Tag Archives: database applications

Looking for New Ways to Bridge the Old Gap: New Ideas After the Conference

Before I went to this conference, I was resentful regarding the fact that the gap between applications and databases will never be closed. Even at the conference focused on both data engineering and software development, there was barely a place for me, and our talk barely got accepted.

I have to admit, I didn’t explore the program much before coming because I had never-ending work crisis, and we had to rework our presentation several times.

But when I took a closer look, I realized that I am way more interested in the SE sessions than in the database sessions. Day four, I could not miss a single moment, and I had several interesting conversations with the speakers.


It turned out that most of them were not even present on day one when I was giving my talk. And they said they would love to come if it won’t be on the first day.

Now I am wondering whether I did it right, never trying to present my work at the SE conferences. On the one hand, I am always saying that my success won’t be possible if I won’t have such an incredible backend team. On the other hand, I routinely say that inefficient programming is all application developers’ fault. That is not true.

One of the talks was about the refactoring techniques, and after the presentation, I asked the speaker whether he ever tried to consider taking into account the factor of accessing a database as a factoring criterion.
He replied that one of his colleagues tried to explore this option, but found it challenging: queries appear to be so entangled, so difficult to extract, that it led to nothing. I told him about my work and suggested that we would love to collaborate if he will find it interesting. He said that he would take a look, and then also mentioned that usually, the database people are not collaborating. He mentioned the lack of constraints and unwillingness to use views. I said that views are horrible because most of the time they decrease performance. But they provide a level of abstraction, he suggested. I replied – the are better ways! He said – well, then give them to us!

This exchange made me think that I am not explaining myself enough when talking about NORM. And if I know that successful implementation depends on cooperation with app developers, I should advocate for them.

Leave a comment

Filed under Development and testing, research, Systems

Finally – Accepted Paper!

Last week I had the first acceptance of my paper to the real Computer Science conference for a very long time (since 2015!). Many things make this acceptance very meaningful for me. 

First, as I said, it’s the first one after four and a half years. 

Second, this is my first acceptance for the scientific conference since I joined Braviant Holdings. When I spoke at ICDE 2016, I was already with Braviant Holding, but the work was done, and the paper was submitted during my Enova tenure.

Third, the topic of the paper is ORIM, which is very difficult to sell to both the academic community and industry alike, and we already had two rejections of the earlier version of this paper.

And the last – this is the paper Boris and I submitted together since I can’t remember how long (I want to say – since 1995 :)). OK, the first accepted paper together:)

We will be presenting at the SOFSEM 2020 conference in January. So yes, this is also my birthday present 🙂

Leave a comment

Filed under events, news, research, talks

Introducing New Term

I’ve been thinking for a while how to name our methodology of data exchange between databases and applications. Several years ago, we came up with the term HDAT – Holistic Database Application Tuning.

This term didn’t get much recognition, partially because I didn’t really like it myself. It is pretty vague and implies that the subject we are talking about is something “nice to have” rather than entirely new technology.

In an attempts to highlight the most important feature of this technology – the fact that it eliminates a need in any ORM – I came up with the term which I am going to introduce today:  NoRM.

The letter ‘o’ stands for both ‘No‘ and ORM, to avoid double ‘o’ and any possible associations with “Noom”:). Also, it highlights the fact that this approach is natural, that this is “a new norm.”

From now on, I am going to call “this JSON thing” a NoRM approach or NoRM technology.  

How do you feel about it? Do you like this term? Will you use it?

1 Comment

Filed under Systems

How to make optimization a priority

One of my usual/perennial rants is that many managers would tell you something like “business needs this functionality, we will optimize later”. And we all know what happens “later” – nothing. The most important thing I love about working at Braviant is that the tech team shares the same values, since I know quite well how often this is not the case, I appreciate it immensely. However, business might still think differently, and what’s good for business… well, that’s what we should do, since we all need to make money.

… After one of our recent releases our DBA noticed that one of the application functions slowed down significantly, specifically instead of executing for about 3 sec, it started to execute for 7-8 seconds. And immediately we all were alarmed. You might wonder – why? The 7 sec execution time is a good time, perfectly acceptable for the end users, especially because this function is executed not so often. Well… Didn’t I just say our tech team agree on priorities? We ll, we believe that the good user experience includes fast response time, and thereby our applications time out on 10 sec. And if a function’s average execution time is over 7 sec, the peak time can easily reach 10 sec!

I had to make a miracle… the sooner the better. Because, as you can imagine, I usually do not write bad queries. Well, most of the time:). Which meant I had to find some unusual optimization.

To tell the truth, I knew right away, why this function starter to perform slower. We added one new field to the distributed query (which was required by business stakeholders, or cause!), and to select this additional field I needed to join one more remote table. And all of a sudden, although all required indexes were on place, the optimizer would choose the full table scan. Of a huge table!

Not much I can do to explain the optimizer that they are wrong (are optimizers male or female, what do you think? 🙂 – they are male in my mother tongue, which explains a lot – they are always sure they know better how to make things work!). So I had to find a way to put this optimizer in such a situation, that there won’t be any way other than to utilize the indexes which were out there. First I thought it will be relatively easy – in all previous cases when a similar issue would occur, I would create a view on the remote node – but this time it didn’t work. I’ve conducted several experiments, and came to the conclusion that the only way to make it work is to implement one new technology, which I’ve played with a couple of months ago, but never implemented in production.

So…
– testing
– making sure it does what I want
– mapping through the foreign data wrapper and making sure it wors
– creating a new version of the function
– testing in lower environments
– QA
– implementing on staging and QA on both our products
– in production on both products

Total execution time of the process described above: around 3 hours.

Result: everybody happy, we’ve got one more groundbreaking technology, which I can guarantee nobody in the world is using (because documentation clearly says it’s impossible:)), and which we will be able to use in many other cases to reduce execution time. And all because we have our priorities right!

P.S. Are you anxious to find out what is this technology? Come to 2Q PgConf in Chicago this December!

4 Comments

Filed under Companies, Data management, Development and testing, SQL

February Chicago PUG – what the conversation was about

It’a almost time for our March PUG, and I never blogged about the February one, I guess now it’s as good as any other, especially because  the March PUG is just several days away, and as usual I hope to attract more people to our next event

As for the February PUG,  I really liked it, even though I managed to completely mess up and accidentally cancelled the meetup! I am still not used to the new Meetup editor. Nevertheless, perhaps it was even better, that I’ve made people to re-confirm their participation at the last minute.

I was presenting our most recent work –  a new framework for efficient communications between a database and a web application. My favorite topic; and I was super -excited to share our success. And I was very glad, that one of our application developers decided to stay for the PUG, because very soon all the questions merged into one big question: what did you do to make it happen? What did it take to change the developers mindset? How did we pull it all together?

And my coworker started to describe, how we did it. And I’ve realized, that I almost forgot about many obstacles, which we had overcome. How many things didn’t work from the very beginning. How many “extra miles” we had to walk in both directions.

Answering the comments on one of my previous posts on that topic: it’s just not so easy to write a “matrix” of decisions which would automatically replace the ORM. Most of the time it’s a customized development. If an app developer would always know that method A involves three joins on a database side, and method B pulls the attributes from the same table as  method C… then probably they won’t start using ORM from the very beginning. But the purpose of ORM is to hide these details!

It’s not easy to do things differently, Especially in a small startup. With all the deadlines, and with clear understanding that there is a potential slowdown in development. But we all tried to do the right thing – as a team. I give a credit to myself for coming up with a framework which at the end of the day is easy and convenient to use in the application. And I give even a bigger credit to the whole team for willingness to work through all the issues toward the best solution.

My fellow chicagoans! If by now you feel sorry you missed the February PUG – please consider coming to the March PUG upcoming Wednesday!  Johnathan Katz from Crunchy Data  will be presenting the talk “An Introduction to Using PostgreSQL with Docker & Kubernetes”. We expect another “bridging the gap” event 🙂

Leave a comment

Filed under Companies, Team and teamwork

The second rejected paper

****Reposting because the previous version didn’t get  shared on LinkedIn****

Object-relational impedance mismatch is by far my favorite research topic, mostly due to the fact that it has a very practical implementation. I would make even stronger statement: the most rewarding optimization is the one when you can reduce the number of SQL statements executed when a web page is rendered and all of a sudden it is loaded 50 times faster (and I mean actually 50 times, not figuratively speaking!).  It always looks like a magic – and I haven’d done anything!

This been said, the ORMs are my worst enemies, and I am always looking for opportunities to promote the better ways of communication between a database and an applications. Most of the time the human factor appears to be more important than the technological challenges, so I always think about these projects as battles.

At Braviant however, first time in my professional career I had nobody to fight about this issue – the app developers were completely on board with my approach since day one. Which allowed us to develop something really cool, and to optimize the interaction between databases and application to the point of absolute perfection. SO, when my husband suggested we’d write a short paper about this project, I had no doubt it will be accepted – because two of my previous papers on the same subject were accepted to the very serious conferences.

Life proved me wrong :), I am not going to name the conference and the workshop, but I have to make some comments about the reviews, so that the level of my frustration can be understood.

One of the reviewers asked: why we think that the number of round trips defines the response time of the web application. Another reviewer asked, whether we tried to use Mongo DB :))). And why we think that (de) serialization of the JSON takes negligible time. And why we think Hibernate is worse.

I think the only valid objection was, that the topic of the paper is not relevant to the workshop topic.  And the latter might explain the whole story.

Several years ago, when I started to attend the database conferences again, after fifteen years of absence, I made an observation that a significant number of the attendees never saw the real applications, and never had deal with performance problems, Fortunately, I’ve also met and got to know some really outstanding researches, whom I admire and feel honored to be aquatinted with, so… I am sure I will find the right place to showcase our work.

And may be it’s time to get back to my old “HDAT” workshop idea,,,

And for my fellow Chicagoans: I will be presenting this work this Tuesday, Feb 13 at the Chicago PUG meetup!

Leave a comment

Filed under research

The second rejected paper: the ORIM again

Object-relational impedance mismatch is by far my favorite research topic, mostly due to the fact that it has a very practical implementation. I would make even stronger statement: the most rewarding optimization is the one when you can reduce the number of SQL statements executed when a web page is rendered and all of a sudden it is loaded 50 times faster (and I mean actually 50 times, not figuratively speaking!).  It always looks like a magic – and I haven’d done anything!

This been said, the ORMs are my worst enemies, and I am always looking for opportunities to promote the better ways of communication between a database and an applications. Most of the time the human factor appears to be more important than the technological challenges, so I always think about these projects as battles.

At Braviant however, first time in my professional career I had nobody to fight about this issue – the app developers were completely on board with my approach since day one. Which allowed us to develop something really cool, and to optimize the interaction between databases and application to the point of absolute perfection. SO, when my husband suggested we’d write a short paper about this project, I had no doubt it will be accepted – because two of my previous papers on the same subject were accepted to the very serious conferences.

Life proved me wrong :), I am not going to name the conference and the workshop, but I have to make some comments about the reviews, so that the level of my frustration can be understood.

One of the reviewers asked: why we think that the number of round trips defines the response time of the web application. Another reviewer asked, whether we tried to use Mongo DB :))). And why we think that (de) serialization of the JSON takes negligible time. And why we think Hibernate is worse.

I think the only valid objection was, that the topic of the paper is not relevant to the workshop topic.  And the latter might explain the whole story.

Several years ago, when I started to attend the database conferences again, after fifteen years of absence, I made an observation that a significant number of the attendees never saw the real applications, and never had deal with performance problems, Fortunately, I’ve also met and got to know some really outstanding researches, whom I admire and feel honored to be aquatinted with, so… I am sure I will find the right place to showcase our work.

And may be it’s time to get back to my old “HDAT” workshop idea,,,

And for my fellow Chicagoans: I will be presenting this work this Tuesday, Feb 13 at the Chicago PUG meetup!

8 Comments

Filed under research