Real Performance Gains? Not In The Database!

When I presented my NORM talk at NYC PG Conf in December, I told the audience that it was the last time I was doing it. Everybody loves this talk, and trust me, I love it as well, but presenting one talk six times feels more than enough. 

However, NORM still is not a new norm, and I am not aware of any successful implementations except for those where I participated at some point in my career. So this year, I resolved to keep promoting NORM and advocate for the development of some NORM-related automation.  

What is NORM? At the moment, NORM does not stand for any tools or libraries. Everything written about NORM so far is gathered in the NORM GitHub repo.

As it is stated in the README,

NORM stands for No ORM framework. NORM is not a PostgreSQL extension, not a library, and not a set of functions. NORM is a methodology that allows object-oriented applications to interact with relational databases directly without involving any ORM. Why do we want to avoid ORM? Because it negatively impacts application performance.

An overall system response time is the most critical factor for the companies financial success. Even a fraction of a second of slowing down response time might cost millions of dollars. The negative impact of ORM comes from the fact that complex objects from the application model are disassembled into atomic objects before communicating with the database, generating too many small queries, which bring down system performance. 

NORM takes a different approach. In addition to an application model and a database model, we introduce a transfer model. The presence of this model is a unique feature of the NORM approach, which makes the mapping symmetrical. We are not trying to build a database model based on an application model, nor are we insisting on creating database objects first. Instead, we call for a contract to be established between the application layer and the database, similar to the way you would see a contract over a RESTful web service.

The contract, or a transfer model, comes in the form of a JSON object. Through this contract, it is possible to simplify the persistence of objects by serializing the objects into JSON payloads that the database can consume. This results in one database call to persist an object regardless of its structure or complexity. Likewise, when retrieving objects, the application can deserialize the result coming back from the database to a model in a single database call. It can also pass additional parameters as a part of the contract to tell the database that it needs additional pieces of the model, similar to an ODATA web service request. Application developers love the simplified implementation of the data access layer on the application side. The fact that NORM uses a contract to determine the inputs and outputs of every call to the database allows application developers to code to the contract and easily mock out any dependencies when testing, as the calls to and from the database will abide by the contract. Thus, after a contract is established, database and application developers can do their part simultaneously and independently from each other. Moreover, different groups of application developers can use the same contract for different projects. On the application side, all modern object-oriented languages have libraries for serializing and deserializing objects. As each new database interaction occurs, it is possible to reuse the same pattern for implementation. This allows application developers to spend more time designing the JSON payload to ensure it meets the current and future needs of the business. Reusing the same pattern of interactions also reduces implementation time, minimizes the likelihood of defects, and allows minimal code changes to impact the entire database access implementation.

In 2021, we included the chapter on NORM in our PostgreSQL Query Optimization book. 

Tom Kincaid (we can never thank him enough for his contribution as a technical reviewer!) published a blog about this chapter. As he points out in this blog, many people remained unconvinced on the benefits of the NORM methodology. Part of the problem is that many readers are unsure what the benefits are and how NORM is different from numerous previous suggestions to leave all the heavy lifting to the database. 

Here, I will list all benefits one more time and highlight the pieces that are still missing.

1. Using JSON as a contract between the database and the application allows passing hierarchies. Yes, we want to send all data which is needed in one roundtrip. However, unless we can do this while preserving the object’s hierarchy, multiple roundtrips for one screen rendering would still be unavoidable.
2. Using JSON as a contract between the database and the application allows application and database developers to work simultaneously. As long as a contract is there, they do not need to wait for each other to complete their portion of work.
3.  Using JSON as a contract between the database and the application makes application developers’ work easier; they do not need to modify their work methods drastically. 

Note that all benefits mentioned above pave the path to adopting NORM by application developers, while the real benefit is drastic performance improvement. In short, unless we force applications to perform bulk operations, we can’t reach real scalability. When we recommend having the size of RAM enough to fit “almost the whole database” into it, we are setting ourselves into a never-ending upgrade race. The true scalability means that when the data size increases, the queries execution time increases not more than logarithmically of the speed of the data growth.

The above characteristic is a feature of SQL queries, not NORM. You do not need NORM to write efficient queries, nor do you need functions or JSONs. Functions and JSONs make these queries easier to use by applications. 

Now, let’s get to the list of what else we need to succeed. What can be done to recruit more developers to use NORM? 

1. The most challenging task is to change people’s mindsets. NORM may be easy to use, but using it the right way may be challenging. A typical example of the wrong usage of NORM. A database developer creates a function that returns a JSON document to populate all fields on the screen. Later, the requirements change, and we need to add one more object to display. Many application developers come to database developers with a request to add one more function rather than a request to modify an existing function. In less obvious cases, an application developer needs to create a new controller, and they would go like, “ok, I have a function which returns that object, and another function that returns another object, so now I only need to request a function which returns this one new object.” The right thing, in this case, would be to create a whole new function.
2. The less challenging task would be to create several automatic generators for user-defined types and functions. I will write a separate blog detailing what needs to be done. Still, one thing is for sure: NORM won’t be adopted by application developers if they have to write everything from scratch. ORMs are easy to use because the SQL code is generated. Postgres should be at a minimum as good as any ORM in this respect.

Why do we want to cater to application developers? Why am I keeping saying that application developers need this and that? The reason is those application developers are our customers. PostgreSQL adds more and more features and extensions every year, but many of these features and extensions remain barely used. PostgreSQL has the best optimizer in the world, but when the application code issues 100,000 short queries per second, even the best optimizer in the world can’t solve this performance problem. The holistic application optimization has been a nobody’s territory for decades, and I believe that it’s time for PostgreSQL to claim this territory.

Just Back From PGCONF NYC 2021

What a fantastic way to start a new job – to spend the first three days attending and presenting at the PGCONF NYC – the first in-person Postgres conference since pre-pandemic! How could things ever be better? 

I felt reunited with the Postgres community on a whole new level. Not like I ever was away from it, but nothing can replace human interactions and face-to-face conversations. The only regret I have about this conference is that I could not be in two or three places at the same time, because there were a lot of excellent talks going in parallel. 

What’s next? The conference is over, and I am about the start the actual work. I feel so empowered, and I am looking forward to immersing myself in new challenges and impacting the Postgres community! 

Job Update: I Joined EDB!

Today, I became a part of the EDB family. I believe it’s the first time in my life I am joining a company where I already know so many people that it feels like I’ve been an honorary member for years!
I am happy to join this extraordinarily talented group of people and looking forward to the most exciting projects coming my way!

Chicago PUG Is Finishing 2021 Strong!

We had another hybrid meetup on Wednesday, and this time, there we five live participants with fourteen more on zoom. It felt even better than in September, though I have to admit that doing a hybrid meetup is harder than doing it all in zoom! I messed up a couple of things, including not starting the video recording, but we will be better next time!

I liked that we had a local speaker; that is something I missed with zoom-only meetups. Somehow it’s easier to get a well-known speaker on zoom than to convince one of the local developers to speak.
I wanted to thank everybody who attended on Wednesday and everybody for continuous participation through the pandemic. We will not meet in December to let everybody focus on the holidays and family, and I hope to see everybody next year, in person or virtually!

Developing Modern Database Applications – A Book Review

Reblogging to share on LinkedIn

The World of Data

Some time ago, I was asked to review a bookDeveloping Modern Database Applications with PostgreSQL. I was happy to submit the review because I really like the book, and I think it fills in many gaps in the technical literature.

I think it happened to many of us: you start something new, you read documentation, and you are still left there with an embarrassing question: OK, so how should I do thisfor real? This book provides enough instructions to spare you from these questions: all the steps are listed, and there are plenty of pictures so that you can be reassured you are doing the right thing. That’s especially important for smaller groups of developers, which often do not have enough expertise in all areas, but still, need to jump-start their development.

Below is my Amazon review.

View original post 502 more words

Developing Modern Database Applications – A Book Review

Some time ago, I was asked to review a book Developing Modern Database Applications with PostgreSQL. I was happy to submit the review because I really like the book, and I think it fills in many gaps in the technical literature.

I think it happened to many of us: you start something new, you read documentation, and you are still left there with an embarrassing question: OK, so how should I do this for real? This book provides enough instructions to spare you from these questions: all the steps are listed, and there are plenty of pictures so that you can be reassured you are doing the right thing. That’s especially important for smaller groups of developers, which often do not have enough expertise in all areas, but still, need to jump-start their development.

Below is my Amazon review.

Is It Worth To Have Live Meetups?

Yesterday, we had the first hybrid meetup, the first not-completely-virtual event since February 2020. It was an experiment, and I had a lot of fears about how it will go. In the end, there were only two people in the room for the “live” part. I suspected it could be the case because in addition to the pandemic, we are now in a new building with tighter security, and people had to comply with lots of new rules. At some point, I thought – is there any good reason to try so hard to make it hybrid? Now I can tell that it really felt differently.

I am not sure why but being in the office while streaming made it feel like everybody was right there. Most speakers complain that it is difficult to get the audience’s feedback when on zoom and that you feel like talking into the dark hole. And I would agree that it is much harder to make a good presentation on zoom. Nevertheless, I felt very different when streaming from the office. I do not know whether other participants felt the difference, but I felt like I talked differently and interacted with people differently. And I really liked it!

I hope that next time, there will be more participants in the live part of the meetup. I think that many people forget how special it feels:)

Let’s Talk About Performance Calibration

Do you know what does performance calibration means? If you are reading this blog post on professional social media, the chance is you know! And if you are not sure about what a performance calibration is, you know what an end-of-year review means.

I can’t imagine anybody could like these end-of-year reviews, and the only reason we put up with them is that we hope that our achievements will be noticed. That means we need to have some objective criteria to compare marketing specialists to accountants.

Although everybody hates writing self-evaluation, it is a necessary evil: there have to be some ways to quantify the quality of each person’s work. And when some objective criteria are in place, a person with ambition would strive to check all boxed and match all requirements for a promotion. But that’s where calibration kicks in. Apparently, it is impossible to have too many employees who performed great – there should be a normal distribution (a Bell Curve) of achievements.

There is a good reason for “not too many top performers.” Most likely, earlier in the year, a company announced the figures for the bonus payoff. Something like “top performers will get 125% of the target bonus, and the average performers would get 100% of the target bonus, and if you didn’t meet some goals, you would be paid less than 100% or nothing at all.” Such an announcement imposes an objective limitation: a certain amount of money can be allocated for bonuses. If there are “too many” top performers, a company can’t meet its obligations (just like the state of Illinois can’t meet its pensions obligations).

However, that obvious consideration is never a part of the announcement. Employees do not know that no matter how hard they work to meet all the requirements and hit all the goals, they still may not qualify for a top bonus because of the evil behavior of the “Cursed Bella.” (Once, I heard a manager calling the Bell’s Curve “the Cursed Bella,” and I liked it:)).

I think that this is unfair on many levels. Employees who didn’t make it are left with questions “what I did wrong” and “how I can make it better.” Managers spend sleepless nights forced to artificially cut the achievements of half of their direct reports. Everybody is upset. And most importantly – people’s performance does not necessarily fit the normal distribution. I remember some periods of my career when I could say: everybody on my team is a top performer. Everybody is exceptionally talented, and everybody contributed their time and effort at the max level.
I think, instead of trying to find the non-existing flaws in somebody’s performance, the company management could announce the projected bonus the following way: “x” percentage of profit would be allocated for bonuses payoff. The entire amount, depending on how many people will be ranked top performers, will be distributed between the top performers and the rest, according to the following formula (there are potentially several different ways to split).
Maybe, I am still hopelessly naive. But I think that this is an honest way to inform employees about what they should expect. Work is not a competitive sport (although many people think that way). The purpose of whatever you are doing is to help your company to achieve its business goals, not competing for the highest bonus. Although ideally, these two should coincide – see the beginning of this post 🙂

Am I still hopelessly idealistic?

Live Postgres Conference is back!

Attention Postgres Community, friends, and co-workers, both current and former! Welcome the comeback of the live Postgres Community conference held Dec 2-3, 2021, in New York! 

Please see the conference page for details.

Several things worth noting:

• The talk submission deadline is September 7. As usual, you do not need to submit the full presentation; you need just a couple of paragraphs to clearly explain what that talk will be about and why everybody should get excited to hear it.
• The hotel discounted price is good till September 4.
• I am on the talks selection committee 🙂

That being said, I am pretty excited to share this news! Please spread the word, submit a proposal and register to attend!

How To Foster Diversity

Some time ago, I had a conversation about supporting diversity in academia. A person I was talking with asked me what concrete and specific actions would help increase diversity and give opportunities to individuals who can’t succeed in the academic world otherwise. For a while, I was thinking about a good answer. But the more I thought, the more I felt I need to answer another question first.

The question is, why do we need diversity and inclusion? Many people genuinely believe that diversity and inclusion are buzz words, that it is “fashionable” to talk about diversity, or even worse, that this concept is invented by democrats, or lefts, or communists, or anybody else who corrupts our youth in the universities – you name it. 

But the truth is that we all – we as a society, we as a country, we as humanity – all need to foster diversity. 

Let me start from a seemingly unrelated episode. Like many others, I came to this country on a work visa. I came because there was a job waiting for me here. There were not many companies that would sponsor a work visa, even with the shortage of qualified candidates. Like many other H1B workers, I was convinced that the only reason a company would sponsor a work visa was an option to pay the immigrant workers less than local candidates. Otherwise – why bother? However, later, when I was already a US citizen, a recruiter for my company mentioned that the company sponsors work visas. I was about to ask – why? when the recruiter continued: that way, we get access to a larger pool of candidates! It was that simple: the more candidates you have to choose from, the better chances of finding a great fit are.

Building a diverse team is not about checking the boxes with race, gender, or ethnic origin. It is about removing boundaries and limits. The person I mentioned at the beginning of that post asked me: aren’t we giving people an unfair advantage? Discrimination is disgusting; there is no excuse for banning women or minorities from attending universities. But after the ban is removed – shouldn’t all the rest be merit-based? Why are we required to maintain a specific proportion of women and minorities on a faculty or a postgrad program? 

But what is merit? How do we measure it? We do not want to measure the skill of passing tests. Nor even a level of knowledge in any specific field. I still remember how my younger son was tested for a gifted program in the second grade. He didn’t make a cut because his reading level was too low, and the reading level was one of the measured skills. But later, his teacher called me because the results of the cognitive skills test were back, and his results were off the charts – literally. He was given a chance to participate in the program. 

Similarly, in the academic environment, if we are looking for talent and potential, for somebody who can contribute, we should not limit our search to those who are good at passing tests. Or the most knowledge. We should look for talent and potential, and those might not be easy to recognize if we won’t give people a chance. 

That is not “reverse discrimination”; it’s allowing talent to thrive. And in the end, everybody will benefit. 

If we understand the goal, it is not difficult to come up with a list of practical actions. 

• Proceeding with an open mind
• Providing extra help and support for people who do not have role models, supporting networks, or any other pre-requisites which we take for granted
• Educate others about the importance of fostering diversity

Please feel free to add to this list!