Optimizing something you can’t control

This is very much  like  finding Pluto! At Braviant, we use several external service providers to perform some business tasks. And then, as I’ve mentioned in one of my presentations about our usage of foreign data wrappers, we need to manage data, when we do not really own the data.

But this time around the task was even more complex, and I’ve spent weeks trying to figure out how to approach it. There is one Really Large Table on the “other” side, and to refresh the Data Mart, we need to select a small subset of records each time, basically “all records starting from the moment we refreshed last time”.

For some reason unknown to me something on the way from “them” to “us” did not work, and we could not push the condition to the external site. No matter what I was selecting, what was really happening (I’ve figured it out by observing the query behavior closely) – the whole table was fetched from the third-party server, and only then the selection criteria was applied.

The problem looked unsolvable, because “everything worked on the other side”. Then I cam up with one crazy idea. I thought: if we can’t push our condition through, may be we can create similar condition on the other side.

So, I’ve asked our service provider tech support, whether they can create a view on their side, which would restrict the size of object, I am selecting from, Note, I’ve asked for just a view, not a materialized view. So it was literally “query is executed locally”. And then I’ve mapped this view to the foreign table, so there was no changes to reporting.

Yes, this view has way more records than I need (it contains “last 24 hours”), while I refresh data every  two hours. However, now I select from way smaller data set, because the view contains only last 24 hours, not the last 2 months!

… and now tell me, which optimizer would be able to execute this kind of optimization?!

Advertisements

Leave a comment

Filed under Data management, SQL

Last Day to submit your talk proposals!

Hey fellow Postgres enthusiasts, especially a Chicago bunch! A friendly reminder, that today is a deadline for the 2Q PGCONF talks submission. If you didn’t submit anything, there is still time, just 1-2 paragraphs highlighting the main idea of  the talk  will be sufficient. Let’s show that Chicago CAN!

Leave a comment

Filed under events, talks

Why I like so much what I am doing

Many years ago, when I was about to graduate from the University, my not-then-husband asked me, what I wanted to do with my professional life: to write “smart” papers about how-everything-should-be, or to do something real? Because it was quite obvious, which answer he had expected at that time, I’ve answered: of cause, the latter one!

But speaking seriously, that was my goal through all my professional life. Yes, I do write the “smart” papers about how things should work, but all these discoveries are of little interest to me until I can make a practical usage out of them, and until I can prove, that what I think is right actually changes things for better.

I like to say, that “a database is a service”. There is nothing else in the world of information technology which is more remote from the end user, than the database internals. Our work manifests itself in a very not-so- straightforward way. And when the the absolutely theoretical approaches which I’ve developed, actually work the best possible way – there is nothing more exciting.

In the system which I am building right now, which is more than just an app, but the whole system, which includes interaction between different online services and the data warehouse(s) I am implementing all the ideas, which has been important for me for most of my professional career.

I am using the bitemporal model I talked so much about through the past two or three years, and it is fascinating to see that things I was hoping will work and have some value to the business, actually produce value!

I work with application developers to bypass the ORM, and to use the output of the database functions for the most efficient communication with the data storage. I did this many times before, but never before I’ve experienced that level of cooperativeness.

I am using the foreign data wrappers in a most extended manner, and literally eliminate the gap between the application databases and the data mart.

Everything I wanted to accomplish in different periods of my professional life – everything is coming together, and I can see that the results are coming out really … how I wanted them to be :).  And I can’t allow it to be different.

Leave a comment

Filed under Data management

Postgres conference in Chicago Nov 9 2017

It is my pleasure to advertise an event, which will be happening in Chicago in November:  2Q PGCONF 2017.

This conference is organized by the 2ndQuadrant, and it will be held in two locations: New York Nov 7-8 and  Chicago Nov 9. Participants can register and/or submit their talks for each of the locations separately.

If you ask me, I think that a one-day conference is great thing. It’s  much more doable, then several days conference, and your manager is way more likely to agree to you being one day away from work, than several :). This being said…

– Please consider participation (please register on the web site)

– Please consider submitting a talk (the deadline in Aug 22!)

– Please help us to find sponsors!

 

 

 

 

Leave a comment

Filed under events, news

Chicago July PUG with Bruce Momjian

I have just one thing to say – it was a great meetup! I was worried, that the even was scheduled not for the best day: it’s the end of July, when people are either heading to vacation, or at least just trying to spend as much time outdoors, as possible. Besides, for some reason nobody likes to meet on Thursdays.

In spite all of the above – the attendance was great, and the audience was really engaged. Do I need to mention that Bruce is the greatest speaker in the Postgres world?!

The presentation was brilliant,  over an hour just passed by, and nobody even asked whether we can have a break 🙂


And now I really hope  this won’t be the last time Bruce is visiting Chicago and talking at PUG!

Leave a comment

Filed under events, People

The joy of team work!

Last week I had to work a lot! I mean, A LOT! But work-wise that was one of the happiest weeks, and I wanted to share with everybody why it felt so happy.

Those of you who worked with me before know, that my favorite activity is working with  applications, because optimizing for applications is way more difficult and way more fun than optimizing reports. Yes, you can impress everybody, including yourself, reducing the report run time from one hour to one minute. But how much cooler is it to reduce the page load time from 30 sec to 0.1 sec?!  Especially when you have a power not only to write the best queries ever, but also to design the data model “the right way”.

When you do the application database work, the most critical part is to work in close contact with the  application developers. And depending on what kind of team you are lucky or unlucky to have, it may be the best or the worst part of your professional experience.

My IT team here in Braviant is one of the best I ever worked with, which was proved one more time last week. The most difficult part has always been connecting the db work and the app work, like: I’ve selected all this data for you, can you read it from the output I am providing? Or: we can give you all input parameters that way, can you process them? Our app developers have already made a huge step “in my direction” agreeing not to use ORM, but to read the output of the database function. Next step – we hit the  wall exactly where I expected. I’ve spent a half of Saturday writing my code, so that the app developers could start using in Monday morning… and now they are saying they can’t process correctly the embedded record sets! I’ve heard that many times before, and each time in a couple of hours I would hear: Hettie, there is no way! Let’s do it “the old way”, we know how… That time, however my team kept trying to find a solution, and watching these efforts made me to start thinking how I can change the output on my side. After several iterations going back and forward, we came up with a pretty neat way to return the records, which could be used right away, and even a better approach, which, however would require more work from me, and could not be done on the spot.

And you know – I totally understand, when people hate rewriting one piece of code multiple times, which makes me appreciate even more the willingness to rewrite later, when I will come up with the more automated solution from my side…

I have already written a lot in this post, and I am not sure whether it all make sense, but let me try to summarize.  I loved that everybody were willing to compromise, to make adjustments, that there were no “just because” statements, that the whole team was focused on the goal to build the application right from the very beginning, so that we won’t need to worry about performance six months down the road.

Hope it will continue that way.  Except of me working on Saturdays part 🙂

 

Leave a comment

Filed under People, Systems, Team and teamwork

Please join us this Thursday for a very special meetup!

Attention Chicago Postgres users, developers, DBAs and everybody who knows what the word “Postgres” means! In the unlikely event you did not hear about it already – this day is coming! Bruce Momjian will be our guest at the July meeting of the Chicago PostgreSQL User Group, and I do not think I need to say anything else! We are just excited that this is finally happening!

Please RSVP, if you are planning to come, and didn’t RSVP yet – we have a new person at the building reception, and I need to give her a guest list! Also, just for this meetup we will extend the time till 9 PM, so that everybody could enjoy the conversation.

Hope to see you there!

Leave a comment

Filed under events, People, talks