Category Archives: Systems

The joy of team work!

Last week I had to work a lot! I mean, A LOT! But work-wise that was one of the happiest weeks, and I wanted to share with everybody why it felt so happy.

Those of you who worked with me before know, that my favorite activity is working with  applications, because optimizing for applications is way more difficult and way more fun than optimizing reports. Yes, you can impress everybody, including yourself, reducing the report run time from one hour to one minute. But how much cooler is it to reduce the page load time from 30 sec to 0.1 sec?!  Especially when you have a power not only to write the best queries ever, but also to design the data model “the right way”.

When you do the application database work, the most critical part is to work in close contact with the  application developers. And depending on what kind of team you are lucky or unlucky to have, it may be the best or the worst part of your professional experience.

My IT team here in Braviant is one of the best I ever worked with, which was proved one more time last week. The most difficult part has always been connecting the db work and the app work, like: I’ve selected all this data for you, can you read it from the output I am providing? Or: we can give you all input parameters that way, can you process them? Our app developers have already made a huge step “in my direction” agreeing not to use ORM, but to read the output of the database function. Next step – we hit the  wall exactly where I expected. I’ve spent a half of Saturday writing my code, so that the app developers could start using in Monday morning… and now they are saying they can’t process correctly the embedded record sets! I’ve heard that many times before, and each time in a couple of hours I would hear: Hettie, there is no way! Let’s do it “the old way”, we know how… That time, however my team kept trying to find a solution, and watching these efforts made me to start thinking how I can change the output on my side. After several iterations going back and forward, we came up with a pretty neat way to return the records, which could be used right away, and even a better approach, which, however would require more work from me, and could not be done on the spot.

And you know – I totally understand, when people hate rewriting one piece of code multiple times, which makes me appreciate even more the willingness to rewrite later, when I will come up with the more automated solution from my side…

I have already written a lot in this post, and I am not sure whether it all make sense, but let me try to summarize.  I loved that everybody were willing to compromise, to make adjustments, that there were no “just because” statements, that the whole team was focused on the goal to build the application right from the very beginning, so that we won’t need to worry about performance six months down the road.

Hope it will continue that way.  Except of me working on Saturdays part 🙂

 

Advertisements

Leave a comment

Filed under People, Systems, Team and teamwork

I am going to be a panelist at the ChicagoSC event

This post is mostly for my Chicago colleagues, but who knows? May be somebody from nearby could also come.

Please check out this event: Panel: ORM’s and Data Management. I will be one of for panelists, and the topic of ORM and ORIM is near and dear to my heart, as all of you know! So I am very excited about this event and honored to be a part of this panel. Please try to  attend :)!

Leave a comment

Filed under events, Systems, talks

Bi-temporal foreign keys support – reloaded

Since I’ve started my new job at the Braviant Holding, I could not find time to continue working on our bi-temporal project. And it was really sad, but to be honest I didn’t feel like I have a single brain cell left to do anything except work.

And I cant’ tell that my today’s life is less stressful than it was for the past four months, but I for some reason about a week ago I’ve resumed that work. It was very difficult to recover details of what I was doing  almost six months ago, and why I didn’t  finish some functions, and so on; especially because I’ve move to a new laptop in between, so all of the intermediate testing, “not worth saving in the git repo” was also gone…

Long story short – today I’ve completed the generator of the functions, which will be used validate bi-temporal foreign key constraints. Than was one of the most complicated pieces of code I’ve written for this project.  But although this is the most difficult part, it’s not the only thing which is necessary for full bi-temporal integrity constraint support. The good par is, that I know exactly what should be done!

Leave a comment

Filed under research, Systems

How I learned to love tests: both using and writing ones – part 1

My fellow database developers, let’s be honest: we do not like writing tests. We are not application developers. We do not understand the test-driven development: at the end of the day, how we can figure out what should be the outcome of our functions and stored procedures, when we do not know what data they will be working with?!  When our managers tell us, we should generate the test data, we think its’ the most ridiculous thing in the world, because if we create some data, obviously the results of the testing will be favorable!

I understand, that not necessarily each and single database developer goes to that extreme, but… pretty close. And I will be the first to admit being guilty with the similar attitude. I only believed in testing on a “copy of the real data”, sort of A/B testing, which is important, but not the only thing to be tested.

Especially these days, when the data structure is not “almost always static”, when the changes the application DB are not a rare catastrophe, but a part of normal life of the application. At a minimum you want to have tests which show you that if you change”something” in the database structure, other “something” won’t break. We need those tests. But… it is so boring to write them! It slows our development process sooooo much! Especially, you know, when you have this big project to complete, and each and every half an hour matters!

At least that’s what I was thinking for the past two months working hard to bring our new Data Warehouse live. And promising to myself, I will write the tests… later :).

But my former co-worker, my forever-mentor, current consultant for my company – Chad – have written his test for “his”part  – which is a part of our system, which is responsible for “taking” to the third-party databases. So… on the night of massive changes on the said third-party database, which were not properly communicated in advance, when some parts of my processes started to fail… and when I fixed the data structures to match the new ones… the tests started to fail!

I was not happy :). Not happy at all. I was thinking – why?  Why I have to sit and fix these tests at 11-30 PM?! But guess what. It took me only 45 minutes to fix each and single test, which was touched by the change, and to validate the new data structures. And I was done before the midnight. And guess, how long it took other people to have their parts of the system updated and running with no issues? Almost the whole night, and almost the whole next day! You might laugh at the next statement, but here is it anyway: at that moment I felt very much protected by these tests.

And that’s what the tests are for, aren’t they?!

Leave a comment

Filed under Development and testing, Systems, Team and teamwork

What I was doing for the past month, and why I was so busy

I’ve just looked at my stats and realized, that I haven’t being posting anything for an entire month (till yesterday :))! In my defense – I didn’t have a single day off for the past 3 weeks, so this weekend was the first time since the beginning of June when I’ve started to get back to normal, back to life. More or less:)

I’ve spent the whole month of June building a new Data Warehouse for my company. I had a pretty good idea what I want to build, but the timing was really-really tight. I know – not only theoretically, but from my experience, that no matter how carefully you plan, there will be unplanned things which will slow you down, and that basically you need to plan for unplanned 🙂 – but it’s always hard. Especially when you know exactly, what you want to do, and you feel really frustrated when you are limited by the speed with which you can type, and by the time it takes to recompile a function or to execute a query.

That was the first time in my professional career, that I had to build the whole system from scratch, and to build it exactly the way I wanted. Ironically (or, may be not!) it was also the first time I had to sacrifice the “purity” to “what business needs”. No, do not take me wrong, actually, the at the end the business needs things to be done right, but I always hated when I was told: we need to build this thing fast now, and later we will optimize – make it right, etc. I used to always scream, that “later” will never happen, and now I made a conscious decision myself to do exactly that! And I do not regret, because now it’s me who has to control, that “later” will actually happen.

Continue reading

Leave a comment

Filed under Companies, People, Systems, Workplace

The database developers: who are they and what they have to do with “development”

Interesting question to ask in this blog, right? But recently I’ve being involved in lot’s of conversations regarding whether this profession even exists.

This is actually quite interesting, because the database development started to be a separate profession quite early in my career. Perhaps in the mid 1990s there was still no clear distinction between the database developer and the DBA, but 2000 job search clearly indicated that these two are distinct (although I’ve applied for both). In a couple of years the “Data Architect” became a new thing, and I’ve started to apply for all three (or rather by that time it was more like jobs chasing me, not the other way around).

What was never a question since 1980s, that “programmers” and “database people whatever you call them” are two separate qualifications (although some people can do both).

There is a common belief, that “small companies” like to hire people who can do both, but I remember how I was hired first time to work in the US. The very small startup initially hired a friend of my friend, who could do it all, but after just several months they realized, that they need to hire somebody who specifically knows what are the databases and what they are for.  And I think that this idea goes along with the broader tendency of professionals becoming more and more specialized, and with the general trend (yes, I know, there are always exceptions) that people who specialize in a certain area become more skillful in this specific area (yea, I know it’s not always the case, I can think about many counter – examples right now, when I type this:)).

All this might seem  contrary to what I am doing these days and what I am advocating for. I am a part of the Software Engineering Department, and I believe firmly (more than ever before:)) that my place is there. But I distinguish clearly between actually doing it all and  understanding what other parties are doing (and yes, sometimes you can do without understanding :)).

The main reason why I wanted to write on this subject one more time, however, is not work-related. I was in process of writing a blog post for my company blog about why the fact that our paper was accepted for ICDE 2016 is so-so-so cool. And it’s not only about the fact, that this conference is so selective, and that that’s a great honor and just amazing; but also because it is clear from the reviews, that it is an important step in recognizing that what we are doing is indeed optimization.

To learn, how non-obvious is it, just ask Karthik Ramachandra, how many times his and his team’s work was not recognized because “it’s not about the databases”. Our work is even more on the application side, than Dbridge project, but that’s where the coolest and the most powerful optimization happens. That’s were we can really improve the user’s experience.

When I’ve received a notification that our paper is accepted, first I was just happy and thought about it mostly as my personal achievement, but now I think that recognizing the direction we are working in is even more important – we can write more papers on that subject :). So although I never submitted a workshop proposal which I though about two years ago, I still think I did my share.

Back to where I’ve started. Just look at our paper: people have been asking me who is the main author, but we do not really have one! (well, I did most of nagging and planning, but not of the actual writing). It’s definitely app-plus-db paper, but all three of us have our own piece of work. Actually we are contemplating to make our presentation this way – so that all three of us could present their parts.

 

 

Leave a comment

Filed under events, Systems

The other day I had another conversation about the business logic with one of our new managers.The good thing – since we were writing another paper, I’ve just reviewed my “split logic” principles and explained it to two of my other co-workers. So I was ready to explain it again.

But I keep wondering, why people are so scared to have “a business logic in the database”? At the end of the day we have multiple applications accessing one database. The database is “more permanent” than the app. The same data can’t be processed using “two different logics”. Isn’t it quite natural, that the business logic actually should be reflected in the database?

Another thing is the fear that the code, which is stored in the db in form of  functions, is somewhat different, that just “a code”. Which leads to another scare – that when something is stored in a database, it becomes a black box. I am not sure how to fight those phobias, but I am pretty sure that they do us a disservice, because those are phycological, not logical, and even more not technical reasons to reject the better performing solution… The more I think about it, the more I agree with the statement, that impedance mismatch os more cultural, than anything else…

Leave a comment

Filed under Systems