Category Archives: Systems

If we need this data for reporting purposes only…

I always thought I have a well established opinion on separating application needs and reporting need. So whenever I was asked to add some attribute to the production system, which is “necessary for reporting”, I would always say: we will build this dada in the reporting system.

So first I could not believe what I just said, when I’ve said something to the effect “I will create this table, which application does not need, and I will use it to record data, which we need for reporting. Literally, after I’ve sadid this I thought – what in the world I am saying?! But then I was very positive that we indeed need this data to be recorded, and I’ve been keeping thinking about it…

And then I’ve realized, that the “reporting” I was referring to is not really a “reporting”. We just historically use the reporting system to go for this information, but in reality, those ones are not analytical reports, but operational reports, which raise alerts when some processing errors occur – mostly in the daily batch jobs. And then I started to think – probably we should treat those operational reports differently? They are not processing big data volumes, they are actually exception reports, and may be we even need to create an “exception processing service” in our production system?

Yes, just to prove, that I am right, of cause 🙂

and data, which is required for reporting. I think it makes lots of sense –

Advertisements

Leave a comment

Filed under Data management, Systems

The best things from 2017 and what I am looking forward to in 2018

When I am talking to people about the year 2017, and what was important, I would repeat over and over: everything I was striving for during my 30+ years of professional career had happened. All wishes have come true. Which would inevitably lead to the question: well, Hettie, what are you going to next then?

And there are plenty and plenty of things I want to do, but first I wanted to reflect on 2017 one more time. I’ve already listed multiple times all our technical accomplishments, all the wonderful things I was able to implement in a real production environment.  But when I look back at what was the best, it is definitely working with our new tech team. As I am reiterating over and over again, “the database is a service”, and whatever we are doing inside the database, can only make any impact, if our results can be utilized by “somebody” – by our end users.

And most of the time our end users are  application developers.  I can’t imagine any serious database development without continuous interaction with the rest of the development team. For years my “golden standard” for the teamwork has been my work for New York Department of Education, where I had a full support of the tech leadership, which helped me to defend my approaches and produce the results I am still happy about.

But what has being going on in the past several months is even better. The way we discuss the user stories. The way we make decisions on what should go into the application, and what – into the database. The way we debug. The way we discuss, what is critical, and what’s not. The willingness of each of the team members “to go extra mile”, to adjust in order to make others job easier. Basically, whatever I could imagine for the teamwork being perfect, is there. 🙂

Being in this industry for 34 years, I know that nothing lasts forever:). I know that every tea,, every organization evolve, and that nothing is granted forever. But I also learned to be thankful for what’s going on right now, and enjoy the moment.

Looking forward to 2018 – I hope that our team will continue to be the most amazing team ever. I am looking forward for the stress -test of our new applications with high data volumes. After all, I was designing the data storage and data access to be super-scalable, and I hope that it will work this way.

I am looking  forward to use the performance data we’ll obtain to improve our bitemporal library, and I already have some ideas of what I exactly I want to do. Actually, I have new technical ideas on almost everything I’ve developed through the last year. I still didn’t get notifications on whether any of the papers I’ve submitted for the next year conferences were accepted,  but I know that both are very good papers :), so even if none will get accepted… I will try for another conference!

 

 

 

 

Leave a comment

Filed under Development and testing, SQL, Systems, Team and teamwork

The joy of team work!

Last week I had to work a lot! I mean, A LOT! But work-wise that was one of the happiest weeks, and I wanted to share with everybody why it felt so happy.

Those of you who worked with me before know, that my favorite activity is working with  applications, because optimizing for applications is way more difficult and way more fun than optimizing reports. Yes, you can impress everybody, including yourself, reducing the report run time from one hour to one minute. But how much cooler is it to reduce the page load time from 30 sec to 0.1 sec?!  Especially when you have a power not only to write the best queries ever, but also to design the data model “the right way”.

When you do the application database work, the most critical part is to work in close contact with the  application developers. And depending on what kind of team you are lucky or unlucky to have, it may be the best or the worst part of your professional experience.

My IT team here in Braviant is one of the best I ever worked with, which was proved one more time last week. The most difficult part has always been connecting the db work and the app work, like: I’ve selected all this data for you, can you read it from the output I am providing? Or: we can give you all input parameters that way, can you process them? Our app developers have already made a huge step “in my direction” agreeing not to use ORM, but to read the output of the database function. Next step – we hit the  wall exactly where I expected. I’ve spent a half of Saturday writing my code, so that the app developers could start using in Monday morning… and now they are saying they can’t process correctly the embedded record sets! I’ve heard that many times before, and each time in a couple of hours I would hear: Hettie, there is no way! Let’s do it “the old way”, we know how… That time, however my team kept trying to find a solution, and watching these efforts made me to start thinking how I can change the output on my side. After several iterations going back and forward, we came up with a pretty neat way to return the records, which could be used right away, and even a better approach, which, however would require more work from me, and could not be done on the spot.

And you know – I totally understand, when people hate rewriting one piece of code multiple times, which makes me appreciate even more the willingness to rewrite later, when I will come up with the more automated solution from my side…

I have already written a lot in this post, and I am not sure whether it all make sense, but let me try to summarize.  I loved that everybody were willing to compromise, to make adjustments, that there were no “just because” statements, that the whole team was focused on the goal to build the application right from the very beginning, so that we won’t need to worry about performance six months down the road.

Hope it will continue that way.  Except of me working on Saturdays part 🙂

 

Leave a comment

Filed under People, Systems, Team and teamwork

I am going to be a panelist at the ChicagoSC event

This post is mostly for my Chicago colleagues, but who knows? May be somebody from nearby could also come.

Please check out this event: Panel: ORM’s and Data Management. I will be one of for panelists, and the topic of ORM and ORIM is near and dear to my heart, as all of you know! So I am very excited about this event and honored to be a part of this panel. Please try to  attend :)!

Leave a comment

Filed under events, Systems, talks

Bi-temporal foreign keys support – reloaded

Since I’ve started my new job at the Braviant Holding, I could not find time to continue working on our bi-temporal project. And it was really sad, but to be honest I didn’t feel like I have a single brain cell left to do anything except work.

And I cant’ tell that my today’s life is less stressful than it was for the past four months, but I for some reason about a week ago I’ve resumed that work. It was very difficult to recover details of what I was doing  almost six months ago, and why I didn’t  finish some functions, and so on; especially because I’ve move to a new laptop in between, so all of the intermediate testing, “not worth saving in the git repo” was also gone…

Long story short – today I’ve completed the generator of the functions, which will be used validate bi-temporal foreign key constraints. Than was one of the most complicated pieces of code I’ve written for this project.  But although this is the most difficult part, it’s not the only thing which is necessary for full bi-temporal integrity constraint support. The good par is, that I know exactly what should be done!

Leave a comment

Filed under research, Systems

How I learned to love tests: both using and writing ones – part 1

My fellow database developers, let’s be honest: we do not like writing tests. We are not application developers. We do not understand the test-driven development: at the end of the day, how we can figure out what should be the outcome of our functions and stored procedures, when we do not know what data they will be working with?!  When our managers tell us, we should generate the test data, we think its’ the most ridiculous thing in the world, because if we create some data, obviously the results of the testing will be favorable!

I understand, that not necessarily each and single database developer goes to that extreme, but… pretty close. And I will be the first to admit being guilty with the similar attitude. I only believed in testing on a “copy of the real data”, sort of A/B testing, which is important, but not the only thing to be tested.

Especially these days, when the data structure is not “almost always static”, when the changes the application DB are not a rare catastrophe, but a part of normal life of the application. At a minimum you want to have tests which show you that if you change”something” in the database structure, other “something” won’t break. We need those tests. But… it is so boring to write them! It slows our development process sooooo much! Especially, you know, when you have this big project to complete, and each and every half an hour matters!

At least that’s what I was thinking for the past two months working hard to bring our new Data Warehouse live. And promising to myself, I will write the tests… later :).

But my former co-worker, my forever-mentor, current consultant for my company – Chad – have written his test for “his”part  – which is a part of our system, which is responsible for “taking” to the third-party databases. So… on the night of massive changes on the said third-party database, which were not properly communicated in advance, when some parts of my processes started to fail… and when I fixed the data structures to match the new ones… the tests started to fail!

I was not happy :). Not happy at all. I was thinking – why?  Why I have to sit and fix these tests at 11-30 PM?! But guess what. It took me only 45 minutes to fix each and single test, which was touched by the change, and to validate the new data structures. And I was done before the midnight. And guess, how long it took other people to have their parts of the system updated and running with no issues? Almost the whole night, and almost the whole next day! You might laugh at the next statement, but here is it anyway: at that moment I felt very much protected by these tests.

And that’s what the tests are for, aren’t they?!

Leave a comment

Filed under Development and testing, Systems, Team and teamwork

What I was doing for the past month, and why I was so busy

I’ve just looked at my stats and realized, that I haven’t being posting anything for an entire month (till yesterday :))! In my defense – I didn’t have a single day off for the past 3 weeks, so this weekend was the first time since the beginning of June when I’ve started to get back to normal, back to life. More or less:)

I’ve spent the whole month of June building a new Data Warehouse for my company. I had a pretty good idea what I want to build, but the timing was really-really tight. I know – not only theoretically, but from my experience, that no matter how carefully you plan, there will be unplanned things which will slow you down, and that basically you need to plan for unplanned 🙂 – but it’s always hard. Especially when you know exactly, what you want to do, and you feel really frustrated when you are limited by the speed with which you can type, and by the time it takes to recompile a function or to execute a query.

That was the first time in my professional career, that I had to build the whole system from scratch, and to build it exactly the way I wanted. Ironically (or, may be not!) it was also the first time I had to sacrifice the “purity” to “what business needs”. No, do not take me wrong, actually, the at the end the business needs things to be done right, but I always hated when I was told: we need to build this thing fast now, and later we will optimize – make it right, etc. I used to always scream, that “later” will never happen, and now I made a conscious decision myself to do exactly that! And I do not regret, because now it’s me who has to control, that “later” will actually happen.

Continue reading

Leave a comment

Filed under Companies, People, Systems, Workplace