A couple of weeks ago a co-worker emailed me the following.
Hettie, last week we asked you whether you think moving our database to remote from local would decrease performance. And you told us, that you do not think we will experience any visible performance degradation. Well, we did move our database yesterday, and performance decreased about six times. And looking at our monitoring tool we can see that all the slowness is on the database side. Is this something you would expect?
I told him: no, definitely not! Let me stop by tomorrow, and we shall see, what’s going on.
When I stopped by with the intension to run parallel tests on both databases, I found out, that they removed all the data volume from the local db, andthereby it became impossible to compare performance. Meanwhile I’ve asked, whether I can see parameter file.
Turned out, they didn’t have access to it, and then I did my little investigation to find out, who has, and who actually installed this DB, and whether he or she changed any parameters. And guess what I found out? I found that the work_mem parameter in their new remote database was set at default – 1 MB, when in should have being at least 128 MB! Then they contacted the person, who could change this parameter and the problem was solved.
Why I am writing about this? Because it is a typical example of “we see what we want to see”. My coworker was so expecting the problems when switching to remote database, and he attributed the problem to this fact without any hesitations… To be honest, I often have a similar problem – looking for the issues, which I expect to see, not the real ones. And we all do – and very often.