Occasionally, I have a day where things go (reasonably) well coding wise, until the very end of the day.
I was in the middle of an appointment yesterday, when my phone started blowing up. Calls, emails, text messages all about the same thing: one of the changes I made to some code during the afternoon was causing problems on our production server. Really bad problems. By that, I mean that no one could connect to the database bad problems. As lot of our clients are physicians and are on the system outside of standard business hours, it was a big problem.
By the time my appointment was over , I was fairly (read: extremely) anxious about the problem, but there wasn’t much I could do without my computer. Fortunately, a single call to a static method (for those of you who aren’t coders, that’s one line of code that tells another piece of code to run) was responsible, so I texted back that it should be commented out for the time being.
You see, the developers of PHP (the programming language we use at work) decided to deprecate (kind of like no longer support) the mysql library, which is what all of our code uses to talk to the database. Instead, the PHP folks implemented the mysqli library. I’m guessing that they have some good reasons for this. Fortunately, the two libraries can work side by side in code, so I started implementing the new library in some code that I’m working on. Most of it isn’t accessible by anyone other than myself and my colleagues, so it isn’t heavily used, but there was one piece that was called Every Time Someone Saved A Web Form! Pushing that code out to our production server caused an overload of connections, resulting in the database connection error “Too many connections.” As someone responding to my StackOverflow question about this pointed out, that mean’s there are too many connections.
Each time I would use that piece of code, 3 new connections were formed to the database. No wonder there were too many connections after a couple hours.
I spent (nearly) all of today debugging this problem. I tried closing my connections to the database and freeing up result sets so they wouldn’t hog memory/ do anything bad. This made the problem less bad, in that new connections are created sometimes, not all the time. Still not ideal, but it may be something we can work with.
It appears that there is some funny interplay between database connections and the Apache web server we use, where the different server connections only know about their own database connections (and not those of other server connections).
For those of you who don’t spend much time looking at code, suffice it to say that this was frustrating. I’ve never dealt with a web infrastructure problem before, and this tells me that it may be time to learn. What appeared to be a code problem is a code + infrastructure issue. For now, at least.
And if anyone has ideas on why things go haywire when you’re not available to deal with them, that would be good to know too…
