asemanfar - a blog about programming

In our cluster, we run many mongrels on several app servers all behind a load balancer. In order to get an idea of how each app server and its mongrels are doing, we use rtomayko's mongrel_proctitle gem. This gem sets up a mongrel handler that extracts request information from the client's request and sets it as the process title so when you do a ps aux or run top you can see each mongrel's queue length and request info:

   1  mongrel_rails [8000/1/4]: handling 127.0.0.1: GET /users
   2  # that's [port/queue length/requests handled]: handling client ip: request

Problem

Last week, we were having some misbehaving mongrels that would lock up and essentially stop serving requests. We'd look at the process titles and see something like this:

   1  mongrel rails [8021/14/6123]: handling 127.0.0.1: GET /status

We have an action designated to be a lightweight health check to let the load balancer know what's up and according to this process title, this action was locking up the mongrel and growing to a queue length of around 14. This didn't make sense, why is the lightest weight action locking up the mongrel?

Solution

With a little digging, I found that the mongrel_proctitle gem is broken. More precisely, it is only accurate when there is a single request being handled and none queued. The handler was setting the process title as soon as it received the request, which then led that client's thread to the next handler, which happens to be the synchronized Rails handler. So the request the process title shows is the most recent received request and not the current request being handled.

With a few small modifications, we now have a more accurate process title that also shows the rest of the queue (up to a character limit) as a comma-separated list:

   1  mongrel rails [8021/2/6123]: handling 127.0.0.1: GET /users, 127.0.0.1: GET /status

The updated version is available on github and also as a gem at gems.github.com as arya-mongrel_proctitle:

   1  gem sources -a http://gems.github.com
   2  sudo gem install arya-mongrel_proctitle

Note, as described in the comments in the code, it's still not exact. There is at least one race case where the entire list won't be ordered accurately, but it's unlikely to occur and still gives a good idea of what's going on.