asemanfar - a blog about programming

Pandemic is a map-reduce framework. You give it the map, process, and reduce methods and it handles the rest. It's designed to serve requests in real-time, but can also be used for offline tasks.

It's different from the typical map-reduce framework in that it doesn't have a master-worker structure. Every node can map, process, and reduce. It also doesn't have the concept of jobs, everything is a request.

The framework is designed to be as flexible as possible, there is no rigid request format, or API, you can specify it however you want. You can send it http-style headers and a body, you can send it JSON, or you can even just send it a single line and have it do whatever you want. The only requirement is that you write your handler to appropriately act on the request and return the response.

Here is how you use it:

Server

   1  require 'rubygems'
   2  require 'pandemic'
   3  
   4  class Handler < Pandemic::ServerSide::Handler
   5    def process(body)
   6      body.reverse
   7    end
   8  end
   9  
  10  pandemic_server = epidemic!
  11  pandemic_server.handler = Handler.new
  12  pandemic_server.start.join

In this example, the handler doesn't define the map or reduce methods, and the defaults are used. The default for each is as follows:

• map: Send the full request body to every connected node
• process: Return the body (do nothing)
• reduce: Concatenate all the responses

Client

   1  require 'rubygems'
   2  require 'pandemic'
   3  
   4  class TextFlipper
   5    include Pandemize
   6    def flip(str)
   7      pandemic.request(str)
   8    end
   9  end

Config

Both the server and client have config files:

   1  # pandemic_server.yml
   2  servers:
   3    - host1:4000
   4    - host2:4000
   5  response_timeout: 0.5

Each value for the server list is the host:port that a node can bind to. The servers value can be a hash or an array of hashes, but I'll get to that later. The response timeout is how long to wait for responses from nodes before returning to the client.

   1  # pandemic_client.yml
   2  servers:
   3    - host1:4000
   4    - host2:4000
   5  max_connections_per_server: 10
   6  min_connections_per_server: 1
   7  response_timeout: 1

The min/max connections refers to how many connections to each node. If you're using the client in Rails, then just use 1 for both min/max since it's single threaded.

More Config

There are three ways to start a server:

• ruby server.rb -i 0
• ruby server.rb -i machine1hostname
• ruby server.rb -a localhost:4000

The first refers to the index in the servers array:

   1  servers:
   2    - host1:4000 # started with ruby server.rb -i 0
   3    - host2:4000 # started with ruby server.rb -i 1

The second refers to the index in the servers hash. This can be particularly useful if you use the hostname as the key.

   1  servers:
   2    machine1: host1:4000 # started with ruby server.rb -i machine1
   3    machine2: host2:4000 # started with ruby server.rb -i machine2

The third is to specify the host and port explicitly. Ensure that the host and port you specify is actually in the config otherwise the other nodes won't be able to communicate with it.

You can also set node-specific configuration options.

   1  servers:
   2    - host1:4000:
   3        database: pandemic_node_1
   4        host: localhost
   5        username: foobar
   6        password: f00bar
   7    - host2:4000:
   8        database: pandemic_node_2
   9        host: localhost
  10        username: fizzbuzz
  11        password: f1zzbuzz

And you can access these additional options using config.get(keys) in your handler:

   1  class Handler < Pandemic::ServerSide::Handler
   2    def initialize
   3      @dbh = Mysql.real_connect(*config.get('host', 'username',
   4                                            'password', 'database')) 
   5    end
   6  end

Code: github repository

Install:
sudo gem -a http://gems.github.com
sudo gem install arya-pandemic

In our cluster, we run many mongrels on several app servers all behind a load balancer. In order to get an idea of how each app server and its mongrels are doing, we use rtomayko's mongrel_proctitle gem. This gem sets up a mongrel handler that extracts request information from the client's request and sets it as the process title so when you do a ps aux or run top you can see each mongrel's queue length and request info:

   1  mongrel_rails [8000/1/4]: handling 127.0.0.1: GET /users
   2  # that's [port/queue length/requests handled]: handling client ip: request

Problem

Last week, we were having some misbehaving mongrels that would lock up and essentially stop serving requests. We'd look at the process titles and see something like this:

   1  mongrel rails [8021/14/6123]: handling 127.0.0.1: GET /status

We have an action designated to be a lightweight health check to let the load balancer know what's up and according to this process title, this action was locking up the mongrel and growing to a queue length of around 14. This didn't make sense, why is the lightest weight action locking up the mongrel?

Solution

With a little digging, I found that the mongrel_proctitle gem is broken. More precisely, it is only accurate when there is a single request being handled and none queued. The handler was setting the process title as soon as it received the request, which then led that client's thread to the next handler, which happens to be the synchronized Rails handler. So the request the process title shows is the most recent received request and not the current request being handled.

With a few small modifications, we now have a more accurate process title that also shows the rest of the queue (up to a character limit) as a comma-separated list:

   1  mongrel rails [8021/2/6123]: handling 127.0.0.1: GET /users, 127.0.0.1: GET /status

The updated version is available on github and also as a gem at gems.github.com as arya-mongrel_proctitle:

   1  gem sources -a http://gems.github.com
   2  sudo gem install arya-mongrel_proctitle

Note, as described in the comments in the code, it's still not exact. There is at least one race case where the entire list won't be ordered accurately, but it's unlikely to occur and still gives a good idea of what's going on.

So I've been cleaning up my blog's code base and decided to switch to unobtrusive javascript. I've heard many reasons advocating its use but for some reason, they never stuck. Until a couple of weeks ago when I worked on a Railsrumble project with a couple of buddies of mine, one of which was a big advocate of unobtrusive javascript using LowPro. Actually using unobtrusive javascript turned out to be much more gratifying that I anticipated. It satisfied some inner OCD tendencies of mine that a lot of programmers share and it made developing javascript'd UIs much easier to develop and much much easier to maintain.

Looking back, I don't think know why I chose to use the Rails javascript helpers or RJS; I like writing javascript, especially when there are great frameworks out there like LowPro and Prototype. The fact that the Rails javascript helpers generate inline javascript, which sometimes turns out being longer than a simple one-liner, is kind of gross. As for RJS, the fact that it's generates code that contains both the data and the logic makes OCD baby Jesus cry.

The separation of HTML (markup), CSS (presentation), Javascript (interactivity) is almost as awesome as the separation of church and state...oh wait, crap.

If you haven't already switched, do it now!

Just as a little background, I'm working on adding the concept of rounds to an application. A "round" is simply an organization unit with a start and end date. Each round has_many :events, and each event has an end date, defined as its moment of expiration. The validation logic for these two models is that an event's end date must be between the round's start/end dates and that rounds cannot overlap. This isn't an issue at all with things like fixtures since that's all manually specified. But I'm using FactoryGirl for generating my test data so things get a little tricky.

Read more...