asemanfar - a blog about programming

 

Posts tagged with "programming"

HashWithIndifferentAccess + Ruby's Assignment Behavior

January 17, 2010

I recently ran into a weird bug related to HashWithIndifferentAccess that led to my discovery of an interesting Ruby behavior.

The relevant code:

   1  require 'rubygems'; require 'activesupport'
   2  class House
   3    def initialize
   4      @rooms = {}.with_indifferent_access
   5    end
   6  
   7    def room_details_for(room_name)
   8      @rooms[room_name] ||= {}
   9    end
  10  end
  11  
  12  house = House.new
  13  kitchen = house.room_details_for(:kitchen)
  14  kitchen[:size] = "5x10"
  15  
  16  puts kitchen.inspect #=> {:size=>"5x10"}
  17  puts house.room_details_for(:kitchen).inspect #=> {}

After a little debugging and digging through HashWithIndifferentAccess's code, it turns out the problem is a consequence of the way Ruby implements the assignment operator. The return value of an assignment expression is the return value of the right hand side, regardless of what the return value of the assignment method that ends up being called:

   1  obj = Object.new
   2  def obj.foo=(value)
   3    print "foo is called with #{value} but is returning 2, puts receives: "
   4    2
   5  end
   6  
   7  puts (obj.foo = 1) #=> foo is called with 1 but is returning 2, puts receives: 1

That, in conjunction with the fact that HashWithIndifferentAccess converts all hashes set as values into HashWithIndifferentAccess, makes the above bug possible.

   1  hash = {}.with_indifferent_access
   2  hash[:key] = {'inner_key' => 2}
   3  puts hash[:key][:inner_key] #=> 2

Note that ActiveSupport creates a new HashWithIndifferentAccess even if it is already one:

   1  hash = {}.with_indifferent_access
   2  inner_hash = (hash[:key] = {}.with_indifferent_access)
   3  hash[:key][:inner_key] = 2
   4  puts inner_hash.inspect #=> {}
   5  puts hash[:key].inspect #=> {"inner_key"=>2}

There are a couple ways to get around this bug:

  1. Always discard the return value of a HWIA assignment statement unless the right hand side is not a Hash or an Array.
  2. Patch ActiveSupport so that calling with_indifferent_access on a HWIA is a no-op and always use HWIA in favor of Hash when it's a value of another HWIA.
  3. Patch ActiveSupport to not convert values assigned to it. This would break params in Rails unless it was explicitly recursively converted to HWIA.

Oh and by the way, using send doesn't have the same behavior, but is ugly:

   1  obj = Object.new
   2  def obj.foo=(value)
   3    print "foo is called with #{value} but is returning 2, puts receives: "
   4    2
   5  end
   6  
   7  puts (obj.send(:foo=, 1)) #=> foo is called with 1 but is returning 2, puts receives: 2

I hope this helps at least one person while Googling for this issue.

Monitoring Unicorn with Bluepill

November 06, 2009

At Serious Business, we recently switched to using unicorn and bluepill. One of the features we implemented in bluepill was the ability to monitor child processes. This was built with Unicorn's workers in mind. Here's the config file we're using to monitor our Unicorn workers:

   1  Bluepill.application("app_name-production") do |app|
   2    app.process("unicorn") do |process|
   3      process.pid_file = File.join(RAILS_ROOT, 'tmp', 'pids', 'unicorn.pid')
   4      process.working_dir = RAILS_ROOT
   5    
   6      process.start_command = "unicorn -Dc config/unicorn.rb -E production"
   7      process.stop_command = "kill -QUIT {{PID}}"
   8      process.restart_command = "kill -USR2 {{PID}}"
   9    
  10      process.uid = process.gid = 'deploy'
  11    
  12      process.start_grace_time = 8.seconds
  13      process.stop_grace_time = 5.seconds
  14      process.restart_grace_time = 13.seconds
  15    
  16    
  17      process.monitor_children do |child_process|
  18        child_process.stop_command = "kill -QUIT {{PID}}"
  19      
  20        child_process.checks :mem_usage, :every => 10.seconds, :below => 150.megabytes, :times => [3,4], :fires => :stop
  21        child_process.checks :cpu_usage, :every => 10.seconds, :below => 20, :times => [3,4], :fires => :stop
  22      end
  23    end
  24  end

One new feature we recently implemented was the support for working_dir, similar to god's config. There is one important difference though: we also set the PWD environmental variable. This is used to reload the unicorn app with a USR2 signal. In order for Unicorn to properly reload the app with the new current symlink, the env variable needs to be set.

Typically, when you chdir into a symlink'd folder, you are stuck in whatever the symlimk referenced, but if we pass the symlink as an environmental variable, unicorn chdir's into the symlink. The alternative would be to put the full path to the config file in the start command.

The montioring of child processes is still fairly new, as is all of bluepill, so please report any bugs you encounter to the github repos' issue page.

Read more about Bluepill.

Bluepill: a new process monitoring tool

November 04, 2009

Over the past several weeks, two co-workers and I wrote a simple process monitoring tool: bluepill, because it keeps things up. Bluepill replaces the existing process monitoring tool we were using which we had some issues with.

Bluepill is now running on over a dozen of our machines with no memory leaks. It's monitoring several of our services, including long-running rake tasks, background workers, unicorn processes, and pandemic workers. It's been idling at 18MB (see related post for graph).

Here is a sample of the DSL:

   1  Bluepill.application("app_name") do |app|
   2    app.process("process_name") do |process|
   3      process.start_command = "/usr/bin/some_start_command"
   4      process.pid_file = "/tmp/some_pid_file.pid"
   5  
   6      process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3        
   7      process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
   8      process.checks :flapping, :times => 2, :within => 30.seconds, :retry_in => 7.seconds
   9    end
  10  end

Check the readme for detailed installation and usage.

read, fork, and report bugs.

Why We Wrote Bluepill

October 31, 2009

At Serious Business, we use god to monitor our long-running processes (mongrel, background workers, and more recently unicorn). We had a basic god config setup that only checks for memory usage, cpu usage, and request queue length (for mongrels only). God was working fine for us except for one problem: the notorious memory leak. If you use god you probably know it leaks memory in correlation with the number of watches on the system. This became a problem for us when god hadn't been restarted for several days; its memory usage would climb and reach several gigs, causing the machine to swap and eventually lock-up. To prevent lock-ups we needed to manually monitor god (what does that make us?) and restart god daily via cron, gross.

Our frustration with this issue eventually reached a point where we decided to write our own process monitoring tool. Rohith Ravi, Gary Tsang, and I got together one weekend and built a first version of what we've come to call bluepill. We spent the next couple weeks massaging the DSL, expanding feature set, and fixing some bugs we found while using it for our apps. The current feature set is small but is sufficient for the most users:

  • DSL for specifying processes and their respective watches
  • Built-in support for monitoring memory usage and CPU usage
  • Support for custom conditions to watch
  • Daemoniziation of non-daemonized processes
  • Monitoring child processes (especially useful for monitoring unicorn workers)
  • Logging
  • Support for triggers (flapping)

While both bluepill internal and the external interface is heavily influenced by god, we decided do some things differently in bluepill:

  • Written with long-running daemon in mind (read: low resource consumption)
  • Simplicity over flexibility:
  • one process per application; forces separation between multiple apps on the same box
  • simple state machine; does only what it needs to to keep the process up

This past week, we ran a test to see how well Bluepill will do in the wild compared to god, so we set up a basic bluepill config file and the equivalent god config on two identical machines and recorded their memory usage every 30 minutes for just over 4 days.

bluepill vs god memory usage

In addition to the memory leak issue, we sought to improve god in a few other ways: sequential CLI command processing and monitoring child processes:

CLI Command Processing
In god, CLI issued commands are sent to the long-running god daemon which starts a separate thread and returns to the CLI; this led to some race cases when you issue two commands sequentially and expected them to execute in that order (i.e god stop <process_name>; god start <process_name>). This is fixed in bluepill by handling CLI issued commands in a single thread fed by a queue.

Monitoring Child Processes
We recently switched in Unicorn which starts its own long-running child processes to handle requests. So in order to monitor the unicorn workers, we needed to add support for monitoring child processes. Child process monitoring differs from regular process monitoring because bluepill is not responsible for starting them back up and the PID comes from the parent process and not a PID file.

We're going to continue working on it to improve its feature set and iron outs any bugs that we find.

Read the readme for usage information. Read the design file for technical details.
Fork and contribute: http://github.com/arya/bluepill
Report bugs: http://github.com/arya/bluepill/issues