Fixie - Home

Concurrency's a bitch

2009-01-03T23:08:00Z

On this Rails application I’m writing, I have a Person model. When People are displayed to the user, the sort order is more complicated than a name asc>

First are featured People with a picture
Then are featured People without a picture
Then People with pictures and bio
Then People with a picture
Then People with a description
Then People with an address and phone number
And so on

What I ended up doing is adding a “priority” integer attribute to the Person model. On a method that runs before the Person is saved, it figures out what attributes the Person has, and then assigns a priority number to the Person. I gave a priority value of 10 to featured People with a picture, a priority value of 9 to a featured Person without a picture, and so on. Indexed this column and whammo, the sorting works great, just add a “ORDER BY priority DESC” SQL clause to your search statements.

Then it occurred to me when I was in the shower: “What happens if the rules change and People should be ordered differently?” It’s happened twice so far, but it’s no problem since the application hasn’t launched yet. I make the code change to the module (PersonRules, need a better name) and then run:

Person.find(:all).each do |Person|
  Person.priority = PersonRules.generate_priority(Person)
  Person.save!
end

And that works. Almost.

What happens if after you have loaded the Person objects in the find call and a user changes the name of Person while you’re looping through the People?

Prior to Rails 2.2, which I haven’t really upgraded to yet for most of my applications, that change would be overridden¹. Say they update the name of the Person. But the above code loop has already loaded the original Person object into memory and is going to be operating on that data—the priority for the original Person object (with the old name) will be updated and then saved. So the user’s change to the Person name would be overwritten with the old name and new priority.

Ok, so you load the Person with “select for update” (I think in postgresql, don’t know if this syntax is what mysql uses). You do this like:

Person.find(:all).each do |Person|
  transaction do
    Person.lock!
    # now this bit of code is guaranteed to have an exclusive lock on the Person
    # if another bit of code is updating the Person at the same time, the database
    # will hang until it's done
    Person.priority = PersonRules.generate_priority(Person)
    Person.save!
  end
end

This should work, afaik. I don’t like it though:

You have to remember to lock the Person object whenever you do something like this.
If generating the priority took a while, the Person object would be locked for a while.
It adds more code.
How do you test this?

There’s another way to do this, that’s using optimistic locking. You add a lock_version column to your model. That version number is updated every time the Person object is updated. So in the above example, the Person object would be loaded with a lock_version of 10, then when the Person’s name is updated by another user, the lock_version would increment to 11, and then when the priority is generated and the original Person object is saved, ActiveRecord would notice that the lock_version value for the Person has been modified and would throw a nasty exception.

Now that works, except that you have to be aware of the possibility of the exception and rescue it if you don’t want the request to die a fiery death. Now, in most circumstances, not rescuing it would work just fine, since this type of problem wouldn’t happen more often, and it’s better to crash and burn as opposed to have the possibility of bad data. So there’s a case to be made for always adding a lock_version column to all of the models that have the remote possibility of concurrent updates.

But say you wanted something that didn’t require locking the Person object, or rescuing lock_version exceptions all the time. I can think of a couple of options that might work—I have not tried them out yet.

Add a version column to the Person model. When the Person is updated, instead of updating the existing Person database row, insert a new one with the Person data and update the version number. Then when you select the Person, add a “order by version desc”. Problem with this approach is that any other objects that refer to the Person would need to be updated with the new Person id. But it has the advantage of keeping old versions of People around, you could then easily rollback or provide “undo” functionality.

Add a new model: PersonPriority. This would have two columns: person_id and priority. When you want to update the priority of the Person, you don’t have to modify the Person itself, you can update this model. Problem: extra table/model, and doing a sort on the Person would be more complex. (Can you do this indexing with sphinx?)

How do you handle concurrency in your applications? Am I missing any approaches here?

¹ I believe in Rails 2.2, Rails only updates the parts of models that change. So if only the name field of a Person has been changed, when the object is saved, only the name field update is included in the generated SQL update statement.

Setting Up Ubuntu

2008-11-09T15:46:00Z

Although my preferred platform is OS X for personal stuff, I use Ubuntu Linux for work. I do all my work in Ubuntu via VMware Fusion.

Great thing about Ubuntu is apt. It makes setting up the machine for development really easy. OS X has Mac Ports, but imo apt is better. Plus, working in a VM allows you to have multiple OS installs, allows you to backup your setup easily, you can take snapshots, and work on the same OS that you deploy your software to.

Read on to read my notes on how to setup Ubuntu for work.

Download Ubuntu 8.04 Server ISO from from http://mirrors.kernel.org/ubuntu-releases/8.04/ubuntu-8.04.1-server-i386.iso.
Purchase / install VMware Fusion 2
Create the VM, point the install disc to the downloaded Ubuntu ISO.
Configure the VM, give it a lot of hard drive space. It doesn’t use all the hard drive space, but you want it to think that it has access to as much space as it needs. I gave it 50GB.
Start the VM, press enter a bunch of times. Install process should be pretty easy.
I enter “localhost.com” as the hostname. Read on to find out why.
When it asks you what software to install, pick OpenSSH, PostgreSQL, and Samba.
Reboot, login.
Type in: sudo apt-get update This will update the package information on your system.
Type in: sudo apt-get dist-upgrade This will update all the software.
Install the software you work with. For me, I type in the following:

Note the VM’s IP address by running ifconfig -a . For me, the IP address 192.168.206.128.
Add this line to your “host” (i.e. the physical OS) /etc/hosts file: 192.168.206.128 localhost.com . This allows you you to use localhost.com as the name of the machine – you don’t have to remember the IP address.
See if you can ssh into localhost.com from your host machine. Hopefully you can.
Install rubygems: curl http://rubyforge-files.ruby-forum.com/rubygems/rubygems-1.3.1.tgz | tar zxv && cd rubygems-1.3.1 && sudo ruby setup.rb && sudo ln -s /usr/bin/gem1.8 /usr/bin/gem
Install any rubygems you work with. I install haml, capistrano, hoe, mongrel, and webby.
Add yourself as a postgers user: sudo su postgres -c “createuser YOUR_USER_NAME”
Install the apache phusion software: sudo /usr/bin/passenger-install-apache2-module

Enterprise Rails

2008-11-09T15:02:00Z

I recently picked up the book Enterprise Rails by Dan Chak.

it’s an excellent book, and was sorely needed by me. The biggest thing for me was that it explains how to use a good (i.e. PostgreSQL) database effectively and shows how to use materialized views, triggers, and so forth in a Rails application.

It also delves into how to organize a large Rails application, separating your models into modules and then into standalone services. Afaik, no other book shows how to do this sort of stuff.

We’re looking to apply it to this site. (note: not all the tests are counted, since “rake stats” only shows the specs, not the “historical” tests in the tests directory)

+----------------------+-------+-------+---------+---------+-----+-------+
| Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Libraries            |  1301 |  1003 |      24 |     168 |   7 |     3 |
| Models               |  6112 |  4927 |     110 |     757 |   6 |     4 |
| Helpers              |  1836 |  1575 |       0 |     176 |   0 |     6 |
| Controllers          |  4342 |  3696 |      90 |     478 |   5 |     5 |
| Model specs          |  2915 |  2417 |       2 |       9 |   4 |   266 |
| View specs           |   370 |   301 |       0 |       1 |   0 |   299 |
| Controller specs     |  1307 |  1043 |       0 |       4 |   0 |   258 |
| Helper specs         |   432 |   286 |       0 |       0 |   0 |     0 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total                | 18615 | 15248 |     226 |    1593 |   7 |     7 |
+----------------------+-------+-------+---------+---------+-----+-------+
  Code LOC: 11201     Test LOC: 4047     Code to Test Ratio: 1:0.4

How To Dynamically Create ActiveRecord Classes

2008-08-23T15:23:00Z

A question came up on IRC on someone commenting on how it was a pain to have a file for each ActiveRecord class if they were all really simple and basically the same. I cooked up the following code for him.

So, in your environment file (or some other appropriate place), you can create these classes with just one line of code per class. Of course, the method that creates the class can get more complicated ~~- you could pass in options if the classes needed to differ from each other. This is a really simple example of the power of metaprogramming -~~ writing code that writes code.

Quite helpful if you don’t want to repeat yourself all the time.

Advertising for useful things

2008-07-31T15:23:00Z

Software that we as a small web consulting company can not live without:

For keeping track of our tasks, we use Pivotal Tracker. The estimation and prioritizing features are what makes it work well for us. And it’s free for now!
To make and share documents and spreadsheets, Google Docs does the trick.<!~~-more~~->
For email, good old Google Apps.
Tickspot lets us keep track of time spent on each client.
Highrise keeps track of people that we know and what’s new with them.
Engine Yard keeps our websites and software running. I love how they handle so much of the system administration work that I would otherwise have to do.
For remote communication, Adium rocks. It’s a perfect instant messaging client. Well, not quite perfect. It needs to do video.
For editing/programming, Jordan uses TextMate, Joe uses vim.

Starling and Facebook

2008-07-30T15:24:00Z

It’s really important to make sure that your Rails controllers complete their actions quickly. If you have some code in their that takes a while to execute (say, a remote network call), you need to have that stuff done in a different process or thread.

For Daily Caption, we use Starling to queue up messages that need to get sent to Facebook. Another program, creatively called “facebook_daemon” watches the Starling queue.

So, you’ll want to install starling and memcached in the usual fashion.

Then, in config/initializers/starling.rb, put:

ports = { 'production' => "memcache-server:22122", "development" => "localhost:22122", "test" => "localhost:22122" }
STARLING = MemCache.new(ports[RAILS_ENV])

To add stuff to the Starling queue, you can do stuff like:

STARLING.set "key-of-some-sort", "value"

In our FacebookPublisher model, we have the following method:

class FacebookPublisher > Facebooker::Rails::Publisher
  def self.queue action, args
    STARLING.set "facebook_actions", [action, *args]
  end
  # and more

If we need to update Facebook, say, with a updated comment notification, the following code does the trick:

FacebookPublisher.queue(:deliver_notify_caption_comment, self.caption, self)

This adds, to the key “facebook_actions”, an array that looks like this: [:deliver_notify_caption_comment, [self.caption, self]]

You following? At this point, we have that array stored in Starling. Now, it’s up to another program to monitor Starling and pop messages off its queue.

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
require File.dirname(__FILE__) + "/../config/environment" 

class FacebookDaemon
  TIMEOUT = 5

  def self.do_publisher_things args
    puts "Running FB publisher" 
    puts args.inspect
    action = args.first
    args = args[1..-1]
    Timeout.timeout(TIMEOUT) { FacebookPublisher.send action, *args}
  end

  def self.do_profile_updates args
    puts "Doing profile update" 
    puts args.inspect
    user, fb_user, content = args
    Timeout.timeout(TIMEOUT) { fb_user.profile_fbml = content }
    Timeout.timeout(TIMEOUT) { user.update_friends! }
  end

  def self.update
    begin
      while fetch = STARLING.get("facebook_actions")         
        do_publisher_things(fetch) 
      end
      while fetch = STARLING.get("facebook_profile_update") 
        do_profile_updates(fetch)  
      end
    rescue StandardError => e
      puts "**** EXCEPTION!!! fetch = #{ fetch.inspect } *** " 
      puts e
      puts e.backtrace.inspect
    end
  end
end

ActiveRecord::Base.logger = Logger.new STDOUT

options = {
  :app_name => "facebook_daemon",
  :ARGV => ARGV,
  :dir_mode => :normal,
  :dir => File.expand_path(File.dirname(__FILE__) + '/../log'),
  :multiple => false,
  :backtrace => true,
  :monitor => false
}

Daemons.run_proc("facebook_daemon", options) do
  loop do
    FacebookDaemon.update
    sleep 1
  end
end

It’s pretty straightforward stuff, FacebookDaemon.update is called once a second. The update method pops stuff off the Starling queue, and then calls do_publisher_things with whatever array was stored. do_publisher_things knows that the method is the first thing in the array, so calls that method. Hence, FacebookPublisher.deliver_notify_caption_comment (or whatever is on the queue to be called) is called, and Facebook gets sent the call.

If you keep the slow code out of the Rails request cycle, everyone will be happy. I’ve outlined one strategy, but there are more. We also use BackgroundJob to move slow stuff out of the cycle.