Archive for the ‘Uncategorized’ Category

Ruby Daemons and Vendoring

Wednesday, April 8th, 2009

In the past in order to gather server side usage stats, we have had a procession of different methods to track and record this data in a way that was outside the main app. The reason for this separation is to ensure that search results on TouchLocal were as quick as possible while still recording the details. The first attempt was to use Stomp, which was interesting but we had problems with stability. After that, and for a long time, we had a backend Merb application that was sent the tracking information from the webserver and returned immediately. While this was quick and pretty fast, it had the downside of the HTTP request from the main site which would time out and make the site crawl to a halt if the backend processes were not responding for whetever reason.

As a result, we went on a third rewrite effort. This time we went back to basics, and decided to use log4r to output a special log format that would be parsed and inserted into the tracking tables aynchronously. This would imply a delay, but only in the vicinity of a few minutes. The main design problems were ensuring stability of the processing platform and also the ability to retry files that might have been missed or errored. The repeatability was achieved by using a combination of a hash of the tracking data + the timestamp + some random information, and a (large) table that tracks this timestamps and hash combination (given that hashes can sometimes collide, it makes sense to add the timestamp as a factor).

The backend process to parse and record the information from this log file format was written in Ruby of course. In order to achieve stability I decided to use the Ruby Daemons gem. This handles PID file management and lots of other neat things for writing a daemon so the basics of long-running processes were not my ‘problem’ as such. So that these Ruby processes could scale up, I ensured as I was writing it that it would be aware of the potential error conditions of multiple processes, such as one process moving a file when another was looking for it (race conditions, etc). While the Daemons gem uses the fork() implementation on *nix for the standard headless run mode, it also support a non-background run mode which works on Windows. I also chose to use ActiveRecord, reusing the AR models from the Merb application.

One of my personal goals was to ensure the daemon was as self-contained as possible. For me, this meant that I wanted the Sysadmin to be able to check it out and run the start command and have it work on a standard base Ruby installation.

Here’s the initialisation code for the Daemon:

################
### REQUIRES ###
################

# This loads gems in vendor/gems (abstracted out so it can be used in rake tasks that haven't loaded ENV yet)
require "lib/local_gem_loader"

require 'rubygems'
require 'daemons'
require 'activerecord'
require 'json/pure'

require 'erb'
require 'cgi'

# +require+ all the models
model_path = File.expand_path(File.join(File.dirname(__FILE__), 'models'))
$LOAD_PATH.unshift model_path
Dir.glob(File.join(model_path, '*.rb')) do |file|
  require file
end


# Log4r does not work because the Daemons gem closes all open file descriptors.
#require 'log4r'
#require 'config/logger'
#logger = ::DEFAULT_LOGGER

#################
### CONSTANTS ###
#################

SLEEP_TIME_SECONDS = 5
FILE_NAME = 'tracking_daemon.rb' # name to report as the process

#####################
### CONFIGURATION ###
#####################

database_configuration_file = File.expand_path(File.join(File.dirname(__FILE__), 'config', 'database.yml'))
database_configuration = YAML::load(ERB.new(IO.read(database_configuration_file)).result)

######################
### INITIALISATION ###
######################

# Parse out the RAILS_ENV=production setting
ARGV.each do |arg|
  if arg.include?('=')
    key, val = arg.split('=', 2)
    ENV[key] ||= val
  elsif database_configuration.keys.include?(arg)
    ENV['RAILS_ENV'] ||= arg
  end
end
RAILS_ENV = (ENV['RAILS_ENV'] || "development").dup
puts "Starting #{FILE_NAME} daemon in #{RAILS_ENV} mode"


ActiveRecord::Base.configurations = database_configuration
ActiveRecord::Base.establish_connection RAILS_ENV

root_files_path       = File.expand_path(File.join(File.dirname(__FILE__), 'files'))
incoming_files_path   = File.expand_path(File.join(root_files_path, 'incoming_files'))

options = {
             :multiple   => true,
             :ontop      => false,
             :backtrace  => true,
             :log_output => true,
             :monitor    => true
           }

##############
### DAEMON ###
##############

Daemons.run_proc(FILE_NAME, options) do
  loop do
    # 1. Get the next file to process
    Dir.glob(File.join(incoming_files_path, '*')) do |incoming_file|
    end
    
    #...

    puts "Sleep #{SLEEP_TIME_SECONDS} sec" if RAILS_ENV == "development"
    sleep(SLEEP_TIME_SECONDS)
  end
end

The local_gem_loader is something of my own invention, based on the vendor/gems loader that was introduced in Rails 2. I wrote it initially to enable our (then) Rails 1.2.6 app to have vendored gems. It was very useful here to allow me to meet my desire to have this thing be checked out from SVN and started. Here it is – it’s pretty simple really:

# Load the gems in /vendor/gems
standard_dirs = ['rails', 'plugins']
gems          = Dir[File.join(__FILE__, "vendor/*/**") ]
if gems.any?
  gems.each do |dir|
    next if standard_dirs.include?(File.basename(dir))
    lib = File.join(dir, 'lib')
    $LOAD_PATH.unshift(lib) if File.directory?(lib)
    src = File.join(dir, 'src')
    $LOAD_PATH.unshift(src) if File.directory?(src)
  end
end

After including that line, I was able to vendor all the gems I needed (activerecord, json-pure, and even daemons) in the vendor/gems directory I created. After that, the ./models directory is loaded with the lines

# +require+ all the models
model_path = File.expand_path(File.join(File.dirname(__FILE__), 'models'))
$LOAD_PATH.unshift model_path
Dir.glob(File.join(model_path, '*.rb')) do |file|
  require file
end

Also note that as per the behaviour of how Daemons is designed, as it starts it closes all open file descriptors. While I read this in the documentation, I still tried to integrate Log4r, and spent a very confused hour wondering why all my log files were erroring on write… anyhoo…

After ensuring the models are in the load path and are ready to go, ActiveRecord needs to be initialised. I added the ability to at runtime choose the environment to write to database-wise, just like Rails does. This is achieved here:

# First load the Rails config/database.yml
database_configuration_file = File.expand_path(File.join(File.dirname(__FILE__), 'config', 'database.yml'))
database_configuration = YAML::load(ERB.new(IO.read(database_configuration_file)).result)

# Parse out the RAILS_ENV=production setting, which can be either in the form
# ruby tracking_daemon.rb start RAILS_ENV=production or
# ruby tracking_daemon.rb start production
# Note that the environments allowed are vaildated against the ones available in the database.yml
ARGV.each do |arg|
  if arg.include?('=')
    key, val = arg.split('=', 2)
    ENV[key] ||= val
  elsif database_configuration.keys.include?(arg)
    ENV['RAILS_ENV'] ||= arg
  end
end

RAILS_ENV = (ENV['RAILS_ENV'] || "development").dup
puts "Starting #{FILE_NAME} daemon in #{RAILS_ENV} mode"

ActiveRecord::Base.configurations = database_configuration
ActiveRecord::Base.establish_connection RAILS_ENV

At the end of this we have loaded the database config and connected to the database. Happy days. The only thing left is to start the daemon, which I chose to do in an inline fashion. Note that Daemons allows you to have the process report whatever name you like in the process lists, but I went with the name of the file itself for clarity:

root_files_path       = File.expand_path(File.join(File.dirname(__FILE__), 'files'))
incoming_files_path   = File.expand_path(File.join(root_files_path, 'incoming_files'))

options = {
             :multiple   => true, # allow multiple concurrent of the same
             :ontop      => false, # daemonise
             :backtrace  => true, # show full failure info
             :log_output => true, 
             :monitor    => true # instantiate a monitor to restart as required
           }

Daemons.run_proc(FILE_NAME, options) do
  loop do
    # 1. Get the next file to process
    Dir.glob(File.join(incoming_files_path, '*')) do |incoming_file|
    end
    
    #...

    puts "Sleep #{SLEEP_TIME_SECONDS} sec" if RAILS_ENV == "development"
    sleep(SLEEP_TIME_SECONDS)
  end
end

All in all it’s been a great success – what was previously using a few backend servers running full whack to process all the incoming information, was now using 2 daemons on a single server in each datacentre. They hardly even show up on the top list. Excellent stuff.

Guide to Electronic Music

Saturday, August 11th, 2007

I found this Guide to Electronic Music – it’s so comprehensive it makes my head spin… but now when someone ask me what I think of weird music variants I’ll be able to give them an answer ;)

WordPress <code> and <pre> formatting

Thursday, June 7th, 2007

After the last paste-heavy post, I finally got around to doing something about wordpress’s formatting of code blocks. This site was very helpful:

http://tjulo.blogspot.com/2007/03/wordpress-and-source-code-posts.html

So, edit the file wp-includes/js/tinymce/plugins/wordpress/editor_plugin.js.

Comment out the section that replaces pre content:

/* var startPos = -1;
while ((startPos = content.indexOf('', startPos+1);
var innerPos = content.indexOf('>', startPos+1);
var chunkBefore = content.substring(0, innerPos);
var chunkAfter = content.substring(endPos);

var innards = content.substring(innerPos, endPos);
innards = innards.replace(/\n/g, '
');
content = chunkBefore + innards + chunkAfter;
}*/

Then I had to update my CSS so that pre used the same class as code and then change my posts over to use pre instead of code.

Next step is to fix code too so that I don’t have to think about it.

It does seem to have the side-effect of not automatically handling the escaping of < and > symbols outside those tags though, which for this post where I started with <code> and <pre> instead of code and pre, made things a bit of a mess.

London tea, disgusting me

Thursday, May 3rd, 2007

Ever since I arrived in London, I’ve been wondering why the black tea I have made here has a horrible oily film on it.

Like the good noodle I am, I googled it, and it turns out that it’s because the water over here is so hard. When you boil water, it excites the calcium and magnesium salts in the water, which the tannins in the tea bind to as it brews. The oxygen in the air makes larger groups form that can no longer stay in solution and then separate from the water.

Why black tea and not green tea or coffee? It’s still there in green tea, but because there is less tannin in the green tea it’s pretty much invisible. Coffee beans have about half the tannin in them as tea leaves, so once again it’s still there but less visible. I think especially instant coffee would mask this effect because it 1. is soluble so may contain agents to assist this, and 2. is weaker in raw content then espresso. Also, I add milk to coffee so that probably masks it as well.

It has no flavour, and is not harmful, but it’s a bit gross and sticks to the inside of the cup. Thus, I have not made it far into the 100 bags I bought. Now, it seems I have to either boil the crap out of the water so that the salts become lime scale in the kettle, or filter the water first.

I’ll buy a filter.

OzRailer

Thursday, March 16th, 2006

A while ago, in a moment of inspiration/impulse purchase, I bought the domain “ozrailer.com” thinking it’d be great to put stuff there about Rails dev from, well, Oz. As of today, that idea is postponed and integrated with this site, “dansketcher.com” mainly because I started to doubt whether it was actually all that clever. It could be a bit wanky. So, Dreamhost mirror configuration, here we are!