Thinking-Sphinx :with => @geodist returns no results

Posted by dansketcher on January 01, 2010
gems, Rails, rant, software development / No Comments

Having just recently started using Thinking Sphinx, I’ve been toying with the geograhical search that it provides to simply do distance-based searches while querying other information. The Geo-searching part of the manual shows how to do searches limiting distance:

# Searching for places within 10km
Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => 0..10_000}
# Searching for places sorted by closest first
Place.search "pancakes", :geo => [@lat, @lng],
  :order => "@geodist ASC, @relevance DESC"

Unfortunately, the first style of query where @geodist is limited in range – when used as-is – does not work I found in my testing that I could order by @geodist as per the second query but when I added the :with range, I had no results.

The solution was actually simple, but really annoying. You have to use Floats instead of Integers for the range numbers! So changing the first query to:

Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => 0.0..10_000.0}

# or more neatly

Place.search "pancakes", :geo => [@lat, @lng],
  :with => {"@geodist" => Range.new(0.0, 10_000.0)}

makes it all work. Now to get someone to update those docs!
Update: Updated!

Rails.cache with Memcache and a File cache for fragments

Posted by dansketcher on December 07, 2009
Uncategorized / No Comments

One of the things that we do at TouchLocal is provide Search Engine Optimised pages that drive traffic to the site through the UK National Directory pages. As you can imagine, there’s quite a lot of data intensive processing that is required to generate these pages, and there are an awful lot of them. As it changes relatively rarely, for a long time we had been caching these pages in a Rails Action cache so that the headers and footers are not cached but the content is.

As we enhanced the caching strategy through the site, and the Rails support for Memcache improved, we changed the Rails cache to use Memcache. This by implication meant that the default cache we had been using for actions (the filestore cache) was no longer used and the infrastructure team could not flush the cache in the same way it previously had.

So that we could use the Rails.cache helper and still have the benefit of the Action cache, we changed our production environment configuration to look like this:

# Namespace for memcache to automatically flush on app version updates.
# APP_REVISION is updated automatically via asking the Version Control System
ns = "dir_#{RAILS_ENV}_#{$persona}_#{APP_REVISION}"

# MC_ADDRESSES constant is put into the root namespace - we use it for other things
::MC_ADDRESSES = ['10.0.0.1:11211', '10.0.0.2:11211', '10.0.0.3:11211']

# Rails.cache is memcache
config.cache_store = :mem_cache_store, MC_ADDRESSES,{:namespace => ns}

# Force the ActionController cache store to be the file one as per the previous default.
config.action_controller.cache_store = :file_store, "#{RAILS_ROOT}/tmp/cache"

Happy days.

Integrating Zeus ZXTM Load Balancer and Capistrano for Ruby on Rails websites

Posted by dansketcher on November 30, 2009
Uncategorized / No Comments

At TouchLocal, one of the difficulties in automating deployment was the tight coupling between the app servers that were currently running and what the Load Balancer said was running. The implication of this is that at deployment time a human had to go into the ZXTM interface and update the list of running nodes before and afterwards. To ease this, and allow fully automatic deployment of Ruby on Rails apps running under Phusion Passenger, I built the ZXTMManager class.

This class implements an interface to the Zeus ZXTM API, allowing (in its current state) the ability to programatically change active nodes and manage other details of the commonly changed data in the Load Balancer. On top of that, there is an included deploy.rb file that shows the deployment recipes that can be simply and easily added to your own Capistrano configuration to use this class quickly and easily.

Enjoy.

Using DataFabric to replace Masochism

Posted by dansketcher on November 25, 2009
Uncategorized / No Comments

When we upgraded from Rails 1.x to 2.x, we also had to migrate from Masochism to DataFabric. Unfortunately, the DataFabric way of doing things requires adding a declaration to each of the model classes. For a site like TouchLocal, where there are literally hundreds of model classes, this was a daunting task.

I’ve just gone through extracting the magic that makes it possible to do this programatically, released as DataFabric::Initializer. I hope it helps someone other than me!

Howto recover lost commits from a git rebase

Posted by dansketcher on November 23, 2009
Uncategorized / No Comments

Yesterday, I was working on a branch of a branch in a git repository, and I wanted to merge the last branch back to master. Following chapter 3.6 of ProGit, I ran

git rebase --onto master branch1 branch2

Unfortunately for me, I seem to have either done it from the wrong place or done the wrong thing for my situation, because git rewound branch2 to the revision that it was branched from master and all the later commits were inaccessible!

However, I was not going to give up quite so easily, and I figured that once a revision was committed, it had to be somewhere.. Fortunately for me, I manged to find this post on how to recover lost commits in a git repository – I subsequently ran:

git reflog show
# find the commit with the last set of changes in your branch - 
# good commit comments are so useful!
git co b6654bd
git co -b branch2_2

With my branch recovered, I did a simple rebase of the new (old) branch onto master and merged it, and all was well.

Workling support for Synchronous AMQP RabbitMQ Clients and Amazon SQS Queues

Posted by dansketcher on November 16, 2009
gems, Rails, software development / No Comments

As a part of contracting work I have been doing for TouchLocal, I have just opensourced some code I wrote to support new Workling clients. As you may know, Workling is a Rails-oriented system for performing asynchronous processing and optionally returning data from these background workers. However, because of the implementation of the original AMQP client, you could not use RabbitMQ queues from non-evented Mongrel or Phusion Passenger servers (only evented Mongrel or Thin).

Building on the work of celdee-bunny and famoseagle-carrot, I implemented a RabbitMQ workling client that could be used from within Phusion Passenger and Mongrel. The Synchronous AMQP Workling client allows RabbitMQ to be used from Workling without requiring complicated changes to deployment scenarios. Also, I implemented the Return Store functionality, so that RabbitMQ users can get data back from the workers, just like when using Starling.

Additionally, it was useful at the time to add support for an Amazon SQS Workling client, more as an exercise in testing its performance than anything else. As with the SyncAmqpClient, support for the Return Store is present. One of the discoveries in working with Amazon SQS (via the RightAws gem) was the discovery that the default key structure for Workling (which uses colon characters for segment delimiters) is not supported by Amazon. As a result, if you define the keys used for AWS configuration, even if you don’t use them, they will change the Workling key structure. This is not a problem for new implementations, but for existing deployments adding this will mean that Workling cannot see the old queues and you may not be able to access them without removing the AWS configuration… not a deal breaker, but something to be aware of.

So, the TouchLocal github account holds the version of Workling that has these two implementation for now, at least until my pull request to the main branch is accepted :)

touchlocal-openx gem released

Posted by dansketcher on November 04, 2009
gems, hosting, Rails, software development / No Comments

After the last post, I spent a bit of time integrating my work (along with other upgrades) into a fork of the openx code on github and have released a gem version of it on GemCuttter.org

As you may or may not know, GemCutter.org will be the new default gem source by becoming rubygems.org. This is pretty exciting, because previously you had to have you project registered on rubygems.org in order to publish to it, or use the non-standard github gem host. Good stuff.

In any case, the new OpenX gem can be installed now by executing

sudo gem install touchlocal-openx --source "http://gemcutter.org"

# Load it using
require 'rubygems'
gem 'touchlocal-openx' 
require 'openx'

In Rails, include it like this in your Rails::Initializer block:

  config.gem "touchlocal-openx", :lib => "openx", :source => "http://gemcutter.org"

I’m a proud parent!

Minimum OpenX XMLRPC Ruby Client

Posted by dansketcher on September 29, 2009
hosting, Rails, software development / No Comments

2009-11-04 Update: Gem version released

For TouchLocal I am currently reworking some of the internal advertising systems to use OpenX. That way we get better reporting and better reliability, but we can use things like OpenX Direct Selection to make the best use of our existing infrastructure.

While there are 2 Ruby projects (1 a more recent fork of the other), both are oriented around the API for administering OpenX rather than serving ads via the API. So, here is the minimum you need to do from Ruby to get a banner served:

(Based on http://www.openx.org/en/docs/tutorials/Advanced+XML-RPC)

require 'xmlrpc/client'
# The settings are the  HTTP Headers - the PHP client sets many, but this is the minimum requirement
settings = {:cookies => [], :remote_addr => 'localhost'}

# 'what' in our case is a Direct Selection
params = [
  what = 'Plumber',
  campaignId = 0,
  target = '',
  source = '',
  withText = false,
  xmlContext = [{'!=' => 'campaignid:2'}]
]

b = nil
begin
  server = XMLRPC::Client.new2("http://www.example.com/www/delivery/axmlrpc.php")
  b = server.call('openads.view', settings, *params)
  b['html']
rescue Exception => e
  puts e.message
end

In particular, note the content of the xmlContext parameter – that took me quite a few hours of digging through the PHP to work out. It’s an array of hashes, that use the keys “==” and “!=” to include or exclude banners based on the values. The values must be of the form “type:id”, with type being one of “campaignid”, “clientid”, “companionid”, and “bannerdid” (although technically, anything other than the first 3 is treated as a banner ID as of version 2.8.1 of OpenX)

I hope that saves someone the pain that I just went through to discover it!

can’t dup NilClass

Posted by dansketcher on May 11, 2009
Uncategorized / No Comments

I’m in the middle of upgrading an old Rails 1.2.6 app to Rails 2.3, and all of a sudden when logging in as a user, I get this:

TypeError in BusinessController#view

can't dup NilClass

C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2189:in `dup'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2189:in `scoped_methods'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2193:in `current_scoped_methods'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:2183:in `scope'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1548:in `find_every'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1588:in `find_one'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1574:in `find_from_ids'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:616:in `find'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations/belongs_to_association.rb:44:in `find_target'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations/association_proxy.rb:240:in `load_target'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations/association_proxy.rb:112:in `reload'
C:/development/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/active_record/associations.rb:1231:in `user'
C:/development/community_s0024_upgrade_upgrade/webapps/touchlocal/app/controllers/application_controller.rb:178:in `check_ticket_and_session'

Of course, that error means nothing as it is. What the underlying cause is though, is that the class reloader is trying to unload a class but it can’t because of included modules that can’t be unloaded. The fix is to use the keyword unloadable in the model to tell the reloader to force a reload on each request:

class User < ActiveRecord::Base
  unloadable

  ...
end

All fixed!

On comments

Posted by dansketcher on April 15, 2009
rant / No Comments

Blog comments. I used to like them. Some of my blog posts even have threads of helpful and useful feedback left by you, dear reader. Or should I say, some of you.

More recently I have started receiving a new sort of comment. The specious or outright false, hit-and-run attacks from anonymous morons who have somehow decided that the access to a keyboard and a place to exercise it allows them to write whatever they want. Well not here.

Should have listened to Alex Payne earlier…

If you want to reply to me, trackbacks are still enabled (for now)…