Freelancing Gods 2008

Following in the steps of the Australian Rails Camps, it’s now time to announce the first UK edition. Running from Friday the 15th to Monday the 18th of August, it will be an extended weekend of hacking, talking, eating, drinking and games, with a bunch of smart and passionate Ruby developers.

Even though the name is “Rails Camp”, previous camps have included talks on topics from Merb to Rack to Extreme Programming – all topics somewhat related to Ruby are welcome.

If you’d like to come along, I’d recommend registering soon, as there’s a very limited number of places.

Okay, it’s well past time for the companion piece to my Sphinx Primer – let’s go through the basic process of using Thinking Sphinx with Rails.

Just to recap: Sphinx is a search engine that indexes data, and then you can query it with search terms to find out which documents are relevant. Why do you want to use it with Rails? Because it saves having to write messy SQL, and it’s so damn fast.

(If you’re getting a feeling of deja-vu, then it’s probably because you’ve read an old post on this blog that dealt with an old version of Thinking Sphinx. I’ve had a few requests for an updated article, so this is it.)

Installation

So: first step is to install Sphinx. This may be tricky on some systems – but I’ve never had a problem with it with Mac OS X or Ubuntu. My process is thus:

curl -O http://sphinxsearch.com/downloads/sphinx-0.9.8-rc2.tar.gz
tar zxvf sphinx-0.9.8-rc2.tar.gz
cd sphinx-0.9.8-rc2
./configure
make
sudo make install

If you’re using Windows, you can just grab the binaries.

Once that’s taken care of, you then want to take your Rails app, and install the plugin. If you’re running edge or 2.1, this is a piece of cake:

script/plugin install git://github.com/freelancing-god/thinking-sphinx.git

Otherwise, you’ve got a couple of options. The first is, if you have git installed, just clone to your vendor/plugins directory:

git clone git://github.com/freelancing-god/thinking-sphinx.git
  vendor/plugins/thinking-sphinx

If you’re not yet using git, then the easiest way is to download the tar file of the code. Try the following:

curl -L http://github.com/freelancing-god/thinking-sphinx/tarball/master
  -o thinking-sphinx.tar.gz
tar -xvf thinking-sphinx.tar.gz -C vendor/plugins
mv vendor/plugins/freelancing-god-thinking-sphinx* vendor/plugins/thinking-sphinx

Oh, and it’s worth noting: if you’re not using MySQL or PostgreSQL, then you’re out of luck – Sphinx doesn’t talk to any other relational databases.

Configuration

Next step: let’s get a model or two indexed. It might be worth refreshing your memory on what fields and attributes are for – can I recommend my Sphinx article (because I’m not at all biased)?

Ok, now let’s work with a simple Person model, and add a few fields:

class Person < ActiveRecord::Base
  define_index do
    indexes [first_name, last_name], :as => :name
    indexes location
  end
end

Nothing too scary – we’ve added two fields. The first is the first and last names of a person combined to one field with the alias ‘name’. The second is simply location.

Adding attributes is just as easy:

define_index do
  # ...

  has birthday
end

This attribute is the datetime value birthday (so you can now sort and filter your results by birthdays).

Managing Sphinx

We’ve set up a basic index – now what? We tell Sphinx to index the data, and then we can start searching. Rake is our friend for this:

rake thinking_sphinx:index
rake thinking_sphinx:start

Searching

Now for the fun stuff:

Person.search "Melbourne"

Or with some sorting:

Person.search "Melbourne", :order => :birthday

Or just people born within a 10 year window:

Person.search "Melbourne", :with => {:birthday => 25.years.ago..15.years.ago}

If you want to keep certain search terms to specific fields, use :conditions:

Person.search :conditions => {:location => "Melbourne"}

Just remember: :conditions is for fields, :with is for attributes (and :without for exclusive attribute filters).

Change

Your data changes – but unfortunately, Sphinx doesn’t update your indexes to match automatically. So there’s two things you need to do. Firstly, run rake thinking_sphinx:index regularly (using cron or something similar). ‘Regularly’ can mean whatever time frame you want – weekly, daily, hourly.

The second step is optional, but it’s needed to have your indexes always up to date. First, add a boolean column to your model, named ‘delta’, and have it default to false. Then, tell your index to use that delta field to keep track of changes:

define_index do
  # ...

  set_property :delta => true
end

Then you need to tell Sphinx about the updates:

rake thinking_sphinx:stop
rake thinking_sphinx:index
rake thinking_sphinx:start

Once that’s done, a delta index will be created – which holds any recent changes (since the last proper indexing), and gets re-indexed whenever a model is edited or created. This doesn’t mean you can stop the regular indexing, as that’s needed to keep delta indexes as small (and fast) as possible.

String Sorting

If you remember the details about fields and attributes, you’ll know that you can’t sort by fields. Which is a pain, but there’s ways around this – and it’s kept pretty damn easy in Thinking Sphinx. Let’s say we wanted to make our name field sortable:

define_index do
  indexes [first_name, last_name], :as => :name, :sortable => true

  # ...
end

Re-index and restart Sphinx, and sorting by name will work.

How is this done? Thinking Sphinx creates an attribute under the hood, called name_sort, and uses that, as Sphinx is quite fine with sorting by strings if they’re converted to ordinal values (which happens automatically when they’re attributes).

Pagination

Sphinx paginates automatically – in fact, there’s no way of turning that off. But that’s okay… as long as you can use your will_paginate helper, right? Never fear, Thinking Sphinx plays nicely with will_paginate, so your views don’t need to change at all:

<%= will_paginate @search_results %>

Associations

Sometimes you’ll want data in your fields (or attributes) from associations. This is a piece of cake:

define_index do
  indexes photos.caption, :as => :captions
  indexes friends.photos.caption, :as => :friends_photos

  # ...
end

Polymorphic associations are fine as well – but keep in mind, the more complex your index fields and attributes, the slower it will be for Sphinx to index (and you’ll definitely need some database indexes on foreign key columns to help it stay as speedy as possible).

Gotchas

In case things aren’t working, here’s some things to keep in mind:

• Added an attribute, but can’t sort or filter by it? Have you reindexed and restarted Sphinx? It doesn’t automatically pick up these changes.
• Sorting not working? If you’re specifying the attribute to sort by as a string, you’ll need to include the direction to sort by, just like with SQL: “birthday ASC”.
• Using name or id columns in your fields or attributes? Make sure you specify them using symbols, as they’re core class methods in Ruby.

define_index do
  indexes :name

  # ...

  has photos(:id), :as => :photo_ids
end

And Next?

I realise this article is pretty light on details – but if you want more information, the first stop should be the extended usage page on the Thinking Sphinx site, quickly followed by the documentation. There’s also an email list to ask questions on.

I’ve just started my round-the-world conferences-and-holiday adventure, and the first stop is RailsConf in Portland – so if you’re in town and see me wandering around looking rather cluelessly, please say hi.

Also, in case you’re on the Twitter bandwagon, you’ll find me with the creative nickname of pat.

In case you’ve not been watching every commit carefully flow through Thinking Sphinx on GitHub – PostgreSQL support has been added. I’ve done a little bit of testing, and I’ve had some excellent support from Björn Andreasson and Tim Riley, so I feel it’s ready for people to start kicking the tires.

I’m no PostgreSQL expert – I definitely fall into the n00b category – so if you think there’s better ways to do what I’m doing, would love to hear them.

If you’re a Ruby developer in or near Australia, I highly recommend attending RailsCamp number 3, which has just opened for registration. The first two were simply amazing, so I’m just a little annoyed that I can’t make it to this one (as I’ll be traveling overseas at the time). I’ve no doubt that this one will be just as fantastic – expect an extended weekend of hacking and talking with a bunch of smart, entertaining and passionate developers, and plenty of drinks and games thrown in for good measure.

You don’t need to be a Rails or Ruby genius to attend – just a desire to discuss, learn, teach and (most importantly) have fun.

Go register now.

Another small sphinx-related post.

In line with the first release candidate release of Sphinx 0.9.8 last week, I’ve updated both my API, Riddle, and my plugin, Thinking Sphinx, to support it. Also, for those inclined, you can now get Riddle as a gem.

I’m slowly making progress on some major changes to Thinking Sphinx, so hopefully I’ll have something cool to show people soon. Oh, but some features that aren’t reflected in the documentation: most of Sphinx’s search options can be passed through when you call Model.search – including :group_by, :group_function, :field_weights, :sort_mode, etc. Consider it an exercise for the reader to figure out the details until I get around to improving the docs.

Here’s a collection of notes made while I was working on migrating this blog from Rails to Merb (no, the Merb version isn’t live yet). These are relevant to version 0.5.2 – but Merb’s moving so fast these days, I wouldn’t be surprised if much of this isn’t relevant any more.

Most of this you can find if you look through the documentation, but if you’ve not yet played with Merb, this will hopefully give you some idea of some of the smaller differences.

Filters

• Use before and after instead of before_filter and after_filter
• Instead of :except, use :exclude.
• To kill the chain of filters and not follow on to the action method, you need to use the throw method:

def confirm_user_session
  throw :halt, Proc.new { |controller|
    controller.redirect url(:new_session)
  } if current_user.nil?
end

Oh, and in case you missed it in the above sample, redirect_to becomes redirect.

Routes

Resource routing isn’t actually any different to Rails – but I had trouble finding any documentation for it, so it wasn’t obvious to start with. This code will work in both Merb and Rails (although in Rails r is usually referenced as map)

r.resources :posts do |post|
  post.resources :comments
end

However, using these routes is definitely different. The url_for, route_url and route_path methods aren’t around – you need to use Merb’s magical url method:

url(:post, @post@)
url(:new_session)

View Helpers

Helpers are pretty minimal in Merb – you don’t get any of the inflectors, or the number formatters. And no form helpers – not in the core gem, anyway. You can get some of those from the merb_helpers gem, but they don’t match Rails’ syntax and method naming, so it can take a bit of time to get this switched over, depending on the size of your app.

No support for Form Builders in that gem, by the way.

Some of the default helpers that do exist have different names to their Rails counterparts. A few of the ones I came across:

• content_tag => tag
• tag => open_tag
• content_for => throw_content
• javascript_include_tag => js_include_tag
• stylesheet_include_tag => css_include_tag

All your favourite plugins

Because Merb plugins are gems, anything you use as a plugin in Rails is pretty unlikely to be ported over. will_paginate was the main one for me, so I ended up pulling the files into my lib directory. Of course, that was only really useful when using ActiveRecord – DataMapper and Sequel users may have to get hacking into any ActiveRecord-focused plugins they want to use.

Partials

No more render :partial – you want something more like the following:

partial "comments/show", :with => @post.comments.active
partial "post", :with => @posts, :as => :post

Again – you’ll find most/all of this in the documentation, but the only potential show-stopper I found was plugins – the rest isn’t that that big a difference to Rails.

Okay, time for another code post. Prompted by a comment on an earlier post about Merb and MYOB, I thought I’d provide some more detail on how to talk to MYOB using Ruby, for any other people stuck in a similar position.

ODBC Bindings

If you like crafting your own SQL, this is the easiest approach. Firstly, you’ll need to download Christian Werner’s ODBC Bindings for Ruby, and install it using the following commands (or something along these lines) from within the source directory:

ruby extconf.rb
make
sudo make install

Documentation is a bit light on examples – and it’s really just a hook into the C libraries for ODBC, so good luck reading the source. Here’s some of the basic things you’ll need to do. (Don’t forget to require 'odbc' first, of course).

Setting up a Connection

You can either go through the DSN collection, or connect using the string name of the DSN (which in this case we’ll assume is ‘MYOB’):

dsn = ODBC.datasources.detect { |source| source.name == "MYOB" }
database = ODBC::Database.connect dsn

# or

database = ODBC::Database.connect "MYOB"

Note that the ODBC.datasources collection only contains system DSNs, not User-level ones. Not sure why, or how to access the latter.

SELECT Statements

statement = database.prepare "SELECT * FROM Customers" 
statement.execute
statement.each_hash d |row|
  # results accessed as row["column_name"]
end
statement.drop

Make sure you drop the statement once you’re done with it, otherwise you’ll get complaints when you stop ruby that the statements weren’t all closed and dropped.

There’s other ways to access results – you can go through results as you see fit using fetch or fetch_hash, depending on whether you want an array or hash of the record.

INSERT Statements

With other SQL statements (although there’s no use of UPDATE or DEETE in MYOB connections), you don’t really care about the results, so that makes things a bit simpler, you can just call database.do("INSERT INTO Import_Customer_Cards (...) VALUES (...)"). The do method creates the statement, executes it and then drops it for you. MYOB makes this a little more complex though, as it all must be done in transactions – not that that’s a bad thing.

database.transaction do |env|
  env.do("INSERT INTO Import_Customer_Cards (...) VALUES (...)")
end

This won’t actually do anything though – and it won’t throw up an error or warning. The problem is you need to turn autocommit mode off, because the MYOB ODBC drivers don’t like it (it’s not a problem with Christian’s bindings). So, back when you create your connection:

database = ODBC::Database.connect "MYOB" 
database.autocommit = false

ActiveRecord

If you prefer some level of abstraction above the messy SQL, then it might be worth looking at OpenLink’s ActiveRecord ODBCAdapter. I recommend the gem instead of the plugin, as the plugin takes it upon itself to modify your code, which I don’t like.

sudo gem install odbc-rails

Now, I’m not an ODBC expert, but there seems to be different flavours of ODBC connections possible – and MYOB’s is not one supported by this gem. So, if you’re using Rails, put the gem into vendor (and for Merb, into the local gems folder), then modify it with this patch. I make no promises for it being stable or reliable – but I’ve not had any problems yet.

Of course, using ActiveRecord is viable if you’re just reading data out – but if you want to write as well as read, then there’s issues. MYOB has separate write tables (prefixed with ‘Import_’), and they have denormalised schemas compared to the read-equivalents.

For database.yml, you’ll need something like the following:

development:
  adapter: odbc
  dsn: MYOB

Everything Else

In both the situations above, I’ve not put usernames or passwords into the connections – you can, but that is already handled by the ODBC DSN, so I keep my code that bit simpler.

For those who are new to coding for MYOB - it does cost money to get a developer account (several hundred dollars per year), which is the only way to get write access. I think read access is a once off fee of a couple of hundred dollars, but that’s specific to a MYOB file.

And if anyone is wondering – while I was initially using the odbc-rails library, I’ve now switched to constructing the SQL myself and just using the bindings, because of the table issues.

Today was the first time I’d come across Ruby’s undef_method – it’s used in a few places in Rails, particularly with ActiveRecord’s associations. While I see the point of it, there were a few methods I wanted back – and I’ve figured out how to do it – you need to grab the method definition from the superclass. Here’s an example:

class AntiString
  undef_method to_s
end

AntiString.new.to_s
  #=> NameError: undefined method `to_s' for class `AntiString'

AntiString.send(:define_method, :to_s,
  AntiString.superclass.instance_method(:to_s))

AntiString.new.to_s #=> "#<AntiString:0x364ef8>"

Now, the obvious caveat – if the method was originally defined in that class, not the superclass, then I think you’re out of luck. Although I’m guessing you’ll rarely be in a position where you need to resurrect a method like this anyway.

For one of the contracts I’m working on at the moment, I’ve been using Merb to construct a web service that interacts with MYOB, and can be consumed with ActiveResource.

The connection to MYOB is ugly, using Christian Werner’s ODBC Bindings and the Rails/ActiveRecord ODBC Adapter, the latter of which had to be hacked slightly. However, the Merb side of things was quite clean. I’m really looking forward to seeing how Merb progresses, especially with their plans for merb_core and merb_more.

One of the rare snippets of code from the Merb app that I think is more verbose than the Rails equivalent is how to go about obtaining the query parameters (as opposed to routing parameters) of a request.

The Rails way:

request.query_parameters

The Merb way:

params.except *request.route_params.keys.collect { |key| key.to_s }

Also, in case you’re as stupid as I am and want to generate Merb controllers on the fly, you can’t use Class.new. The only way is by building the class in a string and eval’ing it:

Object.send(:eval, <<-CODE
class ::Object::#{controller_name} < Application
  # actions and such go here
end
CODE
)

It’s not particularly elegant, but at least it works.

Short post, as befitting the importance of the content: Riddle and Thinking Sphinx have both been updated to support the current version of Sphinx, 0.9.8r1065.

An addition to auspostie.com, prompted by Dr Nic’s suggestion in the comments of the earlier blog post – the postie gem. It allows easy parsing of Postie search results, and also provides a command-line tool for searching.

By suburb:

postie Brunswick

By postcode:

postie 3070

To install:

sudo gem install postie

To use within your own ruby code:

require 'postie'

Postie::Locality.find("Melbourne")

Again, extremely simple, but just makes access to the data that little bit easier.

Now, what would be really cool is a Quicksilver plugin that queries the API. Any volunteers to code that up?

A couple of weeks ago I quickly coded a basic webservice – using Merb – for Australian Postcode data. Just got hosting for it sorted out last night, so now I can blog about it.

Postie

The emphasis is on simple – you can search by postcode or suburb, and get the data back either as a HTML page, XML, or JSON. That’s pretty much it. The request urls aren’t complicated, either:

• auspostie.com/Brunswick
• auspostie.com/Brunswick.js
• auspostie.com/Brunswick.xml
• auspostie.com/2782
• auspostie.com/2782.js
• auspostie.com/2782.xml

Suburb requests use partial matching, so you don’t need the full suburb name. If you want the JSON returned with the MIME type of application/json, use the .json extension.

I’ve no plans at this point on using this, but perhaps it’s useful for someone out there – if so, would love to hear about it.

Another quick Sphinx post – Riddle is updated to support Sphinx’s latest release (0.9.8r985), and Thinking Sphinx now has that new version of Riddle as well.

I’ve not tested any of this with the recently released Ruby 1.9 yet, though (but it’s on my list of things to do).

Also, thank-you to Joost Hietbrink (again) and Jonathan Conway for their patches to Thinking Sphinx – very much appreciated.

I’m always trying to find a better way to write specs for my Rails apps with RSpec – it took a while for me to be comfortable with writing model specs, but just recently, I’ve developed a style of controller specs that I feel work well for me.

While it’s not too hard to write methods that automate some of the repetitive side of things, it can be hard to do so in a manner that fits RSpec’s DSL - but I’ve found the key to (my current style of) controller specs is shared behaviours. An example of a few actions from the news controller at ausdwcon.org:

describe NewsController, "index action" do
  before :each do
    @method = :get
    @action = :index
    @params = {}
  end

  it_should_behave_like "Public-Access Actions" 

  it "should paginate the results" do
    @news = []
    News.stub_method(:paginate => @news)

    get :index

    News.should have_received(:paginate)
    assigns[:news].should == @news
  end

  it "should set the title to 'News'" do
    News.stub_method(:paginate => [])

    get :index

    assigns[:title].should == "News" 
  end
end

describe NewsController, "new action" do
  before :each do
    @method = :get
    @action = :new
    @params = {}
  end

  it_should_behave_like "Admin-Only Actions" 

  it "should set the title to 'News'" do
    controller.current_user = User.stub_instance(:admin? => true)

    get :new

    assigns[:title].should == "News" 
  end
end

You can find the full spec for the controller on pastie.caboo.se. The shared behaviors – ‘Public-Access Actions’ and ‘Admin-Only Actions’ – are (at least for the moment) kept in my spec_helper.rb file – a sample of which is below:

describe "Admin-Only Actions", :shared => true do
  it "should not be accessible without authentication" do
    @controller.current_user = nil

    send @method, @action, @params

    response.should be_redirect
    response.should redirect_to(new_session_url)
  end

  it "should not be accessible by a normal user" do
    @controller.current_user = User.stub_instance(:admin? => false)

    send @method, @action, @params

    response.should be_redirect
    response.should redirect_to(new_session_url)
  end

  it "should be accessible by an admin user" do
    @controller.current_user = User.stub_instance(:admin? => true)

    send @method, @action, @params

    response.should_not redirect_to(new_session_url)
  end
end

Firstly – the not so nice stuff: the use of instance variables to communicate the method, action and parameters of requests to the shared behaviours. It’s not ideal, and apparently there’s plans to add arguments to the it_should_behave_like method, but for the moment it does the job.

I’m using Pete Yandell’s NotAMock for my stubbing – albeit with a few modifications of my own (which may make it back into the plugin itself at some point). I also use my own ActiveMatchers – but that’s more focused on models. It’s also not really feature-complete, but if you like what it offers, feel free to use it.

Oh, and the main caveat? This is my current way of spec’ing controllers – and it’s vastly better than the minimal specs I was writing before this – but it may/will change. I don’t even know if my style is ‘best practice’ – I’m putting them online to get feedback and provoke discussion. So please feel free to critique.

Rails code snippet for the day: validates_uniqueness_of_set. Useful for making sure each specific combination of the specified attributes is unique. Example from the pastie:

validates_uniqueness_of_set :first_name, :last_name

Just like the options it accepts, the code is very similar to validates_uniqueness_of – and the few tests I’ve thrown at it are handled without any problems.

RailsCamp 2.0 finished earlier today – and I think it’s safe to declare it a fantastic success (even given my bias).

Massive thanks to Ben and Karen for their hard work getting everyone fed and co-ordinating people in the kitchen (and the RailsCamp bus from Melbourne to Sunnystones). Thanks too to everyone who helped at various points – both in the organising and over the weekend.

Finally, thank you to everyone who came along – these camps are so much fun because of the calibre of people who attend, and their willingness to share ideas, code and laughs.

Will be posting a version of my talk at some point soon… once I’ve recovered from the weekend.

Built out of the work I’ve done for Thinking Sphinx (which has just got basic support for delta indexes, attributes and sorting – although the documentation doesn’t reflect that), I’ve extracted a new Ruby client that communicates with Sphinx, which I’ve named Riddle.

I’m not going to delve into the code here – because I’m not expecting it to be that useful to many people (and I just wrote examples in the documentation – go read that instead!) – but I’m very happy with how it’s ended up, and it’s got some level of specs to give it a thorough test. It’s also compatible with the most recent release of Sphinx (0.9.8 r871). Should you wish to poke around with it, just check it out from subversion:

svn co
  http://rails-oceania.googlecode.com/svn/patallan/riddle/trunk riddle

It’s also being used in Evan Weaver’s UltraSphinx plugin, which I’m pretty pleased about.

Code snippet for application.rb that makes life just a little easier if multiple actions are doing exactly the same thing (both in the controller and the view):

def self.alias_action(existing, aliased)
  define_method(aliased.to_sym) do
    send(existing.to_sym)
    render :action => existing.to_s
  end
end

And then in the appropriate controller (where edit is an existing action, and show needs to be exactly the same):

alias_action :edit, :show

I know this is a specialised use – but perhaps someone else out there is doing something similar.