Freelancing Gods 2010

It’s just over two months since I asked for – and received – support from the Ruby community to work on Thinking Sphinx for a month. A review of this would be a good idea, hey?

I’m going to write a separate blog post about how it all worked out, but here’s a long overview of the new features.

Internal Cucumber Cleanup

This one’s purely internal, but it’s worth knowing about.

Thinking Sphinx has a growing set of Cucumber features to test behaviour with a live Sphinx daemon. This has made the code far more reliable, but there was a lot of hackery to get it all working. I’ve cleaned this up considerably, and it is now re-usable for other gems that extend Thinking Sphinx.

External Delta Gems

Of course, it was my own re-use that was driving that need: I wanted to use it in gems for the delayed job and datetime delta approaches.

There was a clear need for removing these two pieces of functionality from Thinking Sphinx: to keep the main library as slim as possible, and to make better use of gem dependencies, allowing people to use whichever version of delayed job they like.

So, if you’ve not upgraded in a while, it’s worth re-reading the delta page of the documentation, which covers the new setup pretty well.

Testing Helpers

Internal testing is all very well, but what’s much more useful for everyone using Thinking Sphinx is the new testing class. This provides a clean, simple interface for processing indexes and starting the Sphinx daemon.

There’s also a Cucumber world that simplifies things even further – automatically starting and stopping Sphinx when your features are run. I’ve been using this myself in a project over the last few days, and I’m figuring out a neat workflow. More details soon, but in the meantime, have a read through the documentation.

No Vendored Code for Gems

One of the uglier parts of Thinking Sphinx is the fact that it vendors Riddle and AfterCommit (and for a while, Delayed Job), two essential libraries. This is not ideal at all, particularly when gem dependencies can manage this for you.

So, Thinking Sphinx no longer vendors these libraries if you install it as a gem – instead, the riddle and after_commit gems will get brought along for the ride.

The one catch is that they’re still vendored for plugin installations. I recommend people use Thinking Sphinx as a gem, but there are valid reasons for going down the plugin path.

Default Sphinx Scopes

Thanks to some hard work by Joost Hietbrink of the Netherlands, Thinking Sphinx now supports default sphinx scopes. All I had to do was merge this in – Joost was the first contributor to Thinking Sphinx (and there’s now over 100!), so he knows the code pretty well.

In lieu of any real documentation, here’s a quick sample – define a scope normally, and then set it as the default:

class Article < ActiveRecord::Base
  # ...
  
  sphinx_scope(:by_date) {
    {:order => :created_at_}
  }
  
  default_sphinx_scope :by_date
  
  # ...
end

Thread Safety

I’ve made some changes to improve the thread safety of Thinking Sphinx. It’s not perfect, but I think all critical areas are covered. Most of the dynamic behaviour occurs when the environment is initialised anyway.

That said, I’m anything but an expert in this area, so consider this a tentative feature.

Sphinx Select Option

Another community-sourced patch – this time from Andrei Bocan in Romania: if you’re using Sphinx 0.9.9, you can make use of its custom select statements:

Article.search 'pancakes',
  :sphinx_select => '*, @weight + karma AS superkarma'

This is much like the :select option in ActiveRecord – but make sure you use :sphinx_select (as the former gets passed through to ActiveRecord’s find calls).

Multiple Index Support

You can now have more than one index in a model. I don’t see this as being a widely needed feature, but there’s definitely times when it comes in handy (such as having one index with stemming, and one without). The one thing to note is that all indexes after the first one need explicit names:

define_index 'stemmed' do
  # ...
end

You can then specify explicit indexes when searching:

Article.search 'pancakes',
  :index => 'stemmed_core'
Article.search 'pancakes',
  :index => 'article_core,stemmed_core'

Don’t forget that the default index name is the model’s name in lowercase and underscores. All indexes are prefixed with _core, and if you’ve enabled deltas, then a matching index with the _delta suffix exists as well.

Building on from this, you can also now have indexes on STI subclasses when superclasses are already indexed.

While the commits to this feature are mine, I was reading code from a patch by Jonas von Andrian – so he’s the person to thank, not me.

Lazy Initialisation

Thinking Sphinx needs to know which models have indexes for searching and indexing – and so it would load every single model when the environment is initialised, just to figure this out. While this was necessary, it also is slow for applications with more than a handful of models… and in development mode, this hit happens on every single page load.

Now, though, Thinking Sphinx only runs this load request when you’re searching or indexing. While this doesn’t make a difference in production environments, it should make life on your workstations a little happier.

Lazy Index Definition

In a similar vein, anything within the define_index block is now evaluated when it’s needed. This means you can have it anywhere in your model files, whereas before, it had to appear after association definitions, else Thinking Sphinx would complain that they didn’t exist.

This feature actually introduced a fair few bugs, but (thanks to some patience from early adopters), it now runs smoothly. And if it doesn’t, you know where to find me.

Sphinx Auto-Version detection

Over the course of the month, Thinking Sphinx and Riddle went through some changes as to how they’d be required (depending on your version of Sphinx). First, there was separate gems for 0.9.8 and 0.9.9, and then single gems with different require statements. Neither of these approaches were ideal, which Ben Schwarz clarified for me.

So I spent a day or two working on a solution, and now Thinking Sphinx will automatically detect which version you have installed. You don’t need any version numbers in your require statements.

The one catch with this is that you currently need Sphinx installed on every machine that needs to know about it, including web servers that talk to Sphinx on a separate server. There’s an issue logged for this, and I’ll be figuring out a solution soon.

Sphinx 0.9.9

This isn’t quite a Thinking Sphinx feature, but it’s worth noting that Sphinx 0.9.9 final release is now available. If you’re upgrading (which should be painless), the one thing to note is that the default port for Sphinx has changed from 3312 to 9312.

Upgrading

If you want to grab the latest and greatest Thinking Sphinx, then version 1.3.14 is what to install. And read the documentation on upgrading!

At a recent Melbourne Ruby meet, I was asked to speak about my travelling freelancer lifestyle, and the talk was recorded. I feel a little self-conscious about the topic, but perhaps you’ll find it interesting.

Massive thanks to James Healy for not only recording the talks that night, but producing the neat slides-and-video output. I’m looking forward to the Melbourne Ruby channel building up a good collection of sessions.

Also: I’ll be posting a review of my month working on Thinking Sphinx soon.

Update: I’ve now hit my target. If you want to donate more, I won’t turn you away, but perhaps you should send those funds to other worthy open source projects, or a local charity. A massive thank you to all who have pitched in to the pledgie, your generosity and support is amazing.

Over the past two years, Thinking Sphinx has grown massively – in lines of code, in the numbers of users, in complexity, in time required to support it. I’m regularly amazed and touched by the recommendations I see on Twitter, and the feedback I get in conversations. The fact that there’s been almost one hundred contributors is staggering.

It’s not all fun and games, though… there’s still plenty of features that can be added, and bugs to be fixed, and documentation to write. So, what I’d really like to do is spend November working close to full-time on just Thinking Sphinx. I have a long task list. All I need is a bit of financial help to cover living expenses.

I have an existing pledgie tied to the GitHub project, currently sitting on $600. If I can get another $2000, then I won’t have to worry at all about how I’m going to pay bills or rent for November. Even $1400 will make it viable for me, albeit maybe with some help from my savings.

If you or your workplace can make a donation, that would be very much appreciated. I’m happy to provide weekly updates on where things are at if people request it – but of course, watching the GitHub projects for Thinking Sphinx itself and the documentation site is the most reliable way to keep an eye on my progress.

I’m hoping to get Thinking Sphinx to a point where the documentation is by far the best place for support, and it’s only the really tricky problems (and bug reports) that end up in my inbox.

I want it to be a model Ruby library that doesn’t get in your way, is as fast as possible, and plays nicely with other libraries.

I want the testing suite to be rock-solid. I’ve been much better at writing tests first over the last six months, and using Cucumber has made the test suite so much more reliable, but there’s still some way to go.

This is not a rewrite – it’s polishing.

I’ve been toying with this idea for a while, and it’s time to have a stab at it. Hopefully you can provide some assistance to do this.

If you’re working with Ruby and have been paying attention to Twitter or RSS feeds, then you’ve probably heard of Gemcutter. If not, it’s the latest flavour for publishing gems, and I’m finding the simplicity of it a delight.

Its appearance is doubly useful, as since GitHub has moved to Rackspace, automated gem building from projects has been disabled, perhaps never to return.

Getting Started

If you’ve not clicked the link to Gemcutter yet, let’s run down how easy it is to get it set up on your machine.

sudo gem install gemcutter
gem tumble

That’s it. Any future gem installs will look at Gemcutter’s growing library.

This doesn’t replace RubyForge or GitHub in your sources list, but it does set Gemcutter as the top priority – which is fine, as it has almost all of RubyForge’s gems ready for you anyway.

Publishing

Firstly, get yourself an account, click that confirmation email link, then hunt down a gem you want to publish, and run the following command:

gem push my-awesome-gem-0.0.1.gem

If it’s your first time, you’ll be asked for your login details, and then the gem is online and ready for anyone to download it. No waiting, no forms, no pain.

When you’ve got a new version, just run that same command again, pointing to the new gem file:

gem push my-awesome-gem-0.1.0.gem

One command. No authentication prompts. Available for everyone straight away. Awesome.

Migrating

If you’ve already got gems on RubyForge that you’d like to take ownership of on Gemcutter, it’s another one-step process:

gem migrate my-legacy-gem

You’ll be prompted for your RubyForge account name and password, and then Gemcutter does the rest.

Pretty easy, hey?

My Gems

Over this past weekend, I made Gemcutter the definitive source for all of my gems:

• Thinking Sphinx
• Thinking Sphinx for Sphinx 0.9.9
• Thinking Sphinx/Raspell plugin
• Riddle
• Ginger
• Fakeweb Matcher
• Postie

Incoming Confusion

There’s been some discussion about whether Gemcutter should replace the gem hosting facilities provided by Rubyforge. This may or may not happen, but it is confirmed that Gemcutter will be moving to rubygems.org soon.

Everything will still work fine via the gemcutter.org address, though, so don’t let that hold you back from diving in head first.

Hat-tip

The talented Nick Quaranto has been working hard on this for a while, and it’s great to see the Ruby community embrace Gemcutter so quickly. Here’s hoping it becomes the defacto gem source for all Ruby projects.

This morning I decided to get Nginx and Passenger set up in my local dev environment. I needed an easier way to test of Thinking Sphinx in such environments, but also, I find Nginx configuration syntax so much easier than Apache.

And of course, if I’ve got these components there, it would be great to use them to serve my development versions of rails applications, much like script/server. So I’ve got a script/nginx file that manages that as well. Sit tight, and let’s run through how to make this happen on your machine.

Be Prepared to Think

Firstly, a couple of notes on my development machine – I’m running Snow Leopard, and I compile libraries by source. No MacPorts, no custom versions of Ruby (yet). So, you may need to tweak the instructions to fit your own setup.

Installing Passenger

Before we get to Nginx, you’ll want the Passenger gem installed first.

sudo gem install passenger

You’ll also need to compile Passenger’s nginx module (keep an eye on the file path below – yours may be different):

cd /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/ext/nginx
sudo rake nginx

Installing Nginx

Nginx requires the PCRE library, so that adds an extra step, but it’s nothing too complex. Jump into Terminal or your shell application of choice, create a directory to hold all the source files, and step through the following commands (initially sourced from instructions by Wincent Colaiuta):

curl -O \
  ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.9.tar.bz2
tar xjvf pcre-7.9.tar.bz2
cd pcre-7.9
./configure
make
make check
sudo make install

That should be PCRE taken care of – I didn’t have any issues on my machine, hopefully it’s the same for you. Next up: Nginx itself. Grab the source:

curl -O \
  http://sysoev.ru/nginx/nginx-0.7.62.tar.gz
tar zxvf nginx-0.7.62.tar.gz
cd nginx-0.7.62

Let’s pause for a second before we configure things.

Even though the focus is having Nginx working in a local user setting, not system-wide, I wanted the default file locations to be something approaching Unix/OS X standards, so I’ve gone a bit crazy with configuration flags. You may want to alter them to your own personal tastes:

./configure \
  --prefix=/usr/local/nginx \
  --add-module=/Library/Ruby/Gems/1.8/gems/passenger-2.2.5/ext/nginx \
  --with-http_ssl_module \
  --with-pcre \
  --sbin-path=/usr/sbin/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --pid-path=/var/nginx/nginx.pid \
  --lock-path=/var/nginx/nginx.lock \
  --error-log-path=/var/nginx/error.log \
  --http-log-path=/var/nginx/access.log

And with that slightly painful step out of the way, let’s compile and install:

make
sudo make install

And just to test that Nginx is happy, run the following command:

nginx -v

Do you see the version details? Great! (If you don’t, then review the last couple of steps – did anything go wrong? Do you have the passenger module path correct?)

Configuring for a Rails App

The penultimate section – let’s create a simple configuration file for Rails applications, which can be used by our script/nginx file. I store mine at /etc/nginx/rails.conf, but you can put yours wherever you like.

daemon off;

events {
  worker_connections  1024;
}

http {
  include /etc/nginx/mime.types;
  
  # Assuming path has been set to a Rails application
  access_log            log/nginx.access.log;
  
  client_body_temp_path tmp/nginx.client_body_temp;
  fastcgi_temp_path     tmp/nginx.client_body_temp;
  proxy_temp_path       tmp/nginx.proxy_temp;
  
  passenger_root /Library/Ruby/Gems/1.8/gems/passenger-2.2.5;
  passenger_ruby /usr/bin/ruby;
  
  server {
    listen      3000;
    server_name localhost;
    
    root              public;
    passenger_enabled on;
    rails_env         development;
  }
}

script/nginx

The final piece of the puzzle – the script/nginx file, for the Rails app of your choice:

#!/usr/bin/env bash
nginx -p `pwd`/ -c /etc/nginx/rails.conf \
  -g "error_log `pwd`/log/nginx.error.log; pid `pwd`/log/nginx.pid;";

Don’t forget to make it executable:

chmod +x script/nginx

If you run the script right now, you’ll see a warning that Nginx can’t write to the global error log, but that’s okay. Even with that message, it uses a local error log. I’ve granted full access to the global log just to avoid the message, but if you know a Better Way, I’d love to hear it.

sudo chmod 666 /var/nginx/error.log

Head on over to localhost:3000 – and, after Passenger’s warmed up, your Rails app should load. Success!

Known Limitations

• The environment is hard-coded to development. If this is annoying, the easiest way around it is to create multiple versions of rails.conf, one per environment, and then use the appropriate one in your script/nginx file.
• You can’t specify a custom port either. Patches welcome.
• You won’t see the log output. Either tail log/development.log when necessary, or suggest a patch for script/nginx. I’d prefer the latter.

Beyond that, it should work smoothly. If I’m wrong, that’s what the comments form is for.

Also, you can find all of my config files, as well as other details of how I’ve set up my machine since installing Snow Leopard, on gist.github.com.

This weekend, there’s going to be a Rails Camp. In October, there’s going to be a Rails Camp. Then in November, there’s going to be a Rails Camp. That in itself is pretty freaking cool. What’s even cooler is that they’re in Maine, England and Australia respectively.

Definition

If you’re not quite sure what Rails Camps are – they’re unconference style events, held away from cities, generally without internet, on a weekend from Friday to Monday. The venues are usually scout halls or similar, so the name is slightly inaccurate – most people don’t bring tents, but sleep in dorm rooms instead.

Also, they are events for Rubyists of all level of experience – and not just focused on Rails either. Anything related to Ruby and development in general is a welcome topic for discussion.

The weekends are made up of plenty of hacking, socialising, talks, and partying. Alcohol and guitar hero usually feature. A ton of fun ensues.

Rails Camp New England

A quick rundown in chronological order: first up, from the 17th to 20th of July, is Rails Camp New England. This will (as far as I know) be the first Rails Camp in North America. We’ll be up in the middle of Maine, at the MountainView House (a bit different from most Rails Camp venues) in Bryant Pond.

Unfortunately, if you want to come to this camp, we’re all sold out. Let me know anyway, just in case someone drops out (although it is late notice).

Rails Camp UK 2

Building on the success of last year’s first UK Rails Camp, a second one has been put together by Tom Crinson out in Margate, Kent.

If you’re anywhere in the UK, or even Europe, you really should be keeping the weekend of the 16th to 19th of October free. In fact, go book your spot right now.

Rails Camp Australia 6

Last on this list is the original Rails Camp, that started back in June 2007, run by the inimitable Ben Askins. We’re returning to Melbourne (the host of the second camp, in November 2007), but this time we’re down by the beach in Somers.

November 20th to 23rd are the dates for this, and going by the names of confirmed attendees, alongside what looks to be an fantastic venue, it’s going to rock just as much as the last five (and quite possibly even more). Feel like booking your place?

For all of these events, you should beg, borrow or steal to get your hands on a ticket. The energy, intelligence and passion of past camps has been amazing (which is why I do my best to spread the word), and they are a breath of fresh air compared to the staid and structured setup of RailsConf and most other technical conferences.

Thanks to John Barton, Max Muermann, and Jason Crane for the photos above.

Update: Jesse’s talk is now online should you want to watch it.

Before this all fades from my brain in the post-conference haze, I just wanted to post a few thoughts on the final session of the amazing FutureRuby conference in Toronto. Jesse Hirsh delivered an impassioned argument against what he labels the Imperial Californian Idealogy.

As someone who is most definitely left-of-centre, the bulk of this talk appealed to me. A call to action, highlights of the flaws of the capitalism, railing against the environmental destruction caused by the pursuit of wealth – it ticked the boxes.

There were a lot of references to the prominent place in history that San Francisco holds when it comes to mining, wars and weaponry, corporations and politics. I won’t go into those, because I’ll probably get it wrong. I have no bones to pick with that part of the talk, though.

It wasn’t immediately clear to me that he was attacking (some of) the ideals put forward by Stewart Brand, Kevin Kelly and Chris Anderson – at first, I thought they were the alternative movement to the old San Francisco elite that was initially described. All three names hold some credibility for me, so that was an interesting twist.

That said – and if we take the points on Anderson’s Free as accurate – then I’m happy to buy into at least some of the criticism, particularly around the push for acceptance of waste. I agree completely that the wasteful nature of people has got us into the current ecological mess. It promotes a very narrow, selfish view, instead of a more holistic approach. Indeed, holistic solutions was the end point Hirsh was driving at.

My main issue? The revolutionary, us-vs-them vibe. It felt implied that we are the elite, the creators, the visionaries. The ones who know best, the ideal internet citizens. There’s enough division in the world as it is. Not that I think Jesse is advocating such an approach, but that’s how the message came across to me. Granted, the Snowcrash references were lost on me, so that didn’t help.

Besides, revolutions are (more often than not) ineffective routes to change. Evolution is the road I much prefer to walk down.

Anyway, I’m off to the afterparty to discuss all this further with other attendees. If you weren’t there, then this probably doesn’t make much sense to you – watch the video, see if that helps.