Freelancing Gods 2010

It’s just over two months since I asked for – and received – support from the Ruby community to work on Thinking Sphinx for a month. A review of this would be a good idea, hey?

I’m going to write a separate blog post about how it all worked out, but here’s a long overview of the new features.

Internal Cucumber Cleanup

This one’s purely internal, but it’s worth knowing about.

Thinking Sphinx has a growing set of Cucumber features to test behaviour with a live Sphinx daemon. This has made the code far more reliable, but there was a lot of hackery to get it all working. I’ve cleaned this up considerably, and it is now re-usable for other gems that extend Thinking Sphinx.

External Delta Gems

Of course, it was my own re-use that was driving that need: I wanted to use it in gems for the delayed job and datetime delta approaches.

There was a clear need for removing these two pieces of functionality from Thinking Sphinx: to keep the main library as slim as possible, and to make better use of gem dependencies, allowing people to use whichever version of delayed job they like.

So, if you’ve not upgraded in a while, it’s worth re-reading the delta page of the documentation, which covers the new setup pretty well.

Testing Helpers

Internal testing is all very well, but what’s much more useful for everyone using Thinking Sphinx is the new testing class. This provides a clean, simple interface for processing indexes and starting the Sphinx daemon.

There’s also a Cucumber world that simplifies things even further – automatically starting and stopping Sphinx when your features are run. I’ve been using this myself in a project over the last few days, and I’m figuring out a neat workflow. More details soon, but in the meantime, have a read through the documentation.

No Vendored Code for Gems

One of the uglier parts of Thinking Sphinx is the fact that it vendors Riddle and AfterCommit (and for a while, Delayed Job), two essential libraries. This is not ideal at all, particularly when gem dependencies can manage this for you.

So, Thinking Sphinx no longer vendors these libraries if you install it as a gem – instead, the riddle and after_commit gems will get brought along for the ride.

The one catch is that they’re still vendored for plugin installations. I recommend people use Thinking Sphinx as a gem, but there are valid reasons for going down the plugin path.

Default Sphinx Scopes

Thanks to some hard work by Joost Hietbrink of the Netherlands, Thinking Sphinx now supports default sphinx scopes. All I had to do was merge this in – Joost was the first contributor to Thinking Sphinx (and there’s now over 100!), so he knows the code pretty well.

In lieu of any real documentation, here’s a quick sample – define a scope normally, and then set it as the default:

class Article < ActiveRecord::Base
  # ...
  
  sphinx_scope(:by_date) {
    {:order => :created_at_}
  }
  
  default_sphinx_scope :by_date
  
  # ...
end

Thread Safety

I’ve made some changes to improve the thread safety of Thinking Sphinx. It’s not perfect, but I think all critical areas are covered. Most of the dynamic behaviour occurs when the environment is initialised anyway.

That said, I’m anything but an expert in this area, so consider this a tentative feature.

Sphinx Select Option

Another community-sourced patch – this time from Andrei Bocan in Romania: if you’re using Sphinx 0.9.9, you can make use of its custom select statements:

Article.search 'pancakes',
  :sphinx_select => '*, @weight + karma AS superkarma'

This is much like the :select option in ActiveRecord – but make sure you use :sphinx_select (as the former gets passed through to ActiveRecord’s find calls).

Multiple Index Support

You can now have more than one index in a model. I don’t see this as being a widely needed feature, but there’s definitely times when it comes in handy (such as having one index with stemming, and one without). The one thing to note is that all indexes after the first one need explicit names:

define_index 'stemmed' do
  # ...
end

You can then specify explicit indexes when searching:

Article.search 'pancakes',
  :index => 'stemmed_core'
Article.search 'pancakes',
  :index => 'article_core,stemmed_core'

Don’t forget that the default index name is the model’s name in lowercase and underscores. All indexes are prefixed with _core, and if you’ve enabled deltas, then a matching index with the _delta suffix exists as well.

Building on from this, you can also now have indexes on STI subclasses when superclasses are already indexed.

While the commits to this feature are mine, I was reading code from a patch by Jonas von Andrian – so he’s the person to thank, not me.

Lazy Initialisation

Thinking Sphinx needs to know which models have indexes for searching and indexing – and so it would load every single model when the environment is initialised, just to figure this out. While this was necessary, it also is slow for applications with more than a handful of models… and in development mode, this hit happens on every single page load.

Now, though, Thinking Sphinx only runs this load request when you’re searching or indexing. While this doesn’t make a difference in production environments, it should make life on your workstations a little happier.

Lazy Index Definition

In a similar vein, anything within the define_index block is now evaluated when it’s needed. This means you can have it anywhere in your model files, whereas before, it had to appear after association definitions, else Thinking Sphinx would complain that they didn’t exist.

This feature actually introduced a fair few bugs, but (thanks to some patience from early adopters), it now runs smoothly. And if it doesn’t, you know where to find me.

Sphinx Auto-Version detection

Over the course of the month, Thinking Sphinx and Riddle went through some changes as to how they’d be required (depending on your version of Sphinx). First, there was separate gems for 0.9.8 and 0.9.9, and then single gems with different require statements. Neither of these approaches were ideal, which Ben Schwarz clarified for me.

So I spent a day or two working on a solution, and now Thinking Sphinx will automatically detect which version you have installed. You don’t need any version numbers in your require statements.

The one catch with this is that you currently need Sphinx installed on every machine that needs to know about it, including web servers that talk to Sphinx on a separate server. There’s an issue logged for this, and I’ll be figuring out a solution soon.

Sphinx 0.9.9

This isn’t quite a Thinking Sphinx feature, but it’s worth noting that Sphinx 0.9.9 final release is now available. If you’re upgrading (which should be painless), the one thing to note is that the default port for Sphinx has changed from 3312 to 9312.

Upgrading

If you want to grab the latest and greatest Thinking Sphinx, then version 1.3.14 is what to install. And read the documentation on upgrading!

At a recent Melbourne Ruby meet, I was asked to speak about my travelling freelancer lifestyle, and the talk was recorded. I feel a little self-conscious about the topic, but perhaps you’ll find it interesting.

Massive thanks to James Healy for not only recording the talks that night, but producing the neat slides-and-video output. I’m looking forward to the Melbourne Ruby channel building up a good collection of sessions.

Also: I’ll be posting a review of my month working on Thinking Sphinx soon.

Update: I’ve now hit my target. If you want to donate more, I won’t turn you away, but perhaps you should send those funds to other worthy open source projects, or a local charity. A massive thank you to all who have pitched in to the pledgie, your generosity and support is amazing.

Over the past two years, Thinking Sphinx has grown massively – in lines of code, in the numbers of users, in complexity, in time required to support it. I’m regularly amazed and touched by the recommendations I see on Twitter, and the feedback I get in conversations. The fact that there’s been almost one hundred contributors is staggering.

It’s not all fun and games, though… there’s still plenty of features that can be added, and bugs to be fixed, and documentation to write. So, what I’d really like to do is spend November working close to full-time on just Thinking Sphinx. I have a long task list. All I need is a bit of financial help to cover living expenses.

I have an existing pledgie tied to the GitHub project, currently sitting on $600. If I can get another $2000, then I won’t have to worry at all about how I’m going to pay bills or rent for November. Even $1400 will make it viable for me, albeit maybe with some help from my savings.

If you or your workplace can make a donation, that would be very much appreciated. I’m happy to provide weekly updates on where things are at if people request it – but of course, watching the GitHub projects for Thinking Sphinx itself and the documentation site is the most reliable way to keep an eye on my progress.

I’m hoping to get Thinking Sphinx to a point where the documentation is by far the best place for support, and it’s only the really tricky problems (and bug reports) that end up in my inbox.

I want it to be a model Ruby library that doesn’t get in your way, is as fast as possible, and plays nicely with other libraries.

I want the testing suite to be rock-solid. I’ve been much better at writing tests first over the last six months, and using Cucumber has made the test suite so much more reliable, but there’s still some way to go.

This is not a rewrite – it’s polishing.

I’ve been toying with this idea for a while, and it’s time to have a stab at it. Hopefully you can provide some assistance to do this.

If you’re working with Ruby and have been paying attention to Twitter or RSS feeds, then you’ve probably heard of Gemcutter. If not, it’s the latest flavour for publishing gems, and I’m finding the simplicity of it a delight.

Its appearance is doubly useful, as since GitHub has moved to Rackspace, automated gem building from projects has been disabled, perhaps never to return.

Getting Started

If you’ve not clicked the link to Gemcutter yet, let’s run down how easy it is to get it set up on your machine.

sudo gem install gemcutter
gem tumble

That’s it. Any future gem installs will look at Gemcutter’s growing library.

This doesn’t replace RubyForge or GitHub in your sources list, but it does set Gemcutter as the top priority – which is fine, as it has almost all of RubyForge’s gems ready for you anyway.

Publishing

Firstly, get yourself an account, click that confirmation email link, then hunt down a gem you want to publish, and run the following command:

gem push my-awesome-gem-0.0.1.gem

If it’s your first time, you’ll be asked for your login details, and then the gem is online and ready for anyone to download it. No waiting, no forms, no pain.

When you’ve got a new version, just run that same command again, pointing to the new gem file:

gem push my-awesome-gem-0.1.0.gem

One command. No authentication prompts. Available for everyone straight away. Awesome.

Migrating

If you’ve already got gems on RubyForge that you’d like to take ownership of on Gemcutter, it’s another one-step process:

gem migrate my-legacy-gem

You’ll be prompted for your RubyForge account name and password, and then Gemcutter does the rest.

Pretty easy, hey?

My Gems

Over this past weekend, I made Gemcutter the definitive source for all of my gems:

• Thinking Sphinx
• Thinking Sphinx for Sphinx 0.9.9
• Thinking Sphinx/Raspell plugin
• Riddle
• Ginger
• Fakeweb Matcher
• Postie

Incoming Confusion

There’s been some discussion about whether Gemcutter should replace the gem hosting facilities provided by Rubyforge. This may or may not happen, but it is confirmed that Gemcutter will be moving to rubygems.org soon.

Everything will still work fine via the gemcutter.org address, though, so don’t let that hold you back from diving in head first.

Hat-tip

The talented Nick Quaranto has been working hard on this for a while, and it’s great to see the Ruby community embrace Gemcutter so quickly. Here’s hoping it becomes the defacto gem source for all Ruby projects.

This weekend, there’s going to be a Rails Camp. In October, there’s going to be a Rails Camp. Then in November, there’s going to be a Rails Camp. That in itself is pretty freaking cool. What’s even cooler is that they’re in Maine, England and Australia respectively.

Definition

If you’re not quite sure what Rails Camps are – they’re unconference style events, held away from cities, generally without internet, on a weekend from Friday to Monday. The venues are usually scout halls or similar, so the name is slightly inaccurate – most people don’t bring tents, but sleep in dorm rooms instead.

Also, they are events for Rubyists of all level of experience – and not just focused on Rails either. Anything related to Ruby and development in general is a welcome topic for discussion.

The weekends are made up of plenty of hacking, socialising, talks, and partying. Alcohol and guitar hero usually feature. A ton of fun ensues.

Rails Camp New England

A quick rundown in chronological order: first up, from the 17th to 20th of July, is Rails Camp New England. This will (as far as I know) be the first Rails Camp in North America. We’ll be up in the middle of Maine, at the MountainView House (a bit different from most Rails Camp venues) in Bryant Pond.

Unfortunately, if you want to come to this camp, we’re all sold out. Let me know anyway, just in case someone drops out (although it is late notice).

Rails Camp UK 2

Building on the success of last year’s first UK Rails Camp, a second one has been put together by Tom Crinson out in Margate, Kent.

If you’re anywhere in the UK, or even Europe, you really should be keeping the weekend of the 16th to 19th of October free. In fact, go book your spot right now.

Rails Camp Australia 6

Last on this list is the original Rails Camp, that started back in June 2007, run by the inimitable Ben Askins. We’re returning to Melbourne (the host of the second camp, in November 2007), but this time we’re down by the beach in Somers.

November 20th to 23rd are the dates for this, and going by the names of confirmed attendees, alongside what looks to be an fantastic venue, it’s going to rock just as much as the last five (and quite possibly even more). Feel like booking your place?

For all of these events, you should beg, borrow or steal to get your hands on a ticket. The energy, intelligence and passion of past camps has been amazing (which is why I do my best to spread the word), and they are a breath of fresh air compared to the staid and structured setup of RailsConf and most other technical conferences.

Thanks to John Barton, Max Muermann, and Jason Crane for the photos above.

In case you’ve not booked your ticket yet, but were considering coming and needed one more reason: I’ll be giving a tutorial on Sphinx at RailsConf in Las Vegas next month. To get 15% discount, use the code ‘RC09FOS’. (O’Reilly seem to hand out codes everywhere. If you paid full price this time, keep that in mind for next year.)

If you will be there, I’m more than happy to have a chat – about Sphinx or anything else – so let me know if you want to meet up.

Crowdsourced Research

For those of you who know Sphinx and Thinking Sphinx already, I’d love to hear about the things you found a bit difficult when learning. What are the topics I should make sure I cover in my tutorial, that are maybe lacking in documentation?

There’s been a bit of changes under the hood with Thinking Sphinx lately, and some of the more recent commits are pretty useful.

Small Stuff

First off, something neat but minor – you can now use decimal, date and timestamp columns as attributes – the plugin automatically maps those to float and datetime types as needed.

There’s also now a cucumber-driven set of feature tests, which can run on MySQL and PostgreSQL. While that’s not important to most users, it makes it much less likely that I’ll break things. It’s also useful for the numerous contributors – just over 50 people as of this week! You all rock!

New Delta Possibilities

The major changes are around delta indexing, though. As well as the default delta column approach, there’s now two other methods of getting your changes into Sphinx. The first, requested by some Ultrasphinx users, and heavily influenced by a fork by Ed Hickey, is datetime-driven deltas. You can use a datetime column (the default is updated_at), and then run the thinking_sphinx:index:delta rake task on a regular basis to load recent changes into Sphinx.

Your define_index block would look something like the following:

define_index do
  # ... field and attribute definitions
  
  set_property :delta => :datetime, :threshold => 1.day
end

If you want to use a column other than updated_at, set it with the :delta_column option.

The above situation is if you’re running the rake task once a day. The more often you run it, the lower you can set your threshold. This is a bit different to the normal delta approach, as changes will not appear in search results straight away – only whenever the rake task is run.

Delayed Reaction

One of the biggest complaints with the default delta structure is that it didn’t scale. Your delta index got larger and larger every time records were updated, and that meant each change got slower and slower, because the indexing time increased. When running multiple servers, you could get a few indexer processes running at once. That ain’t good.

So now, we have delayed deltas, using the delayed_job plugin. You’ll need to have the job queue being processed (via the thinking_sphinx:delayed_delta rake task), but everything is pushed off into that, instead of overloading your web server. It means the changes take slightly longer to get into Sphinx, but that’s almost certainly not going to be a problem.

Firstly, you’ll need to create the delayed_jobs table (see the delayed_job readme for example code), and then change your define_index block so it looks something like this:

define_index do
  # ... field and attribute definitions
  
  set_property :delta => :delayed
end

Riddle Update

As part of the restructuring over the last couple of months, I’ve also added some additional code to Riddle, my Ruby API for Sphinx. It now has objects to represent all of the configuration elements of Sphinx (ie: settings for sources, indexes, indexer and searchd), and can generate the configuration file for you. This means you don’t need to worry about doing text manipulation, just do everything in neat, clean Ruby.

Documentation on this is non-existent, mind you, but the source shouldn’t be too hard to grok. I also need to update Thinking Sphinx’s documentation to cover the delta changes – for now, this blog post will have to do. If you get stuck, check out the Google Group.

Sphinx 0.9.9

One more thing: Thinking Sphinx and Riddle now both have Sphinx 0.9.9 branches – not merged into master, as most people are still using Sphinx 0.9.8, but you can find both code sets on GitHub.

Context

The fourth Australian Rails Camp happened back in the middle of November – and it was unsurprisingly and extremely enjoyably awesome, just like the previous four. Ryan and Anthony did a sterling job with putting it all together.

I probably talked a bit too much – I certainly felt I had more than my fair share of peoples’ focus – and while I rabbited on about Sphinx and Ginger, the topic I really enjoyed ranting about was freelancing, because it became far less about me, and far more about sharing the wealth of everybody’s experiences. I provided a few starting points, and then wise RORO minds added their own thoughts and opinions.

I can’t reproduce all that here, though. I wouldn’t do it justice. What I can do is go over the same notes I had then, and you can add your 2 cents (or five dollars) in the comments.

Freelancing Maths

One of the first things you need to be aware of, when you start freelancing, is how much to charge. I didn’t have a clue, but some more business-minded friends put me on the right track, so I’m sharing their advice here – don’t give me any credit for it.

So, let’s assume you want to start freelancing, and you have a target of earning $80,000 over the year (yes, some of you may say that’s too low, but others will say it’s too high – it’s just an example, okay?). You can use this as a basis for figuring out an hourly rate. There’s 52 weeks in a year, 5 days in a week, and 8 hours in a day…

 52 weeks
x 5  days
x 8 hours
x ?  rate
_________
   80,000

But wait a second – are you really going to work all of those 52 weeks? I doubt it. You’ll need time off for annual leave, sick leave and public holidays – the times when an employer would still pay you when you’re not slaving away. Australian annual leave is four weeks, sick leave is usually two, let’s add in another one for public holidays, and that brings us down to 45.

 45 weeks
x 5  days
x 8 hours
x ?  rate
_________
   80,000

What are the odds you’re going to have work all the time though, and are you really going to have eight billable hours each day? Unless you’re some sort of machine, the answer’s no, trust me. So lets drop eight down to six.

     45 weeks
x     5  days
x     6 hours
x 59.25  rate
_____________
       80,000

One thing we’ve missed in our calculations is superannuation. Again, using Australia as the example (because it’s all I can reliably comment on), you’re supposed to be putting away 9% of your income into your super account. Let’s factor that in:

     45 weeks
x     5  days
x     6 hours
x 64.59  rate
_____________
       87,200

Okay, so we can get an hourly rate of about $65 from that maths. And that could be fine… but maybe you’ve been eyeing off RailsConf or RubyConf or other such events. They’re not cheap – and hopefully employers would normally fork out the cash to get you there. You’re the employer now, so how are you going to afford it? Add an allowance into your calculations.

Again, due to the remoteness of Australia, it’s extra expensive to get to any of the major Ruby conferences. If we assume you’ll get to two of them (again, could be extravagant for some of you, but this is all hypothetical), then I’m adding a touch over $12,000 – flights, hotels, insurance, the conference tickets – to bring us to a nice round $100,000 target.

Also, I’ve dropped the number of weeks down another two – it’s not like you’ll be getting anything done for your clients as you jet around.

     43 weeks
x     5  days
x     6 hours
x 77.52  rate
_____________
      100,000

Okay, our final hourly rate is about $77.50.

I know a lot of the more experienced developers are looking at that value and thinking it’s pretty low – and going by market rates (for Ruby developers), it’s definitely below average. Some say a good ballpark figure for a decent Rails developer is $100/hour – USD or AUD (remember when the two currencies were almost on par?). This doesn’t mean you should charge that much (or that little) – but it should factor into your thinking.

All that said, you need to be comfortable with what you’re billing your time at, but don’t be afraid to charge what you’re worth. If the idea of having more cash than you expect scares you, there’s plenty of charities who would like to be your friend. Or, you could just work less, and spend the extra time on cool things (and they don’t even have to involve code!)

Freelancing Profile

Knowing what to charge is useful, but it’s not going to bring in the clients. Being known will help that problem, though – and there’s a few things you can do to help that.

Blog

Interpret how you will – a normal blog, twitter, tumblelog, even gists and pasties – sharing your ideas and knowledge is a great way to get your name known to others. It also helps build some human connections, via comments, emails or directed tweets. If it is valuable, they will find you (and if you think they need help, use a site like RubyFlow or RubyCorner to bring in some eyeballs).

Talk

If there’s a neat bit of code you’ve found, library you’ve come across (or written), or knowledge you think is valuable to others, offer to talk about it! It can be at your local Ruby group, or at something like a Rails Camp or BarCamp, or if you’re really comfortable up on stage, think about applying for a RailsConf or RubyConf slot.

I’m not a natural public speaker – but my confidence has grown in leaps and bounds from giving talks to fellow developers. Granted, I need to build up a bigger repertoire of topics, but I’m a bit less nervous about standing up and announcing my thoughts and opinions to others. It all started with an email from Tim Lucas asking what I was going to talk about at the first Rails Camp – and now Rails Camp folks are probably sick of hearing my voice.

They know who I am, though, and they know what code I’ve written. And that’s led to a referral or two for Rails work (usually Sphinx-related).

Socialise

Networking is a dirty word – and I can see how building connections with others for the purpose of connections, instead of meeting cool people, is a bit dirty. The much more fun alternative is to socialise – go out to social events, find those drinks happening in the evenings of conferences, have a conversation with a person you’ve not met before at your local Ruby meet.

Down the track, you will find these people may throw work your way – or maybe you’ll just learn cool new ways to code, or share some of your own knowledge, or make a good friend. All chalked up as wins in my book.

Release Code

Releasing your own code – from snippets to plug ins to full-blown applications – is a great way to show peers that you know what you’re talking about. It also shows potential clients that too, and reaffirms that you’re worth the rate you’re charging, and that you can be creative.

In my own case, I’ve done the occasional bit of Sphinx consulting due to my work on Thinking Sphinx.

Coincidentally, doing all these things are rewarding in and of themselves. I don’t do them to bring in work, I do them because they’re fun and I meet awesome people, which is (I think) the best approach. The opportunities they lead to are just an added bonus.

Your Turn

So, what’s your advice to a budding freelancer? Is there anything here that’s a bit Ruby or developer-centric? Any more general suggestions to keep in mind?

Also, please keep in mind I’m not an expert. I think the above advice is useful, but it is just advice. There’s no hard and fast rules that should be followed.

And the name of this blog has nothing to do with my work lifestyle, but the idea of deities who freelance for each other. Don’t take it as an indication of my ego. Honest.

A quick note to let anyone using (or interested in using) Sphinx and/or Thinking Sphinx that my Peepcode PDF has just been published, and contains a wealth of information on how to mix Sphinx with ActiveRecord (via Rails or Merb).

It’s been great working with Geoffrey Grosenbach to get this written up, and I’m pretty stoked to see the final results – hopefully others will enjoy it as well.

Also, a massive thank you to all the contributors to Thinking Sphinx – it wouldn’t be quite so cool if it wasn’t for all the patches (facilitated by GitHub’s forking).

A quick question to fellow coders…

Unsurprisingly, there’s a dearth of Ruby developers in Cambodia. I imagine the situation is pretty similar in other developing nations. PHP and Visual Basic seem to be the common languages in the small tech community here.

I’m currently working on building a website for one of the local NGOs here – and of course, Rails is my preferred framework. But looking forward, I don’t wish to be providing ongoing support for the site – and the client shares that sentiment. So to make it easier for local developers to take over, should I be considering using PHP for the project instead?

I have offered to help the IT guy at this organisation learn Ruby, but he won’t be there forever as well. And they’re a small NGO – they don’t have the cash to throw around hiring super-skilled developers. The project itself is pro-bono.

So, what would you do, given the circumstances?

(And for the record, it’s very likely I’ll stick with Ruby – using the Radiant CMS – but I’m interested in others’ opinions.)

One of the things I’ve tried to do with Thinking Sphinx has been to keep it friendly for a few different versions of Rails/ActiveRecord: 1.2.6, 2.0.x and 2.1.×. It’s not that easy, because I know I don’t think about which methods are pre-Rails-2.0, and which are even newer. And add on top of that the large (and very welcome) number of contributors, and it becomes a bit of a headache.

I was introduced to a potential aid for this at RailsConf EU, by fellow Australian Ian White, in the form of his library Garlic. It takes Rails plugins, instantiates them within Rails applications for each defined environment, and then runs their specs.

That in itself is pretty damn cool – except I don’t run Thinking Sphinx’s specs from within Rails – and it’s really only reliant on ActiveRecord (so it can be used in Merb). So, I’ve spent an afternoon coding up something slightly different, but in something of an acknowledgement to Ian’s library, my solution is called Ginger.

Now, it’s not exhaustively tested, but I’ve got it to the point where it runs as I would expect it to with Thinking Sphinx, so consider it ready for release.

If you’re interested in using it, here’s the basic guidelines. First up, install the gem:

sudo gem install freelancing-god-ginger --source=http://gems.github.com

(GitHub doesn’t have the gem loaded at the moment though, so you’ll have to install it manually)

git clone git://github.com/freelancing-god/ginger.git
cd ginger
gem build ginger.gemspec
sudo gem install --local ginger-1.0.0.gem

Then, in your spec_helper.rb file (or equivalent – this should work for Test::Unit as well as RSpec), add:

require 'ginger'

You’ll want to make sure that require is before any other require calls to libraries you want to test multiple versions of.

Next, you need to create a scenario file, outlining the different testing situations Ginger should run through. These details go in a file called ginger_scenarios.rb in the root of your project. Here’s my current one for Thinking Sphinx:

require 'ginger'

Ginger.configure do |config|
  config.aliases["active_record"] = "activerecord"
  
  ar_1_2_6 = Ginger::Scenario.new
  ar_1_2_6[/^active_?record$/] = "1.15.6"
  
  ar_2_0_2 = Ginger::Scenario.new
  ar_2_0_2[/^active_?record$/] = "2.0.2"
  
  ar_2_1_1 = Ginger::Scenario.new
  ar_2_1_1[/^active_?record$/] = "2.1.1"
  
  config.scenarios << ar_1_2_6 << ar_2_0_2 << ar_2_1_1
end

I’m adding three different scenarios – one for each version of ActiveRecord I want to test. I’m also adding an alias, as sometimes people have an underscore in the require calls, and while the require method isn’t fussy, the gem method is. That’s also the reason for the regular expressions, but strings will work as well.

If you want, you can have multiple gem requirements for each scenario – Ginger::Scenario is subclassed from Hash, so just add more key/value pairs as needed.

sphinx_scenario = Ginger::Scenario.new
sphinx_scenario["riddle"] = "0.9.8" 
sphinx_scenario["thinking_sphinx"] = "0.9.8"

And don’t forget to add each scenario to @config@’s collection.

The last step is the most important – running your tests through each scenario! This is done with the ginger command line tool – any arguments given to it get echoed onto rake, so in theory it should work with whatever rake task you like. I’ve only tested it with RSpec though.

ginger spec
ginger test
ginger spec:unit

This was built to scratch my itch, obviously, but happy to accept patches or suggestions. GitHub, as always, is the best way to do this – fork to your heart’s content.

Just over a week ago, the first Rails Camp in the UK was held in Downe, outside Orpington – and I think it was a fantastic success (having been the organiser though, obviously there is some bias).

We had quite an international flavour to the weekend. Of the 30 or so who attended, several were from around Europe, alongside the local British, and a few of us Australians to round it out.

In true Rails Camp style, around the beer, pizza and games, much hacking and discussion was had – assisted by the *jour gems, twetter and SubEthaEdit. Plenty of cool projects were displayed and created – topics ranging from RSpec to EXTJS to in-memory models to plugins to CouchDB to approaches for better browser-server polling (with a neat browser game as an example).

One of the cool creations of this Rails Camp has gone live. The collective talent of the RailsLove guys and Rany Keddo produced a forkable lists web app called Don’t Forget The Wurst – and features William Shatner, which just adds several levels of awesomeness to an already neat idea.

Massive thanks to all who came along and made it such a fantastic weekend – I’m looking forward to hearing about another Rails Camp in this part of the world (even if I won’t be able to attend it).

Also, if you’re in Europe, you might want to check out the German and Danish Rails Camps, which will be happening later this year. Australians, Rails Camp #4 will be happening in November (details are almost finalised). Everyone else: I highly recommend making one happen near you. There’s now a group of us who have dabbled in the organisation of them, and we’re more than happy to help however we can to get more of them happening around the world. It’s not too hard, and it’s an awesome way of strengthening your local Ruby community.

A few random items:

• There’s a Rails Camp happening in Denmark. How awesome is that!?
• I’ve added an About Me page to this blog – filled with opinions. You have been warned.
• I’m talking at NYC Ignite – come along and listen to me talk quickly about non-Ruby stuff for five minutes, if you’re near that part of the world.
• Joss Whedon is awesome.
• So is Pixar. You must see Wall-E. Easily the best film I’ve seen all year.

Following in the steps of the Australian Rails Camps, it’s now time to announce the first UK edition. Running from Friday the 15th to Monday the 18th of August, it will be an extended weekend of hacking, talking, eating, drinking and games, with a bunch of smart and passionate Ruby developers.

Even though the name is “Rails Camp”, previous camps have included talks on topics from Merb to Rack to Extreme Programming – all topics somewhat related to Ruby are welcome.

If you’d like to come along, I’d recommend registering soon, as there’s a very limited number of places.

Okay, it’s well past time for the companion piece to my Sphinx Primer – let’s go through the basic process of using Thinking Sphinx with Rails.

Just to recap: Sphinx is a search engine that indexes data, and then you can query it with search terms to find out which documents are relevant. Why do you want to use it with Rails? Because it saves having to write messy SQL, and it’s so damn fast.

(If you’re getting a feeling of deja-vu, then it’s probably because you’ve read an old post on this blog that dealt with an old version of Thinking Sphinx. I’ve had a few requests for an updated article, so this is it.)

Installation

So: first step is to install Sphinx. This may be tricky on some systems – but I’ve never had a problem with it with Mac OS X or Ubuntu. My process is thus:

curl -O http://sphinxsearch.com/downloads/sphinx-0.9.8-rc2.tar.gz
tar zxvf sphinx-0.9.8-rc2.tar.gz
cd sphinx-0.9.8-rc2
./configure
make
sudo make install

If you’re using Windows, you can just grab the binaries.

Once that’s taken care of, you then want to take your Rails app, and install the plugin. If you’re running edge or 2.1, this is a piece of cake:

script/plugin install git://github.com/freelancing-god/thinking-sphinx.git

Otherwise, you’ve got a couple of options. The first is, if you have git installed, just clone to your vendor/plugins directory:

git clone git://github.com/freelancing-god/thinking-sphinx.git
  vendor/plugins/thinking-sphinx

If you’re not yet using git, then the easiest way is to download the tar file of the code. Try the following:

curl -L http://github.com/freelancing-god/thinking-sphinx/tarball/master
  -o thinking-sphinx.tar.gz
tar -xvf thinking-sphinx.tar.gz -C vendor/plugins
mv vendor/plugins/freelancing-god-thinking-sphinx* vendor/plugins/thinking-sphinx

Oh, and it’s worth noting: if you’re not using MySQL or PostgreSQL, then you’re out of luck – Sphinx doesn’t talk to any other relational databases.

Configuration

Next step: let’s get a model or two indexed. It might be worth refreshing your memory on what fields and attributes are for – can I recommend my Sphinx article (because I’m not at all biased)?

Ok, now let’s work with a simple Person model, and add a few fields:

class Person < ActiveRecord::Base
  define_index do
    indexes [first_name, last_name], :as => :name
    indexes location
  end
end

Nothing too scary – we’ve added two fields. The first is the first and last names of a person combined to one field with the alias ‘name’. The second is simply location.

Adding attributes is just as easy:

define_index do
  # ...
  
  has birthday
end

This attribute is the datetime value birthday (so you can now sort and filter your results by birthdays).

Managing Sphinx

We’ve set up a basic index – now what? We tell Sphinx to index the data, and then we can start searching. Rake is our friend for this:

rake thinking_sphinx:index
rake thinking_sphinx:start

Searching

Now for the fun stuff:

Person.search "Melbourne"

Or with some sorting:

Person.search "Melbourne", :order => :birthday

Or just people born within a 10 year window:

Person.search "Melbourne", :with => {:birthday => 25.years.ago..15.years.ago}

If you want to keep certain search terms to specific fields, use :conditions:

Person.search :conditions => {:location => "Melbourne"}

Just remember: :conditions is for fields, :with is for attributes (and :without for exclusive attribute filters).

Change

Your data changes – but unfortunately, Sphinx doesn’t update your indexes to match automatically. So there’s two things you need to do. Firstly, run rake thinking_sphinx:index regularly (using cron or something similar). ‘Regularly’ can mean whatever time frame you want – weekly, daily, hourly.

The second step is optional, but it’s needed to have your indexes always up to date. First, add a boolean column to your model, named ‘delta’, and have it default to true. Then, tell your index to use that delta field to keep track of changes:

define_index do
  # ...
  
  set_property :delta => true
end

Then you need to tell Sphinx about the updates:

rake thinking_sphinx:stop
rake thinking_sphinx:index
rake thinking_sphinx:start

Once that’s done, a delta index will be created – which holds any recent changes (since the last proper indexing), and gets re-indexed whenever a model is edited or created. This doesn’t mean you can stop the regular indexing, as that’s needed to keep delta indexes as small (and fast) as possible.

String Sorting

If you remember the details about fields and attributes, you’ll know that you can’t sort by fields. Which is a pain, but there’s ways around this – and it’s kept pretty damn easy in Thinking Sphinx. Let’s say we wanted to make our name field sortable:

define_index do
  indexes [first_name, last_name], :as => :name, :sortable => true
  
  # ...
end

Re-index and restart Sphinx, and sorting by name will work.

How is this done? Thinking Sphinx creates an attribute under the hood, called name_sort, and uses that, as Sphinx is quite fine with sorting by strings if they’re converted to ordinal values (which happens automatically when they’re attributes).

Pagination

Sphinx paginates automatically – in fact, there’s no way of turning that off. But that’s okay… as long as you can use your will_paginate helper, right? Never fear, Thinking Sphinx plays nicely with will_paginate, so your views don’t need to change at all:

<%= will_paginate @search_results %>

Associations

Sometimes you’ll want data in your fields (or attributes) from associations. This is a piece of cake:

define_index do
  indexes photos.caption, :as => :captions
  indexes friends.photos.caption, :as => :friends_photos
  
  # ...
end

Polymorphic associations are fine as well – but keep in mind, the more complex your index fields and attributes, the slower it will be for Sphinx to index (and you’ll definitely need some database indexes on foreign key columns to help it stay as speedy as possible).

Gotchas

In case things aren’t working, here’s some things to keep in mind:

• Added an attribute, but can’t sort or filter by it? Have you reindexed and restarted Sphinx? It doesn’t automatically pick up these changes.
• Sorting not working? If you’re specifying the attribute to sort by as a string, you’ll need to include the direction to sort by, just like with SQL: “birthday ASC”.
• Using name or id columns in your fields or attributes? Make sure you specify them using symbols, as they’re core class methods in Ruby.

define_index do
  indexes :name
  
  # ...
  
  has photos(:id), :as => :photo_ids
end

And Next?

I realise this article is pretty light on details – but if you want more information, the first stop should be the extended usage page on the Thinking Sphinx site, quickly followed by the documentation. There’s also an email list to ask questions on.

I’ve just started my round-the-world conferences-and-holiday adventure, and the first stop is RailsConf in Portland – so if you’re in town and see me wandering around looking rather cluelessly, please say hi.

Also, in case you’re on the Twitter bandwagon, you’ll find me with the creative nickname of pat.

In case you’ve not been watching every commit carefully flow through Thinking Sphinx on GitHub – PostgreSQL support has been added. I’ve done a little bit of testing, and I’ve had some excellent support from Björn Andreasson and Tim Riley, so I feel it’s ready for people to start kicking the tires.

I’m no PostgreSQL expert – I definitely fall into the n00b category – so if you think there’s better ways to do what I’m doing, would love to hear them.

If you’re a Ruby developer in or near Australia, I highly recommend attending RailsCamp number 3, which has just opened for registration. The first two were simply amazing, so I’m just a little annoyed that I can’t make it to this one (as I’ll be traveling overseas at the time). I’ve no doubt that this one will be just as fantastic – expect an extended weekend of hacking and talking with a bunch of smart, entertaining and passionate developers, and plenty of drinks and games thrown in for good measure.

You don’t need to be a Rails or Ruby genius to attend – just a desire to discuss, learn, teach and (most importantly) have fun.

Go register now.

Another small sphinx-related post.

In line with the first release candidate release of Sphinx 0.9.8 last week, I’ve updated both my API, Riddle, and my plugin, Thinking Sphinx, to support it. Also, for those inclined, you can now get Riddle as a gem.

I’m slowly making progress on some major changes to Thinking Sphinx, so hopefully I’ll have something cool to show people soon. Oh, but some features that aren’t reflected in the documentation: most of Sphinx’s search options can be passed through when you call Model.search – including :group_by, :group_function, :field_weights, :sort_mode, etc. Consider it an exercise for the reader to figure out the details until I get around to improving the docs.

Here’s a collection of notes made while I was working on migrating this blog from Rails to Merb (no, the Merb version isn’t live yet). These are relevant to version 0.5.2 – but Merb’s moving so fast these days, I wouldn’t be surprised if much of this isn’t relevant any more.

Most of this you can find if you look through the documentation, but if you’ve not yet played with Merb, this will hopefully give you some idea of some of the smaller differences.

Filters

• Use before and after instead of before_filter and after_filter
• Instead of :except, use :exclude.
• To kill the chain of filters and not follow on to the action method, you need to use the throw method:

def confirm_user_session
  throw :halt, Proc.new { |controller|
    controller.redirect url(:new_session)
  } if current_user.nil?
end

Oh, and in case you missed it in the above sample, redirect_to becomes redirect.

Routes

Resource routing isn’t actually any different to Rails – but I had trouble finding any documentation for it, so it wasn’t obvious to start with. This code will work in both Merb and Rails (although in Rails r is usually referenced as map)

r.resources :posts do |post|
  post.resources :comments
end

However, using these routes is definitely different. The url_for, route_url and route_path methods aren’t around – you need to use Merb’s magical url method:

url(:post, @post@)
url(:new_session)

View Helpers

Helpers are pretty minimal in Merb – you don’t get any of the inflectors, or the number formatters. And no form helpers – not in the core gem, anyway. You can get some of those from the merb_helpers gem, but they don’t match Rails’ syntax and method naming, so it can take a bit of time to get this switched over, depending on the size of your app.

No support for Form Builders in that gem, by the way.

Some of the default helpers that do exist have different names to their Rails counterparts. A few of the ones I came across:

• content_tag => tag
• tag => open_tag
• content_for => throw_content
• javascript_include_tag => js_include_tag
• stylesheet_include_tag => css_include_tag

All your favourite plugins

Because Merb plugins are gems, anything you use as a plugin in Rails is pretty unlikely to be ported over. will_paginate was the main one for me, so I ended up pulling the files into my lib directory. Of course, that was only really useful when using ActiveRecord – DataMapper and Sequel users may have to get hacking into any ActiveRecord-focused plugins they want to use.

Partials

No more render :partial – you want something more like the following:

partial "comments/show", :with => @post.comments.active
partial "post", :with => @posts, :as => :post

Again – you’ll find most/all of this in the documentation, but the only potential show-stopper I found was plugins – the rest isn’t that that big a difference to Rails.

Okay, time for another code post. Prompted by a comment on an earlier post about Merb and MYOB, I thought I’d provide some more detail on how to talk to MYOB using Ruby, for any other people stuck in a similar position.

ODBC Bindings

If you like crafting your own SQL, this is the easiest approach. Firstly, you’ll need to download Christian Werner’s ODBC Bindings for Ruby, and install it using the following commands (or something along these lines) from within the source directory:

ruby extconf.rb
make
sudo make install

Documentation is a bit light on examples – and it’s really just a hook into the C libraries for ODBC, so good luck reading the source. Here’s some of the basic things you’ll need to do. (Don’t forget to require 'odbc' first, of course).

Setting up a Connection

You can either go through the DSN collection, or connect using the string name of the DSN (which in this case we’ll assume is ‘MYOB’):

dsn = ODBC.datasources.detect { |source| source.name == "MYOB" }
database = ODBC::Database.connect dsn

# or

database = ODBC::Database.connect "MYOB"

Note that the ODBC.datasources collection only contains system DSNs, not User-level ones. Not sure why, or how to access the latter.

SELECT Statements

statement = database.prepare "SELECT * FROM Customers"
statement.execute
statement.each_hash d |row|
  # results accessed as row["column_name"]
end
statement.drop

Make sure you drop the statement once you’re done with it, otherwise you’ll get complaints when you stop ruby that the statements weren’t all closed and dropped.

There’s other ways to access results – you can go through results as you see fit using fetch or fetch_hash, depending on whether you want an array or hash of the record.

INSERT Statements

With other SQL statements (although there’s no use of UPDATE or DEETE in MYOB connections), you don’t really care about the results, so that makes things a bit simpler, you can just call database.do("INSERT INTO Import_Customer_Cards (...) VALUES (...)"). The do method creates the statement, executes it and then drops it for you. MYOB makes this a little more complex though, as it all must be done in transactions – not that that’s a bad thing.

database.transaction do |env|
  env.do("INSERT INTO Import_Customer_Cards (...) VALUES (...)")
end

This won’t actually do anything though – and it won’t throw up an error or warning. The problem is you need to turn autocommit mode off, because the MYOB ODBC drivers don’t like it (it’s not a problem with Christian’s bindings). So, back when you create your connection:

database = ODBC::Database.connect "MYOB"
database.autocommit = false

ActiveRecord

If you prefer some level of abstraction above the messy SQL, then it might be worth looking at OpenLink’s ActiveRecord ODBCAdapter. I recommend the gem instead of the plugin, as the plugin takes it upon itself to modify your code, which I don’t like.

sudo gem install odbc-rails

Now, I’m not an ODBC expert, but there seems to be different flavours of ODBC connections possible – and MYOB’s is not one supported by this gem. So, if you’re using Rails, put the gem into vendor (and for Merb, into the local gems folder), then modify it with this patch. I make no promises for it being stable or reliable – but I’ve not had any problems yet.

Of course, using ActiveRecord is viable if you’re just reading data out – but if you want to write as well as read, then there’s issues. MYOB has separate write tables (prefixed with ‘Import_’), and they have denormalised schemas compared to the read-equivalents.

For database.yml, you’ll need something like the following:

development:
  adapter: odbc
  dsn: MYOB

Everything Else

In both the situations above, I’ve not put usernames or passwords into the connections – you can, but that is already handled by the ODBC DSN, so I keep my code that bit simpler.

For those who are new to coding for MYOB – it does cost money to get a developer account (several hundred dollars per year), which is the only way to get write access. I think read access is a once off fee of a couple of hundred dollars, but that’s specific to a MYOB file.

And if anyone is wondering – while I was initially using the odbc-rails library, I’ve now switched to constructing the SQL myself and just using the bindings, because of the table issues.