Freelancing Gods 2013

As I developed Flying Sphinx, I found myself both writing and consuming several APIs: from Heroku to Flying Sphinx, Flying Sphinx to Heroku, the flying-sphinx gem in apps to Flying Sphinx, Flying Sphinx to Sphinx servers, and Sphinx servers to Flying Sphinx.

None of that was particularly painful – but when Josh Kalderimis was improving the flying-sphinx gem, he noted that the API it interacts with wasn’t that great. Namely, it was inconsistent with what it returned (sometimes text status messages, sometimes JSON), it was sending authentication credentials as GET/POST parameters instead of in a header, and it wasn’t versioned.

I was thinking that given I control pretty much every aspect of the service, it didn’t matter if the APIs had versions or not. However, as Josh and I worked through improvements, it became clear that the apps using older versions of the flying-sphinx gem were going to have one expectation, and newer versions another. Versioning suddenly became a much more attractive idea.

The next point of discussion was how clients should specify which version they are after. Most APIs put this in the path – here’s Twitter’s as an example, specifying version 1:

https://api.twitter.com/1/statuses/user_timeline.json

However, I’d recently been working with Scalarium’s API, and theirs put the version information in a header (again, version 1):

Accept: application/vnd.scalarium-v1+json

Some research turned up a discussion on Hacker News about best practices for APIs – and it’s argued there that using headers keeps the paths focused on just the resource, which is a more RESTful approach. It also makes for cleaner URLs, which I like as well.

How to implement this in a Rails application though? My routing ended up looking something like this:

namespace :api do
  constrants ApiVersion.new(1) do
    scope :module => :v1 do
      resource :app do
        resources :indices
      end
    end
  end

  constraints ApiVersion.new(2) do
    scope :module => :v2
      resource :app do
        resources :indices
      end
    end
  end
end

The ApiVersion class (which I have saved to app/lib/api_version.rb) is where we check the version header and route accordingly:

class ApiVersion
  def initialize(version)
    @version = version
  end

  def matches?(request)
    versioned_accept_header?(request) || version_one?(request)
  end

  private

  def versioned_accept_header?(request)
    accept = request.headers['Accept']
    accept && accept[/application\/vnd\.flying-sphinx-v#{@version}\+json/]
  end

  def unversioned_accept_header?(request)
    accept = request.headers['Accept']
    accept.blank? || accept[/application\/vnd\.flying-sphinx/].nil?
  end

  def version_one?(request)
    @version == 1 && unversioned_accept_header?(request)
  end
end

You’ll see that I default to version 1 if no header is supplied. This is for the older versions of the flying-sphinx gem – but if I was starting afresh, I may default to the latest version instead.

All of this gives us URLs that look like something like this:

http://flying-sphinx.com/api/app
http://flying-sphinx.com/api/app/indices

My SSL certificate is locked to flying-sphinx.com – if it was wildcarded, then I’d be using a subdomain ‘api’ instead, and clean those URLs up even further.

The controllers are namespaced according to both the path and the version – so we end up with names like Api::V2::AppsController. It does mean you get a new set of controllers for each version, but I’m okay with that (though would welcome suggestions for other approaches).

Authentication is managed by namespaced application controllers – here’s an example for version 2, where I’m using headers:

class Api::V2::ApplicationController < ApplicationController
  skip_before_filter :verify_authenticity_token
  before_filter :check_api_params

  expose(:app) { App.find_by_identifier identifier }

  private

  def check_api_params
    # ensure the response returns with the same header value
    headers['X-Flying-Sphinx-Token'] = request.headers['X-Flying-Sphinx-Token']
    render_json_with_code 403 unless app && app.api_key == api_key
  end

  def api_token
    request.headers['X-Flying-Sphinx-Token']
  end

  def identifier
    api_token && api_token.split(':').first
  end

  def api_key
    api_token && api_token.split(':').last
  end
end

Authentication, in case it’s not clear, is done by a header named X-Flying-Sphinx-Token with a value of the account’s identifier and api_key concatenated together, separated by a colon.

(If you’re not familiar with the expose method, that’s from the excellent decent_exposure gem.)

So where does that leave us? Well, we have an elegantly namespaced API, and both versions and authentication is managed in headers instead of paths and parameters. I also made sure version 2 responses all return JSON. Josh is happy and all versions of the flying-sphinx gem are happy.

The one caveat with all of this? While it works for me, and it suits Flying Sphinx, it’s not the One True Way for API development. We had a great discussion at the most recent Rails Camp up at Lake Ainsworth about different approaches – at the end of the day, it really comes down to the complexity of your API and who it will be used by.

Just a quick note for anyone in or near Eastern Europe – I’ll be heading over to Kiev for RubyC in November. I’m going to be speaking there about how to build gems and the best practices when doing so.

So, if that interests you (or you’d just like to catch up or hear some of the other speakers talk about interesting Ruby-related topics), then hopefully I’ll see you there!

I spent a good part of last month writing my first Rails engine – although it’s not yet released and for a client, so I won’t talk about that too much here.

Very quickly in the development process, I was looking around on how to test Rails engines. It seemed that, beyond some basic unit tests, having a full Rails application within your test or spec directory was the accepted approach for integration testing.

That felt kludgy and bloated to me, so I decided to try something a little different.

The end goal was full stack testing in a clear and manageable fashion – writing specs within my spec directory, not a bundled Rails app’s spec directory. Capybara’s DSL would be nice as well.

This, of course, meant having a Rails application to test through – but it turns out you can get away without the vast majority of files that Rails generates for you. Indeed, the one file a Rails app expects is config/database.yml – and that’s only if you have ActiveRecord in play.

Enter Combustion – my minimal Rails app-as-a-gem for testing engines, with smart defaults for your standard Rails settings.

Setting It Up

A basic setup is as follows:

• Add the gem to your gemspec or Gemfile.
• Run the generator in your engine’s directory to get a small Rails app stub created: combust (or bundle exec combust if you’re referencing the git repository instead).
• Add Combustion.initialize! to your spec/spec_helper.rb (currently only RSpec is supported, but shouldn’t be hard to patch for TestUnit et al).

Here’s a sample spec_helper, mixing in Capybara as well:

require 'rubygems'
require 'bundler'

Bundler.require :default, :development

require 'capybara/rspec'

Combustion.initialize!

require 'rspec/rails'
require 'capybara/rails'

RSpec.configure do |config|
  config.use_transactional_fixtures = true
end

Putting It To Work

Firstly, you’ll want to make sure you’re using your engine within the test Rails application. The generator has likely added the hooks we need for this. If you’re adding routes, then edit spec/internal/config/routes.rb. If you’re dealing with models, make sure you add the tables to spec/internal/db/schema.rb. The README covers this a bit more detail.

And then, get stuck into your specs. Here’s a really simple example:

# spec/controllers/users_controller_spec.rb
require 'spec_helper'

describe UsersController do
  describe '#new' do
    it "runs successfully" do
      get :new

      response.should be_success
    end
  end
end

Or, using Capybara for integration:

# spec/acceptance/visitors_can_sign_up_spec.rb
require 'spec_helper'

describe 'authentication process' do
  it 'allows a visitor to sign up' do
    visit '/'

    click_link 'Sign Up'
    fill_in 'Name',     :with => 'Pat Allan'
    fill_in 'Email',    :with => 'pat@no-spam-please.com'
    fill_in 'Password', :with => 'chunkybacon'
    click_button 'Sign Up'

    page.should have_content('Sign Out')
  end
end

And that’s really the core of it. Write the specs you need to test your engine within the context of a full Rails application. If you need models, controllers or views in the internal application to fully test out your engine, then add them to the appropriate location within spec/internal – but only add what’s necessary.

Rack It Up

Oh, and one of my favourite little helpers is this: Combustion’s generator adds a config.ru file to your engine, which means you can fire up your test application in the browser – just run rackup and visit http://localhost:9292.

Caveats

As already mentioned, Combustion is built with RSpec in mind – but I will happily accept patches for TestUnit as well. Same for Cucumber – should work in theory, but I’m yet to try it.

It’s also written for Rails 3.1 – it may work with Rails 3.0 with some patches, but I very much doubt it’ll play nicely with anything before that. Still, feel free to investigate.

And it’s possible that this could be useful for integration testing for libraries that aren’t engines. If you want to try that, I’d love to hear how it goes.

Final Notes

So, where do we stand?

• You can test your engine within a full Rails stack, without a full Rails app.
• You only add what you need to your Rails app stub (that lives in spec/internal).
• Your testing code is DRYer and easier to maintain.
• You can use standard RSpec and Capybara helpers for integration testing.
• You can view your test application via Rack.

I’m not the first to come up with this idea – after I had finished Combustion, it was pointed out to me that Kaminari’s test suite does a similar thing (just not extracted out into a separate library). It wouldn’t surprise me if others have done the same – but in my searching, I kept coming across well-known libraries with full Rails apps in their test or spec directories.

If you think Combustion could suit your engine, please give it a spin – I’d love to have others kick the tires and ensure it works in a wider set of situations. Patches and feedback are most definitely welcome.

Just over two weeks ago, I released Flying Sphinx – which provides Sphinx search capability for Heroku apps. I’ll talk more about how I built it and the challenges faced at some point, but right now I just want to introduce the service and how you may go about using it.

Why Sphinx?

Perhaps you’re not familiar with Sphinx and how it can be useful. For those who are new to Sphinx, it’s a full-text search tool – think of your own personal Google for within your website. It comes with two main moving parts – the indexer tool for interpreting and storing your search data (indices), and the searchd tool, which runs as a daemon accepting search requests, and returns the most appropriate matches for a given search query.

In most situations, Sphinx is very fast at indexing your data, and connects directly to MySQL and PostgreSQL databases – so it’s quite a good fit for a lot of Rails applications.

Using Sphinx in Rails

I’ve written a gem, Thinking Sphinx, which integrates Sphinx neatly with ActiveRecord. It allows you to define indices in your models, and then use rake tasks to handle the processing of these indices, along with managing the searchd daemon.

If you want to install Sphinx, have a read through of this guide from the Thinking Sphinx documentation – in most cases it should be reasonably painless.

Installing Thinking Sphinx in a Rails 3 application is quite simple – just add the gem to your Gemfile:

gem 'thinking-sphinx', '2.0.5'

For older versions of Rails, the Thinking Sphinx docs have more details.

I’m not going to get too caught up in the details of how to structure indices – this is also covered within the Thinking Sphinx documentation – but here’s a quick example, for user account:

class User < ActiveRecord::Base
  # ...
  
  define_index do
    indexes name, :sortable => true
    indexes location
    
    has admin, created_at
  end
  
  # ...
end

The indexes method defines fields – which are the textual data that people can search for. In this case, we’ve got the user names and locations covered. The has method is for attributes – which are used for filtering and sorting (fields can’t be used for sorting by default). The distinction of fields and attributes is quite important – make sure you understand the difference.

Now that we have our index defined, we can have Sphinx grab the required data from our database, which is done via a rake task:

rake ts:index

What Sphinx does here is grab all the required data from the database, inteprets it and stores it in a custom format. This allows Sphinx to be smarter about ranking search results and matching words within your fields.

Once that’s done, we next start up the Sphinx daemon:

rake ts:start

And now we can search! Either in script/console or in an appropriate action, just use the search method on your model:

User.search 'pat'

This returns the first page of users that match your search query. Sphinx always paginates results – though you can set the page size to be quite large if you wish – and Thinking Sphinx search results can be used by both WillPaginate and Kaminari pagination view helpers.

Instead of sorting by the most relevant matches, here’s examples where we sort by name and created_at:

User.search 'pat', :order => :name
User.search 'pat', :order => :created_at

And if we only want admin users returned in our search, we can filter on the admin attribute:

User.search 'pat', :with => {:admin => true}

There’s many more options for search calls – the documentation (yet again) covers most of them quite well.

One more thing to remember – if you change your index structures, or add/remove index defintions, then you should restart and reindex Sphinx. This can be done in a single rake task:

rake ts:rebuild

If you just want the latest data to be processed into your indices, there’s no need to restart Sphinx – a normal ts:index call is fine.

Using Thinking Sphinx with Heroku

Now that we’ve got a basic search setup working quite nicely, let’s get it sorted out on Heroku as well. Firstly, let’s add the flying-sphinx gem to our Gemfile (below our thinking-sphinx reference):

gem 'flying-sphinx', '0.5.0'

Get that change (along with your indexed model setup) deployed to Heroku, then inform Heroku you’d like to use the Flying Sphinx add-on (the entry level plan costs $12 USD per month):

heroku addons:add flying_sphinx:wooden

And finally, let’s get our data on the site indexed and the daemon running:

heroku rake fs:index
heroku rake fs:start

Note the fs prefix instead of the ts prefix in those rake calls – the normal Thinking Sphinx tasks are only useful on your local machine (or on servers that aren’t Heroku).

When you run those rake tasks, you will probably see the following output:

Sphinx cannot be found on your system. You may need to configure the
following settings in your config/sphinx.yml file:
  * bin_path
  * searchd_binary_name
  * indexer_binary_name

For more information, read the documentation:
http://freelancing-god.github.com/ts/en/advanced_config.html

This is because Thinking Sphinx doesn’t have access to Sphinx locally, and isn’t sure which version of Sphinx is available. To have these warnings silenced, you should add a config/sphinx.yml file to your project, with the version set for the production environment:

production:
  version: 1.10-beta

Push that change up to Heroku, and you won’t see the warnings again.

For the more curious of you: the Sphinx daemon is located on a Flying Sphinx server, also located within the Amazon cloud (just like Heroku) to keep things fast and cheap. This is all managed by the flying-sphinx gem, though – you don’t need to worry about IP addresses or port numbers.

Also: the same rules apply with Flying Sphinx for modifying index structures or adding/removing index definitions – make sure you restart Sphinx so it’s aware of the changes:

heroku rake fs:rebuild

The final thing to note is that you’ll want the data in your Sphinx indices updated regularly – perhaps every day or every hour. This is best done on Heroku via their Cron add-on – since that’s just a rake task as well.

If you don’t have a cron task already, the following (perhaps in lib/tasks/cron.rake) will do the job:

desc 'Have cron index the Sphinx search indices'
task :cron => 'fs:index'

Otherwise, maybe something more like the following suits:

desc 'Have cron index the Sphinx search indices'
task :cron => 'fs:index' do
  # Other things to do when Cron comes calling
end

If you’d like your search data to have your latest changes, then I recommend you read up on delta indexing – both for Thinking Sphinx and for Flying Sphinx.

Further Sources

Keep in mind this is just an introduction – the documentation for Thinking Sphinx is pretty good, and Flying Sphinx is improving regularly. There’s also the Thinking Sphinx google group and the Flying Sphinx support site if you have questions about either, along with numerous blog posts (though the older they are, the more likely they’ll be out of date). And finally – I’m always happy to answer questions about this, so don’t hesitate to get in touch.

A couple of weeks ago, some dear friends of mine started a campaign on Kickstarter to raise funds for their cafe. Now, in most situations, you’d be correct to question whether such a campaign deserves to be on Kickstarter – why should people chip in to see a cafe come into being?

However, Kinyei’s cafe is quite special. To begin with, it’s in the town of Battambang, one of the regional hubs of Cambodia. And it’s not just a cafe – Kinyei are creating a space for co-working, collaboration and social ventures. They are supporting local Khmer entrepreneurs and community leaders in building businesses, running social awareness campaigns, hosting concerts, and conducting workshops. The introductory video does an excellent job at providing some context:

Kinyei have already been doing amazing things in Battambang – but this cafe will help them grow, and assist them in enabling local Khmer to grab opportunities with both hands. Cambodia is inundated with charities and aid organisations, and it’s extremely important that the Khmer people are given opportunities to take command of their lives.

So, if you’ve got some spare cash lying around after Christmas, perhaps you’d like to send it Kinyei’s way – it will definitely be put to good use.

I’ve just devoured the latest Quarterly Essay, titled Trivial Pursuit: Leadership and the End of the Reform Era, by political journalist George Megalogenis.

It’s an interesting read – it’s filled in a few holes in my political knowledge (though that’s not too hard to do), and it’s provided an intelligent take on the current leadership crisis Australian politics is facing.

There’s plenty of thoughts from it that are buzzing around my head which I’d like to share – and would appreciate any thoughts you may have.

Generational support in the Major Parties

The Greens are building a considerable supporter base within Generation Y – and people often don’t switch parties as they get older, so this could lead to them being a serious player in Australian politics within the coming decade. However, the Baby Boomers are predominantly Coalition supporters. Keeping in mind our ageing population, in our most recent federal election 46% of voters were aged 50 or older.

Then factor this in: the ALP have dominated in the eastern states in particular – and yet there’s a decent chance that all three of Victoria, New South Wales and Queensland will swich to Coalition Governments over the coming two years.

So: where does this leave the ALP on a Federal level?

Red and Blue States

Another divide is through the middle of the country: South Australia, Victoria, Tasmania and the ACT all had swings towards the ALP in the most recent election. NSW was rather ambivalent, whereas Queensland and Western Australia favoured the Coalition.

Yes, inner city electorates have always been progressive in comparison to their rural siblings. But arguably this is a little more distinct. How would you go about being a leader for all of Australia?

Lead by Example

Hawke, Keating and early Howard were seen as true leaders – they led by action (ie: reform). Later, Howard, Rudd, Gillard and Abbott have all played by the polls, instead of leading with their own ideals and conviction. The abandoning of the ETS is a prime example of how the second approach failed Labor.

Playing to the polls is not healthy – far better to lead, and bring the country with you. Give them something to vote for, instead of being the least-worst option.

Stick to your guns – people are more willing to forgive you if you’ve got clear convictions. Consider Howard’s GST policy as an example. I’d quote others from Hawke and Keating, but I really don’t have the history chops to be confident in what I’d be saying.

Faster! Faster!

The pace of the media cycle is dangerous – there is the expectation of constant news. Our political leaders must push back against this desire. They need to plan out policy carefully, and not bombard people with a whole lot of small pieces of information.

Depth instead of breadth should be the focus – but that’s a hard thing to manage given our short attention spans. (A curse of the internet, perhaps?)

Be Prepared to Negotiate

This isn’t something from the book, but I think hung parliaments are going to become far more common. The Greens are coming into their own, and the combined ALP and Coalition vote hasn’t come close to 90% since 1993, when Keating beat Hewson. When 15-20% of voters are regularly looking beyond the two major parties, that makes a majority far harder to capture.

Does this all sound reasonable? Are my biases clouding my own perceptions? Is there anything you’d like to add?

This is the fifth (and last) of my guides to Cambodia

Well, this post has been a long time coming… don’t take that as any reflection of quality, mind you. And again, this post is focused on Battambang, as it’s where I’ve spent the vast majority of my time when in Cambodia.

The Smoking Pot Cooking Class

As mentioned in my previous post, Smoking Pot has cooking classes – quite possibly the first of its kind in Cambodia. Usually operating in the morning, they provide a great introduction to a Khmer cooking. Make sure you book a day or two ahead!

Vannak (who runs Smoking Pot) will take you down to the market and purchase all the ingredients for the meals you’ll be preparing – and the market is an experience in itself. He’ll then lead you through making three meals – which you then will eat, so don’t have much in the way of breakfast beforehand! Vannak’s very good with the classes, and quite happy to chat about Cambodia in general.

The Bamboo Train

Cambodia has a very basic and unreliable train system – in most places, there’s no longer proper trains running. However, in and around Battambang there is the Bamboo Train – platforms of bamboo wood, metal wheels on axels, and a motor to power the whole thing along – which can be hired to get from one village to another.

For the most part, these are used by locals to ferry goods around, but can also be taken by tourists. It’s not the most comfortable ride in the world, but you get along at a decent clip, and you get glimpses of the Cambodian countryside.

Also: as you can see in the photo above, it’s a single track – so if there’s another bamboo platform coming in the other direction, whoever has the lightest load has to take their platform off the tracks.

Perhaps it’s the adrenalin rush from such an unsafe adventure – seats, let alone seatbelts, don’t exist – but this is easily one of my favourite things to do in Battambang.

At some point, the train line is probably going to be torn up to make way for a shiny modern system (as part of China’s cross-Asia train line) – so if you’re in town, make this a priority, as you may not get another chance!

Phare Ponleu Selpak Circus

Another highlight of Battambang is the Circus at Phare Ponleu Selpak. These kids are extremely talented – I wouldn’t be surprised if plenty of them find themselves in Cirque du Soleil shows.

The performances they put together are a lot of fun – even though it’s all in Khmer, you’ll easily pick up on what’s happening. There’s performances at least once a week – make sure you get along.

Phnom Sampeau

One of the hills close to Battambang is Phnom Sampeau. If you take the stairs up, you’ll find several temples, as well as the Killing Caves. These caves were where many Khmer were killed or maimed and then left to die. It’s not a happy place, but it does help with understanding what the people of Cambodia have been through – and are still recovering from.

On a lighter note, you get some great views from up on the top of the hill. Also, if you stick around until late afternoon, you’ll get to see millions of small bats streaming out from the many caves to find food for the night. I’ve only managed to see this once, but it’s really quite something to watch.

Phnom Sampeau is probably too far for a tuk-tuk journey – the roads aren’t sealed for most of the way, so taking a moto is a far better option. If you speak to your hotel, or ask at restaurants, you should be able to find tour guides and moto drivers without too much hassle.

Soksabike

One fantastic way of getting around Battambang and the surrounding areas is by bike – and there’s now bike tours, run by an organisation known as Soksabike.

This has only started up recently – a good friend of mine has helped get it going – but from what I’ve heard the guides are getting better and better, and it’s a great experience riding out through Battambang to the nearby villages.

Battambang – much like most of Cambodia – is extremely flat, so it’s really easy to get around by bike. Don’t feel you have to be super fit to give this a shot.

More Temples and Touring

There’s several other temples nearby – Wat Banan is perhaps the best known. Quite old, it’s like a small version of some of the temples you can find in the Angkor complex. The one catch is it’s on top of a hill, and the stairs are a killer.

You can also check out some of the local industries – rice paper, fish paste, rice wine and more – as part of your tours, whether that be by moto, tuk-tuk, or with Soksabike.

In Closing

Well, these Cambodia posts have taken me a long time to write. Hopefully they’re useful for others in providing some perspective on Cambodia, Khmer people and Battambang. I’d love to hear from anyone who has made it to this corner of the world and what you thought of the experience.