Freelancing Gods 2015

God
30 Dec 2008

Freelancing Tips via Rails Camp 4

Context

The fourth Australian Rails Camp happened back in the middle of November – and it was unsurprisingly and extremely enjoyably awesome, just like the previous four. Ryan and Anthony did a sterling job with putting it all together.

I probably talked a bit too much – I certainly felt I had more than my fair share of peoples’ focus – and while I rabbited on about Sphinx and Ginger, the topic I really enjoyed ranting about was freelancing, because it became far less about me, and far more about sharing the wealth of everybody’s experiences. I provided a few starting points, and then wise RORO minds added their own thoughts and opinions.

I can’t reproduce all that here, though. I wouldn’t do it justice. What I can do is go over the same notes I had then, and you can add your 2 cents (or five dollars) in the comments.

Freelancing Maths

One of the first things you need to be aware of, when you start freelancing, is how much to charge. I didn’t have a clue, but some more business-minded friends put me on the right track, so I’m sharing their advice here – don’t give me any credit for it.

So, let’s assume you want to start freelancing, and you have a target of earning $80,000 over the year (yes, some of you may say that’s too low, but others will say it’s too high – it’s just an example, okay?). You can use this as a basis for figuring out an hourly rate. There’s 52 weeks in a year, 5 days in a week, and 8 hours in a day…

 52 weeks
x 5  days
x 8 hours
x ?  rate
_________
   80,000

But wait a second – are you really going to work all of those 52 weeks? I doubt it. You’ll need time off for annual leave, sick leave and public holidays – the times when an employer would still pay you when you’re not slaving away. Australian annual leave is four weeks, sick leave is usually two, let’s add in another one for public holidays, and that brings us down to 45.

 45 weeks
x 5  days
x 8 hours
x ?  rate
_________
   80,000

What are the odds you’re going to have work all the time though, and are you really going to have eight billable hours each day? Unless you’re some sort of machine, the answer’s no, trust me. So lets drop eight down to six.

     45 weeks
x     5  days
x     6 hours
x 59.25  rate
_____________
       80,000

One thing we’ve missed in our calculations is superannuation. Again, using Australia as the example (because it’s all I can reliably comment on), you’re supposed to be putting away 9% of your income into your super account. Let’s factor that in:

     45 weeks
x     5  days
x     6 hours
x 64.59  rate
_____________
       87,200

Okay, so we can get an hourly rate of about $65 from that maths. And that could be fine… but maybe you’ve been eyeing off RailsConf or RubyConf or other such events. They’re not cheap – and hopefully employers would normally fork out the cash to get you there. You’re the employer now, so how are you going to afford it? Add an allowance into your calculations.

Again, due to the remoteness of Australia, it’s extra expensive to get to any of the major Ruby conferences. If we assume you’ll get to two of them (again, could be extravagant for some of you, but this is all hypothetical), then I’m adding a touch over $12,000 – flights, hotels, insurance, the conference tickets – to bring us to a nice round $100,000 target.

Also, I’ve dropped the number of weeks down another two – it’s not like you’ll be getting anything done for your clients as you jet around.

     43 weeks
x     5  days
x     6 hours
x 77.52  rate
_____________
      100,000

Okay, our final hourly rate is about $77.50.

I know a lot of the more experienced developers are looking at that value and thinking it’s pretty low – and going by market rates (for Ruby developers), it’s definitely below average. Some say a good ballpark figure for a decent Rails developer is $100/hour – USD or AUD (remember when the two currencies were almost on par?). This doesn’t mean you should charge that much (or that little) – but it should factor into your thinking.

All that said, you need to be comfortable with what you’re billing your time at, but don’t be afraid to charge what you’re worth. If the idea of having more cash than you expect scares you, there’s plenty of charities who would like to be your friend. Or, you could just work less, and spend the extra time on cool things (and they don’t even have to involve code!)

Freelancing Profile

Knowing what to charge is useful, but it’s not going to bring in the clients. Being known will help that problem, though – and there’s a few things you can do to help that.

Blog

Interpret how you will – a normal blog, twitter, tumblelog, even gists and pasties – sharing your ideas and knowledge is a great way to get your name known to others. It also helps build some human connections, via comments, emails or directed tweets. If it is valuable, they will find you (and if you think they need help, use a site like RubyFlow or RubyCorner to bring in some eyeballs).

Talk

If there’s a neat bit of code you’ve found, library you’ve come across (or written), or knowledge you think is valuable to others, offer to talk about it! It can be at your local Ruby group, or at something like a Rails Camp or BarCamp, or if you’re really comfortable up on stage, think about applying for a RailsConf or RubyConf slot.

I’m not a natural public speaker – but my confidence has grown in leaps and bounds from giving talks to fellow developers. Granted, I need to build up a bigger repertoire of topics, but I’m a bit less nervous about standing up and announcing my thoughts and opinions to others. It all started with an email from Tim Lucas asking what I was going to talk about at the first Rails Camp – and now Rails Camp folks are probably sick of hearing my voice.

They know who I am, though, and they know what code I’ve written. And that’s led to a referral or two for Rails work (usually Sphinx-related).

Socialise

Networking is a dirty word – and I can see how building connections with others for the purpose of connections, instead of meeting cool people, is a bit dirty. The much more fun alternative is to socialise – go out to social events, find those drinks happening in the evenings of conferences, have a conversation with a person you’ve not met before at your local Ruby meet.

Down the track, you will find these people may throw work your way – or maybe you’ll just learn cool new ways to code, or share some of your own knowledge, or make a good friend. All chalked up as wins in my book.

Release Code

Releasing your own code – from snippets to plug ins to full-blown applications – is a great way to show peers that you know what you’re talking about. It also shows potential clients that too, and reaffirms that you’re worth the rate you’re charging, and that you can be creative.

In my own case, I’ve done the occasional bit of Sphinx consulting due to my work on Thinking Sphinx.

Coincidentally, doing all these things are rewarding in and of themselves. I don’t do them to bring in work, I do them because they’re fun and I meet awesome people, which is (I think) the best approach. The opportunities they lead to are just an added bonus.

Your Turn

So, what’s your advice to a budding freelancer? Is there anything here that’s a bit Ruby or developer-centric? Any more general suggestions to keep in mind?

Also, please keep in mind I’m not an expert. I think the above advice is useful, but it is just advice. There’s no hard and fast rules that should be followed.

And the name of this blog has nothing to do with my work lifestyle, but the idea of deities who freelance for each other. Don’t take it as an indication of my ego. Honest.

24 Oct 2008

Thinking Sphinx PDF at Peepcode

A quick note to let anyone using (or interested in using) Sphinx and/or Thinking Sphinx that my Peepcode PDF has just been published, and contains a wealth of information on how to mix Sphinx with ActiveRecord (via Rails or Merb).

It’s been great working with Geoffrey Grosenbach to get this written up, and I’m pretty stoked to see the final results – hopefully others will enjoy it as well.

Also, a massive thank you to all the contributors to Thinking Sphinx – it wouldn’t be quite so cool if it wasn’t for all the patches (facilitated by GitHub’s forking).

23 Oct 2008

Developer Ethics

A quick question to fellow coders…

Unsurprisingly, there’s a dearth of Ruby developers in Cambodia. I imagine the situation is pretty similar in other developing nations. PHP and Visual Basic seem to be the common languages in the small tech community here.

I’m currently working on building a website for one of the local NGOs here – and of course, Rails is my preferred framework. But looking forward, I don’t wish to be providing ongoing support for the site – and the client shares that sentiment. So to make it easier for local developers to take over, should I be considering using PHP for the project instead?

I have offered to help the IT guy at this organisation learn Ruby, but he won’t be there forever as well. And they’re a small NGO – they don’t have the cash to throw around hiring super-skilled developers. The project itself is pro-bono.

So, what would you do, given the circumstances?

(And for the record, it’s very likely I’ll stick with Ruby – using the Radiant CMS – but I’m interested in others’ opinions.)

11 Oct 2008

Flavouring For Your Specs

One of the things I’ve tried to do with Thinking Sphinx has been to keep it friendly for a few different versions of Rails/ActiveRecord: 1.2.6, 2.0.x and 2.1.×. It’s not that easy, because I know I don’t think about which methods are pre-Rails-2.0, and which are even newer. And add on top of that the large (and very welcome) number of contributors, and it becomes a bit of a headache.

I was introduced to a potential aid for this at RailsConf EU, by fellow Australian Ian White, in the form of his library Garlic. It takes Rails plugins, instantiates them within Rails applications for each defined environment, and then runs their specs.

That in itself is pretty damn cool – except I don’t run Thinking Sphinx’s specs from within Rails – and it’s really only reliant on ActiveRecord (so it can be used in Merb). So, I’ve spent an afternoon coding up something slightly different, but in something of an acknowledgement to Ian’s library, my solution is called Ginger.

Now, it’s not exhaustively tested, but I’ve got it to the point where it runs as I would expect it to with Thinking Sphinx, so consider it ready for release.

If you’re interested in using it, here’s the basic guidelines. First up, install the gem:

sudo gem install freelancing-god-ginger --source=http://gems.github.com

(GitHub doesn’t have the gem loaded at the moment though, so you’ll have to install it manually)

git clone git://github.com/freelancing-god/ginger.git
cd ginger
gem build ginger.gemspec
sudo gem install --local ginger-1.0.0.gem

Then, in your spec_helper.rb file (or equivalent – this should work for Test::Unit as well as RSpec), add:

require 'ginger'

You’ll want to make sure that require is before any other require calls to libraries you want to test multiple versions of.

Next, you need to create a scenario file, outlining the different testing situations Ginger should run through. These details go in a file called ginger_scenarios.rb in the root of your project. Here’s my current one for Thinking Sphinx:

require 'ginger'

Ginger.configure do |config|
  config.aliases["active_record"] = "activerecord"
  
  ar_1_2_6 = Ginger::Scenario.new
  ar_1_2_6[/^active_?record$/] = "1.15.6"
  
  ar_2_0_2 = Ginger::Scenario.new
  ar_2_0_2[/^active_?record$/] = "2.0.2"
  
  ar_2_1_1 = Ginger::Scenario.new
  ar_2_1_1[/^active_?record$/] = "2.1.1"
  
  config.scenarios << ar_1_2_6 << ar_2_0_2 << ar_2_1_1
end

I’m adding three different scenarios – one for each version of ActiveRecord I want to test. I’m also adding an alias, as sometimes people have an underscore in the require calls, and while the require method isn’t fussy, the gem method is. That’s also the reason for the regular expressions, but strings will work as well.

If you want, you can have multiple gem requirements for each scenario – Ginger::Scenario is subclassed from Hash, so just add more key/value pairs as needed.

sphinx_scenario = Ginger::Scenario.new
sphinx_scenario["riddle"] = "0.9.8" 
sphinx_scenario["thinking_sphinx"] = "0.9.8"

And don’t forget to add each scenario to @config@’s collection.

The last step is the most important – running your tests through each scenario! This is done with the ginger command line tool – any arguments given to it get echoed onto rake, so in theory it should work with whatever rake task you like. I’ve only tested it with RSpec though.

ginger spec
ginger test
ginger spec:unit

This was built to scratch my itch, obviously, but happy to accept patches or suggestions. GitHub, as always, is the best way to do this – fork to your heart’s content.

07 Sep 2008

RejectConf: Coders Kicking Arse

One of the highlights from RailsConf EU last week was RejectConf – even if it was a bit smaller than last year (going by what I’ve heard, anyway).

I reprised my So You’re A Kick-Arse Coder talk for it – since it was a rejected talk from the main event – and Geoffrey Grosenbach managed to get an audio recording, so I’ve put that together with the slides and onto Viddler. Keep in mind the following caveats:

  • I’m pretty happy with this talk – but I realise I’m not that great a speaker. Imagine what I’d be like on a bad day ;)
  • Geoff didn’t catch the very start of the talk, which went something along the lines of “Hi, my name’s Pat, and I’m Australian [Cheers from Audience] I want to start of with some flattery, because I want to get on your good side.”
  • Geoff’s also the heckler about two-thirds of the way though.

Links to the sites I mention:

XKCD Comics featured:

Photos used thanks to either permission or permissive licences:

26 Aug 2008

Rails Camp UK Report

Just over a week ago, the first Rails Camp in the UK was held in Downe, outside Orpington – and I think it was a fantastic success (having been the organiser though, obviously there is some bias).

We had quite an international flavour to the weekend. Of the 30 or so who attended, several were from around Europe, alongside the local British, and a few of us Australians to round it out.

The Beer Disappears

In true Rails Camp style, around the beer, pizza and games, much hacking and discussion was had – assisted by the *jour gems, twetter and SubEthaEdit. Plenty of cool projects were displayed and created – topics ranging from RSpec to EXTJS to in-memory models to plugins to CouchDB to approaches for better browser-server polling (with a neat browser game as an example).

Railscamp UK 2008 (8 of 12)

One of the cool creations of this Rails Camp has gone live. The collective talent of the RailsLove guys and Rany Keddo produced a forkable lists web app called Don’t Forget The Wurst – and features William Shatner, which just adds several levels of awesomeness to an already neat idea.

Werewolf

Massive thanks to all who came along and made it such a fantastic weekend – I’m looking forward to hearing about another Rails Camp in this part of the world (even if I won’t be able to attend it).

Matt and Simon

Also, if you’re in Europe, you might want to check out the German and Danish Rails Camps, which will be happening later this year. Australians, Rails Camp #4 will be happening in November (details are almost finalised). Everyone else: I highly recommend making one happen near you. There’s now a group of us who have dabbled in the organisation of them, and we’re more than happy to help however we can to get more of them happening around the world. It’s not too hard, and it’s an awesome way of strengthening your local Ruby community.

Balancing

16 Jul 2008

Odds and Ends

A few random items:

  • There’s a Rails Camp happening in Denmark. How awesome is that!?
  • I’ve added an About Me page to this blog – filled with opinions. You have been warned.
  • I’m talking at NYC Ignite – come along and listen to me talk quickly about non-Ruby stuff for five minutes, if you’re near that part of the world.
  • Joss Whedon is awesome.
  • So is Pixar. You must see Wall-E. Easily the best film I’ve seen all year.
12 Jul 2008

Link: Jedlinski.pl devblog » Thinking Sphinx as Windows service

"Thinking Sphinx for Windows - a batch of simple rake tasks dedicated for Windows users."

01 Jul 2008

Rails Camp UK

Following in the steps of the Australian Rails Camps, it’s now time to announce the first UK edition. Running from Friday the 15th to Monday the 18th of August, it will be an extended weekend of hacking, talking, eating, drinking and games, with a bunch of smart and passionate Ruby developers.

Even though the name is “Rails Camp”, previous camps have included talks on topics from Merb to Rack to Extreme Programming – all topics somewhat related to Ruby are welcome.

If you’d like to come along, I’d recommend registering soon, as there’s a very limited number of places.

01 Jul 2008

Link: new controller examples

"refining the examples generated for restful controllers"

12 Jun 2008

A Concise Guide to Using Thinking Sphinx

Okay, it’s well past time for the companion piece to my Sphinx Primer – let’s go through the basic process of using Thinking Sphinx with Rails.

Just to recap: Sphinx is a search engine that indexes data, and then you can query it with search terms to find out which documents are relevant. Why do you want to use it with Rails? Because it saves having to write messy SQL, and it’s so damn fast.

(If you’re getting a feeling of deja-vu, then it’s probably because you’ve read an old post on this blog that dealt with an old version of Thinking Sphinx. I’ve had a few requests for an updated article, so this is it.)

Installation

So: first step is to install Sphinx. This may be tricky on some systems – but I’ve never had a problem with it with Mac OS X or Ubuntu. My process is thus:

curl -O http://sphinxsearch.com/downloads/sphinx-0.9.8-rc2.tar.gz
tar zxvf sphinx-0.9.8-rc2.tar.gz
cd sphinx-0.9.8-rc2
./configure
make
sudo make install

If you’re using Windows, you can just grab the binaries.

Once that’s taken care of, you then want to take your Rails app, and install the plugin. If you’re running edge or 2.1, this is a piece of cake:

script/plugin install git://github.com/freelancing-god/thinking-sphinx.git

Otherwise, you’ve got a couple of options. The first is, if you have git installed, just clone to your vendor/plugins directory:

git clone git://github.com/freelancing-god/thinking-sphinx.git
  vendor/plugins/thinking-sphinx

If you’re not yet using git, then the easiest way is to download the tar file of the code. Try the following:

curl -L http://github.com/freelancing-god/thinking-sphinx/tarball/master
  -o thinking-sphinx.tar.gz
tar -xvf thinking-sphinx.tar.gz -C vendor/plugins
mv vendor/plugins/freelancing-god-thinking-sphinx* vendor/plugins/thinking-sphinx

Oh, and it’s worth noting: if you’re not using MySQL or PostgreSQL, then you’re out of luck – Sphinx doesn’t talk to any other relational databases.

Configuration

Next step: let’s get a model or two indexed. It might be worth refreshing your memory on what fields and attributes are for – can I recommend my Sphinx article (because I’m not at all biased)?

Ok, now let’s work with a simple Person model, and add a few fields:

class Person < ActiveRecord::Base
  define_index do
    indexes [first_name, last_name], :as => :name
    indexes location
  end
end

Nothing too scary – we’ve added two fields. The first is the first and last names of a person combined to one field with the alias ‘name’. The second is simply location.

Adding attributes is just as easy:

define_index do
  # ...
  
  has birthday
end

This attribute is the datetime value birthday (so you can now sort and filter your results by birthdays).

Managing Sphinx

We’ve set up a basic index – now what? We tell Sphinx to index the data, and then we can start searching. Rake is our friend for this:

rake thinking_sphinx:index
rake thinking_sphinx:start

Searching

Now for the fun stuff:

Person.search "Melbourne"

Or with some sorting:

Person.search "Melbourne", :order => :birthday

Or just people born within a 10 year window:

Person.search "Melbourne", :with => {:birthday => 25.years.ago..15.years.ago}

If you want to keep certain search terms to specific fields, use :conditions:

Person.search :conditions => {:location => "Melbourne"}

Just remember: :conditions is for fields, :with is for attributes (and :without for exclusive attribute filters).

Change

Your data changes – but unfortunately, Sphinx doesn’t update your indexes to match automatically. So there’s two things you need to do. Firstly, run rake thinking_sphinx:index regularly (using cron or something similar). ‘Regularly’ can mean whatever time frame you want – weekly, daily, hourly.

The second step is optional, but it’s needed to have your indexes always up to date. First, add a boolean column to your model, named ‘delta’, and have it default to true. Then, tell your index to use that delta field to keep track of changes:

define_index do
  # ...
  
  set_property :delta => true
end

Then you need to tell Sphinx about the updates:

rake thinking_sphinx:stop
rake thinking_sphinx:index
rake thinking_sphinx:start

Once that’s done, a delta index will be created – which holds any recent changes (since the last proper indexing), and gets re-indexed whenever a model is edited or created. This doesn’t mean you can stop the regular indexing, as that’s needed to keep delta indexes as small (and fast) as possible.

String Sorting

If you remember the details about fields and attributes, you’ll know that you can’t sort by fields. Which is a pain, but there’s ways around this – and it’s kept pretty damn easy in Thinking Sphinx. Let’s say we wanted to make our name field sortable:

define_index do
  indexes [first_name, last_name], :as => :name, :sortable => true
  
  # ...
end

Re-index and restart Sphinx, and sorting by name will work.

How is this done? Thinking Sphinx creates an attribute under the hood, called name_sort, and uses that, as Sphinx is quite fine with sorting by strings if they’re converted to ordinal values (which happens automatically when they’re attributes).

Pagination

Sphinx paginates automatically – in fact, there’s no way of turning that off. But that’s okay… as long as you can use your will_paginate helper, right? Never fear, Thinking Sphinx plays nicely with will_paginate, so your views don’t need to change at all:

<%= will_paginate @search_results %>

Associations

Sometimes you’ll want data in your fields (or attributes) from associations. This is a piece of cake:

define_index do
  indexes photos.caption, :as => :captions
  indexes friends.photos.caption, :as => :friends_photos
  
  # ...
end

Polymorphic associations are fine as well – but keep in mind, the more complex your index fields and attributes, the slower it will be for Sphinx to index (and you’ll definitely need some database indexes on foreign key columns to help it stay as speedy as possible).

Gotchas

In case things aren’t working, here’s some things to keep in mind:

  • Added an attribute, but can’t sort or filter by it? Have you reindexed and restarted Sphinx? It doesn’t automatically pick up these changes.
  • Sorting not working? If you’re specifying the attribute to sort by as a string, you’ll need to include the direction to sort by, just like with SQL: “birthday ASC”.
  • Using name or id columns in your fields or attributes? Make sure you specify them using symbols, as they’re core class methods in Ruby.
define_index do
  indexes :name
  
  # ...
  
  has photos(:id), :as => :photo_ids
end

And Next?

I realise this article is pretty light on details – but if you want more information, the first stop should be the extended usage page on the Thinking Sphinx site, quickly followed by the documentation. There’s also an email list to ask questions on.

28 May 2008

RailsConf 2008

I’ve just started my round-the-world conferences-and-holiday adventure, and the first stop is RailsConf in Portland – so if you’re in town and see me wandering around looking rather cluelessly, please say hi.

Also, in case you’re on the Twitter bandwagon, you’ll find me with the creative nickname of pat.

21 May 2008

Sphinx + Rails + PostgreSQL

In case you’ve not been watching every commit carefully flow through Thinking Sphinx on GitHub – PostgreSQL support has been added. I’ve done a little bit of testing, and I’ve had some excellent support from Björn Andreasson and Tim Riley, so I feel it’s ready for people to start kicking the tires.

I’m no PostgreSQL expert – I definitely fall into the n00b category – so if you think there’s better ways to do what I’m doing, would love to hear them.

13 May 2008

Link: lindsaar.net Tip #4 - Validating an Email Address with Ruby on Rails

"TMail has an “Address” class. It will throw an invalid address exception if given an address it can’t handle (and it has about 2,000 test cases of email addresses it can handle, so you are pretty safe.)"

11 May 2008

Updates for Thinking Sphinx

I’ve been working away on Thinking Sphinx when I’ve had the time – and we’re nearing what I feel is a solid 1.0 release. I say we, because I’ve had some very generous people supply patches over the last few weeks – namely Arthur Zapparoli, James Healy, Chris Heald and Jae-Jun Hwang. Switching to git – and GitHub in particular – has made it very easy to accept patches.

Mind you, all of these changes aren’t committed just yet – and even when they are, there’ll still be a few more things to cross off the list before we hit the big 1.0 milestone, namely PostgreSQL support and solid spec coverage. Slowly edging closer.

In other news – to help share the Thinking Sphinx knowledge (after some prompting by a few users of the plugin), I’ve created a Google Group for it – so this will be the ideal place to ask questions about how to implement Sphinx into your app, suggest features, or report bugs.

If you’ve been pondering how to deploy Thinking Sphinx via Capistrano, I recommend you read a blog post by Wade Winningham – or if you’re interested in better ways of handling UTF-8 characters outside of the ‘normal’ set (ie: without accents and so forth), make sure you peruse James Healy’s solution.

And one last reminder – if you’re in Sydney on Wednesday evening and interested in learning a bit more about Sphinx in general and Thinking Sphinx in particular, come along to the monthly Ruby meet at the Crown Hotel in Surry Hills, as those are the topics I’ll be presenting on.

26 Apr 2008

Sphinx: A Primer

On Thursday night I presented to the Melbourne Ruby Group about Sphinx – first with a non-Ruby perspective, and then using Ruby, and more specifically Rails. I’ll be presenting again at the Sydney group in a couple of weeks, but I am also adapting the talk to a few blog posts – to allow a bit more detail in a few doses.

First up: Sphinx itself. Why should you read this? Because understanding Sphinx will help you use whichever library (Ruby or otherwise) smarter. It might also teach you some things you had no idea about (ie: this is the article I should have read when I started using Sphinx).

What is Sphinx?

Sphinx is a search engine. You feed it documents, each with a unique identifier and a bunch of text, and then you can send it search terms, and it will tell you the most relevant documents that match them. If you’re familiar with Lucene, Ferret or Solr, it’s pretty similar to those systems. You get the daemon running, your data indexed, and then using a client of some sort, start searching.

When indexing your data, Sphinx talks directly to your data source itself – which must be one of MySQL, PostgreSQL, or XML files – which means it can be very fast to index (if your SQL statements aren’t too complex, anyway).

Sphinx Structure

A Sphinx daemon (the process known as searchd) can talk to a collection of indexes, and each index can have a collection of sources. Sphinx can be directed to search a specific index, or all of them, but you can’t limit the search to a specific source explicitly.

Each source tracks a set of documents, and each document is made up of fields and attributes. While in other areas of software you could use those two terms interchangeably, they have distinct meanings in Sphinx (and thus require their own sections in this post).

Fields

Fields are the content for your search queries – so if you want words tied to a specific document, you better make sure they’re in a field in your source. They are only string data – you could have numbers and dates and such in your fields, but Sphinx will only treat them as strings, nothing else.

Attributes

Attributes are used for sorting, filtering and grouping your search results. Their values do not get paid any attention by Sphinx for search terms, though, and they’re limited to the following data types: integers, floats, datetimes (as Unix timestamps – and thus integers anyway), booleans, and strings. Take note that string attributes are converted to ordinal integers, which is especially useful for sorting, but not much else.

Multi-Value Attributes

There is also support in Sphinx to handle arrays of attributes for a single document – which go by the name of multi-value attributes. Currently (Sphinx version 0.9.8rc2) only integers are supported, so this isn’t quite as flexible as normal attributes, but it’s worth keeping in mind.

Filters

Filters are useful with attributes to limit your searches to certain sets of results – for example, limiting a forum post search to entries by a specific user id. Sphinx’s filters accept arrays or ranges – so if filtering by a single value, just put that in an array. The range filters are particularly useful for getting results from a certain time span.

Relevance

Relevancy is the default sorting order for Sphinx. I’ve no idea exactly how it is calculated, but there are a couple of things you can do easily enough in your queries to influence it. The first is index-level weighting, where you give specific indexes higher rankings than others. The other, similar in nature, but at a lower level, is field weightings. Generally these are set before each query, but it will depend on the library you use.

Keeping Your Indexes Updated

One thing that sets Sphinx apart from Ferret and other search engines is that there is no way to update fields for a specific document in your indexes. The main approach around this is having delta indexes – a small index with all the recent changes (which will be super-fast to index), so Sphinx will include that and the main index for its searches. Of the Rails plugins, both Thinking Sphinx and Ultrasphinx have support for this – I’ve no idea for other languages, mind you.

What’s next?

Next is when we’ll dive into some actual code – we’ll go through some of the common tasks for setting up Sphinx with Rails using Thinking Sphinx.

10 Apr 2008

Thinking Sphinx Reborn

So, over the last month or so I’ve been working hard on rewriting Thinking Sphinx – and it’s now time to release those changes publicly. The site’s now got a brief quickstart page and a detailed usage page beyond the rdoc files, and there will be more added over the coming weeks.

A quick overview of what’s shiny and new:

Better index definition syntax

This part reworked many times, finally to something I’m pretty happy with:

define_index do
  indexes [first_name, last_name], :as => :name, :sortable => true
  indexes email, location
  indexes [posts.content, posts.subject], :as => :posts
end

Polymorphic association support in indexes

When you’re drilling down into your associations for relevant field data, it’s now safe to use polymorphic associations – Thinking Sphinx is pretty smart about figuring what models to look at. Make sure you put table indexes on your _type columns though.

MVA Support

Multi-Value Attributes now work nice and cleanly – so you can tie an array of integers to any record.

Multi-Model Searching

Just like before, you can search for records of a specific model. This time around though, you can also search across all your models – and the results still use will_paginate if it’s installed.

ThinkingSphinx::Search.search "help"

Better Filter Support

It was kinda in there to start with, but now it’s much smarter – and it all goes into the conditions hash, just like a find call:

User.search :conditions => {:role_id => 5}
Article.search :conditions => {:author_ids => [12, 24, 48]}

Sorting by Fields

As you may have noticed in the first code block of this post, you can mark fields as :sortable – what this does is it uses Sphinx’s string attributes, and creates a matching attribute that acts as a sort-index to the field. When specifying the search options though, you can just use the field’s name – Thinking Sphinx knows what you’re talking about.

User.search "John", :order => :name
User.search "Smith", :order => "name DESC"

Even More

I’m so eager to share this new release that there’s probably a few things that need a bit more documentation – that will appear both on the Thinking Sphinx site and here on the blog. I’m planning on writing some articles that provide a solid overview to Sphinx in general – which will hopefully be some help no matter what plugin you use – and then dive into some regular ‘recipes’ of Thinking Sphinx usage, and some detailed posts of the cool new features as well.

Also in the pipeline is Merb support – just for ActiveRecord initially, but I’d love to get it talking to DataMapper as well.

Update: Jonathan Conway’s got a branch working in Merb and Rails – needless to say, I’ll be updating trunk with his patch as soon as possible.

06 Apr 2008

Link: New in Rails: a request profiler for profiling your app | redemption in a blog

Old news, but I need to remember it's in there

25 Mar 2008

Link: Formtastic Plugin Documentation

"makes it far easier to create beautiful, semantically rich, syntactically awesome, readily stylable and wonderfully accessible HTML forms in your Rails applications."

16 Mar 2008

RailsCamp #3

If you’re a Ruby developer in or near Australia, I highly recommend attending RailsCamp number 3, which has just opened for registration. The first two were simply amazing, so I’m just a little annoyed that I can’t make it to this one (as I’ll be traveling overseas at the time). I’ve no doubt that this one will be just as fantastic – expect an extended weekend of hacking and talking with a bunch of smart, entertaining and passionate developers, and plenty of drinks and games thrown in for good measure.

You don’t need to be a Rails or Ruby genius to attend – just a desire to discuss, learn, teach and (most importantly) have fun.

Go register now.

RssSubscribe to the RSS feed

About Freelancing Gods

Freelancing Gods is written by , who works on the web as a web developer in Melbourne, Australia, specialising in Ruby on Rails.

In case you're wondering what the likely content here will be about (besides code), keep in mind that Pat is passionate about the internet, music, politics, comedy, bringing people together, and making a difference. And pancakes.

His ego isn't as bad as you may think. Honest.

Here's more than you ever wanted to know.

Open Source Projects

Other Sites

Creative Commons Logo All original content on this site is available through a Creative Commons by-nc-sa licence.