Freelancing Gods 2014

God
26 Apr 2008

Sphinx: A Primer

On Thursday night I presented to the Melbourne Ruby Group about Sphinx – first with a non-Ruby perspective, and then using Ruby, and more specifically Rails. I’ll be presenting again at the Sydney group in a couple of weeks, but I am also adapting the talk to a few blog posts – to allow a bit more detail in a few doses.

First up: Sphinx itself. Why should you read this? Because understanding Sphinx will help you use whichever library (Ruby or otherwise) smarter. It might also teach you some things you had no idea about (ie: this is the article I should have read when I started using Sphinx).

What is Sphinx?

Sphinx is a search engine. You feed it documents, each with a unique identifier and a bunch of text, and then you can send it search terms, and it will tell you the most relevant documents that match them. If you’re familiar with Lucene, Ferret or Solr, it’s pretty similar to those systems. You get the daemon running, your data indexed, and then using a client of some sort, start searching.

When indexing your data, Sphinx talks directly to your data source itself – which must be one of MySQL, PostgreSQL, or XML files – which means it can be very fast to index (if your SQL statements aren’t too complex, anyway).

Sphinx Structure

A Sphinx daemon (the process known as searchd) can talk to a collection of indexes, and each index can have a collection of sources. Sphinx can be directed to search a specific index, or all of them, but you can’t limit the search to a specific source explicitly.

Each source tracks a set of documents, and each document is made up of fields and attributes. While in other areas of software you could use those two terms interchangeably, they have distinct meanings in Sphinx (and thus require their own sections in this post).

Fields

Fields are the content for your search queries – so if you want words tied to a specific document, you better make sure they’re in a field in your source. They are only string data – you could have numbers and dates and such in your fields, but Sphinx will only treat them as strings, nothing else.

Attributes

Attributes are used for sorting, filtering and grouping your search results. Their values do not get paid any attention by Sphinx for search terms, though, and they’re limited to the following data types: integers, floats, datetimes (as Unix timestamps – and thus integers anyway), booleans, and strings. Take note that string attributes are converted to ordinal integers, which is especially useful for sorting, but not much else.

Multi-Value Attributes

There is also support in Sphinx to handle arrays of attributes for a single document – which go by the name of multi-value attributes. Currently (Sphinx version 0.9.8rc2) only integers are supported, so this isn’t quite as flexible as normal attributes, but it’s worth keeping in mind.

Filters

Filters are useful with attributes to limit your searches to certain sets of results – for example, limiting a forum post search to entries by a specific user id. Sphinx’s filters accept arrays or ranges – so if filtering by a single value, just put that in an array. The range filters are particularly useful for getting results from a certain time span.

Relevance

Relevancy is the default sorting order for Sphinx. I’ve no idea exactly how it is calculated, but there are a couple of things you can do easily enough in your queries to influence it. The first is index-level weighting, where you give specific indexes higher rankings than others. The other, similar in nature, but at a lower level, is field weightings. Generally these are set before each query, but it will depend on the library you use.

Keeping Your Indexes Updated

One thing that sets Sphinx apart from Ferret and other search engines is that there is no way to update fields for a specific document in your indexes. The main approach around this is having delta indexes – a small index with all the recent changes (which will be super-fast to index), so Sphinx will include that and the main index for its searches. Of the Rails plugins, both Thinking Sphinx and Ultrasphinx have support for this – I’ve no idea for other languages, mind you.

What’s next?

Next is when we’ll dive into some actual code – we’ll go through some of the common tasks for setting up Sphinx with Rails using Thinking Sphinx.

Comments

16 responses to this article

26 Apr 2008
Thuva said:

Thank you for the great overview of Sphinx. I’m looking forward to your next post in this series.

By the way, I’m using Thinking Sphinx with great success in a project that is under development.

27 Apr 2008
Tim Lucas said:

Would love to see such a succinct overview of the basics make it back the official site. The official manual is considerably less accessible.

Look forward to the preso!

05 May 2008
Brian Ablaza said:

What can I do to try and resolve a problem or ask a question?

05 May 2008
pat said:

Brian: Either post a comment here or email me (pat at this domain) – more than happy to help.

06 May 2008
Brian Ablaza said:

I’ll send you an email. Thanks.

06 May 2008
Jae-Jun Hwang said:

Thank you for writing ThinkingSphinx and Riddle. It’s been wonderful product for my development.

BTW, I experimented TS, and made some patch to 1) handle CJK charset_tables, 2) prefix_fields, 3) delta+dirty etc. I’ll send it to you email,
so could you please take a look?

07 May 2008
Brian Ablaza said:

Pat: That fixed it. The .gz file was missing a few components, but I pulled them off of github and now it works beautifully.

Thanks for your help, and thanks for this plugin.

07 May 2008
pat said:

Brian: great to know it’s all working out for you now :)

08 May 2008
Brian Ablaza said:

Pat:

In delta.rb you say:

  1. This module contains all the delta-related code for models. There isn’t
  2. really anything you need to call manually in here – except perhaps
  3. index_delta, but not sure what reason why.

Here’s a reason. My DB is a legacy database, so I can’t change its structure. I’d like to trigger a delta re-index based on the max modifydate (an integer) in the File table.

So how do I call index_delta?

09 May 2008
pat said:

Brian: Not really sure what the best approach for you is – I can’t think of anything simple. I’d suggest you look at reimplementing delta.rb to use the modified date as a reference – but you’re going to need to previous main index date stored somewhere in the database, as a reference point…

You’ll almost certainly need to change things in configuration.rb as well.

The easiest solution is just to skip delta indexes and run normal indexing often.

09 May 2008
Chris Heald said:

Thinking Sphinx looks fantastic. I’m looking forward to giving it a go!

However, in setting it up in my app, I discovered that it chokes on namespaced models in subdirectories (load_models in configuration.rb, specifically) – I’ve fixed it locally, but I’d like to submit a patch, if there’s an appropriate place to do so. I didn’t see any kind of issue tracker or whatnot linked to the project.

09 May 2008
pat said:

Chris: Would love the patch! Either send me the file directly (via email) – or fork a copy on GitHub, commit your change to that, and I’ll pull it straight in from there (and you’ll get the credit for the commit)

09 May 2008
Chris Heald said:

Sure! I’ve just cloned it on github – you’ll find my commit at http://github.com/cheald/thinking-sphinx/commit/de852124fb55df07b607aee0804d71317573fbc4

I actually have two commits in there, with three fixes. I’d suggest checking ‘em before committing, as you know your code better than I, and I don’t know if I mistakenly screwed something up in the process. Seems to work fine against my app, though!

27 Jul 2009
Pratik said:

Thanks for a very well written article!
Do you know how we can restrict the fields from which we search. I mean, i have two indexed columns, “bio” and “status”. When the user is looking for the string “active”, it should only search from the status column and not from bio.

27 Jul 2009
pat said:

Pratik: depends how you want to go about this… is the user entering any query text they like (which may or may not include the word ‘active’)? Or are you specifically directing specific input to specific fields?

If the latter, use “@status active” for your query.

Or, :conditions => {:status => 'active'} if you’re using Thinking Sphinx.

The former, there’s no way to indicate a particular word should only be cared about if it exists in certain fields. You could set field_weights to make the status column more important though:

# with Riddle
client.field_weights = {'status' => 10}
# with Thinking Sphinx
Model.search :field_weights => {'status' => 10}
# Or in define_index block
set_property :field_weights => {'status' => 10}

Any fields not mentioned in the weightings keep their default weight of 1.

08 Sep 2014
Somerset Business said:

Hello to every body, it’s my first visit of this web site; this website
carries awesome and really excellent data designed for readers.

Leave a Comment

Comments are formatted using Textile. Please be respectful of others when posting comments. Be nice.

RssSubscribe to the RSS feed

Related Links

Related Posts

About Freelancing Gods

Freelancing Gods is written by , who works on the web as a web developer in Melbourne, Australia, specialising in Ruby on Rails.

In case you're wondering what the likely content here will be about (besides code), keep in mind that Pat is passionate about the internet, music, politics, comedy, bringing people together, and making a difference. And pancakes.

His ego isn't as bad as you may think. Honest.

Here's more than you ever wanted to know.

Ruby on Rails Projects

Other Sites

Creative Commons Logo All original content on this site is available through a Creative Commons by-nc-sa licence.