Advanced Sphinx Configuration
Thinking Sphinx provides a good set of defaults out of the box, and for some people, those options are exactly what they need. Sometimes, though, you may need to customise how Sphinx works - and this can usually be done by adding some settings to a file named
thinking_sphinx.yml in your
config directory. Much like database.yml, settings are defined for each environment. Here’s an example:
Now, Sphinx has a lot of different settings you can play with, and they’re pretty much all supported by Thinking Sphinx as well.
This page covers most of the more important settings, but there is also a full overview which is far more exhaustive.
Any paths provided in this configuration are expected to be absolute by default. If you want to provide relative paths, you will need to add
absolute_paths: true to each relevant environment in
config/thinking_sphinx.yml. These paths will then be translated to absolute paths from the root of the Rails application, within the generated Sphinx configuration.
Index File Location
You can customise the location of your Sphinx index files using the
Thinking Sphinx defaults to putting these files in db/sphinx/ENVIRONMENT - which makes life easier if you’re running integration tests with a live Sphinx setup. It’s worth keeping this in mind and ensuring your file locations are unique for each environment when they share a machine. Indeed, you’ll probably only want to change this value on your production machine.
Configuration, PID and Log File Locations
In the same vein as the above setting, you can nominate custom locations for your configuration, log and pid files.
Here’s some example syntax, using Thinking Sphinx’s defaults. Uppercase words are placeholders for system variables in the example only - you can’t actually use them in your YAML file.
Daemon Address and Port
If your Sphinx Daemon (also known as searchd) is running on a different machine or port, you’re going to need to tell Thinking Sphinx the critical details:
Hosting via a UNIX Socket
It is possible to run the Sphinx daemon on a UNIX socket. To do this, you will need to specify the path to the socket in your
config/thinking_sphinx.yml file per environment:
If you specify
address, then the daemon will also be available via TCP, but connections from Thinking Sphinx to Sphinx will still be made via the UNIX socket.
This feature is unfortunately not supported in JRuby (as there doesn’t seem to be a way to use UNIX sockets to connect to the MySQL protocol). It also means that Sphinx cannot be interacted with by other servers - only the machine with the Sphinx daemon. So, remote searches and deletions are not possible with this UNIX socket setting.
Indexer Memory Usage
Sphinx indexes your data using the
indexer command-line tool. This tool runs with a fixed memory limit - defaulting to 64 megabytes. You can change this to something else if you’d like - the more memory, the faster your indexes will be processed.
Word Stemming / Morphology
By default, Sphinx and Thinking Sphinx doesn’t get too smart about the words you’re searching for - it assumes you know exactly what you’re after. However, sometimes you may want it to recognise that certain words share pretty much the same meaning. For example: think and thinking.
To enable this kind of behaviour, you need to specify a morphology (or stemming library) to Sphinx. It comes with English (stem_en) and Russian (stem_ru) built-in. You can also use other stemmers via Snowball’s libstemmer library. Have a read of Sphinx’s documentation for more clues.
If you’re using Sphinx 2.2.2 or newer, wildcard syntax will be respected by default (though you’ll also need infixes or prefixes, as covered in the next section).
If you’re using an older version of Sphinx, then you can enable wildcard syntax using the
Infix and Prefix Indexing
If you want partial word matching, then you’re going to need to tell Sphinx to either index prefixes (the beginnings of words) or infixes (substrings of words). You cannot enable both at once, though.
You need to tell Sphinx what the minimum infix or prefix length is - the smaller the number is, the larger your index gets. If you set it to zero, though, that disables this feature. If you want absolutely everything, down to the last character, then set min_infix_len to 1 - but be prepared for the performance hit.
Character Sets and Tables
By default, Sphinx and Thinking Sphinx use the UTF-8 character set. If you’re using an older version of Sphinx and prefer Sphinx’s inbuild sbcs encoding, you’ll need to specify it via the charset_type setting:
This changest the default character mappings, which you can read about in the Sphinx documentation. You can also set your own character mappings - which is recommended when using UTF-8 - to include other characters. James Healy has posted his extensive settings which cover most (if not all) accented characters. If you don’t want to click through, it’s all done via the charset_table setting:
Large Result Sets
To keep searching fast, Sphinx has a default limit of 1000 records being available via pagination, even if there are more matches than that. The reasons for this limit are discussed in the Sphinx documentation.
However, you can change this value. Firstly, in your
config/thinking_sphinx.yml file, you need to set max_matches to your upper limit:
Don’t forget to reconfigure and restart your Sphinx daemon so it is aware of the change.
And you also need to specify it in your searches (Sphinx doesn’t assume you want the higher number by default):
This does not mean you will get 10,000 results returned in one request, but you can paginate up to the ten-thousandth result. If you want them all at once (which will be slow, because you’re asking Rails to instantiate 10,000 records), use the
Word Forms, Exceptions, and Stop Words
To configure Thinking Sphinx for any of these features, simply specify the path to the appropriate file in your
For full details on what these features actually do, please refer to the Sphinx documentation.