Freelancing Gods 2015

God
01 Mar 2015

Diversity, Awareness, and Welcoming Spaces

Yesterday Melbourne was host to yet another Trampoline unconference. This time around, I was taking a break from organising (something else had demanded a lot of my time lately), but that gave me a bit more time to craft together a session I’ve been keen to run for some time: a discussion on diversity.

While nothing is as good as being there on the day and partaking in the dialogue, here’s what I covered (you’ll just have to imagine all the insightful conversation you missed from others who were in the room).

Firstly, it’s worth acknowledging that I am a straight, white guy – thus in some ways it feels like I’m not the best source to discuss things related to diversity, given how dominant my ‘type’ is. I do feel like I’ve learnt a lot over the past few years, and so this is me sharing what I’ve learnt, and how I’ve learnt, but also noting that I may get things wrong, and feedback and corrections are very much welcome.

The target audience for this talk was other straight, white guys – I hope they can connect with my own evolution of thinking. I’m sure plenty of what I have to say is obvious (and possibly condescending – but I hope not!) to those who don’t have as much inherent privilege in Australian society.

At the recent Link Festival, Angus Hervey noted that he was a young white guy, but didn’t want to become a old white guy – or at least, the kind of old white guy that seems to be causing so many problems in our world – and this strongly resonated with me. Our world is wonderfully diverse, and yet our leaders, our media, our world views (especially those of us who fit the dominant type) are not. How do we change that?

What?

And as for what I mean when talking about diversity: a plurality of everything: gender and gender identity, sexuality, race, religion, age, financial situations, physical ability, political views, and so on. I’m not going to even try to provide a definitive list, because there’s just too many things to take into account, and I’m sure I’ll forget some.

I’ve noted this before in other talks I’ve given, but bringing together a wide group of people together and getting them to think in the same way is not diversity. The goal is not just diversity of peoples, but diversity of thought and culture.

Why?

If you’re not sold on why diversity matters, well, here’s just a few reasons, from different viewpoints – pick one that works for you:

  • Diverse teams are more likely to be successful (there are studies that back this up).
  • Nature is a great example of diverse environments/systems being more resilient, and imitating this to have more resilient societies is a good thing.
  • Diverse groups are more interesting! You end up with a wider mix of ideas, which can lead to more innovation and wisdom (which is part of our goal with Trampoline).
  • I like a culture where no one gets left behind, and where everyone matters. Something that strives for fairness and equality – and I think this is only possible in a diverse society.

How?

If you only get one thing from this post, it should be this: practice empathy and compassion as much as possible, towards as many people as possible.

And listen, because your path through life is different to others – be wary of your own assumptions, and be open to hearing others’ perspectives. Keep in mind that society usually serves the dominant ‘type’ – which, certainly in Australia, is straight, white men. You may initially struggle to understand others’ perspectives because of this – we can unconsciously surround ourselves with friends and media that reinforce our presumptions, which makes breaking out of that filter bubble all the more difficult.

This growth in awareness helped me become more aware of the privilege I have, and the power implicit in that privilege. From there, I can then aim to drive that power in ways that can help others. Of course, this is (always) a work in progress, as I’m always learning.

The listening and learning is grounded in a lack of ego – a recognition that it’s not about you. White guys: don’t get carried away with your own righteousness and announce that you’re fixing the system, that you’re an ally. Go and read André Arko’s wise words, and then promote the voices of those who would otherwise not be heard. You already have the privilege and influence – try to share that around.

Another thing to keep in mind is the intersectional nature of discrimination/diversity. Putting people in boxes – whether that’s white box, eg: white, or many, eg: white, straight, male – will not capture a fair representation of who they are, nor will it provide a clear picture of the discrimination they may face. The impact of discrimination and oppression is deep and complex, and proposed actions to deal with this need to understand that complexity as much as possible.

When it comes to people speaking about discrimination they suffer from, they may be angry, and their perspectives may not be calm, and you may not think they’re rational – but you should listen and seek to understand anyway. Their anger is justified, and they shouldn’t have to sugarcoat their perspective just to please you (especially if you’re in a comfortable position of privilege).

My journey has come from a position of blissful ignorance, and once upon a time I would have been in favour of meritocracy. That is most definitely not the case any more – sure, meritocracy could be considered, but only in a society where everyone is on equal footing, and we’ve all had the same opportunities, and no one has any conscious or unconscious biases. Impossible.

And because of this, I think there is value in things like affirmative action and diversity quotas. They’re just one small step, mind you.

I’m going to finish this section the same way I started: please, ground all of this in empathy and compassion.

Practical Examples

I have a habit of running events, and more and more I’m trying to have these events reflect my growing awareness. Through this, one of my goals – particularly in the Melbourne Ruby community, where Mario Visic and I just wrapped up our two year stint of running monthly events – has been to create welcoming, friendly, safe spaces.

Granted, you can’t please everyone, but that doesn’t mean you shouldn’t try. Good satire (and good comedy) should be directed at those higher up on society’s ladder, but good communities should aim at the other end of the spectrum: assist those who don’t get the same opportunities and implicit support as the majority.

This perhaps isn’t so much about diversity in a direct sense, but I believe it can help: when you’re dealing with groups of people, particularly at semi-organised social events, be wary of cliques. Newcomers can find established groups daunting as it is, and cliques just reinforce that. Don’t be afraid to ask those in cliques to make an extra effort to talk to those who don’t have anyone to talk to.

Certainly, at RubyConf AU (which happened last month, and I was one of the organisers) I think this is one area where we can clearly improve. I’d love to see official greeters and social connectors, as a role for volunteers and other regulars: if you spot someone who looks a little lost or lonely, go and have a chat with them.

Also, strive to make events as accessible as possible – and this covers things from dietary requirements, to wheelchair access, to hearing assistance, to facilitating child-care. Don’t be afraid to make a statement on this too (something that we learnt from the eurucamp organisers).

Speaking of statements, I think it’s important to show people that you take creating a safe space seriously. Codes of conduct are a good first step, but make sure they outline the desired behaviour, what will not be tolerated, and how you will go about enforcing this. You need to walk the talk – saying “we’ve not had any problems before” is not enough.

Lastly – and this is another thing I need to be better at – be aware that language matters, from how you describe your events through to how you address people. The term ‘guys’ is an excellent example – some consider it to be gender-neutral, others don’t. You could argue about semantics and context – or, you could find a more welcoming term that suits more people.

At the end of the day, being a great event host is hard work, and maintaining an awareness about peoples’ needs is hard work, but both really help to strengthen an event or community and make them welcoming places for wider audiences.

The Shoulders of Giants

I cannot say this enough: my perspective is constantly growing and evolving and improving, and there are many people to thank for this – in particular, my parents, and my dear friend Melina Chan. A lot of my recent growth has come from Twitter – here’s a selection of folk who’ve helped me (whether knowingly or not):

I don’t want to hold up these people as the token ‘diverse’ Twitter accounts I follow, because that’s not the case (and some have been friends for many years), but they have opened my eyes to a broader and better understanding of diversity, discrimination, and the world. Following all of these wise minds would be an excellent move.

All of what I’ve written above comes with the disclaimer that I don’t think I’m doing this topic justice, and many people have written far better things on this topic (the links I’ve shared are definitely worth exploring). If I can provide people with just a small step towards a much deeper understanding, then that’s a fantastic thing. Thanks for reading, this turned into a longer essay than I expected!

10 Feb 2015

RubyConf AU 2015: Thank You

Last week RubyConf AU 2015 took place in Melbourne. A year prior to that, I’d put my hand up to run it… and over the course of twelve months, had assembled an excellent team, lined up speakers, venues, and a whole bunch of fun.

On Wednesday morning, it became real, as the workshops kicked off. By Saturday evening, it was finished with our after party at the Melbourne Lawn Bowls club in Flagstaff Gardens.

Going by the feedback we’ve received, I think it’s safe to say it was a success – at the very least, I’m thrilled with what we achieved.

But, of course, it would not have been possible without contributions from many, many people. I do want to list them here, even though it’s guaranteed I’ll forget someone and then feel terrible once I realise.

Firstly: to our sponsors, who not only gave us considerable amounts of money (no small thing in itself), but trusted and supported our efforts to grow the Australian Ruby community. Thank you Envato, realestate.com.au, Redbubble, reinteractive, Digital Ocean, JobReady, Torii Recruitment, GitHub, Pluralsight, BuildKite, Lookahead Search, EngineYard, Soundcloud and Travis CI.

To our venues: Jasper, Zinc, and Deakin Edge. You provided fantastic spaces for our community to listen, learn, eat and socialise within. A special thank you to the AV team at Deakin Edge: Blake, Wes and Brad, plus our own video recorder Anthony, returning yet again to make sure our talks are captured for future generations.

To our stenographer Rebekah, who provided live captioning of our conference proceedings. She was not only extremely good at her job, but also responded to Keith and Josh’s banter in style.

To the weather gods – Melbourne’s traditionally fickle weather gave us four days of warm sunshine, which was perfect for showing off our fine city.

To the team behind our ticketing system Tito, who helped us with beta features and late night support.

To the Ruby Australia committee, who were super supportive when I first asked about running this conference, and provide essential and appreciated financial and organisational support. You play a massive part in the health and success of our community.

To our event manager Deborah Langley, and her colleague Sam. Engaging Deb to work on our event made our lives a great deal easier, and helped us to achieve great things. Plus, Deb and Sam helped the running of the conference and events purr along smoothly.

To our volunteers, lead by the inestimable Liam Esler and Mel Sherrin, and our stage manager Maxine Sherrin. You took excellent care of our attendees and speakers, kept things running to schedule, and deserve all of the credit for how calmly the conference ran.

To Amanda Neumann and Darcy Laycock, who worked with me to select presenters from our massive selection of proposals. We agonised over which talks made the cut (and there were many excellent choices that missed out), but I think our choices were great ones!

To our local Rubyists: Healesville guide Pete Yandell, and cycling leaders Gareth Townsend & Gus Gollings, who all ensured our attendees from near and far got to experience a different aspect of Melbourne beyond just the conference sessions.

To our fabulous illustrator Dougal MacPherson, who, with his 15 minute drawings hat on, drew a picture of every session (including workshops), which then became lovely gifts for our speakers.

To Tim Lucas, for his tireless work on our slick website, plus the corralling of our beautiful and popular t-shirts – which were designed by Magdalena Ksiezak (for the conference) and Carla Hackett (for the Rails Girls workshops).

To the organisers of the previous RubyConf AU events – Keith Pitty, Martin Stannard_, Michael Koukoullis, Josh Price, Elle Meredith, Jason Crane, Georgina Robilliard, and Steve Gilles. We have only been able to create this event by standing on your shoulders and reaping the rewards of your hard work.

To Ben Askins, who kicked off the bonding of our fantastic Australian Ruby community by organising the very first Rails Camp. That event changed my life.

To the large number of conferences that provided inspiration, including (but certainly not limited to) JSConf US and EU, FutureRuby, NordicRuby, eurucamp, and Web Directions: Code.

To our speakers, workshop presenters, Rails Girls organisers, and our entertaining and excellent MCs Josh Kalderimis and Keith Pitt. We gave you the stage, and you made us so very proud.

To my fellow organisers: Melissa Kaulfuss, Matt Allen, and Sebastian von Conrad. Through our shared vision and skill-set we have crafted a special event, all contributing in different and most definitely valued ways. I really cannot thank you enough.

To our employers: Inspire9, Envato, Lookahead Search and Icelab, who supported us in our endeavour, with time and patience and suggestions.

To our families, who recognised the commitment we had to give to make this real, and looked after us, loved and supported us. You’re the very definition of amazing.

To everyone else who helped in any way – I was inundated with offers of support and assistance over the past year, and while I didn’t have the opportunity to take everyone up on that, the offers themselves are greatly appreciated.

And finally, to everyone who attended the conference, and the broader Ruby community. It feels far more that we’ve done this with you than for you.

Thank you all, so very, very much.

24 Mar 2014

MICF 2014 Recommendations

The Melbourne International Comedy Festival is about to kick off this year – and I can’t wait! It truly is the best time to be in Melbourne.

Because I often see more than my fair share of comedy shows, I’m often asked for recommendations… and while it’s hard to be super sure what’s good this year before any performances have happened, I’ve scanned through the program to put together a list of performers I regularly enjoy. In alphabetical order…

  • Bane Trilogy – I was lucky enough to catch all three parts of this in Edinburgh a couple of years ago, and they’re all excellent. Mobster-style storytelling, with all roles performed by one man, Joe Bone. Start with part one (on Tuesday and Friday nights), and if it takes your fancy, see the other two as well.
  • The Boy with Tape on his Face – silent comedy, but it’s a tonne of fun, even if you do get called up on stage as part of the audience participation. I’ll put money on him being big this year in Melbourne.
  • Celia Pacquola – Regularly hilarious, clever, and with a bit of heart too (perhaps my favourite comedy recipe).
  • David O’Doherty – All his shows feel kinda the same – but that’s not a bad thing at all. Music and comedy that’s both charming and funny.
  • Felicity Ward – her show in 2012 was nominated for the Barry Award (best show of the festival), and would have been a deserving winner. Hilarious even while sometimes being extremely personal.
  • Hannah Gadsby – last year’s show was very brave and very funny and I wish I could see it again. I’ve already got tickets to see this year’s show.
  • Justin Hamilton – he’s a stalwart of the Australian comedy scene, and yet I only saw his show for the first time in 2012, and it was a clever mix of laughs and storytelling. Cue much regret for missing him in previous years. Odds of me making that mistake again this year are slim.
  • Michael Workman – his show last year, Ave Loretta, was one of the most wonderful and beautiful shows I’ve ever seen. Don’t expect comedy, even though you will laugh, because the storytelling is the highlight. I was wiping tears of sadness from my eyes at the end of Ave Loretta, for Michael Workman brings both the laughs and the feels.
  • Sammy J & Randy – the description on the site sums them up perfectly: catchy songs with chaotic tomfoolery and outbursts of filth. Rarely clean, often very, very funny.
  • Tegan Higginbotham – Tegan’s recent solo shows been great narrative stand-up shows, and she doesn’t get the attention she deserves. Given she’s not performing in the duo of chaotically hilarious Watson this year, I’ll be making an extra effort to see this show.
  • Tim Key – some people hate his odd, dry style, but I’ve loved his shows (at least, the two I’ve had the great pleasure of catching). His last show involved him having a bath on stage. Not your average comedy, but all the more enjoyable if it does float your boat.
  • Wil Anderson – I think Wil’s stand-up shows are better than anything he’s done on television… last year’s Goodwil was an excellent mix of intelligence and wit.
  • And for bonus points, my good friend Ben Hopper is performing in The Law Revue – this is Ben’s first festival performing, so there’s no past performances to judge by. Ben’s a funny guy though, so I’m expecting a top show!

These are just a dozen shows that I feel super happy with recommending to all and sundry. There are a few hundred that are part of the festival, so you don’t lack for choices (or excuses) – make sure you catch a show or two before it all wraps up on April 20th.

30 Dec 2013

Melbourne Ruby Retrospective for 2013

The Melbourne Ruby community has grown and evolved a fair bit in this past year, and I’m extremely proud of what it has become.

Mind you, I’ve always thought it was pretty special. I first started to attend the meets back when Rails was young and the community in Australia was pretty new, towards the end of 2005. The meets themselves started in January of that year – almost nine years ago! – and have continued regularly since, in many shapes, sizes and venues, under the guiding hands of many wise Rubyists.

Given I’ve been around so long, it’s a little surprising I’d not had a turn convening the meetings on a regular basis (though I’d certainly helped out when other organisers couldn’t be present). After the excellent, recent guidance of Dave Goodlad and Justin French, Mario Visic and Ivan Vanderbyl stepped up – and then Ivan made plans to move to the USA. I was recently inspired by discussions around growing and improving the community at the latest New Zealand Rails Camp, and so I offered to take Ivan’s place. (As it turns out, Ivan’s yet to switch sides of the Pacific Ocean. Soon, though!)

And so, since February, Mario and I have added our own touches to the regular events. Borrowing from both Sydney and Christchurch, we’ve added monthly hack nights – evenings where there’s no presentations, but people of all different experience levels bring along their laptops and get some coding done. If anyone gets stuck, there’s plenty of friendly and experienced developers around to help.

More recently, reInteractive have helped to bring InstallFests from Sydney to Melbourne. They are events to help beginners interested in Ruby and Rails get the tools they need installed on their machines and then go through the process of setting up a basic blog, with mentors on hand to help deal with any teething problems.

For the bulk of Melbourne Ruby community’s life, the meets have been announced through Google groups – first the Melbourne Ruby User Group, then in the broader Ruby or Rails Oceania group. It’d become a little more clear over the past couple of years that this wasn’t obvious to outsiders who were curious about Ruby – which prompted the detailing of meeting schedules on ruby.org.au – but there was still room for improvement. reInteractive’s assistance with the InstallFest events was linked to their support with setting up a group on Meetup.com – and almost overnight we’ve had a significant increase in newcomers.

Now, many of us Rubyists are quite opinionated, and I know some find Meetup inelegant and, well, noisy. I certainly don’t think it’s as good as it could be – but it’s the major player in the space, and it’s the site upon which many people go searching for communities like ours. The Google group does okay when it comes to discussions, but highlighting upcoming events (especially if you’re not a regular) is not its forte at all.

We’ve not abandoned the Google group, but now we announce events through both tools – and the change has been so dramatic that, as much as I’m wary of supporting big players in any space, I’d argue that you’d be stupid not to use Meetup. We’ve had so many new faces come along to our events – and while we still have a long way to go for equal gender representation (it’s still predominantly white males), it’s slowly improving.

With the new faces appearing, we held a Newbie Night as one of our presentation evenings (something that’s happened a couple of times before, but certainly not frequently enough). Mario and I were lucky enough to have Jeremy Tennant step up to run this and corral several speakers to provide short, introductory presentations on a variety of topics. (Perhaps this should become a yearly event!)

We’re also blessed to have an excellent array of sponsors – Envato, Inspire9, Zendesk, reInteractive and Lookahead Search have all provided a mixture of money, space and experienced minds. We wouldn’t be where we are now without you, your support is appreciated immensely.

Mario and I have also spent some time thinking a bit deeper about some of the longstanding issues with tech events, and tried to push things in a healthier direction:

At many of the last handful of meetings for this year, instead of pizza, we’ve had finger food from the ASRC Catering service, tacos from The Taco Guy, and a few pancakes as well. In each case we’ve ensured there’s vegetarian, gluten-free and lactose-free options. This trend shall certainly continue!

The drinks fridge at Inspire9 (our wonderful hosts for the past couple of years) now have plenty of soft drinks and sparkling mineral water alongside the alcoholic options – and we’ve been pretty good at making sure jugs of tap water are available too. There’s also tea and coffee, though we need to be better at highlighting this.

We’ve also adopted Ruby Australia’s Code of Conduct for all Melbourne Ruby events. This is to both recognise that our community provides value and opportunity to many, and to make it clear we want it to continue to be a safe and welcoming place, offline and online.

We’re by no means perfect, and I’m keen to help this community grow stronger and smarter over the coming year – but we’ve got some great foundations to build on. The Melbourne Ruby community – and indeed, the broader Australian Ruby community – is growing from strength to strength, and a lot of that is due to the vast array of leaders we have, whose shoulders we are standing on.

Alongside the regular city meets, there are Rails Camps twice a year, RailsGirls events becoming a regular appearance on the calendar, and the second RubyConf Australia is in Sydney this coming February. I’m looking forward to seeing what 2014 brings – thanks to all who’ve been part of the ride thus far!

22 Jul 2013

Rewriting Thinking Sphinx: Introducing Realtime Indices

The one other feature in the rewrite of Thinking Sphinx that I wanted to highlight should most certainly be considered in beta, but it’s gradually getting to the point where it can be used reliably: real-time indices.

Real-time indices are built into Sphinx, and are indices that can be dynamically updated on the fly – and are not backed by a database sources. They do have a defined structure with fields and attributes (so they’re not a NoSQL key/value store), but they remove the need for delta indices, because each record in a real-time index can be updated directly. You also get the benefit, within Thinking Sphinx, to refer to Ruby methods instead of tables and columns.

The recent 3.0.4 release of Thinking Sphinx provides support for this, but the workflow’s a little different from the SQL-backed indices:

Define your indices

Presuming a Product model defined just so:

class Product < ActiveRecord::Base
  has_many :categorisations
  has_many :categories, :through => :categorisations
end

You can put together an index like this:

ThinkingSphinx::Index.define :product, :with => :real_time do
  indexes name

  has category_ids, :type => :integer, :multi => true
end

You can see here that it’s very similar to a SQL-backed index, but we’re referring to Ruby methods (such as category_ids, perhaps auto-generated by associations), and we’re specifying the attribute type explicitly – as we can’t be sure what a method returns.

Add Callbacks

Every time a record is updated in your database, you want those changes to be reflected in Sphinx as well. Sometimes you may want associated models to prompt a change – hence, these callbacks aren’t added automatically.

In our example above, we’d want after_save callbacks in our Product model (of course) and also our Categorisation model – as that will impact a product’s category_ids value.

# within product.rb
after_save ThinkingSphinx::RealTime.callback_for(:product)

# within categorisation.rb
after_save ThinkingSphinx::RealTime.callback_for(:product, [:product])

The first argument is the reference to the indices involved – matching the first argument when you define your index. The second argument in the Categorisation example is the method chain required to get to the objects involved in the index.

Generate the configuration

rake ts:configure

We’ve no need for the old ts:index task as that’s preloading index data via the database.

Start Sphinx

rake ts:start

All of our interactions with Sphinx are through the daemon – and so, Sphinx must be running before we can add records into the indices.

Populate the initial data

rake ts:generate

This will go through each index, load each record for the appropriate model and insert (or update, if it exists) the data for that into the real-time indices. If you’ve got a tonne of records or complex index definitions, then this could take a while.

Everything at once

rake ts:regenerate

The regenerate task will stop Sphinx (if it’s running), clear out all Sphinx index files, generate the configuration file again, start Sphinx, and then repopulate all the data.

Essentially, this is the rake task you want to call when you’ve changed the structure of your Sphinx indices.

Handle with care

Once you have everything in place, then searching will work, and as your models are updated, your indices will be too. In theory, it should be pretty smooth sailing indeed!

Of course, there could be glitches, and so if you spot inconsistencies between your database and Sphinx, consider where you may be making changes to your database without firing the after_save callback. You can run the ts:generate task at any point to update your Sphinx dataset.

I don’t yet have Flying Sphinx providing full support for real-time indices – it should work fine, but there’s not yet any automated backup (whereas SQL-backed indices are backed up every time you process the index files). This means if a server fails it’d be up to you to restore your index files. It’s on my list of things to do!

What’s next?

I’m keen to provide hooks to allow the callbacks to fire off background jobs instead of having that Sphinx update part of the main process – though it’s certainly not as bad as the default delta approach (you’re not shelling out to another process, and you’re only updating a single record).

I’m starting to play with this in my own apps, and am keen to see it used in production. It is a different way of using Sphinx, but it’s certainly one worth considering. If you give it a spin, let me know how you go!

11 Jul 2013

Gutentag: Simple Rails Tagging

The last thing the Rails ecosystem needs is another tagging gem. But I went and built one anyway… it’s called Gutentag, and perhaps worth it for the name alone (something I get an inordinate amount of happiness from).

My reasons for building Gutentag are as follows:

A tutorial example

I ran a workshop at RailsConf earlier this year (as a pair to this talk), and wanted a simple example that people could work through to have the experience of building a gem. Almost every Rails app seems to need tags, so I felt this was a great starting point – and a great way to show off how simple it is to write and publish a gem.

You can work through the tutorial yourself if you like – though keep in mind the focus is more on the process of building a gem rather than the implementation of this gem in particular.

A cleaner code example

Many gems aren’t good object-oriented citizens – and this includes most of the ones I’ve written. They’re built with long, complex classes and modules, are structured in ways that Demeter does not like, and aren’t particularly easy to extend cleanly.

I have the beginnings of a talk on how to structure gems (especially those that work with Rails) sensibly – but I’ve not yet had the opportunity to present this at any conferences.

One point that will definitely feature if I ever do get that opportunity: more and more, I like to avoid including modules into ActiveRecord and other parts of Rails – and if you peruse the source you’ll see I’m only adding the absolute minimum to ActiveRecord::Base, plus I’ve pulled out the logic around the tag names collection and resulting persistence into a separate, simple class.

I got a nice little buzz when I had Code Climate scan the source and give it an A rating without me needing to change anything.

Test-driven design

I started with tests, and wrote them in a way that made it clear how I expected the gem to behave – and then wrote the implementation to match. If you’re particularly keen, you can scan through each commit to see how the gem has evolved – I tried to keep them small and focused.

Or, just have a read through of the acceptance test files – there’s only two, so it won’t take you long.

So?

There are a large number of other tagging gems out there – and if you’re using one of those already, there’s no incentive at all to switch. I’ve used acts-as-taggable-on many times without complaints.

But Gutentag certainly works – the README outlines how you can use it – and at least people might smile every time they add it to a Gemfile. But at the end of the day, if it’s just used as an example of a simple gem done well, I’ll consider this a job well done.

09 Jul 2013

Rewriting Thinking Sphinx: Middleware, Glazes and Panes

Time to discuss more changes to Thinking Sphinx with the v3 releases – this time, the much improved extensibility.

There have been a huge number of contributors to Thinking Sphinx over the years, and each of their commits are greatly appreciated. Sometimes, though, the pull requests that come in cover extreme edge cases, or features that are perhaps only useful to the committer. But running your own hacked version of Thinking Sphinx is not cool, and then you’ve got to keep an especially close eye on new commits, and merge them in manually, and… blergh.

So instead, we now have middleware, glazes and panes.

Middleware

The middleware pattern is pretty well-established in the Ruby community, thanks to Rack – but it’s started to crop up in other libraries too (such as Mike Perham’s excellent Sidekiq).

In Thinking Sphinx, middleware classes are used to process search requests. The default set of middleware are as follows:

  • ThinkingSphinx::Middlewares::StaleIdFilter adding an attribute filter to hide search results that are known to not match any ActiveRecord objects.
  • ThinkingSphinx::Middlewares::SphinxQL generates the SphinxQL query to send to Sphinx.
  • ThinkingSphinx::Middlewares::Geographer modifies the SphinxQL query with geographic co-ordinates if they’re provided via the :geo option.
  • ThinkingSphinx::Middlewares::Inquirer sends the constructed SphinxQL query through to Sphinx itself.
  • ThinkingSphinx::Middlewares::UTF8 ensures all string values returned by Sphinx are encoded as UTF-8.
  • ThinkingSphinx::Middlewares::ActiveRecordTranslator translates Sphinx results into their corresponding ActiveRecord objects.
  • ThinkingSphinx::Middlewares::StaleIdChecker notes any Sphinx results that don’t have corresponding ActiveRecord objects, and retries the search if they exist.
  • ThinkingSphinx::Middlewares::Glazier wraps each search result in a glaze if there’s any panes set for the search (read below for an explanation on this).

Each middleware does its thing, and then passes control through to the next one in the chain. If you want to create your own middleware, your class must respond to two instance methods: initialize(app) and call(contexts).

If you subclass from ThinkingSphinx::Middlewares::Middleware you’ll get the first for free. contexts is an array of search context objects, which provide access to each search object along with the raw search results and other pieces of information to note between middleware objects. Middleware are written to handle multiple search requests, hence why contexts is an array.

If you’re looking for inspiration on how to write your own middleware, have a look through the source – and here’s an extra example I put together when considering approaches to multi-tenancy.

Glazes and Panes

Sometimes it’s useful to have pieces of metadata associated with each search result – and it could be argued the cleanest way to do this is to attach methods directly to each ActiveRecord instance that’s returned by the search.

But inserting methods on objects on the fly is, let’s face it, pretty damn ugly. But that’s precisely what older versions of Thinking Sphinx do. I’ve never liked it, but I’d never spent the time to restructure things to work around that… until now.

There are now a few panes available to provide these helper methods:

  • ThinkingSphinx::Panes::AttributesPane provides a method called sphinx_attributes which is a hash of the raw Sphinx attribute values. This is useful when your Sphinx attributes hold complex values that you don’t want to re-calcuate.
  • ThinkingSphinx::Panes::DistancePane provides the identical distance and geodist methods returning the calculated distance between lat/lng geographical points (and is added automatically if the :geo option is present).
  • ThinkingSphinx::Panes::ExcerptsPane provides access to an excerpts method which you can then chain any call to a method on the search result – and get an excerpted value returned.
  • ThinkingSphinx::Panes::WeightPane provides the weight method, returning Sphinx’s calculated relevance score.

None of these panes are loaded by default – and so the search results you’ll get are the actual ActiveRecord objects. You can add specific panes like so:

# For every search
ThinkingSphinx::Configuration::Defaults::PANES << ThinkingSphinx::Panes::WeightPane

# Or for specific searches:
search = ThinkingSphinx.search('pancakes')
search.context[:panes] << ThinkingSphinx::Panes::WeightPane

When you do add at least pane into the mix, though, the search result gets wrapped in a glaze object. These glaze objects direct any methods called upon themselves with the following logic:

  • If the search result responds to the given method, send it to that search result.
  • Else if any pane responds to the given method, send it to the pane.
  • Otherwise, send it to the search result anyway.

This means that your ActiveRecord instances take priority – so pane methods don’t overwrite your own code. It also allows for method_missing metaprogramming in your models (and ActiveRecord itself) – but otherwise, you can get access to the useful metadata Sphinx can provide, without monkeypatching objects on the fly.

If you’re writing your own panes, the only requirement is that the initializer must accept three arguments: the search context, the underlying search result object, and a hash of the raw values from Sphinx. Again, the source code for the panes is not overly complex – so have a read through that for inspiration.

I’m always keen to hear about any middleware or panes other people write – so please, if you do make use of either of these approaches, let me know!

RssSubscribe to the RSS feed

Recent Links

Recent Posts

Tag Density

About Freelancing Gods

Freelancing Gods is written by , who works on the web as a web developer in Melbourne, Australia, specialising in Ruby on Rails.

In case you're wondering what the likely content here will be about (besides code), keep in mind that Pat is passionate about the internet, music, politics, comedy, bringing people together, and making a difference. And pancakes.

His ego isn't as bad as you may think. Honest.

Here's more than you ever wanted to know.

Ruby on Rails Projects

Other Sites

Creative Commons Logo All original content on this site is available through a Creative Commons by-nc-sa licence.