Freelancing Gods 2009

There’s been a bit of changes under the hood with Thinking Sphinx lately, and some of the more recent commits are pretty useful.

Small Stuff

First off, something neat but minor – you can now use decimal, date and timestamp columns as attributes – the plugin automatically maps those to float and datetime types as needed.

There’s also now a cucumber-driven set of feature tests, which can run on MySQL and PostgreSQL. While that’s not important to most users, it makes it much less likely that I’ll break things. It’s also useful for the numerous contributors – just over 50 people as of this week! You all rock!

New Delta Possibilities

The major changes are around delta indexing, though. As well as the default delta column approach, there’s now two other methods of getting your changes into Sphinx. The first, requested by some Ultrasphinx users, and heavily influenced by a fork by Ed Hickey, is datetime-driven deltas. You can use a datetime column (the default is updated_at), and then run the thinking_sphinx:index:delta rake task on a regular basis to load recent changes into Sphinx.

Your define_index block would look something like the following:

define_index do
  # ... field and attribute definitions

  set_property :delta => :datetime, :threshold => 1.day
end

If you want to use a column other than updated_at, set it with the :delta_column option.

The above situation is if you’re running the rake task once a day. The more often you run it, the lower you can set your threshold. This is a bit different to the normal delta approach, as changes will not appear in search results straight away – only whenever the rake task is run.

Delayed Reaction

One of the biggest complaints with the default delta structure is that it didn’t scale. Your delta index got larger and larger every time records were updated, and that meant each change got slower and slower, because the indexing time increased. When running multiple servers, you could get a few indexer processes running at once. That ain’t good.

So now, we have delayed deltas, using the delayed_job plugin. You’ll need to have the job queue being processed (via the thinking_sphinx:delayed_delta rake task), but everything is pushed off into that, instead of overloading your web server. It means the changes take slightly longer to get into Sphinx, but that’s almost certainly not going to be a problem.

Firstly, you’ll need to create the delayed_jobs table (see the delayed_job readme for example code), and then change your define_index block so it looks something like this:

define_index do
  # ... field and attribute definitions

  set_property :delta => :delayed
end

Riddle Update

As part of the restructuring over the last couple of months, I’ve also added some additional code to Riddle, my Ruby API for Sphinx. It now has objects to represent all of the configuration elements of Sphinx (ie: settings for sources, indexes, indexer and searchd), and can generate the configuration file for you. This means you don’t need to worry about doing text manipulation, just do everything in neat, clean Ruby.

Documentation on this is non-existent, mind you, but the source shouldn’t be too hard to grok. I also need to update Thinking Sphinx’s documentation to cover the delta changes – for now, this blog post will have to do. If you get stuck, check out the Google Group.

Sphinx 0.9.9

One more thing: Thinking Sphinx and Riddle now both have Sphinx 0.9.9 branches – not merged into master, as most people are still using Sphinx 0.9.8, but you can find both code sets on GitHub.

So, a bit over a year ago, as we edged closer and closer to voting John Howard out of office, Rudd’s team make the stupid election promise of an internet filter. Exactly a year ago, in anger and frustration, I sent off a letter to Stephen Conroy, the then new Minister for Broadband, Communications and the Digital Economy.

Twelve months later, what have we learnt?

• Conroy is slow to respond to letters.
• Conroy isn’t particularly good at engaging in reasoned discourse.
• Conroy is good at staying on message.
• The internet filter is unlikely to be completely opt-out – there’ll just be two levels.
• Question Time is a farce and lacking in intelligent discussion.
• Those passionate about the issue have mobilised, with the EFA, GetUp, Steve Hopkins and Elias Bizannes amongst those leading the way.
• Trials of internet filters have focused mainly on HTTP and HTTPS traffic. Not IM, peer-to-peer traffic, or newsgroups.
• ISPs, including iiNet and Internode, are not fans of the proposed legislation
• The Greens and Liberals aren’t supporting it either. Nick Xenephon seems to be on the fence, and unsurprisingly Stephen Fielding of Family First wants to add legal content to be blocked by the filter.
• Conroy insists that our filter will be much like those in Sweden and Canada – yet both of those filters, according to the Government’s own feasibility study, are voluntary.
• There is a live pilot underway.

We’ve had petitions, rallies, phone bombs, emails and letters. It doesn’t seem to be making a dint in Conroy’s plans. I think face-to-face meetings is the best way forward. If your local member is from the ALP, then meet with them. If they’re not, you will have a Senator who is. If you’re uncomfortable about going alone, find some other like-minded souls. A group may well have a larger impact.

If you’re in Melbourne, that’s where Stephen Conroy is based – I’d love to hear some feedback of anyone who has met with him. I’m currently overseas, so I haven’t got around to that yet – I’ve only managed to meet with my then-local member (I’ve since changed electorates), and it was pretty clear that I knew more about the issue than he did.

I’d also recommend not bothering with arguments relating to civil liberties, censorship or keeping legal access to pornography. While I don’t disagree that these are important and valid, it’s not going to win over anyone. Personally, I try to keep the message about how the filter isn’t going to work, just like past filters haven’t worked, and thus it’s a waste of money and time. You need to express understanding that the Government’s goal is laudable, but the approach isn’t. The ends does not justify the means.

Also have alternative plans to suggest – whether that’s recommending parents stay aware of what their children are doing online, an opt-in filter for those who want it, or something like the previous Government’s NetAlert software. (Although that wasn’t downloaded much at all – so is there really the demand for an internet filter?)

Online action is great, but it doesn’t have anything close to the effect that face-to-face communication does. If you really want to make a difference, get into those politicians’ offices.

One of the things that’s really stood out in my time here in Cambodia is that the ex-pat community consumes alcohol and cigarettes much more than back at home in Australia. Leaving alone any judgement over the right and wrong of this, I just wanted to share some of the thoughts a discussion with Mel about this led to.

While the ex-pat community here comes from a variety of nations and backgrounds, and this isn’t a reflection of all of them, it’s widely accepted that volunteer workers in developing countries often drink (and use other substances) more often than at home, wherever home may be.

From my own experience with the circles I mix in regularly, I’m in the minority as a non-smoker, and I’ve not met anyone else here who doesn’t drink – excluding locals, of course (although plenty of Khmer do enjoy their beer).

It’s been suggested to me that this is a coping mechanism – life here is definitely more challenging, especially if you’re not just passing through as a tourist. You’re in the middle of a different culture, facing different food, people, language and locations. Alcohol and smoking can be seen as something familiar – grounding elements in the face of all the other change. Mel suggested it’s much the same with backpackers in hostels. Indeed, religion can fill the same role for other people. For me as I’ve been travelling over the last year, Ruby was my familiarity to some extent – I was able to meet new people and feel a bit more comfortable in new countries and facing new experiences, using common skills and knowledge as a stepping stone.

The smoking in particular is a bit jarring though – it’s pretty common for people to light up in bars and restaurants (things you can’t do in Melbourne anymore), and so I’ve been questioning why it becomes more acceptable here.

Mel’s answer – which makes sense to me – is that there’s an absence of social norms here. You don’t have your home culture around you, and you don’t completely assimilate into Khmer culture (especially in the case of females, as that would mean getting married if you’re not already, and not going out after dark – well, in the less liberal parts of society here). In this absence, the few familiar things can become more common, fill in the gaps in your time.

Maybe it’s possible to draw some parallels to western society – do those who engage in binge-drinking not have other coping mechanisms to deal with stress? Is it related to stress at all? Or is this too long a bow to draw?

Context

The fourth Australian Rails Camp happened back in the middle of November – and it was unsurprisingly and extremely enjoyably awesome, just like the previous four. Ryan and Anthony did a sterling job with putting it all together.

I probably talked a bit too much – I certainly felt I had more than my fair share of peoples’ focus – and while I rabbited on about Sphinx and Ginger, the topic I really enjoyed ranting about was freelancing, because it became far less about me, and far more about sharing the wealth of everybody’s experiences. I provided a few starting points, and then wise RORO minds added their own thoughts and opinions.

I can’t reproduce all that here, though. I wouldn’t do it justice. What I can do is go over the same notes I had then, and you can add your 2 cents (or five dollars) in the comments.

Freelancing Maths

One of the first things you need to be aware of, when you start freelancing, is how much to charge. I didn’t have a clue, but some more business-minded friends put me on the right track, so I’m sharing their advice here – don’t give me any credit for it.

So, let’s assume you want to start freelancing, and you have a target of earning $80,000 over the year (yes, some of you may say that’s too low, but others will say it’s too high – it’s just an example, okay?). You can use this as a basis for figuring out an hourly rate. There’s 52 weeks in a year, 5 days in a week, and 8 hours in a day…

 52 weeks
x 5  days
x 8 hours
x ?  rate
_________
   80,000

But wait a second – are you really going to work all of those 52 weeks? I doubt it. You’ll need time off for annual leave, sick leave and public holidays – the times when an employer would still pay you when you’re not slaving away. Australian annual leave is four weeks, sick leave is usually two, let’s add in another one for public holidays, and that brings us down to 45.

 45 weeks
x 5  days
x 8 hours
x ?  rate
_________
   80,000

What are the odds you’re going to have work all the time though, and are you really going to have eight billable hours each day? Unless you’re some sort of machine, the answer’s no, trust me. So lets drop eight down to six.

     45 weeks
x     5  days
x     6 hours
x 59.25  rate
_____________
       80,000

One thing we’ve missed in our calculations is superannuation. Again, using Australia as the example (because it’s all I can reliably comment on), you’re supposed to be putting away 9% of your income into your super account. Let’s factor that in:

     45 weeks
x     5  days
x     6 hours
x 64.59  rate
_____________
       87,200

Okay, so we can get an hourly rate of about $65 from that maths. And that could be fine… but maybe you’ve been eyeing off RailsConf or RubyConf or other such events. They’re not cheap – and hopefully employers would normally fork out the cash to get you there. You’re the employer now, so how are you going to afford it? Add an allowance into your calculations.

Again, due to the remoteness of Australia, it’s extra expensive to get to any of the major Ruby conferences. If we assume you’ll get to two of them (again, could be extravagant for some of you, but this is all hypothetical), then I’m adding a touch over $12,000 – flights, hotels, insurance, the conference tickets – to bring us to a nice round $100,000 target.

Also, I’ve dropped the number of weeks down another two – it’s not like you’ll be getting anything done for your clients as you jet around.

     43 weeks
x     5  days
x     6 hours
x 77.52  rate
_____________
      100,000

Okay, our final hourly rate is about $77.50.

I know a lot of the more experienced developers are looking at that value and thinking it’s pretty low – and going by market rates (for Ruby developers), it’s definitely below average. Some say a good ballpark figure for a decent Rails developer is $100/hour – USD or AUD (remember when the two currencies were almost on par?). This doesn’t mean you should charge that much (or that little) – but it should factor into your thinking.

All that said, you need to be comfortable with what you’re billing your time at, but don’t be afraid to charge what you’re worth. If the idea of having more cash than you expect scares you, there’s plenty of charities who would like to be your friend. Or, you could just work less, and spend the extra time on cool things (and they don’t even have to involve code!)

Freelancing Profile

Knowing what to charge is useful, but it’s not going to bring in the clients. Being known will help that problem, though – and there’s a few things you can do to help that.

Blog

Interpret how you will – a normal blog, twitter, tumblelog, even gists and pasties – sharing your ideas and knowledge is a great way to get your name known to others. It also helps build some human connections, via comments, emails or directed tweets. If it is valuable, they will find you (and if you think they need help, use a site like RubyFlow or RubyCorner to bring in some eyeballs).

Talk

If there’s a neat bit of code you’ve found, library you’ve come across (or written), or knowledge you think is valuable to others, offer to talk about it! It can be at your local Ruby group, or at something like a Rails Camp or BarCamp, or if you’re really comfortable up on stage, think about applying for a RailsConf or RubyConf slot.

I’m not a natural public speaker – but my confidence has grown in leaps and bounds from giving talks to fellow developers. Granted, I need to build up a bigger repertoire of topics, but I’m a bit less nervous about standing up and announcing my thoughts and opinions to others. It all started with an email from Tim Lucas asking what I was going to talk about at the first Rails Camp – and now Rails Camp folks are probably sick of hearing my voice.

They know who I am, though, and they know what code I’ve written. And that’s led to a referral or two for Rails work (usually Sphinx-related).

Socialise

Networking is a dirty word – and I can see how building connections with others for the purpose of connections, instead of meeting cool people, is a bit dirty. The much more fun alternative is to socialise – go out to social events, find those drinks happening in the evenings of conferences, have a conversation with a person you’ve not met before at your local Ruby meet.

Down the track, you will find these people may throw work your way – or maybe you’ll just learn cool new ways to code, or share some of your own knowledge, or make a good friend. All chalked up as wins in my book.

Release Code

Releasing your own code – from snippets to plug ins to full-blown applications – is a great way to show peers that you know what you’re talking about. It also shows potential clients that too, and reaffirms that you’re worth the rate you’re charging, and that you can be creative.

In my own case, I’ve done the occasional bit of Sphinx consulting due to my work on Thinking Sphinx.

Coincidentally, doing all these things are rewarding in and of themselves. I don’t do them to bring in work, I do them because they’re fun and I meet awesome people, which is (I think) the best approach. The opportunities they lead to are just an added bonus.

Your Turn

So, what’s your advice to a budding freelancer? Is there anything here that’s a bit Ruby or developer-centric? Any more general suggestions to keep in mind?

Also, please keep in mind I’m not an expert. I think the above advice is useful, but it is just advice. There’s no hard and fast rules that should be followed.

And the name of this blog has nothing to do with my work lifestyle, but the idea of deities who freelance for each other. Don’t take it as an indication of my ego. Honest.

Yes, I know it’s been quiet here – there’s several decent blog posts on the way though. I just wanted to post a quick reminder for any Discworld fans in or near Australia that Nullus Anxietas 2 is happening at the end of February, and will be awesome fun. There’s also a discount of $15 off attendee prices (a Hogswatch special) if you get in before the end of the year (going by AEDT/Melbourne time, of course).

The above little clip was done by the fantastically talented Snowgum Films (also responsible for Run Rincewind Run).

The Australian Government’s contentious ‘Clean Feed’ internet censorship proposal has got some media attention lately – and by and large, it’s been rightly critical of Senator Conroy’s plans. If you’re not familiar with it, I recommend you read my letters to Conroy and peruse nocleanfeed.com.

In the middle of last year, the previous Government commissioned a closed environment testing trial. The results of these were released recently, and the values have been used by both sides to tout the usefulness/uselessness of filters. Handily, these results are available to the public, so I’ve skimmed through the extensive PDF - although I claim no solid understanding of it all – to figure out where the figures are from.

Firstly, a few facts:

• Six different filtering approaches were tried (with the codenames Alpha, Beta, Gamma, Delta, Theta and Omega).
• These trials were conducted on a purpose-built network.
• The network is similar in scale to a Tier 3 ISP.
• The trials covered speed changes, the effectiveness of blocking blacklisted material, and the valid sites blocked incorrectly.
• Most filters were only tested against HTTP and HTTPS traffic. Gamma and Omega were also applied to emails, and Delta skipped on HTTPS.

A full grid of numbers is at the bottom of the post, but let’s go through a few comparisons.

Speed vs Blocking

The speed results here are really mixed. One (Delta) doesn’t drop much at all, but two (Alpha and Gamma) are horrific. All filters manage to block at least 87% of the blacklist – but only Beta comes really close, with 98% (losing a third of the speed in the process though).

Speed vs False Positives

Note that the scale on the Y Axis drops a bit, but we still get another set of mixed results. None of them are perfect on the false-positives front, and the closest is Gamma on 1.3% – but that comes with severly limited speeds. And really – there are a lot of websites out there. Even 1% covers a fair chunk of the net.

Blocking vs False Positives

Here there’s something of a trend, although you have to be looking for it: better blocking effectiveness means a higher number of false positives. That’s not good, people.

Takeaways?

There’s really not that much to work off here, no matter what side of the fence you’re on. The main things to keep in mind are:

• None of the solutions are perfect.
• All had issues with false-positives
• This was done on something approaching a Tier 3 ISP - will the performance speeds decrease if we applied these filters on a Tier 1 or 2 ISP? My money’s on yes.
• It wasn’t Conroy who commissioned this study, so it can’t be pinned against him.
• Delta, which is arguably the only viable filter judging by performance, still missed 9% of the blacklisted sites.
• None of the filters were tested against newsgroups, IM(Instant Messaging), or peer-to-peer traffic. I’d imagine HTTP/HTTPS filters are relatively easy, so expecting the same performance and effectiveness for other protocols sounds like a pipe dream to me.

Raw Numbers

	Performance			Effectiveness
Filter	PPI	API	CPI	BRI	OBI
Alpha	92%	16%	17%	90%	2.6%
Beta	99%	67%	68%	98%	7.5%
Gamma	98%	14%	14%	87%	1.3%
Delta	99%	98%	100%	91%	2.4%
Theta	78%	76%	99%	95%	7.8%
Omega	101%	79%	78%	94%	2.9%

Glossary of sorts: PPI (Passive Performance Index) is the relative speed when a filter is attached but not running. API (Active Performance Index) is the relative speed when the filter is running. CPI (Change In Performance Index) is API when using PPI as the reference point (instead of uninhibited network speeds). BRI (Blocking Rate Index) is the percentage of blacklisted sites stopped, and OBI (Overblocking Index) is the percentage of friendly sites overzealously blocked.

A quick note to let anyone using (or interested in using) Sphinx and/or Thinking Sphinx that my Peepcode PDF has just been published, and contains a wealth of information on how to mix Sphinx with ActiveRecord (via Rails or Merb).

It’s been great working with Geoffrey Grosenbach to get this written up, and I’m pretty stoked to see the final results – hopefully others will enjoy it as well.

Also, a massive thank you to all the contributors to Thinking Sphinx – it wouldn’t be quite so cool if it wasn’t for all the patches (facilitated by GitHub’s forking).