Give Real has great execution on technology concepts. We use Rails, Merb and other tools to integrate with a diverse set of APIs. Abstraction and planning for scale are our middle names.

ShardTheLove is a library we use in both Ruby frameworks, Rails and Merb, for scaling our MySQL databases. The code has been humming on our servers for over a year now. There is an alternative library named DataFabric that has similar functionality to ShardTheLove, but the configuration felt clunky and the codebase was getting less maintenance than Mike looks to be giving it today. ShardTheLove is built with support for migrations, testing/RSpec, a flexibility of partitioning patterns, a simple syntax, support for Rails & Merb, and a great name :-) .

Using ShardTheLove

ShardTheLove puts your databases into one of three buckets:

  • System - The default database, and the one where non-sharded data lives.
  • Directory - A database intended for fast queries directing you to the correct shard.
  • Shards - Database servers storing partitioned data.

Partitioning your data by user is a common model. Here’s an example with ShardTheLove randomly choosing a server for your user’s data to live on, storing that location on the directory, then later finding that shard and data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
class User < ActiveRecord::Base
  acts_as_directory

  has_many :reviews

  before_create :choose_shard

private

  def choose_shard
    return unless self.shard.blank?
    self.shard = ShardTheLove.shards[rand(ShardTheLove.shards.length)]
  end

end

class Review < ActiveRecord::Base
  acts_as_shard

  belongs_to :user

end

# Now we can create some reviews.
#

@user = User.create( :email => 'test@example.com' )

ShardTheLove.with(@user.shard) {
  Review.create(
    :user => @user,
    :body => 'Man those aussie meat pies are great!'
}

# And read them back.
#

@user = User.find_by_email( 'test@example.com' )

ShardTheLove.with(@user.shard) {
  @user.reviews
}

If a model calls acts_as_directory or acts_as_shard it behaves as such. If nothing is called, it is assumed to be a model on the system database. This a great way to lazy-configure your app and add sharding without immediately refactoring all your models.

There is documentation up at the ShardTheLove github project page on migrations and testing integration. It’s exciting to get a tool we code with every day out in public! Check back for another post on how ShardTheLove can integrate smoothly with controller actions.

By Matthew Beale
2 comments
  1. Chirag Patel at November 25th, 2009 at 12:19 AM

    Thanks! I’m considering either using this or Project VodelMort (http://project-voldemort.com/). Does ShardTheLove work with PostgreSQL?

  2. Matthew Beale at November 25th, 2009 at 08:59 AM

    Hi Chirag. I haven’t tried using STL on Postges, but the logic is built in on top of ActiveRecord and there are no MySQL specific blocks of code.

    The one place you would want to check is be the rake task db:directory:purge_test. Rails has a DB specific case block there.

    http://github.com/mixonic/ShardTheLove/blob/master/lib/tasks/databases.rake

    Let me know if you need a hand, any feedback would help me update the docs. This is a really complex thing to explain!

Comments are closed for this article.