Magic Scaling Sprinkles

Just another WordPress.com weblog

Introducing Cache Money

with 29 comments

Pre-requisite: please read my article on Write-through caching to understand why this is useful.

Most caching solutions in the Rails world involve something like Cache-Fu: an alternative API to ActiveRecord that explicitly annotates all call sites with cache rules.

  • User.find(1) becomes User.get_cache(1)
  • User.find(:all, ...) becomes User.get_cache("query_name", :ttl => 5.minutes) { User.find(:all, ... )}

I hate this kind of interface, which places the burden on the caller and meekly surrenders any attempt at encapsulation. Your codebase will be littered with haphazard cache rules in your controllers, views, and models.

But even worse are the explicit cache expiry rules. As you cache non-trivial queries, you’ll have to find all the of the writes in your system that could possibly invalidate the results of the query. It’s a tedious and onerous effort: after hundreds of hours of debugging you’ll finally get expire_cache in just the right places.

A solution to this brittle, messy coding style is now available, and ready for production use. `Cache Money` is a plugin for ActiveRecord that transparently provides write-through and read-through caching functionality using Memcached. With `Cache Money`, queries are automatically cached for you; and similarly, cache expiry happens automatically as after_save and after_destroy events.

This doesn’t just apply to trivial queries. Very complex, sophisticated queries are handled effortlessly; the vast majority of ActiveRecord usage is transparently materialized, indexed, and kept fresh in Memcached. Here are some examples:

  • User.find(1)
  • User.find(:all, :conditions => { :screen_name => 'bob' })
  • Friendship.find(:all, :conditions => ['friendships.creator_id = ? AND friendships.receiver_id = ?', ...])
  • users.direct_messages
  • users.direct_messages.find(1)
  • users.direct_messages.count
  • User.find(:all, :limit => 10, :o rder => 'id DESC')

All of these, and much more will automatically be cached and kept fresh as you write to the database. This greatly lessens the load on your database and makes your site impervious to catastrophic replication lag.

The way `Cache Money` works is by materializing the equivalent of database indices in Memcached. It’s as if you store your indices in a distributed hash table instead of an in-process BTree. Just as with a database, you declare your indices:


class User < ActiveRecord::Base
index :screen_name
end

class Friendship
index [:creator_id, :receiver_id]
end

class DirectMessage
index :user_id
index [:user_id, :id]
end

There are lots of configurable options like TTLs:

index :user_id, :ttl => 1.day

You can also specify limits to ensure that your indices do not grow too large:

index :user_id, :limit => 500, :buffer => 20

(This keeps a rolling window of 500 items. The buffer option indicates how many “extra” you want to keep around in case of deletes in order to maintain at least 500 items. If more than 20 are deleted, the index will be repopulated to ensure there are at least 500 items in it).

This is just the tip of the iceberg. Many advanced utilities are included for even more sophisticated use, including shared locks to deal with distributed computations on shared memory, simulated transactions in Memcached (which obviates the need for locks in most cases), high-performance mocks for your tests, and in process-caches to minimize network operations during a single request-response cycle.

A version of this code is in production use at Twitter and is one part of the reason Twitter’s uptime has improved so much over the last several months. This is real, pragmatic, unmagical, production-ready code that can be a big part of your Rails scaling strategy. It is designed with massive datasets and real-world operational challenges in mind. And it’s almost effortless to use, since it requires no changes to how you use ActiveRecord.

Check out `Cache Money` on github.

Written by nkallen

December 11, 2008 at 6:25 am

Posted in Uncategorized

29 Responses

Subscribe to comments with RSS.

  1. Fantastic work nick and co, really looking forward to watching this grow.

    Now I can finally scale my hello world applications!

    Koz

    December 11, 2008 at 9:57 am

  2. Hi Nick.

    Looks like you’ve introduced the killer DB-cache layer for Rails!

    Two minor issues have come up though…

    1. The initializer code in the README has a missing close-bracket, fixed here: http://gist.github.com/34687 (GitHub is taking a worlds-worth of time to fork your repo for some reason)

    2. In lib/cash.rb there is a require ‘cash/request’, but not corresponding file so a MissingSourceFile Exception is encoutered.

    Douglas F Shearer

    December 11, 2008 at 12:48 pm

  3. This looks great, can’t wait to try it out. How do you handle migrations on models that are cached?

    Florian Munz

    December 11, 2008 at 5:23 pm

  4. Generally the way it is done is to have a version # per model. I’ll be supporting that in Cache Money in the coming days.

    nkallen

    December 11, 2008 at 5:50 pm

  5. Angry now! I hope you’re happy that, thanks to you, I won’t be able to make jokes about how Rails can’t scale in the podcast. Way to ruin it for everyone (me).

    Seriously, though, fantastic job.

    Jason Seifer

    December 11, 2008 at 11:14 pm

  6. [...] Introducing Cache Money Pre-requisite: please read my article on Write-through caching to understand why this is useful. Most caching solutions [...] [...]

    Top Posts « WordPress.com

    December 12, 2008 at 12:31 am

  7. ‘Cache Money’ just replaced ‘Vlad the deployer’ as my favourite name for a ruby project.

    This looks very interesting, I look forward to trying it out.

    Ciaran Lee

    December 12, 2008 at 12:54 am

  8. [...] is in the form of cache-money, a write-through caching library for Active Record (announced in a blog entry by Twitter's Nick Kallen). If you need a refresher on write-through caching, it's pretty [...]

  9. [...] Introducing Cache Money – Gem from the Twitter folks to implement write-through caching with Active Record and Memcached. [...]

  10. Would this replace Evan Weaver’s Interlock plugin?

    Steve Easternling

    December 13, 2008 at 2:47 am

  11. [...] Introducing Cache Money A version of this code is in production use at Twitter and is one part of the reason Twitter’s uptime has improved so much over the last several months. This is real, pragmatic, unmagical, production-ready code that can be a big part of your Rails scaling strategy. It is designed with massive datasets and real-world operational challenges in mind. And it’s almost effortless to use, since it requires no changes to how you use ActiveRecord. [...]

  12. [...] Introducing Cache Money « Magic Scaling Sprinkles A solution to this brittle, messy coding style is now available, and ready for production use. `Cache Money` is a plugin for ActiveRecord that transparently provides write-through and read-through caching functionality using Memcached. With `Cache Money`, queries are automatically cached for you; and similarly, cache expiry happens automatically as after_save and after_destroy events. (tags: scalability rubyonrails caching) [...]

  13. Wow, this is great! I’m about a couple weeks out from launching a new site and I start my performance caching today and look what pops up in my newsfeed… fantastic. You just saved me hours if not days!

    Nate Bird

    December 19, 2008 at 3:33 pm

  14. [...] Introducing Cache Money (tags: coding ruby twitter caching memcached activerecord rails) [...]

  15. This is awesome! I’ve hooked this up in my app, and am finally able to wipe out the numerous cached-find wrapper methods i have scattered throughout.

    One problem though, whenever i do a write it seems to hit an exception, getting a NoMethodError on “id_was” (in this case the field id is in play, but if i applied an index such as :user_id and then saved it would fail on “user_id_was”. Here’s a pastie of my output: http://pastie.org/360188.

    Tracing it to the source i see that in cache-money/lib/cash/index.rb line 88, it’s trying to get the original value of the attribute as such:
    original_value = object.send(“#{name}_was”)

    I’m guessing an #{attribute}_was method is dynamically defined somewhere. Any idea why this wouldn’t be working for me? Or can you point me to the dynamic definition, since we have some fairly complex method_missing hookups in our app and I have a suspicion they may be interfering.

    Salman

    January 14, 2009 at 2:35 am

  16. Awesome work, though it seems to choke on STI; any thoughts?

    Matt Darby

    January 14, 2009 at 3:10 pm

  17. Realized the issue was that AR was missing the “Dirty” functionality it gets from rails 2.1 (alternatively, you can patch in just the Dirty functionality into an older rails via this method http://blog.sidu.in/2008/05/get-dirty-objects-in-rails.html). We just happen to be upgrading to 2.2 at this time, and the issue now appears solved.

    P.S. The 07/09 commit (http://github.com/nkallen/cache-money/commit/fc4fb74a50eab3f1682b54b0ab07d314db9fd4d6) caused some pretty nasty “nil.type_cast” errors to show up when doing certain queries (i didn’t see any pattern in the queries that caused the error). I reverted to the version prior to that commit and it fixed my issues. FYI, here’s a pastie of what I was experiencing: http://pastie.org/360861

    Salman

    January 14, 2009 at 9:22 pm

  18. [...] is the lead developer of Cache-Money. Check out his blog for an excellent introduction. Cache-Money is available on github. VN:F [1.0.9_379]please wait…Rating: 0.0/10 (0 votes [...]

  19. [...] is the lead developer of Cache-Money. Check out his blog for an excellent introduction. Cache-Money is available on [...]

  20. [...] Introducting CacheMoney – in-memory database cache. I find it hard to believe this work for anything but small volumes of data. [...]

  21. [...] like to have in a look aside buffer.  Can we ask rails to take care of it all for us?  "Yes we can!" Leave a [...]

  22. CacheMoney is awesome but on saving some models I get this error:

    http://pastie.org/386324

    Basically:

    NoMethodError (undefined method `’ for #):

    When I look at that object in the console it has a “” method so I am not sure why that method is getting removed.

    Its only on some models, not all.

    Rupert

    February 11, 2009 at 7:53 pm

  23. Rupert a lot of people get this error – it has to do with which version of rails you’re using. You can add to ActiveRecord::Base:


    def (other)
    id other.id
    end

    nkallen

    February 18, 2009 at 1:45 am

  24. what about update_all, these calls ain’t update the cache, any solution?

    Bits

    March 28, 2009 at 10:57 pm

  25. Does cache-money also overload the User.find_or_create_by..(..) helper functions introduced in Rails 2.3.x?

    dkindlund

    May 17, 2009 at 10:39 pm

  26. With the help of the KCRUG, I gave a live/video presentation of Cache Money that summarizes this website:
    http://vimeo.com/4927794 (part 1 – 14:30)
    http://vimeo.com/4937895 (part 2 – 05:08)

    I’m not photogenic, but I hope this helps somebody.

    Mike Hoskins

    July 7, 2009 at 7:05 am

  27. If memcached becomes unavailable, cache-money seems to fail catastrophically — all model finders just blow up. How do you use cache-money and still degrade smoothly?

    Ian Kallen

    July 8, 2009 at 12:19 am

  28. Is Cache Money maintained?

    Is there a version planned for Rails 3?

    Mike Hoskins

    July 4, 2010 at 3:21 am

  29. fwiw Nick has told me that Cache Money is no longer maintained because Rails 3 makes the same kind of functionality trivial to implement (Arel, etc.)

    Phil Rosenstein

    July 29, 2010 at 9:55 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.