You wonder why people even read Techcrunch when they write comments like ‘Twitter is a waste of time’, ‘why even bother with Foursquare when Facebook will just kill them’, etc.

Enthusiasm, and even partisanship, is not what I decry as the lack of objective tech journalism. It isn’t like to political journalism; it better resembles scientific journalism: describing the latest images from the Hubble telescope, communicating a sense of wonder at the cosmos. The vocation of tech journalists should be to educate the public. And being enthusiastic is probably necessary to be effective.

I do think journalists should consider themselves public servants. What I dislike is sensationalism, gossip, and a lack of depth. Merely announcing how much money a company has raised, what features a product offers – with this journalists are failing to perform their duty.

It reminds me of Le Petit Prince:

Grown-ups like numbers. When you tell them about a new friend, they never ask questions about what really matters. They never ask: “What does his voice sound like?” “What games does he like best?” “Does he collect butterflies?”. They ask: “How old is he?” “How many brothers does he have?” “How much does he weigh?” “How much money does his father make?” Only then do they think they know him.

I work in a very narrow field that has unfortunate occasions of intense interest. MG Siegler wrote a terrible piece about an outage that Foursquare suffered. The thesis of the article: these clowns can’t keep it up when EC2 goes down on them! But he doesn’t ask the good questions: why did they go down when only one region of EC2 crashed? How do you do multi-region (i.e., multi-master) data-centers? How does a ramen-eating startup deal with suddenly being of historic significance and problems that took google 5 years to solve? Has Foursquare’s unusual technical choices (e.g., EC2, Scala) helped them innovate or increased risk or both?

This would not require long-form journalism. Just ask one question and get the opinion of a few experts. Or just ask a meaningful question and don’t try to answer it. This is not: “Update: 4 hours later Foursquare returns. Will they be another Twitter?’

Perhaps I’m bitter because I work for Twitter.

The audience, of course, lacks reading comprehension skills and any genuine empathy for machines. So there is this unfortunate paradox that the role of the press is to educate the public but the press will not do so until the public demands better and the public is too incompetent to demand anything worthwhile.

The Challenge

Sounds simple right? Wrong! A programmer without control over the source-code of that service must be able to later add enhancements such as statistics collecting, timeouts, memoization, and so forth. There are a few more requirements:

• the “enhancements” must be specified in a configuration object which is consumed at run-time (e.g., it could be based on user-input).
• The enhancements are ordered (stats collecting wraps timeouts, not the other way around) but it must be possible to reverse the order of the enhancements at run-time.
• The enhancements must be “surgical” and not “global”. That is, it must be possible to simultaneously have two query services, one reversed and one not reversed, and even have a query service without any enhancements.
• The code must be thread-safe or support using concurrency by some means. The same query service can be used by multiple threads as well as multiple threads with different query services.

Most programming contests emphasize things that are not modularity. This contest emphasizes things that are modularity. And its the olympics and don’t you want to be olympic? So go ahead and prove to the world that Design Patterns are no longer necessary, or that Haskell is the raddest thing since the movie Rad, or that Peanut Butter is better than Jelly. This is much more fun than inflamatory blog posts and angry reddit comments, yes? Yes.

Context

Please read my blog post on modularity techniques in Scala and various responses (in Clojure, in Ruby, in Ruby again)

Languages & Techniques

You can use any programming language you like (functional, OO, stack-based) and any techniques you like. Your main goal should be clarity and flexibility. Terseness is valuable where it adds to clarity. Creative and unusual solutions are welcome, though. Try to implement all of the functionality (including transactions) of the sample solution.

How to Play

Fork this gist. Your program must produce the same output as mine (included below). I will collect all forks at link them here (it may take me a while, so please be patient).

Sample Program

I’ve tricked out some Factories and wrote this little diddy in Ruby:

config = [
  [:memoizing, []],
  [:timing_out, [2.seconds]],
  [:stats_collecting, [Stats.new]]
]
query_factory = config.inject(Query) do |factory, (decorator_name, settings)|
  "#{decorator_name.to_s.classify}QueryFactory".constantize.new(factory, *settings)
end

[query_factory, ReversingQueryFactory.new(query_factory)].each do |qf|
  query_evaluator = QueryEvaluator.new(ConnectionPool.new(20), qf)
  query_evaluator.transaction do |t|
    t.select("SELECT ... FROM ... FOR UPDATE ...")
    t.execute("INSERT ...")
    t.execute("INSERT ...")
  end
  puts
end

Sample Output

Note the reversing order of the enhancements and the memoization of Query instantiation:

Forward:
Instantiating Query Object
Selecting SELECT ... FROM ... FOR UPDATE ... on #
Did not timeout! Yay fast database!
Measured select at 1.00 seconds
Instantiating Query Object
Executing INSERT ... on #
Did not timeout! Yay fast database!
Measured execute at 1.00 seconds
Executing INSERT ... on #
Did not timeout! Yay fast database!
Measured execute at 1.00 seconds

Backward:
Instantiating Query Object
Selecting SELECT ... FROM ... FOR UPDATE ... on #
Measured select at 1.00 seconds
Did not timeout! Yay fast database!
Instantiating Query Object
Executing INSERT ... on #
Measured execute at 1.00 seconds
Did not timeout! Yay fast database!
Executing INSERT ... on #
Measured execute at 1.00 seconds
Did not timeout! Yay fast database!

Let me explain why. Consider this little Scala program. It uses “futures”, which are a way to schedule computation to be done in parallel from the main flow of a program. They are sometimes a natural way of modeling the most efficient scheduling of program execution. Usually you schedule in advance some expensive work that can be done in parallel and then you do something else in the meantime. Only when you really need the result of the original computation do you block and wait (and hopefully only very briefly since you scheduled the work way in advance!). Here is a “typical” Java-ish Futures library used from Scala:

    private val executor = new ThreadPoolExecutor(
      poolSize, maxPoolSize,
      keepAlive.inSeconds, TimeUnit.SECONDS,
      new LinkedBlockingQueue[Runnable],
      new NamedPoolThreadFactory(name))

    val future = new FutureTask {
      doSomeWork
    }

    executor.execute(future)

If you come from a dynamic language like Ruby or Python you will probably have a visceral reaction like “Yeck! Look at all that horrible boilerplate. Convention over configuration!” Wouldn’t it be nice if you could just do something like:

    val future = new Future {
      doSomeWork
    }

It seems nice but its nicety is just an illusion. All that boilerplate is really important when you work at massive scale and where efficiency really matters. These magic numbers like the thread pool size and the kind of queue you use to schedule work can vastly impact the performance of your application. And the “right” configuration depends entirely on the nature of the problem you’re solving and how callers of this code behave. What all of this weird boilerplate provides is a way to configure the behavior of the system; it doesn’t assume there’s one right way of doing things. And that is precisely how modular software behaves: modular code is code designed to grow past the assumptions of just one user. Modularity really matters when your software isn’t a little throw-away program.

Twitter recently open-sourced Querulous, a minimal database querying library for Scala. We use it in several projects in Twitter, but it was designed principally to meet the extreme demands FlockDB, our distributed, fault-tolerant graph database. FlockDB demands extremely low-latency (sub millisecond) response times for individual queries. Any excessive indirection from an ORM would be unacceptable. Furthermore, because FlockDB processes tens of thousands of queries per second across dozens of shards, FlockDB must collect extensive statistics on the performance and health of the various shards in order to direct traffic to the most efficient place.

So Querulous was designed for querying databases at low latency, massive scale, and with easy operability. It has flexible timeouts, extensive logging, and rich statistics. But as FlockDB became more mature and sophisticated, the demands grew greater. We needed different health-check and timeout strategies in different contexts. It became clear that Querulous would need to be made extremely modular and extremely configurable to work at all.

So we set about to re-write Querulous using my favorite modularity techniques: Dependency Injection, Factories, and Decorators. In other words, everything you hate about Java.

The design patterns of modularity

In order for code to be modular it must have few hard-coded assumptions. In Object-Oriented software this means something very particular since the essence of an Object-Oriented program is that its structure is organized around the types of objects. Therefore, the most fundamental, anti-modular assumption in Object-Oriented software is the concrete type of objects. Any time you write new MyClass in your code you’ve hardcoded an assumption about the concrete class of the object you’re allocating. This makes it impossible, for example, for someone to later add logging around method invocations of that object, or timeouts, or whatever isn’t anticipated a priori.

In a very dynamic language like Ruby, open classes and method aliasing (e.g., alias_method_chain) mitigate this problem, but they don’t solve it. If you manipulate a class to add logging, all instances of that class will have logging; you can’t take a surgical approach and say “just objects instantiated in this context”. (Update: some people are asking “what about metaclasses? Metaclasses do not solve this problem at all because if you do not have control over the caller of Foo.new then you cannot later add new behavior to the metaclass; it has to be hardcoded at the site of manufacture. The point of this technique is to avoid knowing in advance what behavior you will add in, to make it configurable!)

There are standard design patterns to mitigate this, namely Dependency Injection, Factories, and Decorators. By injecting a Factory (a function that manufactures objects) as a parameter to a function that needs to create objects, you allow a programmer to later change his mind about what Factory to inject; and this means the programmer can change the concrete types of objects as his heart desires. And by using Decorators, the programmer can mix and match functionality easily, stack one thing on top of another like so many legos. Let’s look at an example.

Here I have a Query object, with methods like #execute(). I want to add timeouts around all queries. I start by creating a QueryProxy that routes all method invocations through an over-ridable method: #delegate:

    abstract class QueryProxy(query: Query) extends Query {
      def select[A](f: ResultSet => A) = delegate(query.select(f))
      def execute() = delegate(query.execute())
      def cancel() = query.cancel()

      protected def delegate[A](f: => A) = f
    }

Then, to implement timeouts, I create a Query Decorator:

    class TimingOutQuery(timeout: Duration, query: Query) extends QueryProxy(query) {
      override def delegate[A](f: => A) = {
        try {
          Timeout(timeout) {
            f
          } {
            cancel()
          }
        } catch {
          case e: TimeoutException =>
            throw new SqlTimeoutException
        }
      }
    }

This Decorator delegates to the underlying query object the execution of the query, but it wraps that execution in a Timeout.

As an aside, it is interesting to note that the Decorator pattern is just the Object-Oriented equivalent of function composition in a functional language. Scala makes this especially explicit since everything is both an Object and a Function (it is a function if it is an object that responds to the method #apply()). A Decorator around an object that only implements #apply() is pure Function-composition as you would see in Haskell, ML, and so forth. I might phrase this as: function composition is a degenerate case of the Decorator pattern.

The implementation of the Timeout function is shown for the curious. It uses threads and is weird but cool.

    object Timeout {
      val timer = new Timer("Timer thread", true)

      def apply[T](timeout: Duration)(f: => T)(onTimeout: => Unit): T = {
        @volatile var cancelled = false
        val task = if (timeout.inMillis > 0) Some(schedule(timeout, { cancelled = true; onTimeout })) else None
        try {
          f
        } finally {
          task map { t =>
            t.cancel()
            timer.purge()
          }
          if (cancelled) throw new TimeoutException
        }
      }

      private def schedule(timeout: Duration, f: => Unit) = {
        val task = new TimerTask() {
          override def run() { f }
        }
        timer.schedule(task, timeout.inMillis)
        task
      }
    }

(An alternative implementation of Timeout could use Futures, but that’s a subject for another blog post)

Modularity and testing techniques

One of the principal advantages of (or stated another way, one of the principal motivations for) writing Decorator-oriented code is how easy it is to write isolated unit tests of that code. To test the timeout functionality of the TimingOutQuery we don’t need to interact with a database at all. We can write behavioral/mockish tests like this:

    val latch = new CountDownLatch(1)
    val query = new FakeQuery(List(resultSet)) {
      override def cancel() = { latch.countDown() }

      override def select[A](f: ResultSet => A) = {
        latch.await(2.second.inMillis, TimeUnit.MILLISECONDS)
        super.select(f)
      }
    }
    val timingOutQuery = new TimingOutQuery(query, timeout)

    timingOutQuery.select { r => 1 } must throwA[SqlTimeoutException]
    latch.getCount mustEqual 0

If the timeout functionality was just inlined into the #select() method of the source code of the Query class, or “bolted on” as an alias_method_chain in Ruby (or added as “advice” in some AOP shit) you could not write this test without talking to the database and somehow finding a query that takes long enough that it will actually hit the timeout. Because we instead use Decorators, to test the code we can use a fake query that implements the Query interface but that doesn’t talk to the database at all. Here we use a CountDownLatch to “halt” execution for a bounded amount of time, thus triggering the timeout.

Tying it together with Factories

Back to our original mission. So now we have a way of layering on timeout functionality on top of a Query object. But how do we ensure that Timeouts get used when we want them to? The thing that glues this all together is to make sure that everybody that needs to instantiate a Query object never ever calls new Query directly. We provide instead a Factory as a parameter to the method that needs to manufacture the object. The programmer chooses which Factory to provide at runtime. Here is a Factory that makes TimingOutQueries:

    class TimingOutQueryFactory(queryFactory: QueryFactory, timeout: Duration) extends QueryFactory {
      def apply(connection: Connection, query: String, params: Any*) = {
        new TimingOutQuery(queryFactory(connection, query, params: _*), timeout)
      }
    }

Since TimingOutQueries are decorators around regular Queries, to manufacture a TimingOutQuery you have to first manufacture a regular Query. In this example, the TimingOutQueryFactory takes another Factory as an argument. This could be a simple QueryFactory or something more complex–allowing Factories to be composed indefinitely. With this we stack together timeouts, logging, statistics gathering, and debugging like so many pieces of legos. This smacks of the oft-ridiculed Java AbstractFactoryFactoryInterface. But let me put it bluntly: AbstractFactoryFactoryInterface's are how you write real, modular software–not little fart applications.

This seems like a bit of a mind-fuck because we here have Factory Decorators that take Decorated Factories that make Decorated Queries. It’s so meta! (Actually, “meta” in Greek means nothing like “meta” in English. “Meta” plus the accusative means “after” so Aristotle’s Metaphysics is actually just a book “after [the book on] physics”. Anyway.) So all these crazy FactoryFactoryDecorators sound kind-of scary at first but it is just the kind of abstraction on top of abstraction and closure under composition that allows complex software to be made simple. Manage complexity by taking many things and re-conceiving of them as just one thing; this one thing is then combined with many other things and the process is repeated up the ladder of abstraction until you reach the Godhead.

Taking this to the next level

To take this even further, let’s add a new feature: per-query timeouts. At one point in the history of FlockDB, there was a global 3-second timeout. This was really stupid given that our most common query has a latency of 0.5ms and a standard deviation of 2ms. If you have a global timeout you must set your timeout around your most expensive query not your most common query (otherwise, your most expensive query will always timeout!). But for a production system, cheap frequent queries, if they start exceeding 2 standard deviations, can take down your site. So a sensible timeout for these frequent queries is like 5ms. But we had it set to 3,000 ms!! Yikes. So let’s change it!

    class PerQueryTimingOutQueryFactory(queryFactory: QueryFactory, timeouts: Map[String, Duration])
      extends QueryFactory {

      def apply(connection: Connection, query: String, params: Any*) = {
        new TimingOutQuery(queryFactory(connection, query, params: _*), timeouts(query)) // YAY
      }
    }

That’s it. We’ve now implemented a new Timeout strategy in one line of code! And to wire it all together it is a piece of cake! Querulous makes no assumptions about how best to implement a timing-out strategy, it doesn’t even assume you’ll want timeouts (in fact, there are some cases you don’t want any timeouts). Querulous achieves modularity by providing an “injection point” for the programmer to layer on custom functionality. It takes QueryFactories as a parameter to the method, which can return arbitrarily decorated Queries.

I love this example because it’s so simple but yet it’s no toy. It also emphasizes the value of Dependency Injection more generally than just with Factories. We could have written the TimingOutQuery with a static global constant (probably the most common programming technique):

    class TimingOutQuery(query: Query) extends QueryProxy(query) {
      val TIMEOUT = 3.seconds

But intead it is injected as a parameter to the constructor to the TimingOutQuery:

     class TimingOutQueryFactory(queryFactory: QueryFactory, timeout: Duration) extends QueryFactory {
        def apply(connection: Connection, query: String, params: Any*) = {
          new TimingOutQuery(queryFactory(connection, query, params: _*), timeout)
        }
      }

This enables the TimingOutQueryFactory to invoke a function to choose the appropriate timeout for this query. In this case, we just look some shit up in a hash table (timeouts(query)) and we’re done.

Yes, all this FactoryFactory bullshit is exactly what you hate about Java. But it’s amazing not how just short this code is but that it could be configured by any programmer anywhere, regardless of whether they have access to the source code that actually instantiates and executes queries. Any user of Querulous can decide if she want timeouts or not, and she can decide if they also want debugging, stats gathering, and so forth–Querulous hard-codes no assumptions. So, yay modularity.

Postscript:

Check out Querulous.


User.order('users.id DESC').limit(20).includes(:items)


You can see that all queries are now chainable. This has two principal benefits: first, you can programmatically generate complex queries (based on user input or other data); and second, you can refactor your queries to eliminate code duplication in a way that you never could before.

The way that such chaining is possible is that each “query” (e.g., Users.order(...)) returns not an array of data, but a query object that one can manipulate still further. Only when you try to iterate through the data is a SQL query composed and then executed on the database.

If this sounds to you similar to “named_scope”, you’re spot on. After I wrote “named_scope” I immediately asked myself “but what if every query was a named_scope? What if named_scope were the rule and not the exception?”. I saw, unfortunately, that in order to do this I would need to re-write ActiveRecord.

So I set out to re-write ActiveRecord. I announced my intentions (and then began a quixotic quest that would ultimately fail, then later succeed thanks to other people’s help).

I realized, though, that in order to re-write ActiveRecord, it would be best for the purposes of encapsulation to create a whole new self-contained query language; ideally, one that could underlie both DataMapper and ActiveRecord. So I began work on Arel, an Object-Oriented interpretation of the Relational Algebra. The Relational Algebra is a mathematical model for representing “queries” on data. In many ways it resembles SQL (which is modeled on the algebra) but with slightly different jargon (“project” for “select”, “select” for “where”, etc.).

But the Relational Algebra differs greatly from SQL in that it has the property of “closure” under composition. That is, every operation on a relation (such as adding a “where” clause (aka “selection”)) returns a new relation where one can subsequently add more operations. SQL distinguishes queries from result sets: and so queries cannot be composed since queries are just strings (e.g., SELECT * FROM foo). To manipulate a SQL query we must manipulate a string. There is a string algebra, but its operations are things like substring, concatenation, substitution, and so forth–not so useful. In the Relational Algebra, there are no queries per se; everything is either a relation or an operation on a relation. Connect the dots and with the algebra we get something like “everything is named_scope” for free.

Ever since I read the book Structure and Interpretation of Computer Programs I have been fascinated by the concept of closure under composition. (If you haven’t read this book, you must; it’s like Euclid’s Elements). There is a chapter on a Picture Language that represents transformations on pictures; each transformation generates a picture that can be further transformed:


(define wave2 (beside wave (flip-vert wave)))


The picture is ugly but the code is beautiful

In other words, the picture language has closure under composition. To this day, when I work in completely different languages and on somewhat different problems I’m fascinated by the same concept of closure. This gist which I will someday re-write into a blog-post illustrates how I often use the Decorator pattern with Factories and Dependency Injection to structure programs. This is closure writ-large, and is a different way to think about the architecture of your software than the usual “Domain Driven”/Ontological approach. (Actually, it has much in common with the “Test Driven” approach).

Anyway. Just before I began working on Arel I learned of SqlAlchemy. SqlAlchemy was inspiring, but it was written in Python (and thus useless for RoR) and also had a couple design decisions that I thought were fundamentally (not superficially) wrong. I intended to do something radical, something SQL Alchemy couldn’t do: a pure relational algebra with closure under composition even in the presence of aggregations. To explain my motivations though, I want to illustrate two “beautiful” querying problems; one is solved well in SQL, the other not. The first is the technique to do a group-wise maximum of a certain column. In this example, we try to find the most expensive dealer of a given article. for all articles:


SELECT s1.article, s1.dealer, s1.price
FROM shop s1
LEFT JOIN shop s2 ON s1.article = s2.article AND s1.price < s2.price
WHERE s2.article IS NULL;


This, to me, is one of the great mind-blowing SQL queries. It means something like "find the dealer for which there exists no dealer selling the same product at a lesser price". It is the “double negative” restatement of “most expensive”. In other words, “not the lesser expensive”. It’s this kind of radical restatement of problems that I hope to accomplish with my work. Of course this actually has nothing to do with Arel, although it's expressible trivially in Arel. But it provides some insight into my thinking about why I write software and why I am concerned so much about how software is written.

The other query I am fascinated by is the following. Suppose we have a users table and a photos table and we want to select all user data and a *count* of the photos they have created. In SQL, it is expressed like thus:


SELECT users.*, photos_aggregation.cnt
FROM users
LEFT OUTER JOIN (SELECT user_id, count(*) as cnt FROM photos GROUP BY user_id) AS photos_aggregation
ON photos_aggregation.user_id = users.id


You'll note the use of the derived table in the subselect. This is terrible, in my opinion. Only advanced SQL programmers know how to write this (I’ve often asked this question in job interviews I’ve never once seen anybody get it right). And it shouldn’t be hard! It’s just what happens when you lack closure under composition. I was determined to be able to write code like the following:


photo_counts = photos.
group(photos[:user_id]).
project(photos[:user_id], photos[:id].count)


And finally,


users.join(photo_counts).on(users[:id].eq(photo_counts[:user_id]))


If this seems simple to you, you are as naive as I was when I first started working on Arel. It is anything but simple: Arel needs to recognize when you are joining with relations that are aggregations (i.e., have a "group by") and produce a derived table (a subselect with a unique name) and then join with that. Since we have closure under composition, no matter how many joins and how many aggregations you have, Arel needs to generate unique derived tables and scope all attribute names to them. Since derived tables can be nested infinitely the implementation of this is incredibly tricky; one of the hardest problems I've ever solved, and I'm pretty happy how it all turned out.

But therein lies the rub. All of my open-sourced projects share two attributes. First, the source code is extremely stylized and meant to be /read/ more than /used/. They are literary arguments in code much more than a useful piece of infrastructure. If you read the source-code to Arel you will see a highly affected "combinator" and "interpreter" style of coding.

If you look at Screw.Unit , for example, not only is it an argument about how tests should be written but it is probably the most unusual Javascript code you will ever see; it is written in a “concrete” style inspired by Self’s Morphic.

Similarly, Arel is a two-fold argument: this is how you should interact with a database, and this is how software can be written to affect a unique personal style.

The second attribute of all my open-source projects is that I release them and neglect them. This makes me a terrible steward of open-source software. But my aim is to introduce ideas into the world more so than to keep infrastructure alive. (My upcoming open-source project, FlockDB (a distributed graph database) will hopefully be different; we hope to run Twitter off of the open version indefinitely.)

And this explains why I “finished” Arel in May of 2008 but only now has it been integrated into Rails. If it were not for the heroic integration work of Emilio Tagua, the leadership of Bryan Helmkamp, and the support of DHH and Michael Koziarsky, Arel would be but an idea.

Hidden in this story is the true meaning of all information technology.

A brief statement of the problem.

There is an old logical puzzle called the Sorites Paradox, first articulated by the Megarian logician Eubulides of Miletus. It predates the stored program computer by 2,000 years but it similarly concerns the production of pastries:

Would you describe a single grain of wheat as a heap? No. Would you describe two grains of wheat as a heap? No…. You must admit the presence of a heap sooner or later, so where do you draw the line?

This problem was of keen interest to the philosophical community for thousands of years, principally because the Greek recipe for tea cakes called for two heaping tablespoons of sugar. Some philosophers went so far as to vow to grow a beard and engage in pederasty until a solution to the problem was found. But all efforts were in vain; the problem remains unsolved to this day.

Unfortunately, the problem has only become ever more acute in the modern era. In fact, far from only destabilizing the fabrication of pastries, it has further undermined every area of society. Consider the process of voting. If no one voted, one vote would affect the outcome. But if millions of people vote, one vote makes little difference.

In fact, the defining characteristic of the modern era is that every aspect of society is heaping. To understand how this came to be, we must revisit ancient history.

Some scientifical facts.

During the pre-historic era, when mankind lived in trees and swam in lagoons, we lived in small clans and tribes of dozens up to a few hundred people. During this long honeymoon period in homo sapiens history, we evolved our cognitive abilities through careful grooming and diligent fornicating. The brain developed the ability to speak languages and sympathize with other people and feel jealousy and kindness and all that other stuff. In other words, we evolved a social technology that equipped us for living in society and dealing with the ordeals caused by other people.

But that social technology is ill-equipped to deal with the humongous heaps of the modern world. We meet hundreds of people every year and can’t remember any of their names. We evolved language and vocal chords to cover long distances but somebody put skyscrapers in the way and anyway we now live like really long distances away, like you have to fly an airplane to see them or at least ride a bicycle. And then there are these celebrity neanderthals like Ashton Kutcher who are adored for their plumage by a multitude who greedily read every banal detail of his private life in magazines like Star that ship millions of copies to every end of a giant sphere whose radius is 6,378.1 kilometers; or on a Twitter that delivers every inane thing he tweets to 4 million people, 10 times daily. It goes without saying that this would not be possible if not for a defect in our programming in the face of the massive scale of the world.

Everything has become a giant fucking heap.

The modern world is profoundly inhumane. Mankind is incapable of reasoning about the heaping constructs of mass culture using the technes of intimacy that are an hundred thousand years old. For example, we need to be constantly reassured that celebrities are just like us. They eat waffles and pick up dry cleaning. If we do not share this understanding of Ashton Kutcher, we become overwhelmed by existential anomie and commit suicide.

Human beings need to understand one another in terms of primordial intimacies because man has no other tools for understanding the solicitations of man. But if the size of the world no is on longer amenable to intimacy technologies, then mankind must invent information technologies that rehumanize the world.

Thus the proliferation of social software on the web. The reification of the social graph in Friendster; the Facebook Newsfeed and the Twitter; and the Foursquare all serve this one purpose: to rehumanize an inhumane world. Let’s consider each of these technologies one at a time.

1. Friendster’s reification of the social graph makes it possible to understand the ties that bind us all together when we only have room in our brains for the intrigues of a few dozen relationships.
2. The Facebook Newsfeed and the Twitter make it possible to share in the thoughts and intimate moments of those who inhabit different neighborhoods, and different schools, and different jobs, and make different choices than us from amongst the vast cornucopia of mass-produced art sold to us by the culture industry. Finally,
3. the Foursquare coordinates the alienated existence of cosmopolitan voluptuaries into a shared bacchanal.

Technology and self-criticism.

I cannot help but be a technological optimist because technology is mankind’s only bulwark against the barbarism of heaps. But I’ll grant that technology is imperfect; it is sometimes fair to criticize the Tyranny of Technology. The usual argument goes that all these tweets and text messages and notifications that “a software update is available” leave no space quiet, provide no room for contemplation. It is true: we do live in a world of interruptions; interruptions created by information technology. But we should not be surprised by this fact and no more should we despair of it. One generation of technology solves the problems of the previous but causes problems of its own. The next generation of technology repeats this story; a story as old as mankind itself. This is the dialectics of history.

Do not doubt, then, that a technology will arise to solve this problem too. We, the makers, shall fabricate a machine to produce quiet and contemplation. In fact, at this very moment, I have a patent pending on a pair of contemplation goggles. And in just a few weeks I will release a beta of a contemplation-inducing goggular Twitter client. I’m sure it will be received to massive acclaim and to the profound benefit of humanity.

So! Old men: do not fear for the future. We young people, we hackers and makers, we have it all under control. We know the true meaning of information technology. We shall save us all from the giant fucking heaps.

Pre-requisite: please read my article on Write-through caching to understand why this is useful.

Most caching solutions in the Rails world involve something like Cache-Fu: an alternative API to ActiveRecord that explicitly annotates all call sites with cache rules.

• User.find(1) becomes User.get_cache(1)
• User.find(:all, ...) becomes User.get_cache("query_name", :ttl => 5.minutes) { User.find(:all, ... )}

I hate this kind of interface, which places the burden on the caller and meekly surrenders any attempt at encapsulation. Your codebase will be littered with haphazard cache rules in your controllers, views, and models.

But even worse are the explicit cache expiry rules. As you cache non-trivial queries, you’ll have to find all the of the writes in your system that could possibly invalidate the results of the query. It’s a tedious and onerous effort: after hundreds of hours of debugging you’ll finally get expire_cache in just the right places.

A solution to this brittle, messy coding style is now available, and ready for production use. `Cache Money` is a plugin for ActiveRecord that transparently provides write-through and read-through caching functionality using Memcached. With `Cache Money`, queries are automatically cached for you; and similarly, cache expiry happens automatically as after_save and after_destroy events.

This doesn’t just apply to trivial queries. Very complex, sophisticated queries are handled effortlessly; the vast majority of ActiveRecord usage is transparently materialized, indexed, and kept fresh in Memcached. Here are some examples:

• User.find(1)
• User.find(:all, :conditions => { :screen_name => 'bob' })
• Friendship.find(:all, :conditions => ['friendships.creator_id = ? AND friendships.receiver_id = ?', ...])
• users.direct_messages
• users.direct_messages.find(1)
• users.direct_messages.count
• User.find(:all, :limit => 10, rder => 'id DESC')

All of these, and much more will automatically be cached and kept fresh as you write to the database. This greatly lessens the load on your database and makes your site impervious to catastrophic replication lag.

The way `Cache Money` works is by materializing the equivalent of database indices in Memcached. It’s as if you store your indices in a distributed hash table instead of an in-process BTree. Just as with a database, you declare your indices:


class User < ActiveRecord::Base
index :screen_name
end

class DirectMessage
index :user_id
index [:user_id, :id]
end


There are lots of configurable options like TTLs:

index :user_id, :ttl => 1.day


You can also specify limits to ensure that your indices do not grow too large:

index :user_id, :limit => 500, :buffer => 20


(This keeps a rolling window of 500 items. The buffer option indicates how many “extra” you want to keep around in case of deletes in order to maintain at least 500 items. If more than 20 are deleted, the index will be repopulated to ensure there are at least 500 items in it).

This is just the tip of the iceberg. Many advanced utilities are included for even more sophisticated use, including shared locks to deal with distributed computations on shared memory, simulated transactions in Memcached (which obviates the need for locks in most cases), high-performance mocks for your tests, and in process-caches to minimize network operations during a single request-response cycle.

A version of this code is in production use at Twitter and is one part of the reason Twitter’s uptime has improved so much over the last several months. This is real, pragmatic, unmagical, production-ready code that can be a big part of your Rails scaling strategy. It is designed with massive datasets and real-world operational challenges in mind. And it’s almost effortless to use, since it requires no changes to how you use ActiveRecord.

Check out `Cache Money` on github.

With replication you can scale your database. With Master-Slave replication, you can have any number of read-slaves for one master; you perform reads (SELECT queries) on the slaves and issue writes (INSERT/UPDATE/DELETEs) on the master. With this strategy your write capacity wont scale but your read capacity can scale almost indefinitely. And anyway, since write and read actions have different performance characteristics, you’ll get the most out of your beefy machines if you’re into this role-play. By the way, the best Rails plugin for this is masochism.

Replication Lag

Still, Replication has its problems. The latency of information propagation from the master to the slave (and from slave to slave if you have a tree topology) can lead to data inconsistency.

Consider a user who creates a new account on your web site. A new record is INSERTed into the `users` table on the master database. On the next page load their record SELECTed from a random slave. But the information might not yet be propagated to the database slave! So the user gets an HTTP 500 error.

Typically, it takes only a fraction of a second for information to propagate. But it isn’t unusual to measure replication lag in seconds, minutes, or hours because of bugs in mysql, partial outages, and expensive queries. This is an every-day occurrence and you must plan accordingly.

Fortunately there is a solution, and it is write-through caching. At the same time as writing to the master database, write “through” a cache layer such as memcached. If for all reads you first try to read from cache, you will always find data: data that hasn’t yet propagated from database to database is newest and therefore most likely to be in the cache.

Easier said, harder done. You’ll have smear your code with cream-cheese to get this to work. Unless there were some magical plugin that solved this problem for you…

Testing for Replication Lag-induced defects

But you can only obtain code cream cheese if you can even figure out what part of the site needs cacheing. You’d be crazy to test, live in production, bugs caused by non-deterministic replication lag. So you need to simulate replication lag somehow.

One technique that is almost foolproof is to use a comprehensive Selenium test-suite. Create a new Rails environment with your master and slave as two local databases. Load your fixtures into both databases. Don’t establish any replication relationship between the two databases — voila! you’ve just simulated infinite replication lag. Now run your Selenium suite. I’m sure it won’t be pretty, 500s upon 500s, but that’s what will happen to your users unless you implement write-through cacheing.

Other Scaling Strategies

Now, there are other database scaling strategies than master-slave replication. Master-Master replication can skirt replication lag if you use sticky sessions. But MySQL support for master-master replication is arguably not production-ready, and in any case such replication, especially combined with sticky sessions, has worse failure characteristics than master-slave.

The unicorn strategy of “sharding”/Database partitioning does not obviate replication strategies; usually each partition has replication strategies if only for fault tolerance.

My fantasy scaling strategy involves skipping the database altogether. Rather than synchronously write to a database, write to a message queue. Have a daemon read from the queue and write to your (partitioned) (replicated) database at its leisure. In the event of a write-database failure, the daemon gets backed up but there is no downtime since your site is completely decoupled from the write database. If you’re not concerned with in-order delivery, message queues scale effortlessly. (Please in no way consider this an endorsement of either ActiveMQ or RabbitMQ). In any case, writing to the database asynchronously from a daemon is only possible with sufficient write-through cacheing.

This is why I say write-through cacheing is critical to scaling. And in the next few weeks I’ll bring forth some code that shows how to do it right.

Since I’ve left Pivotal and joined Twitter, I’ve been working on a new class of problems, principally: architecture, scaling, and large-scale deployment. I spend as much time thinking about how to safely deploy new functionality as building it. I program as much in system languages like C as in Ruby. I spend half of my time doing science: interpreting profiling data, logs, taking measurements, and performing statistical analysis.

I’m learning so much that my understanding of Software Engineering is completely changing. I love what I’m doing, what I’m learning, and I hope with this blog to share some of that enthusiasm as well as the few lessons I will learn and the many mistakes I will make.

The kinds of things I expect to write about are things like caching strategies, bottleneck analysis, fault tolerance, service-oriented architecture, deployment techniques, iterative development, simplicity, and beauty. Anyway, I hope you all enjoy it.