Easy Twitter Streaming API Testing with Mockingbird

My company makes two applications that focus on Twitter: CheapTweet and TweetReach. Lately, we’ve started transitioning parts of these apps to the Twitter Streaming API. To do that, we’ve been developing the flamingo project to encapsulate all the best practices of working with Twitter streams so that others won’t have to solve all the same problems. I’m not going to talk about flamingo today, though (I’ll do that soon). I’m going to talk about testing.

Testing code that talks to the Twitter Streaming API is hard. The are 8 documented failure HTTP response codes, random disconnections, and second-to-second rate changes. And, since this is Twitter we’re talking about, basically anything can happen at any time – including complete downtime.

So, let me introduce you to mockingbird. Mockingbird is a simple streaming HTTP server written in Ruby that you can use in place of the server at stream.twitter.com when you’re testing your Streaming API code. With mockingbird, all it takes is a few lines of code and you can simulate all the issues I described above and more. Here’s an example config:

Mockingbird.setup(:port=>8080) do
  wait 1
  5.times do
    send '{"foo":"bar"}'
    wait { rand }
  end
  close
end

So what does this do?

1. Starts a mockingbird server on port 8080 on your localhost
2. When there’s a connection, it starts by waiting for 1 second before sending any data
3. Then, it sends ‘{“foo”:”bar”}’ 5 times with each time followed by randoom wait of between 0 and 1 seconds
4. After that, it closes the connection and waits for more

You can do lots of other things with this simple configuration style. See the README and code examples at github.

• Define per connection behaviors with on_connection so you can do multiple-connection tests for retry behavior
• Send lines from a file with a configurable wait between lines using pipe
• Send HTTP statuses like 403 or 500

Mockingbird provides a simple setup/teardown mechanism so it’s easy to start a server and shut it down during your unit tests, specs, or whatever. Here’s an example Test::Unit test:

class TestConnection < Test::Unit::TestCase
  def setup
    # Starts the server running a separate process
    Mockingbird.setup(:port=>8080) do
      status 500, "Error"
      close
    end
  end

  def test_handle_500_errors
    # Your test code here for connecting to localhost:8080
    # and handling a 500 error here
  end

  def teardown
    # Stops the server process
    Mockingbird.teardown
  end
end

We’ve already found mockingbird to be extremely useful for testing flamingo. If you’re using the Streaming API and need to be confident your code can withstand the crazy stuff Twitter might throw at it, I think you’ll probably find it helpful, too. Of course if you have questions, comments, ideas or (even better) pull requests, hit me up.