Monday, November 30, 2009

Update of RabbitMQ Replicator

The little Drizzle RabbitMQ replicator I built last week got quite popular (and even  users!) so I decided to work a bit more on it. These are the changes:

  • Moved it to a real project on launchpad (http://launchpad.net/rabbitreplication
  • Store transaction log position on master so that a transaction is never sent twice over the wire.
  • Major internal refactorings, start using google guice for IoC
  • Make the transport pluggable, it should now be easy to write your own transport and swap out rabbitmq, JMS anyone?
  • Alot more configurations for rabbitmq, see example config
User Guide:
  1. Download binaries here (or check out code: bzr branch lp:rabbitreplication)
  2. Unzip on master
  3. Edit master.properties to reflect your environment (hope the config vars are self-explanatory, let me know if there are any problems)
  4. Start the master: java -jar replication.jar master.properties
  5. Unzip on slave
  6. Edit slave.properties
  7. Start the slave: java -jar replication.jar slave.properties
  8. Watch changes replicate, report issues to me (krummas@gmail.com)
Todo/Ideas:
I'm thinking about using this code to build a framework for replicating changes from Drizzle into other storage forms, for example Hadoop/hbase or Cassandra. My thinking is that it could be useful for moving data that is very hot into a faster storage without changing your application too much. This might need business logic implemented on the "slave" (saying for example that this column should be stored and this should be ignored), and that is what I'm thinking could be built quite nicely using a framework (a DSL could be handy). 

Please let me know if this has been done somewhere or if it is a stupid idea!

1 comment:

Brian Aker said...

Hi!

I've got several requests for replication to Memcached. I suspect it would end up being quite popular. Tyrant maybe? I could see Toru trying to tackle that.

Cheers,
-Brian