Planetaki powered by MongoDB
Over the last year now we’ve been working on developing a new version of Planetaki and finally the fruits of our labour are in production, and wow is it fast! Its an awesome feeling to know that your ideas actually work as intended.
We started working on Planetaki way back in September 2007. At that time, there weren’t many options for alternatives to the traditional databases, and if there were, I certainly didn’t know enough about them to use them on a new project. The obvious choice was to put everything in MySQL and hope for the best. As Planetaki grew it became clear that a traditional database wasn’t going to make the mark. Relational databases are optimised for handling lots of small rows of data in tables, Planetaki caches huge amounts of data from posts all over the internet resulting in very large rows. The result is that the traditional database caching and optimisation techniques just aren’t as effective, and Planetaki was getting progressively slower and harder to manage (you might have noticed if you are a frequent user!)
Welcome MongoDB. MongoDB is a document database, as opposed to a relational database. Data is stored literally as you send it, as opposed to being stored in tables. This removes the restriction of having to fit all you data into columns allowing much greater flexibility and the ability to store more complex structures. The cost of this is a larger database, but given the ever dropping cost of storage, this is not really a problem. Some people call MongoDB a NoSQL database, personally I think this is a crap name as it says nothing about what it is, only what it is not.
Planetaki now uses a hybrid solution for storing data. The complex feeds and their entries are stored in MongoDB, and user data is stored in a traditional Postgresql database. This allows us to split the system into two parts, on one side we have the feed “reaping”, and on the other the website which mainly accesses the tables of data. To ensure we can scale, a queueing system (RabbitMQ) is used so that when a new entry is created, each user’s planet data is updated as soon as the system can allocate CPU time to do so. Eventually this will allows us to add multiple servers and support incoming update requests (coming soon!). You can see me babbling about this on the Vostok blog post: Coming up: a new version of Planetaki
I think the results speak for themselves (fortunately!) and I encourage you to check out Planetaki if you are not already a user. In the upcoming weeks I hope to talk about a bit more of the more technical features, in particular the move to AJAX.
Comments