Trying MongoDB for Cartoque
I recently open-sourced Cartoque, the CMDB tool we use at $work. I want to make it suitable for any medium-sized datacenter in the next few months, but with a bit more than 1 year of specific development for my platform, a significant part of the app isn’t that generic and won’t fit other people needs. Moreover, most of this specific code is poorly tested or not tested at all, because I knew from the beginning that I didn’t want to stick with it. I have to improve that.
The problems with SQL and ActiveRecord
Let’s say it from the beginning, ActiveRecord and AREL are fan-tas-tic tools. I really love them. But it turns out they didn’t fit my needs really well this time.
1- I need first-class citizen custom fields : in Cartoque, users should be able to define structured custom fields (with type,
constraints, optionally defining relations to other “tables”) ; these fields should be accessible with the API the same way normal
fields are (not with cf.{custom_field_47} (yes I’m looking at you Request Tracker). Redmine has a pretty robust solution for this, but the code
is far too complex for my little brain, and it doesn’t provide arbitrary relations to other tables.
2- I need first-class citizen custom “tables” : you may have 2 SaaS products in your company. They each have different properties, hence different custom fields. I need a mechanism for storing and retrieving different attributes for different object types. But maybe you have 3 products, and 2 of them have some common attributes. Same goes for some DCI patterns I’d like to implement (for instance switches, routers, servers and virtual machines all have a processor). If you have some experience with SQL databases, you already imagine the nightmare it would be to implement something in generic tables. Basically it consists in putting denormalized, unstructured data in a structured engine, just for the sake of intellectual pleasure. Not a good idea.
3- I don’t want to deal with secondary joins, callbacks to update a reference table, etc. : today in Cartoque I maintain a secondary
model called ConfigurationItem. Each CI has a configuration item, and common relations (such as contact relations) are linked to this model. It
means I need to update it with a complicated callback chain, and I add some extra joins everywhere as soon as I want to retrieve those informations. It will
become very complicated when I have to expose a dozen CIs in a single page. This is dirty and painful.
4- I need double polymorphic associations, with constraints defined by the user : in a CMDB, nearly every CI could be linked with every other CI. I didn’t find any good solution to provide this kind of link easily, and if possible let the user parameterize it.
5 – SQL databases don’t know how to store complex objects : as I’m writing this, SQL databases don’t support complex column types like JSON. They don’t know how to store Hash, nested Hashes, or Arrays. As soon as you have complex objects to store, modelling them in relational databases becomes a problem, and leads to unnecesay pain.
Enters MongoDB
MongoDB has all this and more.
Kristóf Kovács wrote a fair comparison of so-called “NoSQL” tools. Cartoque is typically the kind of use-case he quotes as an example for MongoDB : “For most things that you would do with MySQL or PostgreSQL, but having predefined columns really holds you back.” I also tried CouchDB (very complicated…), and Riak (which I really love but doesn’t fit this particular use case).
John Nunemaker also wrote an excellent article about ‘Why he thinks Mongo is to databases what Rails was to frameworks’. It’s two years old, a bit old, but still relevant, and I strongly encourage you to check it out. Back in these days MongoDB had some durability problems, and a very tiny ecosystem. But it has resolved its problems and it has gained some popularity in the Rails community. Today there are tons of great gems for the basic functionalities I need in the application (for instance friendly ids, pagination, trees, etc.).
So here I am, now : I’m trying MongoDB with the Mongoid ORM for 3 weeks now, and I’m very happy with it. I might write other posts to cover more technical aspects later, but the best aspect so far is that it forced me to re-think the way I see my data in this application, it put me a bit out of my comfort zone (e.g. SQL databases + ActiveRecord), and it opened a whole new world of great ideas for the future of Cartoque.
Conclusion
Choose the right tool for the job. That’s all ;)