Move Documents From MongoDB to Couchbase

Maybe you remember my post about moving data from CouchDB to Couchbase. If you are not using CouchDB but MongoDB you might have felt let down. I am sorry. Let's fix this. Here's how to move your data out of MongoDB to Couchbase.

One of the great things about using RxJava is that this API is becoming more widespread. We made that choice a while ago and are happy to see more people join that wagon, like Mongo did recently.

The fact that we share a stream API is going to make things really easy. My goal is to get an Observable of Mongo Document, Map it to Couchbase Documents and then write them to Couchbase. As you can see this process is really easy to model with RxJava. In pseudo code it would look like:

getMongoCollectionStream().fromMongoDocumentToCouchbaseDocuments().writeCBdocToCouchbase()

Again this fits very well with RxJava.

MongoDB allows you to open a collection and return it as an Observable of Document. This is exactly what we need and can happen in a couple of lines:

    MongoClient client = MongoClients.create(connectionString);
    MongoDatabase db = client.getDatabase(dbName);
    Observable<org.bson.Document> mongoDocs = db.getCollection(collectionName).find();

Then we need to transform documents coming in into Couchbase documents. We can easily use the Rx map operator to do so. When writing a document on couchbase you need a key or an id. In a Mongo document it is stored in the _id field using the ObjectID abstraction. It's generated by Mongo based on several criteria. Once we get that id all we have to do is get the Doc as a JSON string and create a RawJsonDocument based on that and the id.

 db.getCollection(collectionName).find().toObservable()
                .map(new Func1<org.bson.Document, Document>() {
                    public Document call(org.bson.Document mongoDoc) {
                        mongoDoc.put(typeField, type);
                        RawJsonDocument d = RawJsonDocument.create(mongoDoc
                                .getObjectId("_id").toHexString(), mongoDoc
                                .toJson());
                        return d;
                    };
                });

After this map.operation we are left with an observable of couchbase document. The last step is to write it to Couchbase.

db.getCollection(collectionName).find().toObservable()
                .map(new Func1<org.bson.Document, Document>() {
                    public Document call(org.bson.Document mongoDoc) {
                        mongoDoc.put(typeField, type);
                        RawJsonDocument d = RawJsonDocument.create(mongoDoc
                                .getObjectId("_id").toHexString(), mongoDoc
                                .toJson());
                        return d;
                    };
                }).map(asyncBucket.upsert(doc));;

I have included all this code and made it configurable in the couchbase-java-importer. It's available on Github here.

If you don't need to do anything other than importing the JSON documents from a collection straight into Couchbase, this should be sufficient. Just download the binary and execute it making sure the yml configuration file is in your classpath and contains the right info.

This is one particular way to move data from MongoDB to Couchbase. There are others. Both databases support CSV import for instance and Mongo has a CSV exporter. So, once you have a CSV file you can use cbtransfer or couchbase-java-importer to get its content to Couchbase.

All these examples are assuming you don't want to do any transformation to the data before importing it. Transformation steps would be really easy to plug in the RxJava importer. The code is very simple so please feel free to fork it or send pull requests! Adding more importers to that project can be a great way to enter the CB community by the way.