Migrating Data in Couchbase From One Bucket to Another Using XDCR

Couchbase provides XDCR, which makes your life simpler when it comes to data migration.

Following are my assumptions for this exercise.

1. There is a bucket Metrics that holds data like user events info, push notifications sent from a system, the actual user data, etc.

2. You are running the latest version of Couchbase (current 4.6.3).

3. You have admin access to execute queries.

4. Events document has uuid as a key, i.e. c0f55f62-b990-47bc-8caa-f42313669948.

   {
     "type": "EventType",
     "eventName": "Event name",
     "uuid": "000008F6-DF77-4C9F-B5A9-E94065639A07",
     "timestamp": 1426206242415,
     "localTimestamp": 1426188242415,
     "couchbaseCreationTimestamp": 1426206242536
   }

5. Push notifications documents have uuid key as PUSH_N_XXXXXXX.

   {
     "couchbaseCreationTimestamp": 1457476557919,
     "eventName": "SentPushnotificationEvent",
     "type": "EventtYPE",
     "uuid": "PUSH_N_893579367597",
     "timestamp": 1457476557919
   }

Problem: We need to separate out events and push notifications entries from the Metrics bucket and put them in a new bucket called Events.

Solution steps:

1. Using the Couchbase UI, create a new bucket named Events.

2. Set views if required.

3. Create a Cluster Reference in XDCR. This is required for server reference.

   

4. For events, create a replication filter using the following regex:

   [a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}

5. 

   Once the above replication is complete, delete this replication and create another replication for push notifications.

6. For push notifications, create a replication filter using the following regex:

   PUSH_N_*

   

7. Once the replication is over, delete this filter, as well.

8. Now, we will work on deleting the existing documents in the Metrics bucket.

9. Deletion requires indexes. If the primary index does not exist, create one.

   CREATE PRIMARY INDEX `metrics-primary-index` ON `METRICS` USING GSI;

   

10. Check if the index exists:

    SELECT * FROM system:indexes WHERE name="metrics-primary-index";

11. Delete all events documents from the Metrics bucket:

    DELETE FROM METRICS m WHERE REGEX_LIKE(uuid, "[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}")

12. Delete push notifications:

    DELETE FROM METRICS m WHERE REGEX_LIKE(uuid, "PUSH_N_+.*");

Migration is completed!