DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Your Agents Think Data Federation Means One Truth. It Means a Collage.
  • Ingesting Public Data That Fails Silently: Patterns From a TypeScript and Cloudflare Workers Pipeline
  • Engineering Data Observability: Detecting Schema Drift, Data Drift, and Silent Pipeline Failures
  • Designing a Polling Pipeline for Rate-Limited APIs Without Losing Data

Trending

  • MCP Is Coming to Your Desktop Applications
  • Searching Kafka Without an Index: Lessons From Building Message Search for Kafdrop
  • Your Event Storming Is Not Your Software Design
  • Catching Query Regressions Before Your Users Do

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. Data
  4. Migrating Data in Couchbase From One Bucket to Another Using XDCR

Migrating Data in Couchbase From One Bucket to Another Using XDCR

In this tutorial, learn about XDCR, a feature provided by Couchbase, can make your life simpler when it comes to data migration.

By 
Harsh Shrivastava
·
Oct. 18, 17 · Tutorial
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
4.0K Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

Couchbase provides XDCR, which makes your life simpler when it comes to data migration.

Following are my assumptions for this exercise.

  1. There is a bucket Metrics that holds data like user events info, push notifications sent from a system, the actual user data, etc.

  2. You are running the latest version of Couchbase (current 4.6.3).

  3. You have admin access to execute queries.

  4. Events document has uuid as a key, i.e. c0f55f62-b990-47bc-8caa-f42313669948.

    {
      "type": "EventType",
      "eventName": "Event name",
      "uuid": "000008F6-DF77-4C9F-B5A9-E94065639A07",
      "timestamp": 1426206242415,
      "localTimestamp": 1426188242415,
      "couchbaseCreationTimestamp": 1426206242536
    }
  5. Push notifications documents have uuid key as PUSH_N_XXXXXXX.

    {
      "couchbaseCreationTimestamp": 1457476557919,
      "eventName": "SentPushnotificationEvent",
      "type": "EventtYPE",
      "uuid": "PUSH_N_893579367597",
      "timestamp": 1457476557919
    }

Problem: We need to separate out events and push notifications entries from the Metrics bucket and put them in a new bucket called Events.

Solution steps:

  1. Using the Couchbase UI, create a new bucket named Events.

  2. Set views if required.

  3. Create a Cluster Reference in XDCR. This is required for server reference.

    Image title

  4. For events, create a replication filter using the following regex:

    [a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}
  5. Image title

    Once the above replication is complete, delete this replication and create another replication for push notifications.
  6. For push notifications, create a replication filter using the following regex:

    PUSH_N_*

    Image title

  7. Once the replication is over, delete this filter, as well.

  8. Now, we will work on deleting the existing documents in the Metrics bucket.

  9. Deletion requires indexes. If the primary index does not exist, create one.

    CREATE PRIMARY INDEX `metrics-primary-index` ON `METRICS` USING GSI;

    Image title

  10. Check if the index exists:
    SELECT * FROM system:indexes WHERE name="metrics-primary-index";
  11. Delete all events documents from the Metrics bucket:

    DELETE FROM METRICS m WHERE REGEX_LIKE(uuid, "[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}")
  12. Delete push notifications:

    DELETE FROM METRICS m WHERE REGEX_LIKE(uuid, "PUSH_N_+.*");

Migration is completed!

Data (computing)

Opinions expressed by DZone contributors are their own.

Related

  • Your Agents Think Data Federation Means One Truth. It Means a Collage.
  • Ingesting Public Data That Fails Silently: Patterns From a TypeScript and Cloudflare Workers Pipeline
  • Engineering Data Observability: Detecting Schema Drift, Data Drift, and Silent Pipeline Failures
  • Designing a Polling Pipeline for Rate-Limited APIs Without Losing Data

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook