DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. Migrating Data in Couchbase From One Bucket to Another Using XDCR

Migrating Data in Couchbase From One Bucket to Another Using XDCR

In this tutorial, learn about XDCR, a feature provided by Couchbase, can make your life simpler when it comes to data migration.

Harsh Shrivastava user avatar by
Harsh Shrivastava
·
Oct. 18, 17 · Tutorial
Like (1)
Save
Tweet
Share
3.10K Views

Join the DZone community and get the full member experience.

Join For Free

Couchbase provides XDCR, which makes your life simpler when it comes to data migration.

Following are my assumptions for this exercise.

  1. There is a bucket Metrics that holds data like user events info, push notifications sent from a system, the actual user data, etc.

  2. You are running the latest version of Couchbase (current 4.6.3).

  3. You have admin access to execute queries.

  4. Events document has uuid as a key, i.e. c0f55f62-b990-47bc-8caa-f42313669948.

    {
      "type": "EventType",
      "eventName": "Event name",
      "uuid": "000008F6-DF77-4C9F-B5A9-E94065639A07",
      "timestamp": 1426206242415,
      "localTimestamp": 1426188242415,
      "couchbaseCreationTimestamp": 1426206242536
    }
  5. Push notifications documents have uuid key as PUSH_N_XXXXXXX.

    {
      "couchbaseCreationTimestamp": 1457476557919,
      "eventName": "SentPushnotificationEvent",
      "type": "EventtYPE",
      "uuid": "PUSH_N_893579367597",
      "timestamp": 1457476557919
    }

Problem: We need to separate out events and push notifications entries from the Metrics bucket and put them in a new bucket called Events.

Solution steps:

  1. Using the Couchbase UI, create a new bucket named Events.

  2. Set views if required.

  3. Create a Cluster Reference in XDCR. This is required for server reference.

    Image title

  4. For events, create a replication filter using the following regex:

    [a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}
  5. Image title

    Once the above replication is complete, delete this replication and create another replication for push notifications.
  6. For push notifications, create a replication filter using the following regex:

    PUSH_N_*

    Image title

  7. Once the replication is over, delete this filter, as well.

  8. Now, we will work on deleting the existing documents in the Metrics bucket.

  9. Deletion requires indexes. If the primary index does not exist, create one.

    CREATE PRIMARY INDEX `metrics-primary-index` ON `METRICS` USING GSI;

    Image title

  10. Check if the index exists:
    SELECT * FROM system:indexes WHERE name="metrics-primary-index";
  11. Delete all events documents from the Metrics bucket:

    DELETE FROM METRICS m WHERE REGEX_LIKE(uuid, "[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}")
  12. Delete push notifications:

    DELETE FROM METRICS m WHERE REGEX_LIKE(uuid, "PUSH_N_+.*");

Migration is completed!

Data (computing)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Automated Performance Testing With ArgoCD and Iter8
  • Writing a Modern HTTP(S) Tunnel in Rust
  • How To Generate Code Coverage Report Using JaCoCo-Maven Plugin
  • The 31 Flavors of Data Lineage and Why Vanilla Doesn’t Cut It

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: