DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Data Migration from AWS DocumentDB to Atlas on AWS

Data Migration from AWS DocumentDB to Atlas on AWS

AWS Database Migration Service provides heterogeneous migrations between different platforms. See the migration scenario between DocumentDB and MongoDB Atlas on AWS.

Travis Van user avatar by
Travis Van
·
Sep. 22, 22 · Tutorial
Like (2)
Save
Tweet
Share
4.55K Views

Join the DZone community and get the full member experience.

Join For Free

AWS Database Migration Service (DMS) provides heterogeneous migrations between different database platforms. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. Currently, DMS can use AWS DocumentDB as a database source or target; however, DMS only supports MongoDB as a source endpoint for data replication. 

Following are the steps to replicate data from DocumentDB to Atlas using the DMS. Please be aware that the twisted configuration may not be supported by AWS.

Here is the architecture diagram:

Architecture diagram to replicate data from DocumentDB to Atlas using the DMS

Notes

  1. Security Group: All DocumentDB cluster, EC2, and DMS replication instances share the same security group for simplicity. The inbound rule required is as follows:

    Protocol port source

    TCP

    27017

    <security group name>

    TCP

    22

    <your local IP address for ssh access>

    The outbound rule is the default one:
    Protocol Port Source

    ALL

    ALL

    0.0.0.0/0


  1. According to Amazon Premium Support: "By default, TLS configuration is enabled for Amazon DocumentDB clusters. To disable the TLS configuration, create a new custom Amazon DocumentDB cluster parameter group. Set the tls parameter to disabled, and then modify the cluster to use the new cluster parameter group. For more information, see Managing Amazon DocumentDB cluster parameter groups."

  1. As the AWS DMS service is unable to resolve SRV records, you need to provide the FQDN of the Atlas primary node as the server name for the target endpoint configuration. However, the replication task will fail if the primary member of the replica set in Atlas has been re-elected. You need to stop the task, reconfigure the target endpoint with a new primary host name and then resume the task again. You can’t modify the target endpoint settings while the task is running.    

Alternatively, you can create a single-sharded cluster and use the hostname and port# of one of the Mongo processes for the target endpoint configuration.

  1. DMS supports migrating multiple databases in a single task. As stated in the AWS Database Migration Service User Guide:

    • "In the AWS Management Console for AWS DMS, leave Database name empty under Endpoint configuration on the Create endpoint page."

    • "For each database that you want to migrate from this Amazon DocumentDB source endpoint, specify the name of each database as the name of a schema in the table-mapping for the task using either the guided input in the console or directly in JSON." For example:

 
    {

    "rules": [

        {

            "rule-type": "selection",

            "rule-id": "1",

            "rule-name": "1",

            "object-locator": {

                "schema-name": "Customers",

                "table-name": "%"

            },

            "object-locator": {

                "schema-name": "Orders",

                "table-name": "%"

            },

            "object-locator": {

                "schema-name": "Inventory",

                "table-name": "%"

            },

            "rule-action": "include"

        }

    ]

}


Steps

1. Create a Document DB Cluster

  1. Create a DocumentDB cluster: 
    1. Select the DocumentDB service.
    2. Click the Clusters from the left panel.
    3. Click on the Create button.
    4. Configure the following settings:
  2. Configuration:
    1. Cluster identifier: <name of the DocumentDB cluster>
    2. Engine Version: 4.0.0
    3. Instance class: db.t3.medium (free trial eligible)
    4. Number of instances: 1
  3. Authentication:
    1. Master username: <name of the master user>
    2. Master password/Confirm master password: <password of the master user>
  4. Enable “Show advanced settings.“
  5. Network settings: 
    1. Virtual Private Cloud: <use default or any specific VPC>
    2. Subnet group: <use default or any specific subnet group>
    3. VPC security groups: <use the one specified on Note A>
  6. Cluster options:
    1. Port: 27017
    2. Cluster parameter group: default.docdb4.0. You can take the default values for other settings.
  7. Click the Create cluster button. Reference Note B if you want to disable TLS on the DocumentDB cluster.

2. Create an EC2 Instance to Load the Sample Data

DocumentDB can be accessed directly by Amazon EC2 instances or other AWS services that are deployed in the same Amazon VPC. However, you can only use SSH tunneling to access your Amazon DocumentDB resources if you want to access the resource from outside the cluster's VPC.

An EC2 instance was created with the same network settings (VPC, subnet group, and security group) as the DocumentDB cluster. Once the EC2 instance is ready, install the mongo shell and load some sample data into the DocumentDB cluster. You can reference steps 4 to 6 on the Get Started Guide.

Run the following command to enable change streams on the source database.

 
db.adminCommand({modifyChangeStreams: 1,

    database: "<source database name>",

    collection: "", 

    enable: true});


3. Create a Replication Subnet Group

  1. Select the Data Migration Service service.
  2. Click the Subnet groups from the left panel.
  3. Click on the Create subnet group button.
  4. Provide the VPC and subnet(s) you want the replication instance created on.
  5. Click on the Create subnet group button.

4. Create a DMS Replication Instance

  1. On the same service, Click the Replication instances from the left panel.
  2. Click on the Create replication instance button.
  3. Configure the following settings.
    1. Replication instance configuration: 
  • Name: <name of replication instance>
  • Descriptive Amazon Resource Name (ARN): Leave it blank
  • Description: <description of instance>
  • Instance class: dms.t3.small
  • Engine version: 3.4.6
  • Allocated storage (GiB): 50
  • VPC: <use default or any specific VPC>
  • Multi AZ: Single-AZ
  • Publicly accessible: Tick on the box (add this IP address to the whitelist on Atlas)

b. Advanced security and network configuration:

  • Replication subnet group: <subnet group created on step 3>
  • VPC security group(s): <use the one specified on Note A>

You can take the default values for other settings.

4. Click the Create button.

5. Import DocumentDB Certificate (Optional)

If the DocumentDB cluster is TLS enabled, perform the steps from AWS documentation so that DMS can connect to Amazon DocumentDB using TLS.

6. Create a DocumentDB Source Endpoint

  1. On the same service, Click the Endpoints from the left panel.
  2. Click on the Create endpoint button.
  3. Configure the following settings:
    1. Endpoint type: Source endpoint
    2. Endpoint configuration:
      • Endpoint identifier: < name of the endpoint>
      • Descriptive Amazon Resource Name (ARN): Leave it blank
      • Source engine: Amazon DocumentDB
      • Access to endpoint database: Provide access information manually
      • Server name: < server name of DocumentDB cluster>
        e.g.mdb-ddb-db.cluster-c50ma7ye6nnl.ap-southeast-2.docdb.amazonaws.com
      • Port: 27017
      • Secure Socket Layer (SSL) mode: verify-full (none if tls has been disabled)
      • User name: <name of the master user>
      • Password: <password of the master user>
      • Authentication source: admin
      • Database name: <replicated database name>
      • Metadata mode: document
      • _id as a separate column: Leave it as un-checked
  4. Expand Test endpoint connection (option):
    1. VPC: <select the VPC the replication instance created on>
    2. Replication instance: <select the one created on Step 4>
  5. Click the Run test button.
  6. You may need to troubleshoot the issue if the test failed.
  7. Click Create endpoint button.

7. Import Atlas Certificate

  1. Download and save the Atlas root CA certificate on your local drive.
  2. On the same service, Click the Certificates from the left panel.
  3. Click on the Import Certificate button.
  4. Configure the following settings:
    1. Certificate configuration:
      • Certificate identifier: ISRG-Root-X1
      • Import certificate file: <choose the file saved in step a>
  5. Click the Import certificate button.

8. Update the IP Access List

Update the IP access list of the Atlas project with the public IP address of the DMS replication instance.

9. Create an Altas Target Endpoint

  1. On the same service, Click the Endpoints from the left panel.
  2. Click on the Create endpoint button.
  3. Configure the following settings:
    1. Endpoint type: Target endpoint
    2. Endpoint configuration:
      • Endpoint identifier: < name of the endpoint>
      • Descriptive Amazon Resource Name (ARN): Leave it blank.
      • Target engine: Amazon DocumentDB
      • Access to endpoint database: Provide access information manually.
      • Server name: <primary server name of Atlas replication set>; e.g., cluster1-shard-00-02.2psdk.mongodb.net. The replication task will fail if the primary member has been re-elected (see Note C).
      • Port: 27017
      • Secure Socket Layer (SSL) mode: verify-full
      • CA certificate: ISRG-Root-X1
      • User name: <name of the master user>
      • Password: <password of the master user>
      • Database name: <target database name>
  4. Expand Test endpoint connection (option).
    1. VPC: <use the VPC the replication created on>
    2. Replication instance: <select the one created on Step 4>
  5. Click the Run test button. You may need to troubleshoot the issue if the test failed.
  6. Click Create endpoint button.

10. Create a Full Load + CDC Replication Task

  1. On the same service, click the Database migration tasks from the left panel.
  2. Click on the Create task button.
  3. Configure the following settings.
    1. Task configuration:
      • Task identifier: <name of the replication task>
      • Descriptive Amazon Resource Name (ARN): Leave it blank
      • Replication instance: <select the one created on step 4>
      • Source database endpoint: <select the endpoint created on step 6>
      • Target database endpoint: <select the endpoint created on step 9>
      • Migration type: Migrate existing data and replicate ongoing changes
    2. Task settings:
      • Editing mode: Wizard
      • Target table preparation mode: Drop tables on target
      • Stop task after the full load completes: Don't stop
      • Include LOB columns in replication: Limited LOB mode
      • Maximum LOB size (KB): 32
      • Enable validation: Leave it as un-check
      • Enable CloudWatch logs: Leave it as un-check
    3. Table mappings:
      • Editing mode: Wizard
      • Click on Add new selection rule
      • Schema: Enter a schema
      • Source name: <replicated database name from source>
      • Table name: <replicated collection name from source>
      • Action: Include
      • You can get more information about DMS selection rules from the official documentation.
    4. Premigration assessment: 
      • Enable premigration assessment run: Leave it as un-checked.
    5. Migration task startup configuration:
      • Start migration task: Automatically on create
  4. Click on the Create task button.

11. Monitor the Migration Task

Once the task has started, you can monitor the progress of Full Load and CDC replication on the individual collection with the Table Statistics tab. 

Table Statistics tab


AWS Data migration Database Requirements engineering TLS Virtual private cloud cluster Data (computing) Replication (computing) Task (computing)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Understanding and Solving the AWS Lambda Cold Start Problem
  • Best CI/CD Tools for DevOps: A Review of the Top 10
  • Fixing Bottlenecks in Your Microservices App Flows
  • Public Key and Private Key Pairs: Know the Technical Difference

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: