DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Building AMQP-Based Messaging Framework on MongoDB
  • Spring Boot and React in Harmony
  • Stateless vs. Stateful Widgets: Make the Right Choice for Your Flutter App
  • Adopting Agile Practices for Workforce Management: Benefits, Challenges, and Practices

Trending

  • Auditing Spring Boot Using JPA, Hibernate, and Spring Data JPA
  • An Introduction to Build Servers and Continuous Integration
  • How to Migrate Vector Data from PostgreSQL to MyScale
  • Top 8 Conferences Developers Can Still Attend
  1. DZone
  2. Data Engineering
  3. Databases
  4. Finding Duplicate Keys with MongoDB’s Aggregation Framework

Finding Duplicate Keys with MongoDB’s Aggregation Framework

Chris Chang user avatar by
Chris Chang
·
Mar. 24, 14 · Interview
Like (0)
Save
Tweet
Share
26.42K Views

Join the DZone community and get the full member experience.

Join For Free

Quite frequently our users want to create a unique index on a data set but encounter some form of the following error because of duplicate key value(s):

E11000 duplicate key error index: db.collection.$field_1_field2_1  dup key: { : 1.0 : 1.0 }

While MongoDB supports an option to drop duplicates, dropDups, during index builds, this option forces the creation of a unique index by way of deleting data. If you use the dropDups option, MongoDB will create an index on the first occurrence of a value for a given key and then  delete all subsequent values. While this behavior may be acceptable in some cases, it’s important to be cautious whenever you are deleting data.

Example aggregation query to identify duplicate keys

To avoid blindly deleting data, one of our engineers, Sean, suggests using the aggregation pipeline framework to easily identify documents with duplicate key values.

// Desired unique index: 
// db.collection.ensureIndex({ firstField: 1, secondField: 1 }, { unique: true})

db.collection.aggregate([
  { $group: { 
    _id: { firstField: "$firstField", secondField: "$secondField" }, 
    uniqueIds: { $addToSet: "$_id" },
    count: { $sum: 1 } 
  }}, 
  { $match: { 
    count: { $gt: 1 } 
  }}
])

In the first stage of this example pipeline, we use the $group operator to aggregate documents by the desired index key values and record (in the uniqueIds field) each _id value of the grouped documents. We also count the number of grouped documents.

In the second stage of this example pipeline, we use the $match operator to filter out all documents with a count of 1. The filtered-out documents represent unique index keys.

The remaining documents identify documents in the collection that contain duplicate keys and would prevent the creation of a unique index on { firstField: 1, lastField: 1 }.

Note that the order of operations in the aggregation pipeline matters (i.e., we cannot $match on count before our $group operation creates the count field).

Only one of many aggregation framework uses

MongoDB’s aggregation framework is very flexible and powerful, which allows for many different usage possibilities. This example is only one of many, many use cases – we hope you’re inspired to try it out!


Framework MongoDB

Published at DZone with permission of Chris Chang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Building AMQP-Based Messaging Framework on MongoDB
  • Spring Boot and React in Harmony
  • Stateless vs. Stateful Widgets: Make the Right Choice for Your Flutter App
  • Adopting Agile Practices for Workforce Management: Benefits, Challenges, and Practices

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: