DZone
IoT Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > IoT Zone > Amateur Radio Homebrew (Part 3): Preventing Duplicate Tweets

Amateur Radio Homebrew (Part 3): Preventing Duplicate Tweets

As we continue to work on our radio/social media project, let's focus on what it takes to avoid sending duplicate tweets over to Twitter.

Kevin Hooke user avatar by
Kevin Hooke
·
May. 17, 17 · IoT Zone · Tutorial
Like (2)
Save
Tweet
3.22K Views

Join the DZone community and get the full member experience.

Join For Free

In my last post (part 2, also see part 1 here), I talked about Twitter’s API rate limits. Since many Packet Radio transmissions are duplicates by their nature, for example, beacon packets and ID packets, it’s important to have some kind of mechanism to prevent sending these through to Twitter.

The approach I used was to insert each received packet into a MongoDB database, storing the received packet data, who the packet was from and who to, and additional metadata about the packet, for example, when last sent, and when it was last received.

Here’s an example of what each document stored looks like:

{
    "_id" : ObjectId("5909828e5a2f130fc8039882"),
    "firstHeard" : ISODate("2017-05-03T07:11:10.051Z"),
    "from" : "AE6OR ",
    "heardTimes" : 1,
    "infoString" : "¤¤@à¤@@त@à²@`¨@`¨@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r",
    "lastHeard" : ISODate("2017-05-03T07:11:10.051Z"),
    "lastTweet" : ISODate("2017-05-03T07:11:10.051Z"),
    "to" : "BEACON",
    "tweet" : "Received station: AE6OR  sending to: BEACON : ¤¤@à¤@@त@à²@`¨@`¨@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r"
}


My current logic to check for duplicates and record when a tweet is last sent is:

  1. Search for a matching document (tweet details) with a $lt condition that lastTweet is before ‘now – 12 hours’: 
    document.lastTweet = {'$lt' : moment().utc().subtract(12, 'hours').toDate() };
  2. This is executed as a findOne() query: 
    db.collection('tweets').findOne(document).then(function (olderResult) {
       ...
    }
  3. If an existing document older than 12 hours is found, then update properties to record that this same packet was seen now, and the time it was resent was also now (we’ll resend it to the Twitter API after the DB updates): 
    if (olderResult != null) {
        sendTweet = true;
        updateProperties = {
            lastHeard: now,
            lastTweet: now
        };
    }
    else {
        updateProperties = {
            lastHeard: now
        };
    }

    If not older than 12 hours, set properties to be updated to indicate just the lastHeard property
  4. To either update the existing document or insert a new document if this was the first time this packet was heard, do an ‘upsert’: 
    db.collection('tweets').update(
        document,
        {
            $set: updateProperties,
            $inc: {heardTimes: 1},
            $setOnInsert: {
                firstHeard: now,
                lastTweet: now
            }
        },
        {upsert: true}
    ).then(function (response) {
        ...
    }
  5. Depending on the result indicating if we inserted or updated, set a property so that on return, we know whether to send a new Tweet or not: 
    if(response.upserted != null || sendTweet) {
        response.sendTweet = true;
    }
    else{
        response.sendTweet = false;
    }

The approach is not yet completely foolproof, but it is stopping the majority of duplicate Tweets sent to Twitter so far.

For the full source check the project on GitHub here. 

Homebrew (video games) Database Document twitter Property (programming) API Data (computing) GitHub Metadata IT

Published at DZone with permission of Kevin Hooke, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • MACH Architecture Explained
  • ETL/ELT on Kubernetes With Airbyte
  • Vaadin Apps as Native Executables Using Quarkus Native
  • Blocking Ads on Your Network Using Raspberry Pi 3 + Fedora + Pi-hole

Comments

IoT Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo