Amateur Radio Homebrew (Part 3): Preventing Duplicate Tweets

In my last post (part 2, also see part 1 here), I talked about Twitter’s API rate limits. Since many Packet Radio transmissions are duplicates by their nature, for example, beacon packets and ID packets, it’s important to have some kind of mechanism to prevent sending these through to Twitter.

The approach I used was to insert each received packet into a MongoDB database, storing the received packet data, who the packet was from and who to, and additional metadata about the packet, for example, when last sent, and when it was last received.

Here’s an example of what each document stored looks like:

{
    "_id" : ObjectId("5909828e5a2f130fc8039882"),
    "firstHeard" : ISODate("2017-05-03T07:11:10.051Z"),
    "from" : "AE6OR ",
    "heardTimes" : 1,
    "infoString" : "¤¤@à¤@@त@à²@`¨@`¨@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r",
    "lastHeard" : ISODate("2017-05-03T07:11:10.051Z"),
    "lastTweet" : ISODate("2017-05-03T07:11:10.051Z"),
    "to" : "BEACON",
    "tweet" : "Received station: AE6OR  sending to: BEACON : ¤¤@à¤@@त@à²@`¨@`¨@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r"
}

My current logic to check for duplicates and record when a tweet is last sent is:

1. Search for a matching document (tweet details) with a $lt condition that lastTweet is before ‘now – 12 hours’: 

   document.lastTweet = {'$lt' : moment().utc().subtract(12, 'hours').toDate() };

2. This is executed as a findOne() query: 

   db.collection('tweets').findOne(document).then(function (olderResult) {
      ...
   }

3. If an existing document older than 12 hours is found, then update properties to record that this same packet was seen now, and the time it was resent was also now (we’ll resend it to the Twitter API after the DB updates): 

   if (olderResult != null) {
       sendTweet = true;
       updateProperties = {
           lastHeard: now,
           lastTweet: now
       };
   }
   else {
       updateProperties = {
           lastHeard: now
       };
   }

   
   If not older than 12 hours, set properties to be updated to indicate just the lastHeard property
4. To either update the existing document or insert a new document if this was the first time this packet was heard, do an ‘upsert’: 

   db.collection('tweets').update(
       document,
       {
           $set: updateProperties,
           $inc: {heardTimes: 1},
           $setOnInsert: {
               firstHeard: now,
               lastTweet: now
           }
       },
       {upsert: true}
   ).then(function (response) {
       ...
   }

5. Depending on the result indicating if we inserted or updated, set a property so that on return, we know whether to send a new Tweet or not: 

   if(response.upserted != null || sendTweet) {
       response.sendTweet = true;
   }
   else{
       response.sendTweet = false;
   }

The approach is not yet completely foolproof, but it is stopping the majority of duplicate Tweets sent to Twitter so far.

For the full source check the project on GitHub here.