Using Jolt in Big Data Streams to Remove Nulls

In this article, we're going to learn about using Jolt code in your big data streams to remove null values.

Here's some JSON code for using Jolt in big data streams:

[
  {
    "operation": "default",
    "spec": {
      "address": "",
      "somesensorvalues[]": {
        "*": {
          "sensor1": false
        }
      },
      "startTime": "",
      "onStartTime": "",
      "markId": "",
      "markName": "",
      "stoppedTime": "",
      "startTime2": "",
      "powerSetting": "false",
      "speed": 0,
      "id": 0,
      "city": "",
      "state": ""
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": "&"
    }
  }
]

To help you understand, here's some example source data:

{
  "address" : "2000 Electric Avenue",
  "somesensordata" : [ {
    "sensor1" : null
  } ],
  "city" : "hightstown",
  "deviceId" : 5454545,
  "dateTime" : "2017-08-07 14:56:09",
  "id" : 6831491,
  "idle" : false,
  "startTime" : null,
  "onStartTime" : null,
  "markId" : null,
  "markName" : null,
  "zipCode" : "08520"
}

Yeah, sometimes you really don't want to see any nulls!

The above Jolt script will copy all the values in a source JSON document to a destination. For the explicitly named variables like speed, it will replace nulls with the value on the right. So for speed, it will put in a 0 to represent a null value.

For embedded lists of values (likesensor1  inside of somesensorvalues array), the syntax is a bit different.

References

• Jolt Transform Demo Using v0.1.0

• Jolt GitHub page