DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Google Cloud Document AI Basics
  • Patch Management in the Age of IoT: Challenges and Solutions
  • IoT Communication Protocols for Efficient Device Integration
  • Simplify Your Compliance With Google Cloud Assured Workloads

Trending

  • The Human Side of Logs: What Unstructured Data Is Trying to Tell You
  • Hybrid Cloud vs Multi-Cloud: Choosing the Right Strategy for AI Scalability and Security
  • The Cypress Edge: Next-Level Testing Strategies for React Developers
  • How to Practice TDD With Kotlin
  1. DZone
  2. Data Engineering
  3. Big Data
  4. How to Use Google Core IoT With InfluxData

How to Use Google Core IoT With InfluxData

Want to learn more about how to use Google Core IoT with InfluxData? Check out this tutorial to learn more with the new Telegraf plugin!

By 
David G. Simmons user avatar
David G. Simmons
DZone Core CORE ·
Oct. 15, 18 · Tutorial
Likes (3)
Comment
Save
Tweet
Share
8.7K Views

Join the DZone community and get the full member experience.

Join For Free

As more and more IoT deployments look to leverage the power of cloud-based solutions, we here at InfluxData have been working closely with Cloud Service Providers to make sure that InfluxDB continues to be the go-to Time Series Database for IoT. Google's Core IoT platform is no exception, so we've built a new Telegraf Plugin for Google Core IoT as well as a complete end-to-end tutorial on deploying your IoT solution using Google Core IoT and InfluxData.

This tutorial will assume that you have an instance of InfluxDB running that is accessible.

We will be also be building and installing a version of Telegraf that supports the Google Core IoT Pub/Sub client. That instance of Telegraf must be running on a publicly accessible Internet server that is available with an SSL encryption certificate. Google Pub/Sub can only write to SSL-secured instances of Telegraf.

I've got to say a special thanks to the engineers at Google for their assistance in making this happen. Without our close partnership, this would not have been possible.

What You Will Need

As I said, this is a complete tutorial on building an end-to-end IoT application with Google Core IoT, so we will definitely need to build ourselves an IoT device. There are a couple of ways to do this, including not building a device at all, so we will go through all the options.

What you'll need to complete this tutorial:

  • Google Cloud Account
  • InfluxDB instance, which is available on the Internet

You can, of course, run your InfluxDB instance in the Google Cloud, but getting that set up is beyond the scope of this tutorial.

If you want to use real sensor data, you'll need:

  • Raspberry Pi 3 device
  • Bosch BME 280 sensor board from Adafruit
  • Senseair K30

Otherwise, you can use virtual data generated by the application in simulation mode.

All of the code required for this tutorial is available in the following GitHub repositories:

  • Raspberry Pi Code
  • Telegraf Plugin

You will have to build Telegraf from source (from the link above) in order to use this integration, as the plugin is not yet available in an official build of Telegraf yet. Complete instructions for doing that are included below as well.

IoT Core Setup on the GCP Side

Google Cloud Platform Account

Before you can set up IoT Core, you will need an account with Google. You can start with your Google account, if you have one, or create a new one for this tutorial. Once you have your Google account ready, you will be able to run the cloud commands below. Alternatively, you can use the Google Cloud Console. The instructions for that are in the next section.

Install the Google Cloud Command Line Tool

Start by installing the Google Cloud Command Line Tools according to this document from Google. You'll need to install the components, authenticate to Google Cloud, and create and configure your project:

# gcloud components install beta
# gcloud auth login
# gcloud projects create  <YOUR_PROJECT_NAME>
# gcloud config set project  <YOUR_PROJECT_NAME>
# gcloud beta pubsub topics create  <iot-topic>
# gcloud beta pubsub subscriptions create --topic  <iot-topic>  <iot-subscription> 


The Google Cloud IoT platform requires that all communications be secured with SSL/TLS, so you'll need to generate a Public/Private key pair in order for your device to communicate with the Cloud. First, you'll generate RSA Public and Private keys, and then, you will use those keys to send data.

As long as your computer has OpenSSL installed, you can run the following commands to generate the required key files:

# mkdir PemFiles
# cd PemFiles
# openssl req -x509 -newkey rsa:2048 -keyout private_key.pem -nodes -out public_key.pem -subj "/CN=unused"
Now that you have your Public/Private key pair, your can create a device registry and a device.
# gcloud beta iot registries create <iot-registry> --region us-central1 --event-pubsub-topic=<iot-topic>
# gcloud iot devices create <DEVICE_ID>\
 --project=<PROJECT_ID> \
 --region=<REGION> \
 --registry=<REGISTRY_ID>\
 --public-key path=public_key_file,type=rs256


Note: replace public_key_file with the path to the public key file generated above.

Using the Google Web Console

You will need to login to your Google Cloud Platform account. There's a great Getting Started Guide to get you started.

Create a Registry

You will need to create a Google Core IoT Registry:

Once you've created your Public/Private key pair and cert for your device, you can create a device to write to that registry.

Once you've created the device with the proper security certificate, you can run the client application on that device.

InfluxData Components

This tutorial will assume that you have an instance of InfluxDB running that is accessible.

We will be also be building and installing a version of Telegraf, which supports the Google Core IoT Pub/Sub client. That instance of Telegraf must be running on a publicly accessible Internet server that is available with an SSL encryption certificate. Google Pub/Sub can only write SSL-secured instances of Telegraf.

Building InfluxData Telegraf

Right now, the Google Core IoT Plugin for Telegraf has not been released in a 'released' version of Telegraf. This is a preview release and, hence, must be built from source before running.

One of the prerequisites for this tutorial is understanding how to use Go. Installing and configuring Go is beyond the scope of this tutorial, but you can find detailed instructions for your operating system here.

Note: It is possible to build Telegraf on your local machine and run it on your Internet-accessible server by setting some GO environment variables, which we will also cover later.

Building Telegraf

First, you will want to clone the GoogleIoTCore branch of Telegraf to your machine, then you'll need to build telegraf:

$ git clone https://github.com/davidgs/telegraf/tree/GoogleIoT
$ cd GoogleIoT
$ make


The above code will build the Telegraf agent for your platform. If you are building on a Mac OS X machine, but plan to run it on a Linux machine, you can still build it locally:

$ export GOOS=linux
$ export GOARCH=amd64
$ make


The above code will build a 64-bit linux version of Telegraf.

Configuring Telegraf

Once you have built Telegraf, you will need to create, and then edit, a configuration file to adjust the Google-specific section.

$ telegraf config > telegraf.conf

## Server listens on <server name>:port
## Address and port to host HTTP listener on
service_address = ":9999"
## Path to serve
## default is /write
## path = "/write"
## maximum duration before timing out read of the request
read_timeout = "10s"
## maximum duration before timing out write of the response
write_timeout = "10s"
## precision of the time stamps. can be one of the following:
# second, millisecond, microsecond, nanosecond
# Default is nanosecond
precision = "nanosecond"
# Data Format is either line protocol or json
protocol="line protocol"
## Set one or more allowed client CA certificate file names to
## enable mutually authenticated TLS connections
## tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]
## Add service certificate and key
tls_cert = "/etc/telegraf/cert.pem"
tls_key = "/etc/telegraf/key.pem"
## You **must** provide the certificate and key files!


Once this file is complete, save it to /etc/telegraf/telegraf.conf, and then you can start telegraf:

 $ sudo telegraf -config /etc/telegraf/telegraf.conf 


You can verify that your Telegraf agent is listening with:

 $ netstat -nlp 


And, you can look for the output line:

 tcp6 129 0 :::9999 :::* LISTEN - 


Running the Client App

The client app is also written entirely in Go, so you should be able to run it on virtually any device that supports Golang. For this example, I have been running it on a Raspberry Pi 3, but any device that has a network connection and can run Go code should be fine.

$ git clone https://github.com/davidgs/GoogleIoTClient
$ cd GoogleIoTClient
$ go build GoogleIoT.go 
$ ./GoogleIoT -ca_certs roots.pem -device [deviceName] -private_key [privatekey.pem] -project [projectID] -region [region] -registry [registryName] -format json -virtual


The above command (after you replace the values between []s) will publish 'virtual' (random) values in JSON format.

Sample Output on the Device Without a K30 Sensor:

$ ./GoogleIoT -ca_certs roots.pem -device [deviceName] -private_key [privatekey.pem] -project [projectID] -region [region] -registry [registryName] -format line 


Would publish live sensor data — assuming you have the above-mentioned sensors — in Line Protocol format and show the following output:

2018/09/15 12:45:58 [main] Humidity: 66.24 Temperature: 22.56
2018/09/15 12:45:59 [main] Humidity: 66.26 Temperature: 22.56
2018/09/15 12:46:01 [main] Humidity: 66.24 Temperature: 22.57
2018/09/15 12:46:02 [main] Humidity: 66.23 Temperature: 22.56
2018/09/15 12:46:04 [main] testingGoogle,sensor=bme_280 temp_c=22.56,humidity=66.24 1537029958025504878
testingGoogle,sensor=bme_280 temp_c=22.56,humidity=66.26 1537029959527147539
testingGoogle,sensor=bme_280 temp_c=22.57,humidity=66.24 1537029961028724629
testingGoogle,sensor=bme_280 temp_c=22.56,humidity=66.23 1537029962530306772


Note that the output is four lines of Influx Line Protocol, so four temp and humidity points are written with each iteration.

This writes data to the Google Core IoT MQTT broker, which then sends it to the Google Pub/Sub agent, which pushes to Telegraf using the GoogleIoT branch of Telegraf from davidgs.

Results

If you've gotten everything configured correctly and are running the client app, you should be able to launch Chronograf and see data streaming in to the database in real time.

You now have a complete, end-to-end data pipeline from device to Google Core IoT and into InfluxDB for your IoT sensor data!

A huge thanks to Daz Wilkins for 90% of the code! Couldn't find it on his Github — good developers copy, great developers paste. Happy coding!

Google (verb) IoT

Published at DZone with permission of David G. Simmons, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Google Cloud Document AI Basics
  • Patch Management in the Age of IoT: Challenges and Solutions
  • IoT Communication Protocols for Efficient Device Integration
  • Simplify Your Compliance With Google Cloud Assured Workloads

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!