Using Python to Store Data From Many BLE Devices

BLE (Bluetooth Low Energy) sensor devices like the Hexiwear are great, but they cannot store a large amount of data. For a research project, I have to collect data from many BLE devices for later processing. What I’m using is a Python script running on a Raspberry Pi, which collects the data and stores it on a file:

Raspberry Pi with Python controlling a set of Hexiwear BLE devices

Getting Data From a Single Device

The following script gets sensor data from a single device:

# Using Hexiwear with Python
# Script to get the device data and append it to a file
# Usage
# python GetData.py <device>
# e.g. python GetData.py "00:29:40:08:00:01"
import pexpect
import time
import sys
import os

# ---------------------------------------------------------------------
# function to transform hex string like "0a cd" into signed integer
# ---------------------------------------------------------------------
def hexStrToInt(hexstr):
    val = int(hexstr[0:2],16) + (int(hexstr[3:5],16)<<8)
    if ((val&0x8000)==0x8000): # treat signed 16bits
        val = -((val^0xffff)+1)
    return val
# ---------------------------------------------------------------------

DEVICE = "00:29:40:08:00:01"   # device #24

if len(sys.argv) == 2:
  DEVICE = str(sys.argv[1])

# Run gatttool interactively.
child = pexpect.spawn("gatttool -I")

# Connect to the device.
print("Connecting to:"),
print(DEVICE)

NOF_REMAINING_RETRY = 3

while True:
  try:
    child.sendline("connect {0}".format(DEVICE))
    child.expect("Connection successful", timeout=5)
  except pexpect.TIMEOUT:
    NOF_REMAINING_RETRY = NOF_REMAINING_RETRY-1
    if (NOF_REMAINING_RETRY>0):
      print "timeout, retry..."
      continue
    else:
      print "timeout, giving up."
      break
  else:
    print("Connected!")
    break

if NOF_REMAINING_RETRY>0:
  unixTime = int(time.time())
  unixTime += 60*60 # GMT+1
  unixTime += 60*60 # added daylight saving time of one hour

  # open file
  file = open("data.csv", "a")
  if (os.path.getsize("data.csv")==0):
    file.write("Device\ttime\tAppMode\tBattery\tAmbient\tTemperature\tHumidity\tPressure\tHeartRate\tSteps\tCalorie\tAccX\tAccY\tAccZ\tGyroX\tGyroY\tGyroZ\tMagX\tMagY\tMagZ\n")

  file.write(DEVICE)
  file.write("\t")
  file.write(str(unixTime)) # Unix timestamp in seconds 
  file.write("\t")

  # App mode
  child.sendline("char-read-hnd 0x6d")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("AppMode:  "),
  print(child.before),
  print(str(int(child.before[0:2],16)))

  file.write(str(int(child.before[0:2],16)))
  file.write("\t")

  # Battery
  child.sendline("char-read-hnd 0x28")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Battery:  "),
  print(child.before),
  print(str(int(child.before[0:2],16)))

  file.write(str(int(child.before[0:2],16)))
  file.write("\t")

  # Ambient Light (0x2011)
  child.sendline("char-read-hnd 0x3f")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Ambient:  "),
  print(child.before),
  print(str(int(child.before[0:2],16)))

  file.write(str(int(child.before[0:2],16)))
  file.write("\t")

  # Temperature (0x2012)
  child.sendline("char-read-hnd 0x43")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Temperature:  "),
  print(child.before),
  print(float(hexStrToInt(child.before[0:5]))/100)

  file.write(str(float(hexStrToInt(child.before[0:5]))/100))
  file.write("\t")

  # Humidity (0x2013)
  child.sendline("char-read-hnd 0x47")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Humidity:  "),
  print(child.before),
  print(float(hexStrToInt(child.before[0:5]))/100)

  file.write(str(float(hexStrToInt(child.before[0:5]))/100))
  file.write("\t")

  # Pressure (0x2014)
  child.sendline("char-read-hnd 0x4b")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Pressure:  "),
  print(child.before),
  print(float(hexStrToInt(child.before[0:5]))/100)

  file.write(str(float(hexStrToInt(child.before[0:5]))/100))
  file.write("\t")

  # HeartRate (0x2021)
  child.sendline("char-read-hnd 0x52")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("HeartRate:  "),
  print(child.before),
  print(str(int(child.before[0:2],16)))

  file.write(str(int(child.before[0:2],16)))
  file.write("\t")

  # Steps (0x2022)
  child.sendline("char-read-hnd 0x56")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Steps:  "),
  print(child.before),
  print(hexStrToInt(child.before[0:5]))

  file.write(str(hexStrToInt(child.before[0:5])))
  file.write("\t")

  # Calorie (0x2023)
  child.sendline("char-read-hnd 0x5a")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Calorie:  "),
  print(child.before),
  print(hexStrToInt(child.before[0:5]))

  file.write(str(hexStrToInt(child.before[0:5])))
  file.write("\t")

  # Accelerometer
  child.sendline("char-read-hnd 0x30")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=5)
  print("Accel:  "),
  print(child.before),
  print(float(hexStrToInt(child.before[0:5]))/100),
  print(float(hexStrToInt(child.before[6:11]))/100),
  print(float(hexStrToInt(child.before[12:17]))/100)

  file.write(str(float(hexStrToInt(child.before[0:5]))/100))
  file.write("\t")
  file.write(str(float(hexStrToInt(child.before[6:11]))/100))
  file.write("\t")
  file.write(str(float(hexStrToInt(child.before[12:17]))/100))
  file.write("\t")

  # Gyro
  child.sendline("char-read-hnd 0x34")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=10)
  print("Gyro:   "),
  print(child.before),
  print(float(hexStrToInt(child.before[0:5]))/100),
  print(float(hexStrToInt(child.before[6:11]))/100),
  print(float(hexStrToInt(child.before[12:17]))/100)
  file.write(str(float(hexStrToInt(child.before[0:5]))/100))
  file.write("\t")
  file.write(str(float(hexStrToInt(child.before[6:11]))/100))
  file.write("\t")
  file.write(str(float(hexStrToInt(child.before[12:17]))/100))
  file.write("\t")

  # Magnetometer
  child.sendline("char-read-hnd 0x38")
  child.expect("Characteristic value/descriptor: ", timeout=5)
  child.expect("\r\n", timeout=10)
  print("Magneto:"),
  print(child.before),
  print(hexStrToInt(child.before[0:5])),
  print(hexStrToInt(child.before[6:11])),
  print(hexStrToInt(child.before[12:17]))

  file.write(str(float(hexStrToInt(child.before[0:5]))/100))
  file.write("\t")
  file.write(str(float(hexStrToInt(child.before[6:11]))/100))
  file.write("\t")
  file.write(str(float(hexStrToInt(child.before[12:17]))/100))
  file.write("\t")

  file.write("\n")
  file.close()
  print("done!")

  sys.exit(0)
else:
  print("FAILED!")
  sys.exit(-1)

Calling it with the device ID, it gets the sensor node data, appends it to the file, and shows it on the console:

pi@raspberrypi:~ $ python GetData.py "00:2A:40:0B:00:4E"
Connecting to: 00:2A:40:0B:00:4E
Connected!
AppMode: 02 2
Battery: 64 100
Ambient: 0b 11
Accel: 05 00 02 00 9b ff 0.05 0.02 -1.01
...
done!

The data gets stored into a CSV text file.

Getting Data from Multiple Devices

That brings us to the following script file:

# Python script to get data from multiple devices
# Usage:
# python GetDataAll.py

import subprocess

# list of devices
devices = [
  "00:32:40:08:00:12",  # device 01
  "00:2F:17:03:00:35",  # device 20
  "00:28:22:0C:00:15",  # device 21
  "00:15:77:03:10:01",  # device 22
  "00:2A:40:0B:00:4E",  # device 23
  "00:29:40:08:00:01",  # device 24
   ]

for x in range(0,len(devices)):
  cmd = "python GetData.py " + devices[x]
  subprocess.call(cmd, shell=True)

print("finished all devices!")

The script has a list of BLE devices, which are used to poll the data.

Summary

With a Raspberry Pi and Python scripting, I can collect data from multiple BLE devices and store the information into a file for further processing. Because I don’t need to keep the BLE connection active, I can extend this to an almost unlimited number of devices, without running into the typical BLE connection limit of around eight devices.

I’m using a similar method to update all the devices (e.g. with the current time/date), plus I have implemented a remote shell in each BLE device: That way, I can send commands to each node, which then are executed on the BLE device.

The Python scripts used can be found on GitHub.

Happy saving!

Links

• Using Python, Gatttool, and Bluetooth Low Energy with Hexiwear.
• Python scripts.