IoT at Global Scale: PowerStream Wind Farm Analytics with Spark
Using PowerStream to simulate IoT devices and analyze data in Apache Spark.
Join the DZone community and get the full member experience.Join For Free
at spark summit east in new york , we unveiled powerstream, an internet of things (iot) simulation with visualizations and alerts based on real-time data from 2 million sensors across global wind farms.
renewable energy, such as wind power, is a viable alternative to traditional sources. for example, danish wind turbines set a new world record for generating energy in 2015. according to recently published data, wind power now accounts for 42.1% of the total electricity consumption in denmark. as sensor technology advances, it becomes possible to monitor wind turbines on wind farms, ensuring maximization of air flow and mechanical power.
powerstream processes and analyzes simulated data from approximately 2 million sensors on 197,000 wind turbines installed around the world.
sensors are found on individual wind turbines within wind farms, as illustrated in the diagram below:
with temperature and vibration data points acquired from these sensors, plus a simple machine learning algorithm, powerstream predicts and visualizes the health of turbines. the application predicts both behavior of individual turbines and calculates aggregate behavior of wind farms. it then displays green or red states. a red state predicts out of normal operating bounds, while a green state indicates the turbine or wind farm is within expected bounds.
data producers simulate sensor activity, pushing approximately 1 million data points every second from ten sensors on each turbine. sensor data is sent to an apache kafka queue, which is processed by a memsql streamliner data pipeline. the pipeline predicts the health of each turbine using a pre-trained machine learning model. the sensor, turbine, and wind farm states are stored in memsql and further analyzed to determine their health (green / red). finally, the powerstream ui queries the memsql database to display states in the web interface. these queries, and subsequent visual display, depend on the map geography and zoom level selected by the user.
powerstream also utilizes memsql geospatial capabilities. geolocation (longitude and latitude) of each turbine is stored in a memsql table, which is joined with other data when the user changes the map area in view. if the users zooms in closely, they will see status of specific turbines (depicted below), as opposed to wind farms (depicted above).
large volumes of data are generated and manipulated in this showcase application – here are a few data points:
- memsql streamliner processes approximately 1 million data points per second, then inserts it into a memsql database.
- when a viewer moves the map on screen, several large database queries run and complete in real time. specifically, large database join operations between the sensors table (~2 million rows), turbines table (~200,000 rows) and wind farms table (~20,000 rows) occur in parallel. this produces a geospatial json file that is compressed and rendered instantly (between 50 and 500 ms) in the web-based ui.
- real-time notifications push to the ui based on a `select *` query from an events table, which scales up to 2 million records. powerstream runs on 7 amazon c4.2xlarge instances, at a rough cost of $0.311 hourly apiece, equating to just under $19,100 annually.
take a look at the memsql ops dashboard below, from which powerstream application and database operators manage and monitor the platform:
powerstream exemplifies one way to use modern technology for good. applying iot principles to energy challenges, like harnessing wind power, can inspire energy companies and government organizations to apply resources and contribute to a more efficient future. real-time analytics applies globally, and will enable energy innovation to spread across countries and oceans.
see powerstream live at spark summit east, booth #101. request a demo here: memsql.com/sparkeast
Published at DZone with permission of Steven Camina, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.