First off, this event really expanded on what was done in Munich earlier this year, adding a lot of cool new announcements, a new partnership, and some new projects.
The biggest non-technical news was that IBM will be using HDP as their official Hadoop. This is a big deal, as this helps consolidate the Hadoop market. IBM was already using and is part of ODPi, so this made a lot of sense. EMC Pivotal, Microsoft, IBM, and Hortonworks all being on the same Open Apache version of Hadoop and having guidance from the ODPi makes for a very open release of this critical enterprise platform. To make it even better, IBM Apache committers can now work even closer with Hortonworks to ensure the continued fast development of critical enterprise projects like Apache Ranger and Knox.
HDF 3.0 was released and this includes the new projects of Streaming Analytics Manager (SAM), Apache NiFi 1.2, and Schema Registry. This included revolutionary software the builds real-time streaming applications on top of a Unified API that generates, builds, and deploys Storm applications. The Unified API allows for other streaming options in the near term like Apache Beam, Spark, and Flink.
Silicon Angle has a number of great interviews from their event:
The keynotes provide a lot of direction in the future of big data, AI, and streaming.
Another great talk is by Verizon around their self-service data lake.
There was a great talk by the team on seamless access control with Apache Spark and Ranger.
Here are my Top 10 Talks of the event
If you are in Australia, check out the next Data Works summit there. Otherwise, you are waiting for Berlin in 2018. See you there!