This Week in Hadoop and More: Deep Dive on NiFi, Presentations & Spark

This week, we have Big Data, Fast Data, Data in Motion, and the tools, languages, and techniques to enable better solutions with many types of data. A ton of great presentations on Apache Big Data projects came out of the Apache Big Data Conference that just completed in Vancouver. There's also a couple of interesting Hadoop tutorials for working with Zeppelin and Pig.

Mining Tez App Logs with Pig

Using ZeppelinHub Viewer

There's an interesting new video on working with Hadoop on Cisco: How to Accelerate Time to Value on Your Data (Cisco). A subproject of Apache NiFi - MiniFi for Small IoT Devices is on Github and is very interesting. It helps extend this former NSA project to smaller devices and platforms and turn NIFI into a real IOT powerhouse.

Pivotal has an excellent article on why Agility, Big Data, and Cloud work together in symbiosis.

An interesting use case that effects everyone is Building Big Data for Smart Cities.

As Big Data and Hadoop has added more and more projects and become more complex to get all the pieces in the correct versions to work together, the open consortium (ODPi) is really enabling standards and ODPi becomes Gold Sponsor of ASF. This is helping to push forward the open goals of both organizations.

A very interesting in-depth advanced Spark article this week on Spark RDD Partitioning.

The Apache Big Data 2016 event in Vancouver has produced an amazing stream of presentations on all facets of Spark, NiFi, HBase, Hadoop, Data Science, Zeppelin, YARN, HAWQ, SQL, ProtoBufs, Thrift, and more. If you have time this week, look at these presentations. Not to forget the NJ Hadoop meetup for having a great talk on Apache NiFi.

• New Developments in Hbase

• Protecting Enterprise Data in Apache Hadoop 

• Deep Dive on Apache NiFi

• Introduction to Big Data on Hadoop

• Mining Public Dataset with Zeppelin and Spark

• Data Science for the data center

• Random Forest Clustering on Spark

• Cancer Analysis with Spark

• YARN: Analytic Platform

• Apache HAWQ with YARN

• Apache SOLR and Spark for BOM

• Ozone: Object Store for Hadoop (JIRA)

• SciSpark Atmospheric Science on Spark and Zeppelin (NASA/JPL)

• Speaking The Language of Big Data (ProtoBuf / Thrift)

• Open Spatial Consortium: Working on Open Source and Open Standards including Magellan, Geospatial Analytics on Spark, SciSpark

• Building Large Scale Applications in Apache Hadoop YARN with Apache Twill

That's it for this week; next week we will have all the new items in Spark, NiFi, Hadoop, and related projects in Big Data as this will be a weekly column.