Over a million developers have joined DZone.

This Week in Hadoop and More: Global Presentations

There's a great deal of content from Spark, Flink, Hadoop, and NiFi from conferences around the world. Also, we introduce a few other great new resources.

· Big Data Zone

Learn how you can maximize big data in the cloud with Apache Hadoop. Download this eBook now. Brought to you in partnership with Hortonworks.

FlinkForward Conference 2016

My friend Julian Hyde does a very interesting talk on StreamingSQL which is going to be huge in many of the streaming systems.  With Apache Calcite, the hope is that all the different Streaming SQLs will be using the same tools, syntax and libraries so we can have compatibility across streaming libraries.

Another sign that Apache Flink is becoming more mature, Apache Flink Security Enhancements.

One of the interesting features of Flink is Code Generation in Serializers and Comparators.

Flink is now able to work with Apache Zeppelin, making on par with the ease of data exploration that you get with Apache Spark.

Spark Summit EU 2016

Spark for ETL (Nov 2016)

Spark and Lucene for near-realtime predictive model building

Problem Solving Recipes Learned from Supporting Spark

Elastic Streaming (Spark)

Sparkling Insights at Elsevier

Spark - Universal Computation Engine for Processing Oil Industry Data

Spark and SAP HANA Vora

Online Learning With Structured Streaming


Adopting Dataframes and Parquet in an already existing datawarehouse

Spark ML with Redis

Sparksheet - Transforming Spreadsheets into Spark Data Frames (GitHub)

Boosting Spark Performance on Many-Core Machines

Performance Characterization of In-Memory Data Analytics on a Scale-up Server

Extending Spark with Java Agents (GitHub)

SparkSQL 2.0/2.1 Experiences using TPC-DS

Apache Spark and Object Stores - What you need to know

Deep Dive into Catalyst: Apache Spark's Optimizer

Apache Ignite with Apache Spark

Sparkling Water 2.0:   H2O Deep Water for working with TensorFlow, MXNet, and Caffe and with Zeppelin.  This really expands what you can do on the JVM.

Effective Spark with Alluxio

Mastering Spark Unit Testing

Online Structured Streaming

Apache Spark 2.0 Performance Improvements Investigated with Flame Graphs (Details)

Streaming Outlier Analysis for Fun and Scalability

IoT With NIFI

For complex IoT programming, you need a combination of Apache NiFi, Storm, and Kafka Together.  This is a great talk on Edge Intelligence IoT with MiniFi.  Installing MiniFI on Your Device, will allow you to access all kinds of IoT devices and stream their data into your corporate NIFI servers.  MiniFi in C++ - Very early, but fast small C++ compiled code for devices without Java.

Hadoop Summit Tokyo Oct 2016

Keynote on Connectivity, Big Data, and IoT



Apache NiFi 1.0 in a Nutshell

Zeppelin with Matlib Plots : A cool way to show more complex graph plots.

Hadoop Summit Tokyo Notes

Spark Tuning for Enterprise System Administrators (Cheat Sheet) for Spark 1.6

Apache Spark Crash Course : this is a great introduction to learning Apache Spark.

Apache HiveMall on Spark Dataframes for doing Machine Learning in your Hadoop Database

With expanded talk on Apache HiveMall: ML Lib for Hive/Spark/Pig

Apache NiFi Crash Course:  a great introduction to learning Apache NiFi.

Streamline Hadoop DevOps with Apache Ambari

Hadoop Sandbox Workshop

Other Good Recent Talks

Practical Kerberos with Apache HBase

A nice overview and dive into Hadoop, Hive, HBase and NoSQL

Credit Card Fraud Prevention on the HWX Stack

Cool Libraries

Open Source Bulk Data Loader

FINRA Open Source Libraries for Big DataTesting Hive Scripts in Your IDE

Succinct (Example) (GitHub) from Berkley Labs allows for queries on compressed data in Spark.

TensorFlow for Speech Recognition

Hortonworks DataFlow is an integrated platform that makes data ingestion fast, easy, and secure. Download the white paper now.  Brought to you in partnership with Hortonworks

Topics:
spark ,hadoop ,big data ,machine learning ,nosql

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}