This Week in Hadoop and More: Deep Learning and Tools
The world of Hadoop this week had some major updates on Deep Learning, Hadoop, Spark and AI.
Join the DZone community and get the full member experience.Join For Free
Security is always a big issue, I wrote a few articles about this recently as people are Google Dorking and using Shodan to find vulnerable, unprotected NoSQL stores like MongoDB, Redis, ElasticSearch, and WebHDFS to attack. Check this out.
Old School Production Grade Mature Hadoop Tools keep coming up, here's a great article on SQOOP Performance. SQOOP is a powerful tool for ingres and egress of data into and out of RDBMS' like Oracle and Postgresql. Hard to beat a tool that can harness a whole cluster for data loading.
A nice article on working with Apache HAWQ (HDB) within Hortonworks HDP. A number of great presentations came out of the AI Frontiers conference on Deep Learning.
This week a few software updates of note, Apache Beam is out of incubation. And so is Apache Ranger. Hive as a Service on CloudFoundry is a very interesting project to try. This github repo has a fun Deep Learning application to manipulate images. On my GPU-less machine it took over an hour to run, so not for real-time streams unless you've pretrained and you have some GPU heavy clusters.
Top Articles of the Week
Top Presentations of the Week
A Deep Dive into Structure Streaming in Spark 2.0 : a great talk on the new features via Databricks
Apache Phoenix and Zeppelin Formatting Hack of the Week
%jdbc(phoenix) select '%html @' || handle || ' (' || friends_count || ',' || followers_count || ',' || retweet_count || ') <img width=100 height=100 src="' || COALESCE( profile_image_url, 'https://upload.wikimedia.org/wikipedia/en/4/48/Blank.JPG') || '"> ' || msg || ' @ ' || location || ' <br> Sentiment: ' || stanfordsentiment || ' <img width=50 height=50 src="' || COALESCE( media_url, 'https://upload.wikimedia.org/wikipedia/en/4/48/Blank.JPG') || ' "> ' as Tweets from TweetTable where language = 'en' and location is not null and msg like '%Water%main%break%' order by time desc
Wikimedia has a blank image I am using as a placeholder if I have no image from the tweet. Coalesce is a nice function in Phoenix that displays the second choice if the first one is null. A handy function for many cases, especially this one when you are constructing HTML on the fly.
Opinions expressed by DZone contributors are their own.