Tim Spann

Any updates to this since KRaFT?

https://github.com/tspannhw/pulsar-thermal-pinot/blob/main/weather.md

I have put out three updated release NARs for New DJL 0.8.0 framework.

Put them into 2 docker nodes. Are you using https://hub.docker.com/r/apache/nifi-registry Is configuration right? It has to store data.https://nifi.apache.org/docs/nifi-registry-docs/html/getting-started.html

Upgrade to 1.9. Is it kerberized or have a login?

Spark 1.6 is gone from NiFi 1.8. Livy is installed as part of HDP 2.6/3.0/3.1.

See https://community.hortonworks.com/articles/227560/real-time-stock-processing-with-apache-nifi-and-ap.html

Source: https://community.hortonworks.com/storage/attachments/93299-stock-to-kafka.xml https://community.hortonworks.com/storage/attachments/93298-stocks-copy.json https://github.com/tspannhw/stocks-nifi-kafka

Install Java 8

Install NiFi https://nifi.apache.org/download.html

Install Kafka https://kafka.apache.org/downloads

Or get a linux box or big VM

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/index.html

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/index.html

code is in github https://community.hortonworks.com/articles/198939/using-apache-mxnet-gluoncv-with-apache-nifi-for-de.html https://github.com/tspannhw/UsingGluonCV/tree/master https://github.com/tspannhw/OpenSourceComputerVision/

https://github.com/minimaxir/person-blocker is the imaging tool.

The flow is apache nifi, it's not visualization it's programming

we don't have one to list all topics. you could call kafka-topics.sh to get the list or make an API call

https://community.hortonworks.com/articles/57262/integrating-apache-nifi-and-apache-kafka.html

https://community.hortonworks.com/articles/155527/ingesting-golden-gate-records-from-apache-kafka-an.html

ConsumeKafkaRecord_1_0 (comma list of all topics) to ConvertRecord to PutHDFS you may add ConvertAvroToOrc or PutParquet

it has worked for me. post here https://community.hortonworks.com/gallery/index.html https://community.hortonworks.com/questions/1629/nifi-connection-to-mssql-server-db.html https://community.hortonworks.com/articles/87632/ingesting-sql-server-tables-into-hive-via-apache-n.html

https://github.com/bazaarvoice/jolt/issues/130

I use default values. https://community.hortonworks.com/articles/149910/handling-hl7-records-part-1-hl7-ingest.html

good point, this was a follow up to the other article, should have had a review. sorry.

Josh Long is great, when I worked at Pivotal I got a few articles in the Spring Weekly list.

Spring for Hadoop hasn't been updated in forever. It is stuck on HDP 2.2 and we are on HDP 2.6. Spring Data JDBC and Spring Data Repositories make a lot of sense. I should do that, I'll do an update when I get the chance. Maybe add Java 9 and some other goodies. Thanks for the suggestions. If you want to fork the github repo, please do!

Use getfile or any of the file reading ones. NiFi doesn't care file type. Then to convert to avro just use the ConvertRecord it can do many types including JSON

http://opennlp.sourceforge.net/models-1.5/

You need to install the OpenNLP models and reference that in the processor properties. Also OpenNLP misses a lot of names and locations. Accuracy is kind of hit or miss. https://github.com/tspannhw/nifi-nlp-processor https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html

https://community.hortonworks.com/articles/166802/versioned-dataflows-with-apache-nifi-15-and-apache.html

Jean-Paul,

There's another open source registry that integrates with Kafka and other systems extremely well and has a great REST API and UI:

https://github.com/hortonworks/registry

It has versioning and is moving to adding protocol buffers and going into Apache.

Have you tried that one?

the convertavrotoorc processor generates a ddl attribute with the create external table ddl needed you can send that directly to a puthiveql processor to build the table or di yourself

NiFi can run on a cluster of servers to distribute the load. NiFi generally supports 50 megabytes a second per node

http://www.techrepublic.com/google-amp/article/the-battle-for-apache-cassandra-highlights-major-problem-with-open-source-projects/

google it. apacheissues andonlydatastax cancommit

youd have to hand write that and start an odd project. theres some engines rhat use hbase for that part. you could fork one and do it yourself. but why? unless you work for datastax, cassandra is fir practical purposes closed source

no. spark id useful for processing. no need for cassandra ever

not hadoops but mine. its time to update i will run all my tools and report results

awesome

have you seen the open source Superset from airbnb and hortonworks

paused on that one will try this weekend

https://dzone.com/articles/deep-learning-and-machine-learning-guide-part-ii

https://dzone.com/articles/deep-learning-and-machine-learning-guide-part-ii

Part II is in progress. Anything you want to see.

start with some basic NLTK if you know Python or start with the google TensorFlow examples.

great article.

Definitely no free lunches. Large proprietary interfaces, any programs that cannot handle the extra time required for network calls or extra startup costs. Any existing code that works and doesn't require major changes. Not everything is greenfield. There are a lot of negatives, mostly around networking costs, duplication and transaction concerns.

very cool

tensorflow andphoenixin Pythonandthey scale out fine

JVM is great for concurrency.. You can do java, scala, jython, clojure and a ton other JVM langs

there is Spark in Scala.

Python is used in a lot of visualization and machine learning. At this point and with all the nodes you can run in a big hadoop / spark cluster, it doesn't matter. YARN runs things well.

Scala and Python are nice for data scientists and engineers. But most of the big data infrastructure is in Java. Java is way too verbose for most stuff, but Google chose it for Apache Beam. I like Apache NiFi so I don't have to write too much code when I don't need. Same for Zeppelin, I like doing bits of Scala and Python code when necessary.

Yes Java and Scala are important. The first half of the article is about machine learning. The second half lists 4 python libraries. probably could be 2 different articles, but I was looking at both. H20 is Java and Scala and I like both languages.

I have used Titan, is a really cool tool

Post a blog on your test query performance. I would like to see all the major Open Source SQL on Hadoop engines tested from Hive2 to HAWQ to Presto to Spark SQL to Drill, etc...

Cassandra is definitely an option for some use cases, but it overlaps with Hadoop and HBase. Cassandra requires some specialized knowledge and is it's own thing. If you are are interested in Cassandra evaluate the SMACK stack. It's not as integrated into the big data tools and stack. There's a big learning curve and those skill don't transfer over. If you learn Hadoop and Spark, you can work with tons of tools. Start with the industry standards and then look at exotic databases when you have some special needs.

tim at sparkdeveloper or @paasdev on twitter. or threw meetup http://www.meetup.com/futureofdata-princeton/

Interesting, contact me, I would like to do an article on your product

Trafodion looks very cool. When I was at HP I remember hearing about it. Tweet at me @paasdev if you want to give me details for an article, thanks.

Probably fixing n

twitter message me at PaasDev and then I'll email you.

Fixed, thanks for the catch! If you have any more information on DL4J, I would love to do an article on it.

Some ways to do Java 8. https://dzone.com/articles/zlwell-written-java

i have seen a lot of messy code. You bring it into IntelliJ / Eclipse and format the code, hide the bad comments and run some static code analysis tools. Java is nice for that.

Nice, that is very cool. Thanks!

This is a horrible comment

// don't touch these lines

int crazyVariableDontRemove = -999;

// don't touch above lines

nice slidedeck! Here it is as a link.

Docker will have a tough year ahead

looks great

Typo fixed thanks!

most likely that would be possible with BlueMix

That's a great list!

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

This article was from 2013. Those are good options too. This should probably be upgraded.

Have you seen Spring Data REST ?

Have you seen Spring Data REST ?

What about Gemfire XD?

http://www.pivotal.io/big-data/pivotal-gemfire-xd

What about Gemfire?

http://www.pivotal.io/big-data/pivotal-gemfire

What about Gemfire XD?

http://www.pivotal.io/big-data/pivotal-gemfire-xd

What about Gemfire?

http://www.pivotal.io/big-data/pivotal-gemfire

Good catch. I like your second idea better though.

Good catch. I like your second idea better though.

Ours was dying on hotmail and gmail only. Turned out to be a mail server issue with one particular email address used for sending. Black list security issue.

/**
*
* @param subject
* @param message
* @param emailAddress
* @return
*/
public final static String sendEmail(String subject, String message, String emailAddress) {
// email sent
boolean emailSent = false;

// email message
StringBuilder emailLoggingMessage =
new StringBuilder(UserProvisioningUtility.gkBUFFER_SIZE);

// start of message
emailLoggingMessage
.append(Messages.getString("UserProvisioningProcess.SMTP_MESSAGE_LOG"))
.append(emailAddress).append(System.getProperty("line.separator"));

Email email = new SimpleEmail();
//email.setHostName("smtp.googlemail.com");
//email.setSmtpPort(465);
email.setHostName(Messages.getString("UserProvisioningProcess.EMAIL_SERVER"));

int smtpPort = 25;
if ( null != Messages.getString("UserProvisioningProcess.EMAIL_PORT")) {
try {
smtpPort = Integer.parseInt(Messages.getString("UserProvisioningProcess.EMAIL_PORT"));
}
catch(Throwable t) {
emailLoggingMessage.append(t.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
}
email.setSmtpPort(smtpPort);

// email.setAuthenticator(new DefaultAuthenticator("username", "password"));
// email.setSSLOnConnect(true);
try {
email.setFrom(Messages.getString("UserProvisioningProcess.FROM_EMAIL"));
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
email.setSubject(subject);
try {
email.setMsg(message);
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
try {
email.addTo(emailAddress);
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
try {
email.send();
emailSent = true;
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}

// sent message
if (emailSent) {
emailLoggingMessage.append(
Messages.getString("UserProvisioningProcess.SMTP_MESSAGE_SENT"))
.append(System.getProperty("line.separator"));
}

return emailLoggingMessage.toString();
}

Ours was dying on hotmail and gmail only. Turned out to be a mail server issue with one particular email address used for sending. Black list security issue.

/**
*
* @param subject
* @param message
* @param emailAddress
* @return
*/
public final static String sendEmail(String subject, String message, String emailAddress) {
// email sent
boolean emailSent = false;

// email message
StringBuilder emailLoggingMessage =
new StringBuilder(UserProvisioningUtility.gkBUFFER_SIZE);

// start of message
emailLoggingMessage
.append(Messages.getString("UserProvisioningProcess.SMTP_MESSAGE_LOG"))
.append(emailAddress).append(System.getProperty("line.separator"));

Email email = new SimpleEmail();
//email.setHostName("smtp.googlemail.com");
//email.setSmtpPort(465);
email.setHostName(Messages.getString("UserProvisioningProcess.EMAIL_SERVER"));

int smtpPort = 25;
if ( null != Messages.getString("UserProvisioningProcess.EMAIL_PORT")) {
try {
smtpPort = Integer.parseInt(Messages.getString("UserProvisioningProcess.EMAIL_PORT"));
}
catch(Throwable t) {
emailLoggingMessage.append(t.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
}
email.setSmtpPort(smtpPort);

// email.setAuthenticator(new DefaultAuthenticator("username", "password"));
// email.setSSLOnConnect(true);
try {
email.setFrom(Messages.getString("UserProvisioningProcess.FROM_EMAIL"));
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
email.setSubject(subject);
try {
email.setMsg(message);
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
try {
email.addTo(emailAddress);
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}
try {
email.send();
emailSent = true;
} catch (EmailException e) {
emailLoggingMessage.append(e.getLocalizedMessage()).append(
System.getProperty("line.separator"));
}

// sent message
if (emailSent) {
emailLoggingMessage.append(
Messages.getString("UserProvisioningProcess.SMTP_MESSAGE_SENT"))
.append(System.getProperty("line.separator"));
}

return emailLoggingMessage.toString();
}

Eclipse and other IDEs have automation in the sense of Wizards and configuration tools that do a lot of the basic work for you. There's also plugins to do some things. You can also have scripts in Maven, Roo, Gradle and Ant that can be run from the IDE to automate things.

And once you have a project setup, some shell code and unit tests and version control configured, Jenkins can automate your build/deploy/tests/... That first project setup is usually a wizard, template or a github base project. Spring IDE and JBOSS have some extra templates that really help you get started. That's still a lot of complexity.