Rathnadevi Manivannan

Senior Technical Writer at Treselle Systems

Joined May 2017

http://treselle.com/blog

Stats

Reputation:	1007
Pageviews:	851.1K
Articles:	16
Comments:	4

Articles
Comments

Articles

Ingesting IoT Sensor Data Into S3 With an RPI3

StreamSets Data Collector Edge is a lightweight agent used to create end-to-end data flow pipelines. We'll use it help stream data collected from a sensor.

December 30, 2017

· 10,308 Views · 4 Likes

Sensor Data Quality Management Using PySpark and Seaborn

Learn how to check data for required values, validate data types, and detect integrity violation using data quality management (DQM).

December 2, 2017

· 11,138 Views · 5 Likes

Import and Ingest Data Into HDFS Using Kafka in StreamSets

Learn about reading data from different data sources such as Amazon Simple Storage Service (S3) and flat files, and writing the data into HDFS using Kafka in StreamSets.

October 26, 2017

· 23,492 Views · 5 Likes

API Response Tracking With StreamSets, Elasticsearch, and Kibana

Learn how to track JSON response data from a RESTful API using Elasticsearch and Kibana to capture and visualize the alerts.

October 15, 2017

· 12,155 Views · 3 Likes

Handling Imbalanced Data With R

Imbalanced data is a huge issue. With imbalanced data, accurate predictions cannot be made. Learn how to tackle imbalanced classification problems using R.

October 9, 2017

· 40,672 Views · 3 Likes

Data Analysis Using Apache Hive and Apache Pig

Learn about loading and storing data using Hive, an open-source data warehouse system, and Pig, which can be used for the ETL data pipeline and iterative processing.

August 18, 2017

· 26,089 Views · 5 Likes

Using Airflow to Manage Talend ETL Jobs

Learn how to schedule and execute Talend jobs with Airflow, an open-source platform that programmatically orchestrates workflows as directed acyclic graphs of tasks.

August 3, 2017

· 21,575 Views · 4 Likes

Using NGINX With GeoIP MaxMind Database to Fetch Geolocation Data

Learn how to find the geographical location of a user using their IP address by just configuring NGINX with GeoIP MaxMind databases — without doing ANY coding!

July 29, 2017

· 16,219 Views · 4 Likes

Database Performance Testing With Apache JMeter

Learn how to construct your database performance testing plan with all the most important elements, including a thread group, JDBC request, and summary report.

July 27, 2017

· 37,024 Views · 3 Likes

Data Normalization and Filtration Using Drools

Learn about normalizing and filtering data using Drools by looking at an example of oil well drilling datasets from Arkansas and Oklahoma.

July 20, 2017

· 14,473 Views · 4 Likes

Data Flow Pipeline Using StreamSets

Learn about configuring JDBC Query Consumer, performing JDBC lookup with multiple tables, creating a data flow pipeline, and monitoring the stage and pipeline stats.

July 18, 2017

· 16,167 Views · 7 Likes

Protractor With Cucumber

Learn how to configure the Protractor testing framework to use with the Cucumber Behavior-Driven Development framework for testing AngularJS applications.

July 11, 2017

· 27,368 Views · 4 Likes

Pivoting and Unpivoting Multiple Columns in MS SQL Server

In this article, we'll discuss converting values of rows into columns (PIVOT) and values of columns into rows (UNPIVOT) in MS SQL Server.

Updated July 9, 2017

· 253,104 Views · 5 Likes

Apache Spark Performance Tuning – Degree of Parallelism

Today we learn about improving performance and increasing speed through partition tuning in a Spark application running on YARN.

June 30, 2017

· 101,372 Views · 8 Likes

Apache Spark on YARN: Resource Planning

Apache Spark is an in-memory distributed data processing engine and YARN is a cluster management technology. Learn how to use them effectively to manage your big data.

June 28, 2017

· 37,137 Views · 8 Likes

Apache Spark on YARN – Performance and Bottlenecks

In this series, we learn about performance tuning and fixing bottlenecks in high-level APIs with an Apache Spark application on YARN.

June 27, 2017

· 30,671 Views · 12 Likes

Comments

Pivoting and Unpivoting Multiple Columns in MS SQL Server

Apr 30, 2018 · Rathnadevi Manivannan

Thanks Sandeep

Data Normalization and Filtration Using Drools

Apr 10, 2018 · Rathnadevi Manivannan

Please follow the below steps to test it locally:

Download raw data of Arkansas and Oklahoma.
Move the raw data into MS SQL. The source code is available in - https://github.com/treselle-systems/data_normalization_and_filtration_using_drools.
Change sql server connection url, username, and password in src/common.properties.
Change the table names in input & output queries in the property file.

Using Airflow to Manage Talend ETL Jobs

Jan 08, 2018 · Rathnadevi Manivannan

Hi Ranga Nathan,

Thanks for your review comments.

The scripts required to run the Talend jobs are mentioned in the DAG files. SQL has nothing to do with DAG and scheduling.

Data Analysis Using Apache Hive and Apache Pig

Sep 04, 2017 · Rathnadevi Manivannan

You are welcome