Moving data from system A to system B

This is pretty old age problem to be solved in majority of projects.

History: It comes under Flow based programming:

Our focus is to move data from system A to system B. Only Extraction and Loading. Not much about Transformations.

Option 0: Hand coding in Python / Java / PERL …etc
This is good for small sets of data. Also good for POC.
Not suggested to push to production without failover, managing jobs, scheduling jobs,…etc


Option 1: If system is heavy and need robust solution, better to go with Apache NiFi

The US National Security Agency open-sourced its Niagrafiles, or NiFi, data-flow software.

How to enable security for NiFi?

How to write Java code for NiFi and other languages?

Other directory with date suffix examples

Commercial support available:

Versioning available:

Externalizing variables possible.
Easy to move configurations from QA to Prod

We can slim down the system to minimize its foot print

NiFi support Hadoop HDFS

But Storm objective is different.


Option 2: Use streaming API of Apache Spark
Sqoop Vs Flume


Option 3: If you are using CDAP, better to use Hydrator to generate JSON and use it.
Bit more study required around metrics, management and tracking these jobs.

Better to stay away from CDAP stack. There is not much public acceptance. No response on their forums. If we ask question, they wont respond. If we call them, they will ask us to buy their support/consulting hours. Nothing wrong in this. But we can’t afford.

We can check their poor support in their groups!forum/cdap-user


Option 4: Pentaho Kettle
It is not ready for Big Data as on March 2017
Good for small java enterprise projects (Coding required with Kettle API). Used in the past. – Java documentation quality is not good.


Option 5: Commercial products is good product and it is having better support with (BI/Analytics)


Option 6: Spring Batch

If we want to minimize number of servers, we want minimal solution, Spring Batch is good one.
But it needs continuous maintenance when there is change in Spring / Java version.

Spring Integration:
Spring batch partitioning:
Spring Batch Reference:
Spring Batch UI:


Use Apache NiFi as much as possible. Works well in production and also quick in POCs

As on March 11 2017:!topic/cdap-user/hiuUP3jIxNs
CDAP Hydrator is not in a position to compete with Apache NiFi


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s