This is the third part of our ongoing series on Big Data, how we see it, and how we build products for it. In this installment, we’ll cover the first responsibility of the data engineer: building pipelines.

Related posts:

1. Big Data World: Part 1. Definitions.
2. Big Data World: Part 2. Roles.

Table of contents:

• What a pipeline is
• Orchestrators
• ETL Tools
• Batch vs stream processing
• Apache NiFi
• Conclusion

As we discussed in the previous part, data engineers are responsible for transferring data and for the architecture of DWH, among other things. This sounds quite simple. But in reality, the best way to accomplish these tasks is not always obvious.

#news #big-data #big-data-tools #data-engineering #data-pipelines