Just a couple of years ago, software projects didn’t exceed a bunch of files! You could store a project on a Floppy disk and install it here and there. Nowadays, hardware and software compete with each other to run at a faster pace, yet collaborating with each other to help engineers build faster and faster and more complex systems. The age of creating a huge monolith project has gone and we observe the rise of Distributes systems, Microservices, Big data, and Cloud.

As systems become more complicated, problems also grow with them. Day by day, Technologies, Projects, and Libraries emerge to solve a common problem and make the lives of software engineers easier. (Because their lives are really joyful, just sit behind a desk and stare at a monitor for hours! Can’t be easier than this?)

While architecting software, you need to split it to a number of components, and choose one or more tools to play in each part (if you were lucky to find one, unless you should write something new). You need to benchmark different tools and choose the best one that fits your project. Besides selecting the best fit, you need to decide about the communication protocol and how to integrate them afterward.

These problems also apply to Data projects. You use Kafka as a message broker, Cassandra as a NoSql database, Redshift as a warehouse, Elasticsearch as a search engine, and… then you need a tool to manage the flows among these technologies and integrate them together. There are some projects that address such problems, and Apache Nifi is one of them.

Wikipedia:_ Apache NiFi is a software project from the Apache Software Foundation designed to automate the flow of data between software systems._

#custom-processor #nifi #java