Is it possible to get Nifi to Put to multiple HDFS folders?

I need to stream a bunch of json files to Nifi, which will then go to HDFS. Nifi needs to look at the creation date (UNIX format) within the json file and then route it to the appropriate HDFS folder. So far I have the processors set up like this:

I need to stream a bunch of json files to Nifi, which will then go to HDFS. Nifi needs to look at the creation date (UNIX format) within the json file and then route it to the appropriate HDFS folder. So far I have the processors set up like this:

Consume Kafka -> RouteOnContent (using regex ^"creationDate": \"[0-9]{4}-[0-9]{2}-[0-9]{2}$) -> PutHDFS

There is an HDFS folder for every day, like "2019-01-28", "2019-01-29", "2019-01-30" etc. However, the "PutHDFS" processor will just output to a single directory and I obviously don't want to have 365 processors. And as far as I know, Nifi doesn't have a way to create HDFS folders dynamically so is there an elegant way to handle this?

Hadoop vs Spark | Hadoop MapReduce vs Spark

Hadoop vs Spark | Hadoop MapReduce vs Spark

🔥Intellipaat Big Data Hadoop Course: https://intellipaat.com/big-data-hadoop-training/ In this video on Hadoop vs Spark you will understand about the top Big...

In this video on Hadoop vs Spark you will understand about the top Big Data solutions used in the IT industry, and which one should you use for better performance. So in this Hadoop MapReduce vs Spark comparison some important parameters have been taken into consideration to tell you the difference between Hadoop and Spark also which one is preferred over the other in certain aspects in detail.

Why Hadoop is important

Big data hadoop is one of the best technological advances that is finding increased applications for big data and in a lot of industry domains. Data is being generated hugely in each and every industry domain and to process and distribute effectively hadoop is being deployed everywhere and in every industry.

Big Data & Hadoop Full Course - Hadoop Tutorial For Beginners

Big Data & Hadoop Full Course - Hadoop Tutorial For Beginners

The Big Data & Hadoop Full Course video will help you understand and learn Hadoop concepts in detail. This Big Data & Hadoop Tutorial is ideal for both beginners as well as professionals who want to master the Hadoop Ecosystem.

The Big Data & Hadoop Full Course video will help you understand and learn Hadoop concepts in detail. This Big Data & Hadoop Tutorial is ideal for both beginners as well as professionals who want to master the Hadoop Ecosystem.

Below are the topics covered in this Big Data & Hadoop Tutorial for Beginners video:

Thanks for reading

If you liked this post, share it with all of your programming buddies!

Follow us on Facebook | Twitter

Further reading

Hadoop Administration Tutorial for Beginners

Cheat Sheets for AI, Neural Networks, Machine Learning, Deep Learning & Big Data


Hadoop Administration Tutorial for Beginners

End to end Apache Hadoop Administration. In This i am going to explain apache hadoop administration. first we i will explain bigdata and hadoop introduction.

Next i will explain hadoop architecture and all hadoop deamons and next i will explain about hdfs filesystem

apache hadoop installation in 3 modes they are local mode / standalone mode, pSudo Distribution mode and Fully Distribution mode. i will set cluster on vmware workstation so that you can also replicate same way.


Learn More

The Ultimate Hands-On Hadoop - Tame your Big Data!

Hadoop MAPREDUCE in Depth | A Real-Time course on Mapreduce

Learn Big Data: The Hadoop Ecosystem Masterclass

SQL vs NoSQL or MySQL vs MongoDB

Building Web App using ASP.NET Web API Angular 7 and SQL Server

Build a Basic App with Spring Boot and JPA using PostgreSQL