Motivation

When putting your code into production, you will most likely need to deal with organizing the files of your code. It can be really time-consuming to read, create, and run many files of data. This article will show you how to automatically

• Loop through files in a directory
• Create nested files if they do not exist
• Run one file with different inputs using bash for loop

These tricks have saved me a lot of time while working on my data science projects. I hope you will find them useful as well!

Loop through Files in a Directory

If we have multiple data to read and process like this:

├── data
│   ├── data1.csv
│   ├── data2.csv
│   └── data3.csv
└── main.py

we can try to manually read one file at a time

import pandas as pd 

def process_data(df):
   pass
df = pd.read_csv(data1.csv)
process_data(df)
df2 = pd.read_csv(data2.csv)
process_data(df2)
df3 = pd.read_csv(data3.csv)
process_data(df3)

#automation #data-science #python #bash #programming