Directory and Files Management in Python

Directory and Files Management in Python

In this blog, we will learn about directory management in Python.

In this post, you'll learn about file and directory management in Python, i.e. creating a directory, renaming it, listing all directories and working with them.

Directory class

A directory is a collection of files and sub-directories. Python provides os modules for dealing with directories.

Some important directory method they are listed below:

  1. getcwd()
  2. chdir()
  3. listdir()
  4. path.exists()
  5. mkdir()
  6. rename()
  7. rmdir()
Current Directory

getcwd() method is used to get your current working directory path.

import os  
print(os.getcwd())  
Change Directory

Change your current working directory using chdir() method. It takes a full file path.

import os  
print(os.chdir('D:\\CCA'))  
print(os.getcwd())  
List Directories and files

List all files and sub-directories inside a directory using listdir() method. The listdir() method take a full directory path. If you don't provide directory path then it takes the current working directory.

import os  
print(os.listdir())   
print(os.listdir('D:\\CCA'))  
Check if a directory exists or not

The path.exists() method is used to check if a directory exists or not. If directory exists than returns TRUE else returns FALSE.

import os  
if os.path.exists('D:\\CCA\\Science'):  
print('Directory already exists.')  
else:  
print('Directory doesn\'t exists.') 
Create a new directory

You can create a new directory using the mkdir() method. This method takes the full path of the new directory, if you don't provide a full path than the new directory is created in the current working directory.

import os  
os.chdir('D:\\CCA') #Change the current path  
if os.path.exists('D:\\CCA\\Kumar'): #Check directory exists or not  
print('Directory already exists.')  
else:  
os.mkdir('Kumar') #directory creating  
print('Directory created successfully...')  
Rename a directory

You can rename a directory using rename() method. This method takes two arguments. The first argument is old directory name and the second argument is new directory name.

import os  
os.chdir('D:\\CCA')  
if os.path.exists('D:\\CCA\\Kumar'):  
os.rename('Kumar','Sarfaraj')  
print('Directory is renamed successfully...')  
else:  
print('Drectory doesn\'t exists.')
Delete a directory

The rmdir() method is used to delete an empty directory. It takes a full file path.

import os  
os.chdir('D:\\CCA')  
if os.path.exists('D:\\CCA\\Sarfaraj'):  
os.rmdir('Sarfaraj')  
print('Directory deteted successfully...')  
else:  
print('Drectory doesn\'t exists.')  
Summary

In this post, I covered some directory related methods in python and saw examples of how to use these methods. Thanks for reading.

File management with python

File management with python

File management with python

Most of us have had that one experience where we had a ton of dis-organized files in our machines. It happens. One minute, you're opening a large zip file, the next thing you know, the files are everywhere in the directory, mixing with all your important files and randomly placed leaving you with the task of manually sorting what needs to go where. It's a real pain. To ease this process, we're going to delve into file management with python the smart way.

Work smart, not hard.

Let's begin. We'll be using python 3.4 or greater. 

Assuming you've got python up and running already, we're going to take a walk with the OS module and a few others we will introduce along the way. Most of these come with python, so there's no need to install anything else to follow along.

Creating random files

Create a directory to work with. Call it ManageFiles. Inside this folder create another folder RandomFiles. Your directory structure should now look like this:

ManageFiles/
 |
 |_RandomFiles/

We're going to create random files to play with in the RandomFiles directory

Create a file createrandomfiles.py inside ManageFiles directory. You now have this:

ManageFiles/
 |
 |_ create_random_files.py
 |_RandomFiles/

Done? Now get in the following code, we'll get into its details in a moment.

import os
from pathlib import Path
import random

list_of_extensions = ['.rst','.txt','.md','.docx','.odt','.html','.ppt','.doc']

get into the RandomFiles directory

os.chdir('./RandomFiles')

for item in list_of_extensions:
# create 20 random files for each file extension
for num in range(20):
# let the file begin with a random number between 1 to 50
file_name = random.randint(1,50)
file_to_create = str(file_name) + item
Path(file_to_create).touch()

As of python 3.4, we 've got pathlib, our little magic box. We also import python's random function for creating random numbers; Hold on to that thought, we're going to cover it when as we get to the line that uses it.

First off, we create a list of file extensions from where we will get our random files. Feel free to add to it. 

Next up, we change to the RandomFiles directory, then comes our loop, so here goes.

We are simply saying, take each item in this list_of_extensions and do the following to it. Let's take the .txt for instance. We get into another loop, where to this .txt, we do something to it 20 times.

Remember our import of random? We use it to select a random number between 1 and 50 for our file. In short, what this little loop does is save us, the less creative lot(don't worry, I'm part of this crew), the time of naming random files. We will simply create a file say 23.txt or 14.txt, provided it falls within our range of 50, twenty times. This is just so as to create a mess large enough to give pain when moving manually. The same process will be done with the other extensions. Next? Run this in your terminal.

python create_random_files.py

Congratulations! We now have a mess of a directory. Now to clean it up.

In the same location where our create_random_files.py is, create a file clean_up.py and get the below in.

Method 1:

import os
import shutil
import glob

get into the RandomFiles directory

os.chdir('./RandomFiles')

get the list of files in the directory RandomFiles

files_to_group = []
for random_file in os.listdir('.'):
files_to_group.append(random_file)

get all the file extensions present

file_extensions = []
for our_file in files_to_group:
file_extensions.append(os.path.splitext(our_file)[1])

print(set(file_extensions))

file_types = set(file_extensions)

for type in file_types:
new_directory = type.replace(".", " ")
os.mkdir(new_directory) # create directory with given name

for fname in glob.glob(f'*.{type[1:]}'):
    shutil.move(fname, new_directory)

For this, we import two new libraries; shutil and glob. The shutil will help us move our files while the glob will help find the files to classify. Just like before, this will all become clear as we get to the line.

First off, we get a list of all the files in the directory.

Here, we assume that we do not have a clue of what files are in the directory. This means unlike where you can get all the extensions present manually and use if statements or switch, we want the program to look through the directory and do this for us. What if the file had dozens of extensions or log files? Would you do this manually?

Once we get a list of all the files in the folder, we get into another loop, to get the file extensions of these files.

Notice how we use:

os.path.splitext(our_file)[1]

Currently, the our_file variable looks something like this 5.docx (for instance). When we split it, we get this:

('5', '.docx')

we then get the index [1] from it which in turn takes .docx since 5 is index [0].

So we now have the list of all file extensions present in the folder, whether repeated or not.

To make it non-repetitive, we make a set. This takes all the items from the list and gets only the unique items. In our case, if we had a list where we had an extension say .docx repeating itself over and over in the set would ensure we had only one of it.

# create a set and assign it to a variable
file_types = set(file_extensions)

Remember our list of file types still has the . for every file extension. This would mean if we were to create a folder named exactly the same way, we would end up creating hidden folders and that is something we do not want.

So, as we loop over this set, we create a directory with the same extension name, only this time, we replace the . in the name with an empty string.

new_directory = type.replace(".", " ")

our directory would now be called 'docx'

We still need the .docx extension to move the files.

for fname in glob.glob(f'.{type[1:]}')

This simply implies take any file that ends with the .docx file extension(Notice the spacing used in f'.{type[1:]}').There is no space.

The wild card * means a file can be named anything, provided it ends in .docx. Since we have already placed the period . we take the string we have and have everything else afterwards and that's why we use [1:] which just means take from after the first character, hence take docx.

What next? Move any file with this extension into the directory named as so.

shutil.move(fname, new_directory)

In this way, once a directory for the first file found in the loop has been created, no other duplicates can be made. In short, we will not have a folder to store 5.docx and many others to store 34.docx and so on. Once we have a directory made, all other folders looking like so will move there. That's it!

Method 2

You can alternatively, use generators. This is a fancy way of creating a list with a one liner.

import os
import shutil
import glob

get into the RandomFiles directory

os.chdir('./RandomFiles')

#take every file from the directory and add to a list for all files
all_files = [x for x in os.listdir('.') ]

make a set for the extensions present in the directory

file_types = set((os.path.splitext(f)[1] for f in all_files))

for ftype in file_types:
new_directory = ftype.replace(".", '')
os.mkdir(new_directory)

for fname in glob.glob(f'*.{ftype[1:]}'):
    shutil.move(fname, new_directory)

Both of these will work. You've now got all your files sorted according to extension.

ManageFiles/
|
|_create_random_files.py
|_RandomFiles/
|_doc
|_docx
|_html
|_md
|_odt
|_ppt

Woosh! That was a lot.We did save some time though. Any questions? Feel free to reach out. That's it for now.Stick around as we take it up a notch next week.

Originally published by Marvin  at dev.to

=================================================================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter

Learn More

☞ Complete Python Bootcamp: Go from zero to hero in Python 3

☞ Python for Time Series Data Analysis

☞ Python Programming For Beginners From Scratch

☞ Python Network Programming | Network Apps & Hacking Tools

☞ Intro To SQLite Databases for Python Programming

☞ Ethical Hacking With Python, JavaScript and Kali Linux

☞ Beginner’s guide on Python: Learn python from scratch! (New)

☞ Python for Beginners: Complete Python Programming


Manage Files in Google Cloud Storage With Python

Manage Files in Google Cloud Storage With Python

Manage files in your Google Cloud Storage bucket using the google-cloud-storage Python library...

I recently worked on a project which combined two of my life's greatest passions: coding, and memes. The project was, of course, a chatbot: a fun imaginary friend who sits in your chatroom of choice and loyally waits on your beck and call, delivering memes whenever you might request them. In some cases, the bot would scrape the internet for freshly baked memes, but there were also plenty of instances where the desired memes should be more predictable, namely from a predetermined subset of memes hosted on the cloud which could be updated dynamically. This is where Google Cloud Storage comes in.

Google Cloud Storage is an excellent alternative to S3 for any GCP fanboys out there. Google Cloud provides a dead-simple way of interacting with Cloud Storage via the google-cloud-storage Python SDK: a Python library I've found myself preferring over the clunkier Boto3 library.

We've actually touched on google-cloud-storage briefly when we walked through interacting with BigQuery programmatically, but there's enough functionality available in this library to justify a post in itself.

Getting Set Up

Setting up a Google Cloud bucket is simple enough to skip the details, but there are a couple of things worth mentioning. First on our list: we need to set our bucket's permissions.

Setting Bucket-level Permissions

Making buckets publicly accessible is a big no-no in the vast majority of cases; we should never make a bucket containing sensitive information public (unless you're a contractor for the US government and you decide to store the personal information of all US voters in a public S3 bucket - that's apparently okay). Since I'm working with memes which I've stolen from other sources, I don't mind this bucket being publicly accessible.

Bucket-level permissions aren't enabled on new buckets by default (new buckets abide by object-level permissions). Changing this can be a bit tricky to find at first: we need to click into our bucket of choice and note the prompt at the top of the screen:

New buckets should prompt you for bucket-level permissions.


Clicking "enable" will open a side panel on the right-hand side of your screen. To enable publicly viewable files, we need to attach the Storage Object Viewer role to a keyword called allUsers (allUsers is a reserved type of "member" meaning "everybody in the entire world).

Finding Our Bucket Info

When we access our bucket programmatically, we'll need some information about our bucket like our bucket's URL (we need this to actually know where items in our bucket will be stored). General information about our bucket can be found under the "overview" tab, take this down:

To access the files we modify in our bucket, you'll need to know the URL.


Generating a Service Key

Finally, we need to generate a JSON service key to grant permissions to our script. Check out the credentials page in your GCP console and download a JSON file containing your creds. Please remember not to commit this anywhere.

Configuring our Script

Let's start coding, shall we? Make sure the google-cloud-storage library is installed on your machine with pip3 install google-cloud-storage.

I'm going to set up our project with a config.py file containing relevant information we'll need to work with:

"""Google Cloud Storage Configuration."""
from os import environ
Google Cloud Storage

bucketName = environ.get('GCP_BUCKET_NAME')
bucketFolder = environ.get('GCP_BUCKET_FOLDER_NAME')

Data

localFolder = environ.get('LOCAL_FOLDER')

  • bucketName is our bucket's given name. The google-cloud-storage interacts with buckets by looking for buckets which match a name in your GCP account.
  • bucketFolder is a folder within our bucket that we'll be working with.
  • localFolder is where I'm keeping a bunch of local files to test uploading and downloading to GCP.

With that done, we can start our script by importing these values:

"""Programatically interact with a Google Cloud Storage bucket."""
from google.cloud import storage
from config import bucketName, localFolder, bucketFolder

...

Managing Files in a GCP Bucket

Before we do anything, we need to create an object representing our bucket. I'm creating a global variable named bucket. This is created by calling the get_bucket() method on our storage client and passing the name of our bucket:

"""Programatically interact with a Google Cloud Storage bucket."""
from google.cloud import storage
from config import bucketName, localFolder, bucketFolder

storage_client = storage.Client()
bucket = storage_client.get_bucket(bucketName)

...

To demonstrate how to interact with Google Cloud Storage, we're going to create 5 different functions to handle common tasks: uploading, downloading, listing, deleting, and renaming files.

Upload Files

Our first function will look at a local folder on our machine and upload the contents of that folder:

from os import listdir
from os.path import isfile, join

...

def upload_files(bucketName):
"""Upload files to GCP bucket."""
files = [f for f in listdir(localFolder) if isfile(join(localFolder, f))]
for file in files:
localFile = localFolder + file
blob = bucket.blob(bucketFolder + file)
blob.upload_from_filename(localFile)
return f'Uploaded {files} to "{bucketName}" bucket.'

The first thing we do is fetch all the files we have living in our local folder using listdir(). We verify that each item we fetch is a file (not a folder) by using isfile().

We then loop through each file in our array of files. We set the desired destination of each file using bucket.blob(), which accepts the desired file path where our file will live once uploaded to GCP. We then upload the file with blob.upload_from_filename(localFile):

Uploaded ['sample_csv.csv', 'sample_text.txt', 'peas.jpg', 'sample_image.jpg'] to "hackers-data" bucket.

Listing Files

Knowing which files exist in our bucket is obviously important:

def list_files(bucketName):
"""List all files in GCP bucket."""
files = bucket.list_blobs(prefix=bucketFolder)
fileList = [file.name for file in files if '.' in file.name]
return fileList

list_blobs() gets us a list of files in our bucket. By default this will return all files; we can restrict the files we want to list to those in a bucket by specifying the prefix attribute.

['storage-tutorial/sample_csv.csv', 'storage-tutorial/sample_image.jpg', 'storage-tutorial/sample_text.txt', 'storage-tutorial/test.csv']

Looks like test.csv lives in our bucket, but not in our local folder!

Downloading Files

A feature of the chatbot I built was to fetch a randomized meme per meme keyword. Let’s see how’d we’d accomplish this:

from random import randint

...

def download_random_file(bucketName, bucketFolder, localFolder):
"""Download random file from GCP bucket."""
fileList = list_files(bucketName)
rand = randint(0, len(fileList) - 1)
blob = bucket.blob(fileList[rand])
fileName = blob.name.split('/')[-1]
blob.download_to_filename(localFolder + fileName)
return f'{fileName} downloaded from bucket.'

We leverage the list_files() function we already created to get a list of items in our bucket. We then select a random item by generating a random index using randint.

It's important to note here that .blob() returns a "blob" object as opposed to a string (inspecting our blob with type() results in <class 'google.cloud.storage.blob.Blob'>). This is why we see blob.name come into play when setting our blob's filename.

Finally, we download our target file with download_to_filename().

Deleting Files

Deleting a file is as simple as .delete_blob:

def delete_file(bucketName, bucketFolder, fileName):
"""Delete file from GCP bucket."""
bucket.delete_blob(bucketFolder + fileName)
return f'{fileName} deleted from bucket.'

Renaming Files

To rename a file, we pass a blob object to rename_blob() and set the new name via the new_name attribute:

def rename_file(bucketName, bucketFolder, fileName, newFileName):
"""Rename file in GCP bucket."""
blob = bucket.blob(bucketFolder + fileName)
bucket.rename_blob(blob,
new_name=newFileName)
return f'{fileName} renamed to {newFileName}.'
Managing Buckets

We can also use google-cloud-storage to interact with entire buckets:

  • create_bucket('my_bucket_name') creates a new bucket with the given name.
  • bucket.delete() deletes an existing bucket.

There are also ways to programmatically do things like access details about a bucket, or delete all the objects inside a bucket. Unfortunately, these actions are only supported by the REST API. I don't find these actions particularly useful anyway, so whatever.

The source code for this tutorial can be found here. That's all, folks!

Originally published by Todd Birchard at  hackersandslackers.com

===================================================================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter

Learn More

☞ Complete Python Bootcamp: Go from zero to hero in Python 3

☞ Python for Time Series Data Analysis

☞ Python Programming For Beginners From Scratch

☞ Python Network Programming | Network Apps & Hacking Tools

☞ Intro To SQLite Databases for Python Programming

☞ Ethical Hacking With Python, JavaScript and Kali Linux

☞ Beginner’s guide on Python: Learn python from scratch! (New)

☞ Python for Beginners: Complete Python Programming

Python GUI Programming Projects using Tkinter and Python 3

Python GUI Programming Projects using Tkinter and Python 3

Python GUI Programming Projects using Tkinter and Python 3

Description
Learn Hands-On Python Programming By Creating Projects, GUIs and Graphics

Python is a dynamic modern object -oriented programming language
It is easy to learn and can be used to do a lot of things both big and small
Python is what is referred to as a high level language
Python is used in the industry for things like embedded software, web development, desktop applications, and even mobile apps!
SQL-Lite allows your applications to become even more powerful by storing, retrieving, and filtering through large data sets easily
If you want to learn to code, Python GUIs are the best way to start!

I designed this programming course to be easily understood by absolute beginners and young people. We start with basic Python programming concepts. Reinforce the same by developing Project and GUIs.

Why Python?

The Python coding language integrates well with other platforms – and runs on virtually all modern devices. If you’re new to coding, you can easily learn the basics in this fast and powerful coding environment. If you have experience with other computer languages, you’ll find Python simple and straightforward. This OSI-approved open-source language allows free use and distribution – even commercial distribution.

When and how do I start a career as a Python programmer?

In an independent third party survey, it has been revealed that the Python programming language is currently the most popular language for data scientists worldwide. This claim is substantiated by the Institute of Electrical and Electronic Engineers, which tracks programming languages by popularity. According to them, Python is the second most popular programming language this year for development on the web after Java.

Python Job Profiles
Software Engineer
Research Analyst
Data Analyst
Data Scientist
Software Developer
Python Salary

The median total pay for Python jobs in California, United States is $74,410, for a professional with one year of experience
Below are graphs depicting average Python salary by city
The first chart depicts average salary for a Python professional with one year of experience and the second chart depicts the average salaries by years of experience
Who Uses Python?

This course gives you a solid set of skills in one of today’s top programming languages. Today’s biggest companies (and smartest startups) use Python, including Google, Facebook, Instagram, Amazon, IBM, and NASA. Python is increasingly being used for scientific computations and data analysis
Take this course today and learn the skills you need to rub shoulders with today’s tech industry giants. Have fun, create and control intriguing and interactive Python GUIs, and enjoy a bright future! Best of Luck
Who is the target audience?

Anyone who wants to learn to code
For Complete Programming Beginners
For People New to Python
This course was designed for students with little to no programming experience
People interested in building Projects
Anyone looking to start with Python GUI development
Basic knowledge
Access to a computer
Download Python (FREE)
Should have an interest in programming
Interest in learning Python programming
Install Python 3.6 on your computer
What will you learn
Build Python Graphical User Interfaces(GUI) with Tkinter
Be able to use the in-built Python modules for their own projects
Use programming fundamentals to build a calculator
Use advanced Python concepts to code
Build Your GUI in Python programming
Use programming fundamentals to build a Project
Signup Login & Registration Programs
Quizzes
Assignments
Job Interview Preparation Questions
& Much More