Top 20+ Challenging Python Projects for Experienced Developers (Code Include)
Expand your Python skills with these 20+ challenging projects for experienced developers (code include). This article provides a list of the top 20+ most challenging Python projects for experienced developers, along with code examples for all projects.
From building a web scraper to developing a machine learning model, these projects will challenge you to think outside the box and apply your Python skills to solve real-world problems.
What is an End to End Chatbot?
An end-to-end chatbot refers to a chatbot that can handle a complete conversation from start to finish without requiring human assistance. To create an end-to-end chatbot, you need to write a computer program that can understand user requests, generate appropriate responses, and take action when necessary. This involves collecting data, choosing a programming language and NLP tools, training the chatbot, and testing and refining it before making it available to users.
Once deployed, users can interact with the chatbot by sending it multiple requests and the chatbot can handle the entire conversation itself. To create an end-to-end chatbot using Python, we can follow the steps mentioned below:
- Define Intents
- Create training data
- Train the chatbot
- Build the chatbot
- Test the chatbot
- Deploy the chatbot
I hope you now have understood what an end-to-end chatbot is and the process of creating an end-to-end chatbot. In the section below, I’ll walk you through how to build an end-to-end chatbot using Python.
End to End Chatbot using Python
Now let’s start with creating an end-to-end chatbot using Python. I’ll start this task by importing the necessary Python libraries for this task:
import os
import nltk
import ssl
import streamlit as st
import random
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
ssl._create_default_https_context = ssl._create_unverified_context
nltk.data.path.append(osNow let’s define some intents of the chatbot. You can add more intents to make the chatbot more helpful and more functional:
intents = [
{
"tag": "greeting",
"patterns": ["Hi", "Hello", "Hey", "How are you", "What's up"],
"responses": ["Hi there", "Hello", "Hey", "I'm fine, thank you", "Nothing much"]
},
{
"tag": "goodbye",
"patterns": ["Bye", "See you later", "Goodbye", "Take care"],
"responses": ["Goodbye", "See you later", "Take care"]
},
{
"tag": "thanks",
"patterns": ["Thank you", "Thanks", "Thanks a lot", "I appreciate it"],
"responses": ["You're welcome", "No problem", "Glad I could help"]
},
{
"tag": "about",
"patterns": ["What can you do", "Who are you", "What are you", "What is your purpose"],
"responses": ["I am a chatbot", "My purpose is to assist you", "I can answer questions and provide assistance"]
},
{
"tag": "help",
"patterns": ["Help", "I need help", "Can you help me", "What should I do"],
"responses": ["Sure, what do you need help with?", "I'm here to help. What's the problem?", "How can I assist you?"]
},
{
"tag": "age",
"patterns": ["How old are you", "What's your age"],
"responses": ["I don't have an age. I'm a chatbot.", "I was just born in the digital world.", "Age is just a number for me."]
},
{
"tag": "weather",
"patterns": ["What's the weather like", "How's the weather today"],
"responses": ["I'm sorry, I cannot provide real-time weather information.", "You can check the weather on a weather app or website."]
},
{
"tag": "budget",
"patterns": ["How can I make a budget", "What's a good budgeting strategy", "How do I create a budget"],
"responses": ["To make a budget, start by tracking your income and expenses. Then, allocate your income towards essential expenses like rent, food, and bills. Next, allocate some of your income towards savings and debt repayment. Finally, allocate the remainder of your income towards discretionary expenses like entertainment and hobbies.", "A good budgeting strategy is to use the 50/30/20 rule. This means allocating 50% of your income towards essential expenses, 30% towards discretionary expenses, and 20% towards savings and debt repayment.", "To create a budget, start by setting financial goals for yourself. Then, track your income and expenses for a few months to get a sense of where your money is going. Next, create a budget by allocating your income towards essential expenses, savings and debt repayment, and discretionary expenses."]
},
{
"tag": "credit_score",
"patterns": ["What is a credit score", "How do I check my credit score", "How can I improve my credit score"],
"responseNow let’s prepare the intents and train a Machine Learning model for the chatbot:
# Create the vectorizer and classifier
vectorizer = TfidfVectorizer()
clf = LogisticRegression(random_state=0, max_iter=10000)
# Preprocess the data
tags = []
patterns = []
for intent in intents:
for pattern in intent['patterns']:
tags.append(intent['tag'])
patterns.append(pattern)
# training the model
x = vectorizer.fit_transform(patterns)
y = tags
clf.fit(x, y)Now let’s write a Python function to chat with the chatbot:
def chatbot(input_text):
input_text = vectorizer.transform([input_text])
tag = clf.predict(input_text)[0]
for intent in intents:
if intent['tag'] == tag:
response = random.choice(intent['responses'])
return responseTill now, we have created the chatbot. After running the code, you can interact with the chatbot in the terminal itself. To turn this chatbot into an end-to-end chatbot, we need to deploy it to interact with the chatbot using a user interface. To deploy the chatbot, I will use the streamlit library in Python, which provides amazing features to create a user interface for a Machine Learning application in just a few lines of code.
So, here’s how we can deploy the chatbot using Python:
counter = 0
def main():
global counter
st.title("Chatbot")
st.write("Welcome to the chatbot. Please type a message and press Enter to start the conversation.")
counter += 1
user_input = st.text_input("You:", key=f"user_input_{counter}")
if user_input:
response = chatbot(user_input)
st.text_area("Chatbot:", value=response, height=100, max_chars=None, key=f"chatbot_response_{counter}")
if response.lower() in ['goodbye', 'bye']:
st.write("Thank you for chatting with me. Have a great day!")
st.stop()
if __name__ == '__main__':
main()To run this chatbot, use the command mentioned below in your terminal:
- streamlit run filename.py
After executing the command, you will see a UI of your chatbot in your web browser as shown below:
Output Video
So, this is how you can create an end-to-end chatbot using the Python programming language.
Message Encryption using Python
Before implementing Message Encryption using Python, let’s look at the process of Message Encryption. Here’s the complete process of Message Encryption that messaging apps follow:
- First, choose a secret key that to encrypt the message. This key is a password only the person supposed to read the message knows.
- Then take the message and scramble it using the key. It means turning the message into a secret code like a jumble of letters or numbers.
- Once the message is scrambled, it is delivered to the person who is supposed to read it. But if anyone else tries to read it, they’ll see a jumble of letters or numbers that don’t make sense.
Now let’s get started with the task of Message Encryption using Python. For this task, you need to have a library installed in your system known as cryptography. You can easily install it by executing the command mentioned below in your terminal or command prompt:
- pip install cryptography
Now let’s first create a conversation between two people using a Python dictionary:
message_data = {
"Aman": [
{"message": "Hey Divyansha, how's it going?", "time": "2023-03-21 10:30:00"},
{"message": "Not too bad, just working on some coding projects. Did you hear about the new encryption algorithm?", "time": "2023-03-21 10:35:00"},
{"message": "It's called AES256 and it's supposed to be really secure. Want to give it a try with our messages?", "time": "2023-03-21 10:40:00"},
],
"Divyansha": [
{"message": "Good, thanks! How about you?", "time": "2023-03-21 10:32:00"},
{"message": "No, what's that?", "time": "2023-03-21 10:37:00"},
{"message": "Sure, let's do it!", "time": "2023-03-21 10:42:00"},
]
}Now let’s import the necessary modules and then generate a shared secret key for encryption and decryption:
import os
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
shared_secret_key = os.urandom(32)Now let’s define a function to keep the secret message safe from people who shouldn’t see it:
def encrypt_message(message, key):
iv = os.urandom(16)
cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=default_backend())
encryptor = cipher.encryptor()
padded_message = message + (16 - len(message) % 16) * chr(16 - len(message) % 16)
ciphertext = encryptor.update(padded_message.encode()) + encryptor.finalize()
return iv + ciphertextNow let’s define a function to unscramble the secret message made using a key and encryption:
def decrypt_message(ciphertext, key):
iv = ciphertext[:16]
cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=default_backend())
decryptor = cipher.decryptor()
plaintext = decryptor.update(ciphertext[16:]) + decryptor.finalize()
padding_length = plaintext[-1]
plaintext = plaintext[:-padding_length]
return plaintext.decode()Now here’s how we can encrypt the dictionary with the scrambled messages so that only the people who know the key can unscramble and read the messages:
for person, messages in message_data.items():
for message in messages:
encrypted_message = encrypt_message(message["message"], shared_secret_key)
message["message"] = encrypted_message.hex()
print("Encrypted message_data dictionary:")
print(message_data)Encrypted message_data dictionary:
{'Aman': [{'message': '67e03426da00a7fcc69317820619c7868bb95f6787882b9441db7f89f89367af92bd2ab7386ddb06a9c6036ed764f8b1', 'time': '2023-03-21 10:30:00'}, {'message': 'af78f1cbbbd8103c8db21e042d1c36e8167695a3aa4b2acc3ac5c788abb89260ac6b990bf8344d897a623b8fb556ba4ce04e3255381856815c651b1d93efc0e0424ee1495ab1a72e2a20948c21d51ddbec3e66a493668ad60a9840e436beceb5c34786e05a4f9f71fb5ed2a70ec838994fa545d7c03da15d9850fb5061259df7', 'time': '2023-03-21 10:35:00'}, {'message': '91a5e6eb65ba1550f00dc4b05649d43a3f6792b0c2682e72f09f690f13f0d176ddfb0a6f46abfcd9065735b4cb611e0a3d2119bf0f9b0b6f2ac4501cafc40d71e89dabcd41f5f5b2e544bcbd7556890307a2201e91f70c6e20742bc956886b98ec929eaded5c16036e1449aafb3024ca5dad5dd29b1148e9535586406f8422c7', 'time': '2023-03-21 10:40:00'}], 'Divyansha': [{'message': 'e3ba528d38fe41424434cfd610d73647628867fc059214bda2c177c85b004bb675600b101cf32b4e302264d550091642', 'time': '2023-03-21 10:32:00'}, {'message': '99f1c86dfa530d59b404cc2cf62d278a6124274a365b60397aacb0f82c1b76c596b132678df76943ae31a14a230ea7c4', 'time': '2023-03-21 10:37:00'}, {'message': '6b51e93dcf9e524e94563a5d3b9224b9e30007cbfb61056d2b6a641dee5d3df156e092380f8f8f00f87387ae75c7049c', 'time': '2023-03-21 10:42:00'}]}Now here’s how to decrypt the encrypted messages so that the people having the key can read the messages:
for person, messages in message_data.items():
for message in messages:
ciphertext = bytes.fromhex(message["message"])
decrypted_message = decrypt_message(ciphertext, shared_secret_key)
message["message"] = decrypted_message
print("Decrypted message_data dictionary:")
print(message_data)Decrypted message_data dictionary:
{'Aman': [{'message': "Hey Divyansha, how's it going?", 'time': '2023-03-21 10:30:00'}, {'message': 'Not too bad, just working on some coding projects. Did you hear about the new encryption algorithm?', 'time': '2023-03-21 10:35:00'}, {'message': "It's called AES256 and it's supposed to be really secure. Want to give it a try with our messages?", 'time': '2023-03-21 10:40:00'}], 'Divyansha': [{'message': 'Good, thanks! How about you?', 'time': '2023-03-21 10:32:00'}, {'message': "No, what's that?", 'time': '2023-03-21 10:37:00'}, {'message': "Sure, let's do it!", 'time': '2023-03-21 10:42:00'}]}So this is how we can encrypt and decrypt messages using the Python programming language.
Calculating Distance Between two Locations using Python
We can use the haversine formula to calculate the distance between two locations. The haversine formula calculates the distance between two latitude and longitude points. This formula is defined as:
haversine(d/R) = haversine(latitude2- latitude1 + cos(latitude1 * cos(latitude2 * haversine(longitude2 – longitude1)
In this formula:
- d is the distance between the two points
- r is the radius of the earth
- and haversine is the haversine function defined as:
- haversine(theta) = sin^2(theta/2)
To calculate the final distance in kilometres, you need to multiply the distance between the two points by the radius of the earth.
I hope you have understood how the haversine formula can be used to calculate the distance between two latitude and longitude points. Now, below is how we can implement the haversine formula to calculate the distance between two latitude and longitude points using Python:
import numpy as np
# Set the earth's radius (in kilometers)
r = 6371
# Convert degrees to radians
def deg_to_rad(degrees):
return degrees * (np.pi/180)
# Function to calculate the distance between two points
# using the haversine formula
def distcalculate(lat1, lon1, lat2, lon2):
d_lat = deg_to_rad(lat2-lat1)
d_lon = deg_to_rad(lon2-lon1)
a = np.sin(d_lat/2)**2 + np.cos(deg_to_rad(lat1)) * np.cos(deg_to_rad(lat2)) * np.sin(d_lon/2)**2
c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1-a))
return r Now let’s calculate the distance between two locations with the function we just defined above:
print(distcalculate(22.745049, 75.892471, 22.765049, 75.912471))Output: 3.0251492856773803The above output shows the distance between two locations according to their latitude and longitude points in kilometres.
Netflix Recommendation System using Python
The dataset I am using to build a Netflix recommendation system using Python is downloaded from Kaggle. The dataset contains information about all the movies and TV shows on Netflix as of 2021. You can download the dataset from here.
Now let’s import the necessary Python libraries and the dataset we need for this task:
import numpy as np
import pandas as pd
from sklearn.feature_extraction import text
from sklearn.metrics.pairwise import cosine_similarity
data = pd.read_csv("netflixData.csv")
print(data.head()) Show Id Title \
0 cc1b6ed9-cf9e-4057-8303-34577fb54477 (Un)Well
1 e2ef4e91-fb25-42ab-b485-be8e3b23dedb #Alive
2 b01b73b7-81f6-47a7-86d8-acb63080d525 #AnneFrank - Parallel Stories
3 b6611af0-f53c-4a08-9ffa-9716dc57eb9c #blackAF
4 7f2d4170-bab8-4d75-adc2-197f7124c070 #cats_the_mewvie
Description \
0 This docuseries takes a deep dive into the luc...
1 As a grisly virus rampages a city, a lone man ...
2 Through her diary, Anne Frank's story is retol...
3 Kenya Barris and his family navigate relations...
4 This pawesome documentary explores how our fel...
Director \
0 NaN
1 Cho Il
2 Sabina Fedeli, Anna Migotto
3 NaN
4 Michael Margolis
Genres \
0 Reality TV
1 Horror Movies, International Movies, Thrillers
2 Documentaries, International Movies
3 TV Comedies
4 Documentaries, International Movies
Cast Production Country \
0 NaN United States
1 Yoo Ah-in, Park Shin-hye South Korea
2 Helen Mirren, Gengher Gatti Italy
3 Kenya Barris, Rashida Jones, Iman Benson, Genn... United States
4 NaN Canada
Release Date Rating Duration Imdb Score Content Type Date Added
0 2020.0 TV-MA 1 Season 6.6/10 TV Show NaN
1 2020.0 TV-MA 99 min 6.2/10 Movie September 8, 2020
2 2019.0 TV-14 95 min 6.4/10 Movie July 1, 2020
3 2020.0 TV-MA 1 Season 6.6/10 TV Show NaN
4 2020.0 TV-14 90 min 5.1/10 Movie February 5, 2020 In the first impressions on the dataset, I can see that the Title column needs preparation as it contains # before the name of the movies or tv shows. I will get back to it. For now, let’s have a look at whether the data contains null values or not:
print(data.isnull().sum())Show Id 0
Title 0
Description 0
Director 2064
Genres 0
Cast 530
Production Country 559
Release Date 3
Rating 4
Duration 3
Imdb Score 608
Content Type 0
Date Added 1335
dtype: int64The dataset contains null values, but before removing the null values, let’s select the columns that we can use to build a Netflix recommendation system:
data = data[["Title", "Description", "Content Type", "Genres"]]
print(data.head()) Title \
0 (Un)Well
1 #Alive
2 #AnneFrank - Parallel Stories
3 #blackAF
4 #cats_the_mewvie
Description Content Type \
0 This docuseries takes a deep dive into the luc... TV Show
1 As a grisly virus rampages a city, a lone man ... Movie
2 Through her diary, Anne Frank's story is retol... Movie
3 Kenya Barris and his family navigate relations... TV Show
4 This pawesome documentary explores how our fel... Movie
Genres
0 Reality TV
1 Horror Movies, International Movies, Thrillers
2 Documentaries, International Movies
3 TV Comedies
4 Documentaries, International Movies As the name suggests:
- The title column contains the titles of movies and TV shows on Netflix
- Description column describes the plot of the TV shows and movies
- The Content Type column tells us if it’s a movie or a TV show
- The Genre column contains all the genres of the TV show or the movie
Now let’s drop the rows containing null values and move further:
data = data.dropna()Now I will clean the Title column as it contains some data preparation:
import nltk
import re
nltk.download('stopwords')
stemmer = nltk.SnowballStemmer("english")
from nltk.corpus import stopwords
import string
stopword=set(stopwords.words('english'))
def clean(text):
text = str(text).lower()
text = re.sub('\[.*?\]', '', text)
text = re.sub('https?://\S+|www\.\S+', '', text)
text = re.sub('<.*?>+', '', text)
text = re.sub('[%s]' % re.escape(string.punctuation), '', text)
text = re.sub('\n', '', text)
text = re.sub('\w*\d\w*', '', text)
text = [word for word in text.split(' ') if word not in stopword]
text=" ".join(text)
text = [stemmer.stem(word) for word in text.split(' ')]
text=" ".join(text)
return text
data["Title"] = data["Title"].apply(clean)Now let’s have a look at some samples of the Titles before moving forward:
print(data.Title.sample(10))3111 miniforc super dino power
1822 girl reveng
910 casino tycoon
4075 sand castl
2760 lock
3406 nightflyer
536 bangkok love stori object affect
4365 special
1733 full
2343 jeff dunham map
Name: Title, dtype: objectNow I will use the Genres column as the feature to recommend similar content to the user. I will use the concept of cosine similarity here (used to find similarities in two documents):
feature = data["Genres"].tolist()
tfidf = text.TfidfVectorizer(input=feature, stop_words="english")
tfidf_matrix = tfidf.fit_transform(feature)
similarity = cosine_similarity(tfidf_matrix)Now I will set the Title column as an index so that we can find similar content by giving the title of the movie or TV show as an input:
indices = pd.Series(data.index,
index=data['Title']).drop_duplicates()Now here’s how to write a function to recommend Movies and TV shows on Netflix:
def netFlix_recommendation(title, similarity = similarity):
index = indices[title]
similarity_scores = list(enumerate(similarity[index]))
similarity_scores = sorted(similarity_scores, key=lambda x: x[1], reverse=True)
similarity_scores = similarity_scores[0:10]
movieindices = [i[0] for i in similarity_scores]
return data['Title'].iloc[movieindices]
print(netFlix_recommendation("girlfriend"))
def netFlix_recommendation(title, similarity = similarity):
index = indices[title]
similarity_scores = list(enumerate(similarity[index]))
similarity_scores = sorted(similarity_scores, key=lambda x: x[1], reverse=True)
similarity_scores = similarity_scores[0:10]
movieindices = [i[0] for i in similarity_scores]
return data['Title'].iloc[movieindices]
print(netFlix_recommendation("girlfriend"))So this is how you can build a Netflix Recommendation System using the Python programming language.
Time Series Graph using Python
To visualize a time series graph using Python, I will be using a stock price dataset. There are many libraries in Python for data visualization; I will be using Plotly as it is easy to visualize interactive visualizations using plotly. So let’s start this task by collecting the latest stock price data of Apple:
import pandas as pd
import yfinance as yf
import datetime
from datetime import date, timedelta
today = date.today()
d1 = today.strftime("%Y-%m-%d")
end_date = d1
d2 = date.today() - timedelta(days=360)
d2 = d2.strftime("%Y-%m-%d")
start_date = d2
data = yf.download('AAPL',
start=start_date,
end=end_date,
progress=False)
print(data.head()) Open High ... Adj Close Volume
Date ...
2021-01-19 127.779999 128.710007 ... 127.046783 90757300
2021-01-20 128.660004 132.490005 ... 131.221054 104319500
2021-01-21 133.800003 139.669998 ... 136.031372 120150900
2021-01-22 136.279999 139.850006 ... 138.217926 114459400
2021-01-25 143.070007 145.089996 ... 142.044327 157611700
[5 rows x 6 columns]The above data is collected by using the yfinance API. Now below is how you can visualize a time series graph using Python:
import plotly.express as px
figure = px.line(data, x = data.index, y = "Close")
figure.show()
Get Stock Price Data using Python
Yahoo Finance is one of the most popular websites to collect stock price data. You need to visit the website, enter the company’s name or stock symbol, and you can easily download the dataset. But if you want to get the latest dataset every time you are running your code, you need to use the yfinance API. yfinance is an API provided by Yahoo Finance to collect the latest stock price data.
To use this API, you need to install it by using the pip command in your terminal or command prompt as mentioned below:
- pip install yfinance
I hope you have easily installed this API. Now below is how you can get the latest stock price data using Python:
import pandas as pd
import yfinance as yf
import datetime
from datetime import date, timedelta
today = date.today()
d1 = today.strftime("%Y-%m-%d")
end_date = d1
d2 = date.today() - timedelta(days=360)
d2 = d2.strftime("%Y-%m-%d")
start_date = d2
data = yf.download('AAPL',
start=start_date,
end=end_date,
progress=False)
print(data.head()) Open High ... Adj Close Volume
Date ...
2020-12-28 133.990005 137.339996 ... 135.852509 124486200
2020-12-29 138.050003 138.789993 ... 134.043640 121047300
2020-12-30 135.580002 135.990005 ... 132.900681 96452100
2020-12-31 134.080002 134.740005 ... 131.876999 99116600
2021-01-04 133.520004 133.610001 ... 128.617111 143301900
[5 rows x 6 columns]The above code will collect the stock price data from today to the last 360 days. In this dataset, Date is not a column, it’s the index of this dataset. To use this data for any data science task, we need to convert this index into a column. Below is how you can do that:
data["Date"] = data.index
data = data[["Date", "Open", "High",
"Low", "Close", "Adj Close", "Volume"]]
data.reset_index(drop=True, inplace=True)
print(data.head()) Date Open High ... Close Adj Close Volume
0 2020-12-28 133.990005 137.339996 ... 136.690002 135.852524 124486200
1 2020-12-29 138.050003 138.789993 ... 134.869995 134.043640 121047300
2 2020-12-30 135.580002 135.990005 ... 133.720001 132.900696 96452100
3 2020-12-31 134.080002 134.740005 ... 132.690002 131.876999 99116600
4 2021-01-04 133.520004 133.610001 ... 129.410004 128.617096 143301900
[5 rows x 7 columns]So as you can see, the final dataset is just like the dataset that we download from Yahoo Finance. This is how we can get stock price data using Python.
Candlestick Chart using Python
To visualize a candlestick chart using the Python programming language, you can use any data visualization library in Python like Matplotlib or Plotly. The plotly library provides better features for visualizing a candlestick chart and also makes it interactive at the same time. So in this article, I will be using the plotly library for visualizing a candlestick chart.
As this chart is mainly used to analyze the price movements of financial instruments, so I will be using the Apple stock price data for this task. I have collected data on Apple’s stock price for the past three months from Yahoo Finance. You can either collect a dataset from Yahoo Finance or download the same dataset I’m using for this task from here.
So below is how we can visualize a candlestick chart using Python:
import pandas as pd
data = pd.read_csv("AAPL.csv")
import plotly.graph_objects as go
figure = go.Figure(data=[go.Candlestick(x=data["Date"],
open=data["Open"],
high=data["High"],
low=data["Low"],
close=data["Close"])])
figure.update_layout(title = "Apple Stock Price Analysis",
xaxis_rangeslider_visible=False)
figure.show()
Word Cloud from a Pandas DataFrame in Python
A pandas DataFrame is used to store the data that you use when working on a data science task. Sometimes your dataset contains a column with textual information such as opinions or reviews of people about a product. To understand how most people think about the product, you can visualize a word cloud of that column. So to view a word cloud from a pandas DataFrame in Python, you need to have the wordcloud library installed in your Python environment. You can install this Python library using the pip command as mentioned below:
- pip install wordcloud
Now let’s see how to visualize a word cloud from a pandas DataFrame in Python. For this task, I will first import all the necessary Python libraries and a dataset with textual information:
from wordcloud import WordCloud
from wordcloud import ImageColorGenerator
from wordcloud import STOPWORDS
import matplotlib.pyplot as plt
import pandas as pd
data = pd.read_csv("https://raw.githubusercontent.com/amankharwal/Website-data/master/spam.csv")
print(data.head()) label text
0 ham Go until jurong point, crazy.. Available only ...
1 ham Ok lar... Joking wif u oni...
2 spam Free entry in 2 a wkly comp to win FA Cup fina...
3 ham U dun say so early hor... U c already then say...
4 ham Nah I don't think he goes to usf, he lives aro...There are only two columns in this dataset where the text column contains textual data. So below is how you can visualize a word cloud from the text column of this dataset using Python:
text = " ".join(i for i in data.text)
stopwords = set(STOPWORDS)
wordcloud = WordCloud(stopwords=stopwords, background_color="white").generate(text)
plt.figure( figsize=(15,10))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()
So this is how you can easily visualize a word cloud from any column of your dataset using Python.
Calculation of Accuracy using Python
For the calculation of the accuracy of a classification model, we must first train a model for any classification-based problem. So here’s how we can easily train a classification-based machine learning model:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
nb_samples = 1000
x, y = make_classification(n_samples=nb_samples, n_features=2, n_informative=2, n_redundant=0, n_clusters_per_class=1)
xtrain, xtest, ytrain, ytest = train_test_split(x, y, test_size=0.2, random_state=42)
model = LogisticRegression()
model.fit(xtrain, ytrain)Now here is how we can calculate the accuracy of our trained model:
print(accuracy_score(ytest, model.predict(xtest)))0.99Many people often confuse accuracy and precision(another classification metric) with each other, accuracy is how close the predicted values are to the expected value, while precision is how close the predicted values are with each other.
How to Get Live Weather Updates using Python?
There are many ways to get live weather conditions for any city using the Python programming language. You can either use the weather APIs provided by third-party apps or you can also scrape the live weather data from a particular city.
If you are using weather APIs provided by any platform, you need to connect to their paid services to work with the weather APIs. But if you use your web scraping skills, it will be free. So in the section below, I’m going to walk you through how to get live weather updates from any city using Python.
Live Weather Updates using Python
To get the live weather conditions of any city using Python, you first have to install the BeautifulSoup library in Python. You can easily install this Python library in your system by using the pip command:
- pip install beautifulsoup4
After installing this library, you can now start with this task by importing the necessary Python libraries that we need for this task. After importing the libraries we need to scrape the weather updates of a city by searching about it on Google. Below is how we can do it using Python:
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
def weather(city):
city=city.replace(" ","+")
res = requests.get(f'https://www.google.com/search?q={city}&oq={city}&aqs=chrome.0.35i39l2j0l4j46j69i60.6128j1j7&sourceid=chrome&ie=UTF-8',headers=headers)
print("Searching......\n")
soup = BeautifulSoup(res.text,'html.parser')
location = soup.select('#wob_loc')[0].getText().strip()
time = soup.select('#wob_dts')[0].getText().strip()
info = soup.select('#wob_dc')[0].getText().strip()
weather = soup.select('#wob_tm')[0].getText().strip()
print(location)
print(time)
print(info)
print(weather+"°C")
city=input("Enter the Name of Any City >> ")
city=city+" weather"
weather(city)In the above code, we are searching for a city on Google, and then we are scraping the data about that particular city concerning location, time, info, and weather. After running the above Python code you will see an output as shown below.
Enter the Name of Any City >> New Delhi
Searching......
New Delhi, Delhi
Friday, 4:00 pm
Haze
33°CSteps to Create an OTP Verification System using Python
OTP Verification is the process of verifying a user by sending a unique password so that the user can be verified before completing a registration or payment process. Most of the time, we get an OTP when we make an online payment, or when we forget our password, or when creating an account on any online platform. Thus, the sole purpose of an OTP is to verify the identity of a user by sending a unique password.
We can easily create an application for the task of OTP verification using Python by following the steps mentioned below:
- First, create a 6-digit random number
- Then store the number in a variable
- Then we need to write a program to send emails
- When sending email, we need to use OTP as a message
- Finally, we need to request two user inputs; first for the user’s email and then for the OTP that the user has received.
So this is the complete process of creating an OTP verification application using Python. In the section below, I will take you through how to implement these steps using Python for the task of OTP verification.
OTP Verification using Python
I hope you now have understood what is an OTP and how we can create an application for the task of OTP verification. Now let’s follow the steps mentioned above by using Python to create an application for the task of OTP verification. I will start by importing the necessary Python library that we need for this task:
import os
import math
import random
import smtplibNow I will generate a random number and store it in a variable which I will be using while sending emails to the users:
digits="0123456789"
OTP=""
for i in range(6):
OTP+=digits[math.floor(random.random()*10)]
otp = OTP + " is your OTP"
msg= otpNow, before we go ahead, you need to have your Google app password to be able to send emails using your Gmail account. For this task, you need to follow the steps mentioned here. After you create your app password for your Gmail account you will get a key. Copy that key and paste in the code below to send emails for OTP verification using Python:
s = smtplib.SMTP('smtp.gmail.com', 587)
s.starttls()
s.login("Your Gmail Account", "You app password")
emailid = input("Enter your email: ")
s.sendmail('&&&&&&&&&&&',emailid,msg)
a = input("Enter Your OTP >>: ")
if a == OTP:
print("Verified")
else:
print("Please Check your OTP again")Once you run this code you enter an email where you want to send an OTP and then enter the OTP that you have received in the email. You can get the complete code used in this article for the task of OTP verification from below.
import os
import math
import random
import smtplib
digits="0123456789"
OTP=""
for i in range(6):
OTP+=digits[math.floor(random.random()*10)]
otp = OTP + " is your OTP"
msg= otp
s = smtplib.SMTP('smtp.gmail.com', 587)
s.starttls()
s.login("Your Gmail Account", "You app password")
emailid = input("Enter your email: ")
s.sendmail('&&&&&&&&&&&',emailid,msg)
a = input("Enter Your OTP >>: ")
if a == OTP:
print("Verified")
else:
print("Please Check your OTP again")What is a Chatbot?
A Chatbot is an AI application that mimics human conversations. It is widely used by companies to solve the most common problems they receive from customers daily. For example, if you want to know the CRN of your bank account, a chatbot will assist you by asking for your bank details, then it will give you your CRN. But if you want to know something that is not that common, like asking how you can turn your account into a joint account, chances are the authorized employee will assist you.
So while creating a chatbot for any company you should know what that company deals in and what problems their customers get daily. In the section below, I will take you through how to create a chatbot using Python.
Chatbot with Python
I hope you now have understood what are chatbots and why so many companies use them to solve the most common problems of their customers. Now let’s see how to create a chatbot with Python. Here, I will be using the NLTK library in Python which is one of the best Python libraries for any task of natural language processing:
from nltk.chat.util import Chat, reflectionsNow I will be creating a list of queries and their responses for the chatbot:
#Pairs is a list of patterns and responses.
pairs = [
[
r"(.*)my name is (.*)",
["Hello %2, How are you today ?",]
],
[
r"(.*)help(.*) ",
["I can help you ",]
],
[
r"(.*) your name ?",
["My name is thecleverprogrammer, but you can just call me robot and I'm a chatbot .",]
],
[
r"how are you (.*) ?",
["I'm doing very well", "i am great !"]
],
[
r"sorry (.*)",
["Its alright","Its OK, never mind that",]
],
[
r"i'm (.*) (good|well|okay|ok)",
["Nice to hear that","Alright, great !",]
],
[
r"(hi|hey|hello|hola|holla)(.*)",
["Hello", "Hey there",]
],
[
r"what (.*) want ?",
["Make me an offer I can't refuse",]
],
[
r"(.*)created(.*)",
["Aman Kharwal created me using Python's NLTK library ","top secret ;)",]
],
[
r"(.*) (location|city) ?",
['New Delhi, India',]
],
[
r"(.*)raining in (.*)",
["No rain in the past 4 days here in %2","In %2 there is a 50% chance of rain",]
],
[
r"how (.*) health (.*)",
["Health is very important, but I am a computer, so I don't need to worry about my health ",]
],
[
r"(.*)(sports|game|sport)(.*)",
["I'm a very big fan of Cricket",]
],
[
r"who (.*) (Cricketer|Batsman)?",
["Virat Kohli"]
],
[
r"quit",
["Bye for now. See you soon :) ","It was nice talking to you. See you soon :)"]
],
[
r"(.*)",
['That is nice to hear']
],
]Understanding the Hangman Game
In the hangman game, one player thinks of a word and the other players have to guess the word letter by letter. Whenever the guessed letter is correct, the player who decided on the word will indicate the position of the letter.
To build the Hangman hame with the Python programming language, you need to use basic Python concepts like the random module, integers, strings, characters, input and output operations, and Boolean values.
Hangman Game with Python
The logic of creating the Hangman game with python is that we will have users who will guess a letter and all users will have a very limited number of guesses.
So to code this game with Python, you can create a wordlist from which users can think of the words. In the process, we will also need some helper functions to check if the user entered a single letter or if the letter entered by the user is in the hidden word.
To create the hangman game, I will start by making a list of secret words, then start choosing words at random. In the process, I’ll just represent each word as “_” and whenever the user guesses the correct word, I’ll replace the “_” with the correct word. Now let’s follow this logic to create the hangman game with Python:
import time
import random
name = input("What is your name? ")
print ("Hello, " + name, "Time to play hangman!")
time.sleep(1)
print ("Start guessing...\n")
time.sleep(0.5)
## A List Of Secret Words
words = ['python','programming','treasure','creative','medium','horror']
word = random.choice(words)
guesses = ''
turns = 5
while turns > 0:
failed = 0
for char in word:
if char in guesses:
print (char,end="")
else:
print ("_",end=""),
failed += 1
if failed == 0:
print ("\nYou won")
break
guess = input("\nguess a character:")
guesses += guess
if guess not in word:
turns -= 1
print("\nWrong")
print("\nYou have", + turns, 'more guesses')
if turns == 0:
print ("\nYou Lose") Output:
What is your name? Aman Kharwal
Hello, Aman Kharwal Time to play hangman!
Start guessing…
guess a character:A
Wrong
You have 4 more guesses
m____m
guess a character:e
me___m
guess a character:d
med__m
guess a character:i
medi_m
guess a character:u
medium
You wonHow To Create a URL Shortener with Python?
You must have used various online URL shortening services and they all are doing a great job as well! Even Google forms also use shorten URLs for ease of use. So it’s a widely used service on the Internet.
Have you ever thought about or tried to shorten the length of the URL? Hopefully, there are plenty of libraries and APIs available in the Python programming language to help us do the same using programming without having to visit a website and use anyone’s service. In the section below you will learn how to shorten URLs by using the Python programming language.
URL Shortener with Python
Now let’s see how to write a program to create an application to shorten URLs with Python. We need to write Python code where we can give long URL as an input and we can get short URLs as output, that too in very few lines of code.
It might seem like a difficult task, but using various libraries in Python does it very easily without having to delve into complex topics. Now let’s see how to write a program to create URL Shortener with Python:
from __future__ import with_statement
import contextlib
try:
from urllib.parse import urlencode
except ImportError:
from urllib import urlencode
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
import sys
def make_tiny(url):
request_url = ('http://tinyurl.com/api-create.php?' +
urlencode({'url':url}))
with contextlib.closing(urlopen(request_url)) as response:
return response.read().decode('utf-8')
def main():
for tinyurl in map(make_tiny, sys.argv[1:]):
print(tinyurl)
if __name__ == '__main__':
main()How Does a Desktop Notification App Work?
The main purpose of the desktop notification app that you will learn to develop today is to constantly remind us of the different things that we need to accomplish throughout the day.
This task is similar to a to-do list, where we have a set of goals to achieve. And the desktop notification app will constantly notify us of the different to-do and actions to take throughout the day.
Desktop Notification with Python
I am going to create a desktop notification app to get a reminder to rest after every hour. Your message and alert can be absolutely anything you want. You can have a list of things you need to do in the day, week or month, and the reminder app will constantly remind you of the same.
For this task you need to install a Python library known as Plyer, which is used to access the hardware components of your system. This library can be easily installed by using the pip command; pip install pyler.
Now let’s see how to write a Python program to get desktop notifications:
import time
from plyer import notification
if __name__ == "__main__":
while True:
notification.notify(
title = "ALERT!!!",
message = "Take a break! It has been an hour!",
timeout = 10
)
time.sleep(360Tic Tac Toe GUI with Python
There are two basic logics in this game; when both players are human, and when one is a computer. I will prepare this Tic Tac Toe GUI with Python for two players. Here is your full code:
from tkinter import *
import numpy as np
size_of_board = 600
symbol_size = (size_of_board / 3 - size_of_board / 8) / 2
symbol_thickness = 50
symbol_X_color = '#EE4035'
symbol_O_color = '#0492CF'
Green_color = '#7BC043'
class Tic_Tac_Toe():
# ------------------------------------------------------------------
# Initialization Functions:
# ------------------------------------------------------------------
def __init__(self):
self.window = Tk()
self.window.title('Tic-Tac-Toe')
self.canvas = Canvas(self.window, width=size_of_board, height=size_of_board)
self.canvas.pack()
# Input from user in form of clicks
self.window.bind('<Button-1>', self.click)
self.initialize_board()
self.player_X_turns = True
self.board_status = np.zeros(shape=(3, 3))
self.player_X_starts = True
self.reset_board = False
self.gameover = False
self.tie = False
self.X_wins = False
self.O_wins = False
self.X_score = 0
self.O_score = 0
self.tie_score = 0
def mainloop(self):
self.window.mainloop()
def initialize_board(self):
for i in range(2):
self.canvas.create_line((i + 1) * size_of_board / 3, 0, (i + 1) * size_of_board / 3, size_of_board)
for i in range(2):
self.canvas.create_line(0, (i + 1) * size_of_board / 3, size_of_board, (i + 1) * size_of_board / 3)
def play_again(self):
self.initialize_board()
self.player_X_starts = not self.player_X_starts
self.player_X_turns = self.player_X_starts
self.board_status = np.zeros(shape=(3, 3))
# ------------------------------------------------------------------
# Drawing Functions:
# The modules required to draw required game based object on canvas
# ------------------------------------------------------------------
def draw_O(self, logical_position):
logical_position = np.array(logical_position)
# logical_position = grid value on the board
# grid_position = actual pixel values of the center of the grid
grid_position = self.convert_logical_to_grid_position(logical_position)
self.canvas.create_oval(grid_position[0] - symbol_size, grid_position[1] - symbol_size,
grid_position[0] + symbol_size, grid_position[1] + symbol_size, width=symbol_thickness,
outline=symbol_O_color)
def draw_X(self, logical_position):
grid_position = self.convert_logical_to_grid_position(logical_position)
self.canvas.create_line(grid_position[0] - symbol_size, grid_position[1] - symbol_size,
grid_position[0] + symbol_size, grid_position[1] + symbol_size, width=symbol_thickness,
fill=symbol_X_color)
self.canvas.create_line(grid_position[0] - symbol_size, grid_position[1] + symbol_size,
grid_position[0] + symbol_size, grid_position[1] - symbol_size, width=symbol_thickness,
fill=symbol_X_color)
def display_gameover(self):
if self.X_wins:
self.X_score += 1
text = 'Winner: Player 1 (X)'
color = symbol_X_color
elif self.O_wins:
self.O_score += 1
text = 'Winner: Player 2 (O)'
color = symbol_O_color
else:
self.tie_score += 1
text = 'Its a tie'
color = 'gray'
self.canvas.delete("all")
self.canvas.create_text(size_of_board / 2, size_of_board / 3, font="cmr 60 bold", fill=color, text=text)
score_text = 'Scores \n'
self.canvas.create_text(size_of_board / 2, 5 * size_of_board / 8, font="cmr 40 bold", fill=Green_color,
text=score_text)
score_text = 'Player 1 (X) : ' + str(self.X_score) + '\n'
score_text += 'Player 2 (O): ' + str(self.O_score) + '\n'
score_text += 'Tie : ' + str(self.tie_score)
self.canvas.create_text(size_of_board / 2, 3 * size_of_board / 4, font="cmr 30 bold", fill=Green_color,
text=score_text)
self.reset_board = True
score_text = 'Click to play again \n'
self.canvas.create_text(size_of_board / 2, 15 * size_of_board / 16, font="cmr 20 bold", fill="gray",
text=score_text)
# ------------------------------------------------------------------
# Logical Functions:
# The modules required to carry out game logic
# ------------------------------------------------------------------
def convert_logical_to_grid_position(self, logical_position):
logical_position = np.array(logical_position, dtype=int)
return (size_of_board / 3) * logical_position + size_of_board / 6
def convert_grid_to_logical_position(self, grid_position):
grid_position = np.array(grid_position)
return np.array(grid_position // (size_of_board / 3), dtype=int)
def is_grid_occupied(self, logical_position):
if self.board_status[logical_position[0]][logical_position[1]] == 0:
return False
else:
return True
def is_winner(self, player):
player = -1 if player == 'X' else 1
# Three in a row
for i in range(3):
if self.board_status[i][0] == self.board_status[i][1] == self.board_status[i][2] == player:
return True
if self.board_status[0][i] == self.board_status[1][i] == self.board_status[2][i] == player:
return True
# Diagonals
if self.board_status[0][0] == self.board_status[1][1] == self.board_status[2][2] == player:
return True
if self.board_status[0][2] == self.board_status[1][1] == self.board_status[2][0] == player:
return True
return False
def is_tie(self):
r, c = np.where(self.board_status == 0)
tie = False
if len(r) == 0:
tie = True
return tie
def is_gameover(self):
# Either someone wins or all grid occupied
self.X_wins = self.is_winner('X')
if not self.X_wins:
self.O_wins = self.is_winner('O')
if not self.O_wins:
self.tie = self.is_tie()
gameover = self.X_wins or self.O_wins or self.tie
if self.X_wins:
print('X wins')
if self.O_wins:
print('O wins')
if self.tie:
print('Its a tie')
return gameover
def click(self, event):
grid_position = [event.x, event.y]
logical_position = self.convert_grid_to_logical_position(grid_position)
if not self.reset_board:
if self.player_X_turns:
if not self.is_grid_occupied(logical_position):
self.draw_X(logical_position)
self.board_status[logical_position[0]][logical_position[1]] = -1
self.player_X_turns = not self.player_X_turns
else:
if not self.is_grid_occupied(logical_position):
self.draw_O(logical_position)
self.board_status[logical_position[0]][logical_position[1]] = 1
self.player_X_turns = not self.player_X_turns
# Check if game is concluded
if self.is_gameover():
self.display_gameover()
# print('Done')
else: # Play Again
self.canvas.delete("all")
self.play_again()
self.reset_board = False
game_instance = Tic_Tac_Toe()
game_instance.mainloop()
Calculator GUI with Python
We don’t need to use any file named as file.kv for building a calculator, as a calculator is a very simple application. Let’s see how to build a simple calculator GUI with Python:
from kivy.app import App
from kivy.uix.button import Button
from kivy.uix.boxlayout import BoxLayout
from kivy.uix.gridlayout import GridLayout
from kivy.uix.label import Label
class myApp(App):
def build(self):
root_widget = BoxLayout(orientation='vertical')
output_label = Label(size_hint_y = 0.75, font_size=50)
button_symbols = ('1', '2', '3', '+',
'4', '5', '6', '-',
'7', '8', '9', '.',
'0', '*', '/', '=')
button_grid = GridLayout(cols=4, size_hint_y=2)
for symbol in button_symbols:
button_grid.add_widget(Button(text=symbol))
clear_button = Button(text = 'Clear', size_hint_y=None, height=100)
def print_button_text(instance):
output_label.text += instance.text
for button in button_grid.children[1:]:
button.bind(on_press=print_button_text)
def resize_label_text(label, new_height):
label.fontsize = 0.5*label.height
output_label.bind(height=resize_label_text)
def evaluate_result(instance):
try:
output_label.text = str(eval(output_label.text))
except SyntaxError:
output_label.text = 'Python Syntax error!'
button_grid.children[0].bind(on_press=evaluate_result)
def clear_label(instance):
output_label.text = " "
clear_button.bind(on_press=clear_label)
root_widget.add_widget(output_label)
root_widget.add_widget(button_grid)
root_widget.add_widget(clear_button)
return root_widget
myApp().run()
Building Barcode and QR code Reader with Python and Machine Learning
Now the next step is to write the decode function, where most of the cool stuff will happen. The decode function will mainly do three things and can be listed as follows:
- Recognize and decode the barcode / QR code that we are going to show to the camera.
- Added information stored as text on recognized barcode / QR code.
- And finally, export the stored information as a text document.
Let’s import the libraries we installed before writing to the function:
import cv2
from pyzbar import pyzbarNow let’s define the decoding function:
def read_barcodes(frame):
barcodes = pyzbar.decode(frame)
for barcode in barcodes:
x, y , w, h = barcode.rect
barcode_info = barcode.data.decode('utf-8')
cv2.rectangle(frame, (x, y),(x+w, y+h), (0, 255, 0), 2)
font = cv2.FONT_HERSHEY_DUPLEX
cv2.putText(frame, barcode_info, (x + 6, y - 6), font, 2.0, (255, 255, 255), 1)
with open("barcode_result.txt", mode ='w') as file:
file.write("Recognized Barcode:" + barcode_info)
return frameNow let’s go through the above function to understand what I did:
- First, we decode the barcode or QR code information. And then draw a rectangle around it. It helps us to see if our machine detected the barcode / Qr code.
- Second, we add text above the rectangle that has been created. The text will display the decoded information.
- Third, we export the information to a text document.
Now the next step is to write the main function for building a Barcode and QR code reader with Python. Let’s create our main function:
def main():
camera = cv2.VideoCapture(0)
ret, frame = camera.read()
while ret:
ret, frame = camera.read()
frame = read_barcode(frame)
cv2.imshow('Barcode/QR code reader', frame)
if cv2.waitKey(1) & 0xFF == 27:
break
camera.release()
cv2.destroyAllWindows()
if __name__ == '__main__':
main()Now let’s go through the main function above to understand what I did:
- First of all, we turn on the computer camera using OpenCV. If you have an external camera, you need to change the value 0 to 1 depending on the device.
- Second, we run a while loop to continue performing the decode function until the “Esc” key is pressed. Otherwise, the loop will not stop and cause problems.
- Third, we launch the camera that we turned on in the first step. And then we close the application window. OpenCV does all the work, just call the methods.
- Finally, we call the main function to trigger the program.
Now you can easily run the code and scan any barcode and QR code by showing the code to the camera of your laptop.
You have now created a program that reads barcodes and QR codes for you. Now you have an idea of how to use computer vision and artificial intelligence in real life. Working on hands-on programming projects like this is the best way to sharpen your coding skills.
Hope you liked this article on how to create a barcode and QR code reader with Python and Machine Learning.
Create an Audiobook with Python
Python has a large collection of libraries due to the very active community which serves various purposes. Here we need to use two libraries pyttsx3 and PyPDF2 to create an audiobook with Python.
Both the above libraries can be easily installed by using the pip command; pip install PyPDF2, pip install pyttsx3.
Reading the PDF File
PyPDF2 allows manipulation of pdf in memory. This python library is capable of tasks such as:
- extract information about the document, such as title, author, etc.
- document division by page
- merge documents per page
- cropping pages
- merge multiple pages into one page
- encrypt and decrypt PDF files
- and more.
I will use this library to split the pdf file page by page, then read the text on each page, then send the text to the next step in the process to create an audiobook with Python.
import PyPDF2
pdfReader = PyPDF2.PdfFileReader(open('file.pdf', 'rb'))The pyttsx3 library is capable of converting text to speech offline. The text that we read from a pdf is then fed as an input to the text-to-speech engine:
import pyttsx3
speaker = pyttsx3.init()Now the next step in the process is to loop the process for each page of the pdf file and stop the pyttsx3 speaker engine last:
for page_num in range(pdfReader.numPages):
text = pdfReader.getPage(page_num).extractText()
speaker.say(text)
speaker.runAndWait()
speaker.stop()Now the next step is to save the audio as mp3 file:
engine.save_to_file(text, 'audio.mp3')
engine.runAndWait()This is how we can build an audiobook with Python in a few lines of code. I hope you liked this article on how to create an audiobook with the python programming language.
How To Send Emails with Python?
As the python script will access the Gmail account to send emails, we need to turn to Allow less secure apps to ON in this account. This will make it easier for our python program to access your account. Therefore, it is recommended to create a temporary account for this purpose.
Next, to send emails with Python, we need to create a text file named template.txt. This text file contains the format of the body of the email:
Dear ${PERSON_NAME},
You have secured the following marks in your mid-term exams:
Math - ${MATH}
English - ${ENG}
Science - ${SCI}Then the next file you should have is a CSV file. The file I’ll be using has the details that need to fill in the placeholders in the template file. It contains the details that should be sent to the recipients. It can be an Excel file or a CSV file.
An Example of what I am using
Let’s Code
Once the CSV file and template file are ready, now it’s time to write the code to send custom emails with python. Let’s start by importing the necessary modules:
import smtplib
import csv
from string import Template
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMETextNow I will create a function to read the template.txt file. The function below returns the template object containing the contents of the template.txt file:
ef read_template(filename):
with open(filename, ‘r’, encoding=’utf-8') as template_file:
template_file_content = template_file.read()
return Template(template_file_content)Now we need to configure the SMTP server. It is not a good practice to include the account address, password in the script if you are going to share this script with others. Instead, use input () to allow the user to enter their password when running the script:
MY_ADDRESS = *********@gmail.com #your gmail account address
PASSWORD = *********** #your password
s = smtplib.SMTP(host=’smtp.gmail.com’, port=587)
s.starttls()
s.login(MY_ADDRESS, PASSWORD)Now the last step is to build the body of the email. For this task, we must:
- Browse the CSV file and create a message for each line of the CSV file.
- Create a message using the MIMEMultipart () function, replace the details (of each line) in the template to form the body of the message, and save it in the message variable.
- Then configure parameters such as from and to address, subject of the message. Attach the message variable to the body of the message.
- Finally, send the message via the send_message () function.
# read the message template
message_template = read_template(‘template.txt’)
with open(“details.csv”, “r”) as csv_file:
csv_reader = csv.reader(csv_file, delimiter=’,’)
# skip the first row as it is the header
next(csv_reader)
for row in csv_reader:
msg = MIMEMultipart() # create a message
# add in the actual person name to the message template
message=
message_template.substitute(PERSON_NAME=row[0],MATH=row[2],
ENG=row[3],SCI=row[4])
# Prints out the message body for our sake
print(message)
# setup the parameters of the message
msg[‘From’]=MY_ADDRESS
msg[‘To’]=lines[1]
msg[‘Subject’]=”Mid term grades”
# add in the message body
msg.attach(MIMEText(message, ‘plain’))
# send the message via the server set up earlier.
s.send_message(msg)
del msg
# Terminate the SMTP session and close the connection
s.quit()So here is your final code that you need to run:
import smtplib
import csv
from string import Template
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
def read_template(filename):
with open(filename, ‘r’, encoding=’utf-8') as template_file:
template_file_content = template_file.read()
return Template(template_file_content)
def main():
message_template = read_template(‘template.txt’)
MY_ADDRESS = ‘**********@gmail.com’
PASSWORD = ‘*************’
# set up the SMTP server
s = smtplib.SMTP(host=’smtp.gmail.com’, port=587)
s.starttls()
s.login(MY_ADDRESS, PASSWORD)
with open(“details.csv”, “r”) as csv_file:
csv_reader = csv.reader(csv_file, delimiter=’,’)
# the below statement will skip the first row
next(csv_reader)
for lines in csv_reader:
msg = MIMEMultipart() # create a message
# add in the actual person name to the message template
message = message_template.substitute(PERSON_NAME=row[0],MATH=row[2],
ENG=row[3],SCI=row[4])
print(message)
# setup the parameters of the message
msg[‘From’]=MY_ADDRESS
msg[‘To’]=lines[1]
msg[‘Subject’]=”Mid term grades”
# add in the message body
msg.attach(MIMEText(message, ‘plain’))
# send the message via the server set up earlier.
s.send_message(msg)
del msg
# Terminate the SMTP session and close the connection
s.quit()
if __name__ == ‘__main__’:
main()Finally, don’t forget to close the SMTP connection after sending all messages. Hope you liked this article on how to send personalized emails with Python.
Web Scraping to Create a CSV File
So we need two primary packages for this task, BeautifulSoup and urllib. We can easily install both these packages using the pip command – pip install bs4 and pip install urllib. After successfully installing these packages the next thing you need to do is importing these packages, so let’s import these and scrape the link we need to collect data from:
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq
my_url="https://www.flipkart.com/search?q=samsung+mobiles&sid=tyy%2C4io&as=on&as-show=on&otracker=AS_QueryStore_HistoryAutoSuggest_0_2&otracker1=AS_QueryStore_HistoryAutoSuggest_0_2&as-pos=0&as-type=HISTORY&as-searchtext=sa"
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")Now let’s see how many HTML containers are present in this link:
containers = page_soup.findAll("div", { "class": "_3O0U0u"})
print(len(containers))24print(soup.prettify(containers[0]))<div class="_3O0U0u">
<div data-id="MOBFRZZHMHQVNDFA" style="width:100%">
<div class="_1UoZlX">
<a class="_31qSD5" href="/samsung-galaxy-m01-blue-32-gb/p/itmc068b26305a0d?pid=MOBFRZZHMHQVNDFA&amp;lid=LSTMOBFRZZHMHQVNDFAZXGBO6&amp;marketplace=FLIPKART&amp;srno=s_1_1&amp;otracker=AS_QueryStore_HistoryAutoSuggest_0_2&amp;otracker1=AS_QueryStore_HistoryAutoSuggest_0_2&amp;fm=organic&amp;iid=f9a57085-7ab9-4aba-b59d-5a4cbecd03e9.MOBFRZZHMHQVNDFA.SEARCH&amp;ssid=zu9bg122ao0000001596818422200&amp;qH=0258c7d48242959a" rel="noopener noreferrer" target="_blank">
<div class="_3SQWE6">
<div class="_1OCn9C">
<div>
<div class="_3BTv9X" style="height:200px;width:200px">
<img alt="Samsung Galaxy M01 (Blue, 32 GB)" class="_1Nyybr" src="//img1a.flixcart.com/www/linchpin/fk-cp-zion/img/placeholder_9951d0.svg"/>
</div>
</div>
</div>
<div class="_2lesQu">
<div class="_1O_CiZ">
<span class="_1iHA1p">
<div class="_2kFyHg">
<label>
<input class="_3uUUD5" readonly="" type="checkbox"/>
<div class="_1p7h2j">
</div>
</label>
</div>
</span>
<label class="_10TB-Q">
<span>
Add to Compare
</span>
</label>
</div>
</div>
<div class="_3gDSOa _32A6AP">
<div class="DsQ2eg">
<svg class="_2oLiqr" height="16" viewbox="0 0 20 16" width="16" xmlns="http://www.w3.org/2000/svg">
<path class="_35Y7Yo" d="M8.695 16.682C4.06 12.382 1 9.536 1 6.065 1 3.219 3.178 1 5.95 1c1.566 0 3.069.746 4.05 1.915C10.981 1.745 12.484 1 14.05 1 16.822 1 19 3.22 19 6.065c0 3.471-3.06 6.316-7.695 10.617L10 17.897l-1.305-1.215z" fill="#2874F0" fill-rule="evenodd" opacity=".9" stroke="#FFF">
</path>
</svg>
</div>
</div>
</div>
<div class="_1-2Iqu row">
<div class="col col-7-12">
<div class="_3wU53n">
Samsung Galaxy M01 (Blue, 32 GB)
</div>
<div class="niH0FQ">
<span class="_2_KrJI" id="productRating_LSTMOBFRZZHMHQVNDFAZXGBO6_MOBFRZZHMHQVNDFA_">
<div class="hGSR34">
4.2
<img class="_2lQ_WZ" src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxMyIgaGVpZ2h0PSIxMiI+PHBhdGggZmlsbD0iI0ZGRiIgZD0iTTYuNSA5LjQzOWwtMy42NzQgMi4yMy45NC00LjI2LTMuMjEtMi44ODMgNC4yNTQtLjQwNEw2LjUuMTEybDEuNjkgNC4wMSA0LjI1NC40MDQtMy4yMSAyLjg4Mi45NCA0LjI2eiIvPjwvc3ZnPg=="/>
</div>
</span>
<span class="_38sUEc">
<span>
<span>
5,040 Ratings
</span>
<span class="_1VpSqZ">
&amp;
</span>
<span>
371 Reviews
</span>
</span>
</span>
</div>
<div class="_3ULzGw">
<ul class="vFw0gD">
<li class="tVe95H">
3 GB RAM | 32 GB ROM | Expandable Upto 512 GB
</li>
<li class="tVe95H">
14.48 cm (5.7 inch) HD+ Display
</li>
<li class="tVe95H">
13MP + 2MP | 5MP Front Camera
</li>
<li class="tVe95H">
4000 mAh Lithium-ion Battery
</li>
<li class="tVe95H">
Qualcomm Snapdragon (SDM439) Octa Core Processor
</li>
<li class="tVe95H">
1 Year Manufacturer Warranty for Phone and 6 Months Warranty for in the Box Accessories
</li>
</ul>
</div>
</div>
<div class="col col-5-12 _2o7WAb">
<div class="_6BWGkk">
<div class="_1uv9Cb">
<div class="_1vC4OE _2rQ-NK">
₹9,899
</div>
</div>
</div>
<div class="_3n6o0t">
<img height="21" src="//img1a.flixcart.com/www/linchpin/fk-cp-zion/img/fa_8b4b59.png"/>
</div>
</div>
</div>
</a>
</div>
</div>
</div>Now let’s see the first item present in the page:
container = containers[0]
print(container.div.img["alt"])Samsung Galaxy M01 (Blue, 32 GB)
So we have Samsung Galaxy M01 smartphone with blue colour as the first item on the Flipkart webpage that we have scrapped. Now let’s have a look at the price of this smartphone:
price = container.findAll("div", {"class": "col col-5-12 _2o7WAb"})
print(price[0].text)₹9,899
Now let’s have a look at its ratings from its customers:
ratings = container.findAll("div", {"class": "niH0FQ"})
print(ratings[0].text)4.25,040 Ratings & 371 Reviews
Now let’s create a CSV file and store all the mobile phones with their name, price and ratings:
filename = "products.csv"
f = open(filename, "w")
headers = "Product_Name, Pricing, Ratings \n"
f.write(headers)32
Now let’s have a look at what our CSV file has stored after the web scraping of Flipkart:
for container in containers:
product_name = container.div.img["alt"]
price_container = container.findAll("div", {"class": "col col-5-12 _2o7WAb"})
price = price_container[0].text.strip()
rating_container = container.findAll("div", {"class": "niH0FQ"})
rating = rating_container[0].text
print("Product_Name:"+ product_name)
print("Price: " + price)
print("Ratings:" + rating)Product_Name:Samsung Galaxy M01 (Blue, 32 GB)
Price: ₹9,899
Ratings:4.25,040 Ratings & 371 Reviews
Product_Name:Samsung Galaxy A71 (Haze Crush Silver, 128 GB)
Price: ₹30,999₹34,99911% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.41,311 Ratings & 140 Reviews
Product_Name:Samsung Galaxy A31 (Prism Crush Black, 128 GB)
Price: ₹20,999₹23,99912% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.32,905 Ratings & 272 Reviews
Product_Name:Samsung Galaxy M11 (Violet, 32 GB)
Price: ₹11,899₹12,6996% off
Ratings:4.24,469 Ratings & 381 Reviews
Product_Name:Samsung Galaxy A31 (Prism Crush Blue, 128 GB)
Price: ₹20,999₹23,99912% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.32,905 Ratings & 272 Reviews
Product_Name:Samsung Galaxy A31 (Prism Crush White, 128 GB)
Price: ₹20,999₹23,99912% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.32,905 Ratings & 272 Reviews
Product_Name:Samsung Galaxy A21s (White, 64 GB)
Price: ₹15,999₹17,99911% offUpto ₹13,150 Off on Exchange
Ratings:4.23,257 Ratings & 301 Reviews
Product_Name:Samsung Galaxy A21s (Black, 64 GB)
Price: ₹15,999₹17,99911% offUpto ₹13,150 Off on Exchange
Ratings:4.23,257 Ratings & 301 Reviews
Product_Name:Samsung Galaxy A21s (Blue, 64 GB)
Price: ₹15,999₹17,99911% offUpto ₹13,150 Off on Exchange
Ratings:4.23,257 Ratings & 301 Reviews
Product_Name:Samsung Galaxy M11 (Violet, 64 GB)
Price: ₹13,970₹14,2792% off
Ratings:4.22,817 Ratings & 231 Reviews
Product_Name:Samsung Galaxy M31 (Space Black, 64 GB)
Price: ₹18,922No Cost EMI
Ratings:4.4351 Ratings & 30 Reviews
Product_Name:Samsung Galaxy M11 (Metallic Blue, 64 GB)
Price: ₹13,988₹14,8505% off
Ratings:4.22,817 Ratings & 231 Reviews
Product_Name:Samsung Galaxy M11 (Metallic Blue, 32 GB)
Price: ₹11,925
Ratings:4.24,469 Ratings & 381 Reviews
Product_Name:Samsung Galaxy A71 (Prism Crush Black, 128 GB)
Price: ₹30,999₹34,99911% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.41,311 Ratings & 140 Reviews
Product_Name:Samsung Galaxy A51 (Prism Crush Black, 128 GB)
Price: ₹23,999₹25,9997% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.44,117 Ratings & 470 Reviews
Product_Name:Samsung Galaxy A51 (Prism Crush White, 128 GB)
Price: ₹27,299₹28,9995% off
Ratings:4.4517 Ratings & 33 Reviews
Product_Name:Samsung Guru FM Plus SM-B110E/D
Price: ₹1,662No Cost EMI
Ratings:4.348,448 Ratings & 5,198 Reviews
Product_Name:Samsung Galaxy A51 (Prism Crush Blue, 128 GB)
Price: ₹23,999₹25,9997% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.44,117 Ratings & 470 Reviews
Product_Name:Samsung Galaxy M01 (Black, 32 GB)
Price: ₹9,779₹9,9992% off
Ratings:4.25,040 Ratings & 371 Reviews
Product_Name:Samsung Galaxy M11 (Black, 32 GB)
Price: ₹11,865
Ratings:4.24,469 Ratings & 381 Reviews
Product_Name:Samsung Galaxy M30S (Black, 128 GB)
Price: ₹17,736₹17,9961% offNo Cost EMI
Ratings:4.33,317 Ratings & 295 Reviews
Product_Name:Samsung Galaxy M40 (Seawater Blue, 128 GB)
Price: ₹21,490No Cost EMI
Ratings:4.2691 Ratings & 50 Reviews
Product_Name:Samsung Galaxy S10 Lite (Prism Blue, 128 GB)
Price: ₹42,999₹43,9992% offNo Cost EMIUpto ₹13,150 Off on Exchange
Ratings:4.59,675 Ratings & 2,410 Reviews
Product_Name:Samsung Galaxy A21s (Blue, 64 GB)
Price: ₹17,499₹19,99912% offUpto ₹13,150 Off on Exchange
Ratings:4.31,268 Ratings & 118 Reviews#python #programming #developer #morioh #programmer #coding #softwaredeveloper #computerscience #webdev #webdeveloper #webdevelopment
