1639817150
axyl-stats
is a suite of programs for tracking and displaying stats and info about a GitHub repo.
One is a Discord bot made with Python (with the Hikari API wrapper) and uses PostgreSQL, used as a stats visualizer for a repo. The other is a database program that fetches info from the GitHub API and puts it in the SQL database.
The bot is used to check the download stats of a particular repo either with a bot command (.stats
) or automatically in a set interval (TODO).
Setting up the bot is done through the .env
environment variables.
Right now, the bot's functionality is like this:
Python 3.8
and above is required. PostgreSQL must also be installed, set up with a database and running. axyl-stats will take care of creating and managing the database table.
First, clone this repo:
git clone https://github.com/angelofallars/axyl-stats
Then, change directories into the repo and install the required dependencies:
cd axyl-stats
python3 -m pip install -r requirements.txt
In the same directory, make a .env
file and put the bot token and repo info in there.
The environment variables that axyl-stats will use are:
axyl_stats_bot.py
BOT_TOKEN
: The Discord bot's API token. Make a new Discord application in the Discord Dev Portal and create a bot for it. You will see the copyable token.REPO_OWNER
: The owner of the repo.REPO_NAME
: The name of the repo.DB_NAME
: The database to fetch data from.PostgreSQL
first with the name you will put in DB_NAME
before you can run this app.COUNTER_CHANNEL
: The Discord channel(s) to send automated statistics to. Multiple channels are separated with a comma (,).BOT_PREFIX
(default .stats
): The prefix of the bot for commands.INTERVAL
(default 60
): The interval in minutes in which the bot will fetch the download stats.DB_USER
: The user logging into the DB.DB_PASS
: The DB password.DB_HOST
(default 127.0.0.1
): The host IP address.DB_PORT
(default 5432
): The port of the DB.
axyl_stats_db.py
To run the database module, you must also put in the .env
file:
DB_NAME
: Ditto.REPO_OWNER
: Ditto.REPO_NAME
: Ditto.GITHUB_API_KEY
: The GitHub API key for requesting data. If you don't have an API key, you'll be limited to 60 requests per hour.DB_USER
: Ditto.DB_PASS
: Ditto.DB_HOST
: Ditto.DB_PORT
: Ditto..env
example.env
fileBOT_TOKEN=<your token>
GITHUB_API_KEY=<api key>
REPO_OWNER=axyl-os
REPO_NAME=axyl-iso
INTERVAL=60
COUNTER_CHANNEL=axyl-statistics
DB_NAME=axyl-stats
DB_USER=archie
DB_PASS=hunter2
To run the bot:
python3 axyl_stats_bot.py
To run the program that updates the database with info from the GitHub API, you need to execute:
python3 axyl_stats_db.py
However, this will only create the database (if running for the first time) and insert only one row of data for the current time. If you want more rows of data, you need to run the bot again. If you want to periodically fetch data from the GitHub API every set interval like 10 minutes, 30 minutes or an hour, it is recommended to use cron jobs.
In the configured database, axyl_stats_db.py
will create a table called repo_stats
with the following columns:
repo
: The repository the program is set to fetch data from.total_downloads
: The total number of downloads for every release in the Releases section.latest_downloads
: The number of downloads for the latest release in Releases.stars
watchers
forks
date
: The time the data was fetched.Example table:
repo | total_downloads | latest_downloads | stars | watchers | forks | date
------------------+-----------------+------------------+-------+----------+-------+---------------------
axyl-os/axyl-iso | 1325 | 349 | 56 | 56 | 2 | 2021-11-19 21:50:52
axyl-os/axyl-iso | 1325 | 349 | 56 | 56 | 2 | 2021-11-19 21:55:53
axyl-os/axyl-iso | 1325 | 349 | 56 | 56 | 2 | 2021-11-19 22:00:54
axyl-os/axyl-iso | 1327 | 351 | 56 | 56 | 2 | 2021-11-19 22:05:56
axyl-os/axyl-iso | 1327 | 351 | 56 | 56 | 2 | 2021-11-19 22:10:57
axyl-os/axyl-iso | 1327 | 351 | 56 | 56 | 2 | 2021-11-19 22:15:58
You can run axyl_stats_db.py
without having to run the bot program. In fact, you can just run the database program by itself. You then can access the PostgreSQL database from another program and perhaps run Matplotlib to visualize the growth of your repository.
This program is licensed under the GPLv3 License.
Author: angelofallars
Source Code: https://github.com/angelofallars/axyl-stats
License: GPL-3.0 License
1655630160
Install via pip:
$ pip install pytumblr
Install from source:
$ git clone https://github.com/tumblr/pytumblr.git
$ cd pytumblr
$ python setup.py install
A pytumblr.TumblrRestClient
is the object you'll make all of your calls to the Tumblr API through. Creating one is this easy:
client = pytumblr.TumblrRestClient(
'<consumer_key>',
'<consumer_secret>',
'<oauth_token>',
'<oauth_secret>',
)
client.info() # Grabs the current user information
Two easy ways to get your credentials to are:
interactive_console.py
tool (if you already have a consumer key & secret)client.info() # get information about the authenticating user
client.dashboard() # get the dashboard for the authenticating user
client.likes() # get the likes for the authenticating user
client.following() # get the blogs followed by the authenticating user
client.follow('codingjester.tumblr.com') # follow a blog
client.unfollow('codingjester.tumblr.com') # unfollow a blog
client.like(id, reblogkey) # like a post
client.unlike(id, reblogkey) # unlike a post
client.blog_info(blogName) # get information about a blog
client.posts(blogName, **params) # get posts for a blog
client.avatar(blogName) # get the avatar for a blog
client.blog_likes(blogName) # get the likes on a blog
client.followers(blogName) # get the followers of a blog
client.blog_following(blogName) # get the publicly exposed blogs that [blogName] follows
client.queue(blogName) # get the queue for a given blog
client.submission(blogName) # get the submissions for a given blog
Creating posts
PyTumblr lets you create all of the various types that Tumblr supports. When using these types there are a few defaults that are able to be used with any post type.
The default supported types are described below.
We'll show examples throughout of these default examples while showcasing all the specific post types.
Creating a photo post
Creating a photo post supports a bunch of different options plus the described default options * caption - a string, the user supplied caption * link - a string, the "click-through" url for the photo * source - a string, the url for the photo you want to use (use this or the data parameter) * data - a list or string, a list of filepaths or a single file path for multipart file upload
#Creates a photo post using a source URL
client.create_photo(blogName, state="published", tags=["testing", "ok"],
source="https://68.media.tumblr.com/b965fbb2e501610a29d80ffb6fb3e1ad/tumblr_n55vdeTse11rn1906o1_500.jpg")
#Creates a photo post using a local filepath
client.create_photo(blogName, state="queue", tags=["testing", "ok"],
tweet="Woah this is an incredible sweet post [URL]",
data="/Users/johnb/path/to/my/image.jpg")
#Creates a photoset post using several local filepaths
client.create_photo(blogName, state="draft", tags=["jb is cool"], format="markdown",
data=["/Users/johnb/path/to/my/image.jpg", "/Users/johnb/Pictures/kittens.jpg"],
caption="## Mega sweet kittens")
Creating a text post
Creating a text post supports the same options as default and just a two other parameters * title - a string, the optional title for the post. Supports markdown or html * body - a string, the body of the of the post. Supports markdown or html
#Creating a text post
client.create_text(blogName, state="published", slug="testing-text-posts", title="Testing", body="testing1 2 3 4")
Creating a quote post
Creating a quote post supports the same options as default and two other parameter * quote - a string, the full text of the qote. Supports markdown or html * source - a string, the cited source. HTML supported
#Creating a quote post
client.create_quote(blogName, state="queue", quote="I am the Walrus", source="Ringo")
Creating a link post
#Create a link post
client.create_link(blogName, title="I like to search things, you should too.", url="https://duckduckgo.com",
description="Search is pretty cool when a duck does it.")
Creating a chat post
Creating a chat post supports the same options as default and two other parameters * title - a string, the title of the chat post * conversation - a string, the text of the conversation/chat, with diablog labels (no html)
#Create a chat post
chat = """John: Testing can be fun!
Renee: Testing is tedious and so are you.
John: Aw.
"""
client.create_chat(blogName, title="Renee just doesn't understand.", conversation=chat, tags=["renee", "testing"])
Creating an audio post
Creating an audio post allows for all default options and a has 3 other parameters. The only thing to keep in mind while dealing with audio posts is to make sure that you use the external_url parameter or data. You cannot use both at the same time. * caption - a string, the caption for your post * external_url - a string, the url of the site that hosts the audio file * data - a string, the filepath of the audio file you want to upload to Tumblr
#Creating an audio file
client.create_audio(blogName, caption="Rock out.", data="/Users/johnb/Music/my/new/sweet/album.mp3")
#lets use soundcloud!
client.create_audio(blogName, caption="Mega rock out.", external_url="https://soundcloud.com/skrillex/sets/recess")
Creating a video post
Creating a video post allows for all default options and has three other options. Like the other post types, it has some restrictions. You cannot use the embed and data parameters at the same time. * caption - a string, the caption for your post * embed - a string, the HTML embed code for the video * data - a string, the path of the file you want to upload
#Creating an upload from YouTube
client.create_video(blogName, caption="Jon Snow. Mega ridiculous sword.",
embed="http://www.youtube.com/watch?v=40pUYLacrj4")
#Creating a video post from local file
client.create_video(blogName, caption="testing", data="/Users/johnb/testing/ok/blah.mov")
Editing a post
Updating a post requires you knowing what type a post you're updating. You'll be able to supply to the post any of the options given above for updates.
client.edit_post(blogName, id=post_id, type="text", title="Updated")
client.edit_post(blogName, id=post_id, type="photo", data="/Users/johnb/mega/awesome.jpg")
Reblogging a Post
Reblogging a post just requires knowing the post id and the reblog key, which is supplied in the JSON of any post object.
client.reblog(blogName, id=125356, reblog_key="reblog_key")
Deleting a post
Deleting just requires that you own the post and have the post id
client.delete_post(blogName, 123456) # Deletes your post :(
A note on tags: When passing tags, as params, please pass them as a list (not a comma-separated string):
client.create_text(blogName, tags=['hello', 'world'], ...)
Getting notes for a post
In order to get the notes for a post, you need to have the post id and the blog that it is on.
data = client.notes(blogName, id='123456')
The results include a timestamp you can use to make future calls.
data = client.notes(blogName, id='123456', before_timestamp=data["_links"]["next"]["query_params"]["before_timestamp"])
# get posts with a given tag
client.tagged(tag, **params)
This client comes with a nice interactive console to run you through the OAuth process, grab your tokens (and store them for future use).
You'll need pyyaml
installed to run it, but then it's just:
$ python interactive-console.py
and away you go! Tokens are stored in ~/.tumblr
and are also shared by other Tumblr API clients like the Ruby client.
The tests (and coverage reports) are run with nose, like this:
python setup.py test
Author: tumblr
Source Code: https://github.com/tumblr/pytumblr
License: Apache-2.0 license
1624410000
Create a Twitter bot with Python that tweets images or status updates at a set interval. The Python script also scrapes the web for data.
πΊ The video in this post was made by freeCodeCamp.org
The origin of the article: https://www.youtube.com/watch?v=8u-zJVVVhT4&list=PLWKjhJtqVAbnqBxcdjVGgT3uVR10bzTEB&index=14
π₯ If youβre a beginner. I believe the article below will be useful to you β What You Should Know Before Investing in Cryptocurrency - For Beginner
β β βThe project is of interest to the community. Join to Get free βGEEK coinβ (GEEKCASH coin)!
β **-----CLICK HERE-----**β β β
Thanks for visiting and watching! Please donβt forget to leave a like, comment and share!
#python #a twitter bot #a twitter bot with python #bot #bot with python #create a twitter bot with python
1669003576
In this Python article, let's learn about Mutable and Immutable in Python.
Mutable is a fancy way of saying that the internal state of the object is changed/mutated. So, the simplest definition is: An object whose internal state can be changed is mutable. On the other hand, immutable doesnβt allow any change in the object once it has been created.
Both of these states are integral to Python data structure. If you want to become more knowledgeable in the entire Python Data Structure, take this free course which covers multiple data structures in Python including tuple data structure which is immutable. You will also receive a certificate on completion which is sure to add value to your portfolio.
Mutable is when something is changeable or has the ability to change. In Python, βmutableβ is the ability of objects to change their values. These are often the objects that store a collection of data.
Immutable is the when no change is possible over time. In Python, if the value of an object cannot be changed over time, then it is known as immutable. Once created, the value of these objects is permanent.
Objects of built-in type that are mutable are:
Objects of built-in type that are immutable are:
Object mutability is one of the characteristics that makes Python a dynamically typed language. Though Mutable and Immutable in Python is a very basic concept, it can at times be a little confusing due to the intransitive nature of immutability.
In Python, everything is treated as an object. Every object has these three attributes:
While ID and Type cannot be changed once itβs created, values can be changed for Mutable objects.
Check out this free python certificate course to get started with Python.
I believe, rather than diving deep into the theory aspects of mutable and immutable in Python, a simple code would be the best way to depict what it means in Python. Hence, let us discuss the below code step-by-step:
#Creating a list which contains name of Indian cities
cities = [βDelhiβ, βMumbaiβ, βKolkataβ]
# Printing the elements from the list cities, separated by a comma & space
for city in cities:
print(city, end=β, β)
Output [1]: Delhi, Mumbai, Kolkata
#Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(cities)))
Output [2]: 0x1691d7de8c8
#Adding a new city to the list cities
cities.append(βChennaiβ)
#Printing the elements from the list cities, separated by a comma & space
for city in cities:
print(city, end=β, β)
Output [3]: Delhi, Mumbai, Kolkata, Chennai
#Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(cities)))
Output [4]: 0x1691d7de8c8
The above example shows us that we were able to change the internal state of the object βcitiesβ by adding one more city βChennaiβ to it, yet, the memory address of the object did not change. This confirms that we did not create a new object, rather, the same object was changed or mutated. Hence, we can say that the object which is a type of list with reference variable name βcitiesβ is a MUTABLE OBJECT.
Let us now discuss the term IMMUTABLE. Considering that we understood what mutable stands for, it is obvious that the definition of immutable will have βNOTβ included in it. Here is the simplest definition of immutableβ An object whose internal state can NOT be changed is IMMUTABLE.
Again, if you try and concentrate on different error messages, you have encountered, thrown by the respective IDE; you use you would be able to identify the immutable objects in Python. For instance, consider the below code & associated error message with it, while trying to change the value of a Tuple at index 0.
#Creating a Tuple with variable name βfooβ
foo = (1, 2)
#Changing the index[0] value from 1 to 3
foo[0] = 3
TypeError: 'tuple' object does not support item assignment
Once again, a simple code would be the best way to depict what immutable stands for. Hence, let us discuss the below code step-by-step:
#Creating a Tuple which contains English name of weekdays
weekdays = βSundayβ, βMondayβ, βTuesdayβ, βWednesdayβ, βThursdayβ, βFridayβ, βSaturdayβ
# Printing the elements of tuple weekdays
print(weekdays)
Output [1]: (βSundayβ, βMondayβ, βTuesdayβ, βWednesdayβ, βThursdayβ, βFridayβ, βSaturdayβ)
#Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(weekdays)))
Output [2]: 0x1691cc35090
#tuples are immutable, so you cannot add new elements, hence, using merge of tuples with the # + operator to add a new imaginary day in the tuple βweekdaysβ
weekdays += βPythondayβ,
#Printing the elements of tuple weekdays
print(weekdays)
Output [3]: (βSundayβ, βMondayβ, βTuesdayβ, βWednesdayβ, βThursdayβ, βFridayβ, βSaturdayβ, βPythondayβ)
#Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(weekdays)))
Output [4]: 0x1691cc8ad68
This above example shows that we were able to use the same variable name that is referencing an object which is a type of tuple with seven elements in it. However, the ID or the memory location of the old & new tuple is not the same. We were not able to change the internal state of the object βweekdaysβ. The Python program manager created a new object in the memory address and the variable name βweekdaysβ started referencing the new object with eight elements in it. Hence, we can say that the object which is a type of tuple with reference variable name βweekdaysβ is an IMMUTABLE OBJECT.
Also Read: Understanding the Exploratory Data Analysis (EDA) in Python
Where can you use mutable and immutable objects:
Mutable objects can be used where you want to allow for any updates. For example, you have a list of employee names in your organizations, and that needs to be updated every time a new member is hired. You can create a mutable list, and it can be updated easily.
Immutability offers a lot of useful applications to different sensitive tasks we do in a network centred environment where we allow for parallel processing. By creating immutable objects, you seal the values and ensure that no threads can invoke overwrite/update to your data. This is also useful in situations where you would like to write a piece of code that cannot be modified. For example, a debug code that attempts to find the value of an immutable object.
Watch outs: Non transitive nature of Immutability:
OK! Now we do understand what mutable & immutable objects in Python are. Letβs go ahead and discuss the combination of these two and explore the possibilities. Letβs discuss, as to how will it behave if you have an immutable object which contains the mutable object(s)? Or vice versa? Let us again use a code to understand this behaviourβ
#creating a tuple (immutable object) which contains 2 lists(mutable) as itβs elements
#The elements (lists) contains the name, age & gender
person = (['Ayaan', 5, 'Male'], ['Aaradhya', 8, 'Female'])
#printing the tuple
print(person)
Output [1]: (['Ayaan', 5, 'Male'], ['Aaradhya', 8, 'Female'])
#printing the location of the object created in the memory address in hexadecimal format
print(hex(id(person)))
Output [2]: 0x1691ef47f88
#Changing the age for the 1st element. Selecting 1st element of tuple by using indexing [0] then 2nd element of the list by using indexing [1] and assigning a new value for age as 4
person[0][1] = 4
#printing the updated tuple
print(person)
Output [3]: (['Ayaan', 4, 'Male'], ['Aaradhya', 8, 'Female'])
#printing the location of the object created in the memory address in hexadecimal format
print(hex(id(person)))
Output [4]: 0x1691ef47f88
In the above code, you can see that the object βpersonβ is immutable since it is a type of tuple. However, it has two lists as itβs elements, and we can change the state of lists (lists being mutable). So, here we did not change the object reference inside the Tuple, but the referenced object was mutated.
Also Read: Real-Time Object Detection Using TensorFlow
Same way, letβs explore how it will behave if you have a mutable object which contains an immutable object? Let us again use a code to understand the behaviourβ
#creating a list (mutable object) which contains tuples(immutable) as itβs elements
list1 = [(1, 2, 3), (4, 5, 6)]
#printing the list
print(list1)
Output [1]: [(1, 2, 3), (4, 5, 6)]
#printing the location of the object created in the memory address in hexadecimal format
print(hex(id(list1)))
Output [2]: 0x1691d5b13c8
#changing object reference at index 0
list1[0] = (7, 8, 9)
#printing the list
Output [3]: [(7, 8, 9), (4, 5, 6)]
#printing the location of the object created in the memory address in hexadecimal format
print(hex(id(list1)))
Output [4]: 0x1691d5b13c8
As an individual, it completely depends upon you and your requirements as to what kind of data structure you would like to create with a combination of mutable & immutable objects. I hope that this information will help you while deciding the type of object you would like to select going forward.
Before I end our discussion on IMMUTABILITY, allow me to use the word βCAVITEβ when we discuss the String and Integers. There is an exception, and you may see some surprising results while checking the truthiness for immutability. For instance:
#creating an object of integer type with value 10 and reference variable name βxβ
x = 10
#printing the value of βxβ
print(x)
Output [1]: 10
#Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(x)))
Output [2]: 0x538fb560
#creating an object of integer type with value 10 and reference variable name βyβ
y = 10
#printing the value of βyβ
print(y)
Output [3]: 10
#Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(y)))
Output [4]: 0x538fb560
As per our discussion and understanding, so far, the memory address for x & y should have been different, since, 10 is an instance of Integer class which is immutable. However, as shown in the above code, it has the same memory address. This is not something that we expected. It seems that what we have understood and discussed, has an exception as well.
Quick check β Python Data Structures
Tuples are immutable and hence cannot have any changes in them once they are created in Python. This is because they support the same sequence operations as strings. We all know that strings are immutable. The index operator will select an element from a tuple just like in a string. Hence, they are immutable.
Like all, there are exceptions in the immutability in python too. Not all immutable objects are really mutable. This will lead to a lot of doubts in your mind. Let us just take an example to understand this.
Consider a tuple βtupβ.
Now, if we consider tuple tup = (βGreatLearningβ,[4,3,1,2]) ;
We see that the tuple has elements of different data types. The first element here is a string which as we all know is immutable in nature. The second element is a list which we all know is mutable. Now, we all know that the tuple itself is an immutable data type. It cannot change its contents. But, the list inside it can change its contents. So, the value of the Immutable objects cannot be changed but its constituent objects can. change its value.
Mutable Object | Immutable Object |
State of the object can be modified after it is created. | State of the object canβt be modified once it is created. |
They are not thread safe. | They are thread safe |
Mutable classes are not final. | It is important to make the class final before creating an immutable object. |
list, dictionary, set, user-defined classes.
int, float, decimal, bool, string, tuple, range.
Lists in Python are mutable data types as the elements of the list can be modified, individual elements can be replaced, and the order of elements can be changed even after the list has been created.
(Examples related to lists have been discussed earlier in this blog.)
Tuple and list data structures are very similar, but one big difference between the data types is that lists are mutable, whereas tuples are immutable. The reason for the tupleβs immutability is that once the elements are added to the tuple and the tuple has been created; it remains unchanged.
A programmer would always prefer building a code that can be reused instead of making the whole data object again. Still, even though tuples are immutable, like lists, they can contain any Python object, including mutable objects.
A set is an iterable unordered collection of data type which can be used to perform mathematical operations (like union, intersection, difference etc.). Every element in a set is unique and immutable, i.e. no duplicate values should be there, and the values canβt be changed. However, we can add or remove items from the set as the set itself is mutable.
Strings are not mutable in Python. Strings are a immutable data types which means that its value cannot be updated.
Join Great Learning Academyβs free online courses and upgrade your skills today.
Original article source at: https://www.mygreatlearning.com
1650099327
In this article, We will take you through a tutorial on the tabulate module to create tables using Python.
The tabulate module in Python allows us to create and display data in a tabular format which makes the data look more readable. It can be used to organize your data to make it more understandable. Below are some of the data structures in Python which are supported by the tabulate module:
The tabulate module doesnβt come preinstalled in the Python standard library so you can easily install it by using the pip command; pip install tabulate.
pip install tabulate
We can then load the library:
from tabulate import tabulate
We can then use the following basic syntax to create tables:
print(tabulate(data, headers=col_names, tablefmt="grid", showindex="always"))
The following examples show how to use this function in practice.
The following code shows how to create a basic table with headers:
#create data
data = [["Mavs", 99],
["Suns", 91],
["Spurs", 94],
["Nets", 88]]
#define header names
col_names = ["Team", "Points"]
#display table
print(tabulate(data, headers=col_names))
Team Points
------ --------
Mavs 99
Suns 91
Spurs 94
Nets 88
The following code shows how to create a table with headers and a fancy grid:
#create data
data = [["Mavs", 99],
["Suns", 91],
["Spurs", 94],
["Nets", 88]]
#define header names
col_names = ["Team", "Points"]
#display table
print(tabulate(data, headers=col_names, tablefmt="fancy_grid"))
ββββββββββ€βββββββββββ
β Team β Points β
ββββββββββͺβββββββββββ‘
β Mavs β 99 β
ββββββββββΌβββββββββββ€
β Suns β 91 β
ββββββββββΌβββββββββββ€
β Spurs β 94 β
ββββββββββΌβββββββββββ€
β Nets β 88 β
ββββββββββ§βββββββββββ
Note that the tablefmt argument accepts several different options including:
Refer to the tabulate documentation for a complete list of potential table formats.
The following code shows how to create a table with headers, a fancy grid, and an index column:
#create data
data = [["Mavs", 99],
["Suns", 91],
["Spurs", 94],
["Nets", 88]]
#define header names
col_names = ["Team", "Points"]
#display table
print(tabulate(data, headers=col_names, tablefmt="fancy_grid", showindex="always"))
ββββββ€βββββββββ€βββββββββββ
β β Team β Points β
ββββββͺβββββββββͺβββββββββββ‘
β 0 β Mavs β 99 β
ββββββΌβββββββββΌβββββββββββ€
β 1 β Suns β 91 β
ββββββΌβββββββββΌβββββββββββ€
β 2 β Spurs β 94 β
ββββββΌβββββββββΌβββββββββββ€
β 3 β Nets β 88 β
ββββββ§βββββββββ§βββββββββββ
So this is how you can present your data in the form of tables. It is a good approach to format the data into tables as it makes the data look more readable.
1646698200
What is face recognition? Or what is recognition? When you look at an apple fruit, your mind immediately tells you that this is an apple fruit. This process, your mind telling you that this is an apple fruit is recognition in simple words. So what is face recognition then? I am sure you have guessed it right. When you look at your friend walking down the street or a picture of him, you recognize that he is your friend Paulo. Interestingly when you look at your friend or a picture of him you look at his face first before looking at anything else. Ever wondered why you do that? This is so that you can recognize him by looking at his face. Well, this is you doing face recognition.
But the real question is how does face recognition works? It is quite simple and intuitive. Take a real life example, when you meet someone first time in your life you don't recognize him, right? While he talks or shakes hands with you, you look at his face, eyes, nose, mouth, color and overall look. This is your mind learning or training for the face recognition of that person by gathering face data. Then he tells you that his name is Paulo. At this point your mind knows that the face data it just learned belongs to Paulo. Now your mind is trained and ready to do face recognition on Paulo's face. Next time when you will see Paulo or his face in a picture you will immediately recognize him. This is how face recognition work. The more you will meet Paulo, the more data your mind will collect about Paulo and especially his face and the better you will become at recognizing him.
Now the next question is how to code face recognition with OpenCV, after all this is the only reason why you are reading this article, right? OK then. You might say that our mind can do these things easily but to actually code them into a computer is difficult? Don't worry, it is not. Thanks to OpenCV, coding face recognition is as easier as it feels. The coding steps for face recognition are same as we discussed it in real life example above.
OpenCV comes equipped with built in face recognizer, all you have to do is feed it the face data. It's that simple and this how it will look once we are done coding it.
OpenCV has three built in face recognizers and thanks to OpenCV's clean coding, you can use any of them by just changing a single line of code. Below are the names of those face recognizers and their OpenCV calls.
cv2.face.createEigenFaceRecognizer()
cv2.face.createFisherFaceRecognizer()
cv2.face.createLBPHFaceRecognizer()
We have got three face recognizers but do you know which one to use and when? Or which one is better? I guess not. So why not go through a brief summary of each, what you say? I am assuming you said yes :) So let's dive into the theory of each.
This algorithm considers the fact that not all parts of a face are equally important and equally useful. When you look at some one you recognize him/her by his distinct features like eyes, nose, cheeks, forehead and how they vary with respect to each other. So you are actually focusing on the areas of maximum change (mathematically speaking, this change is variance) of the face. For example, from eyes to nose there is a significant change and same is the case from nose to mouth. When you look at multiple faces you compare them by looking at these parts of the faces because these parts are the most useful and important components of a face. Important because they catch the maximum change among faces, change the helps you differentiate one face from the other. This is exactly how EigenFaces face recognizer works.
EigenFaces face recognizer looks at all the training images of all the persons as a whole and try to extract the components which are important and useful (the components that catch the maximum variance/change) and discards the rest of the components. This way it not only extracts the important components from the training data but also saves memory by discarding the less important components. These important components it extracts are called principal components. Below is an image showing the principal components extracted from a list of faces.
Principal Components source
You can see that principal components actually represent faces and these faces are called eigen faces and hence the name of the algorithm.
So this is how EigenFaces face recognizer trains itself (by extracting principal components). Remember, it also keeps a record of which principal component belongs to which person. One thing to note in above image is that Eigenfaces algorithm also considers illumination as an important component.
Later during recognition, when you feed a new image to the algorithm, it repeats the same process on that image as well. It extracts the principal component from that new image and compares that component with the list of components it stored during training and finds the component with the best match and returns the person label associated with that best match component.
Easy peasy, right? Next one is easier than this one.
This algorithm is an improved version of EigenFaces face recognizer. Eigenfaces face recognizer looks at all the training faces of all the persons at once and finds principal components from all of them combined. By capturing principal components from all the of them combined you are not focusing on the features that discriminate one person from the other but the features that represent all the persons in the training data as a whole.
This approach has drawbacks, for example, images with sharp changes (like light changes which is not a useful feature at all) may dominate the rest of the images and you may end up with features that are from external source like light and are not useful for discrimination at all. In the end, your principal components will represent light changes and not the actual face features.
Fisherfaces algorithm, instead of extracting useful features that represent all the faces of all the persons, it extracts useful features that discriminate one person from the others. This way features of one person do not dominate over the others and you have the features that discriminate one person from the others.
Below is an image of features extracted using Fisherfaces algorithm.
Fisher Faces source
You can see that features extracted actually represent faces and these faces are called fisher faces and hence the name of the algorithm.
One thing to note here is that even in Fisherfaces algorithm if multiple persons have images with sharp changes due to external sources like light they will dominate over other features and affect recognition accuracy.
Getting bored with this theory? Don't worry, only one face recognizer is left and then we will dive deep into the coding part.
I wrote a detailed explaination on Local Binary Patterns Histograms in my previous article on face detection using local binary patterns histograms. So here I will just give a brief overview of how it works.
We know that Eigenfaces and Fisherfaces are both affected by light and in real life we can't guarantee perfect light conditions. LBPH face recognizer is an improvement to overcome this drawback.
Idea is to not look at the image as a whole instead find the local features of an image. LBPH alogrithm try to find the local structure of an image and it does that by comparing each pixel with its neighboring pixels.
Take a 3x3 window and move it one image, at each move (each local part of an image), compare the pixel at the center with its neighbor pixels. The neighbors with intensity value less than or equal to center pixel are denoted by 1 and others by 0. Then you read these 0/1 values under 3x3 window in a clockwise order and you will have a binary pattern like 11100011 and this pattern is local to some area of the image. You do this on whole image and you will have a list of local binary patterns.
LBP Labeling
Now you get why this algorithm has Local Binary Patterns in its name? Because you get a list of local binary patterns. Now you may be wondering, what about the histogram part of the LBPH? Well after you get a list of local binary patterns, you convert each binary pattern into a decimal number (as shown in above image) and then you make a histogram of all of those values. A sample histogram looks like this.
Sample Histogram
I guess this answers the question about histogram part. So in the end you will have one histogram for each face image in the training data set. That means if there were 100 images in training data set then LBPH will extract 100 histograms after training and store them for later recognition. Remember, algorithm also keeps track of which histogram belongs to which person.
Later during recognition, when you will feed a new image to the recognizer for recognition it will generate a histogram for that new image, compare that histogram with the histograms it already has, find the best match histogram and return the person label associated with that best match histogram.
Below is a list of faces and their respective local binary patterns images. You can see that the LBP images are not affected by changes in light conditions.
LBP Faces source
The theory part is over and now comes the coding part! Ready to dive into coding? Let's get into it then.
Coding Face Recognition with OpenCV
The Face Recognition process in this tutorial is divided into three steps.
[There should be a visualization diagram for above steps here]
To detect faces, I will use the code from my previous article on face detection. So if you have not read it, I encourage you to do so to understand how face detection works and its Python coding.
Before starting the actual coding we need to import the required modules for coding. So let's import them first.
#import OpenCV module
import cv2
#import os module for reading training data directories and paths
import os
#import numpy to convert python lists to numpy arrays as
#it is needed by OpenCV face recognizers
import numpy as np
#matplotlib for display our images
import matplotlib.pyplot as plt
%matplotlib inline
The more images used in training the better. Normally a lot of images are used for training a face recognizer so that it can learn different looks of the same person, for example with glasses, without glasses, laughing, sad, happy, crying, with beard, without beard etc. To keep our tutorial simple we are going to use only 12 images for each person.
So our training data consists of total 2 persons with 12 images of each person. All training data is inside training-data
folder. training-data
folder contains one folder for each person and each folder is named with format sLabel (e.g. s1, s2)
where label is actually the integer label assigned to that person. For example folder named s1 means that this folder contains images for person 1. The directory structure tree for training data is as follows:
training-data
|-------------- s1
| |-- 1.jpg
| |-- ...
| |-- 12.jpg
|-------------- s2
| |-- 1.jpg
| |-- ...
| |-- 12.jpg
The test-data
folder contains images that we will use to test our face recognizer after it has been successfully trained.
As OpenCV face recognizer accepts labels as integers so we need to define a mapping between integer labels and persons actual names so below I am defining a mapping of persons integer labels and their respective names.
Note: As we have not assigned label 0
to any person so the mapping for label 0 is empty.
#there is no label 0 in our training data so subject name for index/label 0 is empty
subjects = ["", "Tom Cruise", "Shahrukh Khan"]
You may be wondering why data preparation, right? Well, OpenCV face recognizer accepts data in a specific format. It accepts two vectors, one vector is of faces of all the persons and the second vector is of integer labels for each face so that when processing a face the face recognizer knows which person that particular face belongs too.
For example, if we had 2 persons and 2 images for each person.
PERSON-1 PERSON-2
img1 img1
img2 img2
Then the prepare data step will produce following face and label vectors.
FACES LABELS
person1_img1_face 1
person1_img2_face 1
person2_img1_face 2
person2_img2_face 2
Preparing data step can be further divided into following sub-steps.
s1, s2
.sLabel
where Label
is an integer representing the label we have assigned to that subject. So for example, folder name s1
means that the subject has label 1, s2 means subject label is 2 and so on. The label extracted in this step is assigned to each face detected in the next step.[There should be a visualization for above steps here]
Did you read my last article on face detection? No? Then you better do so right now because to detect faces, I am going to use the code from my previous article on face detection. So if you have not read it, I encourage you to do so to understand how face detection works and its coding. Below is the same code.
#function to detect face using OpenCV
def detect_face(img):
#convert the test image to gray image as opencv face detector expects gray images
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
#load OpenCV face detector, I am using LBP which is fast
#there is also a more accurate but slow Haar classifier
face_cascade = cv2.CascadeClassifier('opencv-files/lbpcascade_frontalface.xml')
#let's detect multiscale (some images may be closer to camera than others) images
#result is a list of faces
faces = face_cascade.detectMultiScale(gray, scaleFactor=1.2, minNeighbors=5);
#if no faces are detected then return original img
if (len(faces) == 0):
return None, None
#under the assumption that there will be only one face,
#extract the face area
(x, y, w, h) = faces[0]
#return only the face part of the image
return gray[y:y+w, x:x+h], faces[0]
I am using OpenCV's LBP face detector. On line 4, I convert the image to grayscale because most operations in OpenCV are performed in gray scale, then on line 8 I load LBP face detector using cv2.CascadeClassifier
class. After that on line 12 I use cv2.CascadeClassifier
class' detectMultiScale
method to detect all the faces in the image. on line 20, from detected faces I only pick the first face because in one image there will be only one face (under the assumption that there will be only one prominent face). As faces returned by detectMultiScale
method are actually rectangles (x, y, width, height) and not actual faces images so we have to extract face image area from the main image. So on line 23 I extract face area from gray image and return both the face image area and face rectangle.
Now you have got a face detector and you know the 4 steps to prepare the data, so are you ready to code the prepare data step? Yes? So let's do it.
#this function will read all persons' training images, detect face from each image
#and will return two lists of exactly same size, one list
# of faces and another list of labels for each face
def prepare_training_data(data_folder_path):
#------STEP-1--------
#get the directories (one directory for each subject) in data folder
dirs = os.listdir(data_folder_path)
#list to hold all subject faces
faces = []
#list to hold labels for all subjects
labels = []
#let's go through each directory and read images within it
for dir_name in dirs:
#our subject directories start with letter 's' so
#ignore any non-relevant directories if any
if not dir_name.startswith("s"):
continue;
#------STEP-2--------
#extract label number of subject from dir_name
#format of dir name = slabel
#, so removing letter 's' from dir_name will give us label
label = int(dir_name.replace("s", ""))
#build path of directory containin images for current subject subject
#sample subject_dir_path = "training-data/s1"
subject_dir_path = data_folder_path + "/" + dir_name
#get the images names that are inside the given subject directory
subject_images_names = os.listdir(subject_dir_path)
#------STEP-3--------
#go through each image name, read image,
#detect face and add face to list of faces
for image_name in subject_images_names:
#ignore system files like .DS_Store
if image_name.startswith("."):
continue;
#build image path
#sample image path = training-data/s1/1.pgm
image_path = subject_dir_path + "/" + image_name
#read image
image = cv2.imread(image_path)
#display an image window to show the image
cv2.imshow("Training on image...", image)
cv2.waitKey(100)
#detect face
face, rect = detect_face(image)
#------STEP-4--------
#for the purpose of this tutorial
#we will ignore faces that are not detected
if face is not None:
#add face to list of faces
faces.append(face)
#add label for this face
labels.append(label)
cv2.destroyAllWindows()
cv2.waitKey(1)
cv2.destroyAllWindows()
return faces, labels
I have defined a function that takes the path, where training subjects' folders are stored, as parameter. This function follows the same 4 prepare data substeps mentioned above.
(step-1) On line 8 I am using os.listdir
method to read names of all folders stored on path passed to function as parameter. On line 10-13 I am defining labels and faces vectors.
(step-2) After that I traverse through all subjects' folder names and from each subject's folder name on line 27 I am extracting the label information. As folder names follow the sLabel
naming convention so removing the letter s
from folder name will give us the label assigned to that subject.
(step-3) On line 34, I read all the images names of of the current subject being traversed and on line 39-66 I traverse those images one by one. On line 53-54 I am using OpenCV's imshow(window_title, image)
along with OpenCV's waitKey(interval)
method to display the current image being traveresed. The waitKey(interval)
method pauses the code flow for the given interval (milliseconds), I am using it with 100ms interval so that we can view the image window for 100ms. On line 57, I detect face from the current image being traversed.
(step-4) On line 62-66, I add the detected face and label to their respective vectors.
But a function can't do anything unless we call it on some data that it has to prepare, right? Don't worry, I have got data of two beautiful and famous celebrities. I am sure you will recognize them!
Let's call this function on images of these beautiful celebrities to prepare data for training of our Face Recognizer. Below is a simple code to do that.
#let's first prepare our training data
#data will be in two lists of same size
#one list will contain all the faces
#and other list will contain respective labels for each face
print("Preparing data...")
faces, labels = prepare_training_data("training-data")
print("Data prepared")
#print total faces and labels
print("Total faces: ", len(faces))
print("Total labels: ", len(labels))
Preparing data...
Data prepared
Total faces: 23
Total labels: 23
This was probably the boring part, right? Don't worry, the fun stuff is coming up next. It's time to train our own face recognizer so that once trained it can recognize new faces of the persons it was trained on. Read? Ok then let's train our face recognizer.
As we know, OpenCV comes equipped with three face recognizers.
cv2.face.createEigenFaceRecognizer()
cv2.face.createFisherFaceRecognizer()
cv2.face.LBPHFisherFaceRecognizer()
I am going to use LBPH face recognizer but you can use any face recognizer of your choice. No matter which of the OpenCV's face recognizer you use the code will remain the same. You just have to change one line, the face recognizer initialization line given below.
#create our LBPH face recognizer
face_recognizer = cv2.face.createLBPHFaceRecognizer()
#or use EigenFaceRecognizer by replacing above line with
#face_recognizer = cv2.face.createEigenFaceRecognizer()
#or use FisherFaceRecognizer by replacing above line with
#face_recognizer = cv2.face.createFisherFaceRecognizer()
Now that we have initialized our face recognizer and we also have prepared our training data, it's time to train the face recognizer. We will do that by calling the train(faces-vector, labels-vector)
method of face recognizer.
#train our face recognizer of our training faces
face_recognizer.train(faces, np.array(labels))
Did you notice that instead of passing labels
vector directly to face recognizer I am first converting it to numpy array? This is because OpenCV expects labels vector to be a numpy
array.
Still not satisfied? Want to see some action? Next step is the real action, I promise!
Now comes my favorite part, the prediction part. This is where we actually get to see if our algorithm is actually recognizing our trained subjects's faces or not. We will take two test images of our celeberities, detect faces from each of them and then pass those faces to our trained face recognizer to see if it recognizes them.
Below are some utility functions that we will use for drawing bounding box (rectangle) around face and putting celeberity name near the face bounding box.
#function to draw rectangle on image
#according to given (x, y) coordinates and
#given width and heigh
def draw_rectangle(img, rect):
(x, y, w, h) = rect
cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2)
#function to draw text on give image starting from
#passed (x, y) coordinates.
def draw_text(img, text, x, y):
cv2.putText(img, text, (x, y), cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 255, 0), 2)
First function draw_rectangle
draws a rectangle on image based on passed rectangle coordinates. It uses OpenCV's built in function cv2.rectangle(img, topLeftPoint, bottomRightPoint, rgbColor, lineWidth)
to draw rectangle. We will use it to draw a rectangle around the face detected in test image.
Second function draw_text
uses OpenCV's built in function cv2.putText(img, text, startPoint, font, fontSize, rgbColor, lineWidth)
to draw text on image.
Now that we have the drawing functions, we just need to call the face recognizer's predict(face)
method to test our face recognizer on test images. Following function does the prediction for us.
#this function recognizes the person in image passed
#and draws a rectangle around detected face with name of the
#subject
def predict(test_img):
#make a copy of the image as we don't want to chang original image
img = test_img.copy()
#detect face from the image
face, rect = detect_face(img)
#predict the image using our face recognizer
label= face_recognizer.predict(face)
#get name of respective label returned by face recognizer
label_text = subjects[label]
#draw a rectangle around face detected
draw_rectangle(img, rect)
#draw name of predicted person
draw_text(img, label_text, rect[0], rect[1]-5)
return img
predict(face)
method. This method will return a lableNow that we have the prediction function well defined, next step is to actually call this function on our test images and display those test images to see if our face recognizer correctly recognized them. So let's do it. This is what we have been waiting for.
print("Predicting images...")
#load test images
test_img1 = cv2.imread("test-data/test1.jpg")
test_img2 = cv2.imread("test-data/test2.jpg")
#perform a prediction
predicted_img1 = predict(test_img1)
predicted_img2 = predict(test_img2)
print("Prediction complete")
#create a figure of 2 plots (one for each test image)
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))
#display test image1 result
ax1.imshow(cv2.cvtColor(predicted_img1, cv2.COLOR_BGR2RGB))
#display test image2 result
ax2.imshow(cv2.cvtColor(predicted_img2, cv2.COLOR_BGR2RGB))
#display both images
cv2.imshow("Tom cruise test", predicted_img1)
cv2.imshow("Shahrukh Khan test", predicted_img2)
cv2.waitKey(0)
cv2.destroyAllWindows()
cv2.waitKey(1)
cv2.destroyAllWindows()
Predicting images...
Prediction complete
wohooo! Is'nt it beautiful? Indeed, it is!
Face Recognition is a fascinating idea to work on and OpenCV has made it extremely simple and easy for us to code it. It just takes a few lines of code to have a fully working face recognition application and we can switch between all three face recognizers with a single line of code change. It's that simple.
Although EigenFaces, FisherFaces and LBPH face recognizers are good but there are even better ways to perform face recognition like using Histogram of Oriented Gradients (HOGs) and Neural Networks. So the more advanced face recognition algorithms are now a days implemented using a combination of OpenCV and Machine learning. I have plans to write some articles on those more advanced methods as well, so stay tuned!
Download Details:
Author: informramiz
Source Code: https://github.com/informramiz/opencv-face-recognition-python
License: MIT License