An Introduction to Pandas

An Introduction to Pandas

Key methods to understanding and utilizing Pandas. Pandas is an open-source Python Library providing high-performance data manipulation and analysis tool using its powerful data structures.

Key methods to understanding and utilizing pandas

I figure since you have found yourself navigating to this page that you probably have a good amount of data that you are looking to analyze, and you may possibly be wondering the best and most efficient way to answer some of your questions about your data. The answer to your questions can be found with the use of the Pandas package.

How to access Pandas

Due to the popularity of Pandas it has its own conventional abbreviation, so anytime you are importing pandas into python, use the nomenclature below:

import pandas as pd

Primary Use of Pandas package is the DataFrame

The pandas API defines a pandas dataframe as:

Two-dimensional, size-mutable, potentially heterogeneous tabular data. Data structure also contains labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure.

Basically, all that means is that you have data contained in the format to what you see below. Data found in rows and columns:

Image for post

Example dataframe with labels for data, rows and columns. Dataset from https://www.kaggle.com/yamaerenay/spotify-dataset-19212020-160k-tracks

DataFrames are extremely useful because they provide an efficient way to visualize data and then manipulate it the way you want it to be. The rows can be easily referenced by the index which are the numbers on the far left of the dataframe. The index will be the corresponding row’s number starting at zero, unless you specify names to each row. The columns can also be easily referenced by the column name such as “Track Name” or by their position in the dataframe. We will talk in more detail about referencing rows and columns later in this article.

Creation Time!

There are several ways to create a pandas dataframe:

  1. Import data from a .csv file (or other file type e.g. Excel, SQL database)
  2. from a list
  3. from a dictionary
  4. from a numpy array
  5. many, many more!

In general, you will mainly be putting data from a .csv file, or some type of data source into a pandas dataframe and not making it from scratch since that would take an insanely long time to do depending on the amount of data you have. Here is a quick, simple example from a python dictionary:

import pandas as pd
dict1 = {'Exercises': ['Running','Walking','Cycling'],
         'Mileage': [250, 1000, 550]}
df = pd.DataFrame(dict1)
df

Output:

Image for post

Basic dataframe made from code above

The dictionary keys (‘Exercises’ and ‘Mileage’) become the corresponding column headers. The values in the dictionary being the lists in this example became the individual data points in the dataframe. The order that the lists are in matters since Running will be placed in first since it is first in the ‘Exercises’ list and 250 will be placed in first in the second column since it is first in the ‘Mileage’ list. Also, you will notice that since I did not specify labels for the index of the dataframe it automatically was labelled 0,1, and 2.

However, like I said before the most likely way that you will be creating a pandas dataframe is from a csv or other type of file that you will import in to analyze the data. This is easily completed with just the following:

df = pd.read_csv("file_location.../file_name.csv")

pd.read_csv() is an extremely strong and versatile method and will be extremely useful depending on how you are looking to import your data. If your csv file already comes with headers or an index, you can specify this while importing and make you life so much easier. In order to understand the full ability of pd.read_csv() I suggest you look at the pandas API here.

First things first

Now you have this huge dataset ready to analyze, you have got to take a look at it and see what it looks like. As a person who is analyzing this data, you first have to become comfortable with the dataset, and really get to know what is going on in the dataset. There are four methods that I love to use in order to get to know my data and which pandas makes super easy.

  1. .head() & .tail()
  2. .info()
  3. .describe()
  4. .sample()
raw_song.head()

The line above is the line that I have in the picture at the top of the page. It with display the first 5 lines of the dataframe and each of the columns to give you an easy summary of what the data looks like. You can also specify a certain number of rows inside the () of the method to show more rows if your heart so desires.

Image for post

.head() method on song data from Spotify dataset

.tail() is the same just displaying the last 5 lines.

raw_song.tail()

Image for post

.tail() method on song data from Spotify dataset

From these two quick methods, I have a general idea or the column names and just what the data looks like just from a small sample of the dataset. These methods are also really useful especially given a dataset such as the Spotify dataset working with over 3 million rows, you can easily display the dataset and get a quick idea, and it won’t take your computer a long time to display the data.

.info() is also useful in that it shows me a succinct list of all of the columns, their datatypes, and whether you have any null datapoints or not.

raw_song.info(verbose=True, null_counts=True)

Image for post

.info() method on song data from Spotify dataset

If you have completely integer or float columns (i.e. ‘Position’, ‘Streams’), then .describe() can be a useful method to understand more about your dataset as it will show many descriptive statistics about those columns.

raw_song.describe()

Image for post

.describe() method on song data from Spotify dataset. Notice that only ‘Position’ and ‘Streams’ columns are shown since they were the only two integer columns, the other columns are strings and do not have descriptive stats.

Lastly, .sample() will allow you to randomly sample your dataframe and see if any manipulation that you made has incorrectly changed something in your dataset, and can also be great when first exploring your dataset just to get an idea of what the dataset contains exactly that was not already shown in the previous methods.

raw_song.sample(10)

Image for post

.sample() method on song data from Spotify dataset.

I use each of these methods consistently while exploring and preparing my datasets for analysis. Anytime I change the data in a column, change a column name, or add/delete a row/column I will then make sure it all changed the way I wanted it to by quickly running at least some of the previous 5 methods.

pandas python data-science data-analytics developer

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Basic Data Types in Python | Python Web Development For Beginners

In the programming world, Data types play an important role. Each Variable is stored in different data types and responsible for various functions. Python had two different objects, and They are mutable and immutable objects.

Hire Python Developers

Are you looking for experienced, reliable, and qualified Python developers? If yes, you have reached the right place. At **[HourlyDeveloper.io](https://hourlydeveloper.io/ "HourlyDeveloper.io")**, our full-stack Python development services...

Applied Data Science with Python Certification Training Course -IgmGuru

Master Applied Data Science with Python and get noticed by the top Hiring Companies with IgmGuru's Data Science with Python Certification Program. Enroll Now

Hire Python Developers India

Looking to build robust, scalable, and dynamic responsive websites and applications in Python? At **[HourlyDeveloper.io](https://hourlydeveloper.io/ "HourlyDeveloper.io")**, we constantly endeavor to give you exactly what you need. If you need to...

Data Science With Python | Python For Data Science | Data Science For Beginners

This Data Science with Python Tutorial will help you understand what is Data Science, basics of Python for data analysis, why learn Python, how to install Python, Python libraries for data analysis, exploratory analysis using Pandas, introduction to series and dataframe, loan prediction problem, data wrangling using Pandas, building a predictive model using Scikit-Learn and implementing logistic regression model using Python.