1612369200
Python: How to Handle Missing Data in Pandas DataFrame
Pandas is a Python library for data analysis and manipulation. Almost all operations in pandas revolve around DataFrames, an abstract data structure tailor-made for handling a metric ton of data.
In the aforementioned metric ton of data, some of it is bound to be missing for various reasons. Resulting in a missing (null/None/Nan) value in our DataFrame.
Which is why, in this article, we’ll be discussing how to handle missing data in a Pandas DataFrame.
Data Inspection
Real-world datasets are rarely perfect. They may contain missing values, wrong data types, unreadable characters, erroneous lines, etc.
The first step to to any proper data analysis is cleaning and organizing the data we’ll later be using. We will discuss a few common problems related to data that might occur in a dataset.
We will be working with small employees dataset for this. The .csv file looks like this:
First Name,Gender,Salary,Bonus %,Senior Management,Team
Douglas,Male,97308,6.945,TRUE,Marketing
Thomas,Male,61933,NaN,TRUE,
Jerry,Male,NA,9.34,TRUE,Finance
Dennis,n.a.,115163,10.125,FALSE,Legal
,Female,0,11.598,,Finance
Angela,,,18.523,TRUE,Engineering
Shawn,Male,111737,6.414,FALSE,na
Rachel,Female,142032,12.599,FALSE,Business Development
Linda,Female,57427,9.557,TRUE,Client Services
Stephanie,Female,36844,5.574,TRUE,Business Development
,,,,,
Let’s import it into a DataFrame:
df = pd.read_csv('out.csv')
df
#python #pandas #tool
What is GEEK
Buddha Community
1620466520
Your Data Architecture: Simple Best Practices for Your Data Strategy
If you accumulate data on which you base your decision-making as an organization, you should probably think about your data architecture and possible best practices.
If you accumulate data on which you base your decision-making as an organization, you most probably need to think about your data architecture and consider possible best practices. Gaining a competitive edge, remaining customer-centric to the greatest extent possible, and streamlining processes to get on-the-button outcomes can all be traced back to an organization’s capacity to build a future-ready data architecture.
In what follows, we offer a short overview of the overarching capabilities of data architecture. These include user-centricity, elasticity, robustness, and the capacity to ensure the seamless flow of data at all times. Added to these are automation enablement, plus security and data governance considerations. These points from our checklist for what we perceive to be an anticipatory analytics ecosystem.
#big data #data science #big data analytics #data analysis #data architecture #data transformation #data platform #data strategy #cloud data platform #data acquisition
1593156510
Basic Data Types in Python | Python Web Development For Beginners
At the end of 2019, Python is one of the fastest-growing programming languages. More than 10% of developers have opted for Python development.
In the programming world, Data types play an important role. Each Variable is stored in different data types and responsible for various functions. Python had two different objects, and They are mutable and immutable objects.
Table of Contents hide
III Built-in data types in Python
Mutable objects
The Size and declared value and its sequence of the object can able to be modified called mutable objects.
Mutable Data Types are list, dict, set, byte array
Immutable objects
The Size and declared value and its sequence of the object can able to be modified.
Immutable data types are int, float, complex, String, tuples, bytes, and frozen sets.
id() and type() is used to know the Identity and data type of the object
a**=25+**85j
type**(a)**
output**:<class’complex’>**
b**={1:10,2:“Pinky”****}**
id**(b)**
output**:**238989244168
Built-in data types in Python
a**=str(“Hello python world”)****#str**
b**=int(18)****#int**
c**=float(20482.5)****#float**
d**=complex(5+85j)****#complex**
e**=list((“python”,“fast”,“growing”,“in”,2018))****#list**
f**=tuple((“python”,“easy”,“learning”))****#tuple**
g**=range(10)****#range**
h**=dict(name=“Vidu”,age=36)****#dict**
i**=set((“python”,“fast”,“growing”,“in”,2018))****#set**
j**=frozenset((“python”,“fast”,“growing”,“in”,2018))****#frozenset**
k**=bool(18)****#bool**
l**=bytes(8)****#bytes**
m**=bytearray(8)****#bytearray**
n**=memoryview(bytes(18))****#memoryview**
Numbers (int,Float,Complex)
Numbers are stored in numeric Types. when a number is assigned to a variable, Python creates Number objects.
#signed interger
age**=**18
print**(age)**
Output**:**18
Python supports 3 types of numeric data.
int (signed integers like 20, 2, 225, etc.)
float (float is used to store floating-point numbers like 9.8, 3.1444, 89.52, etc.)
complex (complex numbers like 8.94j, 4.0 + 7.3j, etc.)
A complex number contains an ordered pair, i.e., a + ib where a and b denote the real and imaginary parts respectively).
String
The string can be represented as the sequence of characters in the quotation marks. In python, to define strings we can use single, double, or triple quotes.
# String Handling
‘Hello Python’
#single (') Quoted String
“Hello Python”
# Double (") Quoted String
“”“Hello Python”“”
‘’‘Hello Python’‘’
# triple (‘’') (“”") Quoted String
In python, string handling is a straightforward task, and python provides various built-in functions and operators for representing strings.
The operator “+” is used to concatenate strings and “*” is used to repeat the string.
“Hello”+“python”
output**:****‘Hello python’**
"python "*****2
'Output : Python python ’
#python web development #data types in python #list of all python data types #python data types #python datatypes #python types #python variable type
1586702221
Python Pandas Objects - Pandas Series and Pandas Dataframe
In this post, we will learn about pandas’ data structures/objects. Pandas provide two type of data structures:-
Pandas Series
Pandas Series is a one dimensional indexed data, which can hold datatypes like integer, string, boolean, float, python object etc. A Pandas Series can hold only one data type at a time. The axis label of the data is called the index of the series. The labels need not to be unique but must be a hashable type. The index of the series can be integer, string and even time-series data. In general, Pandas Series is nothing but a column of an excel sheet with row index being the index of the series.
Pandas Dataframe
Pandas dataframe is a primary data structure of pandas. Pandas dataframe is a two-dimensional size mutable array with both flexible row indices and flexible column names. In general, it is just like an excel sheet or SQL table. It can also be seen as a python’s dict-like container for series objects.
#python #python-pandas #pandas-dataframe #pandas-series #pandas-tutorial
1624088808
How to Reduce the Size of a Pandas Dataframe in Python
Reducing the size of your data can sometimes be tricky. In this quick tutorial, we will demonstrate how to reduce the size of your dataframe in half by down casting allowing you to do more with less.
Background
Whether you are building a model for a hobby project or for work purposes, chances are that your first attempt will include opening up a jupyter notebook and reading in some data. Eventually you will undoubtly run into the memory issues with your notebook. Before you start dropping rows or try complex sampling techniques to reduce the size of your data, you should check the structure of the data.
#data-science #python #downcast #data #pandas #how to reduce the size of a pandas dataframe in python
1602550800
Pandas in Python
Pandas is used for data manipulation, analysis and cleaning.
What are Data Frames and Series?
Dataframe is a two dimensional, size mutable, potentially heterogeneous tabular data.
It contains rows and columns, arithmetic operations can be applied on both rows and columns.
Series is a one dimensional label array capable of holding data of any type. It can be integer, float, string, python objects etc. Panda series is nothing but a column in an excel sheet.
How to create dataframe and series?
s = pd.Series([1,2,3,4,56,np.nan,7,8,90])
print(s)
How to create a dataframe by passing a numpy array?
- d= pd.date_range(‘20200809’,periods=15)
- print(d)
- df = pd.DataFrame(np.random.randn(15,4), index= d, columns = [‘A’,’B’,’C’,’D’])
- print(df)
#pandas-series #pandas #pandas-in-python #pandas-dataframe #python