Using Python’s Garbage Collector with Pandas DataFrames

It’s almost 2021. Memory is inexpensive and it’s easy to access cloud platforms like Amazon Web Services (AWS) or Google Cloud Platform (GCP) and throw vast amount of resources at a data problem. And so, we usually don’t worry about memory (RAM) these days. But there are at least two problems with this line of thinking:

i) if we can use our resource efficiently, we can do more with the same amount of resources (i.e. save money!); and,

ii) “data has mass” in the sense that the rate at which large volume of data moves is slower than smaller volumes of data. In other words, smaller volume of data moves around faster, and hence, processed faster (i.e. save time and money!!).

There are several aspects of managing memory usage. To list a few, we have: garbage collection, the option to use certain data types over others, the option to use tries and directed acyclic word graphs, and option to use probabilistic data structures. Each of these deserves an article (or perhaps several articles!) on their own. So in this article, I’ll stick with just one of them: garbage collection. Often overlooked, it is one of the primary ways Python manages memory. It happens in the background without us doing anything special. But it is possible to control some aspects of it, and knowing them can be really useful when handling large amounts of data.

Before we can dive into the details, it may be useful to provide a little bit of background in variable names in python. In python, variable names are simply symbolic names that are pointers to objects. The schematic below illustrates this.

Internal Representation of Objects in Python [1]

Let’s define a and b separately:

a = "banana"
b = "banana" 

Now let’s take a look at the location of these two variables are referring to:

for name in [a,b]:
    print(object.__repr__(name))

<str object at 0x7f0f901548b0>
<str object at 0x7f0f901548b0>

#data-science #python

What is GEEK

Buddha Community

top 30 Python Tips and Tricks for Beginners

Welcome to my Blog , In this article, you are going to learn the top 10 python tips and tricks.

1) swap two numbers.

2) Reversing a string in Python.

3) Create a single string from all the elements in list.

4) Chaining Of Comparison Operators.

5) Print The File Path Of Imported Modules.

6) Return Multiple Values From Functions.

7) Find The Most Frequent Value In A List.

8) Check The Memory Usage Of An Object.

…

#python #python hacks tricks #python learning tips #python programming tricks #python tips #python tips and tricks #python tips and tricks advanced #python tips and tricks for beginners #python tips tricks and techniques #python tutorial #tips and tricks in python #tips to learn python #top 30 python tips and tricks for beginners

Lambda, Map, Filter functions in python

Welcome to my Blog, In this article, we will learn python lambda function, Map function, and filter function.

Lambda function in python: Lambda is a one line anonymous function and lambda takes any number of arguments but can only have one expression and python lambda syntax is

Syntax: x = lambda arguments : expression

Now i will show you some python lambda function examples:

#python #anonymous function python #filter function in python #lambda #lambda python 3 #map python #python filter #python filter lambda #python lambda #python lambda examples #python map

Python Pandas Objects - Pandas Series and Pandas Dataframe

In this post, we will learn about pandas’ data structures/objects. Pandas provide two type of data structures:-

Pandas Series

Pandas Series is a one dimensional indexed data, which can hold datatypes like integer, string, boolean, float, python object etc. A Pandas Series can hold only one data type at a time. The axis label of the data is called the index of the series. The labels need not to be unique but must be a hashable type. The index of the series can be integer, string and even time-series data. In general, Pandas Series is nothing but a column of an excel sheet with row index being the index of the series.

Pandas Dataframe

Pandas dataframe is a primary data structure of pandas. Pandas dataframe is a two-dimensional size mutable array with both flexible row indices and flexible column names. In general, it is just like an excel sheet or SQL table. It can also be seen as a python’s dict-like container for series objects.

#python #python-pandas #pandas-dataframe #pandas-series #pandas-tutorial

Pandas in Python

Pandas is used for data manipulation, analysis and cleaning.

What are Data Frames and Series?

Dataframe is a two dimensional, size mutable, potentially heterogeneous tabular data.

It contains rows and columns, arithmetic operations can be applied on both rows and columns.

Series is a one dimensional label array capable of holding data of any type. It can be integer, float, string, python objects etc. Panda series is nothing but a column in an excel sheet.

How to create dataframe and series?

s = pd.Series([1,2,3,4,56,np.nan,7,8,90])

print(s)

How to create a dataframe by passing a numpy array?

1. d= pd.date_range(‘20200809’,periods=15)
2. print(d)
3. df = pd.DataFrame(np.random.randn(15,4), index= d, columns = [‘A’,’B’,’C’,’D’])
4. print(df)

#pandas-series #pandas #pandas-in-python #pandas-dataframe #python

Python for Beginners #2 — Importing files to python with pandas

Use pandas to upload CSV, TXT and Excel files

Story time before we begin

Learning Python isn’t the easiest thing to do. But consistency is really the key to arriving at a level that boosts your career.

We hear a lot about millennials wanting things to easy. In reality, there are a lot of young professionals who believe that they can do more for their companies but are being held back by the work cultures they are faced with at the onset of their careers.

Having been lucky enough to have found a job after my studies, I remember immediately feeling a wave of disappointment a very short while after starting my new job. I felt like a cog in a massive machine. I wasn’t really anything other than a ‘resource’. An extra 8–15 hours of daily man power depending on my boss’ whim.

The result, was the eventual disenchantment and lack of motivation simply because, for the most part, I was expected to be quiet and do my job in the hope of one day being senior enough to effect significant changes. And while the older generation would generally tell me to suck it up, I couldn’t see myself sucking it up for 5 years or more. I knew I’d get stale and afraid of change, much like those telling me to stay in my place.

For anyone in a similar situation,**_ do your best to improve on your skills _**and find an environment that works for you. That’s the whole purpose of these articles. To get you on your way to freedom.

Introduction

For this demonstration, I’ll use data from this Kaggle competition. It’s a simple CSV file containing data on individuals in the Titanic and the different profiles i.e. (age, marital status etc.)

I want to import this file to python. I’ll show you how to do this alongside all the possible troubleshoots you may encounter.

Table of Contents

1. Where should you put your files?
2. Reading CSV and TXT files
3. Reading excel (XLSX) files

#python #programming #pandas #python for beginners #importing files to python with pandas #python for beginners #2 — importing files to python with pandas