1596422520
Using String Methods in Pandas
Often in a pandas dataframe we have columns that contain string values. Luckily, pandas provides an easy way of applying string methods to whole columns which are just pandas series objects. We can access the values of these series objects (or columns) as strings and apply string methods to them by using the str attribute of the series. Let’s look at some examples here.
Creating a DataFrame
We will be using the ufo reports found here to create our dataframe. We then will be using the dataframe head method to look at the top 5 rows of our dataframe and the dataframe info method to inspect our column values.
The info method output tells us that all the non-null values in our columns are most likely string objects. We will mainly focus on the City column in this tutorial.
Applying String Methods to a Column
Remember that a pandas dataframe is made up of columns. Each of these columns is a pandas Series object. In pandas, there are methods and attributes of these dataframe and series objects that we can access. Pandas provides us with a number of Series methods designed to work on strings. These string methods will be applied to every single entry in your column.
To apply a string method to a column, we will be using the str attribute of that Series object. So in general, we will be using the following format:
Series.str.<function/property>
Let’s say that we want to convert all the city names in our ufo dataframe to uppercase. First we would need to access that series (or column), then add .str, and finally add the specific method we want to use. To find all the string methods that we have available, go here and locate the String handling section.
We can access the series or column using bracket or dot notation as follows:
# bracket notation
ufo[‘City’]
# dot notation
ufo.City
#machine-learning #python #programming #data-science #data #data analysisa
What is GEEK
Buddha Community
1667425440
Pdf2gerb: Perl Script Converts PDF Files to Gerber format
pdf2gerb
Perl script converts PDF files to Gerber format
Pdf2Gerb generates Gerber 274X photoplotting and Excellon drill files from PDFs of a PCB. Up to three PDFs are used: the top copper layer, the bottom copper layer (for 2-sided PCBs), and an optional silk screen layer. The PDFs can be created directly from any PDF drawing software, or a PDF print driver can be used to capture the Print output if the drawing software does not directly support output to PDF.
The general workflow is as follows:
- Design the PCB using your favorite CAD or drawing software.
- Print the top and bottom copper and top silk screen layers to a PDF file.
- Run Pdf2Gerb on the PDFs to create Gerber and Excellon files.
- Use a Gerber viewer to double-check the output against the original PCB design.
- Make adjustments as needed.
- Submit the files to a PCB manufacturer.
Please note that Pdf2Gerb does NOT perform DRC (Design Rule Checks), as these will vary according to individual PCB manufacturer conventions and capabilities. Also note that Pdf2Gerb is not perfect, so the output files must always be checked before submitting them. As of version 1.6, Pdf2Gerb supports most PCB elements, such as round and square pads, round holes, traces, SMD pads, ground planes, no-fill areas, and panelization. However, because it interprets the graphical output of a Print function, there are limitations in what it can recognize (or there may be bugs).
See docs/Pdf2Gerb.pdf for install/setup, config, usage, and other info.
pdf2gerb_cfg.pm
#Pdf2Gerb config settings:
#Put this file in same folder/directory as pdf2gerb.pl itself (global settings),
#or copy to another folder/directory with PDFs if you want PCB-specific settings.
#There is only one user of this file, so we don't need a custom package or namespace.
#NOTE: all constants defined in here will be added to main namespace.
#package pdf2gerb_cfg;
use strict; #trap undef vars (easier debug)
use warnings; #other useful info (easier debug)
##############################################################################################
#configurable settings:
#change values here instead of in main pfg2gerb.pl file
use constant WANT_COLORS => ($^O !~ m/Win/); #ANSI colors no worky on Windows? this must be set < first DebugPrint() call
#just a little warning; set realistic expectations:
#DebugPrint("${\(CYAN)}Pdf2Gerb.pl ${\(VERSION)}, $^O O/S\n${\(YELLOW)}${\(BOLD)}${\(ITALIC)}This is EXPERIMENTAL software. \nGerber files MAY CONTAIN ERRORS. Please CHECK them before fabrication!${\(RESET)}", 0); #if WANT_DEBUG
use constant METRIC => FALSE; #set to TRUE for metric units (only affect final numbers in output files, not internal arithmetic)
use constant APERTURE_LIMIT => 0; #34; #max #apertures to use; generate warnings if too many apertures are used (0 to not check)
use constant DRILL_FMT => '2.4'; #'2.3'; #'2.4' is the default for PCB fab; change to '2.3' for CNC
use constant WANT_DEBUG => 0; #10; #level of debug wanted; higher == more, lower == less, 0 == none
use constant GERBER_DEBUG => 0; #level of debug to include in Gerber file; DON'T USE FOR FABRICATION
use constant WANT_STREAMS => FALSE; #TRUE; #save decompressed streams to files (for debug)
use constant WANT_ALLINPUT => FALSE; #TRUE; #save entire input stream (for debug ONLY)
#DebugPrint(sprintf("${\(CYAN)}DEBUG: stdout %d, gerber %d, want streams? %d, all input? %d, O/S: $^O, Perl: $]${\(RESET)}\n", WANT_DEBUG, GERBER_DEBUG, WANT_STREAMS, WANT_ALLINPUT), 1);
#DebugPrint(sprintf("max int = %d, min int = %d\n", MAXINT, MININT), 1);
#define standard trace and pad sizes to reduce scaling or PDF rendering errors:
#This avoids weird aperture settings and replaces them with more standardized values.
#(I'm not sure how photoplotters handle strange sizes).
#Fewer choices here gives more accurate mapping in the final Gerber files.
#units are in inches
use constant TOOL_SIZES => #add more as desired
(
#round or square pads (> 0) and drills (< 0):
.010, -.001, #tiny pads for SMD; dummy drill size (too small for practical use, but needed so StandardTool will use this entry)
.031, -.014, #used for vias
.041, -.020, #smallest non-filled plated hole
.051, -.025,
.056, -.029, #useful for IC pins
.070, -.033,
.075, -.040, #heavier leads
# .090, -.043, #NOTE: 600 dpi is not high enough resolution to reliably distinguish between .043" and .046", so choose 1 of the 2 here
.100, -.046,
.115, -.052,
.130, -.061,
.140, -.067,
.150, -.079,
.175, -.088,
.190, -.093,
.200, -.100,
.220, -.110,
.160, -.125, #useful for mounting holes
#some additional pad sizes without holes (repeat a previous hole size if you just want the pad size):
.090, -.040, #want a .090 pad option, but use dummy hole size
.065, -.040, #.065 x .065 rect pad
.035, -.040, #.035 x .065 rect pad
#traces:
.001, #too thin for real traces; use only for board outlines
.006, #minimum real trace width; mainly used for text
.008, #mainly used for mid-sized text, not traces
.010, #minimum recommended trace width for low-current signals
.012,
.015, #moderate low-voltage current
.020, #heavier trace for power, ground (even if a lighter one is adequate)
.025,
.030, #heavy-current traces; be careful with these ones!
.040,
.050,
.060,
.080,
.100,
.120,
);
#Areas larger than the values below will be filled with parallel lines:
#This cuts down on the number of aperture sizes used.
#Set to 0 to always use an aperture or drill, regardless of size.
use constant { MAX_APERTURE => max((TOOL_SIZES)) + .004, MAX_DRILL => -min((TOOL_SIZES)) + .004 }; #max aperture and drill sizes (plus a little tolerance)
#DebugPrint(sprintf("using %d standard tool sizes: %s, max aper %.3f, max drill %.3f\n", scalar((TOOL_SIZES)), join(", ", (TOOL_SIZES)), MAX_APERTURE, MAX_DRILL), 1);
#NOTE: Compare the PDF to the original CAD file to check the accuracy of the PDF rendering and parsing!
#for example, the CAD software I used generated the following circles for holes:
#CAD hole size: parsed PDF diameter: error:
# .014 .016 +.002
# .020 .02267 +.00267
# .025 .026 +.001
# .029 .03167 +.00267
# .033 .036 +.003
# .040 .04267 +.00267
#This was usually ~ .002" - .003" too big compared to the hole as displayed in the CAD software.
#To compensate for PDF rendering errors (either during CAD Print function or PDF parsing logic), adjust the values below as needed.
#units are pixels; for example, a value of 2.4 at 600 dpi = .0004 inch, 2 at 600 dpi = .0033"
use constant
{
HOLE_ADJUST => -0.004 * 600, #-2.6, #holes seemed to be slightly oversized (by .002" - .004"), so shrink them a little
RNDPAD_ADJUST => -0.003 * 600, #-2, #-2.4, #round pads seemed to be slightly oversized, so shrink them a little
SQRPAD_ADJUST => +0.001 * 600, #+.5, #square pads are sometimes too small by .00067, so bump them up a little
RECTPAD_ADJUST => 0, #(pixels) rectangular pads seem to be okay? (not tested much)
TRACE_ADJUST => 0, #(pixels) traces seemed to be okay?
REDUCE_TOLERANCE => .001, #(inches) allow this much variation when reducing circles and rects
};
#Also, my CAD's Print function or the PDF print driver I used was a little off for circles, so define some additional adjustment values here:
#Values are added to X/Y coordinates; units are pixels; for example, a value of 1 at 600 dpi would be ~= .002 inch
use constant
{
CIRCLE_ADJUST_MINX => 0,
CIRCLE_ADJUST_MINY => -0.001 * 600, #-1, #circles were a little too high, so nudge them a little lower
CIRCLE_ADJUST_MAXX => +0.001 * 600, #+1, #circles were a little too far to the left, so nudge them a little to the right
CIRCLE_ADJUST_MAXY => 0,
SUBST_CIRCLE_CLIPRECT => FALSE, #generate circle and substitute for clip rects (to compensate for the way some CAD software draws circles)
WANT_CLIPRECT => TRUE, #FALSE, #AI doesn't need clip rect at all? should be on normally?
RECT_COMPLETION => FALSE, #TRUE, #fill in 4th side of rect when 3 sides found
};
#allow .012 clearance around pads for solder mask:
#This value effectively adjusts pad sizes in the TOOL_SIZES list above (only for solder mask layers).
use constant SOLDER_MARGIN => +.012; #units are inches
#line join/cap styles:
use constant
{
CAP_NONE => 0, #butt (none); line is exact length
CAP_ROUND => 1, #round cap/join; line overhangs by a semi-circle at either end
CAP_SQUARE => 2, #square cap/join; line overhangs by a half square on either end
CAP_OVERRIDE => FALSE, #cap style overrides drawing logic
};
#number of elements in each shape type:
use constant
{
RECT_SHAPELEN => 6, #x0, y0, x1, y1, count, "rect" (start, end corners)
LINE_SHAPELEN => 6, #x0, y0, x1, y1, count, "line" (line seg)
CURVE_SHAPELEN => 10, #xstart, ystart, x0, y0, x1, y1, xend, yend, count, "curve" (bezier 2 points)
CIRCLE_SHAPELEN => 5, #x, y, 5, count, "circle" (center + radius)
};
#const my %SHAPELEN =
#Readonly my %SHAPELEN =>
our %SHAPELEN =
(
rect => RECT_SHAPELEN,
line => LINE_SHAPELEN,
curve => CURVE_SHAPELEN,
circle => CIRCLE_SHAPELEN,
);
#panelization:
#This will repeat the entire body the number of times indicated along the X or Y axes (files grow accordingly).
#Display elements that overhang PCB boundary can be squashed or left as-is (typically text or other silk screen markings).
#Set "overhangs" TRUE to allow overhangs, FALSE to truncate them.
#xpad and ypad allow margins to be added around outer edge of panelized PCB.
use constant PANELIZE => {'x' => 1, 'y' => 1, 'xpad' => 0, 'ypad' => 0, 'overhangs' => TRUE}; #number of times to repeat in X and Y directions
# Set this to 1 if you need TurboCAD support.
#$turboCAD = FALSE; #is this still needed as an option?
#CIRCAD pad generation uses an appropriate aperture, then moves it (stroke) "a little" - we use this to find pads and distinguish them from PCB holes.
use constant PAD_STROKE => 0.3; #0.0005 * 600; #units are pixels
#convert very short traces to pads or holes:
use constant TRACE_MINLEN => .001; #units are inches
#use constant ALWAYS_XY => TRUE; #FALSE; #force XY even if X or Y doesn't change; NOTE: needs to be TRUE for all pads to show in FlatCAM and ViewPlot
use constant REMOVE_POLARITY => FALSE; #TRUE; #set to remove subtractive (negative) polarity; NOTE: must be FALSE for ground planes
#PDF uses "points", each point = 1/72 inch
#combined with a PDF scale factor of .12, this gives 600 dpi resolution (1/72 * .12 = 600 dpi)
use constant INCHES_PER_POINT => 1/72; #0.0138888889; #multiply point-size by this to get inches
# The precision used when computing a bezier curve. Higher numbers are more precise but slower (and generate larger files).
#$bezierPrecision = 100;
use constant BEZIER_PRECISION => 36; #100; #use const; reduced for faster rendering (mainly used for silk screen and thermal pads)
# Ground planes and silk screen or larger copper rectangles or circles are filled line-by-line using this resolution.
use constant FILL_WIDTH => .01; #fill at most 0.01 inch at a time
# The max number of characters to read into memory
use constant MAX_BYTES => 10 * M; #bumped up to 10 MB, use const
use constant DUP_DRILL1 => TRUE; #FALSE; #kludge: ViewPlot doesn't load drill files that are too small so duplicate first tool
my $runtime = time(); #Time::HiRes::gettimeofday(); #measure my execution time
print STDERR "Loaded config settings from '${\(__FILE__)}'.\n";
1; #last value must be truthful to indicate successful load
#############################################################################################
#junk/experiment:
#use Package::Constants;
#use Exporter qw(import); #https://perldoc.perl.org/Exporter.html
#my $caller = "pdf2gerb::";
#sub cfg
#{
# my $proto = shift;
# my $class = ref($proto) || $proto;
# my $settings =
# {
# $WANT_DEBUG => 990, #10; #level of debug wanted; higher == more, lower == less, 0 == none
# };
# bless($settings, $class);
# return $settings;
#}
#use constant HELLO => "hi there2"; #"main::HELLO" => "hi there";
#use constant GOODBYE => 14; #"main::GOODBYE" => 12;
#print STDERR "read cfg file\n";
#our @EXPORT_OK = Package::Constants->list(__PACKAGE__); #https://www.perlmonks.org/?node_id=1072691; NOTE: "_OK" skips short/common names
#print STDERR scalar(@EXPORT_OK) . " consts exported:\n";
#foreach(@EXPORT_OK) { print STDERR "$_\n"; }
#my $val = main::thing("xyz");
#print STDERR "caller gave me $val\n";
#foreach my $arg (@ARGV) { print STDERR "arg $arg\n"; }Download Details:
Author: swannman
Source Code: https://github.com/swannman/pdf2gerb
License: GPL-3.0 license
1624693889
9 Useful Pandas Methods You Might Have Not Heard About
They can make your daily work easier and faster.
In this article, I wanted to quickly show a few useful pandas methods/functions, which can come in handy during your daily work. To manage expectations, this is not an article showing the basic functionalities of pandas and there is no particular theme to the methods. Without further ado, let’s start!
1. hasnans
2. transform
3. merge_asof
4. insert
…
#pandas #data-wrangling #python #data-science #9 useful pandas methods you might have not heard about #pandas methods
1623897480
Practice Problems: How To Use Pandas DataFrames' GroupBy Method
It’s now time for some practice problems! See below for details on how to proceed.
Course Repository & Practice Problems
All of the code for this course’s practice problems can be found in this GitHub repository.
There are two options that you can use to complete the practice problems:
- Open them in your browser with a platform called Binder using this link (recommended)
- Download the repository to your local computer and open them in a Jupyter Notebook using Anaconda (a bit more tedious)
Note that binder can take up to a minute to load the repository, so please be patient.
Within that repository, there is a folder called starter-files and a folder called finished-files. You should open the appropriate practice problems within the starter-files folder and only consult the corresponding file in the finished-files folder if you get stuck.
The repository is public, which means that you can suggest changes using a pull request later in this course if you’d like.
#pandas #groupby methods #pandas dataframe #example #practice problems: how to use pandas dataframes' groupby method #practice problems
1660032308
輕鬆過濾 Pandas 數據框的 14 種方法
每當我們處理任何類型的數據時，我們都需要清楚地了解我們正在處理的數據類型。對於那裡的大多數數據，其中可能包含數千甚至數百萬個包含各種信息的條目，如果沒有任何工具以簡短易讀的格式呈現數據，就真的不可能理解這些數據。
大多數時候，我們需要瀏覽數據、操作數據並將其可視化以獲得洞察力。嗯，有一個很棒的庫，它的名字叫 pandas，它為我們提供了這種能力。最常見的數據操作操作是數據過濾。它與 SQL 中的 WHERE 子句非常相似，或者您必須在 MS Excel 中使用過濾器來根據某些條件選擇特定行。
pandas是一個強大、靈活和開源的數據分析/操作工具，它本質上是一個python 包，提供速度、靈活性和富有表現力的數據結構，以直觀和簡單的方式處理關係”或“標記它是在 Python 中執行實際數據分析的最流行的庫
pandas建立在 NumPy 庫之上，旨在與科學計算環境和眾多其他第三方庫很好地集成。它有兩個主要數據結構，即Series (1D)和Dataframes(2D)，在大多數實際用例中，這是金融、科學計算、工程和統計等許多領域正在處理的數據類型。
讓我們開始在 Pandas Dataframe 的幫助下過濾數據
安裝熊貓
!pip install pandas導入 Pandas 庫，讀取我們的示例數據文件並將其分配給“df” DataFrame
import pandas as pd
df = pd.read_csv(r"C:\Users\rajam\Desktop\sample_data.csv")讓我們看看我們的數據框：
print(df.head())
樣本數據
現在我們有了 DataFrame，我們將應用各種方法來過濾它。
方法 - 1：按列值過濾 DataFrame
我們的 DataFrame 中有一個名為“Total_Sales”的列，我們想要過濾掉所有大於 300 的銷售額。
#Filter a DataFrame for a single column value with a given condition
greater_than = df[df['Total_Sales'] > 300]
print(greater_than.head())
銷售額超過 300
Method – 2 : Filtering DataFrame based on multiple conditions
在這裡，我們過濾“Total_Sales”值大於 300 以及“Units”大於 20 的所有值。我們將不得不使用執行按位與操作的 python 運算符“&”以顯示相應的結果。
#Filter a DataFrame with multiple conditions
filter_sales_units = df[(df['Total_Sales'] > 300) & (df["Units"] > 20)]
print(Filter_sales_units.head())
篩選銷售額和單位
方法 – 3：根據日期值過濾 DataFrame
如果我們想根據某個日期值過濾我們的數據框，例如這裡我們試圖獲取基於特定日期的所有結果，在我們的例子中是日期 '03/10/21' 之後的結果。
#Filter a DataFrame based on specific date
date_filter = df[df['Date'] > '03/10/21']
print(date_filter.head())
按日期過濾
方法四：基於Date值多條件過濾DataFrame
在這裡，我們得到了評估多個日期的 Date 操作的所有結果。
#Filter a DataFrame with multiple conditions our Date value
date_filter2 = df[(df['Date'] >= '3/25/2021') & (df['Date'] <'8/17/2021')]
print(date_filter2.head())
篩選具有多個條件的日期
方法五：根據特定字符串過濾DataFrame
在這裡，我們選擇一個名為“Region”的列並獲取來自“East”區域的所有行，從而根據特定的字符串值進行過濾。
#Filter a DataFrame to a specific string
east = df[df['Region'] == 'East']
print(east.head())
根據特定字符串過濾
方法6：根據字符串中的特定索引值過濾 DataFrame
在這裡，我們選擇一個名為“Region”的列，並獲取所有以字母“E”作為第一個字符的行，即指定列結果中索引 0 處的所有行。
#Filter a DataFrame to show rows starting with a specfic letter
starting_with_e = df[df['Region'].str[0]== 'E']
print(starting_with_e.head())
根據特定字母過濾
方法7：根據值列表過濾 DataFrame
在這裡，我們過濾包含值“West”和“East”的“Region”列中的行，並顯示組合結果。可以使用兩種方法來執行此過濾，即使用管道 | 具有相應所需值集的運算符具有以下語法，或者我們可以使用.isin()函數過濾給定列中的值，在我們的例子中是“區域”，並提供所需集的列表它裡面的值作為一個列表。
#Filter a DataFrame rows based on list of values
#Method 1:
east_west = df[(df['Region'] == 'West') | (df['Region'] == 'East')]
print(east_west)
#Method 2:
east_west_1 = df[df['Region'].isin(['West', 'East'])]
print(east_west_1.head())
方法-2的輸出
方法 – 8：使用 RegEx 根據特定值過濾 DataFrame 行
在這裡，我們想要列 'Region' 中的所有值，並在其字符串值中以 'th'結尾並顯示它們。換句話說，我們希望我們的結果顯示 'Nor th ' 和 'Sout th ' 的值並忽略 'East' 和 'West'。具有指定值的方法.str.contains()以及$ RegEx 模式可用於獲得所需的結果。
有關更多信息，請查看正則表達式文檔
#Filtering the DataFrame rows using regular expressions(REGEX)
regex_df = df[df['Region'].str.contains('th$')]
print(regex_df.head())
基於 REGEX 的過濾器
方法9：過濾 DataFrame 以檢查null
在這裡，我們將在isnull() 函數的幫助下檢查所有列中的空值和非空值。
#Filtering to check for null and not null values in all columns
df_null = df[df.isnull().any(axis=1)]
print(df_null.head())
基於 NULL 或 NOT 空值過濾
方法 - 10：過濾 DataFrame 以檢查特定列中的空值。
#Filtering to check for null values if any in the 'Units' column
units_df = df[df['Units'].isnull()]
print(units_df.head())
在特定列上查找空值
方法 – 11：過濾 DataFrame 以檢查特定列中的非 空值
#Filtering to check for not null values in the 'Units' column
df_not_null = df[df['Units'].notnull()]
print(df_not_null.head())
在特定列上查找非空值
Method – 12: Filtering DataFrame using query()with a condition
#Using query function in pandas
df_query = df.query('Total_Sales > 300')
print(df_query.head())
Query使用函數過濾值
Method – 13: Filtering DataFrame using query()with multiple conditions
#Using query function with multiple conditions in pandas
df_query_1 = df.query('Total_Sales > 300 and Units <18')
print(df_query_1.head())
Query使用函數過濾多列
方法 –loc 14：使用和iloc函數過濾我們的 DataFrame 。
#Creating a sample DataFrame for illustrations
import numpy as np
data = pd.DataFrame({"col1" : np.arange(1, 20 ,2)}, index=[19, 18 ,8, 6, 0, 1, 2, 3, 4, 5])
print(data)
樣本數據
解釋：iloc 根據給定索引的位置考慮行，因此它僅將整數作為值。
有關更多信息，請查看Pandas 文檔
#Filter with iloc
data.iloc[0 : 5]
過濾使用iloc
說明：loc 考慮基於索引標籤的行
#Filter with loc
data.loc[0 : 5]
過濾使用loc
您可能正在思考為什麼loc函數返回 6 行而不是 5 行。這是因為不會根據索引位置產生輸出。它只考慮索引標籤，它也可以是字母表，包括起點和終點。 loc
結論
因此，這些是 pandas 中最常用的一些過濾方法。還有許多其他過濾方法可以使用，但這些是最常見的一些。
鏈接：https ://www.askpython.com/python-modules/pandas/filter-pandas-dataframe
#pandas #python #datafame
1660147320
Top 14 Ways to Filter Pandas Dataframes Easily
Whenever we work with data of any sort, we need a clear picture of the kind of data that we are dealing with. For most of the data out there, which may contain thousands or even millions of entries with a wide variety of information, it’s really impossible to make sense of that data without any tool to present the data in a short and readable format.
Most of the time we need to go through the data, manipulate it, and visualize it for getting insights. Well, there is a great library which goes by the name pandas which provides us with that capability. The most frequent Data manipulation operation is Data Filtering. It is very similar to the WHERE clause in SQL or you must have used a filter in MS Excel for selecting specific rows based on some conditions.
pandas is a powerful, flexible and open source data analysis/manipulation tool which is essentially a python package that provides speed, flexibility and expressive data structures crafted to work with “relational” or “labelled” data in an intuitive and easy manner. It is one of the most popular libraries to perform real-world data analysis in Python.
pandas is built on top of the NumPy library which aims to integrate well with the scientific computing environment and numerous other 3rd party libraries. It has two primary data structures namely Series (1D) and Dataframes(2D), which in most real-world use cases is the type of data that is being dealt with in many sectors of finance, scientific computing, engineering and statistics.
Let’s Start Filtering Data With the Help of Pandas Dataframe
Installing pandas
!pip install pandasImporting the Pandas library, reading our sample data file and assigning it to “df” DataFrame
import pandas as pd
df = pd.read_csv(r"C:\Users\rajam\Desktop\sample_data.csv")Let’s check out our dataframe:
print(df.head())
Sample_data
Now that we have our DataFrame, we will be applying various methods to filter it.
Method – 1: Filtering DataFrame by column value
We have a column named “Total_Sales” in our DataFrame and we want to filter out all the sales value which is greater than 300.
#Filter a DataFrame for a single column value with a given condition
greater_than = df[df['Total_Sales'] > 300]
print(greater_than.head())
Sales with Greater than 300
Method – 2: Filtering DataFrame based on multiple conditions
Here we are filtering all the values whose “Total_Sales” value is greater than 300 and also where the “Units” is greater than 20. We will have to use the python operator “&” which performs a bitwise AND operation in order to display the corresponding result.
#Filter a DataFrame with multiple conditions
filter_sales_units = df[(df['Total_Sales'] > 300) & (df["Units"] > 20)]
print(Filter_sales_units.head())
Filter on Sales and Units
Method – 3: Filtering DataFrame based on Date value
If we want to filter our data frame based on a certain date value, for example here we are trying to get all the results based on a particular date, in our case the results after the date ’03/10/21′.
#Filter a DataFrame based on specific date
date_filter = df[df['Date'] > '03/10/21']
print(date_filter.head())
Filter on Date
Method – 4: Filtering DataFrame based on Date value with multiple conditions
Here we are getting all the results for our Date operation evaluating multiple dates.
#Filter a DataFrame with multiple conditions our Date value
date_filter2 = df[(df['Date'] >= '3/25/2021') & (df['Date'] <'8/17/2021')]
print(date_filter2.head())
Filter on a date with multiple conditions
Method – 5: Filtering DataFrame based on a specific string
Here we are selecting a column called ‘Region’ and getting all the rows that are from the region ‘East’, thus filtering based on a specific string value.
#Filter a DataFrame to a specific string
east = df[df['Region'] == 'East']
print(east.head())
Filter based on a specific string
Method – 6: Filtering DataFrame based on a specific index value in a string
Here we are selecting a column called ‘Region’ and getting all the rows which has the letter ‘E’ as the first character i.e at index 0 in the specified column results.
#Filter a DataFrame to show rows starting with a specfic letter
starting_with_e = df[df['Region'].str[0]== 'E']
print(starting_with_e.head())
Filter based on a specific letter
Method – 7: Filtering DataFrame based on a list of values
Here we are filtering rows in the column ‘Region’ which contains the values ‘West’ as well as ‘East’ and display the combined result. Two methods can be used to perform this filtering namely using a pipe | operator with the corresponding desired set of values with the below syntax OR we can use the .isin() function to filter for the values in a given column, which in our case is the ‘Region’, and provide the list of the desired set of values inside it as a list.
#Filter a DataFrame rows based on list of values
#Method 1:
east_west = df[(df['Region'] == 'West') | (df['Region'] == 'East')]
print(east_west)
#Method 2:
east_west_1 = df[df['Region'].isin(['West', 'East'])]
print(east_west_1.head())
Output of Method -2
Method – 8: Filtering DataFrame rows based on specific values using RegEx
Here we want all the values in the column ‘Region’, which ends with ‘th’ in their string value and display them. In other words, we want our results to show the values of ‘North‘ and ‘South‘ and ignore ‘East’ and ‘West’. The method .str.contains() with the specified values along with the $ RegEx pattern can be used to get the desired results.
For more information please check the Regex Documentation
#Filtering the DataFrame rows using regular expressions(REGEX)
regex_df = df[df['Region'].str.contains('th$')]
print(regex_df.head())
Filter based on REGEX
Method – 9: Filtering DataFrame to check for null
Here, we’ll check for null and not null values in all the columns with the help of isnull() function.
#Filtering to check for null and not null values in all columns
df_null = df[df.isnull().any(axis=1)]
print(df_null.head())
Filter based on NULL or NOT null values
Method – 10: Filtering DataFrame to check for null values in a specific column.
#Filtering to check for null values if any in the 'Units' column
units_df = df[df['Units'].isnull()]
print(units_df.head())
Finding null values on specific columns
Method – 11: Filtering DataFrame to check for not null values in specific columns
#Filtering to check for not null values in the 'Units' column
df_not_null = df[df['Units'].notnull()]
print(df_not_null.head())
Finding not-null values on specific columns
Method – 12: Filtering DataFrame using query() with a condition
#Using query function in pandas
df_query = df.query('Total_Sales > 300')
print(df_query.head())
Filtering values with Query Function
Method – 13: Filtering DataFrame using query() with multiple conditions
#Using query function with multiple conditions in pandas
df_query_1 = df.query('Total_Sales > 300 and Units <18')
print(df_query_1.head())
Filtering multiple columns with Query Function
Method – 14: Filtering our DataFrame using the loc and iloc functions.
#Creating a sample DataFrame for illustrations
import numpy as np
data = pd.DataFrame({"col1" : np.arange(1, 20 ,2)}, index=[19, 18 ,8, 6, 0, 1, 2, 3, 4, 5])
print(data)
sample_data
Explanation: iloc considers rows based on the position of the given index, so that it takes only integers as values.
For more information please check out Pandas Documentation
#Filter with iloc
data.iloc[0 : 5]
Filter using iloc
Explanation: loc considers rows based on index labels
#Filter with loc
data.loc[0 : 5]
Filter using loc
You might be thinking about why the loc function returns 6 rows instead of 5 rows. This is because loc does not produce output based on index position. It considers labels of index only which can be an alphabet as well and includes both starting and endpoint.
Conclusion
So, these were some of the most common filtering methods used in pandas. There are many other filtering methods that could be used, but these are some of the most common.
Link: https://www.askpython.com/python-modules/pandas/filter-pandas-dataframe
#pandas #python #datafame