Fannie  Zemlak

Fannie Zemlak


A Pandas DataFrame Processing CLI Tool


Quick Links


PdpCLI is a pandas DataFrame processing CLI tool which enables you to build a pandas pipeline powered by pdpipe from a configuration file. You can also extend pipeline stages and data readers / writers by using your own python scripts.


  • Process pandas DataFrame from CLI without wrting Python scripts
  • Support multiple configuration file formats: YAML, JSON, Jsonnet
  • Read / write data files in the following formats: CSV, TSV, JSON, JSONL, pickled DataFrame
  • Import / export data with multiple protocols: S3 / Databse (MySQL, Postgres, SQLite, …) / HTTP(S)
  • Extensible pipeline and data readers / writers


Installing the library is simple using pip.

$ pip install "pdpcli[all]"


Basic Usage

  1. Write a pipeline config file config.yml like below. The type fields under pipeline correspond to the snake-cased class names of the PdpipelineStages. Other fields such as stage and columns are the parameters of the __init__ methods of the corresponging classes. Internally, this configuration file is converted to Python objects by colt.
  type: pipeline
      type: col_drop
        - name
        - job

      type: one_hot_encode
      columns: sex

      type: tokenize_text
      columns: content

      type: tfidf_vectorize_token_lists
      column: content
      max_features: 10
  1. Build a pipeline by training on train.csv. The following command generages a pickled pipeline file pipeline.pkl after training. If you specify a URL of file path, it will be automatically downloaded and cached.
$ pdp build config.yml pipeline.pkl --input-file

  1. Apply the fitted pipeline to test.csv and get output of a processed file processed_test.jsonl by the following command. PdpCLI automatically detects the output file format based on the file name. In this example, the processed DataFrame will be exported as the JSON-Lines format.
$ pdp apply pipeline.pkl --output-file processed_test.jsonl

  1. You can also directly run the pipeline from a config file without fitting pipeline.
$ pdp apply config.yml test.csv --output-file processed_test.jsonl

  1. It is possible to override or add parameters by adding command line arguments:
pdp apply config.yml test.csv pipeline.stages.drop_columns.column=name

Data Reader / Writer

PdpCLI automatically detects a suitable data reader / writer based on a given file name. If you need to use the other data reader / writer, add a reader or writer config to config.yml. The following config is an exmaple to use SQL data reader. SQL reader fetches records from the specified database and converts them into a pandas DataFrame.

    type: sql
    dsn: postgres://${env:POSTGRES_USER}:${env:POSTGRES_PASSWORD}@your.posgres.server/your_database

Config files are interpreted by OmegaConf, so ${env:...} is interpolated by environment variables.

Prepare yuor SQL file query.sql to fetch data from the database:

select * from your_table limit 1000

You can execute the pipeline with SQL data reader via:

$ POSTGRES_USER=user POSTGRES_PASSWORD=password pdp apply config.yml query.sql


By using plugins, you can extend PdpCLI. This plugin feature enables you to use your own pipeline stages, data readers / writers and commands.

Add a new stage
  1. Write your plugin script like below. Stage.register("<stage-name>") registers your pipeline stages, and you can specify these stages by writing type: <stage-name> in your config file.
import pdpcli

class PrintStage(pdpcli.Stage):
    def _prec(self, df):
        return True

    def _transform(self, df, verbose):
        return df
  1. Update config.yml to use your plugin.
    type: pipeline

            type: print

  1. Execute command with --module mypdp and you can see the processed DataFrame after running drop_columns.
$ pdp apply config.yml test.csv --module mypdp

Add a new command

You can also add new commands not only stages.

  1. Add the following script to This greet command prints out a greeting message with your name.
    description="say hello",
    help="say hello",
class GreetCommand(pdpcli.Subcommand):
    requires_plugins = False

    def set_arguments(self):
        self.parser.add_argument("--name", default="world")

    def run(self, args):
        print(f"Hello, {}!")
  1. To register this command, you need to create the .pdpcli_plugins file in which module names are listed for each line. Due to module importing order, the --module option is unavailable for command registration.
$ echo "mypdp" > .pdpcli_plugins

  1. Run the following command and get a message like below. By using the .pdpcli_plugins file, it is is not needed to add the --module option to a command line for each execution.
$ pdp greet --name altescy
Hello, altescy!

Download Details:

Author: altescy
The Demo/Documentation: View The Demo/Documentation
Download Link: Download The Source Code
Official Website:
License: MIT

#pandas #python

What is GEEK

Buddha Community

A Pandas DataFrame Processing CLI Tool
Kasey  Turcotte

Kasey Turcotte


Pandas DataFrame vs. Spark DataFrame: When Parallel Computing Matters

With Performance Comparison Analysis and Guided Example of Animated 3D Wireframe Plot

Python is famous for its vast selection of libraries and resources from the open-source community. As a Data Analyst/Engineer/Scientist, one might be familiar with popular packages such as NumpyPandasScikit-learnKeras, and TensorFlow. Together these modules help us extract value out of data and propels the field of analytics. As data continue to become larger and more complex, one other element to consider is a framework dedicated to processing Big Data, such as Apache Spark. In this article, I will demonstrate the capabilities of distributed/cluster computing and present a comparison between the Pandas DataFrame and Spark DataFrame. My hope is to provide more conviction on choosing the right implementation.

Pandas DataFrame

Pandas has become very popular for its ease of use. It utilizes DataFrames to present data in tabular format like a spreadsheet with rows and columns. Importantly, it has very intuitive methods to perform common analytical tasks and a relatively flat learning curve. It loads all of the data into memory on a single machine (one node) for rapid execution. While the Pandas DataFrame has proven to be tremendously powerful in manipulating data, it does have its limits. With data growing at an exponentially rate, complex data processing becomes expensive to handle and causes performance degradation. These operations require parallelization and distributed computing, which the Pandas DataFrame does not support.

Introducing Cluster/Distribution Computing and Spark DataFrame

Apache Spark is an open-source cluster computing framework. With cluster computing, data processing is distributed and performed in parallel by multiple nodes. This is recognized as the MapReduce framework because the division of labor can usually be characterized by sets of the mapshuffle, and reduce operations found in functional programming. Spark’s implementation of cluster computing is unique because processes 1) are executed in-memory and 2) build up a query plan which does not execute until necessary (known as lazy execution). Although Spark’s cluster computing framework has a broad range of utility, we only look at the Spark DataFrame for the purpose of this article. Similar to those found in Pandas, the Spark DataFrame has intuitive APIs, making it easy to implement.

#pandas dataframe vs. spark dataframe: when parallel computing matters #pandas #pandas dataframe #pandas dataframe vs. spark dataframe #spark #when parallel computing matters

Practice Problems: How To Join DataFrames in Pandas

Hey - Nick here! This page is a free excerpt from my $199 course Python for Finance, which is 50% off for the next 50 students.

If you want the full course, click here to sign up.

It’s now time for some practice problems! See below for details on how to proceed.

Course Repository & Practice Problems

All of the code for this course’s practice problems can be found in this GitHub repository.

There are two options that you can use to complete the practice problems:

  • Open them in your browser with a platform called Binder using this link (recommended)
  • Download the repository to your local computer and open them in a Jupyter Notebook using Anaconda (a bit more tedious)

Note that binder can take up to a minute to load the repository, so please be patient.

Within that repository, there is a folder called starter-files and a folder called finished-files. You should open the appropriate practice problems within the starter-files folder and only consult the corresponding file in the finished-files folder if you get stuck.

The repository is public, which means that you can suggest changes using a pull request later in this course if you’d like.

#dataframes #pandas #practice problems: how to join dataframes in pandas #how to join dataframes in pandas #practice #/pandas/issues.

Paula  Hall

Paula Hall


How to add a new column to Pandas DataFrame?

In this tutorial, we are going to discuss different ways to add a new column to pandas data frame.

Table of Contents

What is a pandas data frame?

Pandas data frameis a two-dimensional heterogeneous data structure that stores the data in a tabular form with labeled indexes i.e. rows and columns.

Usually, data frames are used when we have to deal with a large dataset, then we can simply see the summary of that large dataset by loading it into a pandas data frame and see the summary of the data frame.

In the real-world scenario, a pandas data frame is created by loading the datasets from an existing CSV file, Excel file, etc.

But pandas data frame can be also created from the listdictionary, list of lists, list of dictionaries, dictionary of ndarray/lists, etc. Before we start discussing how to add a new column to an existing data frame we require a pandas data frame.

#pandas #dataframe #pandas dataframe #column #add a new column #how to add a new column to pandas dataframe

Udit Vashisht


Python Pandas Objects - Pandas Series and Pandas Dataframe

In this post, we will learn about pandas’ data structures/objects. Pandas provide two type of data structures:-

Pandas Series

Pandas Series is a one dimensional indexed data, which can hold datatypes like integer, string, boolean, float, python object etc. A Pandas Series can hold only one data type at a time. The axis label of the data is called the index of the series. The labels need not to be unique but must be a hashable type. The index of the series can be integer, string and even time-series data. In general, Pandas Series is nothing but a column of an excel sheet with row index being the index of the series.

Pandas Dataframe

Pandas dataframe is a primary data structure of pandas. Pandas dataframe is a two-dimensional size mutable array with both flexible row indices and flexible column names. In general, it is just like an excel sheet or SQL table. It can also be seen as a python’s dict-like container for series objects.

#python #python-pandas #pandas-dataframe #pandas-series #pandas-tutorial

8 Ways to Filter Pandas Dataframes

A practical guide for efficient data analysis

Pandas is a popular data analysis and manipulation library for Python. The core data structure of Pandas is dataframe which stores data in tabular form with labelled rows and columns.

A common operation in data analysis is to filter values based on a condition or multiple conditions. Pandas provides a variety of ways to filter data points (i.e. rows). In this article, we will cover 8 different ways to filter a dataframe.

We start by importing the libraries.

import numpy as np
import pandas as pd

Let’s create a sample dataframe for the examples.

df = pd.DataFrame({

'val2':np.random.randint(1,10, size=7)

#python #programming #data-science #ways to filter pandas dataframes #filter pandas dataframes #pandas dataframes