1593082200

Pandas is a python tool used extensively for data analysis and manipulation. Recently I’ve been using pandas with large DataFrames (>50M rows) and through the PyDataUK May Talks and exploring StackOverflow threads have discovered several tips that have been incredibly useful in optimising my analysis.

This tutorial is part 1 of a series and aims to give an introduction to pandas and some of the useful features it offers while exploring the Palmer Penguin dataset.

In this article, we will go through:

- How to install and import pandas
- Data structures in pandas
- How to input and output data
- Inspecting the data
- Getting started with Data Cleaning

#data-science #python #pandas #data-analysis

1657081614

In this article, We will show how we can use python to automate Excel . A useful Python library is Openpyxl which we will learn to do Excel Automation

Openpyxl is a Python library that is used to read from an Excel file or write to an Excel file. Data scientists use Openpyxl for data analysis, data copying, data mining, drawing charts, styling sheets, adding formulas, and more.

**Workbook:** A spreadsheet is represented as a workbook in openpyxl. A workbook consists of one or more sheets.

**Sheet:** A sheet is a single page composed of cells for organizing data.

**Cell:** The intersection of a row and a column is called a cell. Usually represented by A1, B5, etc.

**Row:** A row is a horizontal line represented by a number (1,2, etc.).

**Column:** A column is a vertical line represented by a capital letter (A, B, etc.).

Openpyxl can be installed using the pip command and it is recommended to install it in a virtual environment.

`pip `

**install** openpyxl

We start by creating a new spreadsheet, which is called a workbook in Openpyxl. We import the workbook module from Openpyxl and use the

which creates a new workbook.**function** **Workbook**()

```
from openpyxl
import Workbook
#creates a new workbook
wb = Workbook()
#Gets the first active worksheet
ws = wb.active
#creating new worksheets by using the create_sheet method
ws1 = wb.create_sheet("sheet1", 0) #inserts at first position
ws2 = wb.create_sheet("sheet2") #inserts at last position
ws3 = wb.create_sheet("sheet3", -1) #inserts at penultimate position
#Renaming the sheet
ws.title = "Example"
#save the workbook
wb.save(filename = "example.xlsx")
```

We load the file using the

which takes the filename as an argument. The file must be saved in the same working directory.**function** **load_Workbook**()

```
#loading a workbook
wb = openpyxl.load_workbook("example.xlsx")
```

```
#getting sheet names
wb.sheetnames
result = ['sheet1', 'Sheet', 'sheet3', 'sheet2']
#getting a particular sheet
sheet1 = wb["sheet2"]
#getting sheet title
sheet1.title
result = 'sheet2'
#Getting the active sheet
sheetactive = wb.active
result = 'sheet1'
```

```
#get a cell from the sheet
sheet1["A1"] <
Cell 'Sheet1'.A1 >
#get the cell value
ws["A1"].value 'Segment'
#accessing cell using row and column and assigning a value
d = ws.cell(row = 4, column = 2, value = 10)
d.value
10
```

```
#looping through each row and column
for x in range(1, 5):
for y in range(1, 5):
print(x, y, ws.cell(row = x, column = y)
.value)
#getting the highest row number
ws.max_row
701
#getting the highest column number
ws.max_column
19
```

There are two functions for iterating through rows and columns.

```
Iter_rows() => returns the rows
Iter_cols() => returns the columns {
min_row = 4, max_row = 5, min_col = 2, max_col = 5
} => This can be used to set the boundaries
for any iteration.
```

**Example:**

```
#iterating rows
for row in ws.iter_rows(min_row = 2, max_col = 3, max_row = 3):
for cell in row:
print(cell) <
Cell 'Sheet1'.A2 >
<
Cell 'Sheet1'.B2 >
<
Cell 'Sheet1'.C2 >
<
Cell 'Sheet1'.A3 >
<
Cell 'Sheet1'.B3 >
<
Cell 'Sheet1'.C3 >
#iterating columns
for col in ws.iter_cols(min_row = 2, max_col = 3, max_row = 3):
for cell in col:
print(cell) <
Cell 'Sheet1'.A2 >
<
Cell 'Sheet1'.A3 >
<
Cell 'Sheet1'.B2 >
<
Cell 'Sheet1'.B3 >
<
Cell 'Sheet1'.C2 >
<
Cell 'Sheet1'.C3 >
```

To get all the rows of the worksheet we use the method worksheet.rows and to get all the columns of the worksheet we use the method worksheet.columns. Similarly, to iterate only through the values we use the method worksheet.values.

**Example:**

```
for row in ws.values:
for value in row:
print(value)
```

Writing to a workbook can be done in many ways such as adding a formula, adding charts, images, updating cell values, inserting rows and columns, etc… We will discuss each of these with an example.

```
#creates a new workbook
wb = openpyxl.Workbook()
#saving the workbook
wb.save("new.xlsx")
```

```
#creating a new sheet
ws1 = wb.create_sheet(title = "sheet 2")
#creating a new sheet at index 0
ws2 = wb.create_sheet(index = 0, title = "sheet 0")
#checking the sheet names
wb.sheetnames['sheet 0', 'Sheet', 'sheet 2']
#deleting a sheet
del wb['sheet 0']
#checking sheetnames
wb.sheetnames['Sheet', 'sheet 2']
```

```
#checking the sheet value
ws['B2'].value
null
#adding value to cell
ws['B2'] = 367
#checking value
ws['B2'].value
367
```

We often require formulas to be included in our Excel datasheet. We can easily add formulas using the Openpyxl module just like you add values to a cell.

**For example:**

```
import openpyxl
from openpyxl
import Workbook
wb = openpyxl.load_workbook("new1.xlsx")
ws = wb['Sheet']
ws['A9'] = '=SUM(A2:A8)'
wb.save("new2.xlsx")
```

The above program will add the formula (=SUM(A2:A8)) in cell A9. The result will be as below.

Two or more cells can be merged to a rectangular area using the method merge_cells(), and similarly, they can be unmerged using the method unmerge_cells().

**For example:****Merge cells**

```
#merge cells B2 to C9
ws.merge_cells('B2:C9')
ws['B2'] = "Merged cells"
```

Adding the above code to the previous example will merge cells as below.

```
#unmerge cells B2 to C9
ws.unmerge_cells('B2:C9')
```

The above code will unmerge cells from B2 to C9.

To insert an image we import the image function from the module openpyxl.drawing.image. We then load our image and add it to the cell as shown in the below example.

**Example:**

```
import openpyxl
from openpyxl
import Workbook
from openpyxl.drawing.image
import Image
wb = openpyxl.load_workbook("new1.xlsx")
ws = wb['Sheet']
#loading the image(should be in same folder)
img = Image('logo.png')
ws['A1'] = "Adding image"
#adjusting size
img.height = 130
img.width = 200
#adding img to cell A3
ws.add_image(img, 'A3')
wb.save("new2.xlsx")
```

**Result:**

Charts are essential to show a visualization of data. We can create charts from Excel data using the Openpyxl module chart. Different forms of charts such as line charts, bar charts, 3D line charts, etc., can be created. We need to create a reference that contains the data to be used for the chart, which is nothing but a selection of cells (rows and columns). I am using sample data to create a 3D bar chart in the below example:

**Example**

```
import openpyxl
from openpyxl
import Workbook
from openpyxl.chart
import BarChart3D, Reference, series
wb = openpyxl.load_workbook("example.xlsx")
ws = wb.active
values = Reference(ws, min_col = 3, min_row = 2, max_col = 3, max_row = 40)
chart = BarChart3D()
chart.add_data(values)
ws.add_chart(chart, "E3")
wb.save("MyChart.xlsx")
```

**Result**

Welcome to another video! In this video, We will cover how we can use python to automate Excel. I'll be going over everything from creating workbooks to accessing individual cells and stylizing cells. There is a ton of things that you can do with Excel but I'll just be covering the core/base things in OpenPyXl.

⭐️ Timestamps ⭐️

00:00 | Introduction

02:14 | Installing openpyxl

03:19 | Testing Installation

04:25 | Loading an Existing Workbook

06:46 | Accessing Worksheets

07:37 | Accessing Cell Values

08:58 | Saving Workbooks

09:52 | Creating, Listing and Changing Sheets

11:50 | Creating a New Workbook

12:39 | Adding/Appending Rows

14:26 | Accessing Multiple Cells

20:46 | Merging Cells

22:27 | Inserting and Deleting Rows

23:35 | Inserting and Deleting Columns

24:48 | Copying and Moving Cells

26:06 | Practical Example, Formulas & Cell Styling

📄 Resources 📄

OpenPyXL Docs: https://openpyxl.readthedocs.io/en/stable/

Code Written in This Tutorial: https://github.com/techwithtim/ExcelPythonTutorial

Subscribe: https://www.youtube.com/c/TechWithTim/featured

1623921720

In this part, we will explore more about pandas library and its uses in data science.

I will continue from where I left. If you haven’t seen part 2. Please visit https://medium.com/analytics-vidhya/getting-started-with-data-science-with-python-part-2-e3cc3411ac70. Because it will help you to understand.

Let’s go.

- Open jupytier notebook.
- Import pandas
- Load the CSV dataset into dataframes.

Now we will mainly focus on commands.

**df.describe(): This commads tells us some important stuff about integer columns. Lets do it and see.**

- count: It gives us the total number of not null values in the column.
- mean: It gives us the mean of column.
- std: A quantity expressing by how much the members of a group differ from the mean value for the group.
- min: The minimum value in the column.
- max: The max value in the column.

#pandas #python #data-analysis #data-visualization #data-science #getting started with data science with python

1623488340

Pandas is one of the predominant data analysis tools which is highly appreciated among data scientists. It provides numerous flexible and versatile functions to perform efficient data analysis.

In this article, we will go over 3 pandas tricks that I think will make you a more happy pandas user. It is better to explain these tricks with some examples. Thus, we start by creating a data frame to wok on.

The data frame contains daily sales quantities of 3 different stores. We first create a period of 10 days using the `date_range`

function of pandas.

```
import numpy as np
import pandas as pd
days = pd.date_range("2020-01-01", periods=10, freq="D")
```

The days variable will be used as a column. We also need a sales quantity column which can be generated by the `randint`

function of numpy. Then, we create a data frame with 3 columns for each store.

#machine-learning #data-science #python #python pandas tricks #efficient data analysis #python pandas tricks for efficient data analysis

1620629020

The opportunities big data offers also come with very real challenges that many organizations are facing today. Often, it’s finding the most cost-effective, scalable way to store and process boundless volumes of data in multiple formats that come from a growing number of sources. Then organizations need the analytical capabilities and flexibility to turn this data into insights that can meet their specific business objectives.

This Refcard dives into how a data lake helps tackle these challenges at both ends — from its enhanced architecture that’s designed for efficient data ingestion, storage, and management to its advanced analytics functionality and performance flexibility. You’ll also explore key benefits and common use cases.

As technology continues to evolve with new data sources, such as IoT sensors and social media churning out large volumes of data, there has never been a better time to discuss the possibilities and challenges of managing such data for varying analytical insights. In this Refcard, we dig deep into how data lakes solve the problem of storing and processing enormous amounts of data. While doing so, we also explore the benefits of data lakes, their use cases, and how they differ from data warehouses (DWHs).

*This is a preview of the Getting Started With Data Lakes Refcard. To read the entire Refcard, please download the PDF from the link above.*

#big data #data analytics #data analysis #business analytics #data warehouse #data storage #data lake #data lake architecture #data lake governance #data lake management

1620466520

If you accumulate data on which you base your decision-making as an organization, you should probably think about your data architecture and possible best practices.

If you accumulate data on which you base your decision-making as an organization, you most probably need to think about your data architecture and consider possible best practices. Gaining a competitive edge, remaining customer-centric to the greatest extent possible, and streamlining processes to get on-the-button outcomes can all be traced back to an organization’s capacity to build a future-ready data architecture.

In what follows, we offer a short overview of the overarching capabilities of data architecture. These include user-centricity, elasticity, robustness, and the capacity to ensure the seamless flow of data at all times. Added to these are automation enablement, plus security and data governance considerations. These points from our checklist for what we perceive to be an anticipatory analytics ecosystem.

#big data #data science #big data analytics #data analysis #data architecture #data transformation #data platform #data strategy #cloud data platform #data acquisition