Python course Lesson2: How to use pandas and get financial data

This article was originally published at:
https://www.blog.duomly.com/python-course-with-building-a-fintech-investment-ai-lesson-2-pandas-and-getting-financial-data

Intro

Welcome to the second lesson of the awesome Python course for AI!

In the last lesson, we built the first database migrations and made the project setup.

Here is the URL:

Python course with building a fintech investment AI – Lesson 1: Start the project

Today, we will build super exciting stuff and will focus on the financial database that we will use to train our AI later.

We will work with pandas, some financial plugins, and will do a bit of refactoring.

As well, I will teach you how to create CLI commands in Python.

I cannot wait to teach you all of those powerful skills, so let's start ASAP!

And if you prefer video, here is the youtube version:

1. Change types in migrations

In the last episode, we created DB migrations and database models with some types.

Today we will change them a bit to have the possibility of using prices with many numbers after the decimal.

To do that, we need just change types in variable "date" to "db.Date()”.

Next, we need to change all Integers to "db.Numeric()”.

Take a look at the example below.

id = db.Column(db.Integer, primary_key=True)

company = db.Column(db.String())

date = db.Column(db.Date())

openPrice = db.Column(db.Numeric())

highPrice = db.Column(db.Numeric())

lowPrice = db.Column(db.Numeric())

closePrice = db.Column(db.Numeric())

volume = db.Column(db.Numeric())

2. Delete migrations folder and migrate again

Now, we should save our changes and update the database with the new changes.

I would suggest initing DB from scratch.

As the first step, you need to delete the "migrations" directory (if you created it in the previous lesson).

Next, open terminal and type:

flask db init

Next:

flask db migrate

Next:

flask db migrate

3. Create module prices

Great!

We can focus on the next features now, and go into the prices module.

Let's start by creating the directory named "prices".

Create that directory in the root of the folder.

Next, go into the created directory and create the file named "prices.py".

4.Install necessary dependencies

Before we go into the proper development, we should remember about installing all of the necessary dependencies that we will use for building AI.

First, we need to create yfinance, that will help us to connect with the financial API, and download all necessary data about the prices.

Open the terminal and type:

pip install yfinance

Next, we need to install pandas:

pip install yfinance

The last one that we need to install is pandas datareader that we will use to handling our financial data:

pip install pandas_datareader

5.Import all necessary dependencies

So, we have prepared all the necessary stuff that we needed for development.

Now, we can go into the prices.py file, and import all of these dependencies.

Let's take a look at the example below:

from datetime import date, timedelta

import yfinance as yf

import pandas as pd

from pandas_datareader import data as pdr

from app import app, db

from models.prices import PriceModel

6. Override yfinance with pandas

Next, we need to override yfinance with pandas.

That line of code should be below imports inside the prices.py file.

yf.pdr_override()

7. Setup CLI

We could do that by typing/commenting on the functions as we had done in golang.

But it'd be a shame if we weren't using the user-friendly features of the flask.

We can define CLI commands very, very easily.

It's enough to use the "app" decorator and define the command that should fire the logic that is below the command definition.

Let's take a look:

@app.cli.command("get-prices")

8. Create function getPrices

Below the CLI command, we need to specify a function that we will want to fire.

In this case, we need a function named "getPrices".

We will use that to start getting prices. 

@app.cli.command("get-prices")

def getPrices():

9. Create a list of stocks that we will need to monitor

In the next step, we should specify company symbols that we would like to get pricing for.

We should create a list and put there all the NASDAQ symbols as strings.

I've used the heaviest 50 of the NASDAQ, but you can specify whatever companies that you would like to get data for.

stockList = ['MSFT', 'AAPL', 'AMZN', 'FB', 'GOOGL', 'GOOG', 'GOOGL', 'INTC', 'NVDA', 'CSCO', 'ADBE', 'NFLX', 'PEP', 'PYP', 'CMCSA', 'TSLA', 'COST', 'GOOGL', 'AMGN', 'TMUS', 'AVGO', 'CHTR', 'TXN', 'GILD', 'QCOM', 'SBUX', 'INTU', 'MDLZ', 'VRTX', 'FISV', 'BKNG', 'ISRG', 'REGN', 'ADP', 'AMD', 'ATVI', 'CSX', 'BIIB', 'ILMN', 'MU', 'AMAT', 'JD', 'ADSK', 'MELI', 'ADI', 'LRCX', 'MNST', 'WBA', 'EXC', 'KHC', 'LULU', 'EA']

10. Iterate through the list of stocks and call function getPrice

In this step, we should take all of the NASDAQ symbols and put them inside the for loop that will iterate through them.

For every iteration of the stock, we should call the function named "getPrice" and pass "stock" as an argument.

for stock in stockList:

 getPrice(stock)

11. Create function getPrice

To have the possibility of using it, we need to create the function "getPrice".

Setup "stock" as a param name in the function.

def getPrice(stock):

12. Setup date

Now, we can start focusing on the core functionality that we are interested in today, fetching data.

As a first step, we need to specify the date range that we are interested in.

I will set up today as the last day, and day that was 31 days ago, that will be my time-range for the stock prices.

today = date.today()

monthAgo = today - timedelta(days=31)

13. Download stock prices

In the next step, we can start getting data!

Look at the example below. We use pandas_datareader.

Next, I've set up stock, that I want to get, the start date, and the end date that is today.

data = pdr.get_data_yahoo(stock, start=monthAgo, end=today)

14. Reformat data by pandas

To have a better life and easier manipulation of data, we need to reformat that a bit.

So we will be able to iterate through that data much more comfortable.

df = pd.DataFrame(data)

15. Iterate through all rows and pass data into the function addPrice

Now is the moment where we can iterate through all of the prices for the xx company.

We use "df.iterrows()" method here because all of the rows are from pandas.

Next, we call "addPrice", and pass necessary data as arguments.

We pass the whole "row" that we will split later, next we pass index (it's data here), but formatted.

And we pass "stock" to know what stock is that price for.

for index, row in df.iterrows():

  addPrice(row, index.strftime("%Y-%m-%d"), stock)

16. Create function addPrice

To make using the "addPrice" possible, we need to define that function.

Use "price", "date", and "stock", as params.

def addPrice(price, date, stock):

17. Setup the data for the PriceModel

We are close to the final.

In this step, we should set up the data that we've got from financial API.

We need to fit that to our PriceModel, that we've created in the Python course Lesson1.

new_price = PriceModel(date=date, company=stock, openPrice=price['Open'], highPrice=price['High'], lowPrice=price['Low'], closePrice=price['Close'], volume=price[‚Volume'])

18. Add price into the database

If our data is ready, we have nothing else to do than just create a new record inside the database.

Don’t forget about „db.session.commit()”, that will update our database.

db.session.add(new_price)

db.session.commit()

19. Import getPrices in app.py

Congratulations!

The prices module is ready now. You can import it inside the app.py.

Add this line of code below the "PriceModel" import.

from prices.prices import getPrices

20. Run get-prices

You can test your application now.

Before you start it, make sure the flask app variable is configured properly (we did it in Lesson 1).

If yes, just open a terminal in the project and type:

flask get-prices

Conclusion

Congratulations!

Now your project has hundreds of financial records.

Those are ready to start working with Artificial Intelligence, which we will build.

What is more important, you've learned where to look for the stock prices, how to select them, and how to save them into the database. 

Code repository for the Python course Lesson 2 is here:

https://github.com/Duomly/python-ai-investment-fintech/tree/Python-AI-course-Lesson-2

In the next lesson, we will focus on building the first AI logic for our algorithm.

Keep learning, and I'm super excited I have a chance to teach you how to build powerful investment AI that can predict stock prices!

Thanks for reading,

Radek from Duomly

#python #web-development #machine-learning #database #sql

What is GEEK

Buddha Community

top 30 Python Tips and Tricks for Beginners

Welcome to my Blog , In this article, you are going to learn the top 10 python tips and tricks.

1) swap two numbers.

2) Reversing a string in Python.

3) Create a single string from all the elements in list.

4) Chaining Of Comparison Operators.

5) Print The File Path Of Imported Modules.

6) Return Multiple Values From Functions.

7) Find The Most Frequent Value In A List.

8) Check The Memory Usage Of An Object.

…

#python #python hacks tricks #python learning tips #python programming tricks #python tips #python tips and tricks #python tips and tricks advanced #python tips and tricks for beginners #python tips tricks and techniques #python tutorial #tips and tricks in python #tips to learn python #top 30 python tips and tricks for beginners

PANDAS: Most Used Functions in Data Science

Most useful functions for data preprocessing

When you get introduced to machine learning, the first step is to learn Python and the basic step of learning Python is to learn pandas library. We can install pandas library by pip install pandas. After installing we have to import pandas each time of the running session. The data used for example is from the UCI repository “https://archive.ics.uci.edu/ml/datasets/Heart+failure+clinical+records ”

1. Read Data

2. Head and Tail

3. Shape, Size and Info

4. isna

…

#pandas: most used functions in data science #pandas #data science #function #used python data #most used functions in data science

Your Data Architecture: Simple Best Practices for Your Data Strategy

If you accumulate data on which you base your decision-making as an organization, you should probably think about your data architecture and possible best practices.

If you accumulate data on which you base your decision-making as an organization, you most probably need to think about your data architecture and consider possible best practices. Gaining a competitive edge, remaining customer-centric to the greatest extent possible, and streamlining processes to get on-the-button outcomes can all be traced back to an organization’s capacity to build a future-ready data architecture.

In what follows, we offer a short overview of the overarching capabilities of data architecture. These include user-centricity, elasticity, robustness, and the capacity to ensure the seamless flow of data at all times. Added to these are automation enablement, plus security and data governance considerations. These points from our checklist for what we perceive to be an anticipatory analytics ecosystem.

#big data #data science #big data analytics #data analysis #data architecture #data transformation #data platform #data strategy #cloud data platform #data acquisition

Data Science With Python Training | Python Data Science Course | Intellipaat

In this Data Science With Python Training video, you will learn everything about data science and python from basic to advance level. This python data science course video will help you learn various python concepts, AI, and lots of projects, hands-on demo, and lastly top trending data science and python interview questions. This is a must-watch video for everyone who wishes o learn data science and python to make a career in it.

#data science with python #python data science course #python data science #data science with python

Basic Data Types in Python | Python Web Development For Beginners

At the end of 2019, Python is one of the fastest-growing programming languages. More than 10% of developers have opted for Python development.

In the programming world, Data types play an important role. Each Variable is stored in different data types and responsible for various functions. Python had two different objects, and They are mutable and immutable objects.

Table of Contents  hide

I Mutable objects

II Immutable objects

III Built-in data types in Python

Mutable objects

The Size and declared value and its sequence of the object can able to be modified called mutable objects.

Mutable Data Types are list, dict, set, byte array

Immutable objects

The Size and declared value and its sequence of the object can able to be modified.

Immutable data types are int, float, complex, String, tuples, bytes, and frozen sets.

id() and type() is used to know the Identity and data type of the object

a**=25+**85j

type**(a)**

output**:<class’complex’>**

b**={1:10,2:“Pinky”****}**

id**(b)**

output**:**238989244168

Built-in data types in Python

a**=str(“Hello python world”)****#str**

b**=int(18)****#int**

c**=float(20482.5)****#float**

d**=complex(5+85j)****#complex**

e**=list((“python”,“fast”,“growing”,“in”,2018))****#list**

f**=tuple((“python”,“easy”,“learning”))****#tuple**

g**=range(10)****#range**

h**=dict(name=“Vidu”,age=36)****#dict**

i**=set((“python”,“fast”,“growing”,“in”,2018))****#set**

j**=frozenset((“python”,“fast”,“growing”,“in”,2018))****#frozenset**

k**=bool(18)****#bool**

l**=bytes(8)****#bytes**

m**=bytearray(8)****#bytearray**

n**=memoryview(bytes(18))****#memoryview**

Numbers (int,Float,Complex)

Numbers are stored in numeric Types. when a number is assigned to a variable, Python creates Number objects.

#signed interger

age**=**18

print**(age)**

Output**:**18

Python supports 3 types of numeric data.

int (signed integers like 20, 2, 225, etc.)

float (float is used to store floating-point numbers like 9.8, 3.1444, 89.52, etc.)

complex (complex numbers like 8.94j, 4.0 + 7.3j, etc.)

A complex number contains an ordered pair, i.e., a + ib where a and b denote the real and imaginary parts respectively).

String

The string can be represented as the sequence of characters in the quotation marks. In python, to define strings we can use single, double, or triple quotes.

# String Handling

‘Hello Python’

#single (') Quoted String

“Hello Python”

# Double (") Quoted String

“”“Hello Python”“”

‘’‘Hello Python’‘’

# triple (‘’') (“”") Quoted String

In python, string handling is a straightforward task, and python provides various built-in functions and operators for representing strings.

The operator “+” is used to concatenate strings and “*” is used to repeat the string.

“Hello”+“python”

output**:****‘Hello python’**

"python "*****2

'Output : Python python ’

#python web development #data types in python #list of all python data types #python data types #python datatypes #python types #python variable type