Web Scraping with Python and BeautifulSoup

There is more information on the Internet than any human can absorb in a lifetime. What you need is not access to that information, but a scalable way to collect, organize, and analyze it.

You need web scraping.

Web scraping automatically extracts data and presents it in a format you can easily make sense of. In this tutorial, we’ll focus on its applications in the financial market, but web scraping can be used in a wide variety of situations.

If you’re an avid investor, getting closing prices every day can be a pain, especially when the information you need is found across several webpages. We’ll make data extraction easier by building a web scraper to retrieve stock indices automatically from the Internet.

Getting Started

We are going to use Python as our scraping language, together with a simple and powerful library, BeautifulSoup.

For Mac users, Python is pre-installed in OS X. Open up Terminal and type python --version. You should see your python version is 2.7.x.
For Windows users, please install Python through the official website.

Next we need to get the BeautifulSoup library using pip, a package management tool for Python.

In the terminal, type:

easy_install pip  
pip install BeautifulSoup4

Note: If you fail to execute the above command line, try adding sudo in front of each line.

The Basics

Before we start jumping into the code, let’s understand the basics of HTML and some rules of scraping.

HTML tags

If you already understand HTML tags, feel free to skip this part.

<!DOCTYPE html>  
<html>  
    <head>
    </head>
    <body>
        <h1> First Scraping </h1>
        <p> Hello World </p>
    <body>
</html>

This is the basic syntax of an HTML webpage. Every <tag> serves a block inside the webpage:

<!DOCTYPE html>: HTML documents must start with a type declaration.
The HTML document is contained between <html> and </html>.
The meta and script declaration of the HTML document is between <head> and </head>.
The visible part of the HTML document is between <body> and </body> tags.
Title headings are defined with the <h1> through <h6> tags.
Paragraphs are defined with the <p> tag.

Other useful tags include <a> for hyperlinks, <table> for tables, <tr> for table rows, and <td> for table columns.

Also, HTML tags sometimes come with id or class attributes. The id attribute specifies a unique id for an HTML tag and the value must be unique within the HTML document. The class attribute is used to define equal styles for HTML tags with the same class. We can make use of these ids and classes to help us locate the data we want.

For more information on HTML tags, id and class, please refer to W3Schools Tutorials.

Scraping Rules

You should check a website’s Terms and Conditions before you scrape it. Be careful to read the statements about legal use of data. Usually, the data you scrape should not be used for commercial purposes.
Do not request data from the website too aggressively with your program (also known as spamming), as this may break the website. Make sure your program behaves in a reasonable manner (i.e. acts like a human). One request for one webpage per second is good practice.
The layout of a website may change from time to time, so make sure to revisit the site and rewrite your code as needed

Inspecting the Page

Let’s take one page from the Bloomberg Quote website as an example.

As someone following the stock market, we would like to get the index name (S&P 500) and its price from this page. First, right-click and open your browser’s inspector to inspect the webpage.

Try hovering your cursor on the price and you should be able to see a blue box surrounding it. If you click it, the related HTML will be selected in the browser console.

From the result, we can see that the price is inside a few levels of HTML tags, which is <div class="basic-quote"> → <div class="price-container up"> → <div class="price">.

Similarly, if you hover and click the name “S&P 500 Index”, it is inside <div class="basic-quote"> and <h1 class="name">.

Now we know the unique location of our data with the help of class tags.

Jump into the Code

Now that we know where our data is, we can start coding our web scraper. Open your text editor now!

First, we need to import all the libraries that we are going to use.

# import libraries
import urllib2
from bs4 import BeautifulSoup

Next, declare a variable for the url of the page.

# specify the url
quote_page = ‘http://www.bloomberg.com/quote/SPX:IND'

Then, make use of the Python urllib2 to get the HTML page of the url declared.

# query the website and return the html to the variable ‘page’
page = urllib2.urlopen(quote_page)

Finally, parse the page into BeautifulSoup format so we can use BeautifulSoup to work on it.

# parse the html using beautiful soup and store in variable `soup`
soup = BeautifulSoup(page, ‘html.parser’)

Now we have a variable, soup, containing the HTML of the page. Here’s where we can start coding the part that extracts the data.

Remember the unique layers of our data? BeautifulSoup can help us get into these layers and extract the content with find(). In this case, since the HTML class name is unique on this page, we can simply query <div class="name">.

# Take out the <div> of name and get its value
name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})

After we have the tag, we can get the data by getting its text.

name = name_box.text.strip() # strip() is used to remove starting and trailing
print name

Similarly, we can get the price too.

# get the index price
price_box = soup.find(‘div’, attrs={‘class’:’price’})
price = price_box.text
print price

When you run the program, you should be able to see that it prints out the current price of the S&P 500 Index.

Export to Excel CSV

Now that we have the data, it is time to save it. The Excel Comma Separated Format is a nice choice. It can be opened in Excel so you can see the data and process it easily.

But first, we have to import the Python csv module and the datetime module to get the record date. Insert these lines to your code in the import section.

import csv
from datetime import datetime

At the bottom of your code, add the code for writing data to a csv file.

# open a csv file with append, so old data will not be erased
with open(‘index.csv’, ‘a’) as csv_file:
 writer = csv.writer(csv_file)
 writer.writerow([name, price, datetime.now()])

Now if you run your program, you should able to export an index.csv file, which you can then open with Excel, where you should see a line of data.

So if you run this program everyday, you will be able to easily get the S&P 500 Index price without rummaging through the website!

Going Further (Advanced uses)

Multiple Indices

So scraping one index is not enough for you, right? We can try to extract multiple indices at the same time.

First, modify the quote_page into an array of URLs.

quote_page = [‘http://www.bloomberg.com/quote/SPX:IND', ‘http://www.bloomberg.com/quote/CCMP:IND']

Then we change the data extraction code into a for loop, which will process the URLs one by one and store all the data into a variable data in tuples.

# for loop
data = []
for pg in quote_page:
 # query the website and return the html to the variable ‘page’
 page = urllib2.urlopen(pg)
# parse the html using beautiful soap and store in variable `soup`
 soup = BeautifulSoup(page, ‘html.parser’)
# Take out the <div> of name and get its value
 name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})
 name = name_box.text.strip() # strip() is used to remove starting and trailing
# get the index price
 price_box = soup.find(‘div’, attrs={‘class’:’price’})
 price = price_box.text
# save the data in tuple
 data.append((name, price))

Also, modify the saving section to save data row by row.

# open a csv file with append, so old data will not be erased
with open(‘index.csv’, ‘a’) as csv_file:
 writer = csv.writer(csv_file)
 # The for loop
 for name, price in data:
 writer.writerow([name, price, datetime.now()])

Rerun the program and you should be able to extract two indices at the same time!

Advanced Scraping Techniques

BeautifulSoup is simple and great for small-scale web scraping. But if you are interested in scraping data at a larger scale, you should consider using these other alternatives:

You should check a website’s Terms and Conditions before you scrape it. Be careful to read the statements about legal use of data. Usually, the data you scrape should not be used for commercial purposes.
Do not request data from the website too aggressively with your program (also known as spamming), as this may break the website. Make sure your program behaves in a reasonable manner (i.e. acts like a human). One request for one webpage per second is good practice.
The layout of a website may change from time to time, so make sure to revisit the site and rewrite your code as needed

Adopt the DRY Method

DRY stands for “Don’t Repeat Yourself”, try to automate your everyday tasks like this person. Some other fun projects to consider might be keeping track of your Facebook friends’ active time (with their consent of course), or grabbing a list of topics in a forum and trying out natural language processing (which is a hot topic for Artificial Intelligence right now)!

Thanks for reading ❤

Mobile App Development Company India | Ecommerce Web Development Company India

Best Mobile App Development Company India, WebClues Global is one of the leading web and mobile app development company. Our team offers complete IT solutions including Cross-Platform App Development, CMS & E-Commerce, and UI/UX Design.

We are custom eCommerce Development Company working with all types of industry verticals and providing them end-to-end solutions for their eCommerce store development.

Know more about Top E-Commerce Web Development Company

Python Tutorial for Beginners (2019) - Learn Python for Machine Learning and Web Development

TABLE OF CONTENT

00:00:00 Introduction

00:01:49 Installing Python

00:06:10 Your First Python Program

00:08:11 How Python Code Gets Executed

00:11:24 How Long It Takes To Learn Python

00:13:03 Variables

00:18:21 Receiving Input

00:22:16 Python Cheat Sheet

00:22:46 Type Conversion

00:29:31 Strings

00:37:36 Formatted Strings

00:40:50 String Methods

00:48:33 Arithmetic Operations

00:51:33 Operator Precedence

00:55:04 Math Functions

00:58:17 If Statements

01:06:32 Logical Operators

01:11:25 Comparison Operators

01:16:17 Weight Converter Program

01:20:43 While Loops

01:24:07 Building a Guessing Game

01:30:51 Building the Car Game

01:41:48 For Loops

01:47:46 Nested Loops

01:55:50 Lists

02:01:45 2D Lists

02:05:11 My Complete Python Course

02:06:00 List Methods

02:13:25 Tuples

02:15:34 Unpacking

02:18:21 Dictionaries

02:26:21 Emoji Converter

02:30:31 Functions

02:35:21 Parameters

02:39:24 Keyword Arguments

02:44:45 Return Statement

02:48:55 Creating a Reusable Function

02:53:42 Exceptions

02:59:14 Comments

03:01:46 Classes

03:07:46 Constructors

03:14:41 Inheritance

03:19:33 Modules

03:30:12 Packages

03:36:22 Generating Random Values

03:44:37 Working with Directories

03:50:47 Pypi and Pip

03:55:34 Project 1: Automation with Python

04:10:22 Project 2: Machine Learning with Python

04:58:37 Project 3: Building a Website with Django

Thanks for reading ❤

If you liked this post, share it with all of your programming buddies!

Top 10 Python Frameworks for Web Development In 2019

In this article, we are going to share our list of the top 10 Python frameworks for web development which we believe will help you to develop awesome applications and your technical abilities.

Given how dynamic web development has become, the popularity of Python frameworks seems to be only increasing. This object-oriented, powerfully composed, interpreted, and interactive programming language is easy to **learn **and effectively lessens the development time with its easy-to-read syntax and simple compilation feature. That’s reason enough why it is continuously gaining popularity.

Also, it has a vast number of Python libraries that support data analysis, visualization, and manipulation. Consequently, it has advanced as the most favored programming language and is now considered the “Next Big Thing” for professionals.

Since **Python **does not accompany the built-in features required to accelerate custom web application development, many developers choose Python’s robust collection of frameworks to deal with the subtleties of execution.

**Python **gives a wide scope of frameworks to developers. There are two types of Python frameworks – **Full Stack Framework **and Non-Full Stack Framework. The full-stack frameworks give full support to Python developers including basic components like form generators, form validation, and template layouts.

There is a cluster of full stack options when we talk of Python frameworks. Listed below are the top 10 full-stack web frameworks for **Python **that you should be using in 2019 valuable for enhancing your technical abilities.

Django

Django is a free and open-source Python framework that enables developers to develop complex code and applications effectively and quickly. This high-level framework streamlines web application development by giving different vigorous features. It has a colossal assortment of libraries and underscores effectiveness, less need for coding, and reusability of components.

A few of the key features of Django, such as authentication mechanism, URL routing, template engine, and database schema migration implements ORM (Object Relational Mapper) for mapping its objects to database tables. The framework underpins numerous **databases **including PostgreSQL, MySQL, Oracle, and SQLite, which implies that a similar coding works with various databases.

Django’s cutting-edge features help developers in achieving basic web development tasks like user authentication, RSS feeds, content services, and sitemap. Due to its incredible features, Django framework is extensively used in several high-traffic sites, which include Pinterest, Instagram, Bitbucket, Mozilla, Disqus, and The Washington Times.

CherryPy

**CherryPy **is an open source Python web development framework that implants its very own multi-strung server. It can keep running on any working framework that supports Python. **CherryPy **features incorporate thread-pooled web server, setup framework, and module framework.

A moderate web framework enables you to utilize any sort of technology for data access, templating, etc. Yet, it can do everything that a web framework can, for instance, handling sessions, static, file uploads, cookies, and so on.

Regardless of the accessible features and advantages like running on multiple platforms, built-in support for profiling, reporting, and testing, some developers may imagine that there is a requirement for easy and enhanced documentation. It doesn’t constrain you to use a specific template engine, ORM, so that you can use anything you wish to use.

Pyramid

**Pyramid **is a Python framework that underpins validation and directing. It is incredible for growing huge web applications, as CMSs, and it is valuable for prototyping an idea and for developers chipping away at API projects. Pyramid is adaptable and can be utilized for both easy as well as difficult projects.

Pyramid is enhanced with features without driving a specific method for completing things, lightweight without abandoning you all alone as your app develops. It is a most valued web framework among experienced Python developers by virtue of its transparency and measured quality. It has been used by a moderate team and tech giants like Mozilla, Yelp, Dropbox, and SurveyMonkey.

The pyramid is reliably known for its security arrangements, which makes it easy to set up and check access control records. Another inventive functionality worth uncovering is Pyramid’s Traversal framework for mapping URLs to code, which makes it simple to develop RESTful APIs.

TurboGears

**TurboGears **is an open-source, free, and data-driven full-stack web application Python framework. It is designed to overcome the inadequacies of various extensively used web development frameworks. It empowers software engineers to begin developing web applications with an insignificant setup.

**TurboGears **enables web developers to streamline web application development utilizing diverse JavaScript development tools. You can develop web applications with the help of elements such as SQLAlchemy, Repoze, WebOb, and Genshi, much faster than other existing frameworks. It supports different **databases **and web servers like Pylons.

The framework pursues an MVC (Model-View-Controller) design and incorporates vigorous formats, an incredible Object Relational Mapper (ORM) and Ajax for the server and program. Organizations using TurboGears incorporate Bisque, ShowMeDo, and SourceForge.

Web2Py

**Web2py **is a free, open source Python framework for web application development. The framework accompanies a debugger, code editor as well as a deployment tool to enable you to build and debug the code, as well as test and keep up web applications.

It’s a cross platform framework that underpins Windows, Unix/Linux, Mac, Google App Engine, and different other platforms. It pursues the MVC (Model-View-Controller) design. The framework streamlines web application development procedure via a web server, SQL database, and an online interface. It enables clients to build, revise, deploy, and manage web applications via web browsers.

The key component of Web2py is a ticketing framework, which issues a ticket when a mistake occurs. This encourages the client to follow the mistake and its status. Also, it has in-built components to manage HTTP requests, reactions, sessions, and cookies.

Bottle

Another interesting Python web framework is Bottle, which falls under the class of small-scale frameworks. Originally, it was developed for building web APIs. Also, Bottle tries to execute everything in a single document, which should give you a short perspective on how small it is designed to be.

The out-of-the-box functionalities include templating, utilities, directing, and some fundamental abstraction over the WSGI standard. Like Flask, you will be coding significantly closer to the metal than with a full-stack framework. Regardless of their, Bottle has been used by Netflix to create web interfaces.

Tornado

Tornado is a Python web framework and offbeat framework library. It utilizes a non-blocking framework I/O and unravels the C10k issue (which means that, whenever configured properly, it can deal with 10,000+ simultaneous connections).

Tornado’s main features comprise of built-in support for user confirmation, superior quality, real-time services, non-blocking HTTP customer, Python-based web templating language, and support for interpretation and localization.

This makes it an extraordinary tool for building applications that require superior and a huge number of simultaneous clients.

Flask

Flask is a Python framework accessible under the BSD license, which is inspired by the Sinatra Ruby framework. Flask relies upon the Werkzeug WSGI toolbox and Jinja2 template. The main purpose is to help develop a strong web application base.

**Developers **can **develop backend frameworks **any way they need, however, it was designed for applications that are open-ended. Flask has been used by big companies, which include LinkedIn and Pinterest. As compared to Django, Flask is best suited for small and easy projects. Thus, you can expect a web server development, **support **for Google App Engine as well as in-built unit testing.

Grok

**Grok framework **has been created, depending on Zope toolbox for giving an agile development experience to developers by concentrating on convention over configuration and DRY (Don’t Repeat Yourself). It is an open-source framework, developed to speed up the application development process.

Developers can choose from a wide scope of network and independent libraries as indicated by the task needs. Grok’s UI (user interface) is like other full-stack frameworks such as **Pylons **and TurboGears.

The Grok component architecture helps developers lessen the unpredictability of development by availing views, content objects, and controller. Grok, likewise, provides the building blocks and other essential assets required to develop custom web applications for business needs.

BlueBream

**BlueBream **is also an open source web application framework, server, and library for website developers. It has been developed by the Zope team which was formerly known as Zope 3.

This framework is best suited for both medium and substantial activities apportioned into various re-usable and well-suited segments.

**BlueBream **relies upon Zoop Toolkit (ZTK). It holds extensive periods of experience ensuring that it meets the main essential for enduring, relentless, and adaptable programming.

Conclusion

Though there are many python web development frameworks that will be popular and in-demand in the coming years, especially in 2019, every framework has its own pros and cons. Every developer has different coding styles and preferences. They will assess every framework as per the requirements of an individual task. In this way, the choice of python web development framework will change from one developer onto the next.

The above-listed are some of the Python frameworks that are widely used as a full-stack backend web application development. Which one are you picking for your next project? Do let us know in the comments section given below.