Web Scraping with Python using Requests, LXML and Splash

Become a Ninja in Web Scraping/Crawling with Python using Requests, the powerful LXML and Splash for JavaScript websites

What is web scraping?

Let’s say your boss at work gave you a task where he wants you to extract about 1000 product from a website, structure the data and save it to a database, would you copy paste manually all the product details from the product name, url and price ? I can imagine you would work days and nights and you wont finish the task, so this is where web scraping shines. So web scraping, or web harvesting or web data extraction is like writing a script that will automate data extraction from websites in a matter of minutes !.

Why learn Web Scraping?

Whether you’re a data analyst, a web developer or even someone who wants to work as a freelancer you should learn web scraping.

For a data analyst building a dataset is extremely important, so without web scraping you simply can’t generate it in addition to that adding web scraping in your resume is a plus for you.

Web scraping can be used in a variety of fields, so let me give you some examples on what you can do with it:

• Generate leads,
• Drop shipping where basically you gonna constantly scrape products from different online stores and show case them on your website to make money,
• Monitor products prices to get the best deals, automation,
• Machine learning,
• Web scraping freelancer

Why LXML and not BeautifulSoup ?

LXML is a lightweight HTML parser even the most popular web scraping framework (Scrapy) is built on the top of LXML, BeautifulSoup is a little bit overloaded with the number of functions exposed to us, it has more functions to use, yes that’s right ! however in Web Scraping most of the time we use XPath and CSS Selectors to navigate and select what to scrape from the HTML web page (tree) so there is no need to learn about new functions and wasting all that time to familiarize yourself with the BeautifulSoup API and the internal architecture, in addition to all of that LXML in terms of performance is way better than BeautifulSoup.

What you’ll learn

• LXML core fundamentals
• XPath & CSS selectors
• How send HTTP requests with Python
• Scraping HTML web pages
• Scraping multiple pages using recursion
• Scraping APIs
• Splash HTTP API
• Scraping JavaScript websites using Splash
• Authentication and Login to websites using Requests
• Web scraping best practices
• Building datasets

#python #web-development #javascript #html