An easy technique for web scraping an interactive web chart

An easy technique for web scraping an interactive web chart

You browse a dynamic website with an interactive chart and it has all the data you need for your next data project. How should you go about web scraping?

You browse a dynamic website with an interactive chart and it has all the data you need for your next data project. How should you go about web scraping?

Image for post

For this process, you only need two libraries

  1. The requests library
  2. The pandas.io.json library → json_normalize

The key to this process is exploring the website’s Network tab before digging into HTML.

For this article, Gold prices in Brazilian Real will be scraped from bullionstar.comfor dates between 01/01/2019 to 11/09/2020.

Steps

  1. Right-click and click on Inspect

    Image for post

2. Reload the page and click on the Network tab

Image for post

3. This is the part that requires exercising your investigative skills!

Make sure to click on the XHR tab, it is an API in the form of an object that transfers data between a web browser and a web server

Within the XHR tab, explore the different objects to see if any house the web chart data by looking at the Preview tab.

After some digging, I'm able to find the web chart dataSeries stored in the “chartsData” object under the Preview tab.

data-science beautifulsoup request web-scraping selenium

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

50 Data Science Jobs That Opened Just Last Week

Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments. Our latest survey report suggests that as the overall Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments, data scientists and AI practitioners should be aware of the skills and tools that the broader community is working on. A good grip in these skills will further help data science enthusiasts to get the best jobs that various industries in their data science functions are offering.

Web Scraping using BeautifulSoup- COVID-19 Data

Web scraping is extracting large amounts of unstructured data from websites and storing it in a structured format in a desired file/database. We’ll see how it’s done in this blog.

Web Scraping E-Commerce Website Using Selenium

A simplified example of scraping an online books store. In this article we will go through a web scraping process of an E-Commerce website.

Data Science Pull Requests — A Method for Data Science Review & Merging

Data Science Pull Requests — A Method for Data Science Review & Merging. A step forward for MLOps and unlocking Open Source Data Science

Applications Of Data Science On 3D Imagery Data

The agenda of the talk included an introduction to 3D data, its applications and case studies, 3D data alignment and more.