You browse a dynamic website with an interactive chart and it has all the data you need for your next data project. How should you go about web scraping?

For this process, you only need two libraries

1. The requests library
2. The pandas.io.json library → json_normalize

The key to this process is exploring the website’s Network tab before digging into HTML.

For this article, Gold prices in Brazilian Real will be scraped from bullionstar.comfor dates between 01/01/2019 to 11/09/2020.

Steps

1. Right-click and click on Inspect

2. Reload the page and click on the Network tab

3. This is the part that requires exercising your investigative skills!

Make sure to click on the XHR tab, it is an API in the form of an object that transfers data between a web browser and a web server

Within the XHR tab, explore the different objects to see if any house the web chart data by looking at the Preview tab.

After some digging, I’m able to find the web chart dataSeries stored in the “chartsData” object under the Preview tab.

#data-science #beautifulsoup #request #web-scraping #selenium