How to automate navigating over a page with infinite data

Photo by Dan Schiumarini on Unsplash.

Selenium is amazing! It gives you the opportunity to scrape everything you want with great simplicity. According to its documentation,_ “_It provides extensions to emulate user interaction with browsers.” Moreover, it is supported by several programming languages and each browser has an interface called a WebDriver that allows the automation.

Anyone who has tried to acquire data from a page has noticed that HTML elements can appear on the page depending on when they are requested. In other words, a website used to contain a lot of data cannot represent all of it at the same time. Therefore, it may be necessary to request the representation of the data step by step, scrolling the page.

I’ve already dealt with this problem several times. I solved it differently each time, so I would like to show you the easiest, fastest, and most effective way to get the data from a page that requires different elements by scrolling the page itself or a section of the page.

#programming #selenium #data-science #python #data-scraping