When I first started web scraping with BeautifulSoup4, I found that the most difficult hoop to jump through was pagination. Getting the elements from a static page seemed fairly straightforward — but what if the data I wanted was not on the initial page I loaded into my script? In this project we will try our hand at pagination using Selenium to cycle through the pages of an Amazon results page, and saving all of the data in a .jsonl file.

What is Selenium?

Selenium is an open-source browser automation tool, mainly used for testing web applications. It’s able to mimic user input such as mouse movements, key presses, and page navigation. There are also many methods which allow for element selection on the page. The main workhorse behind the library is the Webdriver, which makes automation of browser tasks a fairly straightforward affair.

#selenium #data #python #web-scraping #amazon