Scraping involves the extraction of information from HTML Pages available over the Web. In Python, scraping can be performed through the Selenium library.
In this tutorial, I illustrate how to scrape a list of terms, distributed over two levels of nested pages, through Python selenium
. As example, I scrape the list of terms from Bocardi.
The full code of this tutorial can be downloaded from my Github Repository.
The selenium
library can be easily installed via pip
through the command pip install selenium
. In addition to the library, I also need to install the driver for my browser, which depends on the version of my browser. In this tutorial, I exploit the Chrome browser. I can check its version by entering chrome://settings/help
on the address bar of my browser.
In my case the version if 80, thus I can download the chrome driver from this link. Once downloaded, I can put the file into a generic folder of my file system and I need to configure the $PATH
variable with the path to the chrome driver:
.bash_profile
or .profile
file by adding the following line export PATH = "<path to web driver>: $ PATH"
and then restart your computer.In order to scrape data from a Web site, firstly I need to study the URIs structure. In my example, the list of terms is organized alphabetically, and for each letter of the alphabet there is a dedicated page, available at <basic_url>/dizionario/<current_letter>/
(first level of URI). For example, for the letter a
, the dedicated page is available at https://www.brocardi.it/dizionario/a/
. In addition, the list of terms for each letter is paginated in different pages. For each letter, the first page is available the the first level of URI, while starting from the second page, the URI changes and is available at <basic_url>/dizionario/<current_letter>/?page=<page_number>
. For example, for the letter a
, the list of terms in the second page is available at the link https://www.brocardi.it/dizionario/a/?page=2.
jkwxy
). For the external loop, I build a list containing all the letters, but the missing ones. I exploit string.ascii_lowercase
to build the list of letters.#data-collection #python #data-science