Here, I’m going to show you how to scrape data on a website using python and selenium. Here I scrape reviews posted by Tripadvisor users in their user profiles.

first of all, we need to have installed python and selenium in your machines. I’m assuming you know how to use pip.

Downloading the Selenium is simple.

pip install selenium

So now we are ready to code with python selenium. First, we need to import the following libraries because we need to use them during the implementation.

from selenium import webdriver
import csv
import requests
import re
import time

As you can see, we’ve import web driver and here we are going to use chrome web driver.

driver = webdriver.Chrome()
driver.set_page_load_timeout(1)

Then we give URL to open with chrome browser by using driver.get() function

driver.get(URL)
#to maximize the the tab 
driver.maximize_window()

The scraped data will be saved to a CSV file. Therefore we need to create a CSV file beforehand. We can find the name of the Tripadvisor user by using driver.find_element_by_class_name(“class_name”) method. Then create a CSV file by giving that name as username.csv. Then write headings to created CSV files.

username = driver.find_element_by_class_name(“gf69u3Nd”).text
filename1 = username+”.csv”

#open csv file and add titles
 with open(filename1, mode=’w’) as f:
 writer = csv.writer(f)
 writer.writerow([str(‘username’),str(‘reviewTitle’),str(‘reviewDetails’),str(‘reviewDate’),str(‘reviewFor’)])

#python