Ever since Google Web Search API deprecation in 2011, I’ve been searching for an alternative. I need a way to get links from Google search into my Python script. So I made my own, and here is a quick guide on scraping Google searches with requests and Beautiful Soup.

First, let’s install the requirements. Save the following into a text file name requirements.txt

import urllib
import requests
from bs4 import BeautifulSoup

Google returns different search results for `mobile vs. desktop. So depending on the use case, we need to specify appropriate user-agent.

## desktop user-agent
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"
## mobile user-agent
MOBILE_USER_AGENT = "Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"

To perform a search, Google expects the query to be in the parameters of the URL. Additionally, all spaces must be replace with a +. To build the URL, we properly format the query and put it into the q parameter.

query = "hackernoon How To Scrape Google With Python"
query = query.replace(' ', '+')
URL = f"https://google.com/search?q={query}"

#web-scraping #google-search #python #seo #programming #coding