How to Perform an HTTP Request in Python

In this tutorial, we will cover how to download an image, pass an argument to a request, and how to perform a 'post' request to post the data to a particular route. Also, you'll learn how to obtain a JSON response to do a more dynamic operation.

HTTP
Libraries in Python to make HTTP Request
Request in Python
Using GET Request
Downloading and Saving an image using the Request Module
Passing Argument in the Request
Using POST Request
JSON Response

HTTP

HTTP stands for the 'HyperText Transfer Protocol,' where communication is possible by request done by the client and the response made by the server.

For example, you can use the client(browser) to search for a 'dog' image on Google. Then that sends an HTTP request to the server, i.e., a place where a dog image is hosted, and the response from the server is the status code with the requested content. This is a process also known as a request-response cycle. You can also look at this article, What is HTTP for a more detailed explanation.

Libraries in Python to make HTTP Request

There are many libraries to make an HTTP request in Python, which are httplib, urllib, httplib2 , treq, etc., but requests are the simplest and most well-documented libraries among them all.

You'll be using the request library for this tutorial and the command for doing this is below:

pip install requests

Request in Python

According to Wikipedia, "requests are a Python HTTP library, released under the Apache2 License. The goal of the project is to make HTTP requests simpler and more human-friendly. The current version is 2.22.0"

Using GET Request

GET request is the most common method and is used to obtain the requested data from the specific server.

You need to import the required modules in your development environment using the following commands:

import requests

You can retrieve the data from the specific resource by using 'request.get('specific_url')', and 'r' is the response object.

r =requests.get('https://xkcd.com/1906/')

Status Code

According to Wikipedia, "Status codes are issued by a server in response to a client's request made to the server.". There are lots of other status codes and detailed explanations that can be found here: HTTP Status Code.

However, the two most common status code is explained below:

r.status_code

You can see below after running the above code the status code is '200' which is 'OK,' and a request is successful.

Headers

You can view the response headers by using '.headers.' where it returns the Python Dictionaries. They return a lot of additional information containing the case insensitive name with resources types along with the server name, version, etc., and are also included with the code shown below.

r.headers

The vital information obtained in the above code is the Server name as 'Apache', content type, Encoding, etc.

r.headers['Content-Type']

You can see above the type of content of the header by using 'content-type' which is case insensitive, and 'Content-Type' would also give the same result as below.

Response Content

You can get the HTML text of a page by using '.text.' where the request can decode any content automatically from the server, and it is in a Unicode form with the code below:

r.text

You can get an entire page of HTML and parsing can be done by the help of HTML parser.

'\r\n<!DOCTYPE html>\r\n<html>\r\n<head>\r\n\r\n<link href="http://www.smbc-comics.com/comiccontrol/defaultstyles.css?=2" rel="stylesheet" type="text/css" />\r\n<link rel="shortcut icon"
href="http://www.smbc-comics.com/favicon.ico" type="image/x-icon" />\r\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />\r\n<link rel="shortcut icon"
href="http://www.smbc-comics.com/favicon.ico" type="image/x-icon">\r\n<link rel="icon"
href="http://www.smbc-comics.com/favicon.ico"

Downloading and Saving an Image Using Request module

You need to import the module, i.e., using the requests command in your local computer and 'receive' the response object with the 'request.get.' along with the image URL to be download as done below.

import requests
receive = requests.get('https://imgs.xkcd.com/comics/making_progress.png')

with open(r'C:\Users\Dell\Desktop\comics\image5.png','wb') as f: 
     f.write(recieve.content)

You can see the 'with' statement above helps to manage the file stream using the with open function where the required path specifies for doing a specific operation, 'r' which converts the normal string to a raw string. I.e., all of the characters need to remain the same where the location of a local computer contains '\' in C:\Users\Dell\Desktop\comics\image5.png' needs to be preserved and should not be escaped. The mode for opening is 'wb' which is writing the files in a binary way, and 'f' is the file object that has to write a function to write the appropriate content, i.e., downloading the required image.

The final code for downloading and saving an image using the request module is the following:

import requests
receive = requests.get('https://imgs.xkcd.com/comics/making_progress.png')
with open(r'C:\Users\Dell\Desktop\comics\image5.png','wb') as f:
  f.write(recieve.content)

Passing Argument in the Request

You'll be using the httpbin which is what simple HTTP libraries use for testing where the data would have to be given after the 'httpbin.org/get?key=value, but request provides an easy way to make a dictionary where it is passed as an argument using the 'param' keyword.

You can see the example below the 'ploads' as the dictionary containing 'things' and 'total' as key with its respective value as 2 and 25 is passed as an argument to 'get' where 'params=ploads'

import requests
ploads = {'things':2,'total':25}
r = requests.get('https://httpbin.org/get',params=ploads)
print(r.text)
print(r.url)

You can also view the URL using the command 'r.url' as follows:

print(r.url)

The output after doing this is the URL containing the dictionary that you've set up in key-value form after the '?' sign.

Using POST Request

POST is the most common request method used to send data mostly through 'form' to the server for creating/updating in the server.

You can use 'post' a form data to the route where the 'requests.post' and make a dictionary called 'pload' where it is sent as an argument to the post to 'data=pload'.

import requests
pload = {'username':'Olivia','password':'123'}
r = requests.post('https://httpbin.org/post',data = pload)
print(r.text)

The result below is the JSON result from the 'httpbin' website along with necessary data is the 'form', which have both password and username as key and with its value testing and 123 and Olivia respectively.

JSON Response

JSON stands for the 'JavaScript Object Notation' where it is the most common way of interchanging the data format, which is easy to read and write and is based on the common language 'JavaScript'. It is a format which is platform-independent and based on objects where the data is in the form 'key-value' pair. If you want to know more about JSON, have a look at this JSON Data in Python tutorial.

Converting JSON to Python Dictionary

You can see below 'r.json()' creates a Python dictionary from the JSON response given by the 'httpbin' website.

import requests
pload = {'username':'olivia','password':'123'}
r = requests.post('https://httpbin.org/post',data = pload)
print(r.json())

The result obtained from the above code is Python dictionary and is also in ‘key-value’ which is shown below.

Converting JSON to Python dictionary and storing in a variable.

You can have a JSON data to be Python dictionary and do the more dynamic operation by using '‘r.json()’ as done below.

import requests

pload = {‘username’:‘olivia’,‘password’:‘123’}

r = requests.post(‘https://httpbin.org/post’,data = pload)

r_dictionary= r.json()

print(r_dictionary[‘form’])

The dictionary item can be accessed through ‘r_dictionary[‘form’]’ where only the form data is shown below.

Conclusion

Congratulations. You’ve finished the tutorial and learned about the basics of HTTP. Also, you learned about the request library in Python to make different types of request’s like ‘get’ to download an image, passing an argument to a request, and a ‘post’ request to post the data to a particular route. Finally, you learned how to obtain a JSON response to do a more dynamic operation.

Thanks for reading. Keep Visiting. If you liked this post, share it with all of your programming buddies!