Liam Hurst

Liam Hurst

1565162958

An introduction to Python Requests library - How HTTP requests work

In many web apps, it's normal to connect to various third-party services by using APIs. When you use these APIs you can get access to data like weather information, sports scores, movie listings, tweets, search engine results, and pictures. You can also use APIs to add functionality to your app. Examples of these are payments, scheduling, emails, translations, maps, and file transfers. If you were to create any of those on your own it would take a ton of time, but with APIs, it can take only minutes to connect to one and access its features and data.

Table of Contents

  • Quick Overview of HTTP Requests
  • Install Python Requests
  • Our First Request
  • Status Codes
  • Headers
  • Response Text
  • Using the Translate API
  • Translate API Error Cases
  • Conclusion 

And since using an API is simply sending HTTP requests and receiving responses, Requests allows you to use APIs in Python. I'll demonstrate the use of a language translation API here so you can see an example of how it works so you can potentially use it in your own apps.

Quick Overview of HTTP Requests

HTTP requests are how the web works. Every time you navigate to a web page, your browser makes multiple requests to the web page's server. The server then responds with all the data necessary to render the page, and your browser then actually renders the page so you can see it.

The generic process is this: a client (like a browser or Python script using Requests) will send some data to a URL, and then the server located at the URL will read the data, decide what to do with it, and return a response to the client. Finally, the client can decide what to do with the data in the response.

Part of the data the client sends in a request is the request method. Some common request methods are GET, POST, and PUT. GET requests are normally for reading data only without making a change to something, while POST and PUT requests generally are for modifying data on the server. So for example, the Stripe API allows you to use POST requests to create a new charge so a user can purchase something from your app.

This article will cover GET requests only because we won't be modifying any data on a server.

When sending a request from a Python script or inside a web app, you, the developer, gets to decide what gets sent in each request and what to do with the response.

Install Python Requests

Before we can do anything, we need to install the library. So let's go ahead and install requests using pip. It's a good idea to create a virtual environment first if you don't already have one.

pip install requests

Our First Request

To start, let's use Requests for something simple: requesting the Scotch.io site. Create a file called script.py and add the following code to it. In this article, we won't have much code to work with, so when something changes you can just update the existing code instead of adding new lines.

import requests

res = requests.get(‘https://scotch.io’)

print(res)

So all this code is doing is sending a GET request to Scotch.io. This is the same type of request your browser sent to view this page, but the only difference is that Requests can’t actually render the HTML, so instead you will just get the raw HTML and the other response information.

We’re using the .get() function here, but Requests allows you to use other functions like .post() and .put() to send those requests as well.

You can run it by executing the script.py file.

python script.py

And here’s what you get in return:

Status Codes

The first thing we can do is check the status code. HTTP codes range from the 1XX to 5XX. Common status codes that you have probably seen are 200, 404, and 500.

Here’s a quick overview of what each status code means:

  • 1XX - Information
  • 2XX - Success
  • 3XX - Redirect
  • 4XX - Client Error (you messed up)
  • 5XX - Server Error (they messed up)

Generally, what you’re looking for when you perform your own requests are status codes in the 200s.

Requests recognizes that 4XX and 5XX status codes are errors, so if those status codes get returned, the response object from the request evaluates to False.

You can test if a request responded successfully by simply checking the response for truth. For example:

if res:
   print(‘Response OK’)
else:
   print(‘Response Failed’)

The message “Response Failed” will only appear if a 400 or 500 status code returns. Try changing the URL to some nonsense to see the response fail with a 404.

You can take a look at the status code directly by doing:

print(res.status_code)

This will show you the status code directly so you can check the number yourself.

Headers

Another thing you can get from the response are the headers. You can take a look at them by using the headers dictionary on the response object.

print(res.headers)

Headers are sent along with the request and returned in the response. Headers are used so both the client and the server know how to interpret the data that is being sent and received in the response/response.

We see the various headers that are returned. A lot of times you won’t need to use the header information directly, but it’s there if you need it.

The content type is usually the one you may need because it reveals the format of the data, for example HTML, JSON, PDF, text, etc. But the content type is normally handled by Requests so you can easily access the data that gets returned.

Response Text

And finally, if we take a look at res.text (this works for textual data, like a HTML page like we are viewing) we can see all the HTML needed to build the home page of Scotch. It won’t be rendered, but we see that it looks like it belongs to Scotch. If you saved this to a file and opened it, you would see something that resembled the Scotch site. In a real situation, multiple requests are made for a single web page to load things like images, scripts and stylesheets, so if you save only the HTML to a file, it won’t look anything like what the Scotch.io page looks like in your browser because only a single request was performed to get the HTML data.

print(res.text)

Using the Translate API

So now let’s move on to something more interesting. We’ll use the Yandex Translate API to perform a request to translate some text to a different language.

To use the API, first you need to sign up. After you sign up, go to the Translate API and create an API key. Once you have the API key, add it to your file as a constant. Here’s the link where you can do all those things: https://tech.yandex.com/translate/

API_KEY = ‘your yandex api key’

The reason why we need an API key is so Yandex can authenticate us every time we want to use their API. The API key is probably the simplest form of authentication, because it’s simply added on to the end of the request URL when being sent.

To know which URL we need to send to use the API, we can look at the documentation for Yandex here: https://tech.yandex.com/translate/doc/dg/reference/translate-docpage/

If we look there, we’ll see all the information needed to use their Translate API to translate text.

API documentation can be difficult to read at times, but in this case it’s simple. When we see a URL with ampersands (&), question marks (?), and equals signs (=), you can be sure that the URL is for GET requests. Those symbols specify the parameters that go along with the URL.

Normally things in square brackets ([]) will be optional. In this case, format, options, and callback are optional, while the key, text, and lang are required for the request.

And of course it’s easy to see the URL. So let’s add some code to send to that URL. You can replace the first request we created with this:

url = ‘https://translate.yandex.net/api/v1.5/tr.json/translate
res = requests.get(url)

There are two ways we can add the parameters. We can either append it to the end of the URL directly, or we can have Requests do it for us. Having Requests do it for us is much easier.

To do that, we can create a dictionary for our parameters. The three items we need are the key, the text, and the language.

Let’s create the dictionary using the API key, ‘Hello’ for the text, and ‘en-es’ as the lang, which means we want to translate from English to Spanish.

If you need to know any other language codes, you can look here: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes You are looking for the 639-1 column.

We create a params dictionary by using the dict() function and passing in the keys and values we want in our dictionary.

params = dict(key=API_KEY, text=‘Hello’, lang=‘en-es’)

Now we take the parameters dictionary and pass it to the .get() function.

res = requests.get(url, params=params)

When we pass the parameters this way, Requests will go ahead and add the parameters to the URL for us.

Now let’s add a print statement for the response text and view what gets returned in the response.

print(res.text)

We see three things. We see the status code, which is exactly the same status code of the response itself, we see the language that we specified, and we see the translated text inside of the list. So you should see ‘Hola’ for the translated text.

Try again with en-fr as the language code, and you should see ‘Boujour’ in the response now.

params = dict(key=API_KEY, text=‘Hello’, lang=‘en-fr’)

Let’s take a look at the headers for this particular response.

print(res.headers)

Obviously the headers should be different because we’re communicating with a different server, but in this case the content type is application/json instead of text/html. What this means that the data can be interpreted as JSON.

When application/json is the content type of the response, we are able to have Requests convert the response to a dictionary and list so we can access the data easier.

To have the data parsed as JSON, we use the .json() method on the response object.

If you print it, you’ll see that the data looks the same, but the format is slightly different.

json = res.json()
print(json)

The reason why it’s different is because it’s no longer plain text that you get from res.text. This time it’s a printed version of a dictionary.

Let’s say we want to access the text. Since this is now a dictionary, we can use the text key.

print(json[‘text’])

And now we only see the data for that one key. In this case we are looking at a list of one item, so if we wanted to get that text in the list directly, we can access it by the index.

print(json[‘text’][0])

And now the only thing we see is the translated word.

So of course if we change things in our parameters, we’ll get different results. Let’s change the text to be translated from Hello to Goodbye, change the target language back to Spanish, and send the request again.

params = dict(key=API_KEY, text=‘Goodbye’, lang=‘en-es’)

Try translating longer text in different languages and see what responses the API gives you.

Translate API Error Cases

Finally, we’ll take a look at an error case. Everything doesn’t always work, so we need to know when that happens.

Try changing your API KEY by removing one character. When you do this your API key will no longer by valid. Then try sending a request.

If you take a look at the status code, this is what you get:

print(res.status_code)

So when you are using the API, you’ll want to check if things are successful or not so you can handle the error cases according to the needs of your app.

Conclusion

Here’s what we learned:

  • How HTTP requests work
  • The various status codes possible in a response
  • How to send requests and receive responses using the Python Requests library
  • How to use a language translation API to translate text
  • How to convert application/json content responses to dictionariesThat covers the basics of Requests in Python. Of course you can do so much more, but what I talked about in this article is the foundation of most requests. Things may change slightly depending on the circumstances, but the basic ideas will remain the same.

If you want to do more, check out https://apilist.fun/ to see different APIs that are available, and try to use them with Python Requests.

Thanks for reading

If you liked this post, share it with all of your programming buddies!

Follow us on Facebook | Twitter

Further reading about Python

Complete Python Bootcamp: Go from zero to hero in Python 3

Machine Learning A-Z™: Hands-On Python & R In Data Science

Python and Django Full Stack Web Developer Bootcamp

Complete Python Masterclass

Python Tutorial - Python GUI Programming - Python GUI Examples (Tkinter Tutorial)

Computer Vision Using OpenCV

OpenCV Python Tutorial - Computer Vision With OpenCV In Python

Python Tutorial: Image processing with Python (Using OpenCV)

A guide to Face Detection in Python

Machine Learning Tutorial - Image Processing using Python, OpenCV, Keras and TensorFlow

PyTorch Tutorial for Beginners




#python #web-development

What is GEEK

Buddha Community

An introduction to Python Requests library - How HTTP requests work
Ray  Patel

Ray Patel

1619518440

top 30 Python Tips and Tricks for Beginners

Welcome to my Blog , In this article, you are going to learn the top 10 python tips and tricks.

1) swap two numbers.

2) Reversing a string in Python.

3) Create a single string from all the elements in list.

4) Chaining Of Comparison Operators.

5) Print The File Path Of Imported Modules.

6) Return Multiple Values From Functions.

7) Find The Most Frequent Value In A List.

8) Check The Memory Usage Of An Object.

#python #python hacks tricks #python learning tips #python programming tricks #python tips #python tips and tricks #python tips and tricks advanced #python tips and tricks for beginners #python tips tricks and techniques #python tutorial #tips and tricks in python #tips to learn python #top 30 python tips and tricks for beginners

Ray  Patel

Ray Patel

1619510796

Lambda, Map, Filter functions in python

Welcome to my Blog, In this article, we will learn python lambda function, Map function, and filter function.

Lambda function in python: Lambda is a one line anonymous function and lambda takes any number of arguments but can only have one expression and python lambda syntax is

Syntax: x = lambda arguments : expression

Now i will show you some python lambda function examples:

#python #anonymous function python #filter function in python #lambda #lambda python 3 #map python #python filter #python filter lambda #python lambda #python lambda examples #python map

Ray  Patel

Ray Patel

1619571780

Top 20 Most Useful Python Modules or Packages

 March 25, 2021  Deepak@321  0 Comments

Welcome to my blog, In this article, we will learn the top 20 most useful python modules or packages and these modules every Python developer should know.

Hello everybody and welcome back so in this article I’m going to be sharing with you 20 Python modules you need to know. Now I’ve split these python modules into four different categories to make little bit easier for us and the categories are:

  1. Web Development
  2. Data Science
  3. Machine Learning
  4. AI and graphical user interfaces.

Near the end of the article, I also share my personal favorite Python module so make sure you stay tuned to see what that is also make sure to share with me in the comments down below your favorite Python module.

#python #packages or libraries #python 20 modules #python 20 most usefull modules #python intersting modules #top 20 python libraries #top 20 python modules #top 20 python packages

Top 7 Python Libraries Used For Hacking

python is one of the most go-for languages among the developers due to the availability of open-source libraries and frameworks. According to a survey reportPython is the top language preferred for Statistical Modelling, and an overwhelming majority of practitioners prefer Python as the language for statistical works.

Python has become a favourite language for hackers these days. The reason is the presence of pre-built tools and libraries, which makes hacking easy. In fact, the language is adequate for ethical hacking as ethical hackers need to develop smaller scripts, and Python fulfils this criterion.

Below here, we listed down the top 7 Python libraries used in hacking.

1| Requests

Stars: 43.3k

**About: **Requests is a simple HTTP library for Python that allows a user to send HTTP/1.1 requests extremely easily. This library helps in building robust HTTP applications and includes intuitive features such as automatic content decompression and decoding, connection timeouts, basic & digits authentication, among others.

Know more here.

2| Scapy

Stars: 5.5k

About: Scapy is a powerful Python-based interactive packet manipulation program and library. This library is able to forge or decode packets of a wide number of protocols, send them on the wire, capture them, store or read them using pcap files, match requests, and more. It allows the construction of tools that can easily scan or attack networks. It is designed to allow fast packet prototyping by using default values that work. It can also perform tasks such as sending invalid frames, injecting your own 802.11 frames, combining techniques, such as VLAN hopping with ARP cache poisoning, VOIP decoding on WEP encrypted channel, etc., which most other tools cannot.

Know more here.

3| IMpacket

**Stars: **5.3k

**About: **IMpacket is a library that includes a collection of Python classes for working with network protocols. It is focused on providing low-level programmatic access to network packets. It allows Python developers to craft and decode network packets in a simple and consistent manner. The library provides a set of tools as examples of what can be done within the context of this library.

Know more here.

4| Cryptography

**Stars: **3.5k

**About: **Cryptography is a package which provides cryptographic recipes and primitives to Python developers. It includes both high-level recipes and low-level interfaces to common cryptographic algorithms such as symmetric ciphers, message digests and key derivation functions. This library is broadly divided into two levels. One is with safe cryptographic recipes that require little to no configuration choices. The other level is low-level cryptographic primitives, which are often dangerous and can be used incorrectly.

Know more here.

#developers corner #hacking tools #libraries for hacking #python #python libraries #python libraries used for hacking #python tools

August  Larson

August Larson

1625100480

4 Cool Python Libraries That You Should Know About

Discover useful Python libraries that you should try out in your next project

Some of my most popular blogs are about Python libraries. I believe that they are so popular because Python libraries have the power to save us a lot of time and headaches. The problem is that most people focus on those most popular libraries but forget that multiple less-known Python libraries are just as good as their most famous cousins.

Finding new Python libraries can also be problematic. Sometimes we read about these great libraries, and when we try them, they don’t work as we expected. If this has ever happened to you, fear no more. I got your back!

In this blog, I will show you four Python libraries and why you should try them. Let’s get started.

#python #coding #programming #cool python libraries #python libraries #4 cool python libraries