How to extract data using APIs in python

This content originally appeared on CodeSource.io and was authored by Md Niaz Rahman Khan

API stands for Application Programming Interface. It creates a relationship between two applications and makes them share data and talk to each other. In the metaphoric sense, an API can be considered with the restaurant process. Where the customer ordered food and the chef prepared the ordered food. Here, the waiter takes the order from the customer and places the order with the chef. Without the waiter, the process will be incomplete. The waiter plays the role of an API who makes them to talk each other.

Now, compare it with the real-world application. One common task we often do on a daily basis is to log in to some other application using Google. We use Google API for this purpose. A lot of tasks could be done using Google API like Single Sign On,  Scraping Google search results, OCR, Translation, Voice to Text, accessing drive, maintaining calendar, uploading videos on Youtube, analyzing website metrics etc.

Although Google doesn’t provide an official SERP API, scraping Google search results is possible via third party tools. Those tools can deal with enormous amounts of requests due to integrated rotating proxies that ensure the scraper doesn’t get blocked. Different tools provide different export methods, while some of them may provide exporting the results via an API straight to your application.

In this tutorial, we will use this concept of API and extract some data from other applications to ours. The process is simple and at the end of this tutorial, you will be able to extract data using APIs in Python on your own.

Prerequisite:

To complete this tutorial in the most effective way you need to keep some things in your mind.

• A configured computer with an internet connection.
• A text editor – You can choose any text editor on which you may write Python code such as sublime text, nodepad++, etc. In this tutorial, we will use visual studio code.
• Python3 is installed on your computer. We are using Python version 3.10.4. You may use this version or higher. But it is recommended not to use the Python version under 3.0.0 while completing this tutorial.
• Nice to have little knowledge about Request types. If you don’t, do not need to worry we will cover these later in this tutorial.

Things you will learn from this tutorial:

In this tutorial, we are going to cover the following things:

• Introduction and Types of Requests to an API
• Connecting to an API
• Extract Data from an API
• Conclusion

Introduction and Types of Requests to an API

In programming languages like Python, an API is considered some data which are available on the web and can be accessed by using some medium. While working with an API we need to make a connection with web applications and to do so, we need to use some commands. There are basically four types of requests and they are widely used. These are – Get, Post, Put, and Delete. Each of them performed different tasks.

GET Request requests some specific data from a specified resource in a server or an API.

POST Request lets us post or add some data to an API.

PUT Request updates one particular data.

DELETE Request lets us delete data.

These are basic concepts about different types of Requests and what they do. Now, we are ready to jump into the next part of this tutorial.

Connecting to an API

To work with an API first we need to connect the particular API with our application. The process is simple. We need to use the GET Request. But in python, you can not get the support of the Request module by default. This is the reason, we need to install a library named requests. The process of installing it is very easy. You can use pip or conda to install this. See the below command example:

pip install requests

# Or

conda install requests

By giving this command it will install the library for us and we are ready to use it in our application. Follow the below code example of connecting an API:

import requests
connect_API = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print(connect_API.status_code)

# Output: 200

First, we import the requests library that we have installed. This library comes with many methods. Get is one of them. The Get method requests something from an API. We have connected our application with an API using this method.

To check the response of this request we have used the status_code and it returns 200 which means our connection has been successful. There is some other status code available that represents a different meaning. These are-

• 200 – The status code 200 means everything is OK and the connection is successful.
• 301 – This means the server is redirecting to a different endpoint.
• 400 – It refers to a bad request that occurred by the Users.
• 401 – It shows when the user is not authenticated.
• 404 – It appears when the user finds some resource and not found.
• 500 – It indicates an internal server error.

These are some important status codes that will help you to understand the means of the codes. In our case, the code is 200 which means our application has been connected to an API successfully. Now, we are ready to jump into the next section.

Extract Data from an API

We have successfully connected our application with an API. Now, we need to extract some data from the connected API. To do so, we need to follow a few steps. The first step is to pull some data from the API and store it in a variable. To perform this action, follow the below code example:

import requests
connect_API = requests.get('https://jsonplaceholder.typicode.com/todos/1')
pull_data = connect_API.text
print(pull_data)

# Output:
# {
#   "userId": 1,
#   "id": 1,
#   "title": "delectus aut autem",
#   "completed": false
# }

Here, you can see that the data has been pulled out from the API and we have stored it into a pull_data variable in the text format.

The next step is to make this data a proper format so that the browser can understand it. To do so, we need to parse it through JSON(JavaScript Object Notation). The reason behind it is that the browser only understands JavaScript. The process of parsing is a simple task.

We need to import json library to make things workable in Python. This library provides json.load() method, which will let us convert and decode the text into a proper JSON format. Follow the below code example:

import requests
import json
connect_API = requests.get('https://jsonplaceholder.typicode.com/todos/1')
pull_data = connect_API.text
convert_json = json.loads(pull_data)
print(convert_json)

# Output:
# {'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

Here, you can see that we have converted the text into JSON format. The JSON format uses a key–value pair to store data. You notice, each data has a key and each key consists of a value.

We are ready to jump into the final steps where we can extract the data properly. In this step, we will use the key-value format which is known as Dictionary in Python. Python lets us store data with a key value by using Python Dict. Follow the below code example:

import requests
import json
connect_API = requests.get('https://jsonplaceholder.typicode.com/todos/1')
pull_data = connect_API.text
convert_json = json.loads(pull_data)
extract_data = convert_json['title']
print("Title of the User is:", extract_data)

# Output:
# Title of the User is: delectus aut autem

Here, you can see that we have successfully extracted some data from an API and printed them on our application. This is how you can extract data using APIs in python.

Conclusion

If you have made it this far then congratulation from myself to all of you because now you already know how to extract data using APIs in python. In this whole tutorial, we have covered the different types of requests to an API and which requests perform which action, different types of status codes and their meanings, connecting to an API, pulling out the data from an API, converting the data to a proper JSON format so the browser can understand, finally, prints the extracted data in our applications.

Now, it’s time to make your hands dirty with the codes. All you need to do is to take this tutorial as a reference for your learning and try to experiment with other stuff. Some examples can be, extracting data from a Covid-19 API, Gmail API, and so on. You may find them free to use. Finally, if you are stuck somewhere while performing this action, you may follow these steps to extract data using APIs in python.

This content originally appeared on CodeSource.io and was authored by Md Niaz Rahman Khan