How to Get Tweets using IDs with Tweepy, Twitter API and Python

In this post, I am going to explain how to fetch tweets using their IDs using Tweepy. I am going to explain why we are doing such thing? Why we store Tweet IDs instead of directly themselves, after that I am going to explain what is Tweepy and how to use that. And lastly we will see an example with Python.

First of all, the question is why we are using Tweet IDs? This is the result of hardware actually. We don’t want to use lots of memory all the time. We don’t have enough space!

After identification of tweets, we store their IDs, it means that we can fetch them later. But is it really necessary? How much space can we save with this method? Here is one example, since I am working on natural disasters in our world, I am searching for datasets all the time. And if you are data scientist, you need a large amount of data to find out correlation. Here is the example; https://crisisnlp.qcri.org is the website which stores and shares some datasets with data scientists to help them. In that website there is a resource number 5, which is about tweets published during seven major natural disasters.

If you want to download tweets of this datasets you need to download a data which is approximately 1.8GB. But if you want to download only their IDs, you need to download only 79MB. So Tweets are 22 times larger than their IDs. Now think about you are collecting data all the time and you have that kind of datasets for every topic. You need to buy lots of hard drive I guess which is of course not preferred. Because of this reason, we prefer to store tweet IDs after we choose and analyze them. That means we can fetch them later when we need it.

Tweepy and Twitter API

After this brief information, let’s assume that we found a good dataset on the internet that we think it will be useful for our work but it only consist of Tweet IDs. How we can fetch them? Tweepy is my answer. What is Tweepy?

Tweepy explain itself as a “An easy-to-use Python library for accessing the Twitter API.” It is actually a library which uses Twitter API. Twitter has it’s own API to access features of itself. In short with Tweepy, it will be much easier to use Twitter API. I am not going to dive in to Twitter API here, but please check this link to learn more: https://developer.twitter.com/en.html

Let’s continue with how to use Tweepy. To use Tweepy first we need to sign up as a Twitter developer. Please see the above link to sign up as a developer and take your credentials. Otherwise you can’t use the Twitter API which results you can’t use Tweepy as well.

Here are the steps for being a Twitter Developer;

Visit this link; https://developer.twitter.com/en/apply-for-access

Via above link you will apply twitter for a developer account.

Click the button “Apply for a developer account”.

And please login with your account.

So here is the page that welcomes you.

Please choose your reason why you want to use Twitter API. And click “Next”. After that there will be some pages to fill details. Twitter want to know more about your aim. You need to describe your project or idea to Twitter specifically. For example I wanted to fetch tweets for my thesis and I have explained details of my thesis to Twitter. After some detailed explanations. And then click “Next” until you submit your application.

Then you can wait for the answer from Twitter. If you receive an email like that, yey! Now you can start to use Twitter API.

Being able to send a request to Twitter API, you need to have authentication credentials. You need to create an app in order to have credentials. So please create an app under the section “Apps”.

Click the button “Create an app”, and you need to fill some details about your application. Also on the left side of the screen you can see answers of FAQs.

After completion of app creation, you have now your credentials under the section “Keys and tokens”.

Now it is time for coding! You have everything for your application. Install Tweepy, and use it to invoke the Twitter API.

#This code creates the dataset from Corpus.csv which is downloadable from the
#internet well known dataset which is labeled manually by hand. But for the text
#of tweets you need to fetch them with their IDs.
import tweepy

# Twitter Developer keys here
# It is CENSORED
consumer_key = 'XX'
consumer_key_secret = 'XX'
access_token = 'XX-XX'
access_token_secret = 'XX'

auth = tweepy.OAuthHandler(consumer_key, consumer_key_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)

# This method creates the training set
def createTrainingSet(corpusFile, targetResultFile):
    import csv
    import time

    counter = 0
    corpus = []

    with open(corpusFile, 'r') as csvfile:
        lineReader = csv.reader(csvfile, delimiter=',', quotechar="\"")
        for row in lineReader:
            corpus.append({"tweet_id": row[2], "label": row[1], "topic": row[0]})

    sleepTime = 2
    trainingDataSet = []

    for tweet in corpus:
        try:
            tweetFetched = api.get_status(tweet["tweet_id"])
            print("Tweet fetched" + tweetFetched.text)
            tweet["text"] = tweetFetched.text
            trainingDataSet.append(tweet)
            time.sleep(sleepTime)

        except:
            print("Inside the exception - no:2")
            continue

    with open(targetResultFile, 'w') as csvfile:
        linewriter = csv.writer(csvfile, delimiter=',', quotechar="\"")
        for tweet in trainingDataSet:
            try:
                linewriter.writerow([tweet["tweet_id"], tweet["text"], tweet["label"], tweet["topic"]])
            except Exception as e:
                print(e)
    return trainingDataSet

# Code starts here
# This is corpus dataset
corpusFile = "datasets/corpus.csv"
# This is my target file
targetResultFile = "datasets/targetResultFile.csv"
# Call the method
resultFile = createTrainingSet(corpusFile, targetResultFile)

Here is the code that I used for fetching tweets. Please fill your tokens and keys here, I have left them as “XX”. I have used the well-known corpus for that. But you can use same code for any other dataset of course. My corpus file looks like;

This is corpus which is about some tweets about big companies and labeled as positive, negative or neutral. As you can see, last column is tweet ID.

With the above code, we will take this IDs from corpus file, fetch the tweets one by one and write it to new file named “targetResultFile.csv”. Result will looks like that;

There are some key points when running the code. I wanted to explain everything with comments but let’s underline one important point. Twitter has some strict limits. You can’t fetch everything once.

Because of this limitations, you need to add some delays to your code or different business logic. Tweepy has it’s unique exceptions for that. I didn’t add this to my example code for now but, you can add the specific code blocks for specific exceptions for example RateLimitError. You can add some delay if you caught that exception. You may need to up t one hour to be able to fetch new tweets if you caught that exception. You can find more information here.

After you have successfully fetched you tweets, you may now need to preprocess your tweets. You can continue to read following story about preprocessing tweets to continue learn more about this topic.

We have successfully fetched tweets from Twitter and we are ready to continue! Thank you for your time! Please share if you liked it!

#Python #Twitter #Data Science #Machine Learning #Tweepy