Collect and Process Twitter Feeds for Data Analysis: the Pythonic Way

Collect and Process Twitter Feeds for Data Analysis: the Pythonic Way

In part 1: We will look at data acquisition and building a python script to acquire, clean, and transfer data. In part 2: We will look at building visualizations and dashboards in Kibana and Elasticsearch. In Part 3: We will look at operationalizing the script into sustainment.

Introduction

This is a multi-part series on OSINT with Python and ELKstack.

In part 1: We will look at data acquisition and building a python script to acquire, clean, and transfer data.

In part 2: We will look at building visualizations and dashboards in Kibana and Elasticsearch.

In Part 3: We will look at operationalizing the script into sustainment.

Start with Why

I had an opportunity to interview with a large tech company but I didn’t know much about them (I’ve been oblivious to what’s popping up in Silicon Valley). Following due diligence and better prepare for the interview, I spent the weekend working on script to help me better understand the general sentiments and what the company was about. At that point in time, any news is new news, and I wanted to consume it.

Twitter is a great social networking platform that delivers almost real-time news generated and posted by “regular people”. It’s great, because it’s really unfiltered and you’re able to somewhat grasp what people are thinking. The goal of this project is to build a twitter scrapper, dump it into Elasticsearch and create visualization dashboards using Kibana.

Like most people, I’m a visual person. I love looking at dashboards, graphs, and pictures that makes sense of data. It helps me comprehend and gain insight in a particular subject. I’m also a project based learner. I learn better when I have an end goal. In this case, I’m learning: Python, Elasticsearch, and Kibana. I’m also learning how to use the Twitter API for other projects that I’m working on like Natural Language Processing (Sentiment Analysis, bots, docker, etc.)

kibana python data-analysis data-science elasticsearch

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Applied Data Science with Python Certification Training Course -IgmGuru

Master Applied Data Science with Python and get noticed by the top Hiring Companies with IgmGuru's Data Science with Python Certification Program. Enroll Now

Python For Data Science | Python For Data Analysis

Python for Data Science, you will be working on an end-to-end case study to understand different stages in the data science life cycle. This will mostly deal with "data manipulation" with pandas and "data visualization" with seaborn. After this, an ML model will be built on the dataset to get predictions. You will learn about the basics of the sci-kit-learn library to implement the machine learning algorithm.

50 Data Science Jobs That Opened Just Last Week

Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments. Our latest survey report suggests that as the overall Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments, data scientists and AI practitioners should be aware of the skills and tools that the broader community is working on. A good grip in these skills will further help data science enthusiasts to get the best jobs that various industries in their data science functions are offering.

An introduction to exploratory data analysis in python

Many a time, I have seen beginners in data science skip exploratory data analysis (EDA) and jump straight into building a hypothesis function or model. In my opinion, this should not be the case.

Exploratory Data Analysis is a significant part of Data Science

Data science is omnipresent to advanced statistical and machine learning methods. For whatever length of time that there is data to analyse, the need to investigate is obvious.