A framework for Twitter data collection.
For a full description, please check the preliminary version of Twitter Watch’s paper.
The current version merges the core and api containers but you can use the version in the original architecture by using the docker-compose-2.yml file and commenting the second line in the launcher.sh file (nohup python api/main.py > logs_flask.txt &
).
The overall implementation is a sequential execution of groups of tasks until the scheduled (and parallel) tasks are reached.
This tool was used to collect data on the Portuguese Twittersphere, the following figure summarizes the amount of accounts and tweets it collected.
cp example.env .env
and editdocker-compose up
(pass -d
for detached mode)docker exec some-mongo sh -c 'exec mongodump -d twitter --archive' > PATHTOLOCALFILE/dump.archive
docker exec -it twitter-watch_core_1 bash
Check pre-commit.com for more pre-commit functionality and then add it to the pre-commit config file.
To run, execute pre-commit run --all-files
.
Author: msramalho
Demo: https://msramalho.github.io/twitter-watch/
Source Code: https://github.com/msramalho/twitter-watch
#vuejs #javascript #vue