For those who want to see it done in front of your eyes, check out my YouTube video at the bottom of the page.

Scraping the web can be done for a TON of reasons.

Do you want to get stats on your football team so you can algorithmically manage your fantasy team? Boom, make a web scraper that scrapes ESPN. Track your competitor’s activity on different social media? Great, that’s covered here too.

Or maybe you’re a Developer Advocate who is looking for good ways to measure his OKR of hackathon involvement and there is no current good tool out there so you want to build your own.

That last one was oddly specific, and is what we are going to be looking for! This tutorial shows how you can get all the hackathons from devpost that are ending in the next 50 days, based on the keyword blockchain .

Anyway, let’s jump right into how we can scrape anything with python. I’m going to assume you have space where you can code, and are familiar with how to work with python.

The documentation for this is very strong, so be sure to check it out after this tutorial!

NOTE

Beautiful soup works great for **static **web pages. If you follow this and get weird/bad results, you’ll probably need a web driver to scrape the site. Stay tuned for part 2 for that tutorial.

1. pip install requests and beautiful soup

pip install requests

pip install beautifulsoup

Run those two so you can work with the packages.

#beautifulsoup #scrape #devrel #hackathons #python