Building a web scraper with Selenium and BeautifulSoup4.

Building a web scraper with Selenium and BeautifulSoup4.

Building a web scraper with Selenium and BeautifulSoup4. How to automate annoying tasks with Python . Installing Requests and BeautifulSoup. Install pipenv first, and then: pipenv install requests beautifulsoup4.

I recently started to gain interest in web scraping, and I thought it would be fun to try and build something to learn some useful tools.

I needed a project to start, I thought about a problem I had from the first year of college: courses material download.

The Problem

My university website works pretty well, however all the courses material are organized in long trees of folders over folders and theres no bulk download capabilities nor there is a public API that I can take advantage of. Students have to manually download every file for every course they are attending. This require a lot of time to download and extract all the zip, especially when you have something like 6 to 10 courses.

The uni website makes use of the cursed Shibboleth(_Im shivering just by writing its name) _and of javascript dynamic rendering. You need to travel the website to find the links and the navigation is extremely cluttered. Using _only _is not an option.

Image for post

scraping selenium automation programming python

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Guide to Python Programming Language

Guide to Python Programming Language

Automate Instagram Using Python and Selenium.

Automaton Scripting is one of the most interesting and fascinating things to do as a programmer and control the browser with the help of a code. In this particular article, we will see how to**log In**to the Instagram account using Python and the power of**selenium**. Lets understand firstly what is selenium and its advantages and features.

Selenium with Python

Selenium with Python

Web Scraping using Python and Selenium.

The modern way to scrap. Web Scraping Using Selenium Python. How to navigate through multiple pages of a website and scrape large amounts of data using Selenium in Python ...

Selenium Webdriver Automation with Python Programming for Beginners

Selenium Webdriver Automation with Python Programming for Beginners. Implement of Python Selenium Frameworks from scratch. Learn Selenium Webdriver Automation with Python from scratch. Selenium with Python Full Course For Beginners. Learn Pytest tutorial for Beginners