When you need to extract information from the web, you will inevitably come across the term "web scraping". At the same time, you will find a myriad of services and tools, which want to help you in your endeavor.
With such a large number, it, unfortunately, is not always easy to quickly find the right tool for your very own use case and to make the right choice. That's exactly what we want to check out in today's article.
We'll be taking a closer at the tools, both commercial and open-source, available in the data scraping and data extraction landscape and elaborate on their features and how you may use them best for your particular use case.
Web scraping is all about collecting content from websites. Scrapers come in many shapes and forms and the exact details of what a scraper will collect will vary greatly, depending on the use cases.
A very common example is search engines, of course. They continuously crawl and scrape the web for new and updated content, to include in their search index. Other examples include:
Many of us like to play Dart 🎯, but we shouldn't necessarily pick our scraping platform (or technology) like that, right?
So, before we simply jump in at the deep end, let's establish a few key parameters for our scraping project, which should help us narrow down the list of potential scraping solutions.
site:
filter)?Once you have a clearer picture of your requirements, it should be easier to match them against the available technologies and platforms and pick the most appropriate tool for your particular scraping job.
All right, let's now take a closer look at the different types of web scrapers and popular representatives of each category. Here we go 🥳
SaaS scraping platforms typically provide an all-in-one service, where you use their tools to define which sites you'd like to scrape and how retrieved data should be transformed and, ultimately provided to you.
While they usually come at a recurring subscription fee, they also provide quite a few additional services (e.g. proxy management, browser support) that other solutions either do not support at all or only via third-party plugins.
Generally speaking, choosing a SaaS platform for your scraping project will provide you with the most comprehensive package, both, in terms of scalability and maintainability.
ScrapingBee offers a lightweight REST API (along with support libraries for popular language platforms) which provides easy access to all the platform's features.
Among those features, you will find support for data extraction (using CSS selectors), screenshots of pages, access to Google's search API, and traditional (data-center), as well as, premium, residential proxies. Especially the latter is often necessary, in order to avoid being blocked while accessing a site.
ScrapingBee also provides access to a full-fledged Chrome browser engine, which is particularly important when scraping websites which heavily rely on JavaScript and client-side rendering.
ScrapingBee is for developers and tech-companies who want to handle the scraping pipeline themselves without taking care of proxies and headless browsers.
ScrapingBee's black box approach ensure that all the proxy and network management is taken care of by the platform and the user only needs to provide the desired site addresses, along with the applicable request parameters.
👍
👎
Diffbot offers a set of web APIs, which return the the scraped data in a structured format. The service supports sentiment and natural language analysis, though it is rather on the pricey side, with the smallest plan starting at USD 300 a month.
Diffbot's primary audience are developers and tech-companies, whose use cases focus more on data analysis (including sentiments and natural language).
👍
👎
Contrary to SaaS providers, desktop scrapers are (locally) installed applications over which you have full control (very much like your web browser).
While they typically do not come with a subscription tag, and are either freely available or for a one-time license fee, they also require you to maintain any scraper instances you are running. That means you need to provide the hardware, the connectivity, and the overall system maintenance. Depending on your setup, you may also experience issues with scaling your scraper instance.
However, for smaller projects, they are definitely a viable alternative.
ScrapeBox is a desktop scraper, available for Windows and macOS, with a strong focus on SEO related tasks, and the vendor claiming it to be the "Swiss Army Knife of SEO". Though, it does come with a number of other features, which extend the scope of SEO (e.g. YouTube scraping, email aggregation, content posting, and more).
ScrapeBox positions itself primarily for SEO use, but it may be equally useful for bulk scraping of YouTube and for general content scraping.
With its limits on scalability and proxy support, it may be particularly useful for scraping projects with smaller datasets and where it is not necessary to rotate or specify the client location.
👍
👎
ScreamingFrog's SEO spider is a website crawler for Windows, macOS, and Linux. It allows you to crawl URLs to analyze and perform technical audits and onsite SEO. It is able to crawl both small and large websites efficiently, while allowing you to analyze the results in real-time.
The following video provides you with a quick overview of the tool:
Out of all the platforms and services mentioned in this article, ScreamingFrog focuses exclusively on SEO, so it will be most useful for SEO professional and agencies specializing in this field.
👍
👎
Easy Web Extract is a classic Windows desktop application and provides a user-friendly UI, where most data selection steps can be configured without the need for code.
Unlike ScrapeBox and ScreamingFrog, Easy Web Extract does not place its main emphasis on SEO, but markets itself rather as general-purpose scraper.
With the application being limited by local system and network resources, you may experience scalability and site block issues, though. In this context, it may work best for small scraping jobs.
👍
👎
Another rather popular category of web scrapers is based on browser extensions. These scrapers run directly in your web browser instance and make full use of your browser engine and its integrated web technologies (the DOM, CSS styles and selectors, and running JavaScript).
Both Firefox and Chrome have countless such scraper tools in their extension galleries. For example, for Chrome, you can find a full list at https://chrome.google.com/webstore/search/scraper?_category=extensions, but let's take a look at three representatives of this category of scraper.
WebScraper is one of the most popular Chrome scraper extensions. It allows you to scrape websites directly from your browser, without the need to locally set up any tools or or write scraping script code.
Here is a screenshot of the interface (accessible within the Chrome dev tools):
There's also a short video covering the basics of the extension:
They also offer a cloud-based, paid subscription service, which allows you to execute your scraping tasks on their infrastructure. This may be especially useful if your script needs to set a particular scraping location, with the use of proxies.
Companies without developers, marketing teams, product managers...
👍
👎
Instant Data Scraper is the extension of webrobots.io. You simply add it to your Chrome profile and, after which a new button will show up in your browser toolbar, which provides you access to the scraper's features.
The extension is very data table-centric and you only need to select the desired data items with your mouse. Once scraping has finished you can export the data as CSV or Excel file.
Check out the following video to watch Instant Data Scraper live in action - even coupled with nice banjo tune 🪕:
For quick, on-the-fly scraping sessions.
👍
👎
Scraper uses XPath expressions to extract data. You add the extension, open the desired page, right click and select "Scrape similar".
The extension does not pursue any commercial interest at the moment, so it is completely free, but neither offers any additional services (e.g. cloud-based execution).
For basic extraction tasks without complex page structures or browsing requirements.
👍
👎
Last, but not least, there's of course also always the option to build your very own, fully customized scraper in your favorite programming language.
You'll find web scraping libraries and entire frameworks for almost every language and even somewhat more exotic languages, such as the statistcal R, have support for web scraping.
While we will focus in the following examples on Python, PHP, and JavaScript, please also feel free to check out ScrapingBee's blog, where you will find lots of more language examples, among them for Ruby, Groovy, Perl, Go, and C++ - just to name a few.
Scrapy is a free open-source web-crawling framework written in Python. As it handles requests in an asynchronous fashion, it performs quite well with a large number of sites, which contributes to its ability to scale well.
Scrapy definitely is for an audience with a Python background. While it serves as framework and handles lots of the scraping on its own, it still is not an out-of-the-box solution but requires sufficient experience in Python.
It works particularly well in large-scale web scraping, for example:
👍
👎
pyspider is another open-source web crawling tool. It has a web UI that allows you to monitor tasks, edit scripts and view your results.
Similarly to Scrapy, it requires a Python background, but its integrated UI also makes it more suitable for the general public and provides a more user-friendly UX.
👍
👎
Goutte is a PHP library designed for general-purpose web crawling and web scraping. It heavily relies on Symfony components and conveniently combines them to support your scraping tasks.
Goutte provides a nice API to crawl websites and extract data from HTML/XML responses.
It also integrates nicely with the Guzzle requests library, which allows you to customize the framework for more advanced use cases.
Being a PHP library, Goutte certainly is limited to a PHP environment, but if your language choice is PHP, you may definitely want to check it out.
👍
👎
After our short excursion to PHP-land, we are right back to Python. This time with Beautiful Soup.
Unlike Scrapy and pyspider, BS4 - as fans of the library call it affectionately 🤩 - is not a framework but rather a traditional library which you can use in your scraper application.
BS4 is a great choice if you decided to go with Python for your scraper but do not want to be restricted by any framework requirements.
Beautiful Soup offers a straightforward set of functions which will support you in building your own scraper.
👍
👎
If you are familiar with jQuery, you will immediately feel at home with Cheerio.js, it essentially is the server-side counterpart.
Cheerio supports all CSS selector syntax you know from jQuery and allows you to parse HTML documents from a number of sources and extract data with a familiar $('')
call.
Can really be summarized in one sentence: when you have HTML content to parse in a JavaScript or Node.js environment. And yes, you should be familiar with JavaScript, Node.js, and npm.
All right, those were two sentences. 😳
👍
👎
Puppeteer is a Node.js library which acts as a bridge to a Chrome Headless instance, which allows you to control a full-fledged browser setup and scrape the web from your JavaScript code as if you were any regular user.
Full control in this context means you can take screenshots, load SPAs, and send and handle JavaScript events.
Puppeteer will be your go-to tool if your choice of platform is JavaScript and you want to scrape JavaScript-heavy sites (e.g. SPAs) where Cheerio, for example, did not manage to access the desired data. Backed by a proper browser engine, Puppeteer will grant you access to that data treasure in no time.
👍
👎
While not a scraper application or code framework per se, it still is worth mentioning Common Crawl.
The project does not function as data extractor, like the services and tools we so far talked about, but approaches the topic from a very different angle. It essentially crawls and scrapes the web in advance and provides that data as publicly available datasets for everyone to access at no cost.
To put their crawling efforts and the available data into perspective, as of the time these lines are written, their current dataset is close to 400 TB (to have the inevitable comparison to some completely unrelated object 😉, that's about 650,000 traditional CDs).
Common Crawl will be ideal if its datasets match your requirements. If the quality of the data it pre-scraped is sufficient for your use case, it may be the easiest way to evaluate web data.
👍
👎
The world of web scraping is built around a quite diverse landscape. It offers solutions for all sorts of data scraping jobs, ranging from small, local desktop applications to Enterprise platforms which can scale your crawlers up to hundreds of requests per second.
On top of that, there's a vast number of scraper libraries which will support you in almost every programming language, should you decide to go the manual route and build your own platform to crawl and scrape the web.
Whichever technology you choose, make sure you test it well before you use it for production and check for edge cases as well. Two important aspects are how well the platform scales (that is, how it deals with a large number of URLs) and how it ensures your requests do not get blocked in the first place.
We briefly touched upon the latter at What to consider when scraping the web?, and that can really be an important bit in your planning, as many sites employ anti-scraper techniques, which can essentially stop your crawler in its tracks - and you wouldn't want that 🛑.
We hope this article provided you with a good, first overview of the different available technologies in web scraping and that it may make it a bit easier to choose among all these different platforms, services, technolgies, and libraries.
Should you have any further questions on how to best go ahead with your scraping project and how ScrapingBee could possibly help you, please don't even hesitate a second to reach out to us. We have specialized in this field and we are happy to help.
Happy scraping