Vessel is a fast, open source, high-level, web crawling and scraping framework for Ruby, built on top of Ferrum — a minimal-dependency pure-Ruby driver for running headless Google Chrome instances.

Why would you need a web crawler? Perhaps you’re building a search engine for an intranet or a group of public-facing websites, or just need to mirror a website with finer-grained control than tools such as wget offer.

Crawl, walk, run

The best way to demonstrate Vessel’s capabilities is with an example. Don’t worry, just because of what Vessel’s capable of, it doesn’t mean that it’s hard to use.

To get started, add Vessel to your Gemfile:

gem "vessel"

#web-crawling #web-crawlers #web-crawler-tutorial #web-crawling-services #ruby #ruby-on-rails #software-development #framework