Morioh
Login
Tewolde Luwam

Tewolde Luwam

4 years ago

A simple TypeScript framework for declaratively composing web crawlers with Puppeteer

Botmation Crew

Botmation is a simple TypeScript framework to build web bots with Puppeteer in a declarative, functional and composable way.

“Everything should be made as simple as possible, but no simpler.” - Albert Einstein

Why choose Botmation?

Baby Bot

It empowers Puppeteer code with a simple pattern that maximizes code readability, reusability and testability.

It has a compositional design with safe defaults for building bots with less code.

It encourages learning at your own pace, to inspire an appreciation for the possibilities of Functional programming.

It has 100% library code test coverage.

Introduction

Botmation is simple functional framework for Puppeteer to build online Bots in a composable, testable, and declarative way. It provides a simple pattern focused on a single type of function called BotAction.

BotAction’s handle almost everything from simple tasks in crawling and scraping the web to logging in & automating your social media. They are composable. They make assembling Bots easy, declarative, and simple.

You can compose new BotAction’s from ones provided or build your own from scratch, then mix them up.

The possibilities are endless!

Getting Started

Botmation is a NodeJS library written in TypeScript. You’ll need node.js LTS installed and the TypeScript compiler (tsc) installed globally (or have a transpiling code step).

Yellow Bot

Install

Install Botmation with npm and save it as a dependency:

npm install -s botmation

If you’re just getting started, install puppeteer latest v2 & @types/puppeteer v3.0.0:

npm install -s puppeteer@">=2.0.0 <3.0.0" @types/puppeteer@3.0.0

While Botmation source code works with Puppeteer version 3, 4, and 5, the E2E testing has a bug that is limiting the packaging, as of this moment.

If you want to use Botmation with the latest Puppeteer version, for now, clone the repo locally and work from the playground bot. From there you can install the latest @types/puppeteer and puppeteer packages then build Botmation source code with it. But, be wary, the E2E tests will become rather unstable.

Documentation

To get started with Botmation, learn about its design and pattern, view API Doc’s, see examples, advanced techniques, and a tutorial on approaching these Bot problems, visit the official Botmation Documentation site.

Library Reference

After intalling through npm, import any BotAction from the main module:

import { chain, goTo, screenshot } from 'botmation'

As of v2.0.x, there are 12 groups of BotAction to compose with:

Leader Bot

  • assembly-line
    • compose and run BotAction’s in lines
  • console
    • log messages to the nodeJS console
  • cookies
    • read/write page cookies
  • errors
    • try/catch errors in assembly-lines
  • files
    • write files to local disk ie screenshots, pdf’s
  • indexed-db
    • read/write to page’s IndexedDB
  • inject
    • insert new injects into a line of BotAction’s
  • input
    • simulate User input ie typing and clicking with a mouse
  • local-storage
    • read/write/delete from a page’s Local Storage
  • navigation
    • change the page’s URL, wait for form submissions to change page URL, back, forward, refresh
  • utilities
    • handle more complex logic patterns ie if statements and for loops

Examples

In the ./src/examples directory of this repo (excluded from the npm module), exists a small collection of simple bots, to help you get going:

Dev Notes

Library Development

First, clone the repo locally, then install the npm dependencies. You can build the library locally with this command:

npm run build

Orange Bot

The playground_bot is a dedicated spot for trying out new Bot Actions, etc. You can run it’s code, after running the build command, with:

npm run playground

Library Testing

All our testing (e2e, unit, and integration) is done with Jest.

Learn more about the library’s testing strategy and coverage with the Botmation: Tests documentation.

Issues & Feature Requests

Open Issues on Github. Please specify if it’s a feature request or a bug.

When reporting bugs, please provide sample code to recreate the bug, relevant error messages/logs, and any other information that may help.

Contributors

Code

Michael Lage - Blog

Art

Patrick Capeto - Email

Download Details:

Author: mrWh1te

Demo: https://www.botmation.dev/

Source Code: https://github.com/mrWh1te/Botmation

#nodejs #node #javascript #typescript #puppeteer

A simple TypeScript framework for declaratively composing web crawlers with Puppeteer
16.50 GEEK