Using Node 11.7 Worker Threads with RxJS Observable

Using Node 11.7 Worker Threads with RxJS Observable

The latest iteration of the Node.js runtime environment comes with worker threads. Read on to see how to use these worker threads with RxJS observables.


With the release of Node 11.7, the worker_threads module becomes a standard feature and is no longer hidden behind the --experimental-worker switch. The worker_threads module allows developers to run JavaScript asynchronously in light-weight, isolated threads contained within the main Node process. This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

With the release of Node 11.7, the worker_threads module becomes a standard feature and is no longer hidden behind the --experimental-worker switch. The worker_threads module allows developers to run JavaScript asynchronously in light-weight, isolated threads contained within the main Node process. This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

Before we get started, if you want to learn more about worker threads and why you might want to use them, I would recommend reading Node.js multithreading: What are Worker Threads and why do they matter? by Alberto Gimeno. Alberto has done a fantastic job explaining the purpose of the worker_thread module, provided some solid examples of where it makes sense to use it as well as demonstrated some alternate ways to build a multi-threaded Node app.


What are we building?

We are going to be building a simple Node app that creates a worker thread running a simulated long-running task that reports status back at regular intervals until it completes or until time runs out. The worker thread will be wrapped in an RxJS Observable so that the rest of the application can stream messages returned from the worker thread using the powerful RxJS library.

If you want to jump ahead and see the final solution, you can see it out on GitHub at briandesousa/node-worker-thread-rxjs.


Setting up your environment

The first thing we need to do is ensure our environment is ready to go:

  1. Install Node 11.7.0+
  2. Use npm init to initialize a new NPM package
  3. Add a simple start script to the package.json to start the app: node node-parent-thread-rxjs.js
  4. Install the RxJS package with npm install -s rxjs
  5. Create node-parent-thread-rxjs.js which will contain code running on the main thread
  6. Create node-worker-thread-rxjs.js which will contain the implementation of the long-running task running on a separate thread
Creating the worker thread

The worker thread has the logic to simulate a long-running task:

const { workerData, parentPort } = require('worker_threads');

parentPort.postMessage(starting heavy duty work from process ${process.pid} that will take ${workerData}s to complete);

timeLimit = workerData;
timer = 0;

// simulate a long-running process with updates posted back on a regular interval
do {
setTimeout(
(count) => {
parentPort.postMessage(heavy duty work in progress...${count + 1}s);
if (count === timeLimit) {
parentPort.postMessage('done heavy duty work');
}
},
1000 * timer,
timer);
} while (++timer !== timeLimit);

Let’s break this script down a bit:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
  • before the task begins
  • while the task is running (within the do while loop) to provide status back to the parent thread
  • when the task completes
  • We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

This worker thread doesn’t do anything particularly useful but it does demonstrate how a thread might receive instructions from its parent and stream multiple updates back to its parent.


Creating the parent thread

The parent thread has the following responsibilities:

  • Start the worker thread, specifying how long the worker thread should run for. We will call this WORKER_TIME.
  • Receiving updates from the worker thread in an observable stream
  • Exit the application if the worker thread takes too long. We will call this MAX_WAIT_TIME.
const Rxjs = require('rxjs');
const RxjsOperators = require('rxjs/operators');
const { Worker } = require('worker_threads');

console.log("\nNode multi-threading demo using worker_threads module in Node 11.7.0\n");

const COMPLETE_SIGNAL = 'COMPLETE';

function runTask(workerData, completedOnTime) {
return Rxjs.Observable.create(observer => {
const worker = new Worker('./node-worker-thread-rxjs.js', { workerData });
worker.on('message', message => observer.next(message));
worker.on('error', error => observer.error(error));
worker.on('exit', code => {
if (code !== 0) {
observer.error(Worker stopped with exit code ${code});
} else {
completedOnTime();
observer.next(COMPLETE_SIGNAL);
observer.complete();
}
});
});
}

const MAX_WAIT_TIME = 3;
const WORKER_TIME = 10;

function main() {
completedOnTime = false;

console.log(`[Main] Starting worker from process ${process.pid}`);

const worker$ = runTask(WORKER_TIME, () => completedOnTime = true);

// receive messages from worker until it completes but only wait for MAX_WAIT_TIME
worker$.pipe(
    RxjsOperators.takeWhile(message => message !== COMPLETE_SIGNAL),
    RxjsOperators.takeUntil(Rxjs.timer(MAX_WAIT_TIME * 1000))
).subscribe(
    result => console.log(`[Main] worker says: ${result}`),
    error => console.error(`[Main] worker error: ${error}`),
    () => {
        if (!completedOnTime) {
            console.log(`[Main] worker could not complete its work in the allowed ${MAX_WAIT_TIME}s, exiting Node process`);
            process.exit(0);
        } else {
            console.log(`[Main] worker completed its work in the allowed ${WORKER_TIME}s`);
        }
    }
);

}

main();

There is a lot going on here. Let’s focus on the runTask() function first:

  • We use Observerable.create() from the rxjs package to create a new observable. This observable creates an instance of the worker thread and passes some data in.
  • We map events output from the worker thread to the appropriate functions on the Observer interface:
  • message events are returned as normal values pushed to the subscriber through Observer.next()
  • error events are mapped to Observer.error()
  • when an exit message is received, we check the message to determine why the worker thread exited:
  • if a non-zero value is returned, then we know something went wrong and map the result to Observer.error()
  • if zero is returned, we call a callback function to notify the application that the task was completed on time, we send one final special COMPLETE_SIGNAL value and then we complete the observable with Observer.complete()

The runTask() doesn’t have a very descriptive name however you can now see that it encapsulates the mapping logic between worker thread events and the Observable interface.

Next, let’s look at the at the main() function:

  • We create the observable by calling runTask(). We pass in a simple callback that sets the completedOnTime flag to true so that we can report the reason why the observable completed.
  • Note that the observable we just created is a cold observable. It creates the worker thread and starts emitting events only once it has been subscribed to.
  • We pipe some RxJS operator functions into the observable:
  • takeWhile() to stop the stream when the special COMPLETE_SIGNAL value is received
  • takeUntil() to stop the stream when time has run out
  • We subscribe to the observable and log any values or errors that are received to the console. When the observable completes, we log the reason why it completed. If the reason is because we ran out of time, then we forcefully exit the application with process.exit(0).
Running the solution

Run the solution with your npm start command. Assuming MAX_WAIT_TIME is still set to 3 and WORKER_TIME is set to 10, you will see the following output:

Node multi-threading demo using worker_threads module in Node 11.7.0

[Main] Starting worker from process 4764
[Main] worker says: starting heavy duty work from process 4764 that will take 10s to complete
[Main] worker says: heavy duty work in progress...1s
[Main] worker says: heavy duty work in progress...2s
[Main] worker says: heavy duty work in progress...3s
[Main] worker could not complete its work in the allowed 3s, exiting Node process

The worker thread started to do its work, but after 3 seconds, the app signaled to stop the stream. The main process was forcefully exited along with the worker thread before it had a chance to complete its task.

You can also try adjusting the solution to see what happens when:

  • WORKER_TIME is less than MAX_WAIT_TIME
  • multiple worker threads are spawned from the parent thread by calling runTask() multiple times and creating multiple observables with different settings
Wrap up

We have only just scratched the surface of what is possible when you combine the streaming power and beauty of RxJS Observables with the worker_threads module. Happy threading!

Check out the full solution on GitHub at briandesousa/node-worker-thread-rxjs.


Learn More

The Complete Node.js Developer Course (2nd Edition)

Learn and Understand NodeJS

Build a web scraper with Node

MEAN Stack Tutorial MongoDB, ExpressJS, AngularJS and NodeJS

AngularJS tutorial for beginners with NodeJS, ExpressJS and MongoDB

Node.js With Passport Authentication | Full Project

Building A REST API With MongoDB, Mongoose, And Node.js

Machine Learning In Node.js With TensorFlow.js

Originally published by Brian De Sousa at https://briandesousa.net

Hire Node JS Developer from Expert Node JS Development Company

Hire Node JS Developer from Expert Node JS Development Company

NodeJS Development Company-Hire Node JS developer from the most prominent NodeJS development company, Mobiweb and get remarkable Node.js app development services.

Are you looking to hire the most talented and expert Node JS developers for your valuable web application projects and mobile app development projects or you want to migrate application on Node JS framework? Then you have to hire Node JS developer from leading offshore Node JS development company Mobiweb Technologies. We have a team of developers with extensive experience in developing Node JS based applications whether it is web based or mobile app based.

Main Reasons of Using Node JS for Your Web Application Development

Main Reasons of Using Node JS for Your Web Application Development

You have to hire Node JS developer from prestigious and expert Node JS development company Mobiweb Technologies. They are tech enthusiasts with new and latest programming ideas, web development technologies and industry trends.

Node JS is the best JavaScript for utilizing in real-time applications. If you are stressed of using low level web sockets or protocols then with the incredible speed of Node JS you can easily develop real-time applications. According to the business perspective, Node JS is highly advantageous for any online business or business website, so it is very difficult for companies or business owners to avoid Node JS for their web application projects. For the best results in your Node JS development project you must have to hire Node JS developer from the prestigious web development company- Mobiweb Technologies.

Top Vue.js Developers in USA

Top Vue.js Developers in USA

Vue.js is an extensively popular JavaScript framework with which you can create powerful as well as interactive interfaces. Vue.js is the best framework when it comes to building a single web and mobile apps.

We, at HireFullStackDeveloperIndia, implement the right strategic approach to offer a wide variety through customized Vue.js development services to suit your requirements at most competitive prices.

Vue.js is an open-source JavaScript framework that is incredibly progressive and adoptive and majorly used to build a breathtaking user interface. Vue.js is efficient to create advanced web page applications.

Vue.js gets its strength from the flexible JavaScript library to build an enthralling user interface. As the core of Vue.js is concentrated which provides a variety of interactive components for the web and gives real-time implementation. It gives freedom to developers by giving fluidity and eases the integration process with existing projects and other libraries that enables to structure of a highly customizable application.

Vue.js is a scalable framework with a robust in-build stack that can extend itself to operate apps of any proportion. Moreover, vue.js is the best framework to seamlessly create astonishing single-page applications.

Our Vue.js developers have gained tremendous expertise by delivering services to clients worldwide over multiple industries in the area of front-end development. Our adept developers are experts in Vue development and can provide the best value-added user interfaces and web apps.

We assure our clients to have a prime user interface that reaches end-users and target the audience with the exceptional user experience across a variety of devices and platforms. Our expert team of developers serves your business to move ahead on the path of success, where your enterprise can have an advantage over others.

Here are some key benefits that you can avail when you decide to hire vue.js developers in USA from HireFullStackDeveloperIndia:

  • A team of Vue.js developers of your choice
  • 100% guaranteed client satisfaction
  • Integrity and Transparency
  • Free no-obligation quote
  • Portal development solutions
  • Interactive Dashboards over a wide array of devices
  • Vue.js music and video streaming apps
  • Flexible engagement model
  • A free project manager with your team
  • 24*7 communication with your preferred means

If you are looking to hire React Native developers in USA, then choosing HireFullStackDeveloperIndia would be the best as we offer some of the best talents when it comes to Vue.js.