Sofia Kelly

Sofia Kelly

1566545889

Using Node 11.7 Worker Threads With RxJS Observable

This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

Before we get started, if you want to learn more about worker threads and why you might want to use them, I would recommend reading Node.js multithreading: What are Worker Threads and why do they matter? by Alberto Gimeno. Alberto has done a fantastic job explaining the purpose of the worker_thread module, provided some solid examples of where it makes sense to use it as well as demonstrated some alternate ways to build a multi-threaded Node app.

What are we building?

We are going to be building a simple Node app that creates a worker thread running a simulated long-running task that reports status back at regular intervals until it completes or until time runs out. The worker thread will be wrapped in an RxJS Observable so that the rest of the application can stream messages returned from the worker thread using the powerful RxJS library.

If you want to jump ahead and see the final solution, you can see it out on GitHub at briandesousa/node-worker-thread-rxjs.

Setting up your environment

The first thing we need to do is ensure our environment is ready to go:

  1. Install Node 11.7.0+
  2. Use npm init to initialize a new NPM package
  3. Add a simple start script to the package.json to start the app: node node-parent-thread-rxjs.js
  4. Install the RxJS package with npm install -s rxjs
  5. Create node-parent-thread-rxjs.js which will contain code running on the main thread
  6. Create node-worker-thread-rxjs.js which will contain the implementation of the long-running task running on a separate thread

Creating the worker thread

The worker thread has the logic to simulate a long-running task:

const { workerData, parentPort } = require('worker_threads');
	

	parentPort.postMessage(`starting heavy duty work from process ${process.pid} that will take ${workerData}s to complete`);
	

	timeLimit = workerData;
	timer = 0;
	

	// simulate a long-running process with updates posted back on a regular interval
	do {
	    setTimeout(
	        (count) => {
	            parentPort.postMessage(`heavy duty work in progress...${count + 1}s`);
	            if (count === timeLimit) {
	                parentPort.postMessage('done heavy duty work');
	            }
	        },
	        1000 * timer,
	        timer);
	} while (++timer !== timeLimit);

node-worker-thread-rxjs.js

Let’s break this script down a bit:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

This worker thread doesn’t do anything particularly useful but it does demonstrate how a thread might receive instructions from its parent and stream multiple updates back to its parent.

Creating the parent thread

The parent thread has the following responsibilities:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).
const Rxjs = require('rxjs');
	const RxjsOperators = require('rxjs/operators');
	const { Worker } = require('worker_threads');
	

	console.log("\nNode multi-threading demo using worker_threads module in Node 11.7.0\n");
	

	const COMPLETE_SIGNAL = 'COMPLETE';
	

	function runTask(workerData, completedOnTime) {
	    return Rxjs.Observable.create(observer => {
	        const worker = new Worker('./node-worker-thread-rxjs.js', { workerData });
	        worker.on('message', message => observer.next(message));
	        worker.on('error', error => observer.error(error));
	        worker.on('exit', code => {
	            if (code !== 0) {
	                observer.error(`Worker stopped with exit code ${code}`);
	            } else {
	                completedOnTime();
	                observer.next(COMPLETE_SIGNAL);
	                observer.complete();
	            }
	        });
	    });
	}
	

	const MAX_WAIT_TIME = 3;
	const WORKER_TIME = 10;
	

	function main() {
	    completedOnTime = false;
	

	    console.log(`[Main] Starting worker from process ${process.pid}`);
	

	    const worker$ = runTask(WORKER_TIME, () => completedOnTime = true);
	

	    // receive messages from worker until it completes but only wait for MAX_WAIT_TIME
	    worker$.pipe(
	        RxjsOperators.takeWhile(message => message !== COMPLETE_SIGNAL),
	        RxjsOperators.takeUntil(Rxjs.timer(MAX_WAIT_TIME * 1000))
	    ).subscribe(
	        result => console.log(`[Main] worker says: ${result}`),
	        error => console.error(`[Main] worker error: ${error}`),
	        () => {
	            if (!completedOnTime) {
	                console.log(`[Main] worker could not complete its work in the allowed ${MAX_WAIT_TIME}s, exiting Node process`);
	                process.exit(0);
	            } else {
	                console.log(`[Main] worker completed its work in the allowed ${WORKER_TIME}s`);
	            }
	        }
	    );
	}
	

	main();

node-parent-thread-rxjs.js

There is a lot going on here. Let’s focus on the runTask() function first:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

The runTask() doesn’t have a very descriptive name however you can now see that it encapsulates the mapping logic between worker thread events and the Observable interface.

Next, let’s look at the at the main() function:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Running the solution

Run the solution with your npm start command. Assuming MAX_WAIT_TIME is still set to 3 and WORKER_TIME is set to 10, you will see the following output:

Node multi-threading demo using worker_threads module in Node 11.7.0 [Main] Starting worker from process 4764 [Main] worker says: starting heavy duty work from process 4764 that will take 10s to complete [Main] worker says: heavy duty work in progress...1s [Main] worker says: heavy duty work in progress...2s [Main] worker says: heavy duty work in progress...3s [Main] worker could not complete its work in the allowed 3s, exiting Node process

The worker thread started to do its work, but after 3 seconds, the app signaled to stop the stream. The main process was forcefully exited along with the worker thread before it had a chance to complete its task.

You can also try adjusting the solution to see what happens when:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Wrap up

We have only just scratched the surface of what is possible when you combine the streaming power and beauty of RxJS Observables with the worker_threads module. Happy threading!

Check out the full solution on GitHub at briandesousa/node-worker-thread-rxjs.

Further reading:

How to build a command-line chat app using SocketIO

Top 15 Programming Languages by Popularity (2004-2019)

Use MongoDB Node.js Native Driver Without Mongoose

Deploying a Node 12 Function to Cloud Run

How to build a realtime messaging feature in React app with Chatkit

Video Streaming with Node.js

Microservices in Node.js

Building Real-World Microservices with Node.js

This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

Before we get started, if you want to learn more about worker threads and why you might want to use them, I would recommend reading Node.js multithreading: What are Worker Threads and why do they matter? by Alberto Gimeno. Alberto has done a fantastic job explaining the purpose of the worker_thread module, provided some solid examples of where it makes sense to use it as well as demonstrated some alternate ways to build a multi-threaded Node app.

What are we building?

We are going to be building a simple Node app that creates a worker thread running a simulated long-running task that reports status back at regular intervals until it completes or until time runs out. The worker thread will be wrapped in an RxJS Observable so that the rest of the application can stream messages returned from the worker thread using the powerful RxJS library.

If you want to jump ahead and see the final solution, you can see it out on GitHub at briandesousa/node-worker-thread-rxjs.

Setting up your environment

The first thing we need to do is ensure our environment is ready to go:

  1. Install Node 11.7.0+
  2. Use npm init to initialize a new NPM package
  3. Add a simple start script to the package.json to start the app: node node-parent-thread-rxjs.js
  4. Install the RxJS package with npm install -s rxjs
  5. Create node-parent-thread-rxjs.js which will contain code running on the main thread
  6. Create node-worker-thread-rxjs.js which will contain the implementation of the long-running task running on a separate thread

Creating the worker thread

The worker thread has the logic to simulate a long-running task:

const { workerData, parentPort } = require('worker_threads');
	

	parentPort.postMessage(`starting heavy duty work from process ${process.pid} that will take ${workerData}s to complete`);
	

	timeLimit = workerData;
	timer = 0;
	

	// simulate a long-running process with updates posted back on a regular interval
	do {
	    setTimeout(
	        (count) => {
	            parentPort.postMessage(`heavy duty work in progress...${count + 1}s`);
	            if (count === timeLimit) {
	                parentPort.postMessage('done heavy duty work');
	            }
	        },
	        1000 * timer,
	        timer);
	} while (++timer !== timeLimit);

node-worker-thread-rxjs.js

Let’s break this script down a bit:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

This worker thread doesn’t do anything particularly useful but it does demonstrate how a thread might receive instructions from its parent and stream multiple updates back to its parent.

Creating the parent thread

The parent thread has the following responsibilities:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).
const Rxjs = require('rxjs');
	const RxjsOperators = require('rxjs/operators');
	const { Worker } = require('worker_threads');
	

	console.log("\nNode multi-threading demo using worker_threads module in Node 11.7.0\n");
	

	const COMPLETE_SIGNAL = 'COMPLETE';
	

	function runTask(workerData, completedOnTime) {
	    return Rxjs.Observable.create(observer => {
	        const worker = new Worker('./node-worker-thread-rxjs.js', { workerData });
	        worker.on('message', message => observer.next(message));
	        worker.on('error', error => observer.error(error));
	        worker.on('exit', code => {
	            if (code !== 0) {
	                observer.error(`Worker stopped with exit code ${code}`);
	            } else {
	                completedOnTime();
	                observer.next(COMPLETE_SIGNAL);
	                observer.complete();
	            }
	        });
	    });
	}
	

	const MAX_WAIT_TIME = 3;
	const WORKER_TIME = 10;
	

	function main() {
	    completedOnTime = false;
	

	    console.log(`[Main] Starting worker from process ${process.pid}`);
	

	    const worker$ = runTask(WORKER_TIME, () => completedOnTime = true);
	

	    // receive messages from worker until it completes but only wait for MAX_WAIT_TIME
	    worker$.pipe(
	        RxjsOperators.takeWhile(message => message !== COMPLETE_SIGNAL),
	        RxjsOperators.takeUntil(Rxjs.timer(MAX_WAIT_TIME * 1000))
	    ).subscribe(
	        result => console.log(`[Main] worker says: ${result}`),
	        error => console.error(`[Main] worker error: ${error}`),
	        () => {
	            if (!completedOnTime) {
	                console.log(`[Main] worker could not complete its work in the allowed ${MAX_WAIT_TIME}s, exiting Node process`);
	                process.exit(0);
	            } else {
	                console.log(`[Main] worker completed its work in the allowed ${WORKER_TIME}s`);
	            }
	        }
	    );
	}
	

	main();

node-parent-thread-rxjs.js

There is a lot going on here. Let’s focus on the runTask() function first:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

The runTask() doesn’t have a very descriptive name however you can now see that it encapsulates the mapping logic between worker thread events and the Observable interface.

Next, let’s look at the at the main() function:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Running the solution

Run the solution with your npm start command. Assuming MAX_WAIT_TIME is still set to 3 and WORKER_TIME is set to 10, you will see the following output:

Node multi-threading demo using worker_threads module in Node 11.7.0 [Main] Starting worker from process 4764 [Main] worker says: starting heavy duty work from process 4764 that will take 10s to complete [Main] worker says: heavy duty work in progress...1s [Main] worker says: heavy duty work in progress...2s [Main] worker says: heavy duty work in progress...3s [Main] worker could not complete its work in the allowed 3s, exiting Node process

The worker thread started to do its work, but after 3 seconds, the app signaled to stop the stream. The main process was forcefully exited along with the worker thread before it had a chance to complete its task.

You can also try adjusting the solution to see what happens when:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Wrap up

We have only just scratched the surface of what is possible when you combine the streaming power and beauty of RxJS Observables with the worker_threads module. Happy threading!

Check out the full solution on GitHub at briandesousa/node-worker-thread-rxjs.

Further reading:

How to build a command-line chat app using SocketIO

Top 15 Programming Languages by Popularity (2004-2019)

Use MongoDB Node.js Native Driver Without Mongoose

Deploying a Node 12 Function to Cloud Run

How to build a realtime messaging feature in React app with Chatkit

Video Streaming with Node.js

Microservices in Node.js

Building Real-World Microservices with Node.js

#node-js #javascript

What is GEEK

Buddha Community

Using Node 11.7 Worker Threads With RxJS Observable

Pyringe: Debugger Capable Of Attaching to & Injecting Code Into Python

DISCLAIMER: This is not an official google project, this is just something I wrote while at Google.

Pyringe

What this is

Pyringe is a python debugger capable of attaching to running processes, inspecting their state and even of injecting python code into them while they're running. With pyringe, you can list threads, get tracebacks, inspect locals/globals/builtins of running functions, all without having to prepare your program for it.

What this is not

A "Google project". It's my internship project that got open-sourced. Sorry for the confusion.

What do I need?

Pyringe internally uses gdb to do a lot of its heavy lifting, so you will need a fairly recent build of gdb (version 7.4 onwards, and only if gdb was configured with --with-python). You will also need the symbols for whatever build of python you're running.
On Fedora, the package you're looking for is python-debuginfo, on Debian it's called python2.7-dbg (adjust according to version). Arch Linux users: see issue #5, Ubuntu users can only debug the python-dbg binary (see issue #19).
Having Colorama will get you output in boldface, but it's optional.

How do I get it?

Get it from the Github repo, PyPI, or via pip (pip install pyringe).

Is this Python3-friendly?

Short answer: No, sorry. Long answer:
There's three potentially different versions of python in play here:

  1. The version running pyringe
  2. The version being debugged
  3. The version of libpythonXX.so your build of gdb was linked against

2 Is currently the dealbreaker here. Cpython has changed a bit in the meantime[1], and making all features work while debugging python3 will have to take a back seat for now until the more glaring issues have been taken care of.
As for 1 and 3, the 2to3 tool may be able to handle it automatically. But then, as long as 2 hasn't been taken care of, this isn't really a use case in the first place.

[1] - For example, pendingbusy (which is used for injection) has been renamed to busy and been given a function-local scope, making it harder to interact with via gdb.

Will this work with PyPy?

Unfortunately, no. Since this makes use of some CPython internals and implementation details, only CPython is supported. If you don't know what PyPy or CPython are, you'll probably be fine.

Why not PDB?

PDB is great. Use it where applicable! But sometimes it isn't.
Like when python itself crashes, gets stuck in some C extension, or you want to inspect data without stopping a program. In such cases, PDB (and all other debuggers that run within the interpreter itself) are next to useless, and without pyringe you'd be left with having to debug using print statements. Pyringe is just quite convenient in these cases.

I injected a change to a local var into a function and it's not showing up!

This is a known limitation. Things like inject('var = 2') won't work, but inject('var[1] = 1337') should. This is because most of the time, python internally uses a fast path for looking up local variables that doesn't actually perform the dictionary lookup in locals(). In general, code you inject into processes with pyringe is very different from a normal python function call.

How do I use it?

You can start the debugger by executing python -m pyringe. Alternatively:

import pyringe
pyringe.interact()

If that reminds you of the code module, good; this is intentional.
After starting the debugger, you'll be greeted by what behaves almost like a regular python REPL.
Try the following:

==> pid:[None] #threads:[0] current thread:[None]
>>> help()
Available commands:
 attach: Attach to the process with the given pid.
 bt: Get a backtrace of the current position.
 [...]
==> pid:[None] #threads:[0] current thread:[None]
>>> attach(12679)
==> pid:[12679] #threads:[11] current thread:[140108099462912]
>>> threads()
[140108099462912, 140108107855616, 140108116248323, 140108124641024, 140108133033728, 140108224739072, 140108233131776, 140108141426432, 140108241524480, 140108249917184, 140108269324032]

The IDs you see here correspond to what threading.current_thread().ident would tell you.
All debugger functions are just regular python functions that have been exposed to the REPL, so you can do things like the following.

==> pid:[12679] #threads:[11] current thread:[140108099462912]
>>> for tid in threads():
...   if not tid % 10:
...     thread(tid)
...     bt()
... 
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 524, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "./test.py", line 46, in Idle
    Thread_2_Func(1)
  File "./test.py", line 40, in Wait
    time.sleep(n)
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> 

You can access the inferior's locals and inspect them like so:

==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> inflocals()
{'a': <proxy of A object at remote 0x1d9b290>, 'LOL': 'success!', 'b': <proxy of B object at remote 0x1d988c0>, 'n': 1}
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> p('a')
<proxy of A object at remote 0x1d9b290>
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> p('a').attr
'Some_magic_string'
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> 

And sure enough, the definition of a's class reads:

class Example(object):
  cl_attr = False
  def __init__(self):
    self.attr = 'Some_magic_string'

There's limits to how far this proxying of objects goes, and everything that isn't trivial data will show up as strings (like '<function at remote 0x1d957d0>').
You can inject python code into running programs. Of course, there are caveats but... see for yourself:

==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> inject('import threading')
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> inject('print threading.current_thread().ident')
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> 

The output of my program in this case reads:

140108241524480

If you need additional pointers, just try using python's help (pyhelp() in the debugger) on debugger commands.

Author: google
Source Code: https://github.com/google/pyringe
License: Apache-2.0 License

#python 

Zara  Bryant

Zara Bryant

1560830692

Using Node 11.7 Worker Threads with RxJS Observable


With the release of Node 11.7, the worker_threads module becomes a standard feature and is no longer hidden behind the --experimental-worker switch. The worker_threads module allows developers to run JavaScript asynchronously in light-weight, isolated threads contained within the main Node process. This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

With the release of Node 11.7, the worker_threads module becomes a standard feature and is no longer hidden behind the --experimental-worker switch. The worker_threads module allows developers to run JavaScript asynchronously in light-weight, isolated threads contained within the main Node process. This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

Before we get started, if you want to learn more about worker threads and why you might want to use them, I would recommend reading Node.js multithreading: What are Worker Threads and why do they matter? by Alberto Gimeno. Alberto has done a fantastic job explaining the purpose of the worker_thread module, provided some solid examples of where it makes sense to use it as well as demonstrated some alternate ways to build a multi-threaded Node app.


What are we building?

We are going to be building a simple Node app that creates a worker thread running a simulated long-running task that reports status back at regular intervals until it completes or until time runs out. The worker thread will be wrapped in an RxJS Observable so that the rest of the application can stream messages returned from the worker thread using the powerful RxJS library.

If you want to jump ahead and see the final solution, you can see it out on GitHub at briandesousa/node-worker-thread-rxjs.


Setting up your environment

The first thing we need to do is ensure our environment is ready to go:

  1. Install Node 11.7.0+
  2. Use npm init to initialize a new NPM package
  3. Add a simple start script to the package.json to start the app: node node-parent-thread-rxjs.js
  4. Install the RxJS package with npm install -s rxjs
  5. Create node-parent-thread-rxjs.js which will contain code running on the main thread
  6. Create node-worker-thread-rxjs.js which will contain the implementation of the long-running task running on a separate thread

Creating the worker thread

The worker thread has the logic to simulate a long-running task:

const { workerData, parentPort } = require('worker_threads');

parentPort.postMessage(starting heavy duty work from process ${process.pid} that will take ${workerData}s to complete);

timeLimit = workerData;
timer = 0;

// simulate a long-running process with updates posted back on a regular interval
do {
setTimeout(
(count) => {
parentPort.postMessage(heavy duty work in progress...${count + 1}s);
if (count === timeLimit) {
parentPort.postMessage(‘done heavy duty work’);
}
},
1000 * timer,
timer);
} while (++timer !== timeLimit);

Let’s break this script down a bit:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
  • before the task begins
  • while the task is running (within the do while loop) to provide status back to the parent thread
  • when the task completes
  • We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

This worker thread doesn’t do anything particularly useful but it does demonstrate how a thread might receive instructions from its parent and stream multiple updates back to its parent.


Creating the parent thread

The parent thread has the following responsibilities:

  • Start the worker thread, specifying how long the worker thread should run for. We will call this WORKER_TIME.
  • Receiving updates from the worker thread in an observable stream
  • Exit the application if the worker thread takes too long. We will call this MAX_WAIT_TIME.
const Rxjs = require(‘rxjs’);
const RxjsOperators = require(‘rxjs/operators’);
const { Worker } = require(‘worker_threads’);

console.log(“\nNode multi-threading demo using worker_threads module in Node 11.7.0\n”);

const COMPLETE_SIGNAL = ‘COMPLETE’;

function runTask(workerData, completedOnTime) {
return Rxjs.Observable.create(observer => {
const worker = new Worker(‘./node-worker-thread-rxjs.js’, { workerData });
worker.on(‘message’, message => observer.next(message));
worker.on(‘error’, error => observer.error(error));
worker.on(‘exit’, code => {
if (code !== 0) {
observer.error(Worker stopped with exit code ${code});
} else {
completedOnTime();
observer.next(COMPLETE_SIGNAL);
observer.complete();
}
});
});
}

const MAX_WAIT_TIME = 3;
const WORKER_TIME = 10;

function main() {
completedOnTime = false;

console.log(`[Main] Starting worker from process ${process.pid}`);

const worker$ = runTask(WORKER_TIME, () =&gt; completedOnTime = true);

// receive messages from worker until it completes but only wait for MAX_WAIT_TIME
worker$.pipe(
    RxjsOperators.takeWhile(message =&gt; message !== COMPLETE_SIGNAL),
    RxjsOperators.takeUntil(Rxjs.timer(MAX_WAIT_TIME * 1000))
).subscribe(
    result =&gt; console.log(`[Main] worker says: ${result}`),
    error =&gt; console.error(`[Main] worker error: ${error}`),
    () =&gt; {
        if (!completedOnTime) {
            console.log(`[Main] worker could not complete its work in the allowed ${MAX_WAIT_TIME}s, exiting Node process`);
            process.exit(0);
        } else {
            console.log(`[Main] worker completed its work in the allowed ${WORKER_TIME}s`);
        }
    }
);

}

main();

There is a lot going on here. Let’s focus on the runTask() function first:

  • We use Observerable.create() from the rxjs package to create a new observable. This observable creates an instance of the worker thread and passes some data in.
  • We map events output from the worker thread to the appropriate functions on the Observer interface:
  • message events are returned as normal values pushed to the subscriber through Observer.next()
  • error events are mapped to Observer.error()
  • when an exit message is received, we check the message to determine why the worker thread exited:
  • if a non-zero value is returned, then we know something went wrong and map the result to Observer.error()
  • if zero is returned, we call a callback function to notify the application that the task was completed on time, we send one final special COMPLETE_SIGNAL value and then we complete the observable with Observer.complete()

The runTask() doesn’t have a very descriptive name however you can now see that it encapsulates the mapping logic between worker thread events and the Observable interface.

Next, let’s look at the at the main() function:

  • We create the observable by calling runTask(). We pass in a simple callback that sets the completedOnTime flag to true so that we can report the reason why the observable completed.
  • Note that the observable we just created is a cold observable. It creates the worker thread and starts emitting events only once it has been subscribed to.
  • We pipe some RxJS operator functions into the observable:
  • takeWhile() to stop the stream when the special COMPLETE_SIGNAL value is received
  • takeUntil() to stop the stream when time has run out
  • We subscribe to the observable and log any values or errors that are received to the console. When the observable completes, we log the reason why it completed. If the reason is because we ran out of time, then we forcefully exit the application with process.exit(0).

Running the solution

Run the solution with your npm start command. Assuming MAX_WAIT_TIME is still set to 3 and WORKER_TIME is set to 10, you will see the following output:

Node multi-threading demo using worker_threads module in Node 11.7.0

[Main] Starting worker from process 4764
[Main] worker says: starting heavy duty work from process 4764 that will take 10s to complete
[Main] worker says: heavy duty work in progress…1s
[Main] worker says: heavy duty work in progress…2s
[Main] worker says: heavy duty work in progress…3s
[Main] worker could not complete its work in the allowed 3s, exiting Node process

The worker thread started to do its work, but after 3 seconds, the app signaled to stop the stream. The main process was forcefully exited along with the worker thread before it had a chance to complete its task.

You can also try adjusting the solution to see what happens when:

  • WORKER_TIME is less than MAX_WAIT_TIME
  • multiple worker threads are spawned from the parent thread by calling runTask() multiple times and creating multiple observables with different settings

Wrap up

We have only just scratched the surface of what is possible when you combine the streaming power and beauty of RxJS Observables with the worker_threads module. Happy threading!

Check out the full solution on GitHub at briandesousa/node-worker-thread-rxjs.


Learn More

The Complete Node.js Developer Course (2nd Edition)

Learn and Understand NodeJS

Build a web scraper with Node

MEAN Stack Tutorial MongoDB, ExpressJS, AngularJS and NodeJS

AngularJS tutorial for beginners with NodeJS, ExpressJS and MongoDB

Node.js With Passport Authentication | Full Project

Building A REST API With MongoDB, Mongoose, And Node.js

Machine Learning In Node.js With TensorFlow.js

Originally published by Brian De Sousa at https://briandesousa.net

#node-js #web-development

Sofia Kelly

Sofia Kelly

1566545889

Using Node 11.7 Worker Threads With RxJS Observable

This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

Before we get started, if you want to learn more about worker threads and why you might want to use them, I would recommend reading Node.js multithreading: What are Worker Threads and why do they matter? by Alberto Gimeno. Alberto has done a fantastic job explaining the purpose of the worker_thread module, provided some solid examples of where it makes sense to use it as well as demonstrated some alternate ways to build a multi-threaded Node app.

What are we building?

We are going to be building a simple Node app that creates a worker thread running a simulated long-running task that reports status back at regular intervals until it completes or until time runs out. The worker thread will be wrapped in an RxJS Observable so that the rest of the application can stream messages returned from the worker thread using the powerful RxJS library.

If you want to jump ahead and see the final solution, you can see it out on GitHub at briandesousa/node-worker-thread-rxjs.

Setting up your environment

The first thing we need to do is ensure our environment is ready to go:

  1. Install Node 11.7.0+
  2. Use npm init to initialize a new NPM package
  3. Add a simple start script to the package.json to start the app: node node-parent-thread-rxjs.js
  4. Install the RxJS package with npm install -s rxjs
  5. Create node-parent-thread-rxjs.js which will contain code running on the main thread
  6. Create node-worker-thread-rxjs.js which will contain the implementation of the long-running task running on a separate thread

Creating the worker thread

The worker thread has the logic to simulate a long-running task:

const { workerData, parentPort } = require('worker_threads');
	

	parentPort.postMessage(`starting heavy duty work from process ${process.pid} that will take ${workerData}s to complete`);
	

	timeLimit = workerData;
	timer = 0;
	

	// simulate a long-running process with updates posted back on a regular interval
	do {
	    setTimeout(
	        (count) => {
	            parentPort.postMessage(`heavy duty work in progress...${count + 1}s`);
	            if (count === timeLimit) {
	                parentPort.postMessage('done heavy duty work');
	            }
	        },
	        1000 * timer,
	        timer);
	} while (++timer !== timeLimit);

node-worker-thread-rxjs.js

Let’s break this script down a bit:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

This worker thread doesn’t do anything particularly useful but it does demonstrate how a thread might receive instructions from its parent and stream multiple updates back to its parent.

Creating the parent thread

The parent thread has the following responsibilities:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).
const Rxjs = require('rxjs');
	const RxjsOperators = require('rxjs/operators');
	const { Worker } = require('worker_threads');
	

	console.log("\nNode multi-threading demo using worker_threads module in Node 11.7.0\n");
	

	const COMPLETE_SIGNAL = 'COMPLETE';
	

	function runTask(workerData, completedOnTime) {
	    return Rxjs.Observable.create(observer => {
	        const worker = new Worker('./node-worker-thread-rxjs.js', { workerData });
	        worker.on('message', message => observer.next(message));
	        worker.on('error', error => observer.error(error));
	        worker.on('exit', code => {
	            if (code !== 0) {
	                observer.error(`Worker stopped with exit code ${code}`);
	            } else {
	                completedOnTime();
	                observer.next(COMPLETE_SIGNAL);
	                observer.complete();
	            }
	        });
	    });
	}
	

	const MAX_WAIT_TIME = 3;
	const WORKER_TIME = 10;
	

	function main() {
	    completedOnTime = false;
	

	    console.log(`[Main] Starting worker from process ${process.pid}`);
	

	    const worker$ = runTask(WORKER_TIME, () => completedOnTime = true);
	

	    // receive messages from worker until it completes but only wait for MAX_WAIT_TIME
	    worker$.pipe(
	        RxjsOperators.takeWhile(message => message !== COMPLETE_SIGNAL),
	        RxjsOperators.takeUntil(Rxjs.timer(MAX_WAIT_TIME * 1000))
	    ).subscribe(
	        result => console.log(`[Main] worker says: ${result}`),
	        error => console.error(`[Main] worker error: ${error}`),
	        () => {
	            if (!completedOnTime) {
	                console.log(`[Main] worker could not complete its work in the allowed ${MAX_WAIT_TIME}s, exiting Node process`);
	                process.exit(0);
	            } else {
	                console.log(`[Main] worker completed its work in the allowed ${WORKER_TIME}s`);
	            }
	        }
	    );
	}
	

	main();

node-parent-thread-rxjs.js

There is a lot going on here. Let’s focus on the runTask() function first:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

The runTask() doesn’t have a very descriptive name however you can now see that it encapsulates the mapping logic between worker thread events and the Observable interface.

Next, let’s look at the at the main() function:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Running the solution

Run the solution with your npm start command. Assuming MAX_WAIT_TIME is still set to 3 and WORKER_TIME is set to 10, you will see the following output:

Node multi-threading demo using worker_threads module in Node 11.7.0 [Main] Starting worker from process 4764 [Main] worker says: starting heavy duty work from process 4764 that will take 10s to complete [Main] worker says: heavy duty work in progress...1s [Main] worker says: heavy duty work in progress...2s [Main] worker says: heavy duty work in progress...3s [Main] worker could not complete its work in the allowed 3s, exiting Node process

The worker thread started to do its work, but after 3 seconds, the app signaled to stop the stream. The main process was forcefully exited along with the worker thread before it had a chance to complete its task.

You can also try adjusting the solution to see what happens when:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Wrap up

We have only just scratched the surface of what is possible when you combine the streaming power and beauty of RxJS Observables with the worker_threads module. Happy threading!

Check out the full solution on GitHub at briandesousa/node-worker-thread-rxjs.

Further reading:

How to build a command-line chat app using SocketIO

Top 15 Programming Languages by Popularity (2004-2019)

Use MongoDB Node.js Native Driver Without Mongoose

Deploying a Node 12 Function to Cloud Run

How to build a realtime messaging feature in React app with Chatkit

Video Streaming with Node.js

Microservices in Node.js

Building Real-World Microservices with Node.js

This article will be focusing on how use worker threads to execute a task asynchronously and stream data from that task back to the rest of your Node application using RxJS Observables.

Before we get started, if you want to learn more about worker threads and why you might want to use them, I would recommend reading Node.js multithreading: What are Worker Threads and why do they matter? by Alberto Gimeno. Alberto has done a fantastic job explaining the purpose of the worker_thread module, provided some solid examples of where it makes sense to use it as well as demonstrated some alternate ways to build a multi-threaded Node app.

What are we building?

We are going to be building a simple Node app that creates a worker thread running a simulated long-running task that reports status back at regular intervals until it completes or until time runs out. The worker thread will be wrapped in an RxJS Observable so that the rest of the application can stream messages returned from the worker thread using the powerful RxJS library.

If you want to jump ahead and see the final solution, you can see it out on GitHub at briandesousa/node-worker-thread-rxjs.

Setting up your environment

The first thing we need to do is ensure our environment is ready to go:

  1. Install Node 11.7.0+
  2. Use npm init to initialize a new NPM package
  3. Add a simple start script to the package.json to start the app: node node-parent-thread-rxjs.js
  4. Install the RxJS package with npm install -s rxjs
  5. Create node-parent-thread-rxjs.js which will contain code running on the main thread
  6. Create node-worker-thread-rxjs.js which will contain the implementation of the long-running task running on a separate thread

Creating the worker thread

The worker thread has the logic to simulate a long-running task:

const { workerData, parentPort } = require('worker_threads');
	

	parentPort.postMessage(`starting heavy duty work from process ${process.pid} that will take ${workerData}s to complete`);
	

	timeLimit = workerData;
	timer = 0;
	

	// simulate a long-running process with updates posted back on a regular interval
	do {
	    setTimeout(
	        (count) => {
	            parentPort.postMessage(`heavy duty work in progress...${count + 1}s`);
	            if (count === timeLimit) {
	                parentPort.postMessage('done heavy duty work');
	            }
	        },
	        1000 * timer,
	        timer);
	} while (++timer !== timeLimit);

node-worker-thread-rxjs.js

Let’s break this script down a bit:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

This worker thread doesn’t do anything particularly useful but it does demonstrate how a thread might receive instructions from its parent and stream multiple updates back to its parent.

Creating the parent thread

The parent thread has the following responsibilities:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).
const Rxjs = require('rxjs');
	const RxjsOperators = require('rxjs/operators');
	const { Worker } = require('worker_threads');
	

	console.log("\nNode multi-threading demo using worker_threads module in Node 11.7.0\n");
	

	const COMPLETE_SIGNAL = 'COMPLETE';
	

	function runTask(workerData, completedOnTime) {
	    return Rxjs.Observable.create(observer => {
	        const worker = new Worker('./node-worker-thread-rxjs.js', { workerData });
	        worker.on('message', message => observer.next(message));
	        worker.on('error', error => observer.error(error));
	        worker.on('exit', code => {
	            if (code !== 0) {
	                observer.error(`Worker stopped with exit code ${code}`);
	            } else {
	                completedOnTime();
	                observer.next(COMPLETE_SIGNAL);
	                observer.complete();
	            }
	        });
	    });
	}
	

	const MAX_WAIT_TIME = 3;
	const WORKER_TIME = 10;
	

	function main() {
	    completedOnTime = false;
	

	    console.log(`[Main] Starting worker from process ${process.pid}`);
	

	    const worker$ = runTask(WORKER_TIME, () => completedOnTime = true);
	

	    // receive messages from worker until it completes but only wait for MAX_WAIT_TIME
	    worker$.pipe(
	        RxjsOperators.takeWhile(message => message !== COMPLETE_SIGNAL),
	        RxjsOperators.takeUntil(Rxjs.timer(MAX_WAIT_TIME * 1000))
	    ).subscribe(
	        result => console.log(`[Main] worker says: ${result}`),
	        error => console.error(`[Main] worker error: ${error}`),
	        () => {
	            if (!completedOnTime) {
	                console.log(`[Main] worker could not complete its work in the allowed ${MAX_WAIT_TIME}s, exiting Node process`);
	                process.exit(0);
	            } else {
	                console.log(`[Main] worker completed its work in the allowed ${WORKER_TIME}s`);
	            }
	        }
	    );
	}
	

	main();

node-parent-thread-rxjs.js

There is a lot going on here. Let’s focus on the runTask() function first:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

The runTask() doesn’t have a very descriptive name however you can now see that it encapsulates the mapping logic between worker thread events and the Observable interface.

Next, let’s look at the at the main() function:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Running the solution

Run the solution with your npm start command. Assuming MAX_WAIT_TIME is still set to 3 and WORKER_TIME is set to 10, you will see the following output:

Node multi-threading demo using worker_threads module in Node 11.7.0 [Main] Starting worker from process 4764 [Main] worker says: starting heavy duty work from process 4764 that will take 10s to complete [Main] worker says: heavy duty work in progress...1s [Main] worker says: heavy duty work in progress...2s [Main] worker says: heavy duty work in progress...3s [Main] worker could not complete its work in the allowed 3s, exiting Node process

The worker thread started to do its work, but after 3 seconds, the app signaled to stop the stream. The main process was forcefully exited along with the worker thread before it had a chance to complete its task.

You can also try adjusting the solution to see what happens when:

  • We use parentPort from the worker_threads modules to communicate back to the parent thread at 3 different points:
    before the task beginswhile the task is running (within the do while loop) to provide status back to the parent threadwhen the task completes* We use workerData from the worker_threads module to pass in a time limit for how long (in seconds) the task should run for. The task completes when this time limit is reached (line 19).

Wrap up

We have only just scratched the surface of what is possible when you combine the streaming power and beauty of RxJS Observables with the worker_threads module. Happy threading!

Check out the full solution on GitHub at briandesousa/node-worker-thread-rxjs.

Further reading:

How to build a command-line chat app using SocketIO

Top 15 Programming Languages by Popularity (2004-2019)

Use MongoDB Node.js Native Driver Without Mongoose

Deploying a Node 12 Function to Cloud Run

How to build a realtime messaging feature in React app with Chatkit

Video Streaming with Node.js

Microservices in Node.js

Building Real-World Microservices with Node.js

#node-js #javascript

How to Send E-mail Using Queue in Laravel 7/8

Today I will show you How to Send E-mail Using Queue in Laravel 7/8, many time we can see some process take more time to load like payment gateway, email send, etc. Whenever you are sending email for verification then it load time to send mail because it is services. If you don’t want to wait to user for send email or other process on loading server side process then you can use queue.

Read More : How to Send E-mail Using Queue in Laravel 7/8

https://websolutionstuff.com/post/how-to-send-e-mail-using-queue-in-laravel-7-8


Read Also : Send Mail Example In Laravel 8

https://websolutionstuff.com/post/send-mail-example-in-laravel-8

#how to send e-mail using queue in laravel 7/8 #email #laravel #send mail using queue in laravel 7 #laravel 7/8 send mail using queue #laravel 7/8 mail queue example

Aria Barnes

Aria Barnes

1622719015

Why use Node.js for Web Development? Benefits and Examples of Apps

Front-end web development has been overwhelmed by JavaScript highlights for quite a long time. Google, Facebook, Wikipedia, and most of all online pages use JS for customer side activities. As of late, it additionally made a shift to cross-platform mobile development as a main technology in React Native, Nativescript, Apache Cordova, and other crossover devices. 

Throughout the most recent couple of years, Node.js moved to backend development as well. Designers need to utilize a similar tech stack for the whole web project without learning another language for server-side development. Node.js is a device that adjusts JS usefulness and syntax to the backend. 

What is Node.js? 

Node.js isn’t a language, or library, or system. It’s a runtime situation: commonly JavaScript needs a program to work, however Node.js makes appropriate settings for JS to run outside of the program. It’s based on a JavaScript V8 motor that can run in Chrome, different programs, or independently. 

The extent of V8 is to change JS program situated code into machine code — so JS turns into a broadly useful language and can be perceived by servers. This is one of the advantages of utilizing Node.js in web application development: it expands the usefulness of JavaScript, permitting designers to coordinate the language with APIs, different languages, and outside libraries.

What Are the Advantages of Node.js Web Application Development? 

Of late, organizations have been effectively changing from their backend tech stacks to Node.js. LinkedIn picked Node.js over Ruby on Rails since it took care of expanding responsibility better and decreased the quantity of servers by multiple times. PayPal and Netflix did something comparative, just they had a goal to change their design to microservices. We should investigate the motivations to pick Node.JS for web application development and when we are planning to hire node js developers. 

Amazing Tech Stack for Web Development 

The principal thing that makes Node.js a go-to environment for web development is its JavaScript legacy. It’s the most well known language right now with a great many free devices and a functioning local area. Node.js, because of its association with JS, immediately rose in ubiquity — presently it has in excess of 368 million downloads and a great many free tools in the bundle module. 

Alongside prevalence, Node.js additionally acquired the fundamental JS benefits: 

  • quick execution and information preparing; 
  • exceptionally reusable code; 
  • the code is not difficult to learn, compose, read, and keep up; 
  • tremendous asset library, a huge number of free aides, and a functioning local area. 

In addition, it’s a piece of a well known MEAN tech stack (the blend of MongoDB, Express.js, Angular, and Node.js — four tools that handle all vital parts of web application development). 

Designers Can Utilize JavaScript for the Whole Undertaking 

This is perhaps the most clear advantage of Node.js web application development. JavaScript is an unquestionable requirement for web development. Regardless of whether you construct a multi-page or single-page application, you need to know JS well. On the off chance that you are now OK with JavaScript, learning Node.js won’t be an issue. Grammar, fundamental usefulness, primary standards — every one of these things are comparable. 

In the event that you have JS designers in your group, it will be simpler for them to learn JS-based Node than a totally new dialect. What’s more, the front-end and back-end codebase will be basically the same, simple to peruse, and keep up — in light of the fact that they are both JS-based. 

A Quick Environment for Microservice Development 

There’s another motivation behind why Node.js got famous so rapidly. The environment suits well the idea of microservice development (spilling stone monument usefulness into handfuls or many more modest administrations). 

Microservices need to speak with one another rapidly — and Node.js is probably the quickest device in information handling. Among the fundamental Node.js benefits for programming development are its non-obstructing algorithms.

Node.js measures a few demands all at once without trusting that the first will be concluded. Many microservices can send messages to one another, and they will be gotten and addressed all the while. 

Versatile Web Application Development 

Node.js was worked in view of adaptability — its name really says it. The environment permits numerous hubs to run all the while and speak with one another. Here’s the reason Node.js adaptability is better than other web backend development arrangements. 

Node.js has a module that is liable for load adjusting for each running CPU center. This is one of numerous Node.js module benefits: you can run various hubs all at once, and the environment will naturally adjust the responsibility. 

Node.js permits even apportioning: you can part your application into various situations. You show various forms of the application to different clients, in light of their age, interests, area, language, and so on. This builds personalization and diminishes responsibility. Hub accomplishes this with kid measures — tasks that rapidly speak with one another and share a similar root. 

What’s more, Node’s non-hindering solicitation handling framework adds to fast, letting applications measure a great many solicitations. 

Control Stream Highlights

Numerous designers consider nonconcurrent to be one of the two impediments and benefits of Node.js web application development. In Node, at whatever point the capacity is executed, the code consequently sends a callback. As the quantity of capacities develops, so does the number of callbacks — and you end up in a circumstance known as the callback damnation. 

In any case, Node.js offers an exit plan. You can utilize systems that will plan capacities and sort through callbacks. Systems will associate comparable capacities consequently — so you can track down an essential component via search or in an envelope. At that point, there’s no compelling reason to look through callbacks.

 

Final Words

So, these are some of the top benefits of Nodejs in web application development. This is how Nodejs is contributing a lot to the field of web application development. 

I hope now you are totally aware of the whole process of how Nodejs is really important for your web project. If you are looking to hire a node js development company in India then I would suggest that you take a little consultancy too whenever you call. 

Good Luck!

Original Source

#node.js development company in india #node js development company #hire node js developers #hire node.js developers in india #node.js development services #node.js development