How to Use Every Core on your Machine using NodeJS

Background

You have probably used other languages that have developer-friendly ways to multitask complex jobs. Unfortunately, doing this in JavaScript has always been complicated.

For the longest time, JavaScript and NodeJS were limited by the event loop. Code executes asynchronously, but not in true parallel fashion. However, that changed with the release of worker threads in NodeJS.

After discovering this concept, I immediately want to test its full capability. Unfortunately, the existing libraries are overly complex and/or lack true parallel capabilities.

Goal

I want a package that is perfect for small projects. Something that provides a job queue without relying on databases or the filesystem while proving obvious performance benefits.

Problem

Many packages are half-baked implementation of concurrency. For example, some packages have code that look like this.

for (var i = 0; i<numberOfThreads; i++) {
  promises.push(new Promise(...))
}
Promise.all(promises)

Concurrent but fatally flawed

The above code is incorrect because it leaves out some common edge cases:

• What happens if the pool must terminate?
• What happens if the amount of jobs is fewer than the thread count?
• What if one job takes significantly longer than the other?

#programming #nodejs #javascript #node