Morioh
Login
Karim Aya

Karim Aya

4 years ago

Understanding Asynchronous Programming in Node.js

While Asynchronous programming comes with various features like faster execution of programs, it comes with a cost too i.e. usually it is a little bit difficult to program when compare to Synchronous programming.

This tutorial is about explaining each of the Asynchronous scenarios which you may face while coding. We will also learn about how to efficiently avoid callback hell situation.

Getting started

Consider following code.

var fs = require("fs");
var fileContent = fs.readFileSync('sync.js','utf8');
console.log(fileContent);
console.log("something else");

//Output
//Same code content
//something else

In above code, file has been read first and then console.log() executes.

Now consider following code.

var fs = require("fs");
fs.readFile('async.js','utf8',function(err,data){
    if(!err) {
       console.log(data);
    }
});
console.log("something else");

Here output will be as follows :

something else
var fs = require("fs");
fs.readFile('sync.js','utf8',function(err,data){
    if(!err) {
       console.log(data);
    }
});
console.log("something else");

Here we have got console.log() content first and then file content. This is because code is Asynchronous and event loop executes that later.

This also covers one of the interview question where interviewer asks to write program to print same code.

What is callback hell

Familiar with something like this.

var fs = require("fs");
var db = require('somedbfile.js');
var sendEmail = require('someEmail.js');
fs.readFile('async.js','utf8',function(err,data){
    if(!err) {
       console.log(data);
    }
    db.executeQuery('SELECT * FROM test',function(err,rows) {
      if(!err) {
        console.log("Error",err);
      }
      sendEmail(rows,function(err,data) {
        if(!err) {
          console.log("Error",err);
        }
        console.log("Operation done, i am in callback hell");
      });
    });
});

Note : Above code is just an example not a working code.

This happens due to the Asynchronous nature of the JavaScript. We want to execute tasks which are dependent on each other hence we wrap them into the callbacks of each function and hence caught into callback hell situation.

How to avoid callback hell

To avoid callback hell, follow one or combination of the following :

  • Modularise your code.
  • Use generators.
  • Use promises
  • Use event-driven programming.
  • Use Async.js

Modularise your code

Consider following code.

var fs = require("fs");
fs.readFile('async.js','utf8',function(err,data){
    if(!err) {
       console.log(data);
    }
});

The callback function is a closure and can only be accessed inside the function. However you can create separate function by providing some name and pass that function as callback.

Like this.

var fs = require("fs");

fs.readFile('async.js','utf8',fileContent);

function fileContent(err,data) {
  if(!err) {
     console.log(data);
  }
}

The only drawback is you need to create lot of function as code grows.

Use generators

In simple words Generators provides you the ability to convert asynchronous code to synchronous one. Lets learn using example,consider following.

var fs = require("fs");
fs.readFile('async.js','utf8',function(err,data){
    if(!err) {
       console.log(data);
    }
});
console.log("something else");

This is asynchronous code so console.log() will execute prior to readFile().

In order to avoid putting our console.log() inside the callback closure we can use generators to convert the asynchronous nature of readFile() into synchronous one.

Consider following code.

"use strict";
var fs = require('fs');
var co = require('co');

co(function* () {
  var file1 = yield readFile('demo.js');
  console.log(file1);
  console.log("I am after file read even though its Async");
});

function readFile(filename) {
  return function(callback) {
    fs.readFile(filename, 'utf8', callback);
  };
}

Try to execute this code, you will see the content of file first and then console.log().

I will cover generators in more detail in upcoming tutorials.

Use promises

Promise represents the result of asynchronous function. Promises can be used to avoid chaining of callbacks. In Node.js, you can use Q or Bluebird modules to avail the feature of promise.

If we want to convert file read code callback into promises here is how we can do that.

Make sure you have bluebird ( npm install bluebird ) installed.

var Promise = require('bluebird');
 // Converts all function of 'fs' into promises.
var fs = Promise.promisifyAll(require('fs'));

fs.readFileAsync('file.js','utf8')
// 'then' when result comes.
.then(function(data) {
  console.log(data);
})
//'catch' when error comes.
.catch(function(err) {
  console.log(err);
});

Checkout bluebird API reference for more information.

Use event-driven programming

Node.js provides EventEmitter module that can help you to program using events. You can also use it to structure your code and avoid callback hell. However this may not help you in large structure code but it is an option.

Consider below code.

var EventEmitter = require('events').EventEmitter;
var emitter = new EventEmitter();
var fs = require('fs');
// Event to read file - generic function.
emitter.on('start_read',function(file_name) {
  console.log("Started Reading file....\n\n");
  fs.readFile(file_name, 'utf8', function (err,data) {
    if (err) {
      console.log("Error happens.");
    } else {
      console.log("File data", data);
    }
  });
});
// Let's read some file.
emitter.emit('start_read','env.json');
emitter.emit('start_read','envConfig.js');

Here we have created one generic function which perform the file read. You can call this function by emitting events. You can also extend it to emit the events when file reading are done.

Use Async.js

Async.js is utility module provides various functions (around 70) to handle the Asynchronous JavaScript. This is most preferred way and recommended way by us.

Let’s look at some common situation which you may across while writing code and proposed solution for same using Async.js.

Scenario : Run multiple tasks that does not depend on each other and when they all finish do something else

In this scenario, we want to execute multiple asynchronous function which are not dependent on each other i.e We don’t need to block them.

Solution : Use async.parallel() function

Here is code to explain same.

var async = require('async');

async.parallel([
    function(callback) {
      // Some Async task
      callback();
    },
    function(callback) {
      // Some Async task
      callback();
    }
  ],function(err,data) {
  // Code to execute when everything is done.
});

Scenario : Run multiple tasks one after another and once they are finish execute something else
No need to explain the scenario.

Solution : Use async.series() function

Here is code to explain same.

var async = require('async');

async.series([
    function(callback) {
      // some async task
      callback();
    },
    function(callback) {
      // some async task
      callback();
    }
  ],function(err) {
    // Code to execute when everything is done.
});

Scenario : Run multiple tasks one after another and exchange data between them and once they are finish execute something else

This is the scenario very similar to above one except that we need to pass some data to the next function. Async.series() will pass each functions data to final callback function not to the next one.

Solution : Use async.waterfall() function

Here is code to explain same.

var async = require('async');

async.waterfall([
    function(callback) {
      // some code to execute
      // in case to go to next function provide callback like this.
      callback(null,valueForNextFunction);
      // Got some error ? Don't wanna go further.
      // Provide true in callback and execution will stop.
      //callback(true,"Some error");
    },
    function(parameterValue,callback) {
      // Some code to execute.
      callback(null,"Some data");
    }
  ],function(err,data) {
  // Code to execute after everything is done.
});

Run multiple parallel task for by iterating over array ( any collection ) and once all of them are finish execute something else

Consider a scenario where you need to send an email to 1000 users. You have email pushed in an array and you want to execute email() function independent of each other.

Solution : Use async.each()

Here is code to explain same.

var async = require('async');
var emails = ["abc@xyz.com","blahblah@aa.com"];

async.each(emails,function(singleEmail,callback) {
  // Emailer code
  // singleEmail will be one value at a time.
},function(err,data) {
  // Once all done, comes here.
});

Run multiple parallel task by iterating over array ( any collection ) in a set and once all of them are finish execute something else

Same scenario as mentioned above, instead we need limit by which we divide our data. Say you have 1 million email and you want to process 1000 emails at a time.

Solution : use async.eachLimit()

Here is code to explain same.

var async = require('async');
var emails = ["abc@xyz.com","blahblah@aa.com"];

async.eachLimit(emails,1000,function(singleEmail,callback) {
  // Emailer code
  // singleEmail will be one value at a time.
},function(err,data) {
  // Once all done, comes here.
});

Run multiple serial task for by iterating over array ( any collection ) and once all of them are finish execute something else

Same scenario as above, instead of going parallel we need to go for one task at a time.

Solution: You can either use async.eachLimit() with 1 as concurrency limit OR go for async.eachSeries()
Run multiple parallel task by iterating over array ( any collection ) and inside each parallel task run some tasks in series and once all of them are finish execute something else

Consider same email scenario, after sending an email you also need to update the database. That means you need to execute two task for each emails and that too in series manner by passing data from email() function to database one.

Use async.each() with async.waterfall()

Here is code to explain same.

var async = require('async');
var emails = ["abc@xyz.com","blahblah@aa.com"];

async.each(emails,function(singleEmail,callback) {
  async.waterfall([
    function(callback) {
      // code to send email.
      callback(null,Flag);
    },
    function(emailSentOrNot,callback) {
      // Update DB.
    }
  ],function(err,data) {
  });
},function(err,data) {
  // Once all done, comes here.
});

Basically you can use any combination of async.js functions to fulfill your requirement. Here is some combination example.

var async = require('async');

async.forEach(someData,function(singleData,callback){
  async.series();
  //OR
  async.paralle();
  //OR
  async.waterfall();
},function(err,data) {
  // final callback
});

// going more deep.

async.forEach(someData,function(singleData,callback){
  async.waterfall([
      function(callback) {
        async.forEachLimit(somedata,100,function(singleData,callback){
          // You can use more combo here too.
          callback(null);
        },function(err) {
          // final callback
          // Now call the callback of waterfall.
          callback(null,"No error");
        });
      }
    ],function(err,data) {
      // callback of top async.forEach()
      callback(null);
  });
},function(err) {
  // final callback
});

Performing Queue operation

Considering mass mailer in real world scenario, you cannot invoke like millions of callback function at one particular time and it is because of resource limitation and buffer.

In this kind of situation where operation should invoke in batch say 10000 emails at once and keep executing till everything is done you can use async.queue().

Here is very simple queuing example to send an Email. You can refer this tutorial for mailer code.

var async = require('async');

// Send email
var sendEmail = function(email,callback) {
  console.log("Sending email to "+email);
  callback(null);
}

// create a queue object with concurrency 2
var q = async.queue(sendEmail,2);

// Called when every processing is done
q.drain = function() {
    console.log('all emails sent');
}

// add some emails to the queue
q.push(["rwtc66@gmail.com","shahid@codeforgeek.com"]);

// add email to the front of the queue
q.unshift("abc@gmail.com");

//output
/*
Sending email to abc@gmail.com
Sending email to rwtc66@gmail.com
Sending email to shahid@codeforgeek.com
all emails sent
*/

Wrapping up

We have covered some important and useful async.js function, however there are many more and you can surely use them according to your programming scenario.

If you come across any situation which we have not covered and you stuck on it, let us know in comments and we will surely look at it.

Conclusion

Async.js is no doubt very useful package for Node.js developer. It will save a lot of time and make your code looks good as well.

Learn More

#node-js #javascript

Understanding Asynchronous Programming in Node.js
4 Likes33.35 GEEK