Easily identify problems in Node.js applications with Diagnostic Report

Easily identify problems in Node.js applications with Diagnostic Report

The Diagnostic Report utility was recently brought into the Node.js core to help developers identify almost all scenarios of Node.js application anomalies in production. The scenarios include abnormal termination such as a crash, slow performance, memory leak, high CPU, unexpected errors, incorrect output, and more.

The Diagnostic Report utility was recently brought into the Node.js core to help developers identify almost all scenarios of Node.js application anomalies in production. The scenarios include abnormal termination such as a crash, slow performance, memory leak, high CPU, unexpected errors, incorrect output, and more.

While the report does not pinpoint the exact problem or specific fixes, its content-rich diagnostic data offers vital hints about the issue and accelerates the diagnostic process.

The utility was originally available as a npm module, and was brought into the Node.js core because it significantly helps identify the root cause of numerous types of problems, including support issues sent to the different repositories in the Node.js organization. Before it was part of the core, you had to explicitly add the dependency to the npm module in a users’ application, which was a blocker for adoption of the diagnostic tool.

In this blog post, I describe why this tool is important, and then go into some detail on how to interpret the report data, and towards the end of the post, walk you through some example use cases.

Common diagnostic steps

Typically, the starting point to diagnose a problem in an application is to:

  • Capture data to understand the execution environment of the deployment
  • Define a debugging strategy based on the information you obtained
  • Execute one or more investigative steps. Potentially, each step could change what you look for or do based on the inference from the previous steps
  • Iterate until the current theory is confirmed by the data captured

Problem determination of Node.js deployments involves a number of different tools and methodologies. The problem itself determines the action you take to resolve the issue.

For example, if your application crashes, you would:

  • Load the crash dump into a debugger
  • Examine the failing context in the failing thread
  • Go through the execution sequence backwards to identify what code flow or data flow led to the anomaly and what application, third party, runtime code, or configuration is responsible for the said code flow or data flow.

For an issue related to a memory leak, these steps might be different.

Problem determination problems in production-grade deployments

For production-grade deployments, the approach for diagnosing the problem that I outlined above poses a number of challenges, specifically:

  • Unintended business impact. The steps can be iterative, and if followed, can cause unacceptable impact to the business that the production system represents. Even if they’re not iterative, they can cause the debugging task to be delayed. For example, the next data capture needs to wait until the next recycle, which is weeks away.
  • Prone to human error. These diagnostic steps can be error prone depending on the skill level of the person who’s conducting them. For example, an IT admin who carries out the steps at the production site is not skilled enough with Node.js diagnostics.
  • Knowledge gaps. Many times, additional settings need to be set on the system to collect data. For example, you might have to change ulimit settings to enable a dump or enable gc logging to collect the heap statistics and gc activities.
  • Data collection limitations. Many times, the required data may not be easy or even possible to collect. Examples of hard to collect data include: How many handles were in the event loop? What states were those are in? Which SSL library the Node executable was linked against?
  • Binary data hurdles. Because the data is being collected in binary format, many times, the user or admin has no idea what debugging process is being carried out or what data is being collected. Sometimes it’s hard to gain approval for forwarding binary data to the devs who can resolve the issue versus a readable format which can more easily be reviewed and sanitized.
Solution

The solution is useful documentation that explains the most common diagnostic data that is pertinent to your specific execution environment. Diagnostic Report does this using first failure data capture (FFDC). This document is in semi man-machine readable format, so you can read it in its original state if you’re moderately skilled at diagnostics reporting or it can be loaded into a JS program or passed to a monitoring agent.

This document can improve the overall troubleshooting experience because it:

  • Answers many routine questions which can reduce the number of iterations needed to understand the cause of the failure.
  • Offers a comprehensive view of the state of the application and virtual machine at the time of failure. This information can drastically improve the decision making for the next set of data collection, if required.

Ideally, the FFDC enables someone to resolve the issue without any additional information!

Function

Diagnostic Report is an experimental tool that is built into the Node.js core. Its function is to produce a JSON document about points of application misbehavior, or at a point where the user is interested in getting more information. The document produced contains information about the state of the application and the hosting platform, covering all the vital data elements.

The following command line argument runs Diagnostic Report (there are many other ones but this is one).

$ node--experimental-report --diagnostic-report-uncaught-exception w.js

Writing Node.js report to file: report.20190309.102401.47640.001.json
Node.js report completed

Data that it captures could be related to anomalies like fatal errors that terminate the program, application exceptions, or any other common failure scenarios. The data that the tools actually captures could be JavaScript heap statistics, native and application callstack, process’ CPU consumption, and more.

A few command line arguments are available to control the report generation triggers and the report generation behaviors.

You can also generate the report explicitly via an API which is exposed through the Node.js process object. When using the API, the report is available both as a disk file or a JSON string. Another API controls the report generation triggers and reports generation behaviors.

Use cases

In this section, we illustrate some of the benefits of Diagnostic Report through a few different use cases. Keep in mind that this list isn’t exhaustive. Diagnostic Report is a general-purpose tool that can be used in any problem scenarios.

  1. Identify which SSL library the current Node installation is linked against (roughly identify the distribution).
  2. In this case, you can produce a report through the process.report.writeReport() API. As the following image shows, the component versions section contains the SSL linkage information. In this case, it is linked against version 1.1.1b of openssl (line 12).
  3. Reviewing the shared libraries that Node is linked against, you see no external SSL libraries in the list. From there, you can conclude that this is a standard community distribution.
  4. A Node application hangs when you expect it to complete some tasks and then terminate. You have no idea what is causing the event loop to engage.
  5. In this case, produce the report by sending a SIGUSR2 signal to the running process.
  6. The generated report shows an active timer handle lying in the loop that has an expiry time of around 10 hours from the current time. (You can see that on line 4; “firesinMSfromNow” shows how many milliseconds it takes to fire).
  7. Because the application should not be scheduling an event for 10 hours in the future, you now understand the reason for hang. To fix, search in the application that installs a setTimeout handler with the said duration.
  8. You to make sure a web application that you host in a cloud environment is idle outside of business hours.
  9. In this case, you again produce a report by logging into the cloud instance through SSH and sending a signal to the running process. The report generated in the persistent volume showed the resource usage section:
  10. Lines two and three show the time spent by the node process in the user space and kernel space. Because they were only a fraction of a second spent, you can be confident that the application is relatively idle, with no file system activities in the recent past.
We need your feedback

Diagnostic Report is available as an experimental feature from Node.js v11.8.0 and subsequent releases. The tool could exit the experimental status and become a stable and supported feature, based on:

  • The perceived usability in the field
  • Any fine-grained tuning that may be required at the API interface level.

Again, this is based on user feedback.

In software development, ‘feature freeze’ is the inability to refine interfaces because they are already massively used in the field and have many software abstractions built on top of them; any changes to the interfaces can break all these.

To avoid feature freeze with our Diagnostic Report tool, we ask that you evaluate this feature as soon as you get an opportunity and provide your valuable feedback directly in the Node.js Diagnostic User Feedback repo.

Top 7 Most Popular Node.js Frameworks You Should Know

Top 7 Most Popular Node.js Frameworks You Should Know

Node.js is an open-source, cross-platform, runtime environment that allows developers to run JavaScript outside of a browser. In this post, you'll see top 7 of the most popular Node frameworks at this point in time (ranked from high to low by GitHub stars).

Node.js is an open-source, cross-platform, runtime environment that allows developers to run JavaScript outside of a browser.

One of the main advantages of Node is that it enables developers to use JavaScript on both the front-end and the back-end of an application. This not only makes the source code of any app cleaner and more consistent, but it significantly speeds up app development too, as developers only need to use one language.

Node is fast, scalable, and easy to get started with. Its default package manager is npm, which means it also sports the largest ecosystem of open-source libraries. Node is used by companies such as NASA, Uber, Netflix, and Walmart.

But Node doesn't come alone. It comes with a plethora of frameworks. A Node framework can be pictured as the external scaffolding that you can build your app in. These frameworks are built on top of Node and extend the technology's functionality, mostly by making apps easier to prototype and develop, while also making them faster and more scalable.

Below are 7of the most popular Node frameworks at this point in time (ranked from high to low by GitHub stars).

Express

With over 43,000 GitHub stars, Express is the most popular Node framework. It brands itself as a fast, unopinionated, and minimalist framework. Express acts as middleware: it helps set up and configure routes to send and receive requests between the front-end and the database of an app.

Express provides lightweight, powerful tools for HTTP servers. It's a great framework for single-page apps, websites, hybrids, or public HTTP APIs. It supports over fourteen different template engines, so developers aren't forced into any specific ORM.

Meteor

Meteor is a full-stack JavaScript platform. It allows developers to build real-time web apps, i.e. apps where code changes are pushed to all browsers and devices in real-time. Additionally, servers send data over the wire, instead of HTML. The client renders the data.

The project has over 41,000 GitHub stars and is built to power large projects. Meteor is used by companies such as Mazda, Honeywell, Qualcomm, and IKEA. It has excellent documentation and a strong community behind it.

Koa

Koa is built by the same team that built Express. It uses ES6 methods that allow developers to work without callbacks. Developers also have more control over error-handling. Koa has no middleware within its core, which means that developers have more control over configuration, but which means that traditional Node middleware (e.g. req, res, next) won't work with Koa.

Koa already has over 26,000 GitHub stars. The Express developers built Koa because they wanted a lighter framework that was more expressive and more robust than Express. You can find out more about the differences between Koa and Express here.

Sails

Sails is a real-time, MVC framework for Node that's built on Express. It supports auto-generated REST APIs and comes with an easy WebSocket integration.

The project has over 20,000 stars on GitHub and is compatible with almost all databases (MySQL, MongoDB, PostgreSQL, Redis). It's also compatible with most front-end technologies (Angular, iOS, Android, React, and even Windows Phone).

Nest

Nest has over 15,000 GitHub stars. It uses progressive JavaScript and is built with TypeScript, which means it comes with strong typing. It combines elements of object-oriented programming, functional programming, and functional reactive programming.

Nest is packaged in such a way it serves as a complete development kit for writing enterprise-level apps. The framework uses Express, but is compatible with a wide range of other libraries.

LoopBack

LoopBack is a framework that allows developers to quickly create REST APIs. It has an easy-to-use CLI wizard and allows developers to create models either on their schema or dynamically. It also has a built-in API explorer.

LoopBack has over 12,000 GitHub stars and is used by companies such as GoDaddy, Symantec, and the Bank of America. It's compatible with many REST services and a wide variety of databases (MongoDB, Oracle, MySQL, PostgreSQL).

Hapi

Similar to Express, hapi serves data by intermediating between server-side and client-side. As such, it's can serve as a substitute for Express. Hapi allows developers to focus on writing reusable app logic in a modular and prescriptive fashion.

The project has over 11,000 GitHub stars. It has built-in support for input validation, caching, authentication, and more. Hapi was originally developed to handle all of Walmart's mobile traffic during Black Friday.

How to Install Node.js with npm on Debian 10

How to Install Node.js with npm on Debian 10

Install Node.js with npm on Debian 10 In this tutorial, we are going to learn how to install Node.js with npm on Debian 10

Table of Contents

Install Node.js with npm on Debian 10

In this tutorial, we are going to learn how to install Node.js with npm on Debian 10. Node.js is the opensource JavaScript Run-time environment for server-side execution of JavaScript code. Node.js built on Chrome’s V8 JavaScript engine so it can be used to build different types of server-side applications.

Where npm stands for Node Package Manager which is the default package manager for Node.js. npm is the world’s largest software registry for Node.js packages with thousands of packages available.

In this tutorial we will install Node.js in following two ways:

  1. Install Node.js and npm using Debian repository
  2. Install Node.js and npm using nvm
  3. Install Node.js from the NodeSource repository.

1. Install Node.js and npm using Debian repository

First, Update Debian apt package manager index by running the following command.

sudo apt update

Install Node.js from Debian global repository by typing

sudo apt install node

Confirm the installation of Node.js by typing

node --version

Install npm by running following command

sudo apt install npm

Confirm the installation of npm by typing

npm --version

2. Install Node.js and npm using nvm

NVM stands for Node Version Manager which is used to manage multiple Node.js versions. If you want to install or uninstall different versions of Node.js then NVM is there for you.

First, we will install NVM (Node Package Manager) on your system. So download the NVM installation script running the following command.

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash

Check nvm version and confirm installation typing

node --version

Now install Node.js by using the following command.

nvm install node

Verify Node.js installation by typing

node --version

The output should be:

Output

v10.14.0

You can install multiple versions of Node.js. To do so type the following:

nvm install 8.14
nvm install --lts
nvm install 11.3

To list all the versions installed run following command.

nvm ls

You can change the current default version of Node.js by using the following command.

nvm use 8.14

To uninstall a Node.js version type following command

nvm uninstall 11.14

Install Node.js from NodeSource Repository

NodeSource company provides enterprise-grade node support also maintains the repository containing the latest version of Node.js.

To enable the NodeSource repository on your system run following command.

curl -sL https://deb.nodesource.com/setup_10.x | sudo bash -

NOTE: The latest LTS version of Node.js is 10.x if you want to install 8.x version then just replace setup_10.x with setup_8.x

Now install Node.js and npm package typing.

sudo apt install nodejs

Verify installation of Node.js and npm running following command

node --version
npm --version

Install Development Tools

Now install some packages needed for development by running following command

sudo apt install gcc g++ make

Uninstall Node.js and npm

Uninstall Node.js use following command

sudo apt remove nodejs npm
sudo apt autoremove

To uninstall node.js version using nvm type following command

nvm uninstall 10.14

Conclusion

You have successfully learned how to install Node.js with npm on Debian 10. If you have any queries don’t forget to comment below.

Hashing Passwords with Node.js and NPM Bcrypt Library

Hashing Passwords with Node.js and NPM Bcrypt Library

In this tutorial, we will learn to use NPM bcryptjs library to hash and compare the passwords in Node.js

In this tutorial, we will learn to use NPM bcryptjs library to hash and compare the passwords in Node.

To create a secure application, it is always considered a safe practice not to store a user’s password in the database in plain text format. If not in plain text format, then what else we can do?

Here is the solution, generate a hash (complex string and numbers) and store that hash in the database. You can decipher your hashed password later by using the comparing method.

Let’s assume if there was a breach in your database, and all your stored passwords were leaked. Then, you are at significant risk, and password hashing is the best one-way encryption technique to secure the passwords.

In this method, you do not store users’ passwords in the database in its original form. Instead, a password is stored in a complex combination of text and unique characters; this is known as a password hash method.

A hacker can not easily decipher an adequately hashed password. Hackers will get frustrated because it will take lots of time and effort to decrypt the password.

In this tutorial, we will learn how to install and correctly hash a password in node.js.

We will take the help of the NPM BcryptJs package, and it is a widely used encryption module available nowadays via NPM.

Before we begin, you must have Node.js configured in your machine. If not, then you can check out how to install Node in your system tutorial.

Install bcryptjs Npm Module

To get started i assume you already have a Node.js project setup along with Express, and MongoDB.

Run one of the command based on your package manage.

# npm
npm install bcryptjs --save

# yarn
yarn add bcryptjs

Now, once bcryptjs successfully installed. We are ready to go ahead!

Hash A Password with Bcrytp Js

To get started with hashing the password we need node server configuration. In the app.js file, we imported express, bodyParser, mongoose and bcrytpjs module. We defined the MongoDB database connection, user schema and two REST APIs for registering and signing in the user.

const express = require('express');
const mongoose = require('mongoose');
const cors = require('cors');
const bodyParser = require('body-parser');

// Express APIs
const api = require('./routes/auth.routes');

// MongoDB conection
mongoose.Promise = global.Promise;
mongoose.connect("mongodb://localhost:27017/nodedb", {
    useNewUrlParser: true,
    useUnifiedTopology: true
}).then(() => {
    console.log('Database connected')
},
    error => {
        console.log("Database can't be connected: " + error)
    }
)

// Express settings
const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({
    extended: false
}));
app.use(cors());

app.use('/api', api)

// Define PORT
const port = process.env.PORT || 4000;
const server = app.listen(port, () => {
    console.log('Connected to port ' + port)
})

// Express error handling
app.use((req, res, next) => {
    setImmediate(() => {
        next(new Error('Something went wrong'));
    });
});

app.use(function (err, req, res, next) {
    console.error(err.message);
    if (!err.statusCode) err.statusCode = 500;
    res.status(err.statusCode).send(err.message);
});

Hashing a password is very simple, the first argument in the bcrypt.hashSync() method is the password which we are getting from req.body middleware. The second argument is the number of rounds which we set to 10 to generate a salt.

// routes/auth.routes.js

const express = require("express");
const jwt = require("jsonwebtoken");
const bcrypt = require("bcrypt");
const router = express.Router();
const userSchema = require("../models/User");

// Sign-up
router.post("/signup", (req, res, next) => {
    bcrypt.hash(req.body.password, 10).then((hash) => {
        const user = new userSchema({
            name: req.body.name,
            email: req.body.email,
            password: hash
        });
        user.save().then((response) => {
            res.status(201).json({
                message: "User successfully created!",
                result: response
            });
        }).catch(error => {
            res.status(500).json({
                error: error
            });
        });
    });
});

So we are hashing the password when the user makes the signup call after that we are creating a user instance and saving the user data along with the password in the MongoDB database.

Verify or Compare The Password with Bcrypt

When the user logs in the app, API will check the if the email exists in the database with the help of userSchema.findOne() method. Then, we will validate the stored password with the help of bcrypt.compareSync() method. It takes two passwords as an argument stored password and user-entered password.

// routes/auth.routes.js

const express = require("express");
const jwt = require("jsonwebtoken");
const bcrypt = require("bcrypt");
const router = express.Router();
const userSchema = require("../models/User");

// Sign-in
router.post("/signin", (req, res, next) => {
    let getUser;
    userSchema.findOne({
        email: req.body.email
    }).then(user => {
        if (!user) {
            return res.status(401).json({
                message: "Authentication failed"
            });
        }
        return bcrypt.compare(req.body.password, user.password);
    }).then(response => {
        if (!response) {
            return res.status(401).json({
                message: "Authentication failed"
            });
        }
    }).catch(err => {
        return res.status(401).json({
            message: "Authentication failed"
        });
    });
});
Conclusion

We have seen how to store the password in the database securely by making the REST API call with Node/Express.