Node Express Analytics Dashboard with Cube.js

Node Express Analytics Dashboard with Cube.js

In the following tutorial, I’ll show you how to create a basic analytics dashboard with Node, Express, and&nbsp;<a href="https://github.com/statsbotco/cube.js" target="_blank">Cube.js</a>. As a database, we’re going to use MongoDB with MongoDB BI Connector. If you’re not familiar with it, I highly recommend you go through the&nbsp;<a href="https://statsbot.co/blog/building-mongodb-dashboard-using-node.js" target="_blank">Building a MongoDB Dashboard tutorial</a>. It covers the basics of setting up Mongo and its BI Connector.

In the following tutorial, I’ll show you how to create a basic analytics dashboard with Node, Express, and Cube.js. As a database, we’re going to use MongoDB with MongoDB BI Connector. If you’re not familiar with it, I highly recommend you go through the Building a MongoDB Dashboard tutorial. It covers the basics of setting up Mongo and its BI Connector.

Here how the final dashboard would look. You can check the live demo on Heroku here. The full source code is available on Github.

Getting a Sample Dataset


You can skip this step if you already have some data for your dashboard

If you don’t have a local MongoDB instance, please download it here. The BI Connector can be downloaded here.

There is a good repository on Github with a curated list of JSON / BSON datasets from the web in order to practice in MongoDB. We’ll pick a tweets dataset for our dashboard.

Download test data and import it using the following command in your MongoDB directory.

$ bin/mongorestore Your-Downloads-Folder/dump/twitter/tweets.bson

Now make sure both MongoDB and MongoDB BI Connector processes are running.

# Run from MongoDB directory
$ bin/mongod
Run from MongoDB BI Connector directory

$ bin/mongosqld

Setting up a Backend

We’ll use an express application generator to create an application skeleton.

# Install it if you don’t have it already
$ npm install express-generator -g

Next, create a new express app with the view engine set to Handlebars (hbs).

$ express --view=hbs express-analytics-dashboard

We’re going to use an open source framework, Cube.js as our analytical backend. It generates and executes SQL queries, as well as provides caching, data pre-aggregation, security, and API to query results and build visualizations. You can learn more about it here.

Cube.js can be easily embedded into an Express application. Let’s add it to our project dependencies.

$ npm install --save @cubejs-backend/server-core @cubejs-backend/mongobi-driver dotenv

We’ve added a core server package for Cube.js and the Cube.js MongoBI driver. We’ve also added a dotenv package to manage our credentials. Let’s create a .env file with the following credentials; we need them to tell Cube.js how to connect to Mongo.

CUBEJS_DB_HOST=localhost
CUBEJS_DB_NAME=twitter
CUBEJS_DB_PORT=3307
CUBEJS_DB_TYPE=mongobi
CUBEJS_API_SECRET=SECRET

Now, let’s mount the Cube.js Server into our express application. Add the following code right after the routes declaration in your app.js.

var CubejsServerCore = require('@cubejs-backend/server-core');
// ...
app.use('/', indexRouter);

require('dotenv').config();
CubejsServerCore.create().initApp(app);
// ...

With the above two lines of code, we have loaded all required configs from the .env file and mounted Cube.js into our Express app. By default, it’s mounted into the /cubejs-api/v1/ path namespace. But you can change it and a lot of other things by passing the configuration object to the CubejsServerCore.create()method. We’ll keep the default settings for our tutorial.

Now, let’s create a Cube.js Schema for our tweets table. Cube.js uses Data Schema to generate and execute SQL; you can read more about it here.

Create a folder, schema, with a Tweets.js file inside with the following content.

cube(Tweets, {
sql: select * from tweets,

measures: {
count: {
type: count
},

favoriteCount: {
  type: `sum`,
  sql: `favorite_count`
},

retweetCount: {
  type: `sum`,
  sql: `retweet_count`
}

},

dimensions: {
location: {
type: string,
sql: \user.location``
},

lang: {
  type: `string`,
  sql: `lang`
}

}
});

In Cube.js, you can describe your queries in Javascript and then they will be compiled into SQL and executed inside your database. It uses measures and dimensions as basic units to describe various analytics queries. This tutorial is a good place to get started with Cube.js Schema.

Now let’s move on to building a dashboard on the frontend.

Alternative Setup: Run Cube.js in Serverless Mode


If you want to run it as a microservice or as a serverless function - use Cube.js CLI. The code below shows how you can generate a new Cube.js app with Cube.js CLI:

$ npm install -g cubejs-cli
$ cubejs create -d mongobi -t serverless

It will create a new project pre-configured to be deployed to AWS Lambda with Serverless framework. You can learn more about Cube.js serverless deployment here.

Building Analytics Dashboard

We’ll use Bootstrap for styling, the Cube.js client to load data, and Chart.js to display it. Replace the content of views/index.hbs with the following.

<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.7.2/Chart.bundle.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/numeral.js/2.0.6/numeral.min.js"></script>
<script src="/javascripts/cubejs-client-core.umd.js"></script>

<div class="container" id="app">
<div class="row">
<div class="col-md-4">
<div class="card">
<div class="card-body">
<h5 class="card-title">Total Tweets</h5>
<div class="card-text">
<h3 id="total-tweets"></h3>
</div>
</div>
</div>
</div>
<div class="col-md-4">
<div class="card">
<div class="card-body">
<h5 class="card-title">Total Retweets</h5>
<div class="card-text">
<h3 id="total-retweets"></h3>
</div>
</div>
</div>
</div>
<div class="col-md-4">
<div class="card">
<div class="card-body">
<h5 class="card-title">Total Favorites</h5>
<div class="card-text">
<h3 id="total-favorites"></h3>
</div>
</div>
</div>
</div>
</div>
<br />
<br />
<div class="row">
<div class="col-md-6">
<div class="card">
<div class="card-body">
<h5 class="card-title">Top Tweets Locations</h5>
<div class="card-text">
<canvas id="pie-chart"></canvas>
</div>
</div>
</div>
</div>
<div class="col-md-6">
<div class="card">
<div class="card-body">
<h5 class="card-title">Most Popular Languages</h5>
<div class="card-text">
<canvas id="bar-chart"></canvas>
</div>
</div>
</div>
</div>
</div>
</div>

<script>
var cubejsApi = cubejs(
'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1NTIzOTk5MjcsImV4cCI6MTU1MjQ4NjMyN30.SOO-A6GfGH7ar3EoeBb0cjj10BVxO3ffjvmqQziXIZA',
{ apiUrl: 'http://localhost:3000/cubejs-api/v1' }
);

var kpis = [
{ measure: "Tweets.count", element: "total-tweets" },
{ measure: "Tweets.retweetCount", element: "total-retweets" },
{ measure: "Tweets.favoriteCount", element: "total-favorites" }
];

kpis.forEach(kpi => {
cubejsApi.load({
measures: [kpi.measure]
}).then(resultSet => {
document.getElementById(kpi.element).textContent =
numeral(resultSet.totalRow()[kpi.measure]).format('0,0');
})
});

// A helper method to format data for Chart.js
// and add some nice colors
var chartJsData = function(resultSet) {
return {
datasets: [{
data: resultSet.series()[0].series.map(function(r) { return r.value }),
backgroundColor: [
'rgb(255, 99, 132)',
'rgb(255, 159, 64)',
'rgb(255, 205, 86)',
'rgb(75, 192, 192)',
'rgb(54, 162, 235)'
]
}],
labels: resultSet.categories().map(function(c) { return c.category })
}
}

cubejsApi.load({
measures: ["Tweets.count"],
dimensions: ["Tweets.location"],
filters: [
{
dimension: "Tweets.location",
operator: "notEquals",
values: [""]
}
],
limit: 5
}).then(resultSet => {
new Chart(document.getElementById("pie-chart"), {
type: 'pie',
data: chartJsData(resultSet)
})
});

cubejsApi.load({
measures: ["Tweets.count"],
dimensions: ["Tweets.lang"],
limit: 5
}).then(resultSet => {
new Chart(document.getElementById("bar-chart"), {
type: 'bar',
data: chartJsData(resultSet),
options: { legend: { display: false } }
})
});
</script>

Let’s break this down into pieces. First, we’re loading our required libraries. The Cube.js client could be installed in different ways, here we’re just loading a UMD build from CDN. We’re also loading Bootstrap, Chart.js, and numeral.js to format numbers from CDN.

<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.7.2/Chart.bundle.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/numeral.js/2.0.6/numeral.min.js"></script>
<script src="/javascripts/cubejs-client-core.umd.js"></script>

The next part is just a plain HTML markup with the Bootstrap grid.

The last part is where we load and display data in our dashboard widgets. For the sake of this tutorial, we don’t use any frontend libraries. But, if you want, Cube.js has bindings for all popular frontend frameworks, such as React.

First, we’re initializing the Cube.js client and passing an API Token and API URL. Your API Token should be printed to the terminal upon the server start. The URL should be the same.

var cubejsApi = cubejs(
'YOUR-API-TOKEN',
{ apiUrl: 'http://localhost:3000/cubejs-api/v1' }
);

Next, we’re loading and displaying data for the upper row of the dashboard, the KPIs section. Here we display just the plain numbers with some formatting done by numeral.js.

var kpis = [
{ measure: "Tweets.count", element: "total-tweets" },
{ measure: "Tweets.retweetCount", element: "total-retweets" },
{ measure: "Tweets.favoriteCount", element: "total-favorites" }
];

kpis.forEach(kpi => {
cubejsApi
.load({
measures: [kpi.measure]
})
.then(resultSet => {
document.getElementById(kpi.element).textContent = numeral(
resultSet.totalRow()[kpi.measure]
).format("0,0");
});
});

The row has one pie and one bar chart, plotted with Chart.js. To display the bar chart, we’re requesting the Tweets.count measure and grouping it by the Tweets.location dimension. We’re also applying a filter to exclude tweets with an empty location. Finally, we’re setting the limit to 5 to get only the top 5 locations.

You can learn more about the Cube.js Query format here.

For the bar chart, we’re doing a similar grouping, but instead of location, we’re grouping by the Tweets.lang dimension.

// A helper method to format data for Chart.js
// and add some nice colors
var chartJsData = function(resultSet) {
return {
datasets: [
{
data: resultSet.series()[0].series.map(function(r) {
return r.value;
}),
backgroundColor: [
"rgb(255, 99, 132)",
"rgb(255, 159, 64)",
"rgb(255, 205, 86)",
"rgb(75, 192, 192)",
"rgb(54, 162, 235)"
]
}
],
labels: resultSet.categories().map(function(c) {
return c.category;
})
};
};

cubejsApi
.load({
measures: ["Tweets.count"],
dimensions: ["Tweets.location"],
filters: [
{
dimension: "Tweets.location",
operator: "notEquals",
values: [""]
}
],
limit: 5
})
.then(resultSet => {
new Chart(document.getElementById("pie-chart"), {
type: "pie",
data: chartJsData(resultSet)
});
});

cubejsApi
.load({
measures: ["Tweets.count"],
dimensions: ["Tweets.lang"],
limit: 5
})
.then(resultSet => {
new Chart(document.getElementById("bar-chart"), {
type: "bar",
data: chartJsData(resultSet),
options: { legend: { display: false } }
});
});

Now, to see the dashboard in action, start your server.

$ npm start

And visit http://localhost:3000 to see your analytics dashboard in action. Also, we have a live demo of the app hosted on Heroku here. The full source code is available on Github.

Why Cube.js

Why is using Cube.js better than hitting MongoDB directly with SQL queries? Cube.js solves a plethora of different problems every production-ready analytic application needs to solve: analytic SQL generation, query results caching and execution orchestration, data pre-aggregation, security, API for query results fetch, and visualization.

These features allow you to build production-grade analytics applications that are able to handle thousands of concurrent users and billions of data points. They also allow you to do analytics on a production MongoDB read replica or even a MongoDB main node due to their ability to reduce the amount of actual queries issued to a MongoDB instance.

Secure Node.js, Express.js and PostgreSQL API using Passport.js

Secure Node.js, Express.js and PostgreSQL API using Passport.js

The comprehensive step by step tutorial on building secure Node.js, Express.js, Passport.js, and PostgreSQL Restful Web Service

The comprehensive step by step tutorial on building secure Node.js, Express.js, Passport.js, and PostgreSQL Restful Web Service. Previously, we have shown you a combination of Node.js, Express.js, and PostgreSQL tutorial. Now, we just add a security for that RESTful Web Service endpoints. Of course, we will start this tutorial from scratch or from zero application. We will use JWT for this Node.js, Express.js, Passport.js, and PostgreSQL tutorial.

Table of Contents:

The following tools, frameworks, and modules are required for this tutorial:

We assume that you have installed PostgreSQL server in your machine or can use your own remote server (we are using PostgreSQL 9.5.13). Also, you have installed Node.js in your machine and can run node, npm or yarn command in your terminal or command line. Next, check their version by type this commands in your terminal or command line.

node -v
v8.12.0
npm -v
6.4.1
yarn -v
1.10.1

That the versions that we are uses. Let’s continue with the main steps.

1. Create Express.js Project and Install Required Modules

Open your terminal or node command line the go to your projects folder. First, install express generator using this command.

sudo npm install express-generator -g

Next, create an Express.js app using this command.

express secure-node --view=ejs

This will create Express.js project with the EJS view instead of Jade view template because using ‘–view=ejs’ parameter. Next, go to the newly created project folder then install node modules.

cd secure-node && npm install

You should see the folder structure like this.

There’s no view yet using the latest Express generator. We don’t need it because we will create a RESTful API.

2. Add and Configure Sequelize.js Module and Dependencies

Before installing the modules for this project, first, install Sequelize-CLI by type this command.

sudo npm install -g sequelize-cli

To install Sequelize.js module, type this command.

npm install --save sequelize

Then install the module for PostgreSQL.

npm install --save pg pg-hstore

Next, create a new file at the root of the project folder.

touch .sequelizerc

Open and edit that file then add this lines of codes.

const path = require('path');

module.exports = {
&nbsp; "config": path.resolve('./config', 'config.json'),
&nbsp; "models-path": path.resolve('./models'),
&nbsp; "seeders-path": path.resolve('./seeders'),
&nbsp; "migrations-path": path.resolve('./migrations')
};

That files will tell Sequelize initialization to generate config, models, seeders and migrations files to specific directories. Next, type this command to initialize the Sequelize.

sequelize init

That command will create config/config.json, models/index.js, migrations and seeders directories and files. Next, open and edit config/config.json then make it like this.

{
&nbsp; "development": {
&nbsp; &nbsp; "username": "djamware",
&nbsp; &nbsp; "password": "[email&nbsp;protected]@r3",
&nbsp; &nbsp; "database": "secure_node",
&nbsp; &nbsp; "host": "127.0.0.1",
&nbsp; &nbsp; "dialect": "postgres"
&nbsp; },
&nbsp; "test": {
&nbsp; &nbsp; "username": "root",
&nbsp; &nbsp; "password": "[email&nbsp;protected]@r3",
&nbsp; &nbsp; "database": "secure_node",
&nbsp; &nbsp; "host": "127.0.0.1",
&nbsp; &nbsp; "dialect": "postgres"
&nbsp; },
&nbsp; "production": {
&nbsp; &nbsp; "username": "root",
&nbsp; &nbsp; "password": "[email&nbsp;protected]@r3",
&nbsp; &nbsp; "database": "secure_node",
&nbsp; &nbsp; "host": "127.0.0.1",
&nbsp; &nbsp; "dialect": "postgres"
&nbsp; }
}

We use the same configuration for all the environment because we are using the same machine, server, and database for this tutorial.

Before run and test connection, make sure you have created a database as described in the above configuration. You can use the psql command to create a user and database.

psql postgres --u postgres

Next, type this command for creating a new user with password then give access for creating the database.

postgres-# CREATE ROLE djamware WITH LOGIN PASSWORD '[email&nbsp;protected]@r3';
postgres-# ALTER ROLE djamware CREATEDB;

Quit psql then log in again using the new user that previously created.

postgres-# \q
psql postgres -U djamware

Enter the password, then you will enter this psql console.

psql (9.5.13)
Type "help" for help.

postgres=>

Type this command to creating a new database.

postgres=> CREATE DATABASE secure_node;

Then give that new user privileges to the new database then quit the psql.

postgres=> GRANT ALL PRIVILEGES ON DATABASE secure_node TO djamware;
postgres=> \q

3. Create or Generate Models and Migrations

We will use Sequelize-CLI to generating a new model. Type this command to create a model for Products and User model for authentication.

sequelize model:create --name Product --attributes prod_name:string,prod_desc:string,prod_price:float
sequelize model:create --name User --attributes username:string,password:string

That command creates a model file to the model’s folder and a migration file to folder migrations. Next, modify models/user.js and then import this module.

var bcrypt = require('bcrypt-nodejs');

Add the new methods to the User model, so the user.js class will be like this.

module.exports = (sequelize, DataTypes) => {
&nbsp; const User = sequelize.define('User', {
&nbsp; &nbsp; username: DataTypes.STRING,
&nbsp; &nbsp; password: DataTypes.STRING
&nbsp; }, {});
&nbsp; User.beforeSave((user, options) => {
&nbsp; &nbsp; if (user.changed('password')) {
&nbsp; &nbsp; &nbsp; user.password = bcrypt.hashSync(user.password, bcrypt.genSaltSync(10), null);
&nbsp; &nbsp; }
&nbsp; });
&nbsp; User.prototype.comparePassword = function (passw, cb) {
&nbsp; &nbsp; bcrypt.compare(passw, this.password, function (err, isMatch) {
&nbsp; &nbsp; &nbsp; &nbsp; if (err) {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; return cb(err);
&nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; cb(null, isMatch);
&nbsp; &nbsp; });
&nbsp; };
&nbsp; User.associate = function(models) {
&nbsp; &nbsp; // associations can be defined here
&nbsp; };
&nbsp; return User;
};

For the models/product.js there’s no action needed, leave it as default generated the model class.

4. Create Routers for RESTful Web Service and Authentication

To authenticating users and secure the resources or endpoint create this file as a router.

touch routes/api.js

Open and edit routes/api.js then declares all require variables.

const express = require('express');
const jwt = require('jsonwebtoken');
const passport = require('passport');
const router = express.Router();
require('../config/passport')(passport);
const Product = require('../models').Product;
const User = require('../models').User;

Create a router for signup or register the new user.

router.post('/signup', function(req, res) {
&nbsp; console.log(req.body);
&nbsp; if (!req.body.username || !req.body.password) {
&nbsp; &nbsp; res.status(400).send({msg: 'Please pass username and password.'})
&nbsp; } else {
&nbsp; &nbsp; User
&nbsp; &nbsp; &nbsp; .create({
&nbsp; &nbsp; &nbsp; &nbsp; username: req.body.username,
&nbsp; &nbsp; &nbsp; &nbsp; password: req.body.password
&nbsp; &nbsp; &nbsp; })
&nbsp; &nbsp; &nbsp; .then((user) => res.status(201).send(user))
&nbsp; &nbsp; &nbsp; .catch((error) => {
&nbsp; &nbsp; &nbsp; &nbsp; console.log(error);
&nbsp; &nbsp; &nbsp; &nbsp; res.status(400).send(error);
&nbsp; &nbsp; &nbsp; });
&nbsp; }
});

Create a router for sign in or login with username and password.

router.post('/signin', function(req, res) {
&nbsp; User
&nbsp; &nbsp; &nbsp; .find({
&nbsp; &nbsp; &nbsp; &nbsp; where: {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; username: req.body.username
&nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; })
&nbsp; &nbsp; &nbsp; .then((user) => {
&nbsp; &nbsp; &nbsp; &nbsp; if (!user) {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; return res.status(401).send({
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; message: 'Authentication failed. User not found.',
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; });
&nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; user.comparePassword(req.body.password, (err, isMatch) => {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if(isMatch && !err) {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; var token = jwt.sign(JSON.parse(JSON.stringify(user)), 'nodeauthsecret', {expiresIn: 86400 * 30});
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; jwt.verify(token, 'nodeauthsecret', function(err, data){
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; console.log(err, data);
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; })
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; res.json({success: true, token: 'JWT ' + token});
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; } else {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; res.status(401).send({success: false, msg: 'Authentication failed. Wrong password.'});
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; })
&nbsp; &nbsp; &nbsp; })
&nbsp; &nbsp; &nbsp; .catch((error) => res.status(400).send(error));
});

Create a secure router to get and post product data.

router.get('/product', passport.authenticate('jwt', { session: false}), function(req, res) {
&nbsp; var token = getToken(req.headers);
&nbsp; if (token) {
&nbsp; &nbsp; Product
&nbsp; &nbsp; &nbsp; .findAll()
&nbsp; &nbsp; &nbsp; .then((products) => res.status(200).send(products))
&nbsp; &nbsp; &nbsp; .catch((error) => { res.status(400).send(error); });
&nbsp; } else {
&nbsp; &nbsp; return res.status(403).send({success: false, msg: 'Unauthorized.'});
&nbsp; }
});

router.post('/product', passport.authenticate('jwt', { session: false}), function(req, res) {
&nbsp; var token = getToken(req.headers);
&nbsp; if (token) {
&nbsp; &nbsp; Product
&nbsp; &nbsp; &nbsp; .create({
&nbsp; &nbsp; &nbsp; &nbsp; prod_name: req.body.prod_name,
&nbsp; &nbsp; &nbsp; &nbsp; prod_desc: req.body.prod_desc,
&nbsp; &nbsp; &nbsp; &nbsp; prod_price: req.body.prod_price
&nbsp; &nbsp; &nbsp; })
&nbsp; &nbsp; &nbsp; .then((product) => res.status(201).send(product))
&nbsp; &nbsp; &nbsp; .catch((error) => res.status(400).send(error));
&nbsp; } else {
&nbsp; &nbsp; return res.status(403).send({success: false, msg: 'Unauthorized.'});
&nbsp; }
});

Create a function for extract the token.

getToken = function (headers) {
&nbsp; if (headers && headers.authorization) {
&nbsp; &nbsp; var parted = headers.authorization.split(' ');
&nbsp; &nbsp; if (parted.length === 2) {
&nbsp; &nbsp; &nbsp; return parted[1];
&nbsp; &nbsp; } else {
&nbsp; &nbsp; &nbsp; return null;
&nbsp; &nbsp; }
&nbsp; } else {
&nbsp; &nbsp; return null;
&nbsp; }
};

Finally, export the router as a module.

module.exports = router;

5. Run and Test Secure Node.js, Express.js, Passport.js, and PostgreSQL Web Service

To run and test this secure Node.js, Express.js, Passport.js, and PostgreSQL Web Service, run the PostgreSQL instance first then run this command from the Terminal.

nodemon

or

npm start

To test the secure Product endpoint, open the Postman then type fill all required fields like this image.

You should get the response message Unauthorized and status code 401. Next, test signup using the Postman by changing the method to POST, add the address localhost:3000/api/signup, add the header Content-type with value application/json and the body of request raw text like this.

{ "username":"[email&nbsp;protected]", "password":"qqqq1111" }

You should get this response when executing successfully.

Next, test to log in with the above signed/registered username and password by changing the URL to localhost:3000/api/signin. You should get this response when executes successfully.

Now, you can back using the previous GET method with additional header using the token get from the sign in/log in response. You should see the Product data like below.

That it’s, the secure Node.js, Express.js, Passport.js, and PostgreSQL Web Service. You can get the working source code from our GitHub.

Learn More

The Complete Node.js Developer Course (2nd Edition)

Learn and Understand NodeJS

Node JS: Advanced Concepts

GraphQL: Learning GraphQL with Node.Js

Angular (Angular 2+) & NodeJS - The MEAN Stack Guide

The Complete Python & PostgreSQL Developer Course

SQL & Database Design A-Z™: Learn MS SQL Server + PostgreSQL

The Complete SQL Bootcamp

The Complete Oracle SQL Certification Course

Creating a RESTful Web API with Node.js and Express.js from scratch

Creating a RESTful Web API with Node.js and Express.js from scratch

In this article, I’ll show you step by step how to create a RESTful Web API with Node.js and Express.js by building a simple and useful Todo API. This article assumes you have basic javascript knowledge and terminal using capabilities.

In this article, I’ll show you step by step how to create a RESTful Web API with Node.js and Express.js by building a simple and useful Todo API. This article assumes you have basic javascript knowledge and terminal using capabilities.

You can also build a Web API in Node.js by using another framework except Express.js but Express.js is one of the most popular web framework for Node.js.

You can found the final source code of this Web API in this github repository.

Let’s start to create our mentioned Web API.

Before start

If you have never used Node.js or npm package manager you should install them.

To check whether the Node.js is already installed on your computer, open your terminal and run node -v command. If you see your Node.js version it's installed. Otherwise go to below link.

Click here to download and install Node.js (You can choose LTS version)

And if you don’t have any IDE or text editor for writing javascript I advice you Visual Studio Code.

Click here to download VS Code (Optional)

About express-generator

In fact we could use <a href="https://expressjs.com/en/starter/generator.html" target="_blank">express-generator</a> tool which designed to creating an Express Web API quickly but I want to create this API from scratch because of that tool puts some extra files and folder structures that we don't need them now. But you can use this useful tool next time on creating new Web API. I won't use it now due to keep article simple.

Creating Project

Go to your workspace root folder and create a new folder there named "todo-api".

Then create "package.json" and "server.js" files into "todo-api" folder like below.

package.json

{
    "name": "todo-api",
    "version": "1.0.0",
    "scripts": {
        "start": "node server.js"
    },
    "dependencies": {
        "express": "^4.16.4"
    }
}

server.js

const http = require('http');
const express = require('express');
const app = express();
app.use(express.json());
app.use('/', function(req, res) {
    res.send('todo api works');
});
const server = http.createServer(app);
const port = 3000;
server.listen(port);
console.debug('Server listening on port ' + port);

After creating above files open your terminal in the "todo-api" folder and run npm installcommand.

This command will be install your project dependencies which pointed at the "package.json" file.

After finished package download process, downloaded dependency files will be installed into"node_modules" folder at the root of the "todo-api" folder.

After finished package installing then run npm start to start our Web API.

Now our Web API listening. To see result open your web browser then write localhost:3000 to address bar and press enter.

As result you’ll see our request handler response in your browser: “todo api works”.

This is a dead simple Express.js Web API. And it needs the some development. For example we need to an api endpoint to get todo items. So let’s add a new API endpoint for this.

Create a new folder named "routes" in the root of the "todo-api" folder.

Then create a "items.js" file inside of "routes" folder and put following codes inside it.

Your final folder structure should be like below;

/todo-api
/node_modules
/routes
    items.js
package.json
server.js

items.js

const express = require('express');
const router = express.Router();
const data = [
    {id: 1, title: 'Finalize project', order: 1, completed: false, createdOn: new Date()},
    {id: 2, title: 'Book ticket to London', order: 2, completed: false, createdOn: new Date()},
    {id: 3, title: 'Finish last article', order: 3, completed: false, createdOn: new Date()},
    {id: 4, title: 'Get a new t-shirt', order: 4, completed: false, createdOn: new Date()},
    {id: 5, title: 'Create dinner reservation', order: 5, completed: false, createdOn: new Date()},
];
router.get('/', function (req, res) {
    res.status(200).json(data);
});
router.get('/:id', function (req, res) {
    let found = data.find(function (item) {
        return item.id === parseInt(req.params.id);
    });
    if (found) {
        res.status(200).json(found);
    } else {
        res.sendStatus(404);
    }
});
module.exports = router;

Initial code of "items.js" file contains two endpoints. First one gets all todo items and second one gets one item which matches given id parameter.

Before testing items routes we should register it in the "server.js" file.

Modify "server.js" file like below to register new item routes.

server.js

const http = require('http');
const express = require('express');
const itemsRouter = require('./routes/items');
const app = express();
app.use(express.json());
app.use('/items', itemsRouter);
app.use('/', function(req, res) {
    res.send('todo api works');
});
const server = http.createServer(app);
const port = 3000;
server.listen(port);
console.debug('Server listening on port ' + port);

Now run npm start to start our Web API.

Then open your web browser and write localhost:3000/items to address bar and press enter.

You’ll see todo items json array in the response body.

And write localhost:3000/items/3 to address bar and press enter.

You’ll see the todo item which has id 3 in the response body.

But not finished up yet.

CRUD Operations and HTTP methods

I think we’ll need CRUD operations to Create, Read, Update and Delete todo items.

We have already two endpoints for getting items. So we need Create, Update and Delete endpoints.

Let’s add also these endpoints into the items.js file.

Our final "items.js" file and endpoints should be like below.

const express = require('express');
const router = express.Router();

const data = [
  {id: 1, title: 'Finalize project',          order: 1, completed: false, createdOn: new Date()},
  {id: 2, title: 'Book ticket to London',     order: 2, completed: false, createdOn: new Date()},
  {id: 3, title: 'Finish last article',       order: 3, completed: false, createdOn: new Date()},
  {id: 4, title: 'Get a new t-shirt',         order: 4, completed: false, createdOn: new Date()},
  {id: 5, title: 'Create dinner reservation', order: 5, completed: false, createdOn: new Date()},
];

router.get('/', function (req, res) {
  res.status(200).json(data);
});

router.get('/:id', function (req, res) {
  let found = data.find(function (item) {
    return item.id === parseInt(req.params.id);
  });

  if (found) {
    res.status(200).json(found);
  } else {
    res.sendStatus(404);
  }
});

router.post('/', function (req, res) {
  let itemIds = data.map(item => item.id);
  let orderNums = data.map(item => item.order);

  let newId = itemIds.length > 0 ? Math.max.apply(Math, itemIds) + 1 : 1;
  let newOrderNum = orderNums.length > 0 ? Math.max.apply(Math, orderNums) + 1 : 1;

  let newItem = {
    id: newId,
    title: req.body.title,
    order: newOrderNum,
    completed: false,
    createdOn: new Date()
  };

  data.push(newItem);

  res.status(201).json(newItem);
});

router.put('/:id', function (req, res) {
  let found = data.find(function (item) {
    return item.id === parseInt(req.params.id);
  });

  if (found) {
    let updated = {
      id: found.id,
      title: req.body.title,
      order: req.body.order,
      completed: req.body.completed
    };

    let targetIndex = data.indexOf(found);

    data.splice(targetIndex, 1, updated);

    res.sendStatus(204);
  } else {
    res.sendStatus(404);
  }
});

router.delete('/:id', function (req, res) {
  let found = data.find(function (item) {
    return item.id === parseInt(req.params.id);
  });

  if (found) {
    let targetIndex = data.indexOf(found);

    data.splice(targetIndex, 1);
  }

  res.sendStatus(204);
});

module.exports = router;

Short Explanation

I wanna explain shortly some points of our last codes.

First of all you must have noticed that our api works on a static data and keeps it on memory. All of our GET, POST, PUT and DELETE http methods just manipulate a json array. The purpose of this is to keep article simple and draw attention to the Web API structure.

Due to this situation our POST method has some extra logic such as calculating next item ids and order numbers.

So you can modify logic and data structures in these http methods to use a database or whatever you want.

Testing API with Postman

We have tested the GET methods of our Web API in our web browser and seen responses. But we can’t test directly POST, PUT and DELETE http methods in web browser.

If you want to test also other http methods you should use Postman or another http utility.

Now I’ll show you how to test the Web API with Postman

Before we start click here and install Postman.

When you first launch Postman after installing you’ll see start window. Close this start window by clicking close button on top right corner. Then you must see following screen.

An empty Postman request

Sending GET Request

Before sending a request to API we should start it by running npm startcommand as we do before.

After start the Web API and seeing “Server listening on…” message write localhost:3000/itemsto address bar as seen below and click Send button. You'll see todo items array as API response like below.

Sending a GET request with Postman

You can try similarly by giving an item id in request url like this localhost:3000/items/3

Sending POST Request

To sending a POST request and create a new todo item write localhost:3000/items to address bar and change HTTP verb to POST by clicking arrow at front of the address bar as seen below.

Sending a POST request with Postman

Before sending the POST request you should add request data to body of the request by clicking body tab and selecting raw and JSON as seen below.

Attaching a JSON body to POST request in Postman

Now click Send button to send POST request to the Web API. Then you must get “201 Created” http response code and seeing created item in the response body.

To see the last status of todo items send a get request to localhost:3000/itemsaddress. You must see newly created item at the end of the list.

Sending PUT Request

Sending PUT request is very similar to sending POST request.

The most obvious difference is request url should be pointed specific item like this localhost:3000/items/3

And you should choose PUT as http verb instead of POST and send all of the required data in the request body unlike POST.

For example you could send a JSON body in the PUT request as below.

An example JSON body for PUT request

{
    "title": "New title of todo item",
    "order": 3,
    "completed": false
}

When you click Send button you must get “204 No Content” http response code. You can check item you updated by sending a get request.

Sending DELETE Request

To send a DELETE request, change the request url to address a specific item id like this localhost:3000/items/3

And select DELETE as http verb and click Send button.

You must get “204 No Content” http response code as result of the DELETE operation.

Send a get request and see the last status of list.

About the DELETE Http Request

I want to say a few words about DELETE http request. You must have noticed something in our delete code. DELETE request returns “204 No Content” every situation.

Http DELETE requests are idempotent. So what that mean? If you delete a resource on server by sending DELETE request, it’s removed from the collection. And every next DELETE request on the same resource won’t change outcome. So you won’t get “404 Not Found” in the second request. Each request returns same response whether succeed or not. That’s mean idempotent operation.

Conclusion

Finally we’ve tested all http methods of our Web API.

As you can see, it works just fine.

Thanks for reading ❤

If you liked this post, share it with all of your programming buddies!