How to Use MongoDB as a Real-time database with Node.js

How to Use MongoDB as a Real-time database with Node.js

Have you ever run into a scenario where you wanted to update your UI as soon as there is some change to your Mongo database

Have you ever run into a scenario where you wanted to update your UI as soon as there is some change to your Mongo database ? For example, A new user gets added and you wanted that change to reflect in you UI without having to make an API call or constantly polling for changes. If so, then this post is for you.

While the go to would be to use real time databases like firebase or RethinkDB, you actually can achieve this using your MongoDB too and it is pretty simple. MongoDB has something called [change streams](https://docs.mongodb.com/manual/changeStreams/) that allows you to listen to your database.

The method I’m going to show uses [replicaSets](https://docs.mongodb.com/manual/replication/) , you can also achieve the same using [sharded clusters](https://docs.mongodb.com/manual/sharding/). (click those links to read more about replicasets and sharded clusters)

  • So, the first step would be to convert your stand alone MongoDB to replica sets. This is because [change streams](https://docs.mongodb.com/manual/changeStreams/) is not available with standalone MongoDB (Honestly I was about to give up on seeing that, but trust me it’s simple).
$ mongod --port 27017 --replSet rs0

PS: Stop your current standalone mongo and then run the above command.

The above command starts your mongo instance as a [replicaSet](https://docs.mongodb.com/manual/replication/) named as rs0.

  • Now with your mongo running as [replicaSet](https://docs.mongodb.com/manual/replication/), before creating a new database we must initiate our replication set. To do so, open a new window and do the following
$ mongo
$ rs.initiate() 
  • Now let’s go and create our database. I use Robo 3T to manage my databases, you can create it in terminal using $use . I’m creating a db called UserDB.

Create a new Database

  • Now our DB part is set. Scaffold an Express application and install Mongoose. My User model looks like this,
var mongoose = require("mongoose");
var Schema = mongoose.Schema;

var UserSchema = new Schema({
  name: {
    type: String,
    required: [true, 'Need User name']
  }
});

var User = mongoose.model('User', UserSchema);

module.exports = User;

User model

  • Now assume I have two users inside my User collection of UserDB. Now I want to subscribe to my UserDB for any changes on User collection. In your app.js code, the mongoDB connection string should look like this,
require('./models/connection').connectMongoDB('mongodb://localhost/UserDB?replicaSet=rs0');

I’m passing my dbURL to my connection module. (we need to explicitly mention the replicaSet).

const socket_io = require('socket.io');
var io = socket_io();
const User = require('../models/User');

const changeStream = User.watch();

changeStream.on('change', (change) => {
    console.log(change); // You could parse out the needed info and send only that data. 
    io.emit('changeData', change);
}); 

io.on('connection', function () {
    console.log('connected');
});

var socket = io;
module.exports = socket;

I listen to the changes in User collection using the watch() method, and whenever there is any change, the change event listener provided by changeStreams gets fired. It accepts a callback function that receives the changed data as parameter.

We can go ahead and test it to see if it works. Run an update operation onUser collection and at the same time in another tab watch for the console.log messages of your app. You can see the changes real time. At this place, We can now forward this to the front end using socket.io

You’re mongoDB is now acting as a real time DB

_Thank you for reading. If you enjoyed this article, please share it with others who may enjoy it as well.!

How to connect to MongoDB database from Node.js?

How to connect to MongoDB database from Node.js?

Use Node.js? Want to learn MongoDB? In today’s post, we’ll work through connecting to a MongoDB database from a Node.js script, retrieving a list of databases, and printing the results to your console.

In today’s post, we’ll work through connecting to a MongoDB database from a Node.js script, retrieving a list of databases, and printing the results to your console.

Set up

Before we begin, we need to ensure you’ve completed a few prerequisite steps.

Install Node.js

First, make sure you have a supported version of Node.js installed (the MongoDB Node.js Driver requires Node 4.x or greater and for these examples, I've used Node.js 10.16.3).

Install the MongoDB Node.js Driver

The MongoDB Node.js Driver allows you to easily interact with MongoDB databases from within Node.js applications. You’ll need the driver in order to connect to your database and execute the queries described in this Quick Start series.

If you don’t have the MongoDB Node.js Driver installed, you can install it with the following command.

npm install mongodb

At the time of writing, this installed version 3.3.2 of the driver. Running npm list mongodb will display the currently installed driver version number. For more details on the driver and installation, see the official documentation.

Create a free MongoDB Atlas cluster and load the sample data

Next, you’ll need a MongoDB database. Your database will be stored inside of a cluster. At a high level, a cluster is a set of nodes where copies of your database will be stored.

The easiest way to get started with MongoDB is to use Atlas, MongoDB’s fully-managed database-as-a-service. Head over to Atlas and create a new cluster in the free tier. Once your tier is created, load the sample data.

Get started with an M0 cluster on Atlas today. It's free forever, and it’s the easiest way to try out the steps in this blog series.

If you’re not familiar with how to create a new cluster and load the sample data, check out this video tutorial from MongoDB Developer Advocate Maxime Beugnet.

Get your cluster’s connection info

The final step is to prep your cluster for connection.

In Atlas, navigate to your cluster and click CONNECT. The Cluster Connection Wizard will appear.

The Wizard will prompt you to whitelist your current IP address and create a MongoDB user if you haven’t already done so. Be sure to note the username and password you use for the new MongoDB user as you’ll need them in a later step.

Next, the Wizard will prompt you to choose a connection method. Select Connect Your Application. When the Wizard prompts you to select your driver version, select Node.js and 3.0 or later. Copy the provided connection string.

For more details on how to access the Connection Wizard and complete the steps described above, see the official documentation.

Connect to your database from a Node.js application

Now that everything is set up, it’s time to code! Let’s write a Node.js script that connects to your database and lists the databases in your cluster.

Import MongoClient

The MongoDB module exports MongoClient, and that’s what we’ll use to connect to a MongoDB database. We can use an instance of MongoClient to connect to a cluster, access the database in that cluster, and close the connection to that cluster.

const {MongoClient} = require('mongodb');

Create our main function

Let’s create an asynchronous function named main() where we will connect to our MongoDB cluster, call functions that query our database, and disconnect from our cluster.

The first thing we need to do inside of main() is create a constant for our connection URI. The connection URI is the connection string you copied in Atlas in the previous section. When you paste the connection string, don’t forget to update <username> and <password> to be the credentials for the user you created in the previous section. Note: the username and password you provide in the connection string are NOT the same as your Atlas credentials.

/**
 * Connection URI. Update <username>, <password>, and <your-cluster-url> to reflect your cluster.
 * See https://docs.mongodb.com/ecosystem/drivers/node/ for more details
 */
const uri = "mongodb+srv://<username>:<password>@<your-cluster-url>/test?retryWrites=true&w=majority";

Now that we have our URI, we can create an instance of MongoClient.

const client = new MongoClient(uri);

Note: When you run this code, you may see DeprecationWarnings around the URL string parser and the Server Discover and Monitoring engine. If you see these warnings, you can remove them by passing options to the MongoClient. For example, you could instantiate MongoClient by calling new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true }). See the Node.js MongoDB Driver API documentation for more information on these options.

Now we’re ready to use MongoClient to connect to our cluster. client.connect() will return a promise. We will use the await keyword when we call client.connect() to indicate that we should block further execution until that operation has completed.

await client.connect();

Now we are ready to interact with our database. Let’s build a function that prints the names of the databases in this cluster. It’s often useful to contain this logic in well named functions in order to improve the readability of your codebase. Throughout this series, we’ll create new functions similar to the function we’re creating here as we learn how to write different types of queries. For now, let’s call a function named listDatabases().

await listDatabases(client);

Let’s wrap our calls to functions that interact with the database in a try/catch statement so that we handle any unexpected errors.

try {
    await client.connect();

    await listDatabases(client);

} catch (e) {
    console.error(e);
}

We want to be sure we close the connection to our cluster, so we’ll end our try/catch with a finally statement.

finally {
    await client.close();
}

Once we have our main() function written, we need to call it. Let’s send the errors to the console.

main().catch(console.err);

Putting it all together, our main() function and our call to it will look something like the following.

async function main(){
    /**
     * Connection URI. Update <username>, <password>, and <your-cluster-url> to reflect your cluster.
     * See https://docs.mongodb.com/ecosystem/drivers/node/ for more details
     */
    const uri = "mongodb+srv://<username>:<password>@<your-cluster-url>/test?retryWrites=true&w=majority";

    const client = new MongoClient(uri);

    try {
        // Connect to the MongoDB cluster
        await client.connect();

        // Make the appropriate DB calls
        await  listDatabases(client);

    } catch (e) {
        console.error(e);
    } finally {
        await client.close();
    }
}

main().catch(console.err);

List the databases in our cluster

In the previous section, we referenced the listDatabases() function. Let’s implement it!

This function will retrieve a list of databases in our cluster and print the results in the console.

async function listDatabases(client){
    databasesList = await client.db().admin().listDatabases();

    console.log("Databases:");
    databasesList.databases.forEach(db => console.log(` - ${db.name}`));
};

Save Your File

You’ve been implementing a lot of code. Save your changes, and name your file something like connection.js. To see a copy of the complete file, visit the nodejs-quickstart GitHub repo.

Execute Your Node.js Script

Now you’re ready to test your code! Execute your script by running a command like the following in your terminal: node connection.js

You will see output like the following:

Databases:
 - sample_airbnb
 - sample_geospatial
 - sample_mflix
 - sample_supplies
 - sample_training
 - sample_weatherdata
 - admin
 - local

What’s next?

Today, you were able to connect to a MongoDB database from a Node.js script, retrieve a list of databases in your cluster, and view the results in your console. Nice!

In future posts in this series, we’ll dive into each of the CRUD (create, read, update, and delete) operations as well as topics like change streams, transactions, and the aggregation pipeline, so you’ll have the tools you need to successfully interact with data in your databases.

In the meantime, check out the following resources:

Series versions

This examples in this article were created with the following application versions:

MongoDB: 4.0
MongoDB Node.js Driver: 3.3.2
Node.js: 10.16.3

Creating Real-time Chat App with Nodejs, Express, Socket and MongoDB

Creating Real-time Chat App with Nodejs, Express, Socket and MongoDB

In this tutorial, we’ll be building a real-time chat application with NodeJS, Express, Socket.io, and MongoDB.

How to Build a Real-time Chat App With NodeJS, Socket.IO, and MongoDB

In this post, we’ll be building a real-time chat application with NodeJS, Express, Socket.io, and MongoDB.

Here is a screenshot of what we’ll build:

Setup

I’ll assume that you already have NodeJS and NPM installed. You can install it from the Node.js website if you don’t have it installed already.

A basic Knowledge of Javascript is required.

Let’s get started.

Create a directory for the application, open the directory with your favourite editor such as Visual Studio Code. You can use any other editor, I’ll be using VS code in this tutorial:

mkdir chatApplication && cd chatApplication && code .

Next, let’s initialize the directory as a Nodejs application.

npm init

You’ll be prompted to fill in some information — that’s okay. The information will be used to set up your package.json file.

Dependencies Installation

Let’s install our application’s dependencies.

We’ll be using the express web server to serve our static files and body-parserextract the entire body portion of an incoming request stream and exposes it to an API endpoint. So, let's install them. You'll see how they are used later in this tutorial.

npm install express body-parser --save

We added the — save flag so that it’ll be added as a dependency in our package.json file.

Note:

Please, don’t use express generator as I won’t cover how to configure _socket.io_ to work with express generator setup.

Next, install the mongoose node module. It is an ODM (Object Document Mapper) for MongoDB and it’ll make our job a lot easier.

Let’s install it alongside socket.io and bluebird. Socket.IO is a JavaScript library for real-time web applications. Bluebird is a fully-featured Promise library for JavaScript.

npm install mongoose socket.io bluebird --save

That’s it for the Nodejs backend module installation.

Our package.json file should look like this now.

Another way to install the above packages is to copy the package.json file above and paste it into your package.json file and run:

npm install

It’ll install all the required packages.

Let’s set up the client side.

To connect Socket.IO server to the client we add the Socket.IO client-side javascript library.

<script  src="/js/socket.js"></script>

That will be our HTML file for the frontend. You can grab the entire code for the frontend here to follow along. The best way to learn is to follow along.

You can download the client-side socket.io library here as well.

And here /js/chat.js is where we’ll have our custom client-side javascript code.

Setting up our express server:

Create an App.js. You can call it server.js if you like.
It’s my personal preference to call it App.js.

Inside the App.js file let’s create and configure the express server to work with socket.io.

App.js

This is the basic configuration required to set up socket.io in the backend.

Socket.IO works by adding event listeners to an instance of http.Server
which is what we are doing here:

const socket = io(http);

Here is where we listen to new connection events:

socket.on(“connection”, (socket)=>{
console.log(“user connected”);
});

For example, if a new user visits localhost:500 the message “user connected” will be printed on the console.

socket.on() takes an event name and a callback as parameters.

And there is also a special disconnect event that gets fire each time a user closes the tab.

socket.on(“connection”, (socket)=>{
    console.log(“user connected”);
    socket.on("disconnect", ()=>{
    console.log("Disconnected")
})
});
Setting up our frontend code

Open up your js/chat.js file and type the following code:

(function() {
    var  socket  =  io();
    $("form").submit(function(e) {
        e.preventDefault(); // prevents page reloading
        socket.emit("chat message", $("#m").val());
        $("#m").val("");
    return  true;
});
})();

This is a self-executing function it initializes socket.io on the client side and emits the message typed into the input box.

With this line of code, we create a global instance of the soicket.io client on the frontend.

var  socket  =  io();

And inside the submit event handler, socket io is getting our chat from the text box and emitting it to the server.

$("form").submit(function(e) {
    e.preventDefault(); // prevents page reloading
    socket.emit("chat message", $("#m").val());
    $("#m").val("");
 return  true;
});

If you’ve gotten to this point, congratulations, you deserve some accolades.

Great, we have both our express and socket.io server set up to work well. In fact, we’ve been able to send messages to the server by emitting the message from our input box.

socket.emit("chat message", $("#m").val());

Now from the server-side let’s set up an event to listen to the “chat message” event and broadcast it to clients connected on port 500.

App.js

socket.on("chat message", function(msg) {

console.log("message: "  +  msg);

//broadcast message to everyone in port:5000 except yourself.

socket.broadcast.emit("received", { message: msg  });
   });
});

This is the event handler that listens to the “chat message” event and the message received is in the parameter passed to the callback function.

socket.on("chat message", function(msg){
});

Inside this event, we can choose what we do with the message from the client — -insert it into the database, send it back to the client, etc.

In our case, we’ll be saving it into the database and also sending it to the client.

We’ll broadcast it. That means the server will send it to every other person connected to the server apart from the sender.

So, if Mr A sends the message to the server and the server broadcasts it, Mr B, C, D, etc will receive it but Mr A won’t.

We don’t want to receive a message we sent, do we?

That doesn’t mean we can’t receive a message we sent as well. If we remove the broadcast flag we’ll also remove the message.

Here is how to broadcast an event:

socket.broadcast.emit("received",{message:msg})

With that out of the way, we can take the message received and append it to our UI.

If you run your application. You should see something similar to this. Please, don’t laugh at my live chat.

Wawu! Congratulations once again. let’s add some database stuff and display our chats on the frontend.

Database Setup

Install MongoDB

Visit the MongoDB website to download it if you have not done so already.

And make sure your MongoDB server is running. They have an excellent documentation that details how to go about setting it up and to get it up and running. You can find the doc here.

Create Chat Schema

Create a file in the model’s directory called models/ChatSchema.js
Nothing complex, we are just going to have 3 fields in our schema --- a message field, a sender field and a timestamp.

The ChatSchema.js file should look like this:

Connection to the MongoDB database

Create a file and name it dbconnection.js. That's where our database connection will live.

const  mongoose  = require("mongoose");
mongoose.Promise  = require("bluebird");

const  url  =  "mongodb://localhost:27017/chat";
const  connect  =  mongoose.connect(url, { useNewUrlParser: true  });

module.exports  =  connect;
Insert messages into the database

Since we are going to insert the messages in the server-side we’ll be inserting the messages we receive from the frontend in the App.js file.

So, let’s update the App.js file.

...
//database connection

const  Chat  = require("./models/Chat");
const  connect  = require("./dbconnect");

We can now add the

//setup event listener
socket.on("connection", socket  =>  {
    console.log("user connected");
    socket.on("disconnect", function() {
		
    console.log("user disconnected");
    });  
    socket.on("chat message", function(msg) {
        console.log("message: "  +  msg);
        //broadcast message to everyone in port:5000 except yourself.
    socket.broadcast.emit("received", { message: msg  });
		
    //save chat to the database
    connect.then(db  =>  {
    console.log("connected correctly to the server");
		
    let  chatMessage  =  new Chat({ message: msg, sender: "Anonymous"});
    chatMessage.save();
    });
    });
});

We are creating a new document and saving it into the Chat collection in the database.

let  chatMessage  =  new Chat({ message: msg, sender: "Anonymous"});
    chatMessage.save();
Display messages on the frontend

We’ll, first of all, display our message history from the database and append all messages emitted by events.

To achieve this, we need to create an API that sends the data from the database to the client-side when we send a get request.

const  express  = require("express");
const  connectdb  = require("./../dbconnect");
const  Chats  = require("./../models/Chat");

const  router  =  express.Router();

router.route("/").get((req, res, next) =>  {
        res.setHeader("Content-Type", "application/json");
        res.statusCode  =  200;
        connectdb.then(db  =>  {
            Chats.find({}).then(chat  =>  {
            res.json(chat);
        });
    });
});

module.exports  =  router;

In the above code, we query the database and fetch all the messages in the Chat collection.

We’ll import this into the server code App.js file and we'll also import the bodyparser middleware as well.


const bodyParser = require(“body-parser”);
const chatRouter = require(“./route/chatroute”);

//bodyparser middleware
app.use(bodyParser.json());

//routes
app.use(“/chats”, chatRouter);

With this out of the way, we are set to access our API from the frontend and get all the messages in our Chat collection.

So, we got the messages using the fetch API and we appended the messages to the UI.

You’ll also notice that I used formatTimeAgo(data.createdAt)); that is a 1.31kb library I created to manage dates for small projects since moment.js sometimes is rather too big. formatTimeAgo() will display “few seconds ago”, etc. If you are interested, you can find more information here.

Everything seems good at this point, right?

However, since you are not receiving the messages sent to the server back to yourself, let’s grab our own message from our input box and display it on the UI.


(function() {
$(“form”).submit(function(e) {

let li = document.createElement(“li”);

e.preventDefault(); // prevents page reloading

socket.emit(“chat message”, $(“#message”).val());

messages.appendChild(li).append($(“#message”).val());

let span = document.createElement(“span”);

messages.appendChild(span).append(“by “ + “Anonymous” + “: “ + “just now”);

$(“#message”).val(“”);

return false;

  });
})();

And also if we receive messages from the event let’s also output it to the UI.



(function(){

socket.on("received", data => {

let li = document.createElement("li");

let span = document.createElement("span");

var messages = document.getElementById("messages");

messages.appendChild(li).append(data.message);

messages.appendChild(span).append("by " + "anonymous" + ": " + "just now");

});
`
})

Our application is complete now. Go ahead an test it.

Note that if we had our users logged in we wouldn’t have hardcoded the “anonymous” user as it’s in our code right now. We’ll get it from the server.

And also if you want to tell everyone that someone is typing you can also add this code in the frontend.


//isTyping event
messageInput.addEventListener(“keypress”, () => {
socket.emit(“typing”, { user: “Someone”, message: “is typing…” });
});

socket.on(“notifyTyping”, data => {
typing.innerText = data.user + “ “ + data.message;
console.log(data.user + data.message);
});

//stop typing
messageInput.addEventListener(“keyup”, () => {
socket.emit(“stopTyping”, “”);
});

socket.on(“notifyStopTyping”, () => {
typing.innerText = “”;});
`

What it does is that when a user is typing it emits an event to the server and the server broadcasts it to other clients. You listen to the event and update the UI with the message “Someone is typing…” You can add the person’s name if you wish.

Here is the server-side event listener and emitter:

Congratulations.

You can improve this code, add authentication, add groups or make it a one to one chat, re-model the schema to accommodate all of that, etc.

I’ll be super excited to see the real-time applications you’ll build with socket.IO.

I hope this was helpful. The entire code is on Github. You can get it here.

Thank you for reading !

The Missing $some Operator in MongoDB

The Missing $some Operator in MongoDB

There is a void between $in and $all where a new array query operator would come in handy. In MongoDB, querying documents where the value of a field is an array can be straight forward. Using the $in comparison operator on an array field will give you documents where the field’s array holds at least one of your provided values.

There is a void between $in and $all where a new array query operator would come in handy

In MongoDB, querying documents where the value of a field is an array can be straight forward. Using the $in comparison operator on an array field will give you documents where the field’s array holds at least one of your provided values.

You also have the array query operator $all that works the same way, but where the array needs to hold all of your provided values.

These two operators cover a lot of ground, but when you want a query for documents where the array holds at least four values, or exactly eight, then MongoDB is missing a useful operator.

Let’s take a simple example. You have a collection of customers, and each customer document has a purchasedProducts field, which is an array of ObjectIds (references to all products the customer has purchased).

Now, for a marketing campaign, you want to find all your customers who have purchased at least three out of a set of ten selected products, to send a message to these customers.

In this scenario, the $in operator won’t do because it would give you all customers who have purchased at least one of the products. The $all operator won’t work either, because it would only give you the customers who have purchased all ten products in the set.

In this scenario, a $some operator would come in handy.

The $some operator could take an array with two items; the first item being an array of values to match against and the second item being an object declaring the condition for a match.

/*
 * Example of a $some operator where the document would match if purchasedDocuments
 * contain at least two of the three provided ObjectIds
 */
db.collection.find({
  purchasedProducts: {
    $some: [[ObjectId('ID1'), ObjectId('ID2'), ObjectId('ID3')], { $gte: 2 }]
  }
})

While the $some operator doesn’t exist, it is still possible to achieve the result. Using an aggregation pipeline, we get access to the tools we need to create the equivalent of a $some operator.

In theory, what the $some operator would do is that it would create an intersection between an array field on the document and the provided array of values to match against.

The size (array length) of this intersection would then be compared against the provided condition, and if the size is at least two (in the example above), then the document is a match.

To transform this into an aggregation pipeline, we will need four stages:

  1. Narrow down the number of documents to those that match at least one of the provided values.
  2. Create an intersection between the array field on the document and the provided array of values. Get the size of the intersection and compare it with your match condition, and save the result as a temporary field on the document.
  3. Based on the temporary field, filter out any document that didn’t match.
  4. Clean up the documents by removing the temporary field from each document
/*
 * Workaround using aggregation pipeline stages to solve what a
 * $some operator could have done.
 *
 * Inspired by this SO answer: https://stackoverflow.com/a/28161130/678801
 */

const purchasedProducts = [[ObjectId('ID1'), ObjectId('ID2'), ObjectId('ID3')];

db.collection.aggregate([
  // Only include those that match at least one of the values
  // to narrow down the result 
  {
    $match: {
      purchasedProducts: { $in: purchasedProducts },
    },
  },

  // Add a temporary field that holds a boolean value of whether the
  // document matches the conditions
  {
    $addFields: {
      matched: {
        $gte: [
          {
            $size: {
              $setIntersection: ['$purchasedProducts', purchasedProducts]
            }
          },
          2
        ]
      }
    }
  },

  // Filter out anything that did not match
  { $match: { matched: true } },

  // Remove the temporary field from all documents
  { $project: { matched: 0 } }
])

With the_ $expr operator, it is actually possible to make this both shorter and available in the regular collection methods. 👍

/*
 * Example of a $some operator workaround using $expr. The document would match if
 * purchasedProducts contain at least two of the three provided ObjectIds
 */

const purchasedProducts = [[ObjectId('ID1'), ObjectId('ID2'), ObjectId('ID3')]

db.collection.find({
  purchasedProducts: { $in: purchasedProducts },
  $expr: {
    $gt: [
      { $size: { $setIntersection: [‘$purchasedProducts’, purchasedProducts] } },
      2
    ]
  }
})

Even though it is possible to work around the lack of a $some operator, it gets a bit verbose, and a $some operator would make a nice addition to the $in and $all operators.

Let me know what your thoughts are on a $some operator and if you happen to have contacts on the MongoDB dev team, feel free to share the article with them as a feature request.

Thank you and see you!