Building MongoDB Dashboard using Node.js

Building MongoDB Dashboard using Node.js

For a pretty decent amount of time, doing analytics with MongoDB required additional overhead compared to modern SQL RDBMS and Data Warehouses associated with aggregation pipeline and MapReduce practices.

For a pretty decent amount of time, doing analytics with MongoDB required additional overhead compared to modern SQL RDBMS and Data Warehouses associated with aggregation pipeline and MapReduce practices.

While this approach allowed one to craft advanced custom tailored aggregation algorithms it required additional knowledge of how to build and maintain it.

To fill this gap, MongoDB released the MongoDB connector for BI, which acts as a MySQL server on top of your MongoDB data. Under the hood it bridges existing aggregation mechanisms to the MySQL protocol, allowing standard MySQL clients to connect and issue SQL queries.

In this short 30-minutes tutorial we’ll setup the MongoDB connector for your local MongoDB instance and spin up a Node.js application that provides an analytics API, query cache, and orchestration using the Cube.js analytics framework. Cube.js can be used as a standalone server or it can be embedded as part of an existing Node.js web application. You can learn more about it here.

Setting up the MongoDB connector for BI

To install the MongoDB connector for BI locally you can use either quickstart guide or one of the platform dependent installation guides.

Please make sure you use the MongoDB version that supports the MongoDB connector for BI. In this tutorial we use 4.0.5.

If you don’t have a local MongoDB instance please download it here. The BI Connector can be downloaded here.

After the BI connector has been installed please start a mongod instance first. If you use the downloaded installation it can be started from its home directory like so:

1 $ bin/mongod

The BI connector itself can be started the same way:

1 $ bin/mongosqld 

Please note that mongosqld resides in another bin directory. If everything works correctly, you should see a success log message in your shell for the mongosqld process:

1 [initandlisten] waiting for connections at 127.0.0.1:3307

If you’re using the MongoDB Atlas you can use this guide to enable BI connector.

Importing test dataset

You can skip this step if you already have data in your DB that can be analyzed. Otherwise you can use the zip code test dataset from MongoDB to populate your DB with some test data.

Download zips.json and run mongoimport from the MongoDB home directory:

1 $ bin/mongoimport --db test --collection zips --file <path/to/zips.json>

Please make sure to restart the MongoDB BI connector instance in order to generate an up-to-date MySQL schema from the just added collection.

Spinning up a Cube.js application

We’ll use Cube.js to make analytic queries to our MongoDB instance. To install its CLI run:

1 $ npm install -g cubejs-cli

To create a new Cube.js application with MySQL driver run:

1 $ cubejs create mongo-tutorial -d mysql

Go to the just created mongo-tutorial directory and edit the just created .env file: replace placeholders with your MongoDB BI connector credentials.

By default it should be something like:

1 CUBEJS_DB_HOST=localhost
2 CUBEJS_DB_NAME=test
3 CUBEJS_DB_PORT=3307
4 CUBEJS_DB_TYPE=mysql
5 CUBEJS_API_SECRET=941ed7ad8a49bec1b3f87a9f48bff2a5e549e946fc22fcc5f18c3a17bf62c64ed3398e99b271cd66d4521a6fd7caa4bfd268dfccea59ffd9c67de660f3967338

Now generate Cube.js schema files for the zips collection from the test data set or for your own collection:

1 $ cd mongo-tutorial
2 $ cubejs generate -t zips

In order to start a Cube.js dev server you’ll also need a locally running Redis instance, which is used for cache and query queue orchestration. You can download it and run it using these instructions.

If everything went smoothly, you’re able to run the Cube.js dev server:

1 $ npm run dev

If the server started successfully, you can now open [http://localhost:4000](http://localhost:4000 "http://localhost:4000") and navigate to the Cube.js dev environment. There you should see a working example of a pie chart.

Building a Dashboard

The Cube.js dev environment example contains all essential client pieces to build an analytics dashboard. Let’s modify it a little bit so it looks more like a dashboard and uses the zips collection.

Replace the contents of index.js in your dev environment code sandbox:

1 import React from "react";
2 import ReactDOM from "react-dom";
3
4 import "antd/dist/antd.css";
5 import { Row, Col, Card, Layout, Spin } from "antd";
6
7 import cubejs from "@cubejs-client/core";
8 import { QueryRenderer } from "@cubejs-client/react";
9 import { Chart, Axis, Tooltip, Geom, Coord, Legend } from "bizcharts";
10
11 const renderChart = resultSet => (
12   <Chart height={400} data={resultSet.chartPivot()} forceFit>
13     <Coord type="theta" radius={0.75} />
14     <Axis name="Zips.count" />
15     <Legend position="bottom" name="category" />
16     <Tooltip showTitle={false} />
17     <Geom type="intervalStack" position="Zips.count" color="x" />
18   </Chart>
19 );
20
21 const cubejsApi = cubejs(
22   "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1NDkzMDk3NzMsImV4cCI6MTU0OTM5NjE3M30.eXEdfUa_ek2V9MlGTpBMOd_AFfs8laaZj8ZsuM1wqqo",
23   { apiUrl: "http://localhost:4000/cubejs-api/v1" }
24
25
26 const { Header, Footer, Sider, Content } = Layout;
27
28 const App = () => (
29   <Layout>
30     <Header>
31       <h2 style={{ color: "#fff" }}>MongoDB Dashboard</h2>
32     </Header>
33     <Content style={{ padding: "25px", margin: "25px" }}>
34       <Row type="flex" justify="space-around" align="top" gutter={24}>
35         <Col span={24} lg={12}>
36           <Card title="Zip count by state" style={{ marginBottom: "24px" }}>
37             <QueryRenderer
38               query={{ measures: ["Zips.count"], dimensions: ["Zips.state"] }}
39               cubejsApi={cubejsApi}
40               render={({ resultSet, error }) =>
41                (resultSet && renderChart(resultSet)) ||
42 (error && error.toString()) || <Spin />
43               }
44               />
45           </Card>
46         </Col>
47       </Row>
48     </Content>
49   </Layout>
50 );
51 const rootElement = document.getElementById("root");
52 ReactDOM.render(<App />, rootElement);
53

Please make sure to replace the auth token with your own Cube.js temporary token that’s printed in the console while the server starts. Read more about auth tokens here

If everything worked well, you should see the following dashboard:

Why Cube.js?

So why is using Cube.js better than hitting SQL queries to MongoDB directly? Cube.js solves a plethora of different problems every production-ready analytic application needs to solve: analytic SQL generation, query results caching and execution orchestration, data pre-aggregation, security, API for query results fetch, and visualization.

These features allow you to build production grade analytics applications that are able to handle thousands of concurrent users and billions of data points. They also allow you to do analytics on a production MongoDB read replica or even MongoDB main node due to their ability to reduce the amount of actual queries issued to a MongoDB instance. Cube.js schemas also allow you to model everything from simple counts to funnels and cohort retention analysis. You can learn more about it here.

Performance considerations

In order to be able to handle a massive amount of data, Cube.js heavily relies on pre-aggregations. As of now, the MongoDB BI Connector doesn’t support Create Table as Statement, which is required to materialize query results right in your database and create pre-aggregations. If you need to analyze well over 100M of data points in MongoDB please consider using Presto with the MongoDB Connector, which is also supported by Cube.js.

Originally published by Pavel Tiunov at *[https://statsbot.co](https://statsbot.co "https://statsbot.co*")

Learn More

☞ The Complete Developers Guide to MongoDB

☞ Master MongoDB, the NOSQL leader with Clarity and Confidence

☞ MongoDB, NoSQL & Node: Mongoose, Azure & Database Management

☞ Build a ChatApp with: (Nodejs,Socketio, Expressjs ,MongoDB)

☞ GraphQL: Learning GraphQL with Node.Js

mongodb node-js

What's new in Bootstrap 5 and when Bootstrap 5 release date?

How to Build Progressive Web Apps (PWA) using Angular 9

What is new features in Javascript ES2020 ECMAScript 2020

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Random Password Generator Online

HTML Color Picker online | HEX Color Picker | RGB Color Picker

How to Use Express.js, Node.js and MongoDB.js

In this post, I will show you how to use Express.js, Node.js and MongoDB.js. We will be creating a very simple Node application, that will allow users to input data that they want to store in a MongoDB database. It will also show all items that have been entered into the database.

Build a REST API using Node.js, Express.js, Mongoose.js and MongoDB

Node.js, Express.js, Mongoose.js, and MongoDB is a great combination for building easy and fast REST API. You will see how fast that combination than other existing frameworks because of Node.js is a packaged compilation of Google’s V8 JavaScript engine and it works on non-blocking and event-driven I/O. Express.js is a Javascript web server that has a complete function of web development including REST API.

Node.js, ExpressJs, MongoDB and Vue.js (MEVN Stack) Application Tutorial

In this tutorial, you'll learn how to integrate Vue.js with Node.js backend (using Express framework) and MongoDB and how to build application with Node.js, ExpressJs, MongoDB and Vue.js

Hands on with Node.Js Streams | Examples & Approach

The practical implications of having Streams in Node.js are vast. Nodejs Streams are a great way to handle data chunks and uncomplicate development.

MongoDB, Express, Vue.js 2, Node.js (MEVN) and SocketIO Chat App

The comprehensive tutorial on MongoDB, Express, Vue.js 2, Node.js (MEVN) and SocketIO Chat Application