Build a live analytics dashboard using Go and MongoDB

One of the most important step to take while taking a website or app into production is analytics and usage statistics. This is important as it allows you to see how users are actually using your app, improve usability and inform future development decisions.

In this tutorial, I will describe how to monitor all requests an application is going to receive, we will use the data gotten from monitoring to track a few metrics such as:

Most visited links
Response time for each link
Total number of requests
Average response time

Prerequisites

Golang >=1.7. You can install the Golang toolchain by following this guide.
MongoDB >=3. You can install this by following this guide.
A Pusher account.

Starting out

We will start out by setting up our project directory. You will need to create a directory called analytics-dashboard. The location of this directory will depend on the version of the Go toolchain you have:

If you are running <=1.11, you should create the directory in $GOPATH/src/github.com/pusher-tutorials/analytics-dashboard
If you are running 1.12 or greater, you can create the directory anywhere.

In the newly created directory, create a .env in the root directory with the following command:

    $ touch .env

In the .env file, you will need to add your credentials. Copy and paste the following contents into the file:

    // analytics-dashboard/.env
    PUSHER_APP_ID=PUSHER_APP_ID
    PUSHER_APP_KEY=PUSHER_APP_KEY
    PUSHER_APP_SECRET=PUSHER_APP_SECRET
    PUSHER_APP_CLUSTER=PUSHER_APP_CLUSTER
    PUSHER_APP_SECURE="1"

Please make sure to replace the placeholders with your own credentials.## MongoDB

MongoDB is going to be used as a persistent datastore and we are going to make use of it’s calculation abilities to build out the functionality I described above.

Since we are building the application in Golang, we will need to fetch a client library that will assist us in connecting and querying the MongoDB database. To that, you should run the following command:

    $ go get -u -v gopkg.in/mgo.v2/...

Once the above command succeeds, you will need to create a new file called analytics.go. In this file, paste the following code:

    // analytics-dashboard/analytics.go

    package main

    import (
            "gopkg.in/mgo.v2"
            "gopkg.in/mgo.v2/bson"
    )

    const (
            collectionName = "request_analytics"
    )

    type requestAnalytics struct {
            URL         string `json:"url"`
            Method      string `json:"method"`
            RequestTime int64  `json:"request_time"`
            Day         string `json:"day"`
            Hour        int    `json:"hour"`
    }

    type mongo struct {
            sess *mgo.Session
    }

    func (m mongo) Close() error {
            m.sess.Close()
            return nil
    }

    func (m mongo) Write(r requestAnalytics) error {
            return m.sess.DB("pusher_tutorial").C(collectionName).Insert(r)
    }

    func (m mongo) Count() (int, error) {
            return m.sess.DB("pusher_tutorial").C(collectionName).Count()
    }

    type statsPerRoute struct {
            ID struct {
                    Method string `bson:"method" json:"method"`
                    URL    string `bson:"url" json:"url"`
            } `bson:"_id" json:"id"`
            NumberOfRequests int `bson:"numberOfRequests" json:"number_of_requests"`
    }

    func (m mongo) AverageResponseTime() (float64, error) {

            type res struct {
                    AverageResponseTime float64 `bson:"averageResponseTime" json:"average_response_time"`
            }

            var ret = []res{}

            var baseMatch = bson.M{
                    "$group": bson.M{
                            "_id":                 nil,
                            "averageResponseTime": bson.M{"$avg": "$requesttime"},
                    },
            }

            err := m.sess.DB("pusher_tutorial").C(collectionName).
                    Pipe([]bson.M{baseMatch}).All(&ret)

            if len(ret) > 0 {
                    return ret[0].AverageResponseTime, err
            }

            return 0, nil
    }

    func (m mongo) StatsPerRoute() ([]statsPerRoute, error) {

            var ret []statsPerRoute

            var baseMatch = bson.M{
                    "$group": bson.M{
                            "_id":              bson.M{"url": "$url", "method": "$method"},
                            "responseTime":     bson.M{"$avg": "$requesttime"},
                            "numberOfRequests": bson.M{"$sum": 1},
                    },
            }

            err := m.sess.DB("pusher_tutorial").C(collectionName).
                    Pipe([]bson.M{baseMatch}).All(&ret)
            return ret, err
    }

    type requestsPerDay struct {
            ID               string `bson:"_id" json:"id"`
            NumberOfRequests int    `bson:"numberOfRequests" json:"number_of_requests"`
    }

    func (m mongo) RequestsPerHour() ([]requestsPerDay, error) {

            var ret []requestsPerDay

            var baseMatch = bson.M{
                    "$group": bson.M{
                            "_id":              "$hour",
                            "numberOfRequests": bson.M{"$sum": 1},
                    },
            }

            var sort = bson.M{
                    "$sort": bson.M{
                            "numberOfRequests": 1,
                    },
            }

            err := m.sess.DB("pusher_tutorial").C(collectionName).
                    Pipe([]bson.M{baseMatch, sort}).All(&ret)
            return ret, err
    }

    func (m mongo) RequestsPerDay() ([]requestsPerDay, error) {

            var ret []requestsPerDay

            var baseMatch = bson.M{
                    "$group": bson.M{
                            "_id":              "$day",
                            "numberOfRequests": bson.M{"$sum": 1},
                    },
            }

            var sort = bson.M{
                    "$sort": bson.M{
                            "numberOfRequests": 1,
                    },
            }

            err := m.sess.DB("pusher_tutorial").C(collectionName).
                    Pipe([]bson.M{baseMatch, sort}).All(&ret)
            return ret, err
    }

    func newMongo(addr string) (mongo, error) {
            sess, err := mgo.Dial(addr)
            if err != nil {
                    return mongo{}, err
            }

            return mongo{
                    sess: sess,
            }, nil
    }

    type Data struct {
            AverageResponseTime float64          `json:"average_response_time"`
            StatsPerRoute       []statsPerRoute  `json:"stats_per_route"`
            RequestsPerDay      []requestsPerDay `json:"requests_per_day"`
            RequestsPerHour     []requestsPerDay `json:"requests_per_hour"`
            TotalRequests       int              `json:"total_requests"`
    }

    func (m mongo) getAggregatedAnalytics() (Data, error) {

            var data Data

            totalRequests, err := m.Count()
            if err != nil {
                    return data, err
            }

            stats, err := m.StatsPerRoute()
            if err != nil {
                    return data, err
            }

            reqsPerDay, err := m.RequestsPerDay()
            if err != nil {
                    return data, err
            }

            reqsPerHour, err := m.RequestsPerHour()
            if err != nil {
                    return data, err
            }

            avgResponseTime, err := m.AverageResponseTime()
            if err != nil {
                    return data, err
            }

            return Data{
                    AverageResponseTime: avgResponseTime,
                    StatsPerRoute:       stats,
                    RequestsPerDay:      reqsPerDay,
                    RequestsPerHour:     reqsPerHour,
                    TotalRequests:       totalRequests,
            }, nil
    }

In the above, we have implemented a few queries on the MongoDB database:

StatsPerRoute: Analytics for each route visited
RequestsPerDay: Analytics per day
RequestsPerHour: Analytics per hour

The next step is to add some HTTP endpoints a user can visit. Without those, the code above for querying MongoDB for analytics is redundant. You will also need to create a logging middleware that writes analytics to MongoDB. And to make it realtime, Pusher Channels will also be used.

To get started with that, you will need to create a file named main.go. You can do that via the command below:

    $ touch main.go

You will also need to fetch some libraries that will be used while building. You will need to run the command below to fetch them:

    $ go get github.com/go-chi/chi
    $ go get github.com/joho/godotenv
    $ go get github.com/pusher/pusher-http-go

In the newly created main.go file, paste the following code:

    // analytics-dashboard/main.go

    package main

    import (
            "encoding/json"
            "flag"
            "fmt"
            "html/template"
            "log"
            "net/http"
            "os"
            "path/filepath"
            "strconv"
            "strings"
            "sync"
            "time"

            "github.com/go-chi/chi"
            "github.com/joho/godotenv"
            "github.com/pusher/pusher-http-go"
    )

    const defaultSleepTime = time.Second * 2

    func main() {
            httpPort := flag.Int("http.port", 4000, "HTTP Port to run server on")
            mongoDSN := flag.String("mongo.dsn", "localhost:27017", "DSN for mongoDB server")

            flag.Parse()

            if err := godotenv.Load(); err != nil {
                    log.Fatal("Error loading .env file")
            }

            appID := os.Getenv("PUSHER_APP_ID")
            appKey := os.Getenv("PUSHER_APP_KEY")
            appSecret := os.Getenv("PUSHER_APP_SECRET")
            appCluster := os.Getenv("PUSHER_APP_CLUSTER")
            appIsSecure := os.Getenv("PUSHER_APP_SECURE")

            var isSecure bool
            if appIsSecure == "1" {
                    isSecure = true
            }

            client := &pusher.Client{
                    AppId:   appID,
                    Key:     appKey,
                    Secret:  appSecret,
                    Cluster: appCluster,
                    Secure:  isSecure,
                    HttpClient: &http.Client{
                            Timeout: time.Second * 10,
                    },
            }

            mux := chi.NewRouter()

            log.Println("Connecting to MongoDB")
            m, err := newMongo(*mongoDSN)
            if err != nil {
                    log.Fatal(err)
            }

            log.Println("Successfully connected to MongoDB")

            mux.Use(analyticsMiddleware(m, client))

            var once sync.Once
            var t *template.Template

            workDir, _ := os.Getwd()
            filesDir := filepath.Join(workDir, "static")
            fileServer(mux, "/static", http.Dir(filesDir))

            mux.Get("/", func(w http.ResponseWriter, r *http.Request) {

                    once.Do(func() {
                            tem, err := template.ParseFiles("static/index.html")
                            if err != nil {
                                    log.Fatal(err)
                            }

                            t = tem.Lookup("index.html")
                    })

                    t.Execute(w, nil)
            })

            mux.Get("/api/analytics", analyticsAPI(m))
            mux.Get("/wait/{seconds}", waitHandler)

            log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *httpPort), mux))
    }

    func fileServer(r chi.Router, path string, root http.FileSystem) {
            if strings.ContainsAny(path, "{}*") {
                    panic("FileServer does not permit URL parameters.")
            }

            fs := http.StripPrefix(path, http.FileServer(root))

            if path != "/" && path[len(path)-1] != '/' {
                    r.Get(path, http.RedirectHandler(path+"/", 301).ServeHTTP)
                    path += "/"
            }

            path += "*"

            r.Get(path, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                    fs.ServeHTTP(w, r)
            }))
    }

    func analyticsAPI(m mongo) http.HandlerFunc {
            return func(w http.ResponseWriter, r *http.Request) {

                    data, err := m.getAggregatedAnalytics()
                    if err != nil {
                            log.Println(err)

                            json.NewEncoder(w).Encode(&struct {
                                    Message   string `json:"message"`
                                    TimeStamp int64  `json:"timestamp"`
                            }{
                                    Message:   "An error occurred while fetching analytics data",
                                    TimeStamp: time.Now().Unix(),
                            })

                            return
                    }

                    w.Header().Set("Content-Type", "application/json")
                    json.NewEncoder(w).Encode(data)
            }
    }

    func analyticsMiddleware(m mongo, client *pusher.Client) func(next http.Handler) http.Handler {
            return func(next http.Handler) http.Handler {
                    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

                            startTime := time.Now()

                            defer func() {

                                    if strings.HasPrefix(r.URL.String(), "/wait") {

                                            data := requestAnalytics{
                                                    URL:         r.URL.String(),
                                                    Method:      r.Method,
                                                    RequestTime: time.Now().Unix() - startTime.Unix(),
                                                    Day:         startTime.Weekday().String(),
                                                    Hour:        startTime.Hour(),
                                            }

                                            if err := m.Write(data); err != nil {
                                                    log.Println(err)
                                            }

                                            aggregatedData, err := m.getAggregatedAnalytics()
                                            if err == nil {
                                                    client.Trigger("analytics-dashboard", "data", aggregatedData)
                                            }
                                    }
                            }()

                            next.ServeHTTP(w, r)
                    })
            }
    }

    func waitHandler(w http.ResponseWriter, r *http.Request) {
            var sleepTime = defaultSleepTime

            secondsToSleep := chi.URLParam(r, "seconds")
            n, err := strconv.Atoi(secondsToSleep)
            if err == nil && n >= 2 {
                    sleepTime = time.Duration(n) * time.Second
            } else {
                    n = 2
            }

            log.Printf("Sleeping for %d seconds", n)
            time.Sleep(sleepTime)
            w.Write([]byte(`Done`))
    }

While the above might seem like a lot, basically what has been done is:

Line 31 - 33: Parse environment variables from the .env created earlier.

Another reminder to update the .env file to contain your actual credentials* Line 36 - 56: A server side connection to Pusher Channels is established

Line 68 - 95: Build an HTTP server.
Line 139 - 171: A lot is happening here. analyticsMiddleware is used to capture all requests, and for requests that have the path wait/{seconds} , a log is written to MongoDB. It is also sent to Pusher Channels.

Before running the server, you need a frontend to visualize the analytics. The frontend is going to be as simple and usable as can be. You will need to create a new directory called static in your root directory - analytics-dashboard . That can be done with the following command:

    $ mkdir analytics-dashboard/static

In the static directory, create two files - index.html and app.js. You can run the command below to do just that:

    $ touch static/{index.html,app.js}

Open the index.html file and paste the following code:

    // analytics-dashboard/static/index.html

    <!DOCTYPE html>
    <html lang="en">

    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <meta http-equiv="X-UA-Compatible" content="ie=edge">
        <title>Realtime analytics dashboard</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
              integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
    </head>
    <body>
    <div class="container" id="app"></div>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/4.1.2/handlebars.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>
    <script src="https://js.pusher.com/4.3/pusher.min.js"></script>
    <script src="/static/app.js"></script>
    </body>
    </html>

While that is an empty page, you will make use of JavaScript to fill it up with useful data. So you will also need to open up the app.js file. In the app.js file, paste the following code:

    // analytics-dashboard/static/app.js

    const appDiv = document.getElementById('app');

    const tmpl = `
    <div class="row">
        <div class="col-md-5">
            <div class="card">
                <div class="card-body">
                    <h5 class="card-title">Total requests</h5>
                    <div class="card-text">
                        <h3>\{{total_requests}}</h3>
                    </div>
                </div>
            </div>
        </div>
        <div class="col-md-5">
            <div class="card">
                <div class="card-body">
                    <h5 class="card-title">Average response time</h5>
                    <div class="card-text">
                        <h3>\{{ average_response_time }} seconds</h3>
                    </div>
                </div>
            </div>
        </div>
    </div>

    <div class="row">
        <div class="col-md-5">
            <div class="card">
                <div class="card-body">
                    <h5 class="card-title">Busiest days of the week</h5>
                    <div class="card-text" style="width: 18rem">
                        <ul class="list-group list-group-flush">
                            {{#each requests_per_day}}
                            <li class="list-group-item">
                                \{{ this.id }} (\{{ this.number_of_requests }} requests)
                            </li>
                            {{/each }}
                        </ul>
                    </div>
                </div>
            </div>
        </div>
        <div class="col-md-5">
            <div class="card">
                <div class="card-body">
                    <h5 class="card-title">Busiest hours of day</h5>
                    <div class="card-text" style="width: 18rem;">
                        <ul class="list-group list-group-flush">
                            {{#each requests_per_hour}}
                            <li class="list-group-item">
                                \{{ this.id }} (\{{ this.number_of_requests }} requests)
                            </li>
                            {{/each}}
                        </ul>
                    </div>
                </div>
            </div>
        </div>
    </div>

    <div class="row">
        <div class="col-md-5">
            <div class="card">
                <div class="card-body">
                    <h5 class="card-title">Most visited routes</h5>
                    <div class="card-text" style="width: 18rem;">
                        <ul class="list-group list-group-flush">
                            {{#each stats_per_route}}
                            <li class="list-group-item">
                                \{{ this.id.method }} \{{ this.id.url }} (\{{ this.number_of_requests }} requests)
                            </li>
                            {{/each}}
                        </ul>
                    </div>
                </div>
            </div>
        </div>
    </div>
    `;

    const template = Handlebars.compile(tmpl);

    writeData = data => {
      appDiv.innerHTML = template(data);
    };

    axios
      .get('http://localhost:4000/api/analytics', {})
      .then(res => {
        console.log(res.data);
        writeData(res.data);
      })
      .catch(err => {
        console.error(err);
      });

    const APP_KEY = 'PUSHER_APP_KEY';
    const APP_CLUSTER = 'PUSHER_CLUSTER';

    const pusher = new Pusher(APP_KEY, {
      cluster: APP_CLUSTER,
    });

    const channel = pusher.subscribe('analytics-dashboard');

    channel.bind('data', data => {
      writeData(data);
    });

Please replace PUSHER_APP_KEY and PUSHER_CLUSTER with your own credentials.
In the above code, we defined a constant called tmpl, it holds an HTML template which we will run through the Handlebars template engine to fill it up with actual data.

With this done, you can go ahead to run the Golang server one. You will need to go to the root directory - analytics-dashboard and run the following command:

    $ go build
    $ ./analytics-dashboard

Make sure you have a MongoDB instance running. If your MongoDB is running on a port other than the default 27017, make sure to add -mongo.dsn "YOUR_DSN" to the above command> Also make sure your credentials are in .env
At this stage, you will need to open two browser tabs. Visit <a href="http://localhost:4000" target="_blank">http://localhost:4000</a> in one and <a href="http://localhost:4000/wait/2" target="_blank">http://localhost:4000/wait/2</a> in the other. Refresh the tab where you have <a href="http://localhost:4000/wait/2" target="_blank">http://localhost:4000/wait/2</a> and go back to the other tab to see a breakdown of usage activity.
Note you can change the value of 2 in the url to any other digit.## Conclusion

In this tutorial, we’ve built a middleware that tracks every request, and a Golang application that calculates analytics of the tracked requests. We also built a dashboard that displays the relevant data. With Pusher Channels, we’ve been able to update the dashboard in realtime. The full source code can be found on GitHub.

#go #mongodb #html #javascript