In enterprise, presumably multiple data sources are required to be handled because of the possession of vast amount of data. When attempting to build a dashboard to showcase business ideas, typically one needs to integrate data from NoSQL database, relational database and search engine. Thus dashboards like Kibana or Google Data Studio are not suitable choices as the flexibility of multiple data source is limited. One then need an alternative that is as straightforward. For this scenario, I recommend Shiny Dashboard for the task as it fulfills the requirements: flexible, straight forward and aesthetic. However, while building the dashboard, there is an immediate implementation barrier of connecting Amazon Elasticsearch Service and that motivates me to write this article.
This article demonstrates the integration of Elasticsearch data into a Shiny dashboard. The programming language used is mainly R and the back-end connection is performed with Python. Meanwhile, for the data visualization part, graphs are drawn with 3 different graphing packages in R, namely: ggplot2, plotly and wordcloud2.
The contents for the article:
- Elasticsearch Connection (with Amazon Elasticsearch Service)
- Data Manipulation using R.
- Showcase the tool Shiny Dashboard to bring data to live without much styling customization.
Environment used:
- R version 3.4.4
- Python 3.7.4
- Ubuntu 18.04
Elasticsearch is a popular search engine in enterprise. Generally speaking, I would recommend it to be added into the data infrastructure when making summary statistics or locating specific batch of large amount of data** in a timely manner** is necessary. For setup, a convenient way is to make use of Amazon Elasticsearch Service since it one would only need to take care of high-level parameters like number of shards. Moreover, a comprehensive documentation and sample codes are provided and there is not much reason of not using it when a company has already built the infrastructure in AWS. Amazon provides sample code which is well-documented but the supporting languages do not contain R. Although there are various packages for Elasticsearch connection in R but the way to integrate it with Amazon Web Services version 4 authentication (AWS4Auth) is not straight forward. To build the dashboard, the crucial part is to overcoming this implementation barrier.
#r #elasticsearch #shiny #data-visualization #data-science
