Getting started with Elasticsearch and Node.js

Getting started with Elasticsearch and Node.js

In this article, I will tell you the complete integration of Elasticsearch with Node.js. We will create our own database and search stuff from that database. The integration of Elasticsearch with Node.js is quite easy.

Elasticsearch is a search engine (and a database too) which is basically used for searching, and for collecting and analyzing logs. It is becoming very popular and is being used in all kinds of applications.

In this article, I will tell you the complete integration of Elasticsearch with Node.js. We will create our own database and search stuff from that database. The integration of Elasticsearch with Node.js is quite easy.

But, before integrating this, you should have basic knowledge of Elasticsearch.

What Do We Do in This Article?
  • We will install Elasticsearch and run it in our system. We will use Docker, where Elasticsearch will run, it is very easy.
  • We will create a huge data storage for live search.
  • We will create indexes (+100,000 indexes) from our created data storage.
  • We will integrate and search content from our data storage using Elasticsearch.
  • We will analyze the search time.
  • Delete the indexes of Elasticsearch.
Install Elasticsearch

Let’s install Elasticsearch on Ubuntu/Mac/Windows and also install Docker.

Run Elasticsearch with Docker

Before moving forward, I am assuming that you have successfully installed Docker. Let’s run Elasticsearch locally using Docker.

You need to pull the image for Elasticsearch using the command below:

docker pull docker.elastic.co/elasticsearch/elasticsearch:6.4.0

Note: In the above snapshot, the content/message may be different because I have already pulled the image for Elasticsearch (ES). So, don’t get confused.

Run the Docker image

Now its time to run the Docker image using the below command:

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.4.0

By default, this Docker is running on port 9200. Now, open the browser and hit URL localhost:9200 and test the message as the below is showing:

The above message is a great sign that your Elasticsearch on Docker is running.

Keep this terminal open, don't close it. If your Elasticsearch is not running then you will get the following error:

Create a Huge Data Storage for Live Search

To create a huge database, you need to download a file named planet-latest_geonames.tsv.gz from GitHub, which contains a lot of information.

If the above link is not working then go to this link and download file planet-latest_geonames.tsv.gzand extract it.

Check the name of the file, it should be planet-latest-100k_geonames.tsv. If it’s not, rename it to planet-latest-100k_geonames.tsv.

If none of the above links are working then comment me I will share/sent to you personally.

Now, copy this file and create a directory, elasticsearch, and paste in it.

Now, we have a huge database with a lot of data. This is the data we will perform the indexing on so that we can perform our search in real-time.

Write Code for Indexing and Start Working With Elasticsearch

Write the code for creating the indexing for the above database, and start working with Elasticsearch.

Create a file named makeIndex.js and paste the below code into it:

//makeIndex.js 
const _ = require('highland');
const fs = require('fs');
const csv = require('csv-parser');
const elasticsearch = require('elasticsearch');
const indexName = 'demo_elastic_index';

const start = async () => {
  const client = new elasticsearch.Client({
     host: 'localhost:9200',
     // log: 'trace',
  });
  await client.ping({
    requestTimeout: 3000
  }, function (error) {
    if (error) {
      console.trace('elasticsearch cluster is down!');
    } else {
      console.log('Elastic search is running.');
    }
  });
  try {
    await client.indices.create({index: indexName});
    console.log('created index');
   } catch (e) {
      if (e.status === 400) {
        console.log('index alread exists');
      } else {
        throw e;
      }
    }

    // process file
    let currentIndex = 0;
    const stream = _(
      fs.createReadStream('./planet-latest-100k_geonames.tsv').pipe(
        csv({
          separator: '\t',
        })
      )
    )
.map(data => ({
      ...data,
      alternative_names: data.alternative_names.split(','),
      lon_num: parseFloat(data.lon),
      lat_num: parseFloat(data.lat),
      place_rank_num: parseInt(data.place_rank, 10),
      importance_num: parseFloat(data.importance),
    }))
.map(data => [{
  index: {_index: indexName, _type: 'place', _id: data.osm_id},
  },
  data,
])
.batch(100)
.each(async entries => {
      stream.pause();
      const body = entries.reduce((acc, val) => acc.concat(val),[]);
    await client.bulk({body});
    currentIndex += 100;
    console.log('Created index :', currentIndex);
    stream.resume();
})
.on('end', () => {
    console.log('done');
    process.exit();
   });
};
start();

makeIndex.js

Now it’s time to execute makeIndex.js and create the indexing using this command:

node makeIndex.js

If you get the error: No living connections like in the below snapshot:

The above error denotes that you don’t have any living connections for Elasticsearch, which means your Elasticsearch is not running on Docker (check localhost:9200).

In this case, you need to repeat the above step “Run Elasticsearch with Docker” until you get the following message on the terminal while executing the command node makeIndex.js.

If you get the message: Elastic search like in the below snapshot:

And after some time, the indexing will be complete. (It may take some time to complete indexing.)

The number of indexes may vary from the below snapshot because it depends on the download file planet-latest_geonames.tsv.gz.

If you can see the message Indexing Completed in your terminal, it means your indexing is created, now you are ready to search.

Search the Data From the Above Index

To search the data in the index demo_elastic_index, we need to write the code to search the data. So, let’s create a file, searchData.js, and paste the below code in this file.

//searchData.js
const elasticsearch = require('elasticsearch');
const indexName = 'demo_elastic_index';
const query = 'Lewisham';
const searchData = async () => {
  const client = new elasticsearch.Client({
  host: 'localhost:9200',
  // log: 'trace',
  });
await client.ping({
  requestTimeout: 3000
  }, function (error) {
    if (error) {
      console.trace('elasticsearch cluster is down!');
     } else {
      console.log('Elastic search is running.');
    }
  });
try {
  const resp = await client.search({
  index: indexName,
  type: 'place',
  body: {
    sort: [
    {
      place_rank_num: { order: 'desc' },
    },
    {
      importance_num: { order: 'desc' },
    },
],
query: {
    bool: {
    should: [{
              match: {
              lat: '51.4624325',
              }
             },{
              match: {
              alternative_names: query,
             },
          }]
        },
     },
   },
});
  const { hits } = resp.hits;
  console.log(hits);
} catch (e) {
   //   console.log("Error in deleteing index",e);
   if (e.status === 404) {
       console.log('Index Not Found');
    } else {
             throw e;
           }
    }
}
searchData();

searchData.js

  • //1: The query parameter indicates query context.
  • //2 and //3: The bool and match clauses are used in query context, which means that they are used to score how well each document matches.
  • //4: The filter parameter indicates filter context.
  • //5 and //6: The term and range clauses are used in the filter context. They will filter out documents that do not match, but they will not affect the score for matching documents.

Now, run the above code using the below command:

node searchData

You can see the result below:

Score: how well each document matches.

Some other query and their results:

{
  "query": {                                  //1
    "bool": {                                 //2
      "must": [
       { "match":{"address":"Street"}}        //3
      ],
      "filter": [                             //4
        { "term":{"gender":"f"}},             //5
        { "range": { "age": { "gte": 25 }}}   //6
      ]
    }
  }
}

query.js

Delete Index

Let's delete all the indexes.

//deleteIndex.js
const elasticsearch = require('elasticsearch');
const indexName='demo_elastic_index';
const deleteIndex = async () => {
  const client = new elasticsearch.Client({
  host: 'localhost:9200',
  // log: 'trace',
  });
await client.ping({
  requestTimeout: 3000
  }, function (error) {
       if (error) {
          console.trace('elasticsearch cluster is down!');
       } else {
          console.log('Elastic search is running.');
       }
    });
  try {
     await client.indices.delete({index: indexName});
     console.log('All index is deleted');
     } catch (e) {
          //   console.log("Error in deleteing index",e);
               if (e.status === 404) {
                      console.log('Index Not Found');
               } else {
                   throw e;
               }
     }
}
deleteIndex();

deleteIndex.js

Run the above code:

node deleteIndex.js

Let’s search now:

node searchData.js

You can see a message in the terminal:

Index not found. (Because the index is deleted now, you need to create the index again by running the command node makeIndex.js.)

Let’s create the index again:

After the creation of the index, you can search the data again using the searchData file.

So, let’s enjoy Elasticsearch. Thanks for reading. Keep coding.

Node.js for Beginners - Learn Node.js from Scratch (Step by Step)

Node.js for Beginners - Learn Node.js from Scratch (Step by Step)

Node.js for Beginners - Learn Node.js from Scratch (Step by Step) - Learn the basics of Node.js. This Node.js tutorial will guide you step by step so that you will learn basics and theory of every part. Learn to use Node.js like a professional. You’ll learn: Basic Of Node, Modules, NPM In Node, Event, Email, Uploading File, Advance Of Node.

Node.js for Beginners

Learn Node.js from Scratch (Step by Step)

Welcome to my course "Node.js for Beginners - Learn Node.js from Scratch". This course will guide you step by step so that you will learn basics and theory of every part. This course contain hands on example so that you can understand coding in Node.js better. If you have no previous knowledge or experience in Node.js, you will like that the course begins with Node.js basics. otherwise if you have few experience in programming in Node.js, this course can help you learn some new information . This course contain hands on practical examples without neglecting theory and basics. Learn to use Node.js like a professional. This comprehensive course will allow to work on the real world as an expert!
What you’ll learn:

  • Basic Of Node
  • Modules
  • NPM In Node
  • Event
  • Email
  • Uploading File
  • Advance Of Node

Top 7 Most Popular Node.js Frameworks You Should Know

Top 7 Most Popular Node.js Frameworks You Should Know

Node.js is an open-source, cross-platform, runtime environment that allows developers to run JavaScript outside of a browser. In this post, you'll see top 7 of the most popular Node frameworks at this point in time (ranked from high to low by GitHub stars).

Node.js is an open-source, cross-platform, runtime environment that allows developers to run JavaScript outside of a browser.

One of the main advantages of Node is that it enables developers to use JavaScript on both the front-end and the back-end of an application. This not only makes the source code of any app cleaner and more consistent, but it significantly speeds up app development too, as developers only need to use one language.

Node is fast, scalable, and easy to get started with. Its default package manager is npm, which means it also sports the largest ecosystem of open-source libraries. Node is used by companies such as NASA, Uber, Netflix, and Walmart.

But Node doesn't come alone. It comes with a plethora of frameworks. A Node framework can be pictured as the external scaffolding that you can build your app in. These frameworks are built on top of Node and extend the technology's functionality, mostly by making apps easier to prototype and develop, while also making them faster and more scalable.

Below are 7of the most popular Node frameworks at this point in time (ranked from high to low by GitHub stars).

Express

With over 43,000 GitHub stars, Express is the most popular Node framework. It brands itself as a fast, unopinionated, and minimalist framework. Express acts as middleware: it helps set up and configure routes to send and receive requests between the front-end and the database of an app.

Express provides lightweight, powerful tools for HTTP servers. It's a great framework for single-page apps, websites, hybrids, or public HTTP APIs. It supports over fourteen different template engines, so developers aren't forced into any specific ORM.

Meteor

Meteor is a full-stack JavaScript platform. It allows developers to build real-time web apps, i.e. apps where code changes are pushed to all browsers and devices in real-time. Additionally, servers send data over the wire, instead of HTML. The client renders the data.

The project has over 41,000 GitHub stars and is built to power large projects. Meteor is used by companies such as Mazda, Honeywell, Qualcomm, and IKEA. It has excellent documentation and a strong community behind it.

Koa

Koa is built by the same team that built Express. It uses ES6 methods that allow developers to work without callbacks. Developers also have more control over error-handling. Koa has no middleware within its core, which means that developers have more control over configuration, but which means that traditional Node middleware (e.g. req, res, next) won't work with Koa.

Koa already has over 26,000 GitHub stars. The Express developers built Koa because they wanted a lighter framework that was more expressive and more robust than Express. You can find out more about the differences between Koa and Express here.

Sails

Sails is a real-time, MVC framework for Node that's built on Express. It supports auto-generated REST APIs and comes with an easy WebSocket integration.

The project has over 20,000 stars on GitHub and is compatible with almost all databases (MySQL, MongoDB, PostgreSQL, Redis). It's also compatible with most front-end technologies (Angular, iOS, Android, React, and even Windows Phone).

Nest

Nest has over 15,000 GitHub stars. It uses progressive JavaScript and is built with TypeScript, which means it comes with strong typing. It combines elements of object-oriented programming, functional programming, and functional reactive programming.

Nest is packaged in such a way it serves as a complete development kit for writing enterprise-level apps. The framework uses Express, but is compatible with a wide range of other libraries.

LoopBack

LoopBack is a framework that allows developers to quickly create REST APIs. It has an easy-to-use CLI wizard and allows developers to create models either on their schema or dynamically. It also has a built-in API explorer.

LoopBack has over 12,000 GitHub stars and is used by companies such as GoDaddy, Symantec, and the Bank of America. It's compatible with many REST services and a wide variety of databases (MongoDB, Oracle, MySQL, PostgreSQL).

Hapi

Similar to Express, hapi serves data by intermediating between server-side and client-side. As such, it's can serve as a substitute for Express. Hapi allows developers to focus on writing reusable app logic in a modular and prescriptive fashion.

The project has over 11,000 GitHub stars. It has built-in support for input validation, caching, authentication, and more. Hapi was originally developed to handle all of Walmart's mobile traffic during Black Friday.

Node JS Full Course Learn Node.js in 7 Hours

Node JS Full Course Learn Node.js in 7 Hours

This Edureka Node.js Full Course video will help you in learn Node.js along with practical demonstration. This Node.js Tutorial for Beginners is ideal for both beginners as well as professionals who want to master the most prominently used javascript backend framework.

This Edureka Node.js Full Course video will help you in learn Node.js along with practical demonstration. This Node.js Tutorial for Beginners is ideal for both beginners as well as professionals who want to master the most prominently used javascript backend framework. Below are the topics covered in this node.js tutorial video:
2:32 What is Node.js?
3:22 Client-Server Architecture
4:12 Multi-Threaded Model
6:13 Single-Threaded Model
7:43 Multi-Threaded vs Event-Driven
9:45 Uber Old Architecture
11:10 Uber New Architecture
12:30 What is Node.js?
13:05 Sucess Stories
14:20 Node.js Trend
14:40 Node.js Features
16:25 Node.js Installation
16:50 Node.js First Example
17:30 Blocking vs Non-blocking
18:50 Demo
23:50 Node.js Modules
23:50 NPM
25:10 Global Objects
26:55 File System
30:30 Callbacks
31:45 Event
33:05 HTTP
34:50 Hands On
1:09:45 Node.js Tutorial
1:10:45 What is Node.js?
1:12:10 Features of Node.js
1:13:00 Node.js Architecture
1:14:55 NPM(Node Package Manager)
1:16:20 Node.js Modules
1:16:30 Node.js Modules Types
1:16:35 Core Modules
1:16:55 Local Modules
1:17:10 3rd Party Modules
1:18:35 JSON File
1:23:30 Data Types
1:25:35 Variables
1:26:40 Operators
1:27:45 Functions
1:29:10 Objects
1:29:55 File Systems
1:33:50 Events
1:34:20 HTTP Module
1:40:02 Events
1:44:37 HTTP Module
1:45:27 Creating a Web Server using Node.js
1:45:42 Express.js
1:46:57 Demo
1:58:37 Node.js NPM Tutorial
1:59:37 What is NPM?
2:03:12 Main Functions of NPM
2:04:27 Need For NPM
2:08:07 NPM Packages
2:17:42 NPM Installation
2:18:12 JSON File
2:31:32 Node.js Express Tutorial
2:32:02 Introduction to Express.js
2:32:32 Features of Express.js
2:35:27 Getting Started with Express.js
2:39:42 Routing Methods
2:44:57 Hands-On
2:48:12 Building RESTful API with Node.js
2:48:27 What is REST API?
2:49:42 Features of REST API
2:51:12 Principles of REST API
2:56:37 Methods of REST API
2:59:52 Building REST API with Node.js
3:24:07 Node.js MySQL Tutorial
3:24:32 What is MySQL?
3:25:13 Advantages of Using MySQL with Node.js
3:27:38 MySQL Installation
3:44:23 Node.js MongoDB Tutorial
3:44:58 What is NoSQL?
3:47:53 NoSQL Databases
3:48:38 Introduction to MongoDB
3:52:48 Features of MongoDB
3:53:03 MongoDB Installation
4:36:08 Node.js Docker Tutorial
4:36:38 What is Docker?
4:39:13 Docker Working
4:41:43 Docker Basics
4:41:48 DockerFile
4:42:03 Docker Images
4:42:23 Docker Container
4:44:38 Why use Node.js with Docker?
4:45:18 Demo: Node.js with Docker
4:58:38 MEAN Stack Application Tutorial
4:59:18 What is MEAN Application?
4:59:53 MongoDB
5:00:28 Express
5:01:13 Angular
5:01:23 Node.js
5:02:17 RESTful API
5:03:02 Contact List MEAN App
6:17:57 Node.js Interview Questions

Got a question on the topic? Please share it in the comment section below and our experts will answer it for you.