Working With Documents Using Elastic-Search and Node

Introduction

Working with documents in software is fun. It means that storage fit your code not the other way around. This removes the object relational impedance mismatch between how you model your application and how you store those models.

Even if you do not have immediate use of documents, learning how to use documents will broaden your perspective of storage systems.

Recently I have written a post on DZone about document databases in postgreSQL and build a small application using .NET Core. However, this time, I am going to use ElasticSearch as document persistence mechanism and Node.js to work with elastic-search. Elastic Stack (ELK) have other components e.g. Kibana, Log-Stash etc., but I will be only using elastic-search component for discussion.

Elastic Search and Index

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java.

What Is an Index

• The word index itself has different meanings in different context in elastic-search.
• Unlike conventional database, In ES, an index is a place to store related documents.
• An index is a collection of documents that have somewhat similar characteristics.
• The act of storing documents in an index is known indexing.
• An index is identified by a name, which is used to refer to the index when performing indexing, search, update and delete operations against the documents in it.
• We’ll see later how to create index, delete index and store documents in the index etc.

Installation

You can download installer for elastic-search from official website and install it or alternatively you can use some cloud services for same purpose.

For the demo, I installed it on my local machine and then browse to following URL to verify the installation.

Document

Before we start working with elastic-search, lets talk a little bit about documents. You can also read my previous post if you need more details about documents. Documents are essentially a set of key-value pair.

A document is a basic unit of information that can be indexed e.g. you can have a document for a single customer, another document or single product and yet another for a single order.

The document is expressed in JSON which is ubiquitous internet data interchange format.

Within an index/type, you can store as many documents as you want. You can imagine those records in a relational database thinking.

Elastic-Search Restful API

Elastic-Search has quite a few APIS:

• “cluster” API: to manage clusters.
• “index” API: Give access to our indices, mapping, aliases etc.
• “search” API: To query, count, filter, data access multiple indices and types.
• “document” API: to add data etc.

Here are few of the API calls using browser:

ElasticSearch With Node

Now we will build a simple Node.js application to perform some operations on elastic-search. I initialized the source-code folder with npm init command:

Install Elasticsearch Package

Next, I installed elasticsearch npm package as follows:

Connection to ElasticSearch

Connection.js file encapsulate connection to elastic-search.

To verify the connection I add the following code:

Let's run it and see if the connection works:

Ok, our connection is working and lets continue to work with index and documents.

Build an Index API

NPM elasticsearch package exposes many methods that we can use in our application. We can create index, delete them, add documents to them, etc. The index name should be lower case. If index already exists, you get ‘index_already_exists_exception’. You can have an index for customer data, another for product catalog, and yet another index for order data.

For this demo, I will create an API IndexManager to encapsulate these concerns and then we can just use that API. I will keep the implementation simple, however, feel free to adjust as per your style. The API will have the following functionality:

• Create Index
• Delete Index
• Check if Index exist
• Add documents to index

Here is the code for the API which is self explanatory, however if something is not clear, ask in comment:

Here is the method implementation code:

Client Code

Now, as we have functionality related to connection and working with index and documents, let execute these commands:

Create Index ‘Blog’:

Add document to Index

I created a class for Post and this will be searlized to json and save as document into index blog:

and execute the code:

and we can see that data is inserted to elasticsearch:

Import Documents (Blogs) into ElasticSearch Index ‘Blog’

The following code will read the data from a JSON file, parse it, and then save it in the elasticsearch index. you can use this method to populate documents.

JSON Data File

JSON Data Loader

the following code, read JSON data file and return the documents objects:

and also update the client-code, as follows:

Here is the data of all documents after executing the code:

Summary

Elastic-Search is a very powerful and easy to integrate options for your persistence mechanism. The REST API provides very useful services to work with elasticsearch. You can download the code for this application from this git repo. Till next time, Happy Coding.

References

• https://www.elastic.co/
• https://hexquote.com/net-core-postgresql-and-document-database/
• https://github.com/jawadhasan/esdemo.git