How to work with Elasticsearch in .Net

Elasticsearch is an open source, cross-platform, highly scalable distributed search and analytics engine based on Apache Lucene. Lucene is a popular Java-based, full-text search engine that can be used to query large sets of data efficiently. In this article, we’ll discuss the capabilities of Elasticsearch and how we can work with it in .Net using the popular .Net client for Elasticsearch called NEST. 

Elasticsearch gives you web-friendly REST APIs that exchange data in JSON format. Thus you can take advantage of Elasticsearch from your .Net application to store, search, and analyze large volumes of data in real time. Because Elasticsearch is customizable, you can use it to build our own custom search engine quite easily. There are plenty of plug-ins available to help you do this.

Unlike relational databases, Elasticsearch doesn’t store data in separate tables, but as JSON documents. Elasticsearch provides a query language called Query DSL that enables you to build and execute custom queries easily. Elasticsearch will perform better than your traditional relational database if you have a huge amount of data that must be searchable.

Elasticsearch basics

In this section we will examine the basic concepts related to Elasticsearch that will help us to get acquainted with Elasticsearch quickly. These include clusters, nodes, index, shards, and replicas.

In Elasticsearch, the JSON document is the basic unit of information that can be indexed. A document comprises of a collection of fields. Hence fields are the smallest individual unit of representation of data in Elasticsearch.

A cluster in Elasticsearch is defined as a collection of one or more nodes. These nodes communicate with one another and write to an index. A cluster is identified using a unique name. This unique name is “elasticsearch” by default.

A node is a single Elasticsearch server. A node is identified by a random UUID by default, which is assigned at start-up time. Inside a cluster, you can have nodes of different types with different responsibilities, i.e., data nodes, master nodes, client nodes, tribe nodes, and ingestion nodes. Note that all nodes that pertain to the same cluster always share the same cluster name. Note too that you can query any node in a cluster, even if it is a master node.

An index may be defined as a collection of documents of the same type. As an example, you can have an index for product data, one for customer data, one for sales data, yet another for security data, and so on. An index is identified using a name. You can use this name when you are performing search, update, and delete operations on the documents in that index. In a single cluster, you can have as many indexes as you might want.

A shard is an instance of a Lucene index, and it can have zero or more replicas. Note that you can have one or more “types” inside an index. While an index is analogous to a database, a type is similar to a table in the database. So a type in Elasticsearch defines a logical category of your index.

Getting started with Elasticsearch

The first thing to do is download Elasticsearch and install it on your system. You will also need a client to work with Elasticsearch. There are two popular .Net clients available. Elasticsearch.Net is a low-level client, while Nest is a high-level client that comes with a strongly typed query language and maps nicely to the original Elasticsearch client. In this article we will be using Nest.

Once Elasticsearch has been successfully installed and running on your system, point your web browser at this address: 

http://localhost:9200

Here is the JSON response that Elasticsearch returned to the web browser on my system: 

{
  “name” : “wTkxaEG”,
  “cluster_name” : “elasticsearch”,
  “cluster_uuid” : “8azkA-8dTequMRWxOqqcSA”,
  “version” : {
    “number” : “5.5.2”,
    “build_hash” : “b2f0c09”,
    “build_date” : “2017-08-14T12:33:14.154Z”,
    “build_snapshot” : false,
    “lucene_version” : “6.6.0”
  },
  “tagline” : “You Know, for Search”
}

Connecting to Elasticsearch from .Net

Now let’s make the connection to Elasticsearch from .Net. Create a new Console Application project in Visual Studio. To install Nest in your project, type the following command in the Package Manager Console Window within the Visual Studio IDE.

PM> Install-Package NEST

Assuming Nest has been successfully installed, the following code listing illustrates how you can connect to Elasticsearch using Nest in C#.

using Nest;
using System;

namespace IDGElasticSearch
{
    class Program
    {
        static void Main(string[] args)
        {
            var node = new Uri(“http://localhost:9200”);
            var settings = new ConnectionSettings(node);
            var client = new ElasticClient(settings);
            var response = client.ClusterHealth();
            Console.WriteLine(response.Status);
            Console.Read();
        }
    }
}

Elasticsearch is a great choice for building search and analysis engines, and can also serve as a general purpose document store. However, you should not use Elasticsearch as your primary data source. It is not meant to be a replacement for your relational databases.

I will be writing about the more advanced concepts of Elasticsearch, and how we can create, index, search, delete, and update documents using Elasticsearch, in future posts. In the meantime, you can learn more by exploring the official Elasticsearch guide at the Elastic website.