WIXLIB

Elasticsearch Tutorial2 / 54In perspective of Elasticsearch, being RESTful APIs has another advantage: every single piece of data sent to or received fromElasticsearch is itself a human-readable JSON document (although this is not the only protocol Elasticsearch supports as we aregoing to see later on).To keep the discussion relevant and practical, we are going to pretend that we are developing the application to manage thecatalog of books. The data model will include categories, authors, publisher, book details (like publishing date, ISBN, rating)and brief description.Figure 1.1: Book CatalogLet us see how we could leverage Elasticsearch to make our book catalog easily searchable but before that we need to getfamiliarized a bit with the terminology. Although in the next couple of sections we are going to go over most of the conceptsbehind Elasticsearch, please do not hesitate to consult Elasticsearch official documentation any time.1.2.1DocumentsTo put it simply, in context of Elasticsearch document is just an arbitrary piece of data (usually, structured). It could be absolutelyanything which makes sense to your applications (like users, logs, blog posts, articles, products, . . . ) but this is a basic unit ofinformation which Elasticsearch could manipulate.1.2.2IndicesElasticsearch stores documents inside indices and as such, an index is simply a collection of the documents. To be fair, persistingabsolutely different kind of the documents in the same index would be somewhat convenient but quite difficult to work with soevery index may have one or more types. The types group documents logically by defining a set of common properties (or fields)every document of such type should have. Types serve as a metadata about documents and are very useful for exploring thestructure of the data and constructing meaningful queries and aggregations.1.2.3Index SettingsEach index in Elasticsearch could have specific settings associated with it at the time of its creation. The most important ones arenumber of shards and replication factor. Let us talk about that for a moment.Elasticsearch has been built from the ground up to operate over massive amount of indexed data which will very likely exceedthe memory and/or storage capabilities of a single physical (or virtual) machine instance. As such, Elasticsearch uses sharding as

Elasticsearch Tutorial3 / 54a mechanism to split the index into several smaller pieces, called shards, and distribute them among many nodes. Please notice,once set the number of shards could not be altered (although this is not entirely true anymore, the index could be shrunk intofewer shards).Indeed, sharding solves a real problem but it is vulnerable to data loss issues due to individual node failures. To address this problem, Elasticsearch supports high availability by leveraging replication. In this case, depending on replication factor, Elasticsearchmaintains one or more copies of each shard and makes sure that each shard’s replica is placed on different node.1.2.4MappingsThe process of defining the type of the documents and assigning it to a particular index is called index mapping, mapping typeor just a mapping. Coming up with a proper type mapping is, probably, one of the most important design exercises you wouldhave to make in order to get most out of Elasticsearch. Let us take some time and talk about mappings in details.Each mapping consists of optional meta-fields (they usually start from the underscore character like index, id, parent)and regular document fields (or properties). Each field (or property) has a data type, which in Elasticsearch could fall into oneof those categories: Simple data types– text - indexes full-text values– keyword - indexes structured values– date - indexes date/time values– long - indexes signed 64-bit integer values– integer - indexes signed 32-bit integer values– short - indexes signed 16-bit integer values– byte - indexes signed 8-bit integer values– double - indexes double-precision 64-bit IEEE 754 floating point values– float - indexes single-precision 32-bit IEEE 754 floating point values– half float - indexes half-precision 16-bit IEEE 754 floating point values– scaled float - indexes floating point values that is backed by a long and a fixed scaling factor– boolean - indexes boolean values (for example, true/false, on/off, yes/no, 1/0)– ip - indexes either IPv4 or IPv6 address values– binary - indexes any binary value encoded as a Base64 string Composite data types– object - indexes inner objects which, in turn, may contain inner objects themselves– nested - a specialized version of the object data type that allows to index arrays of objects independently of each other Specialized data type– geo point - indexes latitude-longitude pairs– geo shape - indexes an arbitrary geo shapes (such as rectangles and polygons)– completion - dedicated data type to back auto-complete/search-as-you-type functionality– token count - dedicated data type to count the number of tokens in a string– percolator - specialized data type to store the query which is going to be used by percolate query to match the documents Range data types:– integer range - indexes a range of signed 32-bit integers– float range - indexes a range of single-precision 32-bit IEEE 754 floating point values

Elasticsearch Tutorial4 / 54– long range - indexes a range of signed 64-bit integers– double range - indexes a range of double-precision 64-bit IEEE 754 floating point values– date range - indexes a range of date values represented as unsigned 64-bit integer milliseconds elapsed since system epochCannot stress it enough, choosing the proper data type for the fields (properties) of your documents is a key for fast, effectivesearch which delivers really relevant results. There is one catch though: the fields in each mapping type are not entirely independent of each other. The fields with the same name and within the same index but in different mapping types must have thesame mapping definition. The reason is that internally those fields are mapped to the same field.Getting back to our application data model, let us try to define the simplest mapping type for books collections, utilizing ourjust acquired knowledge about data types.Figure 1.2: Mapping Book Catalog: first attemptFor most of the book properties the mapping data types are pretty straightforward but what about authors and categories?Those properties essentially contain the collection of values for which Elasticsearch has no direct data type yet, . . . or has it?1.2.5Advanced MappingsInterestingly, indeed Elasticsearch has no dedicated array or collection type but by default, any field may contain zero or morevalues (of its data type).In case of complex data structures, Elasticsearch supports mapping using object and nested data types as well as establishingparent/child relationships between documents within the same index. There are pros and cons of each approach but in order tolearn how to use those techniques let us store categories as nested property of the books mapping type, while authorsare going to be represented as a dedicated mapping which refers to books as parent.

Elasticsearch Tutorial5 / 54Figure 1.3: Mapping Book Catalog: second (and the last) attemptThese are our close to final mapping types for the catalog index. As we already know, JSON is a first class citizen inElasticsearch, so let us get a feeling of how the typical index mapping looks like in the format Elasticsearch actually understands.{"mappings": {"books": {" source" : {"enabled": true},"properties": {"title": { "type": "text" },"categories" : {"type": "nested","properties" : {"name": { "type": "text" }}},"publisher": { "type": "keyword" },"description": { "type": "text" },"published date": { "type": "date" },"isbn": { "type": "keyword" },"rating": { "type": "byte" }}},"authors": {"properties": {"first name": { "type": "keyword" },"last name": { "type": "keyword" }}," parent": {"type": "books"}}}}

Elasticsearch Tutorial6 / 54You may be surprised but explicit definition of the fields and mapping types could be omitted. Elasticsearch supports dynamicmapping thereby new mapping types and new field names will be added automatically when document is indexed (in this caseElasticsearch makes a decision what the field data types should be).Another important detail to mention is that each mapping type can have custom metadata associated with it by using specialmeta property. It is exceptionally useful technique which will be used by us later on in the tutorial.1.2.6IndexingOnce Elasticsearch has all your indices and their mapping types defined (or inferred using dynamic mapping), it is ready toanalyze and index the documents. It is quite complex but interesting process which involves at least analyzers, tokenizers, tokenfilters and character filters.Elasticsearch supports quite a rich number of mapping parameters which let you tailor the indexing, analysis and search phasesprecisely to your needs. For example, every single field (or property) could be configured to use own index-time and searchtime analyzers, support synonyms, apply stemming, filter out stop words and much, much more. By carefully crafting theseparameters you may end up with superior search capabilities, however the opposite also holds true, having them loose, and a lotof irrelevant and noisy results may be returned every time.If you don’t need all that, you are good to go with the defaults as we have done in the previous section, omitting the parametersaltogether. However, it is rarely the case. To give a realistic example, most of the time our applications have to support multiplelanguages (and locales). Luckily, Elasticsearch shines here as well.Before we move on to the next topic, there is an important constraint you have to be aware of. Once the mapping types areconfigured, in majority of cases they cannot be updated as it automatically assumes that all the documents in the correspondingcollections are not up to date anymore and should be re-indexed.1.2.7Internalization (i18n)The process of indexing and analyzing the documents is very sensitive to the native language of the document. By default,Elasticsearch uses standard analyzer if none is specified in the mapping types. It works well for most of the languages butElasticsearch supplies the dedicated analyzers for Arabic, Armenian, Basque, Brazilian, Bulgarian, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Latvian, Lithuanian, Norwegian, Persian,Portuguese, Romanian, Russian, Spanish, Swedish, Turkish, Thai and a few more.There are couple of ways to approach the indexing of the same document in multiple languages, depending on your data modeland business case. For example, if document instances physically exist (translated) in multiple languages, than it probably makessense to have one index per language.In case when documents are partially translated, Elasticsearch has another interesting option hidden in the sleeves called multifields. Multi-fields allow indexing the same document field (property) in different ways to be used for different purposes (like, forexample, supporting multiple languages). Getting back to our books mapping type, we may have defined the title propertyas a multi-field one, for example:"title": {"type": "text","fields": {"en": { "type": "text", "analyzer": "english" },"fr": { "type": "text", "analyzer": "french" },"de": { "type": "text", "analyzer": "german" },.}}Those are not the only options available but they illustrate well enough the flexibility and maturity of the Elasticsearch in fulfillingquite sophisticated demands.

Elasticsearch Tutorial1.37 / 54Running ElasticsearchElasticsearch embraces simplicity in many ways and one of those is exceptionally easy way to get started on mostly any platformin just two steps: download and run. In the next couple of sections we are going to talk about quite a few different ways to getyour Elasticsearch up and running.1.3.1Standalone InstanceRunning Elasticsearch as a standalone application (or instance) is the fastest and simplest route to take. Just download thepackage of your choice and run the shell script on Linux/Unix/Mac operating systems:bin/elasticsearchOr from the batch file on Windows operating system:bin\elasticsearch.batAnd that is it, pretty straightforward, isn’t it? However, before we go ahead and talk about more advanced options, it wouldbe useful to get a taste what it actually means to run an instance of Elasticsearch. To be more precise, every time we say weare starting the instance of Elasticsearch, we are actually starting an instance of a node. As such, depending on providedconfiguration (by default, it is stored in conf/elastisearch.yml file) there are multiple node types which Elasticsearchsupports at the moment. In this regards, every running standalone instance of Elasticsearch could be configured to run as one (orcombination) of those node types: data node: these kind of nodes are maintaining the data and performing operations over this data (it is controlled by node.data configuration setting which is set to true by default) ingest node: these are special kind of nodes which are able to apply an ingest pipeline in order to transform and enrich thedocument before indexing it (it is controlled by node.ingest configuration setting which is set to true by default)Please take a note that this is not an exhaustive list of node types yet, we are going to learn quite a few more in just a moment.1.3.2ClusteringRunning Elasticsearch as a standalone instance is good for development, learning or testing purposes but certainly is not an optionfor production systems. Generally, in most real-world deployments Elasticsearch is configured to run in a cluster: a collectionof one or more nodes preferably split across multiple physical instances. Elasticsearch cluster manages all the data and alsoprovides federated indexing, aggregations and search capabilities across all its nodes.Every Elasticsearch cluster is identified by a unique name which is controlled by cluster.name configuration setting (setto "elasticsearch" by default). The nodes are joining the cluster by referring to its name so it is quite important pieceof configuration. Last but not least, each cluster has a dedicated master node which is responsible for performing cluster-wideactions and operations.Specifically applicable to the clustered configuration, Elasticsearch supports a couple of more node types, in addition to the oneswe already know about: master-eligible node: these kind of nodes are marked as eligible to be elected as the master node (it is controlled by node.master configuration setting which is set to true by default) coordinating-only node: these are special kind of nodes which are able to only route requests, handle some search phases,and distribute bulk indexing, essentially behaving as load balancers (the node automatically becomes coordinating-only whennode.master, node.data and node.ingest settings are all set to false) tribe node: these are special kind of coordinating-only nodes that can connect to multiple clusters and execute search or otheroperations across all of them (it is controlled by tribe.* configuration settings)

Elasticsearch Tutorial8 / 54By default, if configuration is not specified, each Elasticsearch node is configured to be master-eligible, data node and ingestnode. Similarly to the standalone instance, Elasticsearch cluster instances could be started quickly from the command line:bin/elasticsearch -Ecluster.name cluster-name -Enode.name node-name Or on Windows platform:bin\elasticsearch.bat -Ecluster.name cluster-name -Enode.name node-name Along with sharding and replication, an Elasticsearch cluster has all the properties of a highly available and scalable system whichwill organically evolve to meet the needs of your applications. To be noted, despite significant efforts invested into stabilizingElasticsearch clustering implementation and covering a lot of edge cases related to a different kind of failure scenarios, as of nowElasticsearch is still not recommended to serve as a system of record (or primary storage engine of your data).1.3.3Embedding Into ApplicationNot a long time ago (right till 5.0 release branch) Elasticsearch fully supported the option to be run as part of the application,within the same JVM process (the technique commonly referred as embedding). Although it is certainly not a recommendedpractice, sometimes it was very useful and saves a lot of effort, for example during integration / system / component test runs.The situation has changed recently and embedded version of the Elasticsearch is not officially supported nor recommendedanymore. Luckily, in case you really need the embedded instance, for example while slowly migrating from older Elasticsearchreleases, it is still possible.@Configurationpublic class ElasticsearchEmbeddedConfiguration {private static class EmbeddedNode extends Node {public EmbeddedNode(Settings preparedSettings) (preparedSettings, ));}}@Bean(initMethod "start", destroyMethod "stop")Node elasticSearchTestNode() throws NodeValidationException, IOException {return new TRANSPORT TYPE KEY, "netty4").put(NetworkModule.HTTP TYPE KEY, "netty4").put(NetworkModule.HTTP ENABLED.getKey(), "true").put(Environment.PATH HOME SETTING.getKey(), home().getAbsolutePath()).put(Environment.PATH DATA SETTING.getKey(), data().getAbsolutePath()).build());}@BeanFile home() throws IOException {return oFile();}@BeanFile data() throws IOException {return oFile();}@PreDestroyvoid destroy() throws IOException {FileSystemUtils.deleteRecursively(home());

Elasticsearch Tutorial9 / ough this code snippet is based on the terrific Spring Framework, the idea is pretty simple and could be used in any JVMbased application. With t

Elasticsearch Tutorial v Preface Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is relea