WIXLIB

optimize APIPart II.168Search in Depth12. Structured Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173Finding Exact Valuesterm Filter with Numbersterm Filter with TextInternal Filter OperationCombining FiltersBool FilterNesting Boolean FiltersFinding Multiple Exact ValuesContains, but Does Not EqualEquals ExactlyRangesRanges on DatesRanges on StringsDealing with Null Valuesexists Filtermissing Filterexists/missing on ObjectsAll About CachingIndependent Filter CachingControlling CachingFilter 19019119219219319413. Full-Text Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197Term-Based Versus Full-TextThe match QueryIndex Some DataA Single-Word QueryMultiword QueriesImproving PrecisionControlling PrecisionCombining QueriesScore CalculationControlling PrecisionHow match Uses boolBoosting Query ClausesControlling Analysisviii Table of Contents197199199200201202203204205205206207209

Default AnalyzersConfiguring Analyzers in PracticeRelevance Is Broken!21121321414. Multifield Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217Multiple Query StringsPrioritizing ClausesSingle Query StringKnow Your DataBest Fieldsdis max QueryTuning Best Fields Queriestie breakermulti match QueryUsing Wildcards in Field NamesBoosting Individual FieldsMost FieldsMultifield MappingCross-fields Entity SearchA Naive ApproachProblems with the most fields ApproachField-Centric QueriesProblem 1: Matching the Same Word in Multiple FieldsProblem 2: Trimming the Long TailProblem 3: Term FrequenciesSolutionCustom all Fieldscross-fields QueriesPer-Field BoostingExact-Value 123223223323323423523523623823915. Proximity Matching. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241Phrase MatchingTerm PositionsWhat Is a PhraseMixing It UpMultivalue FieldsCloser Is BetterProximity for RelevanceImproving PerformanceRescoring ResultsFinding Associated Words242242243244245246247249249250Table of Contents ix

Producing ShinglesMultifieldsSearching for ShinglesPerformance25125225325516. Partial Matching. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257Postcodes and Structured Dataprefix Querywildcard and regexp QueriesQuery-Time Search-as-You-TypeIndex-Time OptimizationsNgrams for Partial MatchingIndex-Time Search-as-You-TypePreparing the IndexQuerying the FieldEdge n-grams and PostcodesNgrams for Compound Words25825926026226426426526526727027117. Controlling Relevance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275Theory Behind Relevance ScoringBoolean ModelTerm Frequency/Inverse Document Frequency (TF/IDF)Vector Space ModelLucene’s Practical Scoring FunctionQuery Normalization FactorQuery CoordinationIndex-Time Field-Level BoostingQuery-Time BoostingBoosting an Indext.getBoost()Manipulating Relevance with Query StructureNot Quite Notboosting QueryIgnoring TF/IDFconstant score Queryfunction score QueryBoosting by Popularitymodifierfactorboost modemax boostBoosting Filtered Subsetsx Table of 291291293294296298299301301

filter Versus queryfunctionsscore modeRandom ScoringThe Closer, The BetterUnderstanding the price ClauseScoring with ScriptsPluggable Similarity AlgorithmsOkapi BM25Changing SimilaritiesConfiguring BM25Relevance Tuning Is the Last 10%Part III.302303303303305308308310310313314315Dealing with Human Language18. Getting Started with Languages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319Using Language AnalyzersConfiguring Language AnalyzersPitfalls of Mixing LanguagesAt Index TimeAt Query TimeIdentifying LanguageOne Language per DocumentForeign WordsOne Language per FieldMixed-Language FieldsSplit into Separate FieldsAnalyze Multiple TimesUse n-grams32032132332332432432532632732932932933019. Identifying Words. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333standard Analyzerstandard TokenizerInstalling the ICU Plug-inicu tokenizerTidying Up Input TextTokenizing HTMLTidying Up Punctuation33333433533533733733820. Normalizing Tokens. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341In That Case341Table of Contents xi

You Have an AccentRetaining MeaningLiving in a Unicode WorldUnicode Case FoldingUnicode Character FoldingSorting and CollationsCase-Insensitive SortingDifferences Between LanguagesUnicode Collation AlgorithmUnicode SortingSpecifying a LanguageCustomizing Collations34234334634734935035135335335435535821. Reducing Words to Their Root Form. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359Algorithmic StemmersUsing an Algorithmic StemmerDictionary StemmersHunspell StemmerInstalling a DictionaryPer-Language SettingsCreating a Hunspell Token FilterHunspell Dictionary FormatChoosing a StemmerStemmer PerformanceStemmer QualityStemmer DegreeMaking a ChoiceControlling StemmingPreventing StemmingCustomizing StemmingStemming in situIs Stemming in situ a Good 7237337522. Stopwords: Performance Versus Precision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377Pros and Cons of StopwordsUsing StopwordsStopwords and the Standard AnalyzerMaintaining PositionsSpecifying StopwordsUsing the stop Token FilterUpdating StopwordsStopwords and Performancexii Table of Contents378379379380380381383383

and Operatorminimum should matchDivide and ConquerControlling PrecisionOnly High-Frequency TermsMore Control with Common TermsStopwords and Phrase QueriesPositions DataIndex OptionsStopwordscommon grams Token FilterAt Index TimeUnigram QueriesBigram Phrase QueriesTwo-Word PhrasesStopwords and 339439423. Synonyms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395Using SynonymsFormatting SynonymsExpand or contractSimple ExpansionSimple ContractionGenre ExpansionSynonyms and The Analysis ChainCase-Sensitive SynonymsMultiword Synonyms and Phrase QueriesUse Simple Contraction for Phrase QueriesSynonyms and the query string QuerySymbol Synonyms39639739839839940040140140240440540524. Typoes and Mispelings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409FuzzinessFuzzy QueryImproving PerformanceFuzzy match QueryScoring FuzzinessPhonetic MatchingPart IV.409410411412413413AggregationsTable of Contents xiii

25. High-Level Concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419BucketsMetricsCombining the Two42042042026. Aggregation Test-Drive. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423Adding a Metric to the MixBuckets Inside BucketsOne Final Modification42642742927. Building Bar Charts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43328. Looking at Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437Returning Empty BucketsExtended ExampleThe Sky’s the Limit43944144329. Scoping Aggregations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44530. Filtering Queries and Aggregations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449Filtered QueryFilter BucketPost FilterRecap44945045145231. Sorting Multivalue Buckets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453Intrinsic SortsSorting by a MetricSorting Based on “Deep” Metrics45345445532. Approximate Aggregations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457Finding Distinct CountsUnderstanding the Trade-offsOptimizing for SpeedCalculating PercentilesPercentile MetricPercentile RanksUnderstanding the Trade-offs45846046146246446746933. Significant Terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471significant terms DemoRecommending Based on Popularityxiv Table of Contents472474

Recommending Based on Statistics47834. Controlling Memory Use and Latency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481FielddataAggregations and AnalysisHigh-Cardinality Memory ImplicationsLimiting Memory UsageFielddata SizeMonitoring fielddataCircuit BreakerFielddata FilteringDoc ValuesEnabling Doc ValuesPreloading FielddataEagerly Loading FielddataGlobal OrdinalsIndex WarmersPreventing Combinatorial ExplosionsDepth-First Versus 9649850050235. Closing Thoughts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507Part V.Geolocation36. Geo-Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511Lat/Lon FormatsFiltering by Geo-Pointgeo bounding box FilterOptimizing Bounding Boxesgeo distance FilterFaster Geo-Distance Calculationsgeo distance range FilterCaching geo-filtersReducing Memory UsageSorting by DistanceScoring by Distance51151251351451551651751751952052237. Geohashes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523Mapping Geohashesgeohash cell Filter524525Table of Contents xv

38. Geo-aggregations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527geo distance Aggregationgeohash grid Aggregationgeo bounds Aggregation52753053239. Geo-shapes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535Mapping geo-shapesprecisiondistance error pctIndexing geo-shapesQuerying geo-shapesQuerying with Indexed ShapesGeo-shape Filters and CachingPart VI.536536537537538540541Modeling Your Data40. Handling Relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545Application-side JoinsDenormalizing Your DataField CollapsingDenormalization and ConcurrencyRenaming Files and DirectoriesSolving Concurrency IssuesGlobal LockingDocument LockingTree Locking54654854955255555555655755841. Nested Objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561Nested Object MappingQuerying a Nested ObjectSorting by Nested FieldsNested Aggregationsreverse nested AggregationWhen to Use Nested Objects56356456556756857042. Parent-Child Relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571Parent-Child MappingIndexing Parents and ChildrenFinding Parents by Their Childrenmin children and max childrenFinding Children by Their Parentsxvi Table of Contents572572573575575

Children AggregationGrandparents and GrandchildrenPractical ConsiderationsMemory UseGlobal Ordinals and LatencyMultigenerations and Concluding Thoughts57657757957958058043. Designing for Scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583The Unit of ScaleShard OverallocationKagillion ShardsCapacity PlanningReplica ShardsBalancing Load with ReplicasMultiple IndicesTime-Based DataIndex per Time FrameIndex TemplatesRetiring DataMigrate Old IndicesOptimize IndicesClosing Old IndicesArchiving Old IndicesUser-Based DataShared IndexFaking Index per User with AliasesOne Big UserScale Is Not 596597597600601602Part VII. Administration, Monitoring, and Deployment44. Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607Marvel for MonitoringCluster HealthDrilling Deeper: Finding Problematic IndicesBlocking for Status ChangesMonitoring Individual Nodesindices SectionOS and Process SectionsJVM SectionThreadpool Section607608609611612613616617620Table of Contents xvii

FS and Network SectionsCircuit BreakerCluster StatsIndex StatsPending Taskscat API62262262362362462645. Production Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631HardwareMemoryCPUsDisksNetworkGeneral ConsiderationsJava Virtual MachineTransport Client Versus Node ClientConfiguration ManagementImportant Configuration ChangesAssign NamesPathsMinimum Master NodesRecovery SettingsPrefer Unicast over MulticastDon’t Touch These Settings!Garbage CollectorThreadpoolsHeap: Sizing and SwappingGive Half Your Memory to LuceneDon’t Cross 32 GB!Swapping Is the Death of PerformanceFile Descriptors and MMapRevisit This List Before 3863964064064164164264264464564646. Post-Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647Changing Settings DynamicallyLoggingSlowlogIndexing Performance TipsTest Performance ScientificallyUsing and Sizing Bulk RequestsStorageSegments and Mergingxviii Table of Contents647648648649650650651651

OtherRolling RestartsBacking Up Your ClusterCreating the RepositorySnapshotting All Open IndicesSnapshotting Particular IndicesListing Information About SnapshotsDeleting SnapshotsMonitoring Snapshot ProgressCanceling a SnapshotRestoring from a SnapshotMonitoring Restore OperationsCanceling a RestoreClusters Are Living, Breathing 4Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665Table of Contents xix

ForewordOne of the most nerve-wracking periods when releasing the first version of an opensource project occurs when the IRC channel is created. You are all alone, eagerly hop‐ing and wishing for the first user to come along. I still vividly remember those days.One of the first users that jumped on IRC was Clint, and how excited was I. Well for a brief period, until I found out that Clint was actually a Perl user, no less workingon a website that dealt with obituaries. I remember asking myself why couldn’t we getsomeone from a more “hyped” community, like Ruby or Python (at the time), and aslightly nicer use case.How wrong I was. Clint ended up being instrumental to the success of Elasticsearch.He was the first user to roll out Elasticsearch into production (version 0.4 no less!),and the interaction with Clint was pivotal during the early days in shaping Elastic‐search into what it is today. Clint has a unique insight into what is simple, and he isvery rarely wrong, which has a huge impact on various usability aspects of Elastic‐search, from management, to API design, to day-to-day usability features. It was a nobrainer for us to reach out to Clint and ask if he would join our company immedi‐ately after we formed it.Another one of the first things we did when we formed the company was offer publictraining. It’s hard to express how nervous we were about whether or not peoplewould even sign up for it.We were wrong.The trainings were and still are a rave success with waiting lists in all major cities.One of the people who caught our eye was a young fellow, Zach, who came to one ofour trainings. We knew about Zach from his blog posts about using Elasticsearch(and secretly envied his ability to explain complex concepts in a very simple manner)and from a PHP client he wrote for the software. What we found out was that Zachhad actually paid to attend the Elasticsearch training out of his own pocket! You can’txxi

really ask for more than that, and we reached out to Zach and asked if he would joinour company as well.Both Clint and Zach are pivotal to the success of Elasticsearch. They are wonderfulcommunicators who can explain Elasticsearch from its high-level simplicity, to its(and Apache Lucene’s) low-level internal complexities. It’s a unique skill that wedearly cherish here at Elasticsearch. Clint is also responsible for the Elasticsearch Perlclient, while Zach is responsible for the PHP one - both wonderful pieces of code.And last, both play an instrumental role in most of what happens daily with the Elas‐ticsearch project itself. One of the main reasons why Elasticsearch is so popular is itsability to communicate empathy to its users, and Clint and Zach are both part of thegroup that makes this a reality.xxii Foreword

PrefaceThe world is swimming in data. For years we have been simply overwhelmed by thequantity of data flowing through and produced by our systems. Existing technologyhas focused on how to store and structure warehouses full of data. That’s all well andgood—until you actually need to make decisions in real time informed by that data.Elasticsearch is a distributed, scalable, real-time search and analytics engine. It ena‐bles you to search, analyze, and explore your data, often in ways that you did notanticipate at the start of a project. It exists because raw data sitting on a hard drive isjust not useful.Whether you need full-text search, real-time analytics of structured data, or a combi‐nation of the two, this book introduces you to the fundamental concepts required tostart working with Elasticsearch at a basic level. With these foundations laid, it willmove on to more-advanced search techniques, which you will need to shape thesearch experience

in 2010. When Elasticsearch formed a company in 2012, he joined as a developer and the maintainer of the Perl modules. Zachary Tong has been working with Elasticsearch since 2011, and has written several tutorials to help beginners using the server. Zach is a developer at Elasticsearch and maintains the PHP client. Elasticsearch: The Definitive .