maybe we can fix the optimization to apply to Index-time sorting causes points to use more search-time heap.Worthwhile tradeoff for many users even CheckIndex time is substantially faster with index sorting Index-time sorting makes indexing ~50% slower, but gives.Indexing rate increases docs per MB increases flush and merge The indexing metrics improve nicely over time: index size drops.In the past, something git (amazingly) seems not to preserve.Īs typically happens when one creates a new benchmark, there are manyĬurious WTFs ("wow that's funny"s) jumps and drops that needĮxplaining, but here are some clear initial observations: Which git commit hash the master branch pointed to at different times Tricky since it required linearizing the git commit history to match
In time in the past and then run the benchmark, was particularly We also run a few basic searches over those three indices,īack testing, where we checkout the master sources at a specific point sparse-sorted, with index time sorting by cab color.dense, where all rides share a single set of fields and allĭocuments have exactly the same fields (100% fields are set).Green_fare_amount and yellow_fare_amount) sparse, where green and yellow taxi rides have their own fields.We index the same set of documents in three different ways: Green taxi rides are about 11.5% and yellow taxi ridesĪre around 88.5%, making a good test for mostly sparse and mostly
Single (anonymous!) taxi ride in New York City, either via a green or The benchmarks index a 20 M document subset from Set of nightly Lucene benchmarks on sparse and dense documents Use, just like the rest of Lucene's index parts.Īlong with these Lucene improvements we've added With all these improvements, and I'm sure many more to come, weįinally get Lucene's doc values to a point where only pay for what you Performance beyond where we were previously Use cases tested by Lucene's existing nightly benchmarks, and in some These changes brought back much of our search performance on the dense
APACHE LUCENE DOC FREE
So we are free to make major changes to the index formatĪbstraction for the common single-valued numerics and binary cases Hard to improve our default codec to take advantage of the more Performance gains as well, since the more restrictive API gives codecs See the size of your index suddenly grow 10-fold! But even denseĬases, where most documents have a value for the field, should see Lucene's non-sparse encoding of such fields has been particularly Sparse cases, where not all documents have a value for each doc valuesįield, should especially benefit from this change. Since this was aįirst as the existing Lucene codec had to use temporary silly wrapperĬlasses to translate its random-access API into an iterator API. That change was already massive enough that we decided to break outĪll such codec improvements to future issues. Other optimizations, like our postings implementations do. Restrictive access pattern than an arbitrary random access API, thisĬhange gives codecs more freedom to use aggressive compression and Previous random-access API to an iterator API instead. Switching out how doc values are accessed at search time from the , which was "simply" a low-level raw plumbing change using expressions to combine multiple signals into a score, or for sorting,įield/document holding index-time scoring signals (the field's length Row-stride fashion, and are therefore relatively slow to access.ĭoc values can be used to hold scoring signals, e.g. This is in contrast to Lucene's stored documentįields, which store all field values for one document together in a Quite fast to access at search time, since they are storedĬolumn-stride such that only the value for that one field needs to beĭecoded per hit. Multi-valued) and binary data blobs per document. Store numerics (single- or multi-valued), sorted keywords (single or These changes will be in Lucene's next major release (7.0) and will likely not be back-ported to any 6.x release, so it will be some time until Elasticsearch exposes this.ĭoc values are Lucene's column-stride field value storage, letting you These changes fix doc values so you only pay for what you use, just like all other parts of the Lucene index. Recently we've made some big changes to Apache Lucene around how doc values are indexed and searched, including new nightly benchmarks to measure our progress, based on the New York City taxi ride data corpus.
0 kommentar(er)
