It is possible to notice that retrieving an high number of fields leads to a sensible worsening of performance if DocValues are used. fields marked as “stored” regardless their docValues settings)

Prior to Solr 7, for each SolrDocument instance to be returned in the results the .DocsStreamer class, actual responsible for fetching fields values, executes a sequential scan of we have to make a distinction before and after Solr 7 because an optimisation that has been introduced.Īctually, more than the optimisation itself, the new RetrieveFieldOptimizer class in Solr 7 created a centralised point of responsibility where further enhancements could be implemented in the next future. The third case is a little bit tricky because, as said above, those two options could be both enabled (i.e. With that assumption, case 1 and 2 are easy because stored and docValues fields are two disjoint set: in the first case Solr will use the stored values while in the second scenario will make use of the docValues structure. fl contains a mix of stored and docValues fields?īeing the two options not exclusive, in the first two cases above we assume we have one option enabled and the other disabled.fl contains only docValues (stored=false).One may be wondering: what happens in Solr at retrieval time when Moreover, because the values are sparse in memory, reading more values will cause cache misses and this will degrade performance. However, because we can read directly the value of interest, we can imagine that reading only one field value using docvalues is more performant than using stored fields while reading more fields requires to repeat the same procedure more times. We don’t want to have a pointer for each docvalue, otherwise too much space would be used only for pointers.The field could be empty for some documents.Find the address of document X in the column of field A.įinding the right address is actually a tricky operation because of two reasons:.string), each value has a different size and it is not predictable how much space it is required for each element.Īccessing the value of field A for a document X can be done as following: For each document in the collection, we store consecutively the values of the field A according with the Lucene document id. If you want to return docvalues with fl=*, you need to specify this in the field definition: useDocValuesAsStored=”true” (note this is the default value starting from schema version 1.6)Īs anticipated, fields are serialised by column.Instead, multivalued fields with docValues enabled internally use a Set (SORTED_SET or SORTED_NUMERIC depending on their type), a data structure which pre-orders its member and removes duplicates. Multivalued fields are returned using a different order: when you use stored fields, they are returned in the order they have been indexed (the holding data structure acts as a List).If your field has text type and you want to return it, you are forced to set it as stored. Text fields cannot be set as docvalues.There are 3 important differences in behaviour to take in consideration: This is an important factor to take into account when we define our solr schema.ĭocValues can be used in place of stored fields for retrieving the fields value at query time, but they are serialised in a column fashion. This implies that returning all the stored fields in a row should have more or less the same cost of returning only a subset. Because we have a pointer only at the beginning of the row, we have to read all the stored values in the row at the worst case.

Read all the values in the rows until you reach the value of the field A.

Let’s say you want to retrieve the field A from the document with Lucene id lid: The pointers to each row are stored for allowing fast access to them. string or text fields have variable size). Each row may have a different size dependently on the number of fields defined for that document and data types (e.g. The rows are then stored sequentially on disk according with their Lucene doc id. This means that, given a set of fields, for each document, the values of these fields are concatenated in a row. Stored fields are organised in a row manner.

Apache lucene doc