Description
Currently, for sparse fields (where not all documents have a value), we write two structures into the same .dvd file: the IndexedDISI presence bitmap followed by the packed values. IndexedDISI is a compressed list of docIds which helps in answering question like 'Does document X have a value for this field?'
We already separated out skip index into a different file - #15976. I wonder if we should do the same for indexedDISI structure? As for example, when we are reading skip index, and we hit a block which is YES_IF_PRESENT(all values are in range but some docs are missing), we need to seek into .dvd file to find which docs are present(link). I think this shouldn't be required ie interleaving this metadata with actual values.
We can either separate out this into a new file or stuff it inside skip index. I prefer the former as it is much clean.
Description
Currently, for sparse fields (where not all documents have a value), we write two structures into the same .dvd file: the IndexedDISI presence bitmap followed by the packed values. IndexedDISI is a compressed list of docIds which helps in answering question like 'Does document X have a value for this field?'
We already separated out skip index into a different file - #15976. I wonder if we should do the same for indexedDISI structure? As for example, when we are reading skip index, and we hit a block which is
YES_IF_PRESENT(all values are in range but some docs are missing), we need to seek into .dvd file to find which docs are present(link). I think this shouldn't be required ie interleaving this metadata with actual values.We can either separate out this into a new file or stuff it inside skip index. I prefer the former as it is much clean.