Adding search only mode for index loading to save memory

[jaroslavgratz]

If an index is loaded from disk and used to search only (likely a typical use case) then it is not necessary initialize link_list_locks_ and label_lookup_ data structures. It saves approximately 350MB of memory for 5M dataset. Adding a search_only flag to enable search only mode (default is off).

[yurymalkov]

Hi @jaroslavgratz ,

It seems that the code is missing something. Cannot see a part which omits creating the data structures.

[jaroslavgratz]

Hi @yurymalkov,

These lines omits creating the data structures when loading data:

fdfb030#diff-171628eaa21dab74ca44c386d5a17f05R684

fdfb030#diff-171628eaa21dab74ca44c386d5a17f05R699

In fact the link_list_locks_ and label_lookup_ data structures still exist but they are not loaded.

[yurymalkov]

@jaroslavgratz

Thanks, I see it. Does it give you a good saving?

There might be a simpler way to save the memory, while still preserving insertions. One can lock the elements in buckets. E.g. do link_list_locks_ [i<<8] instead of link_list_locks_ [i]. Not sure it has any measurable performance loss.

Also, the memory can be further saved if VisitedLists are substituted with hash maps. Would you like to see that in this repo?