Release v0.8.0 #523

yurymalkov · 2023-11-19T20:18:49Z

Highlights:

Multi-vector document search and epsilon search (for now, only in C++)
By default, there is no statistic aggregation, which speeds up the multi-threaded search (it does not seem like people are using it anyway: Issue #495).
Various bugfixes and improvements
get_items now have return_type parameter, which can be either 'numpy' or 'list'

Huge thanks to the contributors:

Dmitry Yashunin (dyashuni) - Led the addition of a custom stop condition, multivector and epsilon search functionalities, and various code refinements and bug fixes.
Alexander Vieth (alxvth) - Contributed to resolving global linkage issues and improving Mac setup in the CI environment.
Taras Tsugrii (ttsugriy) - Focused on fixing build warnings, enhancing bruteforce removePoint functionality, and improving memory management.
aurora327 - Implemented more efficient AVX512 instruction sets in functions.
Étienne Mollier (emollier) - added a cap on the 'M' parameter and a typo.
Johan Rade (jrade) - Addressed a linking error when compiling with Visual Studio.
James Melville (jlmelville) - Addressed reordering warning, sign mismatch in loops, and standardized error handling using macros (Provide a macro to override the use of std::cerr #508 has the details)
stephematician - Resolved initialization order warning in GNU compilers.
moritz-h - Added CMake install targets and set CMake version range.
drons - Implemented functions for precise memory footprint control and fixed memory leak issues.
Atsushi Tatsuma (yoshoku) - Added a missing virtual destructor to BaseFilterFunctor.

#438: fix reordering warning

* Add CMake install targets * Set CMake version range

…print control

* Add multithread search for BF index

Add HierarchicalNSW::indexFileSize() function for precise memory footprint control

* Add macos into CI * Fix mac setup * Use macos-13 in CI * Allow asserts, fix checkIntegrity * Revert macos to latest in CI * Fix CI * Windows build warnings

Fix global linkage again

Replace priority_queue::push with emplace.

Fix build warnings.

…efficient AVX512 instruction set

…on consider the size of a Vector that is not divisible by 4

[bruteforce] Fix bruteforce removePoint.

Use unique_ptr to manage visited_list_pool_

Efficient AVX512 implementation in 'InnerProductSIMD16ExtAVX512' Function

Fix memory leak on loadIndex with non-empty HierarchicalNSW object

This patch works around issue #467, also referenced as CVE-2023-37365, by implementing Yury Malkov's suggestion about capping the M value, coding the maximum number of outgoing connections in the graph, to a reasonable enough value of the order of 100000. For the record, the documentation indicates reasonable values for M range from 2 to 100, which are well within the cap; see ALGO_PARAMS.md. The reproducer shown in issue #467 doesn't trigger the double free condition anymore after this change is applied, but completes successfully, although with the below warning popping up on purpose: warning: M parameter exceeds 100000 which may lead to adverse effects. Cap to 100000 will be applied for the rest of the processing. Signed-off-by: Étienne Mollier <[email protected]>

per comment in merge request discussion.

hnswalg.h: cap M to 100000

Fix get_items

Bring back HNSWLIB_NO_NATIVE

Signed-off-by: Étienne Mollier <[email protected]>

python_bindings/bindings.cpp: fix typo.

Fixes linking error when compiling with Visual Studio and including hnswlib.h in several cpp files. before: 2>Tools.lib(Pch.obj) : error LNK2005: "void __cdecl cpuid(int * const,int,int)" (?cpuid@@YAXQEAHHH@Z) already defined in Pch.obj 2>Tools.lib(NearestNeighbors.obj) : error LNK2005: "void __cdecl cpuid(int * const,int,int)" (?cpuid@@YAXQEAHHH@Z) already defined in Pch.obj 2>C:\Users\Rade\Documents\Data Analysis\Code\GenomicsLab Repo\x64\Release\ExpressionDemo.exe : fatal error LNK1169: one or more multiply defined symbols found 2>Done building project "ExpressionDemo.vcxproj" -- FAILED. after: 2>ExpressionDemo.vcxproj -> C:\Users\Rade\Documents\Data Analysis\Code\GenomicsLab Repo\x64\Release\ExpressionDemo.exe

GNU compilers no longer warn with -Wall or -Wreorder that initialisation order does not match declaration order in HierarchicalHNSW

Linking error fix

Provide a macro to override the use of std::cerr

Avoid sign mismatch in loop

Resolve initialisation order warning

* Add stop condition, multivector search, epsilon search * Fix include * Update readme * Update multivector tests * One header file * Add bare_bone_search flag * Fix epsilon search * Refactoring * Adress comments * Fix assert * Add ef to multivector search, return vector, refactoring * Refactoring * Adress comments * Add bare bone search comment Co-authored-by: Yury Malkov <[email protected]> * Remove has_deletions flag --------- Co-authored-by: Yury Malkov <[email protected]>

dyashuni

Looks good

jlmelville and others added 30 commits March 5, 2023 14:10

#438: fix reordering warning

361893c

Merge pull request #443 from jlmelville/438-gcc-reorder

b3fb472

#438: fix reordering warning

Add CMake install targets (#446)

dccd4f9

* Add CMake install targets * Set CMake version range

Add HierarchicalNSW::indexFileSize() function for precise memory foot…

a9de060

…print control

Add multithread search to BF index (#425)

6aac477

* Add multithread search for BF index

Merge pull request #427 from drons/indexFileSize

cd844b5

Add HierarchicalNSW::indexFileSize() function for precise memory footprint control

add virtual destructor to BaseFilterFunctor (#460)

1925428

Add static again

2c538db

Fix mac setup (#461)

a4c8b0b

* Add macos into CI * Fix mac setup * Use macos-13 in CI * Allow asserts, fix checkIntegrity * Revert macos to latest in CI * Fix CI * Windows build warnings

Merge pull request #463 from alxvth/fix_global_linkage_again

643e9dc

Fix global linkage again

[warnings] Fix build warnings.

47c0a32

Replace priority_queue::push with emplace.

3ca227a

[bruteforce] Fix bruteforce removePoint.

013def5

Merge pull request #472 from ttsugriy/emplace

b977d25

Replace priority_queue::push with emplace.

Merge branch 'nmslib:develop' into develop

70ce84e

Use unique_ptr to manage visited_list_pool_.

1ee95db

Merge pull request #471 from ttsugriy/warnings

802a0ec

Fix build warnings.

InnerProductSIMD16ExtAVX512 functions are implemented using the more …

74f14d2

…efficient AVX512 instruction set

InnerProductSIMD16ExtAVX512 Efficient AVX512 instruction implementati…

9291020

…on consider the size of a Vector that is not divisible by 4

Merge pull request #473 from ttsugriy/develop

e7c66c8

[bruteforce] Fix bruteforce removePoint.

Merge pull request #474 from ttsugriy/uniq-ptr

0df757e

Use unique_ptr to manage visited_list_pool_

Fix memory leak on loadIndex with non-empty HierarchicalNSW object

8911d9e

Merge pull request #475 from aurora327/efficient_avx512_instruction

f30b6e1

Efficient AVX512 implementation in 'InnerProductSIMD16ExtAVX512' Function

Merge pull request #477 from drons/fixLoadIndexLeak

006d7b2

Fix memory leak on loadIndex with non-empty HierarchicalNSW object

hnswalg.h: reduce M cap further to 10000

5aba6c6

per comment in merge request discussion.

Merge pull request #484 from emollier/cve-2023-37365

f6d170c

hnswalg.h: cap M to 100000

Bring back HNSWLIB_NO_NATIVE

08569de

Fix

e023f7e

get_items return numpy array

4f7b192

dyashuni and others added 20 commits August 12, 2023 15:03

Fix tests

f22a5b1

Rename flag, use class name to access static variables

eecd540

Add type check for return results of get_items

db19931

Merge pull request #494 from dyashuni/get_items_numpy

39b210d

Fix get_items

Merge pull request #493 from dyashuni/no_native

fd027d4

Bring back HNSWLIB_NO_NATIVE

python_bindings/bindings.cpp: fix typo.

f27913b

Signed-off-by: Étienne Mollier <[email protected]>

Merge pull request #500 from emollier/typo

ca85f0d

python_bindings/bindings.cpp: fix typo.

define a macro for error stream

5b3d81b

use HNSWERR instead of std::cerr directly

f7ec147

Add a comment

92e053a

Avoid sign mismatch in loop

431efa8

Resolve initialisation order warning

3de1d69

GNU compilers no longer warn with -Wall or -Wreorder that initialisation order does not match declaration order in HierarchicalHNSW

Add link to PR in comment

a9e62cb

Merge pull request #501 from jrade/master

ae5ba1b

Linking error fix

Merge pull request #508 from jlmelville/develop

c4418ea

Provide a macro to override the use of std::cerr

Merge pull request #509 from jlmelville/patch-1

898bf5d

Avoid sign mismatch in loop

Merge pull request #511 from stephematician/master

d44bd5d

Resolve initialisation order warning

update repo testing code

ba284f5

yurymalkov requested a review from dyashuni November 20, 2023 03:43

dyashuni approved these changes Dec 2, 2023

View reviewed changes

yurymalkov added 2 commits December 2, 2023 19:25

Bump library version

b9e0597

Update README.md

5a8fd34

yurymalkov merged commit 3f34296 into master Dec 3, 2023
30 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release v0.8.0 #523

Release v0.8.0 #523

yurymalkov commented Nov 19, 2023 •

edited

Loading

dyashuni left a comment

Release v0.8.0 #523

Release v0.8.0 #523

Conversation

yurymalkov commented Nov 19, 2023 • edited Loading

dyashuni left a comment

Choose a reason for hiding this comment

yurymalkov commented Nov 19, 2023 •

edited

Loading