pip installs for wrong architecture and does not build from source

[clstaudt]

Installation, trying to force build from source:

➤ pip install --no-binary hnswlib hnswlib
Collecting hnswlib
  Using cached hnswlib-0.7.0-cp310-cp310-macosx_11_0_arm64.whl
Collecting numpy (from hnswlib)
  Using cached numpy-1.24.3-cp310-cp310-macosx_11_0_arm64.whl (13.9 MB)
Installing collected packages: numpy, hnswlib
Successfully installed hnswlib-0.7.0 numpy-1.24.3

Test:

➤ python
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
i>>> import hnswlib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/cls/miniforge3/envs/hnswlib/lib/python3.10/site-packages/hnswlib.cpython-310-darwin.so, 0x0002): tried: '/Users/cls/miniforge3/envs/hnswlib/lib/python3.10/site-packages/hnswlib.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/cls/miniforge3/envs/hnswlib/lib/python3.10/site-packages/hnswlib.cpython-310-darwin.so' (no such file), '/Users/cls/miniforge3/envs/hnswlib/lib/python3.10/site-packages/hnswlib.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

[yurymalkov]

That seems like a compiler misconfiguration.
When you do pip install hnswlib it compiles the library from scratch using the default compiler.

I guess we should create binary wheels like manylinux, so it would not matter if user's environment is set correctly. If someone can help with instructions on creating those, that would be highly appreciated.

[clstaudt]

I was eventually able to install a working version with

pip install --no-binaries hnswlib hnswlib --no-cache-dir

This finally triggered the build step.

However, if hnswlib is a dependency of some other library, then pip will install mismatched binaries.

[jllllll]

That seems like a compiler misconfiguration. When you do pip install hnswlib it compiles the library from scratch using the default compiler.

I guess we should create binary wheels like manylinux, so it would not matter if user's environment is set correctly. If someone can help with instructions on creating those, that would be highly appreciated.

You can use a GitHub Actions workflow to automatically create wheels and publish them to PyPI using cibuildwheel:
https://cibuildwheel.readthedocs.io/en/stable/setup/#github-actions

They have an example workflow here: https://github.com/pypa/cibuildwheel/blob/main/examples/github-deploy.yml
Another example can be found here: https://github.com/abetlen/llama-cpp-python/blob/main/.github/workflows/build-and-release.yaml

[yurymalkov]

@jllllll thanks, will take a look!

[zhengzangw]

@clstaudt This does not work for me on mac M1.

Python: 3.9 / 3.10 / 3.11, even install from source.

[apang42]

This worked for me on a mac M1: zylon-ai/private-gpt#389 (comment)

# 1. Uninstall hnswlib
> pip uninstall hnswlib

# 2. Clear the pip cache
> pip cache purge

# 3. Reinstall with the arm64 architecture
> ARCHFLAGS="-arch arm64" pip install hnswlib

[riki976]

Here M1 , do not work:

$ pip3 uninstall hnswlib
Found existing installation: hnswlib 0.7.0
Uninstalling hnswlib-0.7.0:
  Would remove:
    /opt/homebrew/lib/python3.11/site-packages/hnswlib-0.7.0.dist-info/*
    /opt/homebrew/lib/python3.11/site-packages/hnswlib.cpython-311-darwin.so
Proceed (Y/n)? pip3 cache purge
Your response ('pip3 cache purge') was not one of the expected responses: y, n,
Proceed (Y/n)? y
  Successfully uninstalled hnswlib-0.7.0

$ ARCHFLAGS="-arch arm64" pip3 install hnswlib
Collecting hnswlib
  Using cached hnswlib-0.7.0-cp311-cp311-macosx_13_0_arm64.whl
Requirement already satisfied: numpy in /opt/homebrew/lib/python3.11/site-packages (from hnswlib) (1.25.1)
Installing collected packages: hnswlib
Successfully installed hnswlib-0.7.0

$ python3 -c "import hnswlib; print(hnswlib.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/opt/homebrew/lib/python3.11/site-packages/hnswlib.cpython-311-darwin.so, 0x0002): tried: '/opt/homebrew/lib/python3.11/site-packages/hnswlib.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/python3.11/site-packages/hnswlib.cpython-311-darwin.so' (no such file), '/opt/homebrew/lib/python3.11/site-packages/hnswlib.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

$ file /opt/homebrew/lib/python3.11/site-packages/hnswlib.cpython-311-darwin.so
/opt/homebrew/lib/python3.11/site-packages/hnswlib.cpython-311-darwin.so: Mach-O 64-bit bundle x86_64