Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install hnswlib fail on python 3.9 #442

Open
taqihaider7 opened this issue Feb 28, 2023 · 50 comments
Open

pip install hnswlib fail on python 3.9 #442

taqihaider7 opened this issue Feb 28, 2023 · 50 comments

Comments

@taqihaider7
Copy link

`Building wheels for collected packages: hnswlib
Building wheel for hnswlib (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for hnswlib (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [5 lines of output]
running bdist_wheel
running build
running build_ext
building 'hnswlib' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for hnswlib
Failed to build hnswlib
ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects`

I have installed the latest Microsoft Visual C++ redistributions despite of that it is not installing

@yurymalkov
Copy link
Member

It seems like something is wrong with the environment. E.g. not running from the command line with MS VC paths set.

@taqihaider7
Copy link
Author

any solution to fix that problem

@yurymalkov
Copy link
Member

Can you check your path variables? Does cl.exe work?

@dcydrob
Copy link

dcydrob commented Mar 27, 2023

I don't know if this is at all helpful to you all, but I (on a Mac) also got failures running
pip install hnswlib
and
python3.9 -m pip install hnswlib
but it worked when I used python 3.7:
python3.7 -m pip install hnswlib
(I don't happen to have 3.8 on my machine, so I didn't try that...)

@anentropic
Copy link

anentropic commented Apr 3, 2023

I have this problem with Python 3.10.3 on macOS 13.2.1 (M1)

...
        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:610:45: error: unknown type name 'nullptr_t'
        operator!=(const unique_ptr<_T1, _D1>& __x, nullptr_t) _NOEXCEPT
                                                    ^
        fatal error: too many errors emitted, stopping now [-ferror-limit=]
        20 errors generated.
        error: command '/usr/bin/clang' failed with exit code 1
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for hnswlib
  Failed to build hnswlib
  ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I'm not using hnswlib directly, but it's needed as a dependency of another package (chroma-core/chroma#250)

Are there some instructions for how to install the python lib successfully?

@anentropic
Copy link

more details around the error:

building 'hnswlib' extension
creating build
creating build/temp.macosx-12.3-arm64-cpython-310
creating build/temp.macosx-12.3-arm64-cpython-310/python_bindings
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-tyw8alra/overlay/lib/python3.10/site-packages/pybind11/include -I/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-tyw8alra/overlay/lib/python3.10/site-packages/numpy/core/include -I./hnswlib/ -I/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/include -I/Users/anentropic/.pyenv/versions/3.10.3/include/python3.10 -c ./python_bindings/bindings.cpp -o build/temp.macosx-12.3-arm64-cpython-310/./python_bindings/bindings.o -O3 -stdlib=libc++ -mmacosx-version-min=10.7 -DVERSION_INFO=\"0.7.0\" -std=c++14 -fvisibility=hidden
In file included from ./python_bindings/bindings.cpp:1:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iostream:41:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/ios:221:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__locale:18:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:841:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/copy.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/unwrap_iter.h:13:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__iterator/iterator_traits.h:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__iterator/incrementable_traits.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/concepts:133:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__concepts/arithmetic.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/type_traits:421:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/invoke.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/apply_cv.h:16:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/remove_reference.h:13:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/cstddef:50:9: error: no member named 'nullptr_t' in the global namespace
using ::nullptr_t;
      ~~^

I know basically nothing about C++

but googling I have learned that nullptr_t was added to the standard in C++11

and we are calling clang with -stdlib=libc++ -std=c++14, which seems like it should work

@anentropic
Copy link

no idea if this is relevant, but I found a similar looking issue here: chipsalliance/verible#260

and they fixed it by adding #include <cstddef> at the top of one of their header files and prefixing as std::nullptr_t
chipsalliance/verible@241da92

@bdunsmuir
Copy link

I have the same problem using python 3.9

Failed to build hnswlib
ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

@snayan06
Copy link

snayan06 commented Apr 5, 2023

same is happening as @anentropic mentioned I was installing chromadb and got a failure with python 3.9, any solutions people have found on this? which can help

@dcydrob
Copy link

dcydrob commented Apr 5, 2023

Hey all. I looked in the code and saw there’s a check for an environment variable. Try
export HNSWLIB_NO_NATIVE=1
and try install again.

@anentropic
Copy link

UsingHNSWLIB_NO_NATIVE=1 I get a different error:

  Command ['/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10', '--no-deps', '/Users/anentropic/Library/Caches/pypoetry/artifacts/d2/32/ed/8674b10968a423c6f1b64821759e55a782ff3b28da34e2a886e85d54a9/hnswlib-0.7.0.tar.gz'] errored with the following return code 1, and output:
  Processing /Users/anentropic/Library/Caches/pypoetry/artifacts/d2/32/ed/8674b10968a423c6f1b64821759e55a782ff3b28da34e2a886e85d54a9/hnswlib-0.7.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error

    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [16 lines of output]
        Traceback (most recent call last):
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
            main()
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
            return self._get_build_requires(config_settings, requirements=['wheel'])
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
            self.run_setup()
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
            exec(code, locals())
          File "<string>", line 74, in <module>
          File "<string>", line 91, in BuildExt
        ValueError: list.remove(x): x not in list
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

@snayan06
Copy link

snayan06 commented Apr 5, 2023

@dcydrob i did that , and added it in ~/.bashrc file still the same error . did it worked for u . am i missing something .

this is how my bashrc looks like . i also tried directly in terminal as well same results

# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi
export HNSWLIB_NO_NATIVE=1
Building wheels for collected packages: hnswlib
  Building wheel for hnswlib (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for hnswlib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      running bdist_wheel
      running build
      running build_ext
      creating tmp
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/engati/faq-indexer/venv/include -I/usr/local/include/python3.9 -c /tmp/tmpfi1kk7tb.cpp -o tmp/tmpfi1kk7tb.o -std=c++14
      gcc: error: unrecognized command line option '-std=c++14'
      error: command '/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hnswlib
Failed to build hnswlib

@jeffleft
Copy link

jeffleft commented Apr 8, 2023

Might be that gcc can't find cc1plus. I had the same problem on Amazon linux and this stackoverflow post fixed it.

@josepmy
Copy link

josepmy commented Apr 11, 2023

Hi, Same problem here, python3.9 on macosx 13.3.1
export HNSWLIB_NO_NATIVE=1 from terminal before install works for me. Thanks 👍

@tourbut
Copy link

tourbut commented Apr 17, 2023

HI.
The same issue occurred in Windows 10, Python 3.9, and VSCODE. I resolved the issue by installing Microsoft C++ Build Tools (selecting the C++ build package during installation).

@OmarSultan85
Copy link

HI. The same issue occurred in Windows 10, Python 3.9, and VSCODE. I resolved the issue by installing Microsoft C++ Build Tools (selecting the C++ build package during installation).

Hey @tourbut , can you please advise which option from the vs installer exactly did u choose , thanks

@shivamMg
Copy link

For windows:

  1. Download Build Tools: https://visualstudio.microsoft.com/visual-cpp-build-tools/
  2. Open, and then in Workloads select "Desktop development with C++" to install:
    image

@bgarza
Copy link

bgarza commented Apr 29, 2023

I have the same problem using python 3.9

Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I have the same error on MacOS 13.1 and python 3.10. Setting the environment variable HNSWLIB_NO_NATIVE did not work for me. I do not have this problem on Windows 11.

@bgarza
Copy link

bgarza commented Apr 29, 2023

I have the same problem using python 3.9
Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I have the same error on MacOS 13.1 and python 3.10. Setting the environment variable HNSWLIB_NO_NATIVE did not work for me. I do not have this problem on Windows 11.

I was able to resolve this by upgrading XCode. I then built hnswlib by running setup.py, which I was not able to do prior. Upgrading XCode may have upgraded or installed Clang, which is now at 14.0.3 for me.

@yurymalkov
Copy link
Member

@bgarza what was the initial environment?
I do not know a way to reproduce this error on my mac.

@anentropic
Copy link

I have clang 14.0.3 and all the problems above:

$ clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0

I have not tried cloning hnswlib repo and building from source though

@bgarza
Copy link

bgarza commented May 1, 2023

@bgarza what was the initial environment? I do not know a way to reproduce this error on my mac.

I believe that XCode was out of date, but not sure at what version. Not sure why, but the XCode upgrade took a very long time via the App Store. After the upgrade I was able to build the project.

@yurymalkov
Copy link
Member

@anentropic I see. Do you use default system python or anaconda?
pip install hnswlibshould be pretty much the same thing as install from source (pip install git+https://github.com/nmslib/hnswlib.git)

@anentropic
Copy link

@anentropic I see. Do you use default system python or anaconda? pip install hnswlibshould be pretty much the same thing as install from source (pip install git+https://github.com/nmslib/hnswlib.git)

I'm not using anaconda

I am using pyenv python with a poetry virtualenv, I think my pyenv is installed via homebrew

@armalko
Copy link

armalko commented May 9, 2023

Was getting the same error while installing on Linux inside Docker Container:

In file included from ./python_bindings/bindings.cpp:6:
      ./hnswlib/hnswlib.h: At global scope:
      ./hnswlib/hnswlib.h:80:13: warning: ‘bool AVX512Capable()’ defined but not used [-Wunused-function]
         80 | static bool AVX512Capable() {
            |             ^~~~~~~~~~~~~
      gcc: fatal error: Killed signal terminated program cc1plus
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

It revealed that problem was in low RAM (1GB). As I created swap-file for another 1GB everything worked just fine.

I used this commands to create SWAP-memory, if anyone needs:

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

And deleted it afterwards with

sudo swapoff /swapfile
sudo rm /swapfile

@konykwong
Copy link

konykwong commented May 18, 2023

how to set the environment var on Windows?
export HNSWLIB_NO_NATIVE=1

what should i type in the terminal?

@ikebo
Copy link

ikebo commented May 20, 2023

export HNSWLIB_NO_NATIVE=1
export ARCHFLAGS="-arch x86_64"
and try again!!

@mariusorani
Copy link

Was getting the same error on mac using python 3.11 in a venv. Switching back to python 3.10 solved the problem

@sanskar-mk2
Copy link

how to set the environment var on Windows? export HNSWLIB_NO_NATIVE=1

what should i type in the terminal?

set HNSWLIB_NO_NATIVE=1 is the equivalent on windows terminal.

@dyashuni
Copy link
Contributor

MACOS installation is fixed in the develop branch
Could you try it?
pip install git+https://github.com/nmslib/hnswlib.git@develop

@yurymalkov
Copy link
Member

pip install git+https://github.com/nmslib/hnswlib.git@develop

This should also work without need for HNSWLIB_NO_NATIVE=1

Also please note that it seems like Windows problem is separate (unless you want to compile it for ARM) and should be solved by #442 (comment)

@peterk
Copy link

peterk commented May 30, 2023

MACOS installation is fixed in the develop branch
Could you try it?
pip install git+https://github.com/nmslib/hnswlib.git@develop

This solution worked for me on Apple M1 with clang 14.0.3 and python 3.10.5. I had the same problem as previous reporters above. Did not need to set HNSWLIB_NO_NATIVE flag either. Thank you!

Installed version was:

pip freeze|grep hnsw
hnswlib @ git+https://github.com/nmslib/hnswlib.git@a4c8b0b9e33a813d94477e7eee79e50c04c42577 

This also makes it possible to install chromadb without any issues. (see chroma-core/chroma#250)

@unmotivatedgene
Copy link

I am having this issue with windows and it was not resolved by installing the build tools or any other suggestion I have found. It works on my windows 11 machine but not my windows 10.

@dgm3333
Copy link

dgm3333 commented Jun 10, 2023

I'm also getting a similar fail (although with latest Python = 3.11) with the error:-
C1083: Cannot open include file: 'crtdbg.h': No such file or directory

Installation command lines tried were:-
python -m pip install hnswlib
python -m pip install . (from the git cloned directory as per the readme.md recommendation to build)

This occurs with a fresh install of Visual Studio 2022 Community 17.6.2 and Python 3.11 on Windows 10
crtdbg.h is present in the VS include tree (as well as various other locations eg StrawberryPerl)
C:\Program Files\Microsoft Visual Studio\2022\Community\SDK\ScopeCppSDK\vc15\SDK\include\ucrt\crtdbg.h

adding ucrt folder to the path also didn't help

  × Building wheel for hnswlib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running bdist_wheel
      running build
      running build_ext
      building 'hnswlib' extension
      creating build
      creating build\temp.win-amd64-cpython-311
      creating build\temp.win-amd64-cpython-311\Release
      creating build\temp.win-amd64-cpython-311\Release\python_bindings
      "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\dgm55\AppData\Local\Temp\pip-build-env-1uoe3q4k\overlay\Lib\site-packages\pybind11\include -IC:\Users\dgm55\AppData\Local\Temp\pip-build-env-1uoe3q4k\overlay\Lib\site-packages\numpy\core\include -I./hnswlib/ -IC:\Users\dgm55\AppData\Local\Programs\Python\Python311\include -IC:\Users\dgm55\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include" /EHsc /Tp./python_bindings/bindings.cpp /Fobuild\temp.win-amd64-cpython-311\Release\./python_bindings/bindings.obj /EHsc /openmp /O2 /DVERSION_INFO=\\\"0.7.0\\\"
      bindings.cpp
      C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\include\yvals.h(17): fatal error C1083: Cannot open include file: 'crtdbg.h': No such file or directory
      error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hnswlib
Failed to build hnswlib
ERROR: Failed to build one or more wheels

@crazygo
Copy link

crazygo commented Jul 20, 2023

My environment is :
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: x86_64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

MAC 2018:
processor: 2.3 GHz Quad-Core Intel Core i5

Python 3.8.16

Question:
I have the same problem when I install by
pip install git+https://github.com/nmslib/hnswlib.git@develop
error: no member named 'nullptr_t' in the global namespace using ::nullptr_t; ~~^

export HNSWLIB_NO_NATIVE=1 does not work for me

@Jero-W
Copy link

Jero-W commented Jul 26, 2023

python3.9 macos13.4.1 worked export HNSWLIB_NO_NATIVE=1

@gomestai
Copy link

@armalko solution worked thanks!!!

@l-cdc
Copy link

l-cdc commented Jul 28, 2023

In case it helps, I was able to make it build under Windows with these two packages:

  • MSVC v143 - VS 2022 C++ x64/x86 build tools
  • Windows 10 SDK

So you don't need to install the full "Desktop development with C++" workload.

@AmazeInteractive
Copy link

Hey all. I looked in the code and saw there’s a check for an environment variable. Try export HNSWLIB_NO_NATIVE=1 and try install again.

thnx... now I need to know why that worked

@tiro2000
Copy link

The following solution was of help to me resolving this after attempting every thing to install or uninstall ( sometimes that work ,sometimes no matter what you install SDK, .... ) , the main thing MS Windows have two Diffrent paths to play with , one which is the C:\Program Files (x86)\Microsoft Visual Studio\ , the other track is C:\Program Files (x86)\Windows Kits\10\bin\ , that is problametic ( at least how I see it ) , any ways I did the following if it would be of help to any one , thanks to Stackoverflow guys : https://stackoverflow.com/questions/14372706/visual-studio-cant-build-due-to-rc-exe/

Add this to your PATH environment variables:
C:\Program Files (x86)\Windows Kits\8.0\bin\x86

Copy these files:
rc.exe
rcdll.dll
From
C:\Program Files (x86)\Windows Kits\8.0\bin\x86 ( should be in your Path of course )
To
C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin

@mbintanveer
Copy link

For Linux users.

sudo apt-get install python3.11-dev

if chromadb is needed then sudo apt-get install chromadb

@pomcho555
Copy link

I got this error along with pipenv even after setting up export HNSWLIB_NO_NATIVE=1

Python 3.11.2

Failed case😡

pipenv install chromadb

Worked case🥰

pipenv shell && pip install chromadb

@DBairdME
Copy link

Hi, I'm also seeing hnswlib install issues with the following environment and install output

Environment:
Python 3.9.6 under RHEL 9, with pip v23.3.1
gcc v11.4.1, g++ v11.4.1

Output from pip:
(.venv) [XXXXXXX]$ pip3 install hnswlib
Collecting hnswlib
Using cached hnswlib-0.7.0.tar.gz (33 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy in ./.venv/lib/python3.9/site-packages (from hnswlib) (1.26.2)
Building wheels for collected packages: hnswlib
Building wheel for hnswlib (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for hnswlib (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [21 lines of output]
running bdist_wheel
running build
running build_ext
creating tmp
gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/xxxxxxxx/.venv/include -I/usr/include/python3.9 -c /tmp/tmpsdn20l8t.cpp -o tmp/tmpsdn20l8t.o -std=c++14
gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/xxxxxxxx/.venv/include -I/usr/include/python3.9 -c /tmp/tmpzeh61qro.cpp -o tmp/tmpzeh61qro.o -fvisibility=hidden
building 'hnswlib' extension
creating build
creating build/temp.linux-x86_64-cpython-39
creating build/temp.linux-x86_64-cpython-39/python_bindings
gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/tmp/pip-build-env-0h0zx8s6/overlay/lib/python3.9/site-packages/pybind11/include -I/tmp/pip-build-env-0h0zx8s6/overlay/lib64/python3.9/site-packages/numpy/core/include -I./hnswlib/ -I/home/xxxxxxxx/.venv/include -I/usr/include/python3.9 -c ./python_bindings/bindings.cpp -o build/temp.linux-x86_64-cpython-39/./python_bindings/bindings.o -O3 -fopenmp -DVERSION_INFO="0.7.0" -std=c++14 -fvisibility=hidden
In file included from /tmp/pip-build-env-0h0zx8s6/overlay/lib/python3.9/site-packages/pybind11/include/pybind11/detail/../attr.h:13,
from /tmp/pip-build-env-0h0zx8s6/overlay/lib/python3.9/site-packages/pybind11/include/pybind11/detail/class.h:12,
from /tmp/pip-build-env-0h0zx8s6/overlay/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h:13,
from /tmp/pip-build-env-0h0zx8s6/overlay/lib/python3.9/site-packages/pybind11/include/pybind11/functional.h:12,
from ./python_bindings/bindings.cpp:2:
/tmp/pip-build-env-0h0zx8s6/overlay/lib/python3.9/site-packages/pybind11/include/pybind11/detail/../detail/common.h:266:10: fatal error: Python.h: No such file or directory
266 | #include <Python.h>
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for hnswlib
Failed to build hnswlib
ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

@FelexTriz
Copy link

image
choose these 2 components and ,it works well on windows11
image

@DBairdME
Copy link

Interesting, thanks for the note. I'll continue to look to see if something similar works for RHEL, as I'm not using Windows in this case.

@VirgilG72
Copy link

Hey all. I looked in the code and saw there’s a check for an environment variable. Try export HNSWLIB_NO_NATIVE=1 and try install again.

It works for me on python3.7 ubuntu16.04 ! Thank you~

@dali2g
Copy link

dali2g commented Mar 14, 2024

Working on Windows 10 , and Python 3.12 , I've installed MSVC Tools and Windows 10 SDK and I'm still getting this error :
Building wheel for chroma-hnswlib (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for chroma-hnswlib (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [5 lines of output]
running bdist_wheel
running build
building 'hnswlib' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

@dltlaos11
Copy link

I'm a mac user and I switched from Python 3.12.xx to Python 3.11.xx and it works fine.

@SauceCat
Copy link

For those who are still struggling, hope this would help: try install chroma-hnswlib instead.
My case: I can install hnswlib without issue on my macbook, however, it kept failing on the remote server. Somehow I found out I could actually install chroma-hnswlib successfully on the server, so I just comment out the hnswlib in pyproject.toml 😹

python3 -m pip install chroma-hnswlib
python3 -c "import hnswlib; print('hnswlib imported successfully')"

https://pypi.org/project/chroma-hnswlib/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests