This project was inspired by the original privateGPT (https://github.com/imartinez/privateGPT). Most of the description here is inspired by the original privateGPT.

In this model, I have replaced the GPT4ALL model with Vicuna-7B model and we are using the InstructorEmbeddings instead of LlamaEmbeddings as used in the original privateGPT. Both Embeddings as well as LLM will run on GPU instead of CPU. It also has CPU support if you do not have a GPU (see below for instruction).

Ask questions to your documents without an internet connection, using the power of LLMs. 100% private, no data leaves your execution environment at any point. You can ingest documents and ask questions without an internet connection!

Built with LangChain and Vicuna-7B and InstructorEmbeddings

In order to set your environment up to run the code here, first install all requirements:

pip install -r requirements.txt

This repo uses a Constitution of USA as an example.

Put any and all of your .txt, .pdf, or .csv files into the SOURCE_DOCUMENTS directory in the load_documents() function, replace the docs_path with the absolute path of your source_documents directory.

Run the following command to ingest all the data.

python ingest.py

It will create an index containing the local vectorstore. Will take time, depending on the size of your documents. You can ingest as many documents as you want, and all will be accumulated in the local embeddings database. If you want to start from an empty database, delete the index.

Note: When you run this for the first time, it will download take time as it has to download the embedding model. In the subseqeunt runs, no data will leave your local enviroment and can be run without internet connection.

In order to ask a question, run a command like:

python run_localGPT.py

And wait for the script to require your input.

> Enter a query:

Hit enter. Wait while the LLM model consumes the prompt and prepares the answer. Once done, it will print the answer and the 4 sources it used as context from your documents; you can then ask another question without re-running the script, just wait for the prompt again.

Note: When you run this for the first time, it will need internet connection to download the vicuna-7B model. After that you can turn off your internet connection, and the script inference would still work. No data gets out of your local environment.

Type exit to finish the script.

By default, localGPT will use your GPU to run both the ingest.py and run_localGPT.py scripts. But if you do not have a GPU and want to run this on CPU, now you can do that (Warning: Its going to be slow!). You will need to use --device_type cpuflag with both scripts.

For Ingestion run the following:

python ingest.py --device_type cpu

In order to ask a question, run a command like:

python run_localGPT.py --device_type cpu

Selecting the right local models and the power of LangChain you can run the entire pipeline locally, without any data leaving your environment, and with reasonable performance.

• ingest.py uses LangChain tools to parse the document and create embeddings locally using InstructorEmbeddings. It then stores the result in a local vector database using Chroma vector store.
• run_localGPT.py uses a local LLM (Vicuna-7B in this case) to understand questions and create answers. The context for the answers is extracted from the local vector store using a similarity search to locate the right piece of context from the docs.
• You can replace this local LLM with any other LLM from the HuggingFace. Make sure whatever LLM you select is in the HF format.

To use this software, you must have Python 3.10 or later installed. Earlier versions of Python will not compile.

If you encounter an error while building a wheel during the pip install process, you may need to install a C++ compiler on your computer.

To install a C++ compiler on Windows 10/11, follow these steps:

1. Install Visual Studio 2022.
2. Make sure the following components are selected:
   • Universal Windows Platform development
   • C++ CMake tools for Windows
3. Download the MinGW installer from the MinGW website.
4. Run the installer and select the "gcc" component.

Follow this page to install NVIDIA Drivers.

1- Follow this [page] (https://developer.apple.com/metal/pytorch/) to build up PyTorch with Metal Performance Shaders (MPS) support. PyTorch uses the new MPS backend for GPU training acceleration. It is good practice to verify mps support using a simple Python script as mentioned in the provided link.

2- By following the page, here is an example of you may initiate in your terminal

% xcode-select --install % conda install pytorch torchvision torchaudio -c pytorch-nightly % pip install chardet % pip install cchardet % pip uninstall charset_normalizer % pip install charset_normalizer % pip install pdfminer.six % pip install xformers

3- Create a new "verifymps.py" in the same directory (localGPT) where you have all files and environment.

import torch if torch.backends.mps.is_available(): mps_device = torch.device("mps") x = torch.ones(1, device=mps_device) print (x)

else: print ("MPS device not found.")

4- Find "instructor.py" and open it in VS Code to edit.

The "instructor.py" is probably embeded similar to this: file_path = "/System/Volumes/Data/Users/USERNAME/anaconda3/envs/LocalGPT/lib/python3.10/site-packages/InstructorEmbedding/instructor.py"

Create a find.py file and run it to place the instructor.py

import os

def find_python_file(root_directory, target_file): for directory, subdirectories, files in os.walk(root_directory): if target_file in files: return os.path.join(directory, target_file) return None

result = find_python_file('/', 'instructor.py') # Replace with your target file name if result: print(f"File found at: {result}") else: print("File not found.")

Once you find it, you can open the "instrictor.py" and then edit it using this code:

Create a open.py and then run the following code: import os subprocess.run(["open", "-a", "Visual Studio Code", file_path])

Once you open "instructor.py" with VS Code, replace the code snippet that has "device_type" with the following codes:

     if device is None:
        device = self._target_device

    # Replace the line: self.to(device)

    if device in ['cpu', 'CPU']:
        device = torch.device('cpu')

    elif device in ['mps', 'MPS']:
        device = torch.device('mps')
    
    else:
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    self.to(device)

This is a test project to validate the feasibility of a fully local solution for question answering using LLMs and Vector embeddings. It is not production ready, and it is not meant to be used in production. Vicuna-7B is based on the Llama model so that has the original Llama license.