LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Header-only C++/python library for fast approximate nearest neighbors

Inference code for Llama models

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡