[Question] Is infinity faster than sentence_transformers for inference? #327

GingerNg · 2024-07-28T14:19:24Z

GingerNg
Jul 28, 2024

2024-07-28T14:19:38Z

To compare the inference speed of Infinity and SentenceTransformers, follow these steps:

Setup Benchmark Environment: Use the provided simple_app.py script in /docs/benchmarks/ to benchmark both Infinity and SentenceTransformers.
Configure Environment Variables:
- Set BENCHMARK_NAME to infinity or sentence_transformers.
- Set DEVICE to cpu or cuda.
- Set BATCH_SIZE as needed.
Run Benchmark:
- For Infinity: BENCHMARK_NAME=infinity DEVICE=cpu BATCH_SIZE=32 python simple_app.py
- For SentenceTransformers: BENCHMARK_NAME=sentence_transformers DEVICE=cpu BATCH_SIZE=32 python simple_app.py
Measure Inference Time: Use the /embeddings endpoint to measure the time taken for embedding requests.

Refer to /libs/infinity_emb/tests/end_to_end/test_optimum_embedding.py for additional test cases and configurations.

This response provides a starting point for your research, not a precise solution.

Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

0 replies

michaelfeil · 2024-07-28T17:42:01Z

But it gets much faster if you implement no batching in your app.

0 replies