Models & platforms
vLLM
An open-source engine for high-throughput large-language-model inference.
vLLM is an open-source engine for high-throughput large-language-model inference. Its reviewed product surface includes efficient llm serving and openai-compatible server and distributed execution. The primary documented workflow is to serve supported language models on controlled compute infrastructure.
Key features
- Efficient LLM serving
- OpenAI-compatible server and distributed execution
Pros and tradeoffs
Strengths
- High-throughput serving with a widely adopted open interface
Consider before choosing
- Hardware sizing and production operations require specialist knowledge
What people use vLLM for
- serve supported language models on controlled compute infrastructure
Frequently asked questions
What is vLLM best suited for?
vLLM is best evaluated for teams or individuals who need to serve supported language models on controlled compute infrastructure. Confirm current limits and terms on the official site.
