vLLM

vLLM is an open-source, high-throughput and memory-efficient inference and serving engine/library for large language models, providing offline inference and online serving with OpenAI-compatible APIs across diverse hardware.