Ray Serve LLM

An open-source Ray Serve framework for deploying large language models in production through an OpenAI-compatible API, with distributed multi-node serving, autoscaling, routing, parallelism, and prefill-decode disaggregation.