vLLM is an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, an attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM serving: it delivers up to 24x higher throughput than HuggingFace Transformers, without requiring any model architecture changes.

GitHub: https://github.com/vllm-project