vllm

Public APIs for LLM inference including offline batch inference (LLM class) and OpenAI-compatible HTTP/WebSocket server with FastAPI endpoints

Engine orchestration layer coordinating request scheduling, KV cache management, model execution, and output processing with support for both sync and async modes

Pluggable attention backends including FlashAttention, FlashInfer, MLA variants, and Triton implementations with CUDA graph and quantization support

Model architectures, neural network layers, and weight loading including transformer models (Llama, Mixtral, Qwen2), quantization methods, MoE layers, and LoRA support

GPU interconnects, NCCL-based collective operations, KV cache transfer for disaggregated inference, and tensor/pipeline parallelism communication

System configuration, sampling parameters, environment variables, logging, tracing, and plugin system for extensibility

Multimodal input processing for vision-language models including image/video/audio encoding, placeholder management, and processor integration