diff --git a/docs/models/pooling_models/README.md b/docs/models/pooling_models/README.md index 2a5357e4fee..d9ce27dd216 100644 --- a/docs/models/pooling_models/README.md +++ b/docs/models/pooling_models/README.md @@ -184,7 +184,7 @@ Our online Server provides endpoints that correspond to the offline APIs: - Corresponding to `LLM.classify`: - [Classification API](classify.md#online-serving)(`/classify`) - Corresponding to `LLM.score`: - - [Score API](scoring.md#score-api)(`/score`) + - [Score API](scoring.md#score-api) (`/score`, `/v1/score`) - [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) - Pooling API (`/pooling`) is similar to `LLM.encode`, being applicable to all types of pooling models. diff --git a/docs/models/pooling_models/scoring.md b/docs/models/pooling_models/scoring.md index c8b4c73cfb3..a4b0fe5d2ea 100644 --- a/docs/models/pooling_models/scoring.md +++ b/docs/models/pooling_models/scoring.md @@ -19,7 +19,7 @@ The score models is designed to compute similarity scores between two input prom - Offline APIs: - `LLM.score` - Online APIs: - - [Score API](scoring.md#score-api) (`/score`) + - [Score API](scoring.md#score-api) (`/score`, `/v1/score`) - [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) !!! note @@ -157,7 +157,7 @@ A code example can be found here: [examples/basic/offline_inference/score.py](.. ### Score API -Our Score API (`/score`) is similar to `LLM.score`, compute similarity scores between two input prompts. +Our Score API (`/score`, `/v1/score`) is similar to `LLM.score`, compute similarity scores between two input prompts. #### Parameters diff --git a/docs/serving/online_serving/README.md b/docs/serving/online_serving/README.md index 9fa1763108c..40fc8b7c426 100644 --- a/docs/serving/online_serving/README.md +++ b/docs/serving/online_serving/README.md @@ -9,12 +9,13 @@ We currently support the following OpenAI APIs: - [Completions API](./openai_compatible_server.md#completions-api) (`/v1/completions`) - Only applicable to [text generation models](../../models/generative_models.md). - *Note: `suffix` parameter is not supported.* -- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`) - - Only applicable to [text generation models](../../models/generative_models.md). - [Chat Completions API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions`) - Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](./openai_compatible_server.md#chat-template). - *Note: `user` parameter is ignored.* - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls. +- [Chat Completions batch API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions/batch`) +- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`) + - Only applicable to [text generation models](../../models/generative_models.md). - [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) - Only applicable to [embedding models](../../models/pooling_models/embed.md). - [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`) @@ -24,7 +25,7 @@ We currently support the following OpenAI APIs: ## Anthropic APIs -- Anthropic messages API (`/v1/messages`) +- Anthropic messages API (`/v1/messages`, `/v1/messages/count_tokens`) ## Cohere APIs @@ -35,10 +36,6 @@ We currently support the following OpenAI APIs: - Implements [Jina AI's v1 rerank API](https://jina.ai/reranker/) - compatible with [Cohere's v1 & v2 rerank APIs](https://docs.cohere.com/v2/reference/rerank) -## SageMaker APIs - -- `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints) - ## Pooling APIs For further details on pooling models, please refer to [this page](../../models/pooling_models/README.md). @@ -51,7 +48,7 @@ For further details on pooling models, please refer to [this page](../../models/ - [OpenAI-compatible Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) - Only applicable to [embedding models](../../models/pooling_models/embed.md). - [Scoring Usages](../../models/pooling_models/scoring.md) - - [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`) + - [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`, `/v1/score`) - [Cohere Rerank API](../../models/pooling_models/scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) - Applicable to [score models](../../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction). - [Pooling API](../../models/pooling_models/README.md#pooling-api) (`/pooling`) @@ -68,17 +65,6 @@ For further details on speech to text, please refer to [this page](speech_to_tex - [Realtime API](./speech_to_text.md#realtime-api) (`/v1/realtime`) - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#realtime-transcription). -## Disaggregated APIs - -### Renderer APIs - -For further details on renderer APIs, please refer to [this page](renderer.md). - -- [Completions Render API](renderer.md) (`/v1/completions/render`) - - Render completion requests -- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`) - - Render chat completions - ## Custom APIs - [Classification API](../../models/pooling_models/classify.md#classification-api) (`/classify`) @@ -91,14 +77,79 @@ For further details on renderer APIs, please refer to [this page](renderer.md). - Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`). - Computes next-token probabilities for specified `label_token_ids`. -## Utility APIs +## Instrumentator APIs + +### Basic APIs + +- `/version` - Version information +- `/load` - Server load metrics +- `/v1/models` - List available models +- `/health` - Health check + +### Metrics APIs + +For further details on metrics, please refer to [this page](../../design/metrics.md). + +- `/metrics` - Prometheus-compatible metrics HTTP endpoint + +### Offline API Documentation + +The FastAPI `/docs` endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the `--enable-offline-docs` flag: + +```bash +vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs +``` + +### LoRA dynamic loading + +LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development! + +- `/v1/load_lora_adapter` - LoRA dynamic loading +- `/v1/unload_lora_adapter` - LoRA dynamic unloading + +### Profiling APIs + +For further details on profiling vLLM, please refer to [this page](../../contributing/profiling.md). + +- `/start_profile` - Start PyTorch profiler +- `/stop_profile` - Stop PyTorch profiler + +### SageMaker APIs + +- `/ping` - SageMaker health check +- `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints) + +## Disaggregated Everything + +### Tokens IN <> Tokens OUT + +- `/inference/v1/generate` - Generate completions +- `/abort_requests` - Abort in-flight requests (only when `--tokens-only` is also set) + +### Renderer APIs + +For further details on renderer APIs, please refer to [this page](renderer.md). + +- [Completions Render API](renderer.md) (`/v1/completions/render`) + - Render completion requests +- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`) + - Render chat completions + +### Derenderer APIs + +- `/v1/completions/derender` - Derenderer completion requests +- `/v1/chat/completions/derender` - Derenderer chat completion requests + +## Tokenize APIs - `/tokenize` - Tokenize text - `/detokenize` - Detokenize tokens -- `/health` - Health check -- `/ping` - SageMaker health check -- `/version` - Version information -- `/load` - Server load metrics +- `/tokenizer_info` - Get comprehensive tokenizer information including chat templates and configuration + +## Elastic Expert Parallelism (EEP) + +- `/scale_elastic_ep` - Trigger scaling operations +- `/is_scaling_elastic_ep` - Check if scaling is in progress ## Server in development mode @@ -120,7 +171,9 @@ For further details on Weight Transfer, please refer to [this page](../../traini - `/resume` - Resume generation - `/is_paused` - Check if generation is paused - `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF +- `/start_weight_update` - Prepares the inference engine for a weight update. - `/update_weights` - Update model weights (can alter model behavior) +- `/finish_weight_update` - Finalizes the weight update - `/get_world_size` - Get distributed world size ### Collective RPC @@ -189,14 +242,6 @@ the detected format, which can be one of: If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument to override which format to use. -## Offline API Documentation - -The FastAPI `/docs` endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the `--enable-offline-docs` flag: - -```bash -vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs -``` - ## Ray Serve LLM Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure. diff --git a/docs/serving/online_serving/openai_compatible_server.md b/docs/serving/online_serving/openai_compatible_server.md index 245de012bff..e50754aa9c0 100644 --- a/docs/serving/online_serving/openai_compatible_server.md +++ b/docs/serving/online_serving/openai_compatible_server.md @@ -9,12 +9,13 @@ We currently support the following OpenAI APIs: - [Completions API](#completions-api) (`/v1/completions`) - Only applicable to [text generation models](../../models/generative_models.md). - *Note: `suffix` parameter is not supported.* -- [Responses API](#responses-api) (`/v1/responses`) - - Only applicable to [text generation models](../../models/generative_models.md). - [Chat Completions API](#chat-api) (`/v1/chat/completions`) - Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](../online_serving/README.md#chat-template). - *Note: `user` parameter is ignored.* - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls. +- [Chat Completions batch API](#chat-api) (`/v1/chat/completions/batch`) +- [Responses API](#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`) + - Only applicable to [text generation models](../../models/generative_models.md). - [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) - Only applicable to [embedding models](../../models/pooling_models/embed.md). - [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`)