diff --git a/docs/serving/online_serving/README.md b/docs/serving/online_serving/README.md index 6d984f1a62d..5df1821f3b3 100644 --- a/docs/serving/online_serving/README.md +++ b/docs/serving/online_serving/README.md @@ -137,8 +137,12 @@ For further details on renderer APIs, please refer to [this page](renderer.md). ### Derenderer APIs -- `/v1/completions/derender` - Derenderer completion requests -- `/v1/chat/completions/derender` - Derenderer chat completion requests +For further details on derenderer APIs, please refer to [this page](derenderer.md). + +- [Chat Completions Derender API](derenderer.md) (`/v1/chat/completions/derender`) + - Derender chat completion requests +- [Completions Derender API](derenderer.md) (`/v1/completions/derender`) + - Derender completion requests ## Tokenize APIs diff --git a/docs/serving/online_serving/derenderer.md b/docs/serving/online_serving/derenderer.md new file mode 100644 index 00000000000..e55cd310949 --- /dev/null +++ b/docs/serving/online_serving/derenderer.md @@ -0,0 +1,98 @@ +# Derenderer APIs + +The derenderer API is the post processing counterpart to the [Renderer APIs](renderer.md). Where `/render` turns a request into token ID (preprocessing), `/derender` turns generated token IDs back into a fully formed OpenAI compatible response (detokenization, reasoning parsing, tool call parsing), all without a GPU. + +This closes the loop for a token-in / token-out engine in disaggregated serving: + +- **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render` +- **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server +- **Non-streaming**: The endpoints expect a complete `GenerateResponse` with all token IDs present and perform one-shot parsing. Streaming derender would require a separate endpoint design and is not currently supported but is in the pipeline + +Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render` +endpoints. + +## Pipeline + +```text + render generate derender + request ───────────────▶ token_ids ─────────▶ token_ids ──────────▶ response + (chat / (GPU less) (token-in / (GPU less) (OpenAI + completion) │ token-out engine) ▲ compatible) + └─────────────── request + prompt_tokens ──┘ +``` + +The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. + +## API Reference + +- Chat Completions Derender API (`/v1/chat/completions/derender`) + - Post process a single `GenerateResponse` into a `ChatCompletionResponse` +- Completions Derender API (`/v1/completions/derender`) + - Post process a list of `GenerateResponse` objects (one per prompt) into a `CompletionResponse` + +## Request format + +Each request wraps the engine's `GenerateResponse`(s) together with the caller metadata needed to reconstruct the final response without a GPU. + +`/v1/chat/completions/derender`: + +??? code + + ```python + --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-chat-request" + ``` + +`/v1/completions/derender`: + +??? code + + ```python + --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-completion-request" + ``` + +Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs. + +## Example + +The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`). + +```python +import httpx + +MODEL = "meta-llama/Llama-3.2-1B-Instruct" +RENDER = "http://localhost:8100" # vllm launch render ... +ENGINE = "http://localhost:8200" # token-in / token-out engine + +chat_request = { + "model": MODEL, + "messages": [{"role": "user", "content": "What is 2+2?"}], + "max_tokens": 32, +} + +with httpx.Client(timeout=60.0) as client: + # 1. Render: request -> token IDs (GPU less) + generate_request = client.post( + f"{RENDER}/v1/chat/completions/render", json=chat_request + ).json() + prompt_tokens = len(generate_request["token_ids"]) + + # 2. Generate: token IDs -> token IDs (token-in / token-out engine) + generate_response = client.post( + f"{ENGINE}/inference/v1/generate", json=generate_request + ).json() + + # 3. Derender: token IDs -> ChatCompletionResponse (GPU less) + response = client.post( + f"{RENDER}/v1/chat/completions/derender", + json={ + "model": MODEL, + "generate_response": generate_response, + "prompt_tokens": prompt_tokens, + "chat_request": chat_request, + }, + ).json() + +print(response["choices"][0]["message"]["content"]) +``` + +Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. Omit `chat_request` for plain detokenization only. diff --git a/docs/serving/online_serving/renderer.md b/docs/serving/online_serving/renderer.md index 9ea2f369db8..517d50f2d30 100644 --- a/docs/serving/online_serving/renderer.md +++ b/docs/serving/online_serving/renderer.md @@ -12,3 +12,5 @@ Our renderer API is designed to disaggregate the render phase(preprocessing) and - Render completion requests - [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`) - Render chat completions + +For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md). diff --git a/docs/usage/security.md b/docs/usage/security.md index ee49374e7b9..2a2e1e886d6 100644 --- a/docs/usage/security.md +++ b/docs/usage/security.md @@ -155,8 +155,10 @@ When `--api-key` is configured, the following `/v1` endpoints require Bearer tok - `/v1/chat/completions` - Chat completions - `/v1/chat/completions/batch` - Batch chat completions - `/v1/chat/completions/render` - Render chat completion requests +- `/v1/chat/completions/derender` - Derender chat completion requests - `/v1/completions` - Text completions - `/v1/completions/render` - Render completion requests +- `/v1/completions/derender` - Derender completion requests - `/v1/embeddings` - Generate embeddings - `/v1/audio/transcriptions` - Audio transcription - `/v1/audio/translations` - Audio translation diff --git a/tests/entrypoints/scale_out/derender/test_derender_parity.py b/tests/entrypoints/scale_out/derender/test_derender_parity.py new file mode 100644 index 00000000000..8e51ff5d42b --- /dev/null +++ b/tests/entrypoints/scale_out/derender/test_derender_parity.py @@ -0,0 +1,268 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Round trip parity CI: render -> generate -> derender -> render == render. + +Pins the coupled parsing path (``/v1/chat/completions`` on a normal GPU +server which parses generated tokens incrementally as they stream out) +against the disaggregated path (``OnlineDerenderer.derender_chat`` via +``/v1/chat/completions/derender`` which parses the same tokens all at once, +out of process). A standard ``vllm serve`` GPU server mounts both, so one +real generation lets both parsers run on identical input. + +Both paths consume the same generated token IDs (extracted from the coupled +response's ``token_ids`` via ``return_token_ids=True``), so generation +nondeterminism is irrelevant. The only variable under test is whether the +two parsing code paths agree. Parity is asserted unconditionally as only the +stronger per case assertions (e.g. "a tool call was produced") are gated +behind the marker actually having been emitted since a 1.5B model is not +guaranteed to emit ```` / ````. +""" + +import json + +import httpx +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +MODEL = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B" +ARGS = [ + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + "--reasoning-parser", + "deepseek_r1", +] + +TOOLS = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + }, + }, + } +] +FORCE_WEATHER_TOOL = {"type": "function", "function": {"name": "get_weather"}} + + +@pytest.fixture(scope="module") +def server(): + with RemoteOpenAIServer(MODEL, ARGS) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with httpx.AsyncClient( + base_url=server.url_for(""), timeout=60.0 + ) as http_client: + yield http_client + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +async def _coupled(client: httpx.AsyncClient, messages: list[dict], **extra) -> dict: + resp = await client.post( + "/v1/chat/completions", + json={ + "model": MODEL, + "messages": messages, + "temperature": 0, + "max_tokens": 128, + "return_token_ids": True, + **extra, + }, + ) + assert resp.status_code == 200, resp.text + return resp.json() + + +async def _disagg( + client: httpx.AsyncClient, + output_ids: list[int], + prompt_tokens: int, + finish_reason: str, + chat_request: dict, + logprobs: dict | None = None, +) -> dict: + resp = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL, + "generate_response": { + "request_id": "parity", + "choices": [ + { + "index": 0, + "token_ids": output_ids, + "finish_reason": finish_reason, + "logprobs": logprobs, + } + ], + }, + "prompt_tokens": prompt_tokens, + "chat_request": chat_request, + }, + ) + assert resp.status_code == 200, resp.text + return resp.json() + + +def _tool_sig(response_choice: dict) -> list[tuple[str, dict]]: + """[(name, json normalized args)] so key ordering / whitespace don't + cause false negatives.""" + return [ + (tc["function"]["name"], json.loads(tc["function"]["arguments"])) + for tc in (response_choice["message"].get("tool_calls") or []) + ] + + +def _assert_parity(coupled: dict, disagg: dict) -> None: + """Both paths saw the same tokens, so they must agree unconditionally.""" + c, d = coupled["choices"][0], disagg["choices"][0] + assert d["message"]["content"] == c["message"]["content"] + assert d["message"].get("reasoning") == c["message"].get("reasoning") + assert _tool_sig(d) == _tool_sig(c) + assert d["finish_reason"] == c["finish_reason"] + assert disagg["usage"]["prompt_tokens"] == coupled["usage"]["prompt_tokens"] + assert disagg["usage"]["completion_tokens"] == len(c["token_ids"]) + + +async def _run_parity_case( + client: httpx.AsyncClient, messages: list[dict], **extra +) -> tuple[dict, dict]: + """Run the coupled request then feed its generated tokens into the + disaggregated derender endpoint. Returns (coupled, disagg).""" + coupled = await _coupled(client, messages, **extra) + ch = coupled["choices"][0] + chat_request = {"model": MODEL, "messages": messages, **extra} + disagg = await _disagg( + client, + ch["token_ids"], + coupled["usage"]["prompt_tokens"], + ch["finish_reason"], + chat_request, + ) + return coupled, disagg + + +# --------------------------------------------------------------------------- +# Parity cases +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_parity_plain(client): + """Plain detokenization parity. No reasoning/tool markers involved.""" + messages = [ + {"role": "user", "content": "What is 2+2? Answer in one short sentence."} + ] + coupled, disagg = await _run_parity_case(client, messages) + _assert_parity(coupled, disagg) + + +@pytest.mark.asyncio +async def test_parity_reasoning(client): + """Reasoning/content split parity for ... outputs.""" + messages = [{"role": "user", "content": "What is 17 times 23? Think it through."}] + coupled, disagg = await _run_parity_case( + client, messages, include_reasoning=True, max_tokens=256 + ) + _assert_parity(coupled, disagg) + + if not coupled["choices"][0]["message"].get("reasoning"): + pytest.skip("Model did not emit a block") + assert disagg["choices"][0]["message"]["reasoning"] + + +@pytest.mark.asyncio +async def test_parity_tool_call(client): + """Tool call name+args parity.""" + messages = [{"role": "user", "content": "What's the weather in Paris?"}] + coupled, disagg = await _run_parity_case( + client, messages, tools=TOOLS, tool_choice=FORCE_WEATHER_TOOL, max_tokens=1024 + ) + _assert_parity(coupled, disagg) + + if not _tool_sig(coupled["choices"][0]): + pytest.skip("Model did not emit a tool call") + assert _tool_sig(disagg["choices"][0]) + + +@pytest.mark.asyncio +async def test_parity_reasoning_and_tool_call(client): + """Combined reasoning + tool call parity means the highest drift risk + since it exercises both parser branches on the same output.""" + messages = [{"role": "user", "content": "What's the weather in Paris?"}] + coupled, disagg = await _run_parity_case( + client, + messages, + tools=TOOLS, + tool_choice=FORCE_WEATHER_TOOL, + include_reasoning=True, + max_tokens=1024, + ) + _assert_parity(coupled, disagg) + + c_msg = coupled["choices"][0]["message"] + if not (c_msg.get("reasoning") and _tool_sig(coupled["choices"][0])): + pytest.skip("Model did not emit both a block and a tool call") + d_msg = disagg["choices"][0]["message"] + assert d_msg["reasoning"] + assert _tool_sig(disagg["choices"][0]) + + +@pytest.mark.asyncio +async def test_parity_logprobs(client): + """token_id:N resolution parity vs. the coupled server's real strings. + + A real disaggregated worker only has token IDs so it emits logprobs + with ``token_id:N`` placeholders (``return_tokens_as_token_ids=True`` + reproduces that shape here). ``/derender`` must resolve those + placeholders to the same token strings/bytes the coupled server + resolves them to directly. + """ + messages = [{"role": "user", "content": "What is 2+2?"}] + extra = {"logprobs": True, "top_logprobs": 3} + + # What a real GPU less worker would hand to /derender is token IDs plus + # logprobs still in token_id:N placeholder form + placeholder = await _coupled( + client, messages, return_tokens_as_token_ids=True, **extra + ) + ch = placeholder["choices"][0] + chat_request = {"model": MODEL, "messages": messages, **extra} + disagg = await _disagg( + client, + ch["token_ids"], + placeholder["usage"]["prompt_tokens"], + ch["finish_reason"], + chat_request, + logprobs=ch["logprobs"], + ) + + # The coupled server resolving the same greedy generation + # to real token strings itself. + resolved = await _coupled(client, messages, **extra) + assert resolved["choices"][0]["token_ids"] == ch["token_ids"], ( + "greedy (temperature=0) generation was expected to be deterministic " + "across the two coupled calls used to build this test's fixtures" + ) + _assert_parity(resolved, disagg) + + r_content = resolved["choices"][0]["logprobs"]["content"] + d_content = disagg["choices"][0]["logprobs"]["content"] + assert len(d_content) == len(r_content) + for d_entry, r_entry in zip(d_content, r_content): + assert d_entry["token"] == r_entry["token"] + assert d_entry["bytes"] == r_entry["bytes"] diff --git a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py index 11308d67c5e..c22e70b014c 100644 --- a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py +++ b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py @@ -266,8 +266,13 @@ class DerenderChatRequest(BaseModel): and ``parser.parse_delta()`` instead of ``parser.parse()``. """ + # --8<-- [start:derender-chat-request] model: str + """Served model name.""" + generate_response: GenerateResponse + """The complete token-in / token-out engine response to derender.""" + prompt_tokens: int | None = None """Prompt token count for usage; defaults to 0 if omitted. @@ -282,6 +287,7 @@ class DerenderChatRequest(BaseModel): request context they expect (request.tools, request.tool_choice, request._grammar_from_tool_parser, etc.). """ + # --8<-- [end:derender-chat-request] class DerenderCompletionRequest(BaseModel): @@ -292,8 +298,14 @@ class DerenderCompletionRequest(BaseModel): returned by /v1/completions/render. """ + # --8<-- [start:derender-completion-request] model: str + """Served model name.""" + generate_responses: list[GenerateResponse] + """One response per prompt, parallel to the list[GenerateRequest] + returned by /v1/completions/render.""" + prompt_tokens: list[int] | None = None """One prompt token count per response; each defaults to 0 if omitted. @@ -306,6 +318,7 @@ class DerenderCompletionRequest(BaseModel): Mirrors chat_request on DerenderChatRequest. Required by the parsing so parsers receive the full request context. """ + # --8<-- [end:derender-completion-request] @model_validator(mode="after") def _validate_prompt_tokens_length(self) -> "DerenderCompletionRequest": diff --git a/vllm/renderers/online_derenderer.py b/vllm/renderers/online_derenderer.py index 20c54eb07ff..3ce4f74d9d8 100644 --- a/vllm/renderers/online_derenderer.py +++ b/vllm/renderers/online_derenderer.py @@ -7,6 +7,7 @@ from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption from vllm.entrypoints.generate.base.serving import resolve_token_id_placeholder from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionLogProbs, + ChatCompletionNamedToolChoiceParam, ChatCompletionRequest, ChatCompletionResponseChoice, ChatMessage, @@ -136,6 +137,13 @@ class OnlineDerenderer: else [] ) + is_named_tool_choice = ( + type(chat_request.tool_choice) is ChatCompletionNamedToolChoiceParam + ) + is_required_tool_choice = chat_request.tool_choice == "required" + if is_named_tool_choice or is_required_tool_choice: + content = content or "" + message = ChatMessage( role="assistant", reasoning=reasoning,