mirror of
https://github.com/vllm-project/vllm.git
synced 2026-08-19 04:00:20 +00:00
[Doc] Fix typos, grammar, and broken commands across docs (#46398)
Signed-off-by: MichaelCaoo <[email protected]> Co-authored-by: Claude <[email protected]>
This commit is contained in:
@@ -338,7 +338,7 @@ vllm bench serve \
|
||||
--model meta-llama/Meta-Llama-3-8B-Instruct \
|
||||
--dataset-name spec_bench \
|
||||
--dataset-path "<YOUR_DOWNLOADED_PATH>/data/spec_bench/question.jsonl" \
|
||||
--num-prompts -1
|
||||
--num-prompts -1 \
|
||||
--spec-bench-category "summarization"
|
||||
```
|
||||
|
||||
@@ -352,7 +352,7 @@ vllm bench serve \
|
||||
First, download the dataset to a folder, using this one liner:
|
||||
|
||||
```bash
|
||||
curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py \| python3 -
|
||||
curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py | python3 -
|
||||
```
|
||||
|
||||
The command supports also the following arguments:
|
||||
@@ -388,7 +388,7 @@ vllm bench serve \
|
||||
--model meta-llama/Llama-3.3-70B-Instruct \
|
||||
--dataset-name speed_bench \
|
||||
--dataset-path "<YOUR_DOWNLOADED_PATH>/data/speed_bench" \
|
||||
--num-prompts -1
|
||||
--num-prompts -1 \
|
||||
--speed-bench-category "multilingual"
|
||||
```
|
||||
|
||||
@@ -398,7 +398,7 @@ Run all categories in the Throughput split (2k ISL):
|
||||
vllm bench serve \
|
||||
--model meta-llama/Llama-3.3-70B-Instruct \
|
||||
--dataset-name speed_bench \
|
||||
--speed-bench-dataset-subset throughput_2k
|
||||
--speed-bench-dataset-subset throughput_2k \
|
||||
--dataset-path "<YOUR_DOWNLOADED_PATH>/data/speed_bench/" \
|
||||
--num-prompts -1
|
||||
```
|
||||
|
||||
@@ -109,7 +109,7 @@ from vllm import LLM
|
||||
|
||||
# Combine pipeline and tensor parallelism
|
||||
llm = LLM(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct,
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
tensor_parallel_size=4,
|
||||
pipeline_parallel_size=2,
|
||||
)
|
||||
|
||||
@@ -161,11 +161,11 @@ class AttentionCGSupport(enum.Enum):
|
||||
ALWAYS = 3
|
||||
"""CUDA Graphs always supported; supports mixed-prefill-decode"""
|
||||
UNIFORM_BATCH = 2
|
||||
"""CUDA Graphs supported for batches the only contain query lengths that are
|
||||
"""CUDA Graphs supported for batches that only contain query lengths that are
|
||||
the same, this can be used for spec-decode
|
||||
i.e. "decodes" are 1 + num_speculative_tokens"""
|
||||
UNIFORM_SINGLE_TOKEN_DECODE = 1
|
||||
"""CUDA Graphs supported for batches the only contain query_len==1 decodes"""
|
||||
"""CUDA Graphs supported for batches that only contain query_len==1 decodes"""
|
||||
NEVER = 0
|
||||
"""NO CUDA Graphs support"""
|
||||
```
|
||||
|
||||
@@ -685,7 +685,7 @@ documentation for this option states:
|
||||
> use of possibly costly and or blocking operations and hence might
|
||||
> have a performance impact.
|
||||
|
||||
The metrics were added by <https://github.com/vllm-project/vllm/pull/7089> and who up in an OpenTelemetry trace
|
||||
The metrics were added by <https://github.com/vllm-project/vllm/pull/7089> and show up in an OpenTelemetry trace
|
||||
as:
|
||||
|
||||
```text
|
||||
|
||||
@@ -27,7 +27,7 @@ In the example above, the KV cache in the first block can be uniquely identified
|
||||
For `vllm serve`, you can control the hashing algorithm via `--prefix-caching-hash-algo`:
|
||||
- `sha256` (default): Uses Python's `pickle` for serialization. Hashes may not be reproducible across different Python or vLLM versions.
|
||||
- `sha256_cbor`: Uses `cbor2` for serialization, providing a reproducible, cross-language compatible hash. This is recommended for deterministic caching across environments.
|
||||
- `xxhash`: `Uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on.
|
||||
- `xxhash`: Uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on.
|
||||
- `xxhash_cbor` combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package.
|
||||
|
||||
**A hashing example with multi-modality inputs**
|
||||
@@ -197,7 +197,7 @@ As can be seen, block 3 is a new full block and is cached. However, it is redund
|
||||
|
||||
When a request is finished, we free all its blocks if no other requests are using them (reference count = 0). In this example, we free request 1 and block 2, 3, 4, 8 associated with it. We can see that the freed blocks are added to the tail of the free queue in the *reverse* order. This is because the last block of a request must hash more tokens and is less likely to be reused by other requests. As a result, it should be evicted first.
|
||||
|
||||

|
||||

|
||||
|
||||
### Eviction (LRU)
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:
|
||||
|
||||
## Running a quantized model with vLLM
|
||||
|
||||
To run an GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:
|
||||
To run a GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:
|
||||
|
||||
```bash
|
||||
python examples/deployment/llm_engine_example.py \
|
||||
|
||||
@@ -78,8 +78,6 @@ def tokenize(sample):
|
||||
ds = ds.map(tokenize, remove_columns=ds.column_names)
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### 3. Applying Quantization
|
||||
|
||||
Now, apply the quantization algorithms:
|
||||
|
||||
@@ -190,7 +190,7 @@ For mitigation strategies, please refer to the FAQ entry *Can the output of a pr
|
||||
|
||||
## Known Feature Incompatibility
|
||||
|
||||
1. Pipeline parallelism is not composible with speculative decoding as of `vllm<=0.15.0`
|
||||
1. Pipeline parallelism is not composable with speculative decoding as of `vllm<=0.15.0`
|
||||
2. Speculative decoding with a draft models is not supported in `vllm<=0.10.0`
|
||||
|
||||
## Resources for vLLM contributors
|
||||
|
||||
@@ -338,7 +338,7 @@ Supported models:
|
||||
|
||||
Flags: `--tool-call-parser deepseek_v31 --chat-template {see_above}`
|
||||
|
||||
### OpenAI OSS Models ('openai`)
|
||||
### OpenAI OSS Models (`openai`)
|
||||
|
||||
Supported models:
|
||||
|
||||
|
||||
@@ -143,7 +143,7 @@ enabling the corresponding APIs.
|
||||
|
||||
The [classify][vllm.LLM.classify] method outputs a probability vector for each prompt.
|
||||
It is primarily designed for [classification models](classify.md).
|
||||
For more information about `LLM.embed`, see [this page](classify.md#offline-inference).
|
||||
For more information about `LLM.classify`, see [this page](classify.md#offline-inference).
|
||||
|
||||
### `LLM.embed`
|
||||
|
||||
@@ -302,7 +302,7 @@ Pooling models now support token-wise task.
|
||||
|
||||
### Score task
|
||||
|
||||
`score` task have has been removed in v0.21, use `classify` instead. Only when a classification model outputs num_labels
|
||||
`score` task has been removed in v0.21, use `classify` instead. Only when a classification model outputs num_labels
|
||||
equal to 1 can it be used as a scoring model and have its scoring API enabled.
|
||||
|
||||
### Pooling multitask support
|
||||
|
||||
@@ -440,7 +440,7 @@ More examples can be found here: [examples/pooling/score](../../../examples/pool
|
||||
|
||||
## Supported Features
|
||||
|
||||
AS cross-encoder models are a subset of classification models that accept two prompts as input and output num_labels equal to 1, cross-encoder features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).
|
||||
As cross-encoder models are a subset of classification models that accept two prompts as input and output num_labels equal to 1, cross-encoder features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).
|
||||
|
||||
### Score Template
|
||||
|
||||
|
||||
Reference in New Issue
Block a user