Zheng Duan
|
c9e2a963e0
|
feat: add kv cache aware router (#3831)
* kv cache aware router
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
* add tests
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
* router config
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
* eviction test
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
add test
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
* eviction detect in worker test
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
* move worker tests to single gpu
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
* reduce memory fraction
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
* fix partial block
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
---------
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
|
2025-05-12 07:23:57 -04:00 |
|
Zheng Duan
|
ae48abefc1
|
bind block key and hasher (#3712)
|
2025-04-21 18:50:57 +08:00 |
|
Zheng Duan
|
bce7ea8c38
|
test: add kv cache event tests for disagg workers (#3602)
|
2025-04-18 18:30:19 +08:00 |
|
Zheng Duan
|
b0cb963199
|
test: torch-flow conditional disagg test (#3410)
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
|
2025-04-15 10:54:14 +08:00 |
|