* server-stream : pimpl
* server-stream: prefix free functions with server_stream_
address review from ggerganov: scope the public stream functions under the
server_stream_ prefix, matching server_stream_session_manager_start/stop.
* server-stream: guard session and manager state with the mutex
address review from ggerganov: make done, completed_ts and the GC running flag plain members under their
mutex and set the condvar predicates under the lock. keep cancelled atomic for
the lock-free should_stop poll.
* server-stream: trim comments to the non-obvious
address review from ggerganov: drop comments that restate the code, keep the
concurrency, lifetime and ordering rationale. de-stale a few comments left by the
pimpl: g_stream_sessions is now internal and the /v1/streams listing is gone.
* server-stream: update dev docs for the pimpl and prefix
reflect server_stream_session_manager_start/stop and the server_stream_ prefix,
note the manager is now a file-static singleton hidden in the .cpp
* server-stream: move stream traces to debug level
keep the bring-up traces for diagnostics but off the default log: skip
drain, draining, drain ended, DELETE evict, attach_pipe, and the router
stream resume proxy.
* server-stream: align router stream resume proxy trace with upstream
the child-side bring-up traces are already SRV_TRC on master, move the
router stream resume proxy trace to the same level.
* server-stream: move stream_read_status enum to the cpp
it is only used by the hidden session and consumer types, so it belongs
with them behind the pimpl boundary, not on the public header surface.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* feat: WIP
* feat: Retire ChatScreenProcessingInfo component, context, and keepStatsVisible settings
* feat: Always-on gauge with active-model /props, conversation stats and live-reactive reading/output/avg
* feat: Add /tokenize endpoint, TokenizeService, FNV-1a and JSON Schema utilities
* feat: Surface enabled-tools token count in context hover card
* refactor(tools): make toolsStore the sole owner of the OpenAI wire format
Previously mcpStore.getToolDefinitionsForLLM() owned the MCP->OpenAI
shape conversion (plus normalizeSchemaProperties). That created two
sources of truth for what gets sent to the LLM, with the
duplication-prone risk of the deduplicated enabled list (which feeds
the token-count cache) drifting from the bytes actually shipped on
chat.
Now:
- mcpStore: pure protocol state + routing. Drop getToolDefinitionsForLLM
and the inline OpenAIToolDefinition conversion + normalizeSchemaProperties.
Doc comment adjusted to declare wire-format ownership as belonging
to toolsStore. Connection lifecycle, health checks, executeTool,
and the connections/toolsIndex remain.
- toolsStore: owns the wire shape (added earlier this series). mcpEntries()
inlines the MCP tool conversion; uses normalizeJsonSchema (the JSON
Schema util extracted in the prior commit) so missing 'type' fields
are inferred from defaults. mcpTools getter iterates mcpEntries() so
the Settings UI and the deduplicated enabled list see the same
definitions. getEnabledToolsForLLM iterates mcpEntries() instead of
calling mcpStore, so the JSON sent to the LLM is identical to what
toolsStore.refreshEnabledToolsTokenCount tokenizes.
- agentic: the chat-completion tools field's type was annotated as
ReturnType<typeof mcpStore.getToolDefinitionsForLLM>, claiming the
shape was owned by mcpStore. Switch to ReturnType<typeof
toolsStore.getEnabledToolsForLLM>, the actual source.
Assisted-by: Claude
* feat: UI WIP
* feat: UI WIP
* feat: UI WIP
* feat: Adjust reasoning submenu layout and spacing
* feat: Adjust context usage gauge thresholds and styling
* feat: Split context usage gauge stats into current and cumulative breakdowns
* chore: Format
* refactor: Cleanup
* refactor: Cleanup
* feat: improve token gauge accuracy and display
* refactor: remove MCP recommendation gating and simplify server visibility
* feat: add token audit logging to ChatStore for debugging
* refactor: Simplify context token reading to use server promptTokens directly
* feat: Replace last-known token tracking with live server-derived stats for accurate streaming gauges
* feat: UI Improvements
* feat: Move prompt processing stats to the preceding user message
* feat: Fix context token double-counting and refine gauge layout
* refactor: remove always-show-agentic-turns setting and simplify agentic turn display
* feat: track and display cache tokens in context gauge
* feat: add diagnostic logging for chat completion requests
* refactor: improve token audit console output with fresh/cached breakdown
* fix: invalidate enabled tools token count cache on tool changes
* test: add unit tests for tools store token count invalidation
* refactor: Remove tools token counting infrastructure
* refactor: Update ChatFormContextGauge to use simplified token tracking
* refactor: Update ChatStore to remove tools token counting
* chore: Formatting
* feat: Improve UI text
* feat: simplify context usage derivation and refine gauge labels
* refactor: cleanup logs
* cleaning
* fix: UI
* refactor: Enums
* refactor: Extract context gauge logic into hook and split UI into sub-components
* refactor: Cleanup comments
---------
Co-authored-by: Pascal <admin@serveurperso.com>
dict.get("key", default) returns None (not default) when the key
exists but its value is explicitly None. This caused an AttributeError
in _escape_html() when a task errored before grading and answer was
set to None.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* ggml : add support for CPU f16->f16 GGML_OP_SET_ROWS
* ggml : add missing type checks in f16 GGML_OP_SET_ROWS
* ggml : merge ggml_compute_forward_set_rows_f32() and ggml_compute_forward_set_rows_f16() into ggml_compute_forward_set_rows_impl()
* chore : replace assert() with GGML_ASSERT()
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* fix: draft model fit vs load inconsistency
* refactor(server): unify draft/mtp parameter initialization, model, and context load
- moves speculative init to speculative.cpp
- changes server_context_impl model_dft and ctx_dft to use raw pointers
- fix: don't throttle progress callback when loading draft model
- refactor: rename draft model/ctx load method
* fix: valign
Before this commit, --cache-ram was not a hard limit:
- The cache always kept at least one entry, even if that entry exceeded the
RAM/token limits.
- Old entries were only evicted for the RAM/token limits after saving the new
one, which could cause the cache to temporarily exceed the RAM/token limits
even if individual entries were below the limit.
Now, ensure that the RAM limit is strict with these changes:
- Skip saving state to cache if by itself it exceeds the RAM limit.
- Evict old entries as necessary to make the new entry fit.
Additionally, token-limit cleanup may now evict the last remaining cache entry
instead of always preserving one.
-ffast-math implies -ffinite-math-only under ROCm/clang 22, which
disables INFINITY/NaN and triggers -Wnan-infinity-disabled (errors
under -Werror in CI). Re-enable infinity handling without dropping
the rest of fast-math.
Fixes#25361
* support op col2im_1d
* update ops.md
* rm unused words
* update for bf16
* optimize 1%-11% as the review comments
* fix the format issue
* update as the review comments
* sycl: add supported types to ggml_sycl_supports_reorder_dmmv
The reordered feature is implemented in ggml_sycl_op_dequantize_mul_mat_vec,
but gated by ggml_sycl_supports_reorder_dmmv. This commit fixes the gate.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* sycl: set K_QUANTS_PER_ITERATION=1 to improve utilization
When combined with opening the reorder gate, this improves GPU
utilization on B70, giving a significant boost to tg t/s.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* sycl: replace QK_WARP_SIZE with WARP_SIZE for QK_5
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* sycl: add missing types to ggml_backend_sycl_buffer_init_tensor
Without this, the extra field is not allocated and the reorder path
will not take effect.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support
* chore : get rid of else
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV
* opencl: improve non FA KQ mv kernels
* opencl: tweaks for multiquery FA
* opencl: some tweaks for FA q1 kernels
* opencl: FA with DK=DV=512 for gemma-4
* opencl: various fixes
* opencl: cleanup
* opencl: fix FA decode crash for DK=512 (gemma-4)
The DK=512 decode-only program does not create the f32_f16 prefill
kernel, so the compiled check in ensure_fa_variant never hit and
supports_op gave inconsistent answers for the same op. block_n is also
unset for DK=512 decode; guard it to avoid an out-of-range read at
dispatch.
* opencl: run DK=512 FA decode on CPU
DK=512 decode is bandwidth-bound and faster on the CPU than the GPU,
increasingly so with depth. Decline it in supports_op; prefill stays on the GPU.
* opencl: compile MQ_GQA=8 FA kernels in a minimal program
The full program compiled with -D MQ_GQA=8 runs the Adreno compiler out
of memory at DK>=256. Only the vec_mq kernels are used from this
program, so compile it with FA_MQ_ONLY, which excludes everything else.
Also include the program name in the compile error log.
* opencl: remove stray token in flash_attn_f32_f16.cl
A stray "." broke the f32_f16 program build.
* opencl: split f16-KV FA decode finer (FD_KV_PER_SPLIT_F16)
The 2048 default under-fills the GPU on single-query f16-KV decode;
use 512 for f16 KV to get more splits. Quantized KV keeps 2048.
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* metal: add col2im_1d op (f32/f16/bf16)
Gather kernel mirroring the CPU/CUDA path: each output (t_out, oc)
reads its ceil(K/s0) source columns with an F32 accumulator, a single
write and no atomics. One thread per output element, 256 per
threadgroup.
* metal: check dst contiguity and type match in supports_op for COL2IM_1D
Align the GGML_OP_COL2IM_1D predicate with the CPU, CUDA, and Vulkan
backends: the kernel writes dst with linear indexing and assumes the
same type as src0, so supports_op must also require a contiguous dst
and op->type == op->src[0]->type.
* Update ggml/src/ggml-metal/ggml-metal.metal
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
---------
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
* server: fix deadlock in load_models() when erasing a finished download
The download monitoring thread acquires the models mutex on its way out,
but load_models() joined it from the erase loop while holding that mutex.
Join it outside the lock via threads_to_join like the other monitoring
threads.
* server: add default timeout to test requests
A hung server now fails the test after 10 minutes instead of stalling
the CI job for hours. Explicit timeouts are unchanged.
The matmul_tiled path uses large local stack buffers for A_pack and B_pack. On AIX this can trigger a segmentation fault, so reduce the buffer footprint there to keep the tiled path usable.
Performance Impact:
~ 2x gains in PP_Speed for FP32, Q4_0 and Q8_0 models tested with llama-bench, llama-batched-bench and llama-cli.
Models used: Llama3.2 3b Instruct F32, qwen 2.5 3b Q4_0 and Q8_0
* meta: fix tensor split metadata for GQA attention
* Tidied the code a bit to match existing style
* Revert "Tidied the code a bit to match existing style"
This reverts commit b90c6c6300.
* Reverted the ggml-backend-meta asset hack.
Oliver Simons