CUDA is compiled with fast math and AMD/HIP is not — this flag lets AMD use fast math too.
We can't use -ffast-math: it implies -ffinite-math-only, which won't compile (ggml uses INFINITY for masking) and produces NaNs. -funsafe-math-optimizations gives the speedup without the NaN problems.
Co-authored-by: Mark Caldwell <mark@cloudhands.ai>
* cuda: fix snake fusion type predicate, a and inv_b are F32
The matcher required a->type == x->type while launch_snake reads both
as const float *, matching the CPU and Metal contract where a and inv_b
stay F32. F16/BF16 chains never fused and fell back to the naive path,
and a hypothetical all F16 chain would have read F16 bits as float.
Aligns the predicate and the comment with ggml-cpu.c
* cuda: reject snake fusion on non-contiguous operands
The kernel reads x[idx] and a[c] / inv_b[c] linearly, so a
non-contiguous view passing the matcher would silently read wrong data.
Mirror the contiguity guard already present in the CPU, Vulkan and
Metal matchers.
* hexagon: add VISION RoPE support
* hexagon: support RoPE on strided half-dim views for all modes
* hex-rope: decouple src0 DMA copy size from row stride
* hex-rope: support non-contiguous dst for RoPE
* hex-rope: fix dst spad pitch for non-contiguous dst
* opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno
Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number.
Route Q6_K dense mul_mat with ne01 % 128 != 0 off the noshuffle path:
decode (ne1==1) uses the correct flat GEMV and the matching GEMM (ne1>1)
falls back to CPU (the flat convert has no verified small-batch GEMM kernel
for these shapes). All standard hidden/FFN/vocab dims are multiples of 128
and keep the noshuffle path.
* opencl: reserve alignment slack for the SOA subbuffer carve in alloc size
set_tensor carves quantized weights into per-component subbuffers (d/q,
ql/qh/s/d, ...) whose origins are each rounded up to the device base
address alignment. When a component's size is not a multiple of the
alignment, the carve extends past ggml_nbytes(tensor) and the last
subbuffer overlaps the next tensor in the pool -- e.g. q6_K [1536, 49155]:
size_s = 49155*96 ends 32 bytes past a 128-byte boundary, so the d
subbuffer ends 96 bytes past the tensor's allocation, and whichever of the
two neighboring tensors is uploaded last silently corrupts the other (here:
the last vocab rows' block scales). This affects any quant type whose
component sizes can be misaligned, on any shape with ne01 not a multiple of
the alignment granularity; standard power-of-two dims are unaffected.
Implement get_alloc_size for the OpenCL buffer type and reserve the
worst-case carve slack (4 aligned gaps; 5 components max, q5_K) for
quantized tensors. Costs at most 512 bytes per quantized tensor at the
observed 128-byte alignment.
* opencl: use lm based q6_k mm when ne1 is not multiple of 128
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* opencl: ragged-tile MoE prefill GEMM (skip padded expert tiles)
The MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles; at low
tokens-per-expert most tiles are mostly padding. When a tile's upper 16 slots
are all padding (router index 0xFFFFFFFF), skip the second dotx16_reduce8 half.
Numerically identical (skipped lanes are padding). Applied to all eight *_f32_ns
MoE GEMMs; default on, opt out with GGML_OPENCL_MOE_RAGGED_FP16=0.
* opencl: quarter-granularity ragged MoE tile-skip (8-col skip-groups)
Replace the two half-tile dotx16_reduce8 calls in the 8 *_f32_ns MoE GEMMs with
four dotx8_reduce4 (8-column) calls, skipping each empty trailing skip-group
independently. Padding is always trailing, so the kernel rounds the valid count
up to the skip granularity and skips fully-padding groups. Byte-identical to the
non-skipped path. New env GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32} (quarter/half/
off); default quarter.
* opencl: move ragged moe env var in cl_init
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hex-fa: refactor kernel param compute to use common layout builder
* hmx: add explicit compiler barriers to make hmx funcs more robust
* hex-vtcm: more generic vtcm layout builder for mm and flash-attn kernels
* hex-hmx: unroll inner kernels
* hex-hmx: use inline asm instead of intrinsics to avoid compiler issues
* hex-hmx: define inline asm macros and simplify code
* hex-hmx: replace leftover intrinsics
* hmx-fa: minor cleanup for hmx asm
* hmx-mm: move per-task stucts out of the kernels header
* hmx-mm: simplify core_dot_chunk
* hmx-mm: simplify inner loops that call hmx instructions
* hmx-mm: proper instrumentation for activation prep work for dma pipelined version
* hmx-mm: update a-prep loop for better prefetch
* hex-vtcm: improved vtcm layout alloc for mm to support overlapping areas
* hmx-mm: reduce the number of act fetch tows to 4 for now, going larger doesnt help here
* hex-hmx: always use hmx-queue in all modes
* hmx-mm: update comments and minor formatting
* hmx-mm: further improve synchro fallback path to prefetch the weights earlier
* hex-fa: further pipeline improvements (earlier prefetch)
* hmx-mm: cleanup dma pipelines to use dst cached in the queue
* hmx-fa: minor cleanup and opts for fa dma pipelines
* hmx-fa: optimize q-prep stage with dma and unrolling
* hmx-fa: use o_tile size from layout instead of computing it
* hmx-mm: cleanup types and size handling
* hmx-mm: replace divs with fastdiv in qprep loops
* hmx-fa: minor update/formatting to q_tile handling
* hmx-fa: cleanup the layout to avoid overpadding
* hmx-fa: simplified and improved cost mode for hmx fa solver that uses vtcm layout funcs
* hmx-queue: add support queue wakeup and make suspend async to avoid hmx-lock latency
* hex-hmx: move queue wakeup / suspend to the op-batch level
* hex-threads: add hybrid polling to workpool
* hex-mm: fix trailing spaces
* cli: move to HTTP-based implementation
* wip
* working
* remote server ok
* cli support router mode
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
* case: router with only one model
* Apply suggestions from code review
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
* remove outdated comment
* use destructor instead
* add ftype
* cli-view --> cli-ui
* pimpl
* no more json in header
* nits fixes
* also show model aliases
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
* server-stream : pimpl
* server-stream: prefix free functions with server_stream_
address review from ggerganov: scope the public stream functions under the
server_stream_ prefix, matching server_stream_session_manager_start/stop.
* server-stream: guard session and manager state with the mutex
address review from ggerganov: make done, completed_ts and the GC running flag plain members under their
mutex and set the condvar predicates under the lock. keep cancelled atomic for
the lock-free should_stop poll.
* server-stream: trim comments to the non-obvious
address review from ggerganov: drop comments that restate the code, keep the
concurrency, lifetime and ordering rationale. de-stale a few comments left by the
pimpl: g_stream_sessions is now internal and the /v1/streams listing is gone.
* server-stream: update dev docs for the pimpl and prefix
reflect server_stream_session_manager_start/stop and the server_stream_ prefix,
note the manager is now a file-static singleton hidden in the .cpp
* server-stream: move stream traces to debug level
keep the bring-up traces for diagnostics but off the default log: skip
drain, draining, drain ended, DELETE evict, attach_pipe, and the router
stream resume proxy.
* server-stream: align router stream resume proxy trace with upstream
the child-side bring-up traces are already SRV_TRC on master, move the
router stream resume proxy trace to the same level.
* server-stream: move stream_read_status enum to the cpp
it is only used by the hidden session and consumer types, so it belongs
with them behind the pimpl boundary, not on the public header surface.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* feat: WIP
* feat: Retire ChatScreenProcessingInfo component, context, and keepStatsVisible settings
* feat: Always-on gauge with active-model /props, conversation stats and live-reactive reading/output/avg
* feat: Add /tokenize endpoint, TokenizeService, FNV-1a and JSON Schema utilities
* feat: Surface enabled-tools token count in context hover card
* refactor(tools): make toolsStore the sole owner of the OpenAI wire format
Previously mcpStore.getToolDefinitionsForLLM() owned the MCP->OpenAI
shape conversion (plus normalizeSchemaProperties). That created two
sources of truth for what gets sent to the LLM, with the
duplication-prone risk of the deduplicated enabled list (which feeds
the token-count cache) drifting from the bytes actually shipped on
chat.
Now:
- mcpStore: pure protocol state + routing. Drop getToolDefinitionsForLLM
and the inline OpenAIToolDefinition conversion + normalizeSchemaProperties.
Doc comment adjusted to declare wire-format ownership as belonging
to toolsStore. Connection lifecycle, health checks, executeTool,
and the connections/toolsIndex remain.
- toolsStore: owns the wire shape (added earlier this series). mcpEntries()
inlines the MCP tool conversion; uses normalizeJsonSchema (the JSON
Schema util extracted in the prior commit) so missing 'type' fields
are inferred from defaults. mcpTools getter iterates mcpEntries() so
the Settings UI and the deduplicated enabled list see the same
definitions. getEnabledToolsForLLM iterates mcpEntries() instead of
calling mcpStore, so the JSON sent to the LLM is identical to what
toolsStore.refreshEnabledToolsTokenCount tokenizes.
- agentic: the chat-completion tools field's type was annotated as
ReturnType<typeof mcpStore.getToolDefinitionsForLLM>, claiming the
shape was owned by mcpStore. Switch to ReturnType<typeof
toolsStore.getEnabledToolsForLLM>, the actual source.
Assisted-by: Claude
* feat: UI WIP
* feat: UI WIP
* feat: UI WIP
* feat: Adjust reasoning submenu layout and spacing
* feat: Adjust context usage gauge thresholds and styling
* feat: Split context usage gauge stats into current and cumulative breakdowns
* chore: Format
* refactor: Cleanup
* refactor: Cleanup
* feat: improve token gauge accuracy and display
* refactor: remove MCP recommendation gating and simplify server visibility
* feat: add token audit logging to ChatStore for debugging
* refactor: Simplify context token reading to use server promptTokens directly
* feat: Replace last-known token tracking with live server-derived stats for accurate streaming gauges
* feat: UI Improvements
* feat: Move prompt processing stats to the preceding user message
* feat: Fix context token double-counting and refine gauge layout
* refactor: remove always-show-agentic-turns setting and simplify agentic turn display
* feat: track and display cache tokens in context gauge
* feat: add diagnostic logging for chat completion requests
* refactor: improve token audit console output with fresh/cached breakdown
* fix: invalidate enabled tools token count cache on tool changes
* test: add unit tests for tools store token count invalidation
* refactor: Remove tools token counting infrastructure
* refactor: Update ChatFormContextGauge to use simplified token tracking
* refactor: Update ChatStore to remove tools token counting
* chore: Formatting
* feat: Improve UI text
* feat: simplify context usage derivation and refine gauge labels
* refactor: cleanup logs
* cleaning
* fix: UI
* refactor: Enums
* refactor: Extract context gauge logic into hook and split UI into sub-components
* refactor: Cleanup comments
---------
Co-authored-by: Pascal <admin@serveurperso.com>
dict.get("key", default) returns None (not default) when the key
exists but its value is explicitly None. This caused an AttributeError
in _escape_html() when a task errored before grading and answer was
set to None.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* ggml : add support for CPU f16->f16 GGML_OP_SET_ROWS
* ggml : add missing type checks in f16 GGML_OP_SET_ROWS
* ggml : merge ggml_compute_forward_set_rows_f32() and ggml_compute_forward_set_rows_f16() into ggml_compute_forward_set_rows_impl()
* chore : replace assert() with GGML_ASSERT()
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* fix: draft model fit vs load inconsistency
* refactor(server): unify draft/mtp parameter initialization, model, and context load
- moves speculative init to speculative.cpp
- changes server_context_impl model_dft and ctx_dft to use raw pointers
- fix: don't throttle progress callback when loading draft model
- refactor: rename draft model/ctx load method
* fix: valign
Before this commit, --cache-ram was not a hard limit:
- The cache always kept at least one entry, even if that entry exceeded the
RAM/token limits.
- Old entries were only evicted for the RAM/token limits after saving the new
one, which could cause the cache to temporarily exceed the RAM/token limits
even if individual entries were below the limit.
Now, ensure that the RAM limit is strict with these changes:
- Skip saving state to cache if by itself it exceeds the RAM limit.
- Evict old entries as necessary to make the new entry fit.
Additionally, token-limit cleanup may now evict the last remaining cache entry
instead of always preserving one.
-ffast-math implies -ffinite-math-only under ROCm/clang 22, which
disables INFINITY/NaN and triggers -Wnan-infinity-disabled (errors
under -Werror in CI). Re-enable infinity handling without dropping
the rest of fast-math.
Fixes#25361
* support op col2im_1d
* update ops.md
* rm unused words
* update for bf16
* optimize 1%-11% as the review comments
* fix the format issue
* update as the review comments
* sycl: add supported types to ggml_sycl_supports_reorder_dmmv
The reordered feature is implemented in ggml_sycl_op_dequantize_mul_mat_vec,
but gated by ggml_sycl_supports_reorder_dmmv. This commit fixes the gate.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* sycl: set K_QUANTS_PER_ITERATION=1 to improve utilization
When combined with opening the reorder gate, this improves GPU
utilization on B70, giving a significant boost to tg t/s.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* sycl: replace QK_WARP_SIZE with WARP_SIZE for QK_5
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* sycl: add missing types to ggml_backend_sycl_buffer_init_tensor
Without this, the extra field is not allocated and the reorder path
will not take effect.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support
* chore : get rid of else
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV
* opencl: improve non FA KQ mv kernels
* opencl: tweaks for multiquery FA
* opencl: some tweaks for FA q1 kernels
* opencl: FA with DK=DV=512 for gemma-4
* opencl: various fixes
* opencl: cleanup
* opencl: fix FA decode crash for DK=512 (gemma-4)
The DK=512 decode-only program does not create the f32_f16 prefill
kernel, so the compiled check in ensure_fa_variant never hit and
supports_op gave inconsistent answers for the same op. block_n is also
unset for DK=512 decode; guard it to avoid an out-of-range read at
dispatch.
* opencl: run DK=512 FA decode on CPU
DK=512 decode is bandwidth-bound and faster on the CPU than the GPU,
increasingly so with depth. Decline it in supports_op; prefill stays on the GPU.
* opencl: compile MQ_GQA=8 FA kernels in a minimal program
The full program compiled with -D MQ_GQA=8 runs the Adreno compiler out
of memory at DK>=256. Only the vec_mq kernels are used from this
program, so compile it with FA_MQ_ONLY, which excludes everything else.
Also include the program name in the compile error log.
* opencl: remove stray token in flash_attn_f32_f16.cl
A stray "." broke the f32_f16 program build.
* opencl: split f16-KV FA decode finer (FD_KV_PER_SPLIT_F16)
The 2048 default under-fills the GPU on single-query f16-KV decode;
use 512 for f16 KV to get more splits. Quantized KV keeps 2048.
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* metal: add col2im_1d op (f32/f16/bf16)
Gather kernel mirroring the CPU/CUDA path: each output (t_out, oc)
reads its ceil(K/s0) source columns with an F32 accumulator, a single
write and no atomics. One thread per output element, 256 per
threadgroup.
* metal: check dst contiguity and type match in supports_op for COL2IM_1D
Align the GGML_OP_COL2IM_1D predicate with the CPU, CUDA, and Vulkan
backends: the kernel writes dst with linear indexing and assumes the
same type as src0, so supports_op must also require a contiguous dst
and op->type == op->src[0]->type.
* Update ggml/src/ggml-metal/ggml-metal.metal
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
---------
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
* server: fix deadlock in load_models() when erasing a finished download
The download monitoring thread acquires the models mutex on its way out,
but load_models() joined it from the erase loop while holding that mutex.
Join it outside the lock via threads_to_join like the other monitoring
threads.
* server: add default timeout to test requests
A hung server now fails the test after 10 minutes instead of stalling
the CI job for hours. Explicit timeouts are unchanged.
RapidMark