mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-07-10 13:10:22 +00:00
Hongqiang Wang
92366df30d
* opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number. Route Q6_K dense mul_mat with ne01 % 128 != 0 off the noshuffle path: decode (ne1==1) uses the correct flat GEMV and the matching GEMM (ne1>1) falls back to CPU (the flat convert has no verified small-batch GEMM kernel for these shapes). All standard hidden/FFN/vocab dims are multiples of 128 and keep the noshuffle path. * opencl: reserve alignment slack for the SOA subbuffer carve in alloc size set_tensor carves quantized weights into per-component subbuffers (d/q, ql/qh/s/d, ...) whose origins are each rounded up to the device base address alignment. When a component's size is not a multiple of the alignment, the carve extends past ggml_nbytes(tensor) and the last subbuffer overlaps the next tensor in the pool -- e.g. q6_K [1536, 49155]: size_s = 49155*96 ends 32 bytes past a 128-byte boundary, so the d subbuffer ends 96 bytes past the tensor's allocation, and whichever of the two neighboring tensors is uploaded last silently corrupts the other (here: the last vocab rows' block scales). This affects any quant type whose component sizes can be misaligned, on any shape with ne01 not a multiple of the alignment granularity; standard power-of-two dims are unaffected. Implement get_alloc_size for the OpenCL buffer type and reserve the worst-case carve slack (4 aligned gaps; 5 components max, q5_K) for quantized tensors. Costs at most 512 bytes per quantized tensor at the observed 128-byte alignment. * opencl: use lm based q6_k mm when ne1 is not multiple of 128 --------- Co-authored-by: Li He <lih@qti.qualcomm.com>