llama : fix Viking pre-tokenizer regex

The order was previously wrong, which caused errors in some tests.
2026-07-09 20:50:21 +00:00 · 2024-07-08 16:34:39 -04:00
parent f9d42c598b
commit 31a1b0eeaa
1 changed files with 1 additions and 1 deletions
@@ -15440,8 +15440,8 @@ struct llm_tokenizer_bpe {
                break;
            case LLAMA_VOCAB_PRE_TYPE_VIKING:
                regex_exprs = {
-                    "\\p{N}",
                    " ?[^(\\s|.,!?…。，、।۔،)]+",
+                    "\\p{N}",
                };
                break;
            default: