From 0b4a8ad4aa66fa4c076569d117080231ad2776da Mon Sep 17 00:00:00 2001 From: jingyaogong Date: Tue, 6 Jan 2026 01:18:10 +0800 Subject: [PATCH] [update] readme --- README.md | 2 +- README_en.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0c79d49..d9415a6 100644 --- a/README.md +++ b/README.md @@ -427,7 +427,7 @@ python train_xxx.py --use_wandb ## Ⅰ Tokenizer 分词器将单词从自然语言通过“词典”映射到`0, 1, 36`这样的数字，可以理解为数字就代表了单词在“词典”中的页码。 -可以选择自己构造词表训练一个“词典”，代码可见`./scripts/train_tokenizer.py`（仅供学习参考，若非必要无需再自行训练，MiniMind已自带tokenizer）。 +可以选择自己构造词表训练一个“词典”，代码可见`./trainer/train_tokenizer.py`（仅供学习参考，若非必要无需再自行训练，MiniMind已自带tokenizer）。 或者选择比较出名的开源大模型分词器， 正如同直接用新华/牛津词典的优点是token编码压缩率很好，缺点是页数太多，动辄数十万个词汇短语； 自己训练的分词器，优点是词表长度和内容随意控制，缺点是压缩率很低（例如"hello"也许会被拆分为"h e l l o" diff --git a/README_en.md b/README_en.md index 961d4f2..296cfda 100644 --- a/README_en.md +++ b/README_en.md @@ -426,7 +426,7 @@ By adding the `--use_wandb` parameter, you can record the training process. Afte ## Ⅰ Tokenizer Tokenizer maps words from natural language to numbers like `0, 1, 36` through a "dictionary," which can be understood as numbers representing the page number of the word in the "dictionary." -You can choose to construct your own vocabulary table to train a "dictionary." The code can be found in `./scripts/train_tokenizer.py` (for learning reference only. It's not necessary to train one yourself unless required. MiniMind comes with a built-in tokenizer). +You can choose to construct your own vocabulary table to train a "dictionary." The code can be found in `./trainer/train_tokenizer.py` (for learning reference only. It's not necessary to train one yourself unless required. MiniMind comes with a built-in tokenizer). Or you can choose tokenizers from well-known open-source large models. Just as using Xinhua/Oxford dictionaries directly has the advantage of good token encoding compression, but the disadvantage of having too many pages—tens of thousands of word phrases; A self-trained tokenizer has the advantage of freely controlling vocabulary length and content, but the disadvantage of low compression ratio (for example, "hello" might be split into "h e l l o"