kanshan/llms-from-scratch-cn
1
0
Fork 0
mirror of https://github.com/datawhalechina/llms-from-scratch-cn.git synced 2026-04-25 08:58:17 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #27 from 0-yy-0/gly

Browse Source 
add：翻译 5.1
This commit is contained in:
Ethan-Chen-plus 2024-05-12 17:48:57 +08:00 committed by GitHub
commit 4d81949de8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 1952 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Translated_Book/ch02/2.6使用滑动窗口进行数据采样.ipynb
View File

@ -75,7 +75,7 @@
"import tiktoken\n",
"\n",
"tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
"with open(\"/Users/zhihu123/Project/other/llms-from-scratch-cn/ch02/01_main-chapter-code/the-verdict.txt\", \"r\", encoding=\"utf-8\") as f:\n",
"with open(\"the-verdict.txt\", \"r\", encoding=\"utf-8\") as f:\n",
" raw_text = f.read()\n",
"enc_text = tokenizer.encode(raw_text)\n",
"print(len(enc_text))"
@ -441,7 +441,7 @@
}
],
"source": [
"with open(\"/Users/zhihu123/Project/other/llms-from-scratch-cn/ch02/01_main-chapter-code/the-verdict.txt\", \"r\", encoding=\"utf-8\") as f:\n",
"with open(\"the-verdict.txt\", \"r\", encoding=\"utf-8\") as f:\n",
" raw_text = f.read()\n",
" dataloader = create_dataloader_v1(\n",
" raw_text, batch_size=1, max_length=4, stride=1, shuffle=False)\n",

1950
Translated_Book/ch05/5.1 在未标记的数据上进行预训练.ipynb Normal file
View File

File diff suppressed because it is too large Load Diff