Merge pull request #39 from Ethan-Chen-plus/main

Browse Source

Add RWKV2~V6 and GLM4

...

This commit is contained in:

Ethan-Chen-plus 2024-06-10 22:50:47 +08:00 committed by GitHub

parent 099c1c620f e2360bc0a2

commit 980692833b

No known key found for this signature in database

GPG Key ID: B5690EEEBB952194

158 changed files with 9053 additions and 55386 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Show Stats Download Patch File Download Diff File Expand all files Collapse all files

0

appendix-A/01_optional-python-setup-preferences/README.md → Codes/appendix-A/01_optional-python-setup-preferences/README.md

Unescape Escape View File

0

appendix-A/01_optional-python-setup-preferences/figures/activate-env.png → Codes/appendix-A/01_optional-python-setup-preferences/figures/activate-env.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 180 KiB After Width:  |  Height:  |  Size: 180 KiB

0

appendix-A/01_optional-python-setup-preferences/figures/check-pip.png → Codes/appendix-A/01_optional-python-setup-preferences/figures/check-pip.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 220 KiB After Width:  |  Height:  |  Size: 220 KiB

0

appendix-A/01_optional-python-setup-preferences/figures/conda-install.png → Codes/appendix-A/01_optional-python-setup-preferences/figures/conda-install.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 186 KiB After Width:  |  Height:  |  Size: 186 KiB

0

appendix-A/01_optional-python-setup-preferences/figures/download.png → Codes/appendix-A/01_optional-python-setup-preferences/figures/download.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 174 KiB After Width:  |  Height:  |  Size: 174 KiB

0

appendix-A/01_optional-python-setup-preferences/figures/miniforge-install.png → Codes/appendix-A/01_optional-python-setup-preferences/figures/miniforge-install.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 258 KiB After Width:  |  Height:  |  Size: 258 KiB

0

appendix-A/01_optional-python-setup-preferences/figures/new-env.png → Codes/appendix-A/01_optional-python-setup-preferences/figures/new-env.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 185 KiB After Width:  |  Height:  |  Size: 185 KiB

0

appendix-A/01_optional-python-setup-preferences/figures/pytorch-installer.jpg → Codes/appendix-A/01_optional-python-setup-preferences/figures/pytorch-installer.jpg

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 94 KiB After Width:  |  Height:  |  Size: 94 KiB

0

appendix-A/02_installing-python-libraries/README.md → Codes/appendix-A/02_installing-python-libraries/README.md

Unescape Escape View File

0

appendix-A/02_installing-python-libraries/figures/check_1.jpg → Codes/appendix-A/02_installing-python-libraries/figures/check_1.jpg

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 107 KiB After Width:  |  Height:  |  Size: 107 KiB

0

appendix-A/02_installing-python-libraries/figures/check_2.jpg → Codes/appendix-A/02_installing-python-libraries/figures/check_2.jpg

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 79 KiB After Width:  |  Height:  |  Size: 79 KiB

0

appendix-A/02_installing-python-libraries/figures/jupyter-issues.jpg → Codes/appendix-A/02_installing-python-libraries/figures/jupyter-issues.jpg

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 103 KiB After Width:  |  Height:  |  Size: 103 KiB

0

appendix-A/02_installing-python-libraries/figures/pytorch-installer.jpg → Codes/appendix-A/02_installing-python-libraries/figures/pytorch-installer.jpg

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 94 KiB After Width:  |  Height:  |  Size: 94 KiB

0

appendix-A/02_installing-python-libraries/figures/watermark.jpg → Codes/appendix-A/02_installing-python-libraries/figures/watermark.jpg

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 36 KiB After Width:  |  Height:  |  Size: 36 KiB

0

appendix-A/02_installing-python-libraries/python_environment_check.ipynb → Codes/appendix-A/02_installing-python-libraries/python_environment_check.ipynb

Unescape Escape View File

0

appendix-A/02_installing-python-libraries/python_environment_check.py → Codes/appendix-A/02_installing-python-libraries/python_environment_check.py

Unescape Escape View File

0

appendix-A/02_installing-python-libraries/requirements.txt → Codes/appendix-A/02_installing-python-libraries/requirements.txt

Unescape Escape View File

0

appendix-A/03_main-chapter-code/DDP-script.py → Codes/appendix-A/03_main-chapter-code/DDP-script.py

Unescape Escape View File

0

appendix-A/03_main-chapter-code/code-part1.ipynb → Codes/appendix-A/03_main-chapter-code/code-part1.ipynb

Unescape Escape View File

0

appendix-A/03_main-chapter-code/code-part2.ipynb → Codes/appendix-A/03_main-chapter-code/code-part2.ipynb

Unescape Escape View File

0

appendix-A/03_main-chapter-code/exercise-solutions.ipynb → Codes/appendix-A/03_main-chapter-code/exercise-solutions.ipynb

Unescape Escape View File

0

appendix-B/README.md → Codes/appendix-B/README.md

Unescape Escape View File

0

ch01/README.md → Codes/ch01/README.md

Unescape Escape View File

0

ch02/01_main-chapter-code/README.md → Codes/ch02/01_main-chapter-code/README.md

Unescape Escape View File

0

ch02/01_main-chapter-code/ch02.ipynb → Codes/ch02/01_main-chapter-code/ch02.ipynb

Unescape Escape View File

0

ch02/01_main-chapter-code/dataloader.ipynb → Codes/ch02/01_main-chapter-code/dataloader.ipynb

Unescape Escape View File

0

ch02/01_main-chapter-code/exercise-solutions.ipynb → Codes/ch02/01_main-chapter-code/exercise-solutions.ipynb

Unescape Escape View File

0

ch02/01_main-chapter-code/the-verdict.txt → Codes/ch02/01_main-chapter-code/the-verdict.txt

Unescape Escape View File

0

ch02/02_bonus_bytepair-encoder/README.md → Codes/ch02/02_bonus_bytepair-encoder/README.md

Unescape Escape View File

0

ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py → Codes/ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py

Unescape Escape View File

0

ch02/02_bonus_bytepair-encoder/compare-bpe-tiktoken.ipynb → Codes/ch02/02_bonus_bytepair-encoder/compare-bpe-tiktoken.ipynb

Unescape Escape View File

0

ch02/02_bonus_bytepair-encoder/gpt2_model/encoder.json → Codes/ch02/02_bonus_bytepair-encoder/gpt2_model/encoder.json

Unescape Escape View File

0

ch02/03_bonus_embedding-vs-matmul/README.md → Codes/ch02/03_bonus_embedding-vs-matmul/README.md

Unescape Escape View File

0

ch02/03_bonus_embedding-vs-matmul/embeddings-and-linear-layers.ipynb → Codes/ch02/03_bonus_embedding-vs-matmul/embeddings-and-linear-layers.ipynb

Unescape Escape View File

0

ch02/03_bonus_embedding-vs-matmul/images/1.png → Codes/ch02/03_bonus_embedding-vs-matmul/images/1.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 133 KiB After Width:  |  Height:  |  Size: 133 KiB

0

ch02/03_bonus_embedding-vs-matmul/images/2.png → Codes/ch02/03_bonus_embedding-vs-matmul/images/2.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 133 KiB After Width:  |  Height:  |  Size: 133 KiB

0

ch02/03_bonus_embedding-vs-matmul/images/3.png → Codes/ch02/03_bonus_embedding-vs-matmul/images/3.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 216 KiB After Width:  |  Height:  |  Size: 216 KiB

0

ch02/03_bonus_embedding-vs-matmul/images/4.png → Codes/ch02/03_bonus_embedding-vs-matmul/images/4.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 291 KiB After Width:  |  Height:  |  Size: 291 KiB

0

ch02/03_bonus_embedding-vs-matmul/images/5.png → Codes/ch02/03_bonus_embedding-vs-matmul/images/5.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 289 KiB After Width:  |  Height:  |  Size: 289 KiB

0

ch02/09_summary/09_summary.ipynb → Codes/ch02/09_summary/09_summary.ipynb

Unescape Escape View File

0

ch02/README.md → Codes/ch02/README.md

Unescape Escape View File

0

ch03/01_main-chapter-code/README.md → Codes/ch03/01_main-chapter-code/README.md

Unescape Escape View File

0

ch03/01_main-chapter-code/ch03.ipynb → Codes/ch03/01_main-chapter-code/ch03.ipynb

Unescape Escape View File

0

ch03/01_main-chapter-code/exercise-solutions.ipynb → Codes/ch03/01_main-chapter-code/exercise-solutions.ipynb

Unescape Escape View File

0

ch03/01_main-chapter-code/figures/attention-matrix.png → Codes/ch03/01_main-chapter-code/figures/attention-matrix.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 136 KiB After Width:  |  Height:  |  Size: 136 KiB

0

ch03/01_main-chapter-code/figures/attention.png → Codes/ch03/01_main-chapter-code/figures/attention.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 67 KiB After Width:  |  Height:  |  Size: 67 KiB

0

ch03/01_main-chapter-code/figures/dot-product.png → Codes/ch03/01_main-chapter-code/figures/dot-product.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 93 KiB After Width:  |  Height:  |  Size: 93 KiB

0

ch03/01_main-chapter-code/figures/dropout.png → Codes/ch03/01_main-chapter-code/figures/dropout.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 63 KiB After Width:  |  Height:  |  Size: 63 KiB

0

ch03/01_main-chapter-code/figures/masked.png → Codes/ch03/01_main-chapter-code/figures/masked.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 59 KiB After Width:  |  Height:  |  Size: 59 KiB

0

ch03/01_main-chapter-code/figures/multi-head.png → Codes/ch03/01_main-chapter-code/figures/multi-head.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 60 KiB After Width:  |  Height:  |  Size: 60 KiB

0

ch03/01_main-chapter-code/figures/single-head.png → Codes/ch03/01_main-chapter-code/figures/single-head.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 71 KiB After Width:  |  Height:  |  Size: 71 KiB

0

ch03/01_main-chapter-code/figures/weight-selfattn-1.png → Codes/ch03/01_main-chapter-code/figures/weight-selfattn-1.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 52 KiB After Width:  |  Height:  |  Size: 52 KiB

0

ch03/01_main-chapter-code/figures/weight-selfattn-2.png → Codes/ch03/01_main-chapter-code/figures/weight-selfattn-2.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 61 KiB After Width:  |  Height:  |  Size: 61 KiB

0

ch03/01_main-chapter-code/figures/weight-selfattn-3.png → Codes/ch03/01_main-chapter-code/figures/weight-selfattn-3.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 53 KiB After Width:  |  Height:  |  Size: 53 KiB

0

ch03/01_main-chapter-code/figures/weight-selfattn-4.png → Codes/ch03/01_main-chapter-code/figures/weight-selfattn-4.png

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 54 KiB After Width:  |  Height:  |  Size: 54 KiB

0

ch03/01_main-chapter-code/multihead-attention.ipynb → Codes/ch03/01_main-chapter-code/multihead-attention.ipynb

Unescape Escape View File

0

ch03/01_main-chapter-code/small-text-sample.txt → Codes/ch03/01_main-chapter-code/small-text-sample.txt

Unescape Escape View File

0

ch03/README.md → Codes/ch03/README.md

Unescape Escape View File

0

ch04/01_main-chapter-code/README.md → Codes/ch04/01_main-chapter-code/README.md

Unescape Escape View File

0

ch04/01_main-chapter-code/ch04.ipynb → Codes/ch04/01_main-chapter-code/ch04.ipynb

Unescape Escape View File

0

ch04/01_main-chapter-code/exercise-solutions.ipynb → Codes/ch04/01_main-chapter-code/exercise-solutions.ipynb

Unescape Escape View File

0

ch04/01_main-chapter-code/figures/chapter-steps.webp → Codes/ch04/01_main-chapter-code/figures/chapter-steps.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 29 KiB After Width:  |  Height:  |  Size: 29 KiB

0

ch04/01_main-chapter-code/figures/ffn.webp → Codes/ch04/01_main-chapter-code/figures/ffn.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 24 KiB After Width:  |  Height:  |  Size: 24 KiB

0

ch04/01_main-chapter-code/figures/generate-text.webp → Codes/ch04/01_main-chapter-code/figures/generate-text.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 36 KiB After Width:  |  Height:  |  Size: 36 KiB

0

ch04/01_main-chapter-code/figures/gpt-in-out.webp → Codes/ch04/01_main-chapter-code/figures/gpt-in-out.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 21 KiB After Width:  |  Height:  |  Size: 21 KiB

0

ch04/01_main-chapter-code/figures/gpt.webp → Codes/ch04/01_main-chapter-code/figures/gpt.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 30 KiB After Width:  |  Height:  |  Size: 30 KiB

0

ch04/01_main-chapter-code/figures/iterative-gen.webp → Codes/ch04/01_main-chapter-code/figures/iterative-gen.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 18 KiB After Width:  |  Height:  |  Size: 18 KiB

0

ch04/01_main-chapter-code/figures/iterative-generate.webp → Codes/ch04/01_main-chapter-code/figures/iterative-generate.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 24 KiB After Width:  |  Height:  |  Size: 24 KiB

0

ch04/01_main-chapter-code/figures/layernorm.webp → Codes/ch04/01_main-chapter-code/figures/layernorm.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 27 KiB After Width:  |  Height:  |  Size: 27 KiB

0

ch04/01_main-chapter-code/figures/layernorm2.webp → Codes/ch04/01_main-chapter-code/figures/layernorm2.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 14 KiB After Width:  |  Height:  |  Size: 14 KiB

0

ch04/01_main-chapter-code/figures/mental-model-2.webp → Codes/ch04/01_main-chapter-code/figures/mental-model-2.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 15 KiB After Width:  |  Height:  |  Size: 15 KiB

0

ch04/01_main-chapter-code/figures/mental-model-3.webp → Codes/ch04/01_main-chapter-code/figures/mental-model-3.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 21 KiB After Width:  |  Height:  |  Size: 21 KiB

0

ch04/01_main-chapter-code/figures/mental-model-final.webp → Codes/ch04/01_main-chapter-code/figures/mental-model-final.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 21 KiB After Width:  |  Height:  |  Size: 21 KiB

0

ch04/01_main-chapter-code/figures/mental-model.webp → Codes/ch04/01_main-chapter-code/figures/mental-model.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 25 KiB After Width:  |  Height:  |  Size: 25 KiB

0

ch04/01_main-chapter-code/figures/overview-after-ln.webp → Codes/ch04/01_main-chapter-code/figures/overview-after-ln.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 21 KiB After Width:  |  Height:  |  Size: 21 KiB

0

ch04/01_main-chapter-code/figures/shortcut-example.webp → Codes/ch04/01_main-chapter-code/figures/shortcut-example.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 32 KiB After Width:  |  Height:  |  Size: 32 KiB

0

ch04/01_main-chapter-code/figures/transformer-block.webp → Codes/ch04/01_main-chapter-code/figures/transformer-block.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 26 KiB After Width:  |  Height:  |  Size: 26 KiB

0

ch04/01_main-chapter-code/figures/use-gpt.webp → Codes/ch04/01_main-chapter-code/figures/use-gpt.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 15 KiB After Width:  |  Height:  |  Size: 15 KiB

0

ch04/01_main-chapter-code/gpt.py → Codes/ch04/01_main-chapter-code/gpt.py

Unescape Escape View File

0

ch04/01_main-chapter-code/previous_chapters.py → Codes/ch04/01_main-chapter-code/previous_chapters.py

Unescape Escape View File

0

ch04/README.md → Codes/ch04/README.md

Unescape Escape View File

0

ch05/01_main-chapter-code/README.md → Codes/ch05/01_main-chapter-code/README.md

Unescape Escape View File

0

ch05/01_main-chapter-code/ch05.ipynb → Codes/ch05/01_main-chapter-code/ch05.ipynb

Unescape Escape View File

0

ch05/01_main-chapter-code/gpt_download.py → Codes/ch05/01_main-chapter-code/gpt_download.py

Unescape Escape View File

0

ch05/01_main-chapter-code/gpt_generate.py → Codes/ch05/01_main-chapter-code/gpt_generate.py

Unescape Escape View File

0

ch05/01_main-chapter-code/gpt_train.py → Codes/ch05/01_main-chapter-code/gpt_train.py

Unescape Escape View File

0

ch05/01_main-chapter-code/images/img-1.webp → Codes/ch05/01_main-chapter-code/images/img-1.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 87 KiB After Width:  |  Height:  |  Size: 87 KiB

0

ch05/01_main-chapter-code/images/img-2.webp → Codes/ch05/01_main-chapter-code/images/img-2.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 72 KiB After Width:  |  Height:  |  Size: 72 KiB

0

ch05/01_main-chapter-code/images/img-3.webp → Codes/ch05/01_main-chapter-code/images/img-3.webp

Unescape Escape View File

Side by Side Swipe Overlay Before Width:  |  Height:  |  Size: 59 KiB After Width:  |  Height:  |  Size: 59 KiB

0

ch05/01_main-chapter-code/previous_chapters.py → Codes/ch05/01_main-chapter-code/previous_chapters.py

Unescape Escape View File

0

ch05/01_main-chapter-code/tests.py → Codes/ch05/01_main-chapter-code/tests.py

Unescape Escape View File

0

ch05/02_alternative_weight_loading/README.md → Codes/ch05/02_alternative_weight_loading/README.md

Unescape Escape View File

0

ch05/02_alternative_weight_loading/previous_chapters.py → Codes/ch05/02_alternative_weight_loading/previous_chapters.py

Unescape Escape View File

0

ch05/02_alternative_weight_loading/weight-loading-hf-transformers.ipynb → Codes/ch05/02_alternative_weight_loading/weight-loading-hf-transformers.ipynb

Unescape Escape View File

0

ch05/03_bonus_pretraining_on_gutenberg/README.md → Codes/ch05/03_bonus_pretraining_on_gutenberg/README.md

Unescape Escape View File

0

ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py → Codes/ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py

Unescape Escape View File

0

ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py → Codes/ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py

Unescape Escape View File

0

ch05/03_bonus_pretraining_on_gutenberg/previous_chapters.py → Codes/ch05/03_bonus_pretraining_on_gutenberg/previous_chapters.py

Unescape Escape View File

0

ch05/04_learning_rate_schedulers/README.md → Codes/ch05/04_learning_rate_schedulers/README.md

Unescape Escape View File

0

ch05/05_bonus_hparam_tuning/README.md → Codes/ch05/05_bonus_hparam_tuning/README.md

Unescape Escape View File

Some files were not shown because too many files have changed in this diff Show More