mirror of
https://github.com/datawhalechina/llms-from-scratch-cn.git
synced 2026-10-01 22:57:26 +00:00
Ethan-Chen-plus
@@ -0,0 +1,729 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "efd0b15a-2694-45e8-806d-7ccc36f71926",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"# 5.3 控制随机性的译码策略"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ccb1f734-42c2-41d1-971d-7d7721a6d935",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"本章节我们会介绍文本生成策略(也称译码策略)来生成更多原始文本.首先我们简单回顾一下在本章节前面的generate_and_print_sample中使用的前面章节的generate_text_simple函数。接着我们会介绍用于优化该函数的两项技术：温度放缩（temperature scaling）和top-k采样"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "893d5d98-b3a0-4a13-b3fb-31816f003d9b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们首先将模型从GPU传输回CPU，因为使用相对较小的模型进行推理不需要 GPU。然后我们将经过训练的模型放入评估模型中以关闭随机组件例如dropout："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d69025d0-7895-4bf1-9f0f-3952e465b4f7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"model.to(\"cpu\")\n",
|
||||
"model.eval()\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5186e848-1366-4cdd-a150-868e9a0eec6e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"接着我们将GPTModel实例（模型）插入到generate_text_simple函数中，该函数使用LLM一次生成一个token："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f6cf33e-be4c-4b48-ac03-e6652c3addcc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```python\n",
|
||||
"tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
|
||||
"token_ids = generate_text_simple(\n",
|
||||
" model=model,\n",
|
||||
" idx=text_to_token_ids(\"Every effort moves you\", tokenizer),\n",
|
||||
" max_new_tokens=25,\n",
|
||||
" context_size=GPT_CONFIG_124M[\"context_length\"]\n",
|
||||
")\n",
|
||||
"print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6efebc02-c777-465f-9c03-7c9032275e43",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"生成的文本如下所示："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9075ecc5-404d-4b8e-b2d3-0a895c7d6b62",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"Output text:\n",
|
||||
"Every effort moves you know,\" was one of the axioms he laid down\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7e36cdea-2ccd-4551-9562-8cf840d7cc76",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"正如前面在5.1.2节所述，在每个生成步骤中选出的生成的token与词汇表中所有token中的最大概率分数相对应。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ad75b490-18ec-4504-88ca-22960118fad5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"这意味着在相同开头的语境中（例如“Every effort moves you”）无论我们运行generate_text_simple函数多少次，LLM都会一直生成同样的结果。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "68cc2760-709f-494e-b2ad-e79488a29b82",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在接下来的小节中会介绍两个控制随机性和多样性的概念：温度放缩（temperature scaling）和top-k采样"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "602b1d5a-5f22-4a3b-b043-dd630594d950",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## 5.3.1 温度放缩（Temperature scaling）"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9f806f31-2541-4be7-b0ab-f618319ee162",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"该部分介绍的温度放缩是一项将概率选择过程添加到下一token生成任务的技术。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a0fd042-7c44-444b-86f8-f89fd3ade4e2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"以前在generate_text_simple函数中，我们总是用torch.argmax 抽取概率最高的令牌作为下一个令牌，这一过程也称为贪婪解码（greedy decoding）。为了生成更具多样性的文本，我们可以将argmax替换成从概率分布中采样的函数（在此特指LLM在每个token生成步骤中为每个词汇条目生成的概率分数）。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a7703c5d-e0ea-4fd9-88b0-bce943c0c5ea",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"为了用具体的例子来阐明概率抽样，让我们用一个非常小的词汇来简要讨论下一个token的生成过程以便说明问题："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d620c46e-f269-4719-a0e7-346ff5b60568",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"vocab = {\n",
|
||||
" \"closer\": 0,\n",
|
||||
" \"every\": 1,\n",
|
||||
" \"effort\": 2,\n",
|
||||
" \"forward\": 3,\n",
|
||||
" \"inches\": 4,\n",
|
||||
" \"moves\": 5,\n",
|
||||
" \"pizza\": 6,\n",
|
||||
" \"toward\": 7,\n",
|
||||
" \"you\": 8,\n",
|
||||
"}\n",
|
||||
"inverse_vocab = {v: k for k, v in vocab.items()}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "114c77aa-df9e-4b7a-ba1e-963345757899",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"下面假设LLM开头语境被设定为“every efforts moves you”，并生成如下所示的的下一token的logit："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9df3a622-299d-4ee2-925f-83f06a49b2b6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"next_token_logits = torch.tensor([4.51, 0.89, -1.90, 6.75, 1.63, -1.62, -1.89, 6.28, 1.79])\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3934cc1f-08b3-4590-bdfb-d8bb05892fb8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"正如上一章在generate_text_simple中所讨论的，我们通过softmax函数将logits转化为概率并通过argmax函数获取与生成的token相应的token ID，然后我们可以通过逆转词汇表将其映射回文本："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "833324af-a23b-4c7e-a938-26b13c3770ab",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"probas = torch.softmax(next_token_logits, dim=0)\n",
|
||||
"next_token_id = torch.argmax(probas).item()\n",
|
||||
"print(inverse_vocab[next_token_id])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "44d84187-de1f-4df9-9e39-87912567882e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"由于最大的logit值以及相应的最大softmax概率分数位于第四个位置（索引位置为3，因为Python采用0索引制），因此生成的单词是“forward”。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c3c54fb7-3b18-4024-9590-3609ba0d497e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"为了实现概率采样过程，我们现在可以将 argmax 替换为 PyTorch 中的多项式函数："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6366c1f3-d7e9-460b-b259-d40024a177e7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"torch.manual_seed(123)\n",
|
||||
"next_token_id = torch.multinomial(probas, num_samples=1).item()\n",
|
||||
"print(inverse_vocab[next_token_id])\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "effa15ec-825b-4cda-a965-bcbdf27f8e25",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"但打印输出结果仍为“forward”。为何如此？这是因为多项式函数对下一个token的采样与其概率分数成正比。换言之，“forward”仍是最具可能性的token并且大多时候都会被多项式选择，但有时也有例外。为了说明这一点，让我们实现一个重复采样1000次的函数："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0a34278f-b0fa-4e61-a0a4-fbdfc6b7cfac",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"def print_sampled_tokens(probas):\n",
|
||||
" torch.manual_seed(123)\n",
|
||||
" sample = [torch.multinomial(probas, num_samples=1).item() for i in range(1_000)]\n",
|
||||
" sampled_ids = torch.bincount(torch.tensor(sample))\n",
|
||||
" for i, freq in enumerate(sampled_ids):\n",
|
||||
" print(f\"{freq} x {inverse_vocab[i]}\")\n",
|
||||
" print_sampled_tokens(probas)\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "70824b91-ff0b-4b09-9255-299d2dc998b6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"采样结果如下所示："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "749d0c86-193f-404b-b08d-a59cd120370d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"73 x closer \n",
|
||||
"0 x every\n",
|
||||
"0 x effort\n",
|
||||
"582 x forward\n",
|
||||
"2 x inches\n",
|
||||
"0 x moves\n",
|
||||
"0 x pizza\n",
|
||||
"343 x toward\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bf7e9a4b-c392-400e-8a6a-98ad285351af",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"基于输出结果我们可以看到单词“forward”大部分时间都被采样（1000次中有582次），但其他token诸如“closer”、“inch”、“toward”有时也会被采样。这意味着，如果我们用generate_and_print_sample函数中的多项式函数代替argmax函数，LLM有时会生成诸如“every effort moves you toward”、“every effort moves you inches”和“every effort moves you closer”而不是“every effort moves you forward”。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1aa12938-8c5e-45d6-af7c-7a9c45904d60",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们可以通过温度放缩（temperature scaling）的概念来进一步控制分布和选择过程，所谓的温度放缩实际上只是对logit除以大于0的数这一过程的一个花哨的描述："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "761f24dd-9cfc-4799-948e-5da01e6c3606",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"def softmax_with_temperature(logits, temperature):\n",
|
||||
"scaled_logits = logits / temperature\n",
|
||||
"return torch.softmax(scaled_logits, dim=0)\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "91d507dd-627b-46c5-b566-9e166d99dc94",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"温度大于1会导致token分布更均匀，温度小于1会导致分布更可靠（更清晰或峰值更高）。让我们通过绘制关于原始概率以及使用不同温度值放缩的概率图来说明这一点："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4147fc9c-fa54-423c-a539-d7b08d1b4cb8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"temperatures = [1, 0.1, 5] # Original, higher, and lower temperature\n",
|
||||
"scaled_probas = [softmax_with_temperature(next_token_logits, T) for T in temperatures]\n",
|
||||
"x = torch.arange(len(vocab))\n",
|
||||
"bar_width = 0.15\n",
|
||||
"fig, ax = plt.subplots(figsize=(5, 3))\n",
|
||||
"for i, T in enumerate(temperatures):\n",
|
||||
" rects = ax.bar(x + i * bar_width, scaled_probas[i],\n",
|
||||
" bar_width, label=f'Temperature = {T}')\n",
|
||||
"ax.set_ylabel('Probability')\n",
|
||||
"ax.set_xticks(x)\n",
|
||||
"ax.set_xticklabels(vocab.keys(), rotation=90)\n",
|
||||
"ax.legend()\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eaa08f06-c363-49c1-8b54-a6844c75f654",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"绘制结果如fig-5-14所示。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65c5265c-f0f1-4453-a392-77b350d03afd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "476dc931-4e8c-4efa-8b3e-46c5afbcadd3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"当温度为1时，将对数除以1然后将其传递给softmax函数来计算概率分数。换言之，使用温度值为1的放缩与不使用任何温度放缩相同。在这种情况下选择的token的概率等同于通过PyTorch中的多项式采样函数获取的原始softmax概率分数。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b686035e-cb54-4a01-a4a6-8edd958b5370",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"例如,如figure-5-14所示，当温度设置为1时，大约有60%的时间会选择与“forward”对应的token。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "be72475f-168a-4b62-9106-2b524033e866",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"此外如figure-5-14所示，施加非常小的温度（例如0.1）将导致分布的差异性更高，从而使得多项式函数类似于argmax函数一样几乎100%表现为选择可能性最大的token（此处为“正向”）。反之亦然，当温度为5时分布会更均衡使得选择其他token的几率增加。这样可以提高生成文本的多样性但也会导致生成更多无意义的文本。举个例子：温度设置为4会导致大约有4%的时间出现诸如“every effort moves you pizza”这样的文本。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "864f8b82-ade6-49e6-a68d-8c4181234045",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**练习 5.1**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "769b1bd4-41d3-4e98-8c1a-0c9356e9dc45",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"使用 print_sampled_tokens 函数绘制 softmax 概率的采样频率，该频率与figure-5-13 所示的温度成比例。在每种情况下，“pizza”这个词多久抽样一次？你能想出一种更快、更准确的方法来确定“pizza”这个词的采样频率吗？"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d90e77f6-59a1-4488-9bf8-3ca0f73d008b",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## 5.3.2 Top-k采样"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "56851ce2-f815-4e02-8bc1-d67d410f5497",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在前面一节中，我们实现了概率采样方法与温度放缩（temperature scaling）相结合来增加输出结果的多样性。可以看到温度值越高下一个token的概率分布越均衡，由于降低了模型重复选择最可能tokrn的可能性导致产生更具多样性的输出。这种方法允许在生成过程中探索可能性低但也许更有趣和更具创造性的路径。然而该方法也存在一个弊端：该方法有时会导致语法错误或者完全无意义的输出，例如“every effort moves you pizza”。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c18d8c21-d61c-4e37-9b9c-6adb21a9ca34",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"本节我们将会介绍另一个概念称作top-k采样，当它与概率采样和温度放缩结合时，可以优化文本生成结果。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5b6f7e3c-2ffb-4187-9802-1be5a8598d0c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"在top-k采样中我们将抽取的token限制为最有可能的top-k的token，并通过屏蔽概率分数的方式排除所有其他的token，如fig-5-15所示。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7ce97ef6-f347-4a14-94eb-07b26d7b05c5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "77c9f1a1-1cbc-4ad1-83ec-d762d998c649",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"fig-5-15中概述的方法将所有为被选择的logits替换为负无穷大（-inf），这样在计算softmax值时非top-k的token的概率分数为0，其余概率总和为1.（细心的读者或许还记得在第3章第3.5.1节“应用因果注意掩码”中实现的来自因果注意模块的掩码技巧）"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d1f5734a-6594-4724-a163-fbc10240d07f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"我们可以用代码实现fig-5-15中概述的 top-k 过程，如下所示，从选择具有最大 logit值的token开始："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e3e08166-ff1b-4114-892d-11193f6ef208",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"top_k = 3\n",
|
||||
"top_logits, top_pos = torch.topk(next_token_logits, top_k)\n",
|
||||
"print(\"Top logits:\", top_logits)\n",
|
||||
"print(\"Top positions:\", top_pos)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c6627a41-eb46-4ab7-8207-38644816ff51",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"前3个token的logit值和token ID（按降序排列）如下所示："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fb6bfb02-080c-4bfa-83da-01707b6c00ff",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"Top logits: tensor([6.7500, 6.2800, 4.5100])\n",
|
||||
"Top positions: tensor([3, 7, 0])\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c1a4f807-870f-4286-ae67-b416d82c3436",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"随后我们运用PyTorch的where函数将低于前3个选项里最低logit值的token的 logit 值设置为负无穷大（-inf）。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a5c26586-7fbd-476d-86fb-45427ab7c3ca",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"new_logits = torch.where(\n",
|
||||
" condition=next_token_logits < top_logits[-1], #A\n",
|
||||
" input=torch.tensor(float('-inf')), #B\n",
|
||||
" other=next_token_logits #C\n",
|
||||
")\n",
|
||||
"print(new_logits)\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "41fab7e4-400c-4826-aa3c-4bcd62b3a33e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"9个token词汇表中下一个token的结果日志如下所示："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f209131b-42a6-464e-ac25-531faa237462",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"tensor([4.5100, -inf, -inf, 6.7500, -inf, -inf, -inf, 6.2800, -inf])\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a60b95fd-3399-49ea-a208-cd80ab920c22",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"最后使用softmax函数将这些转换为下一个token概率："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "915ac3a3-4611-4449-9de8-b4f30988daec",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"topk_probas = torch.softmax(new_logits, dim=0)\n",
|
||||
"print(topk_probas)\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fa2fa765-bb1c-4c96-b894-89d12fb432a5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"可以看到这种top-3方法的结果是3个非零概率分数："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f7e66cc2-8a97-4cd8-868b-36f9d28c38ed",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"tensor([0.0615, 0.0000, 0.0000, 0.5775, 0.0000, 0.0000, 0.0000, 0.3610, 0.0000])\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "160d7e3c-3597-4845-8d59-ae1a69a042e4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"现在我们可以采用上一节介绍的概率抽样的温度放缩和多项式函数，在这3个非零概率分数中选择下一个token来生成下一个token。下一节中我们将通过修改文本生成函数来实现。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "628597d5-7b22-4fa6-9bf3-ea6c5deb1c46",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5.3.3 修改文本生成函数"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f85aea9a-7204-475e-b49f-73362712cbcd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"前面两节介绍了用于增加LLM生成文本多样性的两个概念：温度采样和top-k采样。本节我们会将这些概念结合来修改我们之前通过LLM来生成文本的generate_simple函数，创造一个新的生成函数："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8a684576-5c2b-43c7-9a47-9df46afe14dc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
" **Listing 5.4 具有更多多样性的修改文本生成函数**\n",
|
||||
"```\n",
|
||||
"def generate(model, idx, max_new_tokens, context_size, temperature, top_k=None):\n",
|
||||
" for _ in range(max_new_tokens): #A\n",
|
||||
" idx_cond = idx[:, -context_size:]\n",
|
||||
" with torch.no_grad():\n",
|
||||
" logits = model(idx_cond)\n",
|
||||
" logits = logits[:, -1, :]\n",
|
||||
" if top_k is not None: #B\n",
|
||||
" top_logits, _ = torch.topk(logits, top_k)\n",
|
||||
" min_val = top_logits[:, -1]\n",
|
||||
" logits = torch.where(\n",
|
||||
" logits < min_val,\n",
|
||||
" torch.tensor(float('-inf')).to(logits.device),\n",
|
||||
" logits\n",
|
||||
" )\n",
|
||||
" if temperature > 0.0: #C\n",
|
||||
" logits = logits / temperature\n",
|
||||
" probs = torch.softmax(logits, dim=-1)\n",
|
||||
" idx_next = torch.multinomial(probs, num_samples=1)\n",
|
||||
" else: #D\n",
|
||||
" idx_next = torch.argmax(logits, dim=-1, keepdim=True)\n",
|
||||
" idx = torch.cat((idx, idx_next), dim=1)\n",
|
||||
" return idx\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8d430038-9b1f-4f1c-88c8-9fcad6bd99d9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"来看看这个新的生成函数的实际效果："
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e835bb96-fe55-40c5-bf31-4733dee794e0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"torch.manual_seed(123)\n",
|
||||
"token_ids = generate(\n",
|
||||
" model=model,\n",
|
||||
" idx=text_to_token_ids(\"Every effort moves you\", tokenizer),\n",
|
||||
" max_new_tokens=15,\n",
|
||||
" context_size=GPT_CONFIG_124M[\"context_length\"],\n",
|
||||
" top_k=25,\n",
|
||||
" temperature=1.4\n",
|
||||
")\n",
|
||||
"print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9b7aae98-555c-4c5b-802a-4fa918df4df6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"生成的文本如下所示：\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f1f36b8a-6c57-4268-aae6-b4c4c0eb0248",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```\n",
|
||||
"Output text:\n",
|
||||
"Every effort moves you stand to work on surprise, a one of us had gone with random\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0fbf42a8-767c-4872-ba03-f707e8a81f6a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"可以看到生成的文本与前面在第5.3节开头使用generate_simple 函数生成的文本（以训练集中的一项纪录为例，“Every effort moves you know,\" was one of the axioms he laid...!”）截然不同。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f2c444a-8f6a-425f-8e92-fdad188fdb10",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**练习5.2**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "037d620e-208c-4247-8c46-10a5c0a42a45",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"尝试不同的温度和top-k设置。根据您的观察，您能想到需要较低温度和top-k设置的运用吗？反之亦然，您能想到首选需要更高温度和top-k设置的场景吗？（建议在从OpenAI加载预训练权重后，重新回顾本章节末的此练习。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c8fbb181-9c71-4250-ae81-32015112e32d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**练习5.3**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "78f62496-9fe5-4e21-95ac-e56006016eb8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"生成函数的设置有哪些不同的组合来实现确定性行为（即禁用随机采样使其始终产生接近generate_simple函数的相同输出）？"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d614f26a-e06f-46ed-bea9-8a2ebd0e92de",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"到目前为止，我们介绍了如何预训练LLM并使用它们来生成文本。本章的最后两节将讨论我们如何保存和加载经过训练的LLM，以及如何从OpenAI加载预训练的权重。"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.7"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user