mirror of
https://github.com/datawhalechina/llms-from-scratch-cn.git
synced 2026-05-01 11:58:17 +08:00
352 lines
12 KiB
Plaintext
352 lines
12 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cb87f0a8",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 4.4 增加快捷链接"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b83c4c4e",
|
||
"metadata": {},
|
||
"source": [
|
||
"接下来，让我们讨论快捷连接背后的概念，它也可称为跳过或剩余连接。\n",
|
||
"最初，快捷连接是为计算机视觉中的深度网络（特别是残差网络）而提出的，以缓解梯度消失带来的挑战。\n",
|
||
"梯度消失问题是指梯度（在训练过程中引导权重更新）随着层反向传播而逐渐变小的问题，使得训练早期层变得困难，\n",
|
||
"如图 4.12 所示。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b489704c",
|
||
"metadata": {},
|
||
"source": [
|
||
"**图 4.12 是由 5 层组成的不带快捷连接（左侧）和带快捷连接（右侧）的深度神经网络之间的比较。\n",
|
||
"快捷连接涉及将层的输入添加到其输出，从而有效地创建绕过某些层的备用路径。\n",
|
||
"图 1.1 中所示的梯度表示每层的平均绝对梯度，我们将在下面的代码示例中计算。**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b5c76b5f",
|
||
"metadata": {},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a1388332",
|
||
"metadata": {},
|
||
"source": [
|
||
"如图 4.12 所示，快捷连接通过跳过一个或多个层，为梯度在网络中流动创造了一条更短的备用路径。这是通过将一个层的输出加到后面层的输出上来实现的。\n",
|
||
"这就是为什么这些连接也称为跳过连接。\n",
|
||
"它们在训练过程中的反向传播中起着至关重要的作用，以保持梯度的流动。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "61de7fbe",
|
||
"metadata": {},
|
||
"source": [
|
||
"在下面的代码示例中，我们实现了图 4.12 所示的神经网络，看看如何在前向方法中添加快捷连接："
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b06c73a3",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 代码示例 4.5 说明快捷方式连接的神经网络"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "fb7a3598",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import torch\n",
|
||
"import torch.nn as nn\n",
|
||
"from torch.nn import GELU\n",
|
||
"\n",
|
||
"\n",
|
||
"class ExampleDeepNeuralNetwork(nn.Module):\n",
|
||
" def __init__(self, layer_sizes, use_shortcut):\n",
|
||
" super().__init__()\n",
|
||
" self.use_shortcut = use_shortcut\n",
|
||
" self.layers = nn.ModuleList([\n",
|
||
" # 实现5层神经网络\n",
|
||
" nn.Sequential(nn.Linear(layer_sizes[0], layer_sizes[1]), GELU()),\n",
|
||
" nn.Sequential(nn.Linear(layer_sizes[1], layer_sizes[2]), GELU()),\n",
|
||
" nn.Sequential(nn.Linear(layer_sizes[2], layer_sizes[3]), GELU()),\n",
|
||
" nn.Sequential(nn.Linear(layer_sizes[3], layer_sizes[4]), GELU()),\n",
|
||
" nn.Sequential(nn.Linear(layer_sizes[4], layer_sizes[5]), GELU())\n",
|
||
" ])\n",
|
||
" def forward(self, x):\n",
|
||
" for layer in self.layers:\n",
|
||
" # 计算当前层的输出\n",
|
||
" layer_output = layer(x)\n",
|
||
" # 检查是否可以应用快捷连接\n",
|
||
" if self.use_shortcut and x.shape == layer_output.shape:\n",
|
||
" x = x + layer_output\n",
|
||
" else:\n",
|
||
" x = layer_output\n",
|
||
" return x"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "02f506ca",
|
||
"metadata": {},
|
||
"source": [
|
||
"这段代码实现了一个 5 层的深度神经网络，每层由一个线性层和一个 GELU 激活函数组成。\n",
|
||
"在前向传递中，我们迭代地通过各层传递输入，如果 self.use_shortcut 属性设置为 True，则可以选择添加图 4.12 中所示的快捷方式连接。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "86da8d36",
|
||
"metadata": {},
|
||
"source": [
|
||
"让我们使用此代码来初始化一个没有快捷连接的神经网络。\n",
|
||
"在这里，每个层都将被初始化，以便它接受具有 3 个输入值的示例，并返回 3 个输出值。\n",
|
||
"最后一层返回单个输出值："
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "8ac40afa",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"layer_sizes = [3, 3, 3, 3, 3, 1]\n",
|
||
"sample_input = torch.tensor([[1., 0., -1.]])\n",
|
||
"torch.manual_seed(123) # 为初始权重指定随机种子以确保可重现性\n",
|
||
"model_without_shortcut = ExampleDeepNeuralNetwork(\n",
|
||
" layer_sizes, use_shortcut=False\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c5ec95be",
|
||
"metadata": {},
|
||
"source": [
|
||
"接下来，我们将通过下面代码实现一个函数，在模型的反向传递中计算梯度："
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "64db0e73",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def print_gradients(model, x):\n",
|
||
" # Forward pass\n",
|
||
" output = model(x)\n",
|
||
" target = torch.tensor([[0.]])\n",
|
||
" \n",
|
||
" # 根据目标和输出的接近程度计算损失\n",
|
||
" # 输出是怎样的形式\n",
|
||
" loss = nn.MSELoss()\n",
|
||
" loss = loss(output, target)\n",
|
||
"\n",
|
||
" # 反向传播以计算梯度\n",
|
||
" loss.backward()\n",
|
||
"\n",
|
||
" for name, param in model.named_parameters():\n",
|
||
" if 'weight' in name:\n",
|
||
" # 输出权重的平均绝对梯度\n",
|
||
" print(f\"{name} has gradient mean of {param.grad.abs().mean()}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "62e7a32c",
|
||
"metadata": {},
|
||
"source": [
|
||
"在上面的代码中，我们指定了一个损失函数，用于计算模型输出与用户指定的目标（此处为简单起见，值为 0）的接近程度。\n",
|
||
"然后，当调用 loss.backward() 时，PyTorch 计算模型中每一层的损失梯度。\n",
|
||
"我们可以通过 model.named_parameters() 迭代权重参数。\n",
|
||
"假设给定层有一个 3×3 的权重参数矩阵。\n",
|
||
"在这种情况下，该层将具有 3×3 梯度值，我们打印这些 3×3 梯度值的平均绝对梯度，以获得每层的单个梯度值，以便更容易地比较各层之间的梯度。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c23f78ee",
|
||
"metadata": {},
|
||
"source": [
|
||
"简而言之，.backward() 方法是 PyTorch 中计算模型训练期间所需的损失梯度的一种便捷方法，无需我们自己实现梯度计算的数学运算，从而使深度神经网络的使用变得更加容易。\n",
|
||
"如果不熟悉梯度和神经网络训练的概念，推荐阅读附录 A 中的 A.4“自动微分变得容易”和 A.7 “典型训练循环”部分。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e316b291",
|
||
"metadata": {},
|
||
"source": [
|
||
"现在让我们使用 print_gradients 函数，并将其应用于没有跳跃连接的模型："
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "788c5bf9",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"layers.0.0.weight has gradient mean of 0.0002017411752603948\n",
|
||
"layers.1.0.weight has gradient mean of 0.00012011770741082728\n",
|
||
"layers.2.0.weight has gradient mean of 0.0007152437465265393\n",
|
||
"layers.3.0.weight has gradient mean of 0.0013988513965159655\n",
|
||
"layers.4.0.weight has gradient mean of 0.005049604922533035\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print_gradients(model_without_shortcut, sample_input)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ec213561",
|
||
"metadata": {},
|
||
"source": [
|
||
"输出如下所示："
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2a524796",
|
||
"metadata": {},
|
||
"source": [
|
||
"layers.0.0.weight has gradient mean of 0.00020173587836325169 \\\n",
|
||
"layers.1.0.weight has gradient mean of 0.0001201116101583466 \\\n",
|
||
"layers.2.0.weight has gradient mean of 0.0007152041653171182 \\\n",
|
||
"layers.3.0.weight has gradient mean of 0.001398873864673078 \\\n",
|
||
"layers.4.0.weight has gradient mean of 0.005049646366387606"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "01eb6c8f",
|
||
"metadata": {},
|
||
"source": [
|
||
"从 print_gradients 函数的输出可以看出，梯度从最后一层（layers.4）到第一层（layers.0）逐渐变小，这种现象被称为梯度消失问题。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "71e1a75d",
|
||
"metadata": {},
|
||
"source": [
|
||
"现在，让我们实例化一个带有跳跃连接的模型，看看它与之前的模型有何不同："
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "cd3d189c",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"layers.0.0.weight has gradient mean of 0.22186796367168427\n",
|
||
"layers.1.0.weight has gradient mean of 0.207092747092247\n",
|
||
"layers.2.0.weight has gradient mean of 0.32923877239227295\n",
|
||
"layers.3.0.weight has gradient mean of 0.2667771875858307\n",
|
||
"layers.4.0.weight has gradient mean of 1.3268063068389893\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"torch.manual_seed(123)\n",
|
||
"model_with_shortcut = ExampleDeepNeuralNetwork(\n",
|
||
"layer_sizes, use_shortcut=True\n",
|
||
")\n",
|
||
"print_gradients(model_with_shortcut, sample_input)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4a7c3c36",
|
||
"metadata": {},
|
||
"source": [
|
||
"输出如下所示："
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bbc6e576",
|
||
"metadata": {},
|
||
"source": [
|
||
"layers.0.0.weight has gradient mean of 0.22169792652130127 \\\n",
|
||
"layers.1.0.weight has gradient mean of 0.20694105327129364 \\\n",
|
||
"layers.2.0.weight has gradient mean of 0.32896995544433594 \\\n",
|
||
"layers.3.0.weight has gradient mean of 0.2665732502937317 \\\n",
|
||
"layers.4.0.weight has gradient mean of 1.3258541822433472 \\"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4fa22ff7",
|
||
"metadata": {},
|
||
"source": [
|
||
"正如我们所看到的，根据输出，最后一层（layers.4）仍然比其他层具有更大的梯度。\n",
|
||
"然而，随着我们向第一层（layers.0）进展，梯度值变得更加稳定，并没有缩减到极小的数值。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "028b8b5b",
|
||
"metadata": {},
|
||
"source": [
|
||
"总之，快捷连接对于克服深度神经网络中梯度消失问题所带来的限制非常重要。\n",
|
||
"快捷连接是大型模型（例如 LLMs）的核心构建块，当我们在下一章中训练 GPT 模型时，它们将通过确保跨层梯度流的一致来帮助促进更有效的训练。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "114fddd3",
|
||
"metadata": {},
|
||
"source": [
|
||
"介绍完快捷连接后，我们现在将在下一节的transfomer模块中连接所有先前介绍的概念（层归一化、GELU 激活函数、前馈模块和快捷连接），这是我们编码所需的最终构建块GPT 架构。"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python (cell)",
|
||
"language": "python",
|
||
"name": "cell"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.10.13"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|