llms-from-scratch-cn/Translated_Book/ch04/4.3 实现使用 GELU 激活函数的前馈网络.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1baec45",
   "metadata": {},
   "source": [
    "## 4.3 实现使用 GELU 激活函数的前馈网络"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fffec75",
   "metadata": {},
   "source": [
    "在本节中，我们将实现一个小型神经网络子模块，该子模块在 LLM 的 Transformer 块中起着重要作用。我们首先实现 GELU 激活函数，这是该神经网络子模块的关键部分。（关于在 PyTorch 中实现神经网络的更多信息，请参见附录 A 中的 A.5 节 \"实现多层神经网络\"。）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83289a6",
   "metadata": {},
   "source": [
    "ReLU 激活函数因其简单性和在各种神经网络架构中的有效性，已在深度学习中得到广泛应用。然而，在 LLM 中，除了传统的 ReLU，还有其他几种激活函数被采用。其中，GELU（高斯误差线性单元）和 SwiGLU（Sigmoid-权重线性单元）是两个值得关注的激活函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf1498e",
   "metadata": {},
   "source": [
    "GELU 和 SwiGLU 是更复杂且平滑的激活函数，它们分别使用了高斯和 sigmoid 门控线性单元。与简单的 ReLU 不同，它们能为深度学习模型带来更优秀的性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "055b5653",
   "metadata": {},
   "source": [
    "GELU 激活函数可以通过多种方式实现；其精确版本定义为 $GELU(x)=x \\Phi(x)$，其中 $\\Phi(x)$ 是标准高斯分布的累积分布函数。然而，在实践中，通常会实现一个计算成本更低的近似版本（原始的 GPT-2 模型也是使用这种近似训练的）：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07d92a4f",
   "metadata": {},
   "source": [
    "$$GELU(x) \\approx 0.5 \\cdot x \\cdot (1 + \\tanh[\\sqrt{(2/\\pi)} \\cdot (x + 0.044715 \\cdot x^3)])$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01bde4a6",
   "metadata": {},
   "source": [
    "在代码中，我们可以用如下的 PyTorch 模块实现该功能："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eed1f327",
   "metadata": {},
   "source": [
    "**代码清单 4.3：GELU 激活函数的实现**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0dc0c34f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "class GELU(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "    def forward(self, x):\n",
    "        return 0.5 * x * (1 + torch.tanh(\n",
    "            torch.sqrt(torch.tensor(2.0 / torch.pi)) *\n",
    "            (x + 0.044715 * torch.pow(x, 3))\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f764d7e6",
   "metadata": {},
   "source": [
    "接下来，为了更好地理解 GELU 函数的形状以及它与 ReLU 函数的对比，我们将这两个函数并排绘制出来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5e580fc3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAEiCAYAAABdkh3zAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABCiklEQVR4nO3deVgT5/YH8G8SyMYmO7KKG6goKopFrwv9qWitlVulLm3FtdbCbS2ttnp7q9hbbeveulTrQqVSt1a911oVF7TWFZSKWmlVFGVHIEACSUjm9weSa5qAJCyThPN5njwwM+87c04CHGbmnRkOwzAMCCGEEBPCZTsAQggh5K+oOBFCCDE5VJwIIYSYHCpOhBBCTA4VJ0IIISaHihMhhBCTQ8WJEEKIyaHiRAghxORQcSKEEGJyqDgR0kyWLFkCDofDyrYTEhLA4XBw//79Vt92TU0NFixYAB8fH3C5XERGRrZ6DI3B5ntEDEfFiTRKVlYWYmNj0bVrV4jFYojFYnTv3h0xMTG4fv26Vtu6P9L1vfLz8wEA9+/fB4fDwcqVK+vdbocOHfDiiy/qXZaamgoOh4OEhIRmy/NZZDIZlixZgpSUlFbb5tOWLVuGgwcPsrLt+mzfvh0rVqzAhAkT8O233+Ldd99lNR5TfI+I4azYDoCYvsOHD2PixImwsrLCq6++iuDgYHC5XNy+fRs//vgjNm3ahKysLPj5+Wn127RpE2xtbXXW165du1aKvPnJZDLEx8cDAIYNG6a17KOPPsKHH37YottftmwZJkyYoLN38vrrr2PSpEkQCAQtun19Tp06BS8vL6xZs6bVt62PKb5HxHBUnEiD7t69i0mTJsHPzw8nT55E+/bttZZ//vnn2LhxI7hc3Z3wCRMmwMXFpbVCZZ2VlRWsrNj5leLxeODxeKxsu7Cw0Cz+4WDzPSKGo8N6pEFffPEFpFIpduzYoVOYgNo/yG+//TZ8fHxYiK5xSkpK8P7776Nnz56wtbWFvb09Ro8ejd9++02nbXV1NZYsWYKuXbtCKBSiffv2ePnll3H37l3cv38frq6uAID4+HjNYcolS5YA0D3nFBQUhPDwcJ1tqNVqeHl5YcKECZp5K1euxMCBA+Hs7AyRSISQkBDs379fqx+Hw4FUKsW3336r2fa0adMA1H8+ZePGjejRowcEAgE8PT0RExODsrIyrTbDhg1DUFAQbt26hfDwcIjFYnh5eeGLL75o8H2tOyx7+vRp3Lx5UxNTSkoKUlJSNN/r6/P0odhp06bB1tYWOTk5iIyMhK2tLVxdXfH+++9DpVLpvHfr1q1Dz549IRQK4erqilGjRiE1NdUk3yNiPCpOpEGHDx9G586dMWDAAIP7lpSUoLi4WOv111/61nDv3j0cPHgQL774IlavXo358+cjIyMDQ4cORW5urqadSqXCiy++iPj4eISEhGDVqlV45513IJFIcOPGDbi6umLTpk0AgL///e9ITExEYmIiXn75Zb3bnThxIs6ePas5x1bn3LlzyM3NxaRJkzTz1q1bhz59+mDp0qVYtmwZrKysEBUVhZ9++knTJjExEQKBAIMHD9Zse86cOfXmvWTJEsTExMDT0xOrVq3C+PHjsXnzZowcORJKpVKrbWlpKUaNGoXg4GCsWrUKgYGB+OCDD/Dzzz/Xu35XV1ckJiYiMDAQ3t7empi6detWb5/6qFQqREREwNnZGStXrsTQoUOxatUqbNmyRavdzJkzMW/ePPj4+ODzzz/Hhx9+CKFQiIsXL5rke0SagCGkHhKJhAHAREZG6iwrLS1lioqKNC+ZTKZZtnjxYgaA3ldAQICmXVZWFgOAWbFiRb0x+Pn5MWPGjNG77MqVKwwAZseOHQ3mUV1dzahUKq15WVlZjEAgYJYuXaqZt337dgYAs3r1ap11qNVqhmEYpqioiAHALF68WKdNXd51MjMzGQDMV199pdXurbfeYmxtbbXes6e/ZxiGUSgUTFBQEPP8889rzbexsWGio6N1tr1jxw4GAJOVlcUwDMMUFhYyfD6fGTlypFbu69evZwAw27dv18wbOnQoA4DZuXOnZp5cLmc8PDyY8ePH62zrr4YOHcr06NFDa97p06cZAMzp06e15td95k9/ZtHR0QwArc+CYRimT58+TEhIiGb61KlTDADm7bff1omh7vNhGNN8j4jhaM+J1Ku8vBwA9A5qGDZsGFxdXTWvDRs26LT54YcfkJycrPXasWNHi8f9VwKBQHNOTKVS4fHjx7C1tUVAQACuXr2qFa+Liwv+8Y9/6KzDmCHiXbt2Re/evbFnzx7NPJVKhf3792Ps2LEQiUSa+U9/X1paColEgsGDB2vFZ4gTJ05AoVBg3rx5WucDZ8+eDXt7e609MqD2M37ttdc003w+H6Ghobh3755R2zfGm2++qTU9ePBgre3/8MMP4HA4WLx4sU5fYz4fc3yP2hIaEEHqZWdnBwCorKzUWbZ582ZUVFSgoKBA6xf2aUOGDGmVARHP+sNUd55i48aNyMrK0jqP4ezsrPn+7t27CAgIaNZBDRMnTsSiRYuQk5MDLy8vpKSkoLCwEBMnTtRqd/jwYfz73/9Geno65HK5Zr6x1009ePAAABAQEKA1n8/no2PHjprldby9vXW25ejoqHOZQEupO3/01+2XlpZqpu/evQtPT084OTk1yzbN7T1qa2jPidTLwcEB7du3x40bN3SWDRgwAMOHD8egQYNaNAahUIiqqiq9y2QymaZNQ5YtW4a4uDgMGTIE3333HY4dO4bk5GT06NEDarW62WN+2sSJE8EwDPbt2wcA2Lt3LxwcHDBq1ChNm19++QUvvfQShEIhNm7ciCNHjiA5ORlTpkwBwzAtGl+d+kaxGbv9+orqXwc4PGv7pqS53yPSMCpOpEFjxozBnTt3cPnyZVa27+fnhz/++EPvsszMTE2bhuzfvx/h4eHYtm0bJk2ahJEjR2L48OE6gzM6deqEzMxMnRPhTzN0T8bf3x+hoaHYs2cPampq8OOPPyIyMlLrWpsffvgBQqEQx44dw4wZMzB69GgMHz68Sduve0/q3qM6CoVC7zVpzc3R0REAdN7jv+6NGKJTp07Izc1FSUlJg+3M5T0iDaPiRBq0YMECiMVizJgxAwUFBTrLW/q/xhdeeAGPHj3SueJfLpdj69atcHNzQ9++fRtcB4/H04lz3759yMnJ0Zo3fvx4FBcXY/369TrrqOsvFosB6P7RbcjEiRNx8eJFbN++HcXFxTqH9Hg8HjgcjtZexf379/Xe5cDGxqZR2x4+fDj4fD6+/PJLrdy3bdsGiUSCMWPGNDp+Y/j5+YHH4+Hs2bNa8zdu3Gj0OsePHw+GYTQXQT/t6RzN5T0iDaNzTqRBXbp0QVJSEiZPnoyAgADNHSIYhkFWVhaSkpLA5XLh7e2t03f//v16B1OMGDEC7u7umumTJ0+iurpap11kZCTeeOMNbN++HVFRUZgxYwb69OmDx48fY8+ePbhx4wZ27twJPp/fYA4vvvgili5diunTp2PgwIHIyMjArl270LFjR612U6dOxc6dOxEXF4fLly9j8ODBkEqlOHHiBN566y2MGzcOIpEI3bt3x549e9C1a1c4OTkhKCgIQUFB9W7/lVdewfvvv4/3338fTk5OOntFY8aMwerVqzFq1ChMmTIFhYWF2LBhAzp37qxzPiMkJAQnTpzA6tWr4enpCX9/f73D/F1dXbFw4ULEx8dj1KhReOmll5CZmYmNGzeif//+9Z4nbC4ODg6IiorCV199BQ6Hg06dOuHw4cMoLCw0ep3h4eF4/fXX8eWXX+LPP//EqFGjoFar8csvvyA8PByxsbEAzOc9Is/AziBBYm7u3LnDzJ07l+ncuTMjFAoZkUjEBAYGMm+++SaTnp6u1bahoeR4anhx3bDi+l6JiYkMw9QOW3/33XcZf39/xtramrG3t2fCw8OZn3/+uVGxV1dXM++99x7Tvn17RiQSMYMGDWIuXLjADB06lBk6dKhWW5lMxvzzn//UbMvDw4OZMGECc/fuXU2b8+fPMyEhIQyfz9caVv7XoeRPGzRoEAOAmTVrlt7l27ZtY7p06cIIBAImMDCQ2bFjh9713b59mxkyZAgjEokYAJoh038dJl1n/fr1TGBgIGNtbc24u7szc+fOZUpLS7Xa6BsKzjC1Q7z9/Pz0xtuY/kVFRcz48eMZsVjMODo6MnPmzGFu3Lihdyi5jY2NTn99+dfU1DArVqxgAgMDGT6fz7i6ujKjR49m0tLSNG1M8T0ihuMwDJ3NI4QQYlronBMhhBCTQ8WJEEKIyaHiRAghxORQcSKEEGJyqDgRQggxOVScCCGEmJw2dxGuWq1Gbm4u7OzsjL6pJiGEEMMxDIOKigp4enrqfXr209pcccrNzTXpp7YSQoile/jwod67yjytzRWnusdAPHz4EPb29gb3VyqVOH78OEaOHAlra+vmDo81lpgX5WQ+LDEvyklXeXk5fHx8NH+HG9LmilPdoTx7e3uji5NYLIa9vb3F/MABlpkX5WQ+LDEvyql+jTmlQgMiCCGEmBwqToQQQkwOq8Vp06ZN6NWrl+YQW1hYGH7++ecG++zbtw+BgYEQCoXo2bMnjhw50krREkIIaS2sFidvb2989tlnSEtLQ2pqKp5//nmMGzcON2/e1Nv+/PnzmDx5MmbOnIlr164hMjISkZGReh8jTgghxHyxWpzGjh2LF154AV26dEHXrl3x6aefwtbWFhcvXtTbft26dRg1ahTmz5+Pbt264ZNPPkHfvn31PrmUEEKI+TKZ0XoqlQr79u2DVCpFWFiY3jYXLlxAXFyc1ryIiAi9j7OuI5fLIZfLNdPl5eUAakedKJVKg+Os62NMX1NmiXlRTubDEvOyxJxSbhfgZA4HwxUKo/ob8l6wXpwyMjIQFhaG6upq2Nra4sCBA+jevbvetvn5+VqP9wYAd3d35Ofn17v+5cuXIz4+Xmf+8ePHIRaLjY47OTnZ6L6mzBLzopzMhyXmZSk5PawEvrrJg1zNg/33J9Hf1fDn1Mpkska3Zb04BQQEID09HRKJBPv370d0dDTOnDlTb4Ey1MKFC7X2tuouAhs5cqTR1zklJydjxIgRFnPtAmCZeVFO5sMS87KknB6UyLB0y2XI1Qp0dVDj/Veeh41IYPB66o5cNQbrxYnP56Nz584AgJCQEFy5cgXr1q3D5s2bddp6eHigoKBAa15BQQE8PDzqXb9AIIBAoPsmWltbN+kHpqn9TZUl5kU5mQ9LzMvccyqulGPmzqt4LFWgm4cdpvmUwkYkMConQ/qY3HVOarVa6xzR08LCwnDy5EmtecnJyfWeoyKEEGI8qbwGMxKu4MFjGbwdRdg6tS+ErbRLw+qe08KFCzF69Gj4+vqioqICSUlJSElJwbFjxwAAU6dOhZeXF5YvXw4AeOeddzB06FCsWrUKY8aMwe7du5GamootW7awmQYhhFgcpUqNubuu4vojCRzF1tg5IxRudoYfyjMWq8WpsLAQU6dORV5eHhwcHNCrVy8cO3YMI0aMAABkZ2dr3VZ94MCBSEpKwkcffYRFixahS5cuOHjwIIKCgthKgRBCLA7DMPjgh+s4+0cRRNY8bJ/WHx1dbVt15CGrxWnbtm0NLk9JSdGZFxUVhaioqBaKiBBCyIpjmfjxag54XA42vNoHfXwdWz0GkzvnRAghhD07L9zHxpS7AIDlf++J5wPdn9GjZVBxIoQQAgD4OSMPi/9Te/u490Z0xSv92XswKxUnQgghuJxVgnf2pINhgFcH+CL2+c6sxkPFiRBC2rg/Ciow69srUNSoMaK7O5aOC2rUAwFbEhUnQghpw/IkVYjefhnl1TUI8XPEV5P7gMdltzABVJwIIaTNklQpMW37FeRJqtHR1QZbp/aD0JrHdlgAqDgRQkibJK9RYU5iKjILKuBqJ8C300PhaMNnOywNKk6EENLGqNUM3tv7Gy7eK4GtwAoJ0/vDx8n4pzS0BCpOhBDSxiw78jsOX8+DFZeDTa/1RQ9PB7ZD0kHFiRBC2pBt57Kw9VwWAGBFVC8M7uLKckT6UXEihJA24qfrefj3T7cAAB+MCsTf+3izHFH9qDgRQkgbcOneY7z75CLbqWF+eHNoR7ZDahAVJ0IIsXB/FFRg9s5UKFRqRPRwx+KxPVi/yPZZqDgRQogFKyivxrSnLrJdN8k0LrJ9FipOhBBioSqqlZi24wpyTfAi22eh4kQIIRZIUaPG3O+u4ve8crjYmt5Fts9CxYkQQiwMwzD48MfrOHenGGI+Dzummd5Fts9CxYkQQizM6uQ/nnqSbV/09Da9i2yfhdXitHz5cvTv3x92dnZwc3NDZGQkMjMzG+yTkJAADoej9RIKha0UMSGEmLbdl7Px1ak7AIBlfw9CeIAbyxEZh9XidObMGcTExODixYtITk6GUqnEyJEjIZVKG+xnb2+PvLw8zevBgwetFDEhhJiu05mF+OfBGwCAt5/vjIn9fVmOyHhWbG786NGjWtMJCQlwc3NDWloahgwZUm8/DocDDw+Plg6PEELMRsYjCWJ2XYVKzWBCiDfeHdGV7ZCaxKTOOUkkEgCAk5NTg+0qKyvh5+cHHx8fjBs3Djdv3myN8AghxCQ9LJFhesIVyBQqDO7iguUv9zT5i2yfhdU9p6ep1WrMmzcPgwYNQlBQUL3tAgICsH37dvTq1QsSiQQrV67EwIEDcfPmTXh7694nSi6XQy6Xa6bLy8sBAEqlEkql0uA46/oY09eUWWJelJP5sMS8WisnSZUS0dsvo7hSjkB3W6x7pRegVkGpVjX7tpqakyH9OAzDMEZtpZnNnTsXP//8M86dO6e3yNRHqVSiW7dumDx5Mj755BOd5UuWLEF8fLzO/KSkJIjF5jW0khBCnlajBjbe4uFuBQft+AzeDVKhnYDtqOonk8kwZcoUSCQS2NvbN9jWJIpTbGwsDh06hLNnz8Lf39/g/lFRUbCyssL333+vs0zfnpOPjw+Ki4uf+eboo1QqkZycjBEjRsDa2trg/qbKEvOinMyHJebV0jmp1Qze25+Bwxn5sBVYYfes/gjwsGv27TytqTmVl5fDxcWlUcWJ1cN6DMPgH//4Bw4cOICUlBSjCpNKpUJGRgZeeOEFvcsFAgEEAt1/JaytrZv0A9PU/qbKEvOinMyHJebVUjl9cfQ2Dmfkw4rLwdevhSDIp+Fz9c3J2JwM6cNqcYqJiUFSUhIOHToEOzs75OfnAwAcHBwgEokAAFOnToWXlxeWL18OAFi6dCmee+45dO7cGWVlZVixYgUePHiAWbNmsZYHIYS0pqRL2diYchcA8Nn4XvhbFxeWI2p+rBanTZs2AQCGDRumNX/Hjh2YNm0aACA7Oxtc7v8GFZaWlmL27NnIz8+Ho6MjQkJCcP78eXTv3r21wiaEENaczizEvw7VXsv0zv91wYQQ031gYFOwfljvWVJSUrSm16xZgzVr1rRQRIQQYrpu5koQ++RapvF9vTFveBe2Q2oxJnWdEyGEEP1yy6owI+EKpAoVBnZytohrmRpCxYkQQkxcRbUSMxKuoKBcjq7uttj0Wgj4Vpb959uysyOEEDOnVKnx1q6ruJ1fAVc7AXZMD4WDyLJGNOpDxYkQQkwUwzD418Eb+OXPYoisedge3R9e7URsh9UqqDgRQoiJ2nTmLnZfeQguB/hqch+zfC6Tsag4EUKICfrvb7n44mjt8+0Wj+2B4d3dWY6odVFxIoQQE5P2oATv7fsNADBjkD+iB3ZgNyAWUHEihBAT8uCxFLN3pkFRo8bwbu7455hubIfECipOhBBiIspkCkzfcQUlUgV6ejngy8m9weNa7rVMDaHiRAghJkBRo8acxDTcK5bC00GIbdH9IOabzCP3Wh0VJ0IIYRnDMPjwx+u4lFUCW4EVtk3rDzd7IdthsYqKEyGEsGz9qTv48WoOeFwO1k/pg27tDX/WnKWh4kQIISz6z2+5WJX8BwBgyUs9MCzAjeWITAMVJ0IIYUnag1K8/2TI+My/+eP15/xYjsh0UHEihBAWPCyR4Y2dqZoh44teaJtDxutDxYkQQlpZ+ZO7jD+WKtDD0x7rJrXdIeP1oeJECCGtqEalRsyuq/izsBLu9gJsi+4PG0HbHTJeHypOhBDSShiGQfx/b2nuMr4tuj88HNr2kPH6UHEihJBWknD+PhIvPgCHA6yd1BtBXm3nLuOGYrU4LV++HP3794ednR3c3NwQGRmJzMzMZ/bbt28fAgMDIRQK0bNnTxw5cqQVoiWEEOOdvl2ITw7fAgB8OCoQET08WI7ItLFanM6cOYOYmBhcvHgRycnJUCqVGDlyJKRSab19zp8/j8mTJ2PmzJm4du0aIiMjERkZiRs3brRi5IQQ0niZ+RX4x/fXoGaAV/p5440hHdkOyeSxehbu6NGjWtMJCQlwc3NDWloahgwZorfPunXrMGrUKMyfPx8A8MknnyA5ORnr16/H119/3eIxE0KIIR5XyjEj4Qoq5TUY4O+Ef0f2BIdDI/OexaSGiEgkEgCAk5NTvW0uXLiAuLg4rXkRERE4ePCg3vZyuRxyuVwzXV5eDgBQKpVQKpUGx1jXx5i+pswS86KczIcl5qVUKqFUA3N3XUNOWRX8nMT4alIvcBgVlEoV2+EZpamfkyH9OAzDMEZtpZmp1Wq89NJLKCsrw7lz5+ptx+fz8e2332Ly5MmaeRs3bkR8fDwKCgp02i9ZsgTx8fE685OSkiAWi5sneEII+QuGAb67w0VqMRciHoN3e6rgLmI7KnbJZDJMmTIFEokE9vYN3z/QZPacYmJicOPGjQYLkzEWLlyotadVXl4OHx8fjBw58plvjj5KpRLJyckYMWIErK2tmzNUVlliXpST+bDEvDacvoPU4nvgcTj4+vUQDOzkzHZITdbUz6nuyFVjmERxio2NxeHDh3H27Fl4e3s32NbDw0NnD6mgoAAeHvpHvggEAggEAp351tbWTfolaGp/U2WJeVFO5sNS8jp6Iw9rT90DAHz8YiCGBlrWyDxjPydD+rA6Wo9hGMTGxuLAgQM4deoU/P39n9knLCwMJ0+e1JqXnJyMsLCwlgqTEEIa7UaOBO/uqb2Z6xAPNaaE+rAckXlidc8pJiYGSUlJOHToEOzs7JCfnw8AcHBwgEhUe3B26tSp8PLywvLlywEA77zzDoYOHYpVq1ZhzJgx2L17N1JTU7FlyxbW8iCEEAAoLK/G7J2pqFKqMLizMyJddM+Dk8Zhdc9p06ZNkEgkGDZsGNq3b6957dmzR9MmOzsbeXl5mumBAwciKSkJW7ZsQXBwMPbv34+DBw8iKCiIjRQIIQQAUK1UYXZiGvIk1ejkaoN1E3uBRyPGjcbqnlNjBgqmpKTozIuKikJUVFQLREQIIYZjGAYL9l/Hbw/L0E5sjW3R/WEnNP9zZ2yie+sRQkgTbTh9B//5LRdWXA42vtoXHVxs2A7J7FFxIoSQJjh6Ix8rj9c+Zj1+XA8M7OTCckSWwajDellZWfjll1/w4MEDyGQyuLq6ok+fPggLC4NQSLd/J4S0DbdyyxG3Nx0AMDXMD68OoMesNxeDitOuXbuwbt06pKamwt3dHZ6enhCJRCgpKcHdu3chFArx6quv4oMPPoCfH31IhBDLVVwpx+ydqZApVPhbZxd8/GJ3tkOyKI0uTn369AGfz8e0adPwww8/wMdHe+y+XC7HhQsXsHv3bvTr1w8bN26kQQuEEIukqFFj7ndpyCmrQgdnMdZP6QMrHp0laU6NLk6fffYZIiIi6l0uEAgwbNgwDBs2DJ9++inu37/fHPERQohJYRgG/zp4A1ful8JOaIWt0f3RTsxnOyyL0+ji1FBh+itnZ2c4O5v/faQIIeSvEs7fx57Uh+BygK8m90FnN1u2Q7JIRu2HJiQk6J1fU1ODhQsXNiUeQggxWb/8WaR5mu2iF7phWIAbyxFZLqOK09tvv42oqCiUlpZq5mVmZmLAgAH4/vvvmy04QggxFVnFUsTsugo1A0wI8cbMvz37XqDEeEYVp2vXruHRo0fo2bMnkpOTsWHDBvTt2xeBgYH47bffmjtGQghhVXm1ErO+vYLy6hr09W2HT/8eRE+zbWFGXefUqVMn/Prrr5g3bx5GjRoFHo+n8wBAQgixBCo1g3m703G3SIr2DkJ8/XoIBFY8tsOyeEaPffzpp5+we/duhIWFoV27dti2bRtyc3ObMzZCCGHdimOZOHW7EAIrLra83g9udnSjgdZgVHGaM2cOoqKi8MEHH+CXX37B9evXwefz0bNnT+zdu7e5YySEEFYcSs/B12fuAgC+mNALPb0dWI6o7TDqsN6vv/6KS5cuITg4GEDt02mPHDmCDRs2YMaMGXjllVeaNUhCCGlt1x+VYcH+6wCAucM6YVxvL5YjaluMKk5paWl6H30eExOD4cOHNzkoQghhU2FFNeYkpkFeo8bzgW54f2QA2yG1OUYd1tNXmOoEBNCHSAgxX/IaFeZ+dxV5kmp0dLXB2km9wePSyLzW1ujiNGrUKFy8ePGZ7SoqKvD5559jw4YNTQqMEEJaG8MwWPKfm0h78OTWRFP7wZ4eGsiKRh/Wi4qKwvjx4+Hg4ICxY8eiX79+8PT0hFAoRGlpKW7duoVz587hyJEjGDNmDFasWNGScRNCSLP77uIDfH/5IThPbk3U0ZVuTcSWRu85zZw5E/fu3cOiRYtw69YtvPHGGxg8eDD69++PiIgIfPPNN/D19cWVK1ewZ88e+Pr6PnOdZ8+exdixY+Hp6QkOh4ODBw822D4lJQUcDkfnlZ+f39g0CCFErwt3HyP+v7W3JvpwVCDdmohlBg2IEAgEeO211/Daa68BACQSCaqqquDs7Axra8N3faVSKYKDgzFjxgy8/PLLje6XmZkJe3t7zbSbG/0QEUKM96hUhpikq6hRM3gp2BNvDOnIdkhtnlGj9eo4ODjAwcH4cf+jR4/G6NGjDe7n5uaGdu3aGb1dQgipI1PU4I2daSiRKhDkZY/Px/eiWxOZAIOK05dffql3voODA7p27YqwsLBmCepZevfuDblcjqCgICxZsgSDBg2qt61cLodcLtdMl5eXAwCUSiWUSqXB267rY0xfU2aJeVFO5oOtvBiGwfy9GbiVVw4nG2tsmBQMK44aSqW6yeu2xM+qqTkZ0o/DMAzT2Mb+/vrvwltWVgaJRIKBAwfiP//5D5ycnBodgCYQDgcHDhxAZGRkvW0yMzORkpKCfv36QS6XY+vWrUhMTMSlS5fQt29fvX2WLFmC+Ph4nflJSUkQi8UGx0kIsRwncjj4bzYPXA6D2O4qdLJ/dh9iPJlMhilTpkAikWidmtHHoOLUkHv37uG1115D7969sXHjRoP7N6Y46TN06FD4+voiMTFR73J9e04+Pj4oLi5+5pujj1KpRHJyMkaMGGHUeTZTZYl5UU7mg428zvxRhNnfXQPDAEvGdsOroT7Nun5L/KyamlN5eTlcXFwaVZyadM7paR07dsRnn32GGTNmNNcqGyU0NBTnzp2rd7lAINB70bC1tXWTfmCa2t9UWWJelJP5aK287hVV4t19GWAYYHKoD6IH+rfYeSZL/KyMzcmQPkbflVwfX1/fVh/WnZ6ejvbt27fqNgkh5quiWok3EtNQUV2DED9HxL9Ez2YyRc225wQAGRkZ8PPza3T7yspK3LlzRzOdlZWF9PR0ODk5wdfXFwsXLkROTg527twJAFi7di38/f3Ro0cPVFdXY+vWrTh16hSOHz/enGkQQiyUWs0gbu9vuFNYCXd7ATa91hd8q2b9H500E4OKU91It7+SSCRIS0vDe++9h+jo6EavLzU1FeHh4ZrpuLg4AEB0dDQSEhKQl5eH7OxszXKFQoH33nsPOTk5EIvF6NWrF06cOKG1DkIIqc+Xp/5E8q0C8K242EzPZjJpBhWndu3a1bv7y+FwMGvWLHz44YeNXt+wYcPQ0HiMhIQErekFCxZgwYIFjV4/IYTUOX4zH2tP/AkA+DQyCL192rEbEGmQQcXp9OnTeufb29ujS5cuEAqFKCwshKenZ7MERwghzeHPggq8uycdADBtYAdE9WvekXmk+RlUnIYOHdrg8t9++w19+/aFSqVqUlCEENJcJFW1AyCkChWe6+iEf47pxnZIpBHoTCAhxGKp1Aze2X0NWcVSeLUTYcOUvrDm0Z89c0CfEiHEYq06nomUzCIIrbnY/HoInG3rf1AqMS1UnAghFunw9VxsTLkLAPh8fC8EeRl/k2rS+gw653T9+vUGl2dmZjYpGEIIaQ6/55Vj/r7av1dvDOmIcb29WI6IGMqg4tS7d29wOBy9w7/r5tOV1oQQNpVKFXgjMRVVShUGd3HBgogAtkMiRjCoOGVlZbVUHIQQ0mQ1KjViv7+KhyVV8HUS46vJfWBFAyDMkkHFyZBbExFCSGv77Ofb+PXOY4isedgyNQTtxHy2QyJGMuhfii+++AJVVVWa6V9//VXrcRQVFRV46623mi86QghppAPXHmHrudqjOyujghHoQQ9nMmcGFaeFCxeioqJCMz169Gjk5ORopmUyGTZv3tx80RFCSCNkPJLgwx8yAAAx4Z0wphc9qcDcGVSc/joQopmeU0gIIUYrrpRjTmIq5DVqPB/ohrgRNADCEtCZQkKI2VKq1Hhr11XkSqrR0cUGayb2Bo9LI4YtARUnQojZWvrfW7icVQJbgRW2TA2Bg8iynjjblhn8sMGtW7fC1tYWAFBTU4OEhAS4uLgAgNb5KEIIaUm7L2cj8eIDcDjA2om90dnNju2QSDMyqDj5+vrim2++0Ux7eHggMTFRpw0hhLSktAcl+NehGwCAuOFdMby7O8sRkeZmUHG6f/9+C4VBCCGNkyepwpzEq1CqGIwO8kDs853ZDom0AIOKU3V1NU6cOIEXX3wRQO3Q8qevc7KyssLSpUshFNKjjwkhza9aqcKcxDQUV8oR6GGHlVHBdMs0C2XQgIiEhASt65jWr1+P8+fP49q1a7h27RoSExOxcePGRq/v7NmzGDt2LDw9PcHhcHDw4MFn9klJSUHfvn0hEAjQuXNnnUe5E0IsE8MwWPhjBq4/kqCd2BrfTO0HG4HBp82JmTCoOO3atQtvvPGG1rykpCScPn0ap0+fxooVK7Bv375Gr08qlSI4OBgbNmxoVPusrCyMGTMG4eHhSE9Px7x58zBr1iwcO3bMkDQIIWZo6y9ZOHAtBzwuBxun9IWPk5jtkEgLMujfjjt37qBnz56aaaFQCC73f/UtNDQUMTExjV7f6NGjMXr06Ea3//rrr+Hv749Vq1YBALp164Zz585hzZo1iIiIaPR6CCHm5cwfRVj+8+8AgH+N6YaBnV1Yjoi0NIP2nMrKyrTOMRUVFaFDhw6aabVarbW8uV24cAHDhw/XmhcREYELFy602DYJIey6V1SJ2KSrUDPAK/28ET2wA9shkVZg0J6Tt7c3bty4gYAA/bcHuX79Ory9vZslMH3y8/Ph7q49ZNTd3R3l5eWoqqqCSCTS6SOXy7UKZnl5OQBAqVRCqVQaHENdH2P6mjJLzItyMh/15VVRrcSsb6+goroGfX3b4eMxgaipqWEjRINZ4mfV1JwM6WdQcXrhhRfw8ccfY8yYMToj8qqqqhAfH48xY8YYssoWt3z5csTHx+vMP378OMRi449ZJycnNyUsk2WJeVFO5uPpvNQM8M1tLu6VcdGOz+DvrsU4efwoi9EZxxI/K2NzkslkjW5rUHFatGgR9u7di4CAAMTGxqJr164Aah/Pvn79etTU1GDRokWGRWsADw8PFBQUaM0rKCiAvb293r0moHa4e1xcnGa6vLwcPj4+GDlyJOztDb+lvlKpRHJyMkaMGAFra8u5VYol5kU5mQ99eX1+7A/cKrsPgRUXO2aEIsjLvB6BYYmfVVNzqjty1RgGFSd3d3ecP38ec+fOxYcffqi5KzmHw8GIESOwceNGncNuzSksLAxHjhzRmpecnIywsLB6+wgEAggEAp351tbWTfqBaWp/U2WJeVFO5qMurx/SHmHrufsAap/N1KeDM7uBNYElflbG5mRIH4MvEvD398fRo0dRUlKCO3fuAAA6d+4MJycnQ1eFyspKzTqA2qHi6enpcHJygq+vLxYuXIicnBzs3LkTAPDmm29i/fr1WLBgAWbMmIFTp05h7969+OmnnwzeNiHENF3NLsXCH2ufzRQb3hljgz1Zjoiwwegr2JycnBAaGtqkjaempiI8PFwzXXf4LTo6GgkJCcjLy0N2drZmub+/P3766Se8++67WLduHby9vbF161YaRk6IhciTVGNOYhoUKjVGdndH3IiubIdEWMLq5dXDhg1r8IGF+u7+MGzYMFy7dq0FoyKEsEGuAt7cdQ1FFbW3JlozsTe49GymNovu/UEIYZ1azeC7O1zcKqmAsw2fbk1E6GGDhBD2rTt1F9dLuLDmcbD59RC6NRGh4kQIYdeh9BxsPHMPAPDvcd3Rr4Phg6uI5aHiRAhhTdqDEszfdx0A8LynGi/38WI5ImIqqDgRQljxsESGN3bWjswbHuiKsb5qtkMiJoSKEyGk1ZVXKzHz2yt4LFWgh6c9VkX1BA3MI0+j4kQIaVU1KjVik67hj4JKuNsLsDW6H8R8GplHtFFxIoS0GoZhsPg/N3H2jyKIrHnYOrU/2jvovy8maduoOBFCWs22c1nYdSkbHA6wblJv9PR2YDskYqKoOBFCWsWxm/n49Ejt02w/GtMdI3t4sBwRMWVUnAghLe5adine2X0NDAO8/pwfZgzqwHZIxMRRcSKEtKgHj6WY9W0qqpVqhAe4YvHY7uBwaGgeaRgVJ0JIiymRKjBtR+2Q8SAve6yf0hdWPPqzQ56NfkoIIS2iWqnC7J2pyCqWwqudCNun9aebuZJGo+JECGl2KjWDt7+/hrQHpbAXWiFhen+42QnZDouYESpOhJBmxTAMPj50A8dvFYBvxcU3U/uhi7sd22ERM0PFiRDSrNafuvO/a5km9saAjs5sh0TMEBUnQkiz2XMlG6uS/wAALBnbA6N7tmc5ImKuqDgRQprF0Rt5WPhjBgDgrWGdED2wA7sBEbNmEsVpw4YN6NChA4RCIQYMGIDLly/X2zYhIQEcDkfrJRTSiVZC2HT+bjHe/j4dagaY1N8H8yMC2A6JmDnWi9OePXsQFxeHxYsX4+rVqwgODkZERAQKCwvr7WNvb4+8vDzN68GDB60YMSHkaRmPJJj9bSoUKjUierjj35FBdJEtaTLWi9Pq1asxe/ZsTJ8+Hd27d8fXX38NsViM7du319uHw+HAw8ND83J3d2/FiAkhdf4sqMDU7ZcgVagQ1tEZ6yb1oYtsSbNg9Yo4hUKBtLQ0LFy4UDOPy+Vi+PDhuHDhQr39Kisr4efnB7Vajb59+2LZsmXo0aOH3rZyuRxyuVwzXV5eDgBQKpVQKpUGx1zXx5i+pswS86KcWlZ2iQyvbr2CUpkSPb3ssWFyMHhQQ6k0/Im2ppRXc6Gc6u/fGByGYRijttIMcnNz4eXlhfPnzyMsLEwzf8GCBThz5gwuXbqk0+fChQv4888/0atXL0gkEqxcuRJnz57FzZs34e3trdN+yZIliI+P15mflJQEsVjcvAkR0kaUyYF1N3kokXPgIWLwdg8VbKzZjoqYOplMhilTpkAikcDe3r7BtmZ3L5GwsDCtQjZw4EB069YNmzdvxieffKLTfuHChYiLi9NMl5eXw8fHByNHjnzmm6OPUqlEcnIyRowYAWtry/lttMS8KKeWUVwpx6vbrqBELoOvkwjfzwqFm52gSes0hbyaG+Wkq+7IVWOwWpxcXFzA4/FQUFCgNb+goAAeHo171ou1tTX69OmDO3fu6F0uEAggEOj+4lhbWzfpB6ap/U2VJeZFOTWfx5VyRCek4V6xDO0dhNg16zl4OTXfEQj6rMyDsTkZ0ofVM5d8Ph8hISE4efKkZp5arcbJkye19o4aolKpkJGRgfbt6WI/QlpSqVSBV7dewh8FlXC3F+D72c/BpxkLEyFPY/2wXlxcHKKjo9GvXz+EhoZi7dq1kEqlmD59OgBg6tSp8PLywvLlywEAS5cuxXPPPYfOnTujrKwMK1aswIMHDzBr1iw20yDEopXJFHh9+yXczq+Aq50ASbOfQwcXG7bDIhaM9eI0ceJEFBUV4eOPP0Z+fj569+6No0ePaoaHZ2dng8v93w5eaWkpZs+ejfz8fDg6OiIkJATnz59H9+7d2UqBEItWt8d0K68czjZ8JM0agE6utmyHRSwc68UJAGJjYxEbG6t3WUpKitb0mjVrsGbNmlaIihDyuFKOV7fW7jG52PKRNPs5usM4aRUmUZwIIaanqEKO17ZeQmZB7aG872cPQGc3KkykdVBxIoToyCmrwmtbLyGrWAp3+9pzTHQoj7QmKk6EEC1ZxVK8+s1F5Eqq4dVOhF2zBtDgB9LqqDgRQjR+zyvH69suo7hSjo4uNvhu1gB4thOxHRZpg6g4EUIAABfuPsYbO1NRIa9Bt/b2SJwZChfbpt35gRBjUXEihOBIRh7m7U6HQqVGaAcnfDO1HxzElnVXA2JeqDgR0sbtvHAfi/9zEwwDRPRwx7pJfSC05rEdFmnjqDgR0kap1Aw+/el3bP81CwAwZYAvPhkXBB6XHhRI2EfFiZA2SCqvwTu703Hi99qbLs+PCMBbwzrRE2yJyaDiREgbk1NWhTd2puJmbjn4VlysigrG2GBPtsMiRAsVJ0LakEv3HuOtXVfxWKqAsw0fW6b2Q4ifI9thEaKDihMhbQDDMPju4gPE//cWatQMure3x5apIfB2pEdeENNExYkQCydT1OCjAzfw47UcAMDYYE98Mb4XRHwakUdMFxUnQizYnwUVeGvXVfxZWAkel4MFEQF4Y0hHGvhATB4VJ0IsEMMw2J/2CB8fuokqpQpudgJ8NbkPBnR0Zjs0QhqFihMhFqZMpsCiAxk4kpEPABjU2RlrJ/aBqx3dioiYDypOhFiQc38W4/19vyG/vBpWXA7iRnbFnCGd6MJaYnaoOBFiAcqrlVj20+/YfeUhAKCjiw3WTeqDnt4OLEdGiHG4bAcAABs2bECHDh0gFAoxYMAAXL58ucH2+/btQ2BgIIRCIXr27IkjR460UqSEmJ4TtwowcvVZTWGaGuaHw2//jQoTMWusF6c9e/YgLi4OixcvxtWrVxEcHIyIiAgUFhbqbX/+/HlMnjwZM2fOxLVr1xAZGYnIyEjcuHGjlSMnhF0PS2SYvTMVs3amIr+8Gh2cxdg7JwxLxwVBzKeDIsS8sV6cVq9ejdmzZ2P69Ono3r07vv76a4jFYmzfvl1v+3Xr1mHUqFGYP38+unXrhk8++QR9+/bF+vXrWzlyQtihUAEbUu5hxJozSL5VACsuB3OGdMTP7wxBqL8T2+ER0ixY/fdKoVAgLS0NCxcu1MzjcrkYPnw4Lly4oLfPhQsXEBcXpzUvIiICBw8e1NteLpdDLpdrpsvLywEASqUSSqXS4JhP3MzD5UIOqtIeQmBtBR6Xo3lZcTmw4nFhxeXA+umvPA74PC74VlxYP/U9n8cF10ROVNe9F8a8J6bK0nJSqRn8ePUhvkjnoUxxBwAwwN8Ri8d0Qxd3WwBqKJVqdoM0kqV9VgDl1FD/xmC1OBUXF0OlUsHd3V1rvru7O27fvq23T35+vt72+fn5etsvX74c8fHxOvOPHz8OsdjwW7esvcFDVgUPuPu7wX314XEYWHMBKy7A5wLWT158LsDnMuDzar8X8AABF+DzGAh5tdPCJy+RVe08EQ8QWdW2N/Yay+Tk5GbJy5SYe04MA9ws5eDIQy5yZBwAHDjyGYz1U6OvcxH+TCvCn2wH2UzM/bPSh3L6H5lM1ui2Fn9geuHChVp7WuXl5fDx8cHIkSNhb29v8PquMb9DePsBHJ2cwYADpUoNNQPUqNWoUTFQqRkoVYxmukbNQKlSQ6FSQ6lioKjR/s9WxXCgUgFQ6duacRXGmseBvdAaDiIrtBPz0U5kjXZiaziKreFkw4ejmA9nWz5cbP73lQs1kpOTMWLECFhbW8YTUJVKpVnnxDAMTt4uwvqUu7iZWwEAsBNYIdxdjvjXwmErErIcYfMx989KH8pJV92Rq8ZgtTi5uLiAx+OhoKBAa35BQQE8PDz09vHw8DCovUAggECge/GhtbW1UW/uP8d0wxFOFl54ob9R/RmmtngpVGrIlSrIa9RPXipUK9WoVqpQpVShWqGCTFH7fZVCBamiBjKFClJ5DaTyGlTKVaiUK1FRXYNKeQ3Kq2q/r3lSHB9LFXgsVQBo3H8qdkIriDk8fJ+fDg8HEdzthXC1E6C9gwjt2wnR3kEIV1sBrHisn6Y0mLGfNVuqlSocSs/BtnNZ+KOgEgAg5vMwNawDpof54OKZE7AVCc0qp8Yyt8+qMSgn7X6NxWpx4vP5CAkJwcmTJxEZGQkAUKvVOHnyJGJjY/X2CQsLw8mTJzFv3jzNvOTkZISFhbVCxE3H4XDAt+KAb8WFraB5336GYSBTqCCpUkJSpUSZTIkymQJlVUqUSBUolSpQIlOgRKrA40oFHlfKUVypgEKlRkV1DSrAQUFWKYBSvevncTlwtxPAs50Inu1E8HIUwdtRBK92Ivg4ieHVTkSP926ChyUy7E19iKRL2U/+sQBs+DxED+yAWYM7wsmGb1HnLwhpCOuH9eLi4hAdHY1+/fohNDQUa9euhVQqxfTp0wEAU6dOhZeXF5YvXw4AeOeddzB06FCsWrUKY8aMwe7du5GamootW7awmYZJ4HA4sBFYwUZgBc92okb1YRgG5dU1yC2pxE8nf0GH7r1RIlOioFyO/PJqFEiqkSepRkF5NWrUDHIl1ciVVAMP9Bcwd3sBfJ3E8HESw/fJy89ZDF8nG7jY8umGo38hU9TgxO+F2Jf6EOfuFINhaud7tRNh2sAOmBjqA3uhZf3XTUhjsF6cJk6ciKKiInz88cfIz89H7969cfToUc2gh+zsbHC5/zuUNHDgQCQlJeGjjz7CokWL0KVLFxw8eBBBQUFspWDWOBwOHETWELvZoosDgxeC2+vd9VapGRRXypFbVoXcsmrklMmQU1qFR6VVyCmrwsMSGaQKFQrK5Sgol+PKfd3iZcPnwc/ZBh1cxLVfneu+2sDNTmAyIxdbmkxRg1/+LMbh63k4casAVcr/nXAc3MUFk/r7IqKHu1keQiWkubBenAAgNja23sN4KSkpOvOioqIQFRXVwlGRp/G4HLjbC+FuL0QfX93lDMOgTKZEdokMD0tlyC6RIfuxDA8e136fK6mCVKHCrbxy3MrTPSkqtOaig7MN/JzFT77WFi9fZzHaO4jM+t5wDMPgTmElfr1TjFOZRbh477HWwBgfJxHGBXthYn8f+DjRw/8IAUykOBHzx+Fw4GjDh6MNH8E+7XSWy2tUeFRahQePpcgqliH7sRT3H8tw/7EUj0qrUK1U43Z+BW7nV+j0teZx4OP4v0OFPk4i+DiK4e0ohpejCI5ia5M6XFhercSNHAlu5Ehw9UEZLt8vQcmTc0h1vB1FGNXDAy8GeyLY28Gk4ifEFFBxIq1CYMVDJ1dbdHK11VmmVKmRU1qF+4+luF9cW7QePClej0plUKoY3CuW4l6xVO+6xXwePByE8HQQob2D8MkengBOYitkVQD3H0vh5mADe6FVsxQBhmFQKa9BnqQaOWVVyC2rQlaRFHeKKnG3qBIPS6p0+gituejr64ihXV3xfKAbOrvZUkEipAFUnAjrrHlcdHCxQQcXGyBAe5lKzSBPUoXsJ4cHaw8Z1p7jyimrQlGFHDKFCveKpLhXpK94WWHtjV9rv+NyYC+yhr3QCvYia4iseRDxeRDzebDi1t7Rg8flgAGgVjNQMbXXpUkVKlQpalBRXYMymRIlMoXO9Wp/5e0oQk8vB/T0dsAAfyf09GoHvhWdQyKksag4EZPG43Lg/eQQ3kA9y6uVKuRJqpFXVoXcJ18LK+QorKgdYfiwsAzVjBWkChVq1AxKpAqdQ2zGchBZ1w6rdxCig4vNkz1DG3R1t4OjDb9ZtkFIW0XFiZg1oTUP/i428Hex0VmmVCpx5MgRvPBCBFTgolSmQEV1DSRVSlRUK2svcn5yobPm7h5qNTjggMcFuBwOBFZciPhWsOHzIBZYwUnMh6NN7Z026M7fhLQc+u0ibYLQmld7twt6xBEhZoEOghNCCDE5VJwIIYSYHCpOhBBCTA4VJ0IIISaHihMhhBCTQ8WJEEKIyWlzQ8mZJ88kMOSJjE9TKpWQyWQoLy+3qAeIWWJelJP5sMS8KCdddX936/4ON6TNFaeKitobi/r4+LAcCSGEtE0VFRVwcGj4okMO05gSZkHUajVyc3NhZ2dn1I03y8vL4ePjg4cPH8Le3r4FImSHJeZFOZkPS8yLctLFMAwqKirg6emp9Zw+fdrcnhOXy4W3t3eT12Nvb28xP3BPs8S8KCfzYYl5UU7anrXHVIcGRBBCCDE5VJwIIYSYHCpOBhIIBFi8eDEEAgHboTQrS8yLcjIflpgX5dQ0bW5ABCGEENNHe06EEEJMDhUnQgghJoeKEyGEEJNDxamJXnrpJfj6+kIoFKJ9+/Z4/fXXkZuby3ZYRrt//z5mzpwJf39/iEQidOrUCYsXL4ZCoWA7tCb59NNPMXDgQIjFYrRr147tcIy2YcMGdOjQAUKhEAMGDMDly5fZDqlJzp49i7Fjx8LT0xMcDgcHDx5kO6QmWb58Ofr37w87Ozu4ubkhMjISmZmZbIfVZJs2bUKvXr001zeFhYXh559/btFtUnFqovDwcOzduxeZmZn44YcfcPfuXUyYMIHtsIx2+/ZtqNVqbN68GTdv3sSaNWvw9ddfY9GiRWyH1iQKhQJRUVGYO3cu26EYbc+ePYiLi8PixYtx9epVBAcHIyIiAoWFhWyHZjSpVIrg4GBs2LCB7VCaxZkzZxATE4OLFy8iOTkZSqUSI0eOhFQqZTu0JvH29sZnn32GtLQ0pKam4vnnn8e4ceNw8+bNltsoQ5rVoUOHGA6HwygUCrZDaTZffPEF4+/vz3YYzWLHjh2Mg4MD22EYJTQ0lImJidFMq1QqxtPTk1m+fDmLUTUfAMyBAwfYDqNZFRYWMgCYM2fOsB1Ks3N0dGS2bt3aYuunPadmVFJSgl27dmHgwIEWcxdiAJBIJHBycmI7jDZNoVAgLS0Nw4cP18zjcrkYPnw4Lly4wGJkpCESiQQALOr3R6VSYffu3ZBKpQgLC2ux7VBxagYffPABbGxs4OzsjOzsbBw6dIjtkJrNnTt38NVXX2HOnDlsh9KmFRcXQ6VSwd3dXWu+u7s78vPzWYqKNEStVmPevHkYNGgQgoKC2A6nyTIyMmBrawuBQIA333wTBw4cQPfu3Vtse1Sc9Pjwww/B4XAafN2+fVvTfv78+bh27RqOHz8OHo+HqVOnNup5Ja3J0JwAICcnB6NGjUJUVBRmz57NUuT1MyYnQlpLTEwMbty4gd27d7MdSrMICAhAeno6Ll26hLlz5yI6Ohq3bt1qse3RHSL0KCoqwuPHjxts07FjR/D5fJ35jx49go+PD86fP9+iu7yGMjSn3NxcDBs2DM899xwSEhKeeXt7NhjzOSUkJGDevHkoKytr4eial0KhgFgsxv79+xEZGamZHx0djbKyMovYW+dwODhw4IBWfuYqNjYWhw4dwtmzZ+Hv7892OC1i+PDh6NSpEzZv3twi629zj8xoDFdXV7i6uhrVV61WAwDkcnlzhtRkhuSUk5OD8PBwhISEYMeOHSZZmICmfU7mhs/nIyQkBCdPntT88Var1Th58iRiY2PZDY5oMAyDf/zjHzhw4ABSUlIstjABtT9/Lfl3jopTE1y6dAlXrlzB3/72Nzg6OuLu3bv417/+hU6dOpnUXpMhcnJyMGzYMPj5+WHlypUoKirSLPPw8GAxsqbJzs5GSUkJsrOzoVKpkJ6eDgDo3LkzbG1t2Q2ukeLi4hAdHY1+/fohNDQUa9euhVQqxfTp09kOzWiVlZW4c+eOZjorKwvp6elwcnKCr68vi5EZJyYmBklJSTh06BDs7Ow05wMdHBwgEolYjs54CxcuxOjRo+Hr64uKigokJSUhJSUFx44da7mNttg4wDbg+vXrTHh4OOPk5MQIBAKmQ4cOzJtvvsk8evSI7dCMtmPHDgaA3pc5i46O1pvT6dOn2Q7NIF999RXj6+vL8Pl8JjQ0lLl48SLbITXJ6dOn9X4u0dHRbIdmlPp+d3bs2MF2aE0yY8YMxs/Pj+Hz+Yyrqyvzf//3f8zx48dbdJt0zokQQojJMc2TCYQQQto0Kk6EEEJMDhUnQgghJoeKEyGEEJNDxYkQQojJoeJECCHE5FBxIoQQYnKoOBFCCDE5VJwIIYSYHCpOhBBCTA4VJ0IIISaHihMhJq6oqAgeHh5YtmyZZt758+fB5/Nx8uRJFiMjpOXQjV8JMQNHjhxBZGQkzp8/j4CAAPTu3Rvjxo3D6tWr2Q6NkBZBxYkQMxETE4MTJ06gX79+yMjIwJUrVyAQCNgOi5AWQcWJEDNRVVWFoKAgPHz4EGlpaejZsyfbIRHSYuicEyFm4u7du8jNzYVarcb9+/fZDoeQFkV7ToSYAYVCgdDQUPTu3RsBAQFYu3YtMjIy4ObmxnZohLQIKk6EmIH58+dj//79+O2332Bra4uhQ4fCwcEBhw8fZjs0QloEHdYjxMSlpKRg7dq1SExMhL29PbhcLhITE/HLL79g06ZNbIdHSIugPSdCCCEmh/acCCGEmBwqToQQQkwOFSdCCCEmh4oTIYQQk0PFiRBCiMmh4kQIIcTkUHEihBBicqg4EUIIMTlUnAghhJgcKk6EEEJMDhUnQgghJoeKEyGEEJPz/wYNQM73gQsfAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVcAAAHWCAYAAADD3cplAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABA70lEQVR4nO3deVhTd74G8DdsYUeRJYiIiCuyKVaLbRVbFHfptEzbuV7RVu+0o721dJxKp9elnSkzt2PVqU6t06lMnfHqaFvcqBq1uBS0ioJii6O4y67sSwjJuX9gUhECBJOcJLyf5/F5mpOTnO/5Bd4eTn7neySCIAggIiKDshG7ACIia8RwJSIyAoYrEZERMFyJiIyA4UpEZAQMVyIiI2C4EhEZAcOViMgIGK5EREbAcCWLMG/ePAwYMECUba9cuRISiUSUbdfW1mLBggWQyWSQSCRYsmSJKHV0RswxMlcMVzOQmpoKiUSi/WdnZwd/f3/MmzcPd+7c6dZ7ZmRkQCKRYOfOnTrXkUgkWLx4cbvP7dy5ExKJBBkZGd3afncUFhZi5cqVyMnJMdk2Nerr67Fy5UqT7m9XfPDBB0hNTcVrr72GLVu24D//8z9Fq8Vcx8hc2YldAP3kvffeQ1BQEBobG3Hy5EmkpqbixIkTyMvLg6Ojo9jlGV1hYSFWrVqFAQMGIDIystVzf/3rX6FWq4227fr6eqxatQoAEBMT0+q5d999F8uWLTPatjty5MgRPP7441ixYoUo23+QuY6RuWK4mpGpU6di9OjRAIAFCxbAy8sLf/zjH7F79278/Oc/F7k6cdnb24u2bTs7O9jZifOrUlpaipCQEFG2rQ8xx8hc8bSAGXvqqacAAAUFBa2W5+fn4/nnn4enpyccHR0xevRo7N69W4wScePGDfzqV7/C0KFD4eTkhD59+iAhIQHXr19vs25lZSXefPNNDBgwAFKpFP369cPcuXNRXl6OjIwMPPbYYwCA+fPna0+RpKamAmh9zlWpVMLT0xPz589vs43q6mo4Ojri17/+NQCgqakJy5cvR1RUFDw8PODi4oKnnnoK3377rfY1169fh7e3NwBg1apV2m2vXLkSQPvnE5ubm/H+++8jODgYUqkUAwYMwDvvvAOFQtFqvQEDBmDGjBk4ceIExowZA0dHRwwcOBBffPFFh+OqOa1z7do17Nu3T1vT9evXtaeRHh5jzWse/LM9JiYGoaGh+OGHHzBx4kQ4OzvD398f//u//9tmm42NjVi5ciWGDBkCR0dH+Pn54Wc/+xkKCgrMcozMHcPVjGl+eXr37q1ddvHiRTz++OP48ccfsWzZMqxevRouLi6Ij4/H119/bfIaT58+jczMTLz44ov485//jFdffRWHDx9GTEwM6uvrtevV1tbiqaeewscff4zJkydj3bp1ePXVV5Gfn4/bt29j+PDheO+99wAA//Vf/4UtW7Zgy5YtGD9+fJtt2tvb49lnn0VaWhqamppaPZeWlgaFQoEXX3wRQEvYfvbZZ4iJicEf//hHrFy5EmVlZYiLi9Oe2/X29sYnn3wCAHj22We12/7Zz36mc78XLFiA5cuXY9SoUVizZg0mTJiAlJQU7XYfdOXKFTz//POYNGkSVq9ejd69e2PevHm4ePGizvcfPnw4tmzZAi8vL0RGRmpr0gScPioqKjBlyhRERERg9erVGDZsGN5++21888032nVUKhVmzJiBVatWISoqCqtXr8Ybb7yBqqoq5OXlmeUYmT2BRLd582YBgHDo0CGhrKxMuHXrlrBz507B29tbkEqlwq1bt7TrPvPMM0JYWJjQ2NioXaZWq4Vx48YJgwcP1i779ttvBQDCjh07dG4XgLBo0aJ2n9uxY4cAQPj22287rL2+vr7NsqysLAGA8MUXX2iXLV++XAAgfPXVV23WV6vVgiAIwunTpwUAwubNm9usk5iYKAQGBmofHzhwQAAg7Nmzp9V606ZNEwYOHKh93NzcLCgUilbrVFRUCL6+vsLLL7+sXVZWViYAEFasWNFm2ytWrBAe/FXJyckRAAgLFixotd6vf/1rAYBw5MgR7bLAwEABgHDs2DHtstLSUkEqlQpvvfVWm209LDAwUJg+fXqrZZqfl2vXrrVarvnMH/zMJkyY0OazUCgUgkwmE5577jntss8//1wAIHz00UdtatB8PuY6RuaKR65mJDY2Ft7e3ggICMDzzz8PFxcX7N69G/369QMA3Lt3D0eOHMHPf/5z1NTUoLy8HOXl5bh79y7i4uJw+fLlbs8u6C4nJyftfyuVSty9exeDBg1Cr169cPbsWe1zX375JSIiIvDss8+2eY/uTOF5+umn4eXlhe3bt2uXVVRUQC6X44UXXtAus7W1hYODAwBArVbj3r17aG5uxujRo1vVp4/09HQAQFJSUqvlb731FgBg3759rZaHhIRoT/EALUfKQ4cOxdWrV7u1fX25urpizpw52scODg4YM2ZMq+1/+eWX8PLywuuvv97m9d35fCxtjIyB4WpGNmzYALlcjp07d2LatGkoLy+HVCrVPn/lyhUIgoD/+Z//gbe3d6t/mm+TS0tLDVpTZ79YDQ0NWL58OQICAiCVSuHl5QVvb29UVlaiqqpKu15BQQFCQ0MNVpednR2ee+457Nq1S3sO76uvvoJSqWwVrgDw97//HeHh4XB0dESfPn3g7e2Nffv2tapPHzdu3ICNjQ0GDRrUarlMJkOvXr1w48aNVsv79+/f5j169+6NioqKbm1fX/369WvzOT68/YKCAgwdOtRgX0pZ2hgZA7/eMyNjxozRzhaIj4/Hk08+iV/84he4dOkSXF1dtVORfv3rXyMuLq7d93j4h7kjUqkUDQ0N7T6nOV/a2RSw119/HZs3b8aSJUsQHR0NDw8PSCQSvPjii0adOgUAL774Ij799FN88803iI+Px7/+9S8MGzYMERER2nX+8Y9/YN68eYiPj8fSpUvh4+MDW1tbpKSktPmiUF9dPaKztbVtd7nQzTss6dquSqUyyfb1IdYYmQOGq5nSBMDEiROxfv16LFu2DAMHDgTQ8oVObGzsI28jMDAQly5davc5zfLAwMAO32Pnzp1ITEzE6tWrtcsaGxtRWVnZar3g4GDk5eV1+F76/vk5fvx4+Pn5Yfv27XjyySdx5MgR/Pa3v21T38CBA/HVV1+1ev+H543qs+3AwECo1WpcvnwZw4cP1y4vKSlBZWVlp2P2qDRfcD48xg8fDeojODgYp06dglKp1DntzZLGyBzwtIAZi4mJwZgxY7B27Vo0NjbCx8cHMTEx+PTTT1FUVNRm/bKyMr3ef9q0aTh58iSys7NbLa+srMQ///lPREZGQiaTdfgetra2bY4uPv744zZHUc899xxyc3PbndGgeb2Li4t2+11hY2OD559/Hnv27MGWLVvQ3Nzc5pSA5ojowRpPnTqFrKysVus5Ozt3edvTpk0DAKxdu7bV8o8++ggAMH369C7V313BwcEAgGPHjmmXqVQqbNq0qdvv+dxzz6G8vBzr169v85xm7CxpjMwBj1zN3NKlS5GQkIDU1FS8+uqr2LBhA5588kmEhYVh4cKFGDhwIEpKSpCVlYXbt28jNze31eu//PJL5Ofnt3nfxMRELFu2DDt27MD48ePxy1/+EsOGDUNhYSFSU1NRVFSEzZs3d1rfjBkzsGXLFnh4eCAkJARZWVk4dOgQ+vTp02Y/du7ciYSEBLz88suIiorCvXv3sHv3bmzcuBEREREIDg5Gr169sHHjRri5ucHFxQVjx45FUFCQzu2/8MIL+Pjjj7FixQqEhYW1OkrS1PfVV1/h2WefxfTp03Ht2jVs3LgRISEhqK2t1a7n5OSEkJAQbN++HUOGDIGnpydCQ0PbPU8cERGBxMREbNq0CZWVlZgwYQK+//57/P3vf0d8fDwmTpzY6bg9ihEjRuDxxx9HcnIy7t27B09PT2zbtg3Nzc3dfs+5c+fiiy++QFJSEr7//ns89dRTqKurw6FDh/CrX/0Ks2fPtqgxMgviTVQgDc3UmtOnT7d5TqVSCcHBwUJwcLDQ3NwsCIIgFBQUCHPnzhVkMplgb28v+Pv7CzNmzBB27typfZ1mWo6uf8ePHxcEQRBu374tLFiwQPD39xfs7OwET09PYcaMGcLJkye7VHtFRYUwf/58wcvLS3B1dRXi4uKE/Px8ITAwUEhMTGy17t27d4XFixcL/v7+goODg9CvXz8hMTFRKC8v166za9cuISQkRLCzs2s1LevhqVgaarVaCAgIEAAIv/vd79p9/oMPPhACAwMFqVQqjBw5Uti7d2+775eZmSlERUUJDg4OraYcPTzNSBAEQalUCqtWrRKCgoIEe3t7ISAgQEhOTm41RU4Q2p9KJQgtU6QmTJjQ/qB24fUFBQVCbGysIJVKBV9fX+Gdd94R5HJ5u1OxRowY0eb17e1/fX298Nvf/la7TzKZTHj++eeFgoIC7TrmOEbmSiIIFnzGmIjITPGcKxGRETBciYiMgOFKRGQEDFciIiNguBIRGQHDlYjICHrcRQRqtRqFhYVwc3PjDdWISC+CIKCmpgZ9+/aFjU3Hx6Y9LlwLCwsREBAgdhlEZMFu3bqlbQWqS48LVzc3NwAtg+Pu7i5yNYajVCpx8OBBTJ48WdT7TVk6jqPhWONYVldXIyAgQJsjHelx4ao5FeDu7m514ers7Ax3d3er+UEWA8fRcKx5LLtySpFfaBERGQHDlYjICBiuRERGwHAlIjIChisRkREwXImIjIDhSkRkBAxXIiIjYLgSERkBw5WIyAhEDddPPvkE4eHh2ktRo6Oj8c0333T4mh07dmDYsGFwdHREWFgY0tPTTVQtEVHXiRqu/fr1wx/+8AdkZ2fjzJkzePrppzF79mxcvHix3fUzMzPx0ksv4ZVXXsG5c+cQHx+P+Ph45OXlmbhyIqKOiRquM2fOxLRp0zB48GAMGTIEv//97+Hq6oqTJ0+2u/66deswZcoULF26FMOHD8f777+PUaNGYf369SaunIioY2ZzzlWlUmHbtm2oq6tDdHR0u+tkZWUhNja21bK4uDhkZWWZokQisiKCIODL7NuorG8yyvuL3nLwwoULiI6ORmNjI1xdXfH1118jJCSk3XWLi4vh6+vbapmvry+Ki4t1vr9CoYBCodA+rq6uBtDSDk2pVBpgD8yDZl+saZ/EwHE0HHMfy5xblXhrRy48nOyQ9XYM7G07P9bUZ19ED9ehQ4ciJycHVVVV2LlzJxITE3H06FGdAauvlJQUrFq1qs3ygwcPwtnZ2SDbMCdyuVzsEqwCx9FwzHUsv7xmA8AGg1yaID+wv0uvqa+v7/L7ix6uDg4OGDRoEAAgKioKp0+fxrp16/Dpp5+2WVcmk6GkpKTVspKSEshkMp3vn5ycjKSkJO1jTSfxyZMnW12zbLlcjkmTJlldY2JT4jgajjmPZbNKjfc+PAagCa9NjcKEId5dep3mL9+uED1cH6ZWq1v9Gf+g6OhoHD58GEuWLNEuk8vlOs/RAoBUKoVUKm2z3N7e3uw+cEOw1v0yNY6j4ZjjWGZeK8Pduib0cXHAhGGyLp0SAKDXfogarsnJyZg6dSr69++PmpoabN26FRkZGThw4AAAYO7cufD390dKSgoA4I033sCECROwevVqTJ8+Hdu2bcOZM2ewadMmMXeDiCzMrpw7AIDp4X5dDlZ9iRqupaWlmDt3LoqKiuDh4YHw8HAcOHAAkyZNAgDcvHmz1e1rx40bh61bt+Ldd9/FO++8g8GDByMtLQ2hoaFi7QIRWZiGJhUO5LV8CT470t9o2xE1XP/2t791+HxGRkabZQkJCUhISDBSRURk7Q7nl6CuSYUATyeM6t/LaNsxm3muRESmkHauEAAwO8K/S3dx7S6GKxH1GJX1TTj671IAwOzIvkbdFsOViHqM9AvFUKoEhPi5Y7Cvm1G3xXAloh4j7f4sAWMftQIMVyLqIQorG/D9tXuQSIBZDFciIsPYndvyRdaYAZ7w83Ay+vYYrkTUI+zKaQnX+JHGm9v6IIYrEVm9f5fU4MeiatjbSjA1VHcvEkNiuBKR1dNc7jphiA96OTuYZJsMVyKyaoIgPHBKwPhfZGkwXInIqmXfqMDtiga4ONgidrhv5y8wEIYrEVk1zVFrXKgMjva2Jtsuw5WIrJZSpca+C0UAgHgjdsBqD8OViKzW8ctluFfXBC9XB4wL7mPSbTNcichqaU4JzAjvCzsjNcXWheFKRFapTtGMgxdb7rlnil4CD2O4EpFVOvRjCRqUKgT2cUZkQC+Tb5/hSkRWKe3c/Q5YEX2N2hRbF4YrEVmdu7UKHLtcDgCYbaJeAg9juBKR1Um/UASVWkCYvweCvV1FqYHhSkRWJ+3+LAExvsjSYLgSkVW5da8e2TcqIJEAMyMYrkREBqFpih09sA983R1Fq4PhSkRWQxAE7SwBU1/u+jCGKxFZjR+LanC5tBYOtjaIM1FTbF0YrkRkNXblthy1Pj3MBx5O9qLWwnAlIqugVgvYYwazBDQYrkRkFU5fv4fCqka4Se0wcZiP2OUwXInIOmjmtk4xcVNsXRiuRGTxmprVSL/fFHu2yLMENBiuRGTxjv67DFUNSvi4SRFt4qbYujBcicjiaW6dPTOiL2xtTN8Bqz0MVyKyaLWKZhz6Ubym2LowXInIoh28WIxGpRoDvVwQ5u8hdjlaDFcismiaWQKzIsVpiq0Lw5WILFZ5rQLfXWlpii12L4GHMVyJyGLtzS2ESi0gop8HBni5iF1OKwxXIrJYu3I1l7ua11ErwHAlIgt1424dzt2shI0EmBHhJ3Y5bTBcicgi7b7/RdYTg7zg4yZeU2xdGK5EZHEEQUDa/QsHZol4K5eOMFyJyOJcLKxGQVkdHOxsMEXkpti6MFyJyOJoLneNHe4DN0dxm2LrwnAlIouiUgvamxDOijC/WQIaDFcisiinrt1FSbUC7o52mDjMW+xydGK4EpFF0cwSmBbmB6md+E2xdWG4EpHFUDSrtE2xZ5lRB6z2MFyJyGJkXCpDdWMzZO6OeDzIPJpi6yJquKakpOCxxx6Dm5sbfHx8EB8fj0uXLnX4mtTUVEgkklb/HB3NbwIxERmeZpbArMi+sDGTpti6iBquR48exaJFi3Dy5EnI5XIolUpMnjwZdXV1Hb7O3d0dRUVF2n83btwwUcVEJJaaRiUO/VgKwHwvHHiQnZgb379/f6vHqamp8PHxQXZ2NsaPH6/zdRKJBDKZeU4cJiLj2J9XjKZmNYK9XTCir7vY5XRK1HB9WFVVFQDA09Ozw/Vqa2sRGBgItVqNUaNG4YMPPsCIESPaXVehUEChUGgfV1dXAwCUSiWUSqWBKhefZl+saZ/EwHE0HEOP5dfnbgMAZob7obm52SDvqS999kUiCIJgxFq6TK1WY9asWaisrMSJEyd0rpeVlYXLly8jPDwcVVVV+NOf/oRjx47h4sWL6NevX5v1V65ciVWrVrVZvnXrVjg7Oxt0H4jIOKqagBXZthAgwf+MbIaXSF+z1NfX4xe/+AWqqqrg7t7x0bPZhOtrr72Gb775BidOnGg3JHVRKpUYPnw4XnrpJbz//vttnm/vyDUgIADl5eWdDo4lUSqVkMvlmDRpEuztzfNyQEvAcTQcQ47l5swb+OCbS4gM8MCO/xproAr1V11dDS8vry6Fq1mcFli8eDH27t2LY8eO6RWsAGBvb4+RI0fiypUr7T4vlUohlUrbfZ01/vJY636ZGsfRcAwxlvsuFAMAnh3ZT9TPRZ9tizpbQBAELF68GF9//TWOHDmCoKAgvd9DpVLhwoUL8PMzv2a5RPTorpXXIfd2FWxtJJgebjm/56IeuS5atAhbt27Frl274ObmhuLilv87eXh4wMnJCQAwd+5c+Pv7IyUlBQDw3nvv4fHHH8egQYNQWVmJDz/8EDdu3MCCBQtE2w8iMh7N3NYnB3nBy7XtX6HmStRw/eSTTwAAMTExrZZv3rwZ8+bNAwDcvHkTNjY/HWBXVFRg4cKFKC4uRu/evREVFYXMzEyEhISYqmwiMhFBELArR3OfLPOf2/ogUcO1K9+lZWRktHq8Zs0arFmzxkgVEZE5uXCnCtfK6+Bob4PJIyxrbjt7CxCR2Uo713LUGjvcF65Ss/j+vcsYrkRkllRqAXvOt4RrvBneOrszDFciMktZBXdRVqNAL2d7jB9ivk2xdWG4EpFZ0swSmBbmBwc7y4sqy6uYiKxeo1KF/XktUzNnW0AHrPYwXInI7HybX4oaRTP6ejjisQEdN3IyVwxXIjI7afdPCcy0gKbYujBcicisVDUo8W1+GQDLnCWgwXAlIrOyP68ITSo1hvi6YpjMTexyuo3hSkRm5afLXf0hkVjmKQGA4UpEZqS4qhFZV+8CsIz7ZHWE4UpEZmPv+UIIAjA6sDcCPC37TiEMVyIyG5pZApbWAas9DFciMgtXSmuRd6cadjYSTA9nuBIRGcTu+0et44d4w9PFQeRqHh3DlYhEJwgCduVaZlNsXRiuRCS6nFuVuHG3Hs4OtpgU4it2OQbBcCUi0Wnmtk4O8YWzg2U1xdaF4UpEompWqbH3/E8XDlgLhisRiSqz4C7Ka5vg6eKAJwd7iV2OwTBciUhUmrmtM8L9YG9rPZFkPXtCRBanoUmFA5qm2FYyS0CD4UpEojmcX4K6JhX69XbCqP69xS7HoBiuRCQaza2zZ0f2tegOWO1huBKRKCrrm3D036UALLspti4MVyISRfqFYihVAob7uWOwr+U2xdaF4UpEothlRR2w2sNwJSKTK6xswKlr9yCRWH5TbF0YrkRkcnvuN2kZM8ATfXs5iVyNcTBcicjk0nKs73LXhzFcicik/l1Sgx+LqmFvK8G0MJnY5RgNw5WITErzRdaEIT7o5Wz5TbF1YbgSkckIgqBtLxg/0jq/yNJguBKRyWTfqMDtiga4ONjimWHW0RRbF4YrEZmM5qg1LlQGJwdbkasxLoYrEZmEUqXGvgtFAKx7loAGw5WITOLE5XLcq2uCl6sDngjuI3Y5RsdwJSKT+Kkpdl/YWVFTbF2sfw+JSHT1Tc04eLEEgPX2EngYw5WIjE7+QwkalCoE9nFGZEAvscsxCYYrERmdZpbA7Ajra4qtC8OViIzqXl0Tjv27DAAwqwfMEtBguBKRUX1zsQTNagGh/u4Y5OMqdjkmw3AlIqPak3t/bmtEzzlqBRiuRGREdxuB7JuVkEiAmVbaFFsXhisRGc3Zuy1fXj0e1AcyD0eRqzEthisRGU12WUvEWHsHrPaIGq4pKSl47LHH4ObmBh8fH8THx+PSpUudvm7Hjh0YNmwYHB0dERYWhvT0dBNUS0T6yC+uQVGDBPa2EkwJ9RO7HJMTNVyPHj2KRYsW4eTJk5DL5VAqlZg8eTLq6up0viYzMxMvvfQSXnnlFZw7dw7x8fGIj49HXl6eCSsnos7sOd/yRVbMEG94ONmLXI3p2Ym58f3797d6nJqaCh8fH2RnZ2P8+PHtvmbdunWYMmUKli5dCgB4//33IZfLsX79emzcuNHoNRNR59RqAXvOFwMAZoZb761cOiJquD6sqqoKAODp6alznaysLCQlJbVaFhcXh7S0tHbXVygUUCgU2sfV1dUAAKVSCaVS+YgVmw/NvljTPomB42gYp69XoKiqEY62Ap4K7mU146nPfphNuKrVaixZsgRPPPEEQkNDda5XXFwMX9/WHcx9fX1RXFzc7vopKSlYtWpVm+UHDx6Es7PzoxVthuRyudglWAWO46PZftUGgA0iPAUc+/aI2OUYTH19fZfXNZtwXbRoEfLy8nDixAmDvm9ycnKrI93q6moEBARg8uTJcHd3N+i2xKRUKiGXyzFp0iTY2/e881uGwnF8dE3Naqz436MAlIjyEqxqLDV/+XaFWYTr4sWLsXfvXhw7dgz9+vXrcF2ZTIaSkpJWy0pKSiCTtX9eRyqVQiqVtllub29vNR/4g6x1v0yN49h9Ry+XoLJBCW9XBwz2qLeqsdRnP0SdLSAIAhYvXoyvv/4aR44cQVBQUKeviY6OxuHDh1stk8vliI6ONlaZRKQHTVPs6WEy2PSMBljtEjVcFy1ahH/84x/YunUr3NzcUFxcjOLiYjQ0NGjXmTt3LpKTk7WP33jjDezfvx+rV69Gfn4+Vq5ciTNnzmDx4sVi7AIRPaBW0YxDP7b8ZTkzvOfNbX2QqOH6ySefoKqqCjExMfDz89P+2759u3admzdvoqioSPt43Lhx2Lp1KzZt2oSIiAjs3LkTaWlpHX4JRkSmcfBiMRqVagR5uSDM33q+0+gOUc+5CoLQ6ToZGRltliUkJCAhIcEIFRHRo9A2xY7sOU2xdWFvASIyiPJaBU5cKQfQM26d3RmGKxEZxL7zRVCpBUT080CQl4vY5YiO4UpEBqGZJdCTbuXSEYYrET2ym3frce5mJWwknCWgwXAloke26/5R67hgL/i496ym2LowXInokQiCoD0lMDuy5zXF1oXhSkSP5GJhNQrK6uBgZ4O40J7ZXrA9DFcieiS7c1vmtsYO94G7o3X0EDAEhisRdZtKLWD3/QsHZvWwW2d3huFKRN32/bV7KK5uhLujHSYO8xa7HLPCcCWibtPMEpgW5gepna3I1ZgXhisRdYuiWYX0Cy1NlWZxlkAbDFci6paMS2WobmyGzN0RY4P6iF2O2WG4ElG3aL7ImhnhB9ue3BVbB4YrEemtplGpbYrNDljtY7gSkd4OXCyBolmNYG8XjOjbs5ti68JwJSK97dJe7urf45ti68JwJSK9lNY04jttU2zOEtCF4UpEetmbWwS1AEQG9EJgHzbF1oXhSkR62XW/l0A8j1o7xHAloi67Vl6H3FuVsLWRYHo4w7UjDFci6jLN3NYnBnnB200qcjXmjeFKRF0iCMJPswQieNTaGYYrEXVJ3p1qXC2vg6M9m2J3BcOViLpEcyuX2OG+cJXaiVyN+WO4ElGnVGoBe+7PEuDlrl3DcCWiTp28ehelNQp4ONljwhA2xe4KhisRdSrt3E9NsR3sGBtd0e0TJ9euXcPx48dx48YN1NfXw9vbGyNHjkR0dDQcHXnfciJr0ahUYX9eMQBeOKAPvcP1n//8J9atW4czZ87A19cXffv2hZOTE+7du4eCggI4OjriP/7jP/D2228jMDDQGDUTkQl9m1+KGkUz+no44rEBnmKXYzH0CteRI0fCwcEB8+bNw5dffomAgIBWzysUCmRlZWHbtm0YPXo0/vKXvyAhIcGgBRORae3SNMWO7AsbNsXuMr3C9Q9/+APi4uJ0Pi+VShETE4OYmBj8/ve/x/Xr1x+1PiISUVWDEkfySwEAs3nrbL3oFa4dBevD+vTpgz59eF8dIkt2IK8YTSo1Bvu4Yrifm9jlWJRuf+2Xmpra7vLm5mYkJyd3922JyIxoLhyIH8mm2Prqdrj+93//NxISElBRUaFddunSJYwdOxb/93//Z5DiiEg8JdWNyLp6FwAwi70E9NbtcD137hxu376NsLAwyOVybNiwAaNGjcKwYcOQm5tryBqJSAR7cgshCEBUYG8EeDqLXY7F6fY81+DgYHz33XdYsmQJpkyZAltbW/z973/HSy+9ZMj6iEgkmlkCnNvaPY90qcW+ffuwbds2REdHo1evXvjb3/6GwsJCQ9VGRCIpKKvFhTtVsLWRYFqYn9jlWKRuh+svf/lLJCQk4O2338bx48dx/vx5ODg4ICwsDP/6178MWSMRmZjmqHX8YC/0cWVT7O7o9mmB7777DqdOnUJERAQAQCaTIT09HRs2bMDLL7+Mn//85wYrkohMp1VTbHbA6rZuh2t2djak0rb/R1u0aBFiY2MfqSgiEk/u7SrcuFsPJ3tbTArxFbsci9Xt0wLtBavG0KFDu/u2RCQyTQesySN84cKm2N2mV7hOmTIFJ0+e7HS9mpoa/PGPf8SGDRu6XRgRmV6zSo2954sAALM5S+CR6PW/pYSEBDz33HPw8PDAzJkzMXr0aPTt2xeOjo6oqKjADz/8gBMnTiA9PR3Tp0/Hhx9+aKy6icgIMgvuorxWAU8XBzw1mE2xH4Ve4frKK69gzpw52LFjB7Zv345NmzahqqoKACCRSBASEoK4uDicPn0aw4cPN0rBRGQ8mlkC08P8YG/LptiPQu8TKlKpFHPmzMGcOXMAAFVVVWhoaECfPn1gb29v8AKJyDQalSocuNjSFJunBB7dI5+t9vDwgIeHhyFqISIRHf6xFLWKZvj3ckJUYG+xy7F4eofrn//853aXe3h4YMiQIYiOju7yex07dgwffvghsrOzUVRUhK+//hrx8fE618/IyMDEiRPbLC8qKoJMxvuoEz2KNO3c1r7sgGUAeofrmjVr2l1eWVmJqqoqjBs3Drt374anZ+e3g6irq0NERARefvll/OxnP+tyDZcuXYK7u7v2sY+PT5dfS0RtVdUrkXGppSl2/EheOGAIeofrtWvXdD539epVzJkzB++++y7+8pe/dPpeU6dOxdSpU/UtAT4+PujVq5feryOi9qXnFUGpEjBM5oYhvmyKbQgG/Tpw4MCB+MMf/oCDBw8a8m3biIyMhJ+fHyZNmoTvvvvOqNsi6gl2PdAUmwzD4Jdf9O/fH8XFxYZ+WwCAn58fNm7ciNGjR0OhUOCzzz5DTEwMTp06hVGjRrX7GoVCAYVCoX1cXV0NAFAqlVAqlUapUwyafbGmfRJDTxzHoqpGnLp2DwAwNcTbYPtujWOpz74YPFwvXLhgtFtqDx06tNWltePGjUNBQQHWrFmDLVu2tPualJQUrFq1qs3ygwcPwtnZ+hoAy+VysUuwCj1pHI8USiAItgh2E5CT+S1yDPz+1jSW9fX1XV5X73DVHPk9rKqqCtnZ2XjrrbeQmJio79t225gxY3DixAmdzycnJyMpKUn7uLq6GgEBAZg8eXKrL8UsnVKphFwux6RJkzjf+BH0xHHcuCELQA0SJ4Zg2mMBBntfaxxLXfnXHr3DtVevXjqnaUgkEixYsADLli3T9227LScnB35+upv5SqXSdpvM2NvbW80H/iBr3S9T6ynjeLmkBj8W18DeVoJZkf2Mss/WNJb67Ife4frtt9+2u9zd3R2DBw+Go6MjSktL0bdv51d41NbW4sqVK9rH165dQ05ODjw9PdG/f38kJyfjzp07+OKLLwAAa9euRVBQEEaMGIHGxkZ89tlnOHLkiNG/QCOyVprLXScM8UEvZweRq7EueofrhAkTOnw+NzcXo0aNgkql6vS9zpw50+qiAM2f74mJiUhNTUVRURFu3rypfb6pqQlvvfUW7ty5A2dnZ4SHh+PQoUPtXlhARB0TBAG7cn+6cIAMS9RmjTExMRAEQefzqamprR7/5je/wW9+8xsjV0XUM5y9WYFb9xrg4mCL2OFsim1obHtD1ENpTgnEjZDBycFW5GqsD8OVqAdSPtgUmxcOGIXepwXOnz/f4fOXLl3qdjFEZBonLpfjXl0TvFwd8ERwH7HLsUp6h2tkZCQkEkm750o1y9lRh8i8aS53nRHeF3Zsim0UBm3cQkTmr76pGQd/KAEAzOIsAaPRO1yNdWkrEZmG/IcS1Dep0N/TGSMDeoldjtV6pL8Hjh8/jjlz5iA6Ohp37rT8mbFly5YOL0clInFpZgmwKbZxdTtcv/zyS8TFxcHJyQnnzp3Tdp6qqqrCBx98YLACichw7tU14di/ywDwwgFj63a4/u53v8PGjRvx17/+tdX1tk888QTOnj1rkOKIyLD2XShCs1rAiL7uGOTDptjG1O1wvXTpEsaPH99muYeHByorKx+lJiIykt05vNzVVLodrjKZrFXTFY0TJ05g4MCBj1QUERne7Yp6nL5eAYkEmBXBCweMrdvhunDhQrzxxhs4deoUJBIJCgsL8c9//hNvvfUWXnvtNUPWSEQGsDu35Yusx4P6QObhKHI11q/bjVuWLVsGtVqNZ555BvX19Rg/fjykUimWLl2KBQsWGLJGIjKAXed+miVAxtftI1eJRILf/va3uHfvHvLy8nDy5EmUlZXBw8MDQUFBhqyRiB5RfnE1LpXUwMHWBlNDdTeXJ8PRO1wVCgWSk5MxevRoPPHEE0hPT0dISAguXryIoUOHYt26dXjzzTeNUSsRdVPa/aPWmKHe8HC2jrsCmDu9TwssX74cn376KWJjY5GZmYmEhATMnz8fJ0+exOrVq5GQkABbW7YvIzIXarWAPffPt/LW2aajd7ju2LEDX3zxBWbNmoW8vDyEh4ejubkZubm5vNqDyAyduVGBO5UNcJXa4elhPmKX02PofVrg9u3biIqKAgCEhoZCKpXizTffZLASmSlNB6wpoTI42vOvSlPRO1xVKhUcHH66kZmdnR1cXV0NWhQRGUZTsxr7Ltxvis1ZAial92kBQRAwb9487e2qGxsb8eqrr8LFxaXVel999ZVhKiSibjt+uQyV9Up4uUoxLthL7HJ6FL3DNTExsdXjOXPmGKwYIjKstPsdsGZG+MHWhqfuTEnvcN28ebMx6iAiA6tTNEP+QzEAID6SswRMjfd3ILJSB38oRqNSjQF9nBHez0PscnochiuRldI0xZ4V6c/ZPCJguBJZofJaBY5fLgcAxHOWgCgYrkRWKP1CEVRqAeH9PDDQm1MlxcBwJbJCaec0TbH5RZZYGK5EVubm3XqcvVkJGwkwM5wdsMTCcCWyMrtzW45axwV7wcedTbHFwnAlsiKCIGgvHJjFL7JExXAlsiI/FFXjSmktHOxsMCVUJnY5PRrDlciKaOa2PjPMB+6ObIotJoYrkZVQqwXsztHcJ4uzBMTGcCWyEqeu3UNxdSPcHO0QM9Rb7HJ6PIYrkZXQzBKYFurHpthmgOFKZAUUzSrsO8+m2OaE4UpkBY5eKkN1YzN83aUYO7CP2OUQGK5EVkEzS2BmeF82xTYTDFciC1fTqMShH0sA8NbZ5oThSmThDlwsgaJZjYHeLhjR113scug+hiuRhdPcOjueTbHNCsOVyIKV1jTiuystTbFnRXCWgDlhuBJZsH3ni6AWgMiAXhjg5dL5C8hkGK5EFixNe7krj1rNDcOVyEJdL69D7q1K2NpIMCOc4WpuGK5EFkozt3VccB94u0lFroYexnAlskCCIGBX7k+zBMj8iBqux44dw8yZM9G3b19IJBKkpaV1+pqMjAyMGjUKUqkUgwYNQmpqqtHrJDI3eXeqcbWsDlI7G8SxKbZZEjVc6+rqEBERgQ0bNnRp/WvXrmH69OmYOHEicnJysGTJEixYsAAHDhwwcqVE5kUztzU2xBeuUjuRq6H2iPqpTJ06FVOnTu3y+hs3bkRQUBBWr14NABg+fDhOnDiBNWvWIC4uzlhlEpkVlVrA7tz7swQ4t9VsWdQ516ysLMTGxrZaFhcXh6ysLJEqIjK9U1fvorRGAQ8ne8QM9RG7HNLBov6eKC4uhq+vb6tlvr6+qK6uRkNDA5ycnNq8RqFQQKFQaB9XV1cDAJRKJZRKpXELNiHNvljTPonBEsbxq7O3AQBTRvhAIqigVKpErqh9ljCW+tJnXywqXLsjJSUFq1atarP84MGDcHZ2FqEi45LL5WKXYBXMdRyVamBfri0ACbzrbyI9/YbYJXXKXMeyO+rr67u8rkWFq0wmQ0lJSatlJSUlcHd3b/eoFQCSk5ORlJSkfVxdXY2AgABMnjwZ7u7W00FIqVRCLpdj0qRJsLfnXT+7y9zH8cDFEjScyoXMXYrXXxgPGzPu3WruY9kdmr98u8KiwjU6Ohrp6emtlsnlckRHR+t8jVQqhVTadoK1vb291XzgD7LW/TI1cx3HfXktBxezI/0hlTqIXE3XmOtYdoc++yHqF1q1tbXIyclBTk4OgJapVjk5Obh58yaAlqPOuXPnatd/9dVXcfXqVfzmN79Bfn4+/vKXv+Bf//oX3nzzTTHKJzKp6kYlDueXAgBmsZeA2RM1XM+cOYORI0di5MiRAICkpCSMHDkSy5cvBwAUFRVpgxYAgoKCsG/fPsjlckRERGD16tX47LPPOA2LeoT9ecVoalZjsI8rQvys55SWtRL1tEBMTAwEQdD5fHtXX8XExODcuXNGrIrIPGmbYo9kU2xLYFHzXIl6qpLqRmQW3AXAptiWguFKZAH25BZCEIBR/XshwNP6phBaI4YrkQXQtBfk3V0tB8OVyMwVlNXiwp0q2NpIMD3MT+xyqIsYrkRmTnPUOn6wF/q4sim2pWC4EpkxQRCw+/4sgdlsim1RGK5EZiz3dhWu362Hk70tJoX4dv4CMhsMVyIzppnbOnmEL1zYFNuiMFyJzFSzSo09uUUAeOtsS8RwJTJTWVfvorxWgd7O9nhqsLfY5ZCeGK5EZirtXMssgenhfrC35a+qpeEnRmSGGpUqHLhYDIC3zrZUDFciM3T4x1LUKprh38sJo/r3Frsc6gaGK5EZ2qWd29rXrO82QLoxXInMTFW9EhmXygDwwgFLxnAlMjPf5BWhSaXGMJkbhsrcxC6HuonhSmRm0ni5q1VguBKZkaKqBpy6dg8A75Nl6RiuRGZE0xR7zABP+Pdq/3bxZBkYrkRmRHPhwOyRPGq1dAxXIjNxuaQGPxRVw85GgmmhbIpt6RiuRGZC0xQ7Zqg3ers4iFwNPSqGK5EZEAQBu3JbZgnM4iwBq8BwJTIDZ29W4ta9Bjg72GLScDbFtgYMVyIzoLncNW6EDE4OtiJXQ4bAcCUSmVKlxr7zbIptbRiuRCI7caUcd+ua0MfFAU8O8hK7HDIQhiuRyHbfnyUwI9wPdmyKbTX4SRKJqL6pWdsUm7MErAvDlUhEh34sRX2TCgGeThjVv5fY5ZABMVyJRLTrXMssgfhIf0gkbIptTRiuRCK5V9eEo//WNMXmLAFrw3AlEkn6hSI0qwWM6OuOQT5sim1tGK5EInnwPllkfRiuRCK4XVGP09crIJEAMyMYrtaI4Uokgt25LXNbxwZ5ws+DTbGtEcOVSASaCwd4nyzrxXAlMrH84mrkF9fAwdaGTbGtGMOVyMQebIrt4WwvcjVkLAxXIhNSqwWeEughGK5EJpR9swJ3KhvgKrXDM8N9xC6HjIjhSmRCaed+aortaM+m2NaM4UpkIkqVGukXWppix/PW2VaP4UpkIsf+XYaKeiW8XKWIHthH7HLIyBiuRCaimSUwM4JNsXsCfsJEJlCnaIb8hxIAnCXQUzBciUxA/kMJGpQqDOjjjIh+HmKXQybAcCUygTRtByw2xe4pzCJcN2zYgAEDBsDR0RFjx47F999/r3Pd1NRUSCSSVv8cHR1NWC2Rfu7WKnD8cjkAthfsSUQP1+3btyMpKQkrVqzA2bNnERERgbi4OJSWlup8jbu7O4qKirT/bty4YcKKifSz70IRVGoB4f08MNDbVexyyERED9ePPvoICxcuxPz58xESEoKNGzfC2dkZn3/+uc7XSCQSyGQy7T9fX18TVkykH80sgVns29qj2Im58aamJmRnZyM5OVm7zMbGBrGxscjKytL5utraWgQGBkKtVmPUqFH44IMPMGLEiHbXVSgUUCgU2sfV1dUAAKVSCaVSaaA9EZ9mX6xpn8Rg6HG8VVGP7BstTbGnjvDpUZ+PNf5M6rMvooZreXk5VCpVmyNPX19f5Ofnt/uaoUOH4vPPP0d4eDiqqqrwpz/9CePGjcPFixfRr1+/NuunpKRg1apVbZYfPHgQzs7OhtkRMyKXy8UuwSoYahwP3pYAsMVgdzXOHD9skPe0NNb0M1lfX9/ldUUN1+6Ijo5GdHS09vG4ceMwfPhwfPrpp3j//ffbrJ+cnIykpCTt4+rqagQEBGDy5Mlwd3c3Sc2moFQqIZfLMWnSJNjbs41ddxlyHAVBwMcfZwKow/ynwzBtVM+a32qNP5Oav3y7QtRw9fLygq2tLUpKSlotLykpgUwm69J72NvbY+TIkbhy5Uq7z0ulUkil0nZfZy0f+IOsdb9MzRDj+ENhNa6U1cHBzgbTI/x77OdiTT+T+uyHqF9oOTg4ICoqCocP//TnklqtxuHDh1sdnXZEpVLhwoUL8PNjR3cyL5q7uz4zzAfujtYRLtR1op8WSEpKQmJiIkaPHo0xY8Zg7dq1qKurw/z58wEAc+fOhb+/P1JSUgAA7733Hh5//HEMGjQIlZWV+PDDD3Hjxg0sWLBAzN0gakWtFrQ3IeTc1p5J9HB94YUXUFZWhuXLl6O4uBiRkZHYv3+/9kuumzdvwsbmpwPsiooKLFy4EMXFxejduzeioqKQmZmJkJAQsXaBqI3vr99DUVUj3KR2iBnKptg9kejhCgCLFy/G4sWL230uIyOj1eM1a9ZgzZo1JqiKqPs0c1unhrEpdk8l+kUERNamqfmBptjsgNVjMVyJDCzjUimqGpTwcZNiLJti91gMVyID25X70+WutjbsgNVTMVyJDKimUYlDbIpNYLgSGdTBiyVQNKsx0NsFof7WcwUg6Y/hSmRA2qbYEWyK3dMxXIkMpKxGge+usCk2tWC4EhnI3vOFUAtAREAvDPByEbscEhnDlchANBcOxPOolcBwJTKI6+V1yLlVCRsJMD2cTYSI4UpkEJomLU8M8oKPG2+YSQxXokcmCEKrW2cTAQxXokd2sbAaV8vqILWzQdwI3iyTWjBciR5R2rmWo9bY4b5wY1Nsuo/hSvQIVGoBe86zKTa1xXAlegSnrt5FSbUC7o52mDDUW+xyyIwwXIkegWZu6/RwP0jt2BSbfsJwJeqmRqUK6XktTbFnRXCWALXGcCXqpoxLZahpbIafhyPGBnmKXQ6ZGYYrUTdpbp09M6IvbNgUmx7CcCXqhupGJQ7nlwLgLAFqH8OVqBv25xWjqVmNwT6uCPFjU2xqi+FK1A27c36a28qm2NQehiuRnkqrG5FZ0NIUm7MESBeGK5Ge9pwvgloARvXvhf59nMUuh8wUw5VIT7vYAYu6gOFKpIerZbU4f7sKtjYSNsWmDjFcifSgudz1yUFe8HKVilwNmTOGK1EXCYKgveNA/EjObaWOMVyJuuj87SpcK6+Do70NJofIxC6HzBzDlaiLNKcEJoXI4CK1E7kaMncMV6IueLApNm+dTV3BcCXqgsyCcpTVKNDb2R7jh7ApNnWO4UrUBZpTAtPC/GBvy18b6hx/Sog60ahUYX9eMQAgfiQvHKCuYbgSdeJIfilqFc3w7+WEqP69xS6HLATDlagTmltnz4pkU2zqOoYrUQeq6pXIuFQGAIhnLwHSA8OVqAPf5BWhSaXGMJkbhsrcxC6HLAjDlagDu7RNsXnUSvphuBLpUFzViJPX7gIAZkawAxbph+FKpMOe3EIIAvDYgN7o15tNsUk/DFciHdLYFJseAcOVqB1XSmtxsbAadjYSTA/jKQHSH8OVqB17zrdckTVhiDd6uziIXA1ZIoYr0UMEoeUmhAAwm5e7UjcxXIkecqMWuFXRAGcHW8QO9xG7HLJQZhGuGzZswIABA+Do6IixY8fi+++/73D9HTt2YNiwYXB0dERYWBjS09NNVClZO0EQcLiw5dciboQMzg5sik3dI3q4bt++HUlJSVixYgXOnj2LiIgIxMXFobS0tN31MzMz8dJLL+GVV17BuXPnEB8fj/j4eOTl5Zm4crJGX50rxPl7NrCzkeCVJ4PELocsmOjh+tFHH2HhwoWYP38+QkJCsHHjRjg7O+Pzzz9vd/1169ZhypQpWLp0KYYPH473338fo0aNwvr1601cOVmbG3fr8P6+fADAfz8djFB/D5ErIksm6t88TU1NyM7ORnJysnaZjY0NYmNjkZWV1e5rsrKykJSU1GpZXFwc0tLS2l1foVBAoVBoH1dXVwMAlEollEplpzXuv1iCTcevdbqe2ARBQFWVLf56IwsSCTs3dUdJtQJ1TSoEuwmY/3i/Lv18kG6a8bOmcdRnX0QN1/LycqhUKvj6+rZa7uvri/z8/HZfU1xc3O76xcXF7a6fkpKCVatWtVl+8OBBODt3ftXNiWIJLtyx7XQ98yDBrboasYuwaE62Av5jkApHDh8SuxSrIZfLxS7BYOrr67u8rtWfrU9OTm51pFtdXY2AgABMnjwZ7u7unb4+orIBk0pqjVmiQaiam3EuJwcjIyNha2f1H6vRBHk64ofTxzFp0iTY29uLXY5FUyqVkMvlVjWWmr98u0LU30IvLy/Y2tqipKSk1fKSkhLIZO3fF14mk+m1vlQqhVQqbbPc3t6+Sx/4AG97DPDuPITFplQq0XTjHGJH+FnND7IYlEolfkDXfz6oc9Y0lvrsh6hfaDk4OCAqKgqHDx/WLlOr1Th8+DCio6PbfU10dHSr9YGWPzt0rU9EJAbR/35MSkpCYmIiRo8ejTFjxmDt2rWoq6vD/PnzAQBz586Fv78/UlJSAABvvPEGJkyYgNWrV2P69OnYtm0bzpw5g02bNom5G0RErYgeri+88ALKysqwfPlyFBcXIzIyEvv379d+aXXz5k3Y2Px0gD1u3Dhs3boV7777Lt555x0MHjwYaWlpCA0NFWsXiIjaED1cAWDx4sVYvHhxu89lZGS0WZaQkICEhAQjV0VE1H2iX0RARGSNGK5EREbAcCUiMgKGKxGRETBciYiMgOFKRGQEDFciIiNguBIRGQHDlYjICBiuRERGYBaXv5qSIAgA9OvLaAmUSiXq6+tRXV1tNe3dxMBxNBxrHEtNbmhypCM9Llxralo69QcEBIhcCRFZqpqaGnh4dHyPNYnQlQi2Imq1GoWFhXBzc7Oqe01p7rBw69atLt1hgdrHcTQcaxxLQRBQU1ODvn37turW154ed+RqY2ODfv36iV2G0bi7u1vND7KYOI6GY21j2dkRqwa/0CIiMgKGKxGRETBcrYRUKsWKFSvavRkjdR3H0XB6+lj2uC+0iIhMgUeuRERGwHAlIjIChisRkREwXK3M9evX8corryAoKAhOTk4IDg7GihUr0NTUJHZpFmHDhg0YMGAAHB0dMXbsWHz//fdil2RRUlJS8Nhjj8HNzQ0+Pj6Ij4/HpUuXxC5LFAxXK5Ofnw+1Wo1PP/0UFy9exJo1a7Bx40a88847Ypdm9rZv346kpCSsWLECZ8+eRUREBOLi4lBaWip2aRbj6NGjWLRoEU6ePAm5XA6lUonJkyejrq5O7NJMjrMFeoAPP/wQn3zyCa5evSp2KWZt7NixeOyxx7B+/XoALZdKBwQE4PXXX8eyZctErs4ylZWVwcfHB0ePHsX48ePFLsekeOTaA1RVVcHT01PsMsxaU1MTsrOzERsbq11mY2OD2NhYZGVliViZZauqqgKAHvnzx3C1cleuXMHHH3+MX/7yl2KXYtbKy8uhUqng6+vbarmvry+Ki4tFqsqyqdVqLFmyBE888QRCQ0PFLsfkGK4WYtmyZZBIJB3+y8/Pb/WaO3fuYMqUKUhISMDChQtFqpx6qkWLFiEvLw/btm0TuxRR9LiuWJbqrbfewrx58zpcZ+DAgdr/LiwsxMSJEzFu3Dhs2rTJyNVZPi8vL9ja2qKkpKTV8pKSEshkMpGqslyLFy/G3r17cezYMavuQtcRhquF8Pb2hre3d5fWvXPnDiZOnIioqChs3ry5076TBDg4OCAqKgqHDx9GfHw8gJY/aw8fPozFixeLW5wFEQQBr7/+Or7++mtkZGQgKChI7JJEw3C1Mnfu3EFMTAwCAwPxpz/9CWVlZdrneATWsaSkJCQmJmL06NEYM2YM1q5di7q6OsyfP1/s0izGokWLsHXrVuzatQtubm7a89UeHh5wcnISuTrT4lQsK5OamqozDPhRd279+vX48MMPUVxcjMjISPz5z3/G2LFjxS7LYui6u8fmzZs7Pa1lbRiuRERGwJNxRERGwHAlIjIChisRkREwXImIjIDhSkRkBAxXIiIjYLgSERkBw5WIyAgYrkRERsBwJSIyAoYrEZERMFypxysrK4NMJsMHH3ygXZaZmQkHBwccPnxYxMrIkrFxCxGA9PR0xMfHIzMzE0OHDkVkZCRmz56Njz76SOzSyEIxXInuW7RoEQ4dOoTRo0fjwoULOH36NKRSqdhlkYViuBLd19DQgNDQUNy6dQvZ2dkICwsTuySyYDznSnRfQUEBCgsLoVarcf36dbHLIQvHI1ciAE1NTRgzZgwiIyMxdOhQrF27FhcuXICPj4/YpZGFYrgSAVi6dCl27tyJ3NxcuLq6YsKECfDw8MDevXvFLo0sFE8LUI+XkZGBtWvXYsuWLXB3d4eNjQ22bNmC48eP45NPPhG7PLJQPHIlIjICHrkSERkBw5WIyAgYrkRERsBwJSIyAoYrEZERMFyJiIyA4UpEZAQMVyIiI2C4EhEZAcOViMgIGK5EREbAcCUiMoL/B97AfKxLM+5bAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "gelu, relu = GELU(), nn.ReLU()\n",
    "x = torch.linspace(-3, 3, 100) #A\n",
    "y_gelu, y_relu = gelu(x), relu(x)\n",
    "plt.figure(figsize=(8, 3))\n",
    "for i, (y, label) in enumerate(zip([y_gelu, y_relu], [\"GELU\", \"ReLU\"]), 1):\n",
    "    plt.subplot(1, 2, i)\n",
    "    plt.plot(x, y)\n",
    "    plt.title(f\"{label} activation function\")\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(f\"{label}(x)\")\n",
    "    plt.grid(True)\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e965e0a",
   "metadata": {},
   "source": [
    "如图 4.8 所示，ReLU 是一个分段线性函数，对于正数输入，它会直接输出该输入值；对于负数输入，它的输出为零。而 GELU 是一个平滑的、非线性函数，它与 ReLU 相似，但在处理负数输入时，其梯度不为零。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62905c75",
   "metadata": {},
   "source": [
    "**图 4.8 使用 matplotlib 绘制的 GELU 和 ReLU 函数。x 轴代表函数的输入，y 轴代表函数的输出**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df036b2a",
   "metadata": {},
   "source": [
    "![fig4.8](https://github.com/Pr04Ark/llms-from-scratch-cn/blob/trans01/Translated_Book/img/fig-4-8.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee946234",
   "metadata": {},
   "source": [
    "如图 4.8 所示，GELU 的平滑性在训练过程中可以带来更优的优化性能，因为它允许对模型参数进行更精细的调整。相较之下，ReLU 在零点处有一个尖角，这有时会使优化变得更为困难，尤其是在网络极其深入或架构复杂的情况下。此外，与 ReLU 对所有负输入都输出零不同，GELU 允许负值有微小的、非零的输出。这一特性意味着在训练过程中，接收到负输入的神经元仍然可以在一定程度上参与学习过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bac70b6",
   "metadata": {},
   "source": [
    "接下来，我们将使用 GELU 函数来实现一个小型的神经网络模块 `FeedForward`，这个模块将在后续的 LLM 的 transformer 块中被使用："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1c500d9",
   "metadata": {},
   "source": [
    "**代码清单 4.4 前馈神经网络模块**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9fed9f9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeedForward(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.layers = nn.Sequential(\n",
    "        nn.Linear(cfg[\"emb_dim\"], 4 * cfg[\"emb_dim\"]),\n",
    "        GELU(),\n",
    "        nn.Linear(4 * cfg[\"emb_dim\"], cfg[\"emb_dim\"]),\n",
    "        )\n",
    "    def forward(self, x):\n",
    "        return self.layers(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96ac02fb",
   "metadata": {},
   "source": [
    "如上述代码所示，`FeedForward` 模块是一个由两个线性层和一个 GELU 激活函数组成的小型神经网络。在拥有 12400 万参数的 GPT 模型中，它接收的输入为 768 维的 Embedding 批量输入，这是由`GPT_CONFIG_124M` 字典中的`GPT_CONFIG_124M[\"emb_dim\"] = 768`配置所决定的 。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61566693",
   "metadata": {},
   "source": [
    "图 4.9 展示了当我们向这个小型前馈神经网络输入数据时，Embedding 的维度是如何被处理的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f52945e6",
   "metadata": {},
   "source": [
    "**图 4.9 前馈神经网络各层之间连接的直观示意图。值得注意的是，该神经网络可以适应不同的批次大小和输入的 token 数量。然而，每个 token 的 Embedding 大小在初始化权重时已经确定并被固定下来。**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60136fc",
   "metadata": {},
   "source": [
    "![fig4.9](https://github.com/Pr04Ark/llms-from-scratch-cn/blob/trans01/Translated_Book/img/fig-4-9.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ab9262b",
   "metadata": {},
   "source": [
    "如图 4.9 的示例，我们初始化一个 `toke` 的 `Embedding` 大小为 768 的 `FeedForward` 模块，并向其输入一个批次数据，该批次包含 2 个样本，且每个样本含有 3 个 tokens："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "94559d08",
   "metadata": {},
   "outputs": [],
   "source": [
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,  # 词表大小\n",
    "    \"ctx_len\": 1024,      # 上下文长度\n",
    "    \"emb_dim\": 768,       # 嵌入维度\n",
    "    \"n_heads\": 12,        # 注意力头（attention heads）的数量\n",
    "    \"n_layers\": 12,       # 模型层数\n",
    "    \"drop_rate\": 0.1,     # Dropout rate\n",
    "    \"qkv_bias\": False     # Query-Key-Value bias\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a4455269",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 3, 768])\n"
     ]
    }
   ],
   "source": [
    "ffn = FeedForward(GPT_CONFIG_124M)\n",
    "x = torch.rand(2, 3, 768) #A\n",
    "out = ffn(x)\n",
    "print(out.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135cf7ab",
   "metadata": {},
   "source": [
    "正如我们所看到的，输出张量的形状与输入张量的形状相同："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12cbb9af",
   "metadata": {},
   "source": [
    "```\n",
    "torch.Size([2, 3, 768])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb2c57fc",
   "metadata": {},
   "source": [
    "在本节中，我们实现的 `FeedForward` 模块在增强模型从数据中学习和泛化的能力方面发挥了关键作用。虽然该模块的输入和输出维度相同，但它通过第一个线性层将嵌入维度扩展到更高的空间维度，如图 4.10 所示。这个扩展后面跟随的是一个非线性的 GELU 激活函数，然后通过第二个线性变换将其压缩回原始维度。这种设计使模型能够探索更丰富的表示空间。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f25672e",
   "metadata": {},
   "source": [
    "**图 4.10 展示了前馈神经网络中层输出的扩展和压缩过程。首先，输入维度扩展 4 倍，从 768 维扩展到 3072 维。然后，第二层将这 3072 维压缩回 768 维的表示。**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af678ff",
   "metadata": {},
   "source": [
    "![fig4.10](https://github.com/Pr04Ark/llms-from-scratch-cn/blob/trans01/Translated_Book/img/fig-4-10.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "923a7393",
   "metadata": {},
   "source": [
    "此外，输入和输出维度的一致性简化了架构设计，使得我们可以堆叠多个层，如我们稍后将要做的，而无需在它们之间调整维度，从而提高了模型的可扩展性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fa08967",
   "metadata": {},
   "source": [
    "如图 4.11 所示，我们现在已经实现了构建 LLM 的大部分模块。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f53ecf2b",
   "metadata": {},
   "source": [
    "**图 4.11 展示了本章我们将要涵盖的主题，其中黑色的勾选标记表示我们已经讨论过的主题。**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c145bcb",
   "metadata": {},
   "source": [
    "![fig4.11](https://github.com/Pr04Ark/llms-from-scratch-cn/blob/trans01/Translated_Book/img/fig-4-11.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fe24ad6",
   "metadata": {},
   "source": [
    "在下一节中，我们将探讨在神经网络的不同层之间插入的快捷连接（shortcut connections）的概念，这对于提升深度神经网络架构的训练性能至关重要。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}