google-research
550 строк · 45.1 Кб
1{2"nbformat": 4,3"nbformat_minor": 0,4"metadata": {5"colab": {6"provenance": [],7"collapsed_sections": [8"Ltj0f-flNAi9"9]10},11"kernelspec": {12"name": "python3",13"display_name": "Python 3"14},15"language_info": {16"name": "python"17}18},19"cells": [20{21"cell_type": "markdown",22"source": [23"##### Copyright 2023 Google LLC. SPDX-License-Identifier: Apache-2.0"24],25"metadata": {26"id": "Ltj0f-flNAi9"27}28},29{30"cell_type": "markdown",31"source": [32"Copyright 2023 Google LLC. SPDX-License-Identifier: Apache-2.0\n",33"\n",34"Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",35"\n",36"https://www.apache.org/licenses/LICENSE-2.0\n",37"\n",38"Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."39],40"metadata": {41"id": "I8NlVpAzNB2u"42}43},44{45"cell_type": "markdown",46"source": [47"## **LLMs as General Pattern Machines:** ARC Benchmark\n",48"\n",49"We observe that pretrained large language models (LLMs) are capable of autoregressively completing complex token sequences -- from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstract Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern completion proficiency can be partially retained even when the sequences are expressed using tokens randomly sampled from the vocabulary. These results suggest that without any additional training, LLMs can serve as general sequence modelers, driven by in-context learning. In this work, we investigate how these zero-shot capabilities may be applied to problems in robotics -- from extrapolating sequences of numbers that represent states over time to complete simple motions, to least-to-most prompting of reward-conditioned trajectories that can discover and represent closed-loop policies (e.g., a stabilizing controller for CartPole). While difficult to deploy today for real systems due to latency, context size limitations, and compute costs, the approach of using LLMs to drive low-level control may provide an exciting glimpse into how the patterns among words could be transferred to actions.\n",50"\n",51"This colab runs GPT-3 on the ARC benchmark with consistent tokenization (described more in Sec. 4 of the main paper).\n",52"\n",53"### **Quick Start:**\n",54"\n",55"**Step 1.** Register for an [OpenAI API key](https://openai.com/blog/openai-api/) to use GPT-3 (there's a free trial) and enter it below\n",56"\n",57"**Step 2.** Menu > Runtime > Run all"58],59"metadata": {60"id": "zqTADtDB6zyA"61}62},63{64"cell_type": "code",65"source": [66"openai_api_key = \"your-api-key-here\""67],68"metadata": {69"id": "wwJDOJSz71lk"70},71"execution_count": null,72"outputs": []73},74{75"cell_type": "markdown",76"source": [77"## **Setup**\n",78"\n",79"This does a few things:\n",80"* Installs Python packages and sets OpenAI API key.\n",81"* Downloads the Abstract Reasoning Corpus (ARC) benchmark.\n",82"\n",83"**Note:** only needs a CPU (public) runtime."84],85"metadata": {86"id": "kbWMlIj7XxX8"87}88},89{90"cell_type": "code",91"source": [92"!pip install openai transformers\n",93"\n",94"import json\n",95"import os\n",96"import time\n",97"\n",98"import matplotlib.pyplot as plt\n",99"import numpy as np\n",100"import openai\n",101"import pickle\n",102"from transformers import GPT2Tokenizer\n",103"# import tiktoken # Faster than GPT2Tokenizer.\n",104"\n",105"openai.api_key = openai_api_key\n",106"\n",107"if not os.path.exists(\"ARC\"):\n",108" !git clone https://github.com/fchollet/ARC"109],110"metadata": {111"id": "uI4hX8y5XzeH"112},113"execution_count": null,114"outputs": []115},116{117"cell_type": "markdown",118"source": [119"## **API:** Large Language Models\n",120"\n",121"Define helper functions to call large language models and the tokenizer.\n",122"\n",123"**Note:** this can get expensive."124],125"metadata": {126"id": "2AoMDZ-GZxRP"127}128},129{130"cell_type": "code",131"execution_count": null,132"metadata": {133"id": "-waqt2fUb9ex",134"colab": {135"base_uri": "https://localhost:8080/"136},137"outputId": "f2d6d52e-5502-45e7-91cb-0855fde30c60"138},139"outputs": [140{141"output_type": "execute_result",142"data": {143"text/plain": [144"[\"\\n\\nHello World! It's great to be here.\"]"145]146},147"metadata": {},148"execution_count": 23149}150],151"source": [152"model = \"text-davinci-003\"\n",153"token_limit = 4096\n",154"\n",155"def LLM(prompt, stop=None, max_tokens=256, temperature=0):\n",156" responses = openai.Completion.create(engine=model, prompt=prompt, max_tokens=max_tokens, temperature=temperature, stop=stop)\n",157" text = [response['text'] for response in responses['choices']]\n",158" return text\n",159"\n",160"tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",161"\n",162"LLM(\"hello world!\")"163]164},165{166"cell_type": "markdown",167"source": [168"## **Alphabet:** Token Set\n",169"\n",170"Build a fixed token set by random sampling from the LLM's token vocabulary."171],172"metadata": {173"id": "vMPLptkxatkC"174}175},176{177"cell_type": "code",178"source": [179"item_delim = tokenizer.encode(\",\")\n",180"row_delim = tokenizer.encode(\"\\n\")\n",181"sample_delim = tokenizer.encode(\"---\\n\")\n",182"\n",183"# Handpicked: comma-separated number matrices.\n",184"alphabet = [tokenizer.encode(\" \" + str(a))[0] for a in range(10)]\n",185"value_to_token = lambda x: {i:a for i, a in enumerate(alphabet)}[x]\n",186"print(\"Token Set:\", {i:value_to_token(i) for i in np.arange(10)})\n",187"\n",188"# Random sampled tokens.\n",189"# for seed_offset in range(20):\n",190"# seed_offset = 0\n",191"# np.random.seed(42 + seed_offset)\n",192"# alphabet = [int(i) for i in np.random.randint(tokenizer.vocab_size, size=10)]\n",193"# value_to_token = lambda x: {i:a for i, a in enumerate(alphabet)}[x]"194],195"metadata": {196"id": "tRIQ3AZMUmpK",197"colab": {198"base_uri": "https://localhost:8080/"199},200"outputId": "9e07e590-0986-4320-f164-0268609a9cf7"201},202"execution_count": null,203"outputs": [204{205"output_type": "stream",206"name": "stdout",207"text": [208"Token Set: {0: 657, 1: 352, 2: 362, 3: 513, 4: 604, 5: 642, 6: 718, 7: 767, 8: 807, 9: 860}\n"209]210}211]212},213{214"cell_type": "markdown",215"source": [216"## **Load:** ARC Benchmark\n",217"\n",218"Load tasks from the ARC benchmark."219],220"metadata": {221"id": "SD_-rknea0WU"222}223},224{225"cell_type": "code",226"source": [227"def state_to_tokens(state, value_to_token_fn):\n",228" tokens = []\n",229" for row in state:\n",230" for i, value in enumerate(row):\n",231" tokens +=[value_to_token_fn(value)]\n",232" if i < len(row) - 1:\n",233" tokens += item_delim\n",234" tokens += row_delim\n",235" return tokens\n",236"\n",237"\n",238"def task_json_to_tokens(task_json, value_to_token_fn):\n",239"\n",240" # Training examples.\n",241" train_samples = []\n",242" for sample in task_json[\"train\"]:\n",243" tokens = []\n",244" tokens += tokenizer.encode(\"input:\\n\")\n",245" tokens += state_to_tokens(sample[\"input\"], value_to_token_fn)\n",246" tokens += tokenizer.encode(\"output:\\n\")\n",247" tokens += state_to_tokens(sample[\"output\"], value_to_token_fn)\n",248" tokens += sample_delim\n",249" train_samples.append(tokens)\n",250"\n",251" # Testing examples.\n",252" test_inputs = []\n",253" test_outputs = []\n",254" for sample in task_json[\"test\"]:\n",255" inputs, outputs = [], []\n",256" inputs += tokenizer.encode(\"input:\\n\")\n",257" inputs += state_to_tokens(sample[\"input\"], value_to_token_fn)\n",258" inputs += tokenizer.encode(\"output:\\n\")\n",259" test_inputs.append(inputs)\n",260" outputs += state_to_tokens(sample[\"output\"], value_to_token_fn)\n",261" test_outputs.append(outputs)\n",262" return train_samples, test_inputs, test_outputs"263],264"metadata": {265"id": "5rvACw0XFZWY"266},267"execution_count": null,268"outputs": []269},270{271"cell_type": "code",272"source": [273"tasks_jsons = []\n",274"tasks_names = []\n",275"tasks_len = []\n",276"task_dir = \"ARC/data/training\"\n",277"for task_file in sorted(os.listdir(task_dir)):\n",278" with open(os.path.join(task_dir, task_file)) as fid:\n",279" task_json = json.load(fid)\n",280" tasks_jsons.append(task_json)\n",281" tasks_names.append(task_file)\n",282" tokens, _, _ = task_json_to_tokens(task_json, value_to_token)\n",283" tasks_len.append(np.sum([len(sample) for sample in tokens]))\n",284"\n",285"task_dir = \"ARC/data/evaluation\"\n",286"for task_file in sorted(os.listdir(task_dir)):\n",287" with open(os.path.join(task_dir, task_file)) as fid:\n",288" task_json = json.load(fid)\n",289" tasks_jsons.append(task_json)\n",290" tasks_names.append(task_file)\n",291" tokens, _, _ = task_json_to_tokens(task_json, value_to_token)\n",292" tasks_len.append(np.sum([len(sample) for sample in tokens]))\n",293"\n",294"sorted_task_ids = np.argsort(tasks_len)\n",295"\n",296"print(\"Total number of tasks:\", len(sorted_task_ids))"297],298"metadata": {299"id": "zZY7OSoHbIRk",300"colab": {301"base_uri": "https://localhost:8080/"302},303"outputId": "2fda0d4e-1014-4888-dd2a-d76fb4942ff2"304},305"execution_count": null,306"outputs": [307{308"output_type": "stream",309"name": "stdout",310"text": [311"Total number of tasks: 800\n"312]313}314]315},316{317"cell_type": "markdown",318"source": [319"## **Example:** ARC Problem\n",320"\n",321"Show the LLM prompt for an ARC problem and visualize the grids used as inputs and outputs."322],323"metadata": {324"id": "2wBok5T59kDT"325}326},327{328"cell_type": "code",329"source": [330"colors = [(0, 0, 0),\n",331" (0, 116, 217),\n",332" (255, 65, 54),\n",333" (46, 204, 6),\n",334" (255, 220, 0),\n",335" (170, 170, 170),\n",336" (240, 18, 190),\n",337" (255, 133, 27),\n",338" (127, 219, 255),\n",339" (135, 12, 37)]\n",340"\n",341"def grid_to_img(grid):\n",342" grid = np.int32(grid)\n",343" scale = 10\n",344" img = np.zeros((grid.shape[0] * scale + 1, grid.shape[1] * scale + 1, 3), dtype=np.uint8)\n",345" for r in range(grid.shape[0]):\n",346" for c in range(grid.shape[1]):\n",347" img[r*scale+1:(r+1)*scale, c*scale+1:(c+1)*scale, :] = colors[grid[r, c]]\n",348" new_img = img.copy()\n",349" new_img[0::10, :, :] = np.uint8(np.round((0.7 * np.float32(img[0::10, :, :]) + 0.3 * 255)))\n",350" new_img[:, 0::10, :] = np.uint8(np.round((0.7 * np.float32(img[:, 0::10, :]) + 0.3 * 255)))\n",351" return new_img"352],353"metadata": {354"id": "qCl4heCw88MG"355},356"execution_count": null,357"outputs": []358},359{360"cell_type": "code",361"source": [362"example_json = tasks_jsons[sorted_task_ids[0]]\n",363"\n",364"context = []\n",365"train_xy, test_x, test_y = task_json_to_tokens(example_json, value_to_token)\n",366"for sample in train_xy:\n",367" context += sample\n",368"context += test_x[0]\n",369"\n",370"print(\"PROMPT:\")\n",371"print(tokenizer.decode(context, skip_special_tokens=True))\n",372"print(\"SOLUTION:\")\n",373"print(tokenizer.decode(test_y[0], skip_special_tokens=True))\n",374"\n",375"# Show problem.\n",376"print(\"TRAIN:\")\n",377"for i, ex in enumerate(example_json[\"train\"]):\n",378" in_img = grid_to_img(ex[\"input\"])\n",379" out_img = grid_to_img(ex[\"output\"])\n",380" plt.subplot(1, 2, 1); plt.imshow(grid_to_img(ex[\"input\"]))\n",381" plt.subplot(1, 2, 2); plt.imshow(grid_to_img(ex[\"output\"]))\n",382" plt.show()\n",383"print(\"TEST:\")\n",384"for i, ex in enumerate(example_json[\"test\"]):\n",385" in_img = grid_to_img(ex[\"input\"])\n",386" out_img = grid_to_img(ex[\"output\"])\n",387" plt.subplot(1, 2, 1); plt.imshow(grid_to_img(ex[\"input\"]))\n",388" plt.subplot(1, 2, 2); plt.imshow(grid_to_img(ex[\"output\"]))\n",389" plt.show()"390],391"metadata": {392"id": "orLb7781byY3",393"colab": {394"base_uri": "https://localhost:8080/",395"height": 1000396},397"outputId": "a5e6948a-ccf2-4dc6-bf03-d7902c07e85a"398},399"execution_count": null,400"outputs": [401{402"output_type": "stream",403"name": "stdout",404"text": [405"PROMPT:\n",406"input:\n",407" 3, 3, 8\n",408" 3, 7, 0\n",409" 5, 0, 0\n",410"output:\n",411" 0, 0, 5\n",412" 0, 7, 3\n",413" 8, 3, 3\n",414"---\n",415"input:\n",416" 5, 5, 2\n",417" 1, 0, 0\n",418" 0, 0, 0\n",419"output:\n",420" 0, 0, 0\n",421" 0, 0, 1\n",422" 2, 5, 5\n",423"---\n",424"input:\n",425" 6, 3, 5\n",426" 6, 8, 0\n",427" 4, 0, 0\n",428"output:\n",429"\n",430"SOLUTION:\n",431" 0, 0, 4\n",432" 0, 8, 6\n",433" 5, 3, 6\n",434"\n",435"TRAIN:\n"436]437},438{439"output_type": "display_data",440"data": {441"text/plain": [442"<Figure size 640x480 with 2 Axes>"443],444"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAEPCAYAAABycN8YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZkklEQVR4nO3df0xV9/3H8RcoXK3CpfiDKxEcrU7bWuk3TOmNnbGTiS4abTGZ3RJ1a2pkYGLN0knTVu2W3EaT1bajNNkyqcmsW5ehkaZqi4pphy4yCbVOUg2bNOVia8K9SssV5fP9w/RuV4HrhQufy+X5SE7iPefcc998Gl99eTkXEowxRgAAABYl2h4AAACAQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACsGz1YFy4vL9fOnTvl9XqVm5urN954Q/PmzQv7vO7ubn3xxRdKSUlRQkLCYI0HoA/GGF29elWZmZlKTBy6f7f0NzcksgOwbcC5YQbBvn37THJysvnjH/9oPv30U/PMM8+YtLQ009bWFva5LS0tRhIbG1sMbC0tLYMRET0aSG4YQ3awscXK1t/cSDAm+r9cLz8/X3PnztXvfvc7Sbf+5ZKVlaWNGzdqy5YtfT7X5/MpLS1NixYt0ujRg/YGDoA+3LhxQzU1NWpvb5fT6RyS1xxIbkhkB2DbQHMj6n9rr1+/rvr6epWVlQX3JSYmqqCgQHV1dXecHwgEFAgEgo+vXr16a7DRo5WUlBTt8QBEYKi+9RFpbkhkBxCr+psbUf/m8FdffaWbN28qIyMjZH9GRoa8Xu8d53s8HjmdzuCWlZUV7ZEAxLhIc0MiO4B4Y/1TNmVlZfL5fMGtpaXF9kgAhgGyA4gvUf+WzcSJEzVq1Ci1tbWF7G9ra5PL5brjfIfDIYfDEe0xAAwjkeaGRHYA8SbqhSQ5OVl5eXmqqanRypUrJd26Oa2mpkalpaUDvv6lbUcGfI14kL1tca/HWKNb+lqjvFcPDuEksav+2eW2R5A0+LkhSdXV1VG5znC3bNmyXo+xRrf0tUZr1qwZwkli1549e6J+zUG5FX3z5s1au3atvve972nevHnatWuXOjo69LOf/WwwXg5AHCA3gJFtUArJj3/8Y3355Zd66aWX5PV69cgjj+jQoUN33LAGAN8iN4CRbdA+rF9aWhq1t1oBjAzkBjByWf+UDQAAAIUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgXdQLybZt25SQkBCyzZo1K9ovAyCOkBsARg/GRR966CF9+OGH/32R0YPyMgDiCLkBjGyD8jd+9OjRcrlcg3FpAHGK3ABGtkG5h+Szzz5TZmam7rvvPv30pz/VpUuXBuNlAMQRcgMY2aL+Dkl+fr4qKys1c+ZMtba2avv27fr+97+vs2fPKiUl5Y7zA4GAAoFA8LHf74/2SABiXKS5IZEdQLyJeiFZunRp8M9z5sxRfn6+pk2bpr/85S96+umn7zjf4/Fo+/bt0R4DwDASaW5IZAcQbwb9Y79paWn67ne/qwsXLvR4vKysTD6fL7i1tLQM9kgAYly43JDIDiDeDHohuXbtmi5evKgpU6b0eNzhcCg1NTVkAzCyhcsNiewA4k3UC8kvf/lL1dbW6t///rf+/ve/64knntCoUaP01FNPRfulAMQJcgNA1O8h+fzzz/XUU0/pypUrmjRpkh577DGdPHlSkyZNivZLAYgT5AaAqBeSffv2RfuSAOIcuQGA32UDAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrEowxJpInnDhxQjt37lR9fb1aW1tVVVWllStXBo8bY7R161b9/ve/V3t7u+bPn6+KigrNmDHjrq7v9/vldDpVWFiopKSkiL4YANHR1dWlw4cPy+fzKTU1dcDXG+zckMgOwLaB5kbE75B0dHQoNzdX5eXlPR7fsWOHXn/9db311ls6deqUxo0bp8LCQnV2dkY8HID4QG4ACGd0pE9YunSpli5d2uMxY4x27dqlF154QStWrJAk7dmzRxkZGdq/f79Wr149sGkBDEvkBoBwonoPSXNzs7xerwoKCoL7nE6n8vPzVVdX1+NzAoGA/H5/yAZg5OhPbkhkBxBvolpIvF6vJCkjIyNkf0ZGRvDY7Twej5xOZ3DLysqK5kgAYlx/ckMiO4B4Y/1TNmVlZfL5fMGtpaXF9kgAhgGyA4gvUS0kLpdLktTW1hayv62tLXjsdg6HQ6mpqSEbgJGjP7khkR1AvIn4pta+5OTkyOVyqaamRo888oikWx/FO3XqlIqLi6PyGpe2HYnKdYa77G2Lez120H1mCCeJXcvr/q/XY9XV1UM4SexatmyZ7RGGJDck/pt/q6//5mTHLX1lB/8PuqWv/wf1V8SF5Nq1a7pw4ULwcXNzsxoaGpSenq7s7Gxt2rRJv/nNbzRjxgzl5OToxRdfVGZmZsjPHAAwspAbAMKJuJCcPn1ajz/+ePDx5s2bJUlr165VZWWlnnvuOXV0dGj9+vVqb2/XY489pkOHDmnMmDHRmxrAsEJuAAgn4kKycOFC9fXDXRMSEvTyyy/r5ZdfHtBgAOIHuQEgHOufsgEAAKCQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArIu4kJw4cULLly9XZmamEhIStH///pDj69atU0JCQsi2ZMmSaM0LYBgiNwCEE3Eh6ejoUG5ursrLy3s9Z8mSJWptbQ1u77zzzoCGBDC8kRsAwhkd6ROWLl2qpUuX9nmOw+GQy+Xq91AA4gu5ASCcQbmH5Pjx45o8ebJmzpyp4uJiXblyZTBeBkAcITeAkS3id0jCWbJkiZ588knl5OTo4sWLev7557V06VLV1dVp1KhRd5wfCAQUCASCj/1+f7RHAhDjIs0NiewA4k3UC8nq1auDf3744Yc1Z84c3X///Tp+/LgWLVp0x/kej0fbt2+P9hgAhpFIc0MiO4B4M+gf+73vvvs0ceJEXbhwocfjZWVl8vl8wa2lpWWwRwIQ48LlhkR2APEm6u+Q3O7zzz/XlStXNGXKlB6POxwOORyOwR4DwDASLjcksgOINxEXkmvXroX8q6W5uVkNDQ1KT09Xenq6tm/frqKiIrlcLl28eFHPPfecpk+frsLCwqgODmD4IDcAhBNxITl9+rQef/zx4OPNmzdLktauXauKigo1Njbq7bffVnt7uzIzM7V48WL9+te/5l8ywAhGbgAIJ+JCsnDhQhljej1++PDhAQ0EIP6QGwDC4XfZAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6yIqJB6PR3PnzlVKSoomT56slStXqqmpKeSczs5OlZSUaMKECRo/fryKiorU1tYW1aEBDC9kB4BwEowx5m5PXrJkiVavXq25c+fqxo0bev7553X27FmdO3dO48aNkyQVFxfrvffeU2VlpZxOp0pLS5WYmKiPP/74rl7D7/fL6XSqsLBQSUlJ/fuqAAxIV1eXDh8+LJ/Pp9TU1AFfj+wA4t9AcyOiQnK7L7/8UpMnT1Ztba0WLFggn8+nSZMmae/evVq1apUk6fz583rggQdUV1enRx99NOw1CRXAvmgXktuRHUD8GWhuDOgeEp/PJ0lKT0+XJNXX16urq0sFBQXBc2bNmqXs7GzV1dUN5KUAxBGyA8DtRvf3id3d3dq0aZPmz5+v2bNnS5K8Xq+Sk5OVlpYWcm5GRoa8Xm+P1wkEAgoEAsHHfr+/vyMBGAbIDgA96fc7JCUlJTp79qz27ds3oAE8Ho+cTmdwy8rKGtD1AMQ2sgNAT/pVSEpLS1VdXa1jx45p6tSpwf0ul0vXr19Xe3t7yPltbW1yuVw9XqusrEw+ny+4tbS09GckAMMA2QGgNxEVEmOMSktLVVVVpaNHjyonJyfkeF5enpKSklRTUxPc19TUpEuXLsntdvd4TYfDodTU1JANQHwhOwCEE9E9JCUlJdq7d68OHDiglJSU4Pd2nU6nxo4dK6fTqaefflqbN29Wenq6UlNTtXHjRrnd7ru6S/5urFmzJirXGe727NnT67Hq6uohnCR2LVu2rNdjrNEtfa1RNMVCduS9ejAq1xnu6p9d3uuxS9uODOEksSt72+Jej7FGt/S1Rv0VUSGpqKiQJC1cuDBk/+7du7Vu3TpJ0quvvqrExEQVFRUpEAiosLBQb775ZlSGBTA8kR0AwomokNzNjywZM2aMysvLVV5e3u+hAMQXsgNAOPwuGwAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWBdRIfF4PJo7d65SUlI0efJkrVy5Uk1NTSHnLFy4UAkJCSHbhg0bojo0gOGF7AAQTkSFpLa2ViUlJTp58qQ++OADdXV1afHixero6Ag575lnnlFra2tw27FjR1SHBjC8kB0AwhkdycmHDh0KeVxZWanJkyervr5eCxYsCO6/55575HK5ojMhgGGP7AAQzoDuIfH5fJKk9PT0kP1/+tOfNHHiRM2ePVtlZWX6+uuve71GIBCQ3+8P2QDEN7IDwO0ieofkf3V3d2vTpk2aP3++Zs+eHdz/k5/8RNOmTVNmZqYaGxv1q1/9Sk1NTfrb3/7W43U8Ho+2b9/e3zEADDNkB4Ce9LuQlJSU6OzZs/roo49C9q9fvz7454cfflhTpkzRokWLdPHiRd1///13XKesrEybN28OPvb7/crKyurvWABiHNkBoCf9KiSlpaWqrq7WiRMnNHXq1D7Pzc/PlyRduHChx1BxOBxyOBz9GQPAMEN2AOhNRIXEGKONGzeqqqpKx48fV05OTtjnNDQ0SJKmTJnSrwEBDH9kB4BwIiokJSUl2rt3rw4cOKCUlBR5vV5JktPp1NixY3Xx4kXt3btXP/rRjzRhwgQ1Njbq2Wef1YIFCzRnzpxB+QIAxD6yA0A4ERWSiooKSbd+gNH/2r17t9atW6fk5GR9+OGH2rVrlzo6OpSVlaWioiK98MILURsYwPBDdgAIJ+Jv2fQlKytLtbW1AxoIQPwhOwCEw++yAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1kVUSCoqKjRnzhylpqYqNTVVbrdb77//fvB4Z2enSkpKNGHCBI0fP15FRUVqa2uL+tAAhheyA0A4CcYYc7cnHzx4UKNGjdKMGTNkjNHbb7+tnTt36syZM3rooYdUXFys9957T5WVlXI6nSotLVViYqI+/vjjux7I7/fL6XSqsLBQSUlJ/fqiAAxMV1eXDh8+LJ/Pp9TU1AFfj+wA4t9AcyOiQtKT9PR07dy5U6tWrdKkSZO0d+9erVq1SpJ0/vx5PfDAA6qrq9Ojjz56V9cjVAD7ol1IekJ2APFloLnR73tIbt68qX379qmjo0Nut1v19fXq6upSQUFB8JxZs2YpOztbdXV1/X0ZAHGG7ADQk9GRPuGTTz6R2+1WZ2enxo8fr6qqKj344INqaGhQcnKy0tLSQs7PyMiQ1+vt9XqBQECBQCD42O/3RzoSgGGA7ADQl4jfIZk5c6YaGhp06tQpFRcXa+3atTp37ly/B/B4PHI6ncEtKyur39cCELvIDgB9ibiQJCcna/r06crLy5PH41Fubq5ee+01uVwuXb9+Xe3t7SHnt7W1yeVy9Xq9srIy+Xy+4NbS0hLxFwEg9pEdAPoS8bdsbtfd3a1AIKC8vDwlJSWppqZGRUVFkqSmpiZdunRJbre71+c7HA45HI7g42/vsb1x48ZARwPQT9/+/RvgPe99IjuA+DLg3DAR2LJli6mtrTXNzc2msbHRbNmyxSQkJJgjR44YY4zZsGGDyc7ONkePHjWnT582brfbuN3uSF7CtLS0GElsbGwxsLW0tET095fsYGNj629uRPQOyeXLl7VmzRq1trbK6XRqzpw5Onz4sH74wx9Kkl599VUlJiaqqKhIgUBAhYWFevPNNyN5CWVmZqqlpUUpKSlKSEiQ3+9XVlaWWlpaBu3jh8Mda9Q31ie829fIGKOrV68qMzMzKtcnO2IP6xMeaxTe/65RSkrKgHJjwD+HZLB9+7MFBvPnIQx3rFHfWJ/w4nGN4vFriibWJzzWKLxorhG/ywYAAFhHIQEAANbFfCFxOBzaunVryN30CMUa9Y31CS8e1ygev6ZoYn3CY43Ci+Yaxfw9JAAAIP7F/DskAAAg/lFIAACAdRQSAABgHYUEAABYF9OFpLy8XN/5znc0ZswY5efn6x//+Iftkaw5ceKEli9frszMTCUkJGj//v0hx40xeumllzRlyhSNHTtWBQUF+uyzz+wMa4nH49HcuXOVkpKiyZMna+XKlWpqago5p7OzUyUlJZowYYLGjx+voqIitbW1WZp4aFVUVGjOnDlKTU1Vamqq3G633n///eDxeFobsuO/yI6+kRvhDVV2xGwh+fOf/6zNmzdr69at+uc//6nc3FwVFhbq8uXLtkezoqOjQ7m5uSovL+/x+I4dO/T666/rrbfe0qlTpzRu3DgVFhaqs7NziCe1p7a2ViUlJTp58qQ++OADdXV1afHixero6Aie8+yzz+rgwYN69913VVtbqy+++EJPPvmkxamHztSpU/XKK6+ovr5ep0+f1g9+8AOtWLFCn376qaT4WRuyIxTZ0TdyI7why45+/QacITBv3jxTUlISfHzz5k2TmZlpPB6PxaligyRTVVUVfNzd3W1cLpfZuXNncF97e7txOBzmnXfesTBhbLh8+bKRZGpra40xt9YkKSnJvPvuu8Fz/vWvfxlJpq6uztaYVt17773mD3/4Q1ytDdnRO7IjPHLj7gxGdsTkOyTXr19XfX29CgoKgvsSExNVUFCguro6i5PFpubmZnm93pD1cjqdys/PH9Hr5fP5JEnp6emSpPr6enV1dYWs06xZs5SdnT3i1unmzZvat2+fOjo65Ha742ZtyI7IkB13Ijf6NpjZEdFv+x0qX331lW7evKmMjIyQ/RkZGTp//rylqWKX1+uVpB7X69tjI013d7c2bdqk+fPna/bs2ZJurVNycrLS0tJCzh1J6/TJJ5/I7Xars7NT48ePV1VVlR588EE1NDTExdqQHZEhO0KRG70biuyIyUICDFRJSYnOnj2rjz76yPYoMWXmzJlqaGiQz+fTX//6V61du1a1tbW2xwJiArnRu6HIjpj8ls3EiRM1atSoO+7SbWtrk8vlsjRV7Pp2TVivW0pLS1VdXa1jx45p6tSpwf0ul0vXr19Xe3t7yPkjaZ2Sk5M1ffp05eXlyePxKDc3V6+99lrcrA3ZERmy47/Ijb4NRXbEZCFJTk5WXl6eampqgvu6u7tVU1Mjt9ttcbLYlJOTI5fLFbJefr9fp06dGlHrZYxRaWmpqqqqdPToUeXk5IQcz8vLU1JSUsg6NTU16dKlSyNqnf5Xd3e3AoFA3KwN2REZsoPc6K9ByY7o3ncbPfv27TMOh8NUVlaac+fOmfXr15u0tDTj9Xptj2bF1atXzZkzZ8yZM2eMJPPb3/7WnDlzxvznP/8xxhjzyiuvmLS0NHPgwAHT2NhoVqxYYXJycsw333xjefKhU1xcbJxOpzl+/LhpbW0Nbl9//XXwnA0bNpjs7Gxz9OhRc/r0aeN2u43b7bY49dDZsmWLqa2tNc3NzaaxsdFs2bLFJCQkmCNHjhhj4mdtyI5QZEffyI3whio7YraQGGPMG2+8YbKzs01ycrKZN2+eOXnypO2RrDl27JiRdMe2du1aY8ytj++9+OKLJiMjwzgcDrNo0SLT1NRkd+gh1tP6SDK7d+8OnvPNN9+YX/ziF+bee+8199xzj3niiSdMa2urvaGH0M9//nMzbdo0k5ycbCZNmmQWLVoUDBRj4mttyI7/Ijv6Rm6EN1TZkWCMMf18xwYAACAqYvIeEgAAMLJQSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFj3/zMn9TxTSfy6AAAAAElFTkSuQmCC\n"445},446"metadata": {}447},448{449"output_type": "display_data",450"data": {451"text/plain": [452"<Figure size 640x480 with 2 Axes>"453],454"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAEPCAYAAABycN8YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZdUlEQVR4nO3dcUxV9/3/8RcoXK3CpahwJYKj1WlbC02Y0hs7YycTWWy0xWR2S8StqZGBieW7dNK0dXZLbqP5rbYdpcmWSU1m2boMTWmqbVEx7dBFJqHWSaphk6aArQkXpeWK8vn9YXq7q8D1woXP5fJ8JCfxnnPuuW8+ja++vJwLMcYYIwAAAItibQ8AAABAIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWTR6tC1dUVGjXrl3q6OhQdna2Xn31VS1ZsiTo8/r7+/X5558rISFBMTExozUegCEYY3T58mWlpaUpNnbs/t0y3NyQyA7AthHnhhkF1dXVJj4+3vzpT38yn3zyiXnyySdNUlKS6ezsDPrctrY2I4mNjS0Ctra2ttGIiAGNJDeMITvY2CJlG25uxBgT/l+ul5ubq8WLF+v3v/+9pBv/cklPT9eWLVu0bdu2IZ/r9XqVlJSkFStWaPLkUXsDB8AQrl27prq6OnV1dcnpdI7Ja44kNySyA7BtpLkR9r+1V69eVWNjo8rLy/37YmNjlZeXp4aGhlvO9/l88vl8/seXL1++MdjkyYqLiwv3eABCMFbf+gg1NySyA4hUw82NsH9z+Msvv9T169eVmpoasD81NVUdHR23nO/xeOR0Ov1benp6uEcCEOFCzQ2J7ACijfVP2ZSXl8vr9fq3trY22yMBGAfIDiC6hP1bNjNnztSkSZPU2dkZsL+zs1Mul+uW8x0OhxwOR7jHADCOhJobEtkBRJuwF5L4+Hjl5OSorq5Oa9eulXTj5rS6ujqVlpaO+PobNmwY8TWiwd69ewc9xhrdMNQavd1zcQwniVyPTEuxPYKk0c8NSaqtrQ3Ldca71atXD3qMNbqBNQpuqDUarlG5Fb2srExFRUX63ve+pyVLlmj37t3q6enRz372s9F4OQBRgNwAJrZRKSQ//vGP9cUXX+j5559XR0eHHnjgAR08ePCWG9YA4BvkBjCxjdqH9UtLS8P2ViuAiYHcACYu65+yAQAAoJAAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACsC3sh+fWvf62YmJiAbeHCheF+GQBRhNwAMHk0Lnrffffpgw8++PZFJo/KywCIIuQGMLGNyt/4yZMny+VyjcalAUQpcgOY2EblHpJPP/1UaWlpuuuuu/TTn/5UFy5cGI2XARBFyA1gYgv7OyS5ubmqqqrSggUL1N7erh07duj73/++Tp8+rYSEhFvO9/l88vl8/sfd3d3hHglAhAs1NySyA4g2YS8kBQUF/j9nZWUpNzdXc+fO1V//+lc98cQTt5zv8Xi0Y8eOcI8BYBwJNTcksgOINqP+sd+kpCR997vf1blz5wY8Xl5eLq/X69/a2tpGeyQAES5YbkhkBxBtRr2QXLlyRefPn9fs2bMHPO5wOJSYmBiwAZjYguWGRHYA0SbsheSXv/yl6uvr9Z///Ef/+Mc/9Oijj2rSpEl6/PHHw/1SAKIEuQEg7PeQfPbZZ3r88cd16dIlzZo1Sw899JCOHz+uWbNmhfulAEQJcgNA2AtJdXV1uC8JIMqRGwD4XTYAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALAuxhhjQnnCsWPHtGvXLjU2Nqq9vV01NTVau3at/7gxRtu3b9cf/vAHdXV1aenSpaqsrNT8+fNv6/rd3d1yOp3Kz89XXFxcSF8MgPDo6+vToUOH5PV6lZiYOOLrjXZuSGQHYNtIcyPkd0h6enqUnZ2tioqKAY/v3LlTr7zyil5//XWdOHFC06ZNU35+vnp7e0MeDkB0IDcABDM51CcUFBSooKBgwGPGGO3evVvPPvus1qxZI0nau3evUlNTtX//fq1fv35k0wIYl8gNAMGE9R6S1tZWdXR0KC8vz7/P6XQqNzdXDQ0NAz7H5/Opu7s7YAMwcQwnNySyA4g2YS0kHR0dkqTU1NSA/ampqf5jN/N4PHI6nf4tPT09nCMBiHDDyQ2J7ACijfVP2ZSXl8vr9fq3trY22yMBGAfIDiC6hLWQuFwuSVJnZ2fA/s7OTv+xmzkcDiUmJgZsACaO4eSGRHYA0Sbkm1qHkpmZKZfLpbq6Oj3wwAOSbnwU78SJEyouLg7La9Qu/H9huc54t/rs/w16rLa2dgwniVyrV68e9BhrdMNQazRWxiI3JP6bf4O/F8ENuUb8P0jS0P8PGq6QC8mVK1d07tw5/+PW1lY1NTUpOTlZGRkZ2rp1q377299q/vz5yszM1HPPPae0tLSAnzkAYGIhNwAEE3IhOXnypB5++GH/47KyMklSUVGRqqqq9PTTT6unp0ebNm1SV1eXHnroIR08eFBTpkwJ39QAxhVyA0AwIReS5cuXa6gf7hoTE6MXXnhBL7zwwogGAxA9yA0AwVj/lA0AAACFBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYF3IheTYsWN65JFHlJaWppiYGO3fvz/g+MaNGxUTExOwrVq1KlzzAhiHyA0AwYRcSHp6epSdna2KiopBz1m1apXa29v925tvvjmiIQGMb+QGgGAmh/qEgoICFRQUDHmOw+GQy+Ua9lAAogu5ASCYUbmH5OjRo0pJSdGCBQtUXFysS5cujcbLAIgi5AYwsYX8Dkkwq1at0mOPPabMzEydP39ezzzzjAoKCtTQ0KBJkybdcr7P55PP5/M/7u7uDvdIACJcqLkhkR1AtAl7IVm/fr3/z/fff7+ysrJ099136+jRo1qxYsUt53s8Hu3YsSPcYwAYR0LNDYnsAKLNqH/s96677tLMmTN17ty5AY+Xl5fL6/X6t7a2ttEeCUCEC5YbEtkBRJuwv0Nys88++0yXLl3S7NmzBzzucDjkcDhGewwA40iw3JDIDiDahFxIrly5EvCvltbWVjU1NSk5OVnJycnasWOHCgsL5XK5dP78eT399NOaN2+e8vPzwzo4gPGD3AAQTMiF5OTJk3r44Yf9j8vKyiRJRUVFqqysVHNzs9544w11dXUpLS1NK1eu1G9+8xv+JQNMYOQGgGBCLiTLly+XMWbQ44cOHRrRQACiD7kBIBh+lw0AALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALAupELi8Xi0ePFiJSQkKCUlRWvXrlVLS0vAOb29vSopKdGMGTM0ffp0FRYWqrOzM6xDAxhfyA4AwcQYY8ztnrxq1SqtX79eixcv1rVr1/TMM8/o9OnTOnPmjKZNmyZJKi4u1jvvvKOqqio5nU6VlpYqNjZWH3300W29Rnd3t5xOp/Lz8xUXFze8rwrAiPT19enQoUPyer1KTEwc8fXIDiD6jTQ3QiokN/viiy+UkpKi+vp6LVu2TF6vV7NmzdK+ffu0bt06SdLZs2d1zz33qKGhQQ8++GDQaxIqgH3hLiQ3IzuA6DPS3BjRPSRer1eSlJycLElqbGxUX1+f8vLy/OcsXLhQGRkZamhoGMlLAYgiZAeAm00e7hP7+/u1detWLV26VIsWLZIkdXR0KD4+XklJSQHnpqamqqOjY8Dr+Hw++Xw+/+Pu7u7hjgRgHCA7AAxk2O+QlJSU6PTp06qurh7RAB6PR06n07+lp6eP6HoAIhvZAWAgwyokpaWlqq2t1ZEjRzRnzhz/fpfLpatXr6qrqyvg/M7OTrlcrgGvVV5eLq/X69/a2tqGMxKAcYDsADCYkAqJMUalpaWqqanR4cOHlZmZGXA8JydHcXFxqqur8+9raWnRhQsX5Ha7B7ymw+FQYmJiwAYgupAdAIIJ6R6SkpIS7du3TwcOHFBCQoL/e7tOp1NTp06V0+nUE088obKyMiUnJysxMVFbtmyR2+2+rbvkb0dtbW1YrjPerV69etBjrNENrFFwQ61ROEVCdrzdczEs1xnvHpmWMuixDRs2jOEkkWvv3r2DHmONbhhqjYYrpEJSWVkpSVq+fHnA/j179mjjxo2SpJdeekmxsbEqLCyUz+dTfn6+XnvttbAMC2B8IjsABBNSIbmdH1kyZcoUVVRUqKKiYthDAYguZAeAYPhdNgAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsC6kQuLxeLR48WIlJCQoJSVFa9euVUtLS8A5y5cvV0xMTMC2efPmsA4NYHwhOwAEE1Ihqa+vV0lJiY4fP673339ffX19WrlypXp6egLOe/LJJ9Xe3u7fdu7cGdahAYwvZAeAYCaHcvLBgwcDHldVVSklJUWNjY1atmyZf/8dd9whl8sVngkBjHtkB4BgRnQPidfrlSQlJycH7P/zn/+smTNnatGiRSovL9dXX3016DV8Pp+6u7sDNgDRjewAcLOQ3iH5X/39/dq6dauWLl2qRYsW+ff/5Cc/0dy5c5WWlqbm5mb96le/UktLi/7+978PeB2Px6MdO3YMdwwA4wzZAWAgwy4kJSUlOn36tD788MOA/Zs2bfL/+f7779fs2bO1YsUKnT9/Xnffffct1ykvL1dZWZn/cXd3t9LT04c7FoAIR3YAGMiwCklpaalqa2t17NgxzZkzZ8hzc3NzJUnnzp0bMFQcDoccDsdwxgAwzpAdAAYTUiExxmjLli2qqanR0aNHlZmZGfQ5TU1NkqTZs2cPa0AA4x/ZASCYkApJSUmJ9u3bpwMHDighIUEdHR2SJKfTqalTp+r8+fPat2+ffvSjH2nGjBlqbm7WU089pWXLlikrK2tUvgAAkY/sABBMSIWksrJS0o0fYPS/9uzZo40bNyo+Pl4ffPCBdu/erZ6eHqWnp6uwsFDPPvts2AYGMP6QHQCCCflbNkNJT09XfX39iAYCEH3IDgDB8LtsAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdSEVksrKSmVlZSkxMVGJiYlyu9169913/cd7e3tVUlKiGTNmaPr06SosLFRnZ2fYhwYwvpAdAIKJMcaY2z357bff1qRJkzR//nwZY/TGG29o165dOnXqlO677z4VFxfrnXfeUVVVlZxOp0pLSxUbG6uPPvrotgfq7u6W0+lUfn6+4uLihvVFARiZvr4+HTp0SF6vV4mJiSO+HtkBRL+R5kZIhWQgycnJ2rVrl9atW6dZs2Zp3759WrdunSTp7Nmzuueee9TQ0KAHH3zwtq5HqAD2hbuQDITsAKLLSHNj2PeQXL9+XdXV1erp6ZHb7VZjY6P6+vqUl5fnP2fhwoXKyMhQQ0PDcF8GQJQhOwAMZHKoT/j444/ldrvV29ur6dOnq6amRvfee6+ampoUHx+vpKSkgPNTU1PV0dEx6PV8Pp98Pp//cXd3d6gjARgHyA4AQwn5HZIFCxaoqalJJ06cUHFxsYqKinTmzJlhD+DxeOR0Ov1benr6sK8FIHKRHQCGEnIhiY+P17x585STkyOPx6Ps7Gy9/PLLcrlcunr1qrq6ugLO7+zslMvlGvR65eXl8nq9/q2trS3kLwJA5CM7AAwl5G/Z3Ky/v18+n085OTmKi4tTXV2dCgsLJUktLS26cOGC3G73oM93OBxyOBz+x9/cY3vt2rWRjgZgmL75+zfCe96HRHYA0WXEuWFCsG3bNlNfX29aW1tNc3Oz2bZtm4mJiTHvvfeeMcaYzZs3m4yMDHP48GFz8uRJ43a7jdvtDuUlTFtbm5HExsYWAVtbW1tIf3/JDjY2tuHmRkjvkFy8eFEbNmxQe3u7nE6nsrKydOjQIf3whz+UJL300kuKjY1VYWGhfD6f8vPz9dprr4XyEkpLS1NbW5sSEhIUExOj7u5upaenq62tbdQ+fjjesUZDY32Cu3mNjDG6fPmy0tLSwnJ9siPysD7BsUbB/e8aJSQkjCg3RvxzSEbbNz9bYDR/HsJ4xxoNjfUJLhrXKBq/pnBifYJjjYIL5xrxu2wAAIB1FBIAAGBdxBcSh8Oh7du3B9xNj0Cs0dBYn+CicY2i8WsKJ9YnONYouHCuUcTfQwIAAKJfxL9DAgAAoh+FBAAAWEchAQAA1lFIAACAdRFdSCoqKvSd73xHU6ZMUW5urv75z3/aHsmaY8eO6ZFHHlFaWppiYmK0f//+gOPGGD3//POaPXu2pk6dqry8PH366ad2hrXE4/Fo8eLFSkhIUEpKitauXauWlpaAc3p7e1VSUqIZM2Zo+vTpKiwsVGdnp6WJx1ZlZaWysrKUmJioxMREud1uvfvuu/7j0bQ2ZMe3yI6hkRvBjVV2RGwh+ctf/qKysjJt375d//rXv5Sdna38/HxdvHjR9mhW9PT0KDs7WxUVFQMe37lzp1555RW9/vrrOnHihKZNm6b8/Hz19vaO8aT21NfXq6SkRMePH9f777+vvr4+rVy5Uj09Pf5znnrqKb399tt66623VF9fr88//1yPPfaYxanHzpw5c/Tiiy+qsbFRJ0+e1A9+8AOtWbNGn3zyiaToWRuyIxDZMTRyI7gxy45h/QacMbBkyRJTUlLif3z9+nWTlpZmPB6PxakigyRTU1Pjf9zf329cLpfZtWuXf19XV5dxOBzmzTfftDBhZLh48aKRZOrr640xN9YkLi7OvPXWW/5z/v3vfxtJpqGhwdaYVt15553mj3/8Y1StDdkxOLIjOHLj9oxGdkTkOyRXr15VY2Oj8vLy/PtiY2OVl5enhoYGi5NFptbWVnV0dASsl9PpVG5u7oReL6/XK0lKTk6WJDU2Nqqvry9gnRYuXKiMjIwJt07Xr19XdXW1enp65Ha7o2ZtyI7QkB23IjeGNprZEdJv+x0rX375pa5fv67U1NSA/ampqTp79qylqSJXR0eHJA24Xt8cm2j6+/u1detWLV26VIsWLZJ0Y53i4+OVlJQUcO5EWqePP/5Ybrdbvb29mj59umpqanTvvfeqqakpKtaG7AgN2RGI3BjcWGRHRBYSYKRKSkp0+vRpffjhh7ZHiSgLFixQU1OTvF6v/va3v6moqEj19fW2xwIiArkxuLHIjoj8ls3MmTM1adKkW+7S7ezslMvlsjRV5PpmTVivG0pLS1VbW6sjR45ozpw5/v0ul0tXr15VV1dXwPkTaZ3i4+M1b9485eTkyOPxKDs7Wy+//HLUrA3ZERqy41vkxtDGIjsispDEx8crJydHdXV1/n39/f2qq6uT2+22OFlkyszMlMvlCliv7u5unThxYkKtlzFGpaWlqqmp0eHDh5WZmRlwPCcnR3FxcQHr1NLSogsXLkyodfpf/f398vl8UbM2ZEdoyA5yY7hGJTvCe99t+FRXVxuHw2GqqqrMmTNnzKZNm0xSUpLp6OiwPZoVly9fNqdOnTKnTp0ykszvfvc7c+rUKfPf//7XGGPMiy++aJKSksyBAwdMc3OzWbNmjcnMzDRff/215cnHTnFxsXE6nebo0aOmvb3dv3311Vf+czZv3mwyMjLM4cOHzcmTJ43b7TZut9vi1GNn27Ztpr6+3rS2tprm5mazbds2ExMTY9577z1jTPSsDdkRiOwYGrkR3FhlR8QWEmOMefXVV01GRoaJj483S5YsMcePH7c9kjVHjhwxkm7ZioqKjDE3Pr733HPPmdTUVONwOMyKFStMS0uL3aHH2EDrI8ns2bPHf87XX39tfvGLX5g777zT3HHHHebRRx817e3t9oYeQz//+c/N3LlzTXx8vJk1a5ZZsWKFP1CMia61ITu+RXYMjdwIbqyyI8YYY4b5jg0AAEBYROQ9JAAAYGKhkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALDu/wPBqPU8AVQmZgAAAABJRU5ErkJggg==\n"455},456"metadata": {}457},458{459"output_type": "stream",460"name": "stdout",461"text": [462"TEST:\n"463]464},465{466"output_type": "display_data",467"data": {468"text/plain": [469"<Figure size 640x480 with 2 Axes>"470],471"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAEPCAYAAABycN8YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZnElEQVR4nO3dcUzU9/3H8RconFbhKCqcRHC0Om1roQlTerMz9icTWTTaYjK7LerW1MjAxJKlk6attVtyjSatbUdpsmVSs1m3LkOjTbUtKqYtusgk1jpJNWzSFLA14U5pOVE+vz9MbzsFzoODz3E8H8k38b7f733vzafx1ZfH9yDOGGMEAABgUbztAQAAACgkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwLqxQ3XhyspKbdu2TW1tbcrNzdVrr72mefPmhXxeT0+PvvjiCyUlJSkuLm6oxgPQD2OMLl++rIyMDMXHD9+/WwaaGxLZAdg26NwwQ2D37t0mMTHR/PGPfzSffvqpeeKJJ0xKSoppb28P+dyWlhYjiY2NLQq2lpaWoYiIXg0mN4whO9jYomUbaG7EGRP5X66Xn5+vuXPn6ne/+52kG/9yyczM1IYNG7Rp06Z+n+v1epWSkqJFixZp7NghewMHQD+uXbum2tpadXR0yOl0DstrDiY3JLIDsG2wuRHxv7VXr15VQ0ODKioqAvvi4+NVUFCg+vr6W873+/3y+/2Bx5cvX74x2NixSkhIiPR4AMIwXN/6CDc3JLIDiFYDzY2If3P4q6++0vXr15Wenh60Pz09XW1tbbec7/F45HQ6A1tmZmakRwIQ5cLNDYnsAGKN9U/ZVFRUyOv1BraWlhbbIwEYAcgOILZE/Fs2kydP1pgxY9Te3h60v729XS6X65bzHQ6HHA5HpMcAMIKEmxsS2QHEmogXksTEROXl5am2tlYrVqyQdOPmtNraWpWVlQ36+n/6uHzQ14gFP/v+S30eu/D8e8M4SfTKen5xn8dWr149jJNEr507d9oeQdLQ54Yk7d+/PyLXGemWLl3a5zHW6Ib+1mjfdtZIkpZt7HuNBmpIbkUvLy/XmjVr9L3vfU/z5s3T9u3b1dnZqZ///OdD8XIAYgC5AYxuQ1JIfvzjH+vLL7/Uc889p7a2Nj3wwAM6cODALTesAcC3yA1gdBuyD+uXlZVF7K1WAKMDuQGMXtY/ZQMAAEAhAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWBfxQvL8888rLi4uaJs9e3akXwZADCE3AIwdioved999+uCDD/77ImOH5GUAxBByAxjdhuRv/NixY+VyuYbi0gBiFLkBjG5Dcg/JZ599poyMDN1111366U9/qgsXLgzFywCIIeQGMLpF/B2S/Px8VVdXa9asWWptbdWWLVv0gx/8QKdPn1ZSUtIt5/v9fvn9/sBjn88X6ZEARLlwc0MiO4BYE/FCUlRUFPhzTk6O8vPzNX36dP31r3/V448/fsv5Ho9HW7ZsifQYAEaQcHNDIjuAWDPkH/tNSUnRd7/7XZ07d67X4xUVFfJ6vYGtpaVlqEcCEOVC5YZEdgCxZsgLyZUrV3T+/HlNnTq11+MOh0PJyclBG4DRLVRuSGQHEGsiXkh+9atfqa6uTv/+97/18ccf65FHHtGYMWP02GOPRfqlAMQIcgNAxO8h+fzzz/XYY4/p0qVLmjJlih566CEdO3ZMU6ZMifRLAYgR5AaAiBeS3bt3R/qSAGIcuQGA32UDAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADr4owxJpwnHD16VNu2bVNDQ4NaW1tVU1OjFStWBI4bY7R582b9/ve/V0dHh+bPn6+qqirNnDnztq7v8/nkdDpVWFiohISEsL4YAJHR3d2tgwcPyuv1Kjk5edDXG+rckMgOwLbB5kbY75B0dnYqNzdXlZWVvR7funWrXn31Vb3xxhs6fvy4JkyYoMLCQnV1dYU9HIDYQG4ACGVsuE8oKipSUVFRr8eMMdq+fbueeeYZLV++XJK0c+dOpaena8+ePVq1atXgpgUwIpEbAEKJ6D0kzc3NamtrU0FBQWCf0+lUfn6+6uvre32O3++Xz+cL2gCMHgPJDYnsAGJNRAtJW1ubJCk9PT1of3p6euDYzTwej5xOZ2DLzMyM5EgAotxAckMiO4BYY/1TNhUVFfJ6vYGtpaXF9kgARgCyA4gtES0kLpdLktTe3h60v729PXDsZg6HQ8nJyUEbgNFjILkhkR1ArAn7ptb+ZGdny+Vyqba2Vg888ICkGx/FO378uEpKSiLyGn/6uDwi1xnpfvb9l/o8lvfyvmGcJHo1PLmsz2P79+8fxkmi19KlS22PMCy5IfHf/Fv9/TcnO27oLzv4f9AN/f0/aKDCLiRXrlzRuXPnAo+bm5vV2Nio1NRUZWVlaePGjfrtb3+rmTNnKjs7W88++6wyMjKCfuYAgNGF3AAQStiF5MSJE3r44YcDj8vLb7TFNWvWqLq6Wk899ZQ6Ozu1bt06dXR06KGHHtKBAwc0bty4yE0NYEQhNwCEEnYhWbhwofr74a5xcXF64YUX9MILLwxqMACxg9wAEIr1T9kAAABQSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANaFXUiOHj2qZcuWKSMjQ3FxcdqzZ0/Q8bVr1youLi5oW7JkSaTmBTACkRsAQgm7kHR2dio3N1eVlZV9nrNkyRK1trYGtrfeemtQQwIY2cgNAKGMDfcJRUVFKioq6vcch8Mhl8s14KEAxBZyA0AoQ3IPyZEjR5SWlqZZs2appKREly5dGoqXARBDyA1gdAv7HZJQlixZokcffVTZ2dk6f/68nn76aRUVFam+vl5jxoy55Xy/3y+/3x947PP5Ij0SgCgXbm5IZAcQayJeSFatWhX48/3336+cnBzdfffdOnLkiBYtWnTL+R6PR1u2bIn0GABGkHBzQyI7gFgz5B/7veuuuzR58mSdO3eu1+MVFRXyer2BraWlZahHAhDlQuWGRHYAsSbi75Dc7PPPP9elS5c0derUXo87HA45HI6hHgPACBIqNySyA4g1YReSK1euBP2rpbm5WY2NjUpNTVVqaqq2bNmi4uJiuVwunT9/Xk899ZRmzJihwsLCiA4OYOQgNwCEEnYhOXHihB5++OHA4/LycknSmjVrVFVVpVOnTunNN99UR0eHMjIytHjxYv3mN7/hXzLAKEZuAAgl7EKycOFCGWP6PH7w4MFBDQQg9pAbAELhd9kAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMA6CgkAALCOQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrwiokHo9Hc+fOVVJSktLS0rRixQo1NTUFndPV1aXS0lJNmjRJEydOVHFxsdrb2yM6NICRhewAEEqcMcbc7slLlizRqlWrNHfuXF27dk1PP/20Tp8+rTNnzmjChAmSpJKSEr3zzjuqrq6W0+lUWVmZ4uPj9dFHH93Wa/h8PjmdThUWFiohIWFgXxWAQenu7tbBgwfl9XqVnJw86OuRHUDsG2xuhFVIbvbll18qLS1NdXV1WrBggbxer6ZMmaJdu3Zp5cqVkqSzZ8/qnnvuUX19vR588MGQ1yRUAPsiXUhuRnYAsWewuTGoe0i8Xq8kKTU1VZLU0NCg7u5uFRQUBM6ZPXu2srKyVF9fP5iXAhBDyA4ANxs70Cf29PRo48aNmj9/vubMmSNJamtrU2JiolJSUoLOTU9PV1tbW6/X8fv98vv9gcc+n2+gIwEYAcgOAL0Z8DskpaWlOn36tHbv3j2oATwej5xOZ2DLzMwc1PUARDeyA0BvBlRIysrKtH//fh0+fFjTpk0L7He5XLp69ao6OjqCzm9vb5fL5er1WhUVFfJ6vYGtpaVlICMBGAHIDgB9CauQGGNUVlammpoaHTp0SNnZ2UHH8/LylJCQoNra2sC+pqYmXbhwQW63u9drOhwOJScnB20AYgvZASCUsO4hKS0t1a5du7R3714lJSUFvrfrdDo1fvx4OZ1OPf744yovL1dqaqqSk5O1YcMGud3u27pL/nbs274/ItcZ6ZZtXNrnsf37WSNJWrqUNQqlvzWKpGjIjtWrV0fkOiPdzp07+zx24fn3hnGS6JX1/OI+j/3p4/JhnCR6/ez7L0X8mmEVkqqqKknSwoULg/bv2LFDa9eulSS9/PLLio+PV3Fxsfx+vwoLC/X6669HZFgAIxPZASCUsArJ7fzIknHjxqmyslKVlZUDHgpAbCE7AITC77IBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1YRUSj8ejuXPnKikpSWlpaVqxYoWampqCzlm4cKHi4uKCtvXr10d0aAAjC9kBIJSwCkldXZ1KS0t17Ngxvf/+++ru7tbixYvV2dkZdN4TTzyh1tbWwLZ169aIDg1gZCE7AIQyNpyTDxw4EPS4urpaaWlpamho0IIFCwL777jjDrlcrshMCGDEIzsAhDKoe0i8Xq8kKTU1NWj/n//8Z02ePFlz5sxRRUWFvv766z6v4ff75fP5gjYAsY3sAHCzsN4h+V89PT3auHGj5s+frzlz5gT2/+QnP9H06dOVkZGhU6dO6de//rWampr097//vdfreDwebdmyZaBjABhhyA4AvRlwISktLdXp06f14YcfBu1ft25d4M/333+/pk6dqkWLFun8+fO6++67b7lORUWFysvLA499Pp8yMzMHOhaAKEd2AOjNgApJWVmZ9u/fr6NHj2ratGn9npufny9JOnfuXK+h4nA45HA4BjIGgBGG7ADQl7AKiTFGGzZsUE1NjY4cOaLs7OyQz2lsbJQkTZ06dUADAhj5yA4AoYRVSEpLS7Vr1y7t3btXSUlJamtrkyQ5nU6NHz9e58+f165du/SjH/1IkyZN0qlTp/Tkk09qwYIFysnJGZIvAED0IzsAhBJWIamqqpJ04wcY/a8dO3Zo7dq1SkxM1AcffKDt27ers7NTmZmZKi4u1jPPPBOxgQGMPGQHgFDC/pZNfzIzM1VXVzeogQDEHrIDQCj8LhsAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGAdhQQAAFhHIQEAANZRSAAAgHUUEgAAYB2FBAAAWEchAQAA1lFIAACAdRQSAABgHYUEAABYRyEBAADWUUgAAIB1FBIAAGBdWIWkqqpKOTk5Sk5OVnJystxut959993A8a6uLpWWlmrSpEmaOHGiiouL1d7eHvGhAYwsZAeAUOKMMeZ2T963b5/GjBmjmTNnyhijN998U9u2bdPJkyd13333qaSkRO+8846qq6vldDpVVlam+Ph4ffTRR7c9kM/nk9PpVGFhoRISEgb0RQEYnO7ubh08eFBer1fJycmDvh7ZAcS+weZGWIWkN6mpqdq2bZtWrlypKVOmaNeuXVq5cqUk6ezZs7rnnntUX1+vBx988LauR6gA9kW6kPSG7ABiy2BzY8D3kFy/fl27d+9WZ2en3G63Ghoa1N3drYKCgsA5s2fPVlZWlurr6wf6MgBiDNkBoDdjw33CJ598Irfbra6uLk2cOFE1NTW699571djYqMTERKWkpASdn56erra2tj6v5/f75ff7A499Pl+4IwEYAcgOAP0J+x2SWbNmqbGxUcePH1dJSYnWrFmjM2fODHgAj8cjp9MZ2DIzMwd8LQDRi+wA0J+wC0liYqJmzJihvLw8eTwe5ebm6pVXXpHL5dLVq1fV0dERdH57e7tcLlef16uoqJDX6w1sLS0tYX8RAKIf2QGgP2F/y+ZmPT098vv9ysvLU0JCgmpra1VcXCxJampq0oULF+R2u/t8vsPhkMPhCDz+9h7ba9euDXY0AAP07d+/Qd7z3i+yA4gtg84NE4ZNmzaZuro609zcbE6dOmU2bdpk4uLizHvvvWeMMWb9+vUmKyvLHDp0yJw4ccK43W7jdrvDeQnT0tJiJLGxsUXB1tLSEtbfX7KDjY1toLkR1jskFy9e1OrVq9Xa2iqn06mcnBwdPHhQP/zhDyVJL7/8suLj41VcXCy/36/CwkK9/vrr4byEMjIy1NLSoqSkJMXFxcnn8ykzM1MtLS1D9vHDkY416h/rE9rNa2SM0eXLl5WRkRGR65Md0Yf1CY01Cu1/1ygpKWlQuTHon0My1L792QJD+fMQRjrWqH+sT2ixuEax+DVFEusTGmsUWiTXiN9lAwAArKOQAAAA66K+kDgcDm3evDnobnoEY436x/qEFotrFItfUySxPqGxRqFFco2i/h4SAAAQ+6L+HRIAABD7KCQAAMA6CgkAALCOQgIAAKyL6kJSWVmp73znOxo3bpzy8/P1j3/8w/ZI1hw9elTLli1TRkaG4uLitGfPnqDjxhg999xzmjp1qsaPH6+CggJ99tlndoa1xOPxaO7cuUpKSlJaWppWrFihpqamoHO6urpUWlqqSZMmaeLEiSouLlZ7e7uliYdXVVWVcnJylJycrOTkZLndbr377ruB47G0NmTHf5Ed/SM3Qhuu7IjaQvKXv/xF5eXl2rx5s/75z38qNzdXhYWFunjxou3RrOjs7FRubq4qKyt7Pb5161a9+uqreuONN3T8+HFNmDBBhYWF6urqGuZJ7amrq1NpaamOHTum999/X93d3Vq8eLE6OzsD5zz55JPat2+f3n77bdXV1emLL77Qo48+anHq4TNt2jS9+OKLamho0IkTJ/R///d/Wr58uT799FNJsbM2ZEcwsqN/5EZow5YdA/oNOMNg3rx5prS0NPD4+vXrJiMjw3g8HotTRQdJpqamJvC4p6fHuFwus23btsC+jo4O43A4zFtvvWVhwuhw8eJFI8nU1dUZY26sSUJCgnn77bcD5/zrX/8ykkx9fb2tMa268847zR/+8IeYWhuyo29kR2jkxu0ZiuyIyndIrl69qoaGBhUUFAT2xcfHq6CgQPX19RYni07Nzc1qa2sLWi+n06n8/PxRvV5er1eSlJqaKklqaGhQd3d30DrNnj1bWVlZo26drl+/rt27d6uzs1Nutztm1obsCA/ZcStyo39DmR1h/bbf4fLVV1/p+vXrSk9PD9qfnp6us2fPWpoqerW1tUlSr+v17bHRpqenRxs3btT8+fM1Z84cSTfWKTExUSkpKUHnjqZ1+uSTT+R2u9XV1aWJEyeqpqZG9957rxobG2NibciO8JAdwciNvg1HdkRlIQEGq7S0VKdPn9aHH35oe5SoMmvWLDU2Nsrr9epvf/ub1qxZo7q6OttjAVGB3OjbcGRHVH7LZvLkyRozZswtd+m2t7fL5XJZmip6fbsmrNcNZWVl2r9/vw4fPqxp06YF9rtcLl29elUdHR1B54+mdUpMTNSMGTOUl5cnj8ej3NxcvfLKKzGzNmRHeMiO/yI3+jcc2RGVhSQxMVF5eXmqra0N7Ovp6VFtba3cbrfFyaJTdna2XC5X0Hr5fD4dP358VK2XMUZlZWWqqanRoUOHlJ2dHXQ8Ly9PCQkJQevU1NSkCxcujKp1+l89PT3y+/0xszZkR3jIDnJjoIYkOyJ7323k7N692zgcDlNdXW3OnDlj1q1bZ1JSUkxbW5vt0ay4fPmyOXnypDl58qSRZF566SVz8uRJ85///McYY8yLL75oUlJSzN69e82pU6fM8uXLTXZ2tvnmm28sTz58SkpKjNPpNEeOHDGtra2B7euvvw6cs379epOVlWUOHTpkTpw4Ydxut3G73RanHj6bNm0ydXV1prm52Zw6dcps2rTJxMXFmffee88YEztrQ3YEIzv6R26ENlzZEbWFxBhjXnvtNZOVlWUSExPNvHnzzLFjx2yPZM3hw4eNpFu2NWvWGGNufHzv2WefNenp6cbhcJhFixaZpqYmu0MPs97WR5LZsWNH4JxvvvnG/PKXvzR33nmnueOOO8wjjzxiWltb7Q09jH7xi1+Y6dOnm8TERDNlyhSzaNGiQKAYE1trQ3b8F9nRP3IjtOHKjjhjjBngOzYAAAAREZX3kAAAgNGFQgIAAKyjkAAAAOsoJAAAwDoKCQAAsI5CAgAArKOQAAAA6ygkAADAOgoJAACwjkICAACso5AAAADrKCQAAMC6/wem2PU8xJ4X7QAAAABJRU5ErkJggg==\n"472},473"metadata": {}474}475]476},477{478"cell_type": "markdown",479"source": [480"## **Evaluate:** ARC Benchmark\n",481"\n",482"Evaluate on the available 800 tasks.\n",483"\n",484"**Note:** LLM temperature is set to 0 (deterministic), but your results might still vary depending on stability of the API."485],486"metadata": {487"id": "A6Jz5lc5bQRx"488}489},490{491"cell_type": "code",492"source": [493"success = {}\n",494"for task_id in sorted_task_ids:\n",495" task_json, task_name = tasks_jsons[task_id], tasks_names[task_id]\n",496"\n",497" # Lazy load: skip evals where we already have results.\n",498" if task_name in success:\n",499" continue\n",500"\n",501" # Build context and expected output labels.\n",502" context = []\n",503" batch_prompts = []\n",504" batch_labels = []\n",505" train_xy, test_x, test_y = task_json_to_tokens(task_json, value_to_token)\n",506" test_num_tokens = np.max([len(x) + len(y) for x, y in zip(test_x, test_y)])\n",507" for sample in train_xy:\n",508" if len(context) + len(sample) + test_num_tokens > token_limit: # Ensure both train and test examples can fit in the prompt.\n",509" break\n",510" context += sample\n",511"\n",512" # There can be multiple test examples so put them in the same batch.\n",513" for x, y in zip(test_x, test_y):\n",514" batch_prompts.append(context + x)\n",515" batch_labels.append(y)\n",516"\n",517" # Run LLM.\n",518" try:\n",519" stop_token = tokenizer.decode(sample_delim, skip_special_tokens=True)\n",520" max_tokens = int(np.max([len(y) for y in test_y])) + 10\n",521" batch_responses = LLM(batch_prompts, stop=stop_token, max_tokens=max_tokens, temperature=0)\n",522" except Exception as e:\n",523" print(task_name, f\"LLM failed. {e}\")\n",524" continue\n",525"\n",526" # Check answers and save success rates.\n",527" success[task_name] = 0\n",528" for response, label in zip(batch_responses, batch_labels):\n",529" label_str = tokenizer.decode(label, skip_special_tokens=True)\n",530" is_success = label_str.strip() in response\n",531" success[task_name] += is_success / len(batch_labels)\n",532" success[task_name] = int(success[task_name] > 0.99) # All test cases need to correct.\n",533"\n",534" # Debug prints.\n",535" total_success = np.sum(list(success.values()))\n",536" print(task_name, \"Success:\", success[task_name], \"Total:\", f\"{total_success} / {len(success)}\")\n",537"\n",538" # # Save results.\n",539" # result_file = f\"arc-{model}-alphabet-{'-'.join(map(str, alphabet))}.pkl\"\n",540" # with open(result_file, 'wb') as fid:\n",541" # pickle.dump(success, fid, protocol=pickle.HIGHEST_PROTOCOL)"542],543"metadata": {544"id": "GuFgcjPxsxXG"545},546"execution_count": null,547"outputs": []548}549]550}