LLM-FineTuning-Large-Language-Models

1
{"cells":[{"cell_type":"markdown","metadata":{},"source":["# Fine Tuning DistilBERT for Multiclass Text Classification\n","\n","## Model - 'distilbert-base-uncased'\n","\n","\n","## Dataset Link - https://storage.googleapis.com/dataset-uploader/bbc/bbc-text.csv"]},{"cell_type":"markdown","metadata":{},"source":["## First What is BERT?\n","\n","BERT stands for Bidirectional Encoder Representations from Transformers. The name itself gives us several clues to what BERT is all about.\n","\n","BERT architecture consists of several Transformer encoders stacked together. Each Transformer encoder encapsulates two sub-layers: a self-attention layer and a feed-forward layer.\n","\n","### There are two different BERT models:\n","\n","- BERT base, which is a BERT model consists of 12 layers of Transformer encoder, 12 attention heads, 768 hidden size, and 110M parameters.\n","\n","- BERT large, which is a BERT model consists of 24 layers of Transformer encoder,16 attention heads, 1024 hidden size, and 340 parameters.\n","\n","\n","\n","BERT Input and Output\n","BERT model expects a sequence of tokens (words) as an input. In each sequence of tokens, there are two special tokens that BERT would expect as an input:\n","\n","- [CLS]: This is the first token of every sequence, which stands for classification token.\n","- [SEP]: This is the token that makes BERT know which token belongs to which sequence. This special token is mainly important for a next sentence prediction task or question-answering task. If we only have one sequence, then this token will be appended to the end of the sequence.\n","\n","\n","It is also important to note that the maximum size of tokens that can be fed into BERT model is 512. If the tokens in a sequence are less than 512, we can use padding to fill the unused token slots with [PAD] token. If the tokens in a sequence are longer than 512, then we need to do a truncation.\n","\n","And that’s all that BERT expects as input.\n","\n","BERT model then will output an embedding vector of size 768 in each of the tokens. We can use these vectors as an input for different kinds of NLP applications, whether it is text classification, next sentence prediction, Named-Entity-Recognition (NER), or question-answering.\n","\n","\n","------------\n","\n","**For a text classification task**, we focus our attention on the embedding vector output from the special [CLS] token. This means that we’re going to use the embedding vector of size 768 from [CLS] token as an input for our classifier, which then will output a vector of size the number of classes in our classification task.\n","\n","-----------------------\n","\n","![Imgur](https://imgur.com/NpeB9vb.png)\n","\n","-------------------------"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-09-23T08:06:44.342740Z","iopub.status.busy":"2022-09-23T08:06:44.342112Z","iopub.status.idle":"2022-09-23T08:06:44.347039Z","shell.execute_reply":"2022-09-23T08:06:44.346050Z","shell.execute_reply.started":"2022-09-23T08:06:44.342700Z"},"trusted":true},"outputs":[],"source":["!pip  install transformers==4.22.1 -q"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-09-23T08:06:44.349259Z","iopub.status.busy":"2022-09-23T08:06:44.348566Z","iopub.status.idle":"2022-09-23T08:06:44.452061Z","shell.execute_reply":"2022-09-23T08:06:44.451086Z","shell.execute_reply.started":"2022-09-23T08:06:44.349178Z"},"trusted":true},"outputs":[],"source":["import transformers"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-09-23T08:06:44.454848Z","iopub.status.busy":"2022-09-23T08:06:44.454516Z","iopub.status.idle":"2022-09-23T08:06:44.461681Z","shell.execute_reply":"2022-09-23T08:06:44.460702Z","shell.execute_reply.started":"2022-09-23T08:06:44.454814Z"},"trusted":true},"outputs":[],"source":["print(transformers.__version__)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-09-23T08:06:44.464333Z","iopub.status.busy":"2022-09-23T08:06:44.463179Z","iopub.status.idle":"2022-09-23T08:06:47.251269Z","shell.execute_reply":"2022-09-23T08:06:47.250110Z","shell.execute_reply.started":"2022-09-23T08:06:44.464293Z"},"trusted":true},"outputs":[],"source":["from transformers import DistilBertTokenizer\n","from transformers import TFDistilBertForSequenceClassification\n","from transformers import TextClassificationPipeline\n","\n","import tensorflow as tf\n","import pandas as pd\n","import json\n","import gc\n","\n","from sklearn.model_selection import train_test_split\n","\n","import re\n","import nltk\n","from nltk.corpus import stopwords\n","nltk.download('stopwords')\n","stopw = stopwords.words('english')\n","\n","import seaborn as sns\n","import matplotlib.pyplot as plt\n","from plotly.offline import iplot\n","\n","from tqdm import tqdm"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-09-23T08:06:47.253574Z","iopub.status.busy":"2022-09-23T08:06:47.252641Z","iopub.status.idle":"2022-09-23T08:06:47.259139Z","shell.execute_reply":"2022-09-23T08:06:47.257888Z","shell.execute_reply.started":"2022-09-23T08:06:47.253540Z"},"trusted":true},"outputs":[],"source":["root_path = '../input/bbc-fulltext-and-category/bbc-text.csv'"]},{"cell_type":"code","execution_count":35,"metadata":{"execution":{"iopub.execute_input":"2022-09-23T08:06:47.261633Z","iopub.status.busy":"2022-09-23T08:06:47.260640Z","iopub.status.idle":"2022-09-23T08:06:47.330921Z","shell.execute_reply":"2022-09-23T08:06:47.329991Z","shell.execute_reply.started":"2022-09-23T08:06:47.261592Z"},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>tech</td>\n","      <td>tv future in the hands of viewers with home th...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>business</td>\n","      <td>worldcom boss  left books alone  former worldc...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>sport</td>\n","      <td>tigers wary of farrell  gamble  leicester say ...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>sport</td>\n","      <td>yeading face newcastle in fa cup premiership s...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>entertainment</td>\n","      <td>ocean s twelve raids box office ocean s twelve...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        category                                               text\n","0           tech  tv future in the hands of viewers with home th...\n","1       business  worldcom boss  left books alone  former worldc...\n","2          sport  tigers wary of farrell  gamble  leicester say ...\n","3          sport  yeading face newcastle in fa cup premiership s...\n","4  entertainment  ocean s twelve raids box office ocean s twelve..."]},"execution_count":35,"metadata":{},"output_type":"execute_result"}],"source":["df = pd.read_csv(root_path)\n","df.head()"]},{"cell_type":"code","execution_count":36,"metadata":{"execution":{"iopub.execute_input":"2022-09-23T08:06:47.332851Z","iopub.status.busy":"2022-09-23T08:06:47.332489Z","iopub.status.idle":"2022-09-23T08:06:47.339025Z","shell.execute_reply":"2022-09-23T08:06:47.338074Z","shell.execute_reply.started":"2022-09-23T08:06:47.332815Z"},"trusted":true},"outputs":[{"data":{"text/plain":["(2225, 2)"]},"execution_count":36,"metadata":{},"output_type":"execute_result"}],"source":["df.shape"]},{"cell_type":"markdown","metadata":{},"source":["# Histogram of the count of text"]},{"cell_type":"code","execution_count":38,"metadata":{},"outputs":[],"source":["df['count'] = df['text'].apply(lambda x: len(x.split()))"]},{"cell_type":"code","execution_count":39,"metadata":{},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>text</th>\n","      <th>count</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>tech</td>\n","      <td>tv future in the hands of viewers with home th...</td>\n","      <td>737</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>business</td>\n","      <td>worldcom boss  left books alone  former worldc...</td>\n","      <td>300</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>sport</td>\n","      <td>tigers wary of farrell  gamble  leicester say ...</td>\n","      <td>246</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>sport</td>\n","      <td>yeading face newcastle in fa cup premiership s...</td>\n","      <td>341</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>entertainment</td>\n","      <td>ocean s twelve raids box office ocean s twelve...</td>\n","      <td>260</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        category                                               text  count\n","0           tech  tv future in the hands of viewers with home th...    737\n","1       business  worldcom boss  left books alone  former worldc...    300\n","2          sport  tigers wary of farrell  gamble  leicester say ...    246\n","3          sport  yeading face newcastle in fa cup premiership s...    341\n","4  entertainment  ocean s twelve raids box office ocean s twelve...    260"]},"execution_count":39,"metadata":{},"output_type":"execute_result"}],"source":["df.head()"]},{"cell_type":"code","execution_count":40,"metadata":{},"outputs":[{"data":{"text/plain":["<Figure size 576x576 with 0 Axes>"]},"metadata":{},"output_type":"display_data"},{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAW0AAAF5CAYAAABURnuvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmaUlEQVR4nO3de5wcVZ338c+XICQBEhICmBAg8Bhd0F0Rswqrq+GiRlwXdRVlkUtEWQWeFXVRlMdHfHZ11RVddblFLuEmFxUEkQdECCAuIAjInRAlQJghEBICQgBhfvvHOU0qne6Znsn09JyZ7/v16ld3nTpd/auu6l9XnTpVpYjAzMzKsF6nAzAzs9Y5aZuZFcRJ28ysIE7aZmYFcdI2MyuIk7aZWUFGdNKWNFtSSDqo07EMN5LmSyq2v6ekD0r6vaRVeRnP7nRM60LSQcNhPjoRR6PP7NT3MVyWQ2+KStr5y2z1MaNDMS7On39dk/Hz8/gpQx3bSCHp1cA5wErgcGB/4J4mdc/N3/erGow7MY87q8G4V+ZxPx/k8IdEZYOl9nhJ0gpJd0o6XdIcSRrkzzxG0vsGc5rtkL+bYyRt2ulYBmL9TgfQT/vXDf8tcAgwD/h13bjHgRlDEFMzb5G0d0Rc1MEYRqrZpHX3iIi4pY+6C4AP5/csqhu3G/Ai8PYG79ut8v6SnQNcCgjYBHgN8D7gAOBXkj4UEU9W6p8JnAu8MIDP+gpwOvCzfr5vXT5zIGaTYp0PPNnhWPqtqKQdEWtsEUlan5S0r68fl8cPVWj1HgTGA1+XdElEvNSpQIYLSZtExNODNLlX5uflLdStJd3ZwMmVeKYCrwZOA+ZKelVEVJP67Px89boEWvm8wZz//rilwe/ms8C3gM+Skvq7a+Pyujok62vtOxnKz+zLcIqlmaKaR9aFpLmS7pL0vKQHJX2+Sb1Zki6UtCzXvU/S0fkPolV/Av4N2BE4qIXYmrYv513b+ZXhGbnsGEn7SLott+sukjQ319lG0k8kLZf0tKSzJG3SZPqbSzpD0hOSnpF0paSdm9T9sKTr8jSflXSjpA82i1nSHrn+n4A+mxkkvU3SFZJW5nm6RdLB9dMGvpoHH8iftbjZNCNiIdDF6iRcUxv+Omlre7cG458Ebqt89sdzTKtyjL+U9NYG89Hr/Ev6hKR78/q1SNIRpC3h+ulMlvRdSX+Q9FxeRr+TdGSz+W1FRLwUEZ8DrgPmVOdBjduXx+b17b683J+UdIek/8jjZ1TW3wOrzTKtfCeNPrNi/fzZD+bv63ZJH2nwXa3xO2k2P7nOV/LoByqxHtNbLJKmSDpO0sOSXsjPx0narMnn7S7pX/Kye17SQkkHNpi/fitqS3sdfBLYEjiF9EP8KPBNSUsi4ke1SpLeA1xA2o0+lrQltyvw/4CdgA/14zNPBI4AvirpRxGxap3nYk1/R5qv40lxHgycKukFUiK6CvgS8NfAx4DngI83mM5l+f3HkLZgDweukbRrRNxZqyTp34Cjc/0vAz3A+4EfSzo8Io6rm+4s4B+AH5J2mXsl6b3AhcCjpO/+aeAjwMmSto+Io3PV/YEP5M/+DLCM9CfZmwXAfpJmRsT9uWw2cF9ELJJ0Sx7+YY6lthV+UUT05LJvAp8Hfkv6Xjch7eUtUGoGu7SV+c8J+rvA7/N0xgP/AjzWIO4fA28jrUu3A+OAHXKs/9HHPLfiFOCtwHtICbyZ40jr0BnAd0h5Yyawex7/OGm5nElqppzXZDr9WieybwIbkdZzgLnAOZLGRsT8FqdRdRIwgTXXH0jfb0OSJgL/DbwKOBW4BXgD8Clgd0lvarAX9XXS8joJeD7XnS9pUUT8ZgBxrxYRxT5IW7EBHNRk/Ow8vguYWCkfT1rRrq+UjSUljGuB9eum85k8ndktxLQYuDO//sf8vqMq4+fnsin1ZU2mF8D8yvCMXPYMsG2lfHNSYu4BPls3jQtIbXQbN4jjAkCV8jfmaVxWKds51/16g/h+BjwFbFIXcwB7trgcx5CalJ4EplXKNwB+Q9pdnVkpPyZPf0aL0z841z+kUnYfcGJ+/U3gkcq4fXP9I/Lwa/J3ch2wQaXetBzzYmBMX/MPbJqX293A+Er5dNIfz8vrGDAxDx8/wN/G7Pz+f+mlTm25/rTBb2p2pWw5cGkLn7nGutpgXMN1osln1soeZM3f7sRcthwY19dnN5l20/WnSf2v5bJD6+oelsv/tcH7b61bV7YiJe9zBrI8q4/R0jxyWkSsrA1ExLPADaSthZp3kLbGTwM2zbtDU5R6edS2ot7Zz889h/Sv/AVJkwccfWM/i4gHawMR8TgpEfWQtoyqfg28gsYHZr8Vea3K0/kdcAWwp6SNc/F+pBXx9Or3kr+bi0lbnbvWTff3EfGrFufljcA2wKkR0VWJ5QVS2+t6wN4tTquRart2dUv6mlx+DTBNqVfKy/Uq79ub1HzxrRxTLb4u0vqyLWnLq6rR/L+TtMFwXF4Ha9NZApxdV3cV6Uf+ZrWvJ9RT+XlCH/VWAq+V9Lp1/Lz+rBM1J9T9dleS9jwmsXaTV7u8n7SRV78HcVIuf3+D9xxft648AixkzZwzIKMlaf+xQdkTQLU9aof8fCppQVQf9+ZxW/bnQ3MyPIq0hXV077X7rdE8rQC6I+L5BuWw5vzWNOoqdzdp63fbPLwDKWndy9rfzSm5Tv13s7C34Otsl5/vajCuVrZ9P6a3hoj4I/AQq3/kteda0r6OtDW/W2X8clbvMg8kvkbzX6tzb4Nxd9fF/AKpee11pLbXuyT9QNIeDd47ULVk/VSvtVIck4A7chvtyZL2ltTf/NGfdaKm2foJ67BO9NN2pKa0F6uFeXhhkzhayTkDMlratFs5Glw7EHQklYNPdbqalDcVEVdI+hVwmKTvNavWMKDeD342m6fe5nWg3WlEivHdvUy/PqE927BW5ywgHSR7Dbn7X22rPiKeknQbMFvSxaSt8AureyADsM7zHxEnSrqI1Ob8duCDwOGSzouItQ7GDcBf5ef7+ojjory1v1eOY09Sk9OvJe1Z3aLsQyfWiU7luGa/k3Xu0jZaknYrageonhnALlxfvgDcDPwrjRP0cki9BSKi2o1tKLYkdiA1FVXtSFrpas0v9wNzgIciouFJLOuotlXy2gbjdqyrM1ALgANJW9OzWb2VXXMN6RjE7Er9RvH9YR3iq9X5C+DKJtNZQ0R0k7oqnixpDOlg376Sjo2Im1r4zN7Ueub8oq+Keb08CzhLkoBvkA7M7k06YNouOwD15zo0+s6XA42aIBv9hvr7Z/xH4DWS1q9ubeeNqlez7utmv4yW5pFWXE46gn9Uo/ZnSePUpNtcXyKdAHIuqdfKXzaoUttt3LOu/HMD+bx++nz+EQKg1N1vT+DKiKj1yjgzP389J441SOpXs1EDt5CaL+ZKqvXBRtIrSHs+wdo/3P6qJeF9WbM9u+YaUu+ZT+bhqyvjLs4xHJljqsU3ldSb4UHSgae+XEFqqz5M0vjKdKaT/jColI2v1oGX+xDXmmwGfIxE0hhJ3yb1HLk0eunNkOtuWhdH7UBbfRx/Wpe4mvhU7r1Ri2ciaRk9yZrLcCGwa933Oom0fOrV1utWY/0Z6UB/fe+rT+TyC1uczqDwlnYWEc9IOoC0gO6TdCqp69+mpC2jWjezqwf4Ef+H1N2pUR/oc0hdhOZJ+gvSVsMcYChOdd8WuDw3C0wldflbRUqWAETETbkf6zHAbZJ+TGoqmko6iLgXqafHgETES5IOJ638N0maR+ry92FgF1Kvlft7m0YLn/GQpD+SutDB2kn716SDuG8jdQO7s/Le+5T6JH8euFbSeazu8rcxsF+0cAJVRKyQ9GXg28B/SzqDdGDyk6S9merBzFeTul5emGNZQdrq/BTwAGufAdzMzpI+ml9Xz4jcFvgldX8WDWwCdOf141bShs12OY4VrNn//gbSAewvkP6EIyLObTHOZpYBN0o6LQ/PJR20/nj1YC7wX6Q9gasknUn63X6C9If6StZU27P8pqSzSb2u7oxKF9c63yJ19z0ub9TcSlpWB5Oalr418NkbgHXtftLJB613+VtrPE262ZEO/JwFPELqJreU1Efzy8DkFmJaTO7y12Dc91jd9WlK3bg3k7q3PUdaUeeRVrxmXf6OaTD9q4HFvXxPs+vnn7SlcCbpIMmzpP7db2wS/3tIeyTLST0bHgb+P/DJunpNu3718d29nbQ1+lT+Hm4FDm5Q7xj60eWv8r6T8/seaDL+tjz+J03GfyLH9FyO8QrgbxvU63X+gX8i/difJ20YHEFKRtUuf5uR+nPfRtqqXJXr/icwtYV5ra37tcdLpF4gd5H6SM/p4zdVi2MD4N9J/dOfyDEvJh2wn1n33pmkP4Knap/bynfSZP2sle1JOpnqofzZdwD/2GQ6R5KS9POkA5gfazTtXPfzpGaNP1P5PfVSf3NSX/El+T1LSL206n/HDd/f2++zvw/liZmZWQHcpm1mVhAnbTOzgjhpm5kVxEnbzKwgRXf5mzNnTlx22WWdDsPMrC+DdnH/ore0ly1b1nclM7MRpOikbWY22jhpm5kVxEnbzKwgTtpmZgVx0jYzK4iTtplZQZy0zcwK4qRtZlYQJ20zs4I4aZuZFcRJ28ysIE7aZmYFKfoqf7amnp4eurq6Wqo7bdo01lvP/9lmpXHSHkG6urqYe/zljJvY+03cV61cxmmHvovp06cPUWRmNlictEeYcROnMH7SFp0Ow8zaxPvHZmYFcdI2MyuIk7aZWUGctM3MCuKkbWZWECdtM7OCOGmbmRXESdvMrCBO2mZmBXHSNjMriJO2mVlBnLTNzAripG1mVhAnbTOzgjhpm5kVxEnbzKwgTtpmZgXxnWtGoejpobu7u+X6vp+k2fDhpD0KPff0co48bykTNp/aZ13fT9JseHHSHqXGTtjM95I0K5D3ec3MCuKkbWZWEDePWK/6c9DSByzN2s9J23rV6kFLH7A0GxpO2tYnH7Q0Gz68L2tmVhAnbTOzgjhpm5kVxEnbzKwgTtpmZgVx0jYzK4iTtplZQZy0zcwK4qRtZlYQJ20zs4I4aZuZFcRJ28ysIE7aZmYFcdI2MyuIk7aZWUGctM3MCtK2pC1pa0kLJN0t6S5Jn87lkyVdIen+/Dwpl0vS9yUtknS7pJ3bFZuZWanauaX9IvC5iNgR2AU4TNKOwFHAlRExE7gyDwO8G5iZH4cAJ7QxNjOzIrXtdmMR0Q1059dPS7oH2ArYG5idq50OXA18IZefEREB3CBpU0lT83RGrZ6eHrq6ulqq293dDdHmgMyso4bkHpGSZgBvAG4Etqwk4keBLfPrrYCHK29bksvWSNqSDiFtibPNNtu0L+hhoquri7nHX864iVP6rLvi4YVstMUMxg9BXGbWGW1P2pI2Bn4KHBERT0l6eVxEhKR+bRtGxDxgHsCsWbNGxXbluIlTWrqx7qqVy4YgGjPrpLb2HpH0ClLCPjsiLsjFSyVNzeOnAo/l8keArStvn57LzMwsa2fvEQGnAPdExHcqoy4GDsyvDwQuqpQfkHuR7AKsHO3t2WZm9drZPPIWYH/gDkm35bIvAd8Azpd0MPAgsE8edymwF7AIeBaY28bYzMyK1M7eI9cBajJ6jwb1AzisXfGYmY0EPiPSzKwgTtpmZgVx0jYzK4iTtplZQZy0zcwK4qRtZlYQJ20zs4I4aZuZFcRJ28ysIE7aZmYFcdI2MyuIk7aZWUGctM3MCuKkbWZWECdtM7OCOGmbmRXESdvMrCBO2mZmBXHSNjMriJO2mVlBnLTNzAripG1mVhAnbTOzgjhpm5kVxEnbzKwgTtpmZgVx0jYzK4iTtplZQZy0zcwK4qRtZlYQJ20zs4I4aZuZFcRJ28ysIE7aZmYFcdI2MyuIk7aZWUGctM3MCuKkbWZWECdtM7OCOGmbmRXESdvMrCBO2mZmBXHSNjMriJO2mVlBnLTNzAripG1mVhAnbTOzgjhpm5kVxEnbzKwgTtpmZgVx0jYzK4iTtplZQdqWtCWdKukxSXdWyo6R9Iik2/Jjr8q4L0paJOk+Se9qV1xmZiVr55b2fGBOg/LvRsRO+XEpgKQdgY8Ar83vOV7SmDbGZmZWpLYl7Yi4FljeYvW9gXMj4vmIeABYBLypXbGZmZVq/Q585uGSDgBuBj4XESuArYAbKnWW5DIrRPT00N3d3VLdadOmsd56fW8v9PT00NXVNajTNCvdUCftE4B/BSI/Hwt8rD8TkHQIcAjANttsM9jx2QA99/RyjjxvKRM2n9prvVUrl3Haoe9i+vTpfU6zq6uLucdfzriJUwZtmmalG9KkHRFLa68l/RC4JA8+AmxdqTo9lzWaxjxgHsCsWbOiPZHaQIydsBnjJ20xqNMcN3HKoE/TrGRDuj8pqboZ9n6g1rPkYuAjkjaUtB0wE/jtUMZmZlaCtm1pSzoHmA1MkbQE+AowW9JOpOaRxcA/AUTEXZLOB+4GXgQOi4iX2hWbdU5/2r67u7vTmmJmL2tb0o6IfRsUn9JL/a8BX2tXPDY8tNr2DbDi4YVstMUMxg9BXGal6ETvERvlWm37XrVy2RBEY1YW95EyMyuIk7aZWUGctM3MCuKkbWZWECdtM7OCOGmbmRXEXf46pNWLIfkEEzOrctLukFYvhuQTTMysykm7g1q5GJJPMDGzKrdpm5kVxEnbzKwgTtpmZgVx0jYzK0hLSVvSW1opMzOz9mp1S/sHLZaZmVkb9drlT9KuwN8Am0v6bGXUBGBMOwMzM7O19dVPewNg41xvk0r5U8AH2xWUmZk11mvSjohrgGskzY+IB4coJjMza6LVMyI3lDQPmFF9T0Ts3o6gzMyssVaT9o+BE4GTAd8l3cysQ1pN2i9GxAltjcTMzPrUape/n0s6VNJUSZNrj7ZGZmZma2l1S/vA/HxkpSyA7Qc3HDMz601LSTsitmt3IGZm1reWkrakAxqVR8QZgxuOmZn1ptXmkb+uvB4L7AHcAjhpm5kNoVabR/53dVjSpsC57QjIzMyaG+ilWZ8B3M5tZjbEWm3T/jmr7wk+BtgBOL9dQZmZWWOttml/u/L6ReDBiFjShnjMzKwXLTWP5AtH3Uu60t8k4IV2BmVmZo21eueafYDfAh8C9gFulORLs5qZDbFWm0eOBv46Ih4DkLQ58CvgJ+0KzMzM1tZq75H1agk7e6If7zUzs0HS6pb2ZZIuB87Jwx8GLm1PSGZm1kxf94h8FbBlRBwp6QPAW/Oo64Gz2x2cmZmtqa8t7f8EvggQERcAFwBI+ss87r1tjM3MzOr01S69ZUTcUV+Yy2a0JSIzM2uqry3tTXsZN24Q4zAbsOjpobu7u+X606ZNY731fBzdytRX0r5Z0ici4ofVQkkfB37XvrDMWvfc08s58rylTNh8ap91V61cxmmHvovp06cPQWRmg6+vpH0EcKGk/VidpGcBGwDvb2NcZv0ydsJmjJ+0RafDMGu7XpN2RCwF/kbSbsDrcvEvIuKqtkdmZmZrafV62guABW2OxczM+tDqyTVmI0J/Dlr6gKUNR07aNqq0etDSByxtuHLStlHHBy2tZN73MzMriJO2mVlBnLTNzAripG1mVhAnbTOzgjhpm5kVpG1JW9Kpkh6TdGelbLKkKyTdn58n5XJJ+r6kRZJul7Rzu+IyMytZO7e05wNz6sqOAq6MiJnAlXkY4N3AzPw4BDihjXGZmRWrbUk7Iq4FltcV7w2cnl+fDryvUn5GJDcAm0rq+zqbZmajzFC3aW8ZEbULPzwKbJlfbwU8XKm3JJetRdIhkm6WdPPjjz/evkjNzIahjh2IjIgAYgDvmxcRsyJi1uabb96GyMzMhq+hTtpLa80e+fmxXP4IsHWl3vRcZmZmFUOdtC8GDsyvDwQuqpQfkHuR7AKsrDSjmJlZ1rar/Ek6B5gNTJG0BPgK8A3gfEkHAw8C++TqlwJ7AYuAZ4G57YrLzKxkbUvaEbFvk1F7NKgbwGHtisXMbKTwGZFmZgVx0jYzK4jvXGPWQH/uJQm+n6QNHSdtswZavZck+H6SNrSctM2a8L0kbTjy/pyZWUGctM3MCuKkbWZWECdtM7OCOGmbmRXESdvMrCBO2mZmBXHSNjMriE+uGUQ9PT10dXW1VLe7u3sA9+0xs9HOSXsQdXV1Mff4yxk3cUqfdVc8vJCNtpjB+CGIy8xGDiftQTZu4pSWTn1etXLZEERjZiON27TNzAripG1mVhAnbTOzgjhpm5kVxEnbzKwgTtpmZgVx0jYzK4iTtplZQZy0zcwK4qRtZlYQJ20zs4I4aZuZFcRJ28ysIE7aZmYFcdI2MyuIk7aZWUGctM3MCuKkbWZWECdtM7OC+B6RZusoenro7u5uqe60adNYbz1vK9nAOWmbraPnnl7OkectZcLmU3utt2rlMk479F1Mnz59iCKzkchJ22wQjJ2wGeMnbdHpMGwU8H6amVlBnLTNzAripG1mVhAnbTOzgjhpm5kVxEnbzKwgTtpmZgVx0jYzK4iTtplZQZy0zcwK4tPYzYaILyxlg8FJ22yI+MJSNhictM2GkC8sZevK+19mZgVx0jYzK0hHmkckLQaeBl4CXoyIWZImA+cBM4DFwD4RsaIT8ZmZDVedbNPeLSKWVYaPAq6MiG9IOioPf6EzoZl1Tn96mYB7mow2w+lA5N7A7Pz6dOBqnLRtFGq1lwm4p8lo1KmkHcAvJQVwUkTMA7aMiNrmxaPAlo3eKOkQ4BCAbbbZZihiNRty7mVizXQqab81Ih6RtAVwhaR7qyMjInJCX0tO8PMAZs2a1bCOmdlI1ZGGsIh4JD8/BlwIvAlYKmkqQH5+rBOxmZkNZ0OetCVtJGmT2mvgncCdwMXAgbnagcBFQx2bmdlw14nmkS2BCyXVPv9HEXGZpJuA8yUdDDwI7NOB2MzMhrUhT9oR8Ufg9Q3KnwD2GOp4zMxK4s6dZmYFcdI2MyuIk7aZWUGctM3MCuKkbWZWECdtM7OCOGmbmRVkOF3lz8z6yTcLHn2ctM0K5psFjz5O2maF82VcRxfvK5mZFcRJ28ysIE7aZmYFcdI2MyuIk7aZWUGctM3MCuKkbWZWECdtM7OCOGmbmRXESdvMrCBO2mZmBXHSNjMriJO2mVlBfJU/M1tDT08PXV1dLdX1NbqHnpO2ma2hq6uLucdfzriJU3qt52t0d4aTttko0J873HR3dzNuwhRfo3uYctI2GwVavcMNwIqHF7LRFjMYPwRxWf85aZuNEq3e4WbVymVDEI0NlJO2mbWdD24OHidtM2s7H9wcPE7aZjYkxk30wc3B4H0QM7OCOGmbmRXESdvMrCBO2mZmBXHSNjMriJO2mVlBnLTNzAripG1mVhCfXGNmA9LfKwcSbQ5olHDSNrMBaceVA1v9I+jp6QFo+RolI+l6Jk7aZjZgg33lwFb/CFY8vJAxYzdp6Q/j2Scf498/sBNTp/Zdt4Tk7qRtZsNKK38Eq1YuY8zYCS3/YRx53u/6TPClXKzKSdvMRrxW9whKMLz3A8zMbA1O2mZmBXHSNjMriJO2mVlBnLTNzAripG1mVhAnbTOzNunp6WHJkiWDOk33025BT08PXV1dfdbz9RXMytWfa6m0ehp9d3c3X7rgdq7494PXOb4aJ+0WdHV1Mff4yxk3cUqv9Vq9voKZDT/9vZZKK6fR13LCYBp2SVvSHOB7wBjg5Ij4RodDAmDcxCktnVprZuXqz7VUWjmNvh05YVi1aUsaAxwHvBvYEdhX0o6djcrMbPgYblvabwIWRcQfASSdC+wN3N3RqGjtH/O5p5Yz5oU/8+yGGwxa3ZE2zZE2P+2Y5kibn3ZMs7T5GUzDLWlvBTxcGV4CvLlaQdIhwCF58HlJdw5RbMPBFGA0tcGMpvkdTfMKo2x+deyn7oyI1w3GtIZb0u5TRMwD5gFIujkiZnU4pCHj+R25RtO8wuic38Ga1rBq0wYeAbauDE/PZWZmxvBL2jcBMyVtJ2kD4CPAxR2Oycxs2BhWzSMR8aKkw4HLSV3+To2Iu3p5y7yhiWzY8PyOXKNpXsHzO2CK8Cl8ZmalGG7NI2Zm1gsnbTOzghSbtCXNkXSfpEWSjup0POtK0taSFki6W9Jdkj6dyydLukLS/fl5Ui6XpO/n+b9d0s6dnYOBkTRG0q2SLsnD20m6Mc/XefmANJI2zMOL8vgZHQ18ACRtKuknku6VdI+kXUfq8pX0mbwe3ynpHEljR9KylXSqpMeq54kMZFlKOjDXv1/Sga18dpFJe4Se7v4i8LmI2BHYBTgsz9NRwJURMRO4Mg9DmveZ+XEIcMLQhzwoPg3cUxn+JvDdiHgVsAKoXR7tYGBFLv9urlea7wGXRcRfAK8nzfeIW76StgL+GZiVTygZQ+oJNpKW7XxgTl1Zv5alpMnAV0gnEL4J+Eot0fcqIop7ALsCl1eGvwh8sdNxDfI8XgS8A7gPmJrLpgL35dcnAftW6r9cr5QHqR/+lcDuwCWASGfJrV+/nEk9inbNr9fP9dTpeejHvE4EHqiPeSQuX1af2Tw5L6tLgHeNtGULzADuHOiyBPYFTqqUr1Gv2aPILW0an+6+VYdiGXR59/ANwI3AlhFRu8jvo8CW+fVI+A7+E/g80JOHNwOejIgX83B1nl6e3zx+Za5fiu2Ax4HTcnPQyZI2YgQu34h4BPg28BDQTVpWv2PkLtua/i7LAS3jUpP2iCVpY+CnwBER8VR1XKS/4xHRR1PS3wGPRcTvOh3LEFkf2Bk4ISLeADzD6t1nYOQs37yLvzfpj2oasBFrNyWMaO1clqUm7RF5urukV5AS9tkRcUEuXippah4/FXgsl5f+HbwF+HtJi4FzSU0k3wM2lVQ76as6Ty/Pbx4/EXhiKANeR0uAJRFxYx7+CSmJj8TluyfwQEQ8HhF/Bi4gLe+Rumxr+rssB7SMS03aI+50d0kCTgHuiYjvVEZdDNSOKh9IauuulR+Qj0zvAqys7JoNexHxxYiYHhEzSMvvqojYD1gAfDBXq5/f2vfwwVy/mK3SiHgUeFjSa3LRHqRLDo/E5fsQsIuk8Xm9rs3riFy2Ff1dlpcD75Q0Ke+dvDOX9a7TjfnrcBBgL2Ah8Afg6E7HMwjz81bS7tTtwG35sRepbe9K4H7gV8DkXF+kHjR/AO4gHanv+HwMcN5nA5fk19sDvwUWAT8GNszlY/Pwojx++07HPYD53Am4OS/jnwGTRuryBb4K3AvcCZwJbDiSli1wDqm9/s+kvaiDB7IsgY/l+V4EzG3ls30au5lZQUptHjEzG5WctM3MCuKkbWZWECdtM7OCOGmbmRXESXsUkBQtPBbnuvMlLelwyEWR9F5Jd0h6Ln+Xm3Y6pr5IOkaSu44VaFjdbszaZte64QuB3wPHVMqeH7JoRpB8Bt/ZwH8DhwEvAE93NCgb0Zy0R4GIuKE6LOl5YFl9uQ3IVsAmwPkRcW2ng6mStGFE+M94hHHziDUk6Q2Sfi3p2XyB9k82qLOdpLMlPS7peUm3SXp/C9M+KDcj7JLf/5Skrnyh+LGVerNzvdlN3j+jUrZY0lmS9le6OcaqHP9MSRtJOknSE5KWSjq2cg2M3uKcKukMScvy/N0u6aOV8ccAi/PgKTmmq5tM6x/y+OmVsmNz2ccrZe/IZa+tlM2RdH2ep5WSflY5Hb5W52pJ1+WmmlvzH/OheVxtWT4n6RFJXyadpVcf46eVbs6wStIKSTe3sjxtaDlpWyMTgB8BZ5Gu1nYTcIKk3WoVJG1NunTs64HPAH8P3AL8VNLft/g5Z5JO7f0A6cLwh5GujT5QbyMlqi+Qrv3wv8gX4CI1WXyEdFfsz5IuRt+U0mVTryFdwP5LwPtIpyCfKan23pOBD+XX/0Zqhjq0ySSvIV2mYPdK2e7AqgZlSyPirhzHHOAXwJ+ADwOfAl4HXKd0s4GqVwPfB35Aun71lZKmAFcBU/J3chjpinsfq5vf/YBjSadn7wXsR7qo1eQm82Od0ulz+P0Y+gdp6/CsJuPmk5LLbpWyDUlXXZtXKTuFdH3ozerefwVwWx+ff1D+jK/WlV8CLKwMz871Zjd5/4y6eVoOTKyU/XOud3Ld+28BFvQR4+FNPvtXpKu3jcnDr8r1Dmrhe/89cFp+PZl0HfFjga5KnRuAcyvDN5OuZbF+pWw70jUvvlMpuzpPb6e6z/waqZ1960rZRqQbDUSl7L+AWzq9bvrR98Nb2tbIsxGxoDYQqV10IbBNpc4c4FJgpaT1aw/SVcpeL2lCC5/zi7rhO+o+o7+uj4iVleF783P9ldPuZc1LYjbyNuCRiLi6rvwsYHPSbe766yqgtrcyG3iSdHutqZJ2kLQJ8EbS1fBqW/s7A+fF6psHEBEPAL8B3l43/cURcVtd2a7ADRHx8sX2I+IZ4Od19W4CdpL0A0l7Sho/gPmzIeCkbY2saFD2POlqbDVbAAeQtviqj//I41u588jyBp+xYb8iXVN93C/0Uj6W3k0mXcWt3qOV8f21ANhW0vak5H1NRCwh3X5qN9Ifxfqk5A7pKoDqJY76GBrVmwosbVBeX3YGqenlzaQ/ueWSLlABN9kdbdx7xAbqCeDXNL8Ja9cgfMZz+XmDuvKhuBXVcuA1DcpfWRnfX9cCL5HarXcHTszlV+XhB0lb9/fn8hWkppdXsrZXNoihUb/rblbf9qpqjbJIbSQnASdVru18LHAeKZHbMOEtbRuoy4C/Au6KiJsbPAajq9mD+fl1deXvGYRp9+UaYLqkt9SV/yOpTfvu/k4wIp4EbiUdEN2R1VvUV5GaOvYgN43k+s+Q7q34IUljauWStgX+htSO3ZfrSTckeLk5KDe7vLeXOFdExHnA+az93VuHeUvbBur/ki5Yf62k/yIdCJxE+pFvHxEf6+W9LYmIbknXAF+UtIyULD9Kuph+u80HPg1cIOlo0oXu9wPeAfxTRLw0wOkuAI4k3R/zrlx2NWnvYQrplmtVXya1/V8i6XhgY9INBlaStoT78l1Sj5Zf5i6Kz+fPX1WtJGkeqYfN9aTv+dXA/sAv+zV31nbe0rYBiYiHgFmkHhFfJ/UaOYG0xXhVL2/tr4+SelR8n5RIHyJ1r2urvJX7dlLS+gbp1lGvB/aPiHnrMOkFdc9ExDLSQdg1yvO4y0h7FpuStnxPBO4B3hoRfTZB5WnvQeotcjrpDiqXAafWVf0N6SDo8aRleTTpoOuB2LDiO9eYmRXEW9pmZgVx0jYzK4iTtplZQZy0zcwK4qRtZlYQJ20zs4I4aZuZFcRJ28ysIP8DFGULbOOW9FsAAAAASUVORK5CYII=","text/plain":["<Figure size 360x360 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["plt.figure(figsize= (8, 8))\n","\n","sns.displot(df['count'])\n","\n","plt.xlim(0, 1000)\n","\n","plt.xlabel('The num of words ', fontsize = 16)\n","plt.title(\"The Number of Words Distribution\", fontsize = 18)\n","plt.show()\n","\n"]},{"cell_type":"markdown","metadata":{},"source":["# Bar plot for each of the new category"]},{"cell_type":"code","execution_count":41,"metadata":{},"outputs":[{"data":{"text/plain":["Index(['sport', 'business', 'politics', 'tech', 'entertainment'], dtype='object')"]},"execution_count":41,"metadata":{},"output_type":"execute_result"}],"source":["category_count = df['category'].value_counts()\n","\n","categories = category_count.index\n","\n","categories"]},{"cell_type":"code","execution_count":42,"metadata":{},"outputs":[{"data":{"text/plain":["sport            511\n","business         510\n","politics         417\n","tech             401\n","entertainment    386\n","Name: category, dtype: int64"]},"execution_count":42,"metadata":{},"output_type":"execute_result"}],"source":["category_count"]},{"cell_type":"code","execution_count":45,"metadata":{},"outputs":[{"data":{"text/plain":["Index(['sport', 'business', 'politics', 'tech', 'entertainment'], dtype='object')"]},"execution_count":45,"metadata":{},"output_type":"execute_result"}],"source":["category_count.index"]},{"cell_type":"code","execution_count":46,"metadata":{},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAtgAAAFZCAYAAACi11jlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABWyElEQVR4nO3dd3gc1dXH8e9RtST33m2CK2CqaaG30Duhhl4SAuSFUEMSQigJCSH0GnoJPWCKqcamG7ApNjbuvRdZVu/n/WNG9nq9klb2SivJv8/z7KOdO3dmzszOrs7evXPH3B0REREREUmMlGQHICIiIiLSmijBFhERERFJICXYIiIiIiIJpARbRERERCSBlGCLiIiIiCSQEmwRERERkQRSgi3SgpnZ/mbmZnZOsmNpLsxsnJnNS3Ycm8rMfmtm08ysLHxtByY7pqZmZueE+75/smORlv+eEkkGJdgizUiYVMT7GJjseCWxzOwA4H5gGvAb4ExgZR31x4Xnwhwzy4gx/8Zw/shGC7oVMbNOZnaDmX1jZnlmVm5mi8zsVTM7wcxsE9d7ub4Ei2xZ0pIdgIhs4Myo6X2Ai4BHgE+j5q0EBjZBTNJ0Dgn/nufuuQ1YbivgYuDuxIe0ZTCz3YBRQHfgDeA5IB/oAxwBvApcAjywCau/HJgHPLn5kSbFL4BN+nIhsqVSgi3SjLj7s5HTZpZGkGB/GT0vnN9UoUktzKyduxckaHU9ARqYXJcAc4E/mdnjCYxli2FmPYE3gTbAfu7+WVSVm83sUKBTkweXJGaWCmS6e7G7lyc7HpGWRl1ERFoJMzvXzKaEfXfnm9k1tdQbaWavmdmqsO50M/tjmMzHsx03syfNbE8z+9jMisxstZk9amZto+rG7LtpZgPD9dwYUbauP3nYD3m6mZWa2WQzOyqsM8LM3jWz/HCb95hZei1x/szMRpnZ2rD+a2b2sxj1zMwuNrOJZlZsZoVmNjbsrhEzZjM7JaxfAtwbxzE7zsw+D49VYfj82Oh1A+dGHGM3s3H1rRuoBv4AdAVivuYx4sk0s+vD86U07A7xppntFFWnxMyeilr24TC2u6PKXwyPc1o43dnM7jSz2eE2VofH7Op4Ygylhcd7fniuTjKzU6O2+4OZLTCzjf6fmdkvw1jPqmc7VxO0XF8bI7kGwN3fc/cXItZ9ipm9EW67LHw/vW5m20fF4MAAYD+rpYtXQ96TZnZiuM+l4bb/YmYHW4xrMcysq5ndb2YLLejusjCc7hJVr6bP+8Fm9mczmw2UAieH82t7Hw82s2fMbGm4/nlmdruZ5UTV62dmj0e8jivM7AszOzvmqyHSCqgFW6R1+A3QA3gMyAN+BfzDzBa5+39rKpnZkcD/gFnAHUAusCdwE7Aj8Ms4t7cj8BbwBPBfYH/gfIJk76LN3JdLCFoKHyX4J/874DUz+yXwH+B54HWCn60vA1YAt0StIwcYB3xFkHwOBn4L7GFmO7n7soi6zwCnAa+E+5MJnAF8YGYnuPsbUes+LozpQeAhgm4EtTKz37K+X/VNYfE5wOtm9mt3f4Sgu8+ZBMduH9Z3FVpe17pruPsbZvYZcIWZ3R+1f9HxpAPvAj8n2Pf7gA7AhcDnZravu09w9zIz+wI4IGoVBxG8zgdGrNMIzoFP3b0yLH4Z2JfgGE0CsoDhYb3b49kv4B8Er2VNt4xzgefNrI27PxmW/YfgS84hwHtRy58PrA1jqcuJQDnwVD31Il0KrCbovrUM2Jrg9fvczHZ295lhvTOBO4FVwK0Ry6+Ehr0nzewUgvN/NvBXoBI4Gzg6Ojgz6wB8AQwCHge+BXYi6Ep0oJntFuPXjn8B6QTHNB+YXtvOm9kuwEcEnzcPA4uBHQjeG3uZ2X7uXhF+SfiAoKvNA8AMgvNte4JzvSHHXKTlcHc99NCjmT4IEjEHzqll/v7h/CVAh4jybIJ/4F9GlLUhSAQ+AdKi1nNFuJ7944jJCRKs3aPK3wYqgLYRZeOAeTHWMTBcz40x9mVx1L5sH7HNE6LWMxFYGlU2Lqx/V1T58WH5QzHKLoqqmwZMIOh6YVExVwDD43z9OgGFBMlT+4jy9gRJUgHQMaL8yeBjOe7zYxxQGD7/eYz9uzEsGxnjtT40al3tgQXAuIiyP4Z1B4fT/cPpZ8K/PcLyEeH0leF0h3D6gc087+dHnQsdwrJcICss6wgUAy9FraMfUFVfDEC7cFuTGhhjToyy4UBZ9DYJ+l+Pi1E/7vdkeE4uJvjS1SmiXltgDlGfEwTJvAO/jVrvJWH5zTGO93Qgu5bzbF5U2Q8EXxrb1fI+Oyfq/XvNppwLeujRUh/qIiLSOjzh7mtrJty9GBhP0HJb4xCCVu4ngI7hz8ddzawrMDqs84s4t/elu38VVfYRQRIwcBPij/Rk1L5MImhNW+Lu/4uq+xnQ06K6poRui5xw99cIEojjIop/RZDkvh51PDoS9MkdyIbHEOBtd/8pzn05hKAF9h53X9fSHT6/hyA5OjjOddXJ3b8gaNk/38yG1FH1VwSJ0cSofc4gaGnc28yywrofhX8PjPhbxfrEvab8gKj6JQSJ5u62eaPdPBh1LqwlaBHvRPCFDHfPA14Cjo3q+nAuQTfIx+rZRvvwb52/RERz9yJY18WofXgMVxKcY7vHuZqGvCd3AXoTvD/WRMRRSHBMoh0fxvNIVPnDYfnxMZZ5MPzsqJOZjSBInP8LZEbF/RlQFBF3zet3gJl1r2/dIq2FEmyR1mFOjLLVQGTCMTz8+zjBP9jIx7RwXo/N3B5R29wUsda9hqA1OVZ5rG3meexuEj8BPSL6iA4naMFczsbH5MawTvQxmVFX8FG2Cv9OiTGvpmyjfuGb4Q8Eoz38vY46w4FhbLy/K4HzgFSC/twA3xB8AYlMsCe4+2xgclR5LvA9gAcXxV0ObAfMtaCv971mdlAD9yfWF5mp4d/I4/YIwReEM2Fdl5Vzge/dfWI926hJrNs1JDAz28nM3iI4PmtZfwxHEP/FkA15T9acS7G6bcQq2wqY7uu77AAQTs8g9nkX77ldE/dfY8S9guBLZY9we/MJWtN/ASy1oB/+P81s1zi3JdIiqQ+2SOtQFUedmiFHriZMhGJYkoDtRQ5t4rXUqeuzp7Z1x7vNhjCCpOD0Our8GDVdbwtfsrj7NDN7ArjAzGprRTWC5Pj3daxqZbi+SjP7lKD10QgS6afDOh8RtBqnAPsBH7n7utfb3R8ys1HAkeH8k4BLzexFd9/gQsXN5e5fmNmPBH2u7yLoJz6QoJ90fcsWmNl8YJiZZbl7SX3LmFl/gm4d+cDNBAluEWHXJIJfJuKRyPdkIsR7btfEfQdBf/5YIlvZ/2RmjxOcC/sAFwBXm9k/3f3aTQ1WpDlTgi2y5ai56KrI3T9som3mEvy0HS2RrbaxdDSznjFasYcDK2p+3ic4JkOA8eFP7YlW0xq/LTAmat42UXUS5S8EXxj+QdB3NtpMoBtBQlwdx/o+IhgH+iSCC9Vq9mMMQSv1CQRdaj6KXtDdlxJcrPqoBcO+PQOcZmZ3uPs3cWx7OMHY1JFqO27/Ae62YDzr8wkukH0ujm1AcJHhFQQt4NFdKmI5niCJPsbdx0bOCLuplEXVr+2LZkPek/PCv0NjzItVNgcYamZpka3Y4UWHQ9i8864m7qp4P0vcfQ7Bxaj3mlkbggtSrwnPhRWbEYtIs6QuIiJbjvcIfr69zsw6R880sywza9DP5HGYAbQLk56a7aQQJDON7brICTM7niAReT2i+GmCz8GYXSrMLN4uM7X5gKBl87LIYxs+v4zgAsgPNnMbG3D3JQQ3nNmPIDGO9jTBeNsxW7Bj7HNN4vxXgsTx83D6E4JfFf4aVQ8zyzaz7Ki4qghGEwHY6PyrxcXhaBg16+1AMGJOHvBxVN1nCJLqqwkS4FfD/tnx+CdBq/0/zWzPWBXM7Be2fojAml9TLKrOhYRjmUcpJPY+N+Q9OQFYCpxjZp0i6rQlOCbRXif4InVBVPmFYflrMZaJ13cEv+z8xmIPfZlWsz9m1sGihtJ091LWd//ZYsYWly2LWrBFthDuXmTBeMCvA9PDn2xnEbQ+DiNoiTye2K2em+oR4EqCYfbuJhgK7SQa/7NnFXCCmfUm2J+aYfqWs75vNe7+Stil4lIz25lg6MFVQF+CodIGsRmt7e6eZ8F45PcDX5nZk+Gsc8J1/zryIr4E+gfBkHGx+rneTXBx3e1mdiBBYpxPMELIQQRJauTQfN8T/BIxnGAkjFIILtQ0swkEF/QtjbrwcwjwsZm9RpCIrQmXv5igL330XUlrs4rguD0RTp8bxnlB9MV47r7GzF4huIgTgpbzuLj7MgvGWh8FfGZmr7O+C0hv4DBg7zB+gHcIulM8Y2b3hfu3F8EXmtlsfH6PJ7j49GaCxLIaeLMh78mwu85VBK3yX5vZYwTD9J1DcP3DVmzYUv5PgiH+7g/P7e8Ihuk7n6BLyz/jPT4xjpeb2ZkE586kMO4pBKMXDQrj/gPBqDgHAI+Y2avhdgsJftW6APjK3WsdClCkRUv2MCZ66KFH7Q/iH6Zvo/nUMuQbwYVnzxIM+VVOkHR+AfwZ6BxHTE4wkkFtse4fVX4EQZJWRtCf9B8ELcm1DdMXa1/mEXuYsxvDZQZGlI0L6/+MIGHKJ7gQbRQwqJZ9OpMg6csnSDDnEXQbOCWizsDomBvwOh4fHuOi8PEFcFy8r1kd6x1HOExfjHk1w7xtMExfOC+NYLzibyJimkmQvP0ixrpeDdfz56jymqHgnosq70Iw9vP3BK3NJQSJ411Arwac9wcTtJAvCM+fycDpdSy3T7jcTMLhFRv4OnUm6GIzgeDCxXJgEcEY6cdE1d2XYMSMgnAf3yZ4b41j4yHtuofHMJcguY4+Z+N+TxIkzZPC47EgjLdmaLyTo+p2Ixh7ehHB8JKLCL7sdY3nvRv9nopRPoBgBJN5YdyrCYbO/DvQL6yzVVjnJ4L3V1H4/CYihmDUQ4/W9qgZ31VERKRFC7sifQVc7+51jaTSqpjZlQQ3idnT3ccnOx4RQQm2iIi0Dmb2NHAq0N/ruJtlS2VmGQQXFlZFlLUlaNFuD/T2YIhEEUky9cEWEZEWKxzT/GiCkVp+BTzSGpPr0M+Ad8zsBYK+7L0IbpW+FXCxkmuR5kMt2CIi0mKFd4qcS3Dx3DsEF0A26K6MLUU4BOB9BBdUdie4yHEycKe7v5TM2ERkQ0qwRUREREQSSONgi4iIiIgkUKvrg921a1cfOHBgssMQERERkVZu4sSJq9y9W3R5q0uwBw4cyIQJE5IdhoiIiIi0cmY2P1a5uoiIiIiIiCSQEmwRERERkQRSgi0iIiIikkBKsEVEREREEkgJtoiIiIhIAinBFhERERFJICXYIiIiIiIJpARbRERERCSBlGCLiIiIiCSQEmwRERERkQRSgi0iIiIikkBpyQ6gOdjl6qeTHYI0wMTbz2qS7Sy4aUSTbEcSo/8Nk5MdgoiICKAWbBERERGRhGryBNvM5pnZZDP73swmhGWdzewDM5sZ/u0UlpuZ3WNms8xskpnt3NTxioiIiIg0RLJasA9w9x3dfWQ4fR0wxt0HA2PCaYDDgcHh4yLgwSaPVERERESkAZpLF5FjgafC508Bx0WUP+2B8UBHM+uVhPhEREREROKSjATbgffNbKKZXRSW9XD3peHzZUCP8HkfYGHEsovCMhERERGRZikZo4js7e6Lzaw78IGZTYuc6e5uZt6QFYaJ+kUA/fv3T1ykSdAxJ5NendqSkdY8flyoqKxmWV4RuYWlyQ5FREREpEVo8gTb3ReHf1eY2WvAbsByM+vl7kvDLiArwuqLgX4Ri/cNy6LX+QjwCMDIkSMblJw3F2kpxiE7DKRLuywWri6grKIy2SEB0D4rlR0GdqewtJz3vp9LeWV1skNq8Swjm8x+O5LWeSApGVlgluyQGpc71WVFVKyeS/nC7/HKsmRHJCIi0qiaNME2sxwgxd0Lwue/AG4C3gDOBm4L/44KF3kDuNTMXgB2B9ZGdCVpVX4+rA9llVU89+lUvBl+Rdh7WB/226YfH0yan+xQWjTLyCZnx+OoXD2f4qnvUV1aQNBrqvUySyElqwMZvbcjZ4djKPx+FFSVJzssERGRRtPULdg9gNcsaLFLA/7r7u+a2TfAS2Z2PjAfODmsPxo4ApgFFAPnNnG8TSI1xfhZj4689MW0ZplcA3w1cym/2ncbMtJS1Iq9GTJ6b0vlmkWUzv482aE0GQeqClZQMv0jsrc5lIwegylfMiXZYYmIiDSaJk2w3X0OsEOM8tXAQTHKHbikCUJLqnZtMigtr6S4rHl0C4mloqqavKIyOua0YcXa4mSH02KldR5A6azPkh1G0pSvmElGz+FKsEVEpFVrHlfSbeFSUoyq6mbadB2hsrqa1JRW3l+4kaWkZ1FdVpTsMJLGy4qw9DbJDkNERKRRKcGWdY4euTWvX3tcssPYAmz8Zar3794le8RRCd9S+70voNup9yZ8vZvOW/9FnSIissVLxjB90gylqWW6Vcr/7NFkhyAiIrLFUYLdzJ261zBO33c4HbPbUFRWwVsTZvO/r2bw1vUncvPLX3DW/tvRpW0bJsxZzi0vf8maomC86g7Zmfz+mJHsMaQ3AOOnL+GON74hvyQYveHNP5zAG9/MYuTWPdmmXxceHzOZCw/ZgfTUFD695TQALn/8IybOWZ6cHRcRERFpoZRgN2P9u7bjsiN25sx73mbO8rW0bZPOwO4d1s0/cpetufDBdyktr+Kvp+7FLafvzSX/+RCAW07fm8qqak66PRjx8JbT9ubm0/bm/x7/aN3yx+0+mN8/MZbpS3LJTEtlVUEJ5x80guP+8XqT7qcE0jr1pfs5T5HRcxgVq+ayZvTNlC+ZQudjboHqSnLfunFd3d6/e5e8sfdRPPktUjv0pvNRN5DZZwQ4VOYtYtX/rqVy9Tw67Hcxmf12ZsWzF65brvDbV2iz1e5k9NmeyrzF5L71V8oX/bBu3Tk7nUi73c8grX1PKtcsIm/MnZTO+RKA9J7D6HTYH8joPhj3aipXzWXF85fgpflkb3sYHfa9mNT2PfCKUkpmf0buqD816TEUERFpDpRgN2NV1Q4GW/foyNI1RRSWVvDjglX06pQDwCMf/MDqgqDF+q63JjLquuPp2j4LgJ8P7cNx/3iNgrDF+t9vTuB/1xxH13ZZrCooAeD1r2YyfUkuAGWVVU29exKl7S6/ZOULl1GxfAbt9zybbqc9wJJ7j6h3uY4H/h9Va5ey6IXLoLqK9G6DqC7Jr7V+zo7Hs+rF31Gxai4dD7mKLsfeytL7g/7fOTudSPu9zmPVy1dQsXwmbQbtTdeT72LZwydRuWYhnQ//IyWzP2fFU+eCpZDRaxuoqsDS2tDluL+x4rnfUDbvayw9i4xewxN2bERERFoSXeTYjC3OLeRP//2M43YfzLt/PonHfnsYewzptW7+0jWFGz3v0SGbnh2CBHxJ7vr5i1YXBPM7Zq8rWxKxvCRf0XevUbF0KlRXkv/5Y3hlGVlD9qt3Oa+qILVtV9I69QWvpmLFDKqLc2utXzjxZSpWzgavpui7V0nvMgDLbAtAu93PIP+Th6hYPgNwSmd9Stm8r8ne7vB120pr35PU9j2hupLyxZPwiuALm1dXkt51K1LatMcrSihb8O3mHxQREZEWSC3YzdzYHxcw9scFpKWmcNIeQ7jjnAP41V1vA9CrU1sWrS5c9xxgecQY1cH8ILHu0zmcn7d+fnXUXW2ip6VpVeYt3nB67VJS2/eod7m8D++gwz6/ptup95GSnkXxTx+QN+audYlvtKrCVeueV5cHdVIycqgqKyStYx86Hf5HOh123foFUtKozA/64q9+48902OfX9Dj3aaiqoGjy26z9+EG8spSV//0t7fY4i44H/I7KNYvIH/8UxT+ObuhhEBERafGUYDdjA7q1p3fntnw3ZzmlFVUUlpaDg4eJ8AUHb8/sZXmUVVTxuyN3ZvyMJazKDxKmL6cv4YqjR/KXFz7DzLji6JF89tOidd1DYlldUEKntm3IyUynqKyiSfZR1kvr2GfD6Q69qMpfTlr7nqRkd1w/w1JJyem8brK6eA1r3rsN3ruN1I596XbKPbTf6zzWjru/wTFUrV1K3rgHKPnp/djz8xaT++YNAKR3H0z3Mx6mMm8RRd+/Ttn8CZTNnwCWQtaQ/el68p2UL55E5ZpFDY5DRESkJVOC3Yylp6Zw4cHbs3WPjgAsXF3A1U+PW9df+p1v5/Dobw+jS9s2fDt3OTe8sP4OgX96/lOuPGZX/nfNcQCMn7GEf78xoc7tTZi1jK9mLuWNP5xAaorx+yfH8q1GEWkyOTseR/G0MVSsmEm7Pc7E0ttQMvMTLDWdjgdfQWrHPlQVrKDj/pdiKevfutnbHErZkh+pyluMlxVAVQVevWl96vPHP0OH/S6mMnc+FcunY2mZZPTahqriPCpXzyVn+2MonfMlVYUrqS7ND7ZTXU1KThcy++1E6dzxeFkh1WXBLydeXZ2QYyMiItKSKMFuxmYty+O8+9/dqLzmIscvpi/h9a9nxVw2r6iMPz9f+y25j/77/zYqq6x2rnn6402MVjZX4bev0Omw68joOYzKVfNY+fwleFkhRZPfJrPfTvS68CWqK0rI/+xRqgpWrFsuvddwOh5yJSlZHfDyYkpmjKPgiyc3KYai716Fqgq6HHMzaZ364FWVlC/7ibwP7gAgc6vd6HjQ5VhmDtWlBRRPfpuiSW+S0rYL7XY9lS5H/xVSUqnKX8bq1/9I1doliTg0IiIiLYoSbJFmYMk9hwGQ/8lDG88Mh+iLHKavcMIL656vHXMXa8fcFXO9az9+MOZ2alStXcKCm0ZsUFY06Q2KJr0Rc321DbtXXbiKFc9cEHOeiIjIlkajiIiIiIiIJJBasFugpWuK2OXqp5MdhoiIiIjEoBZsEREREZEEUoItIiIiIpJA6iLSwtx4ys85fKefUR5xa/N73v6Wl7+cDsDgXp247IidGdqnM13bZXH+/e/y/bwVG6zjzyftyYgB3RjQrT1vTZjNza982aT7ICIiItKaKcFugepKiiuqqhk7eQEPvvc9z/7fkTHrzFy6hg8nzeeEPQY3ZpjShDofcws5I47Aq9bfICjvw39TOOFFANJ7DKHjQVeQ0XMYqW27svyJsyhb+N0G60jr1I/OR95ARt/tqS4toGD80xSMV19/ERGRhlKC3crMW7GWeSvW1lnnhc+nAXDoTgObICJpKkU/vLHBUH6RvKqC4p8+ZO3Ye+l54YsbV7AUup16H6Vzx7PyhctI67oV3c94kKr85RRPfa9xAxcREWlllGC3QAeO6M8BI/qTV1TGx1MW8sgHP1BSXpnssKQZq1w1l8pVc2udnzlgF1I79iJvzF14ZSkVy36icOIrtB15shJsERGRBlKC3cK88Nk07nn7W9YUlbJV9w785eS9+NNJe/LH/36a7NAkybKHH0LW8IOpLl5DyfSxrP34QbyiJK5lM3oMpXL1/A3qly+dSttdT22scEVERFotjSLSwkxbnEtuYSnuMGf5Wv795jccvP0A0lP1Um7JCr75L0seOIbFt+/DqpcuJ3PASDoffWPcy1tGDtVlhRuUVZcVkJKZk+BIRUREWj9lZS1ctTsAZkkORJKqYulUqotWA07Fytmsef+fZA8/BFLT41rey4tIyWy7QVlKZjuqy4oaIVoREZHWTQl2C/OLHQbStk2QNPXr2o4rjhrJx1MXUl5Zva5ORloKGWnBS5ueGjxPicjA0yLKUlKMjLQU0tQC3rqEX7wgvm9e5cunk9ZlAJaeta4so9dwKpZPb4TgREREWjf1wW5hTtxzCNedsDsZaSnkFpYy9seFPPL+D+vm9+qUw1vXn7hu+qHf/AKAG1/8nDcnzAbg/gsPZuTWPdfVOWbXQUyYvYxfP/R+E+2FJFr2todRMutzvKyAtM796XTIVZTMGAdV5esrpWZEPE8Ppqsrwaspmz+RqryldDzwd+SNuYu0LgNpu/MvWfPebU2+LyIiIi2dEuwWpr4keOmaIna5uu6xi5VItz5tdzmZTkf8CUtNp7ool+LpH7F23APr5qd26E2f/1s/GkiPsx4DYPWoP1H0wyjwala+cCmdj7qBPld/SnVpAflfPkHxlHebfF9ERERaOiXYIq3AiqfPq3N+1dolLLhpRJ11KtcsZMUzFyYyLBERkS2SOt6KiIiIiCSQEuxmoKraW8RFhulpKVRWVddfUergYM3/tW48FnEBpoiISOu0Jf+nbzbyS8pIS02hXVZG/ZWTpE16Kh2yMllTVJbsUFq06vJiUtq0T3YYSZOS1R4v19B/IiLSuinBbgbcYcaSXPbdpu+64fWak7QUY59t+jF7eZ5asDdT5ep5ZPQcluwwksTI6DmMitXzkh2IiIhIo9JFjs3E17OWsvewPpyxzzasyC+mrKIKT/JP6WZGeloKPTvkMH9VPp9PW5zUeFqDssU/0naHY8ka/gsqlk+jurSg9XeZsBRSsjqQ0XtbwKhYMSvZEYmIiDQqJdjNhDt8+tNivp65jO4ds8lITU12SABUVFUxdu1CSisqkx1K61BVTuEPo8joNZzMfjthGdnEezOYlsvxskIqVs2lfNm0YOxtERGRVkwJdjNTVlnFwlUFyQ5DGlNVOeWLfqB80Q/11xUREZEWZ5M7/JrZMDM7zsx6JzIgEREREZGWLK4E28weNrOHIqZPASYD/wOmmdnPGyk+EREREZEWJd4W7MOATyKmbwaeB3oD74XTIiIiIiJbvHj7YHcHFgKY2WBgEHCCuy8zs0eAFxspPhGRZmWve/dKdggSp88v+zzZIYjIFireFuxcoEf4/GBgmbv/GE4b0DyGvBARERERSbJ4W7DfAW4ysx7ANcBLEfO2A+YlOC4RERERkRYp3hbsK4HxwG8I+mLfEDHveODdBMclIiIiItIixdWC7e5rgfNqmbdPQzdqZqnABGCxux9lZlsBLwBdgInAme5ebmaZwNPALsBq4BR3n9fQ7YmIiIiINJV4h+m72cwOMbO2Cdru/wE/RUz/A7jT3QcBa4Dzw/LzgTVh+Z1hPRERERGRZivePtjHAdcD1Wb2A/BpzcPdVzZkg2bWFzgSuBX4vZkZcCBweljlKeBG4EHg2PA5wCvAfWZm7u4N2aaISEuUlpLGkG5D6N+pP9np2QQfly1PVXUV+aX5zFk9h3lr5iU7HBGRRhdvF5ERZtYJ2CficQmQamYzCRLtC+Pc5l0EF0q2C6e7AHnuXhlOLwL6hM/7EA4P6O6VZrY2rL8qzm2JiLRIaSlpHDr0UMoqy5i6bCr5ZfnQQpsWUlJS6JLdhR377EiPdj34asFXyQ5JRKRRxduCjbuvAd4A3jCzDILh+q4B9gUGA/Um2GZ2FLDC3Sea2f6bEnAt670IuAigf//+iVqtiEjSDO42mPKqcj6c+WGyQ0mI3OJc5q+Zz/Ejjmf6yunkleQlO6SE+njf/ZIdgsRpv08+TnYIsgWIK8E2s/bAXqxvvR4JFAKfA1cTdBeJx17AMWZ2BNAGaA/cDXQ0s7SwFbsvsDisvxjoBywyszSgA8HFjhtw90eARwBGjhzZQtt4RETWG9BpAFOXT012GAlVXlXOvNx5DOg0oNUl2CIikRpyo5lXgIHAs8BId+/m7se5+x3u/nU8K3H3P7h7X3cfCJwKfOTuZwBjgZPCamcDo8Lnb4TThPM/Uv9rEdkSZKVnUVBakOwwEq6grIDs9OxkhyEi0qjiTbC/IWjtPhj4BXCIme1kibvi5lqCCx5nEfSxfiwsfwzoEpb/HrguQdsTEWnWDCPR7Qnn7XYedx1317rpfx3zL07f+fTaF4izTkNUe3Vw/18RkVYs3osc9zSzLGAPgj7XRwG3AJVm9gXwsbs3aAg9dx8HjAufzwF2i1GnFPhlQ9YrIiLxueqNqzaY/vyyz7n4lYuZtHRSrXVERKR+DbnIsYSgK8fYsE/2AcAVwGHAoWiMahERERGRuC9y7MmGQ/RtF86aAtxP/Bc5iojIZnjl7Fd4e+rb7Np/VwZ3HcyCNQu4fdztTFsxjVRL5ayRZ3H48MNpl9mOGStncNcndzE3d27Mdd17/L1MWDiBpyY8xZOnPQnAncfeSbVXM2bmGG776LYN6gD0bNeTS/a6hO17b09mWiZzV8/l2revJb80n4v2uIgjhx9JdkY2a0vX8sJ3L/DKpFea6tBIBEtLI3vAALL69iW1TZsm375XV1Oxdi1Fc+dSvrJBt8sQaRXibcFeApQD3wLvAX8CPnf3vEaKS0REanHcdsdxzVvXMHv1bE7d6VT+dcy/OPmpkzlx+xM5bNhhXPXmVSxdu5QzR57JXcfdxWnPnEZxRXGd6zzn+XP4/LLPuWLUFRt0EYmUmZbJPcffw/j54zn92dMprShlWPdhVFZVslu/3Th8+OFc+PKFrChcQcesjnTL6dYYuy/1sIwMeh56KFWlpRTNnUtVcTE08fgAlppKRpcudD/gAPKnTCF/ypQm3X5zcd+VbyY7BInTpXccndD1xZtgHwh8FXYTERGRJHpr6ltMXzkdgGcnPsvxI45nr6324ojhR/Dst8+yYM0CAJ74+gmO3vZofj7w5wkZT3uvgXuRmZbJ3Z/cTZVXATBleZA4VVRXkJGawVadtyKvJG/dQ5pehxEjKM/LY/Wnyf1xuWTRIgpnzaL3McdQNH8+VYWFSY1HpCnFNYqIu49z9xIL9DOzn5tZTmMHJyIiG1tasHSD6eUFy+nWths92vVgaf76eY6zNH8p3dt1T8h2e7bvyZK1S9Yl15G+W/wdD3/5MGfvejZvXfAWdx57J8O6D0vIdqVhsgcMoGBq82gxrioqonjBfLJ1EzjZwsQ7TB9m9luCG7/MJ+hzPTQs/5+ZXd4o0YmIyEZ6teu1wXSPdj1YWbiS5QXLN5hnGL3a92JFwYq41lvt1XXOX5a/jN7te5Nisf91vDHlDX776m85+rGjmblyJrcecWtc25XESmubQ0Xe2mSHsU7F2rWktW2b7DBEmlRcCbaZXQ38G/gPQXeRyFFMxwGnJDwyERGJ6chtjmRItyGkpqRy+s6n0yatDV/M+4J3pr3D6TufTr+O/UhLSePsXc8m1VL5Yt4Xca03tyiXvh371jr/i3lfUFFdwe/2+R05GTmkWirb9tiW7PRshvcYzg69dyA9JZ2KqgqKK4qprq47YZfGYZaS8DHUo/U47DB2++9zcdX1qmoSd9sMkZYh3j7YlwA3uPs/zSw1at50YEhiwxIRkdqMmjKKy/e9PBhFJG8BV795NUXlRTz37XOkp6Zz57F3kpORw6xVs7hi1BX1XuBY4+HxD3PB7hdw2d6X8dGsj7h97O0bzC+tLOV3r/2OS/e+lBfPfJG01DTmrJ7DtW9dS1Z6FpfudSn9OvajyquYs3oON7x3Q2PsvmyiHe6+izUTJ7Lg6WeSHYpIqxdvgt0TmFjLvGqg6ccAEhHZQi1eu5gnvn5io/Kq6ioe++oxHvvqsRhLweNfP77B9GWvXbbB9OifRjP6p9F11lmSv4TrR1+/0bq/XfQt5714Xlzxi2zJUtNSSEuPbquUplJd7VSUV0IjD6wTb4I9C9gPGBNj3r7A1IRFJCIiIgk16PL/o8P229N+223pf/rplK1axTe/OpOeRx1F35NOJLN7d0qWLGXuww+x5psJ65bruu8+9P/Vr8jq04fq8nKWvj2aeY8+um5+nxNPpN9pp5KSlcXKsWOZ+e87QV2DYmrfJZu+g7qSlZNBZUVVY+d3UovU1KC70qolBSyaubLRRrCMN8G+C3jAzMqBmrsGdDez84HfAxc2QmwiIiKSALPuupucrbbaoItIz6OOov/ppzHlzzdQNGcOnXffjW1vvpkJ519A6eLFdN59d4Zefz0//fUmcr/+mtTMTHK23nrdOjN79iS9Uye+Ou10Mrt3Z+eHH2LtDz+w4oPNHxKytWnXKYutR/Ri7tTlrF1Z2NTDkkuUzKx0Bgzrzs+268XsyUvrX2ATxDtM36PAH4FrCe7eCDAauBu40d3/2yjRiYjIBk566iTen/5+ssOQVqDvSScy/6mnKZo9G9zJHf8Ved99T/eDDgSg94knsHTUKHK//BKqqqgqLiZ/8uR1y1eXlTHv8cfxigpKFy8mb+K3tBs6NFm706z1HNCJhTNXkrdCyXVzUFZSwcwfltC+azYZbeJta26YuNfq7reb2UPAnkBXIBf40t2bz1hAIiIiEpc2vXox6IrLGfS79f3sLTWVsvDW5m169mTVJ7XfrKZizZoNuoNUlZaQmp3deAG3YO07ZzN36vJkhyERvNrJX11M+87ZrFqSn/D1Nyhtd/cCQE0nIiIiLUz00H2ly5cz7/EnWDVuXMz6pcuWkd239mEbJX6paSlUlm98g6aWbJeDBtFzQCfefvybZIeyySrLq0hNi/uWMA1Sa4JtZvs2ZEXu/snmhyMiIolmGA+e9CAjeo3guMePY2XRSgZ1HcRvfv4bhnQdQpecLlz8ysVMWjpp3TI79N6Bfx39rw3Wk5GWwbzceZz9/NlNvQuSAOWrc8nq02fd9KKXXmbguedQsmgRRbNmkZKRQduhQ6lYu5aSBQtY8r/XGH7jX8j77jtyJ0xY1wc7spuIJM+wXfsy8uAhPPv3jzZ7XQeduiPV1dWMfWlS/ZVDE8fM2uztNrZEHqOGqqsFexzBICZ1jQ7vEX8bpxOLiIhsllN2OoXSytINyiqqKvh49sc8Ov5RHjtl42H9fljyA4c8fMi6acN45exXeG/6e40erzSOxS+/zNDrrmWvt9+ibNUqJpx9Dl5ZwdDrriOrV0+qq6oonDGDOQ88CEDu+PHM+Mc/GXjhhQy/8S9UlZay7K23lGC3ImaNPlrdFquupHhEPcv2Ba4BDgDWJCwiERFJmH4d+3HCiBP44+g/8uRpT64rn79mPvPXzI97PXsO3JPOOZ0ZPXV0/ZWlWSqYNo0J55y7Qdnyd99j+bu1f2laOXYsK8eO3ah8+bvvsvzddzcom/732xIT6BYkLT2V3Q8bytYjepGRlcbyBXl88r/JrF1dzPEX78mKRWtp3zmbfkO6UVJYxmdvTGHulOX0HNCJ/U/cntTUFC762+EAvP3Y1yyevZrOPdux9zHb0K1PByorqpnx7SK+enc61dVOu05ZnP2ngxnz4vfstN/WdOiazYQPZzJk5+CXjcE7Bn//88d36NyzPfsevx2de7bDDJbPz+Pj1yaTvzq4cdVuvxhCr606M+rh8QCc9ceDmDJ+Pn0Hd6VH/04U5BYz9pVJLJsXpIgHnbojZkZ1dTVbj+hFRXkln785lTXLCznglzvQqXtbVizK44PnvqUov6ze4wNs8jFqkte2thnuPiVWuZkNBK4DzgbyCEYWeaARYhMRkc1gGH846A/c99l9FJQVbNa6jtvuOMbNGkdeaV5ighMRDjh5ezLbpPPyPZ9SVlLByIMHc9QFu/P87eMAGLZrP95+/GveeXoCO+zzMw4+bSee+OsHLJu/hnGvTtqo+0NW2wxO+O3P+XL0NN567GuycjI58rxdqayo4psPZq6rN2TnPrz+0JeUFpfj1U77Ljkxuog4X783naXz1pCWnsKBJ+/AL07fiVfu/bzW/Rm+W39GP/41a1YUstfR23LwqTvy7G3rv6AN2qEX7zw1gY9e+oFt9xjAAb/cgcWzVjH6yW8oK67gqPN3Y7dDhzL25Un1Hp/qat+kY9RU4u7ZbWZDzewpYAZwOHA1MNDdb3f3osYKUERENs3JO55MbnEun8zZvEtkerTtwR4D9mDUj6MSFJmItMnJYOjOfRn36mRKCsuprnK+fn8G2e0y6TGgEwAzv18ctAA7TBk/n8ysdDp0zal1ncNG9mXVknymjJ9PdZVTlF/KxDGzGDqy3wb1vnl/BsUFZVRXea3DBq5eWsDi2auprqqmvLSSr9+fQc+Bneu8C+WUL+eTuzwYinDqV/Pp2K3tBsPgLZq5ivk/rQCH6RMWkZGZxrQJiyhaW0plRRWzJi2he7+OcR+fTTlGTaXeftNmtj3wJ+BEYA5wMfC0u1c0cmwiIrKJ+nTow6k7ncr5L56/2es6etujWZC3gO+XfL/5gYkIAO07ZwFw2lX7bVCekppC245tACgOu0oA60YhqWvc5nads+m1VScuvOWwDcotZcPL6fJzS+qPr0s2ex21DT0GdCQjM21dIp7VNoOCNbGXL8pff61HRRhvemYa5aWVwfyC9fMrK4L5xQVlG5SlZwb7F8/xgYYfo6ZS1ygiuxMk1kcQ3Ar9TOAFd9c9UEVEmrkdeu1Ax6yOPHN6cNe+FAt+sHz69Kd5ZPwjvDb5tbjWk2qpHLXNUTz37XONFqvIlqgmSX3m7x9RWlS+0fzt9hhQ5/KxsrGCNSUsnLGKtx77uu5lo5qto6cB9j9pe4rXlvLCvz6mtLiCzj3bcfrV+9e53kSq7/jEI5kZa10p/pcEF5eOA/4HdAR+YxZzUBF39wcTHZyIiGyaMbPG8M3C9ePTdm/bnUdOfoQrRl2x7uLGjNSMdfPTUtPISM2gsrqS6oj/SntttRftMtvxzk/vNF3wIluAksJypn+7iP1PHMGnr0+hKL+UjDZp9B3UlYUzVta7fHFBGdltM0jPTKOiLGghnj5hETvttzXDd+vHjG8XU1VVTftO2XTslsOC6bWvszi/jJ4DOgXjxoW5dkZmGmvLqygrqaBNTga7H9q0d+ms7/hUxDGueKxj1FTqa0M3glFCDqinngNKsEVEmomyyjJWVq7/h5qaEvSbXF20mpKKEnq268mr57y6bv69x98LwK0f3MroaetHCjl2u2MZM3MMheWFTRS5NBkzdrz/Pjpstx1fnngS5eEdHHsceigDzjmbjC5dKJozh5n/vpPCGTMASMnIYNgf/0jO4EFk9e7NvMcfZ8HTzyRzL1q0sS9NYpeDBnH8b/cku10bykorWDpnNQviSLAXz1rFwhmrOPuPB2EpxtuPf8OSOat57cEv+PmRw9nj8GGkpadSsKaYH7+se8SgqV8toO/grlxw06GYGY/++V0+e2MKB5y0PRfdejgFeSV8N242W2/fK1G7HpfNOT5Q+zFqChbrZ4GWbOTIkT5hwoQGLbPL1U83UjTSGCbeflaTbGfBTfWNVCnNSf8bmmZs3r3u3atJtnPCiBP4aOZHrW7UjmHdh9EpuxNfzvuy0bf1+WW1j3aQaB/vu1/9lZrIgLPPZv4zz2xwG/NY+p5yMp13351OI0euS7DbjxjB9v+6nSl/+hN53/9A35NOou8pJ/P16WdQVVyMZWTQ+9hjKZo1i60uupDVX35Zb4Ldbvhw0tu3J/errxK5m5tsv08+brJt3Xflm0AwpN3X789osu1KfAYM605pcTnLF+Rx6R1Hb9I6zGyiu4+MLm+c+0OKiIhIs5XVty+9jzuO2Q9s+ONzr6OPYtUnn7Lmmwl4RQULn3+e6ooKuu67DwBeXs7il18m77vvqC7ftH6xIlsCJdgiIs1VXffRbaFquY5HmpIZQ6+7ltkPPEhl4YZdf9puvTUFM6ZvUFY4cyZttx7UlBGKtHhKsEVEmqHSylJyMpI/lmui5WTkUFZRVn9FaTR9TjqJ8txcVn/66UbzUrOzqSzc8NYWlYWFpOZkN1V4Iq2CEmwRkWZoUd4itu6ydbLDSKhUS2Vgp4EsyFuQ7FC2WG369KHfKScz8667Y86vKi4mre2GX+zS2ralqqi4KcITaTWSPxK3iIhsZPrK6Ry1zVHsOXBPpq+YTkFpAdW0zNsQpFoqXXO6sn3v7VldvJpVRauSHdIWq8OIEaR37MiuTz4RFKQE7Wwjn3iceY8+RuHs2bQdPGSDZdoOGsSqTzbvbqAiW5q6bjRTBezp7l+b2ePAze4+t+lCExHZcpVVljH6p9GM6DmCAwcdSHZGy/2J3t3JK81j7uq5TF0+NdnhbNFWjh3LmokT101nduvGzg89yKQrr6J4wQIKZ89m+9v/yfJ3d2btpEn0OelEUjIyWPXJ+u4klp4OZpCSgqWmYhkZUFWFV9U/LrE0X3scPowhO/WhTU46lRXVLJmzms/emEphXglmsMcRwxmycx8y26RTsKaYr9+fwexJS9ctn5aRyp5HDGPQDr1Jz0yjMK+E95/7llWL85O4V8lTVwt2OVBzF4JzgIcAJdgiIk2kpKKErxd+zdcL674rm0i8qsvK1o13DWCpwfjo5bm5VJeUkD95MjPvvIsh11y9bhzsyddcS1Xx+i4iuz37DG16BeMhd9xhBwaeey7L3nmH6X+/rWl3RhJq2sRFfDt2FuWllaSlp7LH4UM59MydefXezxmx11YM26Uvrz34BXkri9hqu54cduYuPL/sY/JWBBfKHnHurlRVVPHy3Z9SmFdK+87ZVJQ37c1dmpO6EuypwI1m9no4fZKZbTTOX0h3chQREWlhypYt22gM7+Xvvcfy996rdZmvTjm1scOSJKhJlAEwcIdO3doC0KFrNotnryJvZXAB7Nwfl1FaXE6Xnu3IW1FI/6Hd6DWwM0/e9AFlJRUA5Odu2f3260qwLwMeBu4kuFPjVXXU1Z0cRURERFqwITv1Yb8TR5CZlU5VVTWfvzEFgKnjF3DIGTvTqUdb8lYU8rMRvUhJsXV3RewzqAv5ucXsfthQBu3Qm4qySmZ+v4Sv35tOdXXruqFhvGpNsN39C2AEgJlVA3u4u36nFBERkfhp7PMWY8Z3i5nx3WKy22UyfLf+rF5aAMDa3GKWzF3N6Vftj7tTVVnNB89/R0lhcLOhNjkZdOnZjgXTVvDULR/StmMWR1+wGxXlVUwcMzOZu5Q08Q7TdwBBlxERERFpxqrKykjNykp2GOuktmmjuz62MMUFZUz9aj5Hnr8bmVnp7H/CCLr16cDTfxvDA9e+zahHxrP/idvTb0g3ACpKq6iuqubL0dOoqqxm7aoiJn8+j59t1yPJe5I8cSXY7v6xuxea2e5mdqWZ3Rr+3b2xAxQREZH4lS5dQvaAAckOI2BG9oD+lCxenOxIpIFSUoyMzDRyOrShW98OTJ+wiII1JeCwbN4als7NZcDw7gCsWrI25jp8y+wdAsSZYJtZjpmNBr4A/g6cF/79wszeNrOWO36UiIhIK5L/4xQ6bL89bYcMCYbUS5L0Dh3out9+VBWXULZiRdLikDgYjNhrIFltg8Hjcjq0Yd8TRpC/upg1KwpZOm8NQ3bpS077NgD06N+RPlt3YeWiILGePXkpJUXl7H7YEFJSjXads9ju5wOZM3lprZts7eK90cw/gT2BU4FX3b3azFKAEwkuhPwHwUWRIiIikkTlq1ez4sMP6bDDDnTeY3e8qppgLIKmYykpVJeVUzR3TsxbskvzM2B4d3Y9ZAjpGamUlVawePZqXn/4S7za+fzNqex11HB+efneZGSmU1xYxncfz2b6xEUAVJRV8cYj49nvhBFcePNhlBSVM+2bhXw3bnaS9yp54k2wTwSudfeXawrcvRp42cw6ATehBFtERKRZKF+1ipVjxoAZlp5OU19m6NXVeOWWOwZyi+Pw1qO1j2NRUVbJuFcnw6uTa62zemkB/7v/i8aIrkWKN8HuACysZd5CoH1iwhEREZGEccfLy5u4/VpE4h1F5AfgYrMNx9oJpy8O54uIiIiIbPHibcG+HngHmGZmrwHLge7A8cBA4PBGiU5ERESkhXN3zLbsUTWao+A1aZwXJd5h+j4Cdga+A34J3AqcDHwL7OzuY+NZj5m1MbOvzewHM5tiZn8Ny7cys6/MbJaZvWhmGWF5Zjg9K5w/sOG7KCIiIpI8ZSUVZLfLTHYYEiWrXSZlxRWNsu54u4jg7lPc/VR339rds8O/p7t7Q25AUwYc6O47ADsCh5nZHgSjkNzp7oOANcD5Yf3zgTVh+Z1hPREREZEWI3d5IT0Hdk52GBKhXacs2uRkkJ9b3Cjrj7eLSEJ40A5fGE6mhw8HDgROD8ufAm4EHgSODZ8DvALcZ2bmjdWeLyIiIpJgS+asZujOfdlm9/7kLiugvKxS/UWSJCU1hXadsujYvS2zf1jaaC9DkybYAGaWCkwEBgH3A7OBPHevGc9nEdAnfN6HcPQSd680s7VAF2BVkwYtIiIisomqq5xpExbSoWsOHbrmkNOhTbJD2mK5O8X5ZSyePZ/y0sYbSrLJE2x3rwJ2NLOOwGvAsM1dp5ldBFwE0L9//81dnYiIiEhCuUPeyiLyVhYlOxRpAnH3wU40d88DxhLcIbKjmdUk+32BxeHzxUA/gHB+B2B1jHU94u4j3X1kt27dGjt0EREREZFa1ZtghyN5nGFmgzd3Y2bWLWy5xsyygEOAnwgS7ZPCamcDo8Lnb4TThPM/Uv9rEREREWnO6k2w3b0MeBTonYDt9QLGmtkk4BvgA3d/C7gW+L2ZzSLoY/1YWP8xoEtY/nvgugTEICIiIiLSaOLtgz0ZGAJ8vDkbc/dJwE4xyucAu8UoLyUYd1tEREREpEWIN8G+AnjSzJYC70aM+CEiIiIiIhHiTbBfB7IJ+ka7ma0hGL96HXfvntjQRERERERanngT7PuJSqhFRERERGRjcSXY7n5jI8chIiIiItIqNOhGM2bWCdiOYGzqd9x9jZm1AcrdvboxAhQRERERaUniutGMmaWZ2T8JbmP+MfAMsFU4+1XgL40TnoiIiIhIyxLvnRxvBS4ELgV+BljEvFHA0QmOS0RERESkRYq3i8hZwHXu/oSZpUbNm02QdIuIiIiIbPHibcHuSJBIx5IBRCfdIiIiIiJbpHgT7B+BY2uZdzjwbWLCERERERFp2eLtInIL8KqZZQEvE4yJvaOZHQ/8GjimkeITEREREWlR4mrBdvdRwOnAwcA7BBc5PgqcA5zp7u81VoAiIiIiIi1J3ONgu/tLwEtmNhToAuQC091dd3gUEREREQk16EYzAO4+vTECERERERFpDeK9yBEzG2Fm/zWzWWZWFP79r5lt35gBioiIiIi0JHG1YJvZccBLBEP1vQKsALoTjCwywcxOdvfXGylGEREREZEWI94uIv8guGPjyZF9rs3sDwSjivwDeD3h0YmIiIiItDDxdhHpBzwafUFjOP2fcL6IiIiIyBYv3gR7ArBtLfO2QzeaEREREREB6ugiYmbZEZO/B14ws3SCriA1fbCPBy4ATm3EGEVEREREWoy6+mAXEtyxsYYBfwf+FlUG8BWQmtjQRERERERanroS7PPYMMEWEREREZF61Jpgu/uTTRiHiIiIiEirEPeNZkREREREpH7x3mgmA7ic4KLGPkCb6Dru3j2hkYmIiIiItEDx3mjmQeAMgpvNfASUN1pEIiIiIiItWLwJ9gnA5e7+UGMGIyIiIiLS0sXbBzsXWNCYgYiIiIiItAbxJtg3AVeaWU5jBiMiIiIi0tLF1UXE3Z8ys22ABWY2EcjbuIqfkujgRERERERamnhHEbkSuBpYBuQA6Y0ZlIiIiIhISxXvRY7XAfcAV7i77u4oIiIiIlKLePtgG/CWkmsRERERkbrFm2A/CZzYiHGIiIiIiLQK8XYRWQT83sw+JLjRTF7UfHf3BxMZmIiIiIhISxRvgv3v8G9f4MAY853gbo8iIiIiIlu0eIfpi7criYiIiIjIFk2Js4iIiIhIAsU7DvYR9dVx99GbH46IiIiISMsWbx/stwj6WVtUeeSwfakJiUhEREREpAWLN8HeKkZZJ+BQ4FzgnEQFJCIiIiLSksV7keP8GMXzge/NrAq4HjgmkYGJiIiIiLREibjI8TtiD923ETPrZ2ZjzWyqmU0xs/8Lyzub2QdmNjP82yksNzO7x8xmmdkkM9s5AfGKiIiIiDSazUqwzSyDoHvI0jgXqQSudPdtgD2AS8xsG+A6YIy7DwbGhNMAhwODw8dFaKxtEREREWnm4h1F5Bs2vKARIAMYCLQj6IddL3dfSpiMu3uBmf0E9AGOBfYPqz0FjAOuDcufdncHxptZRzPrFa5HRERERKTZifcixylsnGCXAi8Dr7v7lIZu2MwGAjsBXwE9IpLmZUCP8HkfYGHEYovCsg0SbDO7iKCFm/79+zc0FBERERGRhIn3IsdzErlRM2sLvApc7u75ZutH/3N3N7PoZL6++B4BHgEYOXJkg5YVEREREUmkJr+To5mlEyTXz7n7/8Li5WbWK5zfC1gRli8G+kUs3jcsExERERFplmptwTazxxuwHnf38+urZEFT9WPAT+7+74hZbwBnA7eFf0dFlF9qZi8AuwNr1f9aRERERJqzurqIjIhj+TbAtgT9s+tNsIG9gDOByWb2fVh2PUFi/ZKZnU8wvvbJ4bzRwBHALKCYOC+mFBERERFJlloTbHfftbZ5ZpYD/Bb4PVBOMPJHvdz9Mza+3XqNg2LUd+CSeNYtIiIiItIcxDuKCABm1gH4XfjIAh4Fbnd39YsWERERESH+cbC7AFcStFobwQ1f7nD3lY0Ym4iIiIhIi1Nngh2O6HE1wRjTZcCdwN3untf4oYmIiIiItDx1jSLyIMFt0POAvwIPuHtR04QlIiIiItIy1dWC/evw72rgl8AvI28IE83dd0tgXCIiIiIiLVJdCfbTbHx7dBERERERqUNdw/Sd04RxiIiIiIi0Ck1+q3QRERERkdZMCbaIiIiISAIpwRYRERERSSAl2CIiIiIiCaQEW0REREQkgeK6VXoNM+sEbAf0A95x9zVm1gYod/fqxghQRERERKQliasF28xSzeyfwCLgY+AZYKtw9qvAXxonPBERERGRliXeLiJ/Ay4ELgV+BkTe0nEUcHSC4xIRERERaZHi7SJyFnCduz9hZqlR82YTJN0iIiIiIlu8eFuwOxIk0rFkANFJt4iIiIjIFineBPtH4Nha5h0OfJuYcEREREREWrZ4u4jcArxqZlnAy4ADO5rZ8cCvgWMaKT4RERERkRYlrhZsdx8FnA4cDLxDcJHjo8A5wJnu/l5jBSgiIiIi0pLEPQ62u78EvGRmQ4CuQC4w3d29sYITEREREWlpGnSjGQB3nwHMaIRYRERERERavLgTbDPrDRwF9AXaRM12d782kYGJiIiIiLREcSXY4cWMzxMMx7cCKI+q4oASbBERERHZ4sXbgv034H3gHHfPbcR4RERERERatHgT7H7AZUquRURERETqFu+NZr4AhjZmICIiIiIirUGtLdhmlh0x+XvgOTMrBD4A8qLru3txwqMTEREREWlh6uoiUkhw8WINA56IKouUmqigRERERERaqroS7HObLAoRERERkVairgR7LvCtuxc2VTAiIiIiIi1dXRc5jgW2aapARERERERag7oSbGuyKEREREREWol4h+kTEREREZE41HejmSPMbFg8K3L3pxMQj4iIiIhIi1Zfgn1DnOtxQAm2iIiIiGzx6kuwDwAmNEUgIiIiIiKtQX0Jdom7FzVJJCIiIiIirYAuchQRERERSSAl2CIiIiIiCVRrFxF3V/ItIiIiItJATZpEm9njZrbCzH6MKOtsZh+Y2czwb6ew3MzsHjObZWaTzGznpoxVRERERGRTNHUr9ZPAYVFl1wFj3H0wMCacBjgcGBw+LgIebKIYRUREREQ2WZMm2O7+CZAbVXws8FT4/CnguIjypz0wHuhoZr2aJFARERERkU3UHPpZ93D3peHzZUCP8HkfYGFEvUVhmYiIiIhIs9UcEux13N0J7grZIGZ2kZlNMLMJK1eubITIRERERETi0xwS7OU1XT/CvyvC8sVAv4h6fcOyjbj7I+4+0t1HduvWrVGDFRERERGpS3NIsN8Azg6fnw2Miig/KxxNZA9gbURXEhERERGRZqm+W6UnlJk9D+wPdDWzRcBfgNuAl8zsfGA+cHJYfTRwBDALKAbObcpYRUREREQ2RZMm2O5+Wi2zDopR14FLGjciEREREZHEag5dREREREREWg0l2CIiIiIiCaQEW0REREQkgZRgi4iIiIgkkBJsEREREZEEUoItIiIiIpJASrBFRERERBJICbaIiIiISAIpwRYRERERSSAl2CIiIiIiCaQEW0REREQkgZRgi4iIiIgkkBJsEREREZEEUoItIiIiIpJASrBFRERERBJICbaIiIiISAIpwRYRERERSSAl2CIiIiIiCaQEW0REREQkgZRgi4iIiIgkkBJsEREREZEEUoItIiIiIpJASrBFRERERBJICbaIiIiISAIpwRYRERERSSAl2CIiIiIiCaQEW0REREQkgZRgi4iIiIgkkBJsEREREZEEUoItIiIiIpJASrBFRERERBJICbaIiIiISAIpwRYRERERSSAl2CIiIiIiCaQEW0REREQkgZRgi4iIiIgkkBJsEREREZEEUoItIiIiIpJASrBFRERERBJICbaIiIiISAIpwRYRERERSSAl2CIiIiIiCdTsE2wzO8zMppvZLDO7LtnxiIiIiIjUpVkn2GaWCtwPHA5sA5xmZtskNyoRERERkdo16wQb2A2Y5e5z3L0ceAE4NskxiYiIiIjUqrkn2H2AhRHTi8IyEREREZFmydw92THUysxOAg5z9wvC6TOB3d390qh6FwEXhZNDgelNGmjz1RVYlewgpNnReSGx6LyQWHReSCw6L9Yb4O7dogvTkhFJAywG+kVM9w3LNuDujwCPNFVQLYWZTXD3kcmOQ5oXnRcSi84LiUXnhcSi86J+zb2LyDfAYDPbyswygFOBN5Ick4iIiIhIrZp1C7a7V5rZpcB7QCrwuLtPSXJYIiIiIiK1atYJNoC7jwZGJzuOFkrdZiQWnRcSi84LiUXnhcSi86IezfoiRxERERGRlqa598EWEREREWlRlGCLNCNmNs/MDk7g+vYxMw1bKeuYmZvZoPD5Q2b25zrqXm9mjzZddNISmNn+ZrYo2XFIcunzoW5KsAUAM3vSzG5JdhySWO7+qbsPTXYc0jy5+2/c/WaInTS5+99q7kMgrUOiv8RL85CI19XMxplZ3O/35vr50ND9aCzN/iJHaXxmlprsGERERKTpmZkBluw4Whu1YLcAZnatmS02swIzm25mB5nZjWb2ipm9GJZ/a2Y7RCwzPPwWl2dmU8zsmIh5T5rZg2Y22syKgPOBM4BrzKzQzN5Mwm7Kerua2VQzW2NmT5hZGzM7x8w+i6wU9VP/EeEyBeG5clVYvkGrZNjKcZWZTTKzteH50yZi/lFm9n143nxhZttHzNvoPAzLdzOzCWaWb2bLzezfjX2AZN1r+YfocyWcd6GZzTKzXDN7w8x617KOJ83sFjPLAd4BeoefAYVm1jv8nHk2ov7e4XmRZ2YLzeycsDzm+SfNi5k9A/QH3gxf42vMbI+I1/QHM9s/on7n8LxaEp5jr0et70ozW2FmS83s3CbdmVYqfN+9amYrzWyumf0uLL/RzF4ys6fD99kUMxsZztvodQ3L63ptx5nZrWb2OVAMPAPsA9wXruO+sN7d4Xs938wmmtk+EetY9/lgZgPD/0lnm9kCM1tlZn+MqvuymT0bxj/ZzIaEn2Erwm38IqJ+BzN7LDy3FoefU6nhvHPM7DMz+1d4Xs41s8PDebfG2o+kcHc9mvGD4NbvC4He4fRAYGvgRqACOAlIB64C5obP04FZwPVABnAgUAAMDdfxJLAW2IvgS1absOyWZO/vlv4A5gE/EtzBtDPwOXALcA7wWVRdBwaFz5cC+4TPOwE7h8/3BxZFrf9roHe4/p+A34TzdgJWALsTjDt/dlg/s7bzMHz+JXBm+LwtsEeyj+OW8KjjXDmQ4BbGO4ev3b3AJ7WcN+ve99HnSlh2I/Bs+HxA+DlyWvgZ0wXYsa7zT4/m9wjPm4PD532A1cAR4f+CQ8LpbuH8t4EXw9c0Hdgv4lypBG4Ky48gSNI6JXv/WvIjfA0mAjcQ/O/+GTAHODR8L5aGxzoV+DswPtbrGudrOw5YAGxL0JshPSy7ICqmX4Xv9TTgSmAZ0CacF/n5MDD8bPkPkAXsAJQBwyPqlob7kgY8TZCz/DHc9oXA3IjtvgY8DOQA3Qn+b/06nHcOQf5zYXgsLgaWsH5kvI32IxkPtWA3f1UE/yS3MbN0d5/n7rPDeRPd/RV3rwD+TZAo7xE+2gK3uXu5u38EvEXwj7HGKHf/3N2r3b206XZH4nCfuy9091zgVjZ83WpTQXCOtHf3Ne7+bR1173H3JeH63wR2DMsvAh5296/cvcrdnyL4gNyDus/DCmCQmXV190J3H9/gPZZNFetcOYPgplzfunsZ8AdgTzMbuJnbOh340N2fd/cKd1/t7t+H8xpy/knz8StgtLuPDv8XfABMAI4ws17A4QRfwNeEr/nHEctWADeF5aOBQoIv4rLpdiVIgG8K/3fPIUhYTw3nfxa+VlUELc471LYi6nhtI+o86e5T3L0yzCM24u7Phu/1Sne/g/UNLrX5q7uXuPsPwA9RMX7q7u+5eyXwMtCNIE+pAF4ABppZRzPrEcZ5ubsXufsK4M6I4wAw393/Ex6Lp4BeQI864mpySrCbOXefBVxO8O1vhZm9EPFz78KIetXAIoKWyd7AwrCsxnyCb7RELyvNTuRrM5/g9azPiQQfSPPN7GMz27OOussinhcTfBmDoIXyyvDnxDwzyyNoHe1dz3l4PjAEmGZm35jZUXHEK4kR61zpHT4HwN0LCVqu+rB5+gGza5nXkPNPmo8BwC+j3vN7EyQr/YBcd19Ty7Krw0SpRuRniWyaAQTdtCJfj+tZnzhGf3a3MbParqWr67WtUW8eYEGXwp8s6FKYB3QAutaxSG3/XwCWRzwvAVaFCXLNNGH9AQSt2ksjYn+YoCV7o+24e3HEss2GEuwWwN3/6+57E5x0DvwjnNWvpo6ZpQB9CX4mWQL0C8tq9AcWR642ejOJjls2Wb+I5/0JXs8iILum0Mx6Ri7g7t+4+7EEH0CvAy9twnYXAre6e8eIR7a7Px9uI+Z56O4z3f20cNv/AF6xoE+vNL5Y58oSgtcIgPC16MKG7/9Y6vsMWEjQPW3jBRNz/knTiHydFwLPRL3nc9z9tnBeZzPrmJQot0wLCbpJRL4e7dz9iHqX3Pj9W9drW9syG0yH/a2vAU4m6P7TkaB7aWNfELmQ4NfTrhGxt3f3beNcvlnkM0qwmzkzG2pmB5pZJkH/pRKgpmV6FzM7IfwGeznBCTke+Irgm+M1ZpYeXthwNMFPMLVZTtDfS5LvEjPra2adCfqnvUjwU9u2ZrajBRey3VhT2cwyzOwMM+sQ/tSWz/pzpCH+A/zGzHa3QI6ZHWlm7eo6D83sV2bWLfzFJC9c16ZsXxou1rnyPHBueK5kAn8DvnL3efWsaznQxcw61DL/OeBgMzvZzNLMrEu4jUSdf9I0Ij/rnwWONrNDzSzVgguq9zezvu6+lODC1wfMrFP4v2TfpEW9ZfgaKLDggvKs8DXZzsx2jWPZ6P/htb62DVhHO4K+9iuBNDO7AWjfsF1quPDcex+4w8zam1mKmW1tZvvFuYpmkc8owW7+MoHbCC5aWkbQQvSHcN4o4BRgDXAmcELYH66cIKE+PFzuAeAsd59Wx3YeI+hDmWdRV4pLk/svwYfLHIKf5G9x9xkEFxR9CMwEPota5kxgnpnlA78h6IfbIO4+geCikfsIzqlZBBeTQN3n4WHAFDMrBO4GTnX3EqQpxDpXPgT+DLxKcPHh1mzYdzGm8PPheWBO+DnQO2r+AoJuIFcCucD3rO9fudnnnzSZvwN/Cn92PwU4lqAbwkqClsOrWZ8bnEnQ13oawQXQlzdxrFuUsLvEUQTXxcwl+Lx9lKBbRn3Wva5mdpW7L6Tu1zaWu4GTLBiZ4x7gPeBdYAZBt7NSmq576VkEF3pOJfh/9Aobdm+pS/R+JEXNFZfSwpjZjQQjAfwq2bGISNMzs3kEV8p/mOxYRERkQ2rBFhERERFJICXYIiIiIiIJpC4iIiIiIiIJpBZsEREREZEEUoItIiIiIpJASrBFRERERBJICbaISBMzsxPN7KNwzNoyM5thZv+OHnu6nnVcE95EqlkysyfNbEKy4xARSQZd5Cgi0oTM7A6CG3Y8QXCzqHxgG4IbtMxx9+PjXM8q4D53v7FxIt08ZrY1kOXuPyY7FhGRppaW7ABERLYUZnY08HvgfHd/PGLWx2b2CPCL5ESWOGaW5e4l7j472bGIiCSLuoiIiDSdK4Bvo5JrILhNsru/A2Bmt5nZZDMrNLNFZvacmfWsqRvexbEL8Bcz8/CxfzgvxcyuM7NZEd1Pzo7clgVuNrMVZpZvZo+b2anhegZG1OtqZk+Z2WozKzazcWY2Mmpd88zsDjP7s5ktImiRj9lFxMz6m9kLZpYbru89MxsaVecPYeylZrbczN6N3HcRkZZACbaISBMws3Tg58C7cVTvDvwNOJKgO8nPgI/MrOYz+3hgLfAYsGf4+Dacdy/wJ+CRcPnXgMfN7KiI9V8OXA88BJwElAD/jBHH68ChwFXAKQT/M8aa2aCoeqcD+wG/DettxMw6A58BQwm6w5wM5AAfmllWWOesMK5/h9u9GJgV1hMRaTHURUREpGl0ATKBBfVVdPfzap6bWSrwJbAI2Bv4xN2/M7NKYJG7j4+oO4ggKT3X3Z8Kiz80s17AX4C3wvVdAzzk7jeEdd43s62AfhHrOgzYC9jf3T8Oyz4C5gFXA7+OCvsody+tY7euIEiUd3T33HB9n4frOw+4H9gNeN/dH4hY7n91rFNEpFlSC7aISNOq98pyMzvczL4ws7VAJUFyDTCknkUPAqqB18wsreYBjAF2DJPrfkBP4I2oZaOndwNW1CTXAO5eBLxFkOhHGlNPcg1wMPABkB8RVwEwEajpdvI9cISZ/dXMdgvjFRFpcZRgi4g0jdVAGdC/rkpmtitBsrsIOJOg+8ce4ew29WyjK5BK0H2kIuLxJMEvlr0IkmuAlVHLRk/3AlbE2MZyoHOMsvp0Jeg+UhH1OID1LeePE3QRORn4ClhuZrco0RaRlkZdREREmoC7V4RdIg4l6CNdm+MJkt1TPBxH1cwGxLmZXIIW770IWrKjrWD95363qHnR00sJ+oJH6xFuJ1I8473mEnxxuDnGvAIAd68G7gTuNLN+wBnArQRfNh6KYxsiIs2CWrBFRJrOXcDI6FE9YN3oH4cBWUCFb3iTgjNirKucjVu0PyJowe7g7hNiPMqBhcAy4NioZY+Jmv4K6G5m+0bEmE1w4eRn9e1oDGOAbYEpMeKaHl3Z3Re6+20EFzluswnbExFJGrVgi4g0EXd/08z+DTxmZnsR3GimEBhGMLLGPOA/wOVmdhfwJsHII7+KsbppwJFm9m64junuPt3MHgJeMLN/AhMIkvBtgSHufoG7V5nZ7cDtZrYS+JwguR4Rrrc6jPU9M/sCeNHMriPo4nIVwReA2zdh9/8d7sdHZnYvsJigNXw/4DN3f97MHiZo6R5P0M3lAGAwcO0mbE9EJGnUgi0i0oTc/UqCvsiDgf8SXPh3JUEL78XuPpogoTyRoEvFfsBRMVZ1NVAEvA18A+wSll9C0A3jLGA0Qf/rI4FPIpa9E/g7wbB6rwKdCIYFhHAc69BxYXx3AS8DBhzo7rM2Yb9XEfQlnxZu/32CoQE7AJPCal8C+xLc5XI0QXeZC9399YZuT0QkmXSrdBERwcweBQ5x93j7e4uISC3URUREZAtjZtsRtKJ/QdAl5HDgXNQVQ0QkIdSCLSKyhQlvKvM4sCPBzV/mAw8Dd7j+KYiIbDYl2CIiIiIiCaSLHEVEREREEkgJtoiIiIhIAinBFhERERFJICXYIiIiIiIJpARbRERERCSBlGCLiIiIiCTQ/wMIhKHtKpzGWgAAAABJRU5ErkJggg==","text/plain":["<Figure size 864x360 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["fig = plt.figure(figsize= (12, 5))\n","\n","ax = fig.add_subplot(111)\n","\n","sns.barplot(x = category_count.index, y = category_count )\n","\n","for a, p in enumerate(ax.patches):\n","    ax.annotate(f'{categories[a]}\\n' + format(p.get_height(), '.0f'), xy = (p.get_x() + p.get_width() / 2.0, p.get_height()), xytext = (0,-25), size = 13, color = 'white' , ha = 'center', va = 'center', textcoords = 'offset points', bbox = dict(boxstyle = 'round', facecolor='none',edgecolor='white', alpha = 0.5) )\n","    \n","plt.xlabel('Categories', size = 15)\n","\n","plt.ylabel('The Number of News', size= 15)\n","\n","plt.xticks(size = 12)\n","\n","plt.title(\"The number of News by Categories\" , size = 18)\n","\n","plt.show()"]},{"cell_type":"code","execution_count":47,"metadata":{},"outputs":[{"data":{"text/plain":["array(['tech', 'business', 'sport', 'entertainment', 'politics'],\n","      dtype=object)"]},"execution_count":47,"metadata":{},"output_type":"execute_result"}],"source":["df['category'].unique()"]},{"cell_type":"code","execution_count":48,"metadata":{},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>text</th>\n","      <th>count</th>\n","      <th>encoded_text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>tech</td>\n","      <td>tv future in the hands of viewers with home th...</td>\n","      <td>737</td>\n","      <td>4</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>business</td>\n","      <td>worldcom boss  left books alone  former worldc...</td>\n","      <td>300</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>sport</td>\n","      <td>tigers wary of farrell  gamble  leicester say ...</td>\n","      <td>246</td>\n","      <td>3</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>sport</td>\n","      <td>yeading face newcastle in fa cup premiership s...</td>\n","      <td>341</td>\n","      <td>3</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>entertainment</td>\n","      <td>ocean s twelve raids box office ocean s twelve...</td>\n","      <td>260</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>politics</td>\n","      <td>howard hits back at mongrel jibe michael howar...</td>\n","      <td>633</td>\n","      <td>2</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>politics</td>\n","      <td>blair prepares to name poll date tony blair is...</td>\n","      <td>269</td>\n","      <td>2</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>sport</td>\n","      <td>henman hopes ended in dubai third seed tim hen...</td>\n","      <td>191</td>\n","      <td>3</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>sport</td>\n","      <td>wilkinson fit to face edinburgh england captai...</td>\n","      <td>157</td>\n","      <td>3</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>entertainment</td>\n","      <td>last star wars  not for children  the sixth an...</td>\n","      <td>237</td>\n","      <td>1</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        category                                               text  count  \\\n","0           tech  tv future in the hands of viewers with home th...    737   \n","1       business  worldcom boss  left books alone  former worldc...    300   \n","2          sport  tigers wary of farrell  gamble  leicester say ...    246   \n","3          sport  yeading face newcastle in fa cup premiership s...    341   \n","4  entertainment  ocean s twelve raids box office ocean s twelve...    260   \n","5       politics  howard hits back at mongrel jibe michael howar...    633   \n","6       politics  blair prepares to name poll date tony blair is...    269   \n","7          sport  henman hopes ended in dubai third seed tim hen...    191   \n","8          sport  wilkinson fit to face edinburgh england captai...    157   \n","9  entertainment  last star wars  not for children  the sixth an...    237   \n","\n","   encoded_text  \n","0             4  \n","1             0  \n","2             3  \n","3             3  \n","4             1  \n","5             2  \n","6             2  \n","7             3  \n","8             3  \n","9             1  "]},"execution_count":48,"metadata":{},"output_type":"execute_result"}],"source":["df['encoded_text'] = df['category'].astype('category').cat.codes\n","\n","df.head(10)"]},{"cell_type":"code","execution_count":49,"metadata":{},"outputs":[],"source":["data_texts = df['text'].to_list()\n","\n","data_labels = df['encoded_text'].to_list()"]},{"cell_type":"markdown","metadata":{},"source":["# Train Test SPlit"]},{"cell_type":"code","execution_count":53,"metadata":{},"outputs":[],"source":["train_texts, val_texts, train_labels, val_labels = train_test_split(data_texts, data_labels, test_size = 0.2, random_state = 0 )\n","\n","\n","train_texts, test_texts, train_labels, test_labels = train_test_split(train_texts, train_labels, test_size = 0.01, random_state = 0 )\n","\n"]},{"cell_type":"markdown","metadata":{},"source":["# Model Definition"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n","\n","train_encodings = tokenizer(train_texts, truncation = True, padding = True  )\n","\n","val_encodings = tokenizer(val_texts, truncation = True, padding = True )"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["train_dataset = tf.data.Dataset.from_tensor_slices((\n","    dict(train_encodings),\n","    train_labels\n","))\n","\n","\n","val_dataset = tf.data.Dataset.from_tensor_slices((\n","    dict(val_encodings),\n","    val_labels\n","))"]},{"cell_type":"markdown","metadata":{},"source":["# Fine-tuning with the TFTrainer class"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=5)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["from transformers import TFDistilBertForSequenceClassification, TFTrainer, TFTrainingArguments\n","\n","\n","training_args = TFTrainingArguments(\n","    output_dir='./results',          \n","    num_train_epochs=7,              \n","    per_device_train_batch_size=16,  \n","    per_device_eval_batch_size=64,   \n","    warmup_steps=500,                \n","    weight_decay=1e-5,               \n","    logging_dir='./logs',            \n","    eval_steps=100                   \n",")\n","\n","with training_args.strategy.scope():\n","    trainer_model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels = 5 )\n","\n","\n","trainer = TFTrainer(\n","    model=trainer_model,                 \n","    args=training_args,                  \n","    train_dataset=train_dataset,         \n","    eval_dataset=val_dataset,            \n",")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["trainer.train()"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["trainer.evaluate()"]},{"cell_type":"markdown","metadata":{},"source":["# Saving & Loading the model"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["save_directory = \"/saved_models\" \n","\n","model.save_pretrained(save_directory)\n","\n","toknizer.save_pretrained(save_directory)"]},{"cell_type":"markdown","metadata":{},"source":["# Loading Pre-Trained Model"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["tokenizer_fine_tuned = DistilBertTokenizer.from_pretrained(save_directory)\n","\n","model_fine_tuned = TFDistilBertForSequenceClassification.from_pretrained(save_directory)"]},{"cell_type":"code","execution_count":54,"metadata":{},"outputs":[{"data":{"text/plain":["'dollar hovers around record lows the us dollar hovered close to record lows against the euro on friday as concern grows about the size of the us budget deficit.  analysts predict that the dollar will remain weak in 2005 as investors worry about the state of the us economy. the bush administration s apparent unwillingness to intervene to support the dollar has caused further concern. however  trading has been volatile over the past week because of technical and automated trading and light demand. this has amplified reactions to news  analysts said  adding that they expect markets to become less jumpy in january.  the dollar was trading at $1.3652 versus the euro on friday morning after hitting a fresh record low of $1.3667 on thursday. one dollar bought 102.55 yen.  disappointing business figures from chicago triggered the us currency s weakness on thursday. the national association of purchasing management-chicago said its manufacturing index dropped to 61.2  a bigger fall than expected.  there are no dollar buyers now  especially after the chicago data yesterday   said abn amro s paul mackel. at the same time  german chancellor gerhard schroeder and italian prime minister silvio berlusconi voiced concerns about the strength of the euro. mr berlusconi said the euro s strength was  absolutely worrying  for italian exports. mr schroeder said in a newspaper article that stability in foreign exchange markets required a correction of global economic imbalances. investors will now look towards february s meeting of finance ministers from the g7 industrialised nations in london for clues as to whether central banks will combine forces to stem the dollar s decline.'"]},"execution_count":54,"metadata":{},"output_type":"execute_result"}],"source":["test_text = test_texts[0]\n","\n","test_text"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["predict_input = tokenizer_fine_tuned.encode(\n","    test_text,\n","    truncation = True,\n","    padding = True,\n","    return_tensors = 'tf'    \n",")\n","\n","output = model_fine_tuned(predict_input)[0]\n","\n","prediction_value = tf.argmax(output, axis = 1).numpy()[0]\n","\n","prediction_value"]},{"cell_type":"markdown","metadata":{},"source":["# Inferencing with Pytorch"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import torch\n","\n","from transformers import DistilBertForSequenceClassification\n","\n","tokenizer_fine_tuned_pt = DistilBertTokenizer.from_pretrained(save_directory)\n","\n","\n","model_fine_tuned_pt = DistilBertForSequenceClassification.from_pretrained(save_directory, from_tf = True )\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["predict_input_pt = tokenizer_fine_tuned_pt(test_text, truncation = True, padding = True, return_tensors = 'pt' )\n","\n","ouput_pt = model_fine_tuned_pt(predict_input_pt)\n","\n","prediction_value_pt = torch.argmax(ouput_pt[0], dim = 1 ).item()\n","\n","prediction_value_pt"]}],"metadata":{"kernelspec":{"display_name":"Python 3.9.14 64-bit","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.14"},"vscode":{"interpreter":{"hash":"36cf16204b8548560b1c020c4e8fb5b57f0e4c58016f52f2d4be01e192833930"}}},"nbformat":4,"nbformat_minor":4}
2