{ "cells": [ { "cell_type": "markdown", "id": "7c5778f2-c097-4f8c-8f54-298e86be7b0d", "metadata": {}, "source": [ "# Cross Validation and Grid Search" ] }, { "cell_type": "markdown", "id": "89733788-abd8-4355-9e7b-8a5aac66705a", "metadata": {}, "source": [ "Here are examples of cross validation (running the same model with different training sets to get statistics on the performance) and grid search (trying an exhaustive search in parameter space.)" ] }, { "cell_type": "code", "execution_count": 185, "id": "0ae0180f-97d9-4a4b-802c-fda6b7ea4196", "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import fetch_openml" ] }, { "cell_type": "code", "execution_count": 186, "id": "83c3e695-8210-473a-b177-dfd2be1adbf0", "metadata": {}, "outputs": [], "source": [ "# Try this -- it may get blocked by LCPS?\n", "\n", "# uncomment next line\n", "# mnist = fetch_openml('mnist_784', as_frame=False)" ] }, { "cell_type": "code", "execution_count": 188, "id": "e01b5321-3631-4e1b-8518-7b091a517ecd", "metadata": {}, "outputs": [], "source": [ "# if above doesn't work, do this.\n", "# first download and gunzip the file from github \"mnist.gzip\"\n", "\n", "# infile = open(\"mnist.pk\", \"rb\")\n", "# mnist = pickle.load(infile)" ] }, { "cell_type": "code", "execution_count": 189, "id": "7e1d470b-f7fb-4beb-9a7c-61df427953c8", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((70000, 784), (70000,))" ] }, "execution_count": 189, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X,y = mnist.data, mnist.target\n", "X.shape, y.shape" ] }, { "cell_type": "code", "execution_count": 190, "id": "263d21b8-df82-4b4b-83a9-50e725db4adf", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "markdown", "id": "6d802781-a83c-40a3-8381-e56b5863a1c0", "metadata": {}, "source": [ "Note we use a subset of the data!" ] }, { "cell_type": "code", "execution_count": 191, "id": "7c4cfa37-15b6-4278-a1e0-847551a86afb", "metadata": {}, "outputs": [], "source": [ "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.01, train_size=0.1)" ] }, { "cell_type": "code", "execution_count": 192, "id": "d8465584-d318-4fe5-8df8-a5c713b0aa40", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((7000, 784), (700, 784))" ] }, "execution_count": 192, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.shape, X_test.shape" ] }, { "cell_type": "markdown", "id": "18a0c32a-571b-46d6-81c4-894123e08b0c", "metadata": {}, "source": [ "Let's make a binary classifier first : \"5\" or \"not 5\"" ] }, { "cell_type": "code", "execution_count": 193, "id": "2486752f-9094-41ca-ad87-3aca8f598dfc", "metadata": {}, "outputs": [], "source": [ "y_train_5 = (y_train == '5')\n", "y_test_5 = (y_test == '5')" ] }, { "cell_type": "markdown", "id": "f83d5a07-b844-4ceb-9339-d1f013bdb351", "metadata": {}, "source": [ "SGD (standard gradient descent) is a type of linear classifier that minimizes a loss function. We use it here just because it's different." ] }, { "cell_type": "code", "execution_count": 194, "id": "ae7c8455-eb83-48bc-a5fa-ae473ae433dc", "metadata": {}, "outputs": [], "source": [ "from sklearn.linear_model import SGDClassifier\n", "from sklearn.metrics import accuracy_score" ] }, { "cell_type": "code", "execution_count": 195, "id": "81fd1fb9-2d04-4d57-a1d0-4c81b93924ef", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
SGDClassifier(random_state=42)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "SGDClassifier(random_state=42)" ] }, "execution_count": 195, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sgd_classifier = SGDClassifier(random_state = 42)\n", "sgd_classifier.fit(X_train, y_train_5)" ] }, { "cell_type": "code", "execution_count": 196, "id": "ea8a3a6b-1df4-4edd-977a-4de04df7acde", "metadata": {}, "outputs": [], "source": [ "y_pred = sgd_classifier.predict(X_test)" ] }, { "cell_type": "code", "execution_count": 197, "id": "823743c5-c7bd-4b8c-93aa-c9832d284f88", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9471428571428572" ] }, "execution_count": 197, "metadata": {}, "output_type": "execute_result" } ], "source": [ "accuracy_score(y_pred, y_test_5)" ] }, { "cell_type": "markdown", "id": "dbe8dc37-2a54-4fab-81e2-2dd1bd0d3ca1", "metadata": {}, "source": [ "Now we try cross validation -- it runs the same model training with 5 different subsets of the training set. It validates it on the remainder. This is called k-fold validation. (you can change the 5)" ] }, { "cell_type": "code", "execution_count": 198, "id": "0810a078-9676-4702-9db9-0b9713c52b8d", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import cross_val_score" ] }, { "cell_type": "code", "execution_count": 199, "id": "a62f04f3-72f6-4208-b5f2-1866c01e523e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.91714286, 0.95142857, 0.95642857, 0.95785714, 0.95214286])" ] }, "execution_count": 199, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cross_val_score(sgd_classifier, X_train, y_train_5, cv=5, scoring=\"accuracy\")" ] }, { "cell_type": "markdown", "id": "ec4a4112-74b1-4588-9dc2-65c9a777d5f2", "metadata": {}, "source": [ "As an example we do a KNN also" ] }, { "cell_type": "code", "execution_count": 200, "id": "9d350183-8461-4dfd-8f96-907376c7af8e", "metadata": {}, "outputs": [], "source": [ "from sklearn.neighbors import KNeighborsClassifier" ] }, { "cell_type": "code", "execution_count": 201, "id": "5cacd61f-14e2-4927-afbf-56d44cdcc696", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
KNeighborsClassifier(n_neighbors=7)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "KNeighborsClassifier(n_neighbors=7)" ] }, "execution_count": 201, "metadata": {}, "output_type": "execute_result" } ], "source": [ "knn_classifier = KNeighborsClassifier(n_neighbors=7)\n", "knn_classifier.fit(X_train, y_train_5)" ] }, { "cell_type": "code", "execution_count": 202, "id": "775924bf-08f2-4f19-907e-9cb10fae4cdc", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.98714286, 0.98785714, 0.98071429, 0.98428571, 0.98928571])" ] }, "execution_count": 202, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cross_val_score(knn_classifier, X_train, y_train_5, cv=5, scoring=\"accuracy\")" ] }, { "cell_type": "markdown", "id": "d8af1519-10c9-4074-9038-7f1772015173", "metadata": {}, "source": [ "KNN can do multiclass, so let's throw the 10-class original training set at it." ] }, { "cell_type": "code", "execution_count": 203, "id": "793acf05-05b4-410e-8b2e-cdf065ca0702", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.92785714, 0.94214286, 0.915 , 0.92857143, 0.92857143])" ] }, "execution_count": 203, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cross_val_score(knn_classifier, X_train, y_train, cv=5, scoring=\"accuracy\")" ] }, { "cell_type": "markdown", "id": "61ac6cb6-f31f-4106-a5b7-e61970714ca4", "metadata": {}, "source": [ "Let's look at some statistics. We'll run it 20 times and use a pandas dataset as a quick way to get stats." ] }, { "cell_type": "code", "execution_count": 204, "id": "d177efb8-8f70-41cb-b8e1-4c6cb5bf7788", "metadata": {}, "outputs": [], "source": [ "knn_results = cross_val_score(knn_classifier, X_train, y_train, cv=20, scoring=\"accuracy\")" ] }, { "cell_type": "code", "execution_count": 205, "id": "00469e09-d012-40d0-9dca-0c7326ee651d", "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 206, "id": "d3db562a-f726-4973-912e-dc522dbcb88b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 20.000000\n", "mean 0.932714\n", "std 0.010466\n", "min 0.914286\n", "25% 0.927857\n", "50% 0.931429\n", "75% 0.937143\n", "max 0.954286\n", "dtype: float64" ] }, "execution_count": 206, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.Series(knn_results).describe()" ] }, { "cell_type": "markdown", "id": "1ba595e6-87a8-4aa5-8d4a-5ebee6f7b3a1", "metadata": {}, "source": [ "## Grid Search" ] }, { "cell_type": "markdown", "id": "2cd477ee-3852-46e8-949c-2bcefeb6ea90", "metadata": {}, "source": [ "A grid search takes a list of parameters and does a cross-validation on each parameter. It is a slow, exhaustive way to find the best setting for your model." ] }, { "cell_type": "code", "execution_count": 209, "id": "4e3869b5-cb32-45c1-a0bb-ea18413cc06b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GridSearchCV(cv=3,\n", " estimator=Pipeline(steps=[('kneighborsclassifier',\n", " KNeighborsClassifier())]),\n", " n_jobs=1,\n", " param_grid={'kneighborsclassifier__n_neighbors': [1, 2, 3, 4, 5,\n", " 10, 15, 20, 25,\n", " 30]},\n", " scoring='accuracy')\n" ] } ], "source": [ "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "# Create pipeline\n", "pipeline = make_pipeline(KNeighborsClassifier())\n", "\n", "# Define parameter grid\n", "param_grid = {\n", " 'kneighborsclassifier__n_neighbors': [1, 2, 3, 4, 5, 10, 15, 20, 25, 30]\n", "}\n", "\n", "# Perform grid search\n", "grid_search = GridSearchCV(pipeline, param_grid, cv=3, scoring='accuracy', n_jobs=1)\n", "\n", "# Print the grid_search to confirm no errors\n", "print(grid_search)" ] }, { "cell_type": "code", "execution_count": 210, "id": "607d643e-edb2-4cc3-9c74-854339c00f6e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
GridSearchCV(cv=3,\n",
       "             estimator=Pipeline(steps=[('kneighborsclassifier',\n",
       "                                        KNeighborsClassifier())]),\n",
       "             n_jobs=1,\n",
       "             param_grid={'kneighborsclassifier__n_neighbors': [1, 2, 3, 4, 5,\n",
       "                                                               10, 15, 20, 25,\n",
       "                                                               30]},\n",
       "             scoring='accuracy')
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "GridSearchCV(cv=3,\n", " estimator=Pipeline(steps=[('kneighborsclassifier',\n", " KNeighborsClassifier())]),\n", " n_jobs=1,\n", " param_grid={'kneighborsclassifier__n_neighbors': [1, 2, 3, 4, 5,\n", " 10, 15, 20, 25,\n", " 30]},\n", " scoring='accuracy')" ] }, "execution_count": 210, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grid_search.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "id": "700ae62a-9306-454b-a27f-49528d4106b6", "metadata": {}, "source": [ "We'll do lots of analysis on these results. The final best model is below." ] }, { "cell_type": "code", "execution_count": 213, "id": "d98582db-a04b-455b-8610-23d079f62262", "metadata": {}, "outputs": [], "source": [ "knn_cv_results = pd.DataFrame(grid_search.cv_results_)" ] }, { "cell_type": "code", "execution_count": 214, "id": "b00c6369-395a-42ea-ae3c-1075b859ffd6", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAG6CAYAAAAroB0qAAAAP3RFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMS5wb3N0MSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8kixA/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABEPUlEQVR4nO3de1hVZf7//9cGhY0i4gFRGBTxfIQUdfCQphRJ44esUccslSn7aqIZlWkxak6jzTSajjlmzsecrEmbDx46KGmU5ikPKGapiEeMBExTkgSVff/+6OeunYBuTFnS83Fd67raa91rrfe998r9Yq17rW0zxhgBAABYmEdFFwAAAHA1BBYAAGB5BBYAAGB5BBYAAGB5BBYAAGB5BBYAAGB5BBYAAGB5BBYAAGB5BBYAAGB5BBYAAGB55Qosc+fOVWhoqOx2u7p06aJt27aV2vbixYuaOnWqmjRpIrvdrvDwcKWkpFzRLjs7Ww8++KDq1KkjHx8ftWvXTjt27ChPeQAAoJJxO7AsXbpUiYmJmjx5snbu3Knw8HDFxMQoLy+vxPZJSUmaP3++5syZo71792rkyJHq37+/du3a5Wzz7bffqlu3bqpatapWr16tvXv3asaMGapVq1b5ewYAACoNm7s/ftilSxd16tRJr7zyiiTJ4XAoJCREY8aM0YQJE65oHxQUpOeee06jR492zrv//vvl4+OjN998U5I0YcIEbdq0SRs2bLievgAAgEqqijuNL1y4oLS0NE2cONE5z8PDQ9HR0dqyZUuJ6xQVFclut7vM8/Hx0caNG52v3333XcXExGjAgAFav369goOD9dhjj2nEiBGl1lJUVKSioiLna4fDodOnT6tOnTqy2WzudAsAAFQQY4y+++47BQUFycOjjAs/xg3Z2dlGktm8ebPL/Kefftp07ty5xHUGDx5sWrdubQ4cOGCKi4vNmjVrjI+Pj/Hy8nK28fb2Nt7e3mbixIlm586dZv78+cZut5tFixaVWsvkyZONJCYmJiYmJqZKMB0/frzMDOLWJaGvv/5awcHB2rx5s6Kiopzzx48fr/Xr12vr1q1XrHPy5EmNGDFC7733nmw2m5o0aaLo6GgtXLhQ58+flyR5eXkpMjJSmzdvdq43duxYbd++vcwzNz89w3L27Fk1bNhQx48fl5+f37V2CQAAVKD8/HyFhITozJkzqlmzZqnt3LokVLduXXl6eio3N9dlfm5ururXr1/iOgEBAVqxYoUKCwt16tQpBQUFacKECQoLC3O2adCggVq3bu2yXqtWrZScnFxqLd7e3vL29r5ivp+fH4EFAIBbzNWGc7h1l5CXl5c6duyo1NRU5zyHw6HU1FSXMy4lsdvtCg4O1qVLl5ScnKy4uDjnsm7duikjI8Ol/YEDB9SoUSN3ygMAAJWUW2dYJCkxMVHDhg1TZGSkOnfurFmzZqmgoEDx8fGSpKFDhyo4OFjTp0+XJG3dulXZ2dmKiIhQdna2pkyZIofDofHjxzu3+cQTT6hr166aNm2aBg4cqG3btum1117Ta6+99gt1EwAA3MrcDiyDBg3SyZMnNWnSJOXk5CgiIkIpKSkKDAyUJGVlZbmM8i0sLFRSUpIOHz4sX19fxcbGavHixfL393e26dSpk5YvX66JEydq6tSpaty4sWbNmqUhQ4Zcfw8BAMAtz+3nsFhVfn6+atasqbNnzzKGBcCvWnFxsS5evFjRZQCSpKpVq8rT07PU5df6/e32GRYAgDUZY5STk6MzZ85UdCmAC39/f9WvX/+6npNGYAGASuJyWKlXr56qVavGQzRR4Ywx+v77750/39OgQYNyb4vAAgCVQHFxsTOs1KlTp6LLAZx8fHwkSXl5eapXr16Zl4fKUq5fawYAWMvlMSvVqlWr4EqAK10+Lq9nbBWBBQAqES4DwYp+ieOSwAIAACyPwAIAACyPQbcAUMmFTvjgpu7v6Iv33NT9VYRFixZp3Lhxv+gt5OvWrdMdd9yhb7/91uXhqvgBZ1gAAMBVXbhwoUL3T2ABAFSoXr16acyYMRo3bpxq1aqlwMBALViwwPk7dTVq1FDTpk21evVq5zpffPGF+vbtK19fXwUGBuqhhx7SN99841yekpKi7t27y9/fX3Xq1NHvfvc7HTp0yLn86NGjstlsWrZsme644w5Vq1ZN4eHh2rJly1XrXbduneLj43X27FnZbDbZbDZNmTJFklRUVKSnnnpKwcHBql69urp06aJ169Y51z127Jj69eunWrVqqXr16mrTpo1WrVqlo0eP6o477pAk1apVSzabTcOHD79qLf/3f/+ndu3aycfHR3Xq1FF0dLQKCgqcyxcuXKg2bdrI29tbDRo0UEJCgnNZVlaW4uLi5OvrKz8/Pw0cOFC5ubnO5VOmTFFERIT+9a9/qXHjxrLb7ZKkM2fO6JFHHlFAQID8/PzUu3dv7d69+6q1Xi8CCwCgwv373/9W3bp1tW3bNo0ZM0ajRo3SgAED1LVrV+3cuVN33XWXHnroIX3//fc6c+aMevfurdtuu007duxQSkqKcnNzNXDgQOf2CgoKlJiYqB07dig1NVUeHh7q37+/HA6Hy36fe+45PfXUU0pPT1fz5s01ePBgXbp0qcxau3btqlmzZsnPz08nTpzQiRMn9NRTT0mSEhIStGXLFi1ZskSff/65BgwYoLvvvluZmZmSpNGjR6uoqEiffvqp9uzZo7/+9a/y9fVVSEiIkpOTJUkZGRk6ceKEZs+eXWYdJ06c0ODBg/XHP/5R+/bt07p163Tffffp8i/uzJs3T6NHj9ajjz6qPXv26N1331XTpk0lSQ6HQ3FxcTp9+rTWr1+vtWvX6vDhwxo0aJDLPg4ePKjk5GQtW7ZM6enpkqQBAwYoLy9Pq1evVlpamjp06KA+ffro9OnTZdZ7vfgtIQCoBAoLC3XkyBGXv4Qvs/oYll69eqm4uFgbNmyQ9MND8GrWrKn77rtPb7zxhqQfnuLboEEDbdmyRR999JE2bNigDz/80LmNr776SiEhIcrIyFDz5s2v2Mc333yjgIAA7dmzR23bttXRo0fVuHFj/etf/9LDDz8sSdq7d6/atGmjffv2qWXLlmXWXNIYlqysLIWFhSkrK0tBQUHO+dHR0ercubOmTZum9u3b6/7779fkyZOv2Ka7Y1h27typjh076ujRo2rUqNEVy4ODgxUfH68XXnjhimVr165V3759deTIEYWEhLj0f9u2berUqZOmTJmiadOmKTs7WwEBAZKkjRs36p577lFeXp68vb2d22vatKnGjx+vRx99tMRayzo+r/X7mzMsAIAK1759e+d/e3p6qk6dOmrXrp1zXmBgoKQfnpa6e/duffLJJ/L19XVOlwPG5cs+mZmZGjx4sMLCwuTn56fQ0FBJP4SK0vZ7+bHxlx8j7649e/aouLhYzZs3d6lt/fr1zrrGjh2rF154Qd26ddPkyZP1+eefl2tfkhQeHq4+ffqoXbt2GjBggBYsWKBvv/3W2Yevv/5affr0KXHdffv2KSQkxBlWJKl169by9/fXvn37nPMaNWrkDCuStHv3bp07d0516tRx6eORI0dcLrndCNwlBACocFWrVnV5bbPZXOZdfvCYw+HQuXPn1K9fP/31r3+9YjuXQ0e/fv3UqFEjLViwQEFBQXI4HGrbtu0VA0dL20d5nDt3Tp6enkpLS7vi8fO+vr6SpEceeUQxMTH64IMPtGbNGk2fPl0zZszQmDFj3N6fp6en1q5dq82bN2vNmjWaM2eOnnvuOW3dulV169YtVx9+rnr16i6vz507pwYNGriMy7nsRt/ZRGABANxSOnTooOTkZIWGhqpKlSu/xk6dOqWMjAwtWLBAPXr0kPTDpYxfkpeXl4qLi13m3XbbbSouLlZeXp5zvyUJCQnRyJEjNXLkSE2cOFELFizQmDFj5OXlJUlXbLcsNptN3bp1U7du3TRp0iQ1atRIy5cvV2JiokJDQ5WamuoczPtTrVq10vHjx3X8+HGXS0JnzpxR69atS91fhw4dlJOToypVqjjPWt0sXBICANxSRo8erdOnT2vw4MHavn27Dh06pA8//FDx8fEqLi5WrVq1VKdOHb322ms6ePCgPv74YyUmJv6iNYSGhurcuXNKTU3VN998o++//17NmzfXkCFDNHToUC1btkxHjhzRtm3bNH36dH3wwQ/jiMaNG6cPP/xQR44c0c6dO/XJJ5+oVatWkn64/GKz2fT+++/r5MmTOnfuXJk1bN26VdOmTdOOHTuUlZWlZcuW6eTJk87tTZkyRTNmzNA//vEPZWZmaufOnZozZ46kH8bVtGvXTkOGDNHOnTu1bds2DR06VD179lRkZGSp+4yOjlZUVJTuvfderVmzRkePHtXmzZv13HPPaceOHb/EW1sqAgsA4JYSFBSkTZs2qbi4WHfddZfatWuncePGyd/fXx4eHvLw8NCSJUuUlpamtm3b6oknntBLL730i9bQtWtXjRw5UoMGDVJAQID+9re/SZJef/11DR06VE8++aRatGihe++9V9u3b1fDhg0l/XD2ZPTo0WrVqpXuvvtuNW/eXP/85z8l/TBI9vnnn9eECRMUGBjocgtySfz8/PTpp58qNjZWzZs3V1JSkmbMmKG+fftKkoYNG6ZZs2bpn//8p9q0aaPf/e53zruVbDabVq5cqVq1aun2229XdHS0wsLCtHTp0jL3abPZtGrVKt1+++2Kj49X8+bN9Yc//EHHjh1zjjO6UbhLCAAqgbLuwgAqGncJAQCAXwUCCwAAP3P5KbolTdOmTbspNWRlZZVag6+v7xW3aFd23CUEAMDP/Otf/9L58+dLXFa7du2bUkNQUJDz6bKlLf81IbAAAPAzwcHBFV2CqlSp4nyUPrgkBAAAbgEEFgCoRCrJjZ+oZH6J45LAAgCVwOVHzH///fcVXAlwpcvH5c9/gsEdjGEBgErA09NT/v7+zh/uq1atmvO3cYCKYozR999/r7y8PPn7+1/xG0vuILAAQCVRv359SeX/tWHgRvH393cen+VFYAGASsJms6lBgwaqV6+eLl68WNHlAJJ+uAx0PWdWLiOwAEAl4+np+Yt8QQBWwqBbAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeQQWAABgeeUKLHPnzlVoaKjsdru6dOmibdu2ldr24sWLmjp1qpo0aSK73a7w8HClpKS4tJkyZYpsNpvL1LJly/KUBgAAKiG3A8vSpUuVmJioyZMna+fOnQoPD1dMTIzy8vJKbJ+UlKT58+drzpw52rt3r0aOHKn+/ftr165dLu3atGmjEydOOKeNGzeWr0cAAKDScTuwzJw5UyNGjFB8fLxat26tV199VdWqVdPChQtLbL948WI9++yzio2NVVhYmEaNGqXY2FjNmDHDpV2VKlVUv35951S3bt3y9QgAAFQ6bgWWCxcuKC0tTdHR0T9uwMND0dHR2rJlS4nrFBUVyW63u8zz8fG54gxKZmamgoKCFBYWpiFDhigrK6vMWoqKipSfn+8yAQCAysmtwPLNN9+ouLhYgYGBLvMDAwOVk5NT4joxMTGaOXOmMjMz5XA4tHbtWi1btkwnTpxwtunSpYsWLVqklJQUzZs3T0eOHFGPHj303XfflVrL9OnTVbNmTecUEhLiTlcAAMAt5IbfJTR79mw1a9ZMLVu2lJeXlxISEhQfHy8Pjx933bdvXw0YMEDt27dXTEyMVq1apTNnzuidd94pdbsTJ07U2bNnndPx48dvdFcAAEAFcSuw1K1bV56ensrNzXWZn5ubq/r165e4TkBAgFasWKGCggIdO3ZM+/fvl6+vr8LCwkrdj7+/v5o3b66DBw+W2sbb21t+fn4uEwAAqJzcCixeXl7q2LGjUlNTnfMcDodSU1MVFRVV5rp2u13BwcG6dOmSkpOTFRcXV2rbc+fO6dChQ2rQoIE75QEAgErK7UtCiYmJWrBggf79739r3759GjVqlAoKChQfHy9JGjp0qCZOnOhsv3XrVi1btkyHDx/Whg0bdPfdd8vhcGj8+PHONk899ZTWr1+vo0ePavPmzerfv788PT01ePDgX6CLAADgVlfF3RUGDRqkkydPatKkScrJyVFERIRSUlKcA3GzsrJcxqcUFhYqKSlJhw8flq+vr2JjY7V48WL5+/s723z11VcaPHiwTp06pYCAAHXv3l2fffaZAgICrr+HAADglmczxpiKLuKXkJ+fr5o1a+rs2bOMZwEA4BZxrd/f/JYQAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwvHIFlrlz5yo0NFR2u11dunTRtm3bSm178eJFTZ06VU2aNJHdbld4eLhSUlJKbf/iiy/KZrNp3Lhx5SkNAABUQm4HlqVLlyoxMVGTJ0/Wzp07FR4erpiYGOXl5ZXYPikpSfPnz9ecOXO0d+9ejRw5Uv3799euXbuuaLt9+3bNnz9f7du3d78nAACg0nI7sMycOVMjRoxQfHy8WrdurVdffVXVqlXTwoULS2y/ePFiPfvss4qNjVVYWJhGjRql2NhYzZgxw6XduXPnNGTIEC1YsEC1atUqX28AAECl5FZguXDhgtLS0hQdHf3jBjw8FB0drS1btpS4TlFRkex2u8s8Hx8fbdy40WXe6NGjdc8997hsGwAAQJKquNP4m2++UXFxsQIDA13mBwYGav/+/SWuExMTo5kzZ+r2229XkyZNlJqaqmXLlqm4uNjZZsmSJdq5c6e2b99+zbUUFRWpqKjI+To/P9+drgAAgFvIDb9LaPbs2WrWrJlatmwpLy8vJSQkKD4+Xh4eP+z6+PHjevzxx/XWW29dcSamLNOnT1fNmjWdU0hIyI3qAgAAqGBuBZa6devK09NTubm5LvNzc3NVv379EtcJCAjQihUrVFBQoGPHjmn//v3y9fVVWFiYJCktLU15eXnq0KGDqlSpoipVqmj9+vX6xz/+oSpVqricifmpiRMn6uzZs87p+PHj7nQFAADcQtwKLF5eXurYsaNSU1Od8xwOh1JTUxUVFVXmuna7XcHBwbp06ZKSk5MVFxcnSerTp4/27Nmj9PR05xQZGakhQ4YoPT1dnp6eJW7P29tbfn5+LhMAAKic3BrDIkmJiYkaNmyYIiMj1blzZ82aNUsFBQWKj4+XJA0dOlTBwcGaPn26JGnr1q3Kzs5WRESEsrOzNWXKFDkcDo0fP16SVKNGDbVt29ZlH9WrV1edOnWumA8AAH6d3A4sgwYN0smTJzVp0iTl5OQoIiJCKSkpzoG4WVlZzvEpklRYWKikpCQdPnxYvr6+io2N1eLFi+Xv7/+LdQIAAFRuNmOMqegifgn5+fmqWbOmzp49y+UhAABuEdf6/c1vCQEAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMsjsAAAAMurUtEFVKTQCR/c8H0cffGeG74PAAAqO86wAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwAAAAyytXYJk7d65CQ0Nlt9vVpUsXbdu2rdS2Fy9e1NSpU9WkSRPZ7XaFh4crJSXFpc28efPUvn17+fn5yc/PT1FRUVq9enV5SgMAAJWQ24Fl6dKlSkxM1OTJk7Vz506Fh4crJiZGeXl5JbZPSkrS/PnzNWfOHO3du1cjR45U//79tWvXLmeb3/zmN3rxxReVlpamHTt2qHfv3oqLi9OXX35Z/p4BAIBKw2aMMe6s0KVLF3Xq1EmvvPKKJMnhcCgkJERjxozRhAkTrmgfFBSk5557TqNHj3bOu//+++Xj46M333yz1P3Url1bL730kh5++OFrqis/P181a9bU2bNn5efnd03rhE744JraXY+jL95zw/cBAMCt6lq/v906w3LhwgWlpaUpOjr6xw14eCg6OlpbtmwpcZ2ioiLZ7XaXeT4+Ptq4cWOJ7YuLi7VkyRIVFBQoKiqq1FqKioqUn5/vMgEAgMrJrcDyzTffqLi4WIGBgS7zAwMDlZOTU+I6MTExmjlzpjIzM+VwOLR27VotW7ZMJ06ccGm3Z88e+fr6ytvbWyNHjtTy5cvVunXrUmuZPn26atas6ZxCQkLc6QoAALiF3PC7hGbPnq1mzZqpZcuW8vLyUkJCguLj4+Xh4brrFi1aKD09XVu3btWoUaM0bNgw7d27t9TtTpw4UWfPnnVOx48fv9FdAQAAFaSKO43r1q0rT09P5ebmuszPzc1V/fr1S1wnICBAK1asUGFhoU6dOqWgoCBNmDBBYWFhLu28vLzUtGlTSVLHjh21fft2zZ49W/Pnzy9xu97e3vL29nanfFgY44kAAGVx6wyLl5eXOnbsqNTUVOc8h8Oh1NTUMsebSJLdbldwcLAuXbqk5ORkxcXFldne4XCoqKjInfIAAEAl5dYZFklKTEzUsGHDFBkZqc6dO2vWrFkqKChQfHy8JGno0KEKDg7W9OnTJUlbt25Vdna2IiIilJ2drSlTpsjhcGj8+PHObU6cOFF9+/ZVw4YN9d133+k///mP1q1bpw8//PAX6iYAALiVuR1YBg0apJMnT2rSpEnKyclRRESEUlJSnANxs7KyXManFBYWKikpSYcPH5avr69iY2O1ePFi+fv7O9vk5eVp6NChOnHihGrWrKn27dvrww8/1J133nn9PQQAALc8t5/DYlU8h+XWxmcBAL9ON+Q5LAAAABWBwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyvSkUXAMBaQid8cMP3cfTFe274PgBULpxhAQAAlkdgAQAAlkdgAQAAlkdgAQAAlsegW+AXxIBVALgxOMMCAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsj8ACAAAsr1yBZe7cuQoNDZXdbleXLl20bdu2UttevHhRU6dOVZMmTWS32xUeHq6UlBSXNtOnT1enTp1Uo0YN1atXT/fee68yMjLKUxoAAKiE3A4sS5cuVWJioiZPnqydO3cqPDxcMTExysvLK7F9UlKS5s+frzlz5mjv3r0aOXKk+vfvr127djnbrF+/XqNHj9Znn32mtWvX6uLFi7rrrrtUUFBQ/p4BAIBKw+3AMnPmTI0YMULx8fFq3bq1Xn31VVWrVk0LFy4ssf3ixYv17LPPKjY2VmFhYRo1apRiY2M1Y8YMZ5uUlBQNHz5cbdq0UXh4uBYtWqSsrCylpaWVv2cAAKDScCuwXLhwQWlpaYqOjv5xAx4eio6O1pYtW0pcp6ioSHa73WWej4+PNm7cWOp+zp49K0mqXbu2O+UBAIBKyq3A8s0336i4uFiBgYEu8wMDA5WTk1PiOjExMZo5c6YyMzPlcDi0du1aLVu2TCdOnCixvcPh0Lhx49StWze1bdu21FqKioqUn5/vMgEAgMrpht8lNHv2bDVr1kwtW7aUl5eXEhISFB8fLw+Pknc9evRoffHFF1qyZEmZ250+fbpq1qzpnEJCQm5E+QAAwAKquNO4bt268vT0VG5ursv83Nxc1a9fv8R1AgICtGLFChUWFurUqVMKCgrShAkTFBYWdkXbhIQEvf/++/r000/1m9/8psxaJk6cqMTEROfr/Px8QgsAp9AJH9zwfRx98Z4bvg8AP3DrDIuXl5c6duyo1NRU5zyHw6HU1FRFRUWVua7dbldwcLAuXbqk5ORkxcXFOZcZY5SQkKDly5fr448/VuPGja9ai7e3t/z8/FwmAABQObl1hkWSEhMTNWzYMEVGRqpz586aNWuWCgoKFB8fL0kaOnSogoODNX36dEnS1q1blZ2drYiICGVnZ2vKlClyOBwaP368c5ujR4/Wf/7zH61cuVI1atRwjoepWbOmfHx8fol+AgCAW5jbgWXQoEE6efKkJk2apJycHEVERCglJcU5EDcrK8tlfEphYaGSkpJ0+PBh+fr6KjY2VosXL5a/v7+zzbx58yRJvXr1ctnX66+/ruHDh7vfKwAAUKm4HVikH8aaJCQklLhs3bp1Lq979uypvXv3lrk9Y0x5ygAAAL8S/JYQAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwPAILAACwvHL9lhCsJXTCBzd0+0dfvOeGbh8AgKvhDAsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALA8AgsAALC8cgWWuXPnKjQ0VHa7XV26dNG2bdtKbXvx4kVNnTpVTZo0kd1uV3h4uFJSUlzafPrpp+rXr5+CgoJks9m0YsWK8pQFAAAqKbcDy9KlS5WYmKjJkydr586dCg8PV0xMjPLy8kpsn5SUpPnz52vOnDnau3evRo4cqf79+2vXrl3ONgUFBQoPD9fcuXPL3xMAAFBpuR1YZs6cqREjRig+Pl6tW7fWq6++qmrVqmnhwoUltl+8eLGeffZZxcbGKiwsTKNGjVJsbKxmzJjhbNO3b1+98MIL6t+/f/l7AgAAKi23AsuFCxeUlpam6OjoHzfg4aHo6Ght2bKlxHWKiopkt9td5vn4+Gjjxo3lKNd1u/n5+S4TAAConKq40/ibb75RcXGxAgMDXeYHBgZq//79Ja4TExOjmTNn6vbbb1eTJk2UmpqqZcuWqbi4uPxVS5o+fbqef/7569oGAFhd6IQPbuj2j754zw3dPvBLueF3Cc2ePVvNmjVTy5Yt5eXlpYSEBMXHx8vD4/p2PXHiRJ09e9Y5HT9+/BeqGAAAWI1bqaFu3bry9PRUbm6uy/zc3FzVr1+/xHUCAgK0YsUKFRQU6NixY9q/f798fX0VFhZW/qoleXt7y8/Pz2UCAACVk1uBxcvLSx07dlRqaqpznsPhUGpqqqKiospc1263Kzg4WJcuXVJycrLi4uLKVzEAAPjVcWsMiyQlJiZq2LBhioyMVOfOnTVr1iwVFBQoPj5ekjR06FAFBwdr+vTpkqStW7cqOztbERERys7O1pQpU+RwODR+/HjnNs+dO6eDBw86Xx85ckTp6emqXbu2GjZseL19BAAAtzi3A8ugQYN08uRJTZo0STk5OYqIiFBKSopzIG5WVpbL+JTCwkIlJSXp8OHD8vX1VWxsrBYvXix/f39nmx07duiOO+5wvk5MTJQkDRs2TIsWLSpn1wAAQGXhdmCRpISEBCUkJJS4bN26dS6ve/bsqb1795a5vV69eskYU55SAADArwC/JQQAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyPwAIAACyvSkUXAACo3EInfHDD93H0xXtu+D5QsTjDAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALI/AAgAALK9KRRcAAMCtIHTCBzd8H0dfvOeG7+NWVa4zLHPnzlVoaKjsdru6dOmibdu2ldr24sWLmjp1qpo0aSK73a7w8HClpKRc1zYBAMCvi9uBZenSpUpMTNTkyZO1c+dOhYeHKyYmRnl5eSW2T0pK0vz58zVnzhzt3btXI0eOVP/+/bVr165ybxMAAPy6uB1YZs6cqREjRig+Pl6tW7fWq6++qmrVqmnhwoUltl+8eLGeffZZxcbGKiwsTKNGjVJsbKxmzJhR7m0CAIBfF7fGsFy4cEFpaWmaOHGic56Hh4eio6O1ZcuWEtcpKiqS3W53mefj46ONGzeWe5uXt1tUVOR8ffbsWUlSfn7+NffHUfT9NbctL3fqKa8b3Y/K0AeJflyrytAHiX5cq8rQB4l+3Mou99kYU3ZD44bs7GwjyWzevNll/tNPP206d+5c4jqDBw82rVu3NgcOHDDFxcVmzZo1xsfHx3h5eZV7m8YYM3nyZCOJiYmJiYmJqRJMx48fLzOD3PC7hGbPnq0RI0aoZcuWstlsatKkieLj46/7cs/EiROVmJjofO1wOHT69GnVqVNHNpvtesu+Qn5+vkJCQnT8+HH5+fn94tu/WeiHdVSGPkiVox+VoQ8S/bCSytAH6eb0wxij7777TkFBQWW2cyuw1K1bV56ensrNzXWZn5ubq/r165e4TkBAgFasWKHCwkKdOnVKQUFBmjBhgsLCwsq9TUny9vaWt7e3yzx/f393ulMufn5+t/TBdxn9sI7K0AepcvSjMvRBoh9WUhn6IN34ftSsWfOqbdwadOvl5aWOHTsqNTXVOc/hcCg1NVVRUVFlrmu32xUcHKxLly4pOTlZcXFx171NAADw6+D2JaHExEQNGzZMkZGR6ty5s2bNmqWCggLFx8dLkoYOHarg4GBNnz5dkrR161ZlZ2crIiJC2dnZmjJlihwOh8aPH3/N2wQAAL9ubgeWQYMG6eTJk5o0aZJycnIUERGhlJQUBQYGSpKysrLk4fHjiZvCwkIlJSXp8OHD8vX1VWxsrBYvXuxy+eZq27QCb29vTZ48+YrLULca+mEdlaEPUuXoR2Xog0Q/rKQy9EGyVj9sxlztPiIAAICKxY8fAgAAyyOwAAAAyyOwAAAAyyOwAAAAyyOwABWIMe8AcG1u+KP5AZTO29tbu3fvVqtWrSq6FKBCbdu2TVu2bFFOTo4kqX79+oqKilLnzp0ruLLyOXLkiA4ePKgGDRqobdu2FV1OpcBtzeV0/PhxTZ48+bp/E+lGO3/+vNLS0lS7dm21bt3aZVlhYaHeeecdDR06tIKqu3b79u3TZ599pqioKLVs2VL79+/X7NmzVVRUpAcffFC9e/eu6BLL9NPfvfqp2bNn68EHH1SdOnUkSTNnzryZZV23goICvfPOO85/mAcPHuzsi9VduHBBK1asuOJLsmvXroqLi5OXl1cFV+g+Y4zWrVvn/DxiYmJUtWrVii6rTHl5ebr//vu1adMmNWzY0Pn8rdzcXGVlZalbt25KTk5WvXr1KrjS0j322GP629/+Jl9fX50/f14PPfSQli9fLmOMbDabevbsqXfffVe+vr4VXeo1sWx4LPOnEVGq9PR04+HhUdFllCkjI8M0atTI2Gw24+HhYW6//Xbz9ddfO5fn5ORYvg/GGLN69Wrj5eVlateubex2u1m9erUJCAgw0dHRpnfv3sbT09OkpqZWdJllstlsJiIiwvTq1ctlstlsplOnTqZXr17mjjvuqOgyr6pVq1bm1KlTxhhjsrKyTGhoqKlZs6bp1KmTqV27tqlXr545fPhwBVd5dZmZmSYsLMzY7XbTs2dPM3DgQDNw4EDTs2dPY7fbTdOmTU1mZmZFl3lVffv2NWfOnDHGGHPq1CnTpUsXY7PZTEBAgPHw8DAtW7Y0eXl5FVxl2e6//34TFRVl9u/ff8Wy/fv3m65du5rf//73FVDZtfPw8DC5ubnGGGMmTpxofvOb35iPP/7YFBQUmI0bN5omTZqYCRMmVHCVV5ebm2u6d+9ubDabadSokencubPp3Lmz83uke/fuzn5WBAJLKVauXFnm9PLLL1v+y/7ee+8199xzjzl58qTJzMw099xzj2ncuLE5duyYMebWCSxRUVHmueeeM8YY8/bbb5tatWqZZ5991rl8woQJ5s4776yo8q7J9OnTTePGja8IVlWqVDFffvllBVXlPpvN5vwHa8iQIaZr167OL8zvvvvOREdHm8GDB1dkidckOjraxMXFmbNnz16x7OzZsyYuLs7cddddFVCZe376eYwaNcq0bt3aGRiPHz9uOnbsaEaOHFmRJV6Vr6+v2blzZ6nLd+zYYXx9fW9iRe776efQtm1b85///Mdl+cqVK03z5s0rojS3WD08ElhKcfmshM1mK3Wy+pd9vXr1zOeff+587XA4zMiRI03Dhg3NoUOHbpnA4ufn5/xrt7i42FSpUsXlH7g9e/aYwMDAiirvmm3bts00b97cPPnkk+bChQvGmFs7sISFhZk1a9a4LN+0aZMJCQmpiNLc4uPjY/bs2VPq8s8//9z4+PjcxIrK56efR4sWLczKlStdln/00UemcePGFVHaNatTp45Zt25dqcs/+eQTU6dOnZtYkftsNpvzTFbdunXNF1984bL86NGjt8TxZPXwyF1CpWjQoIGWLVsmh8NR4rRz586KLvGqzp8/rypVfhxXbbPZNG/ePPXr1089e/bUgQMHKrA699hsNkmSh4eH7Ha7y0+R16hRQ2fPnq2o0q5Zp06dlJaWppMnTyoyMlJffPGFs1+3kss1FxYWqkGDBi7LgoODdfLkyYooyy3+/v46evRoqcuPHj3q8ntnVnb58/j222/VpEkTl2VNmzbV119/XRFlXbNBgwZp2LBhWr58ufLz853z8/PztXz5csXHx2vw4MEVWOG1+dOf/qTExER5eHhc8Z6fOnVK1atXr6DKrp23t7fLZ/Bz3333XYX+phB3CZWiY8eOSktLU1xcXInLbTab5W9JbdmypXbs2HHFHSivvPKKJOl//ud/KqIst4WGhiozM9P5j/GWLVvUsGFD5/KsrKwrvjitytfXV//+97+1ZMkSRUdHq7i4uKJLclufPn1UpUoV5efnKyMjw+UOiGPHjt0Sg24feeQRDR06VH/605/Up08fl4GeqampeuGFFzRmzJgKrvLaDB8+XN7e3rp48aKOHDmiNm3aOJfl5ORYPnjNnDlTDodDf/jDH3Tp0iXnYOcLFy6oSpUqevjhh/X3v/+9gqss2+23366MjAxJUuvWrXXs2DGX5atWrXL5XKzqcnh8+eWX1adPH/n5+Un6ITympqYqMTGxQsMjgaUUTz/9tAoKCkpd3rRpU33yySc3sSL39e/fX2+//bYeeuihK5a98sorcjgcevXVVyugMveMGjXK5Yv957cIrl692vJ3Cf3cH/7wB3Xv3l1paWlq1KhRRZdzzSZPnuzy+ud3Pbz33nvq0aPHzSypXKZOnarq1avrpZde0pNPPuk8S2GMUf369fXMM89o/PjxFVzl1Q0bNsz533Fxcfr+++9dlicnJysiIuImV+Ueb29vzZs3T3/961+VlpbmcmdKx44dnV+aVrZu3boylz/wwAMaPnz4Tanlelg9PHJbM4BftSNHjrh8STZu3LiCK/rlFBQUyNPTU3a7vaJLwS0kPz/fkuGRMSwAftUaN26sqKgoRUVFOcPK8ePH9cc//rGCK7t+p0+f1mOPPVbRZVzV+fPntXHjRu3du/eKZYWFhXrjjTcqoCr3VIY+SD888yo5Odn5XKXbbrtN77zzjsaNG6ePP/64QmvjDAsA/Mzu3bvVoUOHW3KM0U/dCv04cOCA7rrrLmVlZclms6l79+56++23FRQUJOmHcUVBQUG3XB+WLFniHFt3K/RBklJSUhQXFydfX199//33Wr58uYYOHarw8HA5HA6tX79ea9asqbBL8IxhAfCr8+6775a5/PDhwzepkutTGfrxzDPPqG3bttqxY4fOnDmjcePGqXv37lq3bp3L4HorK6kP3bp1u6X6IP0wtuvpp5/WCy+8oCVLluiBBx7QqFGj9Je//EWSNHHiRL344osVFlg4wwLgV8fDw+Oqd/rZbDbL/0VcGfoRGBiojz76SO3atZP0w8Dnxx57TKtWrdInn3yi6tWrW/7sRGXogyTVrFlTaWlpatq0qRwOh7y9vbVt2zbddtttkqQvvvhC0dHRzrEtNxtjWAD86lSG5yxJlaMfleF5UZWhD5dZ+ZlXBBYAvzqXn7NUmlvhOUtS5ejH5edF/dwrr7yiuLi4W+J5UZWhD9KPz7y6zGrPvCKwAPjVefrpp9W1a9dSl98Kz1mSKkc/Lj8vqiSvvPKKBg8ebPnQVRn6IJX8zKufnjmq6GdeMYYFAABYHmdYAACA5RFYAACA5RFYAACA5RFYAACA5RFYgHJYtGiR/P39b/h+bDabVqxYcc3t161bJ5vNpjNnzpTaZsqUKZb8Bd/hw4fr3nvvvSn7Cg0N1axZs5yvc3JydOedd6p69erOz9Xd974y+vn7dDVHjx6VzWZTenp6qW1u1v87qHx4ND9gYSdOnFCtWrUquoxKZ/v27apevbrz9csvv6wTJ04oPT3d+aAs3vsr3yegIhFYcMsyxqi4uNjlOQGVTf369Su6hGt28eJFVa1ataLLuCYBAQEurw8dOqSOHTuqWbNmznnX+95fuHBBXl5e17WNivbz98nKbqXjD+XDJSHcNL169VJCQoISEhJUs2ZN1a1bV3/605+cD1RavHixIiMjVaNGDdWvX18PPPCA8vLynOtfvtyxevVqdezYUd7e3tq4caMOHTqkuLg4BQYGytfXV506ddJHH33ksu/Q0FC98MILGjp0qHx9fdWoUSO9++67OnnypPPXSdu3b1/i0yqvxcmTJxUZGan+/furqKjIWWtqaqoiIyNVrVo1de3aVRkZGS7rrVy5Uh06dJDdbldYWJief/55Xbp0ybn855clNm/erIiICNntdkVGRmrFihUlnoJPS0src7+SNH/+fIWEhKhatWoaOHCgyyO3HQ6Hpk6dqt/85jfy9vZWRESEUlJSnMsvn/pfunSpevbsKbvdrrfeekvHjh1Tv379VKtWLVWvXl1t2rTRqlWrnOt9+eWX+t3vfic/Pz/VqFFDPXr00KFDh0p8T1NSUtS9e3f5+/urTp06+t3vfufS9sKFC0pISFCDBg1kt9vVqFEjTZ8+XdIPYXbKlClq2LChvL29FRQUpLFjxzrX/emljtDQUCUnJ+uNN96QzWbT8OHDS3zvjx8/roEDB8rf31+1a9dWXFycjh496lx++ZLWX/7yFwUFBalFixYl9utahYaGatq0afrjH/+oGjVqqGHDhnrttdeuad3Ln8+yZct0xx13qFq1agoPD9eWLVtc2m3cuFE9evSQj4+PQkJCNHbsWBUUFLjU8NNLQvv371f37t1lt9vVunVrffTRRyVeOjt8+HCZ+5WkFStWqFmzZrLb7YqJidHx48ddls+bN09NmjSRl5eXWrRoocWLF7ssv/z4+//5n/9R9erV9Ze//EXffvuthgwZooCAAPn4+KhZs2Z6/fXXr+k9wy3AADdJz549ja+vr3n88cfN/v37zZtvvmmqVatmXnvtNWOMMf/7v/9rVq1aZQ4dOmS2bNlioqKiTN++fZ3rf/LJJ0aSad++vVmzZo05ePCgOXXqlElPTzevvvqq2bNnjzlw4IBJSkoydrvdHDt2zLluo0aNTO3atc2rr75qDhw4YEaNGmX8/PzM3Xffbd555x2TkZFh7r33XtOqVSvjcDiu2pfXX3/d1KxZ0xhjTFZWlmnRooUZNmyYuXTpkkutXbp0MevWrTNffvml6dGjh+natatzG59++qnx8/MzixYtMocOHTJr1qwxoaGhZsqUKc42kszy5cuNMcacPXvW1K5d2zz44IPmyy+/NKtWrTLNmzc3ksyuXbuueb+TJ0821atXN7179za7du0y69evN02bNjUPPPCAs83MmTONn5+fefvtt83+/fvN+PHjTdWqVc2BAweMMcYcOXLESDKhoaEmOTnZHD582Hz99dfmnnvuMXfeeaf5/PPPzaFDh8x7771n1q9fb4wx5quvvjK1a9c29913n9m+fbvJyMgwCxcuNPv37zfGGDNs2DATFxfnrOH//u//THJyssnMzDS7du0y/fr1M+3atTPFxcXGGGNeeuklExISYj799FNz9OhRs2HDBvOf//zHGGPMf//7X+Pn52dWrVpljh07ZrZu3eo8zi4fDy+//LIxxpi8vDxz9913m4EDB5oTJ06YM2fOXPHeX7hwwbRq1cr88Y9/NJ9//rnZu3eveeCBB0yLFi1MUVGRs35fX1/z0EMPmS+++MJ88cUXVz2OynL5mJ07d67JzMw006dPNx4eHs73qyyXP5+WLVua999/32RkZJjf//73plGjRubixYvGGGMOHjxoqlevbl5++WVz4MABs2nTJnPbbbeZ4cOHl/g+Xbp0ybRo0cLceeedJj093WzYsMF07tzZ5X26lv2+/vrrpmrVqiYyMtJs3rzZ7Nixw3Tu3NnlGF22bJmpWrWqmTt3rsnIyDAzZswwnp6e5uOPP3a2kWTq1atnFi5caA4dOmSOHTtmRo8ebSIiIsz27dvNkSNHzNq1a8277757XZ8DrIPAgpumZ8+eVwSCZ555xrRq1arE9tu3bzeSzHfffWeM+fHLeMWKFVfdV5s2bcycOXOcrxs1amQefPBB5+sTJ04YSeZPf/qTc96WLVuMJHPixImrbv9yYNm/f78JCQkxY8eOdenX5Vo/+ugj57wPPvjASDLnz583xhjTp08fM23aNJftLl682DRo0MD5+qdfBvPmzTN16tRxrm+MMQsWLCgxsJS138mTJxtPT0/z1VdfOdusXr3aeHh4OPseFBRk/vKXv7jU1qlTJ/PYY48ZY378Ypo1a5ZLm3bt2rkErp+aOHGiady4sblw4UKJy38eWH7u5MmTRpLZs2ePMcaYMWPGmN69e5cYMGfMmGGaN29e6r5++kVsjDFxcXFm2LBhLm1++t4vXrzYtGjRwmVfRUVFxsfHx3z44YfO+gMDA50B5nr9/Jh1OBymXr16Zt68eVdd9/Ln869//cs578svvzSSzL59+4wxxjz88MPm0UcfdVlvw4YNxsPDw3ms/PR9Wr16talSpYrL/x9r164tMbCUtd/XX3/dSDKfffaZs82+ffuMJLN161ZjjDFdu3Y1I0aMcKltwIABJjY21vlakhk3bpxLm379+pn4+Pirvj+4NXFJCDfVb3/7W+evgUpSVFSUMjMzVVxcrLS0NPXr108NGzZUjRo11LNnT0k//ODWT0VGRrq8PnfunJ566im1atVK/v7+8vX11b59+65Yr3379s7/DgwMlCTnz8H/dN5PL0OV5fz58+rRo4fuu+8+zZ4926VfJe3z8o+GXd7+7t27NXXqVPn6+jqnESNG6MSJE/r++++v2FZGRobat28vu93unNe5c+cSaytrv5LUsGFDBQcHO19HRUXJ4XAoIyND+fn5+vrrr9WtWzeXbXbr1k379u1zmffzz2Ls2LF64YUX1K1bN02ePFmff/65c1l6erp69OhxzeMMMjMzNXjwYIWFhcnPz0+hoaGSfjwehg8frvT0dLVo0UJjx47VmjVrnOsOGDBA58+fV1hYmEaMGKHly5e7XGpz1+7du3Xw4EHVqFHD+VnVrl1bhYWFLpep2rVr94uOW/np52iz2VS/fv1rPj5/vn5Jx9+iRYtcjr+YmBg5HA4dOXLkim1lZGQoJCTEZWxPeY+/KlWqqFOnTs7XLVu2lL+/v/P42rdvX7mOv1GjRmnJkiWKiIjQ+PHjtXnz5hLrw62JwAJLKCwsVExMjPz8/PTWW29p+/btWr58uaQfxir81M/vWnjqqae0fPlyTZs2TRs2bFB6erratWt3xXo//aK8HC5KmudwOK6pZm9vb0VHR+v9999XdnZ2iW3K2v65c+f0/PPPKz093Tnt2bNHmZmZLqGkPK6nX+74+WfxyCOP6PDhw3rooYe0Z88eRUZGas6cOZIkHx8ft7bdr18/nT59WgsWLNDWrVu1detWST8eDx06dNCRI0f05z//WefPn9fAgQP1+9//XpIUEhKijIwM/fOf/5SPj48ee+wx3X777bp48WK5+nnu3Dl17NjR5bNKT0/XgQMH9MADD5T6flyvn4c7m83m1ud4tePv//2//+fSn927dyszM1NNmjT5xeq+mcdf3759dezYMT3xxBP6+uuv1adPHz311FO/+H5RMQgsuKkuf+lc9tlnn6lZs2bav3+/Tp06pRdffFE9evRQy5Ytr/kvyU2bNmn48OHq37+/2rVrp/r167sMhrxRPDw8tHjxYnXs2FF33HGHvv76a7fW79ChgzIyMtS0adMrJg+PK//XbNGihfbs2aOioiLnvO3bt5er9qysLJd6P/vsM3l4eKhFixby8/NTUFCQNm3a5LLOpk2b1Lp166tuOyQkRCNHjtSyZcv05JNPasGCBZJ++Kt7w4YN1xQaTp06pYyMDCUlJalPnz5q1aqVvv322yva+fn5adCgQVqwYIGWLl2q5ORknT59WtIPAalfv376xz/+oXXr1mnLli3as2fPVfddkg4dOigzM1P16tW74rO6fBv0raZDhw7au3dvicdfSWeJWrRooePHjys3N9c5r7zH36VLl1wGuGdkZOjMmTNq1aqVJKlVq1blPv4CAgI0bNgwvfnmm5o1a9Y1D1SG9RFYcFNlZWUpMTFRGRkZevvttzVnzhw9/vjjatiwoby8vDRnzhwdPnxY7777rv785z9f0zabNWumZcuWOf9CfOCBB27IX3Ml8fT01FtvvaXw8HD17t1bOTk517zupEmT9MYbb+j555/Xl19+qX379mnJkiVKSkoqsf3lfj366KPat2+fPvzwQ/3973+XpBIvR5XFbrdr2LBh2r17tzZs2KCxY8dq4MCBztP9Tz/9tP76179q6dKlysjI0IQJE5Senq7HH3+8zO2OGzdOH374oY4cOaKdO3fqk08+cX4JJSQkKD8/X3/4wx+0Y8cOZWZmavHixSXewVSrVi3VqVNHr732mg4ePKiPP/5YiYmJLm1mzpypt99+W/v379eBAwf03//+V/Xr15e/v78WLVqk//3f/9UXX3yhw4cP680335SPj48aNWrk1vt02ZAhQ1S3bl3FxcVpw4YNOnLkiNatW6exY8fqq6++Ktc2K9ozzzyjzZs3KyEhQenp6crMzNTKlSuVkJBQYvs777xTTZo00bBhw/T5559r06ZNzmPV3eOvatWqGjNmjLZu3aq0tDQNHz5cv/3tb52XmJ5++mktWrRI8+bNU2ZmpmbOnKlly5Zd9WzJpEmTtHLlSh08eFBffvml3n//fefxh1sfgQU31dChQ3X+/Hl17txZo0eP1uOPP65HH31UAQEBWrRokf773/+qdevWevHFF51fxlczc+ZM1apVS127dlW/fv0UExOjDh063OCe/KhKlSp6++231aZNG/Xu3fuazwzFxMTo/fff15o1a9SpUyf99re/1csvv1zql6qfn5/ee+89paenKyIiQs8995wmTZokSW5fQmratKnuu+8+xcbG6q677lL79u31z3/+07l87NixSkxM1JNPPql27dopJSVF7777rstzSkpSXFys0aNHq1WrVrr77rvVvHlz53br1Kmjjz/+WOfOnVPPnj3VsWNHLViwoMQxLR4eHlqyZInS0tLUtm1bPfHEE3rppZdc2tSoUUN/+9vfFBkZqU6dOuno0aNatWqVPDw85O/vrwULFqhbt25q3769PvroI7333nuqU6eOW+/TZdWqVdOnn36qhg0b6r777lOrVq308MMPq7CwUH5+fuXaZkVr37691q9frwMHDqhHjx667bbbNGnSJAUFBZXY3tPTUytWrNC5c+fUqVMnPfLII3ruueckuX/8VatWTc8884weeOABdevWTb6+vlq6dKlz+b333qvZs2fr73//u9q0aaP58+fr9ddfV69evcrcrpeXlyZOnKj27dvr9ttvl6enp5YsWeJWbbAumzH//0MwgBusV69eioiIcOtR3yjbW2+9pfj4eJ09e9btMSLA9dq0aZO6d++ugwcPXve4F+BqKu8jQoFK6I033lBYWJiCg4O1e/duPfPMMxo4cCBhBTfF8uXL5evrq2bNmungwYN6/PHH1a1bN8IKbgouCQEl6Nu3r8vtnj+dpk2bVmF15eTk6MEHH1SrVq30xBNPaMCAAQwqtKgNGzaUegx5enqWuszX1/eq2542bVqp6/bt2/eG9em7777T6NGj1bJlSw0fPlydOnXSypUrb9j+gJ/ikhBQguzsbJ0/f77EZbVr11bt2rVvckW41Zw/f77U293Pnz9f5lmxpk2blrnt06dPO++G+jkfHx+XZ+wAlQWBBQAAWB6XhAAAgOURWAAAgOURWAAAgOURWAAAgOURWAAAgOURWAAAgOURWAAAgOURWAAAgOX9f6psxhBcT8ZKAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "knn_cv_results.plot(x='param_kneighborsclassifier__n_neighbors', y='mean_test_score',\n", " kind='bar',ylim=(0.9,0.96));" ] }, { "cell_type": "code", "execution_count": 215, "id": "5df79199-2787-48ce-bbe8-23d249062148", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mean_fit_timestd_fit_timemean_score_timestd_score_timeparam_kneighborsclassifier__n_neighborsparamssplit0_test_scoresplit1_test_scoresplit2_test_scoremean_test_scorestd_test_scorerank_test_score
00.0080940.0004040.3316920.0130441{'kneighborsclassifier__n_neighbors': 1}0.9288770.9318470.9404200.9337150.0048941
20.0078540.0002860.4410010.0908813{'kneighborsclassifier__n_neighbors': 3}0.9318770.9232750.9339910.9297140.0046342
40.0077640.0004230.3463110.0108525{'kneighborsclassifier__n_neighbors': 5}0.9318770.9194170.9279900.9264280.0052053
30.0090600.0010030.4097700.0989174{'kneighborsclassifier__n_neighbors': 4}0.9310200.9177030.9284180.9257140.0057634
50.0073900.0002230.3272270.00443310{'kneighborsclassifier__n_neighbors': 10}0.9233080.9112730.9249890.9198570.0061085
10.0087930.0013010.3331240.0162382{'kneighborsclassifier__n_neighbors': 2}0.9125960.9159880.9228460.9171440.0042636
60.0078680.0001410.3797070.05623415{'kneighborsclassifier__n_neighbors': 15}0.9177380.8988430.9134160.9099990.0080847
70.0075940.0002290.3593370.03960420{'kneighborsclassifier__n_neighbors': 20}0.9125960.8997000.9069870.9064280.0052808
80.0077590.0000490.3271070.00292025{'kneighborsclassifier__n_neighbors': 25}0.9087400.8906990.9031290.9008560.0075399
90.0076280.0003270.3277240.00492830{'kneighborsclassifier__n_neighbors': 30}0.8980290.8834120.8967000.8927140.00660010
\n", "
" ], "text/plain": [ " mean_fit_time std_fit_time mean_score_time std_score_time \\\n", "0 0.008094 0.000404 0.331692 0.013044 \n", "2 0.007854 0.000286 0.441001 0.090881 \n", "4 0.007764 0.000423 0.346311 0.010852 \n", "3 0.009060 0.001003 0.409770 0.098917 \n", "5 0.007390 0.000223 0.327227 0.004433 \n", "1 0.008793 0.001301 0.333124 0.016238 \n", "6 0.007868 0.000141 0.379707 0.056234 \n", "7 0.007594 0.000229 0.359337 0.039604 \n", "8 0.007759 0.000049 0.327107 0.002920 \n", "9 0.007628 0.000327 0.327724 0.004928 \n", "\n", " param_kneighborsclassifier__n_neighbors \\\n", "0 1 \n", "2 3 \n", "4 5 \n", "3 4 \n", "5 10 \n", "1 2 \n", "6 15 \n", "7 20 \n", "8 25 \n", "9 30 \n", "\n", " params split0_test_score \\\n", "0 {'kneighborsclassifier__n_neighbors': 1} 0.928877 \n", "2 {'kneighborsclassifier__n_neighbors': 3} 0.931877 \n", "4 {'kneighborsclassifier__n_neighbors': 5} 0.931877 \n", "3 {'kneighborsclassifier__n_neighbors': 4} 0.931020 \n", "5 {'kneighborsclassifier__n_neighbors': 10} 0.923308 \n", "1 {'kneighborsclassifier__n_neighbors': 2} 0.912596 \n", "6 {'kneighborsclassifier__n_neighbors': 15} 0.917738 \n", "7 {'kneighborsclassifier__n_neighbors': 20} 0.912596 \n", "8 {'kneighborsclassifier__n_neighbors': 25} 0.908740 \n", "9 {'kneighborsclassifier__n_neighbors': 30} 0.898029 \n", "\n", " split1_test_score split2_test_score mean_test_score std_test_score \\\n", "0 0.931847 0.940420 0.933715 0.004894 \n", "2 0.923275 0.933991 0.929714 0.004634 \n", "4 0.919417 0.927990 0.926428 0.005205 \n", "3 0.917703 0.928418 0.925714 0.005763 \n", "5 0.911273 0.924989 0.919857 0.006108 \n", "1 0.915988 0.922846 0.917144 0.004263 \n", "6 0.898843 0.913416 0.909999 0.008084 \n", "7 0.899700 0.906987 0.906428 0.005280 \n", "8 0.890699 0.903129 0.900856 0.007539 \n", "9 0.883412 0.896700 0.892714 0.006600 \n", "\n", " rank_test_score \n", "0 1 \n", "2 2 \n", "4 3 \n", "3 4 \n", "5 5 \n", "1 6 \n", "6 7 \n", "7 8 \n", "8 9 \n", "9 10 " ] }, "execution_count": 215, "metadata": {}, "output_type": "execute_result" } ], "source": [ "knn_cv_results.sort_values(by=\"mean_test_score\", ascending = False)" ] }, { "cell_type": "code", "execution_count": 216, "id": "cb38df2c-36ae-449f-ac07-aca19bc46a87", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best Parameters: {'kneighborsclassifier__n_neighbors': 1}\n" ] } ], "source": [ "best_params = grid_search.best_params_\n", "print(\"Best Parameters:\", best_params)" ] }, { "cell_type": "code", "execution_count": 217, "id": "f3fe816c-bd6c-45d8-a9cb-e8c9f3a9765e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best Model: Pipeline(steps=[('kneighborsclassifier', KNeighborsClassifier(n_neighbors=1))])\n" ] } ], "source": [ "best_model = grid_search.best_estimator_\n", "print(\"Best Model:\", best_model)" ] }, { "cell_type": "code", "execution_count": 218, "id": "fcd55cd8-8b7c-4aec-9c02-8d7a0a9ece2c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test accuracy with Best Model: 0.9328571428571428\n" ] } ], "source": [ "# Evaluate the best model\n", "y_pred = best_model.predict(X_test)\n", "accuracy = accuracy_score(y_test, y_pred)\n", "print(\"Test accuracy with Best Model:\", accuracy)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }