{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":82695,"databundleVersionId":9551816,"sourceType":"competition"},{"sourceId":6418393,"sourceType":"datasetVersion","datasetId":3702162}],"dockerImageVersionId":30761,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Eedi - Mining Misconceptions in Mathematics πŸ“„πŸ’»

","metadata":{"id":"5zyCB_WFMSk1"}},{"cell_type":"markdown","source":"### CreditsπŸ“‘\n \n\n \n* [Understanding the competition : EDA and Overview](https://www.kaggle.com/code/mbmmurad/understanding-the-competition-eda-and-overview)\n* [Cossine Similarity](https://www.kaggle.com/code/jaytonde/cosine-similarity-bge)\n ","metadata":{}},{"cell_type":"markdown","source":"### Contents πŸ“š\n\n[The Problem](#first-bullet)\n\n[Data](#second-bullet)\n\n[EAD](#second-bullet)\n\n[Cossine similarity](##first-bullet)\n \n[Validation](##first-bullet)\n","metadata":{"id":"I6-_idNeNzhC"}},{"cell_type":"markdown","source":"# Libraries and Setup πŸ“‘","metadata":{"id":"uJuvc1Kv6duu"}},{"cell_type":"code","source":"# !pip install sentence-transformers","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:15.819088Z","iopub.execute_input":"2024-09-23T18:35:15.819625Z","iopub.status.idle":"2024-09-23T18:35:15.824340Z","shell.execute_reply.started":"2024-09-23T18:35:15.819579Z","shell.execute_reply":"2024-09-23T18:35:15.823284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib.colors import ListedColormap, LinearSegmentedColormap\n\nimport numpy as np\nimport math\n\n\nfrom transformers import AutoTokenizer, AutoModel\n\nfrom IPython.display import display, HTML\nfrom tqdm import tqdm\n\nfrom sklearn.manifold import TSNE\nfrom sklearn.metrics.pairwise import cosine_similarity\n\nimport matplotlib as mpl\nimport gc\n\nimport torch\nimport torch.nn as nn\n\n\n\n","metadata":{"id":"p8H3sFWY6GiG","outputId":"c3393edd-f22d-4ef7-d338-c84b14789104","execution":{"iopub.status.busy":"2024-09-23T18:35:15.826328Z","iopub.execute_input":"2024-09-23T18:35:15.827328Z","iopub.status.idle":"2024-09-23T18:35:15.839555Z","shell.execute_reply.started":"2024-09-23T18:35:15.827278Z","shell.execute_reply":"2024-09-23T18:35:15.838644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration","metadata":{}},{"cell_type":"code","source":"class CFG():\n\n data_folder='/kaggle/input/eedi-mining-misconceptions-in-mathematics/'\n embeddings_model = \"/kaggle/input/multi-qa-mpnet-base-cos-v1\"#all-mpnet-base-v2, multi-qa-mpnet-base-cos-v1\n seed=42\n batch_size = 32\n\n","metadata":{"id":"spOxZiyI7BCp","execution":{"iopub.status.busy":"2024-09-23T18:35:15.840722Z","iopub.execute_input":"2024-09-23T18:35:15.841060Z","iopub.status.idle":"2024-09-23T18:35:15.854242Z","shell.execute_reply.started":"2024-09-23T18:35:15.841026Z","shell.execute_reply":"2024-09-23T18:35:15.853236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class clr:\n S = '\\033[1m' + '\\033[94m'\n E = '\\033[0m'\n R = '\\033[31m'\n G = '\\033[1;32m'\n Y = '\\033[33m'\n \nmy_colors = [\"#5EAFD9\", \"#449DD1\", \"#3977BB\", \n \"#2D51A5\", \"#5C4C8F\", \"#8B4679\",\n \"#C53D4C\", \"#E23836\", \"#FF4633\", \"#FF5746\"]\nCMAP1 = ListedColormap(my_colors)\n\nprint(clr.S+\"Notebook Color Schemes:\"+clr.E)\nsns.palplot(sns.color_palette(my_colors))\nplt.show()","metadata":{"id":"_Ju_9uCz9725","outputId":"61fb6e6c-0cde-4c19-e1c2-f103122588f6","execution":{"iopub.status.busy":"2024-09-23T18:35:15.856449Z","iopub.execute_input":"2024-09-23T18:35:15.856744Z","iopub.status.idle":"2024-09-23T18:35:15.950748Z","shell.execute_reply.started":"2024-09-23T18:35:15.856713Z","shell.execute_reply":"2024-09-23T18:35:15.949788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Funtions πŸ“‘","metadata":{"id":"h1McheNb2XlK"}},{"cell_type":"code","source":"def question_print(index):\n mis_cols = ['MisconceptionAId',\n 'MisconceptionBId', \n 'MisconceptionCId', \n 'MisconceptionDId']\n\n sample_q = train.iloc[index]\n sample_mis = sample_q[mis_cols].dropna()\n\n sample_mix_text = ''\n for mis_col, mis_id in zip(sample_mis.index, sample_mis.values):\n mis_id = int(mis_id)\n mis_col = mis_col[:-2]\n mis_text = mis_map.loc[mis_map['MisconceptionId'] == mis_id, 'MisconceptionName'].iloc[0]\n sample_mix_text+=f'\\n{mis_col}: {mis_text}'\n\n # print(sample)\n print('Construct Name:', sample_q['ConstructName'])\n print('Subject Name:', sample_q['SubjectName'])\n print('\\nQuestion Text', sample_q['QuestionText'])\n print(sample_mix_text)\n ","metadata":{"id":"ltYmR_U32M8P","execution":{"iopub.status.busy":"2024-09-23T18:35:15.951879Z","iopub.execute_input":"2024-09-23T18:35:15.953018Z","iopub.status.idle":"2024-09-23T18:35:15.964607Z","shell.execute_reply.started":"2024-09-23T18:35:15.952971Z","shell.execute_reply":"2024-09-23T18:35:15.963505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The Problem\n\nTagging distractors with appropriate misconceptions is essential but time-consuming, and it is difficult to maintain consistency across multiple human labellers.\n\nThis competition challenges you to develop a Natural Language Processing (NLP) model driven by Machine Learning (ML) that predicts the affinity between misconceptions and distractors.","metadata":{"id":"jg-ZsfMzdfY1"}},{"cell_type":"markdown","source":"# Data","metadata":{"id":"hCFAuVso7Gke"}},{"cell_type":"markdown","source":"* **[train/test].csv**\n - `QuestionId` - Unique question identifier (int).\n - `ConstructId` - Unique construct identifier (int) .\n - `ConstructName` - Most granular level of knowledge related to question (str).\n - `CorrectAnswer` - A, B, C or D (char).\n - `SubjectId` - Unique subject identifier (int).\n - `SubjectName` - More general context than the construct (str).\n - `QuestionText` - Question text extracted from the question image using human-in-the-loop OCR (str) .\n - `Answer[A/B/C/D]Text `- Answer option A text extracted from the question image using human-in-the-loop OCR (str).\n - `Misconception[A/B/C/D]Id` - Unique misconception identifier (int). Ground truth labels in train.csv; your task is to predict these labels for test.csv.\n* **misconception_mapping.csv** - maps MisconceptionId to its MisconceptionName\n* **sample_submission.csv** - A submission file in the correct format.\n - `QuestionId_Answer` - Each question has three incorrect answers for which need you predict the MisconceptionId.\n - `MisconceptionId` - You can predict up to 25 values, space delimited.\n","metadata":{"id":"uLBhnzLrP8jB"}},{"cell_type":"code","source":"train = pd.read_csv(CFG.data_folder+'train.csv')\ntest = pd.read_csv(CFG.data_folder+'test.csv')\nmis_map = pd.read_csv(CFG.data_folder+\"misconception_mapping.csv\")\ntrain.head(2)","metadata":{"id":"g8bGY8WC8B3b","execution":{"iopub.status.busy":"2024-09-23T18:35:15.965514Z","iopub.execute_input":"2024-09-23T18:35:15.966168Z","iopub.status.idle":"2024-09-23T18:35:16.016398Z","shell.execute_reply.started":"2024-09-23T18:35:15.966133Z","shell.execute_reply":"2024-09-23T18:35:16.015233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mis_map.head(2)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.018004Z","iopub.execute_input":"2024-09-23T18:35:16.018922Z","iopub.status.idle":"2024-09-23T18:35:16.029702Z","shell.execute_reply.started":"2024-09-23T18:35:16.018872Z","shell.execute_reply":"2024-09-23T18:35:16.028514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Train Columns: \\n', train.columns)\nprint('\\nTest Columns: \\n', test.columns)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.031488Z","iopub.execute_input":"2024-09-23T18:35:16.031998Z","iopub.status.idle":"2024-09-23T18:35:16.041600Z","shell.execute_reply.started":"2024-09-23T18:35:16.031949Z","shell.execute_reply":"2024-09-23T18:35:16.040545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"misTable = {}\nfor i in range(mis_map.shape[0]):\n misTable[mis_map.MisconceptionId.values[i]] = mis_map.MisconceptionName.values[i]","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.042876Z","iopub.execute_input":"2024-09-23T18:35:16.043663Z","iopub.status.idle":"2024-09-23T18:35:16.116176Z","shell.execute_reply.started":"2024-09-23T18:35:16.043616Z","shell.execute_reply":"2024-09-23T18:35:16.115486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.120400Z","iopub.execute_input":"2024-09-23T18:35:16.120685Z","iopub.status.idle":"2024-09-23T18:35:16.130194Z","shell.execute_reply.started":"2024-09-23T18:35:16.120654Z","shell.execute_reply":"2024-09-23T18:35:16.129159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Examples","metadata":{}},{"cell_type":"code","source":"question_print(1)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.131303Z","iopub.execute_input":"2024-09-23T18:35:16.131591Z","iopub.status.idle":"2024-09-23T18:35:16.142758Z","shell.execute_reply.started":"2024-09-23T18:35:16.131560Z","shell.execute_reply":"2024-09-23T18:35:16.141822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"question_print(100)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.144068Z","iopub.execute_input":"2024-09-23T18:35:16.144421Z","iopub.status.idle":"2024-09-23T18:35:16.158003Z","shell.execute_reply.started":"2024-09-23T18:35:16.144385Z","shell.execute_reply":"2024-09-23T18:35:16.157035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploration","metadata":{}},{"cell_type":"code","source":"mis_columns = ['MisconceptionAId','MisconceptionBId', 'MisconceptionCId', 'MisconceptionDId']\ntrain['MisconceptionAllIds'] = train[mis_columns].apply(lambda x: ' '.join(x.dropna().astype(str)), axis = 1)\ntrain['MisconceptionCountIds'] = train[mis_columns].apply(lambda x:len(x.dropna()), axis = 1)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.159369Z","iopub.execute_input":"2024-09-23T18:35:16.159816Z","iopub.status.idle":"2024-09-23T18:35:16.657970Z","shell.execute_reply.started":"2024-09-23T18:35:16.159745Z","shell.execute_reply":"2024-09-23T18:35:16.657146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mis_ids = train[mis_columns].apply(lambda x: x.dropna().astype(str).unique(), axis = 1).values\nmis_ids = pd.Series(np.concatenate(mis_ids))","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:16.659113Z","iopub.execute_input":"2024-09-23T18:35:16.659445Z","iopub.status.idle":"2024-09-23T18:35:17.058976Z","shell.execute_reply.started":"2024-09-23T18:35:16.659409Z","shell.execute_reply":"2024-09-23T18:35:17.058113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (8,5))\ntrain['CorrectAnswer'].value_counts().plot(kind = 'bar', rot = 0, ax=ax)\nax.set_title('Correct Answer Frequency')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:17.060588Z","iopub.execute_input":"2024-09-23T18:35:17.061141Z","iopub.status.idle":"2024-09-23T18:35:17.302858Z","shell.execute_reply.started":"2024-09-23T18:35:17.061095Z","shell.execute_reply":"2024-09-23T18:35:17.301954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (8,5))\ntrain['MisconceptionCountIds'].value_counts().plot(kind = 'bar', rot = 0, ax=ax)\nax.set_title('Count of Misconception per Questions')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:17.304006Z","iopub.execute_input":"2024-09-23T18:35:17.304298Z","iopub.status.idle":"2024-09-23T18:35:17.538647Z","shell.execute_reply.started":"2024-09-23T18:35:17.304264Z","shell.execute_reply":"2024-09-23T18:35:17.537806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top = 15\nfig, ax = plt.subplots(figsize = (6,8))\ntop_mis_ids = mis_ids.value_counts()[:top][::-1]\ntop_mis_ids.plot(kind='barh', ax=ax)\nax.set_title(f'Top {top} Misconception ID')\nax.set_xlabel('Frequency', fontsize = 14)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:17.539661Z","iopub.execute_input":"2024-09-23T18:35:17.539958Z","iopub.status.idle":"2024-09-23T18:35:17.995510Z","shell.execute_reply.started":"2024-09-23T18:35:17.539926Z","shell.execute_reply":"2024-09-23T18:35:17.994548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_ex_mis = mis_map.loc[mis_map['MisconceptionId'] == int(top_mis_ids.index[-1][:-2]), 'MisconceptionName']\n\nprint('TOP MISCONCEPTIONS')\nprint('\\n')\nfor rank,miss_id in enumerate(top_mis_ids[::-1].index):\n top_ex_mis = mis_map.loc[mis_map['MisconceptionId'] == int(miss_id[:-2]), 'MisconceptionName']\n print(f'Top {rank+1} | {miss_id}: ',top_ex_mis.values[0])\n \n","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:17.996863Z","iopub.execute_input":"2024-09-23T18:35:17.997301Z","iopub.status.idle":"2024-09-23T18:35:18.013207Z","shell.execute_reply.started":"2024-09-23T18:35:17.997245Z","shell.execute_reply":"2024-09-23T18:35:18.012182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Embeddings Visualizations\n \n","metadata":{"id":"bYYnP7Zs1U3w"}},{"cell_type":"code","source":"\ndef get_tsne(embeddings,labels,n_components=2):\n\n test_tsne = TSNE(\n n_components=n_components, \n learning_rate=10, \n init='random', \n perplexity=30,#len(embeddings) - 1,#100, \n #n_iter = 3000,\n random_state=CFG.seed).fit_transform(embeddings)\n\n \n columns = [f'comp{i+1}' for i in range(n_components)]\n df_tsne = pd.DataFrame(test_tsne, \n columns=columns)\n \n df_tsne[\"Misconception\"] = labels\n \n return df_tsne\n\n\ndef get_embeddings(text, tokenizer, device, batch_size = CFG.batch_size, normalize = False):\n gc.collect()\n torch.cuda.empty_cache()\n \n all_embeddings = []\n if not isinstance(text, list):\n text = list(text)\n for i,mini_batch in enumerate(tqdm(range(0, len(text[:]), batch_size))):\n \n if i < math.floor(len(text)/batch_size):\n batch = text[mini_batch:mini_batch+ batch_size]\n else:\n batch = text[mini_batch:len(text)]\n \n tokenizer_outputs = tokenizer.batch_encode_plus(\n batch,\n padding = True,\n return_tensors = 'pt',\n max_length = 512,\n truncation = True\n )\n \n with torch.no_grad():\n result = {\n 'input_ids': tokenizer_outputs.input_ids.to(device),\n 'attention_mask': tokenizer_outputs.attention_mask.to(device),\n }\n\n sentence_embeddings = model(**result)[0][:, 0]\n\n\n if normalize:\n sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1)\n\n all_embeddings.append(sentence_embeddings.cpu().numpy())\n \n torch.cuda.empty_cache()\n \n \n all_embeddings = np.concatenate(all_embeddings, axis=0)\n \n return all_embeddings\n\ndef plot_top_misconceptions(top):\n plot_setences = []\n miss_ids_ref = []\n for miss_id in top_mis_ids[::-1][:top].index:\n df_= train.loc[train['MisconceptionAllIds'].str.contains(str(miss_id))]\n miss_ids_ref+=[miss_id for i in range(len(df_))]\n plot_setences_ = df_['all_text'].tolist()\n plot_setences+=plot_setences_\n \n \n miss_embeddings = get_embeddings(\n text = plot_setences, \n tokenizer = tokenizer, \n device= device,\n normalize = True\n \n )\n\n\n \n df_tsne = get_tsne(miss_embeddings,miss_ids_ref,n_components=2)\n \n fig, ax = plt.subplots(figsize = (6,6))\n sns.scatterplot(data=df_tsne, x='comp1', y='comp2', hue='Misconception', \n palette=sns.color_palette(\"tab10\", n_colors = df_tsne['Misconception'].nunique()),\n s = 100, ax=ax)\n ax.set_title(f\"Top {top} Misconception Questions' Texts\")\n plt.tight_layout()\n plt.show()\n \n\n","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:18.015235Z","iopub.execute_input":"2024-09-23T18:35:18.015749Z","iopub.status.idle":"2024-09-23T18:35:18.035001Z","shell.execute_reply.started":"2024-09-23T18:35:18.015699Z","shell.execute_reply":"2024-09-23T18:35:18.034137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenizer = AutoTokenizer.from_pretrained('/kaggle/input/multi-qa-mpnet-base-cos-v1')\nmodel = AutoModel.from_pretrained('/kaggle/input/multi-qa-mpnet-base-cos-v1',\n #torch_dtype=torch.float16, #hal precision\n )\n\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nmodel = nn.DataParallel(model)\nmodel = model.to(device)\n\nmodel.eval()\n\nprint(model.device_ids) ","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:18.036201Z","iopub.execute_input":"2024-09-23T18:35:18.036919Z","iopub.status.idle":"2024-09-23T18:35:18.354367Z","shell.execute_reply.started":"2024-09-23T18:35:18.036875Z","shell.execute_reply":"2024-09-23T18:35:18.353343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !watch -n 1 nvidia-smi\n# !nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:18.355494Z","iopub.execute_input":"2024-09-23T18:35:18.355864Z","iopub.status.idle":"2024-09-23T18:35:18.361687Z","shell.execute_reply.started":"2024-09-23T18:35:18.355828Z","shell.execute_reply":"2024-09-23T18:35:18.360750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['all_text'] = (train['ConstructName'])\\\n +('\\n' + train['SubjectName'])\\\n + ('\\n' + train['QuestionText'])\\\n + ('\\n' + train['AnswerAText'])\\\n + ('\\n' + train['AnswerBText'])\\\n + ('\\n' + train['AnswerCText'])\\\n + ('\\n' + train['AnswerDText'])\n","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:18.363086Z","iopub.execute_input":"2024-09-23T18:35:18.363519Z","iopub.status.idle":"2024-09-23T18:35:18.379014Z","shell.execute_reply.started":"2024-09-23T18:35:18.363474Z","shell.execute_reply":"2024-09-23T18:35:18.378049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_top_misconceptions(top=3)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:18.379856Z","iopub.execute_input":"2024-09-23T18:35:18.380130Z","iopub.status.idle":"2024-09-23T18:35:20.991638Z","shell.execute_reply.started":"2024-09-23T18:35:18.380100Z","shell.execute_reply":"2024-09-23T18:35:20.990716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cossine Similarity\n \nThis is a baseline solution inspired on [cosine similarity - bge](https://www.kaggle.com/code/jaytonde/cosine-similarity-bge)","metadata":{}},{"cell_type":"code","source":"def make_all_question_text(df: pd.DataFrame) -> pd.DataFrame:\n df[\"all_question_text\"] = df[\"ConstructName\"] +\" \" +df[\"QuestionText\"]\n return df\n\ndef make_all_text(df: pd.DataFrame) -> pd.DataFrame:\n df[\"all_text\"] = df[\"all_question_text\"] +\" \" +df[\"value\"]\n return df\n\ndef wide_to_long(df: pd.DataFrame) -> pd.DataFrame:\n df = make_all_question_text(df)\n\n df = pd.melt(\n df[\n [\n \"QuestionId\",\n \"all_question_text\",\n \"CorrectAnswer\",\n \"AnswerAText\",\n \"AnswerBText\",\n \"AnswerCText\",\n \"AnswerDText\"\n ]\n ],\n id_vars = [\"QuestionId\", \"all_question_text\", \"CorrectAnswer\"],\n var_name = 'Answer',\n value_name = 'value'\n )\n \n\n \n df = make_all_text(df).sort_values([\"QuestionId\", \"Answer\"]).reset_index(drop=True)\n\n return df\n","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:20.993044Z","iopub.execute_input":"2024-09-23T18:35:20.993724Z","iopub.status.idle":"2024-09-23T18:35:21.002036Z","shell.execute_reply.started":"2024-09-23T18:35:20.993677Z","shell.execute_reply":"2024-09-23T18:35:21.000998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Validation Split","metadata":{}},{"cell_type":"code","source":"train_long = wide_to_long(train)\ntest_long = wide_to_long(test)\nprint(train_long.shape)\ntest_long.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:21.003167Z","iopub.execute_input":"2024-09-23T18:35:21.003463Z","iopub.status.idle":"2024-09-23T18:35:21.048224Z","shell.execute_reply.started":"2024-09-23T18:35:21.003431Z","shell.execute_reply":"2024-09-23T18:35:21.047368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q_ids = train['QuestionId'].sample(frac = 0.2,random_state = CFG.seed)\nvalid_long = train_long.loc[train_long['QuestionId'].isin(q_ids), :].copy()\n\nmis_sentences = list(mis_map['MisconceptionName'].values)\nvalid_sentences = list(valid_long['all_text'].values)\ntest_sentences = list(test_long['all_text'].values)\n\nprint(valid_long.shape)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:21.049344Z","iopub.execute_input":"2024-09-23T18:35:21.049628Z","iopub.status.idle":"2024-09-23T18:35:21.059810Z","shell.execute_reply.started":"2024-09-23T18:35:21.049597Z","shell.execute_reply":"2024-09-23T18:35:21.058964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Embeddings","metadata":{}},{"cell_type":"code","source":"miss_embeddings = get_embeddings(\n text = mis_sentences, \n tokenizer = tokenizer, \n device= device,\n normalize = True)\n\n\nvalid_embeddings = get_embeddings(\n text = valid_sentences, \n tokenizer = tokenizer, \n device= device,\n normalize = True)\n\n\ntest_embeddings = get_embeddings(\n text = test_sentences, \n tokenizer = tokenizer, \n device= device,\n normalize = True)\n","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:21.061039Z","iopub.execute_input":"2024-09-23T18:35:21.061377Z","iopub.status.idle":"2024-09-23T18:35:40.039836Z","shell.execute_reply.started":"2024-09-23T18:35:21.061342Z","shell.execute_reply":"2024-09-23T18:35:40.038842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict","metadata":{}},{"cell_type":"code","source":"valid_cos_sim_arr = cosine_similarity(valid_embeddings, miss_embeddings)\nvalid_sorted_indices = np.argsort(-valid_cos_sim_arr, axis=1)\ntest_cos_sim_arr = cosine_similarity(test_embeddings, miss_embeddings)\ntest_sorted_indices = np.argsort(-test_cos_sim_arr, axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:40.045564Z","iopub.execute_input":"2024-09-23T18:35:40.045935Z","iopub.status.idle":"2024-09-23T18:35:40.273281Z","shell.execute_reply.started":"2024-09-23T18:35:40.045899Z","shell.execute_reply":"2024-09-23T18:35:40.271823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_sorted_indices[:, :25]","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:40.275345Z","iopub.execute_input":"2024-09-23T18:35:40.276140Z","iopub.status.idle":"2024-09-23T18:35:40.283853Z","shell.execute_reply.started":"2024-09-23T18:35:40.276088Z","shell.execute_reply":"2024-09-23T18:35:40.282750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{"id":"eJB0Ll_9aB_3"}},{"cell_type":"code","source":"def make_sub_df(df_long, similiar_matrix, includ_gtp = False):\n df_long[\"Answer_alphabet\"] = df_long[\"Answer\"].str.extract(r'Answer([A-Z])Text$')\n df_long[\"QuestionId_Answer\"] = df_long[\"QuestionId\"].astype(\"str\") + \"_\" + df_long[\"Answer_alphabet\"]\n df_long[\"MisconceptionId\"] = similiar_matrix[:, :25].tolist()\n df_long[\"MisconceptionId\"] = df_long[\"MisconceptionId\"].apply(lambda x: ' '.join(map(str, x)))\n # filter correct row\n df_long = df_long[df_long[\"CorrectAnswer\"] != df_long[\"Answer_alphabet\"]]\n \n sub_cols = [\"QuestionId_Answer\", \"MisconceptionId\"]\n \n if includ_gtp:\n mis_id_cols = ['MisconceptionAId','MisconceptionBId','MisconceptionCId','MisconceptionDId']\n train['labels'] = train[mis_id_cols].apply(lambda x: ' '.join(map(str, x.dropna().astype(int))), axis = 1)\n df_long = df_long.merge(train[['QuestionId', 'labels']])\n sub_cols.append('labels')\n \n \n submission = df_long[sub_cols].reset_index(drop=True)\n \n return submission","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:40.285611Z","iopub.execute_input":"2024-09-23T18:35:40.286450Z","iopub.status.idle":"2024-09-23T18:35:40.300984Z","shell.execute_reply.started":"2024-09-23T18:35:40.286399Z","shell.execute_reply":"2024-09-23T18:35:40.299705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_sub = make_sub_df(valid_long, valid_sorted_indices, includ_gtp = True)\ntest_sub = make_sub_df(test_long, test_sorted_indices, includ_gtp = False)\ntest_sub.to_csv('./submission.csv', index = False)\nvalid_sub","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:40.302792Z","iopub.execute_input":"2024-09-23T18:35:40.303573Z","iopub.status.idle":"2024-09-23T18:35:40.701165Z","shell.execute_reply.started":"2024-09-23T18:35:40.303519Z","shell.execute_reply":"2024-09-23T18:35:40.700112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The Metric - Validation\n \nIn context of Document/Text retrieval precision represents the ratio of relevant documents the model retrieves\n \n\n![image.png](attachment:53f8fe67-3199-4f58-8e8b-0152244c8f98.png)\n\n \nThe average precision mesure the ability of model to sort the relevant results.\n \n![image.png](attachment:368558fd-6443-4c6e-8f9c-1254493e1eb4.png)\n \n \n* P@k refers to the precision@k\n* rel@k is a relevance function (1 if document is relevant and 0 otherwise)\n* n is the number of documents to retiver (25 in this case)\n \n ","metadata":{},"attachments":{"53f8fe67-3199-4f58-8e8b-0152244c8f98.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAgAAAAA/CAIAAAAZqef8AAAgAElEQVR4Ae19d1hU17r+yvnjnuTc+9xzTm5y74k3OSkaSw5GjSUxxhY1GFEsIFKtFLFRLIgiiKIIKF0UQcpQpDcFkSpV+gADQ+8MdWAYZobps3/z+D2/9ew7IJJYosnef8yzZu9V3732etf3rW99CxHURSFAIUAhQCHwh0QA/SFbTTWaQoBCgEKAQoCgCIDqBBQCFAIUAn9QBCgC+IO+eKrZFAIUAhQCFAFQfYBCgEKAQuAPigBFAH/QF081m0KAQoBCgCKAl98H+Hy+UCicMt+JiQkmk/msp1MmoW6+jQi0tLRwOJzXUHOJRFJfX8/j8V5DWVQRbw4Ccrm8r69vcHDwWVXi8/kTExPPeorvUwSAoXg5ATabHRYW1tbWppIdh8MJDAzcsWPH2rVry8vLVZ5Sf39nCNjb22/atOn48eMlJSWvqGlSqfT8+fO7du0yNjZmMBivqJTfNluFQvHbVuCNLV0sFqelpa1fv/7AgQM5OTkymUylqrm5udXV1So3J/+lCGAyJi90Z3Bw0MvLq6mpiZwLn8/39/dHCBUXF4vF4slvixx5ynBdXd3GjRs///zzK1euTBnhDbwpk8kQQv/617927txZWVn5Btbw1VVJJBIpJQAHB4cvv/yytLT0VRRkYGDwb//2b3Q6XSKRSKXSV1EEzlOhUDCZzKSkJC6Xi2++okB/f/+lS5cWLlyIEHrul9LX12dpaammpoYQYrFYr6hKb2a2UqmUx+MFBQUhhJ48eaJClqmpqTOZaFIE8JJf7uDgoLe3d3NzMznfwcHBH3/88eLFi2w2m3x/5mGBQJCXl4cQOn/+/MxTvfSYAoGAyWTy+fwZ5vzw4UM9Pb1FixbNpC/OMM/fMNrg4GB1dbVIJJphHXp6ehBCmZmZz40vFouFQuHExIRQKBSJRHK5/LlJ5s+fr6+v/9xo00fo6urq7OxUGTsmJ+nr61u9ejVC6NUJNLhQsVjMZDIPHDiAEHpuxYRCIZ1Ot7GxQQj19vbiTN7eAI/HKywsnHkfa2lp0dLSCg4OViHLtLS0mXx0FAG85K4yJQEMDAxs3brVwcFhaGjoRcpTzons7OxeJIcXTNvb22tkZNTR0THDfORyeWho6Jw5c2bSF2eY528YraamxtLScmBgYIZ16OrqQghlZWVNE18ikTAYDH19fWNjY21tbT09PSMjo4SEhLGxsWlSEQSxaNGiPXv2TB/nuU+joqJoNJrK2DE5FY/HCwkJOXnyZEtLy+Snr+KOh4fHTAgAin706NHvhgDq6uqWLVs2c0lLSeH79++nCOBVdMJfk+c0BGBvb/+2E0BdXd2+fftmPtWSyWRBQUG/DwJQKBSPHj2ysbGZuRgHBPDo0aNn9SSpVBobG2ttbZ2Wlobj9Pb23rlz5+bNm9Ms8b0UAlAoFC4uLmFhYbjoNyfg7u4+cwJ4+PDh74YAUlNTEUICgWCG76Kzs/PgwYOXL1+WSCTkJJQEQEbj9YWnJIDBwcGtW7eqEEB9fT2TyWxubu7v76+rq2ttbcUTsbGxscrKyrKyMhWTockSQFtbW2lpaWVlZU9PD0EQUqmUxWI1NTU1Nzc3NjbyeDy5XN7a2gp/YVoxPDxcU1NTWlpaUVHR2dmJ1cd9fX0NDQ11dXVdXV18Pr+mpobBYIyMjIAYDtYmZ8+e3b179+PHj5uamqZRBA0NDdXW1lZXV3d0dERERKgQAJfLrampKSsrq62tnWwq09raWllZWV5e3tTUJBaLRSJRU1MTk8lkMBhcLlcsFnd2djY2NlZVVYGRQ29vL5PJrKurGxwcHBsbo9Pp0C6CICYmJmqeXoODgxhb6AotLS3l5eWAGzzq7u5mMpn19fX9/f2QT319Pa6eUCgsKSlZuXLlkSNHnjx50tjYKBaLIauOjo7S0tKysrK6urqmpiay8N7Z2YkQevjw4bP6X2VlpZ2dXX9//+QI8fHxISEhuJTJERYtWqSjo4Pv83i81tbWhoYGpZgC9e/s7MRP2Wx2VVVVWVlZfX09jBQ8Hi83N3fr1q2urq4NDQ3Nzc3Cp1dzczODweju7u7t7a2urm5tbSUIAsBhMBijo6M4z/Hxceil9fX1MGCNjIw0P72ampqgQ46MjDQ2NjY3N0M+8FLq6+tLS0vpdLrKPHdkZITBYNTU1AwPDzs7O09PAOPj43V1dVVVVQMDAykpKSoEIBaL6+vry8rKqqqqRkZGcJ0JgpDL5T09PVVVVeXl5XQ6XSwWKxSK1tZWJpNZW1s7PDwM7W1sbKytrYVWcLnc+vr62tpaFovF5XKrq6vLy8tBGOLz+QwGo7q6mow2lMJisSoqKsrLyxsbG6ECw8PDTCazpqZmdHSUw+FUV1fT6XQsUAqFwoaGhpMnTyKE6HR6c3MznmooB5CKioqSkhImk1lWVkZuTmdnp6mp6e7du1W6CkUAZJReX3hKAhgYGNDU1LSzsyPP6TQ1Nb/99ltdXV03NzcNDY3t27fX1tZC5/Pw8LCzswsODt6zZ09UVBQeaslrABMTE4WFhdra2snJyba2tsbGxtXV1RMTE5GRkRoaGj/99JOGhkZTU5NQKDQzM1u1atWyZcsYDIZAIPD09Jw1a1Z6erqrq6u+vn5ZWRmMgPHx8ZqamsuWLXNxcYmIiPDx8XFwcLh8+TIMT6D8QQjNnTvX0NBQT08Pf9Iq4CoHR1tb25MnT4aFhV2+fPnkyZNLlizBKqDu7m4vLy9PT8/79+9fu3bt+vXrWJ5QKBQZGRn79u27efOmr6/vzz//XFBQoPyuHBwc1q5du2HDhoaGhpGREXd3923btiGEoGKRkZHq6upr1qxxdnam0Wi+vr7a2toHDhyg0+k0Gs3Pz8/Ly+vMmTPwJUNVHz16ZG5uHh4efvHiRRMTE4A9MDBww4YN69atc3Z2Dg0NvXnzppWV1bVr1+C7YjKZenp6CKHFixfv27dv9+7dMKzU1dXZ2NikpKQkJSXZ2NgcOnQIRhAoCAhgGgkgJiYmNzd3Sk13Q0ODnp7eNIqgxYsXHzhwAIPf3Nx89uxZdXX1ZcuW+fj4rF+/3tjYuKGhgSCIpqYme3t7Ly+vwMDApUuXPnz4UCqV1tbWKtelEEIbNmwwNTU9fvw4i8Xq7+8/fvz4ggULHB0dXVxcvv32Wy0trc7OzpCQkO3bt6upqeHF/O7ubh8fn5MnT4aHh+vo6ISGhvJ4vNLSUjMzsw0bNmzcuDE4OJggiLy8vG+//Xbp0qVnz55VKBQCgSAgIMDW1jY4ONjIyMjd3R0zSm1t7bVr16ytrX19fW/evLlz585pCKC1tdXb29vKyiooKOjOnTtWVlbkRWAejxcZGeng4PDgwQMPDw93d/eqqioASiKR5OXlnT592t3dPTw8fNWqVREREQqFwtbWdu3atf/85z+LiooIgoDO8Ne//jUoKIggCCaTqaGhMWfOHGdn59u3b/v5+dnY2BgaGmZmZkZFRXl4eFy5csXc3LyiogJKkclkNTU1tra2gYGBLi4uW7Zsyc3NBTS2bNnyySef+Pv73717NzAw8ODBg+fPn+/q6iIIoqOjw8bG5p///CdC6NChQ6ampnl5eQRBDAwMuLq6BgYGZmZmnjt3DqH/o7fv6OgwNjY2MDCgCAB/C79l4FkEgBDy8/PDQzlBEMqZmqWlJULI39//3r17CKGioiKpVGpubq6urs5msyUSSWRkJEIId18yAZSUlCCEMjIyCILo6+tbv379sWPHent7BwcHAwICEEJnzpzh8/kKhSI7O9vMzKysrGx8fLytrQ1MJgiCGB4ednV13bVrF8xlQBBBCO3Zsyc3NxdmKBs2bAgJCZFKpXw+Xzm53rp1q56e3jQSQFNT05YtW0JDQ7u6umACrq+vv2zZMiAAkUjk6Oi4evVqmPr19/e7urqeOXMGBs3MzEyEUERExMTERG1tLTSWxWJ1dnYeOHBgwYIFDAZDKpV2dnZ6enoihGDqxGKxsrOzEUI6Ojq5ubnj4+Pl5eWzZ89etWpVYWHh6OhoS0vLxo0b/f39Afzq6mqEUGBgoEwm6+/vP3v2rL6+PofD6e/vv379OkLI3NwcJqdZWVkIofT0dIIgOBxOUVHRjz/+qCIB7Nq1y9HRETpcTk6OoaEhmeOBAKKjo5+1qBsZGQn0M7nLNjU1bd++HYsgKhGam5sRQrhogiB4PF5LS8vBgwcRQr6+vqGhoQih/Pz8np4eY2PjY8eODQ0NiUQiJbl+8MEHT5484fF4ERERH374oZubG0gAExMTIpGouLj40KFDSmIoKCgARXx9fX1fX19MTAxCCMY4uVx+6tSpDz74AESrpKQksHBjs9nNzc2zZ89evnw5UEV/f7+FhUVwcDCYRdy6dWvFihU1NTVSqTQ7O/uvf/1rSEiIQqEYGhr66aefbG1te3p6OBxOW1vbsWPHnkUAo6OjBw4c2L9/f29vL5/Pb25udnR0JBPArVu39uzZ097eThAEl8t1cXFZtWoVzKYbGxtnzZrl6+s7Pj7e09OzZcsWGE9bW1sTEhKg1QRBsFgsANDPzw+wjY2NnT9/PkIoNzd3bGwMlHuffvppWFgY++llZma2aNEieE0DAwNbtmy5deuWUCgcHx8/fvw4Qmh4eHh0dBRKWbVqVU5ODo/H6+3tRQh5eHgQBAGfp5OTE1kCkEgk9+7dc3V1BRkL4pM7g1LwtbS0XLt2LRbl4SklAZBRen3hyQSgnIU5ODgsXLhwsgUF/koFAkFbW5tYLC4oKEAI3bx5E2rc2tq6Y8eOy5cvw94xrALi8/ne3t4IITyBio2N/eSTT2D+0tPTY2houGbNGhBF09PT7969CxmKRKKioiJsIMxgMIB4MEDKgfXw4cPwl8ViWVhYGBoawj4jsVisp6d39OhRPGfHqSAgEAju3LmDEIIZDaikbt++PW/ePCAAWKyLiIjACaG9MFjo6uq+8847MJERCAQFBQV1dXUgnVy8eHHu3Ll4rLx//z4mAIIgBgcHEULBwcF4nN25c+fcuXOhlOHh4YsXL2pqaoLEsG/fPjU1te7ubnialpaGx46WlhaEkJeXFzxqbm7W1tY2NTWFv0otiqampoODAxbMCYJQfqvbt29/8ODBwMAAm80uKSkh78kSCoUwzUxKSppyV05ERERNTQ3kr/IrFotPnTo1mQCkUml9ff3hw4fJbx+ntbe3RwjxeLzx8XHljFIoFD5+/BghhEtRooEQgontkydP/vGPfygnyzg5TAsuXryIEOrr6+NwOK2traAyAkoGAsjPz0cIOTk5QcKOjg5NTU1nZ2egWFDfw9Sko6PjzJkzoB4RCoW4A0PPdHFxUdK8VCr19fX9j//4j/z8fFyTadYAgoODMTFD/AcPHuCXyGKx8JAKT8vKymbNmuXp6SmTyS5dujRr1iz4EsVicVFRUXFxMUQDTi0sLIS/FRUVCKFbt27B39ra2h9++GHdunXj4+MEQYhEomXLlhkYGGB7v6tXrwKXSCSSqKgossUUZAU4M5lMJbHZ29vjLxchZGFhAaUQBAFmnbi38Pl8Pz+/nTt3JiQkgDyXlJSEIxMEoVAokpOTZ82adefOHZwnQRAUAZBRen1hFQKg0WgIITU1NZDmyPWQy+Xh4eF//vOfCwoK8H1fX1+l3cgPP/xw+PBhUO0p5w5Hjx4F/scSAJPJBDXI3r17TU1NTUxM1q1bN2vWLJA0CYJwc3ODIXJiYsLMzOzJkye4CJFIFBcXd+zYMSMjoz179oARMX6qXIA6ffo0/O3t7bWwsNDW1oZBTSAQ6OrqHj58GI/vOBUE+vr6Tp8+jT9FgiAkEklwcPCXX34JA4efnx9C6MGDBzjhkydPEELh4eHNzc3ffffdhg0b8CO5XI51IxcuXCATQHJyMpkABgYGlBsOYmNjcVotLa1vv/0W/rLZ7MuXL6urq/f19REEMXfuXKW+2NjY2NTU1MzMbP369UqJGxREMELBpI8gCKVKXTmR3Lt3L+QzODi4bds2FT1eYmIiQugvf/nL7t27jxw5UlNTg+sMqXg83q1bt0CuIn+f8DQ8PByzGtwh/zo4OEwmABjs3n333SlVcBcuXCCrCEQiUVhYGEJIX1/fzMzMxMQERARoY1FR0X//93+HhYWRF0iGhobs7e03btyIddNQpZqaGiwBwHtcsWKFubk57qVmZmbQQC6Xi4fg7OzskJAQGM5AZl20aNGRI0dMTEyOHDkyb948JSXIZLL//M///Omnn+rr63Hzb9y48SwJAPoYWbNPXgQGcVCpwMRZ1dXVrV69Wk1NTSqVKonN1NQUhAOCIGQyGX5fTU1NWAIgCKK0tJRMAHQ6/ZtvvnFycoIvUSQS6erqmpiYYNU/JgAej7d//36YGZiZmWF8sDYJOjwe4lUIIDAwECEENANNKC8vR08vZR82NTVNSEjATcMBoI0dO3bg9QZqHwAG57UGVAhAJBKNjY05OTktXboU68GhQmAiaWBggLsjQRDQ7/39/WUymUgkEovFEokEf594AqVcF9XR0YGRSyKRQEypVIp7M9gSODs7DwwM2NjYYAj6+vqOHDmCEIqKipJKpTA3IYsmZALo6ek5ceLE7t27oTtOTEwAAfT09HC53Pb2dhXDg/b2diMjo9WrV+NVTSAAvAgMqhuyxQsMCnfu3MnJyVmwYMGWLVtwVcmB6QkAJICYmBicREtLa/ny5fCXzWZfunRp8+bNQACfffbZypUre3t7ybhBTBggMAG0tLTo6Ojs27cPng4NDW3btu3ChQvDw8NsNhsPQE1NTXv37oVPdM2aNeRxE7gWtgKAOg7XEAIRERHTE8BkzpBIJBwOB8zkyesNkKGdnR2ZACQSCahumpqaoL0ikQj3qKKiog8//DA8PFyhUIyNjcFGqqGhoQsXLmzbtk3FYg1UZ0DkXl5eCKHr168rFIrJvVShUGzdulVXV7ekpMTf3x+SEARRVlaGELpw4YJcLhc9vcRiMSgulPPlzZs3M5lMjM80BGBhYUFuI8x28SIwTLl8fX1xVnV1dWvWrPn4449FIhFC6Pjx41j+w3FgpYRMANAzcWeg0+lqamoBAQGwyC8UCvX19ackAD6fDzq0vLw8qVSK8QHxtL6+frLMTZYAgABgyoUZsb29HWQ76GYqrwYEYhqNRt4SSEkA5Jf7+sIqBAAFDw0NIYSUi2Zk/YBcLqfRaIaGhmSzetjtZW1tjWssk8kGBgaAAzABcDgcEJ/xUAuaSvKEcd++fQghd3d3sp0fSNbZ2dmQPwik8fHxkZGRoNmAxQN4ymKxLC0tdXR0QLRX7gLT0dFRiiZ9fX35+fm6uroqw9Pw8PDly5fxpwgqoKCgoPnz54OSJyIiAiGUnJyMWwefGejZt23b9vHHH+NHcrlcIBBAw8+fPz9v3jzs8ADm3VjbPpkAtLW1V6xYAVmNjIxcvnx58+bNgJW2tvaf/vQnskDGZrPhqwaFGP7mW1tbdXV19+/fD/mAKZednd3o6GhsbGxoaChBENu2bQNwCIKAeSieEhIEAWsAU07ZIM/pCcDe3p78QjEyBEFAVclrAPBURQIgCAJWMqKjo/E0Qi6Xw/SzqKjoo48+gobEx8e7u7sTBAESgIaGhsooAyogeI+Q58GDB3GV5HL50NAQnhBA5J9++snR0ZGsMEQIWVpakn1hASsbGhquW7cOj3d4JoTzJwdgKMRKLYw8EFhrayt8azgJg8FYvny5gYGBRCLZtWuXuro6mXT5fD4g09jYiBDCKqDi4mKyBFBdXU0mALFYbGBgQCYAMFsCqVdpX4AQwl8ZQRBisRhaCgSA9U6gRbS0tMS1xQQgk8nU1dVFIhGNRsPxITnZHQCLxTpy5MiGDRuw/hOyoggAQ/paA1MSwODgoKam5vnz5/GYBeO1t7e3lpYWrG3iybuGhgboKxRPr6ioKFdXV/7TCyF06tQp0JKD9jw4OBim5wMDAydPnsTdlyAIUMfv2LGDLE7Csh7MXvl8PvRUDw8PMzMzUBkrNcsnTpyQSqVyubypqcnU1FRTUxOqLRQKzc3N9fX1Ozo6cnNzAwICQBwm4wva4aqqKmhOf3+/gYHBwoULHz9+DIPLoUOHfvjhB2gCn8+n0WjLly+HGRmIsbGxsdCVKyoqTp06BY9gwQMWw7lcLsxAYVYrk8lAdx8aGip5egkEAg0NjSVLlkgkErlc3tnZefbs2fXr14ODJhg6L168CMOQ8tfBwQGUWrAUf+PGDalUKpPJ6HS6pqbmnj17IObQ0JCurq61tfXQ0BCNRgMaA3EeGltXV7d+/Xry7BKWCoHeyCjh8K8mAIIgFi9ejNVToFIXiUSnTp1SDrJjY2N4LG5tbd2/f7+2tjaoucRicU5OTmBgIBi3aGlpbdmyRS6X+/j4JCYmAmmdPHlSXV29q6sLrytKpVIYEPPy8mC4PHz48N///ncWiwW9NDY29saNG7incbncv/3tbwihnJwc3FiCIK5fv/7OO++kpqbCK25ubjY3N4d1KYRQWFgYIMlms8GwZ2RkBNcB5wOyiJ+fH0TmcDigkqqvr4e67dix4+jRoyAeSaVSHx8fpQoOFOh0Ol1pyRYZGYmXmhBCYJIAawAwM1AoFKDScXd3l0gkCoXiyZMnX375JTZbGh4e3rVrl1KJ2tDQIJfLxWIxUC98Ef39/V999ZWbmxt8OEplpr29PRAtaJbS09NFIhHIB7DqBn0Vf7ZjY2MdHR2HDx8WiURubm5hYWHwQhUKBUKIPKXo7Ow0MTHR19eHFmGUKALAULzWwJQEADuBVfYBeHl5ff/99wsWLLCyskpNTcVzvfb29mPHjm3atCk/Pz89Pd3W1pbNZg8PD1+9enXOnDk///yzr69vS0uLWCxOSkpaunTphQsXWlpavLy8EhMTybMA0IyTJwsEQTx58kRPT2/jxo1tbW3x8fGHDx/esWPH1q1bw8LClMrlixcvzpkzZ/PmzYmJiXQ63dHRceXKlUuXLvX09IQhsri4+LPPPvP29vb19cXzcTK+4+PjMTExq1evzs3NVdodWVtb7969GyGE53fV1dXKb/7s2bMVFRWBgYEGBgY4H7lc7uLismzZsqioqMrKSldXVzzLY7PZW7du9fPza2pqSkxMhKWLOXPmREVFPXny5PTp0x9//LHS50RiYmJubu6ZM2cWL178zTff0Gg0BoNx9uzZdevWgQIXRqi4uDh1dXV7e3s6nR4QEBAeHi4Wi3Nzcw0MDObNm7d79+6MjIy8vDwTE5MlS5YsW7YMhkuRSJSQkLB48eKoqCh3d3cY6BFC+/btS01NrampuXLlipOTE36PBEEAAUyzD4BGo+EFeTKMED579qyKjEWOo7IPgMViBQcHq6urz58/39bWliziVFdXGxsb//jjj01NTSkpKVevXoVsJyYmbt++DRoJU1NTHo/H4XC8vLxWrVr1zTffuLq6wqqsRCIBw9kvvvji0KFDZWVlCoWir6/Pyspq2bJlubm5jx49srKyIhvaisXikJCQhQsX0ul0cp1hKFy7dm1sbGxVVZWzszPWWQcGBurp6aWkpFRVVXl6eq5atQoWw8jzaJxVVFTU5s2b4+Pja2pqfH19NTQ0lKYTH374Iaw8d3V1nT59+siRI6WlpeHh4UZGRvfv34e0YCetrq7u7e3NZDIvXbpEo9Hg0eDg4KlTp27fvt3Y2JiamgrKtP/6r/8KDAysqak5d+7cokWLVq5c6enp2d3dbWNjo6amtnbtWnd398LCQn9//3Xr1v3v//6vs7NzfX29QqGorKzcvXv3mTNn6HR6fHw8bNRiMBhnzpz58ssvN2zYEBQUpCSDixcvfvrppxs2bAgMDAQAc3Nz33///eTkZCsrq6KioomJiZs3b37xxRfR0dHNzc3e3t5Hjx7FwhwQtomJiZ6eHkUAuHv8loFpCODChQtkCSA+Pj7n6ZX19CJ/6lwuNzExkUajxcXFgQTd29ubkZGR/fR6+PAhLAAqFIrq6uro6OjQ0NCcnByYEJEbn5SUhK0U8P3GxsaYmJjg4GCwXamqqkpISBgfH6+urs7MzMzJycnKyioqKmIwGFlZWVDDlJQUmD6LxeLCwsKwsLCcnJzJ038oQqFQwIcXFhZWWlra2NgYFRUVERGBdSMDAwNxcXE0Gi0+Pp48akDyrKys8PDwsLCw2tpackdnMpmASVlZWWVlZUJCQnR0dHJyckVFBVQ7MzOzqKiosLAQqp2VlZWbm1tfX5+ampqTk5OdnZ2UlIRVcAwGIzY2lkajFRUVgf4HJ8zMzCx7ej169AgSYp3V+Ph4RkZGaGhobW0tcG1ycjKdTo+KigoNDb1//77KzqbnSgCVlZVkDQl+RxgKrF9SeTR5JzCLxcrNzc3Ozs7JycnMzCSv6xAE0dPTc//+/dDQ0ISEBHJPY7FYKSkpISEhoH/ncDjQ6pycnIyMDNA8SCSSwsLCjIwMyLm2thZm5QKBICkpKSwsLDY2Fr9cXM/R0dHs7OzJ9ZdIJLm5uZGRkeHh4eR1bLlcrpxlR0RE0Gi0ysrK/Pz8uLi46OhoFRkC519ZWRkZGRkWFlZVVVVXVxcfH0+OLBAIUlNTaTRadHQ0zP1xQqlUCq8sJCQEBFP8iMViPXjwgEajPXz4sLu7OzY2Njo6Oj09vaamJi0tDXeGtrY23DeysrKKi4vz8vLg28zIyMArGV1dXUlJSaGhoSkpKSKRSKFQ1NXVAYzZ2dlZWVllZWXwvqDHwregUCjy8/PDwsJgqUwkElVVVT1+/Dg1NTU0NDQ6OhovPkG1QQLQ1dV96wlALpePjo6Sv3n8YsgBkUhE7sHkR29C+FkEoKGh8eK+gN6EBlJ1mDkCzyWAmWc1OeZL8QU0OVvqztuFQFdXF+zje+sJQGlCZ29vP6Vva/Ir8fHxcXNze21OqchFzyQ8JQEMDg5u2rTJxsaGLAHMJDcqzluNAHgDnR3WXbIAABhgSURBVGYn8Iu0bt68eS/uDfRFKkClfRMQaGho0NDQ8PHxUVksefvWAOLi4mCRk+xNZTLEYAhFNiWcHOc3vDMlAQgEAjAGh134v2H1qKJfGwJKUd3BweGTTz4By5mXXq6BgQE2zH/pmVMZvhUIjI+Pw8Ay+TyAt48AWCxWenr6c091YDAYDx48mJ4kfsOXx2azb926hZe2cE3Gx8fj4+M3bdq0efNmlQ0BOA4V+N0gcPjw4e3bt8MSt8rU7GW1kc/nu7i4LFy40NLSkrzR72XlT+XzJiMwMTERGxu7du3aNWvWlJeXT9acUxvBfpvXp3Qs9awzv8BcbGJi4hWNCL9Ng6lSp0IAjnbBtphTRXkJ92QyGZwkQ/Wol4DmW5UFbMETCoUqqn/cCLzdD9+ZMvB/HMtNjgEuJxMTE318fLq7u+l0uqur68OHD/FKNKy/e3h40On02tpaHx8fbNWgUCgyMzOV9mGHDh1SkYJ7e3sDAgKOHz9uaWkJmzL6+vpSU1PDwsIiIiKgSSKRKDs7++DBgxcuXPDx8SkqKpLJZAUFBfHx8Xfv3iWbl42Ojt67d8/MzOz69evYCruvry8tLS0pKcnV1VUoFMbHx9vb209jcje57dQdCgEKAQqB3zcCzyGA0dHRGzduLF++XOl96bvvvrO1tQ0ICNDS0vrwww9Bj5GXl7d27Vql/68DBw6AA7+DBw+Ojo4q7e0cHBw0NTULCwthFw/sgwAnG4sXLz5x4kRFRQW4JSgsLGxtbTU0NEQIaWlpwb6bY8eOHTp0qKSkJCsra8+ePX5+fjKZzNvbe/bs2Qih1NRUeDH19fWrV6++ceNGRUUFjUZbsWIFLA/gg+I+/fTTQ4cOeXp6gslzXFzclPMysVjMZrMHn3fhrS6/725BtY5CgELgj4DAcwgAIAD/SthmlsPhvP/++6ampmC4ymAwwKvqxMSEj49PdHQ0mLEjhLCtzoEDB7S0tOCUCVNTU0NDQ7Aiv3Xr1oYNG2DTSkdHx4EDB/T09EQiEZwnjvdo5ObmxsbGgmQAvmdhc83Q0NCxY8fIG6kzMzNXrFiB95SDOwTYHws+ebZt24ZlFPIL7uvrCw4Ovnr16jXS5ezsTPp3zcnJScW8mpwDFaYQoBCgEHi7EJgRAYCjduwDEmbieBquPJ5JU1MTnA9D4/v7+8FjX2pqasXTy9bWFjy44g3lsI+Gz+eXlpZCuLu729zcXFdXFxMA7Bik0+nZ2dlKd0iw0Qn8wAABgBekgoICvAeqvb0dIZSSkgJ3QKqAWgkEAnA2QN5+8rLellwuz8/PL6YuCgEKAQqBNw+BZw10v4AAsM8yuVyenp6OvXo1NjZu2bLFxMQEl9HQ0PDDDz/861//ioiISE9Pf/jwYWZmZlZWVk9PDzg7w5IBTgL75g8fPqyrqwsqoPz8fH19fTD63LZtW0lJCax0w4kKQAAqTsEIgmhra0MIeXp6wlFKBgYG7777Li4lNzcXIQSbWvHNlxKQyWRKFzrnqYtCgEKAQuBNQuDcuXOXL19+1ig3IwLIyMhQ+jwhEwB4G05JSSEIoqGh4eeffz506BAuo6en5+jRo3CmBL4JARiCyd74cISuri4yARQWFsKyc3x8/A8//LBp0ybYZ08mAFAH5efnYwkACCAoKAhUTPr6+vPnz8dF5OTkPIsAhoaGYE887dlXcHDwlDWH/MFLMBwPRP1SCFAIUAi8IQhM4zrhFxAAVgFJJBI4VgL8NDU2NmpoaOCDkwiCEAqFoGzB2naZTFZcXMxgMOAAkFu3bmHrpb6+Ptgq2d3dfeTIEbwGoKmpCU5rpVIpDPrgrxgGffCwCKcdXb9+HZvB1tXV4YMSCYIwMDD46quvMAFA/CklAIFA0NHR0TTt1djYONkDO86cClAIUAhQCLxdCMyIAGANwNvbG8yNq6qqEELXrl2Dv4WFhUuWLFFXVye3vKmpadeuXT///DN4Fe/v71ceSMRisQQCAbikh60rCoXi2LFjcERUY2Ojjo6O0tYI7OgRQthwKDEx8fvvvwceA5sicF0rk8lcXV2VhzyAWerExITSiMjGxgaGaS6Xu337doQQkA3szHyWBECuPBWmEKAQoBD4IyDwCwjg0KFD8+fPhxP13NzcAJ34+HhQ0yOEPvnkk9u3b2PU2tvb4Wi0zz77bMWKFdhJnkAgAA/dSg/gs2fPhiNYYXEYspo/fz5YAcESLkLovffeg50EmpqauDjgAIIg4BhFuH/q1CkY/XNycr777ju4uXTpUgsLizlz5uC0NBoNVp5xbakAhQCFAIXAHw2BX0AAkz33/tHAotr7EhHgcDjR0dEqx07BMa21tbVw7B+cAPMSC/1ts4JTjsknd05TH4VCIZfLZTLZH22mkpycvHHjxgsXLkypuc7Ozi4pKcFrftMASD2aCQIzIoBHjx4hhCa7/J5JAVQcCoEpEWCz2Y6OjirrMQqFAg4p8/T0rKurA3uwKZO/lJsKhYLL5Q4ODuI1pJeS7bMyuX//fnh4uLm5+XM/JT6fn52dfevWLUNDw2nsDp5V0Ft9f3h4OC8vz9LSkmx4glsUExNz7969Pxop4ua/9MBzCEAsFjc0NICePS0tjXx8+UuvCpXhHwoBNpt95coVlR4lFosRQiqnmP0KWPh8PnYKMk1yDofj5OT02iY3UVFRc+fO/eijj8iOTKas3ujoaEJCws8//4wQekXepKcs95XeVCH76csaGRnZuHHjgQMHVKLBIS2UBKACy6/++xwCGBsbu3fvXnBwcHh4eGBgYGxsLAX9r8aaSkhGYEoCkEgkCCG8wkSO/4vCZWVlP/7443OTcLlcf3//r7/++heNTc/NdpoIPj4+77//Pj7qcpqYBEFERUUpj0XMysqaPtrb8hSh54w2Kg3R1dVdsWKFyk2KAFQAecG/v+yVvGBhVHIKAYzAlAQglUoRQq6urjjarwsEBwd//fXXM0nb399fU1ODjZJnkuRF4vj4+Pztb3+boVPC2NhYhFBmZuaLlPiGpIXD0X5RZXR1ddXU1FSSUASgAsgL/qUI4AUBpJL/SgR+KQFgB7Qq+l82m93a2gpbRqAqQ0ND2traq1atmlwzOLFdZXVR/vQiR+bz+Y2NjeQ8yU+fFebxeG1tbVMe+obXGJycnP7xj39MTwASiQSOuwATOxUCUCgUXV1dz3JKyGazW1payAfTEwTB5/PJDKdQKFSSww4bmUzW1dVFhlclH9zwnp6e5uZmFR/UUqkUdl9OTEwMDAzgyBBwdnaeUgIYHh5uamricDjCpxc5la6u7sKFC8l3CIKgCEAFkBf8SxHACwJIJf+VCExJAKACIksALS0tLk8ve3v7R48eOTk53bhxAxZR5XL53bt3LSwsTj69YGdJYWEhHJX13nvvOTo6Ojk5VVVVtbS0uLm5ubi4ODg4BAUFnTlzJiAgoKam5uHDh46OjmfPnsUDFuw6dHBwMDU1tbGxSU5OVigUlZWVZ86cuX79uru7u6enZ29vb0ZGxrVr11xcXEBbJRaLCwsLz549a2VldezYsevXr2NQxsfHlYpTe3t7qPm+ffvmzp37LAKQSqX37t2ztbW1s7Nzd3cHh1pkAqiurr548aKDg4OdnZ2Liwt5HO/s7Lx9+7alpeWlS5dOnz6dmpo6Pj7u6el55coVa2trMOG7d++eo6PjlStXwJlucXGxo6Oji4tLQEBAXFycg4PD/v37b9y40djYWFBQ4OLicuzYMQ8PDwaDgZvDYrGCg4MtLS0PHDjg6OhYVlZGEERtba2dnZ2Dg0NqampkZKStra2xsXFkZCSg2tDQcPXq1f/5n/9BCF26dOn8+fPQIi6XGxsb6+DgcOnSJUtLy+vXr6t4iKEIAMP+6gIUAbw6bKmcp0NgeHh48iKwXC5XUQEJhcLW1lbYwHH69OmQkBCEUFFREd7/MTIyMjw8HB4evmjRotraWh6PNzIysnXr1iVLloyMjAwMDPD5fJFIVFVVpaOjgxAyMTExNjZGCCUmJrLZbC8vL4QQbFlXKBQhISE7d+4sLi4eHx8PDAxECDU2NgoEgoaGBoTQTz/9lJ+fL5VKWSyWlpYWQgi2p9TX1yOEkpKSuFxuSUnJn/70p5CQEIIgxsbGLl26ZGxsTKfT2Wx2V1fXN998s2DBgikJQKFQnDt3zsjIiE6nj46OVlVVwT4YvAZQU1OzefPmx48fc7lcDodz7ty5o0ePwoS9p6fnxIkTRkZGzc3NPB4PIbRly5b29vb+/n5/f3+8+XF0dDQyMhIhFBwcTBCEQCBISEiYNWvWX/7ylxs3bnA4nMrKys8//3z9+vUeHh5tbW1sNhshZG1tDS+SzWa7urpqamq2traOj4/v3bvXyMiou7tbLBbfu3cPIbRgwYL4+Hg2m81kMgEQKGVkZMTc3BwhxGaz+/v7wU9XWlqagYFBR0fH+Ph4d3c3Qqiuro7cY7S0tJYsWUK+Q0kAKmi8+F+KAF4cQyqHX4PAlAQQHByMEIqIiFDJcc2aNbNnzy4oKBgdHS0sLOTxeHl5eX//+9/xcjGomLHHcm1t7cnrh25ubkoCYLFYExMTMEEmCKKwsBATANCPp6cnlN7Z2amtrW1ubk4QhFwuj4mJQQgVFBTAU2NjYzBhYrPZK1euNDU1xaeZghNDgiCKiopUzHguXrw4a9asKQkA3FjdvXsXt528CDwwMLB+/fqgoCCszBkbG0MI3b59Wy6Xg68UvLa8atWq48ePQz4qDhDHx8fJe+yZTOa2bduwcobL5V6/fl15LAee9dva2n711VfgXx2wqqiogJyB9sLCwgiCqKioWLNmzaeffqo8CIQgiPHx8V27dllZWWFMQJrBTSMIIjo6+r333gMyIAhisgQALoSxCzJIS6mAyBi+eJgigBfHkMrh1yCgQgB4Szn4BVHJcfXq1d988w0+5oEgCDgLW0dHx8LC4vjx4yYmJp9//rmTkxOos7W0tJYvX66SCQxtWB0PT2GMBgkAhtF169aBJsfGxmbdunUIIUjS39+v3LseFhamUCiKi4v19PRACT44OAjCgYWFBZxOsXr16nnz5onF4tOnTy9duhSPmARBXL58+VlrAEZGRu+88w44WIS6wSJwRkYG+MrFDthxuxBCp0+fVirQbWxsPvjgAxUVCkQDAsAu0EE+wE5WOBwOyEMQmcvluru7//jjj9g899y5c2pqagwGQyQS3b17FyF08OBBwNzKykq5/x8YC3y5GxoawuoFj8fbvXu3hYUF9gZ26tQpTDNQ1sOHD/HOfGdnZ2AO3DQI2NnZKcUscAYMdygCUIHoBf9SBPCCAFLJfyUCKgTQ1dUVGBiorq6uHHxhECfnu3Llyq1bt5JXHYEArl69SqfTy///1dXVBYM1JgCpVMrn8yEr2M5CzgRP0mGcSktLgyG1qqoKsqyoqMAuTMbGxs6fP79ly5aOjg5fX9/8/HzIdmBgQDm0OTg4lJaW4lR0Ol25kqyrq7t8+fIZEsCSJUvU1NSwEAPqDmwGymAwEEIq7IgQsrCwaGlp2b59u7q6uspkGaqnIgGoEIBSRUOem3O53Bs3bqxduxbbxZ47d27hwoX19fVcLhdEqMTExMrKSmhpVVVVX18f9uUOnhxBAtDW1ra0tJxMADKZDMZ6iUSSnJxsaWm5detWpQPH77//XmV/XH9//+eff25lZfXgwQMohVIBkT+KlxKmCOClwEhl8osRUCEASN/Y2IgQ8vHxUclu06ZNO3fuJN+k0WiwqIhvisXisbEx0IlraWmBCqi9vR3PdmH8UjkQlCwBgKLfw8MD50kQxODgIN77Ul5ejhCKiYlR6t+xX9ihoSGlO8Ljx4/jwQ5SCYVCR0fHL774AlZKIU8nJ6ePPvoI62rIBcFAnJSUhG/GxcVhM9Cenh5Yt8BPCYJACJ07d045pl+8eJF8AB9BECMjI+A+XYUAOBwOWQU0Ojp66NAhPDcHCWD16tVkAgAJgCAIcAKvor8C66zm5mYDAwNMADweT1tbe0oJoLe319/fnyAIOp0OGLa0tMBxrXAfNxCO8yNbJVEEgMF5WQGKAF4WklQ+vwyBKQlg8iIwZKqlpbV9+3ZyAa2trerq6p988gk21qysrExLSwMV+Y4dO5YtW0YQRGVlZVBQECQEFZCKewkgAJhgyuXy9evX7927F7u9amlpuXr1KhYa2tvbN27c+O677969exeXy+fzQbbIzc2FggQCAZyPBC4OyWY8J06c+Oijj8iUgBtVUFCAEMLHoEokEj8/P4QQnHGtXOs+evToyZMnsZlpbW0tQig+Pp4gCPDXm5aWhofLK1euwJoqEAAWDuBsD0yKDQ0N5DUAPp/v4eGxbt06PBm/cOHC119/DVkxGIzvvvvO2toam4eWlJQoLXngoO+vv/567969wK8g/VhaWuKZu7W1NdBMb28v+H+k0WiJiYmYXP/85z/7+flhNAiC0NHRWbx4MfkORQAqaLz4X4oAXhxDKodfg8CUBDDZDHRkZAROkUMIxcXF4cVJgiBgePX09CwrK3v8+LGfnx9eJABfcvn5+R4eHtXV1ZDJzp07EUJRUVGlpaUEQYhEIgaDcenSJYRQYGBgeXk5TEt37txpbW1dXFxcVFR06tQpsDiCFgqFwtDQ0Mn2KiwW68SJE3v27Hn06FFZWZmHhwe4qh0fH3d2dj5x4sTjx49LSkru378/e/Zs5aKCnZ0d1iyRsQsICFBXV8/MzCwrK7t3797mzZsRQidOnADXEQ0NDfr6+l5eXqWlpXl5eatWrXJ3dwdL0NHRUR8fn/Xr1ycmJlZUVKSmptrZ2QEXdnd3GxkZ3b17t7y8vKioyNTUFCF0+PDh4uLiioqKO3fugCI+Nze3q6srJiYGrJvu3r3b1tZWUFCwfv16hNDNmzfb29slEgkcBejv719eXp6RkWFtba08x2lgYMD5qZn/woULk5OTe3t7Y2Ji3n333e+++y4iIgIUevgk1/j4eFjkj46OhtNbKyoq0tPT1dTUVLweUWag5L7xisIUAbwiYKlsn4PADAlgYGAgMTGxrKysvLz80dOLnG99fX14eLidnV1MTExLSwtW73R2dgYGBt66daugoEAul/f09KSnpxcVFVVUVGRlZSlt3gmCmJiYePz48YMHD+AmHDGkUCja29vv37/v7OwcEhKCD6zGhdbV1QUEBOBdaXBfoVD09/ffv3/f8+lVWFiIZ+Kjo6NpaWnu7u7e3t6PHj26efOmh4fHtWvXyGIBzhyq5OPj4+HhkZWVlZyc7O3t7ebmlpycDH5SOzs7k5OT79y54+/vX1xcTN7RNjo6mp2dfe3pFRcXh3c2gII+Li7O1dU1KSmpoaHhxIkT7u7uERERycnJ2dnZStPP0NBQT09PJpMZERFRWVmZkJBw5coVOp0eFhZWVlZWVFTk5uYGUotAIGAymf7+/q6urhEREbBW3NTUlJKSUlVVFRQU5OTk1NzcHBcXBwKZp6cnHCTF5/MTExOBGkHzk52dnZ6eTqPRXF1dfXx8GhsbsTQAgOzZs4faCIb7xisKUATwioClsn0OAlMSADiDI28Ek8lkYFgCtph4RRfnLhQKh4aGsEIG3+fz+VwuF7Q3UqkUc4NcLodMFAqFUCiECBDGaaVS6ejo6JR2KSKRiM/n4/EdJyEIQiqVcjgc8qAMT2UyGefpJZPJuE+vsbEx2DRLTo7DXC53ZGQEGg7xyZElEsnY2NiUdYOChoeHsakozlMkErHZbFB/DQ0NcblcPp8vEAigIQKBgMvlSiQSiCAWizkcjlgsxmiPjY3htwBbi9lsNq6VRCIBGPl8PofDkUqlUAGpVIpfAUEQsEiDVXA8Hk8mkwkEAjabTd7RhutMuYLAULy6AEUArw5bKufpEJiSAEQiEUKIvJN2uiyoZ79rBPT19b/99luVJlJmoCqAvOBfigBeEEAq+a9EYEoC4PP5CCFzc3NsY/Mrc6eSveUIVFRU/Pu//7vSw4RKOygCUAHkBf9SBPCCAFLJfyUCw8PD165dw9YpOBcWi7V06dKPP/743Llz2BYFP6UCv3sEKisrLSws3nvvvV27dmEtE251fHx8TEzMlCo4HIcKzBwBigBmjhUV82UiIBQKW1payJplnLtUKmUwGCkpKZPpAcehAr9XBGD1+Fnc39XV1d7errJc/HuF4jW0iyKA1wAyVQSFAIUAhcCbiABFAG/iW6HqRCFAIUAh8BoQoAjgNYBMFUEhQCFAIfAmIkARwJv4Vqg6UQhQCFAIvAYEKAJ4DSBTRVAIUAhQCLyJCFAE8Ca+FapOFAIUAhQCrwEBigBeA8hUERQCFAIUAm8iAv8PrtJgSlha6CoAAAAASUVORK5CYII="},"368558fd-6443-4c6e-8f9c-1254493e1eb4.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAPwAAAAzCAYAAACzFWUGAAAIn0lEQVR4Ae2cjZHUOhCENwRiIARiIARiIISLgRCIgRCIgRCIgRT21ac37RrrJK+0u2ewr6lyyZZHP9OanhnJe1wu/vfuEfjz58/127dv15eXl+vPnz+vP378uH79+vVK/bsHxwAYgbMh8P37dxH7+uvXr3IP+SH+2XS1PkbACFwuF4j++fPnheBfvnxZyG+AjIAROBkCpPRcUuvDhw/LvepcGgEjcBIEckRnH8/z79+/y57+JCpaDSPw9xCAVJ8+ffpn0uacznNYB+HT3v7vAeWRjcCRESBqkjrH4ZjKI6vkuRsBIzCIgAk/CJTFjMAZEDDhz7CK1sEIDCJgwg8CZTEjcAYETPgzrKJ1MAKDCJjwg0BZzAicAYFdCB8/ouGHNFfuZy7a9C4+LZ5hEayDEdgLgV0IH9/TC3Hz9/YZJfmcSD98n5cDuLevmXEtawTOhMAuhAewIGch66M/qOGHOfw1HcTHEZxpQayLEXhLBHb76Sokzan9M4iqP6N9S4Dc904IkLr1/h6aev58UgaEt+eZi3va6s8sW9Nl78evzYg0GA3Xlnyrj6PWCTulxvy8Ftz20D/23CUyf/z48SmRmbm37ER67mUj4KcspjWff9Fe8px7DniWK7nPYRxCcPXXUy3AWOzYz61eh/E2oxdEbylHm0dTzdUk/NBEIK2Z0vKm3LMq03irLt/CRiBHOJjVWAd4aHLiXq5M4wDx8Ja3ogAT6gFMBKvf1YRmYrHwZU1wNLXMARbrcFMkq9DB21v/Jxd72gjOJTKno63JK8LXPJjhyjQOLFIYwquJZCS3FlOHOkorILbIzeRZGCI9F/2oX961MgC9d/k4AqyJUu23Pnjb00ZwZDVRHkdrlx5WNv8oV6ZwgGzy+hhFpGRNrbcWs47wWggZmxyBOiajkIxKvXP5fATk0CF8RPznD3K5XHa2kSWojCqDvUdwajZh/gSh5svnVa4IL/t/gCvjOKCg9ACIrbS+t5gRoRfHQX9SgjIOVzRMKXEQci6SlQDGyVjUAwIlF22UNUjW5TgCysIgvbAfbz0muZeNBCkX28UusJFsz70ZY1/YUv2etrdwYVwcJsERu2dMnoMDpUvZKyWY18GuzrKQoyHlLFemccgABZm6pArZhYA8c7WIKCVQOAxthW+P8ACHEswFUNUPjVmoFiC5Y8ai75Hr1uLmfs9yHw4dA3uTSLaHjbAWrJ1sAVvRFVuXm8uFXWW7ZN6j9qAAJ9ukn+BOmVfuh/pGEH3lIJgw/eQ5SYkeV6ZxEEjqmBLAWoPyTouZ5Xv3AoM2Wpgs21MiUs/i7Wqg6It2uR/fzyEgY4XwrHUj+sx1WEnvYSMMSVSFWNhwRLkLush+qmk1H0V65tyz+VbDwGwhuWRUn6M977BjOYSQbRJ+liv0NYUDAwBaviBnz0ves5iA0DIsQJAnlHMQcJQsgN6rHrLHHFS1R1miIQT5R69pDALvos+zHeiONlICicg+DUI0QP86sNzqS8SOchEPZ1OcDve66L9yRE3C38mVcRxa5AlPtHjNRZvJCJ+zBxYFYPF8XIzLhXPhuQKjDImTaCzmCqg8N9/PIRBEb2I/19Naeobw99pI2EWJmgSFVga5nlX7KUf4Ori0W/xfK8LXMsmR1q/q55Ud74IDoAW568mUFKTl+WcWk07ryF2PmRXNk4iUaJW64xRIX5BrOIKlOZkBcx+5ZhZ5GeAkN0oFn63OHjZSk5wIKpvo2XStp8iueuY9ag89wstu68jPGFXdivC8v4crUzhsecVQfEU4JtWrF2itEkUCiNbromgd4WkjcqtRTvFrcCTjcgwBOcMx6TmpPWwE28g2AOFF9NqWWrNHphXQmHvMv9VsqRPhKxKX99lO1YDxKg7Uz0V0litDOOAJg0ylrCfNoLGnKakSz8gAhOpRagRYKcyY9EMbSl30qYWSLCXeFplchxzy9FHPOcv5fhsBsK2d6XaLsbd72gh2mG0Am0CvEZuEeC2yS0tsTNmC6nLJu2hfytpOkaVONsy9bJwS7nAWRB+tcWa48ggOWSffbyCgBZcnz06LezXFIDmHgFyS1YEoi40s9bwPR1rSPurj0HRpJ1nayXg0zkwJIWojmWlvWSPwrhCAeJA2RxcAgEiQV8SljkhSb5vCo78ibR1xGSe+CKzwDbI3I8NKsPFAWxxJlVo2JF1lBIxAOb+oiZlhgaSZ8BCsTtlE+Jp0pH3ZiZAORpTPQ5R7onzv3SvhqGA82tTz6cm73gi8awSCoJuEgbCZ8JArkxgAe4SnPjuBLcKTSRD96763Foh5tfaaW2167yL76L12vRE4PgKQbCSqEn2lbYuQPcLTJstvEX42wrOteBZJcRrhcKSmSyNwPgRI5bfSeWmcSau6XG4RPsv1CK9MY+Q0mv4gZ32OkMcZvUcvZRahw2hTyxmBQyLwFOLMEJ60nYjKhQPgmjmlR37ESdWrAblxLMyVscNhlG3ESJZT9+dnI3A4BNgDPyNSzhL+XqA0Tpz068T/4fJZW4N79XI7I7ALAkTWfCCXByUKQoSIfuW+l9qLiPmALvele6Jz67Oc3t8qaU8K/uyrp9et+fi9ETgUArFnXp2k1wpArlsp716Er+fmZyNgBCYRIMpvpfVbWYCGMuGFhEsjcAAEIDVXndqSoo8QXpnCrZ/Hxl6ZtN7/jIAR+JsIEKUhN6Rkr8weXp/JIoK/mh7vkY9T81LyrHZqgCOhPs4Lyh68lpGsSyNgBIzAFAI4GLYpcUg41dbCRsAIHBABspVeZnJAdTxlI2AEthC49WVhq63fGQEjcCAESOn1+wKiPOcRtw4UD6Sep2oEjEBGgANBfj8A2fnSwF7eh4QZId8bgRMhANn5cuA9/IkW1aoYgR4CpPOk8JzUO7L3UHK9ETgBAnn/rtSetN57+BMsrlUwAjUCEJsf+VAP+Unv/T2+Run4z/8BXQo+n7Zu8oAAAAAASUVORK5CYII="}}},{"cell_type":"code","source":"def ap_a_n(miss, labels, debug = False):\n \n miss = miss.split()\n labels = labels.split()\n gtp = len(set(miss) & set(labels))\n \n gtp = 1 if gtp == 0 else gtp\n \n miss = np.array(miss)\n\n #calculate precision@k\n rel, tp, ks = [],[], []\n for k,mis in enumerate(miss):\n relk = 1 if mis in labels else 0\n tpk = 1 if mis in labels else 0\n rel.append(relk)\n tp.append(tpk)\n ks.append(k+1)\n \n rel, tp, ks = np.array(rel), np.array(tp), np.array(ks)\n AP = (tp/ks) * rel\n \n if not debug: \n OAP = np.sum(AP)/gtp\n else:\n OAP = AP\n \n return OAP\n \n \n \n ","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:40.702624Z","iopub.execute_input":"2024-09-23T18:35:40.703097Z","iopub.status.idle":"2024-09-23T18:35:40.712391Z","shell.execute_reply.started":"2024-09-23T18:35:40.703045Z","shell.execute_reply":"2024-09-23T18:35:40.711226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_ap = ap_a_n(valid_sub['MisconceptionId'].iloc[0], valid_sub['labels'].iloc[0], debug = True)\n\nex_ap_bin = np.where(ex_ap>0,1,0)\n\nx = np.arange(len(ex_ap_bin))\n\n\nfig, ax = plt.subplots(figsize = (10,5))\n\n\nsns.barplot(x = x, y = ex_ap_bin, color = 'g')\n\nx = x[ex_ap>0]\nex_ap= ex_ap[ex_ap>0]\n\nfor i, value in enumerate(ex_ap):\n plt.text(x[i], 1, str(round(value,3)), ha='center', va='bottom')\n \nax.set_ylim(0,1.05)\n\nax.set_title(f'Example of Average Precision for Question {valid_sub.QuestionId_Answer.iloc[0]}', fontsize =15)\nax.set_xlabel('K', fontsize = 16)\nplt.tight_layout()\nplt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:40.713614Z","iopub.execute_input":"2024-09-23T18:35:40.713918Z","iopub.status.idle":"2024-09-23T18:35:41.231040Z","shell.execute_reply.started":"2024-09-23T18:35:40.713886Z","shell.execute_reply":"2024-09-23T18:35:41.230132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This plot shows an example of AP@25, each bar corresponds to a relevant item and the value on top of bar corresponds to the P@K.","metadata":{}},{"cell_type":"code","source":"valid_sub['AP@k'] = valid_sub[['MisconceptionId','labels']].apply(lambda x: ap_a_n(x.iloc[0], x.iloc[1]), axis = 1)\nvalid_sub.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:41.232363Z","iopub.execute_input":"2024-09-23T18:35:41.232754Z","iopub.status.idle":"2024-09-23T18:35:41.346070Z","shell.execute_reply.started":"2024-09-23T18:35:41.232707Z","shell.execute_reply":"2024-09-23T18:35:41.345158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each Question Q, we can calculate a corresponding AP. The mAP is simply the mean of all the queries that the user made.","metadata":{}},{"cell_type":"code","source":"mapk = valid_sub['AP@k'].mean()\n\nfig, ax = plt.subplots(figsize = (7,5))\nsns.histplot(data =valid_sub, x = 'AP@k', ax=ax )\nax.set_title(f'Valid mAP = {mapk}')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-23T18:35:41.347188Z","iopub.execute_input":"2024-09-23T18:35:41.347522Z","iopub.status.idle":"2024-09-23T18:35:41.681359Z","shell.execute_reply.started":"2024-09-23T18:35:41.347484Z","shell.execute_reply":"2024-09-23T18:35:41.680465Z"},"trusted":true},"execution_count":null,"outputs":[]}]}