{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Hi Kagglers!👋🙂\n# 🔬Intro \n\nAs I wrote [there](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400858) and [there](https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn) i made a try to reconstruct model from [Process-BERT: A Framework for Representation Learning on Educational Process Data](https://arxiv.org/pdf/2204.13607.pdf) precisely from authors [GitHub repo](https://github.com/alexscarlatos/clickstream-assessments/) and firstly got promising results 🚀 with F1 near 0.8. But later with help of @thedevastator @shinomoriaoshi I understood that I calculated F1-metric incorrectly. After correct calculation the results decreased to about 0.6.Nevertheless this was a quite huge work (few weeks) and a lot of useful experience 🙂.\n\n# 🗒️ About this notebook:\n<div style=\"background-color:#d4f1f4; padding: 20px;\">\n\n<p>📌 I promised to share my code of this experiment and it is in this notebook. Because of large amount of train data processing takes a lot of time (near the 24 hours with appropriate model training). I refactored code as clean as I could. But as I think it is still quite difficult to read. This is in particular because I modified the existing code.\n<p>📌 I want to share it. Maybe it will be useful for someone who also tries transformers in this competition or if somebody is also trying to reconstruct model from this publication.</p>\n<p>📌 I ran in this notebook a full pipeline to demonstrate that it works correctly. Due to long processing time I ran it for one of tree question groups, for one question and one epoch. You can run it fully if you want but note that you will nead at least 25 Gb RAM and about 24 hours to train for 18 questions 10 epochs per each. If you find this notebook useful -- I will be grateful for your feedback and will be happy to answer your questions.\nCheers!👋\n</div>","metadata":{"id":"_yoWJLiptEJk"}},{"cell_type":"markdown","source":"# Data Prepoc","metadata":{"id":"z0LH-TyDJJot"}},{"cell_type":"markdown","source":"### Imports","metadata":{"id":"IWPqN695yhUp"}},{"cell_type":"markdown","source":"#### Imports, mount drive","metadata":{"id":"DVfOe-0HdR6z"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom tqdm import tqdm, trange\nfrom math import sqrt\nimport sklearn\nimport time\nfrom typing import Dict, List\nimport torch\nfrom torch import nn\nimport numpy as np\nfrom sklearn import metrics\nfrom sklearn import preprocessing\nfrom sklearn.preprocessing import MinMaxScaler\nfrom typing import List, Dict\nfrom sklearn.model_selection import train_test_split\nimport random","metadata":{"id":"KUWm_rW1JoqW"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from google.colab import drive\ndrive.mount('/content/drive')","metadata":{"id":"VhKFeSEqtJLE","outputId":"946610fb-1ce0-4c35-fa26-cc57fb344b6d"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if torch.cuda.is_available(): \n    device = \"cuda:0\" \nelse: \n    device = \"cpu\"","metadata":{"id":"Oneuj3k0oHfW"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Per-event part of \"train\" dataset","metadata":{"id":"BK6A1Owcx-cN"}},{"cell_type":"markdown","source":"#### Raw Data Loading and Categorization","metadata":{"id":"3Xsu8-sGdSlb"}},{"cell_type":"code","source":"dir_path = '/content/drive/MyDrive/ref-predict-student-performance-from-game-play' # '/kaggle/input/predict-student-performance-from-game-play'\ntrain_df = pd.read_csv(f'{dir_path}/train.csv')","metadata":{"id":"P0VY1LPSdbEu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CATS = ['level','page','event_name', 'name', 'fqid', 'room_fqid', 'text', 'text_fqid']\nNUMS = ['elapsed_time','room_coor_x', 'room_coor_y','screen_coor_x', 'screen_coor_y', 'hover_duration']\nEVENTS = ['navigate_click','person_click','cutscene_click','object_click',\n          'map_hover','notification_click','map_click','observation_click','checkpoint']\n\nLEVELS = train_df.level.unique()\nPAGES = train_df.page.unique()\nEVENT_NAMES = train_df.event_name.unique()\nFQIDS = train_df.fqid.unique()\nNAMES = train_df.name.unique()\nROOM_FQIDS = train_df.room_fqid.unique()\nTEXTS = train_df.text.unique()","metadata":{"id":"sKE3otlsdbHv"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LEVELS_types = train_df.level.nunique()\nPAGES_types = train_df.page.nunique()\nEVENT_NAMES_types = train_df.event_name.nunique()\nNAMES_types = train_df.name.nunique()\nFQIDS_types = train_df.fqid.nunique()\nROOM_FQIDS_types = train_df.room_fqid.nunique()\nTEXTS_types = train_df.text.nunique()\nTEXTS_FQIDS_types = train_df.text_fqid.nunique()","metadata":{"id":"sigyv-S8_dO4"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### encode_cats_in_df","metadata":{"id":"AjfaTJ8Zo3DW"}},{"cell_type":"code","source":"def encode_cats_in_df(dataframe):\n    le = preprocessing.LabelEncoder()\n    for column in dataframe.columns:\n        if column in train_df[CATS].columns:\n            dataframe[column] = le.fit_transform(dataframe[column])\n    return dataframe","metadata":{"id":"DKZwzlGSeMpR"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### prepare_per_event_df\n\n\n","metadata":{"id":"efvTB_DXoE-n"}},{"cell_type":"code","source":"def prepare_per_event_df(dataframe):\n    encode_cats_in_df(dataframe)\n    dataframe.fillna(0, inplace=True)\n    scaler = MinMaxScaler()\n    dataframe[NUMS] = scaler.fit_transform(dataframe[NUMS])\n    session_id_mapping = dataframe['session_id'].to_frame().rename(columns = {'session_id':'original_sid'})\n    dataframe['session_id'] = preprocessing.LabelEncoder().fit_transform(dataframe['session_id'])\n    dataframe['session_id'] = dataframe['session_id'].astype(np.int64)\n    dataframe = dataframe.drop('index', axis = 1)\n    session_id_mapping['session_id'] =  dataframe['session_id']\n    session_id_mapping = session_id_mapping.drop_duplicates()\n\n    return dataframe, session_id_mapping","metadata":{"id":"RTvIZbOtLE5r"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\ntrain_df_per_event, session_id_mapping = prepare_per_event_df(train_df)\ntrain_df_per_event.to_csv(f'{dir_path}/train_df_per_event.csv', index = False)\ndel(train_df)\ndel(train_df_per_event)\ngc.collect()","metadata":{"id":"qVg2xlzcwYkX","outputId":"525ee06c-1c5f-48c9-bc80-66468bf8d1c0"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Engineered features \"part\" of train dataset","metadata":{"id":"hERGx9n2xm8t"}},{"cell_type":"markdown","source":"#### Common functions","metadata":{"id":"X86Gasb06coy"}},{"cell_type":"code","source":"def distance(x1, y1, x2, y2):\n    return sqrt((x2 - x1)**2 + (y2 - y1)**2)\n\n\ndef counter(data, event):\n    return len(data[data[\"event_name\"] == event])\n\n\ndef mean_event(data, event, param):\n    return np.mean(list(data[data[\"event_name\"] == event][param]))\n\n\ndef median_event(data, event, param):\n    return np.median(list(data[data[\"event_name\"] == event][param]))\n\n\ndef find_mean_distance_for_coor(data, event):\n    data = data[data[\"event_name\"] == event]\n    xs = list(data[\"room_coor_x\"])\n    ys = list(data[\"room_coor_y\"])\n    res = []\n    for i in range(len(xs) - 1):\n        res.append(distance(xs[i], ys[i], xs[i+1], ys[i+1]))\n    return np.mean(res)","metadata":{"id":"GHkboXuc6coz"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Binary functions","metadata":{"id":"j2ipb5yL6coz"}},{"cell_type":"code","source":"def find_fullscreen(data) -> float:\n    fullscreens = list(data[\"fullscreen\"])\n    return np.mean(fullscreens)\n\n\ndef find_musics(data) -> float:\n    musics = list(data[\"music\"])\n    return np.mean(musics)\n\n\ndef find_hqs(data) -> float:\n    hqs = list(data[\"hq\"])\n    return np.mean(hqs)","metadata":{"id":"OKJH7G6X6co0"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Checkpoints functions","metadata":{"id":"5sTwXEQh6co1"}},{"cell_type":"code","source":"def checkpoints_time(data):\n    checkpoints = data[data[\"event_name\"] == \"checkpoint\"]\n    time1 = int(checkpoints[0:1][\"elapsed_time\"])\n    time2 = int(checkpoints[1:2][\"elapsed_time\"]) - time1\n    time3 = int(data[-2:-1][\"elapsed_time\"]) -time1 - time2\n    return time1, time2, time3\n\ndef checkpoints_index(data):\n    checkpoints = data[data[\"event_name\"] == \"checkpoint\"]\n    index1 = int(checkpoints[0:1][\"index\"])\n    index2 = int(checkpoints[1:2][\"index\"]) - index1\n    index3 = int(data[-2:-1][\"index\"])- index1 - index2\n    return index1, index2, index3","metadata":{"id":"aAmxFMWL6co1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Make DataFrame for Engineered Features","metadata":{"id":"ZdJs0s2n6co2"}},{"cell_type":"code","source":"def click_processing(res, data, click, session_ids):\n    res[f\"{click}_click_counter\"] = [counter(data[data[\"session_id\"] == id], f\"{click}_click\") for id in tqdm(session_ids, desc=f\"{click} click counter\")]\n    res[f\"mean_level_{click}_click\"] = [mean_event(data[data[\"session_id\"] == id], f\"{click}_click\", \"level\") for id in tqdm(session_ids, desc=f\"mean level {click} click\")]\n    res[f\"median_level_{click}_click\"] = [median_event(data[data[\"session_id\"] == id], f\"{click}_click\", \"level\") for id in tqdm(session_ids, desc=f\"median level {click} click\")]\n    res[f\"mean_coor_{click}_click\"] = [find_mean_distance_for_coor(data[data[\"session_id\"] == id], f\"{click}_click\") for id in tqdm(session_ids, desc=f\"mean coordinate distance on room for {click} click\")]\n    return res","metadata":{"id":"6nKcfC0c6co3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def hover_processing(res, data, hover, session_ids):\n    res[f\"{hover}_hover_counter\"] = [counter(data[data[\"session_id\"] == id], f\"{hover}_hover\") for id in tqdm(session_ids, desc=f\"{hover} hover counter\")]\n    res[f\"mean_level_{hover}_hover\"] = [mean_event(data[data[\"session_id\"] == id], f\"{hover}_hover\", \"level\") for id in tqdm(session_ids, desc=f\"mean level {hover} hover\")]\n    res[f\"median_level_{hover}_hover\"] = [median_event(data[data[\"session_id\"] == id], f\"{hover}_hover\", \"level\") for id in tqdm(session_ids, desc=f\"median level {hover} hover\")]\n    res[f\"mean_duration_{hover}_hover\"] = [mean_event(data[data[\"session_id\"] == id], f\"{hover}_hover\", \"hover_duration\") for id in tqdm(session_ids, desc=f\"mean duration {hover} hover\")]\n    res[f\"median_duration_{hover}_hover\"] = [median_event(data[data[\"session_id\"] == id], f\"{hover}_hover\", \"hover_duration\") for id in tqdm(session_ids, desc=f\"median duration {hover} hover\")]\n    return res","metadata":{"id":"mp4X96s26co3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_dict(data):\n    session_ids = list(np.unique(data[\"session_id\"]))\n    res = {\"original_sid\": session_ids}\n\n    clicks = [\"cutscene\", \"map\", \"navigate\", \"notebook\", \"notification\", \"object\", \"observation\", \"person\"]\n    hovers = [\"object\", \"map\"]\n    for click in clicks:\n        res = click_processing(res, data, click, session_ids)\n\n    for hover in hovers:\n        res = hover_processing(res, data, hover, session_ids)\n\n    res[\"music\"] = [find_musics(data[data[\"session_id\"] == id]) for id in tqdm(session_ids, desc=\"calculate music res\")]\n    res[\"fullscreen\"] = [find_fullscreen(data[data[\"session_id\"] == id]) for id in tqdm(session_ids, desc=\"calculate fullscreen res\")]\n    res[\"music\"] = [find_musics(data[data[\"session_id\"] == id]) for id in tqdm(session_ids, desc=\"calculate music res\")]\n    res[\"hq\"] = [find_hqs(data[data[\"session_id\"] == id]) for id in tqdm(session_ids, desc=\"calculate hq res\")]\n\n\n    return pd.DataFrame(res)","metadata":{"id":"DGDJXnnR6co3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def engineered_features_df(dataframe = None):\n    dataframe = dataframe.rename(columns = {'id': 'original_sid'})\n    dataframe = dataframe.rename(columns = {'session_id': 'original_sid'})\n    dataframe.fillna(0, inplace=True)\n    session_id = dataframe['original_sid'].to_frame()\n    scaler = MinMaxScaler()\n    dataframe_normalized = pd.DataFrame(scaler.fit_transform(dataframe.iloc[:, 1:]), columns = dataframe.columns[1:])\n    dataframe_normalized = session_id.merge(dataframe_normalized, left_index=True, right_index=True)\n    dataframe_normalized = session_id_mapping.merge(dataframe_normalized, on = 'original_sid')\n    dataframe_normalized = dataframe_normalized.drop('original_sid', axis = 1)\n\n    return dataframe_normalized","metadata":{"id":"ECl5g5VKpYRS"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to obtain df_preprocessed\n\ndf_preprocessed = make_dict(train_df) but requires a few hours to calculate. So I calculated it previously and now can import it cell bellow","metadata":{"id":"HpPhyB2hBS8D"}},{"cell_type":"code","source":"df_preprocessed = pd.read_csv('/content/drive/MyDrive/predict-student-performance-from-game-play/df.csv').drop('Unnamed: 0', axis = 1)","metadata":{"id":"SXaXwjGDIHud"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataframe_normalized = engineered_features_df(df_preprocessed)","metadata":{"id":"_43A9RAyqaQk"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataframe_normalized.head()","metadata":{"id":"b_vbH2SxSRK8","outputId":"8b15336b-1141-4660-cb60-6c6b6c9abc50"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### convert_df_to_engineered_features_dict","metadata":{"id":"I5bfCNZq6co7"}},{"cell_type":"code","source":"# Convert data_normalized to to engineered features list of dict and save to .json\nimport json\ndef convert_df_to_engineered_features_dict(fe_normalized_data:pd.DataFrame):\n    session_to_sequences: Dict[int, dict] = {}\n    for _, event in fe_normalized_data.iterrows():\n      session_to_sequences[int(event[\"session_id\"])] = event[1:].to_list()\n\n    with open(f'{dir_path}/engineered_features_dict.json', \"w\") as output_file:\n      json.dump(session_to_sequences, output_file)         \n\n    return session_to_sequences","metadata":{"id":"ccoOlFc1rdfA"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert data_normalized to to engineered features list of dict and save to .json\nengineered_features_dict = convert_df_to_engineered_features_dict(dataframe_normalized)","metadata":{"id":"A_FnAny-q69v"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Labels preprocessing and convert data to list of dicts","metadata":{"id":"vUQT0_186co5"}},{"cell_type":"markdown","source":"#### process_lables","metadata":{"id":"oWGM1pIZFOYa"}},{"cell_type":"code","source":"labels = pd.read_csv(f'{dir_path}/train_labels.csv')\nlabels.head()","metadata":{"id":"IQShEkWY6co5","outputId":"6df6aa67-695b-45a2-c3f1-69f508abadac"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def process_lables(labels_dataframe):\n\n    labels_ids = list(labels[\"session_id\"])\n    corrects = list(labels[\"correct\"])\n\n    res = {}\n    res[\"session_id\"] = [session_id.split(\"_\")[0] for session_id in labels_ids]\n    res[\"question\"] = [session_id.split(\"_\")[1][1:] for session_id in labels_ids]\n    res[\"label\"] = corrects\n    processed_labels = pd.DataFrame(res)\n\n    # Encode labels in processed_lables dataset\n    processed_labels['session_id'] = processed_labels['session_id'].astype('int64')\n    processed_labels = processed_labels.rename(columns = {'session_id': 'original_sid'})\n    processed_labels = processed_labels.merge(session_id_mapping, on = 'original_sid')\n\n    return processed_labels","metadata":{"id":"zqZXZrSY6co5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_labels = process_lables(labels)\ndel(labels)\ngc.collect()","metadata":{"id":"HQvgU-Seu88N","outputId":"47e1a28f-84d8-4025-b8f1-85a9001b0d9e"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_labels.head(5)","metadata":{"id":"FKn1foF4AwIc","outputId":"6442dde2-1eeb-44b8-b5b4-bf7c30383c95"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### create_q2l_mapping and convert data to list of dicts","metadata":{"id":"H9mjM6tQbQEq"}},{"cell_type":"code","source":"def create_q2l_mapping(processed_labels):\n  \n    q_lists = processed_labels.groupby('session_id')['question'].apply(list).to_frame()\n    l_lists = processed_labels.groupby('session_id')['label'].apply(list).to_frame()\n    q2l_mapping = q_lists.merge(l_lists, on = 'session_id')\n\n    # Encode labels in q2l_mapping\n    q2l_mapping = q2l_mapping.merge(session_id_mapping, on = 'session_id')\n    q2l_mapping = q2l_mapping.rename(columns ={'question': 'q_list', 'label':'questions'})\n\n    return q2l_mapping","metadata":{"id":"9v4lZnYBKJfT"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q2l_mapping = create_q2l_mapping(processed_labels)","metadata":{"id":"u8qm9vG_w5eh"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# q2l_mapping = pd.read_json('/content/drive/MyDrive/predict-student-performance-from-game-play/q2l_mapping.json', lines=True).drop('Unnamed: 0', axis = 1)","metadata":{"id":"Wa6oB65Dln6v"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q2l_mapping.head()","metadata":{"id":"Aa1kXuzyFe1w","outputId":"bbd260fa-1c9a-46c6-a33e-adcc6584b6c5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processed_labels.head()","metadata":{"id":"ZdYESTDcYOfE","outputId":"d8ffb6f0-c7a5-4931-c3be-031bb872ab72"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Convertation of train dataframe to list of dicts (here 'train dataframe' is still per-event train_df with all questions labels and without engineered features)","metadata":{"id":"-kRbJ4AtpSMC"}},{"cell_type":"code","source":"import json\ndef convert_raw_data_to_list_of_dicts(src_data:pd.DataFrame, output_file_name:str, mode = 'train'):\n    final_data = []\n    student_to_sequences: Dict[int, dict] = {}\n    for _, event in tqdm(src_data.iterrows(), desc=\"calculate res\"):\n      if mode == 'train':\n          sequence: Dict[str, list] = student_to_sequences.setdefault(event[\"session_id\"], {\n                    \"session_id\" : event[\"session_id\"],\n                    \"elapsed_time\" : [], \"event_name\" : [],\n                    \"name\" : [], \"level\" : [],\n                    \"page\" : [], \"room_coor_x\" : [],\n                    \"room_coor_y\" : [], \"screen_coor_x\" : [],\n                    \"screen_coor_y\" : [], \"hover_duration\" : [],\n                    \"text\" : [], \"fqid\" : [],\n                    \"room_fqid\" : [], \"text_fqid\" : [],\n                    \"questions\" :  list(q2l_mapping['questions'][q2l_mapping.index[q2l_mapping['session_id'] == event['session_id']].values[0]])\n                })\n      elif mode == 'test':\n          sequence: Dict[str, list] = student_to_sequences.setdefault(event[\"session_id\"], {\n                    \"session_id\" : event[\"session_id\"],\n                    \"elapsed_time\" : [], \"event_name\" : [],\n                    \"name\" : [], \"level\" : [],\n                    \"page\" : [], \"room_coor_x\" : [],\n                    \"room_coor_y\" : [], \"screen_coor_x\" : [],\n                    \"screen_coor_y\" : [], \"hover_duration\" : [],\n                    \"text\" : [], \"fqid\" : [],\n                    \"room_fqid\" : [], \"text_fqid\" : []\n                })\n      sequence[\"elapsed_time\"].append(event[\"elapsed_time\"])\n      sequence[\"event_name\"].append(event[\"event_name\"])\n      sequence[\"name\"].append(event[\"name\"])\n      sequence[\"level\"].append(event[\"level\"])\n      sequence[\"page\"].append(event[\"page\"])\n      sequence[\"room_coor_x\"].append(event[\"room_coor_x\"])\n      sequence[\"room_coor_y\"].append(event[\"room_coor_y\"])\n      sequence[\"screen_coor_x\"].append(event[\"screen_coor_x\"])\n      sequence[\"screen_coor_y\"].append(event[\"screen_coor_y\"])\n      sequence[\"hover_duration\"].append(event[\"hover_duration\"])\n      sequence[\"text\"].append(event[\"text\"])\n      sequence[\"fqid\"].append(event[\"fqid\"])\n      sequence[\"room_fqid\"].append(event[\"room_fqid\"])\n      sequence[\"text_fqid\"].append(event[\"text_fqid\"])\n\n\n    final_data += list(student_to_sequences.values())\n    with open(f'{dir_path}/{output_file_name}.json', \"w\") as output_file:\n      json.dump(final_data, output_file)\n    return final_data","metadata":{"id":"7NvbqxhFw0O2"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Per-Question Train DataFrames, Loaders and Model","metadata":{"id":"Y-CkVxnZtBpf"}},{"cell_type":"markdown","source":"### Train test splits and convertations to list of dicts","metadata":{"id":"p7HKDI0W-CRZ"}},{"cell_type":"code","source":"def split_by_level_group(dir_path:str, source_name:str, res_1_name:str, res_2_name:str = None, res_3_name:str = None):  \n    reader = pd.read_csv(f'{dir_path}/{source_name}.csv', chunksize=100000)\n    result_template = pd.read_csv(f'{dir_path}/{source_name}.csv', nrows = 10)\n    df_result = pd.DataFrame(columns=result_template.columns)\n    df_result.to_csv(f'{dir_path}/{res_1_name}.csv', index_label=False)\n    df_result.to_csv(f'{dir_path}/{res_2_name}.csv', index_label=False)\n    df_result.to_csv(f'{dir_path}/{res_3_name}.csv', index_label=False) \n  \n    def preprocess(chunk):\n      df1=chunk[chunk['level'] < 5]\n      df1.to_csv(f'{dir_path}/{res_1_name}.csv', mode=\"a\", header=False, index=False)\n      \n      '''\n      df2=chunk[(chunk['level'] >= 5) & (chunk['level'] < 13)]\n      df2.to_csv(f'{dir_path}/{res_2_name}.csv', mode=\"a\", header=False, index=False)\n\n      df3=chunk[chunk['level'] >= 13]\n      df3.to_csv(f'{dir_path}/{res_3_name}.csv', mode=\"a\", header=False, index=False)\n      '''\n    [preprocess(r) for r in reader]","metadata":{"id":"PM_XKJd7XZDE"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"split_by_level_group(dir_path, 'train_df_per_event', 'train_gr_1')\n# split_by_level_group(dir_path, 'train_df_per_event', 'train_gr_1', 'train_gr_2', 'train_gr_3')","metadata":{"id":"cLY2Yqxc1NSB"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def split_to_train_and_test(train_gr_n:str):\n    reader = pd.read_csv(f'{dir_path}/{train_gr_n}.csv', chunksize=100000)\n    result_template = pd.read_csv(f'{dir_path}/{train_gr_n}.csv', nrows = 10)\n    df_result = pd.DataFrame(columns=result_template.columns)\n    df_result.to_csv(f'{dir_path}/{train_gr_n}_train.csv', index_label=False)\n    df_result.to_csv(f'{dir_path}/{train_gr_n}_val.csv', index_label=False)\n  \n    def preprocess_split(chunk):\n      df1, df2 = train_test_split(chunk, test_size=0.2)\n      df1.to_csv(f'{dir_path}/{train_gr_n}_train.csv', mode=\"a\",header=False,index=False)\n      df2.to_csv(f'{dir_path}/{train_gr_n}_val.csv', mode=\"a\",header=False,index=False)\n    \n    [preprocess_split(r) for r in reader]","metadata":{"id":"MiEc_ukV1dyR"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"split_to_train_and_test('train_gr_1')","metadata":{"id":"pmcAMoi33lk3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# split_to_train_and_test('train_gr_2')","metadata":{"id":"NIQj5MZW3x7f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# split_to_train_and_test('train_gr_3')","metadata":{"id":"7WuIzuB43x-6"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train and Val Dataframes Splitted in 3 groups (for '0-4', '5-12' and '13-22' levels)","metadata":{"id":"mTfy7pQUNPeB"}},{"cell_type":"code","source":"import gc\ndir_path = '/content/drive/MyDrive/ref-predict-student-performance-from-game-play'\n\n# Make lists of dicts\ntrain_gr_1_train = pd.read_csv(f'{dir_path}/train_gr_1_train.csv')\ntrain_gr_1_converted_train = convert_raw_data_to_list_of_dicts(train_gr_1_train, 'train_gr_1_converted_train', mode = 'train')\ndel(train_gr_1_train)\ngc.collect()","metadata":{"id":"Y__au-YbxYqP","outputId":"00ad96f3-9ee0-480f-c3aa-58332c20fc76"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_gr_1_val = pd.read_csv(f'{dir_path}/train_gr_1_val.csv')\ntrain_gr_1_converted_val = convert_raw_data_to_list_of_dicts(train_gr_1_val, 'train_gr_1_converted_val', mode = 'train')\ndel(train_gr_1_val)\ngc.collect()","metadata":{"id":"f8vbqlCujeNo","outputId":"e145a130-07b5-4a9e-b67a-c6078e281db8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_gr_2_train = pd.read_csv(f'{dir_path}/train_gr_2_train.csv')\n# train_gr_2_converted_train = convert_raw_data_to_list_of_dicts(train_gr_2_train, 'train_gr_2_converted_train')\n# del(train_gr_2_train)\n# gc.collect()","metadata":{"id":"tqNJVkJlnBMb"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_gr_2_val = pd.read_csv(f'{dir_path}/train_gr_2_val.csv')\n# train_gr_2_converted_val = convert_raw_data_to_list_of_dicts(train_gr_2_val, 'train_gr_2_converted_val')\n# del(train_gr_2_val)\n# gc.collect()","metadata":{"id":"4GDg-wD5nDfl"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_gr_3_train = pd.read_csv(f'{dir_path}/train_gr_3_train.csv')\n# train_gr_3_converted_train = convert_raw_data_to_list_of_dicts(train_gr_3_train, 'train_gr_3_converted_train')\n# del(train_gr_3_train)\n# gc.collect()","metadata":{"id":"tFlfQpetnFCs"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_gr_3_val = pd.read_csv(f'{dir_path}/train_gr_3_val.csv')\n# train_gr_3_converted_val = convert_raw_data_to_list_of_dicts(train_gr_3_val, 'train_gr_3_converted_val')\n# del(train_gr_3_val)\n# gc.collect()","metadata":{"id":"occiVmXdnGpp"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### PerQuestionDataset (torch dataset)\nAggregates and adds list of all labels instead of just one label","metadata":{"id":"PyRt0YoSy_hs"}},{"cell_type":"code","source":"class DatasetPerQuestion(torch.utils.data.Dataset):\n    def __init__(self, data: List[Dict[str, list]], mode = 'train'):\n\n        self.data = []\n\n        # Iterate over given data list and process each sequence\n        for sequence in data:\n            sequence_len = len(sequence[\"elapsed_time\"])\n            sequence[\"session_id\"] = sequence[\"session_id\"]\n            sequence[\"mask\"] = torch.ones(sequence_len, dtype=torch.bool)\n            sequence[\"engineered_features\"] = np.array(engineered_features_dict[sequence[\"session_id\"]], dtype=np.float32)\n            sequence[\"elapsed_time\"] = np.array(sequence[\"elapsed_time\"], dtype=np.float32)\n            sequence[\"event_name\"] = np.array(sequence[\"event_name\"], dtype=np.int64)\n            sequence[\"name\"] = np.array(sequence[\"name\"], dtype=np.int64)\n            sequence[\"level\"] = np.array(sequence[\"level\"], dtype=np.int64)\n            sequence[\"page\"] = np.array(sequence[\"page\"], dtype=np.int64)\n            sequence[\"room_coor_x\"] = np.array(sequence[\"room_coor_x\"], dtype=np.int64)\n            sequence[\"room_coor_y\"] = np.array(sequence[\"room_coor_y\"], dtype=np.int64)\n            sequence[\"screen_coor_x\"] = np.array(sequence[\"screen_coor_x\"], dtype=np.int64)\n            sequence[\"screen_coor_y\"] = np.array(sequence[\"screen_coor_y\"], dtype=np.int64)\n            sequence[\"hover_duration\"] = np.array(sequence[\"hover_duration\"], dtype=np.int64)\n            sequence[\"text\"] = np.array(sequence[\"text\"], dtype=np.int64)\n            sequence[\"fqid\"] = np.array(sequence[\"fqid\"], dtype=np.int64)\n            sequence[\"room_fqid\"] = np.array(sequence[\"room_fqid\"], dtype=np.int64)\n            sequence[\"text_fqid\"] = np.array(sequence[\"text_fqid\"], dtype=np.int64)\n            if mode == 'train':\n              sequence[\"questions\"] = np.array(sequence[\"questions\"], dtype=bool)\n            self.data.append(sequence)\n        print(\"Data size:\", len(self.data))\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, index):\n        return self.data[index]\n","metadata":{"id":"i6KWFWuCChR8"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### PerQuestionCollator \nReturns sequence of labels in batch['label'] instead of just one label","metadata":{"id":"IcSk9xgiy3y8"}},{"cell_type":"code","source":"class CollatorPerQuestion:\n    def __init__(self, random_trim=False, mode = 'train'):\n        self.mode = mode\n\n    def __call__(self, batch: List[Dict]):\n        \n        session_id_batches = []\n        mask_batches = []\n        elapsed_time_batches = []\n        event_name_batches = []\n        name_batches = []\n        level_batches = []\n        page_batches = []\n        room_coor_x_batches = []\n        room_coor_y_batches = []\n        screen_coor_x_batches = []\n        screen_coor_y_batches = []\n        hover_duration_batches = []\n        text_batches = []\n        fqid_batches = []\n        room_fqid_batches = []\n        text_fqid_batches = []\n        engineered_features_batches = []\n        questions_batches = []\n\n        for sequence in batch:\n            # Convert data structures to torch tensors\n\n            mask = sequence[\"mask\"]\n            elapsed_time = torch.from_numpy(sequence[\"elapsed_time\"])\n            event_name = torch.LongTensor(sequence[\"event_name\"])\n            name = torch.from_numpy(sequence[\"name\"])\n            level = torch.from_numpy(sequence[\"level\"])\n            page = torch.from_numpy(sequence[\"page\"])\n            room_coor_x = torch.from_numpy(sequence[\"room_coor_x\"])\n            room_coor_y = torch.from_numpy(sequence[\"room_coor_y\"])\n            screen_coor_x = torch.from_numpy(sequence[\"screen_coor_x\"])\n            screen_coor_y = torch.from_numpy(sequence[\"screen_coor_y\"])\n            hover_duration = torch.from_numpy(sequence[\"hover_duration\"])\n            text = torch.from_numpy(sequence[\"text\"])\n            fqid = torch.from_numpy(sequence[\"fqid\"])\n            room_fqid = torch.from_numpy(sequence[\"room_fqid\"])\n            text_fqid = torch.from_numpy(sequence[\"text_fqid\"])\n            engineered_features = torch.from_numpy(sequence[\"engineered_features\"])\n            if self.mode == 'train':\n                questions = torch.from_numpy(sequence[\"questions\"])\n\n            session_id_batches.append(sequence['session_id'])\n            mask_batches.append(mask)\n            elapsed_time_batches.append(elapsed_time)\n            event_name_batches.append(event_name)\n            name_batches.append(name)\n            level_batches.append(level)\n            page_batches.append(page)\n            room_coor_x_batches.append(room_coor_x)\n            room_coor_y_batches.append(room_coor_y)\n            screen_coor_x_batches.append(screen_coor_x)\n            screen_coor_y_batches.append(screen_coor_y)\n            hover_duration_batches.append(hover_duration)\n            text_batches.append(text)\n            fqid_batches.append(fqid)\n            room_fqid_batches.append(room_fqid)\n            text_fqid_batches.append(text_fqid)\n            engineered_features_batches.append(engineered_features)\n            if self.mode == 'train':\n                questions_batches.append(questions)\n\n        \n        return {          \n            'elapsed_time': torch.nn.utils.rnn.pad_sequence(elapsed_time_batches, batch_first=True).to(device),\n            'event_name' : torch.nn.utils.rnn.pad_sequence(event_name_batches, batch_first=True).to(device), \n            'name' : torch.nn.utils.rnn.pad_sequence(name_batches, batch_first=True).to(device), \n            'level' : torch.nn.utils.rnn.pad_sequence(level_batches, batch_first=True).to(device), \n            'page' : torch.nn.utils.rnn.pad_sequence(page_batches, batch_first=True).to(device), \n            'room_coor_x' : torch.nn.utils.rnn.pad_sequence(room_coor_x_batches, batch_first=True).to(device), \n            'room_coor_y' : torch.nn.utils.rnn.pad_sequence(room_coor_y_batches, batch_first=True).to(device), \n            'screen_coor_x' : torch.nn.utils.rnn.pad_sequence(screen_coor_x_batches, batch_first=True).to(device), \n            'screen_coor_y' : torch.nn.utils.rnn.pad_sequence(screen_coor_y_batches, batch_first=True).to(device), \n            'hover_duration' : torch.nn.utils.rnn.pad_sequence(hover_duration_batches, batch_first=True).to(device), \n            'text' : torch.nn.utils.rnn.pad_sequence(text_batches, batch_first=True).to(device), \n            'fqid' : torch.nn.utils.rnn.pad_sequence(fqid_batches, batch_first=True).to(device), \n            'room_fqid' : torch.nn.utils.rnn.pad_sequence(room_fqid_batches, batch_first=True).to(device), \n            'text_fqid' : torch.nn.utils.rnn.pad_sequence(text_fqid_batches, batch_first=True).to(device), \n            \"engineered_features\": torch.Tensor(engineered_features).to(device),\n            \"mask\": torch.nn.utils.rnn.pad_sequence(mask_batches, batch_first=True).to(device),\n            \"session_id\": torch.LongTensor(session_id_batches).to(device),\n            \"questions\": torch.stack(questions_batches).to(device),\n            \"sequence_lengths\": torch.LongTensor([seq.shape[0] for seq in event_name_batches]) # Must be on CPU\n        } if self.mode == 'train' else {\n            \n            'elapsed_time': torch.nn.utils.rnn.pad_sequence(elapsed_time_batches, batch_first=True).to(device),\n            'event_name' : torch.nn.utils.rnn.pad_sequence(event_name_batches, batch_first=True).to(device), \n            'name' : torch.nn.utils.rnn.pad_sequence(name_batches, batch_first=True).to(device), \n            'level' : torch.nn.utils.rnn.pad_sequence(level_batches, batch_first=True).to(device), \n            'page' : torch.nn.utils.rnn.pad_sequence(page_batches, batch_first=True).to(device), \n            'room_coor_x' : torch.nn.utils.rnn.pad_sequence(room_coor_x_batches, batch_first=True).to(device), \n            'room_coor_y' : torch.nn.utils.rnn.pad_sequence(room_coor_y_batches, batch_first=True).to(device), \n            'screen_coor_x' : torch.nn.utils.rnn.pad_sequence(screen_coor_x_batches, batch_first=True).to(device), \n            'screen_coor_y' : torch.nn.utils.rnn.pad_sequence(screen_coor_y_batches, batch_first=True).to(device), \n            'hover_duration' : torch.nn.utils.rnn.pad_sequence(hover_duration_batches, batch_first=True).to(device), \n            'text' : torch.nn.utils.rnn.pad_sequence(text_batches, batch_first=True).to(device), \n            'fqid' : torch.nn.utils.rnn.pad_sequence(fqid_batches, batch_first=True).to(device), \n            'room_fqid' : torch.nn.utils.rnn.pad_sequence(room_fqid_batches, batch_first=True).to(device), \n            'text_fqid' : torch.nn.utils.rnn.pad_sequence(text_fqid_batches, batch_first=True).to(device), \n            \"engineered_features\": torch.Tensor(engineered_features).to(device),\n            \"mask\": torch.nn.utils.rnn.pad_sequence(mask_batches, batch_first=True).to(device),\n            \"session_id\": torch.LongTensor(session_id_batches).to(device),\n            \"sequence_lengths\": torch.LongTensor([seq.shape[0] for seq in event_name_batches]) # Must be on CPU\n        }","metadata":{"id":"lYO6jpyHfeC1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3 groups of loaders for per-question LSTM\nOne group of loaders (train + val loaders) for each session group ('0-4', '5-12' and '13-22')","metadata":{"id":"iSQwdt73ywCR"}},{"cell_type":"markdown","source":"Group 1 val and train datasets and loaders ","metadata":{"id":"diu-UrY2ocWf"}},{"cell_type":"code","source":"train_dataset_gr_1 = DatasetPerQuestion(train_gr_1_converted_train)\ntrain_loader_gr_1 = torch.utils.data.DataLoader(\n    train_dataset_gr_1,\n    collate_fn=CollatorPerQuestion())\ndel(train_dataset_gr_1)\ngc.collect()\ntorch.save(train_loader_gr_1, f'{dir_path}/train_loader_gr_1.pt')","metadata":{"id":"Amceh6S1IuE6","outputId":"07642cf0-0b04-4396-cea2-72e4a17546ca"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_dataset_gr_1 = DatasetPerQuestion(train_gr_1_converted_val)\nvalidation_loader_gr_1 = torch.utils.data.DataLoader(\n    val_dataset_gr_1,\n    collate_fn=CollatorPerQuestion())\n\ndel(val_dataset_gr_1)\ngc.collect()\ntorch.save(validation_loader_gr_1, f'{dir_path}/validation_loader_gr_1.pt')","metadata":{"id":"jAg3p3pBlgzb","outputId":"b81492b8-883b-4d1d-e106-5bd19a886786"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Group 2 val and train datasets and loaders ","metadata":{"id":"oZ3YseqioZhV"}},{"cell_type":"code","source":"# val_dataset_gr_2 = DatasetPerQuestion(train_gr_2_converted_val)\n# validation_loader_gr_2 = torch.utils.data.DataLoader(\n#     val_dataset_gr_2,\n#     collate_fn=CollatorPerQuestion())\n\n# del(val_dataset_gr_2)\n# gc.collect()\n# torch.save(validation_loader_gr_2, f'{dir_path}/validation_loader_gr_2.pt')","metadata":{"id":"2Zkkc_OIo_UZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_dataset_gr_2 = DatasetPerQuestion(train_gr_2_converted_train)\n# train_loader_gr_2 = torch.utils.data.DataLoader(\n#     train_dataset_gr_2,\n#     collate_fn=CollatorPerQuestion())\n\n# del(train_dataset_gr_2)\n# gc.collect()\n# torch.save(train_loader_gr_2, f'{dir_path}/train_loader_gr_2.pt')","metadata":{"id":"A7yhrGtgqB72"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Group 3 val and train datasets and loaders ","metadata":{"id":"IJI5fdkrsd9E"}},{"cell_type":"code","source":"# val_dataset_gr_3 = DatasetPerQuestion(train_gr_3_converted_val)\n# validation_loader_gr_3 = torch.utils.data.DataLoader(\n#     val_dataset_gr_3,\n#     collate_fn=CollatorPerQuestion())\n\n# torch.save(validation_loader_gr_3, f'{dir_path}/validation_loader_gr_3.pt')\n# del(val_dataset_gr_3)\n# gc.collect()","metadata":{"id":"DWTkjBURp6sB"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_dataset_gr_3 = DatasetPerQuestion(train_gr_3_converted_train)\n# train_loader_gr_3 = torch.utils.data.DataLoader(\n#     train_dataset_gr_3,\n#     collate_fn=CollatorPerQuestion())\n# torch.save(train_loader_gr_3, f'{dir_path}/train_loader_gr_3.pt')\n# del(train_dataset_gr_3)\n# gc.collect()","metadata":{"id":"uzUiagco6EzC"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Per-question LSTM, train & eval functions, train loop","metadata":{"id":"dWETyUoYydMK"}},{"cell_type":"markdown","source":"Per-question LSTM (question_x) receives question number \"x\" and takes accordingly \"x-1\" label from label's list in batch['label']","metadata":{"id":"NzjtRJAYpDkF"}},{"cell_type":"code","source":"num_event_name_types = 11\nname_types = 6\nlevel_types = 23\npage_types = 8\nroom_fqid_types = 19\ntext_fqid_types = 127 \n\nevent_name_embedding_size = 16 \nname_embedding_size = 8\nlevel_embedding_size = 32\npage_embedding_size = 32\nroom_fqid_embedding_size = 32\ntext_fqid_embedding_size = 128\nhidden_size = 300\n\nclass LSTMModelPerQuestion(nn.Module):\n    def __init__(self, question_number: int, num_labels: int = 1, pred_classes: list = None):\n        super().__init__()\n        \n        self.question_number = question_number\n        self.num_event_name_types = num_event_name_types\n        self.name_types = name_types\n        self.level_types = level_types\n        self.page_types = page_types\n        self.room_fqid_types = room_fqid_types\n        self.text_fqid_types = text_fqid_types\n\n        self.event_name_embeddings = nn.Embedding(self.num_event_name_types, event_name_embedding_size)\n        self.name_embeddings = nn.Embedding(self.name_types, name_embedding_size)\n        self.level_embeddings = nn.Embedding(self.level_types, level_embedding_size)\n        self.page_embeddings = nn.Embedding(self.page_types, page_embedding_size)\n        self.room_fqid_embeddings = nn.Embedding(self.room_fqid_types, room_fqid_embedding_size)\n        self.text_fqid_embeddings = nn.Embedding(self.text_fqid_types, text_fqid_embedding_size)\n\n        input_size = event_name_embedding_size + name_embedding_size  + level_embedding_size ++ page_embedding_size + room_fqid_embedding_size + text_fqid_embedding_size + 1 # + level_embedding_size + page_embedding_size + room_fqid_embedding_size + text_fqid_embedding_size + \n\n        self.lstm = nn.LSTM(\n            input_size=input_size,\n            hidden_size=hidden_size,\n            batch_first=True,\n            bidirectional=True\n        )\n\n        self.num_labels = num_labels\n        output_size = hidden_size * 2   \n        final_layer_size = output_size + 45 # engineered_features size == data_normalized.shape[1] - 2 == 45 (features dict mistake is fixed)\n        self.attention = nn.Linear(output_size, 1)\n        self.hidden_layers = nn.Sequential(\n                 nn.Dropout(p=0.2), nn.ReLU(), nn.Linear(output_size, output_size))\n        self.pred_output_layer = nn.Sequential(\n                nn.Dropout(p=0.2), nn.Linear(final_layer_size, num_labels))\n        \n    def forward(self, batch):\n\n        batch_size = batch[\"event_name\"].shape[0]\n\n        # Construct input and run through LSTM\n        event_name = self.event_name_embeddings(batch[\"event_name\"])\n        name = self.name_embeddings(batch[\"name\"])\n        level = self.level_embeddings(batch[\"level\"])\n        page = self.page_embeddings(batch[\"page\"])\n        room_fqid = self.room_fqid_embeddings(batch[\"room_fqid\"])\n        text_fqid = self.text_fqid_embeddings(batch[\"text_fqid\"])\n        \n        # Add a third dimension to be able to concat with embeddings\n        elapsed_time = batch[\"elapsed_time\"].unsqueeze(2) # TO DO : add ['hover_duration'] \n        input_tensors = [event_name, name,  level, page, room_fqid, text_fqid, elapsed_time]\n      \n        lstm_input = torch.cat(input_tensors, dim=-1)\n        packed_lstm_input = torch.nn.utils.rnn.pack_padded_sequence(\n            lstm_input, lengths=batch[\"sequence_lengths\"], batch_first=True, enforce_sorted=False)\n        packed_lstm_output, (hidden, _) = self.lstm(packed_lstm_input) # needs correct train dataset\n        lstm_output, _ = torch.nn.utils.rnn.pad_packed_sequence(packed_lstm_output, batch_first=True)\n\n        attention = self.attention\n        hidden_layers = self.hidden_layers\n        pred_output_layer = self.pred_output_layer\n\n        pred_state = None\n        # Multiply each output vector with learnable attention vector to get attention activations at each timestep\n        activations = attention(lstm_output).squeeze(2) # batch_size x max_seq_len \n        # Apply mask so that output in padding regions gets 0 probability after softmax\n        activations[batch[\"mask\"] == 0] = -torch.inf \n        \n        # Apply softmax to get distribution across timesteps of each sequence\n        attention_weights = nn.Softmax(dim=1)(activations) # batch_size x max_seq_len\n        # Multiply each output vector with its corresponding attention weight\n        weighted_output = lstm_output * attention_weights.unsqueeze(2)\n        # Add weighted output vectors along each sequence in the batch\n        pred_state = torch.sum(weighted_output, dim=1)\n\n        pred_state = hidden_layers(pred_state)\n\n        # Append engineered features to latent state if needed (note that we don't want this )  \n        engineered_features = batch[\"engineered_features\"].unsqueeze(0)\n        pred_state = torch.cat([pred_state, engineered_features], dim=1)\n        predictions = pred_output_layer(pred_state)\n        predictions = predictions.view(-1)\n\n        # Get cross entropy loss of predictions with labels, note that this automatically performs the sigmoid step\n        loss_fn = nn.BCEWithLogitsLoss(reduction=\"mean\")\n        indices = torch.tensor([self.question_number - 1]).to(device)\n        question_number_tensor = torch.index_select(batch[\"questions\"], 1, indices.view(-1))\n        avg_loss = loss_fn(predictions, question_number_tensor.squeeze(1).float())\n\n        return avg_loss, predictions.detach().cpu().numpy()","metadata":{"id":"3sdnUsekgRvK"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Per-Question Train and Evaluate Functions","metadata":{"id":"emzszIZSxYpi"}},{"cell_type":"code","source":"def evaluate_PQ_model(model, validation_loader: torch.utils.data.DataLoader, question_number):\n    total_loss = 0\n    num_batches = 0\n    all_predictions = []\n    all_labels = []\n    with torch.no_grad():\n        for batch in validation_loader:\n            loss, predictions = model(batch)\n            \n            all_predictions.append(predictions)\n            indices = torch.tensor([question_number - 1]).to(device)\n            question_number_tensor = torch.index_select(batch[\"questions\"], 1, indices.view(-1))\n            all_labels.append(question_number_tensor.squeeze(1).float().detach().cpu().numpy())\n            total_loss += float(loss.detach().cpu().numpy())\n            num_batches += 1\n\n    all_preds_np = np.concatenate(all_predictions, axis=0)\n    all_labels_np = np.concatenate(all_labels, axis=0)\n    auc = metrics.roc_auc_score(all_labels_np, all_preds_np)\n    adj_auc = 2 * (auc - .5)\n    # Collapse predictions to calculate accuracy and kappa\n    all_preds_np[all_preds_np < 0] = 0\n    all_preds_np[all_preds_np > 0] = 1\n    # Flatten to handle the multi-label case\n    all_preds_np = all_preds_np.flatten()\n    all_labels_np = all_labels_np.flatten()\n    accuracy = metrics.accuracy_score(all_labels_np, all_preds_np)\n    f_1_score = metrics.f1_score(all_labels_np, all_preds_np, average='macro')\n    kappa = metrics.cohen_kappa_score(all_labels_np, all_preds_np)\n    agg = adj_auc + kappa\n    return total_loss / num_batches, accuracy, f_1_score, auc, kappa, agg","metadata":{"id":"Ez8iasxKxXab"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_stats_all = []\ndef train_PQ(model, question_number, train_loader, validation_loader, model_name: str = 'model', lr=1e-5, weight_decay=1e-6, epochs= 1, patience=1):\n    optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)\n    torch.autograd.set_detect_anomaly(True) # Pause exectuion and get stack trace if something weird happens (ex: NaN grads)\n    best_metric = None\n    best_stats = None\n    cur_stats = None\n    best_epoch = 0\n    for epoch in range(epochs):\n        start_time = time.time()\n        model.train() # Set model to training mode\n        train_loss = 0\n        num_batches = 0\n        for batch in train_loader:\n            optimizer.zero_grad()\n            loss, _ = model(batch)\n            loss.backward()\n            optimizer.step()\n            train_loss += float(loss.detach().cpu().numpy())\n            num_batches += 1\n\n        model.eval() # Set model to evaluation mode\n\n        train_loss, train_accuracy, train_f_1_score, train_auc, train_kappa, train_agg = evaluate_PQ_model(model, train_loader, question_number)\n        val_loss, val_accuracy, val_f_1_score, val_auc, val_kappa, val_agg = evaluate_PQ_model(model, validation_loader, question_number)\n        cur_stats = [epoch, val_loss, val_accuracy, val_f_1_score, val_auc, val_kappa, val_agg]\n        print(f\"Epoch: {epoch + 1}, Question: {question_number}, Train Loss: {train_loss:.3f}, Acc: {train_accuracy:.3f}, F1: {train_f_1_score:.3f}, AUC: {train_auc:.3f}, Kappa: {train_kappa:.3f}, Agg: {train_agg:.3f}, \"\n            f\"Val Loss: {val_loss:.3f}, Acc: {val_accuracy:.3f}, F1: {val_f_1_score:.3f}, AUC: {val_auc:.3f}, Kappa: {val_kappa:.3f}, Agg: {val_agg:.3f}, \"\n            f\"Time: {time.time() - start_time:.2f}\")\n        \n        if not best_metric or val_loss < best_metric:\n            best_metric = val_loss\n            best_epoch = epoch\n            best_stats = cur_stats\n            print(\"Saving model\")\n            torch.save(model.state_dict(), f\"{model_name}_for_question_{question_number}.pt\")\n        \n        best_stats_all.append(best_stats)\n\n    return best_stats","metadata":{"id":"lJtTRF4Zx5V2"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model training \n## Train loop for each question. 3 different dataloaders, each has all labels","metadata":{"id":"ogQnQEBsyQWZ"}},{"cell_type":"code","source":"best_stats_all = []\n# ITERATE THRU QUESTIONS 1 THRU 18\nfor t in range(1,2):\n    \n    # USE THIS TRAIN DATA WITH THESE QUESTIONS\n    if t<=3:\n        train_loader = train_loader_gr_1\n        validation_loader = validation_loader_gr_1\n\n    '''\n    elif t<=13:\n        train_loader = train_loader_gr_2\n        validation_loader = validation_loader_gr_2\n\n    elif t<=22:\n        train_loader = train_loader_gr_3\n        validation_loader = validation_loader_gr_3\n    '''\n    \n    # TRAIN MODEL, SAVE MODEL, PREDICT F1        \n    model_PQ =  LSTMModelPerQuestion(t).to(device)\n    train_PQ(model_PQ, t, train_loader, validation_loader)\n    \nprint()","metadata":{"id":"1vcNoD23eA_D","outputId":"9fb2193d-d462-46a2-a766-742157e8f02d"},"execution_count":null,"outputs":[]}]}