{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Student Performance from Game Play Using TensorFlow Decision Forests\n\n---\n\nThis notebook will take you through the steps needed to train a baseline Gradient Boosted Trees Model using TensorFlow Decision Forests on the `Student Performance from Game Play` dataset made available for this competition, to predict if players will answer questions correctly.\nWe will load the data from a CSV file. Roughly, the code will look as follows:\n\n```\nimport tensorflow_decision_forests as tfdf\nimport pandas as pd\n  \ndataset = pd.read_csv(\"project/dataset.csv\")\ntf_dataset = tfdf.keras.pd_dataframe_to_tf_dataset(dataset, label=\"my_label\")\n\nmodel = tfdf.keras.GradientBoostedTreesModel()\nmodel.fit(tf_dataset)\n  \nprint(model.summary())\n```\n\nWe will also learn how to optimize reading of big datasets, do some feature engineering, data visualization and calculate better results using the F1-score\n\n\nDecision Forests are a family of tree-based models including Random Forests and Gradient Boosted Trees. They are the best place to start when working with tabular data, and will often outperform (or provide a strong baseline) before you begin experimenting with neural networks.","metadata":{"id":"iCTA6kl3n2O1"}},{"cell_type":"markdown","source":"One of the key aspects of TensorFlow Decision Forests that makes it even more suitable for this competition, particularly given the runtime limitations, is that it has been extensively tested for training and inference on CPUs, making it possible to train it on lower-end machines.","metadata":{}},{"cell_type":"markdown","source":"# Import the Required Libraries","metadata":{"id":"zAXHC6-Tn2O5"}},{"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow_addons as tfa\nimport tensorflow_decision_forests as tfdf\nfrom transformers import TFBertForSequenceClassification\n\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt","metadata":{"id":"IanlX-Eqn2O5","execution":{"iopub.status.busy":"2023-05-25T08:12:15.738742Z","iopub.execute_input":"2023-05-25T08:12:15.739503Z","iopub.status.idle":"2023-05-25T08:12:30.445956Z","shell.execute_reply.started":"2023-05-25T08:12:15.739466Z","shell.execute_reply":"2023-05-25T08:12:30.444653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"TensorFlow Decision Forests v\" + tfdf.__version__)\nprint(\"TensorFlow Addons v\" + tfa.__version__)\nprint(\"TensorFlow v\" + tf.__version__)","metadata":{"id":"gLpK2yAen2O7","execution":{"iopub.status.busy":"2023-05-11T09:21:12.192444Z","iopub.execute_input":"2023-05-11T09:21:12.192946Z","iopub.status.idle":"2023-05-11T09:21:12.200711Z","shell.execute_reply.started":"2023-05-11T09:21:12.192903Z","shell.execute_reply":"2023-05-11T09:21:12.199281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the Dataset\n\nSince the dataset is huge, some people may face memory errors while reading the dataset from the csv. To avoid this, we will try to optimize the memory used by Pandas to load and store the dataset.\n\n\nWhen Pandas loads a dataset, by default, it automatically detects the data types of the different columns.\nIrresepective of the maximum value that is stored in these columns, Pandas assigns `int64` for numerical columns, `float64` for float columns, `object` dtype for string columns etc.\n\n\nWe may be able to reduce the size of these columns in memory by downcasting numerical columns to smaller types (like `int8`, `int32`, `float32` etc.), if their maximum values don't need the larger types for storage, (like `int64`, `float64` etc.).\n\n\nSimilarly, Pandas automatically detects string columns as `object` datatype. To reduce memory usage of string columns which store categorical data, we specify their datatype as `category`.\n\n\nMany of the columns in this dataset can be downcast to smaller types.\n\nWe will provide a dict of `dtypes` for columns to pandas while reading the dataset.","metadata":{"id":"22DpLVFLn2O7"}},{"cell_type":"code","source":"# Reference: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/384359\ndtypes={\n    'elapsed_time':np.int32,\n    'event_name':'category',\n    'name':'category',\n    'level':np.uint8,\n    'room_coor_x':np.float32,\n    'room_coor_y':np.float32,\n    'screen_coor_x':np.float32,\n    'screen_coor_y':np.float32,\n    'hover_duration':np.float32,\n    'text':'category',\n    'fqid':'category',\n    'room_fqid':'category',\n    'text_fqid':'category',\n    'fullscreen':'category',\n    'hq':'category',\n    'music':'category',\n    'level_group':'category'}\n\ndataset_df = pd.read_csv('/kaggle/input/predict-student-performance-from-game-play/train.csv', dtype=dtypes)\nprint(\"Full train dataset shape is {}\".format(dataset_df.shape))","metadata":{"id":"_XItl24kn2O7","execution":{"iopub.status.busy":"2023-05-11T09:38:55.585883Z","iopub.execute_input":"2023-05-11T09:38:55.587291Z","iopub.status.idle":"2023-05-11T09:40:47.335729Z","shell.execute_reply.started":"2023-05-11T09:38:55.587226Z","shell.execute_reply":"2023-05-11T09:40:47.334608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The data is composed of 20 columns and 26296946 entries. We can see all 20 dimensions of our dataset by printing out the first 5 entries using the following code:","metadata":{"id":"kwJDjr_Rn2O8"}},{"cell_type":"code","source":"# Display the first 5 examples\ndataset_df.head(5)","metadata":{"id":"-RTRVRiWn2O8","execution":{"iopub.status.busy":"2023-05-11T09:41:17.541346Z","iopub.execute_input":"2023-05-11T09:41:17.541833Z","iopub.status.idle":"2023-05-11T09:41:17.585985Z","shell.execute_reply.started":"2023-05-11T09:41:17.541792Z","shell.execute_reply":"2023-05-11T09:41:17.584687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Please note that `session_id` uniquely identifies a user session.","metadata":{"id":"ZRZu8V9NpTVN"}},{"cell_type":"markdown","source":"# Load the labels\n\nThe labels for the training dataset are stored in the `train_labels.csv`. It consists of the information on whether the user in a particular session answered each question correctly. Load the labels data by running the following code. `","metadata":{"id":"ReNY-i3bn2O8"}},{"cell_type":"code","source":"labels = pd.read_csv('/kaggle/input/predict-student-performance-from-game-play/train_labels.csv')","metadata":{"id":"KD4uayl2n2O9","execution":{"iopub.status.busy":"2023-05-11T09:47:05.398986Z","iopub.execute_input":"2023-05-11T09:47:05.399446Z","iopub.status.idle":"2023-05-11T09:47:05.694216Z","shell.execute_reply.started":"2023-05-11T09:47:05.399406Z","shell.execute_reply":"2023-05-11T09:47:05.693058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each value in the column, `session_id` is a combination of both the session and the question number. \nWe will split these into individual columns for ease of use.","metadata":{"id":"SMKh2KAPn2O9"}},{"cell_type":"code","source":"labels['session'] = labels.session_id.apply(lambda x: int(x.split('_')[0]) )\nlabels['q'] = labels.session_id.apply(lambda x: int(x.split('_')[-1][1:]) )","metadata":{"id":"Kva8_Dbqn2O9","execution":{"iopub.status.busy":"2023-05-11T09:47:28.982673Z","iopub.execute_input":"2023-05-11T09:47:28.983172Z","iopub.status.idle":"2023-05-11T09:47:29.735152Z","shell.execute_reply.started":"2023-05-11T09:47:28.983128Z","shell.execute_reply":"2023-05-11T09:47:29.734006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Let us take a look at the first 5 entries of `labels` using the following code:","metadata":{"id":"0EQFN3vin2O9"}},{"cell_type":"code","source":"# Display the first 5 examples\nlabels.head(5)","metadata":{"id":"0eD-KZMvn2O-","execution":{"iopub.status.busy":"2023-05-11T09:56:03.384543Z","iopub.execute_input":"2023-05-11T09:56:03.385696Z","iopub.status.idle":"2023-05-11T09:56:03.403121Z","shell.execute_reply.started":"2023-05-11T09:56:03.385636Z","shell.execute_reply":"2023-05-11T09:56:03.402146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our goal is to train models for each question to predict the label `correct` for any input user session. ","metadata":{"id":"7rQSOYAYqcZ2"}},{"cell_type":"markdown","source":"# Bar chart for label column: correct","metadata":{"id":"JLHAQ1EEn2O-"}},{"cell_type":"markdown","source":"First we will plot a bar chart for the values of the label `correct`.","metadata":{"id":"a-JzBdfen2O-"}},{"cell_type":"code","source":"plt.figure(figsize=(3, 3))\nplot_df = labels.correct.value_counts()\nplot_df.plot(kind=\"bar\", color=['b', 'c'])","metadata":{"id":"-l9wBCTYn2O_","execution":{"iopub.status.busy":"2023-04-11T14:22:32.873134Z","iopub.execute_input":"2023-04-11T14:22:32.873593Z","iopub.status.idle":"2023-04-11T14:22:33.131746Z","shell.execute_reply.started":"2023-04-11T14:22:32.873554Z","shell.execute_reply":"2023-04-11T14:22:33.130711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let us plot the values of the label column `correct` for each question.","metadata":{"id":"ghnQfNQin2O_"}},{"cell_type":"code","source":"plt.figure(figsize=(10, 20))\nplt.subplots_adjust(hspace=0.5, wspace=0.5)\nplt.suptitle(\"\\\"Correct\\\" column values for each question\", fontsize=14, y=0.94)\nfor n in range(1,19):\n    #print(n, str(n))\n    ax = plt.subplot(6, 3, n)\n\n    # filter df and plot ticker on the new subplot axis\n    plot_df = labels.loc[labels.q == n]\n    plot_df = plot_df.correct.value_counts()\n    plot_df.plot(ax=ax, kind=\"bar\", color=['b', 'c'])\n    \n    # chart formatting\n    ax.set_title(\"Question \" + str(n))\n    ax.set_xlabel(\"\")\n","metadata":{"id":"X238--97n2O_","execution":{"iopub.status.busy":"2023-04-11T14:22:48.162389Z","iopub.execute_input":"2023-04-11T14:22:48.162783Z","iopub.status.idle":"2023-04-11T14:22:50.42859Z","shell.execute_reply.started":"2023-04-11T14:22:48.16275Z","shell.execute_reply":"2023-04-11T14:22:50.427577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare the dataset\n\nAs summarized in the competition overview, the dataset presents the questions and data to us in order of `levels - level segments`(represented by column `level_group`) 0-4, 5-12, and 13-22. We have to predict the correctness of each segment's questions as they are presented. To do this we will create basic aggregate features from the relevant columns. You can create more features to boost your scores. \n\nFirst, we will create two separate lists with names of the Categorical columns and Numerical columns. We will avoid columns `fullscreen`, `hq` and `music` since they don't add any useful value for this problem statement.","metadata":{"id":"y5fK05dsn2O_"}},{"cell_type":"code","source":"CATEGORICAL = ['event_name', 'name','fqid', 'room_fqid', 'text_fqid']\nNUMERICAL = ['elapsed_time','level','page','room_coor_x', 'room_coor_y', \n        'screen_coor_x', 'screen_coor_y', 'hover_duration']","metadata":{"id":"cCZWGiL_n2PA","execution":{"iopub.status.busy":"2023-04-11T14:23:53.357578Z","iopub.execute_input":"2023-04-11T14:23:53.358071Z","iopub.status.idle":"2023-04-11T14:23:53.363563Z","shell.execute_reply.started":"2023-04-11T14:23:53.358029Z","shell.execute_reply":"2023-04-11T14:23:53.362389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each categorical column, we will first group the dataset by `session_id`  and `level_group`. We will then count the number of **distinct elements** in the column for each group and store it temporarily.\n\nFor all numerical columns, we will group the dataset by `session id` and `level_group`. Instead of counting the number of distinct elements, we will calculate the `mean` and `standard deviation` of the numerical column for each group and store it temporarily.\n\nAfter this, we will concatenate the temporary data frames we generated in the earlier step for each column to create our new feature engineered dataset.","metadata":{"id":"Us9sScDSn2PA"}},{"cell_type":"code","source":"# Reference: https://www.kaggle.com/code/cdeotte/random-forest-baseline-0-664/notebook\n\ndef feature_engineer(dataset_df):\n    dfs = []\n    for c in CATEGORICAL:\n        tmp = dataset_df.groupby(['session_id','level_group'])[c].agg('nunique')\n        tmp.name = tmp.name + '_nunique'\n        dfs.append(tmp)\n    for c in NUMERICAL:\n        tmp = dataset_df.groupby(['session_id','level_group'])[c].agg('mean')\n        dfs.append(tmp)\n    for c in NUMERICAL:\n        tmp = dataset_df.groupby(['session_id','level_group'])[c].agg('std')\n        tmp.name = tmp.name + '_std'\n        dfs.append(tmp)\n    dataset_df = pd.concat(dfs,axis=1)\n    dataset_df = dataset_df.fillna(-1)\n    dataset_df = dataset_df.reset_index()\n    dataset_df = dataset_df.set_index('session_id')\n    return dataset_df","metadata":{"id":"nHWhAOtTn2PA","execution":{"iopub.status.busy":"2023-04-11T14:24:34.807357Z","iopub.execute_input":"2023-04-11T14:24:34.807934Z","iopub.status.idle":"2023-04-11T14:24:34.819191Z","shell.execute_reply.started":"2023-04-11T14:24:34.807863Z","shell.execute_reply":"2023-04-11T14:24:34.818058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_df = feature_engineer(dataset_df)\nprint(\"Full prepared dataset shape is {}\".format(dataset_df.shape))","metadata":{"id":"JKcoPoemn2PA","execution":{"iopub.status.busy":"2023-04-11T14:28:01.122096Z","iopub.execute_input":"2023-04-11T14:28:01.122665Z","iopub.status.idle":"2023-04-11T14:28:41.731564Z","shell.execute_reply.started":"2023-04-11T14:28:01.122597Z","shell.execute_reply":"2023-04-11T14:28:41.730639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Our feature engineered dataset is composed of 22 columns and 70686 entries. ","metadata":{"id":"891j7nc1n2PA"}},{"cell_type":"markdown","source":"# Basic exploration of the prepared dataset","metadata":{"id":"Ij7TT3x-n2PB"}},{"cell_type":"markdown","source":"Let us print out the first 5 entries using the following code:","metadata":{"id":"s1c59fMAn2PB"}},{"cell_type":"code","source":"# Display the first 5 examples\ndataset_df.head(5)","metadata":{"id":"mvQEsdV1n2PB","execution":{"iopub.status.busy":"2023-04-11T14:36:28.977997Z","iopub.execute_input":"2023-04-11T14:36:28.978475Z","iopub.status.idle":"2023-04-11T14:36:29.013197Z","shell.execute_reply.started":"2023-04-11T14:36:28.978436Z","shell.execute_reply":"2023-04-11T14:36:29.012009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_df.describe()","metadata":{"id":"DRusg-N1n2PB","execution":{"iopub.status.busy":"2023-04-11T14:36:50.99053Z","iopub.execute_input":"2023-04-11T14:36:50.991295Z","iopub.status.idle":"2023-04-11T14:36:51.133687Z","shell.execute_reply.started":"2023-04-11T14:36:50.991246Z","shell.execute_reply":"2023-04-11T14:36:51.132427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Numerical data distribution¶\n\nLet us plot some numerical columns and their value against each level_group:","metadata":{"id":"xboIq9oDn2PB"}},{"cell_type":"code","source":"figure, axis = plt.subplots(3, 2, figsize=(10, 10))\n\nfor name, data in dataset_df.groupby('level_group'):\n    axis[0, 0].plot(range(1, len(data['room_coor_x_std'])+1), data['room_coor_x_std'], label=name)\n    axis[0, 1].plot(range(1, len(data['room_coor_y_std'])+1), data['room_coor_y_std'], label=name)\n    axis[1, 0].plot(range(1, len(data['screen_coor_x_std'])+1), data['screen_coor_x_std'], label=name)\n    axis[1, 1].plot(range(1, len(data['screen_coor_y_std'])+1), data['screen_coor_y_std'], label=name)\n    axis[2, 0].plot(range(1, len(data['hover_duration'])+1), data['hover_duration_std'], label=name)\n    axis[2, 1].plot(range(1, len(data['elapsed_time_std'])+1), data['elapsed_time_std'], label=name)\n    \n\naxis[0, 0].set_title('room_coor_x')\naxis[0, 1].set_title('room_coor_y')\naxis[1, 0].set_title('screen_coor_x')\naxis[1, 1].set_title('screen_coor_y')\naxis[2, 0].set_title('hover_duration')\naxis[2, 1].set_title('elapsed_time_std')\n\nfor i in range(3):\n    axis[i, 0].legend()\n    axis[i, 1].legend()\n\nplt.show()","metadata":{"id":"mXIiaq_bn2PC","execution":{"iopub.status.busy":"2023-04-11T14:37:01.584599Z","iopub.execute_input":"2023-04-11T14:37:01.58506Z","iopub.status.idle":"2023-04-11T14:37:04.219929Z","shell.execute_reply.started":"2023-04-11T14:37:01.585021Z","shell.execute_reply":"2023-04-11T14:37:04.218926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us split the dataset into training and testing datasets:","metadata":{"id":"W0ex_Jkln2PC"}},{"cell_type":"code","source":"def split_dataset(dataset, test_ratio=0.20):\n    USER_LIST = dataset_df.index.unique()\n    split = int(len(USER_LIST) * (1 - 0.20))\n    return dataset.loc[USER_LIST[:split]], dataset.loc[USER_LIST[split:]]\n\ntrain_x, valid_x = split_dataset(dataset_df)\nprint(\"{} examples in training, {} examples in testing.\".format(\n    len(train_x), len(valid_x)))","metadata":{"id":"OZfTcCJfn2PC","execution":{"iopub.status.busy":"2023-04-11T14:37:26.700647Z","iopub.execute_input":"2023-04-11T14:37:26.701142Z","iopub.status.idle":"2023-04-11T14:37:26.776067Z","shell.execute_reply.started":"2023-04-11T14:37:26.701099Z","shell.execute_reply":"2023-04-11T14:37:26.77508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Select a Model\nThere are several tree-based models for you to choose from.\n\n- RandomForestModel\n- GradientBoostedTreesModel\n- CartModel\n- DistributedGradientBoostedTreesModel\n\nWe can list all the available models in TensorFlow Decision Forests using the following code:","metadata":{"id":"ZnVfKZfzn2PE"}},{"cell_type":"code","source":"tfdf.keras.get_all_models()","metadata":{"id":"KZBdcVU1n2PE","execution":{"iopub.status.busy":"2023-04-11T14:39:52.752222Z","iopub.execute_input":"2023-04-11T14:39:52.752683Z","iopub.status.idle":"2023-04-11T14:39:52.759995Z","shell.execute_reply.started":"2023-04-11T14:39:52.752645Z","shell.execute_reply":"2023-04-11T14:39:52.758877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To get started, we'll work with a Gradient Boosted Trees Model. This is one of the well-known Decision Forest training algorithms.\n\nA Gradient Boosted Decision Tree is a set of shallow decision trees trained sequentially. Each tree is trained to predict and then \"correct\" for the errors of the previously trained trees.","metadata":{"id":"cPJ0fJtsWEKI"}},{"cell_type":"markdown","source":"# How can I configure a tree-based model?\n\nTensorFlow Decision Forests provides good defaults for you (e.g., the top ranking hyperparameters on our benchmarks, slightly modified to run in reasonable time). If you would like to configure the learning algorithm, you will find many options you can explore to get the highest possible accuracy.\n\nYou can select a template and/or set parameters as follows:\n```\nrf = tfdf.keras.GradientBoostedTreesModel(hyperparameter_template=\"benchmark_rank1\")\n```\n\nYou can read more [here](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/GradientBoostedTreesModel).","metadata":{"id":"TbIeh3-En2PE"}},{"cell_type":"markdown","source":"# Training","metadata":{"id":"UdibIrM-XP5-"}},{"cell_type":"markdown","source":"\nWe will train a model for each question to predict if the question will be answered correctly by a user. \nThere are a total of 18 questions in the dataset. Hence, we will be training 18 models, one for each question.\n\nWe need to provide a few data structures to our training loop to store the trained models, predictions on the validation set and evaluation scores for the trained models.\n\nWe will create these using the following code:\n","metadata":{"id":"BtkKMa7sXd65"}},{"cell_type":"code","source":"# Fetch the unique list of user sessions in the validation dataset. We assigned \n# `session_id` as the index of our feature engineered dataset. Hence fetching \n# the unique values in the index column will give us a list of users in the \n# validation set.\nVALID_USER_LIST = valid_x.index.unique()\n\n# Create a dataframe for storing the predictions of each question for all users\n# in the validation set.\n# For this, the required size of the data frame is: \n# (no: of users in validation set  x no of questions).\n# We will initialize all the predicted values in the data frame to zero.\n# The dataframe's index column is the user `session_id`s. \nprediction_df = pd.DataFrame(data=np.zeros((len(VALID_USER_LIST),18)), index=VALID_USER_LIST)\n\n# Create an empty dictionary to store the models created for each question.\nmodels = {}\n\n# Create an empty dictionary to store the evaluation score for each question.\nevaluation_dict ={}","metadata":{"id":"7Brds67Wn2PD","execution":{"iopub.status.busy":"2023-04-11T14:42:04.472171Z","iopub.execute_input":"2023-04-11T14:42:04.472577Z","iopub.status.idle":"2023-04-11T14:42:04.481095Z","shell.execute_reply.started":"2023-04-11T14:42:04.472544Z","shell.execute_reply":"2023-04-11T14:42:04.479771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before training the data we have to understand how `level_groups` and `questions` are associated to each other.\n\nIn this game the first quiz checkpoint(i.e., questions 1 to 3) comes after finishing levels 0 to 4. So for training questions 1 to 3 we will use data from the `level_group` 0-4. Similarly, we will use data from the `level_group` 5-12 to train questions from 4 to 13 and data from the `level_group` 13-22 to train questions from 14 to 18.\n\nWe will train a model for each question and store the trained model in the `models` dict.","metadata":{"id":"p1GoOMRHn2PF"}},{"cell_type":"code","source":"# Iterate through questions 1 to 18 to train models for each question, evaluate\n# the trained model and store the predicted values.\nfor q_no in range(1,19):\n\n    # Select level group for the question based on the q_no.\n    if q_no<=3: grp = '0-4'\n    elif q_no<=13: grp = '5-12'\n    elif q_no<=22: grp = '13-22'\n    print(\"### q_no\", q_no, \"grp\", grp)\n    \n        \n    # Filter the rows in the datasets based on the selected level group. \n    train_df = train_x.loc[train_x.level_group == grp]\n    train_users = train_df.index.values\n    valid_df = valid_x.loc[valid_x.level_group == grp]\n    valid_users = valid_df.index.values\n\n    # Select the labels for the related q_no.\n    train_labels = labels.loc[labels.q==q_no].set_index('session').loc[train_users]\n    valid_labels = labels.loc[labels.q==q_no].set_index('session').loc[valid_users]\n\n    # Add the label to the filtered datasets.\n    train_df[\"correct\"] = train_labels[\"correct\"]\n    valid_df[\"correct\"] = valid_labels[\"correct\"]\n\n    # There's one more step required before we can train the model. \n    # We need to convert the datatset from Pandas format (pd.DataFrame)\n    # into TensorFlow Datasets format (tf.data.Dataset).\n    # TensorFlow Datasets is a high performance data loading library \n    # which is helpful when training neural networks with accelerators like GPUs and TPUs.\n    # We are omitting `level_group`, since it is not needed for training anymore.\n    train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_df.loc[:, train_df.columns != 'level_group'], label=\"correct\")\n    valid_ds = tfdf.keras.pd_dataframe_to_tf_dataset(valid_df.loc[:, valid_df.columns != 'level_group'], label=\"correct\")\n\n    # We will now create the Gradient Boosted Trees Model with default settings. \n    # By default the model is set to train for a classification task.\n    gbtm = tfdf.keras.GradientBoostedTreesModel(verbose=0)\n    gbtm.compile(metrics=[\"accuracy\"])\n\n    # Train the model.\n    gbtm.fit(x=train_ds)\n\n    # Store the model\n    models[f'{grp}_{q_no}'] = gbtm\n\n    # Evaluate the trained model on the validation dataset and store the \n    # evaluation accuracy in the `evaluation_dict`.\n    inspector = gbtm.make_inspector()\n    inspector.evaluation()\n    evaluation = gbtm.evaluate(x=valid_ds,return_dict=True)\n    evaluation_dict[q_no] = evaluation[\"accuracy\"]         \n\n    # Use the trained model to make predictions on the validation dataset and \n    # store the predicted values in the `prediction_df` dataframe.\n    predict = gbtm.predict(x=valid_ds)\n    prediction_df.loc[valid_users, q_no-1] = predict.flatten()     ","metadata":{"id":"VBO3VCOJn2PF","execution":{"iopub.status.busy":"2023-04-11T14:47:02.72488Z","iopub.execute_input":"2023-04-11T14:47:02.725339Z","iopub.status.idle":"2023-04-11T14:49:42.576258Z","shell.execute_reply.started":"2023-04-11T14:47:02.725303Z","shell.execute_reply":"2023-04-11T14:49:42.574741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inspect the Accuracy of the models.\n\nWe trained a model for each question. Now let us check the accuracy of each model and overall accuracy for all the models combined. \n\nNote: Since the label distribution is imbalanced, we can't make an assumption on the model performance from accuracy score alone. ","metadata":{"id":"oOdr1p-Bn2PF"}},{"cell_type":"code","source":"for name, value in evaluation_dict.items():\n  print(f\"question {name}: accuracy {value:.4f}\")\n\nprint(\"\\nAverage accuracy\", sum(evaluation_dict.values())/18)","metadata":{"id":"qPOfPkm7n2PG","execution":{"iopub.status.busy":"2023-04-11T14:52:34.745474Z","iopub.execute_input":"2023-04-11T14:52:34.745979Z","iopub.status.idle":"2023-04-11T14:52:34.753017Z","shell.execute_reply.started":"2023-04-11T14:52:34.745937Z","shell.execute_reply":"2023-04-11T14:52:34.751991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize the model\n\nOne benefit of tree-based models is that we can easily visualize them. The default number of trees used in the Random Forests is 300. \n\nLet us pick one model from `models` dict and select a tree to display below.","metadata":{"id":"qtiCtfNCn2PG"}},{"cell_type":"code","source":"tfdf.model_plotter.plot_model_in_colab(models['0-4_1'], tree_idx=0, max_depth=3)","metadata":{"id":"od__6uAan2PG","execution":{"iopub.status.busy":"2023-04-11T14:52:43.602162Z","iopub.execute_input":"2023-04-11T14:52:43.602581Z","iopub.status.idle":"2023-04-11T14:52:43.618124Z","shell.execute_reply.started":"2023-04-11T14:52:43.602545Z","shell.execute_reply":"2023-04-11T14:52:43.61674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Variable importances\n\nVariable importances generally indicate how much a feature contributes to the model predictions or quality. There are several ways to identify important features using TensorFlow Decision Forests. Let us pick one model from models dict and inspect it.\n\nLet us list the available Variable Importances for Decision Trees:","metadata":{"id":"emVV6509n2PG"}},{"cell_type":"code","source":"inspector = models['0-4_1'].make_inspector()\n\nprint(f\"Available variable importances:\")\nfor importance in inspector.variable_importances().keys():\n  print(\"\\t\", importance)","metadata":{"id":"d_gvL9nbn2PH","execution":{"iopub.status.busy":"2023-04-11T14:53:02.386273Z","iopub.execute_input":"2023-04-11T14:53:02.386699Z","iopub.status.idle":"2023-04-11T14:53:02.395883Z","shell.execute_reply.started":"2023-04-11T14:53:02.386662Z","shell.execute_reply":"2023-04-11T14:53:02.394846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As an example, let us display the important features for the Variable Importance NUM_AS_ROOT.\n\nThe larger the importance score for NUM_AS_ROOT, the more impact it has on the outcome of the model for Question 1(i.e., model\\[\"0-4_1\"\\]).\n\nBy default, the list is sorted from the most important to the least. From the output you can infer that the feature at the top of the list is used as the root node in most number of trees in the gradient boosted trees  than any other feature.","metadata":{"id":"rnfYz_aGn2PH"}},{"cell_type":"code","source":"# Each line is: (feature name, (index of the feature), importance score)\ninspector.variable_importances()[\"NUM_AS_ROOT\"]","metadata":{"id":"ZDsxqRrwn2PH","execution":{"iopub.status.busy":"2023-04-11T14:53:46.07308Z","iopub.execute_input":"2023-04-11T14:53:46.073505Z","iopub.status.idle":"2023-04-11T14:53:46.082043Z","shell.execute_reply.started":"2023-04-11T14:53:46.073468Z","shell.execute_reply":"2023-04-11T14:53:46.080811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Threshold-Moving for Imbalanced Classification\n\nSince the values of the column `correct` is fairly imbalanced, using the default threshold of `0.5` to map the predictions into classes 0 or 1 can result in poor performance. \nIn such cases, to improve performance we will calculate the `F1 score` for a certain range of thresholds and try to find the best threshold aka, threshold with highest `F1 score`. Then we will use this threshold to map the predicted probabilities to class labels 0 or 1.\n\nPlease note that we are using `F1 score` since it is a better metric than `accuracy` to evaluate problems with class imbalance.","metadata":{"id":"PIK5aUH-n2PH"}},{"cell_type":"code","source":"# Create a dataframe of required size:\n# (no: of users in validation set x no: of questions) initialized to zero values\n# to store true values of the label `correct`. \ntrue_df = pd.DataFrame(data=np.zeros((len(VALID_USER_LIST),18)), index=VALID_USER_LIST)\nfor i in range(18):\n    # Get the true labels.\n    tmp = labels.loc[labels.q == i+1].set_index('session').loc[VALID_USER_LIST]\n    true_df[i] = tmp.correct.values\n\nmax_score = 0; best_threshold = 0\n\n# Loop through threshold values from 0.4 to 0.8 and select the threshold with \n# the highest `F1 score`.\nfor threshold in np.arange(0.4,0.8,0.01):\n    metric = tfa.metrics.F1Score(num_classes=2,average=\"macro\",threshold=threshold)\n    y_true = tf.one_hot(true_df.values.reshape((-1)), depth=2)\n    y_pred = tf.one_hot((prediction_df.values.reshape((-1))>threshold).astype('int'), depth=2)\n    metric.update_state(y_true, y_pred)\n    f1_score = metric.result().numpy()\n    if f1_score > max_score:\n        max_score = f1_score\n        best_threshold = threshold\n        \nprint(\"Best threshold \", best_threshold, \"\\tF1 score \", max_score)","metadata":{"id":"2wptRs3In2PH","execution":{"iopub.status.busy":"2023-04-11T15:03:33.348399Z","iopub.execute_input":"2023-04-11T15:03:33.348866Z","iopub.status.idle":"2023-04-11T15:03:34.525896Z","shell.execute_reply.started":"2023-04-11T15:03:33.348829Z","shell.execute_reply":"2023-04-11T15:03:34.524896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission\n\nHere you'll use the `best_threshold` calculate in the previous cell","metadata":{"id":"ezA40GQ4n2PH"}},{"cell_type":"code","source":"# Reference\n# https://www.kaggle.com/code/philculliton/basic-submission-demo\n# https://www.kaggle.com/code/cdeotte/random-forest-baseline-0-664/notebook\n\n\nimport jo_wilder\nenv = jo_wilder.make_env()\niter_test = env.iter_test()\n\nlimits = {'0-4':(1,4), '5-12':(4,14), '13-22':(14,19)}\n\nfor (test, sample_submission) in iter_test:\n    test_df = feature_engineer(test)\n    grp = test_df.level_group.values[0]\n    a,b = limits[grp]\n    for t in range(a,b):\n        gbtm = models[f'{grp}_{t}']\n        test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(test_df.loc[:, test_df.columns != 'level_group'])\n        predictions = gbtm.predict(test_ds)\n        mask = sample_submission.session_id.str.contains(f'q{t}')\n        n_predictions = (predictions > best_threshold).astype(int)\n        sample_submission.loc[mask,'correct'] = n_predictions.flatten()\n    \n    env.predict(sample_submission)","metadata":{"id":"gHiXTnTVn2PI","execution":{"iopub.status.busy":"2023-04-11T15:07:46.992778Z","iopub.execute_input":"2023-04-11T15:07:46.993288Z","iopub.status.idle":"2023-04-11T15:07:48.902986Z","shell.execute_reply.started":"2023-04-11T15:07:46.993246Z","shell.execute_reply":"2023-04-11T15:07:48.901928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! head submission.csv","metadata":{"id":"iYBXokAyn2PI","execution":{"iopub.status.busy":"2023-04-11T15:08:02.43839Z","iopub.execute_input":"2023-04-11T15:08:02.438826Z","iopub.status.idle":"2023-04-11T15:08:03.619129Z","shell.execute_reply.started":"2023-04-11T15:08:02.438789Z","shell.execute_reply":"2023-04-11T15:08:03.617743Z"},"trusted":true},"execution_count":null,"outputs":[]}]}