{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**The current version is suffering issues during training (high/erratic validation loss). That is why there are only 10 epochs. These issues will be fixed when I have the time. Any help on this is appreciated. I know the issues are fixable by improving/tuning the model.**","metadata":{}},{"cell_type":"markdown","source":"First of all, most of the logic and ideas have been taken from @paddykb notebook. However, I have never used FastAI before. After using it and modifying the code from @paddykb I actually really liked the results which got me first place for sometime. As a next step I wanted to port the FastAI code to Tensorflow/Keras as that's what I prefer using usually. Unfortunately, it makes improving and tuning the model **way** harder than using FastAI - tho it helps understand what is happening and thinking about improvements yourself. The low-level approach using keras can most likely give even better results than the FastAI method but requires **a lot** of work to tune. As I am time constraint I **did not** tune the model but feel free to copy my notebook and work on tuning/extending the model.\n\nI also hope that some people who have never used keras can learn a few things - especially how to implement and use a custom data generator which we are using for augmentation. ","metadata":{}},{"cell_type":"code","source":"import random\nimport numpy as np\nimport pandas as pd\nimport gc\nimport tensorflow as tf\nfrom keras.layers import Input, Dense, Dropout, BatchNormalization\nfrom keras.models import Model\nimport keras\nfrom sklearn.utils import shuffle\nfrom sklearn.model_selection import train_test_split\nimport tensorflow_addons as tfa","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:45.658859Z","iopub.execute_input":"2022-10-17T01:14:45.659425Z","iopub.status.idle":"2022-10-17T01:14:45.664978Z","shell.execute_reply.started":"2022-10-17T01:14:45.659390Z","shell.execute_reply":"2022-10-17T01:14:45.663971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we are defining a few arrays that will come in helpful later.","metadata":{}},{"cell_type":"code","source":"features = [\n    'ball_pos_x', 'ball_pos_y','ball_pos_z', 'ball_vel_x', 'ball_vel_y', 'ball_vel_z', \n    'p0_pos_x', 'p0_pos_y', 'p0_pos_z', 'p0_vel_x', 'p0_vel_y', 'p0_vel_z', 'p0_boost', 'p0_na',\n    'p1_pos_x', 'p1_pos_y', 'p1_pos_z', 'p1_vel_x', 'p1_vel_y', 'p1_vel_z', 'p1_boost', 'p1_na',\n    'p2_pos_x', 'p2_pos_y', 'p2_pos_z', 'p2_vel_x', 'p2_vel_y', 'p2_vel_z', 'p2_boost', 'p2_na',\n    'p3_pos_x', 'p3_pos_y', 'p3_pos_z', 'p3_vel_x', 'p3_vel_y', 'p3_vel_z', 'p3_boost', 'p3_na',\n    'p4_pos_x', 'p4_pos_y', 'p4_pos_z', 'p4_vel_x', 'p4_vel_y', 'p4_vel_z', 'p4_boost', 'p4_na',\n    'p5_pos_x', 'p5_pos_y', 'p5_pos_z', 'p5_vel_x', 'p5_vel_y', 'p5_vel_z', 'p5_boost', 'p5_na',\n    'boost0_timer', 'boost1_timer', \n    'boost2_timer', 'boost3_timer',\n    'boost4_timer', 'boost5_timer']\n\nfeatures_x_pos = [pos for pos, feature in enumerate(features) if feature.endswith('_x')]\nfeatures_y_pos = [pos for pos, feature in enumerate(features) if feature.endswith('_y')]\n\ntargets = [\n    'team_A_scoring_within_10sec',\n    'team_B_scoring_within_10sec']","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:45.667119Z","iopub.execute_input":"2022-10-17T01:14:45.667884Z","iopub.status.idle":"2022-10-17T01:14:45.693360Z","shell.execute_reply.started":"2022-10-17T01:14:45.667846Z","shell.execute_reply":"2022-10-17T01:14:45.692413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following functions are uesd to load the training and test data. We also normalize the positions so that our feature range is easier to process for the NN.","metadata":{}},{"cell_type":"code","source":"def fe(x):\n    # indicators for respawns...\n    x['p0_na'] = x['p0_pos_x'].isna().astype('int8')\n    x['p1_na'] = x['p1_pos_x'].isna().astype('int8')\n    x['p2_na'] = x['p2_pos_x'].isna().astype('int8')\n    x['p3_na'] = x['p3_pos_x'].isna().astype('int8')\n    x['p4_na'] = x['p4_pos_x'].isna().astype('int8')\n    x['p5_na'] = x['p5_pos_x'].isna().astype('int8')\n    for feature in features:\n        if feature.endswith('_na'):\n            continue\n        # this is just scaling the features to something reasonable\n        # it might make sense to apply a transformation to the z-dimension.\n        if feature.endswith('_x'):\n            x[feature] = (x[feature]/82).fillna(0).astype('float16')\n        if feature.endswith('_y'):\n            x[feature] = (x[feature]/120).fillna(0).astype('float16')\n        if feature.endswith('_z'):\n            x[feature] = (x[feature]/40).fillna(0).astype('float16')\n        if feature.endswith('_boost'):\n            x[feature] = (x[feature]/100).fillna(0).astype('float16')\n        if feature.endswith('_timer'):\n            x[feature] = (-x[feature]/100).astype('float16')\n    return x\n\ndef read_train(start,end):\n    dfs = []\n    for i in range(start,end):\n        print(f\"The progress is {i}\")\n        dfs.append(fe(pd.read_feather(f\"../input/tpsoct22-feather-files/train_{i}.feather\")))\n    result = pd.concat(dfs)\n    return result.sample(frac=1)\n\ndef read_test():\n    test = pd.read_csv(\"../input/tabular-playground-series-oct-2022/test.csv\")\n    test.drop(labels=['id'], axis=1, inplace=True)\n    return fe(test)\n\n","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:45.694757Z","iopub.execute_input":"2022-10-17T01:14:45.695204Z","iopub.status.idle":"2022-10-17T01:14:45.708509Z","shell.execute_reply.started":"2022-10-17T01:14:45.695169Z","shell.execute_reply":"2022-10-17T01:14:45.707506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following methods are called on each batch during the training/testing proccess of the NN to augment the data.","metadata":{}},{"cell_type":"code","source":"def fe_goal_distance(ball):\n    dist_a = ((ball[:,0:1] - 0) ** 2 + (ball[:,1:2] - 120) ** 2 + (ball[:,2:3] - 0) ** 2) ** 0.5 / 2\n    dist_b = ((ball[:,0:1] - 0) ** 2 + (ball[:,1:2] + 120) ** 2 + (ball[:,2:3] - 0) ** 2) ** 0.5 / 2\n    return dist_a, dist_b\n    \ndef fe_dist_ball_player(ball, player):\n    dist = ((\n        (ball[:,0:1] - player[:,0:1]) ** 2 + \n        (ball[:,1:2] - player[:,1:2]) ** 2 + \n        (ball[:,2:3] - player[:,2:3]) ** 2) ** 0.5) / 12\n    return dist\n\ndef fe_speed_of_thing(thing):\n    return ((thing[:, 3:4]) ** 2 + (thing[:, 4:5]) ** 2 + (thing[:, 5:6]) ** 2) ** 0.5\n\ndef angle_rowwise(A, B):\n    p1 = np.einsum('ij,ij->i',A,B)\n    p2 = np.linalg.norm(A,axis=1)\n    p3 = np.linalg.norm(B,axis=1)\n    p4 = p1 / (p2*p3+0.000001)\n    return np.arccos(np.clip(p4,-1.0,1.0))\n\ndef cos_angle_objs(ob1, ob2, ob3):\n    return ((angle_rowwise(ob1-ob2,ob1-ob3)*180/np.pi)/180).reshape(-1,1)\n\ndef augment_fe(X, Y):\n    ball = X[:, :6]\n    p0 = X[:,  6:14]\n    p1 = X[:, 14:22]\n    p2 = X[:, 22:30]\n    p3 = X[:, 30:38]\n    p4 = X[:, 38:46]\n    p5 = X[:, 46:54]\n    boosts = X[:, 54:]\n    \n    goal_a, goal_b = fe_goal_distance(ball)\n    \n    p0d = fe_dist_ball_player(ball, p0)\n    p1d = fe_dist_ball_player(ball, p1)\n    p2d = fe_dist_ball_player(ball, p2)\n    p3d = fe_dist_ball_player(ball, p3)\n    p4d = fe_dist_ball_player(ball, p4)\n    p5d = fe_dist_ball_player(ball, p5)\n    \n    ball_s = fe_speed_of_thing(ball)\n    p0s = fe_speed_of_thing(p0)\n    p1s = fe_speed_of_thing(p1)\n    p2s = fe_speed_of_thing(p2)\n    p3s = fe_speed_of_thing(p3)\n    p4s = fe_speed_of_thing(p4)\n    p5s = fe_speed_of_thing(p5)\n    \n    pos_goal1=np.zeros((len(ball),2))\n    pos_goal2=np.zeros((len(ball),2))\n    \n    pos_goal1[:,0]=0\n    pos_goal1[:,1]=120\n    \n    pos_goal2[:,0]=0\n    pos_goal2[:,1]=-120\n    \n    p0g1 = cos_angle_objs(ball[:,0:2],p0[:,0:2], pos_goal1[:,0:2])\n    p0g2 = cos_angle_objs(ball[:,0:2],p0[:,0:2], pos_goal2[:,0:2])\n    \n    p1g1 = cos_angle_objs(ball[:,0:2],p1[:,0:2], pos_goal1[:,0:2])\n    p1g2 = cos_angle_objs(ball[:,0:2],p1[:,0:2], pos_goal2[:,0:2])\n    \n    p2g1 = cos_angle_objs(ball[:,0:2],p2[:,0:2], pos_goal1[:,0:2])\n    p2g2 = cos_angle_objs(ball[:,0:2],p2[:,0:2], pos_goal2[:,0:2])\n    \n    p3g1 = cos_angle_objs(ball[:,0:2],p3[:,0:2], pos_goal1[:,0:2])\n    p3g2 = cos_angle_objs(ball[:,0:2],p3[:,0:2], pos_goal2[:,0:2])\n    \n    p4g1 = cos_angle_objs(ball[:,0:2],p4[:,0:2], pos_goal1[:,0:2])\n    p4g2 = cos_angle_objs(ball[:,0:2],p4[:,0:2], pos_goal2[:,0:2])\n    \n    p5g1 = cos_angle_objs(ball[:,0:2],p5[:,0:2], pos_goal1[:,0:2])\n    p5g2 = cos_angle_objs(ball[:,0:2],p5[:,0:2], pos_goal2[:,0:2])\n    \n    new_X = np.concatenate([\n        ball, p0, p1, p2, p3, p4, p5, boosts,\n        goal_a, goal_b,\n        ball_s,\n        p0d, p1d, p2d, p3d, p4d, p5d,\n        p0s, p1s, p2s, p3s, p4s, p5s,\n        p0g1, p0g2,p1g1,p1g2, p2g1,p2g2,p3g1,p3g2,p4g1,p4g2,p5g1,p5g2\n    ], axis=1)\n    \n    return new_X, Y\n\ndef augment_shuffle(X, Y):\n    ball = X[:, :6]\n    p0 = X[:,  6:14]\n    p1 = X[:, 14:22]\n    p2 = X[:, 22:30]\n    p3 = X[:, 30:38]\n    p4 = X[:, 38:46]\n    p5 = X[:, 46:54]\n    boosts = X[:, 54:]\n    \n    pA = np.concatenate(random.sample([p0, p1, p2], 3), axis=1)\n    pB = np.concatenate(random.sample([p3, p4, p5], 3), axis=1)\n    \n    shuffled_X = np.concatenate([ball, pA, pB, boosts], axis=1)\n\n    return shuffled_X, Y\n\ndef augment_flip_x(X, Y):\n    positions = X[:,:54]\n    positions[:, features_x_pos] = -positions[:, features_x_pos]\n    boosts = X[:, [55, 54, 57, 56, 59, 58]]\n    \n    flip_X = np.concatenate([positions, boosts], axis=1)\n    \n    return flip_X, Y\n\ndef augment_mirror(X, Y):\n    positions = X[:,:54]\n    positions[:, features_x_pos] = -positions[:, features_x_pos]\n    positions[:, features_y_pos] = -positions[:, features_y_pos]\n    \n    ball = positions[:, :6]\n    p0 = positions[:,  6:14]\n    p1 = positions[:, 14:22]\n    p2 = positions[:, 22:30]\n    p3 = positions[:, 30:38]\n    p4 = positions[:, 38:46]\n    p5 = positions[:, 46:54]\n    \n    players = np.concatenate([p3, p4, p5, p0, p1, p2], axis=1)\n    boosts = X[:, [59, 58, 57, 56, 55, 54]]\n    \n    flip_X = np.concatenate([ball, players, boosts], axis=1)\n    flip_Y = Y[:,[1,0]]\n    \n    return flip_X, flip_Y","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:45.710937Z","iopub.execute_input":"2022-10-17T01:14:45.711725Z","iopub.status.idle":"2022-10-17T01:14:45.740785Z","shell.execute_reply.started":"2022-10-17T01:14:45.711687Z","shell.execute_reply":"2022-10-17T01:14:45.739904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we are creating a new Custom Data Generator that has the main purpose of creating and returning a new batch. Depending on the settings, it will augment and shuffle the batch using the previously defined functions.","metadata":{}},{"cell_type":"code","source":"class CustomDataGen(tf.keras.utils.Sequence):\n    \n    def __init__(self, df, y,\n                 batch_size,\n                 input_size=(60,),\n                 augment=True, testing = False):\n        \n        self.X = df.copy()\n        self.y = y.copy()\n        self.batch_size = batch_size\n        self.input_size = input_size\n        self.augment = augment\n        self.testing = testing\n\n        self.n = len(self.X)\n    \n    def __get_data(self, x_batch, y_batch):\n        # Generates data containing batch_size samples\n        if self.augment == True:\n            if random.random() < 0.5:\n                x_batch, y_batch = augment_shuffle(x_batch, y_batch)\n            if random.random() < 0.5:\n                x_batch, y_batch = augment_flip_x(x_batch, y_batch)\n            if random.random() < 0.5 and self.testing == False:\n                x_batch, y_batch = augment_mirror(x_batch, y_batch)\n            if self.testing == False:\n                x_batch, y_batch = shuffle(x_batch, y_batch)\n            x_batch,y_batch = augment_fe(x_batch,y_batch)\n            return x_batch,y_batch\n        else:\n            return augment_fe(x_batch,y_batch)\n    \n    def __getitem__(self, index):\n        end = (index + 1) * self.batch_size\n        x_batch = self.X[index * self.batch_size:end]\n        y_batch = self.y[index * self.batch_size:end]\n        X, y = self.__get_data(x_batch,y_batch)        \n        return X, y\n    \n    def __len__(self):\n        return self.n // self.batch_size","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:45.742200Z","iopub.execute_input":"2022-10-17T01:14:45.742535Z","iopub.status.idle":"2022-10-17T01:14:45.755970Z","shell.execute_reply.started":"2022-10-17T01:14:45.742501Z","shell.execute_reply":"2022-10-17T01:14:45.755001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we create our tensorflow model using the functional API.","metadata":{}},{"cell_type":"code","source":"inp = Input(shape=(87,))\n\nx = BatchNormalization(axis=-1,momentum=0.1,epsilon=1e-5)(inp)\n\nx = Dense(512,use_bias=False)(x)\nx = tfa.activations.mish(x)\nx = BatchNormalization(axis=-1,momentum=0.1,epsilon=1e-5)(x)\nx = Dropout(0.3)(x)\n\nx = Dense(256,use_bias=False)(x)\nx = tfa.activations.mish(x)\nx = BatchNormalization(axis=-1,momentum=0.1,epsilon=1e-5)(x)\nx = Dropout(0.3)(x)\n\nx = Dense(64, use_bias=False)(x)\nx = tfa.activations.mish(x)\n\nx = Dense(2)(x)\n\nout = tf.keras.activations.sigmoid(x)\n\nmodel = Model(inputs= inp, outputs=out)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:45.760757Z","iopub.execute_input":"2022-10-17T01:14:45.762019Z","iopub.status.idle":"2022-10-17T01:14:48.689928Z","shell.execute_reply.started":"2022-10-17T01:14:45.761984Z","shell.execute_reply":"2022-10-17T01:14:48.688883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We compile the model using an opitimizer and define the loss we want to track as well as any other metrics.","metadata":{}},{"cell_type":"code","source":"opt = tf.optimizers.Adam(learning_rate=0.001)\nmodel.compile(optimizer=\"adam\",loss = 'binary_crossentropy', metrics = ['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:48.694902Z","iopub.execute_input":"2022-10-17T01:14:48.697700Z","iopub.status.idle":"2022-10-17T01:14:48.717246Z","shell.execute_reply.started":"2022-10-17T01:14:48.697657Z","shell.execute_reply":"2022-10-17T01:14:48.716222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we define some callbacks wich will be activated during the training depeding on their purpose! ","metadata":{}},{"cell_type":"code","source":"earlyStopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, verbose=1, mode='min')\nmcp_save = tf.keras.callbacks.ModelCheckpoint('mdl_wts.hdf5', save_best_only=True, monitor='val_loss', mode='min')\nreduce_lr_loss = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.6, patience=5, verbose=1,min_delta=0.0001 ,min_lr=0.000001, mode='min')","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:48.721419Z","iopub.execute_input":"2022-10-17T01:14:48.723771Z","iopub.status.idle":"2022-10-17T01:14:48.732250Z","shell.execute_reply.started":"2022-10-17T01:14:48.723734Z","shell.execute_reply":"2022-10-17T01:14:48.731158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to get the train data!","metadata":{}},{"cell_type":"code","source":"df_train = read_train(0,10)\n\nBATCH_SIZE = 8192\ngame_nums = df_train['game_num'].unique()\ntrain_game_nums = random.sample(list(game_nums), int(len(game_nums) * 0.80))","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:48.733723Z","iopub.execute_input":"2022-10-17T01:14:48.734388Z","iopub.status.idle":"2022-10-17T01:14:57.798616Z","shell.execute_reply.started":"2022-10-17T01:14:48.734343Z","shell.execute_reply":"2022-10-17T01:14:57.796526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_feature = df_train.query(\"game_num in @train_game_nums\")[features].to_numpy()\ntrain_target = df_train.query(\"game_num in @train_game_nums\")[targets].to_numpy()\nvalid_feature = df_train.query(\"game_num not in @train_game_nums\")[features].to_numpy()\nvalid_target  = df_train.query(\"game_num not in @train_game_nums\")[targets].to_numpy()","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:57.799920Z","iopub.status.idle":"2022-10-17T01:14:57.800684Z","shell.execute_reply.started":"2022-10-17T01:14:57.800407Z","shell.execute_reply":"2022-10-17T01:14:57.800432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = CustomDataGen(train_feature, train_target, BATCH_SIZE, augment=True)\nval = CustomDataGen(valid_feature, valid_target, BATCH_SIZE, augment=False)\n\ndel df_train","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:57.802029Z","iopub.status.idle":"2022-10-17T01:14:57.802754Z","shell.execute_reply.started":"2022-10-17T01:14:57.802481Z","shell.execute_reply":"2022-10-17T01:14:57.802505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's fit the model!","metadata":{}},{"cell_type":"code","source":"model.fit(train, epochs=30, validation_data=val, callbacks=[earlyStopping, mcp_save,reduce_lr_loss])","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:57.804027Z","iopub.status.idle":"2022-10-17T01:14:57.804746Z","shell.execute_reply.started":"2022-10-17T01:14:57.804474Z","shell.execute_reply":"2022-10-17T01:14:57.804498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = read_test()\ntest_data = CustomDataGen(df_test.to_numpy(), np.zeros(len(df_test)), batch_size = 3917, augment=True, testing =True)","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:57.806007Z","iopub.status.idle":"2022-10-17T01:14:57.806717Z","shell.execute_reply.started":"2022-10-17T01:14:57.806449Z","shell.execute_reply":"2022-10-17T01:14:57.806473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = model.evaluate(val, batch_size=1024)\nprint(results)","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:57.808021Z","iopub.status.idle":"2022-10-17T01:14:57.808727Z","shell.execute_reply.started":"2022-10-17T01:14:57.808460Z","shell.execute_reply":"2022-10-17T01:14:57.808484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We use a loop to predict different augmented forms of the test data. The augmentations should not give different results as the game and situation is the same.","metadata":{}},{"cell_type":"code","source":"preds = model.predict_generator(test_data)\n\nfor i in range(3):\n    preds += model.predict_generator(test_data)\npreds = preds/4\nsubmission = pd.read_csv('../input/tabular-playground-series-oct-2022/sample_submission.csv')\nsubmission.iloc[:, 1:] = preds\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-17T01:14:57.809988Z","iopub.status.idle":"2022-10-17T01:14:57.810711Z","shell.execute_reply.started":"2022-10-17T01:14:57.810430Z","shell.execute_reply":"2022-10-17T01:14:57.810455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Feel free to give some feedback and/or copy the kernel to improve the model! ","metadata":{}}]}