{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook is a modified version of the [notebook](https://www.kaggle.com/code/paddykb/tps-2022-10-fastai) shared by [@paddykb](https://www.kaggle.com/paddykb).\n\nIn this notebook I show how using only distance from ball, angle between ball and player velocity vector and the angle formed between player position, goal position and ball position while removing players movement speed does not kill performance of an already really good solution.\n\nApart from some modifications I tried out to avoid Out of memory issues (like using only 8 splits of training data instead of 10) and changing just modifying a bit the architecture of the network the notebook is unchanged.\n\nThis gives us an idea that getting these feature in the model and finding a way to fit everything in memory can allow us to unlock further improvements.\nIt's likely that using 100% training data can give us better results. \n\nAgain many thanks to @paddykb for sharing the original notebook.\nI hope I didn't make mistakes in calculating new features ( it's quite likely as it is 1AM here). I tried to post this as soon as possible to leave a reference to other participants. ","metadata":{}},{"cell_type":"code","source":"import random\nimport numpy as np\nimport pandas as pd\nimport gc\nfrom pathlib import Path\nfrom fastai.tabular.all import *\nimport fastai.losses as loss","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-10-14T21:36:16.282699Z","iopub.execute_input":"2022-10-14T21:36:16.283215Z","iopub.status.idle":"2022-10-14T21:36:18.768303Z","shell.execute_reply.started":"2022-10-14T21:36:16.283103Z","shell.execute_reply":"2022-10-14T21:36:18.767342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Introduction\n\nIn this notebook I'm building a NN using fastai. The \"value add\" is the augmentation of the training batches with random permutations from the 144 possible:\n* Six ways to order team A players.\n* Six ways to order team B players\n* flip teams A and B (removed and to be implementated -> requires manipulation of player angles probably)\n* horizontal reflection (removed and to be implementated -> requires manipulation of player angles probably)\n\nI've made no attempt to tune the model or try different architectures. If you do, please make your notebook public so I can learn from your efforts.\n\nOn the shoulders of giants:\n* @pietromaldini1 [data augmentation discussion](https://www.kaggle.com/competitions/tabular-playground-series-oct-2022/discussion/357577)\n* @hsuyab [Fast loading & High Compression with Feather](https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather)\n* @slawekbiel [preloading data into the GPU](https://www.kaggle.com/code/slawekbiel/fast-fastai-training)\n* @spyrow [Mirroring the board](https://www.kaggle.com/code/spyrow/playground-oct-2022-lgbmclassifier?scriptVersionId=107206095)\n\n**todo**: *Since we have event order in the train set, it would be interesting to model successive positions. I.e. use the current positions as input and the next position of players and ball as the output. (strip off the last layer and use it as an embedding). Take a closer look at borrowing this from @ryancaldwell [Predict next frame](https://www.kaggle.com/code/ryancaldwell/cnn-predict-next-frame)*","metadata":{}},{"cell_type":"code","source":"features = [\n    'ball_pos_x', 'ball_pos_y','ball_pos_z', 'ball_vel_x', 'ball_vel_y', 'ball_vel_z', \n    'p0_pos_x', 'p0_pos_y', 'p0_pos_z', 'p0_vel_x', 'p0_vel_y', 'p0_vel_z', 'p0_boost', 'p0_na',\n    'p1_pos_x', 'p1_pos_y', 'p1_pos_z', 'p1_vel_x', 'p1_vel_y', 'p1_vel_z', 'p1_boost', 'p1_na',\n    'p2_pos_x', 'p2_pos_y', 'p2_pos_z', 'p2_vel_x', 'p2_vel_y', 'p2_vel_z', 'p2_boost', 'p2_na',\n    'p3_pos_x', 'p3_pos_y', 'p3_pos_z', 'p3_vel_x', 'p3_vel_y', 'p3_vel_z', 'p3_boost', 'p3_na',\n    'p4_pos_x', 'p4_pos_y', 'p4_pos_z', 'p4_vel_x', 'p4_vel_y', 'p4_vel_z', 'p4_boost', 'p4_na',\n    'p5_pos_x', 'p5_pos_y', 'p5_pos_z', 'p5_vel_x', 'p5_vel_y', 'p5_vel_z', 'p5_boost', 'p5_na',\n    'boost0_timer', 'boost1_timer', \n    'boost2_timer', 'boost3_timer',\n    'boost4_timer', 'boost5_timer']\nfeatures_x_pos = [pos for pos, feature in enumerate(features) if feature.endswith('_x')]\nfeatures_y_pos = [pos for pos, feature in enumerate(features) if feature.endswith('_y')]\n\ntargets = [\n    'team_A_scoring_within_10sec',\n    'team_B_scoring_within_10sec']","metadata":{"execution":{"iopub.status.busy":"2022-10-14T21:36:18.770098Z","iopub.execute_input":"2022-10-14T21:36:18.770940Z","iopub.status.idle":"2022-10-14T21:36:18.780163Z","shell.execute_reply.started":"2022-10-14T21:36:18.770896Z","shell.execute_reply":"2022-10-14T21:36:18.778729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load Data","metadata":{}},{"cell_type":"code","source":"%%time\n\nDEBUG = False\ninput_path = Path('../input/fast-loading-high-compression-with-feather/feather_data')\n\ndef fe(x):\n    # indicators for respawns...\n    x['p0_na'] = x['p0_pos_x'].isna().astype('int8')\n    x['p1_na'] = x['p1_pos_x'].isna().astype('int8')\n    x['p2_na'] = x['p2_pos_x'].isna().astype('int8')\n    x['p3_na'] = x['p3_pos_x'].isna().astype('int8')\n    x['p4_na'] = x['p4_pos_x'].isna().astype('int8')\n    x['p5_na'] = x['p5_pos_x'].isna().astype('int8')\n    for feature in features:\n        if feature.endswith('_na'):\n            continue\n        # this is just scaling the features to something reasonable\n        # it might make sense to apply a transformation to the z-dimension.\n        if feature.endswith('_x'):\n            x[feature] = (x[feature] / 82).fillna(0).astype('float16')\n        if feature.endswith('_y'):\n            x[feature] = (x[feature] / 120).fillna(0).astype('float16')\n        if feature.endswith('_z'):\n            x[feature] = (x[feature] / 40).fillna(0).astype('float16')\n        if feature.endswith('_boost'):\n            x[feature] = (x[feature] / 100).fillna(0).astype('float16')\n        if feature.endswith('_timer'):\n            x[feature] = (-x[feature] / 100).astype('float16')\n    return x\n\ndef read_train():\n    dfs = []\n    for i in range(10):\n        dfs.append(fe(pd.read_feather(input_path / f'train_{i}_compressed.ftr')))\n    result = pd.concat(dfs)\n    if DEBUG:\n        result = result.sample(frac=0.05)\n    return result.sample(frac=1)\n\ndef read_test():\n    return fe(pd.read_feather(input_path / 'test_compressed.ftr'))\n\ndf_train = read_train()\ngc.collect()\n\nprint(f'Train Rows = {len(df_train):,}  ' \n      f'Memory Usage = {df_train.memory_usage(deep=True).sum() / (1024 * 1024):4.1f} Mb'\n     '\\n')","metadata":{"execution":{"iopub.status.busy":"2022-10-14T21:36:18.781401Z","iopub.execute_input":"2022-10-14T21:36:18.782164Z","iopub.status.idle":"2022-10-14T21:38:02.851726Z","shell.execute_reply.started":"2022-10-14T21:36:18.782127Z","shell.execute_reply":"2022-10-14T21:38:02.850633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Move Training Data to the GPU","metadata":{}},{"cell_type":"code","source":"# split train & validation\nfrac=0.1\ngame_nums = df_train['game_num'].unique()\ntrain_game_nums = random.sample(list(game_nums), int(len(game_nums) * 0.80))\n\ntrain_feature_tensor = torch.tensor(\n    df_train.query(\"game_num in @train_game_nums\")[[\"game_num\"]+features].groupby(\"game_num\").sample(frac=frac, random_state=161194)[features].to_numpy())\ntrain_target_tensor  = torch.tensor(\n    df_train.query(\"game_num in @train_game_nums\")[[\"game_num\"]+targets].groupby(\"game_num\").sample(frac=frac, random_state=161194)[targets].fillna(0).to_numpy())\nvalid_feature_tensor = torch.tensor(\n    df_train.query(\"game_num not in @train_game_nums\")[features].to_numpy())\nvalid_target_tensor  = torch.tensor(\n    df_train.query(\"game_num not in @train_game_nums\")[targets].fillna(0).to_numpy())\n\nvalid_feature_tensor_subsample = torch.tensor(\n    df_train.query(\"game_num not in @train_game_nums\")[[\"game_num\"]+features].groupby(\"game_num\").sample(frac=frac, random_state=161194)[features].to_numpy())\nvalid_target_tensor_subsample  = torch.tensor(\n    df_train.query(\"game_num not in @train_game_nums\")[[\"game_num\"]+targets].groupby(\"game_num\").sample(frac=frac, random_state=161194)[targets].fillna(0).to_numpy())\n\n\ngc.collect()\n\nif torch.cuda.is_available():\n    train_feature_tensor = train_feature_tensor.cuda()\n    train_target_tensor  = train_target_tensor.cuda()\n    valid_feature_tensor = valid_feature_tensor.cuda()\n    valid_target_tensor  = valid_target_tensor.cuda()\n    valid_feature_tensor_subsample=valid_feature_tensor_subsample.cuda()\n    valid_target_tensor_subsample=valid_target_tensor_subsample.cuda()\n\ndel df_train\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-10-14T21:38:02.854589Z","iopub.execute_input":"2022-10-14T21:38:02.855236Z","iopub.status.idle":"2022-10-14T21:40:02.278802Z","shell.execute_reply.started":"2022-10-14T21:38:02.855194Z","shell.execute_reply":"2022-10-14T21:40:02.277617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def augment_mirror(empty, X, Y):\n    # mirror the match\n    # interchange player 1 and 2\n    positions = X[:,:54]\n    positions[:, features_x_pos] = -positions[:, features_x_pos]\n    positions[:, features_y_pos] = -positions[:, features_y_pos]\n    \n    ball = positions[:, :6]\n    p0 = positions[:,  6:14]\n    p1 = positions[:, 14:22]\n    p2 = positions[:, 22:30]\n    p3 = positions[:, 30:38]\n    p4 = positions[:, 38:46]\n    p5 = positions[:, 46:54]\n    \n    players = torch.cat([p3, p4, p5, p0, p1, p2], dim=1)\n    # mirror\n    boosts = X[:, [59, 58, 57, 56, 55, 54]]\n    \n    flip_X = torch.cat([ball, players, boosts], dim=1)\n    flip_Y = Y[:, :, [1,0]]\n    \n    return empty, flip_X, flip_Y\n\ndef augment_flip_x(empty, X, Y):\n    # mirror the match in the Y-axis\n    positions = X[:,:54]\n    positions[:, features_x_pos] = -positions[:, features_x_pos]\n    boosts = X[:, [55, 54, 57, 56, 59, 58]]\n    \n    flip_X = torch.cat([positions, boosts], dim=1)\n    \n    return empty, flip_X, Y\n\ndef augment_shuffle(empty, X, Y):\n    # randomly order players (within teams)\n    ball = X[:, :6]\n    p0 = X[:,  6:14]\n    p1 = X[:, 14:22]\n    p2 = X[:, 22:30]\n    p3 = X[:, 30:38]\n    p4 = X[:, 38:46]\n    p5 = X[:, 46:54]\n    boosts = X[:, 54:]\n    \n    # shuffle player positions\n    pA = torch.cat(random.sample([p0, p1, p2], 3), dim=1)\n    pB = torch.cat(random.sample([p3, p4, p5], 3), dim=1)\n    \n    # shuffled feats\n    shuffled_X = torch.cat([ball, pA, pB, boosts], dim=1)\n    \n    return empty, shuffled_X, Y\n\n#'p0_pos_x', 'p0_pos_y', 'p0_pos_z', 'p0_vel_x', 'p0_vel_y', 'p0_vel_z', 'p0_boost', 'p0_na',\ndef get_single_player_features(player,ball,index=0):\n    pos_player=torch.clone(player[:,:3])\n    pos_ball=torch.clone(ball[:,:3])\n    pos_goal1=torch.zeros_like(pos_player)[:,:2]\n    pos_goal2=torch.zeros_like(pos_player)[:,:2]\n    \n    pos_goal1[:,0]=0\n    pos_goal1[:,1]=120\n    \n    pos_goal2[:,0]=0\n    pos_goal2[:,1]=-120\n    \n    vel_player=torch.clone(player[:,3:6])\n    vel_ball=torch.clone(ball[:,3:])\n    \n    #undo normalization for angle and distance formulation ( otherwise we have distorted space)\n    \n    pos_player[:,0]=pos_player[:,0]*82\n    pos_ball[:,0]=pos_ball[:,0]*82\n    vel_player[:,0]=pos_player[:,0]*82\n    vel_ball[:,0]=pos_ball[:,0]*82\n    \n    pos_player[:,1]=pos_player[:,1]*120\n    pos_ball[:,1]=pos_ball[:,1]*120\n    vel_player[:,1]=pos_player[:,1]*120\n    vel_ball[:,1]=pos_ball[:,1]*120\n    \n    pos_player[:,2]=pos_player[:,2]*40\n    pos_ball[:,2]=pos_ball[:,2]*40\n    vel_player[:,2]=pos_player[:,2]*40\n    vel_ball[:,2]=pos_ball[:,2]*40\n    \n    \n    \n    distance=torch.sqrt(torch.sum(torch.square(pos_player-pos_ball),dim=-1))/300\n    \n    relative_speed=torch.sqrt(torch.sum(torch.square(vel_player-vel_ball),dim=-1))/300\n    \n    cos_angle=torch.sum(torch.tensordot(vel_player,vel_ball,dims=[[-1],[-1]]),dim=-1)/(torch.sqrt(torch.norm(torch.tensordot(vel_player,vel_ball,dims=[[-1],[-1]]),dim=-1))+1e-5)\n    \n    # From here consider projection on 2 axis(ignore z axis)\n    pos_player=pos_player[:,:2]\n    pos_ball=pos_ball[:,:2]\n    vel_player=vel_player[:,:2]\n    vel_ball=vel_ball[:,:2]\n    distance_p_goal1=torch.sqrt(torch.sum(torch.square(pos_player-pos_goal1),dim=-1))/300\n    distance_p_goal2=torch.sqrt(torch.sum(torch.square(pos_player-pos_goal2),dim=-1))/300\n    \n    if index==0:\n        \n\n        distance_b_goal1=torch.sqrt(torch.sum(torch.square(pos_ball-pos_goal1),dim=-1))/300\n        distance_b_goal2=torch.sqrt(torch.sum(torch.square(pos_ball-pos_goal2),dim=-1))/300\n    \n    cos_angle_b_goal1=torch.sum(torch.tensordot(pos_ball-pos_player,pos_ball-pos_goal1),dim=-1)/(torch.sqrt(torch.norm(torch.tensordot(pos_ball-pos_player,pos_ball-pos_goal1,dims=[[-1],[-1]]),dim=-1))+1e-5)\n    \n    cos_angle_b_goal2=torch.sum(torch.tensordot(pos_ball-pos_player,pos_ball-pos_goal2),dim=-1)/(torch.sqrt(torch.norm(torch.tensordot(pos_ball-pos_player,pos_ball-pos_goal2,dims=[[-1],[-1]]),dim=-1))+1e-5)\n    \n    cos_angle_p_goal1=torch.sum(torch.tensordot(pos_ball-pos_player,pos_player-pos_goal1),dim=-1)/(torch.sqrt(torch.norm(torch.tensordot(pos_ball-pos_player,pos_ball-pos_goal1,dims=[[-1],[-1]]),dim=-1))+1e-5)\n    \n    cos_angle_p_goal2=torch.sum(torch.tensordot(pos_ball-pos_player,pos_player-pos_goal2),dim=-1)/(torch.sqrt(torch.norm(torch.tensordot(pos_ball-pos_player,pos_ball-pos_goal2,dims=[[-1],[-1]]),dim=-1))+1e-5)\n    \n    \n    cos_angle_speed_goal1=torch.sum(torch.tensordot(vel_ball,pos_ball-pos_goal1),dim=-1)/(torch.sqrt(torch.norm(torch.tensordot(vel_ball,pos_ball-pos_goal1,dims=[[-1],[-1]]),dim=-1))+1e-5)\n    cos_angle_speed_goal2=torch.sum(torch.tensordot(vel_ball,pos_ball-pos_goal2),dim=-1)/(torch.sqrt(torch.norm(torch.tensordot(vel_ball,pos_ball-pos_goal2,dims=[[-1],[-1]]),dim=-1))+1e-5)\n    if index==0:\n        res= torch.cat([distance.reshape(-1,1),\n                        relative_speed.reshape(-1,1),\n                        cos_angle.reshape(-1,1),\n                        distance_p_goal1.reshape(-1,1),\n                        distance_p_goal2.reshape(-1,1),\n                        distance_b_goal1.reshape(-1,1),\n                        distance_b_goal2.reshape(-1,1),\n                        cos_angle_p_goal1.reshape(-1,1),\n                        cos_angle_p_goal2.reshape(-1,1),\n                        cos_angle_b_goal1.reshape(-1,1),\n                        cos_angle_b_goal2.reshape(-1,1),\n                        cos_angle_speed_goal1.reshape(-1,1),\n                        cos_angle_speed_goal2.reshape(-1,1)\n                       ],dim=1)\n    else:\n         res= torch.cat([distance.reshape(-1,1),\n                        relative_speed.reshape(-1,1),\n                        cos_angle.reshape(-1,1),\n                        distance_p_goal1.reshape(-1,1),\n                        distance_p_goal2.reshape(-1,1),\n                        cos_angle_p_goal1.reshape(-1,1),\n                        cos_angle_p_goal2.reshape(-1,1),\n                        cos_angle_b_goal1.reshape(-1,1),\n                        cos_angle_b_goal2.reshape(-1,1),\n                        cos_angle_speed_goal1.reshape(-1,1),\n                        cos_angle_speed_goal2.reshape(-1,1)\n                       ],dim=1)\n        \n    return res\ndef add_features(empty, X, Y):\n    ball =X[:,:6]\n    p0 = X[:,  6:14]\n    p1 = X[:, 14:22]\n    p2 = X[:, 22:30]\n    p3 = X[:, 30:38]\n    p4 = X[:, 38:46]\n    p5 = X[:, 46:54]\n    feats=[]\n    for i in range(6):\n        p=X[:,6+(i*8):14+(i*8)]\n        feats.append(get_single_player_features(p,ball,index=i))\n    \n    X = torch.cat([X,*feats], dim=1)\n    \n    return empty,X,Y\n#This bespoke dataset classes are based on the ideas of @slawekbiel \nclass BespokeDataset:\n    def __init__(self, feature_tensor, targets, augment=False):\n        store_attr()\n        self.n_inp = 2\n    def __getitem__(self, idx):\n        # convert float16 -> float32 during the minibatch\n        # and apply any augmentation\n        # we might also apply any feature engineering here (as we'll only use a little memory)\n        batch = torch.empty(0), self.feature_tensor[idx].float(), self.targets[idx, None]\n        if self.augment:\n            # shuffle player positions.\n            batch = augment_shuffle(*batch)\n            if random.random() > 0.5:\n                batch = augment_mirror(*batch)\n            if random.random() > 0.5:\n                batch = augment_flip_x(*batch)\n        batch=add_features(*batch)\n        return batch\n    \n    def __len__(self):\n        return len(self.feature_tensor)\n    \nclass BespokeDL(DataLoader):\n    def __iter__(self):\n        if self.shuffle:\n            temp=list(range(0,self.n))\n            random.shuffle(temp)\n            self.__idxs = torch.tensor(temp)\n        else:\n            self.__idxs = torch.tensor(range(0,self.n))\n        for batch_start in range(0, self.n, self.bs):\n            if batch_start + self.bs > self.n and self.drop_last:\n                \n                return \n            indices = self.__idxs[batch_start:batch_start+self.bs]\n            yield self.dataset[indices]\n            \n","metadata":{"execution":{"iopub.status.busy":"2022-10-14T21:44:49.118509Z","iopub.execute_input":"2022-10-14T21:44:49.118995Z","iopub.status.idle":"2022-10-14T21:44:49.360402Z","shell.execute_reply.started":"2022-10-14T21:44:49.118950Z","shell.execute_reply":"2022-10-14T21:44:49.359107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluate Model\n\ntodo: tune / try different layer architectures","metadata":{}},{"cell_type":"code","source":"ds_train = BespokeDataset(train_feature_tensor, train_target_tensor, augment=True)\n#Think what to do here subsample valid or not\nds_val   = BespokeDataset(valid_feature_tensor, valid_target_tensor)\n# we want to keep validation as is since this should have a similar distribution compared with test set\n# setting augment=True gives a better estimate of performance in a general setting, but here we want to maximize performance for test set ! \n\ndls = DataLoaders.from_dsets(ds_train, ds_val, bs=4096, dl_type=BespokeDL, num_workers=0)","metadata":{"execution":{"iopub.status.busy":"2022-10-14T22:02:58.280026Z","iopub.execute_input":"2022-10-14T22:02:58.280496Z","iopub.status.idle":"2022-10-14T22:02:58.292067Z","shell.execute_reply.started":"2022-10-14T22:02:58.280451Z","shell.execute_reply":"2022-10-14T22:02:58.291099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(ds_train), len(ds_val)","metadata":{"execution":{"iopub.status.busy":"2022-10-14T22:02:59.244017Z","iopub.execute_input":"2022-10-14T22:02:59.244435Z","iopub.status.idle":"2022-10-14T22:02:59.252085Z","shell.execute_reply.started":"2022-10-14T22:02:59.244399Z","shell.execute_reply":"2022-10-14T22:02:59.251131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = TabularModel(\n    emb_szs={}, n_cont=128, ps=0.7, out_sz=len(targets), \n    layers=[512,256, 128,64], y_range=(0,1))\nif torch.cuda.is_available():\n    model = model.cuda()\n    \nlearn = Learner(dls, model, loss_func=loss.BCELossFlat())","metadata":{"execution":{"iopub.status.busy":"2022-10-14T22:02:59.894074Z","iopub.execute_input":"2022-10-14T22:02:59.894464Z","iopub.status.idle":"2022-10-14T22:02:59.910962Z","shell.execute_reply.started":"2022-10-14T22:02:59.894426Z","shell.execute_reply":"2022-10-14T22:02:59.910079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model","metadata":{"execution":{"iopub.status.busy":"2022-10-14T22:03:00.505188Z","iopub.execute_input":"2022-10-14T22:03:00.505647Z","iopub.status.idle":"2022-10-14T22:03:00.513304Z","shell.execute_reply.started":"2022-10-14T22:03:00.505603Z","shell.execute_reply":"2022-10-14T22:03:00.512334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfit = learn.fit(25, 1e-3)","metadata":{"execution":{"iopub.status.busy":"2022-10-14T22:03:01.118903Z","iopub.execute_input":"2022-10-14T22:03:01.119272Z","iopub.status.idle":"2022-10-14T22:20:48.578133Z","shell.execute_reply.started":"2022-10-14T22:03:01.119237Z","shell.execute_reply":"2022-10-14T22:20:48.577187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Refit using all training data","metadata":{}},{"cell_type":"code","source":"# refit on full data - ignore the validation... it is meaningless\nds_train = BespokeDataset(\n    torch.cat([train_feature_tensor, valid_feature_tensor_subsample]), \n    torch.cat([train_target_tensor, valid_target_tensor_subsample]), \n    augment=True)\nds_val   = BespokeDataset(valid_feature_tensor_subsample, valid_target_tensor_subsample)\ndls = DataLoaders.from_dsets(ds_train, ds_val, bs=4096, dl_type=BespokeDL, num_workers=0)\n\nmodel = TabularModel(\n    emb_szs={}, n_cont=128, ps=0.7, out_sz=len(targets), \n    layers=[512, 256, 128, 64], y_range=(0,1))\nif torch.cuda.is_available():\n    model = model.cuda()\n    \nlearn = Learner(dls, model, loss_func=loss.BCELossFlat())\n\nfit = learn.fit(25, 1e-3)","metadata":{"execution":{"iopub.status.busy":"2022-10-14T22:21:50.318557Z","iopub.execute_input":"2022-10-14T22:21:50.319027Z","iopub.status.idle":"2022-10-14T22:31:17.304173Z","shell.execute_reply.started":"2022-10-14T22:21:50.318984Z","shell.execute_reply":"2022-10-14T22:31:17.303208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare Submission","metadata":{}},{"cell_type":"code","source":"features = [\n    'ball_pos_x', 'ball_pos_y','ball_pos_z', 'ball_vel_x', 'ball_vel_y', 'ball_vel_z', \n    'p0_pos_x', 'p0_pos_y', 'p0_pos_z', 'p0_vel_x', 'p0_vel_y', 'p0_vel_z', 'p0_boost', 'p0_na',\n    'p1_pos_x', 'p1_pos_y', 'p1_pos_z', 'p1_vel_x', 'p1_vel_y', 'p1_vel_z', 'p1_boost', 'p1_na',\n    'p2_pos_x', 'p2_pos_y', 'p2_pos_z', 'p2_vel_x', 'p2_vel_y', 'p2_vel_z', 'p2_boost', 'p2_na',\n    'p3_pos_x', 'p3_pos_y', 'p3_pos_z', 'p3_vel_x', 'p3_vel_y', 'p3_vel_z', 'p3_boost', 'p3_na',\n    'p4_pos_x', 'p4_pos_y', 'p4_pos_z', 'p4_vel_x', 'p4_vel_y', 'p4_vel_z', 'p4_boost', 'p4_na',\n    'p5_pos_x', 'p5_pos_y', 'p5_pos_z', 'p5_vel_x', 'p5_vel_y', 'p5_vel_z', 'p5_boost', 'p5_na',\n    'boost0_timer', 'boost1_timer', \n    'boost2_timer', 'boost3_timer',\n    'boost4_timer', 'boost5_timer']\ndf_test = read_test()\n\ngc.collect()\n\nds_test = BespokeDataset(torch.tensor(df_test[features].to_numpy()), torch.zeros(len(df_test), 2))\ntest_dl = learn.dls.test_dl(ds_test)\npreds, _ = learn.get_preds(dl=test_dl)\n\nsubmission = pd.read_csv('../input/tabular-playground-series-oct-2022/sample_submission.csv')\nsubmission.iloc[:, 1:] = preds.numpy()\nsubmission.to_csv('model_fastai_v2.csv', index=False)\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-14T22:31:17.309091Z","iopub.execute_input":"2022-10-14T22:31:17.311334Z","iopub.status.idle":"2022-10-14T22:43:09.569122Z","shell.execute_reply.started":"2022-10-14T22:31:17.311293Z","shell.execute_reply":"2022-10-14T22:43:09.568128Z"},"trusted":true},"execution_count":null,"outputs":[]}]}