{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"#Imports\n!pip install torcheval\n!apt install -y ffmpeg\n#!pip install ffmpeg-python\n\nimport pandas as pd\nimport numpy as np\nimport sys\nimport gc\nfrom sklearn.model_selection import GroupKFold\nfrom sklearn import preprocessing\n\nfrom torch.autograd import Variable \nfrom tqdm import tqdm\nimport glob\nimport random\nimport cv2\nimport os\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nimport time\nimport torcheval\nimport timm\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\nimport torch.optim as optim\n\nfrom torcheval.metrics import BinaryAccuracy\n\n\nfrom torch import nn\nfrom sklearn.metrics import matthews_corrcoef\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler\nfrom sklearn.compose import ColumnTransformer\nimport math\n\nfrom timm.scheduler import CosineLRScheduler\n\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-21T09:50:31.823086Z","iopub.execute_input":"2023-07-21T09:50:31.823473Z","iopub.status.idle":"2023-07-21T09:50:49.141940Z","shell.execute_reply.started":"2023-07-21T09:50:31.823441Z","shell.execute_reply":"2023-07-21T09:50:49.140443Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **NFL contact detection 2023**\n\nMy inference notebook for scoring model: https://www.kaggle.com/code/gumm4n/nfl-inference\n\n*Notebook based on public 2,5D CNN inference notebook made by* **zzy**. *Notebook: https://www.kaggle.com/code/zzy990106/nfl-2-5d-cnn-baseline-inference*\n\n\n#### Results: \n* My score: 0.678(Matthews correlation cofficient) *(Training not done on full dataset because of no local gpu and time limitations)*\n* Winning score was 0.79 (0.69 for medal)\n\n#### *Task:* \nThe goal of this competition is to detect external contact experienced by players during an NFL football game. You will use video and player tracking data to identify moments with contact to help improve player safety.\nSubmissions are evaluated on Matthews Correlation Coefficient between the predicted and actual contact events.\n\n<img align=left src = \"attachment:image's http://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F644036%2F65cd663d2c823043b36ecda6c93c1304%2Fcontact-example.gif?generation=1670265252697886&alt=media\" width=\"500\">\n\n\n\n![](http://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F644036%2F65cd663d2c823043b36ecda6c93c1304%2Fcontact-example.gif?generation=1670265252697886&alt=media)\n\n#### **Data:**\n**Videos**\nEach play has four associated videos. Two videos, showing a sideline and endzone view, are time synced and aligned with each other. Additionally, an All29 view is provided but not guaranteed to be time synced. The training set videos are in train/ with corresponding labels in train_labels.csv, while the videos for which you must predict are in the test/ folder.\n\n**Tracking data**\nTracking data is provided with player tracking data for every 6th frame. The data contains postitional data as well as for example velocity, accleration and orientation.\n\n**Helmet data**\nAn imperfect dataset of helmet object detections for all videos exists. Helmet position for each player each frame exists. Data is based on previous Kaggle helmet detection competitions.\n\n\n#### **Model:**\nMain idea was to try old school video classification with (2D CNN + lstm) concatenated with the feature data to be able to understand the temporal video data.\n\n**Input**\n* Tracking features, original data for both players + added distance between players -> MLP\n* Images for each frame, both sideline and endzone view. -> CNN -> LSTM\n\n**Output**\nGiven 2 player id's(or 1 player and G(ground)) the model should be able to tell if there is contact or not, 1 or 0.\nA threshold of 2m was chosen as a maximum distance possibility for possible contact. Thus all inputs where distance between players > 2 was set as 0.\n\n**2D CNN input**\n\nTwo channels were used: One channel with the image centered around the two bounding boxes of the players, and the other channel with only the bounding boxes as well as some features to help the CNN to find where the area of interest are.\n\n*Image channel:*\nDue to the players being different size for different videos/frames, one could make use of the size of the helmet bounding box to estimate a area of interest.\n* Images were cropped to: max(average bbox width, average bbox height) * 4 (centered in the mean coordinate of the two bboxes).\n* Then images were resized to 256x256.\n\n     ![](https://i.ibb.co/Kwv2P7C/nfl-channel1.png)    \n\n\n*Feature channel:*\n* Draw the bounding boxes on blank 256x256 to help the CNN where to look in the image.\n* Naively thinking the distance between the players should be the most important feature to classify contact. Thus the color of the bboxes could be changed depending on the distance between the players.\n    * Player v ground: *color = 100*\n    * Player1 v player2: *color = 255-50*distance* (Ranging from 155(distance = 2) to 255(distance = 0))\n    \n   ![](https://i.ibb.co/YtD1xLb/nfl-channel2.png)    \n\n**LSTM**\nThe ouput the from the CNNs, from the sideline and endzone frames are concatenated together into one feature vector.\nA sequence of 5 frames each with a time step of 12 frames were used as input into the LSTM.\n\n\nOutput from LSTM model were concatenated with the tracking data features from(passed through a MLP layer) to get the final output as the probability of the specific frame having concatct between p1 and p2.\n\n\n**Data**\nRegular data augmentation were done.\nGiven the specific frame a randomized frame +-3 frames apart were used to randomize training.\nOne frame were only used once during training, thus each 6th frame were used. This helped overfitting and minimized the training data alot.\n\n#### *Test:*\nNo real parameter testing was done due to no availble offline GPU so testing was limited to Kaggle GPU quota. Model is more a concept and made for learning.\n\n#### *Ideas for improvement*\n* Split player to player and player to ground detections into two models. Features only of for 1 player, no distance, no 2 boxes and so on. Instead of separating by pixel values and setting features to negative, split the two problems.\n* 3D CNN work instead of LSTM, reached better scores in other competitions.\n* Maybe use a separate XGB/LGBM model for the tracking data instead of MLP. Then do ensemble after DL model.\n\n","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nfolder = \"/kaggle/input/nfl-player-contact-detection/\"\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:50:49.144787Z","iopub.execute_input":"2023-07-21T09:50:49.145547Z","iopub.status.idle":"2023-07-21T09:50:49.233833Z","shell.execute_reply.started":"2023-07-21T09:50:49.145504Z","shell.execute_reply":"2023-07-21T09:50:49.232701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Function for translating contact_id in input data to different features\ndef expand_contact_id(df):\n    \"\"\"\n    Splits out contact_id into seperate columns.\n    \"\"\"\n    df[\"game_play\"] = df[\"contact_id\"].str[:12]\n    df[\"step\"] = df[\"contact_id\"].str.split(\"_\").str[-3].astype(\"int\")\n    df[\"nfl_player_id_1\"] = df[\"contact_id\"].str.split(\"_\").str[-2].astype(\"int\")\n    df[\"nfl_player_id_2\"] = df[\"contact_id\"].str.split(\"_\").str[-1]\n    df[\"game_key\"] = df[\"contact_id\"].str.split(\"_\").str[-5].astype(\"int\")\n    df[\"play_id\"] = df[\"contact_id\"].str.split(\"_\").str[-4].astype(\"int\")\n    return df\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:50:49.254739Z","iopub.execute_input":"2023-07-21T09:50:49.257551Z","iopub.status.idle":"2023-07-21T09:50:49.267363Z","shell.execute_reply.started":"2023-07-21T09:50:49.257514Z","shell.execute_reply":"2023-07-21T09:50:49.266495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Input data\n\ntrain_labels = expand_contact_id(pd.read_csv(folder+\"/train_labels.csv\")[[\"contact_id\",\"contact\"]])\ntrain_tracking = pd.read_csv(folder+\"/train_player_tracking.csv\")\ntrain_helmets = pd.read_csv(folder+\"/train_baseline_helmets.csv\")\ntrain_video_metadata = pd.read_csv(folder+\"/train_video_metadata.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:50:49.272417Z","iopub.execute_input":"2023-07-21T09:50:49.275237Z","iopub.status.idle":"2023-07-21T09:52:39.217065Z","shell.execute_reply.started":"2023-07-21T09:50:49.275131Z","shell.execute_reply":"2023-07-21T09:52:39.215954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Finding indexes with both sideline and endzone frames\n\nlabel_index = train_labels[['game_key', 'play_id',  'nfl_player_id_1', 'step']]\nlabel_index['frame'] = (label_index['step']/10*59.94+5*59.94).astype('int')+1\nlabel_index = label_index[['game_key', 'play_id',  'nfl_player_id_1', 'frame']].reset_index(drop = False)\n\ntrain_helmet = train_helmets[['game_key', 'play_id',  'nfl_player_id', 'frame', 'view']]\nview_list = (train_helmet.groupby(['game_key', 'play_id',  'nfl_player_id', 'frame']).sum())\nall_views = view_list[view_list['view'] == 'EndzoneSideline'].reset_index()\n\ndf_index = label_index.merge(all_views, how = 'inner', right_on = ['game_key', 'play_id',  'nfl_player_id', 'frame'], left_on = ['game_key', 'play_id',  'nfl_player_id_1', 'frame'])\nindex_list = df_index['index'].to_list()\nsorted_labels = train_labels.iloc[index_list]","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:52:39.218572Z","iopub.execute_input":"2023-07-21T09:52:39.218937Z","iopub.status.idle":"2023-07-21T09:52:41.082232Z","shell.execute_reply.started":"2023-07-21T09:52:39.218893Z","shell.execute_reply":"2023-07-21T09:52:41.081154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Concatenate data for two players and the labels into one feature set. Adding distance feature as well as flagging if its pvp or pvg (p= player, g = ground)\n\n\ndef create_features(df, tr_tracking, merge_col=\"step\", use_cols=[\"x_position\", \"y_position\"]):\n    output_cols = []\n    df_combo = (\n        df.astype({\"nfl_player_id_1\": \"str\"})\n        .merge(\n            tr_tracking.astype({\"nfl_player_id\": \"str\"})[\n                [\"game_play\", merge_col, \"nfl_player_id\",] + use_cols\n            ],\n            left_on=[\"game_play\", merge_col, \"nfl_player_id_1\"],\n            right_on=[\"game_play\", merge_col, \"nfl_player_id\"],\n            how=\"left\",\n        )\n        .rename(columns={c: c+\"_1\" for c in use_cols})\n        .drop(\"nfl_player_id\", axis=1)\n        .merge(\n            tr_tracking.astype({\"nfl_player_id\": \"str\"})[\n                [\"game_play\", merge_col, \"nfl_player_id\"] + use_cols\n            ],\n            left_on=[\"game_play\", merge_col, \"nfl_player_id_2\"],\n            right_on=[\"game_play\", merge_col, \"nfl_player_id\"],\n            how=\"left\",\n        )\n        .drop(\"nfl_player_id\", axis=1)\n        .rename(columns={c: c+\"_2\" for c in use_cols})\n        .sort_values([\"game_play\", merge_col, \"nfl_player_id_1\", \"nfl_player_id_2\"])\n        .reset_index(drop=True)\n    )\n    output_cols += [c+\"_1\" for c in use_cols]\n    output_cols += [c+\"_2\" for c in use_cols]\n    \n    if (\"x_position\" in use_cols) & (\"y_position\" in use_cols):\n        index = df_combo['x_position_2'].notnull()\n        \n        distance_arr = np.full(len(index), np.nan)\n        tmp_distance_arr = np.sqrt(\n            np.square(df_combo.loc[index, \"x_position_1\"] - df_combo.loc[index, \"x_position_2\"])\n            + np.square(df_combo.loc[index, \"y_position_1\"]- df_combo.loc[index, \"y_position_2\"])\n        )\n        \n        distance_arr[index] = tmp_distance_arr\n        df_combo['distance'] = distance_arr\n        output_cols += [\"distance\"]\n        \n    df_combo['G_flug'] = (df_combo['nfl_player_id_2']==\"G\")\n    output_cols += [\"G_flug\"]\n    return df_combo, output_cols\n\nuse_cols = [\n    'x_position', 'y_position', 'speed', 'distance',\n    'direction', 'orientation', 'acceleration', 'sa'\n]\n\n\ntrain, feature_cols = create_features(sorted_labels, train_tracking, use_cols=use_cols)\ntrain_filtered = train.query('not distance > 2').reset_index(drop=True)\ntrain_filtered['frame'] = (train_filtered['step']/10*59.94+5*59.94).astype('int')+1\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:52:44.335805Z","iopub.execute_input":"2023-07-21T09:52:44.336187Z","iopub.status.idle":"2023-07-21T09:52:55.467874Z","shell.execute_reply.started":"2023-07-21T09:52:44.336150Z","shell.execute_reply":"2023-07-21T09:52:55.466737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train, train_labels, train_tracking\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:52:55.693543Z","iopub.execute_input":"2023-07-21T09:52:55.693921Z","iopub.status.idle":"2023-07-21T09:52:56.657190Z","shell.execute_reply.started":"2023-07-21T09:52:55.693881Z","shell.execute_reply":"2023-07-21T09:52:56.656337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Scale features\n\nscaled_features = train_filtered.copy()\nfeatures = scaled_features[feature_cols]\nscaler = MinMaxScaler().fit(features.values)\nfeatures = scaler.transform(features.values)\nscaled_features[feature_cols] = features\ntrain_filtered = scaled_features\n\ndel scaled_features\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:52:56.661195Z","iopub.execute_input":"2023-07-21T09:52:56.663361Z","iopub.status.idle":"2023-07-21T09:52:57.951749Z","shell.execute_reply.started":"2023-07-21T09:52:56.663325Z","shell.execute_reply":"2023-07-21T09:52:57.950583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Use every 6th frame\ntrain_filtered.sort_values(by=['game_play', 'nfl_player_id_1', 'nfl_player_id_2', 'frame']).reset_index(drop = True)\ntrain_filtered = train_filtered.iloc[5::6,:].reset_index(drop = True)\n\n\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:52:57.953358Z","iopub.execute_input":"2023-07-21T09:52:57.953818Z","iopub.status.idle":"2023-07-21T09:52:58.233030Z","shell.execute_reply.started":"2023-07-21T09:52:57.953780Z","shell.execute_reply":"2023-07-21T09:52:58.232043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Augmentations\n\n\ntrain_aug = A.Compose([\n    A.HorizontalFlip(p=0.5),\n    A.ShiftScaleRotate(p=0.5),\n    #A.RandomBrightnessContrast(brightness_limit=(-0.05, 0.05), contrast_limit=(-0.05, 0.05), p=0.3),\n    #A.RandomGamma(p=0.5),\n    A.Normalize(mean=[0.], std=[1.]),\n    ToTensorV2()])\n\nvalid_aug = A.Compose([\n    A.Normalize(mean=[0.], std=[1.]),\n    ToTensorV2()\n])","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:52:58.288031Z","iopub.execute_input":"2023-07-21T09:52:58.288380Z","iopub.status.idle":"2023-07-21T09:52:58.296688Z","shell.execute_reply.started":"2023-07-21T09:52:58.288345Z","shell.execute_reply":"2023-07-21T09:52:58.295440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"video2helmets = {}\ntrain_helmets_new = train_helmets.set_index('video')\nfor video in tqdm(train_helmets.video.unique()):\n    video2helmets[video] = train_helmets_new.loc[video].reset_index(drop=True)\n    \ndel train_helmets, train_helmets_new\ngc.collect()\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:52:58.298426Z","iopub.execute_input":"2023-07-21T09:52:58.299591Z","iopub.status.idle":"2023-07-21T09:54:16.004498Z","shell.execute_reply.started":"2023-07-21T09:52:58.299553Z","shell.execute_reply":"2023-07-21T09:54:16.003357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"video2frames = {}\n\nfor game_play in tqdm(train_video_metadata.game_play.unique()):\n    for view in ['Endzone', 'Sideline']:\n        video = game_play + f'_{view}.mp4'\n        video2frames[video] = max(list(map(lambda x:int(x.split('_')[-1].split('.')[0]), \\\n                                           glob.glob(f\"/kaggle/input/nfl-contact-extracted-train-frames/content/work/frames/train/{video}*\"))))","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:54:16.005829Z","iopub.execute_input":"2023-07-21T09:54:16.006509Z","iopub.status.idle":"2023-07-21T09:59:50.386078Z","shell.execute_reply.started":"2023-07-21T09:54:16.006463Z","shell.execute_reply":"2023-07-21T09:59:50.385079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Data loader, returns wanted feature, label and cropped images([20, 256, 256], 10 frames for each view)\n\n\nclass MyDataset_LSTM(Dataset):\n    def __init__(self, df, aug=train_aug):\n        self.df = df\n        self.frame = df.frame.values\n        self.feature = df[feature_cols].fillna(-1).values\n        self.players = df[['nfl_player_id_1','nfl_player_id_2']].values\n        self.game_play = df.game_play.values\n        self.aug = aug\n        self.contact_id = df.contact_id.values\n        \n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):  \n        time_size = 12\n        sequence = 5    #number of frames in lstm sequence\n        frame = self.frame[idx]\n        frame_idx = frame\n        \n        \n        if self.aug == train_aug:\n            frame = frame + random.randint(-6, 6)\n        frame_diff = frame - frame_idx\n        players = []\n        \n      \n        for p in self.players[idx]:\n            if p == 'G':\n                players.append(p)\n            else:\n                players.append(int(p))\n   \n       \n        imgs_arr = []\n        \n        for f in range(frame-((sequence//2)*time_size), frame+((sequence//2)*time_size)+1, time_size):\n            imgs = []\n\n            for view in ['Endzone', 'Sideline']:\n\n                video = self.game_play[idx] + f'_{view}.mp4'\n                \n                tmp = video2helmets[video]           \n                tmp = tmp[tmp.nfl_player_id.isin(players)]       \n                tmp = tmp[tmp['frame'] == f]\n                \n                bboxes = []\n                img_size = 0\n                if len(tmp['left']):\n                  \n                    for i in range(len(tmp.index)):\n                        \n                        x = tmp['left'].tolist()[i]\n                        w = tmp['width'].tolist()[i]\n                        y = tmp['top'].tolist()[i]\n                        h = tmp['height'].tolist()[i]\n                        if math.isfinite(x):\n                            bboxes.append([x, w, y, h])\n                            img_size = max([img_size, w, h])\n                            \n                        \n                        \n                    img_helmet = np.zeros((720,1280), dtype=np.float32)  \n                    if len(tmp) == 2:\n                        \n                        \n                        dist = self.df.distance[idx]\n\n                        if math.isfinite(dist):\n                            color_dist = int(156+(100-(100*dist)))\n                            \n                        else:\n                            color_dist = 100\n                           \n                        color = (color_dist, 0, 0)                        \n                        \n                    else:\n                        color = (100, 0, 0)\n                    \n                    \n                    for i in range(len(tmp)):\n                        \n                        start_point = (bboxes[i][0], bboxes[i][2])\n                        end_point = (bboxes[i][0] +  bboxes[i][1], bboxes[i][2] + bboxes[i][3])\n                        \n\n                        img_helmet = cv2.rectangle(img_helmet, start_point, end_point, color, thickness =-1)\n                    \n                    \n                    img_new = np.zeros((256, 256), dtype=np.float32)         \n                    img = cv2.imread(f\"/kaggle/input/nfl-contact-extracted-train-frames/content/work/frames/train/{video}_{f:04d}.jpg\", 0)\n\n                    \n                    if len(tmp) == 2:  \n                        x = ((bboxes[0][0] + bboxes[1][0])/2) + ((bboxes[0][1] + bboxes[1][1])/4)\n                        y = ((bboxes[0][2] + bboxes[1][2])/2) + ((bboxes[0][3] + bboxes[1][3])/4)   \n                    else: \n                        x = bboxes[0][0] + (bboxes[0][1]/2)\n                        y = bboxes[0][2] + (bboxes[0][3]/2)\n                   \n                    if x < img_size*4:\n                            x = img_size * 4\n                    if y < img_size * 4:\n                            y = img_size * 4\n                 \n                    if img.size == 0:      \n                        img_h = np.zeros((256,256), dtype=np.float32) \n                        img_new = np.zeros((256, 256), dtype=np.float32) \n                        \n                    else:\n                        \n                        \n                        img = img[int(y)-img_size*4:int(y)+img_size*4,int(x)-img_size*4:int(x)+img_size*4].copy()\n                       \n                        \n                        img = cv2.resize(img, dsize=(256, 256), interpolation=cv2.INTER_LINEAR)\n                        \n                        img_new[:img.shape[0], :img.shape[1]] = img \n                        \n                        img_helmet = img_helmet[int(y)-img_size*4:int(y)+img_size*4,int(x)-img_size*4:int(x)+img_size*4].copy()\n                        img_helmet = cv2.resize(img_helmet, dsize=(256, 256), interpolation=cv2.INTER_CUBIC)\n\n                      \n                        \n\n\n\n                        img_h = np.zeros((256, 256), dtype=np.float32)         \n                        img_h[:img_helmet.shape[0], :img_helmet.shape[1]] = img_helmet               \n                             \n                else:\n                    \n                    img_h = np.zeros((256,256), dtype=np.float32) \n                    img_new = np.zeros((256, 256), dtype=np.float32)   \n\n\n                imgs.append(img_new)\n                imgs.append(img_h)\n                \n            imgs_arr.append(imgs)\n       \n                \n        feature = np.float32(self.feature[idx])\n         \n   \n \n        imga = np.array(imgs_arr)\n\n        b, c, h, w = imga.shape\n        imga = imga.reshape(b//b, c*sequence, h, w)\n        imga = np.squeeze(imga)\n        imga = imga.transpose(1,2,0)\n    \n        imga = self.aug(image=imga)[\"image\"]\n        label = np.float32(self.df.contact.values[idx])\n\n        \n        return imga, feature, label\n    \n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:59:50.397996Z","iopub.execute_input":"2023-07-21T09:59:50.400447Z","iopub.status.idle":"2023-07-21T09:59:50.458282Z","shell.execute_reply.started":"2023-07-21T09:59:50.400405Z","shell.execute_reply":"2023-07-21T09:59:50.457018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img, feature, label = MyDataset_LSTM(train_filtered, valid_aug)[15900]\nplt.imshow(img.permute(1,2,0)[:,:,0])\nplt.rcParams[\"figure.figsize\"] = 3,3\n\nplt.show()\nimg.shape, feature, label\n\n\n\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:59:50.463802Z","iopub.execute_input":"2023-07-21T09:59:50.466848Z","iopub.status.idle":"2023-07-21T09:59:51.316164Z","shell.execute_reply.started":"2023-07-21T09:59:50.466804Z","shell.execute_reply":"2023-07-21T09:59:51.315191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train_filtered['G_flug']))\nprint(sum(train_filtered['contact']))\nprint(sum(train_filtered['G_flug']))\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T09:59:51.373960Z","iopub.execute_input":"2023-07-21T09:59:51.375653Z","iopub.status.idle":"2023-07-21T09:59:51.394330Z","shell.execute_reply.started":"2023-07-21T09:59:51.375609Z","shell.execute_reply":"2023-07-21T09:59:51.393094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Model(nn.Module):\n    def __init__(self, num_layers = 1, input_size = 256, hidden_size = 64, seq_length = 5):\n        super(Model, self).__init__()\n        self.num_layers = num_layers #number of layers\n        self.input_size = input_size #input size\n        self.hidden_size = hidden_size #hidden state\n        self.seq_length = seq_length #sequence length\n        \n        #efficientnet_b1\n        self.backbone = timm.create_model('resnet50', pretrained=True, num_classes=128, in_chans=2)\n        \n                \n        self.mlp = nn.Sequential(\n            nn.Linear(18, 32),\n            nn.LayerNorm(32),\n            nn.ReLU(),\n            nn.Dropout(0.2),\n         \n        )\n\n        \n        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,\n                          num_layers=num_layers, batch_first=True) \n        self.fc_lstm = nn.Linear(hidden_size, 128) \n        \n        self.softmax = nn.Softmax()\n        self.fc = nn.Linear(128+32, 1)\n\n    def forward(self, img, x):\n        \n        b, c, h, w = img.shape   \n        img = img.reshape(b*(c//2),c//(c//2), h, w)\n        img = self.backbone(img)\n        img = img.reshape(b,self.seq_length, -1)\n\n\n        h_0 = Variable(torch.zeros(self.num_layers, img.size(0), self.hidden_size)).to(device) #hidden state\n        c_0 = Variable(torch.zeros(self.num_layers, img.size(0), self.hidden_size)).to(device) #internal state\n\n\n\n        output, (hn, cn) = self.lstm(img, (h_0, c_0)) #lstm with input, hidden, and internal state\n        hn = hn.view(-1, self.hidden_size) #reshaping the data for Dense layer next\n        out = self.softmax(hn)\n        out = self.fc_lstm(out)\n        #out = self.softmax(out)\n    \n        feature = self.mlp(x)\n\n        #feature = torch.transpose(feature, 0, 1)\n        y = self.fc(torch.cat([out, feature], dim=1))\n        return y","metadata":{"execution":{"iopub.status.busy":"2023-07-21T10:05:41.354528Z","iopub.execute_input":"2023-07-21T10:05:41.354970Z","iopub.status.idle":"2023-07-21T10:05:41.372019Z","shell.execute_reply.started":"2023-07-21T10:05:41.354931Z","shell.execute_reply":"2023-07-21T10:05:41.370871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n","metadata":{"execution":{"iopub.status.busy":"2023-06-14T18:26:21.904190Z","iopub.execute_input":"2023-06-14T18:26:21.904670Z","iopub.status.idle":"2023-06-14T18:26:21.919744Z","shell.execute_reply.started":"2023-06-14T18:26:21.904631Z","shell.execute_reply":"2023-06-14T18:26:21.918530Z"}}},{"cell_type":"code","source":"train_data,val_data = train_test_split(train_filtered,test_size=0.1, random_state=42,stratify = train_filtered['contact'])\ntrain_data = train_data.reset_index(drop = True)\nval_data = val_data.reset_index(drop = True)\nbatch_size = 16\n\ntrain_set = MyDataset_LSTM(train_data, train_aug)\ntrain_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2, pin_memory=True)\nval_set = MyDataset_LSTM(val_data, valid_aug)\nval_loader = DataLoader(val_set, batch_size=batch_size, shuffle=False, num_workers=2, pin_memory=True)\n\nmodel = Model()\n\nmodel= nn.DataParallel(model)\nmodel.to(device)\n#model.backbone.requires_grad_(False)\n\nmetric = BinaryAccuracy(threshold = 0.0)\nval_metric = BinaryAccuracy(threshold = 0.0)\n\n\ncriterion = nn.BCEWithLogitsLoss()\n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T10:05:43.984405Z","iopub.execute_input":"2023-07-21T10:05:43.984786Z","iopub.status.idle":"2023-07-21T10:05:44.986102Z","shell.execute_reply.started":"2023-07-21T10:05:43.984756Z","shell.execute_reply":"2023-07-21T10:05:44.985097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"optimizer = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay= 1e-6,)\nnbatch = len(train_loader)\nwarmup = 1* nbatch\nnsteps = 6 * nbatch \n\n\nscheduler = CosineLRScheduler(optimizer,warmup_t=warmup, warmup_lr_init=0.0, warmup_prefix=True,t_initial=(nsteps - warmup), lr_min=1e-6)                \n","metadata":{"execution":{"iopub.status.busy":"2023-07-21T10:05:45.648460Z","iopub.execute_input":"2023-07-21T10:05:45.649516Z","iopub.status.idle":"2023-07-21T10:05:45.660189Z","shell.execute_reply.started":"2023-07-21T10:05:45.649468Z","shell.execute_reply":"2023-07-21T10:05:45.658901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_loss = 0\nfor epoch in range(6):\n    i = 0\n    \n    model.train()\n    for batch,(img, feature, label) in tqdm(enumerate(train_loader),total = len(train_loader)):\n        \n        loss_sum = 0.0\n        n_sum = 0\n        img = img.to(device)\n        feature = feature.to(device)\n        label = label.to(device)\n        n = label.size(0)\n\n        \n        output = model(img, feature).reshape(-1)\n   \n        optimizer.zero_grad()\n        metric = BinaryAccuracy(threshold = 0.0)\n\n        metric.update(output.cpu(), label.cpu())\n        \n        \n        loss = criterion(output, label)\n        loss.backward()\n        optimizer.step()\n        scheduler.step(epoch * nbatch + batch + 1)\n\n\n        train_loss = loss.item()\n        loss_sum += n * train_loss\n        n_sum += n\n\n        nn.utils.clip_grad_norm_(model.parameters(), 1000)\n\n        if batch % 10 == 9:    # print every 2000 mini-batches\n            loss_train = loss_sum / n_sum\n            print(f'[{epoch + 1}, {batch + 1:5d}] loss: {loss_train:.3f}')\n            \n        if batch % 100 == 9:  \n            print(output)\n            print(label)\n        del label, img, feature, output\n        gc.collect()\n    \n        n = 0\n        score = 0.0\n    print('Validating model')\n    model.eval()\n    test_loss_sum = 0.0\n    test_n_sum = 0\n    test_n = 0\n    for batch,(img, feature, label) in tqdm(enumerate(val_loader),total = len(val_loader)):\n        test_n = label.size(0)\n        img = img.to(device)\n        feature = feature.to(device)\n        label = label.to(device)\n        \n        with torch.no_grad():\n            output = model(img, feature).reshape(-1)\n            loss = criterion(output, label)\n        test_loss = loss.item()\n        test_loss_sum += test_n * test_loss\n        test_n_sum += test_n\n        i += 1\n        metric = BinaryAccuracy(threshold = 0.0)\n        \n        \n        \n\n        metric.update(output.cpu(), label.cpu())\n        score = score + metric.compute()\n        n = n + 1\n\n\n        del label, img, feature, output\n        gc.collect()\n    val_loss = test_loss_sum / test_n_sum\n    val_score = score/n\n    \n    if val_loss < best_loss:\n        best_loss = val_loss\n        # Save model\n        ofilename = 'nfl_model.pytorch'\n        torch.save(model.state_dict(), ofilename)\n        print(ofilename, 'written')\n    \n    \n    print(f'Validation accuracy: {val_score}')\n    print(f'Validation loss: {val_loss}')\n    \n    ","metadata":{"execution":{"iopub.status.busy":"2023-07-21T10:05:46.000593Z","iopub.execute_input":"2023-07-21T10:05:46.000942Z","iopub.status.idle":"2023-07-21T17:44:33.841628Z","shell.execute_reply.started":"2023-07-21T10:05:46.000913Z","shell.execute_reply":"2023-07-21T17:44:33.838473Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ofilename = 'nfl_model.pytorch'\ntorch.save(model.state_dict(), ofilename)\nprint(ofilename, 'written')","metadata":{"execution":{"iopub.status.busy":"2023-07-21T17:45:17.630609Z","iopub.execute_input":"2023-07-21T17:45:17.631075Z","iopub.status.idle":"2023-07-21T17:45:17.819532Z","shell.execute_reply.started":"2023-07-21T17:45:17.631040Z","shell.execute_reply":"2023-07-21T17:45:17.818440Z"},"trusted":true},"execution_count":null,"outputs":[]}]}