\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fe95e98e9c5e78ffb22708775d0602e32%2Fimportance_p2p.png?generation=1677715572374749&alt=media\")
At the first place, thanks for hosting this wonderful competiton.
\nSince most of metadatas are already created for us, it was way more easier to join this competition than the last one.
Below is my solution writeup.
\nAt first, I devided this competition task as these two:
\nSince I considered these two tasks are completely different ones, I built two pipelines separrately depending on these tasks.
\nFor short, let's call them P2P and P2G.
In each tasks, I constructed the following 3-stage pipeline:
\ntf_efficientnet_b0)In this competition, I aimed to take a lot of experiments quickly, so I first started this competition with tabular data.
\nSince I started to use Polars from Otto competitioin, I also use this tool for this competition.
\nIt was really quick: it tooks only 17-18 sec to extract 1031 features from helmet & tracking data using Polars.
\nThen I train xgboost model. Utilizing GPU, it took for minitutes to train 5-fold models.
Notably, using only tabular data, I got public LB 0.748 (Private LB: 0.736).
\nnfl_player_id_1 and nfl_player_id_2extracting 1031 features (mostly shift & diff features) from helmet & tracking data
\n\nnfl_player_id_1 and nfl_player_id_2Since image classification takes much more time than tabular tasks, I pruned samples using 1st stage model's prediction score.
\nThe performance of pruning is as below:
Thanks to pruning, I can reduced both training & inference time.
\nIt took only 1 hour to training all 5-fold, 3-channel 2.5D-CNNs for 5 epoch for each (NVIDIA RTX-3090 Ti).
One thing to note is, thanks to 1st stage model pruning, I got much boost in TNR & NPV.
\nBecause of few samples, I observed the model easy to overfit. Avoiding this, I adopted domain-specific augmentation as well as the common image augmentations.
\nCommomn image augmentations:
\nDomain-specific image augmentations:
\nI also note that helmet marker should be added after the common augmentation processings because altering marker pattern deteriorates the merit of markers (I obserbed drop of AUC when adding helmet marker before adopting augumentations).
\nThe final stage is very similar to 1st stage, except it uses 2nd-stage's prediction scores of both Endzone and Sideline frames.
I also reused 1st stage feature because it boost both CV & LB scores.
\nThe total features are 1033 for both p2g and p2p models.
\nThe last thing I left for future is finding out the reason of discrepancy between CV & LB scores.
\nIn the 3rd stage, I also tried to extract more features using 2nd-stage prediction scores (e.g. shift features, aggregation features of players, player-player pairs etc.).
\nHowever, although the CV scores increased constantly with using more features, the LB scores decrease as adding more features. So I couldn't increase LB score any more.
\nMy best CV score is 0.833 but I got LB score 0.754 for this submission.
My current hypotheses of this issue are as below:
\nUpdates:
\nI solved this issue using less-player-duplicated fold sprit.
\nMy updated solution is available here.
Throughout the competition, I refered to the following notebooks. I appriciate the authors.
\nI don't remenber who share it first, but I adopted an idea of pruning samples within 2 yard from some notebook or discussion topic.
", "messages": [ { "id": "2165024", "postDate": "03/02/2023 00:03:51", "content": "At the first place, thanks for hosting this wonderful competiton.
\nSince most of metadatas are already created for us, it was way more easier to join this competition than the last one.
Below is my solution writeup.
\nAt first, I devided this competition task as these two:
\nSince I considered these two tasks are completely different ones, I built two pipelines separrately depending on these tasks.
\nFor short, let's call them P2P and P2G.
In each tasks, I constructed the following 3-stage pipeline:
\ntf_efficientnet_b0)In this competition, I aimed to take a lot of experiments quickly, so I first started this competition with tabular data.
\nSince I started to use Polars from Otto competitioin, I also use this tool for this competition.
\nIt was really quick: it tooks only 17-18 sec to extract 1031 features from helmet & tracking data using Polars.
\nThen I train xgboost model. Utilizing GPU, it took for minitutes to train 5-fold models.
Notably, using only tabular data, I got public LB 0.748 (Private LB: 0.736).
\nnfl_player_id_1 and nfl_player_id_2extracting 1031 features (mostly shift & diff features) from helmet & tracking data
\n\nnfl_player_id_1 and nfl_player_id_2Since image classification takes much more time than tabular tasks, I pruned samples using 1st stage model's prediction score.
\nThe performance of pruning is as below:
Thanks to pruning, I can reduced both training & inference time.
\nIt took only 1 hour to training all 5-fold, 3-channel 2.5D-CNNs for 5 epoch for each (NVIDIA RTX-3090 Ti).
One thing to note is, thanks to 1st stage model pruning, I got much boost in TNR & NPV.
\nBecause of few samples, I observed the model easy to overfit. Avoiding this, I adopted domain-specific augmentation as well as the common image augmentations.
\nCommomn image augmentations:
\nDomain-specific image augmentations:
\nI also note that helmet marker should be added after the common augmentation processings because altering marker pattern deteriorates the merit of markers (I obserbed drop of AUC when adding helmet marker before adopting augumentations).
\nThe final stage is very similar to 1st stage, except it uses 2nd-stage's prediction scores of both Endzone and Sideline frames.
I also reused 1st stage feature because it boost both CV & LB scores.
\nThe total features are 1033 for both p2g and p2p models.
\nThe last thing I left for future is finding out the reason of discrepancy between CV & LB scores.
\nIn the 3rd stage, I also tried to extract more features using 2nd-stage prediction scores (e.g. shift features, aggregation features of players, player-player pairs etc.).
\nHowever, although the CV scores increased constantly with using more features, the LB scores decrease as adding more features. So I couldn't increase LB score any more.
\nMy best CV score is 0.833 but I got LB score 0.754 for this submission.
My current hypotheses of this issue are as below:
\nUpdates:
\nI solved this issue using less-player-duplicated fold sprit.
\nMy updated solution is available here.
Throughout the competition, I refered to the following notebooks. I appriciate the authors.
\nI don't remenber who share it first, but I adopted an idea of pruning samples within 2 yard from some notebook or discussion topic.
", "rawMarkdown": "At the first place, thanks for hosting this wonderful competiton.\nSince most of metadatas are already created for us, it was way more easier to join this competition than the last one.\n\nBelow is my solution writeup.\n\n## General description\n\nAt first, I devided this competition task as these two:\n\n- Player-Player contact detection (P2P)\n- Player-Ground contact detection (P2G)\n\nSince I considered these two tasks are completely different ones, I built two pipelines separrately depending on these tasks.\nFor short, let's call them P2P and P2G.\n\nIn each tasks, I constructed the following 3-stage pipeline:\n\n- 1st stage: Candidate Extractor - XGBoost\n- 2nd stage: Image Feature Extractor - small CNN (`tf_efficientnet_b0`)\n- 3rd stage: Binary Classifier - XGBoost\n\n## 1st stage: Candidate Extractor - Public LB: 0.748; Private LB: 0.736\n\nIn this competition, I aimed to take a lot of experiments quickly, so I first started this competition with tabular data.\nSince I started to use Polars from Otto competitioin, I also use this tool for this competition.\nIt was really quick: it tooks only 17-18 sec to extract 1031 features from helmet & tracking data using Polars.\nThen I train xgboost model. Utilizing GPU, it took for minitutes to train 5-fold models.\n\nNotably, using only tabular data, I got **public LB 0.748 (Private LB: 0.736)**.\n\n### Preprocessing\n\n- restoring all player-player matching pairs by swapping `nfl_player_id_1` and `nfl_player_id_2`\n- for p2p pipeline, pruning player pairs within 2 yard\n\n### Feature extraction\n\nextracting **1031** features (mostly shift & diff features) from helmet & tracking data\n\n[feature design list](https://docs.google.com/spreadsheets/d/1u96aSvD1r7jhCsPhlxtfVcK5pbNx5mrNZkvFox_kW5E/edit)\n\n### Postprocess\n\n- for p2p, TTA by swapping `nfl_player_id_1` and `nfl_player_id_2`\n- taking moving average of prediction score through time series\n\n## 2nd stage: Image Feature Extractor\n\n### Pruning samples using 1st stage model\n\nSince image classification takes much more time than tabular tasks, I pruned samples using 1st stage model's prediction score.\nThe performance of pruning is as below:\n\n- p2p: keeping recall 0.992, reduced ~50% of samples\n- p2g: keeping recall 0.992, reduced ~75% of samples\n\nThanks to pruning, I can reduced both training & inference time.\nIt took only 1 hour to training all 5-fold, 3-channel 2.5D-CNNs for 5 epoch for each (NVIDIA RTX-3090 Ti).\n\nOne thing to note is, thanks to 1st stage model pruning, I got much boost in TNR & NPV.\n\n### Image Augmentation\n\nBecause of few samples, I observed the model easy to overfit. Avoiding this, I adopted domain-specific augmentation as well as the common image augmentations.\n\nCommomn image augmentations:\n- horizontal flip, median blur, cutout, affine transform etc. Note that I don't use rotations and translations in case it might cause domain-shift.\n\nDomain-specific image augmentations:\n- Cropping interested 240x240 pixels of regions around helmet. Since the size of the player in the frame differs frame to frame, I adopted cropping based on the helmet size. I found 6x helmet size is the best to identify region of interest. Note that I set center of cropped region a bit below the center of helmet bounding box to capture entire body of the players.\n- Adding helmet marker by uniform noize for each helmet bounding boxes to highlight player pairs of interesk.\n- Adding +-3 frame of shift which expect to simulate actual sensor delay etc.\n\nI also note that helmet marker should be added after the common augmentation processings because altering marker pattern deteriorates the merit of markers (I obserbed drop of AUC when adding helmet marker before adopting augumentations).\n\n## 3rd stage: Binary Classifier - Public LB: 0.771 (+0.023); Private LB 0.767 (+0.031)\n\nThe final stage is very similar to 1st stage, except it uses 2nd-stage's prediction scores of both `Endzone` and `Sideline` frames.\n\nI also reused 1st stage feature because it boost both CV & LB scores.\n\nThe total features are **1033** for both p2g and p2p models.\n\n## Future Works...\n\nThe last thing I left for future is finding out the reason of discrepancy between CV & LB scores.\n\nIn the 3rd stage, I also tried to extract more features using 2nd-stage prediction scores (e.g. shift features, aggregation features of players, player-player pairs etc.).\nHowever, although the CV scores increased constantly with using more features, the LB scores decrease as adding more features. So I couldn't increase LB score any more.\nMy best CV score is **0.833** but I got LB score **0.754** for this submission.\n\nMy current hypotheses of this issue are as below:\n\n- Train & Test samples are sampled by different data source (e.g. samples of non-overlapping teams or players). Beause of this, I only observed CVs that was overfitted to the specific players or teams.\n- My pipeline has some bug.\n\n---\n\n**Updates:**\n\nI solved this issue using less-player-duplicated fold sprit.\nMy updated solution is available [here](https://www.kaggle.com/competitions/nfl-player-contact-detection/discussion/394302).\n\n## Other attempts that didn't worked\n\n- increasing channels of 2.5D CNN: although the AUC in 2nd stage is best for 5-channeled CNN, the final result of 3rd stage is same as 3-channeled models.\n- pseudo labels using previous contest (NFL-2)\n- I don't even remember...\n\n## Aknowledgement\n\nThroughout the competition, I refered to the following notebooks. I appriciate the authors.\n\n- [COLUM2131's XGBoost starter notebook](https://www.kaggle.com/code/columbia2131/nfl-player-contact-detection-simple-xgb-baseline) - as a XGBoost's baseline notebook\n- [ARUTEMA47's 1D-CNN notebook](https://www.kaggle.com/code/kyoshioka47/train-dfl-effnet-1dcnn-cv0-77/notebook) - as a baseline of 2D/2.5D-CNN pipeline\n- [ARUTEMA47's notebook in NFL-1](https://www.kaggle.com/code/kyoshioka47/team-tara-submission-2-stage-3d/notebook) - how to generate video frame images from movie files\n- [ZZY's 2.5D-CNN notebook](https://www.kaggle.com/code/zzy990106/nfl-2-5d-cnn-baseline-inference) - idea of cropping & stacking local picture\n\nI don't remenber who share it first, but I adopted an idea of pruning samples within 2 yard from some notebook or discussion topic.", "votes": null }, { "id": "2165025", "postDate": "03/02/2023 00:07:01", "content": "Here is the features of Top50 importances in the 1st stage.
\nP2P:
\n
P2G:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fce7fd44ef6e681abeaff463a80957e4f%2Fimportance_p2g.png?generation=1677715579910986&alt=media\")
I know my writeup is imperfect. Please ask question if you have ones.
", "rawMarkdown": "I know my writeup is imperfect. Please ask question if you have ones.", "votes": null }, { "id": "2165065", "postDate": "03/02/2023 00:55:51", "content": "Picture of augmented image samples:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F2220b2f5dd870527bd6a22d4c185ef9a%2FScreen%20Shot%202023-03-02%20at%209.54.52.png?generation=1677718505656313&alt=media\")
Here is the features of Top50 importances in the 1st stage.
\nP2P:
\n
P2G:
\n
I know my writeup is imperfect. Please ask question if you have ones.
", "votes": null, "replies": [] }, { "id": 2165065, "author_name": "tatamikenn", "author_url": "", "post_date": "03/02/2023 00:55:51", "content": "Picture of augmented image samples:
\n