{ "id": 391620, "title": "6th Place Solution (TK&penguin46 part)", "url": "/competitions/nfl-player-contact-detection/discussion/391620", "author_name": "", "post_date": "2023-03-02T01:54:32.257799100Z", "votes": 37, "comment_count": 5, "views": 0, "content": "

First, I would like to thank the hosts for organizing the competition.
\nCongratulations to the winning teams. Thanks also to @tanakar, @haqishen, and @boliu0 for competing with me.

\n

In this post, we will explain TK(@tanakar) & penguin46 part of the whole solution.
\n[update] Qishen & Bo part was posted here

\n

Overview

\n

Features created using train.csv, tracking.csv, bbox.csv and pretrained models (yolov7, mmpose) for the video data is input to xgboost along with predictions of the cnn part. CV with csv data only is 0.766, CV with pretrained models added is 0.774, and 0.7955 with the CNN part added.

\n

CNN Part

\n

2.5d CNN is used for contact prediction. In this part, @tanakar in especially made a great contribution.

\n

Input Image

\n

Grayscale images are cropped around the helmet's bbox and resized to 224x224.
\nInputs are different for player & player contacts and player & ground contacts.

\n\n

Downsampling

\n\n

Backbones

\n\n

The simple average of multiple backbones is used as the features of next xgb.

\n

Table & Pretrained Models Part

\n

Player Detection

\n

Player was detected using pretrained YOLOv7 and detected boxes were tracked using REPP, and matched with helmets. We inferred in steps instead of frames to speed up the submission time. In the matching process, bboxes were extracted in the order of the longest tracked length and greedily matched to the helmets of the players that matched best. The features created are as follows

\n\n

Pose Estimation

\n

We used mmpose to estimate the players' posture. mmpose saved inference time by using the players' bboxes that were object detected by yolov7. In addition, inference was performed only once in 2steps because this part was not significant for the computation time.

\n\n

Coordinates Gap Between Field and Camera View

\n

tracking.csv does not include a vertical component. Optimize the homographic transformation from field coordinates to camera coordinates and calculate the displacement of the helmet position. This misalignment can be attributed to the misalignment between the measured position of the field coordinates and the position of the helmet, and the vertical coordinates of the player, from which we can extract the vertical component. This idea is based on this solution by @its7171 from the last year's solution. The homography transformation was optimized using cv2.findHomography.

\n\n

Other Features

\n\n

Other Tips

\n\n

Not Worked

\n", "messages": [ { "id": "2165106", "postDate": "03/02/2023 01:54:32", "content": "

First, I would like to thank the hosts for organizing the competition.
\nCongratulations to the winning teams. Thanks also to @tanakar, @haqishen, and @boliu0 for competing with me.

\n

In this post, we will explain TK(@tanakar) & penguin46 part of the whole solution.
\n[update] Qishen & Bo part was posted here

\n

Overview

\n

Features created using train.csv, tracking.csv, bbox.csv and pretrained models (yolov7, mmpose) for the video data is input to xgboost along with predictions of the cnn part. CV with csv data only is 0.766, CV with pretrained models added is 0.774, and 0.7955 with the CNN part added.

\n

CNN Part

\n

2.5d CNN is used for contact prediction. In this part, @tanakar in especially made a great contribution.

\n

Input Image

\n

Grayscale images are cropped around the helmet's bbox and resized to 224x224.
\nInputs are different for player & player contacts and player & ground contacts.

\n\n

Downsampling

\n\n

Backbones

\n\n

The simple average of multiple backbones is used as the features of next xgb.

\n

Table & Pretrained Models Part

\n

Player Detection

\n

Player was detected using pretrained YOLOv7 and detected boxes were tracked using REPP, and matched with helmets. We inferred in steps instead of frames to speed up the submission time. In the matching process, bboxes were extracted in the order of the longest tracked length and greedily matched to the helmets of the players that matched best. The features created are as follows

\n\n

Pose Estimation

\n

We used mmpose to estimate the players' posture. mmpose saved inference time by using the players' bboxes that were object detected by yolov7. In addition, inference was performed only once in 2steps because this part was not significant for the computation time.

\n\n

Coordinates Gap Between Field and Camera View

\n

tracking.csv does not include a vertical component. Optimize the homographic transformation from field coordinates to camera coordinates and calculate the displacement of the helmet position. This misalignment can be attributed to the misalignment between the measured position of the field coordinates and the position of the helmet, and the vertical coordinates of the player, from which we can extract the vertical component. This idea is based on this solution by @its7171 from the last year's solution. The homography transformation was optimized using cv2.findHomography.

\n\n

Other Features

\n\n

Other Tips

\n\n

Not Worked

\n", "rawMarkdown": "First, I would like to thank the hosts for organizing the competition.\nCongratulations to the winning teams. Thanks also to @tanakar, @haqishen, and @boliu0 for competing with me.\n\nIn this post, we will explain TK(@tanakar) & penguin46 part of the whole solution.\n[update] Qishen & Bo part was posted [here](https://www.kaggle.com/competitions/nfl-player-contact-detection/discussion/391723)\n\n# Overview\nFeatures created using train.csv, tracking.csv, bbox.csv and pretrained models (yolov7, mmpose) for the video data is input to xgboost along with predictions of the cnn part. CV with csv data only is 0.766, CV with pretrained models added is 0.774, and 0.7955 with the CNN part added.\n\n# CNN Part\n2.5d CNN is used for contact prediction. In this part, @tanakar in especially made a great contribution.\n\n## Input Image\nGrayscale images are cropped around the helmet's bbox and resized to 224x224.\nInputs are different for player & player contacts and player & ground contacts.\n- player & player\n - crop range up to 2 times the rectangle surrounding the two bboxes\n - input bbox mask as an additional channel\n - ±2step\n- player & ground\n - crop 6x around the bbox\n - no mask\n - ±2step\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3962786%2F4a505595b255bfda550e795046c0df38%2Fcnn.png?generation=1677721348768987&alt=media)\n\n\n## Downsampling\n- player & player\n - use samples with predictions of 1st stage lightgbm greater than 0.01\n- player & ground\n - remove samples with predictions of Qishen&Bo’s stage1 less than 0.005\n\n## Backbones\n- player & player\n - resnet18d\n - swin_tiny_patch4_window7_224\n - tf_efficientnet_b0_ns\n- player & ground\n - resnet18d\n - swin_tiny_patch4_window7_224\n - efficientnetv2_rw_t\n\nThe simple average of multiple backbones is used as the features of next xgb. \n\n# Table & Pretrained Models Part\n\n## Player Detection\nPlayer was detected using pretrained YOLOv7 and detected boxes were tracked using [REPP](https://github.com/AlbertoSabater/Robust-and-efficient-post-processing-for-video-object-detection), and matched with helmets. We inferred in steps instead of frames to speed up the submission time. In the matching process, bboxes were extracted in the order of the longest tracked length and greedily matched to the helmets of the players that matched best. The features created are as follows\n- size of the bbox\n- aspect ratio of the bbox\n- distance between the center coordinates of the two players' bboxes (dx, dy, dist, dist/width)\n- IoU of the bbox of the two players\n- area of overlap with the bbox of the helmet\n- tracked length\n- matching score (sum of overlapped area)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3962786%2F929fef6a9bf240a7b4faa87bb38e8d9f%2Ftracking.png?generation=1677721445396871&alt=media)\n\n## Pose Estimation\nWe used mmpose to estimate the players' posture. mmpose saved inference time by using the players' bboxes that were object detected by yolov7. In addition, inference was performed only once in 2steps because this part was not significant for the computation time.\n- distance between head and foot, hand and foot\n- knee, hip angle\n- number of keypoints detected and sum of confiences\n- distance between head and center of helmet bbox\n\n## Coordinates Gap Between Field and Camera View\ntracking.csv does not include a vertical component. Optimize the homographic transformation from field coordinates to camera coordinates and calculate the displacement of the helmet position. This misalignment can be attributed to the misalignment between the measured position of the field coordinates and the position of the helmet, and the vertical coordinates of the player, from which we can extract the vertical component. This idea is based on [this solution](https://www.kaggle.com/competitions/nfl-health-and-safety-helmet-assignment/discussion/285112) by @its7171 from the last year's solution. The homography transformation was optimized using cv2.findHomography.\n- Coordinates (x, y) after projection\n- Number of players used to optimize the homography matrix\n- Distance between the coordinates after projection and the center coordinates of the helmet (dx, dy, dist, dx/width, dy/width dist/width)\n- Total distance between the post-projection coordinates and the center coordinates of the helmet in frames\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3962786%2F0b6d775701b80ea65fb25115692ed5e4%2Fgap.png?generation=1677721429648162&alt=media)\n\n## Other Features\n- step, max(step), step / max(step)\n- amount of movement from the start/end of the match (dx, dy, dist)\n- distance to the nth nearest player (n=1, 2, 5, 10, 15)\n- distance from the center of gravity coordinates of all players (dx, dy, dist)\n- difference, product, and sum of velocity, acceleration, distance, and angle of two players\n- aspect ratio of the bboxes of the helmets\n- distance between the center coordinates of the bboxes of the two helmets\n- IoU of the tow helmets\n- position\n- same team or not (0/1)\n- lag features for the most important features among the above (diff, shift, lag=±1, 2, 5, 10, 20)\n\n# Other Tips\n- To make the experiment more efficient, we trained lightgbm, which predicts contact faster with fewer features, and excluded samples that were obviously unnecessary from penguin&tk pipeline.\n- xgboost was about 0.002~4 better than lightgbm.\n- When using cutoff with qishen&bo's predictions, add a mask feature to shows it has been cutoff.\n\n# Not Worked\n- LSTM, Transformer, 1DCNN\n- data augmentation by swapping nfl_player_id_1, nfl_player_id_2 in table part\n- instance segmentation\n- seed averaging improved by less than 0.001", "votes": null }, { "id": "2165297", "postDate": "03/02/2023 05:10:12", "content": "

Nice Detailed Sharing and Congratulations!

", "rawMarkdown": "Nice Detailed Sharing and Congratulations!", "votes": null }, { "id": "2165496", "postDate": "03/02/2023 08:37:58", "content": "

congratulations! Excellent work. specially Not Worked part is very informative. if you don't mind can you please tell me which yolo architecture did you use? yolov7l or yolov7x. because i used yolov8x for contact detection. but when i did the submission it took me 7.5 hours to execute only this model(my idea is just efficiently crop the area with this mode). and feed this data TSM.

", "rawMarkdown": "congratulations! Excellent work. specially Not Worked part is very informative. if you don't mind can you please tell me which yolo architecture did you use? yolov7l or yolov7x. because i used yolov8x for contact detection. but when i did the submission it took me 7.5 hours to execute only this model(my idea is just efficiently crop the area with this mode). and feed this data TSM.", "votes": null }, { "id": "2165536", "postDate": "03/02/2023 09:06:18", "content": "

thanks. we used yolov7.pt and the inference should be finished within 10min when submission. we also tried yolov8x.pt, but the time was not much different. please note that we input images in step units, not frames.

", "rawMarkdown": "thanks. we used `yolov7.pt` and the inference should be finished within 10min when submission. we also tried `yolov8x.pt`, but the time was not much different. please note that we input images in step units, not frames.", "votes": null }, { "id": "2168951", "postDate": "03/04/2023 16:53:37", "content": "

Great great work! There is a lot of interesting research knowledge gained from the different approaches you took. I believe you are one of the few top teams that used pose estimation in your solution. Calculating the displacement of the helmet position vertically is a smart idea, I'd guess it was really helpful for identifying player-ground contact.

\n

Of all the additional features you created (specifically pose and helmet displacement) do you have any idea for which were the most helpful to the model?

\n

Thanks!

", "rawMarkdown": "Great great work! There is a lot of interesting research knowledge gained from the different approaches you took. I believe you are one of the few top teams that used pose estimation in your solution. Calculating the displacement of the helmet position vertically is a smart idea, I'd guess it was really helpful for identifying player-ground contact.\n\nOf all the additional features you created (specifically pose and helmet displacement) do you have any idea for which were the most helpful to the model?\n\nThanks!", "votes": null }, { "id": "2169524", "postDate": "03/05/2023 07:26:06", "content": "

Thank you! We have not analyzed the contribution of the features in the final submission, but below are the changes in cv before and after adding the key features. I hope this will be useful.

\n", "rawMarkdown": "Thank you! We have not analyzed the contribution of the features in the final submission, but below are the changes in cv before and after adding the key features. I hope this will be useful.\n\n- player detection & tracking (769 -> 775)\n- helmet displacement (774 -> 778~779)\n- pose estimation (785 -> 786)", "votes": null } ], "comments": [ { "id": 2165297, "author_name": "chg0901", "author_url": "", "post_date": "03/02/2023 05:10:12", "content": "

Nice Detailed Sharing and Congratulations!

", "votes": null, "replies": [] }, { "id": 2165496, "author_name": "nadhirhasan", "author_url": "", "post_date": "03/02/2023 08:37:58", "content": "

congratulations! Excellent work. specially Not Worked part is very informative. if you don't mind can you please tell me which yolo architecture did you use? yolov7l or yolov7x. because i used yolov8x for contact detection. but when i did the submission it took me 7.5 hours to execute only this model(my idea is just efficiently crop the area with this mode). and feed this data TSM.

", "votes": null, "replies": [ { "id": 2165536, "author_name": "ryotayoshinobu", "author_url": "", "post_date": "03/02/2023 09:06:18", "content": "

thanks. we used yolov7.pt and the inference should be finished within 10min when submission. we also tried yolov8x.pt, but the time was not much different. please note that we input images in step units, not frames.

", "votes": null, "replies": [] } ] }, { "id": 2168951, "author_name": "robikscube", "author_url": "", "post_date": "03/04/2023 16:53:37", "content": "

Great great work! There is a lot of interesting research knowledge gained from the different approaches you took. I believe you are one of the few top teams that used pose estimation in your solution. Calculating the displacement of the helmet position vertically is a smart idea, I'd guess it was really helpful for identifying player-ground contact.

\n

Of all the additional features you created (specifically pose and helmet displacement) do you have any idea for which were the most helpful to the model?

\n

Thanks!

", "votes": null, "replies": [ { "id": 2169524, "author_name": "ryotayoshinobu", "author_url": "", "post_date": "03/05/2023 07:26:06", "content": "

Thank you! We have not analyzed the contribution of the features in the final submission, but below are the changes in cv before and after adding the key features. I hope this will be useful.

\n", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2165106": "First, I would like to thank the hosts for organizing the competition.\nCongratulations to the winning teams. Thanks also to @tanakar, @haqishen, and @boliu0 for competing with me.\n\nIn this post, we will explain TK(@tanakar) & penguin46 part of the whole solution.\n[update] Qishen & Bo part was posted [here](https://www.kaggle.com/competitions/nfl-player-contact-detection/discussion/391723)\n\n# Overview\nFeatures created using train.csv, tracking.csv, bbox.csv and pretrained models (yolov7, mmpose) for the video data is input to xgboost along with predictions of the cnn part. CV with csv data only is 0.766, CV with pretrained models added is 0.774, and 0.7955 with the CNN part added.\n\n# CNN Part\n2.5d CNN is used for contact prediction. In this part, @tanakar in especially made a great contribution.\n\n## Input Image\nGrayscale images are cropped around the helmet's bbox and resized to 224x224.\nInputs are different for player & player contacts and player & ground contacts.\n- player & player\n - crop range up to 2 times the rectangle surrounding the two bboxes\n - input bbox mask as an additional channel\n - ±2step\n- player & ground\n - crop 6x around the bbox\n - no mask\n - ±2step\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3962786%2F4a505595b255bfda550e795046c0df38%2Fcnn.png?generation=1677721348768987&alt=media)\n\n\n## Downsampling\n- player & player\n - use samples with predictions of 1st stage lightgbm greater than 0.01\n- player & ground\n - remove samples with predictions of Qishen&Bo’s stage1 less than 0.005\n\n## Backbones\n- player & player\n - resnet18d\n - swin_tiny_patch4_window7_224\n - tf_efficientnet_b0_ns\n- player & ground\n - resnet18d\n - swin_tiny_patch4_window7_224\n - efficientnetv2_rw_t\n\nThe simple average of multiple backbones is used as the features of next xgb. \n\n# Table & Pretrained Models Part\n\n## Player Detection\nPlayer was detected using pretrained YOLOv7 and detected boxes were tracked using [REPP](https://github.com/AlbertoSabater/Robust-and-efficient-post-processing-for-video-object-detection), and matched with helmets. We inferred in steps instead of frames to speed up the submission time. In the matching process, bboxes were extracted in the order of the longest tracked length and greedily matched to the helmets of the players that matched best. The features created are as follows\n- size of the bbox\n- aspect ratio of the bbox\n- distance between the center coordinates of the two players' bboxes (dx, dy, dist, dist/width)\n- IoU of the bbox of the two players\n- area of overlap with the bbox of the helmet\n- tracked length\n- matching score (sum of overlapped area)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3962786%2F929fef6a9bf240a7b4faa87bb38e8d9f%2Ftracking.png?generation=1677721445396871&alt=media)\n\n## Pose Estimation\nWe used mmpose to estimate the players' posture. mmpose saved inference time by using the players' bboxes that were object detected by yolov7. In addition, inference was performed only once in 2steps because this part was not significant for the computation time.\n- distance between head and foot, hand and foot\n- knee, hip angle\n- number of keypoints detected and sum of confiences\n- distance between head and center of helmet bbox\n\n## Coordinates Gap Between Field and Camera View\ntracking.csv does not include a vertical component. Optimize the homographic transformation from field coordinates to camera coordinates and calculate the displacement of the helmet position. This misalignment can be attributed to the misalignment between the measured position of the field coordinates and the position of the helmet, and the vertical coordinates of the player, from which we can extract the vertical component. This idea is based on [this solution](https://www.kaggle.com/competitions/nfl-health-and-safety-helmet-assignment/discussion/285112) by @its7171 from the last year's solution. The homography transformation was optimized using cv2.findHomography.\n- Coordinates (x, y) after projection\n- Number of players used to optimize the homography matrix\n- Distance between the coordinates after projection and the center coordinates of the helmet (dx, dy, dist, dx/width, dy/width dist/width)\n- Total distance between the post-projection coordinates and the center coordinates of the helmet in frames\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3962786%2F0b6d775701b80ea65fb25115692ed5e4%2Fgap.png?generation=1677721429648162&alt=media)\n\n## Other Features\n- step, max(step), step / max(step)\n- amount of movement from the start/end of the match (dx, dy, dist)\n- distance to the nth nearest player (n=1, 2, 5, 10, 15)\n- distance from the center of gravity coordinates of all players (dx, dy, dist)\n- difference, product, and sum of velocity, acceleration, distance, and angle of two players\n- aspect ratio of the bboxes of the helmets\n- distance between the center coordinates of the bboxes of the two helmets\n- IoU of the tow helmets\n- position\n- same team or not (0/1)\n- lag features for the most important features among the above (diff, shift, lag=±1, 2, 5, 10, 20)\n\n# Other Tips\n- To make the experiment more efficient, we trained lightgbm, which predicts contact faster with fewer features, and excluded samples that were obviously unnecessary from penguin&tk pipeline.\n- xgboost was about 0.002~4 better than lightgbm.\n- When using cutoff with qishen&bo's predictions, add a mask feature to shows it has been cutoff.\n\n# Not Worked\n- LSTM, Transformer, 1DCNN\n- data augmentation by swapping nfl_player_id_1, nfl_player_id_2 in table part\n- instance segmentation\n- seed averaging improved by less than 0.001", "2165297": "Nice Detailed Sharing and Congratulations!", "2165496": "congratulations! Excellent work. specially Not Worked part is very informative. if you don't mind can you please tell me which yolo architecture did you use? yolov7l or yolov7x. because i used yolov8x for contact detection. but when i did the submission it took me 7.5 hours to execute only this model(my idea is just efficiently crop the area with this mode). and feed this data TSM.", "2165536": "thanks. we used `yolov7.pt` and the inference should be finished within 10min when submission. we also tried `yolov8x.pt`, but the time was not much different. please note that we input images in step units, not frames.", "2168951": "Great great work! There is a lot of interesting research knowledge gained from the different approaches you took. I believe you are one of the few top teams that used pose estimation in your solution. Calculating the displacement of the helmet position vertically is a smart idea, I'd guess it was really helpful for identifying player-ground contact.\n\nOf all the additional features you created (specifically pose and helmet displacement) do you have any idea for which were the most helpful to the model?\n\nThanks!", "2169524": "Thank you! We have not analyzed the contribution of the features in the final submission, but below are the changes in cv before and after adding the key features. I hope this will be useful.\n\n- player detection & tracking (769 -> 775)\n- helmet displacement (774 -> 778~779)\n- pose estimation (785 -> 786)" }, "source": "meta" }