I'm seeing extreme reductions in matthews correlation between the train/val sets during training and the test set used for public score. I've narrowed down the performance issue to the tracking files, and get much better generalization exclusively using the helmet files. Here are some notebooks that help show what I'm running into. Extra pair of eyes would be great:
\nBaseline where I define predicted contact as any player1/player2 pair within 1 yard of each other. Using the tracking x,y coordinates to calculate the distance of players gives me a validation matthews correlation around 0.5, but then a terrible public score. Utilizing the helmet data to calculate distance gets me a much better public score (0.238), and much more aligned with the matthews correlation found on the training data: https://www.kaggle.com/code/ryancaldwell/within-x-yards-distance-as-contact-baseline/notebook
Xgboost model utilizing tracking data for player1/player2 distances and step_pct as features. This model shows a matthews correlation around 0.55 on the validation fold during training, but then scores 0.04 on the public leaderboard: https://www.kaggle.com/code/ryancaldwell/xgboost-model?scriptVersionId=113927847
What's interesting is that I created a separate model with only features from the helmet file, which gave a leaderboard score of 0.311 (version 1), and then it dropped way down to 0.035 (version 2) when adding in the player distance and x,y positions from the tracking file: https://www.kaggle.com/code/ryancaldwell/helmet-bounding-box-iou
So it appears maybe I'm doing something wrong when creating the distance features when joining up the test tracking file with submission. That's where I think it could be going wrong.
", "messages": [ { "id": "2066451", "postDate": "12/15/2022 18:00:00", "content": "I'm seeing extreme reductions in matthews correlation between the train/val sets during training and the test set used for public score. I've narrowed down the performance issue to the tracking files, and get much better generalization exclusively using the helmet files. Here are some notebooks that help show what I'm running into. Extra pair of eyes would be great:
\nBaseline where I define predicted contact as any player1/player2 pair within 1 yard of each other. Using the tracking x,y coordinates to calculate the distance of players gives me a validation matthews correlation around 0.5, but then a terrible public score. Utilizing the helmet data to calculate distance gets me a much better public score (0.238), and much more aligned with the matthews correlation found on the training data: https://www.kaggle.com/code/ryancaldwell/within-x-yards-distance-as-contact-baseline/notebook
Xgboost model utilizing tracking data for player1/player2 distances and step_pct as features. This model shows a matthews correlation around 0.55 on the validation fold during training, but then scores 0.04 on the public leaderboard: https://www.kaggle.com/code/ryancaldwell/xgboost-model?scriptVersionId=113927847
What's interesting is that I created a separate model with only features from the helmet file, which gave a leaderboard score of 0.311 (version 1), and then it dropped way down to 0.035 (version 2) when adding in the player distance and x,y positions from the tracking file: https://www.kaggle.com/code/ryancaldwell/helmet-bounding-box-iou
So it appears maybe I'm doing something wrong when creating the distance features when joining up the test tracking file with submission. That's where I think it could be going wrong.
", "rawMarkdown": "I'm seeing extreme reductions in matthews correlation between the train/val sets during training and the test set used for public score. I've narrowed down the performance issue to the tracking files, and get much better generalization exclusively using the helmet files. Here are some notebooks that help show what I'm running into. Extra pair of eyes would be great:\n\n1. Baseline where I define predicted contact as any player1/player2 pair within 1 yard of each other. Using the tracking x,y coordinates to calculate the distance of players gives me a validation matthews correlation around 0.5, but then a terrible public score. Utilizing the helmet data to calculate distance gets me a much better public score (0.238), and much more aligned with the matthews correlation found on the training data: https://www.kaggle.com/code/ryancaldwell/within-x-yards-distance-as-contact-baseline/notebook\n\n2. Xgboost model utilizing tracking data for player1/player2 distances and step_pct as features. This model shows a matthews correlation around 0.55 on the validation fold during training, but then scores 0.04 on the public leaderboard: https://www.kaggle.com/code/ryancaldwell/xgboost-model?scriptVersionId=113927847\n\n3. What's interesting is that I created a separate model with only features from the helmet file, which gave a leaderboard score of 0.311 (version 1), and then it dropped way down to 0.035 (version 2) when adding in the player distance and x,y positions from the tracking file: https://www.kaggle.com/code/ryancaldwell/helmet-bounding-box-iou\n\nSo it appears maybe I'm doing something wrong when creating the distance features when joining up the test tracking file with submission. That's where I think it could be going wrong.", "votes": null }, { "id": "2068648", "postDate": "12/18/2022 07:40:23", "content": "Sometimes messing the order of things screw submissions, be careful with that.
", "rawMarkdown": "Sometimes messing the order of things screw submissions, be careful with that.", "votes": null }, { "id": "2068826", "postDate": "12/18/2022 10:46:25", "content": "What is step_pct?
", "rawMarkdown": "What is step_pct?", "votes": null }, { "id": "2068955", "postDate": "12/18/2022 12:55:36", "content": "It just represents the percentage of the way through a given play is. So, if a play has 10 steps and is currently on step 1, then its step_pct would be 10%. It works as a key with the helmet file, but I don't think it does with the tracking data.
", "rawMarkdown": "It just represents the percentage of the way through a given play is. So, if a play has 10 steps and is currently on step 1, then its step_pct would be 10%. It works as a key with the helmet file, but I don't think it does with the tracking data.", "votes": null }, { "id": "2068961", "postDate": "12/18/2022 13:00:35", "content": "Yeah that makes sense, but I'm not sorting anything. I'm just joining on common keys where the left side of the join is the submission file. I'll do some digging on this, but it doesn't seem like the order of the predictions or submission rows should be changing.
", "rawMarkdown": "Yeah that makes sense, but I'm not sorting anything. I'm just joining on common keys where the left side of the join is the submission file. I'll do some digging on this, but it doesn't seem like the order of the predictions or submission rows should be changing.", "votes": null } ], "comments": [ { "id": 2068648, "author_name": "carloshuertas", "author_url": "", "post_date": "12/18/2022 07:40:23", "content": "Sometimes messing the order of things screw submissions, be careful with that.
", "votes": null, "replies": [ { "id": 2068961, "author_name": "ryancaldwell", "author_url": "", "post_date": "12/18/2022 13:00:35", "content": "Yeah that makes sense, but I'm not sorting anything. I'm just joining on common keys where the left side of the join is the submission file. I'll do some digging on this, but it doesn't seem like the order of the predictions or submission rows should be changing.
", "votes": null, "replies": [] } ] }, { "id": 2068826, "author_name": "louisbunuel", "author_url": "", "post_date": "12/18/2022 10:46:25", "content": "What is step_pct?
", "votes": null, "replies": [ { "id": 2068955, "author_name": "ryancaldwell", "author_url": "", "post_date": "12/18/2022 12:55:36", "content": "It just represents the percentage of the way through a given play is. So, if a play has 10 steps and is currently on step 1, then its step_pct would be 10%. It works as a key with the helmet file, but I don't think it does with the tracking data.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2066451": "I'm seeing extreme reductions in matthews correlation between the train/val sets during training and the test set used for public score. I've narrowed down the performance issue to the tracking files, and get much better generalization exclusively using the helmet files. Here are some notebooks that help show what I'm running into. Extra pair of eyes would be great:\n\n1. Baseline where I define predicted contact as any player1/player2 pair within 1 yard of each other. Using the tracking x,y coordinates to calculate the distance of players gives me a validation matthews correlation around 0.5, but then a terrible public score. Utilizing the helmet data to calculate distance gets me a much better public score (0.238), and much more aligned with the matthews correlation found on the training data: https://www.kaggle.com/code/ryancaldwell/within-x-yards-distance-as-contact-baseline/notebook\n\n2. Xgboost model utilizing tracking data for player1/player2 distances and step_pct as features. This model shows a matthews correlation around 0.55 on the validation fold during training, but then scores 0.04 on the public leaderboard: https://www.kaggle.com/code/ryancaldwell/xgboost-model?scriptVersionId=113927847\n\n3. What's interesting is that I created a separate model with only features from the helmet file, which gave a leaderboard score of 0.311 (version 1), and then it dropped way down to 0.035 (version 2) when adding in the player distance and x,y positions from the tracking file: https://www.kaggle.com/code/ryancaldwell/helmet-bounding-box-iou\n\nSo it appears maybe I'm doing something wrong when creating the distance features when joining up the test tracking file with submission. That's where I think it could be going wrong.", "2068648": "Sometimes messing the order of things screw submissions, be careful with that.", "2068826": "What is step_pct?", "2068955": "It just represents the percentage of the way through a given play is. So, if a play has 10 steps and is currently on step 1, then its step_pct would be 10%. It works as a key with the helmet file, but I don't think it does with the tracking data.", "2068961": "Yeah that makes sense, but I'm not sorting anything. I'm just joining on common keys where the left side of the join is the submission file. I'll do some digging on this, but it doesn't seem like the order of the predictions or submission rows should be changing." }, "source": "meta" }