According to all of our submissions, we found that submissions which only use tabular data for training have huge gap between LB/PB(PB is less than LB around 0.015~0.02). But the submissions which mainly use video data have small gap. What about yours?
", "messages": [ { "id": "2165110", "postDate": "03/02/2023 02:02:34", "content": "According to all of our submissions, we found that submissions which only use tabular data for training have huge gap between LB/PB(PB is less than LB around 0.015~0.02). But the submissions which mainly use video data have small gap. What about yours?
", "rawMarkdown": "According to all of our submissions, we found that submissions which only use tabular data for training have huge gap between LB/PB(PB is less than LB around 0.015~0.02). But the submissions which mainly use video data have small gap. What about yours?", "votes": null }, { "id": "2165137", "postDate": "03/02/2023 02:34:02", "content": "I used only tabular data for training and got LB:0.730, but PB is 0.701.
", "rawMarkdown": "I used only tabular data for training and got LB:0.730, but PB is 0.701.", "votes": null }, { "id": "2165141", "postDate": "03/02/2023 02:47:59", "content": "Our Best single model also tabular data CV 750 and LB 740, gives a little shake, but not huge.
", "rawMarkdown": "Our Best single model also tabular data CV 750 and LB 740, gives a little shake, but not huge.", "votes": null }, { "id": "2165144", "postDate": "03/02/2023 02:49:14", "content": "Huge gap, too. And PB is higher than LB in some of our submission which only use video data.😂
", "rawMarkdown": "Huge gap, too. And PB is higher than LB in some of our submission which only use video data.😂", "votes": null }, { "id": "2165197", "postDate": "03/02/2023 03:48:26", "content": "We found that there is similar gap in our tabular-only submission. CV: 0.729, Public: 0.737 and Private: 0.708.
", "rawMarkdown": "We found that there is similar gap in our tabular-only submission. CV: 0.729, Public: 0.737 and Private: 0.708.", "votes": null }, { "id": "2165672", "postDate": "03/02/2023 10:58:20", "content": "Our xgb model cv 0.726，public 0.720，private 0.697
", "rawMarkdown": "Our xgb model cv 0.726，public 0.720，private 0.697", "votes": null }, { "id": "2165675", "postDate": "03/02/2023 11:03:21", "content": "Could someone with a strong tabular model please post local scores separately for play-to-player and player-to-ground contacts? Or maybe sub it separately?
\nI have some suspicion that there could be a different ratio (or difficulty) of these two types of contacts in public and private. But maybe not.
\nIssue is that test data is quite small, so more shake versus local is expected.
", "rawMarkdown": "Could someone with a strong tabular model please post local scores separately for play-to-player and player-to-ground contacts? Or maybe sub it separately?\n\nI have some suspicion that there could be a different ratio (or difficulty) of these two types of contacts in public and private. But maybe not.\n\nIssue is that test data is quite small, so more shake versus local is expected.", "votes": null }, { "id": "2165678", "postDate": "03/02/2023 11:08:46", "content": "I tried training 2 7-folds XGB models, one for PG, one for PP. with 200k PP data and 400k PG data. The final CV score is same (within 0.001 cv score, both are 0.745)
", "rawMarkdown": "I tried training 2 7-folds XGB models, one for PG, one for PP. with 200k PP data and 400k PG data. The final CV score is same (within 0.001 cv score, both are 0.745)", "votes": null }, { "id": "2165682", "postDate": "03/02/2023 11:13:45", "content": "Each time I improved my local CV score, it always happens in PP group, PG is obviously have a higher performance than PP.
\nBecause I add lots of features between player1 and player2, such as AB corralation, distance etc.
I shared an analysis tool, you could take a look :)
\nhttps://www.kaggle.com/dwchen/oof-analysis
But in your final tabular model, if you just filter out PP and PG, are they on-par?
", "rawMarkdown": "But in your final tabular model, if you just filter out PP and PG, are they on-par?", "votes": null }, { "id": "2165687", "postDate": "03/02/2023 11:18:53", "content": "I guess that sensor data on pb maybe have some problem, such like two players' sensor data makes tabular model always predict 1 on this game play. But video model is more robust.
", "rawMarkdown": "I guess that sensor data on pb maybe have some problem, such like two players' sensor data makes tabular model always predict 1 on this game play. But video model is more robust.", "votes": null }, { "id": "2165689", "postDate": "03/02/2023 11:26:23", "content": "final we put them together. with one column: PG / PP with same team / PP with different team.
\nI guess the main reason is the player2 relation features in PG group all NAN, and Tree method can divide it simple?
", "rawMarkdown": "final we put them together. with one column: PG / PP with same team / PP with different team.\n\nI guess the main reason is the player2 relation features in PG group all NAN, and Tree method can divide it simple?", "votes": null }, { "id": "2165691", "postDate": "03/02/2023 11:28:00", "content": "But overall, at least for us, PG is easier to predict. So I am curious if the score is the same for both groups in tabular model, or also PG is higher.
", "rawMarkdown": "But overall, at least for us, PG is easier to predict. So I am curious if the score is the same for both groups in tabular model, or also PG is higher.", "votes": null }, { "id": "2165692", "postDate": "03/02/2023 11:28:14", "content": "maybe we should add a low pass filter on sensors data.
\nhhhh
", "rawMarkdown": "maybe we should add a low pass filter on sensors data. \n\nhhhh", "votes": null }, { "id": "2165702", "postDate": "03/02/2023 11:34:55", "content": "From experience I would say that this tracking data is quite accurate and consistent. Are you using lots of box features? It could rather be an issue.
", "rawMarkdown": "From experience I would say that this tracking data is quite accurate and consistent. Are you using lots of box features? It could rather be an issue.", "votes": null }, { "id": "2165703", "postDate": "03/02/2023 11:34:58", "content": "PP is much easier to predict in our tabular model.
\nCV MCC in distance < 3:
\nPG=0.62898, PP=0.76083
How do you filter by distance for PG? I think fair comparison is not filtering for both. But it looks like a huge difference.
", "rawMarkdown": "How do you filter by distance for PG? I think fair comparison is not filtering for both. But it looks like a huge difference.", "votes": null }, { "id": "2165707", "postDate": "03/02/2023 11:38:54", "content": "I plot the difference between ground truth and prediction, and find our model always do FP prediction.
\nThen I check some events on video, FP happen at two players close, FN happen at player1 pull player2.
For PG condition, it don't mind Player2 is closer or not.
", "rawMarkdown": "I plot the difference between ground truth and prediction, and find our model always do FP prediction.\nThen I check some events on video, FP happen at two players close, FN happen at player1 pull player2.\n\nFor PG condition, it don't mind Player2 is closer or not.", "votes": null }, { "id": "2165715", "postDate": "03/02/2023 11:51:07", "content": "Oops, we predict all instances for PG. distance < 3 is only for PP.
", "rawMarkdown": "Oops, we predict all instances for PG. distance < 3 is only for PP.", "votes": null }, { "id": "2165717", "postDate": "03/02/2023 11:56:56", "content": "I have different result: distance < 2:
\nPP AUC = 0.96361 and PG AUC = 0.97443
Yes, I use lots of box features. Hope you guys find why shake happended.
", "rawMarkdown": "Yes, I use lots of box features. Hope you guys find why shake happended.", "votes": null }, { "id": "2165749", "postDate": "03/02/2023 12:32:54", "content": "I think there may be other unknown features and some deviation in the tracking data and helmet data to some extent, resulting in 'shake up'.
", "rawMarkdown": "I think there may be other unknown features and some deviation in the tracking data and helmet data to some extent, resulting in 'shake up'.", "votes": null } ], "comments": [ { "id": 2165137, "author_name": "rin120", "author_url": "", "post_date": "03/02/2023 02:34:02", "content": "I used only tabular data for training and got LB:0.730, but PB is 0.701.
", "votes": null, "replies": [ { "id": 2165144, "author_name": "chenxin1991", "author_url": "", "post_date": "03/02/2023 02:49:14", "content": "Huge gap, too. And PB is higher than LB in some of our submission which only use video data.😂
", "votes": null, "replies": [] } ] }, { "id": 2165141, "author_name": "dwchen", "author_url": "", "post_date": "03/02/2023 02:47:59", "content": "Our Best single model also tabular data CV 750 and LB 740, gives a little shake, but not huge.
", "votes": null, "replies": [] }, { "id": 2165197, "author_name": "nyanpn", "author_url": "", "post_date": "03/02/2023 03:48:26", "content": "We found that there is similar gap in our tabular-only submission. CV: 0.729, Public: 0.737 and Private: 0.708.
", "votes": null, "replies": [] }, { "id": 2165672, "author_name": "cjzccc", "author_url": "", "post_date": "03/02/2023 10:58:20", "content": "Our xgb model cv 0.726，public 0.720，private 0.697
", "votes": null, "replies": [] }, { "id": 2165675, "author_name": "philippsinger", "author_url": "", "post_date": "03/02/2023 11:03:21", "content": "Could someone with a strong tabular model please post local scores separately for play-to-player and player-to-ground contacts? Or maybe sub it separately?
\nI have some suspicion that there could be a different ratio (or difficulty) of these two types of contacts in public and private. But maybe not.
\nIssue is that test data is quite small, so more shake versus local is expected.
", "votes": null, "replies": [ { "id": 2165678, "author_name": "dwchen", "author_url": "", "post_date": "03/02/2023 11:08:46", "content": "I tried training 2 7-folds XGB models, one for PG, one for PP. with 200k PP data and 400k PG data. The final CV score is same (within 0.001 cv score, both are 0.745)
", "votes": null, "replies": [ { "id": 2165682, "author_name": "dwchen", "author_url": "", "post_date": "03/02/2023 11:13:45", "content": "Each time I improved my local CV score, it always happens in PP group, PG is obviously have a higher performance than PP.
\nBecause I add lots of features between player1 and player2, such as AB corralation, distance etc.
I shared an analysis tool, you could take a look :)
\nhttps://www.kaggle.com/dwchen/oof-analysis
But in your final tabular model, if you just filter out PP and PG, are they on-par?
", "votes": null, "replies": [ { "id": 2165689, "author_name": "dwchen", "author_url": "", "post_date": "03/02/2023 11:26:23", "content": "final we put them together. with one column: PG / PP with same team / PP with different team.
\nI guess the main reason is the player2 relation features in PG group all NAN, and Tree method can divide it simple?
", "votes": null, "replies": [ { "id": 2165691, "author_name": "philippsinger", "author_url": "", "post_date": "03/02/2023 11:28:00", "content": "But overall, at least for us, PG is easier to predict. So I am curious if the score is the same for both groups in tabular model, or also PG is higher.
", "votes": null, "replies": [ { "id": 2165703, "author_name": "nyanpn", "author_url": "", "post_date": "03/02/2023 11:34:58", "content": "PP is much easier to predict in our tabular model.
\nCV MCC in distance < 3:
\nPG=0.62898, PP=0.76083
How do you filter by distance for PG? I think fair comparison is not filtering for both. But it looks like a huge difference.
", "votes": null, "replies": [ { "id": 2165715, "author_name": "nyanpn", "author_url": "", "post_date": "03/02/2023 11:51:07", "content": "Oops, we predict all instances for PG. distance < 3 is only for PP.
", "votes": null, "replies": [] } ] }, { "id": 2165717, "author_name": "dwchen", "author_url": "", "post_date": "03/02/2023 11:56:56", "content": "I have different result: distance < 2:
\nPP AUC = 0.96361 and PG AUC = 0.97443
I plot the difference between ground truth and prediction, and find our model always do FP prediction.
\nThen I check some events on video, FP happen at two players close, FN happen at player1 pull player2.
For PG condition, it don't mind Player2 is closer or not.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 2165687, "author_name": "hydantess", "author_url": "", "post_date": "03/02/2023 11:18:53", "content": "I guess that sensor data on pb maybe have some problem, such like two players' sensor data makes tabular model always predict 1 on this game play. But video model is more robust.
", "votes": null, "replies": [ { "id": 2165692, "author_name": "dwchen", "author_url": "", "post_date": "03/02/2023 11:28:14", "content": "maybe we should add a low pass filter on sensors data.
\nhhhh
", "votes": null, "replies": [] }, { "id": 2165702, "author_name": "philippsinger", "author_url": "", "post_date": "03/02/2023 11:34:55", "content": "From experience I would say that this tracking data is quite accurate and consistent. Are you using lots of box features? It could rather be an issue.
", "votes": null, "replies": [ { "id": 2165738, "author_name": "hydantess", "author_url": "", "post_date": "03/02/2023 12:22:46", "content": "Yes, I use lots of box features. Hope you guys find why shake happended.
", "votes": null, "replies": [] } ] }, { "id": 2165749, "author_name": "cjzccc", "author_url": "", "post_date": "03/02/2023 12:32:54", "content": "I think there may be other unknown features and some deviation in the tracking data and helmet data to some extent, resulting in 'shake up'.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2165110": "According to all of our submissions, we found that submissions which only use tabular data for training have huge gap between LB/PB(PB is less than LB around 0.015~0.02). But the submissions which mainly use video data have small gap. What about yours?", "2165137": "I used only tabular data for training and got LB:0.730, but PB is 0.701.", "2165141": "Our Best single model also tabular data CV 750 and LB 740, gives a little shake, but not huge.", "2165144": "Huge gap, too. And PB is higher than LB in some of our submission which only use video data.😂", "2165197": "We found that there is similar gap in our tabular-only submission. CV: 0.729, Public: 0.737 and Private: 0.708.", "2165672": "Our xgb model cv 0.726，public 0.720，private 0.697", "2165675": "Could someone with a strong tabular model please post local scores separately for play-to-player and player-to-ground contacts? Or maybe sub it separately?\n\nI have some suspicion that there could be a different ratio (or difficulty) of these two types of contacts in public and private. But maybe not.\n\nIssue is that test data is quite small, so more shake versus local is expected.", "2165678": "I tried training 2 7-folds XGB models, one for PG, one for PP. with 200k PP data and 400k PG data. The final CV score is same (within 0.001 cv score, both are 0.745)", "2165682": "Each time I improved my local CV score, it always happens in PP group, PG is obviously have a higher performance than PP.\nBecause I add lots of features between player1 and player2, such as AB corralation, distance etc.\n\nI shared an analysis tool, you could take a look :)\nhttps://www.kaggle.com/dwchen/oof-analysis", "2165684": "But in your final tabular model, if you just filter out PP and PG, are they on-par?", "2165687": "I guess that sensor data on pb maybe have some problem, such like two players' sensor data makes tabular model always predict 1 on this game play. But video model is more robust.", "2165689": "final we put them together. with one column: PG / PP with same team / PP with different team.\n\nI guess the main reason is the player2 relation features in PG group all NAN, and Tree method can divide it simple?", "2165691": "But overall, at least for us, PG is easier to predict. So I am curious if the score is the same for both groups in tabular model, or also PG is higher.", "2165692": "maybe we should add a low pass filter on sensors data. \n\nhhhh", "2165702": "From experience I would say that this tracking data is quite accurate and consistent. Are you using lots of box features? It could rather be an issue.", "2165703": "PP is much easier to predict in our tabular model.\n\nCV MCC in distance < 3: \nPG=0.62898, PP=0.76083", "2165705": "How do you filter by distance for PG? I think fair comparison is not filtering for both. But it looks like a huge difference.", "2165707": "I plot the difference between ground truth and prediction, and find our model always do FP prediction.\nThen I check some events on video, FP happen at two players close, FN happen at player1 pull player2.\n\nFor PG condition, it don't mind Player2 is closer or not.", "2165715": "Oops, we predict all instances for PG. distance < 3 is only for PP.", "2165717": "I have different result: distance < 2:\nPP AUC = 0.96361 and PG AUC = 0.97443", "2165738": "Yes, I use lots of box features. Hope you guys find why shake happended.", "2165749": "I think there may be other unknown features and some deviation in the tracking data and helmet data to some extent, resulting in 'shake up'." }, "source": "meta" }