…and we sadly missed it!
\nIn this discussion, I’m sharing about my data leak preprocessing in the competition. The raw data can be downloaded from Field Day Lab's Open Game Data website. Our full solution has also been detailed out here.
\nBefore beginning, I would like to thank the host and Kaggle for this competition. Despite its issues and challenges, I'm sure all parties have given their best and this has definitely been a memorable learning opportunity.
\nSecond, I'm grateful to be part of my team with @chaudharypriyanshu , @shinomoriaoshi and @martasprg. Your knowledge and hardwork inspired me a lot. It was fortunate that even when we missed our best shots, we still managed to achieve a shake-up and snatch a gold medal, all thanks to our talented and persisting team members.
\nFinally, thank you all who participated and shared your experience/insights. This community has been and will always be the best part of Kaggle!
\nNow let's begin!
\nI found out about this leaked dataset quite late (about 3 weeks before the deadline) when seeing the thread by @glipko - thank you! I then came across this dataset by @rsakata. Thank you both so much for sharing!
\nI realized that, between these two datasets, there were some fils that exist in one but not the other. When combined together, their total distribution is below
\n| \n | \n |
|---|---|
| Total sessions | \n263,881 | \n
| Overlapping sessions with original train set | \n14,505 | \n
| Sessions completing only level 0-4 | \n57,091 | \n
| Sessions completing only level 0-12 | \n21,623 | \n
| Sessions completing all levels (0-22) | \n11,607 | \n
(the rest are unusable & non-overlapping sessions)
\nI spent the last 3 weeks (until the very last hours) of the competition trying to preprocess the raw leaked data. Assuming that processing stays the same between from the original training to the hidden test set, I used the 14k overlapping sessions as a benchmark to compare my processed version with the original data. Due to some discrepancies from within the raw data itself, we could not obtain complete alignment. We got the similarity to 99.99% for labels and 99.5% for input, and stopped there.
\nHere is the final version of our preprocessing notebook: link to notebook
\nDue to the limited time left in the competition, we could utilize only the 11k sessions that complete all levels.
\nHowever, one big problem remains: using our privately processed leaked data (hereby called private extra data) brings in much lower public LB improvement than using the processed data provided by @glipko here (hereby called public extra data). With @chaudharypriyanshu's best FE and XGB, the public extra data improves LB by 0.001-0.003, but private extra data decreased our best pipeline's public LB score by 0.002. I did several comparisons and concluded that my private version is more similar to the original training data than the public version, and couldn’t understand why its publib LB's performance was worse.
\nWith hindsight, we now all know it was because the public LB's and the private LB's test sets are from different distributions.
\nIn addition, our ensembles (which uses only the public extra data) returned better LB results overall. At the end, we decided not to use the submission with private extra data. Similar to other teams, we missed our best submissions and learned a lesson about not overfitting public LB.
\nBelow are results of some of our models when using the leaked data:
\n| Model | \nCV | \nPublic LB | \nPrivate LB | \nSubmission selected | \n
|---|---|---|---|---|
| XGB 1 (original training data + public extra data) | \n0.70127 | \n0.704 | \n0.701 | \nNo | \n
| XGB 1 (original training data + private extra data) | \n0.70162 | \n0.702 | \n0.704 | \nNo | \n
| XGB 2 (original training data + public extra data) | \n0.70148 | \n0.705 | \n0.700 | \nYes | \n
| XGB 2 (original training data + private extra data) | \n0.7019 | \n0.703 | \n0.704 | \nNo | \n
…and we sadly missed it!
\nIn this discussion, I’m sharing about my data leak preprocessing in the competition. The raw data can be downloaded from Field Day Lab's Open Game Data website. Our full solution has also been detailed out here.
\nBefore beginning, I would like to thank the host and Kaggle for this competition. Despite its issues and challenges, I'm sure all parties have given their best and this has definitely been a memorable learning opportunity.
\nSecond, I'm grateful to be part of my team with @chaudharypriyanshu , @shinomoriaoshi and @martasprg. Your knowledge and hardwork inspired me a lot. It was fortunate that even when we missed our best shots, we still managed to achieve a shake-up and snatch a gold medal, all thanks to our talented and persisting team members.
\nFinally, thank you all who participated and shared your experience/insights. This community has been and will always be the best part of Kaggle!
\nNow let's begin!
\nI found out about this leaked dataset quite late (about 3 weeks before the deadline) when seeing the thread by @glipko - thank you! I then came across this dataset by @rsakata. Thank you both so much for sharing!
\nI realized that, between these two datasets, there were some fils that exist in one but not the other. When combined together, their total distribution is below
\n| \n | \n |
|---|---|
| Total sessions | \n263,881 | \n
| Overlapping sessions with original train set | \n14,505 | \n
| Sessions completing only level 0-4 | \n57,091 | \n
| Sessions completing only level 0-12 | \n21,623 | \n
| Sessions completing all levels (0-22) | \n11,607 | \n
(the rest are unusable & non-overlapping sessions)
\nI spent the last 3 weeks (until the very last hours) of the competition trying to preprocess the raw leaked data. Assuming that processing stays the same between from the original training to the hidden test set, I used the 14k overlapping sessions as a benchmark to compare my processed version with the original data. Due to some discrepancies from within the raw data itself, we could not obtain complete alignment. We got the similarity to 99.99% for labels and 99.5% for input, and stopped there.
\nHere is the final version of our preprocessing notebook: link to notebook
\nDue to the limited time left in the competition, we could utilize only the 11k sessions that complete all levels.
\nHowever, one big problem remains: using our privately processed leaked data (hereby called private extra data) brings in much lower public LB improvement than using the processed data provided by @glipko here (hereby called public extra data). With @chaudharypriyanshu's best FE and XGB, the public extra data improves LB by 0.001-0.003, but private extra data decreased our best pipeline's public LB score by 0.002. I did several comparisons and concluded that my private version is more similar to the original training data than the public version, and couldn’t understand why its publib LB's performance was worse.
\nWith hindsight, we now all know it was because the public LB's and the private LB's test sets are from different distributions.
\nIn addition, our ensembles (which uses only the public extra data) returned better LB results overall. At the end, we decided not to use the submission with private extra data. Similar to other teams, we missed our best submissions and learned a lesson about not overfitting public LB.
\nBelow are results of some of our models when using the leaked data:
\n| Model | \nCV | \nPublic LB | \nPrivate LB | \nSubmission selected | \n
|---|---|---|---|---|
| XGB 1 (original training data + public extra data) | \n0.70127 | \n0.704 | \n0.701 | \nNo | \n
| XGB 1 (original training data + private extra data) | \n0.70162 | \n0.702 | \n0.704 | \nNo | \n
| XGB 2 (original training data + public extra data) | \n0.70148 | \n0.705 | \n0.700 | \nYes | \n
| XGB 2 (original training data + private extra data) | \n0.7019 | \n0.703 | \n0.704 | \nNo | \n
Interesting, thanks for sharing!
\nDid you compare CV excluding private data? From the chart above, I suspect the CV comparisons include CV score of the extra data itself?
\nIn other words, if you trained on either A or A+B, then comparing CV of A to CV of A+B might be less helpful than comparing CV of A to CV only on A (exclude CV on B from result).
", "rawMarkdown": "Interesting, thanks for sharing!\n\nDid you compare CV *excluding* private data? From the chart above, I suspect the CV comparisons include CV score of the extra data itself?\n\nIn other words, if you trained on either A or A+B, then comparing CV of A to CV of A+B might be less helpful than comparing CV of A to CV only on A (exclude CV on B from result).", "votes": null }, { "id": "2326343", "postDate": "07/02/2023 04:34:47", "content": "We compared CVs excluding External data, we only used external data for training. Since we needed previous questions predictions as meta-features, we created CV like this
\n\ntraining_data = pd.concat(train[fold != i],external[fold != i],axis=)\nvalidation_data = train[fold == i] \nexternal_data_for_saving meta_feature = external[fold==i] \nHence, we validated our models only using the original training data provided by Kaggle, however, we did split the external data for training and saving meta-features that can be used across different questions.
\nOur CV was reliable till we reached .703. Above 0.703, the scores became quite random for us (i.e we did not observe the difference >=+0.002, b/w LB and CV that we were observing before, which lead to taking .705 submission due to LB & CV difference>= +0.002 difference).
\nMore about our Validation strategy can be found here:https://www.kaggle.com/code/chaudharypriyanshu/8th-place-solution-training?scriptVersionId=135015129
", "rawMarkdown": "We compared CVs excluding External data, we only used external data for training. Since we needed previous questions predictions as meta-features, we created CV like this\n\n```python\n## for fold=i\ntraining_data = pd.concat(train[fold != i],external[fold != i],axis=0)\nvalidation_data = train[fold == i] #for validation\nexternal_data_for_saving meta_feature = external[fold==i] #for meta feature generation\n```\nHence, **we validated our models only using the original training data provided by Kaggle**, however, we did split the external data for training and saving meta-features that can be used across different questions.\n\nOur CV was reliable till we reached .703. Above 0.703, the scores became quite random for us (i.e we did not observe the difference >=+0.002, b/w LB and CV that we were observing before, which lead to taking .705 submission due to LB & CV difference>= +0.002 difference).\n\nMore about our Validation strategy can be found here:Sorry for the late response.
\nI don't quite understand your last question. Are you saying that we should have done CV on only-A, on only-B, and on (A+B)?
\nIf yes, then I agree what you said would have been more thorough. However, there were two reasons why we could not do so:
\nHope this makes sense, but if it doesn't/is not what you asked about, feel free to comment to continue the discussion!
", "rawMarkdown": "Sorry for the late response.\n\nI don't quite understand your last question. Are you saying that we should have done CV on only-A, on only-B, and on (A+B)?\n\nIf yes, then I agree what you said would have been more thorough. However, there were two reasons why we could not do so:\n1. We did not have enough time - we started working with the extra data 3 weeks before deadline and had to spend half the time left to build & debug the processing, and eventually did not have enough time & submissions to test all possible cases.\n2. Both *private extra data* and *public extra data* differ from the original training data more or less, which we concluded was due to errors in preprocessing & sources of the data. Given this assumption, we believed models trained on **extra data alone** wouldn't be safe to submit, and, going back to point (1) above, decided to spare our submissions for other more hopeful things (FE explorations, ensemble, etc.).\n\nHope this makes sense, but if it doesn't/is not what you asked about, feel free to comment to continue the discussion!", "votes": null } ], "comments": [ { "id": 2325933, "author_name": "roberthatch", "author_url": "", "post_date": "07/01/2023 17:33:49", "content": "Interesting, thanks for sharing!
\nDid you compare CV excluding private data? From the chart above, I suspect the CV comparisons include CV score of the extra data itself?
\nIn other words, if you trained on either A or A+B, then comparing CV of A to CV of A+B might be less helpful than comparing CV of A to CV only on A (exclude CV on B from result).
", "votes": null, "replies": [ { "id": 2326343, "author_name": "chaudharypriyanshu", "author_url": "", "post_date": "07/02/2023 04:34:47", "content": "We compared CVs excluding External data, we only used external data for training. Since we needed previous questions predictions as meta-features, we created CV like this
\n\ntraining_data = pd.concat(train[fold != i],external[fold != i],axis=)\nvalidation_data = train[fold == i] \nexternal_data_for_saving meta_feature = external[fold==i] \nHence, we validated our models only using the original training data provided by Kaggle, however, we did split the external data for training and saving meta-features that can be used across different questions.
\nOur CV was reliable till we reached .703. Above 0.703, the scores became quite random for us (i.e we did not observe the difference >=+0.002, b/w LB and CV that we were observing before, which lead to taking .705 submission due to LB & CV difference>= +0.002 difference).
\nMore about our Validation strategy can be found here:https://www.kaggle.com/code/chaudharypriyanshu/8th-place-solution-training?scriptVersionId=135015129
", "votes": null, "replies": [] }, { "id": 2329401, "author_name": "hoangnguyen719", "author_url": "", "post_date": "07/04/2023 08:55:46", "content": "Sorry for the late response.
\nI don't quite understand your last question. Are you saying that we should have done CV on only-A, on only-B, and on (A+B)?
\nIf yes, then I agree what you said would have been more thorough. However, there were two reasons why we could not do so:
\nHope this makes sense, but if it doesn't/is not what you asked about, feel free to comment to continue the discussion!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2323893": "…and we sadly missed it!\n\nIn this discussion, I’m sharing about my data leak preprocessing in the competition. The raw data can be downloaded from [Field Day Lab's Open Game Data website](https://opengamedata.fielddaylab.wisc.edu/gamedata.php?game=JOWILDER). Our **full solution** has also been detailed out [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420528).\n\nBefore beginning, I would like to thank the host and Kaggle for this competition. Despite its issues and challenges, I'm sure all parties have given their best and this has definitely been a memorable learning opportunity.\n\nSecond, I'm grateful to be part of my team with @chaudharypriyanshu , @shinomoriaoshi and @martasprg. Your knowledge and hardwork inspired me a lot. It was fortunate that even when we missed our best shots, we still managed to achieve a shake-up and snatch a gold medal, all thanks to our talented and persisting team members.\n\nFinally, thank you all who participated and shared your experience/insights. This community has been and will always be the best part of Kaggle!\n\nNow let's begin!\n\n# Leaked Data Collection\nI found out about this leaked dataset quite late (about 3 weeks before the deadline) when seeing the thread by @glipko - thank you! I then came across [this dataset](https://www.kaggle.com/datasets/rsakata/psp-raw-dataset) by @rsakata. Thank you both so much for sharing!\n\nI realized that, between these two datasets, there were some fils that exist in one but not the other. When combined together, their total distribution is below\n| | |\n| --- | --- |\n| Total sessions | 263,881 |\n| Overlapping sessions with original train set | 14,505 |\n| Sessions completing only level 0-4 | 57,091 |\n| Sessions completing only level 0-12 | 21,623 |\n| Sessions completing all levels (0-22) | 11,607 |\n\n*(the rest are unusable & non-overlapping sessions)*\n\n# Leaked Data Preprocessing\nI spent the last 3 weeks (until the very last hours) of the competition trying to preprocess the raw leaked data. Assuming that processing stays the same between from the original training to the hidden test set, I used the 14k overlapping sessions as a benchmark to compare my processed version with the original data. Due to some discrepancies from within the raw data itself, we could not obtain complete alignment. We got the similarity to 99.99% for labels and 99.5% for input, and stopped there.\n\nHere is the final version of our preprocessing notebook: [link to notebook](https://www.kaggle.com/code/hoangnguyen719/pspfgp-additional-data-processed-v3-2)\n\n# Leaked Data Usage\nDue to the limited time left in the competition, we could utilize only the 11k sessions that complete all levels.\n\nHowever, one big problem remains: using our privately processed leaked data (hereby called *private extra data*) brings in much lower **public LB** improvement than using the processed data provided by @glipko [here](https://www.kaggle.com/datasets/glipko/additional-data-predict-students-performance) (hereby called *public extra data*). With @chaudharypriyanshu's best FE and XGB, the public extra data improves LB by 0.001-0.003, but private extra data **decreased** our best pipeline's public LB score by 0.002. I did several comparisons and concluded that my private version is more similar to the original training data than the public version, and couldn’t understand why its publib LB's performance was worse.\n\nWith hindsight, we now all know it was because the public LB's and the private LB's test sets are from different distributions.\n\nIn addition, our ensembles (which uses only the public extra data) returned better LB results overall. At the end, we decided not to use the submission with private extra data. Similar to other teams, we missed our best submissions and learned a lesson about not overfitting public LB.\n\n# Performance Summary\nBelow are results of some of our models when using the leaked data:\n \n| Model | CV | Public LB | Private LB | Submission selected |\n| --- | --- |\n| XGB 1 (original training data + public extra data) | 0.70127 | 0.704 | 0.701 | No |\n| XGB 1 (original training data + private extra data) | 0.70162 | 0.702 | **0.704** | No |\n| XGB 2 (original training data + public extra data) | 0.70148 | **0.705** | 0.700 | **Yes** |\n| XGB 2 (original training data + private extra data) | **0.7019** | 0.703 | **0.704** | No |", "2325933": "Interesting, thanks for sharing!\n\nDid you compare CV *excluding* private data? From the chart above, I suspect the CV comparisons include CV score of the extra data itself?\n\nIn other words, if you trained on either A or A+B, then comparing CV of A to CV of A+B might be less helpful than comparing CV of A to CV only on A (exclude CV on B from result).", "2326343": "We compared CVs excluding External data, we only used external data for training. Since we needed previous questions predictions as meta-features, we created CV like this\n\n```python\n## for fold=i\ntraining_data = pd.concat(train[fold != i],external[fold != i],axis=0)\nvalidation_data = train[fold == i] #for validation\nexternal_data_for_saving meta_feature = external[fold==i] #for meta feature generation\n```\nHence, **we validated our models only using the original training data provided by Kaggle**, however, we did split the external data for training and saving meta-features that can be used across different questions.\n\nOur CV was reliable till we reached .703. Above 0.703, the scores became quite random for us (i.e we did not observe the difference >=+0.002, b/w LB and CV that we were observing before, which lead to taking .705 submission due to LB & CV difference>= +0.002 difference).\n\nMore about our Validation strategy can be found here: