Hi Kagglers!
\nWe made it! On behalf of our team at the Child Mind Institute and our sponsors at the California Department of Health Care Services, Dell Technologies, and NVIDIA, thank you all SO much for participating in our competition over the last three months! We couldn’t be happier to have the participation of over 4,500 of you, submitting over 85,000 submissions to tackle our challenge! Truly, we are grateful for the time and thought you invested in solving our problem. The contributions of each and every one of you, through your submissions, interactions on the discussion board, and exploration of our dataset, are hugely valuable and we look forward to digesting them all and learning from you.
\nIn the past few days it’s been fun and insightful to read the discussion forums and see you, the community, examine the data features, visualize interesting patterns, learn from the data, and help each other with model definition and other data science aspects of the competition.
\nWe also noticed that you started to recognize a trend that we’ve observed behind-the-scenes for a while: there’s plenty of opportunity for overfitting in our competition, and as a result, a large shake-up on the leaderboard. There have been several interesting threads where you’ve each shared what were your signs for detecting overfitting, making predictions about the distribution of Problematic Internet Use in the test set, and guessing what the peak performance in the private test dataset will be (which were generally more conservative than we see in practice). Across each, it’s clear that you have started to recognize the potential for overfitting by optimizing for leaderboard performance over consistency in cross-validation.
\nThere are plenty of other examples of Kaggle competitions with shake-ups (interestingly, according to this Meta-Kaggle post, the largest shake-up in Kaggle history is also in the space of healthcare).
\nTo give some perspective, when I cloned our leaderboard on 12/18 (apologies if it’s slightly out of date at the time of posting), I calculated our shakeup to be 0.255, which is around 15–20th all-time on Kaggle according to the Meta-Kaggle post I mentioned above — putting us right between two NFL-based competitions (1, 2). Now, I guess the more interesting question, what is the source of our shake-up?
\nFirst, a few details. We stratified our sample explicitly across binned age, sex, and the presence of actigraphy data. In practice, we also ended up with a balanced split of all of our instruments (at most, the availability of an instrument differed by 4% between the public and private test set). The proportion of participants in each class was also balanced across the splits, and you can see the breakdown below:
\n| sii | \nOverall | \nTrain | \nPublic | \nPrivate | \n
|---|---|---|---|---|
| 0 | \n0.577 | \n0.581 | \n0.587 | \n0.561 | \n
| 1 | \n0.272 | \n0.275 | \n0.258 | \n0.273 | \n
| 2 | \n0.14 | \n0.133 | \n0.144 | \n0.154 | \n
| 3 | \n0.012 | \n0.012 | \n0.011 | \n0.013 | \n
Given that our data are relatively well balanced, what is going on here?
\nThere are more than a few hypotheses… Let’s first assume that there wasn’t actually any overfitting of submissions in this competition, but that models had different strengths. If this were the case, it’s possible that models optimized with higher sensitivity to sii severity (i.e., were more likely to predict a higher score) were those that raised up the private leaderboard. This is possible given the small change in the distributions of sii classes between the private and public test sets. In this scenario, models that performed the best on the public set could have been doing a better job at predicting participants with an sii=0, of which there was a slightly smaller portion in the private test set.
\nIn practice, we’re not sure that this is the case. We saw some notebooks tinkering with minor model hyperparameters, thresholds, or random seeds to climb the public leaderboard. Another sign of overfitting on our public leaderboard is the submission count. If we look at the top 10 teams in the public leaderboard versus the private leaderboard, the public-leaders on average submitted 212 models, whereas the private-leaders on average submitted 64. If we look at the median, these values are 199 vs 25. This suggests that there was certainly some work to cook scores on the public leaderboard, that didn’t result in success on our held out private data.
\nI think it’s relatively safe to rule out the hypothesis that there isn’t overfitting here, which now begs the question, what about our data lends itself to overfitting or poor generalization?
\nThere have been many (extremely informative!) discussion posts over the course of this competition that highlight possibilities: the possibility of batch effects across our samples; there could be data entry errors that limit reliability in the dataset; or the assessment instrument/prediction target is noisy and biased. All of these are potentially valid hypotheses, and deserve further exploration to understand. While we can evaluate the presence of batch effects in our data with tools like ComBAT, and automatically detect obvious data entry errors, evaluating the bias of our instruments is another can of worms altogether, and often impossible to do after data collection has taken place. If other assessments with similar purposes were deliberately included in data collection, this may be possible, but even in that case, the bias of the other assessments would likely need to be explored independently, perpetuating this challenge. In addition to the data challenges we’ve already mentioned, the nature of working with such heterogeneous datasets with relatively few samples is that there may simply not be strong enough statistical associations to give stable rankings — or critically, predictions when applied in real-world contexts — even in the absence of overfitting.
\nBatch effects, data entry errors, measurement biases, and dataset heterogeneity are all core challenges that many psychological sciences face on a regular basis. This competition reflects those “real-world” challenges, rather than hiding them or proposing sanitized data that researchers, data scientists, and clinicians rarely work with. Moreover, mental health and the definitions of “within expected ranges” or “acceptable” vary across cultures, ages, genders, specific samples (for example, clinical vs non-clinical sample), and more: variation is a challenge inherent to mental health data, and highly generalizable performance is often elusive. This competition showed itself to be no exception!
\nHere, we aimed to offer participants the possibility of working with real data from the mental health space, to collectively learn from the data, identify gaps in data collection practices, bring innovation to the field, and hopefully have fun by participating! We feel that we’ve learned an immense amount from all of you, and are grateful for your company on this adventure. We look forward to debriefing with a handful of you at the winner’s calls soon. 🙂
\nSincerely,
\n-Greg
Gregory Kiar, PhD\nResearch Center for Data Analytics, Innovation, Rigor\nChild Mind Hi Kagglers!
\nWe made it! On behalf of our team at the Child Mind Institute and our sponsors at the California Department of Health Care Services, Dell Technologies, and NVIDIA, thank you all SO much for participating in our competition over the last three months! We couldn’t be happier to have the participation of over 4,500 of you, submitting over 85,000 submissions to tackle our challenge! Truly, we are grateful for the time and thought you invested in solving our problem. The contributions of each and every one of you, through your submissions, interactions on the discussion board, and exploration of our dataset, are hugely valuable and we look forward to digesting them all and learning from you.
\nIn the past few days it’s been fun and insightful to read the discussion forums and see you, the community, examine the data features, visualize interesting patterns, learn from the data, and help each other with model definition and other data science aspects of the competition.
\nWe also noticed that you started to recognize a trend that we’ve observed behind-the-scenes for a while: there’s plenty of opportunity for overfitting in our competition, and as a result, a large shake-up on the leaderboard. There have been several interesting threads where you’ve each shared what were your signs for detecting overfitting, making predictions about the distribution of Problematic Internet Use in the test set, and guessing what the peak performance in the private test dataset will be (which were generally more conservative than we see in practice). Across each, it’s clear that you have started to recognize the potential for overfitting by optimizing for leaderboard performance over consistency in cross-validation.
\nThere are plenty of other examples of Kaggle competitions with shake-ups (interestingly, according to this Meta-Kaggle post, the largest shake-up in Kaggle history is also in the space of healthcare).
\nTo give some perspective, when I cloned our leaderboard on 12/18 (apologies if it’s slightly out of date at the time of posting), I calculated our shakeup to be 0.255, which is around 15–20th all-time on Kaggle according to the Meta-Kaggle post I mentioned above — putting us right between two NFL-based competitions (1, 2). Now, I guess the more interesting question, what is the source of our shake-up?
\nFirst, a few details. We stratified our sample explicitly across binned age, sex, and the presence of actigraphy data. In practice, we also ended up with a balanced split of all of our instruments (at most, the availability of an instrument differed by 4% between the public and private test set). The proportion of participants in each class was also balanced across the splits, and you can see the breakdown below:
\n| sii | \nOverall | \nTrain | \nPublic | \nPrivate | \n
|---|---|---|---|---|
| 0 | \n0.577 | \n0.581 | \n0.587 | \n0.561 | \n
| 1 | \n0.272 | \n0.275 | \n0.258 | \n0.273 | \n
| 2 | \n0.14 | \n0.133 | \n0.144 | \n0.154 | \n
| 3 | \n0.012 | \n0.012 | \n0.011 | \n0.013 | \n
Given that our data are relatively well balanced, what is going on here?
\nThere are more than a few hypotheses… Let’s first assume that there wasn’t actually any overfitting of submissions in this competition, but that models had different strengths. If this were the case, it’s possible that models optimized with higher sensitivity to sii severity (i.e., were more likely to predict a higher score) were those that raised up the private leaderboard. This is possible given the small change in the distributions of sii classes between the private and public test sets. In this scenario, models that performed the best on the public set could have been doing a better job at predicting participants with an sii=0, of which there was a slightly smaller portion in the private test set.
\nIn practice, we’re not sure that this is the case. We saw some notebooks tinkering with minor model hyperparameters, thresholds, or random seeds to climb the public leaderboard. Another sign of overfitting on our public leaderboard is the submission count. If we look at the top 10 teams in the public leaderboard versus the private leaderboard, the public-leaders on average submitted 212 models, whereas the private-leaders on average submitted 64. If we look at the median, these values are 199 vs 25. This suggests that there was certainly some work to cook scores on the public leaderboard, that didn’t result in success on our held out private data.
\nI think it’s relatively safe to rule out the hypothesis that there isn’t overfitting here, which now begs the question, what about our data lends itself to overfitting or poor generalization?
\nThere have been many (extremely informative!) discussion posts over the course of this competition that highlight possibilities: the possibility of batch effects across our samples; there could be data entry errors that limit reliability in the dataset; or the assessment instrument/prediction target is noisy and biased. All of these are potentially valid hypotheses, and deserve further exploration to understand. While we can evaluate the presence of batch effects in our data with tools like ComBAT, and automatically detect obvious data entry errors, evaluating the bias of our instruments is another can of worms altogether, and often impossible to do after data collection has taken place. If other assessments with similar purposes were deliberately included in data collection, this may be possible, but even in that case, the bias of the other assessments would likely need to be explored independently, perpetuating this challenge. In addition to the data challenges we’ve already mentioned, the nature of working with such heterogeneous datasets with relatively few samples is that there may simply not be strong enough statistical associations to give stable rankings — or critically, predictions when applied in real-world contexts — even in the absence of overfitting.
\nBatch effects, data entry errors, measurement biases, and dataset heterogeneity are all core challenges that many psychological sciences face on a regular basis. This competition reflects those “real-world” challenges, rather than hiding them or proposing sanitized data that researchers, data scientists, and clinicians rarely work with. Moreover, mental health and the definitions of “within expected ranges” or “acceptable” vary across cultures, ages, genders, specific samples (for example, clinical vs non-clinical sample), and more: variation is a challenge inherent to mental health data, and highly generalizable performance is often elusive. This competition showed itself to be no exception!
\nHere, we aimed to offer participants the possibility of working with real data from the mental health space, to collectively learn from the data, identify gaps in data collection practices, bring innovation to the field, and hopefully have fun by participating! We feel that we’ve learned an immense amount from all of you, and are grateful for your company on this adventure. We look forward to debriefing with a handful of you at the winner’s calls soon. 🙂
\nSincerely,
\n-Greg
Gregory Kiar, PhD\nResearch Center for Data Analytics, Innovation, Rigor\nChild Mind Thank you for organizing this competition! Recently, a Kaggler pointed out the striking differences between missing values proportion in train vs test data here - this issue was not raised in this post
\nWhat do you think may have caused this ?
", "rawMarkdown": "Thank you for organizing this competition! Recently, a Kaggler pointed out the striking differences between missing values proportion in train vs test data [here](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/552488#3076452) - this issue was not raised in this post\n\nWhat do you think may have caused this ?", "votes": 5 }, { "id": 3077157, "postDate": "2024-12-20T16:19:03.883Z", "content": "Thank you for this challenging real world data, learnt a lot and Many Congratulations to the Winners!
", "rawMarkdown": "Thank you for this challenging real world data, learnt a lot and Many Congratulations to the Winners!", "votes": 1 }, { "id": 3150238, "postDate": "2025-03-15T08:00:50.277Z", "content": "This is really difficult for a kaggle beginner
", "rawMarkdown": "This is really difficult for a kaggle beginner" }, { "id": 3135608, "postDate": "2025-02-27T16:33:20.363Z", "content": "perfect competition
", "rawMarkdown": "perfect competition" }, { "id": 3078498, "postDate": "2024-12-22T11:47:40.687Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3083596, "postDate": "2024-12-29T17:45:58.300Z", "content": "thank you
", "rawMarkdown": "thank you ", "votes": 1 } ], "comments": [ { "id": 3077524, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2024-12-21T04:53:43.517000", "content": "Thank you for organizing this competition! Recently, a Kaggler pointed out the striking differences between missing values proportion in train vs test data here - this issue was not raised in this post
\nWhat do you think may have caused this ?
", "votes": 5, "replies": [] }, { "id": 3077157, "author_name": "Deepak Saldanha", "author_url": "", "post_date": "2024-12-20T16:19:03.883000", "content": "Thank you for this challenging real world data, learnt a lot and Many Congratulations to the Winners!
", "votes": 1, "replies": [] }, { "id": 3150238, "author_name": "Suzie_xtime", "author_url": "", "post_date": "2025-03-15T08:00:50.277000", "content": "This is really difficult for a kaggle beginner
", "votes": 0, "replies": [] }, { "id": 3135608, "author_name": "Đặng Đào Xuân Trúc", "author_url": "", "post_date": "2025-02-27T16:33:20.363000", "content": "perfect competition
", "votes": 0, "replies": [] }, { "id": 3078498, "author_name": "", "author_url": "", "post_date": "2024-12-22T11:47:40.687000", "content": "", "votes": 0, "replies": [] }, { "id": 3083596, "author_name": "benkerrouche abdelbasset", "author_url": "", "post_date": "2024-12-29T17:45:58.300000", "content": "thank you
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "3077021": "Hi Kagglers!\n\nWe made it! On behalf of our team at the Child Mind Institute and our sponsors at the California Department of Health Care Services, Dell Technologies, and NVIDIA, thank you all SO much for participating in our competition over the last three months! We couldn’t be happier to have the participation of over 4,500 of you, submitting over 85,000 submissions to tackle our challenge! Truly, we are grateful for the time and thought you invested in solving our problem. The contributions of each and every one of you, through your submissions, interactions on the discussion board, and exploration of our dataset, are hugely valuable and we look forward to digesting them all and learning from you.\n\nIn the past few days it’s been fun and insightful to read the discussion forums and see you, the community, examine the data features, visualize interesting patterns, learn from the data, and help each other with model definition and other data science aspects of the competition.\n\n### Leaderboard Shake-Up: What Happened?\nWe also noticed that you started to recognize a trend that we’ve observed behind-the-scenes for a while: there’s plenty of opportunity for overfitting in our competition, and as a result, a large shake-up on the leaderboard. There have been several interesting threads where you’ve each shared what were [your signs for detecting overfitting](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/551758), making [predictions about the distribution of Problematic Internet Use in the test set](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/551913), and [guessing what the peak performance in the private test dataset will be](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/551164) (which were generally more conservative than we see in practice). Across each, it’s clear that you have started to recognize the potential for overfitting by optimizing for leaderboard performance over consistency in cross-validation.\n\nThere are plenty of other examples of Kaggle competitions with shake-ups (interestingly, according to this [Meta-Kaggle post](https://www.kaggle.com/code/jtrotman/meta-kaggle-scatter-plot-competition-shake-up), the largest shake-up in Kaggle history is also [in the space of healthcare](https://www.kaggle.com/c/icr-identify-age-related-conditions)). \n\nTo give some perspective, when I cloned our leaderboard on 12/18 (apologies if it’s slightly out of date at the time of posting), I calculated our shakeup to be **0.255**, which is around 15–20th all-time on Kaggle according to the Meta-Kaggle post I mentioned above — putting us right between two NFL-based competitions ([1](https://www.kaggle.com/competitions/data-science-bowl-2017), [2](https://www.kaggle.com/c/nfl-impact-detection)). Now, I guess the more interesting question, *what is the source of our shake-up*?\n\n### Key Hypotheses: Overfitting or Not?\nFirst, a few details. We stratified our sample explicitly across binned age, sex, and the presence of actigraphy data. In practice, we also ended up with a balanced split of all of our instruments (at most, the availability of an instrument differed by 4% between the public and private test set). The proportion of participants in each class was also balanced across the splits, and you can see the breakdown below:\n\n|sii |Overall |Train |Public |Private |\n|:--|:--|:--|:--|:--|\n| 0 | 0.577 | 0.581 | 0.587 | 0.561 |\n| 1 | 0.272 | 0.275 | 0.258 | 0.273 |\n| 2 | 0.14 | 0.133 | 0.144 | 0.154 |\n| 3 | 0.012 | 0.012 | 0.011 | 0.013 |\n\nGiven that our data are relatively well balanced, *what is going on here*? \n\nThere are more than a few hypotheses… Let’s first assume that there wasn’t actually any overfitting of submissions in this competition, but that models had different strengths. If this were the case, it’s possible that models optimized with higher sensitivity to sii severity (i.e., were more likely to predict a higher score) were those that raised up the private leaderboard. This is possible given the small change in the distributions of sii classes between the private and public test sets. In this scenario, models that performed the best on the public set could have been doing a better job at predicting participants with an sii=0, of which there was a slightly smaller portion in the private test set.\n\nIn practice, we’re not sure that this is the case. We saw some notebooks tinkering with minor model hyperparameters, thresholds, or random seeds to climb the public leaderboard. Another sign of overfitting on our public leaderboard is the submission count. If we look at the top 10 teams in the public leaderboard versus the private leaderboard, the public-leaders on average submitted 212 models, whereas the private-leaders on average submitted 64. If we look at the median, these values are 199 vs 25. This suggests that there was certainly some work to cook scores on the public leaderboard, that didn’t result in success on our held out private data.\n\nI think it’s relatively safe to rule out the hypothesis that there isn’t overfitting here, which now begs the question, *what about our data lends itself to overfitting or poor generalization*? \n\nThere have been many (extremely informative!) discussion posts over the course of this competition that highlight possibilities: the possibility of batch effects across our samples; there could be data entry errors that limit reliability in the dataset; or the assessment instrument/prediction target is noisy and biased. All of these are potentially valid hypotheses, and deserve further exploration to understand. While we can evaluate the presence of batch effects in our data with tools like ComBAT, and automatically detect obvious data entry errors, evaluating the bias of our instruments is another can of worms altogether, and often impossible to do after data collection has taken place. If other assessments with similar purposes were deliberately included in data collection, this may be possible, but even in that case, the bias of the other assessments would likely need to be explored independently, perpetuating this challenge. In addition to the data challenges we’ve already mentioned, the nature of working with such heterogeneous datasets with relatively few samples is that there may simply not be strong enough statistical associations to give stable rankings — or critically, predictions when applied in real-world contexts — even in the absence of overfitting.\n\n### Real-World Data, Real-World Challenges\nBatch effects, data entry errors, measurement biases, and dataset heterogeneity are all core challenges that many psychological sciences face on a regular basis. This competition reflects those “real-world” challenges, rather than hiding them or proposing sanitized data that researchers, data scientists, and clinicians rarely work with. Moreover, mental health and the definitions of “within expected ranges” or “acceptable” vary across cultures, ages, genders, specific samples (for example, clinical vs non-clinical sample), and more: variation is a challenge inherent to mental health data, and highly generalizable performance is often elusive. This competition showed itself to be no exception!\n\nHere, we aimed to offer participants the possibility of working with real data from the mental health space, to collectively learn from the data, identify gaps in data collection practices, bring innovation to the field, and hopefully have fun by participating! We feel that we’ve learned an immense amount from all of you, and are grateful for your company on this adventure. We look forward to debriefing with a handful of you at the winner’s calls soon. 🙂\n\n\nSincerely,\n-Greg\n\n\n```\nGregory Kiar, PhD\nResearch Scientist\nDirector, Center for Data Analytics, Innovation, and Rigor\nChild Mind Institute\n```", "3077524": "Thank you for organizing this competition! Recently, a Kaggler pointed out the striking differences between missing values proportion in train vs test data [here](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/552488#3076452) - this issue was not raised in this post\n\nWhat do you think may have caused this ?", "3077157": "Thank you for this challenging real world data, learnt a lot and Many Congratulations to the Winners!", "3150238": "This is really difficult for a kaggle beginner", "3135608": "perfect competition", "3078498": "", "3083596": "thank you " } }