Some high scoring public kernels multiply a certain coefficient (0.996) to the predicted FVC (like this one). This multiply-something-for-better-score, or so-called \"magic\" approach, had been actually useful for some kaggle competitions in the past (e.g. M5), especially in the context of future forecasting.
\nGiven we are predicting mostly declines in the lung function in the future, using a \"magic\" less than 1 (e.g. 0.996) may be actually a good idea for a better private score. I may use it in one of my submissions.
\nWould you use it in your final submission, or would you rather stay away from it? It would be great if you could share your idea on this topic.
", "messages": [ { "id": "1027715", "postDate": "09/26/2020 10:00:59", "content": "Some high scoring public kernels multiply a certain coefficient (0.996) to the predicted FVC (like this one). This multiply-something-for-better-score, or so-called \"magic\" approach, had been actually useful for some kaggle competitions in the past (e.g. M5), especially in the context of future forecasting.
\nGiven we are predicting mostly declines in the lung function in the future, using a \"magic\" less than 1 (e.g. 0.996) may be actually a good idea for a better private score. I may use it in one of my submissions.
\nWould you use it in your final submission, or would you rather stay away from it? It would be great if you could share your idea on this topic.
", "rawMarkdown": "Some high scoring public kernels multiply a certain coefficient (0.996) to the predicted FVC (like [this one](https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter)). This multiply-something-for-better-score, or so-called \"magic\" approach, had been actually useful for some kaggle competitions in the past (e.g. M5), especially in the context of future forecasting.\n\nGiven we are predicting mostly declines in the lung function in the future, using a \"magic\" less than 1 (e.g. 0.996) may be actually a good idea for a better private score. I may use it in one of my submissions.\n\nWould you use it in your final submission, or would you rather stay away from it? It would be great if you could share your idea on this topic.", "votes": null }, { "id": "1027767", "postDate": "09/26/2020 10:51:27", "content": "Since when overfitting to public test set is called \"magic\"?
", "rawMarkdown": "Since when overfitting to public test set is called \"magic\"?", "votes": null }, { "id": "1027788", "postDate": "09/26/2020 11:12:59", "content": "Since the M5, to my knowledge.
", "rawMarkdown": "Since the M5, to my knowledge.", "votes": null }, { "id": "1028234", "postDate": "09/26/2020 17:42:27", "content": "Right, I can't argue with that. I would use a multiplier like that if only I am able to detect higher predictions consistently. I would also test it on oof not public test set.
", "rawMarkdown": "Right, I can't argue with that. I would use a multiplier like that if only I am able to detect higher predictions consistently. I would also test it on oof not public test set.", "votes": null }, { "id": "1028245", "postDate": "09/26/2020 17:49:49", "content": "Yeah, @gunesevitan is right, checked that public kernel and it's going to overfit, so maybe it's not magic!!! Thank you
", "rawMarkdown": "Yeah, @gunesevitan is right, checked that public kernel and it's going to overfit, so maybe it's not magic!!! Thank you", "votes": null }, { "id": "1028455", "postDate": "09/26/2020 21:35:23", "content": "An interesting question is to what extent you would trust something, if it works in cross-validation, when you have picked it based on what happens out-of-fold in cross-validation (I'm worried about overfitting the \"magic\" to the errors in my out-of-fold predictions). I'd assume with way larger datasets that would not be that much of an issue, but with so little data this is presumably much more of a concern. Perhaps the LB is - despite its small size - the way to check whether something like that, which looks good in CV, holds up?
", "rawMarkdown": "An interesting question is to what extent you would trust something, if it works in cross-validation, when you have picked it based on what happens out-of-fold in cross-validation (I'm worried about overfitting the \"magic\" to the errors in my out-of-fold predictions). I'd assume with way larger datasets that would not be that much of an issue, but with so little data this is presumably much more of a concern. Perhaps the LB is - despite its small size - the way to check whether something like that, which looks good in CV, holds up?", "votes": null }, { "id": "1028717", "postDate": "09/27/2020 06:20:36", "content": "Tuning constants and messing around with the random seed - not a good way to go, given how many of the participants have used the 'mloss' constant of a certain public kernel (public kernels?) to get past -6.81 and how seed dependent the kernel is.
", "rawMarkdown": "Tuning constants and messing around with the random seed - not a good way to go, given how many of the participants have used the 'mloss' constant of a certain public kernel (public kernels?) to get past -6.81 and how seed dependent the kernel is.", "votes": null }, { "id": "1029358", "postDate": "09/27/2020 17:19:59", "content": "Does this magic improve cv score too? We can't really rely on public leaderboard since its too small. I remember something similar approach helped in Trends neuroimaging competition.
", "rawMarkdown": "Does this magic improve cv score too? We can't really rely on public leaderboard since its too small. I remember something similar approach helped in Trends neuroimaging competition.", "votes": null }, { "id": "1032089", "postDate": "09/30/2020 00:18:27", "content": "If the \"magic\" number cannot be explained, is it truly magical? How could you know if heretofore unseen data would react similarly?
", "rawMarkdown": "If the \"magic\" number cannot be explained, is it truly magical? How could you know if heretofore unseen data would react similarly?", "votes": null }, { "id": "1032161", "postDate": "09/30/2020 02:28:11", "content": "\n\nIf the \"magic\" number cannot be explained, is it truly magical?
\n
These constants help to \"tune\" your predictions better towards the Public LB, however as Gunes said it's probably a better approach to test it on OOF instead of public test.
\n\n\nHow could you know if heretofore unseen data would react similarly?
\n
We don't, and that's what blew up the M5 leaderboard.
", "rawMarkdown": "> If the \"magic\" number cannot be explained, is it truly magical?\n\nThese constants help to \"tune\" your predictions better towards the Public LB, however as Gunes said it's probably a better approach to test it on OOF instead of public test.\n\n> How could you know if heretofore unseen data would react similarly?\n\nWe don't, and that's what blew up the M5 leaderboard.", "votes": null }, { "id": "1034847", "postDate": "10/02/2020 08:28:18", "content": "I use the pinball loss in my model with quantiles around [0.25, 0.5, 0.75]. For each fold, I try to minimise globally the loss on my fvc and confidence results by using scipy.optimize.minimize; then I use the coefficients before averaging the OOF predictions to produce my final predictions.
\nIt gives me a bit less than +0.01 on my public score.
Since when overfitting to public test set is called \"magic\"?
", "votes": null, "replies": [ { "id": 1027788, "author_name": "code1110", "author_url": "", "post_date": "09/26/2020 11:12:59", "content": "Since the M5, to my knowledge.
", "votes": null, "replies": [] }, { "id": 1028234, "author_name": "gunesevitan", "author_url": "", "post_date": "09/26/2020 17:42:27", "content": "Right, I can't argue with that. I would use a multiplier like that if only I am able to detect higher predictions consistently. I would also test it on oof not public test set.
", "votes": null, "replies": [] }, { "id": 1028245, "author_name": "", "author_url": "", "post_date": "09/26/2020 17:49:49", "content": "Yeah, @gunesevitan is right, checked that public kernel and it's going to overfit, so maybe it's not magic!!! Thank you
", "votes": null, "replies": [] }, { "id": 1028455, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "09/26/2020 21:35:23", "content": "An interesting question is to what extent you would trust something, if it works in cross-validation, when you have picked it based on what happens out-of-fold in cross-validation (I'm worried about overfitting the \"magic\" to the errors in my out-of-fold predictions). I'd assume with way larger datasets that would not be that much of an issue, but with so little data this is presumably much more of a concern. Perhaps the LB is - despite its small size - the way to check whether something like that, which looks good in CV, holds up?
", "votes": null, "replies": [] }, { "id": 1028717, "author_name": "nxrprime", "author_url": "", "post_date": "09/27/2020 06:20:36", "content": "Tuning constants and messing around with the random seed - not a good way to go, given how many of the participants have used the 'mloss' constant of a certain public kernel (public kernels?) to get past -6.81 and how seed dependent the kernel is.
", "votes": null, "replies": [] } ] }, { "id": 1029358, "author_name": "nischaydnk", "author_url": "", "post_date": "09/27/2020 17:19:59", "content": "Does this magic improve cv score too? We can't really rely on public leaderboard since its too small. I remember something similar approach helped in Trends neuroimaging competition.
", "votes": null, "replies": [] }, { "id": 1032089, "author_name": "avrahamadler", "author_url": "", "post_date": "09/30/2020 00:18:27", "content": "If the \"magic\" number cannot be explained, is it truly magical? How could you know if heretofore unseen data would react similarly?
", "votes": null, "replies": [ { "id": 1032161, "author_name": "nxrprime", "author_url": "", "post_date": "09/30/2020 02:28:11", "content": "\n\nIf the \"magic\" number cannot be explained, is it truly magical?
\n
These constants help to \"tune\" your predictions better towards the Public LB, however as Gunes said it's probably a better approach to test it on OOF instead of public test.
\n\n\nHow could you know if heretofore unseen data would react similarly?
\n
We don't, and that's what blew up the M5 leaderboard.
", "votes": null, "replies": [] } ] }, { "id": 1034847, "author_name": "filipix", "author_url": "", "post_date": "10/02/2020 08:28:18", "content": "I use the pinball loss in my model with quantiles around [0.25, 0.5, 0.75]. For each fold, I try to minimise globally the loss on my fvc and confidence results by using scipy.optimize.minimize; then I use the coefficients before averaging the OOF predictions to produce my final predictions.
\nIt gives me a bit less than +0.01 on my public score.