I observed that LB and CV has a relatively large gap(30~50bps) for most people's models(as appeared in discussion threads). Can anyone give some reasonable explanations?
\nFurthermore, our team is working on a 1D model. A very weird thing is that our 1D model somehow only has a gap 10bps or less most of the time, while our 2D model frequently has a ~20-30 bps gap. These two kinds of models share the same preprocessing(high low pass and whiten). This is very confusing to me and I hope people can share some insights about this(and I can increase the gap without overfitting to LB :P ) Thanks!
", "messages": [ { "id": "1507328", "postDate": "09/09/2021 05:20:36", "content": "I observed that LB and CV has a relatively large gap(30~50bps) for most people's models(as appeared in discussion threads). Can anyone give some reasonable explanations?
\nFurthermore, our team is working on a 1D model. A very weird thing is that our 1D model somehow only has a gap 10bps or less most of the time, while our 2D model frequently has a ~20-30 bps gap. These two kinds of models share the same preprocessing(high low pass and whiten). This is very confusing to me and I hope people can share some insights about this(and I can increase the gap without overfitting to LB :P ) Thanks!
", "rawMarkdown": "I observed that LB and CV has a relatively large gap(30~50bps) for most people's models(as appeared in discussion threads). Can anyone give some reasonable explanations? \n\nFurthermore, our team is working on a 1D model. A very weird thing is that our 1D model somehow only has a gap 10bps or less most of the time, while our 2D model frequently has a ~20-30 bps gap. These two kinds of models share the same preprocessing(high low pass and whiten). This is very confusing to me and I hope people can share some insights about this(and I can increase the gap without overfitting to LB :P ) Thanks!", "votes": null }, { "id": "1507334", "postDate": "09/09/2021 05:25:44", "content": "What do you mean by BPS ?
", "rawMarkdown": "What do you mean by BPS ?", "votes": null }, { "id": "1507335", "postDate": "09/09/2021 05:27:03", "content": "@mithilsalunkhe 0.0001 is one bp
", "rawMarkdown": "mithilsalunkhe 0.0001 is one bp", "votes": null }, { "id": "1507434", "postDate": "09/09/2021 07:43:10", "content": "It's not always the case, difference between CV and LB for me was less than 10 bps, even though I'm using 2D model.
\nCV : 0.8757, LB : 0.8766
\nWith release of new 0.8758 kernel it's hard to properly compute CV for now, so I can't judge on that.
As calculated the test data has relative smaller mean value, I guess that's why most of us got higer LB than CV, maybe test data has less noise level?
\n\n", "rawMarkdown": "As calculated the test data has relative smaller mean value, I guess that's why most of us got higer LB than CV, maybe test data has less noise level?\n\n> Traind data: global_mean, global_std = 4.3110404142640676e-26, 6.150793058115705e-21\nTest data: global_mean, global_std = 1.647648937086022e-26, 6.150793058115705e-21", "votes": null }, { "id": "1507464", "postDate": "09/09/2021 08:16:35", "content": "Traind data: global_mean, global_std = 4.3110404142640676e-26, 6.150793058115705e-21
\n
\n Test data: global_mean, global_std = 1.647648937086022e-26, 6.150793058115705e-21
CV-LB difference looks reasonable considering that the same difference can be observed between different folds.
", "rawMarkdown": "CV-LB difference looks reasonable considering that the same difference can be observed between different folds.", "votes": null }, { "id": "1507465", "postDate": "09/09/2021 08:18:52", "content": "you have to tell us the performance of model.
\nfor less discriminative model, the generalization is of course better.
\nas an extreme example, imagine we use a weak model like resnet18. then we have cv low cv and lb, say 0.840. the value are low but generalization gap is almost zero.
\nif a model is highly discriminative (e.g. very deep CNN or transformer), you will learn the \"noise in the data\" and generalization drop and overfitting kicks in (unless you use regularisation methods)
", "rawMarkdown": "you have to tell us the performance of model.\n\nfor less discriminative model, the generalization is of course better.\n\nas an extreme example, imagine we use a weak model like resnet18. then we have cv low cv and lb, say 0.840. the value are low but generalization gap is almost zero.\n\nif a model is highly discriminative (e.g. very deep CNN or transformer), you will learn the \"noise in the data\" and generalization drop and overfitting kicks in (unless you use regularisation methods)", "votes": null }, { "id": "1507476", "postDate": "09/09/2021 08:29:29", "content": "in fact, resnet18 is not that weak. My current best single model is a resnet18d with local CV = 0.874047 and LB = 0.8759
", "rawMarkdown": "in fact, resnet18 is not that weak. My current best single model is a resnet18d with local CV = 0.874047 and LB = 0.8759", "votes": null }, { "id": "1507774", "postDate": "09/09/2021 14:25:51", "content": "Wow, nice CV-LB @fabiendaniel !! If you like to share, you use CWT or CQT ?
", "rawMarkdown": "Wow, nice CV-LB @fabiendaniel !! If you like to share, you use CWT or CQT ?", "votes": null }, { "id": "1507810", "postDate": "09/09/2021 14:59:27", "content": "We didn't see such big difference between folds either(std less than 5bps with only a few runs for 5Fold; we mostly did 1 fold for faster iteration of ideas). If you don't mind, what's your cv std for different folds?
", "rawMarkdown": "We didn't see such big difference between folds either(std less than 5bps with only a few runs for 5Fold; we mostly did 1 fold for faster iteration of ideas). If you don't mind, what's your cv std for different folds?", "votes": null }, { "id": "1507816", "postDate": "09/09/2021 15:07:23", "content": "I used CQT but according to past comments in the forum (I did not try CWT), there's not significant differences between the 2.
", "rawMarkdown": "I used CQT but according to past comments in the forum (I did not try CWT), there's not significant differences between the 2.", "votes": null }, { "id": "1507817", "postDate": "09/09/2021 15:07:41", "content": "@hengck23 Thanks! For the ones we submitted with our 1D model, CV ranges from 0.8697 to 0.8730
", "rawMarkdown": "hengck23 Thanks! For the ones we submitted with our 1D model, CV ranges from 0.8697 to 0.8730", "votes": null }, { "id": "1507822", "postDate": "09/09/2021 15:14:38", "content": "Most people use N-fold training with out-of-fold CV, and therefore use one model per one example on train test. When they submit to LB, they use N models per one example.
\nThats all.
", "rawMarkdown": "Most people use N-fold training with out-of-fold CV, and therefore use one model per one example on train test. When they submit to LB, they use N models per one example.\n\nThats all.", "votes": null }, { "id": "1507825", "postDate": "09/09/2021 15:16:45", "content": "\n\n\nthen we have cv low cv and lb, say 0.840. the value are low but generalisation gap is almost zero.
\n
so if I got it correct, what you suggest is that models with low CV-LB diff (gap) have better generalisation - and hence we should trust more towards final selection (?)
", "rawMarkdown": "hengck23 \n>then we have cv low cv and lb, say 0.840. the value are low but generalisation gap is almost zero.\n\nso if I got it correct, what you suggest is that models with low CV-LB diff (gap) have better generalisation - and hence we should trust more towards final selection (?)", "votes": null }, { "id": "1507833", "postDate": "09/09/2021 15:25:01", "content": "@richx86 congrats you managed to make 1D work!
\nI'm still struggling with wavenet/cnn1d models to make them converge - may I ask what is your input, i.e. if you use whole signal length (4096,3) or window of that?
@imeintanis Thanks! It's tricky to make 1D work. I used to have 0.5000 AUC. Since it's in the late stage of the competition, I will not share too much information here. Hope you would understand.
", "rawMarkdown": "imeintanis Thanks! It's tricky to make 1D work. I used to have 0.5000 AUC. Since it's in the late stage of the competition, I will not share too much information here. Hope you would understand.", "votes": null }, { "id": "1508158", "postDate": "09/10/2021 00:15:43", "content": "The weird thing is that 5Fold doesn't seem to help with increasing the gap for our 1D model.
", "rawMarkdown": "The weird thing is that 5Fold doesn't seem to help with increasing the gap for our 1D model.", "votes": null }, { "id": "1508182", "postDate": "09/10/2021 01:25:53", "content": "\n\"hence we should trust more towards final selection\"
\nif the model lb score is low it is not useful even it has a better generalization gap.
\nthe objective of model learning is to get BOTH the best accuracy AND best generalization.
", "rawMarkdown": "imeintanis \n\n\"hence we should trust more towards final selection\"\n\nif the model lb score is low it is not useful even it has a better generalization gap.\n\nthe objective of model learning is to get BOTH the best accuracy AND best generalization.", "votes": null }, { "id": "1508187", "postDate": "09/10/2021 01:37:46", "content": "@richx86
\n\"For the ones we submitted with our 1D model, CV ranges from 0.8697 to 0.8730\"
Then i would say 1D model generalizes better. I suppose you should also observe:
\nthe good news is that 1D and 2D focuses on different \"view\" of the data and you are likely to get a good boost at ensembling i think
", "rawMarkdown": "richx86 \n\"For the ones we submitted with our 1D model, CV ranges from 0.8697 to 0.8730\"\n\nThen i would say 1D model generalizes better. I suppose you should also observe:\n- std of AUC of k-fold model is lower \n- if you compare AUC for 100%,50%,25%, etc ... of the validation set, you should also see 1D model is less sensitive to the number of validation samples.\n\nthe good news is that 1D and 2D focuses on different \"view\" of the data and you are likely to get a good boost at ensembling i think", "votes": null }, { "id": "1508209", "postDate": "09/10/2021 02:26:36", "content": "\n\nThen i would say 1D model generalizes better. I suppose you should also observe:
\n
std of AUC of k-fold model is lower\nif you compare AUC for 100%,50%,25%, etc … of the validation set, you should also see 1D model is less sensitive to the number of validation samples.\n@hengck23 Thanks a lot for your insight! I will run a few more 5-folds of our models to check std and also AUC sensitivity for difference sizes of validation set. However, if 1D model generalizes better, shouldn't we see larger LB-CV gap for 1D? That's opposite of what we see right now.
\n\n\nthe good news is that 1D and 2D focuses on different \"view\" of the data and you are likely to get a good boost at ensembling i think
\n
We did see a nice boost for ensembling :). The correlation between 1D and 2D model is around 0.95. I guess that evidence support your statement.
", "rawMarkdown": "> Then i would say 1D model generalizes better. I suppose you should also observe:\n\n std of AUC of k-fold model is lower\n if you compare AUC for 100%,50%,25%, etc … of the validation set, you should also see 1D model is less sensitive to the number of validation samples.\n\n@hengck23 Thanks a lot for your insight! I will run a few more 5-folds of our models to check std and also AUC sensitivity for difference sizes of validation set. However, if 1D model generalizes better, shouldn't we see larger LB-CV gap for 1D? That's opposite of what we see right now. \n\n> the good news is that 1D and 2D focuses on different \"view\" of the data and you are likely to get a good boost at ensembling i think\n\nWe did see a nice boost for ensembling :). The correlation between 1D and 2D model is around 0.95. I guess that evidence support your statement.", "votes": null }, { "id": "1508217", "postDate": "09/10/2021 02:35:49", "content": "better generalisation means lower CV-LB gap (assume same distribution of test and train data)
\n\"also AUC sensitivity for difference sizes of validation set.\"
\ndo include this size:
\"This leaderboard is calculated with approximately 16% of the test data.
\nThe final results will be based on the other 84%, so the final standings may be different\"
on a side note the time localisation of both 1d CNN and 2D CNN should be the same. this setup consistency loss for training. maybe there can be some form of unsupervised, semi/weak unsupervised learning as well.
", "rawMarkdown": "better generalisation means lower CV-LB gap (assume same distribution of test and train data)\n\n\"also AUC sensitivity for difference sizes of validation set.\"\ndo include this size:\n\n\"This leaderboard is calculated with approximately 16% of the test data.\nThe final results will be based on the other 84%, so the final standings may be different\"\n\n---\n\non a side note the time localisation of both 1d CNN and 2D CNN should be the same. this setup consistency loss for training. maybe there can be some form of unsupervised, semi/weak unsupervised learning as well.", "votes": null }, { "id": "1508294", "postDate": "09/10/2021 05:30:12", "content": "@richx86 thanks! of course I understand, looking fwd to read your solution afterwards
", "rawMarkdown": "richx86 thanks! of course I understand, looking fwd to read your solution afterwards", "votes": null }, { "id": "1508679", "postDate": "09/10/2021 13:00:49", "content": "it is mentioned here:
\nhttps://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/271353
\n(see section on matched filter CNN)
noise is non stationary.
\n2d CNN can only learn stationary noise.
\n1d CNN however can work with nonstationary noise.
One hypothesis I have right now for my own question is that for 1D model and 2D model with similar performance(CV score), 1D outperform 2D on low SNR signals, while 2D outperform 1D on high SNR signals(which is the case for public LB).
", "rawMarkdown": "One hypothesis I have right now for my own question is that for 1D model and 2D model with similar performance(CV score), 1D outperform 2D on low SNR signals, while 2D outperform 1D on high SNR signals(which is the case for public LB).", "votes": null }, { "id": "1559894", "postDate": "10/27/2021 08:06:44", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
What do you mean by BPS ?
", "votes": null, "replies": [ { "id": 1507335, "author_name": "richx86", "author_url": "", "post_date": "09/09/2021 05:27:03", "content": "@mithilsalunkhe 0.0001 is one bp
", "votes": null, "replies": [] } ] }, { "id": 1507434, "author_name": "martynoveduard", "author_url": "", "post_date": "09/09/2021 07:43:10", "content": "It's not always the case, difference between CV and LB for me was less than 10 bps, even though I'm using 2D model.
\nCV : 0.8757, LB : 0.8766
\nWith release of new 0.8758 kernel it's hard to properly compute CV for now, so I can't judge on that.
As calculated the test data has relative smaller mean value, I guess that's why most of us got higer LB than CV, maybe test data has less noise level?
\n\n", "votes": null, "replies": [] }, { "id": 1507464, "author_name": "aerdem4", "author_url": "", "post_date": "09/09/2021 08:16:35", "content": "Traind data: global_mean, global_std = 4.3110404142640676e-26, 6.150793058115705e-21
\n
\n Test data: global_mean, global_std = 1.647648937086022e-26, 6.150793058115705e-21
CV-LB difference looks reasonable considering that the same difference can be observed between different folds.
", "votes": null, "replies": [ { "id": 1507810, "author_name": "richx86", "author_url": "", "post_date": "09/09/2021 14:59:27", "content": "We didn't see such big difference between folds either(std less than 5bps with only a few runs for 5Fold; we mostly did 1 fold for faster iteration of ideas). If you don't mind, what's your cv std for different folds?
", "votes": null, "replies": [] } ] }, { "id": 1507465, "author_name": "hengck23", "author_url": "", "post_date": "09/09/2021 08:18:52", "content": "you have to tell us the performance of model.
\nfor less discriminative model, the generalization is of course better.
\nas an extreme example, imagine we use a weak model like resnet18. then we have cv low cv and lb, say 0.840. the value are low but generalization gap is almost zero.
\nif a model is highly discriminative (e.g. very deep CNN or transformer), you will learn the \"noise in the data\" and generalization drop and overfitting kicks in (unless you use regularisation methods)
", "votes": null, "replies": [ { "id": 1507476, "author_name": "fabiendaniel", "author_url": "", "post_date": "09/09/2021 08:29:29", "content": "in fact, resnet18 is not that weak. My current best single model is a resnet18d with local CV = 0.874047 and LB = 0.8759
", "votes": null, "replies": [] }, { "id": 1507774, "author_name": "imeintanis", "author_url": "", "post_date": "09/09/2021 14:25:51", "content": "Wow, nice CV-LB @fabiendaniel !! If you like to share, you use CWT or CQT ?
", "votes": null, "replies": [] }, { "id": 1507816, "author_name": "fabiendaniel", "author_url": "", "post_date": "09/09/2021 15:07:23", "content": "I used CQT but according to past comments in the forum (I did not try CWT), there's not significant differences between the 2.
", "votes": null, "replies": [] }, { "id": 1507817, "author_name": "richx86", "author_url": "", "post_date": "09/09/2021 15:07:41", "content": "@hengck23 Thanks! For the ones we submitted with our 1D model, CV ranges from 0.8697 to 0.8730
", "votes": null, "replies": [] }, { "id": 1507825, "author_name": "imeintanis", "author_url": "", "post_date": "09/09/2021 15:16:45", "content": "\n\n\nthen we have cv low cv and lb, say 0.840. the value are low but generalisation gap is almost zero.
\n
so if I got it correct, what you suggest is that models with low CV-LB diff (gap) have better generalisation - and hence we should trust more towards final selection (?)
", "votes": null, "replies": [] }, { "id": 1507833, "author_name": "imeintanis", "author_url": "", "post_date": "09/09/2021 15:25:01", "content": "@richx86 congrats you managed to make 1D work!
\nI'm still struggling with wavenet/cnn1d models to make them converge - may I ask what is your input, i.e. if you use whole signal length (4096,3) or window of that?
@imeintanis Thanks! It's tricky to make 1D work. I used to have 0.5000 AUC. Since it's in the late stage of the competition, I will not share too much information here. Hope you would understand.
", "votes": null, "replies": [] }, { "id": 1508182, "author_name": "hengck23", "author_url": "", "post_date": "09/10/2021 01:25:53", "content": "\n\"hence we should trust more towards final selection\"
\nif the model lb score is low it is not useful even it has a better generalization gap.
\nthe objective of model learning is to get BOTH the best accuracy AND best generalization.
", "votes": null, "replies": [] }, { "id": 1508187, "author_name": "hengck23", "author_url": "", "post_date": "09/10/2021 01:37:46", "content": "@richx86
\n\"For the ones we submitted with our 1D model, CV ranges from 0.8697 to 0.8730\"
Then i would say 1D model generalizes better. I suppose you should also observe:
\nthe good news is that 1D and 2D focuses on different \"view\" of the data and you are likely to get a good boost at ensembling i think
", "votes": null, "replies": [] }, { "id": 1508209, "author_name": "richx86", "author_url": "", "post_date": "09/10/2021 02:26:36", "content": "\n\nThen i would say 1D model generalizes better. I suppose you should also observe:
\n
std of AUC of k-fold model is lower\nif you compare AUC for 100%,50%,25%, etc … of the validation set, you should also see 1D model is less sensitive to the number of validation samples.\n@hengck23 Thanks a lot for your insight! I will run a few more 5-folds of our models to check std and also AUC sensitivity for difference sizes of validation set. However, if 1D model generalizes better, shouldn't we see larger LB-CV gap for 1D? That's opposite of what we see right now.
\n\n\nthe good news is that 1D and 2D focuses on different \"view\" of the data and you are likely to get a good boost at ensembling i think
\n
We did see a nice boost for ensembling :). The correlation between 1D and 2D model is around 0.95. I guess that evidence support your statement.
", "votes": null, "replies": [] }, { "id": 1508217, "author_name": "hengck23", "author_url": "", "post_date": "09/10/2021 02:35:49", "content": "better generalisation means lower CV-LB gap (assume same distribution of test and train data)
\n\"also AUC sensitivity for difference sizes of validation set.\"
\ndo include this size:
\"This leaderboard is calculated with approximately 16% of the test data.
\nThe final results will be based on the other 84%, so the final standings may be different\"
on a side note the time localisation of both 1d CNN and 2D CNN should be the same. this setup consistency loss for training. maybe there can be some form of unsupervised, semi/weak unsupervised learning as well.
", "votes": null, "replies": [] }, { "id": 1508294, "author_name": "imeintanis", "author_url": "", "post_date": "09/10/2021 05:30:12", "content": "@richx86 thanks! of course I understand, looking fwd to read your solution afterwards
", "votes": null, "replies": [] } ] }, { "id": 1507822, "author_name": "fffrrt", "author_url": "", "post_date": "09/09/2021 15:14:38", "content": "Most people use N-fold training with out-of-fold CV, and therefore use one model per one example on train test. When they submit to LB, they use N models per one example.
\nThats all.
", "votes": null, "replies": [ { "id": 1508158, "author_name": "richx86", "author_url": "", "post_date": "09/10/2021 00:15:43", "content": "The weird thing is that 5Fold doesn't seem to help with increasing the gap for our 1D model.
", "votes": null, "replies": [] } ] }, { "id": 1508679, "author_name": "hengck23", "author_url": "", "post_date": "09/10/2021 13:00:49", "content": "it is mentioned here:
\nhttps://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/271353
\n(see section on matched filter CNN)
noise is non stationary.
\n2d CNN can only learn stationary noise.
\n1d CNN however can work with nonstationary noise.
One hypothesis I have right now for my own question is that for 1D model and 2D model with similar performance(CV score), 1D outperform 2D on low SNR signals, while 2D outperform 1D on high SNR signals(which is the case for public LB).
", "votes": null, "replies": [] }, { "id": 1559894, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:06:44", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris