![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2F6f91f11c5459eae070549214810e8146%2Fcv_lb.png?generation=1599639285730839&alt=media\")
I have been experimenting different things with the same validation method(sklearn f1score, both 'samples' and micro')
\nI feel lost as it seems only useful in comparing epoch within the SAME model structure, eg (epoch with better validation result have better LB)
\nBut failed to compare across model structure (eg model A have higher validation score than B but B actually do better in LB)
\nDon't know should I trust local validation or LB in this comp.
\nAnyone can share idea about better validation metrics ? much appreciate!
", "messages": [ { "id": "1003608", "postDate": "09/09/2020 05:57:24", "content": "I have been experimenting different things with the same validation method(sklearn f1score, both 'samples' and micro')
\nI feel lost as it seems only useful in comparing epoch within the SAME model structure, eg (epoch with better validation result have better LB)
\nBut failed to compare across model structure (eg model A have higher validation score than B but B actually do better in LB)
\nDon't know should I trust local validation or LB in this comp.
\nAnyone can share idea about better validation metrics ? much appreciate!
", "rawMarkdown": "I have been experimenting different things with the same validation method(sklearn f1score, both 'samples' and micro') \n\nI feel lost as it seems only useful in comparing epoch within the SAME model structure, eg (epoch with better validation result have better LB)\n\nBut failed to compare across model structure (eg model A have higher validation score than B but B actually do better in LB)\n\nDon't know should I trust local validation or LB in this comp.\n\nAnyone can share idea about better validation metrics ? much appreciate!", "votes": null }, { "id": "1003688", "postDate": "09/09/2020 07:09:49", "content": "My results in this competition were mixed. I dont know exactly which to trust more due to my experience of better CV scoring less on LB. Thats why I will chose once my highest LB and once my highest CV as submissions.
", "rawMarkdown": "My results in this competition were mixed. I dont know exactly which to trust more due to my experience of better CV scoring less on LB. Thats why I will chose once my highest LB and once my highest CV as submissions.", "votes": null }, { "id": "1003758", "postDate": "09/09/2020 08:15:20", "content": "My last few subs cv - lb correlation seems good, but it could be luck… I hope it is not luck.
\n
x is cv, y is lb. Thecv I report here is high because it is computed on first 5 seconds of validation clips.
", "rawMarkdown": "My last few subs cv - lb correlation seems good, but it could be luck... I hope it is not luck.\n\n \n\nx is cv, y is lb. Thecv I report here is high because it is computed on first 5 seconds of validation clips.", "votes": null }, { "id": "1003773", "postDate": "09/09/2020 08:31:58", "content": "Do you train also on 5 second clips?
", "rawMarkdown": "Do you train also on 5 second clips?", "votes": null }, { "id": "1003830", "postDate": "09/09/2020 09:37:12", "content": "I don't trust either :)
", "rawMarkdown": "I don't trust either :)", "votes": null }, { "id": "1003853", "postDate": "09/09/2020 09:57:38", "content": "You should give up this competition IMHO ;)
", "rawMarkdown": "You should give up this competition IMHO ;)", "votes": null }, { "id": "1003860", "postDate": "09/09/2020 10:06:24", "content": "I missed the \"giving up\" deadline unfortunately, it's too late now
", "rawMarkdown": "I missed the \"giving up\" deadline unfortunately, it's too late now", "votes": null }, { "id": "1003908", "postDate": "09/09/2020 10:56:19", "content": "It's never too late to get removed, you can use another account for instance…
", "rawMarkdown": "It's never too late to get removed, you can use another account for instance...", "votes": null }, { "id": "1003922", "postDate": "09/09/2020 11:09:09", "content": "![\"\"](\"https://thumbs.gfycat.com/SmoggyHilariousBaiji-size_restricted.gif\")
I think slight difference (~0.01) in LB doesn't matter. I also don't trust CV.
", "rawMarkdown": "I think slight difference (~0.01) in LB doesn't matter. I also don't trust CV.", "votes": null }, { "id": "1004284", "postDate": "09/09/2020 15:30:57", "content": "Sadly my deviation >0.1 in LB😂
\nCant wait to see your solution after comp ended😄
this looks great! Mine is a joke😭
", "rawMarkdown": "this looks great! Mine is a joke😭", "votes": null }, { "id": "1004309", "postDate": "09/09/2020 16:00:44", "content": "Can't improve my score for 11 days in a row 🤒 #sick
\nHigh chance top 5 teams will keep their positions. Also agree about ~0.01 gap but surprises might happen.
Indeed, the gap between first 5 and the rest looks big enough to secure their rank.
", "rawMarkdown": "Indeed, the gap between first 5 and the rest looks big enough to secure their rank.", "votes": null }, { "id": "1004382", "postDate": "09/09/2020 17:12:40", "content": "I increased my score by 0.005 for 20 days =(
\nAnd a small change reduces it by 0.01
We've been stuck for 11 days as well, nothing has been working since… There's a high chance our score will be enough to secure gold but anything can happen
", "rawMarkdown": "We've been stuck for 11 days as well, nothing has been working since... There's a high chance our score will be enough to secure gold but anything can happen", "votes": null }, { "id": "1004514", "postDate": "09/09/2020 19:05:23", "content": "\n\nDo you train also on 5 second clips?
\n
I suggest you look at high score public notebooks, but not the ones that blend others. In general I give the exact opposite advice but here these notebooks look quite good. The fact that it is hard to beat them (and I haven't beaten the best one yet) is worth taking into account.
", "rawMarkdown": "> Do you train also on 5 second clips?\n\nI suggest you look at high score public notebooks, but not the ones that blend others. In general I give the exact opposite advice but here these notebooks look quite good. The fact that it is hard to beat them (and I haven't beaten the best one yet) is worth taking into account.", "votes": null }, { "id": "1004551", "postDate": "09/09/2020 19:35:11", "content": "Here we're stuck since 5 days despite trying different ideas. It's like there are some \"levels\" in this competition (like in a game where you need to kill a boss at the end of each level 😊). LB 0.57 level which can be achieved quickly by regular approaches, 0.58 level can be reached with additional good training procedure. 0.59 level that can be reached with smarter training. You can be stuck for several days/weeks at the same level which is very frustrating. Next levels seems to be 0.60 and 0.61+ but it might need something new we did not discover yet.
", "rawMarkdown": "Here we're stuck since 5 days despite trying different ideas. It's like there are some \"*levels*\" in this competition (like in a game where you need to kill a boss at the end of each level 😊). LB 0.57 level which can be achieved quickly by regular approaches, 0.58 level can be reached with additional good training procedure. 0.59 level that can be reached with smarter training. You can be stuck for several days/weeks at the same level which is very frustrating. Next levels seems to be 0.60 and 0.61+ but it might need something new we did not discover yet.", "votes": null }, { "id": "1004556", "postDate": "09/09/2020 19:38:20", "content": "Or it is some way to overfit you didn't try.
\nI am not trying to cast doubt on top team work, and I sincerely hope they don't overfit. But we know test data is very different from train data, hence relying on cv only is not really an option. As a result we use LB feedback and it may be that we all overfit to some small test data.
", "rawMarkdown": "Or it is some way to overfit you didn't try.\n\nI am not trying to cast doubt on top team work, and I sincerely hope they don't overfit. But we know test data is very different from train data, hence relying on cv only is not really an option. As a result we use LB feedback and it may be that we all overfit to some small test data.", "votes": null }, { "id": "1004659", "postDate": "09/09/2020 22:50:28", "content": "There can be some overfitting for sure.
\nOn our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01. Well at least I hope.
\nAnd I think other people at the top also have pretty good models :)
", "rawMarkdown": "There can be some overfitting for sure. \n\nOn our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01. Well at least I hope. \n\nAnd I think other people at the top also have pretty good models :)", "votes": null }, { "id": "1004969", "postDate": "09/10/2020 07:06:26", "content": "\n\nOn our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01.
\n
You use the same test data each time, hence you cannot detect overfitting that way.
\n\n\nAnd I think other people at the top also have pretty good models :)
\n
I have no doubt about this.
", "rawMarkdown": "> On our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01. \n\nYou use the same test data each time, hence you cannot detect overfitting that way.\n\n> And I think other people at the top also have pretty good models :)\n\nI have no doubt about this.", "votes": null }, { "id": "1005342", "postDate": "09/10/2020 12:39:08", "content": "My cv f1 score is now 0.8458 and bce loss is 0.0041. Still stuck around 0.570. Need something different!
", "rawMarkdown": "My cv f1 score is now 0.8458 and bce loss is 0.0041. Still stuck around 0.570. Need something different!", "votes": null }, { "id": "1005354", "postDate": "09/10/2020 12:45:04", "content": "Do you use F1-Score with average='samples' and BCELossWithLogits?
Hi @fiyeroleung ,
\nWhat is the activation function did you use? Linear, sigmoid, or softmax?
", "rawMarkdown": "Hi @fiyeroleung , \n\nWhat is the activation function did you use? Linear, sigmoid, or softmax?", "votes": null }, { "id": "1005940", "postDate": "09/10/2020 22:40:09", "content": "sigmoid with threshold 0.5 for prediction
", "rawMarkdown": "sigmoid with threshold 0.5 for prediction", "votes": null }, { "id": "1005991", "postDate": "09/11/2020 01:11:49", "content": "@aliabdin1 I used average='macro' but it's almost sample if I changed to 'samples'.
\nYes, my loss is BCELossWithLogits.
Past experiences always show that you should put more trust on your CV than on the LB score. Keep in mind that LB score is only partial, and fitting the public leaderboard is nearly always leading to a very bad robustness to private lb shakeup.
", "rawMarkdown": "Past experiences always show that you should put more trust on your CV than on the LB score. Keep in mind that LB score is only partial, and fitting the public leaderboard is nearly always leading to a very bad robustness to private lb shakeup.", "votes": null }, { "id": "1006456", "postDate": "09/11/2020 09:10:23", "content": "After fixing quite a few issues and inconstancies in my preprocessing pipeline, my CV score is now within ~0.001 of my public LB score.
", "rawMarkdown": "After fixing quite a few issues and inconstancies in my preprocessing pipeline, my CV score is now within ~0.001 of my public LB score.", "votes": null }, { "id": "1006510", "postDate": "09/11/2020 10:08:26", "content": "That's awesome.
", "rawMarkdown": "That's awesome.", "votes": null }, { "id": "1006657", "postDate": "09/11/2020 13:10:26", "content": "@ramarlina gratz! do you mean map score or?
", "rawMarkdown": "ramarlina gratz! do you mean map score or?", "votes": null }, { "id": "1006794", "postDate": "09/11/2020 14:56:51", "content": "Top LB is moving … it looks 0.63 is going to be broken.
", "rawMarkdown": "Top LB is moving ... it looks 0.63 is going to be broken.", "votes": null }, { "id": "1006859", "postDate": "09/11/2020 15:45:41", "content": "Many top performers in this thread, hope you guys can break the limit!
\nIm totally out of the league and decide to enjoy the weekend
Don't give up, there will be a shakeup and it could be huge depending on how organizers split the public/private dataset.
", "rawMarkdown": "Don't give up, there will be a shakeup and it could be huge depending on how organizers split the public/private dataset.", "votes": null }, { "id": "1006903", "postDate": "09/11/2020 16:21:58", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2062758%2Fbfda21b2b6022b3c9e14f67b6ed3136c%2Fluck.png?generation=1599841260122550&alt=media\")
This is true only when you managed to get a CV setting that mimics test data. Your team probably did it, but many didn't.
", "rawMarkdown": "This is true only when you managed to get a CV setting that mimics test data. Your team probably did it, but many didn't.", "votes": null }, { "id": "1006915", "postDate": "09/11/2020 16:33:22", "content": "So sad, I try a lot, but the result does not improve)=
\nThis is my first competition, where can I get this secret ingredient? =)
Indeed it's really good for a first competition! and we have 4 days left.
", "rawMarkdown": "Indeed it's really good for a first competition! and we have 4 days left.", "votes": null }, { "id": "1006924", "postDate": "09/11/2020 16:43:40", "content": "8 to 10 submits left ^^
", "rawMarkdown": "8 to 10 submits left ^^", "votes": null }, { "id": "1010570", "postDate": "09/14/2020 22:05:19", "content": "Validating on first 5 seconds wasn't that reliable, I spent two days trying to create a CV that is correlated with LB. I should have done it way earlier as this is the only way to get sound progress. Given the number of time I advised people to focus on CV setting first, not doing it myself is hard to understand.
\nI am 100% certain that top teams have a reliable CV setting and therefore do not expect much shakeup, unless private test is very different from public test data.
\nWith 3 subs left I hope it will help us for ensembling.
\nHere are my last subs with cv as x, lb as y. All single models.
\nDon't focus on the high CV values, what matters is the correlation with LB.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2Fdfcec92801057eed209d67eb905e8dcc%2Fcv_lb.png?generation=1600121035663441&alt=media\")
@yaroshevskiy Why do you care about map when the metric is F1?
", "rawMarkdown": "yaroshevskiy Why do you care about map when the metric is F1?", "votes": null }, { "id": "1010597", "postDate": "09/14/2020 22:43:01", "content": "Very impressive CPMP, and impressive to all competitors who joined late and climbed the LB, you are all inspirations. Looking forward to writeups and hope stay in medal zone for private LB!
", "rawMarkdown": "Very impressive CPMP, and impressive to all competitors who joined late and climbed the LB, you are all inspirations. Looking forward to writeups and hope stay in medal zone for private LB!", "votes": null }, { "id": "1010612", "postDate": "09/14/2020 23:15:04", "content": "Thank you. As you hint, good public LB may not be an indication of good private LB… We will describe what we have done anyway.
", "rawMarkdown": "Thank you. As you hint, good public LB may not be an indication of good private LB... We will describe what we have done anyway.", "votes": null }, { "id": "1010919", "postDate": "09/15/2020 06:19:12", "content": "Very interesting. I have CV though it is not consistent with anything :D
", "rawMarkdown": "Very interesting. I have CV though it is not consistent with anything :D", "votes": null }, { "id": "1010920", "postDate": "09/15/2020 06:20:22", "content": "Twos subs left and I have ~5 ideas I wanted to try…
", "rawMarkdown": "Twos subs left and I have ~5 ideas I wanted to try...", "votes": null }, { "id": "1010986", "postDate": "09/15/2020 07:22:13", "content": "I'm confused to select best submission. For the one with best LB my threshold is very high, the other with ~0.575 the CV is not very good. Well its really hard to decide .
", "rawMarkdown": "I'm confused to select best submission. For the one with best LB my threshold is very high, the other with ~0.575 the CV is not very good. Well its really hard to decide .", "votes": null }, { "id": "1011201", "postDate": "09/15/2020 09:55:28", "content": "Since I failed to develop a reliable metrics to correlate CV and LB.
\nI will pick one with highest LB (i overfit it) and close my eyes to randomly pick another for the final submission😁😁😸
Fortunately you may select 2 submissions :)
", "rawMarkdown": "Fortunately you may select 2 submissions :)", "votes": null }, { "id": "1011340", "postDate": "09/15/2020 11:55:54", "content": "Yes, lets hope we don't face any shakeups & get best results.
", "rawMarkdown": "Yes, lets hope we don't face any shakeups & get best results.", "votes": null }, { "id": "1011357", "postDate": "09/15/2020 12:10:41", "content": "May the luck be with you 😸😸😸
", "rawMarkdown": "May the luck be with you 😸😸😸", "votes": null }, { "id": "1011568", "postDate": "09/15/2020 15:07:51", "content": "I also had same problem in earlier competition where i do not select medal winning notebooks and made bad picks
\nhere i select 1 with best CV and 1 with best Public LB… fingers cross. lets see what happen!
\nALL the BEST everyone
\nLets see how the shakeup is in some hours!!
", "rawMarkdown": "I also had same problem in earlier competition where i do not select medal winning notebooks and made bad picks\n\nhere i select 1 with best CV and 1 with best Public LB… fingers cross. lets see what happen!\n\nALL the BEST everyone\n\nLets see how the shakeup is in some hours!!", "votes": null }, { "id": "1011668", "postDate": "09/15/2020 16:06:03", "content": "🖖hope your team can achieve at least gold !
", "rawMarkdown": "🖖hope your team can achieve at least gold !", "votes": null }, { "id": "1011678", "postDate": "09/15/2020 16:14:14", "content": "🖖🖖gd luck.
\nI think money to gold zone wont change much. Half of the silver to bronze zone will shake up.😈
\nIm going to sleep now (mid night at my place), wake up tmr and check where will I end up with😭
all the best to you
\nand everyone!
@fiyeroleung At least gold 😍 Like this and thank you !
", "rawMarkdown": "fiyeroleung At least gold 😍 Like this and thank you !", "votes": null }, { "id": "1015593", "postDate": "09/18/2020 09:24:40", "content": "@louise2001 In your solution writeup Theo wrote:
\n\n\nAs we add no proper validation scheme, private LB was really a coinflip for us.
\n
Then why did you post a misleading statement about trusting CV and not public LB when you do the opposite yourself?
\nMisleading others is about the worst you can do on Kaggle.
\nI stayed polite but if I meet you then I'll say what I really think of this.
", "rawMarkdown": "louise2001 In your solution writeup Theo wrote:\n\n> As we add no proper validation scheme, private LB was really a coinflip for us. \n\nThen why did you post a misleading statement about trusting CV and not public LB when you do the opposite yourself? \n\nMisleading others is about the worst you can do on Kaggle. \n\nI stayed polite but if I meet you then I'll say what I really think of this.", "votes": null }, { "id": "1015632", "postDate": "09/18/2020 10:10:09", "content": "@cpmpml you seem to misunderstand @theoviel statements ! Of course we wasn't blind to the LB which was a great model quality assessor since this competition is all about domain shift (private distribution != public distribution). But we do have our internal validation schemes even if we could not say that they were the proper/best ones !
\nFor instance, @theo says:
\n\n\nWe had no reliable validation strategy, and used stratified 5 folds where the prediction is made on the 5 first second of the validation audios.
\n
We choose our final submissions based on both validation scores and public LB (as does many of us).
\nAnd more again @theo clearly states in this thread that :
\n\n\nI don't trust either :)
\n
Generally, It would be better to avoid blaming others for our own mistakes :)
", "rawMarkdown": "cpmpml you seem to misunderstand @theoviel statements ! Of course we wasn't blind to the LB which was a great model quality assessor since this competition is all about domain shift (private distribution != public distribution). But we do have our internal validation schemes even if we could not say that they were the proper/best ones !\n\nFor instance, @theo says:\n>We had no reliable validation strategy, and used stratified 5 folds where the prediction is made on the 5 first second of the validation audios.\n\nWe choose our final submissions based on both validation scores and public LB (as does many of us).\n\nAnd more again @theo clearly states in this thread that :\n> I don't trust either :)\n\nGenerally, It would be better to avoid blaming others for our own mistakes :)", "votes": null }, { "id": "1015639", "postDate": "09/18/2020 10:18:50", "content": "![](\"https://media0.giphy.com/media/hVTouq08miyVo1a21m/200.gif\")
",
"votes": null
},
{
"id": "1015658",
"postDate": "09/18/2020 10:37:17",
"content": "\n\nGenerally, It would be better to avoid blaming others for our own mistakes :)
\n
Please explain this.
", "rawMarkdown": "> Generally, It would be better to avoid blaming others for our own mistakes :)\n\nPlease explain this.", "votes": null }, { "id": "1015662", "postDate": "09/18/2020 10:42:41", "content": "It means that We don't feel like \"misleading others\" in this competition …
", "rawMarkdown": "It means that We don't feel like *\"misleading others\"* in this competition ...", "votes": null }, { "id": "1015667", "postDate": "09/18/2020 10:46:21", "content": "This was clear. But it is not at all what the sentence I ask about says. I guess you don't have the gut to be more explicit. So be it.
", "rawMarkdown": "This was clear. But it is not at all what the sentence I ask about says. I guess you don't have the gut to be more explicit. So be it.", "votes": null }, { "id": "1015672", "postDate": "09/18/2020 10:51:02", "content": "Hi @cpmpml, I'll send you an email about this issue.
\nI apologize for the contradictory statements between different members of our team, which reveal a lack communication between us.
\nThe purpose of Louise's comment was not to mislead anybody, I assume she wanted to give a vague and general advice that indeed is not really appropriate for this competition.
", "rawMarkdown": "Hi @cpmpml, I'll send you an email about this issue. \n\nI apologize for the contradictory statements between different members of our team, which reveal a lack communication between us. \n\nThe purpose of Louise's comment was not to mislead anybody, I assume she wanted to give a vague and general advice that indeed is not really appropriate for this competition.", "votes": null }, { "id": "1015679", "postDate": "09/18/2020 10:59:42", "content": "After Private LB reveal
\nI choose once my highest CV and once my highest LB at the end I missed choosing my best submission. I was trusting my CV (F1-Score with average='samples') slightly more. It came out, that my best non choosen submission had a CV of: 0.00975 and scored 0.536 on public LB but unexpectatly came on top of my submission on private LB scoring: 0.59 meaning that neither highest LB nor highest CV was an indication to chose this submission.
\nI did not spend a lot of time in this competition which also reflects my score but I still did not figure out what the relation between CV and LB was and why my better trained models were worse, showing no indication of overfitting what so ever.
", "rawMarkdown": "**After Private LB reveal**\n\nI choose once my highest CV and once my highest LB at the end I missed choosing my best submission. I was trusting my CV (F1-Score with average='samples') slightly more. It came out, that my best non choosen submission had a CV of: 0.00975 and scored 0.536 on public LB but unexpectatly came on top of my submission on private LB scoring: 0.59 meaning that neither highest LB nor highest CV was an indication to chose this submission.\n\nI did not spend a lot of time in this competition which also reflects my score but I still did not figure out what the relation between CV and LB was and why my better trained models were worse, showing no indication of overfitting what so ever.", "votes": null }, { "id": "1029603", "postDate": "09/27/2020 23:21:27", "content": "Hi @ramarlina
\nI've been wondering about this comment of yours for a long time.
\nHow did you make it possible?
I've tried many things, but in the end I couldn't achieve it.
\nNow that the competition is over, can you briefly tell me how to do it, if you would?
", "rawMarkdown": "Hi @ramarlina \n\nI've been wondering about this comment of yours for a long time.\nHow did you make it possible?\n\nI've tried many things, but in the end I couldn't achieve it.\n\nNow that the competition is over, can you briefly tell me how to do it, if you would?", "votes": null } ], "comments": [ { "id": 1003688, "author_name": "aliabdin1", "author_url": "", "post_date": "09/09/2020 07:09:49", "content": "My results in this competition were mixed. I dont know exactly which to trust more due to my experience of better CV scoring less on LB. Thats why I will chose once my highest LB and once my highest CV as submissions.
", "votes": null, "replies": [] }, { "id": 1003758, "author_name": "cpmpml", "author_url": "", "post_date": "09/09/2020 08:15:20", "content": "My last few subs cv - lb correlation seems good, but it could be luck… I hope it is not luck.
\n
x is cv, y is lb. Thecv I report here is high because it is computed on first 5 seconds of validation clips.
", "votes": null, "replies": [ { "id": 1003773, "author_name": "aliabdin1", "author_url": "", "post_date": "09/09/2020 08:31:58", "content": "Do you train also on 5 second clips?
", "votes": null, "replies": [] }, { "id": 1004287, "author_name": "fiyeroleung", "author_url": "", "post_date": "09/09/2020 15:34:19", "content": "this looks great! Mine is a joke😭
", "votes": null, "replies": [] }, { "id": 1004514, "author_name": "cpmpml", "author_url": "", "post_date": "09/09/2020 19:05:23", "content": "\n\nDo you train also on 5 second clips?
\n
I suggest you look at high score public notebooks, but not the ones that blend others. In general I give the exact opposite advice but here these notebooks look quite good. The fact that it is hard to beat them (and I haven't beaten the best one yet) is worth taking into account.
", "votes": null, "replies": [] } ] }, { "id": 1003830, "author_name": "theoviel", "author_url": "", "post_date": "09/09/2020 09:37:12", "content": "I don't trust either :)
", "votes": null, "replies": [ { "id": 1003853, "author_name": "cpmpml", "author_url": "", "post_date": "09/09/2020 09:57:38", "content": "You should give up this competition IMHO ;)
", "votes": null, "replies": [] }, { "id": 1003860, "author_name": "theoviel", "author_url": "", "post_date": "09/09/2020 10:06:24", "content": "I missed the \"giving up\" deadline unfortunately, it's too late now
", "votes": null, "replies": [] }, { "id": 1003908, "author_name": "cpmpml", "author_url": "", "post_date": "09/09/2020 10:56:19", "content": "It's never too late to get removed, you can use another account for instance…
", "votes": null, "replies": [] }, { "id": 1003922, "author_name": "theoviel", "author_url": "", "post_date": "09/09/2020 11:09:09", "content": "
I think slight difference (~0.01) in LB doesn't matter. I also don't trust CV.
", "votes": null, "replies": [ { "id": 1004284, "author_name": "fiyeroleung", "author_url": "", "post_date": "09/09/2020 15:30:57", "content": "Sadly my deviation >0.1 in LB😂
\nCant wait to see your solution after comp ended😄
Can't improve my score for 11 days in a row 🤒 #sick
\nHigh chance top 5 teams will keep their positions. Also agree about ~0.01 gap but surprises might happen.
Indeed, the gap between first 5 and the rest looks big enough to secure their rank.
", "votes": null, "replies": [] }, { "id": 1004382, "author_name": "vlomme", "author_url": "", "post_date": "09/09/2020 17:12:40", "content": "I increased my score by 0.005 for 20 days =(
\nAnd a small change reduces it by 0.01
We've been stuck for 11 days as well, nothing has been working since… There's a high chance our score will be enough to secure gold but anything can happen
", "votes": null, "replies": [] } ] }, { "id": 1004551, "author_name": "mpware", "author_url": "", "post_date": "09/09/2020 19:35:11", "content": "Here we're stuck since 5 days despite trying different ideas. It's like there are some \"levels\" in this competition (like in a game where you need to kill a boss at the end of each level 😊). LB 0.57 level which can be achieved quickly by regular approaches, 0.58 level can be reached with additional good training procedure. 0.59 level that can be reached with smarter training. You can be stuck for several days/weeks at the same level which is very frustrating. Next levels seems to be 0.60 and 0.61+ but it might need something new we did not discover yet.
", "votes": null, "replies": [ { "id": 1004556, "author_name": "cpmpml", "author_url": "", "post_date": "09/09/2020 19:38:20", "content": "Or it is some way to overfit you didn't try.
\nI am not trying to cast doubt on top team work, and I sincerely hope they don't overfit. But we know test data is very different from train data, hence relying on cv only is not really an option. As a result we use LB feedback and it may be that we all overfit to some small test data.
", "votes": null, "replies": [] }, { "id": 1004659, "author_name": "theoviel", "author_url": "", "post_date": "09/09/2020 22:50:28", "content": "There can be some overfitting for sure.
\nOn our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01. Well at least I hope.
\nAnd I think other people at the top also have pretty good models :)
", "votes": null, "replies": [] }, { "id": 1004969, "author_name": "cpmpml", "author_url": "", "post_date": "09/10/2020 07:06:26", "content": "\n\nOn our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01.
\n
You use the same test data each time, hence you cannot detect overfitting that way.
\n\n\nAnd I think other people at the top also have pretty good models :)
\n
I have no doubt about this.
", "votes": null, "replies": [] } ] }, { "id": 1005342, "author_name": "bamps53", "author_url": "", "post_date": "09/10/2020 12:39:08", "content": "My cv f1 score is now 0.8458 and bce loss is 0.0041. Still stuck around 0.570. Need something different!
", "votes": null, "replies": [ { "id": 1005354, "author_name": "aliabdin1", "author_url": "", "post_date": "09/10/2020 12:45:04", "content": "Do you use F1-Score with average='samples' and BCELossWithLogits?
@aliabdin1 I used average='macro' but it's almost sample if I changed to 'samples'.
\nYes, my loss is BCELossWithLogits.
Hi @fiyeroleung ,
\nWhat is the activation function did you use? Linear, sigmoid, or softmax?
", "votes": null, "replies": [ { "id": 1005940, "author_name": "fiyeroleung", "author_url": "", "post_date": "09/10/2020 22:40:09", "content": "sigmoid with threshold 0.5 for prediction
", "votes": null, "replies": [] } ] }, { "id": 1006412, "author_name": "louise2001", "author_url": "", "post_date": "09/11/2020 08:29:17", "content": "Past experiences always show that you should put more trust on your CV than on the LB score. Keep in mind that LB score is only partial, and fitting the public leaderboard is nearly always leading to a very bad robustness to private lb shakeup.
", "votes": null, "replies": [ { "id": 1006906, "author_name": "cpmpml", "author_url": "", "post_date": "09/11/2020 16:22:48", "content": "This is true only when you managed to get a CV setting that mimics test data. Your team probably did it, but many didn't.
", "votes": null, "replies": [] }, { "id": 1015593, "author_name": "cpmpml", "author_url": "", "post_date": "09/18/2020 09:24:40", "content": "@louise2001 In your solution writeup Theo wrote:
\n\n\nAs we add no proper validation scheme, private LB was really a coinflip for us.
\n
Then why did you post a misleading statement about trusting CV and not public LB when you do the opposite yourself?
\nMisleading others is about the worst you can do on Kaggle.
\nI stayed polite but if I meet you then I'll say what I really think of this.
", "votes": null, "replies": [] }, { "id": 1015632, "author_name": "kneroma", "author_url": "", "post_date": "09/18/2020 10:10:09", "content": "@cpmpml you seem to misunderstand @theoviel statements ! Of course we wasn't blind to the LB which was a great model quality assessor since this competition is all about domain shift (private distribution != public distribution). But we do have our internal validation schemes even if we could not say that they were the proper/best ones !
\nFor instance, @theo says:
\n\n\nWe had no reliable validation strategy, and used stratified 5 folds where the prediction is made on the 5 first second of the validation audios.
\n
We choose our final submissions based on both validation scores and public LB (as does many of us).
\nAnd more again @theo clearly states in this thread that :
\n\n\nI don't trust either :)
\n
Generally, It would be better to avoid blaming others for our own mistakes :)
", "votes": null, "replies": [] }, { "id": 1015639, "author_name": "aliabdin1", "author_url": "", "post_date": "09/18/2020 10:18:50", "content": "
\n\nGenerally, It would be better to avoid blaming others for our own mistakes :)
\n
Please explain this.
", "votes": null, "replies": [] }, { "id": 1015662, "author_name": "kneroma", "author_url": "", "post_date": "09/18/2020 10:42:41", "content": "It means that We don't feel like \"misleading others\" in this competition …
", "votes": null, "replies": [] }, { "id": 1015667, "author_name": "cpmpml", "author_url": "", "post_date": "09/18/2020 10:46:21", "content": "This was clear. But it is not at all what the sentence I ask about says. I guess you don't have the gut to be more explicit. So be it.
", "votes": null, "replies": [] }, { "id": 1015672, "author_name": "theoviel", "author_url": "", "post_date": "09/18/2020 10:51:02", "content": "Hi @cpmpml, I'll send you an email about this issue.
\nI apologize for the contradictory statements between different members of our team, which reveal a lack communication between us.
\nThe purpose of Louise's comment was not to mislead anybody, I assume she wanted to give a vague and general advice that indeed is not really appropriate for this competition.
", "votes": null, "replies": [] } ] }, { "id": 1006456, "author_name": "ramarlina", "author_url": "", "post_date": "09/11/2020 09:10:23", "content": "After fixing quite a few issues and inconstancies in my preprocessing pipeline, my CV score is now within ~0.001 of my public LB score.
", "votes": null, "replies": [ { "id": 1006510, "author_name": "cpmpml", "author_url": "", "post_date": "09/11/2020 10:08:26", "content": "That's awesome.
", "votes": null, "replies": [] }, { "id": 1006657, "author_name": "yaroshevskiy", "author_url": "", "post_date": "09/11/2020 13:10:26", "content": "@ramarlina gratz! do you mean map score or?
", "votes": null, "replies": [] }, { "id": 1010572, "author_name": "cpmpml", "author_url": "", "post_date": "09/14/2020 22:06:49", "content": "@yaroshevskiy Why do you care about map when the metric is F1?
", "votes": null, "replies": [] }, { "id": 1010919, "author_name": "gaborfodor", "author_url": "", "post_date": "09/15/2020 06:19:12", "content": "Very interesting. I have CV though it is not consistent with anything :D
", "votes": null, "replies": [] }, { "id": 1029603, "author_name": "fkubota", "author_url": "", "post_date": "09/27/2020 23:21:27", "content": "Hi @ramarlina
\nI've been wondering about this comment of yours for a long time.
\nHow did you make it possible?
I've tried many things, but in the end I couldn't achieve it.
\nNow that the competition is over, can you briefly tell me how to do it, if you would?
", "votes": null, "replies": [] } ] }, { "id": 1006794, "author_name": "mpware", "author_url": "", "post_date": "09/11/2020 14:56:51", "content": "Top LB is moving … it looks 0.63 is going to be broken.
", "votes": null, "replies": [ { "id": 1006859, "author_name": "fiyeroleung", "author_url": "", "post_date": "09/11/2020 15:45:41", "content": "Many top performers in this thread, hope you guys can break the limit!
\nIm totally out of the league and decide to enjoy the weekend
Don't give up, there will be a shakeup and it could be huge depending on how organizers split the public/private dataset.
", "votes": null, "replies": [] }, { "id": 1006903, "author_name": "theoviel", "author_url": "", "post_date": "09/11/2020 16:21:58", "content": "
So sad, I try a lot, but the result does not improve)=
\nThis is my first competition, where can I get this secret ingredient? =)
Indeed it's really good for a first competition! and we have 4 days left.
", "votes": null, "replies": [] }, { "id": 1006924, "author_name": "yaroshevskiy", "author_url": "", "post_date": "09/11/2020 16:43:40", "content": "8 to 10 submits left ^^
", "votes": null, "replies": [] }, { "id": 1010920, "author_name": "gaborfodor", "author_url": "", "post_date": "09/15/2020 06:20:22", "content": "Twos subs left and I have ~5 ideas I wanted to try…
", "votes": null, "replies": [] } ] }, { "id": 1010570, "author_name": "cpmpml", "author_url": "", "post_date": "09/14/2020 22:05:19", "content": "Validating on first 5 seconds wasn't that reliable, I spent two days trying to create a CV that is correlated with LB. I should have done it way earlier as this is the only way to get sound progress. Given the number of time I advised people to focus on CV setting first, not doing it myself is hard to understand.
\nI am 100% certain that top teams have a reliable CV setting and therefore do not expect much shakeup, unless private test is very different from public test data.
\nWith 3 subs left I hope it will help us for ensembling.
\nHere are my last subs with cv as x, lb as y. All single models.
\nDon't focus on the high CV values, what matters is the correlation with LB.
\n
Very impressive CPMP, and impressive to all competitors who joined late and climbed the LB, you are all inspirations. Looking forward to writeups and hope stay in medal zone for private LB!
", "votes": null, "replies": [] }, { "id": 1010612, "author_name": "cpmpml", "author_url": "", "post_date": "09/14/2020 23:15:04", "content": "Thank you. As you hint, good public LB may not be an indication of good private LB… We will describe what we have done anyway.
", "votes": null, "replies": [] } ] }, { "id": 1010986, "author_name": "rsinda", "author_url": "", "post_date": "09/15/2020 07:22:13", "content": "I'm confused to select best submission. For the one with best LB my threshold is very high, the other with ~0.575 the CV is not very good. Well its really hard to decide .
", "votes": null, "replies": [ { "id": 1011209, "author_name": "gaborfodor", "author_url": "", "post_date": "09/15/2020 10:08:07", "content": "Fortunately you may select 2 submissions :)
", "votes": null, "replies": [] }, { "id": 1011340, "author_name": "rsinda", "author_url": "", "post_date": "09/15/2020 11:55:54", "content": "Yes, lets hope we don't face any shakeups & get best results.
", "votes": null, "replies": [] } ] }, { "id": 1011201, "author_name": "fiyeroleung", "author_url": "", "post_date": "09/15/2020 09:55:28", "content": "Since I failed to develop a reliable metrics to correlate CV and LB.
\nI will pick one with highest LB (i overfit it) and close my eyes to randomly pick another for the final submission😁😁😸
May the luck be with you 😸😸😸
", "votes": null, "replies": [] }, { "id": 1011668, "author_name": "fiyeroleung", "author_url": "", "post_date": "09/15/2020 16:06:03", "content": "🖖hope your team can achieve at least gold !
", "votes": null, "replies": [] }, { "id": 1012051, "author_name": "vicioussong", "author_url": "", "post_date": "09/15/2020 21:49:20", "content": "@fiyeroleung At least gold 😍 Like this and thank you !
", "votes": null, "replies": [] } ] }, { "id": 1011568, "author_name": "kmldas", "author_url": "", "post_date": "09/15/2020 15:07:51", "content": "I also had same problem in earlier competition where i do not select medal winning notebooks and made bad picks
\nhere i select 1 with best CV and 1 with best Public LB… fingers cross. lets see what happen!
\nALL the BEST everyone
\nLets see how the shakeup is in some hours!!
", "votes": null, "replies": [ { "id": 1011678, "author_name": "fiyeroleung", "author_url": "", "post_date": "09/15/2020 16:14:14", "content": "🖖🖖gd luck.
\nI think money to gold zone wont change much. Half of the silver to bronze zone will shake up.😈
\nIm going to sleep now (mid night at my place), wake up tmr and check where will I end up with😭
all the best to you
\nand everyone!
After Private LB reveal
\nI choose once my highest CV and once my highest LB at the end I missed choosing my best submission. I was trusting my CV (F1-Score with average='samples') slightly more. It came out, that my best non choosen submission had a CV of: 0.00975 and scored 0.536 on public LB but unexpectatly came on top of my submission on private LB scoring: 0.59 meaning that neither highest LB nor highest CV was an indication to chose this submission.
\nI did not spend a lot of time in this competition which also reflects my score but I still did not figure out what the relation between CV and LB was and why my better trained models were worse, showing no indication of overfitting what so ever.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1003608": "I have been experimenting different things with the same validation method(sklearn f1score, both 'samples' and micro') \n\nI feel lost as it seems only useful in comparing epoch within the SAME model structure, eg (epoch with better validation result have better LB)\n\nBut failed to compare across model structure (eg model A have higher validation score than B but B actually do better in LB)\n\nDon't know should I trust local validation or LB in this comp.\n\nAnyone can share idea about better validation metrics ? much appreciate!", "1003688": "My results in this competition were mixed. I dont know exactly which to trust more due to my experience of better CV scoring less on LB. Thats why I will chose once my highest LB and once my highest CV as submissions.", "1003758": "My last few subs cv - lb correlation seems good, but it could be luck... I hope it is not luck.\n\n \n\nx is cv, y is lb. Thecv I report here is high because it is computed on first 5 seconds of validation clips.", "1003773": "Do you train also on 5 second clips?", "1003830": "I don't trust either :)", "1003853": "You should give up this competition IMHO ;)", "1003860": "I missed the \"giving up\" deadline unfortunately, it's too late now", "1003908": "It's never too late to get removed, you can use another account for instance...", "1003922": "", "1004119": "I think slight difference (~0.01) in LB doesn't matter. I also don't trust CV.", "1004284": "Sadly my deviation >0.1 in LB😂\nCant wait to see your solution after comp ended😄", "1004287": "this looks great! Mine is a joke😭", "1004309": "Can't improve my score for 11 days in a row 🤒 #sick \nHigh chance top 5 teams will keep their positions. Also agree about ~0.01 gap but surprises might happen.", "1004315": "Indeed, the gap between first 5 and the rest looks big enough to secure their rank.", "1004382": "I increased my score by 0.005 for 20 days =(\nAnd a small change reduces it by 0.01", "1004403": "We've been stuck for 11 days as well, nothing has been working since... There's a high chance our score will be enough to secure gold but anything can happen", "1004514": "> Do you train also on 5 second clips?\n\nI suggest you look at high score public notebooks, but not the ones that blend others. In general I give the exact opposite advice but here these notebooks look quite good. The fact that it is hard to beat them (and I haven't beaten the best one yet) is worth taking into account.", "1004551": "Here we're stuck since 5 days despite trying different ideas. It's like there are some \"*levels*\" in this competition (like in a game where you need to kill a boss at the end of each level 😊). LB 0.57 level which can be achieved quickly by regular approaches, 0.58 level can be reached with additional good training procedure. 0.59 level that can be reached with smarter training. You can be stuck for several days/weeks at the same level which is very frustrating. Next levels seems to be 0.60 and 0.61+ but it might need something new we did not discover yet.", "1004556": "Or it is some way to overfit you didn't try.\n\nI am not trying to cast doubt on top team work, and I sincerely hope they don't overfit. But we know test data is very different from train data, hence relying on cv only is not really an option. As a result we use LB feedback and it may be that we all overfit to some small test data.", "1004659": "There can be some overfitting for sure. \n\nOn our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01. Well at least I hope. \n\nAnd I think other people at the top also have pretty good models :)", "1004969": "> On our side we're getting consistent at scoring 0.6+ so I'm guessing we're not overfitting by more than 0,01. \n\nYou use the same test data each time, hence you cannot detect overfitting that way.\n\n> And I think other people at the top also have pretty good models :)\n\nI have no doubt about this.", "1005342": "My cv f1 score is now 0.8458 and bce loss is 0.0041. Still stuck around 0.570. Need something different!", "1005354": "Do you use F1-Score with `average='samples'` and `BCELossWithLogits`?", "1005889": "Hi @fiyeroleung , \n\nWhat is the activation function did you use? Linear, sigmoid, or softmax?", "1005940": "sigmoid with threshold 0.5 for prediction", "1005991": "aliabdin1 I used average='macro' but it's almost sample if I changed to 'samples'.\nYes, my loss is BCELossWithLogits.", "1006412": "Past experiences always show that you should put more trust on your CV than on the LB score. Keep in mind that LB score is only partial, and fitting the public leaderboard is nearly always leading to a very bad robustness to private lb shakeup.", "1006456": "After fixing quite a few issues and inconstancies in my preprocessing pipeline, my CV score is now within ~0.001 of my public LB score.", "1006510": "That's awesome.", "1006657": "ramarlina gratz! do you mean map score or?", "1006794": "Top LB is moving ... it looks 0.63 is going to be broken.", "1006859": "Many top performers in this thread, hope you guys can break the limit!\nIm totally out of the league and decide to enjoy the weekend", "1006870": "Don't give up, there will be a shakeup and it could be huge depending on how organizers split the public/private dataset.", "1006903": "", "1006906": "This is true only when you managed to get a CV setting that mimics test data. Your team probably did it, but many didn't.", "1006915": "So sad, I try a lot, but the result does not improve)= \nThis is my first competition, where can I get this secret ingredient? =)", "1006920": "Indeed it's really good for a first competition! and we have 4 days left.", "1006924": "8 to 10 submits left ^^", "1010570": "Validating on first 5 seconds wasn't that reliable, I spent two days trying to create a CV that is correlated with LB. I should have done it way earlier as this is the only way to get sound progress. Given the number of time I advised people to focus on CV setting first, not doing it myself is hard to understand.\n\nI am 100% certain that top teams have a reliable CV setting and therefore do not expect much shakeup, unless private test is very different from public test data.\n\nWith 3 subs left I hope it will help us for ensembling.\n\nHere are my last subs with cv as x, lb as y. All single models.\n\nDon't focus on the high CV values, what matters is the correlation with LB.\n\n", "1010572": "yaroshevskiy Why do you care about map when the metric is F1?", "1010597": "Very impressive CPMP, and impressive to all competitors who joined late and climbed the LB, you are all inspirations. Looking forward to writeups and hope stay in medal zone for private LB!", "1010612": "Thank you. As you hint, good public LB may not be an indication of good private LB... We will describe what we have done anyway.", "1010919": "Very interesting. I have CV though it is not consistent with anything :D", "1010920": "Twos subs left and I have ~5 ideas I wanted to try...", "1010986": "I'm confused to select best submission. For the one with best LB my threshold is very high, the other with ~0.575 the CV is not very good. Well its really hard to decide .", "1011201": "Since I failed to develop a reliable metrics to correlate CV and LB. \nI will pick one with highest LB (i overfit it) and close my eyes to randomly pick another for the final submission😁😁😸", "1011209": "Fortunately you may select 2 submissions :)", "1011340": "Yes, lets hope we don't face any shakeups & get best results.", "1011357": "May the luck be with you 😸😸😸", "1011568": "I also had same problem in earlier competition where i do not select medal winning notebooks and made bad picks\n\nhere i select 1 with best CV and 1 with best Public LB… fingers cross. lets see what happen!\n\nALL the BEST everyone\n\nLets see how the shakeup is in some hours!!", "1011668": "🖖hope your team can achieve at least gold !", "1011678": "🖖🖖gd luck. \nI think money to gold zone wont change much. Half of the silver to bronze zone will shake up.😈 \nIm going to sleep now (mid night at my place), wake up tmr and check where will I end up with😭", "1011682": "all the best to you\nand everyone!", "1012051": "fiyeroleung At least gold 😍 Like this and thank you !", "1015593": "louise2001 In your solution writeup Theo wrote:\n\n> As we add no proper validation scheme, private LB was really a coinflip for us. \n\nThen why did you post a misleading statement about trusting CV and not public LB when you do the opposite yourself? \n\nMisleading others is about the worst you can do on Kaggle. \n\nI stayed polite but if I meet you then I'll say what I really think of this.", "1015632": "cpmpml you seem to misunderstand @theoviel statements ! Of course we wasn't blind to the LB which was a great model quality assessor since this competition is all about domain shift (private distribution != public distribution). But we do have our internal validation schemes even if we could not say that they were the proper/best ones !\n\nFor instance, @theo says:\n>We had no reliable validation strategy, and used stratified 5 folds where the prediction is made on the 5 first second of the validation audios.\n\nWe choose our final submissions based on both validation scores and public LB (as does many of us).\n\nAnd more again @theo clearly states in this thread that :\n> I don't trust either :)\n\nGenerally, It would be better to avoid blaming others for our own mistakes :)", "1015639": "",
"1015658": "> Generally, It would be better to avoid blaming others for our own mistakes :)\n\nPlease explain this.",
"1015662": "It means that We don't feel like *\"misleading others\"* in this competition ...",
"1015667": "This was clear. But it is not at all what the sentence I ask about says. I guess you don't have the gut to be more explicit. So be it.",
"1015672": "Hi @cpmpml, I'll send you an email about this issue. \n\nI apologize for the contradictory statements between different members of our team, which reveal a lack communication between us. \n\nThe purpose of Louise's comment was not to mislead anybody, I assume she wanted to give a vague and general advice that indeed is not really appropriate for this competition.",
"1015679": "**After Private LB reveal**\n\nI choose once my highest CV and once my highest LB at the end I missed choosing my best submission. I was trusting my CV (F1-Score with average='samples') slightly more. It came out, that my best non choosen submission had a CV of: 0.00975 and scored 0.536 on public LB but unexpectatly came on top of my submission on private LB scoring: 0.59 meaning that neither highest LB nor highest CV was an indication to chose this submission.\n\nI did not spend a lot of time in this competition which also reflects my score but I still did not figure out what the relation between CV and LB was and why my better trained models were worse, showing no indication of overfitting what so ever.",
"1029603": "Hi @ramarlina \n\nI've been wondering about this comment of yours for a long time.\nHow did you make it possible?\n\nI've tried many things, but in the end I couldn't achieve it.\n\nNow that the competition is over, can you briefly tell me how to do it, if you would?"
},
"source": "meta"
}