I got an idea this notebook. Thanks for sharing!!
\nmy baseline
\nCV:0.96806077 LB:0.96
Split: StratifiedKFold 4 folds
\nmodel:efnetB0
\nno resize
\nno augumentaion
\nepoch 1
…epoch1→５ CV:0.97831633691 LB:0.96(higher than naive baseline)
\n…epoch1→10 add aug CV:0.98497151 LB:0.97
\n…epoch1→10 add another aug CV:0.9821733582 LB:0.97(lower than above)
I am suffering from the gap between CV and LB and over confidence….
", "messages": [ { "id": 1304897, "postDate": "2021-05-13T01:08:34.707Z", "content": "I got an idea this notebook. Thanks for sharing!!
\nmy baseline
\nCV:0.96806077 LB:0.96
Split: StratifiedKFold 4 folds
\nmodel:efnetB0
\nno resize
\nno augumentaion
\nepoch 1
…epoch1→５ CV:0.97831633691 LB:0.96(higher than naive baseline)
\n…epoch1→10 add aug CV:0.98497151 LB:0.97
\n…epoch1→10 add another aug CV:0.9821733582 LB:0.97(lower than above)
I am suffering from the gap between CV and LB and over confidence….
", "rawMarkdown": "I got an idea [this notebook](https://www.kaggle.com/yasufuminakama/seti-nfnet-l0-starter-training). Thanks for sharing!!\n\nmy baseline\nCV:0.96806077 LB:0.96\n\n Split: StratifiedKFold 4 folds\nmodel:efnetB0\nno resize\nno augumentaion\nepoch 1\n\n...epoch1→５ CV:0.97831633691 LB:0.96(higher than naive baseline)\n...epoch1→10 add aug CV:0.98497151 LB:0.97\n...epoch1→10 add another aug CV:0.9821733582 LB:0.97(lower than above)\n\nI am suffering from the gap between CV and LB and over confidence....\n\n", "votes": 54 }, { "id": 1326592, "postDate": "2021-05-28T14:56:50.077Z", "content": "I think the gap between CV and LB is not only a frustating annoyance, but the key to understand the real challenge of this competition
\nI'am astrophysicis myself, but not related at the SETI project in anyway, and this sentence has been in my mind from the beginning
\n\"While it would be nice to train our algorithms entirely on observations of interplanetary spacecraft, there are not many examples of them, and we also want to be able to find a wider range of signal types. So we’ve turned to simulating technosignature candidates.\"
\nSo, they are specially interested in unknown signals…. literally, they don't know what are looking for !!!
\nThis is only my personal opinion, but if i were one of them i would expose the next problem
\nOBJECTIVE: Finding anomalies that we don't know how they are
\nBuild a Dataset with simulated signals based on the research and information gathered over the last 40 years
Ask the smart guys of the SETI project to build a Neural Network to dectected them…. They already are really good, as you can see in this post of @manabendrarout
https://www.kaggle.com/c/seti-breakthrough-listen/discussion/239245
\nTest the Dataset with this models to achieve a CV/LB greater of 0.99… without gap logically because is the same Dataset for both
Split the original Dataset in Train/Test folds randmnly
**Add a litte batch of positive samples at the Test Dataset **that don't match any kind of pattern of the positive original targets… possibly REAL samples (only guessing).
Check that there is a gap between the CV (testing the train samples) and the LB (testing the test samples) whit the NN of the smart SETI guys
To understand the gap we have to analyze the confussion MAtrix. Por example, with my best model I achiveve CV = 0.988 (and LB = 0.96, but that is not important here).
\nThe confussion matrix say me that:
\n
Relation between False Positives / True Positives (used to compute the ROC Curve) taking true positive when the probability is greater than 0.5 is 21 /664 = 3.16%… A really good result !!
\nBut, the relation between False NEgative / True Positives is 62 / 664 = 9.3 %… not so good
\nSo, even with a magnific 0.988 I fail to detect about the 10% of samples !!!
\nIf, how I am guessing in point 5, they add some new samples at the test dataset, then this relation FN/TP will growth fast in this dataset, but the LB only will decrease a bit…
\nCONCLUSION: SETI guys have found the way to hidde the REAL samples !!… and detect them is the real challenge of this competition
\nI think they are not intereseted in a high CV or LB score (if it's high enough, ie. >0.97), but the submissions with the minimum gap, because they mean they are detecting the interesting REAL samples
\nLoking the submissions and the comments, and Making some fast computations I guess there are about 5 - 10 % of \"REAL samples\", that traslates a 1 % of CV/LB gap
\nMore of two hundred people have LB = 0.97, but only one have LB = 0.99… becasuse this 1% gap.
\nSorry for the long post, and remember I am only guessing here for fun
", "rawMarkdown": "I think the gap between CV and LB is not only a frustating annoyance, but the key to understand the real challenge of this competition\n\nI'am astrophysicis myself, but not related at the SETI project in anyway, and this sentence has been in my mind from the beginning\n\n\"While it would be nice to train our algorithms entirely on observations of interplanetary spacecraft, there are not many examples of them, and we also want to be able to find a wider range of signal types. So we’ve turned to simulating technosignature candidates.\"\n\nSo, they are specially interested in unknown signals.... literally, they don't know what are looking for !!!\n\nThis is only my personal opinion, but if i were one of them i would expose the next problem \n\nOBJECTIVE: Finding anomalies that we don't know how they are\n1. Build a Dataset with simulated signals based on the research and information gathered over the last 40 years\n\n2. Ask the smart guys of the SETI project to build a Neural Network to dectected them.... They already are really good, as you can see in this post of @manabendrarout\n\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/239245\n\n3. Test the Dataset with this models to achieve a CV/LB greater of 0.99... without gap logically because is the same Dataset for both\n\n4. Split the original Dataset in Train/Test folds randmnly\n\n5. **Add a litte batch of positive samples at the Test Dataset **that don't match any kind of pattern of the positive original targets... possibly REAL samples (only guessing).\n\n6. Check that there is a gap between the CV (testing the train samples) and the LB (testing the test samples) whit the NN of the smart SETI guys\n\nTo understand the gap we have to analyze the confussion MAtrix. Por example, with my best model I achiveve CV = 0.988 (and LB = 0.96, but that is not important here).\n\nThe confussion matrix say me that:\n\n\nRelation between False Positives / True Positives (used to compute the ROC Curve) taking true positive when the probability is greater than 0.5 is 21 /664 = 3.16%... A really good result !!\n\nBut, the relation between False NEgative / True Positives is 62 / 664 = 9.3 %... not so good\n\nSo, even with a magnific 0.988 I fail to detect about the 10% of samples !!!\n\nIf, how I am guessing in point 5, they add some new samples at the test dataset, then this relation FN/TP will growth fast in this dataset, but the LB only will decrease a bit...\n\nCONCLUSION: SETI guys have found the way to hidde the REAL samples !!... and detect them is the real challenge of this competition\n\nI think they are not intereseted in a high CV or LB score (if it's high enough, ie. >0.97), but the submissions with the minimum gap, because they mean they are detecting the interesting REAL samples\n\nLoking the submissions and the comments, and Making some fast computations I guess there are about 5 - 10 % of \"REAL samples\", that traslates a 1 % of CV/LB gap\n\nMore of two hundred people have LB = 0.97, but only one have LB = 0.99... becasuse this 1% gap.\n\nSorry for the long post, and remember I am only guessing here for fun", "votes": 17 }, { "id": 1312002, "postDate": "2021-05-17T18:49:05.373Z", "content": "Submitted a single fold (CV ~0.987), LB is 0.97.
\nImage: 256 x 256.
\nModel: b0.
\nSpatial or Channel: spatial.
\nAugmentations: Mixup (added).
\nTricks: maybe
Tricks: maybe😂
", "rawMarkdown": "Tricks: maybe😂", "votes": 3 }, { "id": 1313918, "postDate": "2021-05-18T19:50:55.187Z", "content": "I share my training logs here:
\nTue May 18 02:11:48 2021 Fold 0, Epoch 1, lr: 0.0001000, loss_train: 0.25995, loss_valid: 0.12689, auc: 0.945531.\nTue May 18 02:37:37 2021 Fold 0, Epoch 2, lr: 0.0010000, loss_train: 0.20199, loss_valid: 0.10548, auc: 0.964634.\nTue May 18 03:03:06 2021 Fold 0, Epoch 3, lr: 0.0010000, loss_train: 0.17507, loss_valid: 0.09838, auc: 0.966518.\nTue May 18 03:28:49 2021 Fold 0, Epoch 4, lr: 0.0009844, loss_train: 0.15932, loss_valid: 0.08149, auc: 0.966431.\nTue May 18 03:54:29 2021 Fold 0, Epoch 5, lr: 0.0009652, loss_train: 0.14819, loss_valid: 0.06801, auc: 0.978557.\nTue May 18 04:19:59 2021 Fold 0, Epoch 6, lr: 0.0009388, loss_train: 0.14161, loss_valid: 0.07503, auc: 0.974530.\nTue May 18 04:45:34 2021 Fold 0, Epoch 7, lr: 0.0009055, loss_train: 0.13015, loss_valid: 0.06789, auc: 0.973706.\nTue May 18 05:11:09 2021 Fold 0, Epoch 8, lr: 0.0008658, loss_train: 0.12650, loss_valid: 0.07038, auc: 0.977759.\nTue May 18 05:36:49 2021 Fold 0, Epoch 9, lr: 0.0008205, loss_train: 0.12412, loss_valid: 0.05984, auc: 0.977488.\nTue May 18 06:02:25 2021 Fold 0, Epoch 10, lr: 0.0007702, loss_train: 0.11578, loss_valid: 0.05463, auc: 0.980366.\nTue May 18 06:27:51 2021 Fold 0, Epoch 11, lr: 0.0007158, loss_train: 0.11327, loss_valid: 0.05852, auc: 0.982862.\nTue May 18 06:52:40 2021 Fold 0, Epoch 12, lr: 0.0006580, loss_train: 0.10414, loss_valid: 0.05541, auc: 0.980872.\nTue May 18 07:17:27 2021 Fold 0, Epoch 13, lr: 0.0005978, loss_train: 0.10721, loss_valid: 0.05341, auc: 0.981519.\nTue May 18 07:42:15 2021 Fold 0, Epoch 14, lr: 0.0005361, loss_train: 0.10051, loss_valid: 0.04901, auc: 0.984759.\nTue May 18 08:07:07 2021 Fold 0, Epoch 15, lr: 0.0004739, loss_train: 0.09740, loss_valid: 0.04880, auc: 0.982922.\nTue May 18 08:31:53 2021 Fold 0, Epoch 16, lr: 0.0004122, loss_train: 0.09316, loss_valid: 0.04741, auc: 0.984612.\nTue May 18 08:56:43 2021 Fold 0, Epoch 17, lr: 0.0003520, loss_train: 0.09402, loss_valid: 0.04863, auc: 0.984662.\nTue May 18 09:21:30 2021 Fold 0, Epoch 18, lr: 0.0002942, loss_train: 0.08681, loss_valid: 0.04707, auc: 0.983545.\nTue May 18 09:46:20 2021 Fold 0, Epoch 19, lr: 0.0002398, loss_train: 0.09038, loss_valid: 0.04782, auc: 0.982426.\nTue May 18 10:11:14 2021 Fold 0, Epoch 20, lr: 0.0001895, loss_train: 0.08618, loss_valid: 0.04228, auc: 0.985783.\nTue May 18 10:36:11 2021 Fold 0, Epoch 21, lr: 0.0001442, loss_train: 0.08070, loss_valid: 0.04507, auc: 0.984783.\nTue May 18 11:01:09 2021 Fold 0, Epoch 22, lr: 0.0001045, loss_train: 0.08410, loss_valid: 0.04596, auc: 0.985548.\nTue May 18 11:26:05 2021 Fold 0, Epoch 23, lr: 0.0000712, loss_train: 0.07906, loss_valid: 0.04222, auc: 0.986450.\nTue May 18 11:51:12 2021 Fold 0, Epoch 24, lr: 0.0000448, loss_train: 0.07931, loss_valid: 0.04317, auc: 0.986235.\nTue May 18 12:16:13 2021 Fold 0, Epoch 25, lr: 0.0000256, loss_train: 0.07874, loss_valid: 0.04268, auc: 0.986202.\nI found one trick.
\nAlmost twice as many epochs than me. :) @underwearfitting
@underwearfitting Thanks for the log :)
", "rawMarkdown": "@underwearfitting Thanks for the log :) ", "votes": 1 } ] }, { "id": 1331507, "postDate": "2021-06-01T14:07:28.010Z", "content": "CV 0.9887
\nLB 0.98
\n5 fold ensemble
\nb4
\n512*512
\nweak augmentation
happy to update my LB score😄
\nCV: 0.991, LB: 0.98(5fold-avg)
", "rawMarkdown": "happy to update my LB score😄\n\nCV: 0.991, LB: 0.98(5fold-avg)", "votes": 10, "replies": [ { "id": 1331090, "postDate": "2021-06-01T09:09:37.093Z", "content": "Great work!
\nDo you find something important trick?
I think Mixup works very well rather than some tircks.
", "rawMarkdown": "I think Mixup works very well rather than some tircks. ", "votes": 10 }, { "id": 1331260, "postDate": "2021-06-01T11:18:01.053Z", "content": "Nice!! Is it a small model, if I may ask?
", "rawMarkdown": "Nice!! Is it a small model, if I may ask? " }, { "id": 1331278, "postDate": "2021-06-01T11:31:17.443Z", "content": "Yes, it's a small model. I use resnet34d.
", "rawMarkdown": "Yes, it's a small model. I use resnet34d.", "votes": 6 }, { "id": 1331303, "postDate": "2021-06-01T11:39:26.870Z", "content": "Thanks! It is really interesting to see small models thrive. I would think a correlation between image size and the complexity of model can be attributing to this
", "rawMarkdown": "Thanks! It is really interesting to see small models thrive. I would think a correlation between image size and the complexity of model can be attributing to this ", "votes": 2 }, { "id": 1331328, "postDate": "2021-06-01T11:58:07.313Z", "content": "Perhaps larger models with large images will be successful. But it requires us more and more resources…
", "rawMarkdown": "Perhaps larger models with large images will be successful. But it requires us more and more resources...", "votes": 2 }, { "id": 1331347, "postDate": "2021-06-01T12:12:47.310Z", "content": "@ttahara yes…I have no great access to gpus other than kaggle and colab…so sometimes really have to squeeze dry smaller models ><||
", "rawMarkdown": "@ttahara yes...I have no great access to gpus other than kaggle and colab...so sometimes really have to squeeze dry smaller models ><||" }, { "id": 1331475, "postDate": "2021-06-01T13:48:32.260Z", "content": "Great Job.
\nI guess, I found one as well. Its not network and image size etc, Way of training matters I guess.
\nI wrote this to see if we are on same page or not. 😄
I tried the mixup as recommended by @ttahara, and the CV improved dramatically.
\nSpecifically, the cv has been improved from 0.9803 to 0.9874.
\nThe model is efficientnet b0 and the image size is 336.
\nMore specific results are shown below.
CV: 0.9803 publicLB: 0.96
\nCV: 0.9874 publicLB: 0.96 (higher than above, near 0.97)
\nCV: 0.9849 publicLB: 0.97 (not the best auc, but the best loss)
\nThe all results are from a single fold model out of 5 folds.
@yosukeyama +0.007? That's great !
\nIn my case, mixup improves OOF AUC by 0.003.
I also submit predictions by both best AUC models and best loss model. Best AUC models are always higher than best loss on LB so far.
", "rawMarkdown": "@yosukeyama +0.007? That's great ! \nIn my case, mixup improves OOF AUC by 0.003. \n\nI also submit predictions by both best AUC models and best loss model. Best AUC models are always higher than best loss on LB so far.", "votes": 5 }, { "id": 1332914, "postDate": "2021-06-02T11:33:48.400Z", "content": "In my case, the CV was worse when using the mixup. After seeing @ttahara post, I'm experimenting with mixup, but it's still there. 😂
", "rawMarkdown": "In my case, the CV was worse when using the mixup. After seeing @ttahara post, I'm experimenting with mixup, but it's still there. 😂" }, { "id": 1332973, "postDate": "2021-06-02T12:11:06.597Z", "content": "@piantic At first, as same as you, mixup didn't improve CV. Tuning alpha and training epochs work well for me.
", "rawMarkdown": "@piantic At first, as same as you, mixup didn't improve CV. Tuning alpha and training epochs work well for me.", "votes": 1 }, { "id": 1333050, "postDate": "2021-06-02T13:06:18.727Z", "content": "Here to report my findings. I trained one fold only on eca nfnet l0 and with mixup my cv is 0.989 with LB at 64 place. I’m not sure what is my real score for 0.97 but it’s quite close to 0.98 I guess?
\nI’m using 512 image size and the funny thing is if i transpose time axis w frequency axis my first or second epoch will have super high validation loss.
", "rawMarkdown": "Here to report my findings. I trained one fold only on eca nfnet l0 and with mixup my cv is 0.989 with LB at 64 place. I’m not sure what is my real score for 0.97 but it’s quite close to 0.98 I guess?\n\nI’m using 512 image size and the funny thing is if i transpose time axis w frequency axis my first or second epoch will have super high validation loss. ", "votes": 2 }, { "id": 1333106, "postDate": "2021-06-02T13:44:30.470Z", "content": "Exactly. I have also the same finding regarding the time and freq. axis. Related discussion: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/240277
", "rawMarkdown": "Exactly. I have also the same finding regarding the time and freq. axis. Related discussion: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/240277" }, { "id": 1333605, "postDate": "2021-06-02T22:24:21.760Z", "content": "Hi @reighns , may I ask how long does it take for you to train your eca l0? And how many epoch was that?
", "rawMarkdown": "Hi @reighns , may I ask how long does it take for you to train your eca l0? And how many epoch was that?" }, { "id": 1333785, "postDate": "2021-06-03T04:10:37.190Z", "content": "18 minute and 16 epoch
", "rawMarkdown": "18 minute and 16 epoch " }, { "id": 1333905, "postDate": "2021-06-03T06:22:56.727Z", "content": "Thanks! I'm evaluating the feasibility of this competition.
", "rawMarkdown": "Thanks! I'm evaluating the feasibility of this competition.", "votes": 1 }, { "id": 1333998, "postDate": "2021-06-03T07:47:55.360Z", "content": "It’s definitely feasible! Because anecdotal evidence by many has shown small models work wonders here!
", "rawMarkdown": "It’s definitely feasible! Because anecdotal evidence by many has shown small models work wonders here!", "votes": 1 }, { "id": 1348052, "postDate": "2021-06-13T17:16:37.897Z", "content": "Which prob for mixup works better? 0.3/0.5/1?
", "rawMarkdown": "Which prob for mixup works better? 0.3/0.5/1? " } ] }, { "id": 1313000, "postDate": "2021-05-18T11:11:54.400Z", "content": "CV 0.9884
\nLB 0.98
\n5 fold ensemble
Which model have you used?
", "rawMarkdown": "Which model have you used?" }, { "id": 1313131, "postDate": "2021-05-18T12:20:20.193Z", "content": "NFNet; not tried anything else yet. Sounds like Effnet-B0 works well, but I haven't tried.
", "rawMarkdown": "NFNet; not tried anything else yet. Sounds like Effnet-B0 works well, but I haven't tried.", "votes": 3 }, { "id": 1313921, "postDate": "2021-05-18T19:52:36.007Z", "content": "Effnet is fast, great for initial experiment. I usually use B0 to kick start. Take 2 minutes to train one epoch.
", "rawMarkdown": "Effnet is fast, great for initial experiment. I usually use B0 to kick start. Take 2 minutes to train one epoch.", "votes": 3 }, { "id": 1314791, "postDate": "2021-05-19T11:37:46.773Z", "content": "I don't think model architecture matters much, actually. I'm using a ResNeXt-50 (not even 101), training at twice the speed and getting a similar result to NFNet.
", "rawMarkdown": "I don't think model architecture matters much, actually. I'm using a ResNeXt-50 (not even 101), training at twice the speed and getting a similar result to NFNet.", "votes": 3 }, { "id": 1314804, "postDate": "2021-05-19T11:45:17.967Z", "content": "How much time ResneXt50 is taking for one epoch?
", "rawMarkdown": "How much time ResneXt50 is taking for one epoch?" }, { "id": 1314808, "postDate": "2021-05-19T11:48:41.023Z", "content": "About 20 mins with mixed precision over 2 GPUs, but it's completely dependant on your data formatting, GPUs, mixed precision, etc.
", "rawMarkdown": "About 20 mins with mixed precision over 2 GPUs, but it's completely dependant on your data formatting, GPUs, mixed precision, etc.", "votes": 2 } ] }, { "id": 1319750, "postDate": "2021-05-23T13:25:43.090Z", "content": "CV: 0.9868, LB: 0.97 (5fold-avg)
\nModel: resnet18d
\nInitial weights: ImageNet
\nTrain-Val Split: StratifiedKFold(K=5)
\nAugmentation: HorizontalFlip, VerticalFlip, ShiftScaleRotate, RandomResizedCrop
I'm testing CV data with or without random resizedcrop. Maybe my random resizedcrop super parameters are not adjusted well or are not helpful for training at all? After all, I don't know whether there is a certain proportion of invalid background area in this signal image, so I preliminarily estimate that random resized crop enhancement is not suitable.
\nHorizontalFlip, VerticalFlip, ShiftScaleRotate These should be necessary
Thanks!
\nI'll try experiments without RandomResizedCrop.
", "rawMarkdown": "Thanks!\n\nI'll try experiments without RandomResizedCrop." }, { "id": 1326029, "postDate": "2021-05-28T07:08:40.847Z", "content": "Assuming you have images as the shape of (freq, time), I don't think HorizontalFlip is necessary for images that have Aliens.
\nUsing HorizontalFlip, the order of signals is changed from (ABACAD) to (DACABA). This is good Augmentation for non-alien images, but how about alien images?
\nAugmented images that only A of (DACABA) have \"needle\" are alien images?
\n(I don't know such images that only BCD of (ABACAD) have \"needle\" are included in this competition dataset)
cv:0.9880063998426094 LB:0.97(second) epoch15 baseline+a aug B0
\ncv:0.986708799271502 LB:0.97(best) epoch10 baseline+a aug B0
\ncv:0.9878064170595553 LB:0.97 B0
\noverfit…
How do you know which model of those two is best on LB? You are in first place so can't have gone up a position? :)
", "rawMarkdown": "How do you know which model of those two is best on LB? You are in first place so can't have gone up a position? :)", "votes": 2 }, { "id": 1310475, "postDate": "2021-05-16T17:23:16.363Z", "content": "In\"My Submissions\", sort score by public score
", "rawMarkdown": "In\"My Submissions\", sort score by public score", "votes": 4 }, { "id": 1310489, "postDate": "2021-05-16T17:36:05.860Z", "content": "Wow I didn't know that worked… all this time!
", "rawMarkdown": "Wow I didn't know that worked... all this time!", "votes": 2 }, { "id": 1310618, "postDate": "2021-05-16T19:05:46.517Z", "content": "20% of the data is quite small. I think you are doing well.
", "rawMarkdown": "20% of the data is quite small. I think you are doing well.", "votes": 1 }, { "id": 1311800, "postDate": "2021-05-17T16:14:08.300Z", "content": "😅OMG. I didn't know that.
", "rawMarkdown": "😅OMG. I didn't know that.", "votes": 3 } ] }, { "id": 1323151, "postDate": "2021-05-26T03:40:12.057Z", "content": "Model: ensemble of models which are based on [efficientnet-b0, resnet18d].
\n5 fold
\nAug: HFlip, VFlip, ShiftScale, Mixup
\nCV: 0.989x LB: 0.97
I'm facing the gap between CV and LB. The player who reaches LB 0.98x already achieve CV > 0.99x?
", "rawMarkdown": "Model: ensemble of models which are based on [efficientnet-b0, resnet18d].\n5 fold\nAug: HFlip, VFlip, ShiftScale, Mixup\nCV: 0.989x LB: 0.97\n\nI'm facing the gap between CV and LB. The player who reaches LB 0.98x already achieve CV > 0.99x? ", "votes": 5, "replies": [ { "id": 1323164, "postDate": "2021-05-26T03:58:15.873Z", "content": "me too.
\nfor me, CV:0.987 LB0.98 is single best.
\nI also have CV~0.99 model, but its LB wasn't good…
Now I achieved CV 0.9906x ensembles. Its LB score is still 0.97, but highest of mine.
", "rawMarkdown": "Now I achieved CV 0.9906x ensembles. Its LB score is still 0.97, but highest of mine.", "votes": 3 }, { "id": 1325355, "postDate": "2021-05-27T17:22:14.797Z", "content": "Ensemble of my models got CV 0.9903 but lower LB score than my single best model …
", "rawMarkdown": "Ensemble of my models got CV 0.9903 but lower LB score than my single best model ...", "votes": 4 } ] }, { "id": 1316452, "postDate": "2021-05-20T15:06:59.447Z", "content": "CV vs LB is very strange for me. My best model reaches an 4-fold out-of-fold AUC of 0.982 but submission to LB gives only 0.96…
\nWhat/Where should I investigate? What could be the reason for the mismatch?
(The indivisual folds are stratified and all have AUC>0.98.
\nThe BCE loss of train and val is 0.05. No overfitting here.
\nThe AUC indicates an overfit with train=0.997 and val=0.983
\nI have similar results with DenseNet201/EffcientNet/VGG using pretrained \"imagenet\"-weights.
\nTraining from scratch with a VGG like network works also very good.
\nI use TPU/32batchsizex8 / 256x256 spatial concat / all channels / On and off times color-coded.)
Try Stratified Sampling.
", "rawMarkdown": "Try Stratified Sampling.", "votes": 2 }, { "id": 1316930, "postDate": "2021-05-21T02:56:50.817Z", "content": "My first try.
\nCV 0.974, LB 0.93.
\n…
Issue might be in decoding method of your TPU training pipeline.
\nWhen you encode image in png or jpg and decode it in tensorflow, values change actually.
\nThat pipeline is built for fast processing and decode approximate values.
So try not to save image as an encoded image but use FloatList to encode and decode.
\nThat will help. I guess.
\nBest of luck.
", "rawMarkdown": "Issue might be in decoding method of your TPU training pipeline.\nWhen you encode image in png or jpg and decode it in tensorflow, values change actually. \nThat pipeline is built for fast processing and decode approximate values.\n\nSo try not to save image as an encoded image but use FloatList to encode and decode.\n\nThat will help. I guess.\n\nBest of luck. ", "votes": 1 }, { "id": 1319186, "postDate": "2021-05-23T02:26:14.927Z", "content": "Cutmix & mixup for preventing over fitting and in this project, you will get the result that CV & LB scores are highly correlated. Now I'm thinking about what does cutmix & mixup bring to our project besides preventing over fitting? My CV is closely related to LB, cv97.7 LB 97+
", "rawMarkdown": "Cutmix & mixup for preventing over fitting and in this project, you will get the result that CV & LB scores are highly correlated. Now I'm thinking about what does cutmix & mixup bring to our project besides preventing over fitting? My CV is closely related to LB, cv97.7 LB 97+", "votes": 6 } ] }, { "id": 1305497, "postDate": "2021-05-13T10:12:04.133Z", "content": "So now my single model gives 0.97
\nBut it took 7 hours for 1 Fold.
\nLet's see what we get from 5 Fold pipeline.
\nIt's running….!!!!
CV:0.987460473479882 LB0.98 large model
", "rawMarkdown": "CV:0.987460473479882 LB0.98 large model", "votes": 6, "replies": [ { "id": 1310809, "postDate": "2021-05-17T01:55:44.097Z", "content": "This is a 4 fold CV with a single model?
", "rawMarkdown": "This is a 4 fold CV with a single model?" }, { "id": 1310818, "postDate": "2021-05-17T02:04:33.140Z", "content": "yes,LB is mean of 4 fold
", "rawMarkdown": "yes,LB is mean of 4 fold", "votes": 2 }, { "id": 1310825, "postDate": "2021-05-17T02:10:21.170Z", "content": "Great. Please keep this updated. I want to compare my score as well. My current setup's CV(5fold) is around 0.985 with B0 and 256 image size. Still need to discover more aliens to match you :D.
", "rawMarkdown": "Great. Please keep this updated. I want to compare my score as well. My current setup's CV(5fold) is around 0.985 with B0 and 256 image size. Still need to discover more aliens to match you :D.", "votes": 2 } ] }, { "id": 1353200, "postDate": "2021-06-17T01:32:03.493Z", "content": "lb 1.000
\ni end this Competition.
Congratulation, you did it ;)
", "rawMarkdown": "Congratulation, you did it ;) " }, { "id": 1353203, "postDate": "2021-06-17T01:34:05.320Z", "content": "you are awesome lol.
", "rawMarkdown": "you are awesome lol." } ] }, { "id": 1350256, "postDate": "2021-06-15T11:36:58.843Z", "content": "efnetb0 4fold 768*768
\ncv:0.990752314854183
\nlb:0.984
Are you using all the 6 channels for your experiment?
", "rawMarkdown": "Are you using all the 6 channels for your experiment?" }, { "id": 1351737, "postDate": "2021-06-16T14:57:38.363Z", "content": "yes
\nonly 3ch also work and it's fast
CV 0.9906
\nLB 0.97
\nresnet18d
\n512*512
CV 0.9908
\nLB 0.98
\n5models ensemble
CV 0.9315 LB 0.97
", "rawMarkdown": "CV 0.9315 LB 0.97", "votes": 3, "replies": [ { "id": 1328668, "postDate": "2021-05-30T12:31:57.587Z", "content": "What? impressive. Congrats.
\nMay I ask what did you do?
So I have just checked up my code again , I was trying mixup augmentaion and I accidentally did it on validation dataset. So it is not that impressive lol
", "rawMarkdown": "So I have just checked up my code again , I was trying mixup augmentaion and I accidentally did it on validation dataset. So it is not that impressive lol", "votes": 2 } ] }, { "id": 1315191, "postDate": "2021-05-19T16:10:41.827Z", "content": "CV 0.9826
\nLB 0.97(i guess is around 0.975)
\n4 fold ensemble
\nnfnet_f0
\nspatial
\nAug:Resize256，horizontalflip&verticalflip
CV: 0.9920
\nLB: 0.98
\nEfficientnet-B2, 5fold, 512x512
I have some questions.
\nDid you use Mixup or any other augs?
\nYou used 3 channels or 6 channels?
\nHow many epochs?
\nDid you use BCE or Focal Loss?
Mixup: yes
\n3channels
\n30epoch
\nBCE loss
Yes but my question is did you used any other augmentation as well like (Cutmix or Augmix or something like that)?
", "rawMarkdown": "Yes but my question is did you used any other augmentation as well like (Cutmix or Augmix or something like that)?" }, { "id": 1351806, "postDate": "2021-06-16T16:06:45.750Z", "content": "I do have another question.
\nDid you use any kind of sampler for dataloader as well? Like balanced sampling or something?
CV 0.988119 +/- 0.002054, 4 folds ensemble on LB 0.98
", "rawMarkdown": "CV 0.988119 +/- 0.002054, 4 folds ensemble on LB 0.98", "votes": 3, "replies": [ { "id": 1315110, "postDate": "2021-05-19T15:10:52.687Z", "content": "You also used pure vision models (e.g NFNet or Effnet)?
", "rawMarkdown": "You also used pure vision models (e.g NFNet or Effnet)?" } ] }, { "id": 1347806, "postDate": "2021-06-13T13:54:01.207Z", "content": "Val AUC: 0.9923
\nLB: 0.98
\nefficientnet-b1, single fold, Colab Pro
Can I get some hint?
\nwhich loss function?
\nuse label smoothing?
\nvalidation loss?
\nimage size?
\nefficientnet-b1 better than efficientnet-b0 ?
loss: BCE
\nlabel smoothing: no
\nimage size: 512x512
\nefficientnet-b1 is slightly better than b0 in my case(~0.0001)
Thanks for sharing! How many epochs do you train ?
", "rawMarkdown": "Thanks for sharing! How many epochs do you train ?", "votes": -1 } ] }, { "id": 1343532, "postDate": "2021-06-10T08:58:43.717Z", "content": "My best model is following:
\nb0
\n512 * 512
\n5fold
\nmixup and another augmentation
\nCV 0.991
\nLB 0.98
You used all channels or just 0,2,4 ?
", "rawMarkdown": "You used all channels or just 0,2,4 ?" }, { "id": 1343826, "postDate": "2021-06-10T13:17:56.630Z", "content": "3ch instead of 6ch
", "rawMarkdown": "3ch instead of 6ch", "votes": 3 }, { "id": 1343827, "postDate": "2021-06-10T13:19:11.497Z", "content": "thank you for sharing information
", "rawMarkdown": "thank you for sharing information" }, { "id": 1345200, "postDate": "2021-06-11T11:51:57.800Z", "content": "Thanks for sharing! How long epochs it takes to train one fold ?
", "rawMarkdown": "Thanks for sharing! How long epochs it takes to train one fold ?" }, { "id": 1345251, "postDate": "2021-06-11T12:25:31.217Z", "content": "15epoch
\nOver 15 did not work well in my setting.
Wow.
\nFor me it took 70 epochs each fold to achieve 0.98 LB and 0.99 CV.
\nI am missing something then.
\nThanks.
Still using B0 with Image Size 512x512?
", "rawMarkdown": "Still using B0 with Image Size 512x512?" }, { "id": 1345399, "postDate": "2021-06-11T14:49:16.940Z", "content": "OMG… 70 epoch !? I can't do that lol
\nI tried b4 but b0 is better.
", "rawMarkdown": "OMG... 70 epoch !? I can't do that lol\n\nI tried b4 but b0 is better.", "votes": 1 }, { "id": 1345615, "postDate": "2021-06-11T17:51:59.590Z", "content": "Yeah. I guess max_lr was set to 5e-03. I should increase it a bit I guess.
", "rawMarkdown": "Yeah. I guess max_lr was set to 5e-03. I should increase it a bit I guess." }, { "id": 1346026, "postDate": "2021-06-12T04:00:37.567Z", "content": "I guess it might be some kind of normalization that is required for 0.99 LB.
", "rawMarkdown": "I guess it might be some kind of normalization that is required for 0.99 LB." }, { "id": 1348455, "postDate": "2021-06-14T04:32:55.927Z", "content": "I can't tell you in detail but my LR is not so large.
", "rawMarkdown": "I can't tell you in detail but my LR is not so large." } ] }, { "id": 1306134, "postDate": "2021-05-13T16:16:08.943Z", "content": "There is a gap between CV and LB indeed, but yours seems to correlate well.
", "rawMarkdown": "There is a gap between CV and LB indeed, but yours seems to correlate well.", "votes": 3 }, { "id": 1305547, "postDate": "2021-05-13T11:00:32.577Z", "content": "CV: 0.9736 LB: 0.96
\nCV: 0.9774 LB: 0.96
\nCV: 0.9824 LB: 0.96
\nAll have the same score on public LB, but when sorted, the scores seems to improve outside the indicated digits.
CV: 0.9902
\nLB: 0.98
\nsingle fold model from 5 folds
\nefficientnetb0
\n512*512
\nAfter seeing @ttahara discussion and notebook, I get this effective single fold model by some changes.
Thank you for seeing my posts.
\nBreaking 0.98 by single fold is great🎉
Model: EfficientNetB0
\nImage size: original
\nFolds : 5
\nAugmentations: FMix
\nCV: 0.988
\nLB: 0.97
hflip vflip blur CV:0.98 LB:0.93
", "rawMarkdown": "hflip vflip blur CV:0.98 LB:0.93", "votes": 4, "replies": [ { "id": 1316749, "postDate": "2021-05-20T20:38:22.627Z", "content": "0.98 / 0.93 is the biggest CV gap I've seen so far - what did you do?!
", "rawMarkdown": "0.98 / 0.93 is the biggest CV gap I've seen so far - what did you do?!", "votes": 7 }, { "id": 1340401, "postDate": "2021-06-07T21:41:09.287Z", "content": "screwed up
", "rawMarkdown": "screwed up" } ] }, { "id": 1310798, "postDate": "2021-05-17T01:14:09.047Z", "content": "all train
\nmodel:nfnet_l0
\nno resize
\nno augumentaion
\nepoch 3
\nlb:97
Split: StratifiedKFold 4 folds?
", "rawMarkdown": "Split: StratifiedKFold 4 folds?" }, { "id": 1312290, "postDate": "2021-05-18T01:11:09.583Z", "content": "All images are used for training, no verification set
", "rawMarkdown": "All images are used for training, no verification set", "votes": 1 } ] }, { "id": 1306694, "postDate": "2021-05-14T02:44:11.817Z", "content": "epoch10 add a aug resize hflip vflip CV:0.98614442 LB:0.97
\n(lower than no resize ,higher than \"another aug \")
hflip and flip on spatial images? or on individual channel?
", "rawMarkdown": "hflip and flip on spatial images? or on individual channel?", "votes": 2 }, { "id": 1310752, "postDate": "2021-05-16T23:13:36.950Z", "content": "do augumentation after nakama's preprocess(image = np.vstack(image).transpose((1, 0)).astype(np.float32))
\nI think on each channel is worth trying.
", "rawMarkdown": "do augumentation after nakama's preprocess(image = np.vstack(image).transpose((1, 0)).astype(np.float32))\n\nI think on each channel is worth trying.", "votes": 2 }, { "id": 1325934, "postDate": "2021-05-28T05:42:39.553Z", "content": "Have you tried it on each channel?
", "rawMarkdown": "Have you tried it on each channel?" } ] }, { "id": 1353088, "postDate": "2021-06-16T21:18:26.007Z", "content": "Eff B1
\n5 Fold 512x512
\nOOF 0.9846
\nLB 0.979
\nA little relieved to see that my CV-LB gap is smaller than I thought
CV: 0.943, LB: 0.93
\nModel: EfficientNet-B0
\nInitial weights: imagenet
\n5 fold cross validation, earlystopping, reduce LR on plateau, Adam, Mixup augmentation
Training/validation metrics: https://wandb.ai/ayush-thakur/kaggle-seti
\nWondering why I am stuck with 0.93 even though I am using kinda same strategy.
", "rawMarkdown": "CV: 0.943, LB: 0.93\n\nModel: EfficientNet-B0\nInitial weights: imagenet\n5 fold cross validation, earlystopping, reduce LR on plateau, Adam, Mixup augmentation\n\nTraining/validation metrics: https://wandb.ai/ayush-thakur/kaggle-seti\n\nWondering why I am stuck with 0.93 even though I am using kinda same strategy. ", "votes": 1, "replies": [ { "id": 1319480, "postDate": "2021-05-23T08:26:45.490Z", "content": "You should stack 273 dimension instead of 256 dimension.
\nUse this one: np.vstack(image).transpose((1, 0))
Thanks for the tip. A follow-up question, stack all the 6 spectrograms or just spectrograms with target signal (0, 2 and 4)?
", "rawMarkdown": "Thanks for the tip. A follow-up question, stack all the 6 spectrograms or just spectrograms with target signal (0, 2 and 4)?" }, { "id": 1319486, "postDate": "2021-05-23T08:36:41.300Z", "content": "I am stacking all channels.
", "rawMarkdown": "I am stacking all channels." }, { "id": 1319634, "postDate": "2021-05-23T12:05:14.030Z", "content": "Why should we transpose((0,1)) ??
", "rawMarkdown": "Why should we transpose((0,1)) ??" }, { "id": 1320230, "postDate": "2021-05-23T22:05:51.213Z", "content": "Not necessary to use transpose.
", "rawMarkdown": "Not necessary to use transpose. ", "votes": 1 }, { "id": 1331264, "postDate": "2021-06-01T11:21:02.190Z", "content": "Yes I was also curious on the rationale, because it should not matter in theory.
", "rawMarkdown": "Yes I was also curious on the rationale, because it should not matter in theory. " } ] }, { "id": 1311425, "postDate": "2021-05-17T11:50:19.983Z", "content": "LB0.97+LB0.98=LB0.98(lower than left one)
\nSo, I think we we will have to keep sticking to single model so far …..
Thank you for sharing a lot of results of your experiments!
\nWas your cv improved by blending?
sorry ,I made a mistake to create \"oof.csv \", so i didn't check cv improvement.
", "rawMarkdown": "sorry ,I made a mistake to create \"oof.csv \", so i didn't check cv improvement.", "votes": 1 }, { "id": 1319179, "postDate": "2021-05-23T01:55:55.620Z", "content": "Thank you for your reply.
\nWe can see a strong correlation between CV and LB scores, so we may trust CV.
\nIn this competition, as in other competitions, we believe that ensemble is an important factor, so how we calculate CV is going to be important.
I wish you further success!
", "rawMarkdown": "Thank you for your reply.\nWe can see a strong correlation between CV and LB scores, so we may trust CV.\nIn this competition, as in other competitions, we believe that ensemble is an important factor, so how we calculate CV is going to be important.\n\nI wish you further success!", "votes": 1 }, { "id": 1343205, "postDate": "2021-06-10T05:36:33.853Z", "content": "for my best two, CV0.98978+CV0.99018=CV:0.99099
\nalso LB boost a little.
\nso, ensamble looks good.
\nHas your team already started an ensemble?
CV: 0.988
\nLB: 0.97
\n512 * 512 eff_b1
\nAny Suggestions?
Model: Resnest50
\n5 fold
\nAug: Horizontal, Vertical Flip, Rotation
\nOOF CV: 0.9815 LB: 97
I think the gap between CV and LB is not only a frustating annoyance, but the key to understand the real challenge of this competition
\nI'am astrophysicis myself, but not related at the SETI project in anyway, and this sentence has been in my mind from the beginning
\n\"While it would be nice to train our algorithms entirely on observations of interplanetary spacecraft, there are not many examples of them, and we also want to be able to find a wider range of signal types. So we’ve turned to simulating technosignature candidates.\"
\nSo, they are specially interested in unknown signals…. literally, they don't know what are looking for !!!
\nThis is only my personal opinion, but if i were one of them i would expose the next problem
\nOBJECTIVE: Finding anomalies that we don't know how they are
\nBuild a Dataset with simulated signals based on the research and information gathered over the last 40 years
Ask the smart guys of the SETI project to build a Neural Network to dectected them…. They already are really good, as you can see in this post of @manabendrarout
https://www.kaggle.com/c/seti-breakthrough-listen/discussion/239245
\nTest the Dataset with this models to achieve a CV/LB greater of 0.99… without gap logically because is the same Dataset for both
Split the original Dataset in Train/Test folds randmnly
**Add a litte batch of positive samples at the Test Dataset **that don't match any kind of pattern of the positive original targets… possibly REAL samples (only guessing).
Check that there is a gap between the CV (testing the train samples) and the LB (testing the test samples) whit the NN of the smart SETI guys
To understand the gap we have to analyze the confussion MAtrix. Por example, with my best model I achiveve CV = 0.988 (and LB = 0.96, but that is not important here).
\nThe confussion matrix say me that:
\n
Relation between False Positives / True Positives (used to compute the ROC Curve) taking true positive when the probability is greater than 0.5 is 21 /664 = 3.16%… A really good result !!
\nBut, the relation between False NEgative / True Positives is 62 / 664 = 9.3 %… not so good
\nSo, even with a magnific 0.988 I fail to detect about the 10% of samples !!!
\nIf, how I am guessing in point 5, they add some new samples at the test dataset, then this relation FN/TP will growth fast in this dataset, but the LB only will decrease a bit…
\nCONCLUSION: SETI guys have found the way to hidde the REAL samples !!… and detect them is the real challenge of this competition
\nI think they are not intereseted in a high CV or LB score (if it's high enough, ie. >0.97), but the submissions with the minimum gap, because they mean they are detecting the interesting REAL samples
\nLoking the submissions and the comments, and Making some fast computations I guess there are about 5 - 10 % of \"REAL samples\", that traslates a 1 % of CV/LB gap
\nMore of two hundred people have LB = 0.97, but only one have LB = 0.99… becasuse this 1% gap.
\nSorry for the long post, and remember I am only guessing here for fun
", "votes": 17, "replies": [] }, { "id": 1312002, "author_name": "sin", "author_url": "", "post_date": "2021-05-17T18:49:05.373000", "content": "Submitted a single fold (CV ~0.987), LB is 0.97.
\nImage: 256 x 256.
\nModel: b0.
\nSpatial or Channel: spatial.
\nAugmentations: Mixup (added).
\nTricks: maybe
Tricks: maybe😂
", "votes": 3, "replies": [] }, { "id": 1313918, "author_name": "sin", "author_url": "", "post_date": "2021-05-18T19:50:55.187000", "content": "I share my training logs here:
\nTue May 18 02:11:48 2021 Fold 0, Epoch 1, lr: 0.0001000, loss_train: 0.25995, loss_valid: 0.12689, auc: 0.945531.\nTue May 18 02:37:37 2021 Fold 0, Epoch 2, lr: 0.0010000, loss_train: 0.20199, loss_valid: 0.10548, auc: 0.964634.\nTue May 18 03:03:06 2021 Fold 0, Epoch 3, lr: 0.0010000, loss_train: 0.17507, loss_valid: 0.09838, auc: 0.966518.\nTue May 18 03:28:49 2021 Fold 0, Epoch 4, lr: 0.0009844, loss_train: 0.15932, loss_valid: 0.08149, auc: 0.966431.\nTue May 18 03:54:29 2021 Fold 0, Epoch 5, lr: 0.0009652, loss_train: 0.14819, loss_valid: 0.06801, auc: 0.978557.\nTue May 18 04:19:59 2021 Fold 0, Epoch 6, lr: 0.0009388, loss_train: 0.14161, loss_valid: 0.07503, auc: 0.974530.\nTue May 18 04:45:34 2021 Fold 0, Epoch 7, lr: 0.0009055, loss_train: 0.13015, loss_valid: 0.06789, auc: 0.973706.\nTue May 18 05:11:09 2021 Fold 0, Epoch 8, lr: 0.0008658, loss_train: 0.12650, loss_valid: 0.07038, auc: 0.977759.\nTue May 18 05:36:49 2021 Fold 0, Epoch 9, lr: 0.0008205, loss_train: 0.12412, loss_valid: 0.05984, auc: 0.977488.\nTue May 18 06:02:25 2021 Fold 0, Epoch 10, lr: 0.0007702, loss_train: 0.11578, loss_valid: 0.05463, auc: 0.980366.\nTue May 18 06:27:51 2021 Fold 0, Epoch 11, lr: 0.0007158, loss_train: 0.11327, loss_valid: 0.05852, auc: 0.982862.\nTue May 18 06:52:40 2021 Fold 0, Epoch 12, lr: 0.0006580, loss_train: 0.10414, loss_valid: 0.05541, auc: 0.980872.\nTue May 18 07:17:27 2021 Fold 0, Epoch 13, lr: 0.0005978, loss_train: 0.10721, loss_valid: 0.05341, auc: 0.981519.\nTue May 18 07:42:15 2021 Fold 0, Epoch 14, lr: 0.0005361, loss_train: 0.10051, loss_valid: 0.04901, auc: 0.984759.\nTue May 18 08:07:07 2021 Fold 0, Epoch 15, lr: 0.0004739, loss_train: 0.09740, loss_valid: 0.04880, auc: 0.982922.\nTue May 18 08:31:53 2021 Fold 0, Epoch 16, lr: 0.0004122, loss_train: 0.09316, loss_valid: 0.04741, auc: 0.984612.\nTue May 18 08:56:43 2021 Fold 0, Epoch 17, lr: 0.0003520, loss_train: 0.09402, loss_valid: 0.04863, auc: 0.984662.\nTue May 18 09:21:30 2021 Fold 0, Epoch 18, lr: 0.0002942, loss_train: 0.08681, loss_valid: 0.04707, auc: 0.983545.\nTue May 18 09:46:20 2021 Fold 0, Epoch 19, lr: 0.0002398, loss_train: 0.09038, loss_valid: 0.04782, auc: 0.982426.\nTue May 18 10:11:14 2021 Fold 0, Epoch 20, lr: 0.0001895, loss_train: 0.08618, loss_valid: 0.04228, auc: 0.985783.\nTue May 18 10:36:11 2021 Fold 0, Epoch 21, lr: 0.0001442, loss_train: 0.08070, loss_valid: 0.04507, auc: 0.984783.\nTue May 18 11:01:09 2021 Fold 0, Epoch 22, lr: 0.0001045, loss_train: 0.08410, loss_valid: 0.04596, auc: 0.985548.\nTue May 18 11:26:05 2021 Fold 0, Epoch 23, lr: 0.0000712, loss_train: 0.07906, loss_valid: 0.04222, auc: 0.986450.\nTue May 18 11:51:12 2021 Fold 0, Epoch 24, lr: 0.0000448, loss_train: 0.07931, loss_valid: 0.04317, auc: 0.986235.\nTue May 18 12:16:13 2021 Fold 0, Epoch 25, lr: 0.0000256, loss_train: 0.07874, loss_valid: 0.04268, auc: 0.986202.\nI found one trick.
\nAlmost twice as many epochs than me. :) @underwearfitting
@underwearfitting Thanks for the log :)
", "votes": 1, "replies": [] } ] }, { "id": 1331507, "author_name": "MOONMOON", "author_url": "", "post_date": "2021-06-01T14:07:28.010000", "content": "CV 0.9887
\nLB 0.98
\n5 fold ensemble
\nb4
\n512*512
\nweak augmentation
happy to update my LB score😄
\nCV: 0.991, LB: 0.98(5fold-avg)
", "votes": 10, "replies": [ { "id": 1331090, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-01T09:09:37.093000", "content": "Great work!
\nDo you find something important trick?
I think Mixup works very well rather than some tircks.
", "votes": 10, "replies": [] }, { "id": 1331260, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-01T11:18:01.053000", "content": "Nice!! Is it a small model, if I may ask?
", "votes": 0, "replies": [] }, { "id": 1331278, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-01T11:31:17.443000", "content": "Yes, it's a small model. I use resnet34d.
", "votes": 6, "replies": [] }, { "id": 1331303, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-01T11:39:26.870000", "content": "Thanks! It is really interesting to see small models thrive. I would think a correlation between image size and the complexity of model can be attributing to this
", "votes": 2, "replies": [] }, { "id": 1331328, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-01T11:58:07.313000", "content": "Perhaps larger models with large images will be successful. But it requires us more and more resources…
", "votes": 2, "replies": [] }, { "id": 1331347, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-01T12:12:47.310000", "content": "@ttahara yes…I have no great access to gpus other than kaggle and colab…so sometimes really have to squeeze dry smaller models ><||
", "votes": 0, "replies": [] }, { "id": 1331475, "author_name": "Salman", "author_url": "", "post_date": "2021-06-01T13:48:32.260000", "content": "Great Job.
\nI guess, I found one as well. Its not network and image size etc, Way of training matters I guess.
\nI wrote this to see if we are on same page or not. 😄
I tried the mixup as recommended by @ttahara, and the CV improved dramatically.
\nSpecifically, the cv has been improved from 0.9803 to 0.9874.
\nThe model is efficientnet b0 and the image size is 336.
\nMore specific results are shown below.
CV: 0.9803 publicLB: 0.96
\nCV: 0.9874 publicLB: 0.96 (higher than above, near 0.97)
\nCV: 0.9849 publicLB: 0.97 (not the best auc, but the best loss)
\nThe all results are from a single fold model out of 5 folds.
@yosukeyama +0.007? That's great !
\nIn my case, mixup improves OOF AUC by 0.003.
I also submit predictions by both best AUC models and best loss model. Best AUC models are always higher than best loss on LB so far.
", "votes": 5, "replies": [] }, { "id": 1332914, "author_name": "Heroseo", "author_url": "", "post_date": "2021-06-02T11:33:48.400000", "content": "In my case, the CV was worse when using the mixup. After seeing @ttahara post, I'm experimenting with mixup, but it's still there. 😂
", "votes": 0, "replies": [] }, { "id": 1332973, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-02T12:11:06.597000", "content": "@piantic At first, as same as you, mixup didn't improve CV. Tuning alpha and training epochs work well for me.
", "votes": 1, "replies": [] }, { "id": 1333050, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-02T13:06:18.727000", "content": "Here to report my findings. I trained one fold only on eca nfnet l0 and with mixup my cv is 0.989 with LB at 64 place. I’m not sure what is my real score for 0.97 but it’s quite close to 0.98 I guess?
\nI’m using 512 image size and the funny thing is if i transpose time axis w frequency axis my first or second epoch will have super high validation loss.
", "votes": 2, "replies": [] }, { "id": 1333106, "author_name": "Baran Hashemi", "author_url": "", "post_date": "2021-06-02T13:44:30.470000", "content": "Exactly. I have also the same finding regarding the time and freq. axis. Related discussion: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/240277
", "votes": 0, "replies": [] }, { "id": 1333605, "author_name": "JunYong Tong", "author_url": "", "post_date": "2021-06-02T22:24:21.760000", "content": "Hi @reighns , may I ask how long does it take for you to train your eca l0? And how many epoch was that?
", "votes": 0, "replies": [] }, { "id": 1333785, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-03T04:10:37.190000", "content": "18 minute and 16 epoch
", "votes": 0, "replies": [] }, { "id": 1333905, "author_name": "JunYong Tong", "author_url": "", "post_date": "2021-06-03T06:22:56.727000", "content": "Thanks! I'm evaluating the feasibility of this competition.
", "votes": 1, "replies": [] }, { "id": 1333998, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-03T07:47:55.360000", "content": "It’s definitely feasible! Because anecdotal evidence by many has shown small models work wonders here!
", "votes": 1, "replies": [] }, { "id": 1348052, "author_name": "EnricRovira", "author_url": "", "post_date": "2021-06-13T17:16:37.897000", "content": "Which prob for mixup works better? 0.3/0.5/1?
", "votes": 0, "replies": [] } ] }, { "id": 1313000, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-18T11:11:54.400000", "content": "CV 0.9884
\nLB 0.98
\n5 fold ensemble
Which model have you used?
", "votes": 0, "replies": [] }, { "id": 1313131, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-18T12:20:20.193000", "content": "NFNet; not tried anything else yet. Sounds like Effnet-B0 works well, but I haven't tried.
", "votes": 3, "replies": [] }, { "id": 1313921, "author_name": "sin", "author_url": "", "post_date": "2021-05-18T19:52:36.007000", "content": "Effnet is fast, great for initial experiment. I usually use B0 to kick start. Take 2 minutes to train one epoch.
", "votes": 3, "replies": [] }, { "id": 1314791, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-19T11:37:46.773000", "content": "I don't think model architecture matters much, actually. I'm using a ResNeXt-50 (not even 101), training at twice the speed and getting a similar result to NFNet.
", "votes": 3, "replies": [] }, { "id": 1314804, "author_name": "Salman", "author_url": "", "post_date": "2021-05-19T11:45:17.967000", "content": "How much time ResneXt50 is taking for one epoch?
", "votes": 0, "replies": [] }, { "id": 1314808, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-19T11:48:41.023000", "content": "About 20 mins with mixed precision over 2 GPUs, but it's completely dependant on your data formatting, GPUs, mixed precision, etc.
", "votes": 2, "replies": [] } ] }, { "id": 1319750, "author_name": "Tawara", "author_url": "", "post_date": "2021-05-23T13:25:43.090000", "content": "CV: 0.9868, LB: 0.97 (5fold-avg)
\nModel: resnet18d
\nInitial weights: ImageNet
\nTrain-Val Split: StratifiedKFold(K=5)
\nAugmentation: HorizontalFlip, VerticalFlip, ShiftScaleRotate, RandomResizedCrop
I'm testing CV data with or without random resizedcrop. Maybe my random resizedcrop super parameters are not adjusted well or are not helpful for training at all? After all, I don't know whether there is a certain proportion of invalid background area in this signal image, so I preliminarily estimate that random resized crop enhancement is not suitable.
\nHorizontalFlip, VerticalFlip, ShiftScaleRotate These should be necessary
Thanks!
\nI'll try experiments without RandomResizedCrop.
", "votes": 0, "replies": [] }, { "id": 1326029, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-05-28T07:08:40.847000", "content": "Assuming you have images as the shape of (freq, time), I don't think HorizontalFlip is necessary for images that have Aliens.
\nUsing HorizontalFlip, the order of signals is changed from (ABACAD) to (DACABA). This is good Augmentation for non-alien images, but how about alien images?
\nAugmented images that only A of (DACABA) have \"needle\" are alien images?
\n(I don't know such images that only BCD of (ABACAD) have \"needle\" are included in this competition dataset)
cv:0.9880063998426094 LB:0.97(second) epoch15 baseline+a aug B0
\ncv:0.986708799271502 LB:0.97(best) epoch10 baseline+a aug B0
\ncv:0.9878064170595553 LB:0.97 B0
\noverfit…
How do you know which model of those two is best on LB? You are in first place so can't have gone up a position? :)
", "votes": 2, "replies": [] }, { "id": 1310475, "author_name": "patriot", "author_url": "", "post_date": "2021-05-16T17:23:16.363000", "content": "In\"My Submissions\", sort score by public score
", "votes": 4, "replies": [] }, { "id": 1310489, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-16T17:36:05.860000", "content": "Wow I didn't know that worked… all this time!
", "votes": 2, "replies": [] }, { "id": 1310618, "author_name": "sin", "author_url": "", "post_date": "2021-05-16T19:05:46.517000", "content": "20% of the data is quite small. I think you are doing well.
", "votes": 1, "replies": [] }, { "id": 1311800, "author_name": "FelipeKitamura, MD, PhD", "author_url": "", "post_date": "2021-05-17T16:14:08.300000", "content": "😅OMG. I didn't know that.
", "votes": 3, "replies": [] } ] }, { "id": 1323151, "author_name": "yabea", "author_url": "", "post_date": "2021-05-26T03:40:12.057000", "content": "Model: ensemble of models which are based on [efficientnet-b0, resnet18d].
\n5 fold
\nAug: HFlip, VFlip, ShiftScale, Mixup
\nCV: 0.989x LB: 0.97
I'm facing the gap between CV and LB. The player who reaches LB 0.98x already achieve CV > 0.99x?
", "votes": 5, "replies": [ { "id": 1323164, "author_name": "patriot", "author_url": "", "post_date": "2021-05-26T03:58:15.873000", "content": "me too.
\nfor me, CV:0.987 LB0.98 is single best.
\nI also have CV~0.99 model, but its LB wasn't good…
Now I achieved CV 0.9906x ensembles. Its LB score is still 0.97, but highest of mine.
", "votes": 3, "replies": [] }, { "id": 1325355, "author_name": "Tawara", "author_url": "", "post_date": "2021-05-27T17:22:14.797000", "content": "Ensemble of my models got CV 0.9903 but lower LB score than my single best model …
", "votes": 4, "replies": [] } ] }, { "id": 1316452, "author_name": "AgentAuers", "author_url": "", "post_date": "2021-05-20T15:06:59.447000", "content": "CV vs LB is very strange for me. My best model reaches an 4-fold out-of-fold AUC of 0.982 but submission to LB gives only 0.96…
\nWhat/Where should I investigate? What could be the reason for the mismatch?
(The indivisual folds are stratified and all have AUC>0.98.
\nThe BCE loss of train and val is 0.05. No overfitting here.
\nThe AUC indicates an overfit with train=0.997 and val=0.983
\nI have similar results with DenseNet201/EffcientNet/VGG using pretrained \"imagenet\"-weights.
\nTraining from scratch with a VGG like network works also very good.
\nI use TPU/32batchsizex8 / 256x256 spatial concat / all channels / On and off times color-coded.)
Try Stratified Sampling.
", "votes": 2, "replies": [] }, { "id": 1316930, "author_name": "Tian", "author_url": "", "post_date": "2021-05-21T02:56:50.817000", "content": "My first try.
\nCV 0.974, LB 0.93.
\n…
Issue might be in decoding method of your TPU training pipeline.
\nWhen you encode image in png or jpg and decode it in tensorflow, values change actually.
\nThat pipeline is built for fast processing and decode approximate values.
So try not to save image as an encoded image but use FloatList to encode and decode.
\nThat will help. I guess.
\nBest of luck.
", "votes": 1, "replies": [] }, { "id": 1319186, "author_name": "zhangeng", "author_url": "", "post_date": "2021-05-23T02:26:14.927000", "content": "Cutmix & mixup for preventing over fitting and in this project, you will get the result that CV & LB scores are highly correlated. Now I'm thinking about what does cutmix & mixup bring to our project besides preventing over fitting? My CV is closely related to LB, cv97.7 LB 97+
", "votes": 6, "replies": [] } ] }, { "id": 1305497, "author_name": "Salman", "author_url": "", "post_date": "2021-05-13T10:12:04.133000", "content": "So now my single model gives 0.97
\nBut it took 7 hours for 1 Fold.
\nLet's see what we get from 5 Fold pipeline.
\nIt's running….!!!!
CV:0.987460473479882 LB0.98 large model
", "votes": 6, "replies": [ { "id": 1310809, "author_name": "sin", "author_url": "", "post_date": "2021-05-17T01:55:44.097000", "content": "This is a 4 fold CV with a single model?
", "votes": 0, "replies": [] }, { "id": 1310818, "author_name": "patriot", "author_url": "", "post_date": "2021-05-17T02:04:33.140000", "content": "yes,LB is mean of 4 fold
", "votes": 2, "replies": [] }, { "id": 1310825, "author_name": "sin", "author_url": "", "post_date": "2021-05-17T02:10:21.170000", "content": "Great. Please keep this updated. I want to compare my score as well. My current setup's CV(5fold) is around 0.985 with B0 and 256 image size. Still need to discover more aliens to match you :D.
", "votes": 2, "replies": [] } ] }, { "id": 1353200, "author_name": "patriot", "author_url": "", "post_date": "2021-06-17T01:32:03.493000", "content": "lb 1.000
\ni end this Competition.
Congratulation, you did it ;)
", "votes": 0, "replies": [] }, { "id": 1353203, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-17T01:34:05.320000", "content": "you are awesome lol.
", "votes": 0, "replies": [] } ] }, { "id": 1350256, "author_name": "patriot", "author_url": "", "post_date": "2021-06-15T11:36:58.843000", "content": "efnetb0 4fold 768*768
\ncv:0.990752314854183
\nlb:0.984
Are you using all the 6 channels for your experiment?
", "votes": 0, "replies": [] }, { "id": 1351737, "author_name": "patriot", "author_url": "", "post_date": "2021-06-16T14:57:38.363000", "content": "yes
\nonly 3ch also work and it's fast
CV 0.9906
\nLB 0.97
\nresnet18d
\n512*512
CV 0.9908
\nLB 0.98
\n5models ensemble
CV 0.9315 LB 0.97
", "votes": 3, "replies": [ { "id": 1328668, "author_name": "Baran Hashemi", "author_url": "", "post_date": "2021-05-30T12:31:57.587000", "content": "What? impressive. Congrats.
\nMay I ask what did you do?
So I have just checked up my code again , I was trying mixup augmentaion and I accidentally did it on validation dataset. So it is not that impressive lol
", "votes": 2, "replies": [] } ] }, { "id": 1315191, "author_name": "MOONMOON", "author_url": "", "post_date": "2021-05-19T16:10:41.827000", "content": "CV 0.9826
\nLB 0.97(i guess is around 0.975)
\n4 fold ensemble
\nnfnet_f0
\nspatial
\nAug:Resize256，horizontalflip&verticalflip
CV: 0.9920
\nLB: 0.98
\nEfficientnet-B2, 5fold, 512x512
I have some questions.
\nDid you use Mixup or any other augs?
\nYou used 3 channels or 6 channels?
\nHow many epochs?
\nDid you use BCE or Focal Loss?
Mixup: yes
\n3channels
\n30epoch
\nBCE loss
Yes but my question is did you used any other augmentation as well like (Cutmix or Augmix or something like that)?
", "votes": 0, "replies": [] }, { "id": 1351806, "author_name": "Salman", "author_url": "", "post_date": "2021-06-16T16:06:45.750000", "content": "I do have another question.
\nDid you use any kind of sampler for dataloader as well? Like balanced sampling or something?
CV 0.988119 +/- 0.002054, 4 folds ensemble on LB 0.98
", "votes": 3, "replies": [ { "id": 1315110, "author_name": "Baran Hashemi", "author_url": "", "post_date": "2021-05-19T15:10:52.687000", "content": "You also used pure vision models (e.g NFNet or Effnet)?
", "votes": 0, "replies": [] } ] }, { "id": 1347806, "author_name": "nyanp", "author_url": "", "post_date": "2021-06-13T13:54:01.207000", "content": "Val AUC: 0.9923
\nLB: 0.98
\nefficientnet-b1, single fold, Colab Pro
Can I get some hint?
\nwhich loss function?
\nuse label smoothing?
\nvalidation loss?
\nimage size?
\nefficientnet-b1 better than efficientnet-b0 ?
loss: BCE
\nlabel smoothing: no
\nimage size: 512x512
\nefficientnet-b1 is slightly better than b0 in my case(~0.0001)
Thanks for sharing! How many epochs do you train ?
", "votes": -1, "replies": [] } ] }, { "id": 1343532, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-10T08:58:43.717000", "content": "My best model is following:
\nb0
\n512 * 512
\n5fold
\nmixup and another augmentation
\nCV 0.991
\nLB 0.98
You used all channels or just 0,2,4 ?
", "votes": 0, "replies": [] }, { "id": 1343826, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-10T13:17:56.630000", "content": "3ch instead of 6ch
", "votes": 3, "replies": [] }, { "id": 1343827, "author_name": "assign", "author_url": "", "post_date": "2021-06-10T13:19:11.497000", "content": "thank you for sharing information
", "votes": 0, "replies": [] }, { "id": 1345200, "author_name": "Dmitry", "author_url": "", "post_date": "2021-06-11T11:51:57.800000", "content": "Thanks for sharing! How long epochs it takes to train one fold ?
", "votes": 0, "replies": [] }, { "id": 1345251, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-11T12:25:31.217000", "content": "15epoch
\nOver 15 did not work well in my setting.
Wow.
\nFor me it took 70 epochs each fold to achieve 0.98 LB and 0.99 CV.
\nI am missing something then.
\nThanks.
Still using B0 with Image Size 512x512?
", "votes": 0, "replies": [] }, { "id": 1345399, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-11T14:49:16.940000", "content": "OMG… 70 epoch !? I can't do that lol
\nI tried b4 but b0 is better.
", "votes": 1, "replies": [] }, { "id": 1345615, "author_name": "Salman", "author_url": "", "post_date": "2021-06-11T17:51:59.590000", "content": "Yeah. I guess max_lr was set to 5e-03. I should increase it a bit I guess.
", "votes": 0, "replies": [] }, { "id": 1346026, "author_name": "Salman", "author_url": "", "post_date": "2021-06-12T04:00:37.567000", "content": "I guess it might be some kind of normalization that is required for 0.99 LB.
", "votes": 0, "replies": [] }, { "id": 1348455, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-14T04:32:55.927000", "content": "I can't tell you in detail but my LR is not so large.
", "votes": 0, "replies": [] } ] }, { "id": 1306134, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-05-13T16:16:08.943000", "content": "There is a gap between CV and LB indeed, but yours seems to correlate well.
", "votes": 3, "replies": [] }, { "id": 1305547, "author_name": "YYama", "author_url": "", "post_date": "2021-05-13T11:00:32.577000", "content": "CV: 0.9736 LB: 0.96
\nCV: 0.9774 LB: 0.96
\nCV: 0.9824 LB: 0.96
\nAll have the same score on public LB, but when sorted, the scores seems to improve outside the indicated digits.
CV: 0.9902
\nLB: 0.98
\nsingle fold model from 5 folds
\nefficientnetb0
\n512*512
\nAfter seeing @ttahara discussion and notebook, I get this effective single fold model by some changes.
Thank you for seeing my posts.
\nBreaking 0.98 by single fold is great🎉
Model: EfficientNetB0
\nImage size: original
\nFolds : 5
\nAugmentations: FMix
\nCV: 0.988
\nLB: 0.97
hflip vflip blur CV:0.98 LB:0.93
", "votes": 4, "replies": [ { "id": 1316749, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-20T20:38:22.627000", "content": "0.98 / 0.93 is the biggest CV gap I've seen so far - what did you do?!
", "votes": 7, "replies": [] }, { "id": 1340401, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-07T21:41:09.287000", "content": "screwed up
", "votes": 0, "replies": [] } ] }, { "id": 1310798, "author_name": "zhangeng", "author_url": "", "post_date": "2021-05-17T01:14:09.047000", "content": "all train
\nmodel:nfnet_l0
\nno resize
\nno augumentaion
\nepoch 3
\nlb:97
Split: StratifiedKFold 4 folds?
", "votes": 0, "replies": [] }, { "id": 1312290, "author_name": "zhangeng", "author_url": "", "post_date": "2021-05-18T01:11:09.583000", "content": "All images are used for training, no verification set
", "votes": 1, "replies": [] } ] }, { "id": 1306694, "author_name": "patriot", "author_url": "", "post_date": "2021-05-14T02:44:11.817000", "content": "epoch10 add a aug resize hflip vflip CV:0.98614442 LB:0.97
\n(lower than no resize ,higher than \"another aug \")
hflip and flip on spatial images? or on individual channel?
", "votes": 2, "replies": [] }, { "id": 1310752, "author_name": "patriot", "author_url": "", "post_date": "2021-05-16T23:13:36.950000", "content": "do augumentation after nakama's preprocess(image = np.vstack(image).transpose((1, 0)).astype(np.float32))
\nI think on each channel is worth trying.
", "votes": 2, "replies": [] }, { "id": 1325934, "author_name": "Baran Hashemi", "author_url": "", "post_date": "2021-05-28T05:42:39.553000", "content": "Have you tried it on each channel?
", "votes": 0, "replies": [] } ] }, { "id": 1353088, "author_name": "Tahsin Mostafiz", "author_url": "", "post_date": "2021-06-16T21:18:26.007000", "content": "Eff B1
\n5 Fold 512x512
\nOOF 0.9846
\nLB 0.979
\nA little relieved to see that my CV-LB gap is smaller than I thought
CV: 0.943, LB: 0.93
\nModel: EfficientNet-B0
\nInitial weights: imagenet
\n5 fold cross validation, earlystopping, reduce LR on plateau, Adam, Mixup augmentation
Training/validation metrics: https://wandb.ai/ayush-thakur/kaggle-seti
\nWondering why I am stuck with 0.93 even though I am using kinda same strategy.
", "votes": 1, "replies": [ { "id": 1319480, "author_name": "Salman", "author_url": "", "post_date": "2021-05-23T08:26:45.490000", "content": "You should stack 273 dimension instead of 256 dimension.
\nUse this one: np.vstack(image).transpose((1, 0))
Thanks for the tip. A follow-up question, stack all the 6 spectrograms or just spectrograms with target signal (0, 2 and 4)?
", "votes": 0, "replies": [] }, { "id": 1319486, "author_name": "Salman", "author_url": "", "post_date": "2021-05-23T08:36:41.300000", "content": "I am stacking all channels.
", "votes": 0, "replies": [] }, { "id": 1319634, "author_name": "Rodolphe Lampe", "author_url": "", "post_date": "2021-05-23T12:05:14.030000", "content": "Why should we transpose((0,1)) ??
", "votes": 0, "replies": [] }, { "id": 1320230, "author_name": "Salman", "author_url": "", "post_date": "2021-05-23T22:05:51.213000", "content": "Not necessary to use transpose.
", "votes": 1, "replies": [] }, { "id": 1331264, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-01T11:21:02.190000", "content": "Yes I was also curious on the rationale, because it should not matter in theory.
", "votes": 0, "replies": [] } ] }, { "id": 1311425, "author_name": "", "author_url": "", "post_date": "2021-05-17T11:50:19.983000", "content": "", "votes": 1, "replies": [ { "id": 1311573, "author_name": "", "author_url": "", "post_date": "2021-05-17T13:48:27.667000", "content": "", "votes": 1, "replies": [] }, { "id": 1318238, "author_name": "", "author_url": "", "post_date": "2021-05-22T06:21:12.820000", "content": "", "votes": 1, "replies": [] }, { "id": 1319179, "author_name": "", "author_url": "", "post_date": "2021-05-23T01:55:55.620000", "content": "", "votes": 1, "replies": [] }, { "id": 1343205, "author_name": "", "author_url": "", "post_date": "2021-06-10T05:36:33.853000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1343110, "author_name": "", "author_url": "", "post_date": "2021-06-10T03:38:33.890000", "content": "", "votes": 2, "replies": [] }, { "id": 1322869, "author_name": "", "author_url": "", "post_date": "2021-05-25T18:46:33.127000", "content": "", "votes": 2, "replies": [] }, { "id": 1340390, "author_name": "", "author_url": "", "post_date": "2021-06-07T21:23:06.433000", "content": "", "votes": 5, "replies": [] } ], "raw_markdown_by_id": { "1304897": "I got an idea [this notebook](https://www.kaggle.com/yasufuminakama/seti-nfnet-l0-starter-training). Thanks for sharing!!\n\nmy baseline\nCV:0.96806077 LB:0.96\n\n Split: StratifiedKFold 4 folds\nmodel:efnetB0\nno resize\nno augumentaion\nepoch 1\n\n...epoch1→５ CV:0.97831633691 LB:0.96(higher than naive baseline)\n...epoch1→10 add aug CV:0.98497151 LB:0.97\n...epoch1→10 add another aug CV:0.9821733582 LB:0.97(lower than above)\n\nI am suffering from the gap between CV and LB and over confidence....\n\n", "1326592": "I think the gap between CV and LB is not only a frustating annoyance, but the key to understand the real challenge of this competition\n\nI'am astrophysicis myself, but not related at the SETI project in anyway, and this sentence has been in my mind from the beginning\n\n\"While it would be nice to train our algorithms entirely on observations of interplanetary spacecraft, there are not many examples of them, and we also want to be able to find a wider range of signal types. So we’ve turned to simulating technosignature candidates.\"\n\nSo, they are specially interested in unknown signals.... literally, they don't know what are looking for !!!\n\nThis is only my personal opinion, but if i were one of them i would expose the next problem \n\nOBJECTIVE: Finding anomalies that we don't know how they are\n1. Build a Dataset with simulated signals based on the research and information gathered over the last 40 years\n\n2. Ask the smart guys of the SETI project to build a Neural Network to dectected them.... They already are really good, as you can see in this post of @manabendrarout\n\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/239245\n\n3. Test the Dataset with this models to achieve a CV/LB greater of 0.99... without gap logically because is the same Dataset for both\n\n4. Split the original Dataset in Train/Test folds randmnly\n\n5. **Add a litte batch of positive samples at the Test Dataset **that don't match any kind of pattern of the positive original targets... possibly REAL samples (only guessing).\n\n6. Check that there is a gap between the CV (testing the train samples) and the LB (testing the test samples) whit the NN of the smart SETI guys\n\nTo understand the gap we have to analyze the confussion MAtrix. Por example, with my best model I achiveve CV = 0.988 (and LB = 0.96, but that is not important here).\n\nThe confussion matrix say me that:\n\n\nRelation between False Positives / True Positives (used to compute the ROC Curve) taking true positive when the probability is greater than 0.5 is 21 /664 = 3.16%... A really good result !!\n\nBut, the relation between False NEgative / True Positives is 62 / 664 = 9.3 %... not so good\n\nSo, even with a magnific 0.988 I fail to detect about the 10% of samples !!!\n\nIf, how I am guessing in point 5, they add some new samples at the test dataset, then this relation FN/TP will growth fast in this dataset, but the LB only will decrease a bit...\n\nCONCLUSION: SETI guys have found the way to hidde the REAL samples !!... and detect them is the real challenge of this competition\n\nI think they are not intereseted in a high CV or LB score (if it's high enough, ie. >0.97), but the submissions with the minimum gap, because they mean they are detecting the interesting REAL samples\n\nLoking the submissions and the comments, and Making some fast computations I guess there are about 5 - 10 % of \"REAL samples\", that traslates a 1 % of CV/LB gap\n\nMore of two hundred people have LB = 0.97, but only one have LB = 0.99... becasuse this 1% gap.\n\nSorry for the long post, and remember I am only guessing here for fun", "1312002": "Submitted a single fold (CV ~0.987), LB is 0.97.\nImage: 256 x 256. \nModel: b0. \nSpatial or Channel: spatial. \nAugmentations: Mixup (added). \nTricks: maybe", "1331507": "CV 0.9887\nLB 0.98\n5 fold ensemble\nb4\n512*512\nweak augmentation", "1331084": "happy to update my LB score😄\n\nCV: 0.991, LB: 0.98(5fold-avg)", "1313000": "CV 0.9884\nLB 0.98\n5 fold ensemble", "1319750": "CV: 0.9868, LB: 0.97 (5fold-avg)\n\nModel: resnet18d\nInitial weights: ImageNet\nTrain-Val Split: StratifiedKFold(K=5)\nAugmentation: HorizontalFlip, VerticalFlip, ShiftScaleRotate, RandomResizedCrop", "1310459": "cv:0.9880063998426094 LB:0.97(second) epoch15 baseline+a aug B0\ncv:0.986708799271502 LB:0.97(best) epoch10 baseline+a aug B0\ncv:0.9878064170595553 LB:0.97 B0\noverfit... ", "1323151": "Model: ensemble of models which are based on [efficientnet-b0, resnet18d].\n5 fold\nAug: HFlip, VFlip, ShiftScale, Mixup\nCV: 0.989x LB: 0.97\n\nI'm facing the gap between CV and LB. The player who reaches LB 0.98x already achieve CV > 0.99x? ", "1316452": "CV vs LB is very strange for me. My best model reaches an 4-fold out-of-fold AUC of 0.982 but submission to LB gives only 0.96... \nWhat/Where should I investigate? What could be the reason for the mismatch?\n\n(The indivisual folds are stratified and all have AUC>0.98. \nThe BCE loss of train and val is 0.05. No overfitting here. \nThe AUC indicates an overfit with train=0.997 and val=0.983\nI have similar results with DenseNet201/EffcientNet/VGG using pretrained \"imagenet\"-weights.\nTraining from scratch with a VGG like network works also very good.\nI use TPU/32batchsizex8 / 256x256 spatial concat / all channels / On and off times color-coded.)", "1305497": "So now my single model gives 0.97\nBut it took 7 hours for 1 Fold.\nLet's see what we get from 5 Fold pipeline.\nIt's running....!!!! ", "1310792": "CV:0.987460473479882 LB0.98 large model", "1353200": "lb 1.000\ni end this Competition.", "1350256": "efnetb0 4fold 768*768\ncv:0.990752314854183\nlb:0.984", "1344227": "CV 0.9906\nLB 0.97\nresnet18d\n512*512\n\nCV 0.9908\nLB 0.98\n5models ensemble", "1328649": "CV 0.9315 LB 0.97", "1315191": "CV 0.9826\nLB 0.97(i guess is around 0.975)\n4 fold ensemble\nnfnet_f0\nspatial\nAug:Resize256，horizontalflip&verticalflip", "1349836": "CV: 0.9920\nLB: 0.98\nEfficientnet-B2, 5fold, 512x512", "1312809": "CV 0.988119 +/- 0.002054, 4 folds ensemble on LB 0.98", "1347806": "Val AUC: 0.9923\nLB: 0.98\nefficientnet-b1, single fold, Colab Pro", "1343532": "My best model is following:\nb0\n512 * 512\n5fold\nmixup and another augmentation\nCV 0.991\nLB 0.98", "1306134": "There is a gap between CV and LB indeed, but yours seems to correlate well.", "1305547": "CV: 0.9736 LB: 0.96\nCV: 0.9774 LB: 0.96\nCV: 0.9824 LB: 0.96\nAll have the same score on public LB, but when sorted, the scores seems to improve outside the indicated digits.", "1339374": "CV: 0.9902\nLB: 0.98\nsingle fold model from 5 folds\nefficientnetb0\n512*512\nAfter seeing @ttahara discussion and notebook, I get this effective single fold model by some changes.", "1324466": "Model: EfficientNetB0\nImage size: original\nFolds : `5`\nAugmentations: FMix\nCV: `0.988` \nLB: `0.97`", "1316724": "hflip vflip blur CV:0.98 LB:0.93", "1310798": "all train\nmodel:nfnet_l0\nno resize\nno augumentaion\nepoch 3\nlb:97", "1306694": "epoch10 add a aug resize hflip vflip CV:0.98614442 LB:0.97\n(lower than no resize ,higher than \"another aug \")", "1353088": "Eff B1\n5 Fold 512x512\nOOF **0.9846**\nLB **0.979** \nA little relieved to see that my CV-LB gap is smaller than I thought", "1319477": "CV: 0.943, LB: 0.93\n\nModel: EfficientNet-B0\nInitial weights: imagenet\n5 fold cross validation, earlystopping, reduce LR on plateau, Adam, Mixup augmentation\n\nTraining/validation metrics: https://wandb.ai/ayush-thakur/kaggle-seti\n\nWondering why I am stuck with 0.93 even though I am using kinda same strategy. ", "1311425": "LB0.97+LB0.98=LB0.98(lower than left one)\nSo, I think we we will have to keep sticking to single model so far .....", "1343110": "CV: 0.988\nLB: 0.97\n512 * 512 eff_b1\nAny Suggestions?", "1322869": "Model: Resnest50\n5 fold\nAug: Horizontal, Vertical Flip, Rotation\nOOF CV: 0.9815 LB: 97", "1340390": "" } }