![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2F8adf43c591ce4f78cdd053f409b013ea%2F47n6z2.jpg?generation=1594232067183368&alt=media\")
Thanks to @shonenkov \nnotebook and help with initial code [link], 100s of us are at the baseline score of 0.921
\n\nlooking for inputs and guidance on what was done to improve and move further!
\n\ni tried several parameter changes and most had no impact while some deteriorated the performance. Would be great for newbies to understand what one should focus on!
\n\nAny guidance and suggestions on direction to take is very appreciated. Thanks in advance!\nš š
", "messages": [ { "id": 918165, "postDate": "2020-07-07T03:31:31.743Z", "content": "Thanks to @shonenkov \nnotebook and help with initial code [link], 100s of us are at the baseline score of 0.921
\n\nlooking for inputs and guidance on what was done to improve and move further!
\n\ni tried several parameter changes and most had no impact while some deteriorated the performance. Would be great for newbies to understand what one should focus on!
\n\nAny guidance and suggestions on direction to take is very appreciated. Thanks in advance!\nš š
", "rawMarkdown": "Thanks to @shonenkov \nnotebook and help with [initial code [link]](https://www.kaggle.com/shonenkov/train-inference-gpu-baseline), 100s of us are at the baseline score of 0.921\n\nlooking for inputs and guidance on what was done to improve and move further!\n\ni tried several parameter changes and most had no impact while some deteriorated the performance. Would be great for newbies to understand what one should focus on!\n\nAny guidance and suggestions on direction to take is very appreciated. Thanks in advance!\nš š ", "votes": 15 }, { "id": 920307, "postDate": "2020-07-08T13:45:29.117Z", "content": "I am a bit perplexed by the much higher LB the best pub kernel got over CV. Is this something others are also observing? For me LB is usually worse than CV.
", "rawMarkdown": "I am a bit perplexed by the much higher LB the best pub kernel got over CV. Is this something others are also observing? For me LB is usually worse than CV.", "votes": 6, "replies": [ { "id": 920314, "postDate": "2020-07-08T13:51:45.113Z", "content": "1000 images for the Public LB is very low when you consider there are esentially 10 or 12 classes (Cover plus 3 schemes, all at 3 jpeg ratios) each with quite different detectability. Here is a good analysis of the effect of sample size https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/162936
", "rawMarkdown": "1000 images for the Public LB is very low when you consider there are esentially 10 or 12 classes (Cover plus 3 schemes, all at 3 jpeg ratios) each with quite different detectability. Here is a good analysis of the effect of sample size https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/162936", "votes": 2 }, { "id": 920324, "postDate": "2020-07-08T13:59:09.620Z", "content": "That's correct, yes, at least my LB is in a smaller range though, with higher CV.
\n\nThe public kernel has 0.913x as CV and 0.921 as LB, that's a big jump.
", "rawMarkdown": "That's correct, yes, at least my LB is in a smaller range though, with higher CV.\n\nThe public kernel has 0.913x as CV and 0.921 as LB, that's a big jump.", "votes": 3 }, { "id": 920332, "postDate": "2020-07-08T14:03:32.210Z", "content": "I've been seeing around +-.007 against CV. I suppose it is possible the test set is different from the trainset, not in terms of imagery but in terms of payload. The organiser didn't declare the train/test split strategy AFAIK, but I hope it is random.\nNote that there are whole sequences of similar images of similar objects (as many as 100+), and some fold strategies may be either perplexing or leaking.
", "rawMarkdown": "I've been seeing around +-.007 against CV. I suppose it is possible the test set is different from the trainset, not in terms of imagery but in terms of payload. The organiser didn't declare the train/test split strategy AFAIK, but I hope it is random.\nNote that there are whole sequences of similar images of similar objects (as many as 100+), and some fold strategies may be either perplexing or leaking. ", "votes": 2 }, { "id": 920373, "postDate": "2020-07-08T14:51:24.883Z", "content": "I totally agree that there is leak potential in CV, but as far as I can see the public kernel is doing GroupKFold on image name. Anyways, maybe it was lucky. I was just a bit wondering about that difference there.
", "rawMarkdown": "I totally agree that there is leak potential in CV, but as far as I can see the public kernel is doing GroupKFold on image name. Anyways, maybe it was lucky. I was just a bit wondering about that difference there.", "votes": 2 }, { "id": 920425, "postDate": "2020-07-08T15:38:19.870Z", "content": "\n\n\"The public kernel has 0.913x as CV and 0.921 as LB, that's a big jump.\"
\n\nthis is what i get. CV 0.921~ 0.922, LB 0.906~0.916 (when i train very long, the validation loss values decreases with the same validation metric. LB metric decreases)
\n\nbut for the public kernel https://www.kaggle.com/shonenkov/train-inference-gpu-baseline, the opposite happens (LB 0.921, CV0.912)
\n\nafter reading the ALASKA -1 paper, i won't be surprised that the organizer has hide some private test images taken different cameras. So private score may be different again
", "rawMarkdown": "@philippsinger \n\n\"The public kernel has 0.913x as CV and 0.921 as LB, that's a big jump.\" \n\nthis is what i get. CV 0.921~ 0.922, LB 0.906~0.916 (when i train very long, the validation loss values decreases with the same validation metric. LB metric decreases) \n\nbut for the public kernel https://www.kaggle.com/shonenkov/train-inference-gpu-baseline, the opposite happens (LB 0.921, CV0.912)\n\n---\n\nafter reading the ALASKA -1 paper, i won't be surprised that the organizer has hide some private test images taken different cameras. So private score may be different again", "votes": 4 }, { "id": 920475, "postDate": "2020-07-08T16:11:22.277Z", "content": "Thank you @hengck23 for confirming, I am observing very similar behavior, but using my own pipeline, not the public kernel. Is your setup similar to the public one? I am curious what is different there, or if it was lucky.
", "rawMarkdown": "Thank you @hengck23 for confirming, I am observing very similar behavior, but using my own pipeline, not the public kernel. Is your setup similar to the public one? I am curious what is different there, or if it was lucky." }, { "id": 920499, "postDate": "2020-07-08T16:28:00.690Z", "content": "Our LB is also usually lower than CV. I think it is luck mostly. Even between epoch n and epoch n+1, we could get 0.003 difference on LB while CV only improves by less than 0.001. There will be a medium size local shake-up.
", "rawMarkdown": "Our LB is also usually lower than CV. I think it is luck mostly. Even between epoch n and epoch n+1, we could get 0.003 difference on LB while CV only improves by less than 0.001. There will be a medium size local shake-up.", "votes": 1 }, { "id": 920623, "postDate": "2020-07-08T18:05:45.257Z", "content": "Further finetune Shonenkov kernel CV .915 and LB .919 :) .. There is either something wrong with metrics , or waiting for surprising results after comp . Feels like I am doing an wild goose chase ..
", "rawMarkdown": "Further finetune Shonenkov kernel CV .915 and LB .919 :) .. There is either something wrong with metrics , or waiting for surprising results after comp . Feels like I am doing an wild goose chase ..", "votes": 1 }, { "id": 920627, "postDate": "2020-07-08T18:08:22.233Z", "content": "I got 914-CV, 907-LB with the kernel. Then, further tuning got it to 920-CV, 917-LB.
", "rawMarkdown": "I got 914-CV, 907-LB with the kernel. Then, further tuning got it to 920-CV, 917-LB.", "votes": 2 }, { "id": 920670, "postDate": "2020-07-08T18:41:34.307Z", "content": "\"i won't be surprised that the organizer has hide some private test images taken different cameras\"\nI thought about using the winning solution from Kaggle's Camera Source Identification task but not sure this would comply with rules, since data is scraped from web and rights may breach rules. https://www.kaggle.com/c/sp-society-camera-model-identification\nWith enough labels it may be possible to do PRNU analysis, though the preprocessing pipeline here may destroy it. There are quite a few researchers in this challenge and it is possible they are using such tools and datasets (which they should disclose within 5 days if so :)
", "rawMarkdown": "\"i won't be surprised that the organizer has hide some private test images taken different cameras\"\nI thought about using the winning solution from Kaggle's Camera Source Identification task but not sure this would comply with rules, since data is scraped from web and rights may breach rules. https://www.kaggle.com/c/sp-society-camera-model-identification\nWith enough labels it may be possible to do PRNU analysis, though the preprocessing pipeline here may destroy it. There are quite a few researchers in this challenge and it is possible they are using such tools and datasets (which they should disclose within 5 days if so :)" }, { "id": 920772, "postDate": "2020-07-08T19:34:51.740Z", "content": "FWIW, my validation scores have been pretty spot on with the LB in most cases, +/- 0.002.
", "rawMarkdown": "FWIW, my validation scores have been pretty spot on with the LB in most cases, +/- 0.002.", "votes": 2 }, { "id": 920857, "postDate": "2020-07-08T21:09:30.613Z", "content": "For me LB is usually -0.005 or more. But I cant afford doing more than one fold.
", "rawMarkdown": "For me LB is usually -0.005 or more. But I cant afford doing more than one fold.", "votes": 2 }, { "id": 923281, "postDate": "2020-07-10T17:22:22.020Z", "content": "to find out why the public kernel results is better than mine, i re-run my experiment using his fold splitting [1]. With the same training and validation data, I can get similar high LB score (higher than CV) as in [1], despite different optimizer and i am using my own train framework and model. hence I conclude:
\n\n[1] https://www.kaggle.com/shonenkov/train-inference-gpu-baseline, the
", "rawMarkdown": "to find out why the public kernel results is better than mine, i re-run my experiment using his fold splitting [1]. With the same training and validation data, I can get similar high LB score (higher than CV) as in [1], despite different optimizer and i am using my own train framework and model. hence I conclude:\n\n- results is could be highly train data dependent?\n- some kind of nearest neighbors post processing may help (i.e. find the nearest train neighbors of the train for each of the test image/dct block, etc)\n- take care of your data split, ensemble, etc\n\n\n[1] https://www.kaggle.com/shonenkov/train-inference-gpu-baseline, the \n\n" }, { "id": 923292, "postDate": "2020-07-10T17:39:01.123Z", "content": "@hengck23 interesting - are you sure the GroupKFold is replicating it exactly as he did not set a seed when shuffling?
", "rawMarkdown": "@hengck23 interesting - are you sure the GroupKFold is replicating it exactly as he did not set a seed when shuffling? ", "votes": 1 }, { "id": 923305, "postDate": "2020-07-10T17:54:43.797Z", "content": "I have not checked it clearly the results of the split, but my train_dataset[0] sometimes shows a plant picture and sometimes an elephant's behind , so i dont think its always giving same result .
", "rawMarkdown": "I have not checked it clearly the results of the split, but my train_dataset[0] sometimes shows a plant picture and sometimes an elephant's behind , so i dont think its always giving same result ." }, { "id": 923311, "postDate": "2020-07-10T18:01:33.777Z", "content": "If you do a full kernel restart, you should always get the plant.
", "rawMarkdown": "If you do a full kernel restart, you should always get the plant.", "votes": 1 }, { "id": 923325, "postDate": "2020-07-10T18:12:40.050Z", "content": "@authman . Nice observation, thanks :)
", "rawMarkdown": "@authman . Nice observation, thanks :) " }, { "id": 923393, "postDate": "2020-07-10T19:46:05.027Z", "content": "If you plot local-cv results against id's from published folds, or other folds, you see the lumpiness. Here, results from a stratified fold, plotted against sequential image_names.\n\n0-60000 0.936\n60000-120000 0.922\n120000-180000 0.915\n180000-240000 0.921\n240000-300000 0.924\n
> Psi wrote:\n> \n> as far as I can see the public kernel is doing GroupKFold on image name.
\n\nPlenty of very similar objects. I've not been able to exploit this but see ...
\n\nTrain:
\n\n[25303, 25311, 25333, 25339, 25344, 25348, 25359, 25369, 25373, 25374, 25378, 25381, 25385, 25398, 25413, 25414, 25421, 25422, 25422, 25448, 25467, 25471, 25478, 25485, 25493, 25496, 25501, 25514, 25521, 25525, 25528, 25536, 25537, 25540, 25542, 25544, 25552, 25557, 25562, 25565, 25583, 25586, 25589, 25592, 25599, 25608, 25610, 25626, 25626, 25628, 25654, 25655, 25666, 25667, 25669, 25676, 25677, 25690, 25691, 25705, 25708, 25716, 25742, 25749, 25761, 25769, 25771, 25776, 25780, 25782, 25784, 25795, 25804, 25809, 25820, 25824, 25825, 25835, 25839, 25855, 25861, 25865, 25867, 25866, 25875, 25883, 25884, 25898, 25909, 25912, 25920, 25922, 25930, 25939, 25950, 25955, 25960, 25969, 25970, 25971, 25979, 25982, 25984, 25996, 26020, 26026, 26027, 26029, 26032, 26035, 26038, 26041, 26042]
Test:
\n\n[0033, 0544, 0559, 0566, 0680, 0750, 1005, 1265, 1645, 1826, 2086, 2087, 2108, 2337, 2396, 2713, 2795, 2860, 2925, 3044, 3049, 3126, 3245, 3594, 3599, 3827, 3852, 3915, 4179, 4183, 4269, 4339, 4559]
@philippsinger \n\"are you sure the GroupKFold is replicating it exactly as he did not set a seed when shuffling?\"
\n\ni downloaded and used his split csv file
", "rawMarkdown": "@philippsinger \n\"are you sure the GroupKFold is replicating it exactly as he did not set a seed when shuffling?\"\n\ni downloaded and used his split csv file", "votes": 1 }, { "id": 924113, "postDate": "2020-07-11T09:05:06.173Z", "content": "@hengck23 Oh I see he has it in his dataset.
", "rawMarkdown": "@hengck23 Oh I see he has it in his dataset." } ] }, { "id": 918550, "postDate": "2020-07-07T10:06:37.343Z", "content": "Try EfficientNet B7. You will get a baseline score of 0.93+
", "rawMarkdown": "Try EfficientNet B7. You will get a baseline score of 0.93+", "votes": 4, "replies": [ { "id": 918560, "postDate": "2020-07-07T10:15:23.013Z", "content": "Thank you @davidtong
\n\nappreciate your help and guidance!
", "rawMarkdown": "Thank you @davidtong \n\nappreciate your help and guidance!\n" }, { "id": 920616, "postDate": "2020-07-08T18:00:43.493Z", "content": "It's not easy to be 7.
", "rawMarkdown": "It's not easy to be 7.", "votes": 1 }, { "id": 920636, "postDate": "2020-07-08T18:14:30.067Z", "content": "
a possible way to speedup:\n1. divide into train set T and validation set V\n2. yet divide train set into T1, T2 , T3 ... T4\n3. train on T1 and validate on T2.\n4. sort T2 according to predicted score and uniformly sample, e.g. say 20%. \n5. train with T1 + 20% of T2, and validate on T3.\n6. sort T3 according to predicted score and uniformly sample, e.g. say 20%. \n7. train with T1 + 20% of T2 + 20% T3 and validate on T4.\n...
\n\n.. . finally, train with T1 + 20% of T2 + 20% T3 + 20% T4 and validate on V.\n.... finally, train with T1 + T2 +T3 + T4 and validate on V.
\n\nthe idea is to have a reduced set representative of a larger set. since we do validation very n-th epoch, we use this validation to add to train data for free.
", "rawMarkdown": "a possible way to speedup:\n1. divide into train set T and validation set V\n2. yet divide train set into T1, T2 , T3 ... T4\n3. train on T1 and validate on T2.\n4. sort T2 according to predicted score and uniformly sample, e.g. say 20%. \n5. train with T1 + 20% of T2, and validate on T3.\n6. sort T3 according to predicted score and uniformly sample, e.g. say 20%. \n7. train with T1 + 20% of T2 + 20% T3 and validate on T4.\n...\n\n.. . finally, train with T1 + 20% of T2 + 20% T3 + 20% T4 and validate on V.\n.... finally, train with T1 + T2 +T3 + T4 and validate on V.\n\nthe idea is to have a reduced set representative of a larger set. since we do validation very n-th epoch, we use this validation to add to train data for free.\n", "votes": 3 }, { "id": 926267, "postDate": "2020-07-12T15:54:53.843Z", "content": "Is It going to be like progressive pseudolabeling? (Although we would have the real label in this case ) when you say sort and pick 20% this 20% is 10% with pred < 0.001 and 10% with pred > 10% something like that ?
", "rawMarkdown": "Is It going to be like progressive pseudolabeling? (Although we would have the real label in this case ) when you say sort and pick 20% this 20% is 10% with pred < 0.001 and 10% with pred > 10% something like that ?" }, { "id": 926384, "postDate": "2020-07-12T17:15:45.217Z", "content": "You could use a TPU. It only takes 24 hours for 35 full passes through 285k images at 512x512 on B7. This is on a v2-8 so you may get faster results on at v3-8.
", "rawMarkdown": "You could use a TPU. It *only* takes 24 hours for 35 full passes through 285k images at 512x512 on B7. This is on a v2-8 so you may get faster results on at v3-8.", "votes": 1 }, { "id": 926536, "postDate": "2020-07-12T19:14:15.133Z", "content": "Is this with TFRECORDS @hooong ? I tried with your amazing Kernel but was not able to use TFRECORDS in my colab .
", "rawMarkdown": "Is this with TFRECORDS @hooong ? I tried with your amazing Kernel but was not able to use TFRECORDS in my colab . " }, { "id": 927499, "postDate": "2020-07-13T12:35:45.997Z", "content": "There is a way to use TFRecords in colab. One of my teammates is running colab pro as well.
\n\nI believe you need to host the files on your own GCS bucket for colab
", "rawMarkdown": "There is a way to use TFRecords in colab. One of my teammates is running colab pro as well. \n\nI believe you need to host the files on your own GCS bucket for colab" }, { "id": 928795, "postDate": "2020-07-14T08:13:38.953Z", "content": "stop misguiding
", "rawMarkdown": "stop misguiding" }, { "id": 935013, "postDate": "2020-07-19T02:40:29.377Z", "content": "\"Is It going to be like progressive pseudolabeling? (Although we would have the real label in this case ) when you say sort and pick 20% this 20% is 10% with pred < 0.001 and 10% with pred > 10% something like that ?\"
\n\nit is not pesudo-labeling. it is importance sampling. you can google paper on \"importance sampling + deep learning.\" it is also related to hard mining, curriculum learning. when there are too many samples, we only want to train with less samples.
\n\ni do importance sampling for initial experiments to select a set of representative smaller train samples.
\n\nwhen your model is finalized, you can train with all samples.
", "rawMarkdown": "\"Is It going to be like progressive pseudolabeling? (Although we would have the real label in this case ) when you say sort and pick 20% this 20% is 10% with pred < 0.001 and 10% with pred > 10% something like that ?\"\n\nit is not pesudo-labeling. it is importance sampling. you can google paper on \"importance sampling + deep learning.\" it is also related to hard mining, curriculum learning. when there are too many samples, we only want to train with less samples.\n\ni do importance sampling for initial experiments to select a set of representative smaller train samples.\n\nwhen your model is finalized, you can train with all samples.", "votes": 1 } ] }, { "id": 918387, "postDate": "2020-07-07T08:10:44.883Z", "content": "Hi,
\n\nhonestly, no one is going to tell you how you can beat that benchmark. You have to try and think about what can you do using the same kernel. There are so many things that one can leverage from that kernel.
", "rawMarkdown": "Hi, \n\nhonestly, no one is going to tell you how you can beat that benchmark. You have to try and think about what can you do using the same kernel. There are so many things that one can leverage from that kernel. \n", "votes": 1, "replies": [ { "id": 918503, "postDate": "2020-07-07T09:33:50.807Z", "content": "The community has been gracious in helping reach the base level and a score of 0.921\nsomething I know would not have been possible with my own level of skills
\n\nI think some may provide some guidance on what possible approaches may be ...\nsharing helps in learning! fingers crossed!
", "rawMarkdown": "The community has been gracious in helping reach the base level and a score of 0.921\nsomething I know would not have been possible with my own level of skills\n\nI think some may provide some guidance on what possible approaches may be ...\nsharing helps in learning! fingers crossed!", "votes": 2 } ] }, { "id": 922890, "postDate": "2020-07-10T11:54:38.533Z", "content": "Thank you everyone! All your suggestions and guidance have been invaluable
\n\nYour support and suggestions have helped me get into the top 200! I am yet to try out b7, which I will do this weekend; and march a bit further ahead!
\n\nThank you for being such a supportive community!!
", "rawMarkdown": "Thank you everyone! All your suggestions and guidance have been invaluable\n\nYour support and suggestions have helped me get into the top 200! I am yet to try out b7, which I will do this weekend; and march a bit further ahead!\n\nThank you for being such a supportive community!!", "votes": 1 }, { "id": 926424, "postDate": "2020-07-12T17:53:39.850Z", "content": "awsme!!
", "rawMarkdown": "awsme!!", "votes": -2 }, { "id": 925517, "postDate": "2020-07-12T06:20:52.780Z", "content": "cool
", "rawMarkdown": "cool", "votes": -2 }, { "id": 934505, "postDate": "2020-07-18T13:54:40.697Z", "content": "i am not getting accuracy more than 91.4
", "rawMarkdown": "i am not getting accuracy more than 91.4" }, { "id": 922566, "postDate": "2020-07-10T07:50:13.567Z", "content": "nice
", "rawMarkdown": "nice" }, { "id": 918537, "postDate": "2020-07-07T09:54:47.280Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 926134, "postDate": "2020-07-12T14:18:34.467Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 920307, "author_name": "Psi", "author_url": "", "post_date": "2020-07-08T13:45:29.117000", "content": "I am a bit perplexed by the much higher LB the best pub kernel got over CV. Is this something others are also observing? For me LB is usually worse than CV.
", "votes": 6, "replies": [ { "id": 920314, "author_name": "robga", "author_url": "", "post_date": "2020-07-08T13:51:45.113000", "content": "1000 images for the Public LB is very low when you consider there are esentially 10 or 12 classes (Cover plus 3 schemes, all at 3 jpeg ratios) each with quite different detectability. Here is a good analysis of the effect of sample size https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/162936
", "votes": 2, "replies": [] }, { "id": 920324, "author_name": "Psi", "author_url": "", "post_date": "2020-07-08T13:59:09.620000", "content": "That's correct, yes, at least my LB is in a smaller range though, with higher CV.
\n\nThe public kernel has 0.913x as CV and 0.921 as LB, that's a big jump.
", "votes": 3, "replies": [] }, { "id": 920332, "author_name": "robga", "author_url": "", "post_date": "2020-07-08T14:03:32.210000", "content": "I've been seeing around +-.007 against CV. I suppose it is possible the test set is different from the trainset, not in terms of imagery but in terms of payload. The organiser didn't declare the train/test split strategy AFAIK, but I hope it is random.\nNote that there are whole sequences of similar images of similar objects (as many as 100+), and some fold strategies may be either perplexing or leaking.
", "votes": 2, "replies": [] }, { "id": 920373, "author_name": "Psi", "author_url": "", "post_date": "2020-07-08T14:51:24.883000", "content": "I totally agree that there is leak potential in CV, but as far as I can see the public kernel is doing GroupKFold on image name. Anyways, maybe it was lucky. I was just a bit wondering about that difference there.
", "votes": 2, "replies": [] }, { "id": 920425, "author_name": "hengck23", "author_url": "", "post_date": "2020-07-08T15:38:19.870000", "content": "\n\n\"The public kernel has 0.913x as CV and 0.921 as LB, that's a big jump.\"
\n\nthis is what i get. CV 0.921~ 0.922, LB 0.906~0.916 (when i train very long, the validation loss values decreases with the same validation metric. LB metric decreases)
\n\nbut for the public kernel https://www.kaggle.com/shonenkov/train-inference-gpu-baseline, the opposite happens (LB 0.921, CV0.912)
\n\nafter reading the ALASKA -1 paper, i won't be surprised that the organizer has hide some private test images taken different cameras. So private score may be different again
", "votes": 4, "replies": [] }, { "id": 920475, "author_name": "Psi", "author_url": "", "post_date": "2020-07-08T16:11:22.277000", "content": "Thank you @hengck23 for confirming, I am observing very similar behavior, but using my own pipeline, not the public kernel. Is your setup similar to the public one? I am curious what is different there, or if it was lucky.
", "votes": 0, "replies": [] }, { "id": 920499, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-07-08T16:28:00.690000", "content": "Our LB is also usually lower than CV. I think it is luck mostly. Even between epoch n and epoch n+1, we could get 0.003 difference on LB while CV only improves by less than 0.001. There will be a medium size local shake-up.
", "votes": 1, "replies": [] }, { "id": 920623, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-07-08T18:05:45.257000", "content": "Further finetune Shonenkov kernel CV .915 and LB .919 :) .. There is either something wrong with metrics , or waiting for surprising results after comp . Feels like I am doing an wild goose chase ..
", "votes": 1, "replies": [] }, { "id": 920627, "author_name": "Ų¹Ų«Ł Ų§Ł", "author_url": "", "post_date": "2020-07-08T18:08:22.233000", "content": "I got 914-CV, 907-LB with the kernel. Then, further tuning got it to 920-CV, 917-LB.
", "votes": 2, "replies": [] }, { "id": 920670, "author_name": "robga", "author_url": "", "post_date": "2020-07-08T18:41:34.307000", "content": "\"i won't be surprised that the organizer has hide some private test images taken different cameras\"\nI thought about using the winning solution from Kaggle's Camera Source Identification task but not sure this would comply with rules, since data is scraped from web and rights may breach rules. https://www.kaggle.com/c/sp-society-camera-model-identification\nWith enough labels it may be possible to do PRNU analysis, though the preprocessing pipeline here may destroy it. There are quite a few researchers in this challenge and it is possible they are using such tools and datasets (which they should disclose within 5 days if so :)
", "votes": 0, "replies": [] }, { "id": 920772, "author_name": "Branden Murray", "author_url": "", "post_date": "2020-07-08T19:34:51.740000", "content": "FWIW, my validation scores have been pretty spot on with the LB in most cases, +/- 0.002.
", "votes": 2, "replies": [] }, { "id": 920857, "author_name": "Psi", "author_url": "", "post_date": "2020-07-08T21:09:30.613000", "content": "For me LB is usually -0.005 or more. But I cant afford doing more than one fold.
", "votes": 2, "replies": [] }, { "id": 923281, "author_name": "hengck23", "author_url": "", "post_date": "2020-07-10T17:22:22.020000", "content": "to find out why the public kernel results is better than mine, i re-run my experiment using his fold splitting [1]. With the same training and validation data, I can get similar high LB score (higher than CV) as in [1], despite different optimizer and i am using my own train framework and model. hence I conclude:
\n\n[1] https://www.kaggle.com/shonenkov/train-inference-gpu-baseline, the
", "votes": 0, "replies": [] }, { "id": 923292, "author_name": "Psi", "author_url": "", "post_date": "2020-07-10T17:39:01.123000", "content": "@hengck23 interesting - are you sure the GroupKFold is replicating it exactly as he did not set a seed when shuffling?
", "votes": 1, "replies": [] }, { "id": 923305, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-07-10T17:54:43.797000", "content": "I have not checked it clearly the results of the split, but my train_dataset[0] sometimes shows a plant picture and sometimes an elephant's behind , so i dont think its always giving same result .
", "votes": 0, "replies": [] }, { "id": 923311, "author_name": "Ų¹Ų«Ł Ų§Ł", "author_url": "", "post_date": "2020-07-10T18:01:33.777000", "content": "If you do a full kernel restart, you should always get the plant.
", "votes": 1, "replies": [] }, { "id": 923325, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-07-10T18:12:40.050000", "content": "@authman . Nice observation, thanks :)
", "votes": 0, "replies": [] }, { "id": 923393, "author_name": "robga", "author_url": "", "post_date": "2020-07-10T19:46:05.027000", "content": "If you plot local-cv results against id's from published folds, or other folds, you see the lumpiness. Here, results from a stratified fold, plotted against sequential image_names.\n\n0-60000 0.936\n60000-120000 0.922\n120000-180000 0.915\n180000-240000 0.921\n240000-300000 0.924\n
> Psi wrote:\n> \n> as far as I can see the public kernel is doing GroupKFold on image name.
\n\nPlenty of very similar objects. I've not been able to exploit this but see ...
\n\nTrain:
\n\n[25303, 25311, 25333, 25339, 25344, 25348, 25359, 25369, 25373, 25374, 25378, 25381, 25385, 25398, 25413, 25414, 25421, 25422, 25422, 25448, 25467, 25471, 25478, 25485, 25493, 25496, 25501, 25514, 25521, 25525, 25528, 25536, 25537, 25540, 25542, 25544, 25552, 25557, 25562, 25565, 25583, 25586, 25589, 25592, 25599, 25608, 25610, 25626, 25626, 25628, 25654, 25655, 25666, 25667, 25669, 25676, 25677, 25690, 25691, 25705, 25708, 25716, 25742, 25749, 25761, 25769, 25771, 25776, 25780, 25782, 25784, 25795, 25804, 25809, 25820, 25824, 25825, 25835, 25839, 25855, 25861, 25865, 25867, 25866, 25875, 25883, 25884, 25898, 25909, 25912, 25920, 25922, 25930, 25939, 25950, 25955, 25960, 25969, 25970, 25971, 25979, 25982, 25984, 25996, 26020, 26026, 26027, 26029, 26032, 26035, 26038, 26041, 26042]
Test:
\n\n[0033, 0544, 0559, 0566, 0680, 0750, 1005, 1265, 1645, 1826, 2086, 2087, 2108, 2337, 2396, 2713, 2795, 2860, 2925, 3044, 3049, 3126, 3245, 3594, 3599, 3827, 3852, 3915, 4179, 4183, 4269, 4339, 4559]
@philippsinger \n\"are you sure the GroupKFold is replicating it exactly as he did not set a seed when shuffling?\"
\n\ni downloaded and used his split csv file
", "votes": 1, "replies": [] }, { "id": 924113, "author_name": "Psi", "author_url": "", "post_date": "2020-07-11T09:05:06.173000", "content": "@hengck23 Oh I see he has it in his dataset.
", "votes": 0, "replies": [] } ] }, { "id": 918550, "author_name": "David", "author_url": "", "post_date": "2020-07-07T10:06:37.343000", "content": "Try EfficientNet B7. You will get a baseline score of 0.93+
", "votes": 4, "replies": [ { "id": 918560, "author_name": "Kamal Das", "author_url": "", "post_date": "2020-07-07T10:15:23.013000", "content": "Thank you @davidtong
\n\nappreciate your help and guidance!
", "votes": 0, "replies": [] }, { "id": 920616, "author_name": "Ų¹Ų«Ł Ų§Ł", "author_url": "", "post_date": "2020-07-08T18:00:43.493000", "content": "It's not easy to be 7.
", "votes": 1, "replies": [] }, { "id": 920636, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-07-08T18:14:30.067000", "content": "
a possible way to speedup:\n1. divide into train set T and validation set V\n2. yet divide train set into T1, T2 , T3 ... T4\n3. train on T1 and validate on T2.\n4. sort T2 according to predicted score and uniformly sample, e.g. say 20%. \n5. train with T1 + 20% of T2, and validate on T3.\n6. sort T3 according to predicted score and uniformly sample, e.g. say 20%. \n7. train with T1 + 20% of T2 + 20% T3 and validate on T4.\n...
\n\n.. . finally, train with T1 + 20% of T2 + 20% T3 + 20% T4 and validate on V.\n.... finally, train with T1 + T2 +T3 + T4 and validate on V.
\n\nthe idea is to have a reduced set representative of a larger set. since we do validation very n-th epoch, we use this validation to add to train data for free.
", "votes": 3, "replies": [] }, { "id": 926267, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-07-12T15:54:53.843000", "content": "Is It going to be like progressive pseudolabeling? (Although we would have the real label in this case ) when you say sort and pick 20% this 20% is 10% with pred < 0.001 and 10% with pred > 10% something like that ?
", "votes": 0, "replies": [] }, { "id": 926384, "author_name": "hongy", "author_url": "", "post_date": "2020-07-12T17:15:45.217000", "content": "You could use a TPU. It only takes 24 hours for 35 full passes through 285k images at 512x512 on B7. This is on a v2-8 so you may get faster results on at v3-8.
", "votes": 1, "replies": [] }, { "id": 926536, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-07-12T19:14:15.133000", "content": "Is this with TFRECORDS @hooong ? I tried with your amazing Kernel but was not able to use TFRECORDS in my colab .
", "votes": 0, "replies": [] }, { "id": 927499, "author_name": "hongy", "author_url": "", "post_date": "2020-07-13T12:35:45.997000", "content": "There is a way to use TFRecords in colab. One of my teammates is running colab pro as well.
\n\nI believe you need to host the files on your own GCS bucket for colab
", "votes": 0, "replies": [] }, { "id": 928795, "author_name": "", "author_url": "", "post_date": "2020-07-14T08:13:38.953000", "content": "stop misguiding
", "votes": 0, "replies": [] }, { "id": 935013, "author_name": "hengck23", "author_url": "", "post_date": "2020-07-19T02:40:29.377000", "content": "\"Is It going to be like progressive pseudolabeling? (Although we would have the real label in this case ) when you say sort and pick 20% this 20% is 10% with pred < 0.001 and 10% with pred > 10% something like that ?\"
\n\nit is not pesudo-labeling. it is importance sampling. you can google paper on \"importance sampling + deep learning.\" it is also related to hard mining, curriculum learning. when there are too many samples, we only want to train with less samples.
\n\ni do importance sampling for initial experiments to select a set of representative smaller train samples.
\n\nwhen your model is finalized, you can train with all samples.
", "votes": 1, "replies": [] } ] }, { "id": 918387, "author_name": "Urvish", "author_url": "", "post_date": "2020-07-07T08:10:44.883000", "content": "Hi,
\n\nhonestly, no one is going to tell you how you can beat that benchmark. You have to try and think about what can you do using the same kernel. There are so many things that one can leverage from that kernel.
", "votes": 1, "replies": [ { "id": 918503, "author_name": "Kamal Das", "author_url": "", "post_date": "2020-07-07T09:33:50.807000", "content": "The community has been gracious in helping reach the base level and a score of 0.921\nsomething I know would not have been possible with my own level of skills
\n\nI think some may provide some guidance on what possible approaches may be ...\nsharing helps in learning! fingers crossed!
", "votes": 2, "replies": [] } ] }, { "id": 922890, "author_name": "Kamal Das", "author_url": "", "post_date": "2020-07-10T11:54:38.533000", "content": "Thank you everyone! All your suggestions and guidance have been invaluable
\n\nYour support and suggestions have helped me get into the top 200! I am yet to try out b7, which I will do this weekend; and march a bit further ahead!
\n\nThank you for being such a supportive community!!
", "votes": 1, "replies": [] }, { "id": 926424, "author_name": "sumitttttttt", "author_url": "", "post_date": "2020-07-12T17:53:39.850000", "content": "awsme!!
", "votes": -2, "replies": [] }, { "id": 925517, "author_name": "", "author_url": "", "post_date": "2020-07-12T06:20:52.780000", "content": "cool
", "votes": -2, "replies": [] }, { "id": 934505, "author_name": "shubhamshukla", "author_url": "", "post_date": "2020-07-18T13:54:40.697000", "content": "i am not getting accuracy more than 91.4
", "votes": 0, "replies": [] }, { "id": 922566, "author_name": "", "author_url": "", "post_date": "2020-07-10T07:50:13.567000", "content": "nice
", "votes": 0, "replies": [] }, { "id": 918537, "author_name": "", "author_url": "", "post_date": "2020-07-07T09:54:47.280000", "content": "", "votes": 0, "replies": [ { "id": 926134, "author_name": "", "author_url": "", "post_date": "2020-07-12T14:18:34.467000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "918165": "Thanks to @shonenkov \nnotebook and help with [initial code [link]](https://www.kaggle.com/shonenkov/train-inference-gpu-baseline), 100s of us are at the baseline score of 0.921\n\nlooking for inputs and guidance on what was done to improve and move further!\n\ni tried several parameter changes and most had no impact while some deteriorated the performance. Would be great for newbies to understand what one should focus on!\n\nAny guidance and suggestions on direction to take is very appreciated. Thanks in advance!\nš š ", "920307": "I am a bit perplexed by the much higher LB the best pub kernel got over CV. Is this something others are also observing? For me LB is usually worse than CV.", "918550": "Try EfficientNet B7. You will get a baseline score of 0.93+", "918387": "Hi, \n\nhonestly, no one is going to tell you how you can beat that benchmark. You have to try and think about what can you do using the same kernel. There are so many things that one can leverage from that kernel. \n", "922890": "Thank you everyone! All your suggestions and guidance have been invaluable\n\nYour support and suggestions have helped me get into the top 200! I am yet to try out b7, which I will do this weekend; and march a bit further ahead!\n\nThank you for being such a supportive community!!", "926424": "awsme!!", "925517": "cool", "934505": "i am not getting accuracy more than 91.4", "922566": "nice", "918537": "" } }