It seems that there is no discussion for sharing the best single model yet.
\n\nI will start to share my current best model:\nseresnext101_32x4d with modification on the last classifier, 5 fold CV ~0.595, LB 0.621, image size 299
", "messages": [ { "id": 525403, "postDate": "2019-04-30T22:28:01.243Z", "content": "It seems that there is no discussion for sharing the best single model yet.
\n\nI will start to share my current best model:\nseresnext101_32x4d with modification on the last classifier, 5 fold CV ~0.595, LB 0.621, image size 299
", "rawMarkdown": "It seems that there is no discussion for sharing the best single model yet.\n\nI will start to share my current best model:\nseresnext101_32x4d with modification on the last classifier, 5 fold CV ~0.595, LB 0.621, image size 299", "votes": 29 }, { "id": 537403, "postDate": "2019-05-27T00:21:02.143Z", "content": "LB 0.660 with 6-folds se_resnext101_32x4d. Input size 320x320 and tta 10 times. I probably have to reduce the number of tta from 10 to 4 or 5 for 2nd stage.
", "rawMarkdown": "LB 0.660 with 6-folds se\\_resnext101\\_32x4d. Input size 320x320 and tta 10 times. I probably have to reduce the number of tta from 10 to 4 or 5 for 2nd stage.", "votes": 6, "replies": [ { "id": 537534, "postDate": "2019-05-27T07:38:55.323Z", "content": "What is your best single model CV?
", "rawMarkdown": "What is your best single model CV?" }, { "id": 538336, "postDate": "2019-05-28T12:40:26.840Z", "content": "Hi, does tta help much? for me, tta can only boost 0.001 on LB, which is quite disappointing.
", "rawMarkdown": "Hi, does tta help much? for me, tta can only boost 0.001 on LB, which is quite disappointing." }, { "id": 538737, "postDate": "2019-05-29T03:40:47.627Z", "content": "and why do you reduce tta numbers? since the kernel time limit is 9 hours.
", "rawMarkdown": "and why do you reduce tta numbers? since the kernel time limit is 9 hours." }, { "id": 539335, "postDate": "2019-05-29T23:35:57.057Z", "content": "Because it probably takes 20 hours to run with 10 tta for 2nd stage. Regarding the boost, it depends on how you train and what augmentations you use. In my case, tta is very important as I use random crop.
", "rawMarkdown": "Because it probably takes 20 hours to run with 10 tta for 2nd stage. Regarding the boost, it depends on how you train and what augmentations you use. In my case, tta is very important as I use random crop.", "votes": 1 } ] }, { "id": 527777, "postDate": "2019-05-06T09:12:53.600Z", "content": "My best single model is SE-ResNext101-32x4d (6 folds CV) with public LB score 0.643
", "rawMarkdown": "My best single model is SE-ResNext101-32x4d (6 folds CV) with public LB score 0.643", "votes": 5, "replies": [ { "id": 527898, "postDate": "2019-05-06T15:08:10.120Z", "content": "Wondering the CV and what is your image size?
", "rawMarkdown": "Wondering the CV and what is your image size?" }, { "id": 527949, "postDate": "2019-05-06T17:17:06.190Z", "content": "what is single fold CV score?
", "rawMarkdown": "what is single fold CV score?", "votes": 1 }, { "id": 528098, "postDate": "2019-05-07T04:36:54.747Z", "content": "@allerria validation scores are stable across all folds (0.609 - 0.61). I used a fixed threshold (0.2) for predicting the labels.\n@strideradu variable sized images instead of resizing them into squares. Using this data pre-processing trick boosted my square-sized (224x224) images baseline (same network architecture, same loss, same training procedure) by 7%.
", "rawMarkdown": "@allerria validation scores are stable across all folds (0.609 - 0.61). I used a fixed threshold (0.2) for predicting the labels.\n@strideradu variable sized images instead of resizing them into squares. Using this data pre-processing trick boosted my square-sized (224x224) images baseline (same network architecture, same loss, same training procedure) by 7%.", "votes": 4 }, { "id": 528130, "postDate": "2019-05-07T05:58:07.903Z", "content": "How to make the variable sized images?
", "rawMarkdown": "How to make the variable sized images?", "votes": 1 }, { "id": 528154, "postDate": "2019-05-07T06:52:33.757Z", "content": "@timmmmmms : Are you working on pytorch? If yes, you may use custom collate function. Pls refer on this one: https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3
", "rawMarkdown": "@timmmmmms : Are you working on pytorch? If yes, you may use custom collate function. Pls refer on this one: https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3", "votes": 5 }, { "id": 528334, "postDate": "2019-05-07T14:10:37.933Z", "content": "Thanks for your trick, when variable sized images, how do you determine the size of an image (or ratio compared to the raw image)? Do you use the same number of pixels for each images or something else?
", "rawMarkdown": "Thanks for your trick, when variable sized images, how do you determine the size of an image (or ratio compared to the raw image)? Do you use the same number of pixels for each images or something else?", "votes": 1 }, { "id": 528511, "postDate": "2019-05-08T01:52:15.300Z", "content": "if you do this pre-processing , is there a way to use bn
", "rawMarkdown": "if you do this pre-processing , is there a way to use bn" }, { "id": 528567, "postDate": "2019-05-08T05:09:41.907Z", "content": "When I use se_resnet, the training speed is very slow. How to solve it?
", "rawMarkdown": "When I use se_resnet, the training speed is very slow. How to solve it?" }, { "id": 528996, "postDate": "2019-05-09T02:16:21.577Z", "content": "@andy2709 @projdev \nThanks for your valuable information, but I still could not understand what is the variable input... Even if the image sizes are variable, the number and size of parameters are same, right? If so, how can we deal with variable sized input in the model? And what is the difference compared to zero padding or resize??
", "rawMarkdown": "@andy2709 @projdev \nThanks for your valuable information, but I still could not understand what is the variable input... Even if the image sizes are variable, the number and size of parameters are same, right? If so, how can we deal with variable sized input in the model? And what is the difference compared to zero padding or resize??", "votes": 2 } ] }, { "id": 525644, "postDate": "2019-05-01T12:56:05.387Z", "content": "5fold CV: 0.601, LB: 0.633
", "rawMarkdown": "5fold CV: 0.601, LB: 0.633", "votes": 3, "replies": [ { "id": 525750, "postDate": "2019-05-01T17:07:56.897Z", "content": "May I ask what's your model?
", "rawMarkdown": "May I ask what's your model?" }, { "id": 525756, "postDate": "2019-05-01T17:13:53.523Z", "content": "PNASNet-5
", "rawMarkdown": "PNASNet-5", "votes": 2 }, { "id": 525916, "postDate": "2019-05-02T02:24:21.180Z", "content": "\n\n\n5fold CV: 0.601, LB: 0.635
\n
Your Local CV is much smaller than LB, do you think it is overfitted?
", "rawMarkdown": "> 5fold CV: 0.601, LB: 0.635\n\nYour Local CV is much smaller than LB, do you think it is overfitted?", "votes": -1 }, { "id": 525928, "postDate": "2019-05-02T03:01:05.180Z", "content": "I think if his CV is much higher than LB, that may be because overfitting
", "rawMarkdown": "I think if his CV is much higher than LB, that may be because overfitting", "votes": 1 }, { "id": 525950, "postDate": "2019-05-02T04:15:20.427Z", "content": "IMO, 5 fold CV is the out of fold validation result which consists of 5 parts and each part is the output probability of one single model predict on its validation set. then you calculate one threshold and get prediction. But LB is the average of 5 models, each single model in each fold has its own output probability of the test set, then you take average. so LB is the average ensemble of 5 models, it should be better than CV especially since some samples that only occur once in some folds but never occur in other folds, the single model may miss something.
", "rawMarkdown": "IMO, 5 fold CV is the out of fold validation result which consists of 5 parts and each part is the output probability of one single model predict on its validation set. then you calculate one threshold and get prediction. But LB is the average of 5 models, each single model in each fold has its own output probability of the test set, then you take average. so LB is the average ensemble of 5 models, it should be better than CV especially since some samples that only occur once in some folds but never occur in other folds, the single model may miss something.", "votes": 1 }, { "id": 526287, "postDate": "2019-05-02T18:13:19.913Z", "content": "A lower CV score is expected, since on each fold one typically uses only 4/5 of data, while the final test prediction is made using all data (typically by averaging models from different folds). Another reason for such a gap is TTA.
", "rawMarkdown": "A lower CV score is expected, since on each fold one typically uses only 4/5 of data, while the final test prediction is made using all data (typically by averaging models from different folds). Another reason for such a gap is TTA.", "votes": 6 }, { "id": 526469, "postDate": "2019-05-03T06:05:37.640Z", "content": "NAS need a lot of time to train, I think you are rich in graph cards 👍
", "rawMarkdown": "NAS need a lot of time to train, I think you are rich in graph cards 👍 " }, { "id": 527562, "postDate": "2019-05-05T20:26:08.833Z", "content": "Would you mind to share the image size you are using for your best model?
", "rawMarkdown": "Would you mind to share the image size you are using for your best model?" }, { "id": 527581, "postDate": "2019-05-05T21:24:36.830Z", "content": "Now, my best model is SeResNeXt101, and image size is 320 x 320.
\nbelow is my experience.
\n1. In this competition, difference in accuracy between model is small (e.g. inceptionv3 is a little weaker than SeResNeXt101 (-0.002) ). Train method is very important.
\n2. Very long images exist in dataset, but my model can classified it well. I have no plans to deal with it.
\n3. About overfitting, I don't understand it yet. Now I make effort on creating validation dataset.\n4. I used local machine to train PNASNet-5, but I used only kernel to train all models, except it. use kernels efficiently.
Do you feed your network just images or, you add some kind of image features?
", "rawMarkdown": "Do you feed your network just images or, you add some kind of image features?" }, { "id": 527592, "postDate": "2019-05-05T22:25:27.410Z", "content": "only image.
", "rawMarkdown": "only image.", "votes": 1 }, { "id": 527596, "postDate": "2019-05-05T22:37:00.060Z", "content": "As you said, it's the Train method that makes the difference.
", "rawMarkdown": "As you said, it's the Train method that makes the difference." }, { "id": 527645, "postDate": "2019-05-06T02:23:51.843Z", "content": "@phalanx, you are correct! proper loss function and model training strategies are very important in this type of competition
", "rawMarkdown": "@phalanx, you are correct! proper loss function and model training strategies are very important in this type of competition" }, { "id": 528357, "postDate": "2019-05-07T15:24:24.920Z", "content": "Are your pretrained models based on pytorch?
", "rawMarkdown": "Are your pretrained models based on pytorch? " }, { "id": 528361, "postDate": "2019-05-07T15:29:38.183Z", "content": "I think there's a very little difference between pre-trained models from pytorch, keras or tensorflow. The important factor here is training methods (choosing proper loss function, optimizer, learning scheduler, how large model is. etc..)
", "rawMarkdown": "I think there's a very little difference between pre-trained models from pytorch, keras or tensorflow. The important factor here is training methods (choosing proper loss function, optimizer, learning scheduler, how large model is. etc..)" }, { "id": 528382, "postDate": "2019-05-07T16:15:45.893Z", "content": "You are correct. When I use some pre-trained models based on Pytorch that was implemented by others(such as SeResNext), I need to install some third-party libraries. And this also requires importing these libraries when submitting the kernel. However, it does not seem to be allowed. So I wonder how to deal with this situation.
", "rawMarkdown": "You are correct. When I use some pre-trained models based on Pytorch that was implemented by others(such as SeResNext), I need to install some third-party libraries. And this also requires importing these libraries when submitting the kernel. However, it does not seem to be allowed. So I wonder how to deal with this situation." } ] }, { "id": 532300, "postDate": "2019-05-16T15:54:06.493Z", "content": "By adjusting the augmentation, my CV goes up to 6075 and LB 640, but no idea how to break to 650 level
", "rawMarkdown": "By adjusting the augmentation, my CV goes up to 6075 and LB 640, but no idea how to break to 650 level", "votes": 1, "replies": [ { "id": 532552, "postDate": "2019-05-17T08:27:33.827Z", "content": "Are you using some special augmentations or only standard augmentations?
", "rawMarkdown": "Are you using some special augmentations or only standard augmentations?" }, { "id": 532612, "postDate": "2019-05-17T11:20:45.380Z", "content": "Are you using seresnext101_32x4d with fastai?
", "rawMarkdown": "Are you using seresnext101_32x4d with fastai?\n" }, { "id": 532809, "postDate": "2019-05-17T18:30:48.093Z", "content": "Just normal augmentation in commonly used library
", "rawMarkdown": "Just normal augmentation in commonly used library" }, { "id": 532810, "postDate": "2019-05-17T18:31:04.023Z", "content": "Just pytorch, but it should be same with fastai
", "rawMarkdown": "Just pytorch, but it should be same with fastai" }, { "id": 533167, "postDate": "2019-05-18T15:25:19.857Z", "content": "Are you using the AutoAugment method to adjust the augmentation?
", "rawMarkdown": "Are you using the AutoAugment method to adjust the augmentation?" }, { "id": 533248, "postDate": "2019-05-18T19:17:44.457Z", "content": "No, I didn't try that, have you tried this?
", "rawMarkdown": "No, I didn't try that, have you tried this?" }, { "id": 533350, "postDate": "2019-05-19T03:35:12Z", "content": "The handcrafted augment parameters may not be proper, so I want to try it.
", "rawMarkdown": "The handcrafted augment parameters may not be proper, so I want to try it. " }, { "id": 534259, "postDate": "2019-05-21T02:00:18.890Z", "content": "If you have some results pease share it~
", "rawMarkdown": "If you have some results pease share it~" } ] }, { "id": 529826, "postDate": "2019-05-10T21:23:47.820Z", "content": "I also add the image size I was using for my best model, right now my best result is 384*384 seresnext101_32x4d, CV 0.597, LB 0.630.
\n\nAnd I also tried the PNASanet but the result is not improved, not sure if I miss something
", "rawMarkdown": "I also add the image size I was using for my best model, right now my best result is 384*384 seresnext101_32x4d, CV 0.597, LB 0.630.\n\nAnd I also tried the PNASanet but the result is not improved, not sure if I miss something", "votes": 1, "replies": [ { "id": 529875, "postDate": "2019-05-11T03:37:41.863Z", "content": "Hi, Strideradu. LB 0.630 is obetained by your single model?
", "rawMarkdown": "Hi, Strideradu. LB 0.630 is obetained by your single model?" }, { "id": 530176, "postDate": "2019-05-12T02:12:07.650Z", "content": "5 fold of CV 0.597
", "rawMarkdown": "5 fold of CV 0.597", "votes": 1 }, { "id": 534363, "postDate": "2019-05-21T06:05:47.260Z", "content": "May I ask whether you train the whole model or just some block of the model?
", "rawMarkdown": "May I ask whether you train the whole model or just some block of the model?" }, { "id": 534765, "postDate": "2019-05-21T19:40:54.803Z", "content": "I finetune the whole model, I remembered at the very beginning I try to fine tune the last layers first, but not working. But I may retry it
", "rawMarkdown": "I finetune the whole model, I remembered at the very beginning I try to fine tune the last layers first, but not working. But I may retry it", "votes": 2 }, { "id": 535070, "postDate": "2019-05-22T08:37:14.257Z", "content": "Thank you for your sharing!
", "rawMarkdown": "Thank you for your sharing!" } ] }, { "id": 525667, "postDate": "2019-05-01T13:50:58.683Z", "content": "How can we do 5fold CV with limited running time? The single ResNet50 model has cost me 9 hours for 11 epochs to reach 0.60. Any suggestions will be appreciated.
", "rawMarkdown": "How can we do 5fold CV with limited running time? The single ResNet50 model has cost me 9 hours for 11 epochs to reach 0.60. Any suggestions will be appreciated.", "votes": 2, "replies": [ { "id": 525678, "postDate": "2019-05-01T14:12:58.110Z", "content": "You can train one fold in one kernel and run 7 kernels at same time. Run 8 epochs and save model to output, download them and upload to dataset, then create another 7 kernels to load your models and start from epoch 8 to epoch 11.
", "rawMarkdown": "You can train one fold in one kernel and run 7 kernels at same time. Run 8 epochs and save model to output, download them and upload to dataset, then create another 7 kernels to load your models and start from epoch 8 to epoch 11.", "votes": 3 }, { "id": 525686, "postDate": "2019-05-01T14:36:29.647Z", "content": "But it will warn that the kernel cannot use kernel outputs as a data source for this competition. Are there any ways we can use the pre-trained weight from other kernels?
", "rawMarkdown": "But it will warn that the kernel cannot use kernel outputs as a data source for this competition. Are there any ways we can use the pre-trained weight from other kernels?", "votes": 1 }, { "id": 525710, "postDate": "2019-05-01T15:25:50.077Z", "content": "upload to custom dataset after downloading weights from kernel outputs and import it
", "rawMarkdown": "upload to custom dataset after downloading weights from kernel outputs and import it", "votes": 1 }, { "id": 525797, "postDate": "2019-05-01T18:20:46.717Z", "content": "Thanks a lot!!!
", "rawMarkdown": "Thanks a lot!!!" } ] }, { "id": 525629, "postDate": "2019-05-01T12:23:57.590Z", "content": "densenet201, single fold cv 0.591 lb 0.603, 5 fold lb 0.622
", "rawMarkdown": "densenet201, single fold cv 0.591 lb 0.603, 5 fold lb 0.622", "votes": 2, "replies": [ { "id": 530276, "postDate": "2019-05-12T11:17:51.373Z", "content": "May I ask, what input size do you use? And how long do you train the model? Thank you.
", "rawMarkdown": "May I ask, what input size do you use? And how long do you train the model? Thank you." }, { "id": 530497, "postDate": "2019-05-13T02:52:57.403Z", "content": "288x288, 9 hours in kaggle kernel.
", "rawMarkdown": "288x288, 9 hours in kaggle kernel." }, { "id": 531504, "postDate": "2019-05-15T03:09:54.100Z", "content": "I found that when the input size is 224x224, the batch size will be under 64. I think the small batch size is not good for training. I want to know how to make batch size large. May I ask are there any trick to do that
", "rawMarkdown": "I found that when the input size is 224x224, the batch size will be under 64. I think the small batch size is not good for training. I want to know how to make batch size large. May I ask are there any trick to do that" }, { "id": 531508, "postDate": "2019-05-15T03:26:11.500Z", "content": "There is Image-size Vs Batch size if you increase one you have to decrease the other and the other way.\nThat's because of the limited GPU Memory capacity.
", "rawMarkdown": "There is Image-size Vs Batch size if you increase one you have to decrease the other and the other way.\nThat's because of the limited GPU Memory capacity." } ] }, { "id": 525521, "postDate": "2019-05-01T07:41:58.603Z", "content": "my current best model is single fold resnet50 CV = 0.599, LB = 0.605
", "rawMarkdown": "my current best model is single fold resnet50 CV = 0.599, LB = 0.605", "votes": 2, "replies": [ { "id": 525544, "postDate": "2019-05-01T08:50:17.573Z", "content": "How many epochs did it take for your resnet50 to give a CV of 0.599 ? with one cycle LR
", "rawMarkdown": "How many epochs did it take for your resnet50 to give a CV of 0.599 ? with one cycle LR" }, { "id": 525548, "postDate": "2019-05-01T09:04:09.553Z", "content": "Yes, it's one cycle LR with 20+ epochs, it depends on how you can maximize the best combination between image size, batch size & learning rate. You may check the discussion forum for Bag of Tricks for Image Classification.
", "rawMarkdown": "Yes, it's one cycle LR with 20+ epochs, it depends on how you can maximize the best combination between image size, batch size & learning rate. You may check the discussion forum for Bag of Tricks for Image Classification.", "votes": 3 }, { "id": 525835, "postDate": "2019-05-01T19:51:56.427Z", "content": "@projdev May I ask, what data augmentations did you use?
", "rawMarkdown": "@projdev May I ask, what data augmentations did you use?", "votes": 1 }, { "id": 525977, "postDate": "2019-05-02T05:15:29.673Z", "content": "I used augmentations such as horizontal flip, random rotation, etc. My reference can be found in current public kernels.
", "rawMarkdown": "I used augmentations such as horizontal flip, random rotation, etc. My reference can be found in current public kernels." } ] }, { "id": 534968, "postDate": "2019-05-22T05:58:06.603Z", "content": "I have fine-tuned my best model from 0.597 to 0.618( LB), but ensemble result did not improve at all. so frustrating! so weird！
", "rawMarkdown": "I have fine-tuned my best model from 0.597 to 0.618( LB), but ensemble result did not improve at all. so frustrating! so weird！", "replies": [ { "id": 535208, "postDate": "2019-05-22T13:44:42.160Z", "content": "In this competition, one of the main problem is noisy annotations. Many images lack annotation, so we have to deal with it.
\nGoogle AI Open Images have same problem(task is detection, not recognition), So you may understand something if you refer to it.
May I ask, have you used some methods in your solution to solve this problem?
", "rawMarkdown": "May I ask, have you used some methods in your solution to solve this problem?" }, { "id": 535287, "postDate": "2019-05-22T16:18:05.930Z", "content": "Yes, a little. But I couldn't get quite improvement.
", "rawMarkdown": "Yes, a little. But I couldn't get quite improvement." }, { "id": 535369, "postDate": "2019-05-22T19:27:48.447Z", "content": "have you use 5 fold?
", "rawMarkdown": "have you use 5 fold?" }, { "id": 535416, "postDate": "2019-05-22T21:25:46.047Z", "content": "Yes
", "rawMarkdown": "Yes" }, { "id": 536430, "postDate": "2019-05-24T13:06:34.947Z", "content": "Sadly, I have the same problem as you now. May I ask, did you find any methods to solve this?
", "rawMarkdown": "Sadly, I have the same problem as you now. May I ask, did you find any methods to solve this?" }, { "id": 1220683, "postDate": "2021-02-28T08:49:12.037Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 530606, "postDate": "2019-05-13T09:19:57.300Z", "content": "Are you using a pretrained model or training model from scratch?
", "rawMarkdown": "Are you using a pretrained model or training model from scratch?", "replies": [ { "id": 530727, "postDate": "2019-05-13T14:32:17.513Z", "content": "I think most people use pretrained model
", "rawMarkdown": "I think most people use pretrained model", "votes": 2 } ] }, { "id": 529916, "postDate": "2019-05-11T06:51:32.937Z", "content": "Hi, I used the same model (seresnext101_32x4d ). But I used a package named fnn_finetune. With this package, I can't submit my csv files. I want to know that how did you build the model?
", "rawMarkdown": "Hi, I used the same model (seresnext101_32x4d ). But I used a package named fnn_finetune. With this package, I can't submit my csv files. I want to know that how did you build the model? ", "replies": [ { "id": 531513, "postDate": "2019-05-15T03:41:06.253Z", "content": "We cannot use Internet access for this competition, and we can add pretrainned model by Add Dataset button, someone had post pretrained models on Kaggle Dataset.
LB 0.660 with 6-folds se_resnext101_32x4d. Input size 320x320 and tta 10 times. I probably have to reduce the number of tta from 10 to 4 or 5 for 2nd stage.
", "votes": 6, "replies": [ { "id": 537534, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-27T07:38:55.323000", "content": "What is your best single model CV?
", "votes": 0, "replies": [] }, { "id": 538336, "author_name": "good good study", "author_url": "", "post_date": "2019-05-28T12:40:26.840000", "content": "Hi, does tta help much? for me, tta can only boost 0.001 on LB, which is quite disappointing.
", "votes": 0, "replies": [] }, { "id": 538737, "author_name": "good good study", "author_url": "", "post_date": "2019-05-29T03:40:47.627000", "content": "and why do you reduce tta numbers? since the kernel time limit is 9 hours.
", "votes": 0, "replies": [] }, { "id": 539335, "author_name": "Appian", "author_url": "", "post_date": "2019-05-29T23:35:57.057000", "content": "Because it probably takes 20 hours to run with 10 tta for 2nd stage. Regarding the boost, it depends on how you train and what augmentations you use. In my case, tta is very important as I use random crop.
", "votes": 1, "replies": [] } ] }, { "id": 527777, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2019-05-06T09:12:53.600000", "content": "My best single model is SE-ResNext101-32x4d (6 folds CV) with public LB score 0.643
", "votes": 5, "replies": [ { "id": 527898, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-06T15:08:10.120000", "content": "Wondering the CV and what is your image size?
", "votes": 0, "replies": [] }, { "id": 527949, "author_name": "Georgy Petrov", "author_url": "", "post_date": "2019-05-06T17:17:06.190000", "content": "what is single fold CV score?
", "votes": 1, "replies": [] }, { "id": 528098, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2019-05-07T04:36:54.747000", "content": "@allerria validation scores are stable across all folds (0.609 - 0.61). I used a fixed threshold (0.2) for predicting the labels.\n@strideradu variable sized images instead of resizing them into squares. Using this data pre-processing trick boosted my square-sized (224x224) images baseline (same network architecture, same loss, same training procedure) by 7%.
", "votes": 4, "replies": [] }, { "id": 528130, "author_name": "Timmmmmms", "author_url": "", "post_date": "2019-05-07T05:58:07.903000", "content": "How to make the variable sized images?
", "votes": 1, "replies": [] }, { "id": 528154, "author_name": "FGPC", "author_url": "", "post_date": "2019-05-07T06:52:33.757000", "content": "@timmmmmms : Are you working on pytorch? If yes, you may use custom collate function. Pls refer on this one: https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3
", "votes": 5, "replies": [] }, { "id": 528334, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-07T14:10:37.933000", "content": "Thanks for your trick, when variable sized images, how do you determine the size of an image (or ratio compared to the raw image)? Do you use the same number of pixels for each images or something else?
", "votes": 1, "replies": [] }, { "id": 528511, "author_name": "valencebond", "author_url": "", "post_date": "2019-05-08T01:52:15.300000", "content": "if you do this pre-processing , is there a way to use bn
", "votes": 0, "replies": [] }, { "id": 528567, "author_name": "GUTAR", "author_url": "", "post_date": "2019-05-08T05:09:41.907000", "content": "When I use se_resnet, the training speed is very slow. How to solve it?
", "votes": 0, "replies": [] }, { "id": 528996, "author_name": "Camaro", "author_url": "", "post_date": "2019-05-09T02:16:21.577000", "content": "@andy2709 @projdev \nThanks for your valuable information, but I still could not understand what is the variable input... Even if the image sizes are variable, the number and size of parameters are same, right? If so, how can we deal with variable sized input in the model? And what is the difference compared to zero padding or resize??
", "votes": 2, "replies": [] } ] }, { "id": 525644, "author_name": "phalanx", "author_url": "", "post_date": "2019-05-01T12:56:05.387000", "content": "5fold CV: 0.601, LB: 0.633
", "votes": 3, "replies": [ { "id": 525750, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-01T17:07:56.897000", "content": "May I ask what's your model?
", "votes": 0, "replies": [] }, { "id": 525756, "author_name": "phalanx", "author_url": "", "post_date": "2019-05-01T17:13:53.523000", "content": "PNASNet-5
", "votes": 2, "replies": [] }, { "id": 525916, "author_name": "Dilapsky Lee", "author_url": "", "post_date": "2019-05-02T02:24:21.180000", "content": "\n\n\n5fold CV: 0.601, LB: 0.635
\n
Your Local CV is much smaller than LB, do you think it is overfitted?
", "votes": -1, "replies": [] }, { "id": 525928, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-02T03:01:05.180000", "content": "I think if his CV is much higher than LB, that may be because overfitting
", "votes": 1, "replies": [] }, { "id": 525950, "author_name": "jionie", "author_url": "", "post_date": "2019-05-02T04:15:20.427000", "content": "IMO, 5 fold CV is the out of fold validation result which consists of 5 parts and each part is the output probability of one single model predict on its validation set. then you calculate one threshold and get prediction. But LB is the average of 5 models, each single model in each fold has its own output probability of the test set, then you take average. so LB is the average ensemble of 5 models, it should be better than CV especially since some samples that only occur once in some folds but never occur in other folds, the single model may miss something.
", "votes": 1, "replies": [] }, { "id": 526287, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-05-02T18:13:19.913000", "content": "A lower CV score is expected, since on each fold one typically uses only 4/5 of data, while the final test prediction is made using all data (typically by averaging models from different folds). Another reason for such a gap is TTA.
", "votes": 6, "replies": [] }, { "id": 526469, "author_name": "", "author_url": "", "post_date": "2019-05-03T06:05:37.640000", "content": "NAS need a lot of time to train, I think you are rich in graph cards 👍
", "votes": 0, "replies": [] }, { "id": 527562, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-05T20:26:08.833000", "content": "Would you mind to share the image size you are using for your best model?
", "votes": 0, "replies": [] }, { "id": 527581, "author_name": "phalanx", "author_url": "", "post_date": "2019-05-05T21:24:36.830000", "content": "Now, my best model is SeResNeXt101, and image size is 320 x 320.
\nbelow is my experience.
\n1. In this competition, difference in accuracy between model is small (e.g. inceptionv3 is a little weaker than SeResNeXt101 (-0.002) ). Train method is very important.
\n2. Very long images exist in dataset, but my model can classified it well. I have no plans to deal with it.
\n3. About overfitting, I don't understand it yet. Now I make effort on creating validation dataset.\n4. I used local machine to train PNASNet-5, but I used only kernel to train all models, except it. use kernels efficiently.
Do you feed your network just images or, you add some kind of image features?
", "votes": 0, "replies": [] }, { "id": 527592, "author_name": "phalanx", "author_url": "", "post_date": "2019-05-05T22:25:27.410000", "content": "only image.
", "votes": 1, "replies": [] }, { "id": 527596, "author_name": "JM100", "author_url": "", "post_date": "2019-05-05T22:37:00.060000", "content": "As you said, it's the Train method that makes the difference.
", "votes": 0, "replies": [] }, { "id": 527645, "author_name": "FGPC", "author_url": "", "post_date": "2019-05-06T02:23:51.843000", "content": "@phalanx, you are correct! proper loss function and model training strategies are very important in this type of competition
", "votes": 0, "replies": [] }, { "id": 528357, "author_name": "beyondych11", "author_url": "", "post_date": "2019-05-07T15:24:24.920000", "content": "Are your pretrained models based on pytorch?
", "votes": 0, "replies": [] }, { "id": 528361, "author_name": "FGPC", "author_url": "", "post_date": "2019-05-07T15:29:38.183000", "content": "I think there's a very little difference between pre-trained models from pytorch, keras or tensorflow. The important factor here is training methods (choosing proper loss function, optimizer, learning scheduler, how large model is. etc..)
", "votes": 0, "replies": [] }, { "id": 528382, "author_name": "beyondych11", "author_url": "", "post_date": "2019-05-07T16:15:45.893000", "content": "You are correct. When I use some pre-trained models based on Pytorch that was implemented by others(such as SeResNext), I need to install some third-party libraries. And this also requires importing these libraries when submitting the kernel. However, it does not seem to be allowed. So I wonder how to deal with this situation.
", "votes": 0, "replies": [] } ] }, { "id": 532300, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-16T15:54:06.493000", "content": "By adjusting the augmentation, my CV goes up to 6075 and LB 640, but no idea how to break to 650 level
", "votes": 1, "replies": [ { "id": 532552, "author_name": "QunYang", "author_url": "", "post_date": "2019-05-17T08:27:33.827000", "content": "Are you using some special augmentations or only standard augmentations?
", "votes": 0, "replies": [] }, { "id": 532612, "author_name": "Timmmmmms", "author_url": "", "post_date": "2019-05-17T11:20:45.380000", "content": "Are you using seresnext101_32x4d with fastai?
", "votes": 0, "replies": [] }, { "id": 532809, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-17T18:30:48.093000", "content": "Just normal augmentation in commonly used library
", "votes": 0, "replies": [] }, { "id": 532810, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-17T18:31:04.023000", "content": "Just pytorch, but it should be same with fastai
", "votes": 0, "replies": [] }, { "id": 533167, "author_name": "yzwu", "author_url": "", "post_date": "2019-05-18T15:25:19.857000", "content": "Are you using the AutoAugment method to adjust the augmentation?
", "votes": 0, "replies": [] }, { "id": 533248, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-18T19:17:44.457000", "content": "No, I didn't try that, have you tried this?
", "votes": 0, "replies": [] }, { "id": 533350, "author_name": "yzwu", "author_url": "", "post_date": "2019-05-19T03:35:12", "content": "The handcrafted augment parameters may not be proper, so I want to try it.
", "votes": 0, "replies": [] }, { "id": 534259, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-21T02:00:18.890000", "content": "If you have some results pease share it~
", "votes": 0, "replies": [] } ] }, { "id": 529826, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-10T21:23:47.820000", "content": "I also add the image size I was using for my best model, right now my best result is 384*384 seresnext101_32x4d, CV 0.597, LB 0.630.
\n\nAnd I also tried the PNASanet but the result is not improved, not sure if I miss something
", "votes": 1, "replies": [ { "id": 529875, "author_name": "Zheng Li", "author_url": "", "post_date": "2019-05-11T03:37:41.863000", "content": "Hi, Strideradu. LB 0.630 is obetained by your single model?
", "votes": 0, "replies": [] }, { "id": 530176, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-12T02:12:07.650000", "content": "5 fold of CV 0.597
", "votes": 1, "replies": [] }, { "id": 534363, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-21T06:05:47.260000", "content": "May I ask whether you train the whole model or just some block of the model?
", "votes": 0, "replies": [] }, { "id": 534765, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-21T19:40:54.803000", "content": "I finetune the whole model, I remembered at the very beginning I try to fine tune the last layers first, but not working. But I may retry it
", "votes": 2, "replies": [] }, { "id": 535070, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-22T08:37:14.257000", "content": "Thank you for your sharing!
", "votes": 0, "replies": [] } ] }, { "id": 525667, "author_name": "Aaron Fun", "author_url": "", "post_date": "2019-05-01T13:50:58.683000", "content": "How can we do 5fold CV with limited running time? The single ResNet50 model has cost me 9 hours for 11 epochs to reach 0.60. Any suggestions will be appreciated.
", "votes": 2, "replies": [ { "id": 525678, "author_name": "jionie", "author_url": "", "post_date": "2019-05-01T14:12:58.110000", "content": "You can train one fold in one kernel and run 7 kernels at same time. Run 8 epochs and save model to output, download them and upload to dataset, then create another 7 kernels to load your models and start from epoch 8 to epoch 11.
", "votes": 3, "replies": [] }, { "id": 525686, "author_name": "Aaron Fun", "author_url": "", "post_date": "2019-05-01T14:36:29.647000", "content": "But it will warn that the kernel cannot use kernel outputs as a data source for this competition. Are there any ways we can use the pre-trained weight from other kernels?
", "votes": 1, "replies": [] }, { "id": 525710, "author_name": "Georgy Petrov", "author_url": "", "post_date": "2019-05-01T15:25:50.077000", "content": "upload to custom dataset after downloading weights from kernel outputs and import it
", "votes": 1, "replies": [] }, { "id": 525797, "author_name": "Aaron Fun", "author_url": "", "post_date": "2019-05-01T18:20:46.717000", "content": "Thanks a lot!!!
", "votes": 0, "replies": [] } ] }, { "id": 525629, "author_name": "Georgy Petrov", "author_url": "", "post_date": "2019-05-01T12:23:57.590000", "content": "densenet201, single fold cv 0.591 lb 0.603, 5 fold lb 0.622
", "votes": 2, "replies": [ { "id": 530276, "author_name": "邓雍杰Jay", "author_url": "", "post_date": "2019-05-12T11:17:51.373000", "content": "May I ask, what input size do you use? And how long do you train the model? Thank you.
", "votes": 0, "replies": [] }, { "id": 530497, "author_name": "Georgy Petrov", "author_url": "", "post_date": "2019-05-13T02:52:57.403000", "content": "288x288, 9 hours in kaggle kernel.
", "votes": 0, "replies": [] }, { "id": 531504, "author_name": "邓雍杰Jay", "author_url": "", "post_date": "2019-05-15T03:09:54.100000", "content": "I found that when the input size is 224x224, the batch size will be under 64. I think the small batch size is not good for training. I want to know how to make batch size large. May I ask are there any trick to do that
", "votes": 0, "replies": [] }, { "id": 531508, "author_name": "JM100", "author_url": "", "post_date": "2019-05-15T03:26:11.500000", "content": "There is Image-size Vs Batch size if you increase one you have to decrease the other and the other way.\nThat's because of the limited GPU Memory capacity.
", "votes": 0, "replies": [] } ] }, { "id": 525521, "author_name": "FGPC", "author_url": "", "post_date": "2019-05-01T07:41:58.603000", "content": "my current best model is single fold resnet50 CV = 0.599, LB = 0.605
", "votes": 2, "replies": [ { "id": 525544, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2019-05-01T08:50:17.573000", "content": "How many epochs did it take for your resnet50 to give a CV of 0.599 ? with one cycle LR
", "votes": 0, "replies": [] }, { "id": 525548, "author_name": "FGPC", "author_url": "", "post_date": "2019-05-01T09:04:09.553000", "content": "Yes, it's one cycle LR with 20+ epochs, it depends on how you can maximize the best combination between image size, batch size & learning rate. You may check the discussion forum for Bag of Tricks for Image Classification.
", "votes": 3, "replies": [] }, { "id": 525835, "author_name": "", "author_url": "", "post_date": "2019-05-01T19:51:56.427000", "content": "@projdev May I ask, what data augmentations did you use?
", "votes": 1, "replies": [] }, { "id": 525977, "author_name": "FGPC", "author_url": "", "post_date": "2019-05-02T05:15:29.673000", "content": "I used augmentations such as horizontal flip, random rotation, etc. My reference can be found in current public kernels.
", "votes": 0, "replies": [] } ] }, { "id": 534968, "author_name": "Andy Yu", "author_url": "", "post_date": "2019-05-22T05:58:06.603000", "content": "I have fine-tuned my best model from 0.597 to 0.618( LB), but ensemble result did not improve at all. so frustrating! so weird！
", "votes": 0, "replies": [ { "id": 535208, "author_name": "phalanx", "author_url": "", "post_date": "2019-05-22T13:44:42.160000", "content": "In this competition, one of the main problem is noisy annotations. Many images lack annotation, so we have to deal with it.
\nGoogle AI Open Images have same problem(task is detection, not recognition), So you may understand something if you refer to it.
May I ask, have you used some methods in your solution to solve this problem?
", "votes": 0, "replies": [] }, { "id": 535287, "author_name": "phalanx", "author_url": "", "post_date": "2019-05-22T16:18:05.930000", "content": "Yes, a little. But I couldn't get quite improvement.
", "votes": 0, "replies": [] }, { "id": 535369, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-22T19:27:48.447000", "content": "have you use 5 fold?
", "votes": 0, "replies": [] }, { "id": 535416, "author_name": "phalanx", "author_url": "", "post_date": "2019-05-22T21:25:46.047000", "content": "Yes
", "votes": 0, "replies": [] }, { "id": 536430, "author_name": "QunYang", "author_url": "", "post_date": "2019-05-24T13:06:34.947000", "content": "Sadly, I have the same problem as you now. May I ask, did you find any methods to solve this?
", "votes": 0, "replies": [] }, { "id": 1220683, "author_name": "", "author_url": "", "post_date": "2021-02-28T08:49:12.037000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 530606, "author_name": "云不懂风吹", "author_url": "", "post_date": "2019-05-13T09:19:57.300000", "content": "Are you using a pretrained model or training model from scratch?
", "votes": 0, "replies": [ { "id": 530727, "author_name": "Strideradu", "author_url": "", "post_date": "2019-05-13T14:32:17.513000", "content": "I think most people use pretrained model
", "votes": 2, "replies": [] } ] }, { "id": 529916, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-11T06:51:32.937000", "content": "Hi, I used the same model (seresnext101_32x4d ). But I used a package named fnn_finetune. With this package, I can't submit my csv files. I want to know that how did you build the model?
", "votes": 0, "replies": [ { "id": 531513, "author_name": "云不懂风吹", "author_url": "", "post_date": "2019-05-15T03:41:06.253000", "content": "We cannot use Internet access for this competition, and we can add pretrainned model by Add Dataset button, someone had post pretrained models on Kaggle Dataset.