![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F83f09d3a155652dc55cb68a0a46d8e53%2Ftop.jpg?generation=1592623472281414&alt=media\")
Most Imagenet pretrained CNNs were trained on 224x224 image resolution. It is a common misconception, that when using these pretrained CNN, images need to be resized to 224x224. On the contrary, popular CNN are fully convolutional nets that can accept any input size.
\n\nYou can input any image size and these CNN output feature maps that are 32x times smaller. For example, if you input 224x224 then the CNN outputs feature maps of size 7x7. If you input images of size 512x512, then these CNN outputs feature maps of size 16x16.
\n\nThe only relevance of the pretraining 224x224 size is that these CNN have learned to find certain patterns of certain sizes. For example, maybe they learned to find circles that are 50 pixels diameter, or maybe they learned to find triangles with side length 30 pixels.
\n\nIn the example below, we pretrain CNN on images of size 224x224 and they learn to detect circles of diameter 50 pixels and triangles of side length 30. \n
When you resize input images, you change the size of your circles and triangles. So depending on how you resize your input image, this pretrained CNN may or may not find circles of diameter 50 and triangles of side 30. In the example below, given the original image, the CNN only finds the circles when the input image is resized to 512x512 and finds triangles when resized to 128x128
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F4b883a2a1f34dc271ec3e102161ef1f4%2Ftop2.jpg?generation=1592623537198832&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F8f6e9e0860c47de71de4a58a59e121ce%2Ftop3.jpg?generation=1592624049717100&alt=media\")
In conclusion, a CNN searches for thousands of patterns (not just one circle size and one triangle size). It will find some patterns with some sizes and other patterns with other sizes. Therefore you should experiment with different input sizes to see which has better CV and LB. Also consider ensembling models of different input sizes for maximum CV and LB.
\n\nI have resized all the competition images to TFRecords 768x768 here, 512x512 here, 384x384 here, 256x256 here, 192x192 here, and I provide external data of size 512x512 here. If you ensemble models of different sizes, you can achieve LB 0.960 or higher!
\n\nIf you prefer JPEGs instead of the TFRecords, you can find JPEGs 768x768 here, 512x512 here, 384x384 here, 256x256 here, 192x192 here. And external data 512x512 here. (Also I have resized last year's 2019 data here).
\n\nYou can even experiment with a single model that analyzes multiple image sizes at once with multiple EfficientNet backbones. After the image is inputted, you apply multiple types of downsizing like 2x, 3x, 4x to feed the different backbones. Then apply GlobalAveragePooling2D() after all the backbones then Concatenate all those vectors and finally apply Dense(1, activation='sigmoid') for classification. This worked well in Cloud Comp. An example is posted in Krazy Klassifiers here
Enjoy!
", "messages": [ { "id": 893879, "postDate": "2020-06-20T03:51:34.783Z", "content": "Most Imagenet pretrained CNNs were trained on 224x224 image resolution. It is a common misconception, that when using these pretrained CNN, images need to be resized to 224x224. On the contrary, popular CNN are fully convolutional nets that can accept any input size.
\n\nYou can input any image size and these CNN output feature maps that are 32x times smaller. For example, if you input 224x224 then the CNN outputs feature maps of size 7x7. If you input images of size 512x512, then these CNN outputs feature maps of size 16x16.
\n\nThe only relevance of the pretraining 224x224 size is that these CNN have learned to find certain patterns of certain sizes. For example, maybe they learned to find circles that are 50 pixels diameter, or maybe they learned to find triangles with side length 30 pixels.
\n\nIn the example below, we pretrain CNN on images of size 224x224 and they learn to detect circles of diameter 50 pixels and triangles of side length 30. \n
When you resize input images, you change the size of your circles and triangles. So depending on how you resize your input image, this pretrained CNN may or may not find circles of diameter 50 and triangles of side 30. In the example below, given the original image, the CNN only finds the circles when the input image is resized to 512x512 and finds triangles when resized to 128x128
\n\n
In conclusion, a CNN searches for thousands of patterns (not just one circle size and one triangle size). It will find some patterns with some sizes and other patterns with other sizes. Therefore you should experiment with different input sizes to see which has better CV and LB. Also consider ensembling models of different input sizes for maximum CV and LB.
\n\nI have resized all the competition images to TFRecords 768x768 here, 512x512 here, 384x384 here, 256x256 here, 192x192 here, and I provide external data of size 512x512 here. If you ensemble models of different sizes, you can achieve LB 0.960 or higher!
\n\nIf you prefer JPEGs instead of the TFRecords, you can find JPEGs 768x768 here, 512x512 here, 384x384 here, 256x256 here, 192x192 here. And external data 512x512 here. (Also I have resized last year's 2019 data here).
\n\nYou can even experiment with a single model that analyzes multiple image sizes at once with multiple EfficientNet backbones. After the image is inputted, you apply multiple types of downsizing like 2x, 3x, 4x to feed the different backbones. Then apply GlobalAveragePooling2D() after all the backbones then Concatenate all those vectors and finally apply Dense(1, activation='sigmoid') for classification. This worked well in Cloud Comp. An example is posted in Krazy Klassifiers here
Enjoy!
", "rawMarkdown": "Most Imagenet pretrained CNNs were trained on 224x224 image resolution. It is a common misconception, that when using these pretrained CNN, images need to be resized to 224x224. On the contrary, popular CNN are fully convolutional nets that can accept any input size. \n\nYou can input any image size and these CNN output feature maps that are 32x times smaller. For example, if you input 224x224 then the CNN outputs feature maps of size 7x7. If you input images of size 512x512, then these CNN outputs feature maps of size 16x16.\n\n# Feature Maps\nThe only relevance of the pretraining 224x224 size is that these CNN have learned to find certain patterns of **certain sizes**. For example, maybe they learned to find circles that are 50 pixels diameter, or maybe they learned to find triangles with side length 30 pixels.\n\n# Pretrained Imagenet CNN\nIn the example below, we pretrain CNN on images of size 224x224 and they learn to detect circles of diameter 50 pixels and triangles of side length 30. \n\n\n# Why Resize Input Images\nWhen you resize input images, you change the size of your circles and triangles. So depending on how you resize your input image, this pretrained CNN may or may not find circles of diameter 50 and triangles of side 30. In the example below, given the original image, the CNN only finds the circles when the input image is resized to 512x512 and finds triangles when resized to 128x128\n\n# 512x512 Input\n\n\n#256x256 and 128x128 Input\n\n\n# Conclusion\nIn conclusion, a CNN searches for thousands of patterns (not just one circle size and one triangle size). It will find some patterns with some sizes and other patterns with other sizes. Therefore you should experiment with different input sizes to see which has better CV and LB. Also consider ensembling models of different input sizes for maximum CV and LB.\n\n# Kaggle Datasets\nI have resized all the competition images to TFRecords 768x768 [here][1], 512x512 [here][2], 384x384 [here][3], 256x256 [here][4], 192x192 [here][7], and I provide external data of size 512x512 [here][5]. If you ensemble models of different sizes, you can achieve LB 0.960 or higher!\n\nIf you prefer JPEGs instead of the TFRecords, you can find JPEGs 768x768 [here][8], 512x512 [here][9], 384x384 [here][10], 256x256 [here][11], 192x192 [here][12]. And external data 512x512 [here][13]. (Also I have resized last year's 2019 data [here][14]).\n\nYou can even experiment with a single model that analyzes multiple image sizes at once with multiple EfficientNet backbones. After the image is inputted, you apply multiple types of downsizing like 2x, 3x, 4x to feed the different backbones. Then apply `GlobalAveragePooling2D()` after all the backbones then `Concatenate` all those vectors and finally apply `Dense(1, activation='sigmoid')` for classification. This worked well in Cloud Comp. An example is posted in Krazy Klassifiers [here][6] \n\nEnjoy!\n \n[1]: https://www.kaggle.com/cdeotte/melanoma-768x768\n[2]: https://www.kaggle.com/cdeotte/melanoma-512x512\n[3]: https://www.kaggle.com/cdeotte/melanoma-384x384\n[4]: https://www.kaggle.com/cdeotte/melanoma-256x256\n[5]: https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images\n[6]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/118086\n[7]: https://www.kaggle.com/cdeotte/melanoma-192x192\n[8]: https://www.kaggle.com/cdeotte/jpeg-melanoma-768x768\n[9]: https://www.kaggle.com/cdeotte/jpeg-melanoma-512x512\n[10]: https://www.kaggle.com/cdeotte/jpeg-melanoma-384x384\n[11]: https://www.kaggle.com/cdeotte/jpeg-melanoma-256x256\n[12]: https://www.kaggle.com/cdeotte/jpeg-melanoma-192x192\n[13]: https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg\n[14]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910", "votes": 508 }, { "id": 2185587, "postDate": "2023-03-17T07:13:58.820Z", "content": "Your visualization is beautiful @cdeotte
", "rawMarkdown": "Your visualization is beautiful @cdeotte ", "votes": 4 }, { "id": 918961, "postDate": "2020-07-07T16:01:24.867Z", "content": "you know what? this makes easier to understand than reddit!😂
", "rawMarkdown": "you know what? this makes easier to understand than reddit!😂\n", "votes": 11, "replies": [ { "id": 919756, "postDate": "2020-07-08T05:05:32.130Z", "content": "Thanks Jimmy
", "rawMarkdown": "Thanks Jimmy", "votes": 2 }, { "id": 919764, "postDate": "2020-07-08T05:09:41.473Z", "content": "Already followed you👍
", "rawMarkdown": "Already followed you👍", "votes": 7 } ] }, { "id": 1685746, "postDate": "2022-02-11T14:14:41.377Z", "content": "thanks，from your PetFinder link , congratulations!
", "rawMarkdown": "thanks，from your PetFinder link , congratulations!", "votes": 1 }, { "id": 895277, "postDate": "2020-06-21T08:13:54.457Z", "content": "\n\nthanks for the post.\nwhich is more important, size of image image or depth (complexity) of model.
\n\nit seems difficult to train a deep efficientnetB7 on large size on local GPU and shallower efficientnetB0,1,2 doesn't benefit from large size?
", "rawMarkdown": "@cdeotte \n\nthanks for the post.\nwhich is more important, size of image image or depth (complexity) of model.\n\nit seems difficult to train a deep efficientnetB7 on large size on local GPU and shallower efficientnetB0,1,2 doesn't benefit from large size?", "votes": 5, "replies": [ { "id": 895942, "postDate": "2020-06-21T17:45:21.223Z", "content": "Exactly!\nThis is what has put me in a dilemma.\nNot sure which is important?
", "rawMarkdown": "Exactly!\nThis is what has put me in a dilemma.\nNot sure which is important?" }, { "id": 896220, "postDate": "2020-06-22T01:46:35.867Z", "content": "I don't know which is more \"important\". Note my post is not saying \"bigger is better\". My post is saying that \"bigger is different\". (Likewise different backbones (efn0, efn1, ..., efn7) are different. And deeper is not necessarily better in this comp because there are issues of overfitting. We don't have millions of training images like imagenet).
\n\nTry this experiment. Take any of your models even one with EfficientNetB0 and check the CV using sizes 96, 128, 192, 256, 384, 512, 768, 1024. You will notice that the CV score is different with different image sizes. That is what my post is saying.
", "rawMarkdown": "I don't know which is more \"important\". Note my post is not saying \"bigger is better\". My post is saying that \"bigger is different\". (Likewise different backbones (efn0, efn1, ..., efn7) are different. And deeper is not necessarily better in this comp because there are issues of overfitting. We don't have millions of training images like imagenet).\n\nTry this experiment. Take any of your models even one with EfficientNetB0 and check the CV using sizes 96, 128, 192, 256, 384, 512, 768, 1024. You will notice that the CV score is different with different image sizes. That is what my post is saying.", "votes": 10 }, { "id": 896242, "postDate": "2020-06-22T02:28:54.060Z", "content": "Also note that you can train deep models on large image sizes with limited compute resources by freezing bottom layers.
", "rawMarkdown": "Also note that you can train deep models on large image sizes with limited compute resources by freezing bottom layers.", "votes": 2 }, { "id": 896899, "postDate": "2020-06-22T14:08:43.340Z", "content": "When you say bottom layers here, which ones does that mean exactly? Can you provide an example for EfficientNetB0?
", "rawMarkdown": "When you say bottom layers here, which ones does that mean exactly? Can you provide an example for EfficientNetB0?" }, { "id": 897021, "postDate": "2020-06-22T15:15:14.647Z", "content": "@p4rallax , I think what @cdeotte is trying to say is that majority of parameters come from the linear layers. So freezing them and training would greatly decrease the computational cost. (for deep models)
", "rawMarkdown": "@p4rallax , I think what @cdeotte is trying to say is that majority of parameters come from the linear layers. So freezing them and training would greatly decrease the computational cost. (for deep models)" }, { "id": 897365, "postDate": "2020-06-22T19:59:55.517Z", "content": "I see , that makes sense. Thank you.
", "rawMarkdown": "I see , that makes sense. Thank you." }, { "id": 903519, "postDate": "2020-06-26T22:52:49.840Z", "content": "@p4rallax @rohan1602 I believe \"bottom layers\" refers to the early convolutional layers of the network, while the top typically refers to the final linear layers once spatial features are flattened. When training a network a majority of the memory usage comes from the storage of intermediate hidden representations that will be needed to compute gradients during backprop, rather than the parameters of the model. If you disable the gradient for the early parts of the forward pass (where inputs are large due to spatial dimensions) you will save alot of memory. This can be done because CNNs learn pretty general features and good performance can be achieved by only fine-tuning the final linear layers.
", "rawMarkdown": "@p4rallax @rohan1602 I believe \"bottom layers\" refers to the early convolutional layers of the network, while the top typically refers to the final linear layers once spatial features are flattened. When training a network a majority of the memory usage comes from the storage of intermediate hidden representations that will be needed to compute gradients during backprop, rather than the parameters of the model. If you disable the gradient for the early parts of the forward pass (where inputs are large due to spatial dimensions) you will save alot of memory. This can be done because CNNs learn pretty general features and good performance can be achieved by only fine-tuning the final linear layers.", "votes": 5 }, { "id": 903743, "postDate": "2020-06-27T04:41:39.467Z", "content": "Oh yeah..might be correct..\nSorry..I have recently started Deep Learning..so might have misunderstood the terminology.\nThanks for clarifying :)
", "rawMarkdown": "Oh yeah..might be correct..\nSorry..I have recently started Deep Learning..so might have misunderstood the terminology.\nThanks for clarifying :)", "votes": 1 } ] }, { "id": 896018, "postDate": "2020-06-21T18:49:17.807Z", "content": "That is the one weird looking racing car you have there Chris.
", "rawMarkdown": "That is the one weird looking racing car you have there Chris.", "votes": 6 }, { "id": 991203, "postDate": "2020-08-30T08:11:07.813Z", "content": "I learnt a new lesson today from the discussion
", "rawMarkdown": "I learnt a new lesson today from the discussion", "votes": 3 }, { "id": 904543, "postDate": "2020-06-27T17:45:33.060Z", "content": "Hey Chris! You analysis always helps.
", "rawMarkdown": "Hey Chris! You analysis always helps.", "votes": 3 }, { "id": 901957, "postDate": "2020-06-25T20:16:45.100Z", "content": "Nice Explanation! Thanks :) .As you can see, the images of melanomas are taken with different distances, which can give us false information regarding the real size of the melanoma. do you think it could affect the model ?
", "rawMarkdown": "Nice Explanation! Thanks :) .As you can see, the images of melanomas are taken with different distances, which can give us false information regarding the real size of the melanoma. do you think it could affect the model ?", "votes": 3, "replies": [ { "id": 902096, "postDate": "2020-06-25T23:44:21.820Z", "content": "Great point. It would be nice to extract this distance information and give it to our model as a meta feature.
", "rawMarkdown": "Great point. It would be nice to extract this distance information and give it to our model as a meta feature.", "votes": 1 } ] }, { "id": 900983, "postDate": "2020-06-25T07:30:12.920Z", "content": "Other than trying different image sizes, I started with trying different architectures for a image size of 224, I used a B0 and a Resnet50 arch for this work, as my intuition was that different archs may learn different features.
And, as to validate, I got 0.888 LB with only B0 while I got 0.895 with an ensemble of B0 and Resnet50.
\n\nAlso, I thought of doing this for more image sizes like 384, 512, 768, 1024 and then combining their results. But I think it would not be computationally efficient as it would take a lot of time to do that, what do you think of that, @cdeotte Should I go for it?
", "rawMarkdown": "Other than trying different image sizes, I started with trying different architectures for a image size of `224`, I used a B0 and a Resnet50 arch for this work, as my intuition was that different archs may learn different features.\n\nAnd, as to validate, I got 0.888 LB with only B0 while I got 0.895 with an ensemble of B0 and Resnet50.\n\nAlso, I thought of doing this for more image sizes like 384, 512, 768, 1024 and then combining their results. But I think it would not be computationally efficient as it would take a lot of time to do that, what do you think of that, @cdeotte Should I go for it?", "votes": 3, "replies": [ { "id": 901471, "postDate": "2020-06-25T14:02:23.593Z", "content": "Early in a competition, I suggest that you focus on one model and get the validation score as high as possible. Try different sizes and different model architectures. Also experiment with data augmentation, loss, optimizer, learning schedule. Later in the comp, you can input different sized images into your one model and ensemble the multiple predictions for a CV LB boost.
", "rawMarkdown": "Early in a competition, I suggest that you focus on one model and get the validation score as high as possible. Try different sizes and different model architectures. Also experiment with data augmentation, loss, optimizer, learning schedule. Later in the comp, you can input different sized images into your one model and ensemble the multiple predictions for a CV LB boost.", "votes": 9 }, { "id": 901507, "postDate": "2020-06-25T14:32:22.437Z", "content": "Sure, thanks @cdeotte I will work on Hyper tuning then, actually its my first competition(to some extent), so I was eager to get up in the leaderboard and was being impatient. 😄
\n\nAlso, one more question, I ran my notebook once and submitted the predictions and got .888 LB, but when I again ran it, it only gave me .798, of course there was overfitting and I understand that I should save my models, but I didn't actually save them due to my ignorance.
\n\nSo what should I do in such a case to maintain stability in my models? I used same seed and everything was same too.
", "rawMarkdown": "Sure, thanks @cdeotte I will work on Hyper tuning then, actually its my first competition(to some extent), so I was eager to get up in the leaderboard and was being impatient. 😄 \n\nAlso, one more question, I ran my notebook once and submitted the predictions and got .888 LB, but when I again ran it, it only gave me .798, of course there was overfitting and I understand that I should save my models, but I didn't actually save them due to my ignorance. \n\nSo what should I do in such a case to maintain stability in my models? I used same seed and everything was same too. " }, { "id": 901528, "postDate": "2020-06-25T14:43:44.090Z", "content": "This is the biggest challenge in this competition. With such an unbalanced target (only 1% positive), AUC metric is very volatile when training with cross entropy. You can stabilize your models with folds, bagging, and ensembles. But the best way is to improve the training process (to optimize AUC) but either modifying the loss or using other tricks.
", "rawMarkdown": "This is the biggest challenge in this competition. With such an unbalanced target (only 1% positive), AUC metric is very volatile when training with cross entropy. You can stabilize your models with folds, bagging, and ensembles. But the best way is to improve the training process (to optimize AUC) but either modifying the loss or using other tricks.", "votes": 3 }, { "id": 901543, "postDate": "2020-06-25T14:49:42.987Z", "content": "Sure, I am using folds and ensembles with tta as of now, I think many more things are needed here, I tried augmenting only the melanoma images but the model just got overfitted and was of no use.
", "rawMarkdown": "Sure, I am using folds and ensembles with tta as of now, I think many more things are needed here, I tried augmenting only the melanoma images but the model just got overfitted and was of no use.", "votes": 1 }, { "id": 902203, "postDate": "2020-06-26T01:46:21.717Z", "content": "great insight..
", "rawMarkdown": "great insight.. ", "votes": 2 }, { "id": 903753, "postDate": "2020-06-27T04:59:34.870Z", "content": "Hi there Chris,
\n\nI am still quite unsure on how to perform group Kfold using this tfrec data type, do you care to explain how to approach this problem? Do I separate the training set by its tfrec file?
\n\nThank you so much for your valuable inputs.
", "rawMarkdown": "Hi there Chris,\n\nI am still quite unsure on how to perform group Kfold using this tfrec data type, do you care to explain how to approach this problem? Do I separate the training set by its tfrec file?\n\nThank you so much for your valuable inputs." } ] }, { "id": 896384, "postDate": "2020-06-22T06:05:10.323Z", "content": "Amazing ! I wondered why higher resolution images may not lead to a better score. Now I know why 😄
", "rawMarkdown": "Amazing ! I wondered why higher resolution images may not lead to a better score. Now I know why 😄 \n\n", "votes": 3 }, { "id": 893903, "postDate": "2020-06-20T04:31:55.840Z", "content": "Thank you so very much! Will definitely give this a try!
", "rawMarkdown": "Thank you so very much! Will definitely give this a try!", "votes": 3 }, { "id": 1172017, "postDate": "2021-01-27T08:10:37.427Z", "content": "this lesson help me last-week's question
", "rawMarkdown": "this lesson help me last-week's question", "votes": 1 }, { "id": 1102547, "postDate": "2020-12-05T03:46:46.163Z", "content": "Interesting stuffs! I always assume the input size have to follow the pretrained input size. This is an eye-opening.
", "rawMarkdown": "Interesting stuffs! I always assume the input size have to follow the pretrained input size. This is an eye-opening.", "votes": 1 }, { "id": 1059093, "postDate": "2020-10-24T15:59:29.923Z", "content": "good stuff
", "rawMarkdown": "good stuff", "votes": 1 }, { "id": 1047170, "postDate": "2020-10-12T10:42:41.223Z", "content": "great work
", "rawMarkdown": "great work", "votes": 1 }, { "id": 1002499, "postDate": "2020-09-08T07:18:21.937Z", "content": "This is really insightful and helpful. Thank you so much!
", "rawMarkdown": "This is really insightful and helpful. Thank you so much!", "votes": 1 }, { "id": 971898, "postDate": "2020-08-16T04:02:08.250Z", "content": "Nice Explanation sir. One question is that how pretrained model take any size input if they originally trained on 224*224 size images?
", "rawMarkdown": "Nice Explanation sir. One question is that how pretrained model take any size input if they originally trained on 224*224 size images?", "votes": 1, "replies": [ { "id": 971940, "postDate": "2020-08-16T05:19:19.477Z", "content": "It's a fully convolutional network. That means that it scans the input image with a 3x3 window. Then after a bunch of these layers, the images get reduced to half the size. And so on. At no point does the network care about the original size.
", "rawMarkdown": "It's a fully convolutional network. That means that it scans the input image with a 3x3 window. Then after a bunch of these layers, the images get reduced to half the size. And so on. At no point does the network care about the original size.", "votes": 3, "replies": [ { "id": 971951, "postDate": "2020-08-16T05:44:33.957Z", "content": "Sir, If i want to train my model from scratch and want to change it as you have explained above then what i have to add extra. Can you have any kernel or link regarding this explanation.
", "rawMarkdown": "Sir, If i want to train my model from scratch and want to change it as you have explained above then what i have to add extra. Can you have any kernel or link regarding this explanation." } ] } ] }, { "id": 967321, "postDate": "2020-08-12T06:50:37.103Z", "content": "Intresting
", "rawMarkdown": "Intresting", "votes": 1 }, { "id": 955987, "postDate": "2020-08-03T06:11:49.683Z", "content": "Thanks for the explanation . It was useful .
", "rawMarkdown": "Thanks for the explanation . It was useful .", "votes": 1 }, { "id": 945159, "postDate": "2020-07-25T15:53:02.630Z", "content": "Very good and descriptive explanation!
\n\nScale issues is so important that there are some works that achieve \"Top-of-ImageNet\" position only by properly solving scale issue.\nhttps://arxiv.org/abs/2003.08237\nhttps://arxiv.org/abs/1906.06423
", "rawMarkdown": "Very good and descriptive explanation!\n \nScale issues is so important that there are some works that achieve \"Top-of-ImageNet\" position only by properly solving scale issue.\nhttps://arxiv.org/abs/2003.08237\nhttps://arxiv.org/abs/1906.06423", "votes": 1, "replies": [ { "id": 949629, "postDate": "2020-07-28T18:56:36.380Z", "content": "Hum I would give much of the credit about the smart scaling idea to the original Efficientnet paper from Google Brain :)
", "rawMarkdown": "Hum I would give much of the credit about the smart scaling idea to the original Efficientnet paper from Google Brain :)", "votes": 1 } ] }, { "id": 944411, "postDate": "2020-07-25T05:12:09.587Z", "content": "Thanks much. Your explanation is easy to understand. \nIt is very helpful to a newbie like me.
", "rawMarkdown": "Thanks much. Your explanation is easy to understand. \nIt is very helpful to a newbie like me.", "votes": 1 }, { "id": 934430, "postDate": "2020-07-18T12:33:26.637Z", "content": "Wow!
\nSo much new information to learn from this.
\nThanks!
👍 👍
", "rawMarkdown": "👍 👍 ", "votes": 1 }, { "id": 919844, "postDate": "2020-07-08T06:23:05.733Z", "content": "Thanks for your explaination. This really helps me a lot.
", "rawMarkdown": "Thanks for your explaination. This really helps me a lot.", "votes": 1 }, { "id": 918798, "postDate": "2020-07-07T13:57:22.090Z", "content": "This makes a lot more sense now, thanks for the detailed explanation!
", "rawMarkdown": "This makes a lot more sense now, thanks for the detailed explanation!", "votes": 1, "replies": [ { "id": 919001, "postDate": "2020-07-07T16:34:26.427Z", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": 1 } ] }, { "id": 915240, "postDate": "2020-07-04T15:22:07.903Z", "content": "Great Explanation!
", "rawMarkdown": "Great Explanation!", "votes": 1 }, { "id": 915179, "postDate": "2020-07-04T14:20:29.450Z", "content": "thank you for sharing this knowledge, due to this type of sharing, kaggle is always best for learning new things everyday.
", "rawMarkdown": "thank you for sharing this knowledge, due to this type of sharing, kaggle is always best for learning new things everyday.\n ", "votes": 1 }, { "id": 913780, "postDate": "2020-07-03T12:18:45.933Z", "content": "Thanks for the insightful explanation! Put into a lot of perspective the idea of using different sized images for training.
", "rawMarkdown": "Thanks for the insightful explanation! Put into a lot of perspective the idea of using different sized images for training. ", "votes": 1, "replies": [ { "id": 914656, "postDate": "2020-07-04T06:03:23.460Z", "content": "Thank you
", "rawMarkdown": "Thank you" } ] }, { "id": 913578, "postDate": "2020-07-03T09:39:32.743Z", "content": "I understood it well now, thank you very much.
", "rawMarkdown": "I understood it well now, thank you very much.", "votes": 1 }, { "id": 910957, "postDate": "2020-07-01T13:20:28.390Z", "content": "Thanks @cdeotte
\n\nI'm curious about the effect of weight sharing between EfficientNets vs different nets
", "rawMarkdown": "Thanks @cdeotte\n\nI'm curious about the effect of weight sharing between EfficientNets vs different nets", "votes": 1, "replies": [ { "id": 911786, "postDate": "2020-07-02T02:54:41.750Z", "content": "What do you mean? Are you asking which weights to use for ensemble of EfficientNet with other CNN?
", "rawMarkdown": "What do you mean? Are you asking which weights to use for ensemble of EfficientNet with other CNN?", "votes": 1 }, { "id": 916660, "postDate": "2020-07-05T22:35:13.790Z", "content": "I'm thinking about whether we can improve the performance by training a single network to jointly predict the same image on different scales, for example
\n\nnet = EfficientNetB0(...)\nfeatures0 = keras.layers.GlobalAveragePooling2D()(net(img_512x512))\nfeatures1 = keras.layers.GlobalAveragePooling2D()(net(img_256x256))\npred0 = keras.layers.Dense(1, activation='sigmoid')(features0)\npred1 = keras.layers.Dense(1, activation='sigmoid')(features1)
or if it is necessary to use different nets to have a boost in the performance, like this
\n\nfeatures0 = keras.layers.GlobalAveragePooling2D()(EfficientNetB0(...)(img_512x512))\nfeatures1 = keras.layers.GlobalAveragePooling2D()(EfficientNetB0(...)(img_256x256))\npred0 = keras.layers.Dense(1, activation='sigmoid')(features0)\npred1 = keras.layers.Dense(1, activation='sigmoid')(features1)
Maybe I'll try in the next days
", "rawMarkdown": "I'm thinking about whether we can improve the performance by training a single network to jointly predict the same image on different scales, for example\n\n`net = EfficientNetB0(...)`\n`features0 = keras.layers.GlobalAveragePooling2D()(net(img_512x512))`\n`features1 = keras.layers.GlobalAveragePooling2D()(net(img_256x256))`\n`pred0 = keras.layers.Dense(1, activation='sigmoid')(features0)`\n`pred1 = keras.layers.Dense(1, activation='sigmoid')(features1)`\n\nor if it is necessary to use different nets to have a boost in the performance, like this\n\n`features0 = keras.layers.GlobalAveragePooling2D()(EfficientNetB0(...)(img_512x512))`\n`features1 = keras.layers.GlobalAveragePooling2D()(EfficientNetB0(...)(img_256x256))`\n`pred0 = keras.layers.Dense(1, activation='sigmoid')(features0)`\n`pred1 = keras.layers.Dense(1, activation='sigmoid')(features1)`\n\nMaybe I'll try in the next days" } ] }, { "id": 910465, "postDate": "2020-07-01T06:52:27.770Z", "content": "Very nice effort
", "rawMarkdown": "Very nice effort", "votes": 1 }, { "id": 910401, "postDate": "2020-07-01T06:14:51.977Z", "content": "Thank @cdeotte ,
\n\nI try to implement your idea, and I got 0.912 score. I still work on it.\nLink here [https://www.kaggle.com/truonghoang/multi-size-eff-lb-0-912]
\n\nGreat idea.
", "rawMarkdown": "Thank @cdeotte ,\n\nI try to implement your idea, and I got 0.912 score. I still work on it.\nLink here [https://www.kaggle.com/truonghoang/multi-size-eff-lb-0-912]\n\nGreat idea.", "votes": 1, "replies": [ { "id": 910417, "postDate": "2020-07-01T06:21:15.603Z", "content": "Great work. You should tune the number of epochs separately for each model. The same number is not optimal for all of them.
", "rawMarkdown": "Great work. You should tune the number of epochs separately for each model. The same number is not optimal for all of them.", "votes": 2 } ] }, { "id": 909998, "postDate": "2020-07-01T00:58:09.490Z", "content": "Great!
", "rawMarkdown": "Great!", "votes": 1 }, { "id": 907946, "postDate": "2020-06-30T09:21:29.783Z", "content": "Interesting Stuff
", "rawMarkdown": "Interesting Stuff", "votes": 1 }, { "id": 907945, "postDate": "2020-06-30T09:20:46.867Z", "content": "Interesting stuff
", "rawMarkdown": "**Interesting stuff**", "votes": 1 }, { "id": 907722, "postDate": "2020-06-30T05:54:12.253Z", "content": "Great Explanation!!
", "rawMarkdown": "Great Explanation!!", "votes": 1 }, { "id": 907720, "postDate": "2020-06-30T05:53:12.547Z", "content": "Great Explanation !!
", "rawMarkdown": "Great Explanation !!", "votes": 1 }, { "id": 907617, "postDate": "2020-06-30T04:19:40.033Z", "content": "You are my real life saver, thanks for your great explanation!
", "rawMarkdown": "You are my real life saver, thanks for your great explanation!", "votes": 1, "replies": [ { "id": 911787, "postDate": "2020-07-02T02:54:59.107Z", "content": "Thank you
", "rawMarkdown": "Thank you" } ] }, { "id": 905988, "postDate": "2020-06-29T02:39:12.577Z", "content": "short but practical, very cool effort!
", "rawMarkdown": "short but practical, very cool effort! ", "votes": 1 }, { "id": 905077, "postDate": "2020-06-28T08:24:58.693Z", "content": "Helpful explanation!
", "rawMarkdown": "Helpful explanation!", "votes": 1 }, { "id": 904031, "postDate": "2020-06-27T09:44:26.960Z", "content": "Very Interesting! Thanks for explaining.
", "rawMarkdown": "Very Interesting! Thanks for explaining.", "votes": 1 }, { "id": 903507, "postDate": "2020-06-26T22:24:12.527Z", "content": "thank you @cdeotte for enlightening us about that, is there some public notebook on how to ensemble models with different sizes?
", "rawMarkdown": "thank you @cdeotte for enlightening us about that, is there some public notebook on how to ensemble models with different sizes?", "votes": 1, "replies": [ { "id": 903746, "postDate": "2020-06-27T04:44:04.723Z", "content": "There are two ways. You can either ensemble like normal. Which is w * preds1 + (1-w) * preds2 where you find weight w using OOF predictions from training both models on the same validation holdout or CV folds. The second method is building a single model with multiple backbones and then adding a single classification head.
I will try that for sure.
", "rawMarkdown": "I will try that for sure." }, { "id": 904545, "postDate": "2020-06-27T17:46:19.767Z", "content": "You always get the best w in this case using linear regression.
", "rawMarkdown": "You always get the best w in this case using linear regression." } ] }, { "id": 902244, "postDate": "2020-06-26T02:58:14.530Z", "content": "Very Interesting! Thanks for explaining.
", "rawMarkdown": "Very Interesting! Thanks for explaining.", "votes": 1 }, { "id": 899155, "postDate": "2020-06-24T03:17:58.620Z", "content": "Excellent explanation!
", "rawMarkdown": "Excellent explanation!", "votes": 1 }, { "id": 898998, "postDate": "2020-06-23T21:59:38.380Z", "content": "Great explanation!
", "rawMarkdown": "Great explanation!", "votes": 1 }, { "id": 898706, "postDate": "2020-06-23T17:22:23.893Z", "content": "This is great. Thank you for putting this together. I could relate to my recent Image Classification file (what not to do maybe).
", "rawMarkdown": "This is great. Thank you for putting this together. I could relate to my recent Image Classification file (what not to do maybe).", "votes": 1 }, { "id": 897609, "postDate": "2020-06-23T01:49:30.193Z", "content": "Hi Chris @cdeotte, thank you for sharing! Are the files with the same name in different resized folders based on the same original images? For example, are files \"train00-2071\" in 768x768 folder and 512x512 folder are from the same batch of original images? I'm trying to hold out the same portion of data for validation while using different sizes of image as input. Hope I described my question clearly.
", "rawMarkdown": "Hi Chris @cdeotte, thank you for sharing! Are the files with the same name in different resized folders based on the same original images? For example, are files \"train00-2071\" in 768x768 folder and 512x512 folder are from the same batch of original images? I'm trying to hold out the same portion of data for validation while using different sizes of image as input. Hope I described my question clearly.", "votes": 1, "replies": [ { "id": 897618, "postDate": "2020-06-23T01:53:56.547Z", "content": "Yes they are the same. The file \"train00-2071\" for 512x512 are the same images as \"train00-2071\" for 768x768 and all the sizes 256 and 384.
\n\n(So my 256, 384, 512, and 784 are the same. My 5th dataset is external data of size 512x512 those files are different than the 256, 384, 512, and 784)
", "rawMarkdown": "Yes they are the same. The file \"train00-2071\" for 512x512 are the same images as \"train00-2071\" for 768x768 and all the sizes 256 and 384.\n\n(So my 256, 384, 512, and 784 are the same. My 5th dataset is external data of size 512x512 those files are different than the 256, 384, 512, and 784)" }, { "id": 897724, "postDate": "2020-06-23T04:04:46.970Z", "content": "hi chris, thank you for this amazing work(and all your other amazing works!). May I ask how do you ensemble different models with different image sizes? Do you run them on separate models and mix the results up together or do you concatenate them into one model?
\n\nSorry for newbie question.
", "rawMarkdown": "hi chris, thank you for this amazing work(and all your other amazing works!). May I ask how do you ensemble different models with different image sizes? Do you run them on separate models and mix the results up together or do you concatenate them into one model?\n\nSorry for newbie question." }, { "id": 898376, "postDate": "2020-06-23T13:42:49.537Z", "content": "Thank you so much!!
", "rawMarkdown": "Thank you so much!!" }, { "id": 898491, "postDate": "2020-06-23T14:54:34.320Z", "content": "@FadzlinRafi Both techniques work. For example, let's say you wanted to ensemble 1024, 512, and 256. You could build 3 separate models and ensemble.
\n\nOf you could input just the sizes of 1024 into one model. Then inside your NN, you split the input to 3 parallel nets. In the first net you use 1024. In the second net, you apply pooling 2x2 with stride 2. And in the third net you apply pooling 4x4 with stride 4. Then nets 2 and 3 will process images of size 512 and 256 respectively. In the end you concatenate the nets back together and add a single output sigmoid. (Picture here)
", "rawMarkdown": "@FadzlinRafi Both techniques work. For example, let's say you wanted to ensemble 1024, 512, and 256. You could build 3 separate models and ensemble. \n\nOf you could input just the sizes of 1024 into one model. Then inside your NN, you split the input to 3 parallel nets. In the first net you use 1024. In the second net, you apply pooling 2x2 with stride 2. And in the third net you apply pooling 4x4 with stride 4. Then nets 2 and 3 will process images of size 512 and 256 respectively. In the end you concatenate the nets back together and add a single output sigmoid. (Picture [here][1])\n\n[1]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/118086\n", "votes": 2 } ] }, { "id": 897063, "postDate": "2020-06-22T15:44:27.933Z", "content": "\n\n\nI have resized all the competition images to 768x768 here, 512x512 here, 384x384 here, 256x256 here, and I provide external data of size 512x512 here.
\n
Is there an 'IOS' version of this 'app' available?
\n\nAnd by IOS I mean pytorch. Cause, you know, tfrecords -> tensorflow -> google -> android...\nAnd by version I mean plain jpg image or something.
", "rawMarkdown": "> I have resized all the competition images to 768x768 here, 512x512 here, 384x384 here, 256x256 here, and I provide external data of size 512x512 here.\n\nIs there an 'IOS' version of this 'app' available?\n\nAnd by IOS I mean pytorch. Cause, you know, tfrecords -> tensorflow -> google -> android...\nAnd by version I mean plain jpg image or something.", "votes": 1, "replies": [ { "id": 897089, "postDate": "2020-06-22T15:57:04.063Z", "content": "lol!\nYeah it would be very useful for me too ;)
", "rawMarkdown": "lol!\nYeah it would be very useful for me too ;)" }, { "id": 897172, "postDate": "2020-06-22T17:05:05.077Z", "content": "I started doing it for 128, 384, 512, 768, and 1024, but I could do only 128 and test data of 384 as of now, if you need them, I can make them public. Also, they are png(s). :-)
\n\nP.S. Thanks Roman, your Notebook helped a lot! :)
\n\nUPDATE: You can find jpg images for 384, 512, 768 and 1024 here. :)
", "rawMarkdown": "I started doing it for 128, 384, 512, 768, and 1024, but I could do only 128 and test data of 384 as of now, if you need them, I can make them public. Also, they are png(s). :-)\n\nP.S. Thanks Roman, your Notebook helped a lot! :)\n****************************************************\nUPDATE: You can find jpg images for 384, 512, 768 and 1024 [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161043). :)" } ] }, { "id": 896183, "postDate": "2020-06-22T00:05:25.057Z", "content": "Nice Explanation!
", "rawMarkdown": "Nice Explanation!\n", "votes": 1 }, { "id": 895940, "postDate": "2020-06-21T17:43:24.870Z", "content": "@cdeotte Great insight!\nJust curious though..did increase in size result in a higher score for you?
", "rawMarkdown": "@cdeotte Great insight!\nJust curious though..did increase in size result in a higher score for you?", "votes": 1, "replies": [ { "id": 896221, "postDate": "2020-06-22T01:49:06.790Z", "content": "Note that my post is not saying \"bigger is better\". It is saying that \"bigger is different\". My best single model scores CV 0.950+ and LB 0.950+ and it does not use the biggest size. But when i ensemble models of various sizes (including the biggest size) it pushes my CV LB score higher.
", "rawMarkdown": "Note that my post is not saying \"bigger is better\". It is saying that \"bigger is different\". My best single model scores CV 0.950+ and LB 0.950+ and it does not use the biggest size. But when i ensemble models of various sizes (including the biggest size) it pushes my CV LB score higher.", "votes": 14 }, { "id": 896238, "postDate": "2020-06-22T02:22:37.437Z", "content": "Woah..0.950+ LB for a single model..Great work!!\nOkay, I get the point for 'bigger is different'.\nThanks :)
", "rawMarkdown": "Woah..0.950+ LB for a single model..Great work!!\nOkay, I get the point for 'bigger is different'.\nThanks :)", "votes": 1 } ] }, { "id": 895383, "postDate": "2020-06-21T09:54:55.863Z", "content": "Great explanation.really helpfull👍
", "rawMarkdown": "Great explanation.really helpfull👍 ", "votes": 1 }, { "id": 895199, "postDate": "2020-06-21T07:06:17.023Z", "content": "Thanks, @cdeotte !\nIt is something new to me and this is going to help a lot. 💯
", "rawMarkdown": "Thanks, @cdeotte !\nIt is something new to me and this is going to help a lot. 💯", "votes": 1 }, { "id": 894542, "postDate": "2020-06-20T14:09:49.277Z", "content": "Hi @cdeotte, thanks again for your great explanations. I am planning to enter this competition now, but I don't have a lot of disk space left for the original Dataset ... could I use the datasets that you created resized without losing any information?
", "rawMarkdown": "Hi @cdeotte, thanks again for your great explanations. I am planning to enter this competition now, but I don't have a lot of disk space left for the original Dataset ... could I use the datasets that you created resized without losing any information? ", "votes": 1, "replies": [ { "id": 894559, "postDate": "2020-06-20T14:24:21.560Z", "content": "I can confirm that using only my five datasets (resized 768, 512, 512ext, 314, 256), one can achieve LB 0.955+
\n\nSome of the original images are huge like 4000x3000, perhaps later someone may find a way to use all the information in the high resolution original, but I am currently not.
", "rawMarkdown": "I can confirm that using only my five datasets (resized 768, 512, 512ext, 314, 256), one can achieve LB 0.955+\n\nSome of the original images are huge like 4000x3000, perhaps later someone may find a way to use all the information in the high resolution original, but I am currently not.", "votes": 6 }, { "id": 896050, "postDate": "2020-06-21T19:27:45.400Z", "content": "I can confirm that using only 2 datasets 512 and 512 ext I got LB 0.946.\nCurrently being at 1st position but still pushing everyone fr better LB shows true competitiveness. Thank you @cdeotte for the dataset and explanation.
", "rawMarkdown": "I can confirm that using only 2 datasets 512 and 512 ext I got LB 0.946.\nCurrently being at 1st position but still pushing everyone fr better LB shows true competitiveness. Thank you @cdeotte for the dataset and explanation.", "votes": 3 }, { "id": 896246, "postDate": "2020-06-22T02:38:31.937Z", "content": "Great work Aptha
", "rawMarkdown": "Great work Aptha", "votes": 1 }, { "id": 905823, "postDate": "2020-06-28T20:39:45.743Z", "content": "@apthagowda are you using Efficientnet or some other model ? It would be nice if you share by using which model you got LB 0.946.
", "rawMarkdown": "@apthagowda are you using **Efficientnet** or some other model ? It would be nice if you share by using which model you got LB 0.946." } ] }, { "id": 894178, "postDate": "2020-06-20T07:57:16.420Z", "content": "nice explanation
", "rawMarkdown": "nice explanation", "votes": 1 }, { "id": 894125, "postDate": "2020-06-20T07:29:48.907Z", "content": "Nice Explanation👍
", "rawMarkdown": "Nice Explanation👍 ", "votes": 1 }, { "id": 894073, "postDate": "2020-06-20T06:45:28.773Z", "content": "Thank you, your sharing was really helpful! A small question, your score is currently impressive, it is the result of a single model using one type of image size, or a single model using multiple image sizes then ensemble as you mentioned. in the discussion?
", "rawMarkdown": "Thank you, your sharing was really helpful! A small question, your score is currently impressive, it is the result of a single model using one type of image size, or a single model using multiple image sizes then ensemble as you mentioned. in the discussion?", "votes": 1, "replies": [ { "id": 894482, "postDate": "2020-06-20T13:14:40.417Z", "content": "My current LB score of 0.959, is an ensemble of separate models where one model uses 1024x1024, one uses 768x768, one 512x512 plain, one 512x512 external, one 384x384, and one 256x256. And some or all use meta data.
", "rawMarkdown": "My current LB score of 0.959, is an ensemble of separate models where one model uses 1024x1024, one uses 768x768, one 512x512 plain, one 512x512 external, one 384x384, and one 256x256. And some or all use meta data.", "votes": 8 }, { "id": 894593, "postDate": "2020-06-20T14:57:13.700Z", "content": "Great job! Those models all use the meta feature and and 70k images datasets?
", "rawMarkdown": "Great job! Those models all use the meta feature and and 70k images datasets?", "votes": 1 }, { "id": 894603, "postDate": "2020-06-20T15:10:17.247Z", "content": "I updated my comment. Some models train without external nor meta data. And others include external and/or meta data.
\n\nMy best single model trains on a little of everything and scores LB 0.950+ by itself.
", "rawMarkdown": "I updated my comment. Some models train without external nor meta data. And others include external and/or meta data.\n\nMy best single model trains on a little of everything and scores LB 0.950+ by itself.", "votes": 1 }, { "id": 894705, "postDate": "2020-06-20T17:11:05.107Z", "content": "Thanks again, Chris!
", "rawMarkdown": "Thanks again, Chris!" }, { "id": 897056, "postDate": "2020-06-22T15:39:21.353Z", "content": "\n\n\nI updated my comment. Some models train without external nor meta data. And others include external and/or meta data.
\n
I suppose that you are not using meta data in those models, which are using techniques link mixup and cutmix?
", "rawMarkdown": "> I updated my comment. Some models train without external nor meta data. And others include external and/or meta data.\n\nI suppose that you are not using meta data in those models, which are using techniques link mixup and cutmix?" } ] }, { "id": 967048, "postDate": "2020-08-11T21:34:10.743Z", "content": "A quick question about image sizes and submission averaging. Maybe I'm missing something very obvious for masters here, but does it makes any sense to averages all models trained over different sizes or is it better to average the averages?
\nLike let's sat we have a 5-fold B5 trained on image size 384 and 3-fold B6 trained on image size 512, different models may capture different features in the images. Would it make sense to first average over folds and then average the resulting submissions or maybe it makes sense to average all generated submissions … maybe weighting by the number of folds per size or OOF CV for every model, or this is still where heuristics plays the major role? 🙂
", "rawMarkdown": "A quick question about image sizes and submission averaging. Maybe I'm missing something very obvious for masters here, but does it makes any sense to averages all models trained over different sizes or is it better to average the averages? \n\nLike let's sat we have a 5-fold B5 trained on image size 384 and 3-fold B6 trained on image size 512, different models may capture different features in the images. Would it make sense to first average over folds and then average the resulting submissions or maybe it makes sense to average all generated submissions ... maybe weighting by the number of folds per size or OOF CV for every model, or this is still where heuristics plays the major role? 🙂", "votes": 2, "replies": [ { "id": 967088, "postDate": "2020-08-11T23:07:34.747Z", "content": "I suggest for each model, you combine the folds into a single OOF set of predictions and a single LB submission.csv file. Next use the OOF from each model and find weights to maximize the new_OOF = w1 * OOF_1 + w2 * OOF_2. Then use those weights w1, w2 to blend your two submission files.
Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 930022, "postDate": "2020-07-15T06:26:58.237Z", "content": "Thank you very much, this makes it much easier to understand!
", "rawMarkdown": "Thank you very much, this makes it much easier to understand!", "votes": 2 }, { "id": 919716, "postDate": "2020-07-08T04:32:05.773Z", "content": "👍
", "rawMarkdown": "👍", "votes": 2 }, { "id": 916881, "postDate": "2020-07-06T05:03:24.223Z", "content": "thanks @cdeotte \nmay i know that how do you use tf.data.Dataset to do multi-inputs ?\ni just put it into a list [dataset1,dataset2...]\nand got an error \nValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {\"<class 'tensorflow.python.data.ops.dataset_ops.MapDataset'>\"}), <class 'NoneType'>
There are 3 ways
\n\nThe easiest (to code) way is to make two different models and ensemble them
\n\nThe next way is to just input images of size 512x512. And then reduce the size inside the CNN and branch to two backbones.
\n\nbig = tf.keras.layers.Input(shape=(512,512,3))\nefn1 = efn.EfficientNetB4(input_shape=(512,512,3), \n weights='imagenet', include_top=False)\nx1 = efn1(big); x1 = tf.keras.layers.GlobalAveragePooling2D()(x1)\n\nsmall = tf.keras.layers.AveragePooling2D(pool_size=2, strides=2)(big)\nefn2 = efn.EfficientNetB4(input_shape=(256,256,3), \n weights='imagenet', include_top=False)\nx2 = efn2(small); x2 = tf.keras.layers.GlobalAveragePooling2D()(x2)\n### YOU CAN INSERT LEARNING RATE REDUCTION HERE ###\n\nx = tf.keras.layers.Concatenate()([x1,x2])\nx = tf.keras.layers.Dense(1,activation='sigmoid')(x)\nIf the 256x256 backbone trains too fast, you may need to make its learning rate smaller than the other backbone using x2 = WGT * x2 + (1-WGT)* tf.backend.stop_gradient(x2).
The third way (and less efficient use of RAM) is to input two datasets zipped together:
\n\nGCS_PATH_1 = KaggleDatasets().get_gcs_path('melanoma-256x256')\ntrain_1 = tf.io.gfile.glob(GCS_PATH_1 + '/train*.tfrec')\nds_train_1 = get_image_dataset(train_1)\n\nGCS_PATH_2 = KaggleDatasets().get_gcs_path('melanoma-512x512') \ntrain_2 = tf.io.gfile.glob(GCS_PATH_2 + '/train*.tfrec')\nds_train_2 = get_image_dataset(train_2)\n\nds_image = tf.data.Dataset.zip((ds_train_1, ds_train_2))\nds_label = get_label_dataset(train_1)\nds = tf.data.Dataset.zip((ds_image, ds_label))\nthank you.\nCurrently, I am training separate models, then freeze them to get the features and put them into CNN stacking model. That is the reason why i wanna do multi-inputs. Export the feature one by one take too long
", "rawMarkdown": "thank you.\nCurrently, I am training separate models, then freeze them to get the features and put them into CNN stacking model. That is the reason why i wanna do multi-inputs. Export the feature one by one take too long", "votes": 1 }, { "id": 921065, "postDate": "2020-07-09T04:04:07.060Z", "rawMarkdown": "", "isDeleted": true }, { "id": 921900, "postDate": "2020-07-09T16:30:51.413Z", "content": "Good question. I guess i meant easiest to code and debug. But they really aren't that different. I'm updating my post to just call them 1, 2, 3. Thanks.
\n\n(For those interested, i originally called 1, 2, 3 as easy, medium, hard).
", "rawMarkdown": "Good question. I guess i meant easiest to code and debug. But they really aren't that different. I'm updating my post to just call them 1, 2, 3. Thanks.\n\n(For those interested, i originally called 1, 2, 3 as easy, medium, hard)." }, { "id": 921914, "postDate": "2020-07-09T16:45:06.530Z", "rawMarkdown": "", "isDeleted": true }, { "id": 924965, "postDate": "2020-07-11T18:33:15.097Z", "content": "Dude, you're the best!
", "rawMarkdown": "Dude, you're the best!", "votes": 1 }, { "id": 926449, "postDate": "2020-07-12T18:20:53.497Z", "content": "Does anyone know how to name the EfficientNet layers? When trying to use two or more EfficientNetB4 layers such as in Method 2, I get ValueError:
\n\n\n", "rawMarkdown": "Does anyone know how to name the EfficientNet layers? When trying to use two or more EfficientNetB4 layers such as in Method 2, I get ValueError:\n\n> ValueError: The name \"efficientnet-b3\" is used 3 times in the model. All layer names should be unique." }, { "id": 927569, "postDate": "2020-07-13T13:40:42.393Z", "content": "ValueError: The name \"efficientnet-b3\" is used 3 times in the model. All layer names should be unique.
\n
Same problem. Did you find an answer?
", "rawMarkdown": "Same problem. Did you find an answer?" }, { "id": 927708, "postDate": "2020-07-13T14:56:25.830Z", "content": "Nope, not really. I think I'm gonna go with zipping the datasets?
", "rawMarkdown": "Nope, not really. I think I'm gonna go with zipping the datasets?", "votes": 1 }, { "id": 934706, "postDate": "2020-07-18T16:56:46.750Z", "content": "Try:
\n\n\nfor i, layer in enumerate(eff_base.layers):\n layer._name = layer.name + '2'\n
Thanks for the different approaches @cdeotte Could you explain how to come up with WGT in Method 2. Or what would be probable values in the example above?
Thanks for sharing.
\nI'm curious about two questions:
\nIn Method 2 we use two backbone with different input size, then what input size we should choose when inference.
I find you use 'small = tf.keras.layers.AveragePooling2D(pool_size=2, strides=2)(big)' for resize 512 to 256. Can we use OpenCVor Albumentation for image's resizing instead of Pooling? What's the different of them.
Looking forward to your answer.😄
", "rawMarkdown": "Thanks for sharing.\n\nI'm curious about two questions:\n\n1. In Method 2 we use two backbone with different **input size**, then what **input size** we should choose when inference.\n\n2. I find you use 'small = tf.keras.layers.AveragePooling2D(pool_size=2, strides=2)(big)' for resize 512 to 256. Can we use **OpenCV**or **Albumentation** for image's resizing instead of **Pooling**? What's the different of them.\n\nLooking forward to your answer.😄" }, { "id": 1495384, "postDate": "2021-08-29T14:14:20.130Z", "content": "Method 2 only has 1 input,
\nbig = tf.keras.layers.Input(shape=(512,512,3))
\nthe variable small is not an input
small = tf.keras.layers.AveragePooling2D(pool_size=2, strides=2)(big)
\nSo during inference, we use the one input and input 512x512.
\nWe could modify method 2 to have 2 inputs, and then we can use OpenCV to resize as you suggest. When using OpenCV, we choose a technique to use for resize. For example, you can use interpolation = cv.INTER_CUBIC or cv.INTER_LINEAR or etc. Using pooling is just one of these techniques. Different resize algorithms have different pros and cons.
Thank you, this really helps.
", "rawMarkdown": "Thank you, this really helps.", "votes": 1 } ] }, { "id": 904976, "postDate": "2020-06-28T06:24:00.937Z", "content": "interesting insights, very relatable to real world applications
", "rawMarkdown": "interesting insights, very relatable to real world applications", "votes": 2 }, { "id": 904855, "postDate": "2020-06-28T03:01:50.607Z", "content": "Great explanation! This is my first time working with pre-trained models and never occurred to me how input sizes could affect them.
", "rawMarkdown": "Great explanation! This is my first time working with pre-trained models and never occurred to me how input sizes could affect them.", "votes": 2, "replies": [ { "id": 911792, "postDate": "2020-07-02T02:57:12.813Z", "content": "Transfer learning is an exciting cutting edge topic for both image classification and NLP. Preprocessing your data for transfer learning makes a big difference in the result of transfer learning.
", "rawMarkdown": "Transfer learning is an exciting cutting edge topic for both image classification and NLP. Preprocessing your data for transfer learning makes a big difference in the result of transfer learning.", "votes": 1 } ] }, { "id": 902197, "postDate": "2020-06-26T01:38:04.020Z", "content": "Wow.. great explanation on the importance of input size variation 👍 👍
", "rawMarkdown": "Wow.. great explanation on the importance of input size variation 👍 👍 ", "votes": 2 }, { "id": 902104, "postDate": "2020-06-26T00:02:14.997Z", "content": "Wow! Great Explanation!
", "rawMarkdown": "Wow! Great Explanation!", "votes": 2 }, { "id": 898171, "postDate": "2020-06-23T11:02:06.070Z", "content": "Thanks a lot for your valuable information! I have one question. You made various sizes of the competition images. However, you made only one size (512x512) of external data. Why you didn't make other size of external data? I guess other size of external data, such as 768x768, 384x384 etc., might also be useful.
", "rawMarkdown": "Thanks a lot for your valuable information! I have one question. You made various sizes of the competition images. However, you made only one size (512x512) of external data. Why you didn't make other size of external data? I guess other size of external data, such as 768x768, 384x384 etc., might also be useful.", "votes": 2, "replies": [ { "id": 898480, "postDate": "2020-06-23T14:50:39.690Z", "content": "Good point. That would be useful. So far i have not used different sizes for external data but it could help. Also sizes 128x128, 192x192 for the competition data could help too (but i have not tried these sizes either).
", "rawMarkdown": "Good point. That would be useful. So far i have not used different sizes for external data but it could help. Also sizes 128x128, 192x192 for the competition data could help too (but i have not tried these sizes either).", "votes": 2 }, { "id": 899639, "postDate": "2020-06-24T10:53:45.283Z", "content": "Thanks for your reply! So, I will try to make other sizes of external data and make sure they are really useful.
", "rawMarkdown": "Thanks for your reply! So, I will try to make other sizes of external data and make sure they are really useful." }, { "id": 900773, "postDate": "2020-06-25T03:52:39.140Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 896576, "postDate": "2020-06-22T09:22:20.657Z", "content": "nice work! have a great insight about resizing image! thankyou Chris.
", "rawMarkdown": "nice work! have a great insight about resizing image! thankyou Chris.", "votes": 2 }, { "id": 893905, "postDate": "2020-06-20T04:33:15.833Z", "content": "If the orignal size is 4000*6000, resize to 512*512, then it can't find circle and triangles?
", "rawMarkdown": "If the orignal size is 4000*6000, resize to 512*512, then it can't find circle and triangles?", "votes": 2, "replies": [ { "id": 893958, "postDate": "2020-06-20T05:08:40.887Z", "content": "No. The point is that a pretrained CNN will find different things with different input sizes. A CNN looks for thousands of patterns. It will find some patterns with some input sizes and other patterns with other input sizes. Therefore you should experiment with different sizes to find which produces the best CV and LB. Also consider ensembling the different sized models for maximum CV and LB.
", "rawMarkdown": "No. The point is that a pretrained CNN will find different things with different input sizes. A CNN looks for thousands of patterns. It will find some patterns with some input sizes and other patterns with other input sizes. Therefore you should experiment with different sizes to find which produces the best CV and LB. Also consider ensembling the different sized models for maximum CV and LB.", "votes": 6 }, { "id": 894042, "postDate": "2020-06-20T06:05:50.583Z", "content": "Thanks, I saw \"The original jpegs have been center cropped and then resized using cv2.resize with interpolation = cv2.INTER_AREA.\" no radio change
", "rawMarkdown": "Thanks, I saw \"The original jpegs have been center cropped and then resized using cv2.resize with interpolation = cv2.INTER_AREA.\" no radio change" } ] }, { "id": 3234396, "postDate": "2025-06-27T19:03:59.710Z", "content": "so simple and useful explanation, thank you !
", "rawMarkdown": "so simple and useful explanation, thank you !" }, { "id": 3183687, "postDate": "2025-04-21T06:57:22.207Z", "content": "Cool blog. I struggled with convolutions until now thanks to you
", "rawMarkdown": "Cool blog. I struggled with convolutions until now thanks to you" }, { "id": 3107956, "postDate": "2025-01-27T06:56:49.770Z", "content": "Thanks for the information !! That was really helpful!!
", "rawMarkdown": "Thanks for the information !! That was really helpful!!" }, { "id": 2559678, "postDate": "2023-12-13T02:44:25.123Z", "content": "This, this right here, is info about computer vision explained so gracefully even a 2nd grader can understand. You are a great teacher @cdeotte
", "rawMarkdown": "This, this right here, is info about computer vision explained so gracefully even a 2nd grader can understand. You are a great teacher @cdeotte " }, { "id": 967341, "postDate": "2020-08-12T07:09:44.350Z", "content": "As far as I understand the model learns from the different sizes, and the resultant will be a combination of every size it has learned. Here will it be any change or improvement if the same model is retrained with different sizes, that is first train with some size, save it then train on different size; will the model now be any better than only trained on same image size?
", "rawMarkdown": "As far as I understand the model learns from the different sizes, and the resultant will be a combination of every size it has learned. Here will it be any change or improvement if the **same** model is retrained with different sizes, that is first train with some size, save it then train on different size; will the model now be any better than only trained on same image size?" }, { "id": 937640, "postDate": "2020-07-21T05:40:04.010Z", "content": "Very nice idea,
\nAs I was working on pytorch, we cant set None to the size of the image,
\nSo has anyone implemented this idea on pytorch?
@cdeotte thanks for all great discussions and kernels. I have one question.
\n\nIf a model is performing well on small size image dataset, can we say it will also perform better on large size image. I am asking this question from experimental point of view. For example, if I train effnet-b6 on 128x128 image size and after doing some experiments, I got 0.92 AUC so in this case can I assume that the same model will also give better results on 512x512 image size with same configuration(folds, epochs, loss, optimzer etc.) or there is a possibility that it can give better results with different configurations on different image size datasets?
Great. Trying 1024x1024 but cannot predict.
", "rawMarkdown": "Great. Trying 1024x1024 but cannot predict.", "replies": [ { "id": 927594, "postDate": "2020-07-13T13:57:16.303Z", "content": "What do you mean \"cannot predict\"?
", "rawMarkdown": "What do you mean \"cannot predict\"?" } ] }, { "id": 925004, "postDate": "2020-07-11T19:10:42.260Z", "content": "Thanks @cdeotte.\nI have a stupid question! What happens if I use 512x512 or 768x768 images but I set the CNN input_shape to 224x224?
", "rawMarkdown": "Thanks @cdeotte.\nI have a stupid question! What happens if I use 512x512 or 768x768 images but I set the CNN input_shape to 224x224?", "replies": [ { "id": 925007, "postDate": "2020-07-11T19:12:08.410Z", "content": "You will get an error. But you can set input_shape = (None,None,3) and then you can put in whatever size at whatever time. Then batch 1 can have one size and batch 2 can have a different size! Then for TTA, you can input each test image multiple times with different sizes and average the predictions. (Not sure if this is a good idea, but it can be done).
This is a piece of new knowledge to me. Thanks @cdeotte
", "rawMarkdown": "This is a piece of new knowledge to me. Thanks @cdeotte ", "votes": 1 }, { "id": 935135, "postDate": "2020-07-19T05:21:55.123Z", "content": "Thanks for this valuable info @cdeotte. Actually I am using Pytorch where I cant set my input shape None. So has anyone implemented this great idea in pytorch?
", "rawMarkdown": "Thanks for this valuable info @cdeotte. Actually I am using Pytorch where I cant set my input shape None. So has anyone implemented this great idea in pytorch?" } ] }, { "id": 902708, "postDate": "2020-06-26T10:08:08.993Z", "content": "Hi, firstly, I resizeed all the images to 1200x960, then I randomcrop the images to 224x224x3 for training, do you random crop the images for your training or just input the resize images instead of random-croped ones? Thank you!!!
", "rawMarkdown": "Hi, firstly, I resizeed all the images to 1200x960, then I randomcrop the images to 224x224x3 for training, do you random crop the images for your training or just input the resize images instead of random-croped ones? Thank you!!!", "replies": [ { "id": 903748, "postDate": "2020-06-27T04:44:34.133Z", "content": "Both methods have pros and cons. Try both and see which has the best CV LB
", "rawMarkdown": "Both methods have pros and cons. Try both and see which has the best CV LB" }, { "id": 908036, "postDate": "2020-06-30T10:28:23.623Z", "content": "Hi@cdeotte, could you tell what centercrop method in your dateset? I used albumentations centercrop func and found image lost much info. Thanks
", "rawMarkdown": "Hi[@cdeotte](https://www.kaggle.com/cdeotte), could you tell what centercrop method in your dateset? I used albumentations centercrop func and found image lost much info. Thanks" } ] }, { "id": 901956, "postDate": "2020-06-25T20:14:35.147Z", "content": "new question - what do you do with images originaly smaller than the target size? I mean 640x480 images, do you upscale them or fill them with border?
", "rawMarkdown": "new question - what do you do with images originaly smaller than the target size? I mean 640x480 images, do you upscale them or fill them with border?", "replies": [ { "id": 902097, "postDate": "2020-06-25T23:47:50.070Z", "content": "Currently, my datasets were created by center cropping (with a square) all images and then resize. So in the case of 640x480, i center cropped a square of 480x480 and upsampled to 512 and 768. And downsampled to 384 and 256. This is working well so far, but perhaps there are better ways like you suggest.
", "rawMarkdown": "Currently, my datasets were created by center cropping (with a square) all images and then resize. So in the case of 640x480, i center cropped a square of 480x480 and upsampled to 512 and 768. And downsampled to 384 and 256. This is working well so far, but perhaps there are better ways like you suggest.", "votes": 3 } ] }, { "id": 900666, "postDate": "2020-06-25T02:11:59.160Z", "content": "@cdeotte . If i am understanding your comments below right, you are saying you have a single model using one of these (256, 384, 512, and 784) image sizes giving 0.95+ LB? and an ensemble of different models (b0,b1,b2, etc) trained on different size images gives you 0.955+LB?\nI must be doing some really wrong, i cannot get anything above 0.91 for a single model using any of these image sizes on a groupKFold using Efficientnet1. I suspect my CV set up is contributing to this low score. I should try a different CV or maybe ignore those multiple patient images and train on images without those outliers rather than trying to compensate for this multiple images per patient scenario using groupKfold and all that.
", "rawMarkdown": "@cdeotte . If i am understanding your comments below right, you are saying you have a single model using one of these (256, 384, 512, and 784) image sizes giving 0.95+ LB? and an ensemble of different models (b0,b1,b2, etc) trained on different size images gives you 0.955+LB?\nI must be doing some really wrong, i cannot get anything above 0.91 for a single model using any of these image sizes on a groupKFold using Efficientnet1. I suspect my CV set up is contributing to this low score. I should try a different CV or maybe ignore those multiple patient images and train on images without those outliers rather than trying to compensate for this multiple images per patient scenario using groupKfold and all that. ", "replies": [ { "id": 901559, "postDate": "2020-06-25T15:07:27.813Z", "content": "@cdeotte i guess you missed my post :-) Thank you in advance for any response.🙏
", "rawMarkdown": "@cdeotte i guess you missed my post :-) Thank you in advance for any response.🙏 " } ] }, { "id": 898789, "postDate": "2020-06-23T18:38:18.517Z", "content": "Will using dilated convolutions to process inputs in higher resolutions and capturing broader information work in this scenario? Can this speed up training as it will skip every adjacent pixel and create a more generalized model?
", "rawMarkdown": "Will using dilated convolutions to process inputs in higher resolutions and capturing broader information work in this scenario? Can this speed up training as it will skip every adjacent pixel and create a more generalized model? " }, { "id": 897892, "postDate": "2020-06-23T07:03:58.577Z", "content": "\"Therefore you should experiment with different input sizes to see which has better CV and LB.\" Any thumb rule to hit sweet spot?
", "rawMarkdown": "\"Therefore you should experiment with different input sizes to see which has better CV and LB.\" Any thumb rule to hit sweet spot?" }, { "id": 897789, "postDate": "2020-06-23T05:14:20.260Z", "content": "hello @cdeotte
\n\ncould you explain \"external data\"?
\n\nI have read Official External Data Thread and I have seen older ISIC datasets mentioned by @shonenkov and @andrewmvd but by quickly looking on them I don't really understand how they are used, is there somewhere the code how can you adapt those datasets to this competition?
", "rawMarkdown": "hello @cdeotte \n\ncould you explain \"external data\"? \n\nI have read Official External Data Thread and I have seen older ISIC datasets mentioned by @shonenkov and @andrewmvd but by quickly looking on them I don't really understand how they are used, is there somewhere the code how can you adapt those datasets to this competition?", "replies": [ { "id": 898478, "postDate": "2020-06-23T14:48:15.937Z", "content": "I think Kagglers have posted a few external datasets. The one that I'm familiar with and the most popular is Alex's. It is described here. It includes data from 4 external datasets. It has a total of about 60k extra images.
\n\nAll adaptation is already done. Just include Alex's jpegs here or my TFRecords here. Each external image has a target 0 or 1 just like this competition. Also the external images have meta data just like this competition.
", "rawMarkdown": "I think Kagglers have posted a few external datasets. The one that I'm familiar with and the most popular is Alex's. It is described [here][1]. It includes data from 4 external datasets. It has a total of about 60k extra images.\n\nAll adaptation is already done. Just include Alex's jpegs [here][2] or my TFRecords [here][3]. Each external image has a target 0 or 1 just like this competition. Also the external images have meta data just like this competition.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155859\n[2]: https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg\n[3]: https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images", "votes": 2 }, { "id": 898707, "postDate": "2020-06-23T17:23:07.940Z", "content": "thank you for the link- yes, that's what I needed\nhowever I can't use his or yours dataset, because you have 512x512 images, I need more flexibility
", "rawMarkdown": "thank you for the link- yes, that's what I needed\nhowever I can't use his or yours dataset, because you have 512x512 images, I need more flexibility", "votes": 1 }, { "id": 898753, "postDate": "2020-06-23T17:57:22.260Z", "content": "One easy trick it to load 512x512 into NumPy array and then use image[::2,::2]. That gives you 256x256 without any preprocess time.
Or add a new input layer to your NN where you pool 2x2 stride 2. Then your CNN gets 256x256
", "rawMarkdown": "One easy trick it to load 512x512 into NumPy array and then use `image[::2,::2]`. That gives you 256x256 without any preprocess time.\n\nOr add a new input layer to your NN where you pool 2x2 stride 2. Then your CNN gets 256x256", "votes": 4 } ] }, { "id": 897573, "postDate": "2020-06-23T00:57:47.937Z", "content": "Excelente explicação. Obrigada!
", "rawMarkdown": "Excelente explicação. Obrigada!" }, { "id": 894311, "postDate": "2020-06-20T10:26:21.750Z", "content": "Thanks a lot for sharing. One question for the input size to your model. I used 224x224 (no external data) , B2-eff net, 5 fold CV, it takes me ~3-4 hrs to complete on GPU. imaging that for 512 x512 with external data sets, the time taken would be very significant thought i haven't tried yet. Question, do you directly feed 512x512 to your model or you will don some augmentation to make input image size smaller? Like bounding box or center circle?
", "rawMarkdown": "Thanks a lot for sharing. One question for the input size to your model. I used 224x224 (no external data) , B2-eff net, 5 fold CV, it takes me ~3-4 hrs to complete on GPU. imaging that for 512 x512 with external data sets, the time taken would be very significant thought i haven't tried yet. Question, do you directly feed 512x512 to your model or you will don some augmentation to make input image size smaller? Like bounding box or center circle?", "replies": [ { "id": 894494, "postDate": "2020-06-20T13:35:43.553Z", "content": "If you use a single GPU, you will need to use additional tricks like you discuss.
\n\nFor single GPU, if you wish to do five fold, you can consider trying a variety of smaller sizes 320x320, 256x256, 224x224, 128x128, 96x96, 64x64. Also you can do what you say. Resize the original image to 512x512, then use your data loader to crop to 256x256 where you intelligently crop an important part of the image.
\n\nAlso instead of 5 Fold, you can use a single validation holdout. For example train on a random 80% and validate on 20%. If you do this then using 512x512 on a single GPU should finish in 3-4 hours just like your 224x224 five fold.
\n\n(Note if you have access to multiple GPU, then you can use them together to train any input size quickly).
", "rawMarkdown": "If you use a single GPU, you will need to use additional tricks like you discuss. \n\nFor single GPU, if you wish to do five fold, you can consider trying a variety of smaller sizes 320x320, 256x256, 224x224, 128x128, 96x96, 64x64. Also you can do what you say. Resize the original image to 512x512, then use your data loader to crop to 256x256 where you intelligently crop an important part of the image. \n\nAlso instead of 5 Fold, you can use a single validation holdout. For example train on a random 80% and validate on 20%. If you do this then using 512x512 on a single GPU should finish in 3-4 hours just like your 224x224 five fold.\n\n(Note if you have access to multiple GPU, then you can use them together to train any input size quickly).", "votes": 2 }, { "id": 894506, "postDate": "2020-06-20T13:42:58.847Z", "content": "Thank you Chris! This is my first try on Kaggle, so excited to get advices from a kaggle master like you😃
", "rawMarkdown": "Thank you Chris! This is my first try on Kaggle, so excited to get advices from a kaggle master like you😃 " }, { "id": 894522, "postDate": "2020-06-20T13:53:15.273Z", "content": "Welcome Zhu Chao. Using different sizes to your advantage can be done later. For now, you can continue to focus on your 224x224 model. Keep experimenting with data augmentation, model architecture, loss, learning schedules, meta features, etc and maximize the CV score.
\n\nLater when you want an additional boost on CV and LB, you can use your same model with input sizes 128x128, 256x256, 384x384, 512x512. And ensemble the 4 sets of predictions.
", "rawMarkdown": "Welcome Zhu Chao. Using different sizes to your advantage can be done later. For now, you can continue to focus on your 224x224 model. Keep experimenting with data augmentation, model architecture, loss, learning schedules, meta features, etc and maximize the CV score.\n\nLater when you want an additional boost on CV and LB, you can use your same model with input sizes 128x128, 256x256, 384x384, 512x512. And ensemble the 4 sets of predictions.", "votes": 7 } ] }, { "id": 894225, "postDate": "2020-06-20T08:43:44.417Z", "content": "Hey Chris, this is really amazing info you have shared here, I definitely want to go with this.
\n\nI have one query though, I don't have much experience of working with TPU's, so is it possible to use these TfRecords with GPU? I may go with creating datasets of different image sizes but it may eat up all my GPU quota of the week!
", "rawMarkdown": "Hey Chris, this is really amazing info you have shared here, I definitely want to go with this. \n\nI have one query though, I don't have much experience of working with TPU's, so is it possible to use these TfRecords with GPU? I may go with creating datasets of different image sizes but it may eat up all my GPU quota of the week!", "replies": [ { "id": 894261, "postDate": "2020-06-20T09:21:08.740Z", "content": "You don't have to enable GPU to resize images :)
", "rawMarkdown": "You don't have to enable GPU to resize images :)" }, { "id": 894264, "postDate": "2020-06-20T09:24:35.543Z", "content": "Hey Gilles, Yeah I did so, I forgot, its just that I got an error that time saying \"your notebook tried to write more than the output size available\", maybe I should just break the data into parts and then convert. :/
", "rawMarkdown": "Hey Gilles, Yeah I did so, I forgot, its just that I got an error that time saying \"your notebook tried to write more than the output size available\", maybe I should just break the data into parts and then convert. :/ " }, { "id": 894267, "postDate": "2020-06-20T09:28:20.137Z", "content": "Yeah exactly, Kaggle allows only a max output size of 5 GB, so you will have to divide the data into chunks for larger sizes (or do it locally)
", "rawMarkdown": "Yeah exactly, Kaggle allows only a max output size of 5 GB, so you will have to divide the data into chunks for larger sizes (or do it locally)", "votes": 1 }, { "id": 894270, "postDate": "2020-06-20T09:32:47.820Z", "content": "I will go with the dividing method, my system lack both computing power and memory. 😄
\n\nThanks for the help though! ^_^
", "rawMarkdown": "I will go with the dividing method, my system lack both computing power and memory. 😄 \n\nThanks for the help though! ^_^", "votes": 1 }, { "id": 894283, "postDate": "2020-06-20T09:45:14.063Z", "content": "Hey, btw, Can I use TPU to resize images in any way? My TPU quota goes unused so I can use it here if possible and it will of course give a lot more speed.
", "rawMarkdown": "Hey, btw, Can I use TPU to resize images in any way? My TPU quota goes unused so I can use it here if possible and it will of course give a lot more speed." }, { "id": 894291, "postDate": "2020-06-20T09:53:25.770Z", "content": "Not sure how much faster that would go. The bottleneck is reading/writing from/to disk and not resizing the images I think.
", "rawMarkdown": "Not sure how much faster that would go. The bottleneck is reading/writing from/to disk and not resizing the images I think.", "votes": 1 }, { "id": 894293, "postDate": "2020-06-20T09:58:47.917Z", "content": "Sure, thanks again! :)
", "rawMarkdown": "Sure, thanks again! :)" }, { "id": 894496, "postDate": "2020-06-20T13:38:45.137Z", "content": "Gilles is correct. Turn the GPU/TPU off when making TFRecords (because the bottle neck is reading and writing to disk which GPU/TPU doesn't speed up). I posted a starter notebook here that makes TFRecords using only the CPU.
", "rawMarkdown": "Gilles is correct. Turn the GPU/TPU **off** when making TFRecords (because the bottle neck is reading and writing to disk which GPU/TPU doesn't speed up). I posted a starter notebook [here][1] that makes TFRecords using only the CPU.\n\n[1]: https://www.kaggle.com/cdeotte/how-to-create-tfrecords", "votes": 4 } ] }, { "id": 3187005, "postDate": "2025-04-25T12:15:49.597Z", "rawMarkdown": "", "isDeleted": true }, { "id": 963095, "postDate": "2020-08-08T17:02:23.633Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 913230, "postDate": "2020-07-03T04:40:28.167Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 895059, "postDate": "2020-06-21T04:34:16.487Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 894683, "postDate": "2020-06-20T16:45:44.960Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 896245, "postDate": "2020-06-22T02:37:25.100Z", "content": "Great work dash. Continue to explore attention. I think that is a great idea. I agree that early in a competition it is best to develop a single model and maximize CV LB.
\n\nI don't want to encourage early ensembling with my post here, i just want to point out that different image sizes affect CV LB. And I wanted to provide an explaination.
\n\nLater in the comp an easy way to boost CV LB is to feed multiple image sizes into your model and ensemble the multiple predictions.
", "rawMarkdown": "Great work dash. Continue to explore attention. I think that is a great idea. I agree that early in a competition it is best to develop a single model and maximize CV LB.\n\nI don't want to encourage early ensembling with my post here, i just want to point out that different image sizes affect CV LB. And I wanted to provide an explaination. \n\nLater in the comp an easy way to boost CV LB is to feed multiple image sizes into your model and ensemble the multiple predictions.", "votes": 1 } ] }, { "id": 1154108, "postDate": "2021-01-15T12:07:04.253Z", "content": "thanks. It is helpful :)
", "rawMarkdown": "thanks. It is helpful :)", "votes": 1 }, { "id": 1110802, "postDate": "2020-12-13T04:53:16.757Z", "content": "Thank you Sir!!
", "rawMarkdown": "Thank you Sir!!", "votes": 1 }, { "id": 982398, "postDate": "2020-08-23T10:37:25.117Z", "content": "This itself is GOLD. Thanks!
", "rawMarkdown": "This itself is GOLD. Thanks!", "votes": 1 }, { "id": 972736, "postDate": "2020-08-16T19:10:14.940Z", "content": "Thanks :)
", "rawMarkdown": "Thanks :)", "votes": 1 }, { "id": 940397, "postDate": "2020-07-22T23:12:51.583Z", "content": "Thanks, this is very helpful!
", "rawMarkdown": "Thanks, this is very helpful!", "votes": 1 }, { "id": 933120, "postDate": "2020-07-17T14:00:47.753Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!", "votes": 1 }, { "id": 910072, "postDate": "2020-07-01T01:46:09.637Z", "content": "Thanks for the great explanation!
", "rawMarkdown": "Thanks for the great explanation!", "votes": 1 }, { "id": 907394, "postDate": "2020-06-29T23:45:56.377Z", "content": "Great explanation - thanks!
", "rawMarkdown": "Great explanation - thanks!", "votes": 1 }, { "id": 907310, "postDate": "2020-06-29T20:51:39.067Z", "content": "Great explanation - thanks!
", "rawMarkdown": "Great explanation - thanks!", "votes": 1 }, { "id": 906596, "postDate": "2020-06-29T12:43:23.347Z", "content": "eye opening post for me . Thanks
", "rawMarkdown": "eye opening post for me . Thanks", "votes": 1 }, { "id": 902088, "postDate": "2020-06-25T23:25:57.517Z", "content": "Great explanation, thanks!
", "rawMarkdown": "Great explanation, thanks!", "votes": 1 }, { "id": 900599, "postDate": "2020-06-24T23:27:02.343Z", "content": "Great explanation! thanks
", "rawMarkdown": "Great explanation! thanks", "votes": 1 }, { "id": 899963, "postDate": "2020-06-24T14:50:40.300Z", "content": "Thank you, this really helps.
", "rawMarkdown": "Thank you, this really helps. ", "votes": 1 }, { "id": 897846, "postDate": "2020-06-23T06:19:15.937Z", "content": "Thank you for your explains.
", "rawMarkdown": "Thank you for your explains. ", "votes": 1 }, { "id": 897139, "postDate": "2020-06-22T16:35:54.497Z", "content": "Nice Explanation . Thanks a lot
", "rawMarkdown": "Nice Explanation . Thanks a lot", "votes": 1 }, { "id": 3245480, "postDate": "2025-07-09T12:25:36.757Z", "content": "Really helpful!
", "rawMarkdown": "Really helpful!" }, { "id": 3189224, "postDate": "2025-04-29T01:36:18.697Z", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!" }, { "id": 897492, "postDate": "2020-06-22T22:32:25.240Z", "content": "Thanks for the explanation!!!
", "rawMarkdown": "Thanks for the explanation!!!" } ], "comments": [ { "id": 2185587, "author_name": "Yeakub Sadlil", "author_url": "", "post_date": "2023-03-17T07:13:58.820000", "content": "Your visualization is beautiful @cdeotte
", "votes": 4, "replies": [] }, { "id": 918961, "author_name": "", "author_url": "", "post_date": "2020-07-07T16:01:24.867000", "content": "you know what? this makes easier to understand than reddit!😂
", "votes": 11, "replies": [ { "id": 919756, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-08T05:05:32.130000", "content": "Thanks Jimmy
", "votes": 2, "replies": [] }, { "id": 919764, "author_name": "", "author_url": "", "post_date": "2020-07-08T05:09:41.473000", "content": "Already followed you👍
", "votes": 7, "replies": [] } ] }, { "id": 1685746, "author_name": "Kanlong", "author_url": "", "post_date": "2022-02-11T14:14:41.377000", "content": "thanks，from your PetFinder link , congratulations!
", "votes": 1, "replies": [] }, { "id": 895277, "author_name": "hengck23", "author_url": "", "post_date": "2020-06-21T08:13:54.457000", "content": "\n\nthanks for the post.\nwhich is more important, size of image image or depth (complexity) of model.
\n\nit seems difficult to train a deep efficientnetB7 on large size on local GPU and shallower efficientnetB0,1,2 doesn't benefit from large size?
", "votes": 5, "replies": [ { "id": 895942, "author_name": "", "author_url": "", "post_date": "2020-06-21T17:45:21.223000", "content": "Exactly!\nThis is what has put me in a dilemma.\nNot sure which is important?
", "votes": 0, "replies": [] }, { "id": 896220, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-22T01:46:35.867000", "content": "I don't know which is more \"important\". Note my post is not saying \"bigger is better\". My post is saying that \"bigger is different\". (Likewise different backbones (efn0, efn1, ..., efn7) are different. And deeper is not necessarily better in this comp because there are issues of overfitting. We don't have millions of training images like imagenet).
\n\nTry this experiment. Take any of your models even one with EfficientNetB0 and check the CV using sizes 96, 128, 192, 256, 384, 512, 768, 1024. You will notice that the CV score is different with different image sizes. That is what my post is saying.
", "votes": 10, "replies": [] }, { "id": 896242, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-22T02:28:54.060000", "content": "Also note that you can train deep models on large image sizes with limited compute resources by freezing bottom layers.
", "votes": 2, "replies": [] }, { "id": 896899, "author_name": "Satwik", "author_url": "", "post_date": "2020-06-22T14:08:43.340000", "content": "When you say bottom layers here, which ones does that mean exactly? Can you provide an example for EfficientNetB0?
", "votes": 0, "replies": [] }, { "id": 897021, "author_name": "", "author_url": "", "post_date": "2020-06-22T15:15:14.647000", "content": "@p4rallax , I think what @cdeotte is trying to say is that majority of parameters come from the linear layers. So freezing them and training would greatly decrease the computational cost. (for deep models)
", "votes": 0, "replies": [] }, { "id": 897365, "author_name": "Satwik", "author_url": "", "post_date": "2020-06-22T19:59:55.517000", "content": "I see , that makes sense. Thank you.
", "votes": 0, "replies": [] }, { "id": 903519, "author_name": "chris", "author_url": "", "post_date": "2020-06-26T22:52:49.840000", "content": "@p4rallax @rohan1602 I believe \"bottom layers\" refers to the early convolutional layers of the network, while the top typically refers to the final linear layers once spatial features are flattened. When training a network a majority of the memory usage comes from the storage of intermediate hidden representations that will be needed to compute gradients during backprop, rather than the parameters of the model. If you disable the gradient for the early parts of the forward pass (where inputs are large due to spatial dimensions) you will save alot of memory. This can be done because CNNs learn pretty general features and good performance can be achieved by only fine-tuning the final linear layers.
", "votes": 5, "replies": [] }, { "id": 903743, "author_name": "", "author_url": "", "post_date": "2020-06-27T04:41:39.467000", "content": "Oh yeah..might be correct..\nSorry..I have recently started Deep Learning..so might have misunderstood the terminology.\nThanks for clarifying :)
", "votes": 1, "replies": [] } ] }, { "id": 896018, "author_name": "Roman", "author_url": "", "post_date": "2020-06-21T18:49:17.807000", "content": "That is the one weird looking racing car you have there Chris.
", "votes": 6, "replies": [] }, { "id": 991203, "author_name": "", "author_url": "", "post_date": "2020-08-30T08:11:07.813000", "content": "I learnt a new lesson today from the discussion
", "votes": 3, "replies": [] }, { "id": 904543, "author_name": "itsshavar", "author_url": "", "post_date": "2020-06-27T17:45:33.060000", "content": "Hey Chris! You analysis always helps.
", "votes": 3, "replies": [] }, { "id": 901957, "author_name": "Syphax", "author_url": "", "post_date": "2020-06-25T20:16:45.100000", "content": "Nice Explanation! Thanks :) .As you can see, the images of melanomas are taken with different distances, which can give us false information regarding the real size of the melanoma. do you think it could affect the model ?
", "votes": 3, "replies": [ { "id": 902096, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-25T23:44:21.820000", "content": "Great point. It would be nice to extract this distance information and give it to our model as a meta feature.
", "votes": 1, "replies": [] } ] }, { "id": 900983, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-25T07:30:12.920000", "content": "Other than trying different image sizes, I started with trying different architectures for a image size of 224, I used a B0 and a Resnet50 arch for this work, as my intuition was that different archs may learn different features.
And, as to validate, I got 0.888 LB with only B0 while I got 0.895 with an ensemble of B0 and Resnet50.
\n\nAlso, I thought of doing this for more image sizes like 384, 512, 768, 1024 and then combining their results. But I think it would not be computationally efficient as it would take a lot of time to do that, what do you think of that, @cdeotte Should I go for it?
", "votes": 3, "replies": [ { "id": 901471, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-25T14:02:23.593000", "content": "Early in a competition, I suggest that you focus on one model and get the validation score as high as possible. Try different sizes and different model architectures. Also experiment with data augmentation, loss, optimizer, learning schedule. Later in the comp, you can input different sized images into your one model and ensemble the multiple predictions for a CV LB boost.
", "votes": 9, "replies": [] }, { "id": 901507, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-25T14:32:22.437000", "content": "Sure, thanks @cdeotte I will work on Hyper tuning then, actually its my first competition(to some extent), so I was eager to get up in the leaderboard and was being impatient. 😄
\n\nAlso, one more question, I ran my notebook once and submitted the predictions and got .888 LB, but when I again ran it, it only gave me .798, of course there was overfitting and I understand that I should save my models, but I didn't actually save them due to my ignorance.
\n\nSo what should I do in such a case to maintain stability in my models? I used same seed and everything was same too.
", "votes": 0, "replies": [] }, { "id": 901528, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-25T14:43:44.090000", "content": "This is the biggest challenge in this competition. With such an unbalanced target (only 1% positive), AUC metric is very volatile when training with cross entropy. You can stabilize your models with folds, bagging, and ensembles. But the best way is to improve the training process (to optimize AUC) but either modifying the loss or using other tricks.
", "votes": 3, "replies": [] }, { "id": 901543, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-25T14:49:42.987000", "content": "Sure, I am using folds and ensembles with tta as of now, I think many more things are needed here, I tried augmenting only the melanoma images but the model just got overfitted and was of no use.
", "votes": 1, "replies": [] }, { "id": 902203, "author_name": "Redwan Sony", "author_url": "", "post_date": "2020-06-26T01:46:21.717000", "content": "great insight..
", "votes": 2, "replies": [] }, { "id": 903753, "author_name": "FadzlinRafi", "author_url": "", "post_date": "2020-06-27T04:59:34.870000", "content": "Hi there Chris,
\n\nI am still quite unsure on how to perform group Kfold using this tfrec data type, do you care to explain how to approach this problem? Do I separate the training set by its tfrec file?
\n\nThank you so much for your valuable inputs.
", "votes": 0, "replies": [] } ] }, { "id": 896384, "author_name": "Nitesh Chaudhry", "author_url": "", "post_date": "2020-06-22T06:05:10.323000", "content": "Amazing ! I wondered why higher resolution images may not lead to a better score. Now I know why 😄
", "votes": 3, "replies": [] }, { "id": 893903, "author_name": "Aman Arora", "author_url": "", "post_date": "2020-06-20T04:31:55.840000", "content": "Thank you so very much! Will definitely give this a try!
", "votes": 3, "replies": [] }, { "id": 1172017, "author_name": "CCYang1005", "author_url": "", "post_date": "2021-01-27T08:10:37.427000", "content": "this lesson help me last-week's question
", "votes": 1, "replies": [] }, { "id": 1102547, "author_name": "AndikaRachman", "author_url": "", "post_date": "2020-12-05T03:46:46.163000", "content": "Interesting stuffs! I always assume the input size have to follow the pretrained input size. This is an eye-opening.
", "votes": 1, "replies": [] }, { "id": 1059093, "author_name": "Dmitry Smirnov", "author_url": "", "post_date": "2020-10-24T15:59:29.923000", "content": "good stuff
", "votes": 1, "replies": [] }, { "id": 1047170, "author_name": "Sahil Bhambhu", "author_url": "", "post_date": "2020-10-12T10:42:41.223000", "content": "great work
", "votes": 1, "replies": [] }, { "id": 1002499, "author_name": "Sifat Muhammad Abdullah", "author_url": "", "post_date": "2020-09-08T07:18:21.937000", "content": "This is really insightful and helpful. Thank you so much!
", "votes": 1, "replies": [] }, { "id": 971898, "author_name": "aman2000jaiswal", "author_url": "", "post_date": "2020-08-16T04:02:08.250000", "content": "Nice Explanation sir. One question is that how pretrained model take any size input if they originally trained on 224*224 size images?
", "votes": 1, "replies": [ { "id": 971940, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-16T05:19:19.477000", "content": "It's a fully convolutional network. That means that it scans the input image with a 3x3 window. Then after a bunch of these layers, the images get reduced to half the size. And so on. At no point does the network care about the original size.
", "votes": 3, "replies": [ { "id": 971951, "author_name": "aman2000jaiswal", "author_url": "", "post_date": "2020-08-16T05:44:33.957000", "content": "Sir, If i want to train my model from scratch and want to change it as you have explained above then what i have to add extra. Can you have any kernel or link regarding this explanation.
", "votes": 0, "replies": [] } ] } ] }, { "id": 967321, "author_name": "Harshit Jain", "author_url": "", "post_date": "2020-08-12T06:50:37.103000", "content": "Intresting
", "votes": 1, "replies": [] }, { "id": 955987, "author_name": "Snehal Lokesh", "author_url": "", "post_date": "2020-08-03T06:11:49.683000", "content": "Thanks for the explanation . It was useful .
", "votes": 1, "replies": [] }, { "id": 945159, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2020-07-25T15:53:02.630000", "content": "Very good and descriptive explanation!
\n\nScale issues is so important that there are some works that achieve \"Top-of-ImageNet\" position only by properly solving scale issue.\nhttps://arxiv.org/abs/2003.08237\nhttps://arxiv.org/abs/1906.06423
", "votes": 1, "replies": [ { "id": 949629, "author_name": "Serigne ", "author_url": "", "post_date": "2020-07-28T18:56:36.380000", "content": "Hum I would give much of the credit about the smart scaling idea to the original Efficientnet paper from Google Brain :)
", "votes": 1, "replies": [] } ] }, { "id": 944411, "author_name": "ngxnam", "author_url": "", "post_date": "2020-07-25T05:12:09.587000", "content": "Thanks much. Your explanation is easy to understand. \nIt is very helpful to a newbie like me.
", "votes": 1, "replies": [] }, { "id": 934430, "author_name": "Naman Jaswani", "author_url": "", "post_date": "2020-07-18T12:33:26.637000", "content": "Wow!
\nSo much new information to learn from this.
\nThanks!
👍 👍
", "votes": 1, "replies": [] }, { "id": 919844, "author_name": "Khan Fashee Monowar (Sawrup)", "author_url": "", "post_date": "2020-07-08T06:23:05.733000", "content": "Thanks for your explaination. This really helps me a lot.
", "votes": 1, "replies": [] }, { "id": 918798, "author_name": "Harsh Nagouda", "author_url": "", "post_date": "2020-07-07T13:57:22.090000", "content": "This makes a lot more sense now, thanks for the detailed explanation!
", "votes": 1, "replies": [ { "id": 919001, "author_name": "Rogelio Montemayor", "author_url": "", "post_date": "2020-07-07T16:34:26.427000", "content": "Thanks
", "votes": 1, "replies": [] } ] }, { "id": 915240, "author_name": "Muskan Jain", "author_url": "", "post_date": "2020-07-04T15:22:07.903000", "content": "Great Explanation!
", "votes": 1, "replies": [] }, { "id": 915179, "author_name": "Vikas Kumar", "author_url": "", "post_date": "2020-07-04T14:20:29.450000", "content": "thank you for sharing this knowledge, due to this type of sharing, kaggle is always best for learning new things everyday.
", "votes": 1, "replies": [] }, { "id": 913780, "author_name": "Benjamin", "author_url": "", "post_date": "2020-07-03T12:18:45.933000", "content": "Thanks for the insightful explanation! Put into a lot of perspective the idea of using different sized images for training.
", "votes": 1, "replies": [ { "id": 914656, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-04T06:03:23.460000", "content": "Thank you
", "votes": 0, "replies": [] } ] }, { "id": 913578, "author_name": "Naveenkumar R", "author_url": "", "post_date": "2020-07-03T09:39:32.743000", "content": "I understood it well now, thank you very much.
", "votes": 1, "replies": [] }, { "id": 910957, "author_name": "", "author_url": "", "post_date": "2020-07-01T13:20:28.390000", "content": "", "votes": 1, "replies": [ { "id": 911786, "author_name": "", "author_url": "", "post_date": "2020-07-02T02:54:41.750000", "content": "", "votes": 1, "replies": [] }, { "id": 916660, "author_name": "", "author_url": "", "post_date": "2020-07-05T22:35:13.790000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 910465, "author_name": "", "author_url": "", "post_date": "2020-07-01T06:52:27.770000", "content": "", "votes": 1, "replies": [] }, { "id": 910401, "author_name": "", "author_url": "", "post_date": "2020-07-01T06:14:51.977000", "content": "", "votes": 1, "replies": [ { "id": 910417, "author_name": "", "author_url": "", "post_date": "2020-07-01T06:21:15.603000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 909998, "author_name": "", "author_url": "", "post_date": "2020-07-01T00:58:09.490000", "content": "", "votes": 1, "replies": [] }, { "id": 907946, "author_name": "", "author_url": "", "post_date": "2020-06-30T09:21:29.783000", "content": "", "votes": 1, "replies": [] }, { "id": 907945, "author_name": "", "author_url": "", "post_date": "2020-06-30T09:20:46.867000", "content": "", "votes": 1, "replies": [] }, { "id": 907722, "author_name": "", "author_url": "", "post_date": "2020-06-30T05:54:12.253000", "content": "", "votes": 1, "replies": [] }, { "id": 907720, "author_name": "", "author_url": "", "post_date": "2020-06-30T05:53:12.547000", "content": "", "votes": 1, "replies": [] }, { "id": 907617, "author_name": "", "author_url": "", "post_date": "2020-06-30T04:19:40.033000", "content": "", "votes": 1, "replies": [ { "id": 911787, "author_name": "", "author_url": "", "post_date": "2020-07-02T02:54:59.107000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 905988, "author_name": "", "author_url": "", "post_date": "2020-06-29T02:39:12.577000", "content": "", "votes": 1, "replies": [] }, { "id": 905077, "author_name": "", "author_url": "", "post_date": "2020-06-28T08:24:58.693000", "content": "", "votes": 1, "replies": [] }, { "id": 904031, "author_name": "", "author_url": "", "post_date": "2020-06-27T09:44:26.960000", "content": "", "votes": 1, "replies": [] }, { "id": 903507, "author_name": "", "author_url": "", "post_date": "2020-06-26T22:24:12.527000", "content": "", "votes": 1, "replies": [ { "id": 903746, "author_name": "", "author_url": "", "post_date": "2020-06-27T04:44:04.723000", "content": "", "votes": 4, "replies": [] }, { "id": 903841, "author_name": "", "author_url": "", "post_date": "2020-06-27T06:43:35.080000", "content": "", "votes": 0, "replies": [] }, { "id": 904545, "author_name": "", "author_url": "", "post_date": "2020-06-27T17:46:19.767000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 902244, "author_name": "", "author_url": "", "post_date": "2020-06-26T02:58:14.530000", "content": "", "votes": 1, "replies": [] }, { "id": 899155, "author_name": "", "author_url": "", "post_date": "2020-06-24T03:17:58.620000", "content": "", "votes": 1, "replies": [] }, { "id": 898998, "author_name": "", "author_url": "", "post_date": "2020-06-23T21:59:38.380000", "content": "", "votes": 1, "replies": [] }, { "id": 898706, "author_name": "", "author_url": "", "post_date": "2020-06-23T17:22:23.893000", "content": "", "votes": 1, "replies": [] }, { "id": 897609, "author_name": "", "author_url": "", "post_date": "2020-06-23T01:49:30.193000", "content": "", "votes": 1, "replies": [ { "id": 897618, "author_name": "", "author_url": "", "post_date": "2020-06-23T01:53:56.547000", "content": "", "votes": 0, "replies": [] }, { "id": 897724, "author_name": "", "author_url": "", "post_date": "2020-06-23T04:04:46.970000", "content": "", "votes": 0, "replies": [] }, { "id": 898376, "author_name": "", "author_url": "", "post_date": "2020-06-23T13:42:49.537000", "content": "", "votes": 0, "replies": [] }, { "id": 898491, "author_name": "", "author_url": "", "post_date": "2020-06-23T14:54:34.320000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 897063, "author_name": "", "author_url": "", "post_date": "2020-06-22T15:44:27.933000", "content": "", "votes": 1, "replies": [ { "id": 897089, "author_name": "", "author_url": "", "post_date": "2020-06-22T15:57:04.063000", "content": "", "votes": 0, "replies": [] }, { "id": 897172, "author_name": "", "author_url": "", "post_date": "2020-06-22T17:05:05.077000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 896183, "author_name": "", "author_url": "", "post_date": "2020-06-22T00:05:25.057000", "content": "", "votes": 1, "replies": [] }, { "id": 895940, "author_name": "", "author_url": "", "post_date": "2020-06-21T17:43:24.870000", "content": "", "votes": 1, "replies": [ { "id": 896221, "author_name": "", "author_url": "", "post_date": "2020-06-22T01:49:06.790000", "content": "", "votes": 14, "replies": [] }, { "id": 896238, "author_name": "", "author_url": "", "post_date": "2020-06-22T02:22:37.437000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 895383, "author_name": "", "author_url": "", "post_date": "2020-06-21T09:54:55.863000", "content": "", "votes": 1, "replies": [] }, { "id": 895199, "author_name": "", "author_url": "", "post_date": "2020-06-21T07:06:17.023000", "content": "", "votes": 1, "replies": [] }, { "id": 894542, "author_name": "", "author_url": "", "post_date": "2020-06-20T14:09:49.277000", "content": "", "votes": 1, "replies": [ { "id": 894559, "author_name": "", "author_url": "", "post_date": "2020-06-20T14:24:21.560000", "content": "", "votes": 6, "replies": [] }, { "id": 896050, "author_name": "", "author_url": "", "post_date": "2020-06-21T19:27:45.400000", "content": "", "votes": 3, "replies": [] }, { "id": 896246, "author_name": "", "author_url": "", "post_date": "2020-06-22T02:38:31.937000", "content": "", "votes": 1, "replies": [] }, { "id": 905823, "author_name": "", "author_url": "", "post_date": "2020-06-28T20:39:45.743000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 894178, "author_name": "", "author_url": "", "post_date": "2020-06-20T07:57:16.420000", "content": "", "votes": 1, "replies": [] }, { "id": 894125, "author_name": "", "author_url": "", "post_date": "2020-06-20T07:29:48.907000", "content": "", "votes": 1, "replies": [] }, { "id": 894073, "author_name": "", "author_url": "", "post_date": "2020-06-20T06:45:28.773000", "content": "", "votes": 1, "replies": [ { "id": 894482, "author_name": "", "author_url": "", "post_date": "2020-06-20T13:14:40.417000", "content": "", "votes": 8, "replies": [] }, { "id": 894593, "author_name": "", "author_url": "", "post_date": "2020-06-20T14:57:13.700000", "content": "", "votes": 1, "replies": [] }, { "id": 894603, "author_name": "", "author_url": "", "post_date": "2020-06-20T15:10:17.247000", "content": "", "votes": 1, "replies": [] }, { "id": 894705, "author_name": "", "author_url": "", "post_date": "2020-06-20T17:11:05.107000", "content": "", "votes": 0, "replies": [] }, { "id": 897056, "author_name": "", "author_url": "", "post_date": "2020-06-22T15:39:21.353000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 967048, "author_name": "", "author_url": "", "post_date": "2020-08-11T21:34:10.743000", "content": "", "votes": 2, "replies": [ { "id": 967088, "author_name": "", "author_url": "", "post_date": "2020-08-11T23:07:34.747000", "content": "", "votes": 2, "replies": [] }, { "id": 967102, "author_name": "", "author_url": "", "post_date": "2020-08-11T23:38:42.660000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 930022, "author_name": "", "author_url": "", "post_date": "2020-07-15T06:26:58.237000", "content": "", "votes": 2, "replies": [] }, { "id": 919716, "author_name": "", "author_url": "", "post_date": "2020-07-08T04:32:05.773000", "content": "", "votes": 2, "replies": [] }, { "id": 916881, "author_name": "", "author_url": "", "post_date": "2020-07-06T05:03:24.223000", "content": "", "votes": 2, "replies": [ { "id": 916913, "author_name": "", "author_url": "", "post_date": "2020-07-06T05:41:10.223000", "content": "", "votes": 18, "replies": [] }, { "id": 916967, "author_name": "", "author_url": "", "post_date": "2020-07-06T06:43:31.680000", "content": "", "votes": 1, "replies": [] }, { "id": 921065, "author_name": "", "author_url": "", "post_date": "2020-07-09T04:04:07.060000", "content": "", "votes": 0, "replies": [] }, { "id": 921900, "author_name": "", "author_url": "", "post_date": "2020-07-09T16:30:51.413000", "content": "", "votes": 0, "replies": [] }, { "id": 921914, "author_name": "", "author_url": "", "post_date": "2020-07-09T16:45:06.530000", "content": "", "votes": 0, "replies": [] }, { "id": 924965, "author_name": "", "author_url": "", "post_date": "2020-07-11T18:33:15.097000", "content": "", "votes": 1, "replies": [] }, { "id": 926449, "author_name": "", "author_url": "", "post_date": "2020-07-12T18:20:53.497000", "content": "", "votes": 0, "replies": [] }, { "id": 927569, "author_name": "", "author_url": "", "post_date": "2020-07-13T13:40:42.393000", "content": "", "votes": 0, "replies": [] }, { "id": 927708, "author_name": "", "author_url": "", "post_date": "2020-07-13T14:56:25.830000", "content": "", "votes": 1, "replies": [] }, { "id": 934706, "author_name": "", "author_url": "", "post_date": "2020-07-18T16:56:46.750000", "content": "", "votes": 1, "replies": [] }, { "id": 981187, "author_name": "", "author_url": "", "post_date": "2020-08-22T08:51:37.083000", "content": "", "votes": 0, "replies": [] }, { "id": 1349135, "author_name": "", "author_url": "", "post_date": "2021-06-14T14:31:40.113000", "content": "", "votes": 0, "replies": [] }, { "id": 1495368, "author_name": "", "author_url": "", "post_date": "2021-08-29T14:02:41", "content": "", "votes": 0, "replies": [] }, { "id": 1495384, "author_name": "", "author_url": "", "post_date": "2021-08-29T14:14:20.130000", "content": "", "votes": 1, "replies": [] }, { "id": 1496067, "author_name": "", "author_url": "", "post_date": "2021-08-30T04:27:04.180000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 904976, "author_name": "", "author_url": "", "post_date": "2020-06-28T06:24:00.937000", "content": "", "votes": 2, "replies": [] }, { "id": 904855, "author_name": "", "author_url": "", "post_date": "2020-06-28T03:01:50.607000", "content": "", "votes": 2, "replies": [ { "id": 911792, "author_name": "", "author_url": "", "post_date": "2020-07-02T02:57:12.813000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 902197, "author_name": "", "author_url": "", "post_date": "2020-06-26T01:38:04.020000", "content": "", "votes": 2, "replies": [] }, { "id": 902104, "author_name": "", "author_url": "", "post_date": "2020-06-26T00:02:14.997000", "content": "", "votes": 2, "replies": [] }, { "id": 898171, "author_name": "", "author_url": "", "post_date": "2020-06-23T11:02:06.070000", "content": "", "votes": 2, "replies": [ { "id": 898480, "author_name": "", "author_url": "", "post_date": "2020-06-23T14:50:39.690000", "content": "", "votes": 2, "replies": [] }, { "id": 899639, "author_name": "", "author_url": "", "post_date": "2020-06-24T10:53:45.283000", "content": "", "votes": 0, "replies": [] }, { "id": 900773, "author_name": "", "author_url": "", "post_date": "2020-06-25T03:52:39.140000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 896576, "author_name": "", "author_url": "", "post_date": "2020-06-22T09:22:20.657000", "content": "", "votes": 2, "replies": [] }, { "id": 893905, "author_name": "", "author_url": "", "post_date": "2020-06-20T04:33:15.833000", "content": "", "votes": 2, "replies": [ { "id": 893958, "author_name": "", "author_url": "", "post_date": "2020-06-20T05:08:40.887000", "content": "", "votes": 6, "replies": [] }, { "id": 894042, "author_name": "", "author_url": "", "post_date": "2020-06-20T06:05:50.583000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3234396, "author_name": "", "author_url": "", "post_date": "2025-06-27T19:03:59.710000", "content": "", "votes": 0, "replies": [] }, { "id": 3183687, "author_name": "", "author_url": "", "post_date": "2025-04-21T06:57:22.207000", "content": "", "votes": 0, "replies": [] }, { "id": 3107956, "author_name": "", "author_url": "", "post_date": "2025-01-27T06:56:49.770000", "content": "", "votes": 0, "replies": [] }, { "id": 2559678, "author_name": "", "author_url": "", "post_date": "2023-12-13T02:44:25.123000", "content": "", "votes": 0, "replies": [] }, { "id": 967341, "author_name": "", "author_url": "", "post_date": "2020-08-12T07:09:44.350000", "content": "", "votes": 0, "replies": [] }, { "id": 937640, "author_name": "", "author_url": "", "post_date": "2020-07-21T05:40:04.010000", "content": "", "votes": 0, "replies": [] }, { "id": 936723, "author_name": "", "author_url": "", "post_date": "2020-07-20T13:27:31.363000", "content": "", "votes": 0, "replies": [] }, { "id": 927095, "author_name": "", "author_url": "", "post_date": "2020-07-13T07:44:20.427000", "content": "", "votes": 0, "replies": [ { "id": 927594, "author_name": "", "author_url": "", "post_date": "2020-07-13T13:57:16.303000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 925004, "author_name": "", "author_url": "", "post_date": "2020-07-11T19:10:42.260000", "content": "", "votes": 0, "replies": [ { "id": 925007, "author_name": "", "author_url": "", "post_date": "2020-07-11T19:12:08.410000", "content": "", "votes": 6, "replies": [] }, { "id": 934244, "author_name": "", "author_url": "", "post_date": "2020-07-18T10:36:16.537000", "content": "", "votes": 1, "replies": [] }, { "id": 935135, "author_name": "", "author_url": "", "post_date": "2020-07-19T05:21:55.123000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 902708, "author_name": "", "author_url": "", "post_date": "2020-06-26T10:08:08.993000", "content": "", "votes": 0, "replies": [ { "id": 903748, "author_name": "", "author_url": "", "post_date": "2020-06-27T04:44:34.133000", "content": "", "votes": 0, "replies": [] }, { "id": 908036, "author_name": "", "author_url": "", "post_date": "2020-06-30T10:28:23.623000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 901956, "author_name": "", "author_url": "", "post_date": "2020-06-25T20:14:35.147000", "content": "", "votes": 0, "replies": [ { "id": 902097, "author_name": "", "author_url": "", "post_date": "2020-06-25T23:47:50.070000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 900666, "author_name": "", "author_url": "", "post_date": "2020-06-25T02:11:59.160000", "content": "", "votes": 0, "replies": [ { "id": 901559, "author_name": "", "author_url": "", "post_date": "2020-06-25T15:07:27.813000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 898789, "author_name": "", "author_url": "", "post_date": "2020-06-23T18:38:18.517000", "content": "", "votes": 0, "replies": [] }, { "id": 897892, "author_name": "", "author_url": "", "post_date": "2020-06-23T07:03:58.577000", "content": "", "votes": 0, "replies": [] }, { "id": 897789, "author_name": "", "author_url": "", "post_date": "2020-06-23T05:14:20.260000", "content": "", "votes": 0, "replies": [ { "id": 898478, "author_name": "", "author_url": "", "post_date": "2020-06-23T14:48:15.937000", "content": "", "votes": 2, "replies": [] }, { "id": 898707, "author_name": "", "author_url": "", "post_date": "2020-06-23T17:23:07.940000", "content": "", "votes": 1, "replies": [] }, { "id": 898753, "author_name": "", "author_url": "", "post_date": "2020-06-23T17:57:22.260000", "content": "", "votes": 4, "replies": [] } ] }, { "id": 897573, "author_name": "", "author_url": "", "post_date": "2020-06-23T00:57:47.937000", "content": "", "votes": 0, "replies": [] }, { "id": 894311, "author_name": "", "author_url": "", "post_date": "2020-06-20T10:26:21.750000", "content": "", "votes": 0, "replies": [ { "id": 894494, "author_name": "", "author_url": "", "post_date": "2020-06-20T13:35:43.553000", "content": "", "votes": 2, "replies": [] }, { "id": 894506, "author_name": "", "author_url": "", "post_date": "2020-06-20T13:42:58.847000", "content": "", "votes": 0, "replies": [] }, { "id": 894522, "author_name": "", "author_url": "", "post_date": "2020-06-20T13:53:15.273000", "content": "", "votes": 7, "replies": [] } ] }, { "id": 894225, "author_name": "", "author_url": "", "post_date": "2020-06-20T08:43:44.417000", "content": "", "votes": 0, "replies": [ { "id": 894261, "author_name": "", "author_url": "", "post_date": "2020-06-20T09:21:08.740000", "content": "", "votes": 0, "replies": [] }, { "id": 894264, "author_name": "", "author_url": "", "post_date": "2020-06-20T09:24:35.543000", "content": "", "votes": 0, "replies": [] }, { "id": 894267, "author_name": "", "author_url": "", "post_date": "2020-06-20T09:28:20.137000", "content": "", "votes": 1, "replies": [] }, { "id": 894270, "author_name": "", "author_url": "", "post_date": "2020-06-20T09:32:47.820000", "content": "", "votes": 1, "replies": [] }, { "id": 894283, "author_name": "", "author_url": "", "post_date": "2020-06-20T09:45:14.063000", "content": "", "votes": 0, "replies": [] }, { "id": 894291, "author_name": "", "author_url": "", "post_date": "2020-06-20T09:53:25.770000", "content": "", "votes": 1, "replies": [] }, { "id": 894293, "author_name": "", "author_url": "", "post_date": "2020-06-20T09:58:47.917000", "content": "", "votes": 0, "replies": [] }, { "id": 894496, "author_name": "", "author_url": "", "post_date": "2020-06-20T13:38:45.137000", "content": "", "votes": 4, "replies": [] } ] }, { "id": 3187005, "author_name": "", "author_url": "", "post_date": "2025-04-25T12:15:49.597000", "content": "", "votes": 0, "replies": [] }, { "id": 963095, "author_name": "", "author_url": "", "post_date": "2020-08-08T17:02:23.633000", "content": "", "votes": 1, "replies": [] }, { "id": 913230, "author_name": "", "author_url": "", "post_date": "2020-07-03T04:40:28.167000", "content": "", "votes": 1, "replies": [] }, { "id": 895059, "author_name": "", "author_url": "", "post_date": "2020-06-21T04:34:16.487000", "content": "", "votes": 1, "replies": [] }, { "id": 894683, "author_name": "", "author_url": "", "post_date": "2020-06-20T16:45:44.960000", "content": "", "votes": 1, "replies": [ { "id": 896245, "author_name": "", "author_url": "", "post_date": "2020-06-22T02:37:25.100000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1154108, "author_name": "", "author_url": "", "post_date": "2021-01-15T12:07:04.253000", "content": "", "votes": 1, "replies": [] }, { "id": 1110802, "author_name": "", "author_url": "", "post_date": "2020-12-13T04:53:16.757000", "content": "", "votes": 1, "replies": [] }, { "id": 982398, "author_name": "", "author_url": "", "post_date": "2020-08-23T10:37:25.117000", "content": "", "votes": 1, "replies": [] }, { "id": 972736, "author_name": "", "author_url": "", "post_date": "2020-08-16T19:10:14.940000", "content": "", "votes": 1, "replies": [] }, { "id": 940397, "author_name": "", "author_url": "", "post_date": "2020-07-22T23:12:51.583000", "content": "", "votes": 1, "replies": [] }, { "id": 933120, "author_name": "", "author_url": "", "post_date": "2020-07-17T14:00:47.753000", "content": "", "votes": 1, "replies": [] }, { "id": 910072, "author_name": "", "author_url": "", "post_date": "2020-07-01T01:46:09.637000", "content": "", "votes": 1, "replies": [] }, { "id": 907394, "author_name": "", "author_url": "", "post_date": "2020-06-29T23:45:56.377000", "content": "", "votes": 1, "replies": [] }, { "id": 907310, "author_name": "", "author_url": "", "post_date": "2020-06-29T20:51:39.067000", "content": "", "votes": 1, "replies": [] }, { "id": 906596, "author_name": "", "author_url": "", "post_date": "2020-06-29T12:43:23.347000", "content": "", "votes": 1, "replies": [] }, { "id": 902088, "author_name": "", "author_url": "", "post_date": "2020-06-25T23:25:57.517000", "content": "", "votes": 1, "replies": [] }, { "id": 900599, "author_name": "", "author_url": "", "post_date": "2020-06-24T23:27:02.343000", "content": "", "votes": 1, "replies": [] }, { "id": 899963, "author_name": "", "author_url": "", "post_date": "2020-06-24T14:50:40.300000", "content": "", "votes": 1, "replies": [] }, { "id": 897846, "author_name": "", "author_url": "", "post_date": "2020-06-23T06:19:15.937000", "content": "", "votes": 1, "replies": [] }, { "id": 897139, "author_name": "", "author_url": "", "post_date": "2020-06-22T16:35:54.497000", "content": "", "votes": 1, "replies": [] }, { "id": 3245480, "author_name": "", "author_url": "", "post_date": "2025-07-09T12:25:36.757000", "content": "", "votes": 0, "replies": [] }, { "id": 3189224, "author_name": "", "author_url": "", "post_date": "2025-04-29T01:36:18.697000", "content": "", "votes": 0, "replies": [] }, { "id": 897492, "author_name": "", "author_url": "", "post_date": "2020-06-22T22:32:25.240000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "893879": "Most Imagenet pretrained CNNs were trained on 224x224 image resolution. It is a common misconception, that when using these pretrained CNN, images need to be resized to 224x224. On the contrary, popular CNN are fully convolutional nets that can accept any input size. \n\nYou can input any image size and these CNN output feature maps that are 32x times smaller. For example, if you input 224x224 then the CNN outputs feature maps of size 7x7. If you input images of size 512x512, then these CNN outputs feature maps of size 16x16.\n\n# Feature Maps\nThe only relevance of the pretraining 224x224 size is that these CNN have learned to find certain patterns of **certain sizes**. For example, maybe they learned to find circles that are 50 pixels diameter, or maybe they learned to find triangles with side length 30 pixels.\n\n# Pretrained Imagenet CNN\nIn the example below, we pretrain CNN on images of size 224x224 and they learn to detect circles of diameter 50 pixels and triangles of side length 30. \n\n\n# Why Resize Input Images\nWhen you resize input images, you change the size of your circles and triangles. So depending on how you resize your input image, this pretrained CNN may or may not find circles of diameter 50 and triangles of side 30. In the example below, given the original image, the CNN only finds the circles when the input image is resized to 512x512 and finds triangles when resized to 128x128\n\n# 512x512 Input\n\n\n#256x256 and 128x128 Input\n\n\n# Conclusion\nIn conclusion, a CNN searches for thousands of patterns (not just one circle size and one triangle size). It will find some patterns with some sizes and other patterns with other sizes. Therefore you should experiment with different input sizes to see which has better CV and LB. Also consider ensembling models of different input sizes for maximum CV and LB.\n\n# Kaggle Datasets\nI have resized all the competition images to TFRecords 768x768 [here][1], 512x512 [here][2], 384x384 [here][3], 256x256 [here][4], 192x192 [here][7], and I provide external data of size 512x512 [here][5]. If you ensemble models of different sizes, you can achieve LB 0.960 or higher!\n\nIf you prefer JPEGs instead of the TFRecords, you can find JPEGs 768x768 [here][8], 512x512 [here][9], 384x384 [here][10], 256x256 [here][11], 192x192 [here][12]. And external data 512x512 [here][13]. (Also I have resized last year's 2019 data [here][14]).\n\nYou can even experiment with a single model that analyzes multiple image sizes at once with multiple EfficientNet backbones. After the image is inputted, you apply multiple types of downsizing like 2x, 3x, 4x to feed the different backbones. Then apply `GlobalAveragePooling2D()` after all the backbones then `Concatenate` all those vectors and finally apply `Dense(1, activation='sigmoid')` for classification. This worked well in Cloud Comp. An example is posted in Krazy Klassifiers [here][6] \n\nEnjoy!\n \n[1]: https://www.kaggle.com/cdeotte/melanoma-768x768\n[2]: https://www.kaggle.com/cdeotte/melanoma-512x512\n[3]: https://www.kaggle.com/cdeotte/melanoma-384x384\n[4]: https://www.kaggle.com/cdeotte/melanoma-256x256\n[5]: https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images\n[6]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/118086\n[7]: https://www.kaggle.com/cdeotte/melanoma-192x192\n[8]: https://www.kaggle.com/cdeotte/jpeg-melanoma-768x768\n[9]: https://www.kaggle.com/cdeotte/jpeg-melanoma-512x512\n[10]: https://www.kaggle.com/cdeotte/jpeg-melanoma-384x384\n[11]: https://www.kaggle.com/cdeotte/jpeg-melanoma-256x256\n[12]: https://www.kaggle.com/cdeotte/jpeg-melanoma-192x192\n[13]: https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg\n[14]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910", "2185587": "Your visualization is beautiful @cdeotte ", "918961": "you know what? this makes easier to understand than reddit!😂\n", "1685746": "thanks，from your PetFinder link , congratulations!", "895277": "@cdeotte \n\nthanks for the post.\nwhich is more important, size of image image or depth (complexity) of model.\n\nit seems difficult to train a deep efficientnetB7 on large size on local GPU and shallower efficientnetB0,1,2 doesn't benefit from large size?", "896018": "That is the one weird looking racing car you have there Chris.", "991203": "I learnt a new lesson today from the discussion", "904543": "Hey Chris! You analysis always helps.", "901957": "Nice Explanation! Thanks :) .As you can see, the images of melanomas are taken with different distances, which can give us false information regarding the real size of the melanoma. do you think it could affect the model ?", "900983": "Other than trying different image sizes, I started with trying different architectures for a image size of `224`, I used a B0 and a Resnet50 arch for this work, as my intuition was that different archs may learn different features.\n\nAnd, as to validate, I got 0.888 LB with only B0 while I got 0.895 with an ensemble of B0 and Resnet50.\n\nAlso, I thought of doing this for more image sizes like 384, 512, 768, 1024 and then combining their results. But I think it would not be computationally efficient as it would take a lot of time to do that, what do you think of that, @cdeotte Should I go for it?", "896384": "Amazing ! I wondered why higher resolution images may not lead to a better score. Now I know why 😄 \n\n", "893903": "Thank you so very much! Will definitely give this a try!", "1172017": "this lesson help me last-week's question", "1102547": "Interesting stuffs! I always assume the input size have to follow the pretrained input size. This is an eye-opening.", "1059093": "good stuff", "1047170": "great work", "1002499": "This is really insightful and helpful. Thank you so much!", "971898": "Nice Explanation sir. One question is that how pretrained model take any size input if they originally trained on 224*224 size images?", "967321": "Intresting", "955987": "Thanks for the explanation . It was useful .", "945159": "Very good and descriptive explanation!\n \nScale issues is so important that there are some works that achieve \"Top-of-ImageNet\" position only by properly solving scale issue.\nhttps://arxiv.org/abs/2003.08237\nhttps://arxiv.org/abs/1906.06423", "944411": "Thanks much. Your explanation is easy to understand. \nIt is very helpful to a newbie like me.", "934430": "Wow!\nSo much new information to learn from this.\nThanks!", "919998": "👍 👍 ", "919844": "Thanks for your explaination. This really helps me a lot.", "918798": "This makes a lot more sense now, thanks for the detailed explanation!", "915240": "Great Explanation!", "915179": "thank you for sharing this knowledge, due to this type of sharing, kaggle is always best for learning new things everyday.\n ", "913780": "Thanks for the insightful explanation! Put into a lot of perspective the idea of using different sized images for training. ", "913578": "I understood it well now, thank you very much.", "910957": "Thanks @cdeotte\n\nI'm curious about the effect of weight sharing between EfficientNets vs different nets", "910465": "Very nice effort", "910401": "Thank @cdeotte ,\n\nI try to implement your idea, and I got 0.912 score. I still work on it.\nLink here [https://www.kaggle.com/truonghoang/multi-size-eff-lb-0-912]\n\nGreat idea.", "909998": "Great!", "907946": "Interesting Stuff", "907945": "**Interesting stuff**", "907722": "Great Explanation!!", "907720": "Great Explanation !!", "907617": "You are my real life saver, thanks for your great explanation!", "905988": "short but practical, very cool effort! ", "905077": "Helpful explanation!", "904031": "Very Interesting! Thanks for explaining.", "903507": "thank you @cdeotte for enlightening us about that, is there some public notebook on how to ensemble models with different sizes?", "902244": "Very Interesting! Thanks for explaining.", "899155": "Excellent explanation!", "898998": "Great explanation!", "898706": "This is great. Thank you for putting this together. I could relate to my recent Image Classification file (what not to do maybe).", "897609": "Hi Chris @cdeotte, thank you for sharing! Are the files with the same name in different resized folders based on the same original images? For example, are files \"train00-2071\" in 768x768 folder and 512x512 folder are from the same batch of original images? I'm trying to hold out the same portion of data for validation while using different sizes of image as input. Hope I described my question clearly.", "897063": "> I have resized all the competition images to 768x768 here, 512x512 here, 384x384 here, 256x256 here, and I provide external data of size 512x512 here.\n\nIs there an 'IOS' version of this 'app' available?\n\nAnd by IOS I mean pytorch. Cause, you know, tfrecords -> tensorflow -> google -> android...\nAnd by version I mean plain jpg image or something.", "896183": "Nice Explanation!\n", "895940": "@cdeotte Great insight!\nJust curious though..did increase in size result in a higher score for you?", "895383": "Great explanation.really helpfull👍 ", "895199": "Thanks, @cdeotte !\nIt is something new to me and this is going to help a lot. 💯", "894542": "Hi @cdeotte, thanks again for your great explanations. I am planning to enter this competition now, but I don't have a lot of disk space left for the original Dataset ... could I use the datasets that you created resized without losing any information? ", "894178": "nice explanation", "894125": "Nice Explanation👍 ", "894073": "Thank you, your sharing was really helpful! A small question, your score is currently impressive, it is the result of a single model using one type of image size, or a single model using multiple image sizes then ensemble as you mentioned. in the discussion?", "967048": "A quick question about image sizes and submission averaging. Maybe I'm missing something very obvious for masters here, but does it makes any sense to averages all models trained over different sizes or is it better to average the averages? \n\nLike let's sat we have a 5-fold B5 trained on image size 384 and 3-fold B6 trained on image size 512, different models may capture different features in the images. Would it make sense to first average over folds and then average the resulting submissions or maybe it makes sense to average all generated submissions ... maybe weighting by the number of folds per size or OOF CV for every model, or this is still where heuristics plays the major role? 🙂", "930022": "Thank you very much, this makes it much easier to understand!", "919716": "👍", "916881": "thanks @cdeotte \nmay i know that how do you use tf.data.Dataset to do multi-inputs ?\ni just put it into a list [dataset1,dataset2...]\nand got an error \n`ValueError: Failed to find data adapter that can handle input: (", "904976": "interesting insights, very relatable to real world applications", "904855": "Great explanation! This is my first time working with pre-trained models and never occurred to me how input sizes could affect them.", "902197": "Wow.. great explanation on the importance of input size variation 👍 👍 ", "902104": "Wow! Great Explanation!", "898171": "Thanks a lot for your valuable information! I have one question. You made various sizes of the competition images. However, you made only one size (512x512) of external data. Why you didn't make other size of external data? I guess other size of external data, such as 768x768, 384x384 etc., might also be useful.", "896576": "nice work! have a great insight about resizing image! thankyou Chris.", "893905": "If the orignal size is 4000*6000, resize to 512*512, then it can't find circle and triangles?", "3234396": "so simple and useful explanation, thank you !", "3183687": "Cool blog. I struggled with convolutions until now thanks to you", "3107956": "Thanks for the information !! That was really helpful!!", "2559678": "This, this right here, is info about computer vision explained so gracefully even a 2nd grader can understand. You are a great teacher @cdeotte ", "967341": "As far as I understand the model learns from the different sizes, and the resultant will be a combination of every size it has learned. Here will it be any change or improvement if the **same** model is retrained with different sizes, that is first train with some size, save it then train on different size; will the model now be any better than only trained on same image size?", "937640": "Very nice idea,\nAs I was working on pytorch, we cant set None to the size of the image,\nSo has anyone implemented this idea on pytorch?", "936723": "@cdeotte thanks for all great discussions and kernels. I have one question.\n\nIf a model is performing well on small size image dataset, can we say it will also perform better on large size image. I am asking this question from experimental point of view. For example, if I train `effnet-b6` on `128x128` image size and after doing some experiments, I got 0.92 AUC so in this case can I assume that the same model will also give better results on `512x512` image size with same `configuration(folds, epochs, loss, optimzer etc.)` or there is a possibility that it can give better results with different configurations on different image size datasets?", "927095": "Great. Trying 1024x1024 but cannot predict.", "925004": "Thanks @cdeotte.\nI have a stupid question! What happens if I use 512x512 or 768x768 images but I set the CNN input_shape to 224x224?", "902708": "Hi, firstly, I resizeed all the images to 1200x960, then I randomcrop the images to 224x224x3 for training, do you random crop the images for your training or just input the resize images instead of random-croped ones? Thank you!!!", "901956": "new question - what do you do with images originaly smaller than the target size? I mean 640x480 images, do you upscale them or fill them with border?", "900666": "@cdeotte . If i am understanding your comments below right, you are saying you have a single model using one of these (256, 384, 512, and 784) image sizes giving 0.95+ LB? and an ensemble of different models (b0,b1,b2, etc) trained on different size images gives you 0.955+LB?\nI must be doing some really wrong, i cannot get anything above 0.91 for a single model using any of these image sizes on a groupKFold using Efficientnet1. I suspect my CV set up is contributing to this low score. I should try a different CV or maybe ignore those multiple patient images and train on images without those outliers rather than trying to compensate for this multiple images per patient scenario using groupKfold and all that. ", "898789": "Will using dilated convolutions to process inputs in higher resolutions and capturing broader information work in this scenario? Can this speed up training as it will skip every adjacent pixel and create a more generalized model? ", "897892": "\"Therefore you should experiment with different input sizes to see which has better CV and LB.\" Any thumb rule to hit sweet spot?", "897789": "hello @cdeotte \n\ncould you explain \"external data\"? \n\nI have read Official External Data Thread and I have seen older ISIC datasets mentioned by @shonenkov and @andrewmvd but by quickly looking on them I don't really understand how they are used, is there somewhere the code how can you adapt those datasets to this competition?", "897573": "Excelente explicação. Obrigada!", "894311": "Thanks a lot for sharing. One question for the input size to your model. I used 224x224 (no external data) , B2-eff net, 5 fold CV, it takes me ~3-4 hrs to complete on GPU. imaging that for 512 x512 with external data sets, the time taken would be very significant thought i haven't tried yet. Question, do you directly feed 512x512 to your model or you will don some augmentation to make input image size smaller? Like bounding box or center circle?", "894225": "Hey Chris, this is really amazing info you have shared here, I definitely want to go with this. \n\nI have one query though, I don't have much experience of working with TPU's, so is it possible to use these TfRecords with GPU? I may go with creating datasets of different image sizes but it may eat up all my GPU quota of the week!", "3187005": "", "963095": "", "913230": "", "895059": "", "894683": "", "1154108": "thanks. It is helpful :)", "1110802": "Thank you Sir!!", "982398": "This itself is GOLD. Thanks!", "972736": "Thanks :)", "940397": "Thanks, this is very helpful!", "933120": "Thank you for sharing!", "910072": "Thanks for the great explanation!", "907394": "Great explanation - thanks!", "907310": "Great explanation - thanks!", "906596": "eye opening post for me . Thanks", "902088": "Great explanation, thanks!", "900599": "Great explanation! thanks", "899963": "Thank you, this really helps. ", "897846": "Thank you for your explains. ", "897139": "Nice Explanation . Thanks a lot", "3245480": "Really helpful!", "3189224": "Thanks a lot!", "897492": "Thanks for the explanation!!!" } }