My current setup:
\n\n0.899 CV, 0.903 LB
", "messages": [ { "id": 871656, "postDate": "2020-06-02T14:49:12.193Z", "content": "My current setup:
\n\n0.899 CV, 0.903 LB
", "rawMarkdown": "My current setup:\n\n* ResNet18\n* Images resized to 256x256 \n* Adam\n* 0.0001 LR for ResNet and 0.02 for randomly initialized classification layer\n* 4 rotations + vertical flips (8 options total) on training\n* 10 epochs\n* batch size is 64\n* 3 folds (group + stratify)\n* Average of the last 4 epochs -> gives slightly more stable results than just last/best epoch\n* Training time is less than an hour on a 1080ti\n\n0.899 CV, 0.903 LB", "votes": 39 }, { "id": 871707, "postDate": "2020-06-02T15:29:11.737Z", "content": "Great job. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 256x256 here. Then after you write you code, you can switch the 256x256 dataset with my 512x512 or 768x768 for a higher CV/LB. Enjoy!
\n\nRotation augmentation for TFRecords is explained here. And CutMix and MixUp here
", "rawMarkdown": "Great job. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 256x256 [here][1]. Then after you write you code, you can switch the 256x256 dataset with my 512x512 or 768x768 for a higher CV/LB. Enjoy!\n\nRotation augmentation for TFRecords is explained [here][2]. And CutMix and MixUp [here][3]\n\n[1]: https://www.kaggle.com/cdeotte/melanoma-256x256\n[2]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935", "votes": 7 }, { "id": 871745, "postDate": "2020-06-02T15:55:19.697Z", "content": "Interestingly, so far switching to 512x512 resolutions makes the result worse. I suspect that's because ResNet18 has quite a small receptive field and couldn't fully utilize higher resolution.
", "rawMarkdown": "Interestingly, so far switching to 512x512 resolutions makes the result **worse**. I suspect that's because ResNet18 has quite a small receptive field and couldn't fully utilize higher resolution.", "votes": 3, "replies": [ { "id": 871749, "postDate": "2020-06-02T15:58:58.120Z", "content": "yes, that is interesting
", "rawMarkdown": "yes, that is interesting" }, { "id": 876217, "postDate": "2020-06-06T14:41:14.853Z", "content": "Better try that resolution on which it was pretrained.This will boost the model's efficiency...
", "rawMarkdown": "Better try that resolution on which it was pretrained.This will boost the model's efficiency...", "votes": 1 } ] }, { "id": 884438, "postDate": "2020-06-13T11:28:55.857Z", "content": "> 4 rotations + vertical flips (8 options total)
\n\nIs this a good way of implementing it?\n\nimport albumentations as A\naugs = A.Compose([\n A.OneOf([\n A.Rotate(limit=(0, 0), p=1),\n A.Rotate(limit=(90, 90), p=1),\n A.Rotate(limit=(180, 180), p=1),\n A.Rotate(limit=(270, 270), p=1)]),\n A.VerticalFlip(p=0.5)\n])\n\nOr is there a more concise code?
I guess I used RandomRotate90 :)
Well, it seems this is much more simple :)
", "rawMarkdown": "Well, it seems this is much more simple :)" } ] }, { "id": 876010, "postDate": "2020-06-06T11:08:26.473Z", "content": "@ddanevskyi have you tried more epochs?
", "rawMarkdown": "@ddanevskyi have you tried more epochs?", "votes": -1, "replies": [ { "id": 876028, "postDate": "2020-06-06T11:18:35.860Z", "content": "Yes. For this setup, there was no difference.
", "rawMarkdown": "Yes. For this setup, there was no difference." }, { "id": 876144, "postDate": "2020-06-06T13:27:52.363Z", "content": "are these results are on the provided dataset only or you used external data as well?
", "rawMarkdown": "are these results are on the provided dataset only or you used external data as well?", "votes": -1 }, { "id": 876171, "postDate": "2020-06-06T13:53:26.873Z", "content": "I'm using only the competition data.
", "rawMarkdown": "I'm using only the competition data." } ] }, { "id": 871790, "postDate": "2020-06-02T16:28:18.627Z", "content": "Really interesting results. Could you share a notebook for the same? Also , what exactly do you mean by average of the last 4 epochs? Also, do you think it is possible to run notebooks for this comp on a 6GB 1660Ti?
", "rawMarkdown": "Really interesting results. Could you share a notebook for the same? Also , what exactly do you mean by average of the last 4 epochs? Also, do you think it is possible to run notebooks for this comp on a 6GB 1660Ti?", "votes": -2, "replies": [ { "id": 871860, "postDate": "2020-06-02T17:19:24.323Z", "content": "I just averaged predictions on the test set from the last 4 epochs (I predict both validation and test on each epoch).
\n\nAs for hardware, 1660Ti might be fine for shallow models (such as ResNet18), but deeper models (which are a must for a high score) are better be trained on TPUs.
", "rawMarkdown": "I just averaged predictions on the test set from the last 4 epochs (I predict both validation and test on each epoch). \n\nAs for hardware, 1660Ti might be fine for shallow models (such as ResNet18), but deeper models (which are a must for a high score) are better be trained on TPUs.", "votes": 1 }, { "id": 871896, "postDate": "2020-06-02T18:00:37.773Z", "content": "Oh , I see. Thanks for the reply. Guess I'll just have to work with training with checkpoints on Kaggle TPU's :(
", "rawMarkdown": "Oh , I see. Thanks for the reply. Guess I'll just have to work with training with checkpoints on Kaggle TPU's :(" } ] }, { "id": 888415, "postDate": "2020-06-16T10:34:50.153Z", "content": "hi！I tried to train the dataset with resnet50, the number of pictures with true label is 584, the number of false is 32000+, that is imbalanced, I trained resnet50 using your strategy, when I predict, the output label for the test images are all 0, can you tell me why? Thank you!
", "rawMarkdown": "hi！I tried to train the dataset with resnet50, the number of pictures with true label is 584, the number of false is 32000+, that is imbalanced, I trained resnet50 using your strategy, when I predict, the output label for the test images are all 0, can you tell me why? Thank you! ", "replies": [ { "id": 888437, "postDate": "2020-06-16T11:01:40.483Z", "content": "I can't. What is your AUC score on the training set?
", "rawMarkdown": "I can't. What is your AUC score on the training set?" }, { "id": 888458, "postDate": "2020-06-16T11:21:55.710Z", "content": "I am changing the code to add this function, is there a threshold to tell the bad from the good, I set it 0.5 before, need I change it to a bigger one ?
", "rawMarkdown": "I am changing the code to add this function, is there a threshold to tell the bad from the good, I set it 0.5 before, need I change it to a bigger one ?" }, { "id": 888472, "postDate": "2020-06-16T11:34:12.873Z", "content": "You can keep it 0 or None too, just share the final AUC score on the training set here! :)
Hey, I have a issue with my setup, I am using,
\n\nI am getting a CV of about 0.72 and that too is fluctuating a lot, it sometimes drops to even 0.3. :/
\n\nI am still a newbie, any suggestions on how to improve this would be highly appreciated.
", "rawMarkdown": "Hey, I have a issue with my setup, I am using,\n\n- Efficientnet-b0\n- 224x224\n- Adam\n- 0.0001 LR \n- Random Horizontal and Vertical flips for training data\n- 10 epochs\n- batch_size = 64\n- 5 folds (Group)\n\nI am getting a CV of about 0.72 and that too is fluctuating a lot, it sometimes drops to even 0.3. :/ \n\nI am still a newbie, any suggestions on how to improve this would be highly appreciated. ", "replies": [ { "id": 882271, "postDate": "2020-06-11T17:15:27.297Z", "content": "Sounds like a bug. Do you normalize images? Are you sure your training/validation/testing pipelines are the same?
", "rawMarkdown": "Sounds like a bug. Do you normalize images? Are you sure your training/validation/testing pipelines are the same?" }, { "id": 882278, "postDate": "2020-06-11T17:19:32.947Z", "content": "Yes, I am normalizing them. Yeah, pipeline is same too.
\n\nI am thinking of running it for more epochs now, maybe 20, to see if it starts to converge then.
", "rawMarkdown": "Yes, I am normalizing them. Yeah, pipeline is same too.\n\nI am thinking of running it for more epochs now, maybe 20, to see if it starts to converge then." }, { "id": 882357, "postDate": "2020-06-11T18:30:51.577Z", "content": "It should converge in a couple of epochs, are you reducing your lr during training with a scheduler?
", "rawMarkdown": "It should converge in a couple of epochs, are you reducing your lr during training with a scheduler? " }, { "id": 882627, "postDate": "2020-06-12T02:22:52.707Z", "content": "Yes, I am using ReduceLROnPlateau as a Scheduler with a patience of 3.
\n\nAlso, I tried reducing the number of groups to 3 and auc is now going upto 0.8 now, originally it was only reaching 0.72 and it seems much stable than before.
\n\nAlthough I know I can get at least 0.9 CV with this setup, it still is a long way to go.
", "rawMarkdown": "Yes, I am using ReduceLROnPlateau as a Scheduler with a patience of 3.\n\nAlso, I tried reducing the number of groups to 3 and auc is now going upto 0.8 now, originally it was only reaching 0.72 and it seems much stable than before.\n\nAlthough I know I can get at least 0.9 CV with this setup, it still is a long way to go." }, { "id": 883925, "postDate": "2020-06-13T04:57:42.793Z", "content": "Also I want to add one more thing, I am predicting on 2 classes with cross entropy loss, can it be causing this much fluctuation and less auc?
", "rawMarkdown": "Also I want to add one more thing, I am predicting on 2 classes with cross entropy loss, can it be causing this much fluctuation and less auc?" }, { "id": 883957, "postDate": "2020-06-13T06:31:20.873Z", "content": "I presume almost everyone is using cross-entropy.
", "rawMarkdown": "I presume almost everyone is using cross-entropy." }, { "id": 883973, "postDate": "2020-06-13T06:34:48.163Z", "content": "Okay, then there must be something else which is off, I will be looking for that then. Thanks! :)
", "rawMarkdown": "Okay, then there must be something else which is off, I will be looking for that then. Thanks! :)" }, { "id": 884489, "postDate": "2020-06-13T12:13:17.547Z", "content": "Can I share my notebook with anyone here? it may be a good deal for me to just get it quick checked with someone else, maybe I can't see what's going wrong and I have already dedicated 7 days to this particular problem. :/
", "rawMarkdown": "Can I share my notebook with anyone here? it may be a good deal for me to just get it quick checked with someone else, maybe I can't see what's going wrong and I have already dedicated 7 days to this particular problem. :/" }, { "id": 884617, "postDate": "2020-06-13T13:46:48.153Z", "content": "You could just make your notebook public. Sharing with someone privately without being in a team is prohibited.
", "rawMarkdown": "You could just make your notebook public. Sharing with someone privately without being in a team is prohibited." }, { "id": 884624, "postDate": "2020-06-13T13:53:47.127Z", "content": "Sure, I will do that, will comment here after making it public. Thanks! :)
", "rawMarkdown": "Sure, I will do that, will comment here after making it public. Thanks! :)" }, { "id": 884729, "postDate": "2020-06-13T15:26:26.410Z", "content": "I made the notebook public.
\n\nIt performed a lot better than the last time as that time the score used to drop to 0.4 sometimes but I think it will still be good if you can take a look.
\n\nI took some part of it from other notebooks too as I was just learning how to work with PyTorch. :)
", "rawMarkdown": "I made the notebook [public](https://www.kaggle.com/sarques/melanoma-classification).\n\nIt performed a lot better than the last time as that time the score used to drop to 0.4 sometimes but I think it will still be good if you can take a look. \n\nI took some part of it from other notebooks too as I was just learning how to work with PyTorch. :)" } ] }, { "id": 873057, "postDate": "2020-06-03T18:25:52.160Z", "content": "\"0.0001 LR for ResNet and 0.02 for randomly initialized classification layer\"- How LR can be different for ResNet end classification layer ? How to write it in tf.keras ?
", "rawMarkdown": "\"0.0001 LR for ResNet and 0.02 for randomly initialized classification layer\"- How LR can be different for ResNet end classification layer ? How to write it in tf.keras ?", "replies": [ { "id": 873070, "postDate": "2020-06-03T18:43:50.940Z", "content": "you would do do something like this where model.base is your features and fc is your classifier:
\n\n\noptimizer = optim.AdamW([{'params': model.base.parameters(), 'lr ':1e-4 },\n {'params': model.fc.parameters(), 'lr ':2e-2 },\n ], lr=lr)\n
This is usefull if you are using a pretrained architechture
", "rawMarkdown": "you would do do something like this where model.base is your features and fc is your classifier:\n\n```\noptimizer = optim.AdamW([{'params': model.base.parameters(), 'lr ':1e-4 },\n {'params': model.fc.parameters(), 'lr ':2e-2 },\n ], lr=lr)\n```\n\nThis is usefull if you are using a pretrained architechture", "votes": 6 }, { "id": 873081, "postDate": "2020-06-03T19:08:10.077Z", "content": "how to initialize model.base and model.fc ? Is it example in tf.keras?
", "rawMarkdown": "how to initialize model.base and model.fc ? Is it example in tf.keras?" }, { "id": 873094, "postDate": "2020-06-03T19:31:07.093Z", "content": "Sorry i dont really know how to do it in keras
", "rawMarkdown": "Sorry i dont really know how to do it in keras", "votes": 1 }, { "id": 873176, "postDate": "2020-06-03T21:54:56.533Z", "content": "hmm - did not hear this before - sounds plausible - can one say this gives more often better results than one lr for head and backbone? What is your experience?
", "rawMarkdown": "hmm - did not hear this before - sounds plausible - can one say this gives more often better results than one lr for head and backbone? What is your experience?" }, { "id": 873206, "postDate": "2020-06-03T22:40:48.910Z", "content": "For me it doesnt really change much, but the idea is since the features are pretrained they just need finetuning and the classifier parameters are set randomly so they need more trainning. However i think this idea is more valuable when training with images similar to imagenet, not sure though..
", "rawMarkdown": "For me it doesnt really change much, but the idea is since the features are pretrained they just need finetuning and the classifier parameters are set randomly so they need more trainning. However i think this idea is more valuable when training with images similar to imagenet, not sure though.." }, { "id": 873462, "postDate": "2020-06-04T06:59:38.260Z", "content": "You can do it with tf.keras.backend.stop_gradient()
inp = tf.keras.layers.Input(shape=(*IMAGE_SIZE,3))\n net = efn.EfficientNetB3(weights='imagenet', include_top=False)\n x = net(inp)\n x = tf.keras.layers.GlobalAveragePooling2D()(x)\n x = 0.005*x + 0.995*tf.keras.backend.stop_gradient(x)\n x = tf.keras.layers.Dense(512, activation='relu')(x)\n x = tf.keras.layers.Dense(128, activation='relu')(x)\n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\nThen if LR = 0.02, you stop 99.5% of that, so the EfficientNet (backbone) only gets LR = 0.0001
Are you using TPUs for training the model? I have similar model architecture but while training, I encountered following error:
\n\nNotFoundError: {{function_node __inference_distributed_function_431538}} No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}} .
\n\nRegistered: \n[[PyFunc]] \n[[MultiDeviceIteratorGetNextFromShard]] \n[[RemoteCall]] \n[[IteratorGetNextAsOptional]]
\n\nPlease help.
\n\nThank You!
", "rawMarkdown": "Are you using TPUs for training the model? I have similar model architecture but while training, I encountered following error:\n\nNotFoundError: {{function_node __inference_distributed_function_431538}} No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}} . \n\nRegistered: ", "replies": [ { "id": 872006, "postDate": "2020-06-02T20:01:19.147Z", "content": "I'm currently using GPU but planning to switch to TPU later. Will let you know how it went.
", "rawMarkdown": "I'm currently using GPU but planning to switch to TPU later. Will let you know how it went." } ] }, { "id": 871760, "postDate": "2020-06-02T16:07:08.110Z", "content": "Hey very nice results, ive also experimented with resnet18 and was able to get similar results! However what loss are you using? for me FocalLoss gave me slightly better results!
", "rawMarkdown": "Hey very nice results, ive also experimented with resnet18 and was able to get similar results! However what loss are you using? for me FocalLoss gave me slightly better results!", "replies": [ { "id": 871842, "postDate": "2020-06-02T17:11:53.190Z", "content": "I'm on BCE.
", "rawMarkdown": "I'm on BCE. " } ] }, { "id": 873355, "postDate": "2020-06-04T04:33:53.353Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 873448, "postDate": "2020-06-04T06:45:20.100Z", "content": "4 types of rotations and 2 types of vertical flips (regular image + flipped) makes 8 total different augmentation options for training
", "rawMarkdown": "4 types of rotations and 2 types of vertical flips (regular image + flipped) makes 8 total different augmentation options for training", "votes": 1 }, { "id": 873636, "postDate": "2020-06-04T10:14:11.777Z", "content": "Exactly.
", "rawMarkdown": "Exactly." }, { "id": 873761, "postDate": "2020-06-04T12:09:40.343Z", "content": "Why wouldn't you just randomrotate in a range? Is it better to do 4 different rotation choices?
", "rawMarkdown": "Why wouldn't you just randomrotate in a range? Is it better to do 4 different rotation choices?" }, { "id": 873821, "postDate": "2020-06-04T13:04:01.747Z", "content": "Good question. One advantage of 90 deg rotations is that they do not require interpolation, which results in slightly better image quality. However, since I resize the images anyway, this is unlikely to be a big deal. I guess you could just try different options and compare them.
", "rawMarkdown": "Good question. One advantage of 90 deg rotations is that they do not require interpolation, which results in slightly better image quality. However, since I resize the images anyway, this is unlikely to be a big deal. I guess you could just try different options and compare them." } ] } ], "comments": [ { "id": 871707, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-02T15:29:11.737000", "content": "Great job. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 256x256 here. Then after you write you code, you can switch the 256x256 dataset with my 512x512 or 768x768 for a higher CV/LB. Enjoy!
\n\nRotation augmentation for TFRecords is explained here. And CutMix and MixUp here
", "votes": 7, "replies": [] }, { "id": 871745, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-02T15:55:19.697000", "content": "Interestingly, so far switching to 512x512 resolutions makes the result worse. I suspect that's because ResNet18 has quite a small receptive field and couldn't fully utilize higher resolution.
", "votes": 3, "replies": [ { "id": 871749, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-02T15:58:58.120000", "content": "yes, that is interesting
", "votes": 0, "replies": [] }, { "id": 876217, "author_name": "", "author_url": "", "post_date": "2020-06-06T14:41:14.853000", "content": "Better try that resolution on which it was pretrained.This will boost the model's efficiency...
", "votes": 1, "replies": [] } ] }, { "id": 884438, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2020-06-13T11:28:55.857000", "content": "> 4 rotations + vertical flips (8 options total)
\n\nIs this a good way of implementing it?\n\nimport albumentations as A\naugs = A.Compose([\n A.OneOf([\n A.Rotate(limit=(0, 0), p=1),\n A.Rotate(limit=(90, 90), p=1),\n A.Rotate(limit=(180, 180), p=1),\n A.Rotate(limit=(270, 270), p=1)]),\n A.VerticalFlip(p=0.5)\n])\n\nOr is there a more concise code?
I guess I used RandomRotate90 :)
Well, it seems this is much more simple :)
", "votes": 0, "replies": [] } ] }, { "id": 876010, "author_name": "Gaurav Sharma", "author_url": "", "post_date": "2020-06-06T11:08:26.473000", "content": "@ddanevskyi have you tried more epochs?
", "votes": -1, "replies": [ { "id": 876028, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-06T11:18:35.860000", "content": "Yes. For this setup, there was no difference.
", "votes": 0, "replies": [] }, { "id": 876144, "author_name": "Gaurav Sharma", "author_url": "", "post_date": "2020-06-06T13:27:52.363000", "content": "are these results are on the provided dataset only or you used external data as well?
", "votes": -1, "replies": [] }, { "id": 876171, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-06T13:53:26.873000", "content": "I'm using only the competition data.
", "votes": 0, "replies": [] } ] }, { "id": 871790, "author_name": "Satwik", "author_url": "", "post_date": "2020-06-02T16:28:18.627000", "content": "Really interesting results. Could you share a notebook for the same? Also , what exactly do you mean by average of the last 4 epochs? Also, do you think it is possible to run notebooks for this comp on a 6GB 1660Ti?
", "votes": -2, "replies": [ { "id": 871860, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-02T17:19:24.323000", "content": "I just averaged predictions on the test set from the last 4 epochs (I predict both validation and test on each epoch).
\n\nAs for hardware, 1660Ti might be fine for shallow models (such as ResNet18), but deeper models (which are a must for a high score) are better be trained on TPUs.
", "votes": 1, "replies": [] }, { "id": 871896, "author_name": "Satwik", "author_url": "", "post_date": "2020-06-02T18:00:37.773000", "content": "Oh , I see. Thanks for the reply. Guess I'll just have to work with training with checkpoints on Kaggle TPU's :(
", "votes": 0, "replies": [] } ] }, { "id": 888415, "author_name": "wqz111", "author_url": "", "post_date": "2020-06-16T10:34:50.153000", "content": "hi！I tried to train the dataset with resnet50, the number of pictures with true label is 584, the number of false is 32000+, that is imbalanced, I trained resnet50 using your strategy, when I predict, the output label for the test images are all 0, can you tell me why? Thank you!
", "votes": 0, "replies": [ { "id": 888437, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-16T11:01:40.483000", "content": "I can't. What is your AUC score on the training set?
", "votes": 0, "replies": [] }, { "id": 888458, "author_name": "wqz111", "author_url": "", "post_date": "2020-06-16T11:21:55.710000", "content": "I am changing the code to add this function, is there a threshold to tell the bad from the good, I set it 0.5 before, need I change it to a bigger one ?
", "votes": 0, "replies": [] }, { "id": 888472, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-16T11:34:12.873000", "content": "You can keep it 0 or None too, just share the final AUC score on the training set here! :)
Hey, I have a issue with my setup, I am using,
\n\nI am getting a CV of about 0.72 and that too is fluctuating a lot, it sometimes drops to even 0.3. :/
\n\nI am still a newbie, any suggestions on how to improve this would be highly appreciated.
", "votes": 0, "replies": [ { "id": 882271, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-11T17:15:27.297000", "content": "Sounds like a bug. Do you normalize images? Are you sure your training/validation/testing pipelines are the same?
", "votes": 0, "replies": [] }, { "id": 882278, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-11T17:19:32.947000", "content": "Yes, I am normalizing them. Yeah, pipeline is same too.
\n\nI am thinking of running it for more epochs now, maybe 20, to see if it starts to converge then.
", "votes": 0, "replies": [] }, { "id": 882357, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-06-11T18:30:51.577000", "content": "It should converge in a couple of epochs, are you reducing your lr during training with a scheduler?
", "votes": 0, "replies": [] }, { "id": 882627, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-12T02:22:52.707000", "content": "Yes, I am using ReduceLROnPlateau as a Scheduler with a patience of 3.
\n\nAlso, I tried reducing the number of groups to 3 and auc is now going upto 0.8 now, originally it was only reaching 0.72 and it seems much stable than before.
\n\nAlthough I know I can get at least 0.9 CV with this setup, it still is a long way to go.
", "votes": 0, "replies": [] }, { "id": 883925, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-13T04:57:42.793000", "content": "Also I want to add one more thing, I am predicting on 2 classes with cross entropy loss, can it be causing this much fluctuation and less auc?
", "votes": 0, "replies": [] }, { "id": 883957, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-13T06:31:20.873000", "content": "I presume almost everyone is using cross-entropy.
", "votes": 0, "replies": [] }, { "id": 883973, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-13T06:34:48.163000", "content": "Okay, then there must be something else which is off, I will be looking for that then. Thanks! :)
", "votes": 0, "replies": [] }, { "id": 884489, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-13T12:13:17.547000", "content": "Can I share my notebook with anyone here? it may be a good deal for me to just get it quick checked with someone else, maybe I can't see what's going wrong and I have already dedicated 7 days to this particular problem. :/
", "votes": 0, "replies": [] }, { "id": 884617, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-13T13:46:48.153000", "content": "You could just make your notebook public. Sharing with someone privately without being in a team is prohibited.
", "votes": 0, "replies": [] }, { "id": 884624, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-13T13:53:47.127000", "content": "Sure, I will do that, will comment here after making it public. Thanks! :)
", "votes": 0, "replies": [] }, { "id": 884729, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-06-13T15:26:26.410000", "content": "I made the notebook public.
\n\nIt performed a lot better than the last time as that time the score used to drop to 0.4 sometimes but I think it will still be good if you can take a look.
\n\nI took some part of it from other notebooks too as I was just learning how to work with PyTorch. :)
", "votes": 0, "replies": [] } ] }, { "id": 873057, "author_name": "Jonas Matuzas", "author_url": "", "post_date": "2020-06-03T18:25:52.160000", "content": "\"0.0001 LR for ResNet and 0.02 for randomly initialized classification layer\"- How LR can be different for ResNet end classification layer ? How to write it in tf.keras ?
", "votes": 0, "replies": [ { "id": 873070, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-06-03T18:43:50.940000", "content": "you would do do something like this where model.base is your features and fc is your classifier:
\n\n\noptimizer = optim.AdamW([{'params': model.base.parameters(), 'lr ':1e-4 },\n {'params': model.fc.parameters(), 'lr ':2e-2 },\n ], lr=lr)\n
This is usefull if you are using a pretrained architechture
", "votes": 6, "replies": [] }, { "id": 873081, "author_name": "Jonas Matuzas", "author_url": "", "post_date": "2020-06-03T19:08:10.077000", "content": "how to initialize model.base and model.fc ? Is it example in tf.keras?
", "votes": 0, "replies": [] }, { "id": 873094, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-06-03T19:31:07.093000", "content": "Sorry i dont really know how to do it in keras
", "votes": 1, "replies": [] }, { "id": 873176, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2020-06-03T21:54:56.533000", "content": "hmm - did not hear this before - sounds plausible - can one say this gives more often better results than one lr for head and backbone? What is your experience?
", "votes": 0, "replies": [] }, { "id": 873206, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-06-03T22:40:48.910000", "content": "For me it doesnt really change much, but the idea is since the features are pretrained they just need finetuning and the classifier parameters are set randomly so they need more trainning. However i think this idea is more valuable when training with images similar to imagenet, not sure though..
", "votes": 0, "replies": [] }, { "id": 873462, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-04T06:59:38.260000", "content": "You can do it with tf.keras.backend.stop_gradient()
inp = tf.keras.layers.Input(shape=(*IMAGE_SIZE,3))\n net = efn.EfficientNetB3(weights='imagenet', include_top=False)\n x = net(inp)\n x = tf.keras.layers.GlobalAveragePooling2D()(x)\n x = 0.005*x + 0.995*tf.keras.backend.stop_gradient(x)\n x = tf.keras.layers.Dense(512, activation='relu')(x)\n x = tf.keras.layers.Dense(128, activation='relu')(x)\n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\nThen if LR = 0.02, you stop 99.5% of that, so the EfficientNet (backbone) only gets LR = 0.0001
Are you using TPUs for training the model? I have similar model architecture but while training, I encountered following error:
\n\nNotFoundError: {{function_node __inference_distributed_function_431538}} No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}} .
\n\nRegistered: \n[[PyFunc]] \n[[MultiDeviceIteratorGetNextFromShard]] \n[[RemoteCall]] \n[[IteratorGetNextAsOptional]]
\n\nPlease help.
\n\nThank You!
", "votes": 0, "replies": [ { "id": 872006, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-02T20:01:19.147000", "content": "I'm currently using GPU but planning to switch to TPU later. Will let you know how it went.
", "votes": 0, "replies": [] } ] }, { "id": 871760, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-06-02T16:07:08.110000", "content": "Hey very nice results, ive also experimented with resnet18 and was able to get similar results! However what loss are you using? for me FocalLoss gave me slightly better results!
", "votes": 0, "replies": [ { "id": 871842, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-02T17:11:53.190000", "content": "I'm on BCE.
", "votes": 0, "replies": [] } ] }, { "id": 873355, "author_name": "", "author_url": "", "post_date": "2020-06-04T04:33:53.353000", "content": "", "votes": 0, "replies": [ { "id": 873448, "author_name": "Hunter Mitchell", "author_url": "", "post_date": "2020-06-04T06:45:20.100000", "content": "4 types of rotations and 2 types of vertical flips (regular image + flipped) makes 8 total different augmentation options for training
", "votes": 1, "replies": [] }, { "id": 873636, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-04T10:14:11.777000", "content": "Exactly.
", "votes": 0, "replies": [] }, { "id": 873761, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-06-04T12:09:40.343000", "content": "Why wouldn't you just randomrotate in a range? Is it better to do 4 different rotation choices?
", "votes": 0, "replies": [] }, { "id": 873821, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2020-06-04T13:04:01.747000", "content": "Good question. One advantage of 90 deg rotations is that they do not require interpolation, which results in slightly better image quality. However, since I resize the images anyway, this is unlikely to be a big deal. I guess you could just try different options and compare them.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "871656": "My current setup:\n\n* ResNet18\n* Images resized to 256x256 \n* Adam\n* 0.0001 LR for ResNet and 0.02 for randomly initialized classification layer\n* 4 rotations + vertical flips (8 options total) on training\n* 10 epochs\n* batch size is 64\n* 3 folds (group + stratify)\n* Average of the last 4 epochs -> gives slightly more stable results than just last/best epoch\n* Training time is less than an hour on a 1080ti\n\n0.899 CV, 0.903 LB", "871707": "Great job. If people want to recreate this in Kaggle notebooks using GPU or TPU, I uploaded TFRecords containing 256x256 [here][1]. Then after you write you code, you can switch the 256x256 dataset with my 512x512 or 768x768 for a higher CV/LB. Enjoy!\n\nRotation augmentation for TFRecords is explained [here][2]. And CutMix and MixUp [here][3]\n\n[1]: https://www.kaggle.com/cdeotte/melanoma-256x256\n[2]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132191\n[3]: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/132935", "871745": "Interestingly, so far switching to 512x512 resolutions makes the result **worse**. I suspect that's because ResNet18 has quite a small receptive field and couldn't fully utilize higher resolution.", "884438": "> 4 rotations + vertical flips (8 options total)\n\nIs this a good way of implementing it?\n```\nimport albumentations as A\naugs = A.Compose([\n A.OneOf([\n A.Rotate(limit=(0, 0), p=1),\n A.Rotate(limit=(90, 90), p=1),\n A.Rotate(limit=(180, 180), p=1),\n A.Rotate(limit=(270, 270), p=1)]),\n A.VerticalFlip(p=0.5)\n])\n```\nOr is there a more concise code?", "876010": "@ddanevskyi have you tried more epochs?", "871790": "Really interesting results. Could you share a notebook for the same? Also , what exactly do you mean by average of the last 4 epochs? Also, do you think it is possible to run notebooks for this comp on a 6GB 1660Ti?", "888415": "hi！I tried to train the dataset with resnet50, the number of pictures with true label is 584, the number of false is 32000+, that is imbalanced, I trained resnet50 using your strategy, when I predict, the output label for the test images are all 0, can you tell me why? Thank you! ", "882245": "Hey, I have a issue with my setup, I am using,\n\n- Efficientnet-b0\n- 224x224\n- Adam\n- 0.0001 LR \n- Random Horizontal and Vertical flips for training data\n- 10 epochs\n- batch_size = 64\n- 5 folds (Group)\n\nI am getting a CV of about 0.72 and that too is fluctuating a lot, it sometimes drops to even 0.3. :/ \n\nI am still a newbie, any suggestions on how to improve this would be highly appreciated. ", "873057": "\"0.0001 LR for ResNet and 0.02 for randomly initialized classification layer\"- How LR can be different for ResNet end classification layer ? How to write it in tf.keras ?", "871987": "Are you using TPUs for training the model? I have similar model architecture but while training, I encountered following error:\n\nNotFoundError: {{function_node __inference_distributed_function_431538}} No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}} . \n\nRegistered: ", "871760": "Hey very nice results, ive also experimented with resnet18 and was able to get similar results! However what loss are you using? for me FocalLoss gave me slightly better results!", "873355": "" } }