{ "id": 92159, "title": "cv/lb comparisons between se_resnext50 and 101", "url": "/competitions/imet-2019-fgvc6/discussion/92159", "author_name": "Appian", "post_date": "2019-05-13T22:29:54.483000", "votes": 45, "comment_count": 58, "views": 0, "content": "

Pretrained models are taken from https://github.com/Cadene/pretrained-models.pytorch\nThe results are very predictable. The deeper the better.

\n\n

5-folds se_resnext50_32x4d (320x320)\ncv each: 0.621 | 0.623 | 0.620 | 0.618 | 0.622\nlb each: 0.621 | ---\ncv(concat): 0.621 \nlb(mean): 0.641

\n\n

5-folds se_resnext101_32x4d (320x320)\ncv each: 0.627 | 0.629 | 0.627 | 0.628 | 0.630\nlb each: 0.630 | ---\ncv(concat): 0.628\nlb(mean): 0.652

", "messages": [ { "id": 530889, "postDate": "2019-05-13T22:29:54.483Z", "content": "

Pretrained models are taken from https://github.com/Cadene/pretrained-models.pytorch\nThe results are very predictable. The deeper the better.

\n\n

5-folds se_resnext50_32x4d (320x320)\ncv each: 0.621 | 0.623 | 0.620 | 0.618 | 0.622\nlb each: 0.621 | ---\ncv(concat): 0.621 \nlb(mean): 0.641

\n\n

5-folds se_resnext101_32x4d (320x320)\ncv each: 0.627 | 0.629 | 0.627 | 0.628 | 0.630\nlb each: 0.630 | ---\ncv(concat): 0.628\nlb(mean): 0.652

", "rawMarkdown": "Pretrained models are taken from https://github.com/Cadene/pretrained-models.pytorch\nThe results are very predictable. The deeper the better. \n\n5-folds se_resnext50\\_32x4d (320x320)\ncv each: 0.621 | 0.623 | 0.620 | 0.618 | 0.622\nlb each: 0.621 | ---\ncv(concat): 0.621 \nlb(mean): 0.641 \n\n5-folds se_resnext101\\_32x4d (320x320)\ncv each: 0.627 | 0.629 | 0.627 | 0.628 | 0.630\nlb each: 0.630 | ---\ncv(concat): 0.628\nlb(mean): 0.652\n", "votes": 45 }, { "id": 532440, "postDate": "2019-05-17T01:12:02.040Z", "content": "

Thanks for your sharing. I also use se_resnext50_32x4d, but I got a single fold CV of 0.5947. If you don't mind, could you please give some general guidance about what makes your results so good， loss function or train strategy？ I have tried many methods, but nearly none of them worked...

", "rawMarkdown": "Thanks for your sharing. I also use se_resnext50_32x4d, but I got a single fold CV of 0.5947. If you don't mind, could you please give some general guidance about what makes your results so good， loss function or train strategy？ I have tried many methods, but nearly none of them worked...", "votes": 5 }, { "id": 530997, "postDate": "2019-05-14T05:12:51.020Z", "content": "

May I ask what's your input size? Our results are not as good as you. Did you use metric learning?

", "rawMarkdown": "May I ask what's your input size? Our results are not as good as you. Did you use metric learning?", "votes": 3, "replies": [ { "id": 531176, "postDate": "2019-05-14T12:37:16.827Z", "content": "

320x320 and I don't use metric learning.

", "rawMarkdown": "320x320 and I don't use metric learning." } ] }, { "id": 531821, "postDate": "2019-05-15T15:50:18.733Z", "content": "

Hi Appian, your results are marvelous, may I ask you a few questions?\n- The above pretrained models fix the input size at 224x224. How did you modify the models to receive 320x320 images?\n- Did you crop with the size that is less than 320x320 and then resize to 320x320 or just crop to the size 320x320 with some paddings?

", "rawMarkdown": "Hi Appian, your results are marvelous, may I ask you a few questions?\n- The above pretrained models fix the input size at 224x224. How did you modify the models to receive 320x320 images?\n- Did you crop with the size that is less than 320x320 and then resize to 320x320 or just crop to the size 320x320 with some paddings?", "votes": 1, "replies": [ { "id": 531952, "postDate": "2019-05-15T23:04:08.013Z", "content": "

One way to fix is to alter nn.AvgPool2d to nn.AdaptiveAvgPool2d(1) in SENet class in senet.py. This should enable the net to accept images other than 224x224.

\n\n

UPDATE\n\nnet = senet.se_resnext101_32x4d()\nnet.avg_pool = nn.AdaptiveAvgPool2d(1)\n

", "rawMarkdown": "One way to fix is to alter nn.AvgPool2d to nn.AdaptiveAvgPool2d(1) in SENet class in senet.py. This should enable the net to accept images other than 224x224.\n\nUPDATE\n```\nnet = senet.se_resnext101_32x4d()\nnet.avg_pool = nn.AdaptiveAvgPool2d(1)\n```", "votes": 5 } ] }, { "id": 530905, "postDate": "2019-05-13T23:40:15.880Z", "content": "

How many epochs were each fold trained for if I may ask? I didn't see the exact trend with vanilla resnet models, although I feel I may have to train the larger models for longer compared to the shallower ones.

", "rawMarkdown": "How many epochs were each fold trained for if I may ask? I didn't see the exact trend with vanilla resnet models, although I feel I may have to train the larger models for longer compared to the shallower ones.", "votes": 1 }, { "id": 530911, "postDate": "2019-05-14T00:06:49.373Z", "content": "

@sairam6087\n15 epochs each but the best one came around 12-13.

\n\n

@jionie\nYes, larger image size helps. I haven't tried different n-folds other than 5. Thanks for the info.

", "rawMarkdown": "@sairam6087\n15 epochs each but the best one came around 12-13. \n\n@jionie\nYes, larger image size helps. I haven't tried different n-folds other than 5. Thanks for the info.", "votes": 2, "replies": [ { "id": 531245, "postDate": "2019-05-14T14:41:12.067Z", "content": "

@appian What batch size did you use? 8/16 i guess

", "rawMarkdown": "@appian What batch size did you use? 8/16 i guess" }, { "id": 532283, "postDate": "2019-05-16T15:13:48.437Z", "content": "

Hi appian Can it be completed in nine hours using the se_resnet iteration? How do you use se_resnet?

", "rawMarkdown": "Hi appian Can it be completed in nine hours using the se_resnet iteration? How do you use se_resnet?" }, { "id": 532416, "postDate": "2019-05-16T22:15:08.130Z", "content": "

For se_resnext50_32x4d, yes. You can train 15-16 epochs in 9 hours on kernel.

", "rawMarkdown": "For se\\_resnext50\\_32x4d, yes. You can train 15-16 epochs in 9 hours on kernel. " }, { "id": 533985, "postDate": "2019-05-20T11:36:43.273Z", "content": "

We tried to train the model on kernel.But every time the kernel was stopped in the middle of training process. Did you meet this problem?

", "rawMarkdown": "We tried to train the model on kernel.But every time the kernel was stopped in the middle of training process. Did you meet this problem? " }, { "id": 533999, "postDate": "2019-05-20T12:14:28.733Z", "content": "

@appian Are you using fastai to complete it? Or pytorch?

", "rawMarkdown": "@appian Are you using fastai to complete it? Or pytorch?" }, { "id": 534232, "postDate": "2019-05-21T01:00:10.460Z", "rawMarkdown": "", "isDeleted": true }, { "id": 535149, "postDate": "2019-05-22T11:39:22.167Z", "content": "

It takes about 50 minutes to iterate once using seresnext101. Is it normal?

", "rawMarkdown": "It takes about 50 minutes to iterate once using seresnext101. Is it normal?" } ] }, { "id": 530895, "postDate": "2019-05-13T22:43:57.683Z", "content": "

Thanks for sharing!! Although my results are poor, I found that also the bigger image size and the bigger the cv folds are better.

", "rawMarkdown": "Thanks for sharing!! Although my results are poor, I found that also the bigger image size and the bigger the cv folds are better.", "votes": 2 }, { "id": 536715, "postDate": "2019-05-25T04:14:20.390Z", "content": "

how much batch-size?

", "rawMarkdown": "how much batch-size?", "votes": -1 }, { "id": 538092, "postDate": "2019-05-28T05:56:24.507Z", "content": "

I'm using se_resnext101_32x4d (288 x 288) with proper image augmentation and regularization to get CV 0.61. However, with image size 320 x 320, the model didn't improve at all. May I ask how did you resize the image? Did you simply use RandomSizedCrop or Resize + RandomCrop? Thanks.

", "rawMarkdown": "I'm using se_resnext101_32x4d (288 x 288) with proper image augmentation and regularization to get CV 0.61. However, with image size 320 x 320, the model didn't improve at all. May I ask how did you resize the image? Did you simply use RandomSizedCrop or Resize + RandomCrop? Thanks.", "replies": [ { "id": 538298, "postDate": "2019-05-28T11:20:38.547Z", "content": "

I use RandomResizedCrop from torchvision with a bit of modification.\nI haven't tried the latter case but it seems it changes aspect ratio too much in some cases (ex. from 300x1500 to 300x300?) which might not be good.

", "rawMarkdown": "I use RandomResizedCrop from torchvision with a bit of modification.\nI haven't tried the latter case but it seems it changes aspect ratio too much in some cases (ex. from 300x1500 to 300x300?) which might not be good.", "votes": 1 } ] }, { "id": 537304, "postDate": "2019-05-26T17:00:53.240Z", "content": "

@appian are these vanilla results? without any preprocessing or changing the network at all? Do you train all the layers? Also, when I change the avg_pool layer to AdaptiveAvgPool2d, everything goes out of memory\neven for a batch size of 4. Could you please tell me if I am doing something wrong and how to\nfix it?

", "rawMarkdown": "@appian are these vanilla results? without any preprocessing or changing the network at all? Do you train all the layers? Also, when I change the avg_pool layer to AdaptiveAvgPool2d, everything goes out of memory\neven for a batch size of 4. Could you please tell me if I am doing something wrong and how to\nfix it?", "replies": [ { "id": 537373, "postDate": "2019-05-26T21:27:10.817Z", "content": "

I do some augmentations before feeding images to NN. There are a few other things to make results better such as training a part of the network just like you mentioned but not changing the network itself except replacing avg_pool.\nI have no clue regarding the memory error. All I did was just replacing avg_pool and the number of out classes. How about debugging your forward network by displaying the shape of the tensor and compare it with the network before you modified to see what's causing the memory error.

", "rawMarkdown": "I do some augmentations before feeding images to NN. There are a few other things to make results better such as training a part of the network just like you mentioned but not changing the network itself except replacing avg\\_pool.\nI have no clue regarding the memory error. All I did was just replacing avg\\_pool and the number of out classes. How about debugging your forward network by displaying the shape of the tensor and compare it with the network before you modified to see what's causing the memory error.", "votes": 2 }, { "id": 538178, "postDate": "2019-05-28T09:17:26.447Z", "content": "

May I ask what the loss function you use? Is the focal loss function? Or some else?

", "rawMarkdown": "May I ask what the loss function you use? Is the focal loss function? Or some else?" } ] }, { "id": 537051, "postDate": "2019-05-26T04:52:00.440Z", "content": "

What is the full name of cv and lb?

", "rawMarkdown": "What is the full name of *cv* and *lb*?", "replies": [ { "id": 537069, "postDate": "2019-05-26T05:41:28.140Z", "content": "

CrossValidation score and LeadBoard score

", "rawMarkdown": "CrossValidation score and LeadBoard score", "votes": 2 } ] }, { "id": 534093, "postDate": "2019-05-20T16:58:58.603Z", "content": "

Whats your training time for these models?

", "rawMarkdown": "Whats your training time for these models?", "replies": [ { "id": 534204, "postDate": "2019-05-20T23:01:29.060Z", "content": "

Approx. number till the best epoch.

\n\n\n\n

All models are trained on kaggle kernels.

", "rawMarkdown": "Approx. number till the best epoch.\n\n- se\\_resnext50\\_32x4d (7 hours)\n- se\\_resnext101\\_32x4d (13 hours)\n\nAll models are trained on kaggle kernels.", "votes": 2 }, { "id": 534276, "postDate": "2019-05-21T02:37:18.370Z", "content": "

se_resnext101_32x4d (13 hours)\nHow to train more than 8 hours?

", "rawMarkdown": "`se_resnext101_32x4d (13 hours)`\nHow to train more than 8 hours?" }, { "id": 534483, "postDate": "2019-05-21T09:59:40.320Z", "content": "

Train the model for 9 hours. Download the model. Upload it back and train it from where you left.

", "rawMarkdown": "Train the model for 9 hours. Download the model. Upload it back and train it from where you left.", "votes": 2 } ] }, { "id": 534054, "postDate": "2019-05-20T14:39:28.670Z", "content": "

how to use pretrainedmodels? This is kernels-only competition. Should I rebuild the model and load the weights?

", "rawMarkdown": "how to use pretrainedmodels? This is kernels-only competition. Should I rebuild the model and load the weights?", "replies": [ { "id": 534060, "postDate": "2019-05-20T14:56:26.903Z", "content": "

I find my way to use seresnext. I copy the file from pretrainedmodels and build the model.

", "rawMarkdown": "I find my way to use seresnext. I copy the file from pretrainedmodels and build the model." } ] }, { "id": 533878, "postDate": "2019-05-20T07:12:28.210Z", "content": "

Appian,\nwhat about senet154? In my experiments it is worse than resnext.

", "rawMarkdown": "Appian,\nwhat about senet154? In my experiments it is worse than resnext.", "replies": [ { "id": 534203, "postDate": "2019-05-20T22:59:26.757Z", "content": "

For me it was just as good as se_resnext101. It could be better because I did not do parameter tuning.

", "rawMarkdown": "For me it was just as good as se_resnext101. It could be better because I did not do parameter tuning.", "votes": 1 } ] }, { "id": 531124, "postDate": "2019-05-14T10:58:44.810Z", "content": "

crop or resize_crop?

", "rawMarkdown": "crop or resize_crop?\n", "replies": [ { "id": 531175, "postDate": "2019-05-14T12:36:55.927Z", "content": "

ResizedCrop

", "rawMarkdown": "ResizedCrop" }, { "id": 531251, "postDate": "2019-05-14T14:48:35.093Z", "content": "

Thank you for your sharing!
\nYou mean that your augmentation is Resizedcrop?

", "rawMarkdown": "Thank you for your sharing! \nYou mean that your augmentation is Resizedcrop?" } ] }, { "id": 531105, "postDate": "2019-05-14T10:10:40.380Z", "content": "

Is it possible for you guys to share links on how to perform proper CV Folds on image classification? Many thanks! :-)

", "rawMarkdown": "Is it possible for you guys to share links on how to perform proper CV Folds on image classification? Many thanks! :-)", "replies": [ { "id": 531179, "postDate": "2019-05-14T12:38:45.410Z", "content": "

I recommend Lopuhin's code for making folds. This is very good.\nhttps://github.com/lopuhin/kaggle-imet-2019/blob/master/imet/make_folds.py

", "rawMarkdown": "I recommend Lopuhin's code for making folds. This is very good.\nhttps://github.com/lopuhin/kaggle-imet-2019/blob/master/imet/make_folds.py", "votes": 2 } ] }, { "id": 531023, "postDate": "2019-05-14T06:26:08.297Z", "content": "

May I ask what's your loss function?

", "rawMarkdown": "May I ask what's your loss function?" }, { "id": 531006, "postDate": "2019-05-14T05:41:40.650Z", "content": "

Hi Appian. We usually get a validation score like 0.59-0.60 (0.597 for me), but you get 0.62. That's so amazing. I tried many methods but none of them seems really work well. Can you share some tips about this if you don't mind? Really appreciate.

", "rawMarkdown": "Hi Appian. We usually get a validation score like 0.59-0.60 (0.597 for me), but you get 0.62. That's so amazing. I tried many methods but none of them seems really work well. Can you share some tips about this if you don't mind? Really appreciate.", "replies": [ { "id": 531017, "postDate": "2019-05-14T06:14:34.947Z", "content": "

你好，请问你所使用的模型是 se_resnext吗？

", "rawMarkdown": "你好，请问你所使用的模型是 se_resnext吗？", "votes": -1 }, { "id": 531029, "postDate": "2019-05-14T06:38:51.773Z", "content": "

Resnet101 works for me. CV 0.597 LB 0.597

", "rawMarkdown": "Resnet101 works for me. CV 0.597 LB 0.597" }, { "id": 531110, "postDate": "2019-05-14T10:17:22.600Z", "content": "

Thank you for sharing!

", "rawMarkdown": "Thank you for sharing!" }, { "id": 531235, "postDate": "2019-05-14T14:23:34.230Z", "content": "

What framework do you use with resnet101?

", "rawMarkdown": "What framework do you use with resnet101?" }, { "id": 534371, "postDate": "2019-05-21T06:18:52Z", "content": "

Hellow. Do you use the Kaggle-kernel to train your models? We tried to use it ,but the kernel is very astable. It will be disconnected every time. Do you meet this problem? And may I ask how you solve it.

", "rawMarkdown": "Hellow. Do you use the Kaggle-kernel to train your models? We tried to use it ,but the kernel is very astable. It will be disconnected every time. Do you meet this problem? And may I ask how you solve it." }, { "id": 535920, "postDate": "2019-05-23T16:18:51.600Z", "rawMarkdown": "", "votes": -1 }, { "id": 537122, "postDate": "2019-05-26T08:25:36.377Z", "content": "

@fengziyuan I also faced this problem using Kaggle kernels. It disconnects often. I am not able to test the code in kernel if I have to wait some epochs. One solution is to commit the kernel.

", "rawMarkdown": "@fengziyuan I also faced this problem using Kaggle kernels. It disconnects often. I am not able to test the code in kernel if I have to wait some epochs. One solution is to commit the kernel. " } ] }, { "id": 530953, "postDate": "2019-05-14T03:16:49.717Z", "content": "

Thank you for your sharing. I wonder what is the data-set that your each cv score based on? Is the val of each K-fold model? Or is the test data-set ? Or is the whole train set?

", "rawMarkdown": "Thank you for your sharing. I wonder what is the data-set that your each cv score based on? Is the val of each K-fold model? Or is the test data-set ? Or is the whole train set? ", "replies": [ { "id": 531181, "postDate": "2019-05-14T12:40:41.110Z", "content": "

I meant \"cv each\" for validation score of each fold.\n\"cv concat\" is validation score on entire train data (oof).

", "rawMarkdown": "I meant \"cv each\" for validation score of each fold.\n\"cv concat\" is validation score on entire train data (oof)." }, { "id": 531207, "postDate": "2019-05-14T13:21:41.030Z", "content": "

Thank you for sharing!

", "rawMarkdown": "Thank you for sharing!" } ] }, { "id": 530946, "postDate": "2019-05-14T02:46:10.640Z", "content": "

May I ask how dou you install these external packages when you submit the kernel?（Network connection is not allowed in the kernel.）

", "rawMarkdown": "May I ask how dou you install these external packages when you submit the kernel?（Network connection is not allowed in the kernel.）", "replies": [ { "id": 530962, "postDate": "2019-05-14T03:55:45.537Z", "content": "

the same question

", "rawMarkdown": "the same question" }, { "id": 531003, "postDate": "2019-05-14T05:30:42.017Z", "content": "

the same question

", "rawMarkdown": "the same question" }, { "id": 531065, "postDate": "2019-05-14T08:18:15.417Z", "content": "

You can download the source codes of the external packages and upload them into your dataset. Then import them.

", "rawMarkdown": "You can download the source codes of the external packages and upload them into your dataset. Then import them.", "votes": 3 }, { "id": 531151, "postDate": "2019-05-14T11:48:38.473Z", "content": "

@seefun OK, thanks for your reply!

", "rawMarkdown": "@seefun OK, thanks for your reply!" }, { "id": 531476, "postDate": "2019-05-15T01:43:31.433Z", "content": "

@seefun The \"init.py\" in the source codes could not be uploaded to the dataset( error: this file already exists in competition..). Have you encountered such a problem? Can you share some tips about this if you don't mind? Really appreciate.

", "rawMarkdown": "@seefun The \"__init__.py\" in the source codes could not be uploaded to the dataset( error: this file already exists in competition..). Have you encountered such a problem? Can you share some tips about this if you don't mind? Really appreciate." }, { "id": 531737, "postDate": "2019-05-15T12:50:47.653Z", "content": "

I did not encounter this problem. I only zip the python package source code and upload it to kaggle dataset. And then import them. I guess you didn't zip a python package as a .zip file before upload them.

", "rawMarkdown": "I did not encounter this problem. I only zip the python package source code and upload it to kaggle dataset. And then import them. I guess you didn't zip a python package as a .zip file before upload them.", "votes": 2 }, { "id": 532443, "postDate": "2019-05-17T01:23:23.203Z", "content": "

@seefun Ok, thank you for your guidance.

", "rawMarkdown": "@seefun Ok, thank you for your guidance." }, { "id": 532867, "postDate": "2019-05-17T23:35:40.033Z", "content": "

You can just take SENet.py from pretrainedmodels and include it into your solution, pretty sure this is what Appian did :)

", "rawMarkdown": "You can just take SENet.py from pretrainedmodels and include it into your solution, pretty sure this is what Appian did :)", "votes": 3 }, { "id": 533052, "postDate": "2019-05-18T10:55:57.193Z", "content": "

Yes, that's what I do.

", "rawMarkdown": "Yes, that's what I do. " } ] } ], "comments": [ { "id": 532440, "author_name": "QunYang", "author_url": "", "post_date": "2019-05-17T01:12:02.040000", "content": "

Thanks for your sharing. I also use se_resnext50_32x4d, but I got a single fold CV of 0.5947. If you don't mind, could you please give some general guidance about what makes your results so good， loss function or train strategy？ I have tried many methods, but nearly none of them worked...

", "votes": 5, "replies": [] }, { "id": 530997, "author_name": "seefun", "author_url": "", "post_date": "2019-05-14T05:12:51.020000", "content": "

May I ask what's your input size? Our results are not as good as you. Did you use metric learning?

", "votes": 3, "replies": [ { "id": 531176, "author_name": "Appian", "author_url": "", "post_date": "2019-05-14T12:37:16.827000", "content": "

320x320 and I don't use metric learning.

", "votes": 0, "replies": [] } ] }, { "id": 531821, "author_name": "Putalay", "author_url": "", "post_date": "2019-05-15T15:50:18.733000", "content": "

Hi Appian, your results are marvelous, may I ask you a few questions?\n- The above pretrained models fix the input size at 224x224. How did you modify the models to receive 320x320 images?\n- Did you crop with the size that is less than 320x320 and then resize to 320x320 or just crop to the size 320x320 with some paddings?

", "votes": 1, "replies": [ { "id": 531952, "author_name": "Appian", "author_url": "", "post_date": "2019-05-15T23:04:08.013000", "content": "

One way to fix is to alter nn.AvgPool2d to nn.AdaptiveAvgPool2d(1) in SENet class in senet.py. This should enable the net to accept images other than 224x224.

\n\n

UPDATE\n\nnet = senet.se_resnext101_32x4d()\nnet.avg_pool = nn.AdaptiveAvgPool2d(1)\n

", "votes": 5, "replies": [] } ] }, { "id": 530905, "author_name": "Sairam Sundaresan", "author_url": "", "post_date": "2019-05-13T23:40:15.880000", "content": "

How many epochs were each fold trained for if I may ask? I didn't see the exact trend with vanilla resnet models, although I feel I may have to train the larger models for longer compared to the shallower ones.

", "votes": 1, "replies": [] }, { "id": 530911, "author_name": "Appian", "author_url": "", "post_date": "2019-05-14T00:06:49.373000", "content": "

@sairam6087\n15 epochs each but the best one came around 12-13.

\n\n

@jionie\nYes, larger image size helps. I haven't tried different n-folds other than 5. Thanks for the info.

", "votes": 2, "replies": [ { "id": 531245, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2019-05-14T14:41:12.067000", "content": "

@appian What batch size did you use? 8/16 i guess

", "votes": 0, "replies": [] }, { "id": 532283, "author_name": "GUTAR", "author_url": "", "post_date": "2019-05-16T15:13:48.437000", "content": "

Hi appian Can it be completed in nine hours using the se_resnet iteration? How do you use se_resnet?

", "votes": 0, "replies": [] }, { "id": 532416, "author_name": "Appian", "author_url": "", "post_date": "2019-05-16T22:15:08.130000", "content": "

For se_resnext50_32x4d, yes. You can train 15-16 epochs in 9 hours on kernel.

", "votes": 0, "replies": [] }, { "id": 533985, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-20T11:36:43.273000", "content": "

We tried to train the model on kernel.But every time the kernel was stopped in the middle of training process. Did you meet this problem?

", "votes": 0, "replies": [] }, { "id": 533999, "author_name": "GUTAR", "author_url": "", "post_date": "2019-05-20T12:14:28.733000", "content": "

@appian Are you using fastai to complete it? Or pytorch?

", "votes": 0, "replies": [] }, { "id": 534232, "author_name": "", "author_url": "", "post_date": "2019-05-21T01:00:10.460000", "content": "", "votes": 0, "replies": [] }, { "id": 535149, "author_name": "GUTAR", "author_url": "", "post_date": "2019-05-22T11:39:22.167000", "content": "

It takes about 50 minutes to iterate once using seresnext101. Is it normal?

", "votes": 0, "replies": [] } ] }, { "id": 530895, "author_name": "jionie", "author_url": "", "post_date": "2019-05-13T22:43:57.683000", "content": "

Thanks for sharing!! Although my results are poor, I found that also the bigger image size and the bigger the cv folds are better.

", "votes": 2, "replies": [] }, { "id": 536715, "author_name": "", "author_url": "", "post_date": "2019-05-25T04:14:20.390000", "content": "

how much batch-size?

", "votes": -1, "replies": [] }, { "id": 538092, "author_name": "syoya", "author_url": "", "post_date": "2019-05-28T05:56:24.507000", "content": "

I'm using se_resnext101_32x4d (288 x 288) with proper image augmentation and regularization to get CV 0.61. However, with image size 320 x 320, the model didn't improve at all. May I ask how did you resize the image? Did you simply use RandomSizedCrop or Resize + RandomCrop? Thanks.

", "votes": 0, "replies": [ { "id": 538298, "author_name": "Appian", "author_url": "", "post_date": "2019-05-28T11:20:38.547000", "content": "

I use RandomResizedCrop from torchvision with a bit of modification.\nI haven't tried the latter case but it seems it changes aspect ratio too much in some cases (ex. from 300x1500 to 300x300?) which might not be good.

", "votes": 1, "replies": [] } ] }, { "id": 537304, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2019-05-26T17:00:53.240000", "content": "

@appian are these vanilla results? without any preprocessing or changing the network at all? Do you train all the layers? Also, when I change the avg_pool layer to AdaptiveAvgPool2d, everything goes out of memory\neven for a batch size of 4. Could you please tell me if I am doing something wrong and how to\nfix it?

", "votes": 0, "replies": [ { "id": 537373, "author_name": "Appian", "author_url": "", "post_date": "2019-05-26T21:27:10.817000", "content": "

I do some augmentations before feeding images to NN. There are a few other things to make results better such as training a part of the network just like you mentioned but not changing the network itself except replacing avg_pool.\nI have no clue regarding the memory error. All I did was just replacing avg_pool and the number of out classes. How about debugging your forward network by displaying the shape of the tensor and compare it with the network before you modified to see what's causing the memory error.

", "votes": 2, "replies": [] }, { "id": 538178, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-28T09:17:26.447000", "content": "

May I ask what the loss function you use? Is the focal loss function? Or some else?

", "votes": 0, "replies": [] } ] }, { "id": 537051, "author_name": "", "author_url": "", "post_date": "2019-05-26T04:52:00.440000", "content": "

What is the full name of cv and lb?

", "votes": 0, "replies": [ { "id": 537069, "author_name": "seefun", "author_url": "", "post_date": "2019-05-26T05:41:28.140000", "content": "

CrossValidation score and LeadBoard score

", "votes": 2, "replies": [] } ] }, { "id": 534093, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2019-05-20T16:58:58.603000", "content": "

Whats your training time for these models?

", "votes": 0, "replies": [ { "id": 534204, "author_name": "Appian", "author_url": "", "post_date": "2019-05-20T23:01:29.060000", "content": "

Approx. number till the best epoch.

\n\n\n\n

All models are trained on kaggle kernels.

", "votes": 2, "replies": [] }, { "id": 534276, "author_name": "Timmmmmms", "author_url": "", "post_date": "2019-05-21T02:37:18.370000", "content": "

se_resnext101_32x4d (13 hours)\nHow to train more than 8 hours?

", "votes": 0, "replies": [] }, { "id": 534483, "author_name": "Filemon", "author_url": "", "post_date": "2019-05-21T09:59:40.320000", "content": "

Train the model for 9 hours. Download the model. Upload it back and train it from where you left.

", "votes": 2, "replies": [] } ] }, { "id": 534054, "author_name": "邓雍杰Jay", "author_url": "", "post_date": "2019-05-20T14:39:28.670000", "content": "

how to use pretrainedmodels? This is kernels-only competition. Should I rebuild the model and load the weights?

", "votes": 0, "replies": [ { "id": 534060, "author_name": "邓雍杰Jay", "author_url": "", "post_date": "2019-05-20T14:56:26.903000", "content": "

I find my way to use seresnext. I copy the file from pretrainedmodels and build the model.

", "votes": 0, "replies": [] } ] }, { "id": 533878, "author_name": "DmitryKustikov", "author_url": "", "post_date": "2019-05-20T07:12:28.210000", "content": "

Appian,\nwhat about senet154? In my experiments it is worse than resnext.

", "votes": 0, "replies": [ { "id": 534203, "author_name": "Appian", "author_url": "", "post_date": "2019-05-20T22:59:26.757000", "content": "

For me it was just as good as se_resnext101. It could be better because I did not do parameter tuning.

", "votes": 1, "replies": [] } ] }, { "id": 531124, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2019-05-14T10:58:44.810000", "content": "

crop or resize_crop?

", "votes": 0, "replies": [ { "id": 531175, "author_name": "Appian", "author_url": "", "post_date": "2019-05-14T12:36:55.927000", "content": "

ResizedCrop

", "votes": 0, "replies": [] }, { "id": 531251, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-14T14:48:35.093000", "content": "

Thank you for your sharing!
\nYou mean that your augmentation is Resizedcrop?

", "votes": 0, "replies": [] } ] }, { "id": 531105, "author_name": "FGPC", "author_url": "", "post_date": "2019-05-14T10:10:40.380000", "content": "

Is it possible for you guys to share links on how to perform proper CV Folds on image classification? Many thanks! :-)

", "votes": 0, "replies": [ { "id": 531179, "author_name": "Appian", "author_url": "", "post_date": "2019-05-14T12:38:45.410000", "content": "

I recommend Lopuhin's code for making folds. This is very good.\nhttps://github.com/lopuhin/kaggle-imet-2019/blob/master/imet/make_folds.py

", "votes": 2, "replies": [] } ] }, { "id": 531023, "author_name": "Timmmmmms", "author_url": "", "post_date": "2019-05-14T06:26:08.297000", "content": "

May I ask what's your loss function?

", "votes": 0, "replies": [] }, { "id": 531006, "author_name": "Zheng Li", "author_url": "", "post_date": "2019-05-14T05:41:40.650000", "content": "

Hi Appian. We usually get a validation score like 0.59-0.60 (0.597 for me), but you get 0.62. That's so amazing. I tried many methods but none of them seems really work well. Can you share some tips about this if you don't mind? Really appreciate.

", "votes": 0, "replies": [ { "id": 531017, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-14T06:14:34.947000", "content": "

你好，请问你所使用的模型是 se_resnext吗？

", "votes": -1, "replies": [] }, { "id": 531029, "author_name": "Zheng Li", "author_url": "", "post_date": "2019-05-14T06:38:51.773000", "content": "

Resnet101 works for me. CV 0.597 LB 0.597

", "votes": 0, "replies": [] }, { "id": 531110, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-14T10:17:22.600000", "content": "

Thank you for sharing!

", "votes": 0, "replies": [] }, { "id": 531235, "author_name": "GUTAR", "author_url": "", "post_date": "2019-05-14T14:23:34.230000", "content": "

What framework do you use with resnet101?

", "votes": 0, "replies": [] }, { "id": 534371, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-21T06:18:52", "content": "

Hellow. Do you use the Kaggle-kernel to train your models? We tried to use it ,but the kernel is very astable. It will be disconnected every time. Do you meet this problem? And may I ask how you solve it.

", "votes": 0, "replies": [] }, { "id": 535920, "author_name": "邓雍杰Jay", "author_url": "", "post_date": "2019-05-23T16:18:51.600000", "content": "", "votes": -1, "replies": [] }, { "id": 537122, "author_name": "Filemon", "author_url": "", "post_date": "2019-05-26T08:25:36.377000", "content": "

@fengziyuan I also faced this problem using Kaggle kernels. It disconnects often. I am not able to test the code in kernel if I have to wait some epochs. One solution is to commit the kernel.

", "votes": 0, "replies": [] } ] }, { "id": 530953, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-14T03:16:49.717000", "content": "

Thank you for your sharing. I wonder what is the data-set that your each cv score based on? Is the val of each K-fold model? Or is the test data-set ? Or is the whole train set?

", "votes": 0, "replies": [ { "id": 531181, "author_name": "Appian", "author_url": "", "post_date": "2019-05-14T12:40:41.110000", "content": "

I meant \"cv each\" for validation score of each fold.\n\"cv concat\" is validation score on entire train data (oof).

", "votes": 0, "replies": [] }, { "id": 531207, "author_name": "ZY. Feng", "author_url": "", "post_date": "2019-05-14T13:21:41.030000", "content": "

Thank you for sharing!

", "votes": 0, "replies": [] } ] }, { "id": 530946, "author_name": "beyondych11", "author_url": "", "post_date": "2019-05-14T02:46:10.640000", "content": "

May I ask how dou you install these external packages when you submit the kernel?（Network connection is not allowed in the kernel.）

", "votes": 0, "replies": [ { "id": 530962, "author_name": "", "author_url": "", "post_date": "2019-05-14T03:55:45.537000", "content": "

the same question

", "votes": 0, "replies": [] }, { "id": 531003, "author_name": "He", "author_url": "", "post_date": "2019-05-14T05:30:42.017000", "content": "

the same question

", "votes": 0, "replies": [] }, { "id": 531065, "author_name": "seefun", "author_url": "", "post_date": "2019-05-14T08:18:15.417000", "content": "

You can download the source codes of the external packages and upload them into your dataset. Then import them.

", "votes": 3, "replies": [] }, { "id": 531151, "author_name": "beyondych11", "author_url": "", "post_date": "2019-05-14T11:48:38.473000", "content": "

@seefun OK, thanks for your reply!

", "votes": 0, "replies": [] }, { "id": 531476, "author_name": "beyondych11", "author_url": "", "post_date": "2019-05-15T01:43:31.433000", "content": "

@seefun The \"init.py\" in the source codes could not be uploaded to the dataset( error: this file already exists in competition..). Have you encountered such a problem? Can you share some tips about this if you don't mind? Really appreciate.

", "votes": 0, "replies": [] }, { "id": 531737, "author_name": "seefun", "author_url": "", "post_date": "2019-05-15T12:50:47.653000", "content": "

I did not encounter this problem. I only zip the python package source code and upload it to kaggle dataset. And then import them. I guess you didn't zip a python package as a .zip file before upload them.

", "votes": 2, "replies": [] }, { "id": 532443, "author_name": "beyondych11", "author_url": "", "post_date": "2019-05-17T01:23:23.203000", "content": "

@seefun Ok, thank you for your guidance.

", "votes": 0, "replies": [] }, { "id": 532867, "author_name": "Artyom Palvelev", "author_url": "", "post_date": "2019-05-17T23:35:40.033000", "content": "

You can just take SENet.py from pretrainedmodels and include it into your solution, pretty sure this is what Appian did :)

", "votes": 3, "replies": [] }, { "id": 533052, "author_name": "Appian", "author_url": "", "post_date": "2019-05-18T10:55:57.193000", "content": "

Yes, that's what I do.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "530889": "Pretrained models are taken from https://github.com/Cadene/pretrained-models.pytorch\nThe results are very predictable. The deeper the better. \n\n5-folds se_resnext50\\_32x4d (320x320)\ncv each: 0.621 | 0.623 | 0.620 | 0.618 | 0.622\nlb each: 0.621 | ---\ncv(concat): 0.621 \nlb(mean): 0.641 \n\n5-folds se_resnext101\\_32x4d (320x320)\ncv each: 0.627 | 0.629 | 0.627 | 0.628 | 0.630\nlb each: 0.630 | ---\ncv(concat): 0.628\nlb(mean): 0.652\n", "532440": "Thanks for your sharing. I also use se_resnext50_32x4d, but I got a single fold CV of 0.5947. If you don't mind, could you please give some general guidance about what makes your results so good， loss function or train strategy？ I have tried many methods, but nearly none of them worked...", "530997": "May I ask what's your input size? Our results are not as good as you. Did you use metric learning?", "531821": "Hi Appian, your results are marvelous, may I ask you a few questions?\n- The above pretrained models fix the input size at 224x224. How did you modify the models to receive 320x320 images?\n- Did you crop with the size that is less than 320x320 and then resize to 320x320 or just crop to the size 320x320 with some paddings?", "530905": "How many epochs were each fold trained for if I may ask? I didn't see the exact trend with vanilla resnet models, although I feel I may have to train the larger models for longer compared to the shallower ones.", "530911": "@sairam6087\n15 epochs each but the best one came around 12-13. \n\n@jionie\nYes, larger image size helps. I haven't tried different n-folds other than 5. Thanks for the info.", "530895": "Thanks for sharing!! Although my results are poor, I found that also the bigger image size and the bigger the cv folds are better.", "536715": "how much batch-size?", "538092": "I'm using se_resnext101_32x4d (288 x 288) with proper image augmentation and regularization to get CV 0.61. However, with image size 320 x 320, the model didn't improve at all. May I ask how did you resize the image? Did you simply use RandomSizedCrop or Resize + RandomCrop? Thanks.", "537304": "@appian are these vanilla results? without any preprocessing or changing the network at all? Do you train all the layers? Also, when I change the avg_pool layer to AdaptiveAvgPool2d, everything goes out of memory\neven for a batch size of 4. Could you please tell me if I am doing something wrong and how to\nfix it?", "537051": "What is the full name of *cv* and *lb*?", "534093": "Whats your training time for these models?", "534054": "how to use pretrainedmodels? This is kernels-only competition. Should I rebuild the model and load the weights?", "533878": "Appian,\nwhat about senet154? In my experiments it is worse than resnext.", "531124": "crop or resize_crop?\n", "531105": "Is it possible for you guys to share links on how to perform proper CV Folds on image classification? Many thanks! :-)", "531023": "May I ask what's your loss function?", "531006": "Hi Appian. We usually get a validation score like 0.59-0.60 (0.597 for me), but you get 0.62. That's so amazing. I tried many methods but none of them seems really work well. Can you share some tips about this if you don't mind? Really appreciate.", "530953": "Thank you for your sharing. I wonder what is the data-set that your each cv score based on? Is the val of each K-fold model? Or is the test data-set ? Or is the whole train set? ", "530946": "May I ask how dou you install these external packages when you submit the kernel?（Network connection is not allowed in the kernel.）" } }