{ "id": 99328, "title": "Best score when doing \"just classification\"?", "url": "/competitions/recursion-cellular-image-classification/discussion/99328", "author_name": "Konstantin Lopukhin", "post_date": "2019-07-10T12:54:23.054000", "votes": 23, "comment_count": 52, "views": 0, "content": "

I'm curious what is the best score one can achieve by doing just classification, without using any problem specific trick from controls / batches / plates etc., whether using metric learning or not?

\n\n

For me, it was 0.275 from one fold (could be up to 0.35 with some other non-problem-specific tricks).

", "messages": [ { "id": 572093, "postDate": "2019-07-10T12:54:23.053Z", "content": "

I'm curious what is the best score one can achieve by doing just classification, without using any problem specific trick from controls / batches / plates etc., whether using metric learning or not?

\n\n

For me, it was 0.275 from one fold (could be up to 0.35 with some other non-problem-specific tricks).

", "rawMarkdown": "I'm curious what is the best score one can achieve by doing just classification, without using any problem specific trick from controls / batches / plates etc., whether using metric learning or not?\n\nFor me, it was 0.275 from one fold (could be up to 0.35 with some other non-problem-specific tricks).", "votes": 23 }, { "id": 575326, "postDate": "2019-07-15T09:33:25.780Z", "content": "

0.24LB (0.33 cv) vanilla resnet-18 256x256x3 images single fold, no tta

", "rawMarkdown": "0.24LB (0.33 cv) vanilla resnet-18 256x256x3 images single fold, no tta", "votes": 5, "replies": [ { "id": 575342, "postDate": "2019-07-15T10:08:53.423Z", "content": "

May I ask whether you're using PyTorch or Keras (TF)?

", "rawMarkdown": "May I ask whether you're using PyTorch or Keras (TF)?", "votes": 1 }, { "id": 575343, "postDate": "2019-07-15T10:10:08.900Z", "content": "

pytorch

", "rawMarkdown": "pytorch", "votes": 1 }, { "id": 576797, "postDate": "2019-07-16T03:02:31.877Z", "content": "

Hi, may I know is the reason for using vanilla resnet-18 256x256x3 images single fold, no tta to get a baseline? thanks!

", "rawMarkdown": "Hi, may I know is the reason for using `vanilla resnet-18 256x256x3 images single fold, no tta` to get a baseline? thanks!", "votes": 1 }, { "id": 577069, "postDate": "2019-07-16T10:07:53.110Z", "content": "

mainly because its fast

", "rawMarkdown": "mainly because its fast", "votes": 2 }, { "id": 584228, "postDate": "2019-07-25T15:37:15.963Z", "content": "

hi @christofhenkel, i was trying around what you recommended on resnet-18 256x256x3 images single fold and could not achieve the above result. May I ask whether are you using both site 1 and site 2 for training and you are doing a random train_test_split for single fold validation? Also did you account for both sites for prediction? thank you

\n\n

edit: how many epochs are you training as well?

", "rawMarkdown": "hi @christofhenkel, i was trying around what you recommended on `resnet-18 256x256x3 images single fold` and could not achieve the above result. May I ask whether are you using both site 1 and site 2 for training and you are doing a random `train_test_split` for single fold validation? Also did you account for both sites for prediction? thank you\n\nedit: how many epochs are you training as well?", "votes": 1 } ] }, { "id": 572239, "postDate": "2019-07-10T17:05:23.740Z", "content": "

I am doing \"just classification\" so far, Efficientnet b3 with cross-entropy loss. One fold average of two models trained with different random seed gives my score on board right now.

", "rawMarkdown": "I am doing \"just classification\" so far, Efficientnet b3 with cross-entropy loss. One fold average of two models trained with different random seed gives my score on board right now.", "votes": 6, "replies": [ { "id": 572274, "postDate": "2019-07-10T17:55:53.843Z", "content": "

Thanks for sharing! I suspected that my \"just classification\" part was quite poor. I'm using resnet34 with cross-entropy loss, which gives 0.275, and then a trick which turns this into 0.402.\nThat's impressive that you are able to get 0.397 from EfficientNet b3, given it's reputation of being hard to train.

", "rawMarkdown": "Thanks for sharing! I suspected that my \"just classification\" part was quite poor. I'm using resnet34 with cross-entropy loss, which gives 0.275, and then a trick which turns this into 0.402.\nThat's impressive that you are able to get 0.397 from EfficientNet b3, given it's reputation of being hard to train.", "votes": 2 }, { "id": 572355, "postDate": "2019-07-10T20:01:46.103Z", "content": "

I started with b0 which I got it to .277 one fold and moved on to larger ones. I have trouble get b4 to work.

", "rawMarkdown": "I started with b0 which I got it to .277 one fold and moved on to larger ones. I have trouble get b4 to work.", "votes": 2 }, { "id": 572539, "postDate": "2019-07-11T03:56:21.370Z", "content": "

Classification: 0.226 -> 0.312 with a trick.\nYour trick seems to be stronger than mine ;D

", "rawMarkdown": "Classification: 0.226 -> 0.312 with a trick.\nYour trick seems to be stronger than mine ;D", "votes": 1 }, { "id": 572660, "postDate": "2019-07-11T08:22:56.470Z", "content": "

Sorry for the dumb question, but are you using pre-trained models? Thanks

", "rawMarkdown": "Sorry for the dumb question, but are you using pre-trained models? Thanks" }, { "id": 572689, "postDate": "2019-07-11T09:18:26.653Z", "content": "

Unless it's forbidden, you must give a shot to pretrained

", "rawMarkdown": "Unless it's forbidden, you must give a shot to pretrained ", "votes": 2 }, { "id": 572692, "postDate": "2019-07-11T09:19:48.610Z", "content": "
\n

are you using pre-trained models?

\n
\n\n

Yes, similar to https://www.kaggle.com/leighplt/densenet121-pytorch but with resnet34

", "rawMarkdown": "> are you using pre-trained models?\n\nYes, similar to https://www.kaggle.com/leighplt/densenet121-pytorch but with resnet34", "votes": 1 }, { "id": 572696, "postDate": "2019-07-11T09:23:47.903Z", "content": "

I think it's allowed for this competition. Right?

", "rawMarkdown": "I think it's allowed for this competition. Right?" }, { "id": 572762, "postDate": "2019-07-11T11:29:08.233Z", "content": "

you can check https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/97771

", "rawMarkdown": "you can check https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/97771", "votes": 1 }, { "id": 574765, "postDate": "2019-07-14T13:02:52.380Z", "content": "

If I may, for how many epochs are you training your model? I tried to use ResNet34 (PyTorch) with a \"custom\" initial convolutional layer and a new FC layer but I cannot come even close to your score.

", "rawMarkdown": "If I may, for how many epochs are you training your model? I tried to use ResNet34 (PyTorch) with a \"custom\" initial convolutional layer and a new FC layer but I cannot come even close to your score." }, { "id": 575174, "postDate": "2019-07-15T06:15:18.347Z", "content": "

I guess 100-200 epochs, I haven't tried yet, but folks in other similar competitons used these numbers.

", "rawMarkdown": "I guess 100-200 epochs, I haven't tried yet, but folks in other similar competitons used these numbers.\n", "votes": 2 }, { "id": 576888, "postDate": "2019-07-16T06:25:58.923Z", "content": "

Thanks for the information. May I ask you what's your training loss around at the first several epochs? I'm using CE loss and it's very large at first (>7), then it decreases very slow.

", "rawMarkdown": "Thanks for the information. May I ask you what's your training loss around at the first several epochs? I'm using CE loss and it's very large at first (>7), then it decreases very slow. ", "votes": 1 }, { "id": 576911, "postDate": "2019-07-16T06:57:40.857Z", "content": "

I have the same Cross Entropy loss. I think that is expected because the large number of classes (>1000) in this dataset.

", "rawMarkdown": "I have the same Cross Entropy loss. I think that is expected because the large number of classes (>1000) in this dataset." }, { "id": 576947, "postDate": "2019-07-16T07:33:16.620Z", "content": "

I'm using CE too. After the first epoch it's around 7.0. Then it decreases slowly and I can arrive at 0.01 after 100 epochs. 3 channels only and crop to 256.

", "rawMarkdown": "I'm using CE too. After the first epoch it's around 7.0. Then it decreases slowly and I can arrive at 0.01 after 100 epochs. 3 channels only and crop to 256." }, { "id": 576950, "postDate": "2019-07-16T07:39:40.110Z", "content": "

Damn, my model overfit after a few dozen epochs at loss 3~4. Adding regularization only made it converges slower.

", "rawMarkdown": "Damn, my model overfit after a few dozen epochs at loss 3~4. Adding regularization only made it converges slower.", "votes": 1 }, { "id": 577009, "postDate": "2019-07-16T08:38:33.280Z", "content": "

At the moment I'm not even using any form of regularization. I actually wanted it to overfit but I'm not able to! Seems we have the opposite problem :)

", "rawMarkdown": "At the moment I'm not even using any form of regularization. I actually wanted it to overfit but I'm not able to! Seems we have the opposite problem :)" }, { "id": 577028, "postDate": "2019-07-16T08:56:22.733Z", "content": "

and what is your score after 100 epochs?

", "rawMarkdown": "and what is your score after 100 epochs?", "votes": 1 }, { "id": 577035, "postDate": "2019-07-16T09:03:53.907Z", "content": "

If I remember correctly, this morning I got a 0.05 on a validation set. Which I cannot explain as others who are using the same model can get 0.4 on the LB... ResNet18, 3 channels, 256x256, PyTorch, SGD.

", "rawMarkdown": "If I remember correctly, this morning I got a 0.05 on a validation set. Which I cannot explain as others who are using the same model can get 0.4 on the LB... ResNet18, 3 channels, 256x256, PyTorch, SGD." }, { "id": 577097, "postDate": "2019-07-16T11:16:37.803Z", "content": "

@lorenzofabbri92 do you want to team up?

", "rawMarkdown": "@lorenzofabbri92 do you want to team up?", "votes": 1 }, { "id": 577230, "postDate": "2019-07-16T14:12:28.087Z", "content": "

@lorenzofabbri92 How many epochs have you set ? With resnet34 not pretrained 15 epochs 6 channels I got 0.072, it seems weird that you do not exceed 0.05 with a sufficient number of epochs ...

", "rawMarkdown": "@lorenzofabbri92 How many epochs have you set ? With resnet34 not pretrained 15 epochs 6 channels I got 0.072, it seems weird that you do not exceed 0.05 with a sufficient number of epochs ..." }, { "id": 577235, "postDate": "2019-07-16T14:17:41.240Z", "content": "

First off, thanks for the interest!\nToday I tried for 100 epochs. Yes, indeed. I really do not know what's wrong with my model. Obviously there are a lot of things that you can tune by I used pretty \"standard\" values for basically everything (learning rate, etc...).

", "rawMarkdown": "First off, thanks for the interest!\nToday I tried for 100 epochs. Yes, indeed. I really do not know what's wrong with my model. Obviously there are a lot of things that you can tune by I used pretty \"standard\" values for basically everything (learning rate, etc...)." }, { "id": 577790, "postDate": "2019-07-17T05:03:46.037Z", "content": "

@lorenzofabbri92 I have the same issue as yours. I got 0.06 on a validation set after convergence... What's your init learning rate?

", "rawMarkdown": "@lorenzofabbri92 I have the same issue as yours. I got 0.06 on a validation set after convergence... What's your init learning rate?" }, { "id": 577796, "postDate": "2019-07-17T05:08:02.963Z", "content": "

@ttylacm I tried both SGD and Adam with either 0.1 or 0.01 (or 0.001 for Adam only). I also use a ReduceLROnPlateau. Considering that others are using the same model (and the same framework), I really do not understand what's wrong.

", "rawMarkdown": "@ttylacm I tried both SGD and Adam with either 0.1 or 0.01 (or 0.001 for Adam only). I also use a ReduceLROnPlateau. Considering that others are using the same model (and the same framework), I really do not understand what's wrong." }, { "id": 577799, "postDate": "2019-07-17T05:12:36.437Z", "content": "

@lorenzofabbri92 indeed something strange with your model, resnet 34, 6 channel input, Adam with lr=0.001, and only 10 epochs gave me 0.069 on the leaderboard. Are you using all 70k+ images to train?\nMy transfrom:\ndata_transforms = T.Compose(\n T.RandomCrop(224),\n T.RandomHorizontalFlip(),\n T.RandomVerticalFlip(),\n T.ToTensor()\n )

", "rawMarkdown": "@lorenzofabbri92 indeed something strange with your model, resnet 34, 6 channel input, Adam with lr=0.001, and only 10 epochs gave me 0.069 on the leaderboard. Are you using all 70k+ images to train?\nMy transfrom:\ndata_transforms = T.Compose([\n T.RandomCrop(224),\n T.RandomHorizontalFlip(),\n T.RandomVerticalFlip(),\n T.ToTensor()\n ])", "votes": 2 }, { "id": 577806, "postDate": "2019-07-17T05:27:21.263Z", "content": "

@joven1997 Pretrained or from scratch? You are not normalizing the images?

", "rawMarkdown": "@joven1997 Pretrained or from scratch? You are not normalizing the images?", "votes": 1 }, { "id": 577853, "postDate": "2019-07-17T06:20:55.570Z", "content": "

@lorenzofabbri92 Do you simply decay lr in ReduceLROnPlateau callback? Sometimes it doesn't work well, and probably need a larger lr to escape local minima.

", "rawMarkdown": "@lorenzofabbri92 Do you simply decay lr in ReduceLROnPlateau callback? Sometimes it doesn't work well, and probably need a larger lr to escape local minima. " }, { "id": 577871, "postDate": "2019-07-17T06:36:14.360Z", "content": "

@lorenzofabbri92 i don't normalize, pretrained on imagenet.

", "rawMarkdown": "@lorenzofabbri92 i don't normalize, pretrained on imagenet.", "votes": 1 }, { "id": 578339, "postDate": "2019-07-17T16:14:41.643Z", "content": "

@lorenzofabbri92 Change to 512x512 and use SGD fix my issue. After 30 epochs, validation acc is above 0.24

", "rawMarkdown": "@lorenzofabbri92 Change to 512x512 and use SGD fix my issue. After 30 epochs, validation acc is above 0.24" }, { "id": 578350, "postDate": "2019-07-17T16:24:53.803Z", "content": "

@ttylacm I’m not reducing the dimension and I’m using SGD too. Are you using a pretrained ResNet? If yes, are you normalizing the 6 channels in a particular way? I tried training ResNet34 from scratch and the validation accuracy is 0.1 after almost 20 epochs. But then on the LB it’s 0.05. At the moment I’m using just one site for the test set, though.

", "rawMarkdown": "@ttylacm I’m not reducing the dimension and I’m using SGD too. Are you using a pretrained ResNet? If yes, are you normalizing the 6 channels in a particular way? I tried training ResNet34 from scratch and the validation accuracy is 0.1 after almost 20 epochs. But then on the LB it’s 0.05. At the moment I’m using just one site for the test set, though." }, { "id": 578484, "postDate": "2019-07-17T18:43:05.463Z", "content": "

I haven't used 6 channels yet. I'm using 512x512 rgb with resnet50 imagenet pretrained model.

", "rawMarkdown": "I haven't used 6 channels yet. I'm using 512x512 rgb with resnet50 imagenet pretrained model. " }, { "id": 578500, "postDate": "2019-07-17T19:33:38.077Z", "content": "

When you use pretrained model, are you finetuning the convolutional layers or just training classifier on head of it? That is interesting in order to understand if my approach (1st stage is feature extraction, than fast training and development of different heads https://www.kaggle.com/alexanderkhar/transfer-learning-keras-starter-by-alex-khar) makes sense?

", "rawMarkdown": "When you use pretrained model, are you finetuning the convolutional layers or just training classifier on head of it? That is interesting in order to understand if my approach (1st stage is feature extraction, than fast training and development of different heads https://www.kaggle.com/alexanderkhar/transfer-learning-keras-starter-by-alex-khar) makes sense? " }, { "id": 578974, "postDate": "2019-07-18T11:12:38.857Z", "content": "

@lorenzofabbri92 Check, that you call model.train() and model.eval(), maybe the low CV score is because you evaluate with BatchNormalization/DropOut in train-mode instead of eval-mode. I had good results starting training with a lr between 3e-4 and 6e-5 for pretrained models.

", "rawMarkdown": "@lorenzofabbri92 Check, that you call model.train() and model.eval(), maybe the low CV score is because you evaluate with BatchNormalization/DropOut in train-mode instead of eval-mode. I had good results starting training with a lr between 3e-4 and 6e-5 for pretrained models.", "votes": 1 } ] }, { "id": 583230, "postDate": "2019-07-24T07:40:26.337Z", "content": "

0.49 Val -> 0.36 LB single fold, pre-trained DenseNet201 with CE loss, trained with Adam for 20+ epochs. It seems to me that metric learning & some data-specific tricks should boost score further.

", "rawMarkdown": "0.49 Val -> 0.36 LB single fold, pre-trained DenseNet201 with CE loss, trained with Adam for 20+ epochs. It seems to me that metric learning & some data-specific tricks should boost score further.", "votes": 1, "replies": [ { "id": 583247, "postDate": "2019-07-24T08:07:49.610Z", "content": "

Are you using just one site? PyTorch? Thanks.

", "rawMarkdown": "Are you using just one site? PyTorch? Thanks." }, { "id": 583276, "postDate": "2019-07-24T09:12:29.903Z", "content": "

1) Of course PyTorch:)\n2) Both sites

", "rawMarkdown": "1) Of course PyTorch:)\n2) Both sites\n", "votes": 1 }, { "id": 583280, "postDate": "2019-07-24T09:19:38.793Z", "content": "

Thanks. Interesting. Apparently I'm the only one who cannot get a decent score with pre-trained models... :)\nHave a nice day!

", "rawMarkdown": "Thanks. Interesting. Apparently I'm the only one who cannot get a decent score with pre-trained models... :)\nHave a nice day!" } ] }, { "id": 572604, "postDate": "2019-07-11T07:00:43.817Z", "content": "

What size are you using? We are still at 128x128

", "rawMarkdown": "What size are you using? We are still at 128x128", "votes": 1, "replies": [ { "id": 572610, "postDate": "2019-07-11T07:12:47.267Z", "content": "

I use 512 x 512. With 224x224 I saw much slower convergence

", "rawMarkdown": "I use 512 x 512. With 224x224 I saw much slower convergence" }, { "id": 572649, "postDate": "2019-07-11T08:05:41.763Z", "content": "

I'm not rescaling the image so far.

", "rawMarkdown": "I'm not rescaling the image so far.", "votes": 1 }, { "id": 572891, "postDate": "2019-07-11T14:33:32.753Z", "content": "

512x512 gives for me better result, down-scaling (0.5x) leads to 3-5% metric degradation depending on model choice.

", "rawMarkdown": "512x512 gives for me better result, down-scaling (0.5x) leads to 3-5% metric degradation depending on model choice.", "votes": 2 } ] }, { "id": 584183, "postDate": "2019-07-25T14:28:58.250Z", "content": "

What is the \"trick\" everyone is talking about?! I seem to be the only one not knowing about it.

", "rawMarkdown": "What is the \"trick\" everyone is talking about?! I seem to be the only one not knowing about it. ", "votes": 2 }, { "id": 588309, "postDate": "2019-07-30T12:13:06.347Z", "content": "

For those of you with a >0.5 LB score, can you tell us what accuracy score you get on the validation set after the first few epochs?

", "rawMarkdown": "For those of you with a >0.5 LB score, can you tell us what accuracy score you get on the validation set after the first few epochs?" }, { "id": 583181, "postDate": "2019-07-24T06:19:13.350Z", "content": "

I am curious how many epchs did you use to train your model @lopuhin to get 0.275LB.

", "rawMarkdown": "I am curious how many epchs did you use to train your model @lopuhin to get 0.275LB.", "replies": [ { "id": 583250, "postDate": "2019-07-24T08:20:28.410Z", "content": "

Around 30 if I recall correctly

", "rawMarkdown": "Around 30 if I recall correctly", "votes": 1 } ] }, { "id": 573610, "postDate": "2019-07-12T13:46:06.543Z", "content": "

I got 0.282 with a single fold. I think there's still a lot to improve but if you want to get a good score like the best score, those specific tricks from the biology background must be adopted.

", "rawMarkdown": "I got 0.282 with a single fold. I think there's still a lot to improve but if you want to get a good score like the best score, those specific tricks from the biology background must be adopted." }, { "id": 574255, "postDate": "2019-07-13T14:19:52.597Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 575326, "author_name": "Dieter", "author_url": "", "post_date": "2019-07-15T09:33:25.780000", "content": "

0.24LB (0.33 cv) vanilla resnet-18 256x256x3 images single fold, no tta

", "votes": 5, "replies": [ { "id": 575342, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-15T10:08:53.423000", "content": "

May I ask whether you're using PyTorch or Keras (TF)?

", "votes": 1, "replies": [] }, { "id": 575343, "author_name": "Dieter", "author_url": "", "post_date": "2019-07-15T10:10:08.900000", "content": "

pytorch

", "votes": 1, "replies": [] }, { "id": 576797, "author_name": "wjsheng", "author_url": "", "post_date": "2019-07-16T03:02:31.877000", "content": "

Hi, may I know is the reason for using vanilla resnet-18 256x256x3 images single fold, no tta to get a baseline? thanks!

", "votes": 1, "replies": [] }, { "id": 577069, "author_name": "Dieter", "author_url": "", "post_date": "2019-07-16T10:07:53.110000", "content": "

mainly because its fast

", "votes": 2, "replies": [] }, { "id": 584228, "author_name": "wjsheng", "author_url": "", "post_date": "2019-07-25T15:37:15.963000", "content": "

hi @christofhenkel, i was trying around what you recommended on resnet-18 256x256x3 images single fold and could not achieve the above result. May I ask whether are you using both site 1 and site 2 for training and you are doing a random train_test_split for single fold validation? Also did you account for both sites for prediction? thank you

\n\n

edit: how many epochs are you training as well?

", "votes": 1, "replies": [] } ] }, { "id": 572239, "author_name": "Ren", "author_url": "", "post_date": "2019-07-10T17:05:23.740000", "content": "

I am doing \"just classification\" so far, Efficientnet b3 with cross-entropy loss. One fold average of two models trained with different random seed gives my score on board right now.

", "votes": 6, "replies": [ { "id": 572274, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-07-10T17:55:53.843000", "content": "

Thanks for sharing! I suspected that my \"just classification\" part was quite poor. I'm using resnet34 with cross-entropy loss, which gives 0.275, and then a trick which turns this into 0.402.\nThat's impressive that you are able to get 0.397 from EfficientNet b3, given it's reputation of being hard to train.

", "votes": 2, "replies": [] }, { "id": 572355, "author_name": "Ren", "author_url": "", "post_date": "2019-07-10T20:01:46.103000", "content": "

I started with b0 which I got it to .277 one fold and moved on to larger ones. I have trouble get b4 to work.

", "votes": 2, "replies": [] }, { "id": 572539, "author_name": "yu4u", "author_url": "", "post_date": "2019-07-11T03:56:21.370000", "content": "

Classification: 0.226 -> 0.312 with a trick.\nYour trick seems to be stronger than mine ;D

", "votes": 1, "replies": [] }, { "id": 572660, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-11T08:22:56.470000", "content": "

Sorry for the dumb question, but are you using pre-trained models? Thanks

", "votes": 0, "replies": [] }, { "id": 572689, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2019-07-11T09:18:26.653000", "content": "

Unless it's forbidden, you must give a shot to pretrained

", "votes": 2, "replies": [] }, { "id": 572692, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-07-11T09:19:48.610000", "content": "
\n

are you using pre-trained models?

\n
\n\n

Yes, similar to https://www.kaggle.com/leighplt/densenet121-pytorch but with resnet34

", "votes": 1, "replies": [] }, { "id": 572696, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-11T09:23:47.903000", "content": "

I think it's allowed for this competition. Right?

", "votes": 0, "replies": [] }, { "id": 572762, "author_name": "Vlad Shmyhlo ", "author_url": "", "post_date": "2019-07-11T11:29:08.233000", "content": "

you can check https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/97771

", "votes": 1, "replies": [] }, { "id": 574765, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-14T13:02:52.380000", "content": "

If I may, for how many epochs are you training your model? I tried to use ResNet34 (PyTorch) with a \"custom\" initial convolutional layer and a new FC layer but I cannot come even close to your score.

", "votes": 0, "replies": [] }, { "id": 575174, "author_name": "wayfarer", "author_url": "", "post_date": "2019-07-15T06:15:18.347000", "content": "

I guess 100-200 epochs, I haven't tried yet, but folks in other similar competitons used these numbers.

", "votes": 2, "replies": [] }, { "id": 576888, "author_name": "ttylcc", "author_url": "", "post_date": "2019-07-16T06:25:58.923000", "content": "

Thanks for the information. May I ask you what's your training loss around at the first several epochs? I'm using CE loss and it's very large at first (>7), then it decreases very slow.

", "votes": 1, "replies": [] }, { "id": 576911, "author_name": "Phúc Lê", "author_url": "", "post_date": "2019-07-16T06:57:40.857000", "content": "

I have the same Cross Entropy loss. I think that is expected because the large number of classes (>1000) in this dataset.

", "votes": 0, "replies": [] }, { "id": 576947, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-16T07:33:16.620000", "content": "

I'm using CE too. After the first epoch it's around 7.0. Then it decreases slowly and I can arrive at 0.01 after 100 epochs. 3 channels only and crop to 256.

", "votes": 0, "replies": [] }, { "id": 576950, "author_name": "Phúc Lê", "author_url": "", "post_date": "2019-07-16T07:39:40.110000", "content": "

Damn, my model overfit after a few dozen epochs at loss 3~4. Adding regularization only made it converges slower.

", "votes": 1, "replies": [] }, { "id": 577009, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-16T08:38:33.280000", "content": "

At the moment I'm not even using any form of regularization. I actually wanted it to overfit but I'm not able to! Seems we have the opposite problem :)

", "votes": 0, "replies": [] }, { "id": 577028, "author_name": "wayfarer", "author_url": "", "post_date": "2019-07-16T08:56:22.733000", "content": "

and what is your score after 100 epochs?

", "votes": 1, "replies": [] }, { "id": 577035, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-16T09:03:53.907000", "content": "

If I remember correctly, this morning I got a 0.05 on a validation set. Which I cannot explain as others who are using the same model can get 0.4 on the LB... ResNet18, 3 channels, 256x256, PyTorch, SGD.

", "votes": 0, "replies": [] }, { "id": 577097, "author_name": "wayfarer", "author_url": "", "post_date": "2019-07-16T11:16:37.803000", "content": "

@lorenzofabbri92 do you want to team up?

", "votes": 1, "replies": [] }, { "id": 577230, "author_name": "Nicolas H", "author_url": "", "post_date": "2019-07-16T14:12:28.087000", "content": "

@lorenzofabbri92 How many epochs have you set ? With resnet34 not pretrained 15 epochs 6 channels I got 0.072, it seems weird that you do not exceed 0.05 with a sufficient number of epochs ...

", "votes": 0, "replies": [] }, { "id": 577235, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-16T14:17:41.240000", "content": "

First off, thanks for the interest!\nToday I tried for 100 epochs. Yes, indeed. I really do not know what's wrong with my model. Obviously there are a lot of things that you can tune by I used pretty \"standard\" values for basically everything (learning rate, etc...).

", "votes": 0, "replies": [] }, { "id": 577790, "author_name": "ttylcc", "author_url": "", "post_date": "2019-07-17T05:03:46.037000", "content": "

@lorenzofabbri92 I have the same issue as yours. I got 0.06 on a validation set after convergence... What's your init learning rate?

", "votes": 0, "replies": [] }, { "id": 577796, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-17T05:08:02.963000", "content": "

@ttylacm I tried both SGD and Adam with either 0.1 or 0.01 (or 0.001 for Adam only). I also use a ReduceLROnPlateau. Considering that others are using the same model (and the same framework), I really do not understand what's wrong.

", "votes": 0, "replies": [] }, { "id": 577799, "author_name": "wayfarer", "author_url": "", "post_date": "2019-07-17T05:12:36.437000", "content": "

@lorenzofabbri92 indeed something strange with your model, resnet 34, 6 channel input, Adam with lr=0.001, and only 10 epochs gave me 0.069 on the leaderboard. Are you using all 70k+ images to train?\nMy transfrom:\ndata_transforms = T.Compose(\n T.RandomCrop(224),\n T.RandomHorizontalFlip(),\n T.RandomVerticalFlip(),\n T.ToTensor()\n )

", "votes": 2, "replies": [] }, { "id": 577806, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-17T05:27:21.263000", "content": "

@joven1997 Pretrained or from scratch? You are not normalizing the images?

", "votes": 1, "replies": [] }, { "id": 577853, "author_name": "ttylcc", "author_url": "", "post_date": "2019-07-17T06:20:55.570000", "content": "

@lorenzofabbri92 Do you simply decay lr in ReduceLROnPlateau callback? Sometimes it doesn't work well, and probably need a larger lr to escape local minima.

", "votes": 0, "replies": [] }, { "id": 577871, "author_name": "wayfarer", "author_url": "", "post_date": "2019-07-17T06:36:14.360000", "content": "

@lorenzofabbri92 i don't normalize, pretrained on imagenet.

", "votes": 1, "replies": [] }, { "id": 578339, "author_name": "ttylcc", "author_url": "", "post_date": "2019-07-17T16:14:41.643000", "content": "

@lorenzofabbri92 Change to 512x512 and use SGD fix my issue. After 30 epochs, validation acc is above 0.24

", "votes": 0, "replies": [] }, { "id": 578350, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-17T16:24:53.803000", "content": "

@ttylacm I’m not reducing the dimension and I’m using SGD too. Are you using a pretrained ResNet? If yes, are you normalizing the 6 channels in a particular way? I tried training ResNet34 from scratch and the validation accuracy is 0.1 after almost 20 epochs. But then on the LB it’s 0.05. At the moment I’m using just one site for the test set, though.

", "votes": 0, "replies": [] }, { "id": 578484, "author_name": "ttylcc", "author_url": "", "post_date": "2019-07-17T18:43:05.463000", "content": "

I haven't used 6 channels yet. I'm using 512x512 rgb with resnet50 imagenet pretrained model.

", "votes": 0, "replies": [] }, { "id": 578500, "author_name": "Alexander Khar", "author_url": "", "post_date": "2019-07-17T19:33:38.077000", "content": "

When you use pretrained model, are you finetuning the convolutional layers or just training classifier on head of it? That is interesting in order to understand if my approach (1st stage is feature extraction, than fast training and development of different heads https://www.kaggle.com/alexanderkhar/transfer-learning-keras-starter-by-alex-khar) makes sense?

", "votes": 0, "replies": [] }, { "id": 578974, "author_name": "Marek Wyborski", "author_url": "", "post_date": "2019-07-18T11:12:38.857000", "content": "

@lorenzofabbri92 Check, that you call model.train() and model.eval(), maybe the low CV score is because you evaluate with BatchNormalization/DropOut in train-mode instead of eval-mode. I had good results starting training with a lr between 3e-4 and 6e-5 for pretrained models.

", "votes": 1, "replies": [] } ] }, { "id": 583230, "author_name": "grib0ed0v", "author_url": "", "post_date": "2019-07-24T07:40:26.337000", "content": "

0.49 Val -> 0.36 LB single fold, pre-trained DenseNet201 with CE loss, trained with Adam for 20+ epochs. It seems to me that metric learning & some data-specific tricks should boost score further.

", "votes": 1, "replies": [ { "id": 583247, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-24T08:07:49.610000", "content": "

Are you using just one site? PyTorch? Thanks.

", "votes": 0, "replies": [] }, { "id": 583276, "author_name": "grib0ed0v", "author_url": "", "post_date": "2019-07-24T09:12:29.903000", "content": "

1) Of course PyTorch:)\n2) Both sites

", "votes": 1, "replies": [] }, { "id": 583280, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-24T09:19:38.793000", "content": "

Thanks. Interesting. Apparently I'm the only one who cannot get a decent score with pre-trained models... :)\nHave a nice day!

", "votes": 0, "replies": [] } ] }, { "id": 572604, "author_name": "Dieter", "author_url": "", "post_date": "2019-07-11T07:00:43.817000", "content": "

What size are you using? We are still at 128x128

", "votes": 1, "replies": [ { "id": 572610, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2019-07-11T07:12:47.267000", "content": "

I use 512 x 512. With 224x224 I saw much slower convergence

", "votes": 0, "replies": [] }, { "id": 572649, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-07-11T08:05:41.763000", "content": "

I'm not rescaling the image so far.

", "votes": 1, "replies": [] }, { "id": 572891, "author_name": "grib0ed0v", "author_url": "", "post_date": "2019-07-11T14:33:32.753000", "content": "

512x512 gives for me better result, down-scaling (0.5x) leads to 3-5% metric degradation depending on model choice.

", "votes": 2, "replies": [] } ] }, { "id": 584183, "author_name": "QuantScientist", "author_url": "", "post_date": "2019-07-25T14:28:58.250000", "content": "

What is the \"trick\" everyone is talking about?! I seem to be the only one not knowing about it.

", "votes": 2, "replies": [] }, { "id": 588309, "author_name": "Lorenzo Fabbri", "author_url": "", "post_date": "2019-07-30T12:13:06.347000", "content": "

For those of you with a >0.5 LB score, can you tell us what accuracy score you get on the validation set after the first few epochs?

", "votes": 0, "replies": [] }, { "id": 583181, "author_name": "Kurian Benoy", "author_url": "", "post_date": "2019-07-24T06:19:13.350000", "content": "

I am curious how many epchs did you use to train your model @lopuhin to get 0.275LB.

", "votes": 0, "replies": [ { "id": 583250, "author_name": "Konstantin Lopukhin", "author_url": "", "post_date": "2019-07-24T08:20:28.410000", "content": "

Around 30 if I recall correctly

", "votes": 1, "replies": [] } ] }, { "id": 573610, "author_name": "syoya", "author_url": "", "post_date": "2019-07-12T13:46:06.543000", "content": "

I got 0.282 with a single fold. I think there's still a lot to improve but if you want to get a good score like the best score, those specific tricks from the biology background must be adopted.

", "votes": 0, "replies": [] }, { "id": 574255, "author_name": "", "author_url": "", "post_date": "2019-07-13T14:19:52.597000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "572093": "I'm curious what is the best score one can achieve by doing just classification, without using any problem specific trick from controls / batches / plates etc., whether using metric learning or not?\n\nFor me, it was 0.275 from one fold (could be up to 0.35 with some other non-problem-specific tricks).", "575326": "0.24LB (0.33 cv) vanilla resnet-18 256x256x3 images single fold, no tta", "572239": "I am doing \"just classification\" so far, Efficientnet b3 with cross-entropy loss. One fold average of two models trained with different random seed gives my score on board right now.", "583230": "0.49 Val -> 0.36 LB single fold, pre-trained DenseNet201 with CE loss, trained with Adam for 20+ epochs. It seems to me that metric learning & some data-specific tricks should boost score further.", "572604": "What size are you using? We are still at 128x128", "584183": "What is the \"trick\" everyone is talking about?! I seem to be the only one not knowing about it. ", "588309": "For those of you with a >0.5 LB score, can you tell us what accuracy score you get on the validation set after the first few epochs?", "583181": "I am curious how many epchs did you use to train your model @lopuhin to get 0.275LB.", "573610": "I got 0.282 with a single fold. I think there's still a lot to improve but if you want to get a good score like the best score, those specific tricks from the biology background must be adopted.", "574255": "" } }