{ "id": 248442, "title": "Discussion: CV Vs LB", "url": "/competitions/siim-covid19-detection/discussion/248442", "author_name": "Harshit Sheoran", "post_date": "2021-06-23T14:45:18.342000", "votes": 67, "comment_count": 116, "views": 0, "content": "

Wanna Share your cv score with lb.

\n

Let me start with mine (Study Only)

\n

+Efnv2-L
\n+384 Image Size
\n+Aux Loss
\n+GroupKFold
\n+Pytorch

\n

CV (5 folds):
\n[0.38026270847655175,
\n 0.38925640451414656,
\n 0.3869573594325011,
\n 0.37830235976215315,
\n 0.3903840103131989]

\n

LB w/o tta: 0.456 (~0.401 for actual study only)

\n

I am submitting my score of .617 with study level only the image level part was taken from the public image level notebook of https://www.kaggle.com/micheomaano/siim-cov19-efnb7-yolov5-infer

", "messages": [ { "id": 1362632, "postDate": "2021-06-23T14:45:18.343Z", "content": "

Wanna Share your cv score with lb.

\n

Let me start with mine (Study Only)

\n

+Efnv2-L
\n+384 Image Size
\n+Aux Loss
\n+GroupKFold
\n+Pytorch

\n

CV (5 folds):
\n[0.38026270847655175,
\n 0.38925640451414656,
\n 0.3869573594325011,
\n 0.37830235976215315,
\n 0.3903840103131989]

\n

LB w/o tta: 0.456 (~0.401 for actual study only)

\n

I am submitting my score of .617 with study level only the image level part was taken from the public image level notebook of https://www.kaggle.com/micheomaano/siim-cov19-efnb7-yolov5-infer

", "rawMarkdown": "Wanna Share your cv score with lb.\n\nLet me start with mine (Study Only)\n\n+Efnv2-L\n+384 Image Size\n+Aux Loss\n+GroupKFold\n+Pytorch\n\nCV (5 folds):\n[0.38026270847655175,\n 0.38925640451414656,\n 0.3869573594325011,\n 0.37830235976215315,\n 0.3903840103131989]\n\nLB w/o tta: 0.456 (~0.401 for actual study only)\n\nI am submitting my score of .617 with study level only the image level part was taken from the public image level notebook of https://www.kaggle.com/micheomaano/siim-cov19-efnb7-yolov5-infer", "votes": 67 }, { "id": 1364424, "postDate": "2021-06-25T01:46:39.790Z", "content": "

study-level only (none 1 0 0 1 1 for image-level prediction)

\n

arch : effnet-b7
\nres : 640
\nloss : CE loss (w/o aux loss)
\nvalidation : GroupKFold 5 folds
\nTTA : flip + brightness

\n

CV

\n\n

LB

\n", "rawMarkdown": "study-level only (`none 1 0 0 1 1` for image-level prediction)\n\narch : effnet-b7\nres : 640\nloss : CE loss (w/o aux loss)\nvalidation : GroupKFold 5 folds\nTTA : flip + brightness\n\nCV \n* mAP * 2/3 : about 0.37 ~ 0.38\n* AUC : about 0.87 ~ 0.89\n* top-1 acc : about 0.67 ~ 0.68\n\nLB\n* 0.451\n* maybe about 0.4 w/ null prediction ('') for image-level", "votes": 7, "replies": [ { "id": 1364497, "postDate": "2021-06-25T04:21:29.397Z", "content": "

That's a very impressive score without aux loss. Do you have any other different methods in your model?

", "rawMarkdown": "That's a very impressive score without aux loss. Do you have any other different methods in your model?" }, { "id": 1364564, "postDate": "2021-06-25T04:55:51.600Z", "content": "

yeap. I added more augmentations (e.g. cutout, …) & slightly changed the head network to regularize the model better!

", "rawMarkdown": "yeap. I added more augmentations (e.g. cutout, ...) & slightly changed the head network to regularize the model better!" } ] }, { "id": 1379931, "postDate": "2021-07-07T17:13:30.567Z", "content": "

Efnv2-M
\n512 Image Size
\nBCE + (Lovasz + BCE) AUX
\nGroupKFold
\nPytorch
\nMixed Precision
\nScale : 1.25

\n

CV Score (Group K Fold)

\n

\"scores\": [
\n 0.39388390121767813,
\n 0.39234564782357945,
\n 0.3847761244542205,
\n 0.37818329809109086,
\n 0.38300730435642993
\n ]

\n

LB score: 45.6 (study level only, use 'none 1 0 0 1 1' for image level prediction string)

\n

With help of some invaluable pointers from this thread and @harshitsheoran, thanks for the help

", "rawMarkdown": "Efnv2-M\n512 Image Size\nBCE + (Lovasz + BCE) AUX\nGroupKFold\nPytorch\nMixed Precision\nScale : 1.25\n\nCV Score (Group K Fold)\n\n\"scores\": [\n 0.39388390121767813,\n 0.39234564782357945,\n 0.3847761244542205,\n 0.37818329809109086,\n 0.38300730435642993\n ]\n\nLB score: 45.6 (study level only, use 'none 1 0 0 1 1' for image level prediction string)\n\nWith help of some invaluable pointers from this thread and @harshitsheoran, thanks for the help", "votes": 6, "replies": [ { "id": 1379998, "postDate": "2021-07-07T18:02:45.800Z", "content": "

What exactly is scale 1.25?

", "rawMarkdown": "What exactly is scale 1.25?" }, { "id": 1380005, "postDate": "2021-07-07T18:11:50.387Z", "content": "

increasing the image size from 512 to 640 at inference

", "rawMarkdown": "increasing the image size from 512 to 640 at inference", "votes": 1 }, { "id": 1380527, "postDate": "2021-07-08T06:36:36.853Z", "content": "

Can you reveal the probability of applying scale？

", "rawMarkdown": "Can you reveal the probability of applying scale？" }, { "id": 1383386, "postDate": "2021-07-10T19:23:22.283Z", "content": "

@syxuming its just a fixed scale there is no TTA step for these scores

", "rawMarkdown": "@syxuming its just a fixed scale there is no TTA step for these scores" }, { "id": 1386950, "postDate": "2021-07-13T20:03:20.467Z", "content": "

Hi @varundutt9213

\n

Are you removing duplicates? How are you doing this Group K Fold split? I can't find a way to achieve such high scores in my folds.

", "rawMarkdown": "Hi @varundutt9213 \n\nAre you removing duplicates? How are you doing this Group K Fold split? I can't find a way to achieve such high scores in my folds. " }, { "id": 1388030, "postDate": "2021-07-14T15:52:33.140Z", "content": "

No these scores are without any preprocessing and the splitting has no tricks just simple group kfold

", "rawMarkdown": "No these scores are without any preprocessing and the splitting has no tricks just simple group kfold", "votes": 1 }, { "id": 1391427, "postDate": "2021-07-17T15:27:26.493Z", "content": "

@varundutt9213 Hi guy, I tried to train eff-v2_m with image_size = 512, finetune its with blocks[4] but i get an error \"ValueError: Input contains NaN, infinity or a value too large for dtype('float16').\"
\ni still fix this err but it does not improve. i guest that anything diffirent with eff-b7
\nplease help me!

", "rawMarkdown": "@varundutt9213 Hi guy, I tried to train eff-v2_m with image_size = 512, finetune its with blocks[4] but i get an error \"ValueError: Input contains NaN, infinity or a value too large for dtype('float16').\"\ni still fix this err but it does not improve. i guest that anything diffirent with eff-b7\nplease help me!" }, { "id": 1393328, "postDate": "2021-07-19T14:49:35.980Z", "content": "

What's the purpose of increasing scale at inference time?

", "rawMarkdown": "What's the purpose of increasing scale at inference time?" }, { "id": 1396527, "postDate": "2021-07-22T07:15:04.230Z", "content": "

may be its due to
\nyou are not using pretraining weight
\nbecause efficientnetv2_m is w/0 pre weight in TIMM
\nonly tf_efficientnetv2_** or efficientnetv2_rw_** has weight

", "rawMarkdown": "may be its due to \nyou are not using pretraining weight\nbecause efficientnetv2_m is w/0 pre weight in TIMM\nonly tf_efficientnetv2_** or efficientnetv2_rw_** has weight ", "votes": 3 }, { "id": 1398105, "postDate": "2021-07-23T18:17:00.530Z", "content": "

Oh, i don't no that…
\nVery thanks!!!

", "rawMarkdown": "Oh, i don't no that... \nVery thanks!!!" }, { "id": 1398646, "postDate": "2021-07-24T11:05:48.383Z", "content": "

Hi, i'm rather new to kaggle. When people talk about \"LB score\", do they mean LB score inferenced with ensemble of all KFOLD trained models? so ensemble of K models? Is your's the case too?

", "rawMarkdown": "Hi, i'm rather new to kaggle. When people talk about \"LB score\", do they mean LB score inferenced with ensemble of all KFOLD trained models? so ensemble of K models? Is your's the case too?" } ] }, { "id": 1364125, "postDate": "2021-06-24T16:29:47.093Z", "content": "

Study Only

\n

tf_eff_b5
\n640 Image Size
\nNo Aux Loss
\nGroupKFold
\nPytorch

\n

CV: 0.368 LB: 442

", "rawMarkdown": "Study Only\n\ntf_eff_b5\n640 Image Size\nNo Aux Loss\nGroupKFold\nPytorch\n\nCV: 0.368 LB: 442", "votes": 3, "replies": [ { "id": 1377222, "postDate": "2021-07-05T17:04:41.673Z", "content": "

Update:
\nStudy Only

\n

tf_eff_b5
\n640 Image Size
\nAux Loss BCE Mask [128,128] block 5
\nSnapmix 0.5
\nPreprocessing scale 255.0
\nGroupKFold
\nPytorch

\n

CV: 0.370 LB: 447 LB(TTA): 453

", "rawMarkdown": "Update:\nStudy Only\n\ntf_eff_b5\n640 Image Size\nAux Loss BCE Mask [128,128] block 5\nSnapmix 0.5\nPreprocessing scale 255.0\nGroupKFold\nPytorch\n\nCV: 0.370 LB: 447 LB(TTA): 453\n", "votes": 2 }, { "id": 1377403, "postDate": "2021-07-05T19:40:27.323Z", "content": "

Which TTA was used?

", "rawMarkdown": "Which TTA was used?" }, { "id": 1377406, "postDate": "2021-07-05T19:45:08.163Z", "content": "

@harshitsheoran Hflip + RandomResizedCrop TTA3x

", "rawMarkdown": "@harshitsheoran Hflip + RandomResizedCrop TTA3x" } ] }, { "id": 1396192, "postDate": "2021-07-21T21:48:43.643Z", "content": "

@harshitsheoran Hello, how do you calculate CV score? Are they losses of the models?
\nThanks.

", "rawMarkdown": "@harshitsheoran Hello, how do you calculate CV score? Are they losses of the models?\nThanks.", "votes": 1, "replies": [ { "id": 1397088, "postDate": "2021-07-22T18:54:37.620Z", "content": "

there are public notebooks that could help you
\nnb1 , nb2

", "rawMarkdown": "there are public notebooks that could help you \n[nb1](https://www.kaggle.com/varundutt9213/covid-competition-metric-image-level-map) , [nb2](https://www.kaggle.com/keremt/competition-metric-map-0-5)\n", "votes": 2 }, { "id": 1397142, "postDate": "2021-07-22T20:37:12.193Z", "content": "

Thank you.

", "rawMarkdown": "Thank you.", "votes": 1 } ] }, { "id": 1379489, "postDate": "2021-07-07T12:11:00.057Z", "content": "

Did you use a particular technic to finetune effnetv2? Or just the classical one ? Because I've seen several learning technics were applied to pre-train the network, but I haven't dived deep into this.

", "rawMarkdown": "Did you use a particular technic to finetune effnetv2? Or just the classical one ? Because I've seen several learning technics were applied to pre-train the network, but I haven't dived deep into this.", "votes": 1, "replies": [ { "id": 1379858, "postDate": "2021-07-07T16:38:18.680Z", "content": "

Just the classic one.

", "rawMarkdown": "Just the classic one.", "votes": 1 }, { "id": 1380016, "postDate": "2021-07-07T18:21:55.620Z", "content": "

Could you please clarify what is the classic one?

", "rawMarkdown": "Could you please clarify what is the classic one?" }, { "id": 1380039, "postDate": "2021-07-07T18:39:47.543Z", "content": "

It means just loading the imagenet weights and retraining the backbone on this train data, nothing special or different at all.

", "rawMarkdown": "It means just loading the imagenet weights and retraining the backbone on this train data, nothing special or different at all.", "votes": 2 } ] }, { "id": 1376199, "postDate": "2021-07-04T23:08:37.113Z", "content": "

@harshitsheoran This might be a very vague question but I have very similar CV scores that you have with effv2_l but my lb score is much lower any idea why this might be the case.

\n

Efnv2-L
\n384 Image Size
\nBCE
\nGroupKFold
\nPytorch
\nMixed Precision

\n

CV Score (Group K Fold)

\n

\"scores\": [
\n 0.38221975192699126,
\n 0.3876267083746035,
\n 0.3804051350518518,
\n 0.37546675040210786,
\n 0.38837743907783395
\n ]

\n

LB score: 44.3 (study level only, use 'none 1 0 0 1 1' for image level prediction string)

", "rawMarkdown": "@harshitsheoran This might be a very vague question but I have very similar CV scores that you have with effv2_l but my lb score is much lower any idea why this might be the case.\n\nEfnv2-L\n384 Image Size\nBCE\nGroupKFold\nPytorch\nMixed Precision\n\nCV Score (Group K Fold)\n\n\"scores\": [\n 0.38221975192699126,\n 0.3876267083746035,\n 0.3804051350518518,\n 0.37546675040210786,\n 0.38837743907783395\n ]\n\nLB score: 44.3 (study level only, use 'none 1 0 0 1 1' for image level prediction string)", "votes": 1, "replies": [ { "id": 1376211, "postDate": "2021-07-05T00:28:34.797Z", "content": "

Well, there are many reasons where I saw my score go berserk of tiny changes, learning rate scheduler had huge effects, normalization surely did help the CV just a little but made LB a lot worse and cuz the data in CV is also pretty small so I did not consider it worth doing. I did use a unique aux loss to make it all stable so I dont expect my approach to be stable without it anyways also random seed have effects on CV and LB too, also I am using tf_efficientnetv2_l_in21ft1k and not just the normal weights…

\n

Hope it helps, I know that is a lot of info but I had to try and fail a lot of things after getting to 456 and it is extremely hard to improve even a single point beyond this by just changing model or augs or hyperparameters

\n

Also, changing batch size if especially if its below 32 can cause problems….

", "rawMarkdown": "Well, there are many reasons where I saw my score go berserk of tiny changes, learning rate scheduler had huge effects, normalization surely did help the CV just a little but made LB a lot worse and cuz the data in CV is also pretty small so I did not consider it worth doing. I did use a unique aux loss to make it all stable so I dont expect my approach to be stable without it anyways also random seed have effects on CV and LB too, also I am using tf_efficientnetv2_l_in21ft1k and not just the normal weights...\n\nHope it helps, I know that is a lot of info but I had to try and fail a lot of things after getting to 456 and it is extremely hard to improve even a single point beyond this by just changing model or augs or hyperparameters\n\nAlso, changing batch size if especially if its below 32 can cause problems....", "votes": 2 }, { "id": 1377038, "postDate": "2021-07-05T14:32:42.463Z", "content": "

@harshitsheoran thank you for the detailed answer really appreciate it. By skipping normalization do you mean not using image net mean and std and just dividing image by 255.0?

", "rawMarkdown": "@harshitsheoran thank you for the detailed answer really appreciate it. By skipping normalization do you mean not using image net mean and std and just dividing image by 255.0? " }, { "id": 1377180, "postDate": "2021-07-05T16:25:54.520Z", "content": "

Yes……….

", "rawMarkdown": "Yes.........." }, { "id": 1377221, "postDate": "2021-07-05T17:03:10.313Z", "content": "

That's true. I don't know why but applying normalization lead to a worse LB. Also, batch size is important, but I'm having a hard time fitting a batch like 32 in my GPU, I'm using now batch size 6 for effb5 640x640.

\n

@harshitsheoran can you sure what did you find about learning rate schedulers?

", "rawMarkdown": "That's true. I don't know why but applying normalization lead to a worse LB. Also, batch size is important, but I'm having a hard time fitting a batch like 32 in my GPU, I'm using now batch size 6 for effb5 640x640.\n\n@harshitsheoran can you sure what did you find about learning rate schedulers?" }, { "id": 1377402, "postDate": "2021-07-05T19:40:06.600Z", "content": "

Well, this might not be true for everyone but what I noted was changing the learning rate scheduler after fixing on augmentations would only lead to worse results, which might mean that learning rate scheduler and augmentations are depends on each other and augs might have to be adjusted if the learning rate scheduler is changed.

", "rawMarkdown": "Well, this might not be true for everyone but what I noted was changing the learning rate scheduler after fixing on augmentations would only lead to worse results, which might mean that learning rate scheduler and augmentations are depends on each other and augs might have to be adjusted if the learning rate scheduler is changed." }, { "id": 1388463, "postDate": "2021-07-15T01:37:26.297Z", "content": "

Hi @harshitsheoran, thank you for all your discussions, these help a lot. I though have a question, is there any article you can share that explains the relation between augmentation and scheduler, or any notebook to play with the different augs and scheduler parameters?

", "rawMarkdown": "Hi @harshitsheoran, thank you for all your discussions, these help a lot. I though have a question, is there any article you can share that explains the relation between augmentation and scheduler, or any notebook to play with the different augs and scheduler parameters? " }, { "id": 1388710, "postDate": "2021-07-15T06:47:12.453Z", "content": "

Unfortunately as stated above, its just what I noted, its not something which would be true for every case, to my knowledge it would be applied in the cases where there is a lot of noise or the data is small. This was only my speculation so to my knowledge there exists no article or notebook which would be enough to prove/explain my claims

", "rawMarkdown": "Unfortunately as stated above, its just what I noted, its not something which would be true for every case, to my knowledge it would be applied in the cases where there is a lot of noise or the data is small. This was only my speculation so to my knowledge there exists no article or notebook which would be enough to prove/explain my claims" } ] }, { "id": 1370679, "postDate": "2021-06-30T10:17:12.483Z", "content": "

Could you explain what your aux loss is made up of?

", "rawMarkdown": "Could you explain what your aux loss is made up of?", "votes": 1, "replies": [ { "id": 1370749, "postDate": "2021-06-30T11:38:24.923Z", "content": "

It is just BCE. the predictions are masks and the targets are mask made from annotations.

", "rawMarkdown": "It is just BCE. the predictions are masks and the targets are mask made from annotations." }, { "id": 1370806, "postDate": "2021-06-30T12:19:38.187Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1378235, "postDate": "2021-07-06T11:49:46.150Z", "content": "

but isn't this study level prediction? How can you use BCE while there are 4 output labels?

", "rawMarkdown": "but isn't this study level prediction? How can you use BCE while there are 4 output labels?" }, { "id": 1378543, "postDate": "2021-07-06T15:49:41.597Z", "content": "

BCE does not limit to binary class predictions you know

", "rawMarkdown": "BCE does not limit to binary class predictions you know" }, { "id": 1389933, "postDate": "2021-07-16T08:29:46.417Z", "content": "

Can you help to clarify: how do you encode labels to calculate bce?
\nSo your activation is sigmoid or softmax? Thanks a lot in advance.

", "rawMarkdown": "Can you help to clarify: how do you encode labels to calculate bce? \nSo your activation is sigmoid or softmax? Thanks a lot in advance." }, { "id": 1389995, "postDate": "2021-07-16T09:10:56.640Z", "content": "

SIgmoid…

", "rawMarkdown": "SIgmoid..." } ] }, { "id": 1362910, "postDate": "2021-06-23T18:11:13.460Z", "content": "

Hey, thanks for your post. Do you think your increase in performances compare to the public baseline is due to the aux loss or to effnet V2 l ?

", "rawMarkdown": "Hey, thanks for your post. Do you think your increase in performances compare to the public baseline is due to the aux loss or to effnet V2 l ? ", "votes": 1, "replies": [ { "id": 1362947, "postDate": "2021-06-23T18:50:09.597Z", "content": "

It was both of them, I improved score a lot by make preprocessing specially for v2, aux loss was implemented uniquely as well it did give a boost in score too. v2 had a bigger boost than aux loss. However none of the boosts were anything you could miss.

", "rawMarkdown": "It was both of them, I improved score a lot by make preprocessing specially for v2, aux loss was implemented uniquely as well it did give a boost in score too. v2 had a bigger boost than aux loss. However none of the boosts were anything you could miss.", "votes": 2 }, { "id": 1364303, "postDate": "2021-06-24T20:27:37.053Z", "content": "

What extra things exactly needs to be done for running the V2?

", "rawMarkdown": "What extra things exactly needs to be done for running the V2?" }, { "id": 1365550, "postDate": "2021-06-25T21:21:13.187Z", "content": "

Nothing in specific, just tune the things like image size, scalings, augmentations and your functions to best suit V2

", "rawMarkdown": "Nothing in specific, just tune the things like image size, scalings, augmentations and your functions to best suit V2" } ] }, { "id": 1386571, "postDate": "2021-07-13T14:58:04.490Z", "content": "

Can someone explain to me how masks are used, and what is the relation with aux loss? I see people talking about it in lots of threads, but can't quite figure out what it means.

", "rawMarkdown": "Can someone explain to me how masks are used, and what is the relation with aux loss? I see people talking about it in lots of threads, but can't quite figure out what it means.", "votes": 2, "replies": [ { "id": 1386674, "postDate": "2021-07-13T16:03:05.343Z", "content": "

https://www.kaggle.com/c/siim-covid19-detection/discussion/245323

", "rawMarkdown": "https://www.kaggle.com/c/siim-covid19-detection/discussion/245323", "votes": 4 } ] }, { "id": 1365390, "postDate": "2021-06-25T17:04:56.993Z", "content": "

It seems that if you only submit \"opacity score x0 y0 x1 y1\" without \"none … \" for the image, the lb score is less than one. Let the lb score be S, the S/0.1666666666 gives about 0.50

\n

i did not submit one by one (it would waste too many slots), but i estimate a typical lb results would look like this:

\n
0         Negative for Pneumonia : 0.85\n1             Typical Appearance : 0.85\n2       Indeterminate Appearance : 0.30\n3            Atypical Appearance : 0.30\n4            Opacity : 0.55\n5            None : 0.85\n
\n

i am targeting at 0.42+0.25 for the final lb score

", "rawMarkdown": "It seems that if you only submit \"opacity score x0 y0 x1 y1\" without \"none ... \" for the image, the lb score is less than one. Let the lb score be S, the S/0.1666666666 gives about 0.50\n\ni did not submit one by one (it would waste too many slots), but i estimate a typical lb results would look like this:\n```\n0 Negative for Pneumonia : 0.85\n1 Typical Appearance : 0.85\n2 Indeterminate Appearance : 0.30\n3 Atypical Appearance : 0.30\n4 Opacity : 0.55\n5 None : 0.85\n```\n\ni am targeting at 0.42+0.25 for the final lb score", "votes": 2, "replies": [ { "id": 1365464, "postDate": "2021-06-25T18:50:35.723Z", "content": "

Thanks for the info!, I wish I could do image detection, I currently have pretty much no idea in that but achieving 0.42 study level looks quite possible, I mean with pretraining, you upgrading your model from M to L and doing pseduo labeling not to mention ensembling could give a boost enough!?

", "rawMarkdown": "Thanks for the info!, I wish I could do image detection, I currently have pretty much no idea in that but achieving 0.42 study level looks quite possible, I mean with pretraining, you upgrading your model from M to L and doing pseduo labeling not to mention ensembling could give a boost enough!?", "votes": 1 } ] }, { "id": 1364407, "postDate": "2021-06-25T00:29:55.240Z", "content": "

Each criterion may be different by implement….

", "rawMarkdown": "Each criterion may be different by implement....", "votes": 2 }, { "id": 1364250, "postDate": "2021-06-24T18:39:20.400Z", "content": "

EDIT : LB 0.440 (study level only, use none 1 0 0 1 1 for image level prediction string)

\n

local validation:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
FOLDSmAP*0.66
fold00.3659
fold10.3649
fold20.3459
fold30.3685
fold30.3575
\n

image size = 640 x 640
\nbase = efficientnetb4
\nloss = focal_loss + bce_loss (aux_loss)
\nTTA = null
\ndata = GroupStratifiedKFold

", "rawMarkdown": "EDIT : LB 0.440 (study level only, use `none 1 0 0 1 1` for image level prediction string)\n\nlocal validation:\n\n| FOLDS | mAP*0.66 |\n| --------- | ----------- |\n| fold0 | 0.3659 |\n| fold1 | 0.3649 |\n| fold2 | 0.3459 |\n| fold3 | 0.3685 |\n| fold3 | 0.3575 |\n\n \nimage size = 640 x 640\nbase = efficientnetb4\nloss = focal_loss + bce_loss (aux_loss)\nTTA = null\ndata = [GroupStratifiedKFold](https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html#:~:text=Stratified%20K%2DFolds%20iterator%20variant,of%20samples%20for%20each%20class.)\n\n", "votes": 2, "replies": [ { "id": 1364263, "postDate": "2021-06-24T19:04:12.280Z", "content": "

Please, 0.440 is extremely hard to get as of now if you use null \" for image level prediction string your actual study level score would be ~0.385

", "rawMarkdown": "Please, 0.440 is extremely hard to get as of now if you use null \" for image level prediction string your actual study level score would be ~0.385 ", "votes": 1 }, { "id": 1951021, "postDate": "2022-09-22T17:56:38.963Z", "content": "

@benihime91 I apologize, I was wrong!

", "rawMarkdown": "@benihime91 I apologize, I was wrong!" } ] }, { "id": 1363884, "postDate": "2021-06-24T12:49:07.167Z", "content": "

LB 0.402 (study level only, use null '' for image level prediction string)
\nLB 0.457 (study level only, use 'none 1 0 0 1 1' for image level prediction string)

\n

local validation:

\n
eff2m-512-lovasz        \n            mAP*0.66    CE loss     topk accuracy       \nfold0        0.393118    0.808467    [0.680834   0.85324779 0.95669607 1.        ]               \nfold1        0.388934    0.818398    [0.68243785 0.85404972 0.96230954 1.        ]               \nfold2        0.380347    0.842349    [0.67095736 0.84392599 0.95655672 1.        ]               \nfold3        0.383767    0.891183    [0.66051364 0.83788122 0.94863563 1.        ]               \nfold4        0.388402    0.823305    [0.68167203 0.85691318 0.95016077 1.        ]               \n mean        0.386914                        \n
\n

image size=512x512
\nefficientv2-M
\naux loss : BCE + lovasz
\nTTA: flip + scale1.33

", "rawMarkdown": "LB 0.402 (study level only, use null '' for image level prediction string)\nLB 0.457 (study level only, use 'none 1 0 0 1 1' for image level prediction string)\n\nlocal validation:\n```\neff2m-512-lovasz\t\t\n\t\t\tmAP*0.66\tCE loss\t\ttopk accuracy\t\t\nfold0\t\t0.393118 \t0.808467\t[0.680834 0.85324779 0.95669607 1. ]\t\t\t\t\nfold1\t\t0.388934 \t0.818398\t[0.68243785 0.85404972 0.96230954 1. ]\t\t\t\t\nfold2\t\t0.380347 \t0.842349\t[0.67095736 0.84392599 0.95655672 1. ]\t\t\t\t\nfold3\t\t0.383767 \t0.891183\t[0.66051364 0.83788122 0.94863563 1. ]\t\t\t\t\nfold4\t\t0.388402 \t0.823305\t[0.68167203 0.85691318 0.95016077 1. ]\t\t\t\t\n mean\t\t0.386914 \t\t\t\t\t\t\n\n```\n\nimage size=512x512\nefficientv2-M\naux loss : BCE + lovasz\nTTA: flip + scale1.33", "votes": 2, "replies": [ { "id": 1364021, "postDate": "2021-06-24T14:49:28.980Z", "content": "

Thanks for sharing!, I wish I had experience in scaling images in correspondence to image size as this comeptition preprocessing highly depends on that. It took me a few days to get the scales right for the image size of 384. Yeah increasing image size with scales even through trail and error can give me a better score? Any thoughts on this?

", "rawMarkdown": "Thanks for sharing!, I wish I had experience in scaling images in correspondence to image size as this comeptition preprocessing highly depends on that. It took me a few days to get the scales right for the image size of 384. Yeah increasing image size with scales even through trail and error can give me a better score? Any thoughts on this?", "votes": 2 }, { "id": 1364126, "postDate": "2021-06-24T16:31:00.027Z", "content": "

Hi @harshitsheoran
\nwhat do you mean by scaling in correspondence to image size?

", "rawMarkdown": "Hi @harshitsheoran \nwhat do you mean by scaling in correspondence to image size? " }, { "id": 1364132, "postDate": "2021-06-24T16:34:38.080Z", "content": "

I mean that there are unique scalings to image size like for 384 there is a different \"best\" scaling setting and it is a different setting for 512

", "rawMarkdown": "I mean that there are unique scalings to image size like for 384 there is a different \"best\" scaling setting and it is a different setting for 512", "votes": 1 }, { "id": 1369085, "postDate": "2021-06-29T06:29:47.600Z", "content": "

@hengck23 are these scores for stratified k fold or group k fold?

", "rawMarkdown": "@hengck23 are these scores for stratified k fold or group k fold?" } ] }, { "id": 1363183, "postDate": "2021-06-24T02:01:54.067Z", "content": "

If you want to get the score for the study level only, the image level should be empty instead of none.

", "rawMarkdown": "If you want to get the score for the study level only, the image level should be empty instead of none.", "votes": 2, "replies": [ { "id": 1363299, "postDate": "2021-06-24T05:23:34.943Z", "content": "

Or I can just subtract ~0.055 from study level to get pretty much the same… I am not asking for study only scores here, I am asking to share CV of whatever you have got with the LB Score.

", "rawMarkdown": "Or I can just subtract ~0.055 from study level to get pretty much the same... I am not asking for study only scores here, I am asking to share CV of whatever you have got with the LB Score." }, { "id": 1363364, "postDate": "2021-06-24T06:01:16.867Z", "content": "

anyone has verified that submission of null '' string has resulted in LB score of zero?

", "rawMarkdown": "anyone has verified that submission of null '' string has resulted in LB score of zero?" }, { "id": 1364425, "postDate": "2021-06-25T01:46:42.280Z", "content": "

I made sure of it.

", "rawMarkdown": "I made sure of it." } ] }, { "id": 1393836, "postDate": "2021-07-20T00:44:29.130Z", "content": "

Despite I can get LB 0.453 w/o tta, my cv doesn't go higher than ~0.377. Any tips on what am I doing wrong?
\nAlso, every time I tried effv2 models it got worse results.
\ntf_eff_b5_ns
\n512 Image Size
\nCE + (0.3BCE + 0.7Lovasz)
\nGroupKfold

\n

CV(5 Folds)
\n0.3797073066234588
\n0.3764741718769073
\n0.376079648733139
\n0.3833852708339691
\n0.3772689402103424

\n

mean = 0.3785675637491991
\nLb: 0.453

", "rawMarkdown": "Despite I can get LB 0.453 w/o tta, my cv doesn't go higher than ~0.377. Any tips on what am I doing wrong?\nAlso, every time I tried effv2 models it got worse results.\ntf_eff_b5_ns\n512 Image Size\nCE + (0.3*BCE + 0.7*Lovasz)\nGroupKfold\n\nCV(5 Folds)\n0.3797073066234588\n0.3764741718769073\n0.376079648733139\n0.3833852708339691\n0.3772689402103424\n\nmean = 0.3785675637491991\nLb: 0.453", "replies": [ { "id": 1393928, "postDate": "2021-07-20T03:11:11.807Z", "content": "

me too, i tried so many times but my mean CV from 5 folds about 0.36~0.37

", "rawMarkdown": "me too, i tried so many times but my mean CV from 5 folds about 0.36~0.37 " }, { "id": 1397039, "postDate": "2021-07-22T17:47:10.847Z", "content": "

How do you guys calculate CV score?

", "rawMarkdown": "How do you guys calculate CV score?" }, { "id": 1402283, "postDate": "2021-07-28T04:27:26.170Z", "content": "

@igormunizims What batch size did you use?

", "rawMarkdown": "@igormunizims What batch size did you use?\n" }, { "id": 1402721, "postDate": "2021-07-28T13:37:44.863Z", "content": "

@magiccard batch size 8

", "rawMarkdown": "@magiccard batch size 8", "votes": 1 }, { "id": 1402859, "postDate": "2021-07-28T15:30:45.817Z", "content": "

Thank your reply @igormunizims
\nDid you try increase your batch size? I think it has a pretty big effect.
\nif your GPU doesn't fit you can try tensorflow with TPU
\nAnd the coefficient of lovasz and bce…, try change them

", "rawMarkdown": "Thank your reply @igormunizims \nDid you try increase your batch size? I think it has a pretty big effect.\nif your GPU doesn't fit you can try tensorflow with TPU\nAnd the coefficient of lovasz and bce..., try change them\n" }, { "id": 1405119, "postDate": "2021-07-30T14:03:13.437Z", "content": "

@igormunizims Which number epochs did you use? Increase number of epochs can get better score

", "rawMarkdown": "@igormunizims Which number epochs did you use? Increase number of epochs can get better score" } ] }, { "id": 1387516, "postDate": "2021-07-14T08:42:54.863Z", "content": "

Do you use this dataset ? https://www.kaggle.com/raddar/ricord-covid19-xray-positive-tests

", "rawMarkdown": "Do you use this dataset ? [https://www.kaggle.com/raddar/ricord-covid19-xray-positive-tests](url)", "replies": [ { "id": 1387637, "postDate": "2021-07-14T10:42:14.313Z", "content": "

I have not used it yet…

", "rawMarkdown": "I have not used it yet..." }, { "id": 1387863, "postDate": "2021-07-14T13:28:54.117Z", "content": "

Since this image is from the source data training set, I don't know what to do with them, does anyone have a good idea?

", "rawMarkdown": "Since this image is from the source data training set, I don't know what to do with them, does anyone have a good idea?" } ] }, { "id": 1386903, "postDate": "2021-07-13T19:14:24.547Z", "content": "

Thanks for sharing. Was your effnet custom keras effnet or torch effnet?

", "rawMarkdown": "Thanks for sharing. Was your effnet custom keras effnet or torch effnet?", "replies": [ { "id": 1386996, "postDate": "2021-07-13T21:00:09.827Z", "content": "

torch effnet

", "rawMarkdown": "torch effnet" } ] }, { "id": 1383219, "postDate": "2021-07-10T16:23:56.947Z", "content": "

Any of the aux loss user with scores > 0.456 would care to tell where they attached the aux head ?

", "rawMarkdown": "Any of the aux loss user with scores > 0.456 would care to tell where they attached the aux head ?", "replies": [ { "id": 1383239, "postDate": "2021-07-10T16:38:52.360Z", "content": "

Do you mean in what block to attach the CNN head? If so, I've tried only in block 5, but my best score is 0.453. I built a dynamic code where I can change the block to attach the head and the CNN architecture using the command line, so I'll try different configurations now and let you know what I found.

", "rawMarkdown": "Do you mean in what block to attach the CNN head? If so, I've tried only in block 5, but my best score is 0.453. I built a dynamic code where I can change the block to attach the head and the CNN architecture using the command line, so I'll try different configurations now and let you know what I found.", "votes": 2 }, { "id": 1383254, "postDate": "2021-07-10T16:47:27.967Z", "content": "

Thanks for the answer. Do you use effnet v2 ? I'm using tensorflow, so I got a full graph without blocks …

", "rawMarkdown": "Thanks for the answer. Do you use effnet v2 ? I'm using tensorflow, so I got a full graph without blocks ..." }, { "id": 1383261, "postDate": "2021-07-10T16:49:51.557Z", "content": "

I tried effnetv2 but it's not my best model. Not sure if I'm doing something wrong

", "rawMarkdown": "I tried effnetv2 but it's not my best model. Not sure if I'm doing something wrong", "votes": 1 }, { "id": 1383265, "postDate": "2021-07-10T16:54:08.850Z", "content": "

I've been trying effv2 for two weeks, I can't make something better than with effnetv1 B7 …

", "rawMarkdown": "I've been trying effv2 for two weeks, I can't make something better than with effnetv1 B7 ..." }, { "id": 1383307, "postDate": "2021-07-10T17:43:51.057Z", "content": "

For me ~2 or 3 blocks before the pooling layer worked the best got LB score of 45.6 without any post processing

", "rawMarkdown": "For me ~2 or 3 blocks before the pooling layer worked the best got LB score of 45.6 without any post processing", "votes": 1 }, { "id": 1383310, "postDate": "2021-07-10T17:50:17.733Z", "content": "

@varundutt9213 did you use only one block(e.g block 4 or block 5) or applied mask in more than one?

", "rawMarkdown": "@varundutt9213 did you use only one block(e.g block 4 or block 5) or applied mask in more than one?", "votes": 1 }, { "id": 1383315, "postDate": "2021-07-10T17:58:41.523Z", "content": "

i have not tried training model by taking masks from multiple blocks.

", "rawMarkdown": "i have not tried training model by taking masks from multiple blocks.", "votes": 1 }, { "id": 1383330, "postDate": "2021-07-10T18:17:39.293Z", "content": "

@varundutt9213 how much batch size are you able to fit in ? i am able to fit bs of 8 in effv2 large with image size = 512 , batch size is a key factor due to batch normalization and its increasing the model performance for sure i am curious to know how much you are using .

", "rawMarkdown": "@varundutt9213 how much batch size are you able to fit in ? i am able to fit bs of 8 in effv2 large with image size = 512 , batch size is a key factor due to batch normalization and its increasing the model performance for sure i am curious to know how much you are using .", "votes": 1 }, { "id": 1383339, "postDate": "2021-07-10T18:28:43.197Z", "content": "

Bs of 32*8 so 256, with TPU.

", "rawMarkdown": "Bs of 32*8 so 256, with TPU.", "votes": 1 }, { "id": 1383368, "postDate": "2021-07-10T19:03:19.227Z", "content": "

wow and i am able to fit a bs of 8*8 in torch xla ( TPU ) with resnet101d which is smaller than effv2_l and in gpu for effv2_l its 8 only .

", "rawMarkdown": "wow and i am able to fit a bs of 8*8 in torch xla ( TPU ) with resnet101d which is smaller than effv2_l and in gpu for effv2_l its 8 only . ", "votes": 1 }, { "id": 1383376, "postDate": "2021-07-10T19:11:26.697Z", "content": "

@trooperog On a single gpu i think the largest model with corresponding largest image size that you can use is effnetv2-m with 512x512 image size I think you can fit 21 sample batches, anything larger will just overfit and give a poor performance even with AUX loss.

", "rawMarkdown": "@trooperog On a single gpu i think the largest model with corresponding largest image size that you can use is effnetv2-m with 512x512 image size I think you can fit 21 sample batches, anything larger will just overfit and give a poor performance even with AUX loss.", "votes": 1, "replies": [ { "id": 1383395, "postDate": "2021-07-10T19:30:08.983Z", "content": "

you mean it will give OOM error right ? yes i tested it earlier and it is 20

", "rawMarkdown": "you mean it will give OOM error right ? yes i tested it earlier and it is 20 " } ] }, { "id": 1383383, "postDate": "2021-07-10T19:18:47.347Z", "content": "

I am using 32 batch size with effv2L… lower batch size does give really poor performance and accumulations did not work because the random seeding is disrupted.

", "rawMarkdown": "I am using 32 batch size with effv2L... lower batch size does give really poor performance and accumulations did not work because the random seeding is disrupted.", "votes": 1, "replies": [ { "id": 1383392, "postDate": "2021-07-10T19:28:14.063Z", "content": "

@harshitsheoran yes lower bs is really giving poor performance as i tested it with my v2_medium model and the results were very different , but you are using local machine ? as you cannot fit a effv2 large with a batch size of 32 or even 20 with a image size of 384 or 512 in kaggle or colab gpu .

", "rawMarkdown": "@harshitsheoran yes lower bs is really giving poor performance as i tested it with my v2_medium model and the results were very different , but you are using local machine ? as you cannot fit a effv2 large with a batch size of 32 or even 20 with a image size of 384 or 512 in kaggle or colab gpu . " } ] }, { "id": 1383413, "postDate": "2021-07-10T20:03:36.197Z", "content": "

I'm using tensorflow, maybe that's what makes it possible to fit 32*8 batch size idk

", "rawMarkdown": "I'm using tensorflow, maybe that's what makes it possible to fit 32*8 batch size idk" }, { "id": 1383428, "postDate": "2021-07-10T20:23:27.960Z", "content": "

I am using local machine… Using TPU you should be able to fit them, maybe you could use accumulations if you could live with the seed and redesign everything accordingly but it is a very time consuming thing to do.

", "rawMarkdown": "I am using local machine... Using TPU you should be able to fit them, maybe you could use accumulations if you could live with the seed and redesign everything accordingly but it is a very time consuming thing to do." }, { "id": 1396260, "postDate": "2021-07-22T01:14:29.107Z", "content": "

@josephamigo , I have tried TPU for this competition but I'm not able to use the Sequence generator with model.fit in TPU.

\n

Second, if we use tf.data then is augmentation apply on each epoch in TPU or only applies ones in the beginning?

", "rawMarkdown": "@josephamigo , I have tried TPU for this competition but I'm not able to use the Sequence generator with model.fit in TPU. \n\nSecond, if we use tf.data then is augmentation apply on each epoch in TPU or only applies ones in the beginning?" }, { "id": 1396375, "postDate": "2021-07-22T04:13:40.810Z", "content": "

I am not familiar with tensorflow data module, but I believe it applies on each epoch.

", "rawMarkdown": "I am not familiar with tensorflow data module, but I believe it applies on each epoch." } ] }, { "id": 1380228, "postDate": "2021-07-07T23:50:19.897Z", "content": "

What I meant to say was do you just do segm_loss = (Lovasz + BCE) or something like segm_loss = (p*Lovasz + q*BCE) ? What your average segmentation loss bdw ?

", "rawMarkdown": "What I meant to say was do you just do `segm_loss = (Lovasz + BCE)` or something like `segm_loss = (p*Lovasz + q*BCE)` ? What your average segmentation loss bdw ?", "replies": [ { "id": 1380233, "postDate": "2021-07-08T00:02:04.670Z", "content": "

yeah i am using p=0.75 and q=0.25

", "rawMarkdown": "yeah i am using p=0.75 and q=0.25" } ] }, { "id": 1380135, "postDate": "2021-07-07T20:45:55.710Z", "content": "

@varundutt9213 Did you use weights for Lovasz and BCE ??

", "rawMarkdown": "@varundutt9213 Did you use weights for `Lovasz` and `BCE` ??", "replies": [ { "id": 1380159, "postDate": "2021-07-07T21:30:44.237Z", "content": "

What do you mean by weights? I used both losses at training time as segmentation loss

", "rawMarkdown": "What do you mean by weights? I used both losses at training time as segmentation loss" } ] }, { "id": 1377660, "postDate": "2021-07-06T03:35:26.503Z", "content": "

Thanks for sharing @harshitsheoran

\n

If you don't mind me asking, how are you dealing with the images that fall under a non-negative study but have no bounding boxes?

", "rawMarkdown": "Thanks for sharing @harshitsheoran\n\nIf you don't mind me asking, how are you dealing with the images that fall under a non-negative study but have no bounding boxes?", "replies": [ { "id": 1377750, "postDate": "2021-07-06T05:00:42.583Z", "content": "

Currently I am not doing any data cleaning, I have got the data from my teammate and I will be working on it in future, currently I believe the first thing to do would be removing it. I dont think there are many of those images to cause a noise to even bother with, however at the top lb even 0.001 makes a difference in gold and silver.

", "rawMarkdown": "Currently I am not doing any data cleaning, I have got the data from my teammate and I will be working on it in future, currently I believe the first thing to do would be removing it. I dont think there are many of those images to cause a noise to even bother with, however at the top lb even 0.001 makes a difference in gold and silver.", "votes": 1 } ] }, { "id": 1376099, "postDate": "2021-07-04T19:05:56.783Z", "content": "

Do you use the masks as a 5 channel input or a dual output?

", "rawMarkdown": "Do you use the masks as a 5 channel input or a dual output?", "replies": [ { "id": 1376109, "postDate": "2021-07-04T19:22:25.913Z", "content": "

Dual Output

", "rawMarkdown": "Dual Output" }, { "id": 1376113, "postDate": "2021-07-04T19:26:08.777Z", "content": "

Nice. I am currently experimenting with 5 channel input

", "rawMarkdown": "Nice. I am currently experimenting with 5 channel input" } ] }, { "id": 1371420, "postDate": "2021-07-01T01:43:43.037Z", "content": "

What type of data augmentations do you use?

", "rawMarkdown": "What type of data augmentations do you use?", "replies": [ { "id": 1371484, "postDate": "2021-07-01T03:39:17.947Z", "content": "

Very minimalist acutally, RandomResizedCrop, HFlip, Cutout, RandomBrightnessContrast and Rotate, I did try a lot of augmentatios too and making it augmentation heavy but the lb or cv did not increase

", "rawMarkdown": "Very minimalist acutally, RandomResizedCrop, HFlip, Cutout, RandomBrightnessContrast and Rotate, I did try a lot of augmentatios too and making it augmentation heavy but the lb or cv did not increase", "votes": 2 } ] }, { "id": 1371314, "postDate": "2021-06-30T21:33:42.167Z", "content": "

How do you generate the mask for aux loss? Is it just the bbox crop or is there any other way?

", "rawMarkdown": "How do you generate the mask for aux loss? Is it just the bbox crop or is there any other way?", "replies": [ { "id": 1371328, "postDate": "2021-06-30T22:09:58.517Z", "content": "

It is just bbox crop.

", "rawMarkdown": "It is just bbox crop.", "votes": 1 } ] }, { "id": 1365650, "postDate": "2021-06-26T02:03:09.763Z", "content": "

Are you applying both flips or just the H Flip

", "rawMarkdown": "Are you applying both flips or just the H Flip", "replies": [ { "id": 1365849, "postDate": "2021-06-26T07:26:43.427Z", "content": "

In augmentations just applying H Flip, and if you asking for tta then I am not applying any.

", "rawMarkdown": "In augmentations just applying H Flip, and if you asking for tta then I am not applying any.", "votes": 1 } ] }, { "id": 1365444, "postDate": "2021-06-25T18:09:52.857Z", "content": "

Our First submission with 5-fold-CV YOLOv5 models.
\nCV score was 0.571 and LB 0.563.
\nIn CV score, the study part was 0.35 and the image part 0.22.

", "rawMarkdown": "Our First submission with 5-fold-CV YOLOv5 models.\nCV score was **0.571** and LB **0.563**.\nIn CV score, the study part was 0.35 and the image part 0.22.", "replies": [ { "id": 1365759, "postDate": "2021-06-26T05:25:22.670Z", "content": "

Hello,how do you evaluation them?

", "rawMarkdown": "Hello,how do you evaluation them?" }, { "id": 1365882, "postDate": "2021-06-26T08:02:47.037Z", "content": "

Hi, we evaluated the training set's out-of-fold predictions with the mAP calculator from Kerem's notebook and used 2/3 and 1/3 weights for the study and image parts.
\ntotal_mAP = (2/3) * study_mAP + (1/3) * image_mAP

", "rawMarkdown": "Hi, we evaluated the training set's out-of-fold predictions with the mAP calculator from [Kerem's notebook](https://www.kaggle.com/keremt/competition-metric-map-0-5) and used 2/3 and 1/3 weights for the study and image parts.\n`total_mAP = (2/3) * study_mAP + (1/3) * image_mAP`" }, { "id": 1365902, "postDate": "2021-06-26T08:20:37.317Z", "content": "

But this notebook is only for study_level,is it right?

", "rawMarkdown": "But this notebook is only for study_level,is it right?", "votes": 1 }, { "id": 1365930, "postDate": "2021-06-26T09:11:36.210Z", "content": "

Yes, that's right. The notebook only has the study level scoring implemented, but the image-level score can be calculated with the same VinBigDataEval mAP calculator class. GT annotations and predicted boxes just need to be fed in the correct format.

", "rawMarkdown": "Yes, that's right. The notebook only has the study level scoring implemented, but the image-level score can be calculated with the same `VinBigDataEval` mAP calculator class. GT annotations and predicted boxes just need to be fed in the correct format." }, { "id": 1366071, "postDate": "2021-06-26T12:46:02.207Z", "content": "

Thanks!!!!!!

", "rawMarkdown": "Thanks!!!!!!", "votes": 1 } ] }, { "id": 1364273, "postDate": "2021-06-24T19:18:38.563Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1380378, "postDate": "2021-07-08T04:45:10.357Z", "content": "

Thanks for the info

", "rawMarkdown": "Thanks for the info" } ], "comments": [ { "id": 1364424, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2021-06-25T01:46:39.790000", "content": "

study-level only (none 1 0 0 1 1 for image-level prediction)

\n

arch : effnet-b7
\nres : 640
\nloss : CE loss (w/o aux loss)
\nvalidation : GroupKFold 5 folds
\nTTA : flip + brightness

\n

CV

\n\n

LB

\n", "votes": 7, "replies": [ { "id": 1364497, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-06-25T04:21:29.397000", "content": "

That's a very impressive score without aux loss. Do you have any other different methods in your model?

", "votes": 0, "replies": [] }, { "id": 1364564, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2021-06-25T04:55:51.600000", "content": "

yeap. I added more augmentations (e.g. cutout, …) & slightly changed the head network to regularize the model better!

", "votes": 0, "replies": [] } ] }, { "id": 1379931, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-07T17:13:30.567000", "content": "

Efnv2-M
\n512 Image Size
\nBCE + (Lovasz + BCE) AUX
\nGroupKFold
\nPytorch
\nMixed Precision
\nScale : 1.25

\n

CV Score (Group K Fold)

\n

\"scores\": [
\n 0.39388390121767813,
\n 0.39234564782357945,
\n 0.3847761244542205,
\n 0.37818329809109086,
\n 0.38300730435642993
\n ]

\n

LB score: 45.6 (study level only, use 'none 1 0 0 1 1' for image level prediction string)

\n

With help of some invaluable pointers from this thread and @harshitsheoran, thanks for the help

", "votes": 6, "replies": [ { "id": 1379998, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-07T18:02:45.800000", "content": "

What exactly is scale 1.25?

", "votes": 0, "replies": [] }, { "id": 1380005, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-07T18:11:50.387000", "content": "

increasing the image size from 512 to 640 at inference

", "votes": 1, "replies": [] }, { "id": 1380527, "author_name": "AndyX", "author_url": "", "post_date": "2021-07-08T06:36:36.853000", "content": "

Can you reveal the probability of applying scale？

", "votes": 0, "replies": [] }, { "id": 1383386, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-10T19:23:22.283000", "content": "

@syxuming its just a fixed scale there is no TTA step for these scores

", "votes": 0, "replies": [] }, { "id": 1386950, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-13T20:03:20.467000", "content": "

Hi @varundutt9213

\n

Are you removing duplicates? How are you doing this Group K Fold split? I can't find a way to achieve such high scores in my folds.

", "votes": 0, "replies": [] }, { "id": 1388030, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-14T15:52:33.140000", "content": "

No these scores are without any preprocessing and the splitting has no tricks just simple group kfold

", "votes": 1, "replies": [] }, { "id": 1391427, "author_name": "Ơ con lừa!", "author_url": "", "post_date": "2021-07-17T15:27:26.493000", "content": "

@varundutt9213 Hi guy, I tried to train eff-v2_m with image_size = 512, finetune its with blocks[4] but i get an error \"ValueError: Input contains NaN, infinity or a value too large for dtype('float16').\"
\ni still fix this err but it does not improve. i guest that anything diffirent with eff-b7
\nplease help me!

", "votes": 0, "replies": [] }, { "id": 1393328, "author_name": "Victor", "author_url": "", "post_date": "2021-07-19T14:49:35.980000", "content": "

What's the purpose of increasing scale at inference time?

", "votes": 0, "replies": [] }, { "id": 1396527, "author_name": "Drzhuzhe", "author_url": "", "post_date": "2021-07-22T07:15:04.230000", "content": "

may be its due to
\nyou are not using pretraining weight
\nbecause efficientnetv2_m is w/0 pre weight in TIMM
\nonly tf_efficientnetv2_** or efficientnetv2_rw_** has weight

", "votes": 3, "replies": [] }, { "id": 1398105, "author_name": "Ơ con lừa!", "author_url": "", "post_date": "2021-07-23T18:17:00.530000", "content": "

Oh, i don't no that…
\nVery thanks!!!

", "votes": 0, "replies": [] }, { "id": 1398646, "author_name": "Simo Ryu", "author_url": "", "post_date": "2021-07-24T11:05:48.383000", "content": "

Hi, i'm rather new to kaggle. When people talk about \"LB score\", do they mean LB score inferenced with ensemble of all KFOLD trained models? so ensemble of K models? Is your's the case too?

", "votes": 0, "replies": [] } ] }, { "id": 1364125, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-06-24T16:29:47.093000", "content": "

Study Only

\n

tf_eff_b5
\n640 Image Size
\nNo Aux Loss
\nGroupKFold
\nPytorch

\n

CV: 0.368 LB: 442

", "votes": 3, "replies": [ { "id": 1377222, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-05T17:04:41.673000", "content": "

Update:
\nStudy Only

\n

tf_eff_b5
\n640 Image Size
\nAux Loss BCE Mask [128,128] block 5
\nSnapmix 0.5
\nPreprocessing scale 255.0
\nGroupKFold
\nPytorch

\n

CV: 0.370 LB: 447 LB(TTA): 453

", "votes": 2, "replies": [] }, { "id": 1377403, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-05T19:40:27.323000", "content": "

Which TTA was used?

", "votes": 0, "replies": [] }, { "id": 1377406, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-05T19:45:08.163000", "content": "

@harshitsheoran Hflip + RandomResizedCrop TTA3x

", "votes": 0, "replies": [] } ] }, { "id": 1396192, "author_name": "Furkan K", "author_url": "", "post_date": "2021-07-21T21:48:43.643000", "content": "

@harshitsheoran Hello, how do you calculate CV score? Are they losses of the models?
\nThanks.

", "votes": 1, "replies": [ { "id": 1397088, "author_name": "Ioannis M", "author_url": "", "post_date": "2021-07-22T18:54:37.620000", "content": "

there are public notebooks that could help you
\nnb1 , nb2

", "votes": 2, "replies": [] }, { "id": 1397142, "author_name": "Furkan K", "author_url": "", "post_date": "2021-07-22T20:37:12.193000", "content": "

Thank you.

", "votes": 1, "replies": [] } ] }, { "id": 1379489, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-07-07T12:11:00.057000", "content": "

Did you use a particular technic to finetune effnetv2? Or just the classical one ? Because I've seen several learning technics were applied to pre-train the network, but I haven't dived deep into this.

", "votes": 1, "replies": [ { "id": 1379858, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-07T16:38:18.680000", "content": "

Just the classic one.

", "votes": 1, "replies": [] }, { "id": 1380016, "author_name": "mknzfr", "author_url": "", "post_date": "2021-07-07T18:21:55.620000", "content": "

Could you please clarify what is the classic one?

", "votes": 0, "replies": [] }, { "id": 1380039, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-07T18:39:47.543000", "content": "

It means just loading the imagenet weights and retraining the backbone on this train data, nothing special or different at all.

", "votes": 2, "replies": [] } ] }, { "id": 1376199, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-04T23:08:37.113000", "content": "

@harshitsheoran This might be a very vague question but I have very similar CV scores that you have with effv2_l but my lb score is much lower any idea why this might be the case.

\n

Efnv2-L
\n384 Image Size
\nBCE
\nGroupKFold
\nPytorch
\nMixed Precision

\n

CV Score (Group K Fold)

\n

\"scores\": [
\n 0.38221975192699126,
\n 0.3876267083746035,
\n 0.3804051350518518,
\n 0.37546675040210786,
\n 0.38837743907783395
\n ]

\n

LB score: 44.3 (study level only, use 'none 1 0 0 1 1' for image level prediction string)

", "votes": 1, "replies": [ { "id": 1376211, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-05T00:28:34.797000", "content": "

Well, there are many reasons where I saw my score go berserk of tiny changes, learning rate scheduler had huge effects, normalization surely did help the CV just a little but made LB a lot worse and cuz the data in CV is also pretty small so I did not consider it worth doing. I did use a unique aux loss to make it all stable so I dont expect my approach to be stable without it anyways also random seed have effects on CV and LB too, also I am using tf_efficientnetv2_l_in21ft1k and not just the normal weights…

\n

Hope it helps, I know that is a lot of info but I had to try and fail a lot of things after getting to 456 and it is extremely hard to improve even a single point beyond this by just changing model or augs or hyperparameters

\n

Also, changing batch size if especially if its below 32 can cause problems….

", "votes": 2, "replies": [] }, { "id": 1377038, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-05T14:32:42.463000", "content": "

@harshitsheoran thank you for the detailed answer really appreciate it. By skipping normalization do you mean not using image net mean and std and just dividing image by 255.0?

", "votes": 0, "replies": [] }, { "id": 1377180, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-05T16:25:54.520000", "content": "

Yes……….

", "votes": 0, "replies": [] }, { "id": 1377221, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-05T17:03:10.313000", "content": "

That's true. I don't know why but applying normalization lead to a worse LB. Also, batch size is important, but I'm having a hard time fitting a batch like 32 in my GPU, I'm using now batch size 6 for effb5 640x640.

\n

@harshitsheoran can you sure what did you find about learning rate schedulers?

", "votes": 0, "replies": [] }, { "id": 1377402, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-05T19:40:06.600000", "content": "

Well, this might not be true for everyone but what I noted was changing the learning rate scheduler after fixing on augmentations would only lead to worse results, which might mean that learning rate scheduler and augmentations are depends on each other and augs might have to be adjusted if the learning rate scheduler is changed.

", "votes": 0, "replies": [] }, { "id": 1388463, "author_name": "Vatsal Mavani", "author_url": "", "post_date": "2021-07-15T01:37:26.297000", "content": "

Hi @harshitsheoran, thank you for all your discussions, these help a lot. I though have a question, is there any article you can share that explains the relation between augmentation and scheduler, or any notebook to play with the different augs and scheduler parameters?

", "votes": 0, "replies": [] }, { "id": 1388710, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-15T06:47:12.453000", "content": "

Unfortunately as stated above, its just what I noted, its not something which would be true for every case, to my knowledge it would be applied in the cases where there is a lot of noise or the data is small. This was only my speculation so to my knowledge there exists no article or notebook which would be enough to prove/explain my claims

", "votes": 0, "replies": [] } ] }, { "id": 1370679, "author_name": "AndyX", "author_url": "", "post_date": "2021-06-30T10:17:12.483000", "content": "

Could you explain what your aux loss is made up of?

", "votes": 1, "replies": [ { "id": 1370749, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-30T11:38:24.923000", "content": "

It is just BCE. the predictions are masks and the targets are mask made from annotations.

", "votes": 0, "replies": [] }, { "id": 1370806, "author_name": "", "author_url": "", "post_date": "2021-06-30T12:19:38.187000", "content": "", "votes": 1, "replies": [] }, { "id": 1378235, "author_name": "Furkan K", "author_url": "", "post_date": "2021-07-06T11:49:46.150000", "content": "

but isn't this study level prediction? How can you use BCE while there are 4 output labels?

", "votes": 0, "replies": [] }, { "id": 1378543, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-06T15:49:41.597000", "content": "

BCE does not limit to binary class predictions you know

", "votes": 0, "replies": [] }, { "id": 1389933, "author_name": "Alisher Iglymov", "author_url": "", "post_date": "2021-07-16T08:29:46.417000", "content": "

Can you help to clarify: how do you encode labels to calculate bce?
\nSo your activation is sigmoid or softmax? Thanks a lot in advance.

", "votes": 0, "replies": [] }, { "id": 1389995, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-16T09:10:56.640000", "content": "

SIgmoid…

", "votes": 0, "replies": [] } ] }, { "id": 1362910, "author_name": "Tom Darmon", "author_url": "", "post_date": "2021-06-23T18:11:13.460000", "content": "

Hey, thanks for your post. Do you think your increase in performances compare to the public baseline is due to the aux loss or to effnet V2 l ?

", "votes": 1, "replies": [ { "id": 1362947, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-23T18:50:09.597000", "content": "

It was both of them, I improved score a lot by make preprocessing specially for v2, aux loss was implemented uniquely as well it did give a boost in score too. v2 had a bigger boost than aux loss. However none of the boosts were anything you could miss.

", "votes": 2, "replies": [] }, { "id": 1364303, "author_name": "erkut", "author_url": "", "post_date": "2021-06-24T20:27:37.053000", "content": "

What extra things exactly needs to be done for running the V2?

", "votes": 0, "replies": [] }, { "id": 1365550, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-25T21:21:13.187000", "content": "

Nothing in specific, just tune the things like image size, scalings, augmentations and your functions to best suit V2

", "votes": 0, "replies": [] } ] }, { "id": 1386571, "author_name": "Victor", "author_url": "", "post_date": "2021-07-13T14:58:04.490000", "content": "

Can someone explain to me how masks are used, and what is the relation with aux loss? I see people talking about it in lots of threads, but can't quite figure out what it means.

", "votes": 2, "replies": [ { "id": 1386674, "author_name": "atom1231", "author_url": "", "post_date": "2021-07-13T16:03:05.343000", "content": "

https://www.kaggle.com/c/siim-covid19-detection/discussion/245323

", "votes": 4, "replies": [] } ] }, { "id": 1365390, "author_name": "hengck23", "author_url": "", "post_date": "2021-06-25T17:04:56.993000", "content": "

It seems that if you only submit \"opacity score x0 y0 x1 y1\" without \"none … \" for the image, the lb score is less than one. Let the lb score be S, the S/0.1666666666 gives about 0.50

\n

i did not submit one by one (it would waste too many slots), but i estimate a typical lb results would look like this:

\n
0         Negative for Pneumonia : 0.85\n1             Typical Appearance : 0.85\n2       Indeterminate Appearance : 0.30\n3            Atypical Appearance : 0.30\n4            Opacity : 0.55\n5            None : 0.85\n
\n

i am targeting at 0.42+0.25 for the final lb score

", "votes": 2, "replies": [ { "id": 1365464, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-25T18:50:35.723000", "content": "

Thanks for the info!, I wish I could do image detection, I currently have pretty much no idea in that but achieving 0.42 study level looks quite possible, I mean with pretraining, you upgrading your model from M to L and doing pseduo labeling not to mention ensembling could give a boost enough!?

", "votes": 1, "replies": [] } ] }, { "id": 1364407, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-25T00:29:55.240000", "content": "

Each criterion may be different by implement….

", "votes": 2, "replies": [] }, { "id": 1364250, "author_name": "Ayushman Buragohain", "author_url": "", "post_date": "2021-06-24T18:39:20.400000", "content": "

EDIT : LB 0.440 (study level only, use none 1 0 0 1 1 for image level prediction string)

\n

local validation:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
FOLDSmAP*0.66
fold00.3659
fold10.3649
fold20.3459
fold30.3685
fold30.3575
\n

image size = 640 x 640
\nbase = efficientnetb4
\nloss = focal_loss + bce_loss (aux_loss)
\nTTA = null
\ndata = GroupStratifiedKFold

", "votes": 2, "replies": [ { "id": 1364263, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-24T19:04:12.280000", "content": "

Please, 0.440 is extremely hard to get as of now if you use null \" for image level prediction string your actual study level score would be ~0.385

", "votes": 1, "replies": [] }, { "id": 1951021, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-09-22T17:56:38.963000", "content": "

@benihime91 I apologize, I was wrong!

", "votes": 0, "replies": [] } ] }, { "id": 1363884, "author_name": "hengck23", "author_url": "", "post_date": "2021-06-24T12:49:07.167000", "content": "

LB 0.402 (study level only, use null '' for image level prediction string)
\nLB 0.457 (study level only, use 'none 1 0 0 1 1' for image level prediction string)

\n

local validation:

\n
eff2m-512-lovasz        \n            mAP*0.66    CE loss     topk accuracy       \nfold0        0.393118    0.808467    [0.680834   0.85324779 0.95669607 1.        ]               \nfold1        0.388934    0.818398    [0.68243785 0.85404972 0.96230954 1.        ]               \nfold2        0.380347    0.842349    [0.67095736 0.84392599 0.95655672 1.        ]               \nfold3        0.383767    0.891183    [0.66051364 0.83788122 0.94863563 1.        ]               \nfold4        0.388402    0.823305    [0.68167203 0.85691318 0.95016077 1.        ]               \n mean        0.386914                        \n
\n

image size=512x512
\nefficientv2-M
\naux loss : BCE + lovasz
\nTTA: flip + scale1.33

", "votes": 2, "replies": [ { "id": 1364021, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-24T14:49:28.980000", "content": "

Thanks for sharing!, I wish I had experience in scaling images in correspondence to image size as this comeptition preprocessing highly depends on that. It took me a few days to get the scales right for the image size of 384. Yeah increasing image size with scales even through trail and error can give me a better score? Any thoughts on this?

", "votes": 2, "replies": [] }, { "id": 1364126, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-06-24T16:31:00.027000", "content": "

Hi @harshitsheoran
\nwhat do you mean by scaling in correspondence to image size?

", "votes": 0, "replies": [] }, { "id": 1364132, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-24T16:34:38.080000", "content": "

I mean that there are unique scalings to image size like for 384 there is a different \"best\" scaling setting and it is a different setting for 512

", "votes": 1, "replies": [] }, { "id": 1369085, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-29T06:29:47.600000", "content": "

@hengck23 are these scores for stratified k fold or group k fold?

", "votes": 0, "replies": [] } ] }, { "id": 1363183, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2021-06-24T02:01:54.067000", "content": "

If you want to get the score for the study level only, the image level should be empty instead of none.

", "votes": 2, "replies": [ { "id": 1363299, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-24T05:23:34.943000", "content": "

Or I can just subtract ~0.055 from study level to get pretty much the same… I am not asking for study only scores here, I am asking to share CV of whatever you have got with the LB Score.

", "votes": 0, "replies": [] }, { "id": 1363364, "author_name": "hengck23", "author_url": "", "post_date": "2021-06-24T06:01:16.867000", "content": "

anyone has verified that submission of null '' string has resulted in LB score of zero?

", "votes": 0, "replies": [] }, { "id": 1364425, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2021-06-25T01:46:42.280000", "content": "

I made sure of it.

", "votes": 0, "replies": [] } ] }, { "id": 1393836, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-20T00:44:29.130000", "content": "

Despite I can get LB 0.453 w/o tta, my cv doesn't go higher than ~0.377. Any tips on what am I doing wrong?
\nAlso, every time I tried effv2 models it got worse results.
\ntf_eff_b5_ns
\n512 Image Size
\nCE + (0.3BCE + 0.7Lovasz)
\nGroupKfold

\n

CV(5 Folds)
\n0.3797073066234588
\n0.3764741718769073
\n0.376079648733139
\n0.3833852708339691
\n0.3772689402103424

\n

mean = 0.3785675637491991
\nLb: 0.453

", "votes": 0, "replies": [ { "id": 1393928, "author_name": "Ơ con lừa!", "author_url": "", "post_date": "2021-07-20T03:11:11.807000", "content": "

me too, i tried so many times but my mean CV from 5 folds about 0.36~0.37

", "votes": 0, "replies": [] }, { "id": 1397039, "author_name": "Furkan K", "author_url": "", "post_date": "2021-07-22T17:47:10.847000", "content": "

How do you guys calculate CV score?

", "votes": 0, "replies": [] }, { "id": 1402283, "author_name": "Cloudyy", "author_url": "", "post_date": "2021-07-28T04:27:26.170000", "content": "

@igormunizims What batch size did you use?

", "votes": 0, "replies": [] }, { "id": 1402721, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-28T13:37:44.863000", "content": "

@magiccard batch size 8

", "votes": 1, "replies": [] }, { "id": 1402859, "author_name": "Cloudyy", "author_url": "", "post_date": "2021-07-28T15:30:45.817000", "content": "

Thank your reply @igormunizims
\nDid you try increase your batch size? I think it has a pretty big effect.
\nif your GPU doesn't fit you can try tensorflow with TPU
\nAnd the coefficient of lovasz and bce…, try change them

", "votes": 0, "replies": [] }, { "id": 1405119, "author_name": "Ơ con lừa!", "author_url": "", "post_date": "2021-07-30T14:03:13.437000", "content": "

@igormunizims Which number epochs did you use? Increase number of epochs can get better score

", "votes": 0, "replies": [] } ] }, { "id": 1387516, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2021-07-14T08:42:54.863000", "content": "

Do you use this dataset ? https://www.kaggle.com/raddar/ricord-covid19-xray-positive-tests

", "votes": 0, "replies": [ { "id": 1387637, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-14T10:42:14.313000", "content": "

I have not used it yet…

", "votes": 0, "replies": [] }, { "id": 1387863, "author_name": "AndyX", "author_url": "", "post_date": "2021-07-14T13:28:54.117000", "content": "

Since this image is from the source data training set, I don't know what to do with them, does anyone have a good idea?

", "votes": 0, "replies": [] } ] }, { "id": 1386903, "author_name": "Furkan K", "author_url": "", "post_date": "2021-07-13T19:14:24.547000", "content": "

Thanks for sharing. Was your effnet custom keras effnet or torch effnet?

", "votes": 0, "replies": [ { "id": 1386996, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-13T21:00:09.827000", "content": "

torch effnet

", "votes": 0, "replies": [] } ] }, { "id": 1383219, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-07-10T16:23:56.947000", "content": "

Any of the aux loss user with scores > 0.456 would care to tell where they attached the aux head ?

", "votes": 0, "replies": [ { "id": 1383239, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-10T16:38:52.360000", "content": "

Do you mean in what block to attach the CNN head? If so, I've tried only in block 5, but my best score is 0.453. I built a dynamic code where I can change the block to attach the head and the CNN architecture using the command line, so I'll try different configurations now and let you know what I found.

", "votes": 2, "replies": [] }, { "id": 1383254, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-07-10T16:47:27.967000", "content": "

Thanks for the answer. Do you use effnet v2 ? I'm using tensorflow, so I got a full graph without blocks …

", "votes": 0, "replies": [] }, { "id": 1383261, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-10T16:49:51.557000", "content": "

I tried effnetv2 but it's not my best model. Not sure if I'm doing something wrong

", "votes": 1, "replies": [] }, { "id": 1383265, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-07-10T16:54:08.850000", "content": "

I've been trying effv2 for two weeks, I can't make something better than with effnetv1 B7 …

", "votes": 0, "replies": [] }, { "id": 1383307, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-10T17:43:51.057000", "content": "

For me ~2 or 3 blocks before the pooling layer worked the best got LB score of 45.6 without any post processing

", "votes": 1, "replies": [] }, { "id": 1383310, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-07-10T17:50:17.733000", "content": "

@varundutt9213 did you use only one block(e.g block 4 or block 5) or applied mask in more than one?

", "votes": 1, "replies": [] }, { "id": 1383315, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-10T17:58:41.523000", "content": "

i have not tried training model by taking masks from multiple blocks.

", "votes": 1, "replies": [] }, { "id": 1383330, "author_name": "Shubham Thapa", "author_url": "", "post_date": "2021-07-10T18:17:39.293000", "content": "

@varundutt9213 how much batch size are you able to fit in ? i am able to fit bs of 8 in effv2 large with image size = 512 , batch size is a key factor due to batch normalization and its increasing the model performance for sure i am curious to know how much you are using .

", "votes": 1, "replies": [] }, { "id": 1383339, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-07-10T18:28:43.197000", "content": "

Bs of 32*8 so 256, with TPU.

", "votes": 1, "replies": [] }, { "id": 1383368, "author_name": "Shubham Thapa", "author_url": "", "post_date": "2021-07-10T19:03:19.227000", "content": "

wow and i am able to fit a bs of 8*8 in torch xla ( TPU ) with resnet101d which is smaller than effv2_l and in gpu for effv2_l its 8 only .

", "votes": 1, "replies": [] }, { "id": 1383376, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-10T19:11:26.697000", "content": "

@trooperog On a single gpu i think the largest model with corresponding largest image size that you can use is effnetv2-m with 512x512 image size I think you can fit 21 sample batches, anything larger will just overfit and give a poor performance even with AUX loss.

", "votes": 1, "replies": [ { "id": 1383395, "author_name": "Shubham Thapa", "author_url": "", "post_date": "2021-07-10T19:30:08.983000", "content": "

you mean it will give OOM error right ? yes i tested it earlier and it is 20

", "votes": 0, "replies": [] } ] }, { "id": 1383383, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-10T19:18:47.347000", "content": "

I am using 32 batch size with effv2L… lower batch size does give really poor performance and accumulations did not work because the random seeding is disrupted.

", "votes": 1, "replies": [ { "id": 1383392, "author_name": "Shubham Thapa", "author_url": "", "post_date": "2021-07-10T19:28:14.063000", "content": "

@harshitsheoran yes lower bs is really giving poor performance as i tested it with my v2_medium model and the results were very different , but you are using local machine ? as you cannot fit a effv2 large with a batch size of 32 or even 20 with a image size of 384 or 512 in kaggle or colab gpu .

", "votes": 0, "replies": [] } ] }, { "id": 1383413, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-07-10T20:03:36.197000", "content": "

I'm using tensorflow, maybe that's what makes it possible to fit 32*8 batch size idk

", "votes": 0, "replies": [] }, { "id": 1383428, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-10T20:23:27.960000", "content": "

I am using local machine… Using TPU you should be able to fit them, maybe you could use accumulations if you could live with the seed and redesign everything accordingly but it is a very time consuming thing to do.

", "votes": 0, "replies": [] }, { "id": 1396260, "author_name": "Vatsal Mavani", "author_url": "", "post_date": "2021-07-22T01:14:29.107000", "content": "

@josephamigo , I have tried TPU for this competition but I'm not able to use the Sequence generator with model.fit in TPU.

\n

Second, if we use tf.data then is augmentation apply on each epoch in TPU or only applies ones in the beginning?

", "votes": 0, "replies": [] }, { "id": 1396375, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-22T04:13:40.810000", "content": "

I am not familiar with tensorflow data module, but I believe it applies on each epoch.

", "votes": 0, "replies": [] } ] }, { "id": 1380228, "author_name": "Ayushman Buragohain", "author_url": "", "post_date": "2021-07-07T23:50:19.897000", "content": "

What I meant to say was do you just do segm_loss = (Lovasz + BCE) or something like segm_loss = (p*Lovasz + q*BCE) ? What your average segmentation loss bdw ?

", "votes": 0, "replies": [ { "id": 1380233, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-08T00:02:04.670000", "content": "

yeah i am using p=0.75 and q=0.25

", "votes": 0, "replies": [] } ] }, { "id": 1380135, "author_name": "Ayushman Buragohain", "author_url": "", "post_date": "2021-07-07T20:45:55.710000", "content": "

@varundutt9213 Did you use weights for Lovasz and BCE ??

", "votes": 0, "replies": [ { "id": 1380159, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-07T21:30:44.237000", "content": "

What do you mean by weights? I used both losses at training time as segmentation loss

", "votes": 0, "replies": [] } ] }, { "id": 1377660, "author_name": "Yousef Rabi", "author_url": "", "post_date": "2021-07-06T03:35:26.503000", "content": "

Thanks for sharing @harshitsheoran

\n

If you don't mind me asking, how are you dealing with the images that fall under a non-negative study but have no bounding boxes?

", "votes": 0, "replies": [ { "id": 1377750, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-06T05:00:42.583000", "content": "

Currently I am not doing any data cleaning, I have got the data from my teammate and I will be working on it in future, currently I believe the first thing to do would be removing it. I dont think there are many of those images to cause a noise to even bother with, however at the top lb even 0.001 makes a difference in gold and silver.

", "votes": 1, "replies": [] } ] }, { "id": 1376099, "author_name": "isaac in", "author_url": "", "post_date": "2021-07-04T19:05:56.783000", "content": "

Do you use the masks as a 5 channel input or a dual output?

", "votes": 0, "replies": [ { "id": 1376109, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-04T19:22:25.913000", "content": "

Dual Output

", "votes": 0, "replies": [] }, { "id": 1376113, "author_name": "isaac in", "author_url": "", "post_date": "2021-07-04T19:26:08.777000", "content": "

Nice. I am currently experimenting with 5 channel input

", "votes": 0, "replies": [] } ] }, { "id": 1371420, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-07-01T01:43:43.037000", "content": "

What type of data augmentations do you use?

", "votes": 0, "replies": [ { "id": 1371484, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-07-01T03:39:17.947000", "content": "

Very minimalist acutally, RandomResizedCrop, HFlip, Cutout, RandomBrightnessContrast and Rotate, I did try a lot of augmentatios too and making it augmentation heavy but the lb or cv did not increase

", "votes": 2, "replies": [] } ] }, { "id": 1371314, "author_name": "Debarshi Chanda", "author_url": "", "post_date": "2021-06-30T21:33:42.167000", "content": "

How do you generate the mask for aux loss? Is it just the bbox crop or is there any other way?

", "votes": 0, "replies": [ { "id": 1371328, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-30T22:09:58.517000", "content": "

It is just bbox crop.

", "votes": 1, "replies": [] } ] }, { "id": 1365650, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-26T02:03:09.763000", "content": "

Are you applying both flips or just the H Flip

", "votes": 0, "replies": [ { "id": 1365849, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-06-26T07:26:43.427000", "content": "

In augmentations just applying H Flip, and if you asking for tta then I am not applying any.

", "votes": 1, "replies": [] } ] }, { "id": 1365444, "author_name": "Joni Juvonen", "author_url": "", "post_date": "2021-06-25T18:09:52.857000", "content": "

Our First submission with 5-fold-CV YOLOv5 models.
\nCV score was 0.571 and LB 0.563.
\nIn CV score, the study part was 0.35 and the image part 0.22.

", "votes": 0, "replies": [ { "id": 1365759, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-26T05:25:22.670000", "content": "

Hello,how do you evaluation them?

", "votes": 0, "replies": [] }, { "id": 1365882, "author_name": "Joni Juvonen", "author_url": "", "post_date": "2021-06-26T08:02:47.037000", "content": "

Hi, we evaluated the training set's out-of-fold predictions with the mAP calculator from Kerem's notebook and used 2/3 and 1/3 weights for the study and image parts.
\ntotal_mAP = (2/3) * study_mAP + (1/3) * image_mAP

", "votes": 0, "replies": [] }, { "id": 1365902, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-26T08:20:37.317000", "content": "

But this notebook is only for study_level,is it right?

", "votes": 1, "replies": [] }, { "id": 1365930, "author_name": "Joni Juvonen", "author_url": "", "post_date": "2021-06-26T09:11:36.210000", "content": "

Yes, that's right. The notebook only has the study level scoring implemented, but the image-level score can be calculated with the same VinBigDataEval mAP calculator class. GT annotations and predicted boxes just need to be fed in the correct format.

", "votes": 0, "replies": [] }, { "id": 1366071, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-26T12:46:02.207000", "content": "

Thanks!!!!!!

", "votes": 1, "replies": [] } ] }, { "id": 1364273, "author_name": "", "author_url": "", "post_date": "2021-06-24T19:18:38.563000", "content": "", "votes": 0, "replies": [] }, { "id": 1380378, "author_name": "Ayushman Buragohain", "author_url": "", "post_date": "2021-07-08T04:45:10.357000", "content": "

Thanks for the info

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1362632": "Wanna Share your cv score with lb.\n\nLet me start with mine (Study Only)\n\n+Efnv2-L\n+384 Image Size\n+Aux Loss\n+GroupKFold\n+Pytorch\n\nCV (5 folds):\n[0.38026270847655175,\n 0.38925640451414656,\n 0.3869573594325011,\n 0.37830235976215315,\n 0.3903840103131989]\n\nLB w/o tta: 0.456 (~0.401 for actual study only)\n\nI am submitting my score of .617 with study level only the image level part was taken from the public image level notebook of https://www.kaggle.com/micheomaano/siim-cov19-efnb7-yolov5-infer", "1364424": "study-level only (`none 1 0 0 1 1` for image-level prediction)\n\narch : effnet-b7\nres : 640\nloss : CE loss (w/o aux loss)\nvalidation : GroupKFold 5 folds\nTTA : flip + brightness\n\nCV \n* mAP * 2/3 : about 0.37 ~ 0.38\n* AUC : about 0.87 ~ 0.89\n* top-1 acc : about 0.67 ~ 0.68\n\nLB\n* 0.451\n* maybe about 0.4 w/ null prediction ('') for image-level", "1379931": "Efnv2-M\n512 Image Size\nBCE + (Lovasz + BCE) AUX\nGroupKFold\nPytorch\nMixed Precision\nScale : 1.25\n\nCV Score (Group K Fold)\n\n\"scores\": [\n 0.39388390121767813,\n 0.39234564782357945,\n 0.3847761244542205,\n 0.37818329809109086,\n 0.38300730435642993\n ]\n\nLB score: 45.6 (study level only, use 'none 1 0 0 1 1' for image level prediction string)\n\nWith help of some invaluable pointers from this thread and @harshitsheoran, thanks for the help", "1364125": "Study Only\n\ntf_eff_b5\n640 Image Size\nNo Aux Loss\nGroupKFold\nPytorch\n\nCV: 0.368 LB: 442", "1396192": "@harshitsheoran Hello, how do you calculate CV score? Are they losses of the models?\nThanks.", "1379489": "Did you use a particular technic to finetune effnetv2? Or just the classical one ? Because I've seen several learning technics were applied to pre-train the network, but I haven't dived deep into this.", "1376199": "@harshitsheoran This might be a very vague question but I have very similar CV scores that you have with effv2_l but my lb score is much lower any idea why this might be the case.\n\nEfnv2-L\n384 Image Size\nBCE\nGroupKFold\nPytorch\nMixed Precision\n\nCV Score (Group K Fold)\n\n\"scores\": [\n 0.38221975192699126,\n 0.3876267083746035,\n 0.3804051350518518,\n 0.37546675040210786,\n 0.38837743907783395\n ]\n\nLB score: 44.3 (study level only, use 'none 1 0 0 1 1' for image level prediction string)", "1370679": "Could you explain what your aux loss is made up of?", "1362910": "Hey, thanks for your post. Do you think your increase in performances compare to the public baseline is due to the aux loss or to effnet V2 l ? ", "1386571": "Can someone explain to me how masks are used, and what is the relation with aux loss? I see people talking about it in lots of threads, but can't quite figure out what it means.", "1365390": "It seems that if you only submit \"opacity score x0 y0 x1 y1\" without \"none ... \" for the image, the lb score is less than one. Let the lb score be S, the S/0.1666666666 gives about 0.50\n\ni did not submit one by one (it would waste too many slots), but i estimate a typical lb results would look like this:\n```\n0 Negative for Pneumonia : 0.85\n1 Typical Appearance : 0.85\n2 Indeterminate Appearance : 0.30\n3 Atypical Appearance : 0.30\n4 Opacity : 0.55\n5 None : 0.85\n```\n\ni am targeting at 0.42+0.25 for the final lb score", "1364407": "Each criterion may be different by implement....", "1364250": "EDIT : LB 0.440 (study level only, use `none 1 0 0 1 1` for image level prediction string)\n\nlocal validation:\n\n| FOLDS | mAP*0.66 |\n| --------- | ----------- |\n| fold0 | 0.3659 |\n| fold1 | 0.3649 |\n| fold2 | 0.3459 |\n| fold3 | 0.3685 |\n| fold3 | 0.3575 |\n\n \nimage size = 640 x 640\nbase = efficientnetb4\nloss = focal_loss + bce_loss (aux_loss)\nTTA = null\ndata = [GroupStratifiedKFold](https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html#:~:text=Stratified%20K%2DFolds%20iterator%20variant,of%20samples%20for%20each%20class.)\n\n", "1363884": "LB 0.402 (study level only, use null '' for image level prediction string)\nLB 0.457 (study level only, use 'none 1 0 0 1 1' for image level prediction string)\n\nlocal validation:\n```\neff2m-512-lovasz\t\t\n\t\t\tmAP*0.66\tCE loss\t\ttopk accuracy\t\t\nfold0\t\t0.393118 \t0.808467\t[0.680834 0.85324779 0.95669607 1. ]\t\t\t\t\nfold1\t\t0.388934 \t0.818398\t[0.68243785 0.85404972 0.96230954 1. ]\t\t\t\t\nfold2\t\t0.380347 \t0.842349\t[0.67095736 0.84392599 0.95655672 1. ]\t\t\t\t\nfold3\t\t0.383767 \t0.891183\t[0.66051364 0.83788122 0.94863563 1. ]\t\t\t\t\nfold4\t\t0.388402 \t0.823305\t[0.68167203 0.85691318 0.95016077 1. ]\t\t\t\t\n mean\t\t0.386914 \t\t\t\t\t\t\n\n```\n\nimage size=512x512\nefficientv2-M\naux loss : BCE + lovasz\nTTA: flip + scale1.33", "1363183": "If you want to get the score for the study level only, the image level should be empty instead of none.", "1393836": "Despite I can get LB 0.453 w/o tta, my cv doesn't go higher than ~0.377. Any tips on what am I doing wrong?\nAlso, every time I tried effv2 models it got worse results.\ntf_eff_b5_ns\n512 Image Size\nCE + (0.3*BCE + 0.7*Lovasz)\nGroupKfold\n\nCV(5 Folds)\n0.3797073066234588\n0.3764741718769073\n0.376079648733139\n0.3833852708339691\n0.3772689402103424\n\nmean = 0.3785675637491991\nLb: 0.453", "1387516": "Do you use this dataset ? [https://www.kaggle.com/raddar/ricord-covid19-xray-positive-tests](url)", "1386903": "Thanks for sharing. Was your effnet custom keras effnet or torch effnet?", "1383219": "Any of the aux loss user with scores > 0.456 would care to tell where they attached the aux head ?", "1380228": "What I meant to say was do you just do `segm_loss = (Lovasz + BCE)` or something like `segm_loss = (p*Lovasz + q*BCE)` ? What your average segmentation loss bdw ?", "1380135": "@varundutt9213 Did you use weights for `Lovasz` and `BCE` ??", "1377660": "Thanks for sharing @harshitsheoran\n\nIf you don't mind me asking, how are you dealing with the images that fall under a non-negative study but have no bounding boxes?", "1376099": "Do you use the masks as a 5 channel input or a dual output?", "1371420": "What type of data augmentations do you use?", "1371314": "How do you generate the mask for aux loss? Is it just the bbox crop or is there any other way?", "1365650": "Are you applying both flips or just the H Flip", "1365444": "Our First submission with 5-fold-CV YOLOv5 models.\nCV score was **0.571** and LB **0.563**.\nIn CV score, the study part was 0.35 and the image part 0.22.", "1364273": "", "1380378": "Thanks for the info" } }