{ "id": 243947, "title": "CV scores for Augmentations for this competition", "url": "/competitions/siim-covid19-detection/discussion/243947", "author_name": "Varun Dutt", "post_date": "2021-06-04T15:21:30.155000", "votes": 63, "comment_count": 49, "views": 0, "content": "

This competition seems to be very sensitive to augmentation being applied so I just experimented with common augmentations and recorded their CV scores

\n

Stratified 5 Folds
\nModel: EfficientNet B6
\nImage Size: 512

\n

Augmentations || CV Scores

\n
    \n

  1. H Flip + V Flip || 34.55

    2. \n

  2. Flips + rotate || 35.2

    3. \n

  3. Flips + zoom || 34.87

    4. \n

  4. Flips + shear || 33.89

    5. \n

  5. Flips + shift|| 34.23

    6. \n

  6. Flips + brightness || 35.66

    7. \n

  7. Flips + contrast || 34.71

    8. \n

  8. Flips + hue || 31.9

    9. \n

  9. Flips + saturation || 34.14

    10. \n

  10. Flips + rotate + zoom + brightness || 36.8

    11. \n

  11. Flips + rotate + zoom + brightness + cutout || (37.1/38.5) - CV/LB

    12. \n
\n

Update

\n
    \n
  1. H Flip + rotate + zoom + brightness + cutout || (38.1/38.7) - CV/LB
    2. \n
", "messages": [ { "id": 1336004, "postDate": "2021-06-04T15:21:30.157Z", "content": "

This competition seems to be very sensitive to augmentation being applied so I just experimented with common augmentations and recorded their CV scores

\n

Stratified 5 Folds
\nModel: EfficientNet B6
\nImage Size: 512

\n

Augmentations || CV Scores

\n
    \n

  1. H Flip + V Flip || 34.55

    2. \n

  2. Flips + rotate || 35.2

    3. \n

  3. Flips + zoom || 34.87

    4. \n

  4. Flips + shear || 33.89

    5. \n

  5. Flips + shift|| 34.23

    6. \n

  6. Flips + brightness || 35.66

    7. \n

  7. Flips + contrast || 34.71

    8. \n

  8. Flips + hue || 31.9

    9. \n

  9. Flips + saturation || 34.14

    10. \n

  10. Flips + rotate + zoom + brightness || 36.8

    11. \n

  11. Flips + rotate + zoom + brightness + cutout || (37.1/38.5) - CV/LB

    12. \n
\n

Update

\n
    \n
  1. H Flip + rotate + zoom + brightness + cutout || (38.1/38.7) - CV/LB
    2. \n
", "rawMarkdown": "This competition seems to be very sensitive to augmentation being applied so I just experimented with common augmentations and recorded their CV scores\n\nStratified 5 Folds\nModel: EfficientNet B6\nImage Size: 512\n\nAugmentations || CV Scores\n\n1. H Flip + V Flip || 34.55\n2. Flips + rotate || 35.2\n3. Flips + zoom || 34.87\n4. Flips + shear || 33.89\n5. Flips + shift|| 34.23\n6. Flips + brightness || 35.66\n7. Flips + contrast || 34.71\n8. Flips + hue || 31.9\n9. Flips + saturation || 34.14\n\n10. Flips + rotate + zoom + brightness || 36.8\n11. Flips + rotate + zoom + brightness + cutout || (37.1/38.5) - CV/LB\n\nUpdate\n\n1. H Flip + rotate + zoom + brightness + cutout || (38.1/38.7) - CV/LB", "votes": 63 }, { "id": 1374234, "postDate": "2021-07-03T06:25:18.700Z", "content": "

Thanks for sharing your great experimental results. Thanks a lot!

", "rawMarkdown": "Thanks for sharing your great experimental results. Thanks a lot!", "votes": 1, "replies": [ { "id": 1374299, "postDate": "2021-07-03T07:33:16.823Z", "content": "

My pleasure! Hope it helps!

", "rawMarkdown": "My pleasure! Hope it helps!", "votes": 1 } ] }, { "id": 1372602, "postDate": "2021-07-01T22:03:04.497Z", "content": "

Thanks for sharing your experiments. I have a question about the terms you use. What is \"H Flip\"?

", "rawMarkdown": "Thanks for sharing your experiments. I have a question about the terms you use. What is \"H Flip\"?", "votes": 1, "replies": [ { "id": 1372606, "postDate": "2021-07-01T22:06:59.383Z", "content": "

\"H Flip\" : Horizontal Flip, \"V Flip\": Vertical Flip

", "rawMarkdown": "\"H Flip\" : Horizontal Flip, \"V Flip\": Vertical Flip", "votes": 1 }, { "id": 1372629, "postDate": "2021-07-01T22:24:12.723Z", "content": "

Oh, I see. So is it tf.image.flip_left_right? I acutally have a second question. It would be great if you could answer. \" cutout\" means tfa.image.cutout right?
\ntfa.image.cutout(
\n images: tfa.types.TensorLike,
\n mask_size: tfa.types.TensorLike,
\n offset: tfa.types.TensorLike = (0, 0),
\n constant_values: tfa.types.Number = 0
\n) -> tf.Tensor

\n

I see that the parameters in the function are complicated to me. Could you give me an example how I should use it? For example, tfa.image.cutout(my_image, and what parameters I should put) if the input image size is 512? Thanks!

", "rawMarkdown": "Oh, I see. So is it tf.image.flip_left_right? I acutally have a second question. It would be great if you could answer. \" cutout\" means tfa.image.cutout right? \ntfa.image.cutout(\n images: tfa.types.TensorLike,\n mask_size: tfa.types.TensorLike,\n offset: tfa.types.TensorLike = (0, 0),\n constant_values: tfa.types.Number = 0\n) -> tf.Tensor\n\n\nI see that the parameters in the function are complicated to me. Could you give me an example how I should use it? For example, tfa.image.cutout(my_image, and what parameters I should put) if the input image size is 512? Thanks!", "votes": 1 }, { "id": 1373584, "postDate": "2021-07-02T15:48:16.053Z", "content": "

Yes, \"H flip\" is equivalent to tf.image.flip_left_right. About cutout, @varundutt9213 can confirm that as cutout has slight variants. I don't use TensorFlow so I can not particularly tell about this function.

", "rawMarkdown": "Yes, \"H flip\" is equivalent to tf.image.flip_left_right. About cutout, @varundutt9213 can confirm that as cutout has slight variants. I don't use TensorFlow so I can not particularly tell about this function. \n", "votes": 1 }, { "id": 1373811, "postDate": "2021-07-02T18:13:36.960Z", "content": "

@yus002 Even i use pytorch and not TF, but it should have two main parameters: the length and breadth of the cutouts and the number of cutouts.

", "rawMarkdown": "@yus002 Even i use pytorch and not TF, but it should have two main parameters: the length and breadth of the cutouts and the number of cutouts.", "votes": 1 }, { "id": 1373877, "postDate": "2021-07-02T19:18:01.187Z", "content": "

I see. Thanks! 😃

", "rawMarkdown": "I see. Thanks! 😃 " } ] }, { "id": 1346074, "postDate": "2021-06-12T05:22:35.510Z", "content": "

Hello, did you do only Stratified K Fold or GroupKFold + Stratified K Fold?

", "rawMarkdown": "Hello, did you do only **Stratified K Fold** or **GroupKFold + Stratified K Fold**?", "votes": 1, "replies": [ { "id": 1346950, "postDate": "2021-06-12T19:11:44.210Z", "content": "

The closest thing to GroupKFold + StratifiedKFold I found is Triple Stratified KFold
\nSplitting by group is necessary to avoid leaking, so one way is to apply GroupKFold and manually inspect fold_df.label.value_counts() in each fold to make sure the split is roughly equal

", "rawMarkdown": "The closest thing to GroupKFold + StratifiedKFold I found is [Triple Stratified KFold](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\nSplitting by group is necessary to avoid leaking, so one way is to apply GroupKFold and manually inspect fold_df.label.value_counts() in each fold to make sure the split is roughly equal\n ", "votes": 1 }, { "id": 1346956, "postDate": "2021-06-12T19:16:17.077Z", "content": "

I tried both, these scores are with stratified k fold, stratified worked better for me, what are you using?

", "rawMarkdown": "I tried both, these scores are with stratified k fold, stratified worked better for me, what are you using?", "votes": 1 }, { "id": 1346959, "postDate": "2021-06-12T19:18:39.210Z", "content": "

There are implementations in stack overflow that roughly simulate GroupKFold + Stratified K Fold. Most of them are pretty complex as they are dependent on minimizing some cost function or randomness.

", "rawMarkdown": "There are implementations in stack overflow that roughly simulate GroupKFold + Stratified K Fold. Most of them are pretty complex as they are dependent on minimizing some cost function or randomness. " }, { "id": 1346961, "postDate": "2021-06-12T19:19:35.417Z", "content": "

In this comp I think groupKfold is the best bet. As there are less than 10 labels and 6000+ data points, group k fold gives approximately stratified resuts

", "rawMarkdown": "In this comp I think groupKfold is the best bet. As there are less than 10 labels and 6000+ data points, group k fold gives approximately stratified resuts" }, { "id": 1346963, "postDate": "2021-06-12T19:21:17.913Z", "content": "

@varundutt9213 StratifiedKFold worked better for you because your data is leaking. It will give better CV because there are some common groups in train and validation set.

", "rawMarkdown": "@varundutt9213 StratifiedKFold worked better for you because your data is leaking. It will give better CV because there are some common groups in train and validation set. ", "votes": 2 }, { "id": 1346974, "postDate": "2021-06-12T19:26:55.973Z", "content": "
train['stratify'] = train['label']\nprint('Expected values for stratify column')\nnum_folds = train.fold.nunique()\ntrain.stratify.apply(str).value_counts() / num_folds\n\nfold_dfs = []\nfor fold in range(num_folds): \n    fold_df = train[train.fold == fold]\n    print(f'\\n--- Fold {fold} ---')\n    fold_df.stratify.apply(str).value_counts()\n    print('--------------')\n    fold_dfs.append(fold_df)\n
", "rawMarkdown": "```\ntrain['stratify'] = train['label']\nprint('Expected values for stratify column')\nnum_folds = train.fold.nunique()\ntrain.stratify.apply(str).value_counts() / num_folds\n\nfold_dfs = []\nfor fold in range(num_folds): \n fold_df = train[train.fold == fold]\n print(f'\\n--- Fold {fold} ---')\n fold_df.stratify.apply(str).value_counts()\n print('--------------')\n fold_dfs.append(fold_df)\n```" }, { "id": 1347086, "postDate": "2021-06-12T23:01:56.857Z", "content": "

I'll check for leakage but CV scores with multilabel stratified k fold match very well with LB scores with k=5, so if there is some leakage it will be very minimal.

", "rawMarkdown": "I'll check for leakage but CV scores with multilabel stratified k fold match very well with LB scores with k=5, so if there is some leakage it will be very minimal." } ] }, { "id": 1340486, "postDate": "2021-06-08T03:00:33.207Z", "content": "

How are you computing CV? I mean can you share the code of the metric

", "rawMarkdown": "How are you computing CV? I mean can you share the code of the metric", "votes": 1, "replies": [ { "id": 1340576, "postDate": "2021-06-08T05:48:42.883Z", "content": "

You can use this for 4 class classification model. For detection you can use pycocotools

\n
from sklearn.metrics import average_precision_score\ndef sklearn_mean_ap(preds, targs):\n    \"\"\"\n    Difference from COCO is precision is not interpolated\n    targs: (n), preds: (nx4) \n    \"\"\"\n    return np.mean([average_precision_score(targs==i,preds[:,i]) for i in range(4)])*2/3\n
", "rawMarkdown": "You can use this for 4 class classification model. For detection you can use pycocotools\n\n```\nfrom sklearn.metrics import average_precision_score\ndef sklearn_mean_ap(preds, targs):\n \"\"\"\n Difference from COCO is precision is not interpolated\n targs: (n), preds: (nx4) \n \"\"\"\n return np.mean([average_precision_score(targs==i,preds[:,i]) for i in range(4)])*2/3\n```", "votes": 13 }, { "id": 1341719, "postDate": "2021-06-08T23:09:28.283Z", "content": "

Thank you for sharing evaluation method. I have a question. What does Factor 2/3 mean?

", "rawMarkdown": "Thank you for sharing evaluation method. I have a question. What does Factor 2/3 mean?" }, { "id": 1341773, "postDate": "2021-06-09T01:48:14.933Z", "content": "

mAP is mean average precision, for each class AP is calculated and their average is taken. Here in this competition we have 6 classes: (negative, typical, atypical, indeterminate, none, opacity). Each contribute equally to the final score, and since current approaches separate classification and detection: e.g. train a classifier for negative, typical, atypical, indeterminate and a detector for none, opacity 2/3 comes from 4/6.

", "rawMarkdown": "mAP is mean average precision, for each class AP is calculated and their average is taken. Here in this competition we have 6 classes: (negative, typical, atypical, indeterminate, none, opacity). Each contribute equally to the final score, and since current approaches separate classification and detection: e.g. train a classifier for **negative, typical, atypical, indeterminate** and a detector for **none, opacity** 2/3 comes from 4/6.", "votes": 7 }, { "id": 1342458, "postDate": "2021-06-09T13:22:44.690Z", "content": "

Thank you for replying.

\n

I clearly understood the factor, 2/3.

", "rawMarkdown": "Thank you for replying.\n\nI clearly understood the factor, 2/3." }, { "id": 1344513, "postDate": "2021-06-11T01:17:37.143Z", "content": "

Hello,what is the format of preds and targets?

", "rawMarkdown": "Hello,what is the format of preds and targets?" }, { "id": 1344539, "postDate": "2021-06-11T02:11:01.607Z", "content": "

You can check the documentation: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html. For each target class we create a 1-d binary array by targs==i and preds[:,i] is the indexed probability for that same class again 1-d array.

", "rawMarkdown": "You can check the documentation: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html. For each target class we create a 1-d binary array by `targs==i` and `preds[:,i]` is the indexed probability for that same class again 1-d array.", "votes": 1 }, { "id": 1344552, "postDate": "2021-06-11T02:34:24.367Z", "content": "

Thanks,I got it.

", "rawMarkdown": "Thanks,I got it." }, { "id": 1359926, "postDate": "2021-06-21T16:54:08.037Z", "content": "

@keremt Thanks for your explanations. I still have trouble understanding one part of calculating the metric: how to approach the background class ('none' class in this case) for computing the mAP @ some IoU? I mean when there is only background, we don't have any bboxes so how to compute the IoU in the first place? Sorry if my question is too trivial.

", "rawMarkdown": "@keremt Thanks for your explanations. I still have trouble understanding one part of calculating the metric: how to approach the background class ('none' class in this case) for computing the mAP @ some IoU? I mean when there is only background, we don't have any bboxes so how to compute the IoU in the first place? Sorry if my question is too trivial." } ] }, { "id": 1402982, "postDate": "2021-07-28T17:29:10.053Z", "content": "

Hi, I implemented Coarse dropout (first time I work with it), nevertheless it does not improve my results, do you have any recommendation how to choose Coarse dropout hyperparameters?
\nCoarse dropout Size
\nCoarse dropout number of squares
\nProbability for Coarse dropout

\n

Thanks a lot!

", "rawMarkdown": "Hi, I implemented Coarse dropout (first time I work with it), nevertheless it does not improve my results, do you have any recommendation how to choose Coarse dropout hyperparameters?\nCoarse dropout Size\nCoarse dropout number of squares\nProbability for Coarse dropout \n\nThanks a lot!" }, { "id": 1345579, "postDate": "2021-06-11T17:15:03.027Z", "content": "

Did we need to compute class mAP rather than bbox mAP? I'm confused. Could you please explain me?

", "rawMarkdown": "Did we need to compute class mAP rather than bbox mAP? I'm confused. Could you please explain me?", "replies": [ { "id": 1346071, "postDate": "2021-06-12T05:19:15.493Z", "content": "

There is some issue with object detection scoring of the competition so these scores are for just for the classification task. I used sklearn to get mAP and then multiplied it by 2/3 as we are leaving the two object detection labels. You can check the comments of this discussion to explore it further

", "rawMarkdown": "There is some issue with object detection scoring of the competition so these scores are for just for the classification task. I used sklearn to get mAP and then multiplied it by 2/3 as we are leaving the two object detection labels. You can check the comments of this discussion to explore it further", "votes": 1 } ] }, { "id": 1344821, "postDate": "2021-06-11T06:51:24.480Z", "content": "

what are zoom and cutout meaning of?

", "rawMarkdown": "what are zoom and cutout meaning of?", "replies": [ { "id": 1345547, "postDate": "2021-06-11T16:29:26.023Z", "content": "

zoom is scaling, cutout is removing small patches at random from the image, you can check albumentations library for more details

", "rawMarkdown": "zoom is scaling, cutout is removing small patches at random from the image, you can check albumentations library for more details", "votes": 1 } ] }, { "id": 1343897, "postDate": "2021-06-10T14:00:27.680Z", "content": "

Hello,how did you calculate your cv?

", "rawMarkdown": "Hello,how did you calculate your cv?", "replies": [ { "id": 1345544, "postDate": "2021-06-11T16:28:21.233Z", "content": "

mAP using sklearn

", "rawMarkdown": "mAP using sklearn" } ] }, { "id": 1343446, "postDate": "2021-06-10T07:59:18.783Z", "content": "

Is rotate ShiftScaleRotate or RandomRotate90?

", "rawMarkdown": "Is rotate ShiftScaleRotate or RandomRotate90?", "replies": [ { "id": 1343549, "postDate": "2021-06-10T09:13:03.010Z", "content": "

ShiftScaleRotate with shift being 0.

", "rawMarkdown": "ShiftScaleRotate with shift being 0." } ] }, { "id": 1338987, "postDate": "2021-06-06T23:36:38.837Z", "content": "

Are LB scores from 5 fold average predictions and no TTA?

", "rawMarkdown": "Are LB scores from 5 fold average predictions and no TTA?", "replies": [ { "id": 1338989, "postDate": "2021-06-06T23:40:56.553Z", "content": "

Yes, it's 5 fold average with no TTA, single-fold overfits ill update single-fold LB scores once my submissions get refreshed.

", "rawMarkdown": "Yes, it's 5 fold average with no TTA, single-fold overfits ill update single-fold LB scores once my submissions get refreshed.", "votes": 1 }, { "id": 1338995, "postDate": "2021-06-06T23:53:20.520Z", "content": "

Thanks. I just submitted a single fold without TTA, scored 0.324 another fold scored 0.337. There is some sensitivity to single fold probably due to low data samples. However, 5 fold CV average (stratified kfold shuffle grouped by study ids) scores 0.35 in CV. Now, I will check with 5 fold average predictions as submission to see if LB is similar to what you have, which is always above the CV.

", "rawMarkdown": "Thanks. I just submitted a single fold without TTA, scored 0.324 another fold scored 0.337. There is some sensitivity to single fold probably due to low data samples. However, 5 fold CV average (stratified kfold shuffle grouped by study ids) scores 0.35 in CV. Now, I will check with 5 fold average predictions as submission to see if LB is similar to what you have, which is always above the CV.", "votes": 1 }, { "id": 1339318, "postDate": "2021-06-07T07:20:45.023Z", "content": "

Yeah in my earlier models there was a huge difference in CV and LB scores which was very concerning as the models were almost certainly overfitting LB but for the later models, the gap has decreased.

", "rawMarkdown": "Yeah in my earlier models there was a huge difference in CV and LB scores which was very concerning as the models were almost certainly overfitting LB but for the later models, the gap has decreased.", "votes": 1 }, { "id": 1346969, "postDate": "2021-06-12T19:24:00.757Z", "content": "

@varundutt9213 Use GroupKFold

", "rawMarkdown": "@varundutt9213 Use GroupKFold" } ] }, { "id": 1338697, "postDate": "2021-06-06T16:39:11.077Z", "content": "

Thank you for sharing the results of your wonderful experiment!!

\n

Are all these results based on applying Augmentation to training data only?

\n

I also think that this is an Augmentation sensitive competition.
\nI applied AdjustSaturation on both the training and inference data, and the LB score was greatly improved compared to the other model without it. (0.347 to 0.383)
\n(This is no longer Augmentation, but rather preprocessing.)

\n

I haven't done any parameter tuning yet, but I think that some Augmentations may be more effective when applied to training data only, while others(mainly, Augmentations to change the color of an image) may be more effective when applied to both training data and inference data.

", "rawMarkdown": "Thank you for sharing the results of your wonderful experiment!!\n\nAre all these results based on applying Augmentation to training data only?\n\nI also think that this is an Augmentation sensitive competition.\nI applied AdjustSaturation on both the training and inference data, and the LB score was greatly improved compared to the other model without it. (0.347 to 0.383)\n(This is no longer Augmentation, but rather preprocessing.)\n\nI haven't done any parameter tuning yet, but I think that some Augmentations may be more effective when applied to training data only, while others(mainly, Augmentations to change the color of an image) may be more effective when applied to both training data and inference data.", "votes": 1, "isDeleted": true, "replies": [ { "id": 1338751, "postDate": "2021-06-06T17:40:12.823Z", "content": "

Thank You,

\n

Yes, these augmentations are only applied on the training set, all geometrical transformations I think should be only applied to the training set. But as you mention color pre-processing should work better if it's applied in both test and train time. I'll experiment with it thanks for pointing it out.

\n

Adjusting saturation is very interesting now I think about it, it should work well ill definitely give it a try, thanks for sharing the information.

\n

I have started tuning the parameters, model architecture and depth doesn't seem to be very important parameter tuning and preprocessing and augmentation seems to be a lot more impactful.
\nIll keep updating scores for more augmentations I experiment with.

", "rawMarkdown": "Thank You, \n\nYes, these augmentations are only applied on the training set, all geometrical transformations I think should be only applied to the training set. But as you mention color pre-processing should work better if it's applied in both test and train time. I'll experiment with it thanks for pointing it out.\n\nAdjusting saturation is very interesting now I think about it, it should work well ill definitely give it a try, thanks for sharing the information. \n\nI have started tuning the parameters, model architecture and depth doesn't seem to be very important parameter tuning and preprocessing and augmentation seems to be a lot more impactful.\nIll keep updating scores for more augmentations I experiment with.\n" }, { "id": 1339015, "postDate": "2021-06-07T00:27:16.330Z", "content": "

Very helpful!!

\n
\n

parameter tuning and preprocessing and augmentation seems to be a lot more impactful.

\n
\n

I think so. Thank you!

", "rawMarkdown": "Very helpful!!\n> parameter tuning and preprocessing and augmentation seems to be a lot more impactful.\n\nI think so. Thank you!", "votes": 1, "isDeleted": true }, { "id": 1340560, "postDate": "2021-06-08T05:19:35.703Z", "content": "

Hello , is this single model?

", "rawMarkdown": "Hello , is this single model?" }, { "id": 1340826, "postDate": "2021-06-08T09:35:41.613Z", "content": "

Applying AdjustSaturation to a black and white (or single channel) image doesn't change at all the image.

", "rawMarkdown": "Applying AdjustSaturation to a black and white (or single channel) image doesn't change at all the image." }, { "id": 1341096, "postDate": "2021-06-08T12:29:09.097Z", "content": "

Yes, Efficient B6 with 512 image size

", "rawMarkdown": "Yes, Efficient B6 with 512 image size" }, { "id": 1341116, "postDate": "2021-06-08T12:41:06.817Z", "content": "

It seems to increase the exposure of the image i think saturation is S = [(MaxColor - MinColor) / (MaxColor + MinColor)] ill look into it.

", "rawMarkdown": "It seems to increase the exposure of the image i think saturation is `S = [(MaxColor - MinColor) / (MaxColor + MinColor)]` ill look into it." }, { "id": 1341202, "postDate": "2021-06-08T13:54:53.007Z", "content": "

Thanks…..By the way, what is zoom?

", "rawMarkdown": "Thanks.....By the way, what is zoom?" }, { "id": 1341471, "postDate": "2021-06-08T16:54:01.337Z", "content": "

scaling the image, you can use random or center crop for it.

", "rawMarkdown": "scaling the image, you can use random or center crop for it." }, { "id": 1341775, "postDate": "2021-06-09T01:52:32.723Z", "content": "

Oh,it is similar with crop and resize hhh.

", "rawMarkdown": "Oh,it is similar with crop and resize hhh." } ] } ], "comments": [ { "id": 1374234, "author_name": "kaggler", "author_url": "", "post_date": "2021-07-03T06:25:18.700000", "content": "

Thanks for sharing your great experimental results. Thanks a lot!

", "votes": 1, "replies": [ { "id": 1374299, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-03T07:33:16.823000", "content": "

My pleasure! Hope it helps!

", "votes": 1, "replies": [] } ] }, { "id": 1372602, "author_name": "Yue Sun", "author_url": "", "post_date": "2021-07-01T22:03:04.497000", "content": "

Thanks for sharing your experiments. I have a question about the terms you use. What is \"H Flip\"?

", "votes": 1, "replies": [ { "id": 1372606, "author_name": "Mohammad Zunaed", "author_url": "", "post_date": "2021-07-01T22:06:59.383000", "content": "

\"H Flip\" : Horizontal Flip, \"V Flip\": Vertical Flip

", "votes": 1, "replies": [] }, { "id": 1372629, "author_name": "Yue Sun", "author_url": "", "post_date": "2021-07-01T22:24:12.723000", "content": "

Oh, I see. So is it tf.image.flip_left_right? I acutally have a second question. It would be great if you could answer. \" cutout\" means tfa.image.cutout right?
\ntfa.image.cutout(
\n images: tfa.types.TensorLike,
\n mask_size: tfa.types.TensorLike,
\n offset: tfa.types.TensorLike = (0, 0),
\n constant_values: tfa.types.Number = 0
\n) -> tf.Tensor

\n

I see that the parameters in the function are complicated to me. Could you give me an example how I should use it? For example, tfa.image.cutout(my_image, and what parameters I should put) if the input image size is 512? Thanks!

", "votes": 1, "replies": [] }, { "id": 1373584, "author_name": "Mohammad Zunaed", "author_url": "", "post_date": "2021-07-02T15:48:16.053000", "content": "

Yes, \"H flip\" is equivalent to tf.image.flip_left_right. About cutout, @varundutt9213 can confirm that as cutout has slight variants. I don't use TensorFlow so I can not particularly tell about this function.

", "votes": 1, "replies": [] }, { "id": 1373811, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-07-02T18:13:36.960000", "content": "

@yus002 Even i use pytorch and not TF, but it should have two main parameters: the length and breadth of the cutouts and the number of cutouts.

", "votes": 1, "replies": [] }, { "id": 1373877, "author_name": "Yue Sun", "author_url": "", "post_date": "2021-07-02T19:18:01.187000", "content": "

I see. Thanks! 😃

", "votes": 0, "replies": [] } ] }, { "id": 1346074, "author_name": "Awsaf", "author_url": "", "post_date": "2021-06-12T05:22:35.510000", "content": "

Hello, did you do only Stratified K Fold or GroupKFold + Stratified K Fold?

", "votes": 1, "replies": [ { "id": 1346950, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2021-06-12T19:11:44.210000", "content": "

The closest thing to GroupKFold + StratifiedKFold I found is Triple Stratified KFold
\nSplitting by group is necessary to avoid leaking, so one way is to apply GroupKFold and manually inspect fold_df.label.value_counts() in each fold to make sure the split is roughly equal

", "votes": 1, "replies": [] }, { "id": 1346956, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-12T19:16:17.077000", "content": "

I tried both, these scores are with stratified k fold, stratified worked better for me, what are you using?

", "votes": 1, "replies": [] }, { "id": 1346959, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2021-06-12T19:18:39.210000", "content": "

There are implementations in stack overflow that roughly simulate GroupKFold + Stratified K Fold. Most of them are pretty complex as they are dependent on minimizing some cost function or randomness.

", "votes": 0, "replies": [] }, { "id": 1346961, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2021-06-12T19:19:35.417000", "content": "

In this comp I think groupKfold is the best bet. As there are less than 10 labels and 6000+ data points, group k fold gives approximately stratified resuts

", "votes": 0, "replies": [] }, { "id": 1346963, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2021-06-12T19:21:17.913000", "content": "

@varundutt9213 StratifiedKFold worked better for you because your data is leaking. It will give better CV because there are some common groups in train and validation set.

", "votes": 2, "replies": [] }, { "id": 1346974, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2021-06-12T19:26:55.973000", "content": "
train['stratify'] = train['label']\nprint('Expected values for stratify column')\nnum_folds = train.fold.nunique()\ntrain.stratify.apply(str).value_counts() / num_folds\n\nfold_dfs = []\nfor fold in range(num_folds): \n    fold_df = train[train.fold == fold]\n    print(f'\\n--- Fold {fold} ---')\n    fold_df.stratify.apply(str).value_counts()\n    print('--------------')\n    fold_dfs.append(fold_df)\n
", "votes": 0, "replies": [] }, { "id": 1347086, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-12T23:01:56.857000", "content": "

I'll check for leakage but CV scores with multilabel stratified k fold match very well with LB scores with k=5, so if there is some leakage it will be very minimal.

", "votes": 0, "replies": [] } ] }, { "id": 1340486, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-06-08T03:00:33.207000", "content": "

How are you computing CV? I mean can you share the code of the metric

", "votes": 1, "replies": [ { "id": 1340576, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-06-08T05:48:42.883000", "content": "

You can use this for 4 class classification model. For detection you can use pycocotools

\n
from sklearn.metrics import average_precision_score\ndef sklearn_mean_ap(preds, targs):\n    \"\"\"\n    Difference from COCO is precision is not interpolated\n    targs: (n), preds: (nx4) \n    \"\"\"\n    return np.mean([average_precision_score(targs==i,preds[:,i]) for i in range(4)])*2/3\n
", "votes": 13, "replies": [] }, { "id": 1341719, "author_name": "furu-nag", "author_url": "", "post_date": "2021-06-08T23:09:28.283000", "content": "

Thank you for sharing evaluation method. I have a question. What does Factor 2/3 mean?

", "votes": 0, "replies": [] }, { "id": 1341773, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-06-09T01:48:14.933000", "content": "

mAP is mean average precision, for each class AP is calculated and their average is taken. Here in this competition we have 6 classes: (negative, typical, atypical, indeterminate, none, opacity). Each contribute equally to the final score, and since current approaches separate classification and detection: e.g. train a classifier for negative, typical, atypical, indeterminate and a detector for none, opacity 2/3 comes from 4/6.

", "votes": 7, "replies": [] }, { "id": 1342458, "author_name": "furu-nag", "author_url": "", "post_date": "2021-06-09T13:22:44.690000", "content": "

Thank you for replying.

\n

I clearly understood the factor, 2/3.

", "votes": 0, "replies": [] }, { "id": 1344513, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-11T01:17:37.143000", "content": "

Hello,what is the format of preds and targets?

", "votes": 0, "replies": [] }, { "id": 1344539, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-06-11T02:11:01.607000", "content": "

You can check the documentation: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html. For each target class we create a 1-d binary array by targs==i and preds[:,i] is the indexed probability for that same class again 1-d array.

", "votes": 1, "replies": [] }, { "id": 1344552, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-11T02:34:24.367000", "content": "

Thanks,I got it.

", "votes": 0, "replies": [] }, { "id": 1359926, "author_name": "Moein", "author_url": "", "post_date": "2021-06-21T16:54:08.037000", "content": "

@keremt Thanks for your explanations. I still have trouble understanding one part of calculating the metric: how to approach the background class ('none' class in this case) for computing the mAP @ some IoU? I mean when there is only background, we don't have any bboxes so how to compute the IoU in the first place? Sorry if my question is too trivial.

", "votes": 0, "replies": [] } ] }, { "id": 1402982, "author_name": "OmarArias86", "author_url": "", "post_date": "2021-07-28T17:29:10.053000", "content": "

Hi, I implemented Coarse dropout (first time I work with it), nevertheless it does not improve my results, do you have any recommendation how to choose Coarse dropout hyperparameters?
\nCoarse dropout Size
\nCoarse dropout number of squares
\nProbability for Coarse dropout

\n

Thanks a lot!

", "votes": 0, "replies": [] }, { "id": 1345579, "author_name": "Ichimaru Gin", "author_url": "", "post_date": "2021-06-11T17:15:03.027000", "content": "

Did we need to compute class mAP rather than bbox mAP? I'm confused. Could you please explain me?

", "votes": 0, "replies": [ { "id": 1346071, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-12T05:19:15.493000", "content": "

There is some issue with object detection scoring of the competition so these scores are for just for the classification task. I used sklearn to get mAP and then multiplied it by 2/3 as we are leaving the two object detection labels. You can check the comments of this discussion to explore it further

", "votes": 1, "replies": [] } ] }, { "id": 1344821, "author_name": "mingtai", "author_url": "", "post_date": "2021-06-11T06:51:24.480000", "content": "

what are zoom and cutout meaning of?

", "votes": 0, "replies": [ { "id": 1345547, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-11T16:29:26.023000", "content": "

zoom is scaling, cutout is removing small patches at random from the image, you can check albumentations library for more details

", "votes": 1, "replies": [] } ] }, { "id": 1343897, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-10T14:00:27.680000", "content": "

Hello,how did you calculate your cv?

", "votes": 0, "replies": [ { "id": 1345544, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-11T16:28:21.233000", "content": "

mAP using sklearn

", "votes": 0, "replies": [] } ] }, { "id": 1343446, "author_name": "Shichang Liu", "author_url": "", "post_date": "2021-06-10T07:59:18.783000", "content": "

Is rotate ShiftScaleRotate or RandomRotate90?

", "votes": 0, "replies": [ { "id": 1343549, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-10T09:13:03.010000", "content": "

ShiftScaleRotate with shift being 0.

", "votes": 0, "replies": [] } ] }, { "id": 1338987, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-06-06T23:36:38.837000", "content": "

Are LB scores from 5 fold average predictions and no TTA?

", "votes": 0, "replies": [ { "id": 1338989, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-06T23:40:56.553000", "content": "

Yes, it's 5 fold average with no TTA, single-fold overfits ill update single-fold LB scores once my submissions get refreshed.

", "votes": 1, "replies": [] }, { "id": 1338995, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2021-06-06T23:53:20.520000", "content": "

Thanks. I just submitted a single fold without TTA, scored 0.324 another fold scored 0.337. There is some sensitivity to single fold probably due to low data samples. However, 5 fold CV average (stratified kfold shuffle grouped by study ids) scores 0.35 in CV. Now, I will check with 5 fold average predictions as submission to see if LB is similar to what you have, which is always above the CV.

", "votes": 1, "replies": [] }, { "id": 1339318, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-07T07:20:45.023000", "content": "

Yeah in my earlier models there was a huge difference in CV and LB scores which was very concerning as the models were almost certainly overfitting LB but for the later models, the gap has decreased.

", "votes": 1, "replies": [] }, { "id": 1346969, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2021-06-12T19:24:00.757000", "content": "

@varundutt9213 Use GroupKFold

", "votes": 0, "replies": [] } ] }, { "id": 1338697, "author_name": "", "author_url": "", "post_date": "2021-06-06T16:39:11.077000", "content": "

Thank you for sharing the results of your wonderful experiment!!

\n

Are all these results based on applying Augmentation to training data only?

\n

I also think that this is an Augmentation sensitive competition.
\nI applied AdjustSaturation on both the training and inference data, and the LB score was greatly improved compared to the other model without it. (0.347 to 0.383)
\n(This is no longer Augmentation, but rather preprocessing.)

\n

I haven't done any parameter tuning yet, but I think that some Augmentations may be more effective when applied to training data only, while others(mainly, Augmentations to change the color of an image) may be more effective when applied to both training data and inference data.

", "votes": 1, "replies": [ { "id": 1338751, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-06T17:40:12.823000", "content": "

Thank You,

\n

Yes, these augmentations are only applied on the training set, all geometrical transformations I think should be only applied to the training set. But as you mention color pre-processing should work better if it's applied in both test and train time. I'll experiment with it thanks for pointing it out.

\n

Adjusting saturation is very interesting now I think about it, it should work well ill definitely give it a try, thanks for sharing the information.

\n

I have started tuning the parameters, model architecture and depth doesn't seem to be very important parameter tuning and preprocessing and augmentation seems to be a lot more impactful.
\nIll keep updating scores for more augmentations I experiment with.

", "votes": 0, "replies": [] }, { "id": 1339015, "author_name": "", "author_url": "", "post_date": "2021-06-07T00:27:16.330000", "content": "

Very helpful!!

\n
\n

parameter tuning and preprocessing and augmentation seems to be a lot more impactful.

\n
\n

I think so. Thank you!

", "votes": 1, "replies": [] }, { "id": 1340560, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-08T05:19:35.703000", "content": "

Hello , is this single model?

", "votes": 0, "replies": [] }, { "id": 1340826, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-06-08T09:35:41.613000", "content": "

Applying AdjustSaturation to a black and white (or single channel) image doesn't change at all the image.

", "votes": 0, "replies": [] }, { "id": 1341096, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-08T12:29:09.097000", "content": "

Yes, Efficient B6 with 512 image size

", "votes": 0, "replies": [] }, { "id": 1341116, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-08T12:41:06.817000", "content": "

It seems to increase the exposure of the image i think saturation is S = [(MaxColor - MinColor) / (MaxColor + MinColor)] ill look into it.

", "votes": 0, "replies": [] }, { "id": 1341202, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-08T13:54:53.007000", "content": "

Thanks…..By the way, what is zoom?

", "votes": 0, "replies": [] }, { "id": 1341471, "author_name": "Varun Dutt", "author_url": "", "post_date": "2021-06-08T16:54:01.337000", "content": "

scaling the image, you can use random or center crop for it.

", "votes": 0, "replies": [] }, { "id": 1341775, "author_name": "Zekun", "author_url": "", "post_date": "2021-06-09T01:52:32.723000", "content": "

Oh,it is similar with crop and resize hhh.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1336004": "This competition seems to be very sensitive to augmentation being applied so I just experimented with common augmentations and recorded their CV scores\n\nStratified 5 Folds\nModel: EfficientNet B6\nImage Size: 512\n\nAugmentations || CV Scores\n\n1. H Flip + V Flip || 34.55\n2. Flips + rotate || 35.2\n3. Flips + zoom || 34.87\n4. Flips + shear || 33.89\n5. Flips + shift|| 34.23\n6. Flips + brightness || 35.66\n7. Flips + contrast || 34.71\n8. Flips + hue || 31.9\n9. Flips + saturation || 34.14\n\n10. Flips + rotate + zoom + brightness || 36.8\n11. Flips + rotate + zoom + brightness + cutout || (37.1/38.5) - CV/LB\n\nUpdate\n\n1. H Flip + rotate + zoom + brightness + cutout || (38.1/38.7) - CV/LB", "1374234": "Thanks for sharing your great experimental results. Thanks a lot!", "1372602": "Thanks for sharing your experiments. I have a question about the terms you use. What is \"H Flip\"?", "1346074": "Hello, did you do only **Stratified K Fold** or **GroupKFold + Stratified K Fold**?", "1340486": "How are you computing CV? I mean can you share the code of the metric", "1402982": "Hi, I implemented Coarse dropout (first time I work with it), nevertheless it does not improve my results, do you have any recommendation how to choose Coarse dropout hyperparameters?\nCoarse dropout Size\nCoarse dropout number of squares\nProbability for Coarse dropout \n\nThanks a lot!", "1345579": "Did we need to compute class mAP rather than bbox mAP? I'm confused. Could you please explain me?", "1344821": "what are zoom and cutout meaning of?", "1343897": "Hello,how did you calculate your cv?", "1343446": "Is rotate ShiftScaleRotate or RandomRotate90?", "1338987": "Are LB scores from 5 fold average predictions and no TTA?", "1338697": "Thank you for sharing the results of your wonderful experiment!!\n\nAre all these results based on applying Augmentation to training data only?\n\nI also think that this is an Augmentation sensitive competition.\nI applied AdjustSaturation on both the training and inference data, and the LB score was greatly improved compared to the other model without it. (0.347 to 0.383)\n(This is no longer Augmentation, but rather preprocessing.)\n\nI haven't done any parameter tuning yet, but I think that some Augmentations may be more effective when applied to training data only, while others(mainly, Augmentations to change the color of an image) may be more effective when applied to both training data and inference data." } }