{
  "id": 217853,
  "title": "Question about Inference",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/217853",
  "author_name": "",
  "post_date": "2021-02-08T14:21:55.659693400Z",
  "votes": -2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello kagglers ! I train a  5 folds  seresnext50_32x4d model and my CV  improves to 0.8956.However, my lb score is only 0.827.<br>\nThere is something wrong in my inference kernel? <br>\nOr can you share some tips of inference to get a stable score？<br>\nMy valid transforms and test tramsforms is below.<br>\n`<br>\ndef get_valid_transforms():<br>\n    return Compose([<br>\n            CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),<br>\n            Resize(CFG['img_size'], CFG['img_size']),<br>\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),<br>\n            ToTensorV2(p=1.0),<br>\n        ], p=1.)</p>\n<p>def get_inference_transforms():<br>\n    return Compose([<br>\n            RandomResizedCrop(CFG['img_size'], CFG['img_size'],scale=(0.8,1)),<br>\n            Transpose(p=0.5),<br>\n            HorizontalFlip(p=0.5),<br>\n            VerticalFlip(p=0.5),<br>\n            RandomGridShuffle(grid=(5,5), p=0.8),<br>\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),<br>\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),<br>\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),<br>\n            ToTensorV2(p=1.0),<br>\n        ], p=1.)<br>\n`</p>\n<p>Inference Notebook is here:<br>\n<a href=\"https://www.kaggle.com/whutddmm/seresnext\" target=\"_blank\">https://www.kaggle.com/whutddmm/seresnext</a></p>",
  "messages": [
    {
      "id": "1191541",
      "postDate": "02/08/2021 14:21:55",
      "content": "<p>Hello kagglers ! I train a  5 folds  seresnext50_32x4d model and my CV  improves to 0.8956.However, my lb score is only 0.827.<br>\nThere is something wrong in my inference kernel? <br>\nOr can you share some tips of inference to get a stable score？<br>\nMy valid transforms and test tramsforms is below.<br>\n`<br>\ndef get_valid_transforms():<br>\n    return Compose([<br>\n            CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),<br>\n            Resize(CFG['img_size'], CFG['img_size']),<br>\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),<br>\n            ToTensorV2(p=1.0),<br>\n        ], p=1.)</p>\n<p>def get_inference_transforms():<br>\n    return Compose([<br>\n            RandomResizedCrop(CFG['img_size'], CFG['img_size'],scale=(0.8,1)),<br>\n            Transpose(p=0.5),<br>\n            HorizontalFlip(p=0.5),<br>\n            VerticalFlip(p=0.5),<br>\n            RandomGridShuffle(grid=(5,5), p=0.8),<br>\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),<br>\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),<br>\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),<br>\n            ToTensorV2(p=1.0),<br>\n        ], p=1.)<br>\n`</p>\n<p>Inference Notebook is here:<br>\n<a href=\"https://www.kaggle.com/whutddmm/seresnext\" target=\"_blank\">https://www.kaggle.com/whutddmm/seresnext</a></p>",
      "rawMarkdown": "Hello kagglers ! I train a  5 folds  seresnext50_32x4d model and my CV  improves to 0.8956.However, my lb score is only 0.827.\nThere is something wrong in my inference kernel? \nOr can you share some tips of inference to get a stable score？\nMy valid transforms and test tramsforms is below.\n`\ndef get_valid_transforms():\n    return Compose([\n            CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n            Resize(CFG['img_size'], CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n\ndef get_inference_transforms():\n    return Compose([\n            RandomResizedCrop(CFG['img_size'], CFG['img_size'],scale=(0.8,1)),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            RandomGridShuffle(grid=(5,5), p=0.8),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n`\n\nInference Notebook is here:\nhttps://www.kaggle.com/whutddmm/seresnext",
      "votes": null
    },
    {
      "id": "1191725",
      "postDate": "02/08/2021 16:22:02",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/whutddmm\" target=\"_blank\">@whutddmm</a>, you have a different set of augmentations in <code>get_valid_transforms()</code> and in <code>get_inference_transforms()</code>, where some augmentations are random. It is natural that performance may vary if you process images differently. First, I would recommend to use the same set of deterministic augmentations on inference to get rid of this source of the potential performance discrepancy. If the gap is still large after that, you may continue looking at your notebook to identify other reasons.</p>",
      "rawMarkdown": "Hi @whutddmm, you have a different set of augmentations in `get_valid_transforms()` and in `get_inference_transforms()`, where some augmentations are random. It is natural that performance may vary if you process images differently. First, I would recommend to use the same set of deterministic augmentations on inference to get rid of this source of the potential performance discrepancy. If the gap is still large after that, you may continue looking at your notebook to identify other reasons.",
      "votes": null
    },
    {
      "id": "1192480",
      "postDate": "02/09/2021 06:15:01",
      "content": "<p>Thank you for your suggestion, it is very useful!!!When I change the get_inference_transforms , my lb increased from 0.827 to 0.892, which is a great improvement and is  very close to CV 0.8956.But I think if the augmentations is not random, TTA will not work.Am I right? Or do you have any suggestions for using random augmentations. Thank you  again!</p>\n<pre><code>def get_inference_transforms():\n    return Compose([\n        CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n        Resize(CFG['img_size'], CFG['img_size']),\n        Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0),\n    ], p=1.)\n</code></pre>",
      "rawMarkdown": "Thank you for your suggestion, it is very useful!!!When I change the get_inference_transforms , my lb increased from 0.827 to 0.892, which is a great improvement and is  very close to CV 0.8956.But I think if the augmentations is not random, TTA will not work.Am I right? Or do you have any suggestions for using random augmentations. Thank you  again!\n```\ndef get_inference_transforms():\n    return Compose([\n        CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n        Resize(CFG['img_size'], CFG['img_size']),\n        Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0),\n    ], p=1.)\n```",
      "votes": null
    },
    {
      "id": "1193342",
      "postDate": "02/09/2021 15:29:04",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/whutddmm\" target=\"_blank\">@whutddmm</a>, I think for TTA to work, you have to apply it minimum 5-10 iterations. Also, it's often advised to trust your CV than the LB</p>",
      "rawMarkdown": "Hi @whutddmm, I think for TTA to work, you have to apply it minimum 5-10 iterations. Also, it's often advised to trust your CV than the LB",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1191725,
      "author_name": "kozodoi",
      "author_url": "",
      "post_date": "02/08/2021 16:22:02",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/whutddmm\" target=\"_blank\">@whutddmm</a>, you have a different set of augmentations in <code>get_valid_transforms()</code> and in <code>get_inference_transforms()</code>, where some augmentations are random. It is natural that performance may vary if you process images differently. First, I would recommend to use the same set of deterministic augmentations on inference to get rid of this source of the potential performance discrepancy. If the gap is still large after that, you may continue looking at your notebook to identify other reasons.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1192480,
          "author_name": "whutddmm",
          "author_url": "",
          "post_date": "02/09/2021 06:15:01",
          "content": "<p>Thank you for your suggestion, it is very useful!!!When I change the get_inference_transforms , my lb increased from 0.827 to 0.892, which is a great improvement and is  very close to CV 0.8956.But I think if the augmentations is not random, TTA will not work.Am I right? Or do you have any suggestions for using random augmentations. Thank you  again!</p>\n<pre><code>def get_inference_transforms():\n    return Compose([\n        CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n        Resize(CFG['img_size'], CFG['img_size']),\n        Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0),\n    ], p=1.)\n</code></pre>",
          "votes": null,
          "replies": [
            {
              "id": 1193342,
              "author_name": "suryajrrafl",
              "author_url": "",
              "post_date": "02/09/2021 15:29:04",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/whutddmm\" target=\"_blank\">@whutddmm</a>, I think for TTA to work, you have to apply it minimum 5-10 iterations. Also, it's often advised to trust your CV than the LB</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1191541": "Hello kagglers ! I train a  5 folds  seresnext50_32x4d model and my CV  improves to 0.8956.However, my lb score is only 0.827.\nThere is something wrong in my inference kernel? \nOr can you share some tips of inference to get a stable score？\nMy valid transforms and test tramsforms is below.\n`\ndef get_valid_transforms():\n    return Compose([\n            CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n            Resize(CFG['img_size'], CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n\ndef get_inference_transforms():\n    return Compose([\n            RandomResizedCrop(CFG['img_size'], CFG['img_size'],scale=(0.8,1)),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            RandomGridShuffle(grid=(5,5), p=0.8),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n`\n\nInference Notebook is here:\nhttps://www.kaggle.com/whutddmm/seresnext",
    "1191725": "Hi @whutddmm, you have a different set of augmentations in `get_valid_transforms()` and in `get_inference_transforms()`, where some augmentations are random. It is natural that performance may vary if you process images differently. First, I would recommend to use the same set of deterministic augmentations on inference to get rid of this source of the potential performance discrepancy. If the gap is still large after that, you may continue looking at your notebook to identify other reasons.",
    "1192480": "Thank you for your suggestion, it is very useful!!!When I change the get_inference_transforms , my lb increased from 0.827 to 0.892, which is a great improvement and is  very close to CV 0.8956.But I think if the augmentations is not random, TTA will not work.Am I right? Or do you have any suggestions for using random augmentations. Thank you  again!\n```\ndef get_inference_transforms():\n    return Compose([\n        CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n        Resize(CFG['img_size'], CFG['img_size']),\n        Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0),\n    ], p=1.)\n```",
    "1193342": "Hi @whutddmm, I think for TTA to work, you have to apply it minimum 5-10 iterations. Also, it's often advised to trust your CV than the LB"
  },
  "source": "meta"
}