{
  "id": 207065,
  "title": "Good Validation Accuracy but Bad LB",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/207065",
  "author_name": "",
  "post_date": "2020-12-28T04:05:38.776224700Z",
  "votes": 5,
  "comment_count": 13,
  "views": 0,
  "content": "<p>Hello everyone!</p>\n<p>So, I have made a PyTorch Pipeline with decent augmentations, dataloaders, training and validation functions. <br>\nWhen I train a model (in this case, <code>tf_efficientnet_b4_ns</code> from timm), I get good training accuracy (89% at Epoch #10) and decent validation accuracy (~88% at Epoch #10).</p>\n<p>The problem comes when I run the inference script. When committing, it runs perfectly fine but after submitting and waiting for some time, it gives a very low LB (<code>0.412</code>) which is confusing to me.</p>\n<p>I have used other frameworks like the new <a href=\"https://github.com/abhishekkrthakur/tez\" target=\"_blank\">Tez</a> and doing inference using this results in some acceptable LB (~<code>0.888</code>).</p>\n<p>If anyone knows what maybe the reason behind it, please comment; I am new to Competitions and I have a lot to learn!</p>\n<p>Thank you.</p>",
  "messages": [
    {
      "id": "1129067",
      "postDate": "12/28/2020 04:05:38",
      "content": "<p>Hello everyone!</p>\n<p>So, I have made a PyTorch Pipeline with decent augmentations, dataloaders, training and validation functions. <br>\nWhen I train a model (in this case, <code>tf_efficientnet_b4_ns</code> from timm), I get good training accuracy (89% at Epoch #10) and decent validation accuracy (~88% at Epoch #10).</p>\n<p>The problem comes when I run the inference script. When committing, it runs perfectly fine but after submitting and waiting for some time, it gives a very low LB (<code>0.412</code>) which is confusing to me.</p>\n<p>I have used other frameworks like the new <a href=\"https://github.com/abhishekkrthakur/tez\" target=\"_blank\">Tez</a> and doing inference using this results in some acceptable LB (~<code>0.888</code>).</p>\n<p>If anyone knows what maybe the reason behind it, please comment; I am new to Competitions and I have a lot to learn!</p>\n<p>Thank you.</p>",
      "rawMarkdown": "Hello everyone!\n\nSo, I have made a PyTorch Pipeline with decent augmentations, dataloaders, training and validation functions. \nWhen I train a model (in this case, `tf_efficientnet_b4_ns` from timm), I get good training accuracy (89% at Epoch #10) and decent validation accuracy (~88% at Epoch #10).\n\nThe problem comes when I run the inference script. When committing, it runs perfectly fine but after submitting and waiting for some time, it gives a very low LB (`0.412`) which is confusing to me.\n\nI have used other frameworks like the new [Tez](https://github.com/abhishekkrthakur/tez) and doing inference using this results in some acceptable LB (~`0.888`).\n\nIf anyone knows what maybe the reason behind it, please comment; I am new to Competitions and I have a lot to learn!\n\nThank you.",
      "votes": null
    },
    {
      "id": "1129140",
      "postDate": "12/28/2020 05:41:39",
      "content": "<p>Hi, can you provide whether you did normalization in your augmentations pipeline? If that’s not the problem, then there’s other problems with the unity between your train/inference script.  </p>",
      "rawMarkdown": "Hi, can you provide whether you did normalization in your augmentations pipeline? If that’s not the problem, then there’s other problems with the unity between your train/inference script.",
      "votes": null
    },
    {
      "id": "1129156",
      "postDate": "12/28/2020 05:55:00",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/reighns\" target=\"_blank\">@reighns</a>!<br>\nI am using the following Augmentations as of now:</p>\n<pre><code>train_augments = Compose([\n            RandomResizedCrop(Config.CFG['img_size'], Config.CFG['img_size']),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            ShiftScaleRotate(p=0.5),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            CoarseDropout(p=0.5),\n            Cutout(p=0.5),\n            ToTensorV2(p=1.0),\n        ],p=1.)\n\n\nvalid_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n\n\ntest_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n</code></pre>\n<p>As you can see, I am using Normalization during both training and testing.</p>",
      "rawMarkdown": "Hey @reighns!\nI am using the following Augmentations as of now:\n\n```\ntrain_augments = Compose([\n            RandomResizedCrop(Config.CFG['img_size'], Config.CFG['img_size']),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            ShiftScaleRotate(p=0.5),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            CoarseDropout(p=0.5),\n            Cutout(p=0.5),\n            ToTensorV2(p=1.0),\n        ],p=1.)\n    \n\nvalid_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n\n\ntest_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n```\n\nAs you can see, I am using Normalization during both training and testing.",
      "votes": null
    },
    {
      "id": "1129172",
      "postDate": "12/28/2020 06:08:59",
      "content": "<p><a href=\"https://www.kaggle.com/heyytanay\" target=\"_blank\">@heyytanay</a> </p>\n<p>If the augmentation pipeline is consistent with normalization and resizing, there could be a few more issues that causes this.</p>\n<ol>\n<li><p>Check how you load your model in both train and inference, they must have the exact same structure (the only difference I can think of is setting <code>pretrained=False</code> during inference. Even keys matched successfully can indicate different model architecture if both come from the same family (sometimes).</p></li>\n<li><p>Check you are using the same image size during inference.</p></li>\n</ol>\n<p>You can refer to my notebooks for reference on how to keep track of your configs in the training notebook, however, the inference notebook might seem long because I put in too many details 😂.</p>\n<p>If you still cannot find the bug, I suggest you either publish your code and ask here again, or refer to my pipeline for a (possibly) cleaner approach.</p>",
      "rawMarkdown": "heyytanay \n\nIf the augmentation pipeline is consistent with normalization and resizing, there could be a few more issues that causes this.\n\n1. Check how you load your model in both train and inference, they must have the exact same structure (the only difference I can think of is setting `pretrained=False` during inference. Even keys matched successfully can indicate different model architecture if both come from the same family (sometimes).\n\n2. Check you are using the same image size during inference.\n\nYou can refer to my notebooks for reference on how to keep track of your configs in the training notebook, however, the inference notebook might seem long because I put in too many details 😂.\n\nIf you still cannot find the bug, I suggest you either publish your code and ask here again, or refer to my pipeline for a (possibly) cleaner approach.",
      "votes": null
    },
    {
      "id": "1129197",
      "postDate": "12/28/2020 06:28:05",
      "content": "<p>Thanks for the quick pointers. I am pretty sure that I am using the same image sizes across both scripts but I think the only issue that remains is the 1st point. </p>\n<p>In the training notebook, I am installing timm via pip and then getting the model however in the testing script I am using timm from a dataset and then appending it to a system path since we can't use internet to install it via pip.</p>\n<p>I will use the same method in training notebook and report the results!</p>\n<p>Thank you for helping!</p>",
      "rawMarkdown": "Thanks for the quick pointers. I am pretty sure that I am using the same image sizes across both scripts but I think the only issue that remains is the 1st point. \n\nIn the training notebook, I am installing timm via pip and then getting the model however in the testing script I am using timm from a dataset and then appending it to a system path since we can't use internet to install it via pip.\n\nI will use the same method in training notebook and report the results!\n\nThank you for helping!",
      "votes": null
    },
    {
      "id": "1129234",
      "postDate": "12/28/2020 07:07:46",
      "content": "<p><a href=\"https://www.kaggle.com/heyytanay\" target=\"_blank\">@heyytanay</a> Then that should be the problem! As <code>timm's</code> owner often update his <code>pypi</code>, the different versions do matter a lot! Therefore a good practice is either to follow his latest updates, or pick a stable version to use across all training.</p>",
      "rawMarkdown": "heyytanay Then that should be the problem! As `timm's` owner often update his `pypi`, the different versions do matter a lot! Therefore a good practice is either to follow his latest updates, or pick a stable version to use across all training.",
      "votes": null
    },
    {
      "id": "1129305",
      "postDate": "12/28/2020 08:26:41",
      "content": "<p>The easiest way will be to inference on training set using your inference script and check if that give you reasonable score.</p>",
      "rawMarkdown": "The easiest way will be to inference on training set using your inference script and check if that give you reasonable score.",
      "votes": null
    },
    {
      "id": "1129369",
      "postDate": "12/28/2020 09:45:01",
      "content": "<p>I've had a similar experience for the last week, I use the same augmentations for validation and test (like you) and couldn't get my LB score to go above 0.139. The problem later was a bug in my inference kernel. I had to go over extensive debugging and check every line of code alone.</p>",
      "rawMarkdown": "I've had a similar experience for the last week, I use the same augmentations for validation and test (like you) and couldn't get my LB score to go above 0.139. The problem later was a bug in my inference kernel. I had to go over extensive debugging and check every line of code alone.",
      "votes": null
    },
    {
      "id": "1129490",
      "postDate": "12/28/2020 11:33:22",
      "content": "<p>Yes, that's one of the ways I have in mind, I will try them.</p>",
      "rawMarkdown": "Yes, that's one of the ways I have in mind, I will try them.",
      "votes": null
    },
    {
      "id": "1129491",
      "postDate": "12/28/2020 11:33:53",
      "content": "<p>Can you please tell me what that bug was? Just a general description of it will work.</p>",
      "rawMarkdown": "Can you please tell me what that bug was? Just a general description of it will work.",
      "votes": null
    },
    {
      "id": "1130786",
      "postDate": "12/29/2020 09:56:10",
      "content": "<p>Mine was an indentation in the inference method (silly me), I was not averaging the predictions meaning the list of predictions had only the last prediction.</p>",
      "rawMarkdown": "Mine was an indentation in the inference method (silly me), I was not averaging the predictions meaning the list of predictions had only the last prediction.",
      "votes": null
    },
    {
      "id": "1131988",
      "postDate": "12/30/2020 05:24:02",
      "content": "<p>Thanks for the comment! I tried it and it turns out I was doing the very same thing about indentation! Fixed it and now am getting an improved LB of (<code>0.89</code>).</p>",
      "rawMarkdown": "Thanks for the comment! I tried it and it turns out I was doing the very same thing about indentation! Fixed it and now am getting an improved LB of (`0.89`).",
      "votes": null
    },
    {
      "id": "1131993",
      "postDate": "12/30/2020 05:25:08",
      "content": "<p>It turns out, I was missing an Indentation in my inference script! Fixed it and now my tests are running all fine. Thanks for the help!</p>",
      "rawMarkdown": "It turns out, I was missing an Indentation in my inference script! Fixed it and now my tests are running all fine. Thanks for the help!",
      "votes": null
    },
    {
      "id": "1132133",
      "postDate": "12/30/2020 07:25:35",
      "content": "<p>I'm really glad it helped!</p>",
      "rawMarkdown": "I'm really glad it helped!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1129140,
      "author_name": "reighns",
      "author_url": "",
      "post_date": "12/28/2020 05:41:39",
      "content": "<p>Hi, can you provide whether you did normalization in your augmentations pipeline? If that’s not the problem, then there’s other problems with the unity between your train/inference script.  </p>",
      "votes": null,
      "replies": [
        {
          "id": 1129156,
          "author_name": "heyytanay",
          "author_url": "",
          "post_date": "12/28/2020 05:55:00",
          "content": "<p>Hey <a href=\"https://www.kaggle.com/reighns\" target=\"_blank\">@reighns</a>!<br>\nI am using the following Augmentations as of now:</p>\n<pre><code>train_augments = Compose([\n            RandomResizedCrop(Config.CFG['img_size'], Config.CFG['img_size']),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            ShiftScaleRotate(p=0.5),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            CoarseDropout(p=0.5),\n            Cutout(p=0.5),\n            ToTensorV2(p=1.0),\n        ],p=1.)\n\n\nvalid_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n\n\ntest_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n</code></pre>\n<p>As you can see, I am using Normalization during both training and testing.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1129172,
          "author_name": "reighns",
          "author_url": "",
          "post_date": "12/28/2020 06:08:59",
          "content": "<p><a href=\"https://www.kaggle.com/heyytanay\" target=\"_blank\">@heyytanay</a> </p>\n<p>If the augmentation pipeline is consistent with normalization and resizing, there could be a few more issues that causes this.</p>\n<ol>\n<li><p>Check how you load your model in both train and inference, they must have the exact same structure (the only difference I can think of is setting <code>pretrained=False</code> during inference. Even keys matched successfully can indicate different model architecture if both come from the same family (sometimes).</p></li>\n<li><p>Check you are using the same image size during inference.</p></li>\n</ol>\n<p>You can refer to my notebooks for reference on how to keep track of your configs in the training notebook, however, the inference notebook might seem long because I put in too many details 😂.</p>\n<p>If you still cannot find the bug, I suggest you either publish your code and ask here again, or refer to my pipeline for a (possibly) cleaner approach.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1129197,
          "author_name": "heyytanay",
          "author_url": "",
          "post_date": "12/28/2020 06:28:05",
          "content": "<p>Thanks for the quick pointers. I am pretty sure that I am using the same image sizes across both scripts but I think the only issue that remains is the 1st point. </p>\n<p>In the training notebook, I am installing timm via pip and then getting the model however in the testing script I am using timm from a dataset and then appending it to a system path since we can't use internet to install it via pip.</p>\n<p>I will use the same method in training notebook and report the results!</p>\n<p>Thank you for helping!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1129234,
          "author_name": "reighns",
          "author_url": "",
          "post_date": "12/28/2020 07:07:46",
          "content": "<p><a href=\"https://www.kaggle.com/heyytanay\" target=\"_blank\">@heyytanay</a> Then that should be the problem! As <code>timm's</code> owner often update his <code>pypi</code>, the different versions do matter a lot! Therefore a good practice is either to follow his latest updates, or pick a stable version to use across all training.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1131993,
          "author_name": "heyytanay",
          "author_url": "",
          "post_date": "12/30/2020 05:25:08",
          "content": "<p>It turns out, I was missing an Indentation in my inference script! Fixed it and now my tests are running all fine. Thanks for the help!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1129305,
      "author_name": "louis925",
      "author_url": "",
      "post_date": "12/28/2020 08:26:41",
      "content": "<p>The easiest way will be to inference on training set using your inference script and check if that give you reasonable score.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1129490,
          "author_name": "heyytanay",
          "author_url": "",
          "post_date": "12/28/2020 11:33:22",
          "content": "<p>Yes, that's one of the ways I have in mind, I will try them.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1129369,
      "author_name": "zowlex",
      "author_url": "",
      "post_date": "12/28/2020 09:45:01",
      "content": "<p>I've had a similar experience for the last week, I use the same augmentations for validation and test (like you) and couldn't get my LB score to go above 0.139. The problem later was a bug in my inference kernel. I had to go over extensive debugging and check every line of code alone.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1129491,
          "author_name": "heyytanay",
          "author_url": "",
          "post_date": "12/28/2020 11:33:53",
          "content": "<p>Can you please tell me what that bug was? Just a general description of it will work.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1130786,
          "author_name": "zowlex",
          "author_url": "",
          "post_date": "12/29/2020 09:56:10",
          "content": "<p>Mine was an indentation in the inference method (silly me), I was not averaging the predictions meaning the list of predictions had only the last prediction.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1131988,
          "author_name": "heyytanay",
          "author_url": "",
          "post_date": "12/30/2020 05:24:02",
          "content": "<p>Thanks for the comment! I tried it and it turns out I was doing the very same thing about indentation! Fixed it and now am getting an improved LB of (<code>0.89</code>).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1132133,
          "author_name": "zowlex",
          "author_url": "",
          "post_date": "12/30/2020 07:25:35",
          "content": "<p>I'm really glad it helped!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1129067": "Hello everyone!\n\nSo, I have made a PyTorch Pipeline with decent augmentations, dataloaders, training and validation functions. \nWhen I train a model (in this case, `tf_efficientnet_b4_ns` from timm), I get good training accuracy (89% at Epoch #10) and decent validation accuracy (~88% at Epoch #10).\n\nThe problem comes when I run the inference script. When committing, it runs perfectly fine but after submitting and waiting for some time, it gives a very low LB (`0.412`) which is confusing to me.\n\nI have used other frameworks like the new [Tez](https://github.com/abhishekkrthakur/tez) and doing inference using this results in some acceptable LB (~`0.888`).\n\nIf anyone knows what maybe the reason behind it, please comment; I am new to Competitions and I have a lot to learn!\n\nThank you.",
    "1129140": "Hi, can you provide whether you did normalization in your augmentations pipeline? If that’s not the problem, then there’s other problems with the unity between your train/inference script.",
    "1129156": "Hey @reighns!\nI am using the following Augmentations as of now:\n\n```\ntrain_augments = Compose([\n            RandomResizedCrop(Config.CFG['img_size'], Config.CFG['img_size']),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            ShiftScaleRotate(p=0.5),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            CoarseDropout(p=0.5),\n            Cutout(p=0.5),\n            ToTensorV2(p=1.0),\n        ],p=1.)\n    \n\nvalid_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n\n\ntest_augments = Compose([\n            CenterCrop(Config.CFG['img_size'], Config.CFG['img_size'], p=1.),\n            Resize(Config.CFG['img_size'], Config.CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)\n```\n\nAs you can see, I am using Normalization during both training and testing.",
    "1129172": "heyytanay \n\nIf the augmentation pipeline is consistent with normalization and resizing, there could be a few more issues that causes this.\n\n1. Check how you load your model in both train and inference, they must have the exact same structure (the only difference I can think of is setting `pretrained=False` during inference. Even keys matched successfully can indicate different model architecture if both come from the same family (sometimes).\n\n2. Check you are using the same image size during inference.\n\nYou can refer to my notebooks for reference on how to keep track of your configs in the training notebook, however, the inference notebook might seem long because I put in too many details 😂.\n\nIf you still cannot find the bug, I suggest you either publish your code and ask here again, or refer to my pipeline for a (possibly) cleaner approach.",
    "1129197": "Thanks for the quick pointers. I am pretty sure that I am using the same image sizes across both scripts but I think the only issue that remains is the 1st point. \n\nIn the training notebook, I am installing timm via pip and then getting the model however in the testing script I am using timm from a dataset and then appending it to a system path since we can't use internet to install it via pip.\n\nI will use the same method in training notebook and report the results!\n\nThank you for helping!",
    "1129234": "heyytanay Then that should be the problem! As `timm's` owner often update his `pypi`, the different versions do matter a lot! Therefore a good practice is either to follow his latest updates, or pick a stable version to use across all training.",
    "1129305": "The easiest way will be to inference on training set using your inference script and check if that give you reasonable score.",
    "1129369": "I've had a similar experience for the last week, I use the same augmentations for validation and test (like you) and couldn't get my LB score to go above 0.139. The problem later was a bug in my inference kernel. I had to go over extensive debugging and check every line of code alone.",
    "1129490": "Yes, that's one of the ways I have in mind, I will try them.",
    "1129491": "Can you please tell me what that bug was? Just a general description of it will work.",
    "1130786": "Mine was an indentation in the inference method (silly me), I was not averaging the predictions meaning the list of predictions had only the last prediction.",
    "1131988": "Thanks for the comment! I tried it and it turns out I was doing the very same thing about indentation! Fixed it and now am getting an improved LB of (`0.89`).",
    "1131993": "It turns out, I was missing an Indentation in my inference script! Fixed it and now my tests are running all fine. Thanks for the help!",
    "1132133": "I'm really glad it helped!"
  },
  "source": "meta"
}