{
  "id": 160240,
  "title": "Best CV improvement over folds",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/160240",
  "author_name": "",
  "post_date": "2020-06-20T12:22:59.326014100Z",
  "votes": 3,
  "comment_count": 20,
  "views": 0,
  "content": "<p>I am training my B0 model over 3 folds, I see that the Best CV is increasing over each fold, for example <code>cv = [0.90, 0.92, 0.95]</code>.</p>\n\n<p>Why does this happen, and how do I use these models? Like, should I use only the last one, or averaging the predictions from all three would be a good option?  </p>",
  "messages": [
    {
      "id": "894429",
      "postDate": "06/20/2020 12:22:59",
      "content": "<p>I am training my B0 model over 3 folds, I see that the Best CV is increasing over each fold, for example <code>cv = [0.90, 0.92, 0.95]</code>.</p>\n\n<p>Why does this happen, and how do I use these models? Like, should I use only the last one, or averaging the predictions from all three would be a good option?  </p>",
      "rawMarkdown": "I am training my B0 model over 3 folds, I see that the Best CV is increasing over each fold, for example `cv = [0.90, 0.92, 0.95]`.\n\nWhy does this happen, and how do I use these models? Like, should I use only the last one, or averaging the predictions from all three would be a good option?",
      "votes": null
    },
    {
      "id": "894472",
      "postDate": "06/20/2020 13:05:39",
      "content": "<p>Averaging the predictions for each fold is generally done for k-fold.\nAlso make sure that you don't use the same model for k-fold, i.e. independently train a unique model for each fold. :) </p>",
      "rawMarkdown": "Averaging the predictions for each fold is generally done for k-fold.\nAlso make sure that you don't use the same model for k-fold, i.e. independently train a unique model for each fold. :)",
      "votes": null
    },
    {
      "id": "894478",
      "postDate": "06/20/2020 13:11:03",
      "content": "<p>Yes, I am training 3 different models over 3 folds, that's why I am amused why the CV is increasing over  each fold.</p>\n\n<p>What I am doing now is that I am predicting on all three models and then averaging the predictions. Is it cool?</p>",
      "rawMarkdown": "Yes, I am training 3 different models over 3 folds, that's why I am amused why the CV is increasing over  each fold.\n\nWhat I am doing now is that I am predicting on all three models and then averaging the predictions. Is it cool?",
      "votes": null
    },
    {
      "id": "894481",
      "postDate": "06/20/2020 13:14:11",
      "content": "<p>High CV may be due to lucky seed, so the average should be calculated.</p>",
      "rawMarkdown": "High CV may be due to lucky seed, so the average should be calculated.",
      "votes": null
    },
    {
      "id": "894487",
      "postDate": "06/20/2020 13:22:42",
      "content": "<p>Sure, I am actually averaging as of now, but I saw this trend over many trainings I did, and I think it is just not about a lucky seed. :/</p>",
      "rawMarkdown": "Sure, I am actually averaging as of now, but I saw this trend over many trainings I did, and I think it is just not about a lucky seed. :/",
      "votes": null
    },
    {
      "id": "894504",
      "postDate": "06/20/2020 13:42:17",
      "content": "<p>Sounds correct to me!\nTo make sure, you can check the LB for each fold. If the submissions do not have a lot of variance, It should be fine :)</p>",
      "rawMarkdown": "Sounds correct to me!\nTo make sure, you can check the LB for each fold. If the submissions do not have a lot of variance, It should be fine :)",
      "votes": null
    },
    {
      "id": "894518",
      "postDate": "06/20/2020 13:45:16",
      "content": "<p>Sure, will do that, thanks! :)</p>",
      "rawMarkdown": "Sure, will do that, thanks! :)",
      "votes": null
    },
    {
      "id": "894523",
      "postDate": "06/20/2020 13:53:34",
      "content": "<p>It's really weird about cv increasing over each fold. When this happens frequently, I may carefully check the code haha. Anyway, i think averaging is always right. :)</p>",
      "rawMarkdown": "It's really weird about cv increasing over each fold. When this happens frequently, I may carefully check the code haha. Anyway, i think averaging is always right. :)",
      "votes": null
    },
    {
      "id": "894530",
      "postDate": "06/20/2020 13:59:31",
      "content": "<p>Point noted! 😆 </p>",
      "rawMarkdown": "Point noted! 😆",
      "votes": null
    },
    {
      "id": "894579",
      "postDate": "06/20/2020 14:48:09",
      "content": "<p>What if you change the number of folds and/or reshuffle them? If it's still strictly increasing, there's probably some leakage in the models or scores.</p>",
      "rawMarkdown": "What if you change the number of folds and/or reshuffle them? If it's still strictly increasing, there's probably some leakage in the models or scores.",
      "votes": null
    },
    {
      "id": "894609",
      "postDate": "06/20/2020 15:24:37",
      "content": "<p>Didn't try it yet, will do it, thanks!</p>",
      "rawMarkdown": "Didn't try it yet, will do it, thanks!",
      "votes": null
    },
    {
      "id": "894817",
      "postDate": "06/20/2020 19:24:02",
      "content": "<p>Model weights don't automatically re-initialize, so maybe you are picking up training where you left off, with weights from early folds preserved. See if your loss function for the very beginning of your first epoch is abnormally low for subsequent folds.</p>",
      "rawMarkdown": "Model weights don't automatically re-initialize, so maybe you are picking up training where you left off, with weights from early folds preserved. See if your loss function for the very beginning of your first epoch is abnormally low for subsequent folds.",
      "votes": null
    },
    {
      "id": "894979",
      "postDate": "06/21/2020 02:07:06",
      "content": "<p>This seems very much logical according to my case to me, I think this is what is happening. I will see the loss values though! :) </p>",
      "rawMarkdown": "This seems very much logical according to my case to me, I think this is what is happening. I will see the loss values though! :)",
      "votes": null
    },
    {
      "id": "896957",
      "postDate": "06/22/2020 14:39:24",
      "content": "<p>3 fold is not a lot to validate your models. I'm using 5 fold and when computing the standard deviation over my CV scores, I find std is around +/- 0.01. I would recommend increasing your number of folds while decreasing the number of epochs per fold. That way you'll have more stable results.</p>",
      "rawMarkdown": "3 fold is not a lot to validate your models. I'm using 5 fold and when computing the standard deviation over my CV scores, I find std is around +/- 0.01. I would recommend increasing your number of folds while decreasing the number of epochs per fold. That way you'll have more stable results.",
      "votes": null
    },
    {
      "id": "897062",
      "postDate": "06/22/2020 15:44:20",
      "content": "<p>Sure, thanks for the advice! ^_^</p>",
      "rawMarkdown": "Sure, thanks for the advice! ^_^",
      "votes": null
    },
    {
      "id": "909640",
      "postDate": "06/30/2020 17:53:39",
      "content": "<p>Hey, I saw the loss values (I am training it for 5 folds but could get data for 3 as it is training at this time), the loss is as follows:</p>\n\n<p>| Fold | Start | End |\n| --- | --- | --- |\n| 1 | 0.0451 | 0.0172 |\n| 2 | 0.0285 | 0.0077 |\n| 3 | 0.0204 | 0.0037 |</p>\n\n<p>So, I guess they are training without taking weights from the previous folds model.</p>\n\n<p>Also, I want to ask one thing that when I train my model, its CV increases up to <code>0.998</code>, the trend is same as above( <code>[0.895, 0.934, 0.976, 0.987, 0.998]</code>)  but sometimes its not always increasing.</p>\n\n<p>But when I submit predictions from the ensemble of all 5 models, I am getting a score of only <code>0.865</code>.</p>\n\n<p>I can think of only one reason for this, Leak! but I can't find any of it in my pipeline, is their any more way that my model can overfit the validation?</p>\n\n<p>Also, I am using Average Meter class for loss.</p>",
      "rawMarkdown": "Hey, I saw the loss values (I am training it for 5 folds but could get data for 3 as it is training at this time), the loss is as follows:\n\n| Fold | Start | End |\n| --- | --- | --- |\n| 1 | 0.0451 | 0.0172 |\n| 2 | 0.0285 | 0.0077 |\n| 3 | 0.0204 | 0.0037 |\n\nSo, I guess they are training without taking weights from the previous folds model.\n\nAlso, I want to ask one thing that when I train my model, its CV increases up to `0.998`, the trend is same as above( `[0.895, 0.934, 0.976, 0.987, 0.998]`)  but sometimes its not always increasing.\n\nBut when I submit predictions from the ensemble of all 5 models, I am getting a score of only `0.865`.\n\nI can think of only one reason for this, Leak! but I can't find any of it in my pipeline, is their any more way that my model can overfit the validation?\n\nAlso, I am using Average Meter class for loss.",
      "votes": null
    },
    {
      "id": "909647",
      "postDate": "06/30/2020 18:01:10",
      "content": "<p>Hey, <a href=\"/rftexas\">@rftexas</a> I tried using 5 folds, my std is +/- 0.036. CV for all 5 models is <code>[0.8949, 0.9443, 0.9693, 0.9872, 0.9967]</code> I am getting a score of <code>0.886</code> only with this setup but I can't find any leaks in the pipeline. What else can be happening?</p>",
      "rawMarkdown": "Hey, @rftexas I tried using 5 folds, my std is +/- 0.036. CV for all 5 models is `[0.8949, 0.9443, 0.9693, 0.9872, 0.9967]` I am getting a score of `0.886` only with this setup but I can't find any leaks in the pipeline. What else can be happening?",
      "votes": null
    },
    {
      "id": "909669",
      "postDate": "06/30/2020 18:21:43",
      "content": "<p>I've seen CV scores in the high 0.9x range with much lower LB scores in the setting of using external data. The model trains much better on the external data and does not do as well with the 2020 data. I recommend adding the external data to your train dataset, but not your validation dataset. That way your validation dataset is much more comparable to the LB test set.</p>\n\n<p>With this setup, I still get Train AUC in the 0.97+ range, but my validation AUC is a more realistic 0.88 with a similar LB 0.89. (Obviously still room for a lot of improvement, but importantly my validation AUC tracks the LB closely).</p>\n\n<p>Above numbers are for a single fold, which would presumably improve a bit with full 5-fold CV.</p>\n\n<p>I suspect it is necessary to use External data for the best LB scores, but it cannot just be added to the existing data.</p>",
      "rawMarkdown": "I've seen CV scores in the high 0.9x range with much lower LB scores in the setting of using external data. The model trains much better on the external data and does not do as well with the 2020 data. I recommend adding the external data to your train dataset, but not your validation dataset. That way your validation dataset is much more comparable to the LB test set.\n\nWith this setup, I still get Train AUC in the 0.97+ range, but my validation AUC is a more realistic 0.88 with a similar LB 0.89. (Obviously still room for a lot of improvement, but importantly my validation AUC tracks the LB closely).\n\nAbove numbers are for a single fold, which would presumably improve a bit with full 5-fold CV.\n\nI suspect it is necessary to use External data for the best LB scores, but it cannot just be added to the existing data.",
      "votes": null
    },
    {
      "id": "909683",
      "postDate": "06/30/2020 18:30:18",
      "content": "<p>I am not using any external data, just this 2020 data with size 224. :/</p>",
      "rawMarkdown": "I am not using any external data, just this 2020 data with size 224. :/",
      "votes": null
    },
    {
      "id": "909694",
      "postDate": "06/30/2020 18:48:25",
      "content": "<p>If your train and validation scores are consistently better than the LB score, you might be overfitting. What to do next depends on your model. I'm far from an expert on tweaking models, but consider increasing dropout, decreasing the number of layers, decreasing the size of Dense layers. These steps will make it harder for your model to \"memorize\" the training data and force it to generalize more.</p>\n\n<p>I don't know what the best range is for trainable parameters. I tend to work with pretrained models (set as not trainable) and combine with metadata and add additional layers . Usually my models have around 600,000 trainable parameters. But I don't know what is recommended.</p>",
      "rawMarkdown": "If your train and validation scores are consistently better than the LB score, you might be overfitting. What to do next depends on your model. I'm far from an expert on tweaking models, but consider increasing dropout, decreasing the number of layers, decreasing the size of Dense layers. These steps will make it harder for your model to \"memorize\" the training data and force it to generalize more.\n\nI don't know what the best range is for trainable parameters. I tend to work with pretrained models (set as not trainable) and combine with metadata and add additional layers . Usually my models have around 600,000 trainable parameters. But I don't know what is recommended.",
      "votes": null
    },
    {
      "id": "910002",
      "postDate": "07/01/2020 00:59:19",
      "content": "<p>Sure, will do it, thanks! :)</p>",
      "rawMarkdown": "Sure, will do it, thanks! :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 894472,
      "author_name": "",
      "author_url": "",
      "post_date": "06/20/2020 13:05:39",
      "content": "<p>Averaging the predictions for each fold is generally done for k-fold.\nAlso make sure that you don't use the same model for k-fold, i.e. independently train a unique model for each fold. :) </p>",
      "votes": null,
      "replies": [
        {
          "id": 894478,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/20/2020 13:11:03",
          "content": "<p>Yes, I am training 3 different models over 3 folds, that's why I am amused why the CV is increasing over  each fold.</p>\n\n<p>What I am doing now is that I am predicting on all three models and then averaging the predictions. Is it cool?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 894504,
          "author_name": "",
          "author_url": "",
          "post_date": "06/20/2020 13:42:17",
          "content": "<p>Sounds correct to me!\nTo make sure, you can check the LB for each fold. If the submissions do not have a lot of variance, It should be fine :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 894518,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/20/2020 13:45:16",
          "content": "<p>Sure, will do that, thanks! :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 894481,
      "author_name": "sdeagggg",
      "author_url": "",
      "post_date": "06/20/2020 13:14:11",
      "content": "<p>High CV may be due to lucky seed, so the average should be calculated.</p>",
      "votes": null,
      "replies": [
        {
          "id": 894487,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/20/2020 13:22:42",
          "content": "<p>Sure, I am actually averaging as of now, but I saw this trend over many trainings I did, and I think it is just not about a lucky seed. :/</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 894523,
          "author_name": "sdeagggg",
          "author_url": "",
          "post_date": "06/20/2020 13:53:34",
          "content": "<p>It's really weird about cv increasing over each fold. When this happens frequently, I may carefully check the code haha. Anyway, i think averaging is always right. :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 894530,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/20/2020 13:59:31",
          "content": "<p>Point noted! 😆 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 896957,
          "author_name": "rftexas",
          "author_url": "",
          "post_date": "06/22/2020 14:39:24",
          "content": "<p>3 fold is not a lot to validate your models. I'm using 5 fold and when computing the standard deviation over my CV scores, I find std is around +/- 0.01. I would recommend increasing your number of folds while decreasing the number of epochs per fold. That way you'll have more stable results.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 897062,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/22/2020 15:44:20",
          "content": "<p>Sure, thanks for the advice! ^_^</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 909647,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/30/2020 18:01:10",
          "content": "<p>Hey, <a href=\"/rftexas\">@rftexas</a> I tried using 5 folds, my std is +/- 0.036. CV for all 5 models is <code>[0.8949, 0.9443, 0.9693, 0.9872, 0.9967]</code> I am getting a score of <code>0.886</code> only with this setup but I can't find any leaks in the pipeline. What else can be happening?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 909669,
          "author_name": "richardepstein",
          "author_url": "",
          "post_date": "06/30/2020 18:21:43",
          "content": "<p>I've seen CV scores in the high 0.9x range with much lower LB scores in the setting of using external data. The model trains much better on the external data and does not do as well with the 2020 data. I recommend adding the external data to your train dataset, but not your validation dataset. That way your validation dataset is much more comparable to the LB test set.</p>\n\n<p>With this setup, I still get Train AUC in the 0.97+ range, but my validation AUC is a more realistic 0.88 with a similar LB 0.89. (Obviously still room for a lot of improvement, but importantly my validation AUC tracks the LB closely).</p>\n\n<p>Above numbers are for a single fold, which would presumably improve a bit with full 5-fold CV.</p>\n\n<p>I suspect it is necessary to use External data for the best LB scores, but it cannot just be added to the existing data.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 909683,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/30/2020 18:30:18",
          "content": "<p>I am not using any external data, just this 2020 data with size 224. :/</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 909694,
          "author_name": "richardepstein",
          "author_url": "",
          "post_date": "06/30/2020 18:48:25",
          "content": "<p>If your train and validation scores are consistently better than the LB score, you might be overfitting. What to do next depends on your model. I'm far from an expert on tweaking models, but consider increasing dropout, decreasing the number of layers, decreasing the size of Dense layers. These steps will make it harder for your model to \"memorize\" the training data and force it to generalize more.</p>\n\n<p>I don't know what the best range is for trainable parameters. I tend to work with pretrained models (set as not trainable) and combine with metadata and add additional layers . Usually my models have around 600,000 trainable parameters. But I don't know what is recommended.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 910002,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "07/01/2020 00:59:19",
          "content": "<p>Sure, will do it, thanks! :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 894579,
      "author_name": "sburyachenko",
      "author_url": "",
      "post_date": "06/20/2020 14:48:09",
      "content": "<p>What if you change the number of folds and/or reshuffle them? If it's still strictly increasing, there's probably some leakage in the models or scores.</p>",
      "votes": null,
      "replies": [
        {
          "id": 894609,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/20/2020 15:24:37",
          "content": "<p>Didn't try it yet, will do it, thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 894817,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "06/20/2020 19:24:02",
      "content": "<p>Model weights don't automatically re-initialize, so maybe you are picking up training where you left off, with weights from early folds preserved. See if your loss function for the very beginning of your first epoch is abnormally low for subsequent folds.</p>",
      "votes": null,
      "replies": [
        {
          "id": 894979,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/21/2020 02:07:06",
          "content": "<p>This seems very much logical according to my case to me, I think this is what is happening. I will see the loss values though! :) </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 909640,
          "author_name": "sarques",
          "author_url": "",
          "post_date": "06/30/2020 17:53:39",
          "content": "<p>Hey, I saw the loss values (I am training it for 5 folds but could get data for 3 as it is training at this time), the loss is as follows:</p>\n\n<p>| Fold | Start | End |\n| --- | --- | --- |\n| 1 | 0.0451 | 0.0172 |\n| 2 | 0.0285 | 0.0077 |\n| 3 | 0.0204 | 0.0037 |</p>\n\n<p>So, I guess they are training without taking weights from the previous folds model.</p>\n\n<p>Also, I want to ask one thing that when I train my model, its CV increases up to <code>0.998</code>, the trend is same as above( <code>[0.895, 0.934, 0.976, 0.987, 0.998]</code>)  but sometimes its not always increasing.</p>\n\n<p>But when I submit predictions from the ensemble of all 5 models, I am getting a score of only <code>0.865</code>.</p>\n\n<p>I can think of only one reason for this, Leak! but I can't find any of it in my pipeline, is their any more way that my model can overfit the validation?</p>\n\n<p>Also, I am using Average Meter class for loss.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "894429": "I am training my B0 model over 3 folds, I see that the Best CV is increasing over each fold, for example `cv = [0.90, 0.92, 0.95]`.\n\nWhy does this happen, and how do I use these models? Like, should I use only the last one, or averaging the predictions from all three would be a good option?",
    "894472": "Averaging the predictions for each fold is generally done for k-fold.\nAlso make sure that you don't use the same model for k-fold, i.e. independently train a unique model for each fold. :)",
    "894478": "Yes, I am training 3 different models over 3 folds, that's why I am amused why the CV is increasing over  each fold.\n\nWhat I am doing now is that I am predicting on all three models and then averaging the predictions. Is it cool?",
    "894481": "High CV may be due to lucky seed, so the average should be calculated.",
    "894487": "Sure, I am actually averaging as of now, but I saw this trend over many trainings I did, and I think it is just not about a lucky seed. :/",
    "894504": "Sounds correct to me!\nTo make sure, you can check the LB for each fold. If the submissions do not have a lot of variance, It should be fine :)",
    "894518": "Sure, will do that, thanks! :)",
    "894523": "It's really weird about cv increasing over each fold. When this happens frequently, I may carefully check the code haha. Anyway, i think averaging is always right. :)",
    "894530": "Point noted! 😆",
    "894579": "What if you change the number of folds and/or reshuffle them? If it's still strictly increasing, there's probably some leakage in the models or scores.",
    "894609": "Didn't try it yet, will do it, thanks!",
    "894817": "Model weights don't automatically re-initialize, so maybe you are picking up training where you left off, with weights from early folds preserved. See if your loss function for the very beginning of your first epoch is abnormally low for subsequent folds.",
    "894979": "This seems very much logical according to my case to me, I think this is what is happening. I will see the loss values though! :)",
    "896957": "3 fold is not a lot to validate your models. I'm using 5 fold and when computing the standard deviation over my CV scores, I find std is around +/- 0.01. I would recommend increasing your number of folds while decreasing the number of epochs per fold. That way you'll have more stable results.",
    "897062": "Sure, thanks for the advice! ^_^",
    "909640": "Hey, I saw the loss values (I am training it for 5 folds but could get data for 3 as it is training at this time), the loss is as follows:\n\n| Fold | Start | End |\n| --- | --- | --- |\n| 1 | 0.0451 | 0.0172 |\n| 2 | 0.0285 | 0.0077 |\n| 3 | 0.0204 | 0.0037 |\n\nSo, I guess they are training without taking weights from the previous folds model.\n\nAlso, I want to ask one thing that when I train my model, its CV increases up to `0.998`, the trend is same as above( `[0.895, 0.934, 0.976, 0.987, 0.998]`)  but sometimes its not always increasing.\n\nBut when I submit predictions from the ensemble of all 5 models, I am getting a score of only `0.865`.\n\nI can think of only one reason for this, Leak! but I can't find any of it in my pipeline, is their any more way that my model can overfit the validation?\n\nAlso, I am using Average Meter class for loss.",
    "909647": "Hey, @rftexas I tried using 5 folds, my std is +/- 0.036. CV for all 5 models is `[0.8949, 0.9443, 0.9693, 0.9872, 0.9967]` I am getting a score of `0.886` only with this setup but I can't find any leaks in the pipeline. What else can be happening?",
    "909669": "I've seen CV scores in the high 0.9x range with much lower LB scores in the setting of using external data. The model trains much better on the external data and does not do as well with the 2020 data. I recommend adding the external data to your train dataset, but not your validation dataset. That way your validation dataset is much more comparable to the LB test set.\n\nWith this setup, I still get Train AUC in the 0.97+ range, but my validation AUC is a more realistic 0.88 with a similar LB 0.89. (Obviously still room for a lot of improvement, but importantly my validation AUC tracks the LB closely).\n\nAbove numbers are for a single fold, which would presumably improve a bit with full 5-fold CV.\n\nI suspect it is necessary to use External data for the best LB scores, but it cannot just be added to the existing data.",
    "909683": "I am not using any external data, just this 2020 data with size 224. :/",
    "909694": "If your train and validation scores are consistently better than the LB score, you might be overfitting. What to do next depends on your model. I'm far from an expert on tweaking models, but consider increasing dropout, decreasing the number of layers, decreasing the size of Dense layers. These steps will make it harder for your model to \"memorize\" the training data and force it to generalize more.\n\nI don't know what the best range is for trainable parameters. I tend to work with pretrained models (set as not trainable) and combine with metadata and add additional layers . Usually my models have around 600,000 trainable parameters. But I don't know what is recommended.",
    "910002": "Sure, will do it, thanks! :)"
  },
  "source": "meta"
}