{
  "id": 172183,
  "title": "Validation Batch Size for ROC / AUC",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/172183",
  "author_name": "",
  "post_date": "2020-08-04T03:23:53.039533500Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Ok, updating this post from what it was originally.  I was seeing changes (drastic) in validation score based on validation batch size.  It turned out it was a bug, and I did the test on an older notebook and don't have the issue in my newer notebook.</p>\n\n<p>So lesson learned, one way to find a problem is to check your validation scores across various batch sizes.  If something is amiss you will see an inconsistency depending on batch size.</p>\n\n<p>I used the same data each time (set proper seeds):</p>\n\n<p><code>\nBatch Size          First 3 Scores\n16                       .960, .960, .960\n32                       .960, .960, .952. <br>\n64                       .934, .933, .925\n128                     .882, .881, .873\n256                    .882, .882, .873\n512                     .678, .676, .668 \n1024                   .678, .676, .669\n</code></p>\n\n<p>Validation on batch size 16 was 45 seconds longer per batch than 64-1024.  size 32 only added about 15 extra seconds per batch.  </p>",
  "messages": [
    {
      "id": "957073",
      "postDate": "08/04/2020 03:23:53",
      "content": "<p>Ok, updating this post from what it was originally.  I was seeing changes (drastic) in validation score based on validation batch size.  It turned out it was a bug, and I did the test on an older notebook and don't have the issue in my newer notebook.</p>\n\n<p>So lesson learned, one way to find a problem is to check your validation scores across various batch sizes.  If something is amiss you will see an inconsistency depending on batch size.</p>\n\n<p>I used the same data each time (set proper seeds):</p>\n\n<p><code>\nBatch Size          First 3 Scores\n16                       .960, .960, .960\n32                       .960, .960, .952. <br>\n64                       .934, .933, .925\n128                     .882, .881, .873\n256                    .882, .882, .873\n512                     .678, .676, .668 \n1024                   .678, .676, .669\n</code></p>\n\n<p>Validation on batch size 16 was 45 seconds longer per batch than 64-1024.  size 32 only added about 15 extra seconds per batch.  </p>",
      "rawMarkdown": "Ok, updating this post from what it was originally.  I was seeing changes (drastic) in validation score based on validation batch size.  It turned out it was a bug, and I did the test on an older notebook and don't have the issue in my newer notebook.\n\nSo lesson learned, one way to find a problem is to check your validation scores across various batch sizes.  If something is amiss you will see an inconsistency depending on batch size.\n\nI used the same data each time (set proper seeds):\n\n````\nBatch Size          First 3 Scores\n16                       .960, .960, .960\n32                       .960, .960, .952.  \n64                       .934, .933, .925\n128                     .882, .881, .873\n256                    .882, .882, .873\n512                     .678, .676, .668 \n1024                   .678, .676, .669\n````\n\nValidation on batch size 16 was 45 seconds longer per batch than 64-1024.  size 32 only added about 15 extra seconds per batch.",
      "votes": null
    },
    {
      "id": "957176",
      "postDate": "08/04/2020 05:48:45",
      "content": "<p>Validation batch size should not affect your score? Practically your model isn't training anymore and is just running some predictions. </p>",
      "rawMarkdown": "Validation batch size should not affect your score? Practically your model isn't training anymore and is just running some predictions.",
      "votes": null
    },
    {
      "id": "957182",
      "postDate": "08/04/2020 05:55:56",
      "content": "<p>yes, turns out I had a bug!</p>",
      "rawMarkdown": "yes, turns out I had a bug!",
      "votes": null
    },
    {
      "id": "957185",
      "postDate": "08/04/2020 05:59:40",
      "content": "<p>Glad it's fixed. Good luck :) </p>",
      "rawMarkdown": "Glad it's fixed. Good luck :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 957176,
      "author_name": "aroraaman",
      "author_url": "",
      "post_date": "08/04/2020 05:48:45",
      "content": "<p>Validation batch size should not affect your score? Practically your model isn't training anymore and is just running some predictions. </p>",
      "votes": null,
      "replies": [
        {
          "id": 957182,
          "author_name": "brianfeeny",
          "author_url": "",
          "post_date": "08/04/2020 05:55:56",
          "content": "<p>yes, turns out I had a bug!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 957185,
          "author_name": "aroraaman",
          "author_url": "",
          "post_date": "08/04/2020 05:59:40",
          "content": "<p>Glad it's fixed. Good luck :) </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "957073": "Ok, updating this post from what it was originally.  I was seeing changes (drastic) in validation score based on validation batch size.  It turned out it was a bug, and I did the test on an older notebook and don't have the issue in my newer notebook.\n\nSo lesson learned, one way to find a problem is to check your validation scores across various batch sizes.  If something is amiss you will see an inconsistency depending on batch size.\n\nI used the same data each time (set proper seeds):\n\n````\nBatch Size          First 3 Scores\n16                       .960, .960, .960\n32                       .960, .960, .952.  \n64                       .934, .933, .925\n128                     .882, .881, .873\n256                    .882, .882, .873\n512                     .678, .676, .668 \n1024                   .678, .676, .669\n````\n\nValidation on batch size 16 was 45 seconds longer per batch than 64-1024.  size 32 only added about 15 extra seconds per batch.",
    "957176": "Validation batch size should not affect your score? Practically your model isn't training anymore and is just running some predictions.",
    "957182": "yes, turns out I had a bug!",
    "957185": "Glad it's fixed. Good luck :)"
  },
  "source": "meta"
}