{
  "id": 172586,
  "title": "Strange problem with submission score",
  "url": "/competitions/landmark-recognition-2020/discussion/172586",
  "author_name": "Alex",
  "post_date": "2020-08-05T16:33:04.499000",
  "votes": 5,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I've created a public kernel <a href=\"https://www.kaggle.com/akensert/glrec-resnet50-arcface-tf2-2?scriptVersionId=40194320\">here</a>. </p>\n\n<p>In <code>version 10</code> I get a score of <code>0.0481</code> (training for 12 epochs with under-sampling). In <code>version 11</code>, which is identical to <code>version 10</code> (unless I accidentally changed something) except for the number of epochs (20), I get a score of <code>0.0000</code>. What could be the reason for this? I've thought about it a bit and can't come up with a good reason.</p>\n\n<p>EDIT: The submission times (training/predicting on private train/test set) were also similar between them, which shouldn't be considering that version 11 trained for many more epochs.</p>",
  "messages": [
    {
      "id": 959515,
      "postDate": "2020-08-05T16:33:04.500Z",
      "content": "<p>I've created a public kernel <a href=\"https://www.kaggle.com/akensert/glrec-resnet50-arcface-tf2-2?scriptVersionId=40194320\">here</a>. </p>\n\n<p>In <code>version 10</code> I get a score of <code>0.0481</code> (training for 12 epochs with under-sampling). In <code>version 11</code>, which is identical to <code>version 10</code> (unless I accidentally changed something) except for the number of epochs (20), I get a score of <code>0.0000</code>. What could be the reason for this? I've thought about it a bit and can't come up with a good reason.</p>\n\n<p>EDIT: The submission times (training/predicting on private train/test set) were also similar between them, which shouldn't be considering that version 11 trained for many more epochs.</p>",
      "rawMarkdown": "I've created a public kernel [here](https://www.kaggle.com/akensert/glrec-resnet50-arcface-tf2-2?scriptVersionId=40194320). \n\nIn `version 10` I get a score of `0.0481` (training for 12 epochs with under-sampling). In `version 11`, which is identical to `version 10` (unless I accidentally changed something) except for the number of epochs (20), I get a score of `0.0000`. What could be the reason for this? I've thought about it a bit and can't come up with a good reason.\n\nEDIT: The submission times (training/predicting on private train/test set) were also similar between them, which shouldn't be considering that version 11 trained for many more epochs.",
      "votes": 5
    },
    {
      "id": 960867,
      "postDate": "2020-08-06T18:48:51.723Z",
      "content": "<p>Just copy Baseline and expected 0.461 but obtain 0.4586. Seems there is inconsistency, evaluation changed over time</p>",
      "rawMarkdown": "Just copy Baseline and expected 0.461 but obtain 0.4586. Seems there is inconsistency, evaluation changed over time",
      "votes": 1,
      "replies": [
        {
          "id": 961018,
          "postDate": "2020-08-06T21:04:12.180Z",
          "content": "<p><code>pydegensac</code> doesn't use seeds from Python. Its random values are coming from C code.</p>",
          "rawMarkdown": "`pydegensac` doesn't use seeds from Python. Its random values are coming from C code."
        }
      ]
    },
    {
      "id": 960877,
      "postDate": "2020-08-06T18:57:55.303Z",
      "content": "<p><a href=\"/akensert\">@akensert</a> I've checked your kernel, and you haven't set up a seed. Setting seeds helps getting reproducible results, it doesn't guaranty cause there might be libraries which don't make use of the libraries you set seeds for.\nYou should set the following seeds:\n    numpy.random.seed(42)\n    tf.set_random_seed(42)</p>\n\n<p>btw you don't need to use 42 (i.e the answer to the ultimate question of Life, the Universe, and Everything 😃 ) as seed. you can choose whatever number you'd like</p>",
      "rawMarkdown": "@akensert I've checked your kernel, and you haven't set up a seed. Setting seeds helps getting reproducible results, it doesn't guaranty cause there might be libraries which don't make use of the libraries you set seeds for.\nYou should set the following seeds:\n    numpy.random.seed(42)\n    tf.set_random_seed(42)\n\nbtw you don't need to use 42 (i.e the answer to the ultimate question of Life, the Universe, and Everything 😃 ) as seed. you can choose whatever number you'd like",
      "votes": 2,
      "replies": [
        {
          "id": 960908,
          "postDate": "2020-08-06T19:27:40.660Z",
          "content": "<p>Thank you for clarifying about the seed number <a href=\"/ngcferreira\">@ngcferreira</a>! I learned something new today (that I can pick a number other than 42!) 😄👍 </p>",
          "rawMarkdown": "Thank you for clarifying about the seed number @ngcferreira! I learned something new today (that I can pick a number other than 42!) 😄👍 "
        },
        {
          "id": 961017,
          "postDate": "2020-08-06T21:03:53.963Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 960867,
      "author_name": "Sxwat",
      "author_url": "",
      "post_date": "2020-08-06T18:48:51.723000",
      "content": "<p>Just copy Baseline and expected 0.461 but obtain 0.4586. Seems there is inconsistency, evaluation changed over time</p>",
      "votes": 1,
      "replies": [
        {
          "id": 961018,
          "author_name": "Cam Askew",
          "author_url": "",
          "post_date": "2020-08-06T21:04:12.180000",
          "content": "<p><code>pydegensac</code> doesn't use seeds from Python. Its random values are coming from C code.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 960877,
      "author_name": "Nuno Ferreira",
      "author_url": "",
      "post_date": "2020-08-06T18:57:55.303000",
      "content": "<p><a href=\"/akensert\">@akensert</a> I've checked your kernel, and you haven't set up a seed. Setting seeds helps getting reproducible results, it doesn't guaranty cause there might be libraries which don't make use of the libraries you set seeds for.\nYou should set the following seeds:\n    numpy.random.seed(42)\n    tf.set_random_seed(42)</p>\n\n<p>btw you don't need to use 42 (i.e the answer to the ultimate question of Life, the Universe, and Everything 😃 ) as seed. you can choose whatever number you'd like</p>",
      "votes": 2,
      "replies": [
        {
          "id": 960908,
          "author_name": "Alex",
          "author_url": "",
          "post_date": "2020-08-06T19:27:40.660000",
          "content": "<p>Thank you for clarifying about the seed number <a href=\"/ngcferreira\">@ngcferreira</a>! I learned something new today (that I can pick a number other than 42!) 😄👍 </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 961017,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-08-06T21:03:53.963000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "959515": "I've created a public kernel [here](https://www.kaggle.com/akensert/glrec-resnet50-arcface-tf2-2?scriptVersionId=40194320). \n\nIn `version 10` I get a score of `0.0481` (training for 12 epochs with under-sampling). In `version 11`, which is identical to `version 10` (unless I accidentally changed something) except for the number of epochs (20), I get a score of `0.0000`. What could be the reason for this? I've thought about it a bit and can't come up with a good reason.\n\nEDIT: The submission times (training/predicting on private train/test set) were also similar between them, which shouldn't be considering that version 11 trained for many more epochs.",
    "960867": "Just copy Baseline and expected 0.461 but obtain 0.4586. Seems there is inconsistency, evaluation changed over time",
    "960877": "@akensert I've checked your kernel, and you haven't set up a seed. Setting seeds helps getting reproducible results, it doesn't guaranty cause there might be libraries which don't make use of the libraries you set seeds for.\nYou should set the following seeds:\n    numpy.random.seed(42)\n    tf.set_random_seed(42)\n\nbtw you don't need to use 42 (i.e the answer to the ultimate question of Life, the Universe, and Everything 😃 ) as seed. you can choose whatever number you'd like"
  }
}