{
  "id": 406255,
  "title": "Same model different score",
  "url": "/competitions/asl-signs/discussion/406255",
  "author_name": "",
  "post_date": "2023-05-01T17:06:04.886796800Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Our best model scored 0.74 7 days ago, and we tried to improve the best model this week.  However, the exact same model scored 0.72 3 times in a row.  At first, I thought the inconsistency was because of the random choice of scoring dataset, but I don't think it can explain 0.02 difference in score.  <br>\nHas anyone faced the same problem?  Or Do you have any ideas why this happened?<br>\nDid scoring algorithm or dataset change during the final week of the competition?</p>",
  "messages": [
    {
      "id": "2241638",
      "postDate": "05/01/2023 17:06:04",
      "content": "<p>Our best model scored 0.74 7 days ago, and we tried to improve the best model this week.  However, the exact same model scored 0.72 3 times in a row.  At first, I thought the inconsistency was because of the random choice of scoring dataset, but I don't think it can explain 0.02 difference in score.  <br>\nHas anyone faced the same problem?  Or Do you have any ideas why this happened?<br>\nDid scoring algorithm or dataset change during the final week of the competition?</p>",
      "rawMarkdown": "Our best model scored 0.74 7 days ago, and we tried to improve the best model this week.  However, the exact same model scored 0.72 3 times in a row.  At first, I thought the inconsistency was because of the random choice of scoring dataset, but I don't think it can explain 0.02 difference in score.  \nHas anyone faced the same problem?  Or Do you have any ideas why this happened?\nDid scoring algorithm or dataset change during the final week of the competition?",
      "votes": null
    },
    {
      "id": "2241645",
      "postDate": "05/01/2023 17:18:16",
      "content": "<p>We haven't updated the metric or dataset.</p>",
      "rawMarkdown": "We haven't updated the metric or dataset.",
      "votes": null
    },
    {
      "id": "2241650",
      "postDate": "05/01/2023 17:30:33",
      "content": "<p>May be some randomness in preprocessing  or TTA with random probability 🤔</p>",
      "rawMarkdown": "May be some randomness in preprocessing  or TTA with random probability 🤔",
      "votes": null
    },
    {
      "id": "2241989",
      "postDate": "05/02/2023 01:43:06",
      "content": "<p>Thank you for your reply!  That means the cause is on my notebook…</p>",
      "rawMarkdown": "Thank you for your reply!  That means the cause is on my notebook...",
      "votes": null
    },
    {
      "id": "2241990",
      "postDate": "05/02/2023 01:44:40",
      "content": "<p>I set the random seed for numpy, tensorflow etc. , so I couldn't find where we got the randomness…</p>",
      "rawMarkdown": "I set the random seed for numpy, tensorflow etc. , so I couldn't find where we got the randomness...",
      "votes": null
    },
    {
      "id": "2242009",
      "postDate": "05/02/2023 02:02:19",
      "content": "<p>When you commit the notebook, does it train a new model? Or have you saved the model weights to ensure it is the same model each time?</p>",
      "rawMarkdown": "When you commit the notebook, does it train a new model? Or have you saved the model weights to ensure it is the same model each time?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2241645,
      "author_name": "sohier",
      "author_url": "",
      "post_date": "05/01/2023 17:18:16",
      "content": "<p>We haven't updated the metric or dataset.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2241989,
          "author_name": "lilkoke",
          "author_url": "",
          "post_date": "05/02/2023 01:43:06",
          "content": "<p>Thank you for your reply!  That means the cause is on my notebook…</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2241650,
      "author_name": "rashmibanthia",
      "author_url": "",
      "post_date": "05/01/2023 17:30:33",
      "content": "<p>May be some randomness in preprocessing  or TTA with random probability 🤔</p>",
      "votes": null,
      "replies": [
        {
          "id": 2241990,
          "author_name": "lilkoke",
          "author_url": "",
          "post_date": "05/02/2023 01:44:40",
          "content": "<p>I set the random seed for numpy, tensorflow etc. , so I couldn't find where we got the randomness…</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2242009,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "05/02/2023 02:02:19",
      "content": "<p>When you commit the notebook, does it train a new model? Or have you saved the model weights to ensure it is the same model each time?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2241638": "Our best model scored 0.74 7 days ago, and we tried to improve the best model this week.  However, the exact same model scored 0.72 3 times in a row.  At first, I thought the inconsistency was because of the random choice of scoring dataset, but I don't think it can explain 0.02 difference in score.  \nHas anyone faced the same problem?  Or Do you have any ideas why this happened?\nDid scoring algorithm or dataset change during the final week of the competition?",
    "2241645": "We haven't updated the metric or dataset.",
    "2241650": "May be some randomness in preprocessing  or TTA with random probability 🤔",
    "2241989": "Thank you for your reply!  That means the cause is on my notebook...",
    "2241990": "I set the random seed for numpy, tensorflow etc. , so I couldn't find where we got the randomness...",
    "2242009": "When you commit the notebook, does it train a new model? Or have you saved the model weights to ensure it is the same model each time?"
  },
  "source": "meta"
}