{
  "id": 230416,
  "title": "Reproducibility of top submissions",
  "url": "/competitions/bms-molecular-translation/discussion/230416",
  "author_name": "",
  "post_date": "2021-04-03T15:09:33.356694400Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Are the top submissions supposed to be reproducible? I mean to be able to get to the exact same submission if starting from zero.<br>\nIf so, are they not supposed to set pytorch's cudnn benchmark to True, or if set, then specifying the exact hardware setup on which the submission was created?</p>",
  "messages": [
    {
      "id": "1261914",
      "postDate": "04/03/2021 15:09:33",
      "content": "<p>Are the top submissions supposed to be reproducible? I mean to be able to get to the exact same submission if starting from zero.<br>\nIf so, are they not supposed to set pytorch's cudnn benchmark to True, or if set, then specifying the exact hardware setup on which the submission was created?</p>",
      "rawMarkdown": "Are the top submissions supposed to be reproducible? I mean to be able to get to the exact same submission if starting from zero.\nIf so, are they not supposed to set pytorch's cudnn benchmark to True, or if set, then specifying the exact hardware setup on which the submission was created?",
      "votes": null
    },
    {
      "id": "1262021",
      "postDate": "04/03/2021 17:26:51",
      "content": "<p>I think its important to just save <code>weights</code> of your  best module. If someone re run your code they might get slightly different results .. but I assume it will be within some reasonable margin and not far away from your best module =) </p>",
      "rawMarkdown": "I think its important to just save `weights` of your  best module. If someone re run your code they might get slightly different results .. but I assume it will be within some reasonable margin and not far away from your best module =)",
      "votes": null
    },
    {
      "id": "1262038",
      "postDate": "04/03/2021 17:48:45",
      "content": "<p>As I can see, you tell this from experience. Congratulations for that! :) And thank you for the clarification! I had not found anything related to it on kaggle.</p>",
      "rawMarkdown": "As I can see, you tell this from experience. Congratulations for that! :) And thank you for the clarification! I had not found anything related to it on kaggle.",
      "votes": null
    },
    {
      "id": "1262048",
      "postDate": "04/03/2021 18:02:52",
      "content": "<p>So I guess that all hyperparameters and training code should be that was used to train the model, but seed is not required to set.</p>\n<p>It is complicated to set in pytorch and also requires not to use the cudnn benchmark which speeds up training a little.</p>",
      "rawMarkdown": "So I guess that all hyperparameters and training code should be that was used to train the model, but seed is not required to set.\n\nIt is complicated to set in pytorch and also requires not to use the cudnn benchmark which speeds up training a little.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1262021,
      "author_name": "drhabib",
      "author_url": "",
      "post_date": "04/03/2021 17:26:51",
      "content": "<p>I think its important to just save <code>weights</code> of your  best module. If someone re run your code they might get slightly different results .. but I assume it will be within some reasonable margin and not far away from your best module =) </p>",
      "votes": null,
      "replies": [
        {
          "id": 1262038,
          "author_name": "nofreewill",
          "author_url": "",
          "post_date": "04/03/2021 17:48:45",
          "content": "<p>As I can see, you tell this from experience. Congratulations for that! :) And thank you for the clarification! I had not found anything related to it on kaggle.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1262048,
          "author_name": "nofreewill",
          "author_url": "",
          "post_date": "04/03/2021 18:02:52",
          "content": "<p>So I guess that all hyperparameters and training code should be that was used to train the model, but seed is not required to set.</p>\n<p>It is complicated to set in pytorch and also requires not to use the cudnn benchmark which speeds up training a little.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1261914": "Are the top submissions supposed to be reproducible? I mean to be able to get to the exact same submission if starting from zero.\nIf so, are they not supposed to set pytorch's cudnn benchmark to True, or if set, then specifying the exact hardware setup on which the submission was created?",
    "1262021": "I think its important to just save `weights` of your  best module. If someone re run your code they might get slightly different results .. but I assume it will be within some reasonable margin and not far away from your best module =)",
    "1262038": "As I can see, you tell this from experience. Congratulations for that! :) And thank you for the clarification! I had not found anything related to it on kaggle.",
    "1262048": "So I guess that all hyperparameters and training code should be that was used to train the model, but seed is not required to set.\n\nIt is complicated to set in pytorch and also requires not to use the cudnn benchmark which speeds up training a little."
  },
  "source": "meta"
}