{
  "id": 443506,
  "title": "Are we expecting shake up/down ? ",
  "url": "/competitions/bengaliai-speech/discussion/443506",
  "author_name": "",
  "post_date": "2023-09-27T15:11:21.995367Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Just wondering how badly the shake of the Leaderboard will be.<br>\nOn one hand, the public score is on 46%. That's quite a lot.<br>\nThe scariest part is the statement \"There may be domains in the private test set that are not in the public test set \". <br>\nWe have no way of knowing how our model perform on this \"hidden domain\", we don't even know which one they are out of the 17. The OOD samples are too small (one per category) for feedback.<br>\nDo share your opinion (and perhaps some guide how to handle this secret OOD) </p>",
  "messages": [
    {
      "id": "2458405",
      "postDate": "09/27/2023 15:11:21",
      "content": "<p>Just wondering how badly the shake of the Leaderboard will be.<br>\nOn one hand, the public score is on 46%. That's quite a lot.<br>\nThe scariest part is the statement \"There may be domains in the private test set that are not in the public test set \". <br>\nWe have no way of knowing how our model perform on this \"hidden domain\", we don't even know which one they are out of the 17. The OOD samples are too small (one per category) for feedback.<br>\nDo share your opinion (and perhaps some guide how to handle this secret OOD) </p>",
      "rawMarkdown": "Just wondering how badly the shake of the Leaderboard will be.\nOn one hand, the public score is on 46%. That's quite a lot.\nThe scariest part is the statement \"There may be domains in the private test set that are not in the public test set \". \nWe have no way of knowing how our model perform on this \"hidden domain\", we don't even know which one they are out of the 17. The OOD samples are too small (one per category) for feedback.\nDo share your opinion (and perhaps some guide how to handle this secret OOD)",
      "votes": null
    },
    {
      "id": "2462426",
      "postDate": "09/30/2023 10:42:52",
      "content": "<p>I have the same concern. I wonder if public LB uses the same percentage of OOD test dataset as the private one. Can the administrator disclose about how much of public LB is consisted of OOD and how much Macro Test?</p>",
      "rawMarkdown": "I have the same concern. I wonder if public LB uses the same percentage of OOD test dataset as the private one. Can the administrator disclose about how much of public LB is consisted of OOD and how much Macro Test?",
      "votes": null
    },
    {
      "id": "2462719",
      "postDate": "09/30/2023 14:44:46",
      "content": "<p>It will not use the same percentage. It says in the dataset description that there will be domain \"not\" present in public test. I think domain here means those OOD category<br>\nWow, you manage to improve a lot ! Are you already ensembling or still adding training data ?  It's ok if you don't want to tell</p>",
      "rawMarkdown": "It will not use the same percentage. It says in the dataset description that there will be domain \"not\" present in public test. I think domain here means those OOD category\nWow, you manage to improve a lot ! Are you already ensembling or still adding training data ?  It's ok if you don't want to tell",
      "votes": null
    },
    {
      "id": "2462737",
      "postDate": "09/30/2023 15:02:40",
      "content": "<p>unfortunately, my teammate hasn't got models fit for ensemble. So, probably not lots of improvement can be made anymore from now on. </p>\n<p>TBH, I just got lucky by finding a very useful open source work, and that boosts up my score by a lot. Also, adding data is indeed what I have done, literally blindly finetuning everything right now. ZERO tactic…😂</p>",
      "rawMarkdown": "unfortunately, my teammate hasn't got models fit for ensemble. So, probably not lots of improvement can be made anymore from now on. \n\nTBH, I just got lucky by finding a very useful open source work, and that boosts up my score by a lot. Also, adding data is indeed what I have done, literally blindly finetuning everything right now. ZERO tactic...😂",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2462426,
      "author_name": "renyiwei",
      "author_url": "",
      "post_date": "09/30/2023 10:42:52",
      "content": "<p>I have the same concern. I wonder if public LB uses the same percentage of OOD test dataset as the private one. Can the administrator disclose about how much of public LB is consisted of OOD and how much Macro Test?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2462719,
          "author_name": "nyleve",
          "author_url": "",
          "post_date": "09/30/2023 14:44:46",
          "content": "<p>It will not use the same percentage. It says in the dataset description that there will be domain \"not\" present in public test. I think domain here means those OOD category<br>\nWow, you manage to improve a lot ! Are you already ensembling or still adding training data ?  It's ok if you don't want to tell</p>",
          "votes": null,
          "replies": [
            {
              "id": 2462737,
              "author_name": "renyiwei",
              "author_url": "",
              "post_date": "09/30/2023 15:02:40",
              "content": "<p>unfortunately, my teammate hasn't got models fit for ensemble. So, probably not lots of improvement can be made anymore from now on. </p>\n<p>TBH, I just got lucky by finding a very useful open source work, and that boosts up my score by a lot. Also, adding data is indeed what I have done, literally blindly finetuning everything right now. ZERO tactic…😂</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2458405": "Just wondering how badly the shake of the Leaderboard will be.\nOn one hand, the public score is on 46%. That's quite a lot.\nThe scariest part is the statement \"There may be domains in the private test set that are not in the public test set \". \nWe have no way of knowing how our model perform on this \"hidden domain\", we don't even know which one they are out of the 17. The OOD samples are too small (one per category) for feedback.\nDo share your opinion (and perhaps some guide how to handle this secret OOD)",
    "2462426": "I have the same concern. I wonder if public LB uses the same percentage of OOD test dataset as the private one. Can the administrator disclose about how much of public LB is consisted of OOD and how much Macro Test?",
    "2462719": "It will not use the same percentage. It says in the dataset description that there will be domain \"not\" present in public test. I think domain here means those OOD category\nWow, you manage to improve a lot ! Are you already ensembling or still adding training data ?  It's ok if you don't want to tell",
    "2462737": "unfortunately, my teammate hasn't got models fit for ensemble. So, probably not lots of improvement can be made anymore from now on. \n\nTBH, I just got lucky by finding a very useful open source work, and that boosts up my score by a lot. Also, adding data is indeed what I have done, literally blindly finetuning everything right now. ZERO tactic...😂"
  },
  "source": "meta"
}