{
  "id": 176962,
  "title": "Sometimes extremely low score after long time submission metric calculating",
  "url": "/competitions/birdsong-recognition/discussion/176962",
  "author_name": "",
  "post_date": "2020-08-24T09:01:21.398085300Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello guys~ I'm pretty new to this competiton and I've met a strange and very troublesome problem. I wonder if anyone has met this problem before and know the cause&amp;solution to this:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3756245%2Fd3abfd622a0e7adcc7523e7aa1e05c03%2Fds.png?generation=1598258830643473&amp;alt=media\" alt=\"\"><br>\nThe problem is that the last two submissions' score are extremely low after a pretty long submisson score calculating time that lasts for minutes. But what I've changed comparing to the earliest 0.544 notebook is just the threshold for bird predicting as I found the predicted labels are too many. After the 0.18+ score came out, I directly forked the 0.544 version notebook and changed threshold to 0.58 to see if I accidently put a typo in the later version. However, the score is still extremely low after a long time execution.</p>\n<p>I've met this problem once before in this competition but the second day it came back to normal after I copied a fresh new inference notebook. I doubt that this is caused by simply changing the threshold. Anyone has an idea for the reason of this problem?</p>\n<p>UPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&amp;commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem.</p>",
  "messages": [
    {
      "id": "983384",
      "postDate": "08/24/2020 09:01:21",
      "content": "<p>Hello guys~ I'm pretty new to this competiton and I've met a strange and very troublesome problem. I wonder if anyone has met this problem before and know the cause&amp;solution to this:<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3756245%2Fd3abfd622a0e7adcc7523e7aa1e05c03%2Fds.png?generation=1598258830643473&amp;alt=media\" alt=\"\"><br>\nThe problem is that the last two submissions' score are extremely low after a pretty long submisson score calculating time that lasts for minutes. But what I've changed comparing to the earliest 0.544 notebook is just the threshold for bird predicting as I found the predicted labels are too many. After the 0.18+ score came out, I directly forked the 0.544 version notebook and changed threshold to 0.58 to see if I accidently put a typo in the later version. However, the score is still extremely low after a long time execution.</p>\n<p>I've met this problem once before in this competition but the second day it came back to normal after I copied a fresh new inference notebook. I doubt that this is caused by simply changing the threshold. Anyone has an idea for the reason of this problem?</p>\n<p>UPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&amp;commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem.</p>",
      "rawMarkdown": "Hello guys~ I'm pretty new to this competiton and I've met a strange and very troublesome problem. I wonder if anyone has met this problem before and know the cause&solution to this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3756245%2Fd3abfd622a0e7adcc7523e7aa1e05c03%2Fds.png?generation=1598258830643473&alt=media)\nThe problem is that the last two submissions' score are extremely low after a pretty long submisson score calculating time that lasts for minutes. But what I've changed comparing to the earliest 0.544 notebook is just the threshold for bird predicting as I found the predicted labels are too many. After the 0.18+ score came out, I directly forked the 0.544 version notebook and changed threshold to 0.58 to see if I accidently put a typo in the later version. However, the score is still extremely low after a long time execution.\n\nI've met this problem once before in this competition but the second day it came back to normal after I copied a fresh new inference notebook. I doubt that this is caused by simply changing the threshold. Anyone has an idea for the reason of this problem?\n\nUPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem.",
      "votes": null
    },
    {
      "id": "983516",
      "postDate": "08/24/2020 11:24:48",
      "content": "<p>I can not find the original notebook, that you use. Can you provide a link?</p>",
      "rawMarkdown": "I can not find the original notebook, that you use. Can you provide a link?",
      "votes": null
    },
    {
      "id": "983559",
      "postDate": "08/24/2020 12:04:52",
      "content": "<p>Thanks for your reply. The original link is this public notebook:<a href=\"https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection\" target=\"_blank\">https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection</a> I barely change it as baseline with more epochs:</p>\n<p>I think this is caused by one pretrained weights I trained because this problem only occurs after I use it for prediction. The strange part is that this problem won't occur when I first use the weights but after that it will continue to show up. Maybe this has sth to do with kaggle's save&amp;commit settings that I don't know about.</p>\n<p>Thanks again. I will update this topic after I make submission using the baseline trained weights instead of my weights. And if it's normal, then I will just stop using that particular weights.</p>",
      "rawMarkdown": "Thanks for your reply. The original link is this public notebook:https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection I barely change it as baseline with more epochs:\n\nI think this is caused by one pretrained weights I trained because this problem only occurs after I use it for prediction. The strange part is that this problem won't occur when I first use the weights but after that it will continue to show up. Maybe this has sth to do with kaggle's save&commit settings that I don't know about.\n\nThanks again. I will update this topic after I make submission using the baseline trained weights instead of my weights. And if it's normal, then I will just stop using that particular weights.",
      "votes": null
    },
    {
      "id": "984316",
      "postDate": "08/25/2020 03:20:51",
      "content": "<p>UPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&amp;commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem.</p>",
      "rawMarkdown": "UPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 983516,
      "author_name": "vladimirsydor",
      "author_url": "",
      "post_date": "08/24/2020 11:24:48",
      "content": "<p>I can not find the original notebook, that you use. Can you provide a link?</p>",
      "votes": null,
      "replies": [
        {
          "id": 983559,
          "author_name": "hzhaobang",
          "author_url": "",
          "post_date": "08/24/2020 12:04:52",
          "content": "<p>Thanks for your reply. The original link is this public notebook:<a href=\"https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection\" target=\"_blank\">https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection</a> I barely change it as baseline with more epochs:</p>\n<p>I think this is caused by one pretrained weights I trained because this problem only occurs after I use it for prediction. The strange part is that this problem won't occur when I first use the weights but after that it will continue to show up. Maybe this has sth to do with kaggle's save&amp;commit settings that I don't know about.</p>\n<p>Thanks again. I will update this topic after I make submission using the baseline trained weights instead of my weights. And if it's normal, then I will just stop using that particular weights.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 984316,
      "author_name": "hzhaobang",
      "author_url": "",
      "post_date": "08/25/2020 03:20:51",
      "content": "<p>UPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&amp;commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "983384": "Hello guys~ I'm pretty new to this competiton and I've met a strange and very troublesome problem. I wonder if anyone has met this problem before and know the cause&solution to this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3756245%2Fd3abfd622a0e7adcc7523e7aa1e05c03%2Fds.png?generation=1598258830643473&alt=media)\nThe problem is that the last two submissions' score are extremely low after a pretty long submisson score calculating time that lasts for minutes. But what I've changed comparing to the earliest 0.544 notebook is just the threshold for bird predicting as I found the predicted labels are too many. After the 0.18+ score came out, I directly forked the 0.544 version notebook and changed threshold to 0.58 to see if I accidently put a typo in the later version. However, the score is still extremely low after a long time execution.\n\nI've met this problem once before in this competition but the second day it came back to normal after I copied a fresh new inference notebook. I doubt that this is caused by simply changing the threshold. Anyone has an idea for the reason of this problem?\n\nUPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem.",
    "983516": "I can not find the original notebook, that you use. Can you provide a link?",
    "983559": "Thanks for your reply. The original link is this public notebook:https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection I barely change it as baseline with more epochs:\n\nI think this is caused by one pretrained weights I trained because this problem only occurs after I use it for prediction. The strange part is that this problem won't occur when I first use the weights but after that it will continue to show up. Maybe this has sth to do with kaggle's save&commit settings that I don't know about.\n\nThanks again. I will update this topic after I make submission using the baseline trained weights instead of my weights. And if it's normal, then I will just stop using that particular weights.",
    "984316": "UPDATE: The problem exists after using baseline trained weights. However I've got a sense of the reason. The low score and long time scoring are both caused by too many labels. Inspect the competition evaluation metric and 0.5+ public notebook submissions and you can see fewer labels perform much better. I validate this idea by improve the threshold and my scores gradually improves. So the problem comes to why those many-labels notebook submmissions can score 0.5+ at first time. My guess is that kaggle's save&commit somehow doesn't give the csv file of your notebook but the orginal public notebook as result for scoring. But I'm still confused about the cause of this problem."
  },
  "source": "meta"
}