![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F682c77e9abd75f56c6ec60c6ec3bfffd%2FSelection_028.png?generation=1597497433484980&alt=media\")
I think badcase analysis is very important in ML. But in this competition, I have tried to see some samples that model classifies wrong and want to get some inspiration, I can not even distinguish those birds.
\nSo how do you guys do case study in here?
I think badcase analysis is very important in ML. But in this competition, I have tried to see some samples that model classifies wrong and want to get some inspiration, I can not even distinguish those birds.
\nSo how do you guys do case study in here?
you can make figures like the below:
\nyou can study the local validation results or the sample test soundscape files or BirdCLEF2020 files
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fdbd09bbb2e6aadd76904dc7cb15d0d21%2FSelection_032.png?generation=1597497450806997&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ffbf767f7654e793fe504ea58b4e9c670%2FSelection_030.png?generation=1597497556763483&alt=media\")
Thanks a lot, the figures look great! Could you share some codes to generate it?
", "rawMarkdown": "Thanks a lot, the figures look great! Could you share some codes to generate it?", "votes": null }, { "id": "971682", "postDate": "08/15/2020 19:55:30", "content": "Great pictures! Is it correct : red line is the cases, when the prediction for current ground-truth bird is over some threshhold? What are you gonna do, if the other bird will be predicted? Thanks!
", "rawMarkdown": "Great pictures! Is it correct : red line is the cases, when the prediction for current ground-truth bird is over some threshhold? What are you gonna do, if the other bird will be predicted? Thanks!", "votes": null }, { "id": "971686", "postDate": "08/15/2020 20:00:26", "content": "I think, you can just apply methods like the ones in any other competition, not only with pictures/audio files. We can create table with three rows:
\nAnd then explore cases when a true label doesn't equal a predicted label. For instance, you can find some birds hard to predict, or inapproprite threshhold for some cases.
", "rawMarkdown": "I think, you can just apply methods like the ones in any other competition, not only with pictures/audio files. We can create table with three rows:\n- true label\n- predicted label\n- predicted probobility\n\nAnd then explore cases when a true label doesn't equal a predicted label. For instance, you can find some birds hard to predict, or inapproprite threshhold for some cases.", "votes": null }, { "id": "972088", "postDate": "08/16/2020 08:19:17", "content": "Thanks, I have already done this, but I dont know how to do next. Like how I can impove my model to handle those wrong predictions through this table.
\nIn common competition, I can find the connection between true label and wrong prediction, but these bird data are really a mess for me.
there are only two main issues in this competition (and the solutions could be \"simple\")
\n1.weak labels
\n2.adaption to noisy environment
\nin short, if you have strong labels and noisy train samples, the challenge will be straight forward.
\nduring \"badcase study\", identify the \"strong label\" needed and create them. Also, identify the \"noisy train sample\" needed and create them. measure performance before and after \"these creation\" and see if there is improvement
", "rawMarkdown": "there are only two main issues in this competition (and the solutions could be \"simple\")\n\n---\n\n1.weak labels \n- only clip label is given (or if you assume temporal label = clip label, then you have a noisy label problem)\n- you should estimate how your current prediction accuracy of temporal label.\n- the only way to improve temporal accuracy is simply strong label.\n- think of way to create these strong label automatically (heuristics, ml methods like MIL, weak-supervision, etc ... or manual labels)\n\n---\n\n2.adaption to noisy environment\n- you should make a performance graph of your model : accuracy vs signal-noise-ratio (snr)\n- to make a robust model against snr, the solution is to create noisy train samples.\n\nin short, if you have strong labels and noisy train samples, the challenge will be straight forward.\n\n---\n\nduring \"badcase study\", identify the \"strong label\" needed and create them. Also, identify the \"noisy train sample\" needed and create them. measure performance before and after \"these creation\" and see if there is improvement", "votes": null }, { "id": "972534", "postDate": "08/16/2020 16:07:30", "content": "the model is rained on sigmoid loss, i.e. multi-label. red is the probability of each label.
", "rawMarkdown": "the model is rained on sigmoid loss, i.e. multi-label. red is the probability of each label.", "votes": null } ], "comments": [ { "id": 971375, "author_name": "hengck23", "author_url": "", "post_date": "08/15/2020 13:17:32", "content": "you can make figures like the below:
\nyou can study the local validation results or the sample test soundscape files or BirdCLEF2020 files
\n
Thanks a lot, the figures look great! Could you share some codes to generate it?
", "votes": null, "replies": [] }, { "id": 971682, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "08/15/2020 19:55:30", "content": "Great pictures! Is it correct : red line is the cases, when the prediction for current ground-truth bird is over some threshhold? What are you gonna do, if the other bird will be predicted? Thanks!
", "votes": null, "replies": [] }, { "id": 972534, "author_name": "hengck23", "author_url": "", "post_date": "08/16/2020 16:07:30", "content": "the model is rained on sigmoid loss, i.e. multi-label. red is the probability of each label.
", "votes": null, "replies": [] } ] }, { "id": 971686, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "08/15/2020 20:00:26", "content": "I think, you can just apply methods like the ones in any other competition, not only with pictures/audio files. We can create table with three rows:
\nAnd then explore cases when a true label doesn't equal a predicted label. For instance, you can find some birds hard to predict, or inapproprite threshhold for some cases.
", "votes": null, "replies": [ { "id": 972088, "author_name": "karlyukang", "author_url": "", "post_date": "08/16/2020 08:19:17", "content": "Thanks, I have already done this, but I dont know how to do next. Like how I can impove my model to handle those wrong predictions through this table.
\nIn common competition, I can find the connection between true label and wrong prediction, but these bird data are really a mess for me.
there are only two main issues in this competition (and the solutions could be \"simple\")
\n1.weak labels
\n2.adaption to noisy environment
\nin short, if you have strong labels and noisy train samples, the challenge will be straight forward.
\nduring \"badcase study\", identify the \"strong label\" needed and create them. Also, identify the \"noisy train sample\" needed and create them. measure performance before and after \"these creation\" and see if there is improvement
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "971171": "I think badcase analysis is very important in ML. But in this competition, I have tried to see some samples that model classifies wrong and want to get some inspiration, I can not even distinguish those birds. \nSo how do you guys do case study in here?", "971375": "you can make figures like the below:\n- red : frame-wise prediction\n- yellow : ground truth\n\nyou can study the local validation results or the sample test soundscape files or BirdCLEF2020 files\n\n\n\n\n\n\n\n", "971591": "Thanks a lot, the figures look great! Could you share some codes to generate it?", "971682": "Great pictures! Is it correct : red line is the cases, when the prediction for current ground-truth bird is over some threshhold? What are you gonna do, if the other bird will be predicted? Thanks!", "971686": "I think, you can just apply methods like the ones in any other competition, not only with pictures/audio files. We can create table with three rows:\n- true label\n- predicted label\n- predicted probobility\n\nAnd then explore cases when a true label doesn't equal a predicted label. For instance, you can find some birds hard to predict, or inapproprite threshhold for some cases.", "972088": "Thanks, I have already done this, but I dont know how to do next. Like how I can impove my model to handle those wrong predictions through this table.\nIn common competition, I can find the connection between true label and wrong prediction, but these bird data are really a mess for me.", "972122": "there are only two main issues in this competition (and the solutions could be \"simple\")\n\n---\n\n1.weak labels \n- only clip label is given (or if you assume temporal label = clip label, then you have a noisy label problem)\n- you should estimate how your current prediction accuracy of temporal label.\n- the only way to improve temporal accuracy is simply strong label.\n- think of way to create these strong label automatically (heuristics, ml methods like MIL, weak-supervision, etc ... or manual labels)\n\n---\n\n2.adaption to noisy environment\n- you should make a performance graph of your model : accuracy vs signal-noise-ratio (snr)\n- to make a robust model against snr, the solution is to create noisy train samples.\n\nin short, if you have strong labels and noisy train samples, the challenge will be straight forward.\n\n---\n\nduring \"badcase study\", identify the \"strong label\" needed and create them. Also, identify the \"noisy train sample\" needed and create them. measure performance before and after \"these creation\" and see if there is improvement", "972534": "the model is rained on sigmoid loss, i.e. multi-label. red is the probability of each label." }, "source": "meta" }