I am a machine learning novice. Consequently, I don't understand why most people are using a single model that predicts all 24 species? My inexperienced mind leads me to believe that it would be more accurate to create 24 models that each predict the existence of a single species per sound file.
\nMy best submission using this method is only 0.735, so I'm obviously NOT on to something 😂
", "messages": [ { "id": "1185967", "postDate": "02/04/2021 14:00:19", "content": "I am a machine learning novice. Consequently, I don't understand why most people are using a single model that predicts all 24 species? My inexperienced mind leads me to believe that it would be more accurate to create 24 models that each predict the existence of a single species per sound file.
\nMy best submission using this method is only 0.735, so I'm obviously NOT on to something 😂
", "rawMarkdown": "I am a machine learning novice. Consequently, I don't understand why most people are using a single model that predicts all 24 species? My inexperienced mind leads me to believe that it would be more accurate to create 24 models that each predict the existence of a single species per sound file.\n\nMy best submission using this method is only 0.735, so I'm obviously NOT on to something 😂", "votes": null }, { "id": "1185982", "postDate": "02/04/2021 14:13:16", "content": "Probably you have imbalanced
", "rawMarkdown": "Probably you have imbalanced", "votes": null }, { "id": "1186022", "postDate": "02/04/2021 14:43:06", "content": "I also tried that and had worse results this way.
", "rawMarkdown": "I also tried that and had worse results this way.", "votes": null }, { "id": "1186028", "postDate": "02/04/2021 14:48:23", "content": "Try to evaluate distribution of predictions of the models. One possible problem is that probabilities are not calibrated with each other as here LWRAP is used for evaluation.
", "rawMarkdown": "Try to evaluate distribution of predictions of the models. One possible problem is that probabilities are not calibrated with each other as here LWRAP is used for evaluation.", "votes": null }, { "id": "1186085", "postDate": "02/04/2021 15:21:40", "content": "To my mind, the main theoretical reasons would be
\nThe second point suggests (and in my experience it's often the case) that neural networks benefit (rather than suffer) from having multiple tasks to do. There's quite a few nice examples of this out there, one of my favorites came up in the twiml podcast episode, where they observed that predicting diabetic retinopathy from retinal photographs got better by also predicting things like blood pressure, blood sugar, age, and male/female with the same model.
\nOf course, in practice you want to try it and see what works best. Another example where a single neural network (vs. multiple models) turned out to be clearly the best choice was the Mechanisms of Action (MoA) Prediction Challenge.
", "rawMarkdown": "To my mind, the main theoretical reasons would be \n1. the presence of one species makes the presence of another species more or less likely with a one-vs-all model you are not getting the signal on the other species as clearly, and\n2. for a neural network additional supervision (i.e. other tasks like identifying other noises) does help with creating a good internal representation (kind of closely related to the first point). \n\nThe second point suggests (and in my experience it's often the case) that neural networks benefit (rather than suffer) from having multiple tasks to do. There's quite a few nice examples of this out there, one of my favorites came up in the [twiml podcast episode](https://twimlai.com/twiml-talk-122-predicting-cardiovascular-risk-factors-eye-images-ryan-poplin/), where they observed that predicting diabetic retinopathy from retinal photographs got better by also predicting things like blood pressure, blood sugar, age, and male/female with the same model.\n\nOf course, in practice you want to try it and see what works best. Another example where a single neural network (vs. multiple models) turned out to be clearly the best choice was the [Mechanisms of Action (MoA) Prediction Challenge](https://www.kaggle.com/c/lish-moa).", "votes": null }, { "id": "1186854", "postDate": "02/05/2021 04:55:36", "content": "I agree with @bjoernholzhauer
\nIn one sentence \"birds of the same feather flock together\"…
\nMind you, I believe one can get a decent score with single model approach also, I dont deny that… but the top 3 winners would definitely be those who allowed their model to consider the entire eco system (the entire cacaphony of voices in the audio across species).. For e.g. s3,s12,s18 are almost always there with s0.
", "rawMarkdown": "I agree with @bjoernholzhauer \n\nIn one sentence \"birds of the same feather flock together\"...\n\nMind you, I believe one can get a decent score with single model approach also, I dont deny that... but the top 3 winners would definitely be those who allowed their model to consider the entire eco system (the entire cacaphony of voices in the audio across species).. For e.g. s3,s12,s18 are almost always there with s0.", "votes": null }, { "id": "1186920", "postDate": "02/05/2021 05:50:11", "content": "\"OFF TOPIC\"
\nHey @adityakumarsinha , I was just going through your profile (checking Indian kagglers ). I found something unusual that in competition section you have 2 gold medals but when I went through you competitions list I can only find one competition where you won gold. Does Kaggle have some error ??
![https://drive.google.com/file/d/1nYXe-goc5KN5jHZpI80SNt1rGgOYHGUG/view?usp=sharing]
\nNot able to attach image here. Kaggle have some issue.
", "rawMarkdown": "\"**OFF TOPIC**\"\nHey @adityakumarsinha , I was just going through your profile (checking Indian kagglers ). I found something unusual that in competition section you have 2 gold medals but when I went through you competitions list I can only find one competition where you won gold. Does Kaggle have some error ??\n\n![https://drive.google.com/file/d/1nYXe-goc5KN5jHZpI80SNt1rGgOYHGUG/view?usp=sharing]\n\nNot able to attach image here. Kaggle have some issue.", "votes": null }, { "id": "1186945", "postDate": "02/05/2021 06:00:29", "content": "Kaggle sometimes organizes competition master only competition. can you see this competition: https://www.kaggle.com/c/masters-caesars-customer-gaming-prediction
", "rawMarkdown": "Kaggle sometimes organizes competition master only competition. can you see this competition: https://www.kaggle.com/c/masters-caesars-customer-gaming-prediction", "votes": null }, { "id": "1186949", "postDate": "02/05/2021 06:03:02", "content": "No, I'm not able to see that.
\nWell thank you got to know something new.
\nSo are there any grandmaster's only competitions ??
Yes, there are few competitions restricted by Kaggle to masters and grandmasters only. More details can be found in progression page below.
\nhttps://www.kaggle.com/progression
\n\"Masters in the Competitions category are eligible for exclusive Master-Only competitions.\"
", "rawMarkdown": "Yes, there are few competitions restricted by Kaggle to masters and grandmasters only. More details can be found in progression page below.\n\nhttps://www.kaggle.com/progression\n\n\"*Masters in the Competitions category are eligible for exclusive Master-Only competitions.*\"", "votes": null }, { "id": "1190763", "postDate": "02/08/2021 02:02:33", "content": "From my understanding, I have the same idea as you have and I still believe that this is A VERY GOOD APPROACH. In fact, my opinion is to predict the number of occurrences of each 60s clip for each bird, given the we crop each 60s to the same level, say to 60 chunks, 1s each. And the prediction result would just be within the range num. belongs to [0,60], where num. is an integer. That totally solved the problem mentioned in many others reply. The question now becomes the accuracy of each single model and the metrics used(designed) by the competition host. I tried this experiment before, however, the result is not good due to the not-so-good accuracy of each model: some may get 0.95+ acc but some may just got 0.6 acc or so…
", "rawMarkdown": "From my understanding, I have the same idea as you have and I still believe that this is A VERY GOOD APPROACH. In fact, my opinion is to predict the number of occurrences of each 60s clip for each bird, given the we crop each 60s to the same level, say to 60 chunks, 1s each. And the prediction result would just be within the range num. belongs to [0,60], where num. is an integer. That totally solved the problem mentioned in many others reply. The question now becomes the accuracy of each single model and the metrics used(designed) by the competition host. I tried this experiment before, however, the result is not good due to the not-so-good accuracy of each model: some may get 0.95+ acc but some may just got 0.6 acc or so...", "votes": null } ], "comments": [ { "id": 1185982, "author_name": "gopidurgaprasad", "author_url": "", "post_date": "02/04/2021 14:13:16", "content": "Probably you have imbalanced
", "votes": null, "replies": [] }, { "id": 1186022, "author_name": "nofreewill", "author_url": "", "post_date": "02/04/2021 14:43:06", "content": "I also tried that and had worse results this way.
", "votes": null, "replies": [] }, { "id": 1186028, "author_name": "adityakumarsinha", "author_url": "", "post_date": "02/04/2021 14:48:23", "content": "Try to evaluate distribution of predictions of the models. One possible problem is that probabilities are not calibrated with each other as here LWRAP is used for evaluation.
", "votes": null, "replies": [ { "id": 1186920, "author_name": "rsinda", "author_url": "", "post_date": "02/05/2021 05:50:11", "content": "\"OFF TOPIC\"
\nHey @adityakumarsinha , I was just going through your profile (checking Indian kagglers ). I found something unusual that in competition section you have 2 gold medals but when I went through you competitions list I can only find one competition where you won gold. Does Kaggle have some error ??
![https://drive.google.com/file/d/1nYXe-goc5KN5jHZpI80SNt1rGgOYHGUG/view?usp=sharing]
\nNot able to attach image here. Kaggle have some issue.
", "votes": null, "replies": [] }, { "id": 1186945, "author_name": "adityakumarsinha", "author_url": "", "post_date": "02/05/2021 06:00:29", "content": "Kaggle sometimes organizes competition master only competition. can you see this competition: https://www.kaggle.com/c/masters-caesars-customer-gaming-prediction
", "votes": null, "replies": [] }, { "id": 1186949, "author_name": "rsinda", "author_url": "", "post_date": "02/05/2021 06:03:02", "content": "No, I'm not able to see that.
\nWell thank you got to know something new.
\nSo are there any grandmaster's only competitions ??
Yes, there are few competitions restricted by Kaggle to masters and grandmasters only. More details can be found in progression page below.
\nhttps://www.kaggle.com/progression
\n\"Masters in the Competitions category are eligible for exclusive Master-Only competitions.\"
", "votes": null, "replies": [] } ] }, { "id": 1186085, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "02/04/2021 15:21:40", "content": "To my mind, the main theoretical reasons would be
\nThe second point suggests (and in my experience it's often the case) that neural networks benefit (rather than suffer) from having multiple tasks to do. There's quite a few nice examples of this out there, one of my favorites came up in the twiml podcast episode, where they observed that predicting diabetic retinopathy from retinal photographs got better by also predicting things like blood pressure, blood sugar, age, and male/female with the same model.
\nOf course, in practice you want to try it and see what works best. Another example where a single neural network (vs. multiple models) turned out to be clearly the best choice was the Mechanisms of Action (MoA) Prediction Challenge.
", "votes": null, "replies": [ { "id": 1186854, "author_name": "allohvk", "author_url": "", "post_date": "02/05/2021 04:55:36", "content": "I agree with @bjoernholzhauer
\nIn one sentence \"birds of the same feather flock together\"…
\nMind you, I believe one can get a decent score with single model approach also, I dont deny that… but the top 3 winners would definitely be those who allowed their model to consider the entire eco system (the entire cacaphony of voices in the audio across species).. For e.g. s3,s12,s18 are almost always there with s0.
", "votes": null, "replies": [] } ] }, { "id": 1190763, "author_name": "wubinbai", "author_url": "", "post_date": "02/08/2021 02:02:33", "content": "From my understanding, I have the same idea as you have and I still believe that this is A VERY GOOD APPROACH. In fact, my opinion is to predict the number of occurrences of each 60s clip for each bird, given the we crop each 60s to the same level, say to 60 chunks, 1s each. And the prediction result would just be within the range num. belongs to [0,60], where num. is an integer. That totally solved the problem mentioned in many others reply. The question now becomes the accuracy of each single model and the metrics used(designed) by the competition host. I tried this experiment before, however, the result is not good due to the not-so-good accuracy of each model: some may get 0.95+ acc but some may just got 0.6 acc or so…
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1185967": "I am a machine learning novice. Consequently, I don't understand why most people are using a single model that predicts all 24 species? My inexperienced mind leads me to believe that it would be more accurate to create 24 models that each predict the existence of a single species per sound file.\n\nMy best submission using this method is only 0.735, so I'm obviously NOT on to something 😂", "1185982": "Probably you have imbalanced", "1186022": "I also tried that and had worse results this way.", "1186028": "Try to evaluate distribution of predictions of the models. One possible problem is that probabilities are not calibrated with each other as here LWRAP is used for evaluation.", "1186085": "To my mind, the main theoretical reasons would be \n1. the presence of one species makes the presence of another species more or less likely with a one-vs-all model you are not getting the signal on the other species as clearly, and\n2. for a neural network additional supervision (i.e. other tasks like identifying other noises) does help with creating a good internal representation (kind of closely related to the first point). \n\nThe second point suggests (and in my experience it's often the case) that neural networks benefit (rather than suffer) from having multiple tasks to do. There's quite a few nice examples of this out there, one of my favorites came up in the [twiml podcast episode](https://twimlai.com/twiml-talk-122-predicting-cardiovascular-risk-factors-eye-images-ryan-poplin/), where they observed that predicting diabetic retinopathy from retinal photographs got better by also predicting things like blood pressure, blood sugar, age, and male/female with the same model.\n\nOf course, in practice you want to try it and see what works best. Another example where a single neural network (vs. multiple models) turned out to be clearly the best choice was the [Mechanisms of Action (MoA) Prediction Challenge](https://www.kaggle.com/c/lish-moa).", "1186854": "I agree with @bjoernholzhauer \n\nIn one sentence \"birds of the same feather flock together\"...\n\nMind you, I believe one can get a decent score with single model approach also, I dont deny that... but the top 3 winners would definitely be those who allowed their model to consider the entire eco system (the entire cacaphony of voices in the audio across species).. For e.g. s3,s12,s18 are almost always there with s0.", "1186920": "\"**OFF TOPIC**\"\nHey @adityakumarsinha , I was just going through your profile (checking Indian kagglers ). I found something unusual that in competition section you have 2 gold medals but when I went through you competitions list I can only find one competition where you won gold. Does Kaggle have some error ??\n\n![https://drive.google.com/file/d/1nYXe-goc5KN5jHZpI80SNt1rGgOYHGUG/view?usp=sharing]\n\nNot able to attach image here. Kaggle have some issue.", "1186945": "Kaggle sometimes organizes competition master only competition. can you see this competition: https://www.kaggle.com/c/masters-caesars-customer-gaming-prediction", "1186949": "No, I'm not able to see that.\n\nWell thank you got to know something new.\nSo are there any grandmaster's only competitions ??", "1190467": "Yes, there are few competitions restricted by Kaggle to masters and grandmasters only. More details can be found in progression page below.\n\nhttps://www.kaggle.com/progression\n\n\"*Masters in the Competitions category are eligible for exclusive Master-Only competitions.*\"", "1190763": "From my understanding, I have the same idea as you have and I still believe that this is A VERY GOOD APPROACH. In fact, my opinion is to predict the number of occurrences of each 60s clip for each bird, given the we crop each 60s to the same level, say to 60 chunks, 1s each. And the prediction result would just be within the range num. belongs to [0,60], where num. is an integer. That totally solved the problem mentioned in many others reply. The question now becomes the accuracy of each single model and the metrics used(designed) by the competition host. I tried this experiment before, however, the result is not good due to the not-so-good accuracy of each model: some may get 0.95+ acc but some may just got 0.6 acc or so..." }, "source": "meta" }