Sirish Somanchi shared in this competition forum that there are 78 melanoma in public test data. He submitted a probe where all predictions are 0 except for one known melanoma case where he submitted 1. He got a public LB of 0.5064, and from that computed the number of melanoma in public test.
\n\nI was puzzled by his finding, and decided to check it.
\n\nTL;DR Sirish Somanchi was almost right: the number of melanoma in public test data is 77 or 78. I find it quite amazing that this can be determined with a single submission. Sirish had a great insight.
\n\nI provide details in this notebook on how I compute the number of melanoma:\nhttps://www.kaggle.com/cpmpml/number-of-public-melanoma-is-78-or-77
\n\nLet's say it is an independent confirmation of Sirish finding.
", "messages": [ { "id": "959625", "postDate": "08/05/2020 18:51:43", "content": "Sirish Somanchi shared in this competition forum that there are 78 melanoma in public test data. He submitted a probe where all predictions are 0 except for one known melanoma case where he submitted 1. He got a public LB of 0.5064, and from that computed the number of melanoma in public test.
\n\nI was puzzled by his finding, and decided to check it.
\n\nTL;DR Sirish Somanchi was almost right: the number of melanoma in public test data is 77 or 78. I find it quite amazing that this can be determined with a single submission. Sirish had a great insight.
\n\nI provide details in this notebook on how I compute the number of melanoma:\nhttps://www.kaggle.com/cpmpml/number-of-public-melanoma-is-78-or-77
\n\nLet's say it is an independent confirmation of Sirish finding.
", "rawMarkdown": "Sirish Somanchi shared in this competition forum that there are 78 melanoma in public test data. He submitted a probe where all predictions are 0 except for one known melanoma case where he submitted 1. He got a public LB of 0.5064, and from that computed the number of melanoma in public test.\n\nI was puzzled by his finding, and decided to check it. \n\nTL;DR Sirish Somanchi was almost right: the number of melanoma in public test data is 77 or 78. I find it quite amazing that this can be determined with a single submission. Sirish had a great insight.\n\nI provide details in this notebook on how I compute the number of melanoma:\nhttps://www.kaggle.com/cpmpml/number-of-public-melanoma-is-78-or-77\n\nLet's say it is an independent confirmation of Sirish finding.", "votes": null }, { "id": "959752", "postDate": "08/05/2020 21:29:39", "content": "Very interesting. I am always surprised by how much information can be extracted from something as simple as a few (or even a single) submissions.
", "rawMarkdown": "Very interesting. I am always surprised by how much information can be extracted from something as simple as a few (or even a single) submissions.", "votes": null }, { "id": "960202", "postDate": "08/06/2020 08:09:36", "content": "Here is my independent confirmation with a slightly different approach (no code) https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#932194
", "rawMarkdown": "Here is my independent confirmation with a slightly different approach (no code) https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#932194", "votes": null }, { "id": "960241", "postDate": "08/06/2020 08:56:48", "content": "I had read it ;) Issue is you don't show that adding one known positive prediction always yield the same auc improvement. It looks true, but an you prove it?
", "rawMarkdown": "I had read it ;) Issue is you don't show that adding one known positive prediction always yield the same auc improvement. It looks true, but an you prove it?", "votes": null }, { "id": "960299", "postDate": "08/06/2020 09:43:27", "content": "If you think of AUC as the mean number of good ranking between all possible pairs between set M of malignant and set B of benign lesions then you see that everything scales independently.
\n\nThe auc = 1/nb_pairs * \\sum_{m\\inM, b\\inB} 1_{p_m > p_b} so you can write this with two sums and separate them auc = 1/nb_pairs * \\sum_{m\\inM} \\sum{b\\inB} 1_{p_m > p_b} , so the switch from one m from 0 to 1 would have the exact same impact on the final score independently of what score you have for other malignant (as long as p_bs stay the same).
Note that in the sum above, in case of p_m = p_b , 1_{p_m > p_b} = 0.5
Would this convince you?
\n\n(sorry for the ugly latex like symbols, don't know how to do better)
", "rawMarkdown": "If you think of AUC as the mean number of good ranking between all possible pairs between set M of malignant and set B of benign lesions then you see that everything scales independently.\n\nThe auc = ` 1/nb_pairs * \\sum_{m\\inM, b\\inB} 1_{p_m > p_b}` so you can write this with two sums and separate them auc = ` 1/nb_pairs * \\sum_{m\\inM} \\sum{b\\inB} 1_{p_m > p_b}` , so the switch from one m from 0 to 1 would have the exact same impact on the final score independently of what score you have for other malignant (as long as p_bs stay the same).\n\nNote that in the sum above, in case of p_m = p_b , `1_{p_m > p_b} = 0.5`\n\nWould this convince you?\n\n(sorry for the ugly latex like symbols, don't know how to do better)", "votes": null }, { "id": "960359", "postDate": "08/06/2020 10:36:44", "content": "ok.
", "rawMarkdown": "ok.", "votes": null }, { "id": "960816", "postDate": "08/06/2020 17:37:54", "content": "Good Work Mate
", "rawMarkdown": "Good Work Mate", "votes": null } ], "comments": [ { "id": 959752, "author_name": "kozodoi", "author_url": "", "post_date": "08/05/2020 21:29:39", "content": "Very interesting. I am always surprised by how much information can be extracted from something as simple as a few (or even a single) submissions.
", "votes": null, "replies": [] }, { "id": 960202, "author_name": "optimo", "author_url": "", "post_date": "08/06/2020 08:09:36", "content": "Here is my independent confirmation with a slightly different approach (no code) https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#932194
", "votes": null, "replies": [ { "id": 960241, "author_name": "cpmpml", "author_url": "", "post_date": "08/06/2020 08:56:48", "content": "I had read it ;) Issue is you don't show that adding one known positive prediction always yield the same auc improvement. It looks true, but an you prove it?
", "votes": null, "replies": [] }, { "id": 960299, "author_name": "optimo", "author_url": "", "post_date": "08/06/2020 09:43:27", "content": "If you think of AUC as the mean number of good ranking between all possible pairs between set M of malignant and set B of benign lesions then you see that everything scales independently.
\n\nThe auc = 1/nb_pairs * \\sum_{m\\inM, b\\inB} 1_{p_m > p_b} so you can write this with two sums and separate them auc = 1/nb_pairs * \\sum_{m\\inM} \\sum{b\\inB} 1_{p_m > p_b} , so the switch from one m from 0 to 1 would have the exact same impact on the final score independently of what score you have for other malignant (as long as p_bs stay the same).
Note that in the sum above, in case of p_m = p_b , 1_{p_m > p_b} = 0.5
Would this convince you?
\n\n(sorry for the ugly latex like symbols, don't know how to do better)
", "votes": null, "replies": [] }, { "id": 960359, "author_name": "cpmpml", "author_url": "", "post_date": "08/06/2020 10:36:44", "content": "ok.
", "votes": null, "replies": [] } ] }, { "id": 960816, "author_name": "raoofnaushad", "author_url": "", "post_date": "08/06/2020 17:37:54", "content": "Good Work Mate
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "959625": "Sirish Somanchi shared in this competition forum that there are 78 melanoma in public test data. He submitted a probe where all predictions are 0 except for one known melanoma case where he submitted 1. He got a public LB of 0.5064, and from that computed the number of melanoma in public test.\n\nI was puzzled by his finding, and decided to check it. \n\nTL;DR Sirish Somanchi was almost right: the number of melanoma in public test data is 77 or 78. I find it quite amazing that this can be determined with a single submission. Sirish had a great insight.\n\nI provide details in this notebook on how I compute the number of melanoma:\nhttps://www.kaggle.com/cpmpml/number-of-public-melanoma-is-78-or-77\n\nLet's say it is an independent confirmation of Sirish finding.", "959752": "Very interesting. I am always surprised by how much information can be extracted from something as simple as a few (or even a single) submissions.", "960202": "Here is my independent confirmation with a slightly different approach (no code) https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#932194", "960241": "I had read it ;) Issue is you don't show that adding one known positive prediction always yield the same auc improvement. It looks true, but an you prove it?", "960299": "If you think of AUC as the mean number of good ranking between all possible pairs between set M of malignant and set B of benign lesions then you see that everything scales independently.\n\nThe auc = ` 1/nb_pairs * \\sum_{m\\inM, b\\inB} 1_{p_m > p_b}` so you can write this with two sums and separate them auc = ` 1/nb_pairs * \\sum_{m\\inM} \\sum{b\\inB} 1_{p_m > p_b}` , so the switch from one m from 0 to 1 would have the exact same impact on the final score independently of what score you have for other malignant (as long as p_bs stay the same).\n\nNote that in the sum above, in case of p_m = p_b , `1_{p_m > p_b} = 0.5`\n\nWould this convince you?\n\n(sorry for the ugly latex like symbols, don't know how to do better)", "960359": "ok.", "960816": "Good Work Mate" }, "source": "meta" }