{ "id": 280296, "title": "Expected AUC for \"N\" submissions for a Random Classifier ", "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/280296", "author_name": "", "post_date": "2021-10-20T23:33:47.479768100Z", "votes": 1, "comment_count": 4, "views": 0, "content": "
    \n
  1. In the past, I had made this discussion thread to discuss random classifier (link)
    2. \n
  2. there are some notebook links in it.
    3. \n
  3. I received great feedback from @qitvision on it.
    4. \n
  4. To summarize that discussion thread, there are two main points:
    5. \n
  5. Score after 1 random submission versus score after 10,20, N submissions: for example:
    6. \n
  6. There is \"1/100 chance of random classifier getting 0.65\" for one random submission;
    7. \n
  7. for 10 random submissions (link)
    \n(10 random subs) percentile-50 : 0.5941798941798941
    \n(10 random subs) percentile-90 : 0.6423280423280423
    \n(10 random subs) percentile-95 : 0.6587301587301587
    \n(10 random subs) percentile-99 : 0.6862433862433863
    \n(10 random subs) percentile-99.9 : 0.7285714285714286
    \n(10 random subs) percentile-99.99 : 0.764021164021164
    8. \n
  8. for 20 random submissions:(link)
    \n(20 random subs) percentile-50 : 0.6132275132275132
    \n(20 random subs) percentile-90 : 0.6566137566137566
    \n(20 random subs) percentile-95 : 0.6714285714285714
    \n(20 random subs) percentile-99 : 0.7005291005291006
    \n(20 random subs) percentile-99.9 : 0.7391534391534391
    \n(20 random subs) percentile-99.99 : 0.764021164021164
    9. \n
", "messages": [ { "id": "1551875", "postDate": "10/20/2021 23:33:47", "content": "
    \n
  1. In the past, I had made this discussion thread to discuss random classifier (link)
    2. \n
  2. there are some notebook links in it.
    3. \n
  3. I received great feedback from @qitvision on it.
    4. \n
  4. To summarize that discussion thread, there are two main points:
    5. \n
  5. Score after 1 random submission versus score after 10,20, N submissions: for example:
    6. \n
  6. There is \"1/100 chance of random classifier getting 0.65\" for one random submission;
    7. \n
  7. for 10 random submissions (link)
    \n(10 random subs) percentile-50 : 0.5941798941798941
    \n(10 random subs) percentile-90 : 0.6423280423280423
    \n(10 random subs) percentile-95 : 0.6587301587301587
    \n(10 random subs) percentile-99 : 0.6862433862433863
    \n(10 random subs) percentile-99.9 : 0.7285714285714286
    \n(10 random subs) percentile-99.99 : 0.764021164021164
    8. \n
  8. for 20 random submissions:(link)
    \n(20 random subs) percentile-50 : 0.6132275132275132
    \n(20 random subs) percentile-90 : 0.6566137566137566
    \n(20 random subs) percentile-95 : 0.6714285714285714
    \n(20 random subs) percentile-99 : 0.7005291005291006
    \n(20 random subs) percentile-99.9 : 0.7391534391534391
    \n(20 random subs) percentile-99.99 : 0.764021164021164
    9. \n
", "rawMarkdown": "1. In the past, I had made this discussion thread to discuss random classifier ([link]( https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/273655))\n2. there are some notebook links in it.\n3. I received great feedback from @qitvision on it. \n4. To summarize that discussion thread, there are two main points: \n5. Score after 1 random submission versus score after 10,20, N submissions: for example: \n5. There is \"1/100 chance of random classifier getting 0.65\" for one random submission; \n6. for 10 random submissions ([link]( https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/273655))\n(10 random subs) percentile-50 : 0.5941798941798941\n(10 random subs) percentile-90 : 0.6423280423280423\n(10 random subs) percentile-95 : 0.6587301587301587\n(10 random subs) percentile-99 : 0.6862433862433863\n(10 random subs) percentile-99.9 : 0.7285714285714286\n(10 random subs) percentile-99.99 : 0.764021164021164\n7. for 20 random submissions:([link]( https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/273655))\n(20 random subs) percentile-50 : 0.6132275132275132\n(20 random subs) percentile-90 : 0.6566137566137566\n(20 random subs) percentile-95 : 0.6714285714285714\n(20 random subs) percentile-99 : 0.7005291005291006\n(20 random subs) percentile-99.9 : 0.7391534391534391\n(20 random subs) percentile-99.99 : 0.764021164021164", "votes": null }, { "id": "1551880", "postDate": "10/20/2021 23:39:57", "content": "

In this competition 1,555 teams made 27,466 submissions.
\nI was wondering if anyone had thought about a random classifier with 27,466 submissions ?
\nHow should one account for 1,555 teams ?
\nI am curious how would one simulate 27,466 random submission from 1,555 teams ?
\n(I will check with some statisticians and reply back here if I get some insightful answers)

", "rawMarkdown": "In this competition 1,555 teams made 27,466 submissions. \nI was wondering if anyone had thought about a random classifier with 27,466 submissions ? \nHow should one account for 1,555 teams ? \nI am curious how would one simulate 27,466 random submission from 1,555 teams ? \n(I will check with some statisticians and reply back here if I get some insightful answers)", "votes": null }, { "id": "1552167", "postDate": "10/21/2021 07:49:57", "content": "

There is \"1/100 chance of random classifier getting 0.65\" for one random submission; for public or private score?

", "rawMarkdown": "`There is \"1/100 chance of random classifier getting 0.65\" for one random submission;` for public or private score?", "votes": null }, { "id": "1552174", "postDate": "10/21/2021 08:02:12", "content": "

Good point . I believe that was for the public test set; will try and recompute it for the private test set and report back here.

", "rawMarkdown": "Good point . I believe that was for the public test set; will try and recompute it for the private test set and report back here.", "votes": null }, { "id": "1552479", "postDate": "10/21/2021 13:16:52", "content": "

Yes, that was for the public test set. @anlthms created this private score simulation notebook that shows that even if predictions of all teams were shuffled, the top-10 scores would remain in the same range.

\n

I was also curious about simulating private scores so I modified the previous LB simulation notebook to simulate private score (sorry, I beat you to it @mpsampat).

\n

Note that this used the training set and not the actual private set distribution of MGMT values so the results might vary. It shows that one is likely to get over AUC 0.51 with two all-random submits.

\n

If everyone would predict random values, 1 out of 100 teams would get AUC 0.57 and 1 out of 1000 AUC 0.60.

\n
(2 random subs) percentile-50 : 0.5172779028200715\n(2 random subs) percentile-90 : 0.5502482457301735\n(2 random subs) percentile-95 : 0.5608367536078379\n(2 random subs) percentile-99 : 0.5782470541506686\n(2 random subs) percentile-99.9 : 0.6034704753078249\n(2 random subs) percentile-99.99 : 0.625480759300934\n
", "rawMarkdown": "Yes, that was for the public test set. @anlthms created this [private score simulation notebook](https://www.kaggle.com/anlthms/leaderboard-simulation) that shows that even if predictions of all teams were shuffled, the top-10 scores would remain in the same range.\n\nI was also curious about simulating private scores so I modified the previous LB simulation notebook to [simulate private score](https://www.kaggle.com/qitvision/private-lb-simulation) (sorry, I beat you to it @mpsampat).\n\nNote that this used the training set and not the actual private set distribution of MGMT values so the results might vary. It shows that one is likely to get over `AUC 0.51` with two all-random submits.\n\nIf everyone would predict random values, 1 out of 100 teams would get `AUC 0.57` and 1 out of 1000 `AUC 0.60`.\n\n```\n(2 random subs) percentile-50 : 0.5172779028200715\n(2 random subs) percentile-90 : 0.5502482457301735\n(2 random subs) percentile-95 : 0.5608367536078379\n(2 random subs) percentile-99 : 0.5782470541506686\n(2 random subs) percentile-99.9 : 0.6034704753078249\n(2 random subs) percentile-99.99 : 0.625480759300934\n```", "votes": null } ], "comments": [ { "id": 1551880, "author_name": "mpsampat", "author_url": "", "post_date": "10/20/2021 23:39:57", "content": "

In this competition 1,555 teams made 27,466 submissions.
\nI was wondering if anyone had thought about a random classifier with 27,466 submissions ?
\nHow should one account for 1,555 teams ?
\nI am curious how would one simulate 27,466 random submission from 1,555 teams ?
\n(I will check with some statisticians and reply back here if I get some insightful answers)

", "votes": null, "replies": [] }, { "id": 1552167, "author_name": "mrinath", "author_url": "", "post_date": "10/21/2021 07:49:57", "content": "

There is \"1/100 chance of random classifier getting 0.65\" for one random submission; for public or private score?

", "votes": null, "replies": [ { "id": 1552174, "author_name": "mpsampat", "author_url": "", "post_date": "10/21/2021 08:02:12", "content": "

Good point . I believe that was for the public test set; will try and recompute it for the private test set and report back here.

", "votes": null, "replies": [] }, { "id": 1552479, "author_name": "qitvision", "author_url": "", "post_date": "10/21/2021 13:16:52", "content": "

Yes, that was for the public test set. @anlthms created this private score simulation notebook that shows that even if predictions of all teams were shuffled, the top-10 scores would remain in the same range.

\n

I was also curious about simulating private scores so I modified the previous LB simulation notebook to simulate private score (sorry, I beat you to it @mpsampat).

\n

Note that this used the training set and not the actual private set distribution of MGMT values so the results might vary. It shows that one is likely to get over AUC 0.51 with two all-random submits.

\n

If everyone would predict random values, 1 out of 100 teams would get AUC 0.57 and 1 out of 1000 AUC 0.60.

\n
(2 random subs) percentile-50 : 0.5172779028200715\n(2 random subs) percentile-90 : 0.5502482457301735\n(2 random subs) percentile-95 : 0.5608367536078379\n(2 random subs) percentile-99 : 0.5782470541506686\n(2 random subs) percentile-99.9 : 0.6034704753078249\n(2 random subs) percentile-99.99 : 0.625480759300934\n
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1551875": "1. In the past, I had made this discussion thread to discuss random classifier ([link]( https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/273655))\n2. there are some notebook links in it.\n3. I received great feedback from @qitvision on it. \n4. To summarize that discussion thread, there are two main points: \n5. Score after 1 random submission versus score after 10,20, N submissions: for example: \n5. There is \"1/100 chance of random classifier getting 0.65\" for one random submission; \n6. for 10 random submissions ([link]( https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/273655))\n(10 random subs) percentile-50 : 0.5941798941798941\n(10 random subs) percentile-90 : 0.6423280423280423\n(10 random subs) percentile-95 : 0.6587301587301587\n(10 random subs) percentile-99 : 0.6862433862433863\n(10 random subs) percentile-99.9 : 0.7285714285714286\n(10 random subs) percentile-99.99 : 0.764021164021164\n7. for 20 random submissions:([link]( https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/273655))\n(20 random subs) percentile-50 : 0.6132275132275132\n(20 random subs) percentile-90 : 0.6566137566137566\n(20 random subs) percentile-95 : 0.6714285714285714\n(20 random subs) percentile-99 : 0.7005291005291006\n(20 random subs) percentile-99.9 : 0.7391534391534391\n(20 random subs) percentile-99.99 : 0.764021164021164", "1551880": "In this competition 1,555 teams made 27,466 submissions. \nI was wondering if anyone had thought about a random classifier with 27,466 submissions ? \nHow should one account for 1,555 teams ? \nI am curious how would one simulate 27,466 random submission from 1,555 teams ? \n(I will check with some statisticians and reply back here if I get some insightful answers)", "1552167": "`There is \"1/100 chance of random classifier getting 0.65\" for one random submission;` for public or private score?", "1552174": "Good point . I believe that was for the public test set; will try and recompute it for the private test set and report back here.", "1552479": "Yes, that was for the public test set. @anlthms created this [private score simulation notebook](https://www.kaggle.com/anlthms/leaderboard-simulation) that shows that even if predictions of all teams were shuffled, the top-10 scores would remain in the same range.\n\nI was also curious about simulating private scores so I modified the previous LB simulation notebook to [simulate private score](https://www.kaggle.com/qitvision/private-lb-simulation) (sorry, I beat you to it @mpsampat).\n\nNote that this used the training set and not the actual private set distribution of MGMT values so the results might vary. It shows that one is likely to get over `AUC 0.51` with two all-random submits.\n\nIf everyone would predict random values, 1 out of 100 teams would get `AUC 0.57` and 1 out of 1000 `AUC 0.60`.\n\n```\n(2 random subs) percentile-50 : 0.5172779028200715\n(2 random subs) percentile-90 : 0.5502482457301735\n(2 random subs) percentile-95 : 0.5608367536078379\n(2 random subs) percentile-99 : 0.5782470541506686\n(2 random subs) percentile-99.9 : 0.6034704753078249\n(2 random subs) percentile-99.99 : 0.625480759300934\n```" }, "source": "meta" }