{ "id": 175379, "title": "still interested to know : 0.99 on public LB, how is it done?", "url": "/competitions/siim-isic-melanoma-classification/discussion/175379", "author_name": "", "post_date": "2020-08-18T03:50:46.798487300Z", "votes": 14, "comment_count": 8, "views": 0, "content": "

i saw a couple of teams have 0.99 on public LB before the competition ends. I wonder how it is done?

\n

(even if it is overfitting, i love to know about the details)
\nThanks!

", "messages": [ { "id": "974783", "postDate": "08/18/2020 03:50:46", "content": "

i saw a couple of teams have 0.99 on public LB before the competition ends. I wonder how it is done?

\n

(even if it is overfitting, i love to know about the details)
\nThanks!

", "rawMarkdown": "i saw a couple of teams have 0.99 on public LB before the competition ends. I wonder how it is done?\n\n(even if it is overfitting, i love to know about the details)\nThanks!", "votes": null }, { "id": "974805", "postDate": "08/18/2020 04:00:22", "content": "

I think you are looking for this.

", "rawMarkdown": "I think you are looking for [this](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/174515#970111).", "votes": null }, { "id": "974846", "postDate": "08/18/2020 04:27:23", "content": "

there are some general discussion in the link. but i hope to see how details. e.g. actual submission csv, scores or number of probes, etc

", "rawMarkdown": "there are some general discussion in the link. but i hope to see how details. e.g. actual submission csv, scores or number of probes, etc", "votes": null }, { "id": "974878", "postDate": "08/18/2020 04:39:04", "content": "

I not sure but I think you can download data from ISIC archive and overfit on it, I think it might contained some test samples. But I don't have any evidence it just a hypothesis.

", "rawMarkdown": "I not sure but I think you can download data from [ISIC archive ](https://www.isic-archive.com/#!/topWithHeader/onlyHeaderTop/gallery) and overfit on it, I think it might contained some test samples. But I don't have any evidence it just a hypothesis.", "votes": null }, { "id": "975020", "postDate": "08/18/2020 05:47:58", "content": "

Same curious here! It's amazing.

", "rawMarkdown": "Same curious here! It's amazing.", "votes": null }, { "id": "975064", "postDate": "08/18/2020 06:10:23", "content": "

It is not fresh as far as I know. Please see the original post and kaggle kernel with related discussion. The difference of applying in this comp could be probing top N records (for example top 3000) rather than probing all 10982. Meanwhile, I also find a recent work (paper) done by Google using sequential Monte carlo samplers for COVID-19 group testing that has the potiental to further optimize the method.

", "rawMarkdown": "It is not fresh as far as I know. Please see the original [post ](http://blog.mrtz.org/2015/03/09/competition.html) and kaggle [kernel](https://www.kaggle.com/olegtrott/the-perfect-score-script) with related [discussion](https://www.kaggle.com/c/restaurant-revenue-prediction/discussion/13950). The difference of applying in this comp could be probing top N records (for example top 3000) rather than probing all 10982. Meanwhile, I also find a recent [work ](https://github.com/google-research/google-research/tree/master/grouptesting) ([paper](https://arxiv.org/abs/2004.12508)) done by Google using sequential Monte carlo samplers for COVID-19 group testing that has the potiental to further optimize the method.", "votes": null }, { "id": "975240", "postDate": "08/18/2020 08:01:52", "content": "

@zaharch do you plan to make a write up?
\n@sirishks ?

\n

I would be interested in three things personally:

\n", "rawMarkdown": "zaharch do you plan to make a write up?\n@sirishks ?\n\nI would be interested in three things personally:\n- what was your approach for optimal LB probing?\n- why would you do that? especially @sirishks who did this very early in the competition. Having all this light being at the top while everyone knows you achieved an impossible score... all this in a medical research competition...\n- what did you learn from the experiments and what did you do with labels you got?", "votes": null }, { "id": "975433", "postDate": "08/18/2020 09:47:19", "content": "

I am also waiting for @zaharch writeup, especially the Gurobi MIP part.

\n

When I started working on this competition, I kept getting bad results and was frustrated because of 2 reasons:

\n
    \n
  1. It was impossible to visually identify whether a lesion is a MM or not, especially the Amelanotic kind.
    2. \n
  2. Many images looked like MMs and I wanted to figure out the real count i.e. distribution of test data.
    3. \n
\n

So, I kept making more and more submissions. As others have pointed out, probing Public LB is not very helpful and wastes precious submissions. I only did this experiment to check the accuracy of my model, whether the model is learning what is useful or not.

\n

In this competition I figured out how to climb Public LB, may be next one I will try to stay in a higher rank and avoid a shakeup 😪

\n

Regarding the actual technique, I have already posted the secret recipe in multiple posts - see:

\n", "rawMarkdown": "I am also waiting for [@zaharch](https://www.kaggle.com/zaharch) writeup, especially the [Gurobi MIP part](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/174515#970111).\n\nWhen I started working on this competition, I kept getting bad results and was frustrated because of 2 reasons:\n1. It was impossible to **visually** identify whether a lesion is a MM or not, especially the Amelanotic kind.\n2. Many images looked like MMs and I wanted to figure out the **real count** i.e. distribution of test data.\n\nSo, I kept making more and more submissions. As others have pointed out, probing Public LB is not very helpful and wastes precious submissions. I only did this experiment to check the accuracy of my model, whether the model is learning what is useful or not.\n\nIn this competition I figured out how to climb **Public** LB, may be next one I will try to **stay** in a higher rank and avoid a shakeup 😪\n\nRegarding the actual technique, I have already posted the **secret recipe** in multiple posts - see:\n- [ignoring bottom rows](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161497#901185),\n- [estimating number of MMs in test data](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#930723)\n - ([Optimo confirmed the calculations](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#932194)),\n- [checking duplicates](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171930#956437)\n - ([Chris confirmed the math calculations](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171930#956491)),\n- [power averaging concepts](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165653)", "votes": null }, { "id": "976007", "postDate": "08/18/2020 15:22:26", "content": "

Hi @hengck23, @optimo , @sirishks , I published my story in a different post, please take a look:

\n

https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175559

", "rawMarkdown": "Hi @hengck23, @optimo , @sirishks , I published my story in a different post, please take a look:\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175559", "votes": null } ], "comments": [ { "id": 974805, "author_name": "wuliaokaola", "author_url": "", "post_date": "08/18/2020 04:00:22", "content": "

I think you are looking for this.

", "votes": null, "replies": [ { "id": 974846, "author_name": "hengck23", "author_url": "", "post_date": "08/18/2020 04:27:23", "content": "

there are some general discussion in the link. but i hope to see how details. e.g. actual submission csv, scores or number of probes, etc

", "votes": null, "replies": [] } ] }, { "id": 974878, "author_name": "ivanvoid", "author_url": "", "post_date": "08/18/2020 04:39:04", "content": "

I not sure but I think you can download data from ISIC archive and overfit on it, I think it might contained some test samples. But I don't have any evidence it just a hypothesis.

", "votes": null, "replies": [] }, { "id": 975020, "author_name": "yuanlin08", "author_url": "", "post_date": "08/18/2020 05:47:58", "content": "

Same curious here! It's amazing.

", "votes": null, "replies": [] }, { "id": 975064, "author_name": "dxchen", "author_url": "", "post_date": "08/18/2020 06:10:23", "content": "

It is not fresh as far as I know. Please see the original post and kaggle kernel with related discussion. The difference of applying in this comp could be probing top N records (for example top 3000) rather than probing all 10982. Meanwhile, I also find a recent work (paper) done by Google using sequential Monte carlo samplers for COVID-19 group testing that has the potiental to further optimize the method.

", "votes": null, "replies": [] }, { "id": 975240, "author_name": "optimo", "author_url": "", "post_date": "08/18/2020 08:01:52", "content": "

@zaharch do you plan to make a write up?
\n@sirishks ?

\n

I would be interested in three things personally:

\n", "votes": null, "replies": [] }, { "id": 975433, "author_name": "sirishks", "author_url": "", "post_date": "08/18/2020 09:47:19", "content": "

I am also waiting for @zaharch writeup, especially the Gurobi MIP part.

\n

When I started working on this competition, I kept getting bad results and was frustrated because of 2 reasons:

\n
    \n
  1. It was impossible to visually identify whether a lesion is a MM or not, especially the Amelanotic kind.
    2. \n
  2. Many images looked like MMs and I wanted to figure out the real count i.e. distribution of test data.
    3. \n
\n

So, I kept making more and more submissions. As others have pointed out, probing Public LB is not very helpful and wastes precious submissions. I only did this experiment to check the accuracy of my model, whether the model is learning what is useful or not.

\n

In this competition I figured out how to climb Public LB, may be next one I will try to stay in a higher rank and avoid a shakeup 😪

\n

Regarding the actual technique, I have already posted the secret recipe in multiple posts - see:

\n", "votes": null, "replies": [] }, { "id": 976007, "author_name": "zaharch", "author_url": "", "post_date": "08/18/2020 15:22:26", "content": "

Hi @hengck23, @optimo , @sirishks , I published my story in a different post, please take a look:

\n

https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175559

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "974783": "i saw a couple of teams have 0.99 on public LB before the competition ends. I wonder how it is done?\n\n(even if it is overfitting, i love to know about the details)\nThanks!", "974805": "I think you are looking for [this](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/174515#970111).", "974846": "there are some general discussion in the link. but i hope to see how details. e.g. actual submission csv, scores or number of probes, etc", "974878": "I not sure but I think you can download data from [ISIC archive ](https://www.isic-archive.com/#!/topWithHeader/onlyHeaderTop/gallery) and overfit on it, I think it might contained some test samples. But I don't have any evidence it just a hypothesis.", "975020": "Same curious here! It's amazing.", "975064": "It is not fresh as far as I know. Please see the original [post ](http://blog.mrtz.org/2015/03/09/competition.html) and kaggle [kernel](https://www.kaggle.com/olegtrott/the-perfect-score-script) with related [discussion](https://www.kaggle.com/c/restaurant-revenue-prediction/discussion/13950). The difference of applying in this comp could be probing top N records (for example top 3000) rather than probing all 10982. Meanwhile, I also find a recent [work ](https://github.com/google-research/google-research/tree/master/grouptesting) ([paper](https://arxiv.org/abs/2004.12508)) done by Google using sequential Monte carlo samplers for COVID-19 group testing that has the potiental to further optimize the method.", "975240": "zaharch do you plan to make a write up?\n@sirishks ?\n\nI would be interested in three things personally:\n- what was your approach for optimal LB probing?\n- why would you do that? especially @sirishks who did this very early in the competition. Having all this light being at the top while everyone knows you achieved an impossible score... all this in a medical research competition...\n- what did you learn from the experiments and what did you do with labels you got?", "975433": "I am also waiting for [@zaharch](https://www.kaggle.com/zaharch) writeup, especially the [Gurobi MIP part](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/174515#970111).\n\nWhen I started working on this competition, I kept getting bad results and was frustrated because of 2 reasons:\n1. It was impossible to **visually** identify whether a lesion is a MM or not, especially the Amelanotic kind.\n2. Many images looked like MMs and I wanted to figure out the **real count** i.e. distribution of test data.\n\nSo, I kept making more and more submissions. As others have pointed out, probing Public LB is not very helpful and wastes precious submissions. I only did this experiment to check the accuracy of my model, whether the model is learning what is useful or not.\n\nIn this competition I figured out how to climb **Public** LB, may be next one I will try to **stay** in a higher rank and avoid a shakeup 😪\n\nRegarding the actual technique, I have already posted the **secret recipe** in multiple posts - see:\n- [ignoring bottom rows](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161497#901185),\n- [estimating number of MMs in test data](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#930723)\n - ([Optimo confirmed the calculations](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215#932194)),\n- [checking duplicates](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171930#956437)\n - ([Chris confirmed the math calculations](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171930#956491)),\n- [power averaging concepts](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165653)", "976007": "Hi @hengck23, @optimo , @sirishks , I published my story in a different post, please take a look:\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175559" }, "source": "meta" }