In case of a submission which strictly uses only image processing, my question is whether using patient_id for the purpose of creating clean training/validation splits such that patients included in the Train set are different from those in Test set, cause the submission to become ineligible for the Special Prize Without Context - $5,000 (Top-scoring model without using any patient-level contextual information)?
", "messages": [ { "id": "905695", "postDate": "06/28/2020 18:11:05", "content": "In case of a submission which strictly uses only image processing, my question is whether using patient_id for the purpose of creating clean training/validation splits such that patients included in the Train set are different from those in Test set, cause the submission to become ineligible for the Special Prize Without Context - $5,000 (Top-scoring model without using any patient-level contextual information)?
", "rawMarkdown": "In case of a submission which strictly uses only image processing, my question is whether using patient_id for the purpose of creating **clean** training/validation splits such that patients included in the Train set are different from those in Test set, cause the submission to **become ineligible** for the Special Prize **Without Context** - $5,000 (Top-scoring model without using any patient-level contextual information)?", "votes": null }, { "id": "912410", "postDate": "07/02/2020 13:13:28", "content": "I have been asking this myself for some weeks - pls Julia, can you try to get an answer? @juliaelliott \nI think this is a valid question.
", "rawMarkdown": "I have been asking this myself for some weeks - pls Julia, can you try to get an answer? @juliaelliott \nI think this is a valid question.", "votes": null }, { "id": "912581", "postDate": "07/02/2020 15:31:00", "content": "Thanks, great question which I’ve gone back to the host to confirm. What I believe you are describing is simply performing your own splits on the training data, using patient_id, to create a cleaner cross validation method? This would constitute a “without context” approach as described in the special prizes, since you aren’t using the patient_id substantively to cluster images for inference. I will update the Prizes page with this clarification to specify that splits isolated to the training data do not constitute a “with context” approach.
", "rawMarkdown": "Thanks, great question which I’ve gone back to the host to confirm. What I believe you are describing is simply performing your own splits on the training data, using patient_id, to create a cleaner cross validation method? **This would constitute a “without context” approach** as described in the special prizes, since you aren’t using the patient_id substantively to cluster images for inference. I will update the Prizes page with this clarification to specify that splits isolated to the training data do not constitute a “with context” approach.", "votes": null }, { "id": "920159", "postDate": "07/08/2020 11:42:17", "content": "@juliaelliott: I'm a little confused.
\nIn your reply, you've stated: This would constitute a “without context” approach as described in the special prizes. But the Prizes page states: If Patient ID is used only for splitting the training data into validation subsets, this is still not considered “without context.”
Could you please clarify which is correct?
", "rawMarkdown": "juliaelliott: I'm a little confused. \nIn your reply, you've stated: This would constitute a “without context” approach as described in the special prizes. But the Prizes [page] (https://www.kaggle.com/c/siim-isic-melanoma-classification/overview/prizes) states: If Patient ID is used only for splitting the training data into validation subsets, this is still ***not*** considered “without context.”\n\nCould you please clarify which is correct?", "votes": null }, { "id": "920175", "postDate": "07/08/2020 12:00:56", "content": "Good catch Ria, it's a typo!
\n\n@juliaelliott - could you please remove the word \"not\" highlighted by Ria above.
", "rawMarkdown": "Good catch Ria, it's a typo!\n\n@juliaelliott - could you please remove the word \"not\" highlighted by Ria above.", "votes": null }, { "id": "920883", "postDate": "07/08/2020 22:01:53", "content": "@sirishks @rai555 Oh my, SO sorry. That was definitely a typo of the worst kind. I've updated it. Thanks for catching it!
", "rawMarkdown": "sirishks @rai555 Oh my, SO sorry. That was definitely a typo of the worst kind. I've updated it. Thanks for catching it!", "votes": null } ], "comments": [ { "id": 912410, "author_name": "romanweilguny", "author_url": "", "post_date": "07/02/2020 13:13:28", "content": "I have been asking this myself for some weeks - pls Julia, can you try to get an answer? @juliaelliott \nI think this is a valid question.
", "votes": null, "replies": [] }, { "id": 912581, "author_name": "juliaelliott", "author_url": "", "post_date": "07/02/2020 15:31:00", "content": "Thanks, great question which I’ve gone back to the host to confirm. What I believe you are describing is simply performing your own splits on the training data, using patient_id, to create a cleaner cross validation method? This would constitute a “without context” approach as described in the special prizes, since you aren’t using the patient_id substantively to cluster images for inference. I will update the Prizes page with this clarification to specify that splits isolated to the training data do not constitute a “with context” approach.
", "votes": null, "replies": [ { "id": 920159, "author_name": "rai555", "author_url": "", "post_date": "07/08/2020 11:42:17", "content": "@juliaelliott: I'm a little confused.
\nIn your reply, you've stated: This would constitute a “without context” approach as described in the special prizes. But the Prizes page states: If Patient ID is used only for splitting the training data into validation subsets, this is still not considered “without context.”
Could you please clarify which is correct?
", "votes": null, "replies": [] }, { "id": 920175, "author_name": "sirishks", "author_url": "", "post_date": "07/08/2020 12:00:56", "content": "Good catch Ria, it's a typo!
\n\n@juliaelliott - could you please remove the word \"not\" highlighted by Ria above.
", "votes": null, "replies": [] }, { "id": 920883, "author_name": "juliaelliott", "author_url": "", "post_date": "07/08/2020 22:01:53", "content": "@sirishks @rai555 Oh my, SO sorry. That was definitely a typo of the worst kind. I've updated it. Thanks for catching it!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "905695": "In case of a submission which strictly uses only image processing, my question is whether using patient_id for the purpose of creating **clean** training/validation splits such that patients included in the Train set are different from those in Test set, cause the submission to **become ineligible** for the Special Prize **Without Context** - $5,000 (Top-scoring model without using any patient-level contextual information)?", "912410": "I have been asking this myself for some weeks - pls Julia, can you try to get an answer? @juliaelliott \nI think this is a valid question.", "912581": "Thanks, great question which I’ve gone back to the host to confirm. What I believe you are describing is simply performing your own splits on the training data, using patient_id, to create a cleaner cross validation method? **This would constitute a “without context” approach** as described in the special prizes, since you aren’t using the patient_id substantively to cluster images for inference. I will update the Prizes page with this clarification to specify that splits isolated to the training data do not constitute a “with context” approach.", "920159": "juliaelliott: I'm a little confused. \nIn your reply, you've stated: This would constitute a “without context” approach as described in the special prizes. But the Prizes [page] (https://www.kaggle.com/c/siim-isic-melanoma-classification/overview/prizes) states: If Patient ID is used only for splitting the training data into validation subsets, this is still ***not*** considered “without context.”\n\nCould you please clarify which is correct?", "920175": "Good catch Ria, it's a typo!\n\n@juliaelliott - could you please remove the word \"not\" highlighted by Ria above.", "920883": "sirishks @rai555 Oh my, SO sorry. That was definitely a typo of the worst kind. I've updated it. Thanks for catching it!" }, "source": "meta" }