I could'nt find how public/private sets divided, then how many patients are there in public/private sets?\nAnyway, we know that only 3 data points per patients are used for scoring, so there will be one of the biggest shake in history I think.
", "messages": [ { "id": "955773", "postDate": "08/02/2020 23:50:25", "content": "I could'nt find how public/private sets divided, then how many patients are there in public/private sets?\nAnyway, we know that only 3 data points per patients are used for scoring, so there will be one of the biggest shake in history I think.
", "rawMarkdown": "I could'nt find how public/private sets divided, then how many patients are there in public/private sets?\nAnyway, we know that only 3 data points per patients are used for scoring, so there will be one of the biggest shake in history I think.", "votes": null }, { "id": "956156", "postDate": "08/03/2020 09:50:47", "content": "Oh I found it. The host said that there are around 200 in public+private.\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165723
\n\nThen public is around 30 patients * 3 weeks = 90 points and private is around 170 patients * 3 weeks = 510 points?\nLB seems too shaky so we need concrete CV.
", "rawMarkdown": "Oh I found it. The host said that there are around 200 in public+private.\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165723\n\nThen public is around 30 patients * 3 weeks = 90 points and private is around 170 patients * 3 weeks = 510 points?\nLB seems too shaky so we need concrete CV.", "votes": null }, { "id": "960158", "postDate": "08/06/2020 07:19:54", "content": "But there is only 5 patients in the test set - isn't it 15 cases for public?
", "rawMarkdown": "But there is only 5 patients in the test set - isn't it 15 cases for public?", "votes": null }, { "id": "960424", "postDate": "08/06/2020 11:47:19", "content": "Public and private sets are entirely hidden from us and when we finish a commit with test set the script will be rerun with public and private sets.\nThe test set is not used for scoring, but for sanity check.
", "rawMarkdown": "Public and private sets are entirely hidden from us and when we finish a commit with test set the script will be rerun with public and private sets.\nThe test set is not used for scoring, but for sanity check.", "votes": null }, { "id": "961041", "postDate": "08/06/2020 21:40:06", "content": "Thanks for the explanation; I didn’t realise this.
", "rawMarkdown": "Thanks for the explanation; I didn’t realise this.", "votes": null }, { "id": "961284", "postDate": "08/07/2020 04:41:11", "content": "I'm expecting very few patients in public test set compared to private test set because every model is oversensitive right now. A single parameter change can even return ±0.1 change in score.
", "rawMarkdown": "I'm expecting very few patients in public test set compared to private test set because every model is oversensitive right now. A single parameter change can even return ±0.1 change in score.", "votes": null }, { "id": "961303", "postDate": "08/07/2020 04:58:59", "content": "Yeah this would make sense. I'm trying to create a reliable cross validation for this competition so that my LB score is close to what I think it will be but that may be difficult.
", "rawMarkdown": "Yeah this would make sense. I'm trying to create a reliable cross validation for this competition so that my LB score is close to what I think it will be but that may be difficult.", "votes": null }, { "id": "961624", "postDate": "08/07/2020 10:54:48", "content": "±0.1 change? That's big. Maybe different seeds will yield similar result.
\nGiven that the top score is -6.7599 and the best notebook score is -6.8322, it seems that no one has achieved meaningful result yet.
I think we can detect the size of public set by LB probing, can't we?
", "rawMarkdown": "±0.1 change? That's big. Maybe different seeds will yield similar result.\nGiven that the top score is -6.7599 and the best notebook score is -6.8322, it seems that no one has achieved meaningful result yet.\n\nI think we can detect the size of public set by LB probing, can't we?", "votes": null }, { "id": "961640", "postDate": "08/07/2020 11:05:36", "content": "You can basically create a generator object with df_test.groupby('Patient') and predict all data points of 1 patient, then do that for 2 patients and go on like this. This will consume too many submissions and there must be a more efficient way to do this.
I thought this is enough.
", "rawMarkdown": "1. predict all data points with (10000, 50)\n2. predict all data points with (10000, 100)\n3. predict 1 data points with (10000, 50) and others with (10000, 100)\n\nI thought this is enough.", "votes": null } ], "comments": [ { "id": 961284, "author_name": "gunesevitan", "author_url": "", "post_date": "08/07/2020 04:41:11", "content": "I'm expecting very few patients in public test set compared to private test set because every model is oversensitive right now. A single parameter change can even return ±0.1 change in score.
", "votes": null, "replies": [ { "id": 961303, "author_name": "lgreig", "author_url": "", "post_date": "08/07/2020 04:58:59", "content": "Yeah this would make sense. I'm trying to create a reliable cross validation for this competition so that my LB score is close to what I think it will be but that may be difficult.
", "votes": null, "replies": [] }, { "id": 961624, "author_name": "resistance0108", "author_url": "", "post_date": "08/07/2020 10:54:48", "content": "±0.1 change? That's big. Maybe different seeds will yield similar result.
\nGiven that the top score is -6.7599 and the best notebook score is -6.8322, it seems that no one has achieved meaningful result yet.
I think we can detect the size of public set by LB probing, can't we?
", "votes": null, "replies": [] }, { "id": 961640, "author_name": "gunesevitan", "author_url": "", "post_date": "08/07/2020 11:05:36", "content": "You can basically create a generator object with df_test.groupby('Patient') and predict all data points of 1 patient, then do that for 2 patients and go on like this. This will consume too many submissions and there must be a more efficient way to do this.
I thought this is enough.
", "votes": null, "replies": [] } ] }, { "id": 956156, "author_name": "resistance0108", "author_url": "", "post_date": "08/03/2020 09:50:47", "content": "Oh I found it. The host said that there are around 200 in public+private.\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165723
\n\nThen public is around 30 patients * 3 weeks = 90 points and private is around 170 patients * 3 weeks = 510 points?\nLB seems too shaky so we need concrete CV.
", "votes": null, "replies": [ { "id": 960158, "author_name": "lgreig", "author_url": "", "post_date": "08/06/2020 07:19:54", "content": "But there is only 5 patients in the test set - isn't it 15 cases for public?
", "votes": null, "replies": [] }, { "id": 960424, "author_name": "resistance0108", "author_url": "", "post_date": "08/06/2020 11:47:19", "content": "Public and private sets are entirely hidden from us and when we finish a commit with test set the script will be rerun with public and private sets.\nThe test set is not used for scoring, but for sanity check.
", "votes": null, "replies": [] }, { "id": 961041, "author_name": "lgreig", "author_url": "", "post_date": "08/06/2020 21:40:06", "content": "Thanks for the explanation; I didn’t realise this.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "955773": "I could'nt find how public/private sets divided, then how many patients are there in public/private sets?\nAnyway, we know that only 3 data points per patients are used for scoring, so there will be one of the biggest shake in history I think.", "956156": "Oh I found it. The host said that there are around 200 in public+private.\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165723\n\nThen public is around 30 patients * 3 weeks = 90 points and private is around 170 patients * 3 weeks = 510 points?\nLB seems too shaky so we need concrete CV.", "960158": "But there is only 5 patients in the test set - isn't it 15 cases for public?", "960424": "Public and private sets are entirely hidden from us and when we finish a commit with test set the script will be rerun with public and private sets.\nThe test set is not used for scoring, but for sanity check.", "961041": "Thanks for the explanation; I didn’t realise this.", "961284": "I'm expecting very few patients in public test set compared to private test set because every model is oversensitive right now. A single parameter change can even return ±0.1 change in score.", "961303": "Yeah this would make sense. I'm trying to create a reliable cross validation for this competition so that my LB score is close to what I think it will be but that may be difficult.", "961624": "±0.1 change? That's big. Maybe different seeds will yield similar result.\nGiven that the top score is -6.7599 and the best notebook score is -6.8322, it seems that no one has achieved meaningful result yet.\n\nI think we can detect the size of public set by LB probing, can't we?", "961640": "You can basically create a generator object with `df_test.groupby('Patient')` and predict all data points of 1 patient, then do that for 2 patients and go on like this. This will consume too many submissions and there must be a more efficient way to do this.", "961650": "1. predict all data points with (10000, 50)\n2. predict all data points with (10000, 100)\n3. predict 1 data points with (10000, 50) and others with (10000, 100)\n\nI thought this is enough." }, "source": "meta" }