Just curious to all the people who are using CT scan data as a part of their modelling, what's your maximum CV/LB attained using only CT-scans?
", "messages": [ { "id": "1002743", "postDate": "09/08/2020 11:47:37", "content": "Just curious to all the people who are using CT scan data as a part of their modelling, what's your maximum CV/LB attained using only CT-scans?
", "rawMarkdown": "Just curious to all the people who are using CT scan data as a part of their modelling, what's your maximum CV/LB attained using only CT-scans?", "votes": null }, { "id": "1002934", "postDate": "09/08/2020 14:40:15", "content": "havent used so far.. but plan to..
\nsorry i dint except your invite.. i had people in queue.
I have made use of features like lung volume, tissue area, average ratio of tissue pixels by lung pixels for each patient and a few other features. I am using GroupKFold CV and obtained an overall OOF CV score of -6.7012(much better CV than any other model I have) and the corresponding LB is -6.90. I see no correlation b/w CV and LB whatsoever over multiple exoeriments, even ensembles with good OOF CV dont improve the LB. I'm also sure that all the existing high scoring public notebooks are overfitting the LB as even seed changes lead to a big drop in LB score. I also tried ensembling my model with image features with other public notebooks, the CV improves but LB drops. LB is based on only 15% of the test data and looks very risky at this point, expecting a big shakeup.
", "rawMarkdown": "I have made use of features like lung volume, tissue area, average ratio of tissue pixels by lung pixels for each patient and a few other features. I am using GroupKFold CV and obtained an overall OOF CV score of -6.7012(much better CV than any other model I have) and the corresponding LB is -6.90. I see no correlation b/w CV and LB whatsoever over multiple exoeriments, even ensembles with good OOF CV dont improve the LB. I'm also sure that all the existing high scoring public notebooks are overfitting the LB as even seed changes lead to a big drop in LB score. I also tried ensembling my model with image features with other public notebooks, the CV improves but LB drops. LB is based on only 15% of the test data and looks very risky at this point, expecting a big shakeup.", "votes": null }, { "id": "1006884", "postDate": "09/11/2020 16:00:30", "content": "Thank you for reporting your observations @abhishekgbhat , and yes there is no visible correlation between the CV and LB. Given the fact that 70 percent of top 200 teams is just mloss parameters tuning, the LB is already a house of cards compounded multiple times.
Our experiments in image features have been rather unsuccessful, with a best CV of -6.90 approx and a -7.xxxx LB.
", "rawMarkdown": "Thank you for reporting your observations @abhishekgbhat , and yes there is no visible correlation between the CV and LB. Given the fact that 70 percent of top 200 teams is just `mloss` parameters tuning, the LB is already a house of cards compounded multiple times.\n\nOur experiments in image features have been rather unsuccessful, with a best CV of -6.90 approx and a -7.xxxx LB.", "votes": null }, { "id": "1007903", "postDate": "09/12/2020 16:03:36", "content": "With only CT scan based model, I achieved a maximum -6.92 on LB, but on the CV I got around -6.79, so far couldn't find a good correlation between CV and LB though.
", "rawMarkdown": "With only CT scan based model, I achieved a maximum -6.92 on LB, but on the CV I got around -6.79, so far couldn't find a good correlation between CV and LB though.", "votes": null } ], "comments": [ { "id": 1002934, "author_name": "jaideepvalani", "author_url": "", "post_date": "09/08/2020 14:40:15", "content": "havent used so far.. but plan to..
\nsorry i dint except your invite.. i had people in queue.
I have made use of features like lung volume, tissue area, average ratio of tissue pixels by lung pixels for each patient and a few other features. I am using GroupKFold CV and obtained an overall OOF CV score of -6.7012(much better CV than any other model I have) and the corresponding LB is -6.90. I see no correlation b/w CV and LB whatsoever over multiple exoeriments, even ensembles with good OOF CV dont improve the LB. I'm also sure that all the existing high scoring public notebooks are overfitting the LB as even seed changes lead to a big drop in LB score. I also tried ensembling my model with image features with other public notebooks, the CV improves but LB drops. LB is based on only 15% of the test data and looks very risky at this point, expecting a big shakeup.
", "votes": null, "replies": [ { "id": 1006884, "author_name": "nxrprime", "author_url": "", "post_date": "09/11/2020 16:00:30", "content": "Thank you for reporting your observations @abhishekgbhat , and yes there is no visible correlation between the CV and LB. Given the fact that 70 percent of top 200 teams is just mloss parameters tuning, the LB is already a house of cards compounded multiple times.
Our experiments in image features have been rather unsuccessful, with a best CV of -6.90 approx and a -7.xxxx LB.
", "votes": null, "replies": [] } ] }, { "id": 1007903, "author_name": "furcifer", "author_url": "", "post_date": "09/12/2020 16:03:36", "content": "With only CT scan based model, I achieved a maximum -6.92 on LB, but on the CV I got around -6.79, so far couldn't find a good correlation between CV and LB though.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1002743": "Just curious to all the people who are using CT scan data as a part of their modelling, what's your maximum CV/LB attained using only CT-scans?", "1002934": "havent used so far.. but plan to.. \nsorry i dint except your invite.. i had people in queue.", "1006292": "I have made use of features like lung volume, tissue area, average ratio of tissue pixels by lung pixels for each patient and a few other features. I am using GroupKFold CV and obtained an overall OOF CV score of -6.7012(much better CV than any other model I have) and the corresponding LB is -6.90. I see no correlation b/w CV and LB whatsoever over multiple exoeriments, even ensembles with good OOF CV dont improve the LB. I'm also sure that all the existing high scoring public notebooks are overfitting the LB as even seed changes lead to a big drop in LB score. I also tried ensembling my model with image features with other public notebooks, the CV improves but LB drops. LB is based on only 15% of the test data and looks very risky at this point, expecting a big shakeup.", "1006884": "Thank you for reporting your observations @abhishekgbhat , and yes there is no visible correlation between the CV and LB. Given the fact that 70 percent of top 200 teams is just `mloss` parameters tuning, the LB is already a house of cards compounded multiple times.\n\nOur experiments in image features have been rather unsuccessful, with a best CV of -6.90 approx and a -7.xxxx LB.", "1007903": "With only CT scan based model, I achieved a maximum -6.92 on LB, but on the CV I got around -6.79, so far couldn't find a good correlation between CV and LB though." }, "source": "meta" }