Came upon a recently published paper [1] in nature on Covid-19 detection in CXR and CT images.
\nThe authors reviewed 62 ML detection models (papers) for Covid-19 detection. They make a bold conclusion that
\n\n\"Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases \".
\n
This definitely got my attention I started reading the paper. However I found no constructive advice on how such a bias can be corrected.
\nThough the authors claim
\n\n\n\" We also give detailed recommendations in five domains: (1) considerations when collating COVID-19 imaging datasets that are to be made public; (2) methodological considerations for algorithm developers; (3) specific issues about reproducibility of the results in the literature; (4) considerations for authors to ensure sufficient documentation of methodologies in manuscripts; and (5) considerations for reviewers performing peer review of manuscripts.\"
\n
The only piece of constructive advice I found was that - many studies combined multiple datasets without realizing that the studies within them may be duplicated.
\nHence there was data leakage: Train data -> Test data.
[1] Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
\nhttps://www.nature.com/articles/s42256-021-00307-0
", "messages": [ { "id": 1346178, "postDate": "2021-06-12T07:11:34.130Z", "content": "Came upon a recently published paper [1] in nature on Covid-19 detection in CXR and CT images.
\nThe authors reviewed 62 ML detection models (papers) for Covid-19 detection. They make a bold conclusion that
\n\n\"Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases \".
\n
This definitely got my attention I started reading the paper. However I found no constructive advice on how such a bias can be corrected.
\nThough the authors claim
\n\n\n\" We also give detailed recommendations in five domains: (1) considerations when collating COVID-19 imaging datasets that are to be made public; (2) methodological considerations for algorithm developers; (3) specific issues about reproducibility of the results in the literature; (4) considerations for authors to ensure sufficient documentation of methodologies in manuscripts; and (5) considerations for reviewers performing peer review of manuscripts.\"
\n
The only piece of constructive advice I found was that - many studies combined multiple datasets without realizing that the studies within them may be duplicated.
\nHence there was data leakage: Train data -> Test data.
[1] Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
\n", "rawMarkdown": "Came upon a recently published paper [1] in nature on Covid-19 detection in CXR and CT images.\nThe authors reviewed 62 ML detection models (papers) for Covid-19 detection. They make a bold conclusion that \n> \"*Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases* \".\n\nThis definitely got my attention I started reading the paper. However I found no constructive advice on how such a bias can be corrected. \n\nThough the authors claim\n> \" *We also give detailed recommendations in five domains: (1) considerations when collating COVID-19 imaging datasets that are to be made public; (2) **methodological considerations for algorithm developers;** (3) specific issues about reproducibility of the results in the literature; (4) considerations for authors to ensure sufficient documentation of methodologies in manuscripts; and (5) considerations for reviewers performing peer review of manuscripts*.\"\n\nThe only piece of constructive advice I found was that - many studies combined multiple datasets without realizing that the studies within them may be duplicated. \nHence there was data leakage: Train data -> Test data. \n\n- Do you see any constructive advice from this paper that we could utilize in our solutions ?\n\n\n[1] Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans\n\nhttps://www.nature.com/articles/s42256-021-00307-0", "votes": 6 } ], "comments": [], "raw_markdown_by_id": { "1346178": "Came upon a recently published paper [1] in nature on Covid-19 detection in CXR and CT images.\nThe authors reviewed 62 ML detection models (papers) for Covid-19 detection. They make a bold conclusion that \n> \"*Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases* \".\n\nThis definitely got my attention I started reading the paper. However I found no constructive advice on how such a bias can be corrected. \n\nThough the authors claim\n> \" *We also give detailed recommendations in five domains: (1) considerations when collating COVID-19 imaging datasets that are to be made public; (2) **methodological considerations for algorithm developers;** (3) specific issues about reproducibility of the results in the literature; (4) considerations for authors to ensure sufficient documentation of methodologies in manuscripts; and (5) considerations for reviewers performing peer review of manuscripts*.\"\n\nThe only piece of constructive advice I found was that - many studies combined multiple datasets without realizing that the studies within them may be duplicated. \nHence there was data leakage: Train data -> Test data. \n\n- Do you see any constructive advice from this paper that we could utilize in our solutions ?\n\n\n[1] Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans\n\nhttps://www.nature.com/articles/s42256-021-00307-0" } }