It's well known that COVID-19 has been disproportionately affecting certain demographics. In the United States, the black community represented 52% of the cases but cover only 20% of the US counties. More recently, India has been hit hard with a sudden surge of new cases caused by a new variant.
\nI believe that this competition has the potential to really help those communities. However, from the data description, I can't immediately tell if various ethnicity and age groups are represented. I believe it would be beneficial for the greater data science community if metadata and statistics about the demographic is released. This information can be used by doctors and engineers throughout the world as they build on top of the work that will be produced in this competition in order to help their own community.
", "messages": [ { "id": "1312320", "postDate": "05/18/2021 01:50:31", "content": "It's well known that COVID-19 has been disproportionately affecting certain demographics. In the United States, the black community represented 52% of the cases but cover only 20% of the US counties. More recently, India has been hit hard with a sudden surge of new cases caused by a new variant.
\nI believe that this competition has the potential to really help those communities. However, from the data description, I can't immediately tell if various ethnicity and age groups are represented. I believe it would be beneficial for the greater data science community if metadata and statistics about the demographic is released. This information can be used by doctors and engineers throughout the world as they build on top of the work that will be produced in this competition in order to help their own community.
", "rawMarkdown": "It's well known that COVID-19 has been disproportionately affecting certain demographics. In the United States, the [black community represented 52% of the cases but cover only 20% of the US counties](https://www.sciencedirect.com/science/article/pii/S1047279720301769?via%3Dihub). More recently, India has been hit hard with a [sudden surge](https://www.nytimes.com/interactive/2020/world/asia/india-coronavirus-cases.html) of new cases caused by a new variant.\n\nI believe that this competition has the potential to really help those communities. However, from the data description, I can't immediately tell if various ethnicity and age groups are represented. I believe it would be beneficial for the greater data science community if metadata and statistics about the demographic is released. This information can be used by doctors and engineers throughout the world as they build on top of the work that will be produced in this competition in order to help their own community.", "votes": null }, { "id": "1313776", "postDate": "05/18/2021 17:55:01", "content": "Hi @xhlulu ,
\nI am from India, and reading this discussion thread makes me so happy.
\nUnless the pull for fair representation comes from inside the data community and from experts like you, we wont see real change.
\nThanks for raising this.
It seems to me that the patient's demographic data should not be considered in a system designed to classify this specific pathology. I'm not a physician, but I do not believe race or ethnicity itself directly predisposes a person to Covid19 .. biologically speaking.
\nUsing this kind of demographics would be detrimental in my mind. If a patient is Indian, but lives in a different country, it would be wrong to weight his results with such a feature.
\nKnowing the location where the image was acquired would make more sense to me. Of course any facility or identifiable UID data has been anonymized in this competition.
", "rawMarkdown": "It seems to me that the patient's demographic data **should not** be considered in a system designed to classify this specific pathology. I'm not a physician, but I do not believe race or ethnicity itself directly predisposes a person to Covid19 .. biologically speaking. \n\nUsing this kind of demographics would be detrimental in my mind. If a patient is Indian, but lives in a different country, it would be wrong to weight his results with such a feature.\n\nKnowing the location where the image was acquired would make more sense to me. Of course any facility or identifiable UID data has been anonymized in this competition.", "votes": null }, { "id": "1313928", "postDate": "05/18/2021 20:03:50", "content": "I agree that finegrained patient demographic information should not used inside the model; in fact that would create even more bias.
\nHowever it would be beneficial to have test splits that show the model generalizes across different (potentially underrepresented) demographic groups. This would make the resulting models much more useful for researchers and doctors that want to apply them in a different setting, where the target audience will have a different distribution compared to this competition. Hence I chose the term \"statistics\" and \"metadata\".
", "rawMarkdown": "I agree that finegrained patient demographic information should not used inside the model; in fact that would create even more bias.\n\nHowever it would be beneficial to have test splits that show the model generalizes across different (potentially underrepresented) demographic groups. This would make the resulting models much more useful for researchers and doctors that want to apply them in a different setting, where the target audience will have a different distribution compared to this competition. Hence I chose the term \"statistics\" and \"metadata\".", "votes": null }, { "id": "1313934", "postDate": "05/18/2021 20:07:27", "content": "Thank you for the kind words @kritidoneria. Although I enjoy sharing code in this type of competition, I am not an expert in this - I'd love to see real experts like Drs. @alexandrecc @vaillant weigh in on this type of issues.
", "rawMarkdown": "Thank you for the kind words @kritidoneria. Although I enjoy sharing code in this type of competition, I am not an expert in this - I'd love to see real experts like Drs. @alexandrecc @vaillant weigh in on this type of issues.", "votes": null }, { "id": "1313948", "postDate": "05/18/2021 20:19:12", "content": "@xhlulu , thank you for taking time to respond!
", "rawMarkdown": "xhlulu , thank you for taking time to respond!", "votes": null }, { "id": "1313984", "postDate": "05/18/2021 21:08:10", "content": "Hi @xhlulu ,
\nLooking forward to seeing the works of the experts. Before nationality, I'd like to see how this model does for men vs women, since we know chest proportions differ by sex.
\nThanks
Hi @davidbroberts and @shaz13,
\nIn addition to what @xhlulu has already articulated,
\nThis HBR article article outlines why simply removing protected classes doesn't work.
\nThanks
At least some of the DICOMs have the patients sex. Would it be sensible to make different models for each sex?
", "rawMarkdown": "At least some of the DICOMs have the patients sex. Would it be sensible to make different models for each sex?", "votes": null }, { "id": "1314838", "postDate": "05/19/2021 12:08:03", "content": "I disagree diversification at this point. For following points;
\nRace / Ethinicty
\nFeasibility
\nDeclaring bias is not a fare game on Race itself. Let's bring Age, Sex and other factors in. Now collecting all that data given supressed medical systems and sometimes people not believing in vaccines or take effort to wear masks. I believe its quite impossible to gather such kind of dataset.
May be in 2025 or 2030, from around the world. Still may be?
You are actually totally right on the Covid-19 relating to factors discussed. I believe much more important is at play apart from Chest X-Rays.
\nMore data would be better? Definitely. Feasible now, not much so đź‘Ť
", "rawMarkdown": "I disagree diversification at this point. For following points;\n\n1. Race / Ethinicty \n - Biologically I believe X-Rays are totally independent (my opinion)\n - Also, having those as features in dataset and real bias would set in.**You can actually win competition with biased model.** If test data has proportions wrong\n\n2. Feasibility \n - Declaring bias is not a fare game on Race itself. Let's bring Age, Sex and other factors in. Now collecting all that data given supressed medical systems and sometimes people not believing in vaccines or take effort to wear masks. I believe its quite impossible to gather such kind of dataset.\n\n - May be in 2025 or 2030, from around the world. Still may be? \n\n\nYou are actually totally right on the Covid-19 relating to factors discussed. I believe much more important is at play apart from Chest X-Rays.\n\n1. Diet of Patient\n2. [Blood Group ](https://ashpublications.org/bloodadvances/article/4/20/4981/464437/The-association-of-ABO-blood-group-with-indices-of) \n3. Local Immunity due to country/continent\n4. Hereditary Immunity\n\nMore data would be better? Definitely. Feasible now, not much so đź‘Ť", "votes": null }, { "id": "1315137", "postDate": "05/19/2021 15:33:42", "content": "I'm not an expert in radiology so I can't say if x-ray is dependent on racial factors. However, I'll share this Nature Medicine paper, which has a very relevant abstract section:
\n\n", "rawMarkdown": "I'm not an expert in radiology so I can't say if x-ray is dependent on racial factors. However, I'll share this [Nature Medicine paper](https://cs.stanford.edu/~jure/pubs/pain-nature_medicine21.pdf), which has a very relevant abstract section:\n> We show that the algorithm’s ability to reduce unexplained disparities is rooted in the racial and socioeconomic diversity of the training set. Because algorithmic severity measures better capture underserved patients’ pain, and severity measures influence treatment decisions, algorithmic predictions could potentially redress disparities in access to treatments like arthroplasty. An algorithmic, machine-learning approach to measuring severe pain from osteoarthritis applied to X-ray images of knees suggests that reported disparities in knee pain in underserved populations can be reduced by comparison with use of standard radiographic measures of disease severity.", "votes": null }, { "id": "1321767", "postDate": "05/24/2021 23:49:47", "content": "We show that the algorithm’s ability to reduce unexplained disparities is rooted in the racial and socioeconomic diversity of the training set. Because algorithmic severity measures better capture underserved patients’ pain, and severity measures influence treatment decisions, algorithmic predictions could potentially redress disparities in access to treatments like arthroplasty. An algorithmic, machine-learning approach to measuring severe pain from osteoarthritis applied to X-ray images of knees suggests that reported disparities in knee pain in underserved populations can be reduced by comparison with use of standard radiographic measures of disease severity.
\n
Your statement that \"Race / Ethnicity is mostly due to color, region and culture based\" is incorrect. Race/Ethnicity (and Gender) has biological effects due to genetic expression. This is a well-known problem in medical research not being inclusive of a diverse population and there being negative outcomes for those populations not properly represented in the studies.
\nWe do not necessarily require the application of the demographics of the patient to the model in order to be successful. The issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-sections of the patient population of the study.
", "rawMarkdown": "Your statement that \"Race / Ethnicity is mostly due to color, region and culture based\" is incorrect. Race/Ethnicity (and Gender) has biological effects due to genetic expression. This is a well-known problem in medical research not being inclusive of a diverse population and there being negative outcomes for those populations not properly represented in the studies.\n\nWe do not necessarily require the application of the demographics of the patient to the model in order to be successful. ****The issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-sections of the patient population of the study.****", "votes": null }, { "id": "1321858", "postDate": "05/25/2021 03:42:15", "content": "@lonestart - I didn't knew that. I made changes in the thread. Can you provide us with links/references to study more on it. Thanks for your insight
", "rawMarkdown": "lonestart - I didn't knew that. I made changes in the thread. Can you provide us with links/references to study more on it. Thanks for your insight", "votes": null }, { "id": "1321860", "postDate": "05/25/2021 03:44:06", "content": "\n\nThe issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-section of the patient population of the study.
\n
I couldn't agree more
", "rawMarkdown": "> The issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-section of the patient population of the study.\n\nI couldn't agree more", "votes": null }, { "id": "1328717", "postDate": "05/30/2021 13:14:30", "content": "Since people from different geographical regions have been hit by different variants of the virus, is it possible that their X-Ray scans would also differ due to the variance in the virus?
", "rawMarkdown": "Since people from different geographical regions have been hit by different variants of the virus, is it possible that their X-Ray scans would also differ due to the variance in the virus?", "votes": null } ], "comments": [ { "id": 1313776, "author_name": "kritidoneria", "author_url": "", "post_date": "05/18/2021 17:55:01", "content": "Hi @xhlulu ,
\nI am from India, and reading this discussion thread makes me so happy.
\nUnless the pull for fair representation comes from inside the data community and from experts like you, we wont see real change.
\nThanks for raising this.
Thank you for the kind words @kritidoneria. Although I enjoy sharing code in this type of competition, I am not an expert in this - I'd love to see real experts like Drs. @alexandrecc @vaillant weigh in on this type of issues.
", "votes": null, "replies": [] }, { "id": 1313984, "author_name": "kritidoneria", "author_url": "", "post_date": "05/18/2021 21:08:10", "content": "Hi @xhlulu ,
\nLooking forward to seeing the works of the experts. Before nationality, I'd like to see how this model does for men vs women, since we know chest proportions differ by sex.
\nThanks
At least some of the DICOMs have the patients sex. Would it be sensible to make different models for each sex?
", "votes": null, "replies": [] } ] }, { "id": 1313907, "author_name": "davidbroberts", "author_url": "", "post_date": "05/18/2021 19:36:47", "content": "It seems to me that the patient's demographic data should not be considered in a system designed to classify this specific pathology. I'm not a physician, but I do not believe race or ethnicity itself directly predisposes a person to Covid19 .. biologically speaking.
\nUsing this kind of demographics would be detrimental in my mind. If a patient is Indian, but lives in a different country, it would be wrong to weight his results with such a feature.
\nKnowing the location where the image was acquired would make more sense to me. Of course any facility or identifiable UID data has been anonymized in this competition.
", "votes": null, "replies": [ { "id": 1313928, "author_name": "xhlulu", "author_url": "", "post_date": "05/18/2021 20:03:50", "content": "I agree that finegrained patient demographic information should not used inside the model; in fact that would create even more bias.
\nHowever it would be beneficial to have test splits that show the model generalizes across different (potentially underrepresented) demographic groups. This would make the resulting models much more useful for researchers and doctors that want to apply them in a different setting, where the target audience will have a different distribution compared to this competition. Hence I chose the term \"statistics\" and \"metadata\".
", "votes": null, "replies": [] }, { "id": 1313948, "author_name": "davidbroberts", "author_url": "", "post_date": "05/18/2021 20:19:12", "content": "@xhlulu , thank you for taking time to respond!
", "votes": null, "replies": [] }, { "id": 1313988, "author_name": "kritidoneria", "author_url": "", "post_date": "05/18/2021 21:11:38", "content": "Hi @davidbroberts and @shaz13,
\nIn addition to what @xhlulu has already articulated,
\nThis HBR article article outlines why simply removing protected classes doesn't work.
\nThanks
I disagree diversification at this point. For following points;
\nRace / Ethinicty
\nFeasibility
\nDeclaring bias is not a fare game on Race itself. Let's bring Age, Sex and other factors in. Now collecting all that data given supressed medical systems and sometimes people not believing in vaccines or take effort to wear masks. I believe its quite impossible to gather such kind of dataset.
May be in 2025 or 2030, from around the world. Still may be?
You are actually totally right on the Covid-19 relating to factors discussed. I believe much more important is at play apart from Chest X-Rays.
\nMore data would be better? Definitely. Feasible now, not much so đź‘Ť
", "votes": null, "replies": [ { "id": 1315137, "author_name": "xhlulu", "author_url": "", "post_date": "05/19/2021 15:33:42", "content": "I'm not an expert in radiology so I can't say if x-ray is dependent on racial factors. However, I'll share this Nature Medicine paper, which has a very relevant abstract section:
\n\n", "votes": null, "replies": [] }, { "id": 1321767, "author_name": "lonestar", "author_url": "", "post_date": "05/24/2021 23:49:47", "content": "We show that the algorithm’s ability to reduce unexplained disparities is rooted in the racial and socioeconomic diversity of the training set. Because algorithmic severity measures better capture underserved patients’ pain, and severity measures influence treatment decisions, algorithmic predictions could potentially redress disparities in access to treatments like arthroplasty. An algorithmic, machine-learning approach to measuring severe pain from osteoarthritis applied to X-ray images of knees suggests that reported disparities in knee pain in underserved populations can be reduced by comparison with use of standard radiographic measures of disease severity.
\n
Your statement that \"Race / Ethnicity is mostly due to color, region and culture based\" is incorrect. Race/Ethnicity (and Gender) has biological effects due to genetic expression. This is a well-known problem in medical research not being inclusive of a diverse population and there being negative outcomes for those populations not properly represented in the studies.
\nWe do not necessarily require the application of the demographics of the patient to the model in order to be successful. The issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-sections of the patient population of the study.
", "votes": null, "replies": [] }, { "id": 1321858, "author_name": "shaz13", "author_url": "", "post_date": "05/25/2021 03:42:15", "content": "@lonestart - I didn't knew that. I made changes in the thread. Can you provide us with links/references to study more on it. Thanks for your insight
", "votes": null, "replies": [] }, { "id": 1321860, "author_name": "shaz13", "author_url": "", "post_date": "05/25/2021 03:44:06", "content": "\n\nThe issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-section of the patient population of the study.
\n
I couldn't agree more
", "votes": null, "replies": [] } ] }, { "id": 1328717, "author_name": "tahsintunan", "author_url": "", "post_date": "05/30/2021 13:14:30", "content": "Since people from different geographical regions have been hit by different variants of the virus, is it possible that their X-Ray scans would also differ due to the variance in the virus?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1312320": "It's well known that COVID-19 has been disproportionately affecting certain demographics. In the United States, the [black community represented 52% of the cases but cover only 20% of the US counties](https://www.sciencedirect.com/science/article/pii/S1047279720301769?via%3Dihub). More recently, India has been hit hard with a [sudden surge](https://www.nytimes.com/interactive/2020/world/asia/india-coronavirus-cases.html) of new cases caused by a new variant.\n\nI believe that this competition has the potential to really help those communities. However, from the data description, I can't immediately tell if various ethnicity and age groups are represented. I believe it would be beneficial for the greater data science community if metadata and statistics about the demographic is released. This information can be used by doctors and engineers throughout the world as they build on top of the work that will be produced in this competition in order to help their own community.", "1313776": "Hi @xhlulu ,\nI am from India, and reading this discussion thread makes me so happy.\nUnless the pull for fair representation comes from inside the data community and from experts like you, we wont see real change.\nThanks for raising this.", "1313907": "It seems to me that the patient's demographic data **should not** be considered in a system designed to classify this specific pathology. I'm not a physician, but I do not believe race or ethnicity itself directly predisposes a person to Covid19 .. biologically speaking. \n\nUsing this kind of demographics would be detrimental in my mind. If a patient is Indian, but lives in a different country, it would be wrong to weight his results with such a feature.\n\nKnowing the location where the image was acquired would make more sense to me. Of course any facility or identifiable UID data has been anonymized in this competition.", "1313928": "I agree that finegrained patient demographic information should not used inside the model; in fact that would create even more bias.\n\nHowever it would be beneficial to have test splits that show the model generalizes across different (potentially underrepresented) demographic groups. This would make the resulting models much more useful for researchers and doctors that want to apply them in a different setting, where the target audience will have a different distribution compared to this competition. Hence I chose the term \"statistics\" and \"metadata\".", "1313934": "Thank you for the kind words @kritidoneria. Although I enjoy sharing code in this type of competition, I am not an expert in this - I'd love to see real experts like Drs. @alexandrecc @vaillant weigh in on this type of issues.", "1313948": "xhlulu , thank you for taking time to respond!", "1313984": "Hi @xhlulu ,\nLooking forward to seeing the works of the experts. Before nationality, I'd like to see how this model does for men vs women, since we know chest proportions differ by sex.\nThanks", "1313988": "Hi @davidbroberts and @shaz13,\nIn addition to what @xhlulu has already articulated,\n[This HBR article](https://hbr.org/2020/08/how-to-fight-discrimination-in-ai) article outlines why simply removing protected classes doesn't work.\nThanks", "1314004": "At least some of the DICOMs have the patients sex. Would it be sensible to make different models for each sex?", "1314838": "I disagree diversification at this point. For following points;\n\n1. Race / Ethinicty \n - Biologically I believe X-Rays are totally independent (my opinion)\n - Also, having those as features in dataset and real bias would set in.**You can actually win competition with biased model.** If test data has proportions wrong\n\n2. Feasibility \n - Declaring bias is not a fare game on Race itself. Let's bring Age, Sex and other factors in. Now collecting all that data given supressed medical systems and sometimes people not believing in vaccines or take effort to wear masks. I believe its quite impossible to gather such kind of dataset.\n\n - May be in 2025 or 2030, from around the world. Still may be? \n\n\nYou are actually totally right on the Covid-19 relating to factors discussed. I believe much more important is at play apart from Chest X-Rays.\n\n1. Diet of Patient\n2. [Blood Group ](https://ashpublications.org/bloodadvances/article/4/20/4981/464437/The-association-of-ABO-blood-group-with-indices-of) \n3. Local Immunity due to country/continent\n4. Hereditary Immunity\n\nMore data would be better? Definitely. Feasible now, not much so 👍", "1315137": "I'm not an expert in radiology so I can't say if x-ray is dependent on racial factors. However, I'll share this [Nature Medicine paper](https://cs.stanford.edu/~jure/pubs/pain-nature_medicine21.pdf), which has a very relevant abstract section:\n> We show that the algorithm’s ability to reduce unexplained disparities is rooted in the racial and socioeconomic diversity of the training set. Because algorithmic severity measures better capture underserved patients’ pain, and severity measures influence treatment decisions, algorithmic predictions could potentially redress disparities in access to treatments like arthroplasty. An algorithmic, machine-learning approach to measuring severe pain from osteoarthritis applied to X-ray images of knees suggests that reported disparities in knee pain in underserved populations can be reduced by comparison with use of standard radiographic measures of disease severity.", "1321767": "Your statement that \"Race / Ethnicity is mostly due to color, region and culture based\" is incorrect. Race/Ethnicity (and Gender) has biological effects due to genetic expression. This is a well-known problem in medical research not being inclusive of a diverse population and there being negative outcomes for those populations not properly represented in the studies.\n\nWe do not necessarily require the application of the demographics of the patient to the model in order to be successful. ****The issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-sections of the patient population of the study.****", "1321858": "lonestart - I didn't knew that. I made changes in the thread. Can you provide us with links/references to study more on it. Thanks for your insight", "1321860": "> The issue at hand here is it should be known that the patient demographics are sufficiently diverse to assure that the learned model captures patterns that may be specific to a cross-section of the patient population of the study.\n\nI couldn't agree more", "1328717": "Since people from different geographical regions have been hit by different variants of the virus, is it possible that their X-Ray scans would also differ due to the variance in the virus?" }, "source": "meta" }