As scientists from myriad fields rush to perform algorithmic analyses, Google’s Patrick Riley calls for clear standards in research and reporting.
\nYou can find the article in the link. https://www.nature.com/articles/d41586-019-02307-y
Three problems are: Splitting data inappropriately,Hidden variables and Mistaking the objective.
\n\nFor Splitting data inappropriately, based on the discussion \nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100815#latest-589248\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100471#latest-580157\nThe training data set and public data set are very different, and we don't know what Private data set would be.
\n\nFor Hidden variables. \nhttps://www.kaggle.com/taindow/be-careful-what-you-train-on\nhttps://www.kaggle.com/dimitreoliveira/diabetic-retinopathy-shap-model-explainability
\n\nFor Mistaking the objective\nHe uses diabetic retinopathy as the example.\n\"Diverging goals also cropped up in our work on machine screening for diabetic retinopathy, a complication of diabetes and a leading cause of preventable blindness in the world. The condition can be treated effectively if it is detected early, from images of the back of the eye. As we gathered data and had ophthalmologists offer diagnoses based on the images, we asked our machine-learning tools to predict what the ophthalmologist would say. Two issues emerged.\nFirst, the ophthalmologists often disagreed on the diagnosis. Thus, we realized that we could not base our model on a single prediction. Nor could we use a majority vote, because, when it comes to medical accuracy, sometimes the minority opinion is the right one. Second, the diagnosis of a single disease was not actually the real objective. We should have been asking: ‘should this patient see a doctor?’ We therefore expanded our goal from the diagnosis of a single disease to multiple diseases.\nIt is easy for machine-learning practitioners to become fixated on an ‘obvious’ objective in which the data and labels are clear. But they could be setting up the algorithm to solve the wrong problem. The overall aim must be kept in mind, or we will produce precise systems that answer the wrong questions.\"
\n\nSince we need to avoid these pitfalls in our projects, I hope Kaggle organizers could develop clear standards for how to split the data set(training,public and private) and what's the best objective(evaluation metric).
\n\nSo we can put our time in label cleaning, image cropping and modeling, and hope public leader board reflect these improvements.
", "messages": [ { "id": 589824, "postDate": "2019-08-01T12:25:08.357Z", "content": "As scientists from myriad fields rush to perform algorithmic analyses, Google’s Patrick Riley calls for clear standards in research and reporting.
\nYou can find the article in the link. https://www.nature.com/articles/d41586-019-02307-y
Three problems are: Splitting data inappropriately,Hidden variables and Mistaking the objective.
\n\nFor Splitting data inappropriately, based on the discussion \nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100815#latest-589248\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100471#latest-580157\nThe training data set and public data set are very different, and we don't know what Private data set would be.
\n\nFor Hidden variables. \nhttps://www.kaggle.com/taindow/be-careful-what-you-train-on\nhttps://www.kaggle.com/dimitreoliveira/diabetic-retinopathy-shap-model-explainability
\n\nFor Mistaking the objective\nHe uses diabetic retinopathy as the example.\n\"Diverging goals also cropped up in our work on machine screening for diabetic retinopathy, a complication of diabetes and a leading cause of preventable blindness in the world. The condition can be treated effectively if it is detected early, from images of the back of the eye. As we gathered data and had ophthalmologists offer diagnoses based on the images, we asked our machine-learning tools to predict what the ophthalmologist would say. Two issues emerged.\nFirst, the ophthalmologists often disagreed on the diagnosis. Thus, we realized that we could not base our model on a single prediction. Nor could we use a majority vote, because, when it comes to medical accuracy, sometimes the minority opinion is the right one. Second, the diagnosis of a single disease was not actually the real objective. We should have been asking: ‘should this patient see a doctor?’ We therefore expanded our goal from the diagnosis of a single disease to multiple diseases.\nIt is easy for machine-learning practitioners to become fixated on an ‘obvious’ objective in which the data and labels are clear. But they could be setting up the algorithm to solve the wrong problem. The overall aim must be kept in mind, or we will produce precise systems that answer the wrong questions.\"
\n\nSince we need to avoid these pitfalls in our projects, I hope Kaggle organizers could develop clear standards for how to split the data set(training,public and private) and what's the best objective(evaluation metric).
\n\nSo we can put our time in label cleaning, image cropping and modeling, and hope public leader board reflect these improvements.
", "rawMarkdown": "As scientists from myriad fields rush to perform algorithmic analyses, Google’s Patrick Riley calls for clear standards in research and reporting. \nYou can find the article in the link. https://www.nature.com/articles/d41586-019-02307-y\n\nThree problems are: Splitting data inappropriately,Hidden variables and Mistaking the objective. \n\nFor Splitting data inappropriately, based on the discussion \nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100815#latest-589248\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100471#latest-580157\nThe training data set and public data set are very different, and we don't know what Private data set would be.\n\nFor Hidden variables. \nhttps://www.kaggle.com/taindow/be-careful-what-you-train-on\nhttps://www.kaggle.com/dimitreoliveira/diabetic-retinopathy-shap-model-explainability\n\nFor Mistaking the objective\nHe uses diabetic retinopathy as the example.\n\"Diverging goals also cropped up in our work on machine screening for diabetic retinopathy, a complication of diabetes and a leading cause of preventable blindness in the world. The condition can be treated effectively if it is detected early, from images of the back of the eye. As we gathered data and had ophthalmologists offer diagnoses based on the images, we asked our machine-learning tools to predict what the ophthalmologist would say. Two issues emerged.\nFirst, the ophthalmologists often disagreed on the diagnosis. Thus, we realized that we could not base our model on a single prediction. Nor could we use a majority vote, because, when it comes to medical accuracy, sometimes the minority opinion is the right one. Second, the diagnosis of a single disease was not actually the real objective. We should have been asking: ‘should this patient see a doctor?’ We therefore expanded our goal from the diagnosis of a single disease to multiple diseases.\nIt is easy for machine-learning practitioners to become fixated on an ‘obvious’ objective in which the data and labels are clear. But they could be setting up the algorithm to solve the wrong problem. The overall aim must be kept in mind, or we will produce precise systems that answer the wrong questions.\"\n\nSince we need to avoid these pitfalls in our projects, I hope Kaggle organizers could develop clear standards for how to split the data set(training,public and private) and what's the best objective(evaluation metric). \n\nSo we can put our time in label cleaning, image cropping and modeling, and hope public leader board reflect these improvements.", "votes": 3 } ], "comments": [], "raw_markdown_by_id": { "589824": "As scientists from myriad fields rush to perform algorithmic analyses, Google’s Patrick Riley calls for clear standards in research and reporting. \nYou can find the article in the link. https://www.nature.com/articles/d41586-019-02307-y\n\nThree problems are: Splitting data inappropriately,Hidden variables and Mistaking the objective. \n\nFor Splitting data inappropriately, based on the discussion \nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100815#latest-589248\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/100471#latest-580157\nThe training data set and public data set are very different, and we don't know what Private data set would be.\n\nFor Hidden variables. \nhttps://www.kaggle.com/taindow/be-careful-what-you-train-on\nhttps://www.kaggle.com/dimitreoliveira/diabetic-retinopathy-shap-model-explainability\n\nFor Mistaking the objective\nHe uses diabetic retinopathy as the example.\n\"Diverging goals also cropped up in our work on machine screening for diabetic retinopathy, a complication of diabetes and a leading cause of preventable blindness in the world. The condition can be treated effectively if it is detected early, from images of the back of the eye. As we gathered data and had ophthalmologists offer diagnoses based on the images, we asked our machine-learning tools to predict what the ophthalmologist would say. Two issues emerged.\nFirst, the ophthalmologists often disagreed on the diagnosis. Thus, we realized that we could not base our model on a single prediction. Nor could we use a majority vote, because, when it comes to medical accuracy, sometimes the minority opinion is the right one. Second, the diagnosis of a single disease was not actually the real objective. We should have been asking: ‘should this patient see a doctor?’ We therefore expanded our goal from the diagnosis of a single disease to multiple diseases.\nIt is easy for machine-learning practitioners to become fixated on an ‘obvious’ objective in which the data and labels are clear. But they could be setting up the algorithm to solve the wrong problem. The overall aim must be kept in mind, or we will produce precise systems that answer the wrong questions.\"\n\nSince we need to avoid these pitfalls in our projects, I hope Kaggle organizers could develop clear standards for how to split the data set(training,public and private) and what's the best objective(evaluation metric). \n\nSo we can put our time in label cleaning, image cropping and modeling, and hope public leader board reflect these improvements." } }