![\"RMS](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30134519/errorTable.png\")
This year’s competition was intended to catalyze a change in cardiac diagnostics, so connecting the competition participants and the medical community is an essential part of the DSB. I have done some preliminary analysis of the Data Science Bowl’s (DSB) top 4 team submissions. The goal is to present the results in terms that are meaningful to the medical research community. In doing so I hope to spark a dialog between the communities.
\n\nDSB competitors produced diastolic and systolic volume cumulative probability distribution function (CDF) for each heart. Researchers, clinicians, and technicians will prefer an algorithm to output volumes, ejection fraction and a precision estimate for those predictions. Obtaining those quantities from the submission files requires some level of approximation. I will describe what was done for this post, noting where I made simplifying assumptions. The plots and visualizations presented here are valuable for understanding the significant of the results. Many of the plots were used because they appear in prior medical research on reliability of cardiac MRI.
\n\nPlease feel free to comment on, add to, and critique the approach outlined in this post. I have uploaded the script I used to generate the plots, and the solutions used to score the Private Leaderboard are live now.
\n\nDr. Andrew Arai of NIH provides a medical perspective on these results in another post, here.
\n\nThe DSB challenge was to use the images from a heart study to generate two cumulative probability distribution functions (CDF)—one for the end-systolic volume (SV) and one for the end-diastolic volume (DV). Those volumes are the left ventricle sizes at the beginning and middle of each heartbeat. There are 600 values in each row, giving the the predicted probability that the true volume of the chamber in mL is less than the value index (i.e., if the 34th value in a row is 0.55, then the model is predicting that there is a 55% chance that the volume is less than 34 mL). So, subtracting adjacent values will give the predicted cumulative probability that the volume falls between the value indices (ie, if the 35th value is 60% and the 34th value is 55%, then the predicted probability that the volume is between 34 mL and 35 mL is 5%). Doing this subtraction for each consecutive value in a row generates a discretized probability density function (PDF):
\n\n$$P(i+0.5)=CDF_{i+1}-CDF_{i}$$,
\n\nwhere the value for P(v) can be interpreted as the predicted approximate probability per mL that the true volume is v. For me, the PDF is a more intuitive way to look at the model submissions. A volume that the model is predicting as likely for a heart will show up as a peak in the PDF. I am using the expectation value and the standard deviation of the PDF as the single-value volume prediction and the uncertainty estimate for that prediction respectively. Summing over the bins in the PDF, the prediction and uncertainty estimate are:
\n\n$$V_{pred}=\\sum_{i=0}^{598}(i+0.5)\\times P(i) and \\sigma_{V}^2=\\sum_{i=0}^{598}[V_{pred}-(i+0.5)]^2\\times P(i)$$
\n\nThe entries are already normalized, and the width of the bins are 1 mL, so I am not explicitly writing out the bin width or the normalization for these averages. Note, the PDF necessarily has one fewer bins than the CDF. This may look confusing depending on your background, but all I’m doing is finding the average and standard deviation of the PDF.
\n\nIn order to give you an idea of what these CDFs look like after I turn them into PDFs, I have included a set of 100 random PDFs along with the true values, for the top four teams’ submissions. Two submissions have nearly normal distributions and the other two have some distinct non-normal features. I invite feedback and discussion on better ways to translate the CDFs into single number predictions, or any comments on why there is a better way to look at the problem of digesting model output for consumption by the medical research community.
\n\nPlease see the uploaded files for the PDF comparisons. They are too big to post in-line
\n\nAnother important diagnostic metric is the ejection fraction (EF), which is calculated as
\n\n$$EF=\\frac{\\mbox{Diastolic Volume}-\\mbox{Systolic Volume}}{\\mbox{Diastolic Volume}}$$.
\n\nFor each heart, I simulated EF distributions by collecting EFs calculated from 10,000 SV-DV pairs sampled from the SV and DV PDFs. I then took the expectation value of the simulated EF distribution as the model’s EF prediction. I verified over a subset of heart studies that the sample size I chose produces a stable EF expectation value.
\n\nThis overall approach requires some caveats. Using the expectation value as the “prediction” is only reasonable for PDFs that are well behaved and unimodal. In reality a model could output a prediction that gives high probability to disparate systolic (or diastolic) volumes, in which case, the expectation value of SV (or DV) may be inappropriate. Estimating the EF distribution from the SV and DV distributions may also be problematic because the SV and DV are clearly correlated. This simplest solution is to seek a model that directly produces the EF value.
\n\nMy overall approach will provide, however, an approximation of results that can be used to start a conversation with the medical community. We will eventually need to work with the medical community to develop a more rigorous analysis that takes into account some knowledge of the models. The medical community and the data science community must combine their effort to develop a method to rigorously translate model output into metrics that can be used for clinical evaluation and ultimately clinical use.
\n\nThe table below shows root-mean-square (RMS) errors for the diastolic and systolic volumes and the ejection fraction (EF) from each of the top 4 teams. Remember, this is the RMS error value for the difference between the test value and the single-value prediction calculation. For some clinical perspective on these errors, the precision on human clinical determination of diastolic volume, systolic volume, and ejection fraction is about 13 mL, 14mL, and 6% respectively. Consider those benchmarks hearsay for the moment. I will edit the post with numbers from the literature and some references ASAP. With those benchmarks in mind, the RMS errors look promising.
\n\n
We can glean more about the potential clinical applicability of the solutions than just a RMS error. Rather than approaching this only from a model evaluation standpoint, let’s return to the original motivation of the challenge—diagnosing heart disease.
\n\nAs noted about, the quantity of interest for many diagnoses is the ejection fraction (EF). Roughly speaking, an EF under 35% is a dire emergency, around 60% is normal, and above 73% is considered hyperactive. I calculated the confusion (a.k.a. contingency) matrices for the top 4 competitors using diagnostic EF bins provided by Dr. Andrew Arai. If you take a close look at the matrices, you can get a picture of what might happen in a medical setting if these algorithms were put to use.
\n\n![\"Contingency](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100556/ContingencyMatrix.png\")
The models keep the diagnosis categories pretty tightly grouped together. While the models are not right 100% the time, there is a very low probability of a severely abnormal EF being incorrectly categorized in the mild to hyperdynamic range. The normal to mild diagnoses are very likely to stay within their domain of the matrix. This is a pretty good sign pointing towards suitability for clinical applications.
\n\nOther metrics that are used to evaluate clinical measurement techniques are correlation plots and Bland-Altman (BA) plots. The correlation plots are straight forward linear fits to the scatter pot of the true values versus the predicted data. For the BA plots, $$V_{pred}-V_{true}$$ is plotted versus $$\\mbox{AVG}(V_{pred},V_{true})$$ for each prediction. A non-zero mean on the BA plot would indicate a relative bias between the two measurement techniques being compared (or in our case, a bias in the model, since we are assuming the test values are “true”). The dashed lines in the plot give the 95% confidence interval for the difference between the measurement types. What we would like to compare these BA plots to is other BA plots comparing two human measurements of the same data set. Is the variance in the plots comparable to human to human comparisons? Scroll to the bottom of the post for the correlation and BA plots.
\n\nWe are especially interested in producing reasonable confidence estimates around volume and ejection fraction predictions. Michael Hansen, Ph.D, co-PI for this DSB, noted that the best models can fail, but they should “fail loudly.” We’d like a good measure of confidence for each individual prediction. In other words, “How well does the model estimate its own accuracy?” Ideally, if an individual prediction has a high probability of being wrong, a model will flag that prediction as uncertain. As mentioned before, I’m using the PDF standard deviation as an estimate of the model’s prediction certainty. A sharp peak indicates higher confidence whereas a wide peak indicates greater uncertainty.
\n\nThe plot below is an illustration of a way to visualize a model’s ability to estimate its own accuracy. For each prediction, the standard deviation of the PDF is plotted versus the error of that prediction. The structure of a plot can be conceptualized as shown in the below graphic. Predictions that fall in the bottom left quadrant would be very confident (small width PDF) and accurate (small absolute error). Points in the top right would be low confidence (wide PDF) and inaccurate (large absolute error). Neither classes of points are necessarily bad. If all the predictions for a model fall within the two aforementioned quadrants, one would know when to trust the algorithm’s output and when to consider retaking a measurement. However, points in the other quadrants (large error but purported accuracy) could confuse one’s ability to rely on the algorithm’s uncertainty estimate.
\n\n![\"Quadrant](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/28193219/error-on-prediction.png\")
In the plots that follow, the majority of the data points are shown in grey. The 5% best and worst errors are highlighted as well as the 5% widest and thinnest PDFs. I have looked at this a couple of ways, and the model error and “confidence” distributions are not obviously correlated. For that reason, I suspect that there is a better way to evaluate the models’ estimation of its own accuracy.
\n\n![\"Prediction](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/28193219/individual-prediction-error.png\")
Hopefully, there will be an ongoing conversation between the medical and data science communities regarding the evaluation of these models for clinical use. Are there other metrics that would be useful? What else do you want to know about these models? What information from the model developers would help inform decisions that would be made in a clinical setting? From the data science community, is there more useful information that can be derived from the submission data? Are there better ways to estimate model prediction confidence? Are there better ways to calculate the prediction value?
\n\nThis post is a twin of the post on the DSB, which you can read here.
\n\n![\"TenciaWoshialex](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100610/Tencia_n_WoshialexBlandAltman.png\")
\n![\"kunsthard](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100604/kunsthartBlandAltman.png\")
\n![\"Julian](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100559/Julian_de_WitBlandAltman.png\")
\n![\"Show](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100608/show_me_the_moneyBlandAltman.png\")
![\"TenciaWoshialex](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100612/Tencia_n_WoshialexCorr.png\")
\n![\"kunsthart](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100605/kunsthartCorr.png\")
\n![\"Julian](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100602/Julian_de_WitCorr.png\")
\n![\"Show](\"https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100609/show_me_the_moneyCorr.png\")
This year’s competition was intended to catalyze a change in cardiac diagnostics, so connecting the competition participants and the medical community is an essential part of the DSB. I have done some preliminary analysis of the Data Science Bowl’s (DSB) top 4 team submissions. The goal is to present the results in terms that are meaningful to the medical research community. In doing so I hope to spark a dialog between the communities.
\n\nDSB competitors produced diastolic and systolic volume cumulative probability distribution function (CDF) for each heart. Researchers, clinicians, and technicians will prefer an algorithm to output volumes, ejection fraction and a precision estimate for those predictions. Obtaining those quantities from the submission files requires some level of approximation. I will describe what was done for this post, noting where I made simplifying assumptions. The plots and visualizations presented here are valuable for understanding the significant of the results. Many of the plots were used because they appear in prior medical research on reliability of cardiac MRI.
\n\nPlease feel free to comment on, add to, and critique the approach outlined in this post. I have uploaded the script I used to generate the plots, and the solutions used to score the Private Leaderboard are live now.
\n\nDr. Andrew Arai of NIH provides a medical perspective on these results in another post, here.
\n\nThe DSB challenge was to use the images from a heart study to generate two cumulative probability distribution functions (CDF)—one for the end-systolic volume (SV) and one for the end-diastolic volume (DV). Those volumes are the left ventricle sizes at the beginning and middle of each heartbeat. There are 600 values in each row, giving the the predicted probability that the true volume of the chamber in mL is less than the value index (i.e., if the 34th value in a row is 0.55, then the model is predicting that there is a 55% chance that the volume is less than 34 mL). So, subtracting adjacent values will give the predicted cumulative probability that the volume falls between the value indices (ie, if the 35th value is 60% and the 34th value is 55%, then the predicted probability that the volume is between 34 mL and 35 mL is 5%). Doing this subtraction for each consecutive value in a row generates a discretized probability density function (PDF):
\n\n$$P(i+0.5)=CDF_{i+1}-CDF_{i}$$,
\n\nwhere the value for P(v) can be interpreted as the predicted approximate probability per mL that the true volume is v. For me, the PDF is a more intuitive way to look at the model submissions. A volume that the model is predicting as likely for a heart will show up as a peak in the PDF. I am using the expectation value and the standard deviation of the PDF as the single-value volume prediction and the uncertainty estimate for that prediction respectively. Summing over the bins in the PDF, the prediction and uncertainty estimate are:
\n\n$$V_{pred}=\\sum_{i=0}^{598}(i+0.5)\\times P(i) and \\sigma_{V}^2=\\sum_{i=0}^{598}[V_{pred}-(i+0.5)]^2\\times P(i)$$
\n\nThe entries are already normalized, and the width of the bins are 1 mL, so I am not explicitly writing out the bin width or the normalization for these averages. Note, the PDF necessarily has one fewer bins than the CDF. This may look confusing depending on your background, but all I’m doing is finding the average and standard deviation of the PDF.
\n\nIn order to give you an idea of what these CDFs look like after I turn them into PDFs, I have included a set of 100 random PDFs along with the true values, for the top four teams’ submissions. Two submissions have nearly normal distributions and the other two have some distinct non-normal features. I invite feedback and discussion on better ways to translate the CDFs into single number predictions, or any comments on why there is a better way to look at the problem of digesting model output for consumption by the medical research community.
\n\nPlease see the uploaded files for the PDF comparisons. They are too big to post in-line
\n\nAnother important diagnostic metric is the ejection fraction (EF), which is calculated as
\n\n$$EF=\\frac{\\mbox{Diastolic Volume}-\\mbox{Systolic Volume}}{\\mbox{Diastolic Volume}}$$.
\n\nFor each heart, I simulated EF distributions by collecting EFs calculated from 10,000 SV-DV pairs sampled from the SV and DV PDFs. I then took the expectation value of the simulated EF distribution as the model’s EF prediction. I verified over a subset of heart studies that the sample size I chose produces a stable EF expectation value.
\n\nThis overall approach requires some caveats. Using the expectation value as the “prediction” is only reasonable for PDFs that are well behaved and unimodal. In reality a model could output a prediction that gives high probability to disparate systolic (or diastolic) volumes, in which case, the expectation value of SV (or DV) may be inappropriate. Estimating the EF distribution from the SV and DV distributions may also be problematic because the SV and DV are clearly correlated. This simplest solution is to seek a model that directly produces the EF value.
\n\nMy overall approach will provide, however, an approximation of results that can be used to start a conversation with the medical community. We will eventually need to work with the medical community to develop a more rigorous analysis that takes into account some knowledge of the models. The medical community and the data science community must combine their effort to develop a method to rigorously translate model output into metrics that can be used for clinical evaluation and ultimately clinical use.
\n\nThe table below shows root-mean-square (RMS) errors for the diastolic and systolic volumes and the ejection fraction (EF) from each of the top 4 teams. Remember, this is the RMS error value for the difference between the test value and the single-value prediction calculation. For some clinical perspective on these errors, the precision on human clinical determination of diastolic volume, systolic volume, and ejection fraction is about 13 mL, 14mL, and 6% respectively. Consider those benchmarks hearsay for the moment. I will edit the post with numbers from the literature and some references ASAP. With those benchmarks in mind, the RMS errors look promising.
\n\n
We can glean more about the potential clinical applicability of the solutions than just a RMS error. Rather than approaching this only from a model evaluation standpoint, let’s return to the original motivation of the challenge—diagnosing heart disease.
\n\nAs noted about, the quantity of interest for many diagnoses is the ejection fraction (EF). Roughly speaking, an EF under 35% is a dire emergency, around 60% is normal, and above 73% is considered hyperactive. I calculated the confusion (a.k.a. contingency) matrices for the top 4 competitors using diagnostic EF bins provided by Dr. Andrew Arai. If you take a close look at the matrices, you can get a picture of what might happen in a medical setting if these algorithms were put to use.
\n\n
The models keep the diagnosis categories pretty tightly grouped together. While the models are not right 100% the time, there is a very low probability of a severely abnormal EF being incorrectly categorized in the mild to hyperdynamic range. The normal to mild diagnoses are very likely to stay within their domain of the matrix. This is a pretty good sign pointing towards suitability for clinical applications.
\n\nOther metrics that are used to evaluate clinical measurement techniques are correlation plots and Bland-Altman (BA) plots. The correlation plots are straight forward linear fits to the scatter pot of the true values versus the predicted data. For the BA plots, $$V_{pred}-V_{true}$$ is plotted versus $$\\mbox{AVG}(V_{pred},V_{true})$$ for each prediction. A non-zero mean on the BA plot would indicate a relative bias between the two measurement techniques being compared (or in our case, a bias in the model, since we are assuming the test values are “true”). The dashed lines in the plot give the 95% confidence interval for the difference between the measurement types. What we would like to compare these BA plots to is other BA plots comparing two human measurements of the same data set. Is the variance in the plots comparable to human to human comparisons? Scroll to the bottom of the post for the correlation and BA plots.
\n\nWe are especially interested in producing reasonable confidence estimates around volume and ejection fraction predictions. Michael Hansen, Ph.D, co-PI for this DSB, noted that the best models can fail, but they should “fail loudly.” We’d like a good measure of confidence for each individual prediction. In other words, “How well does the model estimate its own accuracy?” Ideally, if an individual prediction has a high probability of being wrong, a model will flag that prediction as uncertain. As mentioned before, I’m using the PDF standard deviation as an estimate of the model’s prediction certainty. A sharp peak indicates higher confidence whereas a wide peak indicates greater uncertainty.
\n\nThe plot below is an illustration of a way to visualize a model’s ability to estimate its own accuracy. For each prediction, the standard deviation of the PDF is plotted versus the error of that prediction. The structure of a plot can be conceptualized as shown in the below graphic. Predictions that fall in the bottom left quadrant would be very confident (small width PDF) and accurate (small absolute error). Points in the top right would be low confidence (wide PDF) and inaccurate (large absolute error). Neither classes of points are necessarily bad. If all the predictions for a model fall within the two aforementioned quadrants, one would know when to trust the algorithm’s output and when to consider retaking a measurement. However, points in the other quadrants (large error but purported accuracy) could confuse one’s ability to rely on the algorithm’s uncertainty estimate.
\n\n
In the plots that follow, the majority of the data points are shown in grey. The 5% best and worst errors are highlighted as well as the 5% widest and thinnest PDFs. I have looked at this a couple of ways, and the model error and “confidence” distributions are not obviously correlated. For that reason, I suspect that there is a better way to evaluate the models’ estimation of its own accuracy.
\n\n
Hopefully, there will be an ongoing conversation between the medical and data science communities regarding the evaluation of these models for clinical use. Are there other metrics that would be useful? What else do you want to know about these models? What information from the model developers would help inform decisions that would be made in a clinical setting? From the data science community, is there more useful information that can be derived from the submission data? Are there better ways to estimate model prediction confidence? Are there better ways to calculate the prediction value?
\n\nThis post is a twin of the post on the DSB, which you can read here.
\n\n\n\n\n
\n\n\n
I've created a repository for this script, and it's been tidied up significantly.
\n\n", "rawMarkdown": "I've created a repository for this script, and it's been tidied up significantly.\r\n\r\n [The link for the repository][1]\r\n\r\n\r\n [1]: https://github.com/jonrmulholland/dsbAnalysis", "votes": null } ], "comments": [ { "id": 113743, "author_name": "jmulholland", "author_url": "", "post_date": "04/04/2016 17:44:30", "content": "I've created a repository for this script, and it's been tidied up significantly.
\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "113324": "##Can we determine clinical applicability?\r\n\r\nThis year’s competition was intended to catalyze a change in cardiac diagnostics, so connecting the competition participants and the medical community is an essential part of the DSB. I have done some preliminary analysis of the Data Science Bowl’s (DSB) top 4 team submissions. The goal is to present the results in terms that are meaningful to the medical research community. In doing so I hope to spark a dialog between the communities. \r\n\r\nDSB competitors produced diastolic and systolic volume cumulative probability distribution function (CDF) for each heart. Researchers, clinicians, and technicians will prefer an algorithm to output volumes, ejection fraction and a precision estimate for those predictions. Obtaining those quantities from the submission files requires some level of approximation. I will describe what was done for this post, noting where I made simplifying assumptions. The plots and visualizations presented here are valuable for understanding the significant of the results. Many of the plots were used because they appear in prior medical research on reliability of cardiac MRI. \r\n\r\nPlease feel free to comment on, add to, and critique the approach outlined in this post. I have uploaded the script I used to generate the plots, and the solutions used to score the Private Leaderboard are live now.\r\n\r\nDr. Andrew Arai of NIH provides a medical perspective on these results in another post, [here][1].\r\n\r\n##Calculating a Prediction\r\nThe DSB challenge was to use the images from a heart study to generate two cumulative probability distribution functions (CDF)—one for the end-systolic volume (SV) and one for the end-diastolic volume (DV). Those volumes are the left ventricle sizes at the beginning and middle of each heartbeat. There are 600 values in each row, giving the the predicted probability that the true volume of the chamber in mL is less than the value index (i.e., if the 34th value in a row is 0.55, then the model is predicting that there is a 55% chance that the volume is less than 34 mL). So, subtracting adjacent values will give the predicted cumulative probability that the volume falls between the value indices (ie, if the 35th value is 60% and the 34th value is 55%, then the predicted probability that the volume is between 34 mL and 35 mL is 5%). Doing this subtraction for each consecutive value in a row generates a discretized probability density function (PDF):\r\n\r\n$$P(i+0.5)=CDF_{i+1}-CDF_{i}$$,\r\n\r\nwhere the value for P(v) can be interpreted as the predicted approximate probability per mL that the true volume is v. For me, the PDF is a more intuitive way to look at the model submissions. A volume that the model is predicting as likely for a heart will show up as a peak in the PDF. I am using the expectation value and the standard deviation of the PDF as the single-value volume prediction and the uncertainty estimate for that prediction respectively. Summing over the bins in the PDF, the prediction and uncertainty estimate are:\r\n\r\n$$V_{pred}=\\sum_{i=0}^{598}(i+0.5)\\times P(i) and \\sigma_{V}^2=\\sum_{i=0}^{598}[V_{pred}-(i+0.5)]^2\\times P(i)$$\r\n\r\nThe entries are already normalized, and the width of the bins are 1 mL, so I am not explicitly writing out the bin width or the normalization for these averages. Note, the PDF necessarily has one fewer bins than the CDF. This may look confusing depending on your background, but all I’m doing is finding the average and standard deviation of the PDF. \r\n\r\n\r\nIn order to give you an idea of what these CDFs look like after I turn them into PDFs, I have included a set of 100 random PDFs along with the true values, for the top four teams’ submissions. Two submissions have nearly normal distributions and the other two have some distinct non-normal features. I invite feedback and discussion on better ways to translate the CDFs into single number predictions, or any comments on why there is a better way to look at the problem of digesting model output for consumption by the medical research community. \r\n\r\n*Please see the uploaded files for the PDF comparisons. They are too big to post in-line*\r\n\r\nAnother important diagnostic metric is the ejection fraction (EF), which is calculated as \r\n\r\n$$EF=\\frac{\\mbox{Diastolic Volume}-\\mbox{Systolic Volume}}{\\mbox{Diastolic Volume}}$$.\r\n\r\nFor each heart, I simulated EF distributions by collecting EFs calculated from 10,000 SV-DV pairs sampled from the SV and DV PDFs. I then took the expectation value of the simulated EF distribution as the model’s EF prediction. I verified over a subset of heart studies that the sample size I chose produces a stable EF expectation value. \r\n\r\n##Limits of My Approach\r\n\r\nThis overall approach requires some caveats. Using the expectation value as the “prediction” is only reasonable for PDFs that are well behaved and unimodal. In reality a model could output a prediction that gives high probability to disparate systolic (or diastolic) volumes, in which case, the expectation value of SV (or DV) may be inappropriate. Estimating the EF distribution from the SV and DV distributions may also be problematic because the SV and DV are clearly correlated. This simplest solution is to seek a model that directly produces the EF value. \r\n\r\nMy overall approach will provide, however, an approximation of results that can be used to start a conversation with the medical community. We will eventually need to work with the medical community to develop a more rigorous analysis that takes into account some knowledge of the models. The medical community and the data science community must combine their effort to develop a method to rigorously translate model output into metrics that can be used for clinical evaluation and ultimately clinical use. \r\n\r\n##Getting Clinical\r\n\r\nThe table below shows root-mean-square (RMS) errors for the diastolic and systolic volumes and the ejection fraction (EF) from each of the top 4 teams. Remember, this is the RMS error value for the difference between the test value and the single-value prediction calculation. For some clinical perspective on these errors, the precision on human clinical determination of diastolic volume, systolic volume, and ejection fraction is about 13 mL, 14mL, and 6% respectively. Consider those benchmarks hearsay for the moment. I will edit the post with numbers from the literature and some references ASAP. With those benchmarks in mind, the RMS errors look promising. \r\n\r\n![RMS Error Table][2]\r\n\r\nWe can glean more about the potential clinical applicability of the solutions than just a RMS error. Rather than approaching this only from a model evaluation standpoint, let’s return to the original motivation of the challenge—diagnosing heart disease. \r\n\r\nAs noted about, the quantity of interest for many diagnoses is the ejection fraction (EF). Roughly speaking, an EF under 35% is a dire emergency, around 60% is normal, and above 73% is considered hyperactive. I calculated the confusion (a.k.a. contingency) matrices for the top 4 competitors using diagnostic EF bins provided by Dr. Andrew Arai. If you take a close look at the matrices, you can get a picture of what might happen in a medical setting if these algorithms were put to use. \r\n\r\n![Contingency Matrices][3]\r\n\r\nThe models keep the diagnosis categories pretty tightly grouped together. While the models are not right 100% the time, there is a very low probability of a severely abnormal EF being incorrectly categorized in the mild to *hyperdynamic* range. The normal to mild diagnoses are very likely to stay within their domain of the matrix. This is a pretty good sign pointing towards suitability for clinical applications.\r\n\r\nOther metrics that are used to evaluate clinical measurement techniques are correlation plots and Bland-Altman (BA) plots. The correlation plots are straight forward linear fits to the scatter pot of the true values versus the predicted data. For the BA plots, $$V_{pred}-V_{true}$$ is plotted versus $$\\mbox{AVG}(V_{pred},V_{true})$$ for each prediction. A non-zero mean on the BA plot would indicate a relative bias between the two measurement techniques being compared (or in our case, a bias in the model, since we are assuming the test values are “true”). The dashed lines in the plot give the 95% confidence interval for the difference between the measurement types. What we would like to compare these BA plots to is other BA plots comparing two human measurements of the same data set. Is the variance in the plots comparable to human to human comparisons? Scroll to the bottom of the post for the correlation and BA plots. \r\n\r\n##Does it know when it’s failing?\r\n\r\nWe are especially interested in producing reasonable confidence estimates around volume and ejection fraction predictions. Michael Hansen, Ph.D, co-PI for this DSB, noted that the best models can fail, but they should “fail loudly.” We’d like a good measure of confidence for each individual prediction. In other words, “How well does the model estimate its own accuracy?” Ideally, if an individual prediction has a high probability of being wrong, a model will flag that prediction as uncertain. As mentioned before, I’m using the PDF standard deviation as an estimate of the model’s prediction certainty. A sharp peak indicates higher confidence whereas a wide peak indicates greater uncertainty. \r\n\r\nThe plot below is an illustration of a way to visualize a model’s ability to estimate its own accuracy. For each prediction, the standard deviation of the PDF is plotted versus the error of that prediction. The structure of a plot can be conceptualized as shown in the below graphic. Predictions that fall in the bottom left quadrant would be very confident (small width PDF) and accurate (small absolute error). Points in the top right would be low confidence (wide PDF) and inaccurate (large absolute error). Neither classes of points are necessarily bad. If all the predictions for a model fall within the two aforementioned quadrants, one would know when to trust the algorithm’s output and when to consider retaking a measurement. However, points in the other quadrants (large error but purported accuracy) could confuse one’s ability to rely on the algorithm’s uncertainty estimate. \r\n\r\n![Quadrant Description][4]\r\n\r\nIn the plots that follow, the majority of the data points are shown in grey. The 5% best and worst errors are highlighted as well as the 5% widest and thinnest PDFs. I have looked at this a couple of ways, and the model error and “confidence” distributions are not obviously correlated. For that reason, I suspect that there is a better way to evaluate the models’ estimation of its own accuracy. \r\n\r\n![Prediction Error and Associated ST DEV][5]\r\n\r\n##In Closing\r\n\r\nHopefully, there will be an ongoing conversation between the medical and data science communities regarding the evaluation of these models for clinical use. Are there other metrics that would be useful? What else do you want to know about these models? What information from the model developers would help inform decisions that would be made in a clinical setting? From the data science community, is there more useful information that can be derived from the submission data? Are there better ways to estimate model prediction confidence? Are there better ways to calculate the prediction value? \r\n\r\nThis post is a twin of the post on the DSB, [which you can read here.][6]\r\n\r\n\r\n![TenciaWoshialex BA][7]\r\n![kunsthard BA][8]\r\n![Julian de Wit BA][9]\r\n![Show me the Money BA][10]\r\n\r\n![TenciaWoshialex Corr][11]\r\n![kunsthart Corr][12]\r\n![Julian de Wit Corr][13]\r\n![Show me the Money Corr][14]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/19839/a-medical-perspective-on-the-quality-of-the-left-ventricular-volume-and\r\n [2]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30134519/errorTable.png\r\n [3]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100556/ContingencyMatrix.png\r\n [4]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/28193219/error-on-prediction.png\r\n [5]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/28193219/individual-prediction-error.png\r\n [6]: http://www.datasciencebowl.com/leading-and-winning-team-submissions-analysis/\r\n [7]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100610/Tencia_n_WoshialexBlandAltman.png\r\n [8]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100604/kunsthartBlandAltman.png\r\n [9]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100559/Julian_de_WitBlandAltman.png\r\n [10]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100608/show_me_the_moneyBlandAltman.png\r\n [11]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100612/Tencia_n_WoshialexCorr.png\r\n [12]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100605/kunsthartCorr.png\r\n [13]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100602/Julian_de_WitCorr.png\r\n [14]: https://s3.amazonaws.com/datasciencebowl.com/wp-content/uploads/2016/03/30100609/show_me_the_moneyCorr.png", "113743": "I've created a repository for this script, and it's been tidied up significantly.\r\n\r\n [The link for the repository][1]\r\n\r\n\r\n [1]: https://github.com/jonrmulholland/dsbAnalysis" }, "source": "meta" }