Hello here a quick summary of my solution.\nThis thing was that I used only one model. This was both cool and stupid at the same time..
\n\nThe basic gist:
\n\nScale images to real sizes using provided pixel areas in dicom. Crop 180x180 center. Use CLAHE on the images to get good local contrast.
\n\nLabel all train patients frame 1 and frame 12. I already had a lean-and-mean labeling tool that allowed me to label very fast. The big problem was that I did not know how to label. I tried to label as consistently as possible and then later on adjust systematic errors with a calibration step.
\n\nUse a U-net for pixel segmentation (LV yes/no).\nhttp://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/\nI consider this architecture the state of the art in pixel segmentation.\nI implemented the U-net in MXNET which was a breeze to work with. big recommendation. Everything in the U-net paper was useful except that I was lazy with the weight initialization and relied on Batch Normalization.
\n\nAfter the pixel segmentation I counted the LV pixels and integrated over the slices.\nThis is easier said then done since many slices were missing. Slice location was not 100% dependable. Order etc etc.\nA lot of work had gone in slice management. Pandas was invaluable for this.
\n\nAfter I had the predictions I did a calibration step to work out systematic labeling errors. I did this by regressing over the residuals with a gradient booster with some features \n like age, sex etc.
\n\nUse the stdev in the errors (sliding window over heart size) and plug that in a CDF to generate predictions.
\n\nPerformance was only limited by the following issues :
\n- I did not know how to label. I'm not a doctor.
\n- Some labels were provided wrongly (429!!)
\n- Corrupt scans/slices.
MAE in ml was roughly 9 ml.
\n\nKaggle and Booz Allen Hamilton for this cool challenge.
\nMxnet
\nThe authors of the U-net paper..
\nSander Dieleman for pointing it out in reddit.
Leustagos advised me to join a team to get more different models so that I could maybe push for the win. Because big ego and my blind confidence in my single model I continued on my own. You will not believe how many times I banged myself on the head last week when I suddenly dropped to #16.\nIn the end it I'm glad it all turned out well but in hindsight you were so right!
", "messages": [ { "id": "111540", "postDate": "03/15/2016 09:06:28", "content": "Hello here a quick summary of my solution.\nThis thing was that I used only one model. This was both cool and stupid at the same time..
\n\nThe basic gist:
\n\nScale images to real sizes using provided pixel areas in dicom. Crop 180x180 center. Use CLAHE on the images to get good local contrast.
\n\nLabel all train patients frame 1 and frame 12. I already had a lean-and-mean labeling tool that allowed me to label very fast. The big problem was that I did not know how to label. I tried to label as consistently as possible and then later on adjust systematic errors with a calibration step.
\n\nUse a U-net for pixel segmentation (LV yes/no).\nhttp://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/\nI consider this architecture the state of the art in pixel segmentation.\nI implemented the U-net in MXNET which was a breeze to work with. big recommendation. Everything in the U-net paper was useful except that I was lazy with the weight initialization and relied on Batch Normalization.
\n\nAfter the pixel segmentation I counted the LV pixels and integrated over the slices.\nThis is easier said then done since many slices were missing. Slice location was not 100% dependable. Order etc etc.\nA lot of work had gone in slice management. Pandas was invaluable for this.
\n\nAfter I had the predictions I did a calibration step to work out systematic labeling errors. I did this by regressing over the residuals with a gradient booster with some features \n like age, sex etc.
\n\nUse the stdev in the errors (sliding window over heart size) and plug that in a CDF to generate predictions.
\n\nPerformance was only limited by the following issues :
\n- I did not know how to label. I'm not a doctor.
\n- Some labels were provided wrongly (429!!)
\n- Corrupt scans/slices.
MAE in ml was roughly 9 ml.
\n\nKaggle and Booz Allen Hamilton for this cool challenge.
\nMxnet
\nThe authors of the U-net paper..
\nSander Dieleman for pointing it out in reddit.
Leustagos advised me to join a team to get more different models so that I could maybe push for the win. Because big ego and my blind confidence in my single model I continued on my own. You will not believe how many times I banged myself on the head last week when I suddenly dropped to #16.\nIn the end it I'm glad it all turned out well but in hindsight you were so right!
", "rawMarkdown": "Hello here a quick summary of my solution.\r\nThis thing was that I used only **one** model. This was both cool and stupid at the same time..\r\n\r\nThe basic gist:\r\n\r\n 1. Preprocessing\r\n\r\nScale images to real sizes using provided pixel areas in dicom. Crop 180x180 center. Use CLAHE on the images to get good local contrast.\r\n\r\n 2. Hand labeling\r\n\r\nLabel all train patients frame 1 and frame 12. I already had a lean-and-mean labeling tool that allowed me to label very fast. The big problem was that I did not know how to label. I tried to label as consistently as possible and then later on adjust systematic errors with a calibration step.\r\n\r\n 3. Pixelwise segmentation with U-net\r\n\r\nUse a U-net for pixel segmentation (LV yes/no).\r\nhttp://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/\r\nI consider this architecture the state of the art in pixel segmentation.\r\nI implemented the U-net in MXNET which was a breeze to work with. big recommendation. Everything in the U-net paper was useful except that I was lazy with the weight initialization and relied on Batch Normalization.\r\n\r\n4. Integration to a volume\r\n\r\nAfter the pixel segmentation I counted the LV pixels and integrated over the slices.\r\nThis is easier said then done since many slices were missing. Slice location was not 100% dependable. Order etc etc.\r\nA lot of work had gone in slice management. Pandas was invaluable for this.\r\n\r\n 5. Calibration\r\n\r\nAfter I had the predictions I did a calibration step to work out systematic labeling errors. I did this by regressing over the residuals with a gradient booster with some features \r\n like age, sex etc.\r\n\r\n6. Submission\r\n\r\nUse the stdev in the errors (sliding window over heart size) and plug that in a CDF to generate predictions.\r\n\r\n7. Conclusion\r\n\r\nPerformance was only limited by the following issues :Super cool! congratulations!
", "rawMarkdown": "Super cool! congratulations!", "votes": null }, { "id": "111569", "postDate": "03/15/2016 13:52:27", "content": "Julian,
\n\nNice and clean single models are more valuable to the community. It seems like magic to me such high score can be achieved with so few submissions. Congratulations!
\n\nDo you mind opening your U-net implementation with MXNET?
", "rawMarkdown": "Julian,\r\n\r\nNice and clean single models are more valuable to the community. It seems like magic to me such high score can be achieved with so few submissions. Congratulations!\r\n\r\nDo you mind opening your U-net implementation with MXNET?", "votes": null }, { "id": "111574", "postDate": "03/15/2016 14:19:15", "content": "Wei Dong,\nYes I must publish the source code to win the prize I guess.
\n\nBut I have two other goals. \nThe first is to do something with my hand drawn labels + images. Perhaps make it a public dataset.\nI do not know how to go about this and of course they are not 100% correct.\nBut it's bigger and more challenging dataset than sunnybrook.
\n\nThe second goal is to make an even more concise example and, if allowed, add it to the MxNet examples.
", "rawMarkdown": "Wei Dong,\r\nYes I must publish the source code to win the prize I guess.\r\n\r\nBut I have two other goals. \r\nThe first is to do something with my hand drawn labels + images. Perhaps make it a public dataset.\r\nI do not know how to go about this and of course they are not 100% correct.\r\nBut it's bigger and more challenging dataset than sunnybrook.\r\n\r\nThe second goal is to make an even more concise example and, if allowed, add it to the MxNet examples.", "votes": null }, { "id": "111575", "postDate": "03/15/2016 14:19:21", "content": "hnn, I feel the other way that in hindsight it was good that you did not merge with another team...\nIf you did merge with team #2 you'll split the prize with 5 people (so the prize you get is the same)... and it is not 100% sure that your team can get to #1..\nand if you merged with team #3, you probably only be able to get the second position and the amount money you get is still the same...\nAlso, you could have win #2 if you are a little bit luckier... and you have so few submissions so your results were very solid without overfitting to the leaderboard.
", "rawMarkdown": "hnn, I feel the other way that in hindsight it was good that you did not merge with another team...\r\nIf you did merge with team #2 you'll split the prize with 5 people (so the prize you get is the same)... and it is not 100% sure that your team can get to #1..\r\nand if you merged with team #3, you probably only be able to get the second position and the amount money you get is still the same...\r\nAlso, you could have win #2 if you are a little bit luckier... and you have so few submissions so your results were very solid without overfitting to the leaderboard.", "votes": null }, { "id": "111577", "postDate": "03/15/2016 14:28:03", "content": "@woshialex,\nFirst of all congratulations of course.\nYes I did notice (like you) that simple ensembling did not add much.\nThe model that was best at picking up the outliers did best on it's own.
\n\nBut I thought in hindsight that the easy patients could be more accurate with more models.\nWith the pixel based segmentation I had a good measure on the uncertainty of the prediction so I knew which patients were easy. Also, last week I though that other models benefitted more from the 200 extra patients. Anyway.. it's all good now.
", "rawMarkdown": "woshialex,\r\nFirst of all congratulations of course.\r\nYes I did notice (like you) that simple ensembling did not add much.\r\nThe model that was best at picking up the outliers did best on it's own.\r\n\r\nBut I thought in hindsight that the easy patients could be more accurate with more models.\r\nWith the pixel based segmentation I had a good measure on the uncertainty of the prediction so I knew which patients were easy. Also, last week I though that other models benefitted more from the 200 extra patients. Anyway.. it's all good now.", "votes": null }, { "id": "111590", "postDate": "03/15/2016 16:11:45", "content": "@Julian
\n\nYou are more than welcome to push it into MXNet example! Congratulations again!
", "rawMarkdown": "Julian\r\n\r\nYou are more than welcome to push it into MXNet example! Congratulations again!", "votes": null }, { "id": "111627", "postDate": "03/15/2016 19:51:49", "content": "@julian congrats! You really deserved it! Im glad you made it anyway.
", "rawMarkdown": "julian congrats! You really deserved it! Im glad you made it anyway.", "votes": null }, { "id": "111630", "postDate": "03/15/2016 19:56:55", "content": "Congratulations Julian, very good solution!
", "rawMarkdown": "Congratulations Julian, very good solution!", "votes": null }, { "id": "111654", "postDate": "03/15/2016 22:35:31", "content": "What was the total computational time required to train your final model? What about making predictions from it on the test set?
", "rawMarkdown": "What was the total computational time required to train your final model? What about making predictions from it on the test set?", "votes": null }, { "id": "111692", "postDate": "03/16/2016 06:23:16", "content": "@Ben hammer,
\nI used 5 folds for training to not overfit the calibration and the average 5 translations in predictions.\nBoth are not strictly necessary.
Training for one full set (no folds) was around 5 hours.
\n\nPrediction was around 1 hour for 1140 patients (+/- 3 secs per patient).
\n\nPrediction took so long since every patient has 30 frames and +/- 10 slices which comes down to roughly 342000 images on total that need to be segmented.
", "rawMarkdown": "Ben hammer,Hi, we were doing this as a training purpose. First using only dicom data and then using images.
\n\nWe were having problems using directly the binary masks of LV with the same size of input grayscale image to do the neural network training cause we got 100% error with different nets. Are you using directly the binary masks vector? or are you using a contour? or the xy coordinates of the boundaries of the contour?
\n\nif you have a simple code example it would be useful. We were doing the task on Matlab.
", "rawMarkdown": "Hi, we were doing this as a training purpose. First using only dicom data and then using images. \r\n\r\nWe were having problems using directly the binary masks of LV with the same size of input grayscale image to do the neural network training cause we got 100% error with different nets. Are you using directly the binary masks vector? or are you using a contour? or the xy coordinates of the boundaries of the contour?\r\n\r\nif you have a simple code example it would be useful. We were doing the task on Matlab.", "votes": null }, { "id": "112136", "postDate": "03/18/2016 10:05:26", "content": "@liveflow,\nI also used a binary mask image as the target.
\n\nHowever, without good measures this can be very numerically unstable.\nFor instance, when most of the pixels in the target are zero, the net wants to predict all '0' very quickly and does not recover. Mostly I saw my activations explode to float.nan.
\n\nA few important measures were:
\n\n@Julian de Wit
\n\nSorry for self-PR, but I suggest you to try LSUV-init. It solves the most problems with initialization :) \nhttp://arxiv.org/abs/1511.06422
", "rawMarkdown": "Julian de Wit\r\n\r\nSorry for self-PR, but I suggest you to try LSUV-init. It solves the most problems with initialization :) \r\nhttp://arxiv.org/abs/1511.06422", "votes": null }, { "id": "112613", "postDate": "03/22/2016 11:10:27", "content": "Nothing wrong with a little PR :)
\n\nWill take a look.. Hard to keep up.
", "rawMarkdown": "Nothing wrong with a little PR :)\r\n\r\nWill take a look.. Hard to keep up.", "votes": null } ], "comments": [ { "id": 111554, "author_name": "binghsu", "author_url": "", "post_date": "03/15/2016 11:51:30", "content": "Super cool! congratulations!
", "votes": null, "replies": [] }, { "id": 111569, "author_name": "aaalgo", "author_url": "", "post_date": "03/15/2016 13:52:27", "content": "Julian,
\n\nNice and clean single models are more valuable to the community. It seems like magic to me such high score can be achieved with so few submissions. Congratulations!
\n\nDo you mind opening your U-net implementation with MXNET?
", "votes": null, "replies": [] }, { "id": 111574, "author_name": "juliandewit", "author_url": "", "post_date": "03/15/2016 14:19:15", "content": "Wei Dong,\nYes I must publish the source code to win the prize I guess.
\n\nBut I have two other goals. \nThe first is to do something with my hand drawn labels + images. Perhaps make it a public dataset.\nI do not know how to go about this and of course they are not 100% correct.\nBut it's bigger and more challenging dataset than sunnybrook.
\n\nThe second goal is to make an even more concise example and, if allowed, add it to the MxNet examples.
", "votes": null, "replies": [] }, { "id": 111575, "author_name": "woshialex", "author_url": "", "post_date": "03/15/2016 14:19:21", "content": "hnn, I feel the other way that in hindsight it was good that you did not merge with another team...\nIf you did merge with team #2 you'll split the prize with 5 people (so the prize you get is the same)... and it is not 100% sure that your team can get to #1..\nand if you merged with team #3, you probably only be able to get the second position and the amount money you get is still the same...\nAlso, you could have win #2 if you are a little bit luckier... and you have so few submissions so your results were very solid without overfitting to the leaderboard.
", "votes": null, "replies": [] }, { "id": 111577, "author_name": "juliandewit", "author_url": "", "post_date": "03/15/2016 14:28:03", "content": "@woshialex,\nFirst of all congratulations of course.\nYes I did notice (like you) that simple ensembling did not add much.\nThe model that was best at picking up the outliers did best on it's own.
\n\nBut I thought in hindsight that the easy patients could be more accurate with more models.\nWith the pixel based segmentation I had a good measure on the uncertainty of the prediction so I knew which patients were easy. Also, last week I though that other models benefitted more from the 200 extra patients. Anyway.. it's all good now.
", "votes": null, "replies": [] }, { "id": 111590, "author_name": "binghsu", "author_url": "", "post_date": "03/15/2016 16:11:45", "content": "@Julian
\n\nYou are more than welcome to push it into MXNet example! Congratulations again!
", "votes": null, "replies": [] }, { "id": 111627, "author_name": "leustagos", "author_url": "", "post_date": "03/15/2016 19:51:49", "content": "@julian congrats! You really deserved it! Im glad you made it anyway.
", "votes": null, "replies": [] }, { "id": 111630, "author_name": "titericz", "author_url": "", "post_date": "03/15/2016 19:56:55", "content": "Congratulations Julian, very good solution!
", "votes": null, "replies": [] }, { "id": 111654, "author_name": "benhamner", "author_url": "", "post_date": "03/15/2016 22:35:31", "content": "What was the total computational time required to train your final model? What about making predictions from it on the test set?
", "votes": null, "replies": [] }, { "id": 111692, "author_name": "juliandewit", "author_url": "", "post_date": "03/16/2016 06:23:16", "content": "@Ben hammer,
\nI used 5 folds for training to not overfit the calibration and the average 5 translations in predictions.\nBoth are not strictly necessary.
Training for one full set (no folds) was around 5 hours.
\n\nPrediction was around 1 hour for 1140 patients (+/- 3 secs per patient).
\n\nPrediction took so long since every patient has 30 frames and +/- 10 slices which comes down to roughly 342000 images on total that need to be segmented.
", "votes": null, "replies": [] }, { "id": 112043, "author_name": "liveflow", "author_url": "", "post_date": "03/17/2016 22:35:01", "content": "Hi, we were doing this as a training purpose. First using only dicom data and then using images.
\n\nWe were having problems using directly the binary masks of LV with the same size of input grayscale image to do the neural network training cause we got 100% error with different nets. Are you using directly the binary masks vector? or are you using a contour? or the xy coordinates of the boundaries of the contour?
\n\nif you have a simple code example it would be useful. We were doing the task on Matlab.
", "votes": null, "replies": [] }, { "id": 112136, "author_name": "juliandewit", "author_url": "", "post_date": "03/18/2016 10:05:26", "content": "@liveflow,\nI also used a binary mask image as the target.
\n\nHowever, without good measures this can be very numerically unstable.\nFor instance, when most of the pixels in the target are zero, the net wants to predict all '0' very quickly and does not recover. Mostly I saw my activations explode to float.nan.
\n\nA few important measures were:
\n\n@Julian de Wit
\n\nSorry for self-PR, but I suggest you to try LSUV-init. It solves the most problems with initialization :) \nhttp://arxiv.org/abs/1511.06422
", "votes": null, "replies": [] }, { "id": 112613, "author_name": "juliandewit", "author_url": "", "post_date": "03/22/2016 11:10:27", "content": "Nothing wrong with a little PR :)
\n\nWill take a look.. Hard to keep up.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "111540": "Hello here a quick summary of my solution.\r\nThis thing was that I used only **one** model. This was both cool and stupid at the same time..\r\n\r\nThe basic gist:\r\n\r\n 1. Preprocessing\r\n\r\nScale images to real sizes using provided pixel areas in dicom. Crop 180x180 center. Use CLAHE on the images to get good local contrast.\r\n\r\n 2. Hand labeling\r\n\r\nLabel all train patients frame 1 and frame 12. I already had a lean-and-mean labeling tool that allowed me to label very fast. The big problem was that I did not know how to label. I tried to label as consistently as possible and then later on adjust systematic errors with a calibration step.\r\n\r\n 3. Pixelwise segmentation with U-net\r\n\r\nUse a U-net for pixel segmentation (LV yes/no).\r\nhttp://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/\r\nI consider this architecture the state of the art in pixel segmentation.\r\nI implemented the U-net in MXNET which was a breeze to work with. big recommendation. Everything in the U-net paper was useful except that I was lazy with the weight initialization and relied on Batch Normalization.\r\n\r\n4. Integration to a volume\r\n\r\nAfter the pixel segmentation I counted the LV pixels and integrated over the slices.\r\nThis is easier said then done since many slices were missing. Slice location was not 100% dependable. Order etc etc.\r\nA lot of work had gone in slice management. Pandas was invaluable for this.\r\n\r\n 5. Calibration\r\n\r\nAfter I had the predictions I did a calibration step to work out systematic labeling errors. I did this by regressing over the residuals with a gradient booster with some features \r\n like age, sex etc.\r\n\r\n6. Submission\r\n\r\nUse the stdev in the errors (sliding window over heart size) and plug that in a CDF to generate predictions.\r\n\r\n7. Conclusion\r\n\r\nPerformance was only limited by the following issues :