Hey everyone,
\n\nFirst of, congratulations to the winners! You did an awesome job, and we still haven't figured out how to get our scores that low! We are looking forward to hearing how you did it. :-)
\n\nHere's a blog post describing our approach in some detail: http://irakorshunova.github.io/2016/03/15/heart.html and http://317070.github.io/heart/
\n\nCode and documentation to reproduce our solution will follow soon!
\n\nEdit: the code and documentation of our solution: https://github.com/317070/kaggle-heart
", "messages": [ { "id": "111493", "postDate": "03/15/2016 00:18:12", "content": "Hey everyone,
\n\nFirst of, congratulations to the winners! You did an awesome job, and we still haven't figured out how to get our scores that low! We are looking forward to hearing how you did it. :-)
\n\nHere's a blog post describing our approach in some detail: http://irakorshunova.github.io/2016/03/15/heart.html and http://317070.github.io/heart/
\n\nCode and documentation to reproduce our solution will follow soon!
\n\nEdit: the code and documentation of our solution: https://github.com/317070/kaggle-heart
", "rawMarkdown": "Hey everyone,\r\n\r\nFirst of, congratulations to the winners! You did an awesome job, and we still haven't figured out how to get our scores that low! We are looking forward to hearing how you did it. :-)\r\n\r\nHere's a blog post describing our approach in some detail: http://irakorshunova.github.io/2016/03/15/heart.html and http://317070.github.io/heart/\r\n\r\nCode and documentation to reproduce our solution will follow soon!\r\n\r\nEdit: the code and documentation of our solution: https://github.com/317070/kaggle-heart", "votes": null }, { "id": "111494", "postDate": "03/15/2016 00:23:49", "content": "big congratulations, will study it.
\n\nit's really smart to use the cross to find the center.
", "rawMarkdown": "big congratulations, will study it.\r\n\r\nit's really smart to use the cross to find the center.", "votes": null }, { "id": "111495", "postDate": "03/15/2016 00:25:41", "content": "Wow, what large network :O
", "rawMarkdown": "Wow, what large network :O", "votes": null }, { "id": "111497", "postDate": "03/15/2016 00:31:01", "content": "How did the idea? :D
\n\nIt was trial and error?
", "rawMarkdown": "How did the idea? :D\r\n\r\nIt was trial and error?", "votes": null }, { "id": "111498", "postDate": "03/15/2016 00:36:03", "content": "Thanks for sharing your approach.
", "rawMarkdown": "Thanks for sharing your approach.", "votes": null }, { "id": "111499", "postDate": "03/15/2016 00:46:41", "content": "Congrats! Thanks for posting. Your second link doesn't seem to work for me. http://irakorshunova.github.io/2015/03/15/heart.html
\n\nEdit: fixed, thanks
", "rawMarkdown": "Congrats! Thanks for posting. Your second link doesn't seem to work for me. http://irakorshunova.github.io/2015/03/15/heart.html\r\n\r\nEdit: fixed, thanks", "votes": null }, { "id": "111500", "postDate": "03/15/2016 00:47:45", "content": "[quote=Beyond Two Layers;111497]How did the idea? :D\nIt was trial and error?\n[/quote]\nQuote from Yann LeCun:\nLarger networks tend to work better. Make your network bigger and bigger until the accuracy stops increasing. Then regularize the hell out of it. Then make it bigger still and pre-train it (...).\nhttp://fastml.com/yann-lecuns-answers-from-the-reddit-ama/
", "rawMarkdown": "[quote=Beyond Two Layers;111497]How did the idea? :D\r\nIt was trial and error?\r\n[/quote]\r\nQuote from Yann LeCun:\r\n*Larger networks tend to work better. Make your network bigger and bigger until the accuracy stops increasing. Then regularize the hell out of it. Then make it bigger still and pre-train it (...).*\r\nhttp://fastml.com/yann-lecuns-answers-from-the-reddit-ama/", "votes": null }, { "id": "111505", "postDate": "03/15/2016 01:29:08", "content": "may i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? looking forward to the code though, must be a very good piece of study material
", "rawMarkdown": "may i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? looking forward to the code though, must be a very good piece of study material", "votes": null }, { "id": "111506", "postDate": "03/15/2016 01:31:13", "content": "Thank you for posting, really interesting approach and very different from ours. We used a fully convolutional network for segmentation, as in the deep learning tutorial, with the Sunnybrook data as the main NN training set. This was then combined with a few other, simpler models. I think we will have a blog post up eventually too :)
", "rawMarkdown": "Thank you for posting, really interesting approach and very different from ours. We used a fully convolutional network for segmentation, as in the deep learning tutorial, with the Sunnybrook data as the main NN training set. This was then combined with a few other, simpler models. I think we will have a blog post up eventually too :)", "votes": null }, { "id": "111507", "postDate": "03/15/2016 01:35:00", "content": "Similar to my spider architecture too bad I didn't use it. I could only afford 3 conv layers in each branch of a 15 branch network because of memory issues. My question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.
", "rawMarkdown": "Similar to my spider architecture too bad I didn't use it. I could only afford 3 conv layers in each branch of a 15 branch network because of memory issues. My question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.", "votes": null }, { "id": "111509", "postDate": "03/15/2016 01:49:44", "content": "The way you found ROI centers is brilliant. If you consider this to be part of data cleaning, then perhaps it was smart, after all, to give us raw data.
", "rawMarkdown": "The way you found ROI centers is brilliant. If you consider this to be part of data cleaning, then perhaps it was smart, after all, to give us raw data.", "votes": null }, { "id": "111512", "postDate": "03/15/2016 02:04:51", "content": "It's similar to how the Fourier tutorial finds region of interest. The Fourier tutorial actually averages all the images in a slice into a single image then finds roi based on all slices. I would like to stack these roi images on top of each other ordered by slice location, Then run 3D convolutions. This gives a complete view of a patients heart from all slices summarized into a smaller training set.
", "rawMarkdown": "It's similar to how the Fourier tutorial finds region of interest. The Fourier tutorial actually averages all the images in a slice into a single image then finds roi based on all slices. I would like to stack these roi images on top of each other ordered by slice location, Then run 3D convolutions. This gives a complete view of a patients heart from all slices summarized into a smaller training set.", "votes": null }, { "id": "111513", "postDate": "03/15/2016 02:32:06", "content": "I'll be the Devils advocate and say I hope Kaggle is not playing favorites here, you guys were not on the radar in the 1st round and currently you are 24th, any yet you won 2nd place. Not hating, just saying :) granted you won the last dsb and you have an interesting architecture.
\n\nEdit: my bad didn't know you are part of kunsthart, you were on the radar.
", "rawMarkdown": "I'll be the Devils advocate and say I hope Kaggle is not playing favorites here, you guys were not on the radar in the 1st round and currently you are 24th, any yet you won 2nd place. Not hating, just saying :) granted you won the last dsb and you have an interesting architecture. \r\n\r\nEdit: my bad didn't know you are part of kunsthart, you were on the radar.", "votes": null }, { "id": "111545", "postDate": "03/15/2016 10:34:19", "content": "[quote=Yuanfang Guan;111505]\nmay i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? \n[/quote]
\n\nIn total we trained about 250 different models. Most of these could be subdivided into the 4 broad categories we mention in our blog post (slice model / patient model / 2ch / 4ch). Although we only mentioned the architecture of our single best models, we tried out a lot of variants with both major and minor differences to get to that point, and some of those proved to be useful in the end.
", "rawMarkdown": "[quote=Yuanfang Guan;111505]\r\nmay i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? \r\n[/quote]\r\n\r\nIn total we trained about 250 different models. Most of these could be subdivided into the 4 broad categories we mention in our blog post (slice model / patient model / 2ch / 4ch). Although we only mentioned the architecture of our single best models, we tried out a lot of variants with both major and minor differences to get to that point, and some of those proved to be useful in the end.", "votes": null }, { "id": "111546", "postDate": "03/15/2016 10:45:43", "content": "[quote=DavidGbodiOdaibo;111507]\nMy question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.\n[/quote]
\n\nWe trained our models on the NVIDIA GPUs that we have in the lab, which include GTX TITAN X, GTX 980, GTX 680 and Tesla K40 cards.
\n\nFeeding the slices in order is important for the truncated code approximation to work. It is not invariant to it.
", "rawMarkdown": "[quote=DavidGbodiOdaibo;111507]\r\nMy question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.\r\n[/quote]\r\n\r\nWe trained our models on the NVIDIA GPUs that we have in the lab, which include GTX TITAN X, GTX 980, GTX 680 and Tesla K40 cards. \r\n\r\nFeeding the slices in order is important for the truncated code approximation to work. It is not invariant to it.", "votes": null }, { "id": "111655", "postDate": "03/15/2016 22:35:57", "content": "What was the total computational time required to train your final model? What about making predictions from it on the test set?
", "rawMarkdown": "What was the total computational time required to train your final model? What about making predictions from it on the test set?", "votes": null }, { "id": "111712", "postDate": "03/16/2016 09:49:16", "content": "[quote=Ben Hamner;111655]What was the total computational time required to train your final model? What about making predictions from it on the test set?[/quote]\nWe didn't separate the two, but it took us 6 days to retrain everything twice. So about 3 days per submission. However, we needed 14 GPU's to stay within that time frame.
\n\nOn a Titan X, one model would typically train in 4h for the smallest models, up to 36h for the slowest, also depending on the speed of reading from disk (which was the bottleneck on most of our machines). Generating predictions would take about 2h per model, since we predict every patient about 200 times before averaging.
\n\nHowever, a single model without test time augmentation could probably predict the entire set of 440 patients in about 1 minute with a loss of ~0.0107.
\n\nMost of the computation was to make it more robust, since it had to run on an unknown dataset, with maybe unknown data problems. In the end, it didn't look entirely necessary though.
", "rawMarkdown": "[quote=Ben Hamner;111655]What was the total computational time required to train your final model? What about making predictions from it on the test set?[/quote]\r\nWe didn't separate the two, but it took us 6 days to retrain everything twice. So about 3 days per submission. However, we needed 14 GPU's to stay within that time frame.\r\n\r\nOn a Titan X, one model would typically train in 4h for the smallest models, up to 36h for the slowest, also depending on the speed of reading from disk (which was the bottleneck on most of our machines). Generating predictions would take about 2h per model, since we predict every patient about 200 times before averaging.\r\n\r\nHowever, a single model without test time augmentation could probably predict the entire set of 440 patients in about 1 minute with a loss of ~0.0107.\r\n\r\nMost of the computation was to make it more robust, since it had to run on an unknown dataset, with maybe unknown data problems. In the end, it didn't look entirely necessary though.", "votes": null }, { "id": "111778", "postDate": "03/16/2016 18:27:17", "content": "Awesome guys, great work! In the README in your repo, you say you "reckon this data too dirty to do meaningful extrapolations for medical applications". Which specific features of the data do you think make it unsuitable for medical applications? What changes do you think would have to be made to the dataset for it to be "suitable" for medical applications?
", "rawMarkdown": "Awesome guys, great work! In the README in your repo, you say you \"reckon this data too dirty to do meaningful extrapolations for medical applications\". Which specific features of the data do you think make it unsuitable for medical applications? What changes do you think would have to be made to the dataset for it to be \"suitable\" for medical applications?", "votes": null }, { "id": "111916", "postDate": "03/17/2016 10:05:19", "content": "I think the big issue is that it can be pretty wrong for some patients. The result you see on the leaderboard is only an average error, but it is because most are very good and some are very bad outliers.
\n\nThis is typically NOT what you'd want for a medical application.
\n\nThe dataset would need to be cleaner. I reckon however that when the radiologists using this get clear instructions on what data is needed, it can become suitable. Especially since the algorithms seem to be reasonably good at telling when it doesn't know for sure.
\n\nSo, there is this other forum topic where it says "accurate to within ~10%", that's only true on average. And in medicine, averages are no good.
", "rawMarkdown": "I think the big issue is that it can be pretty wrong for some patients. The result you see on the leaderboard is only an average error, but it is because most are very good and some are very bad outliers.\r\n\r\nThis is typically NOT what you'd want for a medical application.\r\n\r\nThe dataset would need to be cleaner. I reckon however that when the radiologists using this get clear instructions on what data is needed, it can become suitable. Especially since the algorithms seem to be reasonably good at telling when it doesn't know for sure.\r\n\r\nSo, there is this other forum topic where it says \"accurate to within ~10%\", that's only true on average. And in medicine, averages are no good.", "votes": null } ], "comments": [ { "id": 111494, "author_name": "gyuanfan", "author_url": "", "post_date": "03/15/2016 00:23:49", "content": "big congratulations, will study it.
\n\nit's really smart to use the cross to find the center.
", "votes": null, "replies": [] }, { "id": 111495, "author_name": "alvaroosvaldo", "author_url": "", "post_date": "03/15/2016 00:25:41", "content": "Wow, what large network :O
", "votes": null, "replies": [] }, { "id": 111497, "author_name": "alvaroosvaldo", "author_url": "", "post_date": "03/15/2016 00:31:01", "content": "How did the idea? :D
\n\nIt was trial and error?
", "votes": null, "replies": [] }, { "id": 111498, "author_name": "niranjanmudhiraj", "author_url": "", "post_date": "03/15/2016 00:36:03", "content": "Thanks for sharing your approach.
", "votes": null, "replies": [] }, { "id": 111499, "author_name": "shannonlantzy", "author_url": "", "post_date": "03/15/2016 00:46:41", "content": "Congrats! Thanks for posting. Your second link doesn't seem to work for me. http://irakorshunova.github.io/2015/03/15/heart.html
\n\nEdit: fixed, thanks
", "votes": null, "replies": [] }, { "id": 111500, "author_name": "de317070", "author_url": "", "post_date": "03/15/2016 00:47:45", "content": "[quote=Beyond Two Layers;111497]How did the idea? :D\nIt was trial and error?\n[/quote]\nQuote from Yann LeCun:\nLarger networks tend to work better. Make your network bigger and bigger until the accuracy stops increasing. Then regularize the hell out of it. Then make it bigger still and pre-train it (...).\nhttp://fastml.com/yann-lecuns-answers-from-the-reddit-ama/
", "votes": null, "replies": [] }, { "id": 111505, "author_name": "gyuanfan", "author_url": "", "post_date": "03/15/2016 01:29:08", "content": "may i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? looking forward to the code though, must be a very good piece of study material
", "votes": null, "replies": [] }, { "id": 111506, "author_name": "tencia", "author_url": "", "post_date": "03/15/2016 01:31:13", "content": "Thank you for posting, really interesting approach and very different from ours. We used a fully convolutional network for segmentation, as in the deep learning tutorial, with the Sunnybrook data as the main NN training set. This was then combined with a few other, simpler models. I think we will have a blog post up eventually too :)
", "votes": null, "replies": [] }, { "id": 111507, "author_name": "godaibo", "author_url": "", "post_date": "03/15/2016 01:35:00", "content": "Similar to my spider architecture too bad I didn't use it. I could only afford 3 conv layers in each branch of a 15 branch network because of memory issues. My question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.
", "votes": null, "replies": [] }, { "id": 111509, "author_name": "udibr1", "author_url": "", "post_date": "03/15/2016 01:49:44", "content": "The way you found ROI centers is brilliant. If you consider this to be part of data cleaning, then perhaps it was smart, after all, to give us raw data.
", "votes": null, "replies": [] }, { "id": 111512, "author_name": "godaibo", "author_url": "", "post_date": "03/15/2016 02:04:51", "content": "It's similar to how the Fourier tutorial finds region of interest. The Fourier tutorial actually averages all the images in a slice into a single image then finds roi based on all slices. I would like to stack these roi images on top of each other ordered by slice location, Then run 3D convolutions. This gives a complete view of a patients heart from all slices summarized into a smaller training set.
", "votes": null, "replies": [] }, { "id": 111513, "author_name": "godaibo", "author_url": "", "post_date": "03/15/2016 02:32:06", "content": "I'll be the Devils advocate and say I hope Kaggle is not playing favorites here, you guys were not on the radar in the 1st round and currently you are 24th, any yet you won 2nd place. Not hating, just saying :) granted you won the last dsb and you have an interesting architecture.
\n\nEdit: my bad didn't know you are part of kunsthart, you were on the radar.
", "votes": null, "replies": [] }, { "id": 111545, "author_name": "jburms", "author_url": "", "post_date": "03/15/2016 10:34:19", "content": "[quote=Yuanfang Guan;111505]\nmay i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? \n[/quote]
\n\nIn total we trained about 250 different models. Most of these could be subdivided into the 4 broad categories we mention in our blog post (slice model / patient model / 2ch / 4ch). Although we only mentioned the architecture of our single best models, we tried out a lot of variants with both major and minor differences to get to that point, and some of those proved to be useful in the end.
", "votes": null, "replies": [] }, { "id": 111546, "author_name": "jburms", "author_url": "", "post_date": "03/15/2016 10:45:43", "content": "[quote=DavidGbodiOdaibo;111507]\nMy question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.\n[/quote]
\n\nWe trained our models on the NVIDIA GPUs that we have in the lab, which include GTX TITAN X, GTX 980, GTX 680 and Tesla K40 cards.
\n\nFeeding the slices in order is important for the truncated code approximation to work. It is not invariant to it.
", "votes": null, "replies": [] }, { "id": 111655, "author_name": "benhamner", "author_url": "", "post_date": "03/15/2016 22:35:57", "content": "What was the total computational time required to train your final model? What about making predictions from it on the test set?
", "votes": null, "replies": [] }, { "id": 111712, "author_name": "de317070", "author_url": "", "post_date": "03/16/2016 09:49:16", "content": "[quote=Ben Hamner;111655]What was the total computational time required to train your final model? What about making predictions from it on the test set?[/quote]\nWe didn't separate the two, but it took us 6 days to retrain everything twice. So about 3 days per submission. However, we needed 14 GPU's to stay within that time frame.
\n\nOn a Titan X, one model would typically train in 4h for the smallest models, up to 36h for the slowest, also depending on the speed of reading from disk (which was the bottleneck on most of our machines). Generating predictions would take about 2h per model, since we predict every patient about 200 times before averaging.
\n\nHowever, a single model without test time augmentation could probably predict the entire set of 440 patients in about 1 minute with a loss of ~0.0107.
\n\nMost of the computation was to make it more robust, since it had to run on an unknown dataset, with maybe unknown data problems. In the end, it didn't look entirely necessary though.
", "votes": null, "replies": [] }, { "id": 111778, "author_name": "kaggalex", "author_url": "", "post_date": "03/16/2016 18:27:17", "content": "Awesome guys, great work! In the README in your repo, you say you "reckon this data too dirty to do meaningful extrapolations for medical applications". Which specific features of the data do you think make it unsuitable for medical applications? What changes do you think would have to be made to the dataset for it to be "suitable" for medical applications?
", "votes": null, "replies": [] }, { "id": 111916, "author_name": "de317070", "author_url": "", "post_date": "03/17/2016 10:05:19", "content": "I think the big issue is that it can be pretty wrong for some patients. The result you see on the leaderboard is only an average error, but it is because most are very good and some are very bad outliers.
\n\nThis is typically NOT what you'd want for a medical application.
\n\nThe dataset would need to be cleaner. I reckon however that when the radiologists using this get clear instructions on what data is needed, it can become suitable. Especially since the algorithms seem to be reasonably good at telling when it doesn't know for sure.
\n\nSo, there is this other forum topic where it says "accurate to within ~10%", that's only true on average. And in medicine, averages are no good.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "111493": "Hey everyone,\r\n\r\nFirst of, congratulations to the winners! You did an awesome job, and we still haven't figured out how to get our scores that low! We are looking forward to hearing how you did it. :-)\r\n\r\nHere's a blog post describing our approach in some detail: http://irakorshunova.github.io/2016/03/15/heart.html and http://317070.github.io/heart/\r\n\r\nCode and documentation to reproduce our solution will follow soon!\r\n\r\nEdit: the code and documentation of our solution: https://github.com/317070/kaggle-heart", "111494": "big congratulations, will study it.\r\n\r\nit's really smart to use the cross to find the center.", "111495": "Wow, what large network :O", "111497": "How did the idea? :D\r\n\r\nIt was trial and error?", "111498": "Thanks for sharing your approach.", "111499": "Congrats! Thanks for posting. Your second link doesn't seem to work for me. http://irakorshunova.github.io/2015/03/15/heart.html\r\n\r\nEdit: fixed, thanks", "111500": "[quote=Beyond Two Layers;111497]How did the idea? :D\r\nIt was trial and error?\r\n[/quote]\r\nQuote from Yann LeCun:\r\n*Larger networks tend to work better. Make your network bigger and bigger until the accuracy stops increasing. Then regularize the hell out of it. Then make it bigger still and pre-train it (...).*\r\nhttp://fastml.com/yann-lecuns-answers-from-the-reddit-ama/", "111505": "may i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? looking forward to the code though, must be a very good piece of study material", "111506": "Thank you for posting, really interesting approach and very different from ours. We used a fully convolutional network for segmentation, as in the deep learning tutorial, with the Sunnybrook data as the main NN training set. This was then combined with a few other, simpler models. I think we will have a blog post up eventually too :)", "111507": "Similar to my spider architecture too bad I didn't use it. I could only afford 3 conv layers in each branch of a 15 branch network because of memory issues. My question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.", "111509": "The way you found ROI centers is brilliant. If you consider this to be part of data cleaning, then perhaps it was smart, after all, to give us raw data.", "111512": "It's similar to how the Fourier tutorial finds region of interest. The Fourier tutorial actually averages all the images in a slice into a single image then finds roi based on all slices. I would like to stack these roi images on top of each other ordered by slice location, Then run 3D convolutions. This gives a complete view of a patients heart from all slices summarized into a smaller training set.", "111513": "I'll be the Devils advocate and say I hope Kaggle is not playing favorites here, you guys were not on the radar in the 1st round and currently you are 24th, any yet you won 2nd place. Not hating, just saying :) granted you won the last dsb and you have an interesting architecture. \r\n\r\nEdit: my bad didn't know you are part of kunsthart, you were on the radar.", "111545": "[quote=Yuanfang Guan;111505]\r\nmay i know how you generated 44 models. looking though the description, i only found three (slice/2ch/4ch). did i miss something? \r\n[/quote]\r\n\r\nIn total we trained about 250 different models. Most of these could be subdivided into the 4 broad categories we mention in our blog post (slice model / patient model / 2ch / 4ch). Although we only mentioned the architecture of our single best models, we tried out a lot of variants with both major and minor differences to get to that point, and some of those proved to be useful in the end.", "111546": "[quote=DavidGbodiOdaibo;111507]\r\nMy question is what kind or hardware can you train this on? with a 22 branch vgg 16 network. You also mention slice ordering with this architecture is important. I don't think so the top layers will be invariant to slice location as long as they are all represented.\r\n[/quote]\r\n\r\nWe trained our models on the NVIDIA GPUs that we have in the lab, which include GTX TITAN X, GTX 980, GTX 680 and Tesla K40 cards. \r\n\r\nFeeding the slices in order is important for the truncated code approximation to work. It is not invariant to it.", "111655": "What was the total computational time required to train your final model? What about making predictions from it on the test set?", "111712": "[quote=Ben Hamner;111655]What was the total computational time required to train your final model? What about making predictions from it on the test set?[/quote]\r\nWe didn't separate the two, but it took us 6 days to retrain everything twice. So about 3 days per submission. However, we needed 14 GPU's to stay within that time frame.\r\n\r\nOn a Titan X, one model would typically train in 4h for the smallest models, up to 36h for the slowest, also depending on the speed of reading from disk (which was the bottleneck on most of our machines). Generating predictions would take about 2h per model, since we predict every patient about 200 times before averaging.\r\n\r\nHowever, a single model without test time augmentation could probably predict the entire set of 440 patients in about 1 minute with a loss of ~0.0107.\r\n\r\nMost of the computation was to make it more robust, since it had to run on an unknown dataset, with maybe unknown data problems. In the end, it didn't look entirely necessary though.", "111778": "Awesome guys, great work! In the README in your repo, you say you \"reckon this data too dirty to do meaningful extrapolations for medical applications\". Which specific features of the data do you think make it unsuitable for medical applications? What changes do you think would have to be made to the dataset for it to be \"suitable\" for medical applications?", "111916": "I think the big issue is that it can be pretty wrong for some patients. The result you see on the leaderboard is only an average error, but it is because most are very good and some are very bad outliers.\r\n\r\nThis is typically NOT what you'd want for a medical application.\r\n\r\nThe dataset would need to be cleaner. I reckon however that when the radiologists using this get clear instructions on what data is needed, it can become suitable. Especially since the algorithms seem to be reasonably good at telling when it doesn't know for sure.\r\n\r\nSo, there is this other forum topic where it says \"accurate to within ~10%\", that's only true on average. And in medicine, averages are no good." }, "source": "meta" }