Hi, I'm trying to get started with this competition. I've prototyped a bunch of different models (3D CNN on the whole scan, 3D Unet with Threat Zone masks, the segmented 2D classifier from Brian Farrar's notebook, and MVCNN to classify on the 16 images at once). The only one that seemed to be working had actually just learned the expected probabilities for each tz. My next ideas are: 3D - segment the threat zones and classify on those crops, and 2D - assemble the 'threat zone plates' mentioned in Farrar's notebook (by joining the cropped images of a TZ in 2D)
\n\nAny chance someone who's solved for the basic pipeline could help by pointing me in the right direction? If so, thank you! It would be much appreciated.
", "messages": [ { "id": 243857, "postDate": "2017-11-15T01:39:25.040Z", "content": "From my experience, segmentation of some kind is really helpful. Its a much harder problem to extract information from a full image when only a tiny area is relevant. You can use a CNN to segment the zone regions or some other ML techniques, but more simply you can just hand-label patches large enough to account for variation in the data. Since everyone is more or less in the same pose, those patches can be surprisingly small. If the one here isn't precise enough for you, it's not too hard to redo more accurately from scratch.
\n\nJust my 2 cents, hope that helps! :)
", "rawMarkdown": "From my experience, segmentation of some kind is really helpful. Its a much harder problem to extract information from a full image when only a tiny area is relevant. You can use a CNN to segment the zone regions or some other ML techniques, but more simply you can just hand-label patches large enough to account for variation in the data. Since everyone is more or less in the same pose, those patches can be surprisingly small. If the one [here][1] isn't precise enough for you, it's not too hard to redo more accurately from scratch.\n\nJust my 2 cents, hope that helps! :)\n\n [1]: https://www.kaggle.com/jbfarrar/exploratory-data-analysis-and-example-generation", "votes": 3, "replies": [ { "id": 244244, "postDate": "2017-11-15T21:35:36.143Z", "content": "Hi Kevin, \nThank you for being so kind to share your valuable insights. Just wanted to say your score in this competition is extremely impressive! I am also using a similar segmentation strategy but feels like I have hit some kind of ceiling for the moment :)
", "rawMarkdown": "Hi Kevin, \nThank you for being so kind to share your valuable insights. Just wanted to say your score in this competition is extremely impressive! I am also using a similar segmentation strategy but feels like I have hit some kind of ceiling for the moment :)", "votes": 2 }, { "id": 244322, "postDate": "2017-11-16T01:42:50.553Z", "content": "Wow, thank you Kevin! Ok, in that case I'll switch back to segmentation and give the tz plates a shot. Thank you James, as well, for sharing that you're also using a segmentation strategy.
", "rawMarkdown": "Wow, thank you Kevin! Ok, in that case I'll switch back to segmentation and give the tz plates a shot. Thank you James, as well, for sharing that you're also using a segmentation strategy." }, { "id": 245853, "postDate": "2017-11-19T21:41:40.263Z", "content": "I thought it would be useful for you (and everybody else) to have some kind of benchmark: I compared the same model and the same parameters, and while I stopped the test after a few hours, my segmentation design scored 0.25 after a few hours of training, while the full image design scored 0.27 after the same training.
\n\nI did have to scale down the convolutions and pooling slightly in order to avoid exhausting my RAM with the increased input size, but I'm confident saying that if you can ensure accurate segmentation of the subject's body, you will definitely see significant improvement in scoring by slicing up the input :)
", "rawMarkdown": "I thought it would be useful for you (and everybody else) to have some kind of benchmark: I compared the same model and the same parameters, and while I stopped the test after a few hours, my segmentation design scored 0.25 after a few hours of training, while the full image design scored 0.27 after the same training.\n\nI did have to scale down the convolutions and pooling slightly in order to avoid exhausting my RAM with the increased input size, but I'm confident saying that if you can ensure accurate segmentation of the subject's body, you will definitely see significant improvement in scoring by slicing up the input :)", "votes": 1 } ] }, { "id": 245888, "postDate": "2017-11-20T01:52:07.493Z", "content": "At the risk of sounding pretentious or something, I'd like to offer people some advice to try and help them avoid the frustration and disappointment of complete failure (which I seem to have finally dug myself out of, thank God.) First of all, I'm a total newcomer to modern machine learning: I took artificial intelligence at university and all, but they still called it artificial intelligence then (and I got a C in that class, ahem...) So take what you find useful and discard the rest.
\n\nAlmost every failure I had seems to have been due to small, unexpected mistakes. Not paying attention to the order in which my nets were expecting channel data, for instance. TensorFlow seems to default to \"channels_last\", i.e. (660, 512, 1), whereas Theano is \"channels_first\", i.e. (1, 660, 512).
\n\nA number of my designs resulted in nothing more than the statistical frequency, regardless of the input I passed to it. When I inspected the output of my convolution layers, the reason became very clear: it was a solid black block and nothing else. If you're having similar problems, go over the way you're processing your data very carefully -- be careful with transpose() and reshape() for example. They won't warn you if you warp your data in to the wrong format.
\n\nTake this in particular with a grain of salt, but over reliance on relu (rectified linear unit) activations can also cause problems. I actually knocked out the majority of my networks accidentally using them in silly ways, haha...
\n\nI think that's about all the advice I can give that I wasn't able to find online myself. Good luck! :)
", "rawMarkdown": "At the risk of sounding pretentious or something, I'd like to offer people some advice to try and help them avoid the frustration and disappointment of complete failure (which I seem to have finally dug myself out of, thank God.) First of all, I'm a total newcomer to modern machine learning: I took artificial intelligence at university and all, but they still called it artificial intelligence then (and I got a C in that class, ahem...) So take what you find useful and discard the rest.\n\nAlmost every failure I had seems to have been due to small, unexpected mistakes. Not paying attention to the order in which my nets were expecting channel data, for instance. TensorFlow seems to default to \"channels_last\", i.e. (660, 512, 1), whereas Theano is \"channels_first\", i.e. (1, 660, 512).\n\nA number of my designs resulted in nothing more than the statistical frequency, regardless of the input I passed to it. When I inspected the output of my convolution layers, the reason became very clear: it was a solid black block and nothing else. If you're having similar problems, go over the way you're processing your data very carefully -- be careful with transpose() and reshape() for example. They won't warn you if you warp your data in to the wrong format.\n\nTake this in particular with a grain of salt, but over reliance on relu (rectified linear unit) activations can also cause problems. I actually knocked out the majority of my networks accidentally using them in silly ways, haha...\n\nI think that's about all the advice I can give that I wasn't able to find online myself. Good luck! :)", "votes": 2, "replies": [ { "id": 245945, "postDate": "2017-11-20T06:25:59.813Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 243393, "postDate": "2017-11-13T23:28:57.360Z", "content": "I don't like the segmenting stuff, as you will never get a robust segment (unless you use a NN on the whole image to get your segment in the first place). I know how frustrating it is (and how relieving when it finally works), use the standard 3 components: convolution, normalization and regularization on a possibly slightly reduced image (but whole) and you should get at least 0.25. Also, make sure you have a GPU. Another tip, look at the images with your own eyes. Make sure the labels make sense and see what you need to recognize the threads,.. the computer is not going to be smarter than you are.
", "rawMarkdown": "I don't like the segmenting stuff, as you will never get a robust segment (unless you use a NN on the whole image to get your segment in the first place). I know how frustrating it is (and how relieving when it finally works), use the standard 3 components: convolution, normalization and regularization on a possibly slightly reduced image (but whole) and you should get at least 0.25. Also, make sure you have a GPU. Another tip, look at the images with your own eyes. Make sure the labels make sense and see what you need to recognize the threads,.. the computer is not going to be smarter than you are.", "votes": 2, "replies": [ { "id": 243400, "postDate": "2017-11-14T00:00:11.203Z", "content": "Ok, thank you! This is really helpful. I will focus on applying a CNN to the full image, making sure to have regularization in the architecture and normalization in the preprocessing pipeline. Since we have 16 images per data point, I'm guessing this means I should join them all into one large image and use that to predict all 17 threat zones at once. Please let me know if you had a different interpretation in mind. Thank you!!
", "rawMarkdown": "Ok, thank you! This is really helpful. I will focus on applying a CNN to the full image, making sure to have regularization in the architecture and normalization in the preprocessing pipeline. Since we have 16 images per data point, I'm guessing this means I should join them all into one large image and use that to predict all 17 threat zones at once. Please let me know if you had a different interpretation in mind. Thank you!!", "votes": 1 }, { "id": 243669, "postDate": "2017-11-14T16:44:38.893Z", "content": "For what it's worth, I feel similarly several times a day. A couple of points: there are 16 images per scan in the .aps files, so you're right about that; but there are 64 in the .a3daps files, and 660 in the .a3d files (those are a bird's eye view of a 360 degree scan going from the subject's feet to their head, which is, I imagine, why the Y axis in the other files is 660 -- the .a3d files are actually 512x512.)
\n\nAlso -- and pardon me if you find this advice patronizing, but I personally could have used it -- if your network's output appears independent of it's input (meaning it predicts the same probability for every zone no matter what data you feed it), you probably screwed up your convolution layer(s).
", "rawMarkdown": "For what it's worth, I feel similarly several times a day. A couple of points: there are 16 images per scan in the .aps files, so you're right about that; but there are 64 in the .a3daps files, and 660 in the .a3d files (those are a bird's eye view of a 360 degree scan going from the subject's feet to their head, which is, I imagine, why the Y axis in the other files is 660 -- the .a3d files are actually 512x512.)\n\nAlso -- and pardon me if you find this advice patronizing, but I personally could have used it -- if your network's output appears independent of it's input (meaning it predicts the same probability for every zone no matter what data you feed it), you probably screwed up your convolution layer(s).", "votes": 2 }, { "id": 258861, "postDate": "2017-12-17T07:07:13.133Z", "content": "Congrats on the score, you did really well! How did you end up doing it?
", "rawMarkdown": "Congrats on the score, you did really well! How did you end up doing it?", "votes": 1 }, { "id": 259772, "postDate": "2017-12-19T01:03:42.597Z", "content": "Thanks! The real tip was hearing that Kevin was using 2D segmentation. That was exactly what I needed. Also, thanks to you for mentioning normalization - that was another important part. I normalized the data by cropping around a certain threshold and resizing that to be the full image, so that threat zone locations would be more consistent. Overall, I concatenated the threat zone crops from the aps files into \"threat plates\" and trained an ensemble of ResNet50 (Imagenet weights) on that.
\n\nMore specifically, I had two key insights:
\n\nExploit the symmetry of the human body. By switching and horizontally flipping certain images I could use the same crops for right and left forearm, right and left shin, etc. Thus I doubled the size of the dataset and only needed 9 networks for a full segmentation approach.
Clipping. This is a bit of a Kaggle trick, which I learned from Fast.AI. Based on the math of log loss, on an inaccurate prediction, overconfidence will be penalized really harshly. So I clipped my predictions to the [0.015, 0.985] interval and this took me from bronze to silver.
Thanks again for your help! And for everyone's help! I think the community spirit is one of the greatest aspects of Kaggle. Here's my repo - https://github.com/jamespeterthornton/DHS
", "rawMarkdown": "Thanks! The real tip was hearing that Kevin was using 2D segmentation. That was exactly what I needed. Also, thanks to you for mentioning normalization - that was another important part. I normalized the data by cropping around a certain threshold and resizing that to be the full image, so that threat zone locations would be more consistent. Overall, I concatenated the threat zone crops from the aps files into \"threat plates\" and trained an ensemble of ResNet50 (Imagenet weights) on that.\n\nMore specifically, I had two key insights: \n\n1. Exploit the symmetry of the human body. By switching and horizontally flipping certain images I could use the same crops for right and left forearm, right and left shin, etc. Thus I doubled the size of the dataset and only needed 9 networks for a full segmentation approach.\n\n2. Clipping. This is a bit of a Kaggle trick, which I learned from Fast.AI. Based on the math of log loss, on an inaccurate prediction, overconfidence will be penalized really harshly. So I clipped my predictions to the [0.015, 0.985] interval and this took me from bronze to silver.\n\nThanks again for your help! And for everyone's help! I think the community spirit is one of the greatest aspects of Kaggle. Here's my repo - https://github.com/jamespeterthornton/DHS", "votes": 3 } ] }, { "id": 243382, "postDate": "2017-11-13T23:06:37.727Z", "content": "Hi, I'm trying to get started with this competition. I've prototyped a bunch of different models (3D CNN on the whole scan, 3D Unet with Threat Zone masks, the segmented 2D classifier from Brian Farrar's notebook, and MVCNN to classify on the 16 images at once). The only one that seemed to be working had actually just learned the expected probabilities for each tz. My next ideas are: 3D - segment the threat zones and classify on those crops, and 2D - assemble the 'threat zone plates' mentioned in Farrar's notebook (by joining the cropped images of a TZ in 2D)
\n\nAny chance someone who's solved for the basic pipeline could help by pointing me in the right direction? If so, thank you! It would be much appreciated.
", "rawMarkdown": "Hi, I'm trying to get started with this competition. I've prototyped a bunch of different models (3D CNN on the whole scan, 3D Unet with Threat Zone masks, the segmented 2D classifier from Brian Farrar's notebook, and MVCNN to classify on the 16 images at once). The only one that seemed to be working had actually just learned the expected probabilities for each tz. My next ideas are: 3D - segment the threat zones and classify on those crops, and 2D - assemble the 'threat zone plates' mentioned in Farrar's notebook (by joining the cropped images of a TZ in 2D)\n \nAny chance someone who's solved for the basic pipeline could help by pointing me in the right direction? If so, thank you! It would be much appreciated.", "votes": 2 } ], "comments": [ { "id": 243857, "author_name": "Kevin H", "author_url": "", "post_date": "2017-11-15T01:39:25.040000", "content": "From my experience, segmentation of some kind is really helpful. Its a much harder problem to extract information from a full image when only a tiny area is relevant. You can use a CNN to segment the zone regions or some other ML techniques, but more simply you can just hand-label patches large enough to account for variation in the data. Since everyone is more or less in the same pose, those patches can be surprisingly small. If the one here isn't precise enough for you, it's not too hard to redo more accurately from scratch.
\n\nJust my 2 cents, hope that helps! :)
", "votes": 3, "replies": [ { "id": 244244, "author_name": "James Requa", "author_url": "", "post_date": "2017-11-15T21:35:36.143000", "content": "Hi Kevin, \nThank you for being so kind to share your valuable insights. Just wanted to say your score in this competition is extremely impressive! I am also using a similar segmentation strategy but feels like I have hit some kind of ceiling for the moment :)
", "votes": 2, "replies": [] }, { "id": 244322, "author_name": "James Thornton", "author_url": "", "post_date": "2017-11-16T01:42:50.553000", "content": "Wow, thank you Kevin! Ok, in that case I'll switch back to segmentation and give the tz plates a shot. Thank you James, as well, for sharing that you're also using a segmentation strategy.
", "votes": 0, "replies": [] }, { "id": 245853, "author_name": "Murray Miron", "author_url": "", "post_date": "2017-11-19T21:41:40.263000", "content": "I thought it would be useful for you (and everybody else) to have some kind of benchmark: I compared the same model and the same parameters, and while I stopped the test after a few hours, my segmentation design scored 0.25 after a few hours of training, while the full image design scored 0.27 after the same training.
\n\nI did have to scale down the convolutions and pooling slightly in order to avoid exhausting my RAM with the increased input size, but I'm confident saying that if you can ensure accurate segmentation of the subject's body, you will definitely see significant improvement in scoring by slicing up the input :)
", "votes": 1, "replies": [] } ] }, { "id": 245888, "author_name": "Murray Miron", "author_url": "", "post_date": "2017-11-20T01:52:07.493000", "content": "At the risk of sounding pretentious or something, I'd like to offer people some advice to try and help them avoid the frustration and disappointment of complete failure (which I seem to have finally dug myself out of, thank God.) First of all, I'm a total newcomer to modern machine learning: I took artificial intelligence at university and all, but they still called it artificial intelligence then (and I got a C in that class, ahem...) So take what you find useful and discard the rest.
\n\nAlmost every failure I had seems to have been due to small, unexpected mistakes. Not paying attention to the order in which my nets were expecting channel data, for instance. TensorFlow seems to default to \"channels_last\", i.e. (660, 512, 1), whereas Theano is \"channels_first\", i.e. (1, 660, 512).
\n\nA number of my designs resulted in nothing more than the statistical frequency, regardless of the input I passed to it. When I inspected the output of my convolution layers, the reason became very clear: it was a solid black block and nothing else. If you're having similar problems, go over the way you're processing your data very carefully -- be careful with transpose() and reshape() for example. They won't warn you if you warp your data in to the wrong format.
\n\nTake this in particular with a grain of salt, but over reliance on relu (rectified linear unit) activations can also cause problems. I actually knocked out the majority of my networks accidentally using them in silly ways, haha...
\n\nI think that's about all the advice I can give that I wasn't able to find online myself. Good luck! :)
", "votes": 2, "replies": [ { "id": 245945, "author_name": "", "author_url": "", "post_date": "2017-11-20T06:25:59.813000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 243393, "author_name": "Bastiaan Bergman", "author_url": "", "post_date": "2017-11-13T23:28:57.360000", "content": "I don't like the segmenting stuff, as you will never get a robust segment (unless you use a NN on the whole image to get your segment in the first place). I know how frustrating it is (and how relieving when it finally works), use the standard 3 components: convolution, normalization and regularization on a possibly slightly reduced image (but whole) and you should get at least 0.25. Also, make sure you have a GPU. Another tip, look at the images with your own eyes. Make sure the labels make sense and see what you need to recognize the threads,.. the computer is not going to be smarter than you are.
", "votes": 2, "replies": [ { "id": 243400, "author_name": "James Thornton", "author_url": "", "post_date": "2017-11-14T00:00:11.203000", "content": "Ok, thank you! This is really helpful. I will focus on applying a CNN to the full image, making sure to have regularization in the architecture and normalization in the preprocessing pipeline. Since we have 16 images per data point, I'm guessing this means I should join them all into one large image and use that to predict all 17 threat zones at once. Please let me know if you had a different interpretation in mind. Thank you!!
", "votes": 1, "replies": [] }, { "id": 243669, "author_name": "Murray Miron", "author_url": "", "post_date": "2017-11-14T16:44:38.893000", "content": "For what it's worth, I feel similarly several times a day. A couple of points: there are 16 images per scan in the .aps files, so you're right about that; but there are 64 in the .a3daps files, and 660 in the .a3d files (those are a bird's eye view of a 360 degree scan going from the subject's feet to their head, which is, I imagine, why the Y axis in the other files is 660 -- the .a3d files are actually 512x512.)
\n\nAlso -- and pardon me if you find this advice patronizing, but I personally could have used it -- if your network's output appears independent of it's input (meaning it predicts the same probability for every zone no matter what data you feed it), you probably screwed up your convolution layer(s).
", "votes": 2, "replies": [] }, { "id": 258861, "author_name": "Bastiaan Bergman", "author_url": "", "post_date": "2017-12-17T07:07:13.133000", "content": "Congrats on the score, you did really well! How did you end up doing it?
", "votes": 1, "replies": [] }, { "id": 259772, "author_name": "James Thornton", "author_url": "", "post_date": "2017-12-19T01:03:42.597000", "content": "Thanks! The real tip was hearing that Kevin was using 2D segmentation. That was exactly what I needed. Also, thanks to you for mentioning normalization - that was another important part. I normalized the data by cropping around a certain threshold and resizing that to be the full image, so that threat zone locations would be more consistent. Overall, I concatenated the threat zone crops from the aps files into \"threat plates\" and trained an ensemble of ResNet50 (Imagenet weights) on that.
\n\nMore specifically, I had two key insights:
\n\nExploit the symmetry of the human body. By switching and horizontally flipping certain images I could use the same crops for right and left forearm, right and left shin, etc. Thus I doubled the size of the dataset and only needed 9 networks for a full segmentation approach.
Clipping. This is a bit of a Kaggle trick, which I learned from Fast.AI. Based on the math of log loss, on an inaccurate prediction, overconfidence will be penalized really harshly. So I clipped my predictions to the [0.015, 0.985] interval and this took me from bronze to silver.
Thanks again for your help! And for everyone's help! I think the community spirit is one of the greatest aspects of Kaggle. Here's my repo - https://github.com/jamespeterthornton/DHS
", "votes": 3, "replies": [] } ] } ], "raw_markdown_by_id": { "243857": "From my experience, segmentation of some kind is really helpful. Its a much harder problem to extract information from a full image when only a tiny area is relevant. You can use a CNN to segment the zone regions or some other ML techniques, but more simply you can just hand-label patches large enough to account for variation in the data. Since everyone is more or less in the same pose, those patches can be surprisingly small. If the one [here][1] isn't precise enough for you, it's not too hard to redo more accurately from scratch.\n\nJust my 2 cents, hope that helps! :)\n\n [1]: https://www.kaggle.com/jbfarrar/exploratory-data-analysis-and-example-generation", "245888": "At the risk of sounding pretentious or something, I'd like to offer people some advice to try and help them avoid the frustration and disappointment of complete failure (which I seem to have finally dug myself out of, thank God.) First of all, I'm a total newcomer to modern machine learning: I took artificial intelligence at university and all, but they still called it artificial intelligence then (and I got a C in that class, ahem...) So take what you find useful and discard the rest.\n\nAlmost every failure I had seems to have been due to small, unexpected mistakes. Not paying attention to the order in which my nets were expecting channel data, for instance. TensorFlow seems to default to \"channels_last\", i.e. (660, 512, 1), whereas Theano is \"channels_first\", i.e. (1, 660, 512).\n\nA number of my designs resulted in nothing more than the statistical frequency, regardless of the input I passed to it. When I inspected the output of my convolution layers, the reason became very clear: it was a solid black block and nothing else. If you're having similar problems, go over the way you're processing your data very carefully -- be careful with transpose() and reshape() for example. They won't warn you if you warp your data in to the wrong format.\n\nTake this in particular with a grain of salt, but over reliance on relu (rectified linear unit) activations can also cause problems. I actually knocked out the majority of my networks accidentally using them in silly ways, haha...\n\nI think that's about all the advice I can give that I wasn't able to find online myself. Good luck! :)", "243393": "I don't like the segmenting stuff, as you will never get a robust segment (unless you use a NN on the whole image to get your segment in the first place). I know how frustrating it is (and how relieving when it finally works), use the standard 3 components: convolution, normalization and regularization on a possibly slightly reduced image (but whole) and you should get at least 0.25. Also, make sure you have a GPU. Another tip, look at the images with your own eyes. Make sure the labels make sense and see what you need to recognize the threads,.. the computer is not going to be smarter than you are.", "243382": "Hi, I'm trying to get started with this competition. I've prototyped a bunch of different models (3D CNN on the whole scan, 3D Unet with Threat Zone masks, the segmented 2D classifier from Brian Farrar's notebook, and MVCNN to classify on the 16 images at once). The only one that seemed to be working had actually just learned the expected probabilities for each tz. My next ideas are: 3D - segment the threat zones and classify on those crops, and 2D - assemble the 'threat zone plates' mentioned in Farrar's notebook (by joining the cropped images of a TZ in 2D)\n \nAny chance someone who's solved for the basic pipeline could help by pointing me in the right direction? If so, thank you! It would be much appreciated." } }