![\"enter](\"http://i.imgur.com/rJBOLir.gif\" "\\"\\"")
Thanks to everyone. I love this competition.
\n\n\n\nI use VGG16 without top, add FC-1024, and FC-5 with linear output.
\n\nSGD optimization and mean_squared_error loss.
\n\nAnd post processing for better regression.
\n\nI think it is a simple way to count. Looking forward to hearing your thoughts, thanks.
", "messages": [ { "id": 196735, "postDate": "2017-06-28T02:28:12.197Z", "content": "Thanks to everyone. I love this competition.
\n\n\n\nI use VGG16 without top, add FC-1024, and FC-5 with linear output.
\n\nSGD optimization and mean_squared_error loss.
\n\nAnd post processing for better regression.
\n\nI think it is a simple way to count. Looking forward to hearing your thoughts, thanks.
", "rawMarkdown": "Thanks to everyone. I love this competition.\n\n[Here is my solution.][1] \n\nI use VGG16 without top, add FC-1024, and FC-5 with linear output.\n\nSGD optimization and mean_squared_error loss.\n\nAnd post processing for better regression.\n\nI think it is a simple way to count. Looking forward to hearing your thoughts, thanks.\n\n [1]: https://www.kaggle.com/outrunner/use-keras-to-count-sea-lions/notebook", "votes": 67 }, { "id": 196751, "postDate": "2017-06-28T03:02:21.327Z", "content": "I'm proud to see my blob detection in the winning solution.
\n\nGood job!
", "rawMarkdown": "I'm proud to see my blob detection in the winning solution.\n\nGood job!", "votes": 8, "replies": [ { "id": 196783, "postDate": "2017-06-28T05:13:23.207Z", "content": "Thanks again.
", "rawMarkdown": "Thanks again.", "votes": 4 } ] }, { "id": 197128, "postDate": "2017-06-28T21:43:51.967Z", "content": "Outstanding work outrunner! Thanks for sharing; may I ask... did you have any intuition, before starting out, that this network would work ? And what do you think made the big difference in this architecture - was it the 1024 FC or the SGD or slow training or sthing else ... i am sure you tried lots of changes in this area. \nI gather you started with pretrained weights.
", "rawMarkdown": "Outstanding work outrunner! Thanks for sharing; may I ask... did you have any intuition, before starting out, that this network would work ? And what do you think made the big difference in this architecture - was it the 1024 FC or the SGD or slow training or sthing else ... i am sure you tried lots of changes in this area. \nI gather you started with pretrained weights. ", "votes": 1, "replies": [ { "id": 197219, "postDate": "2017-06-29T02:39:36.913Z", "content": "Just intuition, and I think it would work undoubtedly even when I got trouble at the beginning. VGG is a basic CNN, I took the 1024 FC as a buffer between FCN and output. The size is according to RAM limit and model size, and intuitively I think it is enough. When I got scale problem, I tried more neurons and nothing happened. SGD is more stable than other aggressive optimizers in my work, maybe it is due to the large variation in output labels.
", "rawMarkdown": "Just intuition, and I think it would work undoubtedly even when I got trouble at the beginning. VGG is a basic CNN, I took the 1024 FC as a buffer between FCN and output. The size is according to RAM limit and model size, and intuitively I think it is enough. When I got scale problem, I tried more neurons and nothing happened. SGD is more stable than other aggressive optimizers in my work, maybe it is due to the large variation in output labels.", "votes": 3 } ] }, { "id": 196752, "postDate": "2017-06-28T03:02:34.203Z", "content": "Congratulations and Excellent work. Your kaggle ranking will be top 40.
", "rawMarkdown": "Congratulations and Excellent work. Your kaggle ranking will be top 40.", "votes": 1 }, { "id": 197692, "postDate": "2017-06-30T03:08:24.533Z", "content": "Something I could share about why misclassification on females and juveniles is worse more than I expected.
\n\n
I generated random dot for explanation. I wish the threshold between females and juveniles in the model could be the green line, but it is too naive. Then I think at least it should be the red line by statistically, but in fact the threshold is dotted red line. (It is about size feature, there are other features like lion distribution etc. in the model let the prediction better.)
\n\nI guess it is because in the small scale patch, lion count is larger, and the squared error even more. I tried add large scale training patch with more juveniles and it helps a little, but overall score is not significant.
", "rawMarkdown": "Something I could share about why misclassification on females and juveniles is worse more than I expected.\n\n![enter image description here][1]\n\nI generated random dot for explanation. I wish the threshold between females and juveniles in the model could be the green line, but it is too naive. Then I think at least it should be the red line by statistically, but in fact the threshold is dotted red line. (It is about size feature, there are other features like lion distribution etc. in the model let the prediction better.)\n\nI guess it is because in the small scale patch, lion count is larger, and the squared error even more. I tried add large scale training patch with more juveniles and it helps a little, but overall score is not significant.\n\n [1]: http://i.imgur.com/rJBOLir.gif", "votes": 2 }, { "id": 198656, "postDate": "2017-07-03T09:16:21.487Z", "content": "Would you mind creating a visualization of the predictions that the network produces similar to the ones by Konstantin Lopuhin, e.g. using for i, p in enumerate(batch_of_predictions): scipy.misc.imsave('preds_%i.png' % i, normalize(np.concatenate(np.moveaxis(p, -1, 0))))? I think it would be interesting to see how well the network manages to tell the classes apart in various cases and what the predicted dots look like.
I don't quite understand what is batch_of_predictions?
", "rawMarkdown": "I don't quite understand what is batch_of_predictions?" }, { "id": 198671, "postDate": "2017-07-03T10:40:24.903Z", "content": "The variable batch_of_predictions is be the output of your network. Usually, all operations in deep learning frameworks operate on batches of training examples. When you run the model to test it on a couple of images, you need to feed a batch of n≥1 images and you get n predictions back. This is how you can save all images from the batch. I basically just copied & modified a line of code from my solution.
I mean, in my network, predictions are only numbers. Or you want to see some testing patches with its output?
", "rawMarkdown": "I mean, in my network, predictions are only numbers. Or you want to see some testing patches with its output?" }, { "id": 198816, "postDate": "2017-07-03T19:17:09.003Z", "content": "Oops, I misunderstood the way you create the trainYs. I thought your trainYs consisted of dots at the animal coordinates. Never mind then. I could imagine your solution works well because only predicting the total counts (not the locations) in a large window like that reduces noise from the label positions (sometimes the labels are even outside of the animals). For your method, imprecise label positions only cause noise at the borders of the patch.
By the way, this is how I verify the prediction: （more count more bright)\n![\"enter](\"http://i.imgur.com/FKQdbpX.jpg\" "\\"\\"")
It is cool! How can you show the picture in this way? Could you please show some code on this?
", "rawMarkdown": "It is cool! How can you show the picture in this way? Could you please show some code on this?" }, { "id": 198901, "postDate": "2017-07-04T03:50:06.773Z", "content": "for p in test_patches: p = p*f(count(p)) #make sure 0 < f < 1 or do some normalized processing
\n\nthen stitch them side by side
", "rawMarkdown": "for p in test_patches: p = p*f(count(p)) #make sure 0 < f < 1 or do some normalized processing\n\nthen stitch them side by side" } ] }, { "id": 198600, "postDate": "2017-07-03T05:14:55.613Z", "content": "Congrats on winning. Great solution!! \nI noticed that the test was also run on the patches that were cut from the original test images and the overall count for an image is the sum of the counts from all the patches cut from the image. Since the patch cut is random, would a sea lion at the boundary of two patches be double-counted? How would the double counting be avoided in the solution?\nThanks!!
", "rawMarkdown": "Congrats on winning. Great solution!! \nI noticed that the test was also run on the patches that were cut from the original test images and the overall count for an image is the sum of the counts from all the patches cut from the image. Since the patch cut is random, would a sea lion at the boundary of two patches be double-counted? How would the double counting be avoided in the solution?\nThanks!!", "replies": [ { "id": 198653, "postDate": "2017-07-03T08:54:14.403Z", "content": "Ideally, output should be 0.5 if the lion has been cut in half. Because in the same situation at training stage, one patch is 1 and the other one is 0.
", "rawMarkdown": "Ideally, output should be 0.5 if the lion has been cut in half. Because in the same situation at training stage, one patch is 1 and the other one is 0." } ] }, { "id": 197253, "postDate": "2017-06-29T06:33:34.793Z", "rawMarkdown": "" }, { "id": 197252, "postDate": "2017-06-29T06:33:22.160Z", "content": "Hi Outrunner,
\n\nCongratulations!
\n\nI'm not very familiar with Keras code. Could you explain the last layers of your neural network? What happens after the convolutional layers? Did you just use linear activations to generate a 5-layer heatmap and compared that to the GT heatmap using RMSE?
", "rawMarkdown": "Hi Outrunner,\n\nCongratulations!\n\nI'm not very familiar with Keras code. Could you explain the last layers of your neural network? What happens after the convolutional layers? Did you just use linear activations to generate a 5-layer heatmap and compared that to the GT heatmap using RMSE?", "replies": [ { "id": 197267, "postDate": "2017-06-29T07:00:52.650Z", "content": "The \"Patches looks like\" section in notebook, image is training patch, [1 0 3 0 1] is label (lion count per type in the patch), the last is 5 neurons fully-connected layer with linear output.
", "rawMarkdown": "The \"Patches looks like\" section in notebook, image is training patch, [1 0 3 0 1] is label (lion count per type in the patch), the last is 5 neurons fully-connected layer with linear output." }, { "id": 197570, "postDate": "2017-06-29T20:52:45.550Z", "content": "Aha! So it's a kind of regression?
", "rawMarkdown": "Aha! So it's a kind of regression?" }, { "id": 197595, "postDate": "2017-06-29T23:11:52.107Z", "content": "I think so.
", "rawMarkdown": " I think so." } ] }, { "id": 197238, "postDate": "2017-06-29T04:45:26.363Z", "content": "Great job here!
", "rawMarkdown": "Great job here!" }, { "id": 197143, "postDate": "2017-06-28T22:17:25.833Z", "content": "I use custom data generator.
I trained a 200x200 and it performed worse. Maybe it is because 300 contains more relationship information between pups and adult_females. Flipping(rotating) 4 times at prediction time improved 0.04 RMSE. Using only convolutional layer in VGG is size free.
Just a normal loss function.
You can reference the table in the section.
Respect! Never underestimate the basics.
", "rawMarkdown": "Respect! Never underestimate the basics." }, { "id": 197074, "postDate": "2017-06-28T18:22:34.977Z", "content": "@outrunner\nCongrats and thanks for the code and the explanations.
\n\nAs you requested, please find attached the bounding boxes. the first batch are the 5000 first bboxes manually labeled and the other 75 to 80,000 are the ones generated using SSD (no guarantee on the quality and some subadult male disappeared in the mix).
\n\nKeep us informed on your progress with them please.
\n\nI generated them as I started this competition as standard detection problem but later switched to a segmentation where they were not the best idea and dragged me to the bottom of the leader-board: I was more concerned by crowed area where bboxes are an handicap rather than image scale...
", "rawMarkdown": "@outrunner\nCongrats and thanks for the code and the explanations.\n\nAs you requested, please find attached the bounding boxes. the first batch are the 5000 first bboxes manually labeled and the other 75 to 80,000 are the ones generated using SSD (no guarantee on the quality and some subadult male disappeared in the mix).\n\nKeep us informed on your progress with them please.\n\nI generated them as I started this competition as standard detection problem but later switched to a segmentation where they were not the best idea and dragged me to the bottom of the leader-board: I was more concerned by crowed area where bboxes are an handicap rather than image scale...", "replies": [ { "id": 197214, "postDate": "2017-06-29T02:14:31.817Z", "content": "Thanks. But I may try it when I have time or need to do farther research on this topic.
", "rawMarkdown": "Thanks. But I may try it when I have time or need to do farther research on this topic." } ] }, { "id": 196987, "postDate": "2017-06-28T14:38:50.380Z", "content": "It is interesting that the model trained at 1.0x scale had a much better performance being evaluated on the test set at 0.67x scale than on 1.0x. What might be the reason for this? Perhaps the receptive field of VGG16 is just not big enough to actually recognize the animals at 1.0x scale. VGG16 has an effective receptive field size of ~220px at the output neurons, but the largest adult males are up to ~260-270px in length. Also the largest distance of the pubs to other animals is about 250px.
", "rawMarkdown": "It is interesting that the model trained at 1.0x scale had a much better performance being evaluated on the test set at 0.67x scale than on 1.0x. What might be the reason for this? Perhaps the receptive field of VGG16 is just not big enough to actually recognize the animals at 1.0x scale. VGG16 has an effective receptive field size of ~220px at the output neurons, but the largest adult males are up to ~260-270px in length. Also the largest distance of the pubs to other animals is about 250px.", "replies": [ { "id": 197003, "postDate": "2017-06-28T15:27:55.667Z", "content": "I think it is typically because the testing image's sea lion is larger than training in average. In my opinion, the model don't know what sea lion is and where it is.
", "rawMarkdown": "I think it is typically because the testing image's sea lion is larger than training in average. In my opinion, the model don't know what sea lion is and where it is." }, { "id": 197088, "postDate": "2017-06-28T19:22:08.303Z", "content": "> In my opinion, the model don't know what sea lion is and where it is.
\n\nI agree. :) 'Not recognizing' was just shorthand for 'it might be harder to find distinguishing features if the animals are often only partially in sight for the network'. My info was incomplete/wrong anyway though: VGG's receptive field is 224 × 224 with zero-padding, but 404 × 404 if the net is applied convolutionally^1.
", "rawMarkdown": "> In my opinion, the model don't know what sea lion is and where it is.\n\nI agree. :) 'Not recognizing' was just shorthand for 'it might be harder to find distinguishing features if the animals are often only partially in sight for the network'. My info was incomplete/wrong anyway though: VGG's receptive field is 224 × 224 with zero-padding, but 404 × 404 if the net is applied convolutionally^[1].\n\n [1]: https://arxiv.org/pdf/1412.7062.pdf" }, { "id": 197204, "postDate": "2017-06-29T01:53:17.547Z", "content": "I haven't studied receptive field in detail yet. So, thanks for sharing.
", "rawMarkdown": "I haven't studied receptive field in detail yet. So, thanks for sharing." } ] }, { "id": 196976, "postDate": "2017-06-28T14:12:33.797Z", "content": "Fantastic result, congrats!
\n\nAm I understanding correctly that the multi-scale model is essentially heavy data augmentation with different scales?
", "rawMarkdown": "Fantastic result, congrats!\n\nAm I understanding correctly that the multi-scale model is essentially heavy data augmentation with different scales?" }, { "id": 196967, "postDate": "2017-06-28T13:48:30.440Z", "content": "Such a simple solution. It really goes to show how far machine learning, convolutional neural networks, and a perfect implementation, application, and execution can take you. Congrats @outrunner on 1st place, and much thanks for the informative write-up.
", "rawMarkdown": "Such a simple solution. It really goes to show how far machine learning, convolutional neural networks, and a perfect implementation, application, and execution can take you. **Congrats @outrunner on 1st place**, and much thanks for the informative write-up. ", "replies": [ { "id": 196970, "postDate": "2017-06-28T13:52:27.350Z", "content": "I also had a question about the values on your table, I don't quite understand why a higher value on the table like the values in your final submission column yielded better results. What is the significance of average of juveniles# / (adult_females# + juveniles#)?
", "rawMarkdown": "I also had a question about the values on your table, I don't quite understand why a higher value on the table like the values in your final submission column yielded better results. What is the significance of **average of juveniles# / (adult_females# + juveniles#)**?", "votes": 1 }, { "id": 196982, "postDate": "2017-06-28T14:27:33.503Z", "content": "I think the numbers just show that on the test set outrunner's models predicted much fewer juveniles vs adult_females than during training, thus he/she corrected roughly for that ratio (with great success). E.g. the the patches with 46-50 juveniles had a number of 0.93 during training with a single zoom level (1x), but on the test set only 0.68, so it was corrected to 0.76. In that particular case it would have been interesting to see the performance of the r0.59 model without post-processing on the public LB.
\n\nThis might imply that the test set had different juvenile/adult_female ratios than the training set.
", "rawMarkdown": "I think the numbers just show that on the test set outrunner's models predicted much fewer juveniles vs adult_females than during training, thus he/she corrected roughly for that ratio (with great success). E.g. the the patches with 46-50 juveniles had a number of 0.93 during training with a single zoom level (1x), but on the test set only 0.68, so it was corrected to 0.76. In that particular case it would have been interesting to see the performance of the r0.59 model without post-processing on the public LB.\n\nThis might imply that the test set had different juvenile/adult_female ratios than the training set." }, { "id": 196997, "postDate": "2017-06-28T15:14:00.110Z", "content": "Sorry for being unclear and thanks to erde. It is the ratio between juveniles and juveniles + adult_females.
\n\nI think the ratio problem comes from scale problem. The CNN basically is a statistical model, simply distinguish juveniles and adult_females by size. Scale down image can get more juveniles, but the problem is if out of range, total lions count will go down. For example:
\n\nThis is the predict result of test/12690 on multi-scale model with different test scale. The juveniles + adult_females keeps around 300 for 0.59~0.43, and began to decrease from 0.39.
\n\nSo, all you are interesting in without post processing's performance. It is sure if I scale down more. But I don't want to take the risk if sea lions in private set are smaller then public.
\n\n\n\n\nthe multi-scale model is essentially heavy data augmentation with different scales?
\n
Yes, and single scale model's keeping number scale range is narrower.
", "rawMarkdown": " Sorry for being unclear and thanks to erde. It is the ratio between juveniles and juveniles + adult_females.\n\nI think the ratio problem comes from scale problem. The CNN basically is a statistical model, simply distinguish juveniles and adult_females by size. Scale down image can get more juveniles, but the problem is if out of range, total lions count will go down. For example:\n\n - scale predict\n - 0.59 [ 13 5 180 120 2]\n - 0.53 [ 14 8 141 149 0]\n - 0.48 [ 10 5 132 165 2]\n - 0.43 [ 9 7 122 182 0]\n - 0.39 [ 6 7 101 185 0]\n - 0.35 [ 7 7 74 176 0]\n - 0.31 [ 4 6 62 168 1]\n - 0.28 [ 5 7 48 137 0]\n\nThis is the predict result of test/12690 on multi-scale model with different test scale. The juveniles + adult_females keeps around 300 for 0.59~0.43, and began to decrease from 0.39.\n\nSo, all you are interesting in without post processing's performance. It is sure if I scale down more. But I don't want to take the risk if sea lions in private set are smaller then public.\n\n> the multi-scale model is essentially heavy data augmentation with different scales?\n\nYes, and single scale model's keeping number scale range is narrower.\n\n", "votes": 1 }, { "id": 197004, "postDate": "2017-06-28T15:29:00.683Z", "content": "Ah I see so scaled down images tend to have more juveniles than adult females versus not scaled down. If it were possible to teach the method how to compare the sizes of sea lions within a single image's scale, then the model could learn to compare the juvenile's relative size to female's relative size and your post processing trick might not have been needed? Maybe if first the image is processed for females then those features are fed to another network that searches for juveniles, the features can be reused. I think your post processing basically gave the model domain knowledge about the problem that could be replaced with a different architecture, would be interesting to test this!
", "rawMarkdown": "Ah I see so scaled down images tend to have more juveniles than adult females versus not scaled down. If it were possible to teach the method how to compare the sizes of sea lions within a single image's scale, then the model could learn to compare the juvenile's relative size to female's relative size and your post processing trick might not have been needed? Maybe if first the image is processed for females then those features are fed to another network that searches for juveniles, the features can be reused. I think your post processing basically gave the model domain knowledge about the problem that could be replaced with a different architecture, would be interesting to test this!" }, { "id": 197007, "postDate": "2017-06-28T15:39:33.807Z", "content": "Sure, and it is not easy work. By the way, if they have grounding truth bounding box, I want to train a object detection for image scale detect by sea lion's size, just a thought.
", "rawMarkdown": "Sure, and it is not easy work. By the way, if they have grounding truth bounding box, I want to train a object detection for image scale detect by sea lion's size, just a thought." }, { "id": 197012, "postDate": "2017-06-28T15:59:07.277Z", "content": "Ideally, there should be preprocessing that makes all images the same scale. Much like satellite images are orthorectified to make objects on the ground to be the same scale and angle (so square on ground is square on image no matter how camera looks at Earth -- at right or some different angle). The same should be done here -- knowing flight path, UAV elevation, camera characteristics, capture angle and so on, it is possible to make all images the same scale.
", "rawMarkdown": "Ideally, there should be preprocessing that makes all images the same scale. Much like satellite images are orthorectified to make objects on the ground to be the same scale and angle (so square on ground is square on image no matter how camera looks at Earth -- at right or some different angle). The same should be done here -- knowing flight path, UAV elevation, camera characteristics, capture angle and so on, it is possible to make all images the same scale. \n", "votes": 5 }, { "id": 197023, "postDate": "2017-06-28T16:20:39.583Z", "content": "We labeled some images from train set according to scale and tried to train a CNN to regress a scale of the image. But it didn't work out. I recon this is due to high variation in terrain and not being able to estimate scale if you look at tiles with a small spatial context (even with my own eyes).
", "rawMarkdown": "We labeled some images from train set according to scale and tried to train a CNN to regress a scale of the image. But it didn't work out. I recon this is due to high variation in terrain and not being able to estimate scale if you look at tiles with a small spatial context (even with my own eyes)." }, { "id": 197028, "postDate": "2017-06-28T16:32:43.247Z", "content": "I tried a little different, and failed as expected.
", "rawMarkdown": "I tried a little different, and failed as expected." }, { "id": 197037, "postDate": "2017-06-28T16:51:15.843Z", "content": "agreed. including such data would make for a more robust approach.
", "rawMarkdown": "agreed. including such data would make for a more robust approach." } ] }, { "id": 196839, "postDate": "2017-06-28T08:16:13.257Z", "content": "Well done & congratz @outrunner.
", "rawMarkdown": "Well done & congratz @outrunner." }, { "id": 196830, "postDate": "2017-06-28T08:02:23.047Z", "content": "Сongratulations!
", "rawMarkdown": "Сongratulations!" }, { "id": 196769, "postDate": "2017-06-28T03:44:00.367Z", "content": "Congrats! Great solution!
\n\nA couple of questions: What was the difference in public/private score between r0.59 and r0.59*1.3 ?
\n\nDid you try other optimizers/did the learning rate schedule make a large difference in score?
\n\nFor the multi-scale model did you take larger images and resize them down to 300x300 or did you train seperate networks on different scaling of the 300x300 patches?
\n\nThanks, and congratulations on the win again.
", "rawMarkdown": "Congrats! Great solution!\n\nA couple of questions: What was the difference in public/private score between r0.59 and r0.59*1.3 ?\n\nDid you try other optimizers/did the learning rate schedule make a large difference in score?\n\nFor the multi-scale model did you take larger images and resize them down to 300x300 or did you train seperate networks on different scaling of the 300x300 patches?\n\nThanks, and congratulations on the win again.\n", "replies": [ { "id": 196780, "postDate": "2017-06-28T05:05:25.763Z", "content": "Sorry, do you mean different score or what make the score difference? I did not submit r0.59, I think it may be 14~15. About post processing: if a image predict in r0.59 = [10,10,100,100,20], then submit r0.59*1.3 = [10,10,70,130,20]
\n\nI tried other optimizers/learning rate schedule, just for observe the learning loss convergence.
\n\nI scaled down the original image (*0.9, ..., *0.6561), cut into 300x300, and use those patches together to train a network.
", "rawMarkdown": "Sorry, do you mean different score or what make the score difference? I did not submit r0.59, I think it may be 14~15. About post processing: if a image predict in r0.59 = [10,10,100,100,20], then submit r0.59*1.3 = [10,10,70,130,20]\n\nI tried other optimizers/learning rate schedule, just for observe the learning loss convergence.\n\nI scaled down the original image (*0.9, ..., *0.6561), cut into 300x300, and use those patches together to train a network.", "votes": 1 }, { "id": 196788, "postDate": "2017-06-28T05:21:11.980Z", "content": "Thanks for the info. You are correct I was wondering what the leaderboard score of r0.59 was. From 14~15 to ~11 is a big boost in score.
", "rawMarkdown": "Thanks for the info. You are correct I was wondering what the leaderboard score of r0.59 was. From 14~15 to ~11 is a big boost in score." }, { "id": 196791, "postDate": "2017-06-28T05:39:33.490Z", "content": "Additionally, post processing also add pups, in this case, it should be [10,10,70,130,24]. But the main contribution comes from juveniles/adult_females adjustment.
", "rawMarkdown": "Additionally, post processing also add pups, in this case, it should be [10,10,70,130,24]. But the main contribution comes from juveniles/adult_females adjustment.", "votes": 1 }, { "id": 196822, "postDate": "2017-06-28T07:34:11.220Z", "content": "Great result, congratulation with winning the competition!
\n\nInteresting, what was your score before post processing? It's amazing a relatively simple cnn managed to predict number of lion so well.
", "rawMarkdown": "Great result, congratulation with winning the competition!\n\nInteresting, what was your score before post processing? It's amazing a relatively simple cnn managed to predict number of lion so well." }, { "id": 196828, "postDate": "2017-06-28T07:56:48.270Z", "content": "I found a submission got 13.46393 private LB score is made by:
\n\nonly cut pups more than 1.8x adult_females after predict, and this affect a little in my experience.
\n\nSo, without post processing on other combination, maybe it could be around 13.
", "rawMarkdown": "I found a submission got 13.46393 private LB score is made by:\n\n - training scale: *1.23~0.66\n - testing scale: *0.48\n\nonly cut pups more than 1.8x adult_females after predict, and this affect a little in my experience.\n\nSo, without post processing on other combination, maybe it could be around 13.\n", "votes": 1 } ] }, { "id": 196766, "postDate": "2017-06-28T03:35:23.173Z", "content": "congratulations and amazing that you achieved #1 position with in four months ..
", "rawMarkdown": "congratulations and amazing that you achieved #1 position with in four months .. " }, { "id": 196761, "postDate": "2017-06-28T03:27:50.767Z", "content": "Congrats for the 1st position. Thanks for sharing the details. Its a nice approach.
\n\nI have a few questions -
\n\nYes, train the new FC layers first, then all layers.
I use linear activation at output layer. (because 3.)
Yes.
Training got exploded sometimes, but it is not problem. When testing, distinguish juveniles and adult_females is difficult. I think the model don't know what is \"relative\" size. Actually, in the training set, we also got a lot of human label noise at juveniles and adult_females.
Thanks.
", "rawMarkdown": "Thanks." }, { "id": 197134, "postDate": "2017-06-28T21:52:43.907Z", "content": "Why linear activation? is that mean that it can predict negative values?
", "rawMarkdown": "Why linear activation? is that mean that it can predict negative values?" }, { "id": 197206, "postDate": "2017-06-29T01:55:00.797Z", "content": "Why not?
", "rawMarkdown": "Why not?" }, { "id": 197257, "postDate": "2017-06-29T06:37:05.960Z", "content": "I mean it will work, but it can produce negative values as side affect? are you perform some sort of clipping?
\n\nWill it work with Relu activation?
", "rawMarkdown": "I mean it will work, but it can produce negative values as side affect? are you perform some sort of clipping?\n\nWill it work with Relu activation?" }, { "id": 197279, "postDate": "2017-06-29T07:36:51.033Z", "content": "Relu got problem at 0, and also need to do clip.
", "rawMarkdown": "Relu got problem at 0, and also need to do clip." }, { "id": 197284, "postDate": "2017-06-29T07:44:24.763Z", "content": "Can you elaborate on this? I mean that if final prediction will be [-10, 0, 0, 4, 5] it should be clipped to [0, 0, 0, 4, 5] (to make things cleaner I don't talking about gradient clipping).
", "rawMarkdown": "Can you elaborate on this? I mean that if final prediction will be [-10, 0, 0, 4, 5] it should be clipped to [0, 0, 0, 4, 5] (to make things cleaner I don't talking about gradient clipping)." }, { "id": 197288, "postDate": "2017-06-29T07:49:59.657Z", "content": "Yes, and if trained well, it is really hard to predict -10, since training labels never less then 0. In statistically, it is no need to clip even use linear, but we can simply clip the noise around 0 and get better performance.
", "rawMarkdown": "Yes, and if trained well, it is really hard to predict -10, since training labels never less then 0. In statistically, it is no need to clip even use linear, but we can simply clip the noise around 0 and get better performance." }, { "id": 199823, "postDate": "2017-07-06T11:51:31.807Z", "rawMarkdown": "", "isDeleted": true }, { "id": 199843, "postDate": "2017-07-06T12:38:03.150Z", "content": "58 training epochs is a example in the kernel. I train the new FC several epochs, observe the loss, then train more layers until the train loss close to 0.3. The total epochs before training the whole network is about 60(I think it is too slow due to my poor skill). And the whole network takes 27 epochs to got 0.126/0.247 train/validation loss with private LB 11.7.
", "rawMarkdown": "58 training epochs is a example in the kernel. I train the new FC several epochs, observe the loss, then train more layers until the train loss close to 0.3. The total epochs before training the whole network is about 60(I think it is too slow due to my poor skill). And the whole network takes 27 epochs to got 0.126/0.247 train/validation loss with private LB 11.7. " } ] }, { "id": 196749, "postDate": "2017-06-28T02:53:58.883Z", "content": "Congratulations, thanks for sharing your solution.
\n\nA little question is, can you share what computer did you use and how long did training 30 epochs took?
", "rawMarkdown": "Congratulations, thanks for sharing your solution.\n\nA little question is, can you share what computer did you use and how long did training 30 epochs took?", "replies": [ { "id": 196758, "postDate": "2017-06-28T03:17:14.697Z", "content": "i7 7770K, 16GB RAM, 1 GTX 1080.
\n\nFor single scale model, positive training patches = 14237, so total training patches = 54100 (positive*4*0.95). If one epoch means total training patches, it took about 30 mins a epoch. (Multi-scale takes 4 times) It took several days for training.
", "rawMarkdown": "i7 7770K, 16GB RAM, 1 GTX 1080.\n\nFor single scale model, positive training patches = 14237, so total training patches = 54100 (positive*4*0.95). If one epoch means total training patches, it took about 30 mins a epoch. (Multi-scale takes 4 times) It took several days for training.", "votes": 2 } ] }, { "id": 197468, "postDate": "2017-06-29T16:37:56.817Z", "content": "Great work! Thanks for sharing!
", "rawMarkdown": "Great work! Thanks for sharing!" }, { "id": 196760, "postDate": "2017-06-28T03:20:34.857Z", "content": "congrats and thanks for sharing.
", "rawMarkdown": "congrats and thanks for sharing." } ], "comments": [ { "id": 196751, "author_name": "Radu Stoicescu", "author_url": "", "post_date": "2017-06-28T03:02:21.327000", "content": "I'm proud to see my blob detection in the winning solution.
\n\nGood job!
", "votes": 8, "replies": [ { "id": 196783, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T05:13:23.207000", "content": "Thanks again.
", "votes": 4, "replies": [] } ] }, { "id": 197128, "author_name": "Darragh", "author_url": "", "post_date": "2017-06-28T21:43:51.967000", "content": "Outstanding work outrunner! Thanks for sharing; may I ask... did you have any intuition, before starting out, that this network would work ? And what do you think made the big difference in this architecture - was it the 1024 FC or the SGD or slow training or sthing else ... i am sure you tried lots of changes in this area. \nI gather you started with pretrained weights.
", "votes": 1, "replies": [ { "id": 197219, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T02:39:36.913000", "content": "Just intuition, and I think it would work undoubtedly even when I got trouble at the beginning. VGG is a basic CNN, I took the 1024 FC as a buffer between FCN and output. The size is according to RAM limit and model size, and intuitively I think it is enough. When I got scale problem, I tried more neurons and nothing happened. SGD is more stable than other aggressive optimizers in my work, maybe it is due to the large variation in output labels.
", "votes": 3, "replies": [] } ] }, { "id": 196752, "author_name": "sfx7rail", "author_url": "", "post_date": "2017-06-28T03:02:34.203000", "content": "Congratulations and Excellent work. Your kaggle ranking will be top 40.
", "votes": 1, "replies": [] }, { "id": 197692, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-30T03:08:24.533000", "content": "Something I could share about why misclassification on females and juveniles is worse more than I expected.
\n\n
I generated random dot for explanation. I wish the threshold between females and juveniles in the model could be the green line, but it is too naive. Then I think at least it should be the red line by statistically, but in fact the threshold is dotted red line. (It is about size feature, there are other features like lion distribution etc. in the model let the prediction better.)
\n\nI guess it is because in the small scale patch, lion count is larger, and the squared error even more. I tried add large scale training patch with more juveniles and it helps a little, but overall score is not significant.
", "votes": 2, "replies": [] }, { "id": 198656, "author_name": "erde", "author_url": "", "post_date": "2017-07-03T09:16:21.487000", "content": "Would you mind creating a visualization of the predictions that the network produces similar to the ones by Konstantin Lopuhin, e.g. using for i, p in enumerate(batch_of_predictions): scipy.misc.imsave('preds_%i.png' % i, normalize(np.concatenate(np.moveaxis(p, -1, 0))))? I think it would be interesting to see how well the network manages to tell the classes apart in various cases and what the predicted dots look like.
I don't quite understand what is batch_of_predictions?
", "votes": 0, "replies": [] }, { "id": 198671, "author_name": "erde", "author_url": "", "post_date": "2017-07-03T10:40:24.903000", "content": "The variable batch_of_predictions is be the output of your network. Usually, all operations in deep learning frameworks operate on batches of training examples. When you run the model to test it on a couple of images, you need to feed a batch of n≥1 images and you get n predictions back. This is how you can save all images from the batch. I basically just copied & modified a line of code from my solution.
I mean, in my network, predictions are only numbers. Or you want to see some testing patches with its output?
", "votes": 0, "replies": [] }, { "id": 198816, "author_name": "erde", "author_url": "", "post_date": "2017-07-03T19:17:09.003000", "content": "Oops, I misunderstood the way you create the trainYs. I thought your trainYs consisted of dots at the animal coordinates. Never mind then. I could imagine your solution works well because only predicting the total counts (not the locations) in a large window like that reduces noise from the label positions (sometimes the labels are even outside of the animals). For your method, imprecise label positions only cause noise at the borders of the patch.
By the way, this is how I verify the prediction: （more count more bright)\n
It is cool! How can you show the picture in this way? Could you please show some code on this?
", "votes": 0, "replies": [] }, { "id": 198901, "author_name": "outrunner", "author_url": "", "post_date": "2017-07-04T03:50:06.773000", "content": "for p in test_patches: p = p*f(count(p)) #make sure 0 < f < 1 or do some normalized processing
\n\nthen stitch them side by side
", "votes": 0, "replies": [] } ] }, { "id": 198600, "author_name": "MingT", "author_url": "", "post_date": "2017-07-03T05:14:55.613000", "content": "Congrats on winning. Great solution!! \nI noticed that the test was also run on the patches that were cut from the original test images and the overall count for an image is the sum of the counts from all the patches cut from the image. Since the patch cut is random, would a sea lion at the boundary of two patches be double-counted? How would the double counting be avoided in the solution?\nThanks!!
", "votes": 0, "replies": [ { "id": 198653, "author_name": "outrunner", "author_url": "", "post_date": "2017-07-03T08:54:14.403000", "content": "Ideally, output should be 0.5 if the lion has been cut in half. Because in the same situation at training stage, one patch is 1 and the other one is 0.
", "votes": 0, "replies": [] } ] }, { "id": 197253, "author_name": "Sander", "author_url": "", "post_date": "2017-06-29T06:33:34.793000", "content": "", "votes": 0, "replies": [] }, { "id": 197252, "author_name": "Sander", "author_url": "", "post_date": "2017-06-29T06:33:22.160000", "content": "Hi Outrunner,
\n\nCongratulations!
\n\nI'm not very familiar with Keras code. Could you explain the last layers of your neural network? What happens after the convolutional layers? Did you just use linear activations to generate a 5-layer heatmap and compared that to the GT heatmap using RMSE?
", "votes": 0, "replies": [ { "id": 197267, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T07:00:52.650000", "content": "The \"Patches looks like\" section in notebook, image is training patch, [1 0 3 0 1] is label (lion count per type in the patch), the last is 5 neurons fully-connected layer with linear output.
", "votes": 0, "replies": [] }, { "id": 197570, "author_name": "Sander", "author_url": "", "post_date": "2017-06-29T20:52:45.550000", "content": "Aha! So it's a kind of regression?
", "votes": 0, "replies": [] }, { "id": 197595, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T23:11:52.107000", "content": "I think so.
", "votes": 0, "replies": [] } ] }, { "id": 197238, "author_name": "Jerome Zhou", "author_url": "", "post_date": "2017-06-29T04:45:26.363000", "content": "Great job here!
", "votes": 0, "replies": [] }, { "id": 197143, "author_name": "mrgloom", "author_url": "", "post_date": "2017-06-28T22:17:25.833000", "content": "I use custom data generator.
I trained a 200x200 and it performed worse. Maybe it is because 300 contains more relationship information between pups and adult_females. Flipping(rotating) 4 times at prediction time improved 0.04 RMSE. Using only convolutional layer in VGG is size free.
Just a normal loss function.
You can reference the table in the section.
Respect! Never underestimate the basics.
", "votes": 0, "replies": [] }, { "id": 197074, "author_name": "eagle4", "author_url": "", "post_date": "2017-06-28T18:22:34.977000", "content": "@outrunner\nCongrats and thanks for the code and the explanations.
\n\nAs you requested, please find attached the bounding boxes. the first batch are the 5000 first bboxes manually labeled and the other 75 to 80,000 are the ones generated using SSD (no guarantee on the quality and some subadult male disappeared in the mix).
\n\nKeep us informed on your progress with them please.
\n\nI generated them as I started this competition as standard detection problem but later switched to a segmentation where they were not the best idea and dragged me to the bottom of the leader-board: I was more concerned by crowed area where bboxes are an handicap rather than image scale...
", "votes": 0, "replies": [ { "id": 197214, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T02:14:31.817000", "content": "Thanks. But I may try it when I have time or need to do farther research on this topic.
", "votes": 0, "replies": [] } ] }, { "id": 196987, "author_name": "erde", "author_url": "", "post_date": "2017-06-28T14:38:50.380000", "content": "It is interesting that the model trained at 1.0x scale had a much better performance being evaluated on the test set at 0.67x scale than on 1.0x. What might be the reason for this? Perhaps the receptive field of VGG16 is just not big enough to actually recognize the animals at 1.0x scale. VGG16 has an effective receptive field size of ~220px at the output neurons, but the largest adult males are up to ~260-270px in length. Also the largest distance of the pubs to other animals is about 250px.
", "votes": 0, "replies": [ { "id": 197003, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T15:27:55.667000", "content": "I think it is typically because the testing image's sea lion is larger than training in average. In my opinion, the model don't know what sea lion is and where it is.
", "votes": 0, "replies": [] }, { "id": 197088, "author_name": "erde", "author_url": "", "post_date": "2017-06-28T19:22:08.303000", "content": "> In my opinion, the model don't know what sea lion is and where it is.
\n\nI agree. :) 'Not recognizing' was just shorthand for 'it might be harder to find distinguishing features if the animals are often only partially in sight for the network'. My info was incomplete/wrong anyway though: VGG's receptive field is 224 × 224 with zero-padding, but 404 × 404 if the net is applied convolutionally^1.
", "votes": 0, "replies": [] }, { "id": 197204, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T01:53:17.547000", "content": "I haven't studied receptive field in detail yet. So, thanks for sharing.
", "votes": 0, "replies": [] } ] }, { "id": 196976, "author_name": "erde", "author_url": "", "post_date": "2017-06-28T14:12:33.797000", "content": "Fantastic result, congrats!
\n\nAm I understanding correctly that the multi-scale model is essentially heavy data augmentation with different scales?
", "votes": 0, "replies": [] }, { "id": 196967, "author_name": "LivingProgram", "author_url": "", "post_date": "2017-06-28T13:48:30.440000", "content": "Such a simple solution. It really goes to show how far machine learning, convolutional neural networks, and a perfect implementation, application, and execution can take you. Congrats @outrunner on 1st place, and much thanks for the informative write-up.
", "votes": 0, "replies": [ { "id": 196970, "author_name": "LivingProgram", "author_url": "", "post_date": "2017-06-28T13:52:27.350000", "content": "I also had a question about the values on your table, I don't quite understand why a higher value on the table like the values in your final submission column yielded better results. What is the significance of average of juveniles# / (adult_females# + juveniles#)?
", "votes": 1, "replies": [] }, { "id": 196982, "author_name": "erde", "author_url": "", "post_date": "2017-06-28T14:27:33.503000", "content": "I think the numbers just show that on the test set outrunner's models predicted much fewer juveniles vs adult_females than during training, thus he/she corrected roughly for that ratio (with great success). E.g. the the patches with 46-50 juveniles had a number of 0.93 during training with a single zoom level (1x), but on the test set only 0.68, so it was corrected to 0.76. In that particular case it would have been interesting to see the performance of the r0.59 model without post-processing on the public LB.
\n\nThis might imply that the test set had different juvenile/adult_female ratios than the training set.
", "votes": 0, "replies": [] }, { "id": 196997, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T15:14:00.110000", "content": "Sorry for being unclear and thanks to erde. It is the ratio between juveniles and juveniles + adult_females.
\n\nI think the ratio problem comes from scale problem. The CNN basically is a statistical model, simply distinguish juveniles and adult_females by size. Scale down image can get more juveniles, but the problem is if out of range, total lions count will go down. For example:
\n\nThis is the predict result of test/12690 on multi-scale model with different test scale. The juveniles + adult_females keeps around 300 for 0.59~0.43, and began to decrease from 0.39.
\n\nSo, all you are interesting in without post processing's performance. It is sure if I scale down more. But I don't want to take the risk if sea lions in private set are smaller then public.
\n\n\n\n\nthe multi-scale model is essentially heavy data augmentation with different scales?
\n
Yes, and single scale model's keeping number scale range is narrower.
", "votes": 1, "replies": [] }, { "id": 197004, "author_name": "LivingProgram", "author_url": "", "post_date": "2017-06-28T15:29:00.683000", "content": "Ah I see so scaled down images tend to have more juveniles than adult females versus not scaled down. If it were possible to teach the method how to compare the sizes of sea lions within a single image's scale, then the model could learn to compare the juvenile's relative size to female's relative size and your post processing trick might not have been needed? Maybe if first the image is processed for females then those features are fed to another network that searches for juveniles, the features can be reused. I think your post processing basically gave the model domain knowledge about the problem that could be replaced with a different architecture, would be interesting to test this!
", "votes": 0, "replies": [] }, { "id": 197007, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T15:39:33.807000", "content": "Sure, and it is not easy work. By the way, if they have grounding truth bounding box, I want to train a object detection for image scale detect by sea lion's size, just a thought.
", "votes": 0, "replies": [] }, { "id": 197012, "author_name": "Cut Onion", "author_url": "", "post_date": "2017-06-28T15:59:07.277000", "content": "Ideally, there should be preprocessing that makes all images the same scale. Much like satellite images are orthorectified to make objects on the ground to be the same scale and angle (so square on ground is square on image no matter how camera looks at Earth -- at right or some different angle). The same should be done here -- knowing flight path, UAV elevation, camera characteristics, capture angle and so on, it is possible to make all images the same scale.
", "votes": 5, "replies": [] }, { "id": 197023, "author_name": "Artem.Sanakoev", "author_url": "", "post_date": "2017-06-28T16:20:39.583000", "content": "We labeled some images from train set according to scale and tried to train a CNN to regress a scale of the image. But it didn't work out. I recon this is due to high variation in terrain and not being able to estimate scale if you look at tiles with a small spatial context (even with my own eyes).
", "votes": 0, "replies": [] }, { "id": 197028, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T16:32:43.247000", "content": "I tried a little different, and failed as expected.
", "votes": 0, "replies": [] }, { "id": 197037, "author_name": "zero zero", "author_url": "", "post_date": "2017-06-28T16:51:15.843000", "content": "agreed. including such data would make for a more robust approach.
", "votes": 0, "replies": [] } ] }, { "id": 196839, "author_name": "Faron", "author_url": "", "post_date": "2017-06-28T08:16:13.257000", "content": "Well done & congratz @outrunner.
", "votes": 0, "replies": [] }, { "id": 196830, "author_name": "Vladimir Savinov", "author_url": "", "post_date": "2017-06-28T08:02:23.047000", "content": "Сongratulations!
", "votes": 0, "replies": [] }, { "id": 196769, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2017-06-28T03:44:00.367000", "content": "Congrats! Great solution!
\n\nA couple of questions: What was the difference in public/private score between r0.59 and r0.59*1.3 ?
\n\nDid you try other optimizers/did the learning rate schedule make a large difference in score?
\n\nFor the multi-scale model did you take larger images and resize them down to 300x300 or did you train seperate networks on different scaling of the 300x300 patches?
\n\nThanks, and congratulations on the win again.
", "votes": 0, "replies": [ { "id": 196780, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T05:05:25.763000", "content": "Sorry, do you mean different score or what make the score difference? I did not submit r0.59, I think it may be 14~15. About post processing: if a image predict in r0.59 = [10,10,100,100,20], then submit r0.59*1.3 = [10,10,70,130,20]
\n\nI tried other optimizers/learning rate schedule, just for observe the learning loss convergence.
\n\nI scaled down the original image (*0.9, ..., *0.6561), cut into 300x300, and use those patches together to train a network.
", "votes": 1, "replies": [] }, { "id": 196788, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2017-06-28T05:21:11.980000", "content": "Thanks for the info. You are correct I was wondering what the leaderboard score of r0.59 was. From 14~15 to ~11 is a big boost in score.
", "votes": 0, "replies": [] }, { "id": 196791, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T05:39:33.490000", "content": "Additionally, post processing also add pups, in this case, it should be [10,10,70,130,24]. But the main contribution comes from juveniles/adult_females adjustment.
", "votes": 1, "replies": [] }, { "id": 196822, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2017-06-28T07:34:11.220000", "content": "Great result, congratulation with winning the competition!
\n\nInteresting, what was your score before post processing? It's amazing a relatively simple cnn managed to predict number of lion so well.
", "votes": 0, "replies": [] }, { "id": 196828, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T07:56:48.270000", "content": "I found a submission got 13.46393 private LB score is made by:
\n\nonly cut pups more than 1.8x adult_females after predict, and this affect a little in my experience.
\n\nSo, without post processing on other combination, maybe it could be around 13.
", "votes": 1, "replies": [] } ] }, { "id": 196766, "author_name": "Kishore M", "author_url": "", "post_date": "2017-06-28T03:35:23.173000", "content": "congratulations and amazing that you achieved #1 position with in four months ..
", "votes": 0, "replies": [] }, { "id": 196761, "author_name": "KapilYadav", "author_url": "", "post_date": "2017-06-28T03:27:50.767000", "content": "Congrats for the 1st position. Thanks for sharing the details. Its a nice approach.
\n\nI have a few questions -
\n\nYes, train the new FC layers first, then all layers.
I use linear activation at output layer. (because 3.)
Yes.
Training got exploded sometimes, but it is not problem. When testing, distinguish juveniles and adult_females is difficult. I think the model don't know what is \"relative\" size. Actually, in the training set, we also got a lot of human label noise at juveniles and adult_females.
Thanks.
", "votes": 0, "replies": [] }, { "id": 197134, "author_name": "mrgloom", "author_url": "", "post_date": "2017-06-28T21:52:43.907000", "content": "Why linear activation? is that mean that it can predict negative values?
", "votes": 0, "replies": [] }, { "id": 197206, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T01:55:00.797000", "content": "Why not?
", "votes": 0, "replies": [] }, { "id": 197257, "author_name": "mrgloom", "author_url": "", "post_date": "2017-06-29T06:37:05.960000", "content": "I mean it will work, but it can produce negative values as side affect? are you perform some sort of clipping?
\n\nWill it work with Relu activation?
", "votes": 0, "replies": [] }, { "id": 197279, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T07:36:51.033000", "content": "Relu got problem at 0, and also need to do clip.
", "votes": 0, "replies": [] }, { "id": 197284, "author_name": "mrgloom", "author_url": "", "post_date": "2017-06-29T07:44:24.763000", "content": "Can you elaborate on this? I mean that if final prediction will be [-10, 0, 0, 4, 5] it should be clipped to [0, 0, 0, 4, 5] (to make things cleaner I don't talking about gradient clipping).
", "votes": 0, "replies": [] }, { "id": 197288, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-29T07:49:59.657000", "content": "Yes, and if trained well, it is really hard to predict -10, since training labels never less then 0. In statistically, it is no need to clip even use linear, but we can simply clip the noise around 0 and get better performance.
", "votes": 0, "replies": [] }, { "id": 199823, "author_name": "", "author_url": "", "post_date": "2017-07-06T11:51:31.807000", "content": "", "votes": 0, "replies": [] }, { "id": 199843, "author_name": "outrunner", "author_url": "", "post_date": "2017-07-06T12:38:03.150000", "content": "58 training epochs is a example in the kernel. I train the new FC several epochs, observe the loss, then train more layers until the train loss close to 0.3. The total epochs before training the whole network is about 60(I think it is too slow due to my poor skill). And the whole network takes 27 epochs to got 0.126/0.247 train/validation loss with private LB 11.7.
", "votes": 0, "replies": [] } ] }, { "id": 196749, "author_name": "InfiniteWing", "author_url": "", "post_date": "2017-06-28T02:53:58.883000", "content": "Congratulations, thanks for sharing your solution.
\n\nA little question is, can you share what computer did you use and how long did training 30 epochs took?
", "votes": 0, "replies": [ { "id": 196758, "author_name": "outrunner", "author_url": "", "post_date": "2017-06-28T03:17:14.697000", "content": "i7 7770K, 16GB RAM, 1 GTX 1080.
\n\nFor single scale model, positive training patches = 14237, so total training patches = 54100 (positive*4*0.95). If one epoch means total training patches, it took about 30 mins a epoch. (Multi-scale takes 4 times) It took several days for training.
", "votes": 2, "replies": [] } ] }, { "id": 197468, "author_name": "Josue De Santiago", "author_url": "", "post_date": "2017-06-29T16:37:56.817000", "content": "Great work! Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 196760, "author_name": "zero zero", "author_url": "", "post_date": "2017-06-28T03:20:34.857000", "content": "congrats and thanks for sharing.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "196735": "Thanks to everyone. I love this competition.\n\n[Here is my solution.][1] \n\nI use VGG16 without top, add FC-1024, and FC-5 with linear output.\n\nSGD optimization and mean_squared_error loss.\n\nAnd post processing for better regression.\n\nI think it is a simple way to count. Looking forward to hearing your thoughts, thanks.\n\n [1]: https://www.kaggle.com/outrunner/use-keras-to-count-sea-lions/notebook", "196751": "I'm proud to see my blob detection in the winning solution.\n\nGood job!", "197128": "Outstanding work outrunner! Thanks for sharing; may I ask... did you have any intuition, before starting out, that this network would work ? And what do you think made the big difference in this architecture - was it the 1024 FC or the SGD or slow training or sthing else ... i am sure you tried lots of changes in this area. \nI gather you started with pretrained weights. ", "196752": "Congratulations and Excellent work. Your kaggle ranking will be top 40.", "197692": "Something I could share about why misclassification on females and juveniles is worse more than I expected.\n\n![enter image description here][1]\n\nI generated random dot for explanation. I wish the threshold between females and juveniles in the model could be the green line, but it is too naive. Then I think at least it should be the red line by statistically, but in fact the threshold is dotted red line. (It is about size feature, there are other features like lion distribution etc. in the model let the prediction better.)\n\nI guess it is because in the small scale patch, lion count is larger, and the squared error even more. I tried add large scale training patch with more juveniles and it helps a little, but overall score is not significant.\n\n [1]: http://i.imgur.com/rJBOLir.gif", "198656": "Would you mind creating a visualization of the predictions that the network produces similar to the ones by Konstantin Lopuhin, e.g. using `for i, p in enumerate(batch_of_predictions): scipy.misc.imsave('preds_%i.png' % i, normalize(np.concatenate(np.moveaxis(p, -1, 0))))`? I think it would be interesting to see how well the network manages to tell the classes apart in various cases and what the predicted dots look like.", "198600": "Congrats on winning. Great solution!! \nI noticed that the test was also run on the patches that were cut from the original test images and the overall count for an image is the sum of the counts from all the patches cut from the image. Since the patch cut is random, would a sea lion at the boundary of two patches be double-counted? How would the double counting be avoided in the solution?\nThanks!!", "197253": "", "197252": "Hi Outrunner,\n\nCongratulations!\n\nI'm not very familiar with Keras code. Could you explain the last layers of your neural network? What happens after the convolutional layers? Did you just use linear activations to generate a 5-layer heatmap and compared that to the GT heatmap using RMSE?", "197238": "Great job here!", "197143": "1. In your kernel you use only 1 image, how training was setup in practice? it was something like Keras ImageDataGenerator or you load all data to RAM?\n2. Does LB score depend on tile size? How you handle images at prediction time, i.e. with tile overlapping? In my opinion in case of pretrained network it should be about 'original' size, i.e. 224x224 for VGG.\n3. Why mean_squared_error was used that is MSE and in competition we have RMSE?\n4. What LB score you get without tricks described in Experience section?", "197139": "Respect! Never underestimate the basics.", "197074": "@outrunner\nCongrats and thanks for the code and the explanations.\n\nAs you requested, please find attached the bounding boxes. the first batch are the 5000 first bboxes manually labeled and the other 75 to 80,000 are the ones generated using SSD (no guarantee on the quality and some subadult male disappeared in the mix).\n\nKeep us informed on your progress with them please.\n\nI generated them as I started this competition as standard detection problem but later switched to a segmentation where they were not the best idea and dragged me to the bottom of the leader-board: I was more concerned by crowed area where bboxes are an handicap rather than image scale...", "196987": "It is interesting that the model trained at 1.0x scale had a much better performance being evaluated on the test set at 0.67x scale than on 1.0x. What might be the reason for this? Perhaps the receptive field of VGG16 is just not big enough to actually recognize the animals at 1.0x scale. VGG16 has an effective receptive field size of ~220px at the output neurons, but the largest adult males are up to ~260-270px in length. Also the largest distance of the pubs to other animals is about 250px.", "196976": "Fantastic result, congrats!\n\nAm I understanding correctly that the multi-scale model is essentially heavy data augmentation with different scales?", "196967": "Such a simple solution. It really goes to show how far machine learning, convolutional neural networks, and a perfect implementation, application, and execution can take you. **Congrats @outrunner on 1st place**, and much thanks for the informative write-up. ", "196839": "Well done & congratz @outrunner.", "196830": "Сongratulations!", "196769": "Congrats! Great solution!\n\nA couple of questions: What was the difference in public/private score between r0.59 and r0.59*1.3 ?\n\nDid you try other optimizers/did the learning rate schedule make a large difference in score?\n\nFor the multi-scale model did you take larger images and resize them down to 300x300 or did you train seperate networks on different scaling of the 300x300 patches?\n\nThanks, and congratulations on the win again.\n", "196766": "congratulations and amazing that you achieved #1 position with in four months .. ", "196761": "Congrats for the 1st position. Thanks for sharing the details. Its a nice approach.\n\nI have a few questions - \n\n 1. Did you make all the VGG16 layers trainable ? ( I tried VGG16, but I couldnt make it stop overfit on train data. :( )\n 2. How were the final output of linear layers compared ? ( I guess, a sigmoid layer was used ... )\n 3. Just a clarification - In cases when a patch has multiple sea-lions, the network will output high values for their respective nodes in the last layer, right? \n 4. Did you face any difficulty for patches where sea-lions were all crowded in one place?", "196749": "Congratulations, thanks for sharing your solution.\n\nA little question is, can you share what computer did you use and how long did training 30 epochs took?", "197468": "Great work! Thanks for sharing!", "196760": "congrats and thanks for sharing." } }