My solution was mostly based on this version of @martinpiotte amazing work in the previous whale competition. I noticed many comments mentioning the variability of results obtained from Siamese networks, which I thought might be advantageous.
\n\nAt first I made no modifications to Martin's network, training it from scratch for 500 epochs. At that point it would consistently get between 0.895-0.905 on lb. Training for another 5-10 epochs resulted in a similar score but the distribution of predicted whales was much more variable though the scores remained very close. I figured this was due to randomness in how the augmentations were applied so I trained several dozen versions of this model with minor variations--changing image size from 224 up to 600 both grayscale and rgb. I would have added TTA, but I do not know how to do it in keras. After this I had about 40 sets of predictions which I used for a hard-voting scheme at each whale position. A simple average of the prediction scores from each model could get a 0.935 with the right threshold and adding this to the voting resulted in 0.941 on public lb.
\n\nAt this point the scores were not increasing and it seemed I had reached the limit of what this network was capable of distinguishing so I added classification and prototype results to the voting. I find the prototype approach really interesting so I wish I had more time to work on it, I was able to get a single prototype model up to only 0.872. Adding several of these other models to the vote got the final score.
\n\nGreat work by everyone, I learned so much and I can't wait to see how you all approached this.
", "messages": [ { "id": "481017", "postDate": "03/01/2019 01:11:08", "content": "My solution was mostly based on this version of @martinpiotte amazing work in the previous whale competition. I noticed many comments mentioning the variability of results obtained from Siamese networks, which I thought might be advantageous.
\n\nAt first I made no modifications to Martin's network, training it from scratch for 500 epochs. At that point it would consistently get between 0.895-0.905 on lb. Training for another 5-10 epochs resulted in a similar score but the distribution of predicted whales was much more variable though the scores remained very close. I figured this was due to randomness in how the augmentations were applied so I trained several dozen versions of this model with minor variations--changing image size from 224 up to 600 both grayscale and rgb. I would have added TTA, but I do not know how to do it in keras. After this I had about 40 sets of predictions which I used for a hard-voting scheme at each whale position. A simple average of the prediction scores from each model could get a 0.935 with the right threshold and adding this to the voting resulted in 0.941 on public lb.
\n\nAt this point the scores were not increasing and it seemed I had reached the limit of what this network was capable of distinguishing so I added classification and prototype results to the voting. I find the prototype approach really interesting so I wish I had more time to work on it, I was able to get a single prototype model up to only 0.872. Adding several of these other models to the vote got the final score.
\n\nGreat work by everyone, I learned so much and I can't wait to see how you all approached this.
", "rawMarkdown": "My solution was mostly based on this [version][1] of @martinpiotte amazing work in the previous whale competition. I noticed many comments mentioning the variability of results obtained from Siamese networks, which I thought might be advantageous. \n\nAt first I made no modifications to Martin's network, training it from scratch for 500 epochs. At that point it would consistently get between 0.895-0.905 on lb. Training for another 5-10 epochs resulted in a similar score but the distribution of predicted whales was much more variable though the scores remained very close. I figured this was due to randomness in how the augmentations were applied so I trained several dozen versions of this model with minor variations--changing image size from 224 up to 600 both grayscale and rgb. I would have added TTA, but I do not know how to do it in keras. After this I had about 40 sets of predictions which I used for a hard-voting scheme at each whale position. A simple average of the prediction scores from each model could get a 0.935 with the right threshold and adding this to the voting resulted in 0.941 on public lb. \n\nAt this point the scores were not increasing and it seemed I had reached the limit of what this network was capable of distinguishing so I added classification and prototype results to the voting. I find the prototype approach really interesting so I wish I had more time to work on it, I was able to get a single prototype model up to only 0.872. Adding several of these other models to the vote got the final score.\n\nGreat work by everyone, I learned so much and I can't wait to see how you all approached this.\n\n\n [1]: https://www.kaggle.com/seesee/siamese-pretrained-0-822", "votes": null }, { "id": "481030", "postDate": "03/01/2019 01:31:16", "content": "Congrats !
", "rawMarkdown": "Congrats !", "votes": null }, { "id": "481122", "postDate": "03/01/2019 04:16:30", "content": "Congratulations @interneuron !!
", "rawMarkdown": "Congratulations @interneuron !!", "votes": null }, { "id": "481124", "postDate": "03/01/2019 04:21:57", "content": "Thanks!\nI guess its 23rd now lol
", "rawMarkdown": "Thanks!\nI guess its 23rd now lol", "votes": null }, { "id": "481247", "postDate": "03/01/2019 07:35:21", "content": "Congratulations @interneuron. Keep up the great work going.
", "rawMarkdown": "Congratulations @interneuron. Keep up the great work going.", "votes": null }, { "id": "481298", "postDate": "03/01/2019 08:18:59", "content": "Congrats @interneuron and thanks for sharing.
", "rawMarkdown": "Congrats @interneuron and thanks for sharing.", "votes": null }, { "id": "481607", "postDate": "03/01/2019 15:55:06", "content": "Congrats !\nI had the same problem and my mpiotte predictions were also stucked into the .90 range.\nI used several leaks (+.02) to get in the .92+ range but how did you diversify your pool of models to get to .935 ?
", "rawMarkdown": "Congrats !\nI had the same problem and my mpiotte predictions were also stucked into the .90 range.\nI used several leaks (+.02) to get in the .92+ range but how did you diversify your pool of models to get to .935 ?", "votes": null }, { "id": "481650", "postDate": "03/01/2019 17:02:49", "content": "Thanks eagle!
\n\nThe mpiotte siamese single models did indeed stall out around 0.9, the highest I got from any one of them was a 0.907, which was a 512x512 RGB. After the initial 500 epochs with 384 grayscale I continued training it with the different image parameters between 5 and 30 epochs at a time then saving the score array and making a submission, it got to about 1400 epochs before I gave up on it. The 0.935 score was from an average of the highest two dozen or so scores (I can check the exact number if you like) with a threshold of 0.65. Adding more that scored below 0.9 did not help, but I didn't play with the threshold that much.
", "rawMarkdown": "Thanks eagle! \n\nThe mpiotte siamese single models did indeed stall out around 0.9, the highest I got from any one of them was a 0.907, which was a 512x512 RGB. After the initial 500 epochs with 384 grayscale I continued training it with the different image parameters between 5 and 30 epochs at a time then saving the score array and making a submission, it got to about 1400 epochs before I gave up on it. The 0.935 score was from an average of the highest two dozen or so scores (I can check the exact number if you like) with a threshold of 0.65. Adding more that scored below 0.9 did not help, but I didn't play with the threshold that much.", "votes": null }, { "id": "481822", "postDate": "03/01/2019 22:16:03", "content": "Congrats, this architecture is pretty interesting.
", "rawMarkdown": "Congrats, this architecture is pretty interesting.", "votes": null } ], "comments": [ { "id": 481030, "author_name": "jmourad100", "author_url": "", "post_date": "03/01/2019 01:31:16", "content": "Congrats !
", "votes": null, "replies": [] }, { "id": 481122, "author_name": "hwasiti", "author_url": "", "post_date": "03/01/2019 04:16:30", "content": "Congratulations @interneuron !!
", "votes": null, "replies": [] }, { "id": 481124, "author_name": "interneuron", "author_url": "", "post_date": "03/01/2019 04:21:57", "content": "Thanks!\nI guess its 23rd now lol
", "votes": null, "replies": [] }, { "id": 481247, "author_name": "karthik7395", "author_url": "", "post_date": "03/01/2019 07:35:21", "content": "Congratulations @interneuron. Keep up the great work going.
", "votes": null, "replies": [] }, { "id": 481298, "author_name": "sheriytm", "author_url": "", "post_date": "03/01/2019 08:18:59", "content": "Congrats @interneuron and thanks for sharing.
", "votes": null, "replies": [] }, { "id": 481607, "author_name": "chabir", "author_url": "", "post_date": "03/01/2019 15:55:06", "content": "Congrats !\nI had the same problem and my mpiotte predictions were also stucked into the .90 range.\nI used several leaks (+.02) to get in the .92+ range but how did you diversify your pool of models to get to .935 ?
", "votes": null, "replies": [ { "id": 481650, "author_name": "interneuron", "author_url": "", "post_date": "03/01/2019 17:02:49", "content": "Thanks eagle!
\n\nThe mpiotte siamese single models did indeed stall out around 0.9, the highest I got from any one of them was a 0.907, which was a 512x512 RGB. After the initial 500 epochs with 384 grayscale I continued training it with the different image parameters between 5 and 30 epochs at a time then saving the score array and making a submission, it got to about 1400 epochs before I gave up on it. The 0.935 score was from an average of the highest two dozen or so scores (I can check the exact number if you like) with a threshold of 0.65. Adding more that scored below 0.9 did not help, but I didn't play with the threshold that much.
", "votes": null, "replies": [] } ] }, { "id": 481822, "author_name": "titericz", "author_url": "", "post_date": "03/01/2019 22:16:03", "content": "Congrats, this architecture is pretty interesting.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "481017": "My solution was mostly based on this [version][1] of @martinpiotte amazing work in the previous whale competition. I noticed many comments mentioning the variability of results obtained from Siamese networks, which I thought might be advantageous. \n\nAt first I made no modifications to Martin's network, training it from scratch for 500 epochs. At that point it would consistently get between 0.895-0.905 on lb. Training for another 5-10 epochs resulted in a similar score but the distribution of predicted whales was much more variable though the scores remained very close. I figured this was due to randomness in how the augmentations were applied so I trained several dozen versions of this model with minor variations--changing image size from 224 up to 600 both grayscale and rgb. I would have added TTA, but I do not know how to do it in keras. After this I had about 40 sets of predictions which I used for a hard-voting scheme at each whale position. A simple average of the prediction scores from each model could get a 0.935 with the right threshold and adding this to the voting resulted in 0.941 on public lb. \n\nAt this point the scores were not increasing and it seemed I had reached the limit of what this network was capable of distinguishing so I added classification and prototype results to the voting. I find the prototype approach really interesting so I wish I had more time to work on it, I was able to get a single prototype model up to only 0.872. Adding several of these other models to the vote got the final score.\n\nGreat work by everyone, I learned so much and I can't wait to see how you all approached this.\n\n\n [1]: https://www.kaggle.com/seesee/siamese-pretrained-0-822", "481030": "Congrats !", "481122": "Congratulations @interneuron !!", "481124": "Thanks!\nI guess its 23rd now lol", "481247": "Congratulations @interneuron. Keep up the great work going.", "481298": "Congrats @interneuron and thanks for sharing.", "481607": "Congrats !\nI had the same problem and my mpiotte predictions were also stucked into the .90 range.\nI used several leaks (+.02) to get in the .92+ range but how did you diversify your pool of models to get to .935 ?", "481650": "Thanks eagle! \n\nThe mpiotte siamese single models did indeed stall out around 0.9, the highest I got from any one of them was a 0.907, which was a 512x512 RGB. After the initial 500 epochs with 384 grayscale I continued training it with the different image parameters between 5 and 30 epochs at a time then saving the score array and making a submission, it got to about 1400 epochs before I gave up on it. The 0.935 score was from an average of the highest two dozen or so scores (I can check the exact number if you like) with a threshold of 0.65. Adding more that scored below 0.9 did not help, but I didn't play with the threshold that much.", "481822": "Congrats, this architecture is pretty interesting." }, "source": "meta" }