![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/481336/11475/Cropping.png\")
First, my congrats to the winners! And, thanks to the organizers! Also, thanks to all peoples who make great tools for deep machine learning in C++, as Dlib (did you ever heard about it?) and OpenCV!
\n\nSo, to be brief. My final training setup could be found here. My best single model architecture definition (it is ResNet variation with 512x192x1 input and 128 feature vector output) could be found here. I have started from public LB 0.748 and finished at private LB 0.948. How this progress has been done:
\n\n1) My training pipeline was assembled when the test competition was held. So, as first attemp, I just retrain my old model on new data (\"new_whales\" set were removed) and have got 0.748. By playing with identification threshold have improved the score to 0.78. Already good, thanks to great metric_loss layer from DLib.
\n\n2) Have taken into account Martin's Piotte findings. In particular, have replaced RGB input to grayscale (with centering and noralization). And have removed all whales with a single sample from the trainnig set. Public score has growed up to 0.84-0.85.
\n\n3) Ok, what to do next? Let's try experimentation with data augmentation (scale,shift,rotation,perspective distortion, but not horizontal flip yet) and train/validation division. Few models/iterations later, by voting among best single submissions (voting code, it is Matlab's script, could be found here) finally come to 0.86 score.
\n\n4) What else Martin recommend? Flukes cropping. But I am not familiar with Python, so I did not want to use any fluke detection model that competitors have done. Ok, I have take this as challenge. And maybe it is not original way, but I have written a code that generates attention heatmap of my model for a picture (by sliding rectangle on top of the picture), then binarizes this heatmap, aligns binary mask by PCA (among mask's pixels coordiantes) and finally crops and resize image. When I generate such crops (heatmaps from my best at those time model) and retrain best model on cropped data, I have got 0.89 score. Interesting that if I have trained new model on cropped flukes sometimes I have got around 0.85. So two steps trainng (first on original pictures, than on cropped) was selected as important for me.
\n\n
5) What to do next? Somehow increase size of training set? So it was time to horizontal flipping (at those time i have already saw by my eyes many of training examples, so I came to understanding that almost all whales photographed without mirroring, it is good because we can mirror them and add this mirrored whales as new classes). This effectivelly doubled training set size. Have checked scrore, got improvement from 0.86 to 0.88 (this is without cropping, because I have already known that cropping definetelly will improves result, so I have performed experimenations without it). I have also made some experimentations with image negatives (multiplied by -1.0) and new classes compositions from left and right pars of different whales. But, none of them (except horizontal flipp) have improved score.
\n\n6) Where else we can get more trainnig examples? At the playground competitions of course. I have collected training set from playground competition (also only classes with more than one samples per whale), added it to my training set. And, LB score have grown from 0.88 to 0.907 (single model, yet without picture cropping). Maybe some test examples resides in the playground trainng set?? Did not check it. But I have found that there is a lot of noise in training data, and it became even more after union with playground data. Partially this noise could be filtered by good model and some hand labeling (note, taht this is not prohibited as we label training data, not test). So, I have wrote another tool to cleanup training data in semi-automatic manner. This allows me to boost my score to 0.928.
\n\n7) It was time to tune prediction procedure. Untill that moment I have only predicted top 1 label. My classifier produce a 128-dimensional vector of loats for a picture (so each whale could be represented as biometric template with the size of 512 bytes). So, after each model has been trained, I generated enrollment templates. One template for each picture in training set (for each \"new_whale\" too, as it improves results). And at submission generation step, for each test picture I compared the distances (Euclidean) between test picture's identification template and all of the enrollment templates. Then sort all of the distances and made decision about particular test sample. If minimum distance between templates was lower than a particular threshold, sample got label from enrollment set, otherwise sample got \"new_whale\" label. But competition's score also rewards you even if right prediction was 2nd, 3rd, or even 5th. So, we can improve score by submit all predictions in distance ascending order. That way I have improved my LB from 0.928 to 0.938 (my best single model).
\n\n8) What's next? Ensembling, of cource! Even first ensemble (consist of four different networks (0.938, 0.935, 0.854, 0.876) allows to get 0.943 LB score. I am ensembling them by simple concatenation of templates (also have tried to train head on this features, but without any luck). At this moment I have started to train different modifications of my base solution with different number of filters and other hyperparameters. At a final point I was able to stack 25 different models, and got public LB 0.948. Some combinations works better than the others. My final submission has been made by 9-models ResNet ensemble, and public LB was 0.953.
", "messages": [ { "id": "481336", "postDate": "03/01/2019 09:25:23", "content": "First, my congrats to the winners! And, thanks to the organizers! Also, thanks to all peoples who make great tools for deep machine learning in C++, as Dlib (did you ever heard about it?) and OpenCV!
\n\nSo, to be brief. My final training setup could be found here. My best single model architecture definition (it is ResNet variation with 512x192x1 input and 128 feature vector output) could be found here. I have started from public LB 0.748 and finished at private LB 0.948. How this progress has been done:
\n\n1) My training pipeline was assembled when the test competition was held. So, as first attemp, I just retrain my old model on new data (\"new_whales\" set were removed) and have got 0.748. By playing with identification threshold have improved the score to 0.78. Already good, thanks to great metric_loss layer from DLib.
\n\n2) Have taken into account Martin's Piotte findings. In particular, have replaced RGB input to grayscale (with centering and noralization). And have removed all whales with a single sample from the trainnig set. Public score has growed up to 0.84-0.85.
\n\n3) Ok, what to do next? Let's try experimentation with data augmentation (scale,shift,rotation,perspective distortion, but not horizontal flip yet) and train/validation division. Few models/iterations later, by voting among best single submissions (voting code, it is Matlab's script, could be found here) finally come to 0.86 score.
\n\n4) What else Martin recommend? Flukes cropping. But I am not familiar with Python, so I did not want to use any fluke detection model that competitors have done. Ok, I have take this as challenge. And maybe it is not original way, but I have written a code that generates attention heatmap of my model for a picture (by sliding rectangle on top of the picture), then binarizes this heatmap, aligns binary mask by PCA (among mask's pixels coordiantes) and finally crops and resize image. When I generate such crops (heatmaps from my best at those time model) and retrain best model on cropped data, I have got 0.89 score. Interesting that if I have trained new model on cropped flukes sometimes I have got around 0.85. So two steps trainng (first on original pictures, than on cropped) was selected as important for me.
\n\n
5) What to do next? Somehow increase size of training set? So it was time to horizontal flipping (at those time i have already saw by my eyes many of training examples, so I came to understanding that almost all whales photographed without mirroring, it is good because we can mirror them and add this mirrored whales as new classes). This effectivelly doubled training set size. Have checked scrore, got improvement from 0.86 to 0.88 (this is without cropping, because I have already known that cropping definetelly will improves result, so I have performed experimenations without it). I have also made some experimentations with image negatives (multiplied by -1.0) and new classes compositions from left and right pars of different whales. But, none of them (except horizontal flipp) have improved score.
\n\n6) Where else we can get more trainnig examples? At the playground competitions of course. I have collected training set from playground competition (also only classes with more than one samples per whale), added it to my training set. And, LB score have grown from 0.88 to 0.907 (single model, yet without picture cropping). Maybe some test examples resides in the playground trainng set?? Did not check it. But I have found that there is a lot of noise in training data, and it became even more after union with playground data. Partially this noise could be filtered by good model and some hand labeling (note, taht this is not prohibited as we label training data, not test). So, I have wrote another tool to cleanup training data in semi-automatic manner. This allows me to boost my score to 0.928.
\n\n7) It was time to tune prediction procedure. Untill that moment I have only predicted top 1 label. My classifier produce a 128-dimensional vector of loats for a picture (so each whale could be represented as biometric template with the size of 512 bytes). So, after each model has been trained, I generated enrollment templates. One template for each picture in training set (for each \"new_whale\" too, as it improves results). And at submission generation step, for each test picture I compared the distances (Euclidean) between test picture's identification template and all of the enrollment templates. Then sort all of the distances and made decision about particular test sample. If minimum distance between templates was lower than a particular threshold, sample got label from enrollment set, otherwise sample got \"new_whale\" label. But competition's score also rewards you even if right prediction was 2nd, 3rd, or even 5th. So, we can improve score by submit all predictions in distance ascending order. That way I have improved my LB from 0.928 to 0.938 (my best single model).
\n\n8) What's next? Ensembling, of cource! Even first ensemble (consist of four different networks (0.938, 0.935, 0.854, 0.876) allows to get 0.943 LB score. I am ensembling them by simple concatenation of templates (also have tried to train head on this features, but without any luck). At this moment I have started to train different modifications of my base solution with different number of filters and other hyperparameters. At a final point I was able to stack 25 different models, and got public LB 0.948. Some combinations works better than the others. My final submission has been made by 9-models ResNet ensemble, and public LB was 0.953.
", "rawMarkdown": "First, my congrats to the winners! And, thanks to the organizers! Also, thanks to all peoples who make great tools for deep machine learning in C++, as Dlib (did you ever heard about it?) and OpenCV! \n\nSo, to be brief. My final training setup could be found [here](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Learner/main.cpp). My best single model architecture definition (it is ResNet variation with 512x192x1 input and 128 feature vector output) could be found [here](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Learner/customnetwork.h). I have started from public LB 0.748 and finished at private LB 0.948. How this progress has been done:\n\n1) My training pipeline was assembled when the test competition was held. So, as first attemp, I just retrain my old model on new data (\"new\\_whales\" set were removed) and have got 0.748. By playing with identification threshold have improved the score to 0.78. Already good, thanks to great [metric_loss](http://dlib.net/dlib/dnn/loss_abstract.h.html#loss_metric_) layer from DLib.\n\n2) Have taken into account Martin's Piotte [findings](https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563). In particular, have replaced RGB input to grayscale (with centering and noralization). And have removed all whales with a single sample from the trainnig set. Public score has growed up to 0.84-0.85. \n\n3) Ok, what to do next? Let's try experimentation with data augmentation (scale,shift,rotation,perspective distortion, but not horizontal flip yet) and train/validation division. Few models/iterations later, by voting among best single submissions (voting code, it is Matlab's script, could be found [here](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Matlab/Postprocess/Script.m)) finally come to 0.86 score.\n\n4) What else Martin recommend? Flukes cropping. But I am not familiar with Python, so I did not want to use any fluke detection model that competitors have done. Ok, I have take this as challenge. And maybe it is not original way, but I have written a [code](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Imgtransformer/main.cpp) that generates attention heatmap of my model for a picture (by sliding rectangle on top of the picture), then binarizes this heatmap, aligns binary mask by PCA (among mask's pixels coordiantes) and finally crops and resize image. When I generate such crops (heatmaps from my best at those time model) and retrain best model on cropped data, I have got 0.89 score. Interesting that if I have trained new model on cropped flukes sometimes I have got around 0.85. So two steps trainng (first on original pictures, than on cropped) was selected as important for me.\n\n![enter image description here][1]\n\n5) What to do next? Somehow increase size of training set? So it was time to horizontal flipping (at those time i have already saw by my eyes many of training examples, so I came to understanding that almost all whales photographed without mirroring, it is good because we can mirror them and add this mirrored whales as new classes). This effectivelly doubled training set size. Have checked scrore, got improvement from 0.86 to 0.88 (this is without cropping, because I have already known that cropping definetelly will improves result, so I have performed experimenations without it). I have also made some experimentations with image negatives (multiplied by -1.0) and new classes compositions from left and right pars of different whales. But, none of them (except horizontal flipp) have improved score. \n\n6) Where else we can get more trainnig examples? At the playground competitions of course. I have collected training set from playground competition (also only classes with more than one samples per whale), added it to my training set. And, LB score have grown from 0.88 to 0.907 (single model, yet without picture cropping). Maybe some test examples resides in the playground trainng set?? Did not check it. But I have found that there is a lot of noise in training data, and it became even more after union with playground data. Partially this noise could be filtered by good model and some hand labeling (note, taht this is not prohibited as we label training data, not test). So, I have wrote another [tool to cleanup training data](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Traincleaner/main.cpp) in semi-automatic manner. This allows me to boost my score to 0.928.\n\n7) It was time to tune prediction procedure. Untill that moment I have only predicted top 1 label. My classifier produce a 128-dimensional vector of loats for a picture (so each whale could be represented as biometric template with the size of 512 bytes). So, after each model has been trained, I generated enrollment templates. One template for each picture in training set (for each \"new\\_whale\" too, as it improves results). And at submission generation step, for each test picture I compared the distances (Euclidean) between test picture's identification template and all of the enrollment templates. Then sort all of the distances and made decision about particular test sample. If minimum distance between templates was lower than a particular threshold, sample got label from enrollment set, otherwise sample got \"new\\_whale\" label. But competition's score also rewards you even if right prediction was 2nd, 3rd, or even 5th. So, we can improve score by submit all predictions in distance ascending order. That way I have improved my LB from 0.928 to 0.938 (my best single model). \n\n8) What's next? Ensembling, of cource! Even first ensemble (consist of four different networks (0.938, 0.935, 0.854, 0.876) allows to get 0.943 LB score. I am ensembling them by simple concatenation of templates (also have tried to train head on this features, but without any luck). At this moment I have started to train different modifications of my base solution with different number of filters and other hyperparameters. At a final point I was able to stack 25 different models, and got public LB 0.948. Some combinations works better than the others. My final submission has been made by 9-models ResNet ensemble, and public LB was 0.953.\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/481336/11475/Cropping.png \"Fluke cropping by means of auto attention map\"", "votes": null }, { "id": "481342", "postDate": "03/01/2019 09:33:06", "content": "Great job @pinullmezon. Congrats for getting the silver medal.
", "rawMarkdown": "Great job @pinullmezon. Congrats for getting the silver medal.", "votes": null }, { "id": "481361", "postDate": "03/01/2019 09:59:37", "content": "Congrats on the result and thank you for the informative write up! Really neat how you flipped the images to generate more classes / examples.
", "rawMarkdown": "Congrats on the result and thank you for the informative write up! Really neat how you flipped the images to generate more classes / examples.", "votes": null }, { "id": "481477", "postDate": "03/01/2019 13:09:51", "content": "Congrats @pinullmezon and thanks for sharing.
", "rawMarkdown": "Congrats @pinullmezon and thanks for sharing.", "votes": null }, { "id": "481832", "postDate": "03/01/2019 22:37:05", "content": "impressive ranking using c++, congrats!
", "rawMarkdown": "impressive ranking using c++, congrats!", "votes": null }, { "id": "487434", "postDate": "03/10/2019 21:44:05", "content": "A lot of things to learn from your writeup, thanks for sharing!
", "rawMarkdown": "A lot of things to learn from your writeup, thanks for sharing!", "votes": null } ], "comments": [ { "id": 481342, "author_name": "karthik7395", "author_url": "", "post_date": "03/01/2019 09:33:06", "content": "Great job @pinullmezon. Congrats for getting the silver medal.
", "votes": null, "replies": [] }, { "id": 481361, "author_name": "radek1", "author_url": "", "post_date": "03/01/2019 09:59:37", "content": "Congrats on the result and thank you for the informative write up! Really neat how you flipped the images to generate more classes / examples.
", "votes": null, "replies": [] }, { "id": 481477, "author_name": "sheriytm", "author_url": "", "post_date": "03/01/2019 13:09:51", "content": "Congrats @pinullmezon and thanks for sharing.
", "votes": null, "replies": [] }, { "id": 481832, "author_name": "titericz", "author_url": "", "post_date": "03/01/2019 22:37:05", "content": "impressive ranking using c++, congrats!
", "votes": null, "replies": [] }, { "id": 487434, "author_name": "yassinealouini", "author_url": "", "post_date": "03/10/2019 21:44:05", "content": "A lot of things to learn from your writeup, thanks for sharing!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "481336": "First, my congrats to the winners! And, thanks to the organizers! Also, thanks to all peoples who make great tools for deep machine learning in C++, as Dlib (did you ever heard about it?) and OpenCV! \n\nSo, to be brief. My final training setup could be found [here](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Learner/main.cpp). My best single model architecture definition (it is ResNet variation with 512x192x1 input and 128 feature vector output) could be found [here](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Learner/customnetwork.h). I have started from public LB 0.748 and finished at private LB 0.948. How this progress has been done:\n\n1) My training pipeline was assembled when the test competition was held. So, as first attemp, I just retrain my old model on new data (\"new\\_whales\" set were removed) and have got 0.748. By playing with identification threshold have improved the score to 0.78. Already good, thanks to great [metric_loss](http://dlib.net/dlib/dnn/loss_abstract.h.html#loss_metric_) layer from DLib.\n\n2) Have taken into account Martin's Piotte [findings](https://www.kaggle.com/martinpiotte/whale-recognition-model-with-score-0-78563). In particular, have replaced RGB input to grayscale (with centering and noralization). And have removed all whales with a single sample from the trainnig set. Public score has growed up to 0.84-0.85. \n\n3) Ok, what to do next? Let's try experimentation with data augmentation (scale,shift,rotation,perspective distortion, but not horizontal flip yet) and train/validation division. Few models/iterations later, by voting among best single submissions (voting code, it is Matlab's script, could be found [here](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Matlab/Postprocess/Script.m)) finally come to 0.86 score.\n\n4) What else Martin recommend? Flukes cropping. But I am not familiar with Python, so I did not want to use any fluke detection model that competitors have done. Ok, I have take this as challenge. And maybe it is not original way, but I have written a [code](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Imgtransformer/main.cpp) that generates attention heatmap of my model for a picture (by sliding rectangle on top of the picture), then binarizes this heatmap, aligns binary mask by PCA (among mask's pixels coordiantes) and finally crops and resize image. When I generate such crops (heatmaps from my best at those time model) and retrain best model on cropped data, I have got 0.89 score. Interesting that if I have trained new model on cropped flukes sometimes I have got around 0.85. So two steps trainng (first on original pictures, than on cropped) was selected as important for me.\n\n![enter image description here][1]\n\n5) What to do next? Somehow increase size of training set? So it was time to horizontal flipping (at those time i have already saw by my eyes many of training examples, so I came to understanding that almost all whales photographed without mirroring, it is good because we can mirror them and add this mirrored whales as new classes). This effectivelly doubled training set size. Have checked scrore, got improvement from 0.86 to 0.88 (this is without cropping, because I have already known that cropping definetelly will improves result, so I have performed experimenations without it). I have also made some experimentations with image negatives (multiplied by -1.0) and new classes compositions from left and right pars of different whales. But, none of them (except horizontal flipp) have improved score. \n\n6) Where else we can get more trainnig examples? At the playground competitions of course. I have collected training set from playground competition (also only classes with more than one samples per whale), added it to my training set. And, LB score have grown from 0.88 to 0.907 (single model, yet without picture cropping). Maybe some test examples resides in the playground trainng set?? Did not check it. But I have found that there is a lot of noise in training data, and it became even more after union with playground data. Partially this noise could be filtered by good model and some hand labeling (note, taht this is not prohibited as we label training data, not test). So, I have wrote another [tool to cleanup training data](https://github.com/pi-null-mezon/Kaggle/blob/master/Whales/Dlib/Traincleaner/main.cpp) in semi-automatic manner. This allows me to boost my score to 0.928.\n\n7) It was time to tune prediction procedure. Untill that moment I have only predicted top 1 label. My classifier produce a 128-dimensional vector of loats for a picture (so each whale could be represented as biometric template with the size of 512 bytes). So, after each model has been trained, I generated enrollment templates. One template for each picture in training set (for each \"new\\_whale\" too, as it improves results). And at submission generation step, for each test picture I compared the distances (Euclidean) between test picture's identification template and all of the enrollment templates. Then sort all of the distances and made decision about particular test sample. If minimum distance between templates was lower than a particular threshold, sample got label from enrollment set, otherwise sample got \"new\\_whale\" label. But competition's score also rewards you even if right prediction was 2nd, 3rd, or even 5th. So, we can improve score by submit all predictions in distance ascending order. That way I have improved my LB from 0.928 to 0.938 (my best single model). \n\n8) What's next? Ensembling, of cource! Even first ensemble (consist of four different networks (0.938, 0.935, 0.854, 0.876) allows to get 0.943 LB score. I am ensembling them by simple concatenation of templates (also have tried to train head on this features, but without any luck). At this moment I have started to train different modifications of my base solution with different number of filters and other hyperparameters. At a final point I was able to stack 25 different models, and got public LB 0.948. Some combinations works better than the others. My final submission has been made by 9-models ResNet ensemble, and public LB was 0.953.\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/481336/11475/Cropping.png \"Fluke cropping by means of auto attention map\"", "481342": "Great job @pinullmezon. Congrats for getting the silver medal.", "481361": "Congrats on the result and thank you for the informative write up! Really neat how you flipped the images to generate more classes / examples.", "481477": "Congrats @pinullmezon and thanks for sharing.", "481832": "impressive ranking using c++, congrats!", "487434": "A lot of things to learn from your writeup, thanks for sharing!" }, "source": "meta" }