Here is a description of my model that I developed for that competition. It has a lower public LB score than the top models (24th actually on the public LB), however, I used a bit different algorithms and would like to share my experience.
\n\nI decided to work only with M-band images. There were two reasons to do that:
\n\nI created individual models for each class of objects because each pixel may belong to a few classes, and it was unclear to me how to distinguish between them within one multiclass model. For each model I used only three algorithms : XGBooost, Random Forest (RF) and Neural Network (ANN) with 2 or 3 hidden layers. XGB and RF were exploited in almost all my models but parameters were set up individually for each class. Especially, they worked pretty well for the highly unbalanced classes (cars and buildings). ANN was good in water and trees detection.
\n\nThen I tried to combine predictions from a different models. Simple averaging of predictions didn't produced a desired result. I think it happened, because XGB and RF algorithm were able to classify almost all positive examples quite successfully, but also produced a lot of false positive examples (noise). Therefore, I used logical_and method in order to combine different predictions, i.e. pixel was classified as 1 only if it was predicted to be 1 by all the models. Such a technique helped me to reduce a noise level to minimum. Especially it helped with the class 5 (Trees) where most of the models classified crop as trees.
\n\nWater class. I created one model for water detection. RF, XGB and ANN were able to recognize water quite successfully. But the question was how to separate waterways and standing water. So, first of all, I realized that if an image contains a waterway (like a river), a waterway must intersect two or more edges of the image. River cannot has a source and mouth on a 1 km^2 area. If object intersects image's edge only once and if object's area is less than some threshold value (I tried 5000 and 10000 in pixel units, but these two values are not optimal) that object is classified as standing water; otherwise it is a waterway. Also, if image contains a waterway, then all recognized objects are classified as a waterway as well. As the result, my best prediction for a waterway was 0.09749, and for a standing water 0.06305.
\n\nLarge and small vehicles. For these two classes I also created one model, because small cars and large trucks are made of the same material and only different in a size. Model consisted from combination of three XGB models. I distinguished small and large cars by area, however, only large vehicles were \"successfully identified\": best public LB score was 0.02743. For small cars I wasn't able to get more than 0.00000.
\n\nRoads. Road detection problem is well studied. I easily found a lot of papers about it in internet, but would like to mention two of them:
\n\nMy best score for the class 3: 0.08141
\n\nAll other classes\nFor all other classes I used XGB and RF algorithms without any specific tricks and don't want to pay attention on them. Only would like to say that my best prediction for the class 1 is 0.06543, for the class 2 is 0.00378 (very low), and for the class 6 is 0.07317. And I don't have individual scores for class 4, because I added prediction for this class directly to my total prediction. For class 5 (Trees) it was hard to get an individual score, and I submitted predictions within a total submission with other classes as well.
\n\nAfterwords. I would like to say thank to all participants, Kaggle team and DSTL for that competition. It is the first competition that I participated from the beginning to the end. And I would like to apologize for my English, it is not my native language. Thank you!
", "messages": [ { "id": "166292", "postDate": "03/09/2017 04:13:07", "content": "Here is a description of my model that I developed for that competition. It has a lower public LB score than the top models (24th actually on the public LB), however, I used a bit different algorithms and would like to share my experience.
\n\nI decided to work only with M-band images. There were two reasons to do that:
\n\nI created individual models for each class of objects because each pixel may belong to a few classes, and it was unclear to me how to distinguish between them within one multiclass model. For each model I used only three algorithms : XGBooost, Random Forest (RF) and Neural Network (ANN) with 2 or 3 hidden layers. XGB and RF were exploited in almost all my models but parameters were set up individually for each class. Especially, they worked pretty well for the highly unbalanced classes (cars and buildings). ANN was good in water and trees detection.
\n\nThen I tried to combine predictions from a different models. Simple averaging of predictions didn't produced a desired result. I think it happened, because XGB and RF algorithm were able to classify almost all positive examples quite successfully, but also produced a lot of false positive examples (noise). Therefore, I used logical_and method in order to combine different predictions, i.e. pixel was classified as 1 only if it was predicted to be 1 by all the models. Such a technique helped me to reduce a noise level to minimum. Especially it helped with the class 5 (Trees) where most of the models classified crop as trees.
\n\nWater class. I created one model for water detection. RF, XGB and ANN were able to recognize water quite successfully. But the question was how to separate waterways and standing water. So, first of all, I realized that if an image contains a waterway (like a river), a waterway must intersect two or more edges of the image. River cannot has a source and mouth on a 1 km^2 area. If object intersects image's edge only once and if object's area is less than some threshold value (I tried 5000 and 10000 in pixel units, but these two values are not optimal) that object is classified as standing water; otherwise it is a waterway. Also, if image contains a waterway, then all recognized objects are classified as a waterway as well. As the result, my best prediction for a waterway was 0.09749, and for a standing water 0.06305.
\n\nLarge and small vehicles. For these two classes I also created one model, because small cars and large trucks are made of the same material and only different in a size. Model consisted from combination of three XGB models. I distinguished small and large cars by area, however, only large vehicles were \"successfully identified\": best public LB score was 0.02743. For small cars I wasn't able to get more than 0.00000.
\n\nRoads. Road detection problem is well studied. I easily found a lot of papers about it in internet, but would like to mention two of them:
\n\nMy best score for the class 3: 0.08141
\n\nAll other classes\nFor all other classes I used XGB and RF algorithms without any specific tricks and don't want to pay attention on them. Only would like to say that my best prediction for the class 1 is 0.06543, for the class 2 is 0.00378 (very low), and for the class 6 is 0.07317. And I don't have individual scores for class 4, because I added prediction for this class directly to my total prediction. For class 5 (Trees) it was hard to get an individual score, and I submitted predictions within a total submission with other classes as well.
\n\nAfterwords. I would like to say thank to all participants, Kaggle team and DSTL for that competition. It is the first competition that I participated from the beginning to the end. And I would like to apologize for my English, it is not my native language. Thank you!
", "rawMarkdown": "Here is a description of my model that I developed for that competition. It has a lower public LB score than the top models (24th actually on the public LB), however, I used a bit different algorithms and would like to share my experience.\n\n## Data selection\nI decided to work only with M-band images. There were two reasons to do that: \n\n 1. Misalignment between 3, A,M and P bands (thanks to @visoft),\n 2. The indices like Normalized Difference Vegetation Index (NDVI), Enhanced Vegetation Indices (EVI) and many others use information from multispectral data. For road detection one can use BAI (built-up area index) and REI (Road extraction index) indices. Most of these indices has the same formula but use different bands. Therefore, I decided to use NDVI formula with different combinations of bands, and derived new features. As the result I got 8 M-band features + 20 combinations of these.\n\n## Algorithms\nI created individual models for each class of objects because each pixel may belong to a few classes, and it was unclear to me how to distinguish between them within one multiclass model. For each model I used only three algorithms : XGBooost, Random Forest (RF) and Neural Network (ANN) with 2 or 3 hidden layers. XGB and RF were exploited in almost all my models but parameters were set up individually for each class. Especially, they worked pretty well for the highly unbalanced classes (cars and buildings). ANN was good in water and trees detection. \n\nThen I tried to combine predictions from a different models. Simple averaging of predictions didn't produced a desired result. I think it happened, because XGB and RF algorithm were able to classify almost all positive examples quite successfully, but also produced a lot of false positive examples (noise). Therefore, I used logical_and method in order to combine different predictions, i.e. pixel was classified as 1 only if it was predicted to be 1 by all the models. Such a technique helped me to reduce a noise level to minimum. Especially it helped with the class 5 (Trees) where most of the models classified crop as trees.\n\n**Water class.** I created one model for water detection. RF, XGB and ANN were able to recognize water quite successfully. But the question was how to separate waterways and standing water. So, first of all, I realized that if an image contains a waterway (like a river), a waterway must intersect two or more edges of the image. River cannot has a source and mouth on a 1 km^2 area. If object intersects image's edge only once and if object's area is less than some threshold value (I tried 5000 and 10000 in pixel units, but these two values are not optimal) that object is classified as standing water; otherwise it is a waterway. Also, if image contains a waterway, then all recognized objects are classified as a waterway as well. As the result, my best prediction for a waterway was 0.09749, and for a standing water 0.06305.\n\n**Large and small vehicles.** For these two classes I also created one model, because small cars and large trucks are made of the same material and only different in a size. Model consisted from combination of three XGB models. I distinguished small and large cars by area, however, only large vehicles were \"successfully identified\": best public LB score was 0.02743. For small cars I wasn't able to get more than 0.00000.\n\n**Roads.** Road detection problem is well studied. I easily found a lot of papers about it in internet, but would like to mention two of them:\n\n 1. \"A novel spectral index to automatically extract roads networks from WorldView-2 satellite imagery\", Kaveh Shahi et al., *The Egyptian Journal of Remote Sensing and Space Science*, 2015. This paper contains description of BAI and REI indices that I used for a road detection.\n 2. \"A review of road extraction from remote sensing images\", Weixing Wang et al., *Journal of traffic and Transportation Engineering*, 2016. This paper describes a lot of different techniques, including ML methods such as ANN and SVM. Unfortunately, I didn't had enough time to try all of them. This paper may be interesting for those who want to make a research in that field.\n\nMy best score for the class 3: 0.08141\n\n**All other classes**\nFor all other classes I used XGB and RF algorithms without any specific tricks and don't want to pay attention on them. Only would like to say that my best prediction for the class 1 is 0.06543, for the class 2 is 0.00378 (very low), and for the class 6 is 0.07317. And I don't have individual scores for class 4, because I added prediction for this class directly to my total prediction. For class 5 (Trees) it was hard to get an individual score, and I submitted predictions within a total submission with other classes as well.\n\n**Afterwords**. I would like to say thank to all participants, Kaggle team and DSTL for that competition. It is the first competition that I participated from the beginning to the end. And I would like to apologize for my English, it is not my native language. Thank you!", "votes": null }, { "id": "166297", "postDate": "03/09/2017 04:39:50", "content": "Awesome solution! It's quite different from mainstream neural network approach almost everyone else used.
", "rawMarkdown": "Awesome solution! It's quite different from mainstream neural network approach almost everyone else used.", "votes": null }, { "id": "166305", "postDate": "03/09/2017 05:28:45", "content": "Glad to see that pixel based did well. Did you use any features derived from nearby pixels or was the score from only the single pixels?
", "rawMarkdown": "Glad to see that pixel based did well. Did you use any features derived from nearby pixels or was the score from only the single pixels?", "votes": null }, { "id": "166308", "postDate": "03/09/2017 05:38:57", "content": "Single pixels only. I thought about interactions of neighboring pixels, but didn't realized how I can implement such interactions within my model.
", "rawMarkdown": "Single pixels only. I thought about interactions of neighboring pixels, but didn't realized how I can implement such interactions within my model.", "votes": null }, { "id": "166339", "postDate": "03/09/2017 08:42:05", "content": "I find very interesting that you were able to get that high score only using M band. Maybe the resolution was not enough for classifying small vehicles though.
", "rawMarkdown": "I find very interesting that you were able to get that high score only using M band. Maybe the resolution was not enough for classifying small vehicles though.", "votes": null }, { "id": "166468", "postDate": "03/09/2017 19:56:07", "content": "I guess it is because different classes of objects consist of different materials (road: oil compounds, iron oxide, cars: steel, aluminum) and their spectral signals is easy to identify. And I think you are absolutely right, the resolution of M-images is too small to identify small cars and a different approach must be used for that.
", "rawMarkdown": "I guess it is because different classes of objects consist of different materials (road: oil compounds, iron oxide, cars: steel, aluminum) and their spectral signals is easy to identify. And I think you are absolutely right, the resolution of M-images is too small to identify small cars and a different approach must be used for that.", "votes": null }, { "id": "166470", "postDate": "03/09/2017 20:00:33", "content": "Thank you very much! By the way, I was happy to see someone from the east Siberia here. I was born and lived in Irkutsk for 17 years!
", "rawMarkdown": "Thank you very much! By the way, I was happy to see someone from the east Siberia here. I was born and lived in Irkutsk for 17 years!", "votes": null }, { "id": "166536", "postDate": "03/10/2017 02:23:19", "content": "This is awesome! I also tried pixel-based methods, but with less success, I use CNN to train a pixel-classifier (use a 65*65 window), but this model seems to too hard to train as there are so many pixels, glad to see traditional methods like xgb, rf, and ann works well, maybe sometimes cnn is a bit too complicated, well thks for sharing!
", "rawMarkdown": "This is awesome! I also tried pixel-based methods, but with less success, I use CNN to train a pixel-classifier (use a 65*65 window), but this model seems to too hard to train as there are so many pixels, glad to see traditional methods like xgb, rf, and ann works well, maybe sometimes cnn is a bit too complicated, well thks for sharing!", "votes": null }, { "id": "166537", "postDate": "03/10/2017 02:26:07", "content": "In spacenet chanlledge , the winner use pixel-based method too and he extract much more features than you have, I believe you will get much better result if with nearby features.
", "rawMarkdown": "In [spacenet chanlledge][1] , the winner use pixel-based method too and he extract much more features than you have, I believe you will get much better result if with nearby features.\n\n\n [1]: https://github.com/SpaceNetChallenge/BuildingDetectors", "votes": null }, { "id": "166894", "postDate": "03/11/2017 21:14:51", "content": "Thank you for the reference! Very interesting solutions! I guess this definitely would help, especially for vehicle detection.
", "rawMarkdown": "Thank you for the reference! Very interesting solutions! I guess this definitely would help, especially for vehicle detection.", "votes": null } ], "comments": [ { "id": 166297, "author_name": "ceperaang", "author_url": "", "post_date": "03/09/2017 04:39:50", "content": "Awesome solution! It's quite different from mainstream neural network approach almost everyone else used.
", "votes": null, "replies": [ { "id": 166470, "author_name": "alexlu", "author_url": "", "post_date": "03/09/2017 20:00:33", "content": "Thank you very much! By the way, I was happy to see someone from the east Siberia here. I was born and lived in Irkutsk for 17 years!
", "votes": null, "replies": [] } ] }, { "id": 166305, "author_name": "devinanzelmo", "author_url": "", "post_date": "03/09/2017 05:28:45", "content": "Glad to see that pixel based did well. Did you use any features derived from nearby pixels or was the score from only the single pixels?
", "votes": null, "replies": [ { "id": 166308, "author_name": "alexlu", "author_url": "", "post_date": "03/09/2017 05:38:57", "content": "Single pixels only. I thought about interactions of neighboring pixels, but didn't realized how I can implement such interactions within my model.
", "votes": null, "replies": [] }, { "id": 166537, "author_name": "zeliek", "author_url": "", "post_date": "03/10/2017 02:26:07", "content": "In spacenet chanlledge , the winner use pixel-based method too and he extract much more features than you have, I believe you will get much better result if with nearby features.
", "votes": null, "replies": [] }, { "id": 166894, "author_name": "alexlu", "author_url": "", "post_date": "03/11/2017 21:14:51", "content": "Thank you for the reference! Very interesting solutions! I guess this definitely would help, especially for vehicle detection.
", "votes": null, "replies": [] } ] }, { "id": 166339, "author_name": "ironbar", "author_url": "", "post_date": "03/09/2017 08:42:05", "content": "I find very interesting that you were able to get that high score only using M band. Maybe the resolution was not enough for classifying small vehicles though.
", "votes": null, "replies": [ { "id": 166468, "author_name": "alexlu", "author_url": "", "post_date": "03/09/2017 19:56:07", "content": "I guess it is because different classes of objects consist of different materials (road: oil compounds, iron oxide, cars: steel, aluminum) and their spectral signals is easy to identify. And I think you are absolutely right, the resolution of M-images is too small to identify small cars and a different approach must be used for that.
", "votes": null, "replies": [] } ] }, { "id": 166536, "author_name": "zeliek", "author_url": "", "post_date": "03/10/2017 02:23:19", "content": "This is awesome! I also tried pixel-based methods, but with less success, I use CNN to train a pixel-classifier (use a 65*65 window), but this model seems to too hard to train as there are so many pixels, glad to see traditional methods like xgb, rf, and ann works well, maybe sometimes cnn is a bit too complicated, well thks for sharing!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "166292": "Here is a description of my model that I developed for that competition. It has a lower public LB score than the top models (24th actually on the public LB), however, I used a bit different algorithms and would like to share my experience.\n\n## Data selection\nI decided to work only with M-band images. There were two reasons to do that: \n\n 1. Misalignment between 3, A,M and P bands (thanks to @visoft),\n 2. The indices like Normalized Difference Vegetation Index (NDVI), Enhanced Vegetation Indices (EVI) and many others use information from multispectral data. For road detection one can use BAI (built-up area index) and REI (Road extraction index) indices. Most of these indices has the same formula but use different bands. Therefore, I decided to use NDVI formula with different combinations of bands, and derived new features. As the result I got 8 M-band features + 20 combinations of these.\n\n## Algorithms\nI created individual models for each class of objects because each pixel may belong to a few classes, and it was unclear to me how to distinguish between them within one multiclass model. For each model I used only three algorithms : XGBooost, Random Forest (RF) and Neural Network (ANN) with 2 or 3 hidden layers. XGB and RF were exploited in almost all my models but parameters were set up individually for each class. Especially, they worked pretty well for the highly unbalanced classes (cars and buildings). ANN was good in water and trees detection. \n\nThen I tried to combine predictions from a different models. Simple averaging of predictions didn't produced a desired result. I think it happened, because XGB and RF algorithm were able to classify almost all positive examples quite successfully, but also produced a lot of false positive examples (noise). Therefore, I used logical_and method in order to combine different predictions, i.e. pixel was classified as 1 only if it was predicted to be 1 by all the models. Such a technique helped me to reduce a noise level to minimum. Especially it helped with the class 5 (Trees) where most of the models classified crop as trees.\n\n**Water class.** I created one model for water detection. RF, XGB and ANN were able to recognize water quite successfully. But the question was how to separate waterways and standing water. So, first of all, I realized that if an image contains a waterway (like a river), a waterway must intersect two or more edges of the image. River cannot has a source and mouth on a 1 km^2 area. If object intersects image's edge only once and if object's area is less than some threshold value (I tried 5000 and 10000 in pixel units, but these two values are not optimal) that object is classified as standing water; otherwise it is a waterway. Also, if image contains a waterway, then all recognized objects are classified as a waterway as well. As the result, my best prediction for a waterway was 0.09749, and for a standing water 0.06305.\n\n**Large and small vehicles.** For these two classes I also created one model, because small cars and large trucks are made of the same material and only different in a size. Model consisted from combination of three XGB models. I distinguished small and large cars by area, however, only large vehicles were \"successfully identified\": best public LB score was 0.02743. For small cars I wasn't able to get more than 0.00000.\n\n**Roads.** Road detection problem is well studied. I easily found a lot of papers about it in internet, but would like to mention two of them:\n\n 1. \"A novel spectral index to automatically extract roads networks from WorldView-2 satellite imagery\", Kaveh Shahi et al., *The Egyptian Journal of Remote Sensing and Space Science*, 2015. This paper contains description of BAI and REI indices that I used for a road detection.\n 2. \"A review of road extraction from remote sensing images\", Weixing Wang et al., *Journal of traffic and Transportation Engineering*, 2016. This paper describes a lot of different techniques, including ML methods such as ANN and SVM. Unfortunately, I didn't had enough time to try all of them. This paper may be interesting for those who want to make a research in that field.\n\nMy best score for the class 3: 0.08141\n\n**All other classes**\nFor all other classes I used XGB and RF algorithms without any specific tricks and don't want to pay attention on them. Only would like to say that my best prediction for the class 1 is 0.06543, for the class 2 is 0.00378 (very low), and for the class 6 is 0.07317. And I don't have individual scores for class 4, because I added prediction for this class directly to my total prediction. For class 5 (Trees) it was hard to get an individual score, and I submitted predictions within a total submission with other classes as well.\n\n**Afterwords**. I would like to say thank to all participants, Kaggle team and DSTL for that competition. It is the first competition that I participated from the beginning to the end. And I would like to apologize for my English, it is not my native language. Thank you!", "166297": "Awesome solution! It's quite different from mainstream neural network approach almost everyone else used.", "166305": "Glad to see that pixel based did well. Did you use any features derived from nearby pixels or was the score from only the single pixels?", "166308": "Single pixels only. I thought about interactions of neighboring pixels, but didn't realized how I can implement such interactions within my model.", "166339": "I find very interesting that you were able to get that high score only using M band. Maybe the resolution was not enough for classifying small vehicles though.", "166468": "I guess it is because different classes of objects consist of different materials (road: oil compounds, iron oxide, cars: steel, aluminum) and their spectral signals is easy to identify. And I think you are absolutely right, the resolution of M-images is too small to identify small cars and a different approach must be used for that.", "166470": "Thank you very much! By the way, I was happy to see someone from the east Siberia here. I was born and lived in Irkutsk for 17 years!", "166536": "This is awesome! I also tried pixel-based methods, but with less success, I use CNN to train a pixel-classifier (use a 65*65 window), but this model seems to too hard to train as there are so many pixels, glad to see traditional methods like xgb, rf, and ann works well, maybe sometimes cnn is a bit too complicated, well thks for sharing!", "166537": "In [spacenet chanlledge][1] , the winner use pixel-based method too and he extract much more features than you have, I believe you will get much better result if with nearby features.\n\n\n [1]: https://github.com/SpaceNetChallenge/BuildingDetectors", "166894": "Thank you for the reference! Very interesting solutions! I guess this definitely would help, especially for vehicle detection." }, "source": "meta" }