![\"enter](\"https://pp.userapi.com/c841633/v841633423/21c6e/Pvzs3anHMLs.jpg\" "\\"\\"")
Our team were created quite late, nevertheless we've managed to create 5 common folds and train bunch of models on them.\nThe original plan was to train a second-level model, but we haven't enough time for that after all.\nIn my opinion, this is the heaviest competition in the terms of hardware used so far (our team has used over 40 GPUs in total).
\n\nHere is the list of the models we've trained with corresponding dice scores.\nAll the networks have been trained on full resolution images.
\n\n
Valeriy Babushkin (VENHEADs)
\n\nValery used custom-built U-Net based on keras framework with different optimizers (Adam, RMSProb, SGD).\nDue to the limited batch size that can be fitted on one GPU he replaced BatchNorm layer with renormaliztion one.
\n\nArtur Kuzin (n01z3)
\n\nMy solution is based on mxnet.\nIn my opinion, it was crucial to use some multi-GPU framework what mxnet fits and the latter is also fast enough what is the bonus.\nHowever the deconvolution layer in mxnet realization is rather strange therefore I wasn’t able to reproduce the LinkNet architecture straight from the article\nand had to invent something to fit the layers’ shapes from hairy resnet graph.
\n\nNevertheless in spite of all the tricks (custom loss, different optimizers, hard negative sampling) had been done,\nI haven’t succeeded with training models to the heights on a par with my teammates’ models.
\n\nEvgeny Nizhibitsky (nizhib)
\n\nEvgeny made the best models of our ensemble with pytorch.\nAfter digging into and translating some arXiv articles\ninto the code, he've discovered that LinkNet does very well in this competition.\nThus he decided to create some mutated forms of LinkNet using other architectures as an encoder.\nAs a result we've got \"incnet\" — inception-v3 + linknet decoders and \"dinknet\" — densenet +\nthe same decoders. Evgeny used Adam with inital LR=1e-4 and several lr drops at fixed steps.
\n\nRoman Trusov (lextal)
\n\nRoman has trained some modified versions of pspnet with pytorch.\nHe replaced a single deconvolutional layer of the original network with 3 intermediate layers\nwith consistent size increases. Networks weights were initialized from resnet18 and resnet34.
", "messages": [ { "id": "224915", "postDate": "09/27/2017 23:02:10", "content": "Our team were created quite late, nevertheless we've managed to create 5 common folds and train bunch of models on them.\nThe original plan was to train a second-level model, but we haven't enough time for that after all.\nIn my opinion, this is the heaviest competition in the terms of hardware used so far (our team has used over 40 GPUs in total).
\n\nHere is the list of the models we've trained with corresponding dice scores.\nAll the networks have been trained on full resolution images.
\n\n
Valeriy Babushkin (VENHEADs)
\n\nValery used custom-built U-Net based on keras framework with different optimizers (Adam, RMSProb, SGD).\nDue to the limited batch size that can be fitted on one GPU he replaced BatchNorm layer with renormaliztion one.
\n\nArtur Kuzin (n01z3)
\n\nMy solution is based on mxnet.\nIn my opinion, it was crucial to use some multi-GPU framework what mxnet fits and the latter is also fast enough what is the bonus.\nHowever the deconvolution layer in mxnet realization is rather strange therefore I wasn’t able to reproduce the LinkNet architecture straight from the article\nand had to invent something to fit the layers’ shapes from hairy resnet graph.
\n\nNevertheless in spite of all the tricks (custom loss, different optimizers, hard negative sampling) had been done,\nI haven’t succeeded with training models to the heights on a par with my teammates’ models.
\n\nEvgeny Nizhibitsky (nizhib)
\n\nEvgeny made the best models of our ensemble with pytorch.\nAfter digging into and translating some arXiv articles\ninto the code, he've discovered that LinkNet does very well in this competition.\nThus he decided to create some mutated forms of LinkNet using other architectures as an encoder.\nAs a result we've got \"incnet\" — inception-v3 + linknet decoders and \"dinknet\" — densenet +\nthe same decoders. Evgeny used Adam with inital LR=1e-4 and several lr drops at fixed steps.
\n\nRoman Trusov (lextal)
\n\nRoman has trained some modified versions of pspnet with pytorch.\nHe replaced a single deconvolutional layer of the original network with 3 intermediate layers\nwith consistent size increases. Networks weights were initialized from resnet18 and resnet34.
", "rawMarkdown": "Our team were created quite late, nevertheless we've managed to create 5 common folds and train bunch of models on them.\nThe original plan was to train a second-level model, but we haven't enough time for that after all.\nIn my opinion, this is the heaviest competition in the terms of hardware used so far (our team has used over 40 GPUs in total).\n\nHere is the list of the models we've trained with corresponding dice scores.\nAll the networks have been trained on full resolution images. \n\n![enter image description here][1]\n\nValeriy Babushkin (VENHEADs)\n\nValery used custom-built U-Net based on keras framework with different optimizers (Adam, RMSProb, SGD).\nDue to the limited batch size that can be fitted on one GPU he replaced BatchNorm layer with renormaliztion one.\n\nArtur Kuzin (n01z3)\n\nMy solution is based on mxnet.\nIn my opinion, it was crucial to use some multi-GPU framework what mxnet fits and the latter is also fast enough what is the bonus.\nHowever the deconvolution layer in mxnet realization is rather strange therefore I wasn’t able to reproduce the LinkNet architecture straight from the article\nand had to invent something to fit the layers’ shapes from hairy resnet graph.\n\nNevertheless in spite of all the tricks (custom loss, different optimizers, hard negative sampling) had been done,\nI haven’t succeeded with training models to the heights on a par with my teammates’ models.\n\nEvgeny Nizhibitsky (nizhib)\n\nEvgeny made the best models of our ensemble with pytorch.\nAfter digging into and translating some arXiv articles\ninto the code, he've discovered that LinkNet does very well in this competition.\nThus he decided to create some mutated forms of LinkNet using other architectures as an encoder.\nAs a result we've got \"incnet\" — inception-v3 + linknet decoders and \"dinknet\" — densenet +\nthe same decoders. Evgeny used Adam with inital LR=1e-4 and several lr drops at fixed steps.\n\nRoman Trusov (lextal)\n\nRoman has trained some modified versions of pspnet with pytorch.\nHe replaced a single deconvolutional layer of the original network with 3 intermediate layers\nwith consistent size increases. Networks weights were initialized from resnet18 and resnet34.\n\n\n [1]: https://pp.userapi.com/c841633/v841633423/21c6e/Pvzs3anHMLs.jpg", "votes": null }, { "id": "224918", "postDate": "09/27/2017 23:08:02", "content": "Have you used all of the predictions (including low-score ones) for your best ensemble?
", "rawMarkdown": "Have you used all of the predictions (including low-score ones) for your best ensemble?", "votes": null }, { "id": "224919", "postDate": "09/27/2017 23:10:14", "content": "Thanks @n01z3.
\n\nYou are right about this being the heaviest competition in terms of hardware so far. I could hardly compete as my system is a CPU. So I managed to make only 3 NN based models by paying for cloud service. It is really hard to experiment when using paid system. But I enjoyed the challenge all the same.
", "rawMarkdown": "Thanks @n01z3. \n\nYou are right about this being the heaviest competition in terms of hardware so far. I could hardly compete as my system is a CPU. So I managed to make only 3 NN based models by paying for cloud service. It is really hard to experiment when using paid system. But I enjoyed the challenge all the same.", "votes": null }, { "id": "224920", "postDate": "09/27/2017 23:15:20", "content": "I used DevBox with 4 x 1080Ti and all the time I had a feeling that I need to invest into better cooling so that GPUs will stop dropping the frequency from overheating and to buy a few more GPUs...
\n\n![\"Not](\"https://habrastorage.org/webt/59/cc/36/59cc36326477f448297183.jpeg\" "\\"\\"")
Hm LinkNet looks really interesting. Personally I used a modified u-net based on this paper https://arxiv.org/pdf/1709.00201.pdf, but the ideas look really similar.
", "rawMarkdown": "Hm LinkNet looks really interesting. Personally I used a modified u-net based on this paper https://arxiv.org/pdf/1709.00201.pdf, but the ideas look really similar.", "votes": null }, { "id": "224923", "postDate": "09/27/2017 23:25:01", "content": "Seems network structure is a crucial aspect in this competition? Using networks with pre-trained weights also gives more benefits?
", "rawMarkdown": "Seems network structure is a crucial aspect in this competition? Using networks with pre-trained weights also gives more benefits?", "votes": null }, { "id": "224925", "postDate": "09/27/2017 23:27:37", "content": "@Vladimir, you have given me something to think about. As I plan on buying a good system in a couple of months but have started researching now. Although I enjoyed participating in contests involving all kinds of data, for example I found the data by Instacart so very interesting, my passion lies in the computer vision/ image processing types of contests. So I need to get a better HW.
", "rawMarkdown": "Vladimir, you have given me something to think about. As I plan on buying a good system in a couple of months but have started researching now. Although I enjoyed participating in contests involving all kinds of data, for example I found the data by Instacart so very interesting, my passion lies in the computer vision/ image processing types of contests. So I need to get a better HW.", "votes": null }, { "id": "224927", "postDate": "09/27/2017 23:31:58", "content": "Absolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D
\n\nAnd still, this competition was particularly tough in this respect. Our team have 5 1080ti and 3 1080 and was still limited by hardware most of the time. Could easily consume 10x that amount.
", "rawMarkdown": "Absolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D \n\nAnd still, this competition was particularly tough in this respect. Our team have 5 1080ti and 3 1080 and was still limited by hardware most of the time. Could easily consume 10x that amount.", "votes": null }, { "id": "224929", "postDate": "09/27/2017 23:33:44", "content": "Thanks @Steven for sharing this paper. I know what I will be reading today on my ride back home.
", "rawMarkdown": "Thanks @Steven for sharing this paper. I know what I will be reading today on my ride back home.", "votes": null }, { "id": "224932", "postDate": "09/27/2017 23:36:48", "content": "@Sergey, I noticed some limitations even with the Floyhub GPU service so, I know what you mean.
", "rawMarkdown": "Sergey, I noticed some limitations even with the Floyhub GPU service so, I know what you mean.", "votes": null }, { "id": "224933", "postDate": "09/27/2017 23:39:21", "content": "You are right on both points, as that is my experience too.
", "rawMarkdown": "You are right on both points, as that is my experience too.", "votes": null }, { "id": "224935", "postDate": "09/27/2017 23:44:14", "content": "is linknet a unet with residual block?
", "rawMarkdown": "is linknet a unet with residual block?", "votes": null }, { "id": "224936", "postDate": "09/27/2017 23:51:40", "content": "\n\n\nSergey Mushinskiy wrote
\n \n\n\nAbsolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D
\n
I bought mine for games, but I guess Kaggle is a game too...
", "rawMarkdown": "> **Sergey Mushinskiy wrote**\n> \n> > Absolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D \n\n[I bought mine for games, but I guess Kaggle is a game too...][1]\n\n\n [1]: https://imgur.com/a/zM6O2", "votes": null }, { "id": "224939", "postDate": "09/27/2017 23:55:51", "content": "It is definitely a lot of fun! Emotional roller-coaster like nothing else, especially in the last minutes before private lb reveal =)
", "rawMarkdown": "It is definitely a lot of fun! Emotional roller-coaster like nothing else, especially in the last minutes before private lb reveal =)", "votes": null }, { "id": "224940", "postDate": "09/27/2017 23:57:37", "content": "My last submission was so disappointing its comical
", "rawMarkdown": "[My last submission was so disappointing its comical][1]\n\n\n [1]: https://imgur.com/a/K6TL4", "votes": null }, { "id": "224969", "postDate": "09/28/2017 02:04:49", "content": "In a nutshell - yes.
", "rawMarkdown": "In a nutshell - yes.", "votes": null }, { "id": "225032", "postDate": "09/28/2017 04:27:34", "content": "Congratulations!
\nWhat was your cross-validation strategy?
We submit all avereged model prediction. But weighted geomitric mean off dinknet, incnet, linknet18, linknet34 was better.
", "rawMarkdown": "We submit all avereged model prediction. But weighted geomitric mean off dinknet, incnet, linknet18, linknet34 was better.", "votes": null }, { "id": "225659", "postDate": "09/29/2017 18:26:23", "content": "Interesting. Thanks for sharing!
", "rawMarkdown": "Interesting. Thanks for sharing!", "votes": null }, { "id": "226518", "postDate": "10/02/2017 14:52:13", "content": "We selected 45 degree side-view for every car id. Than sorted car ids by mask area. And we made the splits so that the distribution of areas was the same in each fold.
", "rawMarkdown": "We selected 45 degree side-view for every car id. Than sorted car ids by mask area. And we made the splits so that the distribution of areas was the same in each fold.", "votes": null } ], "comments": [ { "id": 224918, "author_name": "killthekitten", "author_url": "", "post_date": "09/27/2017 23:08:02", "content": "Have you used all of the predictions (including low-score ones) for your best ensemble?
", "votes": null, "replies": [ { "id": 225486, "author_name": "drn01z3", "author_url": "", "post_date": "09/29/2017 08:16:34", "content": "We submit all avereged model prediction. But weighted geomitric mean off dinknet, incnet, linknet18, linknet34 was better.
", "votes": null, "replies": [] } ] }, { "id": 224919, "author_name": "sheriytm", "author_url": "", "post_date": "09/27/2017 23:10:14", "content": "Thanks @n01z3.
\n\nYou are right about this being the heaviest competition in terms of hardware so far. I could hardly compete as my system is a CPU. So I managed to make only 3 NN based models by paying for cloud service. It is really hard to experiment when using paid system. But I enjoyed the challenge all the same.
", "votes": null, "replies": [ { "id": 224920, "author_name": "iglovikov", "author_url": "", "post_date": "09/27/2017 23:15:20", "content": "I used DevBox with 4 x 1080Ti and all the time I had a feeling that I need to invest into better cooling so that GPUs will stop dropping the frequency from overheating and to buy a few more GPUs...
\n\n
@Vladimir, you have given me something to think about. As I plan on buying a good system in a couple of months but have started researching now. Although I enjoyed participating in contests involving all kinds of data, for example I found the data by Instacart so very interesting, my passion lies in the computer vision/ image processing types of contests. So I need to get a better HW.
", "votes": null, "replies": [] }, { "id": 224927, "author_name": "ceperaang", "author_url": "", "post_date": "09/27/2017 23:31:58", "content": "Absolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D
\n\nAnd still, this competition was particularly tough in this respect. Our team have 5 1080ti and 3 1080 and was still limited by hardware most of the time. Could easily consume 10x that amount.
", "votes": null, "replies": [] }, { "id": 224932, "author_name": "sheriytm", "author_url": "", "post_date": "09/27/2017 23:36:48", "content": "@Sergey, I noticed some limitations even with the Floyhub GPU service so, I know what you mean.
", "votes": null, "replies": [] }, { "id": 224936, "author_name": "stevenknguyen", "author_url": "", "post_date": "09/27/2017 23:51:40", "content": "\n\n\nSergey Mushinskiy wrote
\n \n\n\nAbsolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D
\n
I bought mine for games, but I guess Kaggle is a game too...
", "votes": null, "replies": [] }, { "id": 224939, "author_name": "ceperaang", "author_url": "", "post_date": "09/27/2017 23:55:51", "content": "It is definitely a lot of fun! Emotional roller-coaster like nothing else, especially in the last minutes before private lb reveal =)
", "votes": null, "replies": [] }, { "id": 224940, "author_name": "stevenknguyen", "author_url": "", "post_date": "09/27/2017 23:57:37", "content": "My last submission was so disappointing its comical
", "votes": null, "replies": [] } ] }, { "id": 224921, "author_name": "stevenknguyen", "author_url": "", "post_date": "09/27/2017 23:16:43", "content": "Hm LinkNet looks really interesting. Personally I used a modified u-net based on this paper https://arxiv.org/pdf/1709.00201.pdf, but the ideas look really similar.
", "votes": null, "replies": [ { "id": 224929, "author_name": "sheriytm", "author_url": "", "post_date": "09/27/2017 23:33:44", "content": "Thanks @Steven for sharing this paper. I know what I will be reading today on my ride back home.
", "votes": null, "replies": [] } ] }, { "id": 224923, "author_name": "junhongxu", "author_url": "", "post_date": "09/27/2017 23:25:01", "content": "Seems network structure is a crucial aspect in this competition? Using networks with pre-trained weights also gives more benefits?
", "votes": null, "replies": [ { "id": 224933, "author_name": "sheriytm", "author_url": "", "post_date": "09/27/2017 23:39:21", "content": "You are right on both points, as that is my experience too.
", "votes": null, "replies": [] } ] }, { "id": 224935, "author_name": "hengck23", "author_url": "", "post_date": "09/27/2017 23:44:14", "content": "is linknet a unet with residual block?
", "votes": null, "replies": [ { "id": 224969, "author_name": "asanakoev", "author_url": "", "post_date": "09/28/2017 02:04:49", "content": "In a nutshell - yes.
", "votes": null, "replies": [] } ] }, { "id": 225032, "author_name": "ironbar", "author_url": "", "post_date": "09/28/2017 04:27:34", "content": "Congratulations!
\nWhat was your cross-validation strategy?
We selected 45 degree side-view for every car id. Than sorted car ids by mask area. And we made the splits so that the distribution of areas was the same in each fold.
", "votes": null, "replies": [] } ] }, { "id": 225659, "author_name": "amirbarwin", "author_url": "", "post_date": "09/29/2017 18:26:23", "content": "Interesting. Thanks for sharing!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "224915": "Our team were created quite late, nevertheless we've managed to create 5 common folds and train bunch of models on them.\nThe original plan was to train a second-level model, but we haven't enough time for that after all.\nIn my opinion, this is the heaviest competition in the terms of hardware used so far (our team has used over 40 GPUs in total).\n\nHere is the list of the models we've trained with corresponding dice scores.\nAll the networks have been trained on full resolution images. \n\n![enter image description here][1]\n\nValeriy Babushkin (VENHEADs)\n\nValery used custom-built U-Net based on keras framework with different optimizers (Adam, RMSProb, SGD).\nDue to the limited batch size that can be fitted on one GPU he replaced BatchNorm layer with renormaliztion one.\n\nArtur Kuzin (n01z3)\n\nMy solution is based on mxnet.\nIn my opinion, it was crucial to use some multi-GPU framework what mxnet fits and the latter is also fast enough what is the bonus.\nHowever the deconvolution layer in mxnet realization is rather strange therefore I wasn’t able to reproduce the LinkNet architecture straight from the article\nand had to invent something to fit the layers’ shapes from hairy resnet graph.\n\nNevertheless in spite of all the tricks (custom loss, different optimizers, hard negative sampling) had been done,\nI haven’t succeeded with training models to the heights on a par with my teammates’ models.\n\nEvgeny Nizhibitsky (nizhib)\n\nEvgeny made the best models of our ensemble with pytorch.\nAfter digging into and translating some arXiv articles\ninto the code, he've discovered that LinkNet does very well in this competition.\nThus he decided to create some mutated forms of LinkNet using other architectures as an encoder.\nAs a result we've got \"incnet\" — inception-v3 + linknet decoders and \"dinknet\" — densenet +\nthe same decoders. Evgeny used Adam with inital LR=1e-4 and several lr drops at fixed steps.\n\nRoman Trusov (lextal)\n\nRoman has trained some modified versions of pspnet with pytorch.\nHe replaced a single deconvolutional layer of the original network with 3 intermediate layers\nwith consistent size increases. Networks weights were initialized from resnet18 and resnet34.\n\n\n [1]: https://pp.userapi.com/c841633/v841633423/21c6e/Pvzs3anHMLs.jpg", "224918": "Have you used all of the predictions (including low-score ones) for your best ensemble?", "224919": "Thanks @n01z3. \n\nYou are right about this being the heaviest competition in terms of hardware so far. I could hardly compete as my system is a CPU. So I managed to make only 3 NN based models by paying for cloud service. It is really hard to experiment when using paid system. But I enjoyed the challenge all the same.", "224920": "I used DevBox with 4 x 1080Ti and all the time I had a feeling that I need to invest into better cooling so that GPUs will stop dropping the frequency from overheating and to buy a few more GPUs...\n\n![Not enough GPUs][1]\n\n\n [1]: https://habrastorage.org/webt/59/cc/36/59cc36326477f448297183.jpeg", "224921": "Hm LinkNet looks really interesting. Personally I used a modified u-net based on this paper https://arxiv.org/pdf/1709.00201.pdf, but the ideas look really similar.", "224923": "Seems network structure is a crucial aspect in this competition? Using networks with pre-trained weights also gives more benefits?", "224925": "Vladimir, you have given me something to think about. As I plan on buying a good system in a couple of months but have started researching now. Although I enjoyed participating in contests involving all kinds of data, for example I found the data by Instacart so very interesting, my passion lies in the computer vision/ image processing types of contests. So I need to get a better HW.", "224927": "Absolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D \n\nAnd still, this competition was particularly tough in this respect. Our team have 5 1080ti and 3 1080 and was still limited by hardware most of the time. Could easily consume 10x that amount.", "224929": "Thanks @Steven for sharing this paper. I know what I will be reading today on my ride back home.", "224932": "Sergey, I noticed some limitations even with the Floyhub GPU service so, I know what you mean.", "224933": "You are right on both points, as that is my experience too.", "224935": "is linknet a unet with residual block?", "224936": "> **Sergey Mushinskiy wrote**\n> \n> > Absolutely. GTX 1080 was the most fun and useful purchase for me this year. And I haven't even played any games on it yet :D \n\n[I bought mine for games, but I guess Kaggle is a game too...][1]\n\n\n [1]: https://imgur.com/a/zM6O2", "224939": "It is definitely a lot of fun! Emotional roller-coaster like nothing else, especially in the last minutes before private lb reveal =)", "224940": "[My last submission was so disappointing its comical][1]\n\n\n [1]: https://imgur.com/a/K6TL4", "224969": "In a nutshell - yes.", "225032": "Congratulations! \nWhat was your cross-validation strategy?", "225486": "We submit all avereged model prediction. But weighted geomitric mean off dinknet, incnet, linknet18, linknet34 was better.", "225659": "Interesting. Thanks for sharing!", "226518": "We selected 45 degree side-view for every car id. Than sorted car ids by mask area. And we made the splits so that the distribution of areas was the same in each fold." }, "source": "meta" }