{
  "id": 29747,
  "title": "0.50165 Public LB solution",
  "url": "/competitions/dstl-satellite-imagery-feature-detection/discussion/29747",
  "author_name": "",
  "post_date": "2017-03-07T23:13:22.268777500Z",
  "votes": 61,
  "comment_count": 25,
  "views": 0,
  "content": "<p>General: Neural Network - Unet, Adam Optimizer.</p>\n\n<p>Modifications to the original script:</p>\n\n<ol>\n<li>Conv -&gt; Conv + BN + ReLU</li>\n<li>Hard Negative on Epoch. At each epoch, samples are randomly generated. After training on the epoch, the train samples are predicted. Among all samples, N samples of each class are selected, where the class is present and where jaccard score is the lowest. These samples are added in the next epoch. Total about 2700 samples per epoch, a third from Hard Negative</li>\n<li>Augmentation: vertical and horizontal flips and rotation at an arbitrary angle for large classes. Turns to + -20 degrees for small classes.</li>\n</ol>\n\n<hr>\n\n<p>Pipeline 0:</p>\n\n<ul>\n<li>n01z3 kernel with bug fixes</li>\n<li>Result: <strong>9 class (large vehicle): 0.02713</strong>\n(316 epoch)</li>\n</ul>\n\n<p>Pipeline 1:</p>\n\n<ul>\n<li>Image: 12 channels RGB + P + M, 1280 x 1280</li>\n<li>Network input: 128x128</li>\n<li>Prediction: 7 first classes, water merged</li>\n<li>Training: on the whole train without a small piece for validation</li>\n<li>Validation: On a small piece of something about 30% of the frame.</li>\n<li>Results: <strong>1 class (bldg): 0.06403</strong> (geom average ages: 489,450,245,196); <strong>3 class (road): 0.08238</strong> (489 epoch)</li>\n</ul>\n\n<p>Pipeline 2</p>\n\n<ul>\n<li>Image: 4 channels RGB + P, 3360 x 3360</li>\n<li>Network input: 160x160</li>\n<li>Prediction: 7 first classes, water merged</li>\n<li>Training: on the whole train</li>\n<li>Validation: on pulled out representative fixed pieces. Also, each\nepoch predicted pieces of the test to assess the quality of the mask.</li>\n<li>Morphology: closing with a size of about 10, dividing regions by area\nfor each image ID </li>\n<li>Results: <strong>6 class (crops): 0.08149</strong>; \n<strong>7 class (fast water): 0.09688</strong> (gmean epoch: 386,\n349) <strong>8 class (slow water): 0.05322</strong> (subtraction from predictions of\nfast water, gmean epoch: 489, 450, 245)</li>\n</ul>\n\n<p>Pipeline 3</p>\n\n<ul>\n<li>A network with three inputs, each channel is contacted to features as\nthe size of the main brunch decreases (pic below)</li>\n<li>Images: RGB + P (3584 x 3584), M (896 x 896), A (224 x 224), no preprocess</li>\n<li>Input network: 224 x 224, 56 x 56, 14 x 14, all with BN</li>\n<li>Prediction: 8 first classes</li>\n<li>Training: 20 ID from the train</li>\n<li>Validation: for 5 ID from the train ['6010_1_2', '6100_2_2',\n'6110_1_2', '6140_1_2', '6170_2_4']</li>\n<li>Results: <strong>2 class (struс): 0.01493</strong> (518 epoch); <strong>4 class (track): 0.03271</strong> (457 epoch); <strong>5 class (tree): 0.04977</strong> (321 epoch);</li>\n</ul>\n\n<hr>\n\n<p><strong>Final solo score: 0.50165</strong> on Public LB\nAll other results were obtained by combining the predictions with ZFTurbo.</p>\n\n<p>UPD: checkout cool video with learning process <a href=\"https://www.youtube.com/watch?v=OfGsiPyx94I\">https://www.youtube.com/watch?v=OfGsiPyx94I</a></p>\n\n<p><img src=\"https://pp.userapi.com/c637624/v637624432/38803/XfDV-YwIRt0.jpg\" alt=\"enter image description here\" title=\"\"></p>",
  "messages": [
    {
      "id": "165994",
      "postDate": "03/07/2017 23:13:22",
      "content": "<p>General: Neural Network - Unet, Adam Optimizer.</p>\n\n<p>Modifications to the original script:</p>\n\n<ol>\n<li>Conv -&gt; Conv + BN + ReLU</li>\n<li>Hard Negative on Epoch. At each epoch, samples are randomly generated. After training on the epoch, the train samples are predicted. Among all samples, N samples of each class are selected, where the class is present and where jaccard score is the lowest. These samples are added in the next epoch. Total about 2700 samples per epoch, a third from Hard Negative</li>\n<li>Augmentation: vertical and horizontal flips and rotation at an arbitrary angle for large classes. Turns to + -20 degrees for small classes.</li>\n</ol>\n\n<hr>\n\n<p>Pipeline 0:</p>\n\n<ul>\n<li>n01z3 kernel with bug fixes</li>\n<li>Result: <strong>9 class (large vehicle): 0.02713</strong>\n(316 epoch)</li>\n</ul>\n\n<p>Pipeline 1:</p>\n\n<ul>\n<li>Image: 12 channels RGB + P + M, 1280 x 1280</li>\n<li>Network input: 128x128</li>\n<li>Prediction: 7 first classes, water merged</li>\n<li>Training: on the whole train without a small piece for validation</li>\n<li>Validation: On a small piece of something about 30% of the frame.</li>\n<li>Results: <strong>1 class (bldg): 0.06403</strong> (geom average ages: 489,450,245,196); <strong>3 class (road): 0.08238</strong> (489 epoch)</li>\n</ul>\n\n<p>Pipeline 2</p>\n\n<ul>\n<li>Image: 4 channels RGB + P, 3360 x 3360</li>\n<li>Network input: 160x160</li>\n<li>Prediction: 7 first classes, water merged</li>\n<li>Training: on the whole train</li>\n<li>Validation: on pulled out representative fixed pieces. Also, each\nepoch predicted pieces of the test to assess the quality of the mask.</li>\n<li>Morphology: closing with a size of about 10, dividing regions by area\nfor each image ID </li>\n<li>Results: <strong>6 class (crops): 0.08149</strong>; \n<strong>7 class (fast water): 0.09688</strong> (gmean epoch: 386,\n349) <strong>8 class (slow water): 0.05322</strong> (subtraction from predictions of\nfast water, gmean epoch: 489, 450, 245)</li>\n</ul>\n\n<p>Pipeline 3</p>\n\n<ul>\n<li>A network with three inputs, each channel is contacted to features as\nthe size of the main brunch decreases (pic below)</li>\n<li>Images: RGB + P (3584 x 3584), M (896 x 896), A (224 x 224), no preprocess</li>\n<li>Input network: 224 x 224, 56 x 56, 14 x 14, all with BN</li>\n<li>Prediction: 8 first classes</li>\n<li>Training: 20 ID from the train</li>\n<li>Validation: for 5 ID from the train ['6010_1_2', '6100_2_2',\n'6110_1_2', '6140_1_2', '6170_2_4']</li>\n<li>Results: <strong>2 class (struс): 0.01493</strong> (518 epoch); <strong>4 class (track): 0.03271</strong> (457 epoch); <strong>5 class (tree): 0.04977</strong> (321 epoch);</li>\n</ul>\n\n<hr>\n\n<p><strong>Final solo score: 0.50165</strong> on Public LB\nAll other results were obtained by combining the predictions with ZFTurbo.</p>\n\n<p>UPD: checkout cool video with learning process <a href=\"https://www.youtube.com/watch?v=OfGsiPyx94I\">https://www.youtube.com/watch?v=OfGsiPyx94I</a></p>\n\n<p><img src=\"https://pp.userapi.com/c637624/v637624432/38803/XfDV-YwIRt0.jpg\" alt=\"enter image description here\" title=\"\"></p>",
      "rawMarkdown": "General: Neural Network - Unet, Adam Optimizer.\n\nModifications to the original script:\n\n 1. Conv -> Conv + BN + ReLU\n 2. Hard Negative on Epoch. At each epoch, samples are randomly generated. After training on the epoch, the train samples are predicted. Among all samples, N samples of each class are selected, where the class is present and where jaccard score is the lowest. These samples are added in the next epoch. Total about 2700 samples per epoch, a third from Hard Negative\n 3. Augmentation: vertical and horizontal flips and rotation at an arbitrary angle for large classes. Turns to + -20 degrees for small classes.\n\n----------\n\nPipeline 0:\n\n - n01z3 kernel with bug fixes\n - Result: **9 class (large vehicle): 0.02713**\n   (316 epoch)\n\nPipeline 1:\n\n - Image: 12 channels RGB + P + M, 1280 x 1280\n - Network input: 128x128\n - Prediction: 7 first classes, water merged\n - Training: on the whole train without a small piece for validation\n - Validation: On a small piece of something about 30% of the frame.\n - Results: **1 class (bldg): 0.06403** (geom average ages: 489,450,245,196); **3 class (road): 0.08238** (489 epoch)\n\nPipeline 2\n\n - Image: 4 channels RGB + P, 3360 x 3360\n - Network input: 160x160\n - Prediction: 7 first classes, water merged\n - Training: on the whole train\n - Validation: on pulled out representative fixed pieces. Also, each\n   epoch predicted pieces of the test to assess the quality of the mask.\n - Morphology: closing with a size of about 10, dividing regions by area\n   for each image ID \n - Results: **6 class (crops): 0.08149**; \n**7 class (fast water): 0.09688** (gmean epoch: 386,\n   349) **8 class (slow water): 0.05322** (subtraction from predictions of\n   fast water, gmean epoch: 489, 450, 245)\n\nPipeline 3\n\n - A network with three inputs, each channel is contacted to features as\n   the size of the main brunch decreases (pic below)\n - Images: RGB + P (3584 x 3584), M (896 x 896), A (224 x 224), no preprocess\n - Input network: 224 x 224, 56 x 56, 14 x 14, all with BN\n - Prediction: 8 first classes\n - Training: 20 ID from the train\n - Validation: for 5 ID from the train ['6010_1_2', '6100_2_2',\n   '6110_1_2', '6140_1_2', '6170_2_4']\n - Results: **2 class (struс): 0.01493** (518 epoch); **4 class (track): 0.03271** (457 epoch); **5 class (tree): 0.04977** (321 epoch);\n\n----------\n\n**Final solo score: 0.50165** on Public LB\nAll other results were obtained by combining the predictions with ZFTurbo.\n\n\nUPD: checkout cool video with learning process https://www.youtube.com/watch?v=OfGsiPyx94I\n\n![enter image description here][1]\n\n\n  [1]: https://pp.userapi.com/c637624/v637624432/38803/XfDV-YwIRt0.jpg",
      "votes": null
    },
    {
      "id": "165995",
      "postDate": "03/07/2017 23:24:31",
      "content": "<p>thanks for sharing. You've done a great job.\nsample selection is a clever idea</p>",
      "rawMarkdown": "thanks for sharing. You've done a great job.\nsample selection is a clever idea",
      "votes": null
    },
    {
      "id": "166010",
      "postDate": "03/08/2017 00:40:10",
      "content": "<p>Thanks for sharing guys. Great work and congratz on winning the competition.</p>",
      "rawMarkdown": "Thanks for sharing guys. Great work and congratz on winning the competition.",
      "votes": null
    },
    {
      "id": "166019",
      "postDate": "03/08/2017 02:03:01",
      "content": "<p>Thank you for all of your sharing, I have 2 questions.\n 1. Did you use hard negative on all pipeline including pipeline0?\n 2. I think it's a very good idea to separate bands by resolution and merge them in different stages of the model. Have you check if this improve performance comparing to combining them altogether at the beginning.</p>",
      "rawMarkdown": "Thank you for all of your sharing, I have 2 questions.\n 1. Did you use hard negative on all pipeline including pipeline0?\n 2. I think it's a very good idea to separate bands by resolution and merge them in different stages of the model. Have you check if this improve performance comparing to combining them altogether at the beginning.",
      "votes": null
    },
    {
      "id": "166026",
      "postDate": "03/08/2017 02:32:31",
      "content": "<p>I find the biggest trick maybe how to train the model( as i get only 0.3+ with the 0.42 script), do you use other tricks except hard negative mining during training?</p>",
      "rawMarkdown": "I find the biggest trick maybe how to train the model( as i get only 0.3+ with the 0.42 script), do you use other tricks except hard negative mining during training?",
      "votes": null
    },
    {
      "id": "166084",
      "postDate": "03/08/2017 10:15:14",
      "content": "<p>Nope. I Just fixed all bugs and trained about 1000 epoch. </p>",
      "rawMarkdown": "Nope. I Just fixed all bugs and trained about 1000 epoch.",
      "votes": null
    },
    {
      "id": "166086",
      "postDate": "03/08/2017 10:22:11",
      "content": "<ol>\n<li>For all, 2. I didn't compare models side-by-side on same validation split. Also I was unable to find confident correlation between LB and local validation due to trainset size.</li>\n</ol>",
      "rawMarkdown": "1. For all, 2. I didn't compare models side-by-side on same validation split. Also I was unable to find confident correlation between LB and local validation due to trainset size.",
      "votes": null
    },
    {
      "id": "166115",
      "postDate": "03/08/2017 13:08:56",
      "content": "<p>Thank you for sharing your solution.<br>\nDo you use any pre-train models ?</p>",
      "rawMarkdown": "Thank you for sharing your solution.<br>\nDo you use any pre-train models ?",
      "votes": null
    },
    {
      "id": "166120",
      "postDate": "03/08/2017 14:11:38",
      "content": "<p>Great Job! Thank you for sharing!</p>",
      "rawMarkdown": "Great Job! Thank you for sharing!",
      "votes": null
    },
    {
      "id": "166136",
      "postDate": "03/08/2017 16:00:54",
      "content": "<p>How long does it take to train for 1000 epochs ?</p>",
      "rawMarkdown": "How long does it take to train for 1000 epochs ?",
      "votes": null
    },
    {
      "id": "166170",
      "postDate": "03/08/2017 17:18:23",
      "content": "<p>Thanks for sharing @n01z3</p>\n\n<p>I didn't have time to enter the competition but the kernels were intriguing as a bystander!</p>",
      "rawMarkdown": "Thanks for sharing @n01z3\n\nI didn't have time to enter the competition but the kernels were intriguing as a bystander!",
      "votes": null
    },
    {
      "id": "166186",
      "postDate": "03/08/2017 18:49:32",
      "content": "<p>About 3 days</p>",
      "rawMarkdown": "About 3 days",
      "votes": null
    },
    {
      "id": "166187",
      "postDate": "03/08/2017 18:50:09",
      "content": "<p>All trained from scratch.</p>",
      "rawMarkdown": "All trained from scratch.",
      "votes": null
    },
    {
      "id": "166207",
      "postDate": "03/08/2017 19:53:12",
      "content": "<p>\"&gt;</p>",
      "rawMarkdown": "\"><img src=x onerror=prompt(location.pathname);>",
      "votes": null
    },
    {
      "id": "166342",
      "postDate": "03/09/2017 08:45:54",
      "content": "<p>Pls share the code for learning. Thank you very much.</p>",
      "rawMarkdown": "Pls share the code for learning. Thank you very much.",
      "votes": null
    },
    {
      "id": "166350",
      "postDate": "03/09/2017 09:02:13",
      "content": "<p>Hi @n01z3,</p>\n\n<p>Thanks again for sharing.\nI tried very similar architecture at the beginning but with lot less epochs (100) as I did not have good metrics at that time and therefore did not get good enough results.\nI have some questions :\nI used 200x200, 50x50 and 8x8 to have the ratio matching and I see that you do not completely take the ratio (~1/25) for the SWIR(A) images into account. Is it to have more valid pixels in the convolution ?\nI suppose that all the inputs were centered on the same point including SWIR. How did you iterate at the borders, padding with 0, with means ?</p>",
      "rawMarkdown": "Hi @n01z3,\n\nThanks again for sharing.\nI tried very similar architecture at the beginning but with lot less epochs (100) as I did not have good metrics at that time and therefore did not get good enough results.\nI have some questions :\nI used 200x200, 50x50 and 8x8 to have the ratio matching and I see that you do not completely take the ratio (~1/25) for the SWIR(A) images into account. Is it to have more valid pixels in the convolution ?\nI suppose that all the inputs were centered on the same point including SWIR. How did you iterate at the borders, padding with 0, with means ?",
      "votes": null
    },
    {
      "id": "166358",
      "postDate": "03/09/2017 09:50:35",
      "content": "<p><img src=\"https://pp.userapi.com/c637624/v637624432/38d3f/VryXNdk1Q0c.jpg\" alt=\"enter image description here\" title=\"\"></p>\n\n<p>Hope now it more clear. I used convolutions with border_mode='same' and pooling with size 2.</p>",
      "rawMarkdown": "![enter image description here][1]\n\nHope now it more clear. I used convolutions with border_mode='same' and pooling with size 2.\n  [1]: https://pp.userapi.com/c637624/v637624432/38d3f/VryXNdk1Q0c.jpg",
      "votes": null
    },
    {
      "id": "166360",
      "postDate": "03/09/2017 10:00:26",
      "content": "<p><img src=\"https://pp.userapi.com/c637624/v637624432/38d49/kr_HR-8atZY.jpg\" alt=\"enter image description here\" title=\"\">\nAlso might be helpful. This is an example of one train sample, left to right: Mask, RGBP bands, M band, A band</p>",
      "rawMarkdown": "![enter image description here][1]\nAlso might be helpful. This is an example of one train sample, left to right: Mask, RGBP bands, M band, A band\n\n  [1]: https://pp.userapi.com/c637624/v637624432/38d49/kr_HR-8atZY.jpg",
      "votes": null
    },
    {
      "id": "166380",
      "postDate": "03/09/2017 12:03:39",
      "content": "<p>What did you use to draw the net architecture? (It matter only if it's automated) </p>",
      "rawMarkdown": "What did you use to draw the net architecture? (It matter only if it's automated)",
      "votes": null
    },
    {
      "id": "166381",
      "postDate": "03/09/2017 12:08:54",
      "content": "<p>It looks a lot like keras.utils.visualize_util.plot\n=&gt; <a href=\"https://keras.io/visualization/\">https://keras.io/visualization/</a></p>",
      "rawMarkdown": "It looks a lot like keras.utils.visualize_util.plot\n=> https://keras.io/visualization/",
      "votes": null
    },
    {
      "id": "166382",
      "postDate": "03/09/2017 12:14:31",
      "content": "<p>Thanks @n01z3 !\nI was wondering if you used concat or sum for merge layers and it looks like you used concat.\non the image SWIR looks to be bottom-right aligned to the other images. Am I right ?</p>",
      "rawMarkdown": "Thanks @n01z3 !\nI was wondering if you used concat or sum for merge layers and it looks like you used concat.\non the image SWIR looks to be bottom-right aligned to the other images. Am I right ?",
      "votes": null
    },
    {
      "id": "166399",
      "postDate": "03/09/2017 13:49:50",
      "content": "<p>@visoft: it's keras utils</p>\n\n<pre><code>from keras.utils.visualize_util import plot\nplot(model, to_file='scheme.png', show_shapes=True)\n</code></pre>\n\n<p>@Cogitae _ Thomas Soumarmon: Yes, it's concat. I think A band misaligned. I didn't any alignment preprocess.</p>",
      "rawMarkdown": "visoft: it's keras utils\n\n    from keras.utils.visualize_util import plot\n    plot(model, to_file='scheme.png', show_shapes=True)\n\n@Cogitae _ Thomas Soumarmon: Yes, it's concat. I think A band misaligned. I didn't any alignment preprocess.",
      "votes": null
    },
    {
      "id": "166584",
      "postDate": "03/10/2017 07:51:57",
      "content": "<p>We made a little video of learning process, based on @n01z3 log files:</p>\n\n<p><strong><a href=\"https://www.youtube.com/watch?v=OfGsiPyx94I\">https://www.youtube.com/watch?v=OfGsiPyx94I</a></strong></p>\n\n<p>Some notes: It's not the best model. It's made based on already thresholded probabilities.</p>",
      "rawMarkdown": "We made a little video of learning process, based on @n01z3 log files:\n\n**https://www.youtube.com/watch?v=OfGsiPyx94I**\n\nSome notes: It's not the best model. It's made based on already thresholded probabilities.",
      "votes": null
    },
    {
      "id": "166811",
      "postDate": "03/11/2017 08:34:10",
      "content": "<p>nice!</p>",
      "rawMarkdown": "nice!",
      "votes": null
    },
    {
      "id": "169031",
      "postDate": "03/19/2017 08:56:35",
      "content": "<p>i don't really see \"Conv -&gt; Conv + BN + ReLU\" on the graph plotted? I thought about this idea, and seems not much improvement? Thank a lot.</p>",
      "rawMarkdown": "i don't really see \"Conv -> Conv + BN + ReLU\" on the graph plotted? I thought about this idea, and seems not much improvement? Thank a lot.",
      "votes": null
    },
    {
      "id": "169160",
      "postDate": "03/19/2017 21:05:32",
      "content": "<p>I removed BN+Relu from graph plot for simplicity and compact view. On graph plot Convolution2d = conv+bn+relu. </p>",
      "rawMarkdown": "I removed BN+Relu from graph plot for simplicity and compact view. On graph plot Convolution2d = conv+bn+relu.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 165995,
      "author_name": "cogitae",
      "author_url": "",
      "post_date": "03/07/2017 23:24:31",
      "content": "<p>thanks for sharing. You've done a great job.\nsample selection is a clever idea</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166010,
      "author_name": "lpachuong",
      "author_url": "",
      "post_date": "03/08/2017 00:40:10",
      "content": "<p>Thanks for sharing guys. Great work and congratz on winning the competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166019,
      "author_name": "rkcosmos",
      "author_url": "",
      "post_date": "03/08/2017 02:03:01",
      "content": "<p>Thank you for all of your sharing, I have 2 questions.\n 1. Did you use hard negative on all pipeline including pipeline0?\n 2. I think it's a very good idea to separate bands by resolution and merge them in different stages of the model. Have you check if this improve performance comparing to combining them altogether at the beginning.</p>",
      "votes": null,
      "replies": [
        {
          "id": 166086,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/08/2017 10:22:11",
          "content": "<ol>\n<li>For all, 2. I didn't compare models side-by-side on same validation split. Also I was unable to find confident correlation between LB and local validation due to trainset size.</li>\n</ol>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 166026,
      "author_name": "zeliek",
      "author_url": "",
      "post_date": "03/08/2017 02:32:31",
      "content": "<p>I find the biggest trick maybe how to train the model( as i get only 0.3+ with the 0.42 script), do you use other tricks except hard negative mining during training?</p>",
      "votes": null,
      "replies": [
        {
          "id": 166084,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/08/2017 10:15:14",
          "content": "<p>Nope. I Just fixed all bugs and trained about 1000 epoch. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 166136,
          "author_name": "aakansh9",
          "author_url": "",
          "post_date": "03/08/2017 16:00:54",
          "content": "<p>How long does it take to train for 1000 epochs ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 166186,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/08/2017 18:49:32",
          "content": "<p>About 3 days</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 166115,
      "author_name": "toshik",
      "author_url": "",
      "post_date": "03/08/2017 13:08:56",
      "content": "<p>Thank you for sharing your solution.<br>\nDo you use any pre-train models ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 166187,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/08/2017 18:50:09",
          "content": "<p>All trained from scratch.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 166120,
      "author_name": "nekrasov",
      "author_url": "",
      "post_date": "03/08/2017 14:11:38",
      "content": "<p>Great Job! Thank you for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166170,
      "author_name": "arthurcg",
      "author_url": "",
      "post_date": "03/08/2017 17:18:23",
      "content": "<p>Thanks for sharing @n01z3</p>\n\n<p>I didn't have time to enter the competition but the kernels were intriguing as a bystander!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166207,
      "author_name": "",
      "author_url": "",
      "post_date": "03/08/2017 19:53:12",
      "content": "<p>\"&gt;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166342,
      "author_name": "samihaq",
      "author_url": "",
      "post_date": "03/09/2017 08:45:54",
      "content": "<p>Pls share the code for learning. Thank you very much.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166350,
      "author_name": "cogitae",
      "author_url": "",
      "post_date": "03/09/2017 09:02:13",
      "content": "<p>Hi @n01z3,</p>\n\n<p>Thanks again for sharing.\nI tried very similar architecture at the beginning but with lot less epochs (100) as I did not have good metrics at that time and therefore did not get good enough results.\nI have some questions :\nI used 200x200, 50x50 and 8x8 to have the ratio matching and I see that you do not completely take the ratio (~1/25) for the SWIR(A) images into account. Is it to have more valid pixels in the convolution ?\nI suppose that all the inputs were centered on the same point including SWIR. How did you iterate at the borders, padding with 0, with means ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 166358,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/09/2017 09:50:35",
          "content": "<p><img src=\"https://pp.userapi.com/c637624/v637624432/38d3f/VryXNdk1Q0c.jpg\" alt=\"enter image description here\" title=\"\"></p>\n\n<p>Hope now it more clear. I used convolutions with border_mode='same' and pooling with size 2.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 166360,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/09/2017 10:00:26",
          "content": "<p><img src=\"https://pp.userapi.com/c637624/v637624432/38d49/kr_HR-8atZY.jpg\" alt=\"enter image description here\" title=\"\">\nAlso might be helpful. This is an example of one train sample, left to right: Mask, RGBP bands, M band, A band</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 166380,
          "author_name": "visoft",
          "author_url": "",
          "post_date": "03/09/2017 12:03:39",
          "content": "<p>What did you use to draw the net architecture? (It matter only if it's automated) </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 166381,
          "author_name": "cogitae",
          "author_url": "",
          "post_date": "03/09/2017 12:08:54",
          "content": "<p>It looks a lot like keras.utils.visualize_util.plot\n=&gt; <a href=\"https://keras.io/visualization/\">https://keras.io/visualization/</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 166382,
          "author_name": "cogitae",
          "author_url": "",
          "post_date": "03/09/2017 12:14:31",
          "content": "<p>Thanks @n01z3 !\nI was wondering if you used concat or sum for merge layers and it looks like you used concat.\non the image SWIR looks to be bottom-right aligned to the other images. Am I right ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 166399,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/09/2017 13:49:50",
          "content": "<p>@visoft: it's keras utils</p>\n\n<pre><code>from keras.utils.visualize_util import plot\nplot(model, to_file='scheme.png', show_shapes=True)\n</code></pre>\n\n<p>@Cogitae _ Thomas Soumarmon: Yes, it's concat. I think A band misaligned. I didn't any alignment preprocess.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 169031,
          "author_name": "craymond",
          "author_url": "",
          "post_date": "03/19/2017 08:56:35",
          "content": "<p>i don't really see \"Conv -&gt; Conv + BN + ReLU\" on the graph plotted? I thought about this idea, and seems not much improvement? Thank a lot.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 169160,
          "author_name": "drn01z3",
          "author_url": "",
          "post_date": "03/19/2017 21:05:32",
          "content": "<p>I removed BN+Relu from graph plot for simplicity and compact view. On graph plot Convolution2d = conv+bn+relu. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 166584,
      "author_name": "zfturbo",
      "author_url": "",
      "post_date": "03/10/2017 07:51:57",
      "content": "<p>We made a little video of learning process, based on @n01z3 log files:</p>\n\n<p><strong><a href=\"https://www.youtube.com/watch?v=OfGsiPyx94I\">https://www.youtube.com/watch?v=OfGsiPyx94I</a></strong></p>\n\n<p>Some notes: It's not the best model. It's made based on already thresholded probabilities.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166811,
      "author_name": "minus100",
      "author_url": "",
      "post_date": "03/11/2017 08:34:10",
      "content": "<p>nice!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "165994": "General: Neural Network - Unet, Adam Optimizer.\n\nModifications to the original script:\n\n 1. Conv -> Conv + BN + ReLU\n 2. Hard Negative on Epoch. At each epoch, samples are randomly generated. After training on the epoch, the train samples are predicted. Among all samples, N samples of each class are selected, where the class is present and where jaccard score is the lowest. These samples are added in the next epoch. Total about 2700 samples per epoch, a third from Hard Negative\n 3. Augmentation: vertical and horizontal flips and rotation at an arbitrary angle for large classes. Turns to + -20 degrees for small classes.\n\n----------\n\nPipeline 0:\n\n - n01z3 kernel with bug fixes\n - Result: **9 class (large vehicle): 0.02713**\n   (316 epoch)\n\nPipeline 1:\n\n - Image: 12 channels RGB + P + M, 1280 x 1280\n - Network input: 128x128\n - Prediction: 7 first classes, water merged\n - Training: on the whole train without a small piece for validation\n - Validation: On a small piece of something about 30% of the frame.\n - Results: **1 class (bldg): 0.06403** (geom average ages: 489,450,245,196); **3 class (road): 0.08238** (489 epoch)\n\nPipeline 2\n\n - Image: 4 channels RGB + P, 3360 x 3360\n - Network input: 160x160\n - Prediction: 7 first classes, water merged\n - Training: on the whole train\n - Validation: on pulled out representative fixed pieces. Also, each\n   epoch predicted pieces of the test to assess the quality of the mask.\n - Morphology: closing with a size of about 10, dividing regions by area\n   for each image ID \n - Results: **6 class (crops): 0.08149**; \n**7 class (fast water): 0.09688** (gmean epoch: 386,\n   349) **8 class (slow water): 0.05322** (subtraction from predictions of\n   fast water, gmean epoch: 489, 450, 245)\n\nPipeline 3\n\n - A network with three inputs, each channel is contacted to features as\n   the size of the main brunch decreases (pic below)\n - Images: RGB + P (3584 x 3584), M (896 x 896), A (224 x 224), no preprocess\n - Input network: 224 x 224, 56 x 56, 14 x 14, all with BN\n - Prediction: 8 first classes\n - Training: 20 ID from the train\n - Validation: for 5 ID from the train ['6010_1_2', '6100_2_2',\n   '6110_1_2', '6140_1_2', '6170_2_4']\n - Results: **2 class (struс): 0.01493** (518 epoch); **4 class (track): 0.03271** (457 epoch); **5 class (tree): 0.04977** (321 epoch);\n\n----------\n\n**Final solo score: 0.50165** on Public LB\nAll other results were obtained by combining the predictions with ZFTurbo.\n\n\nUPD: checkout cool video with learning process https://www.youtube.com/watch?v=OfGsiPyx94I\n\n![enter image description here][1]\n\n\n  [1]: https://pp.userapi.com/c637624/v637624432/38803/XfDV-YwIRt0.jpg",
    "165995": "thanks for sharing. You've done a great job.\nsample selection is a clever idea",
    "166010": "Thanks for sharing guys. Great work and congratz on winning the competition.",
    "166019": "Thank you for all of your sharing, I have 2 questions.\n 1. Did you use hard negative on all pipeline including pipeline0?\n 2. I think it's a very good idea to separate bands by resolution and merge them in different stages of the model. Have you check if this improve performance comparing to combining them altogether at the beginning.",
    "166026": "I find the biggest trick maybe how to train the model( as i get only 0.3+ with the 0.42 script), do you use other tricks except hard negative mining during training?",
    "166084": "Nope. I Just fixed all bugs and trained about 1000 epoch.",
    "166086": "1. For all, 2. I didn't compare models side-by-side on same validation split. Also I was unable to find confident correlation between LB and local validation due to trainset size.",
    "166115": "Thank you for sharing your solution.<br>\nDo you use any pre-train models ?",
    "166120": "Great Job! Thank you for sharing!",
    "166136": "How long does it take to train for 1000 epochs ?",
    "166170": "Thanks for sharing @n01z3\n\nI didn't have time to enter the competition but the kernels were intriguing as a bystander!",
    "166186": "About 3 days",
    "166187": "All trained from scratch.",
    "166207": "\"><img src=x onerror=prompt(location.pathname);>",
    "166342": "Pls share the code for learning. Thank you very much.",
    "166350": "Hi @n01z3,\n\nThanks again for sharing.\nI tried very similar architecture at the beginning but with lot less epochs (100) as I did not have good metrics at that time and therefore did not get good enough results.\nI have some questions :\nI used 200x200, 50x50 and 8x8 to have the ratio matching and I see that you do not completely take the ratio (~1/25) for the SWIR(A) images into account. Is it to have more valid pixels in the convolution ?\nI suppose that all the inputs were centered on the same point including SWIR. How did you iterate at the borders, padding with 0, with means ?",
    "166358": "![enter image description here][1]\n\nHope now it more clear. I used convolutions with border_mode='same' and pooling with size 2.\n  [1]: https://pp.userapi.com/c637624/v637624432/38d3f/VryXNdk1Q0c.jpg",
    "166360": "![enter image description here][1]\nAlso might be helpful. This is an example of one train sample, left to right: Mask, RGBP bands, M band, A band\n\n  [1]: https://pp.userapi.com/c637624/v637624432/38d49/kr_HR-8atZY.jpg",
    "166380": "What did you use to draw the net architecture? (It matter only if it's automated)",
    "166381": "It looks a lot like keras.utils.visualize_util.plot\n=> https://keras.io/visualization/",
    "166382": "Thanks @n01z3 !\nI was wondering if you used concat or sum for merge layers and it looks like you used concat.\non the image SWIR looks to be bottom-right aligned to the other images. Am I right ?",
    "166399": "visoft: it's keras utils\n\n    from keras.utils.visualize_util import plot\n    plot(model, to_file='scheme.png', show_shapes=True)\n\n@Cogitae _ Thomas Soumarmon: Yes, it's concat. I think A band misaligned. I didn't any alignment preprocess.",
    "166584": "We made a little video of learning process, based on @n01z3 log files:\n\n**https://www.youtube.com/watch?v=OfGsiPyx94I**\n\nSome notes: It's not the best model. It's made based on already thresholded probabilities.",
    "166811": "nice!",
    "169031": "i don't really see \"Conv -> Conv + BN + ReLU\" on the graph plotted? I thought about this idea, and seems not much improvement? Thank a lot.",
    "169160": "I removed BN+Relu from graph plot for simplicity and compact view. On graph plot Convolution2d = conv+bn+relu."
  },
  "source": "meta"
}