{
  "id": 118086,
  "title": "Krazy Klassifiers - 48th place solution",
  "url": "/competitions/understanding_cloud_organization/discussion/118086",
  "author_name": "Chris Deotte",
  "post_date": "2019-11-19T13:57:07.547000",
  "votes": 70,
  "comment_count": 16,
  "views": 0,
  "content": "<p>If you could predict empty mask for every empty mask and predict full mask, (i.e. predict every pixel with <code>rle = '1 183750'</code>) for every mask, then your CV is 0.686!! Therefore a perfect classifier can win without any segmentation. The following code outputs 0.686:</p>\n\n<pre><code>train = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain['pred'] = np.where(~train.EncodedPixels.isna(),'1 183750','')\ntrain['dice'] = train.apply(lambda x: kaggle_dice(x['EncodedPixels'],x['pred']),axis=1)\nprint( train.dice.mean() )\n</code></pre>\n\n<h1>Classification Models</h1>\n\n<p>I focused most of my energy on building classification models and finally achieved 78% classification validation accuracy (on 33% holdout set, i.e. 3-Fold CV) by ensembling two crazy classifiers. The first has 4 backbones that extract features from 4 different resized input images (half size, quarter size, one sixth size, and one eighth size)</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fdef7ae2825082c95e50f22674c562124%2Fcls1.jpg?generation=1574169946410492&amp;alt=media\" alt=\"\"></p>\n\n<pre><code>base_model0 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nbase_model1 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nbase_model2 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nbase_model3 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nx0 = base_model0.output\nx0 = layers.GlobalAveragePooling2D()(x0)\nx1 = base_model1.output\nx1 = layers.GlobalAveragePooling2D()(x1)\nx2 = base_model2.output\nx2 = layers.GlobalAveragePooling2D()(x2)\nx3 = base_model3.output\nx3 = layers.GlobalAveragePooling2D()(x3)\nx = layers.concatenate([x0,x1,x2,x3])\nx = layers.Dense(4,activation='sigmoid')(x)\nmodel = Model(inputs=(base_model0.input, base_model1.input, base_model2.input, \n    base_model3.input), outputs=x)\n</code></pre>\n\n<p>My second model uses masks in addition to labels and achieves 77% accuracy by itself. The label loss is backpropagated through the mask prediction. Then instead of using the outputted labels, we predict 1 or 0 for label based on whether mask is present or not.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F2445cef6b03266046473736d3d4a8914%2Fcls2.jpg?generation=1574170027626081&amp;alt=media\" alt=\"\"></p>\n\n<pre><code>model0 = Unet('resnet34', input_shape=(None,None,3), classes=4,\n    activation='sigmoid', encoder_freeze=True)\nmodel0.layers[-1].name = 'out1'\nx = model0.output\nx = layers.GlobalAveragePooling2D()(x)\nx = layers.Dense(4, activation='sigmoid', name='out2')(x)\nmodel = Model(inputs = model0.input, outputs = (model0.output,x))\nmodel.compile(optimizer=opt, loss={'out1':loss1,'out2':loss2}, \n    metric = {'out1':metric1, 'out2':metric2})\n</code></pre>\n\n<h1>Segmentation Model</h1>\n\n<p>My segmentation model is a collage of ideas from public kernels. Without post process, it achieves Public LB 0.650. Test time augmentation (TTAx6) increases this to LB 0.655. Using 3-Folds increases this to LB 0.660. Ensembling 7 copies with different choices for 3-Fold achieves LB 0.665. And finally removing false positives with my classifier increases this to LB 0.670. My final solution has CV 0.663 and Private LB 0.663. Here are specific details:</p>\n\n<ul>\n<li>Unet Architecture</li>\n<li>EfficientnetB2 backbone</li>\n<li>Train on 352x544 random crops from 384x576 size images</li>\n<li>Train augmentation of flips and rotate</li>\n<li>Adam Accumulate optimizer</li>\n<li>Jaccard loss</li>\n<li>Kaggle Dice metric, Kaggle accuracy metric</li>\n<li>Reduce LR on plateau and early stopping</li>\n<li>Remove masks less than 20000 pixels</li>\n<li>TTA of flips and shifts</li>\n<li>3-Fold CV and prediction</li>\n<li>Remove false positive masks with classifier</li>\n</ul>\n\n<h1>Kaggle Notebook</h1>\n\n<p>I posted a Kaggle notebook showing my segmentation model <a href=\"https://www.kaggle.com/cdeotte/cloud-solution-lb-0-670\">here</a>. It scores LB 0.665 by itself and LB 0.670 if you ensemble it with 7 copies of itself with different initialization seeds. It loads classification predictions from my offline classifier models for false positive removal.</p>\n\n<p>Thank you everyone for a fun and exciting competition. I learned a lot from reading everyone's discussions and posted code. Thank you Kaggle and Max-Planck-Institite for sharing cloud data and hosting. Congratulations to all the winners.</p>",
  "messages": [
    {
      "id": 676784,
      "postDate": "2019-11-19T13:57:07.547Z",
      "content": "<p>If you could predict empty mask for every empty mask and predict full mask, (i.e. predict every pixel with <code>rle = '1 183750'</code>) for every mask, then your CV is 0.686!! Therefore a perfect classifier can win without any segmentation. The following code outputs 0.686:</p>\n\n<pre><code>train = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain['pred'] = np.where(~train.EncodedPixels.isna(),'1 183750','')\ntrain['dice'] = train.apply(lambda x: kaggle_dice(x['EncodedPixels'],x['pred']),axis=1)\nprint( train.dice.mean() )\n</code></pre>\n\n<h1>Classification Models</h1>\n\n<p>I focused most of my energy on building classification models and finally achieved 78% classification validation accuracy (on 33% holdout set, i.e. 3-Fold CV) by ensembling two crazy classifiers. The first has 4 backbones that extract features from 4 different resized input images (half size, quarter size, one sixth size, and one eighth size)</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fdef7ae2825082c95e50f22674c562124%2Fcls1.jpg?generation=1574169946410492&amp;alt=media\" alt=\"\"></p>\n\n<pre><code>base_model0 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nbase_model1 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nbase_model2 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nbase_model3 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\nx0 = base_model0.output\nx0 = layers.GlobalAveragePooling2D()(x0)\nx1 = base_model1.output\nx1 = layers.GlobalAveragePooling2D()(x1)\nx2 = base_model2.output\nx2 = layers.GlobalAveragePooling2D()(x2)\nx3 = base_model3.output\nx3 = layers.GlobalAveragePooling2D()(x3)\nx = layers.concatenate([x0,x1,x2,x3])\nx = layers.Dense(4,activation='sigmoid')(x)\nmodel = Model(inputs=(base_model0.input, base_model1.input, base_model2.input, \n    base_model3.input), outputs=x)\n</code></pre>\n\n<p>My second model uses masks in addition to labels and achieves 77% accuracy by itself. The label loss is backpropagated through the mask prediction. Then instead of using the outputted labels, we predict 1 or 0 for label based on whether mask is present or not.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F2445cef6b03266046473736d3d4a8914%2Fcls2.jpg?generation=1574170027626081&amp;alt=media\" alt=\"\"></p>\n\n<pre><code>model0 = Unet('resnet34', input_shape=(None,None,3), classes=4,\n    activation='sigmoid', encoder_freeze=True)\nmodel0.layers[-1].name = 'out1'\nx = model0.output\nx = layers.GlobalAveragePooling2D()(x)\nx = layers.Dense(4, activation='sigmoid', name='out2')(x)\nmodel = Model(inputs = model0.input, outputs = (model0.output,x))\nmodel.compile(optimizer=opt, loss={'out1':loss1,'out2':loss2}, \n    metric = {'out1':metric1, 'out2':metric2})\n</code></pre>\n\n<h1>Segmentation Model</h1>\n\n<p>My segmentation model is a collage of ideas from public kernels. Without post process, it achieves Public LB 0.650. Test time augmentation (TTAx6) increases this to LB 0.655. Using 3-Folds increases this to LB 0.660. Ensembling 7 copies with different choices for 3-Fold achieves LB 0.665. And finally removing false positives with my classifier increases this to LB 0.670. My final solution has CV 0.663 and Private LB 0.663. Here are specific details:</p>\n\n<ul>\n<li>Unet Architecture</li>\n<li>EfficientnetB2 backbone</li>\n<li>Train on 352x544 random crops from 384x576 size images</li>\n<li>Train augmentation of flips and rotate</li>\n<li>Adam Accumulate optimizer</li>\n<li>Jaccard loss</li>\n<li>Kaggle Dice metric, Kaggle accuracy metric</li>\n<li>Reduce LR on plateau and early stopping</li>\n<li>Remove masks less than 20000 pixels</li>\n<li>TTA of flips and shifts</li>\n<li>3-Fold CV and prediction</li>\n<li>Remove false positive masks with classifier</li>\n</ul>\n\n<h1>Kaggle Notebook</h1>\n\n<p>I posted a Kaggle notebook showing my segmentation model <a href=\"https://www.kaggle.com/cdeotte/cloud-solution-lb-0-670\">here</a>. It scores LB 0.665 by itself and LB 0.670 if you ensemble it with 7 copies of itself with different initialization seeds. It loads classification predictions from my offline classifier models for false positive removal.</p>\n\n<p>Thank you everyone for a fun and exciting competition. I learned a lot from reading everyone's discussions and posted code. Thank you Kaggle and Max-Planck-Institite for sharing cloud data and hosting. Congratulations to all the winners.</p>",
      "rawMarkdown": "If you could predict empty mask for every empty mask and predict full mask, (i.e. predict every pixel with `rle = '1 183750'`) for every mask, then your CV is 0.686!! Therefore a perfect classifier can win without any segmentation. The following code outputs 0.686:\n\n    train = pd.read_csv('../input/understanding_cloud_organization/train.csv')\n    train['pred'] = np.where(~train.EncodedPixels.isna(),'1 183750','')\n    train['dice'] = train.apply(lambda x: kaggle_dice(x['EncodedPixels'],x['pred']),axis=1)\n    print( train.dice.mean() )\n\n# Classification Models\nI focused most of my energy on building classification models and finally achieved 78% classification validation accuracy (on 33% holdout set, i.e. 3-Fold CV) by ensembling two crazy classifiers. The first has 4 backbones that extract features from 4 different resized input images (half size, quarter size, one sixth size, and one eighth size)\n  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fdef7ae2825082c95e50f22674c562124%2Fcls1.jpg?generation=1574169946410492&amp;alt=media)\n  \n    base_model0 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    base_model1 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    base_model2 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    base_model3 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    x0 = base_model0.output\n    x0 = layers.GlobalAveragePooling2D()(x0)\n    x1 = base_model1.output\n    x1 = layers.GlobalAveragePooling2D()(x1)\n    x2 = base_model2.output\n    x2 = layers.GlobalAveragePooling2D()(x2)\n    x3 = base_model3.output\n    x3 = layers.GlobalAveragePooling2D()(x3)\n    x = layers.concatenate([x0,x1,x2,x3])\n    x = layers.Dense(4,activation='sigmoid')(x)\n    model = Model(inputs=(base_model0.input, base_model1.input, base_model2.input, \n        base_model3.input), outputs=x)\n\nMy second model uses masks in addition to labels and achieves 77% accuracy by itself. The label loss is backpropagated through the mask prediction. Then instead of using the outputted labels, we predict 1 or 0 for label based on whether mask is present or not.\n  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F2445cef6b03266046473736d3d4a8914%2Fcls2.jpg?generation=1574170027626081&amp;alt=media)\n\n    model0 = Unet('resnet34', input_shape=(None,None,3), classes=4,\n        activation='sigmoid', encoder_freeze=True)\n    model0.layers[-1].name = 'out1'\n    x = model0.output\n    x = layers.GlobalAveragePooling2D()(x)\n    x = layers.Dense(4, activation='sigmoid', name='out2')(x)\n    model = Model(inputs = model0.input, outputs = (model0.output,x))\n    model.compile(optimizer=opt, loss={'out1':loss1,'out2':loss2}, \n        metric = {'out1':metric1, 'out2':metric2})\n\n# Segmentation Model\nMy segmentation model is a collage of ideas from public kernels. Without post process, it achieves Public LB 0.650. Test time augmentation (TTAx6) increases this to LB 0.655. Using 3-Folds increases this to LB 0.660. Ensembling 7 copies with different choices for 3-Fold achieves LB 0.665. And finally removing false positives with my classifier increases this to LB 0.670. My final solution has CV 0.663 and Private LB 0.663. Here are specific details:\n\n* Unet Architecture\n* EfficientnetB2 backbone\n* Train on 352x544 random crops from 384x576 size images\n* Train augmentation of flips and rotate\n* Adam Accumulate optimizer\n* Jaccard loss\n* Kaggle Dice metric, Kaggle accuracy metric\n* Reduce LR on plateau and early stopping\n* Remove masks less than 20000 pixels\n* TTA of flips and shifts\n* 3-Fold CV and prediction\n* Remove false positive masks with classifier\n\n# Kaggle Notebook\nI posted a Kaggle notebook showing my segmentation model [here][1]. It scores LB 0.665 by itself and LB 0.670 if you ensemble it with 7 copies of itself with different initialization seeds. It loads classification predictions from my offline classifier models for false positive removal.\n  \nThank you everyone for a fun and exciting competition. I learned a lot from reading everyone's discussions and posted code. Thank you Kaggle and Max-Planck-Institite for sharing cloud data and hosting. Congratulations to all the winners.\n\n[1]: https://www.kaggle.com/cdeotte/cloud-solution-lb-0-670",
      "votes": 70
    },
    {
      "id": 902254,
      "postDate": "2020-06-26T03:06:07.010Z",
      "content": "<p>hi <a href=\"/cdeotte\">@cdeotte</a> </p>\n\n<p>I am trying to do the multiple backbones arch and getting this error. Please help me move forward..</p>\n\n<p>Below is my model and error.</p>\n\n<p>def get_model(input_shape):</p>\n\n<p>input1 = Input(shape = (*input_shape, 3), name = 'input_1')</p>\n\n<p>i2 = Input(shape = (*input_shape, 3), name = 'input_2')\ninput2 = L.MaxPooling2D(2)(i2)</p>\n\n<p>i3 = Input(shape = (*input_shape, 3), name = 'input_3')\ninput3 = L.MaxPooling2D(2)(i3)\ninput3 = L.MaxPooling2D(2)(input3)</p>\n\n<p>base1 = DenseNet201(weights = 'imagenet', include_top = False)(input1)\nbase2 = DenseNet201(weights = 'imagenet', include_top = False)(input2)\nbase3 = DenseNet201(weights = 'imagenet', include_top = False)(input3)</p>\n\n<p>x1 = L.GlobalAveragePooling2D()(base1)\nx2 = L.GlobalAveragePooling2D()(base2)\nx3 = L.GlobalAveragePooling2D()(base3)</p>\n\n<p>x = L.concatenate([x1, x2, x3])\nout = L.Dense(train_targets.shape[1], activation = 'softmax')(x)</p>\n\n<p>model = Model(inputs = [input1, input2, input3], outputs = out)</p>\n\n<p>return model</p>\n\n<p>with strategy.scope():\nmodel = get_model(IMAGE_SIZE)\nmodel.compile(\noptimizer = 'adam',\nloss = 'categorical_crossentropy',\nmetrics = ['categorical_accuracy']\n)</p>\n\n<p>And this is the error I am getting.</p>\n\n<p>ValueError Traceback (most recent call last)\nin \n1 with strategy.scope():\n----&gt; 2 model = get_model(IMAGE_SIZE)\n3 model.compile(\n4 optimizer = 'adam',\n5 loss = 'categorical_crossentropy',</p>\n\n<p>in get_model(input_shape)\n28 out = L.Dense(train_targets.shape[1], activation = 'softmax')(x)\n29\n---&gt; 30 model = Model(inputs = [input1, input2, input3], outputs = out)\n31\n32 return model</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in init(self, *args, **kwargs)\n165\n166 def init(self, *args, **kwargs):\n--&gt; 167 super(Model, self).init(*args, **kwargs)\n168 _keras_api_gauge.get_cell('model').set(True)\n169 # Model must be created under scope of DistStrat it will be trained with.</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in init(self, *args, **kwargs)\n171 'inputs' in kwargs and 'outputs' in kwargs):\n172 # Graph network\n--&gt; 173 self._init_graph_network(*args, **kwargs)\n174 else:\n175 # Subclassed network</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)\n454 self._self_setattr_tracking = False # pylint: disable=protected-access\n455 try:\n--&gt; 456 result = method(self, *args, **kwargs)\n457 finally:\n458 self._self_setattr_tracking = previous_value # pylint: disable=protected-access</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name, **kwargs)\n305 # Keep track of the network's nodes and layers.\n306 nodes, nodes_by_depth, layers, _ = _map_graph_network(\n--&gt; 307 self.inputs, self.outputs)\n308 self._network_nodes = nodes\n309 self._nodes_by_depth = nodes_by_depth</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _map_graph_network(inputs, outputs)\n1790 'The following previous layers '\n1791 'were accessed without issue: ' +\n-&gt; 1792 str(layers_with_complete_input))\n1793 for x in nest.flatten(node.output_tensors):\n1794 computable_tensors.add(id(x))</p>\n\n<p>ValueError: Graph disconnected: cannot obtain value for tensor Tensor(\"input_3_6:0\", shape=(None, 256, 256, 3), dtype=float32) at layer \"input_3\". The following previous layers were accessed without issue: []</p>",
      "rawMarkdown": "hi @cdeotte \n\nI am trying to do the multiple backbones arch and getting this error. Please help me move forward..\n\nBelow is my model and error.\n\ndef get_model(input_shape):\n\ninput1 = Input(shape = (*input_shape, 3), name = 'input_1')\n\ni2 = Input(shape = (*input_shape, 3), name = 'input_2')\ninput2 = L.MaxPooling2D(2)(i2)\n\ni3 = Input(shape = (*input_shape, 3), name = 'input_3')\ninput3 = L.MaxPooling2D(2)(i3)\ninput3 = L.MaxPooling2D(2)(input3)\n\nbase1 = DenseNet201(weights = 'imagenet', include_top = False)(input1)\nbase2 = DenseNet201(weights = 'imagenet', include_top = False)(input2)\nbase3 = DenseNet201(weights = 'imagenet', include_top = False)(input3)\n\nx1 = L.GlobalAveragePooling2D()(base1)\nx2 = L.GlobalAveragePooling2D()(base2)\nx3 = L.GlobalAveragePooling2D()(base3)\n\nx = L.concatenate([x1, x2, x3])\nout = L.Dense(train_targets.shape[1], activation = 'softmax')(x)\n\nmodel = Model(inputs = [input1, input2, input3], outputs = out)\n\nreturn model\n\nwith strategy.scope():\nmodel = get_model(IMAGE_SIZE)\nmodel.compile(\noptimizer = 'adam',\nloss = 'categorical_crossentropy',\nmetrics = ['categorical_accuracy']\n)\n\nAnd this is the error I am getting.\n\nValueError Traceback (most recent call last)\nin \n1 with strategy.scope():\n----&gt; 2 model = get_model(IMAGE_SIZE)\n3 model.compile(\n4 optimizer = 'adam',\n5 loss = 'categorical_crossentropy',\n\nin get_model(input_shape)\n28 out = L.Dense(train_targets.shape[1], activation = 'softmax')(x)\n29\n---&gt; 30 model = Model(inputs = [input1, input2, input3], outputs = out)\n31\n32 return model\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in init(self, *args, **kwargs)\n165\n166 def init(self, *args, **kwargs):\n--&gt; 167 super(Model, self).init(*args, **kwargs)\n168 _keras_api_gauge.get_cell('model').set(True)\n169 # Model must be created under scope of DistStrat it will be trained with.\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in init(self, *args, **kwargs)\n171 'inputs' in kwargs and 'outputs' in kwargs):\n172 # Graph network\n--&gt; 173 self._init_graph_network(*args, **kwargs)\n174 else:\n175 # Subclassed network\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)\n454 self._self_setattr_tracking = False # pylint: disable=protected-access\n455 try:\n--&gt; 456 result = method(self, *args, **kwargs)\n457 finally:\n458 self._self_setattr_tracking = previous_value # pylint: disable=protected-access\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name, **kwargs)\n305 # Keep track of the network's nodes and layers.\n306 nodes, nodes_by_depth, layers, _ = _map_graph_network(\n--&gt; 307 self.inputs, self.outputs)\n308 self._network_nodes = nodes\n309 self._nodes_by_depth = nodes_by_depth\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _map_graph_network(inputs, outputs)\n1790 'The following previous layers '\n1791 'were accessed without issue: ' +\n-&gt; 1792 str(layers_with_complete_input))\n1793 for x in nest.flatten(node.output_tensors):\n1794 computable_tensors.add(id(x))\n\nValueError: Graph disconnected: cannot obtain value for tensor Tensor(\"input_3_6:0\", shape=(None, 256, 256, 3), dtype=float32) at layer \"input_3\". The following previous layers were accessed without issue: []",
      "votes": 1
    },
    {
      "id": 902210,
      "postDate": "2020-06-26T02:00:15.687Z",
      "content": "<p>Thanks <a href=\"/cdeotte\">@cdeotte</a> for sharing your solution. I am gonna use it in another competition. 👍 </p>",
      "rawMarkdown": "Thanks @cdeotte for sharing your solution. I am gonna use it in another competition. 👍 ",
      "votes": 2
    },
    {
      "id": 678412,
      "postDate": "2019-11-21T10:47:54.873Z",
      "content": "<p>Nice approach ! Congratulations Chris !</p>",
      "rawMarkdown": "Nice approach ! Congratulations Chris !",
      "votes": 1
    },
    {
      "id": 677546,
      "postDate": "2019-11-20T10:05:54.957Z",
      "content": "<p>Interesting, nice!</p>",
      "rawMarkdown": "Interesting, nice!",
      "votes": 1
    },
    {
      "id": 677301,
      "postDate": "2019-11-20T02:34:58.937Z",
      "content": "<p>This KKKKlasifier is like KKKKrazy! Thanks for your sharing</p>",
      "rawMarkdown": "This KKKKlasifier is like KKKKrazy! Thanks for your sharing",
      "votes": 1
    },
    {
      "id": 676947,
      "postDate": "2019-11-19T16:36:08.483Z",
      "content": "<p>Best summary so far</p>",
      "rawMarkdown": "Best summary so far",
      "votes": 1
    },
    {
      "id": 676902,
      "postDate": "2019-11-19T15:34:51.653Z",
      "content": "<p>Very nice job <a href=\"/cdeotte\">@cdeotte</a> , it's always good to read your solutions, you always find very creative yet efficient solutions.</p>",
      "rawMarkdown": "Very nice job @cdeotte , it's always good to read your solutions, you always find very creative yet efficient solutions.",
      "votes": 1
    },
    {
      "id": 676879,
      "postDate": "2019-11-19T15:10:29.253Z",
      "content": "<p>Congratulations <a href=\"/cdeotte\">@cdeotte</a>  and thanks so much for sharing your solution with codes.</p>",
      "rawMarkdown": "Congratulations @cdeotte  and thanks so much for sharing your solution with codes.",
      "votes": 1
    },
    {
      "id": 676873,
      "postDate": "2019-11-19T15:05:51.110Z",
      "content": "<p>congratulations and thanks for sharing, you are always impressive:)</p>",
      "rawMarkdown": "congratulations and thanks for sharing, you are always impressive:)",
      "votes": 1
    },
    {
      "id": 676869,
      "postDate": "2019-11-19T15:02:46.013Z",
      "content": "<p>Congratulations ! This is fantastic and unique !</p>",
      "rawMarkdown": "Congratulations ! This is fantastic and unique !",
      "votes": 1
    },
    {
      "id": 676842,
      "postDate": "2019-11-19T14:43:03.063Z",
      "content": "<p>Congratulations\nGreat Write-Up\nThank you for Sharing your Insights &amp; Approach! <a href=\"/cdeotte\">@cdeotte</a> </p>",
      "rawMarkdown": "Congratulations\nGreat Write-Up\nThank you for Sharing your Insights &amp; Approach! @cdeotte ",
      "votes": 1
    },
    {
      "id": 676794,
      "postDate": "2019-11-19T14:06:18.417Z",
      "content": "<p><a href=\"/cdeotte\">@cdeotte</a> I enjoy whenever you participate in any competition Because you approach and knowledge sharing is awesome. Congrats and Thanks for sharing new approach.</p>",
      "rawMarkdown": "@cdeotte I enjoy whenever you participate in any competition Because you approach and knowledge sharing is awesome. Congrats and Thanks for sharing new approach.",
      "votes": 1
    },
    {
      "id": 676881,
      "postDate": "2019-11-19T15:11:50.507Z",
      "content": "<p>Wow. Simply amazing <a href=\"/cdeotte\">@cdeotte</a>. Always getting some unique approaches and unique way to solve a problem be it Tabular Data Competition or Image Segmentation competition :). Thanks for sharing </p>",
      "rawMarkdown": "Wow. Simply amazing @cdeotte. Always getting some unique approaches and unique way to solve a problem be it Tabular Data Competition or Image Segmentation competition :). Thanks for sharing ",
      "votes": 2
    },
    {
      "id": 934036,
      "postDate": "2020-07-18T07:44:45.933Z",
      "content": "<p>Hey <a href=\"/cdeotte\">@cdeotte</a>  thanks for this epic idea, I am using this idea in a different model.\nI am pretty much working in pytorch, so has anyone be able to implement on pytorch?\nits a little hard in porch as we cant set None in the dimensions</p>",
      "rawMarkdown": "Hey @cdeotte  thanks for this epic idea, I am using this idea in a different model.\nI am pretty much working in pytorch, so has anyone be able to implement on pytorch?\nits a little hard in porch as we cant set None in the dimensions"
    },
    {
      "id": 895703,
      "postDate": "2020-06-21T14:49:31.473Z",
      "rawMarkdown": "",
      "votes": -1,
      "isDeleted": true
    },
    {
      "id": 684145,
      "postDate": "2019-11-29T09:46:06.053Z",
      "content": "<p>Interesting, thanks for sharing.</p>",
      "rawMarkdown": "Interesting, thanks for sharing.",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 902254,
      "author_name": "msafi04",
      "author_url": "",
      "post_date": "2020-06-26T03:06:07.010000",
      "content": "<p>hi <a href=\"/cdeotte\">@cdeotte</a> </p>\n\n<p>I am trying to do the multiple backbones arch and getting this error. Please help me move forward..</p>\n\n<p>Below is my model and error.</p>\n\n<p>def get_model(input_shape):</p>\n\n<p>input1 = Input(shape = (*input_shape, 3), name = 'input_1')</p>\n\n<p>i2 = Input(shape = (*input_shape, 3), name = 'input_2')\ninput2 = L.MaxPooling2D(2)(i2)</p>\n\n<p>i3 = Input(shape = (*input_shape, 3), name = 'input_3')\ninput3 = L.MaxPooling2D(2)(i3)\ninput3 = L.MaxPooling2D(2)(input3)</p>\n\n<p>base1 = DenseNet201(weights = 'imagenet', include_top = False)(input1)\nbase2 = DenseNet201(weights = 'imagenet', include_top = False)(input2)\nbase3 = DenseNet201(weights = 'imagenet', include_top = False)(input3)</p>\n\n<p>x1 = L.GlobalAveragePooling2D()(base1)\nx2 = L.GlobalAveragePooling2D()(base2)\nx3 = L.GlobalAveragePooling2D()(base3)</p>\n\n<p>x = L.concatenate([x1, x2, x3])\nout = L.Dense(train_targets.shape[1], activation = 'softmax')(x)</p>\n\n<p>model = Model(inputs = [input1, input2, input3], outputs = out)</p>\n\n<p>return model</p>\n\n<p>with strategy.scope():\nmodel = get_model(IMAGE_SIZE)\nmodel.compile(\noptimizer = 'adam',\nloss = 'categorical_crossentropy',\nmetrics = ['categorical_accuracy']\n)</p>\n\n<p>And this is the error I am getting.</p>\n\n<p>ValueError Traceback (most recent call last)\nin \n1 with strategy.scope():\n----&gt; 2 model = get_model(IMAGE_SIZE)\n3 model.compile(\n4 optimizer = 'adam',\n5 loss = 'categorical_crossentropy',</p>\n\n<p>in get_model(input_shape)\n28 out = L.Dense(train_targets.shape[1], activation = 'softmax')(x)\n29\n---&gt; 30 model = Model(inputs = [input1, input2, input3], outputs = out)\n31\n32 return model</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in init(self, *args, **kwargs)\n165\n166 def init(self, *args, **kwargs):\n--&gt; 167 super(Model, self).init(*args, **kwargs)\n168 _keras_api_gauge.get_cell('model').set(True)\n169 # Model must be created under scope of DistStrat it will be trained with.</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in init(self, *args, **kwargs)\n171 'inputs' in kwargs and 'outputs' in kwargs):\n172 # Graph network\n--&gt; 173 self._init_graph_network(*args, **kwargs)\n174 else:\n175 # Subclassed network</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)\n454 self._self_setattr_tracking = False # pylint: disable=protected-access\n455 try:\n--&gt; 456 result = method(self, *args, **kwargs)\n457 finally:\n458 self._self_setattr_tracking = previous_value # pylint: disable=protected-access</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name, **kwargs)\n305 # Keep track of the network's nodes and layers.\n306 nodes, nodes_by_depth, layers, _ = _map_graph_network(\n--&gt; 307 self.inputs, self.outputs)\n308 self._network_nodes = nodes\n309 self._nodes_by_depth = nodes_by_depth</p>\n\n<p>/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _map_graph_network(inputs, outputs)\n1790 'The following previous layers '\n1791 'were accessed without issue: ' +\n-&gt; 1792 str(layers_with_complete_input))\n1793 for x in nest.flatten(node.output_tensors):\n1794 computable_tensors.add(id(x))</p>\n\n<p>ValueError: Graph disconnected: cannot obtain value for tensor Tensor(\"input_3_6:0\", shape=(None, 256, 256, 3), dtype=float32) at layer \"input_3\". The following previous layers were accessed without issue: []</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 902210,
      "author_name": "Redwan Sony",
      "author_url": "",
      "post_date": "2020-06-26T02:00:15.687000",
      "content": "<p>Thanks <a href=\"/cdeotte\">@cdeotte</a> for sharing your solution. I am gonna use it in another competition. 👍 </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 678412,
      "author_name": "Vlad Vaduva",
      "author_url": "",
      "post_date": "2019-11-21T10:47:54.873000",
      "content": "<p>Nice approach ! Congratulations Chris !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 677546,
      "author_name": "Karl Hornlund",
      "author_url": "",
      "post_date": "2019-11-20T10:05:54.957000",
      "content": "<p>Interesting, nice!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 677301,
      "author_name": "Overfit Queen",
      "author_url": "",
      "post_date": "2019-11-20T02:34:58.937000",
      "content": "<p>This KKKKlasifier is like KKKKrazy! Thanks for your sharing</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676947,
      "author_name": "Mobassir",
      "author_url": "",
      "post_date": "2019-11-19T16:36:08.483000",
      "content": "<p>Best summary so far</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676902,
      "author_name": "DimitreOliveira",
      "author_url": "",
      "post_date": "2019-11-19T15:34:51.653000",
      "content": "<p>Very nice job <a href=\"/cdeotte\">@cdeotte</a> , it's always good to read your solutions, you always find very creative yet efficient solutions.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676879,
      "author_name": "YaGana Sheriff-Hussaini",
      "author_url": "",
      "post_date": "2019-11-19T15:10:29.253000",
      "content": "<p>Congratulations <a href=\"/cdeotte\">@cdeotte</a>  and thanks so much for sharing your solution with codes.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676873,
      "author_name": "liuze",
      "author_url": "",
      "post_date": "2019-11-19T15:05:51.110000",
      "content": "<p>congratulations and thanks for sharing, you are always impressive:)</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676869,
      "author_name": "Nirjhar Roy",
      "author_url": "",
      "post_date": "2019-11-19T15:02:46.013000",
      "content": "<p>Congratulations ! This is fantastic and unique !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676842,
      "author_name": "Ailurophile",
      "author_url": "",
      "post_date": "2019-11-19T14:43:03.063000",
      "content": "<p>Congratulations\nGreat Write-Up\nThank you for Sharing your Insights &amp; Approach! <a href=\"/cdeotte\">@cdeotte</a> </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676794,
      "author_name": "SeshuRaju 🧘‍♂️",
      "author_url": "",
      "post_date": "2019-11-19T14:06:18.417000",
      "content": "<p><a href=\"/cdeotte\">@cdeotte</a> I enjoy whenever you participate in any competition Because you approach and knowledge sharing is awesome. Congrats and Thanks for sharing new approach.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 676881,
      "author_name": "Manoj Prabhakar",
      "author_url": "",
      "post_date": "2019-11-19T15:11:50.507000",
      "content": "<p>Wow. Simply amazing <a href=\"/cdeotte\">@cdeotte</a>. Always getting some unique approaches and unique way to solve a problem be it Tabular Data Competition or Image Segmentation competition :). Thanks for sharing </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 934036,
      "author_name": "DeepUnderstanding",
      "author_url": "",
      "post_date": "2020-07-18T07:44:45.933000",
      "content": "<p>Hey <a href=\"/cdeotte\">@cdeotte</a>  thanks for this epic idea, I am using this idea in a different model.\nI am pretty much working in pytorch, so has anyone be able to implement on pytorch?\nits a little hard in porch as we cant set None in the dimensions</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 895703,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-06-21T14:49:31.473000",
      "content": "",
      "votes": -1,
      "replies": []
    },
    {
      "id": 684145,
      "author_name": "Utkarsh Agarwal",
      "author_url": "",
      "post_date": "2019-11-29T09:46:06.053000",
      "content": "<p>Interesting, thanks for sharing.</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "676784": "If you could predict empty mask for every empty mask and predict full mask, (i.e. predict every pixel with `rle = '1 183750'`) for every mask, then your CV is 0.686!! Therefore a perfect classifier can win without any segmentation. The following code outputs 0.686:\n\n    train = pd.read_csv('../input/understanding_cloud_organization/train.csv')\n    train['pred'] = np.where(~train.EncodedPixels.isna(),'1 183750','')\n    train['dice'] = train.apply(lambda x: kaggle_dice(x['EncodedPixels'],x['pred']),axis=1)\n    print( train.dice.mean() )\n\n# Classification Models\nI focused most of my energy on building classification models and finally achieved 78% classification validation accuracy (on 33% holdout set, i.e. 3-Fold CV) by ensembling two crazy classifiers. The first has 4 backbones that extract features from 4 different resized input images (half size, quarter size, one sixth size, and one eighth size)\n  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fdef7ae2825082c95e50f22674c562124%2Fcls1.jpg?generation=1574169946410492&amp;alt=media)\n  \n    base_model0 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    base_model1 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    base_model2 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    base_model3 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))\n    x0 = base_model0.output\n    x0 = layers.GlobalAveragePooling2D()(x0)\n    x1 = base_model1.output\n    x1 = layers.GlobalAveragePooling2D()(x1)\n    x2 = base_model2.output\n    x2 = layers.GlobalAveragePooling2D()(x2)\n    x3 = base_model3.output\n    x3 = layers.GlobalAveragePooling2D()(x3)\n    x = layers.concatenate([x0,x1,x2,x3])\n    x = layers.Dense(4,activation='sigmoid')(x)\n    model = Model(inputs=(base_model0.input, base_model1.input, base_model2.input, \n        base_model3.input), outputs=x)\n\nMy second model uses masks in addition to labels and achieves 77% accuracy by itself. The label loss is backpropagated through the mask prediction. Then instead of using the outputted labels, we predict 1 or 0 for label based on whether mask is present or not.\n  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F2445cef6b03266046473736d3d4a8914%2Fcls2.jpg?generation=1574170027626081&amp;alt=media)\n\n    model0 = Unet('resnet34', input_shape=(None,None,3), classes=4,\n        activation='sigmoid', encoder_freeze=True)\n    model0.layers[-1].name = 'out1'\n    x = model0.output\n    x = layers.GlobalAveragePooling2D()(x)\n    x = layers.Dense(4, activation='sigmoid', name='out2')(x)\n    model = Model(inputs = model0.input, outputs = (model0.output,x))\n    model.compile(optimizer=opt, loss={'out1':loss1,'out2':loss2}, \n        metric = {'out1':metric1, 'out2':metric2})\n\n# Segmentation Model\nMy segmentation model is a collage of ideas from public kernels. Without post process, it achieves Public LB 0.650. Test time augmentation (TTAx6) increases this to LB 0.655. Using 3-Folds increases this to LB 0.660. Ensembling 7 copies with different choices for 3-Fold achieves LB 0.665. And finally removing false positives with my classifier increases this to LB 0.670. My final solution has CV 0.663 and Private LB 0.663. Here are specific details:\n\n* Unet Architecture\n* EfficientnetB2 backbone\n* Train on 352x544 random crops from 384x576 size images\n* Train augmentation of flips and rotate\n* Adam Accumulate optimizer\n* Jaccard loss\n* Kaggle Dice metric, Kaggle accuracy metric\n* Reduce LR on plateau and early stopping\n* Remove masks less than 20000 pixels\n* TTA of flips and shifts\n* 3-Fold CV and prediction\n* Remove false positive masks with classifier\n\n# Kaggle Notebook\nI posted a Kaggle notebook showing my segmentation model [here][1]. It scores LB 0.665 by itself and LB 0.670 if you ensemble it with 7 copies of itself with different initialization seeds. It loads classification predictions from my offline classifier models for false positive removal.\n  \nThank you everyone for a fun and exciting competition. I learned a lot from reading everyone's discussions and posted code. Thank you Kaggle and Max-Planck-Institite for sharing cloud data and hosting. Congratulations to all the winners.\n\n[1]: https://www.kaggle.com/cdeotte/cloud-solution-lb-0-670",
    "902254": "hi @cdeotte \n\nI am trying to do the multiple backbones arch and getting this error. Please help me move forward..\n\nBelow is my model and error.\n\ndef get_model(input_shape):\n\ninput1 = Input(shape = (*input_shape, 3), name = 'input_1')\n\ni2 = Input(shape = (*input_shape, 3), name = 'input_2')\ninput2 = L.MaxPooling2D(2)(i2)\n\ni3 = Input(shape = (*input_shape, 3), name = 'input_3')\ninput3 = L.MaxPooling2D(2)(i3)\ninput3 = L.MaxPooling2D(2)(input3)\n\nbase1 = DenseNet201(weights = 'imagenet', include_top = False)(input1)\nbase2 = DenseNet201(weights = 'imagenet', include_top = False)(input2)\nbase3 = DenseNet201(weights = 'imagenet', include_top = False)(input3)\n\nx1 = L.GlobalAveragePooling2D()(base1)\nx2 = L.GlobalAveragePooling2D()(base2)\nx3 = L.GlobalAveragePooling2D()(base3)\n\nx = L.concatenate([x1, x2, x3])\nout = L.Dense(train_targets.shape[1], activation = 'softmax')(x)\n\nmodel = Model(inputs = [input1, input2, input3], outputs = out)\n\nreturn model\n\nwith strategy.scope():\nmodel = get_model(IMAGE_SIZE)\nmodel.compile(\noptimizer = 'adam',\nloss = 'categorical_crossentropy',\nmetrics = ['categorical_accuracy']\n)\n\nAnd this is the error I am getting.\n\nValueError Traceback (most recent call last)\nin \n1 with strategy.scope():\n----&gt; 2 model = get_model(IMAGE_SIZE)\n3 model.compile(\n4 optimizer = 'adam',\n5 loss = 'categorical_crossentropy',\n\nin get_model(input_shape)\n28 out = L.Dense(train_targets.shape[1], activation = 'softmax')(x)\n29\n---&gt; 30 model = Model(inputs = [input1, input2, input3], outputs = out)\n31\n32 return model\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in init(self, *args, **kwargs)\n165\n166 def init(self, *args, **kwargs):\n--&gt; 167 super(Model, self).init(*args, **kwargs)\n168 _keras_api_gauge.get_cell('model').set(True)\n169 # Model must be created under scope of DistStrat it will be trained with.\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in init(self, *args, **kwargs)\n171 'inputs' in kwargs and 'outputs' in kwargs):\n172 # Graph network\n--&gt; 173 self._init_graph_network(*args, **kwargs)\n174 else:\n175 # Subclassed network\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)\n454 self._self_setattr_tracking = False # pylint: disable=protected-access\n455 try:\n--&gt; 456 result = method(self, *args, **kwargs)\n457 finally:\n458 self._self_setattr_tracking = previous_value # pylint: disable=protected-access\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name, **kwargs)\n305 # Keep track of the network's nodes and layers.\n306 nodes, nodes_by_depth, layers, _ = _map_graph_network(\n--&gt; 307 self.inputs, self.outputs)\n308 self._network_nodes = nodes\n309 self._nodes_by_depth = nodes_by_depth\n\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in _map_graph_network(inputs, outputs)\n1790 'The following previous layers '\n1791 'were accessed without issue: ' +\n-&gt; 1792 str(layers_with_complete_input))\n1793 for x in nest.flatten(node.output_tensors):\n1794 computable_tensors.add(id(x))\n\nValueError: Graph disconnected: cannot obtain value for tensor Tensor(\"input_3_6:0\", shape=(None, 256, 256, 3), dtype=float32) at layer \"input_3\". The following previous layers were accessed without issue: []",
    "902210": "Thanks @cdeotte for sharing your solution. I am gonna use it in another competition. 👍 ",
    "678412": "Nice approach ! Congratulations Chris !",
    "677546": "Interesting, nice!",
    "677301": "This KKKKlasifier is like KKKKrazy! Thanks for your sharing",
    "676947": "Best summary so far",
    "676902": "Very nice job @cdeotte , it's always good to read your solutions, you always find very creative yet efficient solutions.",
    "676879": "Congratulations @cdeotte  and thanks so much for sharing your solution with codes.",
    "676873": "congratulations and thanks for sharing, you are always impressive:)",
    "676869": "Congratulations ! This is fantastic and unique !",
    "676842": "Congratulations\nGreat Write-Up\nThank you for Sharing your Insights &amp; Approach! @cdeotte ",
    "676794": "@cdeotte I enjoy whenever you participate in any competition Because you approach and knowledge sharing is awesome. Congrats and Thanks for sharing new approach.",
    "676881": "Wow. Simply amazing @cdeotte. Always getting some unique approaches and unique way to solve a problem be it Tabular Data Competition or Image Segmentation competition :). Thanks for sharing ",
    "934036": "Hey @cdeotte  thanks for this epic idea, I am using this idea in a different model.\nI am pretty much working in pytorch, so has anyone be able to implement on pytorch?\nits a little hard in porch as we cant set None in the dimensions",
    "895703": "",
    "684145": "Interesting, thanks for sharing."
  }
}