{
  "id": 165194,
  "title": "Multi-Sample Dropout implement: Better Dropout! Better Accuracy!",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/165194",
  "author_name": "",
  "post_date": "2020-07-08T19:38:13.930301400Z",
  "votes": 13,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Dropout is an efficient regularization instrument for avoiding overfitting of deep neural networks.\nI try to reproduce the results presented in this paper; which introduced a technique called Multi-Sample Dropout. As declared by the author, its scopes are:\n* accelerate training and improve generalization over the original dropout;\n* reduce computational cost because most of the computation time is consumed in the layers \n* below (often convolutional or recurrent) and the weights in the layers at the top are shared;\n* achieve lower error rates and losses.\n</p>",
  "messages": [
    {
      "id": "920779",
      "postDate": "07/08/2020 19:38:13",
      "content": "<p>Dropout is an efficient regularization instrument for avoiding overfitting of deep neural networks.\nI try to reproduce the results presented in this paper; which introduced a technique called Multi-Sample Dropout. As declared by the author, its scopes are:\n* accelerate training and improve generalization over the original dropout;\n* reduce computational cost because most of the computation time is consumed in the layers \n* below (often convolutional or recurrent) and the weights in the layers at the top are shared;\n* achieve lower error rates and losses.\n</p>",
      "rawMarkdown": "Dropout is an efficient regularization instrument for avoiding overfitting of deep neural networks.\nI try to reproduce the results presented in this paper; which introduced a technique called Multi-Sample Dropout. As declared by the author, its scopes are:\n* accelerate training and improve generalization over the original dropout;\n* reduce computational cost because most of the computation time is consumed in the layers \n* below (often convolutional or recurrent) and the weights in the layers at the top are shared;\n* achieve lower error rates and losses.\n![multidropout](https://miro.medium.com/max/700/1*mdBXp3-D7G7mTcKDZHui8g.png)",
      "votes": null
    },
    {
      "id": "920829",
      "postDate": "07/08/2020 20:25:43",
      "content": "<p><a href=\"https://github.com/lonePatient/multi-sample_dropout_pytorch/blob/master/nn.py\">Sample code</a></p>\n\n<p>```\nclass ResNet(nn.Module):\n    def <strong>init</strong>(self, ResidualBlock, num_classes,dropout_num=8,dropout_p=0.5):\n        ...\n        self.fc = nn.Linear(128,num_classes)\n        self.dropouts = nn.ModuleList([nn.Dropout(dropout_p) for _ in range(dropout_num)])</p>\n\n<pre><code>def forward(self, x,y = None,loss_fn = None):\n    ...\n    feature = F.avg_pool2d(out, 4)\n    if len(self.dropouts) == 0:\n        out = feature.view(feature.size(0), -1)\n        out = self.fc(out)\n        if loss_fn is not None:\n            loss = loss_fn(out,y)\n            return out,loss\n        return out,None\n    else:\n        for i,dropout in enumerate(self.dropouts):\n            if i== 0:\n                out = dropout(feature)\n                out = out.view(out.size(0),-1)\n                out = self.fc(out)\n                if loss_fn is not None:\n                    loss = loss_fn(out, y)\n            else:\n                temp_out = dropout(feature)\n                temp_out = temp_out.view(temp_out.size(0),-1)\n                out =out+ self.fc(temp_out)\n                if loss_fn is not None:\n                    loss = loss+loss_fn(temp_out, y)\n        if loss_fn is not None:\n            return out / len(self.dropouts),loss / len(self.dropouts)\n        return out,None\n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "[Sample code](https://github.com/lonePatient/multi-sample_dropout_pytorch/blob/master/nn.py)\n\n```\nclass ResNet(nn.Module):\n    def __init__(self, ResidualBlock, num_classes,dropout_num=8,dropout_p=0.5):\n        ...\n        self.fc = nn.Linear(128,num_classes)\n        self.dropouts = nn.ModuleList([nn.Dropout(dropout_p) for _ in range(dropout_num)])\n\n    def forward(self, x,y = None,loss_fn = None):\n        ...\n        feature = F.avg_pool2d(out, 4)\n        if len(self.dropouts) == 0:\n            out = feature.view(feature.size(0), -1)\n            out = self.fc(out)\n            if loss_fn is not None:\n                loss = loss_fn(out,y)\n                return out,loss\n            return out,None\n        else:\n            for i,dropout in enumerate(self.dropouts):\n                if i== 0:\n                    out = dropout(feature)\n                    out = out.view(out.size(0),-1)\n                    out = self.fc(out)\n                    if loss_fn is not None:\n                        loss = loss_fn(out, y)\n                else:\n                    temp_out = dropout(feature)\n                    temp_out = temp_out.view(temp_out.size(0),-1)\n                    out =out+ self.fc(temp_out)\n                    if loss_fn is not None:\n                        loss = loss+loss_fn(temp_out, y)\n            if loss_fn is not None:\n                return out / len(self.dropouts),loss / len(self.dropouts)\n            return out,None\n```",
      "votes": null
    },
    {
      "id": "920834",
      "postDate": "07/08/2020 20:30:20",
      "content": "<p>I mean Dropout parameters change in each loop. Sample code. It work better!\n<code>\n        dense = []\n        FC = tf.keras.layers.Dense(32, activation='relu')\n        for p in np.linspace(0.1,0.5, 5):\n            x_ = tf.keras.layers.Dropout(p)(x)\n            x_ = FC(x_)\n            x_ = tf.keras.layers.Dense(1, activation='sigmoid')(x_)\n            dense.append(x_)\n        x = tf.keras.layers.Average()(dense)\n</code></p>",
      "rawMarkdown": "I mean Dropout parameters change in each loop. Sample code. It work better!\n```\n        dense = []\n        FC = tf.keras.layers.Dense(32, activation='relu')\n        for p in np.linspace(0.1,0.5, 5):\n            x_ = tf.keras.layers.Dropout(p)(x)\n            x_ = FC(x_)\n            x_ = tf.keras.layers.Dense(1, activation='sigmoid')(x_)\n            dense.append(x_)\n        x = tf.keras.layers.Average()(dense)\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 920829,
      "author_name": "sirishks",
      "author_url": "",
      "post_date": "07/08/2020 20:25:43",
      "content": "<p><a href=\"https://github.com/lonePatient/multi-sample_dropout_pytorch/blob/master/nn.py\">Sample code</a></p>\n\n<p>```\nclass ResNet(nn.Module):\n    def <strong>init</strong>(self, ResidualBlock, num_classes,dropout_num=8,dropout_p=0.5):\n        ...\n        self.fc = nn.Linear(128,num_classes)\n        self.dropouts = nn.ModuleList([nn.Dropout(dropout_p) for _ in range(dropout_num)])</p>\n\n<pre><code>def forward(self, x,y = None,loss_fn = None):\n    ...\n    feature = F.avg_pool2d(out, 4)\n    if len(self.dropouts) == 0:\n        out = feature.view(feature.size(0), -1)\n        out = self.fc(out)\n        if loss_fn is not None:\n            loss = loss_fn(out,y)\n            return out,loss\n        return out,None\n    else:\n        for i,dropout in enumerate(self.dropouts):\n            if i== 0:\n                out = dropout(feature)\n                out = out.view(out.size(0),-1)\n                out = self.fc(out)\n                if loss_fn is not None:\n                    loss = loss_fn(out, y)\n            else:\n                temp_out = dropout(feature)\n                temp_out = temp_out.view(temp_out.size(0),-1)\n                out =out+ self.fc(temp_out)\n                if loss_fn is not None:\n                    loss = loss+loss_fn(temp_out, y)\n        if loss_fn is not None:\n            return out / len(self.dropouts),loss / len(self.dropouts)\n        return out,None\n</code></pre>\n\n<p>```</p>",
      "votes": null,
      "replies": [
        {
          "id": 920834,
          "author_name": "doanquanvietnamca",
          "author_url": "",
          "post_date": "07/08/2020 20:30:20",
          "content": "<p>I mean Dropout parameters change in each loop. Sample code. It work better!\n<code>\n        dense = []\n        FC = tf.keras.layers.Dense(32, activation='relu')\n        for p in np.linspace(0.1,0.5, 5):\n            x_ = tf.keras.layers.Dropout(p)(x)\n            x_ = FC(x_)\n            x_ = tf.keras.layers.Dense(1, activation='sigmoid')(x_)\n            dense.append(x_)\n        x = tf.keras.layers.Average()(dense)\n</code></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "920779": "Dropout is an efficient regularization instrument for avoiding overfitting of deep neural networks.\nI try to reproduce the results presented in this paper; which introduced a technique called Multi-Sample Dropout. As declared by the author, its scopes are:\n* accelerate training and improve generalization over the original dropout;\n* reduce computational cost because most of the computation time is consumed in the layers \n* below (often convolutional or recurrent) and the weights in the layers at the top are shared;\n* achieve lower error rates and losses.\n![multidropout](https://miro.medium.com/max/700/1*mdBXp3-D7G7mTcKDZHui8g.png)",
    "920829": "[Sample code](https://github.com/lonePatient/multi-sample_dropout_pytorch/blob/master/nn.py)\n\n```\nclass ResNet(nn.Module):\n    def __init__(self, ResidualBlock, num_classes,dropout_num=8,dropout_p=0.5):\n        ...\n        self.fc = nn.Linear(128,num_classes)\n        self.dropouts = nn.ModuleList([nn.Dropout(dropout_p) for _ in range(dropout_num)])\n\n    def forward(self, x,y = None,loss_fn = None):\n        ...\n        feature = F.avg_pool2d(out, 4)\n        if len(self.dropouts) == 0:\n            out = feature.view(feature.size(0), -1)\n            out = self.fc(out)\n            if loss_fn is not None:\n                loss = loss_fn(out,y)\n                return out,loss\n            return out,None\n        else:\n            for i,dropout in enumerate(self.dropouts):\n                if i== 0:\n                    out = dropout(feature)\n                    out = out.view(out.size(0),-1)\n                    out = self.fc(out)\n                    if loss_fn is not None:\n                        loss = loss_fn(out, y)\n                else:\n                    temp_out = dropout(feature)\n                    temp_out = temp_out.view(temp_out.size(0),-1)\n                    out =out+ self.fc(temp_out)\n                    if loss_fn is not None:\n                        loss = loss+loss_fn(temp_out, y)\n            if loss_fn is not None:\n                return out / len(self.dropouts),loss / len(self.dropouts)\n            return out,None\n```",
    "920834": "I mean Dropout parameters change in each loop. Sample code. It work better!\n```\n        dense = []\n        FC = tf.keras.layers.Dense(32, activation='relu')\n        for p in np.linspace(0.1,0.5, 5):\n            x_ = tf.keras.layers.Dropout(p)(x)\n            x_ = FC(x_)\n            x_ = tf.keras.layers.Dense(1, activation='sigmoid')(x_)\n            dense.append(x_)\n        x = tf.keras.layers.Average()(dense)\n```"
  },
  "source": "meta"
}