{
  "id": 214799,
  "title": "How do you set up the network head?",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/214799",
  "author_name": "",
  "post_date": "2021-01-27T17:11:40.910226700Z",
  "votes": 5,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I am trying to set up a network head after globalaveragepooling. It works best for me if I use Dropout(0.3) and that's it. What works for you?</p>",
  "messages": [
    {
      "id": "1173077",
      "postDate": "01/27/2021 17:11:40",
      "content": "<p>I am trying to set up a network head after globalaveragepooling. It works best for me if I use Dropout(0.3) and that's it. What works for you?</p>",
      "rawMarkdown": "I am trying to set up a network head after globalaveragepooling. It works best for me if I use Dropout(0.3) and that's it. What works for you?",
      "votes": null
    },
    {
      "id": "1173147",
      "postDate": "01/27/2021 17:35:41",
      "content": "<p>I used it without dropout and get good results. I didn't use dropout because somewhere I learned that GAP extract the features from the last ConvBlock so how can I drop features. What do you think about this?</p>",
      "rawMarkdown": "I used it without dropout and get good results. I didn't use dropout because somewhere I learned that GAP extract the features from the last ConvBlock so how can I drop features. What do you think about this?",
      "votes": null
    },
    {
      "id": "1173271",
      "postDate": "01/27/2021 19:02:35",
      "content": "<p>For me <code>Dropout</code> doesn't make any perceptible CV changes, so I directly put just 5 <code>Dense</code> output layers.</p>",
      "rawMarkdown": "For me `Dropout` doesn't make any perceptible CV changes, so I directly put just 5 `Dense` output layers.",
      "votes": null
    },
    {
      "id": "1173433",
      "postDate": "01/27/2021 21:45:59",
      "content": "<p>I always start with the 'head' architecture from the base network; the head looks a bit different from network to network. If some dropout is included and I see some overfitting I usually increase it. Don't know if this is the best practise though looking at my current scores…</p>",
      "rawMarkdown": "I always start with the 'head' architecture from the base network; the head looks a bit different from network to network. If some dropout is included and I see some overfitting I usually increase it. Don't know if this is the best practise though looking at my current scores...",
      "votes": null
    },
    {
      "id": "1173555",
      "postDate": "01/28/2021 00:36:55",
      "content": "<p>Dropout does not remove features - it just resets the weights for a random selection of the features.</p>\n<p>I always use dropout to help close the gap between training and validation accuracy.  But I do start with low number (0.05).  If I like the model and a large gap exists than I will increment up.  For the models I have created for this competition 0.5 has always been too large - the model never learns.  </p>\n<p>It does slow down the rate of learning which is why I use a very small number in the initial stages of looking at a new model.</p>",
      "rawMarkdown": "Dropout does not remove features - it just resets the weights for a random selection of the features.\n\nI always use dropout to help close the gap between training and validation accuracy.  But I do start with low number (0.05).  If I like the model and a large gap exists than I will increment up.  For the models I have created for this competition 0.5 has always been too large - the model never learns.  \n\nIt does slow down the rate of learning which is why I use a very small number in the initial stages of looking at a new model.",
      "votes": null
    },
    {
      "id": "1174574",
      "postDate": "01/28/2021 15:34:35",
      "content": "<p>Actually dropout doesn't reset the weights, drop connect rate does. Dropout zeroes out connections between layers of neurons, but they essentially serve the same purpose. Just FYI.</p>",
      "rawMarkdown": "Actually dropout doesn't reset the weights, drop connect rate does. Dropout zeroes out connections between layers of neurons, but they essentially serve the same purpose. Just FYI.",
      "votes": null
    },
    {
      "id": "1176004",
      "postDate": "01/29/2021 12:56:57",
      "content": "<p>I use global average pooling, too. And Dropout(0.4). At last, of course, Dense layer with softmax. I think last part of layer is not that important… </p>",
      "rawMarkdown": "I use global average pooling, too. And Dropout(0.4). At last, of course, Dense layer with softmax. I think last part of layer is not that important...",
      "votes": null
    },
    {
      "id": "1179172",
      "postDate": "01/31/2021 11:30:28",
      "content": "<p>WOW! IT IS REALLY INTRESTING<br>\nUPVOTED!<br>\n<a href=\"https://www.kaggle.com/ashokkumarbibbab/covid-19-vaccination-process\" target=\"_blank\">https://www.kaggle.com/ashokkumarbibbab/covid-19-vaccination-process</a><br>\nSEE THIS NOTEBOOK🙄</p>",
      "rawMarkdown": "WOW! IT IS REALLY INTRESTING\nUPVOTED!\nhttps://www.kaggle.com/ashokkumarbibbab/covid-19-vaccination-process\nSEE THIS NOTEBOOK🙄",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1173147,
      "author_name": "vatsalmavani",
      "author_url": "",
      "post_date": "01/27/2021 17:35:41",
      "content": "<p>I used it without dropout and get good results. I didn't use dropout because somewhere I learned that GAP extract the features from the last ConvBlock so how can I drop features. What do you think about this?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1173555,
          "author_name": "pcjimmmy",
          "author_url": "",
          "post_date": "01/28/2021 00:36:55",
          "content": "<p>Dropout does not remove features - it just resets the weights for a random selection of the features.</p>\n<p>I always use dropout to help close the gap between training and validation accuracy.  But I do start with low number (0.05).  If I like the model and a large gap exists than I will increment up.  For the models I have created for this competition 0.5 has always been too large - the model never learns.  </p>\n<p>It does slow down the rate of learning which is why I use a very small number in the initial stages of looking at a new model.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1174574,
          "author_name": "junyingsg",
          "author_url": "",
          "post_date": "01/28/2021 15:34:35",
          "content": "<p>Actually dropout doesn't reset the weights, drop connect rate does. Dropout zeroes out connections between layers of neurons, but they essentially serve the same purpose. Just FYI.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1173271,
      "author_name": "nickuzmenkov",
      "author_url": "",
      "post_date": "01/27/2021 19:02:35",
      "content": "<p>For me <code>Dropout</code> doesn't make any perceptible CV changes, so I directly put just 5 <code>Dense</code> output layers.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1173433,
      "author_name": "morodertobias",
      "author_url": "",
      "post_date": "01/27/2021 21:45:59",
      "content": "<p>I always start with the 'head' architecture from the base network; the head looks a bit different from network to network. If some dropout is included and I see some overfitting I usually increase it. Don't know if this is the best practise though looking at my current scores…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1176004,
      "author_name": "vkehfdl1",
      "author_url": "",
      "post_date": "01/29/2021 12:56:57",
      "content": "<p>I use global average pooling, too. And Dropout(0.4). At last, of course, Dense layer with softmax. I think last part of layer is not that important… </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1179172,
      "author_name": "ashokkumarbibbab",
      "author_url": "",
      "post_date": "01/31/2021 11:30:28",
      "content": "<p>WOW! IT IS REALLY INTRESTING<br>\nUPVOTED!<br>\n<a href=\"https://www.kaggle.com/ashokkumarbibbab/covid-19-vaccination-process\" target=\"_blank\">https://www.kaggle.com/ashokkumarbibbab/covid-19-vaccination-process</a><br>\nSEE THIS NOTEBOOK🙄</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1173077": "I am trying to set up a network head after globalaveragepooling. It works best for me if I use Dropout(0.3) and that's it. What works for you?",
    "1173147": "I used it without dropout and get good results. I didn't use dropout because somewhere I learned that GAP extract the features from the last ConvBlock so how can I drop features. What do you think about this?",
    "1173271": "For me `Dropout` doesn't make any perceptible CV changes, so I directly put just 5 `Dense` output layers.",
    "1173433": "I always start with the 'head' architecture from the base network; the head looks a bit different from network to network. If some dropout is included and I see some overfitting I usually increase it. Don't know if this is the best practise though looking at my current scores...",
    "1173555": "Dropout does not remove features - it just resets the weights for a random selection of the features.\n\nI always use dropout to help close the gap between training and validation accuracy.  But I do start with low number (0.05).  If I like the model and a large gap exists than I will increment up.  For the models I have created for this competition 0.5 has always been too large - the model never learns.  \n\nIt does slow down the rate of learning which is why I use a very small number in the initial stages of looking at a new model.",
    "1174574": "Actually dropout doesn't reset the weights, drop connect rate does. Dropout zeroes out connections between layers of neurons, but they essentially serve the same purpose. Just FYI.",
    "1176004": "I use global average pooling, too. And Dropout(0.4). At last, of course, Dense layer with softmax. I think last part of layer is not that important...",
    "1179172": "WOW! IT IS REALLY INTRESTING\nUPVOTED!\nhttps://www.kaggle.com/ashokkumarbibbab/covid-19-vaccination-process\nSEE THIS NOTEBOOK🙄"
  },
  "source": "meta"
}