{
  "id": 102318,
  "title": "attention transfer learning structure",
  "url": "/competitions/aptos2019-blindness-detection/discussion/102318",
  "author_name": "",
  "post_date": "2019-08-01T10:40:41.272047900Z",
  "votes": -2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi there,</p>\n\n<p>Please do not complain about my basic questions here and ignore my \nignorance. This is my second CV competition. There left much to learn for me.</p>\n\n<p>My question: I learn about attention layer from <a href=\"https://github.com/xiaojiu1414/attention-transfer/blob/master/visualize-attention.ipynb\">here</a>. I cannot understand below code:</p>\n\n<p>```\nclass ResNet34AT(ResNet):</p>\n\n<pre><code>def forward(self, x):\n    x = self.conv1(x)\n    x = self.bn1(x)\n    x = self.relu(x)\n    x = self.maxpool(x)\n\n    g0 = self.layer1(x)\n    g1 = self.layer2(g0)\n    g2 = self.layer3(g1)\n    g3 = self.layer4(g2)\n\n    return [g.pow(2).mean(1) for g in (g0, g1, g2, g3)]\n</code></pre>\n\n<p>model = ResNet34AT(BasicBlock, [3, 4, 6, 3])\nmodel.load_state_dict(base_resnet34.state_dict())\n ```</p>\n\n<p>I want to know what is <code>[3, 4, 6, 3]</code> for?</p>",
  "messages": [
    {
      "id": "589761",
      "postDate": "08/01/2019 10:40:41",
      "content": "<p>Hi there,</p>\n\n<p>Please do not complain about my basic questions here and ignore my \nignorance. This is my second CV competition. There left much to learn for me.</p>\n\n<p>My question: I learn about attention layer from <a href=\"https://github.com/xiaojiu1414/attention-transfer/blob/master/visualize-attention.ipynb\">here</a>. I cannot understand below code:</p>\n\n<p>```\nclass ResNet34AT(ResNet):</p>\n\n<pre><code>def forward(self, x):\n    x = self.conv1(x)\n    x = self.bn1(x)\n    x = self.relu(x)\n    x = self.maxpool(x)\n\n    g0 = self.layer1(x)\n    g1 = self.layer2(g0)\n    g2 = self.layer3(g1)\n    g3 = self.layer4(g2)\n\n    return [g.pow(2).mean(1) for g in (g0, g1, g2, g3)]\n</code></pre>\n\n<p>model = ResNet34AT(BasicBlock, [3, 4, 6, 3])\nmodel.load_state_dict(base_resnet34.state_dict())\n ```</p>\n\n<p>I want to know what is <code>[3, 4, 6, 3]</code> for?</p>",
      "rawMarkdown": "Hi there,\n\nPlease do not complain about my basic questions here and ignore my \nignorance. This is my second CV competition. There left much to learn for me.\n\nMy question: I learn about attention layer from [here](https://github.com/xiaojiu1414/attention-transfer/blob/master/visualize-attention.ipynb). I cannot understand below code:\n\n ```\nclass ResNet34AT(ResNet):\n    \n    def forward(self, x):\n        x = self.conv1(x)\n        x = self.bn1(x)\n        x = self.relu(x)\n        x = self.maxpool(x)\n\n        g0 = self.layer1(x)\n        g1 = self.layer2(g0)\n        g2 = self.layer3(g1)\n        g3 = self.layer4(g2)\n        \n        return [g.pow(2).mean(1) for g in (g0, g1, g2, g3)]\n    \nmodel = ResNet34AT(BasicBlock, [3, 4, 6, 3])\nmodel.load_state_dict(base_resnet34.state_dict())\n ```\n\nI want to know what is `[3, 4, 6, 3]` for?",
      "votes": null
    },
    {
      "id": "589792",
      "postDate": "08/01/2019 11:34:54",
      "content": "<p>Your link points to a repository, can you point it to exact line of the file you are referring to? (go to the code file, click on the line number, copy the url from url bar.)</p>",
      "rawMarkdown": "Your link points to a repository, can you point it to exact line of the file you are referring to? (go to the code file, click on the line number, copy the url from url bar.)",
      "votes": null
    },
    {
      "id": "589868",
      "postDate": "08/01/2019 14:01:48",
      "content": "<p>I could see this line in the 3rd cell at -&gt; <a href=\"https://github.com/szagoruyko/attention-transfer/blob/master/visualize-attention.ipynb\">https://github.com/szagoruyko/attention-transfer/blob/master/visualize-attention.ipynb</a></p>",
      "rawMarkdown": "I could see this line in the 3rd cell at -&gt; https://github.com/szagoruyko/attention-transfer/blob/master/visualize-attention.ipynb",
      "votes": null
    },
    {
      "id": "590978",
      "postDate": "08/03/2019 00:44:01",
      "content": "<p>Sorry. Have changed the link.</p>",
      "rawMarkdown": "Sorry. Have changed the link.",
      "votes": null
    },
    {
      "id": "591136",
      "postDate": "08/03/2019 07:45:55",
      "content": "<p>So, <code>ResNet34AT</code> is inheriting <code>ResNet</code> which comes from <code>torchvision.models.resnet</code> module <a href=\"https://github.com/pytorch/vision/blob/4ec38d496db69833eb0a6f144ebbd6f751cd3912/torchvision/models/resnet.py#L118\">here</a> \nIf you take a closer look, <code>ResNet</code>'s second argument is <code>layers</code>, so this list <code>[3, 4, 6, 3]</code>  is passed as <code>layers</code> variable, now going deeper, the elements of this list are used <a href=\"https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L144\">here</a> in initializing <code>self.layer1</code>, <code>self.layer2</code> etc.. they call <code>self._make_layer</code> function, and are used <a href=\"https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L188\">here</a> it specifies the number of <code>block</code> to use in in each layer. Here <code>block</code> is nothing else than <code>BasicBlock</code> which you passed to <code>ResNet34AT</code>\nIn short, number of times times <code>BasicBlock</code> is to be repeated in each <code>layer</code> i.e., <code>self.layer</code>, <code>self.layer2</code> etc. </p>\n\n<p>here <code>layer</code> is not a single layer but a bunch of basic blocks put together. Hope this clears it up.</p>",
      "rawMarkdown": "So, `ResNet34AT` is inheriting `ResNet` which comes from `torchvision.models.resnet` module [here](https://github.com/pytorch/vision/blob/4ec38d496db69833eb0a6f144ebbd6f751cd3912/torchvision/models/resnet.py#L118) \nIf you take a closer look, `ResNet`'s second argument is `layers`, so this list `[3, 4, 6, 3]`  is passed as `layers` variable, now going deeper, the elements of this list are used [here](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L144) in initializing `self.layer1`, `self.layer2` etc.. they call `self._make_layer` function, and are used [here](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L188) it specifies the number of `block` to use in in each layer. Here `block` is nothing else than `BasicBlock` which you passed to `ResNet34AT`\nIn short, number of times times `BasicBlock` is to be repeated in each `layer` i.e., `self.layer`, `self.layer2` etc. \n\nhere `layer` is not a single layer but a bunch of basic blocks put together. Hope this clears it up.",
      "votes": null
    },
    {
      "id": "591606",
      "postDate": "08/04/2019 02:15:16",
      "content": "<p>Very clear. Thanks for your guidance!</p>",
      "rawMarkdown": "Very clear. Thanks for your guidance!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 589792,
      "author_name": "rishabhiitbhu",
      "author_url": "",
      "post_date": "08/01/2019 11:34:54",
      "content": "<p>Your link points to a repository, can you point it to exact line of the file you are referring to? (go to the code file, click on the line number, copy the url from url bar.)</p>",
      "votes": null,
      "replies": [
        {
          "id": 589868,
          "author_name": "dineshkumarilangovan",
          "author_url": "",
          "post_date": "08/01/2019 14:01:48",
          "content": "<p>I could see this line in the 3rd cell at -&gt; <a href=\"https://github.com/szagoruyko/attention-transfer/blob/master/visualize-attention.ipynb\">https://github.com/szagoruyko/attention-transfer/blob/master/visualize-attention.ipynb</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 590978,
          "author_name": "xiaojiu1414",
          "author_url": "",
          "post_date": "08/03/2019 00:44:01",
          "content": "<p>Sorry. Have changed the link.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 591136,
          "author_name": "rishabhiitbhu",
          "author_url": "",
          "post_date": "08/03/2019 07:45:55",
          "content": "<p>So, <code>ResNet34AT</code> is inheriting <code>ResNet</code> which comes from <code>torchvision.models.resnet</code> module <a href=\"https://github.com/pytorch/vision/blob/4ec38d496db69833eb0a6f144ebbd6f751cd3912/torchvision/models/resnet.py#L118\">here</a> \nIf you take a closer look, <code>ResNet</code>'s second argument is <code>layers</code>, so this list <code>[3, 4, 6, 3]</code>  is passed as <code>layers</code> variable, now going deeper, the elements of this list are used <a href=\"https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L144\">here</a> in initializing <code>self.layer1</code>, <code>self.layer2</code> etc.. they call <code>self._make_layer</code> function, and are used <a href=\"https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L188\">here</a> it specifies the number of <code>block</code> to use in in each layer. Here <code>block</code> is nothing else than <code>BasicBlock</code> which you passed to <code>ResNet34AT</code>\nIn short, number of times times <code>BasicBlock</code> is to be repeated in each <code>layer</code> i.e., <code>self.layer</code>, <code>self.layer2</code> etc. </p>\n\n<p>here <code>layer</code> is not a single layer but a bunch of basic blocks put together. Hope this clears it up.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 591606,
          "author_name": "xiaojiu1414",
          "author_url": "",
          "post_date": "08/04/2019 02:15:16",
          "content": "<p>Very clear. Thanks for your guidance!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "589761": "Hi there,\n\nPlease do not complain about my basic questions here and ignore my \nignorance. This is my second CV competition. There left much to learn for me.\n\nMy question: I learn about attention layer from [here](https://github.com/xiaojiu1414/attention-transfer/blob/master/visualize-attention.ipynb). I cannot understand below code:\n\n ```\nclass ResNet34AT(ResNet):\n    \n    def forward(self, x):\n        x = self.conv1(x)\n        x = self.bn1(x)\n        x = self.relu(x)\n        x = self.maxpool(x)\n\n        g0 = self.layer1(x)\n        g1 = self.layer2(g0)\n        g2 = self.layer3(g1)\n        g3 = self.layer4(g2)\n        \n        return [g.pow(2).mean(1) for g in (g0, g1, g2, g3)]\n    \nmodel = ResNet34AT(BasicBlock, [3, 4, 6, 3])\nmodel.load_state_dict(base_resnet34.state_dict())\n ```\n\nI want to know what is `[3, 4, 6, 3]` for?",
    "589792": "Your link points to a repository, can you point it to exact line of the file you are referring to? (go to the code file, click on the line number, copy the url from url bar.)",
    "589868": "I could see this line in the 3rd cell at -&gt; https://github.com/szagoruyko/attention-transfer/blob/master/visualize-attention.ipynb",
    "590978": "Sorry. Have changed the link.",
    "591136": "So, `ResNet34AT` is inheriting `ResNet` which comes from `torchvision.models.resnet` module [here](https://github.com/pytorch/vision/blob/4ec38d496db69833eb0a6f144ebbd6f751cd3912/torchvision/models/resnet.py#L118) \nIf you take a closer look, `ResNet`'s second argument is `layers`, so this list `[3, 4, 6, 3]`  is passed as `layers` variable, now going deeper, the elements of this list are used [here](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L144) in initializing `self.layer1`, `self.layer2` etc.. they call `self._make_layer` function, and are used [here](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L188) it specifies the number of `block` to use in in each layer. Here `block` is nothing else than `BasicBlock` which you passed to `ResNet34AT`\nIn short, number of times times `BasicBlock` is to be repeated in each `layer` i.e., `self.layer`, `self.layer2` etc. \n\nhere `layer` is not a single layer but a bunch of basic blocks put together. Hope this clears it up.",
    "591606": "Very clear. Thanks for your guidance!"
  },
  "source": "meta"
}