{ "id": 205208, "title": "Multi-Head Approach", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/205208", "author_name": "", "post_date": "2020-12-19T02:33:59.379211700Z", "votes": 44, "comment_count": 15, "views": 0, "content": "

We have 11 targets in this competition and they can be divided into 4 groups: ETT, NGT, CVC, and Swan.
\nMaybe, different groups have different areas in images to focus on.

\n

One possible way to leverage this idea is a multi-head approach. Groups share CNN backbone but have independent classifier heads.
\n
\n

\n

I haven't tried it yet, but I'm thinking it might work effectively.

", "messages": [ { "id": "1118369", "postDate": "12/19/2020 02:33:59", "content": "

We have 11 targets in this competition and they can be divided into 4 groups: ETT, NGT, CVC, and Swan.
\nMaybe, different groups have different areas in images to focus on.

\n

One possible way to leverage this idea is a multi-head approach. Groups share CNN backbone but have independent classifier heads.
\n
\n

\n

I haven't tried it yet, but I'm thinking it might work effectively.

", "rawMarkdown": "We have 11 targets in this competition and they can be divided into 4 groups: `ETT`, `NGT`, `CVC`, and `Swan`. \nMaybe, different groups have different areas in images to focus on.\n\nOne possible way to leverage this idea is a multi-head approach. Groups share CNN backbone but have independent classifier heads. \n\n\n\nI haven't tried it yet, but I'm thinking it might work effectively.", "votes": null }, { "id": "1118385", "postDate": "12/19/2020 02:54:00", "content": "

With the Functional API of Keras i see myself doing something like this, but with PyTorch i not able to \"see\" how to code something like that. So. i'm very curious to see how are you going to do that.

", "rawMarkdown": "With the **Functional API** of Keras i see myself doing something like this, but with PyTorch i not able to \"see\" how to code something like that. So. i'm very curious to see how are you going to do that.", "votes": null }, { "id": "1118392", "postDate": "12/19/2020 03:07:35", "content": "

I didn't know there was such a method. I found the similar method in RSNA 2020 solutions.
\nhttps://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193505

\n

I am a newbie of deep learning, so I'm sorry if I misunderstood what the method is.
\nAnyway, I will try it as soon as possible!

", "rawMarkdown": "I didn't know there was such a method. I found the similar method in RSNA 2020 solutions.\nhttps://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193505\n\nI am a newbie of deep learning, so I'm sorry if I misunderstood what the method is. \nAnyway, I will try it as soon as possible!", "votes": null }, { "id": "1118403", "postDate": "12/19/2020 03:32:15", "content": "

I have tried Multi-Head approach on first day but no improvement for old metric, I will try it again for new metric.

", "rawMarkdown": "I have tried Multi-Head approach on first day but no improvement for old metric, I will try it again for new metric.", "votes": null }, { "id": "1118432", "postDate": "12/19/2020 04:19:12", "content": "

A simple way is replacing the fc layer of pretrained models (e.g. models of torchvision).

\n

For example:

\n
import torch\nfrom torch import nn\nfrom torchvision import models\n\nclass CustomMultiHead(nn.Module):\n\n    def __init__(self, out_dims=[3, 4, 3, 1], in_dim=2048, mid_dim=1024):\n        super(CustomMultiHead, self).__init__()\n\n        self.n_heads = len(out_dims)\n        for i in range(self.n_heads):\n            layer = nn.Sequential(\n                nn.Linear(in_dim, mid_dim),\n                nn.ReLU(inplace=True), nn.Dropout(0.5),\n                nn.Linear(mid_dim, out_dims[i]))\n            setattr(self, f\"head{i}\", layer)\n\n    def forward(self, x):\n        out_list = [getattr(self, f\"head{i}\")(x) for i in range(self.n_heads)]\n        for h in out_list:\n            print(h.shape)\n        out = torch.cat(out_list, 1)\n        return out\n\n\n# # test: replace fc with custom head\nm = models.resnet50(pretrained=True)\ndel m.fc\nm.fc = CustomMultiHead()\n
", "rawMarkdown": "A simple way is replacing the `fc` layer of pretrained models (e.g. models of torchvision).\n\nFor example:\n\n```\nimport torch\nfrom torch import nn\nfrom torchvision import models\n\nclass CustomMultiHead(nn.Module):\n\n def __init__(self, out_dims=[3, 4, 3, 1], in_dim=2048, mid_dim=1024):\n super(CustomMultiHead, self).__init__()\n\n self.n_heads = len(out_dims)\n for i in range(self.n_heads):\n layer = nn.Sequential(\n nn.Linear(in_dim, mid_dim),\n nn.ReLU(inplace=True), nn.Dropout(0.5),\n nn.Linear(mid_dim, out_dims[i]))\n setattr(self, f\"head{i}\", layer)\n\n def forward(self, x):\n out_list = [getattr(self, f\"head{i}\")(x) for i in range(self.n_heads)]\n for h in out_list:\n print(h.shape)\n out = torch.cat(out_list, 1)\n return out\n\n\n# # test: replace fc with custom head\nm = models.resnet50(pretrained=True)\ndel m.fc\nm.fc = CustomMultiHead()\n```", "votes": null }, { "id": "1118434", "postDate": "12/19/2020 04:21:43", "content": "

Thank you for information. That is a little surprising to me. I'll try later.

", "rawMarkdown": "Thank you for information. That is a little surprising to me. I'll try later.", "votes": null }, { "id": "1118523", "postDate": "12/19/2020 06:40:13", "content": "

it actually will not affect results because

\n", "rawMarkdown": "it actually will not affect results because \n- you are using BCE loss\n- if you do the maths, the back gradient propagation is exactly the same in values in both case\n(i.e. you end up learning the same parameters)", "votes": null }, { "id": "1118537", "postDate": "12/19/2020 06:57:27", "content": "

Yes, you are right if we use 1 linear layer as heads.

\n

I think there is still room for consideration on a model's branch point and type of heads.

", "rawMarkdown": "Yes, you are right if we use 1 linear layer as heads.\n\nI think there is still room for consideration on a model's branch point and type of heads.", "votes": null }, { "id": "1118539", "postDate": "12/19/2020 07:00:38", "content": "

\" if we use 1 linear layer as heads.\"
\noh, I and sorry that I missed that out.

\n

you are correct. if the heads are non-linear then the dependency of each group becomes decoupled.
\nThis does indeed reduce overfitting in some cases. There could be an improvement.

\n

I am excited to see the experimental results.

\n

Thanks!

", "rawMarkdown": "\" if we use 1 linear layer as heads.\"\noh, I and sorry that I missed that out. \n\nyou are correct. if the heads are non-linear then the dependency of each group becomes decoupled. \nThis does indeed reduce overfitting in some cases. There could be an improvement. \n\nI am excited to see the experimental results. \n\nThanks!", "votes": null }, { "id": "1118541", "postDate": "12/19/2020 07:03:38", "content": "

I guess heads can be used in another fully connected layer for stacking?

", "rawMarkdown": "I guess heads can be used in another fully connected layer for stacking?", "votes": null }, { "id": "1118545", "postDate": "12/19/2020 07:09:18", "content": "

i am thinking of a transformer like attention head. let the transformer to decide connections (i.e. dependency among groups) instead. But I am not sure if we have enough train samples for that

", "rawMarkdown": "i am thinking of a transformer like attention head. let the transformer to decide connections (i.e. dependency among groups) instead. But I am not sure if we have enough train samples for that", "votes": null }, { "id": "1118573", "postDate": "12/19/2020 07:51:57", "content": "

I tried it again, result is same, no improvement for me. How about you?

", "rawMarkdown": "I tried it again, result is same, no improvement for me. How about you?", "votes": null }, { "id": "1119028", "postDate": "12/19/2020 16:39:41", "content": "

@ttahara, in my case, CV was 0.006 higher than that of the original model. Many thanks!

", "rawMarkdown": "ttahara, in my case, CV was 0.006 higher than that of the original model. Many thanks!", "votes": null }, { "id": "1119338", "postDate": "12/20/2020 00:11:44", "content": "

That is the good news😃 Thanks for sharing your result.

\n

BTW, @yasufuminakama says it did not work for him. I'm interested in the difference between two.

", "rawMarkdown": "That is the good news😃 Thanks for sharing your result.\n\nBTW, @yasufuminakama says it did not work for him. I'm interested in the difference between two.", "votes": null }, { "id": "1119351", "postDate": "12/20/2020 01:25:12", "content": "

Unfortunately, in public LB, the score was not improved.
\nMy CV might not be precise. However, the blend with the original one got slightly better (0.003 higher than the original in LB).

", "rawMarkdown": "Unfortunately, in public LB, the score was not improved.\nMy CV might not be precise. However, the blend with the original one got slightly better (0.003 higher than the original in LB).", "votes": null }, { "id": "1130252", "postDate": "12/28/2020 21:53:19", "content": "

@yasufuminakama
\nI'm sorry to keep you waiting so long. I've published a topic about this approach.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207230

\n

So far, my approach has not gotten good score. I will continue to improve it ;)

", "rawMarkdown": "yasufuminakama \nI'm sorry to keep you waiting so long. I've published a topic about this approach.\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207230\n\nSo far, my approach has not gotten good score. I will continue to improve it ;)", "votes": null } ], "comments": [ { "id": 1118385, "author_name": "hiramcho", "author_url": "", "post_date": "12/19/2020 02:54:00", "content": "

With the Functional API of Keras i see myself doing something like this, but with PyTorch i not able to \"see\" how to code something like that. So. i'm very curious to see how are you going to do that.

", "votes": null, "replies": [ { "id": 1118432, "author_name": "ttahara", "author_url": "", "post_date": "12/19/2020 04:19:12", "content": "

A simple way is replacing the fc layer of pretrained models (e.g. models of torchvision).

\n

For example:

\n
import torch\nfrom torch import nn\nfrom torchvision import models\n\nclass CustomMultiHead(nn.Module):\n\n    def __init__(self, out_dims=[3, 4, 3, 1], in_dim=2048, mid_dim=1024):\n        super(CustomMultiHead, self).__init__()\n\n        self.n_heads = len(out_dims)\n        for i in range(self.n_heads):\n            layer = nn.Sequential(\n                nn.Linear(in_dim, mid_dim),\n                nn.ReLU(inplace=True), nn.Dropout(0.5),\n                nn.Linear(mid_dim, out_dims[i]))\n            setattr(self, f\"head{i}\", layer)\n\n    def forward(self, x):\n        out_list = [getattr(self, f\"head{i}\")(x) for i in range(self.n_heads)]\n        for h in out_list:\n            print(h.shape)\n        out = torch.cat(out_list, 1)\n        return out\n\n\n# # test: replace fc with custom head\nm = models.resnet50(pretrained=True)\ndel m.fc\nm.fc = CustomMultiHead()\n
", "votes": null, "replies": [] } ] }, { "id": 1118392, "author_name": "yosukeyama", "author_url": "", "post_date": "12/19/2020 03:07:35", "content": "

I didn't know there was such a method. I found the similar method in RSNA 2020 solutions.
\nhttps://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193505

\n

I am a newbie of deep learning, so I'm sorry if I misunderstood what the method is.
\nAnyway, I will try it as soon as possible!

", "votes": null, "replies": [ { "id": 1119028, "author_name": "yosukeyama", "author_url": "", "post_date": "12/19/2020 16:39:41", "content": "

@ttahara, in my case, CV was 0.006 higher than that of the original model. Many thanks!

", "votes": null, "replies": [] }, { "id": 1119338, "author_name": "ttahara", "author_url": "", "post_date": "12/20/2020 00:11:44", "content": "

That is the good news😃 Thanks for sharing your result.

\n

BTW, @yasufuminakama says it did not work for him. I'm interested in the difference between two.

", "votes": null, "replies": [] }, { "id": 1119351, "author_name": "yosukeyama", "author_url": "", "post_date": "12/20/2020 01:25:12", "content": "

Unfortunately, in public LB, the score was not improved.
\nMy CV might not be precise. However, the blend with the original one got slightly better (0.003 higher than the original in LB).

", "votes": null, "replies": [] } ] }, { "id": 1118403, "author_name": "yasufuminakama", "author_url": "", "post_date": "12/19/2020 03:32:15", "content": "

I have tried Multi-Head approach on first day but no improvement for old metric, I will try it again for new metric.

", "votes": null, "replies": [ { "id": 1118434, "author_name": "ttahara", "author_url": "", "post_date": "12/19/2020 04:21:43", "content": "

Thank you for information. That is a little surprising to me. I'll try later.

", "votes": null, "replies": [] }, { "id": 1118573, "author_name": "yasufuminakama", "author_url": "", "post_date": "12/19/2020 07:51:57", "content": "

I tried it again, result is same, no improvement for me. How about you?

", "votes": null, "replies": [] }, { "id": 1130252, "author_name": "ttahara", "author_url": "", "post_date": "12/28/2020 21:53:19", "content": "

@yasufuminakama
\nI'm sorry to keep you waiting so long. I've published a topic about this approach.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207230

\n

So far, my approach has not gotten good score. I will continue to improve it ;)

", "votes": null, "replies": [] } ] }, { "id": 1118523, "author_name": "hengck23", "author_url": "", "post_date": "12/19/2020 06:40:13", "content": "

it actually will not affect results because

\n", "votes": null, "replies": [ { "id": 1118537, "author_name": "ttahara", "author_url": "", "post_date": "12/19/2020 06:57:27", "content": "

Yes, you are right if we use 1 linear layer as heads.

\n

I think there is still room for consideration on a model's branch point and type of heads.

", "votes": null, "replies": [] }, { "id": 1118539, "author_name": "hengck23", "author_url": "", "post_date": "12/19/2020 07:00:38", "content": "

\" if we use 1 linear layer as heads.\"
\noh, I and sorry that I missed that out.

\n

you are correct. if the heads are non-linear then the dependency of each group becomes decoupled.
\nThis does indeed reduce overfitting in some cases. There could be an improvement.

\n

I am excited to see the experimental results.

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 1118541, "author_name": "gunesevitan", "author_url": "", "post_date": "12/19/2020 07:03:38", "content": "

I guess heads can be used in another fully connected layer for stacking?

", "votes": null, "replies": [] }, { "id": 1118545, "author_name": "hengck23", "author_url": "", "post_date": "12/19/2020 07:09:18", "content": "

i am thinking of a transformer like attention head. let the transformer to decide connections (i.e. dependency among groups) instead. But I am not sure if we have enough train samples for that

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1118369": "We have 11 targets in this competition and they can be divided into 4 groups: `ETT`, `NGT`, `CVC`, and `Swan`. \nMaybe, different groups have different areas in images to focus on.\n\nOne possible way to leverage this idea is a multi-head approach. Groups share CNN backbone but have independent classifier heads. \n\n\n\nI haven't tried it yet, but I'm thinking it might work effectively.", "1118385": "With the **Functional API** of Keras i see myself doing something like this, but with PyTorch i not able to \"see\" how to code something like that. So. i'm very curious to see how are you going to do that.", "1118392": "I didn't know there was such a method. I found the similar method in RSNA 2020 solutions.\nhttps://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193505\n\nI am a newbie of deep learning, so I'm sorry if I misunderstood what the method is. \nAnyway, I will try it as soon as possible!", "1118403": "I have tried Multi-Head approach on first day but no improvement for old metric, I will try it again for new metric.", "1118432": "A simple way is replacing the `fc` layer of pretrained models (e.g. models of torchvision).\n\nFor example:\n\n```\nimport torch\nfrom torch import nn\nfrom torchvision import models\n\nclass CustomMultiHead(nn.Module):\n\n def __init__(self, out_dims=[3, 4, 3, 1], in_dim=2048, mid_dim=1024):\n super(CustomMultiHead, self).__init__()\n\n self.n_heads = len(out_dims)\n for i in range(self.n_heads):\n layer = nn.Sequential(\n nn.Linear(in_dim, mid_dim),\n nn.ReLU(inplace=True), nn.Dropout(0.5),\n nn.Linear(mid_dim, out_dims[i]))\n setattr(self, f\"head{i}\", layer)\n\n def forward(self, x):\n out_list = [getattr(self, f\"head{i}\")(x) for i in range(self.n_heads)]\n for h in out_list:\n print(h.shape)\n out = torch.cat(out_list, 1)\n return out\n\n\n# # test: replace fc with custom head\nm = models.resnet50(pretrained=True)\ndel m.fc\nm.fc = CustomMultiHead()\n```", "1118434": "Thank you for information. That is a little surprising to me. I'll try later.", "1118523": "it actually will not affect results because \n- you are using BCE loss\n- if you do the maths, the back gradient propagation is exactly the same in values in both case\n(i.e. you end up learning the same parameters)", "1118537": "Yes, you are right if we use 1 linear layer as heads.\n\nI think there is still room for consideration on a model's branch point and type of heads.", "1118539": "\" if we use 1 linear layer as heads.\"\noh, I and sorry that I missed that out. \n\nyou are correct. if the heads are non-linear then the dependency of each group becomes decoupled. \nThis does indeed reduce overfitting in some cases. There could be an improvement. \n\nI am excited to see the experimental results. \n\nThanks!", "1118541": "I guess heads can be used in another fully connected layer for stacking?", "1118545": "i am thinking of a transformer like attention head. let the transformer to decide connections (i.e. dependency among groups) instead. But I am not sure if we have enough train samples for that", "1118573": "I tried it again, result is same, no improvement for me. How about you?", "1119028": "ttahara, in my case, CV was 0.006 higher than that of the original model. Many thanks!", "1119338": "That is the good news😃 Thanks for sharing your result.\n\nBTW, @yasufuminakama says it did not work for him. I'm interested in the difference between two.", "1119351": "Unfortunately, in public LB, the score was not improved.\nMy CV might not be precise. However, the blend with the original one got slightly better (0.003 higher than the original in LB).", "1130252": "yasufuminakama \nI'm sorry to keep you waiting so long. I've published a topic about this approach.\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207230\n\nSo far, my approach has not gotten good score. I will continue to improve it ;)" }, "source": "meta" }