{
  "id": 231330,
  "title": "Confusion about masked attention in the MultiHeadAttention (pytorch).",
  "url": "/competitions/bms-molecular-translation/discussion/231330",
  "author_name": "",
  "post_date": "2021-04-08T02:59:12.111548300Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>So, I am sure that I am clearly doing something wrong here because my cv is 70 even with very large  transformer. </p>\n<p>My main confusion is how to adapt 3d mask for multihead scenario. At first I passed mask as follows</p>\n<ul>\n<li>create separate 2d mask for each element in the batch</li>\n<li>pad all masks to the largest mask</li>\n<li>stack all masks so that the shape is <code>(batch_size, longest_sequence, longest_sequence)</code></li>\n<li>multiply masks by the number of heads as is recommended in the torch docs -&gt; <code>torch.cat(n_head*[mask])</code></li>\n<li>create <code>key_padding_mask</code> for padding</li>\n</ul>\n<p>But then I realized that this approach might be wrong since I incorrectly repeating masks along head dimension, so I switched to:</p>\n<ul>\n<li>repeating created 2d mask <code>n_head</code> times first</li>\n<li>then stacking </li>\n</ul>\n<p>In both cases performance is pretty bad…</p>\n<p>So then I simply switched to one 2d mask with the largest sequence and additionally passed <code>key_padding_mask</code> because I created separate 3d mask in the first place so that model ignores padding. </p>\n<p>In 3rd case results seems to stabilize and improve but by the really small margin.</p>\n<p>So my question is, what is the correct way of masking in <code>MultiHeadAttention</code> to avoid attending to future words, avoid attending to padding, and handle sequences with different lengths. </p>",
  "messages": [
    {
      "id": "1266725",
      "postDate": "04/08/2021 02:59:12",
      "content": "<p>So, I am sure that I am clearly doing something wrong here because my cv is 70 even with very large  transformer. </p>\n<p>My main confusion is how to adapt 3d mask for multihead scenario. At first I passed mask as follows</p>\n<ul>\n<li>create separate 2d mask for each element in the batch</li>\n<li>pad all masks to the largest mask</li>\n<li>stack all masks so that the shape is <code>(batch_size, longest_sequence, longest_sequence)</code></li>\n<li>multiply masks by the number of heads as is recommended in the torch docs -&gt; <code>torch.cat(n_head*[mask])</code></li>\n<li>create <code>key_padding_mask</code> for padding</li>\n</ul>\n<p>But then I realized that this approach might be wrong since I incorrectly repeating masks along head dimension, so I switched to:</p>\n<ul>\n<li>repeating created 2d mask <code>n_head</code> times first</li>\n<li>then stacking </li>\n</ul>\n<p>In both cases performance is pretty bad…</p>\n<p>So then I simply switched to one 2d mask with the largest sequence and additionally passed <code>key_padding_mask</code> because I created separate 3d mask in the first place so that model ignores padding. </p>\n<p>In 3rd case results seems to stabilize and improve but by the really small margin.</p>\n<p>So my question is, what is the correct way of masking in <code>MultiHeadAttention</code> to avoid attending to future words, avoid attending to padding, and handle sequences with different lengths. </p>",
      "rawMarkdown": "So, I am sure that I am clearly doing something wrong here because my cv is 70 even with very large  transformer. \n\nMy main confusion is how to adapt 3d mask for multihead scenario. At first I passed mask as follows\n- create separate 2d mask for each element in the batch\n- pad all masks to the largest mask\n- stack all masks so that the shape is `(batch_size, longest_sequence, longest_sequence)`\n- multiply masks by the number of heads as is recommended in the torch docs -> `torch.cat(n_head*[mask])`\n- create `key_padding_mask` for padding\n\nBut then I realized that this approach might be wrong since I incorrectly repeating masks along head dimension, so I switched to:\n- repeating created 2d mask `n_head` times first\n- then stacking \n\nIn both cases performance is pretty bad...\n\nSo then I simply switched to one 2d mask with the largest sequence and additionally passed `key_padding_mask` because I created separate 3d mask in the first place so that model ignores padding. \n\nIn 3rd case results seems to stabilize and improve but by the really small margin.\n\nSo my question is, what is the correct way of masking in `MultiHeadAttention` to avoid attending to future words, avoid attending to padding, and handle sequences with different lengths.",
      "votes": null
    },
    {
      "id": "1266817",
      "postDate": "04/08/2021 05:15:38",
      "content": "<p>I like to bump this as well. I'm sure this question is also faced by many transformer amateur like myself.</p>\n<p>I have a couple questions for clarification:</p>\n<ul>\n<li>How much data are you using? Using more data helped me lower the CV when I was checking my implementation.</li>\n<li>Are you asking about attention mask for the (transformer) encoder or (transformer) decoder?</li>\n</ul>\n<p>I use the PyTorch implementation <a href=\"https://github.com/yahoo/object_relation_transformer/blob/master/models/TransformerModel.py\" target=\"_blank\">here</a> and <a href=\"https://github.com/wtliao/ImageTransformer/blob/master/models/TransformerModel1.py\" target=\"_blank\">here</a>. The masking implementation works for me.</p>",
      "rawMarkdown": "I like to bump this as well. I'm sure this question is also faced by many transformer amateur like myself.\n\nI have a couple questions for clarification:\n- How much data are you using? Using more data helped me lower the CV when I was checking my implementation.\n- Are you asking about attention mask for the (transformer) encoder or (transformer) decoder?\n\nI use the PyTorch implementation [here](https://github.com/yahoo/object_relation_transformer/blob/master/models/TransformerModel.py) and [here](https://github.com/wtliao/ImageTransformer/blob/master/models/TransformerModel1.py). The masking implementation works for me.",
      "votes": null
    },
    {
      "id": "1266866",
      "postDate": "04/08/2021 06:06:54",
      "content": "<p>I am masking only first MHA in the decoder part, at this point I am only experimenting with 480k images. </p>",
      "rawMarkdown": "I am masking only first MHA in the decoder part, at this point I am only experimenting with 480k images.",
      "votes": null
    },
    {
      "id": "1270578",
      "postDate": "04/11/2021 19:03:10",
      "content": "<p>I guess you are talking about Transformer-Decoder, where you have two kinds of attention. One for your sequence (InChI) and one for the context (encoded image).</p>\n<p>The mask for the sequence should be a triangular mask so that each token at position i can't see anything beyond i - meaning future tokens. For the context attention your shouldn't mask anything, so all your tokens in the sequence can see all the context vectors. The context vectors should still have some positional embedding thou. Note: One sequence in a batch stands for itself and should never see a data-point from another batch. The masks are obviously the same for all data-points (sequence + context) in a batch.</p>\n<p>My own code for PyTorch triangular attention mask (it is 2D, but should be adapted in the normal transformer code to 3D -&gt; the batch dimension):</p>\n<p>`def _generate_square_subsequent_mask(size, x):<br>\n        \"\"\"<br>\n        Generates a left-to-right square attention mask</p>\n<pre><code>    Masked positions are filled with float('-inf'), else 0.0\n    The lower left triangle of the mask with the main diagonal\n    is filled with 0.0\n\n    Parameters\n    ----------\n    size: int\n        The sequence length to be masked, which results in a\n        shape == (size, size) mask\n    x: torch.Tensor\n        Tensor which we need the attention mask for\n    \"\"\"\n    mask = torch.triu(torch.ones(size, size, device=x.device), diagonal=1)\n    mask = mask.to(dtype=x.dtype).masked_fill(mask == 1, float('-inf'))\n    return mask`\n</code></pre>\n<p>Can't get all the code in the box, but still readable.</p>\n<p>The Transformer Encoder, if just taking encoded image as input and further encoding it shouldn't have a triangular mask as well,  because here every image feature vector can see everything else with no problem. So just like in the decoder context attention part.<br>\nIn Pytorch itself context is often called src (source) and the decoder sequence part tgt (target)</p>",
      "rawMarkdown": "I guess you are talking about Transformer-Decoder, where you have two kinds of attention. One for your sequence (InChI) and one for the context (encoded image).\n\nThe mask for the sequence should be a triangular mask so that each token at position i can't see anything beyond i - meaning future tokens. For the context attention your shouldn't mask anything, so all your tokens in the sequence can see all the context vectors. The context vectors should still have some positional embedding thou. Note: One sequence in a batch stands for itself and should never see a data-point from another batch. The masks are obviously the same for all data-points (sequence + context) in a batch.\n\nMy own code for PyTorch triangular attention mask (it is 2D, but should be adapted in the normal transformer code to 3D -> the batch dimension):\n\n`def _generate_square_subsequent_mask(size, x):\n        \"\"\"\n        Generates a left-to-right square attention mask\n\n        Masked positions are filled with float('-inf'), else 0.0\n        The lower left triangle of the mask with the main diagonal\n        is filled with 0.0\n\n        Parameters\n        ----------\n        size: int\n            The sequence length to be masked, which results in a\n            shape == (size, size) mask\n        x: torch.Tensor\n            Tensor which we need the attention mask for\n        \"\"\"\n        mask = torch.triu(torch.ones(size, size, device=x.device), diagonal=1)\n        mask = mask.to(dtype=x.dtype).masked_fill(mask == 1, float('-inf'))\n        return mask`\n\nCan't get all the code in the box, but still readable.\n\nThe Transformer Encoder, if just taking encoded image as input and further encoding it shouldn't have a triangular mask as well,  because here every image feature vector can see everything else with no problem. So just like in the decoder context attention part.\nIn Pytorch itself context is often called src (source) and the decoder sequence part tgt (target)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1266817,
      "author_name": "jy2tong",
      "author_url": "",
      "post_date": "04/08/2021 05:15:38",
      "content": "<p>I like to bump this as well. I'm sure this question is also faced by many transformer amateur like myself.</p>\n<p>I have a couple questions for clarification:</p>\n<ul>\n<li>How much data are you using? Using more data helped me lower the CV when I was checking my implementation.</li>\n<li>Are you asking about attention mask for the (transformer) encoder or (transformer) decoder?</li>\n</ul>\n<p>I use the PyTorch implementation <a href=\"https://github.com/yahoo/object_relation_transformer/blob/master/models/TransformerModel.py\" target=\"_blank\">here</a> and <a href=\"https://github.com/wtliao/ImageTransformer/blob/master/models/TransformerModel1.py\" target=\"_blank\">here</a>. The masking implementation works for me.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1266866,
          "author_name": "skull8888888",
          "author_url": "",
          "post_date": "04/08/2021 06:06:54",
          "content": "<p>I am masking only first MHA in the decoder part, at this point I am only experimenting with 480k images. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1270578,
      "author_name": "cepheidq",
      "author_url": "",
      "post_date": "04/11/2021 19:03:10",
      "content": "<p>I guess you are talking about Transformer-Decoder, where you have two kinds of attention. One for your sequence (InChI) and one for the context (encoded image).</p>\n<p>The mask for the sequence should be a triangular mask so that each token at position i can't see anything beyond i - meaning future tokens. For the context attention your shouldn't mask anything, so all your tokens in the sequence can see all the context vectors. The context vectors should still have some positional embedding thou. Note: One sequence in a batch stands for itself and should never see a data-point from another batch. The masks are obviously the same for all data-points (sequence + context) in a batch.</p>\n<p>My own code for PyTorch triangular attention mask (it is 2D, but should be adapted in the normal transformer code to 3D -&gt; the batch dimension):</p>\n<p>`def _generate_square_subsequent_mask(size, x):<br>\n        \"\"\"<br>\n        Generates a left-to-right square attention mask</p>\n<pre><code>    Masked positions are filled with float('-inf'), else 0.0\n    The lower left triangle of the mask with the main diagonal\n    is filled with 0.0\n\n    Parameters\n    ----------\n    size: int\n        The sequence length to be masked, which results in a\n        shape == (size, size) mask\n    x: torch.Tensor\n        Tensor which we need the attention mask for\n    \"\"\"\n    mask = torch.triu(torch.ones(size, size, device=x.device), diagonal=1)\n    mask = mask.to(dtype=x.dtype).masked_fill(mask == 1, float('-inf'))\n    return mask`\n</code></pre>\n<p>Can't get all the code in the box, but still readable.</p>\n<p>The Transformer Encoder, if just taking encoded image as input and further encoding it shouldn't have a triangular mask as well,  because here every image feature vector can see everything else with no problem. So just like in the decoder context attention part.<br>\nIn Pytorch itself context is often called src (source) and the decoder sequence part tgt (target)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1266725": "So, I am sure that I am clearly doing something wrong here because my cv is 70 even with very large  transformer. \n\nMy main confusion is how to adapt 3d mask for multihead scenario. At first I passed mask as follows\n- create separate 2d mask for each element in the batch\n- pad all masks to the largest mask\n- stack all masks so that the shape is `(batch_size, longest_sequence, longest_sequence)`\n- multiply masks by the number of heads as is recommended in the torch docs -> `torch.cat(n_head*[mask])`\n- create `key_padding_mask` for padding\n\nBut then I realized that this approach might be wrong since I incorrectly repeating masks along head dimension, so I switched to:\n- repeating created 2d mask `n_head` times first\n- then stacking \n\nIn both cases performance is pretty bad...\n\nSo then I simply switched to one 2d mask with the largest sequence and additionally passed `key_padding_mask` because I created separate 3d mask in the first place so that model ignores padding. \n\nIn 3rd case results seems to stabilize and improve but by the really small margin.\n\nSo my question is, what is the correct way of masking in `MultiHeadAttention` to avoid attending to future words, avoid attending to padding, and handle sequences with different lengths.",
    "1266817": "I like to bump this as well. I'm sure this question is also faced by many transformer amateur like myself.\n\nI have a couple questions for clarification:\n- How much data are you using? Using more data helped me lower the CV when I was checking my implementation.\n- Are you asking about attention mask for the (transformer) encoder or (transformer) decoder?\n\nI use the PyTorch implementation [here](https://github.com/yahoo/object_relation_transformer/blob/master/models/TransformerModel.py) and [here](https://github.com/wtliao/ImageTransformer/blob/master/models/TransformerModel1.py). The masking implementation works for me.",
    "1266866": "I am masking only first MHA in the decoder part, at this point I am only experimenting with 480k images.",
    "1270578": "I guess you are talking about Transformer-Decoder, where you have two kinds of attention. One for your sequence (InChI) and one for the context (encoded image).\n\nThe mask for the sequence should be a triangular mask so that each token at position i can't see anything beyond i - meaning future tokens. For the context attention your shouldn't mask anything, so all your tokens in the sequence can see all the context vectors. The context vectors should still have some positional embedding thou. Note: One sequence in a batch stands for itself and should never see a data-point from another batch. The masks are obviously the same for all data-points (sequence + context) in a batch.\n\nMy own code for PyTorch triangular attention mask (it is 2D, but should be adapted in the normal transformer code to 3D -> the batch dimension):\n\n`def _generate_square_subsequent_mask(size, x):\n        \"\"\"\n        Generates a left-to-right square attention mask\n\n        Masked positions are filled with float('-inf'), else 0.0\n        The lower left triangle of the mask with the main diagonal\n        is filled with 0.0\n\n        Parameters\n        ----------\n        size: int\n            The sequence length to be masked, which results in a\n            shape == (size, size) mask\n        x: torch.Tensor\n            Tensor which we need the attention mask for\n        \"\"\"\n        mask = torch.triu(torch.ones(size, size, device=x.device), diagonal=1)\n        mask = mask.to(dtype=x.dtype).masked_fill(mask == 1, float('-inf'))\n        return mask`\n\nCan't get all the code in the box, but still readable.\n\nThe Transformer Encoder, if just taking encoded image as input and further encoding it shouldn't have a triangular mask as well,  because here every image feature vector can see everything else with no problem. So just like in the decoder context attention part.\nIn Pytorch itself context is often called src (source) and the decoder sequence part tgt (target)"
  },
  "source": "meta"
}