{ "id": 79911, "title": "Common pitfalls of public kernels", "url": "/competitions/quora-insincere-questions-classification/discussion/79911", "author_name": "Dmytro Danevskyi", "post_date": "2019-02-08T14:53:42.327000", "votes": 71, "comment_count": 39, "views": 0, "content": "

Now, when this crazy competition is (almost) over, it's a perfect time to rest and do some analysis :)

\n\n

In this topic, I want to spot some common mistakes made in public kernels created during this competition. Some of these mistakes were repeated so many times that they almost stopped to look queerly :)

\n\n

Note that I created this topic for educational purposes only, not to make kernels authors look weird or stupid. We all make occasional mistakes from time to time, especially when we don't have much time to do the proper code check.

\n\n

1. Position-dependent bias

\n\n

Many pytorch kernels use Attention class that has the following line:

\n\n

self.b = nn.Parameter(torch.zeros(step_dim))

\n\n

step_dim is chosen to be 70, which is the length to which all questions are either padded or truncated.

\n\n

Thus, the bias term has the individual component for each sequence step. This is weird, at least because different questions have a different number of tokens. What if you need to process a question which is longer than 70 tokens? It's clear that a model should not be constrained to work only for questions of some particular length. Another argument is that each bias component is \"tied\" to a specific token position in a sentence, but an absolute position rarely has something to do with the token meaning or any other linguistic property.

\n\n

A more reasonable way to use a bias term is to have a single scalar bias that is shared among all sequence elements.

\n\n

2. \"Spatial\" dropout

\n\n

The same pytorch code has another suspicious component: the use of 2d dropout. It's used since pytorch doesn't provide a proper implementation of 1d dropout (that zeros out whole channels of a sequence of 1d vectors). So one option is to use a 2d version on unsqueezed embeddings. However, the implementation used in public kernels is completely wrong :)

\n\n

2d dropout layer is initialized as follows:

\n\n

self.embedding_dropout = nn.Dropout2d(0.1)

\n\n

and then used like this:

\n\n

h = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0)))

\n\n

Let's see what is going on here. First, you have a batch of embeddings, say it has a shape of (N, T, K). N is the number of elements in a batch, T is the sequence length and K is the feature (embedding) dimension. If we apply unsqueeze() operation on the 0 axis now, the tensor now has a shape of (1, N, T, K). 2D dropout zeros random channels (which is the second axis in pytorch). Oh, wait... The second axis is the batch dimension! In this case, 2D dropout will zero several batch elements, not channels.

\n\n

A proper way to apply 1d dropout, in this case, is to reshape the tensor into (N, K, 1, T) tensor, apply the 2d dropout and then make it have (N, T, K) shape again.

\n\n

Please contribute if you also noticed any wrong use of common operations!

\n\n

Good luck and happy kaggling! :0

", "messages": [ { "id": 468241, "postDate": "2019-02-08T14:53:42.327Z", "content": "

Now, when this crazy competition is (almost) over, it's a perfect time to rest and do some analysis :)

\n\n

In this topic, I want to spot some common mistakes made in public kernels created during this competition. Some of these mistakes were repeated so many times that they almost stopped to look queerly :)

\n\n

Note that I created this topic for educational purposes only, not to make kernels authors look weird or stupid. We all make occasional mistakes from time to time, especially when we don't have much time to do the proper code check.

\n\n

1. Position-dependent bias

\n\n

Many pytorch kernels use Attention class that has the following line:

\n\n

self.b = nn.Parameter(torch.zeros(step_dim))

\n\n

step_dim is chosen to be 70, which is the length to which all questions are either padded or truncated.

\n\n

Thus, the bias term has the individual component for each sequence step. This is weird, at least because different questions have a different number of tokens. What if you need to process a question which is longer than 70 tokens? It's clear that a model should not be constrained to work only for questions of some particular length. Another argument is that each bias component is \"tied\" to a specific token position in a sentence, but an absolute position rarely has something to do with the token meaning or any other linguistic property.

\n\n

A more reasonable way to use a bias term is to have a single scalar bias that is shared among all sequence elements.

\n\n

2. \"Spatial\" dropout

\n\n

The same pytorch code has another suspicious component: the use of 2d dropout. It's used since pytorch doesn't provide a proper implementation of 1d dropout (that zeros out whole channels of a sequence of 1d vectors). So one option is to use a 2d version on unsqueezed embeddings. However, the implementation used in public kernels is completely wrong :)

\n\n

2d dropout layer is initialized as follows:

\n\n

self.embedding_dropout = nn.Dropout2d(0.1)

\n\n

and then used like this:

\n\n

h = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0)))

\n\n

Let's see what is going on here. First, you have a batch of embeddings, say it has a shape of (N, T, K). N is the number of elements in a batch, T is the sequence length and K is the feature (embedding) dimension. If we apply unsqueeze() operation on the 0 axis now, the tensor now has a shape of (1, N, T, K). 2D dropout zeros random channels (which is the second axis in pytorch). Oh, wait... The second axis is the batch dimension! In this case, 2D dropout will zero several batch elements, not channels.

\n\n

A proper way to apply 1d dropout, in this case, is to reshape the tensor into (N, K, 1, T) tensor, apply the 2d dropout and then make it have (N, T, K) shape again.

\n\n

Please contribute if you also noticed any wrong use of common operations!

\n\n

Good luck and happy kaggling! :0

", "rawMarkdown": "Now, when this crazy competition is (almost) over, it's a perfect time to rest and do some analysis :)\n\nIn this topic, I want to spot some common mistakes made in public kernels created during this competition. Some of these mistakes were repeated so many times that they almost stopped to look queerly :)\n\nNote that I created this topic for educational purposes only, not to make kernels authors look weird or stupid. We all make occasional mistakes from time to time, especially when we don't have much time to do the proper code check.\n\n**1. Position-dependent bias**\n\nMany pytorch kernels use `Attention` class that has the following line:\n\n`self.b = nn.Parameter(torch.zeros(step_dim))`\n\n`step_dim` is chosen to be 70, which is the length to which all questions are either padded or truncated.\n\nThus, the bias term has the individual component for each sequence step. This is weird, at least because different questions have a different number of tokens. What if you need to process a question which is longer than 70 tokens? It's clear that a model should not be constrained to work only for questions of some particular length. Another argument is that each bias component is \"tied\" to a specific token position in a sentence, but an absolute position rarely has something to do with the token meaning or any other linguistic property.\n\nA more reasonable way to use a bias term is to have a single scalar bias that is shared among all sequence elements. \n\n**2. \"Spatial\" dropout**\n\nThe same pytorch code has another suspicious component: the use of 2d dropout. It's used since pytorch doesn't provide a proper implementation of 1d dropout (that zeros out whole channels of a sequence of 1d vectors). So one option is to use a 2d version on unsqueezed embeddings. However, the implementation used in public kernels is completely wrong :)\n\n2d dropout layer is initialized as follows:\n\n`self.embedding_dropout = nn.Dropout2d(0.1)`\n\nand then used like this:\n\n`h = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0)))`\n\nLet's see what is going on here. First, you have a batch of embeddings, say it has a shape of `(N, T, K)`. `N` is the number of elements in a batch, `T` is the sequence length and `K` is the feature (embedding) dimension. If we apply `unsqueeze()` operation on the `0` axis now, the tensor now has a shape of `(1, N, T, K)`. 2D dropout zeros random channels (which is the second axis in pytorch). Oh, wait... The second axis is the batch dimension! In this case, 2D dropout will zero several batch elements, not channels. \n\nA proper way to apply 1d dropout, in this case, is to reshape the tensor into `(N, K, 1, T)` tensor, apply the 2d dropout and then make it have `(N, T, K)` shape again.\n\nPlease contribute if you also noticed any wrong use of common operations!\n\nGood luck and happy kaggling! :0", "votes": 71 }, { "id": 468329, "postDate": "2019-02-08T17:22:46.143Z", "content": "

Both of these issues originated in my kernel.

\n\n

It is really questionable whether position-dependent bias makes sense. In the popular Keras attention script it is implemented the same way though:

\n\n
  if self.bias:\n          self.b = self.add_weight((input_shape[1],),\n                                   initializer='zero',\n                                   name='{}_b'.format(self.name),\n                                   regularizer=self.b_regularizer,\n                                   constraint=self.b_constraint)\n
\n\n

With a quick google search I was not able to find out whether bias is made position dependent in the original paper. Do you know about that?

\n\n

Now the second issue is certainly a mistake I made when writing the kernel. Sorry to everyone whose kernel this code spread to. I fixed this in a new version for people who might use my kernel as a reference later by doing:

\n\n
h_embedding = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 2)), 2)\n
\n\n

Is that correct?

\n\n

Thanks for pointing out the mistakes!

", "rawMarkdown": "Both of these issues originated in [my kernel](https://www.kaggle.com/bminixhofer/deterministic-neural-networks-using-pytorch). \n\nIt is really questionable whether position-dependent bias makes sense. In the popular Keras attention script it is implemented the same way though:\n\n if self.bias:\n self.b = self.add_weight((input_shape[1],),\n initializer='zero',\n name='{}_b'.format(self.name),\n regularizer=self.b_regularizer,\n constraint=self.b_constraint)\n\nWith a quick google search I was not able to find out whether bias is made position dependent in the original paper. Do you know about that?\n\nNow the second issue is certainly a mistake I made when writing the kernel. Sorry to everyone whose kernel this code spread to. I fixed this in a new version for people who might use my kernel as a reference later by doing:\n\n h_embedding = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 2)), 2)\n\nIs that correct?\n\nThanks for pointing out the mistakes!", "votes": 11, "replies": [ { "id": 468365, "postDate": "2019-02-08T19:05:02.573Z", "content": "

@bminixhofer you can refer to this implementation: https://github.com/TobiasLee/Text-Classification/blob/master/models/modules/attention.py

\n\n

It uses a bit more complicated version of attention, where each RNN output is first squashed using tanh(W @ h + b) transformation, but the bias is clearly shared among sequence elements and is not dependent on the sequence length.

", "rawMarkdown": "@bminixhofer you can refer to this implementation: https://github.com/TobiasLee/Text-Classification/blob/master/models/modules/attention.py\n\nIt uses a bit more complicated version of attention, where each RNN output is first squashed using `tanh(W @ h + b)` transformation, but the bias is clearly shared among sequence elements and is not dependent on the sequence length.", "votes": 3 }, { "id": 468371, "postDate": "2019-02-08T19:16:59.143Z", "content": "

@bminixhofer as for your dropout implementation, to make it correct, you need additionally to permute the dimensions after doing the unsqueeze() operation. If you apply unsqueeze on 2nd axis, tensor will have a shape of (N, T, 1, K) and if you then apply 2d dropout it will zero entire embeddings! So you need to add t.permute(0, 3, 2, 1) to make it correct. After doing a dropout, you need to permute the dimensions to original order using t.permute(0, 3, 2, 1) and do the squeeze().

\n\n

So the entire operation will look something like this:

\n\n

\nembeddings = ...\nembeddings = embeddings.unsqueeze(2) # (N, T, 1, K)\nembeddings = embeddings.permute(0, 3, 2, 1) # (N, K, 1, T)\nembeddings = self.dropout2d(embeddings) # (N, K, 1, T), some features are masked\nembeddings = embeddings.permute(0, 3, 2, 1) # (N, T, 1, K)\nembeddings = embeddings.squeeze(2) # (N, T, K)\n

\n\n

Looks a bit cumbersome, but you can put it into a separate module to make your main forward() function to be more clear.

", "rawMarkdown": "@bminixhofer as for your dropout implementation, to make it correct, you need additionally to permute the dimensions after doing the `unsqueeze()` operation. If you apply unsqueeze on 2nd axis, tensor will have a shape of (N, T, 1, K) and if you then apply 2d dropout it will zero entire embeddings! So you need to add `t.permute(0, 3, 2, 1)` to make it correct. After doing a dropout, you need to permute the dimensions to original order using `t.permute(0, 3, 2, 1)` and do the `squeeze()`.\n\nSo the entire operation will look something like this:\n\n```\nembeddings = ...\nembeddings = embeddings.unsqueeze(2) # (N, T, 1, K)\nembeddings = embeddings.permute(0, 3, 2, 1) # (N, K, 1, T)\nembeddings = self.dropout2d(embeddings) # (N, K, 1, T), some features are masked\nembeddings = embeddings.permute(0, 3, 2, 1) # (N, T, 1, K)\nembeddings = embeddings.squeeze(2) # (N, T, K)\n```\n\nLooks a bit cumbersome, but you can put it into a separate module to make your main `forward()` function to be more clear.\n", "votes": 4 }, { "id": 468392, "postDate": "2019-02-08T20:05:57.667Z", "content": "

suppose if we don't use permute to change the shape, then we are effectively dropping some words in the total sentence.

\n\n

Am I right Dmitriy ?

", "rawMarkdown": "suppose if we don't use permute to change the shape, then we are effectively dropping some words in the total sentence. \n\nAm I right Dmitriy ?", "votes": 2 }, { "id": 468437, "postDate": "2019-02-08T21:25:40.077Z", "content": "

Ok, that makes sense @ddanevskyi. I hope spatial dropout gets added to the standard PyTorch library. And yes, without permutation all features of some positions would be dropped @suchith0312.

", "rawMarkdown": "Ok, that makes sense @ddanevskyi. I hope spatial dropout gets added to the standard PyTorch library. And yes, without permutation all features of some positions would be dropped @suchith0312.", "votes": 2 }, { "id": 469365, "postDate": "2019-02-11T03:55:13.397Z", "content": "

Hi, here is a shorter code with the same concept

\n\n
emb = ...\nemb = self.dropout2d(emb.transpose(1,2).unsqueeze(-1)).squeeze().transpose(1,2)\n
", "rawMarkdown": "Hi, here is a shorter code with the same concept\n\n emb = ...\n emb = self.dropout2d(emb.transpose(1,2).unsqueeze(-1)).squeeze().transpose(1,2)", "votes": 3 }, { "id": 469483, "postDate": "2019-02-11T09:57:29.543Z", "content": "

@radream right! I used a more explicit version mainly to be able to write comments

", "rawMarkdown": "@radream right! I used a more explicit version mainly to be able to write comments" }, { "id": 470043, "postDate": "2019-02-12T09:35:46.977Z", "content": "

Here is the kernel with same implementation. Drastically improved CV 0.6792 to .6842\nhttps://www.kaggle.com/mlwhiz/third-place-model-for-toxic-spatial-dropout

", "rawMarkdown": "Here is the kernel with same implementation. Drastically improved CV 0.6792 to .6842\nhttps://www.kaggle.com/mlwhiz/third-place-model-for-toxic-spatial-dropout", "votes": 1 }, { "id": 470052, "postDate": "2019-02-12T10:15:18.753Z", "content": "

@mlwhiz cool!

", "rawMarkdown": "@mlwhiz cool!" }, { "id": 471372, "postDate": "2019-02-14T11:10:25.147Z", "content": "

@bminixhofer, thanks for your contributions in this competition. Your pytorch kernel and the pytorch starter by @hung96ad (thanks Hung) helped me start using pytorch in this contest. My pytorch model did not score as well as my keras model but I used it as a backup. So I am sad to not see both of you on the stage-2 LB, you easily could have jumped several places up. Anyway, see you in the next one.

", "rawMarkdown": "@bminixhofer, thanks for your contributions in this competition. Your pytorch kernel and the pytorch starter by @hung96ad (thanks Hung) helped me start using pytorch in this contest. My pytorch model did not score as well as my keras model but I used it as a backup. So I am sad to not see both of you on the stage-2 LB, you easily could have jumped several places up. Anyway, see you in the next one.", "votes": 3 }, { "id": 471441, "postDate": "2019-02-14T13:19:02.657Z", "content": "

You're welcome! And congratulations on your placement. I grew frustrated with this competition because I was not able to improve my score much further than the public kernels. It's amazing how many unique approaches people found for a competition with only text input and no external data.

", "rawMarkdown": "You're welcome! And congratulations on your placement. I grew frustrated with this competition because I was not able to improve my score much further than the public kernels. It's amazing how many unique approaches people found for a competition with only text input and no external data.", "votes": 3 }, { "id": 471472, "postDate": "2019-02-14T14:11:01.267Z", "content": "

Thanks @bminixhofer, I know what you mean. I felt just as frustrated but learning pytorch kept me going for a bit. In fact my best model that placed me at my current position was a submission from 2 months ago. So it is really interesting to see other's creatvity and I would surely use some in the future.

", "rawMarkdown": "Thanks @bminixhofer, I know what you mean. I felt just as frustrated but learning pytorch kept me going for a bit. In fact my best model that placed me at my current position was a submission from 2 months ago. So it is really interesting to see other's creatvity and I would surely use some in the future." }, { "id": 471930, "postDate": "2019-02-15T05:09:56.617Z", "content": "

thank @sheriytm and congratulations on your placement. I tried with many Kernel but my score isn't able to improve. Howerver, I know many approaches from this competition.

", "rawMarkdown": "thank @sheriytm and congratulations on your placement. I tried with many Kernel but my score isn't able to improve. Howerver, I know many approaches from this competition.", "votes": 1 }, { "id": 471936, "postDate": "2019-02-15T05:38:35Z", "content": "

Thanks @hung96ad . I feel the same way about approaches learnt here will be useful for future work. I am sure I will be revisiting some of these interesting submissions for inspiration just like I have referred to the highly optimized kernels in the Mercari Price competition a few times.

", "rawMarkdown": "Thanks @hung96ad . I feel the same way about approaches learnt here will be useful for future work. I am sure I will be revisiting some of these interesting submissions for inspiration just like I have referred to the highly optimized kernels in the Mercari Price competition a few times." } ] }, { "id": 468391, "postDate": "2019-02-08T20:04:41.100Z", "content": "

I used this alternative version of attention from https://discuss.pytorch.org/t/self-attention-on-words-and-masking/5671/4. I changed the code a bit to work with pytorch 1.0. This version doesn't use a bias term, but in my opinion, setting a maximum token length of 70 shouldn't decrease the performance of the model by too much. This version however, does have the problem of not masking before running softmax over the the attention weights. Allennlp has a masked_log_softmax function https://github.com/allenai/allennlp/blob/b6cc9d39651273e8ec2a7e334908ffa9de5c2026/allennlp/nn/util.py#L272-L303 that correctly masks the input sequence before applying softmax

\n\n
class SelfAttention(nn.Module):\ndef __init__(self, hidden_size, batch_first=False):\n    super(SelfAttention, self).__init__()\n\n    self.hidden_size = hidden_size\n    self.batch_first = batch_first\n\n    self.att_weights = nn.Parameter(torch.Tensor(1, hidden_size), requires_grad=True)\n    nn.init.xavier_uniform_(self.att_weights.data)\n\ndef get_mask(self):\n    pass\n\ndef forward(self, inputs, lengths):\n    if self.batch_first:\n        batch_size, max_len = inputs.size()[:2]\n    else:\n        max_len, batch_size = inputs.size()[:2]\n\n    # apply attention layer\n    weights = torch.bmm(inputs,\n                        self.att_weights  # (1, hidden_size)\n                        .permute(1, 0)  # (hidden_size, 1)\n                        .unsqueeze(0)  # (1, hidden_size, 1)\n                        .repeat(batch_size, 1, 1) # (batch_size, hidden_size, 1)\n                        )\n\n    attentions = torch.softmax(F.relu(weights.squeeze()), dim=-1)\n\n    # create mask based on the sentence lengths\n    mask = torch.ones(attentions.size(), requires_grad=True).cuda()\n    for i, l in enumerate(lengths):  # skip the first sentence\n        if l < max_len:\n            mask[i, l:] = 0\n\n    # apply mask and renormalize attention scores (weights)\n    masked = attentions * mask\n    _sums = masked.sum(-1).unsqueeze(-1)  # sums per row\n\n    attentions = masked.div(_sums)\n\n    # apply attention weights\n    weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))\n\n    # get the final fixed vector representations of the sentences\n    representations = weighted.sum(1).squeeze()\n\n    return representations, attentions\n
", "rawMarkdown": "I used this alternative version of attention from https://discuss.pytorch.org/t/self-attention-on-words-and-masking/5671/4. I changed the code a bit to work with pytorch 1.0. This version doesn't use a bias term, but in my opinion, setting a maximum token length of 70 shouldn't decrease the performance of the model by too much. This version however, does have the problem of not masking before running softmax over the the attention weights. Allennlp has a masked_log_softmax function https://github.com/allenai/allennlp/blob/b6cc9d39651273e8ec2a7e334908ffa9de5c2026/allennlp/nn/util.py#L272-L303 that correctly masks the input sequence before applying softmax\n\n class SelfAttention(nn.Module):\n def __init__(self, hidden_size, batch_first=False):\n super(SelfAttention, self).__init__()\n\n self.hidden_size = hidden_size\n self.batch_first = batch_first\n\n self.att_weights = nn.Parameter(torch.Tensor(1, hidden_size), requires_grad=True)\n nn.init.xavier_uniform_(self.att_weights.data)\n\n def get_mask(self):\n pass\n\n def forward(self, inputs, lengths):\n if self.batch_first:\n batch_size, max_len = inputs.size()[:2]\n else:\n max_len, batch_size = inputs.size()[:2]\n \n # apply attention layer\n weights = torch.bmm(inputs,\n self.att_weights # (1, hidden_size)\n .permute(1, 0) # (hidden_size, 1)\n .unsqueeze(0) # (1, hidden_size, 1)\n .repeat(batch_size, 1, 1) # (batch_size, hidden_size, 1)\n )\n \n attentions = torch.softmax(F.relu(weights.squeeze()), dim=-1)\n \n # create mask based on the sentence lengths\n mask = torch.ones(attentions.size(), requires_grad=True).cuda()\n for i, l in enumerate(lengths): # skip the first sentence\n if l < max_len:\n mask[i, l:] = 0\n\n # apply mask and renormalize attention scores (weights)\n masked = attentions * mask\n _sums = masked.sum(-1).unsqueeze(-1) # sums per row\n \n attentions = masked.div(_sums)\n\n # apply attention weights\n weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))\n\n # get the final fixed vector representations of the sentences\n representations = weighted.sum(1).squeeze()\n\n return representations, attentions", "votes": 3 }, { "id": 468544, "postDate": "2019-02-09T05:28:15.340Z", "content": "

I rerun my 2 submissions, with that dropout correction & guess what, my CV improved by 0.004. Wish I correct it earlier..

", "rawMarkdown": "I rerun my 2 submissions, with that dropout correction & guess what, my CV improved by 0.004. Wish I correct it earlier..", "votes": 4, "replies": [ { "id": 468792, "postDate": "2019-02-09T17:12:37.027Z", "content": "

For me also it was 0.0015 on CV score.

", "rawMarkdown": "For me also it was 0.0015 on CV score. ", "votes": 1 } ] }, { "id": 471993, "postDate": "2019-02-15T07:30:44.263Z", "content": "

@ddanevskyi, as an experiment I used late submission to check the result of a modified copy of @bminixhofer's kernel and one with his current update in the dropout mentioned. See the result below:-

\n\n\n\n

private LB = 0.70299

\n\n

public LB = 0.69834

\n\n\n\n

private LB = 0.70220

\n\n

public LB = 0.69395

", "rawMarkdown": "@ddanevskyi, as an experiment I used late submission to check the result of a modified copy of @bminixhofer's kernel and one with his current update in the dropout mentioned. See the result below:-\n\n- My modified copy of the kernel scored \n\nprivate LB = 0.70299\n\npublic LB = 0.69834\n\n- My same kernel with the latest corrections for the dropout.\n\nprivate LB = 0.70220\n\npublic LB = 0.69395", "votes": 1, "replies": [ { "id": 472135, "postDate": "2019-02-15T12:08:15.457Z", "content": "

Well, I just pointed out that the way dropout is used in public kernels is incorrect. Albeit some people reported major improvements from switching to the corrected version, it might not work for you out of the box for several reasons. First, \"correct\" dropout increases the batch size (since it doesn't drop batch elements), so you might need to tune your learning rate schedule or use a lower number of training epochs. Second, \"correct\" dropout does use dropout (haha), so you might need to do some additional hyperparameter tuning to make it work better.

", "rawMarkdown": "Well, I just pointed out that the way dropout is used in public kernels is incorrect. Albeit some people reported major improvements from switching to the corrected version, it might not work for you out of the box for several reasons. First, \"correct\" dropout increases the batch size (since it doesn't drop batch elements), so you might need to tune your learning rate schedule or use a lower number of training epochs. Second, \"correct\" dropout *does* use dropout (haha), so you might need to do some additional hyperparameter tuning to make it work better.", "votes": 2 }, { "id": 472155, "postDate": "2019-02-15T12:31:47.587Z", "content": "

I agree with your earlier explanation about it being wrong. I was just curious as to how much did it mess with the model. Hence surprised to see the results. Thought others may want to know since @bminixhofer's kernel does not show scores because he missed stage-2. I forgot to add above that the corrections improved the local CV as well.

", "rawMarkdown": "I agree with your earlier explanation about it being wrong. I was just curious as to how much did it mess with the model. Hence surprised to see the results. Thought others may want to know since @bminixhofer's kernel does not show scores because he missed stage-2. I forgot to add above that the corrections improved the local CV as well. ", "votes": 1 }, { "id": 472316, "postDate": "2019-02-15T17:24:49.217Z", "content": "

I think you need to retune the model when you changed the spatialdropout. It make the more harder to overfit.

", "rawMarkdown": "I think you need to retune the model when you changed the spatialdropout. It make the more harder to overfit.", "votes": 1 } ] }, { "id": 469983, "postDate": "2019-02-12T07:03:33.993Z", "content": "

very good observation :)

", "rawMarkdown": "very good observation :)", "votes": 1 }, { "id": 468778, "postDate": "2019-02-09T16:30:57.037Z", "content": "

Noooooooooo 😱

", "rawMarkdown": "Noooooooooo 😱", "votes": 1 }, { "id": 468330, "postDate": "2019-02-08T17:23:23.777Z", "content": "

Thanks for the post. I did copy the code of dropout and my cv score decreased, so didn't use it.

\n\n

Edit: I started to read more about spatial dropout. I didn't understand somethings quite well. (I may be wrong but this is my view)

\n\n

So in convolution layer spatial dropout zeros out the entire channel. So if input as 64 channels we randomly zero out some channels out of 64.

\n\n

Now in case of text, we push the embeddings to in each time step to RNN/LSTM. So here we randomly zero out some of the time steps out of total T steps(here maybe I am wrong).

\n\n

In pytorch, the nn.dropout2d has channels in 1st axis(if we start counting axis from 0), then we have to shape the tensor into (N, T, 1, K)(because time steps will be channels here) but you wrote (N, K, 1, T).

", "rawMarkdown": "Thanks for the post. I did copy the code of dropout and my cv score decreased, so didn't use it.\n\nEdit: I started to read more about spatial dropout. I didn't understand somethings quite well. (I may be wrong but this is my view)\n\nSo in convolution layer spatial dropout zeros out the entire channel. So if input as 64 channels we randomly zero out some channels out of 64. \n\nNow in case of text, we push the embeddings to in each time step to RNN/LSTM. So here we randomly zero out some of the time steps out of total T steps(here maybe I am wrong). \n\nIn pytorch, the nn.dropout2d has channels in 1st axis(if we start counting axis from 0), then we have to shape the tensor into (N, T, 1, K)(because time steps will be channels here) but you wrote (N, K, 1, T). ", "votes": 1, "replies": [ { "id": 468366, "postDate": "2019-02-08T19:08:36.250Z", "content": "

@suchith0312 It has to be (N, K, 1, T). We clearly want to zero features, not timesteps. You can think of this tensor as a batch of images, having K channels, the width of T and the height of 1.

", "rawMarkdown": "@suchith0312 It has to be (N, K, 1, T). We clearly want to zero *features*, not timesteps. You can think of this tensor as a batch of images, having K channels, the width of T and the height of 1.", "votes": 6 }, { "id": 468389, "postDate": "2019-02-08T20:02:57.883Z", "content": "

Thanks for the reply. I think you are right as I have been checking keras spatial dropout implementation and they use a noise shape of (batch_size,1, features). So eventually they are dropping the one dimension of the embedding for all time steps.

\n\n

But I am not getting the feel of why spatial dropout works in some cases?

\n\n

Suppose we have a sequence length of 3 and embedding dimension of 6.

\n\n

Now if the shape is (N,6,1,3) and dropout2d on that will make to drop some out of 6 channels present.\nLet's suppose we have input as [1,2,3,4,5,6] , [7,8,9,10,11,12] , [-1,-2,-3,-4,-5,-6] (embedding of some word), then after applying dropout we can get [1,2,0,0,5,6] , [7,8,0,0,11,12] , [-1,-2,0,0,-5,-6]. This can be embedding of some other word. So by dropping the features, we are changing the word.

\n\n

So it makes sense to drop some of the words as in a sentence, as these words can be noisy and the meaning of sentence really doesn't depend on these words.

", "rawMarkdown": "Thanks for the reply. I think you are right as I have been checking keras spatial dropout implementation and they use a noise shape of `(batch_size,1, features)`. So eventually they are dropping the one dimension of the embedding for all time steps.\n \nBut I am not getting the feel of why spatial dropout works in some cases?\n \nSuppose we have a sequence length of 3 and embedding dimension of 6.\n\nNow if the shape is `(N,6,1,3)` and dropout2d on that will make to drop some out of 6 channels present.\nLet's suppose we have input as `[1,2,3,4,5,6] , [7,8,9,10,11,12] , [-1,-2,-3,-4,-5,-6]` (embedding of some word), then after applying dropout we can get `[1,2,0,0,5,6] , [7,8,0,0,11,12] , [-1,-2,0,0,-5,-6]`. This can be embedding of some other word. So by dropping the features, we are changing the word. \n\nSo it makes sense to drop some of the words as in a sentence, as these words can be noisy and the meaning of sentence really doesn't depend on these words.", "votes": 2 } ] }, { "id": 468317, "postDate": "2019-02-08T17:03:14.830Z", "content": "

thanks for pointing out the correct usage of dropout! I was so curious why my pytorch and keras giving me very different convergence speed - my guess was the spatial dropout part but didn't pay too much attention and ignored the error = =|||

", "rawMarkdown": "thanks for pointing out the correct usage of dropout! I was so curious why my pytorch and keras giving me very different convergence speed - my guess was the spatial dropout part but didn't pay too much attention and ignored the error = =|||", "votes": 1 }, { "id": 468294, "postDate": "2019-02-08T16:26:52.623Z", "content": "

Thanks for pointing out the correct usage of the dropout. I saw that in many kernels and was pretty confident what they were doing wasnt correct but didnt know enough about pytorch to say for sure. Glad we stuck with keras throughout

", "rawMarkdown": "Thanks for pointing out the correct usage of the dropout. I saw that in many kernels and was pretty confident what they were doing wasnt correct but didnt know enough about pytorch to say for sure. Glad we stuck with keras throughout", "votes": 1 }, { "id": 469302, "postDate": "2019-02-10T23:02:52.523Z", "content": "

Haha, that's fascinating! I used hyperopt to find the best possible hyperparameters and it has suggested that I remove embedding dropout completely.

\n\n

However, this \"dropout\" essentially means sampling of random 90% of samples on every epoch, so it's (sort of) regularization.

", "rawMarkdown": "Haha, that's fascinating! I used hyperopt to find the best possible hyperparameters and it has suggested that I remove embedding dropout completely. \n\nHowever, this \"dropout\" essentially means sampling of random 90% of samples on every epoch, so it's (sort of) regularization.", "votes": 2, "replies": [ { "id": 469321, "postDate": "2019-02-11T00:52:54.957Z", "content": "

@artyomp, yeah but the point is that it was not working as intended :-)

", "rawMarkdown": "@artyomp, yeah but the point is that it was not working as intended :-)", "votes": 2 }, { "id": 469530, "postDate": "2019-02-11T11:30:17.243Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 469009, "postDate": "2019-02-10T09:44:51.100Z", "content": "

Thanks @ddanevskyi for pointing these out. I also noticed one of the issues but was not quite sure if it was really wrong. My pytorch skills is not as defined as my keras hence I hesitated, I wish I asked anyway.

", "rawMarkdown": "Thanks @ddanevskyi for pointing these out. I also noticed one of the issues but was not quite sure if it was really wrong. My pytorch skills is not as defined as my keras hence I hesitated, I wish I asked anyway.", "votes": 2 }, { "id": 471983, "postDate": "2019-02-15T07:16:39.673Z", "content": "

Dmitriy, very good observations and thank you for sharing! But what would be the difference of using regular dropout self.embedding_dropout = nn.Dropout(0.1) with \nh = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0))) afterwards?

", "rawMarkdown": "Dmitriy, very good observations and thank you for sharing! But what would be the difference of using regular dropout `self.embedding_dropout = nn.Dropout(0.1)` with \n`h = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0)))` afterwards?", "replies": [ { "id": 472127, "postDate": "2019-02-15T11:57:40.743Z", "content": "

Regular dropout drops elements independently, while \"spatial\" or \"locked\" dropout applies the same mask for all sequence steps. It was demonstrated that the latter performs better for RNN models. See https://arxiv.org/abs/1512.05287 for details.

", "rawMarkdown": "Regular dropout drops elements independently, while \"spatial\" or \"locked\" dropout applies the same mask for all sequence steps. It was demonstrated that the latter performs better for RNN models. See https://arxiv.org/abs/1512.05287 for details.", "votes": 4 }, { "id": 472165, "postDate": "2019-02-15T12:43:25.690Z", "content": "

Thanks again! Well, regular dropout worked better than the one with the mistake you've pointed out.

", "rawMarkdown": "Thanks again! Well, regular dropout worked better than the one with the mistake you've pointed out. ", "votes": 1 } ] }, { "id": 469181, "postDate": "2019-02-10T16:55:21.730Z", "rawMarkdown": "", "isDeleted": true }, { "id": 468718, "postDate": "2019-02-09T14:32:45.413Z", "rawMarkdown": "", "isDeleted": true }, { "id": 470811, "postDate": "2019-02-13T16:05:08.313Z", "content": "

Thanks for sharing Dmitriy!

", "rawMarkdown": "Thanks for sharing Dmitriy!", "votes": 1 }, { "id": 469712, "postDate": "2019-02-11T17:58:58.650Z", "content": "

Thanks~

", "rawMarkdown": "Thanks~", "votes": 1 } ], "comments": [ { "id": 468329, "author_name": "Benjamin Minixhofer", "author_url": "", "post_date": "2019-02-08T17:22:46.143000", "content": "

Both of these issues originated in my kernel.

\n\n

It is really questionable whether position-dependent bias makes sense. In the popular Keras attention script it is implemented the same way though:

\n\n
  if self.bias:\n          self.b = self.add_weight((input_shape[1],),\n                                   initializer='zero',\n                                   name='{}_b'.format(self.name),\n                                   regularizer=self.b_regularizer,\n                                   constraint=self.b_constraint)\n
\n\n

With a quick google search I was not able to find out whether bias is made position dependent in the original paper. Do you know about that?

\n\n

Now the second issue is certainly a mistake I made when writing the kernel. Sorry to everyone whose kernel this code spread to. I fixed this in a new version for people who might use my kernel as a reference later by doing:

\n\n
h_embedding = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 2)), 2)\n
\n\n

Is that correct?

\n\n

Thanks for pointing out the mistakes!

", "votes": 11, "replies": [ { "id": 468365, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-02-08T19:05:02.573000", "content": "

@bminixhofer you can refer to this implementation: https://github.com/TobiasLee/Text-Classification/blob/master/models/modules/attention.py

\n\n

It uses a bit more complicated version of attention, where each RNN output is first squashed using tanh(W @ h + b) transformation, but the bias is clearly shared among sequence elements and is not dependent on the sequence length.

", "votes": 3, "replies": [] }, { "id": 468371, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-02-08T19:16:59.143000", "content": "

@bminixhofer as for your dropout implementation, to make it correct, you need additionally to permute the dimensions after doing the unsqueeze() operation. If you apply unsqueeze on 2nd axis, tensor will have a shape of (N, T, 1, K) and if you then apply 2d dropout it will zero entire embeddings! So you need to add t.permute(0, 3, 2, 1) to make it correct. After doing a dropout, you need to permute the dimensions to original order using t.permute(0, 3, 2, 1) and do the squeeze().

\n\n

So the entire operation will look something like this:

\n\n

\nembeddings = ...\nembeddings = embeddings.unsqueeze(2) # (N, T, 1, K)\nembeddings = embeddings.permute(0, 3, 2, 1) # (N, K, 1, T)\nembeddings = self.dropout2d(embeddings) # (N, K, 1, T), some features are masked\nembeddings = embeddings.permute(0, 3, 2, 1) # (N, T, 1, K)\nembeddings = embeddings.squeeze(2) # (N, T, K)\n

\n\n

Looks a bit cumbersome, but you can put it into a separate module to make your main forward() function to be more clear.

", "votes": 4, "replies": [] }, { "id": 468392, "author_name": "redshellspy", "author_url": "", "post_date": "2019-02-08T20:05:57.667000", "content": "

suppose if we don't use permute to change the shape, then we are effectively dropping some words in the total sentence.

\n\n

Am I right Dmitriy ?

", "votes": 2, "replies": [] }, { "id": 468437, "author_name": "Benjamin Minixhofer", "author_url": "", "post_date": "2019-02-08T21:25:40.077000", "content": "

Ok, that makes sense @ddanevskyi. I hope spatial dropout gets added to the standard PyTorch library. And yes, without permutation all features of some positions would be dropped @suchith0312.

", "votes": 2, "replies": [] }, { "id": 469365, "author_name": "Yu-ray Li", "author_url": "", "post_date": "2019-02-11T03:55:13.397000", "content": "

Hi, here is a shorter code with the same concept

\n\n
emb = ...\nemb = self.dropout2d(emb.transpose(1,2).unsqueeze(-1)).squeeze().transpose(1,2)\n
", "votes": 3, "replies": [] }, { "id": 469483, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-02-11T09:57:29.543000", "content": "

@radream right! I used a more explicit version mainly to be able to write comments

", "votes": 0, "replies": [] }, { "id": 470043, "author_name": "Rahul Agarwal", "author_url": "", "post_date": "2019-02-12T09:35:46.977000", "content": "

Here is the kernel with same implementation. Drastically improved CV 0.6792 to .6842\nhttps://www.kaggle.com/mlwhiz/third-place-model-for-toxic-spatial-dropout

", "votes": 1, "replies": [] }, { "id": 470052, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-02-12T10:15:18.753000", "content": "

@mlwhiz cool!

", "votes": 0, "replies": [] }, { "id": 471372, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-14T11:10:25.147000", "content": "

@bminixhofer, thanks for your contributions in this competition. Your pytorch kernel and the pytorch starter by @hung96ad (thanks Hung) helped me start using pytorch in this contest. My pytorch model did not score as well as my keras model but I used it as a backup. So I am sad to not see both of you on the stage-2 LB, you easily could have jumped several places up. Anyway, see you in the next one.

", "votes": 3, "replies": [] }, { "id": 471441, "author_name": "Benjamin Minixhofer", "author_url": "", "post_date": "2019-02-14T13:19:02.657000", "content": "

You're welcome! And congratulations on your placement. I grew frustrated with this competition because I was not able to improve my score much further than the public kernels. It's amazing how many unique approaches people found for a competition with only text input and no external data.

", "votes": 3, "replies": [] }, { "id": 471472, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-14T14:11:01.267000", "content": "

Thanks @bminixhofer, I know what you mean. I felt just as frustrated but learning pytorch kept me going for a bit. In fact my best model that placed me at my current position was a submission from 2 months ago. So it is really interesting to see other's creatvity and I would surely use some in the future.

", "votes": 0, "replies": [] }, { "id": 471930, "author_name": "Hung The Nguyen", "author_url": "", "post_date": "2019-02-15T05:09:56.617000", "content": "

thank @sheriytm and congratulations on your placement. I tried with many Kernel but my score isn't able to improve. Howerver, I know many approaches from this competition.

", "votes": 1, "replies": [] }, { "id": 471936, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-15T05:38:35", "content": "

Thanks @hung96ad . I feel the same way about approaches learnt here will be useful for future work. I am sure I will be revisiting some of these interesting submissions for inspiration just like I have referred to the highly optimized kernels in the Mercari Price competition a few times.

", "votes": 0, "replies": [] } ] }, { "id": 468391, "author_name": "bilal2vec", "author_url": "", "post_date": "2019-02-08T20:04:41.100000", "content": "

I used this alternative version of attention from https://discuss.pytorch.org/t/self-attention-on-words-and-masking/5671/4. I changed the code a bit to work with pytorch 1.0. This version doesn't use a bias term, but in my opinion, setting a maximum token length of 70 shouldn't decrease the performance of the model by too much. This version however, does have the problem of not masking before running softmax over the the attention weights. Allennlp has a masked_log_softmax function https://github.com/allenai/allennlp/blob/b6cc9d39651273e8ec2a7e334908ffa9de5c2026/allennlp/nn/util.py#L272-L303 that correctly masks the input sequence before applying softmax

\n\n
class SelfAttention(nn.Module):\ndef __init__(self, hidden_size, batch_first=False):\n    super(SelfAttention, self).__init__()\n\n    self.hidden_size = hidden_size\n    self.batch_first = batch_first\n\n    self.att_weights = nn.Parameter(torch.Tensor(1, hidden_size), requires_grad=True)\n    nn.init.xavier_uniform_(self.att_weights.data)\n\ndef get_mask(self):\n    pass\n\ndef forward(self, inputs, lengths):\n    if self.batch_first:\n        batch_size, max_len = inputs.size()[:2]\n    else:\n        max_len, batch_size = inputs.size()[:2]\n\n    # apply attention layer\n    weights = torch.bmm(inputs,\n                        self.att_weights  # (1, hidden_size)\n                        .permute(1, 0)  # (hidden_size, 1)\n                        .unsqueeze(0)  # (1, hidden_size, 1)\n                        .repeat(batch_size, 1, 1) # (batch_size, hidden_size, 1)\n                        )\n\n    attentions = torch.softmax(F.relu(weights.squeeze()), dim=-1)\n\n    # create mask based on the sentence lengths\n    mask = torch.ones(attentions.size(), requires_grad=True).cuda()\n    for i, l in enumerate(lengths):  # skip the first sentence\n        if l < max_len:\n            mask[i, l:] = 0\n\n    # apply mask and renormalize attention scores (weights)\n    masked = attentions * mask\n    _sums = masked.sum(-1).unsqueeze(-1)  # sums per row\n\n    attentions = masked.div(_sums)\n\n    # apply attention weights\n    weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))\n\n    # get the final fixed vector representations of the sentences\n    representations = weighted.sum(1).squeeze()\n\n    return representations, attentions\n
", "votes": 3, "replies": [] }, { "id": 468544, "author_name": "Prashant Kikani", "author_url": "", "post_date": "2019-02-09T05:28:15.340000", "content": "

I rerun my 2 submissions, with that dropout correction & guess what, my CV improved by 0.004. Wish I correct it earlier..

", "votes": 4, "replies": [ { "id": 468792, "author_name": "Rahul Agarwal", "author_url": "", "post_date": "2019-02-09T17:12:37.027000", "content": "

For me also it was 0.0015 on CV score.

", "votes": 1, "replies": [] } ] }, { "id": 471993, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-15T07:30:44.263000", "content": "

@ddanevskyi, as an experiment I used late submission to check the result of a modified copy of @bminixhofer's kernel and one with his current update in the dropout mentioned. See the result below:-

\n\n\n\n

private LB = 0.70299

\n\n

public LB = 0.69834

\n\n\n\n

private LB = 0.70220

\n\n

public LB = 0.69395

", "votes": 1, "replies": [ { "id": 472135, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-02-15T12:08:15.457000", "content": "

Well, I just pointed out that the way dropout is used in public kernels is incorrect. Albeit some people reported major improvements from switching to the corrected version, it might not work for you out of the box for several reasons. First, \"correct\" dropout increases the batch size (since it doesn't drop batch elements), so you might need to tune your learning rate schedule or use a lower number of training epochs. Second, \"correct\" dropout does use dropout (haha), so you might need to do some additional hyperparameter tuning to make it work better.

", "votes": 2, "replies": [] }, { "id": 472155, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-15T12:31:47.587000", "content": "

I agree with your earlier explanation about it being wrong. I was just curious as to how much did it mess with the model. Hence surprised to see the results. Thought others may want to know since @bminixhofer's kernel does not show scores because he missed stage-2. I forgot to add above that the corrections improved the local CV as well.

", "votes": 1, "replies": [] }, { "id": 472316, "author_name": "Smilence", "author_url": "", "post_date": "2019-02-15T17:24:49.217000", "content": "

I think you need to retune the model when you changed the spatialdropout. It make the more harder to overfit.

", "votes": 1, "replies": [] } ] }, { "id": 469983, "author_name": "Chintankumar Upadhyay", "author_url": "", "post_date": "2019-02-12T07:03:33.993000", "content": "

very good observation :)

", "votes": 1, "replies": [] }, { "id": 468778, "author_name": "Max Schumacher", "author_url": "", "post_date": "2019-02-09T16:30:57.037000", "content": "

Noooooooooo 😱

", "votes": 1, "replies": [] }, { "id": 468330, "author_name": "redshellspy", "author_url": "", "post_date": "2019-02-08T17:23:23.777000", "content": "

Thanks for the post. I did copy the code of dropout and my cv score decreased, so didn't use it.

\n\n

Edit: I started to read more about spatial dropout. I didn't understand somethings quite well. (I may be wrong but this is my view)

\n\n

So in convolution layer spatial dropout zeros out the entire channel. So if input as 64 channels we randomly zero out some channels out of 64.

\n\n

Now in case of text, we push the embeddings to in each time step to RNN/LSTM. So here we randomly zero out some of the time steps out of total T steps(here maybe I am wrong).

\n\n

In pytorch, the nn.dropout2d has channels in 1st axis(if we start counting axis from 0), then we have to shape the tensor into (N, T, 1, K)(because time steps will be channels here) but you wrote (N, K, 1, T).

", "votes": 1, "replies": [ { "id": 468366, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-02-08T19:08:36.250000", "content": "

@suchith0312 It has to be (N, K, 1, T). We clearly want to zero features, not timesteps. You can think of this tensor as a batch of images, having K channels, the width of T and the height of 1.

", "votes": 6, "replies": [] }, { "id": 468389, "author_name": "redshellspy", "author_url": "", "post_date": "2019-02-08T20:02:57.883000", "content": "

Thanks for the reply. I think you are right as I have been checking keras spatial dropout implementation and they use a noise shape of (batch_size,1, features). So eventually they are dropping the one dimension of the embedding for all time steps.

\n\n

But I am not getting the feel of why spatial dropout works in some cases?

\n\n

Suppose we have a sequence length of 3 and embedding dimension of 6.

\n\n

Now if the shape is (N,6,1,3) and dropout2d on that will make to drop some out of 6 channels present.\nLet's suppose we have input as [1,2,3,4,5,6] , [7,8,9,10,11,12] , [-1,-2,-3,-4,-5,-6] (embedding of some word), then after applying dropout we can get [1,2,0,0,5,6] , [7,8,0,0,11,12] , [-1,-2,0,0,-5,-6]. This can be embedding of some other word. So by dropping the features, we are changing the word.

\n\n

So it makes sense to drop some of the words as in a sentence, as these words can be noisy and the meaning of sentence really doesn't depend on these words.

", "votes": 2, "replies": [] } ] }, { "id": 468317, "author_name": "Smilence", "author_url": "", "post_date": "2019-02-08T17:03:14.830000", "content": "

thanks for pointing out the correct usage of dropout! I was so curious why my pytorch and keras giving me very different convergence speed - my guess was the spatial dropout part but didn't pay too much attention and ignored the error = =|||

", "votes": 1, "replies": [] }, { "id": 468294, "author_name": "ryches", "author_url": "", "post_date": "2019-02-08T16:26:52.623000", "content": "

Thanks for pointing out the correct usage of the dropout. I saw that in many kernels and was pretty confident what they were doing wasnt correct but didnt know enough about pytorch to say for sure. Glad we stuck with keras throughout

", "votes": 1, "replies": [] }, { "id": 469302, "author_name": "Artyom Palvelev", "author_url": "", "post_date": "2019-02-10T23:02:52.523000", "content": "

Haha, that's fascinating! I used hyperopt to find the best possible hyperparameters and it has suggested that I remove embedding dropout completely.

\n\n

However, this \"dropout\" essentially means sampling of random 90% of samples on every epoch, so it's (sort of) regularization.

", "votes": 2, "replies": [ { "id": 469321, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-11T00:52:54.957000", "content": "

@artyomp, yeah but the point is that it was not working as intended :-)

", "votes": 2, "replies": [] }, { "id": 469530, "author_name": "", "author_url": "", "post_date": "2019-02-11T11:30:17.243000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 469009, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-10T09:44:51.100000", "content": "

Thanks @ddanevskyi for pointing these out. I also noticed one of the issues but was not quite sure if it was really wrong. My pytorch skills is not as defined as my keras hence I hesitated, I wish I asked anyway.

", "votes": 2, "replies": [] }, { "id": 471983, "author_name": "Kostya Kravchenko", "author_url": "", "post_date": "2019-02-15T07:16:39.673000", "content": "

Dmitriy, very good observations and thank you for sharing! But what would be the difference of using regular dropout self.embedding_dropout = nn.Dropout(0.1) with \nh = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0))) afterwards?

", "votes": 0, "replies": [ { "id": 472127, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-02-15T11:57:40.743000", "content": "

Regular dropout drops elements independently, while \"spatial\" or \"locked\" dropout applies the same mask for all sequence steps. It was demonstrated that the latter performs better for RNN models. See https://arxiv.org/abs/1512.05287 for details.

", "votes": 4, "replies": [] }, { "id": 472165, "author_name": "Kostya Kravchenko", "author_url": "", "post_date": "2019-02-15T12:43:25.690000", "content": "

Thanks again! Well, regular dropout worked better than the one with the mistake you've pointed out.

", "votes": 1, "replies": [] } ] }, { "id": 469181, "author_name": "", "author_url": "", "post_date": "2019-02-10T16:55:21.730000", "content": "", "votes": 0, "replies": [] }, { "id": 468718, "author_name": "", "author_url": "", "post_date": "2019-02-09T14:32:45.413000", "content": "", "votes": 0, "replies": [] }, { "id": 470811, "author_name": "mtodisco10", "author_url": "", "post_date": "2019-02-13T16:05:08.313000", "content": "

Thanks for sharing Dmitriy!

", "votes": 1, "replies": [] }, { "id": 469712, "author_name": "Jalen Fu", "author_url": "", "post_date": "2019-02-11T17:58:58.650000", "content": "

Thanks~

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "468241": "Now, when this crazy competition is (almost) over, it's a perfect time to rest and do some analysis :)\n\nIn this topic, I want to spot some common mistakes made in public kernels created during this competition. Some of these mistakes were repeated so many times that they almost stopped to look queerly :)\n\nNote that I created this topic for educational purposes only, not to make kernels authors look weird or stupid. We all make occasional mistakes from time to time, especially when we don't have much time to do the proper code check.\n\n**1. Position-dependent bias**\n\nMany pytorch kernels use `Attention` class that has the following line:\n\n`self.b = nn.Parameter(torch.zeros(step_dim))`\n\n`step_dim` is chosen to be 70, which is the length to which all questions are either padded or truncated.\n\nThus, the bias term has the individual component for each sequence step. This is weird, at least because different questions have a different number of tokens. What if you need to process a question which is longer than 70 tokens? It's clear that a model should not be constrained to work only for questions of some particular length. Another argument is that each bias component is \"tied\" to a specific token position in a sentence, but an absolute position rarely has something to do with the token meaning or any other linguistic property.\n\nA more reasonable way to use a bias term is to have a single scalar bias that is shared among all sequence elements. \n\n**2. \"Spatial\" dropout**\n\nThe same pytorch code has another suspicious component: the use of 2d dropout. It's used since pytorch doesn't provide a proper implementation of 1d dropout (that zeros out whole channels of a sequence of 1d vectors). So one option is to use a 2d version on unsqueezed embeddings. However, the implementation used in public kernels is completely wrong :)\n\n2d dropout layer is initialized as follows:\n\n`self.embedding_dropout = nn.Dropout2d(0.1)`\n\nand then used like this:\n\n`h = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0)))`\n\nLet's see what is going on here. First, you have a batch of embeddings, say it has a shape of `(N, T, K)`. `N` is the number of elements in a batch, `T` is the sequence length and `K` is the feature (embedding) dimension. If we apply `unsqueeze()` operation on the `0` axis now, the tensor now has a shape of `(1, N, T, K)`. 2D dropout zeros random channels (which is the second axis in pytorch). Oh, wait... The second axis is the batch dimension! In this case, 2D dropout will zero several batch elements, not channels. \n\nA proper way to apply 1d dropout, in this case, is to reshape the tensor into `(N, K, 1, T)` tensor, apply the 2d dropout and then make it have `(N, T, K)` shape again.\n\nPlease contribute if you also noticed any wrong use of common operations!\n\nGood luck and happy kaggling! :0", "468329": "Both of these issues originated in [my kernel](https://www.kaggle.com/bminixhofer/deterministic-neural-networks-using-pytorch). \n\nIt is really questionable whether position-dependent bias makes sense. In the popular Keras attention script it is implemented the same way though:\n\n if self.bias:\n self.b = self.add_weight((input_shape[1],),\n initializer='zero',\n name='{}_b'.format(self.name),\n regularizer=self.b_regularizer,\n constraint=self.b_constraint)\n\nWith a quick google search I was not able to find out whether bias is made position dependent in the original paper. Do you know about that?\n\nNow the second issue is certainly a mistake I made when writing the kernel. Sorry to everyone whose kernel this code spread to. I fixed this in a new version for people who might use my kernel as a reference later by doing:\n\n h_embedding = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 2)), 2)\n\nIs that correct?\n\nThanks for pointing out the mistakes!", "468391": "I used this alternative version of attention from https://discuss.pytorch.org/t/self-attention-on-words-and-masking/5671/4. I changed the code a bit to work with pytorch 1.0. This version doesn't use a bias term, but in my opinion, setting a maximum token length of 70 shouldn't decrease the performance of the model by too much. This version however, does have the problem of not masking before running softmax over the the attention weights. Allennlp has a masked_log_softmax function https://github.com/allenai/allennlp/blob/b6cc9d39651273e8ec2a7e334908ffa9de5c2026/allennlp/nn/util.py#L272-L303 that correctly masks the input sequence before applying softmax\n\n class SelfAttention(nn.Module):\n def __init__(self, hidden_size, batch_first=False):\n super(SelfAttention, self).__init__()\n\n self.hidden_size = hidden_size\n self.batch_first = batch_first\n\n self.att_weights = nn.Parameter(torch.Tensor(1, hidden_size), requires_grad=True)\n nn.init.xavier_uniform_(self.att_weights.data)\n\n def get_mask(self):\n pass\n\n def forward(self, inputs, lengths):\n if self.batch_first:\n batch_size, max_len = inputs.size()[:2]\n else:\n max_len, batch_size = inputs.size()[:2]\n \n # apply attention layer\n weights = torch.bmm(inputs,\n self.att_weights # (1, hidden_size)\n .permute(1, 0) # (hidden_size, 1)\n .unsqueeze(0) # (1, hidden_size, 1)\n .repeat(batch_size, 1, 1) # (batch_size, hidden_size, 1)\n )\n \n attentions = torch.softmax(F.relu(weights.squeeze()), dim=-1)\n \n # create mask based on the sentence lengths\n mask = torch.ones(attentions.size(), requires_grad=True).cuda()\n for i, l in enumerate(lengths): # skip the first sentence\n if l < max_len:\n mask[i, l:] = 0\n\n # apply mask and renormalize attention scores (weights)\n masked = attentions * mask\n _sums = masked.sum(-1).unsqueeze(-1) # sums per row\n \n attentions = masked.div(_sums)\n\n # apply attention weights\n weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))\n\n # get the final fixed vector representations of the sentences\n representations = weighted.sum(1).squeeze()\n\n return representations, attentions", "468544": "I rerun my 2 submissions, with that dropout correction & guess what, my CV improved by 0.004. Wish I correct it earlier..", "471993": "@ddanevskyi, as an experiment I used late submission to check the result of a modified copy of @bminixhofer's kernel and one with his current update in the dropout mentioned. See the result below:-\n\n- My modified copy of the kernel scored \n\nprivate LB = 0.70299\n\npublic LB = 0.69834\n\n- My same kernel with the latest corrections for the dropout.\n\nprivate LB = 0.70220\n\npublic LB = 0.69395", "469983": "very good observation :)", "468778": "Noooooooooo 😱", "468330": "Thanks for the post. I did copy the code of dropout and my cv score decreased, so didn't use it.\n\nEdit: I started to read more about spatial dropout. I didn't understand somethings quite well. (I may be wrong but this is my view)\n\nSo in convolution layer spatial dropout zeros out the entire channel. So if input as 64 channels we randomly zero out some channels out of 64. \n\nNow in case of text, we push the embeddings to in each time step to RNN/LSTM. So here we randomly zero out some of the time steps out of total T steps(here maybe I am wrong). \n\nIn pytorch, the nn.dropout2d has channels in 1st axis(if we start counting axis from 0), then we have to shape the tensor into (N, T, 1, K)(because time steps will be channels here) but you wrote (N, K, 1, T). ", "468317": "thanks for pointing out the correct usage of dropout! I was so curious why my pytorch and keras giving me very different convergence speed - my guess was the spatial dropout part but didn't pay too much attention and ignored the error = =|||", "468294": "Thanks for pointing out the correct usage of the dropout. I saw that in many kernels and was pretty confident what they were doing wasnt correct but didnt know enough about pytorch to say for sure. Glad we stuck with keras throughout", "469302": "Haha, that's fascinating! I used hyperopt to find the best possible hyperparameters and it has suggested that I remove embedding dropout completely. \n\nHowever, this \"dropout\" essentially means sampling of random 90% of samples on every epoch, so it's (sort of) regularization.", "469009": "Thanks @ddanevskyi for pointing these out. I also noticed one of the issues but was not quite sure if it was really wrong. My pytorch skills is not as defined as my keras hence I hesitated, I wish I asked anyway.", "471983": "Dmitriy, very good observations and thank you for sharing! But what would be the difference of using regular dropout `self.embedding_dropout = nn.Dropout(0.1)` with \n`h = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0)))` afterwards?", "469181": "", "468718": "", "470811": "Thanks for sharing Dmitriy!", "469712": "Thanks~" } }