![\"\"](\"https://i.ibb.co/L1S3cW0/Selection-352.png\")
I prepared Resnet + LSTM with attention starter code.
\nThe result is CV: 18.7, LB: 20.3
\nIn inference notebook, I used 2 epochs trained weight. There is much room to improve, for example more epochs, augmentation, larger models, larger size…
\nHope this helps, happy kaggling!
", "messages": [ { "id": "1237631", "postDate": "03/14/2021 10:19:13", "content": "I prepared Resnet + LSTM with attention starter code.
\nThe result is CV: 18.7, LB: 20.3
\nIn inference notebook, I used 2 epochs trained weight. There is much room to improve, for example more epochs, augmentation, larger models, larger size…
\nHope this helps, happy kaggling!
", "rawMarkdown": "I prepared Resnet + LSTM with attention starter code.\n\n1. [preprocess](https://www.kaggle.com/yasufuminakama/inchi-preprocess-2)\n2. [training](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter)\n3. [inference](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-inference)\n\nThe result is CV: 18.7, LB: 20.3\n\nIn inference notebook, I used 2 epochs trained weight. There is much room to improve, for example more epochs, augmentation, larger models, larger size...\n\nHope this helps, happy kaggling!", "votes": null }, { "id": "1237740", "postDate": "03/14/2021 12:15:32", "content": "Wouldn't be better to just explain the workflow in the discussion without publishing the actual code?
\nI can already see 2 scores exactly at 21.9 like yours at 5th and 6th positions…
I understand what you say, but it's easier to join the competition for beginners if there is starter code.
\nWhen I was beginner, I learned a lot from public notebooks, that's why I publish starter codes for other competitions too.
\nYou don't need to be so nervous about current LB position, it's just beginning of the competition. There are a lot of things to improve score.
I agree with you, I learned a lot from public notebooks in my previous competition (MoA) but the problem with beginners is that they tend to deceive themselves with copying and submitting the notebook without trying to understand the code (even at the beginning of the competition) and that's bad but they don't realize it. Also, they are people who try their best to learn and improve their scores and that is really demotivating for people who try their best.
\nI personally never copy past code that I don't understand, and my message to @jacoporepossi
\nis to not give up and like @yasufuminakama said it's the beginning of the competition and there is room for improvement and thank you for sharing your notebook with us, very helpful.
By the way, How can we retrain models when we surpass the 9 hour limit of the Kaggle kernel?
", "rawMarkdown": "I agree with you, I learned a lot from public notebooks in my previous competition (MoA) but the problem with beginners is that they tend to deceive themselves with copying and submitting the notebook without trying to understand the code (even at the beginning of the competition) and that's bad but they don't realize it. Also, they are people who try their best to learn and improve their scores and that is really demotivating for people who try their best.\n\nI personally never copy past code that I don't understand, and my message to @jacoporepossi \nis to not give up and like @yasufuminakama said it's the beginning of the competition and there is room for improvement and thank you for sharing your notebook with us, very helpful.\n\nBy the way, How can we retrain models when we surpass the 9 hour limit of the Kaggle kernel?", "votes": null }, { "id": "1238286", "postDate": "03/14/2021 20:35:03", "content": "You can reload saved model states, optimizer states, scheduler states, and restart training.
", "rawMarkdown": "You can reload saved model states, optimizer states, scheduler states, and restart training.", "votes": null }, { "id": "1238321", "postDate": "03/14/2021 22:01:11", "content": "Don't get me wrong, I'm neither angry nor frustrated by the fact that @yasufuminakama shared his code!! I'm sure it will help a lot of people and that's the beauty of being part of this wonderful community :)
\nI'm trying to put myself in someone else's shoes and imagine his frustration at being passed by people copying a notebook without doing anything.
\nObviously it's part of the game here in Kaggle, as you said it's just the beginning and there's room for improvement, it's not the end of the world. However if the aim was simply to provide beginners with ideas on how to approach the problem, probably a win-win approach could have been simply to discuss here about the implementation, just as they did here.
Btw, there are multiple competitions available for beginners to get started, as well as plenty of open notebooks from past competitions to learn
", "rawMarkdown": "Don't get me wrong, I'm neither angry nor frustrated by the fact that @yasufuminakama shared his code!! I'm sure it will help a lot of people and that's the beauty of being part of this wonderful community :)\nI'm trying to put myself in someone else's shoes and imagine his frustration at being passed by people copying a notebook without doing anything. \nObviously it's part of the game here in Kaggle, as you said it's just the beginning and there's room for improvement, it's not the end of the world. However if the aim was simply to provide beginners with ideas on how to approach the problem, probably a win-win approach could have been simply to discuss here about the implementation, just as they did [here](https://www.kaggle.com/c/bms-molecular-translation/discussion/224257).\n\nBtw, there are multiple competitions available for beginners to get started, as well as plenty of open notebooks from past competitions to learn", "votes": null }, { "id": "1238783", "postDate": "03/15/2021 09:39:53", "content": "I always enjoy your Attention starter discussions! They are very fruitful and helpful for all of us.
\nWouldn't be smarter to simply clarify the work process in the conversation without distributing the real code which you do, as many can have plagiarism.
\nBeing a starter, I learn from you rather than copying from your, but others don't do so🤷♂️ @yasufuminakama
Enjoying reading u, thanks a lot, your discussion and your notebooks are useful
", "rawMarkdown": "Enjoying reading u, thanks a lot, your discussion and your notebooks are useful", "votes": null }, { "id": "1239483", "postDate": "03/15/2021 18:52:49", "content": "Thank you for sharing. This is a fascinating competition. Cross domain (i.e., image to text) problems get to the heart of what true human-like understanding is. I was aware of the Show, Attend, and Tell paper, but prior to you sharing your work it was too dense for me to work with directly.
\nCan I ask, how long did you train the model for to get to CV 18.7?
", "rawMarkdown": "Thank you for sharing. This is a fascinating competition. Cross domain (i.e., image to text) problems get to the heart of what true human-like understanding is. I was aware of the Show, Attend, and Tell paper, but prior to you sharing your work it was too dense for me to work with directly. \n\nCan I ask, how long did you train the model for to get to CV 18.7?", "votes": null }, { "id": "1239700", "postDate": "03/15/2021 23:53:08", "content": "As I wrote, it's 2 epochs, and it took about 7.5h per epoch using kaggle notebooks as you can see training notebook.
", "rawMarkdown": "As I wrote, it's 2 epochs, and it took about 7.5h per epoch using kaggle notebooks as you can see training notebook.", "votes": null }, { "id": "1239747", "postDate": "03/16/2021 01:20:23", "content": "\" being passed by people copying a notebook without doing anything.\"
\nthat is part of kaggle and part of life.
\none can then speed his work by learning from the note book and improve on it. then you can overtake yourself.
\n\"probably a win-win approach\"
\nunfortunately, the devils are in the details and not in the methods. there is a huge difference with and without the code.
my suggestion is:
\none may one to think about, which is more important:
\nthis should work. you can do it again!
\n
@hengck23
\nGlad to see you in this competition!
\nYes, I was thinking this kind of approach, thanks for kindly visualization :)
When RANZCR CLiP is over today and the usual suspects arrive here you will see this baseline start to fall on the leaderboard…
", "rawMarkdown": "When RANZCR CLiP is over today and the usual suspects arrive here you will see this baseline start to fall on the leaderboard...", "votes": null }, { "id": "1243100", "postDate": "03/18/2021 02:45:59", "content": "I think @yasufuminakama has proven that he doesn't need to be worried about people forking his code. That being said, @yasufuminakama thank you for the clearly explained code. Very helpful in making the \"Show, Attend and Tell\" paper understandable!
", "rawMarkdown": "I think @yasufuminakama has proven that he doesn't need to be worried about people forking his code. That being said, @yasufuminakama thank you for the clearly explained code. Very helpful in making the \"Show, Attend and Tell\" paper understandable!", "votes": null }, { "id": "1244269", "postDate": "03/18/2021 21:06:30", "content": "@hengck23 Thanks for the suggestion.
\n@yasufuminakama Can I assume from your jump to the top of the leaderboard, that the teacher-student approach worked very well?!
", "rawMarkdown": "hengck23 Thanks for the suggestion.\n\n@yasufuminakama Can I assume from your jump to the top of the leaderboard, that the teacher-student approach worked very well?!", "votes": null }, { "id": "1244697", "postDate": "03/19/2021 06:49:37", "content": "I haven't tried it yet, I want to know the result if someone tried it :)
", "rawMarkdown": "I haven't tried it yet, I want to know the result if someone tried it :)", "votes": null }, { "id": "1245187", "postDate": "03/19/2021 14:50:19", "content": "How might one go about making the training code parallel across multiple GPUs? I've tried the following idea in the \"training\" notebook:
\n :\n :\n encoder = Encoder(CFG.model_name, pretrained=True)\n if torch.cuda.device_count() > 1:\n print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n encoder = nn.DataParallel(encoder)\n encoder.to(device)\n :\n :\nand the same with the decoder. When I run training I can see the memory use go up evenly across all GPUs so something is working. However, I get the following error:
RuntimeError: Gather got an input of invalid size: got [64, 158, 193], but expected [64, 154, 193] (gather at /opt/conda/conda-bld/pytorch_1579022034529/work/torch/csrc/cuda/comm.cpp:231)\nThe batch size is an even multiple of the number of GPUs so that's fine.
", "rawMarkdown": "How might one go about making the training code parallel across multiple GPUs? I've tried the following idea in the \"training\" notebook:\n\n```\n :\n :\n encoder = Encoder(CFG.model_name, pretrained=True)\n if torch.cuda.device_count() > 1:\n print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n encoder = nn.DataParallel(encoder)\n encoder.to(device)\n :\n :\n```\n\nand the same with the `decoder`. When I run training I can see the memory use go up evenly across all GPUs so something is working. However, I get the following error:\n\n```\nRuntimeError: Gather got an input of invalid size: got [64, 158, 193], but expected [64, 154, 193] (gather at /opt/conda/conda-bld/pytorch_1579022034529/work/torch/csrc/cuda/comm.cpp:231)\n```\n\nThe batch size is an even multiple of the number of GPUs so that's fine.", "votes": null }, { "id": "1245263", "postDate": "03/19/2021 16:18:45", "content": "I haven't tried multiple GPUs training so I'm not sure but it seems label sequence length is not same.
", "rawMarkdown": "I haven't tried multiple GPUs training so I'm not sure but it seems label sequence length is not same.", "votes": null }, { "id": "1245300", "postDate": "03/19/2021 16:54:06", "content": "I found some clues here
\n\n", "rawMarkdown": "I found some clues [here](https://pytorch.org/docs/stable/notes/faq.html#my-recurrent-network-doesn-t-work-with-data-parallelism)\n\n> There is a subtlety in using the pack sequence -> recurrent network -> unpack sequence pattern in a Module with DataParallel or data_parallel(). Input to each the forward() on each device will only be part of the entire input. Because the unpack operation torch.nn.utils.rnn.pad_packed_sequence() by default only pads up to the longest input it sees, i.e., the longest on that particular device, size mismatches will happen when results are gathered together. Therefore, you can instead take advantage of the total_length argument of pad_packed_sequence() to make sure that the forward() calls return sequences of same length.", "votes": null }, { "id": "1246228", "postDate": "03/20/2021 14:59:40", "content": "There is a subtlety in using the pack sequence -> recurrent network -> unpack sequence pattern in a Module with DataParallel or data_parallel(). Input to each the forward() on each device will only be part of the entire input. Because the unpack operation torch.nn.utils.rnn.pad_packed_sequence() by default only pads up to the longest input it sees, i.e., the longest on that particular device, size mismatches will happen when results are gathered together. Therefore, you can instead take advantage of the total_length argument of pad_packed_sequence() to make sure that the forward() calls return sequences of same length.
\n
How do you think about the different between CV and LB ?
\nThe estimation of image orientation (w>h or w<h) is not perfect, is it ?
decode_lengths = (caption_lengths - 1).tolist()the_number_of_gpu = torch.cuda.device_count()\n if the_number_of_gpu > 1:\n encoder = nn.DataParallel(encoder)\n # decoder = nn.DataParallel(decoder) <--- commented\nI haven't used nn.DataParallel, but nn.parallel.DistributedDataParallel doesn't throw any error.
Not the author=) But I can add few cents… They gap between CV and LB is greatly reduced if you add 90 degree rotations augmentations during training. (or perhaps any rotation)
IMHO, I don't mind gap if there is correlation between CV and LB.
", "rawMarkdown": "IMHO, I don't mind gap if there is correlation between CV and LB.", "votes": null }, { "id": "1246835", "postDate": "03/21/2021 06:48:57", "content": "Yes, torch states that DistributedDataParallel (DDP) works faster and better than DataParallel like it is stated here.
\n@drhabib if you would create a kernel with DDP based on @yasufuminakama code, lots of kagglers would be happy as DDP requires some additional implementations. :)
Thanks @drhabib for the link to nn.parallel.DistributedDataParallel. That looks very nice. Will try…
\n\n@drhabib if you would create a kernel with DDP based on @yasufuminakama code, lots of kagglers would be happy as DDP requires some additional implementations. :)
\n
I would hit the upvote so hard that I think it would break my mouse… :-)
", "rawMarkdown": "Thanks @drhabib for the link to `nn.parallel.DistributedDataParallel`. That looks very nice. Will try... \n\n> @drhabib if you would create a kernel with DDP based on @yasufuminakama code, lots of kagglers would be happy as DDP requires some additional implementations. :)\n\nI would hit the upvote so hard that I think it would break my mouse... :-)", "votes": null }, { "id": "1249711", "postDate": "03/23/2021 13:18:44", "content": "I am struggling to get intuition about the shape of the feature tensor coming out of the encoder. This is resnet34 and we have removed the global pooling and fully connected layer and just pass down the last layer. Inspected as
\nfrom torchsummary import summary\nsummary(encoder, (3, 224, 224)\n\n----------------------------------------------------------------\n Layer (type) Output Shape Param #\n================================================================\n Conv2d-1 [-1, 64, 112, 112] 9,408\n BatchNorm2d-2 [-1, 64, 112, 112] 128\n :\n :\n :\nConv2d-119 [-1, 512, 7, 7] 2,359,296\n BatchNorm2d-120 [-1, 512, 7, 7] 1,024\n ReLU-121 [-1, 512, 7, 7] 0\n BasicBlock-122 [-1, 512, 7, 7] 0\n Identity-123 [-1, 512, 7, 7] 0\n Identity-124 [-1, 512, 7, 7] 0\n ResNet-125 [-1, 512, 7, 7] 0\nSo the shape of the feature tensor is [batch size sample index, index of feature map, x, y].
The forward method of the Encoder reshapes this feature tensor:
class Encoder(nn.Module):\n def __init__(self, model_name='resnet18', pretrained=False):\n super().__init__()\n self.cnn = timm.create_model(model_name, pretrained=pretrained)\n self.n_features = self.cnn.fc.in_features\n self.cnn.global_pool = nn.Identity()\n self.cnn.fc = nn.Identity()\n\n def forward(self, x):\n bs = x.size(0)\n features = self.cnn(x)\n features = features.permute(0, 2, 3, 1)\n return features\nfeatures.permute(0, 2, 3, 1) makes the shape of the tensor [batch size sample index, x, y, index of feature map]
I don't understand why we are doing this. I would be grateful if anyone could share some light on this! Thanks!
", "rawMarkdown": "I am struggling to get intuition about the shape of the feature tensor coming out of the encoder. This is resnet34 and we have removed the global pooling and fully connected layer and just pass down the last layer. Inspected as\n\n```\nfrom torchsummary import summary\nsummary(encoder, (3, 224, 224)\n\n----------------------------------------------------------------\n Layer (type) Output Shape Param #\n================================================================\n Conv2d-1 [-1, 64, 112, 112] 9,408\n BatchNorm2d-2 [-1, 64, 112, 112] 128\n :\n :\n :\nConv2d-119 [-1, 512, 7, 7] 2,359,296\n BatchNorm2d-120 [-1, 512, 7, 7] 1,024\n ReLU-121 [-1, 512, 7, 7] 0\n BasicBlock-122 [-1, 512, 7, 7] 0\n Identity-123 [-1, 512, 7, 7] 0\n Identity-124 [-1, 512, 7, 7] 0\n ResNet-125 [-1, 512, 7, 7] 0\n```\n\nSo the shape of the feature tensor is `[batch size sample index, index of feature map, x, y]`.\n\nThe `forward` method of the Encoder reshapes this feature tensor:\n\n```\nclass Encoder(nn.Module):\n def __init__(self, model_name='resnet18', pretrained=False):\n super().__init__()\n self.cnn = timm.create_model(model_name, pretrained=pretrained)\n self.n_features = self.cnn.fc.in_features\n self.cnn.global_pool = nn.Identity()\n self.cnn.fc = nn.Identity()\n\n def forward(self, x):\n bs = x.size(0)\n features = self.cnn(x)\n features = features.permute(0, 2, 3, 1)\n return features\n```\n\n`features.permute(0, 2, 3, 1)` makes the shape of the tensor `[batch size sample index, x, y, index of feature map]`\n\nI don't understand why we are doing this. I would be grateful if anyone could share some light on this! Thanks!", "votes": null }, { "id": "1249735", "postDate": "03/23/2021 13:36:42", "content": "In general you can debug Pytorch code by just putting print statements in the nn.model class
class Encoder(nn.Module):\n def __init__(self, model_name='resnet18', pretrained=False):\n super().__init__()\n self.cnn = timm.create_model(model_name, pretrained=pretrained)\n self.n_features = self.cnn.fc.in_features\n self.cnn.global_pool = nn.Identity()\n self.cnn.fc = nn.Identity()\n def forward(self, x):\n bs = x.size(0)\n features = self.cnn(x)\n print (features.shape)\n print (f'Shape before permutation {features.shape}')\n features = features.permute(0, 2, 3, 1)\n print (f'Shape after permutation {features.shape}')\n return features\n#put model in eval mode\nmodel = Encoder().eval()\nbs = 4\nc = 3\nh = 224\nw = 224\nimage = torch.rand(bs, c, h, w)\n#getting features and printing statement \n#no_grad just tells that we don't want to calculate gradient \nwith torch.no_grad():\n features = model(image)\noutput should look like:
\nShape before permutation torch.Size([4, 512, 7, 7])\nShape after permutation torch.Size([4, 7, 7, 512])\ntorch.permute just helps to switch features axis… Later we will reshape this features more in decoder to (bs, 7 *7, 512) so we can apply attention and initiate LSTM cell
\nNot sure I answered your question =)
It's very detailed here … from the author
", "rawMarkdown": "It's very detailed [here](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning) ... from the author", "votes": null }, { "id": "1249758", "postDate": "03/23/2021 13:51:09", "content": "ahhh perfect! thanks for the link =)
", "rawMarkdown": "ahhh perfect! thanks for the link =)", "votes": null }, { "id": "1249766", "postDate": "03/23/2021 13:59:46", "content": "Thanks! I concur on the output shapes. That's what I intended to communicate. I understand what permute is doing technically. What I am asking is why we have to do this.
\nPerhaps this is the answer
\n\n\nLater we will reshape this features more in decoder to (bs, 7 *7, 512) so we can apply attention and initiate LSTM cell
\n
Can you point in the code where this is happening? I could understand it if the 7x7 feature map was flattened to a vector, but still not getting this. 😢
", "rawMarkdown": "Thanks! I concur on the output shapes. That's what I intended to communicate. I understand what permute is doing technically. What I am asking is **why** we have to do this.\n\nPerhaps this is the answer\n\n> Later we will reshape this features more in decoder to (bs, 7 *7, 512) so we can apply attention and initiate LSTM cell\n\nCan you point in the code where this is happening? I could understand it if the 7x7 feature map was flattened to a vector, but still not getting this. :cry:", "votes": null }, { "id": "1249770", "postDate": "03/23/2021 14:02:57", "content": "\n\nIt's very detailed here … from the author
\n
That link just makes things more confusing to me, because in his README he is not
\npermuting the feature tensor. For example, see
![\"\"](\"https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning/raw/master/img/decoder_att.png\")
So here, we maintain the shape of the tensor in the original CNN.
\nEDIT:
\nHaving read more closely the README.md in this repo, he does talk about reshaping the tensor
\nhttps://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning#attention-1
\n", "rawMarkdown": "> It's very detailed here … from the author\n\nThat link just makes things **more** confusing to me, because in his README he is **not** \npermuting the feature tensor. For example, see\n\n\n\n\nSo here, we maintain the shape of the tensor in the original CNN.\n\nEDIT:\n\nHaving read more closely the README.md in this repo, he does talk about reshaping the tensor\nhttps://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning#attention-1\n\n> The output of the Encoder is received here and flattened to dimensions N, 14 * 14, 2048. This is just convenient and prevents having to reshape the tensor multiple times.", "votes": null }, { "id": "1249774", "postDate": "03/23/2021 14:04:34", "content": "The output of the Encoder is received here and flattened to dimensions N, 14 * 14, 2048. This is just convenient and prevents having to reshape the tensor multiple times.
\n
https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter
\nclass DecoderWithAttention(nn.Module):
encoder_out = encoder_out.view(batch_size, -1, encoder_dim) # (batch_size, num_pixels, encoder_dim)\n num_pixels = encoder_out.size(1)\n# (bs, 7, 7, 512)\n#encoder_dim = 512\nencoder_out.view(batch_size, -1, encoder_dim) #(bs, 7 * 7, encoder_dim) (-1 will flatten axis 1 and 2)\n\n\n\n
encoder_out.view(batch_size, -1, encoder_dim) #(bs, 7 * 7, encoder_dim) (-1 will flatten axis 1 and 2)
That's it!! Thank you!
", "rawMarkdown": "> `encoder_out.view(batch_size, -1, encoder_dim) #(bs, 7 * 7, encoder_dim) (-1 will flatten axis 1 and 2)`\n\nThat's it!! Thank you!", "votes": null }, { "id": "1251592", "postDate": "03/24/2021 23:36:01", "content": "I have tried it. It seems that the simple annotations in the rdkit generated images are not enough for the teacher model to pick up on: stage 1 performance is only marginally better than training with the original images. Perhaps I need to use a stronger encoder to see more significant results. I will share my notebooks this weekend when I have some GPU and hopefully a more experienced Kaggler can find a way of improving it. (I think I just need to generate more annotations via rdkit to make it work i.e. highlight bonds / atoms).
Thanks for sharing your result!
", "rawMarkdown": "Thanks for sharing your result!", "votes": null }, { "id": "1251623", "postDate": "03/25/2021 00:39:22", "content": "I think this level of annotation would be better:
\n![\"\"](\"https://www.rdkit.org/docs/_images/atom_highlights_3.png\")
Will try it next and report results.
\nEdit:
\nSome useful code can be found here. We can use SMART queries to highlight arbitrary substructures of the chemicals. I don't know how to best exploit this, as I know very little about chemistry - I'm a physicist :D
For example, we can pass SMART queries like:
\n[\"B\",\"Br\",\"C\",\"Cl\",\"F\",\"I\",\"N\",\"O\",\"P\",\"S\"]
\nto draw circles around the atoms and we can pass queries like:
\n[\"[r{3-}]\", \"*~*~*\"]
\nto highlight rings of radius 3 or greater and 3 linear atoms respectively.
Why you don't care if there is correlation between CV and LB? Can you explain it?
", "rawMarkdown": "Why you don't care if there is correlation between CV and LB? Can you explain it?", "votes": null }, { "id": "1252095", "postDate": "03/25/2021 12:22:36", "content": "I found einops very helpful for this type of code.
\nx = rearrange(x, 'b c h w -> b (h w) c') is very readable.
I think there must be gap between Single-Fold CV and LB for most people, but I think if we perform N-fold ensemble, I think CV is much closer to LB. I haven't tried it, Just my intuition :)
", "rawMarkdown": "I think there must be gap between Single-Fold CV and LB for most people, but I think if we perform N-fold ensemble, I think CV is much closer to LB. I haven't tried it, Just my intuition :)", "votes": null }, { "id": "1252164", "postDate": "03/25/2021 13:21:52", "content": "einops looks amazing! Very elegant and clear code. Thanks.
Thank you @yasufuminakama for sharing. This is very helpful.
", "rawMarkdown": "Thank you @yasufuminakama for sharing. This is very helpful.", "votes": null }, { "id": "1260702", "postDate": "04/02/2021 10:58:05", "content": "some good explanation here: https://medium.com/analytics-vidhya/image-captioning-with-attention-part-1-e8a5f783f6d3
", "rawMarkdown": "some good explanation here: https://medium.com/analytics-vidhya/image-captioning-with-attention-part-1-e8a5f783f6d3", "votes": null }, { "id": "1271143", "postDate": "04/12/2021 10:39:01", "content": "Wow! This is going to be really helpful for someone like me who has just started to learn about RNNs. Thank you for sharing. 🙏🏼
", "rawMarkdown": "Wow! This is going to be really helpful for someone like me who has just started to learn about RNNs. Thank you for sharing. 🙏🏼", "votes": null }, { "id": "1274464", "postDate": "04/15/2021 10:15:44", "content": "Good work!
", "rawMarkdown": "Good work!", "votes": null }, { "id": "1284992", "postDate": "04/26/2021 13:29:20", "content": "I'm joining this comp a month later and this score is way down the leaderboard. As op points out, this is what normally happens in these comps. The code has been super useful in getting me started on this comp
", "rawMarkdown": "I'm joining this comp a month later and this score is way down the leaderboard. As op points out, this is what normally happens in these comps. The code has been super useful in getting me started on this comp", "votes": null }, { "id": "2747540", "postDate": "04/12/2024 00:10:19", "content": "Super useful!
", "rawMarkdown": "Super useful!", "votes": null }, { "id": "3196935", "postDate": "05/07/2025 16:28:26", "content": "Great Work!
", "rawMarkdown": "Great Work!", "votes": null }, { "id": "3306415", "postDate": "10/24/2025 13:59:44", "content": "Nicely explained
", "rawMarkdown": "Nicely explained", "votes": null } ], "comments": [ { "id": 1237740, "author_name": "jacoporepossi", "author_url": "", "post_date": "03/14/2021 12:15:32", "content": "Wouldn't be better to just explain the workflow in the discussion without publishing the actual code?
\nI can already see 2 scores exactly at 21.9 like yours at 5th and 6th positions…
I understand what you say, but it's easier to join the competition for beginners if there is starter code.
\nWhen I was beginner, I learned a lot from public notebooks, that's why I publish starter codes for other competitions too.
\nYou don't need to be so nervous about current LB position, it's just beginning of the competition. There are a lot of things to improve score.
I agree with you, I learned a lot from public notebooks in my previous competition (MoA) but the problem with beginners is that they tend to deceive themselves with copying and submitting the notebook without trying to understand the code (even at the beginning of the competition) and that's bad but they don't realize it. Also, they are people who try their best to learn and improve their scores and that is really demotivating for people who try their best.
\nI personally never copy past code that I don't understand, and my message to @jacoporepossi
\nis to not give up and like @yasufuminakama said it's the beginning of the competition and there is room for improvement and thank you for sharing your notebook with us, very helpful.
By the way, How can we retrain models when we surpass the 9 hour limit of the Kaggle kernel?
", "votes": null, "replies": [] }, { "id": 1238286, "author_name": "yasufuminakama", "author_url": "", "post_date": "03/14/2021 20:35:03", "content": "You can reload saved model states, optimizer states, scheduler states, and restart training.
", "votes": null, "replies": [] }, { "id": 1238321, "author_name": "jacoporepossi", "author_url": "", "post_date": "03/14/2021 22:01:11", "content": "Don't get me wrong, I'm neither angry nor frustrated by the fact that @yasufuminakama shared his code!! I'm sure it will help a lot of people and that's the beauty of being part of this wonderful community :)
\nI'm trying to put myself in someone else's shoes and imagine his frustration at being passed by people copying a notebook without doing anything.
\nObviously it's part of the game here in Kaggle, as you said it's just the beginning and there's room for improvement, it's not the end of the world. However if the aim was simply to provide beginners with ideas on how to approach the problem, probably a win-win approach could have been simply to discuss here about the implementation, just as they did here.
Btw, there are multiple competitions available for beginners to get started, as well as plenty of open notebooks from past competitions to learn
", "votes": null, "replies": [] }, { "id": 1239747, "author_name": "hengck23", "author_url": "", "post_date": "03/16/2021 01:20:23", "content": "\" being passed by people copying a notebook without doing anything.\"
\nthat is part of kaggle and part of life.
\none can then speed his work by learning from the note book and improve on it. then you can overtake yourself.
\n\"probably a win-win approach\"
\nunfortunately, the devils are in the details and not in the methods. there is a huge difference with and without the code.
my suggestion is:
\none may one to think about, which is more important:
\nWhen RANZCR CLiP is over today and the usual suspects arrive here you will see this baseline start to fall on the leaderboard…
", "votes": null, "replies": [] }, { "id": 1243100, "author_name": "jjinho", "author_url": "", "post_date": "03/18/2021 02:45:59", "content": "I think @yasufuminakama has proven that he doesn't need to be worried about people forking his code. That being said, @yasufuminakama thank you for the clearly explained code. Very helpful in making the \"Show, Attend and Tell\" paper understandable!
", "votes": null, "replies": [] }, { "id": 1284992, "author_name": "alexandersoare", "author_url": "", "post_date": "04/26/2021 13:29:20", "content": "I'm joining this comp a month later and this score is way down the leaderboard. As op points out, this is what normally happens in these comps. The code has been super useful in getting me started on this comp
", "votes": null, "replies": [] } ] }, { "id": 1238783, "author_name": "arpit3043", "author_url": "", "post_date": "03/15/2021 09:39:53", "content": "I always enjoy your Attention starter discussions! They are very fruitful and helpful for all of us.
\nWouldn't be smarter to simply clarify the work process in the conversation without distributing the real code which you do, as many can have plagiarism.
\nBeing a starter, I learn from you rather than copying from your, but others don't do so🤷♂️ @yasufuminakama
Enjoying reading u, thanks a lot, your discussion and your notebooks are useful
", "votes": null, "replies": [] }, { "id": 1239483, "author_name": "marketneutral", "author_url": "", "post_date": "03/15/2021 18:52:49", "content": "Thank you for sharing. This is a fascinating competition. Cross domain (i.e., image to text) problems get to the heart of what true human-like understanding is. I was aware of the Show, Attend, and Tell paper, but prior to you sharing your work it was too dense for me to work with directly.
\nCan I ask, how long did you train the model for to get to CV 18.7?
", "votes": null, "replies": [ { "id": 1239700, "author_name": "yasufuminakama", "author_url": "", "post_date": "03/15/2021 23:53:08", "content": "As I wrote, it's 2 epochs, and it took about 7.5h per epoch using kaggle notebooks as you can see training notebook.
", "votes": null, "replies": [] } ] }, { "id": 1239763, "author_name": "hengck23", "author_url": "", "post_date": "03/16/2021 02:01:13", "content": "\nthis should work. you can do it again!
\n
@hengck23
\nGlad to see you in this competition!
\nYes, I was thinking this kind of approach, thanks for kindly visualization :)
@hengck23 Thanks for the suggestion.
\n@yasufuminakama Can I assume from your jump to the top of the leaderboard, that the teacher-student approach worked very well?!
", "votes": null, "replies": [] }, { "id": 1244697, "author_name": "yasufuminakama", "author_url": "", "post_date": "03/19/2021 06:49:37", "content": "I haven't tried it yet, I want to know the result if someone tried it :)
", "votes": null, "replies": [] }, { "id": 1251592, "author_name": "tuckerarrants", "author_url": "", "post_date": "03/24/2021 23:36:01", "content": "I have tried it. It seems that the simple annotations in the rdkit generated images are not enough for the teacher model to pick up on: stage 1 performance is only marginally better than training with the original images. Perhaps I need to use a stronger encoder to see more significant results. I will share my notebooks this weekend when I have some GPU and hopefully a more experienced Kaggler can find a way of improving it. (I think I just need to generate more annotations via rdkit to make it work i.e. highlight bonds / atoms).
Thanks for sharing your result!
", "votes": null, "replies": [] }, { "id": 1251623, "author_name": "tuckerarrants", "author_url": "", "post_date": "03/25/2021 00:39:22", "content": "I think this level of annotation would be better:
\n
Will try it next and report results.
\nEdit:
\nSome useful code can be found here. We can use SMART queries to highlight arbitrary substructures of the chemicals. I don't know how to best exploit this, as I know very little about chemistry - I'm a physicist :D
For example, we can pass SMART queries like:
\n[\"B\",\"Br\",\"C\",\"Cl\",\"F\",\"I\",\"N\",\"O\",\"P\",\"S\"]
\nto draw circles around the atoms and we can pass queries like:
\n[\"[r{3-}]\", \"*~*~*\"]
\nto highlight rings of radius 3 or greater and 3 linear atoms respectively.
How might one go about making the training code parallel across multiple GPUs? I've tried the following idea in the \"training\" notebook:
\n :\n :\n encoder = Encoder(CFG.model_name, pretrained=True)\n if torch.cuda.device_count() > 1:\n print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n encoder = nn.DataParallel(encoder)\n encoder.to(device)\n :\n :\nand the same with the decoder. When I run training I can see the memory use go up evenly across all GPUs so something is working. However, I get the following error:
RuntimeError: Gather got an input of invalid size: got [64, 158, 193], but expected [64, 154, 193] (gather at /opt/conda/conda-bld/pytorch_1579022034529/work/torch/csrc/cuda/comm.cpp:231)\nThe batch size is an even multiple of the number of GPUs so that's fine.
", "votes": null, "replies": [ { "id": 1245263, "author_name": "yasufuminakama", "author_url": "", "post_date": "03/19/2021 16:18:45", "content": "I haven't tried multiple GPUs training so I'm not sure but it seems label sequence length is not same.
", "votes": null, "replies": [] }, { "id": 1245300, "author_name": "marketneutral", "author_url": "", "post_date": "03/19/2021 16:54:06", "content": "I found some clues here
\n\n", "votes": null, "replies": [] }, { "id": 1246476, "author_name": "ubique", "author_url": "", "post_date": "03/20/2021 19:09:39", "content": "There is a subtlety in using the pack sequence -> recurrent network -> unpack sequence pattern in a Module with DataParallel or data_parallel(). Input to each the forward() on each device will only be part of the entire input. Because the unpack operation torch.nn.utils.rnn.pad_packed_sequence() by default only pads up to the longest input it sees, i.e., the longest on that particular device, size mismatches will happen when results are gathered together. Therefore, you can instead take advantage of the total_length argument of pad_packed_sequence() to make sure that the forward() calls return sequences of same length.
\n
the actual problem is with this line
\ndecode_lengths = (caption_lengths - 1).tolist()
\nin multi-gpu it splits into 4 lists so they get different lengths and on gathering you receive an error.
\nFast and dirty way is to run in parallel encoder, and decoder - not, in other words decoder runs on one gpu.
\nthe_number_of_gpu = torch.cuda.device_count()\n if the_number_of_gpu > 1:\n encoder = nn.DataParallel(encoder)\n # decoder = nn.DataParallel(decoder) <--- commented\n
\nin this case gpus won't be used optimally, unfortunately
I haven't used nn.DataParallel, but nn.parallel.DistributedDataParallel doesn't throw any error.
Yes, torch states that DistributedDataParallel (DDP) works faster and better than DataParallel like it is stated here.
\n@drhabib if you would create a kernel with DDP based on @yasufuminakama code, lots of kagglers would be happy as DDP requires some additional implementations. :)
Thanks @drhabib for the link to nn.parallel.DistributedDataParallel. That looks very nice. Will try…
\n\n@drhabib if you would create a kernel with DDP based on @yasufuminakama code, lots of kagglers would be happy as DDP requires some additional implementations. :)
\n
I would hit the upvote so hard that I think it would break my mouse… :-)
", "votes": null, "replies": [] } ] }, { "id": 1246228, "author_name": "toshik", "author_url": "", "post_date": "03/20/2021 14:59:40", "content": "How do you think about the different between CV and LB ?
\nThe estimation of image orientation (w>h or w<h) is not perfect, is it ?
Not the author=) But I can add few cents… They gap between CV and LB is greatly reduced if you add 90 degree rotations augmentations during training. (or perhaps any rotation)
IMHO, I don't mind gap if there is correlation between CV and LB.
", "votes": null, "replies": [] }, { "id": 1252076, "author_name": "fuxungao", "author_url": "", "post_date": "03/25/2021 12:09:14", "content": "Why you don't care if there is correlation between CV and LB? Can you explain it?
", "votes": null, "replies": [] }, { "id": 1252134, "author_name": "yasufuminakama", "author_url": "", "post_date": "03/25/2021 12:53:07", "content": "I think there must be gap between Single-Fold CV and LB for most people, but I think if we perform N-fold ensemble, I think CV is much closer to LB. I haven't tried it, Just my intuition :)
", "votes": null, "replies": [] } ] }, { "id": 1249711, "author_name": "marketneutral", "author_url": "", "post_date": "03/23/2021 13:18:44", "content": "I am struggling to get intuition about the shape of the feature tensor coming out of the encoder. This is resnet34 and we have removed the global pooling and fully connected layer and just pass down the last layer. Inspected as
\nfrom torchsummary import summary\nsummary(encoder, (3, 224, 224)\n\n----------------------------------------------------------------\n Layer (type) Output Shape Param #\n================================================================\n Conv2d-1 [-1, 64, 112, 112] 9,408\n BatchNorm2d-2 [-1, 64, 112, 112] 128\n :\n :\n :\nConv2d-119 [-1, 512, 7, 7] 2,359,296\n BatchNorm2d-120 [-1, 512, 7, 7] 1,024\n ReLU-121 [-1, 512, 7, 7] 0\n BasicBlock-122 [-1, 512, 7, 7] 0\n Identity-123 [-1, 512, 7, 7] 0\n Identity-124 [-1, 512, 7, 7] 0\n ResNet-125 [-1, 512, 7, 7] 0\nSo the shape of the feature tensor is [batch size sample index, index of feature map, x, y].
The forward method of the Encoder reshapes this feature tensor:
class Encoder(nn.Module):\n def __init__(self, model_name='resnet18', pretrained=False):\n super().__init__()\n self.cnn = timm.create_model(model_name, pretrained=pretrained)\n self.n_features = self.cnn.fc.in_features\n self.cnn.global_pool = nn.Identity()\n self.cnn.fc = nn.Identity()\n\n def forward(self, x):\n bs = x.size(0)\n features = self.cnn(x)\n features = features.permute(0, 2, 3, 1)\n return features\nfeatures.permute(0, 2, 3, 1) makes the shape of the tensor [batch size sample index, x, y, index of feature map]
I don't understand why we are doing this. I would be grateful if anyone could share some light on this! Thanks!
", "votes": null, "replies": [ { "id": 1249735, "author_name": "drhabib", "author_url": "", "post_date": "03/23/2021 13:36:42", "content": "In general you can debug Pytorch code by just putting print statements in the nn.model class
class Encoder(nn.Module):\n def __init__(self, model_name='resnet18', pretrained=False):\n super().__init__()\n self.cnn = timm.create_model(model_name, pretrained=pretrained)\n self.n_features = self.cnn.fc.in_features\n self.cnn.global_pool = nn.Identity()\n self.cnn.fc = nn.Identity()\n def forward(self, x):\n bs = x.size(0)\n features = self.cnn(x)\n print (features.shape)\n print (f'Shape before permutation {features.shape}')\n features = features.permute(0, 2, 3, 1)\n print (f'Shape after permutation {features.shape}')\n return features\n#put model in eval mode\nmodel = Encoder().eval()\nbs = 4\nc = 3\nh = 224\nw = 224\nimage = torch.rand(bs, c, h, w)\n#getting features and printing statement \n#no_grad just tells that we don't want to calculate gradient \nwith torch.no_grad():\n features = model(image)\noutput should look like:
\nShape before permutation torch.Size([4, 512, 7, 7])\nShape after permutation torch.Size([4, 7, 7, 512])\ntorch.permute just helps to switch features axis… Later we will reshape this features more in decoder to (bs, 7 *7, 512) so we can apply attention and initiate LSTM cell
\nNot sure I answered your question =)
It's very detailed here … from the author
", "votes": null, "replies": [] }, { "id": 1249758, "author_name": "drhabib", "author_url": "", "post_date": "03/23/2021 13:51:09", "content": "ahhh perfect! thanks for the link =)
", "votes": null, "replies": [] }, { "id": 1249766, "author_name": "marketneutral", "author_url": "", "post_date": "03/23/2021 13:59:46", "content": "Thanks! I concur on the output shapes. That's what I intended to communicate. I understand what permute is doing technically. What I am asking is why we have to do this.
\nPerhaps this is the answer
\n\n\nLater we will reshape this features more in decoder to (bs, 7 *7, 512) so we can apply attention and initiate LSTM cell
\n
Can you point in the code where this is happening? I could understand it if the 7x7 feature map was flattened to a vector, but still not getting this. 😢
", "votes": null, "replies": [] }, { "id": 1249770, "author_name": "marketneutral", "author_url": "", "post_date": "03/23/2021 14:02:57", "content": "\n\nIt's very detailed here … from the author
\n
That link just makes things more confusing to me, because in his README he is not
\npermuting the feature tensor. For example, see
So here, we maintain the shape of the tensor in the original CNN.
\nEDIT:
\nHaving read more closely the README.md in this repo, he does talk about reshaping the tensor
\nhttps://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning#attention-1
\n", "votes": null, "replies": [] }, { "id": 1249774, "author_name": "drhabib", "author_url": "", "post_date": "03/23/2021 14:04:34", "content": "The output of the Encoder is received here and flattened to dimensions N, 14 * 14, 2048. This is just convenient and prevents having to reshape the tensor multiple times.
\n
https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter
\nclass DecoderWithAttention(nn.Module):
encoder_out = encoder_out.view(batch_size, -1, encoder_dim) # (batch_size, num_pixels, encoder_dim)\n num_pixels = encoder_out.size(1)\n# (bs, 7, 7, 512)\n#encoder_dim = 512\nencoder_out.view(batch_size, -1, encoder_dim) #(bs, 7 * 7, encoder_dim) (-1 will flatten axis 1 and 2)\n\n\n\n
encoder_out.view(batch_size, -1, encoder_dim) #(bs, 7 * 7, encoder_dim) (-1 will flatten axis 1 and 2)
That's it!! Thank you!
", "votes": null, "replies": [] }, { "id": 1252095, "author_name": "ryanzhang", "author_url": "", "post_date": "03/25/2021 12:22:36", "content": "I found einops very helpful for this type of code.
\nx = rearrange(x, 'b c h w -> b (h w) c') is very readable.
einops looks amazing! Very elegant and clear code. Thanks.
Thank you @yasufuminakama for sharing. This is very helpful.
", "votes": null, "replies": [] }, { "id": 1260702, "author_name": "hengck23", "author_url": "", "post_date": "04/02/2021 10:58:05", "content": "some good explanation here: https://medium.com/analytics-vidhya/image-captioning-with-attention-part-1-e8a5f783f6d3
", "votes": null, "replies": [] }, { "id": 1271143, "author_name": "adityakadiwal", "author_url": "", "post_date": "04/12/2021 10:39:01", "content": "Wow! This is going to be really helpful for someone like me who has just started to learn about RNNs. Thank you for sharing. 🙏🏼
", "votes": null, "replies": [] }, { "id": 1274464, "author_name": "zavodrobotov", "author_url": "", "post_date": "04/15/2021 10:15:44", "content": "Good work!
", "votes": null, "replies": [] }, { "id": 2747540, "author_name": "itskavyagupta", "author_url": "", "post_date": "04/12/2024 00:10:19", "content": "Super useful!
", "votes": null, "replies": [] }, { "id": 3196935, "author_name": "meesamr", "author_url": "", "post_date": "05/07/2025 16:28:26", "content": "Great Work!
", "votes": null, "replies": [] }, { "id": 3306415, "author_name": "muhammadehsan02", "author_url": "", "post_date": "10/24/2025 13:59:44", "content": "Nicely explained
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1237631": "I prepared Resnet + LSTM with attention starter code.\n\n1. [preprocess](https://www.kaggle.com/yasufuminakama/inchi-preprocess-2)\n2. [training](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter)\n3. [inference](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-inference)\n\nThe result is CV: 18.7, LB: 20.3\n\nIn inference notebook, I used 2 epochs trained weight. There is much room to improve, for example more epochs, augmentation, larger models, larger size...\n\nHope this helps, happy kaggling!", "1237740": "Wouldn't be better to just explain the workflow in the discussion without publishing the actual code?\nI can already see 2 scores exactly at 21.9 like yours at 5th and 6th positions...", "1237787": "I understand what you say, but it's easier to join the competition for beginners if there is starter code.\nWhen I was beginner, I learned a lot from public notebooks, that's why I publish starter codes for other competitions too. \nYou don't need to be so nervous about current LB position, it's just beginning of the competition. There are a lot of things to improve score.", "1238267": "I agree with you, I learned a lot from public notebooks in my previous competition (MoA) but the problem with beginners is that they tend to deceive themselves with copying and submitting the notebook without trying to understand the code (even at the beginning of the competition) and that's bad but they don't realize it. Also, they are people who try their best to learn and improve their scores and that is really demotivating for people who try their best.\n\nI personally never copy past code that I don't understand, and my message to @jacoporepossi \nis to not give up and like @yasufuminakama said it's the beginning of the competition and there is room for improvement and thank you for sharing your notebook with us, very helpful.\n\nBy the way, How can we retrain models when we surpass the 9 hour limit of the Kaggle kernel?", "1238286": "You can reload saved model states, optimizer states, scheduler states, and restart training.", "1238321": "Don't get me wrong, I'm neither angry nor frustrated by the fact that @yasufuminakama shared his code!! I'm sure it will help a lot of people and that's the beauty of being part of this wonderful community :)\nI'm trying to put myself in someone else's shoes and imagine his frustration at being passed by people copying a notebook without doing anything. \nObviously it's part of the game here in Kaggle, as you said it's just the beginning and there's room for improvement, it's not the end of the world. However if the aim was simply to provide beginners with ideas on how to approach the problem, probably a win-win approach could have been simply to discuss here about the implementation, just as they did [here](https://www.kaggle.com/c/bms-molecular-translation/discussion/224257).\n\nBtw, there are multiple competitions available for beginners to get started, as well as plenty of open notebooks from past competitions to learn", "1238783": "I always enjoy your Attention starter discussions! They are very fruitful and helpful for all of us.\nWouldn't be smarter to simply clarify the work process in the conversation without distributing the real code which you do, as many can have plagiarism.\nBeing a starter, I learn from you rather than copying from your, but others don't do so🤷♂️ @yasufuminakama", "1239430": "Enjoying reading u, thanks a lot, your discussion and your notebooks are useful", "1239483": "Thank you for sharing. This is a fascinating competition. Cross domain (i.e., image to text) problems get to the heart of what true human-like understanding is. I was aware of the Show, Attend, and Tell paper, but prior to you sharing your work it was too dense for me to work with directly. \n\nCan I ask, how long did you train the model for to get to CV 18.7?", "1239700": "As I wrote, it's 2 epochs, and it took about 7.5h per epoch using kaggle notebooks as you can see training notebook.", "1239747": "\" being passed by people copying a notebook without doing anything.\"\n\nthat is part of kaggle and part of life.\n\none can then speed his work by learning from the note book and improve on it. then you can overtake yourself.\n\n\"probably a win-win approach\"\nunfortunately, the devils are in the details and not in the methods. there is a huge difference with and without the code.\n\n---\n\nmy suggestion is:\n- you should capitalize on code posted. e.g. run the code with different parameters and observed how accuracy changes, etc. compare the code with yours and see which part is better (or worse)\n- how to be better than those who just copy the code and do nothing? understand the code and improve on it.\n\n---\none may one to think about, which is more important:\n1. i got a medal, but I am inside my box (i.e. i am using methods developed from myself)\n2. i didn't get any medal, but l learned lots of useful techniques from others (methods that are out of my imagination ... i have stepped out of my box)", "1239763": "yasufuminakama \n\nthis should work. you can do it again!\n\n", "1239785": "hengck23 \nGlad to see you in this competition!\nYes, I was thinking this kind of approach, thanks for kindly visualization :)", "1240575": "When RANZCR CLiP is over today and the usual suspects arrive here you will see this baseline start to fall on the leaderboard...", "1243100": "I think @yasufuminakama has proven that he doesn't need to be worried about people forking his code. That being said, @yasufuminakama thank you for the clearly explained code. Very helpful in making the \"Show, Attend and Tell\" paper understandable!", "1244269": "hengck23 Thanks for the suggestion.\n\n@yasufuminakama Can I assume from your jump to the top of the leaderboard, that the teacher-student approach worked very well?!", "1244697": "I haven't tried it yet, I want to know the result if someone tried it :)", "1245187": "How might one go about making the training code parallel across multiple GPUs? I've tried the following idea in the \"training\" notebook:\n\n```\n :\n :\n encoder = Encoder(CFG.model_name, pretrained=True)\n if torch.cuda.device_count() > 1:\n print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n encoder = nn.DataParallel(encoder)\n encoder.to(device)\n :\n :\n```\n\nand the same with the `decoder`. When I run training I can see the memory use go up evenly across all GPUs so something is working. However, I get the following error:\n\n```\nRuntimeError: Gather got an input of invalid size: got [64, 158, 193], but expected [64, 154, 193] (gather at /opt/conda/conda-bld/pytorch_1579022034529/work/torch/csrc/cuda/comm.cpp:231)\n```\n\nThe batch size is an even multiple of the number of GPUs so that's fine.", "1245263": "I haven't tried multiple GPUs training so I'm not sure but it seems label sequence length is not same.", "1245300": "I found some clues [here](https://pytorch.org/docs/stable/notes/faq.html#my-recurrent-network-doesn-t-work-with-data-parallelism)\n\n> There is a subtlety in using the pack sequence -> recurrent network -> unpack sequence pattern in a Module with DataParallel or data_parallel(). Input to each the forward() on each device will only be part of the entire input. Because the unpack operation torch.nn.utils.rnn.pad_packed_sequence() by default only pads up to the longest input it sees, i.e., the longest on that particular device, size mismatches will happen when results are gathered together. Therefore, you can instead take advantage of the total_length argument of pad_packed_sequence() to make sure that the forward() calls return sequences of same length.", "1246228": "How do you think about the different between CV and LB ?\nThe estimation of image orientation (w>h or w