![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F712087%2Fd82197c21fe72cbe2171f7c3a864080f%2Ffix_loss.png?generation=1580200920211234&alt=media\")
Good evening,
\n\nfirst of all, I'd like to thank Kaggle and the hosts for this awesome challenge! It was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.
\n\nSecondly, congrats to all the winners. I locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.
\n\nMy solution is just a single TF2.0 model. It uses custom heads and a BERT transformers backbone (large version). For modeling and training I am using the great transformers library. I think that the following snippet is useful to understand the modeling:\n```python\nclass TFBertForNaturalQuestionAnswering(TFBertPreTrainedModel):\n def init(self, config, *inputs, **kwargs):\n super().init(config, *inputs, **kwargs)\n self.num_labels = config.num_labels
\n\n self.bert = TFBertMainLayer(config, name='bert')\n self.initializer = get_initializer(config.initializer_range)\n self.qa_outputs = L.Dense(config.num_labels,\n kernel_initializer=self.initializer, name='qa_outputs')\n self.long_outputs = L.Dense(1, kernel_initializer=self.initializer,\n name='long_outputs')\n\ndef call(self, inputs, **kwargs):\n outputs = self.bert(inputs, **kwargs)\n sequence_output = outputs[0]\n logits = self.qa_outputs(sequence_output)\n start_logits, end_logits = tf.split(logits, 2, axis=-1)\n start_logits = tf.squeeze(start_logits, -1)\n end_logits = tf.squeeze(end_logits, -1)\n long_logits = tf.squeeze(self.long_outputs(sequence_output), -1)\n return start_logits, end_logits, long_logits\n```
\n\nAs you can see, the natural question answering task can be treated like SQUAD-2.0 with an additional head for long answers. Note that we just need a single output: The opening tag of the HTML bounding box. I guess most competitors used a similar modeling so I think what made the difference to most other solutions is the sampling.
\n\nI changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer. I started with a really low empty answer ratio which I got from the bert-joint paper, but I couldn't reach a good score. I tuned a few hyper parameters but overall I got good results with a wide range of parameters. Adding HTML tags as custom tokens helped a bit. I also tried different start weights and found that:
bert-large-uncased (~0.70 LB) < bert-large-uncased-whole-word-masking (~0.72 LB) < bert-large-uncased-whole-word-masking-finetuned-squad (~0.73 LB).
That's about it. Thanks to @boliu0, @christofhenkel and @kentaronakanishi for fixing and providing the metric!
\n\nPlease refer to my repository for implementation details and instructions to reproduce:\n* https://github.com/see--/natural-question-answering
\n\nYou can find the 2nd place kernel and pretrained weights on Kaggle:\n* https://www.kaggle.com/seesee/submit-full\n* https://www.kaggle.com/seesee/nq-bert-uncased-68
\n\nFeel free to ask questions and / or create GitHub issues.
", "messages": [ { "id": 726961, "postDate": "2020-01-23T10:42:30.750Z", "content": "Good evening,
\n\nfirst of all, I'd like to thank Kaggle and the hosts for this awesome challenge! It was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.
\n\nSecondly, congrats to all the winners. I locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.
\n\nMy solution is just a single TF2.0 model. It uses custom heads and a BERT transformers backbone (large version). For modeling and training I am using the great transformers library. I think that the following snippet is useful to understand the modeling:\n```python\nclass TFBertForNaturalQuestionAnswering(TFBertPreTrainedModel):\n def init(self, config, *inputs, **kwargs):\n super().init(config, *inputs, **kwargs)\n self.num_labels = config.num_labels
\n\n self.bert = TFBertMainLayer(config, name='bert')\n self.initializer = get_initializer(config.initializer_range)\n self.qa_outputs = L.Dense(config.num_labels,\n kernel_initializer=self.initializer, name='qa_outputs')\n self.long_outputs = L.Dense(1, kernel_initializer=self.initializer,\n name='long_outputs')\n\ndef call(self, inputs, **kwargs):\n outputs = self.bert(inputs, **kwargs)\n sequence_output = outputs[0]\n logits = self.qa_outputs(sequence_output)\n start_logits, end_logits = tf.split(logits, 2, axis=-1)\n start_logits = tf.squeeze(start_logits, -1)\n end_logits = tf.squeeze(end_logits, -1)\n long_logits = tf.squeeze(self.long_outputs(sequence_output), -1)\n return start_logits, end_logits, long_logits\n```
\n\nAs you can see, the natural question answering task can be treated like SQUAD-2.0 with an additional head for long answers. Note that we just need a single output: The opening tag of the HTML bounding box. I guess most competitors used a similar modeling so I think what made the difference to most other solutions is the sampling.
\n\nI changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer. I started with a really low empty answer ratio which I got from the bert-joint paper, but I couldn't reach a good score. I tuned a few hyper parameters but overall I got good results with a wide range of parameters. Adding HTML tags as custom tokens helped a bit. I also tried different start weights and found that:
bert-large-uncased (~0.70 LB) < bert-large-uncased-whole-word-masking (~0.72 LB) < bert-large-uncased-whole-word-masking-finetuned-squad (~0.73 LB).
That's about it. Thanks to @boliu0, @christofhenkel and @kentaronakanishi for fixing and providing the metric!
\n\nPlease refer to my repository for implementation details and instructions to reproduce:\n* https://github.com/see--/natural-question-answering
\n\nYou can find the 2nd place kernel and pretrained weights on Kaggle:\n* https://www.kaggle.com/seesee/submit-full\n* https://www.kaggle.com/seesee/nq-bert-uncased-68
\n\nFeel free to ask questions and / or create GitHub issues.
", "rawMarkdown": "Good evening,\n\nfirst of all, I'd like to thank Kaggle and the hosts for this awesome challenge! It was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.\n\nSecondly, congrats to all the winners. I locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.\n\nMy solution is just a single TF2.0 model. It uses custom heads and a BERT transformers backbone (large version). For modeling and training I am using the great [transformers](https://github.com/huggingface/transformers) library. I think that the [following snippet](https://github.com/see--/natural-question-answering/blob/master/models.py#L8-L28) is useful to understand the modeling:\n```python\nclass TFBertForNaturalQuestionAnswering(TFBertPreTrainedModel):\n def __init__(self, config, *inputs, **kwargs):\n super().__init__(config, *inputs, **kwargs)\n self.num_labels = config.num_labels\n\n self.bert = TFBertMainLayer(config, name='bert')\n self.initializer = get_initializer(config.initializer_range)\n self.qa_outputs = L.Dense(config.num_labels,\n kernel_initializer=self.initializer, name='qa_outputs')\n self.long_outputs = L.Dense(1, kernel_initializer=self.initializer,\n name='long_outputs')\n\n def call(self, inputs, **kwargs):\n outputs = self.bert(inputs, **kwargs)\n sequence_output = outputs[0]\n logits = self.qa_outputs(sequence_output)\n start_logits, end_logits = tf.split(logits, 2, axis=-1)\n start_logits = tf.squeeze(start_logits, -1)\n end_logits = tf.squeeze(end_logits, -1)\n long_logits = tf.squeeze(self.long_outputs(sequence_output), -1)\n return start_logits, end_logits, long_logits\n```\n\nAs you can see, the natural question answering task can be treated like SQUAD-2.0 with an additional head for long answers. Note that we just need a single output: The opening tag of the HTML bounding box. I guess most competitors used a similar modeling so I think what made the difference to most other solutions is the sampling.\n\nI changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer. I started with a really low empty answer ratio which I got from the [`bert-joint` paper](https://arxiv.org/abs/1901.08634), but I couldn't reach a good score. I tuned a few hyper parameters but overall I got good results with a wide range of parameters. Adding HTML tags as custom tokens helped a bit. I also tried different start weights and found that:\n\n`bert-large-uncased` (~0.70 LB) < `bert-large-uncased-whole-word-masking` (~0.72 LB) < `bert-large-uncased-whole-word-masking-finetuned-squad` (~0.73 LB).\n\nThat's about it. Thanks to @boliu0, @christofhenkel and @kentaronakanishi for fixing and providing the metric!\n\nPlease refer to my repository for implementation details and instructions to reproduce:\n* https://github.com/see--/natural-question-answering\n\nYou can find the 2nd place kernel and pretrained weights on Kaggle:\n* https://www.kaggle.com/seesee/submit-full\n* https://www.kaggle.com/seesee/nq-bert-uncased-68\n\nFeel free to ask questions and / or create GitHub issues.", "votes": 92 }, { "id": 726994, "postDate": "2020-01-23T11:04:16.027Z", "content": "we went with
\n\n\n", "rawMarkdown": "we went with \n> taken days on my local 2 x 1080 TI machine", "votes": 3, "replies": [ { "id": 727000, "postDate": "2020-01-23T11:11:28.860Z", "content": "taken days on my local 2 x 1080 TI machine
\n
haha, I ran out of hyperparameters and ideas π
", "rawMarkdown": "haha, I ran out of hyperparameters and ideas π", "votes": 1 } ] }, { "id": 734457, "postDate": "2020-02-01T14:08:53.620Z", "content": "Congratulations See!!! You derserve it =]
", "rawMarkdown": "Congratulations See!!! You derserve it =]", "votes": 1 }, { "id": 731403, "postDate": "2020-01-28T15:43:54.543Z", "content": "Congratulations Mate.
", "rawMarkdown": "Congratulations Mate.", "votes": 1 }, { "id": 727510, "postDate": "2020-01-23T18:59:17.143Z", "content": "Your approach is clean and innovative. I am really enjoying reading it. Thank you so much for sharing your code. You're a good example of what kaggle is all about and you deserve to win the TF2.0 prize as well.
", "rawMarkdown": "Your approach is clean and innovative. I am really enjoying reading it. Thank you so much for sharing your code. You're a good example of what kaggle is all about and you deserve to win the TF2.0 prize as well.", "votes": 1, "replies": [ { "id": 728000, "postDate": "2020-01-24T10:17:22.790Z", "content": "Thanks for the kind words π
", "rawMarkdown": "Thanks for the kind words π " } ] }, { "id": 727327, "postDate": "2020-01-23T16:25:33.677Z", "content": "I have two questions. :)
\n\nCould you please share your long answer F1 and short answer F1 if you evaluated them separately in local validation? I'm wondering how much the long_outputs helps.
For the long_logits, did you treat it as a binary classification obj? Whether the token is part of the long answer or not?
1 Yes I do. The scores are (concat F1, short F1, long F1) and local score is:\n* With ~3500 empty and ~3000 non-empty validation samples: (0.563, 0.481, 0.615)\n* Only ~3000 non-empty: (0.722, 0.576, 0.823)
\n\nMaybe check this script if you are using the same metric: https://github.com/see--/natural-question-answering/blob/master/eval_server.py
\n\nThe validation set is here:\nhttps://github.com/see--/natural-question-answering/blob/master/val_ids.csv
\n\n2 No, I treat it as softmax classification. Whether the token is the opening HTML tag and predict [CLS] token if there is no long answer.
", "rawMarkdown": "1 Yes I do. The scores are (concat F1, short F1, long F1) and local score is:\n* With ~3500 empty and ~3000 non-empty validation samples: (0.563, 0.481, 0.615)\n* Only ~3000 non-empty: (0.722, 0.576, 0.823)\n\nMaybe check this script if you are using the same metric: https://github.com/see--/natural-question-answering/blob/master/eval_server.py\n\nThe validation set is here:\nhttps://github.com/see--/natural-question-answering/blob/master/val_ids.csv\n\n2 No, I treat it as softmax classification. Whether the token is the opening HTML tag and predict [CLS] token if there is no long answer.", "votes": 1 }, { "id": 727516, "postDate": "2020-01-23T19:04:15.243Z", "content": "\n\n\n2 No, I treat it as softmax classification. Whether the token is the opening HTML tag and predict [CLS] token if there is no long answer.
\n
Clever. This and your sampling ratios are big learning points for me so far.
", "rawMarkdown": "> 2 No, I treat it as softmax classification. Whether the token is the opening HTML tag and predict [CLS] token if there is no long answer.\n\nClever. This and your sampling ratios are big learning points for me so far.", "votes": 1 } ] }, { "id": 727308, "postDate": "2020-01-23T16:11:00.210Z", "content": "Big congrats! Thank you so much for the amazing sharing of the source code.
", "rawMarkdown": "Big congrats! Thank you so much for the amazing sharing of the source code.", "votes": 1 }, { "id": 727204, "postDate": "2020-01-23T14:29:53.283Z", "content": "Great Solution! Thank you for sharing the code!! :)
", "rawMarkdown": "Great Solution! Thank you for sharing the code!! :)", "votes": 1 }, { "id": 727176, "postDate": "2020-01-23T14:10:41.140Z", "content": "Congrats, nice write-up and thanks for sharing the code!
", "rawMarkdown": "Congrats, nice write-up and thanks for sharing the code!", "votes": 1 }, { "id": 726987, "postDate": "2020-01-23T10:54:34.120Z", "content": "> I changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer.
\n\nGreat catch is all I can say. Congratulations @seesee π
", "rawMarkdown": "> I changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer.\n\nGreat catch is all I can say. Congratulations @seesee π ", "votes": 1 }, { "id": 739421, "postDate": "2020-02-07T20:24:44.083Z", "content": "@seesee, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)
", "rawMarkdown": "@seesee, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)", "votes": 2 }, { "id": 730804, "postDate": "2020-01-28T00:11:15.623Z", "content": "Thanks for sharing the code @seesee ! I've been having some fun reproducing your results. Was hoping to ask a few questions about it:
\n\nCan you explain what the code is doing with crops? Seems like you are creating a text window around the true short answer for training purposes. Is that better than selecting the relevant long span among the candidates?
Your loss function (below) has a different weight for end_loss than the other components. Did you arrive at this through experimentation? Did it outperform equal-weight?
\nloss = ((tf.reduce_mean(start_loss) + tf.reduce_mean(end_loss) / 2.0) +\n tf.reduce_mean(long_loss)) / 2.0\n
Thanks,\nAlon
", "rawMarkdown": "Thanks for sharing the code @seesee ! I've been having some fun reproducing your results. Was hoping to ask a few questions about it:\n\n1. Can you explain what the code is doing with crops? Seems like you are creating a text window around the true short answer for training purposes. Is that better than selecting the relevant long span among the candidates?\n\n2. Your loss function (below) has a different weight for end_loss than the other components. Did you arrive at this through experimentation? Did it outperform equal-weight?\n \n```\nloss = ((tf.reduce_mean(start_loss) + tf.reduce_mean(end_loss) / 2.0) +\n tf.reduce_mean(long_loss)) / 2.0\n```\n\n3. Why did you choose to train for exactly 2 epochs? Would you recommend using validation every X steps as a stopping condition?\n\nThanks,\nAlon\n\n\n", "votes": 2, "replies": [ { "id": 731016, "postDate": "2020-01-28T08:54:20.183Z", "content": "1)\n> Can you explain what the code is doing with crops? Seems like you are creating a text window around the true short answer for training purposes.
\n\nYes, the crop length is set so that we get the desired ratio between empty and non-empty training samples. By using crops we remove text that would result in empty samples.
\n\n> Is that better than selecting the relevant long span among the candidates?
\n\nThis was not tested. I don't know.
\n\n2) Thank you for writing this post! It's actually a bug. At least, it's not what I thought I was using as loss. My intention was:\n```
\n\nloss = ((tf.reduce_mean(start_loss) + tf.reduce_mean(end_loss)) / 2.0 +\n tf.reduce_mean(long_loss)) / 2.0\n``\nThis is a better approximation of the metric. Long and short answers should be weighted equally! I think it's better than taking themean` of all 3 loss terms (untested). After retraining with this fixed loss, public and private LB improve (entries are sorted by private LB but truncated, thus both show 0.71):
> Why did you choose to train for exactly 2 epochs? Would you recommend using validation every X steps as a stopping condition?
\n\n[1, 2, 3, 4] were tried and 2 gave the best local LB. Personally, I would not do it. It can help, but there are more important hyper parameters (e.g. sample ratios).
", "rawMarkdown": "1)\n> Can you explain what the code is doing with crops? Seems like you are creating a text window around the true short answer for training purposes.\n\nYes, the crop length is set so that we get the desired ratio between empty and non-empty training samples. By using crops we remove text that would result in empty samples.\n\n> Is that better than selecting the relevant long span among the candidates?\n\nThis was not tested. I don't know.\n\n2) Thank you for writing this post! It's actually a bug. At least, it's not what I thought I was using as loss. My intention was:\n```\n# fixed loss\nloss = ((tf.reduce_mean(start_loss) + tf.reduce_mean(end_loss)) / 2.0 +\n tf.reduce_mean(long_loss)) / 2.0\n```\nThis is a better approximation of the metric. Long and short answers should be weighted equally! I think it's better than taking the `mean` of all 3 loss terms (untested). After retraining with this fixed loss, public and private LB improve (entries are sorted by private LB but truncated, thus both show 0.71):\n\n\n\n> Why did you choose to train for exactly 2 epochs? Would you recommend using validation every X steps as a stopping condition?\n\n[1, 2, 3, 4] were tried and 2 gave the best local LB. Personally, I would not do it. It can help, but there are more important hyper parameters (e.g. sample ratios).", "votes": 1 }, { "id": 731487, "postDate": "2020-01-28T17:31:40.280Z", "content": "\n\n\nAfter retraining with this fixed loss, public and private LB improve
\n
Haha... If only we'd partnered up, you might have made an extra few $...
\n\nThanks again for the code and answers.
\n\n-Alon
", "rawMarkdown": "> After retraining with this fixed loss, public and private LB improve\n\nHaha... If only we'd partnered up, you might have made an extra few $... \n\nThanks again for the code and answers.\n\n-Alon" } ] }, { "id": 727375, "postDate": "2020-01-23T17:07:43.187Z", "content": "Congrats to the double win with regular prize + TF2.0 prize! Such a strong performance with a single model. Thanks for sharing the code so fast.
\n\nAlso thanks for helping us debug the kernel prediction error in last week.
\n\n\n\n\nIt was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.
\n
β Cannot agree anymore.
\n\n\n\n\nI locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.
\n
β We also locked submission a fews days before deadline. I guess those TPUs can really dry up one's ideas quickly huh? We then watched us dropping every day to 24th place on public LB during last week. Not the most fun experience.
", "rawMarkdown": "Congrats to the double win with regular prize + TF2.0 prize! Such a strong performance with a single model. Thanks for sharing the code so fast.\n\nAlso thanks for helping us debug the kernel prediction error in last week.\n\n\n> It was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.\n\nβ Cannot agree anymore.\n\n\n> I locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.\n\nβ We also locked submission a fews days before deadline. I guess those TPUs can really dry up one's ideas quickly huh? We then watched us dropping every day to 24th place on public LB during last week. Not the most fun experience.", "votes": 2 }, { "id": 727282, "postDate": "2020-01-23T15:50:25.103Z", "content": "Thanks and congrats for the result! It is funny, I was discussing with Dieter about what we could have done better and my first thing was to revisit sampling of empty windows ;)
", "rawMarkdown": "Thanks and congrats for the result! It is funny, I was discussing with Dieter about what we could have done better and my first thing was to revisit sampling of empty windows ;) ", "votes": 2 }, { "id": 733296, "postDate": "2020-01-30T23:53:07.030Z", "content": "Congrats π Thank you for writing up & sharing code!
", "rawMarkdown": "Congrats π Thank you for writing up & sharing code!" }, { "id": 732664, "postDate": "2020-01-30T04:11:37.833Z", "content": "This is a very good explanation
", "rawMarkdown": "This is a very good explanation" }, { "id": 732388, "postDate": "2020-01-29T19:03:59.947Z", "content": "This is perhaps a naive question about adding tokens to a Bert vocabulary:\nIn the Huggingface documentation, you add special tokens as follows:
\n\n\nnum_added_toks = tokenizer.add_special_tokens(special_tokens_dict)\nprint('We have added', num_added_toks, 'tokens')\nmodel.resize_token_embeddings(len(tokenizer)) # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.\n\nIn this winning solution, I didn't find a call to model.resize_token_embeddings (which is not yet implemented by Huggingface for TF models). Don't we need it? Is the Huggingface documentation wrong?
Great stuff!
", "rawMarkdown": "Great stuff!\n" }, { "id": 731084, "postDate": "2020-01-28T10:37:13.637Z", "content": "Since I locked my submission some days ago, I am only able to checkin today. Congrats @seesee and well done.
\n\nThanks so much for sharing your solution and code. I will try and reproduce you result in a few days. If I have any question then, I will be back to ask here.
", "rawMarkdown": "Since I locked my submission some days ago, I am only able to checkin today. Congrats @seesee and well done.\n\nThanks so much for sharing your solution and code. I will try and reproduce you result in a few days. If I have any question then, I will be back to ask here." }, { "id": 729176, "postDate": "2020-01-25T21:14:13.683Z", "content": "Congratulations and thanks for providing a repository with codes, this really helps!
", "rawMarkdown": "Congratulations and thanks for providing a repository with codes, this really helps!" }, { "id": 728513, "postDate": "2020-01-24T20:32:31.927Z", "content": "Congrats & Thanks for sharing your great solusionsπ π π
", "rawMarkdown": "Congrats & Thanks for sharing your great solusionsπ π π " }, { "id": 727495, "postDate": "2020-01-23T18:46:13.937Z", "content": "Congrats @seesee and thanks for sharing! Did you totally disregard the answer type classification?
\n\n> Adding HTML tags as custom tokens helped a bit
\n\nβ I totally missed this by removing the HTML tags and setting the long answer target position onto the [SEP] token, which performs very poorly, even when there is only the long answer head. Intuitively using separate models for short/long could get even higher?
", "rawMarkdown": "Congrats @seesee and thanks for sharing! Did you totally disregard the answer type classification?\n\n> Adding HTML tags as custom tokens helped a bit\n\nβ I totally missed this by removing the HTML tags and setting the long answer target position onto the [SEP] token, which performs very poorly, even when there is only the long answer head. Intuitively using separate models for short/long could get even higher?", "replies": [ { "id": 728008, "postDate": "2020-01-24T10:29:18.333Z", "content": "\n\n\nIntuitively using separate models for short/long could get even higher?
\n
Maybe, I prefer a single model and multiple targets. It's easier to tune, requires less code and less training time.
", "rawMarkdown": "> Intuitively using separate models for short/long could get even higher?\n\nMaybe, I prefer a single model and multiple targets. It's easier to tune, requires less code and less training time." } ] }, { "id": 727107, "postDate": "2020-01-23T13:16:29.217Z", "content": "Fantastic , kudos :-)
", "rawMarkdown": "Fantastic , kudos :-)" }, { "id": 727097, "postDate": "2020-01-23T13:06:31.043Z", "content": "Congrats mate @seesee! Just had a look at your code, a very clean code!\nSeems like you have played around with RoBERTa, how was the performance? I had issues recalculating the BPE tokenization using longest-common-substring algorithm. Wondering how was your experience.
", "rawMarkdown": "Congrats mate @seesee! Just had a look at your code, a very clean code!\nSeems like you have played around with RoBERTa, how was the performance? I had issues recalculating the BPE tokenization using longest-common-substring algorithm. Wondering how was your experience.", "replies": [ { "id": 727119, "postDate": "2020-01-23T13:24:29.830Z", "content": "Thanks. That's a good question. I should have added a What did not work section. You should be able to use the --model_name_or_path roberta-large flag for the training script. However, the performance was just like bert-large-uncased (i.e. ~0.70 LB). As I saw no gains from using this model I switched back to BERT. I still have no real explanation why Roberta didn't perform a lot better.
Congratulations @seesee. This approach is quite different from which we were using. Thanks for sharing :)
", "rawMarkdown": "Congratulations @seesee. This approach is quite different from which we were using. Thanks for sharing :)" }, { "id": 727045, "postDate": "2020-01-23T12:04:51.887Z", "content": "Congratulations! It is out of my expect that your approach is so simple but works well.
", "rawMarkdown": "Congratulations! It is out of my expect that your approach is so simple but works well.", "replies": [ { "id": 727089, "postDate": "2020-01-23T12:59:37.677Z", "content": "Thanks
\n\n> Build a model is very simple, but build a simple model is the hardest thing there is -- Guanshuo Xu
\n\nπ€
", "rawMarkdown": "Thanks \n\n> Build a model is very simple, but build a simple model is the hardest thing there is -- Guanshuo Xu\n\nπ€ ", "votes": 1 }, { "id": 727222, "postDate": "2020-01-23T14:38:27.720Z", "content": "Hi I have one question: if you don't have threshold, then you will have only 2 situations: blank for both short and long OR slice for both short and long. Right?
", "rawMarkdown": "Hi I have one question: if you don't have threshold, then you will have only 2 situations: blank for both short and long OR slice for both short and long. Right?" }, { "id": 727335, "postDate": "2020-01-23T16:31:20.973Z", "content": "No. If we ignore YES/NO answers there are 4 (short, long) possibilities: (empty, empty), (empty, span), (span, empty), (span, span). They can all be predicted by the model.
Thanks.\n\n min_dist = 1_000_000\n if long_token != -1:\n num_searched_long += 1\n for candidate in candidates[example_id]:\n cstart, cend = candidate.start_token, candidate.end_token\n dist = abs(cstart - long_token)\n if dist < min_dist:\n min_dist = dist\n if long_token == cstart:\n long_answer = f'{cstart}:{cend}'\n found_long = True\n break\n\nhave you tried using condition \ncstart<=long_token<=cend\nbesides \nong_token == cstart? Is score worse?
I didn't test it, but I am quite sure that the score won't change. I'd be surprised if it would get better. I checked that the model always predicts one of the opening tags or the CLS token. long_token == cstart should be fine.
Congrats! Thanks for sharing.
", "rawMarkdown": "Congrats! Thanks for sharing.", "votes": 1 }, { "id": 728691, "postDate": "2020-01-25T05:07:50.837Z", "content": "Thank you for your sharing!
", "rawMarkdown": "Thank you for your sharing!" }, { "id": 727025, "postDate": "2020-01-23T11:46:01.343Z", "content": "Congrats! Thanks for sharing.
", "rawMarkdown": "Congrats! Thanks for sharing." } ], "comments": [ { "id": 726994, "author_name": "Dieter", "author_url": "", "post_date": "2020-01-23T11:04:16.027000", "content": "we went with
\n\n\n", "votes": 3, "replies": [ { "id": 727000, "author_name": "See--", "author_url": "", "post_date": "2020-01-23T11:11:28.860000", "content": "taken days on my local 2 x 1080 TI machine
\n
haha, I ran out of hyperparameters and ideas π
", "votes": 1, "replies": [] } ] }, { "id": 734457, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2020-02-01T14:08:53.620000", "content": "Congratulations See!!! You derserve it =]
", "votes": 1, "replies": [] }, { "id": 731403, "author_name": "Sumit Mishra", "author_url": "", "post_date": "2020-01-28T15:43:54.543000", "content": "Congratulations Mate.
", "votes": 1, "replies": [] }, { "id": 727510, "author_name": "Ken Krige", "author_url": "", "post_date": "2020-01-23T18:59:17.143000", "content": "Your approach is clean and innovative. I am really enjoying reading it. Thank you so much for sharing your code. You're a good example of what kaggle is all about and you deserve to win the TF2.0 prize as well.
", "votes": 1, "replies": [ { "id": 728000, "author_name": "See--", "author_url": "", "post_date": "2020-01-24T10:17:22.790000", "content": "Thanks for the kind words π
", "votes": 0, "replies": [] } ] }, { "id": 727327, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2020-01-23T16:25:33.677000", "content": "I have two questions. :)
\n\nCould you please share your long answer F1 and short answer F1 if you evaluated them separately in local validation? I'm wondering how much the long_outputs helps.
For the long_logits, did you treat it as a binary classification obj? Whether the token is part of the long answer or not?
1 Yes I do. The scores are (concat F1, short F1, long F1) and local score is:\n* With ~3500 empty and ~3000 non-empty validation samples: (0.563, 0.481, 0.615)\n* Only ~3000 non-empty: (0.722, 0.576, 0.823)
\n\nMaybe check this script if you are using the same metric: https://github.com/see--/natural-question-answering/blob/master/eval_server.py
\n\nThe validation set is here:\nhttps://github.com/see--/natural-question-answering/blob/master/val_ids.csv
\n\n2 No, I treat it as softmax classification. Whether the token is the opening HTML tag and predict [CLS] token if there is no long answer.
", "votes": 1, "replies": [] }, { "id": 727516, "author_name": "Ken Krige", "author_url": "", "post_date": "2020-01-23T19:04:15.243000", "content": "\n\n\n2 No, I treat it as softmax classification. Whether the token is the opening HTML tag and predict [CLS] token if there is no long answer.
\n
Clever. This and your sampling ratios are big learning points for me so far.
", "votes": 1, "replies": [] } ] }, { "id": 727308, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2020-01-23T16:11:00.210000", "content": "Big congrats! Thank you so much for the amazing sharing of the source code.
", "votes": 1, "replies": [] }, { "id": 727204, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2020-01-23T14:29:53.283000", "content": "Great Solution! Thank you for sharing the code!! :)
", "votes": 1, "replies": [] }, { "id": 727176, "author_name": "Giba", "author_url": "", "post_date": "2020-01-23T14:10:41.140000", "content": "Congrats, nice write-up and thanks for sharing the code!
", "votes": 1, "replies": [] }, { "id": 726987, "author_name": "Mourad", "author_url": "", "post_date": "2020-01-23T10:54:34.120000", "content": "> I changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer.
\n\nGreat catch is all I can say. Congratulations @seesee π
", "votes": 1, "replies": [] }, { "id": 739421, "author_name": "Vitalii Mokin", "author_url": "", "post_date": "2020-02-07T20:24:44.083000", "content": "@seesee, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)
", "votes": 2, "replies": [] }, { "id": 730804, "author_name": "Alon Bochman", "author_url": "", "post_date": "2020-01-28T00:11:15.623000", "content": "Thanks for sharing the code @seesee ! I've been having some fun reproducing your results. Was hoping to ask a few questions about it:
\n\nCan you explain what the code is doing with crops? Seems like you are creating a text window around the true short answer for training purposes. Is that better than selecting the relevant long span among the candidates?
Your loss function (below) has a different weight for end_loss than the other components. Did you arrive at this through experimentation? Did it outperform equal-weight?
\nloss = ((tf.reduce_mean(start_loss) + tf.reduce_mean(end_loss) / 2.0) +\n tf.reduce_mean(long_loss)) / 2.0\n
Thanks,\nAlon
", "votes": 2, "replies": [ { "id": 731016, "author_name": "See--", "author_url": "", "post_date": "2020-01-28T08:54:20.183000", "content": "1)\n> Can you explain what the code is doing with crops? Seems like you are creating a text window around the true short answer for training purposes.
\n\nYes, the crop length is set so that we get the desired ratio between empty and non-empty training samples. By using crops we remove text that would result in empty samples.
\n\n> Is that better than selecting the relevant long span among the candidates?
\n\nThis was not tested. I don't know.
\n\n2) Thank you for writing this post! It's actually a bug. At least, it's not what I thought I was using as loss. My intention was:\n```
\n\nloss = ((tf.reduce_mean(start_loss) + tf.reduce_mean(end_loss)) / 2.0 +\n tf.reduce_mean(long_loss)) / 2.0\n``\nThis is a better approximation of the metric. Long and short answers should be weighted equally! I think it's better than taking themean` of all 3 loss terms (untested). After retraining with this fixed loss, public and private LB improve (entries are sorted by private LB but truncated, thus both show 0.71):
> Why did you choose to train for exactly 2 epochs? Would you recommend using validation every X steps as a stopping condition?
\n\n[1, 2, 3, 4] were tried and 2 gave the best local LB. Personally, I would not do it. It can help, but there are more important hyper parameters (e.g. sample ratios).
", "votes": 1, "replies": [] }, { "id": 731487, "author_name": "Alon Bochman", "author_url": "", "post_date": "2020-01-28T17:31:40.280000", "content": "\n\n\nAfter retraining with this fixed loss, public and private LB improve
\n
Haha... If only we'd partnered up, you might have made an extra few $...
\n\nThanks again for the code and answers.
\n\n-Alon
", "votes": 0, "replies": [] } ] }, { "id": 727375, "author_name": "Bo", "author_url": "", "post_date": "2020-01-23T17:07:43.187000", "content": "Congrats to the double win with regular prize + TF2.0 prize! Such a strong performance with a single model. Thanks for sharing the code so fast.
\n\nAlso thanks for helping us debug the kernel prediction error in last week.
\n\n\n\n\nIt was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.
\n
β Cannot agree anymore.
\n\n\n\n\nI locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.
\n
β We also locked submission a fews days before deadline. I guess those TPUs can really dry up one's ideas quickly huh? We then watched us dropping every day to 24th place on public LB during last week. Not the most fun experience.
", "votes": 2, "replies": [] }, { "id": 727282, "author_name": "CPMP", "author_url": "", "post_date": "2020-01-23T15:50:25.103000", "content": "Thanks and congrats for the result! It is funny, I was discussing with Dieter about what we could have done better and my first thing was to revisit sampling of empty windows ;)
", "votes": 2, "replies": [] }, { "id": 733296, "author_name": "corochann", "author_url": "", "post_date": "2020-01-30T23:53:07.030000", "content": "Congrats π Thank you for writing up & sharing code!
", "votes": 0, "replies": [] }, { "id": 732664, "author_name": "Sapharn", "author_url": "", "post_date": "2020-01-30T04:11:37.833000", "content": "This is a very good explanation
", "votes": 0, "replies": [] }, { "id": 732388, "author_name": "Alon Bochman", "author_url": "", "post_date": "2020-01-29T19:03:59.947000", "content": "This is perhaps a naive question about adding tokens to a Bert vocabulary:\nIn the Huggingface documentation, you add special tokens as follows:
\n\n\nnum_added_toks = tokenizer.add_special_tokens(special_tokens_dict)\nprint('We have added', num_added_toks, 'tokens')\nmodel.resize_token_embeddings(len(tokenizer)) # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.\n\nIn this winning solution, I didn't find a call to model.resize_token_embeddings (which is not yet implemented by Huggingface for TF models). Don't we need it? Is the Huggingface documentation wrong?
Great stuff!
", "votes": 0, "replies": [] }, { "id": 731084, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-01-28T10:37:13.637000", "content": "Since I locked my submission some days ago, I am only able to checkin today. Congrats @seesee and well done.
\n\nThanks so much for sharing your solution and code. I will try and reproduce you result in a few days. If I have any question then, I will be back to ask here.
", "votes": 0, "replies": [] }, { "id": 729176, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-01-25T21:14:13.683000", "content": "Congratulations and thanks for providing a repository with codes, this really helps!
", "votes": 0, "replies": [] }, { "id": 728513, "author_name": "Miyabon", "author_url": "", "post_date": "2020-01-24T20:32:31.927000", "content": "Congrats & Thanks for sharing your great solusionsπ π π
", "votes": 0, "replies": [] }, { "id": 727495, "author_name": "Zhengkai Tu", "author_url": "", "post_date": "2020-01-23T18:46:13.937000", "content": "Congrats @seesee and thanks for sharing! Did you totally disregard the answer type classification?
\n\n> Adding HTML tags as custom tokens helped a bit
\n\nβ I totally missed this by removing the HTML tags and setting the long answer target position onto the [SEP] token, which performs very poorly, even when there is only the long answer head. Intuitively using separate models for short/long could get even higher?
", "votes": 0, "replies": [ { "id": 728008, "author_name": "See--", "author_url": "", "post_date": "2020-01-24T10:29:18.333000", "content": "\n\n\nIntuitively using separate models for short/long could get even higher?
\n
Maybe, I prefer a single model and multiple targets. It's easier to tune, requires less code and less training time.
", "votes": 0, "replies": [] } ] }, { "id": 727107, "author_name": "aintnosunshine", "author_url": "", "post_date": "2020-01-23T13:16:29.217000", "content": "Fantastic , kudos :-)
", "votes": 0, "replies": [] }, { "id": 727097, "author_name": "Kornesh Kanan", "author_url": "", "post_date": "2020-01-23T13:06:31.043000", "content": "Congrats mate @seesee! Just had a look at your code, a very clean code!\nSeems like you have played around with RoBERTa, how was the performance? I had issues recalculating the BPE tokenization using longest-common-substring algorithm. Wondering how was your experience.
", "votes": 0, "replies": [ { "id": 727119, "author_name": "See--", "author_url": "", "post_date": "2020-01-23T13:24:29.830000", "content": "Thanks. That's a good question. I should have added a What did not work section. You should be able to use the --model_name_or_path roberta-large flag for the training script. However, the performance was just like bert-large-uncased (i.e. ~0.70 LB). As I saw no gains from using this model I switched back to BERT. I still have no real explanation why Roberta didn't perform a lot better.
Congratulations @seesee. This approach is quite different from which we were using. Thanks for sharing :)
", "votes": 0, "replies": [] }, { "id": 727045, "author_name": "SchenbergZ", "author_url": "", "post_date": "2020-01-23T12:04:51.887000", "content": "Congratulations! It is out of my expect that your approach is so simple but works well.
", "votes": 0, "replies": [ { "id": 727089, "author_name": "See--", "author_url": "", "post_date": "2020-01-23T12:59:37.677000", "content": "Thanks
\n\n> Build a model is very simple, but build a simple model is the hardest thing there is -- Guanshuo Xu
\n\nπ€
", "votes": 1, "replies": [] }, { "id": 727222, "author_name": "SchenbergZ", "author_url": "", "post_date": "2020-01-23T14:38:27.720000", "content": "Hi I have one question: if you don't have threshold, then you will have only 2 situations: blank for both short and long OR slice for both short and long. Right?
", "votes": 0, "replies": [] }, { "id": 727335, "author_name": "See--", "author_url": "", "post_date": "2020-01-23T16:31:20.973000", "content": "No. If we ignore YES/NO answers there are 4 (short, long) possibilities: (empty, empty), (empty, span), (span, empty), (span, span). They can all be predicted by the model.
Thanks.\n\n min_dist = 1_000_000\n if long_token != -1:\n num_searched_long += 1\n for candidate in candidates[example_id]:\n cstart, cend = candidate.start_token, candidate.end_token\n dist = abs(cstart - long_token)\n if dist < min_dist:\n min_dist = dist\n if long_token == cstart:\n long_answer = f'{cstart}:{cend}'\n found_long = True\n break\n\nhave you tried using condition \ncstart<=long_token<=cend\nbesides \nong_token == cstart? Is score worse?
I didn't test it, but I am quite sure that the score won't change. I'd be surprised if it would get better. I checked that the model always predicts one of the opening tags or the CLS token. long_token == cstart should be fine.
Congrats! Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 728691, "author_name": "Takuya Makiyama", "author_url": "", "post_date": "2020-01-25T05:07:50.837000", "content": "Thank you for your sharing!
", "votes": 0, "replies": [] }, { "id": 727025, "author_name": "Shayekh Islam", "author_url": "", "post_date": "2020-01-23T11:46:01.343000", "content": "Congrats! Thanks for sharing.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "726961": "Good evening,\n\nfirst of all, I'd like to thank Kaggle and the hosts for this awesome challenge! It was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.\n\nSecondly, congrats to all the winners. I locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.\n\nMy solution is just a single TF2.0 model. It uses custom heads and a BERT transformers backbone (large version). For modeling and training I am using the great [transformers](https://github.com/huggingface/transformers) library. I think that the [following snippet](https://github.com/see--/natural-question-answering/blob/master/models.py#L8-L28) is useful to understand the modeling:\n```python\nclass TFBertForNaturalQuestionAnswering(TFBertPreTrainedModel):\n def __init__(self, config, *inputs, **kwargs):\n super().__init__(config, *inputs, **kwargs)\n self.num_labels = config.num_labels\n\n self.bert = TFBertMainLayer(config, name='bert')\n self.initializer = get_initializer(config.initializer_range)\n self.qa_outputs = L.Dense(config.num_labels,\n kernel_initializer=self.initializer, name='qa_outputs')\n self.long_outputs = L.Dense(1, kernel_initializer=self.initializer,\n name='long_outputs')\n\n def call(self, inputs, **kwargs):\n outputs = self.bert(inputs, **kwargs)\n sequence_output = outputs[0]\n logits = self.qa_outputs(sequence_output)\n start_logits, end_logits = tf.split(logits, 2, axis=-1)\n start_logits = tf.squeeze(start_logits, -1)\n end_logits = tf.squeeze(end_logits, -1)\n long_logits = tf.squeeze(self.long_outputs(sequence_output), -1)\n return start_logits, end_logits, long_logits\n```\n\nAs you can see, the natural question answering task can be treated like SQUAD-2.0 with an additional head for long answers. Note that we just need a single output: The opening tag of the HTML bounding box. I guess most competitors used a similar modeling so I think what made the difference to most other solutions is the sampling.\n\nI changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer. I started with a really low empty answer ratio which I got from the [`bert-joint` paper](https://arxiv.org/abs/1901.08634), but I couldn't reach a good score. I tuned a few hyper parameters but overall I got good results with a wide range of parameters. Adding HTML tags as custom tokens helped a bit. I also tried different start weights and found that:\n\n`bert-large-uncased` (~0.70 LB) < `bert-large-uncased-whole-word-masking` (~0.72 LB) < `bert-large-uncased-whole-word-masking-finetuned-squad` (~0.73 LB).\n\nThat's about it. Thanks to @boliu0, @christofhenkel and @kentaronakanishi for fixing and providing the metric!\n\nPlease refer to my repository for implementation details and instructions to reproduce:\n* https://github.com/see--/natural-question-answering\n\nYou can find the 2nd place kernel and pretrained weights on Kaggle:\n* https://www.kaggle.com/seesee/submit-full\n* https://www.kaggle.com/seesee/nq-bert-uncased-68\n\nFeel free to ask questions and / or create GitHub issues.", "726994": "we went with \n> taken days on my local 2 x 1080 TI machine", "734457": "Congratulations See!!! You derserve it =]", "731403": "Congratulations Mate.", "727510": "Your approach is clean and innovative. I am really enjoying reading it. Thank you so much for sharing your code. You're a good example of what kaggle is all about and you deserve to win the TF2.0 prize as well.", "727327": "I have two questions. :)\n\n1. Could you please share your long answer F1 and short answer F1 if you evaluated them separately in local validation? I'm wondering how much the `long_outputs` helps. \n\n2. For the `long_logits`, did you treat it as a binary classification obj? Whether the token is part of the long answer or not?", "727308": "Big congrats! Thank you so much for the amazing sharing of the source code.", "727204": "Great Solution! Thank you for sharing the code!! :)", "727176": "Congrats, nice write-up and thanks for sharing the code!", "726987": "> I changed the empty answer ratio so that it is similar to the full dataset. I.e. roughly as many empty answers as answers with a long answer.\n\nGreat catch is all I can say. Congratulations @seesee π ", "739421": "@seesee, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)", "730804": "Thanks for sharing the code @seesee ! I've been having some fun reproducing your results. Was hoping to ask a few questions about it:\n\n1. Can you explain what the code is doing with crops? Seems like you are creating a text window around the true short answer for training purposes. Is that better than selecting the relevant long span among the candidates?\n\n2. Your loss function (below) has a different weight for end_loss than the other components. Did you arrive at this through experimentation? Did it outperform equal-weight?\n \n```\nloss = ((tf.reduce_mean(start_loss) + tf.reduce_mean(end_loss) / 2.0) +\n tf.reduce_mean(long_loss)) / 2.0\n```\n\n3. Why did you choose to train for exactly 2 epochs? Would you recommend using validation every X steps as a stopping condition?\n\nThanks,\nAlon\n\n\n", "727375": "Congrats to the double win with regular prize + TF2.0 prize! Such a strong performance with a single model. Thanks for sharing the code so fast.\n\nAlso thanks for helping us debug the kernel prediction error in last week.\n\n\n> It was really fun to work with TF2.0 and TPUs. What would have taken days on my local 2 x 1080 TI machine just took a couple of hours. For example, the actual training time for my final model (excluding tokenization and post-processing) is just a little more than 2 hours.\n\nβ Cannot agree anymore.\n\n\n> I locked my submissions a week ago with +0.04 on public but everyone kept improving. Maybe I should have had continued working on this challenged as well.\n\nβ We also locked submission a fews days before deadline. I guess those TPUs can really dry up one's ideas quickly huh? We then watched us dropping every day to 24th place on public LB during last week. Not the most fun experience.", "727282": "Thanks and congrats for the result! It is funny, I was discussing with Dieter about what we could have done better and my first thing was to revisit sampling of empty windows ;) ", "733296": "Congrats π Thank you for writing up & sharing code!", "732664": "This is a very good explanation", "732388": "This is perhaps a naive question about adding tokens to a Bert vocabulary:\nIn the [Huggingface documentation](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_special_tokens), you add special tokens as follows:\n\n```\nnum_added_toks = tokenizer.add_special_tokens(special_tokens_dict)\nprint('We have added', num_added_toks, 'tokens')\nmodel.resize_token_embeddings(len(tokenizer)) # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.\n```\nIn this winning solution, I didn't find a call to model.resize\\_token\\_embeddings (which is not yet implemented by Huggingface for TF models). Don't we need it? Is the Huggingface documentation wrong?", "731853": "Great stuff!\n", "731084": "Since I locked my submission some days ago, I am only able to checkin today. Congrats @seesee and well done.\n\nThanks so much for sharing your solution and code. I will try and reproduce you result in a few days. If I have any question then, I will be back to ask here.", "729176": "Congratulations and thanks for providing a repository with codes, this really helps!", "728513": "Congrats & Thanks for sharing your great solusionsπ π π ", "727495": "Congrats @seesee and thanks for sharing! Did you totally disregard the answer type classification?\n\n> Adding HTML tags as custom tokens helped a bit\n\nβ I totally missed this by removing the HTML tags and setting the long answer target position onto the [SEP] token, which performs very poorly, even when there is only the long answer head. Intuitively using separate models for short/long could get even higher?", "727107": "Fantastic , kudos :-)", "727097": "Congrats mate @seesee! Just had a look at your code, a very clean code!\nSeems like you have played around with RoBERTa, how was the performance? I had issues recalculating the BPE tokenization using longest-common-substring algorithm. Wondering how was your experience.", "727050": "Congratulations @seesee. This approach is quite different from which we were using. Thanks for sharing :)", "727045": "Congratulations! It is out of my expect that your approach is so simple but works well.", "727004": "", "727920": "Congrats! Thanks for sharing.", "728691": "Thank you for your sharing!", "727025": "Congrats! Thanks for sharing." } }