Thanks kaggle provides this competition and this is a big improvement for me to take an competition solo and reach this place.
\n\nBase from (https://github.com/bojone/bert4keras) My solution is based on a wierd ROBERTA structure inplemented in keras (https://www.kaggle.com/httpwwwfszyc/bert4keras4nq) as it is different from the huggingface in \n1. 2*1024 token_type_id_embedding layer (0 put pretrained weights, 1 set to np.zeros((1,1024)))\n2. no padding masking \n3. wierd zero masking in attention mask:\n```\n def build(self):\n x_in = Input(shape=(512, ), name='Input-Token')\n s_in = Input(shape=(512, ), name='Input-Segment')\n x, s = x_in, s_in
\n\n sequence_mask = Lambda(lambda x: K.cast(K.greater(x, 0), 'float32'),\n name='Sequence-Mask')(x)\n\n # Embedding\n x = Embedding(input_dim=self.vocab_size,\n output_dim=self.embedding_size,\n embeddings_initializer=self.initializer,\n name='Embedding-Token')(x)\n s = Embedding(input_dim=2, #1 or 2 , 2 finally because roberta need to train it\n output_dim=self.embedding_size,\n embeddings_initializer=self.initializer,\n name='Embedding-Segment')(s)\n x = Add(name='Embedding-Token-Segment')([x, s])\n if self.max_position_embeddings == 514:\n x = RobertaPositionEmbeddings(input_dim=self.max_position_embeddings,\n output_dim=self.embedding_size,\n merge_mode='add',\n embeddings_initializer=self.initializer,\n name='Embedding-Position')([x,x_in])\n else:\n x = PositionEmbedding(input_dim=self.max_position_embeddings,\n output_dim=self.embedding_size,\n merge_mode='add',\n embeddings_initializer=self.initializer,\n name='Embedding-Position')(x)\n x = LayerNormalization(name='Embedding-Norm')(x)\n if self.dropout_rate > 0:\n x = Dropout(rate=self.dropout_rate, name='Embedding-Dropout')(x)\n if self.embedding_size != self.hidden_size:\n x = Dense(units=self.hidden_size,\n kernel_initializer=self.initializer,\n name='Embedding-Mapping')(x)\n\n layers = None\n for i in range(self.num_hidden_layers):\n attention_name = 'Encoder-%d-MultiHeadSelfAttention' % (i + 1)\n feed_forward_name = 'Encoder-%d-FeedForward' % (i + 1)\n x, layers = self.transformer_block(\n inputs=x,\n sequence_mask=sequence_mask,\n attention_mask=self.compute_attention_mask(i, s_in),\n attention_name=attention_name,\n feed_forward_name=feed_forward_name,\n input_layers=layers)\n x = self.post_processing(i, x)\n if not self.block_sharing:\n layers = None\n\n outputs = [x]\n```\nI concatenate last 4 layers and put a single linear output for each output head.
\n\nSamples-ratio of non-zero with 256 stride vs zero with stride 128 is 1:4.
\n\n*Total_loss = loss_weights1*sample_weights*start_loss+ loss_weights2*sample_weights*end_loss+ loss_weights3*sample_weights*answertype_loss*
\n\nloss weights for [start, end, answer_type] is 1:1: (1/sampleweight.mean())
\n\nThe result of my solution is: CV: 0.478 because there are too many False Negative samples after I fix my metric error 10 days ago. So searching by my 2 devsets I finally choose a safe threshold ( the one slightly smaller than the threshold which reach max CV in order to lower the risk) . If a short answer score less than 0.5 or long answer smaller than 0.1 the answer will be blank.
\n\nAs I only upload 1 model, I use stride=128 for inference.\nMy result: CV 0.523, public LB 0.63, private LB 0.65
\n\n\n sequence_mask = Lambda(lambda x: K.cast(K.greater(x, 0), 'float32'),\n name='Sequence-Mask')(x)\n\nI have no time to test it. So I just replace it by:\n\n sequence_mask = Lambda(lambda x: K.cast(K.not_equal(x, 1), 'float32'),\n name='Sequence-Mask')(x) \n\nwhich replace token 1 because 1 represent padding. After TPU training I got 0.53 CV score. But when I plug this to gpu, the public LB only reach 0.48, which is wierd. So my question is: is my replacement of attention_mask really matching what I expect (mask token 1 of value matrix in attention layer)?Thanks kaggle provides this competition and this is a big improvement for me to take an competition solo and reach this place.
\n\nBase from (https://github.com/bojone/bert4keras) My solution is based on a wierd ROBERTA structure inplemented in keras (https://www.kaggle.com/httpwwwfszyc/bert4keras4nq) as it is different from the huggingface in \n1. 2*1024 token_type_id_embedding layer (0 put pretrained weights, 1 set to np.zeros((1,1024)))\n2. no padding masking \n3. wierd zero masking in attention mask:\n```\n def build(self):\n x_in = Input(shape=(512, ), name='Input-Token')\n s_in = Input(shape=(512, ), name='Input-Segment')\n x, s = x_in, s_in
\n\n sequence_mask = Lambda(lambda x: K.cast(K.greater(x, 0), 'float32'),\n name='Sequence-Mask')(x)\n\n # Embedding\n x = Embedding(input_dim=self.vocab_size,\n output_dim=self.embedding_size,\n embeddings_initializer=self.initializer,\n name='Embedding-Token')(x)\n s = Embedding(input_dim=2, #1 or 2 , 2 finally because roberta need to train it\n output_dim=self.embedding_size,\n embeddings_initializer=self.initializer,\n name='Embedding-Segment')(s)\n x = Add(name='Embedding-Token-Segment')([x, s])\n if self.max_position_embeddings == 514:\n x = RobertaPositionEmbeddings(input_dim=self.max_position_embeddings,\n output_dim=self.embedding_size,\n merge_mode='add',\n embeddings_initializer=self.initializer,\n name='Embedding-Position')([x,x_in])\n else:\n x = PositionEmbedding(input_dim=self.max_position_embeddings,\n output_dim=self.embedding_size,\n merge_mode='add',\n embeddings_initializer=self.initializer,\n name='Embedding-Position')(x)\n x = LayerNormalization(name='Embedding-Norm')(x)\n if self.dropout_rate > 0:\n x = Dropout(rate=self.dropout_rate, name='Embedding-Dropout')(x)\n if self.embedding_size != self.hidden_size:\n x = Dense(units=self.hidden_size,\n kernel_initializer=self.initializer,\n name='Embedding-Mapping')(x)\n\n layers = None\n for i in range(self.num_hidden_layers):\n attention_name = 'Encoder-%d-MultiHeadSelfAttention' % (i + 1)\n feed_forward_name = 'Encoder-%d-FeedForward' % (i + 1)\n x, layers = self.transformer_block(\n inputs=x,\n sequence_mask=sequence_mask,\n attention_mask=self.compute_attention_mask(i, s_in),\n attention_name=attention_name,\n feed_forward_name=feed_forward_name,\n input_layers=layers)\n x = self.post_processing(i, x)\n if not self.block_sharing:\n layers = None\n\n outputs = [x]\n```\nI concatenate last 4 layers and put a single linear output for each output head.
\n\nSamples-ratio of non-zero with 256 stride vs zero with stride 128 is 1:4.
\n\n*Total_loss = loss_weights1*sample_weights*start_loss+ loss_weights2*sample_weights*end_loss+ loss_weights3*sample_weights*answertype_loss*
\n\nloss weights for [start, end, answer_type] is 1:1: (1/sampleweight.mean())
\n\nThe result of my solution is: CV: 0.478 because there are too many False Negative samples after I fix my metric error 10 days ago. So searching by my 2 devsets I finally choose a safe threshold ( the one slightly smaller than the threshold which reach max CV in order to lower the risk) . If a short answer score less than 0.5 or long answer smaller than 0.1 the answer will be blank.
\n\nAs I only upload 1 model, I use stride=128 for inference.\nMy result: CV 0.523, public LB 0.63, private LB 0.65
\n\n\n sequence_mask = Lambda(lambda x: K.cast(K.greater(x, 0), 'float32'),\n name='Sequence-Mask')(x)\n\nI have no time to test it. So I just replace it by:\n\n sequence_mask = Lambda(lambda x: K.cast(K.not_equal(x, 1), 'float32'),\n name='Sequence-Mask')(x) \n\nwhich replace token 1 because 1 represent padding. After TPU training I got 0.53 CV score. But when I plug this to gpu, the public LB only reach 0.48, which is wierd. So my question is: is my replacement of attention_mask really matching what I expect (mask token 1 of value matrix in attention layer)?You are welcome!
", "rawMarkdown": "You are welcome!" } ] } ], "comments": [ { "id": 726759, "author_name": "SchenbergZ", "author_url": "", "post_date": "2020-01-23T08:13:25.777000", "content": "You are welcome!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "726753": "Thanks kaggle provides this competition and this is a big improvement for me to take an competition solo and reach this place.\n\n## **1. Wierd ROBERTA Achitecture**\nBase from (https://github.com/bojone/bert4keras) My solution is based on a wierd ROBERTA structure inplemented in keras (https://www.kaggle.com/httpwwwfszyc/bert4keras4nq) as it is different from the huggingface in \n1. 2*1024 token_type_id_embedding layer (0 put pretrained weights, 1 set to np.zeros((1,1024)))\n2. no padding masking \n3. wierd zero masking in attention mask:\n```\n def build(self):\n x_in = Input(shape=(512, ), name='Input-Token')\n s_in = Input(shape=(512, ), name='Input-Segment')\n x, s = x_in, s_in\n\n sequence_mask = Lambda(lambda x: K.cast(K.greater(x, 0), 'float32'),\n name='Sequence-Mask')(x)\n\n # Embedding\n x = Embedding(input_dim=self.vocab_size,\n output_dim=self.embedding_size,\n embeddings_initializer=self.initializer,\n name='Embedding-Token')(x)\n s = Embedding(input_dim=2, #1 or 2 , 2 finally because roberta need to train it\n output_dim=self.embedding_size,\n embeddings_initializer=self.initializer,\n name='Embedding-Segment')(s)\n x = Add(name='Embedding-Token-Segment')([x, s])\n if self.max_position_embeddings == 514:\n x = RobertaPositionEmbeddings(input_dim=self.max_position_embeddings,\n output_dim=self.embedding_size,\n merge_mode='add',\n embeddings_initializer=self.initializer,\n name='Embedding-Position')([x,x_in])\n else:\n x = PositionEmbedding(input_dim=self.max_position_embeddings,\n output_dim=self.embedding_size,\n merge_mode='add',\n embeddings_initializer=self.initializer,\n name='Embedding-Position')(x)\n x = LayerNormalization(name='Embedding-Norm')(x)\n if self.dropout_rate > 0:\n x = Dropout(rate=self.dropout_rate, name='Embedding-Dropout')(x)\n if self.embedding_size != self.hidden_size:\n x = Dense(units=self.hidden_size,\n kernel_initializer=self.initializer,\n name='Embedding-Mapping')(x)\n\n layers = None\n for i in range(self.num_hidden_layers):\n attention_name = 'Encoder-%d-MultiHeadSelfAttention' % (i + 1)\n feed_forward_name = 'Encoder-%d-FeedForward' % (i + 1)\n x, layers = self.transformer_block(\n inputs=x,\n sequence_mask=sequence_mask,\n attention_mask=self.compute_attention_mask(i, s_in),\n attention_name=attention_name,\n feed_forward_name=feed_forward_name,\n input_layers=layers)\n x = self.post_processing(i, x)\n if not self.block_sharing:\n layers = None\n\n outputs = [x]\n```\nI concatenate last 4 layers and put a single linear output for each output head. \n## **2. data distribution**\nSamples-ratio of non-zero with 256 stride vs zero with stride 128 is 1:4.\n## **3. Training**\n1. UseRadam with warmup 0.05 and train 1 epoch\n2. set different weights to match the distribution of dev set (I use 2 dev set for the 135000th to 140000th and for the 302373 to end section). As a result my loss is:\n\n*Total\\_loss = loss_weights1\\*sample\\_weights\\*start\\_loss+ loss\\_weights2\\*sample\\_weights\\*end_loss+ loss\\_weights3\\*sample\\_weights\\*answertype\\_loss*\n\nloss weights for [start, end, answer_type] is 1:1: (1/sampleweight.mean())\n\n## **4. Threshold killing False Positive**\nThe result of my solution is: CV: 0.478 because there are too many False Negative samples after I fix my metric error 10 days ago. So searching by my 2 devsets I finally choose a safe threshold ( the one slightly smaller than the threshold which reach max CV in order to lower the risk) . If a short answer score less than 0.5 or long answer smaller than 0.1 the answer will be blank.\n\nAs I only upload 1 model, I use stride=128 for inference.\nMy result: CV 0.523, public LB 0.63, private LB 0.65\n\n## **5 My question**\n1. how to mask padding:\nI tried to add a padding mask before embedding layer but which will raise error because layers after does not support mask... So I have to use this wierd ROBERTA architecture.\n2. why my attention mask never works:\nBase on time limit for me, when I realise there are some mistake on attention mask because of the code\n```\n sequence_mask = Lambda(lambda x: K.cast(K.greater(x, 0), 'float32'),\n name='Sequence-Mask')(x)\n```\nI have no time to test it. So I just replace it by:\n```\n sequence_mask = Lambda(lambda x: K.cast(K.not_equal(x, 1), 'float32'),\n name='Sequence-Mask')(x) \n```\nwhich replace token 1 because 1 represent padding. After TPU training I got 0.53 CV score. But when I plug this to gpu, the public LB only reach 0.48, which is wierd. So my question is: is my replacement of attention_mask really matching what I expect (mask token 1 of value matrix in attention layer)?", "726759": "## Additions: \n1. Because the private data seems close to the public, so my weighting strategy might be not good.\n2. RobertaPositionEmbedding comes from huggingface transformers", "726779": "" } }