In data preprocessing, we did as followed:
\nnormalization, NaN_filling and Padding
Extract the left and right hand from the entire dot, and only consider the relative position between the dots of the hands.
lhand = x[:,(LIP):(LIP)+(LHAND)]\nrhand = x[:,(LIP)+(LHAND):]\nrelative_lhand =norm_xy (lhand.unsqueeze() - lhand.unsqueeze())\nrelative_rhand =norm_xy (rhand.unsqueeze() - rhand.unsqueeze())\nrelative_back = torch.zeros(x.shape[],x.shape[],n//,)\nrelative_front = torch.zeros(x.shape[],x.shape[],n//,)\n i (n//):\n off =x[:,i+:]-x[:,:-i-]\n relative_back[:,i+:,i] = off\n relative_front[:,:-i-,i] = -off\nrelative = torch.cat([relative_back,relative_front],-)\nrelative_xy = norm_xy(relative).permute(,,,)\nWe performed the followings in data augmentation:
\n self.p>:\n indices=[]\n (indices) ==:\n indices = (torch.rand(sample.shape[]) >=self.fd).nonzero().squeeze()\n sample = sample[indices]\n sample\nx = x_max - x + x_min\n| \n | Layer Name | \nOutput Shape | \nParameters | \n
|---|---|---|---|
| TransformerEmbedding | \nx_embed.fc1 | \n(batch_size, 128, 135) | \n17,408 | \n
| \n | x_embed.fc2 | \n(batch_size, 128, 135) | \n17,408 | \n
| \n | x_embed.fc3 | \n(batch_size, 128, 672) | \n87,168 | \n
| \n | x_embed.fc4 | \n(batch_size, 128, 672) | \n87,168 | \n
| \n | x_embed.fc5 | \n(batch_size, 128, 882) | \n113,376 | \n
| \n | x_embed.fc6 | \n(batch_size, 128, 882) | \n113,376 | \n
| \n | x_embed.fc | \n(batch_size, 128, 512) | \n442,496 | \n
| LayerNorm | \nnorm | \n(batch_size, 512) | \n1,024 | \n
| TransformerBlock | \nencoder.attn.fc_q | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.attn.fc_k | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.attn.fc_v | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.attn.fc_o | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.norm1 | \n(batch_size, 512) | \n1,024 | \n
| \n | encoder.norm2 | \n(batch_size, 512) | \n1,024 | \n
| Logit | \nLinear | \n(batch_size, 250) | \n12,8250 | \n
Learning Rate : $1.0 × 10^{-4}$
Dropout Probability : $p = 0.3$
Epoches : $200$
We use $AdamW$ optimizer for training.(The best-performing in our experiments)
The activation function we used is called $SWISH$, which was proposed by Google.
\nThe SWISH activation function is a non-linear function similar to ReLU, but it performs better in some cases than ReLU. This is because it can produce stronger regularization effects.
CrossEntropy (with label smoothing which is useful)
Finally, our single model can achieve 0.77.
\nThe model ensemble was not so great, so we won't post it here to mislead anyone. We just took a simple average of the class probabilities obtained from each model.
\nThe differences between models are not significant, which may be the main reason why the fusion did not improve significantly.
The fusion method did not work well.
Thank 🐸 very much for providing the code and ideas.
", "messages": [ { "id": "2242732", "postDate": "05/02/2023 13:28:23", "content": "In data preprocessing, we did as followed:
\nnormalization, NaN_filling and Padding
Extract the left and right hand from the entire dot, and only consider the relative position between the dots of the hands.
lhand = x[:,(LIP):(LIP)+(LHAND)]\nrhand = x[:,(LIP)+(LHAND):]\nrelative_lhand =norm_xy (lhand.unsqueeze() - lhand.unsqueeze())\nrelative_rhand =norm_xy (rhand.unsqueeze() - rhand.unsqueeze())\nrelative_back = torch.zeros(x.shape[],x.shape[],n//,)\nrelative_front = torch.zeros(x.shape[],x.shape[],n//,)\n i (n//):\n off =x[:,i+:]-x[:,:-i-]\n relative_back[:,i+:,i] = off\n relative_front[:,:-i-,i] = -off\nrelative = torch.cat([relative_back,relative_front],-)\nrelative_xy = norm_xy(relative).permute(,,,)\nWe performed the followings in data augmentation:
\n self.p>:\n indices=[]\n (indices) ==:\n indices = (torch.rand(sample.shape[]) >=self.fd).nonzero().squeeze()\n sample = sample[indices]\n sample\nx = x_max - x + x_min\n| \n | Layer Name | \nOutput Shape | \nParameters | \n
|---|---|---|---|
| TransformerEmbedding | \nx_embed.fc1 | \n(batch_size, 128, 135) | \n17,408 | \n
| \n | x_embed.fc2 | \n(batch_size, 128, 135) | \n17,408 | \n
| \n | x_embed.fc3 | \n(batch_size, 128, 672) | \n87,168 | \n
| \n | x_embed.fc4 | \n(batch_size, 128, 672) | \n87,168 | \n
| \n | x_embed.fc5 | \n(batch_size, 128, 882) | \n113,376 | \n
| \n | x_embed.fc6 | \n(batch_size, 128, 882) | \n113,376 | \n
| \n | x_embed.fc | \n(batch_size, 128, 512) | \n442,496 | \n
| LayerNorm | \nnorm | \n(batch_size, 512) | \n1,024 | \n
| TransformerBlock | \nencoder.attn.fc_q | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.attn.fc_k | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.attn.fc_v | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.attn.fc_o | \n(batch_size, 512) | \n262,656 | \n
| \n | encoder.norm1 | \n(batch_size, 512) | \n1,024 | \n
| \n | encoder.norm2 | \n(batch_size, 512) | \n1,024 | \n
| Logit | \nLinear | \n(batch_size, 250) | \n12,8250 | \n
Learning Rate : $1.0 × 10^{-4}$
Dropout Probability : $p = 0.3$
Epoches : $200$
We use $AdamW$ optimizer for training.(The best-performing in our experiments)
The activation function we used is called $SWISH$, which was proposed by Google.
\nThe SWISH activation function is a non-linear function similar to ReLU, but it performs better in some cases than ReLU. This is because it can produce stronger regularization effects.
CrossEntropy (with label smoothing which is useful)
Finally, our single model can achieve 0.77.
\nThe model ensemble was not so great, so we won't post it here to mislead anyone. We just took a simple average of the class probabilities obtained from each model.
\nThe differences between models are not significant, which may be the main reason why the fusion did not improve significantly.
The fusion method did not work well.
Thank 🐸 very much for providing the code and ideas.
", "rawMarkdown": "## Data Preprocessing\nIn data preprocessing, we did as followed:\n\n* normalization, NaN_filling and Padding\n\n\n* Extract the left and right hand from the entire dot, and only consider the relative position between the dots of the hands.\n```python\nlhand = x[:,len(LIP):len(LIP)+len(LHAND)]\nrhand = x[:,len(LIP)+len(LHAND):]\nrelative_lhand =norm_xy (lhand.unsqueeze(1) - lhand.unsqueeze(2))\nrelative_rhand =norm_xy (rhand.unsqueeze(1) - rhand.unsqueeze(2))\n```\n\n* Add the relative position changes between frames.(If frame=16, we are computing the relative distances between the current frame and the previous 8 frames and the next 8 frames.)\n\n```python\nrelative_back = torch.zeros(x.shape[0],x.shape[1],n//2,2)\nrelative_front = torch.zeros(x.shape[0],x.shape[1],n//2,2)\nfor i in range(n//2):\n off =x[:,i+1:]-x[:,:-i-1]\n relative_back[:,i+1:,i] = off\n relative_front[:,:-i-1,i] = -off\nrelative = torch.cat([relative_back,relative_front],-2)#shape (len(LHAND)*2, max_length, n, 2)\nrelative_xy = norm_xy(relative).permute(1,0,2,3)\n```\n\n* add Original features.\n\n## Augmentation\nWe performed the followings in data augmentation:\n\n* Random frame dropout.(It seems to be a very common processing method in other sign language recognition studies.)\n```python\nif self.p>0:\n indices=[]\n while len(indices) ==0:\n indices = (torch.rand(sample.shape[0]) >=self.fd).nonzero().squeeze(1)\n sample = sample[indices]\nreturn sample\n```\n\n* Flip (without flipping hands).\n```python\nx = x_max - x + x_min\n```\n\n## Model\n### Model Construction and Hyperparameter \n* Model Construction(a typical one):\n\n| | Layer Name | Output Shape | Parameters |\n| --- | --- | --- | --- |\n| TransformerEmbedding | x_embed.fc1 | (batch_size, 128, 135) | 17,408 |\n| | x_embed.fc2 | (batch_size, 128, 135) | 17,408 |\n| | x_embed.fc3 | (batch_size, 128, 672) | 87,168 |\n| | x_embed.fc4 | (batch_size, 128, 672) | 87,168 |\n| | x_embed.fc5 | (batch_size, 128, 882) | 113,376 |\n| | x_embed.fc6 | (batch_size, 128, 882) | 113,376 |\n| | x_embed.fc | (batch_size, 128, 512) | 442,496 |\n| LayerNorm | norm | (batch_size, 512) | 1,024 |\n| TransformerBlock | encoder.attn.fc_q | (batch_size, 512) | 262,656 |\n| | encoder.attn.fc_k | (batch_size, 512) | 262,656 |\n| | encoder.attn.fc_v | (batch_size, 512) | 262,656 |\n| | encoder.attn.fc_o | (batch_size, 512) | 262,656 |\n| | encoder.norm1 | (batch_size, 512) | 1,024 |\n| | encoder.norm2 | (batch_size, 512) | 1,024 |\n| Logit | Linear | (batch_size, 250) | 12,8250 |\n\n* Learning Rate : $1.0 × 10^{-4}$\n \n* Dropout Probability : $p = 0.3$ \n\n* Epoches : $200$\n### Train\n* We use $AdamW$ optimizer for training.(The best-performing in our experiments)\n \n* The activation function we used is called $SWISH$, which was proposed by Google.\nThe SWISH activation function is a non-linear function similar to ReLU, but it performs better in some cases than ReLU. This is because it can produce stronger regularization effects.\n\n* CrossEntropy (with label smoothing which is useful)\n\n\n**Finally, our single model can achieve 0.77.**\n\n## Ensemble\nThe model ensemble was not so great, so we won't post it here to mislead anyone. We just took a simple average of the class probabilities obtained from each model.\n\n## Shortcoming\n* The differences between models are not significant, which may be the main reason why the fusion did not improve significantly.\n \n* The fusion method did not work well.\n\n## Thanks\nThank [🐸](https://www.kaggle.com/hengck23) very much for providing the code and ideas.", "votes": null }, { "id": "2242932", "postDate": "05/02/2023 15:28:39", "content": "@bugooman thanks for sharing your approach! 🤝👍 Congrats with the silver!🎉
", "rawMarkdown": "bugooman thanks for sharing your approach! 🤝👍 Congrats with the silver!🎉", "votes": null } ], "comments": [ { "id": 2242932, "author_name": "ivanisaev", "author_url": "", "post_date": "05/02/2023 15:28:39", "content": "@bugooman thanks for sharing your approach! 🤝👍 Congrats with the silver!🎉
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2242732": "## Data Preprocessing\nIn data preprocessing, we did as followed:\n\n* normalization, NaN_filling and Padding\n\n\n* Extract the left and right hand from the entire dot, and only consider the relative position between the dots of the hands.\n```python\nlhand = x[:,len(LIP):len(LIP)+len(LHAND)]\nrhand = x[:,len(LIP)+len(LHAND):]\nrelative_lhand =norm_xy (lhand.unsqueeze(1) - lhand.unsqueeze(2))\nrelative_rhand =norm_xy (rhand.unsqueeze(1) - rhand.unsqueeze(2))\n```\n\n* Add the relative position changes between frames.(If frame=16, we are computing the relative distances between the current frame and the previous 8 frames and the next 8 frames.)\n\n```python\nrelative_back = torch.zeros(x.shape[0],x.shape[1],n//2,2)\nrelative_front = torch.zeros(x.shape[0],x.shape[1],n//2,2)\nfor i in range(n//2):\n off =x[:,i+1:]-x[:,:-i-1]\n relative_back[:,i+1:,i] = off\n relative_front[:,:-i-1,i] = -off\nrelative = torch.cat([relative_back,relative_front],-2)#shape (len(LHAND)*2, max_length, n, 2)\nrelative_xy = norm_xy(relative).permute(1,0,2,3)\n```\n\n* add Original features.\n\n## Augmentation\nWe performed the followings in data augmentation:\n\n* Random frame dropout.(It seems to be a very common processing method in other sign language recognition studies.)\n```python\nif self.p>0:\n indices=[]\n while len(indices) ==0:\n indices = (torch.rand(sample.shape[0]) >=self.fd).nonzero().squeeze(1)\n sample = sample[indices]\nreturn sample\n```\n\n* Flip (without flipping hands).\n```python\nx = x_max - x + x_min\n```\n\n## Model\n### Model Construction and Hyperparameter \n* Model Construction(a typical one):\n\n| | Layer Name | Output Shape | Parameters |\n| --- | --- | --- | --- |\n| TransformerEmbedding | x_embed.fc1 | (batch_size, 128, 135) | 17,408 |\n| | x_embed.fc2 | (batch_size, 128, 135) | 17,408 |\n| | x_embed.fc3 | (batch_size, 128, 672) | 87,168 |\n| | x_embed.fc4 | (batch_size, 128, 672) | 87,168 |\n| | x_embed.fc5 | (batch_size, 128, 882) | 113,376 |\n| | x_embed.fc6 | (batch_size, 128, 882) | 113,376 |\n| | x_embed.fc | (batch_size, 128, 512) | 442,496 |\n| LayerNorm | norm | (batch_size, 512) | 1,024 |\n| TransformerBlock | encoder.attn.fc_q | (batch_size, 512) | 262,656 |\n| | encoder.attn.fc_k | (batch_size, 512) | 262,656 |\n| | encoder.attn.fc_v | (batch_size, 512) | 262,656 |\n| | encoder.attn.fc_o | (batch_size, 512) | 262,656 |\n| | encoder.norm1 | (batch_size, 512) | 1,024 |\n| | encoder.norm2 | (batch_size, 512) | 1,024 |\n| Logit | Linear | (batch_size, 250) | 12,8250 |\n\n* Learning Rate : $1.0 × 10^{-4}$\n \n* Dropout Probability : $p = 0.3$ \n\n* Epoches : $200$\n### Train\n* We use $AdamW$ optimizer for training.(The best-performing in our experiments)\n \n* The activation function we used is called $SWISH$, which was proposed by Google.\nThe SWISH activation function is a non-linear function similar to ReLU, but it performs better in some cases than ReLU. This is because it can produce stronger regularization effects.\n\n* CrossEntropy (with label smoothing which is useful)\n\n\n**Finally, our single model can achieve 0.77.**\n\n## Ensemble\nThe model ensemble was not so great, so we won't post it here to mislead anyone. We just took a simple average of the class probabilities obtained from each model.\n\n## Shortcoming\n* The differences between models are not significant, which may be the main reason why the fusion did not improve significantly.\n \n* The fusion method did not work well.\n\n## Thanks\nThank [🐸](https://www.kaggle.com/hengck23) very much for providing the code and ideas.", "2242932": "bugooman thanks for sharing your approach! 🤝👍 Congrats with the silver!🎉" }, "source": "meta" }