![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F980bafe93f77c3d2418f5512ebfb2aa3%2Fpipeline.png?generation=1682985827786639&alt=media\")
I thank Kaggle administraror & host for holding this competition. Although struggling with Tensorflow's unfriendly errors and lots of try-and-errors for failing TFLite conversion was really, really tough, these low-layer experience was valuable for me.
\nBelow is my solution writeup of this competition.
I tried over 377 different patterns of training models for this competition, however, the best architecture is only minor-changed one from Mark Wijkhuizen's great public notebook.
\n
I tested a) PostLN, b) PreLN, c) Mixed architectures, and found Mixed architecture provides the best result. This architecture was originally (perhaps unintentionally) implemented in the Mark Wijkhuizen's public notebook (in earlier version).
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fe045ca21bb34a220006c06d9306446aa%2Ftransformer_block.png?generation=1682985844553627&alt=media\"){kind=spacer}
0.5 * ArcFace + 0.5 * CrossEntropy (This setting was shared by Med Ali Bouchhioua). Using ArcFace loss together with cross entropy loss converges faster, as well as the accuracy gets better.The augmentation strategy is almost the same as that of @hengck23 was shared.
\nFrame masking simulates the mis-detection of keypoins. It also act as cutout augmentation in image tasks.
\nWhen FP16 quantization is applied for the Mark's original implementation, the Transformer block outputs NaN value. So I rewrote transformer block based on @henck23's implementation.
\nFP16 quantization made model size about 1/2, without losing model accuracy.
I thank Kaggle administraror & host for holding this competition. Although struggling with Tensorflow's unfriendly errors and lots of try-and-errors for failing TFLite conversion was really, really tough, these low-layer experience was valuable for me.
\nBelow is my solution writeup of this competition.
I tried over 377 different patterns of training models for this competition, however, the best architecture is only minor-changed one from Mark Wijkhuizen's great public notebook.
\n
I tested a) PostLN, b) PreLN, c) Mixed architectures, and found Mixed architecture provides the best result. This architecture was originally (perhaps unintentionally) implemented in the Mark Wijkhuizen's public notebook (in earlier version).
\n
0.5 * ArcFace + 0.5 * CrossEntropy (This setting was shared by Med Ali Bouchhioua). Using ArcFace loss together with cross entropy loss converges faster, as well as the accuracy gets better.The augmentation strategy is almost the same as that of @hengck23 was shared.
\nFrame masking simulates the mis-detection of keypoins. It also act as cutout augmentation in image tasks.
\nWhen FP16 quantization is applied for the Mark's original implementation, the Transformer block outputs NaN value. So I rewrote transformer block based on @henck23's implementation.
\nFP16 quantization made model size about 1/2, without losing model accuracy.
Fantastic approach note and excellent result @tatamikenn, hearty congratulations for the result! All the best and regards!
", "rawMarkdown": "Fantastic approach note and excellent result @tatamikenn, hearty congratulations for the result! All the best and regards!", "votes": null } ], "comments": [ { "id": 2241961, "author_name": "ravi20076", "author_url": "", "post_date": "05/02/2023 01:18:37", "content": "Fantastic approach note and excellent result @tatamikenn, hearty congratulations for the result! All the best and regards!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2241888": "# Summary\n\nI thank Kaggle administraror & host for holding this competition. Although struggling with Tensorflow's unfriendly errors and lots of try-and-errors for failing TFLite conversion was really, really tough, these low-layer experience was valuable for me.\nBelow is my solution writeup of this competition.\n\n## Pipeline\n\nI tried over 377 different patterns of training models for this competition, however, the best architecture is only minor-changed one from [Mark Wijkhuizen's great public notebook](https://www.kaggle.com/code/markwijkhuizen/gislr-tf-data-processing-transformer-training).\n\n\n\n### Mixed PostLN & PreLN Architecture\n\nI tested a) PostLN, b) PreLN, c) Mixed architectures, and found Mixed architecture provides the best result. This architecture was originally (perhaps unintentionally) implemented in the Mark Wijkhuizen's public notebook (in earlier version).\n\n\n\n### Other modifications\n\n* keep frames with no hands (instead of dropping) in pre-processing.\n* increase number of layers in the keypoint encoder. With more layers, the accuracy gets better. The 4x layer seetting is the best tradeoff for accuracy and inference time.\n* set number of hidden units in keypoint encoder independently for each parts (lips=192, left_hand=256, right_hand=256, pose=128). This reduces inference time without losing accuracy.\n* attach ArcFace layer on training.\n\n## Training Setting\n\n* loss function: `0.5 * ArcFace + 0.5 * CrossEntropy` (This setting was shared by [Med Ali Bouchhioua](https://www.kaggle.com/code/medali1992/gislr-nn-arcface-baseline)). Using ArcFace loss together with cross entropy loss converges faster, as well as the accuracy gets better.\n* I tested 50, 80, 100, 120 epochs, but 100 epoch is the best on LB.\n* Label smoothing (0.20-0.25) can avoid overfitting, but ArcFace is better. Using both label smoothing and ArcFace didn't increase CV/LB.\n\n### Data Augmentation\n\nThe augmentation strategy is almost the same as [that of @hengck23 was shared](https://www.kaggle.com/competitions/asl-signs/discussion/391265).\n\n* handedness swapping (p=0.5)\n* global 2D Affine transformation (p=0.5, shift=(-0.1, 0.1), rotation=(-30, 30), scale=(0.9, 1.1), shear=(-1.5, 1.5))\n* random frame masking (p=0.5, mask_ratio=0.75).\n\nFrame masking simulates the mis-detection of keypoins. It also act as cutout augmentation in image tasks.\n\n## TFLite Conversion\n\nWhen FP16 quantization is applied for the Mark's original implementation, the Transformer block outputs NaN value. So I rewrote transformer block based on [@henck23's implementation](https://www.kaggle.com/code/hengck23/lb-0-73-single-fold-transformer-architecture/notebook).\nFP16 quantization made model size about 1/2, without losing model accuracy.\n\n### TTA\n\n* apply handedness swapping augmentation to 2/4 of ensemble-seed models\n\n### The output TFLite model\n\n* model size=26MB\n* scoring time=56-59 min\n\n## Not-worked Experiments\n\n* distance/velocity features didn't contribute to increase accuracy.\n* using synthesized hand pose dataset, I trained z-axis prediction model. I used this model's prediction result for a) 3D-affine transform augmentation and b) sub-task to predict z-axis from key-point encoder's output, but both trials didn't contribute to increase accuracy.", "2241961": "Fantastic approach note and excellent result @tatamikenn, hearty congratulations for the result! All the best and regards!" }, "source": "meta" }