![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5487737%2F49828424985feacbedc991a356c4f9d1%2Fmixup_example.jpg?generation=1683018709427551&alt=media\")
Here is a quick overview of the 8th place solution.
\n3 transformers models, 2 layers each (384 hidden, 512 hidden ffn), with an ffn encoder (512->384), trained from scratch. LR 8e-4 with cosine schedule trained for ~300 epochs, dropout 0.1, batch size 1024, label smoothing 0.1. Using hands, lips and pose (above waist only). On one transformer all pose and a subset of lips were used for diversity.
\nAugmentations
\nPreprocessing
\n[1, 2, 3, 4, 6, 8, 12, 16]. It was important to forward fill NANs as opposed to 0-fill. Tflite
\nTraining was all in pytorch. Converting to tflite, the feature preprocessing was rewritten in tensorflow, and the base models were converted via onnx to tf (this turned out to be a mistake from looking at #2 solution, I should have rewritten the transformer encoder). It was a great opportunity to learn tensorflow. I like it.
\nA funny thing⦠doing the following on the pytorch model model = model.half().float() before converting to onnx, gave a good speed up in the final tflite inference. I tried quantizing the pytorch model and it did not further increase the speed.
In the last week I found in preprocessing, normalising points across the whole sequence channel wise gave a good lift. Particularly what I tried to do was for each body part get the channel wise range (max-min), then get the average range over all frames, and use this to normalise the time deltas and the raw coordinates - as opposed to normalising each frame independently. Every time I submitted this it gave a very low score after ~25 mins. I rewrote it in different ways a number of times and it worked with tfilte in kaggle kernels but not when submitted. It was very frustrating at the end. However, I do not think the boost would have brought me into the prize range which was the goal π€
\nFailed attempts
\nHere is a quick overview of the 8th place solution.
\n3 transformers models, 2 layers each (384 hidden, 512 hidden ffn), with an ffn encoder (512->384), trained from scratch. LR 8e-4 with cosine schedule trained for ~300 epochs, dropout 0.1, batch size 1024, label smoothing 0.1. Using hands, lips and pose (above waist only). On one transformer all pose and a subset of lips were used for diversity.
\nAugmentations
\nPreprocessing
\n[1, 2, 3, 4, 6, 8, 12, 16]. It was important to forward fill NANs as opposed to 0-fill. Tflite
\nTraining was all in pytorch. Converting to tflite, the feature preprocessing was rewritten in tensorflow, and the base models were converted via onnx to tf (this turned out to be a mistake from looking at #2 solution, I should have rewritten the transformer encoder). It was a great opportunity to learn tensorflow. I like it.
\nA funny thing⦠doing the following on the pytorch model model = model.half().float() before converting to onnx, gave a good speed up in the final tflite inference. I tried quantizing the pytorch model and it did not further increase the speed.
In the last week I found in preprocessing, normalising points across the whole sequence channel wise gave a good lift. Particularly what I tried to do was for each body part get the channel wise range (max-min), then get the average range over all frames, and use this to normalise the time deltas and the raw coordinates - as opposed to normalising each frame independently. Every time I submitted this it gave a very low score after ~25 mins. I rewrote it in different ways a number of times and it worked with tfilte in kaggle kernels but not when submitted. It was very frustrating at the end. However, I do not think the boost would have brought me into the prize range which was the goal π€
\nFailed attempts
\nCongratulations! Actually it is very impressive that you could find out what caused a worse result in such a short time period!
", "rawMarkdown": "Congratulations! Actually it is very impressive that you could find out what caused a worse result in such a short time period!", "votes": null }, { "id": "2242424", "postDate": "05/02/2023 09:12:18", "content": "We didn't fine-tune without mixup, beta was chosen to be default=1.0 (thus uniform lambda factor sampling) and it worked well, - gray curve with mixup, green curve without it (loss is BCE w/ label-smoothing 0.8)
\n
Label smoothing 0.8 ! π β¦ I did not try that.
\nUnderstood on the mixup - I tried mixing up the FFN output, I think it helped my CV but hurt public so I gave up. Your chart looks compelling though. Thanks for sharing.
Congrats with a gold medal!
\n\n\nlooked like I had an incorrect normalisation from #2 solution
\n
Normalization for us improves with only ~0.005 LB. Maybe you weren't lucky enough with starting hyper parameters π
\n\n\nLooks like I missed to rewrite it into tf, or train in tf
\n
Also efficientnet with basic torch-ONNX-TFLite converters has 60-70ms inference. But you need to use some specific converters like this or this because they dealing with HWC tensor. While the ordinal pipeline use transposes before each Conv layer which make it veeery slow. (You can check the graph on netron)
", "rawMarkdown": "Congrats with a gold medal!\n\n> looked like I had an incorrect normalisation from #2 solution\n\nNormalization for us improves with only ~0.005 LB. Maybe you weren't lucky enough with starting hyper parameters π\n\n> Looks like I missed to rewrite it into tf, or train in tf\n\nAlso efficientnet with basic torch-ONNX-TFLite converters has 60-70ms inference. But you need to use some specific converters like [this](https://github.com/AlexanderLutsenko/nobuco) or [this](https://github.com/MPolaris/onnx2tflite) because they dealing with HWC tensor. While the ordinal pipeline use transposes before each Conv layer which make it veeery slow. (You can check the graph on netron)", "votes": null }, { "id": "2242504", "postDate": "05/02/2023 10:24:48", "content": "Looking at the logs for efficientnet-b1 it did as well as the transformers using 1e-3 lr with mixup; but this was passing points thru an ffn first which has extra overhead. If I remember correctly when I submitted it was getting 3 or 4 samples per second which was way too slow. Will know next time if I am doing this work with onnx to check netron and the links you mentioned. I gave up too early here.
", "rawMarkdown": "Looking at the logs for efficientnet-b1 it did as well as the transformers using 1e-3 lr with mixup; but this was passing points thru an ffn first which has extra overhead. If I remember correctly when I submitted it was getting 3 or 4 samples per second which was way too slow. Will know next time if I am doing this work with onnx to check netron and the links you mentioned. I gave up too early here.", "votes": null }, { "id": "2242821", "postDate": "05/02/2023 14:29:24", "content": "@darraghdog congratulations with gold medal!π Thank you for sharing explanation of your solution - in particular it very helpful for someone like me to improve skills!π
", "rawMarkdown": "darraghdog congratulations with gold medal!π Thank you for sharing explanation of your solution - in particular it very helpful for someone like me to improve skills!π", "votes": null } ], "comments": [ { "id": 2242415, "author_name": "skybookreader", "author_url": "", "post_date": "05/02/2023 08:55:46", "content": "Congratulations! Actually it is very impressive that you could find out what caused a worse result in such a short time period!
", "votes": null, "replies": [] }, { "id": 2242424, "author_name": "martynoveduard", "author_url": "", "post_date": "05/02/2023 09:12:18", "content": "We didn't fine-tune without mixup, beta was chosen to be default=1.0 (thus uniform lambda factor sampling) and it worked well, - gray curve with mixup, green curve without it (loss is BCE w/ label-smoothing 0.8)
\n
Label smoothing 0.8 ! π β¦ I did not try that.
\nUnderstood on the mixup - I tried mixing up the FFN output, I think it helped my CV but hurt public so I gave up. Your chart looks compelling though. Thanks for sharing.
Congrats with a gold medal!
\n\n\nlooked like I had an incorrect normalisation from #2 solution
\n
Normalization for us improves with only ~0.005 LB. Maybe you weren't lucky enough with starting hyper parameters π
\n\n\nLooks like I missed to rewrite it into tf, or train in tf
\n
Also efficientnet with basic torch-ONNX-TFLite converters has 60-70ms inference. But you need to use some specific converters like this or this because they dealing with HWC tensor. While the ordinal pipeline use transposes before each Conv layer which make it veeery slow. (You can check the graph on netron)
", "votes": null, "replies": [ { "id": 2242504, "author_name": "darraghdog", "author_url": "", "post_date": "05/02/2023 10:24:48", "content": "Looking at the logs for efficientnet-b1 it did as well as the transformers using 1e-3 lr with mixup; but this was passing points thru an ffn first which has extra overhead. If I remember correctly when I submitted it was getting 3 or 4 samples per second which was way too slow. Will know next time if I am doing this work with onnx to check netron and the links you mentioned. I gave up too early here.
", "votes": null, "replies": [] } ] }, { "id": 2242821, "author_name": "ivanisaev", "author_url": "", "post_date": "05/02/2023 14:29:24", "content": "@darraghdog congratulations with gold medal!π Thank you for sharing explanation of your solution - in particular it very helpful for someone like me to improve skills!π
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2242371": "Here is a quick overview of the 8th place solution. \n\n3 transformers models, 2 layers each (384 hidden, 512 hidden ffn), with an ffn encoder (512->384), trained from scratch. LR 8e-4 with cosine schedule trained for ~300 epochs, dropout 0.1, batch size 1024, label smoothing 0.1. Using hands, lips and pose (above waist only). On one transformer all pose and a subset of lips were used for diversity. \n\n**Augmentations**\n- most important was sequence cutout. On each sample, and each body part (left hand, right hand, lips, pose) with a 0.4 proba convert to nan 5 random slices of 0.15 x SequenceLength. It was hard to overfit with this in.\n- mirror left\n- random rotate. \n\n**Preprocessing**\n- Linear interpolation of longer sequences to max length of 96. \n- Normalise each body part, using min max - I found this better than mean/std. In one model I used mean/std for diversity. \n- Create time shift delta features on a subset of points, using time shifts of `[1, 2, 3, 4, 6, 8, 12, 16]`. It was important to forward fill NANs as opposed to 0-fill. \n- Angle features between xyz position, and xy positions, on a few different joints - corner of mouth, hand, arms. \n- Some point to point distances, interestingly this did not help much. \n\n**Tflite**\nTraining was all in pytorch. Converting to tflite, the feature preprocessing was rewritten in tensorflow, and the base models were converted via onnx to tf (this turned out to be a mistake from looking at #2 solution, I should have rewritten the transformer encoder). It was a great opportunity to learn tensorflow. I like it. \nA funny thing... doing the following on the pytorch model `model = model.half().float()` before converting to onnx, gave a good speed up in the final tflite inference. I tried quantizing the pytorch model and it did not further increase the speed. \n\nIn the last week I found in preprocessing, normalising points across the whole sequence channel wise gave a good lift. Particularly what I tried to do was for each body part get the channel wise range (max-min), then get the average range over all frames, and use this to normalise the time deltas and the raw coordinates - as opposed to normalising each frame independently. Every time I submitted this it gave a very low score after ~25 mins. I rewrote it in different ways a number of times and it worked with tfilte in kaggle kernels but not when submitted. It was very frustrating at the end. However, I do not think the boost would have brought me into the prize range which was the goal π€\n\n**Failed attempts**\n- Mixup worked (~~as mentioned by #9 team~~ worked for #9 team), I tried it on the embeddings and it worked ok, ~~but I did not finetune the result on the data without mixup, which probably would have helped~~. See comments below on mixup.\n- CNNs with mixup - tried efficientnet and edgenext. I tried on raw points and it did not work, looked like I had an incorrect normalisation from #2 solution. It worked well with an FFN encoder, but was too slow in inference. Looks like I missed to rewrite it into tf, or train in tf.", "2242415": "Congratulations! Actually it is very impressive that you could find out what caused a worse result in such a short time period!", "2242424": "We didn't fine-tune without mixup, beta was chosen to be default=1.0 (thus uniform lambda factor sampling) and it worked well, - gray curve with mixup, green curve without it (loss is BCE w/ label-smoothing 0.8)\n\n", "2242431": "Label smoothing 0.8 ! π ... I did not try that. \nUnderstood on the mixup - I tried mixing up the FFN output, I think it helped my CV but hurt public so I gave up. Your chart looks compelling though. Thanks for sharing.", "2242460": "Congrats with a gold medal!\n\n> looked like I had an incorrect normalisation from #2 solution\n\nNormalization for us improves with only ~0.005 LB. Maybe you weren't lucky enough with starting hyper parameters π\n\n> Looks like I missed to rewrite it into tf, or train in tf\n\nAlso efficientnet with basic torch-ONNX-TFLite converters has 60-70ms inference. But you need to use some specific converters like [this](https://github.com/AlexanderLutsenko/nobuco) or [this](https://github.com/MPolaris/onnx2tflite) because they dealing with HWC tensor. While the ordinal pipeline use transposes before each Conv layer which make it veeery slow. (You can check the graph on netron)", "2242504": "Looking at the logs for efficientnet-b1 it did as well as the transformers using 1e-3 lr with mixup; but this was passing points thru an ffn first which has extra overhead. If I remember correctly when I submitted it was getting 3 or 4 samples per second which was way too slow. Will know next time if I am doing this work with onnx to check netron and the links you mentioned. I gave up too early here.", "2242821": "darraghdog congratulations with gold medal!π Thank you for sharing explanation of your solution - in particular it very helpful for someone like me to improve skills!π" }, "source": "meta" }