yy = F.interpolate(yy[None, None, :], size=self.new_size, mode='nearest').(X, Y, Z) axes. ![\"Preprocessing\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2Fc47290891a7ac6497a6a0c296973f071%2Fdata_prep.jpg?generation=1682986293067532&alt=media\")
We used an approach similar to audio spectrogram classification using the EfficientNet-B0 model, with numerous augmentations and transformer models such as BERT and DeBERTa as helper models. The final solution consists of one EfficientNet-B0 with an input size of 160x80, trained on a single fold from 8 randomly split folds, as well as DeBERTa and BERT trained on the full dataset. A single fold model using EfficientNet has a CV score of 0.898 and a leaderboard score of ~0.8.
\nWe used only competition data.
\nyy = F.interpolate(yy[None, None, :], size=self.new_size, mode='nearest').(X, Y, Z) axes. Only 61 points were kept, including 40 lip points and 21 hand points. For left and right hand, the one with less NaN was kept. If right hand was kept, mirror it to left hand.
Augmentations, normalization and NaN-filling were applied sequentially.
Sequences longer than 96 were interpolated to 96. Sequences shorter than 96 were unchanged.
Apart from raw positions, hand-crafted features were also used, including motion, distances, and cosine of angles.
Motion features consist of future motion and history motion, which can be denoted as:
$$
\n Motion_{future} = position_{t+1} - position_{t}
\n$$
\n$$
\n Motion_{history} = position_{t} - position_{t-1}
\n$$
Full 210 pairwise distances among 21 hand points were included.
There are 5 vertices in a finger (e.g. thumb is [0,1,2,3,4]), and therefore, there are 3 angles: <0,1,2>, <1,2,3>, <2,3,4>. So 15 angles of 5 fingers were included.
Randomly selected 190 pairwise distances and randomly selected 8 angles among 40 lip points were included.
\n\nThese augmentations are used in both CNN training and transformer training
\n
Random affine: Same as @hengck23 shared. In CNN, after global affine, shift-scale-rotate was also applied to each part separately (e.g. hand, lip, body-pose).
Random interpolation: Slightly scale and shift the time dimension.
Flip pose: Flip the x-coordinates of all points. In CNN, x_new = x_max - x_old. In transformer, x_new = 2 * frame[:,0,0] - x_old.
Finger tree rotate: There are 4 root-children pairs in a finger with 5-vertices. E.g. in thumb ([0,1,2,3,4]), these 4 root-children pairs are: 0-[1,2,3,4],1-[2,3,4],2-[3,4],3-[4]. We randomly choose some of these pairs, and rotate the children points around root point with a small random angle.
Mixup: Implement basic mixup augmentation (only works with CNNs, not transformers).Replace augmentation: Replace some random parts from other samples of the same class.Time and frequence masking: This basic torchaudio augmentation works exceptionally well.freq_m = torchaudio.transforms.FrequencyMasking() \ntime_m = torchaudio.transforms.TimeMasking() \nBefore augmentation:
\n![\"aug1\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F5d2b97cf6754c5f4063724181bfe7172%2Fbefore_aug.png?generation=1682986332091937&alt=media\")
After augmentation:
\n ![\"aug2\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F516f0790f6f9903ad8943977bc392965%2Fafter_aug.png?generation=1682986413818912&alt=media\")
CrossEntropyLoss. Increase the weights for poorly predicted classes and classes with semantically similar pairs (such as kitty and cat)Since we trained only one fold and used smaller models, we decided to tune most parameters with Optuna.
\nHere is the parameters list of CNN training (transformer training has a similar param-list):
\nAll augmentations probabilities (0.1 - 0.5+)
Learning rate (2e-3 - 3e-3)
Drop out (0.1 - 0.25)
Num of epochs (170-185)
Loss weights powers (0.75 - 2)
Optimizer (Lookahead_RAdam, RAdam)
Label smoothing (0.5 - 0.7)
We rewrote all our models in Keras and transferred PyTorch weights to them, resulting in a speed boost of around 30%. For transformer model, pytorch-onnx-tf-tflite will generate too much useless tensor shape operations, a fully rewriting can reduce these manually. For CNN model, we rewrote DepthwiseConv2D with a hard-coded way, whose speed is 200%~300% of its original version of tflite DepthwiseConv2D.
After that, we aggregated all these models in the tf.Module class. Converting directly from Keras resulted in lower speed (don't know why).
We calculated ensemble weights for models trained on fold 0 using the local fold 0 score and applied these weights to the full dataset models.
EfficientNet-B0 achieved a leaderboard score of approximately 0.8, and transformers improved the score to 0.81. The final ensemble included:
\nInterestingly, a key feature was using the ensemble without softmax, which consistently provided a boost of around 0.01.
\nDepthwise convolution models performed very well for these tasks, outperforming other CNN and ViT models (rexnet_100 was also good).
\nWe spent a lot of time dealing with the conversion of DepthwiseConv2D operation. Here are some strange results:
Given a input image with 82x42x32 (HWC), there are two ways to do a 3x3 depthwise convolution in Keras. One is Conv2D(32, 3, groups = 32), the other is DepthwiseConv2D(3). However, after converting these two to tflite, the running time of the Conv2D is 5.05ms, and the running time of DepthwiseConv2D is 3.70ms. More strangely, a full convolution Conv2D(32, 3, groups = 1) with FLOPs = HWC^2 only takes 2.09ms, even faster than previous two with FLOPs = HWC.
Then we rewrote the depthwise-conv like this:
\n ():\n out = x[:,:self.H_out:self.strides,:self.W_out:self.strides] * self.weight[,]\n i (self.kernel_size):\n j (self.kernel_size):\n i == j == :\n \n out += x[:,i:self.H_out + i:self.strides,j:self.W_out + j:self.strides] * self.weight[i,j]\n self.bias :\n out = out + self.bias\n out\nThe running time of this is 1.24 ms.
\nIn summary, our version (1.24ms) > full Conv2D with larger FLOPs (2.09ms) > DepthwiseConv2D (3.70ms) > Conv2D(C, groups = C) (5.05ms).
However, our version introduced too much nodes in tflite graph, which is not stable in running time. If the tensorflow team has a better implementation of DepthwiseConv2D, we can even ensemble two CNN models, which is expected to reach 0.82 LB.
\nBy the way, EfficientNet with ONNX was ~5 times faster than TFLite.
\nWe used an approach similar to audio spectrogram classification using the EfficientNet-B0 model, with numerous augmentations and transformer models such as BERT and DeBERTa as helper models. The final solution consists of one EfficientNet-B0 with an input size of 160x80, trained on a single fold from 8 randomly split folds, as well as DeBERTa and BERT trained on the full dataset. A single fold model using EfficientNet has a CV score of 0.898 and a leaderboard score of ~0.8.
\nWe used only competition data.
\nyy = F.interpolate(yy[None, None, :], size=self.new_size, mode='nearest').(X, Y, Z) axes. Only 61 points were kept, including 40 lip points and 21 hand points. For left and right hand, the one with less NaN was kept. If right hand was kept, mirror it to left hand.
Augmentations, normalization and NaN-filling were applied sequentially.
Sequences longer than 96 were interpolated to 96. Sequences shorter than 96 were unchanged.
Apart from raw positions, hand-crafted features were also used, including motion, distances, and cosine of angles.
Motion features consist of future motion and history motion, which can be denoted as:
$$
\n Motion_{future} = position_{t+1} - position_{t}
\n$$
\n$$
\n Motion_{history} = position_{t} - position_{t-1}
\n$$
Full 210 pairwise distances among 21 hand points were included.
There are 5 vertices in a finger (e.g. thumb is [0,1,2,3,4]), and therefore, there are 3 angles: <0,1,2>, <1,2,3>, <2,3,4>. So 15 angles of 5 fingers were included.
Randomly selected 190 pairwise distances and randomly selected 8 angles among 40 lip points were included.
\n\nThese augmentations are used in both CNN training and transformer training
\n
Random affine: Same as @hengck23 shared. In CNN, after global affine, shift-scale-rotate was also applied to each part separately (e.g. hand, lip, body-pose).
Random interpolation: Slightly scale and shift the time dimension.
Flip pose: Flip the x-coordinates of all points. In CNN, x_new = x_max - x_old. In transformer, x_new = 2 * frame[:,0,0] - x_old.
Finger tree rotate: There are 4 root-children pairs in a finger with 5-vertices. E.g. in thumb ([0,1,2,3,4]), these 4 root-children pairs are: 0-[1,2,3,4],1-[2,3,4],2-[3,4],3-[4]. We randomly choose some of these pairs, and rotate the children points around root point with a small random angle.
Mixup: Implement basic mixup augmentation (only works with CNNs, not transformers).Replace augmentation: Replace some random parts from other samples of the same class.Time and frequence masking: This basic torchaudio augmentation works exceptionally well.freq_m = torchaudio.transforms.FrequencyMasking() \ntime_m = torchaudio.transforms.TimeMasking() \nBefore augmentation:
\n
After augmentation:
\n
CrossEntropyLoss. Increase the weights for poorly predicted classes and classes with semantically similar pairs (such as kitty and cat)Since we trained only one fold and used smaller models, we decided to tune most parameters with Optuna.
\nHere is the parameters list of CNN training (transformer training has a similar param-list):
\nAll augmentations probabilities (0.1 - 0.5+)
Learning rate (2e-3 - 3e-3)
Drop out (0.1 - 0.25)
Num of epochs (170-185)
Loss weights powers (0.75 - 2)
Optimizer (Lookahead_RAdam, RAdam)
Label smoothing (0.5 - 0.7)
We rewrote all our models in Keras and transferred PyTorch weights to them, resulting in a speed boost of around 30%. For transformer model, pytorch-onnx-tf-tflite will generate too much useless tensor shape operations, a fully rewriting can reduce these manually. For CNN model, we rewrote DepthwiseConv2D with a hard-coded way, whose speed is 200%~300% of its original version of tflite DepthwiseConv2D.
After that, we aggregated all these models in the tf.Module class. Converting directly from Keras resulted in lower speed (don't know why).
We calculated ensemble weights for models trained on fold 0 using the local fold 0 score and applied these weights to the full dataset models.
EfficientNet-B0 achieved a leaderboard score of approximately 0.8, and transformers improved the score to 0.81. The final ensemble included:
\nInterestingly, a key feature was using the ensemble without softmax, which consistently provided a boost of around 0.01.
\nDepthwise convolution models performed very well for these tasks, outperforming other CNN and ViT models (rexnet_100 was also good).
\nWe spent a lot of time dealing with the conversion of DepthwiseConv2D operation. Here are some strange results:
Given a input image with 82x42x32 (HWC), there are two ways to do a 3x3 depthwise convolution in Keras. One is Conv2D(32, 3, groups = 32), the other is DepthwiseConv2D(3). However, after converting these two to tflite, the running time of the Conv2D is 5.05ms, and the running time of DepthwiseConv2D is 3.70ms. More strangely, a full convolution Conv2D(32, 3, groups = 1) with FLOPs = HWC^2 only takes 2.09ms, even faster than previous two with FLOPs = HWC.
Then we rewrote the depthwise-conv like this:
\n ():\n out = x[:,:self.H_out:self.strides,:self.W_out:self.strides] * self.weight[,]\n i (self.kernel_size):\n j (self.kernel_size):\n i == j == :\n \n out += x[:,i:self.H_out + i:self.strides,j:self.W_out + j:self.strides] * self.weight[i,j]\n self.bias :\n out = out + self.bias\n out\nThe running time of this is 1.24 ms.
\nIn summary, our version (1.24ms) > full Conv2D with larger FLOPs (2.09ms) > DepthwiseConv2D (3.70ms) > Conv2D(C, groups = C) (5.05ms).
However, our version introduced too much nodes in tflite graph, which is not stable in running time. If the tensorflow team has a better implementation of DepthwiseConv2D, we can even ensemble two CNN models, which is expected to reach 0.82 LB.
\nBy the way, EfficientNet with ONNX was ~5 times faster than TFLite.
\n
Congrats for the 2nd place!
\nI haven't even imagine treating this data as image⦠(I like audio spectrogram analogy by the way!)
\nWell deserved!
Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 2242329, "postDate": "2023-05-02T07:31:49.030Z", "content": "Very nice work for both branches (CNN, Transformer) of the solution. How did you identify Depthwise convolution particularly as the bottleneck ? I tried efficientnet but it was too slow for me⦠however, I didnt rewrite the base model in to tf, only the prepreprocessing. I see that transferring the base model to keras gave a 30% speed up.
\nAlso, was the ordering of the xyz points important in the CNN. I guess you would want adjacent points beside each other in the CNN. But some points, like wrist, have multiple adjacent points.
", "rawMarkdown": "Very nice work for both branches (CNN, Transformer) of the solution. How did you identify Depthwise convolution particularly as the bottleneck ? I tried efficientnet but it was too slow for me... however, I didnt rewrite the base model in to tf, only the prepreprocessing. I see that transferring the base model to keras gave a 30% speed up. \n\nAlso, was the ordering of the xyz points important in the CNN. I guess you would want adjacent points beside each other in the CNN. But some points, like wrist, have multiple adjacent points. ", "votes": 4, "replies": [ { "id": 2242520, "postDate": "2023-05-02T10:36:24.830Z", "content": "About identification I hope Carno will answer to you later (he found that)
\nAbout the ordering:
\nWe arranged them in groups (lips, left hand, pose, right hand), and within groups, in the order in which they are arranged in the raw data. Didn't tried any other orderings. But tried different amount of points and this combination of 80 points was the best
I use Tensorflow's benchmark tool to analyze Node-wise running time. But DepthwiseConv is slower than what I expected.
", "rawMarkdown": "I use Tensorflow's benchmark tool to analyze Node-wise running time. But DepthwiseConv is slower than what I expected.", "votes": 3 } ] }, { "id": 2241922, "postDate": "2023-05-02T00:44:29.260Z", "content": "By the way, for us in public LB score for full data train CNN versions was worse and we haven't select it. But in private they were better. But still lower than @hoyso48 . That's a pure victory on last hours, GZ!
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F10ba2123e7c100e364688e095a50b569%2Ftg_image_455350557.jpeg?generation=1682988215699105&alt=media\")
Congrats on the great result. Thanks for the detailed write-up. Found a lot of useful tricks that are gonna help in the future π₯
", "rawMarkdown": "Congrats on the great result. Thanks for the detailed write-up. Found a lot of useful tricks that are gonna help in the future π₯", "votes": 1 }, { "id": 2242511, "postDate": "2023-05-02T10:30:59.163Z", "content": "Congrats great approach and good idea to use it as image ππππ₯
", "rawMarkdown": "Congrats great approach and good idea to use it as image ππππ₯", "votes": 1 }, { "id": 2241912, "postDate": "2023-05-02T00:34:15.397Z", "content": "Congratz with the result!
\nAbout DepthwiseConv2D, - I had the same problem when depthwise conv1d was converted not efficiently when using pytorch-onnx-tf pipeline, I used it for positional encoding in transformer.
You can try to check output conversion nodes via tensorflow benchmark that hengck shared, for me it appeared that whole transformer takes 200 ops and single depthwise conv1d takes 1800 ops (???) when using pytorch->onnx->tf conversion pipeline, and it was indeed very slow.
\nI fixed it when I used nobuco, - depthwise convolution converted succesfully using built-in tf function.
", "rawMarkdown": "Congratz with the result! \nAbout DepthwiseConv2D, - I had the same problem when depthwise conv1d was converted not efficiently when using pytorch-onnx-tf pipeline, I used it for positional encoding in transformer.\n\nYou can try to check output conversion nodes via tensorflow benchmark that hengck shared, for me it appeared that whole transformer takes 200 ops and single depthwise conv1d takes 1800 ops (???) when using pytorch->onnx->tf conversion pipeline, and it was indeed very slow.\n\nI fixed it when I used [nobuco](https://github.com/AlexanderLutsenko/nobuco), - depthwise convolution converted succesfully using built-in tf function.", "votes": 1, "replies": [ { "id": 2241917, "postDate": "2023-05-02T00:37:31.773Z", "content": "Thanks. Congrats to you too!
\nYeah, we tried 'nobuco' and some other converters with ONNX. But they are all slower than our keras version. For example with nobuco we was able to add only one transformer model, with our current two transformers
", "rawMarkdown": "Thanks. Congrats to you too!\n\nYeah, we tried 'nobuco' and some other converters with ONNX. But they are all slower than our keras version. For example with nobuco we was able to add only one transformer model, with our current two transformers", "votes": 2 }, { "id": 2241925, "postDate": "2023-05-02T00:48:17.833Z", "content": "I think what causes DepthwiseConv2D slow is the low-level code of tflite. We can not solve this unless we can use custom ops. In our solution, we use many other ops like Mul, Add, StrideSlice to replace DWConv.
", "rawMarkdown": "I think what causes DepthwiseConv2D slow is the low-level code of tflite. We can not solve this unless we can use custom ops. In our solution, we use many other ops like Mul, Add, StrideSlice to replace DWConv. ", "votes": 2 } ] }, { "id": 2243391, "postDate": "2023-05-02T21:49:33.037Z", "content": "Congratz on the result and on the overall impressive performance !
\n0.81 public was impressive, we guessed you were using CNNs but could not get them to match our transformer performances.
Thanks. Congratulations to you too!
\n\n\nwe guessed you were using CNNs but could not get them to match our transformer performances.
\n
I guess it because we have started CNN at early stage, when transformers didn't get 0.78+ yet, and with simply implementation it has 0.74 LB score (top1 at that moment)
", "rawMarkdown": "Thanks. Congratulations to you too!\n\n>we guessed you were using CNNs but could not get them to match our transformer performances.\n\nI guess it because we have started CNN at early stage, when transformers didn't get 0.78+ yet, and with simply implementation it has 0.74 LB score (top1 at that moment)", "votes": 1 } ] }, { "id": 2242007, "postDate": "2023-05-02T01:59:03.923Z", "content": "Big congratulations, and thanks for write-ups. It's amazing to read your struggle to speed up the inference π
", "rawMarkdown": "Big congratulations, and thanks for write-ups. It's amazing to read your struggle to speed up the inference π", "votes": 2, "replies": [ { "id": 2242012, "postDate": "2023-05-02T02:03:15.260Z", "content": "Thanks to you π
", "rawMarkdown": "Thanks to you π", "votes": 2 } ] }, { "id": 2433850, "postDate": "2023-09-11T22:29:49.607Z", "content": "Thank you so much for posting your solution. Great job!!
", "rawMarkdown": "Thank you so much for posting your solution. Great job!!" }, { "id": 2392704, "postDate": "2023-08-15T20:08:21.507Z", "content": "Hi,
\nCongrats on second place!
We are trying to run the code on the colab file training_notebook_example, and I saw that there is a possibility to run TREF model, which for what I understand run both CNN and transformers model. the problem I faced was choosing this model because I got an error when the name was sent to get_model() and I noticed that there is no option there. Can you help me with that.
\nThank you and congrats again!
", "rawMarkdown": "Hi,\nCongrats on second place!\n\nWe are trying to run the code on the colab file training_notebook_example, and I saw that there is a possibility to run TREF model, which for what I understand run both CNN and transformers model. the problem I faced was choosing this model because I got an error when the name was sent to get_model() and I noticed that there is no option there. Can you help me with that.\n\nThank you and congrats again!" }, { "id": 2381121, "postDate": "2023-08-09T02:25:50.227Z", "content": "Could someone explain this part for the normalization with how you found those values to be replaced by (lign 237 to 243 of data.py) :
\nself.interesting_idx = np.array(LIPS + l_hand + pose + r_hand)
\n # norm params
\n self.x_mean = torch.tensor(0.48282188177108765).float()
\n self.y_mean = torch.tensor(0.6022918224334717).float()
\n self.z_mean = torch.tensor(-0.4806228578090668).float()
\n self.x_std = torch.tensor(0.22193588316440582).float()
\n self.y_std = torch.tensor(0.252733051776886).float()
\n self.z_std = torch.tensor(0.68902188539505).float()
@kolyaforrat great work. I could not find the transfer_transformer function used in inference code. Could you help me find it?
", "rawMarkdown": "@kolyaforrat great work. I could not find the transfer_transformer function used in inference code. Could you help me find it? " }, { "id": 2246172, "postDate": "2023-05-05T00:08:55.600Z", "content": "Great work, 2nd place is impressive. Hope to learn from your approach and use it to solve similar problems.
", "rawMarkdown": "Great work, 2nd place is impressive. Hope to learn from your approach and use it to solve similar problems. " }, { "id": 2245705, "postDate": "2023-05-04T14:34:39.260Z", "content": "Great work! I'm very inspired by your spectrogram classification approach. Very clever! I've been trying to recreate it locally from scratch using your code on github a a guide. Any chance you could share a screenshot of the training curves for the train/val loss and accuracy and learning rate?
", "rawMarkdown": "Great work! I'm very inspired by your spectrogram classification approach. Very clever! I've been trying to recreate it locally from scratch using your code on github a a guide. Any chance you could share a screenshot of the training curves for the train/val loss and accuracy and learning rate?", "replies": [ { "id": 2245792, "postDate": "2023-05-04T15:25:01.003Z", "content": "Thanks!
\nSure, that's my last attempts to train, here is 16 random fold split train (accuracy ~0.005 higher than 8 folds split)
LR:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F0d2b6a6b595b8c5e323fb27a16bc74c8%2Ftg_image_762402010.jpeg?generation=1683213719007236&alt=media\")
Train/Val Loss
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F16063ab6e8101a1f6f88c08be4f7548f%2Ftg_image_1545167309.jpeg?generation=1683213729689919&alt=media\")
Accuracy:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2Feee8989b2b68e37bb0279e44984fa101%2Ftg_image_2756323583.jpeg?generation=1683213753844440&alt=media\")
TopK3:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F622f783745ea25768373fdd9d23a1f36%2Ftg_image_3839074706.jpeg?generation=1683213791599040&alt=media\")
Epochs-accuracy-LR:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F0cd11d2e85c7097bff3c898d1e598aaa%2Ftg_image_3735457178.jpeg?generation=1683213779294885&alt=media\")
This is great! Thanks for sharing. I'm noticing your loss function uses bad_weights.npy and common_weights.npy which you'ver provided in your repo- do you have any more detail about how those were created and how you decided to implement the loss function like this? Thanks!
full_wgt = torch.tensor(bad_wgt ** cfg.pw_bad + common_wgt ** cfg.pw_com, dtype=torch.float32)\nhere: https://github.com/ffs333/2nd_place_GISLR/blob/main/GISLR_utils/losses.py#L13
\nIt looks like your best config used:
\nuse_loss_wgt:True\npw_bad:0.8219710950094926\npw_com:1.3979208248868304\nFirst we trained models with basic CE loss. And then we analyzed the errors of our models as provided here
\nThen just convert it from json to numpy arrays
\nPowers applied to these weights were just hyper parameter we tuned in optuna. At the beginning we started with wide range from 0 to 3
Also we had problems with samples with low lengths (<8). But we didn't found the solution to predict them better. Tried to train or finetune two models:
\nAnd predict it with if else statements in submission. But it has ~the same score and even little worse
", "rawMarkdown": "First we trained models with basic CE loss. And then we analyzed the errors of our models as provided [here](https://www.kaggle.com/kolyaforrat/error-analyse)\n\n* Bad weights - increase weights of bad predicted classes\n* Common weights - increase weights of classes which has similar pairs (like kitty and cat)\n\nThen just convert it from json to numpy arrays\nPowers applied to these weights were just hyper parameter we tuned in optuna. At the beginning we started with wide range from 0 to 3\n\nAlso we had problems with samples with low lengths (<8). But we didn't found the solution to predict them better. Tried to train or finetune two models:\n1. On train data with increased amount of low length data (<=10)\n2. On only long data (>10)\n\nAnd predict it with if else statements in submission. But it has ~the same score and even little worse", "votes": 1 }, { "id": 2246073, "postDate": "2023-05-04T20:33:58.303Z", "content": "Thanks again for answering. Very interesting. I did something similar with looking at common words - but I did the opposite and added additional label smoothing for them. Maybe I should have been doing like you did an increased their weights.
\nI have another quick question, because your pw_bad value is < 1 then you are effectively reducing the weights of the \"bad\" words. Am I understanding that correctly?
Yeah, they become lower than calculated by errors, but higher that just 1 :)
\nFor example weight for class cat = 2, and it becomes 2**0.75~=1.68 in current run if pw_bad = 0.75
\nAs I said we tuned range of loss weights powers from 0, because we didn't know will they help or not
", "rawMarkdown": "Yeah, they become lower than calculated by errors, but higher that just 1 :)\n\nFor example weight for class cat = 2, and it becomes 2**0.75~=1.68 in current run if pw_bad = 0.75\n\nAs I said we tuned range of loss weights powers from 0, because we didn't know will they help or not", "votes": 1 } ] } ] }, { "id": 2244071, "postDate": "2023-05-03T12:14:36.877Z", "content": "So cool to read about your approach, thank you for sharing! Interestingly, as probably many of us, I could not get CNN to work on this problem, now I see why. \"Devil is in the details\". Congrats on the 2nd place, well deserved!
", "rawMarkdown": "So cool to read about your approach, thank you for sharing! Interestingly, as probably many of us, I could not get CNN to work on this problem, now I see why. \"Devil is in the details\". Congrats on the 2nd place, well deserved! " }, { "id": 2243854, "postDate": "2023-05-03T08:23:05.040Z", "content": "Congrats ! You did a great work
", "rawMarkdown": "Congrats ! You did a great work" }, { "id": 2243335, "postDate": "2023-05-02T20:32:58.400Z", "content": "Now just tried to submit single fold solo CNN model and got this score:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4212496%2F867040d30c2f4535cd33fd19989de3d3%2Ftg_image_380692410.jpeg?generation=1683059371986310&alt=media\")
It's 5th place in public and 4th in private with 17.5 MB and 42 minutes
", "rawMarkdown": "Now just tried to submit single fold solo CNN model and got this score:\n\n\nIt's 5th place in public and 4th in private with 17.5 MB and 42 minutes" }, { "id": 2243334, "postDate": "2023-05-02T20:32:33.493Z", "content": "wow great !this will be very helpful this is what in summary i understood-
\nThey used an approach similar to audio spectrogram classification for their solution.
\nThey used EfficientNet-B0 as the main model and applied various augmentations and transformer models such as BERT and DeBERTa as helper models.
\nThe team extracted 80 points for CNN preprocessing and 61 points for Transformer preprocessing.
\nThey used various augmentations such as random affine, random interpolation, flip pose, finger tree rotate, mixup, replace augmentation, time and frequency masking.
\nThey trained on one fold with a random split (8 folds in total) or the full dataset using the best parameters.
\nThey used a onecycle scheduler with 0.1 warmup and a weighted CrossEntropyLoss for CNN training, and a Ranger optimizer with 60% flat and 40% cosine annealing learning rate schedule for Transformer training.
\nThey implemented hyperparameter tuning with Optuna for most parameters.
\nThey rewrote all their models in Keras and transferred PyTorch models to Keras models.
\nThey used an ensemble of different models to improve their performance.
@kolyaforrat @artemtprv @carnozhaocongrats with the second highest place in so competitive contest!π₯π The 5-th paragraph of you post is especially interesting as well as Efficientnet+BERT+DeBERTa ensemble approachππͺπ
", "rawMarkdown": "@kolyaforrat @artemtprv @carnozhaocongrats with the second highest place in so competitive contest!π₯π The 5-th paragraph of you post is especially interesting as well as Efficientnet+BERT+DeBERTa ensemble approachππͺπ" }, { "id": 2243019, "postDate": "2023-05-02T16:32:14.707Z", "content": "Congrats on the perfect result!
\nMay I ask you to clarify a way you use cross-validation? You mentioned Train on one fold with a random split (8 folds in total) or the full dataset using the best parameters. Do you mean you run all 8 folds to validate your model? Or the single fold was used most of the time? Am I also wondering, how do you validate an ensemble of models? I trained all the folds for each model, collect OOF predictions and calculate the ensemble using CV over OOF. It was a time-consuming process and I am not sure whether this is the best way to follow.
\nWould love to learn from your experience
Thanks!
\nWe tested few times for 5 folds, and with random split OOF scores was almost the same for all folds. That's why after this we trained only fold 0 for all splits. We tried 5 folds, 8 folds, 16 folds. 8 folds version was the best
\nAfter finding of some best checkpoint for fold 0 we trained full data checkpoint with the same params.
\nFor getting best ensemble weights we trained all models (eff-b0, bert, deberta) with fold 0 and calculate best weights for ensemble. It was 4 numbers:
\ncnn_wgt * CNN + trans_wgt * (bert * bert_wgt + deberta * deberta_wgt)
\n Also we have tried manually tune weights on LB score but always that fold0 weights was the best.
You can check how we did it in last 2 commented cells in our inference notebook
\nAlso sorry about your shakedown, I know that feeling π
", "rawMarkdown": "Thanks!\n\nWe tested few times for 5 folds, and with random split OOF scores was almost the same for all folds. That's why after this we trained only fold 0 for all splits. We tried 5 folds, 8 folds, 16 folds. 8 folds version was the best\n\nAfter finding of some best checkpoint for fold 0 we trained full data checkpoint with the same params.\n\nFor getting best ensemble weights we trained all models (eff-b0, bert, deberta) with fold 0 and calculate best weights for ensemble. It was 4 numbers: \n```cnn_wgt * CNN + trans_wgt * (bert * bert_wgt + deberta * deberta_wgt)```\n Also we have tried manually tune weights on LB score but always that fold0 weights was the best.\n\nYou can check how we did it in last 2 commented cells in [our inference notebook](https://github.com/ffs333/2nd_place_GISLR/blob/main/kaggle_inference_notebook_example.ipynb)\n\nAlso sorry about your shakedown, I know that feeling π", "votes": 1, "replies": [ { "id": 2243169, "postDate": "2023-05-02T18:02:02.593Z", "content": "Thank you so much for your answer!
\n\n\nAlso sorry about your shakedown, I know that feeling π
\n
Ah, that is painful, taking into account that it was my first real competition I participate from start to end almost evening. Nonetheless, I am looking forward to meeting you in one of the next competitions!
", "rawMarkdown": "Thank you so much for your answer!\n\n> Also sorry about your shakedown, I know that feeling π\n\nAh, that is painful, taking into account that it was my first real competition I participate from start to end almost evening. Nonetheless, I am looking forward to meeting you in one of the next competitions! " } ] } ] }, { "id": 2242218, "postDate": "2023-05-02T06:01:22.063Z", "content": "Could you further explain the meaning of your xyz heatmap? I do not understand why you put it in the repoβ¦
", "rawMarkdown": "Could you further explain the meaning of your xyz heatmap? I do not understand why you put it in the repo...", "replies": [ { "id": 2242237, "postDate": "2023-05-02T06:15:41.597Z", "content": "data is:
\nsample = train.loc[i] \nyy = load_relevant_data_subset(base_dir + sample[]) \nIn training dataset class we just combine all these yy to list and iterate through it. We keep it RAM for faster training.
\nAnd load it like this data = np.load(_cfg.base_path + 'gen_xyz/data.npy', allow_pickle=True)
\nYou can see it in repo in GISLR_utils/mixup_data.py DatasetImageSmall80Mixup
Answer here
", "rawMarkdown": "Answer [here](https://github.com/ffs333/2nd_place_GISLR/issues/1#issuecomment-1555364919)" } ] } ] }, { "id": 2242200, "postDate": "2023-05-02T05:40:05.650Z", "content": "Thanks for sharing, and congratulations!
", "rawMarkdown": "Thanks for sharing, and congratulations!" }, { "id": 2242198, "postDate": "2023-05-02T05:32:59.060Z", "content": "Amazing!Thanks for sharing the detail approaches here.
", "rawMarkdown": "Amazing!Thanks for sharing the detail approaches here." }, { "id": 2242141, "postDate": "2023-05-02T04:06:32.953Z", "content": "@kolyaforrat cool solution! Congrats with 2nd place!
", "rawMarkdown": "@kolyaforrat cool solution! Congrats with 2nd place!" }, { "id": 2242098, "postDate": "2023-05-02T03:25:04.227Z", "content": "Thank you for the explanation! Is the source code going to be published publicly? That would be great.
", "rawMarkdown": "Thank you for the explanation! Is the source code going to be published publicly? That would be great.", "replies": [ { "id": 2242150, "postDate": "2023-05-02T04:20:45.850Z", "content": "Sure, it posted here
", "rawMarkdown": "Sure, it posted [here](https://github.com/ffs333/2nd_place_GISLR)", "votes": 2 } ] }, { "id": 2241955, "postDate": "2023-05-02T01:13:55.517Z", "content": "Hearty congratulations to you for the fantastic result and associated gold medal and best regards @kolyaforrat
", "rawMarkdown": "Hearty congratulations to you for the fantastic result and associated gold medal and best regards @kolyaforrat " }, { "id": 2241929, "postDate": "2023-05-02T00:50:25.557Z", "content": "Very impressive. Thanks for sharing the detail approaches here.
", "rawMarkdown": "Very impressive. Thanks for sharing the detail approaches here. " }, { "id": 2970705, "postDate": "2024-08-26T12:17:03.667Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2242422, "postDate": "2023-05-02T09:07:43Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2246448, "postDate": "2023-05-05T06:47:22.663Z", "content": "Amazing work!
\nThanks for sharing.
Congrats for the 2nd place!
\nI haven't even imagine treating this data as image⦠(I like audio spectrogram analogy by the way!)
\nWell deserved!
Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 2242329, "author_name": "Darragh", "author_url": "", "post_date": "2023-05-02T07:31:49.030000", "content": "Very nice work for both branches (CNN, Transformer) of the solution. How did you identify Depthwise convolution particularly as the bottleneck ? I tried efficientnet but it was too slow for me⦠however, I didnt rewrite the base model in to tf, only the prepreprocessing. I see that transferring the base model to keras gave a 30% speed up.
\nAlso, was the ordering of the xyz points important in the CNN. I guess you would want adjacent points beside each other in the CNN. But some points, like wrist, have multiple adjacent points.
", "votes": 4, "replies": [ { "id": 2242520, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-02T10:36:24.830000", "content": "About identification I hope Carno will answer to you later (he found that)
\nAbout the ordering:
\nWe arranged them in groups (lips, left hand, pose, right hand), and within groups, in the order in which they are arranged in the raw data. Didn't tried any other orderings. But tried different amount of points and this combination of 80 points was the best
I use Tensorflow's benchmark tool to analyze Node-wise running time. But DepthwiseConv is slower than what I expected.
", "votes": 3, "replies": [] } ] }, { "id": 2241922, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-02T00:44:29.260000", "content": "By the way, for us in public LB score for full data train CNN versions was worse and we haven't select it. But in private they were better. But still lower than @hoyso48 . That's a pure victory on last hours, GZ!
\n
Congrats on the great result. Thanks for the detailed write-up. Found a lot of useful tricks that are gonna help in the future π₯
", "votes": 1, "replies": [] }, { "id": 2242511, "author_name": "mhdaw", "author_url": "", "post_date": "2023-05-02T10:30:59.163000", "content": "Congrats great approach and good idea to use it as image ππππ₯
", "votes": 1, "replies": [] }, { "id": 2241912, "author_name": "slime", "author_url": "", "post_date": "2023-05-02T00:34:15.397000", "content": "Congratz with the result!
\nAbout DepthwiseConv2D, - I had the same problem when depthwise conv1d was converted not efficiently when using pytorch-onnx-tf pipeline, I used it for positional encoding in transformer.
You can try to check output conversion nodes via tensorflow benchmark that hengck shared, for me it appeared that whole transformer takes 200 ops and single depthwise conv1d takes 1800 ops (???) when using pytorch->onnx->tf conversion pipeline, and it was indeed very slow.
\nI fixed it when I used nobuco, - depthwise convolution converted succesfully using built-in tf function.
", "votes": 1, "replies": [ { "id": 2241917, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-02T00:37:31.773000", "content": "Thanks. Congrats to you too!
\nYeah, we tried 'nobuco' and some other converters with ONNX. But they are all slower than our keras version. For example with nobuco we was able to add only one transformer model, with our current two transformers
", "votes": 2, "replies": [] }, { "id": 2241925, "author_name": "Carno Zhao", "author_url": "", "post_date": "2023-05-02T00:48:17.833000", "content": "I think what causes DepthwiseConv2D slow is the low-level code of tflite. We can not solve this unless we can use custom ops. In our solution, we use many other ops like Mul, Add, StrideSlice to replace DWConv.
", "votes": 2, "replies": [] } ] }, { "id": 2243391, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-05-02T21:49:33.037000", "content": "Congratz on the result and on the overall impressive performance !
\n0.81 public was impressive, we guessed you were using CNNs but could not get them to match our transformer performances.
Thanks. Congratulations to you too!
\n\n\nwe guessed you were using CNNs but could not get them to match our transformer performances.
\n
I guess it because we have started CNN at early stage, when transformers didn't get 0.78+ yet, and with simply implementation it has 0.74 LB score (top1 at that moment)
", "votes": 1, "replies": [] } ] }, { "id": 2242007, "author_name": "yukiya", "author_url": "", "post_date": "2023-05-02T01:59:03.923000", "content": "Big congratulations, and thanks for write-ups. It's amazing to read your struggle to speed up the inference π
", "votes": 2, "replies": [ { "id": 2242012, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-02T02:03:15.260000", "content": "Thanks to you π
", "votes": 2, "replies": [] } ] }, { "id": 2433850, "author_name": "Johnpaul Nwagwu", "author_url": "", "post_date": "2023-09-11T22:29:49.607000", "content": "Thank you so much for posting your solution. Great job!!
", "votes": 0, "replies": [] }, { "id": 2392704, "author_name": "Adir Eliza", "author_url": "", "post_date": "2023-08-15T20:08:21.507000", "content": "Hi,
\nCongrats on second place!
We are trying to run the code on the colab file training_notebook_example, and I saw that there is a possibility to run TREF model, which for what I understand run both CNN and transformers model. the problem I faced was choosing this model because I got an error when the name was sent to get_model() and I noticed that there is no option there. Can you help me with that.
\nThank you and congrats again!
", "votes": 0, "replies": [] }, { "id": 2381121, "author_name": "Quentin Parrenin", "author_url": "", "post_date": "2023-08-09T02:25:50.227000", "content": "Could someone explain this part for the normalization with how you found those values to be replaced by (lign 237 to 243 of data.py) :
\nself.interesting_idx = np.array(LIPS + l_hand + pose + r_hand)
\n # norm params
\n self.x_mean = torch.tensor(0.48282188177108765).float()
\n self.y_mean = torch.tensor(0.6022918224334717).float()
\n self.z_mean = torch.tensor(-0.4806228578090668).float()
\n self.x_std = torch.tensor(0.22193588316440582).float()
\n self.y_std = torch.tensor(0.252733051776886).float()
\n self.z_std = torch.tensor(0.68902188539505).float()
@kolyaforrat great work. I could not find the transfer_transformer function used in inference code. Could you help me find it?
", "votes": 0, "replies": [] }, { "id": 2246172, "author_name": "Hasan Patel Rodriguez", "author_url": "", "post_date": "2023-05-05T00:08:55.600000", "content": "Great work, 2nd place is impressive. Hope to learn from your approach and use it to solve similar problems.
", "votes": 0, "replies": [] }, { "id": 2245705, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-05-04T14:34:39.260000", "content": "Great work! I'm very inspired by your spectrogram classification approach. Very clever! I've been trying to recreate it locally from scratch using your code on github a a guide. Any chance you could share a screenshot of the training curves for the train/val loss and accuracy and learning rate?
", "votes": 0, "replies": [ { "id": 2245792, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-04T15:25:01.003000", "content": "Thanks!
\nSure, that's my last attempts to train, here is 16 random fold split train (accuracy ~0.005 higher than 8 folds split)
LR:
\n
Train/Val Loss
\n
Accuracy:
\n
TopK3:
\n
Epochs-accuracy-LR:
\n
This is great! Thanks for sharing. I'm noticing your loss function uses bad_weights.npy and common_weights.npy which you'ver provided in your repo- do you have any more detail about how those were created and how you decided to implement the loss function like this? Thanks!
full_wgt = torch.tensor(bad_wgt ** cfg.pw_bad + common_wgt ** cfg.pw_com, dtype=torch.float32)\nhere: https://github.com/ffs333/2nd_place_GISLR/blob/main/GISLR_utils/losses.py#L13
\nIt looks like your best config used:
\nuse_loss_wgt:True\npw_bad:0.8219710950094926\npw_com:1.3979208248868304\nFirst we trained models with basic CE loss. And then we analyzed the errors of our models as provided here
\nThen just convert it from json to numpy arrays
\nPowers applied to these weights were just hyper parameter we tuned in optuna. At the beginning we started with wide range from 0 to 3
Also we had problems with samples with low lengths (<8). But we didn't found the solution to predict them better. Tried to train or finetune two models:
\nAnd predict it with if else statements in submission. But it has ~the same score and even little worse
", "votes": 1, "replies": [] }, { "id": 2246073, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-05-04T20:33:58.303000", "content": "Thanks again for answering. Very interesting. I did something similar with looking at common words - but I did the opposite and added additional label smoothing for them. Maybe I should have been doing like you did an increased their weights.
\nI have another quick question, because your pw_bad value is < 1 then you are effectively reducing the weights of the \"bad\" words. Am I understanding that correctly?
Yeah, they become lower than calculated by errors, but higher that just 1 :)
\nFor example weight for class cat = 2, and it becomes 2**0.75~=1.68 in current run if pw_bad = 0.75
\nAs I said we tuned range of loss weights powers from 0, because we didn't know will they help or not
", "votes": 1, "replies": [] } ] } ] }, { "id": 2244071, "author_name": "Siarhei T", "author_url": "", "post_date": "2023-05-03T12:14:36.877000", "content": "So cool to read about your approach, thank you for sharing! Interestingly, as probably many of us, I could not get CNN to work on this problem, now I see why. \"Devil is in the details\". Congrats on the 2nd place, well deserved!
", "votes": 0, "replies": [] }, { "id": 2243854, "author_name": "Hemangi atale", "author_url": "", "post_date": "2023-05-03T08:23:05.040000", "content": "Congrats ! You did a great work
", "votes": 0, "replies": [] }, { "id": 2243335, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-02T20:32:58.400000", "content": "Now just tried to submit single fold solo CNN model and got this score:
\n
It's 5th place in public and 4th in private with 17.5 MB and 42 minutes
", "votes": 0, "replies": [] }, { "id": 2243334, "author_name": "asjad2024", "author_url": "", "post_date": "2023-05-02T20:32:33.493000", "content": "wow great !this will be very helpful this is what in summary i understood-
\nThey used an approach similar to audio spectrogram classification for their solution.
\nThey used EfficientNet-B0 as the main model and applied various augmentations and transformer models such as BERT and DeBERTa as helper models.
\nThe team extracted 80 points for CNN preprocessing and 61 points for Transformer preprocessing.
\nThey used various augmentations such as random affine, random interpolation, flip pose, finger tree rotate, mixup, replace augmentation, time and frequency masking.
\nThey trained on one fold with a random split (8 folds in total) or the full dataset using the best parameters.
\nThey used a onecycle scheduler with 0.1 warmup and a weighted CrossEntropyLoss for CNN training, and a Ranger optimizer with 60% flat and 40% cosine annealing learning rate schedule for Transformer training.
\nThey implemented hyperparameter tuning with Optuna for most parameters.
\nThey rewrote all their models in Keras and transferred PyTorch models to Keras models.
\nThey used an ensemble of different models to improve their performance.
@kolyaforrat @artemtprv @carnozhaocongrats with the second highest place in so competitive contest!π₯π The 5-th paragraph of you post is especially interesting as well as Efficientnet+BERT+DeBERTa ensemble approachππͺπ
", "votes": 0, "replies": [] }, { "id": 2243019, "author_name": "Mykola", "author_url": "", "post_date": "2023-05-02T16:32:14.707000", "content": "Congrats on the perfect result!
\nMay I ask you to clarify a way you use cross-validation? You mentioned Train on one fold with a random split (8 folds in total) or the full dataset using the best parameters. Do you mean you run all 8 folds to validate your model? Or the single fold was used most of the time? Am I also wondering, how do you validate an ensemble of models? I trained all the folds for each model, collect OOF predictions and calculate the ensemble using CV over OOF. It was a time-consuming process and I am not sure whether this is the best way to follow.
\nWould love to learn from your experience
Thanks!
\nWe tested few times for 5 folds, and with random split OOF scores was almost the same for all folds. That's why after this we trained only fold 0 for all splits. We tried 5 folds, 8 folds, 16 folds. 8 folds version was the best
\nAfter finding of some best checkpoint for fold 0 we trained full data checkpoint with the same params.
\nFor getting best ensemble weights we trained all models (eff-b0, bert, deberta) with fold 0 and calculate best weights for ensemble. It was 4 numbers:
\ncnn_wgt * CNN + trans_wgt * (bert * bert_wgt + deberta * deberta_wgt)
\n Also we have tried manually tune weights on LB score but always that fold0 weights was the best.
You can check how we did it in last 2 commented cells in our inference notebook
\nAlso sorry about your shakedown, I know that feeling π
", "votes": 1, "replies": [ { "id": 2243169, "author_name": "Mykola", "author_url": "", "post_date": "2023-05-02T18:02:02.593000", "content": "Thank you so much for your answer!
\n\n\nAlso sorry about your shakedown, I know that feeling π
\n
Ah, that is painful, taking into account that it was my first real competition I participate from start to end almost evening. Nonetheless, I am looking forward to meeting you in one of the next competitions!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2242218, "author_name": "zj yang", "author_url": "", "post_date": "2023-05-02T06:01:22.063000", "content": "Could you further explain the meaning of your xyz heatmap? I do not understand why you put it in the repoβ¦
", "votes": 0, "replies": [ { "id": 2242237, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-02T06:15:41.597000", "content": "data is:
\nsample = train.loc[i] \nyy = load_relevant_data_subset(base_dir + sample[]) \nIn training dataset class we just combine all these yy to list and iterate through it. We keep it RAM for faster training.
\nAnd load it like this data = np.load(_cfg.base_path + 'gen_xyz/data.npy', allow_pickle=True)
\nYou can see it in repo in GISLR_utils/mixup_data.py DatasetImageSmall80Mixup
Answer here
", "votes": 0, "replies": [] } ] } ] }, { "id": 2242200, "author_name": "JimmyWang", "author_url": "", "post_date": "2023-05-02T05:40:05.650000", "content": "Thanks for sharing, and congratulations!
", "votes": 0, "replies": [] }, { "id": 2242198, "author_name": "gutendzx", "author_url": "", "post_date": "2023-05-02T05:32:59.060000", "content": "Amazing!Thanks for sharing the detail approaches here.
", "votes": 0, "replies": [] }, { "id": 2242141, "author_name": "serangu", "author_url": "", "post_date": "2023-05-02T04:06:32.953000", "content": "@kolyaforrat cool solution! Congrats with 2nd place!
", "votes": 0, "replies": [] }, { "id": 2242098, "author_name": "Emre Kurtoglu", "author_url": "", "post_date": "2023-05-02T03:25:04.227000", "content": "Thank you for the explanation! Is the source code going to be published publicly? That would be great.
", "votes": 0, "replies": [ { "id": 2242150, "author_name": "Kolya Forrat", "author_url": "", "post_date": "2023-05-02T04:20:45.850000", "content": "Sure, it posted here
", "votes": 2, "replies": [] } ] }, { "id": 2241955, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2023-05-02T01:13:55.517000", "content": "Hearty congratulations to you for the fantastic result and associated gold medal and best regards @kolyaforrat
", "votes": 0, "replies": [] }, { "id": 2241929, "author_name": "joejeo1", "author_url": "", "post_date": "2023-05-02T00:50:25.557000", "content": "Very impressive. Thanks for sharing the detail approaches here.
", "votes": 0, "replies": [] }, { "id": 2970705, "author_name": "", "author_url": "", "post_date": "2024-08-26T12:17:03.667000", "content": "", "votes": 0, "replies": [] }, { "id": 2242422, "author_name": "", "author_url": "", "post_date": "2023-05-02T09:07:43", "content": "", "votes": 0, "replies": [] }, { "id": 2246448, "author_name": "", "author_url": "", "post_date": "2023-05-05T06:47:22.663000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2241899": "## TLDR\nWe used an approach similar to audio spectrogram classification using the EfficientNet-B0 model, with numerous augmentations and transformer models such as BERT and DeBERTa as helper models. The final solution consists of one EfficientNet-B0 with an input size of 160x80, trained on a single fold from 8 randomly split folds, as well as DeBERTa and BERT trained on the full dataset. A single fold model using EfficientNet has a CV score of 0.898 and a leaderboard score of ~0.8.\n\nWe used only competition data.\n\n\n\n## 1. Data Preprocessing\n\n### 1.1 CNN Preprocessing\n\n* We extracted 18 lip points, 20 pose points (including arms, shoulders, eyebrows, and nose), and all hand points, resulting in a total of 80 points.\n* During training, we applied various augmentations.\n* We implemented standard normalization.\n* Instead of dropping NaN values, we filled them with zeros after normalization.\n* We interpolated the time axis to a size of 160 using 'nearest' interpolation: `yy = F.interpolate(yy[None, None, :], size=self.new_size, mode='nearest')`.\n* Finally, we obtained a tensor with dimensions 160x80x3, where 3 represents the `(X, Y, Z)` axes. \n\n### 1.2 Transformer Preprocessing\n\n* Only 61 points were kept, including 40 lip points and 21 hand points. For left and right hand, the one with less NaN was kept. If right hand was kept, mirror it to left hand.\n\n* Augmentations, normalization and NaN-filling were applied sequentially.\n\n* Sequences longer than 96 were interpolated to 96. Sequences shorter than 96 were unchanged.\n\n* Apart from raw positions, hand-crafted features were also used, including motion, distances, and cosine of angles.\n\n* Motion features consist of future motion and history motion, which can be denoted as:\n \n$$\n Motion_{future} = position_{t+1} - position_{t}\n$$\n$$\n Motion_{history} = position_{t} - position_{t-1}\n$$\n\n* Full 210 pairwise distances among 21 hand points were included. \n* There are 5 vertices in a finger (e.g. thumb is `[0,1,2,3,4]`), and therefore, there are 3 angles: `<0,1,2>, <1,2,3>, <2,3,4>`. So 15 angles of 5 fingers were included.\n\n* Randomly selected 190 pairwise distances and randomly selected 8 angles among 40 lip points were included.\n\n## 2. Augmentation\n\n### 2.1 Common Augmentations\n\n> These augmentations are used in both CNN training and transformer training\n\n1. `Random affine`: Same as @hengck23 shared. In CNN, after global affine, shift-scale-rotate was also applied to each part separately (e.g. hand, lip, body-pose).\n\n2. `Random interpolation`: Slightly scale and shift the time dimension.\n\n3. `Flip pose`: Flip the x-coordinates of all points. In CNN, `x_new = x_max - x_old`. In transformer, `x_new = 2 * frame[:,0,0] - x_old`.\n\n4. `Finger tree rotate`: There are 4 root-children pairs in a finger with 5-vertices. E.g. in thumb (`[0,1,2,3,4]`), these 4 root-children pairs are: `0-[1,2,3,4]`,`1-[2,3,4]`,`2-[3,4]`,`3-[4]`. We randomly choose some of these pairs, and rotate the children points around root point with a small random angle.\n\n### 2.2 CNN Specific Augmentations \n\n* `Mixup`: Implement basic mixup augmentation (only works with CNNs, not transformers).\n* `Replace augmentation`: Replace some random parts from other samples of the same class.\n* `Time and frequence masking`: This basic torchaudio augmentation works exceptionally well.\n\n```python\nfreq_m = torchaudio.transforms.FrequencyMasking(80) # it's time axis\ntime_m = torchaudio.transforms.TimeMasking(18) # it's points axis\n```\n\n### 2.3 Augmented Sample Example\n\nBefore augmentation:\n\n