![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2212893%2Fa66298421b178fa887022e93194583ab%2FScreenshot%202023-05-01%20at%205.33.08%20AM.png?generation=1682899415191246&alt=media\")
Since this competition is about to end, I decided to document my learnings. Why a failed attempt? Well, I am not topping the leaderboard. Why?
\nNow that I am done sulking about not reaching the top of LB and preparing myself for another competition, it's important to share what didn't work in this competition. Even though these ideas didn't turn out to be great for this competition, I am positive that more can be squeezed out of them. By sharing my failed attempts, I am illuminating a corner of the overall space of potential solutions, IMO.
\nI converted the provided data (in .parquet) to TFRecords to improve the I/O bottleneck of the data pipeline. It was also done to leverage the TPUs.
\nparticipant_id: https://www.kaggle.com/datasets/ayuraj/asl-tfrecords-participantsAll the training were done with the stratified split (20 splits for training and 4 splits for validation).
\nWell, I decided to use TensorFlow/Keras entirely for my workflow. This resulted in a few limitations - no SOTA action recognition model, had to implement the papers from scratch. Why stick with it then? It's my core area of strength + one can leverage the same pipeline to do quantised aware training (I didn't go to this point though).
\nI used Weights and Biases for tracking everything. Here is my W&B dashboard: https://wandb.ai/ayush-thakur/kaggle-asl
\nThere is a lot of unfiltered information on this dashboard but you might still find it useful especially to organise your own experiments.
\nOne of the first initial things I tried was good-old ConvLSTM1D based model. The implementation can be found in this notebook.
The Data pipeline uses the tf.cond method to zero pad the sequence if num of frames were lower than certain threshold. If the num frames was greater than threshold, the sequence was sliced.
I used Pose and Right and Left hands for training. Adding extra body types didn't help as reported in multiple notebooks and discussions.
\nThis was inspired by the Video ViT paper.
Aritra and I implemented this architecture in Keras and the key idea is to use a Tubelet Embedding that basically extract patches from a sequence and embeds them. It's works really well for videos but didn't translate to skeleton data sequence.
\nThis idea was introduced in Make Skeleton-based Action Recognition Model Smaller, Faster and Better paper. The idea is to represent the skeleton as Joint Collection Distances (JCD) and use motion features. You can find the implementation in this notebook.
\nHaving done a few experimentations I noticed that motion features worked well and JCD wasn't working at all.
\nI did a lot of research to use graph based neural networks. I didn't have much experience with graph based nets thus I moved to # 5 but I would still like to documents the papers I read and my what imo should work in this competition. I will not be surprised if any of the top solution is using one of these papers for inspiration.
\nThe paper Revisiting Skeleton-based Action Recognition introduces the idea of converting graph data (skeleton) to heat map and use Conv3D based image-style model (ResNet). This paper though old, is still the SOTA in skeleton based action recognition.
I spent the last two-three weeks implementing this paper. The hardest part was managing the dataset size. You can find my dataset creation step here. The resulting dataset was visualised here: https://wandb.ai/ayush-thakur/kaggle-asl/runs/nkn0h3sw?workspace=user-ayush-thakur
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2212893%2F9d7b9a8bc270fec4e9b0e74d69dbd0f4%2FScreenshot%202023-05-01%20at%205.57.35%20AM.png?generation=1682900877237841&alt=media\")
I created a simple Conv3D based model and was able to reach the same score as that of ConvLSTM1D. I think this method has a lot of promise but requires compute to train it well.
You can find the training code here.
\nIt was a fun competition. Thanks to the organizers for arranging this and congrats to everyone who participated in this competition. The impact of ML to solve problem statement like this is why we all are trying to push this technology ahead. What could I have done better?
\nSince this competition is about to end, I decided to document my learnings. Why a failed attempt? Well, I am not topping the leaderboard. Why?
\nNow that I am done sulking about not reaching the top of LB and preparing myself for another competition, it's important to share what didn't work in this competition. Even though these ideas didn't turn out to be great for this competition, I am positive that more can be squeezed out of them. By sharing my failed attempts, I am illuminating a corner of the overall space of potential solutions, IMO.
\nI converted the provided data (in .parquet) to TFRecords to improve the I/O bottleneck of the data pipeline. It was also done to leverage the TPUs.
\nparticipant_id: https://www.kaggle.com/datasets/ayuraj/asl-tfrecords-participantsAll the training were done with the stratified split (20 splits for training and 4 splits for validation).
\nWell, I decided to use TensorFlow/Keras entirely for my workflow. This resulted in a few limitations - no SOTA action recognition model, had to implement the papers from scratch. Why stick with it then? It's my core area of strength + one can leverage the same pipeline to do quantised aware training (I didn't go to this point though).
\nI used Weights and Biases for tracking everything. Here is my W&B dashboard: https://wandb.ai/ayush-thakur/kaggle-asl
\nThere is a lot of unfiltered information on this dashboard but you might still find it useful especially to organise your own experiments.
\nOne of the first initial things I tried was good-old ConvLSTM1D based model. The implementation can be found in this notebook.
The Data pipeline uses the tf.cond method to zero pad the sequence if num of frames were lower than certain threshold. If the num frames was greater than threshold, the sequence was sliced.
I used Pose and Right and Left hands for training. Adding extra body types didn't help as reported in multiple notebooks and discussions.
\nThis was inspired by the Video ViT paper.
Aritra and I implemented this architecture in Keras and the key idea is to use a Tubelet Embedding that basically extract patches from a sequence and embeds them. It's works really well for videos but didn't translate to skeleton data sequence.
\nThis idea was introduced in Make Skeleton-based Action Recognition Model Smaller, Faster and Better paper. The idea is to represent the skeleton as Joint Collection Distances (JCD) and use motion features. You can find the implementation in this notebook.
\nHaving done a few experimentations I noticed that motion features worked well and JCD wasn't working at all.
\nI did a lot of research to use graph based neural networks. I didn't have much experience with graph based nets thus I moved to # 5 but I would still like to documents the papers I read and my what imo should work in this competition. I will not be surprised if any of the top solution is using one of these papers for inspiration.
\nThe paper Revisiting Skeleton-based Action Recognition introduces the idea of converting graph data (skeleton) to heat map and use Conv3D based image-style model (ResNet). This paper though old, is still the SOTA in skeleton based action recognition.
I spent the last two-three weeks implementing this paper. The hardest part was managing the dataset size. You can find my dataset creation step here. The resulting dataset was visualised here: https://wandb.ai/ayush-thakur/kaggle-asl/runs/nkn0h3sw?workspace=user-ayush-thakur
\n
I created a simple Conv3D based model and was able to reach the same score as that of ConvLSTM1D. I think this method has a lot of promise but requires compute to train it well.
You can find the training code here.
\nIt was a fun competition. Thanks to the organizers for arranging this and congrats to everyone who participated in this competition. The impact of ML to solve problem statement like this is why we all are trying to push this technology ahead. What could I have done better?
\nYour willingness to share your journey and learnings is inspiring, and I'm sure you'll do great in the next competition. Keep pushing forward!
", "rawMarkdown": "Your willingness to share your journey and learnings is inspiring, and I'm sure you'll do great in the next competition. Keep pushing forward!", "votes": null }, { "id": "2243368", "postDate": "05/02/2023 21:23:49", "content": "Thanks @drindramanisinha.
", "rawMarkdown": "Thanks @drindramanisinha.", "votes": null } ], "comments": [ { "id": 2242489, "author_name": "drindramanisinha", "author_url": "", "post_date": "05/02/2023 10:17:47", "content": "Your willingness to share your journey and learnings is inspiring, and I'm sure you'll do great in the next competition. Keep pushing forward!
", "votes": null, "replies": [ { "id": 2243368, "author_name": "ayuraj", "author_url": "", "post_date": "05/02/2023 21:23:49", "content": "Thanks @drindramanisinha.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2240883": "Since this competition is about to end, I decided to document my learnings. Why a failed attempt? Well, I am not topping the leaderboard. Why?\n* Lack of computing\n* Could not focus on the competition consistently\n\nNow that I am done sulking about not reaching the top of LB and preparing myself for another competition, it's important to share what didn't work in this competition. Even though these ideas didn't turn out to be great for this competition, I am positive that more can be squeezed out of them. By sharing my failed attempts, I am illuminating a corner of the overall space of potential solutions, IMO.\n\n# Setup\n\n## Data\n\nI converted the provided data (in .parquet) to TFRecords to improve the I/O bottleneck of the data pipeline. It was also done to leverage the TPUs.\n* Stratified split of the dataset: https://www.kaggle.com/datasets/ayuraj/asl-tfrecords\n* TFRecords by `participant_id`: https://www.kaggle.com/datasets/ayuraj/asl-tfrecords-participants\n\nAll the training were done with the stratified split (20 splits for training and 4 splits for validation).\n\n## Framework\n\nWell, I decided to use TensorFlow/Keras entirely for my workflow. This resulted in a few limitations - no SOTA action recognition model, had to implement the papers from scratch. Why stick with it then? It's my core area of strength + one can leverage the same pipeline to do quantised aware training (I didn't go to this point though).\n\n## Experiment Tracking\n\nI used Weights and Biases for tracking everything. Here is my W&B dashboard: https://wandb.ai/ayush-thakur/kaggle-asl\n\nThere is a lot of unfiltered information on this dashboard but you might still find it useful especially to organise your own experiments.\n\n# 1. ConvLSTM1D\n\nOne of the first initial things I tried was good-old [`ConvLSTM1D`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/ConvLSTM1D) based model. The implementation can be found in this [notebook](https://github.com/ayulockin/kaggle-asl/blob/main/notebooks/03_Train%20Baseline.ipynb).\n\nThe Data pipeline uses the `tf.cond` method to zero pad the sequence if num of frames were lower than certain threshold. If the num frames was greater than threshold, the sequence was sliced.\n\nI used Pose and Right and Left hands for training. Adding extra body types didn't help as reported in multiple notebooks and discussions.\n\n# 2. Tubelet Embedding\n\nThis was inspired by the [Video ViT paper](https://arxiv.org/abs/2103.15691). \n\nAritra and I [implemented this architecture in Keras](https://keras.io/examples/vision/vivit/) and the key idea is to use a Tubelet Embedding that basically extract patches from a sequence and embeds them. It's works really well for videos but didn't translate to skeleton data sequence.\n\n# 3. DDNet (Double-motion Network)\n\nThis idea was introduced in [Make Skeleton-based Action Recognition Model Smaller, Faster and Better](https://arxiv.org/abs/1907.09658) paper. The idea is to represent the skeleton as Joint Collection Distances (JCD) and use motion features. You can find the implementation in this [notebook](https://github.com/ayulockin/kaggle-asl/blob/main/notebooks/11_JCD_and_Motion_Features.ipynb).\n\nHaving done a few experimentations I noticed that motion features worked well and JCD wasn't working at all.\n\n# 4. Graph based networks\n\nI did a lot of research to use graph based neural networks. I didn't have much experience with graph based nets thus I moved to # 5 but I would still like to documents the papers I read and my what imo should work in this competition. I will not be surprised if any of the top solution is using one of these papers for inspiration.\n\n* STGCN: https://arxiv.org/abs/1801.07455\n* InfoGCN: https://openaccess.thecvf.com/content/CVPR2022/papers/Chi_InfoGCN_Representation_Learning_for_Human_Skeleton-Based_Action_Recognition_CVPR_2022_paper.pdf One could have used this network to learn a meaningful representation from extra data (pretrain in a sense) and use the encoder to be fine-tuned on this competition's data.\n* Hyperformer: https://arxiv.org/abs/2211.09590 This was the most promising paper imo. I was able to find an official PyTorch [implementation](https://github.com/ZhouYuxuanYX/Hyperformer) but was too late for me to start porting it to TF.\n\n# 5. PoseConv3D\n\nThe paper [Revisiting Skeleton-based Action Recognition](https://arxiv.org/pdf/2104.13586.pdf) introduces the idea of converting graph data (skeleton) to heat map and use `Conv3D` based image-style model (ResNet). This paper though old, is still the SOTA in skeleton based action recognition.\n\nI spent the last two-three weeks implementing this paper. The hardest part was managing the dataset size. You can find my dataset creation step [here](https://github.com/ayulockin/kaggle-asl/blob/main/scripts/create_heatmap.py). **The resulting dataset was visualised here**: https://wandb.ai/ayush-thakur/kaggle-asl/runs/nkn0h3sw?workspace=user-ayush-thakur\n\n\n\nI created a simple `Conv3D` based model and was able to reach the same score as that of `ConvLSTM1D`. I think this method has a lot of promise but requires compute to train it well.\n\nYou can find the training code [here](https://github.com/ayulockin/kaggle-asl/blob/main/train_poseconv3d.py).\n\n# Conclusion\n\nIt was a fun competition. Thanks to the organizers for arranging this and congrats to everyone who participated in this competition. The impact of ML to solve problem statement like this is why we all are trying to push this technology ahead. What could I have done better?\n\n* Collaborated - I am in search of a team with whom I can participate more seriously.\n* Experimented more with a idea before moving ahead.", "2242489": "Your willingness to share your journey and learnings is inspiring, and I'm sure you'll do great in the next competition. Keep pushing forward!", "2243368": "Thanks @drindramanisinha." }, "source": "meta" }