![\"https://i.ibb.co/9TvfrhW/Selection-999-1855.png\"](\"https://i.ibb.co/9TvfrhW/Selection-999-1855.png\")
experiments in progress …
\n
previous results:
\nhttps://www.kaggle.com/competitions/asl-signs/discussion/391265#2197663
after some experiments for the first week, here is the master plan:
\n[1] one (or two) layer transformer is sufficient. for one fold, memory is only 2mb ( tf.lite.Optimize.DEFAULT, i.e. int8 in storage) and 25min for all 40_000 test videos (or 30 msec per video). Future because of missing frame, long term attention has its advantage
\nin fact two layer transformer would lead to over fitting.
\n[2] next step is to spend more time on data representation, especially normalization
\n[3] use of external data, maybe need to use mediapipe to extract landmarks on external ASL video.
\nOther ways to use increase data increases, adversarial/diffusion augmentation, masked token prediction, etc
I feel that SSL with masked token prediction (e.g. generative pretraining) fits this problem well as there are high proportion of missing frames.
\n[4], There no much of algorithm design because network is quite shallow. if there is, it would be more on increasing parameters for accuracy and speeds. There are many VIT tricks that can be used here, e.g. window attention, poolforming, convolutional positional encoding, grouped conv, …
\nFirst results
\nCV one fold using train/validation random split: cv 0.72~0.74 (probably the limit if good generalisation)
\nCV one fold using train/validation participant non-overlap split: cv 0.61 (compared with above, show the severity of lack of train data)
inference:
\nhttps://www.kaggle.com/code/hengck23/lb-0-65-one-pytorch-transformer-solution
training code:
\nhttps://www.kaggle.com/datasets/hengck23/asl-demo
experiments in progress …
\n
previous results:
\nhttps://www.kaggle.com/competitions/asl-signs/discussion/391265#2197663
after some experiments for the first week, here is the master plan:
\n[1] one (or two) layer transformer is sufficient. for one fold, memory is only 2mb ( tf.lite.Optimize.DEFAULT, i.e. int8 in storage) and 25min for all 40_000 test videos (or 30 msec per video). Future because of missing frame, long term attention has its advantage
\nin fact two layer transformer would lead to over fitting.
\n[2] next step is to spend more time on data representation, especially normalization
\n[3] use of external data, maybe need to use mediapipe to extract landmarks on external ASL video.
\nOther ways to use increase data increases, adversarial/diffusion augmentation, masked token prediction, etc
I feel that SSL with masked token prediction (e.g. generative pretraining) fits this problem well as there are high proportion of missing frames.
\n[4], There no much of algorithm design because network is quite shallow. if there is, it would be more on increasing parameters for accuracy and speeds. There are many VIT tricks that can be used here, e.g. window attention, poolforming, convolutional positional encoding, grouped conv, …
\nFirst results
\nCV one fold using train/validation random split: cv 0.72~0.74 (probably the limit if good generalisation)
\nCV one fold using train/validation participant non-overlap split: cv 0.61 (compared with above, show the severity of lack of train data)
inference:
\nhttps://www.kaggle.com/code/hengck23/lb-0-65-one-pytorch-transformer-solution
training code:
\nhttps://www.kaggle.com/datasets/hengck23/asl-demo
i have been playing with controlnet and i come across this:
\n![\"https://i.ibb.co/0cdwBSL/Selection-999-1366.png\"](\"https://i.ibb.co/0cdwBSL/Selection-999-1366.png\")
maybe can make some fun animation video while waiting for training
", "rawMarkdown": "i have been playing with controlnet and i come across this:\n\n\nmaybe can make some fun animation video while waiting for training", "votes": 14, "replies": [ { "id": 2180654, "postDate": "2023-03-14T02:26:27.493Z", "content": "landmark --> controlnet -->stable diffusion --> image+style+augment --> media pose -->landmark+augment ???
\nnote that you can have depthmap as control etc and you can change viewpoint, etc
", "rawMarkdown": "landmark --> controlnet -->stable diffusion --> image+style+augment --> media pose -->landmark+augment ???\n\nnote that you can have depthmap as control etc and you can change viewpoint, etc", "votes": 2 }, { "id": 2202379, "postDate": "2023-03-30T02:22:09.557Z", "content": "controlnet based on face landmark
\nhttps://huggingface.co/georgefen/Face-Landmark-ControlNet
Hi, that is really interesting. I am guessing this will create images frame by frame and we merge all the frames to create a video. Can you give me gist of it? I'm still learning, but seems like a fun learning project. Thank you!
", "rawMarkdown": "Hi, that is really interesting. I am guessing this will create images frame by frame and we merge all the frames to create a video. Can you give me gist of it? I'm still learning, but seems like a fun learning project. Thank you!" } ] }, { "id": 2201909, "postDate": "2023-03-29T16:19:45.363Z", "content": "flip augmentation magic:
\nCV 0.703 (single fold)
\nLB 0.72
LHAND = np.arange(468, 489).tolist() # 21\nRHAND = np.arange(522, 543).tolist() # 21\nPOSE = np.arange(489, 522).tolist() # 33\nFACE = np.arange(0,468).tolist() #468\n\nREYE = [\n 33, 7, 163, 144, 145, 153, 154, 155, 133,\n 246, 161, 160, 159, 158, 157, 173,\n]\nLEYE = [\n 263, 249, 390, 373, 374, 380, 381, 382, 362,\n 466, 388, 387, 386, 385, 384, 398,\n]\nNOSE=[\n 1,2,98,327\n]\nSLIP = [\n 78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n 191, 80, 81, 82, 13, 312, 311, 310, 415,\n]\nSPOSE = (np.array([\n 11,13,15,12,14,16,23,24,\n])+489).tolist()\n\n\n...\n## assume zero mean xyz. so flip can be implement by multuiplication of -1\ndef do_hflip_hand(lhand, rhand):\n rhand[...,0] *= -1\n lhand[...,0] *= -1\n rhand, lhand = lhand,rhand\n return lhand, rhand\n\ndef do_hflip_spose(spose):\n spose[...,0] *= -1\n spose = spose[:,[3,4,5,0,1,2,7,6]]\n return spose\n\ndef do_hflip_slip(slip):\n slip[...,0] *= -1\n slip = slip[:,[10,9,8,7,6,5,4,3,2,1,0]+[19,18,17,16,15,14,13,12,11]]\n return slip\n\n...\n...\n xyz = load_relevant_data_subset(pq_file)\n lhand = xyz[:,LHAND]\n rhand = xyz[:,RHAND]\n spose = xyz[:,SPOSE]\n leye = xyz[:,LEYE]\n reye = xyz[:,REYE]\n slip = xyz[:,SLIP]\n nose = xyz[:,NOSE]\n\n\n if is_aug==1:\n if np.random.rand()<0.5:\n lhand, rhand = do_hflip_hand(lhand, rhand)\n spose = do_hflip_spose(spose)\n leye, reye = do_hflip_eye(leye, reye)\n slip = do_hflip_slip(slip)\n nose = do_hflip_nose(nose)\ntrain log:
\nfold_type = kaggle-part\nfold = 0\ntrain_dataset : \n len = 71518\n num_participant_id = 16\n\nvalid_dataset : \n len = 22959\n num_participant_id = 5\n...\n batch_size = 64 \n experiment = ['tx025-no-z-flip', 'run_train_fold0.py']\n |---------------- VALID---------|---- TRAIN/BATCH ----------------------\nrate iter epoch | loss top1 top2 top5 | loss | time \n---------------------------------------------------------------------------------------------------\n...\n1.00e-4 00050265* 45.00 | 2.025 0.698 0.7992 0.870 | 3.887 0.000 0.000 | 1 hr 48 min\n1.00e-4 00053616* 48.00 | 2.042 0.697 0.7969 0.870 | 3.879 0.000 0.000 | 1 hr 56 min\none may want to try \"cut and paste augment\": assemble body parts from different signers
", "rawMarkdown": "one may want to try \"cut and paste augment\": assemble body parts from different signers", "votes": 2, "replies": [ { "id": 2202195, "postDate": "2023-03-29T20:06:22.660Z", "content": "I've already tried something similar when I changed body parts. On my pipeline, the result is controversial. On the one hand, this gave an increase of 1 fold, but on the other hand, the ensemble did not improve. Of course I could have made a mistake somewhere in the code. And this reduced val_loss
", "rawMarkdown": "I've already tried something similar when I changed body parts. On my pipeline, the result is controversial. On the one hand, this gave an increase of 1 fold, but on the other hand, the ensemble did not improve. Of course I could have made a mistake somewhere in the code. And this reduced val_loss", "votes": 1 }, { "id": 2202204, "postDate": "2023-03-29T20:12:24.370Z", "content": "the parts have to be synchronised correctly in time. a better way is to use gan or pddm
", "rawMarkdown": "the parts have to be synchronised correctly in time. a better way is to use gan or pddm", "votes": 1 }, { "id": 2202381, "postDate": "2023-03-30T02:30:20.223Z", "content": "instead of cut and paste, imagine:
\nbasically, we are learning a better normalisation
\nhttps://www.youtube.com/watch?v=U9G2VlG9HbA
🤔Do other landmarks really help? In my experiments, event simply add pose landmarks (arms position) decrease CV.
", "rawMarkdown": "🤔Do other landmarks really help? In my experiments, event simply add pose landmarks (arms position) decrease CV.", "votes": 1, "replies": [ { "id": 2202395, "postDate": "2023-03-30T03:08:03.123Z", "content": "it depends on how you normalised i think.
\nthere is many noise in the data becuase each signer is of different size and their camera may have different frame rate. They also move differently.
\nin theory:
\neye: should help. for sign words \"awake\", the eye is closed then opened.
\npose (arm): should help. this is global motion of hand.
but if your nomralisation is poor or your model cannot filter out the noise, then there will be easy overfitting to noise
", "rawMarkdown": "it depends on how you normalised i think.\n\nthere is many noise in the data becuase each signer is of different size and their camera may have different frame rate. They also move differently.\n\nin theory:\neye: should help. for sign words \"awake\", the eye is closed then opened.\npose (arm): should help. this is global motion of hand.\n\nbut if your nomralisation is poor or your model cannot filter out the noise, then there will be easy overfitting to noise\n", "votes": 4 }, { "id": 2202398, "postDate": "2023-03-30T03:12:11.127Z", "content": "yes, for me, normalization method also matters
", "rawMarkdown": "yes, for me, normalization method also matters" } ] }, { "id": 2202578, "postDate": "2023-03-30T06:40:03.967Z", "content": "this is not too difficult if you think about it
\n![\"https://i.ibb.co/5FLVDDw/Selection-999-1630.png\"](\"https://i.ibb.co/5FLVDDw/Selection-999-1630.png\")
I tried a very similar approach to this, but unfortunately, I didn't see any performance improvement compared to simple augmentation(maybe my approach was too naive). However, I think it's a fascinating idea, and I'm excited to hear about your progress if you decide to give it a go:).
", "rawMarkdown": "I tried a very similar approach to this, but unfortunately, I didn't see any performance improvement compared to simple augmentation(maybe my approach was too naive). However, I think it's a fascinating idea, and I'm excited to hear about your progress if you decide to give it a go:)." } ] }, { "id": 2229296, "postDate": "2023-04-21T08:27:47.767Z", "content": "According to LANDMARKS_REFINEMENT_LEFT_EYE_CONFIG and LANDMARKS_REFINEMENT_RIGHT_EYE_CONFIG (from https://github.com/tensorflow/tfjs-models/blob/master/face-landmarks-detection/src/tfjs/constants.ts) , your REYE if a left eye, and LEYE is a right eye. Are you sure that's correct?
", "rawMarkdown": "According to LANDMARKS_REFINEMENT_LEFT_EYE_CONFIG and LANDMARKS_REFINEMENT_RIGHT_EYE_CONFIG (from https://github.com/tensorflow/tfjs-models/blob/master/face-landmarks-detection/src/tfjs/constants.ts) , your REYE if a left eye, and LEYE is a right eye. Are you sure that's correct?" } ] }, { "id": 2193356, "postDate": "2023-03-23T08:28:19.040Z", "content": "I love your experiments records table! thanks for sharing this!
", "rawMarkdown": "I love your experiments records table! thanks for sharing this!", "votes": 8, "replies": [ { "id": 2193995, "postDate": "2023-03-23T16:50:13.230Z", "content": "+1 @hengck23
", "rawMarkdown": "+1 @hengck23 " } ] }, { "id": 2178946, "postDate": "2023-03-12T19:46:44.203Z", "content": "[unorganized] some proven intermediate results … will write in more detail later
\nadd motion (xyz at current frame - xyz at previous frame). i.e. each point now has 6 dim. need to handle NaN motion correctly
\nimprove CV by 0.015
add framerate interpolation augmentation scipy.interpolate.interp1d()
\nimprove CV by 0.015
ideally, if there is lots and lots of training samples, you don't have to do anything, just add more parameters and the model would learn scale, rotation invariant features.
\nif there is not much training samples, you can do augmentation, e.g. rotate/shift the hands
\nyou can also handcraft features, e.g. pairwise distance, pairwise angles of the hand points (these are invariant to rotation)
\nbetter still, graph network CGN or pointnet features are for this.
", "rawMarkdown": "ideally, if there is lots and lots of training samples, you don't have to do anything, just add more parameters and the model would learn scale, rotation invariant features.\n\nif there is not much training samples, you can do augmentation, e.g. rotate/shift the hands\n\nyou can also handcraft features, e.g. pairwise distance, pairwise angles of the hand points (these are invariant to rotation)\n\nbetter still, graph network CGN or pointnet features are for this.\n ", "votes": 3, "replies": [ { "id": 2180165, "postDate": "2023-03-13T16:21:10.243Z", "content": "convergence is magical if you do things right !
\ntrain/validation participant id does not overlap shown below:
\n![\"https://i.ibb.co/g3C1Gq6/Selection-999-1379.png\"](\"https://i.ibb.co/g3C1Gq6/Selection-999-1379.png\")
distance matrix is the poor man's graph network D x A x inv(D) …
\ni need to google for better joint kinematics features ….
i wondered if i renderd the point into 48x48 images and use 3dCNN, what will be the speed and accuracy …
", "rawMarkdown": "i wondered if i renderd the point into 48x48 images and use 3dCNN, what will be the speed and accuracy ...", "votes": 3 }, { "id": 2181090, "postDate": "2023-03-14T10:11:12.587Z", "content": "possible of aux loss:
\n….
", "rawMarkdown": "possible of aux loss:\n- predict if word is one-hand or two-hand word\n- predict if the signer is using left, right or both hands\n- predict signer identity\n\n....", "votes": 1 }, { "id": 2182554, "postDate": "2023-03-15T06:51:09.113Z", "content": "I didn’t try 3D CNN yet, but I have tried 2D CNN. I use a (batch_size, num_frame, num_landmark, 3) tensor as input. I didn’t do much experiments, the best val acc I got is around 0.53. I didn’t record the speed. There are many skeleton-based motion recognition papers that use CNN, but according to the paper, they all achieve a fantastic result, which is much higher than 0.53… I guess we need to process the data in some ways to make it suitable to CNN.
", "rawMarkdown": "I didn’t try 3D CNN yet, but I have tried 2D CNN. I use a (batch_size, num_frame, num_landmark, 3) tensor as input. I didn’t do much experiments, the best val acc I got is around 0.53. I didn’t record the speed. There are many skeleton-based motion recognition papers that use CNN, but according to the paper, they all achieve a fantastic result, which is much higher than 0.53… I guess we need to process the data in some ways to make it suitable to CNN." }, { "id": 2182562, "postDate": "2023-03-15T06:59:03.907Z", "content": "![\"Selection-999-1426\"](\"https://i.ibb.co/1q45K1y/Selection-999-1426.png\")
\n![\"Selection-999-1422\"](\"https://i.ibb.co/SVNsqhK/Selection-999-1422.png\")
\n",
"votes": 1
},
{
"id": 2182671,
"postDate": "2023-03-15T08:41:20.820Z",
"content": "@jay2333 Papers in the skeleton-based action recognition domain rely on high-quality pose data. In most cases, they use HRNet pose estimator to extract the skeleton landmarks. Furthermore, there are only a few missing values.
\nThe landmarks in our dataset are accurate, but just not as good as landmarks from HRNet. Plus, we have a lot of missing values and we need to find a way to deal with them.
", "rawMarkdown": "@jay2333 Papers in the skeleton-based action recognition domain rely on high-quality pose data. In most cases, they use HRNet pose estimator to extract the skeleton landmarks. Furthermore, there are only a few missing values. \n\nThe landmarks in our dataset are accurate, but just not as good as landmarks from HRNet. Plus, we have a lot of missing values and we need to find a way to deal with them. " }, { "id": 2183125, "postDate": "2023-03-15T13:29:50.920Z", "content": "@gregorlied Totally agree with that. According to the current public notebook, a simple fully connected network can perform well. So the main issue is the data process part.
", "rawMarkdown": "@gregorlied Totally agree with that. According to the current public notebook, a simple fully connected network can perform well. So the main issue is the data process part.", "votes": 1 }, { "id": 2183239, "postDate": "2023-03-15T14:42:51.547Z", "content": "\" fully connected network can perform well. \"
\nthe reason is that the seq are not really long
", "rawMarkdown": "\" fully connected network can perform well. \"\n\nthe reason is that the seq are not really long", "votes": 1 }, { "id": 2183267, "postDate": "2023-03-15T15:03:07.460Z", "content": "there is a trick to speedup 3d CNN or 2d CNN
\nbasically it is like CLIP (aligned text and image)
\nhere you trained joint encoder for (pose, rendered image) pair.
\nrendered image has more information so they have better embedding.
you can think of we are distilling image embedding to point embedding.
\nsince both embedding are \"aligned\", in testing we are only using point embedding (though we trained them jointly)
try to use different color for part, shade for depth when rendering ….
", "rawMarkdown": "there is a trick to speedup 3d CNN or 2d CNN\n\nbasically it is like CLIP (aligned text and image)\n\nhere you trained joint encoder for (pose, rendered image) pair.\nrendered image has more information so they have better embedding.\n\nyou can think of we are distilling image embedding to point embedding.\nsince both embedding are \"aligned\", in testing we are only using point embedding (though we trained them jointly)\n\n---\ntry to use different color for part, shade for depth when rendering ....\n", "votes": 1 }, { "id": 2185509, "postDate": "2023-03-17T05:45:15.100Z", "content": "yet another aux loss,
\nMay I ask you how you actually apply interpolation? I'm trying to apply it, but even with soft parameters the score is lower
", "rawMarkdown": "May I ask you how you actually apply interpolation? I'm trying to apply it, but even with soft parameters the score is lower" } ] }, { "id": 2228313, "postDate": "2023-04-20T12:56:37.840Z", "content": "watch this thread!
\nreceipe for LB0.76 one fold:
\ni seems that there is somthing very strange about the dataset … maybe very noisy. i have to use unconventional strong regularisation and training method ….
\nhint:
\n[1] Dropout Reduces Underfitting
\nhttps://arxiv.org/abs/2303.01500
i find that if i reduce learning rate, the model easily overfits. instead, according to paper[1], by setting dropout to different values at different training epoches, we can reduce validation loss (it has the same effects as reducing learning rate) without overfitting:
\n![\"https://i.ibb.co/mz15RwQ/Selection-999-1847.png\"](\"https://i.ibb.co/mz15RwQ/Selection-999-1847.png\")
![\"https://i.ibb.co/GvkZB5j/Selection-999-1848.png\"](\"https://i.ibb.co/GvkZB5j/Selection-999-1848.png\")
fold_type = kaggle-random (20% validation, 80% train. pure random)\nfold = 0\nvalid_dataset : \n len = 18896\n num_participant_id = 21\n num_external_video = 0\n\n\ntime_taken = 0 min 27 sec\nloss = 1.7932500839233398\ntopk[0] = 0.8607112616426758 (LB 0.76)\ntopk[1] = 0.9180249788314987\ntopk[2] = 0.9374470787468248\ntopk[3] = 0.9457557154953429\ntopk[4] = 0.9513653683319221\ncheckpoint_file = /home/titanx/hengck/share1/kaggle/2022/hand-sign/result/run202/tx90-face-norm-480-0b/fold-0-kaggle-random/checkpoint/swa.model.pth\nHave you tried using ViT or are you still using Conv1d + MHA?
", "rawMarkdown": "Have you tried using ViT or are you still using Conv1d + MHA?", "replies": [ { "id": 2229582, "postDate": "2023-04-21T13:40:39.923Z", "content": "dropout has the effect of flatten the valley of loss landscape.
\n(this is the same as SWA, deeply supervised loss, etc)
Of course, adeverisal training SAM (sharpness aware optimizer, aka sample perturbation via weight perturbation) also flattens the valley.
\nBelow is results of SAM, which is comparable to late dropout
fold_type = kaggle-part\nfold = 0\nvalid_dataset : \n len = 22959\n num_participant_id = 5\n num_external_video = 0\n\n\n 22959 / 22959 0 min 30 sec\n\ntime_taken = 0 min 30 sec\nloss = 2.5410375595092773\ntopk[0] = 0.7265560346704996\ntopk[1] = 0.8181105448843591\ntopk[2] = 0.856265516790801\ntopk[3] = 0.8760398972080665\ntopk[4] = 0.8882791062328499\ncheckpoint_file = /home/user/share1/kaggle/2022/hand-sign/result/run202/tx90-face-norm-sam-fine-rho0.1/fold-0-kaggle-part/checkpoint/swa.model.pth\n----- end -----\nIt may be worthwhile to try additional parameters in conjunction with a 'late' configuration
", "rawMarkdown": "It may be worthwhile to try additional parameters in conjunction with a 'late' configuration" } ] } ] }, { "id": 2163630, "postDate": "2023-03-01T00:49:08.950Z", "content": "[reference]
\nhttps://github.com/matyasbohacek/spoter
\nhttps://www.matyasbohacek.com/
\n(uses MediaPipe library pose too)
https://huggingface.co/spaces/matyasbohacek/spoter-demo-test
\nhttps://www.youtube.com/watch?v=6i5ODpHMny8
\n![\"https://i.ibb.co/r2NHnkh/Selection-999-1166.png\"](\"https://i.ibb.co/r2NHnkh/Selection-999-1166.png\")
SLGTFORMER: AN ATTENTION-BASED APPROACH TO SIGN LANGUAGE RECOGNITION
\nhttps://github.com/neilsong/SLGTformer
[external data]
\n… to be updated …
HOW TO SIGN
\nhttps://how2sign.github.io/
\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Duarte_How2Sign_A_Large-Scale_Multimodal_Dataset_for_Continuous_American_Sign_Language_CVPR_2021_paper.pdf
\n(vocab=16k, time=79 hr, signer=11)
MS-ASL American Sign Language Dataset
\nhttps://microsoft.github.io/data-for-society/dataset?d=MS-ASL-American-Sign-Language-Dataset
\nhttps://arxiv.org/pdf/1812.01053.pdf
\n(vocab=1000, num video=25513, signer=222)
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
\nhttps://www.robots.ox.ac.uk/~vgg/research/bsl1k/
\n(vocab=1000, num video=1000)
ChaLearn LAP Large Scale Signer Independent Isolated Sign Language
\nRecognition Challenge: Design, Results and Future Research
ASL-Skeleton3D and ASL-Phono: Two Novel
\nDatasets for the American Sign Language
how to read media pipline to extra keypoint for external data
\n… to be udpated …
more about PopSign learning app
\nhttps://devpost.com/software/pop-sign-learning
\nhttps://www.youtube.com/watch?v=nbQAfiJzp2g&t=430s
\nhttps://github.com/Accessible-Technology-in-Sign/PopSign
https://github.com/Accessible-Technology-in-Sign/ASLRT
\n\"A pipeline to run tracking Kinect, MediaPipe and AlphaPose experiments on sign language feature data using either HMMs, SBHMMs, or Transformers.\"
collect data you from youtube?
", "rawMarkdown": "[external data]\n... to be updated ...\n\nHOW TO SIGN\nhttps://how2sign.github.io/\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Duarte_How2Sign_A_Large-Scale_Multimodal_Dataset_for_Continuous_American_Sign_Language_CVPR_2021_paper.pdf\n(vocab=16k, time=79 hr, signer=11)\n\nMS-ASL American Sign Language Dataset\nhttps://microsoft.github.io/data-for-society/dataset?d=MS-ASL-American-Sign-Language-Dataset\nhttps://arxiv.org/pdf/1812.01053.pdf\n(vocab=1000, num video=25513, signer=222)\n\nBSL-1K: Scaling up co-articulated sign language recognition using mouthing cues\nhttps://www.robots.ox.ac.uk/~vgg/research/bsl1k/\n(vocab=1000, num video=1000)\n\n\nChaLearn LAP Large Scale Signer Independent Isolated Sign Language\nRecognition Challenge: Design, Results and Future Research\n\nASL-Skeleton3D and ASL-Phono: Two Novel\nDatasets for the American Sign Language\n\n---\n\n- how to read media pipline to extra keypoint for external data \n... to be udpated ...\n\n- more about PopSign learning app\nhttps://devpost.com/software/pop-sign-learning\nhttps://www.youtube.com/watch?v=nbQAfiJzp2g&t=430s\nhttps://github.com/Accessible-Technology-in-Sign/PopSign\n\nhttps://github.com/Accessible-Technology-in-Sign/ASLRT\n\"A pipeline to run tracking Kinect, MediaPipe and AlphaPose experiments on sign language feature data using either HMMs, SBHMMs, or Transformers.\"\n\ncollect data you from youtube?\n\n\n\n\n\n\n\n", "votes": 6, "replies": [ { "id": 2176272, "postDate": "2023-03-10T14:03:14.067Z", "content": "pytorch mediapipe … hmm
\nhttps://github.com/zmurez/MediaPipePyTorch
this image explain why hand are missing.
\nyou probably can ignore the legs
![\"https://i.ibb.co/237HvmF/original.png\"](\"https://i.ibb.co/237HvmF/original.png\")
there is a related product called copycat:
\nCopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills
\nhttps://gvu.gatech.edu/research/projects/copycat-using-sign-language-recognition-help-deaf-children-acquire-language-skills#:~:text=CopyCat%20is%20a%20game%20where,language%20skills%20and%20working%20memory.
https://www.youtube.com/watch?v=WMFZVcey8FU
\nhttps://github.com/Accessible-Technology-in-Sign/ASLRT
much information on HMM + handcraft feature here
", "rawMarkdown": "there is a related product called copycat:\nCopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills\nhttps://gvu.gatech.edu/research/projects/copycat-using-sign-language-recognition-help-deaf-children-acquire-language-skills#:~:text=CopyCat%20is%20a%20game%20where,language%20skills%20and%20working%20memory.\n\nhttps://www.youtube.com/watch?v=WMFZVcey8FU\nhttps://github.com/Accessible-Technology-in-Sign/ASLRT\n\nmuch information on HMM + handcraft feature here" } ] }, { "id": 2226810, "postDate": "2023-04-19T08:43:02.580Z", "content": "i forget an import detail at the data card. suggestions for augmentation:
\n\"While the app provided a video example of the sign desired, signers routinely made variants of the sign based on their background and region.\"
\n\"More rarely, signers might fingerspell a sign, miss it completely, or produce the wrong sign.\"
\n\" Extraneous movements, such as scratching an itch, or the ending movement from the previous sign or the onset of the next sign, are sometimes included.\"
\n\" Conversely, some signers pressed the button late or released the button early, causing cropping in some sign examples. \"
\n\"Some signers sign with their left hand; others sign with their right. Some signers switch their signing hand. All of these situations must be handled by the game’s recognition system.\"
\na also note that some signer make repeated signs in one video
", "rawMarkdown": "i forget an import detail at the data card. suggestions for augmentation:\n\n\"While the app provided a video example of the sign desired, signers routinely made variants of the sign based on their background and region.\" \n- you need external data or create your variant\n\n\"More rarely, signers might fingerspell a sign, miss it completely, or produce the wrong sign.\" \n- need to measure label noise and decide if you want to exclude some data\n\n\" Extraneous movements, such as scratching an itch, or the ending movement from the previous sign or the onset of the next sign, are sometimes included.\" \n- random add hand (from other video at start and end). actually CAM heat map is about to hightlight the useful frame.\n\n\" Conversely, some signers pressed the button late or released the button early, causing cropping in some sign examples. \" \n- cut and paste (mix up)\n\n\"Some signers sign with their left hand; others sign with their right. Some signers switch their signing hand. All of these situations must be handled by the game’s recognition system.\" \n- flip\n\n\n---\n\na also note that some signer make repeated signs in one video\n\n", "votes": 3, "replies": [ { "id": 2229212, "postDate": "2023-04-21T06:49:13.603Z", "content": "\"need to measure label noise and decide if you want to exclude some data\"
\nSo we need to specify some kind of \"ideal\" signs, and compare train data with it?
", "rawMarkdown": "\"need to measure label noise and decide if you want to exclude some data\"\n\nSo we need to specify some kind of \"ideal\" signs, and compare train data with it?", "replies": [ { "id": 2229217, "postDate": "2023-04-21T06:51:29.157Z", "content": "those that have validation/train error higher than normal are supsicous outliers
", "rawMarkdown": "those that have validation/train error higher than normal are supsicous outliers ", "votes": 1 }, { "id": 2229256, "postDate": "2023-04-21T07:37:43.810Z", "content": "If the hidden dataset is very cleanly drawn, I think data with a low frame count can be left out.
", "rawMarkdown": "If the hidden dataset is very cleanly drawn, I think data with a low frame count can be left out." }, { "id": 2229468, "postDate": "2023-04-21T11:49:10.123Z", "content": "there must be some reason why some video are very long (much longer than average).
\nat the signer moving very slowly? or are they not cropping the video (press start and end) correctly? or are they signing wrongly?
You're right, long frame data has some problems.
", "rawMarkdown": "You're right, long frame data has some problems. " }, { "id": 2229604, "postDate": "2023-04-21T14:07:24.983Z", "content": "this is explain why conv1d does not work but conv1d+max pool works (about LB 0.72, compared to transformer of 0.74)
\n self.conv2 = nn.Sequential(\n nn.Conv1d(1024, 768, kernel_size=3, padding=1, stride=1),\n LayerNorm1D(768),\n nn.ReLU(inplace=True),\n nn.Dropout(p=0.1),\n nn.MaxPool1d(kernel_size=3, padding=1, stride=2),\n )\nmax pool is poor man's attention pool(softmax versus hard max)
\nalso, why adding noise (dropout, dropframe, random values, mix frame …) before mha works. it forces the model to select the correct signal.
\ni am very surprise of the small num of paramaters and performance of transformer compared to other model design (like conv1d, dense, etc)
", "rawMarkdown": "this is explain why conv1d does not work but conv1d+max pool works (about LB 0.72, compared to transformer of 0.74)\n\n``` \n\t\tself.conv2 = nn.Sequential(\n\t\t\tnn.Conv1d(1024, 768, kernel_size=3, padding=1, stride=1),\n\t\t\tLayerNorm1D(768),\n\t\t\tnn.ReLU(inplace=True),\n\t\t\tnn.Dropout(p=0.1),\n\t\t\tnn.MaxPool1d(kernel_size=3, padding=1, stride=2),\n\t\t)\n\n```\n\nmax pool is poor man's attention pool(softmax versus hard max)\n\nalso, why adding noise (dropout, dropframe, random values, mix frame ...) before mha works. it forces the model to select the correct signal.\n\ni am very surprise of the small num of paramaters and performance of transformer compared to other model design (like conv1d, dense, etc)" } ] } ] }, { "id": 2224078, "postDate": "2023-04-17T01:26:37.653Z", "content": "it is suprised for me to achieve 0.75 in one fold with participant split train data(num participant = 5 in validation data). This makes me think that 0.76 is possible.
\nfold_type = kaggle-part\nfold = 0\nvalid_dataset : \n len = 22959\n num_participant_id = 5 \n\n 22959 / 22959 0 min 42 sec\n\ntime_taken = 0 min 42 sec\nloss = 2.406057596206665\ntopk[0] = 0.7277320440785748\ntopk[1] = 0.8219870203406072\ntopk[2] = 0.858530423798946\ntopk[3] = 0.8757785617840498\ntopk[4] = 0.8888888888888888\ncheckpoint_file = /home/titanx/hengck/share1/kaggle/2022/hand-sign/result/run201/tx61a-1-frame-first-scale-2/fold-0-kaggle-part/checkpoint/swa.model.pth\nand i hit the jackpot 888888888
", "rawMarkdown": "it is suprised for me to achieve 0.75 in one fold with participant split train data(num participant = 5 in validation data). This makes me think that 0.76 is possible. \n\n```\n\nfold_type = kaggle-part\nfold = 0\nvalid_dataset : \n\tlen = 22959\n\tnum_participant_id = 5 \n \n 22959 / 22959 0 min 42 sec\n\ntime_taken = 0 min 42 sec\nloss = 2.406057596206665\ntopk[0] = 0.7277320440785748\ntopk[1] = 0.8219870203406072\ntopk[2] = 0.858530423798946\ntopk[3] = 0.8757785617840498\ntopk[4] = 0.8888888888888888\ncheckpoint_file = /home/titanx/hengck/share1/kaggle/2022/hand-sign/result/run201/tx61a-1-frame-first-scale-2/fold-0-kaggle-part/checkpoint/swa.model.pth\n \n\n```\n\nand i hit the jackpot 888888888", "votes": 3, "replies": [ { "id": 2224095, "postDate": "2023-04-17T02:08:50.303Z", "content": "888888888!
", "rawMarkdown": "888888888!", "votes": 1 }, { "id": 2225136, "postDate": "2023-04-17T23:29:35.293Z", "content": "Are these accuracies shown for the 5 holdout participants?
", "rawMarkdown": "Are these accuracies shown for the 5 holdout participants?" } ] }, { "id": 2206909, "postDate": "2023-04-03T02:01:33.027Z", "content": "a couple on conv1d (as spatial temporal encoder) + MHA attention seems to have good results.
\nsome good ideas to design efficient transformer.
[1] FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization
\nhttps://arxiv.org/pdf/2303.14189.pdf
[2] Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios
\nhttps://arxiv.org/abs/2207.05501
to avoid uncessary permute reshape, i can write my q,v,k linear operation in terms of conv1d.
\ni will be removing absolute position embedding and use convolutional position embedding instead (convolutional FFN, i.e. position information is implicitly embedding on 1d conv)
actually it is aslo possible to use 2d convolution, where (W,H) = (x,y,z of spatial, t of temporal).
", "rawMarkdown": "a couple on conv1d (as spatial temporal encoder) + MHA attention seems to have good results.\nsome good ideas to design efficient transformer.\n\n[1] FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization\nhttps://arxiv.org/pdf/2303.14189.pdf\n\n[2] Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios\nhttps://arxiv.org/abs/2207.05501\n\nto avoid uncessary permute reshape, i can write my q,v,k linear operation in terms of conv1d.\ni will be removing absolute position embedding and use convolutional position embedding instead (convolutional FFN, i.e. position information is implicitly embedding on 1d conv)\n\n\nactually it is aslo possible to use 2d convolution, where (W,H) = (x,y,z of spatial, t of temporal).", "votes": 3, "replies": [ { "id": 2206947, "postDate": "2023-04-03T02:55:46.753Z", "content": "Won't conv bring much more parameters?
\nI suppose what you mean is :
\n\ninputs_embeds = inputs_embeds + position_embeds \n\n\ninputs_embeds = inputs_embeds.permute(, , ) \ninputs_embeds = conv1d(inputs_embeds) \ninputs_embeds = inputs_embeds.permute(, , ) \nwhere the number of parameters in original is TxH, and the conv has HxHx3
\nI also tried this idea, but found that the conv is much harder to train than MHSA, maybe because of more parameters.
", "rawMarkdown": "Won't conv bring much more parameters?\n\nI suppose what you mean is :\n\n```python\n# original Transformer\ninputs_embeds = inputs_embeds + position_embeds # BxTxH <- BxTxH + BxTxH \n\n# conv as position\ninputs_embeds = inputs_embeds.permute(0, 2, 1) # BxHxT <- BxTxH\ninputs_embeds = conv1d(inputs_embeds) # BxHxT <- BxHxT @ HxHx3 (e.g. conv1d kernel=3)\ninputs_embeds = inputs_embeds.permute(0, 2, 1) # BxTxH <- BxHxT\n```\n\nwhere the number of parameters in original is TxH, and the conv has HxHx3\n\nI also tried this idea, but found that the conv is much harder to train than MHSA, maybe because of more parameters.", "votes": 1, "replies": [ { "id": 2207339, "postDate": "2023-04-03T10:49:08.177Z", "content": "\n1d convolution multi-head-attention:
\nclass MyMultiHeadAttention(nn.Module):\n def __init__(self,\n embed_dim,\n out_dim,\n qk_dim,\n v_dim,\n num_head,\n ):\n super().__init__()\n self.embed_dim = embed_dim\n self.num_head = num_head\n self.qk_dim = qk_dim\n self.v_dim = v_dim\n\n self.q = nn.Conv1d(embed_dim, qk_dim*num_head,1)\n self.k = nn.Conv1d(embed_dim, qk_dim*num_head,1) # stride=2 for token reduction, kernel>1 for mixing\n self.v = nn.Conv1d(embed_dim, v_dim*num_head,1)\n\n self.out = nn.Conv1d(v_dim*num_head, out_dim, 1)\n self.scale = 1/(qk_dim**0.5)\n\n #https://github.com/pytorch/pytorch/issues/40497\n def forward(self, x, x_mask):\n B,dim,L= x.shape\n\n num_head = self.num_head\n qk_dim = self.qk_dim\n v_dim = self.v_dim\n\n q = self.q(x) #B,qk_dim,L\n k = self.k(x)\n v = self.v(x)\n q = q.reshape(B, num_head, qk_dim, L).permute(0,1,3,2).contiguous()\n k = k.reshape(B, num_head, qk_dim, L)#.permute(0,1,2,3).contiguous()\n v = v.reshape(B, num_head, v_dim, L).permute(0,1,3,2).contiguous()\n\n dot = torch.matmul(q, k) *self.scale # H L L\n x_mask = x_mask.reshape(B,1,1,L).expand(-1,num_head,L,-1)\n #dot[x_mask]= -1e4\n dot.masked_fill_(x_mask, -1e4)\n attn = F.softmax(dot, -1) # L L\n\n v = torch.matmul(attn, v) # L H dim\n v = v.permute(0,1,3,2).reshape(B, v_dim*num_head,L).contiguous()\n out = self.out(v)\n return out\n\n#---\nembed_dim = 512\nout_dim = 512\nqk_dim = 512//4 #for one head\nv_dim = 512//4\nnum_head = 4\nmax_length = 96\nbatch_size = 4\n\nmha = MyMultiHeadAttention(\n embed_dim,\n out_dim,\n qk_dim,\n v_dim,\n num_head,\n)\nx = torch.from_numpy( np.random.uniform(-1,1,(batch_size, embed_dim, max_length))).float()\nx_mask = torch.from_numpy( np.random.uniform(0,1,(batch_size, max_length)))\nx_mask = x_mask>0.5\n\ny=mha(x,x_mask)\nprint(y.shape) #torch.Size([4, 512, 96])\nas a comparison:
\noriginal linear multi-head-attention:
\nclass MyMultiHeadAttention(nn.Module):\n def __init__(self,\n embed_dim,\n out_dim,\n qk_dim,\n v_dim,\n num_head,\n ):\n super().__init__()\n self.embed_dim = embed_dim\n self.num_head = num_head\n self.qk_dim = qk_dim\n self.v_dim = v_dim\n\n self.q = nn.Linear(embed_dim, qk_dim*num_head)\n self.k = nn.Linear(embed_dim, qk_dim*num_head)\n self.v = nn.Linear(embed_dim, v_dim*num_head)\n\n self.out = nn.Linear(v_dim*num_head, out_dim)\n self.scale = 1/(qk_dim**0.5)\n\n #https://github.com/pytorch/pytorch/issues/40497\n def forward(self, x, x_mask):\n B,L,dim = x.shape\n #out, _ = self.mha(x,x,x, key_padding_mask=x_mask)\n num_head = self.num_head\n qk_dim = self.qk_dim\n v_dim = self.v_dim\n\n q = self.q(x)\n k = self.k(x)\n v = self.v(x)\n q = q.reshape(B, L, num_head, qk_dim).permute(0,2,1,3).contiguous()\n k = k.reshape(B, L, num_head, qk_dim).permute(0,2,3,1).contiguous()\n v = v.reshape(B, L, num_head, v_dim ).permute(0,2,1,3).contiguous()\n\n dot = torch.matmul(q, k) *self.scale # H L L\n x_mask = x_mask.reshape(B,1,1,L).expand(-1,num_head,L,-1)\n #dot[x_mask]= -1e4\n dot.masked_fill_(x_mask, -1e4)\n attn = F.softmax(dot, -1) # L L\n\n v = torch.matmul(attn, v) # L H dim\n v = v.permute(0,2,1,3).reshape(B,L, v_dim*num_head).contiguous()\n out = self.out(v)\n return out\nMy conv1d layers are not getting through the pytorch to tflite conversion. Do you still do pytorch -> onnx ->tflite conversion or do you have a tf/keras twin which you convert to tflite?
", "rawMarkdown": "My conv1d layers are not getting through the pytorch to tflite conversion. Do you still do pytorch -> onnx ->tflite conversion or do you have a tf/keras twin which you convert to tflite?", "votes": 2 } ] }, { "id": 2204321, "postDate": "2023-03-31T13:31:22.793Z", "content": "this is how much the face (upper triangle = eyes and lip) and body(lower quadliteral = shoulders and hips) have moved for one hand sign.
\nI wonder if there will be improvement in accuracy if i remove these noise motion?
\n![\"https://i.ibb.co/xgPNq9L/Selection-999-1647.png\"](\"https://i.ibb.co/xgPNq9L/Selection-999-1647.png\")
Tried to apply moving average across frames but it doesn't work for me
", "rawMarkdown": "Tried to apply moving average across frames but it doesn't work for me " } ] }, { "id": 2206111, "postDate": "2023-04-02T09:49:09.440Z", "content": "simplest normalisation code
\nREF = [500, 501, 512, 513, 159, 386, 13,]\n\ndef do_normalise_by_ref(xyz, ref): \n K = xyz.shape[-1]\n xyz_flat = ref.reshape(-1,K)\n m = np.nanmean(xyz_flat,0).reshape(1,1,K)\n s = np.nanstd(xyz_flat, 0).mean() \n xyz = xyz - m\n xyz = xyz / s\n return xyz\n\nxyz = load_relevant_data_subset(pq_file)[...,:2]\nxyz = do_normalise_by_ref(xyz,xyz[:,REF])\nREF = [500, 501, 512, 513, 159, 386, 13,]\nref = xyz[:,REF]\nxyz_flat = ref.reshape(-1,K)\nusing ref=xyz will be more accurate
\nyou can conduct experiments to find the subset of most stable normalising points
[self-study]
\n[ensemble trick]
\n[augmentation and tricks]
\nsince landmark is xyz, we are having a 3d motion capture. would be fun to use GAN to create more data. distangle motion and shape
\npossible to synthetic animate motion for new person via motion transfer. But we need to find some media pipe human xyz models.
", "rawMarkdown": "[augmentation and tricks]\n- simulate different frame length \n- simulate different NaN (as in train data)\n- simulate different missing frame or differnt frame rate? \n- 3d guassian noise\n- type of landmark. use ['face', 'left_hand', 'pose', 'right_hand'] for normalisation and augmentation\n (e.g. affine transform to group of point --> bigger mouth, longer arm ..., rotate hand)\n\nsince landmark is xyz, we are having a 3d motion capture. would be fun to use GAN to create more data. distangle motion and shape\n\npossible to synthetic animate motion for new person via motion transfer. But we need to find some media pipe human xyz models. ", "votes": 3, "replies": [ { "id": 2178001, "postDate": "2023-03-12T02:16:37.907Z", "content": "augmentation paper:
\n[1] PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation
\nhttps://arxiv.org/pdf/2105.02465.pdf
[2] DH-AUG: DH Forward Kinematics Model Driven Augmentation for 3D Human Pose Estimation
\nhttps://arxiv.org/pdf/2207.09303.pdf
i check github for ASL generator
\n\"3D Avatar ASL github \"
i wonder if motion transfer would work?
\ni.e. transfer from one signer to another
https://deepmotionediting.github.io/style_transfer
", "rawMarkdown": "i wonder if motion transfer would work?\ni.e. transfer from one signer to another\n\nhttps://deepmotionediting.github.io/style_transfer", "votes": 2 } ] } ] }, { "id": 2170203, "postDate": "2023-03-05T18:52:03.680Z", "content": "[normalization]
\nnormalization is all you need!
\ndef pre_process(xyz):\n lip = xyz[:, LIP]\n lhand = xyz[:, LHAND]\n rhand = xyz[:, RHAND]\n xyz = torch.cat([ #(none, 82, 3)\n lip,\n lhand,\n rhand,\n ],1)\n xyz = xyz - xyz[~torch.isnan(xyz)].mean(0,keepdims=True) #noramlisation to common maen\n xyz = xyz / xyz[~torch.isnan(xyz)].std(0, keepdims=True)\n xyz[torch.isnan(xyz)] = 0\n xyz = xyz[:max_length]\n return xyz\n** start training here! **\n batch_size = 64 \n experiment = ['transformer-pool-03-hand_lip', 'run_train_fold0.py']\n |---------------- VALID---------|---- TRAIN/BATCH ----------------------\nrate iter epoch | loss top1 top2 top5 | loss | time \n---------------------------------------------------------------------------------------------------\n0.00e+0 00000000* 0.00 | 5.544 0.005 0.0078 0.019 | 0.000 0.000 0.000 | 0 hr 00 min\n1.00e-4 00003600* 3.00 | 3.183 0.293 0.4145 0.572 | 2.815 0.000 0.000 | 0 hr 04 min\n1.00e-4 00007200* 6.00 | 2.637 0.406 0.5310 0.678 | 2.038 0.000 0.000 | 0 hr 08 min\n1.00e-4 00010800* 9.00 | 2.404 0.451 0.5784 0.719 | 1.671 0.000 0.000 | 0 hr 11 min\n1.00e-4 00014400* 12.00 | 2.363 0.475 0.6005 0.729 | 1.540 0.000 0.000 | 0 hr 15 min\n1.00e-4 00018000* 15.00 | 2.198 0.505 0.6321 0.758 | 1.296 0.000 0.000 | 0 hr 19 min\n1.00e-4 00021600* 18.00 | 2.146 0.519 0.6440 0.765 | 1.225 0.000 0.000 | 0 hr 23 min\n1.00e-4 00025200* 21.00 | 2.141 0.524 0.6510 0.765 | 1.090 0.000 0.000 | 0 hr 27 min\n1.00e-4 00028800* 24.00 | 2.101 0.529 0.6570 0.775 | 0.985 0.000 0.000 | 0 hr 31 min\n1.00e-4 00032400* 27.00 | 2.071 0.540 0.6621 0.777 | 0.934 0.000 0.000 | 0 hr 35 min\n1.00e-4 00036000* 30.00 | 2.131 0.538 0.6573 0.772 | 0.911 0.000 0.000 | 0 hr 39 min\n1.00e-4 00039600* 33.00 | 2.070 0.544 0.6674 0.778 | 0.823 0.000 0.000 | 0 hr 43 min\n1.00e-4 00043200* 36.00 | 2.104 0.543 0.6666 0.779 | 0.736 0.000 0.000 | 0 hr 47 min\n1.00e-4 00046800* 39.00 | 2.114 0.544 0.6701 0.779 | 0.738 0.000 0.000 | 0 hr 50 min\nfast convergence after normalisation for one layer transformer encoder with cls pooling
\nvalidation split is by participant id, i.e participant id does not overlap in test and train
learning based shape normalisation:
\nSpatial Transformer Networks
\nhttps://arxiv.org/pdf/1506.02025.pdf
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
\nhttps://openaccess.thecvf.com/content_cvpr_2017/papers/Qi_PointNet_Deep_Learning_CVPR_2017_paper.pdf
\"We predict an affine transformation matrix by a mini-network (T-net in Fig 2) and directly apply this transformation to the coordinates of input points\", This is basically a 3d spatial transformer network (not the modern seq-to-seq transformer)
", "rawMarkdown": "learning based shape normalisation:\n\n- Spatial Transformer Networks\nhttps://arxiv.org/pdf/1506.02025.pdf\n\n- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation\nhttps://openaccess.thecvf.com/content_cvpr_2017/papers/Qi_PointNet_Deep_Learning_CVPR_2017_paper.pdf\n\n\"We predict an affine transformation matrix by a mini-network (T-net in Fig 2) and directly apply this transformation to the coordinates of input points\", This is basically a 3d spatial transformer network (not the modern seq-to-seq transformer)\n\n\n\n", "votes": 1 }, { "id": 2174824, "postDate": "2023-03-09T12:24:09.203Z", "content": "effects of normalization
\n![\"https://i.ibb.co/bLYJG27/Selection-999-1295.png\"](\"https://i.ibb.co/bLYJG27/Selection-999-1295.png\")
I'm fascinated by this.
\nAs an aside, what are your thoughts about leveraging angles between keypoint-keypoint vectors within a feature set (primarily hands and pose maybe -- face wouldn't work) instead of the points themselves? I believe, although I may be wrong, that this might make the feature set more agnostic to drift, scale, etc.
\nI plan to try a simple model shortly where I replace the hand features with the representative keypoint-keypoint vector angles.
\nAnyway, definitely following, thanks for sharing !
", "rawMarkdown": "I'm fascinated by this. \n* Did you use the methods you linked to do the normalization? \n* It looks like there is an upward swinging arm (right side of left image) in the original but this looks to be shifted into the mass in the right image. Do you think that's the correct transformation?\n* How did you visualize this? (X,Y) coordinates scatter plotted clearly, but I like the soft faded way it looks. Any tips?\n\nAs an aside, what are your thoughts about leveraging angles between keypoint-keypoint vectors within a feature set (primarily hands and pose maybe -- face wouldn't work) instead of the points themselves? I believe, although I may be wrong, that this might make the feature set more agnostic to drift, scale, etc.\n\nI plan to try a simple model shortly where I replace the hand features with the representative keypoint-keypoint vector angles.\n\n---\n\nAnyway, definitely following, thanks for sharing !" }, { "id": 2176238, "postDate": "2023-03-10T13:27:31.560Z", "content": "it is just simple mean and std normalization.
\nthere is probably not enough resource (100 msec) to do complicated normalisation
fig, ax = plt.subplots()\n ax.invert_yaxis()\n\n kaggle_df = pd.read_csv(f'{root_dir}/data/asl-signs/train.ver01.csv')\n for t,d in kaggle_df.iterrows():\n print('\\r', f'{t}/{len(kaggle_df)}', end='', flush=True)\n pq_file = f'{root_dir}/data/asl-signs/{d.path}'\n xyz = load_relevant_data_subset(pq_file)\n\n\n pose = xyz[:,POSE]\n pose = pose.reshape(-1,3)\n\n non_nan = np.all(~np.isnan(pose), axis=-1)\n pose = pose[non_nan]\n\n pose = pose-pose.mean(0,keepdims=True) \n pose = pose/pose.std(0,keepdims=True)\n\n ax.scatter(pose[:,0], pose[:,1], alpha=0.01)\n\n #plt.show()\n plt.waitforbuttonpress()\nAwesome. Makes sense!
", "rawMarkdown": "Awesome. Makes sense!" }, { "id": 2177309, "postDate": "2023-03-11T11:10:47.910Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2179776, "postDate": "2023-03-13T11:55:35.670Z", "content": "Hi, I'm having trouble reproducing your results. I'm using one-layer transformer with participants 26734, 28656, 16069 as validation, and my top1 val caps at 0.35 using the following hyperparams:
\nmost likely there is some bug in your training code
\nthe training loss is shown at:
\nhttps://www.kaggle.com/competitions/asl-signs/discussion/391265#2170328
possible bug ares:
\nIt looks like I was normalizing each landmark separately and that's why it didn't work. Thanks for your help!
", "rawMarkdown": "It looks like I was normalizing each landmark separately and that's why it didn't work. Thanks for your help!" } ] } ] }, { "id": 2240056, "postDate": "2023-04-30T07:22:23.977Z", "content": "Excellent sharing！I learned a lot from this.Thank you very much!Wish you can get the top prize!
", "rawMarkdown": "Excellent sharing！I learned a lot from this.Thank you very much!Wish you can get the top prize!", "votes": 1 }, { "id": 2224702, "postDate": "2023-04-17T15:03:30.517Z", "content": "https://github.com/facebookresearch/dropout
\nDropout Reduces Underfitting
![\"https://user-images.githubusercontent.com/8370623/222586143-3500fa5b-c294-48c9-a5cf-5fac2659e519.png\"](\"https://user-images.githubusercontent.com/8370623/222586143-3500fa5b-c294-48c9-a5cf-5fac2659e519.png\")
late dropout works for me
", "rawMarkdown": "https://github.com/facebookresearch/dropout\nDropout Reduces Underfitting\n\n\n\nlate dropout works for me", "votes": 1 }, { "id": 2224196, "postDate": "2023-04-17T04:45:42.443Z", "content": "replace the word \"academics\" with \"kaggle\"
\n![\"https://i.ibb.co/rxL6Thm/Selection-999-1831.png\"](\"https://i.ibb.co/rxL6Thm/Selection-999-1831.png\")
i wonder if (inverse) DTW can be used as augmentation?
\nhttps://ai.googleblog.com/2021/01/recognizing-pose-similarity-in-images.html
Hi, I've noticed that you use xyz = xyz - xyz[~torch.isnan(xyz)].mean(0,keepdim=True) to normalize the input. However, xyz[~torch.isnan(xyz)] (or np.isnan) returns a flattened tensor of size (num_frames * num_landmarks * 3 - num_nans), which means that calling mean will return a scalar. In other words, what you're really doing is subtracting the mean of everything in the sequence instead of normalizing each landmark across frames (passing dim=0 and keepdim=True). Am I getting it correctly?
Nevertheless, I believe the correct mean normalization here is to subtract the mean of all landmark positions for each frame by doing something like xyz -= xyz.nanmean(dim=1, keepdim=True). It might be even better to normalize x, y, and z separately but I haven't figured out how to do it yet.
do a few print statement:
\nprint(xyz.shape)\nprint((torch.isnan(xyz)).shape)\nprint((xyz[~torch.isnan(xyz)]).shape)\nprint((xyz[~torch.isnan(xyz)].mean(0,keepdim=True)).shape)\nHere are the sizes of everything, which are literally the same as what I said above. Was it your original intention to normalize by the single mean of all landmarks in the sequence? Just curious about this as dim=0, keepdim=True means the opposite.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6782247%2F733adc2538a07c8859e8a7b41bcb027a%2FScreenshot%20from%202023-04-01%2008-36-47.png?generation=1680313038017988&alt=media\")
\n
\n
Glad to help. I'm not sure how to do the same for unit variance as there is no such thing as nanstd. Please share if you know.
i make the mistake becuase of
\nhttps://pytorch.org/docs/stable/generated/torch.nanmean.html
quote; \"(torch.nanmean(a) is equivalent to torch.mean(a[~a.isnan()])).\"
\ni suggest you use np for preparaing pytorch train dataset.
\nfor tflite, you will be using tf functions, so it is not a problem
\n(tf.experimental.numpy.std)
sorry for the confusion. the original implementation is \"almost correct\"
\n![\"https://i.ibb.co/095CCVD/Selection-999-1671.png\"](\"https://i.ibb.co/095CCVD/Selection-999-1671.png\")
I wonder if it's better to normalize per frame, though. I'll test that out.
\nSince you're normalizing everything at once, I suggest you can just delete the arguments for mean as they have no effects on a single scalar and only create confusion.
\n[Edit] Wait, what do you mean by \"almost correct\"? Can you clarify what kind of normalization are we using here?
if your xyz is already zero mean, then the average distance of points from center(0,0,0) should be
\nvar = np.nanstd(xyz.reshape(-1,3))**2 # mean of x**2, y**2, zz*2\ndistance_sq = var.sum()\ndistance = distance_sq**0.5\nthis should be the normalising factor
", "rawMarkdown": "if your xyz is already zero mean, then the average distance of points from center(0,0,0) should be\n```\nvar = np.nanstd(xyz.reshape(-1,3))**2 # mean of x**2, y**2, zz*2\ndistance_sq = var.sum()\ndistance = distance_sq**0.5\n```\n\nthis should be the normalising factor", "votes": 2 } ] }, { "id": 2206355, "postDate": "2023-04-02T13:54:54.773Z", "content": "By normalising x, y, z separately you meant sth like this?
\nxyz.nanmean(dim=(0, 1), keepdim=True)'
It results in normalisation of each of dimensions separately, so that mean and std for x, y, and z are the same (notice that means are around 0):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2423410%2F20c87321212b07a3e5d3c1be4eb9ff3f%2FScreenshot%202023-04-02%20at%2015.59.16.png?generation=1680443969118291&alt=media\")
Yes, but I also wanted to normalize the spatial coordinates for each frame, which means to only reduce dim=1. But it's just my intuition and I'm not sure if it will benefit the model. You can check out hengck23's replies above.
\nWe're getting the means close to 0 and the stds close to 1 as the original data are already normalized. This step is just for those artifacts that are outside of [0,1].
", "rawMarkdown": "Yes, but I also wanted to normalize the spatial coordinates for each frame, which means to only reduce dim=1. But it's just my intuition and I'm not sure if it will benefit the model. You can check out hengck23's replies above.\n\nWe're getting the means close to 0 and the stds close to 1 as the original data are already normalized. This step is just for those [artifacts](https://www.kaggle.com/competitions/asl-signs/discussion/392286#2171542) that are outside of [0,1]." } ] } ] }, { "id": 2197663, "postDate": "2023-03-26T09:48:45.067Z", "content": "[previous results]
\n15-mar
\n![\"https://i.ibb.co/Yf52njT/Selection-999-1544.png\"](\"https://i.ibb.co/Yf52njT/Selection-999-1544.png\")
03-apr
\n![\"https://i.ibb.co/TgWJk32/Selection-999-1677.png\"](\"https://i.ibb.co/TgWJk32/Selection-999-1677.png\")
i just realise my work flow is:
\n1.click edit notebook
\n2.click open data in new tab
3.click submit
\ni wish to hvae one button just update dataset directly from the notebook page.
\nno need all the \"click check for update\"
I wish you could just submit the model.tflite directly like csv submissions. But it's handy enough😗
\nI like this new kind of kaggle competitions focused on edge computing. Kudos to the kaggle team making it happen!
https://www.kaggle.com/discussions/general/395353
\nmaybe one day we can have a chatgpt AI bot to help us with that:
\nprompt 'upload new tflitemodel and submit'
and i think this is useful
\nprompt 'check license issue for this external dataset at www.xxx.xxx. if there is an issue, send email to moderator and ask'
did another try mirror augmentation?
\nif yes, can i have your feedbacks?
I've tried mirror augmentation with a simple MLP model, but didn't get better results for now, I'm still stuck at 0.63LB with and without this augmentation.
\nI'll experiment further this weekend and I'm interested in other feedbacks on this topic as I supposed it would mitigate the right-handed/left-handed issue.
How do you implement the mirror augmentation?
", "rawMarkdown": "How do you implement the mirror augmentation?" }, { "id": 2210891, "postDate": "2023-04-05T17:13:06.600Z", "content": "I simply multiply the x coordinates by -1 after normalization
", "rawMarkdown": "I simply multiply the x coordinates by -1 after normalization" } ] } ] }, { "id": 2176454, "postDate": "2023-03-10T17:03:18.613Z", "content": "[pretrain model]
\nhttps://github.com/AI4Bharat/OpenHands
\npretrain model based on mediapipe !!!!
this is one thing what i want to do:
\ngenerative pretraining
\n[1]SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
\nhttps://arxiv.org/pdf/2110.05382.pdf
![\"https://i.ibb.co/bd6YCGS/Selection-999-1327.png\"](\"https://i.ibb.co/bd6YCGS/Selection-999-1327.png\")
It is interesting! Do you have some thoughts about creating baseline using that pretrained model?
", "rawMarkdown": "It is interesting! Do you have some thoughts about creating baseline using that pretrained model?", "replies": [ { "id": 2178093, "postDate": "2023-03-12T05:19:08.527Z", "content": "related: tokenization as pretraining
\n![\"https://i.ibb.co/ZYrHjCv/Selection-999-1357.png\"](\"https://i.ibb.co/ZYrHjCv/Selection-999-1357.png\")
BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization
\nhttps://arxiv.org/pdf/2302.05075.pdf
diffusion-based autoencoder (for augmentation?)
\nVector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation
\nhttps://arxiv.org/pdf/2208.09141.pdf
It's too large architecture, I think.
", "rawMarkdown": "It's too large architecture, I think." }, { "id": 2178214, "postDate": "2023-03-12T08:16:53.277Z", "content": "you can distill later
", "rawMarkdown": "you can distill later", "votes": 1 }, { "id": 2181076, "postDate": "2023-03-14T09:48:18.840Z", "content": "in theory, the max numbers of hand pattern is fixed(e.g. limited by rotation angle of finger joints, etc). hand you can actually build a \"tokenizer\" (aka lookup table)
\nit is interesting to make a tsne plot of the the hands
", "rawMarkdown": "in theory, the max numbers of hand pattern is fixed(e.g. limited by rotation angle of finger joints, etc). hand you can actually build a \"tokenizer\" (aka lookup table)\n\nit is interesting to make a tsne plot of the the hands", "votes": 1 } ] } ] }, { "id": 2165058, "postDate": "2023-03-02T00:46:44.770Z", "content": "[framework conversion]
\nkeras, pytorch:
\nhttps://github.com/leondgarse/keras_cv_attention_models
pytorch, onnx, tf
\nhttps://github.com/PINTO0309/onnx2tf
i check some of the confusing words …. i wonder if a second classifier for specific pairs (or groups) of confusing words would help (e.g. using more features, number of frames, etc)
\n167 pen (99)\n 43, 0.43434, 167 pen \n 28, 0.28283, 168 pencil \n 9, 0.09091, 40 chair \n 7, 0.07071, 176 potty \n 2, 0.02020, 0 TV \n\n168 pencil (96)\n 66, 0.68750, 168 pencil \n 15, 0.15625, 167 pen \n 2, 0.02083, 6 another \n 2, 0.02083, 176 potty \n\n129 kitty (103)\n 77, 0.74757, 129 kitty \n 10, 0.09709, 38 cat \n 3, 0.02913, 19 bee \n 3, 0.02913, 173 please \n\n\n11 awake (96)\n 46, 0.47917, 11 awake \n 34, 0.35417, 233 wake \n 7, 0.07292, 131 later \nhttps://arxiv.org/pdf/2303.12080.pdf
\nNatural Language-Assisted Sign Language Recognition
you can see that the confused words has similary meaning (e.g. kitty/cat, pencil/pen)
\ncheck the paper on how to improve on this.
\n(this may the the reason why label smooth work for me, but according to the paper word-based label smooth works better)
Can I ask the simple question, which classes you didn't apply flip-agumentations. Thankss
", "rawMarkdown": "Can I ask the simple question, which classes you didn't apply flip-agumentations. Thankss " } ] } ] }, { "id": 2211075, "postDate": "2023-04-05T19:24:21.707Z", "content": "distangle hand shape and hand motion:
\nPhonologically-meaningful Subunits for Deep Learning-based Sign Language Recognition
\nhttps://www.slrtp.com/papers/full_papers/SLRTP.FP.02.012.paper.pdf
motion token?
\n\"The large majority of sign language recognition systems based on deep learning adopt a word model approach. Here we present a system that works with subunits, rather than word models. We propose a pipelined approach to deep learning that uses a factorisation algorithm to derive hand motion features, embedded within a low-rank trajectory space.\"
\n\"To separate the real hand motions from the camera and whole body movements of the signer, we propose to use a non-rigid structure from motion (NRSfM) technique based on the factorisation method [54].\"
", "rawMarkdown": "distangle hand shape and hand motion:\nPhonologically-meaningful Subunits for Deep Learning-based Sign Language Recognition\nhttps://www.slrtp.com/papers/full_papers/SLRTP.FP.02.012.paper.pdf\n\nmotion token?\n\n\"The large majority of sign language recognition systems based on deep learning adopt a word model approach. Here we present a system that works with subunits, rather than word models. We propose a pipelined approach to deep learning that uses a factorisation algorithm to derive hand motion features, embedded within a low-rank trajectory space.\"\n\n\n\"To separate the real hand motions from the camera and whole body movements of the signer, we propose to use a non-rigid structure from motion (NRSfM) technique based on the factorisation method [54].\"\n\n", "votes": 2 }, { "id": 2200211, "postDate": "2023-03-28T11:23:42.713Z", "content": "i probably got it all wrong.
\nthe correct way to shape normalise and do feature extraction is given by:
https://google.github.io/mediapipe/solutions/pose_classification.html
\nhttps://www.tensorflow.org/lite/tutorials/pose_classification
Next, convert the landmark coordinates to a feature vector by:\n\n- Moving the pose center to the origin.\n- Scaling the pose so that the pose size becomes 1\n- Flattening these coordinates into a feature vector\nI am worried about \"scaling the pose so that the pose size becomes 1\". Let's say we are focused on the hand skeleton. If we have flat palm and fist - both of them would be scaled to the size of 1. Would it be beneficial for the model? I guess without per sample scaling it would be possible to extract additional information right from a scale of a given sign. What do you think about it?
", "rawMarkdown": "```\nNext, convert the landmark coordinates to a feature vector by:\n\n- Moving the pose center to the origin.\n- Scaling the pose so that the pose size becomes 1\n- Flattening these coordinates into a feature vector\n```\n\nI am worried about \"scaling the pose so that the pose size becomes 1\". Let's say we are focused on the hand skeleton. If we have flat palm and fist - both of them would be scaled to the size of 1. Would it be beneficial for the model? I guess without per sample scaling it would be possible to extract additional information right from a scale of a given sign. What do you think about it?", "replies": [ { "id": 2202120, "postDate": "2023-03-29T18:46:10.453Z", "content": "\"i probably got it all wrong.\"
\nbecuase if your feature is pairwise distance, then your normalisation is also pairwise distance.
\nobjective of normalisation is to make feature values the same for similar class
Hi, my pairwise distance calculation code reduces training speed dramatically from 20 batches/s to 4 batches/s. Do you precalculate pairwise distances or do you calculate it live? Do you have similar speed reduction?
", "rawMarkdown": "Hi, my pairwise distance calculation code reduces training speed dramatically from 20 batches/s to 4 batches/s. Do you precalculate pairwise distances or do you calculate it live? Do you have similar speed reduction?" }, { "id": 2205319, "postDate": "2023-04-01T12:32:13.977Z", "content": "If you are using tf.data.Dataset I would recommend you to use .cache() after feature calculation. It will precalculate all features and cache them. All other epochs would use cache instead of calculation all from scatch
", "rawMarkdown": "If you are using tf.data.Dataset I would recommend you to use .cache() after feature calculation. It will precalculate all features and cache them. All other epochs would use cache instead of calculation all from scatch" } ] } ] }, { "id": 2189046, "postDate": "2023-03-20T07:37:41.097Z", "content": "we have an interesting results of LB0.71 with single fold using all train data
", "rawMarkdown": "we have an interesting results of LB0.71 with single fold using all train data", "votes": 2, "replies": [ { "id": 2193611, "postDate": "2023-03-23T11:37:21.013Z", "content": "Yes, I felt it on the ensemble. When adding a new model, the increase in lb to 0.02 is too large. This means that unit models do not extract everything from the data. I used lb models 0.6-0.64 which resulted in 0.69. Good luck
", "rawMarkdown": "Yes, I felt it on the ensemble. When adding a new model, the increase in lb to 0.02 is too large. This means that unit models do not extract everything from the data. I used lb models 0.6-0.64 which resulted in 0.69. Good luck" }, { "id": 2195003, "postDate": "2023-03-24T10:34:29.210Z", "content": "I reach 0.71 with only one fold just now.🤓🤓
", "rawMarkdown": "I reach 0.71 with only one fold just now.🤓🤓", "votes": 1 } ] }, { "id": 2178024, "postDate": "2023-03-12T02:33:03.533Z", "content": "[opensource implementation]
\n[0] ST-GCN
\nhttps://github.com/yysijie/st-gcn
[1] SL-GCN
\nhttps://github.com/jackyjsy/SAM-SLR-v2
[2] ECCV 2022 Sign Spotting Challenge 3rd ranked solution
\nhttps://chalearnlap.cvc.uab.cat/media/results/None/top3-track1_H1KGRhQ.pdf
\nhttps://github.com/ycmin95/Chalearn_2022_Sign_Spotting_MSSL_track/tree/master/dataset
include how to use mediapipe to generate data
", "rawMarkdown": "[opensource implementation]\n\n[0] ST-GCN \nhttps://github.com/yysijie/st-gcn\n\n[1] SL-GCN \nhttps://github.com/jackyjsy/SAM-SLR-v2\n\n[2] ECCV 2022 Sign Spotting Challenge 3rd ranked solution\nhttps://chalearnlap.cvc.uab.cat/media/results/None/top3-track1_H1KGRhQ.pdf \nhttps://github.com/ycmin95/Chalearn_2022_Sign_Spotting_MSSL_track/tree/master/dataset\n\ninclude how to use mediapipe to generate data\n", "votes": 2 }, { "id": 2163489, "postDate": "2023-02-28T21:14:11.653Z", "content": "[tflite conversion]
\nextra attribute for training (aux loss). if you have external training set, you can now use non kaggle gloss (word labels) for training too!
\nTowards Zero-shot Sign Language Recognition
\nhttps://arxiv.org/pdf/2201.05914.pdf
\nhttps://bmvc2019.org/wp-content/uploads/papers/0122-paper.pdf
\"We further annotate the datasets with high-level attributes that are gathered from American Sign Language Hand Shape Dictionary [87].\"
\na page from American Sign Language Hand Shape Dictionary:
\n![\"https://i.ibb.co/Zzdt317/6e980c3584164c851f4b2d6788bb46626712a27c.jpg\"](\"https://i.ibb.co/Zzdt317/6e980c3584164c851f4b2d6788bb46626712a27c.jpg\")
so you have attribute like:
\n![\"https://i.ibb.co/3TdzLM5/Selection-999-1856.png\"](\"https://i.ibb.co/3TdzLM5/Selection-999-1856.png\")
how do you get the point_dim 1414 in the lastest update?
", "rawMarkdown": "how do you get the point_dim 1414 in the lastest update?" }, { "id": 2222116, "postDate": "2023-04-14T22:59:25.597Z", "content": "there is another way (maybe better?) to approach the problem:
\nsegmentation :
an extension will be classify every N-frame intervals, etc to learn videolet features
\nyou can using classification to perform vector quantisation (VQ) … kinda fake AE
", "rawMarkdown": "there is another way (maybe better?) to approach the problem:\nsegmentation : \n- classify every frame (instead of one video), i.e. making frame features\n- you can combine several videos during training\n- if you want you can train an aggregator at the end to combine frame features into video features to do video prediction\n\n---\n\nan extension will be classify every N-frame intervals, etc to learn videolet features \n\n---\n\nyou can using classification to perform vector quantisation (VQ) ... kinda fake AE" }, { "id": 2211093, "postDate": "2023-04-05T19:39:46.310Z", "content": "joint angle as a feature:
\nhttps://github.com/google/mediapipe/issues/2999
\nhttps://github.com/TemugeB/joint_angles_calculate
Could you say which of participants are in fold 0 in your table?
", "rawMarkdown": "Could you say which of participants are in fold 0 in your table?", "replies": [ { "id": 2216136, "postDate": "2023-04-09T19:52:14.493Z", "content": "I think its [49445, 61333, 4718, 2044, 37779]
", "rawMarkdown": "I think its [49445, 61333, 4718, 2044, 37779]" } ] }, { "id": 2209724, "postDate": "2023-04-04T21:08:42.600Z", "content": "how to augment:
\nconceptually:
\n![\"https://i.ibb.co/FsskMh7/Selection-999-1690.png\"](\"https://i.ibb.co/FsskMh7/Selection-999-1690.png\")
need to constraint the augmentation correctly.
\nmore operations will be stretch shear
you can observe a few coordinate images of the same word of the same/different signers
\nactually one can perform classification on these coordinate images (i.e. treat it as an image problem) if you can render perform 2d convolution fast enough …
", "rawMarkdown": "how to augment:\n\nconceptually:\n\n\nneed to constraint the augmentation correctly.\nmore operations will be stretch shear\n\nyou can observe a few coordinate images of the same word of the same/different signers\n\n---\n\nactually one can perform classification on these coordinate images (i.e. treat it as an image problem) if you can render perform 2d convolution fast enough ... " }, { "id": 2209702, "postDate": "2023-04-04T20:46:35.190Z", "content": "a possible way to select pairwise distance feature:
\nSequential Attention for Feature Selection
\nhttps://arxiv.org/abs/2209.14881
Thank you for sharing. Did you use entire dataset as training data? How can you check the accuracy of a model at each checkpoint when using all train samples?
", "rawMarkdown": "Thank you for sharing. Did you use entire dataset as training data? How can you check the accuracy of a model at each checkpoint when using all train samples?", "replies": [ { "id": 2204423, "postDate": "2023-03-31T14:59:33.130Z", "content": "![\"https://i.ibb.co/syVRXGD/Selection-999-1648.png\"](\"https://i.ibb.co/syVRXGD/Selection-999-1648.png\")
this is a bit tricky. but if you can set the correct regularization (and if the data has no noise), you may have this
", "rawMarkdown": "\n\nthis is a bit tricky. but if you can set the correct regularization (and if the data has no noise), you may have this", "votes": 1, "replies": [ { "id": 2204436, "postDate": "2023-03-31T15:03:47.210Z", "content": "my train log:
\nyou can see my loss are almost flat
\n** start training here! **\n batch_size = 64 \n experiment = ['tx025', 'run_train_fold0.py']\n |---------------- VALID---------|---- TRAIN/BATCH ----------------------\nrate iter epoch | loss top1 top2 top5 | loss | time \n---------------------------------------------------------------------------------------------------\n0.00e+0 00000000* 0.00 | 0.000 0.000 0.0000 0.000 | 0.000 0.000 0.000 | 0 hr 00 min\n1.00e-4 00003351* 3.00 | 3.658 0.477 0.6162 0.760 | 5.278 0.000 0.000 | 0 hr 08 min\n1.00e-4 00006702* 6.00 | 3.250 0.574 0.7079 0.823 | 5.191 0.000 0.000 | 0 hr 16 min\n1.00e-4 00010053* 9.00 | 3.088 0.621 0.7418 0.845 | 5.148 0.000 0.000 | 0 hr 24 min\n1.00e-4 00013404* 12.00 | 3.047 0.631 0.7559 0.850 | 5.120 0.000 0.000 | 0 hr 32 min\n1.00e-4 00016755* 15.00 | 3.012 0.643 0.7622 0.854 | 5.109 0.000 0.000 | 0 hr 40 min\n1.00e-4 00020106* 18.00 | 3.002 0.645 0.7631 0.853 | 5.095 0.000 0.000 | 0 hr 48 min\n1.00e-4 00023457* 21.00 | 2.970 0.645 0.7624 0.855 | 5.083 0.000 0.000 | 0 hr 56 min\n1.00e-4 00026808* 24.00 | 2.977 0.654 0.7672 0.858 | 5.075 0.000 0.000 | 1 hr 04 min\n1.00e-4 00030159* 27.00 | 2.972 0.655 0.7690 0.856 | 5.068 0.000 0.000 | 1 hr 12 min\n1.00e-4 00033510* 30.00 | 2.953 0.660 0.7722 0.858 | 5.060 0.000 0.000 | 1 hr 20 min\n1.00e-4 00036861* 33.00 | 2.949 0.663 0.7742 0.860 | 5.058 0.000 0.000 | 1 hr 28 min\n1.00e-4 00040212* 36.00 | 2.952 0.663 0.7802 0.863 | 5.048 0.000 0.000 | 1 hr 35 min\n1.00e-4 00043563* 39.00 | 2.958 0.669 0.7794 0.859 | 5.047 0.000 0.000 | 1 hr 43 min\n1.00e-4 00046914* 42.00 | 3.010 0.663 0.7742 0.856 | 5.045 0.000 0.000 | 1 hr 51 min\n1.00e-4 00050265* 45.00 | 2.954 0.665 0.7728 0.858 | 5.040 0.000 0.000 | 1 hr 59 min\n1.00e-4 00053616* 48.00 | 2.976 0.665 0.7741 0.858 | 5.035 0.000 0.000 | 2 hr 07 min\n1.00e-4 00056967* 51.00 | 3.035 0.668 0.7744 0.857 | 5.028 0.000 0.000 | 2 hr 15 min\n1.00e-4 00060318* 54.00 | 2.952 0.669 0.7762 0.862 | 5.030 0.000 0.000 | 2 hr 23 min\n1.00e-4 00063669* 57.00 | 2.966 0.667 0.7776 0.861 | 5.023 0.000 0.000 | 2 hr 31 min\n1.00e-4 00067020* 60.00 | 2.973 0.671 0.7791 0.860 | 5.028 0.000 0.000 | 2 hr 39 min\n1.00e-4 00070371* 63.00 | 2.954 0.671 0.7773 0.861 | 5.020 0.000 0.000 | 2 hr 47 min\n1.00e-4 00073722* 66.00 | 2.966 0.666 0.7757 0.858 | 5.018 0.000 0.000 | 2 hr 55 min\n1.00e-4 00077073* 69.00 | 3.002 0.667 0.7763 0.857 | 5.016 0.000 0.000 | 3 hr 03 min\n1.00e-4 00080424* 72.00 | 2.986 0.670 0.7760 0.859 | 5.013 0.000 0.000 | 3 hr 12 min\n1.00e-4 00083775* 75.00 | 3.026 0.673 0.7779 0.858 | 5.011 0.000 0.000 | 3 hr 20 min\n1.00e-4 00087126* 78.00 | 2.970 0.671 0.7793 0.858 | 5.009 0.000 0.000 | 3 hr 28 min\n1.00e-4 00090477* 81.00 | 2.983 0.676 0.7801 0.858 | 5.008 0.000 0.000 | 3 hr 36 min\n1.00e-4 00093828* 84.00 | 2.952 0.677 0.7807 0.861 | 5.006 0.000 0.000 | 3 hr 45 min\n1.00e-4 00097179* 87.00 | 3.004 0.667 0.7775 0.858 | 5.004 0.000 0.000 | 3 hr 53 min\n1.00e-4 00100530* 90.00 | 2.985 0.674 0.7786 0.859 | 5.003 0.000 0.000 | 4 hr 01 min\n1.00e-4 00103881* 93.00 | 3.067 0.655 0.7676 0.850 | 5.001 0.000 0.000 | 4 hr 09 min\n1.00e-4 00107232* 96.00 | 2.987 0.674 0.7768 0.856 | 4.997 0.000 0.000 | 4 hr 16 min\n1.00e-4 00110583* 99.00 | 2.980 0.673 0.7795 0.858 | 5.000 0.000 0.000 | 4 hr 24 min\n1.00e-4 00113934* 102.00 | 3.001 0.669 0.7751 0.856 | 4.994 0.000 0.000 | 4 hr 32 min\n1.00e-4 00117285* 105.00 | 3.035 0.666 0.7719 0.852 | 4.994 0.000 0.000 | 4 hr 40 min\n1.00e-4 00120636* 108.00 | 3.020 0.668 0.7744 0.855 | 4.992 0.000 0.000 | 4 hr 48 min\n1.00e-4 00123987* 111.00 | 3.019 0.670 0.7756 0.858 | 4.994 0.000 0.000 | 4 hr 55 min\n1.00e-4 00127338* 114.00 | 2.999 0.676 0.7800 0.858 | 4.991 0.000 0.000 | 5 hr 03 min\n1.00e-4 00130689* 117.00 | 2.995 0.674 0.7771 0.856 | 4.990 0.000 0.000 | 5 hr 11 min\n1.00e-4 00134040* 120.00 | 3.037 0.669 0.7742 0.850 | 4.988 0.000 0.000 | 5 hr 19 min\n1.00e-4 00137391* 123.00 | 3.013 0.667 0.7762 0.855 | 4.988 0.000 0.000 | 5 hr 27 min\n1.00e-4 00140742* 126.00 | 3.045 0.668 0.7736 0.853 | 4.984 0.000 0.000 | 5 hr 35 min\n1.00e-4 00144093* 129.00 | 3.013 0.673 0.7757 0.854 | 4.985 0.000 0.000 | 5 hr 43 min\n1.00e-4 00147444* 132.00 | 3.051 0.674 0.7790 0.854 | 4.980 0.000 0.000 | 5 hr 50 min\n1.00e-4 00150795* 135.00 | 3.060 0.667 0.7693 0.849 | 4.983 0.000 0.000 | 5 hr 58 min\n1.00e-4 00154146* 138.00 | 3.054 0.669 0.7747 0.852 | 4.982 0.000 0.000 | 6 hr 06 min\n1.00e-4 00157497* 141.00 | 3.038 0.669 0.7746 0.852 | 4.981 0.000 0.000 | 6 hr 14 min\ncheck also the loss curve at: https://www.kaggle.com/code/sabinaabdurakhmanova/transformer-training-with-learnable-weight-pe
", "rawMarkdown": "check also the loss curve at: https://www.kaggle.com/code/sabinaabdurakhmanova/transformer-training-with-learnable-weight-pe" }, { "id": 2204464, "postDate": "2023-03-31T15:32:47.537Z", "content": "Edit: I think this is possibly due to number of frames - TF code has only 32 frames and with PyTorch I was trying much higher number of frames like 256/512.
\nThis is interesting @hengck23 , it seems one epoch is taking 8mins. I converted this code to PyTorch and it ran in like <30sec per epoch. (I haven't been able to create inference yet, which is a different problem) ~~
\nMy current dataset is based on your dataset and code posted here. However when I try to use .npy file / pickle file with this best I get is ~2.5mins per epoch and I haven't understood why. Any thoughts much appreciated, Thank you 🙏
", "rawMarkdown": "Edit: I think this is possibly due to number of frames - TF code has only 32 frames and with PyTorch I was trying much higher number of frames like 256/512. \n\nThis is interesting @hengck23 , it seems one epoch is taking 8mins. I converted [this code](https://www.kaggle.com/code/markwijkhuizen/gislr-tf-data-processing-transformer-training) to PyTorch and it ran in like <30sec per epoch. (I haven't been able to create inference yet, which is a different problem) ~~\n\nMy current dataset is based on your dataset and code posted [here](https://www.kaggle.com/datasets/hengck23/asl-demo). However when I try to use .npy file / pickle file with this best I get is ~2.5mins per epoch and I haven't understood why. Any thoughts much appreciated, Thank you 🙏" }, { "id": 2205038, "postDate": "2023-04-01T07:36:22.143Z", "content": "it depends on if embed dim and point dim.
\naugment that use scipy interpolation is slow, etc
", "rawMarkdown": "it depends on if embed dim and point dim.\n\naugment that use scipy interpolation is slow, etc", "votes": 1 } ] } ] }, { "id": 2203287, "postDate": "2023-03-30T17:05:00.597Z", "content": "I am getting out of memory error, even when taking emb_dim = 256, head = 4, transformer_layer = 1. any optimization i am missing out on?
", "rawMarkdown": "I am getting out of memory error, even when taking emb_dim = 256, head = 4, transformer_layer = 1. any optimization i am missing out on?", "replies": [ { "id": 2203309, "postDate": "2023-03-30T17:33:06.200Z", "content": "I had similar problem, make sure to use TF Input layer rather than pytorch
\nSee this - https://www.kaggle.com/competitions/asl-signs/discussion/390935#2176801
Edit: see discussion below
", "rawMarkdown": "I had similar problem, make sure to use TF Input layer rather than pytorch \nSee this - https://www.kaggle.com/competitions/asl-signs/discussion/390935#2176801 \n\nEdit:[ see discussion below ](https://www.kaggle.com/competitions/asl-signs/discussion/391265#2188739)", "replies": [ { "id": 2205331, "postDate": "2023-04-01T12:38:25.987Z", "content": "Thanks !!!
\nhow you load(reading file and preprocessing) data, it is taking a lot of time, am loading data during training. for 1 epoch it takes 1 hour on Kaggle gpu.
May I ask are you still using 1-layer transformer? I can't believe that one attention pooling can achieve 73 single-fold score.
\nOr maybe your deeper embedding layer (fc-norm-act-fc-norm-act) can do the same feature processing as more transformer layers?
", "rawMarkdown": "May I ask are you still using 1-layer transformer? I can't believe that one attention pooling can achieve 73 single-fold score. \n\nOr maybe your deeper embedding layer (fc-norm-act-fc-norm-act) can do the same feature processing as more transformer layers?", "replies": [ { "id": 2202945, "postDate": "2023-03-30T12:15:06.343Z", "content": "\"using 1-layer transformer? \"
\nyes.
the embedding (below) is not very deep either
\nclass XEmbed(nn.Module):\n def __init__(self,\n ):\n super().__init__()\n self.v = nn.Sequential(\n nn.Linear(point_dim, embed_dim*2, bias=True),\n nn.LayerNorm(embed_dim*2),\n nn.ReLU(inplace=True),\n nn.Linear(embed_dim*2, embed_dim, bias=True),\n nn.LayerNorm(embed_dim),\n nn.ReLU(inplace=True),\n ) \n def forward(self, x, x_mask):\n B,L,_ = x.shape\n v = self.v(x)\n x = v\n return x, x_mask\nGot it, seems feature preprocessing is more important than cross-frame interaction
", "rawMarkdown": "Got it, seems feature preprocessing is more important than cross-frame interaction " }, { "id": 2202972, "postDate": "2023-03-30T12:33:43.527Z", "content": "here is the full netowrk structure for LB0.73 single fold
\nhttps://www.kaggle.com/code/hengck23/lb-0-73-single-fold-transformer-architecture
", "rawMarkdown": "here is the full netowrk structure for LB0.73 single fold\n\nhttps://www.kaggle.com/code/hengck23/lb-0-73-single-fold-transformer-architecture", "votes": 1 }, { "id": 2202977, "postDate": "2023-03-30T12:40:41.453Z", "content": "\" seems feature preprocessing is more important than cross-frame interaction\"
\nnote that my input feature include frame difference motion feature.
\nit really depends how much data we have. maybe just 21 signers (in train data) is not enough to learn the motion features. (if they all move differently) … or maybe the sign words we used don't have that much motion?
\ni believe if we can get external data (i.e. use media pipe to generator your own data) or use GAN, PDDM, augmentation, etc. motion will become important and i am expecting LB of 0.80
", "rawMarkdown": "\" seems feature preprocessing is more important than cross-frame interaction\"\n\nnote that my input feature include frame difference motion feature.\n\nit really depends how much data we have. maybe just 21 signers (in train data) is not enough to learn the motion features. (if they all move differently) ... or maybe the sign words we used don't have that much motion?\n\ni believe if we can get external data (i.e. use media pipe to generator your own data) or use GAN, PDDM, augmentation, etc. motion will become important and i am expecting LB of 0.80\n" }, { "id": 2202980, "postDate": "2023-03-30T12:43:23.397Z", "content": "Amazing work! I use 9-layer transformer to reach 0.72 single fold.
", "rawMarkdown": "Amazing work! I use 9-layer transformer to reach 0.72 single fold." }, { "id": 2203191, "postDate": "2023-03-30T15:44:17.403Z", "content": "Having a Dense layer for embedding calculation, didn't we lose temporal axis here?
", "rawMarkdown": "Having a Dense layer for embedding calculation, didn't we lose temporal axis here?" }, { "id": 2203306, "postDate": "2023-03-30T17:28:17.947Z", "content": "\"Amazing work! I use 9-layer transformer\"
\nout of curiousity, i tried a 4-layer transformer. Local CV
\n1-layer: 0.706302539309203
\n4-layer: 0.7132279280456466
it is either the same or slightly better (note that slightly different hyper-parameters used).
", "rawMarkdown": "\"Amazing work! I use 9-layer transformer\"\n\nout of curiousity, i tried a 4-layer transformer. Local CV\n1-layer: 0.706302539309203\n4-layer: 0.7132279280456466\n\nit is either the same or slightly better (note that slightly different hyper-parameters used).\n " }, { "id": 2205366, "postDate": "2023-04-01T13:35:07.010Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2205487, "postDate": "2023-04-01T15:54:15.407Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] } ] }, { "id": 2202855, "postDate": "2023-03-30T11:24:00.570Z", "content": "Label smoothing and flip didn't work for me.
", "rawMarkdown": "Label smoothing and flip didn't work for me." }, { "id": 2202383, "postDate": "2023-03-30T02:40:57.767Z", "content": "Hi @hengck23 , what is cls and what is all token mean pooling?
", "rawMarkdown": "Hi @hengck23 , what is cls and what is all token mean pooling?" }, { "id": 2194498, "postDate": "2023-03-24T01:46:16.067Z", "content": "i find it easier to write your own attention.
\nyou can control q,k,v dim differently etc.
\nyou can control local attention, e.g. which query is attent to which values (according hand, lip parts or joints)
you can learned the affinity matrix of graph CGN if you treat affinity=attention weights
", "rawMarkdown": "i find it easier to write your own attention.\nyou can control q,k,v dim differently etc.\nyou can control local attention, e.g. which query is attent to which values (according hand, lip parts or joints)\n\nyou can learned the affinity matrix of graph CGN if you treat affinity=attention weights" }, { "id": 2189929, "postDate": "2023-03-20T22:31:03.683Z", "content": "Many thanks for you work @hengck23.
\nDo you ensemble your models or do you just take the best model through CV?
\nWhen I ensemble, it seems that inference exceeds an hour…
can you do validation without validation data?
\ne.g. if i used all the train images for training, how do i know if there is over fitting?
in theory, yes ….
\nyou can still use the following to judge if there is overfitting:
you should think of how these indicators change for an optimum, over-fitted, under-fitted trained models.
\nBut of course having another set of data is still the best way to check over-fitting. so instead of finding methods and indicators, you may just find another set of data (external or synthetically created)
", "rawMarkdown": "can you do validation without validation data?\ne.g. if i used all the train images for training, how do i know if there is over fitting?\n\nin theory, yes ....\nyou can still use the following to judge if there is overfitting:\n1. we usually have a training set and validation set. then we monitor the training and validation loss.\n2. But there are other indicators that is correlated to degree of overfitting. Validation loss is only one of them\n3. say if you have only train data, you can measure \n- degree of regularization and effects on train loss (for different models of different design)\n- rate of change of loss/prediction as data is perturbed\n- rate of change of loss/prediction as parameters is perturbed\n- observe attention, heatmap, etc ....\n\nyou should think of how these indicators change for an optimum, over-fitted, under-fitted trained models.\n\nBut of course having another set of data is still the best way to check over-fitting. so instead of finding methods and indicators, you may just find another set of data (external or synthetically created)\n\n\n" }, { "id": 2188739, "postDate": "2023-03-20T00:06:49.330Z", "content": "Thank you so much for sharing your approach @hengck23 I was able to train the model from here
\nI pretty much changed no settings except epochs set to 20, because I want the pipeline to work first.
\nI used the inference however I'm getting OOM even with max_length=40
\nAny suggestions ?
https://www.kaggle.com/competitions/asl-signs/discussion/390935#2176801
\nhttps://www.kaggle.com/competitions/asl-signs/discussion/393655#2177124
class InputNet(tf.keras.layers.Layer):\n def __init__(self, ):\n super(InputNet, self).__init__()\n self.lip = tf.constant([ \n......\n\n#if your input net is a keras model, you can directly use ....\nclass TFModel(tf.Module):\n def __init__(self,):\n super(TFModel, self).__init__()\n self.input_net = InputNet()\n self.single_net = .... e.g. load from file ...\n\n\n @tf.function(input_signature=[tf.TensorSpec(shape=[None, 543, 3], dtype=tf.float32, name='inputs')])\n def __call__(self, inputs):\n outputs = self.input_net(inputs) \n .... \ni just added a new folder convert1 to https://www.kaggle.com/datasets/hengck23/asl-demo
\nit shows how to use keras input model.
Thank you so much, I'll try these !
", "rawMarkdown": "Thank you so much, I'll try these ! " }, { "id": 2189403, "postDate": "2023-03-20T13:19:55.457Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] } ] }, { "id": 2188724, "postDate": "2023-03-19T23:16:49.727Z", "content": "thinking of implementing a differentiable topk feature selector …
", "rawMarkdown": "thinking of implementing a differentiable topk feature selector ... " }, { "id": 2187997, "postDate": "2023-03-19T08:22:03.700Z", "content": "Thanks for your share！！ can I ask what do you mean of \"use triu\"
", "rawMarkdown": "Thanks for your share！！ can I ask what do you mean of \"use triu\"", "replies": [ { "id": 2188014, "postDate": "2023-03-19T08:29:31.507Z", "content": "upper triangle maxtrix index (you only need half of the pairwise distance matrix)
", "rawMarkdown": "upper triangle maxtrix index (you only need half of the pairwise distance matrix)", "votes": 1, "replies": [ { "id": 2188043, "postDate": "2023-03-19T08:50:39.050Z", "content": "Got it, thanks a lot! I noticed that point_dim down to 960 from 1002 when apply \"fixed triu\", what's the meaning of \"fixed triu\"👀
", "rawMarkdown": "Got it, thanks a lot! I noticed that point_dim down to 960 from 1002 when apply \"fixed triu\", what's the meaning of \"fixed triu\"👀" } ] } ] }, { "id": 2183926, "postDate": "2023-03-16T03:29:44.980Z", "content": "\n\none (or two) layer transformer is sufficient
\n
hmmm I have different results to this, deeper transformer lead to a higher CV:
\n1 layer 0.50
\n2 layers 0.53
\n3 layers 0.55
\n4 layers 0.56
performanced is limited by data. model configuration and feature extraction are just means to ahieve it.
\nthere is better feature represenation with less transfomer layer
", "rawMarkdown": "performanced is limited by data. model configuration and feature extraction are just means to ahieve it.\n\nthere is better feature represenation with less transfomer layer\n" } ] }, { "id": 2183228, "postDate": "2023-03-15T14:37:10.477Z", "content": "Hi~ You said that one (or two) layer transformer is sufficient, is that means we only need 1 layer of MultiHeadAttention? In the paper \"Attention is all you need\", a complete transformer contains 6 duplicated encoders and decoders, and each encoder contains one MultiHeadAttention Layer and something else. I learned your code lb-0-67-one-pytorch-transformer-solution, and I think you mean we just need one layer of MultiHeadAttention but not just one layer of transformer?
Best wishes
\nhappyOldMan
one or two layer of MultiHeadAttention.
\nit also depends on the strength of your emebedding.
it may work well if you have gru or/and MultiHeadAttention
\nanyway, the key is really how you normalize and extract/represent your feature from the 2d/3d points.
", "rawMarkdown": "one or two layer of MultiHeadAttention.\nit also depends on the strength of your emebedding.\n\nit may work well if you have gru or/and MultiHeadAttention\n\n---\n\nanyway, the key is really how you normalize and extract/represent your feature from the 2d/3d points.", "replies": [ { "id": 2183317, "postDate": "2023-03-15T15:26:18.740Z", "content": "Got it, thanks!
", "rawMarkdown": "Got it, thanks!" } ] } ] }, { "id": 2182925, "postDate": "2023-03-15T11:41:14.323Z", "content": "[team forming advertisement]
\nLooking for experienced Kagglers to form a team. We are interested in working with 8-bit quantization aware training with Keras or pytorch and implementation in tflite.
\nIt's important to note that our main objective is to learn from experts in deep learning technology, specifically embedded implementation. While we haven't yet decided if we will use an int8 model for our submission, as we are aware that only TFLite supports 8-bit computation for ARM CPU.
\nWe're committed to achieving the highest ranking on LB, but we also want to emphasize that Kaggle is about having fun and enjoying the competition.
\nIf this sounds like a team you'd like to be a part of, please submit your application and state your experiences in 8-bit quantization.
\nPlease note that only shortlisted candidates will be contacted.
Thank you!
\n(drafted by chatgpt)
\nlast update 15-mar
last update 03-apr
\nthe key to winning is to create more data. Here, wre inviting external data creator to join our team.
\nTo apply, you should show your work for:
1) use mediapipe to create 543 landmarks on some open source public dataset. Show some samples using public dataset or notebook.\n2) apply some model (your model or public model) and the difference for top-1 accuracy : kaggle dataset - your dataset < 15%\nPlease note that only shortlisted candidates will be contacted.
\nThank you!
", "rawMarkdown": "last update 03-apr\n\nthe key to winning is to create more data. Here, wre inviting external data creator to join our team.\nTo apply, you should show your work for:\n\n```\n1) use mediapipe to create 543 landmarks on some open source public dataset. Show some samples using public dataset or notebook.\n2) apply some model (your model or public model) and the difference for top-1 accuracy : kaggle dataset - your dataset < 15%\n```\nPlease note that only shortlisted candidates will be contacted.\n\nThank you!" } ] }, { "id": 2182591, "postDate": "2023-03-15T07:36:04.037Z", "content": "Thanks for sharing your experiment result, I’m curious about the val acc you got. I’ve seen many got a pretty high val acc. (around 0.75), your top1 acc is lower that that, what may be the reason？
\nIn my transformer model, I tried many different modifications to the transformer block and it’s hard to break 0.7 on val acc, I got my highest acc at 0.69.
there are two validation set i use:
\n[1] one is just random split (monitor the point when validation loss = training loss, and top-1 accuracy)
\n[2] one is straification based on participant id
cv for [1] can be up to 0.79 (after SWA averaging) / LB 0.69?
\ncv for [2] can be up to 0.66 (after SWA averaging) / LB 0.67
\n\ntime_taken = 35 sec (gpu, btach=32)
\n
May I ask if this is the time of one epoch? Or is it a time of 100 epoch? I'm training on tensorflow, but it feels slow.
", "rawMarkdown": ">time_taken = 35 sec (gpu, btach=32)\n\nMay I ask if this is the time of one epoch? Or is it a time of 100 epoch? I'm training on tensorflow, but it feels slow.", "replies": [ { "id": 2168595, "postDate": "2023-03-04T11:07:50.940Z", "content": "total validation time for 20_000 videos
", "rawMarkdown": "total validation time for 20_000 videos" } ] }, { "id": 2187722, "postDate": "2023-03-19T01:56:45.423Z", "rawMarkdown": "", "votes": -5, "isDeleted": true, "replies": [ { "id": 2187785, "postDate": "2023-03-19T03:45:59.167Z", "rawMarkdown": "", "votes": 3, "isDeleted": true } ] }, { "id": 2179510, "postDate": "2023-03-13T08:14:48.807Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2206052, "postDate": "2023-04-02T08:05:45.377Z", "content": "Thanks for sharing!!
", "rawMarkdown": "Thanks for sharing!!" } ], "comments": [ { "id": 2179326, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-13T04:40:49.920000", "content": "i have been playing with controlnet and i come across this:
\n
maybe can make some fun animation video while waiting for training
", "votes": 14, "replies": [ { "id": 2180654, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-14T02:26:27.493000", "content": "landmark --> controlnet -->stable diffusion --> image+style+augment --> media pose -->landmark+augment ???
\nnote that you can have depthmap as control etc and you can change viewpoint, etc
", "votes": 2, "replies": [] }, { "id": 2202379, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-30T02:22:09.557000", "content": "controlnet based on face landmark
\nhttps://huggingface.co/georgefen/Face-Landmark-ControlNet
Hi, that is really interesting. I am guessing this will create images frame by frame and we merge all the frames to create a video. Can you give me gist of it? I'm still learning, but seems like a fun learning project. Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 2201909, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-29T16:19:45.363000", "content": "flip augmentation magic:
\nCV 0.703 (single fold)
\nLB 0.72
LHAND = np.arange(468, 489).tolist() # 21\nRHAND = np.arange(522, 543).tolist() # 21\nPOSE = np.arange(489, 522).tolist() # 33\nFACE = np.arange(0,468).tolist() #468\n\nREYE = [\n 33, 7, 163, 144, 145, 153, 154, 155, 133,\n 246, 161, 160, 159, 158, 157, 173,\n]\nLEYE = [\n 263, 249, 390, 373, 374, 380, 381, 382, 362,\n 466, 388, 387, 386, 385, 384, 398,\n]\nNOSE=[\n 1,2,98,327\n]\nSLIP = [\n 78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n 191, 80, 81, 82, 13, 312, 311, 310, 415,\n]\nSPOSE = (np.array([\n 11,13,15,12,14,16,23,24,\n])+489).tolist()\n\n\n...\n## assume zero mean xyz. so flip can be implement by multuiplication of -1\ndef do_hflip_hand(lhand, rhand):\n rhand[...,0] *= -1\n lhand[...,0] *= -1\n rhand, lhand = lhand,rhand\n return lhand, rhand\n\ndef do_hflip_spose(spose):\n spose[...,0] *= -1\n spose = spose[:,[3,4,5,0,1,2,7,6]]\n return spose\n\ndef do_hflip_slip(slip):\n slip[...,0] *= -1\n slip = slip[:,[10,9,8,7,6,5,4,3,2,1,0]+[19,18,17,16,15,14,13,12,11]]\n return slip\n\n...\n...\n xyz = load_relevant_data_subset(pq_file)\n lhand = xyz[:,LHAND]\n rhand = xyz[:,RHAND]\n spose = xyz[:,SPOSE]\n leye = xyz[:,LEYE]\n reye = xyz[:,REYE]\n slip = xyz[:,SLIP]\n nose = xyz[:,NOSE]\n\n\n if is_aug==1:\n if np.random.rand()<0.5:\n lhand, rhand = do_hflip_hand(lhand, rhand)\n spose = do_hflip_spose(spose)\n leye, reye = do_hflip_eye(leye, reye)\n slip = do_hflip_slip(slip)\n nose = do_hflip_nose(nose)\ntrain log:
\nfold_type = kaggle-part\nfold = 0\ntrain_dataset : \n len = 71518\n num_participant_id = 16\n\nvalid_dataset : \n len = 22959\n num_participant_id = 5\n...\n batch_size = 64 \n experiment = ['tx025-no-z-flip', 'run_train_fold0.py']\n |---------------- VALID---------|---- TRAIN/BATCH ----------------------\nrate iter epoch | loss top1 top2 top5 | loss | time \n---------------------------------------------------------------------------------------------------\n...\n1.00e-4 00050265* 45.00 | 2.025 0.698 0.7992 0.870 | 3.887 0.000 0.000 | 1 hr 48 min\n1.00e-4 00053616* 48.00 | 2.042 0.697 0.7969 0.870 | 3.879 0.000 0.000 | 1 hr 56 min\none may want to try \"cut and paste augment\": assemble body parts from different signers
", "votes": 2, "replies": [ { "id": 2202195, "author_name": "Andrij", "author_url": "", "post_date": "2023-03-29T20:06:22.660000", "content": "I've already tried something similar when I changed body parts. On my pipeline, the result is controversial. On the one hand, this gave an increase of 1 fold, but on the other hand, the ensemble did not improve. Of course I could have made a mistake somewhere in the code. And this reduced val_loss
", "votes": 1, "replies": [] }, { "id": 2202204, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-29T20:12:24.370000", "content": "the parts have to be synchronised correctly in time. a better way is to use gan or pddm
", "votes": 1, "replies": [] }, { "id": 2202381, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-30T02:30:20.223000", "content": "instead of cut and paste, imagine:
\nbasically, we are learning a better normalisation
\nhttps://www.youtube.com/watch?v=U9G2VlG9HbA
🤔Do other landmarks really help? In my experiments, event simply add pose landmarks (arms position) decrease CV.
", "votes": 1, "replies": [ { "id": 2202395, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-30T03:08:03.123000", "content": "it depends on how you normalised i think.
\nthere is many noise in the data becuase each signer is of different size and their camera may have different frame rate. They also move differently.
\nin theory:
\neye: should help. for sign words \"awake\", the eye is closed then opened.
\npose (arm): should help. this is global motion of hand.
but if your nomralisation is poor or your model cannot filter out the noise, then there will be easy overfitting to noise
", "votes": 4, "replies": [] }, { "id": 2202398, "author_name": "Carno Zhao", "author_url": "", "post_date": "2023-03-30T03:12:11.127000", "content": "yes, for me, normalization method also matters
", "votes": 0, "replies": [] } ] }, { "id": 2202578, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-30T06:40:03.967000", "content": "this is not too difficult if you think about it
\n
I tried a very similar approach to this, but unfortunately, I didn't see any performance improvement compared to simple augmentation(maybe my approach was too naive). However, I think it's a fascinating idea, and I'm excited to hear about your progress if you decide to give it a go:).
", "votes": 0, "replies": [] } ] }, { "id": 2229296, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-04-21T08:27:47.767000", "content": "According to LANDMARKS_REFINEMENT_LEFT_EYE_CONFIG and LANDMARKS_REFINEMENT_RIGHT_EYE_CONFIG (from https://github.com/tensorflow/tfjs-models/blob/master/face-landmarks-detection/src/tfjs/constants.ts) , your REYE if a left eye, and LEYE is a right eye. Are you sure that's correct?
", "votes": 0, "replies": [] } ] }, { "id": 2193356, "author_name": "KKY", "author_url": "", "post_date": "2023-03-23T08:28:19.040000", "content": "I love your experiments records table! thanks for sharing this!
", "votes": 8, "replies": [ { "id": 2193995, "author_name": "Dheeraj Pandey", "author_url": "", "post_date": "2023-03-23T16:50:13.230000", "content": "+1 @hengck23
", "votes": 0, "replies": [] } ] }, { "id": 2178946, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-12T19:46:44.203000", "content": "[unorganized] some proven intermediate results … will write in more detail later
\nadd motion (xyz at current frame - xyz at previous frame). i.e. each point now has 6 dim. need to handle NaN motion correctly
\nimprove CV by 0.015
add framerate interpolation augmentation scipy.interpolate.interp1d()
\nimprove CV by 0.015
ideally, if there is lots and lots of training samples, you don't have to do anything, just add more parameters and the model would learn scale, rotation invariant features.
\nif there is not much training samples, you can do augmentation, e.g. rotate/shift the hands
\nyou can also handcraft features, e.g. pairwise distance, pairwise angles of the hand points (these are invariant to rotation)
\nbetter still, graph network CGN or pointnet features are for this.
", "votes": 3, "replies": [ { "id": 2180165, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-13T16:21:10.243000", "content": "convergence is magical if you do things right !
\ntrain/validation participant id does not overlap shown below:
\n
distance matrix is the poor man's graph network D x A x inv(D) …
\ni need to google for better joint kinematics features ….
i wondered if i renderd the point into 48x48 images and use 3dCNN, what will be the speed and accuracy …
", "votes": 3, "replies": [] }, { "id": 2181090, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-14T10:11:12.587000", "content": "possible of aux loss:
\n….
", "votes": 1, "replies": [] }, { "id": 2182554, "author_name": "Lau2664", "author_url": "", "post_date": "2023-03-15T06:51:09.113000", "content": "I didn’t try 3D CNN yet, but I have tried 2D CNN. I use a (batch_size, num_frame, num_landmark, 3) tensor as input. I didn’t do much experiments, the best val acc I got is around 0.53. I didn’t record the speed. There are many skeleton-based motion recognition papers that use CNN, but according to the paper, they all achieve a fantastic result, which is much higher than 0.53… I guess we need to process the data in some ways to make it suitable to CNN.
", "votes": 0, "replies": [] }, { "id": 2182562, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-15T06:59:03.907000", "content": "
\n
@jay2333 Papers in the skeleton-based action recognition domain rely on high-quality pose data. In most cases, they use HRNet pose estimator to extract the skeleton landmarks. Furthermore, there are only a few missing values.
\nThe landmarks in our dataset are accurate, but just not as good as landmarks from HRNet. Plus, we have a lot of missing values and we need to find a way to deal with them.
", "votes": 0, "replies": [] }, { "id": 2183125, "author_name": "Lau2664", "author_url": "", "post_date": "2023-03-15T13:29:50.920000", "content": "@gregorlied Totally agree with that. According to the current public notebook, a simple fully connected network can perform well. So the main issue is the data process part.
", "votes": 1, "replies": [] }, { "id": 2183239, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-15T14:42:51.547000", "content": "\" fully connected network can perform well. \"
\nthe reason is that the seq are not really long
", "votes": 1, "replies": [] }, { "id": 2183267, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-15T15:03:07.460000", "content": "there is a trick to speedup 3d CNN or 2d CNN
\nbasically it is like CLIP (aligned text and image)
\nhere you trained joint encoder for (pose, rendered image) pair.
\nrendered image has more information so they have better embedding.
you can think of we are distilling image embedding to point embedding.
\nsince both embedding are \"aligned\", in testing we are only using point embedding (though we trained them jointly)
try to use different color for part, shade for depth when rendering ….
", "votes": 1, "replies": [] }, { "id": 2185509, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-17T05:45:15.100000", "content": "yet another aux loss,
\nMay I ask you how you actually apply interpolation? I'm trying to apply it, but even with soft parameters the score is lower
", "votes": 0, "replies": [] } ] }, { "id": 2228313, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-20T12:56:37.840000", "content": "watch this thread!
\nreceipe for LB0.76 one fold:
\ni seems that there is somthing very strange about the dataset … maybe very noisy. i have to use unconventional strong regularisation and training method ….
\nhint:
\n[1] Dropout Reduces Underfitting
\nhttps://arxiv.org/abs/2303.01500
i find that if i reduce learning rate, the model easily overfits. instead, according to paper[1], by setting dropout to different values at different training epoches, we can reduce validation loss (it has the same effects as reducing learning rate) without overfitting:
\n
fold_type = kaggle-random (20% validation, 80% train. pure random)\nfold = 0\nvalid_dataset : \n len = 18896\n num_participant_id = 21\n num_external_video = 0\n\n\ntime_taken = 0 min 27 sec\nloss = 1.7932500839233398\ntopk[0] = 0.8607112616426758 (LB 0.76)\ntopk[1] = 0.9180249788314987\ntopk[2] = 0.9374470787468248\ntopk[3] = 0.9457557154953429\ntopk[4] = 0.9513653683319221\ncheckpoint_file = /home/titanx/hengck/share1/kaggle/2022/hand-sign/result/run202/tx90-face-norm-480-0b/fold-0-kaggle-random/checkpoint/swa.model.pth\nHave you tried using ViT or are you still using Conv1d + MHA?
", "votes": 0, "replies": [ { "id": 2229582, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-21T13:40:39.923000", "content": "dropout has the effect of flatten the valley of loss landscape.
\n(this is the same as SWA, deeply supervised loss, etc)
Of course, adeverisal training SAM (sharpness aware optimizer, aka sample perturbation via weight perturbation) also flattens the valley.
\nBelow is results of SAM, which is comparable to late dropout
fold_type = kaggle-part\nfold = 0\nvalid_dataset : \n len = 22959\n num_participant_id = 5\n num_external_video = 0\n\n\n 22959 / 22959 0 min 30 sec\n\ntime_taken = 0 min 30 sec\nloss = 2.5410375595092773\ntopk[0] = 0.7265560346704996\ntopk[1] = 0.8181105448843591\ntopk[2] = 0.856265516790801\ntopk[3] = 0.8760398972080665\ntopk[4] = 0.8882791062328499\ncheckpoint_file = /home/user/share1/kaggle/2022/hand-sign/result/run202/tx90-face-norm-sam-fine-rho0.1/fold-0-kaggle-part/checkpoint/swa.model.pth\n----- end -----\nIt may be worthwhile to try additional parameters in conjunction with a 'late' configuration
", "votes": 0, "replies": [] } ] } ] }, { "id": 2163630, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-01T00:49:08.950000", "content": "[reference]
\nhttps://github.com/matyasbohacek/spoter
\nhttps://www.matyasbohacek.com/
\n(uses MediaPipe library pose too)
https://huggingface.co/spaces/matyasbohacek/spoter-demo-test
\nhttps://www.youtube.com/watch?v=6i5ODpHMny8
\n
SLGTFORMER: AN ATTENTION-BASED APPROACH TO SIGN LANGUAGE RECOGNITION
\nhttps://github.com/neilsong/SLGTformer
[external data]
\n… to be updated …
HOW TO SIGN
\nhttps://how2sign.github.io/
\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Duarte_How2Sign_A_Large-Scale_Multimodal_Dataset_for_Continuous_American_Sign_Language_CVPR_2021_paper.pdf
\n(vocab=16k, time=79 hr, signer=11)
MS-ASL American Sign Language Dataset
\nhttps://microsoft.github.io/data-for-society/dataset?d=MS-ASL-American-Sign-Language-Dataset
\nhttps://arxiv.org/pdf/1812.01053.pdf
\n(vocab=1000, num video=25513, signer=222)
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
\nhttps://www.robots.ox.ac.uk/~vgg/research/bsl1k/
\n(vocab=1000, num video=1000)
ChaLearn LAP Large Scale Signer Independent Isolated Sign Language
\nRecognition Challenge: Design, Results and Future Research
ASL-Skeleton3D and ASL-Phono: Two Novel
\nDatasets for the American Sign Language
how to read media pipline to extra keypoint for external data
\n… to be udpated …
more about PopSign learning app
\nhttps://devpost.com/software/pop-sign-learning
\nhttps://www.youtube.com/watch?v=nbQAfiJzp2g&t=430s
\nhttps://github.com/Accessible-Technology-in-Sign/PopSign
https://github.com/Accessible-Technology-in-Sign/ASLRT
\n\"A pipeline to run tracking Kinect, MediaPipe and AlphaPose experiments on sign language feature data using either HMMs, SBHMMs, or Transformers.\"
collect data you from youtube?
", "votes": 6, "replies": [ { "id": 2176272, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-10T14:03:14.067000", "content": "pytorch mediapipe … hmm
\nhttps://github.com/zmurez/MediaPipePyTorch
this image explain why hand are missing.
\nyou probably can ignore the legs
there is a related product called copycat:
\nCopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills
\nhttps://gvu.gatech.edu/research/projects/copycat-using-sign-language-recognition-help-deaf-children-acquire-language-skills#:~:text=CopyCat%20is%20a%20game%20where,language%20skills%20and%20working%20memory.
https://www.youtube.com/watch?v=WMFZVcey8FU
\nhttps://github.com/Accessible-Technology-in-Sign/ASLRT
much information on HMM + handcraft feature here
", "votes": 0, "replies": [] } ] }, { "id": 2226810, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-19T08:43:02.580000", "content": "i forget an import detail at the data card. suggestions for augmentation:
\n\"While the app provided a video example of the sign desired, signers routinely made variants of the sign based on their background and region.\"
\n\"More rarely, signers might fingerspell a sign, miss it completely, or produce the wrong sign.\"
\n\" Extraneous movements, such as scratching an itch, or the ending movement from the previous sign or the onset of the next sign, are sometimes included.\"
\n\" Conversely, some signers pressed the button late or released the button early, causing cropping in some sign examples. \"
\n\"Some signers sign with their left hand; others sign with their right. Some signers switch their signing hand. All of these situations must be handled by the game’s recognition system.\"
\na also note that some signer make repeated signs in one video
", "votes": 3, "replies": [ { "id": 2229212, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-04-21T06:49:13.603000", "content": "\"need to measure label noise and decide if you want to exclude some data\"
\nSo we need to specify some kind of \"ideal\" signs, and compare train data with it?
", "votes": 0, "replies": [ { "id": 2229217, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-21T06:51:29.157000", "content": "those that have validation/train error higher than normal are supsicous outliers
", "votes": 1, "replies": [] }, { "id": 2229256, "author_name": "Jackson You", "author_url": "", "post_date": "2023-04-21T07:37:43.810000", "content": "If the hidden dataset is very cleanly drawn, I think data with a low frame count can be left out.
", "votes": 0, "replies": [] }, { "id": 2229468, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-21T11:49:10.123000", "content": "there must be some reason why some video are very long (much longer than average).
\nat the signer moving very slowly? or are they not cropping the video (press start and end) correctly? or are they signing wrongly?
You're right, long frame data has some problems.
", "votes": 0, "replies": [] }, { "id": 2229604, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-21T14:07:24.983000", "content": "this is explain why conv1d does not work but conv1d+max pool works (about LB 0.72, compared to transformer of 0.74)
\n self.conv2 = nn.Sequential(\n nn.Conv1d(1024, 768, kernel_size=3, padding=1, stride=1),\n LayerNorm1D(768),\n nn.ReLU(inplace=True),\n nn.Dropout(p=0.1),\n nn.MaxPool1d(kernel_size=3, padding=1, stride=2),\n )\nmax pool is poor man's attention pool(softmax versus hard max)
\nalso, why adding noise (dropout, dropframe, random values, mix frame …) before mha works. it forces the model to select the correct signal.
\ni am very surprise of the small num of paramaters and performance of transformer compared to other model design (like conv1d, dense, etc)
", "votes": 0, "replies": [] } ] } ] }, { "id": 2224078, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-17T01:26:37.653000", "content": "it is suprised for me to achieve 0.75 in one fold with participant split train data(num participant = 5 in validation data). This makes me think that 0.76 is possible.
\nfold_type = kaggle-part\nfold = 0\nvalid_dataset : \n len = 22959\n num_participant_id = 5 \n\n 22959 / 22959 0 min 42 sec\n\ntime_taken = 0 min 42 sec\nloss = 2.406057596206665\ntopk[0] = 0.7277320440785748\ntopk[1] = 0.8219870203406072\ntopk[2] = 0.858530423798946\ntopk[3] = 0.8757785617840498\ntopk[4] = 0.8888888888888888\ncheckpoint_file = /home/titanx/hengck/share1/kaggle/2022/hand-sign/result/run201/tx61a-1-frame-first-scale-2/fold-0-kaggle-part/checkpoint/swa.model.pth\nand i hit the jackpot 888888888
", "votes": 3, "replies": [ { "id": 2224095, "author_name": "Rob Mulla", "author_url": "", "post_date": "2023-04-17T02:08:50.303000", "content": "888888888!
", "votes": 1, "replies": [] }, { "id": 2225136, "author_name": "chemdatafarmer", "author_url": "", "post_date": "2023-04-17T23:29:35.293000", "content": "Are these accuracies shown for the 5 holdout participants?
", "votes": 0, "replies": [] } ] }, { "id": 2206909, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-03T02:01:33.027000", "content": "a couple on conv1d (as spatial temporal encoder) + MHA attention seems to have good results.
\nsome good ideas to design efficient transformer.
[1] FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization
\nhttps://arxiv.org/pdf/2303.14189.pdf
[2] Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios
\nhttps://arxiv.org/abs/2207.05501
to avoid uncessary permute reshape, i can write my q,v,k linear operation in terms of conv1d.
\ni will be removing absolute position embedding and use convolutional position embedding instead (convolutional FFN, i.e. position information is implicitly embedding on 1d conv)
actually it is aslo possible to use 2d convolution, where (W,H) = (x,y,z of spatial, t of temporal).
", "votes": 3, "replies": [ { "id": 2206947, "author_name": "Carno Zhao", "author_url": "", "post_date": "2023-04-03T02:55:46.753000", "content": "Won't conv bring much more parameters?
\nI suppose what you mean is :
\n\ninputs_embeds = inputs_embeds + position_embeds \n\n\ninputs_embeds = inputs_embeds.permute(, , ) \ninputs_embeds = conv1d(inputs_embeds) \ninputs_embeds = inputs_embeds.permute(, , ) \nwhere the number of parameters in original is TxH, and the conv has HxHx3
\nI also tried this idea, but found that the conv is much harder to train than MHSA, maybe because of more parameters.
", "votes": 1, "replies": [ { "id": 2207339, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-03T10:49:08.177000", "content": "\n1d convolution multi-head-attention:
\nclass MyMultiHeadAttention(nn.Module):\n def __init__(self,\n embed_dim,\n out_dim,\n qk_dim,\n v_dim,\n num_head,\n ):\n super().__init__()\n self.embed_dim = embed_dim\n self.num_head = num_head\n self.qk_dim = qk_dim\n self.v_dim = v_dim\n\n self.q = nn.Conv1d(embed_dim, qk_dim*num_head,1)\n self.k = nn.Conv1d(embed_dim, qk_dim*num_head,1) # stride=2 for token reduction, kernel>1 for mixing\n self.v = nn.Conv1d(embed_dim, v_dim*num_head,1)\n\n self.out = nn.Conv1d(v_dim*num_head, out_dim, 1)\n self.scale = 1/(qk_dim**0.5)\n\n #https://github.com/pytorch/pytorch/issues/40497\n def forward(self, x, x_mask):\n B,dim,L= x.shape\n\n num_head = self.num_head\n qk_dim = self.qk_dim\n v_dim = self.v_dim\n\n q = self.q(x) #B,qk_dim,L\n k = self.k(x)\n v = self.v(x)\n q = q.reshape(B, num_head, qk_dim, L).permute(0,1,3,2).contiguous()\n k = k.reshape(B, num_head, qk_dim, L)#.permute(0,1,2,3).contiguous()\n v = v.reshape(B, num_head, v_dim, L).permute(0,1,3,2).contiguous()\n\n dot = torch.matmul(q, k) *self.scale # H L L\n x_mask = x_mask.reshape(B,1,1,L).expand(-1,num_head,L,-1)\n #dot[x_mask]= -1e4\n dot.masked_fill_(x_mask, -1e4)\n attn = F.softmax(dot, -1) # L L\n\n v = torch.matmul(attn, v) # L H dim\n v = v.permute(0,1,3,2).reshape(B, v_dim*num_head,L).contiguous()\n out = self.out(v)\n return out\n\n#---\nembed_dim = 512\nout_dim = 512\nqk_dim = 512//4 #for one head\nv_dim = 512//4\nnum_head = 4\nmax_length = 96\nbatch_size = 4\n\nmha = MyMultiHeadAttention(\n embed_dim,\n out_dim,\n qk_dim,\n v_dim,\n num_head,\n)\nx = torch.from_numpy( np.random.uniform(-1,1,(batch_size, embed_dim, max_length))).float()\nx_mask = torch.from_numpy( np.random.uniform(0,1,(batch_size, max_length)))\nx_mask = x_mask>0.5\n\ny=mha(x,x_mask)\nprint(y.shape) #torch.Size([4, 512, 96])\nas a comparison:
\noriginal linear multi-head-attention:
\nclass MyMultiHeadAttention(nn.Module):\n def __init__(self,\n embed_dim,\n out_dim,\n qk_dim,\n v_dim,\n num_head,\n ):\n super().__init__()\n self.embed_dim = embed_dim\n self.num_head = num_head\n self.qk_dim = qk_dim\n self.v_dim = v_dim\n\n self.q = nn.Linear(embed_dim, qk_dim*num_head)\n self.k = nn.Linear(embed_dim, qk_dim*num_head)\n self.v = nn.Linear(embed_dim, v_dim*num_head)\n\n self.out = nn.Linear(v_dim*num_head, out_dim)\n self.scale = 1/(qk_dim**0.5)\n\n #https://github.com/pytorch/pytorch/issues/40497\n def forward(self, x, x_mask):\n B,L,dim = x.shape\n #out, _ = self.mha(x,x,x, key_padding_mask=x_mask)\n num_head = self.num_head\n qk_dim = self.qk_dim\n v_dim = self.v_dim\n\n q = self.q(x)\n k = self.k(x)\n v = self.v(x)\n q = q.reshape(B, L, num_head, qk_dim).permute(0,2,1,3).contiguous()\n k = k.reshape(B, L, num_head, qk_dim).permute(0,2,3,1).contiguous()\n v = v.reshape(B, L, num_head, v_dim ).permute(0,2,1,3).contiguous()\n\n dot = torch.matmul(q, k) *self.scale # H L L\n x_mask = x_mask.reshape(B,1,1,L).expand(-1,num_head,L,-1)\n #dot[x_mask]= -1e4\n dot.masked_fill_(x_mask, -1e4)\n attn = F.softmax(dot, -1) # L L\n\n v = torch.matmul(attn, v) # L H dim\n v = v.permute(0,2,1,3).reshape(B,L, v_dim*num_head).contiguous()\n out = self.out(v)\n return out\nMy conv1d layers are not getting through the pytorch to tflite conversion. Do you still do pytorch -> onnx ->tflite conversion or do you have a tf/keras twin which you convert to tflite?
", "votes": 2, "replies": [] } ] }, { "id": 2204321, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-31T13:31:22.793000", "content": "this is how much the face (upper triangle = eyes and lip) and body(lower quadliteral = shoulders and hips) have moved for one hand sign.
\nI wonder if there will be improvement in accuracy if i remove these noise motion?
\n
Tried to apply moving average across frames but it doesn't work for me
", "votes": 0, "replies": [] } ] }, { "id": 2206111, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-02T09:49:09.440000", "content": "simplest normalisation code
\nREF = [500, 501, 512, 513, 159, 386, 13,]\n\ndef do_normalise_by_ref(xyz, ref): \n K = xyz.shape[-1]\n xyz_flat = ref.reshape(-1,K)\n m = np.nanmean(xyz_flat,0).reshape(1,1,K)\n s = np.nanstd(xyz_flat, 0).mean() \n xyz = xyz - m\n xyz = xyz / s\n return xyz\n\nxyz = load_relevant_data_subset(pq_file)[...,:2]\nxyz = do_normalise_by_ref(xyz,xyz[:,REF])\nREF = [500, 501, 512, 513, 159, 386, 13,]\nref = xyz[:,REF]\nxyz_flat = ref.reshape(-1,K)\nusing ref=xyz will be more accurate
\nyou can conduct experiments to find the subset of most stable normalising points
[self-study]
\n[ensemble trick]
\n[augmentation and tricks]
\nsince landmark is xyz, we are having a 3d motion capture. would be fun to use GAN to create more data. distangle motion and shape
\npossible to synthetic animate motion for new person via motion transfer. But we need to find some media pipe human xyz models.
", "votes": 3, "replies": [ { "id": 2178001, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-12T02:16:37.907000", "content": "augmentation paper:
\n[1] PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation
\nhttps://arxiv.org/pdf/2105.02465.pdf
[2] DH-AUG: DH Forward Kinematics Model Driven Augmentation for 3D Human Pose Estimation
\nhttps://arxiv.org/pdf/2207.09303.pdf
i check github for ASL generator
\n\"3D Avatar ASL github \"
i wonder if motion transfer would work?
\ni.e. transfer from one signer to another
https://deepmotionediting.github.io/style_transfer
", "votes": 2, "replies": [] } ] } ] }, { "id": 2170203, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-05T18:52:03.680000", "content": "[normalization]
\nnormalization is all you need!
\ndef pre_process(xyz):\n lip = xyz[:, LIP]\n lhand = xyz[:, LHAND]\n rhand = xyz[:, RHAND]\n xyz = torch.cat([ #(none, 82, 3)\n lip,\n lhand,\n rhand,\n ],1)\n xyz = xyz - xyz[~torch.isnan(xyz)].mean(0,keepdims=True) #noramlisation to common maen\n xyz = xyz / xyz[~torch.isnan(xyz)].std(0, keepdims=True)\n xyz[torch.isnan(xyz)] = 0\n xyz = xyz[:max_length]\n return xyz\n** start training here! **\n batch_size = 64 \n experiment = ['transformer-pool-03-hand_lip', 'run_train_fold0.py']\n |---------------- VALID---------|---- TRAIN/BATCH ----------------------\nrate iter epoch | loss top1 top2 top5 | loss | time \n---------------------------------------------------------------------------------------------------\n0.00e+0 00000000* 0.00 | 5.544 0.005 0.0078 0.019 | 0.000 0.000 0.000 | 0 hr 00 min\n1.00e-4 00003600* 3.00 | 3.183 0.293 0.4145 0.572 | 2.815 0.000 0.000 | 0 hr 04 min\n1.00e-4 00007200* 6.00 | 2.637 0.406 0.5310 0.678 | 2.038 0.000 0.000 | 0 hr 08 min\n1.00e-4 00010800* 9.00 | 2.404 0.451 0.5784 0.719 | 1.671 0.000 0.000 | 0 hr 11 min\n1.00e-4 00014400* 12.00 | 2.363 0.475 0.6005 0.729 | 1.540 0.000 0.000 | 0 hr 15 min\n1.00e-4 00018000* 15.00 | 2.198 0.505 0.6321 0.758 | 1.296 0.000 0.000 | 0 hr 19 min\n1.00e-4 00021600* 18.00 | 2.146 0.519 0.6440 0.765 | 1.225 0.000 0.000 | 0 hr 23 min\n1.00e-4 00025200* 21.00 | 2.141 0.524 0.6510 0.765 | 1.090 0.000 0.000 | 0 hr 27 min\n1.00e-4 00028800* 24.00 | 2.101 0.529 0.6570 0.775 | 0.985 0.000 0.000 | 0 hr 31 min\n1.00e-4 00032400* 27.00 | 2.071 0.540 0.6621 0.777 | 0.934 0.000 0.000 | 0 hr 35 min\n1.00e-4 00036000* 30.00 | 2.131 0.538 0.6573 0.772 | 0.911 0.000 0.000 | 0 hr 39 min\n1.00e-4 00039600* 33.00 | 2.070 0.544 0.6674 0.778 | 0.823 0.000 0.000 | 0 hr 43 min\n1.00e-4 00043200* 36.00 | 2.104 0.543 0.6666 0.779 | 0.736 0.000 0.000 | 0 hr 47 min\n1.00e-4 00046800* 39.00 | 2.114 0.544 0.6701 0.779 | 0.738 0.000 0.000 | 0 hr 50 min\nfast convergence after normalisation for one layer transformer encoder with cls pooling
\nvalidation split is by participant id, i.e participant id does not overlap in test and train
learning based shape normalisation:
\nSpatial Transformer Networks
\nhttps://arxiv.org/pdf/1506.02025.pdf
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
\nhttps://openaccess.thecvf.com/content_cvpr_2017/papers/Qi_PointNet_Deep_Learning_CVPR_2017_paper.pdf
\"We predict an affine transformation matrix by a mini-network (T-net in Fig 2) and directly apply this transformation to the coordinates of input points\", This is basically a 3d spatial transformer network (not the modern seq-to-seq transformer)
", "votes": 1, "replies": [] }, { "id": 2174824, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-09T12:24:09.203000", "content": "effects of normalization
\n
I'm fascinated by this.
\nAs an aside, what are your thoughts about leveraging angles between keypoint-keypoint vectors within a feature set (primarily hands and pose maybe -- face wouldn't work) instead of the points themselves? I believe, although I may be wrong, that this might make the feature set more agnostic to drift, scale, etc.
\nI plan to try a simple model shortly where I replace the hand features with the representative keypoint-keypoint vector angles.
\nAnyway, definitely following, thanks for sharing !
", "votes": 0, "replies": [] }, { "id": 2176238, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-10T13:27:31.560000", "content": "it is just simple mean and std normalization.
\nthere is probably not enough resource (100 msec) to do complicated normalisation
fig, ax = plt.subplots()\n ax.invert_yaxis()\n\n kaggle_df = pd.read_csv(f'{root_dir}/data/asl-signs/train.ver01.csv')\n for t,d in kaggle_df.iterrows():\n print('\\r', f'{t}/{len(kaggle_df)}', end='', flush=True)\n pq_file = f'{root_dir}/data/asl-signs/{d.path}'\n xyz = load_relevant_data_subset(pq_file)\n\n\n pose = xyz[:,POSE]\n pose = pose.reshape(-1,3)\n\n non_nan = np.all(~np.isnan(pose), axis=-1)\n pose = pose[non_nan]\n\n pose = pose-pose.mean(0,keepdims=True) \n pose = pose/pose.std(0,keepdims=True)\n\n ax.scatter(pose[:,0], pose[:,1], alpha=0.01)\n\n #plt.show()\n plt.waitforbuttonpress()\nAwesome. Makes sense!
", "votes": 0, "replies": [] }, { "id": 2177309, "author_name": "", "author_url": "", "post_date": "2023-03-11T11:10:47.910000", "content": "", "votes": 0, "replies": [] }, { "id": 2179776, "author_name": "Vladimir Bogachev", "author_url": "", "post_date": "2023-03-13T11:55:35.670000", "content": "Hi, I'm having trouble reproducing your results. I'm using one-layer transformer with participants 26734, 28656, 16069 as validation, and my top1 val caps at 0.35 using the following hyperparams:
\nmost likely there is some bug in your training code
\nthe training loss is shown at:
\nhttps://www.kaggle.com/competitions/asl-signs/discussion/391265#2170328
possible bug ares:
\nIt looks like I was normalizing each landmark separately and that's why it didn't work. Thanks for your help!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2240056, "author_name": "Bob_Fan91", "author_url": "", "post_date": "2023-04-30T07:22:23.977000", "content": "Excellent sharing！I learned a lot from this.Thank you very much!Wish you can get the top prize!
", "votes": 1, "replies": [] }, { "id": 2224702, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-17T15:03:30.517000", "content": "https://github.com/facebookresearch/dropout
\nDropout Reduces Underfitting
late dropout works for me
", "votes": 1, "replies": [] }, { "id": 2224196, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-17T04:45:42.443000", "content": "replace the word \"academics\" with \"kaggle\"
\n
i wonder if (inverse) DTW can be used as augmentation?
\nhttps://ai.googleblog.com/2021/01/recognizing-pose-similarity-in-images.html
Hi, I've noticed that you use xyz = xyz - xyz[~torch.isnan(xyz)].mean(0,keepdim=True) to normalize the input. However, xyz[~torch.isnan(xyz)] (or np.isnan) returns a flattened tensor of size (num_frames * num_landmarks * 3 - num_nans), which means that calling mean will return a scalar. In other words, what you're really doing is subtracting the mean of everything in the sequence instead of normalizing each landmark across frames (passing dim=0 and keepdim=True). Am I getting it correctly?
Nevertheless, I believe the correct mean normalization here is to subtract the mean of all landmark positions for each frame by doing something like xyz -= xyz.nanmean(dim=1, keepdim=True). It might be even better to normalize x, y, and z separately but I haven't figured out how to do it yet.
do a few print statement:
\nprint(xyz.shape)\nprint((torch.isnan(xyz)).shape)\nprint((xyz[~torch.isnan(xyz)]).shape)\nprint((xyz[~torch.isnan(xyz)].mean(0,keepdim=True)).shape)\nHere are the sizes of everything, which are literally the same as what I said above. Was it your original intention to normalize by the single mean of all landmarks in the sequence? Just curious about this as dim=0, keepdim=True means the opposite.
\n
\n
Glad to help. I'm not sure how to do the same for unit variance as there is no such thing as nanstd. Please share if you know.
i make the mistake becuase of
\nhttps://pytorch.org/docs/stable/generated/torch.nanmean.html
quote; \"(torch.nanmean(a) is equivalent to torch.mean(a[~a.isnan()])).\"
\ni suggest you use np for preparaing pytorch train dataset.
\nfor tflite, you will be using tf functions, so it is not a problem
\n(tf.experimental.numpy.std)
sorry for the confusion. the original implementation is \"almost correct\"
\n
I wonder if it's better to normalize per frame, though. I'll test that out.
\nSince you're normalizing everything at once, I suggest you can just delete the arguments for mean as they have no effects on a single scalar and only create confusion.
\n[Edit] Wait, what do you mean by \"almost correct\"? Can you clarify what kind of normalization are we using here?
if your xyz is already zero mean, then the average distance of points from center(0,0,0) should be
\nvar = np.nanstd(xyz.reshape(-1,3))**2 # mean of x**2, y**2, zz*2\ndistance_sq = var.sum()\ndistance = distance_sq**0.5\nthis should be the normalising factor
", "votes": 2, "replies": [] } ] }, { "id": 2206355, "author_name": "Denys Shushpanov", "author_url": "", "post_date": "2023-04-02T13:54:54.773000", "content": "By normalising x, y, z separately you meant sth like this?
\nxyz.nanmean(dim=(0, 1), keepdim=True)'
It results in normalisation of each of dimensions separately, so that mean and std for x, y, and z are the same (notice that means are around 0):
\n
Yes, but I also wanted to normalize the spatial coordinates for each frame, which means to only reduce dim=1. But it's just my intuition and I'm not sure if it will benefit the model. You can check out hengck23's replies above.
\nWe're getting the means close to 0 and the stds close to 1 as the original data are already normalized. This step is just for those artifacts that are outside of [0,1].
", "votes": 0, "replies": [] } ] } ] }, { "id": 2197663, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-26T09:48:45.067000", "content": "[previous results]
\n15-mar
\n
03-apr
\n
i just realise my work flow is:
\n1.click edit notebook
\n2.click open data in new tab
3.click submit
\ni wish to hvae one button just update dataset directly from the notebook page.
\nno need all the \"click check for update\"
I wish you could just submit the model.tflite directly like csv submissions. But it's handy enough😗
\nI like this new kind of kaggle competitions focused on edge computing. Kudos to the kaggle team making it happen!
https://www.kaggle.com/discussions/general/395353
\nmaybe one day we can have a chatgpt AI bot to help us with that:
\nprompt 'upload new tflitemodel and submit'
and i think this is useful
\nprompt 'check license issue for this external dataset at www.xxx.xxx. if there is an issue, send email to moderator and ask'
did another try mirror augmentation?
\nif yes, can i have your feedbacks?
I've tried mirror augmentation with a simple MLP model, but didn't get better results for now, I'm still stuck at 0.63LB with and without this augmentation.
\nI'll experiment further this weekend and I'm interested in other feedbacks on this topic as I supposed it would mitigate the right-handed/left-handed issue.
How do you implement the mirror augmentation?
", "votes": 0, "replies": [] }, { "id": 2210891, "author_name": "Muai", "author_url": "", "post_date": "2023-04-05T17:13:06.600000", "content": "I simply multiply the x coordinates by -1 after normalization
", "votes": 0, "replies": [] } ] } ] }, { "id": 2176454, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-10T17:03:18.613000", "content": "[pretrain model]
\nhttps://github.com/AI4Bharat/OpenHands
\npretrain model based on mediapipe !!!!
this is one thing what i want to do:
\ngenerative pretraining
\n[1]SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
\nhttps://arxiv.org/pdf/2110.05382.pdf
It is interesting! Do you have some thoughts about creating baseline using that pretrained model?
", "votes": 0, "replies": [ { "id": 2178093, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-12T05:19:08.527000", "content": "related: tokenization as pretraining
\n
BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization
\nhttps://arxiv.org/pdf/2302.05075.pdf
diffusion-based autoencoder (for augmentation?)
\nVector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation
\nhttps://arxiv.org/pdf/2208.09141.pdf
It's too large architecture, I think.
", "votes": 0, "replies": [] }, { "id": 2178214, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-12T08:16:53.277000", "content": "you can distill later
", "votes": 1, "replies": [] }, { "id": 2181076, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-14T09:48:18.840000", "content": "in theory, the max numbers of hand pattern is fixed(e.g. limited by rotation angle of finger joints, etc). hand you can actually build a \"tokenizer\" (aka lookup table)
\nit is interesting to make a tsne plot of the the hands
", "votes": 1, "replies": [] } ] } ] }, { "id": 2165058, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-02T00:46:44.770000", "content": "[framework conversion]
\nkeras, pytorch:
\nhttps://github.com/leondgarse/keras_cv_attention_models
pytorch, onnx, tf
\nhttps://github.com/PINTO0309/onnx2tf
i check some of the confusing words …. i wonder if a second classifier for specific pairs (or groups) of confusing words would help (e.g. using more features, number of frames, etc)
\n167 pen (99)\n 43, 0.43434, 167 pen \n 28, 0.28283, 168 pencil \n 9, 0.09091, 40 chair \n 7, 0.07071, 176 potty \n 2, 0.02020, 0 TV \n\n168 pencil (96)\n 66, 0.68750, 168 pencil \n 15, 0.15625, 167 pen \n 2, 0.02083, 6 another \n 2, 0.02083, 176 potty \n\n129 kitty (103)\n 77, 0.74757, 129 kitty \n 10, 0.09709, 38 cat \n 3, 0.02913, 19 bee \n 3, 0.02913, 173 please \n\n\n11 awake (96)\n 46, 0.47917, 11 awake \n 34, 0.35417, 233 wake \n 7, 0.07292, 131 later \nhttps://arxiv.org/pdf/2303.12080.pdf
\nNatural Language-Assisted Sign Language Recognition
you can see that the confused words has similary meaning (e.g. kitty/cat, pencil/pen)
\ncheck the paper on how to improve on this.
\n(this may the the reason why label smooth work for me, but according to the paper word-based label smooth works better)
Can I ask the simple question, which classes you didn't apply flip-agumentations. Thankss
", "votes": 0, "replies": [] } ] } ] }, { "id": 2211075, "author_name": "hengck23", "author_url": "", "post_date": "2023-04-05T19:24:21.707000", "content": "distangle hand shape and hand motion:
\nPhonologically-meaningful Subunits for Deep Learning-based Sign Language Recognition
\nhttps://www.slrtp.com/papers/full_papers/SLRTP.FP.02.012.paper.pdf
motion token?
\n\"The large majority of sign language recognition systems based on deep learning adopt a word model approach. Here we present a system that works with subunits, rather than word models. We propose a pipelined approach to deep learning that uses a factorisation algorithm to derive hand motion features, embedded within a low-rank trajectory space.\"
\n\"To separate the real hand motions from the camera and whole body movements of the signer, we propose to use a non-rigid structure from motion (NRSfM) technique based on the factorisation method [54].\"
", "votes": 2, "replies": [] }, { "id": 2200211, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-28T11:23:42.713000", "content": "i probably got it all wrong.
\nthe correct way to shape normalise and do feature extraction is given by:
https://google.github.io/mediapipe/solutions/pose_classification.html
\nhttps://www.tensorflow.org/lite/tutorials/pose_classification
Next, convert the landmark coordinates to a feature vector by:\n\n- Moving the pose center to the origin.\n- Scaling the pose so that the pose size becomes 1\n- Flattening these coordinates into a feature vector\nI am worried about \"scaling the pose so that the pose size becomes 1\". Let's say we are focused on the hand skeleton. If we have flat palm and fist - both of them would be scaled to the size of 1. Would it be beneficial for the model? I guess without per sample scaling it would be possible to extract additional information right from a scale of a given sign. What do you think about it?
", "votes": 0, "replies": [ { "id": 2202120, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-29T18:46:10.453000", "content": "\"i probably got it all wrong.\"
\nbecuase if your feature is pairwise distance, then your normalisation is also pairwise distance.
\nobjective of normalisation is to make feature values the same for similar class
Hi, my pairwise distance calculation code reduces training speed dramatically from 20 batches/s to 4 batches/s. Do you precalculate pairwise distances or do you calculate it live? Do you have similar speed reduction?
", "votes": 0, "replies": [] }, { "id": 2205319, "author_name": "Mykola", "author_url": "", "post_date": "2023-04-01T12:32:13.977000", "content": "If you are using tf.data.Dataset I would recommend you to use .cache() after feature calculation. It will precalculate all features and cache them. All other epochs would use cache instead of calculation all from scatch
", "votes": 0, "replies": [] } ] } ] }, { "id": 2189046, "author_name": "", "author_url": "", "post_date": "2023-03-20T07:37:41.097000", "content": "", "votes": 2, "replies": [ { "id": 2193611, "author_name": "", "author_url": "", "post_date": "2023-03-23T11:37:21.013000", "content": "", "votes": 0, "replies": [] }, { "id": 2195003, "author_name": "", "author_url": "", "post_date": "2023-03-24T10:34:29.210000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2178024, "author_name": "", "author_url": "", "post_date": "2023-03-12T02:33:03.533000", "content": "", "votes": 2, "replies": [] }, { "id": 2163489, "author_name": "", "author_url": "", "post_date": "2023-02-28T21:14:11.653000", "content": "", "votes": 2, "replies": [] }, { "id": 2230043, "author_name": "", "author_url": "", "post_date": "2023-04-22T00:40:49.027000", "content": "", "votes": 0, "replies": [] }, { "id": 2229823, "author_name": "", "author_url": "", "post_date": "2023-04-21T18:14:11.380000", "content": "", "votes": 0, "replies": [] }, { "id": 2222116, "author_name": "", "author_url": "", "post_date": "2023-04-14T22:59:25.597000", "content": "", "votes": 0, "replies": [] }, { "id": 2211093, "author_name": "", "author_url": "", "post_date": "2023-04-05T19:39:46.310000", "content": "", "votes": 0, "replies": [] }, { "id": 2209841, "author_name": "", "author_url": "", "post_date": "2023-04-05T01:01:04.810000", "content": "", "votes": 0, "replies": [ { "id": 2216136, "author_name": "", "author_url": "", "post_date": "2023-04-09T19:52:14.493000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2209724, "author_name": "", "author_url": "", "post_date": "2023-04-04T21:08:42.600000", "content": "", "votes": 0, "replies": [] }, { "id": 2209702, "author_name": "", "author_url": "", "post_date": "2023-04-04T20:46:35.190000", "content": "", "votes": 0, "replies": [] }, { "id": 2204403, "author_name": "", "author_url": "", "post_date": "2023-03-31T14:34:11.150000", "content": "", "votes": 0, "replies": [ { "id": 2204423, "author_name": "", "author_url": "", "post_date": "2023-03-31T14:59:33.130000", "content": "", "votes": 1, "replies": [ { "id": 2204436, "author_name": "", "author_url": "", "post_date": "2023-03-31T15:03:47.210000", "content": "", "votes": 3, "replies": [] }, { "id": 2204457, "author_name": "", "author_url": "", "post_date": "2023-03-31T15:28:02.627000", "content": "", "votes": 0, "replies": [] }, { "id": 2204464, "author_name": "", "author_url": "", "post_date": "2023-03-31T15:32:47.537000", "content": "", "votes": 0, "replies": [] }, { "id": 2205038, "author_name": "", "author_url": "", "post_date": "2023-04-01T07:36:22.143000", "content": "", "votes": 1, "replies": [] } ] } ] }, { "id": 2203287, "author_name": "", "author_url": "", "post_date": "2023-03-30T17:05:00.597000", "content": "", "votes": 0, "replies": [ { "id": 2203309, "author_name": "", "author_url": "", "post_date": "2023-03-30T17:33:06.200000", "content": "", "votes": 0, "replies": [ { "id": 2205331, "author_name": "", "author_url": "", "post_date": "2023-04-01T12:38:25.987000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2202934, "author_name": "", "author_url": "", "post_date": "2023-03-30T12:07:27.730000", "content": "", "votes": 0, "replies": [ { "id": 2202945, "author_name": "", "author_url": "", "post_date": "2023-03-30T12:15:06.343000", "content": "", "votes": 2, "replies": [ { "id": 2202963, "author_name": "", "author_url": "", "post_date": "2023-03-30T12:25:20.450000", "content": "", "votes": 0, "replies": [] }, { "id": 2202972, "author_name": "", "author_url": "", "post_date": "2023-03-30T12:33:43.527000", "content": "", "votes": 1, "replies": [] }, { "id": 2202977, "author_name": "", "author_url": "", "post_date": "2023-03-30T12:40:41.453000", "content": "", "votes": 0, "replies": [] }, { "id": 2202980, "author_name": "", "author_url": "", "post_date": "2023-03-30T12:43:23.397000", "content": "", "votes": 0, "replies": [] }, { "id": 2203191, "author_name": "", "author_url": "", "post_date": "2023-03-30T15:44:17.403000", "content": "", "votes": 0, "replies": [] }, { "id": 2203306, "author_name": "", "author_url": "", "post_date": "2023-03-30T17:28:17.947000", "content": "", "votes": 0, "replies": [] }, { "id": 2205366, "author_name": "", "author_url": "", "post_date": "2023-04-01T13:35:07.010000", "content": "", "votes": 0, "replies": [] }, { "id": 2205487, "author_name": "", "author_url": "", "post_date": "2023-04-01T15:54:15.407000", "content": "", "votes": 1, "replies": [] } ] } ] }, { "id": 2202855, "author_name": "", "author_url": "", "post_date": "2023-03-30T11:24:00.570000", "content": "", "votes": 0, "replies": [] }, { "id": 2202383, "author_name": "", "author_url": "", "post_date": "2023-03-30T02:40:57.767000", "content": "", "votes": 0, "replies": [] }, { "id": 2194498, "author_name": "", "author_url": "", "post_date": "2023-03-24T01:46:16.067000", "content": "", "votes": 0, "replies": [] }, { "id": 2189929, "author_name": "", "author_url": "", "post_date": "2023-03-20T22:31:03.683000", "content": "", "votes": 0, "replies": [] }, { "id": 2189299, "author_name": "", "author_url": "", "post_date": "2023-03-20T11:43:27.953000", "content": "", "votes": 0, "replies": [] }, { "id": 2188739, "author_name": "", "author_url": "", "post_date": "2023-03-20T00:06:49.330000", "content": "", "votes": 0, "replies": [ { "id": 2188788, "author_name": "", "author_url": "", "post_date": "2023-03-20T01:40:20.897000", "content": "", "votes": 1, "replies": [] }, { "id": 2188797, "author_name": "", "author_url": "", "post_date": "2023-03-20T01:59:21.083000", "content": "", "votes": 1, "replies": [ { "id": 2188822, "author_name": "", "author_url": "", "post_date": "2023-03-20T02:50:25.633000", "content": "", "votes": 0, "replies": [] }, { "id": 2189403, "author_name": "", "author_url": "", "post_date": "2023-03-20T13:19:55.457000", "content": "", "votes": 1, "replies": [] } ] } ] }, { "id": 2188724, "author_name": "", "author_url": "", "post_date": "2023-03-19T23:16:49.727000", "content": "", "votes": 0, "replies": [] }, { "id": 2187997, "author_name": "", "author_url": "", "post_date": "2023-03-19T08:22:03.700000", "content": "", "votes": 0, "replies": [ { "id": 2188014, "author_name": "", "author_url": "", "post_date": "2023-03-19T08:29:31.507000", "content": "", "votes": 1, "replies": [ { "id": 2188043, "author_name": "", "author_url": "", "post_date": "2023-03-19T08:50:39.050000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2183926, "author_name": "", "author_url": "", "post_date": "2023-03-16T03:29:44.980000", "content": "", "votes": 0, "replies": [ { "id": 2183941, "author_name": "", "author_url": "", "post_date": "2023-03-16T03:51:01.303000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2183228, "author_name": "", "author_url": "", "post_date": "2023-03-15T14:37:10.477000", "content": "", "votes": 0, "replies": [ { "id": 2183236, "author_name": "", "author_url": "", "post_date": "2023-03-15T14:41:51.207000", "content": "", "votes": 0, "replies": [ { "id": 2183317, "author_name": "", "author_url": "", "post_date": "2023-03-15T15:26:18.740000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2182925, "author_name": "", "author_url": "", "post_date": "2023-03-15T11:41:14.323000", "content": "", "votes": 0, "replies": [ { "id": 2207158, "author_name": "", "author_url": "", "post_date": "2023-04-03T06:34:50.307000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2182591, "author_name": "", "author_url": "", "post_date": "2023-03-15T07:36:04.037000", "content": "", "votes": 0, "replies": [ { "id": 2183260, "author_name": "", "author_url": "", "post_date": "2023-03-15T14:58:31.307000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2167655, "author_name": "", "author_url": "", "post_date": "2023-03-03T16:25:25.970000", "content": "", "votes": 0, "replies": [ { "id": 2168595, "author_name": "", "author_url": "", "post_date": "2023-03-04T11:07:50.940000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2187722, "author_name": "", "author_url": "", "post_date": "2023-03-19T01:56:45.423000", "content": "", "votes": -5, "replies": [ { "id": 2187785, "author_name": "", "author_url": "", "post_date": "2023-03-19T03:45:59.167000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 2179510, "author_name": "", "author_url": "", "post_date": "2023-03-13T08:14:48.807000", "content": "", "votes": 0, "replies": [] }, { "id": 2206052, "author_name": "", "author_url": "", "post_date": "2023-04-02T08:05:45.377000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2163485": "experiments in progress ...\n\n\n\n---\n\nprevious results:\nhttps://www.kaggle.com/competitions/asl-signs/discussion/391265#2197663\n\n\n---\n\nafter some experiments for the first week, here is the master plan:\n\n[1] one (or two) layer transformer is sufficient. for one fold, memory is only 2mb ( tf.lite.Optimize.DEFAULT, i.e. int8 in storage) and 25min for all 40\\_000 test videos (or 30 msec per video). Future because of missing frame, long term attention has its advantage\n\nin fact two layer transformer would lead to over fitting.\n\n[2] next step is to spend more time on data representation, especially normalization\n\n[3] use of external data, maybe need to use mediapipe to extract landmarks on external ASL video. \nOther ways to use increase data increases, adversarial/diffusion augmentation, masked token prediction, etc\n\nI feel that SSL with masked token prediction (e.g. generative pretraining) fits this problem well as there are high proportion of missing frames.\n\n[4], There no much of algorithm design because network is quite shallow. if there is, it would be more on increasing parameters for accuracy and speeds. There are many VIT tricks that can be used here, e.g. window attention, poolforming, convolutional positional encoding, grouped conv, ...\n\n\n----\nFirst results \n- one fold, max video length=60 (crop first) : LB 0.65 \n- 1 fold, max video length=512 (crop center) : LB 0.65 25 min\n- 4 fold, max video length=512 (crop center) : LB 0.68 32 min\n- 5 fold, max video length=512 (crop center) : LB (better) 0.68 35 min\n\nCV one fold using train/validation random split: cv 0.72~0.74 (probably the limit if good generalisation)\nCV one fold using train/validation participant non-overlap split: cv 0.61 (compared with above, show the severity of lack of train data) \n\ninference:\nhttps://www.kaggle.com/code/hengck23/lb-0-65-one-pytorch-transformer-solution\n\ntraining code:\nhttps://www.kaggle.com/datasets/hengck23/asl-demo\n\n\n", "2179326": "i have been playing with controlnet and i come across this:\n\n\nmaybe can make some fun animation video while waiting for training", "2201909": "flip augmentation magic:\nCV 0.703 (single fold)\nLB 0.72\n\n```\nLHAND = np.arange(468, 489).tolist() # 21\nRHAND = np.arange(522, 543).tolist() # 21\nPOSE = np.arange(489, 522).tolist() # 33\nFACE = np.arange(0,468).tolist() #468\n\nREYE = [\n\t33, 7, 163, 144, 145, 153, 154, 155, 133,\n\t246, 161, 160, 159, 158, 157, 173,\n]\nLEYE = [\n\t263, 249, 390, 373, 374, 380, 381, 382, 362,\n\t466, 388, 387, 386, 385, 384, 398,\n]\nNOSE=[\n\t1,2,98,327\n]\nSLIP = [\n\t78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n\t191, 80, 81, 82, 13, 312, 311, 310, 415,\n]\nSPOSE = (np.array([\n\t11,13,15,12,14,16,23,24,\n])+489).tolist()\n\n\n...\n## assume zero mean xyz. so flip can be implement by multuiplication of -1\ndef do_hflip_hand(lhand, rhand):\n\trhand[...,0] *= -1\n\tlhand[...,0] *= -1\n\trhand, lhand = lhand,rhand\n\treturn lhand, rhand\n\ndef do_hflip_spose(spose):\n\tspose[...,0] *= -1\n\tspose = spose[:,[3,4,5,0,1,2,7,6]]\n\treturn spose\n\ndef do_hflip_slip(slip):\n\tslip[...,0] *= -1\n\tslip = slip[:,[10,9,8,7,6,5,4,3,2,1,0]+[19,18,17,16,15,14,13,12,11]]\n\treturn slip\n\n...\n...\n\t\txyz = load_relevant_data_subset(pq_file)\n\t\tlhand = xyz[:,LHAND]\n\t\trhand = xyz[:,RHAND]\n\t\tspose = xyz[:,SPOSE]\n\t\tleye = xyz[:,LEYE]\n\t\treye = xyz[:,REYE]\n\t\tslip = xyz[:,SLIP]\n\t\tnose = xyz[:,NOSE]\n\n\n\tif is_aug==1:\n\t\tif np.random.rand()<0.5:\n\t\t\tlhand, rhand = do_hflip_hand(lhand, rhand)\n\t\t\tspose = do_hflip_spose(spose)\n\t\t\tleye, reye = do_hflip_eye(leye, reye)\n\t\t\tslip = do_hflip_slip(slip)\n\t\t\tnose = do_hflip_nose(nose)\n\n```\n\ntrain log:\n\n```\nfold_type = kaggle-part\nfold = 0\ntrain_dataset : \n\tlen = 71518\n\tnum_participant_id = 16\n\nvalid_dataset : \n\tlen = 22959\n\tnum_participant_id = 5\n...\n batch_size = 64 \n experiment = ['tx025-no-z-flip', 'run_train_fold0.py']\n |---------------- VALID---------|---- TRAIN/BATCH ----------------------\nrate iter epoch | loss top1 top2 top5 | loss | time \n---------------------------------------------------------------------------------------------------\n...\n1.00e-4 00050265* 45.00 | 2.025 0.698 0.7992 0.870 | 3.887 0.000 0.000 | 1 hr 48 min\n1.00e-4 00053616* 48.00 | 2.042 0.697 0.7969 0.870 | 3.879 0.000 0.000 | 1 hr 56 min\n\n```\n\n\n", "2193356": "I love your experiments records table! thanks for sharing this!", "2178946": "[unorganized] some proven intermediate results ... will write in more detail later\n\n- add motion (xyz at current frame - xyz at previous frame). i.e. each point now has 6 dim. need to handle NaN motion correctly\nimprove CV by 0.015\n\n\n- add framerate interpolation augmentation scipy.interpolate.interp1d()\nimprove CV by 0.015\n", "2228313": "watch this thread!\n\nreceipe for LB0.76 one fold:\n\n---\n\ni seems that there is somthing very strange about the dataset ... maybe very noisy. i have to use unconventional strong regularisation and training method ....\n \n\nhint: \n[1] Dropout Reduces Underfitting\nhttps://arxiv.org/abs/2303.01500\n\ni find that if i reduce learning rate, the model easily overfits. instead, according to paper[1], by setting dropout to different values at different training epoches, we can reduce validation loss (it has the same effects as reducing learning rate) without overfitting:\n- early: train without dropout\n- late: train with large dropout\n- final: tune with small dropout\n\n\n\n\n\n```\nfold_type = kaggle-random (20% validation, 80% train. pure random)\nfold = 0\nvalid_dataset : \n\tlen = 18896\n\tnum_participant_id = 21\n\tnum_external_video = 0\n\n\ntime_taken = 0 min 27 sec\nloss = 1.7932500839233398\ntopk[0] = 0.8607112616426758 (LB 0.76)\ntopk[1] = 0.9180249788314987\ntopk[2] = 0.9374470787468248\ntopk[3] = 0.9457557154953429\ntopk[4] = 0.9513653683319221\ncheckpoint_file = /home/titanx/hengck/share1/kaggle/2022/hand-sign/result/run202/tx90-face-norm-480-0b/fold-0-kaggle-random/checkpoint/swa.model.pth\n\n```", "2163630": "[reference]\n- 'Sign Pose-based Transformer for Word-level Sign Language Recognition' - Matyáš Boháček\n- Combining Efficient and Precise Sign Language Recognition: Good pose estimation library is all you need- Matyáš Boháček\n\nhttps://github.com/matyasbohacek/spoter\nhttps://www.matyasbohacek.com/\n(uses MediaPipe library pose too)\n\nhttps://huggingface.co/spaces/matyasbohacek/spoter-demo-test\n\nhttps://www.youtube.com/watch?v=6i5ODpHMny8\n\n\n\n\n---\nSLGTFORMER: AN ATTENTION-BASED APPROACH TO SIGN LANGUAGE RECOGNITION\nhttps://github.com/neilsong/SLGTformer\n", "2163638": "[external data]\n... to be updated ...\n\nHOW TO SIGN\nhttps://how2sign.github.io/\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Duarte_How2Sign_A_Large-Scale_Multimodal_Dataset_for_Continuous_American_Sign_Language_CVPR_2021_paper.pdf\n(vocab=16k, time=79 hr, signer=11)\n\nMS-ASL American Sign Language Dataset\nhttps://microsoft.github.io/data-for-society/dataset?d=MS-ASL-American-Sign-Language-Dataset\nhttps://arxiv.org/pdf/1812.01053.pdf\n(vocab=1000, num video=25513, signer=222)\n\nBSL-1K: Scaling up co-articulated sign language recognition using mouthing cues\nhttps://www.robots.ox.ac.uk/~vgg/research/bsl1k/\n(vocab=1000, num video=1000)\n\n\nChaLearn LAP Large Scale Signer Independent Isolated Sign Language\nRecognition Challenge: Design, Results and Future Research\n\nASL-Skeleton3D and ASL-Phono: Two Novel\nDatasets for the American Sign Language\n\n---\n\n- how to read media pipline to extra keypoint for external data \n... to be udpated ...\n\n- more about PopSign learning app\nhttps://devpost.com/software/pop-sign-learning\nhttps://www.youtube.com/watch?v=nbQAfiJzp2g&t=430s\nhttps://github.com/Accessible-Technology-in-Sign/PopSign\n\nhttps://github.com/Accessible-Technology-in-Sign/ASLRT\n\"A pipeline to run tracking Kinect, MediaPipe and AlphaPose experiments on sign language feature data using either HMMs, SBHMMs, or Transformers.\"\n\ncollect data you from youtube?\n\n\n\n\n\n\n\n", "2226810": "i forget an import detail at the data card. suggestions for augmentation:\n\n\"While the app provided a video example of the sign desired, signers routinely made variants of the sign based on their background and region.\" \n- you need external data or create your variant\n\n\"More rarely, signers might fingerspell a sign, miss it completely, or produce the wrong sign.\" \n- need to measure label noise and decide if you want to exclude some data\n\n\" Extraneous movements, such as scratching an itch, or the ending movement from the previous sign or the onset of the next sign, are sometimes included.\" \n- random add hand (from other video at start and end). actually CAM heat map is about to hightlight the useful frame.\n\n\" Conversely, some signers pressed the button late or released the button early, causing cropping in some sign examples. \" \n- cut and paste (mix up)\n\n\"Some signers sign with their left hand; others sign with their right. Some signers switch their signing hand. All of these situations must be handled by the game’s recognition system.\" \n- flip\n\n\n---\n\na also note that some signer make repeated signs in one video\n\n", "2224078": "it is suprised for me to achieve 0.75 in one fold with participant split train data(num participant = 5 in validation data). This makes me think that 0.76 is possible. \n\n```\n\nfold_type = kaggle-part\nfold = 0\nvalid_dataset : \n\tlen = 22959\n\tnum_participant_id = 5 \n \n 22959 / 22959 0 min 42 sec\n\ntime_taken = 0 min 42 sec\nloss = 2.406057596206665\ntopk[0] = 0.7277320440785748\ntopk[1] = 0.8219870203406072\ntopk[2] = 0.858530423798946\ntopk[3] = 0.8757785617840498\ntopk[4] = 0.8888888888888888\ncheckpoint_file = /home/titanx/hengck/share1/kaggle/2022/hand-sign/result/run201/tx61a-1-frame-first-scale-2/fold-0-kaggle-part/checkpoint/swa.model.pth\n \n\n```\n\nand i hit the jackpot 888888888", "2206909": "a couple on conv1d (as spatial temporal encoder) + MHA attention seems to have good results.\nsome good ideas to design efficient transformer.\n\n[1] FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization\nhttps://arxiv.org/pdf/2303.14189.pdf\n\n[2] Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios\nhttps://arxiv.org/abs/2207.05501\n\nto avoid uncessary permute reshape, i can write my q,v,k linear operation in terms of conv1d.\ni will be removing absolute position embedding and use convolutional position embedding instead (convolutional FFN, i.e. position information is implicitly embedding on 1d conv)\n\n\nactually it is aslo possible to use 2d convolution, where (W,H) = (x,y,z of spatial, t of temporal).", "2204321": "this is how much the face (upper triangle = eyes and lip) and body(lower quadliteral = shoulders and hips) have moved for one hand sign.\n\nI wonder if there will be improvement in accuracy if i remove these noise motion?\n\n", "2206111": "simplest normalisation code\n\n```\n\nREF = [500, 501, 512, 513, 159, 386, 13,]\n\ndef do_normalise_by_ref(xyz, ref): \n\tK = xyz.shape[-1]\n\txyz_flat = ref.reshape(-1,K)\n\tm = np.nanmean(xyz_flat,0).reshape(1,1,K)\n\ts = np.nanstd(xyz_flat, 0).mean() \n\txyz = xyz - m\n\txyz = xyz / s\n\treturn xyz\n\nxyz = load_relevant_data_subset(pq_file)[...,:2]\nxyz = do_normalise_by_ref(xyz,xyz[:,REF])\n\n```\n", "2178029": "[self-study]\n- Sign Language Translation with Transformers\nhttps://www.youtube.com/watch?v=E5nKeEvoAK0\n\n", "2163502": "[ensemble trick]\n- since only one model can be submitted and there is a time limit, we can distilled \"ensemble knowledge\" into one model.\n- a better solution is distilled encoder (or multiple encoders) and multiple heads. heads are using different \"view\" of encoded features, final prediction is averaged over the heads\n \n\nmore on this later", "2163494": "[augmentation and tricks]\n- simulate different frame length \n- simulate different NaN (as in train data)\n- simulate different missing frame or differnt frame rate? \n- 3d guassian noise\n- type of landmark. use ['face', 'left_hand', 'pose', 'right_hand'] for normalisation and augmentation\n (e.g. affine transform to group of point --> bigger mouth, longer arm ..., rotate hand)\n\nsince landmark is xyz, we are having a 3d motion capture. would be fun to use GAN to create more data. distangle motion and shape\n\npossible to synthetic animate motion for new person via motion transfer. But we need to find some media pipe human xyz models. ", "2170203": "[normalization]\n\nnormalization is all you need!\n```\n\n\ndef pre_process(xyz):\n\tlip = xyz[:, LIP]\n\tlhand = xyz[:, LHAND]\n\trhand = xyz[:, RHAND]\n\txyz = torch.cat([ #(none, 82, 3)\n\t\tlip,\n\t\tlhand,\n\t\trhand,\n\t],1)\n\txyz = xyz - xyz[~torch.isnan(xyz)].mean(0,keepdims=True) #noramlisation to common maen\n\txyz = xyz / xyz[~torch.isnan(xyz)].std(0, keepdims=True)\n\txyz[torch.isnan(xyz)] = 0\n\txyz = xyz[:max_length]\n\treturn xyz\n``` ", "2240056": "Excellent sharing！I learned a lot from this.Thank you very much!Wish you can get the top prize!", "2224702": "https://github.com/facebookresearch/dropout\nDropout Reduces Underfitting\n\n\n\nlate dropout works for me", "2224196": "replace the word \"academics\" with \"kaggle\"\n\n", "2210622": "i wonder if (inverse) DTW can be used as augmentation?\nhttps://ai.googleblog.com/2021/01/recognizing-pose-similarity-in-images.html", "2204573": "Hi, I've noticed that you use `xyz = xyz - xyz[~torch.isnan(xyz)].mean(0,keepdim=True)` to normalize the input. However, `xyz[~torch.isnan(xyz)]` (or `np.isnan`) returns a *flattened* tensor of size `(num_frames * num_landmarks * 3 - num_nans)`, which means that calling `mean` will return a scalar. In other words, what you're really doing is subtracting the mean of everything in the sequence instead of normalizing each landmark **across frames** (passing `dim=0` and `keepdim=True`). Am I getting it correctly?\n\nNevertheless, I believe the correct mean normalization here is to subtract the mean of all landmark positions **for each frame** by doing something like `xyz -= xyz.nanmean(dim=1, keepdim=True)`. It might be even better to normalize x, y, and z separately but I haven't figured out how to do it yet.", "2197663": "[previous results]\n\n15-mar\n\n\n03-apr\n\n", "2185116": "i just realise my work flow is:\n\n1.click edit notebook \n2.click open data in new tab \n - upload new tflite file and wait \n - back to notebook page click check for update \n \n3.click submit \n\ni wish to hvae one button just update dataset directly from the notebook page.\nno need all the \"click check for update\" ", "2182036": "did another try mirror augmentation?\nif yes, can i have your feedbacks?", "2176454": "[pretrain model]\n\nhttps://github.com/AI4Bharat/OpenHands\npretrain model based on mediapipe !!!!", "2165058": "[framework conversion]\n\n- keras, pytorch:\nhttps://github.com/leondgarse/keras_cv_attention_models\n\n- pytorch, onnx, tf\nhttps://github.com/PINTO0309/onnx2tf", "2229717": "i check some of the confusing words .... i wonder if a second classifier for specific pairs (or groups) of confusing words would help (e.g. using more features, number of frames, etc)\n\n```\n\n167 pen (99)\n\t 43, 0.43434, 167 pen \n\t 28, 0.28283, 168 pencil \n\t 9, 0.09091, 40 chair \n\t 7, 0.07071, 176 potty \n\t 2, 0.02020, 0 TV \n\n168 pencil (96)\n\t 66, 0.68750, 168 pencil \n\t 15, 0.15625, 167 pen \n\t 2, 0.02083, 6 another \n\t 2, 0.02083, 176 potty \n\n129 kitty (103)\n\t 77, 0.74757, 129 kitty \n\t 10, 0.09709, 38 cat \n\t 3, 0.02913, 19 bee \n\t 3, 0.02913, 173 please \n\n\n11 awake (96)\n\t 46, 0.47917, 11 awake \n\t 34, 0.35417, 233 wake \n\t 7, 0.07292, 131 later \n```\n\n\n\n\n", "2211075": "distangle hand shape and hand motion:\nPhonologically-meaningful Subunits for Deep Learning-based Sign Language Recognition\nhttps://www.slrtp.com/papers/full_papers/SLRTP.FP.02.012.paper.pdf\n\nmotion token?\n\n\"The large majority of sign language recognition systems based on deep learning adopt a word model approach. Here we present a system that works with subunits, rather than word models. We propose a pipelined approach to deep learning that uses a factorisation algorithm to derive hand motion features, embedded within a low-rank trajectory space.\"\n\n\n\"To separate the real hand motions from the camera and whole body movements of the signer, we propose to use a non-rigid structure from motion (NRSfM) technique based on the factorisation method [54].\"\n\n", "2200211": "i probably got it all wrong.\nthe correct way to shape normalise and do feature extraction is given by:\n\nhttps://google.github.io/mediapipe/solutions/pose_classification.html\nhttps://www.tensorflow.org/lite/tutorials/pose_classification\n", "2189046": "we have an interesting results of LB0.71 with single fold using all train data", "2178024": "[opensource implementation]\n\n[0] ST-GCN \nhttps://github.com/yysijie/st-gcn\n\n[1] SL-GCN \nhttps://github.com/jackyjsy/SAM-SLR-v2\n\n[2] ECCV 2022 Sign Spotting Challenge 3rd ranked solution\nhttps://chalearnlap.cvc.uab.cat/media/results/None/top3-track1_H1KGRhQ.pdf \nhttps://github.com/ycmin95/Chalearn_2022_Sign_Spotting_MSSL_track/tree/master/dataset\n\ninclude how to use mediapipe to generate data\n", "2163489": "[tflite conversion]\n\n- https://github.com/onnx/onnx-tensorflow\n- fallback plan (aka write my own converter):\n - re-code in keras. need to perform one to one check for each nn.Module(), tf.keras.layers().\n - copy weights from pytorch to keras (trick is to give good names to layer for easier code)\n - keras to tflite", "2230043": "extra attribute for training (aux loss). if you have external training set, you can now use non kaggle gloss (word labels) for training too!\n\n\nTowards Zero-shot Sign Language Recognition\nhttps://arxiv.org/pdf/2201.05914.pdf\nhttps://bmvc2019.org/wp-content/uploads/papers/0122-paper.pdf\n\n\"We further annotate the datasets with high-level attributes that are gathered from American Sign Language Hand Shape Dictionary [87].\"\n\na page from American Sign Language Hand Shape Dictionary:\n\n\nso you have attribute like:\n- palm orientation (in, out, up, down, left, or right)\n- movement (up, down, left, right, inward, outward, circular, wrist movement, finger movement)\n- hand shape (A-hand, S-hand, 5-hand, etc.)\n\n\n\n\n", "2229823": "how do you get the point_dim 1414 in the lastest update?", "2222116": "there is another way (maybe better?) to approach the problem:\nsegmentation : \n- classify every frame (instead of one video), i.e. making frame features\n- you can combine several videos during training\n- if you want you can train an aggregator at the end to combine frame features into video features to do video prediction\n\n---\n\nan extension will be classify every N-frame intervals, etc to learn videolet features \n\n---\n\nyou can using classification to perform vector quantisation (VQ) ... kinda fake AE", "2211093": "joint angle as a feature:\nhttps://github.com/google/mediapipe/issues/2999\nhttps://github.com/TemugeB/joint_angles_calculate", "2209841": "Could you say which of participants are in fold 0 in your table?", "2209724": "how to augment:\n\nconceptually:\n\n\nneed to constraint the augmentation correctly.\nmore operations will be stretch shear\n\nyou can observe a few coordinate images of the same word of the same/different signers\n\n---\n\nactually one can perform classification on these coordinate images (i.e. treat it as an image problem) if you can render perform 2d convolution fast enough ... ", "2209702": "a possible way to select pairwise distance feature:\n\n\nSequential Attention for Feature Selection\nhttps://arxiv.org/abs/2209.14881", "2204403": "Thank you for sharing. Did you use entire dataset as training data? How can you check the accuracy of a model at each checkpoint when using all train samples?", "2203287": "I am getting out of memory error, even when taking emb_dim = 256, head = 4, transformer_layer = 1. any optimization i am missing out on?", "2202934": "May I ask are you still using 1-layer transformer? I can't believe that one attention pooling can achieve 73 single-fold score. \n\nOr maybe your deeper embedding layer (fc-norm-act-fc-norm-act) can do the same feature processing as more transformer layers?", "2202855": "Label smoothing and flip didn't work for me.", "2202383": "Hi @hengck23 , what is cls and what is all token mean pooling?", "2194498": "i find it easier to write your own attention.\nyou can control q,k,v dim differently etc.\nyou can control local attention, e.g. which query is attent to which values (according hand, lip parts or joints)\n\nyou can learned the affinity matrix of graph CGN if you treat affinity=attention weights", "2189929": "Many thanks for you work @hengck23.\nDo you ensemble your models or do you just take the best model through CV?\nWhen I ensemble, it seems that inference exceeds an hour...", "2189299": "can you do validation without validation data?\ne.g. if i used all the train images for training, how do i know if there is over fitting?\n\nin theory, yes ....\nyou can still use the following to judge if there is overfitting:\n1. we usually have a training set and validation set. then we monitor the training and validation loss.\n2. But there are other indicators that is correlated to degree of overfitting. Validation loss is only one of them\n3. say if you have only train data, you can measure \n- degree of regularization and effects on train loss (for different models of different design)\n- rate of change of loss/prediction as data is perturbed\n- rate of change of loss/prediction as parameters is perturbed\n- observe attention, heatmap, etc ....\n\nyou should think of how these indicators change for an optimum, over-fitted, under-fitted trained models.\n\nBut of course having another set of data is still the best way to check over-fitting. so instead of finding methods and indicators, you may just find another set of data (external or synthetically created)\n\n\n", "2188739": "Thank you so much for sharing your approach @hengck23 I was able to train the model from [here](https://www.kaggle.com/datasets/hengck23/asl-demo) \nI pretty much changed no settings except epochs set to 20, because I want the pipeline to work first. \nI used the [inference](https://www.kaggle.com/code/hengck23/lb-0-65-one-pytorch-transformer-solution) however I'm getting OOM even with `max_length=40` \nAny suggestions ? ", "2188724": "thinking of implementing a differentiable topk feature selector ... ", "2187997": "Thanks for your share！！ can I ask what do you mean of \"use triu\"", "2183926": "> one (or two) layer transformer is sufficient\n\nhmmm I have different results to this, deeper transformer lead to a higher CV:\n\n1 layer 0.50\n2 layers 0.53\n3 layers 0.55\n4 layers 0.56", "2183228": "Hi~ You said that `one (or two) layer transformer is sufficient`, is that means we only need 1 layer of MultiHeadAttention? In the paper \"Attention is all you need\", a complete transformer contains 6 duplicated encoders and decoders, and each encoder contains one MultiHeadAttention Layer and something else. I learned your code [lb-0-67-one-pytorch-transformer-solution](https://www.kaggle.com/code/hengck23/lb-0-67-one-pytorch-transformer-solution), and I think you mean we just need one layer of MultiHeadAttention but not just one layer of transformer?\n\nBest wishes\nhappyOldMan", "2182925": "[team forming advertisement]\n\nLooking for experienced Kagglers to form a team. We are interested in working with 8-bit quantization aware training with Keras or pytorch and implementation in tflite. \n\nIt's important to note that our main objective is to learn from experts in deep learning technology, specifically embedded implementation. While we haven't yet decided if we will use an int8 model for our submission, as we are aware that only TFLite supports 8-bit computation for ARM CPU.\n\nWe're committed to achieving the highest ranking on LB, but we also want to emphasize that Kaggle is about having fun and enjoying the competition.\n\nIf this sounds like a team you'd like to be a part of, please submit your application and state your experiences in 8-bit quantization.\nPlease note that only shortlisted candidates will be contacted.\n\nThank you!\n\n(drafted by chatgpt)\nlast update 15-mar", "2182591": "Thanks for sharing your experiment result, I’m curious about the val acc you got. I’ve seen many got a pretty high val acc. (around 0.75), your top1 acc is lower that that, what may be the reason？\nIn my transformer model, I tried many different modifications to the transformer block and it’s hard to break 0.7 on val acc, I got my highest acc at 0.69.", "2167655": ">time_taken = 35 sec (gpu, btach=32)\n\nMay I ask if this is the time of one epoch? Or is it a time of 100 epoch? I'm training on tensorflow, but it feels slow.", "2187722": "", "2179510": "", "2206052": "Thanks for sharing!!" } }