The model we used is
\n\nWe finetuned it further on this model bengali_wav2vec2_finetuned (kaggle.com)
\nSince the data quality of the competition dataset is low, we used the train split of Common Voice 13 | Bengali (Normalized) (kaggle.com)to train. We trained about 10 epochs with the 20000 data in the dataset, and splited about 700 data for validation.
\nThe data here is normalized and removed punctuations.
\nHere is the training arguments:
training_args = TrainingArguments(\n group_by_length=,\n weight_decay=,\n num_train_epochs=,\n fp16=,\n learning_rate=,\n warmup_steps=,\n)\nAnd we used cosine optimizer.
\nThe best model got local wer 0.15, and Improved the Public Score from 0.445->0.434
\nHowever, my teammate also trained a model with the same data for just 60 steps( different args), and got the same score. (even 0.001 better on the private LB). How interesting and confusing.
\nWe trained a 6gram with lm_no_punc (kaggle.com).
\nMention that there is a error in the LM provided in YellowKing_DLSprint_Model (kaggle.com).
There is no terminator arpa . So we need </s> .\nThanks to this notebook Build an n-gram with KenLM | MaCro | Kaggle, we are able to realize this.
\nAdding the improved the LB from 0.445->0.422
\nAnd building an 6gram with lm_no_punc (kaggle.com).Improved about 0.001
Punctuation really matters. Thanks to this post Bengali.AI Speech Recognition | Kaggle, we are able to realize this. And an response under this post showed us a way to restore the punctuation:
\nxashru/punctuation-restoration: Punctuation Restoration using Transformer Models for High-and Low-Resource Languages (github.com) First we trained a model with the dataset provided in this repositorie. It can restore 3 punctuations :
{: , : , : }\nThis improved the LB from 0.422->0.400
\nCombining with the finetuned model, we have 0.400->0.397
\nAfterwards, we thought that 3 punctuations might be not enouth. So we made a dataset with oscar · Datasets at Hugging Face, filterd datas that have only have bengali words. We chose 7
\npunctuations:
{: , : , : , : , : , : , : }\nWe trained 6 epochs with the default pharams.
\nAnd we have 0.393->0.387
We simply ensembled our model like this:
\n y = model_1(x).logits* + model_2(x).logits* + model_3(x).logits*\nWait, This works???
\nYes, thouth the predictions may not be aligned, But since the three models are trained on same datasets, the no-aligning problem is paritially solved.
\nThis improved our performance about 0.001
There are three main pharams for the decoder:
\nalpha: weight language model during shallow fusion\nbeta: weight score adjustment during scoring\nbeam_width: determines candidate output sequences retained step.\nTo find the best pharams, we used optuna [0.444] Optimize Decoding Parameters with Optuna | Kaggle to search the best pharams. We searched the pharams with the example datas in the dataset, which is ood data, brought us better LB score.
\nThe final decoder pharams are:
{'alpha': , 'beta': , 'beam_width': }\nThe model we used is
\n\nWe finetuned it further on this model bengali_wav2vec2_finetuned (kaggle.com)
\nSince the data quality of the competition dataset is low, we used the train split of Common Voice 13 | Bengali (Normalized) (kaggle.com)to train. We trained about 10 epochs with the 20000 data in the dataset, and splited about 700 data for validation.
\nThe data here is normalized and removed punctuations.
\nHere is the training arguments:
training_args = TrainingArguments(\n group_by_length=,\n weight_decay=,\n num_train_epochs=,\n fp16=,\n learning_rate=,\n warmup_steps=,\n)\nAnd we used cosine optimizer.
\nThe best model got local wer 0.15, and Improved the Public Score from 0.445->0.434
\nHowever, my teammate also trained a model with the same data for just 60 steps( different args), and got the same score. (even 0.001 better on the private LB). How interesting and confusing.
\nWe trained a 6gram with lm_no_punc (kaggle.com).
\nMention that there is a error in the LM provided in YellowKing_DLSprint_Model (kaggle.com).
There is no terminator arpa . So we need </s> .\nThanks to this notebook Build an n-gram with KenLM | MaCro | Kaggle, we are able to realize this.
\nAdding the improved the LB from 0.445->0.422
\nAnd building an 6gram with lm_no_punc (kaggle.com).Improved about 0.001
Punctuation really matters. Thanks to this post Bengali.AI Speech Recognition | Kaggle, we are able to realize this. And an response under this post showed us a way to restore the punctuation:
\nxashru/punctuation-restoration: Punctuation Restoration using Transformer Models for High-and Low-Resource Languages (github.com) First we trained a model with the dataset provided in this repositorie. It can restore 3 punctuations :
{: , : , : }\nThis improved the LB from 0.422->0.400
\nCombining with the finetuned model, we have 0.400->0.397
\nAfterwards, we thought that 3 punctuations might be not enouth. So we made a dataset with oscar · Datasets at Hugging Face, filterd datas that have only have bengali words. We chose 7
\npunctuations:
{: , : , : , : , : , : , : }\nWe trained 6 epochs with the default pharams.
\nAnd we have 0.393->0.387
We simply ensembled our model like this:
\n y = model_1(x).logits* + model_2(x).logits* + model_3(x).logits*\nWait, This works???
\nYes, thouth the predictions may not be aligned, But since the three models are trained on same datasets, the no-aligning problem is paritially solved.
\nThis improved our performance about 0.001
There are three main pharams for the decoder:
\nalpha: weight language model during shallow fusion\nbeta: weight score adjustment during scoring\nbeam_width: determines candidate output sequences retained step.\nTo find the best pharams, we used optuna [0.444] Optimize Decoding Parameters with Optuna | Kaggle to search the best pharams. We searched the pharams with the example datas in the dataset, which is ood data, brought us better LB score.
\nThe final decoder pharams are:
{'alpha': , 'beta': , 'beam_width': }\nI can confirm that way of ensembling somehow works :)
\nThank you for sharing
Did you used DTW to align logits? You said before in previous ensemble discussion thread?
", "rawMarkdown": "Did you used DTW to align logits? You said before in previous ensemble discussion thread?", "replies": [ { "id": 2488891, "postDate": "2023-10-19T15:10:19.800Z", "content": "@aifahim Sadly no, I didn't have time. I found a publication that supports it though. If you're interested, I can point you to this paper: https://ieeexplore.ieee.org/document/9687953
\n\"Warped Ensembles: A Novel Technique for Improving CTC Based End-to-End Speech Recognition\"
\nThey mentioned the use of DTW there.
牛但是得凑够十个字才能发送
", "rawMarkdown": "牛但是得凑够十个字才能发送" }, { "id": 2488159, "postDate": "2023-10-19T04:35:46.877Z", "content": "Congratulation for your great achievement. Still didn't get you ensemble part! Can you share your pseudocode how you do this?
", "rawMarkdown": "Congratulation for your great achievement. Still didn't get you ensemble part! Can you share your pseudocode how you do this? ", "replies": [ { "id": 2488514, "postDate": "2023-10-19T09:47:32.540Z", "content": "Yes, here is our main inference code:
\n torch.no_grad():\n batch tqdm(test_loader):\n x = batch[]\n x = x.to(device, non_blocking=)\n\n torch.cuda.amp.autocast():\n y = model_1(x).logits* + model_2(x).logits* + model_3(x).logits*\n y = y.detach().cpu().numpy()\n\n l y: \n sentence = processor_with_lm.decode(l, **best_params).text\n pred_sentence_list.append(sentence)\nHowever, it is the wrong way to do the ensemble. Because the format of the logits of wav2vec2 model is not aligned. There are other ways to do this, like (https://www.kaggle.com/competitions/bengaliai-speech/discussion/448074#2487921) this solution.
", "rawMarkdown": "Yes, here is our main inference code:\n```python\nwith torch.no_grad():\n for batch in tqdm(test_loader):\n x = batch[\"input_values\"]\n x = x.to(device, non_blocking=True)\n# x = x.squeeze(0)\n with torch.cuda.amp.autocast(True):\n y = model_1(x).logits*0.7 + model_2(x).logits*0.2 + model_3(x).logits*0.1\n y = y.detach().cpu().numpy()\n \n for l in y: \n sentence = processor_with_lm.decode(l, **best_params).text\n pred_sentence_list.append(sentence)\n```\nHowever, it is the wrong way to do the ensemble. Because the format of the logits of wav2vec2 model is not aligned. There are other ways to do this, like (https://www.kaggle.com/competitions/bengaliai-speech/discussion/448074#2487921) this solution.", "votes": 1, "replies": [ { "id": 2488980, "postDate": "2023-10-19T15:58:58.650Z", "content": "Thank you so much @hongori 😃
", "rawMarkdown": "Thank you so much @hongori 😃" } ] } ] }, { "id": 2487858, "postDate": "2023-10-18T19:57:10.077Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 2487449, "author_name": "yukiya", "author_url": "", "post_date": "2023-10-18T15:43:27.847000", "content": "I can confirm that way of ensembling somehow works :)
\nThank you for sharing
Did you used DTW to align logits? You said before in previous ensemble discussion thread?
", "votes": 0, "replies": [ { "id": 2488891, "author_name": "yukiya", "author_url": "", "post_date": "2023-10-19T15:10:19.800000", "content": "@aifahim Sadly no, I didn't have time. I found a publication that supports it though. If you're interested, I can point you to this paper: https://ieeexplore.ieee.org/document/9687953
\n\"Warped Ensembles: A Novel Technique for Improving CTC Based End-to-End Speech Recognition\"
\nThey mentioned the use of DTW there.
牛但是得凑够十个字才能发送
", "votes": 0, "replies": [] }, { "id": 2488159, "author_name": "AIFahim", "author_url": "", "post_date": "2023-10-19T04:35:46.877000", "content": "Congratulation for your great achievement. Still didn't get you ensemble part! Can you share your pseudocode how you do this?
", "votes": 0, "replies": [ { "id": 2488514, "author_name": "Mutian Hong", "author_url": "", "post_date": "2023-10-19T09:47:32.540000", "content": "Yes, here is our main inference code:
\n torch.no_grad():\n batch tqdm(test_loader):\n x = batch[]\n x = x.to(device, non_blocking=)\n\n torch.cuda.amp.autocast():\n y = model_1(x).logits* + model_2(x).logits* + model_3(x).logits*\n y = y.detach().cpu().numpy()\n\n l y: \n sentence = processor_with_lm.decode(l, **best_params).text\n pred_sentence_list.append(sentence)\nHowever, it is the wrong way to do the ensemble. Because the format of the logits of wav2vec2 model is not aligned. There are other ways to do this, like (https://www.kaggle.com/competitions/bengaliai-speech/discussion/448074#2487921) this solution.
", "votes": 1, "replies": [ { "id": 2488980, "author_name": "AIFahim", "author_url": "", "post_date": "2023-10-19T15:58:58.650000", "content": "Thank you so much @hongori 😃
", "votes": 0, "replies": [] } ] } ] }, { "id": 2487858, "author_name": "", "author_url": "", "post_date": "2023-10-18T19:57:10.077000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2487380": "First of all, we want to thank the organizers who held this wonderful competition. This is my first kaggle competition, I fell satisfied with the final outcome. Thanks my teammate @nisshokuitsuki , who worked with me during this three months journel. Thanks everyone who contributed to the discussion and the notebooks, your works gave us a lot insperiations. \n---\n# Model\nThe model we used is\n```\nWav2Vec2ForCTC\n```\nWe finetuned it further on this model [bengali_wav2vec2_finetuned (kaggle.com)](https://www.kaggle.com/datasets/nischaydnk/bengali-wav2vec2-finetuned)\nSince the data quality of the competition dataset is low, we used the train split of [Common Voice 13 | Bengali (Normalized) (kaggle.com)](https://www.kaggle.com/datasets/umongsain/common-voice-13-bengali-normalized?select=train.tsv)to train. We trained about 10 epochs with the 20000 data in the dataset, and splited about 700 data for validation. \nThe data here is normalized and removed punctuations.\nHere is the training arguments:\n```python\ntraining_args = TrainingArguments(\n group_by_length=False,\n weight_decay=0.01,\n num_train_epochs=10,\n fp16=True,\n learning_rate=4e-5,\n warmup_steps=600,\n)\n```\nAnd we used cosine optimizer.\n\nThe best model got local wer 0.15, and Improved the Public Score from *0.445->0.434*\n\nHowever, my teammate also trained a model with the same data for just 60 steps( different args), and got the same score. (even 0.001 better on the private LB). How interesting and confusing.\n\n# Language Model\nWe trained a 6gram with [lm_no_punc (kaggle.com)](https://www.kaggle.com/datasets/mbmmurad/lm-no-punc). \nMention that there is a error in the LM provided in [YellowKing_DLSprint_Model (kaggle.com)](https://www.kaggle.com/datasets/sameen53/yellowking-dlsprint-model). \n```\nThere is no terminator in the arpa file. So we need to add an into the file.\n```\nThanks to this notebook [Build an n-gram with KenLM | MaCro | Kaggle](https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro), we are able to realize this. \nAdding the improved the LB from *0.445->0.422*\nAnd building an 6gram with [lm_no_punc (kaggle.com)](https://www.kaggle.com/datasets/mbmmurad/lm-no-punc).Improved about *0.001*\n# Punctuation Restoration\nPunctuation really matters. Thanks to this post [Bengali.AI Speech Recognition | Kaggle](https://www.kaggle.com/competitions/bengaliai-speech/discussion/432305), we are able to realize this. And an response under this post showed us a way to restore the punctuation:\n[xashru/punctuation-restoration: Punctuation Restoration using Transformer Models for High-and Low-Resource Languages (github.com)](https://github.com/xashru/punctuation-restoration) First we trained a model with the dataset provided in this repositorie. It can restore 3 punctuations : \n```python\n{1: ',', 2: '।', 3: '?'}\n```\n\nThis improved the LB from *0.422->0.400*\nCombining with the finetuned model, we have *0.400->0.397*\nAfterwards, we thought that 3 punctuations might be not enouth. So we made a dataset with [oscar · Datasets at Hugging Face](https://huggingface.co/datasets/oscar), filterd datas that have only have bengali words. We chose 7\npunctuations: \n```python\n{1: ',', 2: '।', 3: '?', 4: '!', 5: '-', 6: '\"', 7: ':'}\n```\nWe trained 6 epochs with the default pharams.\nAnd we have *0.393->0.387*\n\n# Model ensemble\nWe simply ensembled our model like this: \n```python\n y = model_1(x).logits*0.7 + model_2(x).logits*0.2 + model_3(x).logits*0.1\n```\nWait, This works???\nYes, thouth the predictions may not be aligned, But since the three models are trained on same datasets, the no-aligning problem is paritially solved. \nThis improved our performance about *0.001*\n\n# Decoder pharams selection\nThere are three main pharams for the decoder:\n```\nalpha: weight for language model during shallow fusion\nbeta: weight for length score adjustment of during scoring\nbeam_width: determines the number of candidate output sequences retained at each time step.\n```\nTo find the best pharams, we used optuna [[0.444] Optimize Decoding Parameters with Optuna | Kaggle](https://www.kaggle.com/code/snnclsr/0-444-optimize-decoding-parameters-with-optuna) to search the best pharams. We searched the pharams with the example datas in the dataset, which is ood data, brought us better LB score.\nThe final decoder pharams are:\n```\n{'alpha': 0.46570704474381447, 'beta': 0.8635977171858652, 'beam_width': 768}\n```\n\n\n# What doesn't work for us\n- Data augmentation. We added background noise downloaded from https://pixabay.com/sound-effects/search/noise/ and also pitch shift , time stretch etc. But The LB got worse (*0.397->0.415*). Every experiment of data augmentation takes too much time and I fells to tired to do more experiments. Maybe I could write some codes to do it automatically\n- Denoise model. We tried three denoising models: \n\t\tUVR: Notebook Run Out of time\n\t\t[facebookresearch/denoiser](https://github.com/facebookresearch/denoiser): decreased about 0.01\n\t\t[CleanUNet](https://github.com/NVIDIA/CleanUNet/blob/main/exp/DNS-large-high/checkpoint/pretrained.pkl): decreased about 0.005\n\tWe thought that denoising harms the features and makes some short syllables unrecornizable.\n- Train a bigger LM with more data. We used 15G normalized benglai data to build an kenlm, the score got worse. We still haven't found the cause of the problem. \n", "2487449": "I can confirm that way of ensembling somehow works :)\nThank you for sharing", "3106246": "牛但是得凑够十个字才能发送", "2488159": "Congratulation for your great achievement. Still didn't get you ensemble part! Can you share your pseudocode how you do this? ", "2487858": "" } }