\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9268451%2Fc66948ac5c61830bd513e6af4564cfa4%2Fdenoising_solution.png?generation=1697586862740699&alt=media\")
First of all, we want to thank the competition organizers who hosted this fantastic competition and gave us an excellent opportunity to learn how to improve the ASR model against low-resource languages like Bengali and other competitors who generously shared their knowledge.
\nThe following is a brief explanation of our solution. We will open-source more detailed codes later, but if you have questions, ask us freely.
We fine-tuned \"ai4bharat/indicwav2vec_v1_bengali\" with competition data(CD).
\nWe observed low-quality samples in CD, and mindlessly fine-tuning a model with all the CD deteriorated its performance. So first, we fine-tuned a model only with split=”valid” CD (this improved the model’s performance) and predicted with it against split=”train” CD. After that, we included a high-quality split=’train’ CD (WER<0.75) to split=’valid’ CD and fine-tuned \"ai4bharat/indicwav2vec_v1_bengali\" from scratch.
\nThis improved the public baseline to LB=0.405.
Because there are many out-of-vocabulary in OOD data, we thought training strong LM with large external text data is important. So we downloaded text data and script of ASR datasets(CD, indicCorp v2, common voice, fleurs, openslr, openslr37, and oscar) and trained 5-gram LM.
\nThis LM improved LB score by about 0.01 compared with \"arijitx/wav2vec2-xls-r-300m-bengali\"
We used only the CD. As mentioned in the Model Architecture section, we did something like an adversarial validation using a CTC model trained with split=’valid’ and used about 70% of the all CD.
\nWe used text data and script of ASR datasets(CD, indicCorp v2, common voice, fleurs, openslr, openslr37, and oscar). As a preprocessing, we normalized the text with bnUnicodeNormalizer and removed some characters('[\\,\\?.!-\\;:\\\"\\।\\—]').
\nPadding at the end of audios negatively affected CTC, so we sorted the data based on the audio length and dynamically padded each batch. This increased prediction speed and performance.
\nWe utilized Demucs to denoise audios. This improved the LB score by about 0.003.
\nDemucs sometimes make the audio worse, so we evaluated if the audio gets worse and we switched the audio used in a prediction. This improved the LB score by about 0.001.
\nThe procedure is as follows.
We built models that predict punctuations that go into the spaces between words. These improved LB by more than 0.030.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9268451%2Fdc61bcc13bd81e9a0259a3df7b95ac6f%2Fesprit_solution.png?generation=1697586896853219&alt=media\")
As a tip for training, the CV score was better improved by setting the loss weight of \"PAD\" to 0.0.
\nBackborn: xlm-roberta-large, xlm-roberta-base
\nTrainer: XLMRobertaForTokenClassification
\nDataset: train.csv (given), indicCorp v2
\nPunctuations: [ ,।?-]
・CTC and LM train code
\n・punctuation model code
\n・inference notebook
First of all, we want to thank the competition organizers who hosted this fantastic competition and gave us an excellent opportunity to learn how to improve the ASR model against low-resource languages like Bengali and other competitors who generously shared their knowledge.
\nThe following is a brief explanation of our solution. We will open-source more detailed codes later, but if you have questions, ask us freely.
We fine-tuned \"ai4bharat/indicwav2vec_v1_bengali\" with competition data(CD).
\nWe observed low-quality samples in CD, and mindlessly fine-tuning a model with all the CD deteriorated its performance. So first, we fine-tuned a model only with split=”valid” CD (this improved the model’s performance) and predicted with it against split=”train” CD. After that, we included a high-quality split=’train’ CD (WER<0.75) to split=’valid’ CD and fine-tuned \"ai4bharat/indicwav2vec_v1_bengali\" from scratch.
\nThis improved the public baseline to LB=0.405.
Because there are many out-of-vocabulary in OOD data, we thought training strong LM with large external text data is important. So we downloaded text data and script of ASR datasets(CD, indicCorp v2, common voice, fleurs, openslr, openslr37, and oscar) and trained 5-gram LM.
\nThis LM improved LB score by about 0.01 compared with \"arijitx/wav2vec2-xls-r-300m-bengali\"
We used only the CD. As mentioned in the Model Architecture section, we did something like an adversarial validation using a CTC model trained with split=’valid’ and used about 70% of the all CD.
\nWe used text data and script of ASR datasets(CD, indicCorp v2, common voice, fleurs, openslr, openslr37, and oscar). As a preprocessing, we normalized the text with bnUnicodeNormalizer and removed some characters('[\\,\\?.!-\\;:\\\"\\।\\—]').
\nPadding at the end of audios negatively affected CTC, so we sorted the data based on the audio length and dynamically padded each batch. This increased prediction speed and performance.
\nWe utilized Demucs to denoise audios. This improved the LB score by about 0.003.
\nDemucs sometimes make the audio worse, so we evaluated if the audio gets worse and we switched the audio used in a prediction. This improved the LB score by about 0.001.
\nThe procedure is as follows.
We built models that predict punctuations that go into the spaces between words. These improved LB by more than 0.030.
\n
As a tip for training, the CV score was better improved by setting the loss weight of \"PAD\" to 0.0.
\nBackborn: xlm-roberta-large, xlm-roberta-base
\nTrainer: XLMRobertaForTokenClassification
\nDataset: train.csv (given), indicCorp v2
\nPunctuations: [ ,।?-]
・CTC and LM train code
\n・punctuation model code
\n・inference notebook
Congratulations on the 3rd place!
\nDid xlm-roberta-large fit into GPU memory? I was always getting CUDA OOM
I was able to train with batch_size=32 using a single RTX3090 or 4090.
\nAnd, your public notebook was very helpful. Thank you very much.
Glad I could help. Thanks for your kind words!
", "rawMarkdown": "Glad I could help. Thanks for your kind words!", "votes": 1 } ] } ] }, { "id": 2486567, "postDate": "2023-10-18T03:08:13.700Z", "content": "Congratulations on the 3rd place and thanks for sharing.
\nYesterday, I was contemplating two optimizations, one being a punctuation model. Considering the percentage of words with punctuation it seemed a good bet to improve a few points. I first thought about predicting punctuation in free spaces with a classification model, but after compiling occurrences, I noticed that there are many combinations of two or more consecutive characters, e.g., \"| and |||. After a quick look online (unfortunately I'm totally ignorant on Bengali), I felt that a few might be mistakes, but most reflected correct punctuation.
\nHence, to predict punctuation in blank spaces I'd need to consider multiple possibilities, even after discarding the ones with few occurrences. I felt that a seq2seq model would be better suited to learn how to do that than a classification model. I ended up pursuing the other idea (it had a higher potential, though unfortunately I ran out of time), but I'm still curious about what would have been the way to go. I'd appreciate your thoughts on it.
", "rawMarkdown": "Congratulations on the 3rd place and thanks for sharing. \n\nYesterday, I was contemplating two optimizations, one being a punctuation model. Considering the percentage of words with punctuation it seemed a good bet to improve a few points. I first thought about predicting punctuation in free spaces with a classification model, but after compiling occurrences, I noticed that there are many combinations of two or more consecutive characters, e.g., \"| and |||. After a quick look online (unfortunately I'm totally ignorant on Bengali), I felt that a few might be mistakes, but most reflected correct punctuation. \n\nHence, to predict punctuation in blank spaces I'd need to consider multiple possibilities, even after discarding the ones with few occurrences. I felt that a seq2seq model would be better suited to learn how to do that than a classification model. I ended up pursuing the other idea (it had a higher potential, though unfortunately I ran out of time), but I'm still curious about what would have been the way to go. I'd appreciate your thoughts on it.", "votes": 1, "replies": [ { "id": 2486571, "postDate": "2023-10-18T03:23:31.457Z", "content": "In our model, consecutive [।!?] in the train data were replaced by a single character \"।\" or \"?\". We also removed sentences containing consecutive [,-] from the train data. However, we have not tried anything else, so we cannot say for sure if this was the best idea.
For punctuation frequency, we decided to predict \"-\" as well, since there were many \"-\" in given train.csv.
We also tried to create a seq2seq model (backborn: T5), but this did not work at all. If there is a success solution I would like to see it too.
", "rawMarkdown": "In our model, consecutive `[।!?]` in the train data were replaced by a single character `\"।\"` or `\"?\"`. We also removed sentences containing consecutive `[,-]` from the train data. However, we have not tried anything else, so we cannot say for sure if this was the best idea.\n\nFor punctuation frequency, we decided to predict `\"-\"` as well, since there were many `\"-\"` in given train.csv.\n\nWe also tried to create a seq2seq model (backborn: T5), but this did not work at all. If there is a success solution I would like to see it too.", "votes": 1 } ] }, { "id": 2486454, "postDate": "2023-10-18T00:10:55.273Z", "content": "Thank you for sharing! Liked the postprocessing steps.
\nFor how many epochs did you train your final model and with how much data in total?
", "rawMarkdown": "Thank you for sharing! Liked the postprocessing steps.\n\nFor how many epochs did you train your final model and with how much data in total?", "votes": 1, "replies": [ { "id": 2486494, "postDate": "2023-10-18T01:16:17.297Z", "content": "Which model (CTC, LM, punctuation)?
", "rawMarkdown": "Which model (CTC, LM, punctuation)?", "votes": 1 }, { "id": 2486539, "postDate": "2023-10-18T02:36:15.130Z", "content": "In the training of the punctuation model, 17M Bengali sentences were trained for 1~2 epochs.
", "rawMarkdown": "In the training of the punctuation model, 17M Bengali sentences were trained for 1~2 epochs.", "votes": 1, "replies": [ { "id": 2487949, "postDate": "2023-10-18T22:36:40.603Z", "content": "I was curious about the CTC model, thanks for sharing the details for the punctuation model as well though.
", "rawMarkdown": "I was curious about the CTC model, thanks for sharing the details for the punctuation model as well though.", "votes": 1 }, { "id": 2488041, "postDate": "2023-10-19T01:22:35.647Z", "content": "We trained the CTC model for 10 epochs with 671231 data(split='valid': 28855, cleaned split='train': 642376).
", "rawMarkdown": "We trained the CTC model for 10 epochs with 671231 data(split='valid': 28855, cleaned split='train': 642376).", "votes": 1 }, { "id": 2488400, "postDate": "2023-10-19T08:39:49.550Z", "content": "Oh, nice! Thanks for sharing. Was wondering if it was manageable or not and apparently yes! Well done again!
", "rawMarkdown": "Oh, nice! Thanks for sharing. Was wondering if it was manageable or not and apparently yes! Well done again! ", "votes": 1 } ] } ] }, { "id": 2486505, "postDate": "2023-10-18T01:27:31.777Z", "content": "Congratulations for your 3rd place. \"Predicting punctuations that go into the spaces between words\" sounds amazing for a newbie like me.
", "rawMarkdown": "Congratulations for your 3rd place. \"Predicting punctuations that go into the spaces between words\" sounds amazing for a newbie like me.", "votes": 2 }, { "id": 2491953, "postDate": "2023-10-22T06:55:28.637Z", "content": "Congratulations on the 3rd place 🎉
\nand thanks for sharing your knowledge!
Two questions about \"sort data by audio length\".
\nIt means shuffle=False?
\nref. https://github.com/sagawatatsuya/BengaliAI_Speech_Recognition_3rd_solution/blob/bedefebe763409cbc6b2a0461a325d8f90a5e166/train_CTC/stage1.py#L239
Sorted by ascending order?
\nref. https://github.com/sagawatatsuya/BengaliAI_Speech_Recognition_3rd_solution/blob/bedefebe763409cbc6b2a0461a325d8f90a5e166/train_CTC/stage1.py#L199
During inference, we sorted data based on audio_length in ascending order and set the dataloader's variable shuffle=False.
", "rawMarkdown": "During inference, we sorted data based on audio_length in ascending order and set the dataloader's variable shuffle=False.", "votes": 1, "replies": [ { "id": 2492458, "postDate": "2023-10-22T14:33:49.100Z", "content": "thank you for answering!
", "rawMarkdown": "thank you for answering!" } ] } ] }, { "id": 2486459, "postDate": "2023-10-18T00:12:49.757Z", "content": "Congratulations! I have a question:
\n\n\nSo first, we fine-tuned a model only with split=”valid” CD (this improved the model’s performance) and predicted with it against split=”train” CD. After that, we included a high-quality split=’train’ CD (WER<0.75) to split=’valid’ CD
\n
In the second step did you use the original train split data or the new pseudo labeled data to train the model?
", "rawMarkdown": "Congratulations! I have a question:\n>So first, we fine-tuned a model only with split=”valid” CD (this improved the model’s performance) and predicted with it against split=”train” CD. After that, we included a high-quality split=’train’ CD (WER<0.75) to split=’valid’ CD\n\nIn the second step did you use the original train split data or the new pseudo labeled data to train the model?", "replies": [ { "id": 2486472, "postDate": "2023-10-18T00:41:50.950Z", "content": "Thank you!
\nWe used the original train split data at 2nd step.
Awesome solution! I wonder how to let the LM be uploaded in kaggle input without exceeding the 20G(?) capacity limit. ARPA may seem too large when I train a LM with too much corpus.
", "rawMarkdown": "Awesome solution! I wonder how to let the LM be uploaded in kaggle input without exceeding the 20G(?) capacity limit. ARPA may seem too large when I train a LM with too much corpus.", "replies": [ { "id": 2486477, "postDate": "2023-10-18T00:50:27.227Z", "content": "Thank you.
\nI don't understand what 20GB limit is. The limit of data uploaded to kaggle 107GB. By deleting or making public your datasets, you can upload over 20GB files.
Congratulations on the 3rd place!
\nDid xlm-roberta-large fit into GPU memory? I was always getting CUDA OOM
I was able to train with batch_size=32 using a single RTX3090 or 4090.
\nAnd, your public notebook was very helpful. Thank you very much.
Glad I could help. Thanks for your kind words!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2486567, "author_name": "vialactea", "author_url": "", "post_date": "2023-10-18T03:08:13.700000", "content": "Congratulations on the 3rd place and thanks for sharing.
\nYesterday, I was contemplating two optimizations, one being a punctuation model. Considering the percentage of words with punctuation it seemed a good bet to improve a few points. I first thought about predicting punctuation in free spaces with a classification model, but after compiling occurrences, I noticed that there are many combinations of two or more consecutive characters, e.g., \"| and |||. After a quick look online (unfortunately I'm totally ignorant on Bengali), I felt that a few might be mistakes, but most reflected correct punctuation.
\nHence, to predict punctuation in blank spaces I'd need to consider multiple possibilities, even after discarding the ones with few occurrences. I felt that a seq2seq model would be better suited to learn how to do that than a classification model. I ended up pursuing the other idea (it had a higher potential, though unfortunately I ran out of time), but I'm still curious about what would have been the way to go. I'd appreciate your thoughts on it.
", "votes": 1, "replies": [ { "id": 2486571, "author_name": "esprit", "author_url": "", "post_date": "2023-10-18T03:23:31.457000", "content": "In our model, consecutive [।!?] in the train data were replaced by a single character \"।\" or \"?\". We also removed sentences containing consecutive [,-] from the train data. However, we have not tried anything else, so we cannot say for sure if this was the best idea.
For punctuation frequency, we decided to predict \"-\" as well, since there were many \"-\" in given train.csv.
We also tried to create a seq2seq model (backborn: T5), but this did not work at all. If there is a success solution I would like to see it too.
", "votes": 1, "replies": [] } ] }, { "id": 2486454, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-10-18T00:10:55.273000", "content": "Thank you for sharing! Liked the postprocessing steps.
\nFor how many epochs did you train your final model and with how much data in total?
", "votes": 1, "replies": [ { "id": 2486494, "author_name": "luddite^", "author_url": "", "post_date": "2023-10-18T01:16:17.297000", "content": "Which model (CTC, LM, punctuation)?
", "votes": 1, "replies": [] }, { "id": 2486539, "author_name": "esprit", "author_url": "", "post_date": "2023-10-18T02:36:15.130000", "content": "In the training of the punctuation model, 17M Bengali sentences were trained for 1~2 epochs.
", "votes": 1, "replies": [ { "id": 2487949, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-10-18T22:36:40.603000", "content": "I was curious about the CTC model, thanks for sharing the details for the punctuation model as well though.
", "votes": 1, "replies": [] }, { "id": 2488041, "author_name": "luddite^", "author_url": "", "post_date": "2023-10-19T01:22:35.647000", "content": "We trained the CTC model for 10 epochs with 671231 data(split='valid': 28855, cleaned split='train': 642376).
", "votes": 1, "replies": [] }, { "id": 2488400, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2023-10-19T08:39:49.550000", "content": "Oh, nice! Thanks for sharing. Was wondering if it was manageable or not and apparently yes! Well done again!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2486505, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-10-18T01:27:31.777000", "content": "Congratulations for your 3rd place. \"Predicting punctuations that go into the spaces between words\" sounds amazing for a newbie like me.
", "votes": 2, "replies": [] }, { "id": 2491953, "author_name": "sadahry", "author_url": "", "post_date": "2023-10-22T06:55:28.637000", "content": "Congratulations on the 3rd place 🎉
\nand thanks for sharing your knowledge!
Two questions about \"sort data by audio length\".
\nIt means shuffle=False?
\nref. https://github.com/sagawatatsuya/BengaliAI_Speech_Recognition_3rd_solution/blob/bedefebe763409cbc6b2a0461a325d8f90a5e166/train_CTC/stage1.py#L239
Sorted by ascending order?
\nref. https://github.com/sagawatatsuya/BengaliAI_Speech_Recognition_3rd_solution/blob/bedefebe763409cbc6b2a0461a325d8f90a5e166/train_CTC/stage1.py#L199
During inference, we sorted data based on audio_length in ascending order and set the dataloader's variable shuffle=False.
", "votes": 1, "replies": [ { "id": 2492458, "author_name": "sadahry", "author_url": "", "post_date": "2023-10-22T14:33:49.100000", "content": "thank you for answering!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2486459, "author_name": "Man of the year", "author_url": "", "post_date": "2023-10-18T00:12:49.757000", "content": "Congratulations! I have a question:
\n\n\nSo first, we fine-tuned a model only with split=”valid” CD (this improved the model’s performance) and predicted with it against split=”train” CD. After that, we included a high-quality split=’train’ CD (WER<0.75) to split=’valid’ CD
\n
In the second step did you use the original train split data or the new pseudo labeled data to train the model?
", "votes": 0, "replies": [ { "id": 2486472, "author_name": "luddite^", "author_url": "", "post_date": "2023-10-18T00:41:50.950000", "content": "Thank you!
\nWe used the original train split data at 2nd step.
Awesome solution! I wonder how to let the LM be uploaded in kaggle input without exceeding the 20G(?) capacity limit. ARPA may seem too large when I train a LM with too much corpus.
", "votes": 0, "replies": [ { "id": 2486477, "author_name": "luddite^", "author_url": "", "post_date": "2023-10-18T00:50:27.227000", "content": "Thank you.
\nI don't understand what 20GB limit is. The limit of data uploaded to kaggle 107GB. By deleting or making public your datasets, you can upload over 20GB files.