![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8577629%2F04724b5e0fab1e4b550271153025dcd3%2Fdomain.png?generation=1697591004781091&alt=media\")
\n(dataset paper: https://arxiv.org/abs/2305.09688 C.1.1. Data Scraping Roadmap & Prerequisites)
Thanks to the organizers and Kaggle staff for holding the competition, and congratulations to the winners!
\nBusiness context: https://www.kaggle.com/competitions/bengaliai-speech/overview
\nData context: https://www.kaggle.com/competitions/bengaliai-speech/data
My approach addressed two main challenges:
\nChallenge:
\nApproach:
\nindicwav2vec_v1_bengali model using competition data.This approach led to a Public Leaderboard score of 0.38.
\nI began with the foundation provided by @ttahara’s notebook.
\nDiverse Speaker Recognition:
\nThe test audio data comes mostly from YouTube, which means that the speakers' identities are often unknown. To create a versatile model, I trained it on diverse audio data.
\n(dataset paper: https://arxiv.org/abs/2305.09688 C.1.1. Data Scraping Roadmap & Prerequisites)
Punctuation Restoration:
\nIt's known that the labels are normalized, which implies that punctuation is preserved, as mentioned here.
\nPredicting punctuation during transcription is difficult, and leaving it out would result in word errors.
By restoring punctuation, Word Error Rate (WER) can be reduced:
\nWhile the public notebook appends periods at the end of sentences, the test data has an average of 34.42 words per sample and a Macro Train/Validation set with averages of 8.42/9.21. This suggests that multiple sentences may exist in one audio file, making it necessary to restore punctuation at points other than sentence endings.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8577629%2F1aa92dc382aaefb5ee7797fdc42a5c94%2Fwps.png?generation=1697591051306745&alt=media\")
\n(dataset paper: https://arxiv.org/abs/2305.09688 Table 1: OOD-Speech Dataset Statistics)
ai4bharat/indicwav2vec_v1_bengaliarijitx/wav2vec2-xls-r-300m-bengaliFine-tuning Wav2vec2CTC with Transformers involved choosing datasets based on Yellowking’s WER, CER, and MOS_PRED metrics, as outlined in this notebook. Two dataset splits were used:
\nThe first dataset helped the model adapt to a variety of voices, while the second dataset allowed it to handle audio with higher WER.
\nAdditionally, the inclusion of white noise during training led to a slight improvement in the Leaderboard score by 0.001.
\nI only used LB for Validate.
\nI couldn't rely on local cross-validation because the domain shift between the training data and the test data was too significant.
improved things from public notebook
\nPunctuation restoration (-0.023)
\nusing unigrams.txt for KenLM ( -0.005)
( / , encoding=) :\n unigram_list = [t.() t f.().().()]\n\ndecoder = pyctcdecode.(\n (sorted_vocab_dict.()),\n ( / ),\n unigram_list,\n)\n| \n | Public | \nImprovement | \n\n | Private | \n
|---|---|---|---|---|
| Baseline | \n0.471 | \n\n | \n | 0.564 | \n
| Training (Easy Data) | \n0.425 | \n-0.046 | \n\n | 0.508 | \n
| Beam width 1500 | \n0.423 | \n-0.002 | \n\n | 0.506 | \n
| Punctuation restoration | \n0.400 | \n-0.023 | \n\n | 0.488 | \n
| Using unigrams.txt | \n0.395 | \n-0.005 | \n\n | 0.48 | \n
| Training (Hard Data) | \n0.380 | \n-0.015 | \n\n | 0.458 | \n
At the beginning of the competition, I tried adding noise to adapt to different domains, but it didn't improve the LB. From this, I thought that there were other challenges to address besides noise.
\nThis competition challenged participants to achieve generalization and deal with label noise (punctuation). It required addressing the question of how much generalization is necessary and how to handle punctuation noise effectively.
\nI am grateful for the opportunity to learn from this competition.
\nThanks to the organizers and Kaggle staff for holding the competition, and congratulations to the winners!
\nBusiness context: https://www.kaggle.com/competitions/bengaliai-speech/overview
\nData context: https://www.kaggle.com/competitions/bengaliai-speech/data
My approach addressed two main challenges:
\nChallenge:
\nApproach:
\nindicwav2vec_v1_bengali model using competition data.This approach led to a Public Leaderboard score of 0.38.
\nI began with the foundation provided by @ttahara’s notebook.
\nDiverse Speaker Recognition:
\nThe test audio data comes mostly from YouTube, which means that the speakers' identities are often unknown. To create a versatile model, I trained it on diverse audio data.
\n(dataset paper: https://arxiv.org/abs/2305.09688 C.1.1. Data Scraping Roadmap & Prerequisites)
Punctuation Restoration:
\nIt's known that the labels are normalized, which implies that punctuation is preserved, as mentioned here.
\nPredicting punctuation during transcription is difficult, and leaving it out would result in word errors.
By restoring punctuation, Word Error Rate (WER) can be reduced:
\nWhile the public notebook appends periods at the end of sentences, the test data has an average of 34.42 words per sample and a Macro Train/Validation set with averages of 8.42/9.21. This suggests that multiple sentences may exist in one audio file, making it necessary to restore punctuation at points other than sentence endings.
\n
\n(dataset paper: https://arxiv.org/abs/2305.09688 Table 1: OOD-Speech Dataset Statistics)
ai4bharat/indicwav2vec_v1_bengaliarijitx/wav2vec2-xls-r-300m-bengaliFine-tuning Wav2vec2CTC with Transformers involved choosing datasets based on Yellowking’s WER, CER, and MOS_PRED metrics, as outlined in this notebook. Two dataset splits were used:
\nThe first dataset helped the model adapt to a variety of voices, while the second dataset allowed it to handle audio with higher WER.
\nAdditionally, the inclusion of white noise during training led to a slight improvement in the Leaderboard score by 0.001.
\nI only used LB for Validate.
\nI couldn't rely on local cross-validation because the domain shift between the training data and the test data was too significant.
improved things from public notebook
\nPunctuation restoration (-0.023)
\nusing unigrams.txt for KenLM ( -0.005)
( / , encoding=) :\n unigram_list = [t.() t f.().().()]\n\ndecoder = pyctcdecode.(\n (sorted_vocab_dict.()),\n ( / ),\n unigram_list,\n)\n| \n | Public | \nImprovement | \n\n | Private | \n
|---|---|---|---|---|
| Baseline | \n0.471 | \n\n | \n | 0.564 | \n
| Training (Easy Data) | \n0.425 | \n-0.046 | \n\n | 0.508 | \n
| Beam width 1500 | \n0.423 | \n-0.002 | \n\n | 0.506 | \n
| Punctuation restoration | \n0.400 | \n-0.023 | \n\n | 0.488 | \n
| Using unigrams.txt | \n0.395 | \n-0.005 | \n\n | 0.48 | \n
| Training (Hard Data) | \n0.380 | \n-0.015 | \n\n | 0.458 | \n
At the beginning of the competition, I tried adding noise to adapt to different domains, but it didn't improve the LB. From this, I thought that there were other challenges to address besides noise.
\nThis competition challenged participants to achieve generalization and deal with label noise (punctuation). It required addressing the question of how much generalization is necessary and how to handle punctuation noise effectively.
\nI am grateful for the opportunity to learn from this competition.
\nI found good result after changing beam width 512 -> 1024. But you got more improvement with 1500! Kudos to you for your achievement :)
", "rawMarkdown": "I found good result after changing beam width 512 -> 1024. But you got more improvement with 1500! Kudos to you for your achievement :) ", "votes": 1, "replies": [ { "id": 2486792, "postDate": "2023-10-18T07:04:37.710Z", "content": "Thank you!👍
", "rawMarkdown": "Thank you!👍" } ] }, { "id": 2487171, "postDate": "2023-10-18T12:00:47.813Z", "content": "Has xlm-roberta-large been fine-tuned?
", "rawMarkdown": "Has xlm-roberta-large been fine-tuned?", "replies": [ { "id": 2487194, "postDate": "2023-10-18T12:26:08.367Z", "content": "Hasn't. using pre-trained model from github repo.
\nI tried fine-tune with MacRo Train sentence.
\nbut LB became worse (0.400 -> 0.405) so I gave up.
I found good result after changing beam width 512 -> 1024. But you got more improvement with 1500! Kudos to you for your achievement :)
", "votes": 1, "replies": [ { "id": 2486792, "author_name": "neilus", "author_url": "", "post_date": "2023-10-18T07:04:37.710000", "content": "Thank you!👍
", "votes": 0, "replies": [] } ] }, { "id": 2487171, "author_name": "answer-qzd2", "author_url": "", "post_date": "2023-10-18T12:00:47.813000", "content": "Has xlm-roberta-large been fine-tuned?
", "votes": 0, "replies": [ { "id": 2487194, "author_name": "neilus", "author_url": "", "post_date": "2023-10-18T12:26:08.367000", "content": "Hasn't. using pre-trained model from github repo.
\nI tried fine-tune with MacRo Train sentence.
\nbut LB became worse (0.400 -> 0.405) so I gave up.