![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4772132%2Fc86799e65036ebc9228f9f709a98a309%2FScreenshot%20from%202023-10-20%2020-15-23.png?generation=1697811401235581&alt=media\")
First of all, thank you to the organizers for arranging this exciting competition, and congratulations to the winners. I have learned a lot throughout this journey. The discussion section and public notebooks were quite helpful. I feel a bit regretful because I couldn't dedicate enough time due to my final exams.
\nA few months back, I improved the public notebook by @nischaydnk just by training my own n-gram language model. It improved both my public LB (0.445 -> 0.436) and private LB (0.534 -> 0.515).
\nI normalized the training texts using bnunicodenormalizer as well as the normalizer developed by CSE BUET NLP. Some data could not be normalized because of the confusion which contained the character \" ঃ\" (বিসর্গ) as a separate word. As it is mostly used in place of \":\" (colon), I replaced them with colons instead.
\n
Before training the n-gram model, I removed the punctuation from the normalized text data because Wav2Vec2 does not predict any punctuation. I trained the n-gram model using pyctcdecode.
\nLater, I tried to fine-tune multiple XLS-R and MMS models from scratch. As I had limited resources, I focused on low-resource training, using only 30k samples from the Common Voice 13. Unfortunately, they could not beat my previous baseline. But what I found is that MMS 1B works quite well for low-resource languages. Sadly, it is available only for non-commercial uses. The checkpoints can be found here:
\nI had plans to use QLoRa to fine-tune on a few out-of-domain (OOD) hand-labeled samples for domain adaptation after fine-tuning on the crowd-sourced data. I wish I had a bit more free time.
\nI am glad to have my 1st Bronze competition medal. Again, thanks to the organizers.
", "messages": [ { "id": 2487361, "postDate": "2023-10-18T14:31:38.157Z", "content": "First of all, thank you to the organizers for arranging this exciting competition, and congratulations to the winners. I have learned a lot throughout this journey. The discussion section and public notebooks were quite helpful. I feel a bit regretful because I couldn't dedicate enough time due to my final exams.
\nA few months back, I improved the public notebook by @nischaydnk just by training my own n-gram language model. It improved both my public LB (0.445 -> 0.436) and private LB (0.534 -> 0.515).
\nI normalized the training texts using bnunicodenormalizer as well as the normalizer developed by CSE BUET NLP. Some data could not be normalized because of the confusion which contained the character \" ঃ\" (বিসর্গ) as a separate word. As it is mostly used in place of \":\" (colon), I replaced them with colons instead.
\n
Before training the n-gram model, I removed the punctuation from the normalized text data because Wav2Vec2 does not predict any punctuation. I trained the n-gram model using pyctcdecode.
\nLater, I tried to fine-tune multiple XLS-R and MMS models from scratch. As I had limited resources, I focused on low-resource training, using only 30k samples from the Common Voice 13. Unfortunately, they could not beat my previous baseline. But what I found is that MMS 1B works quite well for low-resource languages. Sadly, it is available only for non-commercial uses. The checkpoints can be found here:
\nI had plans to use QLoRa to fine-tune on a few out-of-domain (OOD) hand-labeled samples for domain adaptation after fine-tuning on the crowd-sourced data. I wish I had a bit more free time.
\nI am glad to have my 1st Bronze competition medal. Again, thanks to the organizers.
", "rawMarkdown": "First of all, thank you to the organizers for arranging this exciting competition, and congratulations to the winners. I have learned a lot throughout this journey. The discussion section and public notebooks were quite helpful. I feel a bit regretful because I couldn't dedicate enough time due to my final exams.\n\n## My Approach\nA few months back, I improved [the public notebook](https://www.kaggle.com/code/nischaydnk/bengali-finetuning-baseline-wav2vec2-inference) by @nischaydnk just by training my own [n-gram language model](https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro). It improved both my public LB (0.445 -> 0.436) and private LB (0.534 -> 0.515).\n\n## Preparing the Data\nI normalized the training texts using [bnunicodenormalizer](https://github.com/mnansary/bnUnicodeNormalizer) as well as the normalizer developed by [CSE BUET NLP](https://github.com/csebuetnlp/normalizer). Some data could not be normalized because of the confusion which contained the character \" ঃ\" ([বিসর্গ](https://bn.wikipedia.org/wiki/%E0%A6%AC%E0%A6%BF%E0%A6%B8%E0%A6%B0%E0%A7%8D%E0%A6%97)) as a separate word. As it is mostly used in place of \":\" (colon), I replaced them with colons instead.\n\n\n\n## Training the N-gram Model\nBefore training the n-gram model, I removed the punctuation from the normalized text data because Wav2Vec2 does not predict any punctuation. I trained the n-gram model using [pyctcdecode](https://github.com/kensho-technologies/pyctcdecode).\n\n## Solution Code\n- Data preparation: https://www.kaggle.com/code/umongsain/macro-normalization\n- Training code: https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro/notebook\n- Inference code: https://www.kaggle.com/code/umongsain/67th-place-solution-inference-notebook\n\n## Things I Tried But Did Not Work\nLater, I tried to fine-tune multiple XLS-R and MMS models from scratch. As I had limited resources, I focused on low-resource training, using only 30k samples from the Common Voice 13. Unfortunately, they could not beat my previous baseline. But what I found is that [MMS 1B](https://huggingface.co/facebook/mms-1b) works quite well for low-resource languages. Sadly, it is available only for non-commercial uses. The checkpoints can be found here:\n\n- https://huggingface.co/Umong/wav2vec2-xls-r-300m-bengali\n- https://huggingface.co/Umong/wav2vec2-large-mms-1b-bengali\n\n## Things That I Could Have Tried\nI had plans to use QLoRa to fine-tune on a few out-of-domain (OOD) hand-labeled samples for domain adaptation after fine-tuning on the crowd-sourced data. I wish I had a bit more free time.\n\nI am glad to have my 1st Bronze competition medal. Again, thanks to the organizers.", "votes": 9 }, { "id": 2487865, "postDate": "2023-10-18T20:00:03.470Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2487865, "author_name": "", "author_url": "", "post_date": "2023-10-18T20:00:03.470000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2487361": "First of all, thank you to the organizers for arranging this exciting competition, and congratulations to the winners. I have learned a lot throughout this journey. The discussion section and public notebooks were quite helpful. I feel a bit regretful because I couldn't dedicate enough time due to my final exams.\n\n## My Approach\nA few months back, I improved [the public notebook](https://www.kaggle.com/code/nischaydnk/bengali-finetuning-baseline-wav2vec2-inference) by @nischaydnk just by training my own [n-gram language model](https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro). It improved both my public LB (0.445 -> 0.436) and private LB (0.534 -> 0.515).\n\n## Preparing the Data\nI normalized the training texts using [bnunicodenormalizer](https://github.com/mnansary/bnUnicodeNormalizer) as well as the normalizer developed by [CSE BUET NLP](https://github.com/csebuetnlp/normalizer). Some data could not be normalized because of the confusion which contained the character \" ঃ\" ([বিসর্গ](https://bn.wikipedia.org/wiki/%E0%A6%AC%E0%A6%BF%E0%A6%B8%E0%A6%B0%E0%A7%8D%E0%A6%97)) as a separate word. As it is mostly used in place of \":\" (colon), I replaced them with colons instead.\n\n\n\n## Training the N-gram Model\nBefore training the n-gram model, I removed the punctuation from the normalized text data because Wav2Vec2 does not predict any punctuation. I trained the n-gram model using [pyctcdecode](https://github.com/kensho-technologies/pyctcdecode).\n\n## Solution Code\n- Data preparation: https://www.kaggle.com/code/umongsain/macro-normalization\n- Training code: https://www.kaggle.com/code/umongsain/build-an-n-gram-with-kenlm-macro/notebook\n- Inference code: https://www.kaggle.com/code/umongsain/67th-place-solution-inference-notebook\n\n## Things I Tried But Did Not Work\nLater, I tried to fine-tune multiple XLS-R and MMS models from scratch. As I had limited resources, I focused on low-resource training, using only 30k samples from the Common Voice 13. Unfortunately, they could not beat my previous baseline. But what I found is that [MMS 1B](https://huggingface.co/facebook/mms-1b) works quite well for low-resource languages. Sadly, it is available only for non-commercial uses. The checkpoints can be found here:\n\n- https://huggingface.co/Umong/wav2vec2-xls-r-300m-bengali\n- https://huggingface.co/Umong/wav2vec2-large-mms-1b-bengali\n\n## Things That I Could Have Tried\nI had plans to use QLoRa to fine-tune on a few out-of-domain (OOD) hand-labeled samples for domain adaptation after fine-tuning on the crowd-sourced data. I wish I had a bit more free time.\n\nI am glad to have my 1st Bronze competition medal. Again, thanks to the organizers.", "2487865": "" } }