STT Model:
\nPunctuation Model:
\nIn my daily job, I do speech speech recognition for low resource central Asian languages. From my experience, OpenAI Whisper works really well for OOD audios and can even transribe song lyrics. The downside is it is very sensitive to the annotation noise. So fixing the annotation noise, is the most crucial part of this competition.
\nBecause the competition dataset was not validated, the initial model was trained on OpenSLR datasets. We normalized the texts and filtered out texts containing Bengali digits. All punctuation was also removed. Additionally, we sampled 420k texts from the IndicCorp and synthesized audios using GoogleTTS, which were then used as training datasets.
\nFollowing the training of an initial Whisper-medium model on OpenSLR and GoogleTTS, we conducted inference on MadASR, Shrutilipi, Macro, and Kathbath. We included audios with a WER of less than 15% in the next training phase. After three rounds of training, the model achieved an 8% WER on the Macro validation dataset and a public leaderboard score of approximately 0.380.
\nSince most of the training set audios were short, we merged some short audios to create around 70k longer audios. Subsequently, we achieved a public leaderboard score of approximately 0.370.
\nWhisper with the original tokenizer was slow on Bengali audios. Therefore, we trained a Whisper tokenizer with a 12k vocabulary on Bengali texts. With this tokenizer, we were able to perform inference with a num_beam value of up to 8 and a chunk_length_s of 20.1 seconds in less than 7 hours.
\nIn the next step, we applied pseudo labeling to some YouTube videos, which enabled us to achieve a public leaderboard score of approximately 0.360. When we combined the predictions of four punctuation models, our public leaderboard score improved to around 0.325.
\nBy adding more pseudo-labeled YouTube videos, our public leaderboard score further improved to 0.312 (private LB 0.372)
\nmodel weight and inference notebook: https://www.kaggle.com/competitions/bengaliai-speech/discussion/447970
\ncleaned/long/pseudo data: https://www.kaggle.com/competitions/bengaliai-speech/discussion/448110
STT Model:
\nPunctuation Model:
\nIn my daily job, I do speech speech recognition for low resource central Asian languages. From my experience, OpenAI Whisper works really well for OOD audios and can even transribe song lyrics. The downside is it is very sensitive to the annotation noise. So fixing the annotation noise, is the most crucial part of this competition.
\nBecause the competition dataset was not validated, the initial model was trained on OpenSLR datasets. We normalized the texts and filtered out texts containing Bengali digits. All punctuation was also removed. Additionally, we sampled 420k texts from the IndicCorp and synthesized audios using GoogleTTS, which were then used as training datasets.
\nFollowing the training of an initial Whisper-medium model on OpenSLR and GoogleTTS, we conducted inference on MadASR, Shrutilipi, Macro, and Kathbath. We included audios with a WER of less than 15% in the next training phase. After three rounds of training, the model achieved an 8% WER on the Macro validation dataset and a public leaderboard score of approximately 0.380.
\nSince most of the training set audios were short, we merged some short audios to create around 70k longer audios. Subsequently, we achieved a public leaderboard score of approximately 0.370.
\nWhisper with the original tokenizer was slow on Bengali audios. Therefore, we trained a Whisper tokenizer with a 12k vocabulary on Bengali texts. With this tokenizer, we were able to perform inference with a num_beam value of up to 8 and a chunk_length_s of 20.1 seconds in less than 7 hours.
\nIn the next step, we applied pseudo labeling to some YouTube videos, which enabled us to achieve a public leaderboard score of approximately 0.360. When we combined the predictions of four punctuation models, our public leaderboard score improved to around 0.325.
\nBy adding more pseudo-labeled YouTube videos, our public leaderboard score further improved to 0.312 (private LB 0.372)
\nmodel weight and inference notebook: https://www.kaggle.com/competitions/bengaliai-speech/discussion/447970
\ncleaned/long/pseudo data: https://www.kaggle.com/competitions/bengaliai-speech/discussion/448110
So the whisper do performs better. But why most of us didn't successed with finetuning whisper? Do you have any insights into it?
", "rawMarkdown": "So the whisper do performs better. But why most of us didn't successed with finetuning whisper? Do you have any insights into it? ", "votes": 5, "replies": [ { "id": 2486470, "postDate": "2023-10-18T00:39:57.673Z", "content": "I have updated the solution summary, the important part is you have to remove wrong annotations from the dataset.
", "rawMarkdown": "I have updated the solution summary, the important part is you have to remove wrong annotations from the dataset.", "votes": 9 } ] }, { "id": 2493597, "postDate": "2023-10-23T13:49:51.150Z", "content": "it's amazing congrats for achievement
", "rawMarkdown": "it's amazing congrats for achievement\n", "votes": 1 }, { "id": 2491249, "postDate": "2023-10-21T13:05:39.633Z", "content": "congratulations on your 1st place and thanks for sharing.
", "rawMarkdown": "congratulations on your 1st place and thanks for sharing.", "votes": 1 }, { "id": 2491230, "postDate": "2023-10-21T12:39:24.543Z", "content": "Congratz on 1st place!!! that is crazy and thank you for sharing
", "rawMarkdown": "Congratz on 1st place!!! that is crazy and thank you for sharing \n", "votes": 1 }, { "id": 2490782, "postDate": "2023-10-21T04:21:16.427Z", "content": "Good job to you my sir!
", "rawMarkdown": "Good job to you my sir!", "votes": 1 }, { "id": 2489470, "postDate": "2023-10-20T03:00:52.900Z", "content": "Thanks for sharing your approach @tugstugi
", "rawMarkdown": "Thanks for sharing your approach @tugstugi ", "votes": 1 }, { "id": 2488738, "postDate": "2023-10-19T13:08:02.387Z", "content": "Congratulations on 1st place and thank you for sharing a great solution write-up.
\nCould you tell me how you downloaded YouTube videos which specifically contains Bengali speaks? Is there any criteria to which audio used?
Browse YouTube and identify Bangle channels like news, cartoon, poetry etc. Then download all videos from the channels and do pseudo labeling. Nothing fancy. I have released the data here: https://www.kaggle.com/datasets/tugstugi/bengali-asr-data The audio file name starts with the channel name + YouTube id.
", "rawMarkdown": "Browse YouTube and identify Bangle channels like news, cartoon, poetry etc. Then download all videos from the channels and do pseudo labeling. Nothing fancy. I have released the data here: https://www.kaggle.com/datasets/tugstugi/bengali-asr-data The audio file name starts with the channel name + YouTube id.", "votes": 3, "replies": [ { "id": 2489323, "postDate": "2023-10-19T22:21:53.947Z", "content": "I understood. Thank you!
", "rawMarkdown": "I understood. Thank you!" } ] } ] }, { "id": 2487792, "postDate": "2023-10-18T18:54:06.917Z", "content": "impressive score.
\nCongratulations
It is very good and nice speech
", "rawMarkdown": "It is very good and nice speech ", "votes": 1 }, { "id": 2487089, "postDate": "2023-10-18T11:05:40.580Z", "content": "Very impressive score. Congratulations on your win.
\nI have a question about the pseudo labels.
\nHow did you handle the alignment between audio and transcription? For example, a simple approach of audio separation at 30 seconds and transcription?
Long youtube audios are splitted using with VAD. Reject too short or too long audio segments and let's say use only segments between 5 and 22 seconds.
", "rawMarkdown": "Long youtube audios are splitted using with VAD. Reject too short or too long audio segments and let's say use only segments between 5 and 22 seconds.", "votes": 2, "replies": [ { "id": 2489385, "postDate": "2023-10-20T00:48:53.893Z", "content": "Thank you. An additional question, did you use any libraries for the VAD?
", "rawMarkdown": "Thank you. An additional question, did you use any libraries for the VAD?", "votes": 3 }, { "id": 2492009, "postDate": "2023-10-22T07:49:51.883Z", "content": "you can use any VAD library, but i have used webrtcvad.
", "rawMarkdown": "you can use any VAD library, but i have used webrtcvad.", "votes": 3 }, { "id": 2493148, "postDate": "2023-10-23T07:14:40.910Z", "content": "Thanks for the clarification!
", "rawMarkdown": "Thanks for the clarification!", "votes": 1 } ] } ] }, { "id": 2486703, "postDate": "2023-10-18T05:49:15.990Z", "content": "Unique approach in this competition, outstanding score in private and public lb. Impressive! Well done!
", "rawMarkdown": "Unique approach in this competition, outstanding score in private and public lb. Impressive! Well done!", "votes": 1 }, { "id": 2486503, "postDate": "2023-10-18T01:25:10.607Z", "content": "A huge congratulations Tugstugi for your 1st place!
", "rawMarkdown": "A huge congratulations Tugstugi for your 1st place!", "votes": 1 }, { "id": 2490749, "postDate": "2023-10-21T02:44:30.287Z", "content": "Congratulations on your 1st place. Thanks for sharing this great solution. I will certainlyCongratulations on your 1st place. Thanks for sharing this great solution. I will certain get a 1sst like that .
", "rawMarkdown": "Congratulations on your 1st place. Thanks for sharing this great solution. I will certainlyCongratulations on your 1st place. Thanks for sharing this great solution. I will certain get a 1sst like that .", "votes": 2 }, { "id": 2489757, "postDate": "2023-10-20T07:23:32.973Z", "content": "amazing, congratulations!
", "rawMarkdown": "amazing, congratulations! ", "votes": 2 }, { "id": 2488163, "postDate": "2023-10-19T04:39:16.587Z", "content": "That's so cool!
", "rawMarkdown": "That's so cool!", "votes": 2 }, { "id": 2486467, "postDate": "2023-10-18T00:21:02.333Z", "content": "Learned a lot from Whisper performance with long train time, outperforming Wav2Vec2π
", "rawMarkdown": "Learned a lot from Whisper performance with long train time, outperforming Wav2Vec2π", "votes": 2 }, { "id": 2606989, "postDate": "2024-01-18T01:58:19.450Z", "content": "Hi Tugstugi,
\nCongratulations on your phenomenal victory.
\nWould it be possible to replicate your training scripts on a 24GB RTX4090 by reducing batch sizes? How many GPU hours did it take to train your model 8x48GB RTX A6000 setup? What was the aggregated size of the datasets for the STT models? Thank you.
", "rawMarkdown": "Hi Tugstugi,\n\nCongratulations on your phenomenal victory.\n\nWould it be possible to replicate your training scripts on a 24GB RTX4090 by reducing batch sizes? How many GPU hours did it take to train your model 8x48GB RTX A6000 setup? What was the aggregated size of the datasets for the STT models? Thank you. " }, { "id": 2590693, "postDate": "2024-01-07T11:39:22.077Z", "content": "oh good good.
", "rawMarkdown": "oh good good." }, { "id": 2588964, "postDate": "2024-01-05T22:45:59.767Z", "content": "hey pal congratulations π
\nsorry if my question is off-topic; it is just that your use of external GPU power contradicts my understanding of code competition, has this competition changed rules after your submission? or does code competition allow the use of external cloud resources during training?
", "rawMarkdown": "hey pal congratulations π\n\nsorry if my question is off-topic; it is just that your use of external GPU power contradicts my understanding of code competition, has this competition changed rules after your submission? or does code competition allow the use of external cloud resources during training?\n\n", "replies": [ { "id": 2614873, "postDate": "2024-01-22T20:44:34.750Z", "content": "in a code competition, only the inference time is limited. So you can upload the trained weights (as a dataset) beforehand, then make a blank notebook that imports those weights and performs the inference within the timeframe.
", "rawMarkdown": "in a code competition, only the inference time is limited. So you can upload the trained weights (as a dataset) beforehand, then make a blank notebook that imports those weights and performs the inference within the timeframe." } ] }, { "id": 2499682, "postDate": "2023-10-26T07:12:55.287Z", "content": "Congratulations @tugstugi inspirational news.
", "rawMarkdown": "Congratulations @tugstugi inspirational news." }, { "id": 2499672, "postDate": "2023-10-26T07:09:02.807Z", "content": "can you please share the training notebook as well?
", "rawMarkdown": "can you please share the training notebook as well?" }, { "id": 2496590, "postDate": "2023-10-24T06:39:26.500Z", "content": "amazing, congratulations!!!. On your 1st place.
", "rawMarkdown": "amazing, congratulations!!!. On your 1st place. \n" }, { "id": 2486923, "postDate": "2023-10-18T08:41:20.410Z", "content": "Just curious how much was the cost of the training? Was a cloud hosted GPU used? Fantastic write-up by the way!
", "rawMarkdown": "Just curious how much was the cost of the training? Was a cloud hosted GPU used? Fantastic write-up by the way!" }, { "id": 2486480, "postDate": "2023-10-18T00:57:37.300Z", "content": "Did you preprocess the dataset beforehand or did you read the audio files before each batch?
", "rawMarkdown": "Did you preprocess the dataset beforehand or did you read the audio files before each batch? ", "replies": [ { "id": 2486482, "postDate": "2023-10-18T01:00:48.290Z", "content": "We don't do any preprocessing except converting to 16khz. During the training, audios are read from the disk and applied ramdom speed/pitch etc before computing features.
", "rawMarkdown": "We don't do any preprocessing except converting to 16khz. During the training, audios are read from the disk and applied ramdom speed/pitch etc before computing features.", "votes": 2 } ] }, { "id": 2486479, "postDate": "2023-10-18T00:56:02.317Z", "content": "Great solution! I also noticed how inefficient the tokenizer was for whisper. How did you initialize the word embeddings after changing the tokenizer?
", "rawMarkdown": "Great solution! I also noticed how inefficient the tokenizer was for whisper. How did you initialize the word embeddings after changing the tokenizer? ", "replies": [ { "id": 2486518, "postDate": "2023-10-18T01:38:54.157Z", "content": "Thanks, @nbroad, The original Whisper tokenizer employs character-level tokens for low-resource languages, which can be time-consuming. So we trained a BPE tokenizer with 12,000 tokens specifically for Bengali text. We then replaced some tokens in the Whisper tokenizer with these. We carefully replaced tokens after the 10,000th position of the original Whisper's tokens.
", "rawMarkdown": "Thanks, @nbroad, The original Whisper tokenizer employs character-level tokens for low-resource languages, which can be time-consuming. So we trained a BPE tokenizer with 12,000 tokens specifically for Bengali text. We then replaced some tokens in the Whisper tokenizer with these. We carefully replaced tokens after the 10,000th position of the original Whisper's tokens.", "votes": 8, "replies": [ { "id": 2491715, "postDate": "2023-10-21T23:53:09.053Z", "content": "Wow that's a great approach and idea. For the text, do you use BPE for just text from speech examples or did you even include written text like stuff from news articles or wiki dumps? My first thought is that written data would would be different enough from speech text to cause issue, but maybe those minor difference are drowned out from the overall tokenization benefit?
", "rawMarkdown": "Wow that's a great approach and idea. For the text, do you use BPE for just text from speech examples or did you even include written text like stuff from news articles or wiki dumps? My first thought is that written data would would be different enough from speech text to cause issue, but maybe those minor difference are drowned out from the overall tokenization benefit?" } ] } ] }, { "id": 2861401, "postDate": "2024-06-08T07:00:49.323Z", "content": "Can you please tell me how you handled the noise annotation , I am facing the same issue in whisper-medium , its keep getting hallucinated whenever it hears annotation noise, can you tell me how to reduce the hallucination and handle the annotation noise , I've tried the repetition penalty , changed the beam size , changed the -log probability , compression ratio , nothing seems to me working good
", "rawMarkdown": "Can you please tell me how you handled the noise annotation , I am facing the same issue in whisper-medium , its keep getting hallucinated whenever it hears annotation noise, can you tell me how to reduce the hallucination and handle the annotation noise , I've tried the repetition penalty , changed the beam size , changed the -log probability , compression ratio , nothing seems to me working good", "isDeleted": true }, { "id": 2487845, "postDate": "2023-10-18T19:46:39.940Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2486466, "author_name": "Mutian Hong", "author_url": "", "post_date": "2023-10-18T00:20:57.933000", "content": "So the whisper do performs better. But why most of us didn't successed with finetuning whisper? Do you have any insights into it?
", "votes": 5, "replies": [ { "id": 2486470, "author_name": "tugstugi", "author_url": "", "post_date": "2023-10-18T00:39:57.673000", "content": "I have updated the solution summary, the important part is you have to remove wrong annotations from the dataset.
", "votes": 9, "replies": [] } ] }, { "id": 2493597, "author_name": "Sameer", "author_url": "", "post_date": "2023-10-23T13:49:51.150000", "content": "it's amazing congrats for achievement
", "votes": 1, "replies": [] }, { "id": 2491249, "author_name": "Ali Aghayari", "author_url": "", "post_date": "2023-10-21T13:05:39.633000", "content": "congratulations on your 1st place and thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 2491230, "author_name": "Max Marriott-Clarke", "author_url": "", "post_date": "2023-10-21T12:39:24.543000", "content": "Congratz on 1st place!!! that is crazy and thank you for sharing
", "votes": 1, "replies": [] }, { "id": 2490782, "author_name": "Brian The Messiah 410", "author_url": "", "post_date": "2023-10-21T04:21:16.427000", "content": "Good job to you my sir!
", "votes": 1, "replies": [] }, { "id": 2489470, "author_name": "Indra Sonowal", "author_url": "", "post_date": "2023-10-20T03:00:52.900000", "content": "Thanks for sharing your approach @tugstugi
", "votes": 1, "replies": [] }, { "id": 2488738, "author_name": "luddite^", "author_url": "", "post_date": "2023-10-19T13:08:02.387000", "content": "Congratulations on 1st place and thank you for sharing a great solution write-up.
\nCould you tell me how you downloaded YouTube videos which specifically contains Bengali speaks? Is there any criteria to which audio used?
Browse YouTube and identify Bangle channels like news, cartoon, poetry etc. Then download all videos from the channels and do pseudo labeling. Nothing fancy. I have released the data here: https://www.kaggle.com/datasets/tugstugi/bengali-asr-data The audio file name starts with the channel name + YouTube id.
", "votes": 3, "replies": [ { "id": 2489323, "author_name": "luddite^", "author_url": "", "post_date": "2023-10-19T22:21:53.947000", "content": "I understood. Thank you!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2487792, "author_name": "Abdallah Hesham", "author_url": "", "post_date": "2023-10-18T18:54:06.917000", "content": "impressive score.
\nCongratulations
It is very good and nice speech
", "votes": 1, "replies": [] }, { "id": 2487089, "author_name": "ktr", "author_url": "", "post_date": "2023-10-18T11:05:40.580000", "content": "Very impressive score. Congratulations on your win.
\nI have a question about the pseudo labels.
\nHow did you handle the alignment between audio and transcription? For example, a simple approach of audio separation at 30 seconds and transcription?
Long youtube audios are splitted using with VAD. Reject too short or too long audio segments and let's say use only segments between 5 and 22 seconds.
", "votes": 2, "replies": [ { "id": 2489385, "author_name": "ktr", "author_url": "", "post_date": "2023-10-20T00:48:53.893000", "content": "Thank you. An additional question, did you use any libraries for the VAD?
", "votes": 3, "replies": [] }, { "id": 2492009, "author_name": "tugstugi", "author_url": "", "post_date": "2023-10-22T07:49:51.883000", "content": "you can use any VAD library, but i have used webrtcvad.
", "votes": 3, "replies": [] }, { "id": 2493148, "author_name": "ktr", "author_url": "", "post_date": "2023-10-23T07:14:40.910000", "content": "Thanks for the clarification!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2486703, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2023-10-18T05:49:15.990000", "content": "Unique approach in this competition, outstanding score in private and public lb. Impressive! Well done!
", "votes": 1, "replies": [] }, { "id": 2486503, "author_name": "MarΓlia Prata", "author_url": "", "post_date": "2023-10-18T01:25:10.607000", "content": "A huge congratulations Tugstugi for your 1st place!
", "votes": 1, "replies": [] }, { "id": 2490749, "author_name": "Tran Le Minh Nhat", "author_url": "", "post_date": "2023-10-21T02:44:30.287000", "content": "Congratulations on your 1st place. Thanks for sharing this great solution. I will certainlyCongratulations on your 1st place. Thanks for sharing this great solution. I will certain get a 1sst like that .
", "votes": 2, "replies": [] }, { "id": 2489757, "author_name": "YuanXu", "author_url": "", "post_date": "2023-10-20T07:23:32.973000", "content": "amazing, congratulations!
", "votes": 2, "replies": [] }, { "id": 2488163, "author_name": "Vishrut Grover", "author_url": "", "post_date": "2023-10-19T04:39:16.587000", "content": "That's so cool!
", "votes": 2, "replies": [] }, { "id": 2486467, "author_name": "iiiiitsu", "author_url": "", "post_date": "2023-10-18T00:21:02.333000", "content": "Learned a lot from Whisper performance with long train time, outperforming Wav2Vec2π
", "votes": 2, "replies": [] }, { "id": 2606989, "author_name": "tahsin1221", "author_url": "", "post_date": "2024-01-18T01:58:19.450000", "content": "Hi Tugstugi,
\nCongratulations on your phenomenal victory.
\nWould it be possible to replicate your training scripts on a 24GB RTX4090 by reducing batch sizes? How many GPU hours did it take to train your model 8x48GB RTX A6000 setup? What was the aggregated size of the datasets for the STT models? Thank you.
", "votes": 0, "replies": [] }, { "id": 2590693, "author_name": "BIraj Ar", "author_url": "", "post_date": "2024-01-07T11:39:22.077000", "content": "oh good good.
", "votes": 0, "replies": [] }, { "id": 2588964, "author_name": "Mansour Hassan", "author_url": "", "post_date": "2024-01-05T22:45:59.767000", "content": "hey pal congratulations π
\nsorry if my question is off-topic; it is just that your use of external GPU power contradicts my understanding of code competition, has this competition changed rules after your submission? or does code competition allow the use of external cloud resources during training?
", "votes": 0, "replies": [ { "id": 2614873, "author_name": "Curtis Chong", "author_url": "", "post_date": "2024-01-22T20:44:34.750000", "content": "in a code competition, only the inference time is limited. So you can upload the trained weights (as a dataset) beforehand, then make a blank notebook that imports those weights and performs the inference within the timeframe.
", "votes": 0, "replies": [] } ] }, { "id": 2499682, "author_name": "Tariq Mahmood", "author_url": "", "post_date": "2023-10-26T07:12:55.287000", "content": "Congratulations @tugstugi inspirational news.
", "votes": 0, "replies": [] }, { "id": 2499672, "author_name": "Shubham Agnihotri", "author_url": "", "post_date": "2023-10-26T07:09:02.807000", "content": "can you please share the training notebook as well?
", "votes": 0, "replies": [] }, { "id": 2496590, "author_name": "zisa123", "author_url": "", "post_date": "2023-10-24T06:39:26.500000", "content": "amazing, congratulations!!!. On your 1st place.
", "votes": 0, "replies": [] }, { "id": 2486923, "author_name": "Wayne Lau", "author_url": "", "post_date": "2023-10-18T08:41:20.410000", "content": "Just curious how much was the cost of the training? Was a cloud hosted GPU used? Fantastic write-up by the way!
", "votes": 0, "replies": [] }, { "id": 2486480, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-10-18T00:57:37.300000", "content": "Did you preprocess the dataset beforehand or did you read the audio files before each batch?
", "votes": 0, "replies": [ { "id": 2486482, "author_name": "tugstugi", "author_url": "", "post_date": "2023-10-18T01:00:48.290000", "content": "We don't do any preprocessing except converting to 16khz. During the training, audios are read from the disk and applied ramdom speed/pitch etc before computing features.
", "votes": 2, "replies": [] } ] }, { "id": 2486479, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-10-18T00:56:02.317000", "content": "Great solution! I also noticed how inefficient the tokenizer was for whisper. How did you initialize the word embeddings after changing the tokenizer?
", "votes": 0, "replies": [ { "id": 2486518, "author_name": "lhagiimn", "author_url": "", "post_date": "2023-10-18T01:38:54.157000", "content": "Thanks, @nbroad, The original Whisper tokenizer employs character-level tokens for low-resource languages, which can be time-consuming. So we trained a BPE tokenizer with 12,000 tokens specifically for Bengali text. We then replaced some tokens in the Whisper tokenizer with these. We carefully replaced tokens after the 10,000th position of the original Whisper's tokens.
", "votes": 8, "replies": [ { "id": 2491715, "author_name": "Matt S.", "author_url": "", "post_date": "2023-10-21T23:53:09.053000", "content": "Wow that's a great approach and idea. For the text, do you use BPE for just text from speech examples or did you even include written text like stuff from news articles or wiki dumps? My first thought is that written data would would be different enough from speech text to cause issue, but maybe those minor difference are drowned out from the overall tokenization benefit?
", "votes": 0, "replies": [] } ] } ] }, { "id": 2861401, "author_name": "Navaneetha Krishnan", "author_url": "", "post_date": "2024-06-08T07:00:49.323000", "content": "Can you please tell me how you handled the noise annotation , I am facing the same issue in whisper-medium , its keep getting hallucinated whenever it hears annotation noise, can you tell me how to reduce the hallucination and handle the annotation noise , I've tried the repetition penalty , changed the beam size , changed the -log probability , compression ratio , nothing seems to me working good
", "votes": 0, "replies": [] }, { "id": 2487845, "author_name": "", "author_url": "", "post_date": "2023-10-18T19:46:39.940000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2486461": "STT Model:\n- OpenAI whisper-medium\n- Huggingface trainer\n- Trained on 8x 48GB RTX A6000\n- bs=8 and lr=1e-5\n- Train steps 50k\n- Spectrogram dithering\n- Spectrogram time and frequency masking\n- Resampling 16khz->8khz->16khz as augmentation\n- Inference with max_length=260, num_beams=4 and chunk_length_s=20.1s\n- Libsonic based speed/pitch augmentation\n- Datasets: OpenSLR 37, OpenSLR 53, MadASR, Shrutilipi, Macro, Kathbath, GoogleTTS generated audios and pseudo labeled YouTube videos\n\n\nPunctuation Model:\n- AutoModelForTokenClassification google/muril-base-cased\n- Huggingface trainer\n- Labels: period, comma and question mark\n- bs=64, lr=2e-4 and max_seq_length=512\n- Ensemble of 4 models (using 6, 8, 11 and 12 layers of google/muril-base-cased)\n- Normalized IndicCorp v2 Bangla dataset\n\nIn my daily job, I do speech speech recognition for low resource central Asian languages. From my experience, OpenAI Whisper works really well for OOD audios and can even transribe song lyrics. The downside is it is very sensitive to the annotation noise. So fixing the annotation noise, is the most crucial part of this competition.\n\nBecause the competition dataset was not validated, the initial model was trained on OpenSLR datasets. We normalized the texts and filtered out texts containing Bengali digits. All punctuation was also removed. Additionally, we sampled 420k texts from the IndicCorp and synthesized audios using GoogleTTS, which were then used as training datasets.\n\nFollowing the training of an initial Whisper-medium model on OpenSLR and GoogleTTS, we conducted inference on MadASR, Shrutilipi, Macro, and Kathbath. We included audios with a WER of less than 15% in the next training phase. After three rounds of training, the model achieved an 8% WER on the Macro validation dataset and a public leaderboard score of approximately 0.380.\n\nSince most of the training set audios were short, we merged some short audios to create around 70k longer audios. Subsequently, we achieved a public leaderboard score of approximately 0.370.\n\nWhisper with the original tokenizer was slow on Bengali audios. Therefore, we trained a Whisper tokenizer with a 12k vocabulary on Bengali texts. With this tokenizer, we were able to perform inference with a num_beam value of up to 8 and a chunk_length_s of 20.1 seconds in less than 7 hours.\n\nIn the next step, we applied pseudo labeling to some YouTube videos, which enabled us to achieve a public leaderboard score of approximately 0.360. When we combined the predictions of four punctuation models, our public leaderboard score improved to around 0.325.\n\nBy adding more pseudo-labeled YouTube videos, our public leaderboard score further improved to 0.312 (private LB 0.372)\n\nmodel weight and inference notebook: https://www.kaggle.com/competitions/bengaliai-speech/discussion/447970\ncleaned/long/pseudo data: https://www.kaggle.com/competitions/bengaliai-speech/discussion/448110", "2486466": "So the whisper do performs better. But why most of us didn't successed with finetuning whisper? Do you have any insights into it? ", "2493597": "it's amazing congrats for achievement\n", "2491249": "congratulations on your 1st place and thanks for sharing.", "2491230": "Congratz on 1st place!!! that is crazy and thank you for sharing \n", "2490782": "Good job to you my sir!", "2489470": "Thanks for sharing your approach @tugstugi ", "2488738": "Congratulations on 1st place and thank you for sharing a great solution write-up.\nCould you tell me how you downloaded YouTube videos which specifically contains Bengali speaks? Is there any criteria to which audio used?", "2487792": "impressive score. \n**Congratulations**", "2487668": "It is very good and nice speech ", "2487089": "Very impressive score. Congratulations on your win.\nI have a question about the pseudo labels.\nHow did you handle the alignment between audio and transcription? For example, a simple approach of audio separation at 30 seconds and transcription?", "2486703": "Unique approach in this competition, outstanding score in private and public lb. Impressive! Well done!", "2486503": "A huge congratulations Tugstugi for your 1st place!", "2490749": "Congratulations on your 1st place. Thanks for sharing this great solution. I will certainlyCongratulations on your 1st place. Thanks for sharing this great solution. I will certain get a 1sst like that .", "2489757": "amazing, congratulations! ", "2488163": "That's so cool!", "2486467": "Learned a lot from Whisper performance with long train time, outperforming Wav2Vec2π", "2606989": "Hi Tugstugi,\n\nCongratulations on your phenomenal victory.\n\nWould it be possible to replicate your training scripts on a 24GB RTX4090 by reducing batch sizes? How many GPU hours did it take to train your model 8x48GB RTX A6000 setup? What was the aggregated size of the datasets for the STT models? Thank you. ", "2590693": "oh good good.", "2588964": "hey pal congratulations π\n\nsorry if my question is off-topic; it is just that your use of external GPU power contradicts my understanding of code competition, has this competition changed rules after your submission? or does code competition allow the use of external cloud resources during training?\n\n", "2499682": "Congratulations @tugstugi inspirational news.", "2499672": "can you please share the training notebook as well?", "2496590": "amazing, congratulations!!!. On your 1st place. \n", "2486923": "Just curious how much was the cost of the training? Was a cloud hosted GPU used? Fantastic write-up by the way!", "2486480": "Did you preprocess the dataset beforehand or did you read the audio files before each batch? ", "2486479": "Great solution! I also noticed how inefficient the tokenizer was for whisper. How did you initialize the word embeddings after changing the tokenizer? ", "2861401": "Can you please tell me how you handled the noise annotation , I am facing the same issue in whisper-medium , its keep getting hallucinated whenever it hears annotation noise, can you tell me how to reduce the hallucination and handle the annotation noise , I've tried the repetition penalty , changed the beam size , changed the -log probability , compression ratio , nothing seems to me working good", "2487845": "" } }