![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd1010fbf802e033d78dfb6ed2043dbea%2FSelection_999(2747).png?generation=1689742315359191&alt=media\")
https://huggingface.co/ai4bharat/indicwav2vec_v1_bengali (CTC model only)
\nadd a LM yourself, you can train one or use e.g. https://huggingface.co/shahruk10/wav2vec2-xls-r-300m-bengali-commonvoice
\nnormalise + dari
\nsome lbscore:\n[]+[b]+[c]: 0.481\n[]+[c]: 0.520\n[]+[b]: 0.550\n[]: \n[]+[b]+[c/only normalise]: 0.490\n
https://www.kaggle.com/code/hengck23/local-wer-0-2600-nemo-baseline-conformer
\ninitial results …. to be updated as experiments progress
https://huggingface.co/ai4bharat/indicwav2vec_v1_bengali (CTC model only)
\nadd a LM yourself, you can train one or use e.g. https://huggingface.co/shahruk10/wav2vec2-xls-r-300m-bengali-commonvoice
\nnormalise + dari
\nsome lbscore:\n[]+[b]+[c]: 0.481\n[]+[c]: 0.520\n[]+[b]: 0.550\n[]: \n[]+[b]+[c/only normalise]: 0.490\n
https://www.kaggle.com/code/hengck23/local-wer-0-2600-nemo-baseline-conformer
\ninitial results …. to be updated as experiments progress
some important todo list:
\nthe trick to winning is really the OOD (out of distribution) private test set
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fa28e51cd1564da2a042d70893ec20d51%2FSelection_999(2748).png?generation=1689742664897606&alt=media\")
Note:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F261acc12207cdb46587c27f675c55cdb%2FSelection_999(2750).png?generation=1689743292178787&alt=media\")
you should read the paper very carefully
", "rawMarkdown": ".png?generation=1689743292178787&alt=media)\n\nyou should read the paper very carefully", "votes": null }, { "id": "2350229", "postDate": "07/19/2023 05:38:34", "content": "papers related to out-of-distribution and asr:
\nusing lanuage model is another trick
\nhttps://huggingface.co/arijitx/wav2vec2-xls-r-300m-bengali
https://huggingface.co/blog/wav2vec2-with-ngram
\n language model :\n\n: .\n: .\n gram language model trained M sentences randomly chosen from AI4Bharat IndicCorp dataset :\n\n: .\n: .\ni think i find some OOD dataset:
\nhttps://github.com/Open-Speech-EkStep/ULCA-asr-dataset-corpus
\ne.g. youtube data
\nEntertainment Mirchi_Bangla_1 Labelled Mirchi_Bangla_1 54 (hrs)
\nEntertainment Mirchi_Bangla_2 Labelled Mirchi_Bangla_2
\nEntertainment Mirchi_Bangla_3 Labelled Mirchi_Bangla_3
\nGeneral CTVN_AKD_PLUS_1 Labelled CTVN_AKD_PLUS_1 21.7 (hrs)
\nGeneral CTVN_AKD_PLUS_2 Labelled CTVN_AKD_PLUS_2
however, the data are not available ….
\nbut the pretrain model are here
\nhttps://github.com/Open-Speech-EkStep/vakyansh-models
\nVakyansh-Conformer-SSL
\"This model was pre-trained using Nemo toolkit with 34,000 hours unlabeled audio in 39 Indian languages. This includes 15,000 hours of news recordings available on the internet, 10,000 hours of YouTube audios and other audio data.\"
\nmodel config yaml:
\ntrain_ds:
\n manifest_filepath: /mnt/lustre/megh/indic-ssl/manifest/crisil_noa_nptel_yt_indic_ssl_train.json
i am guessing
\nnoa : news on air
\nyt: youtube
paper:
\nVakyansh: ASR Toolkit for Low Resource Indic languages
\nhttps://arxiv.org/pdf/2203.16512.pdf
sampling rate as TTA?
\nsampling_rate=\n\n : one\n d = valid_df\n mp3_file = f\n\n = (mp3_file) \n with (mp3_file, ) as f:\n bpayload = f()\n = ( bpayload , sampling_rate)\n = (a) \n (, p)\n (, d.sentence)\n Hz\npredict ও বলেছে আপনার টিকাপ\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict ওবলেছে আপনার টিকা\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict এগুন শ অ\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict অবিচয় আনাটি\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict ও বলেছে আপনার টিকা\ntruth ও বলেছে আপনার ঠিকানা!\nthis youtube channel has many tutorials:
\nhttps://www.youtube.com/@ai4bharat/videos
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3e96ec049ef64542d26eac80c7249bc4%2FSelection_999(2766).png?generation=1690008184783299&alt=media\")
e.g
\nhttps://www.youtube.com/watch?v=iSyipKKgleo
visualise distance edit
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fa32c0720b98a08aedccc73603163a0bf%2FSelection_999(2767).png?generation=1690128936873031&alt=media\")
https://github.com/ukiuki-satoshi/visedit
", "rawMarkdown": "visualise distance edit\n\n.png?generation=1690128936873031&alt=media)\n\nhttps://github.com/ukiuki-satoshi/visedit", "votes": null }, { "id": "2356341", "postDate": "07/24/2023 05:36:49", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd719f707bd940500540adeb99191780d%2FSelection_999(2811).png?generation=1690403372321495&alt=media\")
\n
\n
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F11ad141e94c7a32b3148618428ab7b11%2FSelection_999(2797).png?generation=1690186435573149&alt=media\")
OOD results
", "rawMarkdown": ".png?generation=1690186435573149&alt=media)\n\nOOD results", "votes": null }, { "id": "2356515", "postDate": "07/24/2023 08:56:36", "content": "\" We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type\"
\nhttps://arxiv.org/pdf/2307.03183.pdf
\nWhisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
End-to-end Music-mixed Speech Recognition
\nhttps://arxiv.org/pdf/2008.12048.pdf
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3fbba9c75392b0f594274f017d7bbeec%2FSelection_999(2798).png?generation=1690189696857157&alt=media\")
Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition
", "rawMarkdown": ".png?generation=1690189696857157&alt=media)\n\nWav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition", "votes": null }, { "id": "2356736", "postDate": "07/24/2023 10:52:08", "content": "https://sites.google.com/iitdh.ac.in/vssasr2021/resources
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F55dbc9f676130c29ea69dd771132459e%2FSelection_999(2799).png?generation=1690195926421740&alt=media\")
CTC bert?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F40e8bbfbd5da3f9ddd2af5e56f66ed62%2FSelection_999(2803).png?generation=1690298599425663&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F231e9aa6db041b8f6fe453217fbedd6f%2FSelection_999(2804).png?generation=1690298610687049&alt=media\")
@hengck23 How did you get the ground truth for the test_example to calculate the wer ? It's not available in the Data section
\nEDIT: sorry, I've found the source https://www.kaggle.com/competitions/bengaliai-speech/discussion/425932
", "rawMarkdown": "hengck23 How did you get the ground truth for the test_example to calculate the wer ? It's not available in the Data section \n\nEDIT: sorry, I've found the source https://www.kaggle.com/competitions/bengaliai-speech/discussion/425932", "votes": null }, { "id": "2359216", "postDate": "07/26/2023 04:31:33", "content": "massive external data:
\nhttps://github.com/AI4Bharat/vistaar#download-training-datasets-and-benchmarks
i made a mistake. i forget about normalisation and others in post-processing
", "rawMarkdown": "i made a mistake. i forget about normalisation and others in post-processing", "votes": null }, { "id": "2360720", "postDate": "07/27/2023 00:33:30", "content": "paper on effects of in and out domain :
\nROBUST WAV2VEC 2.0: ANALYZING DOMAIN SHIFT IN SELF-SUPERVISED PRE-TRAINING
\nhttps://arxiv.org/pdf/2104.01027.pdf
\"In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain\"
", "rawMarkdown": "paper on effects of in and out domain :\nROBUST WAV2VEC 2.0: ANALYZING DOMAIN SHIFT IN SELF-SUPERVISED PRE-TRAINING\nhttps://arxiv.org/pdf/2104.01027.pdf\n\n\"In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain\"", "votes": null }, { "id": "2360878", "postDate": "07/27/2023 04:36:08", "content": "yet another external data
\nhttps://globalrecordings.net
i wonder if the each sentence in the test is unique?
\nif not, this is a leak?
\n(e.g. one may collect speech data by asking different people to speak the same sentence)
Thanks for posting all the insights. May I ask how long does it take you to run whsiper for leaderboard submssion?
", "rawMarkdown": "Thanks for posting all the insights. May I ask how long does it take you to run whsiper for leaderboard submssion?", "votes": null }, { "id": "2368194", "postDate": "08/01/2023 04:04:52", "content": "whisper takes in 30sec audio chunk. so i did a probe on audio clip duration of hidden test data:
\nvalid_df = pd()\nmp3_dir = f\n\nduration=\n t,d valid_df():\n audio_file = f\n\n sampling_rate = _000\n with (audio_file, ) as f:\n bpayload = f()\n = (bpayload, sampling_rate)\n dur = (a)/sampling_rate\n (dur)\n duration(dur)\n\nmax_duration=np(duration)\n\n\nsubmission failed for : max_duration<=30 sec
\nsubmission passed for : max_duration<=32 sec
Thank you for the info … so much research on the topic great …all the best
", "rawMarkdown": "Thank you for the info ... so much research on the topic great ...all the best", "votes": null }, { "id": "2369358", "postDate": "08/01/2023 17:34:50", "content": "another dataset
\nDataset - RESPIN
\nhttps://slt2022.org/projects/13%20-%20Dialectical%20speech%20recognition%20for%20two%20Indian%20languages%20-%20Bengali%20and%20Bhojpuri/13%20-%20Dialectical%20speech%20recognition%20for%20two%20Indian%20languages%20-%20Bengali%20and%20Bhojpuri.pdf
https://sites.google.com/view/respinasrchallenge2023/home
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F6ff8e18a7c97388e287e41fd5bcf58bb%2FSelection_999(2829).png?generation=1690911460374350&alt=media\")
code switch dataset:
\nhttps://github.com/navana-tech/baseline_recipe_is21s_indic_asr_challenge
Hi, is indicwav2vec_v1_bengali simply transformed from the fairseq model on indicwav2vec github repo?
", "rawMarkdown": "Hi, is indicwav2vec_v1_bengali simply transformed from the fairseq model on indicwav2vec github repo?", "votes": null }, { "id": "2370496", "postDate": "08/02/2023 13:09:32", "content": "i have submitted the version from hugging face.
\ni may choose to use the github version later
", "rawMarkdown": "i have submitted the version from hugging face.\n\ni may choose to use the github version later", "votes": null }, { "id": "2370590", "postDate": "08/02/2023 14:04:55", "content": "
\nThey are the same model
In my experiment, directly binding IndicWav2Vec2 to Yellowking's decoder has led to some problems, that is the vocabularie sizes have not matched The former is 87 and the latter is 112. How can I solve it? Thanks!
", "rawMarkdown": "In my experiment, directly binding IndicWav2Vec2 to Yellowking's decoder has led to some problems, that is the vocabularie sizes have not matched The former is 87 and the latter is 112. How can I solve it? Thanks!", "votes": null }, { "id": "2372494", "postDate": "08/03/2023 17:58:12", "content": "Is this OK by just setting ignore_mismatched_sizes=True
There may be some problems, because the vocab.json of indicwav2vec_v1_bengali and other models (bengali, with LM) are not the same. I tried rewriting the alphabet.json and the model infers some nonsense.
this is my code
\n transformers pipeline\n transformers Wav2Vec2CTCTokenizer, Wav2Vec2FeatureExtractor, Wav2Vec2Processor,Wav2Vec2ProcessorWithLM\n transformers Wav2Vec2ForCTC\n\n\npretrain_model = \\\n \nvocab_dir = \\\n \narpa_file= \\\n \n \n\n\n\nsampling_rate=\ntokenizer = Wav2Vec2CTCTokenizer.from_pretrained(\n vocab_dir, \n unk_token=,\n pad_token=,\n word_delimiter_token=, \n bos_token=,\n eos_token=,\n)\n\n\n\nfeature_extractor = Wav2Vec2FeatureExtractor(\n feature_size=,\n sampling_rate=sampling_rate,\n padding_value=,\n padding_side=,\n do_normalize=,\n return_attention_mask=,\n)\n\n :\n pyctcdecode BeamSearchDecoderCTC\n pyctcdecode build_ctcdecoder\n\n vocab_dict = tokenizer.get_vocab()\n\n vocab_dict = {k: v k, v (vocab_dict.items(), key= item: item[])}\n decoder = build_ctcdecoder(\n labels=(vocab_dict.keys()),\n kenlm_model_path=arpa_file,\n )\n\nprocessor = Wav2Vec2ProcessorWithLM(\n feature_extractor=feature_extractor,\n tokenizer=tokenizer,\n decoder=decoder,\n)\n\n\n (nn.Module):\n ():\n ().__init__()\n self.output_type = []\n self.model = Wav2Vec2ForCTC.from_pretrained(\n pretrain_model,\n ignore_mismatched_sizes=,\n attention_dropout= ,\n hidden_dropout= ,\n feat_proj_dropout= ,\n mask_time_prob= ,\n layerdrop= ,\n ctc_loss_reduction=,\n pad_token_id=processor.tokenizer.pad_token_id,\n vocab_size=(processor.tokenizer),\n )\n \n ():\n B,L = batch[].shape\n\n \n out = self.model (\n batch[],\n batch[],\n labels = batch.get(,),\n )\n\n\n output = {}\n self.output_type:\n output[] = out.logits\n \n \n output[] = postprocess_lm_to_text(out, batch[])\n \n\n self.output_type:\n output[] = out.loss\n\n output\n\n\n ():\n B = (out.logits)\n \n logit = out.logits\n token = logit.argmax(dim=-)\n text = []\n b (B):\n t = token[b][:length[b]//]\n t = tokenizer.decode(t, skip_special_tokens=)\n text.append(t)\n text\n\n ():\n B = (out.logits)\n \n logit = out.logits.data.cpu().numpy()\n text = []\n b (B):\n l = logit[b]\n beam = decoder.decode_beams(l, beam_width=)\n t = beam[][]\n text.append(t)\n text\nLM model and wave2vec models are independent.
\ni have been mixing different wav2vec and LM from different sources
i also tried Flashlight decoder, and it would also work
", "rawMarkdown": "LM model and wave2vec models are independent.\ni have been mixing different wav2vec and LM from different sources\n\ni also tried Flashlight decoder, and it would also work", "votes": null }, { "id": "2381973", "postDate": "08/09/2023 13:44:30", "content": "Thanks a lot for your detailed help! I'm really appreciating that
", "rawMarkdown": "Thanks a lot for your detailed help! I'm really appreciating that", "votes": null }, { "id": "2438832", "postDate": "09/14/2023 14:10:06", "content": "Thank you
", "rawMarkdown": "Thank you", "votes": null } ], "comments": [ { "id": 2350184, "author_name": "hengck23", "author_url": "", "post_date": "07/19/2023 04:54:18", "content": "some important todo list:
\nThanks for posting all the insights. May I ask how long does it take you to run whsiper for leaderboard submssion?
", "votes": null, "replies": [] } ] }, { "id": 2350187, "author_name": "hengck23", "author_url": "", "post_date": "07/19/2023 05:00:08", "content": "the trick to winning is really the OOD (out of distribution) private test set
\n
Note:
\n
you should read the paper very carefully
", "votes": null, "replies": [ { "id": 2351573, "author_name": "hengck23", "author_url": "", "post_date": "07/20/2023 08:40:56", "content": "using lanuage model is another trick
\nhttps://huggingface.co/arijitx/wav2vec2-xls-r-300m-bengali
https://huggingface.co/blog/wav2vec2-with-ngram
\n language model :\n\n: .\n: .\n gram language model trained M sentences randomly chosen from AI4Bharat IndicCorp dataset :\n\n: .\n: .\npapers related to out-of-distribution and asr:
\ni think i find some OOD dataset:
\nhttps://github.com/Open-Speech-EkStep/ULCA-asr-dataset-corpus
\ne.g. youtube data
\nEntertainment Mirchi_Bangla_1 Labelled Mirchi_Bangla_1 54 (hrs)
\nEntertainment Mirchi_Bangla_2 Labelled Mirchi_Bangla_2
\nEntertainment Mirchi_Bangla_3 Labelled Mirchi_Bangla_3
\nGeneral CTVN_AKD_PLUS_1 Labelled CTVN_AKD_PLUS_1 21.7 (hrs)
\nGeneral CTVN_AKD_PLUS_2 Labelled CTVN_AKD_PLUS_2
however, the data are not available ….
\nbut the pretrain model are here
\nhttps://github.com/Open-Speech-EkStep/vakyansh-models
\nVakyansh-Conformer-SSL
\"This model was pre-trained using Nemo toolkit with 34,000 hours unlabeled audio in 39 Indian languages. This includes 15,000 hours of news recordings available on the internet, 10,000 hours of YouTube audios and other audio data.\"
\nmodel config yaml:
\ntrain_ds:
\n manifest_filepath: /mnt/lustre/megh/indic-ssl/manifest/crisil_noa_nptel_yt_indic_ssl_train.json
i am guessing
\nnoa : news on air
\nyt: youtube
paper:
\nVakyansh: ASR Toolkit for Low Resource Indic languages
\nhttps://arxiv.org/pdf/2203.16512.pdf
sampling rate as TTA?
\nsampling_rate=\n\n : one\n d = valid_df\n mp3_file = f\n\n = (mp3_file) \n with (mp3_file, ) as f:\n bpayload = f()\n = ( bpayload , sampling_rate)\n = (a) \n (, p)\n (, d.sentence)\n Hz\npredict ও বলেছে আপনার টিকাপ\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict ওবলেছে আপনার টিকা\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict এগুন শ অ\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict অবিচয় আনাটি\ntruth ও বলেছে আপনার ঠিকানা!\n\n Hz\npredict ও বলেছে আপনার টিকা\ntruth ও বলেছে আপনার ঠিকানা!\nthis youtube channel has many tutorials:
\nhttps://www.youtube.com/@ai4bharat/videos
\n
e.g
\nhttps://www.youtube.com/watch?v=iSyipKKgleo
visualise distance edit
\n
https://github.com/ukiuki-satoshi/visedit
", "votes": null, "replies": [] }, { "id": 2356341, "author_name": "hengck23", "author_url": "", "post_date": "07/24/2023 05:36:49", "content": "
\n
\n
\n
i made a mistake. i forget about normalisation and others in post-processing
", "votes": null, "replies": [] } ] }, { "id": 2356475, "author_name": "hengck23", "author_url": "", "post_date": "07/24/2023 08:14:01", "content": "
OOD results
", "votes": null, "replies": [ { "id": 2359089, "author_name": "nyleve", "author_url": "", "post_date": "07/26/2023 01:41:46", "content": "@hengck23 How did you get the ground truth for the test_example to calculate the wer ? It's not available in the Data section
\nEDIT: sorry, I've found the source https://www.kaggle.com/competitions/bengaliai-speech/discussion/425932
", "votes": null, "replies": [] } ] }, { "id": 2356515, "author_name": "hengck23", "author_url": "", "post_date": "07/24/2023 08:56:36", "content": "\" We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type\"
\nhttps://arxiv.org/pdf/2307.03183.pdf
\nWhisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
End-to-end Music-mixed Speech Recognition
\nhttps://arxiv.org/pdf/2008.12048.pdf
Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition
", "votes": null, "replies": [ { "id": 2358541, "author_name": "hengck23", "author_url": "", "post_date": "07/25/2023 15:23:32", "content": "CTC bert?
\n
https://sites.google.com/iitdh.ac.in/vssasr2021/resources
\n
massive external data:
\nhttps://github.com/AI4Bharat/vistaar#download-training-datasets-and-benchmarks
yet another external data
\nhttps://globalrecordings.net
paper on effects of in and out domain :
\nROBUST WAV2VEC 2.0: ANALYZING DOMAIN SHIFT IN SELF-SUPERVISED PRE-TRAINING
\nhttps://arxiv.org/pdf/2104.01027.pdf
\"In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain\"
", "votes": null, "replies": [] }, { "id": 2365899, "author_name": "hengck23", "author_url": "", "post_date": "07/30/2023 14:44:37", "content": "i wonder if the each sentence in the test is unique?
\nif not, this is a leak?
\n(e.g. one may collect speech data by asking different people to speak the same sentence)
whisper takes in 30sec audio chunk. so i did a probe on audio clip duration of hidden test data:
\nvalid_df = pd()\nmp3_dir = f\n\nduration=\n t,d valid_df():\n audio_file = f\n\n sampling_rate = _000\n with (audio_file, ) as f:\n bpayload = f()\n = (bpayload, sampling_rate)\n dur = (a)/sampling_rate\n (dur)\n duration(dur)\n\nmax_duration=np(duration)\n\n\nsubmission failed for : max_duration<=30 sec
\nsubmission passed for : max_duration<=32 sec
Thank you for the info … so much research on the topic great …all the best
", "votes": null, "replies": [] }, { "id": 2369358, "author_name": "hengck23", "author_url": "", "post_date": "08/01/2023 17:34:50", "content": "another dataset
\nDataset - RESPIN
\nhttps://slt2022.org/projects/13%20-%20Dialectical%20speech%20recognition%20for%20two%20Indian%20languages%20-%20Bengali%20and%20Bhojpuri/13%20-%20Dialectical%20speech%20recognition%20for%20two%20Indian%20languages%20-%20Bengali%20and%20Bhojpuri.pdf
https://sites.google.com/view/respinasrchallenge2023/home
\n
code switch dataset:
\nhttps://github.com/navana-tech/baseline_recipe_is21s_indic_asr_challenge
Hi, is indicwav2vec_v1_bengali simply transformed from the fairseq model on indicwav2vec github repo?
", "votes": null, "replies": [ { "id": 2370496, "author_name": "hengck23", "author_url": "", "post_date": "08/02/2023 13:09:32", "content": "i have submitted the version from hugging face.
\ni may choose to use the github version later
", "votes": null, "replies": [] }, { "id": 2370590, "author_name": "aphysict", "author_url": "", "post_date": "08/02/2023 14:04:55", "content": "
\nThey are the same model
In my experiment, directly binding IndicWav2Vec2 to Yellowking's decoder has led to some problems, that is the vocabularie sizes have not matched The former is 87 and the latter is 112. How can I solve it? Thanks!
", "votes": null, "replies": [ { "id": 2372494, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "08/03/2023 17:58:12", "content": "Is this OK by just setting ignore_mismatched_sizes=True
There may be some problems, because the vocab.json of indicwav2vec_v1_bengali and other models (bengali, with LM) are not the same. I tried rewriting the alphabet.json and the model infers some nonsense.
this is my code
\n transformers pipeline\n transformers Wav2Vec2CTCTokenizer, Wav2Vec2FeatureExtractor, Wav2Vec2Processor,Wav2Vec2ProcessorWithLM\n transformers Wav2Vec2ForCTC\n\n\npretrain_model = \\\n \nvocab_dir = \\\n \narpa_file= \\\n \n \n\n\n\nsampling_rate=\ntokenizer = Wav2Vec2CTCTokenizer.from_pretrained(\n vocab_dir, \n unk_token=,\n pad_token=,\n word_delimiter_token=, \n bos_token=,\n eos_token=,\n)\n\n\n\nfeature_extractor = Wav2Vec2FeatureExtractor(\n feature_size=,\n sampling_rate=sampling_rate,\n padding_value=,\n padding_side=,\n do_normalize=,\n return_attention_mask=,\n)\n\n :\n pyctcdecode BeamSearchDecoderCTC\n pyctcdecode build_ctcdecoder\n\n vocab_dict = tokenizer.get_vocab()\n\n vocab_dict = {k: v k, v (vocab_dict.items(), key= item: item[])}\n decoder = build_ctcdecoder(\n labels=(vocab_dict.keys()),\n kenlm_model_path=arpa_file,\n )\n\nprocessor = Wav2Vec2ProcessorWithLM(\n feature_extractor=feature_extractor,\n tokenizer=tokenizer,\n decoder=decoder,\n)\n\n\n (nn.Module):\n ():\n ().__init__()\n self.output_type = []\n self.model = Wav2Vec2ForCTC.from_pretrained(\n pretrain_model,\n ignore_mismatched_sizes=,\n attention_dropout= ,\n hidden_dropout= ,\n feat_proj_dropout= ,\n mask_time_prob= ,\n layerdrop= ,\n ctc_loss_reduction=,\n pad_token_id=processor.tokenizer.pad_token_id,\n vocab_size=(processor.tokenizer),\n )\n \n ():\n B,L = batch[].shape\n\n \n out = self.model (\n batch[],\n batch[],\n labels = batch.get(,),\n )\n\n\n output = {}\n self.output_type:\n output[] = out.logits\n \n \n output[] = postprocess_lm_to_text(out, batch[])\n \n\n self.output_type:\n output[] = out.loss\n\n output\n\n\n ():\n B = (out.logits)\n \n logit = out.logits\n token = logit.argmax(dim=-)\n text = []\n b (B):\n t = token[b][:length[b]//]\n t = tokenizer.decode(t, skip_special_tokens=)\n text.append(t)\n text\n\n ():\n B = (out.logits)\n \n logit = out.logits.data.cpu().numpy()\n text = []\n b (B):\n l = logit[b]\n beam = decoder.decode_beams(l, beam_width=)\n t = beam[][]\n text.append(t)\n text\nLM model and wave2vec models are independent.
\ni have been mixing different wav2vec and LM from different sources
i also tried Flashlight decoder, and it would also work
", "votes": null, "replies": [ { "id": 2381973, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "08/09/2023 13:44:30", "content": "Thanks a lot for your detailed help! I'm really appreciating that
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2438832, "author_name": "barnobarno", "author_url": "", "post_date": "09/14/2023 14:10:06", "content": "Thank you
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2350183": "#more details later, ... here is the lb0.481 recipe:\n#[a]model: \nhttps://huggingface.co/ai4bharat/indicwav2vec_v1_bengali (CTC model only)\n\n#[b]decoder: \nadd a LM yourself, you can train one or use e.g. https://huggingface.co/shahruk10/wav2vec2-xls-r-300m-bengali-commonvoice\n\n#[c]post-process:\nnormalise + dari\n\n```\nsome lbscore:\n[a]+[b]+[c]: 0.481\n[a]+[c]: 0.520\n[a]+[b]: 0.550\n[a]: 0.585\n[a]+[b]+[c/only normalise]: 0.490\n```\n\n----\n\n\n.png?generation=1689742315359191&alt=media)\n\nhttps://www.kaggle.com/code/hengck23/local-wer-0-2600-nemo-baseline-conformer\ninitial results .... to be updated as experiments progress", "2350184": "some important todo list:\n- speed up whsiper inference (e.g. JAX or openAI C api)", "2350187": "the trick to winning is really the OOD (out of distribution) private test set\n\n.png?generation=1689742664897606&alt=media)\n\nNote:\n1. from https://www.kaggle.com/competitions/bengaliai-speech/data\nThe full test set contains about 20 hours of speech in almost 8000 MP3 audio files, public LB is 46% of the test data\n2. from dataset paper\n- OOD = 2681\n- Macro Test = 4872\n- all = 7553", "2350194": ".png?generation=1689743292178787&alt=media)\n\nyou should read the paper very carefully", "2350229": "papers related to out-of-distribution and asr:\n- to be updated", "2351573": "using lanuage model is another trick\nhttps://huggingface.co/arijitx/wav2vec2-xls-r-300m-bengali\n\nhttps://huggingface.co/blog/wav2vec2-with-ngram\n```\nWithout language model :\n\nWER: 0.21726385291857586\nCER: 0.04725010353701041\nWith 5 gram language model trained on 30M sentences randomly chosen from AI4Bharat IndicCorp dataset :\n\nWER: 0.15322879016421437\nCER: 0.03413696666806267\n\n```", "2353514": "i think i find some OOD dataset:\n\nhttps://github.com/Open-Speech-EkStep/ULCA-asr-dataset-corpus\n\ne.g. youtube data\nEntertainment\tMirchi_Bangla_1\tLabelled\tMirchi_Bangla_1\t54 (hrs)\nEntertainment\tMirchi_Bangla_2\tLabelled\tMirchi_Bangla_2\t\nEntertainment\tMirchi_Bangla_3\tLabelled\tMirchi_Bangla_3\t\nGeneral\tCTVN_AKD_PLUS_1\tLabelled\tCTVN_AKD_PLUS_1\t21.7 (hrs)\nGeneral\tCTVN_AKD_PLUS_2\tLabelled\tCTVN_AKD_PLUS_2\t\n\n\nhowever, the data are not available ....\nbut the pretrain model are here\nhttps://github.com/Open-Speech-EkStep/vakyansh-models\nVakyansh-Conformer-SSL\n\n\"This model was pre-trained using Nemo toolkit with 34,000 hours unlabeled audio in 39 Indian languages. This includes 15,000 hours of news recordings available on the internet, 10,000 hours of YouTube audios and other audio data.\"\n\nmodel config yaml:\ntrain_ds:\n manifest_filepath: /mnt/lustre/megh/indic-ssl/manifest/crisil_noa_nptel_yt_indic_ssl_train.json\n\ni am guessing\nnoa : news on air\nyt: youtube\n\n\npaper: \nVakyansh: ASR Toolkit for Low Resource Indic languages\nhttps://arxiv.org/pdf/2203.16512.pdf", "2353528": "sampling rate as TTA?\n\n```\nsampling_rate=16000\n\nif 1: #test one\n\td = valid_df.iloc[0]\n\tmp3_file = f'{mp3_dir}/{d[\"id\"]}.mp3'\n\n\t#p = pipe(mp3_file)['text'] \n\twith open(mp3_file, 'rb') as f:\n\t\tbpayload = f.read()\n\ta = ffmpeg_read( bpayload , sampling_rate)\n\tp = pipe(a)['text'] \n\tprint('predict', p)\n\tprint('truth ', d.sentence)\n\n```\n\n```\n18000 Hz\npredict ও বলেছে আপনার টিকাপ\ntruth ও বলেছে আপনার ঠিকানা!\n\n16000 Hz\npredict ওবলেছে আপনার টিকা\ntruth ও বলেছে আপনার ঠিকানা!\n\n32000 Hz\npredict এগুন শ অ\ntruth ও বলেছে আপনার ঠিকানা!\n\n14000 Hz\npredict অবিচয় আনাটি\ntruth ও বলেছে আপনার ঠিকানা!\n\n20000 Hz\npredict ও বলেছে আপনার টিকা\ntruth ও বলেছে আপনার ঠিকানা!\n```", "2353969": "this youtube channel has many tutorials:\n\nhttps://www.youtube.com/@ai4bharat/videos\n\n.png?generation=1690008184783299&alt=media)\n\ne.g\nhttps://www.youtube.com/watch?v=iSyipKKgleo", "2355813": "visualise distance edit\n\n.png?generation=1690128936873031&alt=media)\n\nhttps://github.com/ukiuki-satoshi/visedit", "2356341": ".png?generation=1690403372321495&alt=media)\n\n\n~~should we be using kaggle dataset \"as it is\" at all?~~\n~~i note that the baseline conformer-CTC has good results on train.csv valid split (clean label). But it only has LB0.68.~~\n~~On the other hands, public huggingface/ai4bharat/indicwav2vec_v1_bengali only trained on their datase ~~~~(radio boradcast and youtube) has less performance gap: valid split 0.39, LB0.52.~~\n\n~~This makes me suspect kaggle data is not good on hidden OOD public dataset.~~\n~~In my experiments abovem, the better results you get for train.csv valid split (or train split), you get worse ~~~~results for publuc LB.~~\n\n~~Kagglers may want to confirm this observation~~", "2356475": ".png?generation=1690186435573149&alt=media)\n\nOOD results", "2356515": "\" We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type\"\n\nhttps://arxiv.org/pdf/2307.03183.pdf\nWhisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers", "2356526": "End-to-end Music-mixed Speech Recognition\nhttps://arxiv.org/pdf/2008.12048.pdf", "2356542": ".png?generation=1690189696857157&alt=media)\n\nWav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition", "2356736": "https://sites.google.com/iitdh.ac.in/vssasr2021/resources\n.png?generation=1690195926421740&alt=media)", "2358541": "CTC bert?\n\n.png?generation=1690298599425663&alt=media)\n\n.png?generation=1690298610687049&alt=media)", "2359089": "hengck23 How did you get the ground truth for the test_example to calculate the wer ? It's not available in the Data section \n\nEDIT: sorry, I've found the source https://www.kaggle.com/competitions/bengaliai-speech/discussion/425932", "2359216": "massive external data:\nhttps://github.com/AI4Bharat/vistaar#download-training-datasets-and-benchmarks", "2360593": "i made a mistake. i forget about normalisation and others in post-processing", "2360720": "paper on effects of in and out domain :\nROBUST WAV2VEC 2.0: ANALYZING DOMAIN SHIFT IN SELF-SUPERVISED PRE-TRAINING\nhttps://arxiv.org/pdf/2104.01027.pdf\n\n\"In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain\"", "2360878": "yet another external data\nhttps://globalrecordings.net", "2365899": "i wonder if the each sentence in the test is unique?\nif not, this is a leak?\n(e.g. one may collect speech data by asking different people **to speak the same sentence**)", "2367644": "Thanks for posting all the insights. May I ask how long does it take you to run whsiper for leaderboard submssion?", "2368194": "whisper takes in 30sec audio chunk. so i did a probe on audio clip duration of hidden test data:\n```\nvalid_df = pd.read_csv('/kaggle/input/bengaliai-speech/sample_submission.csv')\nmp3_dir = f'/kaggle/input/bengaliai-speech/test_mp3s'\n\nduration=[]\nfor t,d in valid_df.iterrows():\n audio_file = f'{mp3_dir}/{d[\"id\"]}.mp3'\n\n sampling_rate = 16_000\n with open(audio_file, 'rb') as f:\n bpayload = f.read()\n a = ffmpeg_read(bpayload, sampling_rate)\n dur = len(a)/sampling_rate\n print(dur)\n duration.append(dur)\n\nmax_duration=np.max(duration)\nprint('max_duration',max_duration)\nassert(max_duration<=DURATION)\n\n```\n\nsubmission failed for : max_duration<=30 sec\nsubmission passed for : max_duration<=32 sec", "2368354": "Thank you for the info ... so much research on the topic great ...all the best", "2369358": "another dataset\nDataset - RESPIN\nhttps://slt2022.org/projects/13%20-%20Dialectical%20speech%20recognition%20for%20two%20Indian%20languages%20-%20Bengali%20and%20Bhojpuri/13%20-%20Dialectical%20speech%20recognition%20for%20two%20Indian%20languages%20-%20Bengali%20and%20Bhojpuri.pdf\n\nhttps://sites.google.com/view/respinasrchallenge2023/home\n.png?generation=1690911460374350&alt=media)", "2369394": "code switch dataset:\nhttps://github.com/navana-tech/baseline_recipe_is21s_indic_asr_challenge", "2370469": "Hi, is indicwav2vec_v1_bengali simply transformed from the fairseq model on indicwav2vec github repo?", "2370496": "i have submitted the version from hugging face.\n\ni may choose to use the github version later", "2370590": "~~I try the fairseq model a little, it's fast but the wer is high. Maybe they are not the same model, or I did something wrong. ~~\nThey are the same model", "2372470": "In my experiment, directly binding IndicWav2Vec2 to Yellowking's decoder has led to some problems, that is the vocabularie sizes have not matched The former is 87 and the latter is 112. How can I solve it? Thanks!", "2372494": "Is this OK by just setting `ignore_mismatched_sizes=True`", "2379732": "There may be some problems, because the vocab.json of `indicwav2vec_v1_bengali` and other models (bengali, with LM) are not the same. I tried rewriting the alphabet.json and the model infers some nonsense.", "2381841": "this is my code\n\n```\nfrom transformers import pipeline\nfrom transformers import Wav2Vec2CTCTokenizer, Wav2Vec2FeatureExtractor, Wav2Vec2Processor,Wav2Vec2ProcessorWithLM\nfrom transformers import Wav2Vec2ForCTC\n\n\npretrain_model = \\\n '/kaggle/input/ai4bharat-indicwav2vec-v1-bengali'\nvocab_dir = \\\n '/kaggle/input/ai4bharat-indicwav2vec-v1-bengali'\narpa_file= \\\n '/kaggle/input/my-weight-bengali-asr-01/my-lm-5gram.arpa' #or binary file\n #'/kaggle/input/my-weight-bengali-asr-01/language_model/commonvoice-bn.5.arpa'\n\n##########################################################################\n#model \nsampling_rate=16_000\ntokenizer = Wav2Vec2CTCTokenizer.from_pretrained(\n vocab_dir, #'my_tokenizer',\n unk_token='