![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1310697%2F353eddf6b198743d96e68da92839dfd4%2FScreenshot%202023-07-31%20at%2022-23-37%20whisper.pdf.png?generation=1690822464550409&alt=media\")
When I read through the Whisper paper ie Robust Speech Recognition via Large-Scale Weak Supervision by OpenAI team. I noticed the performance of bengali is really bad based on bench-marking results in two datasets - Common Voice9 and Fleurs.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1310697%2Fc25dc14105013abbc40adcd6598a5687%2FScreenshot%202023-07-31%20at%2022-23-57%20whisper.pdf.png?generation=1690822545094901&alt=media\")
I suspect because of this directly using whisper model weights might be a bad idea. Yet fine-tuning whisper might be promising as in low-resource languages like Malayalam we have been able to fine tune and reach an WER of below 0.1 in Common Voice 11 dataset.
", "messages": [ { "id": "2367609", "postDate": "07/31/2023 16:58:12", "content": "When I read through the Whisper paper ie Robust Speech Recognition via Large-Scale Weak Supervision by OpenAI team. I noticed the performance of bengali is really bad based on bench-marking results in two datasets - Common Voice9 and Fleurs.
\n
I suspect because of this directly using whisper model weights might be a bad idea. Yet fine-tuning whisper might be promising as in low-resource languages like Malayalam we have been able to fine tune and reach an WER of below 0.1 in Common Voice 11 dataset.
", "rawMarkdown": "When I read through the Whisper paper ie Robust Speech Recognition via Large-Scale Weak Supervision by OpenAI team. I noticed the performance of bengali is really bad based on bench-marking results in two datasets - Common Voice9 and Fleurs.\n\n- COMMON VOICE 9\n\n\n\n- FLEURS\n\n\n\n\nI suspect because of this directly using whisper model weights might be a bad idea. Yet fine-tuning whisper might be promising as in low-resource languages like Malayalam we have been able to fine tune and reach an WER of below 0.1 in Common Voice 11 dataset.", "votes": null }, { "id": "2367622", "postDate": "07/31/2023 17:03:25", "content": "this is very interesting to see, thanks
", "rawMarkdown": "this is very interesting to see, thanks", "votes": null }, { "id": "2367746", "postDate": "07/31/2023 19:07:19", "content": "it depends on your training data, check this instead:
\nhttps://github.com/AI4Bharat/vistaar
the downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)
", "rawMarkdown": "it depends on your training data, check this instead:\nhttps://github.com/AI4Bharat/vistaar\n\nthe downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)", "votes": null }, { "id": "2369253", "postDate": "08/01/2023 16:08:20", "content": "Thanks for sharing about vistaar. TIL moment.
\n\n\nthe downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)
\n
I agree the inference of whisper is pretty slow. Yet faster-whisper project kind of makes the inference lot more faster now.
", "rawMarkdown": "Thanks for sharing about vistaar. TIL moment.\n\n> the downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)\n\nI agree the inference of whisper is pretty slow. Yet [faster-whisper](https://github.com/guillaumekln/faster-whisper) project kind of makes the inference lot more faster now.", "votes": null }, { "id": "2369333", "postDate": "08/01/2023 17:05:55", "content": "after my experiments, here are the good news:
\nhuggingface whisper-medium:
\ni think whisper-medium may performance better than wav2vec-large?
\ni tried faster-whisper, but there is some loss of accuracy (maybe there is a bug in my code). it is faster but cannot do batching? there is another wispherX = faster-whisper + batch
\nhttps://github.com/m-bain/whisperX
do note that most of SOTA asr benchmark you found in papers and github may be different from our kaggle competition.
\nFor example, the punctuation are removed from ground truth and text normalization/capitalization are applied.
actually the trick to win this competition may be these post processing, e.g.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fb60f4f52a147ceb769724e5d0c067965%2FSelection_999(2830).png?generation=1690911869019500&alt=media\")
another whisper results
", "rawMarkdown": ".png?generation=1690911869019500&alt=media)\n\nanother whisper results", "votes": null }, { "id": "2369372", "postDate": "08/01/2023 17:46:48", "content": "Under review as a conference paper at ICLR 2023
\nESC: A BENCHMARK FOR MULTI-DOMAIN END-TO-END SPEECH RECOGNITION
\nhttps://openreview.net/pdf?id=9OL2fIfDLK
yet another whisper results
", "rawMarkdown": "Under review as a conference paper at ICLR 2023\nESC: A BENCHMARK FOR MULTI-DOMAIN END-TO-END SPEECH RECOGNITION\nhttps://openreview.net/pdf?id=9OL2fIfDLK\n\nyet another whisper results", "votes": null }, { "id": "2369657", "postDate": "08/01/2023 23:13:09", "content": "I tried submitting whisper-medium using the same code that worked for whisper-small and it failed with a scoring error. Were you able to get it submit fully?
", "rawMarkdown": "I tried submitting whisper-medium using the same code that worked for whisper-small and it failed with a scoring error. Were you able to get it submit fully?", "votes": null }, { "id": "2369763", "postDate": "08/02/2023 02:36:33", "content": "scoring error probably due to this
\nhttps://www.kaggle.com/competitions/bengaliai-speech/discussion/425942
Interesting! I thought punctuation was not taken into account when calculating WER. It is a pity that this is not specified anywhere in the description.
", "rawMarkdown": "Interesting! I thought punctuation was not taken into account when calculating WER. It is a pity that this is not specified anywhere in the description.", "votes": null } ], "comments": [ { "id": 2367622, "author_name": "anivana", "author_url": "", "post_date": "07/31/2023 17:03:25", "content": "this is very interesting to see, thanks
", "votes": null, "replies": [] }, { "id": 2367746, "author_name": "hengck23", "author_url": "", "post_date": "07/31/2023 19:07:19", "content": "it depends on your training data, check this instead:
\nhttps://github.com/AI4Bharat/vistaar
the downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)
", "votes": null, "replies": [ { "id": 2369253, "author_name": "kurianbenoy", "author_url": "", "post_date": "08/01/2023 16:08:20", "content": "Thanks for sharing about vistaar. TIL moment.
\n\n\nthe downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)
\n
I agree the inference of whisper is pretty slow. Yet faster-whisper project kind of makes the inference lot more faster now.
", "votes": null, "replies": [ { "id": 2369333, "author_name": "hengck23", "author_url": "", "post_date": "08/01/2023 17:05:55", "content": "after my experiments, here are the good news:
\nhuggingface whisper-medium:
\ni think whisper-medium may performance better than wav2vec-large?
\ni tried faster-whisper, but there is some loss of accuracy (maybe there is a bug in my code). it is faster but cannot do batching? there is another wispherX = faster-whisper + batch
\nhttps://github.com/m-bain/whisperX
another whisper results
", "votes": null, "replies": [ { "id": 2369372, "author_name": "hengck23", "author_url": "", "post_date": "08/01/2023 17:46:48", "content": "Under review as a conference paper at ICLR 2023
\nESC: A BENCHMARK FOR MULTI-DOMAIN END-TO-END SPEECH RECOGNITION
\nhttps://openreview.net/pdf?id=9OL2fIfDLK
yet another whisper results
", "votes": null, "replies": [ { "id": 2369657, "author_name": "nbroad", "author_url": "", "post_date": "08/01/2023 23:13:09", "content": "I tried submitting whisper-medium using the same code that worked for whisper-small and it failed with a scoring error. Were you able to get it submit fully?
", "votes": null, "replies": [ { "id": 2369763, "author_name": "hengck23", "author_url": "", "post_date": "08/02/2023 02:36:33", "content": "scoring error probably due to this
\nhttps://www.kaggle.com/competitions/bengaliai-speech/discussion/425942
do note that most of SOTA asr benchmark you found in papers and github may be different from our kaggle competition.
\nFor example, the punctuation are removed from ground truth and text normalization/capitalization are applied.
actually the trick to win this competition may be these post processing, e.g.
\nInteresting! I thought punctuation was not taken into account when calculating WER. It is a pity that this is not specified anywhere in the description.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2367609": "When I read through the Whisper paper ie Robust Speech Recognition via Large-Scale Weak Supervision by OpenAI team. I noticed the performance of bengali is really bad based on bench-marking results in two datasets - Common Voice9 and Fleurs.\n\n- COMMON VOICE 9\n\n\n\n- FLEURS\n\n\n\n\nI suspect because of this directly using whisper model weights might be a bad idea. Yet fine-tuning whisper might be promising as in low-resource languages like Malayalam we have been able to fine tune and reach an WER of below 0.1 in Common Voice 11 dataset.", "2367622": "this is very interesting to see, thanks", "2367746": "it depends on your training data, check this instead:\nhttps://github.com/AI4Bharat/vistaar\n\nthe downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)", "2369253": "Thanks for sharing about vistaar. TIL moment.\n\n> the downside of whisper i think is its speed and( unexpected behavior which results in erroneous prediction for some few simple cases)\n\nI agree the inference of whisper is pretty slow. Yet [faster-whisper](https://github.com/guillaumekln/faster-whisper) project kind of makes the inference lot more faster now.", "2369333": "after my experiments, here are the good news:\n\nhuggingface whisper-medium:\n- on old pascal titan x gpu (12 gb) : batch=8 takes about 2 hr for 8000 mp3s files.\n- since whisper pad all audio input to 30sec, i think we can use whisper in submission.\n- i also probe that hidden test mp3s are mostly less than 30sec, maybe with a few within 30 to32 sec\n\ni think whisper-medium may performance better than wav2vec-large?\n\n---\n\ni tried faster-whisper, but there is some loss of accuracy (maybe there is a bug in my code). it is faster but cannot do batching? there is another wispherX = faster-whisper + batch\nhttps://github.com/m-bain/whisperX", "2369346": "do note that most of SOTA asr benchmark you found in papers and github may be different from our kaggle competition.\nFor example, the punctuation are removed from ground truth and text normalization/capitalization are applied.\n\nactually the trick to win this competition may be these post processing, e.g.\n1. according to my statistics, test mp3 has average length of e.g. 5sec, with e.g. 10 words.\n2. there is on average 1.5 punctuation per clip.\n3. the bari (full stop is attached to the last word) and so if you don't detect it. it would be a substitution error in WER.\n4. even if you detect all words perfectly without punctuation marks, you have a WER of 0.15.\n5. just by building good punctuation model (either within your model or as good post-processing), you probably gain +0.05 to 0.10 in LB score", "2369370": ".png?generation=1690911869019500&alt=media)\n\nanother whisper results", "2369372": "Under review as a conference paper at ICLR 2023\nESC: A BENCHMARK FOR MULTI-DOMAIN END-TO-END SPEECH RECOGNITION\nhttps://openreview.net/pdf?id=9OL2fIfDLK\n\nyet another whisper results", "2369657": "I tried submitting whisper-medium using the same code that worked for whisper-small and it failed with a scoring error. Were you able to get it submit fully?", "2369763": "scoring error probably due to this\nhttps://www.kaggle.com/competitions/bengaliai-speech/discussion/425942", "2369855": "Interesting! I thought punctuation was not taken into account when calculating WER. It is a pity that this is not specified anywhere in the description." }, "source": "meta" }