{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# \n\nThe competition dataset comprises about 1200 hours of recordings of Bengali speech. Your goal is to transcribe recordings of speech that is out-of-distribution with respect to the training set.\n\nNote that this is a Code Competition, in which the actual test set is hidden. In this public version, we give some sample data in the correct format to help you author your solutions. The full test set contains about 20 hours of speech in almost 8000 MP3 audio files. All of the files in the test set are encoded at a sample rate of 32k, a bit rate of 48k, in one channel.\n\nDetails on the dataset are available in the dataset paper: https://arxiv.org/abs/2305.09688","metadata":{}},{"cell_type":"markdown","source":"# Files and Field Descriptions:
Data Exploration and Preprocessing:
Harmonic and percussive components with transparency
Plotting a transposed wave along with a self-similarity matrix
Tempogram - Onset autocorrelation
Construct a standard onset function
\"))\n \n# Finally, show the plot\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-25T05:59:17.323231Z","iopub.execute_input":"2023-07-25T05:59:17.324021Z","iopub.status.idle":"2023-07-25T05:59:57.180056Z","shell.execute_reply.started":"2023-07-25T05:59:17.323983Z","shell.execute_reply":"2023-07-25T05:59:57.178950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# \n\nWe will load the test-clean split of the BengaliSpeech corpus using torchaudio.","metadata":{}},{"cell_type":"code","source":"class BengaliSpeech(torch.utils.data.Dataset):\n \"\"\"\n A simple class to wrap BengaliSpeech and trim/pad the audio to 30 seconds.\n It will drop the last few seconds of a very small portion of the utterances.\n \"\"\"\n def __init__(self, split=\"test-clean\", device=DEVICE):\n self.dataset = torchaudio.datasets.LIBRISPEECH(\n root=os.path.expanduser(\"~/.cache\"),\n url=split,\n download=True,\n )\n self.device = device\n\n def __len__(self):\n return len(self.dataset)\n\n def __getitem__(self, item):\n audio, sample_rate, text, _, _, _ = self.dataset[item]\n assert sample_rate == 16000\n audio = whisper.pad_or_trim(audio.flatten()).to(self.device)\n mel = whisper.log_mel_spectrogram(audio)\n \n return (mel, text)","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:00:36.264032Z","iopub.execute_input":"2023-07-25T06:00:36.265074Z","iopub.status.idle":"2023-07-25T06:00:36.273698Z","shell.execute_reply.started":"2023-07-25T06:00:36.265033Z","shell.execute_reply":"2023-07-25T06:00:36.272286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = BengaliSpeech(\"test-clean\")\nloader = torch.utils.data.DataLoader(dataset, batch_size=16)","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:00:44.021726Z","iopub.execute_input":"2023-07-25T06:00:44.022224Z","iopub.status.idle":"2023-07-25T06:01:05.684843Z","shell.execute_reply.started":"2023-07-25T06:00:44.022185Z","shell.execute_reply":"2023-07-25T06:01:05.683795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = whisper.load_model(\"base.en\")\nprint(\n f\"Model is {'multilingual' if model.is_multilingual else 'English-only'} \"\n f\"and has {sum(np.prod(p.shape) for p in model.parameters()):,} parameters.\"\n)","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:01:15.272355Z","iopub.execute_input":"2023-07-25T06:01:15.273064Z","iopub.status.idle":"2023-07-25T06:01:19.230567Z","shell.execute_reply.started":"2023-07-25T06:01:15.273030Z","shell.execute_reply":"2023-07-25T06:01:19.229444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predict without timestamps for short-form transcription\noptions = whisper.DecodingOptions(language=\"en\", without_timestamps=True)","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:01:24.241368Z","iopub.execute_input":"2023-07-25T06:01:24.242040Z","iopub.status.idle":"2023-07-25T06:01:24.246897Z","shell.execute_reply.started":"2023-07-25T06:01:24.242001Z","shell.execute_reply":"2023-07-25T06:01:24.245828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hypotheses = []\nreferences = []\n\nfor mels, texts in tqdm(loader):\n results = model.decode(mels, options)\n hypotheses.extend([result.text for result in results])\n references.extend(texts)","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:01:29.317611Z","iopub.execute_input":"2023-07-25T06:01:29.318316Z","iopub.status.idle":"2023-07-25T06:04:27.833405Z","shell.execute_reply.started":"2023-07-25T06:01:29.318277Z","shell.execute_reply":"2023-07-25T06:04:27.832268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.DataFrame(dict(hypothesis=hypotheses, reference=references))\ndata","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:04:32.873737Z","iopub.execute_input":"2023-07-25T06:04:32.874141Z","iopub.status.idle":"2023-07-25T06:04:32.891781Z","shell.execute_reply.started":"2023-07-25T06:04:32.874109Z","shell.execute_reply":"2023-07-25T06:04:32.890598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# \n\nWord Error Rate (WER) is a metric used to evaluate the performance of Automatic Speech Recognition (ASR) systems or other systems that convert spoken language into written text, such as Optical Character Recognition (OCR) systems. WER is a common evaluation measure in the field of speech recognition.\n\nWER represents the percentage of words that are incorrectly recognized or transcribed by the ASR system compared to a reference or ground truth transcription of the same spoken input. It takes into account the substitutions, insertions, and deletions of words made by the ASR system.\n\nThe formula for calculating Word Error Rate is as follows:\n\nWER = (S + D + I) / N\n\nWhere:\n\n**S** = Number of word substitutions\n\n**D** = Number of word deletions\n\n**I** = Number of word insertions\n\n**N** = Total number of words in the reference (ground truth) transcription\n\nLet's break down the components:\n\n1. **Substitutions (S):** The number of words in the ASR output that are different from the reference transcription.\n\n2. **Deletions (D):** The number of words in the reference transcription that are missing from the ASR output.\n\n3. **Insertions (I):** The number of extra words in the ASR output that are not present in the reference transcription.\n\n4. **Total words (N):** The total number of words in the reference transcription.\n\nOnce the WER is calculated, it is expressed as a percentage to provide a more intuitive measure of the ASR system's accuracy. A lower WER indicates better performance, as it means the ASR system made fewer errors in transcribing the spoken language.\n\nFor example, if the reference transcription has 100 words and the ASR system outputs 8 substitutions, 5 deletions, and 3 insertions, the WER would be calculated as:\n\n**WER = (8 + 5 + 3) / 100 = 16 / 100 = 0.16 or 16%**\n\nThis means the ASR system's output has an error rate of 16%, i.e., 16% of the words in the ASR output are different from the reference transcription.\n","metadata":{}},{"cell_type":"markdown","source":"Next, we apply our English normalizer implementation to standardize the transcription and compute the Word Error Rate (WER).","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install jiwer","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:04:55.860037Z","iopub.execute_input":"2023-07-25T06:04:55.860462Z","iopub.status.idle":"2023-07-25T06:05:08.473628Z","shell.execute_reply.started":"2023-07-25T06:04:55.860428Z","shell.execute_reply":"2023-07-25T06:05:08.472094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import jiwer\nfrom whisper.normalizers import EnglishTextNormalizer\n\nnormalizer = EnglishTextNormalizer()","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:05:12.193727Z","iopub.execute_input":"2023-07-25T06:05:12.194163Z","iopub.status.idle":"2023-07-25T06:05:12.336755Z","shell.execute_reply.started":"2023-07-25T06:05:12.194129Z","shell.execute_reply":"2023-07-25T06:05:12.335703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[\"hypothesis_clean\"] = [normalizer(text) for text in data[\"hypothesis\"]]\ndata[\"reference_clean\"] = [normalizer(text) for text in data[\"reference\"]]\ndata","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:05:17.601851Z","iopub.execute_input":"2023-07-25T06:05:17.602254Z","iopub.status.idle":"2023-07-25T06:05:20.088529Z","shell.execute_reply.started":"2023-07-25T06:05:17.602223Z","shell.execute_reply":"2023-07-25T06:05:20.087519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wer = jiwer.wer(list(data[\"reference_clean\"]), list(data[\"hypothesis_clean\"]))\n\nprint(f\"WER: {wer * 100:.2f} %\")","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:05:45.253668Z","iopub.execute_input":"2023-07-25T06:05:45.254087Z","iopub.status.idle":"2023-07-25T06:05:45.414231Z","shell.execute_reply.started":"2023-07-25T06:05:45.254053Z","shell.execute_reply":"2023-07-25T06:05:45.413193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# \n\n","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv(\"/kaggle/input/bengaliai-speech/sample_submission.csv\")\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:05:49.313980Z","iopub.execute_input":"2023-07-25T06:05:49.314758Z","iopub.status.idle":"2023-07-25T06:05:49.332470Z","shell.execute_reply.started":"2023-07-25T06:05:49.314712Z","shell.execute_reply":"2023-07-25T06:05:49.331415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# The WER results stored in a list called `wer_results`\n# Assuming you have the WER results stored in a list called `wer_results`\nwer_results = [4.27, 5.32, 3.80] # Replace with your actual WER results\n\n\nsub = pd.read_csv(\"/kaggle/input/bengaliai-speech/sample_submission.csv\")\n\nsubmission_df = pd.DataFrame(sub)\n\n# Adding the WER results to the DataFrame\nsubmission_df[\"4.27%\"] = wer_results\n\n# Save the DataFrame as a submission CSV\nsubmission_df.to_csv(\"submission_with_wer.csv\", index=False)\n\n# Optionally, you can also display the DataFrame\nsubmission_df.head()\n","metadata":{"execution":{"iopub.status.busy":"2023-07-25T06:05:53.571493Z","iopub.execute_input":"2023-07-25T06:05:53.571930Z","iopub.status.idle":"2023-07-25T06:05:53.602086Z","shell.execute_reply.started":"2023-07-25T06:05:53.571900Z","shell.execute_reply":"2023-07-25T06:05:53.601035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reference:\n\n[Website:Librosa](https://librosa.org/)\n\n[OpenAI Whisper - MultiLingual AI Speech Recognition Live App Tutorial](https://www.youtube.com/watch?v=ywIyc8l1K1Q&t=237s)\n\n[Acknowledgement: jong wook kim](https://github.com/openai/whisper/blob/main/notebooks/LibriSpeech.ipynb)\n\n[Acknowledgement: Umar Farooqi](https://github.com/openai/whisper/blob/main/notebooks/Multilingual_ASR.ipynb)\n","metadata":{}},{"cell_type":"markdown","source":" 📌 \"Hey there! Your positive feedback and support for my notebook mean the world to me! It motivates me to create more valuable content. If you can spare a moment to give it an upvote, it would help others discover and benefit from it too. Together, let's foster a vibrant community of knowledge-sharing and empowerment. Thank you for considering it, and continued success on your learning journey!\"😃
","metadata":{}}]}