{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<center><div class=\"alert alert-block alert-info\" style=\" line-height: 1.7em; font-family: Verdana;\">\n    <b style=\"font-size: 18px; color:green\"> &nbsp; Bengali.AI Speech Recognition : Our very first submission</b><br><br><b style=\"font-size: 18px; color:green\">HELLO EVERYONE!</b><br>\n</div></center>\n\nIn this Notebook we'll try to submit our first submission by using a publicly available model.\n\nWe'll be using this [BanglaASR](https://huggingface.co/bangla-speech-processing/BanglaASR) model. It is actually a finetuned [whisper](https://openai.com/research/whisper) model on rained [Bangla Mozilla Common Voice Dataset](https://arxiv.org/abs/2206.14053). It's reported performance is 4.58% on 7k validation set which seems pretty good to be a starting point. So let's start with that.\n\nI'll be trying to use the other publicly available models one by one and see how they perform. Then we'll get an idea from where we can start the training/finetuning on this huge dataset.","metadata":{"execution":{"iopub.status.busy":"2023-07-18T02:29:22.175543Z","iopub.execute_input":"2023-07-18T02:29:22.175858Z","iopub.status.idle":"2023-07-18T02:29:36.971462Z","shell.execute_reply.started":"2023-07-18T02:29:22.175830Z","shell.execute_reply":"2023-07-18T02:29:36.970344Z"}}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport librosa\nimport torch\nimport torchaudio\nimport numpy as np\n\nfrom transformers import WhisperTokenizer\nfrom transformers import WhisperProcessor\nfrom transformers import WhisperFeatureExtractor\nfrom transformers import WhisperForConditionalGeneration\nfrom tqdm import tqdm\nimport pandas as pd\nimport soundfile as sf\nfrom pydub import AudioSegment","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:41:43.401035Z","iopub.execute_input":"2023-07-18T10:41:43.401370Z","iopub.status.idle":"2023-07-18T10:41:56.538811Z","shell.execute_reply.started":"2023-07-18T10:41:43.401341Z","shell.execute_reply":"2023-07-18T10:41:56.537864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the model","metadata":{}},{"cell_type":"markdown","source":"While submitting notebooks, you need to have internet disabled. So you can't download models from huggingface directly. One workaround is to download the models from huggingface, upload them to kaggle as datasets and then use them!","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nmodel_path = \"/kaggle/input/bangla-speech-processing-banglaasr/BanglaASR\"\nfeature_extractor = WhisperFeatureExtractor.from_pretrained(model_path)\ntokenizer = WhisperTokenizer.from_pretrained(model_path)\nprocessor = WhisperProcessor.from_pretrained(model_path)\nmodel = WhisperForConditionalGeneration.from_pretrained(model_path).to(device)","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:41:56.540715Z","iopub.execute_input":"2023-07-18T10:41:56.541025Z","iopub.status.idle":"2023-07-18T10:42:12.028790Z","shell.execute_reply.started":"2023-07-18T10:41:56.540999Z","shell.execute_reply":"2023-07-18T10:42:12.027637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Demo inference","metadata":{}},{"cell_type":"markdown","source":"We'll write an inference function where we'll give the path as input and it will return the transcription!\n\nFirst we'll need to convert the audio sampling rate to 16k since WhisperFeatureExtractor requires audios to be sampled at 16k.","metadata":{}},{"cell_type":"code","source":"#This code is modified from the source code provided here https://huggingface.co/bangla-speech-processing/BanglaASR\ndef inference_fn(path):\n    speech_array, sampling_rate = sf.read(mp3_path)\n    speech_array = librosa.resample(np.asarray(speech_array), orig_sr=sampling_rate, target_sr=16000)\n    input_features = feature_extractor(speech_array, sampling_rate=16000, return_tensors=\"pt\").input_features\n    predicted_ids = model.generate(inputs=input_features.to(device))[0]\n    transcription = processor.decode(predicted_ids, skip_special_tokens=True)\n    return transcription\n","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:12.030311Z","iopub.execute_input":"2023-07-18T10:42:12.030763Z","iopub.status.idle":"2023-07-18T10:42:12.037850Z","shell.execute_reply.started":"2023-07-18T10:42:12.030727Z","shell.execute_reply":"2023-07-18T10:42:12.036662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now let's look at some examples from the train set and see how this model performs**","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/bengaliai-speech/train.csv\")\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:12.040885Z","iopub.execute_input":"2023-07-18T10:42:12.041235Z","iopub.status.idle":"2023-07-18T10:42:16.229791Z","shell.execute_reply.started":"2023-07-18T10:42:12.041204Z","shell.execute_reply":"2023-07-18T10:42:16.228683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"okay so let's hear the an audio from the train set. ","metadata":{}},{"cell_type":"code","source":"root_path = \"/kaggle/input/bengaliai-speech/train_mp3s/\"\nidx = 1\nmp3_path = root_path + df['id'].iloc[idx]+\".mp3\"\n\nprint(f\"File name :\",mp3_path)\nAudioSegment.from_file(mp3_path)","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:16.232313Z","iopub.execute_input":"2023-07-18T10:42:16.233012Z","iopub.status.idle":"2023-07-18T10:42:17.006818Z","shell.execute_reply.started":"2023-07-18T10:42:16.232977Z","shell.execute_reply":"2023-07-18T10:42:17.005864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's see what it's actual transcription is and what this model is predicting?","metadata":{}},{"cell_type":"code","source":"print(f\"Original Transcription : {df['sentence'].iloc[idx]}\")\nprint(f\"Predicted Text : {inference_fn(mp3_path)}\")","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:17.010110Z","iopub.execute_input":"2023-07-18T10:42:17.010411Z","iopub.status.idle":"2023-07-18T10:42:31.411049Z","shell.execute_reply.started":"2023-07-18T10:42:17.010386Z","shell.execute_reply":"2023-07-18T10:42:31.410110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"okay that's quite bad ! :3 The main reason behind that is the original audio is sampled at 32k sampling rate, but this model(Whisper) needs the inputs to be sampled at 16k. So in the inference function we're resampling the audio to 16k sampling rate and at that time the audio quality is degrading!\nThis will be a big challenge in this competition. \n\nLet's look at some more examples:","metadata":{}},{"cell_type":"code","source":"import time\ntotal = 0\nfor idx in range(5,10):\n    mp3_path = root_path + df['id'].iloc[idx]+\".mp3\"\n    print(f\"File name :\",mp3_path)\n    print(f\"Original Transcription : {df['sentence'].iloc[idx]}\")\n    start = time.time()\n    print(f\"Predicted Text : {inference_fn(mp3_path)}\")\n    end = time.time()\n    total+=end-start\n    print(\"\\n\")\nprint(f\"Total Inference time : {total} seconds\")","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:31.415211Z","iopub.execute_input":"2023-07-18T10:42:31.418848Z","iopub.status.idle":"2023-07-18T10:42:44.813262Z","shell.execute_reply.started":"2023-07-18T10:42:31.418810Z","shell.execute_reply":"2023-07-18T10:42:44.812169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like it's doing quite good for longer sentences and common words!","metadata":{}},{"cell_type":"markdown","source":"# Inference on test data","metadata":{}},{"cell_type":"markdown","source":"Let's look at the test data first. From the competition description :\n\n```Note that this is a Code Competition, in which the actual test set is hidden. In this public version, we give some sample data in the correct format to help you author your solutions. The full test set contains about 20 hours of speech in almost 8000 MP3 audio files. All of the files in the test set are encoded at a sample rate of 32k, a bit rate of 48k, in one channel.```\n\nSo we're not seeing all the test data here. in the ***test_mp3s*** folder there's only three files available. Let's look at what they have to say!","metadata":{}},{"cell_type":"code","source":"mp3_path = \"/kaggle/input/bengaliai-speech/test_mp3s/0f3dac00655e.mp3\"\nprint(f\"File name :\",mp3_path)\nAudioSegment.from_file(mp3_path)","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:44.814827Z","iopub.execute_input":"2023-07-18T10:42:44.815206Z","iopub.status.idle":"2023-07-18T10:42:45.167877Z","shell.execute_reply.started":"2023-07-18T10:42:44.815172Z","shell.execute_reply":"2023-07-18T10:42:45.166957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What's she saying? Well I can understand it since Bengali is my mother tongue.\n\n<span style=\"color:purple\">Transcription : একটু বয়েস হলে একটি বিদেশী</span>\n\nNow let's see what our model understands from this audio. \n","metadata":{}},{"cell_type":"code","source":"print(f\"Prediction : {inference_fn(mp3_path)}\")","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:45.169312Z","iopub.execute_input":"2023-07-18T10:42:45.169643Z","iopub.status.idle":"2023-07-18T10:42:45.973154Z","shell.execute_reply.started":"2023-07-18T10:42:45.169611Z","shell.execute_reply":"2023-07-18T10:42:45.972147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Pretty close!\n\nNow let's look at the **sample_submission.csv** file to understand what to submit .","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv(\"/kaggle/input/bengaliai-speech/sample_submission.csv\")\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:42:45.976438Z","iopub.execute_input":"2023-07-18T10:42:45.976738Z","iopub.status.idle":"2023-07-18T10:42:45.992961Z","shell.execute_reply.started":"2023-07-18T10:42:45.976712Z","shell.execute_reply":"2023-07-18T10:42:45.991923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now to submit in the competition, we'll have to infer on all the audio files(approx 8k) and then generate a csv file containing the \"id\" of the file and the predicted sentence. Pretty easy, eh?","metadata":{"execution":{"iopub.status.busy":"2023-07-18T03:00:48.556355Z","iopub.execute_input":"2023-07-18T03:00:48.556774Z","iopub.status.idle":"2023-07-18T03:00:48.565196Z","shell.execute_reply.started":"2023-07-18T03:00:48.556741Z","shell.execute_reply":"2023-07-18T03:00:48.564074Z"}}},{"cell_type":"code","source":"import os\ntest_path = \"/kaggle/input/bengaliai-speech/test_mp3s/\"\nfiles = os.listdir(test_path)\nids = []\nsentences = []\nfor file in tqdm(files):\n    ids.append(file.split(\".\")[0])\n    mp3_path = os.path.join(test_path,file)\n    prediction = inference_fn(mp3_path)\n    \n    #sanity check\n    if len(prediction)==0:\n        prediction = \"\\n\"\n    \n    sentences.append(prediction)\n    \n    ","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:44:25.636814Z","iopub.execute_input":"2023-07-18T10:44:25.637182Z","iopub.status.idle":"2023-07-18T10:44:29.689356Z","shell.execute_reply.started":"2023-07-18T10:44:25.637148Z","shell.execute_reply":"2023-07-18T10:44:29.688406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Predictions ready. Let's make the csv file","metadata":{}},{"cell_type":"code","source":"df = pd.DataFrame({\"id\":ids,\"sentence\":sentences})\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:44:29.691670Z","iopub.execute_input":"2023-07-18T10:44:29.692398Z","iopub.status.idle":"2023-07-18T10:44:29.702479Z","shell.execute_reply.started":"2023-07-18T10:44:29.692362Z","shell.execute_reply":"2023-07-18T10:44:29.701466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.to_csv(\"submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-18T10:44:41.002623Z","iopub.execute_input":"2023-07-18T10:44:41.003205Z","iopub.status.idle":"2023-07-18T10:44:41.011862Z","shell.execute_reply.started":"2023-07-18T10:44:41.003163Z","shell.execute_reply":"2023-07-18T10:44:41.010917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All set. Let's hit the \"submit\" option and pray!","metadata":{}}]}