{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"> In this Notebook, we'll see how to preprocess the dataset for training Wav2Vec2. \n> The notebook is mostly modified from YellowKing's data preprocessing [Notebook](https://www.kaggle.com/code/sameen53/yellowking-dlsprint-datapreprocessingv1)","metadata":{"execution":{"iopub.status.busy":"2023-08-03T18:11:26.651706Z","iopub.execute_input":"2023-08-03T18:11:26.653104Z","iopub.status.idle":"2023-08-03T18:12:03.067322Z","shell.execute_reply.started":"2023-08-03T18:11:26.653032Z","shell.execute_reply":"2023-08-03T18:12:03.065501Z"}}},{"cell_type":"markdown","source":"
\n
Preprocessing Steps
\n","metadata":{}},{"cell_type":"markdown","source":"**Preprocessing steps** :\n> * [Create Dataset and Resample audios at 16000 sampling rate](#1)\n> * [Remove special characters and Normalize using bnunicodenormalizer](#2)\n> * [Tokenize using Wav2Vec2Processor](#3)\n> * [Trim silences](#4)\n> * [Filter audios of duration 1-10s](#5)","metadata":{}},{"cell_type":"markdown","source":"We'll preprocess ```30k audios for training``` and ```5k audios for validation```. If we want to preprocess more audios we'll have to do that in chunks because kaggle won't allow us to have more than 19GB data in the disk.","metadata":{}},{"cell_type":"markdown","source":"# Install Dependencies","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install transformers\n!pip install jiwer\n!apt install git-lfs","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:52:34.977919Z","iopub.execute_input":"2023-08-13T09:52:34.978425Z","iopub.status.idle":"2023-08-13T09:53:11.450128Z","shell.execute_reply.started":"2023-08-13T09:52:34.978389Z","shell.execute_reply":"2023-08-13T09:53:11.448566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install bnunicodenormalizer","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:11.452927Z","iopub.execute_input":"2023-08-13T09:53:11.453416Z","iopub.status.idle":"2023-08-13T09:53:28.339976Z","shell.execute_reply.started":"2023-08-13T09:53:11.453373Z","shell.execute_reply":"2023-08-13T09:53:28.338937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Imports**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport json\nfrom datasets import Audio\nfrom datasets import Dataset\nfrom bnunicodenormalizer import Normalizer \nbnorm=Normalizer()\nfrom datasets import concatenate_datasets","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:28.341736Z","iopub.execute_input":"2023-08-13T09:53:28.342214Z","iopub.status.idle":"2023-08-13T09:53:29.501287Z","shell.execute_reply.started":"2023-08-13T09:53:28.342161Z","shell.execute_reply":"2023-08-13T09:53:29.500007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
Create Dataset and Resample audios at 16000 sampling rate
","metadata":{}},{"cell_type":"code","source":"#Read the dataframe\ndf = pd.read_csv(\"/kaggle/input/bengaliai-speech/train.csv\")\ntrain = df[df.split==\"train\"]\nval = df[df.split==\"valid\"]\ndisplay(train.head())\ndisplay(val.head())\nprint(\"Train shape : \",train.shape)\nprint(\"Validation set shape : \",val.shape)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:29.504482Z","iopub.execute_input":"2023-08-13T09:53:29.505348Z","iopub.status.idle":"2023-08-13T09:53:36.023387Z","shell.execute_reply.started":"2023-08-13T09:53:29.505293Z","shell.execute_reply":"2023-08-13T09:53:36.022119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll take a subset of training and validation set for this notebook. Let's shuffle the dataset and take 30k training samples and 10k validation samples randomly","metadata":{}},{"cell_type":"code","source":"train = train.sample(frac=1,random_state=42)\nval = val.sample(frac=1,random_state=42)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:36.024985Z","iopub.execute_input":"2023-08-13T09:53:36.025451Z","iopub.status.idle":"2023-08-13T09:53:36.466970Z","shell.execute_reply.started":"2023-08-13T09:53:36.025404Z","shell.execute_reply":"2023-08-13T09:53:36.465613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.iloc[30000:50000]\nval = val.iloc[:5000]\nprint(\"Train shape : \",train.shape)\nprint(\"Validation set shape : \",val.shape)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:36.468257Z","iopub.execute_input":"2023-08-13T09:53:36.468653Z","iopub.status.idle":"2023-08-13T09:53:36.476859Z","shell.execute_reply.started":"2023-08-13T09:53:36.468617Z","shell.execute_reply":"2023-08-13T09:53:36.475633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Now we'll create pytorch datasets from the audio paths.\n\n> We'll resample the audios at 16000 sampling rate here since we'll need to have the audios at thi sampling rate to train Wav2Vec2 model","metadata":{}},{"cell_type":"code","source":"audio_dir = \"/kaggle/input/bengaliai-speech/train_mp3s/\"\ntrain_paths = train['id'].apply(lambda x:audio_dir+x+\".mp3\")\ntrain_ds = Dataset.from_dict({\"audio\":train_paths ,\"sentence\":train['sentence'].tolist()}).cast_column(\"audio\", Audio(sampling_rate=16000))","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:36.478858Z","iopub.execute_input":"2023-08-13T09:53:36.479211Z","iopub.status.idle":"2023-08-13T09:53:36.581089Z","shell.execute_reply.started":"2023-08-13T09:53:36.479180Z","shell.execute_reply":"2023-08-13T09:53:36.580200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at what we have here","metadata":{}},{"cell_type":"code","source":"train_ds","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:36.582361Z","iopub.execute_input":"2023-08-13T09:53:36.583684Z","iopub.status.idle":"2023-08-13T09:53:36.591311Z","shell.execute_reply.started":"2023-08-13T09:53:36.583646Z","shell.execute_reply":"2023-08-13T09:53:36.589934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at a sample","metadata":{}},{"cell_type":"code","source":"train_ds[0]","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:36.592985Z","iopub.execute_input":"2023-08-13T09:53:36.593702Z","iopub.status.idle":"2023-08-13T09:53:41.081402Z","shell.execute_reply.started":"2023-08-13T09:53:36.593645Z","shell.execute_reply":"2023-08-13T09:53:41.078848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> We have two columns here. \n> * **audio** - contains the information about the audio. sample rate, audio array , path etc.\n> * **sentence** - contains the transcription","metadata":{}},{"cell_type":"markdown","source":"Let's listen the audio","metadata":{}},{"cell_type":"code","source":"import IPython.display as ipd\nipd.Audio(train_ds[0]['audio']['array'],rate = 16000)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:41.086777Z","iopub.execute_input":"2023-08-13T09:53:41.088101Z","iopub.status.idle":"2023-08-13T09:53:41.130902Z","shell.execute_reply.started":"2023-08-13T09:53:41.088056Z","shell.execute_reply":"2023-08-13T09:53:41.129410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
Remove special characters and Normalize using bnunicodenormalizer
","metadata":{}},{"cell_type":"markdown","source":"We'll remove the punctuations from the sentences and also normalize the sentences using bnunicodenormalizer. ","metadata":{}},{"cell_type":"code","source":"import re\nchars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\\"\\—\\‘\\'\\‚\\“\\”\\…]'\n\ndef remove_special_characters(batch):\n batch[\"sentence\"] = re.sub(chars_to_ignore_regex, '', batch[\"sentence\"]) + \" \"\n return batch\n\ndef normalize(batch):\n _words = [bnorm(word)['normalized'] for word in batch[\"sentence\"].split()]\n batch[\"sentence\"] = \" \".join([word for word in _words if word is not None])\n return batch\n","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:41.133026Z","iopub.execute_input":"2023-08-13T09:53:41.133928Z","iopub.status.idle":"2023-08-13T09:53:41.145361Z","shell.execute_reply.started":"2023-08-13T09:53:41.133878Z","shell.execute_reply":"2023-08-13T09:53:41.143115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = train_ds.map(remove_special_characters)\ntrain_ds = train_ds.map(normalize)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:41.146787Z","iopub.execute_input":"2023-08-13T09:53:41.147849Z","iopub.status.idle":"2023-08-13T09:53:46.897880Z","shell.execute_reply.started":"2023-08-13T09:53:41.147810Z","shell.execute_reply":"2023-08-13T09:53:46.896260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
Tokenize using pretrained processor
","metadata":{}},{"cell_type":"markdown","source":"In this step we'll tokenize using the pretrained processor from the [publicly available model ](https://huggingface.co/arijitx/wav2vec2-xls-r-300m-bengali) using Wav2Vec2Processor.from_pretrained(\"arijitx/wav2vec2-xls-r-300m-bengali\")","metadata":{}},{"cell_type":"code","source":"from transformers import Wav2Vec2Processor\n\nprocessor = Wav2Vec2Processor.from_pretrained(\"arijitx/wav2vec2-xls-r-300m-bengali\")","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:46.899472Z","iopub.execute_input":"2023-08-13T09:53:46.899883Z","iopub.status.idle":"2023-08-13T09:53:50.272169Z","shell.execute_reply.started":"2023-08-13T09:53:46.899850Z","shell.execute_reply":"2023-08-13T09:53:50.270786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def prepare_dataset(batch):\n\n batch[\"audio\"][\"array\"] = np.trim_zeros(batch[\"audio\"][\"array\"], 'fb')\n audio = batch[\"audio\"]\n \n\n # batched output is \"un-batched\" to ensure mapping is correct\n batch[\"input_values\"] = processor(audio[\"array\"], sampling_rate=16000).input_values[0]\n batch[\"input_length\"] = len(batch[\"input_values\"])\n \n with processor.as_target_processor():\n batch[\"labels\"] = processor(batch[\"sentence\"]).input_ids\n return batch\n","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:50.273953Z","iopub.execute_input":"2023-08-13T09:53:50.274688Z","iopub.status.idle":"2023-08-13T09:53:50.282847Z","shell.execute_reply.started":"2023-08-13T09:53:50.274644Z","shell.execute_reply":"2023-08-13T09:53:50.281410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = train_ds.map(prepare_dataset, remove_columns=train_ds.column_names)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:53:50.284143Z","iopub.execute_input":"2023-08-13T09:53:50.284521Z","iopub.status.idle":"2023-08-13T09:54:23.313627Z","shell.execute_reply.started":"2023-08-13T09:53:50.284468Z","shell.execute_reply":"2023-08-13T09:54:23.311995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
Trim silences
","metadata":{}},{"cell_type":"markdown","source":"Now we'll trim silences with the threshold (amplitude < max_amplitude/30 ). This makes training faster","metadata":{}},{"cell_type":"code","source":"def trim_silence(batch):\n arr = batch['input_values']\n \n try:\n _max = max(max(arr), -min(arr))\n old_length = len(arr)\n \n threshold = 30\n\n for i,e in enumerate(arr):\n if threshold*e>_max:\n break\n\n for j,e in enumerate(reversed(arr)):\n if threshold*e>_max:\n break\n\n batch['input_values'] = arr[i:old_length-j]\n batch['input_length'] = old_length -i -j\n except:\n print(batch['input_length'])\n return batch","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:54:23.315416Z","iopub.execute_input":"2023-08-13T09:54:23.315977Z","iopub.status.idle":"2023-08-13T09:54:23.326591Z","shell.execute_reply.started":"2023-08-13T09:54:23.315941Z","shell.execute_reply":"2023-08-13T09:54:23.324785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = train_ds.map(trim_silence)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:55:37.986225Z","iopub.execute_input":"2023-08-13T09:55:37.986674Z","iopub.status.idle":"2023-08-13T09:56:45.696720Z","shell.execute_reply.started":"2023-08-13T09:55:37.986638Z","shell.execute_reply":"2023-08-13T09:56:45.695277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n
Filter audios of duration 1-10s
","metadata":{}},{"cell_type":"code","source":"max_input_length_in_sec = 10.0\nmin_input_length_in_sec = 1\n\ntrain_ds = train_ds.filter(lambda x: x < max_input_length_in_sec * 16000, input_columns=[\"input_length\"])\ntrain_ds = train_ds.filter(lambda x: x > min_input_length_in_sec * 16000, input_columns=[\"input_length\"])\n","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:56:52.625445Z","iopub.execute_input":"2023-08-13T09:56:52.626739Z","iopub.status.idle":"2023-08-13T09:56:52.705894Z","shell.execute_reply.started":"2023-08-13T09:56:52.626695Z","shell.execute_reply":"2023-08-13T09:56:52.704585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's save the dataset in the disk","metadata":{}},{"cell_type":"code","source":"train_ds.save_to_disk(\"train\")","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:56:52.708652Z","iopub.execute_input":"2023-08-13T09:56:52.709052Z","iopub.status.idle":"2023-08-13T09:57:40.049592Z","shell.execute_reply.started":"2023-08-13T09:56:52.709018Z","shell.execute_reply":"2023-08-13T09:57:40.048077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**A complete function with all the steps**","metadata":{}},{"cell_type":"code","source":"def create_dataset(df):\n paths = df['id'].apply(lambda x:audio_dir+x+\".mp3\")\n dataset = Dataset.from_dict({\"audio\":paths ,\"sentence\":df['sentence'].tolist()}).cast_column(\"audio\", Audio(sampling_rate=16000))\n dataset=dataset.map(remove_special_characters)\n dataset = dataset.map(normalize)\n dataset_train = dataset.map(prepare_dataset, remove_columns=dataset.column_names)\n dataset = dataset_train.map(trim_silence)\n\n dataset = dataset.filter(lambda x: x < max_input_length_in_sec * 16000, input_columns=[\"input_length\"])\n dataset = dataset.filter(lambda x: x > min_input_length_in_sec * 16000, input_columns=[\"input_length\"])\n \n return dataset","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:57:40.052882Z","iopub.execute_input":"2023-08-13T09:57:40.053413Z","iopub.status.idle":"2023-08-13T09:57:40.062392Z","shell.execute_reply.started":"2023-08-13T09:57:40.053366Z","shell.execute_reply":"2023-08-13T09:57:40.061025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_ds = create_dataset(val)","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:57:40.065158Z","iopub.execute_input":"2023-08-13T09:57:40.065579Z","iopub.status.idle":"2023-08-13T09:58:41.857284Z","shell.execute_reply.started":"2023-08-13T09:57:40.065538Z","shell.execute_reply":"2023-08-13T09:58:41.855858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_ds.save_to_disk(\"val\")","metadata":{"execution":{"iopub.status.busy":"2023-08-13T09:58:41.858559Z","iopub.execute_input":"2023-08-13T09:58:41.858912Z","iopub.status.idle":"2023-08-13T09:59:29.378584Z","shell.execute_reply.started":"2023-08-13T09:58:41.858881Z","shell.execute_reply":"2023-08-13T09:59:29.377345Z"},"trusted":true},"execution_count":null,"outputs":[]}]}