{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook , we'll try to build a basic end to end pipeline for this competition. I'll be following this [keras tutorial.](http://https://keras.io/examples/audio/ctc_asr/)","metadata":{}},{"cell_type":"markdown","source":"Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers. It is also known as automatic speech recognition (ASR), computer speech recognition or speech to text (STT). It incorporates knowledge and research in the computer science, linguistics and computer engineering fields.\n\nThis demonstration shows how to combine a 2D CNN, RNN and a Connectionist Temporal Classification (CTC) loss to build an ASR. CTC is an algorithm used to train deep neural networks in speech recognition, handwriting recognition and other sequence problems. CTC is used when we don’t know how the input aligns with the output (how the characters in the transcript align to the audio). The model we create is similar to DeepSpeech2.\n\nWe will evaluate the quality of the model using Word Error Rate (WER). WER is obtained by adding up the substitutions, insertions, and deletions that occur in a sequence of recognized words. Divide that number by the total number of words originally spoken. The result is the WER. To get the WER score you need to install the jiwer package. You can use the following command line:","metadata":{}},{"cell_type":"code","source":"!pip install jiwer\n!pip install bnunicodenormalizer","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:07:40.147908Z","iopub.execute_input":"2022-07-31T10:07:40.148715Z","iopub.status.idle":"2022-07-31T10:08:07.00675Z","shell.execute_reply.started":"2022-07-31T10:07:40.148604Z","shell.execute_reply":"2022-07-31T10:08:07.005619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Setup","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport matplotlib.pyplot as plt\nfrom IPython import display\nfrom jiwer import wer","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:08:26.818663Z","iopub.execute_input":"2022-07-31T10:08:26.819277Z","iopub.status.idle":"2022-07-31T10:08:31.891181Z","shell.execute_reply.started":"2022-07-31T10:08:26.819232Z","shell.execute_reply":"2022-07-31T10:08:31.890198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Loading Datasets","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/dlsprint/train.csv\")\nvalid = pd.read_csv(\"../input/dlsprint/validation.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:08:40.38306Z","iopub.execute_input":"2022-07-31T10:08:40.383979Z","iopub.status.idle":"2022-07-31T10:08:42.241334Z","shell.execute_reply.started":"2022-07-31T10:08:40.383941Z","shell.execute_reply":"2022-07-31T10:08:42.240352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T08:21:07.336316Z","iopub.execute_input":"2022-07-31T08:21:07.33689Z","iopub.status.idle":"2022-07-31T08:21:07.371431Z","shell.execute_reply.started":"2022-07-31T08:21:07.336849Z","shell.execute_reply":"2022-07-31T08:21:07.370469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We only need the path and sentence column","metadata":{}},{"cell_type":"code","source":"train = train[[\"path\",\"sentence\"]]\nval = valid[[\"path\",\"sentence\"]]","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:08:45.889265Z","iopub.execute_input":"2022-07-31T10:08:45.889876Z","iopub.status.idle":"2022-07-31T10:08:45.919096Z","shell.execute_reply.started":"2022-07-31T10:08:45.88984Z","shell.execute_reply":"2022-07-31T10:08:45.918076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T08:21:24.353704Z","iopub.execute_input":"2022-07-31T08:21:24.354177Z","iopub.status.idle":"2022-07-31T08:21:24.367465Z","shell.execute_reply.started":"2022-07-31T08:21:24.354134Z","shell.execute_reply":"2022-07-31T08:21:24.366341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Size of the training set: {len(train)}\")\nprint(f\"Size of the training set: {len(valid)}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:08:50.06164Z","iopub.execute_input":"2022-07-31T10:08:50.062001Z","iopub.status.idle":"2022-07-31T10:08:50.067764Z","shell.execute_reply.started":"2022-07-31T10:08:50.061969Z","shell.execute_reply":"2022-07-31T10:08:50.066524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Although we will not use the full 206k audios for training. We'll use [this](http://https://www.kaggle.com/datasets/nazmuddhohaansary/train-wavs-voted-dl-sprint) 37k wav files as training audios.","metadata":{}},{"cell_type":"markdown","source":"# Preprocessing","metadata":{}},{"cell_type":"markdown","source":"We first prepare the vocabulary to be used.","metadata":{}},{"cell_type":"code","source":"characters = [x for x in ['।', 'ঁ', 'ং', 'ঃ', 'অ', 'আ', 'ই', 'ঈ', 'উ', 'ঋ', 'এ', 'ঐ', 'ও', 'ক', 'খ', 'গ', 'ঘ', 'ঙ', 'চ', 'ছ', 'জ', 'ঝ', 'ঞ', 'ট', 'ঠ', 'ড', 'ঢ', 'ণ', 'ত', 'থ', 'দ', 'ধ', 'ন', 'প', 'ফ', 'ব', 'ভ', 'ম', 'য', 'র', 'ল', 'শ', 'ষ', 'স', 'হ', '়', 'া', 'ি', 'ী', 'ু', 'ূ', 'ৃ', 'ে', 'ৈ', 'ো', 'ৌ', '্', 'ৎ', 'ড়', 'য়', '০', '১', '২', '৩', '৪', '৫', '৬', '৭', '৮', '৯',' ']]\n# Mapping characters to integers\nchar_to_num = keras.layers.StringLookup(vocabulary=characters, oov_token=\"\")\n# Mapping integers back to original characters\nnum_to_char = keras.layers.StringLookup(\n    vocabulary=char_to_num.get_vocabulary(), oov_token=\"\", invert=True\n)\n\nprint(\n    f\"The vocabulary is: {char_to_num.get_vocabulary()} \"\n    f\"(size ={char_to_num.vocabulary_size()})\"\n)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:08:57.493433Z","iopub.execute_input":"2022-07-31T10:08:57.493802Z","iopub.status.idle":"2022-07-31T10:08:59.923246Z","shell.execute_reply.started":"2022-07-31T10:08:57.493772Z","shell.execute_reply":"2022-07-31T10:08:59.922049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I directly used the main characters of Bengali Language. A lot of other preprocessing stuffs can be done here. We'll not dive into that in this NB.","metadata":{}},{"cell_type":"markdown","source":"Next, we create the function that describes the transformation that we apply to each audio element of our dataset.","metadata":{}},{"cell_type":"code","source":"# An integer scalar Tensor. The window length in samples.\nframe_length = 200\n# An integer scalar Tensor. The number of samples to step.\nframe_step = 80\n# An integer scalar Tensor. The size of the FFT to apply.\n# If not provided, uses the smallest power of 2 enclosing frame_length.\nfft_length = 256\n\n\ndef encode_single_sample(wav_file, label):\n    ###########################################\n    ##  Process the Audio\n    ##########################################\n    # 1. Read wav file\n    file = tf.io.read_file(wavs_path + wav_file)\n    # 2. Decode the wav file\n    audio, _ = tf.audio.decode_wav(file)\n    audio = tf.squeeze(audio, axis=-1)\n    # 3. Change type to float\n    audio = tf.cast(audio, tf.float32)\n    # 4. Get the spectrogram\n    spectrogram = tf.signal.stft(\n        audio, frame_length=frame_length, frame_step=frame_step, fft_length=fft_length\n    )\n    # 5. We only need the magnitude, which can be derived by applying tf.abs\n    spectrogram = tf.abs(spectrogram)\n    spectrogram = tf.math.pow(spectrogram, 0.5)\n    # 6. normalisation\n    means = tf.math.reduce_mean(spectrogram, 1, keepdims=True)\n    stddevs = tf.math.reduce_std(spectrogram, 1, keepdims=True)\n    spectrogram = (spectrogram - means) / (stddevs + 1e-10)\n    ###########################################\n    ##  Process the label\n    ##########################################\n    # 7. Convert label to Lower case\n    #label = tf.strings.lower(label)\n    # 8. Split the label\n    label = tf.strings.unicode_split(label, input_encoding=\"UTF-8\")\n    # 9. Map the characters in label to numbers\n    label = char_to_num(label)\n    # 10. Return a dict as our model is expecting two inputs\n    return spectrogram, label","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:09:03.894719Z","iopub.execute_input":"2022-07-31T10:09:03.8952Z","iopub.status.idle":"2022-07-31T10:09:03.91223Z","shell.execute_reply.started":"2022-07-31T10:09:03.895135Z","shell.execute_reply":"2022-07-31T10:09:03.911013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating Dataset objects","metadata":{}},{"cell_type":"markdown","source":"We create a tf.data.Dataset object that yields the transformed elements, in the same order as they appeared in the input.\n\nBut first we have to make new csv files since we are not using all the audios from training csv and also the path names has .mp3 in as extension. We have to replace it with .wav","metadata":{}},{"cell_type":"code","source":"# Collecting training filenames\n\nimport os\nfilenames = os.listdir(\"../input/train-wavs-voted-dl-sprint/train_files_wav\")","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:09:10.957646Z","iopub.execute_input":"2022-07-31T10:09:10.958006Z","iopub.status.idle":"2022-07-31T10:09:11.507381Z","shell.execute_reply.started":"2022-07-31T10:09:10.957975Z","shell.execute_reply":"2022-07-31T10:09:11.506366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"path\"] = train[\"path\"].apply(lambda x:x.replace(\".mp3\",\".wav\"))","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:09:15.29117Z","iopub.execute_input":"2022-07-31T10:09:15.291552Z","iopub.status.idle":"2022-07-31T10:09:15.399375Z","shell.execute_reply.started":"2022-07-31T10:09:15.291519Z","shell.execute_reply":"2022-07-31T10:09:15.39839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm import tqdm\n\ndf = train.iloc[:len(filenames)]\nto_take = []\nlabel = []\n\nfor i in tqdm(range(len(train))):\n    if train.path.iloc[i] in filenames:\n        to_take.append(train.path.iloc[i])\n        label.append(train.sentence.iloc[i])\ndf.path = to_take\ndf.sentence = label\n\ntrain = df\nprint(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:09:18.616534Z","iopub.execute_input":"2022-07-31T10:09:18.616898Z","iopub.status.idle":"2022-07-31T10:13:11.047801Z","shell.execute_reply.started":"2022-07-31T10:09:18.616867Z","shell.execute_reply":"2022-07-31T10:13:11.046626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val[\"path\"] = val[\"path\"].apply(lambda x:x.replace(\".mp3\",\".wav\"))\nval.path.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:13:46.79789Z","iopub.execute_input":"2022-07-31T10:13:46.798371Z","iopub.status.idle":"2022-07-31T10:13:46.812773Z","shell.execute_reply.started":"2022-07-31T10:13:46.798337Z","shell.execute_reply":"2022-07-31T10:13:46.811581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Voila! We're back on track again. Let's make the dataset objects","metadata":{}},{"cell_type":"code","source":"wavs_path = \"../input/train-wavs-voted-dl-sprint/train_files_wav/\"\n\nbatch_size = 32\n# Define the trainig dataset\ntrain_dataset = tf.data.Dataset.from_tensor_slices(\n    (list(train[\"path\"]), list(train[\"sentence\"]))\n)\ntrain_dataset = (\n    train_dataset.map(encode_single_sample, num_parallel_calls=tf.data.AUTOTUNE)\n    .padded_batch(batch_size)\n    .prefetch(buffer_size=tf.data.AUTOTUNE)\n)\n\nwavs_path = \"../input/validation-fileswav-format/validation_files_wav/\"\n# Define the validation dataset\nvalidation_dataset = tf.data.Dataset.from_tensor_slices(\n    (list(val[\"path\"]), list(val[\"sentence\"]))\n)\nvalidation_dataset = (\n    validation_dataset.map(encode_single_sample, num_parallel_calls=tf.data.AUTOTUNE)\n    .padded_batch(batch_size)\n    .prefetch(buffer_size=tf.data.AUTOTUNE)\n)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:13:50.620499Z","iopub.execute_input":"2022-07-31T10:13:50.62086Z","iopub.status.idle":"2022-07-31T10:13:51.593106Z","shell.execute_reply.started":"2022-07-31T10:13:50.620828Z","shell.execute_reply":"2022-07-31T10:13:51.592161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize the data","metadata":{}},{"cell_type":"markdown","source":"Let's visualize an example in our dataset, including the audio clip, the spectrogram and the corresponding label.","metadata":{}},{"cell_type":"code","source":"wavs_path = \"../input/train-wavs-voted-dl-sprint/train_files_wav/\"\n\nfig = plt.figure(figsize=(8, 5))\nfor batch in train_dataset.take(1):\n    spectrogram = batch[0][0].numpy()\n    spectrogram = np.array([np.trim_zeros(x) for x in np.transpose(spectrogram)])\n    label = batch[1][0]\n    # Spectrogram\n    label = tf.strings.reduce_join(num_to_char(label)).numpy().decode(\"utf-8\")\n    print(label)\n    ax = plt.subplot(2, 1, 1)\n    ax.imshow(spectrogram, vmax=1)\n    ax.set_title(label)\n    ax.axis(\"off\")\n    # Wav\n    file = tf.io.read_file(wavs_path + list(train[\"path\"])[0])\n    audio, _ = tf.audio.decode_wav(file)\n    print(_)\n    audio = audio.numpy()\n    ax = plt.subplot(2, 1, 2)\n    plt.plot(audio)\n    ax.set_title(\"Signal Wave\")\n    ax.set_xlim(0, len(audio))\n    display.display(display.Audio(np.transpose(audio), rate=16000))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:13:54.385369Z","iopub.execute_input":"2022-07-31T10:13:54.385976Z","iopub.status.idle":"2022-07-31T10:13:55.915521Z","shell.execute_reply.started":"2022-07-31T10:13:54.385941Z","shell.execute_reply":"2022-07-31T10:13:55.91455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"markdown","source":"We first define the CTC Loss function","metadata":{}},{"cell_type":"code","source":"def CTCLoss(y_true, y_pred):\n    # Compute the training-time loss value\n    batch_len = tf.cast(tf.shape(y_true)[0], dtype=\"int64\")\n    input_length = tf.cast(tf.shape(y_pred)[1], dtype=\"int64\")\n    label_length = tf.cast(tf.shape(y_true)[1], dtype=\"int64\")\n\n    input_length = input_length * tf.ones(shape=(batch_len, 1), dtype=\"int64\")\n    label_length = label_length * tf.ones(shape=(batch_len, 1), dtype=\"int64\")\n    print(input_length,label_length)\n    loss = keras.backend.ctc_batch_cost(y_true, y_pred, input_length, label_length)\n    return loss","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:14:02.309759Z","iopub.execute_input":"2022-07-31T10:14:02.310429Z","iopub.status.idle":"2022-07-31T10:14:02.320161Z","shell.execute_reply.started":"2022-07-31T10:14:02.310378Z","shell.execute_reply":"2022-07-31T10:14:02.319088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We now define our model. We will define a model similar to [DeepSpeech2](http://https://nvidia.github.io/OpenSeq2Seq/html/speech-recognition/deepspeech2.html).","metadata":{}},{"cell_type":"code","source":"def build_model(input_dim, output_dim, rnn_layers=5, rnn_units=128):\n    \"\"\"Model similar to DeepSpeech2.\"\"\"\n    # Model's input\n    input_spectrogram = layers.Input((None, input_dim), name=\"input\")\n    # Expand the dimension to use 2D CNN.\n    x = layers.Reshape((-1, input_dim, 1), name=\"expand_dim\")(input_spectrogram)\n    # Convolution layer 1\n    x = layers.Conv2D(\n        filters=34,\n        kernel_size=[3, 3],\n        strides=[2, 2],\n        padding=\"same\",\n        use_bias=False,\n        name=\"conv_1\",\n    )(x)\n    x = layers.BatchNormalization(name=\"conv_1_bn\")(x)\n    x = layers.ReLU(name=\"conv_1_relu\")(x)\n    # Convolution layer 2\n    x = layers.Conv2D(\n        filters=34,\n        kernel_size=[3, 3],\n        strides=[1, 2],\n        padding=\"same\",\n        use_bias=False,\n        name=\"conv_2\",\n    )(x)\n    x = layers.BatchNormalization(name=\"conv_2_bn\")(x)\n    x = layers.ReLU(name=\"conv_2_relu\")(x)\n    # Convolution layer 3\n    x = layers.Conv2D(\n        filters=128,\n        kernel_size=[7, 7],\n        strides=[1, 2],\n        padding=\"same\",\n        use_bias=False,\n        name=\"conv_3\",\n    )(x)\n    x = layers.BatchNormalization(name=\"conv_3_bn\")(x)\n    x = layers.ReLU(name=\"conv_3_relu\")(x)\n    # Reshape the resulted volume to feed the RNNs layers\n    x = layers.Reshape((-1, x.shape[-2] * x.shape[-1]))(x)\n    # RNN layers\n    for i in range(1, rnn_layers + 1):\n        recurrent = layers.GRU(\n            units=rnn_units,\n            activation=\"tanh\",\n            recurrent_activation=\"sigmoid\",\n            use_bias=True,\n            return_sequences=True,\n            reset_after=True,\n            name=f\"gru_{i}\",\n        )\n        x = layers.Bidirectional(\n            recurrent, name=f\"bidirectional_{i}\", merge_mode=\"concat\"\n        )(x)\n        if i < rnn_layers:\n            x = layers.Dropout(rate=0.5)(x)\n    # Dense layer\n    x = layers.Dense(units=rnn_units * 2, name=\"dense_1\")(x)\n    x = layers.ReLU(name=\"dense_1_relu\")(x)\n    x = layers.Dropout(rate=0.5)(x)\n    # Classification layer\n    output = layers.Dense(units=output_dim + 1, activation=\"softmax\")(x)\n    # Model\n    model = keras.Model(input_spectrogram, output, name=\"DeepSpeech_2\")\n    # Optimizer\n    opt = keras.optimizers.Adam(learning_rate=0.001)\n    # Compile the model and return\n    model.compile(optimizer=opt, loss=CTCLoss)\n    return model\n\n\n# Get the model\nmodel = build_model(\n    input_dim=fft_length // 2 + 1,\n    output_dim=char_to_num.vocabulary_size(),\n    rnn_units=128,\n)\nmodel.summary(line_length=110)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:16:13.394503Z","iopub.execute_input":"2022-07-31T10:16:13.394935Z","iopub.status.idle":"2022-07-31T10:16:15.326147Z","shell.execute_reply.started":"2022-07-31T10:16:13.394903Z","shell.execute_reply":"2022-07-31T10:16:15.325029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training and Evaluating","metadata":{}},{"cell_type":"code","source":"# A utility function to decode the output of the network\ndef decode_batch_predictions(pred):\n    input_len = np.ones(pred.shape[0]) * pred.shape[1]\n    # Use greedy search. For complex tasks, you can use beam search\n    results = keras.backend.ctc_decode(pred, input_length=input_len, greedy=True)[0][0]\n    # Iterate over the results and get back the text\n    output_text = []\n    for result in results:\n        result = tf.strings.reduce_join(num_to_char(result)).numpy().decode(\"utf-8\")\n        output_text.append(result)\n    return output_text\n\n\n# A callback class to output a few transcriptions during training\nclass CallbackEval(keras.callbacks.Callback):\n    \"\"\"Displays a batch of outputs after every epoch.\"\"\"\n\n    def __init__(self, dataset):\n        super().__init__()\n        self.dataset = dataset\n\n    def on_epoch_end(self, epoch: int, logs=None):\n        predictions = []\n        targets = []\n        for batch in self.dataset:\n            X, y = batch\n            batch_predictions = model.predict(X)\n            batch_predictions = decode_batch_predictions(batch_predictions)\n            predictions.extend(batch_predictions)\n            for label in y:\n                label = (\n                    tf.strings.reduce_join(num_to_char(label)).numpy().decode(\"utf-8\")\n                )\n                targets.append(label)\n        wer_score = wer(targets, predictions)\n        print(\"-\" * 100)\n        print(f\"Word Error Rate: {wer_score:.4f}\")\n        print(\"-\" * 100)\n        for i in np.random.randint(0, len(predictions), 2):\n            print(f\"Target    : {targets[i]}\")\n            print(f\"Prediction: {predictions[i]}\")\n            print(\"-\" * 100)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:16:40.328304Z","iopub.execute_input":"2022-07-31T10:16:40.32867Z","iopub.status.idle":"2022-07-31T10:16:40.342205Z","shell.execute_reply.started":"2022-07-31T10:16:40.328639Z","shell.execute_reply":"2022-07-31T10:16:40.340361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Let's start the training process.","metadata":{}},{"cell_type":"code","source":"# Define the number of epochs.\nepochs = 2\n# Callback function to check transcription on the val set.\nvalidation_callback = CallbackEval(validation_dataset)\n# Train the model\nhistory = model.fit(\n    train_dataset,\n    validation_data=validation_dataset,\n    epochs=epochs,\n    callbacks=[validation_callback],\n)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T10:16:50.935064Z","iopub.execute_input":"2022-07-31T10:16:50.936011Z","iopub.status.idle":"2022-07-31T11:59:05.011839Z","shell.execute_reply.started":"2022-07-31T10:16:50.935975Z","shell.execute_reply":"2022-07-31T11:59:05.010758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's evaluate the model for the validation dataset","metadata":{}},{"cell_type":"code","source":"predictions = []\ntargets = []\ni=0\nfor batch in tqdm(validation_dataset):\n    X, y = batch\n    batch_predictions = model.predict(X)\n    batch_predictions = decode_batch_predictions(batch_predictions)\n    print(batch_predictions)\n    i+=1\n    if i>5:\n        break\n    predictions.extend(batch_predictions)\n    for label in y:\n        label = tf.strings.reduce_join(num_to_char(label)).numpy().decode(\"utf-8\")\n        targets.append(label)\nwer_score = wer(targets, predictions)\nprint(\"-\" * 100)\nprint(f\"Word Error Rate: {wer_score:.4f}\")\nprint(\"-\" * 100)\nfor i in np.random.randint(0, len(predictions), 5):\n    print(f\"Target    : {targets[i]}\")\n    print(f\"Prediction: {predictions[i]}\")\n    print(\"-\" * 100)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T12:06:59.878702Z","iopub.execute_input":"2022-07-31T12:06:59.879081Z","iopub.status.idle":"2022-07-31T12:07:09.067184Z","shell.execute_reply.started":"2022-07-31T12:06:59.879049Z","shell.execute_reply":"2022-07-31T12:07:09.066188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inference","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv(\"../input/dlsprint/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:08:09.098223Z","iopub.execute_input":"2022-07-08T00:08:09.099212Z","iopub.status.idle":"2022-07-08T00:08:09.135994Z","shell.execute_reply.started":"2022-07-08T00:08:09.099161Z","shell.execute_reply":"2022-07-08T00:08:09.135093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.path = test.path.apply(lambda x:x.replace(\".mp3\",\".wav\"))\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:08:09.137797Z","iopub.execute_input":"2022-07-08T00:08:09.138473Z","iopub.status.idle":"2022-07-08T00:08:09.154329Z","shell.execute_reply.started":"2022-07-08T00:08:09.138436Z","shell.execute_reply":"2022-07-08T00:08:09.15331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wavs_path = \"../input/test-wav-files-dl-sprint/test_files_wav/\"","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:08:09.155567Z","iopub.execute_input":"2022-07-08T00:08:09.156505Z","iopub.status.idle":"2022-07-08T00:08:09.163312Z","shell.execute_reply.started":"2022-07-08T00:08:09.156454Z","shell.execute_reply":"2022-07-08T00:08:09.162406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataset = tf.data.Dataset.from_tensor_slices(\n    (list(test[\"path\"]), list(test[\"sentence\"]))\n)\ntest_dataset = (\n    test_dataset.map(encode_single_sample, num_parallel_calls=tf.data.AUTOTUNE)\n    .padded_batch(batch_size)\n    .prefetch(buffer_size=tf.data.AUTOTUNE)\n)","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:08:09.166483Z","iopub.execute_input":"2022-07-08T00:08:09.166884Z","iopub.status.idle":"2022-07-08T00:08:09.316979Z","shell.execute_reply.started":"2022-07-08T00:08:09.166848Z","shell.execute_reply":"2022-07-08T00:08:09.31608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = []\ntargets = []\nfor batch in tqdm(validation_dataset):\n    X, y = batch\n    batch_predictions = model.predict(X)\n    batch_predictions = decode_batch_predictions(batch_predictions)\n    predictions.extend(batch_predictions)\n    for label in y:\n        label = tf.strings.reduce_join(num_to_char(label)).numpy().decode(\"utf-8\")\n        targets.append(label)\nwer_score = wer(targets, predictions)\nprint(\"-\" * 100)\nprint(f\"Word Error Rate: {wer_score:.4f}\")\nprint(\"-\" * 100)\nfor i in np.random.randint(0, len(predictions), 5):\n    print(f\"Target    : {targets[i]}\")\n    print(f\"Prediction: {predictions[i]}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:08:09.318734Z","iopub.execute_input":"2022-07-08T00:08:09.319083Z","iopub.status.idle":"2022-07-08T00:15:31.466048Z","shell.execute_reply.started":"2022-07-08T00:08:09.31905Z","shell.execute_reply":"2022-07-08T00:15:31.463086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Making Submission csv","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv(\"../input/dlsprint/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:15:31.467444Z","iopub.execute_input":"2022-07-08T00:15:31.467903Z","iopub.status.idle":"2022-07-08T00:15:31.490466Z","shell.execute_reply.started":"2022-07-08T00:15:31.467858Z","shell.execute_reply":"2022-07-08T00:15:31.489559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.sentence = predictions\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:15:31.4935Z","iopub.execute_input":"2022-07-08T00:15:31.493752Z","iopub.status.idle":"2022-07-08T00:15:31.505766Z","shell.execute_reply.started":"2022-07-08T00:15:31.493729Z","shell.execute_reply":"2022-07-08T00:15:31.504688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv(\"submission_ctc_tutorial\",index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-08T00:15:31.50727Z","iopub.execute_input":"2022-07-08T00:15:31.507704Z","iopub.status.idle":"2022-07-08T00:15:31.534595Z","shell.execute_reply.started":"2022-07-08T00:15:31.50767Z","shell.execute_reply":"2022-07-08T00:15:31.533756Z"},"trusted":true},"execution_count":null,"outputs":[]}]}