{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"```\nN-gram language model can boost the performance of Wav2Vec2 model by a significant amount. In this notebook, we'll see how we can create an n-gram language model, combine it with a wav2vec2 model and the difference it makes in the performance of the wav2vec2 model.\n```","metadata":{}},{"cell_type":"markdown","source":"# What is n-gram Language Model?\n\n```\nN-gram language models are a type of probabilistic language model where \"n\" represents the number of consecutive items (usually words or characters) in a sequence. For instance, a unigram is a single word, a bigram is a pair of consecutive words, a trigram is a sequence of three consecutive words, and so on. These models compute the likelihood of a word or character given the previous \"n-1\" words or characters. N-gram models estimate the probability distribution of sequences of words or characters based on the frequency of their occurrences in the training data.\n```\n\n\n**Suppose we have a sentence: \"The quick brown fox jumps over the lazy dog.\"**\n\n**Unigram (1-gram):**\n\n> \"The\", \"quick\", \"brown\", \"fox\", \"jumps\", \"over\", \"the\", \"lazy\", \"dog.\"\n\n**Bigram (2-gram):**\n\n> \"The quick\", \"quick brown\", \"brown fox\", \"fox jumps\", \"jumps over\", \"over the\", \"the lazy\", \"lazy dog.\"\n\n**Trigram (3-gram):**\n\n> \"The quick brown\", \"quick brown fox\", \"brown fox jumps\", \"fox jumps over\", \"jumps over the\", \"over the lazy\", \"the lazy dog.\"\n\n**4-gram (Quadgram):**\n\n> \"The quick brown fox\", \"quick brown fox jumps\", \"brown fox jumps over\", \"fox jumps over the\", \"jumps over the lazy\", \"over the lazy dog.\"\n\nHere, each word is predicted based on the three preceding words.\n\n**N-gram language models are trained on large text corpora to estimate the probabilities of these sequences**. This information is then used to predict the most likely word or character given the context. ","metadata":{}},{"cell_type":"markdown","source":"# Why do we need Language Model here?","metadata":{}},{"cell_type":"markdown","source":"```\nWav2vec2 is an acoustic model, that means it can map the speech audios with characters/phonemes. However, it might struggle with punctuations,spellings etc since these are not directly mapped with the audios/ different spellings might have the same or close pronunciation. In some cases, a character might be silence while pronunciation. \n```\n\n> For example, **'Knight' and 'Night'** have the same pronunciation, so the correct spelling will depend on the context.\n> **Homophones** : weak/week , too/two etc\n\nN-gram models can boost the performance of the acoustic models by leveraging it's linguistic charactersitics, by understanding the context and thus modify the words that seems to have the best probabilty to be in that place.\n","metadata":{}},{"cell_type":"markdown","source":"[Reference tutorial to understand N-gram models and implementation](https://huggingface.co/blog/wav2vec2-with-ngram)","metadata":{}},{"cell_type":"markdown","source":"# Installations","metadata":{}},{"cell_type":"code","source":"import resource\nimport os\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:22:49.492718Z","iopub.execute_input":"2023-09-03T20:22:49.493125Z","iopub.status.idle":"2023-09-03T20:22:49.499146Z","shell.execute_reply.started":"2023-09-03T20:22:49.493095Z","shell.execute_reply":"2023-09-03T20:22:49.497807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%capture ts\n!pip install jiwer\n!pip install bnunicodenormalizer\n!pip -q install https://github.com/kpu/kenlm/archive/master.zip pyctcdecode\n! sudo apt -y install build-essential cmake libboost-system-dev libboost-thread-dev libboost-program-options-dev libboost-test-dev libeigen3-dev zlib1g-dev libbz2-dev liblzma-dev\n! wget -O - https://kheafield.com/code/kenlm.tar.gz | tar xz\n! mkdir kenlm/build && cd kenlm/build && cmake .. && make -j2\n! ls kenlm/build/bin","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:22:49.501413Z","iopub.execute_input":"2023-09-03T20:22:49.501786Z","iopub.status.idle":"2023-09-03T20:23:54.497741Z","shell.execute_reply.started":"2023-09-03T20:22:49.501755Z","shell.execute_reply":"2023-09-03T20:23:54.496103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from bnunicodenormalizer import Normalizer \nbnorm = Normalizer()\ndef normalize(sen):\n    _words = [bnorm(word)['normalized']  for word in sen.split()]\n    return \" \".join([word for word in _words if word is not None])","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:23:54.499381Z","iopub.execute_input":"2023-09-03T20:23:54.499760Z","iopub.status.idle":"2023-09-03T20:23:54.507083Z","shell.execute_reply.started":"2023-09-03T20:23:54.499725Z","shell.execute_reply":"2023-09-03T20:23:54.505908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the Data","metadata":{}},{"cell_type":"markdown","source":"We'll be using all of our training sentences to build our 5-gram langauge model","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/bengaliai-speech/train.csv\")\ntrain_df = df[df.split==\"train\"]\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:23:54.510420Z","iopub.execute_input":"2023-09-03T20:23:54.510899Z","iopub.status.idle":"2023-09-03T20:23:58.891882Z","shell.execute_reply.started":"2023-09-03T20:23:54.510858Z","shell.execute_reply":"2023-09-03T20:23:58.890667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dump the sentences in a text file","metadata":{}},{"cell_type":"code","source":"all_sentences = list(set(train_df['sentence'].tolist()))\nlen(all_sentences)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:23:58.893451Z","iopub.execute_input":"2023-09-03T20:23:58.896543Z","iopub.status.idle":"2023-09-03T20:23:59.305469Z","shell.execute_reply.started":"2023-09-03T20:23:58.896497Z","shell.execute_reply":"2023-09-03T20:23:59.304133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import re\nfrom tqdm import tqdm\nchars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\\"\\“\\%\\‘\\”\\�\\']'\n\nwith open('text.txt', 'w') as f:\n    for sentence in tqdm(all_sentences):\n        f.write(normalize(sentence))\n        f.write('\\n')\n    with open(\"/kaggle/input/dataset-for-lm/hasan-etal-2020-low/2.75M/original_corpus.bn\",'r') as f2:\n        lines = f2.readlines()\n        \n        for line in tqdm(lines):\n            f.write(normalize(line))\n            f.write('\\n')","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:23:59.307252Z","iopub.execute_input":"2023-09-03T20:23:59.307721Z","iopub.status.idle":"2023-09-03T20:24:11.687465Z","shell.execute_reply.started":"2023-09-03T20:23:59.307681Z","shell.execute_reply":"2023-09-03T20:24:11.686130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now build 5-gram arpa model using kenlm. As it's relatively common in speech recognition, we build a 5-gram by passing the -o 5 parameter.**","metadata":{}},{"cell_type":"code","source":"pwd","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:24:11.689358Z","iopub.execute_input":"2023-09-03T20:24:11.689818Z","iopub.status.idle":"2023-09-03T20:24:11.698942Z","shell.execute_reply.started":"2023-09-03T20:24:11.689775Z","shell.execute_reply":"2023-09-03T20:24:11.697372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!kenlm/build/bin/lmplz -o 5 < \"text.txt\" > \"5gram.arpa\"","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:24:11.701094Z","iopub.execute_input":"2023-09-03T20:24:11.701604Z","iopub.status.idle":"2023-09-03T20:26:18.998213Z","shell.execute_reply.started":"2023-09-03T20:24:11.701569Z","shell.execute_reply":"2023-09-03T20:26:18.996172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at what is inside 5gram.arpa","metadata":{}},{"cell_type":"code","source":"!head -20 5gram.arpa","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:26:19.002579Z","iopub.execute_input":"2023-09-03T20:26:19.003037Z","iopub.status.idle":"2023-09-03T20:26:20.193404Z","shell.execute_reply.started":"2023-09-03T20:26:19.003000Z","shell.execute_reply":"2023-09-03T20:26:20.191940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Voila. Now we're done with our language model. But now we have a problem here. \n```\nThere is a small problem that 🤗 Transformers will not be happy about later on. The 5-gram correctly includes a \"Unknown\" or <unk>, as well as a begin-of-sentence, <s> token, but no end-of-sentence, </s> token. This sadly has to be corrected currently after the build.\n\nWe can simply add the end-of-sentence token by adding the line 0 </s> -0.11831701 below the begin-of-sentence token and increasing the ngram 1 count by 1\n```\n\n> [Reference](https://huggingface.co/blog/wav2vec2-with-ngram)","metadata":{}},{"cell_type":"markdown","source":"Let's fix it","metadata":{}},{"cell_type":"code","source":"with open(\"5gram.arpa\", \"r\") as read_file, open(\"5gram_correct.arpa\", \"w\") as write_file:\n    has_added_eos = False\n    for line in read_file:\n        if not has_added_eos and \"ngram 1=\" in line:\n            count=line.strip().split(\"=\")[-1]\n            write_file.write(line.replace(f\"{count}\", f\"{int(count)+1}\"))\n        elif not has_added_eos and \"<s>\" in line:\n            write_file.write(line)\n            write_file.write(line.replace(\"<s>\", \"</s>\"))\n            has_added_eos = True\n        else:\n            write_file.write(line)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:26:20.200907Z","iopub.execute_input":"2023-09-03T20:26:20.201323Z","iopub.status.idle":"2023-09-03T20:27:49.957803Z","shell.execute_reply.started":"2023-09-03T20:26:20.201289Z","shell.execute_reply":"2023-09-03T20:27:49.956449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm 5gram.arpa","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:27:49.959602Z","iopub.execute_input":"2023-09-03T20:27:49.960010Z","iopub.status.idle":"2023-09-03T20:27:52.220065Z","shell.execute_reply.started":"2023-09-03T20:27:49.959977Z","shell.execute_reply":"2023-09-03T20:27:52.218180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay now our LM is done! But wait, what do we do with this LM? It's of no use seperately, we need to integrate it with our wav2vec2 model","metadata":{}},{"cell_type":"markdown","source":"# Combine LM with wav2vec2","metadata":{}},{"cell_type":"markdown","source":"We'll use the [yellowking model](https://www.kaggle.com/code/sameen53/yellowking-dlsprint-inference) as our demo wav2vec2 model. It already came with a LM, we'll remove it and use our LM here. ","metadata":{}},{"cell_type":"code","source":"!cp -av /kaggle/input/yellowking-dlsprint-model/YellowKing_processor .","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:27:52.224494Z","iopub.execute_input":"2023-09-03T20:27:52.224958Z","iopub.status.idle":"2023-09-03T20:28:24.674867Z","shell.execute_reply.started":"2023-09-03T20:27:52.224921Z","shell.execute_reply":"2023-09-03T20:28:24.673160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -rf YellowKing_processor/language_model","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:28:24.677823Z","iopub.execute_input":"2023-09-03T20:28:24.678332Z","iopub.status.idle":"2023-09-03T20:28:26.465203Z","shell.execute_reply.started":"2023-09-03T20:28:24.678294Z","shell.execute_reply":"2023-09-03T20:28:26.463482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's load the processor without LM","metadata":{}},{"cell_type":"code","source":"from transformers import Wav2Vec2ProcessorWithLM,Wav2Vec2Processor\n\nprocessor = Wav2Vec2Processor.from_pretrained(\"/kaggle/working/YellowKing_processor\")\nvocab_dict = processor.tokenizer.get_vocab()\nsorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:28:26.467374Z","iopub.execute_input":"2023-09-03T20:28:26.467774Z","iopub.status.idle":"2023-09-03T20:28:29.356972Z","shell.execute_reply.started":"2023-09-03T20:28:26.467741Z","shell.execute_reply":"2023-09-03T20:28:29.355592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"``` \nThe \"labels\" and the previously built 5gram_correct.arpa file is all that's needed to build the decoder.\n```","metadata":{}},{"cell_type":"code","source":"from pyctcdecode import build_ctcdecoder\n\ndecoder = build_ctcdecoder(\n    labels=list(sorted_vocab_dict.keys()),\n    kenlm_model_path=\"5gram_correct.arpa\",\n)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:28:29.358552Z","iopub.execute_input":"2023-09-03T20:28:29.359212Z","iopub.status.idle":"2023-09-03T20:30:58.038051Z","shell.execute_reply.started":"2023-09-03T20:28:29.359179Z","shell.execute_reply":"2023-09-03T20:30:58.036890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's wrap the just created decoder, together with the processor's tokenizer and feature_extractor into a Wav2Vec2ProcessorWithLM class.","metadata":{}},{"cell_type":"code","source":"processor_with_lm = Wav2Vec2ProcessorWithLM(\n    feature_extractor=processor.feature_extractor,\n    tokenizer=processor.tokenizer,\n    decoder=decoder\n)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:30:58.039521Z","iopub.execute_input":"2023-09-03T20:30:58.039875Z","iopub.status.idle":"2023-09-03T20:30:58.046290Z","shell.execute_reply.started":"2023-09-03T20:30:58.039846Z","shell.execute_reply":"2023-09-03T20:30:58.044931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"yaay! We're done. Let's save the processor.","metadata":{}},{"cell_type":"code","source":"processor_with_lm.save_pretrained(\"Yellowking_Processor_New_LM\")","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:30:58.047821Z","iopub.execute_input":"2023-09-03T20:30:58.048193Z","iopub.status.idle":"2023-09-03T20:31:12.260455Z","shell.execute_reply.started":"2023-09-03T20:30:58.048162Z","shell.execute_reply":"2023-09-03T20:31:12.259023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -rf /kaggle/working/kenlm","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:31:12.262611Z","iopub.execute_input":"2023-09-03T20:31:12.263131Z","iopub.status.idle":"2023-09-03T20:31:13.607101Z","shell.execute_reply.started":"2023-09-03T20:31:12.263086Z","shell.execute_reply":"2023-09-03T20:31:13.605752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inference","metadata":{}},{"cell_type":"code","source":"class CFG:\n    my_model_name = '../input/yellowking-dlsprint-model/YellowKing_model'\n    processor_name = '/kaggle/working/Yellowking_Processor_New_LM'\n    processor_without_LM = '/kaggle/working/YellowKing_processor'\n    \nfrom transformers import Wav2Vec2ProcessorWithLM,pipeline\n\nprocessor = Wav2Vec2ProcessorWithLM.from_pretrained(CFG.processor_name)\nasr_w_LM = pipeline(\"automatic-speech-recognition\", model=CFG.my_model_name ,feature_extractor =processor.feature_extractor, tokenizer= processor.tokenizer,decoder=processor.decoder)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:31:13.610936Z","iopub.execute_input":"2023-09-03T20:31:13.611462Z","iopub.status.idle":"2023-09-03T20:34:30.491612Z","shell.execute_reply.started":"2023-09-03T20:31:13.611415Z","shell.execute_reply":"2023-09-03T20:34:30.490319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import librosa\ndef infer(audio_path):\n    speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n    my_LM_prediction = asr_w_LM(\n                speech\n            )\n\n    return normalize(my_LM_prediction['text'])","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:34:30.493206Z","iopub.execute_input":"2023-09-03T20:34:30.493570Z","iopub.status.idle":"2023-09-03T20:34:30.506968Z","shell.execute_reply.started":"2023-09-03T20:34:30.493537Z","shell.execute_reply":"2023-09-03T20:34:30.505766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"infer(\"/kaggle/input/bengaliai-speech/train_mp3s/00001e0bc131.mp3\")","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:34:30.508992Z","iopub.execute_input":"2023-09-03T20:34:30.509465Z","iopub.status.idle":"2023-09-03T20:34:46.267812Z","shell.execute_reply.started":"2023-09-03T20:34:30.509424Z","shell.execute_reply":"2023-09-03T20:34:46.266897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Performance comparison :  with LM and without LM","metadata":{}},{"cell_type":"code","source":"processor_without_LM = Wav2Vec2Processor.from_pretrained(CFG.processor_without_LM)\nasr_wo_LM = pipeline(\"automatic-speech-recognition\", model=CFG.my_model_name ,feature_extractor =processor_without_LM.feature_extractor, tokenizer= processor_without_LM.tokenizer)\n\ndef infer_wo_LM(audio_path):\n    speech, sr = librosa.load(audio_path, sr=processor.feature_extractor.sampling_rate)\n    my_LM_prediction = asr_wo_LM(\n                speech\n            )\n\n    return normalize(my_LM_prediction['text'])","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:34:46.269642Z","iopub.execute_input":"2023-09-03T20:34:46.271418Z","iopub.status.idle":"2023-09-03T20:34:51.096821Z","shell.execute_reply.started":"2023-09-03T20:34:46.271370Z","shell.execute_reply":"2023-09-03T20:34:51.095488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/bengaliai-speech/train.csv\")\nval = df[df.split==\"valid\"]\nval.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:34:51.098787Z","iopub.execute_input":"2023-09-03T20:34:51.099526Z","iopub.status.idle":"2023-09-03T20:34:57.737346Z","shell.execute_reply.started":"2023-09-03T20:34:51.099482Z","shell.execute_reply":"2023-09-03T20:34:57.736065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from jiwer import wer","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:34:57.739444Z","iopub.execute_input":"2023-09-03T20:34:57.739793Z","iopub.status.idle":"2023-09-03T20:34:57.833470Z","shell.execute_reply.started":"2023-09-03T20:34:57.739765Z","shell.execute_reply":"2023-09-03T20:34:57.832302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import IPython.display as ipd\n\ndef example(idx):\n    path = val['id'].iloc[idx]\n    full_path = \"/kaggle/input/bengaliai-speech/train_mp3s/\"+path+\".mp3\"\n    display(ipd.Audio(full_path))\n    \n    sentence = normalize(val.sentence.iloc[idx])\n    print(\"Grounf Truth : \",normalize(sentence))\n    print(\"Prediction With LM : \",infer(full_path))\n    print(\"Prediction Without LM : \",infer_wo_LM(full_path))\n    print(\"Word Error Rate with LM : \",wer(sentence,infer(full_path)))\n    print(\"Word Error Rate without LM : \",wer(sentence,infer_wo_LM(full_path)))\n    ","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:34:57.835271Z","iopub.execute_input":"2023-09-03T20:34:57.835625Z","iopub.status.idle":"2023-09-03T20:34:57.843431Z","shell.execute_reply.started":"2023-09-03T20:34:57.835597Z","shell.execute_reply":"2023-09-03T20:34:57.841901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 0\nexample(idx)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:34:57.845165Z","iopub.execute_input":"2023-09-03T20:34:57.845605Z","iopub.status.idle":"2023-09-03T20:35:13.325379Z","shell.execute_reply.started":"2023-09-03T20:34:57.845573Z","shell.execute_reply":"2023-09-03T20:35:13.324491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 5\nexample(idx)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:35:13.326870Z","iopub.execute_input":"2023-09-03T20:35:13.327714Z","iopub.status.idle":"2023-09-03T20:35:26.168274Z","shell.execute_reply.started":"2023-09-03T20:35:13.327681Z","shell.execute_reply":"2023-09-03T20:35:26.166918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 9\nexample(idx)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:35:26.175375Z","iopub.execute_input":"2023-09-03T20:35:26.175786Z","iopub.status.idle":"2023-09-03T20:35:32.041971Z","shell.execute_reply.started":"2023-09-03T20:35:26.175754Z","shell.execute_reply":"2023-09-03T20:35:32.040764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 12\nexample(idx)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:35:32.043511Z","iopub.execute_input":"2023-09-03T20:35:32.043980Z","iopub.status.idle":"2023-09-03T20:35:46.201064Z","shell.execute_reply.started":"2023-09-03T20:35:32.043943Z","shell.execute_reply":"2023-09-03T20:35:46.199840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 13\nexample(idx)","metadata":{"execution":{"iopub.status.busy":"2023-09-03T20:35:46.202925Z","iopub.execute_input":"2023-09-03T20:35:46.203308Z","iopub.status.idle":"2023-09-03T20:35:58.140822Z","shell.execute_reply.started":"2023-09-03T20:35:46.203275Z","shell.execute_reply":"2023-09-03T20:35:58.139381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see, the model with LM is performing better than without LM! ","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}