{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### About this kernel\nThis kernel ONLY describe the data preparation scripts for preparing necessary files to run on Kaldi and it cannot run end to end on Kaggle kernel. I will not say any further about how to run it on Kaldi. There is a detail tutorial [here](http://kaldi-asr.org/doc/kaldi_for_dummies.html).\nI'm also getting familar with Kaldi for nearly a month, so please feel free to discuss and make question or suggestion to have a better result.\nThe folder to put on running script (run.sh, data, etc...) is uploaded on this [Github repository](https://github.com/minhnq97/asr-commands)."},{"metadata":{},"cell_type":"markdown","source":"## Data preparation\nSo let begin with creating data for training.\nJust to remind, preparing data for Kaldi needs three files:\n* *wav.scp*: Each line of file is followed by pattern: utterance_id path_to_audio\n* *text*: Pattern of file is: utterance_id transcript\n* *utt2spk*: Pattern of file is: utterance_id speaker"},{"metadata":{},"cell_type":"markdown","source":"### Training and validation data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os.path\nimport random\n\nall_info = []\ntranscript = {}\npath = '../input/train/audio/'\n\nall_info = []\ncount = 0\n# r=root, d=directories, f = files\nfor r, d, f in os.walk(path):\n    for file in f:\n        if '.wav' in file:\n            count+=1\n            trans = r.split(\"/\")[-1]\n            file_id = file.split(\".\")[0] + \"_\" + trans\n            spk_id = file_id.split(\"_\")[0]\n            transcript[file_id] = trans\n            all_info.append([spk_id,file_id,os.path.join(r, file)])\n\ncounter = int(len(all_info) * 0.1)\nrandom.shuffle(all_info)\nall_train_info = all_info[counter:]\nall_test_info = all_info[:counter]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Collect all data in a list [*speaker_id*, *utterance_id*, *file_to_audio*]  \nChoose 10% from them to be a test list. (Although I named it test list, it should be considered as validation list).    \n**Note** that to avoid some error from Kaldi, the utterance_id should begin with the speaker_id."},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists(os.path.dirname('data/train_command/text')):\n    os.makedirs(os.path.dirname('data/train_command/text'))\nif not os.path.exists(os.path.dirname('data/test_command/text')):\n    os.makedirs(os.path.dirname('data/test_command/text'))\n\ndef text(file_infos):\n    results = []\n    # folder_path = os.path.abspath(\"recordings\")\n    for info in file_infos:\n        utt_id = info[1]\n        trans = transcript[utt_id]\n        results.append(\"{} {}\".format(utt_id, trans))\n    return '\\n'.join(sorted(results))\n\nwith open(\"data/train_command/text\",\"wt\") as f:\n    f.writelines(text(all_train_info))\nwith open(\"data/test_command/text\",\"wt\") as f:\n    f.writelines(text(all_test_info))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create text file with format: utterance_id -> transcript"},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists(os.path.dirname('data/train_command/wav.scp')):\n    os.makedirs(os.path.dirname('data/train_command/wav.scp'))\nif not os.path.exists(os.path.dirname('data/test_command/wav.scp')):\n    os.makedirs(os.path.dirname('data/test_command/wav.scp'))\n\ndef wavscp(file_infos):\n    results = []\n    for info in file_infos:\n        results.append(\"{} {}\".format(info[1], info[2]))\n    return '\\n'.join(sorted(results))\n\nwith open(\"data/train_command/wav.scp\",\"wt\") as f:\n    f.writelines(wavscp(all_train_info))\nwith open(\"data/test_command/wav.scp\",\"wt\") as f:\n    f.writelines(wavscp(all_test_info))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create wav.scp file with format: utterance_id -> audio path"},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists(os.path.dirname('data/train_command/utt2spk')):\n    os.makedirs(os.path.dirname('data/train_command/utt2spk'))\nif not os.path.exists(os.path.dirname('data/test_command/utt2spk')):\n    os.makedirs(os.path.dirname('data/test_command/utt2spk'))\n\ndef utt2spk(file_infos):\n    results = []\n    for info in file_infos:\n        speaker = info[0]\n        utt_id = info[1]\n        results.append(\"{} {}\".format(utt_id, speaker))\n    return '\\n'.join(sorted(results))\n\nwith open(\"data/train_command/utt2spk\",\"wt\") as f:\n    f.writelines(utt2spk(all_train_info))\nwith open(\"data/test_command/utt2spk\",\"wt\") as f:\n    f.writelines(utt2spk(all_test_info))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create utt2spk file with format: utterance_id -> speaker"},{"metadata":{},"cell_type":"markdown","source":"### Evaluation data"},{"metadata":{"trusted":true},"cell_type":"code","source":"all_info = []\ntranscript = {}\npath = '../input/data/test'\n\nall_info = []\ncount = 0\n# r=root, d=directories, f = files\nfor r, d, f in os.walk(path):\n    for file in f:\n        if '.wav' in file:\n            count+=1\n            file_name = file.split(\".\")[0]\n            spk_id = file_name.split(\"_\")[1]\n            all_info.append([spk_id,spk_id + \"_\" + file_name,os.path.join(r, file)])\n\n\nif not os.path.exists(os.path.dirname('data/eval_command/wav.scp')):\n    os.makedirs(os.path.dirname('data/eval_command/wav.scp'))\n\ndef wavscp(file_infos):\n    results = []\n    for info in file_infos:\n        results.append(\"{} {}\".format(info[1], info[2]))\n    return '\\n'.join(sorted(results))\n\nwith open(\"data/eval_command/wav.scp\",\"wt\") as f:\n    f.writelines(wavscp(all_info))\n\n\nif not os.path.exists(os.path.dirname('data/eval_command/utt2spk')):\n    os.makedirs(os.path.dirname('data/eval_command/utt2spk'))\n\ndef utt2spk(file_infos):\n    results = []\n    for info in file_infos:\n        speaker = info[0]\n        utt_id = info[1]\n        results.append(\"{} {}\".format(utt_id, speaker))\n    return '\\n'.join(sorted(results))\n\nwith open(\"data/eval_command/utt2spk\",\"wt\") as f:\n    f.writelines(utt2spk(all_info))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Similarly, evaluation data contain utt2spk and wav.scp file. "},{"metadata":{},"cell_type":"markdown","source":"### Prepare language data\nLanguage data required for running Kaldi contain 4 files:  \n* *lexicon.txt*: Contain every word in the dataset and its phonemes. Pattern: <word> <phone1> <phone2>...\n* *nonsilence_phones*: Every phonemes you have. Pattern: <phone>\n* *optional_phones*: List of optional silence phone. I use only <sil>\n* *silence_phones*: List of silence phone. I also use only <sil>\n"},{"metadata":{},"cell_type":"markdown","source":"So the most important part is how to create your own phoneme list. For me, I list all the label and search the phoneme on this [website](http://www.speech.cs.cmu.edu/tools/lextool.html).  \nMy *lexicon.txt* is here:  \n> bed\tb eh d  \nbird\tb er d  \ncat\tk ae t  \ndog\td ao g  \ndown\td aw n  \neight\tey t  \nfive\tf ay v  \nfour\tf ao r  \ngo\tg ow  \nhappy\thh ae p iy  \nhouse\thh aw s  \nleft\tl eh f t  \nmarvin\tm aa r v ih n  \nnine\tn ay n  \nno\tn ow  \noff\tao f  \non\taa n  \non\tao n  \none\tw ah n  \none\thh w ah n  \nright\tr ay t  \nseven\ts eh v ah n  \nsheila\tsh iy l ah  \nsix\ts ih k s  \nstop\ts t aa p  \nthree\tth r iy  \ntree\tt r iy  \ntwo\tt uw  \nup\tah p  \nwow\tw aw  \nyes\ty eh s  \nzero\tz iy r ow  "},{"metadata":{},"cell_type":"markdown","source":"## Running script on Kaldi"},{"metadata":{},"cell_type":"markdown","source":"The detail of how to run the script is in the README of Github repository, which I gave on the top of this kernel.  \nI just want to note that I'm using HMM + GMM and train with triphone.  \nThe result of validation set is relatively **5% WER** (Word Error Average) after finish the training."},{"metadata":{},"cell_type":"markdown","source":"## Postprocess for submission"},{"metadata":{},"cell_type":"markdown","source":"After training, you can file the output process at log folder from Kaldi. Something like this: *$kaldi_path/egs/command/exp/tri3b/decode*/log/*  \nIn this log folder, there are many files decode.*.log (the number of files depends on how much number of job you use for decoding in Kaldi script, it should be lower than your CPU-core and lower than the number of speakers). And each file of decode.*.log will look like:  \n\n>000044442_clip_000044442 no   \nLOG (gmm-latgen-faster[5.5.382~1-c2163]:DecodeUtteranceLatticeFaster():decoder-wrappers.cc:289) Log-like per frame for utterance 000044442_clip_000044442 is -3.39615 over 98 frames.  \n0000adecb_clip_0000adecb happy   \nLOG (gmm-latgen-faster[5.5.382~1-c2163]:DecodeUtteranceLatticeFaster():decoder-wrappers.cc:289) Log-like per frame for utterance 0000adecb_clip_0000adecb is -4.63466 over 98 frames."},{"metadata":{},"cell_type":"markdown","source":"There will be the utterance_id<whitespace>transcript, and if the transcript is an empty string, we will consider it as silence sound.  \nLet make a sample of it! We will create a *decode.1.log* file inside *log/* folder."},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists(os.path.dirname('log/decode.1.log')):\n    os.makedirs(os.path.dirname('log/decode.1.log'))\n    \nwith open(\"log/decode.1.log\",\"wt\") as f:\n    f.write(\"000044442_clip_000044442 no\\n\")\n    f.write(\"LOG (gmm-latgen-faster[5.5.382~1-c2163]:DecodeUtteranceLatticeFaster():decoder-wrappers.cc:289) Log-like per frame for utterance 000044442_clip_000044442 is -3.39615 over 98 frames.\\n\")\n    f.write(\"0000adecb_clip_0000adecb happy\\n\")\n    f.write(\"LOG (gmm-latgen-faster[5.5.382~1-c2163]:DecodeUtteranceLatticeFaster():decoder-wrappers.cc:289) Log-like per frame for utterance 0000adecb_clip_0000adecb is -4.63466 over 98 frames.\\n\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cat log/decode.1.log","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The postprocess script have to handle this file to get a submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"import re\n\nall_info = []\npath = 'log' #Path to log folder\neval = {}\ncount = 0\npattern = r'.{9}_clip_.{9}.*'\n\ndef _read_decode_file(filepath):\n    with open(filepath, \"rt\") as f:\n        for line in f.read().splitlines():\n            if line.startswith(\"LOG\"):\n                continue\n            x = re.search(pattern,line)\n            if x is not None:\n                res = x.group(0)\n                info = res.split()\n                utt_id = info[0]\n                wav_id = utt_id[10:] + \".wav\"\n                if len(info) == 1:\n                    trans = \"silence\"\n                else:\n                    trans = \" \".join(info[1:])\n                eval[wav_id] = trans\n    pass\n\n\n# r=root, d=directories, f = files\nfor r, d, f in os.walk(path):\n    for file in f:\n        if 'decode.' in file:\n            count+=1\n            _read_decode_file(\"/\".join([r,file]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To keep the order of submission, I will read the file id from sample_submission.csv and get the relative transcript from the above dictionary."},{"metadata":{"trusted":true},"cell_type":"code","source":"all_lines=[]\nwith open(\"../input/sample_submission.csv\",\"rt\") as f:\n    for line in f.read().splitlines():\n        if line.startswith(\"fname\"):\n            all_lines.append(line)\n            continue\n        line = line.split(\",\")\n        try:\n            trans = eval[line[0]]\n        except KeyError:\n            trans = \"silence\"\n        all_lines.append(\",\".join([line[0],trans]))\n\nwith open(\"submission.csv\",\"wt\") as f:\n    f.writelines(\"\\n\".join(all_lines))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\nThis kernel is only a reference if you want to run the challenge by Kaldi. I do search keyword Kaldi on Kaggle but it seems not very popular, so I decided to make this as a tutorial. However please give me some advises if you see something wrong.  \nThe final result I got is 75% on Public LB and 77% on Private LB."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}