{"cells":[{"metadata":{},"cell_type":"markdown","source":"### What is this?\nI demonstrate here how to fix malformed lines in train&test dataset.\n\n#### Malformed line example\nAt line 623 in the train file below. There are **two** data in **one line**, TYPE_BEACON and TYPE_ACCELEROMETER."},{"metadata":{"trusted":true},"cell_type":"code","source":"!sed '623q;d' /kaggle/input/indoor-location-navigation/train/5cd56ba1e2acfd2d33b603af/B2/5cf75ac5e36a480008125f9d.txt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's create fixed directory and copy the train file to the directory so that we can compare before & after."},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir fixed\n!cp /kaggle/input/indoor-location-navigation/train/5cd56ba1e2acfd2d33b603af/B2/5cf75ac5e36a480008125f9d.txt fixed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's run the following sed command to fix the malformed line. Note that the command fix the file inplace because we use -i option."},{"metadata":{"trusted":true},"cell_type":"code","source":"!cd fixed && find ./ -type f -exec sed -i -e 's/[^^]\\([[:digit:]]\\{13\\}\\)\\tTYPE/\\n\\1\\tTYPE/g' {} \\;","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Alright, let's use diff command to see before and after. It looks great."},{"metadata":{"trusted":true},"cell_type":"code","source":"!diff -ur /kaggle/input/indoor-location-navigation/train/5cd56ba1e2acfd2d33b603af/B2/5cf75ac5e36a480008125f9d.txt fixed/5cf75ac5e36a480008125f9d.txt","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}