{"cells":[{"metadata":{"trusted":true,"_uuid":"7a6d5ef92752b663ef29b835642045c69f5892df","_kg_hide-input":false},"cell_type":"code","source":"import numpy as np\nimport pandas as pd","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"11c6c3b8ecc742ae2803903cbf08b38f262a2cd1"},"cell_type":"markdown","source":"This kernel does not predict anything. Instead, it tries to address a problem that many newcomers to this competition may have: **The training data is *huge*, simply reading the full csv at once may be too much**. If you want to play around with the data on a local machine, you need something better. Fortunately, the data can be encoded much more efficiently:\n* time_to_failure is *for all practical purposes* (see explanation below) linear and does not need to be saved. It can be generated by numpy.linspace as needed\n* acoustic_data can be saved in a compressed npz file\n\nThe resulting file will be ~400MB in total, can be easily saved in the cloud, contains the full acoustic_data and can be loaded in ~ 10 sec on most machines.\n\n**Run this notebook as a Kaggle kernel and download the output files *train_acoustic_data.npz* and *train_info.csv* .**\n\nEDIT: 2019-04-17: added the same procedure for the test data as well."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"%time df_train = pd.read_csv('../input/train.csv', dtype = {'acoustic_data': np.int16, 'time_to_failure': np.float32} ) # float32 is enough :)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5f3f65366c11f6e9368532ee9e257c490ca938c"},"cell_type":"markdown","source":"Yes, that took forever.\n\nThere are 17 \"earthquake periods\" (the last one ends long before its quake). Let's find the beginning of each period:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"ttf = df_train['time_to_failure'].values\nindex_start = np.nonzero(np.diff(ttf) > 0)[0] + 1\nindex_start = np.insert(index_start, 0, 0) # insert 1st period start manually\nprint(index_start)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"276bde195bf3509b701d8af016593cf3ba0f2246"},"cell_type":"markdown","source":"How many data points are in each period?"},{"metadata":{"trusted":true,"_uuid":"15167c4fb2a72bf46aba7df0382f767eb40e3f89"},"cell_type":"code","source":"chunk_length = np.diff(np.append(index_start, df_train.shape[0]))\nprint(chunk_length)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e0196ecaf4e5b7ac4b234066796181623bf3588"},"cell_type":"markdown","source":"What are the time_to_failure values of the 1st and last data point in each period?"},{"metadata":{"trusted":true,"_uuid":"ac19f006567370a6eb489e9908232df17f0c4ce3"},"cell_type":"code","source":"t_start = ttf[index_start]\nt_end = ttf[index_start + chunk_length - 1]\nprint('t_start =', t_start)\nprint('t_end =', t_end)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3130b3dcf3da7196b92ca7f2c5dac36895334d04"},"cell_type":"markdown","source":"Assuming that the acoustic_data is contiguous (that's probably not strictly true, see the discussions, but it doesn't really matter anyway), what is the length of the time steps between data points in each period? The values should be approximately equal, and they are:"},{"metadata":{"trusted":true,"_uuid":"6bc58632303bf246a108ee7ea4ae63105aa15c00"},"cell_type":"code","source":"dt_step = (t_start - t_end) / chunk_length\ndt_step","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9aa6657f718fc236a039cc8e2f99352bee2d2953"},"cell_type":"markdown","source":"Now it gets interesting. If we approximate the time_to_failure column for each period with a numpy.linspace from t_start to t_end, what is the maximal difference (i.e. error) between this approximation and the original data? Result: 1.15 ms. **This is 3 orders of magnitude below the prediction accuracy that we can reasonably hope to achieve (O(1 sec)). We don't need the original time_to_failure column anymore.**"},{"metadata":{"trusted":true,"_uuid":"00b652875ad700d709bc1417f04afe25b3555077"},"cell_type":"code","source":"linsp_diff_max = []\nfor i in range(len(index_start)):\n ttf_orig = ttf[index_start[i] : index_start[i] + chunk_length[i]]\n ttf_linsp = np.linspace(t_start[i], t_end[i], chunk_length[i])\n linsp_diff_max.append(np.abs(ttf_orig - ttf_linsp).max())\nlinsp_diff_max = np.array(linsp_diff_max)\nprint(linsp_diff_max)\nprint('Max error =', linsp_diff_max.max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f2e3342d12661d893238eac832acadca86f815b"},"cell_type":"markdown","source":"Let's collect all of this into a dataframe which will be saved (we will need it again to generate time_to_failure!)"},{"metadata":{"trusted":true,"_uuid":"1f639dddc899edb636415a7f8d987b4a04d235e8"},"cell_type":"code","source":"df_train_info = pd.DataFrame({\n 'index_start':index_start,\n 'chunk_length':chunk_length,\n 't_start':t_start,\n 't_end':t_end,\n 'dt_step':dt_step,\n 'linsp_diff_max':linsp_diff_max\n})\ndf_train_info.to_csv('train_info.csv', index=False)\ndf_train_info","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"99df2fe10f235f7487502c4f3ee146dc4e127e30"},"cell_type":"markdown","source":"Let's save the acoustic data in compressed form. This is only done once. Yep, we're below 400MB:"},{"metadata":{"trusted":true,"_uuid":"09b6a85edffdcbc53f0e35b14075364da4f167e2"},"cell_type":"code","source":"%time np.savez_compressed('train_acoustic_data.npz', acoustic_data=df_train['acoustic_data'].values)\n%ls -lh","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7a827272a08011455dec6b8eed2f79118fe3eb79"},"cell_type":"markdown","source":"Now we're done. From now on, you only need *train_acoustic_data.npz* , *train_info.csv* and a function to extract a quake period and generate the time_to_failure data:"},{"metadata":{"trusted":true,"_uuid":"20511b664c30c3148a4e92e6dfc08056bd1e5964"},"cell_type":"code","source":"%%time\ndf_train_info = pd.read_csv('train_info.csv')\nac_data = np.load('train_acoustic_data.npz')['acoustic_data']\n\ndef get_quake_period(i):\n index_start, chunk_length = df_train_info['index_start'][i], df_train_info['chunk_length'][i]\n t_start, t_end = df_train_info['t_start'][i], df_train_info['t_end'][i]\n ac_data_period = ac_data[ index_start : index_start + chunk_length ]\n ttf_data_period = np.linspace(t_start, t_end, chunk_length, dtype=np.float32)\n return ac_data_period, ttf_data_period","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa3d9d8d5d9006c25be9ea6adfd26c9008fbc02b"},"cell_type":"markdown","source":"Let's extract period #3:"},{"metadata":{"trusted":true,"_uuid":"c8b59ade56a24bdc2b415ee3ea0a3fa943adbeb4"},"cell_type":"code","source":"get_quake_period(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3796b0e2698fa0089b8faeb893e486c1941577e0"},"cell_type":"markdown","source":"That's all. Happy Predicting!"},{"metadata":{},"cell_type":"markdown","source":"Well, almost: let's bring the huge amount of litte test files into a manageable format as well:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_ssub = pd.read_csv('../input/sample_submission.csv')\ndf_ssub.tail()","execution_count":2,"outputs":[{"output_type":"execute_result","execution_count":2,"data":{"text/plain":" seg_id time_to_failure\n2619 seg_ff4236 0\n2620 seg_ff7478 0\n2621 seg_ff79d9 0\n2622 seg_ffbd6a 0\n2623 seg_ffe7cc 0","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
seg_idtime_to_failure
2619seg_ff42360
2620seg_ff74780
2621seg_ff79d90
2622seg_ffbd6a0
2623seg_ffe7cc0
\n
"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Pack the 2624 test segments into a 2624 x 150000 numpy array and save it:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ac_data = []\n\nfor fname in df_ssub['seg_id'].values:\n ac_data.append(pd.read_csv('../input/test/' + fname + '.csv').acoustic_data.values.astype(np.int16))\n \nac_data = np.array(ac_data)\n\nnp.savez_compressed('test_acoustic_data.npz', acoustic_data=ac_data)\n\nprint('test data shape :', ac_data.shape)","execution_count":15,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}