{"cells":[{"metadata":{"_uuid":"07ad96c71a881cca01c4d79bc5864b5eff44df7b"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"68ecc4db9c43a52caf4c590b6c22c688222e6a97"},"cell_type":"markdown","source":"### Intro\n\nLooking through the kernels in the competition I noticed most of the kernels were using an iterative way to calculate features. So I'm sharing a template for vectorizing the feature extraction to improve the performance. Hope it would be useful for you. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport pandas as pd\nimport numpy as np\nfrom tqdm import tqdm_notebook","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"%%time\ntrain_df = pd.read_csv('../input/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float32}) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd241a35e0ffaf30bb6a7061e7b5c7ee3a5a8714"},"cell_type":"markdown","source":"### The iterative way \nAs you might have seen in many of the kernels, one way of extracting features is to calculate segments of data and then iterate through each segment to calculate the features. Here I put a simple example (to extract only 6 features) "},{"metadata":{"trusted":true,"_uuid":"df400c9bb27146da161c77a744b2488aad71b23e"},"cell_type":"code","source":"# Segmenting \nrows = 150_000\nsegments = int(np.floor(train_df.shape[0] / rows))\nprint(\"Number of segments: \", segments)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"519e04f47201cb01ad3f07aa1a6abd0802698d60"},"cell_type":"code","source":"def create_features(seg_id, seg, X):\n    xc = pd.Series(seg['acoustic_data'].values)\n    \n    X.loc[seg_id, 'mean'] = xc.mean()\n    X.loc[seg_id, 'std'] = xc.std()\n    X.loc[seg_id, 'max'] = xc.max()\n    X.loc[seg_id, 'min'] = xc.min()\n    X.loc[seg_id, 'sum'] = xc.sum()\n    X.loc[seg_id, 'median'] = xc.median()\n    \n    \n    return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd9b8b83bc6c60f2a5ccdfcfc4b65be0f12cc07d"},"cell_type":"code","source":"train_X = pd.DataFrame(index=range(segments), dtype=np.float64)\ntrain_y = pd.DataFrame(index=range(segments), dtype=np.float64, columns=['time_to_failure'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"076c69f172e666799473416796b42613242fe9b3"},"cell_type":"code","source":"%%time\nfor seg_id in tqdm_notebook(range(segments)):\n    seg = train_df.iloc[seg_id*rows:seg_id*rows+rows]\n    create_features(seg_id, seg, train_X)\n    train_y.loc[seg_id, 'time_to_failure'] = seg['time_to_failure'].values[-1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4503f144b71a38d0c01d3d4a60a84b293ac21681"},"cell_type":"markdown","source":"### The vectorize way\nSo let's take a look at anothe way of handling the featur extraction. \n\nWe all know that Numpy is known for its performance in handling matrices, so let's leverage it to calcualte the features.\n\nIn this project, we only have one input parameter (acoustic data) and it makes it much easier to prepare the input matrix. \nThe idea is very seimple, we trim the input data to be dividable by our desired number rows (in this case 150,000) and then turn the 1D matrix into a 2D matrix and calculate features amongs its vertical axis. \n\nSo assuming our input data has **n** rows,  so our input matrix is (n x 1), we make it (m x 1) where  m = n -  n mod 150,000 and then reshape it to (150,000, x) where x = m / 150,000 \n\nNow we can calculate features amongst the axis =0 of the new matrix. \n"},{"metadata":{"trusted":true,"_uuid":"ae5ed41deed70e62b11cc6462ed18e42804dd7d9"},"cell_type":"code","source":"# This function makes sure the input matrix is dividable by the target number of rows\ndef prep_df_for_separation(df,rows):\n    mod_value = df.shape[0] % rows \n    if mod_value > 0:\n        lastRow = df.shape[0] - mod_value\n        df = df.iloc[:lastRow]\n    return df ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"383ba3b316a7311bc51e1e0dcaf077285c27cad9"},"cell_type":"markdown","source":"The following function is a very simple example of vectorized feature extraction. We know the input matrix shape is (rows, x) and we want to calculate features for each row so our output can be (x, n_features) "},{"metadata":{"trusted":true,"_uuid":"04675244976a7064349182a2a383095c90eeb841"},"cell_type":"code","source":"# an example of vactorized feature exraction function\ndef vectorized_features(data):\n    n_features = 6\n    output_matrix = np.empty(shape=(data.shape[1], n_features))\n\n    output_matrix[:,0] = np.mean(data,axis=0)\n    output_matrix[:,1] = np.std(data,axis=0)\n    output_matrix[:,2] = np.max(data,axis=0)\n    output_matrix[:,3] = np.min(data,axis=0)\n    output_matrix[:,4] = np.sum(data,axis=0)\n    output_matrix[:,5] = np.median(data,axis=0)\n    \n    return output_matrix","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5a4e698902c689b27c530105798a874f91a7970"},"cell_type":"markdown","source":"we use *prep_df_for_separation* function we defined earlier to prepare our train data frame and then separate x and y and process them separately. \n\nFor y, in this case we know the time_to_failure is reducing during each 150,000 section, so we can simply use min function to get the desired value (notice we used axis= 0) "},{"metadata":{"trusted":true,"_uuid":"157aaaf0ff8355417fa724bd8d09e5b61cac10e1"},"cell_type":"code","source":"%%time\ntrain_df = prep_df_for_separation(train_df,rows)\ndata_matrix = train_df.acoustic_data.values.reshape(-1,rows).T\noutput_matrix_all = train_df.time_to_failure.values.reshape(-1,rows).T\noutput_matrix = np.min(output_matrix_all,axis=0)\n\nprint(\"data matrix shape\", data_matrix.shape)\nprint(\"output matrix shape\", output_matrix.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3d513af291d2adbc4c5a006d135f18931f5e3e1"},"cell_type":"markdown","source":"Finally we can calculate features. You can compare its process time with the iterative method above and consider this is only for 6 features and if you would incrase the number of features, the difference woud be much more significant. "},{"metadata":{"trusted":true,"_uuid":"2c54f28f9ec95861ab23eecd9266e5d903d88711"},"cell_type":"code","source":"%%time\nfeatures = vectorized_features(data_matrix)\nprint(\"data matrix shape\", data_matrix.shape, \"\\t| Output matrix shape:\", output_matrix.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a3a32775fb7c8e25f63952f24ecba5e50b0de173"},"cell_type":"markdown","source":"### Conclusion\nSome of the features are a bit harder to calculate using this method. Moving averag for instance, is possible but requires forming a larger matrix which consumes memory. I'd add more feature ideas later on if requested. "},{"metadata":{"trusted":true,"_uuid":"f2d2ee61d53ea8ca355bc6ae1110c5dadad2f4eb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}