{"cells":[{"metadata":{"_uuid":"8462fb6798c7b3b46536d007092f6a9df3129b33"},"cell_type":"markdown","source":"****-This kernel is summary of kernesl I have seen for the dataset as well as some modification.****"},{"metadata":{"id":"LfMwWkTzBAuD","colab_type":"code","outputId":"93cbfa0f-39d5-481f-8ef2-f35c282ee812","executionInfo":{"status":"ok","timestamp":1552573228773,"user_tz":-60,"elapsed":6459,"user":{"displayName":"ehsan olyaee","photoUrl":"https://lh6.googleusercontent.com/-jgQPfjJNWIA/AAAAAAAAAAI/AAAAAAAACaw/Rwy-pWgjDUU/s64/photo.jpg","userId":"04555599038073960403"}},"colab":{"base_uri":"https://localhost:8080/","height":35},"trusted":true,"_uuid":"1ebbc647137573024e71b2bb1d567f2a90a606f3"},"cell_type":"code","source":"import keras\nimport keras.backend as K\nfrom keras.layers import LSTM,Dropout,Dense,TimeDistributed,Conv1D,MaxPooling1D,Flatten\nfrom keras.models import Sequential\nimport tensorflow as tf\nimport pandas as pd\n# import pyarrow as pa\nimport pyarrow.parquet as pq\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n#Both numpy and scipy has utilities for FFT which is an endlessly useful algorithm\nfrom numpy.fft import *\nfrom scipy import fftpack","execution_count":null,"outputs":[]},{"metadata":{"id":"JzZ86TEBBDnf","colab_type":"text","_uuid":"48b7b3ccaa3db95e5dfcdf1d73982a28d3213d80"},"cell_type":"markdown","source":"**Signals:**\n\n- 800.000 measurement points for 8712 signals.\n- The signals are **three-phased** so there are 2904 distinct signaling instances.\n- Three phase signals:\n    - Sums to zero.\n    - When one fails other continue to carry the current.\n    - Can be rectified to be converted to DC current.\n    - Ripples in rectification can be seen on failure.\n    \n![](https://upload.wikimedia.org/wikipedia/commons/thumb/4/48/3-phase_flow.gif/357px-3-phase_flow.gif)"},{"metadata":{"id":"67XYLSIQBHxQ","colab_type":"text","_uuid":"9d13520f50e8ecfcf161a9e98542a58091cb67d3"},"cell_type":"markdown","source":"### What is partial discharge?\n\n - Typical situation of PD: imagine there is an internal cavity/void or **impurity in insulation**. \n - When **High Voltage** is applied on conductor, a field is also induced on the cavity. Further, when the field increases, this **defect breaks down** and **discharges** different forms of energy which result in partial discharge.\n - This phenomenon is damaging over a long period of time. It is not event that occurs suddenly. "},{"metadata":{"id":"XPyOvEquBKQ2","colab_type":"text","_uuid":"37ef420b3b791d5b62557cc824df3802e28a76e5"},"cell_type":"markdown","source":"### Classical Modes of Detection\n- Partial Discharges can be detected by **measuring the emissions** they give off: Ultrasonic Sound, Transient Earth Voltages (TEV and UHF energy).\n- Is it possible to enhance the modes of detection by **better feature extraction** for the classifiers?\n- **Intel Mobile ODT** challenge on 2017 was about topping **classical image processing** methods by automatic feature extaction using pre-trained CNN models and **transfer learning**.\n- **Two possible approaches**:\n    - FE on signals and feeding them into NNs for classification.\n    - Using NNs further as feature extractors and then use shallow classifiers (XGBoost) for binary classification\n"},{"metadata":{"id":"9BrZyDgIBMuf","colab_type":"text","_uuid":"53bbe20e3deff3739da71dd6b26448439f9573a6"},"cell_type":"markdown","source":"### **TASK:** Classify long-term failure of covered conductors based on signal characteristics:\n- Extract features from time series data for classification.\n- Use **CNN** for further FE and **LSTM** to get temporal dependencies and perform time series classification on the top layer."},{"metadata":{"id":"fyaMmwjEBPnl","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":72},"outputId":"b8054b42-2653-4024-ce1e-52d6497c9786","executionInfo":{"status":"ok","timestamp":1552573245913,"user_tz":-60,"elapsed":19731,"user":{"displayName":"ehsan olyaee","photoUrl":"https://lh6.googleusercontent.com/-jgQPfjJNWIA/AAAAAAAAAAI/AAAAAAAACaw/Rwy-pWgjDUU/s64/photo.jpg","userId":"04555599038073960403"}},"trusted":true,"_uuid":"cc18bd9f66d11fd59c184df453d200efe98d6820"},"cell_type":"code","source":"signals = pq.read_table('../input/train.parquet', columns=[str(i) for i in range(999)]).to_pandas()\nprint('signals shape is: ',signals.shape)\n#Since data size is big we just load one third of it for now\nsignals = np.array(signals).T.reshape((999//3, 3, 800000))\nprint('signals shape after reshaping is: ', signals.shape)\ntrain_df = pd.read_csv('../input/metadata_train.csv')\nprint('metadata shape is: ',train_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"id":"-QT9CIr8E0UK","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":745},"outputId":"c4345861-19f0-4571-b634-783f820284f0","executionInfo":{"status":"ok","timestamp":1552571993369,"user_tz":-60,"elapsed":15660,"user":{"displayName":"ehsan olyaee","photoUrl":"https://lh6.googleusercontent.com/-jgQPfjJNWIA/AAAAAAAAAAI/AAAAAAAACaw/Rwy-pWgjDUU/s64/photo.jpg","userId":"04555599038073960403"}},"trusted":true,"_uuid":"200e13ba2b55d3ca9af2659a28c2ebd14c5aa9f0"},"cell_type":"code","source":"fig, axs = plt.subplots(2, 2, constrained_layout=True,figsize=(15, 10))\n\naxs[0,0].set_title('normal wires')\naxs[0,0].plot(signals[0, 0, :], label='Phase 0')\naxs[0,0].plot(signals[0, 1, :], label='Phase 1')\naxs[0,0].plot(signals[0, 2, :], label='Phase 2')\naxs[0,0].legend()\n\naxs[1,0].set_title('damaged wires')\naxs[1,0].plot(signals[1, 0, :], label='Phase 0')\naxs[1,0].plot(signals[1, 1, :], label='Phase 1')\naxs[1,0].plot(signals[1, 2, :], label='Phase 2')\naxs[1,0].legend()\n\naxs[0,1].set_title('normal wires')\naxs[0,1].plot(signals[77, 0, :], label='Phase 0')\naxs[0,1].plot(signals[77, 1, :], label='Phase 1')\naxs[0,1].plot(signals[77, 2, :], label='Phase 2')\naxs[0,1].legend()\n\naxs[1,1].set_title('damaged wires')\naxs[1,1].plot(signals[76, 0, :], label='Phase 0')\naxs[1,1].plot(signals[76, 1, :], label='Phase 1')\naxs[1,1].plot(signals[76, 2, :], label='Phase 2')\naxs[1,1].legend()\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"aQqtyOveeJ_W","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":680},"outputId":"1eb94423-a799-4d41-cf39-9fc951728df6","executionInfo":{"status":"ok","timestamp":1552572573901,"user_tz":-60,"elapsed":2793,"user":{"displayName":"ehsan olyaee","photoUrl":"https://lh6.googleusercontent.com/-jgQPfjJNWIA/AAAAAAAAAAI/AAAAAAAACaw/Rwy-pWgjDUU/s64/photo.jpg","userId":"04555599038073960403"}},"trusted":true,"_uuid":"7a19685167d48d7a8a5c473bd539782ae15216a5"},"cell_type":"code","source":"target = train_df['target']\nplt.figure(figsize=(15, 10))\nsns.countplot(target)\nplt.show()\n\nprint('number of damaged samples', sum(target))\nprint('number of normal sampes', target.shape[0]-sum(target))","execution_count":null,"outputs":[]},{"metadata":{"id":"Xzntzc9eg6Ep","colab_type":"code","colab":{},"trusted":true,"_uuid":"5da755b630abf6a3f17499877d9dfd59a5bc3cbe"},"cell_type":"code","source":"#FFT to filter out HF components and get main signal profile\ndef low_pass(s, threshold=1e4):\n    fourier = rfft(s)\n    frequencies = rfftfreq(s.size, d=2e-2/s.size)\n    fourier[frequencies > threshold] = 0\n    return irfft(fourier)","execution_count":null,"outputs":[]},{"metadata":{"id":"9IUNz4QAj6AB","colab_type":"code","colab":{},"trusted":true,"_uuid":"6d34a6bb62c1368158631318bba4763aa544e3e0"},"cell_type":"code","source":"#normal one\nlf_normal1_1 = low_pass(signals[0, 0, :])\nlf_normal1_2 = low_pass(signals[0, 1, :])\nlf_normal1_3 = low_pass(signals[0, 2, :])\n#normal two\nlf_normal2_1 = low_pass(signals[77, 0, :])\nlf_normal2_2 = low_pass(signals[77, 1, :])\nlf_normal2_3 = low_pass(signals[77, 2, :])\n#damaged one\nlf_damaged1_1 = low_pass(signals[1, 0, :])\nlf_damaged1_2 = low_pass(signals[1, 1, :])\nlf_damaged1_3 = low_pass(signals[1, 2, :])\n#damaged two\nlf_damaged2_1 = low_pass(signals[76, 0, :])\nlf_damaged2_2 = low_pass(signals[76, 1, :])\nlf_damaged2_3 = low_pass(signals[76, 2, :])","execution_count":null,"outputs":[]},{"metadata":{"id":"GXHp4pLUlEE_","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":745},"outputId":"ae2d4955-4a2d-416a-b28a-462eba6572be","executionInfo":{"status":"ok","timestamp":1552575687660,"user_tz":-60,"elapsed":38979,"user":{"displayName":"ehsan olyaee","photoUrl":"https://lh6.googleusercontent.com/-jgQPfjJNWIA/AAAAAAAAAAI/AAAAAAAACaw/Rwy-pWgjDUU/s64/photo.jpg","userId":"04555599038073960403"}},"trusted":true,"_uuid":"150151d161fce2fb6144393d20905eba21c40b82"},"cell_type":"code","source":"fig, axs = plt.subplots(2, 2, constrained_layout=True,figsize=(15, 10))\n\naxs[0,0].set_title('normal wires')\naxs[0,0].plot(lf_normal1_1, label='Phase 0')\naxs[0,0].plot(lf_normal1_2, label='Phase 1')\naxs[0,0].plot(lf_normal1_3, label='Phase 2')\naxs[0,0].legend()\n\naxs[1,0].set_title('damaged wires')\naxs[1,0].plot(lf_damaged1_1, label='Phase 0')\naxs[1,0].plot(lf_damaged1_2, label='Phase 1')\naxs[1,0].plot(lf_damaged1_3, label='Phase 2')\naxs[1,0].legend()\n\naxs[0,1].set_title('normal wires')\naxs[0,1].plot(lf_normal2_1, label='Phase 0')\naxs[0,1].plot(lf_normal2_2, label='Phase 1')\naxs[0,1].plot(lf_normal2_3, label='Phase 2')\naxs[0,1].legend()\n\naxs[1,1].set_title('damaged wires')\naxs[1,1].plot(lf_damaged2_1, label='Phase 0')\naxs[1,1].plot(lf_damaged2_2, label='Phase 1')\naxs[1,1].plot(lf_damaged2_3, label='Phase 2')\naxs[1,1].legend()\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"he4tc1XxmeCX","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":745},"outputId":"d1ddfd46-d33a-4c0e-bce7-d96e5621e90f","executionInfo":{"status":"ok","timestamp":1552574210687,"user_tz":-60,"elapsed":58682,"user":{"displayName":"ehsan olyaee","photoUrl":"https://lh6.googleusercontent.com/-jgQPfjJNWIA/AAAAAAAAAAI/AAAAAAAACaw/Rwy-pWgjDUU/s64/photo.jpg","userId":"04555599038073960403"}},"trusted":true,"_uuid":"19bfd7412f9ad28d651cd5051844f06797c34c11"},"cell_type":"code","source":"fig, axs = plt.subplots(2, 2, constrained_layout=True,figsize=(15, 10))\n\naxs[0,0].set_title('normal wires')\naxs[0,0].plot((np.abs(lf_normal1_1)+np.abs(lf_normal1_2)+np.abs(lf_normal1_3)))\naxs[0,0].plot(lf_normal1_1, label='Phase 0')\naxs[0,0].plot(lf_normal1_2, label='Phase 1')\naxs[0,0].plot(lf_normal1_3, label='Phase 2')\naxs[0,0].legend()\n\naxs[1,0].set_title('damaged wires')\naxs[1,0].plot((np.abs(lf_damaged1_1)+np.abs(lf_damaged1_2)+np.abs(lf_damaged1_3)))\naxs[1,0].plot(lf_damaged1_1, label='Phase 0')\naxs[1,0].plot(lf_damaged1_2, label='Phase 1')\naxs[1,0].plot(lf_damaged1_3, label='Phase 2')\naxs[1,0].legend()\n\naxs[0,1].set_title('normal wires')\naxs[0,1].plot((np.abs(lf_normal2_1)+np.abs(lf_normal2_2)+np.abs(lf_normal2_3)))\naxs[0,1].plot(lf_normal2_1, label='Phase 0')\naxs[0,1].plot(lf_normal2_2, label='Phase 1')\naxs[0,1].plot(lf_normal2_3, label='Phase 2')\naxs[0,1].legend()\n\naxs[1,1].set_title('damaged wires')\naxs[1,1].plot((np.abs(lf_damaged2_1)+np.abs(lf_damaged2_2)+np.abs(lf_damaged2_3)))\naxs[1,1].plot(lf_damaged2_1, label='Phase 0')\naxs[1,1].plot(lf_damaged2_2, label='Phase 1')\naxs[1,1].plot(lf_damaged2_3, label='Phase 2')\naxs[1,1].legend()\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"S86kAZ2UovO4","colab_type":"code","colab":{},"trusted":true,"_uuid":"f4ba1c32a2cc4b407f9df2d353b131f2c8623b69"},"cell_type":"code","source":"###Filter out low frequencies from the signal to get HF characteristics\ndef high_pass(s, threshold=1e7):\n    fourier = rfft(s)\n    frequencies = rfftfreq(s.size, d=2e-2/s.size)\n    fourier[frequencies < threshold] = 0\n    return irfft(fourier)","execution_count":null,"outputs":[]},{"metadata":{"id":"YyRY4YM2o0pX","colab_type":"code","colab":{},"trusted":true,"_uuid":"e2480a87519665aa64b19a84de999c98e7b5fcaa"},"cell_type":"code","source":"#normal one\nhf_normal1_1 = high_pass(signals[0, 0, :])\nhf_normal1_2 = high_pass(signals[0, 1, :])\nhf_normal1_3 = high_pass(signals[0, 2, :])\n#normal two\nhf_normal2_1 = high_pass(signals[77, 0, :])\nhf_normal2_2 = high_pass(signals[77, 1, :])\nhf_normal2_3 = high_pass(signals[77, 2, :])\n#damaged one\nhf_damaged1_1 = high_pass(signals[1, 0, :])\nhf_damaged1_2 = high_pass(signals[1, 1, :])\nhf_damaged1_3 = high_pass(signals[1, 2, :])\n#damaged two\nhf_damaged2_1 = high_pass(signals[76, 0, :])\nhf_damaged2_2 = high_pass(signals[76, 1, :])\nhf_damaged2_3 = high_pass(signals[76, 2, :])","execution_count":null,"outputs":[]},{"metadata":{"id":"DnmcuncBpAuQ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":745},"outputId":"78de333d-6049-4bf5-bd1d-945ac0bff681","executionInfo":{"status":"ok","timestamp":1552574402581,"user_tz":-60,"elapsed":12133,"user":{"displayName":"ehsan olyaee","photoUrl":"https://lh6.googleusercontent.com/-jgQPfjJNWIA/AAAAAAAAAAI/AAAAAAAACaw/Rwy-pWgjDUU/s64/photo.jpg","userId":"04555599038073960403"}},"trusted":true,"_uuid":"d4dd6ed6f5990713c5db31a46e6474ce624463c0"},"cell_type":"code","source":"fig, axs = plt.subplots(2, 2, constrained_layout=True,figsize=(15, 10))\n\naxs[0,0].set_title('normal wires')\naxs[0,0].plot(hf_normal1_1, label='Phase 0')\n# axs[0,0].plot(hf_normal1_2, label='Phase 1')\n# axs[0,0].plot(hf_normal1_3, label='Phase 2')\naxs[0,0].legend()\n\naxs[1,0].set_title('damaged wires')\naxs[1,0].plot(hf_damaged1_1, label='Phase 0')\naxs[1,0].plot(hf_damaged1_2, label='Phase 1')\naxs[1,0].plot(hf_damaged1_3, label='Phase 2')\naxs[1,0].legend()\n\naxs[0,1].set_title('normal wires')\naxs[0,1].plot(hf_normal2_1, label='Phase 0')\naxs[0,1].plot(hf_normal2_2, label='Phase 1')\naxs[0,1].plot(hf_normal2_3, label='Phase 2')\naxs[0,1].legend()\n\naxs[1,1].set_title('damaged wires')\naxs[1,1].plot(hf_damaged2_1, label='Phase 0')\naxs[1,1].plot(hf_damaged2_2, label='Phase 1')\naxs[1,1].plot(hf_damaged2_3, label='Phase 2')\naxs[1,1].legend()\n\n\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"colab":{"name":"VSB.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}