{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Universidad de Antioquia\n## Especialización en Analítica y Ciencia de Datos\n\nIntegrantes:\n\nJose Daniel Alvear Acevedo\n\nAlfonso Cubillos Delgado","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(url='https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Three_Phase_Electric_Power_Transmission.jpg/1200px-Three_Phase_Electric_Power_Transmission.jpg')","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2021-06-01T22:41:54.726076Z","iopub.execute_input":"2021-06-01T22:41:54.726536Z","iopub.status.idle":"2021-06-01T22:41:54.734743Z","shell.execute_reply.started":"2021-06-01T22:41:54.726501Z","shell.execute_reply":"2021-06-01T22:41:54.733789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **--> Primera Iteración**","metadata":{}},{"cell_type":"markdown","source":"# **Contexto del problema**","metadata":{}},{"cell_type":"markdown","source":"Las líneas eléctricas aéreas de media tensión recorren cientos de millas para suministrar energía a las ciudades. Estas grandes distancias hacen que sea costoso inspeccionar manualmente las líneas en busca de daños que no provoquen inmediatamente un corte de energía, como una rama de un árbol golpeando la línea o una falla en el aislante. Estos modos de daño conducen a un fenómeno conocido como descarga parcial, una descarga eléctrica que no une completamente los electrodos entre un sistema de aislamiento. Las descargas parciales dañan lentamente la línea eléctrica, por lo que si no se reparan, eventualmente provocarán un corte de energía o provocarán un incendio.\n\nSu desafío es detectar patrones de descarga parcial en señales adquiridas de estas líneas eléctricas con un nuevo medidor diseñado en el Centro ENET en VŠB . Los clasificadores efectivos que utilizan estos datos permitirán monitorear continuamente las líneas eléctricas en busca de fallas.\n\nEl Centro ENET investiga y desarrolla recursos energéticos renovables con el objetivo de reducir o eliminar los impactos ambientales nocivos. Sus esfuerzos se centran en desarrollar soluciones tecnológicas en torno al transporte y procesamiento de materias primas energéticas.\n\nAl desarrollar una solución para detectar descargas parciales, ayudará a reducir los costos de mantenimiento y evitará cortes de energía.","metadata":{}},{"cell_type":"markdown","source":"# **Importanto librerias**","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport pyarrow.parquet as pq # Leer archivos parquet\nimport matplotlib.pyplot as plt\nimport os\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:42:08.189187Z","iopub.execute_input":"2021-06-01T22:42:08.189664Z","iopub.status.idle":"2021-06-01T22:42:09.292177Z","shell.execute_reply.started":"2021-06-01T22:42:08.189633Z","shell.execute_reply":"2021-06-01T22:42:09.291374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Tomamos las 2000 primeras filas**","metadata":{}},{"cell_type":"code","source":"INIT_DIR = '../input'\ntamaño = 2001","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:42:32.114916Z","iopub.execute_input":"2021-06-01T22:42:32.115239Z","iopub.status.idle":"2021-06-01T22:42:32.119816Z","shell.execute_reply.started":"2021-06-01T22:42:32.115211Z","shell.execute_reply":"2021-06-01T22:42:32.118543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pq.read_pandas(os.path.join(INIT_DIR, 'vsb-power-line-fault-detection/train.parquet'), columns=[str(i) for i in range(tamaño)]).to_pandas()\nmetadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:42:35.448421Z","iopub.execute_input":"2021-06-01T22:42:35.4488Z","iopub.status.idle":"2021-06-01T22:42:44.954367Z","shell.execute_reply.started":"2021-06-01T22:42:35.448768Z","shell.execute_reply":"2021-06-01T22:42:44.953684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.tail()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:42:48.547675Z","iopub.execute_input":"2021-06-01T22:42:48.548274Z","iopub.status.idle":"2021-06-01T22:42:48.572732Z","shell.execute_reply.started":"2021-06-01T22:42:48.548225Z","shell.execute_reply":"2021-06-01T22:42:48.571946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:43:13.009951Z","iopub.execute_input":"2021-06-01T22:43:13.010425Z","iopub.status.idle":"2021-06-01T22:43:13.01527Z","shell.execute_reply.started":"2021-06-01T22:43:13.010386Z","shell.execute_reply":"2021-06-01T22:43:13.01461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metadata.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:43:18.795588Z","iopub.execute_input":"2021-06-01T22:43:18.796046Z","iopub.status.idle":"2021-06-01T22:43:18.805959Z","shell.execute_reply.started":"2021-06-01T22:43:18.796016Z","shell.execute_reply":"2021-06-01T22:43:18.804797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metadata.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:43:21.73727Z","iopub.execute_input":"2021-06-01T22:43:21.737619Z","iopub.status.idle":"2021-06-01T22:43:21.742449Z","shell.execute_reply.started":"2021-06-01T22:43:21.737588Z","shell.execute_reply":"2021-06-01T22:43:21.74181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata = metadata[:tamaño]\ntrain_metadata","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:43:28.57556Z","iopub.execute_input":"2021-06-01T22:43:28.576023Z","iopub.status.idle":"2021-06-01T22:43:28.586919Z","shell.execute_reply.started":"2021-06-01T22:43:28.575993Z","shell.execute_reply":"2021-06-01T22:43:28.586239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:43:35.015587Z","iopub.execute_input":"2021-06-01T22:43:35.016081Z","iopub.status.idle":"2021-06-01T22:43:35.021877Z","shell.execute_reply.started":"2021-06-01T22:43:35.01604Z","shell.execute_reply":"2021-06-01T22:43:35.020883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cada columna representa una señal, por lo tanto para manejar el mismo formato de pandas, trasponemos la base de datos","metadata":{}},{"cell_type":"code","source":"train = train.T\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:43:40.199077Z","iopub.execute_input":"2021-06-01T22:43:40.199458Z","iopub.status.idle":"2021-06-01T22:43:40.205828Z","shell.execute_reply.started":"2021-06-01T22:43:40.199425Z","shell.execute_reply":"2021-06-01T22:43:40.205076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:46:09.711287Z","iopub.execute_input":"2021-06-01T22:46:09.711851Z","iopub.status.idle":"2021-06-01T22:46:09.744588Z","shell.execute_reply.started":"2021-06-01T22:46:09.711806Z","shell.execute_reply":"2021-06-01T22:46:09.743628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Adicionamos el Id de la señal en el dataframe principal","metadata":{}},{"cell_type":"code","source":"train['signal_id'] = list(train_metadata['signal_id'])","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:51:45.734433Z","iopub.execute_input":"2021-06-01T22:51:45.734932Z","iopub.status.idle":"2021-06-01T22:51:45.849705Z","shell.execute_reply.started":"2021-06-01T22:51:45.73489Z","shell.execute_reply":"2021-06-01T22:51:45.848992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:51:47.372594Z","iopub.execute_input":"2021-06-01T22:51:47.373098Z","iopub.status.idle":"2021-06-01T22:51:47.440087Z","shell.execute_reply.started":"2021-06-01T22:51:47.373057Z","shell.execute_reply":"2021-06-01T22:51:47.439187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Unión de metadatos y datos de señales basados en signal_id**","metadata":{}},{"cell_type":"code","source":"train = train.merge(train_metadata, on='signal_id')","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:51:50.259546Z","iopub.execute_input":"2021-06-01T22:51:50.260097Z","iopub.status.idle":"2021-06-01T22:51:53.200067Z","shell.execute_reply.started":"2021-06-01T22:51:50.260049Z","shell.execute_reply":"2021-06-01T22:51:53.199334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:51:58.921209Z","iopub.execute_input":"2021-06-01T22:51:58.921724Z","iopub.status.idle":"2021-06-01T22:51:58.953822Z","shell.execute_reply.started":"2021-06-01T22:51:58.921679Z","shell.execute_reply":"2021-06-01T22:51:58.952899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Revisando valores nulos en el dataframe**","metadata":{}},{"cell_type":"code","source":"train.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:52:10.303227Z","iopub.execute_input":"2021-06-01T22:52:10.303555Z","iopub.status.idle":"2021-06-01T22:52:16.676732Z","shell.execute_reply.started":"2021-06-01T22:52:10.303527Z","shell.execute_reply":"2021-06-01T22:52:16.675559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Gráficos de conteo frente a gráficos de destino para verificar el desequilibrio de datos**","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 4))\nsns.countplot(x=\"target\", data=train, ax=ax1)\nsns.countplot(x=\"target\", data=train, hue=\"phase\", ax=ax2);","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:53:05.306975Z","iopub.execute_input":"2021-06-01T22:53:05.307331Z","iopub.status.idle":"2021-06-01T22:53:05.725048Z","shell.execute_reply.started":"2021-06-01T22:53:05.307303Z","shell.execute_reply":"2021-06-01T22:53:05.724107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_count = train.target.value_counts()\nprint(\"negative(target=0) target: {}\".format(target_count[0]))\nprint(\"positive(target=1) target: {}\".format(target_count[1]))\nprint(\"positive data {:.3}%\".format((target_count[1]/(target_count[0]+target_count[1]))*100))","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:56:06.37503Z","iopub.execute_input":"2021-06-01T22:56:06.37566Z","iopub.status.idle":"2021-06-01T22:56:06.385954Z","shell.execute_reply.started":"2021-06-01T22:56:06.375609Z","shell.execute_reply":"2021-06-01T22:56:06.385264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[['id_measurement', 'phase']]","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:56:27.137488Z","iopub.execute_input":"2021-06-01T22:56:27.138025Z","iopub.status.idle":"2021-06-01T22:56:27.158432Z","shell.execute_reply.started":"2021-06-01T22:56:27.137977Z","shell.execute_reply":"2021-06-01T22:56:27.157613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_mismatch = train[[\"id_measurement\", \"target\"]].groupby([\"id_measurement\"]).sum().query(\"target != 3 & target != 0\")\nprint(\"Valores de la variable objetivo que no son positivos ni negativos: {}\".format(target_mismatch.shape[0]))\ntarget_mismatch","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:56:33.334617Z","iopub.execute_input":"2021-06-01T22:56:33.334965Z","iopub.status.idle":"2021-06-01T22:56:33.358098Z","shell.execute_reply.started":"2021-06-01T22:56:33.334937Z","shell.execute_reply":"2021-06-01T22:56:33.357167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Revisemos el Id == 67, donde la variable objetivo es diferente en diferentes fases","metadata":{}},{"cell_type":"code","source":"train[train['id_measurement'] == 67]","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:56:42.195112Z","iopub.execute_input":"2021-06-01T22:56:42.19547Z","iopub.status.idle":"2021-06-01T22:56:42.248701Z","shell.execute_reply.started":"2021-06-01T22:56:42.195439Z","shell.execute_reply":"2021-06-01T22:56:42.247706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observación:**\n\nLas variables objetivo pueden ser diferentes para la misma señal en diferentes fases","metadata":{}},{"cell_type":"markdown","source":"**Hallamos el único valor del id_measurement**","metadata":{}},{"cell_type":"code","source":"print(\"id_measurement tiene {} valores únicos\".format(train.id_measurement.nunique()))","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:56:56.764511Z","iopub.execute_input":"2021-06-01T22:56:56.764834Z","iopub.status.idle":"2021-06-01T22:56:56.771029Z","shell.execute_reply.started":"2021-06-01T22:56:56.764806Z","shell.execute_reply":"2021-06-01T22:56:56.770014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Descripción de la columna id_measurement**","metadata":{}},{"cell_type":"code","source":"train.id_measurement.value_counts().describe()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:57:02.413717Z","iopub.execute_input":"2021-06-01T22:57:02.414051Z","iopub.status.idle":"2021-06-01T22:57:02.426794Z","shell.execute_reply.started":"2021-06-01T22:57:02.414019Z","shell.execute_reply":"2021-06-01T22:57:02.425858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Valores únicos de la columna phase**","metadata":{}},{"cell_type":"code","source":"print(\"La variable fase tiene {} valores únicos en train {}\".format(len(train.phase.unique()),train.phase.unique()))","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:57:22.055972Z","iopub.execute_input":"2021-06-01T22:57:22.056302Z","iopub.status.idle":"2021-06-01T22:57:22.063197Z","shell.execute_reply.started":"2021-06-01T22:57:22.056274Z","shell.execute_reply":"2021-06-01T22:57:22.062107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(train['phase']);","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:57:31.819912Z","iopub.execute_input":"2021-06-01T22:57:31.820315Z","iopub.status.idle":"2021-06-01T22:57:31.934452Z","shell.execute_reply.started":"2021-06-01T22:57:31.82028Z","shell.execute_reply":"2021-06-01T22:57:31.933502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Graficando las señales**","metadata":{}},{"cell_type":"markdown","source":"**Graficando una señal normal**","metadata":{}},{"cell_type":"code","source":"train.loc[1]['target']","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:57:41.566996Z","iopub.execute_input":"2021-06-01T22:57:41.567299Z","iopub.status.idle":"2021-06-01T22:57:42.316271Z","shell.execute_reply.started":"2021-06-01T22:57:41.567273Z","shell.execute_reply":"2021-06-01T22:57:42.315418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[1].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:57:47.589352Z","iopub.execute_input":"2021-06-01T22:57:47.589704Z","iopub.status.idle":"2021-06-01T22:57:48.608421Z","shell.execute_reply.started":"2021-06-01T22:57:47.589674Z","shell.execute_reply":"2021-06-01T22:57:48.6074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Grficando una señal defectuosa**","metadata":{}},{"cell_type":"code","source":"train.loc[201]['target']","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:57:59.565867Z","iopub.execute_input":"2021-06-01T22:57:59.566189Z","iopub.status.idle":"2021-06-01T22:58:00.290291Z","shell.execute_reply.started":"2021-06-01T22:57:59.566161Z","shell.execute_reply":"2021-06-01T22:58:00.289225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[201].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:58:03.993598Z","iopub.execute_input":"2021-06-01T22:58:03.994075Z","iopub.status.idle":"2021-06-01T22:58:05.042601Z","shell.execute_reply.started":"2021-06-01T22:58:03.994036Z","shell.execute_reply":"2021-06-01T22:58:05.041798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Graficando las tres fases de una señal**","metadata":{}},{"cell_type":"code","source":"train.loc[0:2][['target', 'id_measurement']]","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:58:22.326222Z","iopub.execute_input":"2021-06-01T22:58:22.326579Z","iopub.status.idle":"2021-06-01T22:58:22.344557Z","shell.execute_reply.started":"2021-06-01T22:58:22.326549Z","shell.execute_reply":"2021-06-01T22:58:22.343499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[0].values), alpha=0.7);\nplt.plot((train.loc[1].values), alpha=0.7);\nplt.plot((train.loc[2].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:58:27.450399Z","iopub.execute_input":"2021-06-01T22:58:27.450723Z","iopub.status.idle":"2021-06-01T22:58:30.154793Z","shell.execute_reply.started":"2021-06-01T22:58:27.450685Z","shell.execute_reply":"2021-06-01T22:58:30.153764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Graficando las tres fases de una señal defectuosa**","metadata":{}},{"cell_type":"code","source":"train.loc[3:5][['target', 'id_measurement']]","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:58:38.095349Z","iopub.execute_input":"2021-06-01T22:58:38.095858Z","iopub.status.idle":"2021-06-01T22:58:38.112498Z","shell.execute_reply.started":"2021-06-01T22:58:38.095825Z","shell.execute_reply":"2021-06-01T22:58:38.111508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[3].values), alpha=0.7);\nplt.plot((train.loc[4].values), alpha=0.7);\nplt.plot((train.loc[5].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"execution":{"iopub.status.busy":"2021-06-01T22:58:49.0125Z","iopub.execute_input":"2021-06-01T22:58:49.012808Z","iopub.status.idle":"2021-06-01T22:58:51.732155Z","shell.execute_reply.started":"2021-06-01T22:58:49.012781Z","shell.execute_reply":"2021-06-01T22:58:51.731316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Primera Iteración**","metadata":{}},{"cell_type":"markdown","source":"Dado el tamaño de los datos, se recomienda reiniciar el entorno para ejecutar las sigientes líneas.","metadata":{}},{"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cargamos los datos nuevamente","metadata":{}},{"cell_type":"code","source":"train_data = pq.read_pandas(os.path.join(INIT_DIR, 'vsb-power-line-fault-detection/train.parquet')).to_pandas()\ntrain_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"Ahora realizamos un análisis exploratorio de los datos","metadata":{}},{"cell_type":"code","source":"train_metadata.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.tail(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vamos a tomar el primer 1% de los datos para realizar una primera iteración del modelo, ya que no poseemos recursos computacionales para poder tomar la base de datos completa.","metadata":{}},{"cell_type":"code","source":"# Generar codigo para carga total de los datos \n\n\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train_data.iloc[:8000]\ntarget = train_metadata.target[:8000]\nprint(train.shape)\nprint(target.shape)\ntrain","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A continuación realizaremos un análisis de componentes principales (PCA por sus siglas en inglés) para determinar si podemos utilizar una menor cantidad de variables.","metadata":{}},{"cell_type":"code","source":"from sklearn.decomposition import PCA\npca = PCA(n_components=0.5, whiten=True)\n\nX_pca = pca.fit_transform(train)\n\nprint('Número original de atributos:', train.shape[1])\nprint('Número reducido de atributos:', X_pca.shape[1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Con el análisis previo, vamos a implementar inicialmente una regresión logística como primera iteración del conjunto de datos.\n\nPara ello, vamos a utilizar la regresión logística con los datos originales, con el PCA y los datos estandarizados.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import accuracy_score\n\nX_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=42)\n\nmodel = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=100)\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\nprint('Precisión de datos originales:', accuracy_score(y_test, y_pred))\n#######\n\nX_train, X_test, y_train, y_test = train_test_split(X_pca, target, test_size=0.3, random_state=42)\n\nmodel = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=100)\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\nprint('Precisión de datos reducidos:', accuracy_score(y_test, y_pred))\n\nX_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=42)\n\npca = PCA(n_components=0.5, whiten=True)\n\nX_train = pca.fit_transform(X_train)\nX_test = pca.transform(X_test)\n\nmodel = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=100)\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\nprint('Precisión de datos originales estandarizados:', accuracy_score(y_test, y_pred))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Plan de trabajo para las siguientes iteraciones**","metadata":{}},{"cell_type":"markdown","source":"De los anteriores resultados, podemos observar que se obtuvo una precisión muy alta en los tres modelos. Por una análisis posterior podemos afirmar que el modelo se ve afectado por el desbalanceo de los datos, por lo tanto el conjunto de entrenamiento y validación no se están particionando de forma adecuada.\n\nAsí, para las siguientes iteraciones vamos a aplicar técnicas para el análisis de datos desbalanceados tales como: submuestreo inteligente, sobremuestreo inteligente, muestreo durante la validación y pesos diferentes para el error durante el entrenamiento.\n\nAdemás determinar si existe independencia entre las muestras para así poder realizar un análisis suponiendo normalidad de los datos y poder tomar muestras las cuales no van a interferir con las demás.       ","metadata":{}},{"cell_type":"markdown","source":"**--------------------------------------------------------------------------------------------------------------------------------------------**","metadata":{}},{"cell_type":"markdown","source":"# **--> Segunda Iteración**","metadata":{}},{"cell_type":"markdown","source":"Luego del analisis realizado en la iteración 1, se determina que no es posible hacer un entrenamiento optimo con solo el 1% de los datos; por lo tanto se toman medidas drasticas para abordar el problema","metadata":{}},{"cell_type":"markdown","source":"**--------------- Primera parte--------------------**","metadata":{}},{"cell_type":"markdown","source":"**Importamos las librerias**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport pyarrow.parquet as pq\nfrom sklearn.linear_model import SGDRegressor\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, scale, RobustScaler\nimport gc\nfrom skimage.restoration import denoise_wavelet\nimport scipy.signal as signal\nimport scipy.stats as stats\nimport time\nimport itertools\n\npd.set_option(\"max_columns\", 200)\npd.set_option(\"max_rows\", 200)\ngc.enable()","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:25:06.994250Z","iopub.execute_input":"2021-06-12T02:25:06.994660Z","iopub.status.idle":"2021-06-12T02:25:09.014237Z","shell.execute_reply.started":"2021-06-12T02:25:06.994628Z","shell.execute_reply":"2021-06-12T02:25:09.013674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def metrica(y_true, y_pred):\n    assert y_true.shape[0] == y_pred.shape[0]\n    \n    tp = np.sum((y_true == 1) & (y_pred == 1))\n    tn = np.sum((y_true == 0) & (y_pred == 0))\n    fp = np.sum((y_true == 0) & (y_pred == 1))\n    fn = np.sum((y_true == 1) & (y_pred == 0))\n\n    numerator = (tp * tn - fp * fn) \n    denominator = ((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)) ** .5\n\n    return numerator / (denominator + 1e-15)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_metadata():\n    \"\"\"\n    Lectura de los metadatos de Entrenamiento y de Test\n    \"\"\"\n    train_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')\n    test_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_test.csv')\n    return (train_metadata, test_metadata)\n\ndef resample_train():\n    \"\"\"\n    Lectura del archivo que contiene las mediciones de las señales, se trasponen las columnas y se agrega \n    la columna target de los datasets anteriores.\n    \"\"\"\n    data = pq.read_pandas('../input/vsb-power-line-fault-detection/train.parquet').to_pandas().transpose()\n    target = read_metadata()[0]['target'].values\n    data['target'] = target\n    p_indices = data[data.target == 0].index\n    np.random.seed(311)\n    random_indices = np.random.choice(p_indices, 1777, replace=False)\n    df = pd.concat([data.loc[random_indices][['target']], \n                    data[target == 1][['target']]]).sample(frac=1.0, random_state=311)\n    df.to_csv('train_us_target.csv', index=False)\n    return df.index","metadata":{"execution":{"iopub.status.busy":"2021-05-29T20:38:35.218478Z","iopub.execute_input":"2021-05-29T20:38:35.219369Z","iopub.status.idle":"2021-05-29T20:38:35.231641Z","shell.execute_reply.started":"2021-05-29T20:38:35.21932Z","shell.execute_reply":"2021-05-29T20:38:35.230429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata, test_metadata = read_metadata()\n","metadata":{"execution":{"iopub.status.busy":"2021-05-29T20:38:37.490765Z","iopub.execute_input":"2021-05-29T20:38:37.491569Z","iopub.status.idle":"2021-05-29T20:38:37.557737Z","shell.execute_reply.started":"2021-05-29T20:38:37.491523Z","shell.execute_reply":"2021-05-29T20:38:37.556626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.tail()","metadata":{"execution":{"iopub.status.busy":"2021-05-29T20:41:13.780315Z","iopub.execute_input":"2021-05-29T20:41:13.780735Z","iopub.status.idle":"2021-05-29T20:41:13.794135Z","shell.execute_reply.started":"2021-05-29T20:41:13.780702Z","shell.execute_reply":"2021-05-29T20:41:13.792191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr_limits = []\nstart = 0\nend = 8712\nwhile True:\n    if (start+1000) <= 8712:\n        tr_limits.append((start, start+1000))\n        start=start+1000\n    else:\n        tr_limits.append((start, end))\n        break\n        \nts_limits = []\nstart = 8712\nend = 29049\nwhile True:\n    if (start+1000) <= 29049:\n        ts_limits.append((start, start+1000))\n        start=start+1000\n    else:\n        ts_limits.append((start, end))\n        break","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:25:13.643451Z","iopub.execute_input":"2021-06-12T02:25:13.643726Z","iopub.status.idle":"2021-06-12T02:25:13.650572Z","shell.execute_reply.started":"2021-06-12T02:25:13.643703Z","shell.execute_reply":"2021-06-12T02:25:13.649667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def peak_calc(isTrain, filterSignal=True):\n    \"\"\"\n    Calculo de picos de las señales\n    \n    \"\"\"\n    print(\"peak_calc | start:\", time.strftime(\"%H:%M\"))\n    peak_counts = {}\n    # peak_counts expected values: [[1,3,5,1,6], [4,5,2,2,1], ..]\n    \n    def f1(df):\n        nonlocal peak_counts\n        df = df.transpose().values\n        def filter_sg(sg):\n            wv = denoise_wavelet(sg, sigma=None, wavelet='haar', mode='hard', \n                                 wavelet_levels=15, multichannel=False, \n                                 convert2ycbcr=False, method='VisuShrink') #ELiminando el Ruido ------>\n            B, A = signal.butter(N=3, Wn=0.1, output='ba')\n            smooth_data = signal.filtfilt(B, A, wv)\n            res = (wv - smooth_data) * 100\n            return res\n        \n        if filterSignal:\n            df = np.apply_along_axis(filter_sg, 1, df) #Aplicar una función a cortes 1-D a lo largo del eje dado.\n        \n        print(\"height\", time.strftime(\"%H:%M\"))\n        for h in [3, 5, 8, 10, 15, 25, 50, 100]:\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=h)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            if ('height_more_' + str(h)) in peak_counts:\n                peak_counts['height_more_' + str(h)].extend(num_peaks)\n            else:\n                peak_counts['height_more_' + str(h)] = num_peaks\n        \n        print(\"threshold\", time.strftime(\"%H:%M\"))\n        for t in [3, 10, 16, 50]:\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, threshold=t)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            if ('threshold_more_' + str(t)) in peak_counts:\n                peak_counts['threshold_more_' + str(t)].extend(num_peaks)\n            else:\n                peak_counts['threshold_more_' + str(t)] = num_peaks\n                \n        print(\"height and distance\", time.strftime(\"%H:%M\"))\n        for h, d in itertools.product([3, 5, 8, 10, 15, 25, 50, 100],\n                                      [5, 7, 25, 75, 111, 1000, 11111, 100000]):\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=h, distance=d)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            col_name = 'h_bt_' + str(h) + '_dist_bt_' + str(d)\n            if (col_name) in peak_counts:\n                peak_counts[col_name].extend(num_peaks)\n            else:\n                peak_counts[col_name] = num_peaks\n                \n        print(\"height and width\", time.strftime(\"%H:%M\"))\n        for h, w in itertools.product([3, 5, 8, 10, 15, 25, 50, 100],\n                                      [5, 10, 30, 70, 100, 200, 500]):\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=h, width=w, rel_height=0.4)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            col_name = 'h_bt_' + str(h) + '_w_bt_' + str(w)\n            if (col_name) in peak_counts:\n                peak_counts[col_name].extend(num_peaks)\n            else:\n                peak_counts[col_name] = num_peaks\n        del peaks\n        \n        print(\"max/min widths/proms\", time.strftime(\"%H:%M\"))\n        max_wdth_pk = []\n        min_wdth_pk = []\n        max_prom_pk = []\n        min_prom_pk = []\n        for n in range(df.shape[0]):\n            seq = df[n, :]\n            peaks, _ = signal.find_peaks(seq, height=7, threshold=None, distance=None, width=None)\n            if len(peaks) == 0:\n                pk_proms = [[0]]\n                pk_wdths = [0]\n            else:\n                pk_proms = signal.peak_prominences(seq, peaks, wlen=None)\n                pk_wdths = signal.peak_widths(seq, peaks, rel_height=0.4, \n                                              prominence_data=pk_proms, wlen=None)[0]\n            max_wdth_pk.append(max(pk_wdths))\n            min_wdth_pk.append(min(pk_wdths))\n            max_prom_pk.append(max(pk_proms[0]))\n            min_prom_pk.append(min(pk_proms[0]))\n        if ('max_peak_width') in peak_counts:\n            peak_counts['max_peak_width'].extend(max_wdth_pk)\n        else:\n            peak_counts['max_peak_width'] = max_wdth_pk\n        if ('min_peak_width') in peak_counts:\n            peak_counts['min_peak_width'].extend(min_wdth_pk)\n        else:\n            peak_counts['min_peak_width'] = min_wdth_pk\n        if ('max_peak_prom') in peak_counts:\n            peak_counts['max_peak_prom'].extend(max_prom_pk)\n        else:\n            peak_counts['max_peak_prom'] = max_prom_pk\n        if ('min_peak_prom') in peak_counts:\n            peak_counts['min_peak_prom'].extend(min_prom_pk)\n        else:\n            peak_counts['min_peak_prom'] = min_prom_pk\n            \n    if isTrain:\n        limits = tr_limits\n        path = '../input/vsb-power-line-fault-detection/train.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f1(df)\n    else:\n        limits = ts_limits\n        path = '../input/vsb-power-line-fault-detection/test.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f1(df)\n            \n    del df; gc.collect()\n    print(\"peak_calc | end:\", time.strftime(\"%H:%M\"))\n    return peak_counts","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pulse_stats(isTrain):\n    \"\"\"\n    Estadisticas de las señales\n    \n    \"\"\"\n    print(\"pulse_stats | start:\", time.strftime(\"%H:%M\"))\n    sg_max = np.array([], dtype=np.float16)\n    sg_min = np.array([], dtype=np.float16)\n    sg_mean = np.array([], dtype=np.float16)\n    sg_std = np.array([], dtype=np.float16)\n    perc1 = np.array([], dtype=np.float16)\n    perc3 = np.array([], dtype=np.float16)\n    perc5 = np.array([], dtype=np.float16)\n    perc7 = np.array([], dtype=np.float16)\n    perc9 = np.array([], dtype=np.float16)\n    sg_skew = np.array([], dtype=np.float16)\n    def f2(df):\n        nonlocal sg_max, sg_min, sg_mean, sg_std, perc1, perc3, perc5, perc7, perc9, sg_skew\n        df = df.transpose()\n        sg_max = np.append(sg_max, df.max(axis=1))\n        sg_min = np.append(sg_min, df.min(axis=1))\n        sg_mean = np.append(sg_mean, df.mean(axis=1))\n        sg_std = np.append(sg_std, df.apply(func=np.std, axis=1, raw=True))\n        perc1 = np.append(perc1, df.quantile(q=0.1, axis=1))\n        perc3 = np.append(perc3, df.quantile(q=0.3, axis=1))\n        perc5 = np.append(perc5, df.quantile(q=0.5, axis=1))\n        perc7 = np.append(perc7, df.quantile(q=0.7, axis=1))\n        perc9 = np.append(perc9, df.quantile(q=0.9, axis=1))\n        sg_skew = np.append(sg_skew, df.apply(func=stats.skew, axis=1, raw=True))\n\n    if isTrain:\n        limits = tr_limits\n        path = '../input/vsb-power-line-fault-detection/train.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f2(df)\n    else:\n        limits = ts_limits\n        path = '../input/vsb-power-line-fault-detection/test.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f2(df)\n    del df; gc.collect()\n    res = pd.DataFrame({'sg_max': sg_max,\n                        'sg_min': sg_min,\n                        'sg_mean': sg_mean,\n                        'sg_std': sg_std,\n                        'perc1': perc1,\n                        'perc3': perc3,\n                        'perc5': perc5,\n                        'perc7': perc7,\n                        'perc9': perc9,\n                        'sg_skew': sg_skew\n                       })\n    print(\"pulse_stats | end:\", time.strftime(\"%H:%M\"))\n    return res","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Nota:**\n* Estos bucles son bastante demorados (Calculo de picos), ya que se esta construyendo las bases de datos ('final_v1_tr' y 'final_v1_ts') para el entrenamiento y test, por lo tanto se dejan los resultados en el GitHub ( https://github.com/joseDalvear/VSB-Power-Line-Fault-Detection ) para que no tengas que esperar la ejeción (Tiempo Aproximado de 5 Horas).","metadata":{}},{"cell_type":"markdown","source":"*Nota: En caso de descargar los archivos dispuestos en el repositorio, es necesario que los cargue manualmente a Kaggle y cambie las rutas de cada uno segun lo requiera el codigo. En caso de dudas visite nuevamente el repositorio y encontrara mas instrucciones para la ejecución del proyecto.*","metadata":{}},{"cell_type":"code","source":"tr = pulse_stats(isTrain=True)\npeak_counts = peak_calc(isTrain=True)\nfor k, v in peak_counts.items():\n    tr[k] = v\n\nts = pulse_stats(isTrain=False)\npeak_counts = peak_calc(isTrain=False)\nfor k, v in peak_counts.items():\n    ts[k] = v\n    \ndel peak_counts; gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr.to_csv('final_v1_tr', index=False)\nts.to_csv('final_v1_ts', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evalerror(preds, dtrain):\n    labels = dtrain.get_label()\n    return ('matthews', metrica(labels, preds), True)\n\n\ntrain_data = lgb.Dataset(tr, label=train_metadata['target'])\nparams={'learning_rate': 0.1, 'objective':'binary', 'metric':'None', \n        'num_leaves': 777, 'verbose': 1, 'random_state':311, 'max_depth': 11,\n        'bagging_fraction': 0.7, 'feature_fraction': 1.0, 'min_data_in_leaf': 33,\n        'is_unbalance': True}\nnum_round = 15000\nlight = lgb.train(params, train_data, num_round, feval=evalerror)\npred = light.predict(ts)\nfeature_importances = light.feature_importance()\nfeature_names = ts.columns.values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(pred, bins=100);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = np.where(pred > 0.6, 1, 0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n        \"signal_id\": test_metadata['signal_id'],\n        \"target\": pred\n})\n\nsubmission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_feat_importances(feature_names, fi, figsize=(12,8), color=\"royalblue\"):\n    feature_importances = fi\n    feature_importances = pd.Series(\n        feature_importances, index=feature_names\n        ).sort_values(ascending=False).iloc[:100]\n    fig, ax = plt.subplots(figsize=figsize)\n    sns.barplot(x=feature_importances, \n                y=feature_importances.index, \n                color=color);\n    plt.xlabel('Feature Importance');\n    plt.ylabel('Feature');\n\nplot_feat_importances(feature_names, feature_importances, figsize=(12, 30))\nlen(feature_names)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"resample_train()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val = tr.copy()\nval['target'] = pd.read_csv('train_us_target.csv')\nval = val.sample(frac=0.7, replace=True, random_state=2)\ntr_pred = light.predict(val.drop('target', axis=1))\ntr_pred = np.where(tr_pred > 0.8, 1, 0)\nmetrica(val['target'], tr_pred)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**--------------- Segunda parte--------------------**","metadata":{}},{"cell_type":"markdown","source":"* El proceso es muy similar al anterior","metadata":{}},{"cell_type":"code","source":"def metrica(y_true, y_pred):\n    \"\"\"\n    Metrica del proyecto\n    \n    \"\"\"\n    assert y_true.shape[0] == y_pred.shape[0]\n    \n    tp = np.sum((y_true == 1) & (y_pred == 1))\n    tn = np.sum((y_true == 0) & (y_pred == 0))\n    fp = np.sum((y_true == 0) & (y_pred == 1))\n    fn = np.sum((y_true == 1) & (y_pred == 0))\n\n    numerator = (tp * tn - fp * fn) \n    denominator = ((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)) ** .5\n\n    return numerator / (denominator + 1e-15)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:07:51.768811Z","iopub.execute_input":"2021-06-12T02:07:51.769263Z","iopub.status.idle":"2021-06-12T02:07:51.780984Z","shell.execute_reply.started":"2021-06-12T02:07:51.769225Z","shell.execute_reply":"2021-06-12T02:07:51.778536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_metadata():\n    train_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')\n    test_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_test.csv')\n    return (train_metadata, test_metadata)\ndef resample_train():\n    data = pq.read_pandas('../input/vsb-power-line-fault-detection/train.parquet').to_pandas().transpose()\n    target = read_metadata()[0]['target'].values\n    data['target'] = target\n    p_indices = data[data.target == 0].index\n    np.random.seed(311)\n    random_indices = np.random.choice(p_indices, 1777, replace=False)\n    df = pd.concat([data.loc[random_indices][['target']], \n                    data[target == 1][['target']]]).sample(frac=1.0, random_state=311)\n    df.to_csv('train_us_target.csv', index=False)\n    return df.index","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:09:10.813786Z","iopub.execute_input":"2021-06-12T02:09:10.814554Z","iopub.status.idle":"2021-06-12T02:09:10.825664Z","shell.execute_reply.started":"2021-06-12T02:09:10.814478Z","shell.execute_reply":"2021-06-12T02:09:10.824283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata, test_metadata = read_metadata()","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:09:18.654777Z","iopub.execute_input":"2021-06-12T02:09:18.655395Z","iopub.status.idle":"2021-06-12T02:09:18.699066Z","shell.execute_reply.started":"2021-06-12T02:09:18.655354Z","shell.execute_reply":"2021-06-12T02:09:18.697265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.tail()","metadata":{"execution":{"iopub.status.busy":"2021-05-24T23:04:58.880107Z","iopub.execute_input":"2021-05-24T23:04:58.880679Z","iopub.status.idle":"2021-05-24T23:04:58.901058Z","shell.execute_reply.started":"2021-05-24T23:04:58.880629Z","shell.execute_reply":"2021-05-24T23:04:58.900224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr_limits = []\nstart = 0\nend = 8712\nwhile True:\n    if (start+1000) <= 8712:\n        tr_limits.append((start, start+1000))\n        start=start+1000\n    else:\n        tr_limits.append((start, end))\n        break\n        \nts_limits = []\nstart = 8712\nend = 29049\nwhile True:\n    if (start+1000) <= 29049:\n        ts_limits.append((start, start+1000))\n        start=start+1000\n    else:\n        ts_limits.append((start, end))\n        break","metadata":{"execution":{"iopub.status.busy":"2021-05-24T23:05:00.539028Z","iopub.execute_input":"2021-05-24T23:05:00.539354Z","iopub.status.idle":"2021-05-24T23:05:00.546915Z","shell.execute_reply.started":"2021-05-24T23:05:00.539323Z","shell.execute_reply":"2021-05-24T23:05:00.546053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def peak_calc(isTrain, filterSignal=True):\n    print(\"peak_calc | start:\", time.strftime(\"%H:%M\"))\n    peak_counts = {}\n    # peak_counts expected values: [[1,3,5,1,6], [4,5,2,2,1], ..]\n    \n    def f1(df):\n        nonlocal peak_counts\n        df = df.transpose().values\n        \n        def filter_sg(sg):\n            wv = denoise_wavelet(sg, sigma=None, wavelet='haar', mode='hard', \n                                 wavelet_levels=15, multichannel=False, \n                                 convert2ycbcr=False, method='VisuShrink')\n            B, A = signal.butter(N=3, Wn=0.1, output='ba')\n            smooth_data = signal.filtfilt(B, A, wv)\n            res = (wv - smooth_data) * 100\n            return res\n        \n        if filterSignal:\n            df = np.apply_along_axis(filter_sg, 1, df)\n        \n        print(\"height\", time.strftime(\"%H:%M\"))\n        for h in [3, 5, 8, 10, 15, 25, 50, 100]:\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=(None, -h))[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            if ('height_more_' + str(h)) in peak_counts:\n                peak_counts['height_more_' + str(h)].extend(num_peaks)\n            else:\n                peak_counts['height_more_' + str(h)] = num_peaks\n        \n        print(\"threshold\", time.strftime(\"%H:%M\"))\n        for t in [3, 10, 16, 50]:\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, threshold=(None, -t))[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            if ('threshold_more_' + str(t)) in peak_counts:\n                peak_counts['threshold_more_' + str(t)].extend(num_peaks)\n            else:\n                peak_counts['threshold_more_' + str(t)] = num_peaks\n                \n        print(\"height and distance\", time.strftime(\"%H:%M\"))\n        for h, d in itertools.product([3, 5, 8, 10, 15, 25, 50, 100],\n                                      [5, 7, 25, 75, 111, 1000, 11111, 100000]):\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=(None, -h), distance=d)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            col_name = 'h_bt_' + str(h) + '_dist_bt_' + str(d)\n            if (col_name) in peak_counts:\n                peak_counts[col_name].extend(num_peaks)\n            else:\n                peak_counts[col_name] = num_peaks\n                \n        print(\"height and width\", time.strftime(\"%H:%M\"))\n        for h, w in itertools.product([3, 5, 8, 10, 15, 25, 50, 100],\n                                      [5, 10, 30, 70, 100, 200, 500]):\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=(None, -h), width=w, rel_height=0.4)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            col_name = 'h_bt_' + str(h) + '_w_bt_' + str(w)\n            if (col_name) in peak_counts:\n                peak_counts[col_name].extend(num_peaks)\n            else:\n                peak_counts[col_name] = num_peaks\n        del peaks\n        \n        print(\"max/min widths/proms\", time.strftime(\"%H:%M\"))\n        max_wdth_pk = []\n        std_wdth_pk = []\n        skew_wdth_pk = []\n        min_wdth_pk = []\n        max_prom_pk = []\n        std_prom_pk = []\n        skew_prom_pk = []\n        min_prom_pk = []\n        for n in range(df.shape[0]):\n            seq = df[n, :]\n            peaks, _ = signal.find_peaks(seq, height=(None, -7), threshold=None, distance=None, width=None)\n            if len(peaks) == 0:\n                pk_proms = [[0]]\n                pk_wdths = [0]\n            else:\n                pk_proms = signal.peak_prominences(seq, peaks, wlen=None)\n                pk_wdths = signal.peak_widths(seq, peaks, rel_height=0.4, \n                                              prominence_data=pk_proms, wlen=None)[0]\n            max_wdth_pk.append(max(pk_wdths))\n            std_wdth_pk.append(np.std(pk_wdths))\n            skew_wdth_pk.append(stats.skew(pk_wdths))\n            min_wdth_pk.append(min(pk_wdths))\n            max_prom_pk.append(max(pk_proms[0]))\n            std_prom_pk.append(np.std(pk_proms[0]))\n            skew_prom_pk.append(stats.skew(pk_proms[0]))\n            min_prom_pk.append(min(pk_proms[0]))\n        if ('max_peak_width') in peak_counts:\n            peak_counts['max_peak_width'].extend(max_wdth_pk)\n        else:\n            peak_counts['max_peak_width'] = max_wdth_pk\n        if ('std_peak_width') in peak_counts:\n            peak_counts['std_peak_width'].extend(std_wdth_pk)\n        else:\n            peak_counts['std_peak_width'] = std_wdth_pk\n        if ('skew_peak_width') in peak_counts:\n            peak_counts['skew_peak_width'].extend(skew_wdth_pk)\n        else:\n            peak_counts['skew_peak_width'] = skew_wdth_pk\n        if ('min_peak_width') in peak_counts:\n            peak_counts['min_peak_width'].extend(min_wdth_pk)\n        else:\n            peak_counts['min_peak_width'] = min_wdth_pk\n        if ('max_peak_prom') in peak_counts:\n            peak_counts['max_peak_prom'].extend(max_prom_pk)\n        else:\n            peak_counts['max_peak_prom'] = max_prom_pk\n        if ('std_peak_prom') in peak_counts:\n            peak_counts['std_peak_prom'].extend(std_prom_pk)\n        else:\n            peak_counts['std_peak_prom'] = std_prom_pk\n        if ('skew_peak_prom') in peak_counts:\n            peak_counts['skew_peak_prom'].extend(skew_prom_pk)\n        else:\n            peak_counts['skew_peak_prom'] = skew_prom_pk\n        if ('min_peak_prom') in peak_counts:\n            peak_counts['min_peak_prom'].extend(min_prom_pk)\n        else:\n            peak_counts['min_peak_prom'] = min_prom_pk\n            \n    if isTrain:\n        limits = tr_limits\n        path = '../input/vsb-power-line-fault-detection/train.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f1(df)\n    else:\n        limits = ts_limits\n        path = '../input/vsb-power-line-fault-detection/test.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f1(df)\n            \n    del df; gc.collect()\n    print(\"peak_calc | end:\", time.strftime(\"%H:%M\"))\n    return peak_counts","metadata":{"execution":{"iopub.status.busy":"2021-05-24T23:05:02.090245Z","iopub.execute_input":"2021-05-24T23:05:02.090745Z","iopub.status.idle":"2021-05-24T23:05:02.124761Z","shell.execute_reply.started":"2021-05-24T23:05:02.09071Z","shell.execute_reply":"2021-05-24T23:05:02.123675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pulse_stats(isTrain):\n    print(\"pulse_stats | start:\", time.strftime(\"%H:%M\"))\n    sg_max = np.array([], dtype=np.float16)\n    sg_min = np.array([], dtype=np.float16)\n    sg_mean = np.array([], dtype=np.float16)\n    sg_std = np.array([], dtype=np.float16)\n    perc1 = np.array([], dtype=np.float16)\n    perc3 = np.array([], dtype=np.float16)\n    perc5 = np.array([], dtype=np.float16)\n    perc7 = np.array([], dtype=np.float16)\n    perc9 = np.array([], dtype=np.float16)\n    sg_skew = np.array([], dtype=np.float16)\n    def f2(df):\n        nonlocal sg_max, sg_min, sg_mean, sg_std, perc1, perc3, perc5, perc7, perc9, sg_skew\n        df = df.transpose()\n        sg_max = np.append(sg_max, df.max(axis=1))\n        sg_min = np.append(sg_min, df.min(axis=1))\n        sg_mean = np.append(sg_mean, df.mean(axis=1))\n        sg_std = np.append(sg_std, df.apply(func=np.std, axis=1, raw=True))\n        perc1 = np.append(perc1, df.quantile(q=0.1, axis=1))\n        perc3 = np.append(perc3, df.quantile(q=0.3, axis=1))\n        perc5 = np.append(perc5, df.quantile(q=0.5, axis=1))\n        perc7 = np.append(perc7, df.quantile(q=0.7, axis=1))\n        perc9 = np.append(perc9, df.quantile(q=0.9, axis=1))\n        sg_skew = np.append(sg_skew, df.apply(func=stats.skew, axis=1, raw=True))\n\n    if isTrain:\n        limits = tr_limits\n        path = '../input/vsb-power-line-fault-detection/train.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f2(df)\n    else:\n        limits = ts_limits\n        path = '../input/vsb-power-line-fault-detection/test.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f2(df)\n    del df; gc.collect()\n    res = pd.DataFrame({'sg_max': sg_max,\n                        'sg_min': sg_min,\n                        'sg_mean': sg_mean,\n                        'sg_std': sg_std,\n                        'perc1': perc1,\n                        'perc3': perc3,\n                        'perc5': perc5,\n                        'perc7': perc7,\n                        'perc9': perc9,\n                        'sg_skew': sg_skew\n                       })\n    print(\"pulse_stats | end:\", time.strftime(\"%H:%M\"))\n    return res","metadata":{"execution":{"iopub.status.busy":"2021-05-24T23:05:03.559044Z","iopub.execute_input":"2021-05-24T23:05:03.559388Z","iopub.status.idle":"2021-05-24T23:05:03.576514Z","shell.execute_reply.started":"2021-05-24T23:05:03.559344Z","shell.execute_reply":"2021-05-24T23:05:03.575466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Nota:**\n* Estos bucles son bastante demorados (Calculo de picos), ya que se esta construyendo las bases de datos ('final_v2_tr' y 'final_v2_ts') para el entrenamiento y test, por lo tanto se dejan los resultados en el GitHub ( https://github.com/joseDalvear/VSB-Power-Line-Fault-Detection ) para que no tengas que esperar la ejeción (Tiempo Aproximado de 4.5 Horas).","metadata":{}},{"cell_type":"markdown","source":"*Nota: En caso de descargar los archivos dispuestos en el repositorio, es necesario que los cargue manualmente a Kaggle y cambie las rutas de cada uno segun lo requiera el codigo. En caso de dudas visite nuevamente el repositorio y encontrara mas instrucciones para la ejecución del proyecto.*","metadata":{}},{"cell_type":"code","source":"tr = pulse_stats(isTrain=True)\npeak_counts = peak_calc(isTrain=True)\nfor k, v in peak_counts.items():\n    tr[k] = v\n\nts = pulse_stats(isTrain=False)\npeak_counts = peak_calc(isTrain=False)\nfor k, v in peak_counts.items():\n    ts[k] = v\n    \ndel peak_counts; gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-05-24T23:05:05.126446Z","iopub.execute_input":"2021-05-24T23:05:05.126763Z","iopub.status.idle":"2021-05-25T02:18:05.840622Z","shell.execute_reply.started":"2021-05-24T23:05:05.126737Z","shell.execute_reply":"2021-05-25T02:18:05.839602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr.to_csv('final_v2_tr', index=False)\nts.to_csv('final_v2_ts', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:27:33.640312Z","iopub.execute_input":"2021-05-25T02:27:33.641844Z","iopub.status.idle":"2021-05-25T02:27:35.048839Z","shell.execute_reply.started":"2021-05-25T02:27:33.641789Z","shell.execute_reply":"2021-05-25T02:27:35.047862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evalerror(preds, dtrain):\n    labels = dtrain.get_label()\n    return ('matthews', metrica(labels, preds), True)\n\n\ntrain_data = lgb.Dataset(tr, label=train_metadata['target'])\nparams={'learning_rate': 0.1, 'objective':'binary', 'metric':'None', \n        'num_leaves': 777, 'verbose': 1, 'random_state':311, 'max_depth': 11,\n        'bagging_fraction': 0.7, 'feature_fraction': 1.0, 'min_data_in_leaf': 33,\n        'is_unbalance': True}\nnum_round = 15000\nlight = lgb.train(params, train_data, num_round, feval=evalerror)\npred = light.predict(ts)\nfeature_importances = light.feature_importance()\nfeature_names = ts.columns.values","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:27:58.119125Z","iopub.execute_input":"2021-05-25T02:27:58.119567Z","iopub.status.idle":"2021-05-25T02:28:26.961289Z","shell.execute_reply.started":"2021-05-25T02:27:58.119532Z","shell.execute_reply":"2021-05-25T02:28:26.960337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(pred, bins=100);","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:28:39.211322Z","iopub.execute_input":"2021-05-25T02:28:39.211718Z","iopub.status.idle":"2021-05-25T02:28:39.517407Z","shell.execute_reply.started":"2021-05-25T02:28:39.211685Z","shell.execute_reply":"2021-05-25T02:28:39.516408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = np.where(pred > 0.6, 1, 0)","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:28:43.132994Z","iopub.execute_input":"2021-05-25T02:28:43.13332Z","iopub.status.idle":"2021-05-25T02:28:43.137069Z","shell.execute_reply.started":"2021-05-25T02:28:43.133292Z","shell.execute_reply":"2021-05-25T02:28:43.136398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n        \"signal_id\": test_metadata['signal_id'],\n        \"target\": pred\n})\n\nsubmission.to_csv('submission.csv2', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:28:52.225637Z","iopub.execute_input":"2021-05-25T02:28:52.226238Z","iopub.status.idle":"2021-05-25T02:28:52.254345Z","shell.execute_reply.started":"2021-05-25T02:28:52.226201Z","shell.execute_reply":"2021-05-25T02:28:52.253645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_feat_importances(feature_names, fi, figsize=(12,8), color=\"royalblue\"):\n    feature_importances = fi\n    feature_importances = pd.Series(\n        feature_importances, index=feature_names\n        ).sort_values(ascending=False).iloc[:100]\n    fig, ax = plt.subplots(figsize=figsize)\n    sns.barplot(x=feature_importances, \n                y=feature_importances.index, \n                color=color);\n    plt.xlabel('Feature Importance');\n    plt.ylabel('Feature');\n    print(feature_importances.head(25).index)\n\nplot_feat_importances(feature_names, feature_importances, figsize=(12, 30))","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:29:14.271088Z","iopub.execute_input":"2021-05-25T02:29:14.271668Z","iopub.status.idle":"2021-05-25T02:29:15.510226Z","shell.execute_reply.started":"2021-05-25T02:29:14.271631Z","shell.execute_reply":"2021-05-25T02:29:15.509414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"resample_train()","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:29:46.787941Z","iopub.execute_input":"2021-05-25T02:29:46.788451Z","iopub.status.idle":"2021-05-25T02:30:29.703915Z","shell.execute_reply.started":"2021-05-25T02:29:46.788416Z","shell.execute_reply":"2021-05-25T02:30:29.70037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val = tr.copy()\nval['target'] = train_metadata['target']\nval = val.sample(frac=0.7, replace=True, random_state=2)\ntr_pred = light.predict(val.drop('target', axis=1))\ntr_pred = np.where(tr_pred > 0.8, 1, 0)\nmetrica(val['target'], tr_pred)","metadata":{"execution":{"iopub.status.busy":"2021-05-25T02:30:49.600574Z","iopub.execute_input":"2021-05-25T02:30:49.60102Z","iopub.status.idle":"2021-05-25T02:30:51.96258Z","shell.execute_reply.started":"2021-05-25T02:30:49.60099Z","shell.execute_reply":"2021-05-25T02:30:51.961565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**--------------- Tercera parte--------------------**","metadata":{}},{"cell_type":"markdown","source":"**Modelos**\n\n* Utilizamos los modelos con las bases de datos generadas anteriormente en la primera y segunda parte ","metadata":{}},{"cell_type":"markdown","source":"Importamos los modelos y librerias a utilizar ","metadata":{}},{"cell_type":"code","source":"from skimage.restoration import denoise_wavelet\nimport scipy.signal as signal\nimport scipy.stats as stats\nimport time\nimport itertools\nfrom sklearn.preprocessing import PolynomialFeatures\nimport pandas as pd\nimport numpy as np\nimport pyarrow.parquet as pq\nfrom sklearn.linear_model import SGDRegressor\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, scale, RobustScaler\nimport gc\n\n\npd.set_option(\"max_columns\", 200)\npd.set_option(\"max_rows\", 200)\ngc.enable()","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:25:39.196098Z","iopub.execute_input":"2021-06-12T02:25:39.196403Z","iopub.status.idle":"2021-06-12T02:25:39.204172Z","shell.execute_reply.started":"2021-06-12T02:25:39.196355Z","shell.execute_reply":"2021-06-12T02:25:39.202780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def metrica(y_true, y_pred):\n    assert y_true.shape[0] == y_pred.shape[0]\n    \n    tp = np.sum((y_true == 1) & (y_pred == 1))\n    tn = np.sum((y_true == 0) & (y_pred == 0))\n    fp = np.sum((y_true == 0) & (y_pred == 1))\n    fn = np.sum((y_true == 1) & (y_pred == 0))\n\n    numerator = (tp * tn - fp * fn) \n    denominator = ((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)) ** .5\n\n    return numerator / (denominator + 1e-15)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:25:49.148852Z","iopub.execute_input":"2021-06-12T02:25:49.149137Z","iopub.status.idle":"2021-06-12T02:25:49.156855Z","shell.execute_reply.started":"2021-06-12T02:25:49.149112Z","shell.execute_reply":"2021-06-12T02:25:49.155607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_metadata():\n    train_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')\n    test_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_test.csv')\n    return (train_metadata, test_metadata)\n\ndef resample_train():\n    data = pq.read_pandas('../input/vsb-power-line-fault-detection/train.parquet').to_pandas().transpose()\n    target = read_metadata()[0]['target'].values\n    data['target'] = target\n    p_indices = data[data.target == 0].index\n    np.random.seed(311)\n    random_indices = np.random.choice(p_indices, 1777, replace=False)\n    df = pd.concat([data.loc[random_indices][['target']], \n                    data[target == 1][['target']]]).sample(frac=1.0, random_state=311)\n    df.to_csv('train_us_target.csv', index=False)\n    return df.index","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:25:58.227484Z","iopub.execute_input":"2021-06-12T02:25:58.227798Z","iopub.status.idle":"2021-06-12T02:25:58.234829Z","shell.execute_reply.started":"2021-06-12T02:25:58.227769Z","shell.execute_reply":"2021-06-12T02:25:58.234211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata, test_metadata = read_metadata()\n# resampling_index = resample_train()","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:26:56.897026Z","iopub.execute_input":"2021-06-12T02:26:56.897309Z","iopub.status.idle":"2021-06-12T02:26:56.938187Z","shell.execute_reply.started":"2021-06-12T02:26:56.897281Z","shell.execute_reply":"2021-06-12T02:26:56.936652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Nota**\nEs muy probable que toque organziar las rutas de los archivos para que no genere error, para hacerlo es solo ir al panel derecho dispuesto para \"Data\" y dar clic en \"Copy file path\" a la data que se pretende seleccionar; copiar la ruta y pegarla en medio de las comillas simples donde se hace referencia a la ruta del archivo.","metadata":{}},{"cell_type":"markdown","source":"* En caso de presentar errores visite: https://github.com/joseDalvear/VSB-Power-Line-Fault-Detection  Donde encontrar mas instrucciones para la ejecución del proyecto","metadata":{}},{"cell_type":"markdown","source":"*Nota: En caso de descargar los archivos dispuestos en el repositorio, es necesario que los cargue manualmente a Kaggle y cambie las rutas de cada uno segun lo requiera el codigo. En caso de dudas visite nuevamente el repositorio y encontrara mas instrucciones para la ejecución del proyecto.*","metadata":{}},{"cell_type":"code","source":"tr_v1 = pd.read_csv('../input/output-notebook-1/final_v1_tr')\ntr_v2 = pd.read_csv('../input/outputnotebook2/final_v2_tr')\n# tr_v3 = pd.read_csv('final_v3_tr.csv')\n# tr_v4 = pd.read_csv('final_v4_tr.csv')\n\nts_v1 = pd.read_csv('../input/output-notebook-1/final_v1_ts')\nts_v2 = pd.read_csv('../input/outputnotebook2/final_v2_ts')\n# ts_v3 = pd.read_csv('final_v3_ts.csv')\n# ts_v4 = pd.read_csv('final_v4_ts.csv')\n\ntr_v2.drop(['std_peak_width', 'skew_peak_width', 'std_peak_prom', 'skew_peak_prom'], axis=1, inplace=True)\nts_v2.drop(['std_peak_width', 'skew_peak_width', 'std_peak_prom', 'skew_peak_prom'], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:27:05.164712Z","iopub.execute_input":"2021-06-12T02:27:05.165102Z","iopub.status.idle":"2021-06-12T02:27:06.030872Z","shell.execute_reply.started":"2021-06-12T02:27:05.165078Z","shell.execute_reply":"2021-06-12T02:27:06.030011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr_v3 = tr_v1 - tr_v2\nts_v3 = ts_v1 - ts_v2\nfeat_names = ['h_bt_8_dist_bt_5', 'h_bt_8_dist_bt_1000', 'height_more_15', 'height_more_8', 'h_bt_3_w_bt_5', \n              'threshold_more_16', 'h_bt_5_dist_bt_5', 'h_bt_8_dist_bt_11111', 'sg_skew', \n              'h_bt_5_dist_bt_11111', 'h_bt_8_dist_bt_7', 'max_peak_prom', 'sg_min', 'h_bt_3_dist_bt_11111', \n              'sg_mean', 'sg_std', 'h_bt_5_dist_bt_75', 'h_bt_3_dist_bt_1000', 'h_bt_3_dist_bt_75', \n              'min_peak_prom', 'max_peak_width', 'h_bt_3_dist_bt_7', 'sg_max', 'h_bt_3_w_bt_10', \n              'height_more_3', 'h_bt_3_dist_bt_25', 'h_bt_5_dist_bt_111', 'min_peak_width', \n              'threshold_more_3', 'height_more_5', 'h_bt_5_dist_bt_1000', 'threshold_more_10', \n              'h_bt_5_dist_bt_25', 'h_bt_10_dist_bt_1000']\n\ntr_v1 = tr_v1[feat_names]\nts_v1 = ts_v1[feat_names]\ntr_v2 = tr_v2[feat_names]\nts_v2 = ts_v2[feat_names]\ntr_v3 = tr_v3[feat_names]\nts_v3 = ts_v3[feat_names]\n\ntr_v2.columns = ['ng-df_' + x for x in tr_v2.columns.values]\n# tr_v3.columns = ['abs-df_' + x for x in tr_v3.columns.values]\n# tr_v4.columns = ['tsfresh_' + x for x in tr_v4.columns.values]\ntr_v3.columns = ['sub-df_' + x for x in tr_v3.columns.values]\nts_v3.columns = ['sub-df_' + x for x in ts_v3.columns.values]\n\nts_v2.columns = ['ng-df_' + x for x in ts_v2.columns.values]\n# ts_v3.columns = ['abs-df_' + x for x in ts_v3.columns.values]\n# ts_v4.columns = ['tsfresh_' + x for x in ts_v4.columns.values]","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:27:12.319563Z","iopub.execute_input":"2021-06-12T02:27:12.319853Z","iopub.status.idle":"2021-06-12T02:27:12.370207Z","shell.execute_reply.started":"2021-06-12T02:27:12.319822Z","shell.execute_reply":"2021-06-12T02:27:12.369466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr_v3.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:27:23.099700Z","iopub.execute_input":"2021-06-12T02:27:23.100248Z","iopub.status.idle":"2021-06-12T02:27:23.129147Z","shell.execute_reply.started":"2021-06-12T02:27:23.100215Z","shell.execute_reply":"2021-06-12T02:27:23.128465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr_list = [tr_v1, tr_v2, tr_v3]#, tr_v4]\nts_list = [ts_v1, ts_v2, ts_v3]#, ts_v4]\n\nfor df in tr_list+ts_list:\n    print(df.shape)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:27:28.754244Z","iopub.execute_input":"2021-06-12T02:27:28.754730Z","iopub.status.idle":"2021-06-12T02:27:28.761768Z","shell.execute_reply.started":"2021-06-12T02:27:28.754705Z","shell.execute_reply":"2021-06-12T02:27:28.760162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntr = pd.concat(tr_list, axis=1)\nts = pd.concat(ts_list, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:27:36.075732Z","iopub.execute_input":"2021-06-12T02:27:36.076120Z","iopub.status.idle":"2021-06-12T02:27:36.096783Z","shell.execute_reply.started":"2021-06-12T02:27:36.076095Z","shell.execute_reply":"2021-06-12T02:27:36.095505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evalerror(preds, dtrain):\n    labels = dtrain.get_label()\n    return ('matthews', metrica(labels, preds), True)\n\ntrain_data = lgb.Dataset(tr, label=train_metadata['target'])\nparams={'learning_rate': 0.1, 'objective':'binary', 'metric':'None', \n        'num_leaves': 777, 'verbose': 1, 'seed':311, 'max_depth': 11,\n        'bagging_fraction': 0.7, 'feature_fraction': 1.0, \n        'feature_fraction_seed': 311, 'min_data_in_leaf': 33, \n        'is_unbalance': True}#, 'lambda_l1': 500, 'histogram_pool_size': 6000}\nnum_round = 15000\nlight = lgb.train(params, train_data, num_round, feval=evalerror)\nlight_pred = light.predict(ts)\n\nlight_pred = np.zeros(ts.shape[0])\nfor i in range(1,4):\n    params['bagging_fraction'] = 1.0 - (i/10)\n    params['seed'] = i*110\n    params['learning_rate'] = 0.03 * i + 0.01\n    params['max_depth'] = 9 + i*2\n    params['num_leaves'] = 553 + 100*i\n    light = lgb.train(params, train_data, num_round, feval=evalerror)\n    light_pred += light.predict(ts)\nlight_pred /= 3","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:30:44.467573Z","iopub.execute_input":"2021-06-12T02:30:44.467862Z","iopub.status.idle":"2021-06-12T02:33:10.388052Z","shell.execute_reply.started":"2021-06-12T02:30:44.467835Z","shell.execute_reply":"2021-06-12T02:33:10.387419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 12))\nplt.hist(light_pred, bins=100);\nprint(pd.Series(np.where(light_pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:34:22.707143Z","iopub.execute_input":"2021-06-12T02:34:22.707440Z","iopub.status.idle":"2021-06-12T02:34:23.000427Z","shell.execute_reply.started":"2021-06-12T02:34:22.707412Z","shell.execute_reply":"2021-06-12T02:34:22.999286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from catboost import CatBoostClassifier, Pool\n\ntrain_pool = Pool(tr, label=train_metadata['target'])\ntest_pool = Pool(ts) \n\ncat = CatBoostClassifier(random_seed=1000)\ncat.fit(train_pool)\ncat_pred = cat.predict_proba(test_pool)[:,1]","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:34:35.792224Z","iopub.execute_input":"2021-06-12T02:34:35.792597Z","iopub.status.idle":"2021-06-12T02:34:51.231763Z","shell.execute_reply.started":"2021-06-12T02:34:35.792572Z","shell.execute_reply":"2021-06-12T02:34:51.230214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 12))\nplt.hist(cat_pred, bins=100);\nprint(pd.Series(np.where(cat_pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:19:07.577817Z","iopub.execute_input":"2021-06-12T02:19:07.578403Z","iopub.status.idle":"2021-06-12T02:19:07.945596Z","shell.execute_reply.started":"2021-06-12T02:19:07.578355Z","shell.execute_reply":"2021-06-12T02:19:07.944136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import xgboost\n\ndtrain = xgboost.DMatrix(tr, label=train_metadata['target'])\ndtest = xgboost.DMatrix(ts)\nparams = {\"nthread\": 4, \"seed\": 3, \"subsample\": 0.9, \"reg_lambda\": 11, \"reg_alpha\": 11, \n          \"learning_rate\": 0.15, \"gamma\": 0, \"colsample_bytree\": 0.8, \n          \"colsample_bylevel\": 0.9, \"max_depth\": 50, \"objective\": \"binary:logistic\",\n          'min_child_weight': 33} #hinge\nnum_round = 3000\n\nxgb_pred = np.zeros(ts.shape[0])\nfor i in range(1,4):\n    params['subsample'] = 1.0 - i/10\n    params['seed'] = i*19\n    params['learning_rate'] = 0.005 + i/1000 \n    params['min_child_weight'] = 11*i\n    xgb = xgboost.train(params, dtrain, num_round)\n    xgb_pred += xgb.predict(dtest)\nxgb_pred /= 3","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:43:12.363358Z","iopub.execute_input":"2021-06-12T02:43:12.363818Z","iopub.status.idle":"2021-06-12T02:44:51.542514Z","shell.execute_reply.started":"2021-06-12T02:43:12.363782Z","shell.execute_reply":"2021-06-12T02:44:51.541834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 12))\nplt.hist(xgb_pred, bins=100);\nprint(pd.Series(np.where(xgb_pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:51:30.152570Z","iopub.execute_input":"2021-06-12T02:51:30.152914Z","iopub.status.idle":"2021-06-12T02:51:30.499952Z","shell.execute_reply.started":"2021-06-12T02:51:30.152884Z","shell.execute_reply":"2021-06-12T02:51:30.498500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.ensemble import BaggingClassifier\n\nscaler = StandardScaler()\nscaler.fit(tr)\n\nknn = KNeighborsClassifier(n_neighbors=5, weights='uniform', algorithm='ball_tree', leaf_size=30, \n                           p=2, metric='minkowski', metric_params=None, n_jobs=None)\n\nknn_bg = BaggingClassifier(base_estimator=knn, n_estimators=3, max_samples=0.8, \n                      max_features=1.0, bootstrap=True, bootstrap_features=False, \n                      oob_score=False, warm_start=False, n_jobs=1, \n                      random_state=7, verbose=3)\n\nknn_bg.fit(scaler.transform(tr), train_metadata['target'])\nknn_pred = knn_bg.predict_proba(scaler.transform(ts))[:,1]\nprint(knn_bg.classes_)\nknn_pred.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:51:39.108345Z","iopub.execute_input":"2021-06-12T02:51:39.108696Z","iopub.status.idle":"2021-06-12T02:52:21.763277Z","shell.execute_reply.started":"2021-06-12T02:51:39.108666Z","shell.execute_reply":"2021-06-12T02:52:21.762687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 12))\nplt.hist(knn_pred, bins=100);\nprint(pd.Series(np.where(knn_pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:52:39.178666Z","iopub.execute_input":"2021-06-12T02:52:39.178952Z","iopub.status.idle":"2021-06-12T02:52:39.444442Z","shell.execute_reply.started":"2021-06-12T02:52:39.178927Z","shell.execute_reply":"2021-06-12T02:52:39.443516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.neural_network import MLPClassifier\n\nnn = MLPClassifier(hidden_layer_sizes=(146, 146, 146), \n                   activation=\"relu\", solver=\"adam\", alpha=1e-7, \n                   batch_size=128, learning_rate=\"constant\", learning_rate_init=0.001, power_t=0.5, \n                   max_iter=5000, shuffle=True, random_state=2, tol=0.0001, verbose=False, \n                   warm_start=False, momentum=0.9, nesterovs_momentum=True, early_stopping=False, \n                   validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08, n_iter_no_change=10)\n\nnn_pred = np.zeros(ts.shape[0])\nfor i in range(1,4):\n    nn.set_params(batch_size = 108 + i*10)\n    nn.set_params(learning_rate_init = 0.0005 + i/2000)\n    nn.set_params(alpha = i * 1e-7)\n    nn.fit(scaler.transform(tr), train_metadata['target'])\n    nn_pred += nn.predict_proba(scaler.transform(ts))[:,1]\n    print(nn_pred, nn_pred.shape)\nnn_pred /= 3\nprint(nn.classes_)\nnn_pred","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:52:45.776651Z","iopub.execute_input":"2021-06-12T02:52:45.777050Z","iopub.status.idle":"2021-06-12T02:54:09.336272Z","shell.execute_reply.started":"2021-06-12T02:52:45.777004Z","shell.execute_reply":"2021-06-12T02:54:09.335192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 12))\nplt.hist(nn_pred, bins=100);\nprint(pd.Series(np.where(nn_pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:54:29.929058Z","iopub.execute_input":"2021-06-12T02:54:29.929332Z","iopub.status.idle":"2021-06-12T02:54:30.193176Z","shell.execute_reply.started":"2021-06-12T02:54:29.929309Z","shell.execute_reply":"2021-06-12T02:54:30.192164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\n\nlr = LogisticRegression(penalty='l2', dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, \n                        class_weight=None, random_state=11, solver='liblinear', max_iter=1000, \n                        multi_class='ovr', verbose=0, warm_start=False, n_jobs=4)\n\nlr_pred = np.zeros(ts.shape[0])\nfor i in range(1,4):\n    lr.set_params(random_state = 11 + i)\n    lr.set_params(C = 1.0 * i)\n    lr.fit(tr, train_metadata['target'])\n    lr_pred += lr.predict_proba(ts)[:,1]\nlr_pred /= 3\nlr.classes_","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:54:38.113036Z","iopub.execute_input":"2021-06-12T02:54:38.113385Z","iopub.status.idle":"2021-06-12T02:54:46.899063Z","shell.execute_reply.started":"2021-06-12T02:54:38.113335Z","shell.execute_reply":"2021-06-12T02:54:46.898133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 12))\nplt.hist(lr_pred, bins=100);\nprint(pd.Series(np.where(lr_pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:56:28.393626Z","iopub.execute_input":"2021-06-12T02:56:28.393937Z","iopub.status.idle":"2021-06-12T02:56:28.754710Z","shell.execute_reply.started":"2021-06-12T02:56:28.393904Z","shell.execute_reply":"2021-06-12T02:56:28.753508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.naive_bayes import GaussianNB\n\ngnb = GaussianNB()\n\ngnb_bg = BaggingClassifier(base_estimator=gnb, n_estimators=3, max_samples=0.8, \n                      max_features=1.0, bootstrap=True, bootstrap_features=False, \n                      oob_score=False, warm_start=False, n_jobs=1, \n                      random_state=7, verbose=3)\n\ngnb_bg.fit(tr, train_metadata['target'])\ngnb_pred = gnb_bg.predict_proba(ts)[:,1]\ngnb_bg.classes_","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:56:36.137322Z","iopub.execute_input":"2021-06-12T02:56:36.137687Z","iopub.status.idle":"2021-06-12T02:56:36.267002Z","shell.execute_reply.started":"2021-06-12T02:56:36.137658Z","shell.execute_reply":"2021-06-12T02:56:36.265781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 12))\nplt.hist(gnb_pred, bins=100);\nprint(pd.Series(np.where(gnb_pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:56:44.004137Z","iopub.execute_input":"2021-06-12T02:56:44.004440Z","iopub.status.idle":"2021-06-12T02:56:44.272228Z","shell.execute_reply.started":"2021-06-12T02:56:44.004412Z","shell.execute_reply":"2021-06-12T02:56:44.271120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = [light_pred, cat_pred, xgb_pred, knn_pred, nn_pred, lr_pred, gnb_pred]\nweights = [3, 1, 1, 1, 1, 1, 1]\npred = np.zeros(ts.shape[0])\nfor i in range(len(preds)):\n    pred += weights[i] * preds[i]\npred = pred / np.sum(weights)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:56:53.625827Z","iopub.execute_input":"2021-06-12T02:56:53.626096Z","iopub.status.idle":"2021-06-12T02:56:53.633270Z","shell.execute_reply.started":"2021-06-12T02:56:53.626073Z","shell.execute_reply":"2021-06-12T02:56:53.632310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,17))\nplt.hist(pred, bins=100);\nprint(pd.Series(np.where(pred > 0.51, 1, 0).ravel()).value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:57:01.161095Z","iopub.execute_input":"2021-06-12T02:57:01.161578Z","iopub.status.idle":"2021-06-12T02:57:01.501492Z","shell.execute_reply.started":"2021-06-12T02:57:01.161548Z","shell.execute_reply":"2021-06-12T02:57:01.500188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = np.where(pred > 0.51, 1, 0)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:57:16.365115Z","iopub.execute_input":"2021-06-12T02:57:16.365519Z","iopub.status.idle":"2021-06-12T02:57:16.369113Z","shell.execute_reply.started":"2021-06-12T02:57:16.365492Z","shell.execute_reply":"2021-06-12T02:57:16.368432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n        \"signal_id\": test_metadata['signal_id'],\n        \"target\": pred\n})\n\nsubmission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:57:24.211958Z","iopub.execute_input":"2021-06-12T02:57:24.212327Z","iopub.status.idle":"2021-06-12T02:57:24.556185Z","shell.execute_reply.started":"2021-06-12T02:57:24.212302Z","shell.execute_reply":"2021-06-12T02:57:24.554912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_feat_importances(feature_names, fi, figsize=(12,8), color=\"blue\"):\n    feature_importances = fi\n    feature_importances = pd.Series(\n        feature_importances, index=feature_names\n        ).sort_values(ascending=False).iloc[:100]\n    fig, ax = plt.subplots(figsize=figsize)\n    sns.barplot(x=feature_importances, \n                y=feature_importances.index, \n                color=color);\n    plt.xlabel('Feature Importance');\n    plt.ylabel('Feature');\n    print(feature_importances.head(25).index)\nfeature_importances = light.feature_importance()\nfeature_names = ts.columns.values\nplot_feat_importances(feature_names, feature_importances, figsize=(12, 30))","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:57:30.825434Z","iopub.execute_input":"2021-06-12T02:57:30.825717Z","iopub.status.idle":"2021-06-12T02:57:32.003978Z","shell.execute_reply.started":"2021-06-12T02:57:30.825687Z","shell.execute_reply":"2021-06-12T02:57:32.003001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nval = tr.copy()\nval['target'] = train_metadata['target']\nval = val.sample(frac=0.7, replace=True, random_state=2)\ntr_pred = light.predict(val.drop('target', axis=1))\ntr_pred = np.where(tr_pred > 0.8, 1, 0)\nmetrica(val['target'], tr_pred)","metadata":{"execution":{"iopub.status.busy":"2021-06-12T02:57:42.928207Z","iopub.execute_input":"2021-06-12T02:57:42.928499Z","iopub.status.idle":"2021-06-12T02:57:44.595843Z","shell.execute_reply.started":"2021-06-12T02:57:42.928475Z","shell.execute_reply":"2021-06-12T02:57:44.594934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# --> Conclusiones","metadata":{}},{"cell_type":"markdown","source":"En este estudio se aplicaron las fases de la construcción de un proyecto de aprendizaje automático, lo que permitió terminar a satisfacción la presente monografía. La cual tenía como objetivo proponer un modelo de análisis de datos utilizando técnicas de aprendizaje automático, donde se quería predecir las fallas en las señales eléctricas tomadas por medio de sensores.\n\nLos resultados obtenidos no son perfectos, pero se logró un alto grado de satisfacción, puesto que estamos clasificando las señales por medio de no solo un modelo de aprendizaje automático; se están utilizando varios modelos al tiempo, lo que nos beneficia en temas de confiabilidad al momento de clasificar una señal.\n\nTodavía es posible mejorar aún más los resultados, utilizando técnicas más sofisticadas y potentes del Deep Learning; además, utilizando tensor flow que ofrece una gran cantidad de ayudas para la solución de este tipo de problemas; sin hablar de temas de preprocesamiento en donde se aumentaría la capacidad trabajando desde la nube, hadoop, etc.\n\nEstos resultados pueden ser de gran utilidad para la empresa en caso de querer poner este modelo en productivo, ya que se ahorran altos costos de desplazamiento para el monitoreo manual de las señales, además de la mano de obra de los técnicos encargados de realizarlas. También como uno de los factores más importantes y motivantes de este proyecto, es que beneficiara el medio ambiente y se evitaran gran cantidad de los incendios forestales que eran causados por esta problemática.\n\nEs necesario comentar que fue posible llevar a cabo la ejecución de este proyecto gracias a las técnicas vistas en la especialización en Analitica y Ciencia de Datos de la Universidad de Antioquia; también a la dedicación y acompañamiento del profesor Raul Ramos Pollan donde nos apoyó y nos guió en la idea de realizar una limpieza del ruido a las señales y también de realizar todo un proceso de featuring engineer para obtener unos datos más manipulables para el entrenamiento de los modelos.\n","metadata":{}},{"cell_type":"markdown","source":"**Nota:**\n* En caso de querer profundizar mas en los resultado y las tecnicas utilizadas, es recomendable pasar por el repositorio publico: https://github.com/joseDalvear/VSB-Power-Line-Fault-Detection Donde podra encontrar una amplia explicación sobre las metodologías utilizadas a donde se pretendia llegar con el proyecto","metadata":{}}]}