{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Universidad de Antioquia\n## Especialización en Analítica y Ciencia de Datos\n\nIntegrantes:\n\nJose Daniel Alvear Acevedo\n\nAlfonso Cubillos Delgado","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(url='https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Three_Phase_Electric_Power_Transmission.jpg/1200px-Three_Phase_Electric_Power_Transmission.jpg')","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Contexto del problema**","metadata":{}},{"cell_type":"markdown","source":"Las líneas eléctricas aéreas de media tensión recorren cientos de millas para suministrar energía a las ciudades. Estas grandes distancias hacen que sea costoso inspeccionar manualmente las líneas en busca de daños que no provoquen inmediatamente un corte de energía, como una rama de un árbol golpeando la línea o una falla en el aislante. Estos modos de daño conducen a un fenómeno conocido como descarga parcial, una descarga eléctrica que no une completamente los electrodos entre un sistema de aislamiento. Las descargas parciales dañan lentamente la línea eléctrica, por lo que si no se reparan, eventualmente provocarán un corte de energía o provocarán un incendio.\n\nSu desafío es detectar patrones de descarga parcial en señales adquiridas de estas líneas eléctricas con un nuevo medidor diseñado en el Centro ENET en VŠB . Los clasificadores efectivos que utilizan estos datos permitirán monitorear continuamente las líneas eléctricas en busca de fallas.\n\nEl Centro ENET investiga y desarrolla recursos energéticos renovables con el objetivo de reducir o eliminar los impactos ambientales nocivos. Sus esfuerzos se centran en desarrollar soluciones tecnológicas en torno al transporte y procesamiento de materias primas energéticas.\n\nAl desarrollar una solución para detectar descargas parciales, ayudará a reducir los costos de mantenimiento y evitará cortes de energía.","metadata":{}},{"cell_type":"markdown","source":"# **Importanto librerias**","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport pyarrow.parquet as pq # Leer archivos parquet\nimport matplotlib.pyplot as plt\nimport os\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2021-05-28T02:13:48.578057Z","iopub.execute_input":"2021-05-28T02:13:48.578459Z","iopub.status.idle":"2021-05-28T02:13:49.69094Z","shell.execute_reply.started":"2021-05-28T02:13:48.578428Z","shell.execute_reply":"2021-05-28T02:13:49.6901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport pyarrow.parquet as pq\nfrom sklearn.linear_model import SGDRegressor\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport lightgbm as lgb\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, scale, RobustScaler\nimport gc\nfrom skimage.restoration import denoise_wavelet\nimport scipy.signal as signal\nimport scipy.stats as stats\nimport time\nimport itertools\n\npd.set_option(\"max_columns\", 200)\npd.set_option(\"max_rows\", 200)\ngc.enable()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Tomamos las 2000 primeras filas**","metadata":{}},{"cell_type":"code","source":"INIT_DIR = '../input'\ntamaño = 2001","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:47:11.722941Z","iopub.execute_input":"2021-05-28T01:47:11.723341Z","iopub.status.idle":"2021-05-28T01:47:11.730105Z","shell.execute_reply.started":"2021-05-28T01:47:11.723306Z","shell.execute_reply":"2021-05-28T01:47:11.728695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pq.read_pandas(os.path.join(INIT_DIR, 'vsb-power-line-fault-detection/train.parquet'), columns=[str(i) for i in range(tamaño)]).to_pandas()\nmetadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:47:14.721698Z","iopub.execute_input":"2021-05-28T01:47:14.722173Z","iopub.status.idle":"2021-05-28T01:47:23.796838Z","shell.execute_reply.started":"2021-05-28T01:47:14.722122Z","shell.execute_reply":"2021-05-28T01:47:23.795928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:48:35.952493Z","iopub.execute_input":"2021-05-28T01:48:35.952943Z","iopub.status.idle":"2021-05-28T01:48:35.979482Z","shell.execute_reply.started":"2021-05-28T01:48:35.952907Z","shell.execute_reply":"2021-05-28T01:48:35.978122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:48:42.117529Z","iopub.execute_input":"2021-05-28T01:48:42.118092Z","iopub.status.idle":"2021-05-28T01:48:42.12462Z","shell.execute_reply.started":"2021-05-28T01:48:42.118035Z","shell.execute_reply":"2021-05-28T01:48:42.123816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metadata.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:48:46.490983Z","iopub.execute_input":"2021-05-28T01:48:46.491627Z","iopub.status.idle":"2021-05-28T01:48:46.501679Z","shell.execute_reply.started":"2021-05-28T01:48:46.491588Z","shell.execute_reply":"2021-05-28T01:48:46.500463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metadata.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:48:50.481586Z","iopub.execute_input":"2021-05-28T01:48:50.481979Z","iopub.status.idle":"2021-05-28T01:48:50.48689Z","shell.execute_reply.started":"2021-05-28T01:48:50.48195Z","shell.execute_reply":"2021-05-28T01:48:50.486223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata = metadata[:tamaño]\ntrain_metadata","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:48:53.805017Z","iopub.execute_input":"2021-05-28T01:48:53.805516Z","iopub.status.idle":"2021-05-28T01:48:53.816605Z","shell.execute_reply.started":"2021-05-28T01:48:53.805482Z","shell.execute_reply":"2021-05-28T01:48:53.815955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:46:07.40301Z","iopub.execute_input":"2021-05-28T01:46:07.403376Z","iopub.status.idle":"2021-05-28T01:46:07.409287Z","shell.execute_reply.started":"2021-05-28T01:46:07.403346Z","shell.execute_reply":"2021-05-28T01:46:07.408321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cada columna representa una señal, por lo tanto para manejar el mismo formato de pandas, trasponemos la base de datos","metadata":{}},{"cell_type":"code","source":"train = train.T\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:49:02.266808Z","iopub.execute_input":"2021-05-28T01:49:02.26735Z","iopub.status.idle":"2021-05-28T01:49:02.28438Z","shell.execute_reply.started":"2021-05-28T01:49:02.267296Z","shell.execute_reply":"2021-05-28T01:49:02.280841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(5)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:46:13.480998Z","iopub.execute_input":"2021-05-28T01:46:13.481414Z","iopub.status.idle":"2021-05-28T01:46:13.51015Z","shell.execute_reply.started":"2021-05-28T01:46:13.481376Z","shell.execute_reply":"2021-05-28T01:46:13.509396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Adicionamos el Id de la señal en el dataframe principal","metadata":{}},{"cell_type":"code","source":"train['signal_id'] = list(train_metadata['signal_id'])","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:49:08.788755Z","iopub.execute_input":"2021-05-28T01:49:08.789275Z","iopub.status.idle":"2021-05-28T01:49:08.908103Z","shell.execute_reply.started":"2021-05-28T01:49:08.789241Z","shell.execute_reply":"2021-05-28T01:49:08.907253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(5)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:46:19.927673Z","iopub.execute_input":"2021-05-28T01:46:19.928012Z","iopub.status.idle":"2021-05-28T01:46:19.995092Z","shell.execute_reply.started":"2021-05-28T01:46:19.927984Z","shell.execute_reply":"2021-05-28T01:46:19.994193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Unión de metadatos y datos de señales basados en signal_id**","metadata":{}},{"cell_type":"code","source":"train = train.merge(train_metadata, on='signal_id')","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:49:13.39632Z","iopub.execute_input":"2021-05-28T01:49:13.396868Z","iopub.status.idle":"2021-05-28T01:49:29.907151Z","shell.execute_reply.started":"2021-05-28T01:49:13.396835Z","shell.execute_reply":"2021-05-28T01:49:29.905981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(5)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:46:48.930462Z","iopub.execute_input":"2021-05-28T01:46:48.930846Z","iopub.status.idle":"2021-05-28T01:46:48.965606Z","shell.execute_reply.started":"2021-05-28T01:46:48.930811Z","shell.execute_reply":"2021-05-28T01:46:48.964755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Revisando valores nulos en el dataframe**","metadata":{}},{"cell_type":"code","source":"train.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:49:29.909087Z","iopub.execute_input":"2021-05-28T01:49:29.909385Z","iopub.status.idle":"2021-05-28T01:49:36.109707Z","shell.execute_reply.started":"2021-05-28T01:49:29.909354Z","shell.execute_reply":"2021-05-28T01:49:36.108986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Gráficos de conteo frente a gráficos de destino para verificar el desequilibrio de datos**","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 4))\nsns.countplot(x=\"target\", data=train, ax=ax1)\nsns.countplot(x=\"target\", data=train, hue=\"phase\", ax=ax2);","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:49:39.971617Z","iopub.execute_input":"2021-05-28T01:49:39.972106Z","iopub.status.idle":"2021-05-28T01:49:40.309541Z","shell.execute_reply.started":"2021-05-28T01:49:39.972077Z","shell.execute_reply":"2021-05-28T01:49:40.30882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[train['target']==0]/train['target']","metadata":{"execution":{"iopub.status.busy":"2021-05-28T02:13:12.291825Z","iopub.execute_input":"2021-05-28T02:13:12.292356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_count = train.target.value_counts()\nprint(\"negative(target=0) target: {}\".format(target_count[0]))\nprint(\"positive(target=1) target: {}\".format(target_count[1]))\nprint(\"positive data {:.3}%\".format((target_count[1]/(target_count[0]+target_count[1]))*100))","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:40:53.660093Z","iopub.execute_input":"2021-05-28T01:40:53.660439Z","iopub.status.idle":"2021-05-28T01:40:53.684377Z","shell.execute_reply.started":"2021-05-28T01:40:53.660409Z","shell.execute_reply":"2021-05-28T01:40:53.68319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[['id_measurement', 'phase']]","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:45:13.522366Z","iopub.execute_input":"2021-05-28T01:45:13.522848Z","iopub.status.idle":"2021-05-28T01:45:13.552384Z","shell.execute_reply.started":"2021-05-28T01:45:13.522809Z","shell.execute_reply":"2021-05-28T01:45:13.551157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_mismatch = train[[\"id_measurement\", \"target\"]].groupby([\"id_measurement\"]).sum().query(\"target != 3 & target != 0\")\nprint(\"Valores de la variable objetivo que no son positivos ni negativos: {}\".format(target_mismatch.shape[0]))\ntarget_mismatch","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:45:19.137574Z","iopub.execute_input":"2021-05-28T01:45:19.138001Z","iopub.status.idle":"2021-05-28T01:45:19.159328Z","shell.execute_reply.started":"2021-05-28T01:45:19.13796Z","shell.execute_reply":"2021-05-28T01:45:19.157923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Revisemos el Id == 67, donde la variable objetivo es diferente en diferentes fases","metadata":{}},{"cell_type":"code","source":"train[train['id_measurement'] == 67]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observación:**\n\nLas variables objetivo pueden ser diferentes para la misma señal en diferentes fases","metadata":{}},{"cell_type":"markdown","source":"**Hallamos el único valor del id_measurement**","metadata":{}},{"cell_type":"code","source":"print(\"id_measurement tiene {} valores únicos\".format(train.id_measurement.nunique()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Descripción de la columna id_measurement**","metadata":{}},{"cell_type":"code","source":"train.id_measurement.value_counts().describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Valores únicos de la columna phase**","metadata":{}},{"cell_type":"code","source":"print(\"La variable fase tiene {} valores únicos en train {}\".format(len(train.phase.unique()),train.phase.unique()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(train['phase']);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Graficando las señales**","metadata":{}},{"cell_type":"markdown","source":"**Graficando una señal normal**","metadata":{}},{"cell_type":"code","source":"train.loc[1]['target']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[1].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Grficando una señal defectuosa**","metadata":{}},{"cell_type":"code","source":"train.loc[201]['target']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[201].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Graficando las tres fases de una señal**","metadata":{}},{"cell_type":"code","source":"train.loc[0:2][['target', 'id_measurement']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[0].values), alpha=0.7);\nplt.plot((train.loc[1].values), alpha=0.7);\nplt.plot((train.loc[2].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Graficando las tres fases de una señal defectuosa**","metadata":{}},{"cell_type":"code","source":"train.loc[3:5][['target', 'id_measurement']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(24, 8))\nplt.plot((train.loc[3].values), alpha=0.7);\nplt.plot((train.loc[4].values), alpha=0.7);\nplt.plot((train.loc[5].values), alpha=0.7);\nplt.ylim([-100, 100])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Primera Iteración**","metadata":{}},{"cell_type":"markdown","source":"Dado el tamaño de los datos, se recomienda reiniciar el entorno para ejecutar las sigientes líneas.","metadata":{}},{"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:45:54.040226Z","iopub.execute_input":"2021-05-28T00:45:54.04084Z","iopub.status.idle":"2021-05-28T00:45:54.047073Z","shell.execute_reply.started":"2021-05-28T00:45:54.040784Z","shell.execute_reply":"2021-05-28T00:45:54.045749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cargamos los datos nuevamente","metadata":{}},{"cell_type":"code","source":"train_data = pq.read_pandas(os.path.join(INIT_DIR, 'vsb-power-line-fault-detection/train.parquet')).to_pandas()\ntrain_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:45:57.957543Z","iopub.execute_input":"2021-05-28T00:45:57.957938Z","iopub.status.idle":"2021-05-28T00:46:33.124836Z","shell.execute_reply.started":"2021-05-28T00:45:57.957907Z","shell.execute_reply":"2021-05-28T00:46:33.123529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"Ahora realizamos un análisis exploratorio de los datos","metadata":{}},{"cell_type":"code","source":"train_metadata.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:49:20.492884Z","iopub.execute_input":"2021-05-28T00:49:20.493302Z","iopub.status.idle":"2021-05-28T00:49:20.501945Z","shell.execute_reply.started":"2021-05-28T00:49:20.493269Z","shell.execute_reply":"2021-05-28T00:49:20.500584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.head(10)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:49:37.59626Z","iopub.execute_input":"2021-05-28T00:49:37.59668Z","iopub.status.idle":"2021-05-28T00:49:37.615598Z","shell.execute_reply.started":"2021-05-28T00:49:37.596647Z","shell.execute_reply":"2021-05-28T00:49:37.614438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:49:45.786232Z","iopub.execute_input":"2021-05-28T00:49:45.786649Z","iopub.status.idle":"2021-05-28T00:49:45.792661Z","shell.execute_reply.started":"2021-05-28T00:49:45.786596Z","shell.execute_reply":"2021-05-28T00:49:45.791805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head(10)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:49:50.923063Z","iopub.execute_input":"2021-05-28T00:49:50.923627Z","iopub.status.idle":"2021-05-28T00:49:50.950156Z","shell.execute_reply.started":"2021-05-28T00:49:50.92357Z","shell.execute_reply":"2021-05-28T00:49:50.94899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vamos a tomar el primer 1% de los datos para realizar una primera iteración del modelo, ya que no poseemos recursos computacionales para poder tomar la base de datos completa.","metadata":{}},{"cell_type":"code","source":"train = train_data.iloc[:8000]\ntarget = train_metadata.target[:8000]\nprint(train.shape)\nprint(target.shape)\ntrain","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:50:01.102704Z","iopub.execute_input":"2021-05-28T00:50:01.103106Z","iopub.status.idle":"2021-05-28T00:50:01.133139Z","shell.execute_reply.started":"2021-05-28T00:50:01.103074Z","shell.execute_reply":"2021-05-28T00:50:01.132393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:50:08.008587Z","iopub.execute_input":"2021-05-28T00:50:08.009156Z","iopub.status.idle":"2021-05-28T00:50:08.459931Z","shell.execute_reply.started":"2021-05-28T00:50:08.009121Z","shell.execute_reply":"2021-05-28T00:50:08.458754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A continuación realizaremos un análisis de componentes principales (PCA por sus siglas en inglés) para determinar si podemos utilizar una menor cantidad de variables.","metadata":{}},{"cell_type":"code","source":"from sklearn.decomposition import PCA\npca = PCA(n_components=0.5, whiten=True)\n\nX_pca = pca.fit_transform(train)\n\nprint('Número original de atributos:', train.shape[1])\nprint('Número reducido de atributos:', X_pca.shape[1])","metadata":{"execution":{"iopub.status.busy":"2021-05-28T00:50:16.78078Z","iopub.execute_input":"2021-05-28T00:50:16.781283Z","iopub.status.idle":"2021-05-28T00:53:50.70167Z","shell.execute_reply.started":"2021-05-28T00:50:16.781251Z","shell.execute_reply":"2021-05-28T00:53:50.700807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Con el análisis previo, vamos a implementar inicialmente una regresión logística como primera iteración del conjunto de datos.\n\nPara ello, vamos a utilizar la regresión logística con los datos originales, con el PCA y los datos estandarizados.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import accuracy_score\n\nX_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=42)\n\nmodel = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=100)\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\nprint('Precisión de datos originales:', accuracy_score(y_test, y_pred))\n#######\n\nX_train, X_test, y_train, y_test = train_test_split(X_pca, target, test_size=0.3, random_state=42)\n\nmodel = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=100)\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\nprint('Precisión de datos reducidos:', accuracy_score(y_test, y_pred))\n\nX_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=42)\n\npca = PCA(n_components=0.5, whiten=True)\n\nX_train = pca.fit_transform(X_train)\nX_test = pca.transform(X_test)\n\nmodel = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=100)\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\nprint('Precisión de datos originales estandarizados:', accuracy_score(y_test, y_pred))","metadata":{"execution":{"iopub.status.busy":"2021-05-28T01:00:55.700325Z","iopub.execute_input":"2021-05-28T01:00:55.701515Z","iopub.status.idle":"2021-05-28T01:03:39.919611Z","shell.execute_reply.started":"2021-05-28T01:00:55.701453Z","shell.execute_reply":"2021-05-28T01:03:39.918317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_metadata():\n    train_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_train.csv')\n    test_metadata = pd.read_csv('../input/vsb-power-line-fault-detection/metadata_test.csv')\n    return (train_metadata, test_metadata)\n\ndef resample_train():\n    data = pq.read_pandas('../input/vsb-power-line-fault-detection/train.parquet').to_pandas().transpose()\n    #data = pq.read_pandas(os.path.join(INIT_DIR, 'vsb-power-line-fault-detection/train.parquet')).to_pandas().transpose()\n    target = read_metadata()[0]['target'].values\n    data['target'] = target\n    p_indices = data[data.target == 0].index\n    np.random.seed(311)\n    random_indices = np.random.choice(p_indices, 1777, replace=False)\n    df = pd.concat([data.loc[random_indices][['target']], \n                    data[target == 1][['target']]]).sample(frac=1.0, random_state=311)\n    df.to_csv('train_us_target.csv', index=False)\n    return df.index","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.infex","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pq.read_pandas('../input/vsb-power-line-fault-detection/train.parquet').to_pandas().transpose()\n#data = pq.read_pandas(os.path.join(INIT_DIR, 'vsb-power-line-fault-detection/train.parquet')).to_pandas().transpose()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr_limits = []\nstart = 0\nend = 8712\nwhile True:\n    if (start+1000) <= 8712:\n        tr_limits.append((start, start+1000))\n        start=start+1000\n    else:\n        tr_limits.append((start, end))\n        break\n        \nts_limits = []\nstart = 8712\nend = 29049\nwhile True:\n    if (start+1000) <= 29049:\n        ts_limits.append((start, start+1000))\n        start=start+1000\n    else:\n        ts_limits.append((start, end))\n        break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata, test_metadata = read_metadata()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def peak_calc(isTrain, filterSignal=True):\n    print(\"peak_calc | start:\", time.strftime(\"%H:%M\"))\n    peak_counts = {}\n    # peak_counts expected values: [[1,3,5,1,6], [4,5,2,2,1], ..]\n    \n    def f1(df):\n        nonlocal peak_counts\n        df = df.transpose().values\n        def filter_sg(sg):\n            wv = denoise_wavelet(sg, sigma=None, wavelet='haar', mode='hard', \n                                 wavelet_levels=15, multichannel=False, \n                                 convert2ycbcr=False, method='VisuShrink')\n            B, A = signal.butter(N=3, Wn=0.1, output='ba')\n            smooth_data = signal.filtfilt(B, A, wv)\n            res = (wv - smooth_data) * 100\n            return res\n        \n        if filterSignal:\n            df = np.apply_along_axis(filter_sg, 1, df)\n        \n        print(\"height\", time.strftime(\"%H:%M\"))\n        for h in [3, 5, 8, 10, 15, 25, 50, 100]:\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=h)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            if ('height_more_' + str(h)) in peak_counts:\n                peak_counts['height_more_' + str(h)].extend(num_peaks)\n            else:\n                peak_counts['height_more_' + str(h)] = num_peaks\n        \n        print(\"threshold\", time.strftime(\"%H:%M\"))\n        for t in [3, 10, 16, 50]:\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, threshold=t)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            if ('threshold_more_' + str(t)) in peak_counts:\n                peak_counts['threshold_more_' + str(t)].extend(num_peaks)\n            else:\n                peak_counts['threshold_more_' + str(t)] = num_peaks\n                \n        print(\"height and distance\", time.strftime(\"%H:%M\"))\n        for h, d in itertools.product([3, 5, 8, 10, 15, 25, 50, 100],\n                                      [5, 7, 25, 75, 111, 1000, 11111, 100000]):\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=h, distance=d)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            col_name = 'h_bt_' + str(h) + '_dist_bt_' + str(d)\n            if (col_name) in peak_counts:\n                peak_counts[col_name].extend(num_peaks)\n            else:\n                peak_counts[col_name] = num_peaks\n                \n        print(\"height and width\", time.strftime(\"%H:%M\"))\n        for h, w in itertools.product([3, 5, 8, 10, 15, 25, 50, 100],\n                                      [5, 10, 30, 70, 100, 200, 500]):\n            peaks = np.apply_along_axis(signal.find_peaks, 1, df, height=h, width=w, rel_height=0.4)[:, 0]\n            num_peaks = [len(x) for x in peaks] \n            col_name = 'h_bt_' + str(h) + '_w_bt_' + str(w)\n            if (col_name) in peak_counts:\n                peak_counts[col_name].extend(num_peaks)\n            else:\n                peak_counts[col_name] = num_peaks\n        del peaks\n        \n        print(\"max/min widths/proms\", time.strftime(\"%H:%M\"))\n        max_wdth_pk = []\n        min_wdth_pk = []\n        max_prom_pk = []\n        min_prom_pk = []\n        for n in range(df.shape[0]):\n            seq = df[n, :]\n            peaks, _ = signal.find_peaks(seq, height=7, threshold=None, distance=None, width=None)\n            if len(peaks) == 0:\n                pk_proms = [[0]]\n                pk_wdths = [0]\n            else:\n                pk_proms = signal.peak_prominences(seq, peaks, wlen=None)\n                pk_wdths = signal.peak_widths(seq, peaks, rel_height=0.4, \n                                              prominence_data=pk_proms, wlen=None)[0]\n            max_wdth_pk.append(max(pk_wdths))\n            min_wdth_pk.append(min(pk_wdths))\n            max_prom_pk.append(max(pk_proms[0]))\n            min_prom_pk.append(min(pk_proms[0]))\n        if ('max_peak_width') in peak_counts:\n            peak_counts['max_peak_width'].extend(max_wdth_pk)\n        else:\n            peak_counts['max_peak_width'] = max_wdth_pk\n        if ('min_peak_width') in peak_counts:\n            peak_counts['min_peak_width'].extend(min_wdth_pk)\n        else:\n            peak_counts['min_peak_width'] = min_wdth_pk\n        if ('max_peak_prom') in peak_counts:\n            peak_counts['max_peak_prom'].extend(max_prom_pk)\n        else:\n            peak_counts['max_peak_prom'] = max_prom_pk\n        if ('min_peak_prom') in peak_counts:\n            peak_counts['min_peak_prom'].extend(min_prom_pk)\n        else:\n            peak_counts['min_peak_prom'] = min_prom_pk\n            \n    if isTrain:\n        limits = tr_limits\n        path = '../input/vsb-power-line-fault-detection/train.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f1(df)\n    else:\n        limits = ts_limits\n        path = '../input/vsb-power-line-fault-detection/test.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f1(df)\n            \n    del df; gc.collect()\n    print(\"peak_calc | end:\", time.strftime(\"%H:%M\"))\n    return peak_counts","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pulse_stats(isTrain):\n    print(\"pulse_stats | start:\", time.strftime(\"%H:%M\"))\n    sg_max = np.array([], dtype=np.float16)\n    sg_min = np.array([], dtype=np.float16)\n    sg_mean = np.array([], dtype=np.float16)\n    sg_std = np.array([], dtype=np.float16)\n    perc1 = np.array([], dtype=np.float16)\n    perc3 = np.array([], dtype=np.float16)\n    perc5 = np.array([], dtype=np.float16)\n    perc7 = np.array([], dtype=np.float16)\n    perc9 = np.array([], dtype=np.float16)\n    sg_skew = np.array([], dtype=np.float16)\n    def f2(df):\n        nonlocal sg_max, sg_min, sg_mean, sg_std, perc1, perc3, perc5, perc7, perc9, sg_skew\n        df = df.transpose()\n        sg_max = np.append(sg_max, df.max(axis=1))\n        sg_min = np.append(sg_min, df.min(axis=1))\n        sg_mean = np.append(sg_mean, df.mean(axis=1))\n        sg_std = np.append(sg_std, df.apply(func=np.std, axis=1, raw=True))\n        perc1 = np.append(perc1, df.quantile(q=0.1, axis=1))\n        perc3 = np.append(perc3, df.quantile(q=0.3, axis=1))\n        perc5 = np.append(perc5, df.quantile(q=0.5, axis=1))\n        perc7 = np.append(perc7, df.quantile(q=0.7, axis=1))\n        perc9 = np.append(perc9, df.quantile(q=0.9, axis=1))\n        sg_skew = np.append(sg_skew, df.apply(func=stats.skew, axis=1, raw=True))\n\n    if isTrain:\n        limits = tr_limits\n        path = '../input/vsb-power-line-fault-detection/train.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f2(df)\n    else:\n        limits = ts_limits\n        path = '../input/vsb-power-line-fault-detection/test.parquet'\n        for i, j in limits:\n            df = pq.read_pandas(path, columns=[str(i) for i in range(i, j)]).to_pandas()\n            f2(df)\n    del df; gc.collect()\n    res = pd.DataFrame({'sg_max': sg_max,\n                        'sg_min': sg_min,\n                        'sg_mean': sg_mean,\n                        'sg_std': sg_std,\n                        'perc1': perc1,\n                        'perc3': perc3,\n                        'perc5': perc5,\n                        'perc7': perc7,\n                        'perc9': perc9,\n                        'sg_skew': sg_skew\n                       })\n    print(\"pulse_stats | end:\", time.strftime(\"%H:%M\"))\n    return res","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr = pulse_stats(isTrain=True)\npeak_counts = peak_calc(isTrain=True)\nfor k, v in peak_counts.items():\n    tr[k] = v\n\nts = pulse_stats(isTrain=False)\npeak_counts = peak_calc(isTrain=False)\nfor k, v in peak_counts.items():\n    ts[k] = v\n    \ndel peak_counts; gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr.to_csv('final_v1_tr', index=False)\nts.to_csv('final_v1_ts', index=False)\ntr.shape, ts.shape\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ts = pd.read_csv('../final_v1_ts.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evalerror(preds, dtrain):\n    labels = dtrain.get_label()\n    return ('matthews', mat_cor(labels, preds), True)\n\n\ntrain_data = lgb.Dataset(tr, label=train_metadata['target'])\nparams={'learning_rate': 0.1, 'objective':'binary', 'metric':'None', \n        'num_leaves': 777, 'verbose': 1, 'random_state':311, 'max_depth': 11,\n        'bagging_fraction': 0.7, 'feature_fraction': 1.0, 'min_data_in_leaf': 33,\n        'is_unbalance': True}\nnum_round = 15000\nlight = lgb.train(params, train_data, num_round, feval=evalerror)\npred = light.predict(ts)\nfeature_importances = light.feature_importance()\nfeature_names = ts.columns.values\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(pred, bins=100);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = np.where(pred > 0.6, 1, 0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n        \"signal_id\": test_metadata['signal_id'],\n        \"target\": pred\n})\n\nsubmission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_feat_importances(feature_names, fi, figsize=(12,8), color=\"royalblue\"):\n    feature_importances = fi\n    feature_importances = pd.Series(\n        feature_importances, index=feature_names\n        ).sort_values(ascending=False).iloc[:100]\n    fig, ax = plt.subplots(figsize=figsize)\n    sns.barplot(x=feature_importances, \n                y=feature_importances.index, \n                color=color);\n    plt.xlabel('Feature Importance');\n    plt.ylabel('Feature');\n\nplot_feat_importances(feature_names, feature_importances, figsize=(12, 30))\nlen(feature_names)\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val = tr.copy()\nval['target'] = pd.read_csv('train_us_target.csv')\nval = val.sample(frac=0.7, replace=True, random_state=2)\ntr_pred = light.predict(val.drop('target', axis=1))\ntr_pred = np.where(tr_pred > 0.8, 1, 0)\nmat_cor(val['target'], tr_pred)\n\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Plan de trabajo para las siguientes iteraciones**","metadata":{}},{"cell_type":"markdown","source":"De los anteriores resultados, podemos observar que se obtuvo una precisión muy alta en los tres modelos. Por una análisis posterior podemos afirmar que el modelo se ve afectado por el desbalanceo de los datos, por lo tanto el conjunto de entrenamiento y validación no se están particionando de forma adecuada.\n\nAsí, para las siguientes iteraciones vamos a aplicar técnicas para el análisis de datos desbalanceados tales como: submuestreo inteligente, sobremuestreo inteligente, muestreo durante la validación y pesos diferentes para el error durante el entrenamiento.\n\nAdemás determinar si existe independencia entre las muestras para así poder realizar un análisis suponiendo normalidad de los datos y poder tomar muestras las cuales no van a interferir con las demás.       ","metadata":{}}]}