{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Trabajo practico Nro 3, haciendo ciencia de datos","metadata":{"id":"fFdluZsZ_aHk","papermill":{"duration":0.077031,"end_time":"2022-08-04T00:20:23.546829","exception":false,"start_time":"2022-08-04T00:20:23.469798","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Introduccion\n\nEn este informe intentaremos predecir la falta de pago de las tarjetas de credito de clientes de American Express. Para ello usaremos un dataset reducido del provisto por la empresa en [esta competencia](https://www.kaggle.com/competitions/amex-default-prediction/overview/description).\n\nPara este trabajo primero vamos a reducir el [dataset original de entrenamiento](https://www.kaggle.com/competitions/amex-default-prediction/data?select=train_data.csv) en un 95% para poder analizarlo completamente desde Kaggle.\n\nEl dataset original cuenta con informacion del perfil de varios clientes dividido en 18 meses, cada registro del dataset es una abstraccion del resumen de la tarjeta de credito de un cliente para una fecha dada. y si el cliente no paga 120 dias despues de haber recibido su ultimo resumen se lo considera un cliente moroso.\n\nComo nuestro dataset reducido solo considera un 5% de los registros del dataset original, no vamos a poder ver en detalle cada resumen de cada cliente, por lo que nuestro analisis se limitara a lo que podemos observar con el dataset reducido.","metadata":{"papermill":{"duration":0.073301,"end_time":"2022-08-04T00:20:23.693985","exception":false,"start_time":"2022-08-04T00:20:23.620684","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Dependencias\n\nPara el analisis de datos y prediccion de morosos, haremos uso de las siguientes bibliotecas","metadata":{"papermill":{"duration":0.072542,"end_time":"2022-08-04T00:20:23.839735","exception":false,"start_time":"2022-08-04T00:20:23.767193","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install visualkeras","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:20:23.988978Z","iopub.status.busy":"2022-08-04T00:20:23.988291Z","iopub.status.idle":"2022-08-04T00:20:37.337018Z","shell.execute_reply":"2022-08-04T00:20:37.335895Z"},"id":"GGYdfbFt_aHm","outputId":"1d17c966-0567-49e2-a5c9-ee4f3efe10b8","papermill":{"duration":13.426721,"end_time":"2022-08-04T00:20:37.339547","exception":false,"start_time":"2022-08-04T00:20:23.912826","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport scipy\nimport scipy.stats as st\nimport sklearn as sk\nimport visualkeras\nimport tensorflow as tf\nimport plotly.graph_objects as go\nimport xgboost as xgb\nimport math\nimport io\nimport warnings\nfrom math import pi\nfrom tensorflow import keras\nfrom sklearn import svm, preprocessing\nfrom sklearn.calibration import CalibratedClassifierCV\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.tree import DecisionTreeClassifier, export_text\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split, StratifiedKFold, KFold, RandomizedSearchCV, GridSearchCV, cross_validate\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer\nfrom sklearn.naive_bayes import MultinomialNB\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.metrics import confusion_matrix, accuracy_score, recall_score, f1_score, make_scorer,classification_report, precision_score\nfrom sklearn.neighbors import LocalOutlierFactor\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.ensemble import VotingClassifier\nfrom plotly.subplots import make_subplots\nfrom keras.models import Model\nfrom keras.layers import Dense, Input, Dropout\nfrom plotly.offline import init_notebook_mode\ninit_notebook_mode(connected=True)\n\nwarnings.filterwarnings('ignore')\ntf.random.set_seed(42)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:20:37.493437Z","iopub.status.busy":"2022-08-04T00:20:37.493036Z","iopub.status.idle":"2022-08-04T00:20:45.780648Z","shell.execute_reply":"2022-08-04T00:20:45.779578Z"},"id":"5pL_xTxn_aHn","papermill":{"duration":8.367797,"end_time":"2022-08-04T00:20:45.782861","exception":false,"start_time":"2022-08-04T00:20:37.415064","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Anexo - Generacion del dataset reducido\n\nDejamos el algoritmo utilizado para la generacion del dataset reducido. Se hace uso de la funcion ***sample_without_replacement*** con un *random state* particular de nuestro grupo. Con ello mantenemos la proporcion original del dataset y nos permite realizar un analisis mas cercano al que se haria con el dataset original.","metadata":{"id":"yTbEtpR4_aHo","papermill":{"duration":0.141929,"end_time":"2022-08-04T00:20:46.000142","exception":false,"start_time":"2022-08-04T00:20:45.858213","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"```py\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom sklearn.utils.random import sample_without_replacement\nfrom pathlib import Path\n\nfilename = '.../train_data.csv'\n\ndef row_count():\n    with open(filename) as f:\n        return sum(1 for line in f)\n\nstate = (31416 * 9) % 1000 #Grupo 9\ncount = row_count()\nkeep_rows = sample_without_replacement(n_population=count, n_samples=int(count * (0.05)), random_state=state)\nkeep_rows = np.insert(keep_rows, 0,0, axis=0)\ndata = pd.read_csv(filename, skiprows=lambda x: x not in keep_rows)\nfilepath = Path('.../reduced_data.csv')  \nfilepath.parent.mkdir(parents=True, exist_ok=True)  \ndata.to_csv(filepath)\n```","metadata":{"papermill":{"duration":0.074551,"end_time":"2022-08-04T00:20:46.151088","exception":false,"start_time":"2022-08-04T00:20:46.076537","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Cargamos el dataset reducido y dropeamos la primer columna, que duplica la informacion del index.","metadata":{"papermill":{"duration":0.074791,"end_time":"2022-08-04T00:20:46.300887","exception":false,"start_time":"2022-08-04T00:20:46.226096","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data = pd.read_csv('../input/amexreducido/reduced_data.csv')\n\ndata.drop('Unnamed: 0', axis=1, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:20:46.453187Z","iopub.status.busy":"2022-08-04T00:20:46.452519Z","iopub.status.idle":"2022-08-04T00:21:05.06343Z","shell.execute_reply":"2022-08-04T00:21:05.062605Z"},"id":"kKX2iKTw_aHo","papermill":{"duration":18.689644,"end_time":"2022-08-04T00:21:05.065789","exception":false,"start_time":"2022-08-04T00:20:46.376145","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:05.224754Z","iopub.status.busy":"2022-08-04T00:21:05.223625Z","iopub.status.idle":"2022-08-04T00:21:05.333258Z","shell.execute_reply":"2022-08-04T00:21:05.332305Z"},"id":"vvVsYTiv_aHp","outputId":"2ae3bc68-33b8-48d0-c248-5fcdef791f5b","papermill":{"duration":0.19502,"end_time":"2022-08-04T00:21:05.335918","exception":false,"start_time":"2022-08-04T00:21:05.140898","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Tambien cargamos el dataset de labels, con la variable objetivo **target**\n\n| _Variable_              \t| _Tipo_                \t| _Descripción_                                                   \t|\n|-------------------------\t|-----------------------\t|-----------------------------------------------------------------\t|\n| Target             \t| Cualitstiva binaria                      \t| Indica si un cliente es moroso                                                 \t|","metadata":{"papermill":{"duration":0.076307,"end_time":"2022-08-04T00:21:05.488949","exception":false,"start_time":"2022-08-04T00:21:05.412642","status":"completed"},"tags":[]}},{"cell_type":"code","source":"labels = pd.read_csv(\"../input/amex-default-prediction/train_labels.csv\")\nlabels","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:05.643551Z","iopub.status.busy":"2022-08-04T00:21:05.643141Z","iopub.status.idle":"2022-08-04T00:21:06.505405Z","shell.execute_reply":"2022-08-04T00:21:06.504649Z"},"id":"FAHXrjMF_aHu","outputId":"b21297f4-ab38-4598-d1c8-54b973eadff3","papermill":{"duration":0.94227,"end_time":"2022-08-04T00:21:06.507693","exception":false,"start_time":"2022-08-04T00:21:05.565423","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ciencia de datos","metadata":{"id":"DXW0miOg_aHv","papermill":{"duration":0.075679,"end_time":"2022-08-04T00:21:06.659172","exception":false,"start_time":"2022-08-04T00:21:06.583493","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Introducción al dataset\nA continuación exploraremos el dataset para tener una mejor comprensión de sus variables. Hacemos un merge entre los datos y la variable objetivo para analizarlas en conjunto.","metadata":{"id":"v6XET5_d_aHw","papermill":{"duration":0.075842,"end_time":"2022-08-04T00:21:06.81124","exception":false,"start_time":"2022-08-04T00:21:06.735398","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_analysis = pd.merge(data, labels, on='customer_ID', how='inner')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:06.965771Z","iopub.status.busy":"2022-08-04T00:21:06.965371Z","iopub.status.idle":"2022-08-04T00:21:08.170501Z","shell.execute_reply":"2022-08-04T00:21:08.169582Z"},"id":"zQ760J-a_aHw","papermill":{"duration":1.28448,"end_time":"2022-08-04T00:21:08.172963","exception":false,"start_time":"2022-08-04T00:21:06.888483","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Variables del dataset segun su representacion**\n\n| _Nomenclatura_\t| _Representa_\t| _Numeracion_                      \t|\n|------------------\t|--------------\t|-----------------------------------\t|\n|D_*            \t| Delincuencia\t| D_39 - D_145 (96 variables)       \t|\n|S_*            \t| Gasto     \t| S_2 - S_27 (22 variables)         \t|\n|P_*            \t| Pago      \t| P_2 - P_4 (3 variables)           \t|\n|B_*            \t| Balance      \t| B_1 - B_42 (40 variables)         \t|\n|R_*            \t| Riesgo      \t| R_1 - R_28 (28 variables)         \t|\n|               \t|           \t| **Total: 189 variables**           \t|","metadata":{"papermill":{"duration":0.075995,"end_time":"2022-08-04T00:21:08.32514","exception":false,"start_time":"2022-08-04T00:21:08.249145","status":"completed"},"tags":[]}},{"cell_type":"code","source":"categories=['Delincuencia', 'Gasto','Pago','Balance','Riesgo']\nvalues= [len(data.filter(regex='D_').columns), len(data.filter(regex='S_').columns),len(data.filter(regex='P_').columns), len(data.filter(regex='B_').columns),len(data.filter(regex='R_').columns)]\n\n\nfig = go.Figure()\nfig.add_trace(go.Pie(values = values,labels = categories,hole = 0.6, \n                     hoverinfo ='label+percent', textfont=dict(color=\"white\")))\nfig.update_layout(title='Distribución de las categorías de las variables', \n                  legend=dict(traceorder='reversed',y=1.05,x=0),\n                  uniformtext_minsize=15, uniformtext_mode='hide',width=700)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:08.481296Z","iopub.status.busy":"2022-08-04T00:21:08.480873Z","iopub.status.idle":"2022-08-04T00:21:08.728721Z","shell.execute_reply":"2022-08-04T00:21:08.727972Z"},"id":"z_u4H33F_l0s","outputId":"caf3a81b-ac6d-4a95-e5ea-3bc7247fb4fd","papermill":{"duration":0.329983,"end_time":"2022-08-04T00:21:08.731046","exception":false,"start_time":"2022-08-04T00:21:08.401063","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**De las variables anteriormente descriptas, las siguientes son informadas como categoricas segun American Express**\n\n| _Variable_    \t| _Tipo de variable_\t| _Valores_                |\n|------------------\t|-------------------\t|--------------------------|\n|B_30            \t| Cualitativa ordinal\t| {0, 1, 2}                |\n|B_38            \t| Cualitativa ordinal  \t| {1, 2, 3, 4, 5, 6, 7}    |\n|D_114            \t| Cualitativa binaria  \t| {0, 1}                   |\n|D_116            \t| Cualitativa binaria  \t| {0, 1}                   |\n|D_117            \t| Cualitativa ordinal  \t| {-1, 1, 2, 3, 4, 5, 6}   |\n|D_120            \t| Cualitativa binaria  \t| {0, 1}                   |\n|D_126             \t| Cualitativa ordinal  \t| {-1, 0, 1}               | \n|D_63             \t| Cualitativa nominal  \t| {CL, CO, CR, XL, XM, XZ} |\n|D_64             \t| Cualitativa nominal  \t| {-1, O, R, U}            |\n|D_66              \t| Cualitativa binaria  \t| {0, 1}                   |\n|D_68             \t| Cualitativa ordinal  \t| {0, 1, 2, 3, 4, 5, 6}    |\n|**Total: 10 variables**               \t| | |","metadata":{"id":"Ayuwn7yL_aHw","papermill":{"duration":0.082144,"end_time":"2022-08-04T00:21:08.896109","exception":false,"start_time":"2022-08-04T00:21:08.813965","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"**Analisis de variables categoricas**","metadata":{"id":"X0r8QfE8_aHy","papermill":{"duration":0.085958,"end_time":"2022-08-04T00:21:09.065306","exception":false,"start_time":"2022-08-04T00:21:08.979348","status":"completed"},"tags":[]}},{"cell_type":"code","source":"categorical = data_analysis.nunique().sort_values(ascending=True).reset_index(name='count').head(15)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:09.231631Z","iopub.status.busy":"2022-08-04T00:21:09.230599Z","iopub.status.idle":"2022-08-04T00:21:11.831048Z","shell.execute_reply":"2022-08-04T00:21:11.829922Z"},"id":"WaORkuQg_aHw","papermill":{"duration":2.686709,"end_time":"2022-08-04T00:21:11.833579","exception":false,"start_time":"2022-08-04T00:21:09.14687","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A continuacion vemos las 15 variables con menor cantidad de valores diferentes, notamos que hay algunas variables que tienen pocos valores pero que no estan informadas en el detalle del dataset como categoricas, de igual manera las analizaremos en conjunto.","metadata":{"id":"mS6wufqx_aHx","papermill":{"duration":0.08072,"end_time":"2022-08-04T00:21:11.996304","exception":false,"start_time":"2022-08-04T00:21:11.915584","status":"completed"},"tags":[]}},{"cell_type":"code","source":"categorical","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:12.163627Z","iopub.status.busy":"2022-08-04T00:21:12.163219Z","iopub.status.idle":"2022-08-04T00:21:12.172927Z","shell.execute_reply":"2022-08-04T00:21:12.172029Z"},"id":"lx-r6lVV_aHx","outputId":"4e6fb6c0-0a77-4405-94a0-82e21819502f","papermill":{"duration":0.095136,"end_time":"2022-08-04T00:21:12.175139","exception":false,"start_time":"2022-08-04T00:21:12.080003","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical = categorical.head(14)\ncategorical_columns = categorical['index'].values\ncategorical","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:12.341234Z","iopub.status.busy":"2022-08-04T00:21:12.340287Z","iopub.status.idle":"2022-08-04T00:21:12.350816Z","shell.execute_reply":"2022-08-04T00:21:12.349689Z"},"id":"TrJSZnzb_aHx","papermill":{"duration":0.095039,"end_time":"2022-08-04T00:21:12.352881","exception":false,"start_time":"2022-08-04T00:21:12.257842","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"En los siguientes histogramas vemos la frecuencia de los valores de cada variable segun el target (en rojo estan los morosos y en azul los pagos). Observamos que las variables *D_87* y *B_31* tienen frecuencias particulares. Por un lado *D_87* tiene mayoria de valores nulos y *B_31* mayormente el valor 1","metadata":{"papermill":{"duration":0.083512,"end_time":"2022-08-04T00:21:12.519509","exception":false,"start_time":"2022-08-04T00:21:12.435997","status":"completed"},"tags":[]}},{"cell_type":"code","source":"categorical_ignore_target = [x for x in categorical_columns if x != \"target\"]\nfig = make_subplots(rows=5, cols=3, \n                    subplot_titles=categorical_ignore_target, \n                    vertical_spacing=0.1)\npal, color=['#48D0FF','#A61B1B'], ['#6FD6FF','#C12323']\nrow=0\nc=[1,2,3]*5\nplot_df=data_analysis[categorical_columns]\nfor i,col in enumerate(categorical_ignore_target):\n    if i%3==0:\n        row+=1\n    df=plot_df.replace(np.nan, 'NA').replace('', 'NA').groupby(col)['target'].value_counts().rename('count').reset_index()\n    x_label = [str(int(item)) if isinstance(item, (int, float)) else str(item) for item in df[col].drop_duplicates().reset_index()[col].values]\n    fig.add_trace(go.Bar(x=x_label, y=df[df.target==1]['count'],\n                         marker_color=pal[1], marker_line=dict(color=pal[1],width=2), \n                         hovertemplate='Impagos. Frecuencia = %{y}',\n                         name='Impago', showlegend=(True if i==0 else False)),\n                  row=row, col=c[i])\n    fig.add_trace(go.Bar(x=x_label, y=df[df.target==0]['count'],\n                         marker_color=pal[0], marker_line=dict(color=pal[0],width=2),\n                         hovertemplate='Pagos. Frecuencia = %{y}',\n                         name='Pago', showlegend=(True if i==0 else False)),\n                  row=row, col=c[i])\n    if i%3==0:\n        fig.update_yaxes(title='Frecuencia',row=row,col=c[i])\nfig.update_layout(legend=dict(orientation=\"h\",yanchor=\"bottom\",y=1.03,xanchor=\"right\",x=0.85),\n                  barmode='group',height=1500,width=900)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:12.68939Z","iopub.status.busy":"2022-08-04T00:21:12.688694Z","iopub.status.idle":"2022-08-04T00:21:20.068846Z","shell.execute_reply":"2022-08-04T00:21:20.06793Z"},"id":"IdNjW-OR_aHy","outputId":"cf7602df-93e2-4270-d582-6760b650f1cd","papermill":{"duration":7.469687,"end_time":"2022-08-04T00:21:20.073043","exception":false,"start_time":"2022-08-04T00:21:12.603356","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Analisis de la distribucion de resumenes por fecha**\n\nEn el siguiente grafico podemos observar como se distribuye el porcentaje de deudores y no deudores por fecha de resumen. Notamos que se trata de una distribucion casi uniforme, por lo que podemos considerar que la distribucion de resumenes de clientes deudores y no deudores es uniforme para todo el dataset, con valores cercanos al 80% de resumenes de clientes no deudores cada fecha","metadata":{"papermill":{"duration":0.095196,"end_time":"2022-08-04T00:21:20.323642","exception":false,"start_time":"2022-08-04T00:21:20.228446","status":"completed"},"tags":[]}},{"cell_type":"code","source":"target=pd.DataFrame(data={'Deudores':data_analysis.groupby('S_2')['target'].mean()*100})\ntarget['Pagos']=np.abs(data_analysis.groupby('S_2')['target'].mean()-1)*100\nfig=go.Figure()\nfig.add_trace(go.Bar(x=target.index, y=target.Pagos, name='Pagos',\n                     text=target.Pagos, marker=dict(color=color[0],line=dict(color=pal[0],width=1.5)),\n                     hovertemplate = \"<b>%{x}</b><br>Clientes pagos: %{y:.2f}%\"))\nfig.add_trace(go.Bar(x=target.index, y=target.Deudores, name='Deudores',\n                     marker=dict(color=color[1],line=dict(color=pal[1],width=1.5)),\n                     hovertemplate = \"<b>%{x}</b><br>Default accounts: %{y:.2f}%\"))\nfig.update_layout(barmode='relative', yaxis_ticksuffix='%', width=1400,\n                  legend=dict(orientation=\"h\", traceorder=\"reversed\", yanchor=\"bottom\",y=1.1,xanchor=\"left\", x=0))\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:20.515077Z","iopub.status.busy":"2022-08-04T00:21:20.514386Z","iopub.status.idle":"2022-08-04T00:21:20.608117Z","shell.execute_reply":"2022-08-04T00:21:20.606993Z"},"id":"laYKEqoV_aHy","outputId":"54eb8501-54e7-41b8-ff3b-a053ce9ce364","papermill":{"duration":0.19234,"end_time":"2022-08-04T00:21:20.610752","exception":false,"start_time":"2022-08-04T00:21:20.418412","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Analisis del balance de los datos**","metadata":{"id":"RrUsIgUU_aHx","papermill":{"duration":0.103932,"end_time":"2022-08-04T00:21:20.82014","exception":false,"start_time":"2022-08-04T00:21:20.716208","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Observamos que existe un desbalanceo de los datos con una tendencia a clientes que no tienen deudas. Esto ya se podia presuponer del grafico anterior, ya que cada dia se cuenta con mas resumenes de clientes no deudores que de clientes deudores.","metadata":{"id":"OdRPOruS_aHx","papermill":{"duration":0.10561,"end_time":"2022-08-04T00:21:21.031722","exception":false,"start_time":"2022-08-04T00:21:20.926112","status":"completed"},"tags":[]}},{"cell_type":"code","source":"target=data_analysis.target.value_counts(normalize=True)\ntarget.rename(index={1:'Deudores',0:'Pagos'},inplace=True)\npal, color=['#48D0FF','#A61B1B'], ['#6FD6FF','#C12323']\nfig=go.Figure()\nfig.add_trace(go.Pie(labels=target.index, values=target*100, hole=.45, \n                     showlegend=True,sort=False, \n                     marker=dict(colors=color,line=dict(color=pal,width=2.5)),\n                     hovertemplate = \"Clientes %{label}: %{value:.2f}%<extra></extra>\"))\nfig.update_layout(title='Balance de los datos', \n                  legend=dict(traceorder='reversed',y=1.05,x=0),\n                  uniformtext_minsize=15, uniformtext_mode='hide',width=700)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:21.241936Z","iopub.status.busy":"2022-08-04T00:21:21.240748Z","iopub.status.idle":"2022-08-04T00:21:21.28983Z","shell.execute_reply":"2022-08-04T00:21:21.288891Z"},"id":"zClt2R6u_aHx","outputId":"ef1a6cd3-e361-4248-c537-ffdef0a1155d","papermill":{"duration":0.156704,"end_time":"2022-08-04T00:21:21.292209","exception":false,"start_time":"2022-08-04T00:21:21.135505","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Análisis de las variables de pago**\n\nAnalizamos el dataset enfocandonos en las variables de pago.","metadata":{"id":"lS4-lm6b_aHy","papermill":{"duration":0.111421,"end_time":"2022-08-04T00:21:21.512883","exception":false,"start_time":"2022-08-04T00:21:21.401462","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_analysis_p = data_analysis.filter(regex='P_')\ndata_analysis_p.head()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:21.737672Z","iopub.status.busy":"2022-08-04T00:21:21.736942Z","iopub.status.idle":"2022-08-04T00:21:21.750438Z","shell.execute_reply":"2022-08-04T00:21:21.749357Z"},"id":"Ww3dqfBu_aHy","papermill":{"duration":0.127052,"end_time":"2022-08-04T00:21:21.752552","exception":false,"start_time":"2022-08-04T00:21:21.6255","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_p.describe().applymap(\"{0:.2f}\".format)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:21.975842Z","iopub.status.busy":"2022-08-04T00:21:21.975036Z","iopub.status.idle":"2022-08-04T00:21:22.047058Z","shell.execute_reply":"2022-08-04T00:21:22.046126Z"},"id":"ooQzMmng_aHz","outputId":"d27014ad-4601-428a-a1af-afa9588b1b8e","papermill":{"duration":0.186549,"end_time":"2022-08-04T00:21:22.049177","exception":false,"start_time":"2022-08-04T00:21:21.862628","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Distribucion de las variables de pago segun la situacion de la deuda impaga o paga de cada cliente. Notamos que la variable P_2 tiene distribuciones distintas cuando se trata de clientes deudores y pagos, mientras que P_3 y P_4 tienen distribuciones parecidas.\n\nEsto puede indicarnos que P_2 puede llegar a ser un factor interesante para decidir la situación de la deuda.","metadata":{"id":"zzcrkSyY_aHz","papermill":{"duration":0.109494,"end_time":"2022-08-04T00:21:22.270281","exception":false,"start_time":"2022-08-04T00:21:22.160787","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols=[col for col in data_analysis.columns if (col.startswith(('P','t')))]\nplot_df=data_analysis[cols]\nfig, ax = plt.subplots(1,3, figsize=(16,5))\nfig.suptitle('Distribucion de las variables de pago',fontsize=16)\nfor i, col in enumerate(plot_df.columns[:-1]):\n    sns.kdeplot(x=col, hue='target', palette=pal[::-1], hue_order=[1,0], \n                label=['Impago','Pago'], data=plot_df, \n                fill=True, linewidth=2, legend=False, ax=ax[i])\n    ax[i].tick_params(left=False,bottom=False)\n    ax[i].set(title='{}'.format(col), xlabel='', ylabel=('Densidad' if i==0 else ''))\nhandles, _ = ax[0].get_legend_handles_labels() \nfig.legend(labels=['Impago','Pago'], handles=reversed(handles), ncol=2, bbox_to_anchor=(0.18, 1))\nsns.despine(bottom=True, trim=True)\nplt.tight_layout(rect=[0, 0.2, 1, 0.99])","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2022-08-04T00:21:22.493257Z","iopub.status.busy":"2022-08-04T00:21:22.49218Z","iopub.status.idle":"2022-08-04T00:21:26.899931Z","shell.execute_reply":"2022-08-04T00:21:26.898957Z"},"id":"x6BqwwKk_aHz","outputId":"71f5500c-d8a7-4ae3-a088-359fda8d79b2","papermill":{"duration":4.521793,"end_time":"2022-08-04T00:21:26.9026","exception":false,"start_time":"2022-08-04T00:21:22.380807","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos tambien que P_4 se comporta casi como una variable discreta, tomando valores muy cercanos a 0 o muy cercanos a 1.\n\nAhora analizaremos la correlación entre las variables de pago y la variable objetivo","metadata":{"id":"e4MKQmj9_aHz","papermill":{"duration":0.111744,"end_time":"2022-08-04T00:21:27.129161","exception":false,"start_time":"2022-08-04T00:21:27.017417","status":"completed"},"tags":[]}},{"cell_type":"code","source":"corr=plot_df.corr()\nmask=np.triu(np.ones_like(corr, dtype=bool))[1:,:-1]\ncorr=corr.iloc[1:,:-1].copy()\nfig, ax = plt.subplots()   \nsns.heatmap(corr, mask=mask, vmin=-1, vmax=1, center=0, annot=True, fmt='.2f', \n            cmap='coolwarm', annot_kws={'fontsize':12,'fontweight':'bold'}, cbar=False)\nax.tick_params(left=False,bottom=False)\nax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right',fontsize=12)\nax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\nplt.title('Correlacion entre variables de pago\\n', fontsize=16)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:27.357213Z","iopub.status.busy":"2022-08-04T00:21:27.356731Z","iopub.status.idle":"2022-08-04T00:21:27.572356Z","shell.execute_reply":"2022-08-04T00:21:27.57127Z"},"id":"9x_LT8-B_aHz","outputId":"a50e0abc-0fd0-443b-e89b-3ca323a0c1a8","papermill":{"duration":0.332751,"end_time":"2022-08-04T00:21:27.574729","exception":false,"start_time":"2022-08-04T00:21:27.241978","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,3, figsize=(16,5))\nfig.suptitle('Relaciones entre las variables de pago, transformacion logaritmica',fontsize=16)\nax[0].hexbin(x='P_2', y='P_3', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[0].text(-.2,2.2, 'Correlation: {:.2f}'.format(plot_df[['P_2','P_3']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[0].set(xlabel='P_2',ylabel='P_3')\nax[1].hexbin(x='P_3', y='P_4', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[1].text(-.6,1.35, 'Correlation: {:.2f}'.format(plot_df[['P_3','P_4']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[1].set(xlabel='P_3',ylabel='P_4')\nax[2].hexbin(x='P_4', y='P_2', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[2].text(.25,1.1, 'Correlation: {:.2f}'.format(plot_df[['P_4','P_2']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[2].set(xlabel='P_4',ylabel='P_2')\nfor i in range(3):\n    ax[i].tick_params(left=False,bottom=False)\nsns.despine()\nplt.tight_layout(rect=[0, 0, 1, 0.99])\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:27.857948Z","iopub.status.busy":"2022-08-04T00:21:27.857535Z","iopub.status.idle":"2022-08-04T00:21:28.920308Z","shell.execute_reply":"2022-08-04T00:21:28.919227Z"},"id":"QTVYlYIQ_aHz","outputId":"cbf8b4d5-af9c-42e5-eaa5-415b9ebb8191","papermill":{"duration":1.237257,"end_time":"2022-08-04T00:21:28.924278","exception":false,"start_time":"2022-08-04T00:21:27.687021","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Analizamos los nulos y vemos que no representan una gran cantidad respecto a la totalidad del dataset","metadata":{"papermill":{"duration":0.11555,"end_time":"2022-08-04T00:21:29.158667","exception":false,"start_time":"2022-08-04T00:21:29.043117","status":"completed"},"tags":[]}},{"cell_type":"code","source":"null_p=round((data_analysis_p.isna().sum()/data_analysis_p.shape[0]*100),2).sort_values(ascending=False).astype(str)+('%')\nnull_p=null_p.to_frame().rename(columns={0:'Nulos'})\nnull_p","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:29.389783Z","iopub.status.busy":"2022-08-04T00:21:29.389375Z","iopub.status.idle":"2022-08-04T00:21:29.404357Z","shell.execute_reply":"2022-08-04T00:21:29.403412Z"},"id":"SxQTwAPp_aHz","outputId":"87d6cf45-3fdc-4f20-d8af-3ff58f75cab4","papermill":{"duration":0.133389,"end_time":"2022-08-04T00:21:29.406525","exception":false,"start_time":"2022-08-04T00:21:29.273136","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Análisis de las variables de riesgo**\n\nAnalizamos el dataset enfocandonos en las variables de pago.","metadata":{"id":"scdIPoGI_aHz","papermill":{"duration":0.114972,"end_time":"2022-08-04T00:21:29.636481","exception":false,"start_time":"2022-08-04T00:21:29.521509","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_analysis_r = data_analysis.filter(regex='R_')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:29.86806Z","iopub.status.busy":"2022-08-04T00:21:29.867595Z","iopub.status.idle":"2022-08-04T00:21:29.885451Z","shell.execute_reply":"2022-08-04T00:21:29.884427Z"},"id":"LnEQiPOQ_aHz","papermill":{"duration":0.1368,"end_time":"2022-08-04T00:21:29.887781","exception":false,"start_time":"2022-08-04T00:21:29.750981","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_r.head()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:30.12464Z","iopub.status.busy":"2022-08-04T00:21:30.124021Z","iopub.status.idle":"2022-08-04T00:21:30.150009Z","shell.execute_reply":"2022-08-04T00:21:30.148954Z"},"id":"ATZxhZkf_aH0","outputId":"a2a88d57-0bec-45e3-eb3c-cc6d89701e12","papermill":{"duration":0.148528,"end_time":"2022-08-04T00:21:30.152364","exception":false,"start_time":"2022-08-04T00:21:30.003836","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Analizamos las distribuciones de variables para las de riesgo. De nuevo vemos distribuciones parecidas entre los clientes deudores y no deudores. Observamos tambien variables con valores atipicos muy marcados, con máximos muy alejados de su media y cuartiles.","metadata":{"id":"IV6qGqCk_aH0","papermill":{"duration":0.114617,"end_time":"2022-08-04T00:21:30.384752","exception":false,"start_time":"2022-08-04T00:21:30.270135","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols=[col for col in data_analysis.columns if (col.startswith(('R','t'))) & (col not in categorical_ignore_target)]\nplot_df=data_analysis[cols]\nfig, ax = plt.subplots(6,5, figsize=(16,24))\nfig.suptitle('Distribucion de las variables de riesgo',fontsize=16)\nrow=0\ncol=[0,1,2,3,4]*6\nfor i, column in enumerate(plot_df.columns[:-1]):\n    if (i!=0)&(i%5==0):\n        row+=1\n    sns.kdeplot(x=column, hue='target', palette=pal[::-1], hue_order=[1,0], \n                label=['Impago','Pago'], data=plot_df, \n                fill=True, linewidth=2, legend=False, ax=ax[row,col[i]])\n    ax[row,col[i]].tick_params(left=False,bottom=False)\n    ax[row,col[i]].set(title='\\n\\n{}'.format(column), xlabel='', ylabel=('Densidad' if i%5==0 else ''))\nfor i in range(3,5):\n    ax[5,i].set_visible(False)\nhandles, _ = ax[0,0].get_legend_handles_labels() \nfig.legend(labels=['Impago','Pago'], handles=reversed(handles), ncol=2, bbox_to_anchor=(0.18, 0.984))\nsns.despine(bottom=True, trim=True)\nplt.tight_layout(rect=[0, 0.2, 1, 0.99])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:21:30.619636Z","iopub.status.busy":"2022-08-04T00:21:30.618547Z","iopub.status.idle":"2022-08-04T00:22:06.826051Z","shell.execute_reply":"2022-08-04T00:22:06.824926Z"},"id":"gD57cz52_aH0","outputId":"8e7bcb43-ee2b-4720-b95e-fe352cff9d74","papermill":{"duration":36.447626,"end_time":"2022-08-04T00:22:06.947204","exception":false,"start_time":"2022-08-04T00:21:30.499578","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_r[['R_5', 'R_7', 'R_8', 'R_14', 'R_26', 'R_20']].describe().applymap(\"{0:.2f}\".format)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:07.183666Z","iopub.status.busy":"2022-08-04T00:22:07.183277Z","iopub.status.idle":"2022-08-04T00:22:07.297286Z","shell.execute_reply":"2022-08-04T00:22:07.296281Z"},"id":"ZHAWjWVc_aH0","outputId":"2d9ed810-2b32-4fb4-cdf2-82a51c8ffc25","papermill":{"duration":0.234831,"end_time":"2022-08-04T00:22:07.299595","exception":false,"start_time":"2022-08-04T00:22:07.064764","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr=plot_df.corr()\nmask=np.triu(np.ones_like(corr, dtype=bool))[1:,:-1]\ncorr=corr.iloc[1:,:-1].copy()\nfig, ax = plt.subplots(figsize=(24,18))   \nsns.heatmap(corr, mask=mask, vmin=-1, vmax=1, center=0, annot=True, fmt='.2f', \n            cmap='coolwarm', annot_kws={'fontsize':12,'fontweight':'bold'}, cbar=False)\nax.tick_params(left=False,bottom=False)\nax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right',fontsize=12)\nax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\nplt.title('Correlaciones entre variables de riesgo\\n', fontsize=16)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:07.538556Z","iopub.status.busy":"2022-08-04T00:22:07.538027Z","iopub.status.idle":"2022-08-04T00:22:10.202955Z","shell.execute_reply":"2022-08-04T00:22:10.202158Z"},"id":"ZJLDoDgw_aH0","outputId":"de324643-4688-4d12-cb15-ebbc6eefe70f","papermill":{"duration":2.788167,"end_time":"2022-08-04T00:22:10.207179","exception":false,"start_time":"2022-08-04T00:22:07.419012","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Encontramos correlaciones fuertes entre algunas variables (R_2/R_4, R_8/R_5), pero tambien vemos otras muy debiles (R_18, R_23, R_28) con el resto de variables.\n\nAnalizando los valores nulos para las variables de riesgo, encontramos R_9 y R_26 con porcentajes muy altos de registros sin informacion.","metadata":{"id":"5FtllOoc_aH0","papermill":{"duration":0.122593,"end_time":"2022-08-04T00:22:10.511231","exception":false,"start_time":"2022-08-04T00:22:10.388638","status":"completed"},"tags":[]}},{"cell_type":"code","source":"null_r=round((data_analysis_r.isna().sum()/data_analysis_r.shape[0]*100),2).sort_values(ascending=False).astype(str)+('%')\nnull_r=null_r.to_frame().rename(columns={0:'Nulos'})\nnull_r.head(10)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:10.758276Z","iopub.status.busy":"2022-08-04T00:22:10.757596Z","iopub.status.idle":"2022-08-04T00:22:10.786216Z","shell.execute_reply":"2022-08-04T00:22:10.785425Z"},"id":"eNJCD6ge_aH0","outputId":"27644073-4c4e-4acf-e82a-fcfda059755d","papermill":{"duration":0.154605,"end_time":"2022-08-04T00:22:10.788341","exception":false,"start_time":"2022-08-04T00:22:10.633736","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Análisis de las variables de balance**\n\nAnalizamos el dataset enfocandonos en las variables de balance.","metadata":{"id":"WFHSl6ss_aH0","papermill":{"duration":0.125093,"end_time":"2022-08-04T00:22:11.03443","exception":false,"start_time":"2022-08-04T00:22:10.909337","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_analysis_b = data_analysis.filter(regex='B_')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:11.283423Z","iopub.status.busy":"2022-08-04T00:22:11.282712Z","iopub.status.idle":"2022-08-04T00:22:11.314521Z","shell.execute_reply":"2022-08-04T00:22:11.313471Z"},"id":"th2v9ISC_aH0","papermill":{"duration":0.160399,"end_time":"2022-08-04T00:22:11.317004","exception":false,"start_time":"2022-08-04T00:22:11.156605","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_b.head()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:11.560461Z","iopub.status.busy":"2022-08-04T00:22:11.560037Z","iopub.status.idle":"2022-08-04T00:22:11.586919Z","shell.execute_reply":"2022-08-04T00:22:11.58588Z"},"id":"SAZXjD8C_aH0","outputId":"9b71022c-6fd1-4ea0-9aec-9b82d43d3346","papermill":{"duration":0.151368,"end_time":"2022-08-04T00:22:11.589254","exception":false,"start_time":"2022-08-04T00:22:11.437886","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Analizamos las distribuciones de las variables de balance. De nuevo vemos distribuciones parecidas para deudores y no deudores, junto con variables con valores atipicos marcados.","metadata":{"id":"8BUhHAsq_aH0","papermill":{"duration":0.127531,"end_time":"2022-08-04T00:22:11.838222","exception":false,"start_time":"2022-08-04T00:22:11.710691","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols=[col for col in data_analysis.columns if (col.startswith(('B','t'))) & (col not in categorical_ignore_target)]\nplot_df=data_analysis[cols]\nfig, ax = plt.subplots(8,5, figsize=(16,32))\nfig.suptitle('Distribucion de las variables de balance',fontsize=16)\nrow=0\ncol=[0,1,2,3,4]*8\nfor i, column in enumerate(plot_df.columns[:-1]):\n    if (i!=0)&(i%5==0):\n        row+=1\n    sns.kdeplot(x=column, hue='target', palette=pal[::-1], hue_order=[1,0], \n                label=['Impago','Pago'], data=plot_df, \n                fill=True, linewidth=2, legend=False, ax=ax[row,col[i]])\n    ax[row,col[i]].tick_params(left=False,bottom=False)\n    ax[row,col[i]].set(title='\\n\\n{}'.format(column), xlabel='', ylabel=('Densidad' if i%5==0 else ''))\nfor i in range(3,5):\n    ax[7,i].set_visible(False)\nhandles, _ = ax[0,0].get_legend_handles_labels() \nfig.legend(labels=['Impago','Paid'], handles=reversed(handles), ncol=2, bbox_to_anchor=(0.18, 0.984))\nsns.despine(bottom=True, trim=True)\nplt.tight_layout(rect=[0, 0.2, 1, 0.99])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:12.091682Z","iopub.status.busy":"2022-08-04T00:22:12.091254Z","iopub.status.idle":"2022-08-04T00:22:58.156221Z","shell.execute_reply":"2022-08-04T00:22:58.155336Z"},"id":"vQB6IgtJ_aH0","outputId":"664fda85-1397-4134-c1b8-6745d73194ee","papermill":{"duration":46.326772,"end_time":"2022-08-04T00:22:58.287585","exception":false,"start_time":"2022-08-04T00:22:11.960813","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_b[['B_6', 'B_10', 'B_12', 'B_13', 'B_14', 'B_15', 'B_26', 'B_40']].describe().applymap(\"{0:.2f}\".format)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:58.542633Z","iopub.status.busy":"2022-08-04T00:22:58.541954Z","iopub.status.idle":"2022-08-04T00:22:58.694106Z","shell.execute_reply":"2022-08-04T00:22:58.693012Z"},"id":"q9VNj1Rk_aH0","outputId":"521feb16-566c-488a-ee38-23b4deeadbc5","papermill":{"duration":0.282563,"end_time":"2022-08-04T00:22:58.696724","exception":false,"start_time":"2022-08-04T00:22:58.414161","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Analizamos la correlacion entre variables. En este caso encontramos correlaciones lineales mucho mas marcadas que con las variables anteriores. Graficamos algunas de ellas.","metadata":{"id":"EL29qeZ9_aH0","papermill":{"duration":0.125278,"end_time":"2022-08-04T00:22:58.949686","exception":false,"start_time":"2022-08-04T00:22:58.824408","status":"completed"},"tags":[]}},{"cell_type":"code","source":"corr=plot_df.corr()\nmask=np.triu(np.ones_like(corr, dtype=bool))[1:,:-1]\ncorr=corr.iloc[1:,:-1].copy()\nfig, ax = plt.subplots(figsize=(24,22))   \nsns.heatmap(corr, mask=mask, vmin=-1, vmax=1, center=0, annot=True, fmt='.2f', \n            cmap='coolwarm', annot_kws={'fontsize':12,'fontweight':'bold'}, cbar=False)\nax.tick_params(left=False,bottom=False)\nax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right',fontsize=12)\nax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\nplt.title('Correlaciones entre variables de balance\\n', fontsize=16)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:22:59.206167Z","iopub.status.busy":"2022-08-04T00:22:59.205741Z","iopub.status.idle":"2022-08-04T00:23:03.40916Z","shell.execute_reply":"2022-08-04T00:23:03.408347Z"},"id":"JwvQAZSK_aH0","outputId":"81b64a2f-f5d8-4df6-d800-8059cbc1a1b4","papermill":{"duration":4.336512,"end_time":"2022-08-04T00:23:03.414916","exception":false,"start_time":"2022-08-04T00:22:59.078404","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,3, figsize=(16,5))\nfig.suptitle('Relaciones entre las variables de balance, transformacion logaritmica',fontsize=16)\nax[0].hexbin(x='B_37', y='B_1', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[0].text(-2.2,1.1, 'Correlacion: {:.2f}'.format(plot_df[['B_37','B_1']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[0].set(xlabel='B_37',ylabel='B_1')\nax[1].hexbin(x='B_37', y='B_11', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[1].text(-2.2,1.6, 'Correlacion: {:.2f}'.format(plot_df[['B_37','B_11']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[1].set(xlabel='B_37',ylabel='B_11')\nax[2].hexbin(x='B_23', y='B_7', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[2].text(.25,1.1, 'Correlacion: {:.2f}'.format(plot_df[['B_23','B_7']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[2].set(xlabel='B_23',ylabel='B_7')\nfor i in range(3):\n    ax[i].tick_params(left=False,bottom=False)\nsns.despine()\nplt.tight_layout(rect=[0, 0, 1, 0.99])\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:03.743937Z","iopub.status.busy":"2022-08-04T00:23:03.743281Z","iopub.status.idle":"2022-08-04T00:23:04.73654Z","shell.execute_reply":"2022-08-04T00:23:04.735754Z"},"id":"XokDAivY_aH1","outputId":"33ab6399-d822-4101-fcb5-f69f5d4a6f00","papermill":{"duration":1.129729,"end_time":"2022-08-04T00:23:04.739423","exception":false,"start_time":"2022-08-04T00:23:03.609694","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Encontramos a las variables B_39, B_42, B_29 y B_17 con porcentajes elevados de registros sin informacion.","metadata":{"id":"2WfcWFfu_aH1","papermill":{"duration":0.135259,"end_time":"2022-08-04T00:23:05.010834","exception":false,"start_time":"2022-08-04T00:23:04.875575","status":"completed"},"tags":[]}},{"cell_type":"code","source":"null_b=round((data_analysis_b.isna().sum()/data_analysis_b.shape[0]*100),2).sort_values(ascending=False).astype(str)+('%')\nnull_b=null_b.to_frame().rename(columns={0:'Nulos'})\nnull_b.head(10)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:05.281988Z","iopub.status.busy":"2022-08-04T00:23:05.281005Z","iopub.status.idle":"2022-08-04T00:23:05.317371Z","shell.execute_reply":"2022-08-04T00:23:05.31661Z"},"id":"abQaITm__aH1","outputId":"c2d72058-86d4-4c12-ce0c-def3594e7a2c","papermill":{"duration":0.174521,"end_time":"2022-08-04T00:23:05.31954","exception":false,"start_time":"2022-08-04T00:23:05.145019","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Análisis de las variables de gasto**\n\nAnalizamos el dataset enfocandonos en las variables de gasto.","metadata":{"id":"TDcPmJKm_aH1","papermill":{"duration":0.133894,"end_time":"2022-08-04T00:23:05.58721","exception":false,"start_time":"2022-08-04T00:23:05.453316","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_analysis_s = data_analysis.filter(regex='S_')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:05.857253Z","iopub.status.busy":"2022-08-04T00:23:05.856241Z","iopub.status.idle":"2022-08-04T00:23:05.873402Z","shell.execute_reply":"2022-08-04T00:23:05.872493Z"},"id":"krVBWIQe_aH1","papermill":{"duration":0.155478,"end_time":"2022-08-04T00:23:05.875812","exception":false,"start_time":"2022-08-04T00:23:05.720334","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_s.head()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:06.148988Z","iopub.status.busy":"2022-08-04T00:23:06.148429Z","iopub.status.idle":"2022-08-04T00:23:06.17633Z","shell.execute_reply":"2022-08-04T00:23:06.174804Z"},"id":"rUvT-YC__aH1","outputId":"caeb7d11-0c2f-4a45-a637-a08de11cc353","papermill":{"duration":0.17082,"end_time":"2022-08-04T00:23:06.180696","exception":false,"start_time":"2022-08-04T00:23:06.009876","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Viendo la distribución de las variables respecto a la variable objetivo, observamos el mismo comportamiento que con las variables ya analizadas.","metadata":{"papermill":{"duration":0.13822,"end_time":"2022-08-04T00:23:06.464636","exception":false,"start_time":"2022-08-04T00:23:06.326416","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols=[col for col in data_analysis.columns if (col.startswith(('S','t'))) & (col != 'S_2')]\nplot_df=data_analysis[cols]\nfig, ax = plt.subplots(5,5, figsize=(16,20))\nfig.suptitle('Distribucion de las variables de gasto',fontsize=16)\nrow=0\ncol=[0,1,2,3,4]*5\nfor i, column in enumerate(plot_df.columns[:-1]):\n    if (i!=0)&(i%5==0):\n        row+=1\n    sns.kdeplot(x=column, hue='target', palette=pal[::-1], hue_order=[1,0], \n                label=['Impago','Pago'], data=plot_df, \n                fill=True, linewidth=2, legend=False, ax=ax[row,col[i]])\n    ax[row,col[i]].tick_params(left=False,bottom=False)\n    ax[row,col[i]].set(title='\\n\\n{}'.format(column), xlabel='', ylabel=('Densidad' if i%5==0 else ''))\nfor i in range(1,5):\n    ax[4,i].set_visible(False)\nhandles, _ = ax[0,0].get_legend_handles_labels() \nfig.legend(labels=['Impago','Pago'], handles=reversed(handles), ncol=2, bbox_to_anchor=(0.18, 0.985))\nsns.despine(bottom=True, trim=True)\nplt.tight_layout(rect=[0, 0.2, 1, 0.99])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:06.736519Z","iopub.status.busy":"2022-08-04T00:23:06.736118Z","iopub.status.idle":"2022-08-04T00:23:34.425064Z","shell.execute_reply":"2022-08-04T00:23:34.423864Z"},"id":"dFCAqAal_aH1","outputId":"64395220-4197-4e38-c5a9-a6e7ae0b8585","papermill":{"duration":27.827623,"end_time":"2022-08-04T00:23:34.427823","exception":false,"start_time":"2022-08-04T00:23:06.6002","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_s[['S_5', 'S_12', 'S_16', 'S_22', 'S_23', 'S_24', 'S_26']].describe().applymap(\"{0:.2f}\".format)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:34.707159Z","iopub.status.busy":"2022-08-04T00:23:34.706752Z","iopub.status.idle":"2022-08-04T00:23:34.84797Z","shell.execute_reply":"2022-08-04T00:23:34.847121Z"},"id":"jCG2bHV4_aH2","outputId":"7434edd2-9441-4198-b26a-8c9350754b4f","papermill":{"duration":0.282138,"end_time":"2022-08-04T00:23:34.849988","exception":false,"start_time":"2022-08-04T00:23:34.56785","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observando la correlación entre las distintas variables de gasto, sobresalen 2 relaciones fuertes, que separaremos para visualizar más de cerca.","metadata":{"papermill":{"duration":0.137499,"end_time":"2022-08-04T00:23:35.124627","exception":false,"start_time":"2022-08-04T00:23:34.987128","status":"completed"},"tags":[]}},{"cell_type":"code","source":"corr=plot_df.corr()\nmask=np.triu(np.ones_like(corr, dtype=bool))[1:,:-1]\ncorr=corr.iloc[1:,:-1].copy()\nfig, ax = plt.subplots(figsize=(24,22))   \nsns.heatmap(corr, mask=mask, vmin=-1, vmax=1, center=0, annot=True, fmt='.2f', \n            cmap='coolwarm', annot_kws={'fontsize':12,'fontweight':'bold'}, cbar=False)\nax.tick_params(left=False,bottom=False)\nax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right',fontsize=12)\nax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\nplt.title('Correlaciones entre variables de gasto\\n', fontsize=16)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:35.458748Z","iopub.status.busy":"2022-08-04T00:23:35.458097Z","iopub.status.idle":"2022-08-04T00:23:37.251567Z","shell.execute_reply":"2022-08-04T00:23:37.250236Z"},"id":"HAPMllZ1_aH2","outputId":"c39b8e7d-5631-4b72-9d7a-a7b76431bb37","papermill":{"duration":1.935458,"end_time":"2022-08-04T00:23:37.255983","exception":false,"start_time":"2022-08-04T00:23:35.320525","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,2, figsize=(12,5))\nfig.suptitle('Relaciones entre las variables de balance, transformacion logaritmica',fontsize=16)\nax[0].hexbin(x='S_7', y='S_3', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[0].text(0,5, 'Correlacion: {:.2f}'.format(plot_df[['S_7','S_3']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[0].set(xlabel='S_7',ylabel='S_3')\nax[1].hexbin(x='S_24', y='S_22', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[1].text(-80,0, 'Correlacion: {:.2f}'.format(plot_df[['S_24','S_22']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[1].set(xlabel='S_24',ylabel='S_22')\nfor i in range(2):\n    ax[i].tick_params(left=False,bottom=False)\nsns.despine()\nplt.tight_layout(rect=[0, 0, 1, 0.99])\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:37.550212Z","iopub.status.busy":"2022-08-04T00:23:37.549499Z","iopub.status.idle":"2022-08-04T00:23:38.284531Z","shell.execute_reply":"2022-08-04T00:23:38.283787Z"},"id":"LK_sVIt3_aH3","outputId":"d2644c73-c854-4f75-d313-f459d3bbfa9c","papermill":{"duration":0.88503,"end_time":"2022-08-04T00:23:38.287072","exception":false,"start_time":"2022-08-04T00:23:37.402042","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Analizando los registros nulos, si bien no son tan elevados como en la variable anterior, S_9 supera el 50% del dataset.\n\nOtro dato interesante es que tanto S_7 como S_3 presentan el mismo porcentaje de nulos, y de los gráficos de correlación habíamos observado que se encontraban fuertemente relacionados, por lo tanto tiene sentido que los nulos provengan de la misma fuente de datos.","metadata":{"papermill":{"duration":0.140428,"end_time":"2022-08-04T00:23:38.567746","exception":false,"start_time":"2022-08-04T00:23:38.427318","status":"completed"},"tags":[]}},{"cell_type":"code","source":"null_s=round((data_analysis_s.isna().sum()/data_analysis_s.shape[0]*100),2).sort_values(ascending=False).astype(str)+('%')\nnull_s=null_s.to_frame().rename(columns={0:'Nulos'})\nnull_s.head(10)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:38.854689Z","iopub.status.busy":"2022-08-04T00:23:38.854083Z","iopub.status.idle":"2022-08-04T00:23:38.891722Z","shell.execute_reply":"2022-08-04T00:23:38.891038Z"},"id":"Wz8Am7Uu_aH3","outputId":"948217bf-d995-46fa-df43-d1833fae521e","papermill":{"duration":0.184245,"end_time":"2022-08-04T00:23:38.893616","exception":false,"start_time":"2022-08-04T00:23:38.709371","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Análisis de las variables de delincuencia**\n\nAnalizamos el dataset enfocandonos en las variables de delincuencia.","metadata":{"id":"8EYa1Ek1_aH3","papermill":{"duration":0.14577,"end_time":"2022-08-04T00:23:39.182344","exception":false,"start_time":"2022-08-04T00:23:39.036574","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_analysis_d = data_analysis.filter(regex='D_')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:39.465799Z","iopub.status.busy":"2022-08-04T00:23:39.465355Z","iopub.status.idle":"2022-08-04T00:23:39.538508Z","shell.execute_reply":"2022-08-04T00:23:39.537369Z"},"id":"redQHGrF_aH3","papermill":{"duration":0.218036,"end_time":"2022-08-04T00:23:39.541079","exception":false,"start_time":"2022-08-04T00:23:39.323043","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_analysis_d.head()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:39.825485Z","iopub.status.busy":"2022-08-04T00:23:39.825079Z","iopub.status.idle":"2022-08-04T00:23:39.851087Z","shell.execute_reply":"2022-08-04T00:23:39.850078Z"},"id":"lWgPGXdf_aH3","outputId":"300852f4-762e-4e51-de12-0290b1c3b11a","papermill":{"duration":0.170747,"end_time":"2022-08-04T00:23:39.853274","exception":false,"start_time":"2022-08-04T00:23:39.682527","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Volvemos a observar distribuciones parecidas respecto al resto de las variables, donde no se ve de forma marcada que una variable de delincuencia contribuya a la detección de una deuda impaga. Para D_88 por ejemplo si bien se observa mayor cantidad de deudas impagas, no nos alcanza para asumir que en la totalidad de los casos donde esto ocurra, la deuda será impaga.","metadata":{"papermill":{"duration":0.147387,"end_time":"2022-08-04T00:23:40.144951","exception":false,"start_time":"2022-08-04T00:23:39.997564","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols=[col for col in data_analysis.columns if (col.startswith(('D','t'))) & (col not in categorical_ignore_target)]\nplot_df=data_analysis[cols]\nfig, ax = plt.subplots(18,5, figsize=(16,54))\nfig.suptitle('Distribucion de las variables de delincuencia',fontsize=16)\nrow=0\ncol=[0,1,2,3,4]*18\nfor i, column in enumerate(plot_df.columns[:-1]):\n    if (i!=0)&(i%5==0):\n        row+=1\n    sns.kdeplot(x=column, hue='target', palette=pal[::-1], hue_order=[1,0], \n                label=['Impago','Pago'], data=plot_df, \n                fill=True, linewidth=2, legend=False, ax=ax[row,col[i]])\n    ax[row,col[i]].tick_params(left=False,bottom=False)\n    ax[row,col[i]].set(title='\\n\\n{}'.format(column), xlabel='', ylabel=('Densidad' if i%5==0 else ''))\nfor i in range(2,5):\n    ax[17,i].set_visible(False)\nhandles, _ = ax[0,0].get_legend_handles_labels() \nfig.legend(labels=['Impago','Pago'], handles=reversed(handles), ncol=2, bbox_to_anchor=(0.18, 0.984))\nsns.despine(bottom=True, trim=True)\nplt.tight_layout(rect=[0, 0.2, 1, 0.99])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:23:40.462592Z","iopub.status.busy":"2022-08-04T00:23:40.461515Z","iopub.status.idle":"2022-08-04T00:25:14.396201Z","shell.execute_reply":"2022-08-04T00:25:14.3949Z"},"id":"xS857k29_aH3","outputId":"bfee65b8-40e0-41f8-f017-64266237c609","papermill":{"duration":94.235043,"end_time":"2022-08-04T00:25:14.552177","exception":false,"start_time":"2022-08-04T00:23:40.317134","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Analizando la correlación, vemos correlaciones marcadas entre diferentes variables. Hacemos zoom en 3 correlaciones particulares:","metadata":{"papermill":{"duration":0.147451,"end_time":"2022-08-04T00:25:14.907034","exception":false,"start_time":"2022-08-04T00:25:14.759583","status":"completed"},"tags":[]}},{"cell_type":"code","source":"corr=plot_df.corr()\nmask=np.triu(np.ones_like(corr, dtype=bool))[1:,:-1]\ncorr=corr.iloc[1:,:-1].copy()\nfig, ax = plt.subplots(figsize=(60,60))   \nsns.heatmap(corr, mask=mask, vmin=-1, vmax=1, center=0, annot=True, fmt='.2f', \n            cmap='coolwarm', annot_kws={'fontsize':12,'fontweight':'bold'}, cbar=False)\nax.tick_params(left=False,bottom=False)\nax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right',fontsize=12)\nax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\nplt.title('Correlaciones entre variables de delincuencia\\n', fontsize=16)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:15.20412Z","iopub.status.busy":"2022-08-04T00:25:15.203701Z","iopub.status.idle":"2022-08-04T00:25:36.64771Z","shell.execute_reply":"2022-08-04T00:25:36.646532Z"},"id":"FY9aCiR9_aH3","outputId":"44cbde4b-8288-408c-a260-38820e788f8b","papermill":{"duration":21.612691,"end_time":"2022-08-04T00:25:36.668075","exception":false,"start_time":"2022-08-04T00:25:15.055384","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,3, figsize=(12,5))\nfig.suptitle('Relaciones entre las variables de delincuencia, transformacion logaritmica',fontsize=16)\nax[0].hexbin(x='D_75', y='D_74', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[0].text(1,4, 'Correlacion: {:.2f}'.format(plot_df[['D_75','D_74']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[0].set(xlabel='D_75',ylabel='D_74')\nax[1].hexbin(x='D_77', y='D_62', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[1].text(2,11, 'Correlacion: {:.2f}'.format(plot_df[['D_77','D_62']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[1].set(xlabel='D_77',ylabel='D_62')\nax[2].hexbin(x='D_119', y='D_118', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[2].text(0.5,1.8, 'Correlacion: {:.2f}'.format(plot_df[['D_119','D_118']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[2].set(xlabel='D_119',ylabel='D_118')\nfor i in range(3):\n    ax[i].tick_params(left=False,bottom=False)\nsns.despine()\nplt.tight_layout(rect=[0, 0, 1, 0.99])\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:37.023814Z","iopub.status.busy":"2022-08-04T00:25:37.023162Z","iopub.status.idle":"2022-08-04T00:25:37.962117Z","shell.execute_reply":"2022-08-04T00:25:37.961081Z"},"id":"bdgRtwzP_aH3","outputId":"6014ae9f-f0e1-4f0d-98e8-efc8b7fbd2bf","papermill":{"duration":1.119479,"end_time":"2022-08-04T00:25:37.964345","exception":false,"start_time":"2022-08-04T00:25:36.844866","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A diferencia del resto de las variables, para las de delincuencia se observa una cantidad bastante más marcada de nulos, donde hay 14 columnas con más del 90% de nulos.","metadata":{"papermill":{"duration":0.173893,"end_time":"2022-08-04T00:25:38.310379","exception":false,"start_time":"2022-08-04T00:25:38.136486","status":"completed"},"tags":[]}},{"cell_type":"code","source":"null_d=round((data_analysis_d.isna().sum()/data_analysis_d.shape[0]*100),2).sort_values(ascending=False).astype(str)+('%')\nnull_d=null_d.to_frame().rename(columns={0:'Nulos'})\nnull_d.head(30)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:38.656024Z","iopub.status.busy":"2022-08-04T00:25:38.655535Z","iopub.status.idle":"2022-08-04T00:25:38.744656Z","shell.execute_reply":"2022-08-04T00:25:38.743661Z"},"id":"H6zKoAMN_aH3","outputId":"7e4afda0-581e-4a3c-957b-793ea65ecf05","papermill":{"duration":0.264732,"end_time":"2022-08-04T00:25:38.746841","exception":false,"start_time":"2022-08-04T00:25:38.482109","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Análisis y limpieza de valores nulos\n\nSi bien ya vimos los nulos separados por cada tipo de variable, vamos a realizar un análisis más general de los nulos del dataset para decidir que hacer con ellos.","metadata":{"id":"-6Im-Gvi_aH3","papermill":{"duration":0.174489,"end_time":"2022-08-04T00:25:39.099097","exception":false,"start_time":"2022-08-04T00:25:38.924608","status":"completed"},"tags":[]}},{"cell_type":"code","source":"nulls=round((data_analysis.isna().sum()/data_analysis.shape[0]*100),2).sort_values(ascending=False)\nnulls=nulls.to_frame().rename(columns={0:'Nulos (%)'})\nnulls.head(30)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:39.447525Z","iopub.status.busy":"2022-08-04T00:25:39.447066Z","iopub.status.idle":"2022-08-04T00:25:39.618614Z","shell.execute_reply":"2022-08-04T00:25:39.617399Z"},"id":"-_q2P_M8_aH3","outputId":"bc6f0629-d18d-43b5-b4c5-85450ca42e70","papermill":{"duration":0.349064,"end_time":"2022-08-04T00:25:39.621058","exception":false,"start_time":"2022-08-04T00:25:39.271994","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dada la cantidad de columnas con valores nulos, eliminaremos aquellas con un porcentaje mayor al 80% porque de los valores restantes, si los completamos con un valor (la media o la mediana), no nos daría mucha información de la columna. Y si eliminamos los registros nulos, perderíamos casi la totalidad del dataset.","metadata":{"id":"ceL45c6A_aH3","papermill":{"duration":0.174546,"end_time":"2022-08-04T00:25:39.965981","exception":false,"start_time":"2022-08-04T00:25:39.791435","status":"completed"},"tags":[]}},{"cell_type":"code","source":"nulls_cols = nulls[nulls['Nulos (%)'] >= 80]\nnulls_cols","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:40.32089Z","iopub.status.busy":"2022-08-04T00:25:40.320173Z","iopub.status.idle":"2022-08-04T00:25:40.332721Z","shell.execute_reply":"2022-08-04T00:25:40.331526Z"},"id":"hFWdfTp0_aH3","outputId":"cbd8e31d-6062-411b-cd87-9a2c7e946343","papermill":{"duration":0.192385,"end_time":"2022-08-04T00:25:40.334969","exception":false,"start_time":"2022-08-04T00:25:40.142584","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.drop(columns = nulls_cols.index.values, inplace = True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:40.681437Z","iopub.status.busy":"2022-08-04T00:25:40.680998Z","iopub.status.idle":"2022-08-04T00:25:40.850896Z","shell.execute_reply":"2022-08-04T00:25:40.849718Z"},"id":"pnyEWPMCG14Y","papermill":{"duration":0.346698,"end_time":"2022-08-04T00:25:40.853521","exception":false,"start_time":"2022-08-04T00:25:40.506823","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:41.205867Z","iopub.status.busy":"2022-08-04T00:25:41.204816Z","iopub.status.idle":"2022-08-04T00:25:41.333998Z","shell.execute_reply":"2022-08-04T00:25:41.332992Z"},"id":"PpSFurpk_aH3","outputId":"b52860be-b050-45aa-a0df-b3a0bfeb42b1","papermill":{"duration":0.31116,"end_time":"2022-08-04T00:25:41.336342","exception":false,"start_time":"2022-08-04T00:25:41.025182","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Guardamos todas las variables con nulos en una lista para analizarlos","metadata":{"id":"ZTvJbzkU_aH3","papermill":{"duration":0.170173,"end_time":"2022-08-04T00:25:41.732727","exception":false,"start_time":"2022-08-04T00:25:41.562554","status":"completed"},"tags":[]}},{"cell_type":"code","source":"analisis_nulos = nulls_cols.index.values\nanalisis_nulos","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:42.090718Z","iopub.status.busy":"2022-08-04T00:25:42.090058Z","iopub.status.idle":"2022-08-04T00:25:42.095668Z","shell.execute_reply":"2022-08-04T00:25:42.094898Z"},"id":"5ed0Gsaz_aH3","outputId":"aaa6759c-61f9-4f37-b709-fcc802885241","papermill":{"duration":0.18884,"end_time":"2022-08-04T00:25:42.097682","exception":false,"start_time":"2022-08-04T00:25:41.908842","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos si para las variables que quedaron alguna esta muy correlacionada con otra para poder borrarla","metadata":{"id":"0LSQ0rk4_aH4","papermill":{"duration":0.17474,"end_time":"2022-08-04T00:25:42.442223","exception":false,"start_time":"2022-08-04T00:25:42.267483","status":"completed"},"tags":[]}},{"cell_type":"code","source":"analisis_cols = [x for x in analisis_nulos if (x not in categorical_ignore_target)]\ndata_columns = [x for x in data_analysis.columns.values if (x != 'customer_ID') & (x != 'S_2') & (x not in categorical_ignore_target) & (x not in analisis_cols)]\n#Create a new dictionary\nplotDict = {}\n# Loop across each of the two lists that contain the items you want to compare\nfor gene1 in list(analisis_cols):\n    for gene2 in list(data_columns):\n        df_nonulls = data_analysis[[gene1, gene2]].dropna(axis=0)\n        if(len(df_nonulls[gene1]) < 2):\n            continue\n        # Do a pearsonR comparison between the two items you want to compare\n        tempDict = {(gene1, gene2): \n                    scipy.stats.pearsonr(df_nonulls[gene1],df_nonulls[gene2])\n                    }\n        # Update the dictionary each time you do a comparison\n        plotDict.update(tempDict)\n# Unstack the dictionary into a DataFrame\ndfOutput = pd.Series(plotDict, dtype='float64').unstack()\n# Optional: Take just the pearsonR value out of the output tuple\ncorr = dfOutput.apply(lambda x: x.apply(lambda x:x[0]))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:25:42.78928Z","iopub.status.busy":"2022-08-04T00:25:42.788804Z","iopub.status.idle":"2022-08-04T00:26:11.373378Z","shell.execute_reply":"2022-08-04T00:26:11.371548Z"},"id":"9nv1KdQ0_aH4","papermill":{"duration":28.764368,"end_time":"2022-08-04T00:26:11.378507","exception":false,"start_time":"2022-08-04T00:25:42.614139","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Debido a que es una gran cantidad de variables las que queremos comparar, vamos a separar el heatmap en dos partes","metadata":{"papermill":{"duration":0.174088,"end_time":"2022-08-04T00:26:11.751644","exception":false,"start_time":"2022-08-04T00:26:11.577556","status":"completed"},"tags":[]}},{"cell_type":"code","source":"mid = int(len(corr.columns) / 2)\ncorr_a = corr[corr.columns[0:mid]]\ncorr_b = corr[corr.columns[mid:]]\nfig, ax = plt.subplots(2, figsize=(32,12))\nfig.suptitle('Correlaciones entre variables eliminadas y el resto del dataset', fontsize=16)\nsns.heatmap(corr_a, vmin=-1, vmax=1, center=0, square=True, linewidth=0.01, \n            cmap='coolwarm', cbar=False, ax=ax[0])\nax[0].tick_params(left=False,bottom=False)\nax[0].set_xticklabels(ax[0].get_xticklabels(), horizontalalignment='right',fontsize=8)\nax[0].set_yticklabels(ax[0].get_yticklabels(), fontsize=8)\nsns.heatmap(corr_b, vmin=-1, vmax=1, center=0, square=True, linewidth=0.01, \n            cmap='coolwarm', cbar=False, ax=ax[1])\nax[1].tick_params(left=False,bottom=False)\nax[1].set_xticklabels(ax[1].get_xticklabels(), horizontalalignment='right',fontsize=8)\nax[1].set_yticklabels(ax[1].get_yticklabels(), fontsize=8)\nfig.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:12.115645Z","iopub.status.busy":"2022-08-04T00:26:12.115166Z","iopub.status.idle":"2022-08-04T00:26:16.497496Z","shell.execute_reply":"2022-08-04T00:26:16.496563Z"},"id":"INHS5WFr_aH4","outputId":"be64b35c-1577-4f88-c15a-cb5db13fd1d6","papermill":{"duration":4.572123,"end_time":"2022-08-04T00:26:16.500126","exception":false,"start_time":"2022-08-04T00:26:11.928003","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Definimos un valor de correlación como threshold de 0.7 para aquellas que podemos borrar.","metadata":{"papermill":{"duration":0.173234,"end_time":"2022-08-04T00:26:16.86302","exception":false,"start_time":"2022-08-04T00:26:16.689786","status":"completed"},"tags":[]}},{"cell_type":"code","source":"threshold = .5\ndelete_spectre = .7\ncorr_columns = [corr.columns[j] for i, j in zip(*np.where(np.abs(corr.values) >= delete_spectre))]\ncorr_elevadas = [f\"{corr.index[i]} y {corr.columns[j]} = {corr.iloc[i][j]:.2f}\" for i, j in zip(*np.where(np.abs(corr.values) >= threshold))]\nprint(\"\\n\" + f\"Columnas a eliminar (corr > {delete_spectre}): \" + \", \".join(corr_columns) + \"\\n\")\nprint(\"Columnas con correlaciones elevadas: \" + \", \".join(corr_elevadas))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:17.211146Z","iopub.status.busy":"2022-08-04T00:26:17.210398Z","iopub.status.idle":"2022-08-04T00:26:17.220068Z","shell.execute_reply":"2022-08-04T00:26:17.218981Z"},"id":"aSSanjVoHYeP","outputId":"553f96fb-2ca4-4e6b-f96f-fa1728818378","papermill":{"duration":0.186305,"end_time":"2022-08-04T00:26:17.222969","exception":false,"start_time":"2022-08-04T00:26:17.036664","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(2,2, figsize=(8,8))\ncols=[col for col in data_analysis.columns if (col.startswith(('B','D'))) & (col not in categorical_ignore_target)]\nplot_df=data_analysis[cols]\nfig.suptitle('Relaciones entre las variables de delincuencia, transformacion logaritmica',fontsize=16)\nax[0][0].hexbin(x='D_131', y='D_132', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[0][0].text(1.1,6, 'Correlacion: {:.2f}'.format(plot_df[['D_131','D_132']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[0][0].set(xlabel='D_131',ylabel='D_132')\nax[0][1].hexbin(x='D_142', y='D_141', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[0][1].text(0.5, 1.3, 'Correlacion: {:.2f}'.format(plot_df[['D_142','D_141']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[0][1].set(xlabel='D_142',ylabel='D_141')\nax[1][0].hexbin(x='B_17', y='B_39', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[1][0].text(0.2, 1.15, 'Correlacion: {:.2f}'.format(plot_df[['B_17','B_39']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[1][0].set(xlabel='B_17',ylabel='B_39')\nax[1][1].hexbin(x='D_59', y='D_111', data=plot_df, bins='log', gridsize=40, cmap='coolwarm')\nax[1][1].text(0.4, 0.8, 'Correlacion: {:.2f}'.format(plot_df[['D_59','D_111']].corr().iloc[1,0]), \n           ha=\"center\", va=\"center\",bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\nax[1][1].set(xlabel='D_59',ylabel='D_111')\n\nfor i in range(2):\n    for j in range(2):\n        ax[i][j].tick_params(left=False,bottom=False)\nsns.despine()\nplt.tight_layout(rect=[0, 0, 1, 0.99])\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:17.572497Z","iopub.status.busy":"2022-08-04T00:26:17.571628Z","iopub.status.idle":"2022-08-04T00:26:18.800552Z","shell.execute_reply":"2022-08-04T00:26:18.799774Z"},"id":"DeuC1f8i_aH4","outputId":"54eeb02f-0c40-4aa7-9e5d-1ffadb18867c","papermill":{"duration":1.406959,"end_time":"2022-08-04T00:26:18.803302","exception":false,"start_time":"2022-08-04T00:26:17.396343","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.drop(columns = corr_columns, inplace = True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:19.173133Z","iopub.status.busy":"2022-08-04T00:26:19.172224Z","iopub.status.idle":"2022-08-04T00:26:19.330018Z","shell.execute_reply":"2022-08-04T00:26:19.329239Z"},"id":"-yz9OlnL_aH5","papermill":{"duration":0.338924,"end_time":"2022-08-04T00:26:19.332352","exception":false,"start_time":"2022-08-04T00:26:18.993428","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:19.692232Z","iopub.status.busy":"2022-08-04T00:26:19.691571Z","iopub.status.idle":"2022-08-04T00:26:19.80299Z","shell.execute_reply":"2022-08-04T00:26:19.802162Z"},"id":"L_Fmcx7h_aH5","outputId":"659af2f6-1ced-48fb-95a0-094e652cfa43","papermill":{"duration":0.297986,"end_time":"2022-08-04T00:26:19.805185","exception":false,"start_time":"2022-08-04T00:26:19.507199","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Luedo de haber tratado las de > 80% y filtrado las que están muy correlacionadas, vemos qué variables quedaron con nulos:","metadata":{"id":"45PTdI_-_aH5","papermill":{"duration":0.179739,"end_time":"2022-08-04T00:26:20.258154","exception":false,"start_time":"2022-08-04T00:26:20.078415","status":"completed"},"tags":[]}},{"cell_type":"code","source":"fill_nulls=round((data.isna().sum()/data.shape[0]*100),2).sort_values(ascending=False)\nfill_nulls=fill_nulls.to_frame().rename(columns={0:'Nulos (%)'})\nfill_nulls.head(30)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:20.620558Z","iopub.status.busy":"2022-08-04T00:26:20.619966Z","iopub.status.idle":"2022-08-04T00:26:20.781871Z","shell.execute_reply":"2022-08-04T00:26:20.780882Z"},"id":"NuOAsd05IK6r","outputId":"6dfccf22-f0cc-45ef-addd-26918c8fa939","papermill":{"duration":0.345178,"end_time":"2022-08-04T00:26:20.783956","exception":false,"start_time":"2022-08-04T00:26:20.438778","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"De todas las variables la mayoría presenta una baja cantidad de nulos; para las que presentan un porcentaje importante ser verá de llenar esos nulos.","metadata":{"id":"ZM03Ol2AIQUt","papermill":{"duration":0.185051,"end_time":"2022-08-04T00:26:21.150025","exception":false,"start_time":"2022-08-04T00:26:20.964974","status":"completed"},"tags":[]}},{"cell_type":"code","source":"fill_nulls=fill_nulls[fill_nulls['Nulos (%)'] > 10]\nfill_nulls","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:21.521894Z","iopub.status.busy":"2022-08-04T00:26:21.521459Z","iopub.status.idle":"2022-08-04T00:26:21.533152Z","shell.execute_reply":"2022-08-04T00:26:21.532154Z"},"id":"9RxglWEY_aH5","outputId":"e21f442f-6de2-4be2-f6bb-a1bc7cd62374","papermill":{"duration":0.201393,"end_time":"2022-08-04T00:26:21.536011","exception":false,"start_time":"2022-08-04T00:26:21.334618","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos las medidas de resumen para esas columnas:","metadata":{"id":"KUbNQVxd_aH5","papermill":{"duration":0.258561,"end_time":"2022-08-04T00:26:21.970732","exception":false,"start_time":"2022-08-04T00:26:21.712171","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data[fill_nulls.index.values].describe().applymap(lambda x: f\"{x:0.2f}\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:22.327424Z","iopub.status.busy":"2022-08-04T00:26:22.326798Z","iopub.status.idle":"2022-08-04T00:26:22.706503Z","shell.execute_reply":"2022-08-04T00:26:22.705432Z"},"id":"gt3XXwmA_aH5","outputId":"3b74f64c-27d4-49bc-9053-fe51adfb7c7e","papermill":{"duration":0.559927,"end_time":"2022-08-04T00:26:22.70878","exception":false,"start_time":"2022-08-04T00:26:22.148853","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos que la desviación estandar es baja para todas las variables (< 0.7) por lo que decidimos llenar los nulos de cada una con sus medianas.","metadata":{"id":"RaQY1t8n_aH6","papermill":{"duration":0.179309,"end_time":"2022-08-04T00:26:23.066104","exception":false,"start_time":"2022-08-04T00:26:22.886795","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data[fill_nulls.index.values] = data[fill_nulls.index.values].fillna(data[fill_nulls.index.values].median())","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:23.427574Z","iopub.status.busy":"2022-08-04T00:26:23.426879Z","iopub.status.idle":"2022-08-04T00:26:23.614725Z","shell.execute_reply":"2022-08-04T00:26:23.614008Z"},"id":"WCTbRh4M_aH6","papermill":{"duration":0.373304,"end_time":"2022-08-04T00:26:23.616963","exception":false,"start_time":"2022-08-04T00:26:23.243659","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"remain_nulls = round((data.isna().sum()/data.shape[0]*100),2).sort_values(ascending=False)\nremain_nulls=remain_nulls.to_frame().rename(columns={0:'Nulos (%)'})\nremain_nulls.head(30)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:23.981342Z","iopub.status.busy":"2022-08-04T00:26:23.980111Z","iopub.status.idle":"2022-08-04T00:26:24.141028Z","shell.execute_reply":"2022-08-04T00:26:24.139999Z"},"id":"yHY8LyvR_aH6","outputId":"0a01e923-6e95-4a73-fafc-87df42d1dce2","papermill":{"duration":0.345281,"end_time":"2022-08-04T00:26:24.143365","exception":false,"start_time":"2022-08-04T00:26:23.798084","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos que como resultado de todo el análisis de las columans con nulos, las columnas que nos quedaron con nulos sin tratar representan un porcentaje muy bajo. Para cada modelo luego vamos a decidir qué hacer con estos nulos.","metadata":{"id":"piuAJJSU_aH6","papermill":{"duration":0.18158,"end_time":"2022-08-04T00:26:24.504512","exception":false,"start_time":"2022-08-04T00:26:24.322932","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Análisis de outliers","metadata":{"id":"S8SyHYRn_aH6","papermill":{"duration":0.182417,"end_time":"2022-08-04T00:26:24.925436","exception":false,"start_time":"2022-08-04T00:26:24.743019","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Comenzaremos el análisis graficando un boxplot para cada variable del dataset. Utilizaremos un boxplot ya que permite ver para todas las variables al mismo tiempo de forma práctica si hay valores atípicos.  \nEl boxplot permite realizar un análisis univariado de los datos para cada columna, resultando en posibles outliers moderados aquellos que se encuentran entre la caja (espacio entre el primer y tercer cuartil) y los bigotes (mínimo/máximo), y outliers severos aquellos que se encuentran más allá de los bigotes.","metadata":{"id":"OyYNnl1e_aH6","papermill":{"duration":0.203234,"end_time":"2022-08-04T00:26:25.313611","exception":false,"start_time":"2022-08-04T00:26:25.110377","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_bp = data.select_dtypes(exclude=['object'])\nplots_per_column = 4\ntotal_rows = math.ceil(len(data_bp.columns) / plots_per_column)\nfig, axes = plt.subplots(total_rows,plots_per_column, figsize=(20, 80)) # create figure and axes\nrow=0\ncol = list(range(0, plots_per_column)) * total_rows\ndata_bp_columns = sorted(data_bp.columns)\nfor i, column in enumerate(data_bp_columns):\n    if (i!=0)&(i%plots_per_column==0):\n        row+=1\n        \n    sns.boxplot(y=column, data=data_bp, ax=axes[row,col[i]])\n    \nfor i in range(1, plots_per_column):\n    axes[total_rows-1,i].set_visible(False)\n\nplt.tight_layout()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:25.66993Z","iopub.status.busy":"2022-08-04T00:26:25.669192Z","iopub.status.idle":"2022-08-04T00:26:50.283356Z","shell.execute_reply":"2022-08-04T00:26:50.282416Z"},"id":"SltUXZBC_aH6","papermill":{"duration":24.796891,"end_time":"2022-08-04T00:26:50.286607","exception":false,"start_time":"2022-08-04T00:26:25.489716","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado el problema planteado, en donde hay que predecir si un cliente pagará o no, creemos que no tiene sentido eliminar todos los outliers ya que son los que pueden aportar la información más importante. Por lo tanto analizaremos las variables que tengan mayor presencia de outliers. Haciendo zoom en este boxplot se puede ver que son: B_6, B_10, D_69, R_7, B_26, S_16, R_14, S_26 y B_40.\n","metadata":{"id":"NuFwZrm3_aH6","papermill":{"duration":0.183317,"end_time":"2022-08-04T00:26:50.672627","exception":false,"start_time":"2022-08-04T00:26:50.48931","status":"completed"},"tags":[]}},{"cell_type":"code","source":"analisis_outliers = ['B_6', 'B_10', 'D_69', 'R_7', 'B_26', 'S_16', 'R_14', 'S_26', 'B_40']","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:51.044205Z","iopub.status.busy":"2022-08-04T00:26:51.043269Z","iopub.status.idle":"2022-08-04T00:26:51.048154Z","shell.execute_reply":"2022-08-04T00:26:51.047372Z"},"id":"XdghKU0Y_aH6","papermill":{"duration":0.193333,"end_time":"2022-08-04T00:26:51.050203","exception":false,"start_time":"2022-08-04T00:26:50.85687","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Para seguir con el análisis univariado sobre cada una de las variables seleccionadas utilizaremos el método de la métrica Z-Score. Esta métrica se basa en cuan desviado está un dato de la media, asumiendo una distribución gaussiana, resultando los posibles outliers aquellos que den con módulo mayor a 3.","metadata":{"id":"KyItPxLG_aH6","papermill":{"duration":0.193299,"end_time":"2022-08-04T00:26:51.427586","exception":false,"start_time":"2022-08-04T00:26:51.234287","status":"completed"},"tags":[]}},{"cell_type":"code","source":"z_cols = []\nfor col in analisis_outliers:\n    z_col = 'z_' + col\n    media = np.mean(data[col])\n    std = np.std(data[col])\n    data[z_col] = (data[col] - media) / std\n    z_cols.append(z_col)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:51.810696Z","iopub.status.busy":"2022-08-04T00:26:51.809959Z","iopub.status.idle":"2022-08-04T00:26:51.864873Z","shell.execute_reply":"2022-08-04T00:26:51.864023Z"},"id":"4QO3VPp2_aH6","papermill":{"duration":0.249729,"end_time":"2022-08-04T00:26:51.867286","exception":false,"start_time":"2022-08-04T00:26:51.617557","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Así quedan los resultados para cada variable de la métrica de Z-Score","metadata":{"id":"UXOYC8oQ_aH6","papermill":{"duration":0.194636,"end_time":"2022-08-04T00:26:52.248577","exception":false,"start_time":"2022-08-04T00:26:52.053941","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data[z_cols]","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:52.63982Z","iopub.status.busy":"2022-08-04T00:26:52.638073Z","iopub.status.idle":"2022-08-04T00:26:52.98426Z","shell.execute_reply":"2022-08-04T00:26:52.9834Z"},"id":"LVzVSpod_aH6","outputId":"ba6f3e9e-4d8d-4206-b77c-c3848c44ae49","papermill":{"duration":0.54428,"end_time":"2022-08-04T00:26:52.987058","exception":false,"start_time":"2022-08-04T00:26:52.442778","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A continuación verificamos cuántos registros tienen valores mayores a 3 (en módulo)","metadata":{"id":"i4oKUFPG_aH6","papermill":{"duration":0.186985,"end_time":"2022-08-04T00:26:53.357139","exception":false,"start_time":"2022-08-04T00:26:53.170154","status":"completed"},"tags":[]}},{"cell_type":"code","source":"sns.boxplot(y=data['B_6'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:53.728781Z","iopub.status.busy":"2022-08-04T00:26:53.728372Z","iopub.status.idle":"2022-08-04T00:26:53.872649Z","shell.execute_reply":"2022-08-04T00:26:53.871963Z"},"id":"qw-XkF8G_aH6","outputId":"2ecb6e5c-d4b7-4a91-cf4f-5581b91b7862","papermill":{"duration":0.335219,"end_time":"2022-08-04T00:26:53.875978","exception":false,"start_time":"2022-08-04T00:26:53.540759","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[(abs(data['z_B_6']) >= 3)][['B_6', 'z_B_6']].sort_values(by='z_B_6', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:54.321026Z","iopub.status.busy":"2022-08-04T00:26:54.320333Z","iopub.status.idle":"2022-08-04T00:26:54.339509Z","shell.execute_reply":"2022-08-04T00:26:54.338806Z"},"id":"1XK9ABTS_aH6","outputId":"72a8e1c0-f3e8-4582-d3fb-f9bf3da30663","papermill":{"duration":0.26779,"end_time":"2022-08-04T00:26:54.341286","exception":false,"start_time":"2022-08-04T00:26:54.073496","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 580 registros representan sólo el 0.21% del total decidimos eliminarlos.","metadata":{"id":"rmbHnonu_aH6","papermill":{"duration":0.185609,"end_time":"2022-08-04T00:26:54.712495","exception":false,"start_time":"2022-08-04T00:26:54.526886","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[(abs(data['z_B_6']) >= 3)].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:55.091485Z","iopub.status.busy":"2022-08-04T00:26:55.090404Z","iopub.status.idle":"2022-08-04T00:26:55.358319Z","shell.execute_reply":"2022-08-04T00:26:55.357298Z"},"id":"bBJ22_T3_aH6","papermill":{"duration":0.46087,"end_time":"2022-08-04T00:26:55.360829","exception":false,"start_time":"2022-08-04T00:26:54.899959","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(y=data['B_10'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:55.733973Z","iopub.status.busy":"2022-08-04T00:26:55.733504Z","iopub.status.idle":"2022-08-04T00:26:55.926289Z","shell.execute_reply":"2022-08-04T00:26:55.925509Z"},"id":"Gu_9irLl_aH6","outputId":"77e98dec-8210-42fd-ba96-77eb7249f10b","papermill":{"duration":0.382923,"end_time":"2022-08-04T00:26:55.928463","exception":false,"start_time":"2022-08-04T00:26:55.54554","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[(abs(data['z_B_10']) >= 3)][['B_10', 'z_B_10']].sort_values(by='z_B_10', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:56.298439Z","iopub.status.busy":"2022-08-04T00:26:56.29751Z","iopub.status.idle":"2022-08-04T00:26:56.314596Z","shell.execute_reply":"2022-08-04T00:26:56.313956Z"},"id":"2st7eALO_aH6","outputId":"fb8b5282-0aec-4d1a-ade7-a268a60f3569","papermill":{"duration":0.203302,"end_time":"2022-08-04T00:26:56.316499","exception":false,"start_time":"2022-08-04T00:26:56.113197","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 172 registros representan sólo el 0.062% del total decidimos eliminarlos.","metadata":{"id":"zPxSh6ET_aH7","papermill":{"duration":0.182681,"end_time":"2022-08-04T00:26:56.685621","exception":false,"start_time":"2022-08-04T00:26:56.50294","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[(abs(data['z_B_10']) >= 3)].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:57.059523Z","iopub.status.busy":"2022-08-04T00:26:57.058753Z","iopub.status.idle":"2022-08-04T00:26:57.320574Z","shell.execute_reply":"2022-08-04T00:26:57.319288Z"},"id":"im-pKqm3_aH7","papermill":{"duration":0.451596,"end_time":"2022-08-04T00:26:57.323299","exception":false,"start_time":"2022-08-04T00:26:56.871703","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(y=data['D_69'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:57.698872Z","iopub.status.busy":"2022-08-04T00:26:57.698459Z","iopub.status.idle":"2022-08-04T00:26:57.89311Z","shell.execute_reply":"2022-08-04T00:26:57.892255Z"},"id":"rRq04mlh_aH7","outputId":"99c72e35-1e25-4780-f4b3-1136eb01f5ba","papermill":{"duration":0.385509,"end_time":"2022-08-04T00:26:57.895145","exception":false,"start_time":"2022-08-04T00:26:57.509636","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[(abs(data['z_D_69']) >= 3)][['D_69', 'z_D_69']].sort_values(by='z_D_69', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:58.290067Z","iopub.status.busy":"2022-08-04T00:26:58.289018Z","iopub.status.idle":"2022-08-04T00:26:58.306621Z","shell.execute_reply":"2022-08-04T00:26:58.305976Z"},"id":"BX2AuxLv_aH7","outputId":"ee79da93-e23b-4efb-c634-e15451646f05","papermill":{"duration":0.225112,"end_time":"2022-08-04T00:26:58.308503","exception":false,"start_time":"2022-08-04T00:26:58.083391","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 73 registros representan sólo el 0.026% del total decidimos eliminarlos.","metadata":{"id":"l3_W3T31_aH7","papermill":{"duration":0.184794,"end_time":"2022-08-04T00:26:58.67617","exception":false,"start_time":"2022-08-04T00:26:58.491376","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[(abs(data['z_D_69']) >= 3)].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:59.057884Z","iopub.status.busy":"2022-08-04T00:26:59.056882Z","iopub.status.idle":"2022-08-04T00:26:59.323967Z","shell.execute_reply":"2022-08-04T00:26:59.322876Z"},"id":"FMxqCASn_aH7","papermill":{"duration":0.460617,"end_time":"2022-08-04T00:26:59.326668","exception":false,"start_time":"2022-08-04T00:26:58.866051","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(y=data['R_7'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:26:59.708682Z","iopub.status.busy":"2022-08-04T00:26:59.708273Z","iopub.status.idle":"2022-08-04T00:26:59.901136Z","shell.execute_reply":"2022-08-04T00:26:59.900119Z"},"id":"xCthY3r5_aH7","outputId":"abb0b113-44e0-496a-a7ab-d36c03cd0cda","papermill":{"duration":0.386642,"end_time":"2022-08-04T00:26:59.903217","exception":false,"start_time":"2022-08-04T00:26:59.516575","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[(abs(data['z_R_7']) >= 3)][['R_7', 'z_R_7']].sort_values(by='z_R_7', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:00.286891Z","iopub.status.busy":"2022-08-04T00:27:00.286203Z","iopub.status.idle":"2022-08-04T00:27:00.306119Z","shell.execute_reply":"2022-08-04T00:27:00.30519Z"},"id":"crzA5kV__aH7","outputId":"af378794-1a9a-4b80-87d4-b269b06b19aa","papermill":{"duration":0.213662,"end_time":"2022-08-04T00:27:00.308254","exception":false,"start_time":"2022-08-04T00:27:00.094592","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 918 registros representan sólo el 0.332% del total decidimos eliminarlos.","metadata":{"id":"irXB0hkn_aH7","papermill":{"duration":0.190856,"end_time":"2022-08-04T00:27:00.687138","exception":false,"start_time":"2022-08-04T00:27:00.496282","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[(abs(data['z_R_7']) >= 3)].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:01.072197Z","iopub.status.busy":"2022-08-04T00:27:01.07115Z","iopub.status.idle":"2022-08-04T00:27:01.334932Z","shell.execute_reply":"2022-08-04T00:27:01.333969Z"},"id":"LbapU4wh_aH7","papermill":{"duration":0.457758,"end_time":"2022-08-04T00:27:01.337439","exception":false,"start_time":"2022-08-04T00:27:00.879681","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(y=data['B_26'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:01.769226Z","iopub.status.busy":"2022-08-04T00:27:01.768328Z","iopub.status.idle":"2022-08-04T00:27:01.915632Z","shell.execute_reply":"2022-08-04T00:27:01.914863Z"},"id":"R5TzeTda_aH8","outputId":"53bde37a-5bed-4294-c275-f6e8b2ff0af6","papermill":{"duration":0.395609,"end_time":"2022-08-04T00:27:01.918197","exception":false,"start_time":"2022-08-04T00:27:01.522588","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[(abs(data['z_B_26']) >= 3)][['B_26', 'z_B_26']].sort_values(by='z_B_26', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:02.289735Z","iopub.status.busy":"2022-08-04T00:27:02.288979Z","iopub.status.idle":"2022-08-04T00:27:02.306307Z","shell.execute_reply":"2022-08-04T00:27:02.30529Z"},"id":"5AcFuFfK_aH8","outputId":"3e413e74-2334-49bc-acdc-89123355e9e9","papermill":{"duration":0.204788,"end_time":"2022-08-04T00:27:02.308468","exception":false,"start_time":"2022-08-04T00:27:02.10368","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 370 registros representan sólo el 0.134% del total decidimos eliminarlos.","metadata":{"id":"0Qbi4ADz_aH8","papermill":{"duration":0.18699,"end_time":"2022-08-04T00:27:02.680303","exception":false,"start_time":"2022-08-04T00:27:02.493313","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[(abs(data['z_B_26']) >= 3)].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:03.052842Z","iopub.status.busy":"2022-08-04T00:27:03.052029Z","iopub.status.idle":"2022-08-04T00:27:03.313395Z","shell.execute_reply":"2022-08-04T00:27:03.31239Z"},"id":"Ppd2w72c_aH8","papermill":{"duration":0.450887,"end_time":"2022-08-04T00:27:03.315979","exception":false,"start_time":"2022-08-04T00:27:02.865092","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(y=data['S_16'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:03.695946Z","iopub.status.busy":"2022-08-04T00:27:03.695509Z","iopub.status.idle":"2022-08-04T00:27:03.892969Z","shell.execute_reply":"2022-08-04T00:27:03.892124Z"},"id":"o39bpH0X_aH8","outputId":"5e31af26-07d7-4d0e-cbaa-4ec860114b59","papermill":{"duration":0.395356,"end_time":"2022-08-04T00:27:03.895116","exception":false,"start_time":"2022-08-04T00:27:03.49976","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[(abs(data['z_S_16']) >= 3)][['S_16', 'z_S_16']].sort_values(by='z_S_16', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:04.281422Z","iopub.status.busy":"2022-08-04T00:27:04.280968Z","iopub.status.idle":"2022-08-04T00:27:04.299916Z","shell.execute_reply":"2022-08-04T00:27:04.299139Z"},"id":"a1CstRYg_aH8","outputId":"8882786a-f1a3-4b92-aca9-f4b64159b355","papermill":{"duration":0.21416,"end_time":"2022-08-04T00:27:04.301801","exception":false,"start_time":"2022-08-04T00:27:04.087641","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 1475 registros representan sólo el 0.533% del total decidimos eliminarlos.","metadata":{"id":"QffEL_61_aH8","papermill":{"duration":0.186561,"end_time":"2022-08-04T00:27:04.694896","exception":false,"start_time":"2022-08-04T00:27:04.508335","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[(abs(data['z_S_16']) >= 3)].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:05.081152Z","iopub.status.busy":"2022-08-04T00:27:05.080462Z","iopub.status.idle":"2022-08-04T00:27:05.352443Z","shell.execute_reply":"2022-08-04T00:27:05.351439Z"},"id":"egwyn6Zc_aH8","papermill":{"duration":0.4712,"end_time":"2022-08-04T00:27:05.355323","exception":false,"start_time":"2022-08-04T00:27:04.884123","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(y=data['R_14'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:05.779132Z","iopub.status.busy":"2022-08-04T00:27:05.778302Z","iopub.status.idle":"2022-08-04T00:27:05.986751Z","shell.execute_reply":"2022-08-04T00:27:05.985832Z"},"id":"jp5N51s9_aH8","outputId":"1fd7ab25-bb2b-40a3-aaca-4ddd0272fd38","papermill":{"duration":0.420721,"end_time":"2022-08-04T00:27:05.989491","exception":false,"start_time":"2022-08-04T00:27:05.56877","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[abs(data['z_R_14']) >= 3][['R_14', 'z_R_14']].sort_values(by='z_R_14', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:06.382505Z","iopub.status.busy":"2022-08-04T00:27:06.381838Z","iopub.status.idle":"2022-08-04T00:27:06.398608Z","shell.execute_reply":"2022-08-04T00:27:06.397953Z"},"id":"37HstOKP_aH8","outputId":"208888ac-ad0b-425e-bd8e-31a31ce230ee","papermill":{"duration":0.2092,"end_time":"2022-08-04T00:27:06.400527","exception":false,"start_time":"2022-08-04T00:27:06.191327","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 432 registros representan sólo el 0.156% del total decidimos eliminarlos.","metadata":{"id":"aesDPkKb_aH8","papermill":{"duration":0.189963,"end_time":"2022-08-04T00:27:06.776162","exception":false,"start_time":"2022-08-04T00:27:06.586199","status":"completed"},"tags":[]}},{"cell_type":"code","source":"sns.boxplot(y=data['S_26'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:07.156836Z","iopub.status.busy":"2022-08-04T00:27:07.155751Z","iopub.status.idle":"2022-08-04T00:27:07.461108Z","shell.execute_reply":"2022-08-04T00:27:07.459985Z"},"id":"jzMjd7Q8_aH8","outputId":"ae0e844b-9037-4e84-effa-516f4018a60d","papermill":{"duration":0.499255,"end_time":"2022-08-04T00:27:07.463709","exception":false,"start_time":"2022-08-04T00:27:06.964454","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[abs(data['z_S_26']) >= 3][['S_26', 'z_S_26']].sort_values(by='z_S_26', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:07.852463Z","iopub.status.busy":"2022-08-04T00:27:07.851772Z","iopub.status.idle":"2022-08-04T00:27:07.871781Z","shell.execute_reply":"2022-08-04T00:27:07.870791Z"},"id":"SMXPqMwk_aH8","outputId":"0e99f9d4-fed6-4f34-e991-d82d527fac50","papermill":{"duration":0.218351,"end_time":"2022-08-04T00:27:07.873963","exception":false,"start_time":"2022-08-04T00:27:07.655612","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 1235 registros representan sólo el 0.447% del total decidimos eliminarlos.","metadata":{"id":"keidmr44_aH8","papermill":{"duration":0.196357,"end_time":"2022-08-04T00:27:08.261833","exception":false,"start_time":"2022-08-04T00:27:08.065476","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[abs(data['z_S_26']) >= 3].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:08.644117Z","iopub.status.busy":"2022-08-04T00:27:08.64336Z","iopub.status.idle":"2022-08-04T00:27:08.905726Z","shell.execute_reply":"2022-08-04T00:27:08.904418Z"},"id":"Ha0c5Pb7_aH8","papermill":{"duration":0.456307,"end_time":"2022-08-04T00:27:08.908436","exception":false,"start_time":"2022-08-04T00:27:08.452129","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(y=data['B_40'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:09.301104Z","iopub.status.busy":"2022-08-04T00:27:09.300642Z","iopub.status.idle":"2022-08-04T00:27:09.506714Z","shell.execute_reply":"2022-08-04T00:27:09.505731Z"},"id":"RqCNCojK_aH8","outputId":"29f1e787-e28e-4a64-a846-59ec79f55535","papermill":{"duration":0.410643,"end_time":"2022-08-04T00:27:09.509036","exception":false,"start_time":"2022-08-04T00:27:09.098393","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[abs(data['z_B_40']) >= 3][['B_40', 'z_B_40']].sort_values(by='z_B_40', ascending=False)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:09.890072Z","iopub.status.busy":"2022-08-04T00:27:09.889372Z","iopub.status.idle":"2022-08-04T00:27:09.905474Z","shell.execute_reply":"2022-08-04T00:27:09.904487Z"},"id":"lVTfdmR__aH8","outputId":"9c18c3fd-b631-451a-b91b-6ab559e62248","papermill":{"duration":0.213087,"end_time":"2022-08-04T00:27:09.907737","exception":false,"start_time":"2022-08-04T00:27:09.69465","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que los 259 registros representan sólo el 0.094% del total decidimos eliminarlos.","metadata":{"id":"VhQJln_j_aH9","papermill":{"duration":0.197022,"end_time":"2022-08-04T00:27:10.296398","exception":false,"start_time":"2022-08-04T00:27:10.099376","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(data[abs(data['z_B_40']) >= 3].index, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:10.687883Z","iopub.status.busy":"2022-08-04T00:27:10.687204Z","iopub.status.idle":"2022-08-04T00:27:10.945354Z","shell.execute_reply":"2022-08-04T00:27:10.944446Z"},"id":"ciUGf4V6_aH9","papermill":{"duration":0.456876,"end_time":"2022-08-04T00:27:10.947622","exception":false,"start_time":"2022-08-04T00:27:10.490746","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos cómo quedó el dataset finalmente","metadata":{"id":"8iUi97Ho_aH9","papermill":{"duration":0.338959,"end_time":"2022-08-04T00:27:11.479177","exception":false,"start_time":"2022-08-04T00:27:11.140218","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:11.856551Z","iopub.status.busy":"2022-08-04T00:27:11.855853Z","iopub.status.idle":"2022-08-04T00:27:12.003517Z","shell.execute_reply":"2022-08-04T00:27:12.001848Z"},"id":"Nazzzt_9_aH9","outputId":"ada45fba-709a-4eee-9e96-28a9f41f2bb1","papermill":{"duration":0.342078,"end_time":"2022-08-04T00:27:12.008765","exception":false,"start_time":"2022-08-04T00:27:11.666687","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"De los 276572 registros originales quedaron 271490, lo que equivale al 98.16% del dataset. ","metadata":{"id":"u0TGybMd_aH9","papermill":{"duration":0.187555,"end_time":"2022-08-04T00:27:12.38341","exception":false,"start_time":"2022-08-04T00:27:12.195855","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Eliminamos las columnas que agregamos para evaluar los outliers.","metadata":{"id":"uWbGLIiL_aH9","papermill":{"duration":0.189145,"end_time":"2022-08-04T00:27:12.760302","exception":false,"start_time":"2022-08-04T00:27:12.571157","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(columns=z_cols, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:13.143966Z","iopub.status.busy":"2022-08-04T00:27:13.143511Z","iopub.status.idle":"2022-08-04T00:27:13.275329Z","shell.execute_reply":"2022-08-04T00:27:13.274167Z"},"id":"w_YKZnsR_aH9","papermill":{"duration":0.327784,"end_time":"2022-08-04T00:27:13.277878","exception":false,"start_time":"2022-08-04T00:27:12.950094","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:13.663754Z","iopub.status.busy":"2022-08-04T00:27:13.663348Z","iopub.status.idle":"2022-08-04T00:27:13.803204Z","shell.execute_reply":"2022-08-04T00:27:13.80209Z"},"id":"6u-MKENp_aH9","outputId":"dde234c0-47e0-45e4-e0fa-829f0cb0f2d7","papermill":{"duration":0.337394,"end_time":"2022-08-04T00:27:13.805744","exception":false,"start_time":"2022-08-04T00:27:13.46835","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Columnas a agregar\n\nRevisamos si hay datos que aporten información como para crear columnas nuevas.\n\nDe las columnas que estan representadas en forma de texto, nos fijamos qué información presentan:","metadata":{"id":"qqTv1OJG_aH9","papermill":{"duration":0.189439,"end_time":"2022-08-04T00:27:14.18359","exception":false,"start_time":"2022-08-04T00:27:13.994151","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols_txt = data.select_dtypes(include=['object']).columns\ncols_txt","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:14.560147Z","iopub.status.busy":"2022-08-04T00:27:14.559729Z","iopub.status.idle":"2022-08-04T00:27:14.584781Z","shell.execute_reply":"2022-08-04T00:27:14.583688Z"},"id":"Jmzro0SI_aH9","outputId":"58435a99-328a-481d-877a-4357af0bb2aa","papermill":{"duration":0.215888,"end_time":"2022-08-04T00:27:14.587035","exception":false,"start_time":"2022-08-04T00:27:14.371147","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[cols_txt]","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:14.969351Z","iopub.status.busy":"2022-08-04T00:27:14.968946Z","iopub.status.idle":"2022-08-04T00:27:14.993956Z","shell.execute_reply":"2022-08-04T00:27:14.992926Z"},"id":"jFzJ8H1d_aH9","outputId":"47a49388-4538-4060-cf6f-d8957c1cac18","papermill":{"duration":0.220809,"end_time":"2022-08-04T00:27:14.996177","exception":false,"start_time":"2022-08-04T00:27:14.775368","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Customer ID es el identificador del deudor al cual queremos clasificar. S_2 representa una fecha la cual podemos dividir en 3 variables numéricas (año, mes y día), mientras que D_63 y D_64 son strings que representan diferentes categorías.\n\nCrearemos una columna para el año, el mes y el día extraidos de la columna S_2","metadata":{"papermill":{"duration":0.191869,"end_time":"2022-08-04T00:27:15.381381","exception":false,"start_time":"2022-08-04T00:27:15.189512","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data['S_2'] = pd.to_datetime(data['S_2']) \ndata['S_2_Year'] = data['S_2'].dt.year\ndata['S_2_Month'] = data['S_2'].dt.month\ndata['S_2_Day'] = data['S_2'].dt.day","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:15.765673Z","iopub.status.busy":"2022-08-04T00:27:15.765237Z","iopub.status.idle":"2022-08-04T00:27:15.933521Z","shell.execute_reply":"2022-08-04T00:27:15.932449Z"},"id":"fuZcyrkG_aH-","papermill":{"duration":0.362831,"end_time":"2022-08-04T00:27:15.936011","exception":false,"start_time":"2022-08-04T00:27:15.57318","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[['S_2', 'S_2_Year', 'S_2_Month', 'S_2_Day']]","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:16.509788Z","iopub.status.busy":"2022-08-04T00:27:16.50902Z","iopub.status.idle":"2022-08-04T00:27:16.535749Z","shell.execute_reply":"2022-08-04T00:27:16.534771Z"},"id":"od8UIf8v_aH-","outputId":"f3b42a64-cf30-44b2-b8da-e618e87f624c","papermill":{"duration":0.410643,"end_time":"2022-08-04T00:27:16.538029","exception":false,"start_time":"2022-08-04T00:27:16.127386","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Borramos S_2 asi no duplicamos informacion","metadata":{"papermill":{"duration":0.189806,"end_time":"2022-08-04T00:27:16.938157","exception":false,"start_time":"2022-08-04T00:27:16.748351","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.drop(columns=['S_2'], inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:17.31723Z","iopub.status.busy":"2022-08-04T00:27:17.316579Z","iopub.status.idle":"2022-08-04T00:27:17.444723Z","shell.execute_reply":"2022-08-04T00:27:17.443664Z"},"papermill":{"duration":0.321141,"end_time":"2022-08-04T00:27:17.447166","exception":false,"start_time":"2022-08-04T00:27:17.126025","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reducción de dimensionalidad","metadata":{"id":"aInYhfj6_aH-","papermill":{"duration":0.192137,"end_time":"2022-08-04T00:27:17.82693","exception":false,"start_time":"2022-08-04T00:27:17.634793","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.shape","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:18.207302Z","iopub.status.busy":"2022-08-04T00:27:18.20668Z","iopub.status.idle":"2022-08-04T00:27:18.212928Z","shell.execute_reply":"2022-08-04T00:27:18.211985Z"},"id":"IBs7ECtt_aH-","outputId":"faa2d6f7-11de-4ef8-a6ea-23a4d0cc8ed4","papermill":{"duration":0.197338,"end_time":"2022-08-04T00:27:18.215018","exception":false,"start_time":"2022-08-04T00:27:18.01768","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que hay muchas columnas vemos si es posible reducir la cantidad de columnas para mejorar los tiempos de ejecución. Para ello utilizaremos PCA (Análisis de Componentes Principales).","metadata":{"id":"RBR5lCxC_aH_","papermill":{"duration":0.186619,"end_time":"2022-08-04T00:27:18.588883","exception":false,"start_time":"2022-08-04T00:27:18.402264","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.decomposition import PCA","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:18.976601Z","iopub.status.busy":"2022-08-04T00:27:18.975789Z","iopub.status.idle":"2022-08-04T00:27:18.980661Z","shell.execute_reply":"2022-08-04T00:27:18.979978Z"},"id":"7rSKGxxC_aH_","papermill":{"duration":0.202608,"end_time":"2022-08-04T00:27:18.982706","exception":false,"start_time":"2022-08-04T00:27:18.780098","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que sólo funciona con variables numéricas filtraremos las columnas que cumplan con este requisito","metadata":{"id":"bm6cpA7w_aH_","papermill":{"duration":0.19482,"end_time":"2022-08-04T00:27:19.375202","exception":false,"start_time":"2022-08-04T00:27:19.180382","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols = data.select_dtypes(include=['number']).columns\ncols = [x for x in cols if (x not in ['S_2_Year', 'S_2_Month', 'S_2_Day'])]\nnot_cols = [x for x in data.columns if (x not in cols)]\nnot_cols","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:19.76987Z","iopub.status.busy":"2022-08-04T00:27:19.769204Z","iopub.status.idle":"2022-08-04T00:27:19.889Z","shell.execute_reply":"2022-08-04T00:27:19.887927Z"},"id":"6ozJe4U6_aIB","outputId":"6abec2d6-76e7-4de2-8cf4-89c33027e32f","papermill":{"duration":0.320071,"end_time":"2022-08-04T00:27:19.89148","exception":false,"start_time":"2022-08-04T00:27:19.571409","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_parte_a = data[not_cols].copy()\ndata_parte_b = data[cols].copy()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:20.371865Z","iopub.status.busy":"2022-08-04T00:27:20.371466Z","iopub.status.idle":"2022-08-04T00:27:20.650749Z","shell.execute_reply":"2022-08-04T00:27:20.649772Z"},"id":"xiGL-2D2_aIB","papermill":{"duration":0.566256,"end_time":"2022-08-04T00:27:20.653205","exception":false,"start_time":"2022-08-04T00:27:20.086949","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nulos=round((data_parte_b.isna().sum()/data_parte_b.shape[0]*100),5).sort_values(ascending=False)\nnulos=nulos.to_frame().rename(columns={0:'Nulos (%)'})\nnulos = nulos[nulos['Nulos (%)'] > 0]\nnulos","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:21.0399Z","iopub.status.busy":"2022-08-04T00:27:21.039154Z","iopub.status.idle":"2022-08-04T00:27:21.150965Z","shell.execute_reply":"2022-08-04T00:27:21.150002Z"},"id":"-Oey_eoq_aIB","outputId":"9d01ea6b-6a02-4361-e270-d31dcb9cb102","papermill":{"duration":0.310097,"end_time":"2022-08-04T00:27:21.153121","exception":false,"start_time":"2022-08-04T00:27:20.843024","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dado que PCA no permite las variables con nulos, vamos a llenar los nulos con la mediana de cada variable dado que los porcentajes son chicos y no se modificará de una forma importante su distribución.","metadata":{"id":"8DJdrgVe_aIC","papermill":{"duration":0.366942,"end_time":"2022-08-04T00:27:21.710503","exception":false,"start_time":"2022-08-04T00:27:21.343561","status":"completed"},"tags":[]}},{"cell_type":"code","source":"for col in nulos.index.values:\n    data_parte_b[col] = data_parte_b[col].fillna(data_parte_b[col].median())","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:22.091212Z","iopub.status.busy":"2022-08-04T00:27:22.090707Z","iopub.status.idle":"2022-08-04T00:27:22.658068Z","shell.execute_reply":"2022-08-04T00:27:22.656965Z"},"id":"3-PTjzsb_aIC","papermill":{"duration":0.760793,"end_time":"2022-08-04T00:27:22.660625","exception":false,"start_time":"2022-08-04T00:27:21.899832","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tipos_de_variables = \"DSPBR\"\nfor tipo in tipos_de_variables:\n    var = tipo + '_'\n    print(f\"{var}: {len(data_parte_b.filter(regex=var).columns)}\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:23.041988Z","iopub.status.busy":"2022-08-04T00:27:23.04125Z","iopub.status.idle":"2022-08-04T00:27:23.163096Z","shell.execute_reply":"2022-08-04T00:27:23.161873Z"},"id":"g-mW5eE-_aIC","outputId":"5de9773a-dc57-40d8-fe4a-a2343ee7a6ec","papermill":{"duration":0.314972,"end_time":"2022-08-04T00:27:23.165405","exception":false,"start_time":"2022-08-04T00:27:22.850433","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_d = data_parte_b.filter(regex=\"D_\").copy()\ndata_s = data_parte_b.filter(regex=\"S_\").copy()\ndata_b = data_parte_b.filter(regex=\"B_\").copy()\ndata_r = data_parte_b.filter(regex=\"R_\").copy()\ndata_p = data_parte_b.filter(regex=\"P_\").copy()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:23.548266Z","iopub.status.busy":"2022-08-04T00:27:23.547828Z","iopub.status.idle":"2022-08-04T00:27:23.813653Z","shell.execute_reply":"2022-08-04T00:27:23.812473Z"},"id":"uO_P09h2_aID","papermill":{"duration":0.460739,"end_time":"2022-08-04T00:27:23.8163","exception":false,"start_time":"2022-08-04T00:27:23.355561","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Al momento de reducir la dimensionalidad, buscaremos quedarnos con las variables que cubran el 97.5% de la varianza ya que de quedarnos con el 95% podríamos perder valores _\"borde\"_ que aporten buena información para este problema puntual.","metadata":{"id":"i9q3m7EI_aID","papermill":{"duration":0.194237,"end_time":"2022-08-04T00:27:24.260182","exception":false,"start_time":"2022-08-04T00:27:24.065945","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Como las variables respectivas a la delincuencia son las más abundantes, empezaremos por reducir esas variables.","metadata":{"id":"dPQthtGN_aIE","papermill":{"duration":0.189049,"end_time":"2022-08-04T00:27:24.641094","exception":false,"start_time":"2022-08-04T00:27:24.452045","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pca_data = PCA()\npca_data.fit(data_d)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:25.021267Z","iopub.status.busy":"2022-08-04T00:27:25.020842Z","iopub.status.idle":"2022-08-04T00:27:26.803673Z","shell.execute_reply":"2022-08-04T00:27:26.802984Z"},"id":"_eSTOFcR_aIE","outputId":"a5ddd167-21f2-4c78-f1c6-346a2ca4a245","papermill":{"duration":1.975319,"end_time":"2022-08-04T00:27:26.805585","exception":false,"start_time":"2022-08-04T00:27:24.830266","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Calculamos la cantidad de componentes principales a utilizar","metadata":{"id":"FnZx-WYa_aIE","papermill":{"duration":0.188592,"end_time":"2022-08-04T00:27:27.182782","exception":false,"start_time":"2022-08-04T00:27:26.99419","status":"completed"},"tags":[]}},{"cell_type":"code","source":"var_cumu = np.cumsum(pca_data.explained_variance_ratio_) * 100","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:27.562393Z","iopub.status.busy":"2022-08-04T00:27:27.561646Z","iopub.status.idle":"2022-08-04T00:27:27.565681Z","shell.execute_reply":"2022-08-04T00:27:27.565071Z"},"id":"MgF-GwLB_aIE","papermill":{"duration":0.195859,"end_time":"2022-08-04T00:27:27.567536","exception":false,"start_time":"2022-08-04T00:27:27.371677","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"k = np.argmax(var_cumu > 97.5)\nprint(\"El numero minimo de componentes para explicar el 97.5% de la varianza es: \" + str(k))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:28.00302Z","iopub.status.busy":"2022-08-04T00:27:28.002549Z","iopub.status.idle":"2022-08-04T00:27:28.00793Z","shell.execute_reply":"2022-08-04T00:27:28.0072Z"},"id":"VbjO0sj3_aIE","outputId":"ad0d169b-42ac-4ac5-c187-35035d48b942","papermill":{"duration":0.255661,"end_time":"2022-08-04T00:27:28.01074","exception":false,"start_time":"2022-08-04T00:27:27.755079","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=[10, 5])\nplt.title('Varianza acumulada explicada por componente')\nplt.ylabel('Varianza acumulada explicada')\nplt.xlabel('Componentes principales')\nplt.axvline(x=k, color=\"k\", linestyle=\"--\")\nplt.axhline(y=97.5, color=\"r\", linestyle=\"--\")\nax = plt.plot(var_cumu)\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:28.393965Z","iopub.status.busy":"2022-08-04T00:27:28.392472Z","iopub.status.idle":"2022-08-04T00:27:28.625219Z","shell.execute_reply":"2022-08-04T00:27:28.624207Z"},"id":"gcC39CIq_aIE","outputId":"51cfc764-6d7c-4894-daa1-77d530e15110","papermill":{"duration":0.426961,"end_time":"2022-08-04T00:27:28.627612","exception":false,"start_time":"2022-08-04T00:27:28.200651","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Continuamos con la cantidad de componentes principales que nos dan el 97.5% de variabilidad explicada","metadata":{"id":"sVo0HSyp_aIE","papermill":{"duration":0.191425,"end_time":"2022-08-04T00:27:29.011011","exception":false,"start_time":"2022-08-04T00:27:28.819586","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pca = PCA(n_components=k)\npca_transform=pca.fit_transform(data_d)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:29.392015Z","iopub.status.busy":"2022-08-04T00:27:29.391254Z","iopub.status.idle":"2022-08-04T00:27:34.127458Z","shell.execute_reply":"2022-08-04T00:27:34.12631Z"},"id":"_8R6dlLS_aIE","papermill":{"duration":4.928996,"end_time":"2022-08-04T00:27:34.129886","exception":false,"start_time":"2022-08-04T00:27:29.20089","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"columnas = [\"D_PCA_\" + f\"{i}\" for i in range(41)]\npca_d = pd.DataFrame(data = pca_transform, columns=columnas)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:34.512419Z","iopub.status.busy":"2022-08-04T00:27:34.512025Z","iopub.status.idle":"2022-08-04T00:27:34.517841Z","shell.execute_reply":"2022-08-04T00:27:34.516871Z"},"id":"B7k9gs4X_aIE","papermill":{"duration":0.199254,"end_time":"2022-08-04T00:27:34.519959","exception":false,"start_time":"2022-08-04T00:27:34.320705","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_d","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:34.90248Z","iopub.status.busy":"2022-08-04T00:27:34.901715Z","iopub.status.idle":"2022-08-04T00:27:35.030954Z","shell.execute_reply":"2022-08-04T00:27:35.029999Z"},"id":"F17mczar_aIF","outputId":"728e68b1-7095-46d8-d5e3-2ed38ba63bf2","papermill":{"duration":0.322945,"end_time":"2022-08-04T00:27:35.033142","exception":false,"start_time":"2022-08-04T00:27:34.710197","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Repetimos el proceso ahora para las variables respectivas al gasto","metadata":{"id":"MpXuQXJE_aIF","papermill":{"duration":0.191114,"end_time":"2022-08-04T00:27:35.414511","exception":false,"start_time":"2022-08-04T00:27:35.223397","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pca_data = PCA()\npca_data.fit(data_s)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:35.858054Z","iopub.status.busy":"2022-08-04T00:27:35.857272Z","iopub.status.idle":"2022-08-04T00:27:36.125633Z","shell.execute_reply":"2022-08-04T00:27:36.124629Z"},"id":"VhilnDlG_aIF","outputId":"23a27a65-840c-43b3-c84e-b81739038c98","papermill":{"duration":0.52459,"end_time":"2022-08-04T00:27:36.129039","exception":false,"start_time":"2022-08-04T00:27:35.604449","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Calculamos la cantidad de componentes principales a utilizar","metadata":{"id":"qRVxeFpS_aIF","papermill":{"duration":0.189146,"end_time":"2022-08-04T00:27:36.551865","exception":false,"start_time":"2022-08-04T00:27:36.362719","status":"completed"},"tags":[]}},{"cell_type":"code","source":"var_cumu = np.cumsum(pca_data.explained_variance_ratio_) * 100\n\nk = np.argmax(var_cumu > 97.5)\nprint(\"El numero minimo de componentes para explicar el 97.5% de la varianza es: \" + str(k))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:36.934755Z","iopub.status.busy":"2022-08-04T00:27:36.934366Z","iopub.status.idle":"2022-08-04T00:27:36.94049Z","shell.execute_reply":"2022-08-04T00:27:36.939527Z"},"id":"EfELSdir_aIF","outputId":"dd3687d8-e245-48d2-8559-07f85a5e2bd3","papermill":{"duration":0.200653,"end_time":"2022-08-04T00:27:36.942733","exception":false,"start_time":"2022-08-04T00:27:36.74208","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=[10, 5])\nplt.title('Varianza acumulada explicada por componente')\nplt.ylabel('Varianza acumulada explicada')\nplt.xlabel('Componentes principales')\nplt.axvline(x=k, color=\"k\", linestyle=\"--\")\nplt.axhline(y=97.5, color=\"r\", linestyle=\"--\")\nax = plt.plot(var_cumu)\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:37.327446Z","iopub.status.busy":"2022-08-04T00:27:37.326421Z","iopub.status.idle":"2022-08-04T00:27:37.553233Z","shell.execute_reply":"2022-08-04T00:27:37.552211Z"},"id":"JMShtJ1i_aIF","outputId":"a49ebc48-30d8-478d-f1dc-a4e6a047d1b1","papermill":{"duration":0.421063,"end_time":"2022-08-04T00:27:37.555471","exception":false,"start_time":"2022-08-04T00:27:37.134408","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Continuamos con la cantidad de componentes principales que nos dan el 97.5% de variabilidad explicada","metadata":{"id":"a5GVhW3-_aIF","papermill":{"duration":0.192032,"end_time":"2022-08-04T00:27:37.946352","exception":false,"start_time":"2022-08-04T00:27:37.75432","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pca = PCA(n_components=k)\npca_transform=pca.fit_transform(data_s)\n\ncolumnas = [\"S_PCA_\" + f\"{i}\" for i in range(14)]\npca_s = pd.DataFrame(data = pca_transform, columns=columnas)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:38.334014Z","iopub.status.busy":"2022-08-04T00:27:38.331897Z","iopub.status.idle":"2022-08-04T00:27:40.593715Z","shell.execute_reply":"2022-08-04T00:27:40.592741Z"},"id":"LIGozg92_aIF","papermill":{"duration":2.457952,"end_time":"2022-08-04T00:27:40.596289","exception":false,"start_time":"2022-08-04T00:27:38.138337","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_s","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:40.985351Z","iopub.status.busy":"2022-08-04T00:27:40.984889Z","iopub.status.idle":"2022-08-04T00:27:41.008975Z","shell.execute_reply":"2022-08-04T00:27:41.008241Z"},"id":"0t8byHEq_aIF","outputId":"581f84cd-013b-457e-9643-f4d3ccc3289b","papermill":{"duration":0.223401,"end_time":"2022-08-04T00:27:41.011014","exception":false,"start_time":"2022-08-04T00:27:40.787613","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Repetimos el proceso ahora para las variables respectivas al balance","metadata":{"id":"7zr_PGiu_aIF","papermill":{"duration":0.19359,"end_time":"2022-08-04T00:27:41.396952","exception":false,"start_time":"2022-08-04T00:27:41.203362","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pca_data = PCA()\npca_data.fit(data_b)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:41.794969Z","iopub.status.busy":"2022-08-04T00:27:41.794198Z","iopub.status.idle":"2022-08-04T00:27:42.527758Z","shell.execute_reply":"2022-08-04T00:27:42.526488Z"},"id":"zffOZDXo_aIF","outputId":"63bbe0fa-0a06-42be-dabd-1aa92ba3dfb6","papermill":{"duration":0.93852,"end_time":"2022-08-04T00:27:42.531459","exception":false,"start_time":"2022-08-04T00:27:41.592939","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Calculamos la cantidad de componentes principales a utilizar","metadata":{"id":"kRDyrnIy_aIF","papermill":{"duration":0.195086,"end_time":"2022-08-04T00:27:42.989208","exception":false,"start_time":"2022-08-04T00:27:42.794122","status":"completed"},"tags":[]}},{"cell_type":"code","source":"var_cumu = np.cumsum(pca_data.explained_variance_ratio_) * 100\n\nk = np.argmax(var_cumu > 97.5)\nprint(\"El numero minimo de componentes para explicar el 97.5% de la varianza es: \" + str(k))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:43.391467Z","iopub.status.busy":"2022-08-04T00:27:43.391068Z","iopub.status.idle":"2022-08-04T00:27:43.396939Z","shell.execute_reply":"2022-08-04T00:27:43.395897Z"},"id":"q7sFT5xt_aIF","outputId":"1b9a2025-1d9f-4456-c80a-0c2eb93ed048","papermill":{"duration":0.212066,"end_time":"2022-08-04T00:27:43.398926","exception":false,"start_time":"2022-08-04T00:27:43.18686","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=[10, 5])\nplt.title('Varianza acumulada explicada por componente')\nplt.ylabel('Varianza acumulada explicada')\nplt.xlabel('Componentes principales')\nplt.axvline(x=k, color=\"k\", linestyle=\"--\")\nplt.axhline(y=97.5, color=\"r\", linestyle=\"--\")\nax = plt.plot(var_cumu)\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:43.786857Z","iopub.status.busy":"2022-08-04T00:27:43.786206Z","iopub.status.idle":"2022-08-04T00:27:44.508682Z","shell.execute_reply":"2022-08-04T00:27:44.507989Z"},"id":"cj-yYpwe_aIF","outputId":"5df211f2-96a4-44cb-de33-c2f268132c94","papermill":{"duration":0.920572,"end_time":"2022-08-04T00:27:44.510872","exception":false,"start_time":"2022-08-04T00:27:43.5903","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Continuamos con la cantidad de componentes principales que nos dan el 95% de variabilidad explicada","metadata":{"id":"g6ofA4I1_aIF","papermill":{"duration":0.199921,"end_time":"2022-08-04T00:27:44.906738","exception":false,"start_time":"2022-08-04T00:27:44.706817","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pca = PCA(n_components=k)\npca_transform=pca.fit_transform(data_b)\n\ncolumnas = [\"B_PCA_\" + f\"{i}\" for i in range(19)]\npca_b = pd.DataFrame(data = pca_transform, columns=columnas)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:45.300145Z","iopub.status.busy":"2022-08-04T00:27:45.299395Z","iopub.status.idle":"2022-08-04T00:27:47.883217Z","shell.execute_reply":"2022-08-04T00:27:47.882198Z"},"id":"97sY3CZQ_aIG","papermill":{"duration":2.779858,"end_time":"2022-08-04T00:27:47.886086","exception":false,"start_time":"2022-08-04T00:27:45.106228","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_b","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:48.272793Z","iopub.status.busy":"2022-08-04T00:27:48.271612Z","iopub.status.idle":"2022-08-04T00:27:48.299343Z","shell.execute_reply":"2022-08-04T00:27:48.298343Z"},"id":"TYbAlenw_aIG","outputId":"dbf5b27f-1345-4fdd-feb0-330cc2fd8859","papermill":{"duration":0.222933,"end_time":"2022-08-04T00:27:48.301672","exception":false,"start_time":"2022-08-04T00:27:48.078739","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Repetimos el proceso ahora para las variables respectivas al riesgo","metadata":{"id":"xN7rwf9O_aIG","papermill":{"duration":0.197876,"end_time":"2022-08-04T00:27:48.69155","exception":false,"start_time":"2022-08-04T00:27:48.493674","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pca_data = PCA()\npca_data.fit(data_r)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:49.091397Z","iopub.status.busy":"2022-08-04T00:27:49.090415Z","iopub.status.idle":"2022-08-04T00:27:49.452192Z","shell.execute_reply":"2022-08-04T00:27:49.450899Z"},"id":"Ajr9P8NR_aIJ","outputId":"4291ea96-8b73-4a5c-c176-358646f4193e","papermill":{"duration":0.566043,"end_time":"2022-08-04T00:27:49.455774","exception":false,"start_time":"2022-08-04T00:27:48.889731","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Calculamos la cantidad de componentes principales a utilizar","metadata":{"id":"R0MnM-Jd_aIK","papermill":{"duration":0.195622,"end_time":"2022-08-04T00:27:49.878988","exception":false,"start_time":"2022-08-04T00:27:49.683366","status":"completed"},"tags":[]}},{"cell_type":"code","source":"var_cumu = np.cumsum(pca_data.explained_variance_ratio_) * 100\n\nk = np.argmax(var_cumu > 97.5)\nprint(\"El numero minimo de componentes para explicar el 97.5% de la varianza es: \" + str(k))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:50.274195Z","iopub.status.busy":"2022-08-04T00:27:50.27351Z","iopub.status.idle":"2022-08-04T00:27:50.280943Z","shell.execute_reply":"2022-08-04T00:27:50.280235Z"},"id":"XZT-ORvG_aIK","outputId":"a21d7a95-ec60-4ede-d616-78838e3bdfc8","papermill":{"duration":0.208219,"end_time":"2022-08-04T00:27:50.283597","exception":false,"start_time":"2022-08-04T00:27:50.075378","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como no tiene sentido eliminar variables, nos vamos a quedar con 5.","metadata":{"id":"48w4YdWV_aIK","papermill":{"duration":0.196654,"end_time":"2022-08-04T00:27:50.678108","exception":false,"start_time":"2022-08-04T00:27:50.481454","status":"completed"},"tags":[]}},{"cell_type":"code","source":"k = 5","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:51.075985Z","iopub.status.busy":"2022-08-04T00:27:51.074928Z","iopub.status.idle":"2022-08-04T00:27:51.080603Z","shell.execute_reply":"2022-08-04T00:27:51.079442Z"},"id":"utMBoq2i_aIL","papermill":{"duration":0.20791,"end_time":"2022-08-04T00:27:51.082675","exception":false,"start_time":"2022-08-04T00:27:50.874765","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=[10, 5])\nplt.title('Varianza acumulada explicada por componente')\nplt.ylabel('Varianza acumulada explicada')\nplt.xlabel('Componentes principales')\nplt.axvline(x=k, color=\"k\", linestyle=\"--\")\nplt.axhline(y=97.5, color=\"r\", linestyle=\"--\")\nax = plt.plot(var_cumu)\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:51.482013Z","iopub.status.busy":"2022-08-04T00:27:51.480931Z","iopub.status.idle":"2022-08-04T00:27:51.703297Z","shell.execute_reply":"2022-08-04T00:27:51.70229Z"},"id":"FRIEDUrn_aIL","outputId":"7cb1dfdb-05a0-492a-ec26-e0f31f00f6da","papermill":{"duration":0.424903,"end_time":"2022-08-04T00:27:51.705728","exception":false,"start_time":"2022-08-04T00:27:51.280825","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca = PCA(n_components=k)\npca_transform=pca.fit_transform(data_r)\n\ncolumnas = [\"R_PCA_\" + f\"{i}\" for i in range(5)]\npca_r = pd.DataFrame(data = pca_transform, columns=columnas)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:52.119208Z","iopub.status.busy":"2022-08-04T00:27:52.118099Z","iopub.status.idle":"2022-08-04T00:27:53.698527Z","shell.execute_reply":"2022-08-04T00:27:53.69754Z"},"id":"m0ZFegJL_aIL","papermill":{"duration":1.778877,"end_time":"2022-08-04T00:27:53.701031","exception":false,"start_time":"2022-08-04T00:27:51.922154","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_r","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:54.096579Z","iopub.status.busy":"2022-08-04T00:27:54.096189Z","iopub.status.idle":"2022-08-04T00:27:54.112468Z","shell.execute_reply":"2022-08-04T00:27:54.111478Z"},"id":"je8enjVf_aIL","outputId":"d1a036ea-3375-435c-cd68-be906c0b109e","papermill":{"duration":0.212936,"end_time":"2022-08-04T00:27:54.114954","exception":false,"start_time":"2022-08-04T00:27:53.902018","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ahora unimos el dataset","metadata":{"id":"ti-x4heF_aIM","papermill":{"duration":0.19266,"end_time":"2022-08-04T00:27:54.556757","exception":false,"start_time":"2022-08-04T00:27:54.364097","status":"completed"},"tags":[]}},{"cell_type":"code","source":"datasets = [data_parte_a, pca_d, pca_s, pca_b, pca_r, data_p]\nfor d in datasets:\n    d.reset_index(inplace=True)\nnuevo = pd.concat(datasets, axis=1)\nnuevo.drop(columns=['index'], inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:54.945493Z","iopub.status.busy":"2022-08-04T00:27:54.945086Z","iopub.status.idle":"2022-08-04T00:27:55.488166Z","shell.execute_reply":"2022-08-04T00:27:55.487196Z"},"id":"y8hlUauv_aIM","papermill":{"duration":0.74122,"end_time":"2022-08-04T00:27:55.490711","exception":false,"start_time":"2022-08-04T00:27:54.749491","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nuevo","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:55.879138Z","iopub.status.busy":"2022-08-04T00:27:55.878707Z","iopub.status.idle":"2022-08-04T00:27:56.046332Z","shell.execute_reply":"2022-08-04T00:27:56.045092Z"},"id":"7Yp0L8fa_aIM","outputId":"e7502e90-3e29-4135-a28d-9f3806efbe41","papermill":{"duration":0.365024,"end_time":"2022-08-04T00:27:56.048822","exception":false,"start_time":"2022-08-04T00:27:55.683798","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:56.435842Z","iopub.status.busy":"2022-08-04T00:27:56.435097Z","iopub.status.idle":"2022-08-04T00:27:56.573577Z","shell.execute_reply":"2022-08-04T00:27:56.572468Z"},"id":"lcv7ApX6_aIM","outputId":"b43fc343-90e5-4c34-b968-b461c2d3e95c","papermill":{"duration":0.334055,"end_time":"2022-08-04T00:27:56.575862","exception":false,"start_time":"2022-08-04T00:27:56.241807","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nos quedamos con el nuevo dataset","metadata":{"id":"kWz0vzbE_aIM","papermill":{"duration":0.201395,"end_time":"2022-08-04T00:27:56.975326","exception":false,"start_time":"2022-08-04T00:27:56.773931","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data = nuevo","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:57.451862Z","iopub.status.busy":"2022-08-04T00:27:57.451432Z","iopub.status.idle":"2022-08-04T00:27:57.456291Z","shell.execute_reply":"2022-08-04T00:27:57.455228Z"},"id":"B9xornmW_aIM","papermill":{"duration":0.277263,"end_time":"2022-08-04T00:27:57.459136","exception":false,"start_time":"2022-08-04T00:27:57.181873","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Normalización de los datos**  \nEvaluamos la posible normalizacion de los datos para cada variable, para eso veremos la antidad de valores que son mayores a 1.","metadata":{"id":"C_yn6tWD_aIM","papermill":{"duration":0.194827,"end_time":"2022-08-04T00:27:57.850212","exception":false,"start_time":"2022-08-04T00:27:57.655385","status":"completed"},"tags":[]}},{"cell_type":"code","source":"contador_total = 0\ncontador_variables = 0\nfor col in data.columns:\n    contador = 0\n    for elemento in data[col]:\n        contador_total += 1\n        if(isinstance(elemento, float) and elemento > 1):\n            contador += 1\n            contador_variables += 1\n    print(f\"{col}: {contador}\")\nprint(f\"Porcentaje total: {round(contador_variables/contador_total, 2)}\" + \"%\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:27:58.240896Z","iopub.status.busy":"2022-08-04T00:27:58.240508Z","iopub.status.idle":"2022-08-04T00:28:08.333343Z","shell.execute_reply":"2022-08-04T00:28:08.332414Z"},"id":"HfgkiTNC_aIN","outputId":"3980e064-e930-4415-c222-191681ba8c49","papermill":{"duration":10.291525,"end_time":"2022-08-04T00:28:08.337642","exception":false,"start_time":"2022-08-04T00:27:58.046117","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:08.736566Z","iopub.status.busy":"2022-08-04T00:28:08.73572Z","iopub.status.idle":"2022-08-04T00:28:08.889694Z","shell.execute_reply":"2022-08-04T00:28:08.888986Z"},"id":"B9VBaqgk_aIN","outputId":"f9481285-52dc-4459-9df3-81a774c8754e","papermill":{"duration":0.353956,"end_time":"2022-08-04T00:28:08.89162","exception":false,"start_time":"2022-08-04T00:28:08.537664","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = data.reindex(sorted(data.columns), axis=1)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:09.289442Z","iopub.status.busy":"2022-08-04T00:28:09.288538Z","iopub.status.idle":"2022-08-04T00:28:09.359443Z","shell.execute_reply":"2022-08-04T00:28:09.358414Z"},"id":"9l-kcN-O_aIN","papermill":{"duration":0.272222,"end_time":"2022-08-04T00:28:09.361985","exception":false,"start_time":"2022-08-04T00:28:09.089763","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:09.763699Z","iopub.status.busy":"2022-08-04T00:28:09.762763Z","iopub.status.idle":"2022-08-04T00:28:09.902784Z","shell.execute_reply":"2022-08-04T00:28:09.901791Z"},"id":"CmojoqwK_aIN","outputId":"7e56e10c-771b-4463-cc78-b53d8749b57e","papermill":{"duration":0.342525,"end_time":"2022-08-04T00:28:09.905013","exception":false,"start_time":"2022-08-04T00:28:09.562488","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in data.columns:\n    if(col not in [\"customer_ID\", \"S_2_Year\", \"S_2_Month\", \"S_2_Day\"]):\n        for elemento in data[col]:\n            if(isinstance(elemento, float) and elemento > 1):\n                data[col]=(data[col]-data[col].min())/(data[col].max()-data[col].min())\n    \ndata","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:10.329872Z","iopub.status.busy":"2022-08-04T00:28:10.328832Z","iopub.status.idle":"2022-08-04T00:28:18.494708Z","shell.execute_reply":"2022-08-04T00:28:18.493684Z"},"id":"pGdaHd19_aIN","outputId":"5e0afe01-2b23-4272-e6e7-b46e613f26a4","papermill":{"duration":8.375317,"end_time":"2022-08-04T00:28:18.497185","exception":false,"start_time":"2022-08-04T00:28:10.121868","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:18.962156Z","iopub.status.busy":"2022-08-04T00:28:18.961414Z","iopub.status.idle":"2022-08-04T00:28:19.094352Z","shell.execute_reply":"2022-08-04T00:28:19.093602Z"},"papermill":{"duration":0.338551,"end_time":"2022-08-04T00:28:19.096196","exception":false,"start_time":"2022-08-04T00:28:18.757645","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tareas de limpieza y transformación extras\n\nVamos a pasar a numeros todas las variables del dataset para no tener problemas con los distintos modelos. Aquellas que no puedan pasarse no serán tenidas en cuenta al momento de dividir el set en train y test.","metadata":{"id":"WGL5Ak_J_aIP","papermill":{"duration":0.202052,"end_time":"2022-08-04T00:28:19.498945","exception":false,"start_time":"2022-08-04T00:28:19.296893","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cols = data.select_dtypes(include=['number']).columns\nnot_cols = [x for x in data.columns if (x not in cols)]\nnot_cols","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:19.903521Z","iopub.status.busy":"2022-08-04T00:28:19.902597Z","iopub.status.idle":"2022-08-04T00:28:19.976494Z","shell.execute_reply":"2022-08-04T00:28:19.975442Z"},"papermill":{"duration":0.279316,"end_time":"2022-08-04T00:28:19.979081","exception":false,"start_time":"2022-08-04T00:28:19.699765","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como habíamos visto en la sección Columnas a agregar, D_63 y D_64 eran columnas con información de tipo Object (string). Veamos cuántos valores pueden tomar","metadata":{"papermill":{"duration":0.200446,"end_time":"2022-08-04T00:28:20.383182","exception":false,"start_time":"2022-08-04T00:28:20.182736","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print(f'D_63 toma {data[\"D_63\"].nunique()} valores posibles')\nprint(f'D_64 toma {data[\"D_64\"].nunique()} valores posibles')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:20.788827Z","iopub.status.busy":"2022-08-04T00:28:20.788169Z","iopub.status.idle":"2022-08-04T00:28:20.822684Z","shell.execute_reply":"2022-08-04T00:28:20.821534Z"},"papermill":{"duration":0.240023,"end_time":"2022-08-04T00:28:20.824892","exception":false,"start_time":"2022-08-04T00:28:20.584869","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Son pocos los valores categorícos que toma, por lo tanto decidimos aplicar One Hot Encoding por medio de get_dummies, y modelamos los nulos como una categoría más:","metadata":{"papermill":{"duration":0.201064,"end_time":"2022-08-04T00:28:21.227233","exception":false,"start_time":"2022-08-04T00:28:21.026169","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data = pd.get_dummies(data, columns=['D_64'], dummy_na=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:21.643027Z","iopub.status.busy":"2022-08-04T00:28:21.642135Z","iopub.status.idle":"2022-08-04T00:28:21.860861Z","shell.execute_reply":"2022-08-04T00:28:21.859841Z"},"papermill":{"duration":0.431647,"end_time":"2022-08-04T00:28:21.863169","exception":false,"start_time":"2022-08-04T00:28:21.431522","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.get_dummies(data, columns=['D_63'], dummy_na=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:22.275876Z","iopub.status.busy":"2022-08-04T00:28:22.27546Z","iopub.status.idle":"2022-08-04T00:28:22.599706Z","shell.execute_reply":"2022-08-04T00:28:22.598805Z"},"papermill":{"duration":0.535444,"end_time":"2022-08-04T00:28:22.601919","exception":false,"start_time":"2022-08-04T00:28:22.066475","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:23.009026Z","iopub.status.busy":"2022-08-04T00:28:23.008428Z","iopub.status.idle":"2022-08-04T00:28:23.069963Z","shell.execute_reply":"2022-08-04T00:28:23.068843Z"},"papermill":{"duration":0.27051,"end_time":"2022-08-04T00:28:23.072266","exception":false,"start_time":"2022-08-04T00:28:22.801756","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Balanceo del dataset\n\nHasta el momento se trabajó con la parte de datos del dataset, ahora haremos un merge con los labels para ver qué tan balanceado se encuentra","metadata":{"id":"tbvxiGzR_aIN","papermill":{"duration":0.202473,"end_time":"2022-08-04T00:28:23.477457","exception":false,"start_time":"2022-08-04T00:28:23.274984","status":"completed"},"tags":[]}},{"cell_type":"code","source":"completo = pd.merge(data, labels, on='customer_ID', how='inner')\ncompleto","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:23.944646Z","iopub.status.busy":"2022-08-04T00:28:23.944003Z","iopub.status.idle":"2022-08-04T00:28:24.421567Z","shell.execute_reply":"2022-08-04T00:28:24.420349Z"},"id":"6nHx0fS3_aIO","outputId":"67b261e6-c9b9-444f-db06-b79b12997460","papermill":{"duration":0.741405,"end_time":"2022-08-04T00:28:24.423974","exception":false,"start_time":"2022-08-04T00:28:23.682569","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"colores=[\"#75201a\",\"#194373\"]\ncompleto['target'].value_counts().plot.bar(alpha=0.9, color=colores)\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:24.833945Z","iopub.status.busy":"2022-08-04T00:28:24.833525Z","iopub.status.idle":"2022-08-04T00:28:25.04086Z","shell.execute_reply":"2022-08-04T00:28:25.039895Z"},"id":"I8XLueMk_aIO","outputId":"e9a5482a-4e97-445f-e391-2e452e4b57f3","papermill":{"duration":0.411494,"end_time":"2022-08-04T00:28:25.043123","exception":false,"start_time":"2022-08-04T00:28:24.631629","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vemos que el dataset está desbalanceado, hay mucha cantidad de registros para los cuales no se hizo el pago (target == 0).\n\nEs por ello que vamos a armar un segundo dataset con un balanceo por cliente manteniendo una proporcion 70 / 30 entre clientes no deudores y clientes deudores","metadata":{"id":"QBBdTNgv_aIO","papermill":{"duration":0.200708,"end_time":"2022-08-04T00:28:25.447397","exception":false,"start_time":"2022-08-04T00:28:25.246689","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cant_total = completo.shape[0]\ncant_clientes = len(completo['customer_ID'].unique())\ncant_clientes_morosos = len(completo[completo['target'] == 1]['customer_ID'].unique())\ncant_clientes_pagos = len(completo[completo['target'] == 0]['customer_ID'].unique())\ncant_target1 = completo[completo['target'] == 1].shape[0]\ncant_target0 = completo[completo['target'] == 0].shape[0]\ncant_elim = cant_target0 - cant_target1\nratio = 70/30\n\nclientes_unicos = completo.drop_duplicates(subset=['customer_ID'])[['customer_ID', 'target']]\nsample_clientes_unicos_pagos = clientes_unicos[clientes_unicos['target']==0].sample(n=math.ceil(cant_clientes_morosos * ratio), random_state=3)\n\nsample_clientes_pagos = completo[completo['customer_ID'].isin(sample_clientes_unicos_pagos['customer_ID'].values)]\nclientes_morosos = completo[completo['target']==1]\n\ncompleto_balanceado = pd.concat([sample_clientes_pagos, clientes_morosos])\n\ncant_quedan_clientes = len(completo_balanceado['customer_ID'].unique())\ncant_quedan = completo_balanceado.shape[0]\n\nprint(f\"\\nEl dataset tiene {cant_total} registros y {cant_clientes} clientes, de los cuales \" + \\\n      f\"{cant_target1} registros corresponden a {cant_clientes_morosos} clientes target 1 y \" + \\\n      f\"{cant_target0} registros corresponden a {cant_clientes_pagos} clientes target 0\")\nprint(f\"\\nEl dataset balanceado quedaría con {math.ceil(cant_clientes_morosos * ratio)} (cantidad de clientes target 0) + {cant_clientes_morosos} \" + \\\n      f\"(cantidad de clientes target 1) = {cant_quedan_clientes} clientes \" + \\\n      f\"y {cant_quedan} registros\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:25.853663Z","iopub.status.busy":"2022-08-04T00:28:25.852749Z","iopub.status.idle":"2022-08-04T00:28:27.099612Z","shell.execute_reply":"2022-08-04T00:28:27.098663Z"},"id":"4zwGMcL6_aIO","outputId":"e678e226-88d5-4340-cfbd-1adcda28992f","papermill":{"duration":1.453789,"end_time":"2022-08-04T00:28:27.102407","exception":false,"start_time":"2022-08-04T00:28:25.648618","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"completo_balanceado","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:27.517231Z","iopub.status.busy":"2022-08-04T00:28:27.516478Z","iopub.status.idle":"2022-08-04T00:28:27.571811Z","shell.execute_reply":"2022-08-04T00:28:27.570832Z"},"id":"soaLQdSf_aIP","outputId":"fcb8d834-b0d9-40f1-e0b8-dc42c9663a51","papermill":{"duration":0.265076,"end_time":"2022-08-04T00:28:27.574268","exception":false,"start_time":"2022-08-04T00:28:27.309192","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"colores=[\"#75201a\",\"#194373\"]\ncompleto_balanceado['target'].value_counts().plot.bar(alpha=0.9, color=colores)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:28.00419Z","iopub.status.busy":"2022-08-04T00:28:28.003735Z","iopub.status.idle":"2022-08-04T00:28:28.194917Z","shell.execute_reply":"2022-08-04T00:28:28.193881Z"},"id":"Zwj6jbTC_aIP","outputId":"29ca6500-f74f-45dc-911a-a53ef6ad51ae","papermill":{"duration":0.397993,"end_time":"2022-08-04T00:28:28.197316","exception":false,"start_time":"2022-08-04T00:28:27.799323","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Definicion de metricas y prediccion por cliente\n\nComo podemos observar existe mas de un resumen de tarjeta de credito por cliente en nuestro dataset. Ademas, como solo consideramos un 5% del mismo, no tenemos los 18 resumenes de cada cliente, por lo que algunos van a tener mas resumenes que otros, pero con un mismo target por cliente.\n\nPara nuestro analisis vamos a definir metricas de precision, recall y F1 agrupadas por cliente, en donde cada resumen se predice de forma individual, para luego quedarnos con la media de cada cliente.\n\nTambien debemos agregar un split del dataset por cliente para mantener una proporcion de clentes conocida y tener todos los resumenes de un cliente en el mismo split (ya sea para entrenar el modelo o para predecir)\n\nPor ultimo vamos a agregar variables que puedan relacionar resumenes de un mismo cliente. Para ello vamos a entrenar un clasificador que nos pueda indicar cuales fueron las variables mas influyentes en su prediccion y luego nos quedamos con una media por cliente de esas variables y las usamos para predecir con el resto de modelos.","metadata":{"papermill":{"duration":0.212012,"end_time":"2022-08-04T00:28:28.612574","exception":false,"start_time":"2022-08-04T00:28:28.400562","status":"completed"},"tags":[]}},{"cell_type":"code","source":"all_metrics = {}\nmetrics_key_order = []\n\ndef save_metric(name, acc, rec, f1):\n    metrics_key_order.append(name)\n    all_metrics[name] = {\n        \"accuracy\": acc,\n        \"recall\": rec,\n        \"f1\": f1\n    }\n    \ndef get_all_metrics():\n    return all_metrics\n\ndef get_metrics_key_order():\n    return metrics_key_order","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:29.040048Z","iopub.status.busy":"2022-08-04T00:28:29.039433Z","iopub.status.idle":"2022-08-04T00:28:29.046314Z","shell.execute_reply":"2022-08-04T00:28:29.045235Z"},"papermill":{"duration":0.225185,"end_time":"2022-08-04T00:28:29.048584","exception":false,"start_time":"2022-08-04T00:28:28.823399","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from functools import reduce\n\ndef custom_predict(serie):\n    return 0 if ((reduce(lambda x, y: x + y, serie)) / len(serie)) < 0.5 else 1\n\ndef split_por_cliente(balanceado, desbalanceado, porcentaje):\n    balanceado_customer_target = balanceado[['customer_ID', 'target']]\n    balanceado_clientes_unicos = balanceado.drop_duplicates(subset=['customer_ID'])[['customer_ID', 'target']]\n    morosos_unicos = balanceado_customer_target[balanceado_customer_target[\"target\"]==1][\"customer_ID\"].unique()\n    pagos_unicos = balanceado_customer_target[balanceado_customer_target[\"target\"]==0][\"customer_ID\"].unique()\n    cantidad_morosos = math.ceil(len(morosos_unicos) * porcentaje)\n    cantidad_pagos = math.ceil(len(pagos_unicos) * porcentaje)\n    morosos_unicos_porcentaje = balanceado_clientes_unicos[balanceado_clientes_unicos['target']==1].sample(n=cantidad_morosos, random_state=3)\n    pagos_unicos_porcentaje = balanceado_clientes_unicos[balanceado_clientes_unicos['target']==0].sample(n=cantidad_pagos, random_state=3)\n    train_morosos = balanceado[balanceado['customer_ID'].isin(morosos_unicos_porcentaje['customer_ID'].values)]\n    train_pagos = balanceado[balanceado['customer_ID'].isin(pagos_unicos_porcentaje['customer_ID'].values)]\n    train_final = pd.concat([train_morosos, train_pagos])\n    test_final = desbalanceado[~desbalanceado['customer_ID'].isin(train_final['customer_ID'].values)]\n    y_train_final = train_final['target']\n    y_test_final = test_final['target']\n    return train_final.drop(['target'], axis='columns', inplace=False), test_final.drop(['target'], axis='columns', inplace=False), y_train_final, y_test_final\n\ndef get_metrics(real, predict):\n    acc = accuracy_score(real, predict)\n    recall = recall_score(real, predict)\n    f1 = f1_score(real, predict, average='macro')\n    \n    return acc, recall, f1\n\n    \ndef metrics_by_client(name, real, predict):\n    test_by_customer_predict = x_test_copy.copy()\n    test_by_customer_predict['target'] = predict\n    test_by_customer_predict = test_by_customer_predict[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})        \n\n    test_by_customer_real = x_test_copy.copy()\n    test_by_customer_real['target'] = real\n    test_by_customer_real = test_by_customer_real[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})\n    \n    acc, recall, f1 = get_metrics(test_by_customer_real['target'].values, test_by_customer_predict['target'].values)\n    \n    save_metric(name, acc, recall, f1)\n    \n    return acc, recall, f1\n\ndef confusion_matrix_by_client(real, predict):\n    test_by_customer_predict = x_test_copy.copy()\n    test_by_customer_predict['target'] = predict\n    test_by_customer_predict = test_by_customer_predict[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})\n    \n    test_by_customer_real = x_test_copy.copy()\n    test_by_customer_real['target'] = real\n    test_by_customer_real = test_by_customer_real[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})\n    \n    return confusion_matrix(test_by_customer_real['target'].values, test_by_customer_predict['target'].values)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:29.515772Z","iopub.status.busy":"2022-08-04T00:28:29.515024Z","iopub.status.idle":"2022-08-04T00:28:29.534273Z","shell.execute_reply":"2022-08-04T00:28:29.533328Z"},"papermill":{"duration":0.223967,"end_time":"2022-08-04T00:28:29.53659","exception":false,"start_time":"2022-08-04T00:28:29.312623","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Split de conjunto de pruebas y entrenamiento**\n\nA continuacion creamos los sets de entrenamiento y prueba. En este caso usaremos un 40% del dataset balanceado, por lo que se obtiene un dataset con el 40% de clientes target 1 del dataset original reducido, y el resto son clientes target 0 que completan una la misma proporcion que el dataset _completo_balanceado_.\n\nEl set de pruebas contiene el resto de clientes que no fueron utilizados para entrenar, por lo que vamos a poder analizar cada modelo con la totalidad del dataset original.","metadata":{"papermill":{"duration":0.20061,"end_time":"2022-08-04T00:28:29.939125","exception":false,"start_time":"2022-08-04T00:28:29.738515","status":"completed"},"tags":[]}},{"cell_type":"code","source":"x_train, x_test, y_train, y_test = split_por_cliente(completo_balanceado, completo, 0.4)\n\nx_train_copy = x_train.copy()\nx_test_copy = x_test.copy()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:30.350804Z","iopub.status.busy":"2022-08-04T00:28:30.349644Z","iopub.status.idle":"2022-08-04T00:28:31.329476Z","shell.execute_reply":"2022-08-04T00:28:31.328497Z"},"papermill":{"duration":1.187877,"end_time":"2022-08-04T00:28:31.331966","exception":false,"start_time":"2022-08-04T00:28:30.144089","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unicos_train = len(x_train['customer_ID'].unique())\nx_train_target = x_train\nx_train_target['target'] = y_train\nunicos_morosos_train = len(x_train_target[x_train_target['target']==1]['customer_ID'].unique())\nunicos_pagos_train = len(x_train_target[x_train_target['target']==0]['customer_ID'].unique())\nregistros_train = x_train.shape[0]\nregistros_test = x_test.shape[0]\nregistros_balanceado = completo_balanceado.shape[0]\nregistros_completo = completo.shape[0]\n\nprint(f\"\\nLa cantidad de clientes unicos en train es: {unicos_train}\")\nprint(f\"La cantidad de clientes morosos unicos en train es: {unicos_morosos_train}, (Porcentaje del total: {round((unicos_morosos_train / unicos_train) * 100, 2)}%)\")\nprint(f\"La cantidad de clientes pagos unicos en train es: {unicos_pagos_train}, (Porcentaje del total: {round((unicos_pagos_train / unicos_train) * 100, 2)}%)\")\nprint(f\"La cantidad de registros en train es: {registros_train}, (Porcentaje respecto de balanceado: {round((registros_train / registros_balanceado) * 100, 2)}%, porcentaje respecto del total: {round((registros_train / registros_completo) * 100, 2)}%\")\nprint(f\"La cantidad de registros en test es: {registros_test}, (Porcentaje del total: {round((registros_test / registros_completo) * 100, 2)}%)\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:31.74044Z","iopub.status.busy":"2022-08-04T00:28:31.739411Z","iopub.status.idle":"2022-08-04T00:28:31.871182Z","shell.execute_reply":"2022-08-04T00:28:31.870164Z"},"papermill":{"duration":0.339213,"end_time":"2022-08-04T00:28:31.874003","exception":false,"start_time":"2022-08-04T00:28:31.53479","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Analisis de variables mas influyentes**\n\nEntrenamos el clasificador que nos va a indicar que variables fueron las que tuvieron mas influencia durante la prediccion, en este caso usaremos Random Forest","metadata":{"papermill":{"duration":0.206036,"end_time":"2022-08-04T00:28:32.285094","exception":false,"start_time":"2022-08-04T00:28:32.079058","status":"completed"},"tags":[]}},{"cell_type":"code","source":"model = RandomForestClassifier()\nx_train_sin_customer = x_train.drop(['customer_ID', 'target'], axis='columns', inplace=False)\nmodel.fit(x_train_sin_customer,y_train)\nfeat_importances = pd.Series(model.feature_importances_, index=x_train_sin_customer.columns)\nfeat_importances.nlargest(10).plot(kind='barh')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:28:32.692054Z","iopub.status.busy":"2022-08-04T00:28:32.691568Z","iopub.status.idle":"2022-08-04T00:30:48.485518Z","shell.execute_reply":"2022-08-04T00:30:48.484704Z"},"papermill":{"duration":136.202514,"end_time":"2022-08-04T00:30:48.691101","exception":false,"start_time":"2022-08-04T00:28:32.488587","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prediccion con el conjunto de prueba balanceado","metadata":{"papermill":{"duration":0.205617,"end_time":"2022-08-04T00:30:49.101044","exception":false,"start_time":"2022-08-04T00:30:48.895427","status":"completed"},"tags":[]}},{"cell_type":"code","source":"x_test_sin_customer = x_test.drop(['customer_ID'], axis='columns', inplace=False)\n\npredicted_categories = model.predict(x_test_sin_customer)\n\nbf_acc, bf_recall, bf_f1 = metrics_by_client(\"Random Forest cust_id feature importance\", y_test, predicted_categories)\n\nprint(\"La precision es {}\".format(bf_acc))\n\nprint(\"El recall es {}\".format(bf_recall))\n\nprint(\"F1 es {}\".format(bf_f1))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:30:49.671211Z","iopub.status.busy":"2022-08-04T00:30:49.670156Z","iopub.status.idle":"2022-08-04T00:30:58.901378Z","shell.execute_reply":"2022-08-04T00:30:58.900077Z"},"papermill":{"duration":9.463852,"end_time":"2022-08-04T00:30:58.903579","exception":false,"start_time":"2022-08-04T00:30:49.439727","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observamos la matriz de confusion para tener mas detalle sobre la prediccion de clientes deudores y no deudores.","metadata":{"papermill":{"duration":0.206655,"end_time":"2022-08-04T00:30:59.318419","exception":false,"start_time":"2022-08-04T00:30:59.111764","status":"completed"},"tags":[]}},{"cell_type":"code","source":"balanceado_important_columns_cm = confusion_matrix_by_client(y_test, predicted_categories)\nsns.heatmap(balanceado_important_columns_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:30:59.730598Z","iopub.status.busy":"2022-08-04T00:30:59.729986Z","iopub.status.idle":"2022-08-04T00:31:02.579136Z","shell.execute_reply":"2022-08-04T00:31:02.578212Z"},"papermill":{"duration":3.057824,"end_time":"2022-08-04T00:31:02.581511","exception":false,"start_time":"2022-08-04T00:30:59.523687","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nos quedamos con las siguientes variables, que fueron las que mas influyeron con diferencia respecto del resto.","metadata":{"papermill":{"duration":0.202103,"end_time":"2022-08-04T00:31:02.987479","exception":false,"start_time":"2022-08-04T00:31:02.785376","status":"completed"},"tags":[]}},{"cell_type":"code","source":"columnas_importantes = [\"B_PCA_0\",\"P_2\"]","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:31:03.393555Z","iopub.status.busy":"2022-08-04T00:31:03.393121Z","iopub.status.idle":"2022-08-04T00:31:03.397924Z","shell.execute_reply":"2022-08-04T00:31:03.396736Z"},"papermill":{"duration":0.210668,"end_time":"2022-08-04T00:31:03.399975","exception":false,"start_time":"2022-08-04T00:31:03.189307","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Para las columnas con mayor importancia, agregamos una nueva feature por columna con la media agrupada por cliente","metadata":{"papermill":{"duration":0.202457,"end_time":"2022-08-04T00:31:03.804255","exception":false,"start_time":"2022-08-04T00:31:03.601798","status":"completed"},"tags":[]}},{"cell_type":"code","source":"completo_medias = completo.copy()\ncompleto_balanceado_medias = completo_balanceado.copy()\n\nfor column in columnas_importantes:\n    completo_medias[column + \"_mean\"] = (completo_medias.groupby(\"customer_ID\")[column].transform('mean'))\n    completo_balanceado_medias[column + \"_mean\"] = (completo_balanceado_medias.groupby(\"customer_ID\")[column].transform('mean'))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:31:04.212584Z","iopub.status.busy":"2022-08-04T00:31:04.212198Z","iopub.status.idle":"2022-08-04T00:31:05.252994Z","shell.execute_reply":"2022-08-04T00:31:05.251978Z"},"papermill":{"duration":1.24697,"end_time":"2022-08-04T00:31:05.255518","exception":false,"start_time":"2022-08-04T00:31:04.008548","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"completo_medias","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:31:05.664615Z","iopub.status.busy":"2022-08-04T00:31:05.664072Z","iopub.status.idle":"2022-08-04T00:31:05.748265Z","shell.execute_reply":"2022-08-04T00:31:05.747226Z"},"papermill":{"duration":0.292218,"end_time":"2022-08-04T00:31:05.750758","exception":false,"start_time":"2022-08-04T00:31:05.45854","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"completo_balanceado_medias","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:31:06.219325Z","iopub.status.busy":"2022-08-04T00:31:06.218855Z","iopub.status.idle":"2022-08-04T00:31:06.273649Z","shell.execute_reply":"2022-08-04T00:31:06.272735Z"},"papermill":{"duration":0.32219,"end_time":"2022-08-04T00:31:06.276175","exception":false,"start_time":"2022-08-04T00:31:05.953985","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Dividimos al dataset con las columnas nuevas y creamos dos dataset para testear cada modelo entrenado. En uno guardamos el conjunto de test agrupado por cliente y el target real de cada uno, en el otro vamos a guardar una copia del conjunto de pruebas sin la columna de target, para completarla luego con las predicciones y tomar la media del target predicho por cliente (si la media es menor a 0.5 se considera target = 0, si no se considera target = 1)","metadata":{"id":"bIK96unT_aIQ","papermill":{"duration":0.205357,"end_time":"2022-08-04T00:31:06.683986","exception":false,"start_time":"2022-08-04T00:31:06.478629","status":"completed"},"tags":[]}},{"cell_type":"code","source":"x_train, x_test, y_train, y_test = split_por_cliente(completo_balanceado_medias, completo_medias, 0.4)\n\nx_train_copy = x_train.copy()\nx_test_copy = x_test.copy()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:31:07.102773Z","iopub.status.busy":"2022-08-04T00:31:07.102157Z","iopub.status.idle":"2022-08-04T00:31:08.214618Z","shell.execute_reply":"2022-08-04T00:31:08.213532Z"},"papermill":{"duration":1.321332,"end_time":"2022-08-04T00:31:08.217085","exception":false,"start_time":"2022-08-04T00:31:06.895753","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Para hacer las pruebas de cada modelo, vamos a considerar en nuestro conjunto de test a todos los registros que no consideramos en el set de entrenamiento, y vamos a entrenar cada modelo con el set balanceado a un ratio de 60-40 (target 0 (%) - target 1 (%))","metadata":{"papermill":{"duration":0.204849,"end_time":"2022-08-04T00:31:08.62781","exception":false,"start_time":"2022-08-04T00:31:08.422961","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Vamos a realizar una transformación en el dataset de forma que los datos tengan la forma de una desviación estandar.","metadata":{"id":"pMdwtRDx_aIQ","papermill":{"duration":0.206714,"end_time":"2022-08-04T00:31:09.040778","exception":false,"start_time":"2022-08-04T00:31:08.834064","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#Armo una version estandarizada\nstand_scaler = preprocessing.StandardScaler()\nx_train = stand_scaler.fit_transform(x_train.drop(['customer_ID'], axis='columns', inplace=False))\nx_test = stand_scaler.fit_transform(x_test.drop(['customer_ID'], axis='columns', inplace=False))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:31:09.461442Z","iopub.status.busy":"2022-08-04T00:31:09.460651Z","iopub.status.idle":"2022-08-04T00:31:10.032997Z","shell.execute_reply":"2022-08-04T00:31:10.031963Z"},"id":"8Ts_6mfD_aIR","papermill":{"duration":0.787118,"end_time":"2022-08-04T00:31:10.035567","exception":false,"start_time":"2022-08-04T00:31:09.248449","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generación y evaluación de modelos\n\nVamos a buscar los mejores hiperparámetros con KFOLD CV Random Search, usaremos sólo 2 folds ya que el dataset es muy grande y puede demorar demasiado tiempo.","metadata":{"id":"CG5Ctzxt_aIR","papermill":{"duration":0.208048,"end_time":"2022-08-04T00:31:10.45854","exception":false,"start_time":"2022-08-04T00:31:10.250492","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Random Forest\n\nBuscamos los mejores hiperparámetros.","metadata":{"id":"jx7VT60G_aIS","papermill":{"duration":0.203819,"end_time":"2022-08-04T00:31:10.864595","exception":false,"start_time":"2022-08-04T00:31:10.660776","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n##KFOLD CV Random Search para buscar el mejor arbol (los mejores atributos, hiperparametros,etc)\nfrom sklearn.model_selection import StratifiedKFold, KFold,RandomizedSearchCV\nfrom sklearn.metrics import make_scorer\n\nn=10\n\n#Conjunto de parámetros que quiero usar\nparams_grid = {'criterion':['gini','entropy'],\n               'bootstrap': [True, False],\n               'n_estimators': [75, 100, 150],\n               'min_samples_leaf':[1,3,5],\n               'max_depth':[3,5,8,10,15]}\n                \n#Cantidad de splits para el Cross Validation\nfolds=2\n\n#Kfold estratificado\nkfoldcv = StratifiedKFold(n_splits=folds)\n\n#Clasificador\nbase_tree = RandomForestClassifier(n_jobs=-1) \n\n#Metrica que quiero optimizar F1 Score\nscorer_fn = make_scorer(sk.metrics.f1_score)\n\n#Random Search Cross Validation\nrandomcv = RandomizedSearchCV(estimator=base_tree,\n                              param_distributions=params_grid,\n                              scoring=scorer_fn,\n                              cv=kfoldcv,\n                              n_iter=n,\n                              n_jobs=-1,\n                              random_state=420\n                             ) \n\n#Busco los hiperparamtros que optimizan F1 Score\nrandomcv.fit(x_train,y_train);\n\n#Mejores hiperparametros del arbol\nprint(f\"Best params: {randomcv.best_params_}\")\n#Mejor métrica\nprint(f\"Best score: {randomcv.best_score_}\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:31:11.357864Z","iopub.status.busy":"2022-08-04T00:31:11.357405Z","iopub.status.idle":"2022-08-04T00:37:05.930369Z","shell.execute_reply":"2022-08-04T00:37:05.929026Z"},"id":"H8UBWxmD_aIS","papermill":{"duration":355.083355,"end_time":"2022-08-04T00:37:06.151074","exception":false,"start_time":"2022-08-04T00:31:11.067719","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Entrenamos el clasificador con pesos que permitan ajustar mejor el target:","metadata":{"id":"ctP1AgHM_aIS","papermill":{"duration":0.22159,"end_time":"2022-08-04T00:37:06.590433","exception":false,"start_time":"2022-08-04T00:37:06.368843","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n#Creo el árbol\narbol = RandomForestClassifier(n_jobs=-1,\n                               random_state=420\n                              ).set_params(**randomcv.best_params_)\n\nn=10\n\n#Entreno el arbol en todo el set\narbol.fit(x_train,y_train)\n\npredicted_categories1 = arbol.predict(x_test)\n\nrf_acc1, rf_recall1, rf_f11 = metrics_by_client(\"Random Forest\", y_test, predicted_categories1)\n\nprint(\"La precision es {}\".format(rf_acc1))\n\nprint(\"El recall es {}\".format(rf_recall1))\n\nprint(\"F1 es {}\".format(rf_f11))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:37:07.041857Z","iopub.status.busy":"2022-08-04T00:37:07.040606Z","iopub.status.idle":"2022-08-04T00:38:04.545469Z","shell.execute_reply":"2022-08-04T00:38:04.544435Z"},"id":"LhpE1IHx_aIS","outputId":"89b640ee-0a2b-41cd-ff88-ffff0f99ec01","papermill":{"duration":57.940936,"end_time":"2022-08-04T00:38:04.752541","exception":false,"start_time":"2022-08-04T00:37:06.811605","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Observamos que las columnas que agregamos se estan teniendo en cuenta por los arboles de clasificacion.","metadata":{"papermill":{"duration":0.207805,"end_time":"2022-08-04T00:38:05.166583","exception":false,"start_time":"2022-08-04T00:38:04.958778","status":"completed"},"tags":[]}},{"cell_type":"code","source":"feat_importances_rfc = pd.Series(arbol.feature_importances_, index=x_train_copy.drop(['customer_ID'], axis='columns', inplace=False).columns)\nfeat_importances_rfc.nlargest(10).plot(kind='barh')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:38:05.575291Z","iopub.status.busy":"2022-08-04T00:38:05.574605Z","iopub.status.idle":"2022-08-04T00:38:05.931691Z","shell.execute_reply":"2022-08-04T00:38:05.930703Z"},"papermill":{"duration":0.564319,"end_time":"2022-08-04T00:38:05.933988","exception":false,"start_time":"2022-08-04T00:38:05.369669","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Luego la prediccion por customer ","metadata":{"papermill":{"duration":0.203874,"end_time":"2022-08-04T00:38:06.351552","exception":false,"start_time":"2022-08-04T00:38:06.147678","status":"completed"},"tags":[]}},{"cell_type":"code","source":"rfc_cm = confusion_matrix_by_client(y_test, predicted_categories1)\nsns.heatmap(rfc_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:38:06.76459Z","iopub.status.busy":"2022-08-04T00:38:06.764162Z","iopub.status.idle":"2022-08-04T00:38:09.664056Z","shell.execute_reply":"2022-08-04T00:38:09.662993Z"},"papermill":{"duration":3.109577,"end_time":"2022-08-04T00:38:09.666547","exception":false,"start_time":"2022-08-04T00:38:06.55697","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### XGBoost","metadata":{"id":"BcIwyg4Z_aIS","papermill":{"duration":0.282257,"end_time":"2022-08-04T00:38:10.157942","exception":false,"start_time":"2022-08-04T00:38:09.875685","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Como primer approach vemos cómo se comporta XGBoost sin tunear sus hiperparámetros","metadata":{"papermill":{"duration":0.205323,"end_time":"2022-08-04T00:38:10.575613","exception":false,"start_time":"2022-08-04T00:38:10.37029","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nxgb_model_default = xgb.XGBClassifier(random_state=420)\n\nxgb_model_default.fit(x_train, y_train)\n\npredicted_categories_xgb_def = xgb_model_default.predict(x_test)\n\nxgb_acc, xgb_recall, xgb_f1 = metrics_by_client(\"XGBoost default\", y_test, predicted_categories_xgb_def)\n\nprint(\"La precision es {}\".format(xgb_acc))\n\nprint(\"El recall es {}\".format(xgb_recall))\n\nprint(\"F1 es {}\".format(xgb_f1))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:38:10.992671Z","iopub.status.busy":"2022-08-04T00:38:10.992002Z","iopub.status.idle":"2022-08-04T00:39:41.654506Z","shell.execute_reply":"2022-08-04T00:39:41.653641Z"},"papermill":{"duration":91.076571,"end_time":"2022-08-04T00:39:41.858704","exception":false,"start_time":"2022-08-04T00:38:10.782133","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xgb_cm = confusion_matrix_by_client(y_test, predicted_categories_xgb_def)\nsns.heatmap(xgb_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:39:42.275339Z","iopub.status.busy":"2022-08-04T00:39:42.27451Z","iopub.status.idle":"2022-08-04T00:39:45.060466Z","shell.execute_reply":"2022-08-04T00:39:45.059499Z"},"papermill":{"duration":2.997449,"end_time":"2022-08-04T00:39:45.062695","exception":false,"start_time":"2022-08-04T00:39:42.065246","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Buscamos los mejores hiperparámetros","metadata":{"id":"-_dLgnJ__aIS","papermill":{"duration":0.204544,"end_time":"2022-08-04T00:39:45.472337","exception":false,"start_time":"2022-08-04T00:39:45.267793","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n##KFOLD CV Random Search para buscar el mejor clasificador (los mejores atributos, hiperparametros,etc)\n\nn=10\n\n#Conjunto de parámetros que quiero usar\nparams_grid = {'n_estimators':[50,75,100],\n               'learning_rate': [0.3, 0.1, 0.01],\n               'max_depth':list(range(3,7)),\n               'subsample': [0.6, 1],\n               'colsample_bytree': [0.6, 1],\n               'eval_metric':list([\"auc\", \"error\"])\n              }\n                \n#Cantidad de splits para el Cross Validation\nfolds=2\n\n#Kfold estratificado\nkfoldcv = StratifiedKFold(n_splits=folds)\n\n#Clasificador\nbase_model = xgb.XGBClassifier() \n\n#Metrica que quiero optimizar F1 Score\nscorer_fn = make_scorer(sk.metrics.f1_score)\n\n#Random Search Cross Validation\nrandomcv_xgb = RandomizedSearchCV(estimator=base_model,\n                              param_distributions=params_grid,\n                              scoring=scorer_fn,\n                              cv=kfoldcv,\n                              n_iter=n,\n                              n_jobs=-1,\n                              random_state=420) \n\n#Busco los hiperparamtros que optimizan F1 Score\nrandomcv_xgb.fit(x_train,y_train);\n\n#Mejores hiperparametros del arbol\nprint(f\"Best params: {randomcv_xgb.best_params_}\")\n#Mejor métrica\nprint(f\"Best scope: {randomcv_xgb.best_score_}\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:39:45.882752Z","iopub.status.busy":"2022-08-04T00:39:45.882082Z","iopub.status.idle":"2022-08-04T00:45:17.041029Z","shell.execute_reply":"2022-08-04T00:45:17.039846Z"},"id":"c9DUC5DY_aIS","papermill":{"duration":331.582791,"end_time":"2022-08-04T00:45:17.258224","exception":false,"start_time":"2022-08-04T00:39:45.675433","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Creamos el modelo con los hiperparámetros calculados","metadata":{"id":"T_p6BqZ0_aIT","papermill":{"duration":0.205694,"end_time":"2022-08-04T00:45:17.733637","exception":false,"start_time":"2022-08-04T00:45:17.527943","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nxgb_model = xgb.XGBClassifier(random_state=420).set_params(**randomcv_xgb.best_params_)\n\nxgb_model.fit(x_train, y_train)\n\npredicted_categories2 = xgb_model.predict(x_test)\n\nxgb_acc, xgb_recall, xgb_f1 = metrics_by_client(\"XGBoost\", y_test, predicted_categories2)\n\nprint(\"La precision es {}\".format(xgb_acc))\n\nprint(\"El recall es {}\".format(xgb_recall))\n\nprint(\"F1 es {}\".format(xgb_f1))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:45:18.15224Z","iopub.status.busy":"2022-08-04T00:45:18.151283Z","iopub.status.idle":"2022-08-04T00:46:17.657961Z","shell.execute_reply":"2022-08-04T00:46:17.656961Z"},"id":"eUwIxpHa_aIT","outputId":"e2cd1cb0-ed6e-462a-87d8-74215d01dfd4","papermill":{"duration":59.930439,"end_time":"2022-08-04T00:46:17.870142","exception":false,"start_time":"2022-08-04T00:45:17.939703","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_features = x_train_copy.drop(['customer_ID'], axis='columns', inplace=False).columns\nfeature_names = [train_features[int(x[1:])] for x in xgb_model.get_booster().get_score(importance_type='gain').keys()]\nfeat_importances_xgb = pd.Series(list(xgb_model.get_booster().get_score(importance_type='gain').values()), index=feature_names)\nfeat_importances_xgb.nlargest(10).plot(kind='barh')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:46:18.288124Z","iopub.status.busy":"2022-08-04T00:46:18.287027Z","iopub.status.idle":"2022-08-04T00:46:18.534192Z","shell.execute_reply":"2022-08-04T00:46:18.53324Z"},"papermill":{"duration":0.45966,"end_time":"2022-08-04T00:46:18.536554","exception":false,"start_time":"2022-08-04T00:46:18.076894","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xgb_cm = confusion_matrix_by_client(y_test, predicted_categories2)\nsns.heatmap(xgb_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:46:18.954997Z","iopub.status.busy":"2022-08-04T00:46:18.953964Z","iopub.status.idle":"2022-08-04T00:46:21.774885Z","shell.execute_reply":"2022-08-04T00:46:21.773962Z"},"papermill":{"duration":3.033452,"end_time":"2022-08-04T00:46:21.777008","exception":false,"start_time":"2022-08-04T00:46:18.743556","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### SVM","metadata":{"id":"bFYhb2d8_aIT","papermill":{"duration":0.209132,"end_time":"2022-08-04T00:46:22.195617","exception":false,"start_time":"2022-08-04T00:46:21.986485","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Primero probamos el modelo sin tuneo de hiperparámetros","metadata":{"papermill":{"duration":0.205788,"end_time":"2022-08-04T00:46:22.60912","exception":false,"start_time":"2022-08-04T00:46:22.403332","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n# Usamos svm.LinearSVC ya que está hecho para datasets grandes\nsvm_model = svm.LinearSVC(random_state=420)\n\nsvm_model.fit(x_train, y_train)\n\npredicted_categories3_def = svm_model.predict(x_test)\n\nsvm_acc_def, svm_recall_def, svm_f1_def = metrics_by_client(\"LinearSVC default\", y_test, predicted_categories3_def)\n\nprint(\"La precision es {}\".format(svm_acc_def))\n\nprint(\"El recall es {}\".format(svm_recall_def))\n\nprint(\"F1 es {}\".format(svm_f1_def))\n\nsvm_cm = confusion_matrix_by_client(y_test, predicted_categories3_def)\nsns.heatmap(svm_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:46:23.089193Z","iopub.status.busy":"2022-08-04T00:46:23.088347Z","iopub.status.idle":"2022-08-04T00:47:51.390405Z","shell.execute_reply":"2022-08-04T00:47:51.389469Z"},"papermill":{"duration":88.575415,"end_time":"2022-08-04T00:47:51.394843","exception":false,"start_time":"2022-08-04T00:46:22.819428","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Buscamos los mejores hiperparámetros.","metadata":{"id":"vuY3j65k_aIT","papermill":{"duration":0.20585,"end_time":"2022-08-04T00:47:51.810637","exception":false,"start_time":"2022-08-04T00:47:51.604787","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n##KFOLD CV Random Search para buscar el mejor clasificador (los mejores atributos, hiperparametros,etc)\nn=10\n#Conjunto de parámetros que quiero usar\nparams_grid = {'fit_intercept': list([True, False]),\n               'dual': list([True, False]),\n               'C': [0.1, 1, 10, 100],\n               'tol': [0.1, 0.01, 0.001, 0.0001]\n              }\n                \n#Cantidad de splits para el Cross Validation\nfolds=2\n\n#Kfold estratificado\nkfoldcv = StratifiedKFold(n_splits=folds)\n\n#Clasificador\nbase_model = svm.LinearSVC() \n\n#Metrica que quiero optimizar F1 Score\nscorer_fn = make_scorer(sk.metrics.f1_score)\n\n#Random Search Cross Validation\nrandomcv_svm = RandomizedSearchCV(estimator=base_model,\n                              param_distributions=params_grid,\n                              scoring=scorer_fn,\n                              cv=kfoldcv,\n                              n_iter=n,\n                              n_jobs=-1,\n                              random_state=420) \n\n#Busco los hiperparamtros que optimizan F1 Score\nrandomcv_svm.fit(x_train,y_train);\n\n#Mejores hiperparametros del arbol\nprint(f\"Best Params: {randomcv_svm.best_params_}\")\n#Mejor métrica\nprint(f\"Best score: {randomcv_svm.best_score_}\")","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:47:52.248038Z","iopub.status.busy":"2022-08-04T00:47:52.246573Z","iopub.status.idle":"2022-08-04T00:52:24.91905Z","shell.execute_reply":"2022-08-04T00:52:24.917813Z"},"id":"GZzWOZss_aIT","papermill":{"duration":273.113052,"end_time":"2022-08-04T00:52:25.135369","exception":false,"start_time":"2022-08-04T00:47:52.022317","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Creamos el modelo.","metadata":{"id":"5RFa4m5F_aIT","papermill":{"duration":0.209507,"end_time":"2022-08-04T00:52:25.55317","exception":false,"start_time":"2022-08-04T00:52:25.343663","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n# Usamos svm.LinearSVC ya que está hecho para datasets grandes\nsvm_model = svm.LinearSVC(random_state=420).set_params(**randomcv_svm.best_params_)\n\nsvm_model.fit(x_train, y_train)\n\npredicted_categories3 = svm_model.predict(x_test)\n\nsvm_acc, svm_recall, svm_f1 = metrics_by_client(\"LinearSVC tuneado\", y_test, predicted_categories3)\n\nprint(\"La precision es {}\".format(svm_acc))\n\nprint(\"El recall es {}\".format(svm_recall))\n\nprint(\"F1 es {}\".format(svm_f1))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:52:25.976654Z","iopub.status.busy":"2022-08-04T00:52:25.97596Z","iopub.status.idle":"2022-08-04T00:53:59.888961Z","shell.execute_reply":"2022-08-04T00:53:59.887777Z"},"id":"gqh0xnLm_aIT","outputId":"79141175-409b-405a-f473-743ee7fbc663","papermill":{"duration":94.35515,"end_time":"2022-08-04T00:54:00.119484","exception":false,"start_time":"2022-08-04T00:52:25.764334","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.inspection import permutation_importance\n\nperm_importance = permutation_importance(svm_model, x_test, y_test)\n\nfeature_names = x_train_copy.drop(['customer_ID'], axis='columns', inplace=False).columns\nfeatures = np.array(feature_names)\n\nsorted_idx = perm_importance.importances_mean.argsort()[::-1][:10]\nplt.barh(features[sorted_idx], perm_importance.importances_mean[sorted_idx])\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:54:00.610116Z","iopub.status.busy":"2022-08-04T00:54:00.609485Z","iopub.status.idle":"2022-08-04T00:54:54.595301Z","shell.execute_reply":"2022-08-04T00:54:54.594537Z"},"papermill":{"duration":54.419714,"end_time":"2022-08-04T00:54:54.815255","exception":false,"start_time":"2022-08-04T00:54:00.395541","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"svm_cm = confusion_matrix_by_client(y_test, predicted_categories3)\nsns.heatmap(svm_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:54:55.236171Z","iopub.status.busy":"2022-08-04T00:54:55.23509Z","iopub.status.idle":"2022-08-04T00:54:57.980049Z","shell.execute_reply":"2022-08-04T00:54:57.979227Z"},"papermill":{"duration":2.956801,"end_time":"2022-08-04T00:54:57.982166","exception":false,"start_time":"2022-08-04T00:54:55.025365","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ensamble","metadata":{"id":"Frh5Q4rQ_aIT","papermill":{"duration":0.206016,"end_time":"2022-08-04T00:54:58.398623","exception":false,"start_time":"2022-08-04T00:54:58.192607","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Realizamos un ensamble de tipo VotingClassifier.","metadata":{"id":"hBivUIl5_aIU","papermill":{"duration":0.208676,"end_time":"2022-08-04T00:54:58.816699","exception":false,"start_time":"2022-08-04T00:54:58.608023","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n#Creo ensemble de Votación\nvot_clf = VotingClassifier(estimators = [('RF', arbol), ('xGBoost', xgb_model), ('SVM', svm_model)], voting = 'hard')\n\n#Entreno el ensamble\nvot_clf.fit(x_train, y_train)\n\npredicted_categoriesE = vot_clf.predict(x_test)\n\nens_acc, ens_recall, ens_f1 = metrics_by_client(\"VotingClassifier\", y_test, predicted_categoriesE)\n\nprint(\"La precision es {}\".format(ens_acc))\n\nprint(\"El recall es {}\".format(ens_recall))\n\nprint(\"F1 es {}\".format(ens_f1))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:54:59.239771Z","iopub.status.busy":"2022-08-04T00:54:59.238999Z","iopub.status.idle":"2022-08-04T00:58:25.343232Z","shell.execute_reply":"2022-08-04T00:58:25.341968Z"},"id":"9wSGAcxG_aIU","outputId":"90b7927d-49d7-459b-b45e-c4b66576a7c9","papermill":{"duration":206.543664,"end_time":"2022-08-04T00:58:25.572264","exception":false,"start_time":"2022-08-04T00:54:59.0286","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cm = confusion_matrix_by_client(y_test, predicted_categoriesE)\nsns.heatmap(cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:58:26.082416Z","iopub.status.busy":"2022-08-04T00:58:26.082003Z","iopub.status.idle":"2022-08-04T00:58:28.841802Z","shell.execute_reply":"2022-08-04T00:58:28.840543Z"},"id":"FTrIfjrI_aIU","outputId":"fe3e417a-899f-4ec1-cfae-2e09b15b2c0e","papermill":{"duration":3.052038,"end_time":"2022-08-04T00:58:28.845026","exception":false,"start_time":"2022-08-04T00:58:25.792988","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Red neuronal","metadata":{"id":"mi11HE9a_aIU","papermill":{"duration":0.210286,"end_time":"2022-08-04T00:58:29.27971","exception":false,"start_time":"2022-08-04T00:58:29.069424","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Vamos a empezar por un modelo de red neuronal básico, y luego le vamos a ir agregando complejidad hasta llegar a un modelo predictor que supere a los anteriores","metadata":{"papermill":{"duration":0.210051,"end_time":"2022-08-04T00:58:29.69839","exception":false,"start_time":"2022-08-04T00:58:29.488339","status":"completed"},"tags":[]}},{"cell_type":"code","source":"columnas_predictoras=completo.drop(columns=['target', 'customer_ID']).columns.to_list()\nd_in=len(columnas_predictoras)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:58:30.12134Z","iopub.status.busy":"2022-08-04T00:58:30.120702Z","iopub.status.idle":"2022-08-04T00:58:30.192181Z","shell.execute_reply":"2022-08-04T00:58:30.189972Z"},"id":"ofg_ocLS_aIU","papermill":{"duration":0.288418,"end_time":"2022-08-04T00:58:30.195991","exception":false,"start_time":"2022-08-04T00:58:29.907573","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Definimos funciones que nos van a ayudar a comparar a los diferentes modelos","metadata":{"papermill":{"duration":0.212392,"end_time":"2022-08-04T00:58:30.654133","exception":false,"start_time":"2022-08-04T00:58:30.441741","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def plot_hist(history):\n    acc = history.history['accuracy']\n    loss = history.history['loss']\n    \n    epochs = range(1, len(acc) + 1)\n\n    plt.plot(epochs, acc, 'b', label='Training acc')\n    plt.title('Training accuracy')\n    plt.legend()\n\n    plt.figure()\n\n    plt.plot(epochs, loss, 'b', label='Training loss')\n    plt.title('Training loss')\n    plt.legend()\n\n    plt.show()\n    \ndef print_score(name, y_test, pred_red):\n    \n    red_acc, red_recall, red_f1 = metrics_by_client(name, y_test, pred_red)\n\n    print(\"La precision es {}\".format(red_acc))\n\n    print(\"El recall es {}\".format(red_recall))\n\n    print(\"F1 es {}\".format(red_f1))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:58:31.090652Z","iopub.status.busy":"2022-08-04T00:58:31.089799Z","iopub.status.idle":"2022-08-04T00:58:31.097624Z","shell.execute_reply":"2022-08-04T00:58:31.096918Z"},"id":"9Yn2xZ3w_aIY","papermill":{"duration":0.229458,"end_time":"2022-08-04T00:58:31.099721","exception":false,"start_time":"2022-08-04T00:58:30.870263","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"El primer modelo que vamos a probar es uno básico, con 1 hidden layer de 16 nodos cuya función de regularización L2 es de tipo RELU. Como capa de output, elegimos una función de activación sigmoidea porque queremos clasificar 2 clases excluyentes. Como optimizador elegimos inicialmente SGD","metadata":{"papermill":{"duration":0.20754,"end_time":"2022-08-04T00:58:31.520601","exception":false,"start_time":"2022-08-04T00:58:31.313061","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def baseline_red_neuronal():\n    features = completo_medias.drop(['customer_ID', 'target'], axis='columns', inplace=False).columns.to_list()\n    regularization = 4e-4\n    activation_func = 'relu'\n    inputs = Input(shape = (len(features)))\n\n    x = Dense(16,\n           kernel_regularizer = tf.keras.regularizers.l2(regularization),\n           activation = activation_func)(inputs)\n\n    x = Dense(1,\n              activation='sigmoid')(x)\n\n    model = Model(inputs, x)\n    model.compile(\n        optimizer=tf.keras.optimizers.SGD(),\n        loss=tf.keras.losses.BinaryCrossentropy(),\n        metrics=['accuracy'],\n    )\n\n    \n    return model","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:58:32.000461Z","iopub.status.busy":"2022-08-04T00:58:32.000049Z","iopub.status.idle":"2022-08-04T00:58:32.008389Z","shell.execute_reply":"2022-08-04T00:58:32.006975Z"},"papermill":{"duration":0.221447,"end_time":"2022-08-04T00:58:32.010654","exception":false,"start_time":"2022-08-04T00:58:31.789207","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"baseline_red = baseline_red_neuronal()\nbaseline_red.summary()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:58:32.431218Z","iopub.status.busy":"2022-08-04T00:58:32.430477Z","iopub.status.idle":"2022-08-04T00:58:32.673145Z","shell.execute_reply":"2022-08-04T00:58:32.672045Z"},"papermill":{"duration":0.456096,"end_time":"2022-08-04T00:58:32.677181","exception":false,"start_time":"2022-08-04T00:58:32.221085","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Entrenamiento del modelo\nhistory_base = baseline_red.fit(x_train,y_train,epochs=10,batch_size=8,verbose=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T00:58:33.105665Z","iopub.status.busy":"2022-08-04T00:58:33.104729Z","iopub.status.idle":"2022-08-04T01:00:55.890896Z","shell.execute_reply":"2022-08-04T01:00:55.889037Z"},"papermill":{"duration":143.002597,"end_time":"2022-08-04T01:00:55.893529","exception":false,"start_time":"2022-08-04T00:58:32.890932","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como la función de activación que estamos utilizando en la capa de salida es una función sigmoidea, el output de la red va a ser un número entre 0 y 1. Es por ello que vamos a aproximar según esté más cercano a 0 o a 1 para definir la clase a la que pertenece (pago o impago)","metadata":{"papermill":{"duration":0.387445,"end_time":"2022-08-04T01:00:56.68316","exception":false,"start_time":"2022-08-04T01:00:56.295715","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pred_base_red = baseline_red.predict(x_test)\n\nprint_score(\"NN Baseline SGD\", y_test, pred_base_red)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:00:58.066277Z","iopub.status.busy":"2022-08-04T01:00:58.06088Z","iopub.status.idle":"2022-08-04T01:01:08.155558Z","shell.execute_reply":"2022-08-04T01:01:08.154319Z"},"papermill":{"duration":10.645598,"end_time":"2022-08-04T01:01:08.157777","exception":false,"start_time":"2022-08-04T01:00:57.512179","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rn1_cm = confusion_matrix_by_client(y_test, pred_base_red)\nsns.heatmap(rn1_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:01:09.036057Z","iopub.status.busy":"2022-08-04T01:01:09.035284Z","iopub.status.idle":"2022-08-04T01:01:11.785173Z","shell.execute_reply":"2022-08-04T01:01:11.784375Z"},"papermill":{"duration":3.221086,"end_time":"2022-08-04T01:01:11.787386","exception":false,"start_time":"2022-08-04T01:01:08.5663","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(history_base)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:01:12.581531Z","iopub.status.busy":"2022-08-04T01:01:12.581116Z","iopub.status.idle":"2022-08-04T01:01:12.997277Z","shell.execute_reply":"2022-08-04T01:01:12.996268Z"},"papermill":{"duration":0.814596,"end_time":"2022-08-04T01:01:12.999422","exception":false,"start_time":"2022-08-04T01:01:12.184826","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vamos a partir de la misma base del modelo anterior pero ahora modificando el optimizador por Adam, con los valores default (LR 0,001, Beta_1 0,9 y Beta_2 0,999)","metadata":{"papermill":{"duration":0.395047,"end_time":"2022-08-04T01:01:13.784279","exception":false,"start_time":"2022-08-04T01:01:13.389232","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def baseline_red_neuronal_v2():\n    features = completo_medias.drop(['customer_ID', 'target'], axis='columns', inplace=False).columns.to_list()\n    regularization = 4e-4\n    activation_func = 'relu'\n    inputs = Input(shape = (len(features)))\n\n    x = Dense(16,\n           kernel_regularizer = tf.keras.regularizers.l2(regularization),\n           activation = activation_func)(inputs)\n\n    x = Dense(1,\n              activation='sigmoid')(x)\n\n    model = Model(inputs, x)\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999),\n        loss=tf.keras.losses.BinaryCrossentropy(),\n        metrics=['accuracy'],\n    )\n\n    \n    return model","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:01:14.582733Z","iopub.status.busy":"2022-08-04T01:01:14.582042Z","iopub.status.idle":"2022-08-04T01:01:14.592625Z","shell.execute_reply":"2022-08-04T01:01:14.591647Z"},"papermill":{"duration":0.405776,"end_time":"2022-08-04T01:01:14.594542","exception":false,"start_time":"2022-08-04T01:01:14.188766","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"baseline_red_v2 = baseline_red_neuronal_v2()\nbaseline_red_v2.summary()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:01:15.474733Z","iopub.status.busy":"2022-08-04T01:01:15.473767Z","iopub.status.idle":"2022-08-04T01:01:15.580435Z","shell.execute_reply":"2022-08-04T01:01:15.579618Z"},"papermill":{"duration":0.527049,"end_time":"2022-08-04T01:01:15.582658","exception":false,"start_time":"2022-08-04T01:01:15.055609","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Entrenamiento del modelo\nhistory_base_v2 = baseline_red_v2.fit(x_train,y_train,epochs=10,batch_size=8,verbose=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:01:16.363445Z","iopub.status.busy":"2022-08-04T01:01:16.36302Z","iopub.status.idle":"2022-08-04T01:04:16.573886Z","shell.execute_reply":"2022-08-04T01:04:16.571692Z"},"papermill":{"duration":180.607073,"end_time":"2022-08-04T01:04:16.576626","exception":false,"start_time":"2022-08-04T01:01:15.969553","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_base_red_v2 = baseline_red_v2.predict(x_test)\n\nprint_score(\"NN Baseline Adam\", y_test, pred_base_red_v2)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:04:17.841381Z","iopub.status.busy":"2022-08-04T01:04:17.840976Z","iopub.status.idle":"2022-08-04T01:04:26.019626Z","shell.execute_reply":"2022-08-04T01:04:26.018078Z"},"papermill":{"duration":8.804213,"end_time":"2022-08-04T01:04:26.022694","exception":false,"start_time":"2022-08-04T01:04:17.218481","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rn2_cm = confusion_matrix_by_client(y_test, pred_base_red_v2)\nsns.heatmap(rn2_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:04:27.323508Z","iopub.status.busy":"2022-08-04T01:04:27.322791Z","iopub.status.idle":"2022-08-04T01:04:30.198638Z","shell.execute_reply":"2022-08-04T01:04:30.19744Z"},"papermill":{"duration":3.565174,"end_time":"2022-08-04T01:04:30.201056","exception":false,"start_time":"2022-08-04T01:04:26.635882","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(history_base_v2)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:04:31.430996Z","iopub.status.busy":"2022-08-04T01:04:31.430065Z","iopub.status.idle":"2022-08-04T01:04:31.8317Z","shell.execute_reply":"2022-08-04T01:04:31.830607Z"},"papermill":{"duration":1.015781,"end_time":"2022-08-04T01:04:31.833961","exception":false,"start_time":"2022-08-04T01:04:30.81818","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Probamos otro modelo agregando una hidden layer después de la input layer, y sumando una capa de dropout entre ambas hidden layers para evitar el overfitting","metadata":{"papermill":{"duration":0.616231,"end_time":"2022-08-04T01:04:33.178487","exception":false,"start_time":"2022-08-04T01:04:32.562256","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def red_neuronal_2_hidden_1_drop():\n    features = completo_medias.drop(['customer_ID', 'target'], axis='columns', inplace=False).columns.to_list()\n    regularization = 4e-4\n    activation_func = 'relu'\n    inputs = Input(shape = (len(features)))\n    \n    x = Dense(32,\n           kernel_regularizer = tf.keras.regularizers.l2(regularization),\n           activation = activation_func)(inputs)\n    \n    x = Dropout(0.1)(x)\n\n\n    x = Dense(16,\n           kernel_regularizer = tf.keras.regularizers.l2(regularization),\n           activation = activation_func)(x)\n\n    x = Dense(1,\n              activation='sigmoid')(x)\n\n    model = Model(inputs, x)\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999),\n        loss=tf.keras.losses.BinaryCrossentropy(),\n        metrics=['accuracy'],\n    )\n\n    \n    return model","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:04:34.519471Z","iopub.status.busy":"2022-08-04T01:04:34.518741Z","iopub.status.idle":"2022-08-04T01:04:34.529132Z","shell.execute_reply":"2022-08-04T01:04:34.527857Z"},"papermill":{"duration":0.714298,"end_time":"2022-08-04T01:04:34.531444","exception":false,"start_time":"2022-08-04T01:04:33.817146","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"red_2h1d = red_neuronal_2_hidden_1_drop()\nred_2h1d.summary()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:04:35.831323Z","iopub.status.busy":"2022-08-04T01:04:35.830239Z","iopub.status.idle":"2022-08-04T01:04:35.941958Z","shell.execute_reply":"2022-08-04T01:04:35.941154Z"},"papermill":{"duration":0.765999,"end_time":"2022-08-04T01:04:35.944286","exception":false,"start_time":"2022-08-04T01:04:35.178287","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Entrenamiento del modelo\nhistory_2h1d = red_2h1d.fit(x_train,y_train,epochs=10,batch_size=8,verbose=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:04:37.323312Z","iopub.status.busy":"2022-08-04T01:04:37.322581Z","iopub.status.idle":"2022-08-04T01:08:07.842992Z","shell.execute_reply":"2022-08-04T01:08:07.842082Z"},"papermill":{"duration":211.24914,"end_time":"2022-08-04T01:08:07.846272","exception":false,"start_time":"2022-08-04T01:04:36.597132","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_red_2h1d = red_2h1d.predict(x_test)\nprint_score(\"NN 2 hidden layers con 1 Dropout\", y_test, pred_red_2h1d)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:08:09.641051Z","iopub.status.busy":"2022-08-04T01:08:09.640382Z","iopub.status.idle":"2022-08-04T01:08:18.700226Z","shell.execute_reply":"2022-08-04T01:08:18.69895Z"},"papermill":{"duration":9.946771,"end_time":"2022-08-04T01:08:18.702354","exception":false,"start_time":"2022-08-04T01:08:08.755583","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rn3_cm = confusion_matrix_by_client(y_test, pred_base_red_v2)\nsns.heatmap(rn3_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:08:20.592002Z","iopub.status.busy":"2022-08-04T01:08:20.591353Z","iopub.status.idle":"2022-08-04T01:08:23.410104Z","shell.execute_reply":"2022-08-04T01:08:23.409198Z"},"papermill":{"duration":3.712854,"end_time":"2022-08-04T01:08:23.412123","exception":false,"start_time":"2022-08-04T01:08:19.699269","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(history_2h1d)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:08:25.378578Z","iopub.status.busy":"2022-08-04T01:08:25.37765Z","iopub.status.idle":"2022-08-04T01:08:25.767189Z","shell.execute_reply":"2022-08-04T01:08:25.766007Z"},"papermill":{"duration":1.374814,"end_time":"2022-08-04T01:08:25.769655","exception":false,"start_time":"2022-08-04T01:08:24.394841","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Probamos con un modelo similar, pero con menos nodos en las hidden layers","metadata":{"papermill":{"duration":0.879669,"end_time":"2022-08-04T01:08:27.528013","exception":false,"start_time":"2022-08-04T01:08:26.648344","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def red_neuronal_2_hidden_1_drop_v2():\n    features = completo_medias.drop(['customer_ID', 'target'], axis='columns', inplace=False).columns.to_list()\n    regularization = 4e-4\n    activation_func = 'relu'\n    inputs = Input(shape = (len(features)))\n    \n    x = Dense(16,\n           kernel_regularizer = tf.keras.regularizers.l2(regularization),\n           activation = activation_func)(inputs)\n    \n    x = Dropout(0.1)(x)\n\n\n    x = Dense(8,\n           kernel_regularizer = tf.keras.regularizers.l2(regularization),\n           activation = activation_func)(x)\n\n    x = Dense(1,\n              activation='sigmoid')(x)\n\n    model = Model(inputs, x)\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999),\n        loss=tf.keras.losses.BinaryCrossentropy(),\n        metrics=['accuracy'],\n    )\n\n    \n    return model","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:08:29.371714Z","iopub.status.busy":"2022-08-04T01:08:29.371008Z","iopub.status.idle":"2022-08-04T01:08:29.379287Z","shell.execute_reply":"2022-08-04T01:08:29.378458Z"},"papermill":{"duration":0.893949,"end_time":"2022-08-04T01:08:29.381428","exception":false,"start_time":"2022-08-04T01:08:28.487479","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"red_2h1d_v2= red_neuronal_2_hidden_1_drop_v2()\nred_2h1d_v2.summary()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:08:31.292725Z","iopub.status.busy":"2022-08-04T01:08:31.291039Z","iopub.status.idle":"2022-08-04T01:08:31.392992Z","shell.execute_reply":"2022-08-04T01:08:31.391654Z"},"papermill":{"duration":0.982799,"end_time":"2022-08-04T01:08:31.396051","exception":false,"start_time":"2022-08-04T01:08:30.413252","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"También, a diferencia del resto de los modelos, esta vez entrenamos con más iteraciones (epochs 20 en lugar de 10) y un batch_size mayor (32 en lugar de 8). La idea es entrenar durante más iteraciones y aprovechar la memoria de la máquina para agarrar una muestra más grande para recorrer toda la red.","metadata":{"papermill":{"duration":0.870607,"end_time":"2022-08-04T01:08:33.208213","exception":false,"start_time":"2022-08-04T01:08:32.337606","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n# Entrenamiento del modelo\nhistory_2h1d_v2 = red_2h1d_v2.fit(x_train,y_train,epochs=20,batch_size=32,verbose=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:08:35.037503Z","iopub.status.busy":"2022-08-04T01:08:35.036656Z","iopub.status.idle":"2022-08-04T01:10:21.145338Z","shell.execute_reply":"2022-08-04T01:10:21.14379Z"},"papermill":{"duration":106.995837,"end_time":"2022-08-04T01:10:21.148361","exception":false,"start_time":"2022-08-04T01:08:34.152524","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_red_2h1d_v2 = red_2h1d_v2.predict(x_test)\n\nprint_score(\"NN 2 hidden layers con 1 Dropout y menos nodos por capa y más iteraciones\", y_test, pred_red_2h1d_v2)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:10:23.337165Z","iopub.status.busy":"2022-08-04T01:10:23.336462Z","iopub.status.idle":"2022-08-04T01:10:31.162995Z","shell.execute_reply":"2022-08-04T01:10:31.16172Z"},"papermill":{"duration":8.849569,"end_time":"2022-08-04T01:10:31.165172","exception":false,"start_time":"2022-08-04T01:10:22.315603","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rn4_cm = confusion_matrix_by_client(y_test,pred_red_2h1d_v2)\nsns.heatmap(rn4_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:10:33.275926Z","iopub.status.busy":"2022-08-04T01:10:33.275506Z","iopub.status.idle":"2022-08-04T01:10:36.072988Z","shell.execute_reply":"2022-08-04T01:10:36.072094Z"},"papermill":{"duration":3.819125,"end_time":"2022-08-04T01:10:36.075072","exception":false,"start_time":"2022-08-04T01:10:32.255947","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(history_2h1d_v2)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:10:38.283664Z","iopub.status.busy":"2022-08-04T01:10:38.282817Z","iopub.status.idle":"2022-08-04T01:10:38.698776Z","shell.execute_reply":"2022-08-04T01:10:38.697702Z"},"papermill":{"duration":1.51279,"end_time":"2022-08-04T01:10:38.701037","exception":false,"start_time":"2022-08-04T01:10:37.188247","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nuevamente volvemos a generar un modelo con una configuración similar, pero luego vamos a cambiar parámetros al entrenar el modelo.","metadata":{"papermill":{"duration":1.099765,"end_time":"2022-08-04T01:10:40.829166","exception":false,"start_time":"2022-08-04T01:10:39.729401","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def red_neuronal(x_train, y_train):\n    features = completo_medias.drop(['customer_ID', 'target'], axis='columns', inplace=False).columns.to_list()\n    regularization = 4e-4\n    activation_func = 'relu'\n    inputs = Input(shape = (len(features)))\n    \n    x = Dense(32,\n           kernel_regularizer = tf.keras.regularizers.l2(regularization),\n           activation = activation_func)(inputs)\n    \n    x = Dropout(0.1)(x)\n\n    x = Dense(16,\n              kernel_regularizer=tf.keras.regularizers.l2(regularization),\n              activation=activation_func)(x)\n\n    x = Dense(1,\n              activation='sigmoid')(x)\n\n    model = Model(inputs, x)\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999),\n        loss=tf.keras.losses.BinaryCrossentropy(),\n        metrics=['accuracy'],\n    )\n\n    \n    return model","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:10:42.929097Z","iopub.status.busy":"2022-08-04T01:10:42.928672Z","iopub.status.idle":"2022-08-04T01:10:42.937702Z","shell.execute_reply":"2022-08-04T01:10:42.936504Z"},"id":"3dzpk5O6_aIV","papermill":{"duration":1.098327,"end_time":"2022-08-04T01:10:42.939712","exception":false,"start_time":"2022-08-04T01:10:41.841385","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_nr = red_neuronal(x_train, y_train)\nmodel_nr.summary()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:10:45.036684Z","iopub.status.busy":"2022-08-04T01:10:45.035142Z","iopub.status.idle":"2022-08-04T01:10:45.139584Z","shell.execute_reply":"2022-08-04T01:10:45.138851Z"},"id":"K-R71E6x_aIV","outputId":"77e216f3-388c-46de-92a0-27498e31a016","papermill":{"duration":1.197791,"end_time":"2022-08-04T01:10:45.144203","exception":false,"start_time":"2022-08-04T01:10:43.946412","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualkeras.layered_view(model_nr,legend=True) ","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:10:47.253182Z","iopub.status.busy":"2022-08-04T01:10:47.251849Z","iopub.status.idle":"2022-08-04T01:10:47.275595Z","shell.execute_reply":"2022-08-04T01:10:47.272894Z"},"id":"f7g_foOI_aIV","outputId":"a8ab86c3-9f28-44fd-d8e1-bbd03bf68cc9","papermill":{"duration":1.126915,"end_time":"2022-08-04T01:10:47.279047","exception":false,"start_time":"2022-08-04T01:10:46.152132","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vamos a agregar una función de callback de Early Stopping: si después de 5 iteraciones, la métrica de pérdida se mantiene sin bajar, cortar el entrenamiento. Aprovechando esto, vamos a entrenar una mayor cantidad de epochs, y aumentar el batch_size respecto al modelo anterior.","metadata":{"papermill":{"duration":1.070306,"end_time":"2022-08-04T01:10:49.445061","exception":false,"start_time":"2022-08-04T01:10:48.374755","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n# Entrenamiento del modelo\nes = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5)\nhistory = model_nr.fit(x_train,y_train,epochs=100, batch_size=128, callbacks=[es], verbose=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:10:51.540195Z","iopub.status.busy":"2022-08-04T01:10:51.539414Z","iopub.status.idle":"2022-08-04T01:12:42.047877Z","shell.execute_reply":"2022-08-04T01:12:42.045964Z"},"id":"ymjjtMmv_aIV","outputId":"7bbaf941-9fa2-44f0-f9f9-897f0b3ae623","papermill":{"duration":111.520057,"end_time":"2022-08-04T01:12:42.050233","exception":false,"start_time":"2022-08-04T01:10:50.530176","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_red_r = model_nr.predict(x_test)\n\nprint_score(\"NN 2 hidden layers con 1 Dropout y muchas iteraciones pero con Early Stopping\", y_test, pred_red_r)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:12:44.457547Z","iopub.status.busy":"2022-08-04T01:12:44.456581Z","iopub.status.idle":"2022-08-04T01:12:52.819802Z","shell.execute_reply":"2022-08-04T01:12:52.818741Z"},"id":"AryVXaEE_aIW","papermill":{"duration":9.597211,"end_time":"2022-08-04T01:12:52.823185","exception":false,"start_time":"2022-08-04T01:12:43.225974","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rn5_cm = confusion_matrix_by_client(y_test,pred_red_r)\nsns.heatmap(rn5_cm, cmap='Blues', annot=True, fmt='g')\nplt.xlabel('Predicted')\nplt.ylabel('True')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:12:55.227652Z","iopub.status.busy":"2022-08-04T01:12:55.226943Z","iopub.status.idle":"2022-08-04T01:12:58.032144Z","shell.execute_reply":"2022-08-04T01:12:58.030336Z"},"papermill":{"duration":3.962774,"end_time":"2022-08-04T01:12:58.034649","exception":false,"start_time":"2022-08-04T01:12:54.071875","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(history)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:13:00.542795Z","iopub.status.busy":"2022-08-04T01:13:00.542204Z","iopub.status.idle":"2022-08-04T01:13:00.853352Z","shell.execute_reply":"2022-08-04T01:13:00.852329Z"},"id":"TFD2xIbb_aIY","outputId":"c24f5bfa-b679-4332-e42c-a2cbdcf4268d","papermill":{"duration":1.539259,"end_time":"2022-08-04T01:13:00.855556","exception":false,"start_time":"2022-08-04T01:12:59.316297","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Se ve como a partir de las 20 epochs no hay mucha ganancia en accuracy y el modelo no aprende nada nuevo, por lo que podemos reducir la cantidad de epochs","metadata":{"id":"aRGm6yxX_aIY","papermill":{"duration":1.209665,"end_time":"2022-08-04T01:13:03.288301","exception":false,"start_time":"2022-08-04T01:13:02.078636","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Ensamble en cascada\n\nCreamos un nuevo clasificador en base a los modelos ya generados y a la red neuronal. Funciona de la siguiente manera:\n\nSea X un cliente\n\nX -> RED(X) = P(\"X NO ES DEUDOR\") < ESPECTRO -> RF(X) = P(\"X NO ES DEUDOR\") < ESPECTRO -> XGB(X) = P(\"X NO ES DEUDOR\") < ESPECTRO -> SVM(X) = P(\"X NO ES DEUDOR\") < ESPECTRO -> X ES DEUDOR (PREDICT TARGET = 1)\n\nSi algun modelo predice una probabilidad mayor que el espectro de que ese cliente no sea deudor, se lo considera no deudor (target = 0)","metadata":{"id":"dCaIpVUH_aIY","papermill":{"duration":1.145748,"end_time":"2022-08-04T01:13:05.672164","exception":false,"start_time":"2022-08-04T01:13:04.526416","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#usar con un solo cliente\ndef predict_by_client(model, x_test):\n    predicts = model.predict_proba(x_test)\n    predicts = [x[0] for x in predicts] #me quedo con la prob de target == 0 para cada registro\n    temp_x_test = x_test_copy.copy()\n    temp_x_test['target_proba'] = predicts\n    temp_x_test = temp_x_test[['customer_ID', 'target_proba']].groupby('customer_ID').agg({'target_proba':np.mean}).reset_index()\n    return temp_x_test.values.tolist()\n\nclass svm_decorator(object):\n    def __init__(self, model):\n        fittable = CalibratedClassifierCV(model)\n        fittable.fit(x_train, y_train)\n        self._model = fittable\n    \n    def predict_proba(self, x_test):\n        return self._model.predict_proba(x_test)\n    \nclass red_decorator(object):\n    def __init__(self, model):\n        self._model = model\n        \n    def predict_proba(self, x_test):\n        probas = self._model.predict(x_test)\n        return [1 - x for x in probas]\n\ndef cascade_predict(x_test, spectre):\n    svm_proba = svm_decorator(svm_model)\n    red_model = red_decorator(red_2h1d_v2)\n    models = [red_model, arbol, xgb_model, svm_proba]\n    predicts_by_customer = []\n    predicts_clients = [predict_by_client(x, x_test) for x in models]\n    \n    for n in range(len(predicts_clients[0])):\n        n_client = [item[n] for item in predicts_clients]\n        target0 = False\n        customer_ID = ''\n        for client in n_client:\n            customer_ID = client[0]\n            proba = client[1]\n            if (proba > spectre):\n                target0 = True\n                predicts_by_customer.append([customer_ID, 0])\n                break\n        if not target0:\n            predicts_by_customer.append([customer_ID, 1])\n    \n    return predicts_by_customer\n\ndef cascade_metrics(name, y_test, cascade_results):\n    test_real = x_test_copy\n    test_real['target'] = y_test\n    test_real = test_real[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})\n    \n    acc = accuracy_score(test_real['target'].values, cascade_results)\n    recall = recall_score(test_real['target'].values, cascade_results)\n    f1 = f1_score(test_real['target'].values, cascade_results, average='macro')\n    \n    save_metric(name, acc, recall, f1)\n        \n    return acc, recall, f1\n\ndef cascade_cm(y_test, cascade_results):\n    test_real = x_test_copy\n    test_real['target'] = y_test\n    test_real = test_real[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})\n    \n    return confusion_matrix(test_real['target'].values, cascade_results)\n\ndef predict_cascade(x_test, spectre):\n    cascade_results = cascade_predict(x_test, spectre)\n    predicts = [x[1] for x in cascade_results]\n    cascade_acc, cascade_recall, cascade_f1 = cascade_metrics(f\"Cascade spectre={spectre:.2f}\", y_test, predicts)\n\n    print(f\"Metricas con spectre={spectre:.2f}:\")\n    print(\"La precision es {}\".format(cascade_acc))\n\n    print(\"El recall es {}\".format(cascade_recall))\n\n    print(\"F1 es {}\".format(cascade_f1))\n    \n    cascada_cm = cascade_cm(y_test, predicts)\n    sns.heatmap(cascada_cm, cmap='Blues', annot=True, fmt='g')\n    plt.title(f\"Confusion Matrix spectre={spectre:.2f}\")\n    plt.xlabel('Predicted')\n    plt.ylabel('True')\n    plt.show()\n    \n    return predicts","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:13:08.13012Z","iopub.status.busy":"2022-08-04T01:13:08.129687Z","iopub.status.idle":"2022-08-04T01:13:08.149709Z","shell.execute_reply":"2022-08-04T01:13:08.148976Z"},"id":"vVgFivSa_aIY","papermill":{"duration":1.260953,"end_time":"2022-08-04T01:13:08.1519","exception":false,"start_time":"2022-08-04T01:13:06.890947","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Vamos a probar con varios valores de spectre y analizar los resultados:","metadata":{"papermill":{"duration":1.275946,"end_time":"2022-08-04T01:13:10.580459","exception":false,"start_time":"2022-08-04T01:13:09.304513","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Probamos desde valores altos hasta valores más bajos y viendo cómo se comporta el ensamble en cascada. Vamos desde 0.99 hasta 0.51","metadata":{"papermill":{"duration":1.19745,"end_time":"2022-08-04T01:13:13.083661","exception":false,"start_time":"2022-08-04T01:13:11.886211","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nspectre_values = [0.99, 0.9, 0.85, 0.8, 0.7, 0.51]\nspectre_predicts = {}\nfor s in spectre_values:\n    spectre_predicts[s] = predict_cascade(x_test, s)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:13:15.661453Z","iopub.status.busy":"2022-08-04T01:13:15.661028Z","iopub.status.idle":"2022-08-04T01:49:55.571829Z","shell.execute_reply":"2022-08-04T01:49:55.570813Z"},"papermill":{"duration":2201.252589,"end_time":"2022-08-04T01:49:55.610796","exception":false,"start_time":"2022-08-04T01:13:14.358207","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A medida que vamos disminuyendo el valor del espectro, vemos como nuestro modelo predice mejor, disminuyendo la cantidad de Falsos Positivos. Esto tiene sentido, porque quiere cuanto más seguros tenemos que estar para dar una predicción de 0, menor cantidad de 0s voy a predecir, pero los pocos que prediga, la certeza será mayor. Como se aprecia, nuestros modelos no nos garantizan probabilidades muy altas para valores de espectro altas, y a medida que vamos cediendo confianza para la detección de no deudores, mayor es la cantidad de Falsos Negativos que van apareciendo.\n\nPara un valor de spectre=0.51, esto significa que vamos a fijarnos en el modelo 1 si la predicción está más cerca del 0 que del 1. Si está cerca del 0 entonces confirmamos que es 0, caso opuesto se lo pasamos al siguiente modelo y así sucesivamente. Si al menos uno de esos modelos predice que está cerca del 0, entonces confirmaríamos que no es deudor. Al ser 51% una probabilidad muy cercana a la otra clase, perdemos la confianza en nuestra predicción y por eso es que se aprecia mayor cantidad de Falsos Negativos.\n\nUn valor razonable podría ser spectre=0.8, es decir que tenemos que estar 80% seguros de que nuestra predicción sea 0 (no deudor) antes de confirmarla como tal. Si no estamos seguros, se lo pasamos al siguiente modelo y así hasta tener la certeza que sea no deudor. Caso opuesto, asumimos que es deudor no pago.","metadata":{"papermill":{"duration":1.236591,"end_time":"2022-08-04T01:49:57.999734","exception":false,"start_time":"2022-08-04T01:49:56.763143","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Conclusión","metadata":{"papermill":{"duration":1.232868,"end_time":"2022-08-04T01:50:00.411487","exception":false,"start_time":"2022-08-04T01:49:59.178619","status":"completed"},"tags":[]}},{"cell_type":"code","source":"metrics_models = get_all_metrics()\nmetrics_keys = get_metrics_key_order()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:02.81225Z","iopub.status.busy":"2022-08-04T01:50:02.811598Z","iopub.status.idle":"2022-08-04T01:50:02.815672Z","shell.execute_reply":"2022-08-04T01:50:02.814986Z"},"papermill":{"duration":1.15715,"end_time":"2022-08-04T01:50:02.817663","exception":false,"start_time":"2022-08-04T01:50:01.660513","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics_data = []\n\nfor k in metrics_keys:\n    model_metrics = metrics_models[k]\n    metrics_data.append([k, model_metrics['accuracy'], model_metrics['recall'], model_metrics['f1']])\n\nmetrics_df = pd.DataFrame(metrics_data, columns=['Name', 'Accuracy', 'Recall', 'F1'])\nmetrics_df","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:05.212266Z","iopub.status.busy":"2022-08-04T01:50:05.210797Z","iopub.status.idle":"2022-08-04T01:50:05.228712Z","shell.execute_reply":"2022-08-04T01:50:05.227727Z"},"papermill":{"duration":1.172239,"end_time":"2022-08-04T01:50:05.230676","exception":false,"start_time":"2022-08-04T01:50:04.058437","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Graficamos a continuación una comparativa entre las distintas redes neuronales","metadata":{"papermill":{"duration":1.147866,"end_time":"2022-08-04T01:50:07.621473","exception":false,"start_time":"2022-08-04T01:50:06.473607","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Buscamos cuál de todas es la mejor red neuronal, para luego analizarla junto al resto de los modelos.","metadata":{"papermill":{"duration":1.265242,"end_time":"2022-08-04T01:50:10.125958","exception":false,"start_time":"2022-08-04T01:50:08.860716","status":"completed"},"tags":[]}},{"cell_type":"code","source":"nn_metrics_df = metrics_df[metrics_df['Name'].str.startswith('NN')]","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:12.524612Z","iopub.status.busy":"2022-08-04T01:50:12.523955Z","iopub.status.idle":"2022-08-04T01:50:12.529885Z","shell.execute_reply":"2022-08-04T01:50:12.529088Z"},"papermill":{"duration":1.249694,"end_time":"2022-08-04T01:50:12.531938","exception":false,"start_time":"2022-08-04T01:50:11.282244","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=nn_metrics_df[\"Accuracy\"] - 0.80, data=nn_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Accuracy (0.80 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:14.915791Z","iopub.status.busy":"2022-08-04T01:50:14.915157Z","iopub.status.idle":"2022-08-04T01:50:15.161667Z","shell.execute_reply":"2022-08-04T01:50:15.160854Z"},"papermill":{"duration":1.488969,"end_time":"2022-08-04T01:50:15.163692","exception":false,"start_time":"2022-08-04T01:50:13.674723","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=nn_metrics_df[\"Recall\"] - 0.80, data=nn_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Recall (0.80 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:17.565013Z","iopub.status.busy":"2022-08-04T01:50:17.564284Z","iopub.status.idle":"2022-08-04T01:50:17.81372Z","shell.execute_reply":"2022-08-04T01:50:17.812638Z"},"papermill":{"duration":1.490633,"end_time":"2022-08-04T01:50:17.81595","exception":false,"start_time":"2022-08-04T01:50:16.325317","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=nn_metrics_df[\"F1\"] - 0.80, data=nn_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"F1 (0.80 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:20.354979Z","iopub.status.busy":"2022-08-04T01:50:20.35449Z","iopub.status.idle":"2022-08-04T01:50:20.608001Z","shell.execute_reply":"2022-08-04T01:50:20.606952Z"},"papermill":{"duration":1.462533,"end_time":"2022-08-04T01:50:20.61029","exception":false,"start_time":"2022-08-04T01:50:19.147757","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"De los modelos de las redes se observan comportamientos muy similares. Las redes que presentan mejores métricas son las últimas 2, ya que si bien son las que peor recall tienen, presentan mejor accuracy y mejor F1. La que mejor recall presenta, si bien es la que más disminuye los Falsos Negativos, el aumento de los Falsos Positivos no permiten que sea la mejor de todas las redes. Por ello, nos vamos a quedar con la anteúltima red como la mejor red.","metadata":{"papermill":{"duration":1.145174,"end_time":"2022-08-04T01:50:22.996448","exception":false,"start_time":"2022-08-04T01:50:21.851274","status":"completed"},"tags":[]}},{"cell_type":"code","source":"metrics_df['Name'] = metrics_df['Name'].str.replace(\"NN 2 hidden layers con 1 Dropout y menos nodos por capa y más iteraciones\", 'Red Neuronal')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:25.409018Z","iopub.status.busy":"2022-08-04T01:50:25.408577Z","iopub.status.idle":"2022-08-04T01:50:25.414285Z","shell.execute_reply":"2022-08-04T01:50:25.413296Z"},"papermill":{"duration":1.161689,"end_time":"2022-08-04T01:50:25.416357","exception":false,"start_time":"2022-08-04T01:50:24.254668","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ahora, con la misma motivación, analizaremos los ensambles en cascada","metadata":{"papermill":{"duration":1.243904,"end_time":"2022-08-04T01:50:27.894796","exception":false,"start_time":"2022-08-04T01:50:26.650892","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cascade_metrics_df = metrics_df[metrics_df['Name'].str.startswith('Cascade')]","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:30.293046Z","iopub.status.busy":"2022-08-04T01:50:30.292261Z","iopub.status.idle":"2022-08-04T01:50:30.298474Z","shell.execute_reply":"2022-08-04T01:50:30.297771Z"},"papermill":{"duration":1.249256,"end_time":"2022-08-04T01:50:30.300558","exception":false,"start_time":"2022-08-04T01:50:29.051302","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=cascade_metrics_df[\"Accuracy\"] - 0.50, data=cascade_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Accuracy (0.50 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:32.712559Z","iopub.status.busy":"2022-08-04T01:50:32.711957Z","iopub.status.idle":"2022-08-04T01:50:32.958321Z","shell.execute_reply":"2022-08-04T01:50:32.957289Z"},"papermill":{"duration":1.484035,"end_time":"2022-08-04T01:50:32.960469","exception":false,"start_time":"2022-08-04T01:50:31.476434","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=cascade_metrics_df[\"Recall\"] - 0.60, data=cascade_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Recall (0.60 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:35.423332Z","iopub.status.busy":"2022-08-04T01:50:35.422182Z","iopub.status.idle":"2022-08-04T01:50:35.683032Z","shell.execute_reply":"2022-08-04T01:50:35.681823Z"},"papermill":{"duration":1.568753,"end_time":"2022-08-04T01:50:35.68534","exception":false,"start_time":"2022-08-04T01:50:34.116587","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=cascade_metrics_df[\"F1\"] - 0.40, data=cascade_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"F1 (0.40 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:38.099702Z","iopub.status.busy":"2022-08-04T01:50:38.099227Z","iopub.status.idle":"2022-08-04T01:50:38.351869Z","shell.execute_reply":"2022-08-04T01:50:38.351064Z"},"papermill":{"duration":1.417331,"end_time":"2022-08-04T01:50:38.354396","exception":false,"start_time":"2022-08-04T01:50:36.937065","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Para los distintos modelos, en este caso los cambios son mucho más notorios. Como ya vimos en la sección del ensamble en cascada, la única forma que tenemos de confirmar que se trata de un target 0 (cliente no deudor), es que estemos seguros en un porcentaje definido por el espectro (spectre). Para el caso de spectre=0.99, esto implica que tenemos que estar 99% seguros de que pertenezca a esta clase para poder confirmarlo. Es decir, dicho de otra forma, la probabilidad de que nuestro modelo de cascada prediga un 0 depende de que uno de los modelos esté prediciendo la clase 0 con una probabilidad del 99%. Esto hace que el recall, sea muy elevado, casi del 100%, porque hay muy pocos Falsos Negativos. El problema de este modelo es que los Falsos Positivos y por ende que la métrica de accuracy dé muy baja, del 50%, lo cual lo hace un modelo aleatorio y no nos sirve. A medida que vamos bajando nuestra medida de certeza (spectre), vamos viendo el comportamiento inverso... predecimos 0 con más facilidad, pero a costa de aumentar mucho los Falsos Negativos.\n\nViendo las métricas y teniendo en cuenta un modelo balanceado, los mejores son los de spectre=0.8 y spectre=0.7, siendo este último un poco mejor que el otro. Pero si bien la diferencia es mínima, preferimos quedarnos con el de 0.8 ya que tenemos mayor certeza al predecir.","metadata":{"papermill":{"duration":1.156367,"end_time":"2022-08-04T01:50:40.752704","exception":false,"start_time":"2022-08-04T01:50:39.596337","status":"completed"},"tags":[]}},{"cell_type":"code","source":"metrics_df['Name'] = metrics_df['Name'].str.replace(\"Cascade spectre=0.80\", 'Ensamble en cascada -- >80%')","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:43.173104Z","iopub.status.busy":"2022-08-04T01:50:43.172683Z","iopub.status.idle":"2022-08-04T01:50:43.178651Z","shell.execute_reply":"2022-08-04T01:50:43.1779Z"},"papermill":{"duration":1.186792,"end_time":"2022-08-04T01:50:43.180722","exception":false,"start_time":"2022-08-04T01:50:41.99393","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ahora analizamos el resto de los modelos junto a la mejor red y mejor ensamble:","metadata":{"papermill":{"duration":1.236385,"end_time":"2022-08-04T01:50:45.656156","exception":false,"start_time":"2022-08-04T01:50:44.419771","status":"completed"},"tags":[]}},{"cell_type":"code","source":"best_metrics_df = metrics_df[((~metrics_df['Name'].str.startswith('NN')) & ((~metrics_df['Name'].str.startswith('Cascade'))))]","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:48.113127Z","iopub.status.busy":"2022-08-04T01:50:48.112659Z","iopub.status.idle":"2022-08-04T01:50:48.120811Z","shell.execute_reply":"2022-08-04T01:50:48.119867Z"},"papermill":{"duration":1.247408,"end_time":"2022-08-04T01:50:48.123177","exception":false,"start_time":"2022-08-04T01:50:46.875769","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=best_metrics_df[\"Accuracy\"] - 0.8, data=best_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Accuracy (0.80 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:50.513797Z","iopub.status.busy":"2022-08-04T01:50:50.512987Z","iopub.status.idle":"2022-08-04T01:50:50.786486Z","shell.execute_reply":"2022-08-04T01:50:50.785402Z"},"papermill":{"duration":1.511156,"end_time":"2022-08-04T01:50:50.788841","exception":false,"start_time":"2022-08-04T01:50:49.277685","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=best_metrics_df[\"Recall\"] - 0.7, data=best_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Recall (0.70 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:53.346585Z","iopub.status.busy":"2022-08-04T01:50:53.346168Z","iopub.status.idle":"2022-08-04T01:50:53.607631Z","shell.execute_reply":"2022-08-04T01:50:53.606657Z"},"papermill":{"duration":1.550536,"end_time":"2022-08-04T01:50:53.609803","exception":false,"start_time":"2022-08-04T01:50:52.059267","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=best_metrics_df[\"F1\"] - 0.6, data=best_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"F1 (0.60 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:50:56.01197Z","iopub.status.busy":"2022-08-04T01:50:56.01151Z","iopub.status.idle":"2022-08-04T01:50:56.271979Z","shell.execute_reply":"2022-08-04T01:50:56.271037Z"},"papermill":{"duration":1.425838,"end_time":"2022-08-04T01:50:56.274414","exception":false,"start_time":"2022-08-04T01:50:54.848576","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Respecto a SVM, notamos que al tunear el modelo con los mejores hiperparámetros, aumenta bastante el recall. Esto se puede apreciar viendo que los Falsos Negativos en la matriz de confusión son muy bajos respecto al resto de los modelos. Pero esto es a costa de sacrificar accuracy y la métrica de F1. Si necesitáramos un buen modelo donde lo que importe es no perdernos estos casos de deudores no pagos (Falsos Negativos), sin duda sería un modelo a tener en cuenta. Lo mismo se aprecia para los ensambles en cascada con un espectro alto, como analizamos anteriormente.\n\nSiguiendo el análisis, con Random Forest pudimos hacer un análisis inicial bastante acertado y que nos permitió ver con certeza cuáles eran los features más importantes de manera rápida. Al tunear los hiperparámetros, si bien la F1 se mantuvo parecida y el accuracy bajó un 1%, donde más se aprecia la mejora es en el recall. Al tener árboles un poco más complejos pero sin llegar a overfittear, logramos reducir las predicciones de falsos no deudores sin perder tanto respecto a las demás métricas. Con XGBoost, que también usa árboles ya que usamos el booster gbtree, observamos lo mismo, con la diferencia que todas las métricas se ven mejoradas levemente al tunearlo, viendo la mayor mejora respecto al recall.\n\nEl ensamble de todos estos modelos con Voting, por todo lo mencionado anteriormente, junta lo mejor de todos estos modelos, generando un modelo balanceado con las mejores cualidades de los modelos ya analizados. Gracias a SVM, el recall es el más alto, solo por detrás de justamente SVM, y a diferencia de este, las métricas de accuracy y F1 son altas, es decir que lo que no predice bien SVM, lo predice bien gracias a Random Forest y XGBoost. Este consideramos que es un buen modelo y es candidato a ser uno de los mejores.\n\nLa mejor de las redes neuronales también es un buen modelo, ya que si bien la métrica de recall no es tan buena como la del VotingClassifier, en el resto de las métricas compite par a par, logrando mejor accuracy y F1. Es por esto que también lo consideramos un buen modelo y candidato a ser uno de los mejores.","metadata":{"papermill":{"duration":1.161758,"end_time":"2022-08-04T01:50:58.708843","exception":false,"start_time":"2022-08-04T01:50:57.547085","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### El mejor modelo","metadata":{"papermill":{"duration":1.162778,"end_time":"2022-08-04T01:51:01.114465","exception":false,"start_time":"2022-08-04T01:50:59.951687","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Vamos a hacer un voting manual con las predicciones ya hechas de los mejores modelos, aprovechando que los modelos ya estan entrenados.","metadata":{"papermill":{"duration":1.31064,"end_time":"2022-08-04T01:51:03.659825","exception":false,"start_time":"2022-08-04T01:51:02.349185","status":"completed"},"tags":[]}},{"cell_type":"code","source":"test_by_customer_real = x_test_copy.copy()\ntest_by_customer_real['target'] = y_test\ntest_by_customer_real = test_by_customer_real[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})\n\ny_test_grouped = test_by_customer_real['target'].values","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:51:06.113404Z","iopub.status.busy":"2022-08-04T01:51:06.112247Z","iopub.status.idle":"2022-08-04T01:51:07.386038Z","shell.execute_reply":"2022-08-04T01:51:07.384945Z"},"papermill":{"duration":2.531186,"end_time":"2022-08-04T01:51:07.38864","exception":false,"start_time":"2022-08-04T01:51:04.857454","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict_by_client(predict):\n    test_by_customer_predict = x_test_copy.copy()\n    test_by_customer_predict['target'] = predict\n    test_by_customer_predict = test_by_customer_predict[['customer_ID', 'target']].groupby('customer_ID').agg({'target':custom_predict})\n    \n    return np.array(test_by_customer_predict['target'])","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:51:09.815549Z","iopub.status.busy":"2022-08-04T01:51:09.814369Z","iopub.status.idle":"2022-08-04T01:51:09.820671Z","shell.execute_reply":"2022-08-04T01:51:09.819941Z"},"papermill":{"duration":1.275065,"end_time":"2022-08-04T01:51:09.822714","exception":false,"start_time":"2022-08-04T01:51:08.547649","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\npredicted_ensamble = predict_by_client(predicted_categoriesE)\npredicted_red = predict_by_client(pred_red_2h1d_v2)\npredicted_cascada = spectre_predicts[0.8]\n\npredicted = np.around((predicted_ensamble + predicted_red + predicted_cascada) / 3)\npredicted = [int(x) for x in predicted]\n\nbest_acc, best_recall, best_f1 = get_metrics(y_test_grouped, predicted)\n\n\nprint(\"La precision es {}\".format(best_acc))\n\nprint(\"El recall es {}\".format(best_recall))\n\nprint(\"F1 es {}\".format(best_f1))","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:51:12.233762Z","iopub.status.busy":"2022-08-04T01:51:12.233156Z","iopub.status.idle":"2022-08-04T01:51:15.136131Z","shell.execute_reply":"2022-08-04T01:51:15.134574Z"},"papermill":{"duration":4.152249,"end_time":"2022-08-04T01:51:15.139506","exception":false,"start_time":"2022-08-04T01:51:10.987257","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_metrics_df = best_metrics_df.append({\n    'Name': \"Best Model\",\n    'Accuracy': best_acc,\n    'Recall': best_recall,\n    'F1': best_f1\n}, ignore_index=True)","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:51:17.542749Z","iopub.status.busy":"2022-08-04T01:51:17.542035Z","iopub.status.idle":"2022-08-04T01:51:17.550775Z","shell.execute_reply":"2022-08-04T01:51:17.549711Z"},"papermill":{"duration":1.162608,"end_time":"2022-08-04T01:51:17.552929","exception":false,"start_time":"2022-08-04T01:51:16.390321","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Con este nuevo modelo calculado, volvemos a ver las métricas comparadas:","metadata":{"papermill":{"duration":1.155622,"end_time":"2022-08-04T01:51:19.949691","exception":false,"start_time":"2022-08-04T01:51:18.794069","status":"completed"},"tags":[]}},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=best_metrics_df[\"Accuracy\"] - 0.8, data=best_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Accuracy (0.80 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:51:22.413977Z","iopub.status.busy":"2022-08-04T01:51:22.413239Z","iopub.status.idle":"2022-08-04T01:51:22.700967Z","shell.execute_reply":"2022-08-04T01:51:22.699864Z"},"papermill":{"duration":1.508879,"end_time":"2022-08-04T01:51:22.703231","exception":false,"start_time":"2022-08-04T01:51:21.194352","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=best_metrics_df[\"Recall\"] - 0.7, data=best_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"Recall (0.70 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:51:25.24928Z","iopub.status.busy":"2022-08-04T01:51:25.248741Z","iopub.status.idle":"2022-08-04T01:51:25.527289Z","shell.execute_reply":"2022-08-04T01:51:25.525932Z"},"papermill":{"duration":1.582348,"end_time":"2022-08-04T01:51:25.529629","exception":false,"start_time":"2022-08-04T01:51:23.947281","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,8))\nax = sns.barplot(y=\"Name\", x=best_metrics_df[\"F1\"] - 0.6, data=best_metrics_df, alpha=0.5, orient='h')\nax.set_xlabel(\"F1 (0.60 + x)\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2022-08-04T01:51:27.967629Z","iopub.status.busy":"2022-08-04T01:51:27.966874Z","iopub.status.idle":"2022-08-04T01:51:28.239769Z","shell.execute_reply":"2022-08-04T01:51:28.238989Z"},"papermill":{"duration":1.524661,"end_time":"2022-08-04T01:51:28.242109","exception":false,"start_time":"2022-08-04T01:51:26.717448","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A modo de conclusión, si bien varios de los modelos resultaron en métricas parecidas, el que creemos que es el mejor modelo es este último voting entre el VotingClassifier, la mejor red neuronal y el ensamble en cascada con espectro 80%.","metadata":{"papermill":{"duration":1.24403,"end_time":"2022-08-04T01:51:30.651672","exception":false,"start_time":"2022-08-04T01:51:29.407642","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"**Sintesis sobre los modelos entrenados y los resultados obtenidos**\n\nEn este informe pudimos observar resumenes de clientes que puso American Express a disposicion y entrenar modelos que puedan detectar aquellos que pueden llegar a no pagar a fin de mes. \n\nLos resultados obtenidos para cada modelo muestran que existen features que estan relacionadas con deudores morosos y que pueden hallarse valores que indiquen una probable morosidad a futuro. \n\nLas matrices de confusion de cada modelo nos indican que existen casos borde en donde un cliente puede reunir todas las caracteristicas de un deudor y resultar no serlo, para explicar este comportamiento debiera analizarse los falsos positivos de cada prediccion y encontrar similitudes y diferencias con respecto a los clasificados correctamente. \n\nPara poder hacer un analisis exhaustivo de los clientes que no pueden ser clasificados como morosos, habria que considerar todos los resumenes disponibles en el dataset original de esos clientes y buscar diferencias con los que se utilizaron para entrenar nuestros modelos. Una primera aproximacion seria observar la distribucion de las variables que fueron mas importantes para cada modelo a la hora de decidir su clasificacion y encontrar diferencias entre los deudores mal clasificados y bien clasificados.","metadata":{}},{"cell_type":"markdown","source":"#### Comentarios finales sobre despliegue en producción","metadata":{"papermill":{"duration":1.304982,"end_time":"2022-08-04T01:51:33.123713","exception":false,"start_time":"2022-08-04T01:51:31.818731","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Si el sistema entrase en producción, se podría incluir actividad humana para la revisión de casos específicos en donde la predicción del modelo esté en el límite de ambas clases (como valores entre 0.4 a 0.6), ya que para los cuales la probabilidad de error sea mayor. También se puede incluir para la revisión de registros random con el fin de hacer pruebas de calidad e introducir mejoras, correcciones y nuevos features al sistema. Otra de las revisiones que se puede realizar es hacer un énfasis en los casos que no pudimos resolver, poder buscar los features más importantes del modelo, y fijarnos si esos features muestran una distribución distinta de las que si se pudieron clasificar. Por ejemplo, en nuestros modelos, predominaba como feature más importante la columna **P_2**, de la categoría de pagos. Esta variable, por la importancia que posee, podría tratarse de antecedentes en falta de pago, o algún campo similar que denote la probabilidad en la falta de pago. Al ver las distribuciones de P_2 durante el análisis de datos, encontramos que la distribución entre deudores y no deudores estaba bien marcada.\n\nUn dato no menor es que el dataset del cual partimos para el análisis representa un 5% del dataset original. En el dataset de la competencia se observan los statements de los clientes por un período de 18 meses, y en nuestro split el máximo de registros que tenemos para un mismo cliente es de 6. Si en el modelo pudiéramos incluir la totalidad de los statements, no sólo tendríamos más datos sino que podríamos calcular mejores features, para analizar el comportamiento del cliente para todo el período presentado. En sintonía con la presencia de un período de meses, si quisiéramos llevar este modelo a producción, con el paso del tiempo el comportamiento de los clientes puede variar, y esto por ende implicaria que nuestro modelo también pueda cambiar. No solo deberíamos nutrir al modelo de los nuevos registros, sino que también deberíamos hacer revisiones como búsqueda de nuevos features y entrenamiento de los modelos. Un ejemplo de esto podría ser un pipeline del estilo cascada, donde lo que no puede resolver el modelo productivo, se lo paso a un siguiente modelo que contenga la información novedosa, para identificar casos nuevos o preexistentes.\n\n","metadata":{"papermill":{"duration":1.162746,"end_time":"2022-08-04T01:51:35.543859","exception":false,"start_time":"2022-08-04T01:51:34.381113","status":"completed"},"tags":[]}},{"cell_type":"code","source":"","metadata":{"papermill":{"duration":1.189858,"end_time":"2022-08-04T01:51:37.9903","exception":false,"start_time":"2022-08-04T01:51:36.800442","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]}]}