{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introdução\n\nA chamada do problema traz a seguinte pergunta: \"Can you identify who will make a transaction?\". A contextualização do problema informa que o objetivo é identificar quais clientes farão uma transação. Embora não haja detalhes sobre a transação, pode ser a compra de um produto, a aquisição de um serviço ou algo do tipo.\n\nTrata-se de um problema de Aprendizado Supervisionado. Os dados fornecidos possum um rótulo, identificado pela coluna `target`, que possui valor binário (0 | 1) e deve ser a coluna usada para inferência.\n\nDois fatores chamam a atenção para este problema:\n\n1. Há um desbalanceamento entre as classes de saída. Aproximadamente 90% dos dados são referentes à classe 0.\n2. Não há uma identificação clara dos dados. Uma hipótese é que estes dados tenham sido gerados por algum processo de compactação / criptografia para esconder os dados reais (provavelmente por questões de sigilo ou algo do tipo). Embora isso não venha a ser um problema de fato, traz algumas questões que podem dificultar a análise. Por exemplo: Os dados originais possuiam dados categóricos? Como estes dados foram tratados?\n\nApesar do objetivo do problema ser inferir a classe (0 | 1), as regras de avaliação informam o uso da métrica ROC / AUC.\n> _\"Submissions are evaluated on area under the ROC curve between the predicted probability and the observed target.\"_","metadata":{}},{"cell_type":"markdown","source":"# Metodologia\n\nEsta análise tem como objetivo explorar intensivamente os dados e testar modelos de classificação.\n\nConforme referências [1] e [2], foi verificado que os dados de teste possuem registros sintéticos, identificados por não possuírem valores únicos. Explorar esta questão nos dados de teste está fora do escopo desta análise, que possui um foco em explorar os dados de treinamento e usar os dados de teste apenas para a submissão. No entanto, a idéia de explorar valores únicos será aproveitada nos modelos de classificação.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-29T10:46:33.916481Z","iopub.execute_input":"2022-07-29T10:46:33.916898Z","iopub.status.idle":"2022-07-29T10:46:33.952381Z","shell.execute_reply.started":"2022-07-29T10:46:33.916816Z","shell.execute_reply":"2022-07-29T10:46:33.951677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Imports\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, PolynomialFeatures, PowerTransformer\nfrom sklearn.decomposition import PCA\nfrom sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier, AdaBoostClassifier\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import roc_auc_score, make_scorer, confusion_matrix, ConfusionMatrixDisplay, classification_report\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\n\nfrom sklearn.exceptions import ConvergenceWarning\nimport warnings\nwarnings.filterwarnings(action=\"ignore\", category=ConvergenceWarning)\nwarnings.filterwarnings(action='once')\n\nfrom keras.layers import Input, Dense\nfrom keras.models import Model, Sequential\nfrom keras import regularizers\nfrom tensorflow.keras.callbacks import EarlyStopping\n\nfrom lightgbm import LGBMClassifier\n\nfrom imblearn.over_sampling import RandomOverSampler as ROS\n\nfrom scipy import stats\n\nfrom tqdm import tqdm\nimport seaborn as sns\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:33.953966Z","iopub.execute_input":"2022-07-29T10:46:33.954550Z","iopub.status.idle":"2022-07-29T10:46:42.447282Z","shell.execute_reply.started":"2022-07-29T10:46:33.954513Z","shell.execute_reply":"2022-07-29T10:46:42.446455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Variável auxiliar para teste de código. Caso esteja marcada como True, o código usara apenas uma pequena parte dos dados para execução\nFAST_RUN = False","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:42.373303Z","iopub.execute_input":"2022-07-29T10:49:42.373815Z","iopub.status.idle":"2022-07-29T10:49:42.378570Z","shell.execute_reply.started":"2022-07-29T10:49:42.373772Z","shell.execute_reply":"2022-07-29T10:49:42.377544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Importação dos dados","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/train.csv')\nif FAST_RUN == True:\n    df_train = df_train.sample(10000)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:42.458313Z","iopub.execute_input":"2022-07-29T10:46:42.458882Z","iopub.status.idle":"2022-07-29T10:46:52.243990Z","shell.execute_reply.started":"2022-07-29T10:46:42.458845Z","shell.execute_reply":"2022-07-29T10:46:52.243151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.246261Z","iopub.execute_input":"2022-07-29T10:46:52.246683Z","iopub.status.idle":"2022-07-29T10:46:52.278164Z","shell.execute_reply.started":"2022-07-29T10:46:52.246643Z","shell.execute_reply":"2022-07-29T10:46:52.277329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.279331Z","iopub.execute_input":"2022-07-29T10:46:52.279763Z","iopub.status.idle":"2022-07-29T10:46:52.285673Z","shell.execute_reply.started":"2022-07-29T10:46:52.279727Z","shell.execute_reply":"2022-07-29T10:46:52.284760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explorando os dados","metadata":{}},{"cell_type":"code","source":"df_train.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.286976Z","iopub.execute_input":"2022-07-29T10:46:52.287514Z","iopub.status.idle":"2022-07-29T10:46:52.497078Z","shell.execute_reply.started":"2022-07-29T10:46:52.287478Z","shell.execute_reply":"2022-07-29T10:46:52.496178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"O dataset é composto de 202 colunas, sendo 200 do tipo float, 1 do tipo int e 1 do tipo string.\n\n`Columns: 202 entries, ID_code to var_199`","metadata":{}},{"cell_type":"code","source":"# Listando colunas do tipo float\ndf_train.select_dtypes(float).columns","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.498352Z","iopub.execute_input":"2022-07-29T10:46:52.498788Z","iopub.status.idle":"2022-07-29T10:46:52.509777Z","shell.execute_reply.started":"2022-07-29T10:46:52.498751Z","shell.execute_reply":"2022-07-29T10:46:52.508914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Listando colunas do tipo int\ndf_train.select_dtypes(int).columns","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.511217Z","iopub.execute_input":"2022-07-29T10:46:52.511857Z","iopub.status.idle":"2022-07-29T10:46:52.521608Z","shell.execute_reply.started":"2022-07-29T10:46:52.511820Z","shell.execute_reply":"2022-07-29T10:46:52.520727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Listando colunas do tipo string\ndf_train.select_dtypes(object).columns","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.523161Z","iopub.execute_input":"2022-07-29T10:46:52.523495Z","iopub.status.idle":"2022-07-29T10:46:52.532745Z","shell.execute_reply.started":"2022-07-29T10:46:52.523462Z","shell.execute_reply":"2022-07-29T10:46:52.531825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Verificando nulos\ndf_train.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.536290Z","iopub.execute_input":"2022-07-29T10:46:52.537991Z","iopub.status.idle":"2022-07-29T10:46:52.551472Z","shell.execute_reply.started":"2022-07-29T10:46:52.537945Z","shell.execute_reply":"2022-07-29T10:46:52.550767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Balanceamento da classe de saída\nsns.countplot(x='target', data=df_train);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.552417Z","iopub.execute_input":"2022-07-29T10:46:52.552661Z","iopub.status.idle":"2022-07-29T10:46:52.701957Z","shell.execute_reply.started":"2022-07-29T10:46:52.552639Z","shell.execute_reply":"2022-07-29T10:46:52.701083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['target'].value_counts() / df_train.shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.706168Z","iopub.execute_input":"2022-07-29T10:46:52.706668Z","iopub.status.idle":"2022-07-29T10:46:52.718330Z","shell.execute_reply.started":"2022-07-29T10:46:52.706623Z","shell.execute_reply":"2022-07-29T10:46:52.716867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As classes de saída possuem alto nível de desbalanceamento. A classe minoritária (classe 1) aparece em somente 10% dos registros.","metadata":{}},{"cell_type":"markdown","source":"## Valores únicos","metadata":{}},{"cell_type":"code","source":"# Percentual de valores únicos por atributo\ncols = [c for c in df_train.columns if 'var' in c]\ndf_unique = df_train[cols].nunique().to_frame()\ndf_unique.rename(columns={0: 'count'}, inplace=True)\ndf_unique['perc'] = df_unique['count'] / df_train.shape[0]\ndf_unique.sort_values(by='count', ascending=True, inplace=True)\ndf_unique.head()\n\n# Distribuição de valores únicos\nsns.displot(x='perc', data=df_unique, kde='True');","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:52.719434Z","iopub.execute_input":"2022-07-29T10:46:52.719962Z","iopub.status.idle":"2022-07-29T10:46:53.127292Z","shell.execute_reply.started":"2022-07-29T10:46:52.719924Z","shell.execute_reply":"2022-07-29T10:46:53.126534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aqui ressalto uma das dificuldades em trabalhar com dados anonimizados. Existem colunas com pouquíssimos dados únicos e não é possível obter maiores informações sobre o que estas variáveis representam. Apesar disso, a informação da frequência de valores únicos pode ser útil para a análise e será utilizada um um dos modelos de classificação.","metadata":{}},{"cell_type":"markdown","source":"## Testando","metadata":{}},{"cell_type":"code","source":"# df = df_train.drop(columns=['ID_code', 'target'])\n# list_resp = []\n# for idx, c1 in tqdm(enumerate(df.columns)):\n#     for c2 in df.columns[idx:]:\n#         if c1==c2: continue\n#         list_resp.append([c1, c2, (df[c1] > df[c2]).sum() / df.shape[0]])\n\n# df_resp = pd.DataFrame(columns=['C1', 'C2', 'RESP'], data=list_resp)\n# df_resp.sort_values(by='RESP', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.128461Z","iopub.execute_input":"2022-07-29T10:46:53.128890Z","iopub.status.idle":"2022-07-29T10:46:53.133383Z","shell.execute_reply.started":"2022-07-29T10:46:53.128852Z","shell.execute_reply":"2022-07-29T10:46:53.132581Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_resp = df_resp.loc[df_resp['RESP'].isin([0, 1])].copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.134846Z","iopub.execute_input":"2022-07-29T10:46:53.135468Z","iopub.status.idle":"2022-07-29T10:46:53.145145Z","shell.execute_reply.started":"2022-07-29T10:46:53.135429Z","shell.execute_reply":"2022-07-29T10:46:53.144383Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_resp","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.146550Z","iopub.execute_input":"2022-07-29T10:46:53.147202Z","iopub.status.idle":"2022-07-29T10:46:53.156293Z","shell.execute_reply.started":"2022-07-29T10:46:53.147164Z","shell.execute_reply":"2022-07-29T10:46:53.155565Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df['var_194'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.158759Z","iopub.execute_input":"2022-07-29T10:46:53.159081Z","iopub.status.idle":"2022-07-29T10:46:53.165644Z","shell.execute_reply.started":"2022-07-29T10:46:53.159056Z","shell.execute_reply":"2022-07-29T10:46:53.164978Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# np.sort(np.unique(np.concatenate([df_resp['C1'].unique(), df_resp['C2'].unique()])))","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.167199Z","iopub.execute_input":"2022-07-29T10:46:53.167434Z","iopub.status.idle":"2022-07-29T10:46:53.175710Z","shell.execute_reply.started":"2022-07-29T10:46:53.167411Z","shell.execute_reply":"2022-07-29T10:46:53.175090Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_train.describe().T","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.176909Z","iopub.execute_input":"2022-07-29T10:46:53.177337Z","iopub.status.idle":"2022-07-29T10:46:53.185968Z","shell.execute_reply.started":"2022-07-29T10:46:53.177301Z","shell.execute_reply":"2022-07-29T10:46:53.185283Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_x = df_unique.sort_values(by='count')\n# df_x['count'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.187265Z","iopub.execute_input":"2022-07-29T10:46:53.187638Z","iopub.status.idle":"2022-07-29T10:46:53.194254Z","shell.execute_reply.started":"2022-07-29T10:46:53.187605Z","shell.execute_reply":"2022-07-29T10:46:53.193449Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# x = df_unique.reset_index()\n# # x.loc[x['index'].isin(['var_6', 'var_165', 'var_53', 'var_110', 'var_12', 'var_99', 'var_174', 'var_1', 'var_13', 'var_170'])]\n# x","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.195470Z","iopub.execute_input":"2022-07-29T10:46:53.195900Z","iopub.status.idle":"2022-07-29T10:46:53.204258Z","shell.execute_reply.started":"2022-07-29T10:46:53.195866Z","shell.execute_reply":"2022-07-29T10:46:53.203539Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# for var in df_train.columns:\n#     if 'var' not in var: continue\n#     var_unique_sorted = np.sort(df_train[var].unique())\n#     var_diff = np.diff(var_unique_sorted)\n#     var_diff = np.round(var_diff, 2)\n#     qtde_diff = len(set(var_diff))\n# #     print(var, qtde_diff)\n    ","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.205572Z","iopub.execute_input":"2022-07-29T10:46:53.205921Z","iopub.status.idle":"2022-07-29T10:46:53.218609Z","shell.execute_reply.started":"2022-07-29T10:46:53.205888Z","shell.execute_reply":"2022-07-29T10:46:53.217521Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pd.factorize(df_train['var_68'])","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.221361Z","iopub.execute_input":"2022-07-29T10:46:53.221601Z","iopub.status.idle":"2022-07-29T10:46:53.228546Z","shell.execute_reply.started":"2022-07-29T10:46:53.221579Z","shell.execute_reply":"2022-07-29T10:46:53.227810Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# var = 'var_12'\n# df_train[var].hist();","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.229426Z","iopub.execute_input":"2022-07-29T10:46:53.229722Z","iopub.status.idle":"2022-07-29T10:46:53.238122Z","shell.execute_reply.started":"2022-07-29T10:46:53.229677Z","shell.execute_reply":"2022-07-29T10:46:53.237225Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# var_unique_sorted = np.sort(df_train[var].unique())\n# np.diff(var_unique_sorted).tolist()[:100]","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.239588Z","iopub.execute_input":"2022-07-29T10:46:53.240216Z","iopub.status.idle":"2022-07-29T10:46:53.247931Z","shell.execute_reply.started":"2022-07-29T10:46:53.240150Z","shell.execute_reply":"2022-07-29T10:46:53.247138Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plt.plot(np.log(np.diff(var_unique_sorted).tolist()));","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.248787Z","iopub.execute_input":"2022-07-29T10:46:53.249027Z","iopub.status.idle":"2022-07-29T10:46:53.258016Z","shell.execute_reply.started":"2022-07-29T10:46:53.249004Z","shell.execute_reply":"2022-07-29T10:46:53.257277Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# np.diff(var_68_unique_sorted / 1.0e-04)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.259354Z","iopub.execute_input":"2022-07-29T10:46:53.259768Z","iopub.status.idle":"2022-07-29T10:46:53.267753Z","shell.execute_reply.started":"2022-07-29T10:46:53.259734Z","shell.execute_reply":"2022-07-29T10:46:53.267082Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# var_68_int = df_train['var_68']/1.0e-03\n# var_68_int.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.274051Z","iopub.execute_input":"2022-07-29T10:46:53.274305Z","iopub.status.idle":"2022-07-29T10:46:53.279419Z","shell.execute_reply.started":"2022-07-29T10:46:53.274283Z","shell.execute_reply":"2022-07-29T10:46:53.278522Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# x = var_68_int - var_68_int.min()\n# x.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.280861Z","iopub.execute_input":"2022-07-29T10:46:53.281210Z","iopub.status.idle":"2022-07-29T10:46:53.288162Z","shell.execute_reply.started":"2022-07-29T10:46:53.281177Z","shell.execute_reply":"2022-07-29T10:46:53.287475Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Correlação entre variáveis\n\nDevido à quantidade de variáveis, fica inviável analisar uma matriz de correlação. Alternativamente, esta análise será feita analisando a distribuição das correlações entre variáveis.\n\nAnálise aproveitada da referência [3].","metadata":{}},{"cell_type":"code","source":"# Correlação entre as variáveis preditoras\ndf_corr = df_train.drop(['ID_code', 'target'], axis=1).corr()\ndf_corr = df_corr.values.flatten()\ndf_corr = df_corr[df_corr != 1]\n\nprint(f'Min correlation = {df_corr.min()}')\nprint(f'Max correlation = {df_corr.max()}')\n\nsns.displot(data=df_corr, kde=True, height=8, aspect=2)\nplt.title(\"Correlação entre variáveis preditoras\");","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:53.289833Z","iopub.execute_input":"2022-07-29T10:46:53.290602Z","iopub.status.idle":"2022-07-29T10:46:54.793328Z","shell.execute_reply.started":"2022-07-29T10:46:53.290554Z","shell.execute_reply":"2022-07-29T10:46:54.792407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Não há correlação linear relavante entre as variáveis. Estou reforçando aqui a hipótese de que os dados tenham sido gerados por algum processo de compactação que aproveitou os principais componentes dos dados (ex.: PCA).","metadata":{}},{"cell_type":"code","source":"# Correlação com a variável de resposta\ndf_corr_target = df_train.drop(['ID_code'], axis=1).corr()[['target']]\ndf_corr_target.loc[df_corr_target.index!='target'].boxplot();\nplt.title(\"Resumo das correlações entre as variáveis de entrada e a variável de saída\");","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:54.794810Z","iopub.execute_input":"2022-07-29T10:46:54.795165Z","iopub.status.idle":"2022-07-29T10:46:55.887435Z","shell.execute_reply.started":"2022-07-29T10:46:54.795124Z","shell.execute_reply":"2022-07-29T10:46:55.886586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Correlação com o index\ndf_corr_idx = df_train.drop(['ID_code'], axis=1).reset_index()\ndf_corr_idx = df_corr_idx.corr()\ndf_corr_idx_input = df_corr_idx[['index']].drop(['index', 'target'], axis=0)\nprint(f\"Correlação entre o índice dos registros e a variável de saída {df_corr_idx.loc['index', 'target']}\\n\\n\")\n\ndf_corr_idx_input.boxplot()\nplt.title(\"Resumo das correlações entre o índice dos registros e as variáveis de entrada\");\n","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:55.890712Z","iopub.execute_input":"2022-07-29T10:46:55.893745Z","iopub.status.idle":"2022-07-29T10:46:56.982588Z","shell.execute_reply.started":"2022-07-29T10:46:55.893687Z","shell.execute_reply":"2022-07-29T10:46:56.981613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Teste de normalidade\n\nComo os modelos de ML em geral possuem melhor desempenho com dados normalmente distribuídos, esta seção realiza o teste de normalidade para verificar se cada variável de entrada se aproxima de uma distribuição normal.\n\nEsta análise de normalidade foi adaptada da referência [4].","metadata":{}},{"cell_type":"code","source":"# Armazena o p-value em dataframe\ncols = [c for c in df_train.columns if c not in ['ID_code', 'target']]\ndf_normal_test = pd.DataFrame(columns=['VAR', 'PVALUE'], index=range(len(cols)))\nfor idx, col in enumerate(cols):\n    p_value = stats.normaltest(df_train[col]).pvalue\n    df_normal_test.loc[idx][['VAR', 'PVALUE']] = [col, p_value]\ndf_normal_test['PVALUE'] = df_normal_test['PVALUE'].astype(float)\ndf_normal_test.sort_values(by='PVALUE', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:56.986747Z","iopub.execute_input":"2022-07-29T10:46:56.987143Z","iopub.status.idle":"2022-07-29T10:46:57.408467Z","shell.execute_reply.started":"2022-07-29T10:46:56.987105Z","shell.execute_reply":"2022-07-29T10:46:57.407636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_normal_test.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:57.413826Z","iopub.execute_input":"2022-07-29T10:46:57.415924Z","iopub.status.idle":"2022-07-29T10:46:57.436356Z","shell.execute_reply.started":"2022-07-29T10:46:57.415884Z","shell.execute_reply":"2022-07-29T10:46:57.435666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Todas as variáveis se aproximam de uma distribuição normal com base no método utilizado. Plotando duas variáveis para comparação.","metadata":{}},{"cell_type":"code","source":"var_min_p = df_normal_test.iloc[0]['VAR']\nvar_max_p = df_normal_test.iloc[-1]['VAR']\nprint(df_normal_test.iloc[0])\nprint(df_normal_test.iloc[-1])","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:57.439990Z","iopub.execute_input":"2022-07-29T10:46:57.442045Z","iopub.status.idle":"2022-07-29T10:46:57.454449Z","shell.execute_reply.started":"2022-07-29T10:46:57.442009Z","shell.execute_reply":"2022-07-29T10:46:57.453554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"figure, axes = plt.subplots(1, 2, sharex=True, figsize=(32,5))\nfigure.suptitle('Comparação entre duas variáveis com os p-valor extremos')\naxes[0].set_title(f'Menor pvalue: {var_min_p}')\naxes[1].set_title(f'Maior pvalue: {var_max_p}')\n\nsns.histplot(ax=axes[0], x=var_min_p, data=df_train, kde=True);\nsns.histplot(ax=axes[1], x=var_max_p, data=df_train, kde=True);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:57.458467Z","iopub.execute_input":"2022-07-29T10:46:57.460462Z","iopub.status.idle":"2022-07-29T10:46:58.038228Z","shell.execute_reply.started":"2022-07-29T10:46:57.460427Z","shell.execute_reply":"2022-07-29T10:46:58.037469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Correlação entre registros de dados","metadata":{}},{"cell_type":"code","source":"# Define tamanho da amostra para análise\nn_samples = 50 if FAST_RUN==True else 1000","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:58.039370Z","iopub.execute_input":"2022-07-29T10:46:58.040058Z","iopub.status.idle":"2022-07-29T10:46:58.044989Z","shell.execute_reply.started":"2022-07-29T10:46:58.040014Z","shell.execute_reply":"2022-07-29T10:46:58.044131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Correlação entre as variáveis preditoras\ndf_corr = df_train.sample(n_samples).drop(['ID_code', 'target'], axis=1).T.corr()\ndf_corr = df_corr.values.flatten()\ndf_corr = df_corr[df_corr != 1]\n\nprint(f'Min correlation = {df_corr.min()}')\nprint(f'Max correlation = {df_corr.max()}')\n\nsns.displot(data=df_corr, kde=True, height=8, aspect=2)\nplt.title(\"Correlação entre registros de dados\");","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:58.046815Z","iopub.execute_input":"2022-07-29T10:46:58.047168Z","iopub.status.idle":"2022-07-29T10:46:58.388275Z","shell.execute_reply.started":"2022-07-29T10:46:58.047132Z","shell.execute_reply":"2022-07-29T10:46:58.387526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Apesar de representar uma amostra de dados (por limitação de recurso), este gráfico mostra que os registros possuem correlação entre si.\n\nSerá que os registros fortemente correlacionados possuem mesmo target?","metadata":{}},{"cell_type":"code","source":"df_sample = df_train.sample(n_samples).drop(columns='ID_code')\ndic_target = df_sample['target'].to_dict()\n\ndf_corr = df_sample.drop(['target'], axis=1).T.corr()\ndf_corr['VAR_A'] = df_corr.index\n\ndf_corr = df_corr.melt(id_vars=['VAR_A'], var_name='VAR_B', value_name='CORR')\ndf_corr['PAIR'] = df_corr.apply(lambda row: tuple(sorted([row['VAR_A'], row['VAR_B']])), axis=1)\ndf_corr.drop_duplicates(subset='PAIR', inplace=True)\ndf_corr = df_corr.loc[df_corr['VAR_A']!=df_corr['VAR_B']].copy()\ndf_corr = df_corr[['VAR_A', 'VAR_B', 'CORR']].copy()\ndf_corr.sort_values(by='CORR', ascending=False, inplace=True)\ndf_corr['TARGET_VAR_A'] = df_corr['VAR_A'].apply(lambda x: int(dic_target[x]))\ndf_corr['TARGET_VAR_B'] = df_corr['VAR_B'].apply(lambda x: int(dic_target[x]))\ndf_corr['SAME_TARGET'] = df_corr.apply(lambda row: int(row['TARGET_VAR_A']==row['TARGET_VAR_B']), axis=1)\ndf_corr.reset_index(inplace=True, drop=True)\ndf_corr.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:58.389403Z","iopub.execute_input":"2022-07-29T10:46:58.390304Z","iopub.status.idle":"2022-07-29T10:46:58.486500Z","shell.execute_reply.started":"2022-07-29T10:46:58.390261Z","shell.execute_reply":"2022-07-29T10:46:58.485621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Busca valores para divisão por sessão de valores\ndivs = np.linspace(df_corr['CORR'].min(), df_corr['CORR'].max(), 10)\n\n# Insere no dataframe a sessão a depender do valor de correlação\ndf_corr['SECTION'] = df_corr['CORR'].apply(lambda x: np.argwhere(divs >= x)[0, 0])\n\n# Retira a primeira seção (somente com o menor valor)\ndf_corr = df_corr.loc[df_corr['SECTION']!=0].copy()\n\n# Ajusta nomenclatura\ndf_corr['SECTION'] = df_corr['SECTION'].apply(\n    lambda x:\n    f'{divs[x-1]:.2f} ~ {divs[x]:.2f}' if x>0\n    else f'{df_corr[\"CORR\"].min():.2f} ~ {divs[x]:.2f}'      \n)\n\n# Calcula o percentual de valores SAME_TARGET por seção\ndf_piv = df_corr.copy()\ndf_piv['AUX'] = 1\ndf_piv = df_piv.pivot_table(values='AUX', columns='SAME_TARGET', index='SECTION', fill_value=0, aggfunc=np.sum).reset_index()\ndf_piv['PERC_SAME_TARGET'] = df_piv[1] / df_piv[[0, 1]].sum(axis=1)\n\n\n# Plota o percentual de valores SAME_TARGET por seção\nsns.set(rc={'figure.figsize':(32, 8)})\nsns.barplot(x='SECTION', y='PERC_SAME_TARGET', data = df_piv)\nplt.title('Percentual de registros com o mesmo target por faixa de correlação')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:58.488174Z","iopub.execute_input":"2022-07-29T10:46:58.488537Z","iopub.status.idle":"2022-07-29T10:46:58.782255Z","shell.execute_reply.started":"2022-07-29T10:46:58.488499Z","shell.execute_reply":"2022-07-29T10:46:58.781507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Era esperado que o percentual de registros com o mesmo target aumente à medida que a correlação aumenta. No entanto, pode-se observar que mesmo com correlação forte existe um percentual razoável de registros com target diferente (percentual > 10%).","metadata":{}},{"cell_type":"code","source":"# Busca os registros com maior correlação e target diferente\nvar_a, var_b, corr = df_corr.loc[df_corr['SAME_TARGET']==0].iloc[0][['VAR_A', 'VAR_B', 'CORR']]\ndf_plot = df_train.loc[df_train.index.isin([var_a, var_b])].drop(columns=['target', 'ID_code'])\ndf_plot['IDX'] = df_plot.index\n\n# Plota em linhas\nfig, ax = plt.subplots()\nax_line = sns.lineplot(x='variable', y='value', hue='IDX', data=df_plot.melt(id_vars='IDX'), palette=\"flare\");\nax.set(xlabel=None)\nax.set(xticklabels=[])\nfig.set_size_inches(32, 8)\nplt.title(f\"Registros com maior correlação (corr={corr:.2f}) e target diferente\");","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:58.783617Z","iopub.execute_input":"2022-07-29T10:46:58.784192Z","iopub.status.idle":"2022-07-29T10:46:59.777338Z","shell.execute_reply.started":"2022-07-29T10:46:58.784153Z","shell.execute_reply":"2022-07-29T10:46:59.776611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Analisando a coluna `ID_Code`","metadata":{}},{"cell_type":"code","source":"# Verifica se ID_Code representa a mesma informação do índice\ndf_id_code = df_train[['ID_code']].copy()\ndf_id_code['ID'] = df_id_code['ID_code'].apply(lambda x: int(x.split('_')[1]))\ndf_id_code['COMP'] = (df_id_code['ID'] == df_id_code.index)\ndf_id_code['COMP'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:59.778764Z","iopub.execute_input":"2022-07-29T10:46:59.779754Z","iopub.status.idle":"2022-07-29T10:46:59.824680Z","shell.execute_reply.started":"2022-07-29T10:46:59.779708Z","shell.execute_reply":"2022-07-29T10:46:59.823965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Removendo coluna ID_code por não trazer informação relevante de acordo com análise acima\ndf_train.drop(columns='ID_code', inplace=True)\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:59.829638Z","iopub.execute_input":"2022-07-29T10:46:59.832257Z","iopub.status.idle":"2022-07-29T10:46:59.882371Z","shell.execute_reply.started":"2022-07-29T10:46:59.832226Z","shell.execute_reply":"2022-07-29T10:46:59.881577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizando distribuição das variáveis","metadata":{}},{"cell_type":"code","source":"# Criando dataframe com valores padronizados\ndf_train_scale = df_train.drop(columns='target')\nscale_model = MinMaxScaler().fit(df_train_scale)\nscale_values = scale_model.transform(df_train_scale)\ndf_train_scale = pd.DataFrame(columns=df_train_scale.columns, data=scale_values)\ndf_train_scale['target'] = df_train['target'].values\ndf_train_scale.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:59.887084Z","iopub.execute_input":"2022-07-29T10:46:59.889439Z","iopub.status.idle":"2022-07-29T10:46:59.971198Z","shell.execute_reply.started":"2022-07-29T10:46:59.889401Z","shell.execute_reply":"2022-07-29T10:46:59.970362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Criando dataframe com valores normalizados\ndf_train_norm = df_train.drop(columns='target')\nnorm_model = StandardScaler().fit(df_train_norm)\nnorm_values = norm_model.transform(df_train_norm)\ndf_train_norm = pd.DataFrame(columns=df_train_norm.columns, data=norm_values)\ndf_train_norm['target'] = df_train['target'].values\ndf_train_norm.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:46:59.976117Z","iopub.execute_input":"2022-07-29T10:46:59.978632Z","iopub.status.idle":"2022-07-29T10:47:00.064973Z","shell.execute_reply.started":"2022-07-29T10:46:59.978591Z","shell.execute_reply":"2022-07-29T10:47:00.064088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Comparando a variância das variáveis\ndf_melt = df_train_scale.copy()\ndf_melt.drop(columns='target', inplace=True)\ncols_melt = df_melt.columns.tolist()\ndf_melt = df_melt.melt(var_name='VAR', value_vars=cols_melt, value_name='VALUE')\n\n# Plotando\nfig, ax = plt.subplots()\nax_bar = sns.boxplot(x='VAR', y='VALUE', data=df_melt);\nax.set(xlabel=None)\nax.set(xticklabels=[])\nfig.set_size_inches(32, 8)\nplt.title(\"Variância dos dados padronizados\");","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:00.069671Z","iopub.execute_input":"2022-07-29T10:47:00.072200Z","iopub.status.idle":"2022-07-29T10:47:04.559432Z","shell.execute_reply.started":"2022-07-29T10:47:00.072131Z","shell.execute_reply":"2022-07-29T10:47:04.558505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Comparando a variância das variáveis\ndf_melt = df_train_norm.copy()\ndf_melt.drop(columns='target', inplace=True)\ncols_melt = df_melt.columns.tolist()\ndf_melt = df_melt.melt(var_name='VAR', value_vars=cols_melt, value_name='VALUE')\n\n# Plotando\nfig, ax = plt.subplots()\nax_bar = sns.boxplot(x='VAR', y='VALUE', data=df_melt);\nax.set(xlabel=None)\nax.set(xticklabels=[])\nfig.set_size_inches(32, 8)\nplt.title(\"Variância dos dados normalizados\");","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:04.561414Z","iopub.execute_input":"2022-07-29T10:47:04.562075Z","iopub.status.idle":"2022-07-29T10:47:09.026725Z","shell.execute_reply.started":"2022-07-29T10:47:04.562031Z","shell.execute_reply":"2022-07-29T10:47:09.025922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aparentemente as variáveis possuem comportamento parecidos (principalmente nos quartis superiores e inferiores) após normalização.","metadata":{}},{"cell_type":"code","source":"# Verificando variância acumulada dos dados padronizados\npca_model = PCA(n_components=0.9999).fit(df_train_scale.drop(columns='target'))\npca_model.explained_variance_ratio_\ndf_pca_scale = pd.DataFrame(columns=['ACM_VARIANCE'], data=pca_model.explained_variance_ratio_)\ndf_pca_scale['ACM_VARIANCE'] = df_pca_scale['ACM_VARIANCE'].cumsum()\ndf_pca_scale['COMPONENT'] = df_pca_scale.index\n\n# Verificando variância acumulada dos dados normalizados\npca_model = PCA(n_components=0.9999).fit(df_train_norm.drop(columns='target'))\npca_model.explained_variance_ratio_\ndf_pca_norm = pd.DataFrame(columns=['ACM_VARIANCE'], data=pca_model.explained_variance_ratio_)\ndf_pca_norm['ACM_VARIANCE'] = df_pca_norm['ACM_VARIANCE'].cumsum()\ndf_pca_norm['COMPONENT'] = df_pca_norm.index\n\n# Concatena\ndf_pca_scale['TYPE'] = 'MinMaxScaler'\ndf_pca_norm['TYPE'] = 'StandardScaler'\ndf_pca = pd.concat([df_pca_scale, df_pca_norm], sort=True, ignore_index=True)\ndf_pca.sort_values(by='COMPONENT', inplace=True)\n\n# Plotando\nfig, ax = plt.subplots()\nax.set_title('Variância explicada acumulada por componente')\nax_bar = sns.lineplot(x='COMPONENT', y='ACM_VARIANCE', hue='TYPE', data=df_pca, ax=ax);\nfig.set_size_inches(20, 8)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:09.027917Z","iopub.execute_input":"2022-07-29T10:47:09.028595Z","iopub.status.idle":"2022-07-29T10:47:09.810385Z","shell.execute_reply.started":"2022-07-29T10:47:09.028552Z","shell.execute_reply":"2022-07-29T10:47:09.809417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Comparando variância explicada PCA dos dados transformados por MinMaxScaler e por StandardScaler. Aqui mais um reforço para a hipótese de os dados terem sido gerados por algum processo de compactação. O crescimento da variância explicada segue um padrão aparentemente linear (StandardScaler) / quase linear (MinMaxScaler).","metadata":{}},{"cell_type":"markdown","source":"## Visualizando dados em dimensão reduzida","metadata":{}},{"cell_type":"code","source":"pca_model = PCA(n_components=2).fit(df_train_scale.drop(columns='target'))\npca_values = pca_model.transform(df_train_scale.drop(columns='target'))\ndf_pca = pd.DataFrame(columns=['D1', 'D2'], data=pca_values)\ndf_pca['target'] = df_train_scale['target'].values\nsns.set(rc={'figure.figsize':(20, 8)})\nsns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:09.811547Z","iopub.execute_input":"2022-07-29T10:47:09.812407Z","iopub.status.idle":"2022-07-29T10:47:10.682891Z","shell.execute_reply.started":"2022-07-29T10:47:09.812362Z","shell.execute_reply":"2022-07-29T10:47:10.682171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_model = PCA(n_components=2).fit(df_train_norm.drop(columns='target'))\npca_values = pca_model.transform(df_train_norm.drop(columns='target'))\ndf_pca = pd.DataFrame(columns=['D1', 'D2'], data=pca_values)\ndf_pca['target'] = df_train_norm['target'].values\nsns.set(rc={'figure.figsize':(20, 8)})\nsns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:10.684207Z","iopub.execute_input":"2022-07-29T10:47:10.684808Z","iopub.status.idle":"2022-07-29T10:47:11.475193Z","shell.execute_reply.started":"2022-07-29T10:47:10.684768Z","shell.execute_reply":"2022-07-29T10:47:11.474525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Como visualmente a transformação StandardScaler realiza uma melhor separação entre as classes, esta será a transformação utilizada antes de submeter os dados aos modelos de classificação.","metadata":{}},{"cell_type":"code","source":"# Busca valores para divisão por sessão de valores\ndivs = np.linspace(df_pca['D1'].min(), df_pca['D1'].max(), 10)\n\n# Insere no dataframe pca a sessão a depender do valor da dimensão D1\ndf_pca['SECTION'] = df_pca['D1'].apply(lambda x: np.argwhere(divs >= x)[0, 0])\n\n# Calcula o percentual de valores 1 por seção\ndf_piv = df_pca.copy()\ndf_piv['AUX'] = 1\ndf_piv = df_piv.pivot_table(values='AUX', columns='target', index='SECTION', fill_value=0, aggfunc=np.sum).reset_index()\ndf_piv['PERC_1'] = df_piv[1] / df_piv[[0, 1]].sum(axis=1)\n\n# Plota o PCA\nsns.set(rc={'figure.figsize':(20, 8)})\nsns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5)\nfor div in divs:\n    plt.axvline(div, color='red')\n    plt.text(div, df_pca['D2'].min(), f'SECTION={list(divs).index(div)}', horizontalalignment='left', size='medium', color='black', weight='semibold')\nplt.title('PCA em duas dimensões segmentado por valor no eixo X')\nplt.legend(loc='upper right')\nplt.show();\n\n# Plota o percentual de valores = 1 por seção\nsns.set(rc={'figure.figsize':(20, 4)})\nsns.barplot(x='SECTION', y='PERC_1', data = df_piv)\nplt.title('Percentual de target=1 por cada segmento da visualização PCA')\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:11.476624Z","iopub.execute_input":"2022-07-29T10:47:11.477494Z","iopub.status.idle":"2022-07-29T10:47:12.487304Z","shell.execute_reply.started":"2022-07-29T10:47:11.477456Z","shell.execute_reply":"2022-07-29T10:47:12.486549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualizando em duas dimensões, o componente D1 possui uma boa influência na separação das classes.","metadata":{}},{"cell_type":"markdown","source":"# Reduzindo dataset de treino para ganho de performance\n\nÁrea destinada para reduzir o tamanho do dataset na fase de testagem de código","metadata":{}},{"cell_type":"code","source":"# Lê o arquivo de dados\ndf_train = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/train.csv')\ndf_train.drop(columns='ID_code', inplace=True)\nif FAST_RUN == True:\n    df_train, _ = train_test_split(df_train, stratify=df_train['target'], train_size=10000, random_state=1)\nelse:\n    df_train, _ = train_test_split(df_train, stratify=df_train['target'], train_size=100_000, random_state=1)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:12.488773Z","iopub.execute_input":"2022-07-29T10:47:12.489460Z","iopub.status.idle":"2022-07-29T10:47:19.202330Z","shell.execute_reply.started":"2022-07-29T10:47:12.489421Z","shell.execute_reply":"2022-07-29T10:47:19.200791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Criando modelos de classificação","metadata":{}},{"cell_type":"markdown","source":"Para a execução dos modelos de classificação, foram utilizadas as seguintes estratégias:\n\n1. **Separação de dados entre treinamento e teste**\n\n    O dataset original `df_train` é processado na função `split_train_test_idx`. Esta função retorna os índices das linhas que serão usadas para treinamento e para validação. Esta divisão é feita usando o parâmetro `stratify=df_train['target']` garantindo assim que o balanceamento entre as classes 0 e 1, existente no dataset original, será mantido nos dois conjuntos criados (treino e validação)\n    \n2. **Execução dos modelos de classificação**\n\n    Embora a tarefa seja de classificação, a avaliação é feita em métrica de probabilidade (ROC - AUC). Então é necessário utilizar algoritmos que tenham capacidade de fornecer uma probablidade e não apenas a classe.\n    \n    Foram testados os seguintes algoritmos:\n    \n    - GradientBoostingClassifier: motivado por ser um conhecido algoritmo do tipo `Boosting`. Optei por incluir algoritmos caracterizados pela metodologia boosting por particularmente acreditar que conseguem obter melhores níveis de acerto. A utilização deste algoritmo estava levando tempo considerável para execução e por este motivo sua execução está sendo ignorada (código comentado).\n    - LogisticRegression: motivado por ser um dos algoritmos mais conhecidos para classificação. Este algoritmo realiza a classificação a partir de função que indica um percentual de chance de um registro pertencer a cada classe.\n    - LGBMClassifier: escolhido devido a ser um algoritmo do tipo `Boosting` e que foi utilizado em vários notebooks que exploraram este mesmo problema. Conforme informado em [5], este algoritmo se destaca por ter um tempo de processamento menor (_\"Light GBM is prefixed as ‘Light’ because of its high speed. Light GBM can handle the large size of data and takes lower memory to run\"_) e por este motivo veio a ser uma alternativa ao GradientBoostingClassifier para este trabalho.\n    - RandomForest: escolhido por ser um algoritmo bem conhecido para tarefas de classificação. No entanto, devido ao mesmo problema de tempo processamento encontrado no algoritmo GradientBoostingClassifier, este foi ignorado no processamento deste código.\n\n\n3. **Validação cruzada e variação de parâmetros**\n\n    Para cada algoritmo, os dados de treinamento foram submetidos à estratégia de validação cruzada, utilizando 5 conjuntos. Desta forma, a cada \"rodada\" 1 desses 5 conjuntos é utlizado como conjunto de validação, evitando assim que o modelo caia no problema de realizar overfitting no conjunto de validação (no caso de usar apenas 1 conjunto de validação).\n    \n    Os parâmetros de execução dos algoritmos foram variados buscando otimizar os resultados. Esta variação foi feita com auxílio da funcionalidade GridSearchCV. A escolha dos valores dos parâmetros foi feita utilizando como primeira opção o valor padrão do parâmetro, seguido de um valor inferior e outro valor superior, como neste exemplo: `'n_estimators': [100, 50, 200]`, onde 100 é o valor padrão, 50 é um valor inferior ao padrão e 200 um valor superior ao padrão.\n    \n4. **Tratamento do desbalanceamento de classes**\n\n    Para o algoritmo LogisticRegression, foi definido o parâmetro `class_weight` com o valor `balanced`. Seguem trechos da documentação [6], que indicam o uso desse parâmetro para o caso de ajustar os pesos do modelo de forma balanceada, evitando assim priorizar uma classe com mais dados.\n    \n    - \"_Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one._\"\n    - \"_The “balanced” mode uses the values of y to **automatically adjust weights inversely proportional to class frequencies** in the input data as n_samples / (n_classes * np.bincount(y))._\"\n\n    Para o algoritmo LGBMClassifier, este ajuste é feito pelo parâmetro `is_unbalance`, conforme indicado na documentação [7]: \"_set this to true if training data are unbalanced_\".","metadata":{}},{"cell_type":"code","source":"def split_train_test_idx(df_train):\n    \"\"\"\n    Separa os índices de dados que serão usados para treinamento e teste.\n    O objetivo é garantir que os mesmos dados de treino e teste sejam usados para as diversas rodadas de teste com modificação de features.\n    \"\"\"\n    # Seleciona uma amostra dos dados para treinamento e teste\n    df_train_sample, df_test_sample = train_test_split(df_train, stratify=df_train['target'], train_size=0.8, random_state=1)\n    \n    # Retorno\n    return df_train_sample.index, df_test_sample.index","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:19.203538Z","iopub.execute_input":"2022-07-29T10:47:19.203946Z","iopub.status.idle":"2022-07-29T10:47:19.219495Z","shell.execute_reply.started":"2022-07-29T10:47:19.203908Z","shell.execute_reply":"2022-07-29T10:47:19.218745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def run_classifiers(df_train_sample, df_test_sample, use_ros=False):\n    \"\"\"\n    Executa os classificadores nos dados recebidos e retorna o que tiver melhor métrica.\n    O parâmetro user_ros define se o balanceamento dos dados será feito na separação dos dados (treinamento e teste)\n    \"\"\"\n\n    # Separa entrada e saída\n    if use_ros:\n        x_train, y_train = ROS().fit_resample(df_train_sample.drop(columns='target'), df_train_sample['target'])\n    \n    else:\n        x_train = df_train_sample.drop(columns='target')\n        y_train = df_train_sample['target']\n\n    x_test = df_test_sample.drop(columns='target')\n    y_test = df_test_sample['target']\n    \n    # LogisticRegression\n    params_lr = {\n#         'penalty': ['l2', 'none'],\n        'penalty': ['none'],\n#         'tol': [1e-3, 1e-4, 1e-2],\n        'tol': [1e-2],\n#         'class_weight': ['balanced', None],\n        'class_weight': ['balanced'] \n    }\n    lr_model = GridSearchCV(LogisticRegression(max_iter=5000), params_lr, cv=5, scoring='roc_auc', n_jobs=-1).fit(x_train, y_train)\n\n\n    # RandomForest\n    params_rf = { \n        'n_estimators': [100, 50, 200],\n        'max_features': ['auto', 'sqrt', 'log2'],\n        'max_depth' : [None, 15, 20],\n        'class_weight': [None, 'balanced']\n    }\n#     rf_model = GridSearchCV(RandomForestClassifier(), params_rf, cv=5, scoring='roc_auc', n_jobs=-1).fit(x_train, y_train)\n\n    # GradientBoostingClassifier\n    params_gbc = {\n        \"learning_rate\": [0.1, 0.05, 0.15],\n        'max_depth' : [None, 15, 20],\n        'max_features': ['auto', 'sqrt', 'log2'],\n        \"subsample\":[0.8, 1.0],\n        'n_estimators': [100, 50, 200],\n        }\n#     gbc_model = GridSearchCV(GradientBoostingClassifier(), params_gbc, cv=5, scoring='roc_auc', n_jobs=-1).fit(x_train, y_train)\n    \n\n    # LGBMClassifier\n    param_lgbm = {\n        'device_type': ['gpu'],\n        'is_unbalance': [True],\n        'max_depth' : [None],\n#         'max_depth' : [None, 5, 15],\n        'metric': ['auc'],\n        'num_leaves': [60],\n#         'num_leaves': [31, 60, 20],\n        'objective': ['binary'],\n#         'reg_alpha': [0.1, 5, 10],\n#         'reg_lambda':  [0.1, 5, 10],\n        'reg_alpha': [0.1],\n        'reg_lambda':  [0.1],\n#         'learning_rate': [0.1, 0.05, 0.15],\n        'learning_rate': [0.15],\n        'is_unbalance': [True],\n        }\n    if FAST_RUN: param_lgbm = {}\n    lgbm_model = GridSearchCV(LGBMClassifier(objective = \"binary\"), param_lgbm, cv=5, scoring='roc_auc', n_jobs=-1).fit(x_train, y_train)\n\n    # Seleciona o melhor modelo\n    models_to_evaluate = [\n#         rf_model,\n#         gbc_model,\n        lgbm_model,\n        lr_model,\n    ]\n    best_score, best_model = 0, None\n    for model in models_to_evaluate:\n        score = roc_auc_score(y_true=y_test, y_score=model.predict_proba(x_test)[:, 1])\n        print(f'{model.best_estimator_}: {score}\\n')\n        if score > best_score:\n            best_score, best_model = score, model.best_estimator_\n            \n    # Retorno\n    return best_model, best_score","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:19.222054Z","iopub.execute_input":"2022-07-29T10:47:19.223040Z","iopub.status.idle":"2022-07-29T10:47:19.254268Z","shell.execute_reply.started":"2022-07-29T10:47:19.222997Z","shell.execute_reply":"2022-07-29T10:47:19.253344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Rodando modelos de classificação","metadata":{}},{"cell_type":"markdown","source":"## Sorteia índices para treino e teste\n\nSepara os índices que serão usados para treinamento e teste. Estes índices serão usados pelos diversos modelos testados, garantindo assim que todos os modelos vão utilizar os mesmos dados de treinamento e de teste.\n\nNa divisão dos dados, é mantido o balanceamento original da classe de saída em ambos os conjuntos.","metadata":{}},{"cell_type":"code","source":"list_idx_train, list_idx_test = split_train_test_idx(df_train)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:19.256207Z","iopub.execute_input":"2022-07-29T10:47:19.256810Z","iopub.status.idle":"2022-07-29T10:47:19.288556Z","shell.execute_reply.started":"2022-07-29T10:47:19.256772Z","shell.execute_reply":"2022-07-29T10:47:19.287750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelo 1 - Realiza classificação com os dados originais","metadata":{}},{"cell_type":"code","source":"# print(\"Rodando modelo v1\")\n\n# # Define os dados para a rodada\n# df_v1 = df_train.copy()\n\n# # Busca os dados de treinamento e teste\n# df_train_v1 = df_v1.loc[list_idx_train]\n# df_test_v1 = df_v1.loc[list_idx_test]\n\n# # Verifica o resultado da classificação\n# model_v1, score_v1 = run_classifiers(df_train_v1, df_test_v1)\n# model_v1_ros, score_v1_ros = run_classifiers(df_train_v1, df_test_v1, use_ros=True)\n# df_v1, df_train_v1, df_test_v1 = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:19.290008Z","iopub.execute_input":"2022-07-29T10:47:19.291214Z","iopub.status.idle":"2022-07-29T10:47:19.297840Z","shell.execute_reply.started":"2022-07-29T10:47:19.291175Z","shell.execute_reply":"2022-07-29T10:47:19.297093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Lê os dados de teste, separa entrada e infere a saída\n# df_test = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/test.csv')\n# x_test = df_test.drop(columns=['ID_code'])\n# y_test = model_v1.predict_proba(x_test)\n# y_test_ros = model_v1_ros.predict_proba(x_test)\n\n# # Gera o dataframe submission\n# df_submission = df_test[['ID_code']].copy()\n# df_submission['target'] = y_test[:, 1]\n# df_submission.to_csv('submission_v1.csv', index=False)\n# df_submission['target'] = y_test_ros[:, 1]\n# df_submission.to_csv('submission_v1_ros.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:19.300241Z","iopub.execute_input":"2022-07-29T10:47:19.301719Z","iopub.status.idle":"2022-07-29T10:47:19.309963Z","shell.execute_reply.started":"2022-07-29T10:47:19.301668Z","shell.execute_reply":"2022-07-29T10:47:19.309202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelo 2 - Realiza classificação com os dados normalizados","metadata":{}},{"cell_type":"code","source":"print(\"Rodando modelo v2\")\n\n# Define os dados para a rodada\ndf_v2 = df_train.copy()\n\n# Busca os dados de treinamento e teste\ndf_train_v2 = df_v2.loc[list_idx_train]\ndf_test_v2 = df_v2.loc[list_idx_test]\n\n# Define o modelo de normalização\ndf_model_norm = df_train_v2.drop(columns='target')\nnorm_model = StandardScaler().fit(df_model_norm)\n\n# Gera os dados de treinamento normalizados\nnorm_values = norm_model.transform(df_train_v2.drop(columns='target'))\ndf_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ndf_aux['target'] = df_train_v2['target'].values\ndf_train_v2 = df_aux.copy()\n\n# Gera os dados de teste normalizados\nnorm_values = norm_model.transform(df_test_v2.drop(columns='target'))\ndf_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ndf_aux['target'] = df_test_v2['target'].values\ndf_test_v2 = df_aux.copy()\n\n# Verifica o resultado da classificação\nmodel_v2, score_v2 = run_classifiers(df_train_v2, df_test_v2)\nmodel_v2_ros, score_v2_ros = run_classifiers(df_train_v2, df_test_v2, use_ros=True)\n# df_v2, df_train_v2, df_test_v2 = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:47:19.311088Z","iopub.execute_input":"2022-07-29T10:47:19.312483Z","iopub.status.idle":"2022-07-29T10:48:05.402549Z","shell.execute_reply.started":"2022-07-29T10:47:19.312446Z","shell.execute_reply":"2022-07-29T10:48:05.401535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lê os dados de teste, separa entrada, normaliza valores e infere saída\ndf_test = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/test.csv')\nx_test = df_test.drop(columns=['ID_code'])\nnorm_values = norm_model.transform(x_test)\nx_test = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ny_test = model_v2.predict_proba(x_test)\ny_test_ros = model_v2_ros.predict_proba(x_test)\n\n# Gera o dataframe submission\ndf_submission = df_test[['ID_code']].copy()\ndf_submission['target'] = y_test[:, 1]\ndf_submission.to_csv('submission_v2.csv', index=False)\ndf_submission['target'] = y_test_ros[:, 1]\ndf_submission.to_csv('submission_v2_ros.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:05.404208Z","iopub.execute_input":"2022-07-29T10:48:05.406198Z","iopub.status.idle":"2022-07-29T10:48:17.173581Z","shell.execute_reply.started":"2022-07-29T10:48:05.406153Z","shell.execute_reply":"2022-07-29T10:48:17.172742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelo 3 - Analisando relevância das features com base no `model_v2`","metadata":{}},{"cell_type":"markdown","source":"Neste modelo, as features mais relevantes serão utilizadas para criação de novas features com base em transformação polinomial.","metadata":{}},{"cell_type":"code","source":"# print(\"Rodando modelo v3\")\n\n# # Relevância das features\n# try:\n#     feat_importance = abs(model_v2.coef_[0])\n# except:\n#     feat_importance = model_v2.feature_importances_\n# df_feat = pd.DataFrame(columns=x_test.columns, data=feat_importance.reshape(1, -1))\n# df_feat = df_feat.melt(var_name='VAR', value_name='VALUE')","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.174773Z","iopub.execute_input":"2022-07-29T10:48:17.175133Z","iopub.status.idle":"2022-07-29T10:48:17.180164Z","shell.execute_reply.started":"2022-07-29T10:48:17.175096Z","shell.execute_reply":"2022-07-29T10:48:17.179389Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Plota a relevância das features\n# sns.set(rc={'figure.figsize':(32, 8)})\n# ax = sns.barplot(x='VAR', y='VALUE', data=df_feat)\n# for item in ax.get_xticklabels():\n#     item.set_rotation(90)\n# plt.title('Relevância das features com base no classificador')\n# plt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.181563Z","iopub.execute_input":"2022-07-29T10:48:17.182102Z","iopub.status.idle":"2022-07-29T10:48:17.192509Z","shell.execute_reply.started":"2022-07-29T10:48:17.182064Z","shell.execute_reply":"2022-07-29T10:48:17.191771Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Seleciona as features mais importantes (as que correspondem ao limiar percentual)\n# threshold_quantile = 0.95\n# threshold_value = df_feat['VALUE'].quantile(threshold_quantile)\n# df_sel_feat = df_feat.loc[df_feat['VALUE'] >= threshold_value].copy()\n# df_sel_feat.sort_values(by=['VALUE'], ascending=False, inplace=True)\n\n# # Plota \n# sns.set(rc={'figure.figsize':(32, 4)})\n# ax = sns.barplot(x='VAR', y='VALUE', data=df_sel_feat)\n# for item in ax.get_xticklabels():\n#     item.set_rotation(90)\n# plt.title(f'{df_sel_feat.shape[0]} features selecionadas usando o percentil {threshold_quantile}')\n# plt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.193755Z","iopub.execute_input":"2022-07-29T10:48:17.194155Z","iopub.status.idle":"2022-07-29T10:48:17.202421Z","shell.execute_reply.started":"2022-07-29T10:48:17.194120Z","shell.execute_reply":"2022-07-29T10:48:17.201445Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cria features com base em transformação polinomial","metadata":{}},{"cell_type":"code","source":"# # Cria dataframe com as features selecionadas e realiza transformação polinomial\n# sel_features = df_sel_feat['VAR'].tolist()[:30] # Limita ao mãximo de 30 features\n# df_train_feat = df_train[sel_features].copy()\n# poly_model = PolynomialFeatures(include_bias=False).fit(df_train_feat)\n# df_feat = pd.DataFrame(columns=poly_model.get_feature_names_out(), data=poly_model.transform(df_train_feat))\n# df_feat.index = df_train.index\n\n# # Concatena com o dataframe principal\n# df_v3 = pd.concat([df_train.drop(columns=sel_features), df_feat], axis=1)\n# df_v3.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.204687Z","iopub.execute_input":"2022-07-29T10:48:17.205318Z","iopub.status.idle":"2022-07-29T10:48:17.212831Z","shell.execute_reply.started":"2022-07-29T10:48:17.205281Z","shell.execute_reply":"2022-07-29T10:48:17.212039Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Criando dataframe com valores normalizados\n# df_train_norm = df_v3.drop(columns='target')\n# norm_model = StandardScaler().fit(df_train_norm)\n# norm_values = norm_model.transform(df_train_norm)\n# df_train_norm = pd.DataFrame(columns=df_train_norm.columns, data=norm_values)\n# df_train_norm['target'] = df_v3['target'].values\n# df_train_norm.head()\n\n# # Plotando visualização\n# pca_model = PCA(n_components=2).fit(df_train_norm.drop(columns='target'))\n# pca_values = pca_model.transform(df_train_norm.drop(columns='target'))\n# df_pca = pd.DataFrame(columns=['D1', 'D2'], data=pca_values)\n# df_pca['target'] = df_train_norm['target'].values\n# sns.set(rc={'figure.figsize':(20, 8)})\n# sns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.214139Z","iopub.execute_input":"2022-07-29T10:48:17.214557Z","iopub.status.idle":"2022-07-29T10:48:17.223241Z","shell.execute_reply.started":"2022-07-29T10:48:17.214514Z","shell.execute_reply":"2022-07-29T10:48:17.222404Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Define os dados para a rodada\n# df_v3 = df_train.copy()\n\n# # Busca os dados de treinamento e teste\n# df_train_v3 = df_v3.loc[list_idx_train]\n# df_test_v3 = df_v3.loc[list_idx_test]\n\n# # Realiza as transformações polinomiais para treinamento\n# df_feat = pd.DataFrame(columns=poly_model.get_feature_names_out(), data=poly_model.transform(df_train_v3[sel_features]))\n# df_feat.index = df_train_v3.index\n# df_train_v3 = pd.concat([df_train_v3.drop(columns=sel_features), df_feat], axis=1)\n\n# # Realiza as transformações polinomiais para teste\n# df_feat = pd.DataFrame(columns=poly_model.get_feature_names_out(), data=poly_model.transform(df_test_v3[sel_features]))\n# df_feat.index = df_test_v3.index\n# df_test_v3 = pd.concat([df_test_v3.drop(columns=sel_features), df_feat], axis=1)\n\n# # Define o modelo de normalização\n# df_model_norm = df_train_v3.drop(columns='target')\n# norm_model = StandardScaler().fit(df_model_norm)\n\n# # Gera os dados de treinamento normalizados\n# norm_values = norm_model.transform(df_train_v3.drop(columns='target'))\n# df_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# df_aux['target'] = df_train_v3['target'].values\n# df_train_v3 = df_aux.copy()\n\n# # Gera os dados de teste normalizados\n# norm_values = norm_model.transform(df_test_v3.drop(columns='target'))\n# df_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# df_aux['target'] = df_test_v3['target'].values\n# df_test_v3 = df_aux.copy()\n\n# # Verifica o resultado da classificação\n# model_v3, score_v3 = run_classifiers(df_train_v3, df_test_v3)\n# model_v3_ros, score_v3_ros = run_classifiers(df_train_v3, df_test_v3, use_ros=True)\n# df_v3, df_train_v3, df_test_v3 = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.225790Z","iopub.execute_input":"2022-07-29T10:48:17.226164Z","iopub.status.idle":"2022-07-29T10:48:17.236829Z","shell.execute_reply.started":"2022-07-29T10:48:17.226130Z","shell.execute_reply":"2022-07-29T10:48:17.236157Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Lê os dados de teste\n# df_test = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/test.csv')\n# x_test = df_test.drop(columns=['ID_code'])\n\n# # Realiza as transformações polinomiais para treinamento\n# df_feat = pd.DataFrame(columns=poly_model.get_feature_names_out(), data=poly_model.transform(x_test[sel_features]))\n# df_feat.index = x_test.index\n# x_test = pd.concat([x_test.drop(columns=sel_features), df_feat], axis=1)\n\n# # Normaliza os valores\n# norm_values = norm_model.transform(x_test)\n# x_test = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# y_test = model_v3.predict_proba(x_test)\n# y_test_ros = model_v3_ros.predict_proba(x_test)\n\n# # Gera o dataframe submission\n# df_submission = df_test[['ID_code']].copy()\n# df_submission['target'] = y_test[:, 1]\n# df_submission.to_csv('submission_v3.csv', index=False)\n# df_submission['target'] = y_test_ros[:, 1]\n# df_submission.to_csv('submission_v3_ros.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.238409Z","iopub.execute_input":"2022-07-29T10:48:17.238762Z","iopub.status.idle":"2022-07-29T10:48:17.250097Z","shell.execute_reply.started":"2022-07-29T10:48:17.238728Z","shell.execute_reply":"2022-07-29T10:48:17.249358Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelo 4 - Separando a parte decimal das features","metadata":{}},{"cell_type":"markdown","source":"Este modelo duplica cada variável de entrada e mantém nessas variáveis apenas a parte decimal do valor.","metadata":{}},{"cell_type":"code","source":"# print(\"Rodando modelo v4\")\n\n# # Define os dados para a rodada\n# df_v4 = df_train.copy()\n\n# # Separando parte decimal e parte inteira\n# df_dec = df_v4.drop(columns='target')\n# df_dec.columns = [c+'_dec' for c in df_dec.columns]\n# for col in tqdm(df_dec.columns):\n#     df_dec[col] = df_dec[col].apply(lambda x: x - int(x))\n\n# # Concatena com o dataframe principal\n# df_v4 = pd.concat([df_v4, df_dec], axis=1)\n\n# # Busca os dados de treinamento e teste\n# df_train_v4 = df_v4.loc[list_idx_train]\n# df_test_v4 = df_v4.loc[list_idx_test]\n\n# # Define o modelo de normalização\n# df_model_norm = df_train_v4.drop(columns='target')\n# norm_model = StandardScaler().fit(df_model_norm)\n\n# # Gera os dados de treinamento normalizados\n# norm_values = norm_model.transform(df_train_v4.drop(columns='target'))\n# df_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# df_aux['target'] = df_train_v4['target'].values\n# df_train_v4 = df_aux.copy()\n\n# # Gera os dados de teste normalizados\n# norm_values = norm_model.transform(df_test_v4.drop(columns='target'))\n# df_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# df_aux['target'] = df_test_v4['target'].values\n# df_test_v4 = df_aux.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.251550Z","iopub.execute_input":"2022-07-29T10:48:17.251974Z","iopub.status.idle":"2022-07-29T10:48:17.261892Z","shell.execute_reply.started":"2022-07-29T10:48:17.251939Z","shell.execute_reply":"2022-07-29T10:48:17.261093Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Plotando visualização\n# pca_model = PCA(n_components=2).fit(df_train_v4.drop(columns='target'))\n# pca_values = pca_model.transform(df_train_v4.drop(columns='target'))\n# df_pca = pd.DataFrame(columns=['D1', 'D2'], data=pca_values)\n# df_pca['target'] = df_train_v4['target'].values\n# sns.set(rc={'figure.figsize':(20, 8)})\n# sns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.263235Z","iopub.execute_input":"2022-07-29T10:48:17.263626Z","iopub.status.idle":"2022-07-29T10:48:17.274795Z","shell.execute_reply.started":"2022-07-29T10:48:17.263590Z","shell.execute_reply":"2022-07-29T10:48:17.274121Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Verifica o resultado da classificação\n# model_v4, score_v4 = run_classifiers(df_train_v4, df_test_v4)\n# model_v4_ros, score_v4_ros = run_classifiers(df_train_v4, df_test_v4, use_ros=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.276572Z","iopub.execute_input":"2022-07-29T10:48:17.276929Z","iopub.status.idle":"2022-07-29T10:48:17.284006Z","shell.execute_reply.started":"2022-07-29T10:48:17.276893Z","shell.execute_reply":"2022-07-29T10:48:17.283258Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Relevância das features\n# try:\n#     feat_importance = abs(model_v4.coef_[0])\n# except:\n#     feat_importance = model_v4.feature_importances_\n# df_feat = pd.DataFrame(columns=df_train_v4.drop(columns='target').columns, data=feat_importance.reshape(1, -1))\n# df_feat = df_feat.melt(var_name='VAR', value_name='VALUE')\n\n# # Verifica em que posição a primeira fetaures criada se encaixa\n# df_feat = df_feat.sort_values(by=['VALUE'], ascending=False).reset_index(drop=True)\n# df_feat.loc[df_feat['VAR'].str.contains('_dec')].head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.285057Z","iopub.execute_input":"2022-07-29T10:48:17.285532Z","iopub.status.idle":"2022-07-29T10:48:17.293833Z","shell.execute_reply.started":"2022-07-29T10:48:17.285495Z","shell.execute_reply":"2022-07-29T10:48:17.292932Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_v4, df_train_v4, df_test_v4 = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.294889Z","iopub.execute_input":"2022-07-29T10:48:17.295291Z","iopub.status.idle":"2022-07-29T10:48:17.306442Z","shell.execute_reply.started":"2022-07-29T10:48:17.295254Z","shell.execute_reply":"2022-07-29T10:48:17.305651Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Lê os dados de teste\n# df_test = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/test.csv')\n# x_test = df_test.drop(columns=['ID_code'])\n\n# # Separando parte decimal e parte inteira\n# df_dec = x_test.copy()\n# df_dec.columns = [c+'_dec' for c in df_dec.columns]\n# for col in tqdm(df_dec.columns):\n#     df_dec[col] = df_dec[col].apply(lambda x: x - int(x))\n\n# # Concatena com o dataframe principal\n# x_test = pd.concat([x_test, df_dec], axis=1)\n\n# # Normaliza os valores\n# norm_values = norm_model.transform(x_test)\n# x_test = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# y_test = model_v4.predict_proba(x_test)\n# y_test_ros = model_v4_ros.predict_proba(x_test)\n\n# # Gera o dataframe submission\n# df_submission = df_test[['ID_code']].copy()\n# df_submission['target'] = y_test[:, 1]\n# df_submission.to_csv('submission_v4.csv', index=False)\n# df_submission['target'] = y_test_ros[:, 1]\n# df_submission.to_csv('submission_v4_ros.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.307610Z","iopub.execute_input":"2022-07-29T10:48:17.308152Z","iopub.status.idle":"2022-07-29T10:48:17.315986Z","shell.execute_reply.started":"2022-07-29T10:48:17.308068Z","shell.execute_reply":"2022-07-29T10:48:17.315219Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelo 5 - Utilizando informação de valores únicos","metadata":{}},{"cell_type":"markdown","source":"Este modelo cria 4 colunas com informação sobre valores únicos de cada coluna, além de inserir colunas para cada variável identificando a quantidade de valores únicos da variável em todo o dataframe.","metadata":{}},{"cell_type":"code","source":"def get_train_frequences(df):\n    cols = [c for c in df.columns if c!='target']\n    dic_freq_values = {}\n    for col in tqdm(cols):\n        dic_freq_values[col] = df[col].value_counts().to_dict()\n    return dic_freq_values\n    \n\ndef create_features_uq(dic_freq_values, df):\n    cols = [c for c in df.columns if c!='target']\n    for col in tqdm(cols):\n        dic_freq_values = df[col].value_counts().to_dict()\n        df[col] = df[col].apply(lambda x: dic_freq_values[x] if x in dic_freq_values else 1)\n    \n    df['SUM'] = df[cols].sum(axis=1)\n    df['MIN'] = df[cols].min(axis=1)\n    df['MAX'] = df[cols].max(axis=1)\n    df['COUNT_UQ'] = df.apply(lambda row: (row[cols]==1).sum(), axis=1)\n\n    dic_cols_uq = {col: f'UQ_{col}' for col in cols}\n    df.rename(columns=dic_cols_uq, inplace=True)\n\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.317379Z","iopub.execute_input":"2022-07-29T10:48:17.317735Z","iopub.status.idle":"2022-07-29T10:48:17.326378Z","shell.execute_reply.started":"2022-07-29T10:48:17.317684Z","shell.execute_reply":"2022-07-29T10:48:17.325635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Rodando modelo V5')\n\n# Define os dados\ndf_v5 = df_train.copy()\n\n# Busca os dados de treinamento e teste\ndf_train_v5 = df_v5.loc[list_idx_train]\ndf_test_v5 = df_v5.loc[list_idx_test]\n\n# Cria o dicionário de frequências\ndic_freq_values = get_train_frequences(df_v5)\n\n# Cria o dataframe de frequencia e concatena ao dataframe de treinamento\ndf_aux = create_features_uq(dic_freq_values, df_train_v5.copy())\ndf_train_v5 = pd.concat([df_train_v5, df_aux.drop(columns='target')], axis=1)\n\n# Cria o dataframe de frequencia e concatena ao dataframe de teste\ndf_aux = create_features_uq(dic_freq_values, df_test_v5.copy())\ndf_test_v5 = pd.concat([df_test_v5, df_aux.drop(columns='target')], axis=1)\n\ndf_aux = None\ndf_train_v5.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.327793Z","iopub.execute_input":"2022-07-29T10:48:17.328223Z","iopub.status.idle":"2022-07-29T10:48:17.339142Z","shell.execute_reply.started":"2022-07-29T10:48:17.328188Z","shell.execute_reply":"2022-07-29T10:48:17.338383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.pairplot(df_train_v5[['SUM', 'MIN', 'MAX', 'COUNT_UQ']]);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.340473Z","iopub.execute_input":"2022-07-29T10:48:17.341087Z","iopub.status.idle":"2022-07-29T10:48:17.347475Z","shell.execute_reply.started":"2022-07-29T10:48:17.341051Z","shell.execute_reply":"2022-07-29T10:48:17.346632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.pairplot(df_test_v5[['SUM', 'MIN', 'MAX', 'COUNT_UQ']]);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.349276Z","iopub.execute_input":"2022-07-29T10:48:17.350035Z","iopub.status.idle":"2022-07-29T10:48:17.357195Z","shell.execute_reply.started":"2022-07-29T10:48:17.349998Z","shell.execute_reply":"2022-07-29T10:48:17.356492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotando visualização\npca_model = PCA(n_components=2).fit(df_v5.drop(columns='target'))\npca_values = pca_model.transform(df_v5.drop(columns='target'))\ndf_pca = pd.DataFrame(columns=['D1', 'D2'], data=pca_values)\ndf_pca['target'] = df_v5['target'].values\nsns.set(rc={'figure.figsize':(20, 8)})\nsns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.358637Z","iopub.execute_input":"2022-07-29T10:48:17.359149Z","iopub.status.idle":"2022-07-29T10:48:17.367185Z","shell.execute_reply.started":"2022-07-29T10:48:17.359109Z","shell.execute_reply":"2022-07-29T10:48:17.366412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define o modelo de normalização\ndf_model_norm = df_train_v5.drop(columns='target')\nnorm_model = StandardScaler().fit(df_model_norm)\n\n# Gera os dados de treinamento normalizados\nnorm_values = norm_model.transform(df_train_v5.drop(columns='target'))\ndf_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ndf_aux['target'] = df_train_v5['target'].values\ndf_train_v5 = df_aux.copy()\n\n# Gera os dados de teste normalizados\nnorm_values = norm_model.transform(df_test_v5.drop(columns='target'))\ndf_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ndf_aux['target'] = df_test_v5['target'].values\ndf_test_v5 = df_aux.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.368159Z","iopub.execute_input":"2022-07-29T10:48:17.368976Z","iopub.status.idle":"2022-07-29T10:48:17.379516Z","shell.execute_reply.started":"2022-07-29T10:48:17.368938Z","shell.execute_reply":"2022-07-29T10:48:17.378743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Verifica o resultado da classificação\nmodel_v5, score_v5 = run_classifiers(df_train_v5, df_test_v5)\nmodel_v5_ros, score_v5_ros = run_classifiers(df_train_v5, df_test_v5, use_ros=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.380796Z","iopub.execute_input":"2022-07-29T10:48:17.381230Z","iopub.status.idle":"2022-07-29T10:48:17.389524Z","shell.execute_reply.started":"2022-07-29T10:48:17.381195Z","shell.execute_reply":"2022-07-29T10:48:17.388668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Relevância das novas features\ntry:\n    feat_importance = abs(model_v5.coef_[0])\nexcept:\n    feat_importance = model_v5.feature_importances_\ndf_feat = pd.DataFrame(columns=df_train_v5.drop(columns='target').columns, data=feat_importance.reshape(1, -1))\ndf_feat = df_feat.melt(var_name='VAR', value_name='VALUE')\n\n# Verifica em que posição a primeira fetaures criada se encaixa\ndf_feat = df_feat.sort_values(by=['VALUE'], ascending=False).reset_index(drop=True)\ndf_feat.loc[df_feat['VAR'].str.startswith('var_')==False].head()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.390538Z","iopub.execute_input":"2022-07-29T10:48:17.391157Z","iopub.status.idle":"2022-07-29T10:48:17.398543Z","shell.execute_reply.started":"2022-07-29T10:48:17.391120Z","shell.execute_reply":"2022-07-29T10:48:17.397773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_v5, df_train_v5, df_test_v5 = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.400977Z","iopub.execute_input":"2022-07-29T10:48:17.401641Z","iopub.status.idle":"2022-07-29T10:48:17.409256Z","shell.execute_reply.started":"2022-07-29T10:48:17.401614Z","shell.execute_reply":"2022-07-29T10:48:17.408545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lê os dados de teste\ndf_test = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/test.csv')\nx_test = df_test.drop(columns=['ID_code'])\n\n# Realiza a criação das features\ndf_aux = create_features_uq(dic_freq_values, x_test.copy())\nx_test = pd.concat([x_test, df_aux], axis=1)\n\n# Normaliza os valores\nnorm_values = norm_model.transform(x_test)\nx_test = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n\n# Roda o modelo\ny_test = model_v5.predict_proba(x_test)\ny_test_ros = model_v5_ros.predict_proba(x_test)\n\n# Gera o dataframe submission\ndf_submission = df_test[['ID_code']].copy()\ndf_submission['target'] = y_test[:, 1]\ndf_submission.to_csv('submission_v5.csv', index=False)\ndf_submission['target'] = y_test_ros[:, 1]\ndf_submission.to_csv('submission_v5_ros.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.417891Z","iopub.execute_input":"2022-07-29T10:48:17.418139Z","iopub.status.idle":"2022-07-29T10:48:17.421939Z","shell.execute_reply.started":"2022-07-29T10:48:17.418116Z","shell.execute_reply":"2022-07-29T10:48:17.421173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelo 6 - Extraindo características dos dados com AutoEncoder","metadata":{}},{"cell_type":"markdown","source":"Este modelo tenta criar uma representação alternativa das features através de um modelo de rede neural AutoEncoder. O vetor da estrutura interna (encoder) é utilizado como entrada para os modelos de classificação.","metadata":{}},{"cell_type":"code","source":"# print(\"Rodando modelo v6\")\n\n# # Define os dados\n# df_v6 = df_train.copy()\n\n# # Busca os dados de treinamento e teste\n# df_train_v6 = df_v6.loc[list_idx_train]\n# df_test_v6 = df_v6.loc[list_idx_test]","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.422956Z","iopub.execute_input":"2022-07-29T10:48:17.423639Z","iopub.status.idle":"2022-07-29T10:48:17.435287Z","shell.execute_reply.started":"2022-07-29T10:48:17.423601Z","shell.execute_reply":"2022-07-29T10:48:17.434312Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Define o modelo de normalização\n# df_model_norm = df_train_v6.drop(columns='target')\n# norm_model = StandardScaler().fit(df_model_norm)\n\n# # Gera os dados de treinamento normalizados\n# norm_values = norm_model.transform(df_train_v6.drop(columns='target'))\n# df_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# df_aux['target'] = df_train_v6['target'].values\n# df_train_v6 = df_aux.copy()\n\n# # Gera os dados de teste normalizados\n# norm_values = norm_model.transform(df_test_v6.drop(columns='target'))\n# df_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n# df_aux['target'] = df_test_v6['target'].values\n# df_test_v6 = df_aux.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.436348Z","iopub.execute_input":"2022-07-29T10:48:17.437663Z","iopub.status.idle":"2022-07-29T10:48:17.446318Z","shell.execute_reply.started":"2022-07-29T10:48:17.437636Z","shell.execute_reply":"2022-07-29T10:48:17.445500Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Separa as classes\n# df_c0 = df_train_v6.loc[df_train_v6['target']==0].copy()\n# df_c1 = df_train_v6.loc[df_train_v6['target']==1].copy()\n\n# # Define o conjunto de treinamento com base na classe 0 (especializar o modelo na classe maioritária)\n# num_samples = 10000\n# if df_c0.shape[1] < num_samples: num_samples = df_c0.shape[1]\n# x_treino = df_c0.sample(num_samples).drop(columns='target')","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.447164Z","iopub.execute_input":"2022-07-29T10:48:17.447397Z","iopub.status.idle":"2022-07-29T10:48:17.456201Z","shell.execute_reply.started":"2022-07-29T10:48:17.447374Z","shell.execute_reply":"2022-07-29T10:48:17.455384Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # AutoEncoder - Input layer\n# input_layer = Input(shape=(df_v6.drop(columns=\"target\").shape[1],))\n\n# # Encoding part\n# encoded_0 = Dense(200, activation='tanh', activity_regularizer=regularizers.l1(10e-5))(input_layer)\n# encoded_1 = Dense(100, activation='relu')(encoded_0)\n\n# # Decoding part\n# decoded_1 = Dense(100, activation='relu')(encoded_1)\n# decoded_0 = Dense(200, activation='tanh', activity_regularizer=regularizers.l1(10e-5))(decoded_1)\n\n# # Output layer\n# output_layer = Dense(df_v6.drop(columns=\"target\").shape[1], activation='relu')(decoded_0)\n\n# # Modelo\n# autoencoder_model = Model(input_layer, output_layer)\n# autoencoder_model.compile(optimizer=\"sgd\", loss=\"mse\")\n\n# # Treina o modelo\n# callback = EarlyStopping(monitor='val_loss', patience=10, min_delta=0.0001)\n# autoencoder_model.fit(x_treino, x_treino, \n#                 batch_size = 256, epochs = 50, \n#                 shuffle = True, validation_split = 0.20, callbacks=[callback]);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.457580Z","iopub.execute_input":"2022-07-29T10:48:17.458001Z","iopub.status.idle":"2022-07-29T10:48:17.469075Z","shell.execute_reply.started":"2022-07-29T10:48:17.457966Z","shell.execute_reply":"2022-07-29T10:48:17.468446Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Cria o modelo para recuparação da estrutura de encoder\n# hidden_representation = Sequential()\n# hidden_representation.add(autoencoder_model.layers[0])\n# hidden_representation.add(autoencoder_model.layers[1])\n# hidden_representation.add(autoencoder_model.layers[2])\n\n# # Realiza inferência da representação\n# hid_rep = hidden_representation.predict(df_train_v6.drop(columns='target'))\n\n# # Consolida em dataframe\n# cols = [f'ae_{i}' for i in range(hid_rep.shape[1])]\n# df_aux = pd.DataFrame(columns=cols, data=hid_rep, index=df_train_v6.index)\n# df_aux['target'] = df_train_v6['target']\n# df_train_v6 = df_aux.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.470461Z","iopub.execute_input":"2022-07-29T10:48:17.470825Z","iopub.status.idle":"2022-07-29T10:48:17.479077Z","shell.execute_reply.started":"2022-07-29T10:48:17.470791Z","shell.execute_reply":"2022-07-29T10:48:17.478276Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_train_v6.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.480710Z","iopub.execute_input":"2022-07-29T10:48:17.481243Z","iopub.status.idle":"2022-07-29T10:48:17.488342Z","shell.execute_reply.started":"2022-07-29T10:48:17.481204Z","shell.execute_reply":"2022-07-29T10:48:17.487585Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Plotando visualização\n# pca_model = PCA(n_components=2).fit(df_train_v6.drop(columns='target'))\n# pca_values = pca_model.transform(df_train_v6.drop(columns='target'))\n# df_pca = pd.DataFrame(columns=['D1', 'D2'], data=pca_values)\n# df_pca['target'] = df_train_v6['target'].values\n# sns.set(rc={'figure.figsize':(20, 8)})\n# sns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.489256Z","iopub.execute_input":"2022-07-29T10:48:17.489512Z","iopub.status.idle":"2022-07-29T10:48:17.499537Z","shell.execute_reply.started":"2022-07-29T10:48:17.489481Z","shell.execute_reply":"2022-07-29T10:48:17.498723Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Realiza inferência da representação para os dados de teste\n# hid_rep = hidden_representation.predict(df_test_v6.drop(columns='target'))\n\n# # Consolida em dataframe\n# cols = [f'ae_{i}' for i in range(hid_rep.shape[1])]\n# df_aux = pd.DataFrame(columns=cols, data=hid_rep, index=df_test_v6.index)\n# df_aux['target'] = df_test_v6['target']\n# df_test_v6 = df_aux.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.500827Z","iopub.execute_input":"2022-07-29T10:48:17.501192Z","iopub.status.idle":"2022-07-29T10:48:17.513523Z","shell.execute_reply.started":"2022-07-29T10:48:17.501155Z","shell.execute_reply":"2022-07-29T10:48:17.512778Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Verifica o resultado da classificação\n# model_v6, score_v6 = run_classifiers(df_train_v6, df_test_v6)\n# model_v6_ros, score_v6_ros = run_classifiers(df_train_v6, df_test_v6, use_ros=True)\n# # df_v6, df_train_v6, df_test_v6 = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.516342Z","iopub.execute_input":"2022-07-29T10:48:17.516823Z","iopub.status.idle":"2022-07-29T10:48:17.522400Z","shell.execute_reply.started":"2022-07-29T10:48:17.516798Z","shell.execute_reply":"2022-07-29T10:48:17.521568Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Lê os dados de teste\n# df_test = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/test.csv')\n# x_test = df_test.drop(columns=['ID_code'])\n\n# # Normaliza os valores\n# norm_values = norm_model.transform(x_test)\n# x_test = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\n\n# # Realiza inferência da representação para os dados de teste\n# hid_rep = hidden_representation.predict(x_test)\n\n# # Consolida em dataframe\n# cols = [f'ae_{i}' for i in range(hid_rep.shape[1])]\n# x_test = pd.DataFrame(columns=cols, data=hid_rep, index=x_test.index)\n\n# # Roda o modelo\n# y_test = model_v6.predict_proba(x_test)\n# y_test_ros = model_v6_ros.predict_proba(x_test)\n\n# # Gera o dataframe submission\n# df_submission = df_test[['ID_code']].copy()\n# df_submission['target'] = y_test[:, 1]\n# df_submission.to_csv('submission_v6.csv', index=False)\n# df_submission['target'] = y_test_ros[:, 1]\n# df_submission.to_csv('submission_v6_ros.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.523845Z","iopub.execute_input":"2022-07-29T10:48:17.524339Z","iopub.status.idle":"2022-07-29T10:48:17.533651Z","shell.execute_reply.started":"2022-07-29T10:48:17.524301Z","shell.execute_reply":"2022-07-29T10:48:17.532900Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(\"Rodando modelo v7\")\n\n# # Define a lista dos dataframes a serem usados\n# list_df = [\n#     [df_train_v2, df_test_v2],\n#     [df_train_v3, df_test_v3],\n#     [df_train_v4, df_test_v4],\n#     [df_train_v5, df_test_v5],\n#     [df_train_v6, df_test_v6],\n# ]\n\n# # Cria os dataframes da rodada 7\n# df_train_v7 = pd.DataFrame(index=df_train_v2.index)\n# df_test_v7 = pd.DataFrame(index=df_test_v2.index)\n\n# # Insere o target\n# df_train_v7['target'] = df_train_v2['target']\n# df_test_v7['target'] = df_test_v2['target']\n\n# # Percorre cada dataframe e concatena as colunas não existentes\n# for df_train, df_test in list_df:\n#     # Treinamento\n#     for col in df_train.columns:\n#         if col not in df_train_v7.columns:\n#             df_train_v7[col] = df_train[col]\n            \n#     # Teste\n#     for col in df_test.columns:\n#         if col not in df_test_v7.columns:\n#             df_test_v7[col] = df_test[col]","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.534955Z","iopub.execute_input":"2022-07-29T10:48:17.535315Z","iopub.status.idle":"2022-07-29T10:48:17.543149Z","shell.execute_reply.started":"2022-07-29T10:48:17.535279Z","shell.execute_reply":"2022-07-29T10:48:17.542386Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Plotando visualização\n# pca_model = PCA(n_components=2).fit(df_train_v7.drop(columns='target'))\n# pca_values = pca_model.transform(df_train_v7.drop(columns='target'))\n# df_pca = pd.DataFrame(columns=['D1', 'D2'], data=pca_values)\n# df_pca['target'] = df_train_v6['target'].values\n# sns.set(rc={'figure.figsize':(20, 8)})\n# sns.scatterplot(data=df_pca, x=\"D1\", y=\"D2\", hue=\"target\", alpha=0.5);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.544317Z","iopub.execute_input":"2022-07-29T10:48:17.544790Z","iopub.status.idle":"2022-07-29T10:48:17.555800Z","shell.execute_reply.started":"2022-07-29T10:48:17.544752Z","shell.execute_reply":"2022-07-29T10:48:17.555065Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Verifica o resultado da classificação\n# model_v7, score_v7 = run_classifiers(df_train_v7, df_test_v7)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.558664Z","iopub.execute_input":"2022-07-29T10:48:17.558983Z","iopub.status.idle":"2022-07-29T10:48:17.565795Z","shell.execute_reply.started":"2022-07-29T10:48:17.558959Z","shell.execute_reply":"2022-07-29T10:48:17.565038Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modelo 7 - Realiza classificação com os dados normalizados, removendo outliers\n\nA remoção de outliers é feita no conjunto de treinamento removendo valores, que após o StandardScaler, que ultrapassgem o limiar definido na variável `THRESHOLD`. Esta lógica está retirando dados a partir da variação do desvio padrão.","metadata":{}},{"cell_type":"code","source":"# Definição de limites superior e inferior\nTHRESHOLD = 2.5","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.566925Z","iopub.execute_input":"2022-07-29T10:48:17.567394Z","iopub.status.idle":"2022-07-29T10:48:17.575886Z","shell.execute_reply.started":"2022-07-29T10:48:17.567358Z","shell.execute_reply":"2022-07-29T10:48:17.575061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Rodando modelo v7\")\n\n# Define os dados para a rodada\ndf_v7 = df_train.copy()\n\n# Busca os dados de treinamento e teste\ndf_train_v7 = df_v7.loc[list_idx_train]\ndf_test_v7 = df_v7.loc[list_idx_test]\n\n# Define o modelo de normalização\ndf_model_norm = df_train_v7.drop(columns='target')\nnorm_model = StandardScaler().fit(df_model_norm)\n\n# Gera os dados de treinamento normalizados\nnorm_values = norm_model.transform(df_train_v7.drop(columns='target'))\ndf_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ndf_aux['target'] = df_train_v7['target'].values\ndf_train_v7 = df_aux.copy()\n\n# Gera os dados de teste normalizados\nnorm_values = norm_model.transform(df_test_v7.drop(columns='target'))\ndf_aux = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ndf_aux['target'] = df_test_v7['target'].values\ndf_test_v7 = df_aux.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.576830Z","iopub.execute_input":"2022-07-29T10:48:17.577878Z","iopub.status.idle":"2022-07-29T10:48:17.641225Z","shell.execute_reply.started":"2022-07-29T10:48:17.577848Z","shell.execute_reply":"2022-07-29T10:48:17.640349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Identifica e remove as linhas que possuem algum outlier\ncols = [c for c in df_train_v7.columns if c!='target']\ndf_train_v7['OUTLIER_COUNT_POS'] = df_train_v7[cols].apply(lambda row: (row>THRESHOLD).sum(), axis=1)\ndf_train_v7['OUTLIER_COUNT_NEG'] = df_train_v7[cols].apply(lambda row: (row<-THRESHOLD).sum(), axis=1)\ndf_train_v7 = df_train_v7.loc[\n    (df_train_v7['OUTLIER_COUNT_POS']==0)&\n    (df_train_v7['OUTLIER_COUNT_NEG']==0)\n].copy()\ndf_train_v7.drop(columns=['OUTLIER_COUNT_POS', 'OUTLIER_COUNT_NEG'], inplace=True)\ndf_train_v7.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:17.642485Z","iopub.execute_input":"2022-07-29T10:48:17.643073Z","iopub.status.idle":"2022-07-29T10:48:20.062478Z","shell.execute_reply.started":"2022-07-29T10:48:17.643033Z","shell.execute_reply":"2022-07-29T10:48:20.061669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Comparando a variância das variáveis\ndf_melt = df_train_v7.copy()\ndf_melt.drop(columns='target', inplace=True)\ncols_melt = df_melt.columns.tolist()\ndf_melt = df_melt.melt(var_name='VAR', value_vars=cols_melt, value_name='VALUE')\n\n# Plotando\nfig, ax = plt.subplots()\nax_bar = sns.boxplot(x='VAR', y='VALUE', data=df_melt);\nax.set(xlabel=None)\nax.set(xticklabels=[])\nfig.set_size_inches(32, 8)\nplt.title(\"Variância dos dados normalizados\");","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:20.063531Z","iopub.execute_input":"2022-07-29T10:48:20.064031Z","iopub.status.idle":"2022-07-29T10:48:23.762575Z","shell.execute_reply.started":"2022-07-29T10:48:20.063995Z","shell.execute_reply":"2022-07-29T10:48:23.761630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Verifica o resultado da classificação\nmodel_v7, score_v7 = run_classifiers(df_train_v7, df_test_v7)\nmodel_v7_ros, score_v7_ros = run_classifiers(df_train_v7, df_test_v7, use_ros=True)\ndf_v7, df_train_v7, df_test_v7 = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:23.763969Z","iopub.execute_input":"2022-07-29T10:48:23.764428Z","iopub.status.idle":"2022-07-29T10:48:54.995716Z","shell.execute_reply.started":"2022-07-29T10:48:23.764388Z","shell.execute_reply":"2022-07-29T10:48:54.995100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lê os dados de teste, separa entrada, normaliza valores e infere saída\ndf_test = pd.read_csv('/kaggle/input/santander-customer-transaction-prediction/test.csv')\nx_test = df_test.drop(columns=['ID_code'])\nnorm_values = norm_model.transform(x_test)\nx_test = pd.DataFrame(columns=df_model_norm.columns, data=norm_values)\ny_test = model_v7.predict_proba(x_test)\ny_test_ros = model_v7_ros.predict_proba(x_test)\n\n# Gera o dataframe submission\ndf_submission = df_test[['ID_code']].copy()\ndf_submission['target'] = y_test[:, 1]\ndf_submission.to_csv('submission_v7.csv', index=False)\ndf_submission['target'] = y_test_ros[:, 1]\ndf_submission.to_csv('submission_v7_ros.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:48:54.999141Z","iopub.execute_input":"2022-07-29T10:48:55.000883Z","iopub.status.idle":"2022-07-29T10:49:02.738573Z","shell.execute_reply.started":"2022-07-29T10:48:55.000837Z","shell.execute_reply":"2022-07-29T10:49:02.737746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Compara os resultados dos modelos","metadata":{}},{"cell_type":"code","source":"models = [\n#     ['V1', model_v1, score_v1],\n    ['V2', model_v2, score_v2],\n#     ['V3', model_v3, score_v3],\n#     ['V4', model_v4, score_v4],\n    ['V5', model_v5, score_v5],\n#     ['V6', model_v6, score_v6],\n    ['V7', model_v7, score_v7],\n#     ['V1_ROS', model_v1_ros, score_v1_ros],\n    ['V2_ROS', model_v2_ros, score_v2_ros],\n#     ['V3_ROS', model_v3_ros, score_v3_ros],\n#     ['V4_ROS', model_v4_ros, score_v4_ros],\n    ['V5_ROS', model_v5_ros, score_v5_ros],\n#     ['V6_ROS', model_v6_ros, score_v6_ros],\n    ['V7_ROS', model_v7_ros, score_v7_ros],\n]\ndf_results = pd.DataFrame(columns=['RODADA', 'MODEL', 'SCORE'])\n\nfor idx, row in enumerate(models):\n    rodada, model, score = row\n    df_results.loc[idx, ['RODADA', 'MODEL', 'SCORE']] = rodada, str(model), score\n    \ndf_results.sort_values(by='SCORE', inplace=True)\n\nsns.barplot(x='RODADA', y='SCORE', data=df_results);","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:02.739750Z","iopub.execute_input":"2022-07-29T10:49:02.740187Z","iopub.status.idle":"2022-07-29T10:49:02.935807Z","shell.execute_reply.started":"2022-07-29T10:49:02.740150Z","shell.execute_reply":"2022-07-29T10:49:02.935046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_results","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:02.937194Z","iopub.execute_input":"2022-07-29T10:49:02.937540Z","iopub.status.idle":"2022-07-29T10:49:02.949948Z","shell.execute_reply.started":"2022-07-29T10:49:02.937502Z","shell.execute_reply":"2022-07-29T10:49:02.949127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explorando os resultados do melhor modelo","metadata":{}},{"cell_type":"code","source":"# Seleciona o melhor modelo\nbest_model = model_v2\n\n# Seleciona os dados do melhor modelo\ndf_best_model_train = df_train_v2\ndf_best_model_test = df_test_v2","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:02.951507Z","iopub.execute_input":"2022-07-29T10:49:02.951903Z","iopub.status.idle":"2022-07-29T10:49:02.957587Z","shell.execute_reply.started":"2022-07-29T10:49:02.951866Z","shell.execute_reply":"2022-07-29T10:49:02.956872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Realiza a inferência nos dados de treino e teste\ny_pred_train = best_model.predict(df_best_model_train.drop(columns='target'))\ny_pred_test = best_model.predict(df_best_model_test.drop(columns='target'))","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:02.959138Z","iopub.execute_input":"2022-07-29T10:49:02.959772Z","iopub.status.idle":"2022-07-29T10:49:02.984360Z","shell.execute_reply.started":"2022-07-29T10:49:02.959736Z","shell.execute_reply":"2022-07-29T10:49:02.983375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Matriz de confusão do conjunto de treinamento')\nmatriz_confusao = confusion_matrix(df_best_model_train['target'], y_pred_train, labels=best_model.classes_, normalize='true')\nmc_plot = ConfusionMatrixDisplay(confusion_matrix=matriz_confusao, display_labels=best_model.classes_, )\nmc_plot = mc_plot.plot(cmap='Blues')\nplt.grid(False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:02.985919Z","iopub.execute_input":"2022-07-29T10:49:02.986502Z","iopub.status.idle":"2022-07-29T10:49:03.214871Z","shell.execute_reply.started":"2022-07-29T10:49:02.986464Z","shell.execute_reply":"2022-07-29T10:49:03.214089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"True Positive = {matriz_confusao[0,0]}\")\nprint(f\"False Negative = {matriz_confusao[0,1]}\")\nprint(f\"False Positive = {matriz_confusao[1,0]}\")\nprint(f\"True Negative = {matriz_confusao[1,1]}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:03.216298Z","iopub.execute_input":"2022-07-29T10:49:03.216871Z","iopub.status.idle":"2022-07-29T10:49:03.222418Z","shell.execute_reply.started":"2022-07-29T10:49:03.216832Z","shell.execute_reply":"2022-07-29T10:49:03.221419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Matriz de confusão do conjunto de teste')\nmatriz_confusao = confusion_matrix(df_best_model_test['target'], y_pred_test, labels=best_model.classes_, normalize='true')\nmc_plot = ConfusionMatrixDisplay(confusion_matrix=matriz_confusao, display_labels=best_model.classes_, )\nmc_plot = mc_plot.plot(cmap='Blues')\nplt.grid(False)","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:03.223861Z","iopub.execute_input":"2022-07-29T10:49:03.224650Z","iopub.status.idle":"2022-07-29T10:49:03.419857Z","shell.execute_reply.started":"2022-07-29T10:49:03.224502Z","shell.execute_reply":"2022-07-29T10:49:03.419017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"True Positive = {matriz_confusao[0,0]}\")\nprint(f\"False Negative = {matriz_confusao[0,1]}\")\nprint(f\"False Positive = {matriz_confusao[1,0]}\")\nprint(f\"True Negative = {matriz_confusao[1,1]}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:03.421318Z","iopub.execute_input":"2022-07-29T10:49:03.421930Z","iopub.status.idle":"2022-07-29T10:49:03.427408Z","shell.execute_reply.started":"2022-07-29T10:49:03.421890Z","shell.execute_reply":"2022-07-29T10:49:03.426713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(classification_report(df_best_model_test['target'], y_pred_test))","metadata":{"execution":{"iopub.status.busy":"2022-07-29T10:49:03.428844Z","iopub.execute_input":"2022-07-29T10:49:03.429528Z","iopub.status.idle":"2022-07-29T10:49:03.444668Z","shell.execute_reply.started":"2022-07-29T10:49:03.429489Z","shell.execute_reply":"2022-07-29T10:49:03.443754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Pode-se constatar que o balanceamento entre as classes tentou evitar que o algoritmo se _viciasse_ em inferir apenas a classe majoritária. Isso pode ser observado pela distribuição da matriz de confusão, onde há uma concentração (menor no conjunto de teste) de registros classificados corretamente (True Positive e True Negative).\n\nNo entanto, avaliando as métricas do melhor modelo, observa-se que o desbalanceamento ainda é um fator que influencia na qualidade do modelo. Mesmo com os tratamentos utilizados (balanceando pesos no modelo de classificação ou realizando reamostragem através da biblioteca ROS), as métricas `precision` e por consequência a `f1-score` obtiveram valores baixos.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Referências:\n- [1] https://github.com/btrotta/kaggle-santander-2019/blob/master/Readme.pdf\n- [2] https://www.kaggle.com/code/yag320/list-of-fake-samples-and-public-private-lb-split/notebook\n- [3] https://www.kaggle.com/code/allunia/santander-customer-transaction-eda/notebook\n- [4] https://www.kaggle.com/code/claudiohfg/santander-lightgbm-random-over-sampler-l1-l2\n- [5] https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc\n- [6] https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n- [7] https://lightgbm.readthedocs.io/en/latest/Parameters.html#is_unbalance","metadata":{}}]}