{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":81933,"databundleVersionId":9643020,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Definição do Problema\nO uso excessivo da internet pode levar a problemas como dependência, isolamento social e queda de desempenho acadêmico, especialmente em crianças e jovens. Identificar comportamentos que indicam uso problemático da internet pode ajudar pesquisadores, pais e educadores a tomar ações preventivas.\n\nO objetivo é desenvolver um modelo preditivo que analise dados de atividade física e condicionamento físico de crianças para identificar sinais de uso problemático da internet. Identificar esses padrões pode ajudar a desencadear intervenções para encorajar hábitos digitais mais saudáveis.","metadata":{}},{"cell_type":"markdown","source":"# Descrição do conjunto de dados\n\n1. **Basic_Demos-Age:** Idade do participante.\n1. **Basic_Demos-Sex:** Sexo do participante.\n1. **Physical-BMI:** O Índice de Massa Corporal (BMI, ou IMC) é uma medida de gordura corporal baseada na altura e no peso do indivíduo, expressa em kg/m²\n1. **Physical-Height:** Medida da altura do participante em polegadas (in).\n1. **Physical-Weight:** Peso do participante em libras (lbs).\n1. **Physical-Waist_Circumference:** Medida da circunferência da cintura em polegadas (in).\n1. **Physical-HeartRate:** A frequência cardíaca em repouso mede o número de batimentos por minuto (bpm).\n1. **Physical-Systolic_BP:** Medida da pressão arterial sistólica, que é a pressão nas artérias quando o coração se contrai.\n1. **Physical-Diastolic_BP:** Medida da pressão arterial diastólica, que é a pressão nas artérias quando o coração está em repouso \n1. **FGC-FGC_CU (Curl Up Total):** Esta coluna indica o número total de curl-ups (exercícios de abdominal) que o participante completou.\n1. **FGC-FGC_CU_Zone (Curl Up Fitness Zone):** Esta coluna indica a zona de aptidão que o participante alcançou no teste de curl-up.\n1. **FGC-FGC_PU (Push-up Total):** Esta coluna indica o número total de flexões (push-ups) que o participante conseguiu completar. O teste de flexão mede a resistência muscular dos músculos superiores, especialmente peito, ombros e tríceps.\n1. **FGC-FGC_PU_Zone (Push-up Fitness Zone):** Esta coluna representa a zona de aptidão para o teste de flexões, avaliando o desempenho em comparação com as zonas de saúde do FitnessGram, que variam por idade e sexo.\n1. **FGC-FGC_SRL (Sit & Reach Total - Left Side):** Esta coluna indica o resultado total do teste de Sit & Reach para o lado esquerdo do corpo. Nesse teste, o participante se senta com as pernas estendidas e alcança o mais longe possível à frente, tocando uma régua ou medidor. O valor é medido em centímetros ou polegadas, dependendo do protocolo.\n1. **FGC-FGC_SRL_Zone (Sit & Reach Fitness Zone - Left Side):** Esta coluna indica a zona de aptidão para o teste de Sit & Reach do lado esquerdo, baseada nas diretrizes do FitnessGram. A zona de aptidão classifica o desempenho do participante em relação a padrões de flexibilidade para sua idade e sexo.\n1. **FGC-FGC_SRR (Sit & Reach Total - Right Side):** Esta coluna mostra o resultado total do teste de Sit & Reach para o lado direito do corpo. Este teste mede a flexibilidade da parte inferior das costas e dos músculos posteriores das pernas, especialmente os isquiotibiais.\n1. **FGC-FGC_SRR_Zone (Sit & Reach Fitness Zone - Right Side):** Esta coluna indica a zona de aptidão para o teste de Sit & Reach no lado direito, categorizando a flexibilidade em comparação com padrões de saúde para idade e sexo.\n1. **FGC-FGC_TL (Trunk Lift Total):** Esta coluna indica o valor total do teste de Trunk Lift (Elevação do Tronco), que mede a altura que o participante consegue levantar a parte superior do corpo a partir de uma posição deitada de bruços, até um limite recomendado.\n1. **FGC-FGC_TL_Zone (Trunk Lift Fitness Zone):** Esta coluna indica a zona de aptidão do participante para o teste de Trunk Lift, categorizando o desempenho em relação aos critérios de saúde e desenvolvimento estabelecidos pelo FitnessGram para idade e sexo\n1. **FGC-FGC_GSND (Grip Strength Total - Non-Dominant Hand):** Esta coluna registra a força de preensão total da mão não-dominante do participante. \n1. **FGC-FGC_GSND_Zone (Grip Strength Fitness Zone - Non-Dominant Hand):** Esta coluna indica a zona de aptidão alcançada no teste de força de preensão da mão não-dominante. \n1. **FGC-FGC_GSD (Grip Strength Total - Dominant Hand):** Esta coluna registra a força de preensão total da mão dominante do participante, medida em quilogramas (kg).\n1. **FGC-FGC_GSD_Zone (Grip Strength Fitness Zone - Dominant Hand):** Esta coluna indica a zona de aptidão para o teste de força de preensão da mão dominante. \n1. **BIA-BIA_Activity_Level_num (Nível de Atividade Física):** Esta coluna registra o nível de atividade física do participante, geralmente em uma escala numérica. \n1. **BIA-BIA_BMC (Bone Mineral Content):** Esta coluna indica o conteúdo mineral ósseo do participante, uma medida da quantidade de minerais presentes nos ossos, geralmente expressa em quilogramas (kg). \n1. **BIA-BIA_BMI (Body Mass Index):** Esta coluna representa o Índice de Massa Corporal (IMC), calculado a partir da altura e do peso do participante. O IMC é uma medida da gordura corporal com base na altura e no peso, expressa em kg/m².\n1. **BIA-BIA_BMR (Basal Metabolic Rate):** Esta coluna mede a taxa metabólica basal (TMB), que é a quantidade mínima de calorias que o corpo precisa para manter funções vitais em repouso, como respiração, circulação e digestão. \n1. **BIA-BIA_DEE (Daily Energy Expenditure):** Esta coluna indica o gasto energético diário (DEE), que é uma estimativa de quantas calorias o participante queima em um dia. O DEE é calculado considerando a taxa metabólica basal (TMB) e o nível de atividade física.\n1. **BIA-BIA_ECW (Extracellular Water):** Esta coluna mede a água extracelular (ECW), que é a quantidade de água fora das células, incluindo fluidos intersticiais e no plasma sanguíneo, geralmente expressa em litros ou como uma porcentagem da água corporal total.\n1. **BIA-BIA_FFM (Fat Free Mass):** Esta coluna indica a massa livre de gordura (FFM), que é a massa total do corpo excluindo a gordura. Inclui músculos, ossos, água e outros tecidos. Geralmente expressa em quilogramas (kg).\n1. **BIA-BIA_FFMI (Fat Free Mass Index):** Esta coluna representa o Índice de Massa Livre de Gordura (FFMI), que é a FFM ajustada para a altura do indivíduo. Calculado como: FFMI= FFM (kg)​ / Altura (m) 2.\n1. **BIA-BIA_FMI (Fat Mass Index):** Esta coluna indica o Índice de Massa de Gordura (FMI), que representa a massa de gordura em relação à altura do participante. O FMI é semelhante ao IMC, mas foca apenas na gordura corporal. É calculado como: FMI= Massa de Gordura (kg)​ / Altura (m) 2\n1. **BIA-BIA_Fat (Body Fat Percentage):** Esta coluna mede o percentual de gordura corporal, que é a proporção da massa de gordura em relação ao peso total do corpo. Geralmente expressa em porcentagem (%).\n1. **BIA-BIA_Frame_num (Body Frame):** Esta coluna classifica o porte corporal do participante, geralmente em uma escala numérica para diferenciar entre tamanhos de estrutura corporal (por exemplo, pequena, média, grande).\n1. **BIA-BIA_ICW (Intracellular Water):** Esta coluna mede a quantidade de água intracelular (ICW), que é a água contida dentro das células do corpo. A água intracelular é geralmente expressa em litros e é um componente importante para a função celular.\n1. **BIA-BIA_LDM (Lean Dry Mass):** Esta coluna representa a massa magra seca (LDM), que é a massa magra do corpo excluindo a gordura e a água. Em outras palavras, é a massa dos tecidos do corpo sem a água, incluindo músculos, ossos, e órgãos.\n1. **BIA-BIA_LST (Lean Soft Tissue):** Esta coluna mede o tecido mole magro (LST), que inclui a massa de tecidos moles do corpo sem gordura, como músculos, tendões e órgãos. Diferente da LDM, o LST inclui a água presente nesses tecidos.\n1. **BIA-BIA_SMM (Skeletal Muscle Mass):** Esta coluna representa a massa muscular esquelética (SMM), que é a quantidade de músculo nos músculos esqueléticos (músculos que movem os ossos e estão sob controle voluntário), excluindo órgãos e outros tecidos.\n1. **BIA-BIA_TBW (Total Body Water):** Esta coluna indica a água corporal total (TBW), que é a quantidade de água presente em todo o corpo, incluindo a água intracelular (dentro das células) e extracelular (fora das células). Geralmente expressa em litros.\n1. **PAQ_A-PAQ_A_Total (Activity Summary Score for Adolescents):** Esta coluna indica o escore de resumo da atividade física para adolescentes com base no Physical Activity Questionnaire for Adolescents (PAQ-A). Esse questionário avalia a frequência e intensidade das atividades vigorosas realizadas pelos adolescentes nos últimos 7 dias, como correr, esportes de alta intensidade e atividades físicas regulares.\n1. **PAQ_C-PAQ_C_Total (Activity Summary Score for Children):** Esta coluna representa o escore de resumo da atividade física para crianças, calculado a partir do Physical Activity Questionnaire for Children (PAQ-C). Esse questionário é adaptado para avaliar o nível de atividade física de crianças mais jovens (geralmente entre 8 e 14 anos), considerando atividades vigorosas realizadas nos últimos 7 dias.\n1. **Estágio Máximo Alcançado:** Esta coluna indica o nível máximo de intensidade alcançado pelo participante durante o teste de esteira. \n1. **Tempo Exato Completado (Minutos):** Esta coluna registra o tempo total (em minutos) que o participante conseguiu manter o exercício na esteira antes de parar.\n1. **Tempo Exato Completado (Segundos):** Esta coluna registra os segundos adicionais ao tempo em minutos, permitindo a medição precisa da duração do teste.\n1. **SDS-SDS_Total_Raw (Total Raw Score):** Esta coluna representa o escore bruto total da escala SDS. O escore bruto é a soma das respostas dadas em cada item do questionário SDS, que avalia diferentes aspectos do sono, como insônia, distúrbios respiratórios, comportamentos noturnos, parassonias e sonolência diurna.\n1. **SDS-SDS_Total_T (Total T-Score):** Esta coluna indica o escore T total, que é uma versão padronizada do escore bruto. O escore T é ajustado para levar em conta a idade e, às vezes, o gênero do participante, permitindo uma comparação mais precisa entre diferentes crianças.\n1. **Children’s Global Assessment Scale (CGAS):** A coluna CGAS-CGAS_Score representa a pontuação na Children's Global Assessment Scale (CGAS), que é uma escala numérica usada por profissionais de saúde mental para avaliar o nível geral de funcionamento de crianças e adolescentes com menos de 18 anos.\n1. **PreInt_EduHx-computerinternet_hoursday:** representa o número de horas diárias que o participante passa usando o computador ou a internet. \n","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.preprocessing import FunctionTransformer\nfrom sklearn.preprocessing import label_binarize\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics import silhouette_score\nfrom sklearn.ensemble import StackingClassifier\nfrom sklearn.ensemble import VotingClassifier\nfrom sklearn.linear_model import Lasso\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.ensemble import GradientBoostingRegressor\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\nfrom sklearn.metrics import accuracy_score, classification_report, confusion_matrix, precision_score\nfrom sklearn.metrics import recall_score, f1_score, ConfusionMatrixDisplay, RocCurveDisplay, PrecisionRecallDisplay\nfrom sklearn.decomposition import PCA\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.neural_network import MLPRegressor\nfrom sklearn.svm import SVR\nfrom sklearn.ensemble import ExtraTreesRegressor\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.model_selection import train_test_split, cross_val_score\n\n\nfrom xgboost import XGBRegressor\nfrom xgboost import XGBClassifier\n\nfrom lightgbm import LGBMRegressor\nfrom lightgbm import LGBMClassifier\n\nfrom catboost import CatBoostRegressor\nfrom catboost import CatBoostClassifier\n\nfrom sklearn.compose import ColumnTransformer\n\n#from sklearn.pipeline import Pipeline\nfrom imblearn.pipeline import Pipeline\nfrom imblearn.under_sampling import RandomUnderSampler\nfrom imblearn.over_sampling import SMOTE\n\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport warnings","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:28.151000Z","iopub.execute_input":"2025-04-21T14:17:28.151549Z","iopub.status.idle":"2025-04-21T14:17:28.163860Z","shell.execute_reply.started":"2025-04-21T14:17:28.151501Z","shell.execute_reply":"2025-04-21T14:17:28.162397Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"warnings.filterwarnings(\"ignore\", category=FutureWarning)\n\ntrain_df = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/train.csv')\ntest_df = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/test.csv')\ndata_dict = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/data_dictionary.csv')","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:28.170483Z","iopub.execute_input":"2025-04-21T14:17:28.171661Z","iopub.status.idle":"2025-04-21T14:17:28.238181Z","shell.execute_reply.started":"2025-04-21T14:17:28.171606Z","shell.execute_reply":"2025-04-21T14:17:28.237209Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Exploratory analysis","metadata":{}},{"cell_type":"code","source":"df = train_df.copy()\n\ndf['Basic_Demos-Age_Zone'] = pd.cut(df['Basic_Demos-Age'], \n                                        bins=[0, 13, 18, 41, 66], \n                                        labels=['criança', 'adolescente', 'adulto', 'idoso'])","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:28.239741Z","iopub.execute_input":"2025-04-21T14:17:28.240121Z","iopub.status.idle":"2025-04-21T14:17:28.248909Z","shell.execute_reply.started":"2025-04-21T14:17:28.240086Z","shell.execute_reply":"2025-04-21T14:17:28.247708Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"num_registros, num_colunas = df.shape\n\nprint(f\"Quantidade de registros: {num_registros}\")\nprint(f\"Quantidade de colunas: {num_colunas}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:28.250167Z","iopub.execute_input":"2025-04-21T14:17:28.250485Z","iopub.status.idle":"2025-04-21T14:17:28.260577Z","shell.execute_reply.started":"2025-04-21T14:17:28.250454Z","shell.execute_reply":"2025-04-21T14:17:28.259628Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def listar_correlacoes(df, campo_analise, threshold=0.5):\n    # Calcular a matriz de correlação\n    correlacoes = df.corr()\n    \n    # Selecionar as correlações da coluna de interesse e aplicar o threshold\n    correlacao_alvo = correlacoes[campo_analise]\n    correlacao_filtrada = correlacao_alvo[abs(correlacao_alvo) > threshold]\n    \n    # Remover a autocorrelação da própria coluna\n    correlacao_filtrada = correlacao_filtrada.drop(index=campo_analise)\n    \n    # Ordenar as correlações em ordem decrescente de valor absoluto\n    correlacao_ordenada = correlacao_filtrada.abs().sort_values(ascending=False)\n    \n    # Exibir o resultado\n    print(f\"Colunas com correlação maior que {threshold} com '{campo_analise}':\")\n    for coluna, valor in correlacao_ordenada.items():\n        print(f\"{coluna}: {valor:.3f}\")\n\ndf_temp = df[df.select_dtypes(include=['number']).columns.tolist()]\nlistar_correlacoes(df_temp, 'sii', threshold=0.7)","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:28.262788Z","iopub.execute_input":"2025-04-21T14:17:28.263601Z","iopub.status.idle":"2025-04-21T14:17:28.336790Z","shell.execute_reply.started":"2025-04-21T14:17:28.263551Z","shell.execute_reply":"2025-04-21T14:17:28.335755Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def analysis_pipeline(df1, df2, x, y, z):      \n    fig, axes = plt.subplots(1, 2, figsize=(20, 5))\n    \n    # Identificar possíveis outliers usando um scatter plot no primeiro subplot\n    scatter1 = axes[0].scatter(df1[x], df1[y], c=df1[z], cmap='viridis', s=50)\n    fig.colorbar(scatter1, ax=axes[0], label=z)\n    axes[0].set_title('DF Original')\n    axes[0].set_xlabel(x)\n    axes[0].set_ylabel(y)\n    \n    # Scatter plot no segundo subplot\n    scatter2 = axes[1].scatter(df2[x], df2[y], c=df2[z], cmap='viridis', s=50)\n    fig.colorbar(scatter2, ax=axes[1], label=z)\n    axes[1].set_title('DF com valores previstos')\n    axes[1].set_xlabel(x)\n    axes[1].set_ylabel(y)\n\n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:28.338476Z","iopub.execute_input":"2025-04-21T14:17:28.338792Z","iopub.status.idle":"2025-04-21T14:17:28.346579Z","shell.execute_reply.started":"2025-04-21T14:17:28.338759Z","shell.execute_reply":"2025-04-21T14:17:28.345363Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def statistical_analysis(df, coluna):\n    stats = df[coluna].describe()\n\n    # Calcular quantidade de nulos e percentual\n    missing_count = df[coluna].isnull().sum()\n    missing_percent = (missing_count / len(df)) * 100\n\n    # Adicionar informações de nulos ao DataFrame de estatísticas descritivas\n    stats['Valores Nulos'] = missing_count\n    stats['Percentual Nulos (%)'] = missing_percent\n    \n    # Calcular a correlação entre idade e sii\n    correlation = df[coluna].corr(df['sii'])\n    stats['Correlação (sii)'] = correlation\n    print(f'Analise estatística da coluna {coluna}')\n    print(stats)\n\nstatistical_analysis(df, 'sii')","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:28.347964Z","iopub.execute_input":"2025-04-21T14:17:28.348296Z","iopub.status.idle":"2025-04-21T14:17:28.367331Z","shell.execute_reply.started":"2025-04-21T14:17:28.348266Z","shell.execute_reply":"2025-04-21T14:17:28.365706Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def analysis_categorical_variables(df_original, coluna):\n    df = df_original.copy()\n    \n    # Atualiza tamanho padrão da fonte\n    plt.rcParams.update({'font.size': 20})\n    \n    fig1, axes1 = plt.subplots(1, 2, figsize=(30, 10))\n    \n    sns.kdeplot(data=df, x='Basic_Demos-Age', hue=coluna, fill=True, palette='viridis', ax=axes1[0])\n    axes1[0].set_title(f'Densidade de Idade por {coluna}')\n    \n    sns.countplot(x=coluna, data=df, palette='viridis', ax=axes1[1])\n    axes1[1].set_title(f'Frequência de Ocorrências de {coluna}')\n    total = len(df)\n    for p in axes1[1].patches:\n        altura = p.get_height()\n        percentual = f'{100 * altura / total:.1f}%'\n        axes1[1].text(p.get_x() + p.get_width() / 2, altura + 0.1, percentual, ha='center')\n    \n     \n    plt.savefig(f'/kaggle/working/{coluna}_fig1.png')\n    \n    fig2, axes2 = plt.subplots(1, 3, figsize=(30, 10))\n    sns.countplot(data=df, x=coluna, hue='Basic_Demos-Sex', palette=[\"#56B4E9\", \"#FBAFE4\"], ax=axes2[0])\n    axes2[0].set_title(f\"Distribuição de {coluna} por Sexo\")    \n\n    #df['Basic_Demos-Age_Zone'] = pd.cut(df['Basic_Demos-Age'], \n    #                                    bins=[0, 10, 15, 20, 25, 30], \n    #                                    labels=['1', '2', '3', '4', '5'])\n    sns.countplot(data=df, x='sii', hue=coluna, palette='pastel', ax=axes2[1])\n    axes2[1].set_title(f\"Distribuição do Target (sii) por {coluna}\")\n    \n    sns.countplot(data=df, x=coluna, hue='sii', palette='pastel', ax=axes2[2])\n    axes2[2].set_title(f\"Distribuição de {coluna} por Target (sii)\")\n    plt.savefig(f'/kaggle/working/{coluna}_fig2.png')\n    plt.show()\n\nanalysis_categorical_variables(df, 'sii')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:28.369426Z","iopub.execute_input":"2025-04-21T14:17:28.369748Z","iopub.status.idle":"2025-04-21T14:17:30.636474Z","shell.execute_reply.started":"2025-04-21T14:17:28.369715Z","shell.execute_reply":"2025-04-21T14:17:30.635196Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def analysis_continuous_variables(df_original, coluna):\n    df = df_original.copy()\n    fig1, axes1 = plt.subplots(1, 2, figsize=(20, 5))\n    \n    # Identificar possíveis outliers usando um boxplot\n    sns.boxplot(x=df[coluna], ax=axes1[0])\n    axes1[0].set_title(f'Boxplot da Variável {coluna}')\n    \n    # Histograma para visualizar a distribuição da idade\n    sns.histplot(df[coluna], bins=15, kde=True, ax=axes1[1])\n    axes1[1].set_title(f'Histograma da Variável {coluna}')\n\n    # Salvar o gráfico\n    plt.savefig(f'/kaggle/working/{coluna}_fig1.png')\n    \n    fig2, axes2 = plt.subplots(1, 3, figsize=(20, 5))\n    \n    # Gráfico de Violin Plot\n    sns.violinplot(x='sii', y=coluna, hue='Basic_Demos-Sex', data=df, \n                   split=True, palette=[\"#56B4E9\", \"#FBAFE4\"], ax=axes2[0])\n    axes2[0].set_title(f\"Gráfico de Violino de {coluna} por Sexo\")\n    \n    #sns.boxplot(x='sii', y=coluna, data=df, palette='pastel', ax=axes2[1])\n    sns.kdeplot(data=df, x=coluna, hue='sii', fill=True, palette='tab10', ax=axes2[1])\n    axes2[1].set_title(f\"Distribuição de {coluna} por Target (sii)\")\n\n    df['Basic_Demos-Age_Zone'] = pd.cut(df['Basic_Demos-Age'], \n                                        bins=[0, 6, 10, 13, 18, 30], \n                                        labels=['0-5', '6-10', '11-13', '14-18', '19-30'])\n    sns.kdeplot(data=df, x=coluna, hue='Basic_Demos-Age_Zone', fill=True, palette='tab10', ax=axes2[2])\n    axes2[2].set_title(f\"Distribuição de {coluna} por Faixa Etaria\")\n\n    # Salvar o gráfico\n    plt.savefig(f'/kaggle/working/{coluna}_fig2.png')\n\n    plt.show()\n\nanalysis_continuous_variables(df, 'Physical-BMI')","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:30.638374Z","iopub.execute_input":"2025-04-21T14:17:30.638730Z","iopub.status.idle":"2025-04-21T14:17:32.407773Z","shell.execute_reply.started":"2025-04-21T14:17:30.638695Z","shell.execute_reply":"2025-04-21T14:17:32.406713Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 'BIA-BIA_Activity_Level_num','BIA-BIA_BMI','BIA-BIA_FFMI',\n# 'BIA-BIA_FMI','BIA-BIA_Frame_num',\ncolumns = ['sii','BIA-BIA_BMR', 'BIA-BIA_DEE', 'BIA-BIA_ECW', \n           'BIA-BIA_FFM', 'BIA-BIA_Fat', 'BIA-BIA_ICW', 'BIA-BIA_LDM', \n           'BIA-BIA_LST', 'BIA-BIA_SMM', 'BIA-BIA_TBW']\n\ndf_temp =  df[columns]\ncorrelation_matrix = df_temp.corr()\nplt.figure(figsize=(10,8))\nsns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', fmt='.1f', vmin=-1, vmax=1)\n#plt.rcParams.update({'font.size': 14})\nplt.savefig('/kaggle/working/matrix-BIA.png')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:32.409001Z","iopub.execute_input":"2025-04-21T14:17:32.409308Z","iopub.status.idle":"2025-04-21T14:17:33.323678Z","shell.execute_reply.started":"2025-04-21T14:17:32.409277Z","shell.execute_reply":"2025-04-21T14:17:33.322632Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"columns = ['Physical-BMI','Physical-Weight',\n    'BIA-BIA_FFMI','BIA-BIA_FMI','BIA-BIA_TBW','BIA-BIA_BMC']\n\ndf_temp =  df[columns]\nindicadores = df_temp.describe()\nindicadores","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.326870Z","iopub.execute_input":"2025-04-21T14:17:33.327353Z","iopub.status.idle":"2025-04-21T14:17:33.356407Z","shell.execute_reply.started":"2025-04-21T14:17:33.327300Z","shell.execute_reply":"2025-04-21T14:17:33.355314Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"missing_data = df_temp.isnull().sum().sort_values(ascending=False)\nprint(missing_data)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.357884Z","iopub.execute_input":"2025-04-21T14:17:33.358265Z","iopub.status.idle":"2025-04-21T14:17:33.367709Z","shell.execute_reply.started":"2025-04-21T14:17:33.358229Z","shell.execute_reply":"2025-04-21T14:17:33.366616Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Ajusta para exibir todas as linhas\n# pd.set_option(\"display.max_rows\", None)\n\nmissing_percentage = (df_temp.isnull().sum() / len(df_temp)) * 100\nprint(missing_percentage.sort_values(ascending=False))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.369470Z","iopub.execute_input":"2025-04-21T14:17:33.369936Z","iopub.status.idle":"2025-04-21T14:17:33.377986Z","shell.execute_reply.started":"2025-04-21T14:17:33.369888Z","shell.execute_reply":"2025-04-21T14:17:33.376886Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"counts = df['sii'].value_counts()\nprint(counts)","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.379381Z","iopub.execute_input":"2025-04-21T14:17:33.379806Z","iopub.status.idle":"2025-04-21T14:17:33.392582Z","shell.execute_reply.started":"2025-04-21T14:17:33.379759Z","shell.execute_reply":"2025-04-21T14:17:33.391046Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Pré-processamento dos Dados","metadata":{}},{"cell_type":"markdown","source":"## Intervalo dos valores posso para cada coluna\n\n1. **Basic_Demos-Age:** Valores Impossíveis: < 0 ou > 120 (expectativa de vida máxima).\n2. **Physical-BMI e BIA-BIA_BMI (Índice de Massa Corporal - kg/m²):** Valores Impossíveis: < 10 (indicando desnutrição extrema) ou > 100 (IMC extremos em humanos raramente passam de 80-100, mesmo com obesidade severa).\n3. **Basic_Demos-Sex:** Valores Impossíveis: Qualquer valor que não seja 1 (masculino) ou 2 (feminino), se a codificação for binária.\n4. **Physical-Weight (Peso - lbs):** Valores Impossíveis: < 2 ou > 662 lbs (o peso humano mais alto já registrado é de aproximadamente 662 lbs).\n5. **BIA-BIA_DEE (Gasto Energético Diário - kcal/dia):** Valores Impossíveis: < 500 kcal/dia (metabolismo basal mínimo) ou > 10,000 kcal/dia (atividade física extrema, como de atletas olímpicos).\n6. **BIA-BIA_ECW (Água Extracelular - L):** Valores Impossíveis: < 3 L ou > 50 L (a água extracelular total raramente ultrapassa esse limite).\n7. **BIA-BIA_FFM (Massa Livre de Gordura - kg):** Valores Impossíveis: < 5 kg (desnutrição severa) ou > 150 kg (mesmo em fisiculturistas, o limite máximo é em torno de 70-100 kg).\n8. **BIA-BIA_FFMI (Índice de Massa Livre de Gordura - kg/m²):** Valores Impossíveis: < 10 ou > 35 (um FFMI acima de 25 já indica altíssimo nível de massa muscular, como em fisiculturistas).\n9. **BIA-BIA_FMI (Índice de Massa de Gordura - kg/m²):** Valores Impossíveis: < 1 ou > 30 (acima de 30 é raríssimo, mesmo em obesidade extrema).\n10. **BIA-BIA_Fat (Percentual de Gordura Corporal - %):** Valores Impossíveis: < 2% (inviável para sobrevivência humana) ou > 70% (obesidade extrema raramente ultrapassa 70%).\n11. **BIA-BIA_BMR (Taxa Metabólica Basal - kcal/dia):** Valores Impossíveis: < 500 kcal/dia (mínimo absoluto para metabolismo basal) ou > 5000 kcal/dia (mesmo para atletas, raramente excede esse valor).\n12. **BIA-BIA_ICW (Água Intracelular - L):** Valores Impossíveis: < 3 L ou > 70 L.\n14. **BIA-BIA_TBW (Água Corporal Total - L):** Valores Impossíveis: < 5 L ou > 100 L.\n15. **Physical-Height (Altura - in):** Valores Impossíveis: < 11 in (não compatível com sobrevivência) ou > 108 in (o humano mais alto registrado tinha 108 in).\n16. **BIA-BIA_LST (Tecido Mole Magro - kg):** Valores Impossíveis: < 5 kg ou > 150 kg.\n17. **BIA-BIA_SMM (Massa Muscular Esquelética - kg):** Valores Impossíveis: < 5 kg ou > 100 kg (mesmo em fisiculturistas, o limite é próximo de 50-70 kg).\n18. **BIA-BIA_LDM (Massa Magra Seca - kg):** Valores Impossíveis: < 5 kg ou > 70 kg.\n1. **BIA-BIA_BMC ((Conteúdo Mineral Ósseo):** Valores Impossíveis: < 0 kg ou > 15 kg, Mesmo em indivíduos muito altos ou musculosos, o conteúdo mineral ósseo dificilmente ultrapassa 10-12 kg, com 15 kg sendo um limite extremo.","metadata":{}},{"cell_type":"code","source":"class RangePossibleValues(BaseEstimator, TransformerMixin):\n    def __init__(self):\n        pass  \n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n        \n        X.loc[X['Physical-BMI'] < 10, 'Physical-BMI'] = np.nan\n        X.loc[X['Physical-BMI'] > 100, 'Physical-BMI'] = np.nan\n        \n        X.loc[X['Physical-Weight'] < 2, 'Physical-Weight'] = np.nan\n        X.loc[X['Physical-Weight'] > 400, 'Physical-Weight'] = np.nan\n        \n        X.loc[X['Physical-Height'] < 11, 'Physical-Height'] = np.nan\n        X.loc[X['Physical-Height'] > 108, 'Physical-Height'] = np.nan\n        \n        X.loc[X['BIA-BIA_BMI'] < 10, 'BIA-BIA_BMI'] = np.nan\n        X.loc[X['BIA-BIA_BMI'] > 100, 'BIA-BIA_BMI'] = np.nan\n\n        X.loc[X['BIA-BIA_DEE'] < 500, 'BIA-BIA_DEE'] = np.nan\n        X.loc[X['BIA-BIA_DEE'] > 10000, 'BIA-BIA_DEE'] = np.nan\n        \n        X.loc[X['BIA-BIA_ECW'] < 3, 'BIA-BIA_ECW'] = np.nan\n        X.loc[X['BIA-BIA_ECW'] > 50, 'BIA-BIA_ECW'] = np.nan\n        \n        X.loc[X['BIA-BIA_FFM'] < 5, 'BIA-BIA_FFM'] = np.nan\n        X.loc[X['BIA-BIA_FFM'] > 130, 'BIA-BIA_FFM'] = np.nan\n        \n        X.loc[X['BIA-BIA_FFMI'] < 10, 'BIA-BIA_FFMI'] = np.nan\n        X.loc[X['BIA-BIA_FFMI'] > 25, 'BIA-BIA_FFMI'] = np.nan\n        \n        X.loc[X['BIA-BIA_FMI'] < 1, 'BIA-BIA_FMI'] = np.nan\n        X.loc[X['BIA-BIA_FMI'] > 30, 'BIA-BIA_FMI'] = np.nan\n        \n        X.loc[X['BIA-BIA_Fat'] < 2, 'BIA-BIA_Fat'] = np.nan\n        X.loc[X['BIA-BIA_Fat'] > 70, 'BIA-BIA_Fat'] = np.nan\n        \n        X.loc[X['BIA-BIA_BMR'] < 500, 'BIA-BIA_BMR'] = np.nan\n        X.loc[X['BIA-BIA_BMR'] > 5000, 'BIA-BIA_BMR'] = np.nan\n        \n        X.loc[X['BIA-BIA_ICW'] < 3, 'BIA-BIA_ICW'] = np.nan\n        X.loc[X['BIA-BIA_ICW'] > 70, 'BIA-BIA_ICW'] = np.nan\n        \n        X.loc[X['BIA-BIA_TBW'] < 5, 'BIA-BIA_TBW'] = np.nan\n        X.loc[X['BIA-BIA_TBW'] > 100, 'BIA-BIA_TBW'] = np.nan\n        \n        X.loc[X['BIA-BIA_LST'] < 5, 'BIA-BIA_LST'] = np.nan\n        X.loc[X['BIA-BIA_LST'] > 150, 'BIA-BIA_LST'] = np.nan\n        \n        X.loc[X['BIA-BIA_SMM'] < 5, 'BIA-BIA_SMM'] = np.nan\n        X.loc[X['BIA-BIA_SMM'] > 100, 'BIA-BIA_SMM'] = np.nan\n        \n        X.loc[X['BIA-BIA_LDM'] < 5, 'BIA-BIA_LDM'] = np.nan\n        X.loc[X['BIA-BIA_LDM'] > 70, 'BIA-BIA_LDM'] = np.nan\n        \n        X.loc[X['BIA-BIA_BMC'] < 0, 'BIA-BIA_BMC'] = np.nan\n        X.loc[X['BIA-BIA_BMC'] > 15, 'BIA-BIA_BMC'] = np.nan\n\n        X.loc[X['FGC-FGC_SRL'] < 1, 'FGC-FGC_SRL'] = np.nan\n        X.loc[X['FGC-FGC_SRR'] < 1, 'FGC-FGC_SRR'] = np.nan\n        X.loc[X['FGC-FGC_TL'] < 1, 'FGC-FGC_TL'] = np.nan\n        X.loc[X['FGC-FGC_CU'] < 1, 'FGC-FGC_CU'] = np.nan\n        X.loc[X['FGC-FGC_PU'] < 1, 'FGC-FGC_PU'] = np.nan\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.394367Z","iopub.execute_input":"2025-04-21T14:17:33.394813Z","iopub.status.idle":"2025-04-21T14:17:33.413102Z","shell.execute_reply.started":"2025-04-21T14:17:33.394762Z","shell.execute_reply":"2025-04-21T14:17:33.412141Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Transformador para remover colunas específicas\nclass DropColumns(BaseEstimator, TransformerMixin):\n    def __init__(self, columns):\n        self.columns = columns\n        \n    def fit(self, X, y=None):\n        return self\n    \n    def transform(self, X):\n        return X.drop(columns=self.columns)  ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.414584Z","iopub.execute_input":"2025-04-21T14:17:33.415015Z","iopub.status.idle":"2025-04-21T14:17:33.431210Z","shell.execute_reply.started":"2025-04-21T14:17:33.414946Z","shell.execute_reply":"2025-04-21T14:17:33.430143Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Transformador para normalizar colunas discretas\nclass Scaler(BaseEstimator, TransformerMixin):\n    def __init__(self, columns):\n        self.columns = columns\n        self.scaler = MinMaxScaler(feature_range=(-1, 1)) \n    \n    def fit(self, X, y=None):\n        # Ajusta o scaler apenas nas colunas passadas\n        self.scaler.fit(X[self.columns])\n        return self\n    \n    def transform(self, X):\n        X = X.copy()\n        X[self.columns] = self.scaler.transform(X[self.columns])\n        return X","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.435715Z","iopub.execute_input":"2025-04-21T14:17:33.436160Z","iopub.status.idle":"2025-04-21T14:17:33.445581Z","shell.execute_reply.started":"2025-04-21T14:17:33.436126Z","shell.execute_reply":"2025-04-21T14:17:33.444662Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def categorize_by_age(row, column, bins_dict, labels):\n    age = row['Basic_Demos-Age']\n    value = row[column]\n\n    if age <= 5:\n        bins = bins_dict['0-5']\n    elif age <= 9:\n        bins = bins_dict['6-9']\n    elif age <= 12:\n        bins = bins_dict['10-12']\n    elif age <= 17:\n        bins = bins_dict['13-17']\n    else:\n        bins = bins_dict['18+']\n\n    return pd.cut([value], bins=bins, labels=labels, right=False)[0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.446866Z","iopub.execute_input":"2025-04-21T14:17:33.447215Z","iopub.status.idle":"2025-04-21T14:17:33.458031Z","shell.execute_reply.started":"2025-04-21T14:17:33.447183Z","shell.execute_reply":"2025-04-21T14:17:33.457037Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class CategorizeByAge(BaseEstimator, TransformerMixin):\n    def __init__(self):\n        pass  \n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n\n        bins_pu = {\n            '0-5':   [0, 1, 3, 7, 9, np.inf],\n            '6-9':   [0, 2, 4, 9, 10, np.inf],\n            '10-12': [0, 3, 6, 12, 15, np.inf],\n            '13-17': [0, 4, 7, 16, 20, np.inf],\n            '18+':   [0, 5, 7, 23, 28, np.inf]\n        }\n\n        bins_cu = {\n            '0-5':   [0, 1, 2, 4, 6, np.inf],\n            '6-9':   [0, 2, 3, 6, 8, np.inf],\n            '10-12': [0, 5, 10, 19, 26, np.inf],\n            '13-17': [0, 10, 18, 30, 35, np.inf],\n            '18+':   [0, 11, 20, 28, 31, np.inf]\n        }\n\n        bins_srl = {\n            '0-5':   [0, 5, 7, 10, 12, np.inf],\n            '6-9':   [0, 5, 7, 10, 12, np.inf],\n            '10-12': [0, 5, 7, 10, 12, np.inf],\n            '13-17': [0, 5, 7, 10, 12, np.inf],\n            '18+':   [0, 5, 7, 10, 12, np.inf]\n        }\n\n        bins_tl = {\n            '0-5':   [0, 3, 5, 9, 11, np.inf],\n            '6-9':   [0, 3, 5, 9, 11, np.inf],\n            '10-12': [0, 6, 8, 10, 12, np.inf],\n            '13-17': [0, 6, 8, 11, 12, np.inf],\n            '18+':   [0, 7, 9, 11, 12, np.inf]\n        }\n        \n        bins_srr = {\n            '0-5':   [0, 5, 7, 10, 12, np.inf],\n            '6-9':   [0, 5, 7, 10, 12, np.inf],\n            '10-12': [0, 5, 7, 10, 12, np.inf],\n            '13-17': [0, 5, 7, 10, 12, np.inf],\n            '18+':   [0, 5, 7, 10, 12, np.inf]\n        }\n\n        \n\n        bins_dee = {'0-5': [0, 500, 800, 1200, 1500, np.inf],'6-9': [0, 800, 1200, 1600, 2000, np.inf],\n            '10-12': [0, 1000, 1500, 2000, 2500, np.inf],'13-17': [0, 1200, 1800, 2500, 3000, np.inf],\n            '18+': [0, 1500, 2000, 2500, 3500, np.inf]\n        }\n\n        bins_ffm = {'0-5': [0, 8, 15, 20, 25, np.inf],'6-9': [0, 15, 20, 25, 35, np.inf],\n            '10-12': [0, 20, 25, 35, 45, np.inf],'13-17': [0, 25, 35, 45, 55, np.inf],\n            '18+': [0, 30, 45, 60, 75, np.inf]\n        }\n\n        bins_ffmi = {'0-5': [0, 10, 14, 16, 18, np.inf],'6-9': [0, 14, 16, 18, 20, np.inf],\n            '10-12': [0, 16, 18, 20, 22, np.inf],'13-17': [0, 18, 20, 22, 24, np.inf],\n            '18+': [0, 20, 22, 24, 26, np.inf]\n        }\n\n        bins_fmi = {'0-5': [0, 1, 2, 4, 6, np.inf],'6-9': [0, 2, 4, 6, 8, np.inf],\n            '10-12': [0, 3, 5, 8, 10, np.inf],'13-17': [0, 4, 7, 10, 12, np.inf],\n            '18+': [0, 5, 8, 10, 15, np.inf]\n        }\n\n        bins_fat = {'0-5': [0, 10, 15, 20, 25, np.inf],'6-9': [0, 15, 20, 25, 30, np.inf],\n            '10-12': [0, 20, 25, 30, 35, np.inf],'13-17': [0, 25, 30, 35, 40, np.inf],\n            '18+': [0, 30, 35, 40, 50, np.inf]\n        }\n\n        bins_bmr = {'0-5': [0, 500, 800, 1200, 1500, np.inf],'6-9': [0, 800, 1000, 1500, 2000, np.inf],\n            '10-12': [0, 1000, 1500, 2000, 2500, np.inf],'13-17': [0, 1200, 1800, 2500, 3000, np.inf],\n            '18+': [0, 1500, 2000, 2500, 3500, np.inf]\n        }\n        \n        bins_tbw = {'0-5': [0, 5, 10, 15, 20, np.inf],'6-9': [0, 10, 15, 20, 25, np.inf],\n            '10-12': [0, 15, 20, 25, 30, np.inf],'13-17': [0, 20, 25, 30, 40, np.inf],\n            '18+': [0, 25, 30, 40, 50, np.inf]\n        }\n        \n        bins_lst = {'0-5': [0, 8, 15, 20, 25, np.inf],'6-9': [0, 15, 20, 25, 35, np.inf],\n            '10-12': [0, 20, 25, 35, 45, np.inf],'13-17': [0, 25, 35, 45, 55, np.inf],\n            '18+': [0, 30, 45, 60, 75, np.inf]\n        }\n        \n        bins_bmc = {'0-5': [0, 0.5, 1, 2, 3, np.inf],'6-9': [0, 1, 2, 3, 4, np.inf],\n            '10-12': [0, 1.5, 2.5, 3.5, 5, np.inf],'13-17': [0, 2, 3.5, 5, 7, np.inf],\n            '18+': [0, 2.5, 4, 6, 8, np.inf]\n        }\n\n        bins_weight = {'0-5': [0, 20, 40, 60, 80, np.inf],'6-9': [0, 40, 60, 80, 100, np.inf],\n            '10-12': [0, 60, 80, 100, 120, np.inf],'13-17': [0, 80, 120, 160, 200, np.inf],\n            '18+': [0, 100, 150, 200, 300, np.inf]\n        }\n        \n        bins_height = {'0-5': [0, 20, 30, 40, 50, np.inf],'6-9': [0, 40, 50, 60, 65, np.inf],\n            '10-12': [0, 48, 55, 63, 68, np.inf],'13-17': [0, 55, 63, 68, 72, np.inf],\n            '18+': [0, 60, 65, 70, 75, np.inf]\n        }\n                \n        bins_heartrate = {'0-5': [0, 70, 80, 120, 140, np.inf],'6-9': [0, 60, 70, 110, 130, np.inf],\n            '10-12': [0, 55, 65, 100, 120, np.inf],'13-17': [0, 50, 60, 90, 110, np.inf],\n            '18+': [0, 50, 60, 100, 120, np.inf]\n        }\n        \n        labels5 = [0, 1, 2, 3, 4]\n        X['FGC-FGC_PU_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'FGC-FGC_PU', bins_pu, labels5), axis=1)\n        X['FGC-FGC_CU_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'FGC-FGC_CU', bins_cu, labels5), axis=1)\n        X['FGC-FGC_SRR_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'FGC-FGC_SRR', bins_srr, labels5), axis=1)\n        X['FGC-FGC_SRL_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'FGC-FGC_SRL', bins_srl, labels5), axis=1)\n        X['FGC-FGC_TL_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'FGC-FGC_TL', bins_tl, labels5), axis=1)\n\n        X['BIA-BIA_DEE_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_DEE', bins_dee, labels5), axis=1)\n        X['BIA-BIA_FFM_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_FFM', bins_ffm, labels5), axis=1)\n        X['BIA-BIA_FFMI_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_FFMI', bins_ffmi, labels5), axis=1)\n        X['BIA-BIA_FMI_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_FMI', bins_fmi, labels5), axis=1)\n        X['BIA-BIA_Fat_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_Fat', bins_fat, labels5), axis=1)\n        X['BIA-BIA_BMR_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_BMR', bins_bmr, labels5), axis=1)\n        X['BIA-BIA_TBW_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_TBW', bins_tbw, labels5), axis=1)\n        X['BIA-BIA_LST_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_LST', bins_lst, labels5), axis=1)\n        X['BIA-BIA_BMC_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_BMC', bins_bmc, labels5), axis=1)\n\n        X['Physical-Weight_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'Physical-Weight', bins_weight, labels5), axis=1)\n        X['Physical-Height_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'Physical-Height', bins_height, labels5), axis=1)\n        X['Physical-HeartRate_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'Physical-HeartRate', bins_heartrate, labels5), axis=1)\n\n        bins_ecw = {\n            '0-5':   [0, 6, 8, 9, np.inf],\n            '6-9':   [0, 7, 9, 16, np.inf],\n            '10-12': [0, 13, 16, 26, np.inf],\n            '13-17': [0, 22, 27, 37, np.inf],\n            '18+':   [0, 28, 34, 44, np.inf]\n        }\n\n        bins_icw = {\n            '0-5':   [0, 20, 22, 23, np.inf],\n            '6-9':   [0, 22, 27, 29, np.inf],\n            '10-12': [0, 26, 29, 37, np.inf],\n            '13-17': [0, 33, 38, 50, np.inf],\n            '18+':   [0, 38, 44, 50, np.inf]\n        }\n\n        labels4 = [0, 1, 2, 3]\n        X['BIA-BIA_ECW_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_ECW', bins_ecw, labels4), axis=1)\n        X['BIA-BIA_ICW_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'BIA-BIA_ICW', bins_icw, labels4), axis=1)\n        \n        bins_systolic = {\n            '0-5': [0, 70, 80, 100, 120, 140, np.inf],\n            '6-9': [0, 80, 90, 110, 120, 140, np.inf],\n            '10-12': [0, 90, 100, 120, 130, 150, np.inf],\n            '13-17': [0, 100, 110, 130, 140, 160, np.inf],\n            '18+': [0, 110, 120, 130, 140, 160, np.inf]\n        }\n\n        bins_diastolic = {\n            '0-5': [0, 40, 50, 70, 80, 90, np.inf],\n            '6-9': [0, 50, 60, 80, 90, 100, np.inf],\n            '10-12': [0, 60, 70, 80, 90, 100, np.inf],\n            '13-17': [0, 60, 70, 85, 90, 100, np.inf],\n            '18+': [0, 60, 70, 80, 90, 100, np.inf]\n        }\n        \n        labels6 = [0, 1, 2, 3, 4, 5]\n        X['Physical-Systolic_BP_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'Physical-Systolic_BP', bins_systolic, labels6), axis=1)\n        X['Physical-Diastolic_BP_Zone2'] = X.apply(lambda row: categorize_by_age(row, 'Physical-Diastolic_BP', bins_diastolic, labels6), axis=1)\n        \n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.459488Z","iopub.execute_input":"2025-04-21T14:17:33.459802Z","iopub.status.idle":"2025-04-21T14:17:33.502726Z","shell.execute_reply.started":"2025-04-21T14:17:33.459771Z","shell.execute_reply":"2025-04-21T14:17:33.501782Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class CreatingCategoricalVariables(BaseEstimator, TransformerMixin):\n    def __init__(self):\n        pass  \n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n        \n        X['Basic_Demos-Age_Zone'] = pd.cut(X['Basic_Demos-Age'], bins=[0, 6, 10, 13, 18, float('inf')], labels=[0, 1, 2, 3, 4])\n        X['BMI_Zone'] = pd.cut(X['Physical-BMI'], bins=[-np.inf, 16, 18.4, 24.9, 29.9, 34.9, 39.9, np.inf], labels=[0, 1, 2, 3, 4, 5, 6])\n        \n        X['PAQ_Total_Zone'] = pd.cut(X['PAQ_Total'], bins=[-float('inf'), 1.9, 2.9, 3.9, float('inf')], labels=[0, 1, 2, 3])\n        X['CGAS-CGAS_Score_Zone'] = pd.cut(X['CGAS-CGAS_Score'], bins=[-float('inf'), 31, 51, 71, 91, float('inf')], labels=[0, 1, 2, 3, 4])\n        \n        X['SDS-SDS_Total_T_Zone'] = pd.cut(X['SDS-SDS_Total_T'], bins=[-float('inf'), 40, 60, float('inf')], labels=[0, 1, 2])\n        X['SDS-SDS_Total_Raw_Zone'] = pd.cut(X['SDS-SDS_Total_Raw'], bins=[-float('inf'), 50, 60, float('inf')], labels=[0, 1, 2])\n        \n        #X['BIA-BIA_DEE_Zone'] = pd.cut(X['BIA-BIA_DEE'], bins=[-float('inf'), 1500, 2001, 2501, 3500, float('inf')], labels=[0, 1, 2, 3, 4])\n        #X['BIA-BIA_ECW_Zone'] = pd.cut(X['BIA-BIA_ECW'], bins=[-float('inf'), 5, 10, 20, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_ICW_Zone'] = pd.cut(X['BIA-BIA_ICW'], bins=[-float('inf'), 5, 10, 20, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_FFM_Zone'] = pd.cut(X['BIA-BIA_FFM'], bins=[-float('inf'), 20, 41, 70, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_FFMI_Zone'] = pd.cut(X['BIA-BIA_FFMI'], bins=[-float('inf'), 14, 18, 25, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_FMI'] = pd.cut(X['BIA-BIA_FMI'], bins=[-float('inf'), 2, 4, 10, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_Fat_Zone'] = pd.cut(X['BIA-BIA_Fat'], bins=[-float('inf'), 10, 21, 31, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_BMR_Zone'] = pd.cut(X['BIA-BIA_BMR'], bins=[-float('inf'), 1000, 1500, 2500, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_TBW_Zone'] = pd.cut(X['BIA-BIA_TBW'], bins=[-float('inf'), 20, 30, 50, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_LST_Zone'] = pd.cut(X['BIA-BIA_LST'], bins=[-float('inf'), 10, 31, 70, float('inf')], labels=[0, 1, 2, 3])\n        #X['BIA-BIA_BMC_Zone'] = pd.cut(X['BIA-BIA_BMC'], bins=[-float('inf'), 2, 4, 8, 12, float('inf')], labels=[0, 1, 2, 3, 4])\n        \n        #X['Physical-Systolic_BP_Zone'] = pd.cut(X['Physical-Systolic_BP'], bins=[-float('inf'), 120, 130, 140, 180, float('inf')], labels=[0, 1, 2, 3, 4])\n        #X['Physical-Diastolic_BP_Zone'] = pd.cut(X['Physical-Diastolic_BP'], bins=[-float('inf'), 80, 90, 100, 119, float('inf')], labels=[0, 1, 2, 3, 4])\n        #X['Physical-HeartRate_Zone'] = pd.cut(X['Physical-HeartRate'], bins=[-float('inf'), 60, 101, 120, float('inf')], labels=[0, 1, 2, 3])\n        \n        \n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.504178Z","iopub.execute_input":"2025-04-21T14:17:33.504485Z","iopub.status.idle":"2025-04-21T14:17:33.520429Z","shell.execute_reply.started":"2025-04-21T14:17:33.504453Z","shell.execute_reply":"2025-04-21T14:17:33.518657Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class CombinePAQColumns(BaseEstimator, TransformerMixin):\n    def __init__(self, adolescent_column, child_column, new_column):\n        self.adolescent_column = adolescent_column\n        self.child_column = child_column\n        self.new_column = new_column\n\n    def fit(self, X, y=None):\n        return self\n    \n    def transform(self, X):\n        X = X.copy()\n        \n        X[self.new_column] = np.nan\n        X.loc[~X[self.adolescent_column].isna(), self.new_column] = X[self.adolescent_column]\n        X.loc[~X[self.child_column].isna(), self.new_column] = X[self.child_column]\n\n        # Remover as colunas originais\n        X = X.drop(columns=[self.adolescent_column, self.child_column])\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.522135Z","iopub.execute_input":"2025-04-21T14:17:33.522584Z","iopub.status.idle":"2025-04-21T14:17:33.537223Z","shell.execute_reply.started":"2025-04-21T14:17:33.522533Z","shell.execute_reply":"2025-04-21T14:17:33.536174Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class TreatSeasonColumns(BaseEstimator, TransformerMixin):\n    def __init__(self, columns):\n        self.columns = columns\n\n    def fit(self, X, y=None):\n        return self\n    \n    def transform(self, X):\n        X = X.copy()\n        for column in self.columns:\n            X[column] = X[column].replace({\n                'Spring': 1,\n                'Summer': 2,\n                'Fall': 3,\n                'Winter': 4\n            })\n            X[column] = pd.to_numeric(X[column], errors='coerce').astype('Int64')\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.538709Z","iopub.execute_input":"2025-04-21T14:17:33.539170Z","iopub.status.idle":"2025-04-21T14:17:33.550086Z","shell.execute_reply.started":"2025-04-21T14:17:33.539123Z","shell.execute_reply":"2025-04-21T14:17:33.548996Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Transformador para imputação com LogisticRegression\nclass PipModeloLogisticRegression(BaseEstimator, TransformerMixin):\n    def __init__(self, faltante):\n        self.faltante = faltante\n        self.model = LogisticRegression(class_weight='balanced',solver='liblinear',random_state=42)\n\n    def fit(self, X, y=None):\n        # Identificar colunas sem valores nulos, exceto a coluna com valores nulos\n        self.colunas_sem_nulos = X.drop(columns=[self.faltante]).dropna(axis=1).columns.tolist()\n        \n        # Criar um subset sem valores nulos nas colunas correlacionadas e na coluna faltante\n        #df_temp = X[self.colunas_correlacionadas + [self.faltante]].dropna()\n        df_temp = X[self.colunas_sem_nulos + [self.faltante]].dropna()\n\n        # Treina o modelo apenas se houver dados suficientes após a remoção de outliers\n        if df_temp.shape[0] > 0:\n            #self.model.fit(df_temp[self.colunas_correlacionadas], df_temp[self.faltante])\n            self.model.fit(df_temp[self.colunas_sem_nulos], df_temp[self.faltante])\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n\n        # Identificar linhas com valores ausentes na faltante\n        missing_data = X[self.faltante].isna()\n        if missing_data.sum() > 0:\n            # Preencher os valores ausentes com as previsões do modelo\n            #X.loc[missing_data, self.faltante] = self.model.predict(X[self.colunas_correlacionadas][missing_data])\n            X.loc[missing_data, self.faltante] = self.model.predict(X[self.colunas_sem_nulos][missing_data])\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.551432Z","iopub.execute_input":"2025-04-21T14:17:33.551758Z","iopub.status.idle":"2025-04-21T14:17:33.570267Z","shell.execute_reply.started":"2025-04-21T14:17:33.551717Z","shell.execute_reply":"2025-04-21T14:17:33.569246Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Transformador para imputação com\nclass PipModeloMLPRegressor(BaseEstimator, TransformerMixin):\n    def __init__(self, faltante, hidden_layer_sizes=(100,), random_state=42, **model_params):\n        self.faltante = faltante\n        self.hidden_layer_sizes = hidden_layer_sizes\n        self.random_state = random_state\n        self.model_params = model_params\n        self.model = MLPRegressor(hidden_layer_sizes=self.hidden_layer_sizes,\n                                  random_state=self.random_state,\n                                  **self.model_params)\n\n    def fit(self, X, y=None):\n        # Verifica se X é um DataFrame ou um array NumPy\n        #if isinstance(X, np.ndarray):\n        #    X = pd.DataFrame(X, columns=self.feature_names_in_)  # Reconstrói como DataFrame\n        \n        # Identificar colunas sem valores nulos, exceto a coluna faltante\n        self.colunas_sem_nulos = X.drop(columns=[self.faltante]).dropna(axis=1).columns.tolist()\n\n        # Criar um subset de treino removendo linhas com valores nulos na coluna faltante\n        df_temp = X[self.colunas_sem_nulos + [self.faltante]].dropna()\n\n        # Treinar o modelo apenas se houver dados suficientes\n        if df_temp.shape[0] > 0:\n            self.model.fit(df_temp[self.colunas_sem_nulos], df_temp[self.faltante])\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n\n        # Identificar linhas com valores ausentes na coluna faltante\n        missing_data = X[self.faltante].isna()\n        if missing_data.sum() > 0:\n            # Preencher os valores ausentes usando as colunas sem nulos\n            X.loc[missing_data, self.faltante] = self.model.predict(X[self.colunas_sem_nulos][missing_data])\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.573743Z","iopub.execute_input":"2025-04-21T14:17:33.574175Z","iopub.status.idle":"2025-04-21T14:17:33.587752Z","shell.execute_reply.started":"2025-04-21T14:17:33.574141Z","shell.execute_reply":"2025-04-21T14:17:33.586888Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Transformador para imputação com LinearRegression\nclass PipModeloRegression(BaseEstimator, TransformerMixin):\n    def __init__(self, faltante, limite_min=None, limite_max=None):\n        self.faltante = faltante\n        self.model = LinearRegression()\n        self.limite_min = limite_min\n        self.limite_max = limite_max\n\n    def fit(self, X, y=None):\n        # Identificar colunas sem valores nulos, exceto a coluna faltante\n        self.colunas_sem_nulos = X.drop(columns=[self.faltante]).dropna(axis=1).columns.tolist()\n\n        # Criar um subset de treino removendo linhas com valores nulos na coluna faltante\n        df_temp = X[self.colunas_sem_nulos + [self.faltante]].dropna()\n        \n        # Treinar o modelo apenas se houver dados suficientes\n        if df_temp.shape[0] > 0:\n            self.model.fit(df_temp[self.colunas_sem_nulos], df_temp[self.faltante])\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n        \n        # Identificar linhas com valores ausentes na coluna faltante\n        missing_data = X[self.faltante].isna()\n        if missing_data.sum() > 0:\n            # Preencher os valores ausentes usando as colunas sem nulos\n            #X.loc[missing_data, self.faltante] = self.model.predict(X[self.colunas_sem_nulos][missing_data])\n            \n            # Prever valores ausentes\n            previsoes = self.model.predict(X[self.colunas_sem_nulos][missing_data])\n            \n            # Aplicar limites se definidos\n            if self.limite_min is not None or self.limite_max is not None:\n                previsoes = np.clip(previsoes, self.limite_min, self.limite_max)\n            \n            # Substituir valores ausentes pela previsão ajustada\n            X.loc[missing_data, self.faltante] = previsoes\n            \n            #print(f'{self.faltante}: {self.colunas_sem_nulos}')\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.589267Z","iopub.execute_input":"2025-04-21T14:17:33.590337Z","iopub.status.idle":"2025-04-21T14:17:33.604803Z","shell.execute_reply.started":"2025-04-21T14:17:33.590277Z","shell.execute_reply":"2025-04-21T14:17:33.603911Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class PreencherIMC(BaseEstimator, TransformerMixin):\n    def __init__(self, coluna_imc, coluna_nova):\n        self.coluna_peso = 'Physical-Weight'\n        self.coluna_altura = 'Physical-Height'\n        self.coluna_nova = coluna_nova\n        self.coluna_imc = coluna_imc\n\n    def fit(self, X, y=None):\n        # Não há necessidade de aprendizado para esta transformação\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n        \n        # Inicializar a coluna de marcação\n        X[self.coluna_nova] = False\n        \n        # Calcular IMC apenas para linhas onde a coluna IMC é nula\n        missing_imc = X[self.coluna_imc].isna()\n        if missing_imc.sum() > 0:\n            peso_kg = X.loc[missing_imc, self.coluna_peso] * 0.453592\n            altura_m = X.loc[missing_imc, self.coluna_altura] * 0.0254\n            X.loc[missing_imc, self.coluna_imc] = peso_kg / (altura_m ** 2)\n            \n            # Marcar as linhas onde o IMC foi calculado\n            X.loc[missing_imc, self.coluna_nova] = True\n        \n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.606168Z","iopub.execute_input":"2025-04-21T14:17:33.606462Z","iopub.status.idle":"2025-04-21T14:17:33.622293Z","shell.execute_reply.started":"2025-04-21T14:17:33.606433Z","shell.execute_reply":"2025-04-21T14:17:33.621150Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class CalculaIMC_All(BaseEstimator, TransformerMixin):\n    def __init__(self, coluna_imc):\n        self.coluna_peso = 'Physical-Weight'\n        self.coluna_altura = 'Physical-Height'\n        self.coluna_imc = coluna_imc\n\n    def fit(self, X, y=None):\n        # Não há necessidade de aprendizado para esta transformação\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n        peso_kg = X[self.coluna_peso] * 0.453592\n        altura_m = X[self.coluna_altura] * 0.0254\n        X[self.coluna_imc] = peso_kg / (altura_m ** 2)\n        \n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.623472Z","iopub.execute_input":"2025-04-21T14:17:33.623770Z","iopub.status.idle":"2025-04-21T14:17:33.642367Z","shell.execute_reply.started":"2025-04-21T14:17:33.623740Z","shell.execute_reply":"2025-04-21T14:17:33.641280Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class CalculaIMC(BaseEstimator, TransformerMixin):\n    def __init__(self, coluna_imc=\"IMC\"):\n        self.coluna_peso = 'Physical-Weight'\n        self.coluna_altura = 'Physical-Height'\n        self.coluna_imc = coluna_imc\n\n    def fit(self, X, y=None):\n        # Nenhum ajuste necessário, pois o cálculo é determinístico\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n\n        # Identificar linhas onde a altura é nula, mas peso e IMC estão preenchidos\n        missing = X[self.coluna_imc].isna() & X[self.coluna_peso].notna() & X[self.coluna_altura].notna()\n\n        # Calcular altura para essas linhas\n        if missing.sum() > 0:\n            peso_kg = X.loc[missing, self.coluna_peso] * 0.453592  # Converter peso para kg\n            altura_m = X.loc[missing, self.coluna_altura] * 0.0254  # Converter altura para m\n            imc = peso_kg / (altura_m ** 2)\n            \n            X.loc[missing, self.coluna_imc] = imc\n        return X","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.643734Z","iopub.execute_input":"2025-04-21T14:17:33.644327Z","iopub.status.idle":"2025-04-21T14:17:33.655114Z","shell.execute_reply.started":"2025-04-21T14:17:33.644275Z","shell.execute_reply":"2025-04-21T14:17:33.654119Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class CalcularAltura(BaseEstimator, TransformerMixin):\n    def __init__(self, coluna_imc=\"IMC\"):\n        self.coluna_peso = 'Physical-Weight'\n        self.coluna_altura = 'Physical-Height'\n        self.coluna_imc = coluna_imc\n\n    def fit(self, X, y=None):\n        # Nenhum ajuste necessário, pois o cálculo é determinístico\n        return self\n\n    def transform(self, X):\n        X = X.copy()\n\n        # Identificar linhas onde a altura é nula, mas peso e IMC estão preenchidos\n        missing_altura = X[self.coluna_altura].isna() & X[self.coluna_peso].notna() & X[self.coluna_imc].notna()\n\n        # Calcular altura para essas linhas\n        if missing_altura.sum() > 0:\n            peso_kg = X.loc[missing_altura, self.coluna_peso] * 0.453592  # Converter peso para kg\n            altura_m = np.sqrt(peso_kg / X.loc[missing_altura, self.coluna_imc])  # Calcular altura em metros\n            X.loc[missing_altura, self.coluna_altura] = altura_m * 39.3701  # Converter altura para polegadas\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.656639Z","iopub.execute_input":"2025-04-21T14:17:33.657640Z","iopub.status.idle":"2025-04-21T14:17:33.670254Z","shell.execute_reply.started":"2025-04-21T14:17:33.657590Z","shell.execute_reply":"2025-04-21T14:17:33.669307Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class ScalerWithInverse(BaseEstimator, TransformerMixin):\n    def __init__(self, columns, feature_range=(-1, 1)):\n        self.columns = columns\n        self.scaler = MinMaxScaler(feature_range=feature_range)\n\n    def fit(self, X, y=None):\n        # Ajusta o scaler apenas para as colunas especificadas\n        self.scaler.fit(X[self.columns])\n        return self\n\n    def transform(self, X):\n        # Aplica a normalização nas colunas especificadas\n        X = X.copy()\n        X[self.columns] = self.scaler.transform(X[self.columns])\n        return X\n\n    def inverse_transform(self, X):\n        # Reverte a normalização nas colunas especificadas\n        X = X.copy()\n        X[self.columns] = self.scaler.inverse_transform(X[self.columns])\n        return X","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.671698Z","iopub.execute_input":"2025-04-21T14:17:33.672130Z","iopub.status.idle":"2025-04-21T14:17:33.682170Z","shell.execute_reply.started":"2025-04-21T14:17:33.672084Z","shell.execute_reply":"2025-04-21T14:17:33.681263Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class KMeansClustering(BaseEstimator, TransformerMixin):\n    def __init__(self, columns, n_clusters, cluster_column_name='cluster'):\n        self.columns = columns\n        self.n_clusters = n_clusters\n        self.cluster_column_name = cluster_column_name\n        self.kmeans = KMeans(n_clusters=n_clusters, random_state=42)\n\n    def fit(self, X, y=None):\n        # Converte para DataFrame se necessário\n        if isinstance(X, pd.DataFrame):\n            self.feature_names_ = X.columns.tolist()  # Armazena os nomes das colunas\n        else:\n            raise ValueError(\"Esperado um DataFrame como entrada para o transformador.\")\n\n        # Ajusta o KMeans apenas nas colunas especificadas\n        self.kmeans.fit(X[self.columns])\n        return self\n\n    def transform(self, X):\n        # Se X for um array, reconverta para DataFrame usando os nomes das colunas armazenados\n        if not isinstance(X, pd.DataFrame):\n            X = pd.DataFrame(X, columns=self.feature_names_)\n\n        X = X.copy()\n\n        # Gera os rótulos dos clusters e adiciona como uma nova coluna\n        X[self.cluster_column_name] = self.kmeans.predict(X[self.columns])\n\n        # Remove as colunas usadas para clustering\n        #X.drop(columns=self.columns, inplace=True)\n\n        return X","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.683580Z","iopub.execute_input":"2025-04-21T14:17:33.684003Z","iopub.status.idle":"2025-04-21T14:17:33.698196Z","shell.execute_reply.started":"2025-04-21T14:17:33.683937Z","shell.execute_reply":"2025-04-21T14:17:33.697182Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class PCAWithColumns(BaseEstimator, TransformerMixin):\n    def __init__(self, n_components=2, prefix=\"PCA\"):\n        self.n_components = n_components\n        self.prefix = prefix\n        self.pca = PCA(n_components=n_components)\n\n    def fit(self, X, y=None):\n        self.feature_names_ = X.columns.tolist()  # Salva os nomes das colunas originais\n        self.pca.fit(X)\n        return self\n\n    def transform(self, X):\n        # Aplica o PCA\n        components = self.pca.transform(X)\n        \n        # Gera os nomes das colunas do PCA\n        column_names = [f\"{self.prefix}_{i+1}\" for i in range(self.n_components)]\n        \n        # Converte os componentes para DataFrame\n        pca_df = pd.DataFrame(components, columns=column_names, index=X.index)\n        \n        # Combina o DataFrame original com os componentes do PCA\n        return pd.concat([X, pca_df], axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.699441Z","iopub.execute_input":"2025-04-21T14:17:33.699762Z","iopub.status.idle":"2025-04-21T14:17:33.712668Z","shell.execute_reply.started":"2025-04-21T14:17:33.699732Z","shell.execute_reply":"2025-04-21T14:17:33.711742Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pipeline_with_null = Pipeline([\n    ('drop_columns_1', DropColumns(columns=['Fitness_Endurance-Time_Sec','Fitness_Endurance-Time_Mins',\n                                          'Fitness_Endurance-Max_Stage','Physical-Waist_Circumference',\n                                          'FGC-FGC_GSND_Zone','FGC-FGC_GSD_Zone','FGC-FGC_GSD','FGC-FGC_GSND',\n                                          'Fitness_Endurance-Season'])),\n    ('drop_columns_2', DropColumns(columns=['Basic_Demos-Enroll_Season', 'CGAS-Season', \n                                          'Physical-Season', 'FGC-Season', 'BIA-Season', 'PAQ_A-Season',\n                                          'PAQ_C-Season', 'SDS-Season','PreInt_EduHx-Season'])),\n\n    ('paq_total', CombinePAQColumns('PAQ_A-PAQ_A_Total', 'PAQ_C-PAQ_C_Total', 'PAQ_Total')),\n    ('drop_outlier', RangePossibleValues()),\n    ('drop_columns_21', DropColumns(columns=['BIA-BIA_BMR', 'BIA-BIA_DEE', 'BIA-BIA_ECW', 'BIA-BIA_FFM', \n                                            'BIA-BIA_Fat', 'BIA-BIA_ICW', 'BIA-BIA_LDM', 'BIA-BIA_BMC', \n                                            'BIA-BIA_SMM', 'BIA-BIA_TBW','BIA-BIA_BMI'])),\n    ('drop_columns_22', DropColumns(columns=['FGC-FGC_PU','FGC-FGC_CU','FGC-FGC_SRL','FGC-FGC_SRR',\n                                             'FGC-FGC_TL','Physical-Weight','Physical-Height'])),\n])","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.716145Z","iopub.execute_input":"2025-04-21T14:17:33.716515Z","iopub.status.idle":"2025-04-21T14:17:33.728881Z","shell.execute_reply.started":"2025-04-21T14:17:33.716481Z","shell.execute_reply":"2025-04-21T14:17:33.727892Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preprocessing_pipeline = Pipeline([\n    ('drop_columns_1', DropColumns(columns=['Fitness_Endurance-Time_Sec','Fitness_Endurance-Time_Mins',\n                                          'Fitness_Endurance-Max_Stage','Physical-Waist_Circumference',\n                                          'FGC-FGC_GSND_Zone','FGC-FGC_GSD_Zone','FGC-FGC_GSD','FGC-FGC_GSND',\n                                          'Fitness_Endurance-Season'])),\n    ('drop_columns_2', DropColumns(columns=['Basic_Demos-Enroll_Season', 'CGAS-Season', \n                                          'Physical-Season', 'FGC-Season', 'BIA-Season', 'PAQ_A-Season',\n                                          'PAQ_C-Season', 'SDS-Season','PreInt_EduHx-Season'])),\n\n    ('paq_total', CombinePAQColumns('PAQ_A-PAQ_A_Total', 'PAQ_C-PAQ_C_Total', 'PAQ_Total')),\n    ('drop_outlier', RangePossibleValues()),\n    ('drop_columns_21', DropColumns(columns=['BIA-BIA_BMR', 'BIA-BIA_DEE', 'BIA-BIA_ECW', 'BIA-BIA_FFM', \n                                            'BIA-BIA_Fat', 'BIA-BIA_ICW', 'BIA-BIA_LDM', 'BIA-BIA_BMC', \n                                            'BIA-BIA_SMM', 'BIA-BIA_TBW','BIA-BIA_BMI'])),\n    ('drop_columns_22', DropColumns(columns=['FGC-FGC_PU','FGC-FGC_CU','FGC-FGC_SRL','FGC-FGC_SRR',\n                                             'FGC-FGC_TL','Physical-Weight','Physical-Height'])),\n    \n    ('fill_bmi', PipModeloRegression(faltante='Physical-BMI')),\n    ('fill_total_t', PipModeloRegression(faltante='SDS-SDS_Total_T')),\n    ('fill_total_raw', PipModeloRegression(faltante='SDS-SDS_Total_Raw')),\n    ('fill_ffmi', PipModeloRegression(faltante='BIA-BIA_FFMI')),\n    ('fill_lst', PipModeloRegression(faltante='BIA-BIA_LST')),\n    ('fill_fmi', PipModeloRegression(faltante='BIA-BIA_FMI')),\n    ('fill_srl', PipModeloLogisticRegression(faltante='FGC-FGC_SRL_Zone')),\n    ('fill_srr', PipModeloLogisticRegression(faltante='FGC-FGC_SRR_Zone')),\n    ('fill_tl', PipModeloLogisticRegression(faltante='FGC-FGC_TL_Zone')),\n    ('fill_int', PipModeloLogisticRegression(faltante='PreInt_EduHx-computerinternet_hoursday')),\n    ('fill_paq', PipModeloRegression(faltante='PAQ_Total')),\n    ('fill_cgas', PipModeloRegression(faltante='CGAS-CGAS_Score')),\n    \n    # deletadas pq os modelos não tiveram metricas boas\n    ('drop_columns_23', DropColumns(columns=['Physical-Systolic_BP','Physical-Diastolic_BP',\n                                             'Physical-HeartRate','BIA-BIA_Frame_num',\n                                             'BIA-BIA_Activity_Level_num','FGC-FGC_CU_Zone',\n                                             'FGC-FGC_PU_Zone'])),\n    #('drop_columns_24', DropColumns(columns=['Basic_Demos-Age','Basic_Demos-Sex'])),\n    \n    ('undersample', RandomUnderSampler(sampling_strategy={0: 1200}, random_state=42)),\n    ('oversample', SMOTE(sampling_strategy={1: 900, 2: 700, 3: 300}, k_neighbors=5, random_state=42)),\n])","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.730548Z","iopub.execute_input":"2025-04-21T14:17:33.731088Z","iopub.status.idle":"2025-04-21T14:17:33.745274Z","shell.execute_reply.started":"2025-04-21T14:17:33.731038Z","shell.execute_reply":"2025-04-21T14:17:33.744208Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"prediction_pipeline = Pipeline([\n    #('pca', PCAWithColumns(n_components=10, prefix=\"PCA\")),\n    #('drop_columns_5', DropColumns(columns=['Basic_Demos-Age','Basic_Demos-Sex','CGAS-CGAS_Score','PAQ_Total',\n    #                                       'PreInt_EduHx-computerinternet_hoursday','SDS-SDS_Total_T',\n    #                                       'SDS-SDS_Total_Raw','BIA-BIA_LST','BIA-BIA_FMI','BIA-BIA_FFMI',\n    #                                       'FGC-FGC_TL_Zone','FGC-FGC_SRR_Zone','FGC-FGC_SRL_Zone',\n    #                                       'Physical-BMI'])),\n    ('scale', Scaler(columns=['Basic_Demos-Age','Basic_Demos-Sex','CGAS-CGAS_Score','PAQ_Total',\n                                           'PreInt_EduHx-computerinternet_hoursday','SDS-SDS_Total_T',\n                                           'SDS-SDS_Total_Raw','BIA-BIA_LST','BIA-BIA_FMI','BIA-BIA_FFMI',\n                                           'FGC-FGC_TL_Zone','FGC-FGC_SRR_Zone','FGC-FGC_SRL_Zone',\n                                           'Physical-BMI'])),\n    #('classifier', RandomForestClassifier(n_estimators=100, random_state=42))\n    \n    #('kmeans_bia', KMeansClustering(columns=['Basic_Demos-Age','BIA-BIA_FFMI','BIA-BIA_FMI','BIA-BIA_LST'], \n    #                                       n_clusters=4, \n    #                                       cluster_column_name='BIA-kmeans')),\n    #('kmeans_sds', KMeansClustering(columns=['Basic_Demos-Age','SDS-SDS_Total_Raw','SDS-SDS_Total_T'], \n    #                                       n_clusters=4, \n    #                                       cluster_column_name='SDS-kmeans')),\n    #('kmeans_bmi', KMeansClustering(columns=['Physical-BMI','Basic_Demos-Age'], \n    #                                       n_clusters=4, \n    #                                       cluster_column_name='BMI-kmeans')),\n    #('kmeans_cgas', KMeansClustering(columns=['Basic_Demos-Age','CGAS-CGAS_Score'], \n    #                                       n_clusters=4, \n    #                                       cluster_column_name='CGAS-kmeans')),\n    \n    #('classifier', XGBClassifier(random_state=42, use_label_encoder=False, eval_metric='logloss'))  # Modelo de classificação\n])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:33.753412Z","iopub.execute_input":"2025-04-21T14:17:33.753786Z","iopub.status.idle":"2025-04-21T14:17:33.762798Z","shell.execute_reply.started":"2025-04-21T14:17:33.753752Z","shell.execute_reply":"2025-04-21T14:17:33.761819Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/train.csv')\n\nX_train = train_df.dropna(subset=['sii'])\ny_train = X_train['sii']\n\nX_train = X_train.drop(columns=['PCIAT-PCIAT_01','PCIAT-PCIAT_02','PCIAT-PCIAT_03','PCIAT-PCIAT_04','PCIAT-PCIAT_05','PCIAT-PCIAT_06',\n             'PCIAT-PCIAT_07','PCIAT-PCIAT_08','PCIAT-PCIAT_09','PCIAT-PCIAT_10','PCIAT-PCIAT_11','PCIAT-PCIAT_12',\n             'PCIAT-PCIAT_13','PCIAT-PCIAT_14','PCIAT-PCIAT_15','PCIAT-PCIAT_16','PCIAT-PCIAT_17','PCIAT-PCIAT_18',\n             'PCIAT-PCIAT_19','PCIAT-PCIAT_20','PCIAT-PCIAT_20','PCIAT-PCIAT_Total','PCIAT-Season','sii','id'])\n\n# Aplicando o pipeline no conjunto de treino\ndf_with_null = pipeline_with_null.fit_transform(X_train)\n\n# Roda 1 etapa do Pipeline (preprocessing)\nX_train_resampled, y_train_resampled = preprocessing_pipeline.fit_resample(X_train, y_train) \ndf_preprocessing = pd.DataFrame(X_train_resampled)\ndf_preprocessing['sii'] = y_train_resampled\n\nX_train_final = prediction_pipeline.fit_transform(X_train_resampled, y_train_resampled)\ndf_train = pd.DataFrame(X_train_final)\ndf_train['sii'] = y_train_resampled","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:33.764307Z","iopub.execute_input":"2025-04-21T14:17:33.764746Z","iopub.status.idle":"2025-04-21T14:17:34.160147Z","shell.execute_reply.started":"2025-04-21T14:17:33.764696Z","shell.execute_reply":"2025-04-21T14:17:34.158582Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"missing_percentage = (df_train.isnull().sum() / len(df_train)) * 100\nprint(missing_percentage.sort_values(ascending=False))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:34.161301Z","iopub.execute_input":"2025-04-21T14:17:34.163196Z","iopub.status.idle":"2025-04-21T14:17:34.183708Z","shell.execute_reply.started":"2025-04-21T14:17:34.163148Z","shell.execute_reply":"2025-04-21T14:17:34.182055Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"x = 'Basic_Demos-Age'\ny = 'Physical-BMI'\nz = 'Physical-BMI'\n\nanalysis_pipeline(df_with_null, df_preprocessing, x, y, z)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:34.185301Z","iopub.execute_input":"2025-04-21T14:17:34.187248Z","iopub.status.idle":"2025-04-21T14:17:35.055009Z","shell.execute_reply.started":"2025-04-21T14:17:34.187199Z","shell.execute_reply":"2025-04-21T14:17:35.054018Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"analysis_categorical_variables(df_preprocessing, 'sii')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:35.056429Z","iopub.execute_input":"2025-04-21T14:17:35.056743Z","iopub.status.idle":"2025-04-21T14:17:37.361146Z","shell.execute_reply.started":"2025-04-21T14:17:35.056711Z","shell.execute_reply":"2025-04-21T14:17:37.360059Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Agrupamento de Participantes (Clustering) para Criar Perfis","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import davies_bouldin_score\nfrom sklearn.metrics import calinski_harabasz_score","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:37.362889Z","iopub.execute_input":"2025-04-21T14:17:37.363203Z","iopub.status.idle":"2025-04-21T14:17:37.369039Z","shell.execute_reply.started":"2025-04-21T14:17:37.363173Z","shell.execute_reply":"2025-04-21T14:17:37.368034Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def clustering(df, columns, new_column, n_cluster=3):\n    kmeans_bia = KMeans(n_clusters=n_cluster, random_state=0)\n    df[new_column] = kmeans_bia.fit_predict(df[columns])\n    \n    print(\"N Cluster:\", n_cluster)\n    \n    silhouette = silhouette_score(df[columns], df[new_column])\n    print(\"Silhouette Score :\", silhouette)\n\n    correlation = df[new_column].corr(df['sii'])\n    print('Correlação com sii: ', correlation)\n\n    plt.figure(figsize=(8, 6))\n    plt.scatter(df_total[columns[0]], df_total[columns[1]], c=df_total[new_column], cmap='viridis', alpha=0.7)\n    plt.colorbar(label=new_column)\n    plt.xlabel(columns[0])\n    plt.ylabel(columns[1])\n    plt.title('Clustering')\n    plt.grid()\n    plt.show()\n\n    # Coeficiente de Davies-Bouldin\n    #davies_bouldin = davies_bouldin_score(df[columns], df[new_column])\n    #print(\"Davies-Bouldin Index:\", davies_bouldin)\n\n    # Calinski-Harabasz Index\n    #calinski_harabasz = calinski_harabasz_score(df[columns], df[new_column])\n    #print(\"Calinski-Harabasz Index:\", calinski_harabasz)\n\n    # Inércia\n    #inertia = kmeans_bia.inertia_\n    #print(\"Inertia (SSE):\", inertia)","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:37.370349Z","iopub.execute_input":"2025-04-21T14:17:37.370680Z","iopub.status.idle":"2025-04-21T14:17:37.380931Z","shell.execute_reply.started":"2025-04-21T14:17:37.370648Z","shell.execute_reply":"2025-04-21T14:17:37.379901Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"columns = ['BIA-BIA_FFMI','BIA-BIA_FMI','BIA-BIA_LST']\ndf_total = df_preprocessing.copy()\n\n#clustering(df_total, columns, 'N_2', n_cluster=2)\n#clustering(df_total, columns, 'N_3', n_cluster=3)\nclustering(df_total, columns, 'N_4', n_cluster=4)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:37.382381Z","iopub.execute_input":"2025-04-21T14:17:37.382719Z","iopub.status.idle":"2025-04-21T14:17:38.379948Z","shell.execute_reply.started":"2025-04-21T14:17:37.382687Z","shell.execute_reply":"2025-04-21T14:17:38.378986Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# PCA (Principal Component Analysis)\nO PCA é útil para reduzir a quantidade de variáveis, especialmente se há colinearidade ou muitas dimensões irrelevantes. Ele transforma os dados originais em componentes principais, preservando a maior variância possível.","metadata":{}},{"cell_type":"code","source":"def applyPCA(df, target=None, n_components=None, scale_data=True):\n    df_temp = df.copy()\n    df_temp.dropna(inplace=True)\n    \n    # Separar a variável-alvo, se houver\n    if target is not None:\n        X = df_temp.drop(columns=target)\n        y = df_temp[target]\n    else:\n        X = df_temp\n        y = None\n    \n    # Padronizar os dados (opcional)\n    if scale_data:\n        scaler = StandardScaler()\n        X_scaled = scaler.fit_transform(X)\n    else:\n        X_scaled = X.values\n    \n    # Aplicar PCA\n    pca = PCA(n_components=n_components)\n    X_pca = pca.fit_transform(X_scaled)\n    \n    # Criar DataFrame com os componentes principais\n    component_names = [f'PC{i+1}' for i in range(X_pca.shape[1])]\n    pca_result = pd.DataFrame(X_pca, columns=component_names)\n    \n    # Adicionar a variável-alvo de volta, se houver\n    if y is not None:\n        pca_result[target] = y.values\n    \n    # Variância explicada por componente\n    explained_variance = pd.Series(pca.explained_variance_ratio_, index=component_names)\n    \n    # Resultados\n    print(\"\\nProporção de variância explicada por componente:\")\n    print(explained_variance)\n    print(\"\\nVariância explicada acumulada:\")\n    print(explained_variance.cumsum())\n    \n    return pca_result, explained_variance","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:38.381605Z","iopub.execute_input":"2025-04-21T14:17:38.382448Z","iopub.status.idle":"2025-04-21T14:17:38.392444Z","shell.execute_reply.started":"2025-04-21T14:17:38.382398Z","shell.execute_reply":"2025-04-21T14:17:38.391418Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Aplicar PCA no DataFrame (sem variável-alvo)\npca_result, explained_variance = applyPCA(df_preprocessing, target='sii', n_components=0.95)","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:38.393756Z","iopub.execute_input":"2025-04-21T14:17:38.394136Z","iopub.status.idle":"2025-04-21T14:17:38.421632Z","shell.execute_reply.started":"2025-04-21T14:17:38.394092Z","shell.execute_reply":"2025-04-21T14:17:38.420778Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plotando os dois primeiros componentes principais\nplt.figure(figsize=(8, 6))\nplt.scatter(pca_result['PC1'], pca_result['PC2'], c=pca_result['sii'], cmap='viridis', alpha=0.7)\nplt.colorbar(label='Classes (sii)')\nplt.xlabel('PC1')\nplt.ylabel('PC2')\nplt.title('Dados no Espaço dos Componentes Principais (PC1 vs PC2)')\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:38.422685Z","iopub.execute_input":"2025-04-21T14:17:38.423041Z","iopub.status.idle":"2025-04-21T14:17:38.856327Z","shell.execute_reply.started":"2025-04-21T14:17:38.423001Z","shell.execute_reply":"2025-04-21T14:17:38.855075Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Treinamento do Modelo","metadata":{}},{"cell_type":"code","source":"def add_df(df, colunas, modelo, accuracy=None, precision=None, recall=None, \n           f1=None, mse=None, mae=None, r2=None):\n    df.loc[len(df)] = {\n        'colunas': colunas,\n        'modelo': modelo,\n        'accuracy': accuracy,\n        'precision': precision,\n        'recall': recall,\n        'f1': f1,\n        'mse': mse,\n        'mae': mae,\n        'r2': r2\n    }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.857933Z","iopub.execute_input":"2025-04-21T14:17:38.858387Z","iopub.status.idle":"2025-04-21T14:17:38.865919Z","shell.execute_reply.started":"2025-04-21T14:17:38.858339Z","shell.execute_reply":"2025-04-21T14:17:38.864724Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def preprocess_dataframe(df, target, colunas=None, test_size=0.2, random_state=42):\n    # Remover linhas onde o target é nulo\n    df = df[df[target].notnull()]\n    \n    # Remover colunas especificadas, se fornecido\n    if colunas:\n        df = df.drop(columns=colunas, errors='ignore')\n    \n    # Remover colunas que possuem qualquer valor nulo\n    df = df.dropna(axis=1)\n    \n    # Separar variáveis explicativas e variável-alvo\n    X = df.drop(columns=[target])\n    y = df[target]\n    \n    # Dividir os dados em treino e teste\n    X_train, X_test, y_train, y_test = train_test_split(X, y, \n                                                        test_size=test_size, # 20% para o conjunto de teste\n                                                        random_state=random_state, # Para garantir reprodutibilidade\n                                                        stratify=y  # Garantir que a distribuição das classes seja mantida\n                                                       )\n    return X_train, X_test, y_train, y_test","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.867387Z","iopub.execute_input":"2025-04-21T14:17:38.867812Z","iopub.status.idle":"2025-04-21T14:17:38.882725Z","shell.execute_reply.started":"2025-04-21T14:17:38.867756Z","shell.execute_reply":"2025-04-21T14:17:38.881721Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Modelo de regressão","metadata":{}},{"cell_type":"code","source":"def modeloExtraTreesRegressor(X_train, X_test, y_train, y_test, test_size=0.2, \n                              random_state=42, n_estimators=100, **model_params):\n    \n    # Configurar e treinar o modelo Extra Trees Regressor\n    et_model = ExtraTreesRegressor(n_estimators=n_estimators, random_state=random_state, **model_params)\n    et_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = et_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    mse = mean_squared_error(y_test, y_pred)\n    mae = mean_absolute_error(y_test, y_pred)\n    r2 = r2_score(y_test, y_pred)\n\n    return mse, mae, r2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.884165Z","iopub.execute_input":"2025-04-21T14:17:38.884483Z","iopub.status.idle":"2025-04-21T14:17:38.894599Z","shell.execute_reply.started":"2025-04-21T14:17:38.884451Z","shell.execute_reply":"2025-04-21T14:17:38.893148Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloSVR(X_train, X_test, y_train, y_test, test_size=0.2,\n              random_state=42, kernel='rbf', scale_data=True, **model_params):\n    \n    # Padronizar os dados (recomendado para SVR)\n    if scale_data:\n        scaler = StandardScaler()\n        X_train = scaler.fit_transform(X_train)\n        X_test = scaler.transform(X_test)\n    \n    # Configurar e treinar o modelo SVR\n    svr_model = SVR(kernel=kernel, **model_params)\n    svr_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = svr_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    mse = mean_squared_error(y_test, y_pred)\n    mae = mean_absolute_error(y_test, y_pred)\n    r2 = r2_score(y_test, y_pred)\n\n    return mse, mae, r2 ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.898092Z","iopub.execute_input":"2025-04-21T14:17:38.898532Z","iopub.status.idle":"2025-04-21T14:17:38.910035Z","shell.execute_reply.started":"2025-04-21T14:17:38.898496Z","shell.execute_reply":"2025-04-21T14:17:38.909038Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloMLPRegressor(X_train, X_test, y_train, y_test, test_size=0.2, random_state=42, \n                       hidden_layer_sizes=(100,), max_iter=1000, scale_data=True, **model_params):\n    \n    # Padronizar os dados (opcional, recomendado para redes neurais)\n    if scale_data:\n        scaler = StandardScaler()\n        X_train = scaler.fit_transform(X_train)\n        X_test = scaler.transform(X_test)\n    \n    # Configurar e treinar o modelo MLP Regressor\n    mlp_model = MLPRegressor(hidden_layer_sizes=hidden_layer_sizes, max_iter=max_iter, random_state=random_state, **model_params)\n    mlp_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = mlp_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    mse = mean_squared_error(y_test, y_pred)\n    mae = mean_absolute_error(y_test, y_pred)\n    r2 = r2_score(y_test, y_pred)\n\n    return mse, mae, r2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.911438Z","iopub.execute_input":"2025-04-21T14:17:38.911769Z","iopub.status.idle":"2025-04-21T14:17:38.923909Z","shell.execute_reply.started":"2025-04-21T14:17:38.911735Z","shell.execute_reply":"2025-04-21T14:17:38.922668Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloRegressaoLinear(X_train, X_test, y_train, y_test, test_size=0.2, random_state=42):\n    \n    # Configurar e treinar o modelo de Regressão Linear\n    lr_model = LinearRegression()\n    lr_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = lr_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    mse = mean_squared_error(y_test, y_pred)\n    mae = mean_absolute_error(y_test, y_pred)\n    r2 = r2_score(y_test, y_pred)\n    \n    return mse, mae, r2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.927431Z","iopub.execute_input":"2025-04-21T14:17:38.928379Z","iopub.status.idle":"2025-04-21T14:17:38.940785Z","shell.execute_reply.started":"2025-04-21T14:17:38.928314Z","shell.execute_reply":"2025-04-21T14:17:38.939753Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloGradientBoostingRegressor(X_train, X_test, y_train, y_test, model_type=\"xgboost\", \n                                    test_size=0.2, random_state=42, **model_params):\n    \n    # Escolher o modelo de acordo com o tipo\n    if model_type.lower() == \"xgboost\":\n        model = XGBRegressor(random_state=random_state, **model_params)\n    elif model_type.lower() == \"lightgbm\":\n        model = LGBMRegressor(random_state=random_state, **model_params)\n    elif model_type.lower() == \"catboost\":\n        model = CatBoostRegressor(random_state=random_state, verbose=0, **model_params)\n    else:\n        raise ValueError(\"model_type must be 'xgboost', 'lightgbm', or 'catboost'\")\n    \n    # Treinar o modelo\n    model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    mse = mean_squared_error(y_test, y_pred)\n    mae = mean_absolute_error(y_test, y_pred)\n    r2 = r2_score(y_test, y_pred)\n    \n    return mse, mae, r2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.942076Z","iopub.execute_input":"2025-04-21T14:17:38.942405Z","iopub.status.idle":"2025-04-21T14:17:38.954086Z","shell.execute_reply.started":"2025-04-21T14:17:38.942372Z","shell.execute_reply":"2025-04-21T14:17:38.952874Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloRandomForestRegressor(X_train, X_test, y_train, y_test, test_size=0.2, random_state=42, n_estimators=100):\n    \n    # Configurar e treinar o modelo Random Forest\n    rf_model = RandomForestRegressor(n_estimators=n_estimators, random_state=random_state)\n    rf_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = rf_model.predict(X_test)\n    \n    # Calcular métricas de avaliação\n    mse = mean_squared_error(y_test, y_pred)\n    mae = mean_absolute_error(y_test, y_pred)\n    r2 = r2_score(y_test, y_pred)\n\n    return mse, mae, r2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.955470Z","iopub.execute_input":"2025-04-21T14:17:38.955790Z","iopub.status.idle":"2025-04-21T14:17:38.968369Z","shell.execute_reply.started":"2025-04-21T14:17:38.955749Z","shell.execute_reply":"2025-04-21T14:17:38.967414Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modelo_regressor(df, df_model_result, target):\n    X_train, X_test, y_train, y_test = preprocess_dataframe(df, target)\n    columns_name = df.columns.tolist()\n    \n    mse, mae, r2 = modeloExtraTreesRegressor(X_train, X_test, y_train, y_test)\n    add_df(df_model_result, columns_name, 'modeloExtraTreesRegressor', mse=mse, mae=mae, r2=r2)\n    \n    mse, mae, r2 = modeloSVR(X_train, X_test, y_train, y_test)\n    add_df(df_model_result, columns_name, 'modeloSVR', mse=mse, mae=mae, r2=r2)\n    \n    mse, mae, r2 = modeloMLPRegressor(X_train, X_test, y_train, y_test)\n    add_df(df_model_result, columns_name, 'modeloMLPRegressor', mse=mse, mae=mae, r2=r2)\n    \n    mse, mae, r2 = modeloRegressaoLinear(X_train, X_test, y_train, y_test)\n    add_df(df_model_result, columns_name, 'modeloRegressaoLinear', mse=mse, mae=mae, r2=r2)\n    \n    mse, mae, r2 = modeloGradientBoostingRegressor(X_train, X_test, y_train, y_test)\n    add_df(df_model_result, columns_name, 'modeloGradientBoostingRegressor', mse=mse, mae=mae, r2=r2)\n    \n    mse, mae, r2 = modeloRandomForestRegressor(X_train, X_test, y_train, y_test)\n    add_df(df_model_result, columns_name, 'modeloRandomForestRegressor', mse=mse, mae=mae, r2=r2)\n\n    return df_model_result","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.969636Z","iopub.execute_input":"2025-04-21T14:17:38.970019Z","iopub.status.idle":"2025-04-21T14:17:38.981005Z","shell.execute_reply.started":"2025-04-21T14:17:38.969953Z","shell.execute_reply":"2025-04-21T14:17:38.979935Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#df_model_result = pd.DataFrame(columns=['colunas', 'modelo', 'accuracy', 'precision', 'recall', 'f1','mse', 'mae', 'r2'])\n#df_model_result = modelo_regressor(X_train_processed, df_model_result, 'FGC-FGC_PU')\n#df_model_result","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.982399Z","iopub.execute_input":"2025-04-21T14:17:38.982943Z","iopub.status.idle":"2025-04-21T14:17:38.994757Z","shell.execute_reply.started":"2025-04-21T14:17:38.982897Z","shell.execute_reply":"2025-04-21T14:17:38.993761Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Modelo de classificação","metadata":{}},{"cell_type":"code","source":"def print_details_modelo(y_test, y_pred, y_proba, rf_model, nome_modelo):\n    # Feature Importance\n    \n    # Matriz de Confusão\n    cm = confusion_matrix(y_test, y_pred)\n    disp = ConfusionMatrixDisplay(confusion_matrix=cm)\n    disp.plot(cmap='Blues')\n    plt.title(\"Matriz de Confusão\")\n    plt.savefig(f'/kaggle/working/confusion_matrix_RandomForestClassifier.png')\n    plt.show()\n\n    # Importância das Features\n    importances = rf_model.feature_importances_\n    features = X_train.columns\n    df_feat = pd.DataFrame({'feature': features, 'importance': importances})\n    df_feat = df_feat.sort_values(by='importance', ascending=False)\n\n    sns.barplot(x='importance', y='feature', data=df_feat.head(10))\n    plt.title(\"Top 10 Features - Random Forest\")\n    plt.tight_layout()\n    plt.savefig(f'/kaggle/working/feature_importance_{nome_modelo}.png')\n    plt.show()\n\n    # Curva ROC para cada classe\n    classes = np.unique(y_test)\n    y_test_bin = label_binarize(y_test, classes=classes)\n    for i in range(len(classes)):\n        RocCurveDisplay.from_predictions(y_test_bin[:, i], y_proba[:, i], name=f'Classe {classes[i]}')\n    plt.title(\"Curva ROC por Classe\")\n    plt.savefig(f'/kaggle/working/roc_curve_{nome_modelo}.png')\n    plt.show()\n\n    # Curva Precision-Recall para cada classe\n    for i in range(len(classes)):\n        PrecisionRecallDisplay.from_predictions(y_test_bin[:, i], y_proba[:, i], name=f'Classe {classes[i]}')\n        plt.savefig(f'/kaggle/working/precision_recall_{nome_modelo}_{classes[i]}.png')\n    plt.title(\"Curva Precision-Recall por Classe\")\n    plt.show()\n\n    # Confiança das previsões\n    max_proba = np.max(y_proba, axis=1)\n    plt.hist(max_proba, bins=20, color='skyblue', edgecolor='black')\n    plt.title(\"Confiança das Previsões\")\n    plt.xlabel(\"Maior probabilidade predita\")\n    plt.ylabel(\"Número de amostras\")\n    plt.savefig(f'/kaggle/working/prediction_confidence_{nome_modelo}.png')\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:38.996183Z","iopub.execute_input":"2025-04-21T14:17:38.996571Z","iopub.status.idle":"2025-04-21T14:17:39.009509Z","shell.execute_reply.started":"2025-04-21T14:17:38.996538Z","shell.execute_reply":"2025-04-21T14:17:39.008458Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloRandomForestClassifier(X_train, X_test, y_train, y_test, random_state=42, n_estimators=100, \n                                 print_details=False, average='macro'):\n    # Configurar e treinar o modelo Random Forest Classifier\n    rf_model = RandomForestClassifier(n_estimators=n_estimators, random_state=random_state)\n    rf_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = rf_model.predict(X_test)\n    y_proba = rf_model.predict_proba(X_test)\n    \n    # Calcular métricas de desempenho\n    accuracy = accuracy_score(y_test, y_pred)\n    precision = precision_score(y_test, y_pred, average=average, zero_division=1)\n    recall = recall_score(y_test, y_pred, average=average)\n    f1 = f1_score(y_test, y_pred, average=average, zero_division=1)\n    \n    if print_details:\n        print_details_modelo(y_test, y_pred, y_proba, rf_model, 'RandomForestClassifier')\n    \n    return accuracy, precision, recall, f1","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:39.011183Z","iopub.execute_input":"2025-04-21T14:17:39.011904Z","iopub.status.idle":"2025-04-21T14:17:39.024613Z","shell.execute_reply.started":"2025-04-21T14:17:39.011855Z","shell.execute_reply":"2025-04-21T14:17:39.023636Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloKNN(X_train, X_test, y_train, y_test, n_neighbors=5, scale_data=True, print_details=False, average='macro'):    \n    # Padronizar os dados (opcional, recomendado para KNN)\n    if scale_data:\n        scaler = StandardScaler()\n        X_train = scaler.fit_transform(X_train)\n        X_test = scaler.transform(X_test)\n    \n    # Configurar e treinar o modelo KNN\n    knn_model = KNeighborsClassifier(n_neighbors=n_neighbors)\n    knn_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = knn_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    accuracy = accuracy_score(y_test, y_pred)\n    precision = precision_score(y_test, y_pred, average=average, zero_division=1)\n    recall = recall_score(y_test, y_pred, average=average)\n    f1 = f1_score(y_test, y_pred, average=average, zero_division=1)\n    \n    report = classification_report(y_test, y_pred, output_dict=True)\n    confusion = confusion_matrix(y_test, y_pred)\n    \n    if print_details:\n        print('\\n\\033[1m K-Nearest Neighbors (KNN) \\033[0m')\n        print(f\"Accuracy: {accuracy:.4f}\")\n        print(\"\\nClassification Report:\")\n        print(classification_report(y_test, y_pred))\n        print(\"\\nConfusion Matrix:\")\n        print(confusion)\n    \n    return accuracy, precision, recall, f1","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:39.028113Z","iopub.execute_input":"2025-04-21T14:17:39.028644Z","iopub.status.idle":"2025-04-21T14:17:39.040344Z","shell.execute_reply.started":"2025-04-21T14:17:39.028604Z","shell.execute_reply":"2025-04-21T14:17:39.039344Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloGradientBoostingClassifier(X_train, X_test, y_train, y_test, random_state=42, average='macro',\n                                     model_type=\"xgboost\", print_details=False, **model_params):    \n    # Escolher o modelo de acordo com o tipo\n    if model_type.lower() == \"xgboost\":\n        model = XGBClassifier(random_state=random_state, use_label_encoder=False, eval_metric='logloss', **model_params)\n    elif model_type.lower() == \"lightgbm\":\n        model = LGBMClassifier(random_state=random_state, **model_params)\n    elif model_type.lower() == \"catboost\":\n        model = CatBoostClassifier(random_state=random_state, verbose=0, **model_params)\n    else:\n        raise ValueError(\"model_type must be 'xgboost', 'lightgbm', or 'catboost'\")\n    \n    # Treinar o modelo\n    model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    accuracy = accuracy_score(y_test, y_pred)\n    precision = precision_score(y_test, y_pred, average=average, zero_division=1)\n    recall = recall_score(y_test, y_pred, average=average)\n    f1 = f1_score(y_test, y_pred, average=average, zero_division=1)\n    \n    report = classification_report(y_test, y_pred, output_dict=True)\n    confusion = confusion_matrix(y_test, y_pred)\n    \n    if print_details:\n        print('\\n\\033[1m Gradient Boosting Classifiers (XGBoost, LightGBM, CatBoost) \\033[0m')\n        print(f\"Accuracy: {accuracy:.4f}\")\n        print(\"\\nClassification Report:\")\n        print(classification_report(y_test, y_pred))\n        print(\"\\nConfusion Matrix:\")\n        print(confusion)\n\n    return accuracy, precision, recall, f1","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:39.042082Z","iopub.execute_input":"2025-04-21T14:17:39.042511Z","iopub.status.idle":"2025-04-21T14:17:39.056579Z","shell.execute_reply.started":"2025-04-21T14:17:39.042465Z","shell.execute_reply":"2025-04-21T14:17:39.055378Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloLogisticRegression(X_train, X_test, y_train, y_test, random_state=42, average='macro' ,\n                             scale_data=True, print_details=False, **model_params):\n    # Padronizar os dados (opcional, recomendado para regressão logística)\n    if scale_data:\n        scaler = StandardScaler()\n        X_train = scaler.fit_transform(X_train)\n        X_test = scaler.transform(X_test)\n    \n    # Configurar e treinar o modelo LogisticRegression\n    lr_model = LogisticRegression(random_state=random_state, **model_params)\n    lr_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = lr_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    accuracy = accuracy_score(y_test, y_pred)\n    precision = precision_score(y_test, y_pred, average=average, zero_division=1)\n    recall = recall_score(y_test, y_pred, average=average)\n    f1 = f1_score(y_test, y_pred, average=average, zero_division=1)\n    \n    report = classification_report(y_test, y_pred, output_dict=True)\n    confusion = confusion_matrix(y_test, y_pred)\n    \n    if print_details: \n        print('\\n\\033[1m Regressão Logística Multinomial (Logistic Regression) \\033[0m')\n        print(f\"Accuracy: {accuracy:.4f}\")\n        print(\"\\nClassification Report:\")\n        print(classification_report(y_test, y_pred))\n        print(\"\\nConfusion Matrix:\")\n        print(confusion)\n\n        # Obter os coeficientes das variáveis\n        coefficients = pd.DataFrame({\n            'Feature': X_train.columns,\n            'Coefficient': lr_model.coef_[0] if lr_model.coef_.ndim > 1 else lr_model.coef_\n        }).sort_values(by='Coefficient', ascending=False)\n        \n        print(\"\\nCoeficientes das Variáveis:\")\n        print(coefficients)\n        \n    return accuracy, precision, recall, f1","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:39.058162Z","iopub.execute_input":"2025-04-21T14:17:39.058539Z","iopub.status.idle":"2025-04-21T14:17:39.075120Z","shell.execute_reply.started":"2025-04-21T14:17:39.058490Z","shell.execute_reply":"2025-04-21T14:17:39.073702Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloNaiveBayes(X_train, X_test, y_train, y_test, scale_data=True, print_details=False, average='macro'):\n    # Padronizar os dados (opcional, pode ajudar para GaussianNB com dados contínuos)\n    if scale_data:\n        scaler = StandardScaler()\n        X_train = scaler.fit_transform(X_train)\n        X_test = scaler.transform(X_test)\n    \n    # Configurar e treinar o modelo Naive Bayes (GaussianNB para dados contínuos)\n    nb_model = GaussianNB()\n    nb_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = nb_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    accuracy = accuracy_score(y_test, y_pred)\n    precision = precision_score(y_test, y_pred, average=average, zero_division=1)\n    recall = recall_score(y_test, y_pred, average=average)\n    f1 = f1_score(y_test, y_pred, average=average, zero_division=1)\n    \n    report = classification_report(y_test, y_pred, output_dict=True)\n    confusion = confusion_matrix(y_test, y_pred)\n    \n    if print_details:\n        print('\\n\\033[1m Naive Bayes \\033[0m')\n        print(f\"Accuracy: {accuracy:.4f}\")\n        print(\"\\nClassification Report:\")\n        print(classification_report(y_test, y_pred))\n        print(\"\\nConfusion Matrix:\")\n        print(confusion)\n        \n    return accuracy, precision, recall, f1","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:39.076839Z","iopub.execute_input":"2025-04-21T14:17:39.077237Z","iopub.status.idle":"2025-04-21T14:17:39.090652Z","shell.execute_reply.started":"2025-04-21T14:17:39.077195Z","shell.execute_reply":"2025-04-21T14:17:39.089381Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modeloMLPClassifier(X_train, X_test, y_train, y_test, random_state=42, print_details=False,\n                        hidden_layer_sizes=(100,), max_iter=200, scale_data=True, average='macro'):    \n    # Padronizar os dados (opcional, recomendado para MLP)\n    if scale_data:\n        scaler = StandardScaler()\n        X_train = scaler.fit_transform(X_train)\n        X_test = scaler.transform(X_test)\n    \n    # Configurar e treinar o modelo MLP\n    mlp_model = MLPClassifier(hidden_layer_sizes=hidden_layer_sizes, max_iter=max_iter, random_state=random_state)\n    mlp_model.fit(X_train, y_train)\n    \n    # Fazer previsões no conjunto de teste\n    y_pred = mlp_model.predict(X_test)\n    \n    # Calcular métricas de desempenho\n    accuracy = accuracy_score(y_test, y_pred)\n    precision = precision_score(y_test, y_pred, average=average, zero_division=1)\n    recall = recall_score(y_test, y_pred, average=average)\n    f1 = f1_score(y_test, y_pred, average=average, zero_division=1)\n    \n    report = classification_report(y_test, y_pred, output_dict=True)\n    confusion = confusion_matrix(y_test, y_pred)\n    \n    if print_details: \n        print('\\n\\033[1m MLP Classifier (Rede Neural Artificial) \\033[0m')\n        print(f\"Accuracy: {accuracy:.4f}\")\n        print(\"\\nClassification Report:\")\n        print(classification_report(y_test, y_pred))\n        print(\"\\nConfusion Matrix:\")\n        print(confusion)\n    \n    return accuracy, precision, recall, f1","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:39.092140Z","iopub.execute_input":"2025-04-21T14:17:39.092546Z","iopub.status.idle":"2025-04-21T14:17:39.106735Z","shell.execute_reply.started":"2025-04-21T14:17:39.092511Z","shell.execute_reply":"2025-04-21T14:17:39.105727Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def modelo_classificacao(df, df_model_result, target, average='macro', scale_data=True):\n    X_train, X_test, y_train, y_test = preprocess_dataframe(df, target)\n    columns_name = df.columns.tolist()\n    \n    accuracy, precision, recall, f1 = modeloRandomForestClassifier(X_train, X_test, y_train, y_test, average=average)\n    add_df(df_model_result, columns_name, 'modeloRandomForestClassifier', accuracy=accuracy, precision=precision, recall=recall, f1=f1)\n    \n    accuracy, precision, recall, f1 = modeloKNN(X_train, X_test, y_train, y_test, average=average, scale_data=scale_data)\n    add_df(df_model_result, columns_name, 'modeloKNN', accuracy=accuracy, precision=precision, recall=recall, f1=f1)\n    \n    ##accuracy = modeloGradientBoostingClassifier(X_train, X_test, y_train, y_test)\n    ##add_df(df_model_result, columns_name, 'modeloGradientBoostingClassifier', accuracy)\n    \n    accuracy, precision, recall, f1 = modeloNaiveBayes(X_train, X_test, y_train, y_test, average=average, scale_data=scale_data)\n    add_df(df_model_result, columns_name, 'modeloNaiveBayes', accuracy=accuracy, precision=precision, recall=recall, f1=f1)\n    \n    accuracy, precision, recall, f1 = modeloMLPClassifier(X_train, X_test, y_train, y_test, max_iter=500, \n                                                          average=average, scale_data=scale_data)\n    add_df(df_model_result, columns_name, 'modeloMLPClassifier', accuracy=accuracy, precision=precision, recall=recall, f1=f1)\n    \n    accuracy, precision, recall, f1 = modeloLogisticRegression(X_train, X_test, y_train, y_test, \n                                                               scale_data=scale_data, max_iter=300, average=average)\n    add_df(df_model_result, columns_name, 'modeloLogisticRegression', accuracy=accuracy, precision=precision, recall=recall, f1=f1)\n\n    return df_model_result","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:39.110391Z","iopub.execute_input":"2025-04-21T14:17:39.110723Z","iopub.status.idle":"2025-04-21T14:17:39.122625Z","shell.execute_reply.started":"2025-04-21T14:17:39.110689Z","shell.execute_reply":"2025-04-21T14:17:39.121706Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"t = 'Physical-BMI'\nmax_value = X_train_resampled[t].max()\nmin_value = X_train_resampled[t].min()\n\nprint(f\"Valor máximo da coluna 'A': {max_value}\")\nprint(f\"Valor mínimo da coluna 'A': {min_value}\")","metadata":{"execution":{"iopub.status.busy":"2025-04-21T14:17:39.123893Z","iopub.execute_input":"2025-04-21T14:17:39.124216Z","iopub.status.idle":"2025-04-21T14:17:39.139947Z","shell.execute_reply.started":"2025-04-21T14:17:39.124186Z","shell.execute_reply":"2025-04-21T14:17:39.138575Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Variáveis contínuas\n#df_result_regressor = pd.DataFrame(columns=['colunas', 'modelo', 'accuracy', 'precision', 'recall', 'f1','mse', 'mae', 'r2'])\n#df_result_regressor = modelo_regressor(X_train_resampled, df_result_regressor, t)\n\n#df_sorted = df_result_regressor.sort_values(by='r2', ascending=False)\n#df_sorted = df_sorted.reset_index(drop=True)\n#df_sorted.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:39.141376Z","iopub.execute_input":"2025-04-21T14:17:39.141778Z","iopub.status.idle":"2025-04-21T14:17:39.153342Z","shell.execute_reply.started":"2025-04-21T14:17:39.141719Z","shell.execute_reply":"2025-04-21T14:17:39.152274Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Variáveis categóricas \n#df_result_classificacao = pd.DataFrame(columns=['colunas', 'modelo', 'accuracy', 'precision', 'recall', 'f1','mse', 'mae', 'r2'])\n#df_result_classificacao = modelo_classificacao(df_train, df_result_classificacao, 'FGC-FGC_SRL_Zone', average='binary')\n\n#df_sorted = df_result_classificacao.sort_values(by='precision', ascending=False)\n#df_sorted = df_sorted.reset_index(drop=True)\n#df_sorted.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:39.154742Z","iopub.execute_input":"2025-04-21T14:17:39.155114Z","iopub.status.idle":"2025-04-21T14:17:39.165985Z","shell.execute_reply.started":"2025-04-21T14:17:39.155082Z","shell.execute_reply":"2025-04-21T14:17:39.164836Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Variáveis TARGET \ndf_result_classificacao = pd.DataFrame(columns=['colunas', 'modelo', 'accuracy', 'precision', 'recall', 'f1','mse', 'mae', 'r2'])\ndf_result_classificacao = modelo_classificacao(df_train, df_result_classificacao, 'sii') \n\ndf_sorted = df_result_classificacao.sort_values(by='precision', ascending=False)\ndf_sorted = df_sorted.reset_index(drop=True)\ndf_sorted.head(5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:39.167377Z","iopub.execute_input":"2025-04-21T14:17:39.167734Z","iopub.status.idle":"2025-04-21T14:17:51.050156Z","shell.execute_reply.started":"2025-04-21T14:17:39.167699Z","shell.execute_reply":"2025-04-21T14:17:51.049061Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = preprocess_dataframe(df_train, 'sii')\nmodeloRandomForestClassifier(X_train, X_test, y_train, y_test, print_details=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:51.051377Z","iopub.execute_input":"2025-04-21T14:17:51.051694Z","iopub.status.idle":"2025-04-21T14:17:55.236763Z","shell.execute_reply.started":"2025-04-21T14:17:51.051663Z","shell.execute_reply":"2025-04-21T14:17:55.235803Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Criar o arquivo de submissão","metadata":{}},{"cell_type":"code","source":"prediction_pipeline = Pipeline([\n    ('scale', Scaler(columns=['Basic_Demos-Age','Basic_Demos-Sex','CGAS-CGAS_Score','PAQ_Total',\n                                           'PreInt_EduHx-computerinternet_hoursday','SDS-SDS_Total_T',\n                                           'SDS-SDS_Total_Raw','BIA-BIA_LST','BIA-BIA_FMI','BIA-BIA_FFMI',\n                                           'FGC-FGC_TL_Zone','FGC-FGC_SRR_Zone','FGC-FGC_SRL_Zone',\n                                           'Physical-BMI'])),\n    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))\n])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:55.238305Z","iopub.execute_input":"2025-04-21T14:17:55.238768Z","iopub.status.idle":"2025-04-21T14:17:55.248056Z","shell.execute_reply.started":"2025-04-21T14:17:55.238722Z","shell.execute_reply":"2025-04-21T14:17:55.247014Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_df = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/test.csv')\ntrain_df = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/train.csv')\n\nid_test = test_df['id']\nX_test = test_df.drop(columns=['id'])\n\nX_train = train_df.dropna(subset=['sii'])\ny_train = X_train['sii']\nX_train = X_train.drop(columns=['PCIAT-PCIAT_01','PCIAT-PCIAT_02','PCIAT-PCIAT_03','PCIAT-PCIAT_04','PCIAT-PCIAT_05','PCIAT-PCIAT_06',\n             'PCIAT-PCIAT_07','PCIAT-PCIAT_08','PCIAT-PCIAT_09','PCIAT-PCIAT_10','PCIAT-PCIAT_11','PCIAT-PCIAT_12',\n             'PCIAT-PCIAT_13','PCIAT-PCIAT_14','PCIAT-PCIAT_15','PCIAT-PCIAT_16','PCIAT-PCIAT_17','PCIAT-PCIAT_18',\n             'PCIAT-PCIAT_19','PCIAT-PCIAT_20','PCIAT-PCIAT_20','PCIAT-PCIAT_Total','PCIAT-Season','sii','id'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:55.249366Z","iopub.execute_input":"2025-04-21T14:17:55.249729Z","iopub.status.idle":"2025-04-21T14:17:55.308658Z","shell.execute_reply.started":"2025-04-21T14:17:55.249694Z","shell.execute_reply":"2025-04-21T14:17:55.307509Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Processar os dados de treino\nX_train_balanced, y_train_balanced = preprocessing_pipeline.fit_resample(X_train, y_train)\n\n# Pré-processar os dados de teste (sem balanceamento)\nX_test_processed = preprocessing_pipeline[:-2].transform(X_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:55.309904Z","iopub.execute_input":"2025-04-21T14:17:55.310262Z","iopub.status.idle":"2025-04-21T14:17:55.634545Z","shell.execute_reply.started":"2025-04-21T14:17:55.310230Z","shell.execute_reply":"2025-04-21T14:17:55.632728Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Treinar o modelo\nprediction_pipeline.fit(X_train_balanced, y_train_balanced)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:55.635674Z","iopub.execute_input":"2025-04-21T14:17:55.636258Z","iopub.status.idle":"2025-04-21T14:17:56.398128Z","shell.execute_reply.started":"2025-04-21T14:17:55.636205Z","shell.execute_reply":"2025-04-21T14:17:56.396995Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Fazer previsões no conjunto de teste\ntest_predictions = prediction_pipeline.predict(X_test_processed)\n\n# Exibir resultados\nprint(\"Previsões para o conjunto de teste:\")\nprint(test_predictions)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:56.399573Z","iopub.execute_input":"2025-04-21T14:17:56.400018Z","iopub.status.idle":"2025-04-21T14:17:56.422690Z","shell.execute_reply.started":"2025-04-21T14:17:56.399953Z","shell.execute_reply":"2025-04-21T14:17:56.421677Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Criar o arquivo de submissão\nsubmission = pd.DataFrame({'id': id_test, 'sii': test_predictions})\nsubmission.to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-21T14:17:56.424004Z","iopub.execute_input":"2025-04-21T14:17:56.424293Z","iopub.status.idle":"2025-04-21T14:17:56.431489Z","shell.execute_reply.started":"2025-04-21T14:17:56.424264Z","shell.execute_reply":"2025-04-21T14:17:56.430393Z"}},"outputs":[],"execution_count":null}]}