{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":81933,"databundleVersionId":9643020,"sourceType":"competition"},{"sourceId":165404,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":140738,"modelId":163343}],"dockerImageVersionId":30786,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kaggle Competition : Child Mind Institute — Problematic Internet Use\n### Relating Physical Activity to Problematic Internet Use\n### Competition Link : <https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/overview>\n### Compeition Dataset Link : <https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/data>","metadata":{}},{"cell_type":"markdown","source":"\n## train.csv and test.csv comprises measurements from a variety of instruments\n\n- **Demographics** - Information about age and sex of participants.\n- **Internet Use** - Number of hours of using computer/internet per day.\n- **Children's Global Assessment Scale** - Numeric scale used by mental health clinicians to rate the general functioning of youths under the age of 18.\n- **Physical Measures** - Collection of blood pressure, heart rate, height, weight and waist, and hip measurements.\n- **FitnessGram Vitals and Treadmill** - Measurements of cardiovascular fitness assessed using the NHANES treadmill protocol.\n- **FitnessGram Child** - Health related physical fitness assessment measuring five different parameters including aerobic capacity, muscular strength, muscular endurance, flexibility, and body composition.\n- **Bio-electric Impedance Analysis** - Measure of key body composition elements, including BMI, fat, muscle, and water content.\n- **Physical Activity Questionnaire** - Information about children's participation in vigorous activities over the last 7 days.\n- **Sleep Disturbance Scale** - Scale to categorize sleep disorders in children.\n- **Actigraphy** - Objective measure of ecological physical activity through a research-grade biotracker.\n- **Parent-Child Internet Addiction Test** - 20-item scale that measures characteristics and behaviors associated with compulsive use of the Internet including compulsivity, escapism, and dependency.\n\n- **Note** in particular the field **PCIAT-PCIAT_Total.** The target **sii** for this competition is derived from this field as described in the data dictionary: **0 for None, 1 for Mild, 2 for Moderate, and 3 for Severe.** Additionally, each participant has been assigned a unique identifier id.\n\n### Actigraphy Files and Field Descriptions\nDuring their participation in the HBN study, some participants were given an accelerometer to wear for up to 30 days continually while at home and going about their regular daily lives.\n\n- **series_{train|test}.parquet/id={id}** - Series to be used as training data, partitioned by id. Each series is a continuous recording of accelerometer data for a single subject spanning many days.\n\n- **id** - The patient identifier corresponding to the id field in train/test.csv.\n- **step** - An integer timestep for each observation within a series.\n- **X, Y, Z** - Measure of acceleration, in g, experienced by the wrist-worn watch along each standard axis.\n- **enmo** - As calculated and described by the wristpy package, ENMO is the Euclidean Norm Minus One of all accelerometer signals (along each of the x-, y-, and z-axis, measured in g-force) with negative values rounded to zero. Zero values are indicative of periods of no motion. While no standard measure of acceleration exists in this space, this is one of the several commonly computed features.\n- **anglez** - As calculated and described by the wristpy package, Angle-Z is a metric derived from individual accelerometer components and refers to the angle of the arm relative to the horizontal plane.\n- **non-wear_flag** - A flag (0: watch is being worn, 1: the watch is not worn) to help determine periods when the watch has been removed, based on the GGIR definition, which uses the standard deviation and range of the accelerometer data.\n- **light** - Measure of ambient light in lux. See ​​here for details.\n- **battery_voltage** - A measure of the battery voltage in mV.\n- **time_of_day** - Time of day representing the start of a 5s window that the data has been sampled over, with format **%H:%M:%S.%9f.**\n- **weekday** - The day of the week, coded as an integer with 1 being Monday and 7 being Sunday.\n- **quarter** - The quarter of the year, an integer from 1 to 4.\n- **relative_date_PCIAT** - The number of days (integer) since the PCIAT test was administered (negative days indicate that the actigraphy data has been collected before the test was administered).","metadata":{}},{"cell_type":"markdown","source":"# Install library","metadata":{}},{"cell_type":"code","source":"installpath = \"/kaggle/input/tabnet/pytorch/v1/1/pytorch_tabnet-4.1.0-py3-none-any.whl\"\n!pip install pytorch-tabnet --no-index --no-deps --find-links=file://{installpath}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:27.056063Z","iopub.execute_input":"2024-12-05T04:25:27.056467Z","iopub.status.idle":"2024-12-05T04:25:29.232446Z","shell.execute_reply.started":"2024-12-05T04:25:27.056424Z","shell.execute_reply":"2024-12-05T04:25:29.230938Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"class CFG:\n    \n    USE_GPU= False\n    test =0.2\n    normalize = False#True\n    overSampling = True # handling imbalance class\n    crossValidate = True#True # Support cross Validation \n    nFold = 5#4 # 8  No remainder  for oversampling \n\n    #Ensemble Learning Algorithm Select\n    USE_ENSEMBLE = True #False #True\n    # USE_VOTING = True\n    # USE_WEIGHT = False\n    # USE_STACK = False\n    \n\n    # XGBoost\n    xgbEstimate = 350 #300 #250 #150 #800 #500 #200 # 250 , 300 \n    xgbDepth = 3 #5 #7 #5 #6, low value avoid overfit\n    xgbLR = 0.03 #0.05 # 0.08\n    xgbEarlyStop= 8 #10 #20\n    colsample_bytree = 0.5 #0.8\n\n    # Tabnet model hyperparameter \n    maxEpochs = 1000 #500 #300 #150 #100#200 #300\n    patience = 30 #20 #50 #50 #10 #8 #5\n    n_d = 64#32 #8\n    n_a = 64#32 # 8\n    n_steps = 5 # 3\n    mask_type = \"sparsemax\" #\"entmax\" # \"sparsemax\"\n    learningRate = 1e-2 #1e-3 # 1e-2\n\n    # lightGBM\n    lgbmEstimators = 200\n    \n    trainSeriesParquet = \"series_train.parquet\"\n    testSeriesParquet = \"series_test.parquet\"\n    rootDir =\"/kaggle/input/child-mind-institute-problematic-internet-use/\"\n    dataDictionary = \"/kaggle/input/child-mind-institute-problematic-internet-use/data_dictionary.csv\"\n    trainDataFile= \"/kaggle/input/child-mind-institute-problematic-internet-use/train.csv\"\n    testDataFile = \"/kaggle/input/child-mind-institute-problematic-internet-use/test.csv\"\n    samplesub = \"/kaggle/input/child-mind-institute-problematic-internet-use/sample_submission.csv\"\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:29.234305Z","iopub.execute_input":"2024-12-05T04:25:29.234860Z","iopub.status.idle":"2024-12-05T04:25:29.244074Z","shell.execute_reply.started":"2024-12-05T04:25:29.234800Z","shell.execute_reply":"2024-12-05T04:25:29.242565Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nimport os, gc, time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nfrom IPython.display import Markdown\nimport ctypes\nimport seaborn as sns\nfrom tqdm import tqdm\n\nimport torch\nfrom pytorch_tabnet.tab_model import TabNetClassifier\nfrom pytorch_tabnet.callbacks import Callback\n\n\nfrom sklearn.preprocessing import LabelEncoder, StandardScaler\nfrom sklearn.model_selection import (train_test_split, KFold, GridSearchCV, StratifiedKFold,\n                                    cross_val_score)\nfrom sklearn.metrics import roc_curve, roc_auc_score, accuracy_score\nfrom sklearn.metrics import confusion_matrix, classification_report, ConfusionMatrixDisplay\n\n# over Sample \nfrom imblearn.over_sampling import SMOTE\n\nfrom sklearn.metrics import ( mean_absolute_error, \n                             mean_squared_error, \n                             r2_score, \n                             cohen_kappa_score)\n\nfrom sklearn.model_selection import (train_test_split, \n                                     KFold, \n                                     GridSearchCV, \n                                     StratifiedKFold)\n\n\n# import torch\n# ML library \nfrom xgboost import XGBRFRegressor, XGBClassifier# plot_importance, plot_tree\nimport xgboost\nfrom lightgbm import LGBMRegressor, LGBMClassifier\nimport lightgbm\nfrom catboost import CatBoostRegressor, CatBoostClassifier\nimport catboost\nfrom sklearn.ensemble import (RandomForestClassifier ,VotingClassifier )\n\n# multiple processing lib\nfrom joblib import Parallel , delayed\nfrom concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor\n\n\nfrom warnings import simplefilter \nsimplefilter(action=\"ignore\", category=pd.errors.PerformanceWarning)\n\npd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', None)\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:29.246561Z","iopub.execute_input":"2024-12-05T04:25:29.246938Z","iopub.status.idle":"2024-12-05T04:25:33.901319Z","shell.execute_reply.started":"2024-12-05T04:25:29.246901Z","shell.execute_reply":"2024-12-05T04:25:33.899762Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ndevice","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:33.903659Z","iopub.execute_input":"2024-12-05T04:25:33.904594Z","iopub.status.idle":"2024-12-05T04:25:33.914731Z","shell.execute_reply.started":"2024-12-05T04:25:33.904531Z","shell.execute_reply":"2024-12-05T04:25:33.913130Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Load dataset","metadata":{"execution":{"iopub.status.busy":"2024-10-08T11:48:45.817552Z","iopub.execute_input":"2024-10-08T11:48:45.818114Z","iopub.status.idle":"2024-10-08T11:48:45.823731Z","shell.execute_reply.started":"2024-10-08T11:48:45.818063Z","shell.execute_reply":"2024-10-08T11:48:45.822277Z"}}},{"cell_type":"markdown","source":"### Load meta Data \n#### help us to understand the data feature","metadata":{}},{"cell_type":"code","source":"%%time\ndataDictDF = pd.read_csv(CFG.dataDictionary)\ndataDictDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:33.916842Z","iopub.execute_input":"2024-12-05T04:25:33.917306Z","iopub.status.idle":"2024-12-05T04:25:33.957720Z","shell.execute_reply.started":"2024-12-05T04:25:33.917267Z","shell.execute_reply":"2024-12-05T04:25:33.956481Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.set_option('display.max_rows', 50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:33.959087Z","iopub.execute_input":"2024-12-05T04:25:33.959412Z","iopub.status.idle":"2024-12-05T04:25:33.964846Z","shell.execute_reply.started":"2024-12-05T04:25:33.959382Z","shell.execute_reply":"2024-12-05T04:25:33.963517Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\ntrainDataDF = pd.read_csv(CFG.trainDataFile)\ntrainDataDF\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:33.966662Z","iopub.execute_input":"2024-12-05T04:25:33.967259Z","iopub.status.idle":"2024-12-05T04:25:34.135871Z","shell.execute_reply.started":"2024-12-05T04:25:33.967208Z","shell.execute_reply":"2024-12-05T04:25:34.134575Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainDataDF.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.137250Z","iopub.execute_input":"2024-12-05T04:25:34.137650Z","iopub.status.idle":"2024-12-05T04:25:34.321706Z","shell.execute_reply.started":"2024-12-05T04:25:34.137609Z","shell.execute_reply":"2024-12-05T04:25:34.320394Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\ntestDataDF = pd.read_csv(CFG.testDataFile)\ntestDataDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.325229Z","iopub.execute_input":"2024-12-05T04:25:34.325691Z","iopub.status.idle":"2024-12-05T04:25:34.417785Z","shell.execute_reply.started":"2024-12-05T04:25:34.325630Z","shell.execute_reply":"2024-12-05T04:25:34.416596Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"testDataDF.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.419137Z","iopub.execute_input":"2024-12-05T04:25:34.419467Z","iopub.status.idle":"2024-12-05T04:25:34.426580Z","shell.execute_reply.started":"2024-12-05T04:25:34.419436Z","shell.execute_reply":"2024-12-05T04:25:34.425355Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submit = pd.read_csv(CFG.samplesub)\nsubmit","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.428361Z","iopub.execute_input":"2024-12-05T04:25:34.428896Z","iopub.status.idle":"2024-12-05T04:25:34.451365Z","shell.execute_reply.started":"2024-12-05T04:25:34.428847Z","shell.execute_reply":"2024-12-05T04:25:34.450282Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def cleanMemory():\n    gc.collect()\n    ctypes.CDLL(\"libc.so.6\").malloc_trim(0)\n    torch.cuda.empty_cache()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.452867Z","iopub.execute_input":"2024-12-05T04:25:34.453225Z","iopub.status.idle":"2024-12-05T04:25:34.460231Z","shell.execute_reply.started":"2024-12-05T04:25:34.453190Z","shell.execute_reply":"2024-12-05T04:25:34.458815Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"cleanMemory()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.461393Z","iopub.execute_input":"2024-12-05T04:25:34.461791Z","iopub.status.idle":"2024-12-05T04:25:34.642119Z","shell.execute_reply.started":"2024-12-05T04:25:34.461752Z","shell.execute_reply":"2024-12-05T04:25:34.640880Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Print Columns Values ","metadata":{}},{"cell_type":"code","source":"def printAllcolumnsValue(df, showAll=True):\n    for col in df.columns:\n        if showAll :\n            print(f\"{col} : {df[col].unique()}\") # print unique value\n        else: # only print catergory column\n            if df[col].dtype == \"object\":\n                print(f\"{col} : {df[col].unique()}\") # print unique value","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.643752Z","iopub.execute_input":"2024-12-05T04:25:34.644140Z","iopub.status.idle":"2024-12-05T04:25:34.654967Z","shell.execute_reply.started":"2024-12-05T04:25:34.644104Z","shell.execute_reply":"2024-12-05T04:25:34.653597Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def printSerieUnquieValue(df):\n    print(f\" {df.unique()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.656563Z","iopub.execute_input":"2024-12-05T04:25:34.656966Z","iopub.status.idle":"2024-12-05T04:25:34.667531Z","shell.execute_reply.started":"2024-12-05T04:25:34.656930Z","shell.execute_reply":"2024-12-05T04:25:34.665931Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printSerieUnquieValue(trainDataDF[\"Basic_Demos-Age\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.669029Z","iopub.execute_input":"2024-12-05T04:25:34.669370Z","iopub.status.idle":"2024-12-05T04:25:34.684562Z","shell.execute_reply.started":"2024-12-05T04:25:34.669334Z","shell.execute_reply":"2024-12-05T04:25:34.683049Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printSerieUnquieValue(trainDataDF[\"Physical-Waist_Circumference\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.686088Z","iopub.execute_input":"2024-12-05T04:25:34.686488Z","iopub.status.idle":"2024-12-05T04:25:34.698417Z","shell.execute_reply.started":"2024-12-05T04:25:34.686454Z","shell.execute_reply":"2024-12-05T04:25:34.697146Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printSerieUnquieValue(trainDataDF[\"Physical-Height\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.700027Z","iopub.execute_input":"2024-12-05T04:25:34.700403Z","iopub.status.idle":"2024-12-05T04:25:34.713167Z","shell.execute_reply.started":"2024-12-05T04:25:34.700367Z","shell.execute_reply":"2024-12-05T04:25:34.711803Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printSerieUnquieValue(trainDataDF[\"BIA-BIA_Fat\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.714775Z","iopub.execute_input":"2024-12-05T04:25:34.715236Z","iopub.status.idle":"2024-12-05T04:25:34.728851Z","shell.execute_reply.started":"2024-12-05T04:25:34.715182Z","shell.execute_reply":"2024-12-05T04:25:34.727497Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Get Feature ","metadata":{}},{"cell_type":"code","source":"for item in trainDataDF.columns:\n    print(item)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.730283Z","iopub.execute_input":"2024-12-05T04:25:34.730880Z","iopub.status.idle":"2024-12-05T04:25:34.742139Z","shell.execute_reply.started":"2024-12-05T04:25:34.730823Z","shell.execute_reply":"2024-12-05T04:25:34.740801Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"noColumnInTest = [ item for item in trainDataDF.columns  if item not in testDataDF.columns]\nnoColumnInTest","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.743930Z","iopub.execute_input":"2024-12-05T04:25:34.744370Z","iopub.status.idle":"2024-12-05T04:25:34.757345Z","shell.execute_reply.started":"2024-12-05T04:25:34.744321Z","shell.execute_reply":"2024-12-05T04:25:34.756124Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"featureCols = trainDataDF.columns.tolist()\nfeatureCols.remove(\"id\")\nfeatureCols , len(featureCols)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.759178Z","iopub.execute_input":"2024-12-05T04:25:34.759715Z","iopub.status.idle":"2024-12-05T04:25:34.775273Z","shell.execute_reply.started":"2024-12-05T04:25:34.759659Z","shell.execute_reply":"2024-12-05T04:25:34.774122Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## load Time serise data","metadata":{}},{"cell_type":"code","source":"# read one of time series data\ntrainSerise1 = pd.read_parquet(\"/kaggle/input/child-mind-institute-problematic-internet-use/series_train.parquet/id=00115b9f/part-0.parquet\")\ntrainSerise1","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.776735Z","iopub.execute_input":"2024-12-05T04:25:34.777141Z","iopub.status.idle":"2024-12-05T04:25:34.826991Z","shell.execute_reply.started":"2024-12-05T04:25:34.777104Z","shell.execute_reply":"2024-12-05T04:25:34.825715Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainSerise1.drop(\"step\", axis=1, inplace=True)\ntrainSerise1","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.828711Z","iopub.execute_input":"2024-12-05T04:25:34.829097Z","iopub.status.idle":"2024-12-05T04:25:34.860085Z","shell.execute_reply.started":"2024-12-05T04:25:34.829059Z","shell.execute_reply":"2024-12-05T04:25:34.858689Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sensorFeature =trainSerise1.columns.to_list()\nsensorFeature , len(sensorFeature)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.861930Z","iopub.execute_input":"2024-12-05T04:25:34.862410Z","iopub.status.idle":"2024-12-05T04:25:34.870349Z","shell.execute_reply.started":"2024-12-05T04:25:34.862356Z","shell.execute_reply":"2024-12-05T04:25:34.869095Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# get Statistic Feature list\nstatFeat = trainSerise1.describe().index.to_list() \nstatFeat, len(statFeat)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.871604Z","iopub.execute_input":"2024-12-05T04:25:34.871949Z","iopub.status.idle":"2024-12-05T04:25:34.924369Z","shell.execute_reply.started":"2024-12-05T04:25:34.871908Z","shell.execute_reply":"2024-12-05T04:25:34.923137Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# generate senor feature statistic column list","metadata":{}},{"cell_type":"code","source":"# generate senor feature statistic column list\nsensorStatCol = []\nfor sta in statFeat: # start statist row \n    # print(sta)\n    for item in sensorFeature:\n        # print(item)\n        sensorStatCol.append(f\"{item}_{sta}\")\n        \nsensorStatCol, len(sensorStatCol)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.930193Z","iopub.execute_input":"2024-12-05T04:25:34.930625Z","iopub.status.idle":"2024-12-05T04:25:34.940223Z","shell.execute_reply.started":"2024-12-05T04:25:34.930584Z","shell.execute_reply":"2024-12-05T04:25:34.938599Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Result 12(paramter) x 8 (statistic) = 96 (Data vector) summary (statistic for each type of sensor feature)\ntrainSerise1.describe().values.reshape(-1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:34.941913Z","iopub.execute_input":"2024-12-05T04:25:34.942613Z","iopub.status.idle":"2024-12-05T04:25:35.009179Z","shell.execute_reply.started":"2024-12-05T04:25:34.942556Z","shell.execute_reply":"2024-12-05T04:25:35.008006Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def loadParquetFile(directory, fileName):\n    \"\"\"\n    read parquet file \n    \"\"\"\n    path = os.path.join(directory, fileName, \"part-0.parquet\")\n    df = pd.read_parquet(path)\n    df.drop(\"step\", axis=1, inplace=True) # drop step column\n    statDF = df.describe().values.reshape(-1)\n    return statDF, fileName.split(\"=\")[1] # get ids\n    \n\ndef loadTimeSeriesData(directory):\n    \"\"\"\n    input : root direction\n    \"\"\"\n#     print(\"DIR :\", directory)\n    filesIds = os.listdir(directory) # get list of folder name (files ids)\n#     print(filesIds)\n    with ThreadPoolExecutor() as executor:\n#     with ProcessPoolExecutor() as executor:\n        results = list(tqdm(executor.map(lambda fname: loadParquetFile(directory, fname), filesIds), total=len(filesIds)))\n#     print(results)\n    statistic, ids = zip(*results) # pack into Statistic and Ids \n    # create new dataframe with n statistic sensor data\n#     data = pd.DataFrame(statistic, columns=[f\"stat_{i}\" for i in range(len(statistic[0]))])\n    data = pd.DataFrame(statistic, columns=sensorStatCol)\n    data[\"id\"] = ids # add ids into dataframe\n#     print(data)\n    \n    return data","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:35.010581Z","iopub.execute_input":"2024-12-05T04:25:35.011047Z","iopub.status.idle":"2024-12-05T04:25:35.020615Z","shell.execute_reply.started":"2024-12-05T04:25:35.011008Z","shell.execute_reply":"2024-12-05T04:25:35.019235Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\ntrainTSData = loadTimeSeriesData((CFG.rootDir + CFG.trainSeriesParquet))\ntestTSData = loadTimeSeriesData((CFG.rootDir + CFG.testSeriesParquet))","metadata":{"scrolled":true,"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:25:35.022641Z","iopub.execute_input":"2024-12-05T04:25:35.023892Z","iopub.status.idle":"2024-12-05T04:27:05.768387Z","shell.execute_reply.started":"2024-12-05T04:25:35.023830Z","shell.execute_reply":"2024-12-05T04:27:05.767001Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainTSData.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:05.770314Z","iopub.execute_input":"2024-12-05T04:27:05.771380Z","iopub.status.idle":"2024-12-05T04:27:05.856552Z","shell.execute_reply.started":"2024-12-05T04:27:05.771325Z","shell.execute_reply":"2024-12-05T04:27:05.855351Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainTSData.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:05.858032Z","iopub.execute_input":"2024-12-05T04:27:05.858420Z","iopub.status.idle":"2024-12-05T04:27:05.877040Z","shell.execute_reply.started":"2024-12-05T04:27:05.858387Z","shell.execute_reply":"2024-12-05T04:27:05.875578Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"trainTSData.isna().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:05.878611Z","iopub.execute_input":"2024-12-05T04:27:05.879002Z","iopub.status.idle":"2024-12-05T04:27:05.894299Z","shell.execute_reply.started":"2024-12-05T04:27:05.878964Z","shell.execute_reply":"2024-12-05T04:27:05.893030Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"testTSData","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:05.895867Z","iopub.execute_input":"2024-12-05T04:27:05.896283Z","iopub.status.idle":"2024-12-05T04:27:05.968680Z","shell.execute_reply.started":"2024-12-05T04:27:05.896245Z","shell.execute_reply":"2024-12-05T04:27:05.967331Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Combine Time series data into dataset","metadata":{"execution":{"iopub.status.busy":"2024-10-14T11:06:00.194887Z","iopub.execute_input":"2024-10-14T11:06:00.195376Z","iopub.status.idle":"2024-10-14T11:06:00.202283Z","shell.execute_reply.started":"2024-10-14T11:06:00.195332Z","shell.execute_reply":"2024-10-14T11:06:00.200996Z"}}},{"cell_type":"code","source":"combinedTrainDF =  pd.merge(trainDataDF, trainTSData, how=\"left\", on='id')\ncombinedTestDF = pd.merge(testDataDF, testTSData, how=\"left\", on='id')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:05.970081Z","iopub.execute_input":"2024-12-05T04:27:05.970412Z","iopub.status.idle":"2024-12-05T04:27:05.987789Z","shell.execute_reply.started":"2024-12-05T04:27:05.970381Z","shell.execute_reply":"2024-12-05T04:27:05.986600Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:05.989519Z","iopub.execute_input":"2024-12-05T04:27:05.990856Z","iopub.status.idle":"2024-12-05T04:27:06.163007Z","shell.execute_reply.started":"2024-12-05T04:27:05.990812Z","shell.execute_reply":"2024-12-05T04:27:06.161408Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.164563Z","iopub.execute_input":"2024-12-05T04:27:06.164970Z","iopub.status.idle":"2024-12-05T04:27:06.366305Z","shell.execute_reply.started":"2024-12-05T04:27:06.164931Z","shell.execute_reply":"2024-12-05T04:27:06.365082Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF = combinedTrainDF.drop(\"id\", axis=1)\ncombinedTestDF = combinedTestDF.drop(\"id\", axis=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.368007Z","iopub.execute_input":"2024-12-05T04:27:06.368522Z","iopub.status.idle":"2024-12-05T04:27:06.379030Z","shell.execute_reply.started":"2024-12-05T04:27:06.368451Z","shell.execute_reply":"2024-12-05T04:27:06.377887Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.380486Z","iopub.execute_input":"2024-12-05T04:27:06.380931Z","iopub.status.idle":"2024-12-05T04:27:06.554382Z","shell.execute_reply.started":"2024-12-05T04:27:06.380892Z","shell.execute_reply":"2024-12-05T04:27:06.553048Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Update the feature column list after combined time series featureS","metadata":{}},{"cell_type":"code","source":"featureCols = combinedTrainDF.columns.tolist()\nfeatureCols,  len(featureCols)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.556057Z","iopub.execute_input":"2024-12-05T04:27:06.556567Z","iopub.status.idle":"2024-12-05T04:27:06.568297Z","shell.execute_reply.started":"2024-12-05T04:27:06.556497Z","shell.execute_reply":"2024-12-05T04:27:06.567123Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.569991Z","iopub.execute_input":"2024-12-05T04:27:06.570373Z","iopub.status.idle":"2024-12-05T04:27:06.772246Z","shell.execute_reply.started":"2024-12-05T04:27:06.570330Z","shell.execute_reply":"2024-12-05T04:27:06.770962Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data cleaning","metadata":{}},{"cell_type":"code","source":"combinedTrainDF[\"sii\"].isnull().sum() # count number of Nan row in sii target column","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.774005Z","iopub.execute_input":"2024-12-05T04:27:06.774484Z","iopub.status.idle":"2024-12-05T04:27:06.784423Z","shell.execute_reply.started":"2024-12-05T04:27:06.774433Z","shell.execute_reply":"2024-12-05T04:27:06.782911Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF = combinedTrainDF.dropna(subset=\"sii\")\ncombinedTrainDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.786130Z","iopub.execute_input":"2024-12-05T04:27:06.786572Z","iopub.status.idle":"2024-12-05T04:27:06.961192Z","shell.execute_reply.started":"2024-12-05T04:27:06.786493Z","shell.execute_reply":"2024-12-05T04:27:06.959842Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF[\"sii\"].isnull().sum() # check Nan in sii column","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.962572Z","iopub.execute_input":"2024-12-05T04:27:06.962911Z","iopub.status.idle":"2024-12-05T04:27:06.971581Z","shell.execute_reply.started":"2024-12-05T04:27:06.962878Z","shell.execute_reply":"2024-12-05T04:27:06.970256Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## find caterogy column","metadata":{}},{"cell_type":"code","source":"#find \nprintAllcolumnsValue(combinedTrainDF, showAll= False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.973309Z","iopub.execute_input":"2024-12-05T04:27:06.973959Z","iopub.status.idle":"2024-12-05T04:27:06.992102Z","shell.execute_reply.started":"2024-12-05T04:27:06.973895Z","shell.execute_reply":"2024-12-05T04:27:06.990894Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# find which column if catargory\ncolumnsCaterogy = []\nfor col in combinedTrainDF.columns:\n    if combinedTrainDF[col].dtype == \"object\":\n        columnsCaterogy.append(col)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:06.993656Z","iopub.execute_input":"2024-12-05T04:27:06.994098Z","iopub.status.idle":"2024-12-05T04:27:07.008277Z","shell.execute_reply.started":"2024-12-05T04:27:06.994041Z","shell.execute_reply":"2024-12-05T04:27:07.007045Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"columnsCaterogy , len(columnsCaterogy)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.009844Z","iopub.execute_input":"2024-12-05T04:27:07.010242Z","iopub.status.idle":"2024-12-05T04:27:07.023226Z","shell.execute_reply.started":"2024-12-05T04:27:07.010204Z","shell.execute_reply":"2024-12-05T04:27:07.022011Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# find which colum if numunic\ncolumnsNum = []\nfor col in combinedTrainDF.columns:\n    if combinedTrainDF[col].dtype != \"object\" and col !=\"sii\":\n        columnsNum.append(col)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.024824Z","iopub.execute_input":"2024-12-05T04:27:07.025228Z","iopub.status.idle":"2024-12-05T04:27:07.037178Z","shell.execute_reply.started":"2024-12-05T04:27:07.025178Z","shell.execute_reply":"2024-12-05T04:27:07.035919Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"len(columnsNum)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.038834Z","iopub.execute_input":"2024-12-05T04:27:07.039344Z","iopub.status.idle":"2024-12-05T04:27:07.052207Z","shell.execute_reply.started":"2024-12-05T04:27:07.039290Z","shell.execute_reply":"2024-12-05T04:27:07.050616Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"columnsTestCaterogy = []\nfor col in combinedTestDF.columns:\n    if combinedTestDF[col].dtype == \"object\":\n        columnsTestCaterogy.append(col)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.053985Z","iopub.execute_input":"2024-12-05T04:27:07.054384Z","iopub.status.idle":"2024-12-05T04:27:07.070145Z","shell.execute_reply.started":"2024-12-05T04:27:07.054345Z","shell.execute_reply":"2024-12-05T04:27:07.068845Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"columnsTestCaterogy , len(columnsTestCaterogy)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.071491Z","iopub.execute_input":"2024-12-05T04:27:07.072018Z","iopub.status.idle":"2024-12-05T04:27:07.084257Z","shell.execute_reply.started":"2024-12-05T04:27:07.071978Z","shell.execute_reply":"2024-12-05T04:27:07.083073Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# find which colum if numunic\ncolumnsTestNum = []\nfor col in combinedTestDF.columns:\n    if combinedTestDF[col].dtype != \"object\" and col !=\"sii\":\n        columnsTestNum.append(col)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.085800Z","iopub.execute_input":"2024-12-05T04:27:07.086261Z","iopub.status.idle":"2024-12-05T04:27:07.099734Z","shell.execute_reply.started":"2024-12-05T04:27:07.086220Z","shell.execute_reply":"2024-12-05T04:27:07.098162Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"len(columnsTestNum)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.101416Z","iopub.execute_input":"2024-12-05T04:27:07.101854Z","iopub.status.idle":"2024-12-05T04:27:07.116183Z","shell.execute_reply.started":"2024-12-05T04:27:07.101815Z","shell.execute_reply":"2024-12-05T04:27:07.114834Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Seem the Training dataset column more than test(submi) dataset, the training dataset should same feature size with test(submit) feature size \n### select feature for training dateset same with testing dataset","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tempCol =columnsTestCaterogy + columnsTestNum +[\"sii\"]\ntempCol2 = columnsTestCaterogy + columnsTestNum\nlen(tempCol) , len(tempCol2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.117753Z","iopub.execute_input":"2024-12-05T04:27:07.118137Z","iopub.status.idle":"2024-12-05T04:27:07.132003Z","shell.execute_reply.started":"2024-12-05T04:27:07.118100Z","shell.execute_reply":"2024-12-05T04:27:07.130374Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tempCol2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.133368Z","iopub.execute_input":"2024-12-05T04:27:07.133900Z","iopub.status.idle":"2024-12-05T04:27:07.147740Z","shell.execute_reply.started":"2024-12-05T04:27:07.133847Z","shell.execute_reply":"2024-12-05T04:27:07.146341Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# select column for match both training and testing feature\ncombinedTrainDF =combinedTrainDF[tempCol]\ncombinedTestDF = combinedTestDF[tempCol2]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.149305Z","iopub.execute_input":"2024-12-05T04:27:07.149694Z","iopub.status.idle":"2024-12-05T04:27:07.168003Z","shell.execute_reply.started":"2024-12-05T04:27:07.149655Z","shell.execute_reply":"2024-12-05T04:27:07.166575Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF.describe() # final train dataset feature is 155 with label","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.169985Z","iopub.execute_input":"2024-12-05T04:27:07.170448Z","iopub.status.idle":"2024-12-05T04:27:07.544097Z","shell.execute_reply.started":"2024-12-05T04:27:07.170404Z","shell.execute_reply":"2024-12-05T04:27:07.542855Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Other column Nan count","metadata":{}},{"cell_type":"code","source":"combinedTrainDF.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.545375Z","iopub.execute_input":"2024-12-05T04:27:07.545746Z","iopub.status.idle":"2024-12-05T04:27:07.558449Z","shell.execute_reply.started":"2024-12-05T04:27:07.545708Z","shell.execute_reply":"2024-12-05T04:27:07.557112Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## cleaning/handling other column Nan value","metadata":{}},{"cell_type":"code","source":"combinedTrainDF.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.560168Z","iopub.execute_input":"2024-12-05T04:27:07.560837Z","iopub.status.idle":"2024-12-05T04:27:07.578659Z","shell.execute_reply.started":"2024-12-05T04:27:07.560699Z","shell.execute_reply":"2024-12-05T04:27:07.577282Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.580304Z","iopub.execute_input":"2024-12-05T04:27:07.580698Z","iopub.status.idle":"2024-12-05T04:27:07.592672Z","shell.execute_reply.started":"2024-12-05T04:27:07.580661Z","shell.execute_reply":"2024-12-05T04:27:07.591291Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.604863Z","iopub.execute_input":"2024-12-05T04:27:07.605319Z","iopub.status.idle":"2024-12-05T04:27:07.613187Z","shell.execute_reply.started":"2024-12-05T04:27:07.605273Z","shell.execute_reply":"2024-12-05T04:27:07.611972Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# fill the Nan (missing) in for category\nfor c in columnsTestCaterogy:\n    combinedTrainDF[c] = combinedTrainDF[c].fillna('missing') # fill na for missing caterogy\n    combinedTestDF[c] = combinedTestDF[c].fillna(\"missing\") # fill na for missing caterogy\n    ","metadata":{"scrolled":true,"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.615321Z","iopub.execute_input":"2024-12-05T04:27:07.615822Z","iopub.status.idle":"2024-12-05T04:27:07.643280Z","shell.execute_reply.started":"2024-12-05T04:27:07.615764Z","shell.execute_reply":"2024-12-05T04:27:07.642031Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# fill the Nan numberic with median value \nfor c in columnsTestNum:\n    combinedTrainDF[c] = combinedTrainDF[c].fillna(combinedTrainDF[c].median())\n    combinedTestDF[c] = combinedTestDF[c].fillna(combinedTestDF[c].median())","metadata":{"scrolled":true,"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.645059Z","iopub.execute_input":"2024-12-05T04:27:07.645576Z","iopub.status.idle":"2024-12-05T04:27:07.825548Z","shell.execute_reply.started":"2024-12-05T04:27:07.645520Z","shell.execute_reply":"2024-12-05T04:27:07.824316Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF.isnull().sum() # check is any null in column","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.826911Z","iopub.execute_input":"2024-12-05T04:27:07.827286Z","iopub.status.idle":"2024-12-05T04:27:07.850567Z","shell.execute_reply.started":"2024-12-05T04:27:07.827248Z","shell.execute_reply":"2024-12-05T04:27:07.849379Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.852086Z","iopub.execute_input":"2024-12-05T04:27:07.852479Z","iopub.status.idle":"2024-12-05T04:27:07.872709Z","shell.execute_reply.started":"2024-12-05T04:27:07.852444Z","shell.execute_reply":"2024-12-05T04:27:07.871205Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Encode the Cateroy column into numberical ","metadata":{}},{"cell_type":"code","source":"# define Category dictionary for encode \ncatDic = {\"missing\": 0, \"Spring\": 1, \"Summer\": 2, \"Fall\": 3, \"Winter\" : 4}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.874195Z","iopub.execute_input":"2024-12-05T04:27:07.874609Z","iopub.status.idle":"2024-12-05T04:27:07.887603Z","shell.execute_reply.started":"2024-12-05T04:27:07.874570Z","shell.execute_reply":"2024-12-05T04:27:07.886043Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# labelEncoder = LabelEncoder()\n# for c in columnsTestCaterogy:\n#     combinedTrainDF[c] = labelEncoder.fit_transform(combinedTrainDF[c]).astype(int)\n#     combinedTestDF[c] = labelEncoder.fit_transform(combinedTestDF[c]).astype(int)\nfor c in columnsTestCaterogy:\n    combinedTrainDF[c] = combinedTrainDF[c].map(catDic)\n    combinedTestDF[c] =combinedTestDF[c].map(catDic)","metadata":{"scrolled":true,"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.889405Z","iopub.execute_input":"2024-12-05T04:27:07.890534Z","iopub.status.idle":"2024-12-05T04:27:07.918030Z","shell.execute_reply.started":"2024-12-05T04:27:07.890451Z","shell.execute_reply":"2024-12-05T04:27:07.916724Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## find incorrect value in weight","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.max_rows', None)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.919575Z","iopub.execute_input":"2024-12-05T04:27:07.919971Z","iopub.status.idle":"2024-12-05T04:27:07.926979Z","shell.execute_reply.started":"2024-12-05T04:27:07.919927Z","shell.execute_reply":"2024-12-05T04:27:07.925252Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF[combinedTrainDF['Physical-Weight'] ==0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:07.928218Z","iopub.execute_input":"2024-12-05T04:27:07.928719Z","iopub.status.idle":"2024-12-05T04:27:08.357696Z","shell.execute_reply.started":"2024-12-05T04:27:07.928675Z","shell.execute_reply":"2024-12-05T04:27:08.356493Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF['Physical-Weight'].replace(0, combinedTrainDF['Physical-Weight'].median() , \n                                           inplace=True )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.359362Z","iopub.execute_input":"2024-12-05T04:27:08.360038Z","iopub.status.idle":"2024-12-05T04:27:08.369374Z","shell.execute_reply.started":"2024-12-05T04:27:08.359979Z","shell.execute_reply":"2024-12-05T04:27:08.367808Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF[combinedTrainDF['Physical-Weight'] == 0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.371020Z","iopub.execute_input":"2024-12-05T04:27:08.371543Z","iopub.status.idle":"2024-12-05T04:27:08.413967Z","shell.execute_reply.started":"2024-12-05T04:27:08.371470Z","shell.execute_reply":"2024-12-05T04:27:08.412799Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# find incorrect value in BMI","metadata":{}},{"cell_type":"code","source":"# 703  *  75.8/ (60.50**2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.415310Z","iopub.execute_input":"2024-12-05T04:27:08.415695Z","iopub.status.idle":"2024-12-05T04:27:08.421828Z","shell.execute_reply.started":"2024-12-05T04:27:08.415658Z","shell.execute_reply":"2024-12-05T04:27:08.420608Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF[combinedTrainDF[\"Physical-BMI\"]== 0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.423349Z","iopub.execute_input":"2024-12-05T04:27:08.423868Z","iopub.status.idle":"2024-12-05T04:27:08.557314Z","shell.execute_reply.started":"2024-12-05T04:27:08.423815Z","shell.execute_reply":"2024-12-05T04:27:08.556124Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF[\"Physical-BMI\"].replace(0, combinedTrainDF[\"Physical-BMI\"].median() , inplace=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.559070Z","iopub.execute_input":"2024-12-05T04:27:08.559586Z","iopub.status.idle":"2024-12-05T04:27:08.568277Z","shell.execute_reply.started":"2024-12-05T04:27:08.559531Z","shell.execute_reply":"2024-12-05T04:27:08.567036Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF[combinedTrainDF[\"Physical-BMI\"]== 0]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.569873Z","iopub.execute_input":"2024-12-05T04:27:08.570436Z","iopub.status.idle":"2024-12-05T04:27:08.610853Z","shell.execute_reply.started":"2024-12-05T04:27:08.570381Z","shell.execute_reply":"2024-12-05T04:27:08.609716Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.set_option('display.max_rows', 50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.612179Z","iopub.execute_input":"2024-12-05T04:27:08.612569Z","iopub.status.idle":"2024-12-05T04:27:08.617429Z","shell.execute_reply.started":"2024-12-05T04:27:08.612525Z","shell.execute_reply":"2024-12-05T04:27:08.616325Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Feature Engineering, create new feature for improve model training/prediction","metadata":{}},{"cell_type":"markdown","source":"# Feature Engineering for generate new column","metadata":{}},{"cell_type":"markdown","source":"# Define Complex Health Analysis function","metadata":{}},{"cell_type":"code","source":"def getBMIInfo(df):\n    \"\"\"\n    classify Weight caterogy by BMI reference from World Health Organization  \n    \"\"\"\n    # print(df)\n    bmi = df[\"Physical-BMI\"]\n    age = df[\"Basic_Demos-Age\"]\n    gender = df[\"Basic_Demos-Sex\"]\n    bodyweightClass = 0\n    if df[\"Basic_Demos-Sex\"] == 0: # for female\n        if age < 8 :\n            adaptThreshold1 = 12.0\n            adaptThreshold2 = 13.0\n            adaptThreshold3 = 17.7\n            adaptThreshold4 = 0.5*(age-5) + 19.0 \n            if bmi < adaptThreshold1:\n                bodyweightClass = 0 # Severe thinness\n            elif bmi <adaptThreshold2:\n                bodyweightClass = 1 # thiness\n            elif bmi <adaptThreshold3:\n                bodyweightClass = 2 # normal \n            elif bmi <adaptThreshold4:\n                 bodyweightClass = 3 # Overweight \n            else :\n                 bodyweightClass = 4 # Obesity\n\n            return bodyweightClass\n            \n        elif age < 15:\n            # apply linear regression \n            adaptThreshold1 = 0.357 * (age-8) + 12.0\n            adaptThreshold2 = 0.429 * (age-8) + 13.0\n            adaptThreshold3 = 0.828 * (age-8) + 17.7\n            adaptThreshold4 = 1.1*(age-8) + 20.5 \n            if bmi < adaptThreshold1:\n                bodyweightClass = 0 # Severe thinness\n            elif bmi <adaptThreshold2:\n                bodyweightClass = 1 # thiness\n            elif bmi <adaptThreshold3:\n                bodyweightClass = 2 # normal \n            elif bmi <adaptThreshold4:\n                 bodyweightClass = 3 # Overweight \n            else :\n                 bodyweightClass = 4 # Obesity\n\n            return bodyweightClass\n        \n        elif age < 19:\n            # apply linear regression \n            adaptThreshold1 = 14.7\n            adaptThreshold2 = 16.5\n            adaptThreshold3 = 0.5 * (age-15) + 23.5\n            adaptThreshold4 = 0.325 *(age-15) + 28.2 \n            if bmi < adaptThreshold1:\n                bodyweightClass = 0 # Severe thinness\n            elif bmi <adaptThreshold2:\n                bodyweightClass = 1 # thiness\n            elif bmi <adaptThreshold3:\n                bodyweightClass = 2 # normal \n            elif bmi <adaptThreshold4:\n                 bodyweightClass = 3 # Overweight \n            else :\n                 bodyweightClass = 4 # Obesity\n\n            return bodyweightClass\n        else: \n            #use adult\n            if bmi < 18.5: \n                bodyweightClass = 0 # thiness \n            elif bmi < 24.9:\n                bodyweightClass = 2 # noraml \n            elif bmi < 29.9:\n                 bodyweightClass = 3 # Overweight \n            else:\n                bodyweightClass = 4 # Obesity\n\n            return bodyweightClass\n\n\n    else: # for male\n        if age < 8 :\n            adaptThreshold1 = 12.5\n            adaptThreshold2 = 13.2\n            adaptThreshold3 = 0.266 *(age-5) + 16.7\n            adaptThreshold4 = 0.5*(age-5) + 18.2 \n            if bmi < adaptThreshold1:\n                bodyweightClass = 0 # Severe thinness\n            elif bmi <adaptThreshold2:\n                bodyweightClass = 1 # thiness\n            elif bmi <adaptThreshold3:\n                bodyweightClass = 2 # normal \n            elif bmi <adaptThreshold4:\n                 bodyweightClass = 3 # Overweight \n            else :\n                 bodyweightClass = 4 # Obesity\n\n            return bodyweightClass\n            \n        elif age < 15:\n            # apply linear regression \n            adaptThreshold1 = 0.314 * (age-8) + 12.5\n            adaptThreshold2 = 0.4 * (age-8) + 13.2\n            adaptThreshold3 = 0.742 * (age-8) + 17.5\n            adaptThreshold4 = 1.04 *(age-8) + 19.7 \n            if bmi < adaptThreshold1:\n                bodyweightClass = 0 # Severe thinness\n            elif bmi <adaptThreshold2:\n                bodyweightClass = 1 # thiness\n            elif bmi <adaptThreshold3:\n                bodyweightClass = 2 # normal \n            elif bmi <adaptThreshold4:\n                 bodyweightClass = 3 # Overweight \n            else :\n                 bodyweightClass = 4 # Obesity\n\n            return bodyweightClass\n        \n        elif age < 19:\n            # apply linear regression \n            adaptThreshold1 = 15.7\n            adaptThreshold2 = 0.375 * (age-15) + 16.0\n            adaptThreshold3 = 0.7 * (age-15) + 22.7\n            adaptThreshold4 = 0.675 *(age-15) + 27.0\n            if bmi < adaptThreshold1:\n                bodyweightClass = 0 # Severe thinness\n            elif bmi <adaptThreshold2:\n                bodyweightClass = 1 # thiness\n            elif bmi <adaptThreshold3:\n                bodyweightClass = 2 # normal \n            elif bmi <adaptThreshold4:\n                 bodyweightClass = 3 # Overweight \n            else :\n                 bodyweightClass = 4 # Obesity\n\n            return bodyweightClass\n        else: \n            #use adult\n            if bmi < 18.5: \n                bodyweightClass = 0 # thiness \n            elif bmi < 24.9:\n                bodyweightClass = 2 # noraml \n            elif bmi < 29.9:\n                 bodyweightClass = 3 # Overweight \n            else:\n                bodyweightClass = 4 # Obesity\n            return bodyweightClass","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.619326Z","iopub.execute_input":"2024-12-05T04:27:08.619835Z","iopub.status.idle":"2024-12-05T04:27:08.638758Z","shell.execute_reply.started":"2024-12-05T04:27:08.619784Z","shell.execute_reply":"2024-12-05T04:27:08.637335Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def getWaistHeightInfo(df):\n    age = df[\"Basic_Demos-Age\"]\n    gender = df[\"Basic_Demos-Sex\"]\n    ratio = df[\"Physical-Waist_Circumference\"] / df[\"Physical-Height\"]\n    ratioClass = 0\n    if age < 15:\n        if ratio < 0.34:\n            ratioClass =0 # Extremely Slim\n        elif ratio < 0.45:\n            ratioClass  = 1 # slim\n        elif ratio < 0.51:\n            ratioClass = 2 # Healthy\n        elif ratio < 0.63:\n            ratioClass = 3 # over weight\n        else:\n            ratioClass = 5 # Obese\n        return ratioClass \n\n    else : # for adult \n        if gender == 0: # for female\n            if ratio < 0.34:\n                ratioClass =0 # Extremely Slim\n            elif ratio < 0.41:\n                ratioClass  = 1 # slim\n            elif ratio < 0.48:\n                ratioClass = 2 # Healthy\n            elif ratio < 0.53:\n                ratioClass = 3 # over weight\n            elif ratio < 0.57:\n                ratioClass = 4 # Very Overweight\n            else:\n                ratioClass = 5 # Obese\n            return ratioClass\n        else : # for male\n            if ratio < 0.34:\n                ratioClass =0 # Extremely Slim\n            elif ratio < 0.42:\n                ratioClass  = 1 # slim\n            elif ratio < 0.52:\n                ratioClass = 2 # Healthy\n            elif ratio < 0.57:\n                ratioClass = 3 # over weight\n            elif ratio < 0.62:\n                ratioClass = 4 # Very Overweight\n            else:\n                ratioClass = 5 # Obese\n            return ratioClass","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.640335Z","iopub.execute_input":"2024-12-05T04:27:08.641422Z","iopub.status.idle":"2024-12-05T04:27:08.655881Z","shell.execute_reply.started":"2024-12-05T04:27:08.641364Z","shell.execute_reply":"2024-12-05T04:27:08.654717Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def getBodyFatPercentInfo(df):\n    \"\"\"\n    classify the user health status by body fat percentage\n    \"\"\"\n    gender = df[\"Basic_Demos-Sex\"]\n    fatPercent = df[\"BIA-BIA_Fat\"]\n    fatclass =0\n    if gender == 0: # for female\n        if fatPercent < 13:  \n            fatclass = 0 # Essential Fat\n        elif fatPercent < 20:\n            fatclass = 1 # Athletes\n        elif fatPercent < 24:\n            fatclass = 2 # Fitness\n        elif fatPercent < 31:\n            fatclass = 3 # acceptable \n        else:\n            fatclass = 4 # acceptable \n        return fatclass\n    else: # for male\n        if fatPercent < 5:  \n            fatclass = 0 # Essential Fat\n        elif fatPercent < 13:\n            fatclass = 1 # Athletes\n        elif fatPercent < 17:\n            fatclass = 2 # Fitness\n        elif fatPercent < 24:\n            fatclass = 3 # acceptable \n        else:\n            fatclass = 4 # acceptable\n        return fatclass","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.657341Z","iopub.execute_input":"2024-12-05T04:27:08.657810Z","iopub.status.idle":"2024-12-05T04:27:08.672455Z","shell.execute_reply.started":"2024-12-05T04:27:08.657761Z","shell.execute_reply":"2024-12-05T04:27:08.671150Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:08.674393Z","iopub.execute_input":"2024-12-05T04:27:08.674838Z","iopub.status.idle":"2024-12-05T04:27:09.208409Z","shell.execute_reply.started":"2024-12-05T04:27:08.674799Z","shell.execute_reply":"2024-12-05T04:27:09.207285Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Extract Feature","metadata":{}},{"cell_type":"code","source":"def featureEngineering(df):\n    df[\"BMI-AGE\"] = df[\"Physical-BMI\"] * df[\"Basic_Demos-Age\"]\n    df[\"BMI-Classify\"]= df.apply(getBMIInfo, axis=1)\n    df[\"Waist-Height-Ratio\"] = df[\"Physical-Waist_Circumference\"] / df[\"Physical-Height\"]\n    df[\"Waist-Height-Ratio-Classify\"] = df.apply(getWaistHeightInfo, axis=1) \n    df['Internet-Hours-Age'] = df['PreInt_EduHx-computerinternet_hoursday'] * df['Basic_Demos-Age']\n    df['BMI-Internet-Hours'] = df['Physical-BMI'] * df['PreInt_EduHx-computerinternet_hoursday']\n    df[\"FATPercent-Classify\"]= df.apply(getBodyFatPercentInfo, axis= 1)\n    df['ICW_TBW'] = df['BIA-BIA_ICW'] / df['BIA-BIA_TBW']\n    df['FFMI_BFP'] = df['BIA-BIA_FFMI'] / df['BIA-BIA_Fat'] #  \n    df['FMI_BFP'] = df['BIA-BIA_FMI'] / df['BIA-BIA_Fat']\n    df['LST_TBW'] = df['BIA-BIA_LST'] / df['BIA-BIA_TBW']\n    df['BFP_BMR'] = df['BIA-BIA_Fat'] * df['BIA-BIA_BMR']\n    df['BFP_DEE'] = df['BIA-BIA_Fat'] * df['BIA-BIA_DEE']\n    df['BMR_Weight'] = df['BIA-BIA_BMR'] / df['Physical-Weight']\n    df['BMR_Weight'] =df['BMR_Weight'].fillna(0)\n    df['DEE_Weight'] = df['BIA-BIA_DEE'] / df['Physical-Weight']\n    df['DEE_Weight'] = df['DEE_Weight'].fillna(0)\n    df['SMM_Height'] = df['BIA-BIA_SMM'] / df['Physical-Height']\n    # Feature 3\n    df['Muscle_to_Fat'] = df['BIA-BIA_SMM'] / df['BIA-BIA_FMI']\n    df['Hydration_Status'] = df['BIA-BIA_TBW'] / df['Physical-Weight']\n    df['BFP_BMI'] = df['BIA-BIA_Fat'] / df['BIA-BIA_BMI']\n    \n    return df\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.209427Z","iopub.execute_input":"2024-12-05T04:27:09.209785Z","iopub.status.idle":"2024-12-05T04:27:09.219705Z","shell.execute_reply.started":"2024-12-05T04:27:09.209709Z","shell.execute_reply":"2024-12-05T04:27:09.218476Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# ","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printSerieUnquieValue(combinedTrainDF[\"BIA-BIA_DEE\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.221153Z","iopub.execute_input":"2024-12-05T04:27:09.221530Z","iopub.status.idle":"2024-12-05T04:27:09.238663Z","shell.execute_reply.started":"2024-12-05T04:27:09.221472Z","shell.execute_reply":"2024-12-05T04:27:09.237262Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printSerieUnquieValue(combinedTrainDF[\"BIA-BIA_Fat\"])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.240087Z","iopub.execute_input":"2024-12-05T04:27:09.240433Z","iopub.status.idle":"2024-12-05T04:27:09.252946Z","shell.execute_reply.started":"2024-12-05T04:27:09.240398Z","shell.execute_reply":"2024-12-05T04:27:09.251573Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# printSerieUnquieValue(combinedTrainDF['Physical-Weight'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.254270Z","iopub.execute_input":"2024-12-05T04:27:09.254636Z","iopub.status.idle":"2024-12-05T04:27:09.266653Z","shell.execute_reply.started":"2024-12-05T04:27:09.254604Z","shell.execute_reply":"2024-12-05T04:27:09.264465Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF= featureEngineering(combinedTestDF)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.268059Z","iopub.execute_input":"2024-12-05T04:27:09.268397Z","iopub.status.idle":"2024-12-05T04:27:09.295352Z","shell.execute_reply.started":"2024-12-05T04:27:09.268366Z","shell.execute_reply":"2024-12-05T04:27:09.294248Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF = featureEngineering(combinedTrainDF)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.296740Z","iopub.execute_input":"2024-12-05T04:27:09.297209Z","iopub.status.idle":"2024-12-05T04:27:09.429105Z","shell.execute_reply.started":"2024-12-05T04:27:09.297174Z","shell.execute_reply":"2024-12-05T04:27:09.427698Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# combinedTestDF[\"BMI-Classify\"] = combinedTestDF.apply(getBMIInfo, axis=1)\ncombinedTrainDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.430545Z","iopub.execute_input":"2024-12-05T04:27:09.430887Z","iopub.status.idle":"2024-12-05T04:27:09.601172Z","shell.execute_reply.started":"2024-12-05T04:27:09.430855Z","shell.execute_reply":"2024-12-05T04:27:09.599987Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.602423Z","iopub.execute_input":"2024-12-05T04:27:09.602916Z","iopub.status.idle":"2024-12-05T04:27:09.831850Z","shell.execute_reply.started":"2024-12-05T04:27:09.602879Z","shell.execute_reply":"2024-12-05T04:27:09.829913Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printAllcolumnsValue(combinedTrainDF, showAll= False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.833929Z","iopub.execute_input":"2024-12-05T04:27:09.834328Z","iopub.status.idle":"2024-12-05T04:27:09.846955Z","shell.execute_reply.started":"2024-12-05T04:27:09.834289Z","shell.execute_reply":"2024-12-05T04:27:09.845134Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printAllcolumnsValue(combinedTestDF, showAll= False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.848563Z","iopub.execute_input":"2024-12-05T04:27:09.849070Z","iopub.status.idle":"2024-12-05T04:27:09.866900Z","shell.execute_reply.started":"2024-12-05T04:27:09.849016Z","shell.execute_reply":"2024-12-05T04:27:09.865337Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# update Festure columns list","metadata":{}},{"cell_type":"code","source":"tempCol = combinedTrainDF.columns.tolist()\ntempCol,  len(tempCol)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.868252Z","iopub.execute_input":"2024-12-05T04:27:09.868692Z","iopub.status.idle":"2024-12-05T04:27:09.883339Z","shell.execute_reply.started":"2024-12-05T04:27:09.868657Z","shell.execute_reply":"2024-12-05T04:27:09.881994Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tempCol2 = combinedTestDF.columns.tolist()\ntempCol2,  len(tempCol2)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.885105Z","iopub.execute_input":"2024-12-05T04:27:09.885443Z","iopub.status.idle":"2024-12-05T04:27:09.907031Z","shell.execute_reply.started":"2024-12-05T04:27:09.885412Z","shell.execute_reply":"2024-12-05T04:27:09.905528Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# combinedTestDF[columnsTestCaterogy[0]].values","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.908851Z","iopub.execute_input":"2024-12-05T04:27:09.909255Z","iopub.status.idle":"2024-12-05T04:27:09.918107Z","shell.execute_reply.started":"2024-12-05T04:27:09.909219Z","shell.execute_reply":"2024-12-05T04:27:09.917022Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printAllcolumnsValue(combinedTrainDF, showAll= False) # ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.919873Z","iopub.execute_input":"2024-12-05T04:27:09.920377Z","iopub.status.idle":"2024-12-05T04:27:09.934571Z","shell.execute_reply.started":"2024-12-05T04:27:09.920323Z","shell.execute_reply":"2024-12-05T04:27:09.933360Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"printAllcolumnsValue(combinedTestDF, showAll= False) # ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.936044Z","iopub.execute_input":"2024-12-05T04:27:09.936427Z","iopub.status.idle":"2024-12-05T04:27:09.951365Z","shell.execute_reply.started":"2024-12-05T04:27:09.936381Z","shell.execute_reply":"2024-12-05T04:27:09.950059Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF[columnsTestCaterogy[0]].value_counts() # check Encoded caterogy value","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.953387Z","iopub.execute_input":"2024-12-05T04:27:09.953817Z","iopub.status.idle":"2024-12-05T04:27:09.969825Z","shell.execute_reply.started":"2024-12-05T04:27:09.953779Z","shell.execute_reply":"2024-12-05T04:27:09.968484Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF[columnsTestCaterogy[0]].value_counts() # check Encoded caterogy value","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.971531Z","iopub.execute_input":"2024-12-05T04:27:09.972129Z","iopub.status.idle":"2024-12-05T04:27:09.988588Z","shell.execute_reply.started":"2024-12-05T04:27:09.972072Z","shell.execute_reply":"2024-12-05T04:27:09.987130Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:09.990033Z","iopub.execute_input":"2024-12-05T04:27:09.990486Z","iopub.status.idle":"2024-12-05T04:27:10.010283Z","shell.execute_reply.started":"2024-12-05T04:27:09.990437Z","shell.execute_reply":"2024-12-05T04:27:10.009006Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF.isnull().sum()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:10.011461Z","iopub.execute_input":"2024-12-05T04:27:10.011838Z","iopub.status.idle":"2024-12-05T04:27:10.027258Z","shell.execute_reply.started":"2024-12-05T04:27:10.011803Z","shell.execute_reply":"2024-12-05T04:27:10.026081Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## seem The Dataset is very imbalance","metadata":{}},{"cell_type":"code","source":"combinedTrainDF[\"sii\"].value_counts().plot(kind=\"bar\", title=\"sii target catergory Distribution\");","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:10.028853Z","iopub.execute_input":"2024-12-05T04:27:10.029232Z","iopub.status.idle":"2024-12-05T04:27:10.325905Z","shell.execute_reply.started":"2024-12-05T04:27:10.029198Z","shell.execute_reply":"2024-12-05T04:27:10.324798Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTrainDF.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:10.327174Z","iopub.execute_input":"2024-12-05T04:27:10.327486Z","iopub.status.idle":"2024-12-05T04:27:10.729184Z","shell.execute_reply.started":"2024-12-05T04:27:10.327456Z","shell.execute_reply":"2024-12-05T04:27:10.727807Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"combinedTestDF.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:10.730744Z","iopub.execute_input":"2024-12-05T04:27:10.731269Z","iopub.status.idle":"2024-12-05T04:27:11.120971Z","shell.execute_reply.started":"2024-12-05T04:27:10.731217Z","shell.execute_reply":"2024-12-05T04:27:11.119746Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train/Validation Split\n## K-fold Cross Vailation","metadata":{}},{"cell_type":"code","source":"xFeature = combinedTrainDF.drop(\"sii\", axis=1)\nyLabel = combinedTrainDF[\"sii\"]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.122448Z","iopub.execute_input":"2024-12-05T04:27:11.122919Z","iopub.status.idle":"2024-12-05T04:27:11.137103Z","shell.execute_reply.started":"2024-12-05T04:27:11.122856Z","shell.execute_reply":"2024-12-05T04:27:11.135830Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"xFeature.shape  , yLabel.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.138600Z","iopub.execute_input":"2024-12-05T04:27:11.138948Z","iopub.status.idle":"2024-12-05T04:27:11.146663Z","shell.execute_reply.started":"2024-12-05T04:27:11.138914Z","shell.execute_reply":"2024-12-05T04:27:11.145482Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Handling imbalance with SMOTE\nif CFG.overSampling:\n    smote = SMOTE()\n    X_resampled, y_resampled = smote.fit_resample(xFeature, yLabel)\n    print(len(X_resampled))\n    xFeature = X_resampled\n    yLabel = y_resampled\n\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.148745Z","iopub.execute_input":"2024-12-05T04:27:11.149098Z","iopub.status.idle":"2024-12-05T04:27:11.267393Z","shell.execute_reply.started":"2024-12-05T04:27:11.149066Z","shell.execute_reply":"2024-12-05T04:27:11.266178Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.utils import compute_sample_weight, class_weight","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.268857Z","iopub.execute_input":"2024-12-05T04:27:11.269276Z","iopub.status.idle":"2024-12-05T04:27:11.275066Z","shell.execute_reply.started":"2024-12-05T04:27:11.269240Z","shell.execute_reply":"2024-12-05T04:27:11.273855Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"xFeature.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.276635Z","iopub.execute_input":"2024-12-05T04:27:11.277124Z","iopub.status.idle":"2024-12-05T04:27:11.291233Z","shell.execute_reply.started":"2024-12-05T04:27:11.277073Z","shell.execute_reply":"2024-12-05T04:27:11.289887Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"xFeature","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.292826Z","iopub.execute_input":"2024-12-05T04:27:11.293330Z","iopub.status.idle":"2024-12-05T04:27:11.437465Z","shell.execute_reply.started":"2024-12-05T04:27:11.293279Z","shell.execute_reply":"2024-12-05T04:27:11.436321Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# scale \nif CFG.normalize:\n    scale = StandardScaler()\n    scaledXFeature = scale.fit_transform(xFeature)\n    xFeature = scaledXFeature\n    scaledTestDF    = scale.fit_transform(combinedTestDF)\n    combinedTestDF = scaledTestDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.438858Z","iopub.execute_input":"2024-12-05T04:27:11.439217Z","iopub.status.idle":"2024-12-05T04:27:11.444964Z","shell.execute_reply.started":"2024-12-05T04:27:11.439184Z","shell.execute_reply":"2024-12-05T04:27:11.443468Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nxFeature","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.446353Z","iopub.execute_input":"2024-12-05T04:27:11.446747Z","iopub.status.idle":"2024-12-05T04:27:11.594031Z","shell.execute_reply.started":"2024-12-05T04:27:11.446707Z","shell.execute_reply":"2024-12-05T04:27:11.592415Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\ncombinedTestDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.595378Z","iopub.execute_input":"2024-12-05T04:27:11.595769Z","iopub.status.idle":"2024-12-05T04:27:11.813914Z","shell.execute_reply.started":"2024-12-05T04:27:11.595724Z","shell.execute_reply":"2024-12-05T04:27:11.812796Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.crossValidate:\n    straKFold = StratifiedKFold(n_splits=CFG.nFold, random_state=42, shuffle=True)\nelse:\n    X_train, X_test, y_train, y_test = train_test_split(xFeature, yLabel, test_size=CFG.test, \n                                                    random_state=42, stratify=yLabel)\n    print(X_train.shape)\n    print(X_test.shape)  \n    print(y_train.shape)\n    print(y_test.shape)\n    # Compute sample_weight using compute_sample_weight\n    sampleWeight = compute_sample_weight('balanced', y_train)\n    classes_weights = class_weight.compute_sample_weight(\n        class_weight='balanced',\n        y=y_train)\n    print(classes_weights)\n    print(len(sampleWeight))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.815468Z","iopub.execute_input":"2024-12-05T04:27:11.815956Z","iopub.status.idle":"2024-12-05T04:27:11.825026Z","shell.execute_reply.started":"2024-12-05T04:27:11.815895Z","shell.execute_reply":"2024-12-05T04:27:11.823491Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# inital Model","metadata":{}},{"cell_type":"code","source":"# XGBoost parameters\nif device.type == \"cuda\":\n    XGBparams = {\n    'learning_rate': CFG.xgbLR, #0.05 #0.08, #0.01,#0.05,\n    'max_depth': CFG.xgbDepth,  #5, #7, #6, low value avoid overfit\n    'n_estimators':  CFG.xgbEstimate, # large number lead to overfit, small number lead to underfit\n    'subsample': 0.8,\n    'colsample_bytree': CFG.colsample_bytree, # 0.5, #0.8,\n    'reg_alpha': 1,\n    'reg_lambda': 5,\n    'random_state': 42,\n    'tree_method': 'hist',\n    'device': 'cuda',\n    'verbosity': 0,\n    }\n   \n\n\nelse: # cpu base\n     XGBparams = {\n    'learning_rate': CFG.xgbLR, # 0.05 #0.08,#0.01, #0.05,\n    'max_depth': CFG.xgbDepth, #7, #6, low value avoid overfit\n    'n_estimators': CFG.xgbEstimate, #300, #200,  # large number lead to overfit, small number lead to underfit\n    'subsample': 0.8,\n    'colsample_bytree': CFG.colsample_bytree, #0.5, #0.8,\n    'reg_alpha': 1,\n    'reg_lambda': 5,\n    'random_state': 42,\n    'tree_method': 'hist',\n    'device': 'cpu',\n    'verbosity': 0,\n    }\n\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.826722Z","iopub.execute_input":"2024-12-05T04:27:11.827223Z","iopub.status.idle":"2024-12-05T04:27:11.843864Z","shell.execute_reply.started":"2024-12-05T04:27:11.827171Z","shell.execute_reply":"2024-12-05T04:27:11.842550Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# xgb = XGBRFRegressor(**XGBparams ,  verbose=-1)\nif CFG.crossValidate:\n    xgb = XGBClassifier(**XGBparams ,  verbose=0, objective='multi:softmax', \n                    num_class=3, eval_metric=[\"merror\",\"mlogloss\"], early_stopping_rounds= CFG.xgbEarlyStop)\n    tabNet = TabNetClassifier(\n             n_d=CFG.n_d, n_a=CFG.n_a, n_steps=CFG.n_steps,\n                        optimizer_params=dict(lr=CFG.learningRate),\n                        optimizer_fn=torch.optim.Adam,\n                        scheduler_params={\"step_size\":10, \n                                         \"gamma\":0.9},\n                        scheduler_fn=torch.optim.lr_scheduler.StepLR,\n                        mask_type= CFG.mask_type, #'entmax' # \"sparsemax\"\n                        verbose= False,\n                        \n        )\n    \n    lgbm= LGBMClassifier(\n            boosting_type=\"gbdt\",\n            device= \"cuda\" if torch.cuda.is_available() else \"cpu\",\n            gpu_platform_id= 0,\n            gpu_device_id= 0,\n#             n_jobs= -1, # CPU\n            random_state=1,\n            learning_rate= 0.001,#0.04125, #learrning rate\n            subsample=0.9,\n            num_leaves =50,\n            reg_alpha=0.05,\n            reg_lambda=0.05,\n            n_estimators=CFG.lgbmEstimators,\n            max_depth= 5,\n            verbose =-1,\n            class_weight='balanced',\n            verbosity= -1,\n            )\n    \n    catBoost = CatBoostClassifier(\n                thread_count=-1,\n#                 task_type=\"GPU\", not support gpu\n#                 devices='0:1',\n                random_state=42,\n                # loss_function=\"CrossEntropy\",\n                loss_function='MultiClass', \n                verbose=False,\n                learning_rate= 0.001,#0.04125,#0.04125, #learrning rate\n                n_estimators=200,\n                max_depth= 5,\n               )\n\n        \nelse:\n    xgb = XGBClassifier(**XGBparams ,  verbose=0, objective='multi:softmax', \n                    num_class=3, sample_weight=sampleWeight, eval_metric=[\"merror\",\"mlogloss\"],\n                        early_stopping_rounds= CFG.xgbEarlyStop)\n\n    tabNet = TabNetClassifier(\n             n_d=CFG.n_d, n_a=CFG.n_a, n_steps=CFG.n_steps,\n                        optimizer_params=dict(lr=CFG.learningRate),\n                        optimizer_fn=torch.optim.Adam,\n                        scheduler_params={\"step_size\":10, \n                                         \"gamma\":0.9},\n                        scheduler_fn=torch.optim.lr_scheduler.StepLR,\n                        mask_type= CFG.mask_type, #'entmax' # \"sparsemax\"\n                        \n        )\n    lgbm= LGBMClassifier(\n            boosting_type=\"gbdt\",\n            device= \"cuda\" if torch.cuda.is_available() else \"cpu\",\n            gpu_platform_id= 0,\n            gpu_device_id= 0,\n#             n_jobs= -1, # CPU\n            random_state=1,\n            learning_rate= 0.001,#0.04125, #learrning rate\n            subsample=0.9,\n            num_leaves =50,\n            reg_alpha=0.05,\n            reg_lambda=0.05,\n            n_estimators=CFG.lgbmEstimators,\n            max_depth= 5,\n            class_weight='balanced',\n            verbosity= -1)\n    catBoost = CatBoostClassifier(\n                thread_count=-1,\n#                 task_type=\"GPU\", not support gpu\n#                 devices='0:1',\n                random_state=42,\n                # loss_function=\"CrossEntropy\",\n                loss_function='MultiClass', \n                verbose=False,\n                learning_rate= 0.001,#0.04125,#0.04125, #learrning rate\n                n_estimators=200,\n                max_depth= 5,\n               )","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.863558Z","iopub.execute_input":"2024-12-05T04:27:11.863931Z","iopub.status.idle":"2024-12-05T04:27:11.878633Z","shell.execute_reply.started":"2024-12-05T04:27:11.863898Z","shell.execute_reply":"2024-12-05T04:27:11.877579Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"xgb.get_params()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.880117Z","iopub.execute_input":"2024-12-05T04:27:11.881115Z","iopub.status.idle":"2024-12-05T04:27:11.900106Z","shell.execute_reply.started":"2024-12-05T04:27:11.881062Z","shell.execute_reply":"2024-12-05T04:27:11.898847Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lgbm","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.901463Z","iopub.execute_input":"2024-12-05T04:27:11.901861Z","iopub.status.idle":"2024-12-05T04:27:11.918029Z","shell.execute_reply.started":"2024-12-05T04:27:11.901825Z","shell.execute_reply":"2024-12-05T04:27:11.916789Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tabNet","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.919574Z","iopub.execute_input":"2024-12-05T04:27:11.920011Z","iopub.status.idle":"2024-12-05T04:27:11.931948Z","shell.execute_reply.started":"2024-12-05T04:27:11.919974Z","shell.execute_reply":"2024-12-05T04:27:11.930878Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"catBoost.get_params()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.933431Z","iopub.execute_input":"2024-12-05T04:27:11.934182Z","iopub.status.idle":"2024-12-05T04:27:11.944185Z","shell.execute_reply.started":"2024-12-05T04:27:11.934131Z","shell.execute_reply":"2024-12-05T04:27:11.942868Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plotConfustionMatrix(actualVal, predictVal, modelName, trainORVal=\"Training\"):\n    CM = confusion_matrix(actualVal, predictVal)\n    classifyReport =  classification_report(actualVal, predictVal)\n    classifyReportDF =  classification_report(actualVal, predictVal, output_dict=True)\n    print(f\"\\n\\rClassification Report For {modelName} {trainORVal}:\\n\\r\", classifyReport)\n    cmd =ConfusionMatrixDisplay(CM)\n    cmd.plot()\n    plt.title(f\"Confusion Matrix for {modelName} {trainORVal}\")\n    plt.show()\n    return classifyReportDF","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.945626Z","iopub.execute_input":"2024-12-05T04:27:11.946084Z","iopub.status.idle":"2024-12-05T04:27:11.956085Z","shell.execute_reply.started":"2024-12-05T04:27:11.946045Z","shell.execute_reply":"2024-12-05T04:27:11.955004Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plotKappa(trainKappa, valKappa, modelName):\n    kFoldList = [item for item in range(1, len(trainKappa) + 1)]\n    plt.figure(figsize=(6, 4))\n    plt.plot(kFoldList, trainKappa, label='Training  Kappa')\n    plt.plot(kFoldList, valKappa, label='Validation Kappa')\n    plt.title(f'{modelName} Kappa (Training/Validation)')\n    plt.xlabel('K-Fold')\n    plt.ylabel('Kappa')\n    plt.legend()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.957664Z","iopub.execute_input":"2024-12-05T04:27:11.958147Z","iopub.status.idle":"2024-12-05T04:27:11.968820Z","shell.execute_reply.started":"2024-12-05T04:27:11.958097Z","shell.execute_reply":"2024-12-05T04:27:11.967855Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"modelHistory= {} # record all trained Model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.970203Z","iopub.execute_input":"2024-12-05T04:27:11.970585Z","iopub.status.idle":"2024-12-05T04:27:11.984338Z","shell.execute_reply.started":"2024-12-05T04:27:11.970538Z","shell.execute_reply":"2024-12-05T04:27:11.983084Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# modelList =[(\"XGBoost\", xgb ), (\"LightBGM\", lgbm), (\"TabNet\", tabNet), (\"CatBoost\", catBoost) ]\n# modelList","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.985810Z","iopub.execute_input":"2024-12-05T04:27:11.986227Z","iopub.status.idle":"2024-12-05T04:27:11.995270Z","shell.execute_reply.started":"2024-12-05T04:27:11.986189Z","shell.execute_reply":"2024-12-05T04:27:11.994267Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# for modelName, model  in modelList:\n#     print(\"Name : \", modelName)\n#     print(\"Model : \",model)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:11.996681Z","iopub.execute_input":"2024-12-05T04:27:11.997153Z","iopub.status.idle":"2024-12-05T04:27:12.006256Z","shell.execute_reply.started":"2024-12-05T04:27:11.997104Z","shell.execute_reply":"2024-12-05T04:27:12.005088Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# training Model\n## Predict Train/Validation for Evalution Training performance","metadata":{}},{"cell_type":"code","source":"if CFG.USE_ENSEMBLE:\n    modelWeight = [0.25, 0.3, 0.25, 0.2]        ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:12.007979Z","iopub.execute_input":"2024-12-05T04:27:12.008467Z","iopub.status.idle":"2024-12-05T04:27:12.019179Z","shell.execute_reply.started":"2024-12-05T04:27:12.008417Z","shell.execute_reply":"2024-12-05T04:27:12.017910Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def ensembleTrainValFunc():\n    \"\"\"\n    for Ensemble learning with multiple model \n    \"\"\"\n    #history result collection\n    xgbtrainKappaHist =[]\n    lgbmtrainKappaHist =[]\n    tabnettrainKappaHist =[]\n    catBoosttrainKappaHist =[]\n    \n    xgbvalKappaHist = []\n    lgbmvalKappaHist = []\n    tabnetvalKappaHist = []\n    catBoostvalKappaHist = []\n    \n    xgbtrainClassReportHist =[]\n    lgbmtrainClassReportHist =[]\n    tabnettrainClassReportHist =[]\n    catBoosttrainClassReportHist =[]\n    \n    xgbvalClassReportHist =[]\n    lgbmvalClassReportHist =[]\n    tabnetvalClassReportHist =[]\n    catBoostvalClassReportHist =[]\n    \n    if CFG.crossValidate:\n        for i, (trainIdx, valIdx) in tqdm(enumerate(straKFold.split(xFeature, yLabel))):\n            print(f\"K-Fold: {i} \") \n            if CFG.normalize:\n                X_train = xFeature[trainIdx, :] # filter/select train idx for X_Train\n                X_test = xFeature[valIdx, :] # filter/select validation idx for X_Train\n                y_train = yLabel[trainIdx]\n                y_test  = yLabel[valIdx]\n            else:\n                X_train = xFeature.iloc[trainIdx, :] # filter/select train idx for X_Train\n                X_test = xFeature.iloc[valIdx, :] # filter/select validation idx for X_Train\n                y_train = yLabel.iloc[trainIdx]\n                y_test  = yLabel.iloc[valIdx]\n            \n            # train all model\n            print(\"Start Training XGBoost\")\n            evalSet = [(X_train, y_train), (X_test, y_test)]\n            xgb.fit(X_train, y_train, eval_set= evalSet)\n            print(\"Start Training LightGBM\")\n            lgbm.fit(X_train, y_train, eval_set= evalSet)\n            print(\"Start Training TabNet\")\n            tabNet.fit(X_train.values, \n                        y_train,\n                        patience= CFG.patience, max_epochs=CFG.maxEpochs,\n                        eval_name=[\"Train\", \"Val\"],\n                        eval_set=[(X_train.values, y_train), (X_test.values, y_test)],\n                        # eval_metric=['accuracy', \"balanced_accuracy\"],\n                        eval_metric = [\"logloss\"],\n                        weights=1,\n                        drop_last=False)\n            print(\"Start Training CatBoost\")\n            catBoost.fit(X_train, y_train, eval_set= evalSet, verbose=False)\n            \n            \n            # predict train data \n            xgbYTrainPredict = xgb.predict(X_train)\n            lgbmYTrainPredict = lgbm.predict(X_train)\n            tabNetYTrainPredict = tabNet.predict(X_train.values)\n            catBoostYTrainPredict = catBoost.predict(X_train)\n            \n            # predict validation data\n            xgbYValPredict = xgb.predict(X_test)\n            lgbmYValPredict = lgbm.predict(X_test)\n            tabNetYValPredict = tabNet.predict(X_test.values)\n            catBoostYValPredict = catBoost.predict(X_test)\n\n            # plot XGBoost logloss\n            results= xgb.evals_result()\n            epochs = len(results['validation_0']['merror'])\n            x_axis = range(0, epochs)\n            # plot log loss\n            fig, ax = plt.subplots()\n            ax.plot(x_axis, results['validation_0']['mlogloss'], label='Train')\n            ax.plot(x_axis, results['validation_1']['mlogloss'], label='Val')\n            ax.legend()\n            plt.ylabel('Log Loss')\n            plt.title('XGBoost Log Loss')\n            plt.show()\n\n            # plot Tab\n            tabList = [item for item in range(1, len(tabNet.history[\"Train_logloss\"]) + 1)]\n            plt.figure(figsize=(6, 4))\n            plt.plot(tabList, tabNet.history[\"Train_logloss\"], label=\"Train\")\n            plt.plot(tabList, tabNet.history[\"Val_logloss\"], label=\"Val\")\n            plt.title(\"TabNet Log Loss\")\n            plt.legend()\n            plt.show()\n\n            # plot catBoost\n            catResult = catBoost.get_evals_result()\n            plt.figure(figsize=(6, 4))\n            plt.plot(catResult[\"validation_0\"][\"MultiClass\"], label=\"Train\")\n            plt.plot(catResult[\"validation_1\"][\"MultiClass\"], label=\"Val\")\n            plt.title(\"CatBoost Log Loss\")\n            plt.legend()\n            plt.show()\n  \n\n            # plot LightGBM\n            plt.figure(figsize=(6, 4))\n            plt.plot(lgbm.evals_result_[\"training\"][\"multi_logloss\"], label=\"Train\")\n            plt.plot(lgbm.evals_result_[\"valid_1\"][\"multi_logloss\"], label=\"Val\")\n            plt.title(\"LightGBM Log Loss\")\n            plt.legend()\n            plt.show()\n            \n            # Training Classification Report \n            trainClassifyReport = plotConfustionMatrix(y_train, xgbYTrainPredict, \n                                                       \"XGBoost\", trainORVal=\"Training\")\n            xgbtrainClassReportHist.append(trainClassifyReport)\n            trainClassifyReport = plotConfustionMatrix(y_train, lgbmYTrainPredict, \n                                                       \"LightGBM\", trainORVal=\"Training\")\n            lgbmtrainClassReportHist.append(trainClassifyReport)\n            trainClassifyReport = plotConfustionMatrix(y_train, tabNetYTrainPredict, \n                                                       \"TabNet\", trainORVal=\"Training\")\n            tabnettrainClassReportHist.append(trainClassifyReport)\n            trainClassifyReport = plotConfustionMatrix(y_train, catBoostYTrainPredict, \n                                                       \"CatBoost\", trainORVal=\"Training\")\n            catBoosttrainClassReportHist.append(trainClassifyReport)\n            \n            \n            # Validation Classification Report\n            valClassifyReport = plotConfustionMatrix(y_test, xgbYValPredict, \n                                                       \"XGBoost\", trainORVal=\"Validation\")\n            xgbvalClassReportHist.append(valClassifyReport)\n            valClassifyReport = plotConfustionMatrix(y_test, lgbmYValPredict, \n                                                       \"LightGBM\", trainORVal=\"Validation\")\n            lgbmvalClassReportHist.append(valClassifyReport)\n        \n            valClassifyReport = plotConfustionMatrix(y_test, tabNetYValPredict, \n                                                       \"TabNet\", trainORVal=\"Validation\")\n            tabnetvalClassReportHist.append(valClassifyReport)\n            valClassifyReport = plotConfustionMatrix(y_test, catBoostYValPredict, \n                                                       \"CatBoost\", trainORVal=\"Validation\")\n            catBoostvalClassReportHist.append(valClassifyReport)\n\n            # Train Kappa Score\n            trainKappa = cohen_kappa_score(y_train, xgbYTrainPredict)\n            xgbtrainKappaHist.append(trainKappa)            \n            print(f\"XGBoost Training cohen Kappa score: {trainKappa}\")\n            trainKappa = cohen_kappa_score(y_train, lgbmYTrainPredict)\n            lgbmtrainKappaHist.append(trainKappa)            \n            print(f\"LightGBM Training cohen Kappa score: {trainKappa}\")\n            trainKappa = cohen_kappa_score(y_train, tabNetYTrainPredict)\n            tabnettrainKappaHist.append(trainKappa)            \n            print(f\"TabNet Training cohen Kappa score: {trainKappa}\")\n            trainKappa = cohen_kappa_score(y_train, catBoostYTrainPredict)\n            catBoosttrainKappaHist.append(trainKappa)            \n            print(f\"CatBoost Training cohen Kappa score: {trainKappa}\")\n\n            # Validation Kappa Score\n            valKappa = cohen_kappa_score(y_test, xgbYValPredict)\n            xgbvalKappaHist.append(valKappa)\n            print(f\"XGBoost Val cohen Kappa score: {valKappa}\")\n            valKappa = cohen_kappa_score(y_test, lgbmYValPredict)\n            lgbmvalKappaHist.append(valKappa)\n            print(f\"LightGBM Val cohen Kappa score: {valKappa}\")\n            valKappa = cohen_kappa_score(y_test, tabNetYValPredict)\n            tabnetvalKappaHist.append(valKappa)\n            print(f\"TabNet Val cohen Kappa score: {valKappa}\")\n            valKappa = cohen_kappa_score(y_test, catBoostYValPredict)\n            catBoostvalKappaHist.append(valKappa)\n            print(f\"CatBoost Val cohen Kappa score: {valKappa}\")\n            \n        \n            \n        #store is Model History\n        modelHistory[\"XGBoost\"] = {\n            \"k_Fold\": CFG.nFold,\n            \"train_classify_report\" : xgbtrainClassReportHist,\n            \"val_calssify_report\": xgbvalClassReportHist,\n            \"train_kappa_score\" : xgbtrainKappaHist,\n            \"val_kappa_score\": xgbvalKappaHist\n        }\n        modelHistory[\"LightGBM\"] = {\n            \"k_Fold\": CFG.nFold,\n            \"train_classify_report\" : lgbmtrainClassReportHist,\n            \"val_calssify_report\": lgbmvalClassReportHist,\n            \"train_kappa_score\" : lgbmtrainKappaHist,\n            \"val_kappa_score\": lgbmvalKappaHist\n        }\n        modelHistory[\"TabNet\"] = {\n            \"k_Fold\": CFG.nFold,\n            \"train_classify_report\" : tabnettrainClassReportHist,\n            \"val_calssify_report\": tabnetvalClassReportHist,\n            \"train_kappa_score\" : tabnettrainKappaHist,\n            \"val_kappa_score\": tabnetvalKappaHist\n        }\n        modelHistory[\"CatBoost\"] = {\n            \"k_Fold\": CFG.nFold,\n            \"train_classify_report\" : catBoosttrainClassReportHist,\n            \"val_calssify_report\": catBoostvalClassReportHist,\n            \"train_kappa_score\" : catBoosttrainKappaHist,\n            \"val_kappa_score\": catBoostvalKappaHist\n        }\n            \n            \n#             \n    else:\n            # Train model\n            print(\"Start Training XGBoost\")\n            evalSet = [(X_train, y_train), (X_test, y_test)]\n            xgb.fit(X_train, y_train, eval_set= evalSet)\n            print(\"Start Training LightGBM\")\n            lgbm.fit(X_train, y_train, eval_set= evalSet)\n            print(\"Start Training TabNet\")\n            tabNet.fit(X_train.values, \n                        y_train,\n                        patience= CFG.patience, max_epochs=CFG.maxEpochs,\n                        eval_set=[(X_train.values, y_train), (X_test.values, y_test)],\n                        # eval_metric=['accuracy', \"balanced_accuracy\"],\n                        eval_metric = [\"logloss\"],\n                        weights=1,\n                        drop_last=False)\n            print(\"Start Training CatBoost\")\n            catBoost.fit(X_train, y_train, eval_set= evalSet, verbose=False)\n            \n            \n            # predict train data \n            xgbYTrainPredict = xgb.predict(X_train)\n            lgbmYTrainPredict = lgbm.predict(X_train)\n            tabNetYTrainPredict = tabNet.predict(X_train.values)\n            catBoostYTrainPredict = catBoost.predict(X_train)\n            \n            # predict validation data\n            xgbYValPredict = xgb.predict(X_test)\n            lgbmYValPredict = lgbm.predict(X_test)\n            tabNetYValPredict = tabNet.predict(X_test.values)\n            catBoostYValPredict = catBoost.predict(X_test)\n\n            # plot XGBoost logloss\n            results= xgb.evals_result()\n            epochs = len(results['validation_0']['merror'])\n            x_axis = range(0, epochs)\n            # plot log loss\n            fig, ax = plt.subplots()\n            ax.plot(x_axis, results['validation_0']['mlogloss'], label='Train')\n            ax.plot(x_axis, results['validation_1']['mlogloss'], label='Val')\n            ax.legend()\n            plt.ylabel('Log Loss')\n            plt.title('XGBoost Log Loss')\n            plt.show()\n\n            # Training Classification Report \n            trainClassifyReport = plotConfustionMatrix(y_train, xgbYTrainPredict, \n                                                       \"XGBoost\", trainORVal=\"Training\")\n            xgbtrainClassReportHist.append(trainClassifyReport)\n            trainClassifyReport = plotConfustionMatrix(y_train, lgbmYTrainPredict, \n                                                       \"LightGBM\", trainORVal=\"Training\")\n            lgbmtrainClassReportHist.append(trainClassifyReport)\n            trainClassifyReport = plotConfustionMatrix(y_train, tabNetYTrainPredict, \n                                                       \"TabNet\", trainORVal=\"Training\")\n            tabnettrainClassReportHist.append(trainClassifyReport)\n            trainClassifyReport = plotConfustionMatrix(y_train, catBoostYTrainPredict, \n                                                       \"CatBoost\", trainORVal=\"Training\")\n            catBoosttrainClassReportHist.append(trainClassifyReport)\n            \n            \n            # Validation Classification Report\n            valClassifyReport = plotConfustionMatrix(y_test, xgbYValPredict, \n                                                       \"XGBoost\", trainORVal=\"Validation\")\n            xgbvalClassReportHist.append(valClassifyReport)\n            valClassifyReport = plotConfustionMatrix(y_test, lgbmYValPredict, \n                                                       \"LightGBM\", trainORVal=\"Validation\")\n            lgbmvalClassReportHist.append(valClassifyReport)\n        \n            valClassifyReport = plotConfustionMatrix(y_test, tabNetYValPredict, \n                                                       \"TabNet\", trainORVal=\"Validation\")\n            tabnetvalClassReportHist.append(valClassifyReport)\n            valClassifyReport = plotConfustionMatrix(y_test, catBoostYValPredict, \n                                                       \"CatBoost\", trainORVal=\"Validation\")\n            catBoostvalClassReportHist.append(valClassifyReport)\n\n            # Train Kappa Score\n            trainKappa = cohen_kappa_score(y_train, xgbYTrainPredict)\n            xgbtrainKappaHist.append(trainKappa)            \n            print(f\"XGBoost Training cohen Kappa score: {trainKappa}\")\n            trainKappa = cohen_kappa_score(y_train, lgbmYTrainPredict)\n            lgbmtrainKappaHist.append(trainKappa)            \n            print(f\"LightGBM Training cohen Kappa score: {trainKappa}\")\n            trainKappa = cohen_kappa_score(y_train, tabNetYTrainPredict)\n            tabnettrainKappaHist.append(trainKappa)            \n            print(f\"TabNet Training cohen Kappa score: {trainKappa}\")\n            trainKappa = cohen_kappa_score(y_train, catBoostYTrainPredict)\n            catBoosttrainKappaHist.append(trainKappa)            \n            print(f\"CatBoost Training cohen Kappa score: {trainKappa}\")\n\n            # Validation Kappa Score\n            valKappa = cohen_kappa_score(y_test, xgbYValPredict)\n            xgbvalKappaHist.append(valKappa)\n            print(f\"XGBoost Val cohen Kappa score: {valKappa}\")\n            valKappa = cohen_kappa_score(y_test, lgbmYValPredict)\n            lgbmvalKappaHist.append(valKappa)\n            print(f\"LightGBM Val cohen Kappa score: {valKappa}\")\n            valKappa = cohen_kappa_score(y_test, tabNetYValPredict)\n            tabnetvalKappaHist.append(valKappa)\n            print(f\"TabNet Val cohen Kappa score: {valKappa}\")\n            valKappa = cohen_kappa_score(y_test, catBoostYValPredict)\n            catBoostvalKappaHist.append(valKappa)\n            print(f\"CatBoost Val cohen Kappa score: {valKappa}\")\n            #store is Model History\n            modelHistory[\"XGBoost\"] = {\n                \"k_Fold\": CFG.nFold,\n                \"train_classify_report\" : xgbtrainClassReportHist,\n                \"val_calssify_report\": xgbvalClassReportHist,\n                \"train_kappa_score\" : xgbtrainKappaHist,\n                \"val_kappa_score\": xgbvalKappaHist\n            }\n            modelHistory[\"LightGBM\"] = {\n                \"k_Fold\": CFG.nFold,\n                \"train_classify_report\" : lgbmtrainClassReportHist,\n                \"val_calssify_report\": lgbmvalClassReportHist,\n                \"train_kappa_score\" : lgbmtrainKappaHist,\n                \"val_kappa_score\": lgbmvalKappaHist\n            }\n            modelHistory[\"TabNet\"] = {\n                \"k_Fold\": CFG.nFold,\n                \"train_classify_report\" : tabnettrainClassReportHist,\n                \"val_calssify_report\": tabnetvalClassReportHist,\n                \"train_kappa_score\" : tabnettrainKappaHist,\n                \"val_kappa_score\": tabnetvalKappaHist\n            }\n            modelHistory[\"CatBoost\"] = {\n                \"k_Fold\": CFG.nFold,\n                \"train_classify_report\" : catBoosttrainClassReportHist,\n                \"val_calssify_report\": catBoostvalClassReportHist,\n                \"train_kappa_score\" : catBoosttrainKappaHist,\n                \"val_kappa_score\": catBoostvalKappaHist\n            }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:12.021036Z","iopub.execute_input":"2024-12-05T04:27:12.021479Z","iopub.status.idle":"2024-12-05T04:27:12.066074Z","shell.execute_reply.started":"2024-12-05T04:27:12.021440Z","shell.execute_reply":"2024-12-05T04:27:12.064564Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# train single Model \ndef trainValFunc(model , modelName):\n    trainKappaHist =[]\n    valKappaHist = []\n    trainClassReportHist =[]\n    valClassReportHist =[]\n    \n    if CFG.crossValidate:\n        for i, (trainIdx, valIdx) in tqdm(enumerate(straKFold.split(xFeature, yLabel))):\n            print(f\"K-Fold: {i} \")\n#             print(f\"Train Idx : {trainIdx}\")\n#             print(f\"Validate Idx: {valIdx}\")\n            # Set extract X_Train , X_test , y_train, y_test new split dataset \n            if CFG.normalize:\n                X_train = xFeature[trainIdx, :] # filter/select train idx for X_Train\n                X_test = xFeature[valIdx, :] # filter/select validation idx for X_Train\n                y_train = yLabel[trainIdx]\n                y_test  = yLabel[valIdx]\n            else:\n                X_train = xFeature.iloc[trainIdx, :] # filter/select train idx for X_Train\n                X_test = xFeature.iloc[valIdx, :] # filter/select validation idx for X_Train\n                y_train = yLabel.iloc[trainIdx]\n                y_test  = yLabel.iloc[valIdx]\n            # train model\n            evalSet = [(X_train, y_train), (X_test, y_test)]\n            model.fit(X_train, y_train, eval_set= evalSet, \n                      eval_metric=[\"merror\",\"mlogloss\"], early_stopping_rounds= CFG.xgbEarlyStop,  verbose=False)\n\n            results= model.evals_result()\n            epochs = len(results['validation_0']['merror'])\n            x_axis = range(0, epochs)\n            # plot log loss\n            fig, ax = plt.subplots()\n            ax.plot(x_axis, results['validation_0']['mlogloss'], label='Train')\n            ax.plot(x_axis, results['validation_1']['mlogloss'], label='Val')\n            ax.legend()\n            plt.ylabel('Log Loss')\n            plt.title('XGBoost Log Loss')\n            plt.show()\n            # predict train dataset \n            yTrainPredict = model.predict(X_train)\n#             print(yTrainPredict)\n            yValPredict = model.predict(X_test)\n#             print(yValPredict)\n            trainClassifyReport = plotConfustionMatrix(y_train, yTrainPredict, \n                                                       modelName, trainORVal=\"Training\")\n            trainClassReportHist.append(trainClassifyReport)\n            \n            valClassifyReport = plotConfustionMatrix(y_test, yValPredict, \n                                                       modelName, trainORVal=\"Validation\")\n            valClassReportHist.append(valClassifyReport)\n#             print(trainClassifyReport)\n#             print(valClassifyReport)\n            trainKappa = cohen_kappa_score(y_train, yTrainPredict)\n            trainKappaHist.append(trainKappa)\n            print(f\"Training cohen Kappa score: {trainKappa}\")\n            valKappa = cohen_kappa_score(y_test, yValPredict)\n            valKappaHist.append(valKappa)\n            print(f\"Val cohen Kappa score: {valKappa}\")\n            \n        \n            \n        #store is Model History\n        modelHistory[modelName] = {\n            \"k_Fold\": CFG.nFold,\n            \"train_classify_report\" : trainClassReportHist,\n            \"val_calssify_report\": valClassReportHist,\n            \"train_kappa_score\" : trainKappaHist,\n            \"val_kappa_score\": valKappaHist\n        }\n            \n            \n#             \n    else:\n        # Train model\n        evalSet = [(X_train, y_train), (X_test, y_test)]\n        model.fit(X_train, y_train, eval_set= evalSet,  \n                  early_stopping_rounds= CFG.xgbEarlyStop,  eval_metric=[\"merror\",\"mlogloss\"], verbose=False)\n        # Train dateset prediction \n        yTrainPredict = model.predict(X_train)\n#         print(yTrainPredict)\n        yValPredict = model.predict(X_test)\n#         print(yValPredict)\n        trainClassifyReport = plotConfustionMatrix(y_train, yTrainPredict, \n                                                       modelName, trainORVal=\"Training\")\n        trainClassReportHist.append(trainClassifyReport)\n        valClassifyReport = plotConfustionMatrix(y_test, yValPredict, \n                                                       modelName, trainORVal=\"Validation\")\n        valClassReportHist.append(valClassifyReport)\n#         print(trainClassifyReport)\n#         print(valClassifyReport)\n        trainKappa = cohen_kappa_score(y_train, yTrainPredict)\n        trainKappaHist.append(trainKappa)\n        print(f\"Training cohen Kappa score: {trainKappa}\")\n        valKappa = cohen_kappa_score(y_test, yValPredict)\n        print(f\"Val cohen Kappa score: {valKappa}\")\n        valKappaHist.append(valKappa)\n        \n        #store is Model History\n        modelHistory[modelName] = {\n            \"train_classify_report\" : trainClassReportHist,\n            \"val_calssify_report\": valClassReportHist,\n            \"train_kappa_score\" : trainKappaHist,\n            \"val_kappa_score\": valKappaHist\n        }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:12.067884Z","iopub.execute_input":"2024-12-05T04:27:12.068373Z","iopub.status.idle":"2024-12-05T04:27:12.086778Z","shell.execute_reply.started":"2024-12-05T04:27:12.068320Z","shell.execute_reply":"2024-12-05T04:27:12.085432Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\nif CFG.USE_ENSEMBLE:\n    ensembleTrainValFunc()\nelse:\n    trainValFunc(xgb, \"XGBoost\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-05T04:27:12.088192Z","iopub.execute_input":"2024-12-05T04:27:12.088583Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# tabNet.history[\"Val_logloss\"]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# if CFG.USE_ENSEMBLE:\n#     # ensembleTrainValFunc()\n#     for model, val in modelHistory.items():\n#         print(f\"Model: {model}\\n\\r{val}\")\n# else:\n#     modelHistory[\"XGBoost\"]","metadata":{"scrolled":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# modelHistory[\"XGBoost\"][\"val_kappa_score\"]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for model in modelHistory.keys():\n    trainKappa = np.mean(modelHistory[model][\"train_kappa_score\"])\n    valKappa = np.mean(modelHistory[model][\"val_kappa_score\"])                     \n    print(f\"Average Training Kappa Score for {model} : {trainKappa}\")\n    print(f\"Average Val Kappa Score for {model} :  {valKappa}\")\n    ","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# print(\"Average Training Kappa Score for XGBoost :\", np.mean(modelHistory[\"XGBoost\"][\"train_kappa_score\"]))\n# print(\"Average Val Kappa Score for XGBoost :\", np.mean(modelHistory[\"XGBoost\"][\"val_kappa_score\"]))\n# print(\"Average Training Kappa Score for XGBoost :\", np.mean(modelHistory[\"XGBoost\"][\"train_kappa_score\"]))\n# print(\"Average Val Kappa Score for XGBoost :\", np.mean(modelHistory[\"XGBoost\"][\"val_kappa_score\"]))\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for model in modelHistory.keys():\n    plotKappa(modelHistory[model][\"train_kappa_score\"], \n          modelHistory[model][\"val_kappa_score\"], model)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# plotKappa(modelHistory[\"XGBoost\"][\"train_kappa_score\"], \n#           modelHistory[\"XGBoost\"][\"val_kappa_score\"],\n#          \"XGBoost\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# lgbm.evals_result_.keys()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# plt.figure(figsize=(6, 4))\n# plt.plot(lgbm.evals_result_[\"training\"][\"multi_logloss\"], label=\"Train\")\n# plt.plot(lgbm.evals_result_[\"valid_1\"][\"multi_logloss\"], label=\"Val\")\n# plt.title(\"LightGBM Log Loss\")\n# plt.legend()\n# plt.show()\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Ensemble Learning Algriothm, use multi-model for improve ","metadata":{}},{"cell_type":"markdown","source":"## Explainable AI \n#### Study and Understand How XGBoost trained model make decision","metadata":{}},{"cell_type":"code","source":"feature_impXGB = pd.Series(xgb.feature_importances_, index=tempCol2).sort_values(ascending=False)\nxgbTop20 = feature_impXGB[:30]\nplt.figure(figsize=(10,6))\nsns.barplot(x=xgbTop20, y=xgbTop20.index)\nplt.xlabel('Feature Importance Score')\nplt.ylabel('Features')\nplt.tight_layout()\nplt.title(\"Top 30 importenace feature for XGBoost\");\nplt.show()\n# xgboost.plot_importance(xgb);","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(16, 16))\nxgboost.plot_tree(xgb,  num_trees=0, rankdir='LR')\nplt.savefig(\"xgb_out.png\",  dpi=600)\nplt.show();","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model Prediction for Submission","metadata":{}},{"cell_type":"code","source":"# \ndef roundoff(arr, thresholds=[0.5, 1.5, 2.5]):\n    return np.where(arr < thresholds[0], 0, \n                np.where(arr < thresholds[1], 1, \n                    np.where(arr < thresholds[2], 2, 3)))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if CFG.USE_ENSEMBLE:\n    xgbYValPredict = xgb.predict(combinedTestDF)\n    lgbmYValPredict = lgbm.predict(combinedTestDF)\n    tabNetYValPredict = tabNet.predict(combinedTestDF.values)\n    catBoostYValPredict = catBoost.predict(combinedTestDF)\n    temp1 = np.add(modelWeight[0] * xgbYValPredict ,modelWeight[1] * lgbmYValPredict)\n    temp2 = np.add(temp1, modelWeight[2] * tabNetYValPredict)\n    submitPredict = np.add(temp2, (modelWeight[3] * np.squeeze(catBoostYValPredict)))\n    submitPredict = roundoff(submitPredict)\nelse: \n     submitPredict = xgb.predict(combinedTestDF)\nsubmitPredict","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"roundedPredict = submitPredict # roundoff(submitPredict)\nroundedPredict","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submit[\"sii\"] = roundedPredict","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submit","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submit.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sub = pd.read_csv(\"submission.csv\")\nsub","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}