{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Table of Contents\n\n#### [1 What is TrueFoundry?]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#1-What-is-TrueFoundry?')\n#### [2 Tabular Playground Series]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#-2-Tabular-Playground-Series')\n#### [3 Preparation]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#-3-Preparation')\n\n*  [3.1 Essential Packages]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#3.1-Essential-Packages')\n\n#### [4 Data Loading and Preprocessing]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#4-Data-Loading-and-Preprocessing')\n\n*  [4.1 Filling missing values]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#4.1-Filling-missing-values')\n*  [4.2 Encoding]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#4.2-Encoding')\n\n#### [5 Models]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#5-Models')\n\n* [5.1 LGBM Classifier]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#5.1-LGBM-Classifier')\n* [5.2 XGBoost Classifier]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#5.2-XGBoost-Classifier')\n* [5.3 Ensemble]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#5.3-Ensemble')\n* [5.4 Submission]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#5.4-Submission')\n\n#### [6 TrueFoundry Platform]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6-TrueFoundry-Platform')\n* [6.1 Projects]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6.1-Projects')\n* [6.2 Runs]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6.2-Runs')\n\n  * [6.2.1 Overview]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6.2.1-Overview')\n  * [6.2.2 Run Metrics]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6.2.2-Run-Metrics')\n  * [6.2.3 Data & Feature Metrics]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6.2.3-Data-&-Feature-Metrics')\n  * [6.2.4 General Artifact]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6.2.4-General-Artifact')\n  \n* [6.3 Models Comparison]('https://www.kaggle.com/code/khotijahs1/ml-experiment-tracking-with-truefoundry-platform/notebook#6.3-Models-Comparison')\n\n\n","metadata":{"_cell_guid":"008626d2-4396-4781-a7dd-951c8a251ae6","_uuid":"81664643-41f5-4e54-a71c-8fb5943b90dc","papermill":{"duration":0.007224,"end_time":"2022-06-25T16:36:55.594482","exception":false,"start_time":"2022-06-25T16:36:55.587258","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## 1 What is [TrueFoundry]('https://app.truefoundry.com/signin')?\n\n\n","metadata":{"_cell_guid":"c51c705e-a837-42a9-9acf-1cd21b245576","_uuid":"bdf746f0-2c33-4189-ba81-439846b67b71","papermill":{"duration":0.005747,"end_time":"2022-06-25T16:36:55.606507","exception":false,"start_time":"2022-06-25T16:36:55.60076","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n[TrueFoundry]('https://app.truefoundry.com/mlfoundry') aims to provide the different components in a Machine learning stack - all wound together in a way that they talk to each other seamlessly and teams don't have to spend time gluing pieces together. While all the pieces are knit tightly together, we also design all the components in a way that they can be seamlessly integrated with other tools in the future. TrueFoundry can be accessed at https://app.truefoundry.com/mlfoundry.\n\nTruefoundry comprises of the following pieces to tie together things seamlessly:\n\n1. ```MlFoundry```: used during model training to log your model artifacts, parameters, data & code so as to be able to collaborate with your team and reproduce Machine Learning Experiments.\n2. ```ServiceFoundry```: single API which containerizes and deploys your model to a managed Kubernetes Cluster. This also generates a Grafana cluster with complete visibility of your Service Health, System Logs, and Kubernetes Workspace.\n3. ```Monitoring```: model input-output monitoring, data drift charts, and root-cause analysis when things break. Coming soon!","metadata":{}},{"cell_type":"markdown","source":"## 2 Tabular Playground Series\n\nThe dataset is used for this competition is synthetic but based on a real dataset (in this case, the actual Titanic data!) and generated using a CTGAN. The statistical properties of this dataset are very similar to the original Titanic dataset, but there's no way to \"cheat\" by using public labels for predictions. ","metadata":{"_cell_guid":"769586a9-de86-43fe-aeb2-da58af17ec0b","_uuid":"b4329fb3-7819-4873-9021-6f1a51d99f66","papermill":{"duration":0.005747,"end_time":"2022-06-25T16:36:55.618363","exception":false,"start_time":"2022-06-25T16:36:55.612616","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## 3 Preparation\n\nPrepare packages and data that will be used in the analysis process and we will use [TrueFoundry]('https://app.truefoundry.com/signin') to track our experiments. \n\n","metadata":{"_cell_guid":"6049d354-f9e9-434f-b49a-978791ea3d26","_uuid":"07ae36c1-10d9-4e56-9d70-b186b1f62c95","papermill":{"duration":0.005678,"end_time":"2022-06-25T16:36:55.630029","exception":false,"start_time":"2022-06-25T16:36:55.624351","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### 3.1 Essential Packages","metadata":{"_cell_guid":"b1cea804-bf4e-4de3-938e-91fac51cc3dc","_uuid":"c3459020-78f7-40ec-8bdf-1a6cde805957","papermill":{"duration":0.005655,"end_time":"2022-06-25T16:36:55.641734","exception":false,"start_time":"2022-06-25T16:36:55.636079","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport random\nimport os\n\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.preprocessing import LabelEncoder, StandardScaler\nfrom sklearn.model_selection import train_test_split, KFold, StratifiedKFold\n\nimport lightgbm as lgb\nimport catboost as ctb\nfrom lightgbm import LGBMClassifier\nfrom xgboost import XGBClassifier \nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.tree import DecisionTreeClassifier, export_graphviz\n\nimport graphviz\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython.display import Image\n\nimport warnings\nwarnings.simplefilter('ignore')\n","metadata":{"execution":{"iopub.status.busy":"2022-06-26T15:36:03.790617Z","iopub.execute_input":"2022-06-26T15:36:03.791144Z","iopub.status.idle":"2022-06-26T15:36:03.798837Z","shell.execute_reply.started":"2022-06-26T15:36:03.791113Z","shell.execute_reply":"2022-06-26T15:36:03.797846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"api_key\")","metadata":{"_cell_guid":"abc272ba-b6da-4d81-b961-76366c4d9250","_uuid":"6ecb520e-e708-498d-ae3d-b429b0dd3485","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.254499,"end_time":"2022-06-25T16:36:57.955278","exception":false,"start_time":"2022-06-25T16:36:57.700779","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T15:36:03.800308Z","iopub.execute_input":"2022-06-26T15:36:03.80079Z","iopub.status.idle":"2022-06-26T15:36:04.114212Z","shell.execute_reply.started":"2022-06-26T15:36:03.800761Z","shell.execute_reply":"2022-06-26T15:36:04.113382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install mlfoundry\nimport mlfoundry as mlf","metadata":{"_cell_guid":"0403d007-ed09-42bc-a6ea-5d49c4dc891f","_kg_hide-output":true,"_uuid":"ba853bbe-da7a-49ba-9194-821dc24bd545","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":34.967201,"end_time":"2022-06-25T16:37:32.929167","exception":false,"start_time":"2022-06-25T16:36:57.961966","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-26T15:36:04.116112Z","iopub.execute_input":"2022-06-26T15:36:04.116882Z","iopub.status.idle":"2022-06-26T15:36:15.153914Z","shell.execute_reply.started":"2022-06-26T15:36:04.116836Z","shell.execute_reply":"2022-06-26T15:36:15.152379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Login to TrueFoundry platform and get the API key. Create New API keys if we don't have it or copy the generated API keys before. API key can be found at https://app.truefoundry.com/settings.","metadata":{"_cell_guid":"d8396a8d-b63a-44e3-a0d2-72bcb81ea7d5","_uuid":"bc31c46b-b191-41c9-bb24-a8af13adf098","papermill":{"duration":0.013038,"end_time":"2022-06-25T16:37:32.955583","exception":false,"start_time":"2022-06-25T16:37:32.942545","status":"completed"},"tags":[]}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/API_Keys.png')","metadata":{"_cell_guid":"07dcdf11-3a3c-44d4-83af-06d69a58c331","_uuid":"2194f4a6-1d53-4185-90a1-0ebdcfe85013","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.031195,"end_time":"2022-06-25T16:37:33.000063","exception":false,"start_time":"2022-06-25T16:37:32.968868","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-26T15:36:15.155671Z","iopub.execute_input":"2022-06-26T15:36:15.156118Z","iopub.status.idle":"2022-06-26T15:36:15.164783Z","shell.execute_reply.started":"2022-06-26T15:36:15.156071Z","shell.execute_reply":"2022-06-26T15:36:15.163853Z"},"_kg_hide-output":false,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\nWe will login using ```mlf.get_client``` and use pass our ```API Keys``` on api_key. We also create a project name.","metadata":{"_cell_guid":"fdc0ac67-3fae-4635-8652-8f39e36e98ae","_uuid":"ffd76cd7-3ef3-49f1-a7d5-2e39d1abd6fa","papermill":{"duration":0.013982,"end_time":"2022-06-25T16:37:33.027778","exception":false,"start_time":"2022-06-25T16:37:33.013796","status":"completed"},"tags":[]}},{"cell_type":"code","source":"client = mlf.get_client(api_key=secret_value_0)\nproject_name = 'synthanic'","metadata":{"_cell_guid":"5315ae7b-9bd5-4c18-a1a8-3655f38ab736","_uuid":"0db1d655-1661-402c-8ca0-a23c559f9476","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":1.557051,"end_time":"2022-06-25T16:37:34.598075","exception":false,"start_time":"2022-06-25T16:37:33.041024","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-26T15:36:15.166256Z","iopub.execute_input":"2022-06-26T15:36:15.166557Z","iopub.status.idle":"2022-06-26T15:36:15.721562Z","shell.execute_reply.started":"2022-06-26T15:36:15.166527Z","shell.execute_reply":"2022-06-26T15:36:15.720668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4 Data Loading and Preprocessing\n\n","metadata":{"_cell_guid":"6e6be2ea-fc96-456c-a250-fab6937fd298","_uuid":"0ab247a1-608f-47c5-9c69-de08bd2f1df8","papermill":{"duration":0.013225,"end_time":"2022-06-25T16:37:34.624774","exception":false,"start_time":"2022-06-25T16:37:34.611549","status":"completed"},"tags":[]}},{"cell_type":"code","source":"TARGET = 'Survived'\n\nN_ESTIMATORS = 2000\nN_SPLITS = 5\nSEED = 2021\nEARLY_STOPPING_ROUNDS = 100\nVERBOSE = 100\n\n\ndef set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    \nset_seed(SEED)","metadata":{"execution":{"iopub.status.busy":"2022-06-26T15:36:15.722517Z","iopub.execute_input":"2022-06-26T15:36:15.722801Z","iopub.status.idle":"2022-06-26T15:36:15.729862Z","shell.execute_reply.started":"2022-06-26T15:36:15.722776Z","shell.execute_reply":"2022-06-26T15:36:15.728595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv('../input/tabular-playground-series-apr-2021/train.csv')\ntest_df = pd.read_csv('../input/tabular-playground-series-apr-2021/test.csv')\nsubmission = pd.read_csv('../input/tabular-playground-series-apr-2021/sample_submission.csv')\ntest_df[TARGET] = pd.read_csv(\"../input/local-tps-apr/pseudo_label.csv\")[TARGET]\n\nall_df = pd.concat([train_df, test_df]).reset_index(drop=True)","metadata":{"_cell_guid":"896ff512-192a-41ba-ab64-47931556ab1f","_uuid":"0d0b4129-eab0-4d0b-b4ce-42c759a10687","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":0.029279,"end_time":"2022-06-25T16:37:34.667516","exception":false,"start_time":"2022-06-25T16:37:34.638237","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-26T15:36:15.732063Z","iopub.execute_input":"2022-06-26T15:36:15.732662Z","iopub.status.idle":"2022-06-26T15:36:16.293919Z","shell.execute_reply.started":"2022-06-26T15:36:15.732616Z","shell.execute_reply":"2022-06-26T15:36:16.292709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.1 Filling missing values\n","metadata":{}},{"cell_type":"code","source":"# Age fillna with mean age for each class\nall_df['Age'] = all_df['Age'].fillna(all_df['Age'].mean())\n\n# Cabin, fillna with 'X' and take first letter\nall_df['Cabin'] = all_df['Cabin'].fillna('X').map(lambda x: x[0].strip())\n\n# Ticket, fillna with 'X', split string and take first split \nall_df['Ticket'] = all_df['Ticket'].fillna('X').map(lambda x:str(x).split()[0] if len(str(x).split()) > 1 else 'X')\n\n# Fare, fillna with mean value\nfare_map = all_df[['Fare', 'Pclass']].dropna().groupby('Pclass').median().to_dict()\nall_df['Fare'] = all_df['Fare'].fillna(all_df['Pclass'].map(fare_map['Fare']))\nall_df['Fare'] = np.log1p(all_df['Fare'])\n\n# Embarked, fillna with 'X' value\nall_df['Embarked'] = all_df['Embarked'].fillna('X')\n\n# Name, take only surnames\nall_df['Name'] = all_df['Name'].map(lambda x: x.split(',')[0])","metadata":{"papermill":{"duration":0.077511,"end_time":"2022-06-25T16:37:34.758323","exception":true,"start_time":"2022-06-25T16:37:34.680812","status":"failed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-26T15:36:16.295537Z","iopub.execute_input":"2022-06-26T15:36:16.295879Z","iopub.status.idle":"2022-06-26T15:36:16.702151Z","shell.execute_reply.started":"2022-06-26T15:36:16.295846Z","shell.execute_reply":"2022-06-26T15:36:16.70136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.2 Encoding","metadata":{}},{"cell_type":"code","source":"label_cols = ['Name', 'Ticket', 'Sex']\nonehot_cols = ['Cabin', 'Embarked']\nnumerical_cols = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']","metadata":{"execution":{"iopub.status.busy":"2022-06-26T15:36:16.703228Z","iopub.execute_input":"2022-06-26T15:36:16.704166Z","iopub.status.idle":"2022-06-26T15:36:16.709205Z","shell.execute_reply.started":"2022-06-26T15:36:16.704129Z","shell.execute_reply":"2022-06-26T15:36:16.708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def label_encoder(c):\n    le = LabelEncoder()\n    return le.fit_transform(c)\n\nscaler = StandardScaler()\n\nonehot_encoded_df = pd.get_dummies(all_df[onehot_cols])\nlabel_encoded_df = all_df[label_cols].apply(label_encoder)\nnumerical_df = pd.DataFrame(scaler.fit_transform(all_df[numerical_cols]), columns=numerical_cols)\ntarget_df = all_df[TARGET]\n\nall_df = pd.concat([numerical_df, label_encoded_df, onehot_encoded_df, target_df], axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-06-26T15:36:16.710618Z","iopub.execute_input":"2022-06-26T15:36:16.710918Z","iopub.status.idle":"2022-06-26T15:36:17.116129Z","shell.execute_reply.started":"2022-06-26T15:36:16.71089Z","shell.execute_reply":"2022-06-26T15:36:17.115176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5 Models\n\nWe are going to use 2 models: LGBM Classifier,and XGB Classifier. We will also create 5 folds cross validation. \n\n### TrueFoundry Experiment Tracking\n\nWe will track our experiment using TrueFoundry. Below are some explanations in the code related to TrueFoundry experiment tracking. We are using XGB Classifier as an example and consistently being used for other models:\n\n- Create ```run = client.create_run(project_name=project_name, run_name=\"XGB\")``` to start logging our experiment by creating project name and the run name. In this case we log the project name as ```synthanic``` that has been setup before and naming our run as ```XGB``` ```for ```XGBoost Classifier``` model.\n- We can track our dataset including target prediction and target actual using ```run.log_dataset(features=train_df[features], dataset_name=\"full\", actuals=train_df['Transported'], predictions=train_oof)```. Logging our actual and prediction target will help us to compare them in TrueFoundry platform. This line of codes are putted the end of the code as we will need to wait until all the prediction in each fold finished.\n- We will also do the same thing for each fold-dataset, we ```userun.log_dataset(dataset_name=\"fold_\"+str(fold), features=X_valid, actuals=y_valid, predictions=temp_oof) ``` but we will perform this after a fold prediction finished.\n- To log hyperparameters from the model, we use ```run.log_params(model.get_xgb_params())```. We can only log 1 set hyperparameters, that's why we put it at the end of the code.\n- We can also log our validation accuracy metrics over time using below code:\n    ```results = model.evals_result()```\n   ``` epochs = len(results['validation_0']['error'])```\n    ```accuracy_fold = [1-err for err in results['validation_0']['error']]```\n   ``` for global_step in range(epochs):```\n    ```run.log_metrics(metric_dict={f'Accuracy_fold_{fold}':accuracy_fold[global_step]}, step=global_step)```\n    We can compare the ```accuracy``` of each ```fold``` and ```OOF``` accuracy across all model over time.\n    The last but not least, we need to end our run using ```run.end()```.\n\n","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"### 5.1 LGBM Classifier\n\nLGBM ((Light Gradient Boosting Machine) is a gradient boosting framework based on decision trees to increases the efficiency of the model and reduces memory usage. It uses two novel techniques: Gradient-based One Side Sampling and Exclusive Feature Bundling (EFB) which fulfills the limitations of histogram-based algorithm that is primarily used in all GBDT (Gradient Boosting Decision Tree) frameworks. The two techniques of GOSS and EFB described below form the characteristics of LightGBM Algorithm. They comprise together to make the model work efficiently and provide it a cutting edge over other GBDT frameworks ","metadata":{}},{"cell_type":"code","source":"params = {\n    'metric': 'binary_logloss',\n    'n_estimators': N_ESTIMATORS,\n    'objective': 'binary',\n    'random_state': SEED,\n    'learning_rate': 0.01,\n    'min_child_samples': 150,\n    'reg_alpha': 3e-5,\n    'reg_lambda': 9e-2,\n    'num_leaves': 20,\n    'max_depth': 16,\n    'colsample_bytree': 0.8,\n    'subsample': 0.8,\n    'subsample_freq': 2,\n    'max_bin': 240,\n}\n","metadata":{"_cell_guid":"7fcb62a1-18f9-4642-948a-351ae24fa96c","_uuid":"60762c9c-fab2-4384-8932-19a6870b6106","collapsed":false,"jupyter":{"outputs_hidden":false},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-06-26T15:36:17.11784Z","iopub.execute_input":"2022-06-26T15:36:17.118268Z","iopub.status.idle":"2022-06-26T15:36:17.124437Z","shell.execute_reply.started":"2022-06-26T15:36:17.118224Z","shell.execute_reply":"2022-06-26T15:36:17.123634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run = client.create_run(project_name=project_name, run_name=\"LGBMClassifier\") #TrueFoundry\nlgb_oof  = np.zeros(train_df.shape[0])\nlgb_preds = np.zeros(test_df.shape[0])\n\nskf = StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED)\n\nfor fold, (train_idx, valid_idx) in enumerate(skf.split(all_df, all_df[TARGET])):\n    print(f\"===== FOLD {fold} =====\")\n    oof_idx = np.array([idx for idx in valid_idx if idx < train_df.shape[0]])\n    preds_idx = np.array([idx for idx in valid_idx if idx >= train_df.shape[0]])\n\n    X_train, y_train = all_df.iloc[train_idx].drop(TARGET, axis=1), all_df.iloc[train_idx][TARGET]\n    X_valid, y_valid = all_df.iloc[oof_idx].drop(TARGET, axis=1), all_df.iloc[oof_idx][TARGET]\n    X_test = all_df.iloc[preds_idx].drop(TARGET, axis=1)\n    \n    pre_model = lgb.LGBMClassifier(**params)\n    pre_model.fit(\n        X_train, y_train,\n        eval_set=[(X_train, y_train),(X_valid, y_valid)],\n        early_stopping_rounds=EARLY_STOPPING_ROUNDS,\n        verbose=VERBOSE\n    )\n\n    params2 = params.copy()\n    params2['learning_rate'] = params['learning_rate'] * 0.1\n    model = lgb.LGBMClassifier(**params2)\n    model.fit(\n        X_train, y_train,\n        eval_set=[(X_train, y_train),(X_valid, y_valid)],\n        early_stopping_rounds=EARLY_STOPPING_ROUNDS,\n        verbose=VERBOSE,\n        init_model=pre_model\n    )\n        \n    temp_oof= model.predict(X_valid)\n    lgb_oof[oof_idx]=temp_oof\n    lgb_preds[preds_idx-train_df.shape[0]] = model.predict(X_test)\n    \n    print(f'Fold {fold} Accuracy: ', accuracy_score(y_valid, temp_oof))\n    results = model.evals_result_\n    epochs = len(results['valid_1']['binary_logloss'])\n    accuracy_fold = [1-err for err in results['valid_1']['binary_logloss']]\n    for global_step in range(epochs):\n        run.log_metrics(metric_dict={f'Accuracy_fold_{fold}':accuracy_fold[global_step]}, step=global_step) #TrueFoundry\n    run.log_dataset(dataset_name=\"fold_\"+str(fold), features=X_valid, actuals=y_valid, predictions=temp_oof) #TrueFoundry\n\n    \naccuracy_final = accuracy_score(train_df['Survived'], lgb_oof)    \nprint(f'OOF AUC: ', accuracy_final)\n\nrun.log_params(model.get_params()) #TrueFoundry\nrun.log_metrics(metric_dict={'Accuracy_OOF':accuracy_final}) #TrueFoundry\nrun.end() #TrueFoundry\n \n    ","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-26T15:36:17.125601Z","iopub.execute_input":"2022-06-26T15:36:17.12627Z","iopub.status.idle":"2022-06-26T15:48:27.345943Z","shell.execute_reply.started":"2022-06-26T15:36:17.126238Z","shell.execute_reply":"2022-06-26T15:48:27.344859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.2 XGBoost Classifier\n\nXGBoost, which stands for Extreme Gradient Boosting, is a scalable, distributed gradient-boosted decision tree (GBDT) machine learning library. It provides parallel tree boosting and is the leading machine learning library for regression, classification, and ranking problems.\n\nXGBoost is a scalable and highly accurate implementation of gradient boosting that pushes the limits of computing power for boosted tree algorithms, being built largely for energizing machine learning model performance and computational speed. With XGBoost, trees are built in parallel, instead of sequentially like GBDT. It follows a level-wise strategy, scanning across gradient values and using these partial sums to evaluate the quality of splits at every possible split in the training set. ","metadata":{}},{"cell_type":"code","source":"params = {\n'max_depth':7,\n'n_estimators':5000,\n'objective':'binary:logistic',\n'booster':'gbtree',\n'n_jobs':1,\n'min_child_weight':1,\n'colsample_bytree':0.7,\n}\n\n\n","metadata":{"execution":{"iopub.status.busy":"2022-06-26T15:48:27.3473Z","iopub.execute_input":"2022-06-26T15:48:27.347715Z","iopub.status.idle":"2022-06-26T15:48:27.355116Z","shell.execute_reply.started":"2022-06-26T15:48:27.347683Z","shell.execute_reply":"2022-06-26T15:48:27.353639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run = client.create_run(project_name=project_name, run_name=\"XGB\") #TrueFoundry\n\nxgb_oof  = np.zeros(train_df.shape[0])\nxgb_preds = np.zeros(test_df.shape[0])\n\nskf = StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED)\n\nfor fold, (train_idx, valid_idx) in enumerate(skf.split(all_df, all_df[TARGET])):\n    print(f\"===== FOLD {fold} =====\")\n    oof_idx = np.array([idx for idx in valid_idx if idx < train_df.shape[0]])\n    preds_idx = np.array([idx for idx in valid_idx if idx >= train_df.shape[0]])\n\n    X_train, y_train = all_df.iloc[train_idx].drop(TARGET, axis=1), all_df.iloc[train_idx][TARGET]\n    X_valid, y_valid = all_df.iloc[oof_idx].drop(TARGET, axis=1), all_df.iloc[oof_idx][TARGET]\n    X_test = all_df.iloc[preds_idx].drop(TARGET, axis=1)\n    \n    model = XGBClassifier(**params)\n    model.fit(\n        X_train, y_train,\n        eval_set=[(X_train, y_train),(X_valid, y_valid)],\n        early_stopping_rounds=EARLY_STOPPING_ROUNDS,\n        verbose=VERBOSE\n    )\n\n            \n    temp_oof= model.predict(X_valid)\n    xgb_oof[oof_idx]=temp_oof\n    xgb_preds[preds_idx-train_df.shape[0]] = model.predict(X_test)\n    \n    print(f'Fold {fold} Accuracy: ', accuracy_score(y_valid, temp_oof))\n    results =model.evals_result()\n    epochs = len(results['validation_0']['logloss'])\n    accuracy_fold = [1-err for err in results['validation_0']['logloss']]\n    for global_step in range(epochs):\n        run.log_metrics(metric_dict={f'Accuracy_fold_{fold}':accuracy_fold[global_step]}, step=global_step) #TrueFoundry\n    run.log_dataset(dataset_name=\"fold_\"+str(fold), features=X_valid, actuals=y_valid, predictions=temp_oof) #TrueFoundry\n\n    \naccuracy_final = accuracy_score(train_df['Survived'], xgb_oof)    \nprint(f'OOF AUC: ', accuracy_final)\n\nrun.log_params(model.get_params()) #TrueFoundry\nrun.log_metrics(metric_dict={'Accuracy_OOF':accuracy_final}) #TrueFoundry\nrun.end() #TrueFoundry\n \n    ","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-26T15:48:27.356973Z","iopub.execute_input":"2022-06-26T15:48:27.357282Z","iopub.status.idle":"2022-06-26T16:00:37.459265Z","shell.execute_reply.started":"2022-06-26T15:48:27.357255Z","shell.execute_reply":"2022-06-26T16:00:37.458077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.3 Ensemble\n\nThis is only for competition purpose (to boost score).","metadata":{}},{"cell_type":"code","source":"submission['submit_lgb'] = np.where(lgb_preds>0.5, 1, 0)\nsubmission['submit_xgb'] = np.where(xgb_preds>0.5, 1, 0)\n","metadata":{"execution":{"iopub.status.busy":"2022-06-26T16:00:37.46053Z","iopub.execute_input":"2022-06-26T16:00:37.46082Z","iopub.status.idle":"2022-06-26T16:00:37.469288Z","shell.execute_reply.started":"2022-06-26T16:00:37.460793Z","shell.execute_reply":"2022-06-26T16:00:37.46806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission[[col for col in submission.columns if col.startswith('submit_')]].sum(axis = 1).value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-26T16:00:37.470605Z","iopub.execute_input":"2022-06-26T16:00:37.471235Z","iopub.status.idle":"2022-06-26T16:00:37.48805Z","shell.execute_reply.started":"2022-06-26T16:00:37.471195Z","shell.execute_reply":"2022-06-26T16:00:37.486902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission[TARGET] = (submission[[col for col in submission.columns if col.startswith('submit_')]].sum(axis=1) >= 2).astype(int)\nsubmission.drop([col for col in submission.columns if col.startswith('submit_')], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-06-26T16:00:37.489965Z","iopub.execute_input":"2022-06-26T16:00:37.490489Z","iopub.status.idle":"2022-06-26T16:00:37.503613Z","shell.execute_reply.started":"2022-06-26T16:00:37.490443Z","shell.execute_reply":"2022-06-26T16:00:37.502651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.4 Submission","metadata":{}},{"cell_type":"code","source":"submission['submit_1'] = submission[TARGET].copy()\nsubmission['submit_2'] = pd.read_csv(\"../input/local-tps-apr/voting_submission.csv\")[TARGET]\nsubmission['submit_3'] = pd.read_csv(\"../input/local-tps-apr/dae.csv\")[TARGET]","metadata":{"execution":{"iopub.status.busy":"2022-06-26T16:00:37.504676Z","iopub.execute_input":"2022-06-26T16:00:37.505222Z","iopub.status.idle":"2022-06-26T16:00:37.561687Z","shell.execute_reply.started":"2022-06-26T16:00:37.505165Z","shell.execute_reply":"2022-06-26T16:00:37.560631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission[[col for col in submission.columns if col.startswith('submit_')]].sum(axis = 1).value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-26T16:00:37.563008Z","iopub.execute_input":"2022-06-26T16:00:37.563418Z","iopub.status.idle":"2022-06-26T16:00:37.576295Z","shell.execute_reply.started":"2022-06-26T16:00:37.563386Z","shell.execute_reply":"2022-06-26T16:00:37.575243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission[TARGET] = (submission[[col for col in submission.columns if col.startswith('submit_')]].sum(axis=1) >= 2).astype(int)\n","metadata":{"execution":{"iopub.status.busy":"2022-06-26T16:00:37.577444Z","iopub.execute_input":"2022-06-26T16:00:37.578331Z","iopub.status.idle":"2022-06-26T16:00:37.588143Z","shell.execute_reply.started":"2022-06-26T16:00:37.578299Z","shell.execute_reply":"2022-06-26T16:00:37.586941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission[['PassengerId', TARGET]].to_csv(\"voting_submission.csv\", index = False)","metadata":{"execution":{"iopub.status.busy":"2022-06-26T16:00:37.589734Z","iopub.execute_input":"2022-06-26T16:00:37.590528Z","iopub.status.idle":"2022-06-26T16:00:37.761237Z","shell.execute_reply.started":"2022-06-26T16:00:37.590478Z","shell.execute_reply":"2022-06-26T16:00:37.76005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6 TrueFoundry Platform\n\nIn this section we will see how our ```dataset```, ```hyperparameters``` and ```metrics``` have been logged in TrueFoundry. We will see into 2 sections: Projects and Runs.","metadata":{}},{"cell_type":"markdown","source":"### 6.1 Projects\n\nWe can check all of our projects in ML Foundry section. In this case we are looking for our ```synthanic``` projects.","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/all_project.jpg')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.772349Z","iopub.execute_input":"2022-06-26T16:00:37.772698Z","iopub.status.idle":"2022-06-26T16:00:37.78098Z","shell.execute_reply.started":"2022-06-26T16:00:37.772665Z","shell.execute_reply":"2022-06-26T16:00:37.780099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\nChoose the project and we will see model runs that have been logged before which are ``LGBMClassifier`` and ``XGB ``.\n","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/model.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.782369Z","iopub.execute_input":"2022-06-26T16:00:37.782696Z","iopub.status.idle":"2022-06-26T16:00:37.794786Z","shell.execute_reply.started":"2022-06-26T16:00:37.782654Z","shell.execute_reply":"2022-06-26T16:00:37.793737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6.2 Runs\n\nLet's check our XGB run. In the upper side, we can see our ``Run Name``, ``Run Id``, ``Author``, ``Status``, ``Last Updated On``, ``Tags`` and ``Run Duration``. We can also put ``tags`` and ``notes``.","metadata":{}},{"cell_type":"markdown","source":"#### 6.2.1 Overview\n\n\n* On the left side, we can see our log``Key Metrics`` which is ``accuracy``. It logs ``fold_0`` through ``fold_4`` including ``Accuracy_OOF `` metrics.\n* On the right side, we can see our  ``hyperparameters`` that has been logged. It also represents the latest hyperparameters and it would be the same across the folds.\n\n","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/xgb_overview.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.796408Z","iopub.execute_input":"2022-06-26T16:00:37.797409Z","iopub.status.idle":"2022-06-26T16:00:37.808422Z","shell.execute_reply.started":"2022-06-26T16:00:37.797363Z","shell.execute_reply":"2022-06-26T16:00:37.807435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 6.2.2 Run Metrics\n\nWe can see each of our validation ``fold (0 through 4) accuracy `` in a line graph. Meaning we can see how it performs in each steps.","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/xgb_result.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.809762Z","iopub.execute_input":"2022-06-26T16:00:37.810458Z","iopub.status.idle":"2022-06-26T16:00:37.822216Z","shell.execute_reply.started":"2022-06-26T16:00:37.810426Z","shell.execute_reply":"2022-06-26T16:00:37.821034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 6.2.3 Data & Feature Metrics\n\nWe can see our folds dataset, in this case we can see ``fold_0`` through ``fold_4`` and ``full dataset`` as we have logged them before. We can also see more details on each features by clicking the Details.","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/xgb_feature.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.823801Z","iopub.execute_input":"2022-06-26T16:00:37.824581Z","iopub.status.idle":"2022-06-26T16:00:37.834947Z","shell.execute_reply.started":"2022-06-26T16:00:37.824549Z","shell.execute_reply":"2022-06-26T16:00:37.833872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\nWe can also see the comparsion between our predictions and actual performance for each ``fold (0 to 4)`` and also in the ``full dataset``.\n","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/distribution_label.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.836396Z","iopub.execute_input":"2022-06-26T16:00:37.837305Z","iopub.status.idle":"2022-06-26T16:00:37.846115Z","shell.execute_reply.started":"2022-06-26T16:00:37.837268Z","shell.execute_reply":"2022-06-26T16:00:37.84533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 6.2.4 General Artifact\nIn here we can see our dataset that have been stored in csv format and can be re-downloaded.","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/general_artifact.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.847615Z","iopub.execute_input":"2022-06-26T16:00:37.848714Z","iopub.status.idle":"2022-06-26T16:00:37.858071Z","shell.execute_reply.started":"2022-06-26T16:00:37.848666Z","shell.execute_reply":"2022-06-26T16:00:37.857072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6.3 Models Comparison\n\nWe can perform model comparison by clicking all the models that we want to compare. ","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/model_comparison1.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.859689Z","iopub.execute_input":"2022-06-26T16:00:37.860385Z","iopub.status.idle":"2022-06-26T16:00:37.870486Z","shell.execute_reply.started":"2022-06-26T16:00:37.860331Z","shell.execute_reply":"2022-06-26T16:00:37.86934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\nWe can see the comparison of our models ``(LGBMClassifier and XGB)`` .The graphics are based on the validation accuracy metric for each fold. \n","metadata":{}},{"cell_type":"code","source":"Image('../input/truefoundry-2022/model_comparison2.png')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-26T16:00:37.872252Z","iopub.execute_input":"2022-06-26T16:00:37.872922Z","iopub.status.idle":"2022-06-26T16:00:37.882079Z","shell.execute_reply.started":"2022-06-26T16:00:37.87288Z","shell.execute_reply":"2022-06-26T16:00:37.880847Z"},"trusted":true},"execution_count":null,"outputs":[]}]}