{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"},{"sourceId":34377,"databundleVersionId":3220602,"sourceType":"competition"},{"sourceId":35332,"databundleVersionId":3723648,"sourceType":"competition"},{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"},{"sourceId":52784,"databundleVersionId":5687476,"sourceType":"competition"},{"sourceId":57236,"databundleVersionId":7292407,"sourceType":"competition"},{"sourceId":57891,"databundleVersionId":7056235,"sourceType":"competition"},{"sourceId":70089,"databundleVersionId":9515283,"sourceType":"competition"},{"sourceId":23498,"sourceType":"datasetVersion","datasetId":310},{"sourceId":23812,"sourceType":"datasetVersion","datasetId":17810},{"sourceId":2654038,"sourceType":"datasetVersion","datasetId":434238},{"sourceId":147704369,"sourceType":"kernelVersion"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n

All Best Tabular Classifiers - Comparative Study
\n

\n

Created by Alexandre Le Mercier on the 8th of September 2024
\n

\n \n \n\n![TITLE IMAGE](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F17037041%2Fa176d8ab625c73936eb9db7ea29687a6%2FClassifiers%20Showdown%20Enhanced.png?generation=1725790655686010&alt=media)\n\n\n\n
\n

Table of Contents

\n
\n \n![Final Scores](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F17037041%2F41b11e901cb19c339e09884f7097406c%2F_scores1.png?generation=1726314422176735&alt=media)","metadata":{}},{"cell_type":"markdown","source":"\n
\n

1. Introduction - Motivation and Game Rules

\n
\n\n[Back to table of contents](#TOC)\n\n
\n Remark: if you are runing this notebook for the first time and getting errors, think about checking the library versions compatibility! Some updates sometimes make older codes crash. You can force a given version to be installed with !pip uninstall the_library -y, then !pip install the_library==1.1.1 if you want e.g. to install version 1.1.1. The versions I use are listed in section 2.\n
\n\n## Motivation\n\nWe are all using ensemble methods for tabular classification. Usually, `RandomForestClassifier` from scikit-learn is the perfect tool you need: it is easy to use, needs very few data preprocessing and generally reaches outstanding performances without needing hyperparameter tuning. However, scikit-learn doesn't have the monopole of tabular classification. In the [machine learning intermediate tutorial on Kaggle](https://www.kaggle.com/learn/intermediate-machine-learning) for instance, [XGBoost](https://xgboost.readthedocs.io/en/stable/) is designed as \"*the most accurate modeling technique for structured data.*\".\n\nNevertheless, from my personal experience and the fellow AI students I met, XGBoost didn't reach expected performances. Is `XGBoostClassifier` harder to use than its scikit-learn's fellow? Maybe. Does it work in more specific contexts, that do not correspond to the use cases my mates worked on? Maybe. The purpose of this study is to **give a general overview of those classifier's performances and best use case**, so that you have some experimental knowledge about those models. The goal is to help you deciding which classifier to pick up for your specific use case, and a few tips to use them correctly (though I do not intend to make an in-depth analysis of the libraries insights, there already exists several interesting notebooks on Kaggle about that).\n\n## Classifiers\n\nI will compare the following classifiers over very similar tests. You will find below a table of the used classifiers and their corresponding preprocessing needs:\n\n| Model | Can Skip Normalization | Can Skip Scaling | Can Skip Missing Value Handling | Can Skip Categorical Value Handling |\n|----------------|------------------------|------------------|-------------------------------|------------------------------------|\n| Random Forest from Sklearn | Yes | Yes | No | No |\n| Gradient Boosting from Sklearn| Yes | Yes | No | No |\n| Histogram Gradient Boosting from Sklearn| Yes | Yes | No | No |\n| Bagging Classifier from Sklearn| Yes | Yes | No | No |\n| XGBoost | Yes | Yes | Yes | No |\n| LightGBM | Yes | Yes | Yes | Yes (with native support) |\n| CatBoost | Yes | Yes | Yes | Yes (handles categorical directly) |\n| AdaBoost from Sklearn | Yes | Yes | No | No |\n| ExtraTrees from Sklearn | Yes | Yes | No | No |\n| Sequential from Keras | No | No | No | No |\n\nOf note, GPU acceleration will **not** be used through this notebook, to guarantee a good balance into computational ressources used.\n\n## Datasets\n\nI choose 3 datasets (so far) among the most popular ones in Kaggle, that are all very different from each other, plus 3 other that I added later (v1.1) for diffent reasons. Those are the [credit card fraud detection](https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud) (imbalanced binary classification), the [Netflix TV shows](https://www.kaggle.com/datasets/shivamb/netflix-shows) (multi-label classification), the [chest Xray pneumonia](https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia) (image classification, because it is still interesting to see how tabular models work on images), the space Titanic, the MNIST and the ICR datasets.\n\nRegarding the pneumonia dataset, I will extract some features from the image itself using the library I developped: `fast-skimage`. CatBoost classification has proved to be very efficient in my [previous study about brain cancer classification](https://www.kaggle.com/code/alexandrelemercier/99-8-accuracy-on-brain-tumor-classification), so I think it is worth the shot trying it again. Tabular classifiers are much faster than Keras ones, and we could hence spare lots of intense training resources.\n\n
\n Remark: you are looking at version 1.3 of this study. Other datasets might be added in later versions.\n
\n\n## Experimental Tests\n\nThe following experimental tests will be conducted to compare those classifiers:\n\n1. Accuracy and F1-score computing with minimum possible preprocessing and no parameter tuning. This will give an insight about **training time** and **ease of use** (because we don't only want to optimize computational resources, but also time we spend developping models);\n2. The same after an Optuna hyperparameter optimization (I am not a big fan of grid search). This will give an insight about **performance**. Note that the ease of use score can be derived from the difference between initial performance and optimized performance.\n3. Same models, but training on a very small portion of each dataset. A good model should extract patterns from as few data as possible. This will give an insight about **data dependance**.\n4. Finally, conduct [evasion attacks](https://www.ibm.com/docs/en/watsonx/saas?topic=atlas-evasion-attack) on the best models using the [Adversarial Robustness Toolbox (ART)](https://github.com/Trusted-AI/adversarial-robustness-toolbox). This will give an insight on **adversarial robustness**.\n\n## Disclaimer\n\nThe conclusions of this study are drawn from pure empirical analyses over a limited number of publicly available datasets. At version 1.0, there were only 3 of them. The confidence into my results is limited by the small number of datasets, and my limited experience about each library. **Feel free to improve this study by suggesting new implementation techniques or/and new datasets.**\n\nAlso, I will use ChatGPT 4o for several tasks. I believe it is an incredible tool that can make us all win a lot of time. Some will disagree, but I don't see any problem in using it for machine learning projects, and even strongly encourage collegues to do so.\n\n## Previous Work\n\nThis work is inspired from my first experiments on linear models (cf. [The Regularization Rumble](https://www.kaggle.com/code/alexandrelemercier/mlmfo-episode-1-the-regularization-rumble)). This is my most successfull notebook on Kaggle (at the time I am writing those lines), but I was much less experimented as I am now. I hope this study will be more usefull that the previous one.\n\nAs mentionned before, the methods used on image classification derives from my [brain tumor study](https://www.kaggle.com/code/alexandrelemercier/99-8-accuracy-on-brain-tumor-classification). I am particularly proud of this work, which was very well received by my image processing teacher and allowed me to obtain an internship at Sony R&D.\n\nFinally, I discovered that autoencoders (AE) could reach high performance in anomaly detection during my work on [Quantized Autoencoders (QAE)](https://www.kaggle.com/code/alexandrelemercier/quantized-autoencoder-qae-ids-for-iot-devices) where I built the extensive QAE class to detect cyberattacks. This is also a notebook I am proud of.\n\n## Updates Log\n\n- 1.0: Initial study\n- 1.1: Add 3 datasets (ICR, MNIST, Spaceship Titanic) and more conclusion plots\n- 1.2: Add Keras' Sequential Model for all datasets and CIC dataset\n- 1.3: Apply second experiment (Optuna optimization) for the 5 datasets that do not lead to near-perfect classification\n\nYou are currently looking at version **1.3** of this study.","metadata":{}},{"cell_type":"markdown","source":"\n
\n

2. Imports and Constants

\n
\n\n[Back to table of contents](#TOC)","metadata":{}},{"cell_type":"code","source":"# Main Imports\n!pip install tensorflow==2.17.0\n!pip install numpy\n!pip install matplotlib\n!pip install seaborn\n!pip install tqdm\n!pip install pandas\n!pip install scikit-learn\n!pip install xgboost\n!pip install catboost\n!pip install lightgbm\n#!pip install fast-skimage==0.3.1\n\n# Other Important Imports\n!pip install optuna\n!pip install adversarial-robustness-toolbox\n\n# Create Output Directories (c.f. constants names)\n!mkdir Figures\n!mkdir Models\n!mkdir Dataframes\n\n# Extensions\n%load_ext autoreload","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-09-24T09:39:41.996251Z","iopub.execute_input":"2024-09-24T09:39:41.997228Z","iopub.status.idle":"2024-09-24T09:43:57.375289Z","shell.execute_reply.started":"2024-09-24T09:39:41.997181Z","shell.execute_reply":"2024-09-24T09:43:57.373955Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#%autoreload\nimport tensorflow as tf\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\nimport sklearn\nimport xgboost\nimport lightgbm\nimport catboost\nimport sys\nimport os\nimport optuna\nimport pickle\nimport warnings\n#import art\n#import fast_skimage\n\nfrom tqdm.notebook import tqdm\nfrom sklearn.model_selection import train_test_split\n\nimports = [tf, np, sns, pd, sklearn, xgboost, lightgbm, catboost, optuna]\n\nfor imp in imports:\n print(imp.__name__.ljust(20), imp.__version__)\n \nRANDOM_SEED = 5\nOPTUNA_TRIALS = 15\nFIGURES_DIR = \"Figures/\"\nMODELS_DIR = \"Models/\"\nOUTPUT_DATA_DIR = \"Dataframes/\"\nINPUT_DATA_PATHS = [\"/kaggle/input/netflix-shows\",\n \"/kaggle/input/creditcardfraud\",\n \"/kaggle/input/chest-xray-pneumonia/chest_xray\"] \n\nnp.random.seed(RANDOM_SEED)\ntf.random.set_seed(RANDOM_SEED)\n\nwarnings.filterwarnings(\"ignore\",category=FutureWarning)\n\nfor path in INPUT_DATA_PATHS:\n sys.path.append(os.path.abspath(path))\n\n# Utility Functions\ndef load(model_name):\n if model_name.endswith(\".pkl\"):\n with open(model_name, 'rb') as file:\n return pickle.load(file)\n else:\n return tf.keras.models.load_model(model_name)\n\ndef save(model, model_name):\n if model_name.endswith(\".pkl\"):\n with open(MODELS_DIR+model_name, 'wb') as file:\n pickle.dump(model, file)\n else:\n model.save(MODELS_DIR+model_name)\n\ndef to_numpy(arr):\n try:\n return np.ndarray(arr)\n except:\n return arr","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-09-24T09:43:57.378753Z","iopub.execute_input":"2024-09-24T09:43:57.379945Z","iopub.status.idle":"2024-09-24T09:44:08.364700Z","shell.execute_reply.started":"2024-09-24T09:43:57.379879Z","shell.execute_reply":"2024-09-24T09:44:08.363508Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

3. Data Preprocessing

\n
\n\n[Back to table of contents](#TOC)\n\n\n## 3.1 Minimum Preprocessing for Tabular Data\n\nThis includes categorical data and missing value handling. Catboost will have a specially dedicated pipeline using the Pool object. We will create `df_credit`, `df_netflix` along with `X_train_credit`, etc. knowing that the label for credit card fraud is \"Class\" and the one for Netflix is \"duration\". For Netflix, we need to drop every row which \"duration\" is not in the form \"X Seasons\" or \"1 Season\". The labels will be the number of seasons, so we will just encode \"1\", \"2\", etc. in a new column \"Class\". Of course, we need to drop the previous \"duration\" column.\n\nThe Netflix dataset needs in-depth preprocessing, because most of features are categorical:\n- \"show_id\": to drop\n- \"type\": OHE encoding because few labels (2)\n- \"title\": extract the number of words in the title, then drop the title\n- \"director\": lots of null values, sometimes several directors, lots of different directors... keep for Pool's Catboost but drop for the others\n- \"cast\": extract actors in a list with \", \" separator, detect the 20 most cited actors, OHE encoding for them. There are null values here too. In this case, put \"0\" everywhere for OHE\n- \"country\": 5017 different values. OHE over the 20 most cited, \"other\" for the others.\n- \"date_added\": extract the month (the content before the space character) and OHE other the 12 months\n- \"realease_year\": the only numerical feature\n- \"rating\": 3k+ different values, OHE for the 20 most common\n- \"listed_in\": 8k+ different values, do the same as for the actors (OHE over most cited categories)\n- \"description\": drop\n\nWe also make some categorical preprocessing for the Spaceship Titanic.\n\nOf note, the columns from which OHE was extracted should be dropped. No OHE should be applied to Pool's Catboost.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.impute import SimpleImputer\n\n# Load tabular datasets\ndf_credit = pd.read_csv('/kaggle/input/creditcardfraud/creditcard.csv')\ndf_netflix = pd.read_csv('/kaggle/input/netflix-shows/netflix_titles.csv')\ndf_ICR = pd.read_csv('/kaggle/input/icr-identify-age-related-conditions/train.csv')\ndf_sptit = pd.read_csv('/kaggle/input/spaceship-titanic/train.csv')\ndf_mnist = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ndf_cic = pd.read_csv('/kaggle/input/creating-a-smaller-dataset-for-ciciot2023/0.1percent_2classes.csv')","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:44:08.366632Z","iopub.execute_input":"2024-09-24T09:44:08.367646Z","iopub.status.idle":"2024-09-24T09:45:02.726570Z","shell.execute_reply.started":"2024-09-24T09:44:08.367577Z","shell.execute_reply":"2024-09-24T09:45:02.725256Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 3.1 Minimum Preprocessing for Credit Card Dataset (df_credit)\n# Handling missing values\ndf_credit.dropna(inplace=True)\n\n# Split credit card dataset\nX_credit = df_credit.drop('Class', axis=1)\ny_credit = df_credit['Class']\nX_train_credit, X_test_credit, y_train_credit, y_test_credit = train_test_split(X_credit, y_credit, test_size=0.2, random_state=RANDOM_SEED)\n\n# 3.2 Minimum Preprocessing for Netflix Dataset (df_netflix)\n\n# 1. Drop rows where \"duration\" is not in \"X Seasons\" or \"1 Season\" form\ndf_netflix = df_netflix[df_netflix['duration'].str.contains(r'\\d+ Season', na=False)]\n\n# 2. Encode \"duration\" into a \"Class\" column with numerical values\ndf_netflix['Class'] = df_netflix['duration'].str.extract('(\\d+)').astype(int)\ndf_netflix.drop('duration', axis=1, inplace=True)\n\n# 3. Drop unnecessary columns\ndf_netflix.drop(['show_id', 'title', 'description'], axis=1, inplace=True)\n\n# 4. One-Hot Encode \"type\" (few labels)\nohe_type = OneHotEncoder(drop='first', sparse=False)\nencoded_type = ohe_type.fit_transform(df_netflix[['type']])\ndf_netflix[ohe_type.get_feature_names_out(['type'])] = encoded_type\ndf_netflix.drop('type', axis=1, inplace=True)\n\n# 5. Handle \"director\" column (keep for CatBoost, drop for others)\ndf_netflix_director_catboost = df_netflix[['director']] # Store for CatBoost\ndf_netflix.drop('director', axis=1, inplace=True)\n\n# 6. Handle \"cast\" column: Extract top 20 actors, One-Hot Encode\ntop_actors = df_netflix['cast'].str.split(', ').explode().value_counts().head(20).index\nfor actor in top_actors:\n df_netflix[f'actor_{actor}'] = df_netflix['cast'].apply(lambda x: 1 if pd.notnull(x) and actor in x else 0)\ndf_netflix.drop('cast', axis=1, inplace=True)\n\n# 7. Handle \"country\" column: One-Hot Encode top 20 countries, set others to \"other\"\ntop_countries = df_netflix['country'].value_counts().head(20).index\ndf_netflix['country'] = df_netflix['country'].apply(lambda x: x if x in top_countries else 'other')\nohe_country = OneHotEncoder(drop='first', sparse=False)\nencoded_country = ohe_country.fit_transform(df_netflix[['country']])\ndf_netflix[ohe_country.get_feature_names_out(['country'])] = encoded_country\ndf_netflix.drop('country', axis=1, inplace=True)\n\n# 8. Handle \"date_added\" column: Extract month and One-Hot Encode\ndf_netflix['month_added'] = df_netflix['date_added'].str.split(' ').str[0]\ndf_netflix.drop('date_added', axis=1, inplace=True)\nohe_month = OneHotEncoder(drop='first', sparse=False)\nencoded_month = ohe_month.fit_transform(df_netflix[['month_added']])\ndf_netflix[ohe_month.get_feature_names_out(['month_added'])] = encoded_month\ndf_netflix.drop('month_added', axis=1, inplace=True)\n\n# 9. \"release_year\" is numeric, no changes needed\n\n# 10. Handle \"rating\" column: One-Hot Encode top 20 ratings\ntop_ratings = df_netflix['rating'].value_counts().head(20).index\ndf_netflix['rating'] = df_netflix['rating'].apply(lambda x: x if x in top_ratings else 'other')\nohe_rating = OneHotEncoder(drop='first', sparse=False)\nencoded_rating = ohe_rating.fit_transform(df_netflix[['rating']])\ndf_netflix[ohe_rating.get_feature_names_out(['rating'])] = encoded_rating\ndf_netflix.drop('rating', axis=1, inplace=True)\n\n# 11. Handle \"listed_in\" column: Extract top 20 categories, One-Hot Encode\ntop_categories = df_netflix['listed_in'].str.split(', ').explode().value_counts().head(20).index\nfor category in top_categories:\n df_netflix[f'category_{category}'] = df_netflix['listed_in'].apply(lambda x: 1 if pd.notnull(x) and category in x else 0)\ndf_netflix.drop('listed_in', axis=1, inplace=True)\n\n# Split Netflix dataset into training and test sets\nX_netflix = df_netflix.drop('Class', axis=1)\ny_netflix = df_netflix['Class']\nX_train_netflix, X_test_netflix, y_train_netflix, y_test_netflix = train_test_split(X_netflix, y_netflix, test_size=0.2, random_state=RANDOM_SEED)\n\n\n# At this point, the preprocessed data is ready for model training\nprint(\"Preprocessing successfull!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:02.728746Z","iopub.execute_input":"2024-09-24T09:45:02.729297Z","iopub.status.idle":"2024-09-24T09:45:03.171220Z","shell.execute_reply.started":"2024-09-24T09:45:02.729244Z","shell.execute_reply":"2024-09-24T09:45:03.170102Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\nfrom sklearn.model_selection import train_test_split\nimport pandas as pd\n\n# Load ICR dataset\ndf_ICR = pd.read_csv('/kaggle/input/icr-identify-age-related-conditions/train.csv')\n\n# 1. Drop the 'Id' column\ndf_ICR.drop(['Id', 'EJ'], axis=1, inplace=True)\n\n# 2. Handle missing values by dropping rows containing NaNs\ndf_ICR.dropna(inplace=True) # Drop rows with missing values across the entire dataset\n\n# 3. Separate features and target\nX_ICR = df_ICR.drop('Class', axis=1)\ny_ICR = df_ICR['Class']\n\nX_train_ICR, X_test_ICR, y_train_ICR, y_test_ICR = train_test_split(X_ICR, y_ICR, test_size=0.2, random_state=RANDOM_SEED)\n\nprint(\"ICR preprocessing successful!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:03.173950Z","iopub.execute_input":"2024-09-24T09:45:03.174360Z","iopub.status.idle":"2024-09-24T09:45:03.198466Z","shell.execute_reply.started":"2024-09-24T09:45:03.174321Z","shell.execute_reply":"2024-09-24T09:45:03.197269Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_train_ICR.columns","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:03.199899Z","iopub.execute_input":"2024-09-24T09:45:03.200257Z","iopub.status.idle":"2024-09-24T09:45:03.209970Z","shell.execute_reply.started":"2024-09-24T09:45:03.200222Z","shell.execute_reply":"2024-09-24T09:45:03.208784Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_cic = df_cic.astype({col: 'int' for col in df_cic.select_dtypes(include='bool').columns})\n\n# Drop object columns\ndf_cic = df_cic.drop(columns=df_cic.select_dtypes(include='object').columns)\n\nX_CIC = df_cic.drop('benign', axis=1)\ny_CIC = df_cic['benign']\n\nX_train_CIC, X_test_CIC, y_train_CIC, y_test_CIC = train_test_split(X_CIC, y_CIC, test_size=0.2, random_state=RANDOM_SEED)\n\nprint(\"CIC preprocessing successful!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:03.211722Z","iopub.execute_input":"2024-09-24T09:45:03.212244Z","iopub.status.idle":"2024-09-24T09:45:11.318868Z","shell.execute_reply.started":"2024-09-24T09:45:03.212166Z","shell.execute_reply":"2024-09-24T09:45:11.317539Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.model_selection import train_test_split\nimport warnings\n\n# 3.4 Minimum Preprocessing for Spaceship Titanic Dataset (df_sptit)\nwarnings.filterwarnings('ignore')\n\n# Drop unnecessary columns: PassengerId, HomePlanet, Name, Destination\ndf_sptit.drop(['PassengerId', 'HomePlanet', 'Name', 'Destination'], axis=1, inplace=True)\n\n# Handle missing values (optional: decide how to handle missing values)\ndf_sptit.fillna(df_sptit.mean(numeric_only=True), inplace=True) # Fill numeric columns with mean\ndf_sptit.fillna(df_sptit.mode().iloc[0], inplace=True) # Fill categorical columns with mode\n\n# Split the 'Cabin' column into three components (X/Y/Z)\ndf_sptit[['Cabin_X', 'Cabin_Y', 'Cabin_Z']] = df_sptit['Cabin'].str.split('/', expand=True)\ndf_sptit.drop('Cabin', axis=1, inplace=True) # Drop original Cabin column\n\n# One-Hot Encode the Cabin_X component\nohe_cabin_x = OneHotEncoder(drop='first', sparse=False)\nencoded_cabin_x = ohe_cabin_x.fit_transform(df_sptit[['Cabin_X']])\nencoded_cabin_x_df = pd.DataFrame(encoded_cabin_x, columns=ohe_cabin_x.get_feature_names_out(['Cabin_X'])).astype(int) # Convert to int\ndf_sptit = pd.concat([df_sptit.drop('Cabin_X', axis=1), encoded_cabin_x_df], axis=1)\n\n# One-Hot Encode the Cabin_Y component\nohe_cabin_y = OneHotEncoder(drop='first', sparse=False)\nencoded_cabin_y = ohe_cabin_y.fit_transform(df_sptit[['Cabin_Y']])\nencoded_cabin_y_df = pd.DataFrame(encoded_cabin_y, columns=ohe_cabin_y.get_feature_names_out(['Cabin_Y'])).astype(int) # Convert to int\ndf_sptit = pd.concat([df_sptit.drop('Cabin_Y', axis=1), encoded_cabin_y_df], axis=1)\n\n# One-Hot Encode the Cabin_Z component\nohe_cabin_z = OneHotEncoder(drop='first', sparse=False)\nencoded_cabin_z = ohe_cabin_z.fit_transform(df_sptit[['Cabin_Z']])\nencoded_cabin_z_df = pd.DataFrame(encoded_cabin_z, columns=ohe_cabin_z.get_feature_names_out(['Cabin_Z'])).astype(int) # Convert to int\ndf_sptit = pd.concat([df_sptit.drop('Cabin_Z', axis=1), encoded_cabin_z_df], axis=1)\n\n# One-Hot Encode CryoSleep and VIP\nohe_cryosleep_vip = OneHotEncoder(drop='first', sparse=False)\nencoded_cryosleep_vip = ohe_cryosleep_vip.fit_transform(df_sptit[['CryoSleep', 'VIP']])\nencoded_cryosleep_vip_df = pd.DataFrame(encoded_cryosleep_vip, columns=ohe_cryosleep_vip.get_feature_names_out(['CryoSleep', 'VIP'])).astype(int) # Convert to int\ndf_sptit = pd.concat([df_sptit.drop(['CryoSleep', 'VIP'], axis=1), encoded_cryosleep_vip_df], axis=1)\n\ndf_sptit = df_sptit.astype({col: 'int' for col in df_sptit.select_dtypes(include='object').columns})\n\n# Split Spaceship Titanic dataset into X and y\nX_sptit = df_sptit.drop('Transported', axis=1)\ny_sptit = df_sptit['Transported'].astype(int) # Convert to 0/1 for binary classification\n\nX_sptit = X_sptit.drop(X_sptit.select_dtypes(include=['object']).columns, axis=1)\n\n# Train-test split for the Titanic dataset\nX_train_sptit, X_test_sptit, y_train_sptit, y_test_sptit = train_test_split(X_sptit, y_sptit, test_size=0.2, random_state=RANDOM_SEED)\n\nprint(\"Spaceship Titanic preprocessing successful!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:11.320613Z","iopub.execute_input":"2024-09-24T09:45:11.321042Z","iopub.status.idle":"2024-09-24T09:45:13.075015Z","shell.execute_reply.started":"2024-09-24T09:45:11.320968Z","shell.execute_reply":"2024-09-24T09:45:13.073596Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 3.5 Minimum Preprocessing for MNIST Dataset (df_mnist)\n\ndf_mnist = df_mnist[:3000]\n\n# Normalize the pixel values (0-255) to the range (0-1)\nX_mnist = df_mnist.drop('label', axis=1) / 255.0 # Scale pixel values\ny_mnist = df_mnist['label'] # Labels (digits from 0 to 9)\n\n# Train-test split for the MNIST dataset\nX_train_mnist, X_test_mnist, y_train_mnist, y_test_mnist = train_test_split(X_mnist, y_mnist, test_size=0.2, random_state=RANDOM_SEED)\n\nprint(\"MNIST preprocessing successful!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.076896Z","iopub.execute_input":"2024-09-24T09:45:13.077293Z","iopub.status.idle":"2024-09-24T09:45:13.118737Z","shell.execute_reply.started":"2024-09-24T09:45:13.077255Z","shell.execute_reply":"2024-09-24T09:45:13.117508Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\nimport numpy as np\nfrom PIL import Image\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\nfrom scipy.stats import kurtosis, skew # Importing kurtosis and skew functions\nfrom skimage.feature import graycomatrix, graycoprops\nfrom skimage.measure import shannon_entropy\n\ntry: # check if the files already exist. We don't want to recompute this dataset every time!\n \n X_train_file = os.path.join(OUTPUT_DATA_DIR, 'X_train_xray.npy')\n y_train_file = os.path.join(OUTPUT_DATA_DIR, 'y_train_xray.npy')\n X_test_file = os.path.join(OUTPUT_DATA_DIR, 'X_test_xray.npy')\n y_test_file = os.path.join(OUTPUT_DATA_DIR, 'y_test_xray.npy')\n \n X_train_xray = np.load(X_train_file)\n y_train_xray = np.load(y_train_file)\n X_test_xray = np.load(X_test_file)\n y_test_xray = np.load(y_test_file)\n print(\"Loaded saved datasets successfully.\")\n \nexcept:\n # Define directories\n data_dir = '/kaggle/input/chest-xray-pneumonia/chest_xray'\n train_dir = os.path.join(data_dir, 'train')\n test_dir = os.path.join(data_dir, 'test')\n\n # Image size (ensuring all images are resized to the same dimensions)\n img_size = (299, 299) # You can adjust the size depending on your needs\n\n # Helper function to calculate first-order features\n def calculate_first_order_features(image_array):\n mean = np.mean(image_array)\n variance = np.var(image_array)\n std_dev = np.std(image_array)\n skewness = skew(image_array.flatten())\n kurt = kurtosis(image_array.flatten())\n\n return [mean, variance, std_dev, skewness, kurt]\n\n # Helper function to calculate second-order features using Grey Level Co-occurrence Matrix (GLCM)\n def calculate_second_order_features(image_array):\n # Convert to grayscale if necessary\n if len(image_array.shape) == 3: # If RGB, convert to grayscale\n image_array = np.mean(image_array, axis=2).astype(np.uint8)\n\n # Calculate GLCM (graycomatrix requires uint8 image)\n glcm = graycomatrix(image_array, distances=[1], angles=[0], levels=256, symmetric=True, normed=True)\n\n contrast = graycoprops(glcm, 'contrast')[0, 0]\n energy = graycoprops(glcm, 'energy')[0, 0]\n homogeneity = graycoprops(glcm, 'homogeneity')[0, 0]\n correlation = graycoprops(glcm, 'correlation')[0, 0]\n asm = graycoprops(glcm, 'ASM')[0, 0]\n dissimilarity = graycoprops(glcm, 'dissimilarity')[0, 0]\n entropy = shannon_entropy(image_array)\n\n return [contrast, energy, homogeneity, correlation, asm, dissimilarity, entropy]\n\n # Helper function to extract features from a directory\n def extract_features_from_directory(directory):\n features = []\n labels = []\n for label in ['NORMAL', 'PNEUMONIA']:\n path = os.path.join(directory, label)\n for file_name in tqdm(os.listdir(path), desc=f\"Processing {label} images\"):\n file_path = os.path.join(path, file_name)\n try:\n # Load image and resize\n img = Image.open(file_path).resize(img_size)\n img_array = np.array(img)\n\n # First-order features\n first_order_features = calculate_first_order_features(img_array)\n\n # Second-order features\n second_order_features = calculate_second_order_features(img_array)\n\n # Combine all features\n all_features = first_order_features + second_order_features\n features.append(all_features)\n\n # Append label: 0 for 'NORMAL', 1 for 'PNEUMONIA'\n labels.append(0 if label == 'NORMAL' else 1)\n except Exception as e:\n print(f\"Error processing {file_name}: {e}\")\n\n return np.array(features), np.array(labels)\n\n # Extract features from train and test directories\n X_train_xray, y_train_xray = extract_features_from_directory(train_dir)\n X_test_xray, y_test_xray = extract_features_from_directory(test_dir)\n \n # Save the extracted data into separate files\n np.save(os.path.join(OUTPUT_DATA_DIR, 'X_train_xray.npy'), X_train_xray)\n np.save(os.path.join(OUTPUT_DATA_DIR, 'y_train_xray.npy'), y_train_xray)\n np.save(os.path.join(OUTPUT_DATA_DIR, 'X_test_xray.npy'), X_test_xray)\n np.save(os.path.join(OUTPUT_DATA_DIR, 'y_test_xray.npy'), y_test_xray)\n\n print(\"Files saved successfully in the output directory.\")\n\n\n# To check the shapes of the resulting feature arrays:\nprint(f\"X_train shape: {X_train_xray.shape}\")\nprint(f\"y_train shape: {y_train_xray.shape}\")\nprint(f\"X_test shape: {X_test_xray.shape}\")\nprint(f\"y_test shape: {y_test_xray.shape}\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.120570Z","iopub.execute_input":"2024-09-24T09:45:13.121192Z","iopub.status.idle":"2024-09-24T09:45:13.255389Z","shell.execute_reply.started":"2024-09-24T09:45:13.121139Z","shell.execute_reply":"2024-09-24T09:45:13.254248Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom catboost import Pool\n\n# Spaceship Titanic Dataset (df_sptit)\ndf_sptit = pd.read_csv('/kaggle/input/spaceship-titanic/train.csv') # Reload the Titanic dataset\ndf_sptit.drop(['PassengerId', 'HomePlanet', 'Name', 'Destination'], axis=1, inplace=True) # Drop unnecessary columns\ndf_sptit.fillna(df_sptit.mean(numeric_only=True), inplace=True) # Fill numeric columns with mean\ndf_sptit.fillna(df_sptit.mode().iloc[0], inplace=True) # Fill categorical columns with mode\n\n# Split the 'Cabin' column into three components (X/Y/Z)\ndf_sptit[['Cabin_X', 'Cabin_Y', 'Cabin_Z']] = df_sptit['Cabin'].str.split('/', expand=True)\ndf_sptit.drop('Cabin', axis=1, inplace=True) # Drop original Cabin column\n\n# Prepare X and y for Titanic\nX_sptit = df_sptit.drop('Transported', axis=1)\ny_sptit = df_sptit['Transported'].astype(int) # Convert to 0/1 for binary classification\n\n# Handle missing values in categorical features for Titanic\ncategorical_features_sptit = ['Cabin_X', 'Cabin_Y', 'Cabin_Z', 'CryoSleep', 'VIP']\nX_sptit[categorical_features_sptit] = X_sptit[categorical_features_sptit].fillna('Unknown')\n\n# Split the Titanic dataset into train and test sets\nX_train_sptit, X_test_sptit, y_train_sptit, y_test_sptit = train_test_split(X_sptit, y_sptit, test_size=0.2, random_state=RANDOM_SEED)\n\n# Create Pool objects for Spaceship Titanic\npool_sptit_train = Pool(data=X_train_sptit, label=y_train_sptit, cat_features=categorical_features_sptit)\npool_sptit_test = Pool(data=X_test_sptit, label=y_test_sptit, cat_features=categorical_features_sptit)\n\nprint(\"Spaceship Titanic Pool preprocessing successful!\")\n\n# Netflix Dataset (df_netflix2)\ndf_netflix2 = pd.read_csv('/kaggle/input/netflix-shows/netflix_titles.csv')\ndf_netflix2 = df_netflix2[df_netflix2['duration'].str.contains(r'\\d+ Season', na=False)]\ndf_netflix2['Class'] = df_netflix2['duration'].str.extract('(\\d+)').astype(int)\ndf_netflix2.drop('duration', axis=1, inplace=True)\ndf_netflix2.drop(['show_id', 'title', 'description'], axis=1, inplace=True)\n\n# Handle missing values in categorical features for Netflix\ncategorical_features_netflix = ['director', 'cast', 'listed_in', 'type', 'country', 'date_added', 'release_year', 'rating']\nX_netflix2 = df_netflix2.drop('Class', axis=1)\ny_netflix2 = df_netflix2['Class']\nX_netflix2[categorical_features_netflix] = X_netflix2[categorical_features_netflix].fillna('Unknown')\n\n# Split the Netflix dataset into train and test sets\nX_train_netflix2, X_test_netflix2, y_train_netflix2, y_test_netflix2 = train_test_split(X_netflix2, y_netflix2, test_size=0.2, random_state=RANDOM_SEED)\n\n# Create Pool objects for Netflix\npool_netflix_train = Pool(data=X_train_netflix2, label=y_train_netflix2, cat_features=categorical_features_netflix)\npool_netflix_test = Pool(data=X_test_netflix2, label=y_test_netflix2, cat_features=categorical_features_netflix)\n\n# Credit Card Dataset (df_credit)\n# No categorical features in this dataset, so no cat_features are needed.\npool_credit_train = Pool(data=X_train_credit, label=y_train_credit)\npool_credit_test = Pool(data=X_test_credit, label=y_test_credit)\n\n# Separate features and target\nX_ICR = df_ICR.drop('Class', axis=1)\ny_ICR = df_ICR['Class']\n\n# Train-test split\nX_train_ICR, X_test_ICR, y_train_ICR, y_test_ICR = train_test_split(X_ICR, y_ICR, test_size=0.2, random_state=RANDOM_SEED)\n\n\"\"\"# Specifying the categorical column index for 'EJ'\ncat_features = ['EJ'] # Specify 'EJ' as categorical feature\"\"\"\ncat_features= list()\n\n# Create the CatBoost Pool with the 'EJ' column marked as categorical\npool_ICR_train = Pool(data=X_train_ICR, label=y_train_ICR, cat_features=cat_features)\npool_ICR_test = Pool(data=X_test_ICR, label=y_test_ICR, cat_features=cat_features)\n\n# MNIST Dataset (df_mnist)\n# No categorical features, directly create Pool objects\npool_mnist_train = Pool(data=X_train_mnist, label=y_train_mnist)\npool_mnist_test = Pool(data=X_test_mnist, label=y_test_mnist)\n\n# X-Ray Dataset (train_dir and test_dir are preprocessed using feature extraction)\n# No categorical features, directly create Pool objects\npool_xray_train = Pool(data=X_train_xray, label=y_train_xray)\npool_xray_test = Pool(data=X_test_xray, label=y_test_xray)\n\n# CIC Dataset\n# No categorical features, directly create Pool objects\npool_cic_train = Pool(data=X_train_CIC, label=y_train_CIC)\npool_cic_test = Pool(data=X_test_CIC, label=y_test_CIC)\n\nprint(\"Pool preprocessing successful for all datasets!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.257224Z","iopub.execute_input":"2024-09-24T09:45:13.258072Z","iopub.status.idle":"2024-09-24T09:45:13.739380Z","shell.execute_reply.started":"2024-09-24T09:45:13.258028Z","shell.execute_reply":"2024-09-24T09:45:13.737973Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_credit.info()","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.740921Z","iopub.execute_input":"2024-09-24T09:45:13.741340Z","iopub.status.idle":"2024-09-24T09:45:13.783284Z","shell.execute_reply.started":"2024-09-24T09:45:13.741302Z","shell.execute_reply":"2024-09-24T09:45:13.782177Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_netflix.info()","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.784668Z","iopub.execute_input":"2024-09-24T09:45:13.785070Z","iopub.status.idle":"2024-09-24T09:45:13.810620Z","shell.execute_reply.started":"2024-09-24T09:45:13.785031Z","shell.execute_reply":"2024-09-24T09:45:13.809518Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\n# Encode class labels to ensure they are sequential starting from 0\nlabel_encoder_netflix = LabelEncoder()\ny_train_netflix_encoded = label_encoder_netflix.fit_transform(y_train_netflix)\ny_test_netflix_encoded = label_encoder_netflix.transform(y_test_netflix)","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.816160Z","iopub.execute_input":"2024-09-24T09:45:13.816605Z","iopub.status.idle":"2024-09-24T09:45:13.825905Z","shell.execute_reply.started":"2024-09-24T09:45:13.816564Z","shell.execute_reply":"2024-09-24T09:45:13.824686Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_netflix.Class.unique()","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.827416Z","iopub.execute_input":"2024-09-24T09:45:13.827866Z","iopub.status.idle":"2024-09-24T09:45:13.847201Z","shell.execute_reply.started":"2024-09-24T09:45:13.827819Z","shell.execute_reply":"2024-09-24T09:45:13.845725Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Function to clean column names\ndef sanitize_column_names(df):\n df.columns = df.columns.str.replace(r'[^\\w\\s]', '_', regex=True) # Replace special characters with underscores\n return df\n\nX_train_netflix = sanitize_column_names(X_train_netflix)\nX_test_netflix = sanitize_column_names(X_test_netflix)\nX_train_credit = sanitize_column_names(X_train_credit)\nX_test_credit = sanitize_column_names(X_test_credit)\nX_test_netflix.columns","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.848648Z","iopub.execute_input":"2024-09-24T09:45:13.849175Z","iopub.status.idle":"2024-09-24T09:45:13.866141Z","shell.execute_reply.started":"2024-09-24T09:45:13.849134Z","shell.execute_reply":"2024-09-24T09:45:13.864905Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Check the shapes of the training and testing data\nprint(\"X_train_credit shape:\", X_train_credit.shape)\nprint(\"y_train_credit shape:\", y_train_credit.shape)\nprint(\"X_test_credit shape:\", X_test_credit.shape)\nprint(\"y_test_credit shape:\", y_test_credit.shape)\n\n# Ensure the dimensions match\nassert X_train_credit.shape[0] == y_train_credit.shape[0], \"Mismatch between X_train_credit and y_train_credit\"\nassert X_test_credit.shape[0] == y_test_credit.shape[0], \"Mismatch between X_test_credit and y_test_credit\"","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.868268Z","iopub.execute_input":"2024-09-24T09:45:13.868699Z","iopub.status.idle":"2024-09-24T09:45:13.883910Z","shell.execute_reply.started":"2024-09-24T09:45:13.868659Z","shell.execute_reply":"2024-09-24T09:45:13.882672Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_sptit = df_sptit.drop('Transported', axis=1)\ny_sptit = df_sptit['Transported'].astype(int) # Convert to 0/1 for binary classification\nX_train_sptit, X_test_sptit, y_train_sptit, y_test_sptit = train_test_split(X_sptit, y_sptit, test_size=0.2, random_state=RANDOM_SEED)\nX_train_sptit.select_dtypes(include='object').columns","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.885571Z","iopub.execute_input":"2024-09-24T09:45:13.886008Z","iopub.status.idle":"2024-09-24T09:45:13.917334Z","shell.execute_reply.started":"2024-09-24T09:45:13.885968Z","shell.execute_reply":"2024-09-24T09:45:13.915938Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Helper function to check and print rows containing NaNs\ndef check_for_nans(df, df_name):\n nan_rows = df[df.isnull().any(axis=1)]\n if not nan_rows.empty:\n print(f\"Dropping {len(nan_rows)} rows containing NaN values from {df_name} dataset.\")\n df.dropna(inplace=True)\n\n# Check for object columns and handle them for df_credit\ncheck_for_nans(df_credit, 'df_credit')\nfor col in df_credit.select_dtypes(include='object').columns:\n try:\n df_credit[col] = df_credit[col].astype(int) # Attempt to convert to int\n except ValueError:\n print(f\"Dropping column '{col}' from df_credit as it cannot be converted to int.\")\n df_credit.drop(col, axis=1, inplace=True)\n\n# Check for object columns and handle them for df_netflix\ncheck_for_nans(df_netflix, 'df_netflix')\nfor col in df_netflix.select_dtypes(include='object').columns:\n try:\n df_netflix[col] = df_netflix[col].astype(int) # Attempt to convert to int\n except ValueError:\n print(f\"Dropping column '{col}' from df_netflix as it cannot be converted to int.\")\n df_netflix.drop(col, axis=1, inplace=True)\n\n# Check for object columns and handle them for df_ICR\ncheck_for_nans(df_ICR, 'df_ICR')\nfor col in df_ICR.select_dtypes(include='object').columns:\n try:\n df_ICR[col] = df_ICR[col].astype(int) # Attempt to convert to int\n except ValueError:\n print(f\"Dropping column '{col}' from df_ICR as it cannot be converted to int.\")\n df_ICR.drop(col, axis=1, inplace=True)\n X_ICR = df_ICR.drop('Class', axis=1)\n y_ICR = df_ICR['Class']\n X_train_ICR, X_test_ICR, y_train_ICR, y_test_ICR = train_test_split(X_ICR, y_ICR, test_size=0.2, random_state=RANDOM_SEED)\n\n# Check for object columns and handle them for df_sptit\ncheck_for_nans(df_sptit, 'df_sptit')\nfor col in df_sptit.select_dtypes(include='object').columns:\n try:\n df_sptit[col] = df_sptit[col].astype(int) # Attempt to convert to int\n except ValueError:\n print(f\"Dropping column '{col}' from df_sptit as it cannot be converted to int.\")\n df_sptit.drop(col, axis=1, inplace=True)\n X_sptit = df_sptit.drop('Transported', axis=1)\n y_sptit = df_sptit['Transported'].astype(int) # Convert to 0/1 for binary classification\n X_train_sptit, X_test_sptit, y_train_sptit, y_test_sptit = train_test_split(X_sptit, y_sptit, test_size=0.2, random_state=RANDOM_SEED)\n\n\n# Check for object columns and handle them for df_mnist\ncheck_for_nans(df_mnist, 'df_mnist')\nfor col in df_mnist.select_dtypes(include='object').columns:\n try:\n df_mnist[col] = df_mnist[col].astype(int) # Attempt to convert to int\n except ValueError:\n print(f\"Dropping column '{col}' from df_mnist as it cannot be converted to int.\")\n df_mnist.drop(col, axis=1, inplace=True)\n\n\nprint(\"Check over\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.919596Z","iopub.execute_input":"2024-09-24T09:45:13.920072Z","iopub.status.idle":"2024-09-24T09:45:13.976079Z","shell.execute_reply.started":"2024-09-24T09:45:13.920029Z","shell.execute_reply":"2024-09-24T09:45:13.974664Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

4. First Experiment - Initial Performance

\n
\n\n[Back to table of contents](#TOC)\n\nNow, we will apply `RandomForestClassifier`, `XGBoostClassifier`, `GradientBoostingClassifier`, `AdaBoostClassifier`, `ExtraTreesClassifier` and `LGBMClassifier` and other sklearn classifiers to all datasets. Then, we will apply `CatBoostClassifier` to the Pool objects.\n\nWe first try to recycle results from the previous run. Model fitting takes a long time, so we always first check if `df_results` wasn't saved before:","metadata":{}},{"cell_type":"code","source":"df_exists = False\n\ntry:\n df_results = pd.read_csv('/kaggle/working/Dataframes/model_results.csv')\n df_exists = True\n print(df_results)\n print(\"df exists and was successfully loaded!\")\nexcept:\n print(\"df_results not found. Starting all computations from scratch.\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.977479Z","iopub.execute_input":"2024-09-24T09:45:13.977848Z","iopub.status.idle":"2024-09-24T09:45:13.996632Z","shell.execute_reply.started":"2024-09-24T09:45:13.977812Z","shell.execute_reply":"2024-09-24T09:45:13.995378Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's define the \"results\" dictionnary and import classifiers:","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier, \n AdaBoostClassifier, ExtraTreesClassifier, \n HistGradientBoostingClassifier, BaggingClassifier)\nfrom xgboost import XGBClassifier\nfrom lightgbm import LGBMClassifier\nfrom catboost import CatBoostClassifier\nfrom sklearn.metrics import accuracy_score, f1_score\nfrom sklearn.model_selection import train_test_split\nfrom time import time\nfrom sklearn.svm import SVC\n\n\n# Define a function to train and evaluate the models\ndef train_and_evaluate_model(model, X_train, y_train, X_test, y_test):\n t1 = time() # Start time\n model.fit(X_train, y_train) # Fit the model\n y_pred = model.predict(X_test) # Make predictions\n acc = accuracy_score(y_test, y_pred) # Accuracy metric\n f1 = f1_score(y_test, y_pred, average='weighted') # F1 score metric\n t = time() - t1 # Compute elapsed time\n print(\" Time (seconds) taken for fitting and computing metrics:\", t)\n return acc, f1, t\n\n# Reordered classifiers based on speed for Credit dataset\nmodels = {\n 'AdaBoost': AdaBoostClassifier(random_state=RANDOM_SEED),\n 'ExtraTrees': ExtraTreesClassifier(random_state=RANDOM_SEED),\n 'LightGBM': LGBMClassifier(verbose=-1, random_state=RANDOM_SEED),\n 'RandomForest': RandomForestClassifier(random_state=RANDOM_SEED),\n 'GradientBoosting': GradientBoostingClassifier(random_state=RANDOM_SEED),\n 'HistGradientBoosting': HistGradientBoostingClassifier(random_state=RANDOM_SEED),\n 'BaggingClassifier': BaggingClassifier(random_state=RANDOM_SEED),\n 'SVC': SVC()\n}\n\n# Netflix dataset (df_netflix is preprocessed as per previous sections)\nX_train_netflix, X_test_netflix, y_train_netflix, y_test_netflix = train_test_split(\n df_netflix.drop('Class', axis=1), df_netflix['Class'], test_size=0.2, random_state=RANDOM_SEED\n)\n\n# Credit card dataset (df_credit is preprocessed as per previous sections)\nX_train_credit, X_test_credit, y_train_credit, y_test_credit = train_test_split(\n df_credit.drop('Class', axis=1), df_credit['Class'], test_size=0.2, random_state=RANDOM_SEED\n)\n\nX_train_netflix = sanitize_column_names(X_train_netflix)\nX_test_netflix = sanitize_column_names(X_test_netflix)\nX_train_credit = sanitize_column_names(X_train_credit)\nX_test_credit = sanitize_column_names(X_test_credit)\n\n# Evaluate each model on all datasets\nX_train_ICR.dropna(inplace=True)\nX_test_ICR.dropna(inplace=True)\n\n# Dictionary to store results\nresults = {}\n\nprint(\"Results dictionnary was reinitialized. Think about filling it again before saving it to dataframe format!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:13.998528Z","iopub.execute_input":"2024-09-24T09:45:13.998917Z","iopub.status.idle":"2024-09-24T09:45:14.275667Z","shell.execute_reply.started":"2024-09-24T09:45:13.998880Z","shell.execute_reply":"2024-09-24T09:45:14.274307Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now, an utility function to load those results into the dictionnary \"results\":","metadata":{}},{"cell_type":"code","source":"# Function to populate results from df_results for a specific model\ndef populate_results_from_df(df, model_name):\n model_data = df[df['Model'] == model_name]\n datasets = model_data['Dataset'].unique()\n \n results[model_name] = {}\n \n for dataset in datasets:\n dataset_data = model_data[model_data['Dataset'] == dataset]\n accuracy = dataset_data['Accuracy'].values[0]\n f1_score = dataset_data['F1-score'].values[0]\n time_taken = dataset_data['Time (seconds)'].values[0]\n \n results[model_name][dataset] = {\n 'Accuracy': accuracy,\n 'F1-score': f1_score,\n 'Time': time_taken\n }","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.277105Z","iopub.execute_input":"2024-09-24T09:45:14.277500Z","iopub.status.idle":"2024-09-24T09:45:14.286709Z","shell.execute_reply.started":"2024-09-24T09:45:14.277459Z","shell.execute_reply":"2024-09-24T09:45:14.285350Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## XGBoost\n\nXGBoost is a rapid and efficient algorithm. Its fitting is a little bit different from sklearn models, so we make a separated pipeline for it.","metadata":{}},{"cell_type":"code","source":"# Check if 'XGBoost' is already in the dataframe\nif df_exists and 'XGBoost' in df_results['Model'].values:\n # Count the number of datasets where XGBoost was used\n xgboost_count = df_results[df_results['Model'] == 'XGBoost'].shape[0]\n print(f\"XGBoost has already been evaluated on {xgboost_count} datasets.\")\n print(df_results[df_results['Model'] == 'XGBoost'])\n \n populate_results_from_df(df_results, 'XGBoost')\n print(f\"Results for 'XGBoost' populated from df_results.\")\nelse:\n # If XGBoost is not in the dataframe, proceed with execution\n print(\"XGBoost is not in the results, proceeding with execution.\")\n\n # Separate execution for XGBoost\n print(\"\\nEvaluating XGBoost separately...\")\n\n # XGBoost for ICR dataset (Binary Classification)\n print(f\"Training and evaluating XGBoost on ICR dataset...\")\n model = XGBClassifier(use_label_encoder=False, eval_metric='logloss', objective='binary:logistic')\n acc_ICR, f1_ICR, t_ICR = train_and_evaluate_model(model, X_train_ICR, y_train_ICR, X_test_ICR, y_test_ICR)\n\n # XGBoost for Netflix dataset\n print(f\"Training and evaluating XGBoost on Netflix dataset...\")\n model = XGBClassifier(use_label_encoder=False, eval_metric='mlogloss')\n acc_netflix, f1_netflix, t_netflix = train_and_evaluate_model(model, X_train_netflix, y_train_netflix_encoded, X_test_netflix, y_test_netflix_encoded)\n\n # XGBoost for Credit dataset (Binary Classification)\n print(f\"Training and evaluating XGBoost on Credit dataset...\")\n model = XGBClassifier(use_label_encoder=False, eval_metric='logloss', objective='binary:logistic')\n acc_credit, f1_credit, t_credit = train_and_evaluate_model(model, X_train_credit, y_train_credit, X_test_credit, y_test_credit)\n\n # XGBoost for Spaceship Titanic dataset (Binary Classification)\n print(f\"Training and evaluating XGBoost on Spaceship Titanic dataset...\")\n model = XGBClassifier(use_label_encoder=False, eval_metric='logloss', objective='binary:logistic')\n acc_sptit, f1_sptit, t_sptit = train_and_evaluate_model(model, X_train_sptit, y_train_sptit, X_test_sptit, y_test_sptit)\n\n # XGBoost for MNIST dataset (Multi-class Classification)\n print(f\"Training and evaluating XGBoost on MNIST dataset...\")\n model = XGBClassifier(use_label_encoder=False, eval_metric='mlogloss', objective='multi:softmax')\n acc_mnist, f1_mnist, t_mnist = train_and_evaluate_model(model, X_train_mnist, y_train_mnist, X_test_mnist, y_test_mnist)\n\n # XGBoost for Chest X-ray dataset (Binary Classification)\n print(f\"Training and evaluating XGBoost on Chest X-ray dataset...\")\n model = XGBClassifier(use_label_encoder=False, eval_metric='logloss', objective='binary:logistic')\n acc_xray, f1_xray, t_xray = train_and_evaluate_model(model, X_train_xray, y_train_xray, X_test_xray, y_test_xray)\n\n # XGBoost for CIC dataset (Binary Classification)\n print(f\"Training and evaluating XGBoost on CIC dataset...\")\n model = XGBClassifier(use_label_encoder=False, eval_metric='logloss', objective='binary:logistic')\n acc_cic, f1_cic, t_cic = train_and_evaluate_model(model, X_train_CIC, y_train_CIC, X_test_CIC, y_test_CIC)\n\n # Store XGBoost results\n results[\"XGBoost\"] = {\n 'Netflix': {'Accuracy': acc_netflix, 'F1-score': f1_netflix, 'Time': t_netflix},\n 'Credit': {'Accuracy': acc_credit, 'F1-score': f1_credit, 'Time': t_credit},\n 'ICR': {'Accuracy': acc_ICR, 'F1-score': f1_ICR, 'Time': t_ICR},\n 'Spaceship Titanic': {'Accuracy': acc_sptit, 'F1-score': f1_sptit, 'Time': t_sptit},\n 'MNIST': {'Accuracy': acc_mnist, 'F1-score': f1_mnist, 'Time': t_mnist},\n 'Chest X-ray': {'Accuracy': acc_xray, 'F1-score': f1_xray, 'Time': t_xray},\n 'CIC': {'Accuracy': acc_cic, 'F1-score': f1_cic, 'Time': t_cic}\n }\n \n print(results[\"XGBoost\"])\n\nprint(\"\\nXGBoost results stored successfully.\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.288810Z","iopub.execute_input":"2024-09-24T09:45:14.289328Z","iopub.status.idle":"2024-09-24T09:45:14.323524Z","shell.execute_reply.started":"2024-09-24T09:45:14.289275Z","shell.execute_reply":"2024-09-24T09:45:14.322110Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Scikit-Learn and LightGBM\n\nMost of the models will be fitted in this section. As LightGBM doesn't need a special treatment, we can use it the same way as for scikit-learn models for fitting.","metadata":{}},{"cell_type":"code","source":"for model_name, model in tqdm(models.items()):\n print(f\"\\nTraining and evaluating {model_name} on all datasets...\")\n # Check if the model is already in the dataframe\n if df_exists and model_name in df_results['Model'].values:\n # Count the number of datasets where the model was used\n count = df_results[df_results['Model'] == model_name].shape[0]\n print(f\"{model_name} has already been evaluated on {count} datasets.\")\n print(df_results[df_results['Model'] == model_name])\n populate_results_from_df(df_results, model_name)\n print(f\"Results for {model_name} populated from df_results.\")\n else:\n # If the model is not in the dataframe, proceed with execution\n print(f\"{model_name} is not in the results, proceeding with execution.\")\n \n try:\n # Spaceship Titanic dataset\n print(f\"Training and evaluating {model_name} on Spaceship Titanic dataset...\")\n acc_sptit, f1_sptit, t_sptit = train_and_evaluate_model(model, X_train_sptit, y_train_sptit, X_test_sptit, y_test_sptit)\n except Exception as e:\n print(e)\n\n try:\n # Credit card dataset\n print(f\"Training and evaluating {model_name} on Credit Card dataset...\")\n acc_credit, f1_credit, t_credit = train_and_evaluate_model(model, X_train_credit, y_train_credit, X_test_credit, y_test_credit)\n except Exception as e:\n print(e)\n\n try:\n # Netflix dataset\n print(f\"Training and evaluating {model_name} on Netflix dataset...\")\n acc_netflix, f1_netflix, t_netflix = train_and_evaluate_model(model, X_train_netflix, y_train_netflix, X_test_netflix, y_test_netflix)\n except Exception as e:\n print(e)\n\n try:\n # ICR dataset\n print(f\"Training and evaluating {model_name} on ICR dataset...\")\n acc_ICR, f1_ICR, t_ICR = train_and_evaluate_model(model, X_train_ICR, y_train_ICR, X_test_ICR, y_test_ICR)\n except Exception as e:\n print(e)\n\n try:\n # MNIST dataset\n print(f\"Training and evaluating {model_name} on MNIST dataset...\")\n acc_mnist, f1_mnist, t_mnist = train_and_evaluate_model(model, X_train_mnist, y_train_mnist, X_test_mnist, y_test_mnist)\n except Exception as e:\n print(e)\n\n try:\n # Chest X-ray dataset (Pneumonia detection)\n print(f\"Training and evaluating {model_name} on Chest X-ray dataset...\")\n acc_xray, f1_xray, t_xray = train_and_evaluate_model(model, X_train_xray, y_train_xray, X_test_xray, y_test_xray)\n except Exception as e:\n print(e)\n\n try:\n # CIC dataset (Internet Traffic Anomaly Detection)\n print(f\"Training and evaluating {model_name} on CIC dataset...\")\n if model_name != \"SVC\":\n acc_cic, f1_cic, t_cic = train_and_evaluate_model(model, X_train_CIC, y_train_CIC, X_test_CIC, y_test_CIC)\n else:\n print(\"Different training for SVC due to its long training time for large datasets...\")\n acc_cic, f1_cic, t_cic = train_and_evaluate_model(model, X_train_CIC[:100000], y_train_CIC[:100000], X_test_CIC, y_test_CIC)\n #t_cic_real = t_cic * (X_train_CIC.shape[0] / 100000) ** 2\n #t_cic = t_cic_real\n except Exception as e:\n print(e)\n\n\n # Store results for each dataset\n results[model_name] = {\n 'Credit': {'Accuracy': acc_credit, 'F1-score': f1_credit, 'Time': t_credit},\n 'Netflix': {'Accuracy': acc_netflix, 'F1-score': f1_netflix, 'Time': t_netflix},\n 'ICR': {'Accuracy': acc_ICR, 'F1-score': f1_ICR, 'Time': t_ICR},\n 'Spaceship Titanic': {'Accuracy': acc_sptit, 'F1-score': f1_sptit, 'Time': t_sptit},\n 'MNIST': {'Accuracy': acc_mnist, 'F1-score': f1_mnist, 'Time': t_mnist},\n 'Chest X-ray': {'Accuracy': acc_xray, 'F1-score': f1_xray, 'Time': t_xray},\n 'CIC': {'Accuracy': acc_cic, 'F1-score': f1_cic, 'Time': t_cic}\n }\n\n# Print results for all models and datasets\nprint(\"\\nInitial Performance Results for Tabular Classifiers:\")\nfor model_name, result in results.items():\n print(f\"\\n{model_name} Results:\")\n for dataset, metrics in result.items():\n print(f\"{dataset} - Accuracy: {metrics['Accuracy']:.4f}, F1-score: {metrics['F1-score']:.4f}, Time: {metrics['Time']:.2f} seconds\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.325818Z","iopub.execute_input":"2024-09-24T09:45:14.326334Z","iopub.status.idle":"2024-09-24T09:45:14.420695Z","shell.execute_reply.started":"2024-09-24T09:45:14.326280Z","shell.execute_reply":"2024-09-24T09:45:14.419469Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## CatBoost\n\nCatboost lives on its own planet and needs a pipeline specially designed for it, using the `Pool` preprocessing object.","metadata":{}},{"cell_type":"code","source":"from catboost import CatBoostClassifier\nfrom sklearn.metrics import accuracy_score, f1_score\nfrom time import time\n\nmodel_name = 'CatBoost'\n# Check if the model is already in the dataframe\nif df_exists and model_name in df_results['Model'].values:\n # Count the number of datasets where the model was used\n count = df_results[df_results['Model'] == model_name].shape[0]\n print(f\"{model_name} has already been evaluated on {count} datasets.\")\n print(df_results[df_results['Model'] == model_name])\n populate_results_from_df(df_results, model_name)\n print(f\"Results for {model_name} populated from df_results.\")\n\nelse:\n # CatBoost Classifier\n catboost_model = CatBoostClassifier(silent=True)\n\n # Dictionary to store CatBoost results\n catboost_results = {}\n\n # Evaluate CatBoost on Netflix dataset\n print(\"Training and evaluating CatBoost on Netflix dataset using Pool...\")\n t1 = time()\n catboost_model.fit(pool_netflix_train)\n y_pred_netflix_cat = catboost_model.predict(pool_netflix_test)\n acc_netflix_cat = accuracy_score(y_test_netflix2, y_pred_netflix_cat)\n f1_netflix_cat = f1_score(y_test_netflix2, y_pred_netflix_cat, average='weighted')\n t_netflix_cat = time() - t1\n print(\" Time (seconds) taken for fitting and computing metrics:\", t_netflix_cat)\n\n # Evaluate CatBoost on Credit dataset\n print(\"Training and evaluating CatBoost on Credit dataset using Pool...\")\n t1 = time()\n catboost_model.fit(pool_credit_train)\n y_pred_credit_cat = catboost_model.predict(pool_credit_test)\n acc_credit_cat = accuracy_score(y_test_credit, y_pred_credit_cat)\n f1_credit_cat = f1_score(y_test_credit, y_pred_credit_cat, average='weighted')\n t_credit_cat = time() - t1\n print(\" Time (seconds) taken for fitting and computing metrics:\", t_credit_cat)\n # Evaluate CatBoost on ICR dataset\n print(\"Training and evaluating CatBoost on ICR dataset using Pool...\")\n t1 = time()\n catboost_model.fit(pool_ICR_train)\n y_pred_ICR_cat = catboost_model.predict(pool_ICR_test)\n acc_ICR_cat = accuracy_score(y_test_ICR, y_pred_ICR_cat)\n f1_ICR_cat = f1_score(y_test_ICR, y_pred_ICR_cat, average='weighted')\n t_ICR_cat = time() - t1\n print(\" Time (seconds) taken for fitting and computing metrics:\", t_ICR_cat)\n\n # Evaluate CatBoost on Spaceship Titanic dataset\n print(\"Training and evaluating CatBoost on Spaceship Titanic dataset using Pool...\")\n t1 = time()\n catboost_model.fit(pool_sptit_train)\n y_pred_sptit_cat = catboost_model.predict(pool_sptit_test)\n acc_sptit_cat = accuracy_score(y_test_sptit, y_pred_sptit_cat)\n f1_sptit_cat = f1_score(y_test_sptit, y_pred_sptit_cat, average='weighted')\n t_sptit_cat = time() - t1\n print(\" Time (seconds) taken for fitting and computing metrics:\", t_sptit_cat)\n\n # Evaluate CatBoost on MNIST dataset\n print(\"Training and evaluating CatBoost on MNIST dataset using Pool...\")\n t1 = time()\n catboost_model.fit(pool_mnist_train)\n y_pred_mnist_cat = catboost_model.predict(pool_mnist_test)\n acc_mnist_cat = accuracy_score(y_test_mnist, y_pred_mnist_cat)\n f1_mnist_cat = f1_score(y_test_mnist, y_pred_mnist_cat, average='weighted')\n t_mnist_cat = time() - t1\n print(\" Time (seconds) taken for fitting and computing metrics:\", t_mnist_cat)\n\n # Evaluate CatBoost on Chest X-ray dataset\n print(\"Training and evaluating CatBoost on Chest X-ray dataset using Pool...\")\n t1 = time()\n catboost_model.fit(pool_xray_train)\n y_pred_xray_cat = catboost_model.predict(pool_xray_test)\n acc_xray_cat = accuracy_score(y_test_xray, y_pred_xray_cat)\n f1_xray_cat = f1_score(y_test_xray, y_pred_xray_cat, average='weighted')\n t_xray_cat = time() - t1\n print(\" Time (seconds) taken for fitting and computing metrics:\", t_xray_cat)\n\n # Evaluate CatBoost on Chest CIC dataset\n print(\"Training and evaluating CatBoost on CIC dataset using Pool...\")\n t1 = time()\n catboost_model.fit(pool_cic_train)\n y_pred_cic_cat = catboost_model.predict(pool_cic_test)\n acc_cic_cat = accuracy_score(y_test_CIC, y_pred_cic_cat)\n f1_cic_cat = f1_score(y_test_CIC, y_pred_cic_cat, average='weighted')\n t_cic_cat = time() - t1\n print(\" Time (seconds) taken for fitting and computing metrics:\", t_cic_cat)\n\n # Store CatBoost results\n catboost_results['CatBoost'] = {\n 'Netflix': {'Accuracy': acc_netflix_cat, 'F1-score': f1_netflix_cat, 'Time': t_netflix_cat},\n 'Credit': {'Accuracy': acc_credit_cat, 'F1-score': f1_credit_cat, 'Time': t_credit_cat},\n 'ICR': {'Accuracy': acc_ICR_cat, 'F1-score': f1_ICR_cat, 'Time': t_ICR_cat},\n 'Spaceship Titanic': {'Accuracy': acc_sptit_cat, 'F1-score': f1_sptit_cat, 'Time': t_sptit_cat},\n 'MNIST': {'Accuracy': acc_mnist_cat, 'F1-score': f1_mnist_cat, 'Time': t_mnist_cat},\n 'Chest X-ray': {'Accuracy': acc_xray_cat, 'F1-score': f1_xray_cat, 'Time': t_xray_cat},\n 'CIC': {'Accuracy': acc_cic_cat, 'F1-score': f1_cic_cat, 'Time': t_cic_cat}\n }\n\n # Print CatBoost results\n print(\"\\nCatBoost Results:\")\n for dataset_name, metrics in catboost_results['CatBoost'].items():\n print(f\"{dataset_name} - Accuracy: {metrics['Accuracy']:.4f}, F1-score: {metrics['F1-score']:.4f}, Time: {metrics['Time']:.2f} seconds\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.422547Z","iopub.execute_input":"2024-09-24T09:45:14.423030Z","iopub.status.idle":"2024-09-24T09:45:14.457937Z","shell.execute_reply.started":"2024-09-24T09:45:14.422971Z","shell.execute_reply":"2024-09-24T09:45:14.456496Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Keras Sequential\n\nNow, a difficult challenge: creating a Sequential model for all datasets. DNNs need more preprocessing than forest models, this is why we need to add a scaler and a normalizer in the pipeline. Remember that DNNs are very sensitive to hyperparameter tuning.","metadata":{}},{"cell_type":"code","source":"import time\nimport tensorflow as tf\nfrom sklearn.metrics import accuracy_score, f1_score\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout\nfrom tensorflow.keras.optimizers import Adam\nfrom sklearn.preprocessing import StandardScaler, Normalizer\nfrom sklearn.pipeline import Pipeline\n\ntf.random.set_seed(RANDOM_SEED)\n\n# Function to create a Keras Sequential model\ndef create_sequential_classifier(input_dim, num_classes):\n model = Sequential()\n model.add(Dense(128, input_dim=input_dim, activation='relu'))\n model.add(Dropout(0.3)) # Dropout layer to prevent overfitting\n model.add(Dense(64, activation='relu'))\n model.add(Dropout(0.3)) # Dropout layer to prevent overfitting\n model.add(Dense(32, activation='relu'))\n model.add(Dropout(0.3)) # Dropout layer to prevent overfitting\n \n if num_classes == 2:\n model.add(Dense(1, activation='sigmoid'))\n model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])\n else:\n model.add(Dense(num_classes, activation='softmax'))\n model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['accuracy'])\n \n return model\n\n# Function to preprocess data, train, evaluate the model, and compute metrics\ndef train_and_evaluate(X_train, X_test, y_train, y_test, num_classes, dataset_name):\n # Preprocessing pipeline (scaling + normalization)\n pipeline = Pipeline(steps=[\n ('scaler', StandardScaler()),\n ('normalizer', Normalizer())\n ])\n \n # Preprocess the data\n X_train_scaled = pipeline.fit_transform(X_train)\n X_test_scaled = pipeline.transform(X_test)\n \n # For multi-class classification, one-hot encode the labels\n if num_classes > 2:\n y_train = tf.keras.utils.to_categorical(y_train, num_classes=num_classes)\n y_test = tf.keras.utils.to_categorical(y_test, num_classes=num_classes)\n\n # Create Sequential model\n model = create_sequential_classifier(input_dim=X_train_scaled.shape[1], num_classes=num_classes)\n \n # Record start time\n start_time = time.time()\n \n # Train the model\n model.fit(X_train_scaled, y_train, epochs=10, batch_size=32, validation_data=(X_test_scaled, y_test), verbose=1)\n\n # Record end time and compute elapsed time\n elapsed_time = time.time() - start_time\n\n # Predictions\n if num_classes == 2:\n y_pred = (model.predict(X_test_scaled) > 0.5).astype(int)\n else:\n y_pred = model.predict(X_test_scaled).argmax(axis=1)\n y_test = y_test.argmax(axis=1) # Convert one-hot back to labels for accuracy/F1 computation\n\n # Compute accuracy and F1-score\n acc = accuracy_score(y_test, y_pred)\n f1 = f1_score(y_test, y_pred, average='weighted')\n\n # Print results\n print(f\"{dataset_name} - Accuracy: {acc:.4f}, F1-score: {f1:.4f}, Time: {elapsed_time:.2f} seconds\")\n \n # Return the computed metrics\n return acc, f1, elapsed_time\n\nmodel_name = 'Keras Sequential'\n\n# Check if the model is already in the dataframe\nif df_exists and model_name in df_results['Model'].values:\n count = df_results[df_results['Model'] == model_name].shape[0]\n print(f\"{model_name} has already been evaluated on {count} datasets.\")\n print(df_results[df_results['Model'] == model_name])\n populate_results_from_df(df_results, model_name)\n print(f\"Results for {model_name} populated from df_results.\")\nelse:\n # Load each dataset and train/evaluate the model\n datasets = {\n 'Netflix': (X_train_netflix, X_test_netflix, y_train_netflix_encoded, y_test_netflix_encoded, 15),\n 'Credit': (X_train_credit, X_test_credit, y_train_credit, y_test_credit, 2),\n 'ICR': (X_train_ICR, X_test_ICR, y_train_ICR, y_test_ICR, 2),\n 'Spaceship Titanic': (X_train_sptit, X_test_sptit, y_train_sptit, y_test_sptit, 2),\n 'MNIST': (X_train_mnist, X_test_mnist, y_train_mnist, y_test_mnist, 10),\n 'Chest X-ray': (X_train_xray, X_test_xray, y_train_xray, y_test_xray, 2),\n 'CIC': (X_train_CIC, X_test_CIC, y_train_CIC, y_test_CIC, 2)\n }\n\n for dataset_name, (X_train, X_test, y_train, y_test, num_classes) in tqdm(datasets.items()):\n acc, f1, elapsed_time = train_and_evaluate(X_train, X_test, y_train, y_test, num_classes, dataset_name)\n if dataset_name == 'Credit':\n acc_credit, f1_credit, t_credit = acc, f1, elapsed_time\n elif dataset_name == 'Netflix':\n acc_netflix, f1_netflix, t_netflix = acc, f1, elapsed_time\n elif dataset_name == 'ICR':\n acc_ICR, f1_ICR, t_ICR = acc, f1, elapsed_time\n elif dataset_name == 'Spaceship Titanic':\n acc_sptit, f1_sptit, t_sptit = acc, f1, elapsed_time\n elif dataset_name == 'MNIST':\n acc_mnist, f1_mnist, t_mnist = acc, f1, elapsed_time\n elif dataset_name == 'Chest X-ray':\n acc_xray, f1_xray, t_xray = acc, f1, elapsed_time\n elif dataset_name == 'CIC':\n acc_cic, f1_cic, t_cic = acc, f1, elapsed_time\n\n # Store results for each dataset\n results[model_name] = {\n 'Netflix': {'Accuracy': acc_netflix, 'F1-score': f1_netflix, 'Time': t_netflix},\n 'Credit': {'Accuracy': acc_credit, 'F1-score': f1_credit, 'Time': t_credit},\n 'ICR': {'Accuracy': acc_ICR, 'F1-score': f1_ICR, 'Time': t_ICR},\n 'Spaceship Titanic': {'Accuracy': acc_sptit, 'F1-score': f1_sptit, 'Time': t_sptit},\n 'MNIST': {'Accuracy': acc_mnist, 'F1-score': f1_mnist, 'Time': t_mnist},\n 'Chest X-ray': {'Accuracy': acc_xray, 'F1-score': f1_xray, 'Time': t_xray},\n 'CIC': {'Accuracy': acc_cic, 'F1-score': f1_cic, 'Time': t_cic}\n }\n\n print(f\"{model_name} evaluation completed and results stored.\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.459729Z","iopub.execute_input":"2024-09-24T09:45:14.460177Z","iopub.status.idle":"2024-09-24T09:45:14.593569Z","shell.execute_reply.started":"2024-09-24T09:45:14.460137Z","shell.execute_reply":"2024-09-24T09:45:14.592328Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Save The Results","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# Convert the results dictionary into a DataFrame\ndef results_to_dataframe(results):\n data = []\n for model_name, model_results in results.items():\n # Loop through all datasets dynamically\n for dataset_name, dataset_results in model_results.items():\n data.append({\n 'Model': model_name,\n 'Dataset': dataset_name,\n 'Accuracy': dataset_results['Accuracy'],\n 'F1-score': dataset_results['F1-score'],\n 'Time (seconds)': dataset_results['Time']\n })\n \n # Create a DataFrame\n df_results = pd.DataFrame(data)\n return df_results\n\n# Transform the results into a DataFrame\ndf_results = results_to_dataframe(results)","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.595065Z","iopub.execute_input":"2024-09-24T09:45:14.595472Z","iopub.status.idle":"2024-09-24T09:45:14.606154Z","shell.execute_reply.started":"2024-09-24T09:45:14.595410Z","shell.execute_reply":"2024-09-24T09:45:14.604249Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Convert CatBoost results into a DataFrame\ndef catboost_results_to_dataframe(catboost_results):\n data = []\n for dataset_name, metrics in catboost_results['CatBoost'].items():\n data.append({\n 'Model': 'CatBoost',\n 'Dataset': dataset_name,\n 'Accuracy': metrics['Accuracy'],\n 'F1-score': metrics['F1-score'],\n 'Time (seconds)': metrics['Time']\n })\n \n # Create a DataFrame for CatBoost results\n df_catboost = pd.DataFrame(data)\n return df_catboost\n\ntry:\n # Create the CatBoost results DataFrame\n df_catboost_results = catboost_results_to_dataframe(catboost_results)\n\n # Append CatBoost results to the existing df_results DataFrame\n df_results = pd.concat([df_results, df_catboost_results], ignore_index=True)\n\n # Print updated results\n print(\"\\nUpdated Results including CatBoost:\")\n print(df_results)\nexcept Exception as e:\n print(e)","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.608305Z","iopub.execute_input":"2024-09-24T09:45:14.608724Z","iopub.status.idle":"2024-09-24T09:45:14.628190Z","shell.execute_reply.started":"2024-09-24T09:45:14.608685Z","shell.execute_reply":"2024-09-24T09:45:14.626869Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Filepath to save the results\noutput_file = OUTPUT_DATA_DIR + \"model_results.csv\"\n\n# Save the DataFrame to a CSV file\ndf_results.to_csv(output_file, index=False)\n\nprint(f\"Results saved to {output_file}\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.629620Z","iopub.execute_input":"2024-09-24T09:45:14.630061Z","iopub.status.idle":"2024-09-24T09:45:14.648800Z","shell.execute_reply.started":"2024-09-24T09:45:14.630020Z","shell.execute_reply":"2024-09-24T09:45:14.647165Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

5. Second Experiment - Optuna Optimization

\n
\n\n[Back to table of contents](#TOC)\n\nWe will apply Optuna on the datasets that do not lead to near-perfect classification (like CIC and credit card) because it is not interesting to optimize a 99% F1-score classifier.\n\n## Optimized XGBoost","metadata":{}},{"cell_type":"code","source":"try:\n df_results_optimized = pd.read_csv(\"/kaggle/working/Dataframes/model_results_optimized.csv\")\n print(\"Loaded previously saved results!\\n\")\n print(df_results_optimized)\nexcept:\n print(\"No file found at this path. The optimization will start from scratch.\")\n df_results_optimized = pd.DataFrame(columns=['Model', 'Metric', 'Value'])","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:14.650523Z","iopub.execute_input":"2024-09-24T09:45:14.650966Z","iopub.status.idle":"2024-09-24T09:45:14.666030Z","shell.execute_reply.started":"2024-09-24T09:45:14.650928Z","shell.execute_reply":"2024-09-24T09:45:14.664369Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import optuna\nimport xgboost as xgb\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import train_test_split\nimport pandas as pd\n\n# Datasets to optimize XGBoost on\ndatasets_to_optimize = {\n 'Netflix': (X_train_netflix, X_test_netflix, y_train_netflix_encoded, y_test_netflix_encoded),\n 'ICR': (X_train_ICR, X_test_ICR, y_train_ICR, y_test_ICR),\n 'Spaceship Titanic': (X_train_sptit, X_test_sptit, y_train_sptit, y_test_sptit),\n #'MNIST': (X_train_mnist, X_test_mnist, y_train_mnist, y_test_mnist),\n 'Chest X-ray': (X_train_xray, X_test_xray, y_train_xray, y_test_xray)\n}\n\n# Define an objective function for Optuna\ndef objective(trial, X_train, X_test, y_train, y_test, num_classes):\n # Define the hyperparameters to optimize\n param = {\n 'objective': 'binary:logistic' if num_classes == 2 else 'multi:softmax',\n 'num_class': num_classes if num_classes > 2 else None,\n #'tree_method': 'gpu_hist', # if you're using GPUs\n 'verbosity': 0,\n 'max_depth': trial.suggest_int('max_depth', 3, 10),\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),\n 'n_estimators': trial.suggest_int('n_estimators', 100, 1000),\n 'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),\n 'gamma': trial.suggest_float('gamma', 0, 5),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0)\n }\n \n # Train the model\n model = xgb.XGBClassifier(**param, use_label_encoder=False)\n model.fit(X_train, y_train)\n \n # Make predictions\n y_pred = model.predict(X_test)\n \n # Compute F1-score\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\nmodel_name = 'XGBoost'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n optimized_results = {}\n\n # Iterate through each dataset\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing XGBoost for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study\n study = optuna.create_study(direction='maximize')\n study.optimize(lambda trial: objective(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params = study.best_params\n\n # Train and evaluate\n model = xgb.XGBClassifier(**best_params, use_label_encoder=False)\n model.fit(X_train, y_train)\n y_pred = model.predict(X_test)\n optimized_f1 = f1_score(y_test, y_pred, average='weighted')\n\n # Compare F1 before and after\n before_f1 = df_results[\n (df_results['Model'] == 'XGBoost') & \n (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization: {before_f1:.4f}\")\n print(f\"F1-score after optimization: {optimized_f1:.4f}\\n\")\n\n optimized_results[dataset_name] = optimized_f1\n\n # Build a results DataFrame (including 'Model' column)\n df_results_optimized_xgb = pd.DataFrame.from_dict(\n optimized_results, orient='index', columns=['Optimized F1-score']\n ).reset_index().rename(columns={'index': 'Dataset'})\n df_results_optimized_xgb['Model'] = 'XGBoost'\n\n # If df_results_optimized already exists and has data, we append:\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_xgb], ignore_index=True)\n\n # Finally, save\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n print(f\"Optimized results saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T10:34:13.650913Z","iopub.execute_input":"2024-09-24T10:34:13.651394Z","iopub.status.idle":"2024-09-24T10:34:13.678030Z","shell.execute_reply.started":"2024-09-24T10:34:13.651355Z","shell.execute_reply":"2024-09-24T10:34:13.676780Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized LightGBM","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb\n\n# Function for LightGBM optimization using Optuna\ndef objective_lgb(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'objective': 'binary' if num_classes == 2 else 'multiclass',\n 'num_class': num_classes if num_classes > 2 else None,\n 'metric': 'binary_logloss' if num_classes == 2 else 'multi_logloss',\n 'boosting_type': 'gbdt',\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),\n 'max_depth': trial.suggest_int('max_depth', 3, 12),\n 'num_leaves': trial.suggest_int('num_leaves', 10, 150),\n 'min_child_samples': trial.suggest_int('min_child_samples', 20, 100),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),\n 'n_estimators': trial.suggest_int('n_estimators', 100, 1000),\n 'lambda_l1': trial.suggest_float('lambda_l1', 0, 5),\n 'lambda_l2': trial.suggest_float('lambda_l2', 0, 5),\n 'verbose': -1\n }\n\n model = lgb.LGBMClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if LightGBM results already exist\nmodel_name = 'LightGBM'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for LightGBM\n optimized_results_lgb = {}\n\n # Iterate through each dataset for LightGBM optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing LightGBM for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for LightGBM\n study_lgb = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_lgb.optimize(lambda trial: objective_lgb(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_lgb = study_lgb.best_params\n\n # Train the model using the best hyperparameters\n model_lgb = lgb.LGBMClassifier(**best_params_lgb)\n model_lgb.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_lgb = model_lgb.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_lgb = f1_score(y_test, y_pred_lgb, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_lgb[dataset_name] = optimized_f1_lgb\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_lgb = df_results[(df_results['Model'] == 'LightGBM') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (LightGBM): {before_f1_lgb:.4f}\")\n print(f\"F1-score after optimization (LightGBM): {optimized_f1_lgb:.4f}\\n\")\n\n # Build new DataFrame for LightGBM optimized\n df_results_optimized_lgb = pd.DataFrame.from_dict(\n optimized_results_lgb, orient='index', columns=['Optimized F1-score']\n ).reset_index().rename(columns={'index': 'Dataset'})\n df_results_optimized_lgb['Model'] = 'LightGBM'\n\n # Append to existing results\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_lgb], ignore_index=True)\n\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n print(f\"Optimized results (including LightGBM) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.694978Z","iopub.execute_input":"2024-09-24T09:45:14.695515Z","iopub.status.idle":"2024-09-24T09:45:14.722963Z","shell.execute_reply.started":"2024-09-24T09:45:14.695461Z","shell.execute_reply":"2024-09-24T09:45:14.721264Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized Histogram Gradient Boosting","metadata":{}},{"cell_type":"code","source":"from sklearn.experimental import enable_hist_gradient_boosting\nfrom sklearn.ensemble import HistGradientBoostingClassifier\n\n# Function for HistGradientBoostingClassifier optimization using Optuna\ndef objective_hgb(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),\n 'max_depth': trial.suggest_int('max_depth', 3, 12),\n 'max_leaf_nodes': trial.suggest_int('max_leaf_nodes', 10, 255),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 10, 100),\n 'l2_regularization': trial.suggest_float('l2_regularization', 0, 10),\n 'max_iter': trial.suggest_int('max_iter', 100, 1000),\n 'scoring': 'loss' if num_classes == 2 else 'log_loss'\n }\n\n model = HistGradientBoostingClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if HistGradientBoostingClassifier results already exist\nmodel_name = 'HistGradientBoosting'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n # Dictionary to store optimized results for HistGradientBoostingClassifier\n optimized_results_hgb = {}\n\n # Iterate through each dataset for HistGradientBoostingClassifier optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing HistGradientBoosting for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for HistGradientBoostingClassifier\n study_hgb = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_hgb.optimize(lambda trial: objective_hgb(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_hgb = study_hgb.best_params\n\n # Train the model using the best hyperparameters\n model_hgb = HistGradientBoostingClassifier(**best_params_hgb)\n model_hgb.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_hgb = model_hgb.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_hgb = f1_score(y_test, y_pred_hgb, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_hgb[dataset_name] = optimized_f1_hgb\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_hgb = df_results[(df_results['Model'] == 'HistGradientBoosting') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (HistGradientBoosting): {before_f1_hgb:.4f}\")\n print(f\"F1-score after optimization (HistGradientBoosting): {optimized_f1_hgb:.4f}\\n\")\n\n # Create a DataFrame for optimized HistGradientBoosting results\n df_results_optimized_hgb = pd.DataFrame.from_dict(\n optimized_results_hgb, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_hgb.reset_index(inplace=True)\n df_results_optimized_hgb.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_hgb['Model'] = 'HistGradientBoosting'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_hgb], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including HistGradientBoostingClassifier) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.725196Z","iopub.execute_input":"2024-09-24T09:45:14.725779Z","iopub.status.idle":"2024-09-24T09:45:14.757455Z","shell.execute_reply.started":"2024-09-24T09:45:14.725721Z","shell.execute_reply":"2024-09-24T09:45:14.756114Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized Extra Trees","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesClassifier\n\n# Function for ExtraTreesClassifier optimization using Optuna\ndef objective_extratrees(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 1000),\n 'max_depth': trial.suggest_int('max_depth', 2, 20),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 20),\n 'max_features': trial.suggest_categorical('max_features', ['auto', 'sqrt', 'log2']),\n 'random_state': RANDOM_SEED,\n 'bootstrap': trial.suggest_categorical('bootstrap', [True, False])\n }\n\n model = ExtraTreesClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if ExtraTreesClassifier results already exist\nmodel_name = 'ExtraTrees'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for ExtraTreesClassifier\n optimized_results_extratrees = {}\n\n # Iterate through each dataset for ExtraTreesClassifier optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing ExtraTreesClassifier for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for ExtraTreesClassifier\n study_extratrees = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_extratrees.optimize(lambda trial: objective_extratrees(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_extratrees = study_extratrees.best_params\n\n # Train the model using the best hyperparameters\n model_extratrees = ExtraTreesClassifier(**best_params_extratrees)\n model_extratrees.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_extratrees = model_extratrees.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_extratrees = f1_score(y_test, y_pred_extratrees, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_extratrees[dataset_name] = optimized_f1_extratrees\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_extratrees = df_results[(df_results['Model'] == 'ExtraTrees') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (ExtraTreesClassifier): {before_f1_extratrees:.4f}\")\n print(f\"F1-score after optimization (ExtraTreesClassifier): {optimized_f1_extratrees:.4f}\\n\")\n\n # Create a DataFrame for optimized ExtraTreesClassifier results\n df_results_optimized_extratrees = pd.DataFrame.from_dict(\n optimized_results_extratrees, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_extratrees.reset_index(inplace=True)\n df_results_optimized_extratrees.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_extratrees['Model'] = 'ExtraTrees'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_extratrees], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including ExtraTreesClassifier) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.759055Z","iopub.execute_input":"2024-09-24T09:45:14.759498Z","iopub.status.idle":"2024-09-24T09:45:14.780064Z","shell.execute_reply.started":"2024-09-24T09:45:14.759451Z","shell.execute_reply":"2024-09-24T09:45:14.778526Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized Bagging Classifier","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import BaggingClassifier\n\n# Function for BaggingClassifier optimization using Optuna\ndef objective_bagging(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 10, 500),\n 'max_samples': trial.suggest_float('max_samples', 0.1, 1.0),\n 'max_features': trial.suggest_float('max_features', 0.1, 1.0),\n 'bootstrap': trial.suggest_categorical('bootstrap', [True, False]),\n 'bootstrap_features': trial.suggest_categorical('bootstrap_features', [True, False]),\n 'random_state': RANDOM_SEED\n }\n\n model = BaggingClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if BaggingClassifier results already exist\nmodel_name = 'BaggingClassifier'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for BaggingClassifier\n optimized_results_bagging = {}\n\n # Iterate through each dataset for BaggingClassifier optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing BaggingClassifier for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for BaggingClassifier\n study_bagging = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_bagging.optimize(lambda trial: objective_bagging(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_bagging = study_bagging.best_params\n\n # Train the model using the best hyperparameters\n model_bagging = BaggingClassifier(**best_params_bagging)\n model_bagging.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_bagging = model_bagging.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_bagging = f1_score(y_test, y_pred_bagging, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_bagging[dataset_name] = optimized_f1_bagging\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_bagging = df_results[(df_results['Model'] == 'BaggingClassifier') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (BaggingClassifier): {before_f1_bagging:.4f}\")\n print(f\"F1-score after optimization (BaggingClassifier): {optimized_f1_bagging:.4f}\\n\")\n\n # Create a DataFrame for optimized BaggingClassifier results\n df_results_optimized_bagging = pd.DataFrame.from_dict(\n optimized_results_bagging, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_bagging.reset_index(inplace=True)\n df_results_optimized_bagging.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_bagging['Model'] = 'BaggingClassifier'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_bagging], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including BaggingClassifier) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.781705Z","iopub.execute_input":"2024-09-24T09:45:14.782117Z","iopub.status.idle":"2024-09-24T09:45:14.809091Z","shell.execute_reply.started":"2024-09-24T09:45:14.782077Z","shell.execute_reply":"2024-09-24T09:45:14.807839Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized AdaBoost","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import AdaBoostClassifier\n\n# Function for AdaBoost optimization using Optuna\ndef objective_adaboost(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 500),\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 2.0),\n 'random_state': RANDOM_SEED\n }\n\n model = AdaBoostClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if AdaBoost results already exist\nmodel_name = 'AdaBoost'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for AdaBoostClassifier\n optimized_results_adaboost = {}\n\n # Iterate through each dataset for AdaBoostClassifier optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing AdaBoostClassifier for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for AdaBoostClassifier\n study_adaboost = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_adaboost.optimize(lambda trial: objective_adaboost(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_adaboost = study_adaboost.best_params\n\n # Train the model using the best hyperparameters\n model_adaboost = AdaBoostClassifier(**best_params_adaboost)\n model_adaboost.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_adaboost = model_adaboost.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_adaboost = f1_score(y_test, y_pred_adaboost, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_adaboost[dataset_name] = optimized_f1_adaboost\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_adaboost = df_results[(df_results['Model'] == 'AdaBoost') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (AdaBoostClassifier): {before_f1_adaboost:.4f}\")\n print(f\"F1-score after optimization (AdaBoostClassifier): {optimized_f1_adaboost:.4f}\\n\")\n\n # Create a DataFrame for optimized AdaBoostClassifier results\n df_results_optimized_adaboost = pd.DataFrame.from_dict(\n optimized_results_adaboost, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_adaboost.reset_index(inplace=True)\n df_results_optimized_adaboost.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_adaboost['Model'] = 'AdaBoost'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_adaboost], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including AdaBoostClassifier) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.810758Z","iopub.execute_input":"2024-09-24T09:45:14.811184Z","iopub.status.idle":"2024-09-24T09:45:14.841970Z","shell.execute_reply.started":"2024-09-24T09:45:14.811135Z","shell.execute_reply":"2024-09-24T09:45:14.840311Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized Random Forest","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\n# Function for RandomForest optimization using Optuna\ndef objective_randomforest(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 500),\n 'max_depth': trial.suggest_int('max_depth', 2, 32),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 20),\n 'bootstrap': trial.suggest_categorical('bootstrap', [True, False]),\n 'random_state': RANDOM_SEED\n }\n\n model = RandomForestClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if RandomForest results already exist\nmodel_name = 'RandomForest'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n # Launch the Optuna study for RandomForestClassifier\n print(\"Launching Optuna study for RandomForestClassifier...\")\n\n # Dictionary to store optimized results for RandomForestClassifier\n optimized_results_randomforest = {}\n\n # Iterate through each dataset for RandomForestClassifier optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing RandomForestClassifier for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for RandomForestClassifier\n study_randomforest = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_randomforest.optimize(lambda trial: objective_randomforest(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_randomforest = study_randomforest.best_params\n\n # Train the model using the best hyperparameters\n model_randomforest = RandomForestClassifier(**best_params_randomforest)\n model_randomforest.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_randomforest = model_randomforest.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_randomforest = f1_score(y_test, y_pred_randomforest, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_randomforest[dataset_name] = optimized_f1_randomforest\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_randomforest = df_results[(df_results['Model'] == 'RandomForest') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (RandomForestClassifier): {before_f1_randomforest:.4f}\")\n print(f\"F1-score after optimization (RandomForestClassifier): {optimized_f1_randomforest:.4f}\\n\")\n\n # Create a DataFrame for optimized RandomForestClassifier results\n df_results_optimized_randomforest = pd.DataFrame.from_dict(\n optimized_results_randomforest, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_randomforest.reset_index(inplace=True)\n df_results_optimized_randomforest.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_randomforest['Model'] = 'RandomForest'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_randomforest], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including RandomForestClassifier) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.844005Z","iopub.execute_input":"2024-09-24T09:45:14.844713Z","iopub.status.idle":"2024-09-24T09:45:14.874883Z","shell.execute_reply.started":"2024-09-24T09:45:14.844642Z","shell.execute_reply":"2024-09-24T09:45:14.873364Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized Gradient Boosting","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import GradientBoostingClassifier\n\n# Function for GradientBoosting optimization using Optuna\ndef objective_gradientboosting(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 500),\n 'max_depth': trial.suggest_int('max_depth', 2, 32),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 20),\n 'learning_rate': trial.suggest_loguniform('learning_rate', 1e-4, 1),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'random_state': RANDOM_SEED\n }\n\n model = GradientBoostingClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if GradientBoosting results already exist\nmodel_name = 'GradientBoosting'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for GradientBoostingClassifier\n optimized_results_gradientboosting = {}\n\n # Iterate through each dataset for GradientBoostingClassifier optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing GradientBoostingClassifier for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for GradientBoostingClassifier\n study_gradientboosting = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_gradientboosting.optimize(lambda trial: objective_gradientboosting(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_gradientboosting = study_gradientboosting.best_params\n\n # Train the model using the best hyperparameters\n model_gradientboosting = GradientBoostingClassifier(**best_params_gradientboosting)\n model_gradientboosting.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_gradientboosting = model_gradientboosting.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_gradientboosting = f1_score(y_test, y_pred_gradientboosting, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_gradientboosting[dataset_name] = optimized_f1_gradientboosting\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_gradientboosting = df_results[(df_results['Model'] == 'GradientBoosting') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (GradientBoostingClassifier): {before_f1_gradientboosting:.4f}\")\n print(f\"F1-score after optimization (GradientBoostingClassifier): {optimized_f1_gradientboosting:.4f}\\n\")\n\n # Create a DataFrame for optimized GradientBoostingClassifier results\n df_results_optimized_gradientboosting = pd.DataFrame.from_dict(\n optimized_results_gradientboosting, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_gradientboosting.reset_index(inplace=True)\n df_results_optimized_gradientboosting.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_gradientboosting['Model'] = 'GradientBoosting'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_gradientboosting], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including GradientBoostingClassifier) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.876659Z","iopub.execute_input":"2024-09-24T09:45:14.877177Z","iopub.status.idle":"2024-09-24T09:45:14.903689Z","shell.execute_reply.started":"2024-09-24T09:45:14.877124Z","shell.execute_reply":"2024-09-24T09:45:14.902486Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized CatBoost\n\nWe need to introduce early stopping mechanisms to avoid very long optimization processes.","metadata":{}},{"cell_type":"code","source":"from catboost import CatBoostClassifier\nimport optuna\n\n# Function for CatBoost optimization using Optuna\ndef objective_catboost(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'iterations': trial.suggest_int('iterations', 50, 300), # Reduced the max number of iterations\n 'depth': trial.suggest_int('depth', 2, 10), # Reduced the maximum depth\n 'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.1), # Slightly adjusted range\n 'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1e-4, 10),\n 'border_count': trial.suggest_int('border_count', 32, 255),\n 'random_state': RANDOM_SEED,\n 'early_stopping_rounds': 50, # Early stopping after 50 rounds without improvement\n 'verbose': 0\n }\n\n model = CatBoostClassifier(**param)\n model.fit(X_train, y_train, eval_set=(X_test, y_test), verbose=0)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\n# Check if CatBoost results already exist\nmodel_name = 'CatBoost'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for CatBoostClassifier\n optimized_results_catboost = {}\n\n # Iterate through each dataset for CatBoostClassifier optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing CatBoostClassifier for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for CatBoostClassifier\n study_catboost = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_catboost.optimize(lambda trial: objective_catboost(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params_catboost = study_catboost.best_params\n\n # Train the model using the best hyperparameters\n model_catboost = CatBoostClassifier(**best_params_catboost)\n model_catboost.fit(X_train, y_train, eval_set=(X_test, y_test), verbose=0)\n\n # Make predictions with the optimized model\n y_pred_catboost = model_catboost.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_catboost = f1_score(y_test, y_pred_catboost, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_catboost[dataset_name] = optimized_f1_catboost\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_catboost = df_results[(df_results['Model'] == 'CatBoost') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (CatBoostClassifier): {before_f1_catboost:.4f}\")\n print(f\"F1-score after optimization (CatBoostClassifier): {optimized_f1_catboost:.4f}\\n\")\n\n # Create a DataFrame for optimized CatBoostClassifier results\n df_results_optimized_catboost = pd.DataFrame.from_dict(\n optimized_results_catboost, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_catboost.reset_index(inplace=True)\n df_results_optimized_catboost.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_catboost['Model'] = 'CatBoost'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_catboost], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including CatBoostClassifier) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.942411Z","iopub.execute_input":"2024-09-24T09:45:14.942942Z","iopub.status.idle":"2024-09-24T09:45:14.966927Z","shell.execute_reply.started":"2024-09-24T09:45:14.942883Z","shell.execute_reply":"2024-09-24T09:45:14.965683Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized Sequential\n","metadata":{}},{"cell_type":"code","source":"import optuna\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Dropout\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Sequential\nfrom sklearn.preprocessing import StandardScaler, Normalizer\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.metrics import f1_score\nfrom tensorflow.keras.regularizers import l2\n\n# Function to create a Keras Sequential model based on trial parameters\ndef create_keras_model(trial, input_dim, num_classes):\n model = Sequential()\n \n # Define the architecture type: funnel, uniform, etc.\n architecture = trial.suggest_categorical('architecture', ['funnel', 'uniform', 'custom'])\n\n # Define number of layers before determining sizes\n num_layers = trial.suggest_int('num_layers', 2, 6) # Choose between 2 and 4 layers\n \n if architecture == 'funnel':\n # For funnel, choose only the first layer's size and reduce by half for each subsequent layer\n first_layer_size = trial.suggest_int('units_funnel', 64, 256)\n layer_sizes = [first_layer_size // (2**i) for i in range(num_layers)] # Halve the size each time\n elif architecture == 'uniform':\n # For uniform, use the same size for all layers\n units = trial.suggest_int('units_uniform', 32, 256)\n layer_sizes = [units] * num_layers\n else: # Custom architecture\n # For custom, choose different sizes for each layer\n layer_sizes = [trial.suggest_int(f'units_custom_{i}', 16, 256) for i in range(num_layers)]\n\n # Dropout rate and L2 regularization\n dropout_rate = trial.suggest_float('dropout', 0.2, 0.5)\n l2_reg = trial.suggest_loguniform('l2_reg', 1e-5, 1e-2)\n \n # Build the Sequential model\n for units in layer_sizes:\n model.add(Dense(units, input_dim=input_dim, activation='relu', kernel_regularizer=l2(l2_reg)))\n model.add(Dropout(dropout_rate))\n \n # Output layer\n if num_classes == 2:\n model.add(Dense(1, activation='sigmoid'))\n model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])\n else:\n model.add(Dense(num_classes, activation='softmax'))\n model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['accuracy'])\n \n return model\n\n# Function to preprocess data, train and evaluate the model, and compute metrics\ndef objective_keras(trial, X_train, X_test, y_train, y_test, num_classes):\n # Preprocessing pipeline (scaling + normalization)\n pipeline = Pipeline(steps=[\n ('scaler', StandardScaler()),\n ('normalizer', Normalizer())\n ])\n \n # Preprocess the data\n X_train_scaled = pipeline.fit_transform(X_train)\n X_test_scaled = pipeline.transform(X_test)\n \n # For multi-class classification, one-hot encode the labels\n if num_classes > 2:\n y_train = tf.keras.utils.to_categorical(y_train, num_classes=num_classes)\n y_test = tf.keras.utils.to_categorical(y_test, num_classes=num_classes)\n\n # Create the Keras model using trial parameters\n model = create_keras_model(trial, input_dim=X_train_scaled.shape[1], num_classes=num_classes)\n \n # Train the model with a limited number of epochs\n model.fit(X_train_scaled, y_train, epochs=trial.suggest_int('epochs', 10, 50), batch_size=32, verbose=0, validation_data=(X_test_scaled, y_test))\n \n # Predictions\n if num_classes == 2:\n y_pred = (model.predict(X_test_scaled) > 0.5).astype(int)\n else:\n y_pred = model.predict(X_test_scaled).argmax(axis=1)\n y_test = y_test.argmax(axis=1) # Convert one-hot back to labels for accuracy/F1 computation\n\n # Compute F1 score\n f1 = f1_score(y_test, y_pred, average='weighted')\n\n return f1\n\n# Check if Keras Sequential results already exist in df_results_optimized\nmodel_name = 'Keras Sequential'\nif model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for Keras Sequential\n optimized_results_keras = {}\n\n # Iterate through each dataset for Keras Sequential optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing Keras Sequential for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for Keras Sequential\n study_keras = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_keras.optimize(lambda trial: objective_keras(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters and F1 score\n best_params_keras = study_keras.best_params\n optimized_f1_keras = study_keras.best_value\n\n # Print the best F1 score\n print(f\"Optimized F1-score (Keras Sequential) for {dataset_name}: {optimized_f1_keras:.4f}\")\n\n # Store the optimized F1-score without refitting the model\n optimized_results_keras[dataset_name] = optimized_f1_keras\n\n # Compare F1 before and after optimization (from df_results_optimized)\n before_f1_keras = df_results[(df_results['Model'] == 'Keras Sequential') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (Keras Sequential): {before_f1_keras:.4f}\")\n print(f\"F1-score after optimization (Keras Sequential): {optimized_f1_keras:.4f}\\n\")\n\n # Create a DataFrame for optimized Keras Sequential results\n df_results_optimized_keras = pd.DataFrame.from_dict(\n optimized_results_keras, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_keras.reset_index(inplace=True)\n df_results_optimized_keras.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_keras['Model'] = 'Keras Sequential'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_keras], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including Keras Sequential) saved to {output_file_optimized}\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-09-24T09:45:14.975663Z","iopub.execute_input":"2024-09-24T09:45:14.976179Z","iopub.status.idle":"2024-09-24T09:45:15.014877Z","shell.execute_reply.started":"2024-09-24T09:45:14.976112Z","shell.execute_reply":"2024-09-24T09:45:15.013530Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Optimized SVM\n\n
\n Remark: the Optuna optimization part of SVC freezes the kernel for no apparent reason. Until this problem is solved, we won't consider SVC for the optimization part.\n
","metadata":{}},{"cell_type":"code","source":"drop_svc = True","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:03:06.322521Z","iopub.execute_input":"2024-09-24T10:03:06.323073Z","iopub.status.idle":"2024-09-24T10:03:06.330405Z","shell.execute_reply.started":"2024-09-24T10:03:06.323029Z","shell.execute_reply":"2024-09-24T10:03:06.328790Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import optuna\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import f1_score\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split\n\nif drop_svc:\n print(\"Ignored.\")\nelse:\n # Function for SVM optimization using Optuna\n def objective_svm(trial, X_train, X_test, y_train, y_test, num_classes):\n # Hyperparameters to be optimized\n kernel = trial.suggest_categorical('kernel', ['linear', 'rbf', 'poly', 'sigmoid'])\n\n param = {\n 'C': trial.suggest_loguniform('C', 1e-3, 1e2),\n 'kernel': kernel,\n 'gamma': trial.suggest_categorical('gamma', ['scale', 'auto']),\n 'degree': trial.suggest_int('degree', 2, 5) if kernel == 'poly' else 3, # Only optimize 'degree' if kernel is 'poly'\n 'random_state': RANDOM_SEED\n }\n\n # SVM Model with preprocessing pipeline\n model = Pipeline([\n ('scaler', StandardScaler()), # Scale features\n ('svc', SVC(**param)) # Apply SVM with the chosen parameters\n ])\n\n # Train the model\n model.fit(X_train, y_train)\n\n # Make predictions\n y_pred = model.predict(X_test)\n\n # Compute the F1 score\n f1 = f1_score(y_test, y_pred, average='weighted')\n\n return f1\n\n # Check if SVM results already exist\n model_name = 'SVC'\n if model_name in df_results_optimized['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_optimized[df_results_optimized['Model'] == model_name])\n else:\n print(f\"Launching Optuna study for {model_name}...\")\n\n # Dictionary to store optimized results for SVM\n optimized_results_svm = {}\n\n # Iterate through each dataset for SVM optimization\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n print(f\"Optimizing SVM for {dataset_name} dataset...\")\n\n num_classes = len(set(y_train))\n\n # Create an Optuna study for SVM\n study_svm = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study_svm.optimize(lambda trial: objective_svm(trial, X_train, X_test, y_train, y_test, num_classes), \n n_trials=OPTUNA_TRIALS, timeout=60) # 1 minute per trial\n\n\n # Best hyperparameters\n best_params_svm = study_svm.best_params\n\n # Train the model using the best hyperparameters\n model_svm = Pipeline([\n ('scaler', StandardScaler()),\n ('svc', SVC(**best_params_svm))\n ])\n model_svm.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred_svm = model_svm.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1_svm = f1_score(y_test, y_pred_svm, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results_svm[dataset_name] = optimized_f1_svm\n\n # Compare F1 before and after optimization (from df_results)\n before_f1_svm = df_results[(df_results['Model'] == 'SVC') & (df_results['Dataset'] == dataset_name)]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (SVM): {before_f1_svm:.4f}\")\n print(f\"F1-score after optimization (SVM): {optimized_f1_svm:.4f}\\n\")\n\n # Create a DataFrame for optimized SVM results\n df_results_optimized_svm = pd.DataFrame.from_dict(\n optimized_results_svm, \n orient='index', \n columns=['Optimized F1-score']\n )\n df_results_optimized_svm.reset_index(inplace=True)\n df_results_optimized_svm.rename(columns={'index': 'Dataset'}, inplace=True)\n df_results_optimized_svm['Model'] = 'SVC'\n \n # Merge with existing df_results_optimized\n df_results_optimized = pd.concat([df_results_optimized, df_results_optimized_svm], ignore_index=True)\n \n # Save the updated results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_optimized.csv\"\n df_results_optimized.to_csv(output_file_optimized, index=False)\n \n print(f\"Optimized results (including SVC) saved to {output_file_optimized}\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:03:07.996571Z","iopub.execute_input":"2024-09-24T10:03:07.997011Z","iopub.status.idle":"2024-09-24T10:03:08.018296Z","shell.execute_reply.started":"2024-09-24T10:03:07.996969Z","shell.execute_reply":"2024-09-24T10:03:08.017102Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_results_optimized.fillna(\"XGBoost\", inplace=True)\ndf_results_optimized.to_csv(OUTPUT_DATA_DIR + \"optuna_backup.csv\", index=False)\ndf_results_optimized","metadata":{"execution":{"iopub.status.busy":"2024-09-24T09:45:15.037691Z","iopub.execute_input":"2024-09-24T09:45:15.038271Z","iopub.status.idle":"2024-09-24T09:45:15.066775Z","shell.execute_reply.started":"2024-09-24T09:45:15.038217Z","shell.execute_reply":"2024-09-24T09:45:15.065505Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

6. Third Experiment - Data Dependance

\n
\n\n[Back to table of contents](#TOC)\n\n\nThe idea is to significantly reduce the amount of data available and then to make tests again. In some ML problems, we can face situations with only a 100 rows available, or even less. It is important to know what models to use then.","metadata":{}},{"cell_type":"code","source":"# Count and print the number of rows and columns for each training set\nprint(f\"ICR dataset - Training set: {X_train_ICR.shape[0]} rows, {X_train_ICR.shape[1]} columns\")\nprint(f\"Netflix dataset - Training set: {X_train_netflix.shape[0]} rows, {X_train_netflix.shape[1]} columns\")\nprint(f\"Credit dataset - Training set: {X_train_credit.shape[0]} rows, {X_train_credit.shape[1]} columns\")\nprint(f\"Spaceship Titanic dataset - Training set: {X_train_sptit.shape[0]} rows, {X_train_sptit.shape[1]} columns\")\nprint(f\"MNIST dataset - Training set: {X_train_mnist.shape[0]} rows, {X_train_mnist.shape[1]} columns\")\nprint(f\"Chest X-ray dataset - Training set: {X_train_xray.shape[0]} rows, {X_train_xray.shape[1]} columns\")\nprint(f\"CIC dataset - Training set: {X_train_CIC.shape[0]} rows, {X_train_CIC.shape[1]} columns\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:11:53.856154Z","iopub.execute_input":"2024-09-24T10:11:53.856689Z","iopub.status.idle":"2024-09-24T10:11:53.867553Z","shell.execute_reply.started":"2024-09-24T10:11:53.856647Z","shell.execute_reply":"2024-09-24T10:11:53.865927Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We will limit the dataset to 100 rows:","metadata":{}},{"cell_type":"code","source":"# Saving the current datasets to files so they can be reloaded later\nX_train_ICR.to_csv(OUTPUT_DATA_DIR + \"X_train_ICR_full.csv\", index=False)\ny_train_ICR.to_csv(OUTPUT_DATA_DIR + \"y_train_ICR_full.csv\", index=False)\n\nX_train_netflix.to_csv(OUTPUT_DATA_DIR + \"X_train_netflix_full.csv\", index=False)\nnp.savetxt(OUTPUT_DATA_DIR + \"y_train_netflix_full.csv\", y_train_netflix_encoded, delimiter=\",\")\n\nX_train_credit.to_csv(OUTPUT_DATA_DIR + \"X_train_credit_full.csv\", index=False)\ny_train_credit.to_csv(OUTPUT_DATA_DIR + \"y_train_credit_full.csv\", index=False)\n\nX_train_sptit.to_csv(OUTPUT_DATA_DIR + \"X_train_sptit_full.csv\", index=False)\ny_train_sptit.to_csv(OUTPUT_DATA_DIR + \"y_train_sptit_full.csv\", index=False)\n\nX_train_mnist.to_csv(OUTPUT_DATA_DIR + \"X_train_mnist_full.csv\", index=False)\ny_train_mnist.to_csv(OUTPUT_DATA_DIR + \"y_train_mnist_full.csv\", index=False)\n\nX_train_xray = pd.DataFrame(X_train_xray)\ny_train_xray = pd.DataFrame(y_train_xray)\nX_train_xray.to_csv(OUTPUT_DATA_DIR + \"X_train_xray_full.csv\", index=False)\ny_train_xray.to_csv(OUTPUT_DATA_DIR + \"y_train_xray_full.csv\", index=False)\n\nX_train_CIC.to_csv(OUTPUT_DATA_DIR + \"X_train_CIC_full.csv\", index=False)\ny_train_CIC.to_csv(OUTPUT_DATA_DIR + \"y_train_CIC_full.csv\", index=False)\n\n# Now pruning datasets to the first 100 rows\nX_train_ICR = X_train_ICR.head(100)\ny_train_ICR = y_train_ICR.head(100)\n\nX_train_netflix = X_train_netflix.head(100)\ny_train_netflix_encoded = y_train_netflix_encoded[:100]\n\nX_train_credit = X_train_credit.head(100)\ny_train_credit = y_train_credit.head(100)\n\nX_train_sptit = X_train_sptit.head(100)\ny_train_sptit = y_train_sptit.head(100)\n\nX_train_mnist = X_train_mnist.head(100)\ny_train_mnist = y_train_mnist.head(100)\n\nX_train_xray = X_train_xray.head(100)\ny_train_xray = y_train_xray.head(100)\n\nX_train_CIC = X_train_CIC.head(100)\ny_train_CIC = y_train_CIC.head(100)\n\nprint(\"Completed!\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:26:51.988713Z","iopub.execute_input":"2024-09-24T10:26:51.989360Z","iopub.status.idle":"2024-09-24T10:28:28.734434Z","shell.execute_reply.started":"2024-09-24T10:26:51.989306Z","shell.execute_reply":"2024-09-24T10:28:28.733207Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now let's run the optimizations again. We will remove Netflix from the list because it is too complex for a 100-rows problem.\n\n## Optuna Optimization over Small Datasets","metadata":{}},{"cell_type":"code","source":"try:\n df_results_tiny = pd.read_csv(\"/kaggle/working/Dataframes/model_results_tiny.csv\")\n print(\"Loaded previously saved results!\\n\")\n print(df_results_tiny)\nexcept:\n df_results_tiny = pd.DataFrame(columns=['Model'])\n print(\"No file found at this path. The optimization will start from scratch.\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T11:14:49.903969Z","iopub.execute_input":"2024-09-24T11:14:49.904419Z","iopub.status.idle":"2024-09-24T11:14:49.914479Z","shell.execute_reply.started":"2024-09-24T11:14:49.904379Z","shell.execute_reply":"2024-09-24T11:14:49.912930Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import optuna\nimport xgboost as xgb\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import train_test_split\nimport pandas as pd\n\n# Datasets to optimize XGBoost on\ndatasets_to_optimize = {\n #'Netflix': (X_train_netflix, X_test_netflix, y_train_netflix_encoded, y_test_netflix_encoded),\n 'ICR': (X_train_ICR, X_test_ICR, y_train_ICR, y_test_ICR),\n 'Spaceship Titanic': (X_train_sptit, X_test_sptit, y_train_sptit, y_test_sptit),\n 'MNIST': (X_train_mnist, X_test_mnist, y_train_mnist, y_test_mnist),\n 'Chest X-ray': (X_train_xray, X_test_xray, y_train_xray, y_test_xray),\n 'Credit': (X_train_credit, X_test_credit, y_train_credit, y_test_credit),\n 'CIC': (X_train_CIC, X_test_CIC, y_train_CIC, y_test_CIC)\n}\n\n\n# --------------------- XGBoost Objective ---------------------\ndef objective(trial, X_train, X_test, y_train, y_test, num_classes):\n # Define the hyperparameters to optimize\n param = {\n 'objective': 'binary:logistic' if num_classes == 2 else 'multi:softmax',\n 'num_class': num_classes if num_classes > 2 else None,\n 'verbosity': 0,\n 'max_depth': trial.suggest_int('max_depth', 3, 10),\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),\n 'n_estimators': trial.suggest_int('n_estimators', 100, 1000),\n 'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),\n 'gamma': trial.suggest_float('gamma', 0, 5),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0)\n }\n \n # Train the model\n model = xgb.XGBClassifier(**param, use_label_encoder=False)\n model.fit(X_train, y_train)\n \n # Make predictions\n y_pred = model.predict(X_test)\n \n # Compute F1-score\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'XGBoost'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n optimized_results = {}\n\n # Iterate through each dataset\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n # --- NEW CHECK: Ensure at least 2 classes in y_train ---\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing XGBoost for {dataset_name} dataset with {num_classes} unique classes...\")\n\n # Create an Optuna study\n study = optuna.create_study(direction='maximize')\n\n # Optimize the study for 100 iterations\n study.optimize(lambda trial: objective(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n # Best hyperparameters\n best_params = study.best_params\n\n # Train the model using the best hyperparameters\n model = xgb.XGBClassifier(**best_params, use_label_encoder=False)\n model.fit(X_train, y_train)\n\n # Make predictions with the optimized model\n y_pred = model.predict(X_test)\n\n # Compute optimized F1-score\n optimized_f1 = f1_score(y_test, y_pred, average='weighted')\n\n # Store the optimized F1-score in the dictionary\n optimized_results[dataset_name] = optimized_f1\n\n # Compare F1 before and after optimization (from df_results)\n before_f1 = df_results[\n (df_results['Model'] == 'XGBoost') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization: {before_f1:.4f}\")\n print(f\"F1-score after optimization: {optimized_f1:.4f}\\n\")\n\n # Create a DataFrame for optimized results\n df_tmp = pd.DataFrame.from_dict(optimized_results, orient='index', columns=['Optimized F1-score'])\n df_tmp.reset_index(inplace=True)\n df_tmp.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp['Model'] = 'XGBoost'\n\n # Merge or concatenate with df_results_tiny\n df_results_tiny = pd.concat([df_results_tiny, df_tmp], ignore_index=True)\n\n # Save optimized results to CSV\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results saved to {output_file_optimized}\")\n\n\n# --------------------- LightGBM Objective ---------------------\nimport lightgbm as lgb\n\ndef objective_lgb(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'objective': 'binary' if num_classes == 2 else 'multiclass',\n 'num_class': num_classes if num_classes > 2 else None,\n 'metric': 'binary_logloss' if num_classes == 2 else 'multi_logloss',\n 'boosting_type': 'gbdt',\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),\n 'max_depth': trial.suggest_int('max_depth', 3, 12),\n 'num_leaves': trial.suggest_int('num_leaves', 10, 150),\n 'min_child_samples': trial.suggest_int('min_child_samples', 20, 100),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),\n 'n_estimators': trial.suggest_int('n_estimators', 100, 1000),\n 'lambda_l1': trial.suggest_float('lambda_l1', 0, 5),\n 'lambda_l2': trial.suggest_float('lambda_l2', 0, 5),\n 'verbose': -1\n }\n\n model = lgb.LGBMClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n \n return f1\n\nmodel_name = 'LightGBM'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n optimized_results_lgb = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing LightGBM for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_lgb = optuna.create_study(direction='maximize')\n study_lgb.optimize(lambda trial: objective_lgb(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n best_params_lgb = study_lgb.best_params\n model_lgb = lgb.LGBMClassifier(**best_params_lgb)\n model_lgb.fit(X_train, y_train)\n\n y_pred_lgb = model_lgb.predict(X_test)\n optimized_f1_lgb = f1_score(y_test, y_pred_lgb, average='weighted')\n optimized_results_lgb[dataset_name] = optimized_f1_lgb\n\n before_f1_lgb = df_results[\n (df_results['Model'] == 'LightGBM') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (LightGBM): {before_f1_lgb:.4f}\")\n print(f\"F1-score after optimization (LightGBM): {optimized_f1_lgb:.4f}\\n\")\n\n df_tmp_lgb = pd.DataFrame.from_dict(optimized_results_lgb, orient='index', columns=['Optimized F1-score'])\n df_tmp_lgb.reset_index(inplace=True)\n df_tmp_lgb.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_lgb['Model'] = 'LightGBM'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_lgb], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including LightGBM) saved to {output_file_optimized}\")\n\n\n# --------------------- HistGradientBoostingClassifier ---------------------\nfrom sklearn.experimental import enable_hist_gradient_boosting\nfrom sklearn.ensemble import HistGradientBoostingClassifier\n\ndef objective_hgb(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),\n 'max_depth': trial.suggest_int('max_depth', 3, 12),\n 'max_leaf_nodes': trial.suggest_int('max_leaf_nodes', 10, 255),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 10, 100),\n 'l2_regularization': trial.suggest_float('l2_regularization', 0, 10),\n 'max_iter': trial.suggest_int('max_iter', 100, 1000),\n 'scoring': 'loss' if num_classes == 2 else 'log_loss'\n }\n\n model = HistGradientBoostingClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'HistGradientBoosting'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n optimized_results_hgb = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing HistGradientBoosting for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_hgb = optuna.create_study(direction='maximize')\n study_hgb.optimize(lambda trial: objective_hgb(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n best_params_hgb = study_hgb.best_params\n model_hgb = HistGradientBoostingClassifier(**best_params_hgb)\n model_hgb.fit(X_train, y_train)\n\n y_pred_hgb = model_hgb.predict(X_test)\n optimized_f1_hgb = f1_score(y_test, y_pred_hgb, average='weighted')\n optimized_results_hgb[dataset_name] = optimized_f1_hgb\n\n before_f1_hgb = df_results[\n (df_results['Model'] == 'HistGradientBoosting') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (HistGradientBoosting): {before_f1_hgb:.4f}\")\n print(f\"F1-score after optimization (HistGradientBoosting): {optimized_f1_hgb:.4f}\\n\")\n\n df_tmp_hgb = pd.DataFrame.from_dict(optimized_results_hgb, orient='index', columns=['Optimized F1-score'])\n df_tmp_hgb.reset_index(inplace=True)\n df_tmp_hgb.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_hgb['Model'] = 'HistGradientBoosting'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_hgb], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including HistGradientBoostingClassifier) saved to {output_file_optimized}\")\n\n\n# --------------------- ExtraTreesClassifier ---------------------\nfrom sklearn.ensemble import ExtraTreesClassifier\n\ndef objective_extratrees(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 1000),\n 'max_depth': trial.suggest_int('max_depth', 2, 20),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 20),\n 'max_features': trial.suggest_categorical('max_features', ['auto', 'sqrt', 'log2']),\n 'random_state': RANDOM_SEED,\n 'bootstrap': trial.suggest_categorical('bootstrap', [True, False])\n }\n\n model = ExtraTreesClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'ExtraTrees'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n optimized_results_extratrees = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing ExtraTreesClassifier for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_extratrees = optuna.create_study(direction='maximize')\n study_extratrees.optimize(lambda trial: objective_extratrees(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n best_params_extratrees = study_extratrees.best_params\n model_extratrees = ExtraTreesClassifier(**best_params_extratrees)\n model_extratrees.fit(X_train, y_train)\n\n y_pred_extratrees = model_extratrees.predict(X_test)\n optimized_f1_extratrees = f1_score(y_test, y_pred_extratrees, average='weighted')\n optimized_results_extratrees[dataset_name] = optimized_f1_extratrees\n\n before_f1_extratrees = df_results[\n (df_results['Model'] == 'ExtraTrees') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (ExtraTreesClassifier): {before_f1_extratrees:.4f}\")\n print(f\"F1-score after optimization (ExtraTreesClassifier): {optimized_f1_extratrees:.4f}\\n\")\n\n df_tmp_extratrees = pd.DataFrame.from_dict(optimized_results_extratrees, orient='index', columns=['Optimized F1-score'])\n df_tmp_extratrees.reset_index(inplace=True)\n df_tmp_extratrees.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_extratrees['Model'] = 'ExtraTrees'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_extratrees], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including ExtraTreesClassifier) saved to {output_file_optimized}\")\n\n\n# --------------------- BaggingClassifier ---------------------\nfrom sklearn.ensemble import BaggingClassifier\n\ndef objective_bagging(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 10, 500),\n 'max_samples': trial.suggest_float('max_samples', 0.1, 1.0),\n 'max_features': trial.suggest_float('max_features', 0.1, 1.0),\n 'bootstrap': trial.suggest_categorical('bootstrap', [True, False]),\n 'bootstrap_features': trial.suggest_categorical('bootstrap_features', [True, False]),\n 'random_state': RANDOM_SEED\n }\n\n model = BaggingClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'BaggingClassifier'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n optimized_results_bagging = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing BaggingClassifier for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_bagging = optuna.create_study(direction='maximize')\n study_bagging.optimize(lambda trial: objective_bagging(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n best_params_bagging = study_bagging.best_params\n model_bagging = BaggingClassifier(**best_params_bagging)\n model_bagging.fit(X_train, y_train)\n\n y_pred_bagging = model_bagging.predict(X_test)\n optimized_f1_bagging = f1_score(y_test, y_pred_bagging, average='weighted')\n optimized_results_bagging[dataset_name] = optimized_f1_bagging\n\n before_f1_bagging = df_results[\n (df_results['Model'] == 'BaggingClassifier') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (BaggingClassifier): {before_f1_bagging:.4f}\")\n print(f\"F1-score after optimization (BaggingClassifier): {optimized_f1_bagging:.4f}\\n\")\n\n df_tmp_bagging = pd.DataFrame.from_dict(optimized_results_bagging, orient='index', columns=['Optimized F1-score'])\n df_tmp_bagging.reset_index(inplace=True)\n df_tmp_bagging.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_bagging['Model'] = 'BaggingClassifier'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_bagging], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including BaggingClassifier) saved to {output_file_optimized}\")\n\n\n# --------------------- AdaBoostClassifier ---------------------\nfrom sklearn.ensemble import AdaBoostClassifier\n\ndef objective_adaboost(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 500),\n 'learning_rate': trial.suggest_float('learning_rate', 0.01, 2.0),\n 'random_state': RANDOM_SEED\n }\n\n model = AdaBoostClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'AdaBoost'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n optimized_results_adaboost = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing AdaBoostClassifier for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_adaboost = optuna.create_study(direction='maximize')\n study_adaboost.optimize(lambda trial: objective_adaboost(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n best_params_adaboost = study_adaboost.best_params\n model_adaboost = AdaBoostClassifier(**best_params_adaboost)\n model_adaboost.fit(X_train, y_train)\n\n y_pred_adaboost = model_adaboost.predict(X_test)\n optimized_f1_adaboost = f1_score(y_test, y_pred_adaboost, average='weighted')\n optimized_results_adaboost[dataset_name] = optimized_f1_adaboost\n\n before_f1_adaboost = df_results[\n (df_results['Model'] == 'AdaBoost') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (AdaBoostClassifier): {before_f1_adaboost:.4f}\")\n print(f\"F1-score after optimization (AdaBoostClassifier): {optimized_f1_adaboost:.4f}\\n\")\n\n df_tmp_adaboost = pd.DataFrame.from_dict(optimized_results_adaboost, orient='index', columns=['Optimized F1-score'])\n df_tmp_adaboost.reset_index(inplace=True)\n df_tmp_adaboost.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_adaboost['Model'] = 'AdaBoost'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_adaboost], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including AdaBoostClassifier) saved to {output_file_optimized}\")\n\n\n# --------------------- RandomForestClassifier ---------------------\nfrom sklearn.ensemble import RandomForestClassifier\n\ndef objective_randomforest(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 500),\n 'max_depth': trial.suggest_int('max_depth', 2, 32),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 20),\n 'bootstrap': trial.suggest_categorical('bootstrap', [True, False]),\n 'random_state': RANDOM_SEED\n }\n\n model = RandomForestClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'RandomForest'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n optimized_results_randomforest = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing RandomForestClassifier for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_randomforest = optuna.create_study(direction='maximize')\n study_randomforest.optimize(lambda trial: objective_randomforest(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n best_params_randomforest = study_randomforest.best_params\n model_randomforest = RandomForestClassifier(**best_params_randomforest)\n model_randomforest.fit(X_train, y_train)\n\n y_pred_randomforest = model_randomforest.predict(X_test)\n optimized_f1_randomforest = f1_score(y_test, y_pred_randomforest, average='weighted')\n optimized_results_randomforest[dataset_name] = optimized_f1_randomforest\n\n before_f1_randomforest = df_results[\n (df_results['Model'] == 'RandomForest') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (RandomForestClassifier): {before_f1_randomforest:.4f}\")\n print(f\"F1-score after optimization (RandomForestClassifier): {optimized_f1_randomforest:.4f}\\n\")\n\n df_tmp_rf = pd.DataFrame.from_dict(optimized_results_randomforest, orient='index', columns=['Optimized F1-score'])\n df_tmp_rf.reset_index(inplace=True)\n df_tmp_rf.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_rf['Model'] = 'RandomForest'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_rf], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including RandomForestClassifier) saved to {output_file_optimized}\")\n\n\n# --------------------- GradientBoostingClassifier ---------------------\nfrom sklearn.ensemble import GradientBoostingClassifier\n\ndef objective_gradientboosting(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'n_estimators': trial.suggest_int('n_estimators', 50, 500),\n 'max_depth': trial.suggest_int('max_depth', 2, 32),\n 'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),\n 'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 20),\n 'learning_rate': trial.suggest_loguniform('learning_rate', 1e-4, 1),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'random_state': RANDOM_SEED\n }\n\n model = GradientBoostingClassifier(**param)\n model.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'GradientBoosting'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n optimized_results_gradientboosting = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing GradientBoostingClassifier for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_gradientboosting = optuna.create_study(direction='maximize')\n study_gradientboosting.optimize(\n lambda trial: objective_gradientboosting(trial, X_train, X_test, y_train, y_test, num_classes),\n n_trials=OPTUNA_TRIALS\n )\n\n best_params_gradientboosting = study_gradientboosting.best_params\n model_gradientboosting = GradientBoostingClassifier(**best_params_gradientboosting)\n model_gradientboosting.fit(X_train, y_train)\n\n y_pred_gradientboosting = model_gradientboosting.predict(X_test)\n optimized_f1_gradientboosting = f1_score(y_test, y_pred_gradientboosting, average='weighted')\n optimized_results_gradientboosting[dataset_name] = optimized_f1_gradientboosting\n\n before_f1_gradientboosting = df_results[\n (df_results['Model'] == 'GradientBoosting') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (GradientBoostingClassifier): {before_f1_gradientboosting:.4f}\")\n print(f\"F1-score after optimization (GradientBoostingClassifier): {optimized_f1_gradientboosting:.4f}\\n\")\n\n df_tmp_gb = pd.DataFrame.from_dict(optimized_results_gradientboosting, orient='index', columns=['Optimized F1-score'])\n df_tmp_gb.reset_index(inplace=True)\n df_tmp_gb.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_gb['Model'] = 'GradientBoosting'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_gb], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including GradientBoostingClassifier) saved to {output_file_optimized}\")\n\n\n# --------------------- CatBoostClassifier ---------------------\nfrom catboost import CatBoostClassifier\n\ndef objective_catboost(trial, X_train, X_test, y_train, y_test, num_classes):\n param = {\n 'iterations': trial.suggest_int('iterations', 50, 300),\n 'depth': trial.suggest_int('depth', 2, 10),\n 'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.1),\n 'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1e-4, 10),\n 'border_count': trial.suggest_int('border_count', 32, 255),\n 'random_state': RANDOM_SEED,\n 'early_stopping_rounds': 50,\n 'verbose': 0\n }\n\n model = CatBoostClassifier(**param)\n model.fit(X_train, y_train, eval_set=(X_test, y_test), verbose=0)\n\n y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'CatBoost'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n optimized_results_catboost = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n \n print(f\"Optimizing CatBoostClassifier for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_catboost = optuna.create_study(direction='maximize')\n study_catboost.optimize(lambda trial: objective_catboost(trial, X_train, X_test, y_train, y_test, num_classes), n_trials=OPTUNA_TRIALS)\n\n best_params_catboost = study_catboost.best_params\n model_catboost = CatBoostClassifier(**best_params_catboost)\n model_catboost.fit(X_train, y_train, eval_set=(X_test, y_test), verbose=0)\n\n y_pred_catboost = model_catboost.predict(X_test)\n optimized_f1_catboost = f1_score(y_test, y_pred_catboost, average='weighted')\n optimized_results_catboost[dataset_name] = optimized_f1_catboost\n\n before_f1_catboost = df_results[\n (df_results['Model'] == 'CatBoost') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (CatBoostClassifier): {before_f1_catboost:.4f}\")\n print(f\"F1-score after optimization (CatBoostClassifier): {optimized_f1_catboost:.4f}\\n\")\n\n df_tmp_cb = pd.DataFrame.from_dict(optimized_results_catboost, orient='index', columns=['Optimized F1-score'])\n df_tmp_cb.reset_index(inplace=True)\n df_tmp_cb.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_cb['Model'] = 'CatBoost'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_cb], ignore_index=True)\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including CatBoostClassifier) saved to {output_file_optimized}\")\n\n\n# --------------------- Keras Sequential ---------------------\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Dropout\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Sequential\nfrom sklearn.preprocessing import StandardScaler, Normalizer\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.metrics import f1_score\nfrom tensorflow.keras.regularizers import l2\n\ndef create_keras_model(trial, input_dim, num_classes):\n model = Sequential()\n \n # Architecture selection\n architecture = trial.suggest_categorical('architecture', ['funnel', 'uniform', 'custom'])\n num_layers = trial.suggest_int('num_layers', 2, 6)\n \n if architecture == 'funnel':\n first_layer_size = trial.suggest_int('units_funnel', 64, 256)\n layer_sizes = [first_layer_size // (2**i) for i in range(num_layers)]\n elif architecture == 'uniform':\n units = trial.suggest_int('units_uniform', 32, 256)\n layer_sizes = [units] * num_layers\n else:\n layer_sizes = [trial.suggest_int(f'units_custom_{i}', 16, 256) for i in range(num_layers)]\n\n dropout_rate = trial.suggest_float('dropout', 0.2, 0.5)\n l2_reg = trial.suggest_loguniform('l2_reg', 1e-5, 1e-2)\n \n for i, units in enumerate(layer_sizes):\n # For the first layer, specify input_dim; for subsequent layers, Keras infers automatically\n if i == 0:\n model.add(Dense(units, input_dim=input_dim, activation='relu', kernel_regularizer=l2(l2_reg)))\n else:\n model.add(Dense(units, activation='relu', kernel_regularizer=l2(l2_reg)))\n model.add(Dropout(dropout_rate))\n \n if num_classes == 2:\n model.add(Dense(1, activation='sigmoid'))\n model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])\n else:\n model.add(Dense(num_classes, activation='softmax'))\n model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['accuracy'])\n \n return model\n\ndef objective_keras(trial, X_train, X_test, y_train, y_test, num_classes):\n pipeline = Pipeline(steps=[\n ('scaler', StandardScaler()),\n ('normalizer', Normalizer())\n ])\n \n X_train_scaled = pipeline.fit_transform(X_train)\n X_test_scaled = pipeline.transform(X_test)\n\n # One-hot encode y if multi-class\n if num_classes > 2:\n y_train = tf.keras.utils.to_categorical(y_train, num_classes=num_classes)\n y_test = tf.keras.utils.to_categorical(y_test, num_classes=num_classes)\n\n model = create_keras_model(trial, input_dim=X_train_scaled.shape[1], num_classes=num_classes)\n\n epochs = trial.suggest_int('epochs', 10, 50)\n model.fit(X_train_scaled, y_train, epochs=epochs, batch_size=32, verbose=0, validation_data=(X_test_scaled, y_test))\n\n if num_classes == 2:\n y_pred = (model.predict(X_test_scaled) > 0.5).astype(int)\n y_pred = y_pred.flatten() # ensure shape is correct for f1_score\n # y_test is still the original binary array\n else:\n # For multi-class, predictions are the argmax\n y_pred = model.predict(X_test_scaled).argmax(axis=1)\n y_test = y_test.argmax(axis=1)\n\n f1 = f1_score(y_test, y_pred, average='weighted')\n return f1\n\nmodel_name = 'Keras Sequential'\nif model_name in df_results_tiny['Model'].values:\n print(f\"{model_name} results already exist in df_results:\")\n print(df_results_tiny[df_results_tiny['Model'] == model_name])\nelse:\n print(f\"Launching Optuna study for {model_name}...\")\n\n optimized_results_keras = {}\n\n for dataset_name, (X_train, X_test, y_train, y_test) in datasets_to_optimize.items():\n num_classes = len(set(y_train))\n if num_classes < 2:\n print(f\"Skipping dataset '{dataset_name}' because it has {num_classes} unique class(es).\")\n continue\n\n print(f\"Optimizing Keras Sequential for {dataset_name} dataset with {num_classes} unique classes...\")\n\n study_keras = optuna.create_study(direction='maximize')\n study_keras.optimize(\n lambda trial: objective_keras(trial, X_train, X_test, y_train, y_test, num_classes),\n n_trials=OPTUNA_TRIALS\n )\n\n # Best F1 score directly from study\n optimized_f1_keras = study_keras.best_value\n optimized_results_keras[dataset_name] = optimized_f1_keras\n\n before_f1_keras = df_results[\n (df_results['Model'] == 'Keras Sequential') & (df_results['Dataset'] == dataset_name)\n ]['F1-score'].values[0]\n\n print(f\"Dataset: {dataset_name}\")\n print(f\"F1-score before optimization (Keras Sequential): {before_f1_keras:.4f}\")\n print(f\"F1-score after optimization (Keras Sequential): {optimized_f1_keras:.4f}\\n\")\n\n df_tmp_keras = pd.DataFrame.from_dict(optimized_results_keras, orient='index', columns=['Optimized F1-score'])\n df_tmp_keras.reset_index(inplace=True)\n df_tmp_keras.rename(columns={'index': 'Dataset'}, inplace=True)\n df_tmp_keras['Model'] = 'Keras Sequential'\n\n df_results_tiny = pd.concat([df_results_tiny, df_tmp_keras], ignore_index=True)\n\n output_file_optimized = OUTPUT_DATA_DIR + \"model_results_tiny.csv\"\n df_results_tiny.to_csv(output_file_optimized, index=False)\n\n print(f\"Optimized results (including Keras Sequential) saved to {output_file_optimized}\")","metadata":{"execution":{"iopub.status.busy":"2024-09-24T11:15:47.675819Z","iopub.execute_input":"2024-09-24T11:15:47.676285Z","iopub.status.idle":"2024-09-24T11:27:15.215246Z","shell.execute_reply.started":"2024-09-24T11:15:47.676242Z","shell.execute_reply":"2024-09-24T11:27:15.213259Z"},"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

7. Fourth Experiment - Evasion Attacks

\n
\n\n[Back to table of contents](#TOC)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

8. Results and Conclusions

\n
\n\n[Back to table of contents](#TOC)\n\n## Results from First Experiment","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.max_rows', None) # Show all rows\nif drop_svc:\n df_results = df_results[df_results['Model'] != 'SVC']\ndf_results","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:04:37.764364Z","iopub.execute_input":"2024-09-24T10:04:37.764913Z","iopub.status.idle":"2024-09-24T10:04:37.797119Z","shell.execute_reply.started":"2024-09-24T10:04:37.764870Z","shell.execute_reply":"2024-09-24T10:04:37.795855Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom scipy.stats import gmean\n\n# Function to compute mean of F1-scores across all datasets\ndef compute_mean(df, method=\"geometric\"):\n if method == \"geometric\":\n df['Mean_F1'] = df[['F1-score_Credit', 'F1-score_ChestXray', 'F1-score_ICR', 'F1-score_MNIST', 'F1-score_Netflix', 'F1-score_SpaceshipTitanic', 'F1-score_CIC']].apply(lambda row: gmean(row), axis=1)\n elif method == \"arithmetic\":\n df['Mean_F1'] = df[['F1-score_Credit', 'F1-score_ChestXray', 'F1-score_ICR', 'F1-score_MNIST', 'F1-score_Netflix', 'F1-score_SpaceshipTitanic', 'F1-score_CIC']].mean(axis=1)\n return df\n\n# Let the user choose between 'geometric' or 'arithmetic'\nmean_method = \"geometric\" # Change to 'arithmetic' if you want to compute arithmetic mean\n\n# Pivot the DataFrame to have one row per model, with F1-scores from different datasets as columns\ndf_pivot = df_results.pivot(index='Model', columns='Dataset', values='F1-score').reset_index()\n\n# Rename columns for clarity\ndf_pivot.columns.name = None\ndf_pivot.columns = ['Model', 'F1-score_Credit', 'F1-score_ChestXray', 'F1-score_ICR', 'F1-score_MNIST', 'F1-score_Netflix', 'F1-score_SpaceshipTitanic', 'F1-score_CIC']\n\n# Apply the function to compute the chosen mean\ndf_pivot = compute_mean(df_pivot, method=mean_method)\n\n# Sort the DataFrame by Mean F1-scores in descending order\ndf_sorted = df_pivot.sort_values(by=\"Mean_F1\", ascending=False)\n\n# Add the mean to the x-axis labels\ndf_sorted['Model_with_Mean'] = df_sorted.apply(lambda row: f\"{row['Model']} ({row['Mean_F1']:.4f})\", axis=1)\n\n# Set the plot aesthetics\nsns.set(style=\"whitegrid\")\n\n# Define a fixed color palette for each dataset\ndataset_colors = {\n \"F1-score_Netflix\": \"C2\", # Blue\n \"F1-score_Credit\": \"C2\", # Orange\n \"F1-score_ICR\": \"C2\", # Green\n \"F1-score_SpaceshipTitanic\": \"C2\", # Red\n \"F1-score_MNIST\": \"C2\", # Purple\n \"F1-score_ChestXray\": \"C2\", # Brown\n \"F1-score_CIC\": \"C2\" # Brown\n}\n\n# Create the plot\nplt.figure(figsize=(16, 10))\n\nA = 0.3\n\n# Plot F1-scores for all datasets with fixed colors\n\"\"\"barplot_netflix = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_Netflix\", data=df_sorted, color=dataset_colors[\"F1-score_Netflix\"], label=\"Netflix\", alpha=A)\nbarplot_credit = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_Credit\", data=df_sorted, color=dataset_colors[\"F1-score_Credit\"], label=\"Credit\", alpha=A)\nbarplot_ICR = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_ICR\", data=df_sorted, color=dataset_colors[\"F1-score_ICR\"], label=\"ICR\", alpha=A)\nbarplot_sptit = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_SpaceshipTitanic\", data=df_sorted, color=dataset_colors[\"F1-score_SpaceshipTitanic\"], label=\"Spaceship Titanic\", alpha=A)\nbarplot_mnist = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_MNIST\", data=df_sorted, color=dataset_colors[\"F1-score_MNIST\"], label=\"MNIST\", alpha=A)\nbarplot_xray = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_ChestXray\", data=df_sorted, color=dataset_colors[\"F1-score_ChestXray\"], label=\"Chest X-ray\", alpha=A)\n\"\"\"\nbarplot_netflix = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_Netflix\", data=df_sorted, color=dataset_colors[\"F1-score_Netflix\"], alpha=A)\nbarplot_credit = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_Credit\", data=df_sorted, color=dataset_colors[\"F1-score_Credit\"], alpha=A)\nbarplot_ICR = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_ICR\", data=df_sorted, color=dataset_colors[\"F1-score_ICR\"], alpha=A)\nbarplot_sptit = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_SpaceshipTitanic\", data=df_sorted, color=dataset_colors[\"F1-score_SpaceshipTitanic\"], alpha=A)\nbarplot_mnist = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_MNIST\", data=df_sorted, color=dataset_colors[\"F1-score_MNIST\"], alpha=A)\nbarplot_xray = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_ChestXray\", data=df_sorted, color=dataset_colors[\"F1-score_ChestXray\"], alpha=A)\nbarplot_cic = sns.barplot(x=\"Model_with_Mean\", y=\"F1-score_CIC\", data=df_sorted, color=dataset_colors[\"F1-score_CIC\"], alpha=A)\n\n# Add annotations for Netflix bar values\nfor p in barplot_netflix.patches:\n barplot_netflix.annotate(format(p.get_height(), '.4f'),\n (p.get_x() + p.get_width() / 2., p.get_height()),\n ha='center', va='center', xytext=(0, 9),\n textcoords='offset points')\n\n# Add annotations for Credit bar values\nfor p in barplot_credit.patches:\n barplot_credit.annotate(format(p.get_height(), '.4f'),\n (p.get_x() + p.get_width() / 2., p.get_height()),\n ha='center', va='center', xytext=(0, 9),\n textcoords='offset points')\n\n# Add labels and title\nplt.title(f\"Model F1-Score Comparison on All Datasets with {mean_method.capitalize()} Mean Ranking\", fontsize=16)\nplt.xlabel(f\"Model ({mean_method.capitalize()} Mean F1-score)\", fontsize=12)\nplt.ylabel(\"F1-score\", fontsize=12)\nplt.ylim([0.45, 1.03])\n\n# Rotate x-axis labels for better readability\nplt.xticks(rotation=45, ha=\"right\")\n\n# Add legend\n#plt.legend(title=\"Dataset\")\n\n# Save the plot\nfigure_file = FIGURES_DIR + f\"model_f1_score_comparison_{mean_method}_mean_all_datasets.png\"\nplt.tight_layout()\nplt.savefig(figure_file)\n\n# Display the sorted DataFrame for confirmation\ndf_sorted","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:04:49.049570Z","iopub.execute_input":"2024-09-24T10:04:49.050076Z","iopub.status.idle":"2024-09-24T10:04:52.006470Z","shell.execute_reply.started":"2024-09-24T10:04:49.050032Z","shell.execute_reply":"2024-09-24T10:04:52.004982Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Compute geometric mean for F1-scores (already done in previous steps)\ndf_pivot = compute_mean(df_pivot, method=\"geometric\")\n\n# Aggregating the total execution time for each model across datasets\ndf_time = df_results.groupby('Model')['Time (seconds)'].sum().reset_index()\ndf_time.columns = ['Model', 'Total_Time_Seconds']\n\n# Merge the geometric mean F1 scores with the total time data\ndf_merged_time = df_pivot[['Model', 'Mean_F1']].merge(df_time, on='Model')\n\n# Sort by total execution time in ascending order\ndf_merged_time_sorted = df_merged_time.sort_values(by=\"Total_Time_Seconds\", ascending=True)\n\n# Plotting the scatter plot\nplt.figure(figsize=(10, 6))\n\n# Scatter plot of Geometric Mean F1-score vs Total Time\nplt.scatter(df_merged_time_sorted['Total_Time_Seconds'], df_merged_time_sorted['Mean_F1'], color='blue', s=100)\n\n# Add annotations for each point (Model names)\nfor i in range(df_merged_time_sorted.shape[0]):\n plt.text(df_merged_time_sorted['Total_Time_Seconds'].iloc[i], \n df_merged_time_sorted['Mean_F1'].iloc[i], \n df_merged_time_sorted['Model'].iloc[i], \n fontsize=9, ha='right')\n\n# Add labels and title\nplt.title(\"Geometric Mean F1-score vs Execution Time\", fontsize=16)\nplt.xlabel(\"Total Execution Time (Seconds)\", fontsize=12)\nplt.ylabel(\"Geometric Mean F1-score\", fontsize=12)\n\n# Save the plot\nfigure_file = FIGURES_DIR + f\"model_f1_score_comparison_{mean_method}_mean_all_datasets_with_execution_time.png\"\nplt.tight_layout()\nplt.savefig(figure_file)\n\n# Show the plot\nplt.grid(True)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:05:00.003976Z","iopub.execute_input":"2024-09-24T10:05:00.004406Z","iopub.status.idle":"2024-09-24T10:05:00.899168Z","shell.execute_reply.started":"2024-09-24T10:05:00.004367Z","shell.execute_reply":"2024-09-24T10:05:00.897907Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Pivot the DataFrame to have one row per model, with F1-scores for different datasets as columns\ndf_f1_matrix = df_results.pivot(index='Model', columns='Dataset', values='F1-score')\n\n# Set plot size and aesthetics\nplt.figure(figsize=(10, 8))\nsns.set(style=\"whitegrid\")\n\n# Create the heatmap using seaborn\nsns.heatmap(df_f1_matrix, annot=True, cmap=\"coolwarm\", cbar_kws={'label': 'F1-score'}, linewidths=0.5)\n\n# Add labels and title\nplt.title(\"F1-Score Heatmap for Each Model and Dataset\", fontsize=16)\nplt.xlabel(\"Dataset\", fontsize=12)\nplt.ylabel(\"Model\", fontsize=12)\n\n# Save the plot\nfigure_file = FIGURES_DIR + f\"model_f1_score_comparison_grid_plot.png\"\nplt.tight_layout()\nplt.savefig(figure_file)\n\n# Show the plot\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:05:04.008773Z","iopub.execute_input":"2024-09-24T10:05:04.009243Z","iopub.status.idle":"2024-09-24T10:05:05.243847Z","shell.execute_reply.started":"2024-09-24T10:05:04.009195Z","shell.execute_reply":"2024-09-24T10:05:05.242532Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Sizes of the individual datasets\ndataset_sizes = {\n 'Credit': X_train_credit.shape[0], # Size of Credit Card Fraud dataset\n 'Netflix': X_train_netflix.shape[0], # Size of Netflix dataset\n 'ICR': X_train_ICR.shape[0], # Size of ICR dataset\n 'Spaceship Titanic': X_train_sptit.shape[0], # Size of Spaceship Titanic dataset\n 'MNIST': X_train_mnist.shape[0], # Size of MNIST dataset\n 'Chest X-ray': X_train_xray.shape[0], # Size of Chest X-ray dataset\n 'CIC': X_train_CIC.shape[0] # Size of Chest X-ray dataset\n}\n\n# Convert dataset sizes into a DataFrame and sort by size\ndf_sizes = pd.DataFrame(list(dataset_sizes.items()), columns=['Dataset', 'Size']).sort_values(by='Size')\n\n# Pivot the F1-score results DataFrame to get F1-scores for each dataset and model\ndf_f1_scores = df_results.pivot(index='Model', columns='Dataset', values='F1-score').reset_index()\n\n# Sort columns by dataset size\ndf_f1_scores = df_f1_scores[['Model'] + df_sizes['Dataset'].tolist()]\n\n# Melt the DataFrame to get it into long format for seaborn\ndf_f1_long = pd.melt(df_f1_scores, id_vars='Model', var_name='Dataset', value_name='F1-score')\n\n# Merge with dataset sizes to get size info in the plot\ndf_f1_long = pd.merge(df_f1_long, df_sizes, on='Dataset')\n\n# Plotting the line plot\nplt.figure(figsize=(12, 8))\nsns.set(style=\"whitegrid\")\n\n# Line plot for each model\nsns.lineplot(data=df_f1_long, x='Dataset', y='F1-score', hue='Model', marker='o', palette='Set2')\n\n# Add dataset sizes as labels on the x-axis\nplt.xticks(ticks=range(len(df_sizes)), labels=[f\"{dataset} ({size:,})\" for dataset, size in zip(df_sizes['Dataset'], df_sizes['Size'])], rotation=45)\n\n# Add labels and title\nplt.title(\"F1-score Across Datasets Ordered by Size\", fontsize=16)\nplt.xlabel(\"Dataset (Size in Rows)\", fontsize=12)\nplt.ylabel(\"F1-score\", fontsize=12)\n\n# Save the plot\nfigure_file = FIGURES_DIR + f\"model_f1_score_size_comparison.png\"\nplt.tight_layout()\nplt.savefig(figure_file)\n\n# Show the plot\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:05:08.914542Z","iopub.execute_input":"2024-09-24T10:05:08.915026Z","iopub.status.idle":"2024-09-24T10:05:10.094142Z","shell.execute_reply.started":"2024-09-24T10:05:08.914944Z","shell.execute_reply":"2024-09-24T10:05:10.092871Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Results from Second Experiment","metadata":{}},{"cell_type":"code","source":"df_results_optimized","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:05:15.854009Z","iopub.execute_input":"2024-09-24T10:05:15.854476Z","iopub.status.idle":"2024-09-24T10:05:15.874796Z","shell.execute_reply.started":"2024-09-24T10:05:15.854432Z","shell.execute_reply":"2024-09-24T10:05:15.873521Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\ndf_results_optimized[\"F1-score\"] = df_results_optimized[\"Optimized F1-score\"]\n\n# Pivot the DataFrame to have one row per model, with F1-scores for different datasets as columns\ndf_f1_matrix = df_results_optimized.pivot(index='Model', columns='Dataset', values='F1-score')\n\n# Set plot size and aesthetics\nplt.figure(figsize=(10, 8))\nsns.set(style=\"whitegrid\")\n\n# Create the heatmap using seaborn\nsns.heatmap(df_f1_matrix, annot=True, cmap=\"coolwarm\", cbar_kws={'label': 'F1-score'}, linewidths=0.5)\n\n# Add labels and title\nplt.title(\"F1-Score Heatmap for Each Model and Dataset\", fontsize=16)\nplt.xlabel(\"Dataset\", fontsize=12)\nplt.ylabel(\"Model\", fontsize=12)\n\n# Save the plot\nfigure_file = FIGURES_DIR + f\"OPTUNA_model_f1_score_comparison_grid_plot.png\"\nplt.tight_layout()\nplt.savefig(figure_file)\n\n# Show the plot\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-24T10:05:19.003306Z","iopub.execute_input":"2024-09-24T10:05:19.003925Z","iopub.status.idle":"2024-09-24T10:05:20.120247Z","shell.execute_reply.started":"2024-09-24T10:05:19.003875Z","shell.execute_reply":"2024-09-24T10:05:20.118770Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"

Thank you! If you found this useful please like 👍

","metadata":{}}]}