{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7602123,"sourceType":"competition"},{"sourceId":7658638,"sourceType":"datasetVersion","datasetId":4465434}],"dockerImageVersionId":30635,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Example Notebook\n\nWelcome to the example notebook for the Home Credit Kaggle competition. The goal of this competition is to determine how likely a customer is going to default on an issued loan. The main difference between the [first](https://www.kaggle.com/c/home-credit-default-risk) and this competition is that now your submission will be scored with a custom metric that will take into account how well the model performs in future. A decline in performance will be penalized. The goal is to create a model that is stable and performs well in the future.\n\nIn this notebook you will see how to:\n* Load the data\n* Join tables with Polars - a DataFrame library implemented in Rust language, designed to be blazingy fast and memory efficient.  \n* Create simple aggregation features\n* Train a LightGBM model\n* Create a submission table\n\n## Load the data","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import polars as pl\nimport numpy as np\nimport pandas as pd\nimport lightgbm as lgb\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score \n\ndataPath = \"/kaggle/input/home-credit-credit-risk-model-stability/\"","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:09:58.996156Z","iopub.execute_input":"2024-02-19T18:09:58.996804Z","iopub.status.idle":"2024-02-19T18:10:02.943692Z","shell.execute_reply.started":"2024-02-19T18:09:58.996713Z","shell.execute_reply":"2024-02-19T18:10:02.942362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_table_dtypes(df: pl.DataFrame) -> pl.DataFrame:\n    # implement here all desired dtypes for tables\n    # the following is just an example\n    for col in df.columns:\n        # last letter of column name will help you determine the type\n        if col[-1] in (\"P\", \"A\"):\n            df = df.with_columns(pl.col(col).cast(pl.Float64).alias(col))\n\n    return df\n\ndef convert_strings(df: pd.DataFrame) -> pd.DataFrame:\n    for col in df.columns:  \n        if df[col].dtype.name in ['object', 'string']:\n            df[col] = df[col].astype(\"string\").astype('category')\n            current_categories = df[col].cat.categories\n            new_categories = current_categories.to_list() + [\"Unknown\"]\n            new_dtype = pd.CategoricalDtype(categories=new_categories, ordered=True)\n            df[col] = df[col].astype(new_dtype)\n    return df","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:14:12.505126Z","iopub.execute_input":"2024-02-19T18:14:12.506101Z","iopub.status.idle":"2024-02-19T18:14:12.516411Z","shell.execute_reply.started":"2024-02-19T18:14:12.506048Z","shell.execute_reply":"2024-02-19T18:14:12.514746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_basetable = pl.read_csv(dataPath + \"csv_files/train/train_base.csv\")\ntrain_static = pl.concat(\n    [\n        pl.read_csv(dataPath + \"csv_files/train/train_static_0_0.csv\").pipe(set_table_dtypes),\n        pl.read_csv(dataPath + \"csv_files/train/train_static_0_1.csv\").pipe(set_table_dtypes),\n    ],\n    how=\"vertical_relaxed\",\n)\ntrain_static_cb = pl.read_csv(dataPath + \"csv_files/train/train_static_cb_0.csv\").pipe(set_table_dtypes)\ntrain_person_1 = pl.read_csv(dataPath + \"csv_files/train/train_person_1.csv\").pipe(set_table_dtypes) \ntrain_credit_bureau_b_2 = pl.read_csv(dataPath + \"csv_files/train/train_credit_bureau_b_2.csv\").pipe(set_table_dtypes) ","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:26:59.999025Z","iopub.execute_input":"2024-02-19T18:26:59.999476Z","iopub.status.idle":"2024-02-19T18:27:24.492175Z","shell.execute_reply.started":"2024-02-19T18:26:59.999445Z","shell.execute_reply":"2024-02-19T18:27:24.489854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_basetable = pl.read_csv(dataPath + \"csv_files/test/test_base.csv\")\ntest_static = pl.concat(\n    [\n        pl.read_csv(dataPath + \"csv_files/test/test_static_0_0.csv\").pipe(set_table_dtypes),\n        pl.read_csv(dataPath + \"csv_files/test/test_static_0_1.csv\").pipe(set_table_dtypes),\n        pl.read_csv(dataPath + \"csv_files/test/test_static_0_2.csv\").pipe(set_table_dtypes),\n    ],\n    how=\"vertical_relaxed\",\n)\ntest_static_cb = pl.read_csv(dataPath + \"csv_files/test/test_static_cb_0.csv\").pipe(set_table_dtypes)\ntest_person_1 = pl.read_csv(dataPath + \"csv_files/test/test_person_1.csv\").pipe(set_table_dtypes) \ntest_credit_bureau_b_2 = pl.read_csv(dataPath + \"csv_files/test/test_credit_bureau_b_2.csv\").pipe(set_table_dtypes) ","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:27:30.298179Z","iopub.execute_input":"2024-02-19T18:27:30.299270Z","iopub.status.idle":"2024-02-19T18:27:30.421763Z","shell.execute_reply.started":"2024-02-19T18:27:30.299223Z","shell.execute_reply":"2024-02-19T18:27:30.416992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature engineering\n\nIn this part, we can see a simple example of joining tables via `case_id`. Here the loading and joining is done with polars library. Polars library is blazingly fast and has much smaller memory footprint than pandas. ","metadata":{}},{"cell_type":"code","source":"# We need to use aggregation functions in tables with depth > 1, so tables that contain num_group1 column or \n# also num_group2 column.\ntrain_person_1_feats_1 = train_person_1.group_by(\"case_id\").agg(\n    pl.col(\"mainoccupationinc_384A\").max().alias(\"mainoccupationinc_384A_max\"),\n    (pl.col(\"incometype_1044T\") == \"SELFEMPLOYED\").max().alias(\"mainoccupationinc_384A_any_selfemployed\")\n)\n\n# Here num_group1=0 has special meaning, it is the person who applied for the loan.\ntrain_person_1_feats_2 = train_person_1.select([\"case_id\", \"num_group1\", \"housetype_905L\"]).filter(\n    pl.col(\"num_group1\") == 0\n).drop(\"num_group1\").rename({\"housetype_905L\": \"person_housetype\"})\n\n# Here we have num_goup1 and num_group2, so we need to aggregate again.\ntrain_credit_bureau_b_2_feats = train_credit_bureau_b_2.group_by(\"case_id\").agg(\n    pl.col(\"pmts_pmtsoverdue_635A\").max().alias(\"pmts_pmtsoverdue_635A_max\"),\n    (pl.col(\"pmts_dpdvalue_108P\") > 31).max().alias(\"pmts_dpdvalue_108P_over31\")\n)\n\n# We will process in this examples only A-type and M-type columns, so we need to select them.\nselected_static_cols = []\nfor col in train_static.columns:\n    if col[-1] in (\"A\", \"M\"):\n        selected_static_cols.append(col)\nprint(selected_static_cols)\n\nselected_static_cb_cols = []\nfor col in train_static_cb.columns:\n    if col[-1] in (\"A\", \"M\"):\n        selected_static_cb_cols.append(col)\nprint(selected_static_cb_cols)\n\n# Join all tables together.\ndata = train_basetable.join(\n    train_static.select([\"case_id\"]+selected_static_cols), how=\"left\", on=\"case_id\"\n).join(\n    train_static_cb.select([\"case_id\"]+selected_static_cb_cols), how=\"left\", on=\"case_id\"\n).join(\n    train_person_1_feats_1, how=\"left\", on=\"case_id\"\n).join(\n    train_person_1_feats_2, how=\"left\", on=\"case_id\"\n).join(\n    train_credit_bureau_b_2_feats, how=\"left\", on=\"case_id\"\n)","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:27:35.850324Z","iopub.execute_input":"2024-02-19T18:27:35.850816Z","iopub.status.idle":"2024-02-19T18:27:38.970089Z","shell.execute_reply.started":"2024-02-19T18:27:35.850766Z","shell.execute_reply":"2024-02-19T18:27:38.968198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"case_ids = data[\"case_id\"].unique().shuffle(seed=1)\ncase_ids_train, case_ids_test = train_test_split(case_ids, train_size=0.6, random_state=1)\ncase_ids_valid, case_ids_test = train_test_split(case_ids_test, train_size=0.5, random_state=1)\n\ncols_pred = []\nfor col in data.columns:\n    if col[-1].isupper() and col[:-1].islower():\n        cols_pred.append(col)\n\nprint(cols_pred)\n\ndef from_polars_to_pandas(case_ids: pl.DataFrame) -> pl.DataFrame:\n    return (\n        data.filter(pl.col(\"case_id\").is_in(case_ids))[[\"case_id\", \"WEEK_NUM\", \"target\"]].to_pandas(),\n        data.filter(pl.col(\"case_id\").is_in(case_ids))[cols_pred].to_pandas(),\n        data.filter(pl.col(\"case_id\").is_in(case_ids))[\"target\"].to_pandas()\n    )\n\nbase_train, X_train, y_train = from_polars_to_pandas(case_ids_train)\nbase_valid, X_valid, y_valid = from_polars_to_pandas(case_ids_valid)\nbase_test, X_test, y_test = from_polars_to_pandas(case_ids_test)\n\nfor df in [X_train, X_valid, X_test]:\n    df = convert_strings(df)","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:27:52.352438Z","iopub.execute_input":"2024-02-19T18:27:52.353020Z","iopub.status.idle":"2024-02-19T18:28:00.419259Z","shell.execute_reply.started":"2024-02-19T18:27:52.352976Z","shell.execute_reply":"2024-02-19T18:28:00.418311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Train: {X_train.shape}\")\nprint(f\"Valid: {X_valid.shape}\")\nprint(f\"Test: {X_test.shape}\")","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:16:47.506660Z","iopub.execute_input":"2024-02-19T18:16:47.507114Z","iopub.status.idle":"2024-02-19T18:16:47.514243Z","shell.execute_reply.started":"2024-02-19T18:16:47.507077Z","shell.execute_reply":"2024-02-19T18:16:47.512645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_pd = data.to_pandas()\ndata_train_total = data_pd","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndef plot_count(data, columns):\n    for col in columns:\n        value_counts = data[col].value_counts()\n        plt.figure(figsize=(10, 6))\n        plt.bar(value_counts.index, value_counts.values)\n        plt.title(f'Count of {col}')\n        plt.xlabel(col)\n        plt.ylabel('Count')\n        plt.xticks(rotation=45)\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T09:16:29.908588Z","iopub.execute_input":"2024-02-20T09:16:29.909117Z","iopub.status.idle":"2024-02-20T09:16:29.918019Z","shell.execute_reply.started":"2024-02-20T09:16:29.909079Z","shell.execute_reply":"2024-02-20T09:16:29.916918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_count(base_train, ['target'])\nplot_count(data_train_total, ['target'])\n","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:17:46.265375Z","iopub.execute_input":"2024-02-19T18:17:46.265829Z","iopub.status.idle":"2024-02-19T18:17:47.964392Z","shell.execute_reply.started":"2024-02-19T18:17:46.265784Z","shell.execute_reply":"2024-02-19T18:17:47.962993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Inspect shapes of X_train and y_train\nprint(f\"X_train shape: {X_train.shape}\")\nprint(f\"y_train shape: {y_train.shape}\")\n\n# Numerical feature visualizations (adapt based on features)\nfor col in X_train.select_dtypes(include=['int64', 'float64']):\n    # Check for missing values\n    if X_train[col].isna().any():\n        print(f\"Warning: Feature {col} has missing values. Consider imputation.\")\n\n        continue\n\n    # Histogram for distribution and outliers\n    plt.figure(figsize=(8, 6))\n    sns.histplot(data=X_train, x=col, kde=True)\n    plt.title(f\"Distribution of feature: {col}\")\n    plt.show()\n\n    # Box plot for outliers\n    plt.figure(figsize=(8, 6))\n    sns.boxplot(y=col, data=X_train)\n    plt.title(f\"Box plot of feature: {col}\")\n    plt.show()\n\n# Categorical feature visualizations (adapt based on your features)\nfor col in X_train.select_dtypes(include=['category']):\n    # Count or bar plot for category distribution\n    plt.figure(figsize=(8, 6))\n    sns.countplot(x=col, data=X_train)\n    plt.title(f\"Distribution of categories in feature: {col}\")\n    plt.xticks(rotation=45)\n    plt.show()\n\n# Target variable visualization\nplt.figure(figsize=(8, 6))\nsns.histplot(data=y_train, kde=True)\nplt.title(\"Distribution of target variable\")\nplt.show()\n\n# Further considerations:\n# - Investigate relationships between features using scatter matrices or pair plots.\n# - Create feature importance plots if you have trained a model already.\n# - Adjust plot configurations (e.g., colors, bin sizes) for better visualization.","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:18:19.479307Z","iopub.execute_input":"2024-02-19T18:18:19.479762Z","iopub.status.idle":"2024-02-19T18:18:58.894106Z","shell.execute_reply.started":"2024-02-19T18:18:19.479726Z","shell.execute_reply":"2024-02-19T18:18:58.892471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Handling missing with mean (for now)","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Color palette for visualizations\ncolors = sns.color_palette(\"hls\", len(X_train.select_dtypes(include=['int64', 'float64'])))\n\n# Inspect shapes of X_train and y_train\nprint(f\"X_train shape: {X_train.shape}\")\nprint(f\"y_train shape: {y_train.shape}\")\n\n# Numerical feature visualizations (adapt based on your features)\nfor i, col in enumerate(X_train.select_dtypes(include=['int64', 'float64'])):\n    # Count missing values\n    missing_count = X_train[col].isna().sum()\n    if missing_count > 0:    # there is one missing value in annuitynextmonth_57A\n        print(f\"Warning: Feature {col} has {missing_count} missing values ({missing_count/len(X_train):.2%}). Consider imputation.\")\n\n    # Histogram with custom bins and colors based on normal distribution\n    if X_train[col].isna().any():\n        ''' Handle missing values (replace with appropriate strategy) '''\n        X_train[col] = X_train[col].fillna(X_train[col].mean())\n\n    # Calculate mean and standard deviation\n    mean_val = X_train[col].mean()\n    std_dev = X_train[col].std()\n\n    # Define bins based on standard deviations\n    bins = [\n        mean_val - 3 * std_dev,\n        mean_val - 2 * std_dev,\n        mean_val - std_dev,\n        mean_val,\n        mean_val + std_dev,\n        mean_val + 2 * std_dev,\n        mean_val + 3 * std_dev,\n    ]\n\n    # Histogram with custom bins and colors\n    plt.figure(figsize=(8, 6))\n    plt.hist(X_train[col], bins=bins, ec=\"k\", color=colors[i])\n    plt.title(f\"Distribution of feature: {col}\")\n    plt.xlabel(f\"{col}\")\n    plt.ylabel(\"Frequency\")\n    plt.show()\n\n    # Box plot for outliers\n    plt.figure(figsize=(8, 6))\n    sns.boxplot(y=col, data=X_train, color=colors[i])\n    plt.title(f\"Box plot of feature: {col}\")\n    plt.xlabel(f\"{col}\")\n    plt.ylabel(\"\")  # Remove default y-axis label\n    plt.show()\n\n# Categorical feature visualizations\nfor col in X_train.select_dtypes(include=['category']):\n    # Count or bar plot for category distribution\n    plt.figure(figsize=(8, 6))\n    sns.countplot(x=col, data=X_train, palette=colors[:len(X_train[col].unique())])\n    plt.title(f\"Distribution of categories in feature: {col}\")\n    plt.xticks(rotation=45)\n    plt.xlabel(\"\")  # Remove default x-axis label\n    plt.ylabel(\"Frequency\")\n    plt.show()\n\n# Target variable visualization\nplt.figure(figsize=(8, 6))\nsns.histplot(data=y_train, kde=True, color=colors[0])\nplt.title(\"Distribution of target variable\")\nplt.xlabel(\"Target variable\")\nplt.ylabel(\"Frequency\")\nplt.show()\n\n# Further considerations:\n# - Investigate relationships between features using scatter matrices or pair plots with rotated labels and informative color schemes.\n# - Create feature importance plots if you have trained a model already, using a color scheme and rotated labels for clarity.\n# - Adjust plot configurations (e.g., colors, bin sizes, labelpad) for better visualization based on your needs.\n\n","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:19:54.707595Z","iopub.execute_input":"2024-02-19T18:19:54.708091Z","iopub.status.idle":"2024-02-19T18:20:49.193443Z","shell.execute_reply.started":"2024-02-19T18:19:54.708056Z","shell.execute_reply":"2024-02-19T18:20:49.191562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BOX PLOTS\n(Run this before the previous code block, to prevent mean imputation)\nThis plot is without imputing the null values","metadata":{}},{"cell_type":"code","source":"features_with_missing_values = {}\n# Numerical feature visualizations (adapt based on your features)\nfor i, col in enumerate(X_train.select_dtypes(include=['int64', 'float64'])):\n    # Count missing values\n    missing_count = X_train[col].isna().sum()\n    if missing_count > 0:\n        print(f\"Warning: Feature {col} has {missing_count} missing values ({missing_count/len(X_train):.2%}). Consider imputation.\")\n        features_with_missing_values[col] = missing_count\n\n    # Handle missing values (replace with appropriate strategy)\n    # X_train[col] = X_train[col].fillna(X_train[col].mean())  # Example imputation\n\n    # Box plot for outliers\n    plt.figure(figsize=(8, 6))\n    sns.boxplot(y=col, data=X_train, color=colors[i])\n    plt.title(f\"Box plot of feature: {col}\")\n    plt.xlabel(f\"{col}\")\n    plt.ylabel(\"\")\n    plt.show()\n\n","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:29:52.703507Z","iopub.execute_input":"2024-02-19T18:29:52.703988Z","iopub.status.idle":"2024-02-19T18:30:05.802138Z","shell.execute_reply.started":"2024-02-19T18:29:52.703951Z","shell.execute_reply":"2024-02-19T18:30:05.800534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Bar graphs (rotated, with null)","metadata":{}},{"cell_type":"code","source":"for col in X_train.select_dtypes(include=['category']):\n    # Create a count plot with horizontal bars\n    plt.figure(figsize=(8, 24))\n    sns.countplot(y=col, data=df, palette=sns.color_palette(\"hls\", len(df[col].unique())))\n    plt.title(f\"Distribution of categories in feature: {col}\")\n    plt.xlabel(\"Frequency\")\n    plt.ylabel(\"Categories\")\n    plt.show()\n\n\n# Target variable visualization\nplt.figure(figsize=(8, 6))\nsns.histplot(data=y_train, kde=True, color=colors[0])","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:31:48.374518Z","iopub.execute_input":"2024-02-19T18:31:48.375084Z","iopub.status.idle":"2024-02-19T18:32:05.710687Z","shell.execute_reply.started":"2024-02-19T18:31:48.375046Z","shell.execute_reply":"2024-02-19T18:32:05.708741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Missing","metadata":{}},{"cell_type":"code","source":"print(\"Features with missing values:\")\nfor feature, count in features_with_missing_values.items():\n    print(f\"{feature}: {count} missing values, which is {count/len(X_train):.2%} \")","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:30:33.352056Z","iopub.execute_input":"2024-02-19T18:30:33.352452Z","iopub.status.idle":"2024-02-19T18:30:33.360040Z","shell.execute_reply.started":"2024-02-19T18:30:33.352418Z","shell.execute_reply":"2024-02-19T18:30:33.358764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# One-Hot Encoding and Oversampling\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.preprocessing import OneHotEncoder","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:45:35.116345Z","iopub.execute_input":"2024-02-19T18:45:35.116894Z","iopub.status.idle":"2024-02-19T18:45:37.255911Z","shell.execute_reply.started":"2024-02-19T18:45:35.116849Z","shell.execute_reply":"2024-02-19T18:45:37.254294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_pd1 = data.to_pandas()\nfraud = data_pd1[data_pd1[\"target\"] == 1]\nnormal = data_pd1[data_pd1[\"target\"] == 0]\n\nprint(len(fraud),\"\\n\", len(normal))","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:34:22.585227Z","iopub.execute_input":"2024-02-19T18:34:22.586084Z","iopub.status.idle":"2024-02-19T18:34:24.836524Z","shell.execute_reply.started":"2024-02-19T18:34:22.586019Z","shell.execute_reply":"2024-02-19T18:34:24.835195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering 2","metadata":{}},{"cell_type":"code","source":"data_encoded = data_pd1.copy()\ndata_encoded.fillna(0, inplace=True)\ndata_encoded[\"date_decision\"] = pd.to_datetime(data_encoded[\"date_decision\"], format='%Y-%m-%d', errors='coerce').dt.year * 10000 + \\\n                                pd.to_datetime(data_encoded[\"date_decision\"], format='%Y-%m-%d', errors='coerce').dt.month * 100 + \\\n                                pd.to_datetime(data_encoded[\"date_decision\"], format='%Y-%m-%d', errors='coerce').dt.day","metadata":{"execution":{"iopub.status.busy":"2024-02-19T18:35:08.198225Z","iopub.execute_input":"2024-02-19T18:35:08.198673Z","iopub.status.idle":"2024-02-19T18:35:14.329521Z","shell.execute_reply.started":"2024-02-19T18:35:08.198633Z","shell.execute_reply":"2024-02-19T18:35:14.327406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Checkpoint :)","metadata":{}},{"cell_type":"code","source":"# data_encoded.to_csv(\"data_encoded_saved_kaggle.csv\", index=False)\ndata_encoded = pd.read_csv(\"/kaggle/input/data-encoded-saved-csv/data_encoded_saved.csv\", low_memory=False)","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:51:57.121906Z","iopub.execute_input":"2024-02-19T19:51:57.122416Z","iopub.status.idle":"2024-02-19T19:52:29.722658Z","shell.execute_reply.started":"2024-02-19T19:51:57.122375Z","shell.execute_reply":"2024-02-19T19:52:29.720838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dtypes = {55: 'object', 57: 'object'}  # Adjust based on actual types\ndata_encoded = pd.read_csv(\"data_encoded_saved_kaggle.csv\", dtype=dtypes)","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:38:48.169694Z","iopub.execute_input":"2024-02-19T19:38:48.170195Z","iopub.status.idle":"2024-02-19T19:39:05.854309Z","shell.execute_reply.started":"2024-02-19T19:38:48.170160Z","shell.execute_reply":"2024-02-19T19:39:05.852538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_encoded.describe(include='all')","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:28:52.394478Z","iopub.execute_input":"2024-02-19T19:28:52.394968Z","iopub.status.idle":"2024-02-19T19:28:57.313085Z","shell.execute_reply.started":"2024-02-19T19:28:52.394926Z","shell.execute_reply":"2024-02-19T19:28:57.311604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_encoded_copy1 = data_encoded.copy()\ncategorical_features = [col for col in data_encoded_copy1.columns if data_encoded_copy1[col].dtype == object]","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:52:50.755328Z","iopub.execute_input":"2024-02-19T19:52:50.755952Z","iopub.status.idle":"2024-02-19T19:52:51.377115Z","shell.execute_reply.started":"2024-02-19T19:52:50.755906Z","shell.execute_reply":"2024-02-19T19:52:51.375253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_encoded_copy1.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:52:54.715417Z","iopub.execute_input":"2024-02-19T19:52:54.715969Z","iopub.status.idle":"2024-02-19T19:52:54.756826Z","shell.execute_reply.started":"2024-02-19T19:52:54.715924Z","shell.execute_reply":"2024-02-19T19:52:54.755464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_encoded_copy1.describe(include='all')","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:32:08.145686Z","iopub.execute_input":"2024-02-19T19:32:08.146235Z","iopub.status.idle":"2024-02-19T19:32:13.299568Z","shell.execute_reply.started":"2024-02-19T19:32:08.146189Z","shell.execute_reply":"2024-02-19T19:32:13.298322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_features","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:40:46.712156Z","iopub.execute_input":"2024-02-19T19:40:46.712675Z","iopub.status.idle":"2024-02-19T19:40:46.719983Z","shell.execute_reply.started":"2024-02-19T19:40:46.712614Z","shell.execute_reply":"2024-02-19T19:40:46.718705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"# Iterate through categorical features and perform one-hot encoding\nfor col in categorical_features[9:-2]:\n    # Check if the feature has the expected data type (string)\n    if data_encoded_copy[col].dtype == object:\n        encoder = OneHotEncoder(sparse=False, handle_unknown='ignore')\n        encoded_features = encoder.fit_transform(data_encoded_copy[[col]])  # Encode single feature\n\n        # Add encoded features back to the DataFrame, dropping the original categorical feature\n        data_encoded_copy.drop(col, axis=1, inplace=True)\n        for i, new_col in enumerate(encoder.get_feature_names_out()):\n            data_encoded_copy[new_col] = encoded_features[:, i]\n    else:\n        print(f\"Warning: Skipping feature '{col}' because it's not a string (categorical).\")\n\n# Now your data_encoded DataFrame contains one-hot encoded features\nprint(data_encoded.head())\n\"\"\"\n### KEEPS CRASHING, PROBABLY BECAUSE IT IS MEMEORY INTENSIVE ###\n\n## Using Concat to avoid crashing\n\"\"\" \nencoded_data = pd.DataFrame()  # Initialize an empty DataFrame to store encoded features\n\nfor col in categorical_features:\n    if data_encoded_copy[col].dtype == object:\n        encoder = OneHotEncoder(sparse=False, handle_unknown='ignore')\n        encoded_features = encoder.fit_transform(data_encoded_copy[[col]])\n        encoded_data = pd.concat([encoded_data, pd.DataFrame(encoded_features, columns=encoder.get_feature_names_out())], axis=1)\n    else:\n        print(f\"Warning: Skipping feature '{col}' because it's not a string (categorical).\")\n\n# Combine encoded features with numerical features (if any)\nfinal_df = pd.concat([data_encoded_copy.drop(categorical_features, axis=1), encoded_data], axis=1)\n\nprint(final_df.head())\nprint(len(final_df.columns))\n\n###### STILL CRASHING!! ####\n\"\"\"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Performing Concatenations in batches to avoid crashing","metadata":{}},{"cell_type":"code","source":"print(encoded_features[:5])","metadata":{"execution":{"iopub.status.busy":"2024-02-19T19:21:14.463071Z","iopub.execute_input":"2024-02-19T19:21:14.464987Z","iopub.status.idle":"2024-02-19T19:21:14.474623Z","shell.execute_reply.started":"2024-02-19T19:21:14.464932Z","shell.execute_reply":"2024-02-19T19:21:14.472659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Identify potential categorical features by data type and unique values \n# potential_categorical = [col for col in data_encoded_copy.columns if data_encoded_copy[col].dtype == object]\n# #  or data_encoded_copy[col].nunique() > 10]\n\n# Define batch size\nbatch_size = 1 # crashing for 3\n\n# Create an empty DataFrame to store encoded features\nencoded_data = pd.DataFrame()\nfinal_df = pd.DataFrame()\n# Process features in batches\nfor i in range(0, len(categorical_features[:-2]), batch_size):\n    features_to_encode = potential_categorical[i:i+batch_size]\n    encoded_features_set = set()  # Reset encoded features set for each batch\n\n    # Iterate through the batch and perform one-hot encoding\n    for col in features_to_encode:\n        if col not in encoded_features_set and data_encoded_copy1[col].dtype == object:\n            print(col)\n            encoder = OneHotEncoder(sparse_output=True, handle_unknown='ignore') # make sparse=True to save memory\n            encoded_features = encoder.fit_transform(data_encoded_copy1[[col]])\n            encoded_data = pd.concat([encoded_data, pd.DataFrame(encoded_features, columns=encoder.get_feature_names_out())], axis=1)\n            encoded_features_set.add(col)\n        else:\n            print(f\"Skipping feature '{col}' because it's already encoded or not a string (categorical).\")\n\n    # Combine encoded features with numerical features and update final_df\n    final_df = pd.concat([final_df if any(final_df) else data_encoded_copy1.drop(potential_categorical, axis=1), encoded_data], axis=1)\n    encoded_data = pd.DataFrame()  # Reset encoded_data for the next batch","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Tensorflow with batching for one-hot encoding","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.preprocessing import OneHotEncoder\n\ndata_encoded = pd.read_csv(\"/kaggle/input/data-encoded-saved-csv/data_encoded_saved.csv\")\ndata_encoded_copy = data_encoded.copy()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:35:09.321427Z","iopub.execute_input":"2024-02-20T06:35:09.322341Z","iopub.status.idle":"2024-02-20T06:35:37.093074Z","shell.execute_reply.started":"2024-02-20T06:35:09.322288Z","shell.execute_reply":"2024-02-20T06:35:37.091833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow_addons.utils import keras_utils \n\n# Identify potential categorical features (unchanged)\npotential_categorical = [col for col in data_encoded_copy.columns if data_encoded_copy[col].dtype == object]\n#  or data_encoded_copy[col].nunique() > 10]\n\n# Define batch size (unchanged)\nbatch_size = 2\n\n# Create empty DataFrames for encoded data and final result (unchanged)\nencoded_data = pd.DataFrame()\nfinal_df = pd.DataFrame()\n\n# Process features in batches (with GPU utilization)\nfor i in range(0, len(potential_categorical[:-2]), batch_size):\n    features_to_encode = potential_categorical[i:i+batch_size]\n    encoded_features_set = set()\n\n    # --- Correction: Convert categorical data to integers before moving to GPU ---\n    data_batch_int = pd.get_dummies(data_encoded_copy[features_to_encode], columns=features_to_encode)\n    data_batch_gpu = tf.convert_to_tensor(data_batch_int, dtype=tf.int64)\n\n    with tf.device('/gpu:0'):\n        for col in features_to_encode:\n            if col not in encoded_features_set and data_encoded_copy[col].dtype == object:\n                print(col)\n\n                # # Use tf.one_hot directly on the GPU for efficiency\n                # encoded_features = tf.one_hot(data_batch_gpu[:, col], depth=data_encoded_copy[col].nunique())\n\n                # Move encoded features back to CPU and convert to DataFrame\n                col_index = features_to_encode.index(col)  # Get the integer index of the column\n                depth = data_encoded_copy[col].nunique()\n                encoded_features = tf.one_hot(data_batch_gpu[:, col_index], depth=data_encoded_copy[col].nunique())\n\n                # encoded_data = pd.concat([encoded_data, pd.DataFrame(encoded_features.numpy(), columns=encoder.get_feature_names_out())], axis=1)\n                encoded_data = pd.concat([encoded_data, pd.DataFrame(encoded_features.numpy(), columns=[f\"{col}_{i}\" for i in range(depth)])], axis=1)\n                encoded_features_set.add(col)\n            else:\n                print(f\"Skipping feature '{col}' because it's already encoded or not a string (categorical).\")\n\n    # Combine encoded features with numerical features and update final_df (unchanged)\n    final_df = pd.concat([final_df if any(final_df) else data_encoded_copy.drop(potential_categorical, axis=1), encoded_data], axis=1)\n    encoded_data = pd.DataFrame()  # Reset encoded_data for the next batch\n\n# Check the resulting DataFrame (unchanged)\nprint(final_df.head())","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:36:37.894950Z","iopub.execute_input":"2024-02-20T06:36:37.895413Z","iopub.status.idle":"2024-02-20T06:37:56.846081Z","shell.execute_reply.started":"2024-02-20T06:36:37.895363Z","shell.execute_reply":"2024-02-20T06:37:56.844941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_df.fillna(0, inplace=True)\nfinal_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T03:13:03.936841Z","iopub.execute_input":"2024-02-20T03:13:03.937818Z","iopub.status.idle":"2024-02-20T03:13:08.027301Z","shell.execute_reply.started":"2024-02-20T03:13:03.937780Z","shell.execute_reply":"2024-02-20T03:13:08.025600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Checkpoint 2 :')","metadata":{}},{"cell_type":"code","source":"final_df.to_csv(\"final_df_after_tf_ohc.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-02-19T21:48:26.748538Z","iopub.execute_input":"2024-02-19T21:48:26.748966Z","iopub.status.idle":"2024-02-19T22:06:12.898745Z","shell.execute_reply.started":"2024-02-19T21:48:26.748932Z","shell.execute_reply":"2024-02-19T22:06:12.897561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import pandas as pd\n# final_df = pd.read_csv(\"/kaggle/working/final_df_after_tf_ohc.csv\")\nfinal_df.fillna(0, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T02:48:01.815228Z","iopub.execute_input":"2024-02-20T02:48:01.816521Z","iopub.status.idle":"2024-02-20T02:48:07.189755Z","shell.execute_reply.started":"2024-02-20T02:48:01.816451Z","shell.execute_reply":"2024-02-20T02:48:07.188059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Read 5.6 GB csv file generated, chunkwise, using dask","metadata":{}},{"cell_type":"code","source":"# # import pandas as pd\n# # final_df = pd.read_csv(\"/content/drive/MyDrive/final_df_after_tf_ohc.csv\")\n# import dask.dataframe as dd\n\n# # Replace with your actual file path\n# filepath = \"/kaggle/working/final_df_after_tf_ohc.csv\"\n\n# # Define chunk size (adjust based on available memory)\n# chunksize = 1000  # Number of rows per chunk\n\n# # Read the file in chunks\n# # Create a Dask DataFrame by defining chunks manually\n# ddf = dd.read_csv(filepath, blocksize=chunksize * 40)  # Adjust blocksize for optimal performance\n\n# # Collect the results and create the final DataFrame\n# final_df = ddf.compute()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T02:08:48.764984Z","iopub.execute_input":"2024-02-20T02:08:48.765495Z","iopub.status.idle":"2024-02-20T02:13:19.969265Z","shell.execute_reply.started":"2024-02-20T02:08:48.765457Z","shell.execute_reply":"2024-02-20T02:13:19.967799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import pandas as pd\n# import logging\n\n# def read_csv_chunks(filepath, chunksize=10000):\n#   \"\"\"Reads a large CSV file in chunks and returns a final DataFrame with logging.\n\n#   Args:\n#     filepath: Path to the CSV file.\n#     chunksize: Number of rows to read in each chunk.\n\n#   Returns:\n#     A final DataFrame containing all data from the file.\n#   \"\"\"\n#   chunks = []\n#   processed_lines = 0\n#   total_lines = sum(1 for _ in open(filepath, 'r'))  # Count total lines (excluding header)\n#   print(f\"Total lines to process: {total_lines}\")\n\n#   with open(filepath, 'r') as f:\n#     # Skip header row if present\n#     header = f.readline()\n#     if header:\n#       columns = header.strip().split(',')\n    \n#     while True:\n#       chunk = []\n#       for _ in range(chunksize):\n#         line = f.readline()\n#         if not line:\n#           break\n#         processed_lines += 1\n#         chunk.append(line.strip().split(','))\n#       # Append chunk to list and log progress\n#       if chunk:\n#         chunks.append(pd.DataFrame(chunk, columns=columns))\n#         print(f\"Processed {processed_lines} lines ({processed_lines/total_lines:.2%})\")\n#       else:\n#         break\n\n#   # Concatenate chunks into final DataFrame\n#   final_df = pd.concat(chunks, ignore_index=True)\n#   return final_df\n\n# filepath = \"/kaggle/working/final_df_after_tf_ohc.csv\"\n# chunksize = 50 # Adjust based on available memory\n\n# logging.basicConfig(level=logging.INFO)  # Configure logging to show info messages\n\n# final_df = read_csv_chunks(filepath, chunksize)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T02:26:57.111645Z","iopub.execute_input":"2024-02-20T02:26:57.112136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n# X = final_df.drop(\"target\", axis=1)\ny = final_df[\"target\"]\nX = final_df.loc[:, final_df.columns != \"target\"]\n\n# # Split the data into training and testing sets\n# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# # Print the shapes of the split data\n# print(\"X_train shape:\", X_train.shape)\n# print(\"X_test shape:\", X_test.shape)\n# print(\"y_train shape:\", y_train.shape)\n# print(\"y_test shape:\", y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T03:20:36.010636Z","iopub.execute_input":"2024-02-20T03:20:36.011163Z","iopub.status.idle":"2024-02-20T03:20:43.007275Z","shell.execute_reply.started":"2024-02-20T03:20:36.011123Z","shell.execute_reply":"2024-02-20T03:20:43.005981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.shape","metadata":{"execution":{"iopub.status.busy":"2024-02-20T03:55:52.911451Z","iopub.execute_input":"2024-02-20T03:55:52.911862Z","iopub.status.idle":"2024-02-20T03:55:53.338616Z","shell.execute_reply.started":"2024-02-20T03:55:52.911830Z","shell.execute_reply":"2024-02-20T03:55:53.335130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Oversampling underrepresented targets","metadata":{}},{"cell_type":"code","source":"def oversample_chunk(X_chunk, y_chunk):\n    \"\"\"Oversamples a single chunk of data using RandomOverSampler.\"\"\"\n    os = RandomOverSampler()\n    X_res_chunk, y_res_chunk = os.fit_resample(X_chunk, y_chunk)\n    return X_res_chunk, y_res_chunk","metadata":{"execution":{"iopub.status.busy":"2024-02-20T03:22:25.931546Z","iopub.execute_input":"2024-02-20T03:22:25.931980Z","iopub.status.idle":"2024-02-20T03:22:25.939324Z","shell.execute_reply.started":"2024-02-20T03:22:25.931944Z","shell.execute_reply":"2024-02-20T03:22:25.937774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import dask.dataframe as dd\nfrom imblearn.over_sampling import RandomOverSampler\n","metadata":{"execution":{"iopub.status.busy":"2024-02-20T03:23:04.941403Z","iopub.execute_input":"2024-02-20T03:23:04.941893Z","iopub.status.idle":"2024-02-20T03:23:06.474705Z","shell.execute_reply.started":"2024-02-20T03:23:04.941856Z","shell.execute_reply":"2024-02-20T03:23:06.473410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_dask = dd.from_pandas(X, npartitions=2)  # Adjust npartitions based on your system\ny_dask = dd.from_pandas(y, npartitions=2)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-20T03:31:23.326408Z","iopub.execute_input":"2024-02-20T03:31:23.326875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_col_index= final_df.columns.get_loc(\"target\")\ntarget_col_index","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:38:09.918981Z","iopub.execute_input":"2024-02-20T06:38:09.919408Z","iopub.status.idle":"2024-02-20T06:38:09.926376Z","shell.execute_reply.started":"2024-02-20T06:38:09.919375Z","shell.execute_reply":"2024-02-20T06:38:09.925433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_1 = final_df.iloc[: 381663, :target_col_index] + final_df.iloc[: 381663, target_col_index + 1:]\ny_1 = final_df[\"target\"][:381663]","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:38:19.770315Z","iopub.execute_input":"2024-02-20T06:38:19.770789Z","iopub.status.idle":"2024-02-20T06:38:21.949983Z","shell.execute_reply.started":"2024-02-20T06:38:19.770743Z","shell.execute_reply":"2024-02-20T06:38:21.949096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_1.fillna(0, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:40:44.145601Z","iopub.execute_input":"2024-02-20T06:40:44.146066Z","iopub.status.idle":"2024-02-20T06:40:50.764162Z","shell.execute_reply.started":"2024-02-20T06:40:44.146032Z","shell.execute_reply":"2024-02-20T06:40:50.762903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from imblearn.over_sampling import RandomOverSampler\nos = RandomOverSampler()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:42:01.352172Z","iopub.execute_input":"2024-02-20T06:42:01.352645Z","iopub.status.idle":"2024-02-20T06:42:02.311172Z","shell.execute_reply.started":"2024-02-20T06:42:01.352610Z","shell.execute_reply":"2024-02-20T06:42:02.309901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_res_1, y_res_1 = os.fit_resample(X_1, y_1)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:42:06.032399Z","iopub.execute_input":"2024-02-20T06:42:06.033261Z","iopub.status.idle":"2024-02-20T06:42:19.098937Z","shell.execute_reply.started":"2024-02-20T06:42:06.033219Z","shell.execute_reply":"2024-02-20T06:42:19.097026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# (y_res_1 == 0).sum() 369857\n(y_res_1 == 1).sum() 369857","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:15:04.411128Z","iopub.execute_input":"2024-02-20T07:15:04.411877Z","iopub.status.idle":"2024-02-20T07:15:04.421774Z","shell.execute_reply.started":"2024-02-20T07:15:04.411830Z","shell.execute_reply":"2024-02-20T07:15:04.420552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_1.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:00:17.826102Z","iopub.execute_input":"2024-02-20T07:00:17.826610Z","iopub.status.idle":"2024-02-20T07:00:17.875683Z","shell.execute_reply.started":"2024-02-20T07:00:17.826567Z","shell.execute_reply":"2024-02-20T07:00:17.874704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_res_1.to_csv(\"X_res_1.csv\")\ny_res_1.to_csv(\"y_res_1.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:42:30.040074Z","iopub.execute_input":"2024-02-20T06:42:30.040598Z","iopub.status.idle":"2024-02-20T06:55:42.998688Z","shell.execute_reply.started":"2024-02-20T06:42:30.040560Z","shell.execute_reply":"2024-02-20T06:55:42.997324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_2 = final_df.iloc[381664: 763328, :target_col_index] + final_df.iloc[381664: 763328, target_col_index + 1:]\ny_2 = final_df[\"target\"][381664: 763328]","metadata":{"execution":{"iopub.status.busy":"2024-02-20T06:59:34.736838Z","iopub.execute_input":"2024-02-20T06:59:34.737966Z","iopub.status.idle":"2024-02-20T06:59:45.205003Z","shell.execute_reply.started":"2024-02-20T06:59:34.737892Z","shell.execute_reply":"2024-02-20T06:59:45.203593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_2 = final_df.iloc[381664: 763328, :target_col_index] + final_df.iloc[381664: 763328, target_col_index + 1:]\ny_2 = final_df[\"target\"][381664: 763328]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%whos","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:27:33.120376Z","iopub.execute_input":"2024-02-20T07:27:33.120893Z","iopub.status.idle":"2024-02-20T07:27:33.377792Z","shell.execute_reply.started":"2024-02-20T07:27:33.120850Z","shell.execute_reply":"2024-02-20T07:27:33.376529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:34:00.753657Z","iopub.execute_input":"2024-02-20T07:34:00.754103Z","iopub.status.idle":"2024-02-20T07:34:02.124392Z","shell.execute_reply.started":"2024-02-20T07:34:00.754069Z","shell.execute_reply":"2024-02-20T07:34:02.122742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del y_1","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:27:17.299356Z","iopub.execute_input":"2024-02-20T07:27:17.299803Z","iopub.status.idle":"2024-02-20T07:27:17.305649Z","shell.execute_reply.started":"2024-02-20T07:27:17.299767Z","shell.execute_reply":"2024-02-20T07:27:17.304199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_2.fillna(0, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:06:55.824998Z","iopub.execute_input":"2024-02-20T07:06:55.825426Z","iopub.status.idle":"2024-02-20T07:07:02.557608Z","shell.execute_reply.started":"2024-02-20T07:06:55.825392Z","shell.execute_reply":"2024-02-20T07:07:02.556418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_2.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:23:04.035364Z","iopub.execute_input":"2024-02-20T07:23:04.035982Z","iopub.status.idle":"2024-02-20T07:23:04.079418Z","shell.execute_reply.started":"2024-02-20T07:23:04.035930Z","shell.execute_reply":"2024-02-20T07:23:04.078421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_2.to_csv(\"X_2.csv\")\ny_2.to_csv(\"y_2.csv\") ","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:36:29.083408Z","iopub.execute_input":"2024-02-20T07:36:29.083941Z","iopub.status.idle":"2024-02-20T07:41:44.038271Z","shell.execute_reply.started":"2024-02-20T07:36:29.083896Z","shell.execute_reply":"2024-02-20T07:41:44.036276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del y_res_2","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:34:10.326370Z","iopub.execute_input":"2024-02-20T07:34:10.327385Z","iopub.status.idle":"2024-02-20T07:34:10.375162Z","shell.execute_reply.started":"2024-02-20T07:34:10.327339Z","shell.execute_reply":"2024-02-20T07:34:10.373259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_3 = final_df.iloc[763328: 1144992, :target_col_index] + final_df.iloc[763328: 1144992, target_col_index + 1:]\ny_3 = final_df[\"target\"][763328: 1144992]","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:31:58.437422Z","iopub.execute_input":"2024-02-20T07:31:58.437967Z","iopub.status.idle":"2024-02-20T07:32:02.169878Z","shell.execute_reply.started":"2024-02-20T07:31:58.437929Z","shell.execute_reply":"2024-02-20T07:32:02.168559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_3.fillna(0, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:32:34.436985Z","iopub.execute_input":"2024-02-20T07:32:34.437507Z","iopub.status.idle":"2024-02-20T07:32:42.930877Z","shell.execute_reply.started":"2024-02-20T07:32:34.437441Z","shell.execute_reply":"2024-02-20T07:32:42.929562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_3.to_csv(\"X_3.csv\")\ny_3.to_csv(\"y_3.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:51:14.382423Z","iopub.execute_input":"2024-02-20T07:51:14.383540Z","iopub.status.idle":"2024-02-20T07:56:23.965451Z","shell.execute_reply.started":"2024-02-20T07:51:14.383490Z","shell.execute_reply":"2024-02-20T07:56:23.963930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_4 = final_df.iloc[1144992: , :target_col_index] + final_df.iloc[1144992: , target_col_index + 1:]\ny_4 = final_df[\"target\"][1144992: ]","metadata":{"execution":{"iopub.status.busy":"2024-02-20T09:06:32.878309Z","iopub.execute_input":"2024-02-20T09:06:32.878906Z","iopub.status.idle":"2024-02-20T09:06:48.607236Z","shell.execute_reply.started":"2024-02-20T09:06:32.878862Z","shell.execute_reply":"2024-02-20T09:06:48.605987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_4.fillna(0, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T09:08:02.774329Z","iopub.execute_input":"2024-02-20T09:08:02.774911Z","iopub.status.idle":"2024-02-20T09:08:12.402984Z","shell.execute_reply.started":"2024-02-20T09:08:02.774864Z","shell.execute_reply":"2024-02-20T09:08:12.401539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_4","metadata":{"execution":{"iopub.status.busy":"2024-02-20T09:08:42.661476Z","iopub.execute_input":"2024-02-20T09:08:42.661941Z","iopub.status.idle":"2024-02-20T09:08:42.772841Z","shell.execute_reply.started":"2024-02-20T09:08:42.661907Z","shell.execute_reply":"2024-02-20T09:08:42.771289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_4.to_csv(\"X_4.csv\")\ny_4.to_csv(\"y_4.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-02-20T09:08:49.406304Z","iopub.execute_input":"2024-02-20T09:08:49.406932Z","iopub.status.idle":"2024-02-20T09:14:04.262047Z","shell.execute_reply.started":"2024-02-20T09:08:49.406866Z","shell.execute_reply":"2024-02-20T09:14:04.260553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from imblearn.combine import SMOTETomek\nos_smote = SMOTETomek(random_state=42)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:28:35.659028Z","iopub.execute_input":"2024-02-20T07:28:35.659545Z","iopub.status.idle":"2024-02-20T07:28:35.666004Z","shell.execute_reply.started":"2024-02-20T07:28:35.659509Z","shell.execute_reply":"2024-02-20T07:28:35.664515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_res_2, y_res_2 = os_smote.fit_resample(X_2, y_2)","metadata":{"execution":{"iopub.status.busy":"2024-02-20T07:28:42.116137Z","iopub.execute_input":"2024-02-20T07:28:42.116615Z","iopub.status.idle":"2024-02-20T07:29:05.109340Z","shell.execute_reply.started":"2024-02-20T07:28:42.116574Z","shell.execute_reply":"2024-02-20T07:29:05.106204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_res_2.to_csv(\"X_res_2.csv\")\ny_res_2.to_csv(\"y_res_2.csv\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"number_of_zeros = (y_res_1 == 0).sum()\nnumber_of_zeros","metadata":{"execution":{"iopub.status.busy":"2024-02-20T05:17:20.912830Z","iopub.execute_input":"2024-02-20T05:17:20.913273Z","iopub.status.idle":"2024-02-20T05:17:20.923939Z","shell.execute_reply.started":"2024-02-20T05:17:20.913239Z","shell.execute_reply":"2024-02-20T05:17:20.922366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"number_of_ones = (y_res_1 == 1).sum()\nnumber_of_ones","metadata":{"execution":{"iopub.status.busy":"2024-02-20T05:17:30.555000Z","iopub.execute_input":"2024-02-20T05:17:30.555488Z","iopub.status.idle":"2024-02-20T05:17:30.566470Z","shell.execute_reply.started":"2024-02-20T05:17:30.555453Z","shell.execute_reply":"2024-02-20T05:17:30.564917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ","metadata":{}},{"cell_type":"markdown","source":"BALANCED DATASET (with synthetic oversampling)","metadata":{}},{"cell_type":"code","source":"df_train = pd.concat([X_res_1, y_res_1], axis=1)\nplot_count(df_train, [\"target\"])","metadata":{"execution":{"iopub.status.busy":"2024-02-20T09:27:40.587706Z","iopub.execute_input":"2024-02-20T09:27:40.588211Z","iopub.status.idle":"2024-02-20T09:27:41.051628Z","shell.execute_reply.started":"2024-02-20T09:27:40.588169Z","shell.execute_reply":"2024-02-20T09:27:41.050149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training LightGBM\n\nMinimal example of LightGBM training is shown below.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nX_res_1 = pd.read_csv(\"/kaggle/working/X_res_1.csv\")\ny_res_1 = pd.read_csv(\"/kaggle/working/y_res_1.csv\")\ny_2 = pd.read_csv(\"/kaggle/working/y_2.csv\")\nX_2 = pd.read_csv(\"/kaggle/working/X_2.csv\")\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb_train = lgb.Dataset(X_train, label=y_train)\nlgb_valid = lgb.Dataset(X_valid, label=y_valid, reference=lgb_train)\n\nparams = {\n    \"boosting_type\": \"gbdt\",\n    \"objective\": \"binary\",\n    \"metric\": \"auc\",\n    \"max_depth\": 3,\n    \"num_leaves\": 31,\n    \"learning_rate\": 0.05,\n    \"feature_fraction\": 0.9,\n    \"bagging_fraction\": 0.8,\n    \"bagging_freq\": 5,\n    \"n_estimators\": 1000,\n    \"verbose\": -1,\n}\n\ngbm = lgb.train(\n    params,\n    lgb_train,\n    valid_sets=lgb_valid,\n    callbacks=[lgb.log_evaluation(50), lgb.early_stopping(10)]\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Evaluation with AUC and then comparison with the stability metric is shown below.","metadata":{}},{"cell_type":"code","source":"for base, X in [(base_train, X_train), (base_valid, X_valid), (base_test, X_test)]:\n    y_pred = gbm.predict(X, num_iteration=gbm.best_iteration)\n    base[\"score\"] = y_pred\n\nprint(f'The AUC score on the train set is: {roc_auc_score(base_train[\"target\"], base_train[\"score\"])}') \nprint(f'The AUC score on the valid set is: {roc_auc_score(base_valid[\"target\"], base_valid[\"score\"])}') \nprint(f'The AUC score on the test set is: {roc_auc_score(base_test[\"target\"], base_test[\"score\"])}')  ","metadata":{"execution":{"iopub.status.busy":"2024-02-16T01:44:33.994352Z","iopub.execute_input":"2024-02-16T01:44:33.995189Z","iopub.status.idle":"2024-02-16T01:44:50.489250Z","shell.execute_reply.started":"2024-02-16T01:44:33.995166Z","shell.execute_reply":"2024-02-16T01:44:50.488404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def gini_stability(base, w_fallingrate=88.0, w_resstd=-0.5):\n    gini_in_time = base.loc[:, [\"WEEK_NUM\", \"target\", \"score\"]]\\\n        .sort_values(\"WEEK_NUM\")\\\n        .groupby(\"WEEK_NUM\")[[\"target\", \"score\"]]\\\n        .apply(lambda x: 2*roc_auc_score(x[\"target\"], x[\"score\"])-1).tolist()\n    \n    x = np.arange(len(gini_in_time))\n    y = gini_in_time\n    a, b = np.polyfit(x, y, 1)\n    y_hat = a*x + b\n    residuals = y - y_hat\n    res_std = np.std(residuals)\n    avg_gini = np.mean(gini_in_time)\n    return avg_gini + w_fallingrate * min(0, a) + w_resstd * res_std\n\nstability_score_train = gini_stability(base_train)\nstability_score_valid = gini_stability(base_valid)\nstability_score_test = gini_stability(base_test)\n\nprint(f'The stability score on the train set is: {stability_score_train}') \nprint(f'The stability score on the valid set is: {stability_score_valid}') \nprint(f'The stability score on the test set is: {stability_score_test}')  ","metadata":{"execution":{"iopub.status.busy":"2024-02-16T01:44:50.491040Z","iopub.execute_input":"2024-02-16T01:44:50.491545Z","iopub.status.idle":"2024-02-16T01:44:51.280549Z","shell.execute_reply.started":"2024-02-16T01:44:50.491510Z","shell.execute_reply":"2024-02-16T01:44:51.279612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission\n\nScoring the submission dataset is below, we need to take care of new categories. Then we save the score as a last step. ","metadata":{}},{"cell_type":"code","source":"X_submission = data_submission[cols_pred].to_pandas()\nX_submission = convert_strings(X_submission)\ncategorical_cols = X_train.select_dtypes(include=['category']).columns\n\nfor col in categorical_cols:\n    train_categories = set(X_train[col].cat.categories)\n    submission_categories = set(X_submission[col].cat.categories)\n    new_categories = submission_categories - train_categories\n    X_submission.loc[X_submission[col].isin(new_categories), col] = \"Unknown\"\n    new_dtype = pd.CategoricalDtype(categories=train_categories, ordered=True)\n    X_train[col] = X_train[col].astype(new_dtype)\n    X_submission[col] = X_submission[col].astype(new_dtype)\n\ny_submission_pred = gbm.predict(X_submission, num_iteration=gbm.best_iteration)","metadata":{"execution":{"iopub.status.busy":"2024-02-16T02:04:45.781509Z","iopub.execute_input":"2024-02-16T02:04:45.781884Z","iopub.status.idle":"2024-02-16T02:04:45.848034Z","shell.execute_reply.started":"2024-02-16T02:04:45.781857Z","shell.execute_reply":"2024-02-16T02:04:45.847292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n    \"case_id\": data_submission[\"case_id\"].to_numpy(),\n    \"score\": y_submission_pred\n}).set_index('case_id')\nsubmission.to_csv(\"./submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-02-16T02:04:50.143357Z","iopub.execute_input":"2024-02-16T02:04:50.143712Z","iopub.status.idle":"2024-02-16T02:04:50.151580Z","shell.execute_reply.started":"2024-02-16T02:04:50.143681Z","shell.execute_reply":"2024-02-16T02:04:50.150534Z"},"trusted":true},"execution_count":null,"outputs":[]}]}