{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":35332,"databundleVersionId":3723648,"sourceType":"competition"},{"sourceId":3727003,"sourceType":"datasetVersion","datasetId":2213609}],"dockerImageVersionId":30197,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AMEX EDA which makes sense ⭐️⭐️⭐️⭐️⭐️\n\nThis EDA analyzes the data and gives some insight which is useful for designing a machine learning pipeline and selecting a model.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport pickle, gc\nfrom matplotlib import pyplot as plt","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:29:45.343340Z","iopub.execute_input":"2025-05-08T09:29:45.343972Z","iopub.status.idle":"2025-05-08T09:29:45.349591Z","shell.execute_reply.started":"2025-05-08T09:29:45.343924Z","shell.execute_reply":"2025-05-08T09:29:45.348633Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# The labels\n\nWe start by reading the labels for the training data. There are neither missing values nor duplicated customer_IDs. Of the 458913 customer_IDs, 340000 (74 %) have a label of 0 (good customer, no default) and 119000 (26 %) have a label of 1 (bad customer, default).\n\nWe know that the good customers have been subsampled by a factor of 20; this means that in reality there are 6.8 million good customers. 98 % of the customers are good; 2 % are bad.\n\n**Insight:**\n- The classes are imbalanced. A StratifiedKFold for cross-validation is recommended.\n- Because the classes are imbalanced, accuracy would be a bad metric to evaluate a classifier. The [competition metric](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327464) is a mix of area under the roc curve (auc) and recall.","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ntrain_labels.head(2)","metadata":{"execution":{"iopub.status.busy":"2025-05-08T09:29:48.974774Z","iopub.execute_input":"2025-05-08T09:29:48.975860Z","iopub.status.idle":"2025-05-08T09:29:49.979697Z","shell.execute_reply.started":"2025-05-08T09:29:48.975818Z","shell.execute_reply":"2025-05-08T09:29:49.978626Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Check for missing data and duplicated customer_IDs\ntrain_labels.isna().any().any(), train_labels.customer_ID.duplicated().any()","metadata":{"execution":{"iopub.status.busy":"2025-05-08T09:29:55.195321Z","iopub.execute_input":"2025-05-08T09:29:55.195871Z","iopub.status.idle":"2025-05-08T09:29:55.325587Z","shell.execute_reply.started":"2025-05-08T09:29:55.195823Z","shell.execute_reply":"2025-05-08T09:29:55.324457Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"label_stats = pd.DataFrame({'absolute': train_labels.target.value_counts(),\n              'relative': train_labels.target.value_counts() / len(train_labels)})\nlabel_stats['absolute upsampled'] =  label_stats.absolute * np.array([20, 1])\nlabel_stats['relative upsampled'] = label_stats['absolute upsampled'] / label_stats['absolute upsampled'].sum()\nlabel_stats","metadata":{"execution":{"iopub.status.busy":"2025-05-08T09:29:59.643498Z","iopub.execute_input":"2025-05-08T09:29:59.643923Z","iopub.status.idle":"2025-05-08T09:29:59.669782Z","shell.execute_reply.started":"2025-05-08T09:29:59.643889Z","shell.execute_reply":"2025-05-08T09:29:59.668652Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# The data\n\nThe dataset of this competition has a considerable size. If you read the original csv files, the data barely fits into memory. That's why we read the data from @munumbutt's [AMEX-Feather-Dataset](https://www.kaggle.com/datasets/munumbutt/amexfeather). In this [Feather](https://arrow.apache.org/docs/python/feather.html) file, the floating point precision has been reduced from 64 bit to 16 bit. And reading a Feather file is faster than reading a csv file because the Feather file format is binary.\n\nThere are 5.5 million rows for training and 11 million rows of test data.","metadata":{}},{"cell_type":"code","source":"%%time\ntrain = pd.read_feather('../input/amexfeather/train_data.ftr')\ntest = pd.read_feather('../input/amexfeather/test_data.ftr')\nwith pd.option_context(\"display.min_rows\", 6):\n    display(train)\n    display(test)","metadata":{"execution":{"iopub.status.busy":"2025-05-08T09:30:02.983714Z","iopub.execute_input":"2025-05-08T09:30:02.984119Z","iopub.status.idle":"2025-05-08T09:30:52.965635Z","shell.execute_reply.started":"2025-05-08T09:30:02.984087Z","shell.execute_reply":"2025-05-08T09:30:52.964401Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The target column of the train dataframe corresponds to the target column of train_labels.csv. In the csv file of the train data, there is no target column; it has been joined into the Feather file as a convenience.\n\nS_2 is the statement date. All train statement dates are between March of 2017 and March of 2018 (13 months), and no statement dates are missing. All test statement dates are between April of 2018 and October of 2019. This means that the statement dates of train and test don't overlap:","metadata":{}},{"cell_type":"code","source":"print('Train statement dates: ', train.S_2.min(), train.S_2.max(), train.S_2.isna().any())\nprint('Test statement dates: ',  test.S_2.min(), test.S_2.max(), test.S_2.isna().any())\n","metadata":{"execution":{"iopub.status.busy":"2025-05-08T09:30:59.936097Z","iopub.execute_input":"2025-05-08T09:30:59.936615Z","iopub.status.idle":"2025-05-08T09:31:00.079531Z","shell.execute_reply.started":"2025-05-08T09:30:59.936548Z","shell.execute_reply":"2025-05-08T09:31:00.078490Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n\n**Insight:**\n- The test data come from a different phase in the economic cycle than the training data. Our models have no way of learning the effect of the economic cycle.","metadata":{}},{"cell_type":"code","source":"print(f'Train data memory usage: {train.memory_usage().sum() / 1e9} GBytes')\nprint(f'Test data memory usage:  {test.memory_usage().sum() / 1e9} GBytes')\n","metadata":{"execution":{"iopub.status.busy":"2025-05-08T09:31:02.987441Z","iopub.execute_input":"2025-05-08T09:31:02.988527Z","iopub.status.idle":"2025-05-08T09:31:03.015884Z","shell.execute_reply.started":"2025-05-08T09:31:02.988484Z","shell.execute_reply":"2025-05-08T09:31:03.014775Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The training data takes 2.2 GBytes of RAM. The test data is twice the size of the training data.\n\n**Insight:**\n- With that much data, we need to have an eye on memory efficiency. Avoid keeping unnecessary copies of the data in memory, and avoid keeping unnecessary copies of models!\n- Whereas most machine learning algorithms expect the whole training data to be in memory, we don't need to load all the test data at once. The test data can be processed in batches.\n- You may want to separate training and inference code into two notebooks so that you never have training and test data in memory at the same time.","metadata":{}},{"cell_type":"markdown","source":"The info function shows that most other features have missing values:\n","metadata":{}},{"cell_type":"code","source":"train.info(max_cols=200, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2025-05-08T09:31:06.249021Z","iopub.execute_input":"2025-05-08T09:31:06.249419Z","iopub.status.idle":"2025-05-08T09:31:11.852303Z","shell.execute_reply.started":"2025-05-08T09:31:06.249385Z","shell.execute_reply":"2025-05-08T09:31:11.851359Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insight:**\n- There are many columns with missing values: Dropping all columns which have missing values is not a sensible strategy.\n- There are many rows with missing values: Dropping all rows which have missing values is not a sensible strategy.\n- Many decision-tree based algorithms can deal with missing values. If we choose such a model, we don't need to change the missing values.\n- Neural networks and other estimators cannot deal with missing values. If we choose such a model, we need to impute values. See [this guide](https://www.kaggle.com/code/parulpandey/a-guide-to-handling-missing-values-in-python) for an overview of the many imputation options.\n- Most features are 16-bit floats. The original data (in the csv file) has higher precision. By rounding it to 16-bit precision, some information is lost. To make this information loss more tangible: Every float16 number between 1 and 2 is a multiple of 1/1024. These numbers have only three digits behind the decimal point! This precision is enough to start the competition; maybe we'll have to switch to higher precision towards the end.","metadata":{}},{"cell_type":"markdown","source":"# Counting the statements per customer","metadata":{}},{"cell_type":"markdown","source":"Now we can count how many rows (credit card statements) there are per customer. We see that 80 % of the customers have 13 statements; the other 20 % of the customers have between 1 and 12 statements.\n\n**Insight:** Our model will have to deal with a variable-sized input per customer (unless we simplify our life and look only at the most recent statement as @inversion suggests [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327094) or at the average over all statements).","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))\ntrain_sc = train.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Train statements per customer')\nax1.pie(train_sc, labels=train_sc.index)\nax1.set_title(train_sc.name)\ntest_sc = test.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Test statements per customer')\nax2.pie(test_sc, labels=test_sc.index)\nax2.set_title(test_sc.name)\nplt.show()\n\n# display(train.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Train statements per customer'))\n# display(train.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Test statements per customer'))\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-05-08T09:31:21.958716Z","iopub.execute_input":"2025-05-08T09:31:21.959109Z","iopub.status.idle":"2025-05-08T09:31:24.644127Z","shell.execute_reply.started":"2025-05-08T09:31:21.959076Z","shell.execute_reply":"2025-05-08T09:31:24.643062Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's find out when these customers got their last statement. The histogram of the last statement dates shows that every train customer got his last statement in March of 2018. The first four Saturdays (March 3, 10, 17, 24) have more statements than an average day.\n\nThe test customers are split in two: half of them got their last statement in April of 2019 and half in October of 2019. As was [discussed here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327602), the April 2019 data is used for the public leaderboard and the October 2019 data is used for the private leaderboard.","metadata":{}},{"cell_type":"code","source":"temp = train.S_2.groupby(train.customer_ID).max()\nplt.figure(figsize=(16, 4))\nplt.hist(temp, bins=pd.date_range(\"2018-03-01\", \"2018-04-01\", freq=\"d\"),\n         rwidth=0.8, color='#ffd700')\nplt.title('When did the train customers get their last statements?', fontsize=20)\nplt.xlabel('Last statement date per customer')\nplt.ylabel('Count')\nplt.gca().set_facecolor('#0057b8')\nplt.show()\ndel temp\n\ntemp = test.S_2.groupby(test.customer_ID).max()\nplt.figure(figsize=(16, 4))\nplt.hist(temp, bins=pd.date_range(\"2019-04-01\", \"2019-11-01\", freq=\"d\"),\n         rwidth=0.74, color='#ffd700')\nplt.title('When did the test customers get their last statements?', fontsize=20)\nplt.xlabel('Last statement date per customer')\nplt.ylabel('Count')\nplt.gca().set_facecolor('#0057b8')\nplt.show()\ndel temp","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-05-08T09:31:28.701690Z","iopub.execute_input":"2025-05-08T09:31:28.702097Z","iopub.status.idle":"2025-05-08T09:31:33.790202Z","shell.execute_reply.started":"2025-05-08T09:31:28.702063Z","shell.execute_reply":"2025-05-08T09:31:33.788410Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insight:** Although the data are a kind of time series, we cannot cross-validate with a TimeSeriesSplit because all training happens in the same month.\n\nFor most customers, the first and last statement is about a year apart. Together with the fact that we typically have 13 statements per customer, this indicates that the customers get one credit card statement every month.","metadata":{}},{"cell_type":"code","source":"temp = train.S_2.groupby(train.customer_ID).agg(['max', 'min'])\nplt.figure(figsize=(16, 3))\nplt.hist((temp['max'] - temp['min']).dt.days, bins=400, color='#ffd700')\nplt.xlabel('days')\nplt.ylabel('count')\nplt.title('Number of days between first and last statement of customer (train)', fontsize=20)\nplt.gca().set_facecolor('#0057b8')\nplt.show()\n\ntemp = test.S_2.groupby(test.customer_ID).agg(['max', 'min'])\nplt.figure(figsize=(16, 3))\nplt.hist((temp['max'] - temp['min']).dt.days, bins=400, color='#ffd700')\nplt.xlabel('days')\nplt.ylabel('count')\nplt.title('Number of days between first and last statement of customer (test)', fontsize=20)\nplt.gca().set_facecolor('#0057b8')\nplt.show()\ndel temp","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-05-08T09:31:41.944055Z","iopub.execute_input":"2025-05-08T09:31:41.944473Z","iopub.status.idle":"2025-05-08T09:31:48.149975Z","shell.execute_reply.started":"2025-05-08T09:31:41.944437Z","shell.execute_reply":"2025-05-08T09:31:48.148698Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"If we color every statement (i.e. row of train or test) according to the dataset it belongs (training, public lb, and private lb), we see that every dataset covers thirteen months. Train and test don't overlap, but public and private lb periods overlap.","metadata":{}},{"cell_type":"code","source":"temp = pd.concat([train[['customer_ID', 'S_2']], test[['customer_ID', 'S_2']]], axis=0)\ntemp.set_index('customer_ID', inplace=True)\ntemp['last_month'] = temp.groupby('customer_ID').S_2.max().dt.month\nlast_month = temp['last_month'].values\n\nplt.figure(figsize=(16, 4))\nplt.hist([temp.S_2[temp.last_month == 3],   # ending 03/18 -> training\n          temp.S_2[temp.last_month == 4],   # ending 04/19 -> public lb\n          temp.S_2[temp.last_month == 10]], # ending 10/19 -> private lb\n         bins=pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"MS\"),\n         label=['Training', 'Public leaderboard', 'Private leaderboard'],\n         stacked=True)\nplt.xticks(pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"QS\"))\nplt.xlabel('Statement date')\nplt.ylabel('Count')\nplt.title('The three datasets over time', fontsize=20)\nplt.legend()\nplt.show()\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2025-05-08T09:31:51.447248Z","iopub.execute_input":"2025-05-08T09:31:51.447757Z","iopub.status.idle":"2025-05-08T09:32:03.538855Z","shell.execute_reply.started":"2025-05-08T09:31:51.447718Z","shell.execute_reply":"2025-05-08T09:32:03.536756Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now we'll look at the distribution of missing values over time. B_29 is most interesting. Given the each of the three datasets has almost half a million customers, we see that until May of 2019 fewer than a tenth of the customers have a value for B_29. The other nine tenths are missing. Starting in June of 2019, we have B_29 data for almost every customer. \n\n**Insight:** The distribution of the missing B_29 differs between train and test datasets. Whereas in the training and public leaderboard data >90 % are missing, during the last five months of private leaderboard, we have B_29 data for almost every customer. If we use this feature, we should be prepared for surprises in the private leaderboard. Is it better to drop the feature?","metadata":{}},{"cell_type":"code","source":"for f in [ 'B_29', 'S_9','D_87']:#, 'D_88', 'R_26', 'R_27', 'D_108', 'D_110', 'D_111', 'B_39', 'B_42']:\n    temp = pd.concat([train[[f, 'S_2']], test[[f, 'S_2']]], axis=0)\n    temp['last_month'] = last_month\n    temp['has_f'] = ~temp[f].isna() \n\n    plt.figure(figsize=(16, 4))\n    plt.hist([temp.S_2[temp.has_f & (temp.last_month == 3)],   # ending 03/18 -> training\n              temp.S_2[temp.has_f & (temp.last_month == 4)],   # ending 04/19 -> public lb\n              temp.S_2[temp.has_f & (temp.last_month == 10)]], # ending 10/19 -> private lb\n             bins=pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"MS\"),\n             label=['Training', 'Public leaderboard', 'Private leaderboard'],\n             stacked=True)\n    plt.xticks(pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"QS\"))\n    plt.xlabel('Statement date')\n    plt.ylabel(f'Count of {f} non-null values')\n    plt.title(f'{f} non-null values over time', fontsize=20)\n    plt.legend()\n    plt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-05-08T09:32:20.849248Z","iopub.execute_input":"2025-05-08T09:32:20.849673Z","iopub.status.idle":"2025-05-08T09:32:25.441573Z","shell.execute_reply.started":"2025-05-08T09:32:20.849638Z","shell.execute_reply":"2025-05-08T09:32:25.439708Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# The categorical features\n\nAccording to the [data description](https://www.kaggle.com/competitions/amex-default-prediction/data), there are eleven categorical features. We plot histograms for target=0 and target=1. For the ten features which have missing values, the missing values are represented by the rightmost bar of the histogram.\n","metadata":{}},{"cell_type":"code","source":"cat_features = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\nplt.figure(figsize=(16, 16))\nfor i, f in enumerate(cat_features):\n    plt.subplot(4, 3, i+1)\n    temp = pd.DataFrame(train[f][train.target == 0].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=0')\n    temp = pd.DataFrame(train[f][train.target == 1].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=1')\n    plt.xlabel(f)\n    plt.ylabel('frequency')\n    plt.legend()\n    plt.xticks(temp.index, temp.value)\nplt.suptitle('Categorical features', fontsize=20, y=0.93)\nplt.show()\ndel temp\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-05-08T09:32:34.424172Z","iopub.execute_input":"2025-05-08T09:32:34.424666Z","iopub.status.idle":"2025-05-08T09:32:37.227776Z","shell.execute_reply.started":"2025-05-08T09:32:34.424626Z","shell.execute_reply":"2025-05-08T09:32:37.226627Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insight:**\n- Every feature has at most eight categories (including a nan category). One-hot encodings are feasible.\n- The distributions for target=0 and target=1 differ. This means that every feature gives some information about the target.\n","metadata":{}},{"cell_type":"markdown","source":"# The binary features\n\nTwo features are binary:\n- B_31 is always 0 or 1.\n- D_87 is always 1 or missing.","metadata":{}},{"cell_type":"code","source":"bin_features = ['B_31', 'D_87']\nplt.figure(figsize=(16, 4))\nfor i, f in enumerate(bin_features):\n    plt.subplot(1, 2, i+1)\n    temp = pd.DataFrame(train[f][train.target == 0].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=0')\n    temp = pd.DataFrame(train[f][train.target == 1].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=1')\n    plt.xlabel(f)\n    plt.ylabel('frequency')\n    plt.legend()\n    plt.xticks(temp.index, temp.value)\nplt.suptitle('Binary features', fontsize=20)\nplt.show()\ndel temp","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-05-08T09:32:44.755418Z","iopub.execute_input":"2025-05-08T09:32:44.755877Z","iopub.status.idle":"2025-05-08T09:32:45.268740Z","shell.execute_reply.started":"2025-05-08T09:32:44.755842Z","shell.execute_reply":"2025-05-08T09:32:45.267119Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insight:** If you impute missing values for D_87, don't fall into the trap of imputing the mean - the feature would become useless...","metadata":{}},{"cell_type":"markdown","source":"# The numerical features\n\nIf we plot histograms of the 175 numerical features, we see that they have all kinds of distributions:","metadata":{}},{"cell_type":"code","source":"cont_features = sorted([f for f in train.columns if f not in cat_features + bin_features + ['customer_ID', 'target', 'S_2']])\nprint(len(cont_features))\n# print(cont_features)\nncols = 4\nfor i, f in enumerate(cont_features):\n    if i % ncols == 0: \n        if i > 0: plt.show()\n        plt.figure(figsize=(16, 3))\n        if i == 0: plt.suptitle('Continuous features', fontsize=20, y=1.02)\n    plt.subplot(1, ncols, i % ncols + 1)\n    plt.hist(train[f], bins=200)\n    plt.xlabel(f)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-05-08T09:32:49.093171Z","iopub.execute_input":"2025-05-08T09:32:49.093581Z","iopub.status.idle":"2025-05-08T09:36:00.627764Z","shell.execute_reply.started":"2025-05-08T09:32:49.093527Z","shell.execute_reply":"2025-05-08T09:36:00.626646Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insight:** Histograms with white space at the left or right end can indicate that the data contain outliers. We will have to deal with these outliers. But are these data really outliers? Maybe they are, but they could as well be legitimate traces of rare events. We do not know...\n","metadata":{}},{"cell_type":"markdown","source":"# The artificial noise in the data\n\nThe data contain artificial noise. To understand the noise, we need to analyze the data at full precision. Because we cannot load the whole dataset at full precision into memory, we load only two interesting columns. For a complete analysis, see @raddar's [discussion post](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514).","metadata":{}},{"cell_type":"code","source":"def read_columns(name, features):\n    \"\"\"Read the specified columns of the train/test csv at full precision\"\"\"\n    chunksize = 1000000\n    chunklist = []\n    with pd.read_csv(f\"../input/amex-default-prediction/{name}_data.csv\", chunksize=chunksize) as reader:\n        for i, chunk in enumerate(reader):\n            chunk = chunk[features] # keep only selected columns\n            chunklist.append(chunk)\n            print(i, end=' ')\n            if i == 5: break\n        print()\n    df = pd.concat(chunklist, axis=0)\n    return df\n\ndf = read_columns('train', ['B_19', 'S_13'])\ndf.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:36:12.343798Z","iopub.execute_input":"2025-05-08T09:36:12.344335Z","iopub.status.idle":"2025-05-08T09:44:00.547044Z","shell.execute_reply.started":"2025-05-08T09:36:12.344293Z","shell.execute_reply":"2025-05-08T09:44:00.545809Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Let's look at B_19. All values are between 0 and 1.01. To get a high-resolution histogram, we spread it over eleven diagrams. The histogram show only rectangles of width 0.01, which indicate that the values are uniformly distributed in intervals of width 0.01, but every interval has another probability. This means that B_19 originally had some other range, but was scaled, rounded and got added some uniform noise by applying the following function:\n\n```\ndef anonymize(data):\n    data -= data.min()\n    data /= data.max()\n    data = data.round(2)\n    rng = np.random.default_rng()\n    data += rng.uniform(0, 0.01, len(data))\n    return data\n```\n","metadata":{}},{"cell_type":"code","source":"y = df.B_19\nfor i in np.linspace(0, 1, 11):\n    plt.figure(figsize=(16, 3))\n    plt.hist(y, bins=np.linspace(i, i+0.1, 101), rwidth=0.8, color='m')\n    plt.xticks(np.linspace(i, i+0.1, 11))\n    plt.title(f\"B_19 histogram, part {int(i*10+1)}\")\n    plt.show()\n","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:44:13.804262Z","iopub.execute_input":"2025-05-08T09:44:13.805764Z","iopub.status.idle":"2025-05-08T09:44:23.172313Z","shell.execute_reply.started":"2025-05-08T09:44:13.805716Z","shell.execute_reply":"2025-05-08T09:44:23.171016Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Insight:** We don't care about the scaling, we cannot do anything against the rounding, but we should remove the artificial noise, e.g. by applying a function such as `x['B_19'] = x['B_19'].apply(lambda t: np.floor(t*100))`","metadata":{}},{"cell_type":"markdown","source":"S_13 looks similar, the rectangles again have width 0.01, but they start at multiples of 1/1034. This means that S_13 originally had integer values in the range 0..1034 and was treated with\n```\ndef anonymize(data):\n    data -= data.min()\n    data /= data.max() # divides by 1034\n    rng = np.random.default_rng()\n    data += rng.uniform(0, 0.01, len(data))\n    return data\n```\nUnfortunately some rectangles overlap (e.g. between 0.68 and 0.7) so that we cannot remove the noise.","metadata":{}},{"cell_type":"code","source":"y = df.S_13\nfor i in np.linspace(0, 1, 11):\n    plt.figure(figsize=(16, 3))\n    plt.hist(y, bins=np.linspace(i, i+0.1, 101), rwidth=0.8, color='c')\n    plt.xticks(np.linspace(i, i+0.1, 11))\n    plt.title(f\"S_13 histogram, part {int(i*10+1)}\")\n    plt.show()","metadata":{"_kg_hide-input":true,"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:44:37.127996Z","iopub.execute_input":"2025-05-08T09:44:37.128436Z","iopub.status.idle":"2025-05-08T09:44:46.688932Z","shell.execute_reply.started":"2025-05-08T09:44:37.128399Z","shell.execute_reply":"2025-05-08T09:44:46.686997Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Data Cleaning – Handling Missing Values\n\nMissing data can significantly impact the quality of analysis and model performance. In this step, we identify and assess the extent of missing values across the dataset.\n\nTo ensure clean and consistent input for downstream modeling, we apply the following approach:\n\n- **Quantify the percentage of missing values** for each feature.\n- **Identify the top 20 features** with the highest proportion of missing entries.\n- **Prioritize these features** for imputation using strategies such as mean, median, or domain-specific techniques.\n\nThe summary below highlights the most affected features and helps guide robust preprocessing decisions.\n\n","metadata":{}},{"cell_type":"code","source":"# Check missing values\nmissing_values = train.isnull().mean().sort_values(ascending=False)\nmissing_values[missing_values > 0].head(20)  # Show top 20 columns with missing values\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:44:55.728001Z","iopub.execute_input":"2025-05-08T09:44:55.728415Z","iopub.status.idle":"2025-05-08T09:45:00.973664Z","shell.execute_reply.started":"2025-05-08T09:44:55.728379Z","shell.execute_reply":"2025-05-08T09:45:00.972597Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Group the features\nmissing = train.isnull().mean()\nmissing = missing[missing > 0]\n\n# Separate categorical, binary, and continuous features\ncat_features = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\nbin_features = ['B_31', 'D_87']\ncont_features = [f for f in train.columns if f not in cat_features + bin_features + ['customer_ID', 'target', 'S_2']]\n\nmissing_cat = [col for col in cat_features if col in missing.index]\nmissing_bin = [col for col in bin_features if col in missing.index]\nmissing_cont = [col for col in cont_features if col in missing.index]\n\nprint(\"Missing categorical features:\", missing_cat)\nprint(\"Missing binary features:\", missing_bin)\nprint(\"Missing continuous features:\", missing_cont[:10])  \n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:45:05.086712Z","iopub.execute_input":"2025-05-08T09:45:05.087170Z","iopub.status.idle":"2025-05-08T09:45:10.274716Z","shell.execute_reply.started":"2025-05-08T09:45:05.087133Z","shell.execute_reply":"2025-05-08T09:45:10.273273Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Categorical: Fill with the most frequent value (mode)\nfor col in missing_cat:\n    train[col].fillna(train[col].mode()[0], inplace=True)\n\n# Binary: Fill with the most frequent value (mode)\nfor col in missing_bin:\n    train[col].fillna(train[col].mode()[0], inplace=True)\n\n# Continuous: Fill with the median (mean is also possible, but median is more robust to outliers)\nfor col in missing_cont:\n    train[col].fillna(train[col].median(), inplace=True)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:45:14.491753Z","iopub.execute_input":"2025-05-08T09:45:14.492138Z","iopub.status.idle":"2025-05-08T09:45:35.247096Z","shell.execute_reply.started":"2025-05-08T09:45:14.492108Z","shell.execute_reply":"2025-05-08T09:45:35.245644Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Imputing Missing Values by Feature Type\n\nAfter identifying the missing values, we apply tailored imputation strategies based on feature types:\n- **Categorical features** are filled with their mode (most frequent value).\n- **Binary features** are also filled with mode for consistency.\n- **Continuous features** are filled with the median, which is more robust to outliers.\n\nA final validation confirms that all missing values have been successfully imputed.\n","metadata":{}},{"cell_type":"code","source":"# Final check to ensure all missing values have been filled\nmissing_after = train.isnull().mean()\nmissing_after[missing_after > 0] \n\n# If the output is empty, it means all missing values have been filled","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:45:39.330813Z","iopub.execute_input":"2025-05-08T09:45:39.331197Z","iopub.status.idle":"2025-05-08T09:45:44.445445Z","shell.execute_reply.started":"2025-05-08T09:45:39.331167Z","shell.execute_reply":"2025-05-08T09:45:44.444503Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Outlier Detection\n\nOutliers are data points that significantly deviate from the overall distribution of the dataset. These values may result from data entry errors, rare events, or fundamentally different generating processes.\n\nIn this section, we use the **Interquartile Range (IQR)** method to identify outliers in the continuous features. The IQR is the difference between the 75th percentile (Q3) and the 25th percentile (Q1) of a distribution. A data point is considered an outlier if it falls outside the following range:\n\n- **Lower bound** = Q1 − 1.5 × IQR  \n- **Upper bound** = Q3 + 1.5 × IQR\n\nOutliers can distort statistical analyses and negatively affect model performance. Therefore, it is crucial to detect and handle them appropriately. In this analysis, we visualize the distribution of selected features and identify the outliers. Depending on the downstream task, these outliers may be removed, transformed, or imputed.\n","metadata":{}},{"cell_type":"code","source":"outlier_summary = {}\n\nfor col in cont_features:\n    q1 = train[col].quantile(0.25)\n    q3 = train[col].quantile(0.75)\n    iqr = q3 - q1\n    lower = q1 - 1.5 * iqr\n    upper = q3 + 1.5 * iqr\n    outlier_count = ((train[col] < lower) | (train[col] > upper)).sum()\n    outlier_summary[col] = outlier_count\n\n# Top 5 features with the most outliers\ntop_outlier_cols = pd.Series(outlier_summary).sort_values(ascending=False).head(5)\ntop_outlier_cols\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:46:34.087910Z","iopub.execute_input":"2025-05-08T09:46:34.088522Z","iopub.status.idle":"2025-05-08T09:47:39.805824Z","shell.execute_reply.started":"2025-05-08T09:46:34.088480Z","shell.execute_reply":"2025-05-08T09:47:39.804616Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\nfor col in top_outlier_cols.index:\n    plt.figure(figsize=(12, 4))\n    sns.boxplot(data=train, x=train[col])\n    plt.title(f'Boxplot of {col} (outliers: {top_outlier_cols[col]})')\n    plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T09:48:51.167259Z","iopub.execute_input":"2025-05-08T09:48:51.167696Z","iopub.status.idle":"2025-05-08T09:49:32.491235Z","shell.execute_reply.started":"2025-05-08T09:48:51.167660Z","shell.execute_reply":"2025-05-08T09:49:32.489717Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Outlier Detection using Isolation Forest\n\nIn this step, we use the **Isolation Forest** algorithm to detect outliers in continuous features. This model-based method is more flexible and effective than IQR for identifying complex and high-dimensional anomalies.\n\nTo optimize performance on the large dataset, we applied the following strategies:\n\n- **Sampled 100,000 rows** from the training data to reduce runtime.\n- **Limited the analysis to the first 10 continuous features**.\n- **Reduced the number of trees** to 50 (`n_estimators=50`) to speed up training.\n- Enabled **parallel processing** using `n_jobs=-1` to utilize all CPU cores.\n\nThe algorithm labels each observation as `-1` (outlier) or `1` (inlier). We count the number of outliers detected for each selected feature.\n\nThis approach offers a scalable and efficient solution for detecting non-obvious or nonlinear outliers in large datasets.\n","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import IsolationForest\n\nisof_results = {}\nsample_train = train.sample(n=100_000, random_state=42)  # Alt küme\n\nfor col in cont_features[:10]:\n    clf = IsolationForest(n_estimators=50, contamination='auto', random_state=42, n_jobs=-1)\n    preds = clf.fit_predict(sample_train[[col]])\n    outlier_count = (preds == -1).sum()\n    isof_results[col] = outlier_count\n\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T10:05:39.082299Z","iopub.execute_input":"2025-05-08T10:05:39.082962Z","iopub.status.idle":"2025-05-08T10:05:56.013174Z","shell.execute_reply.started":"2025-05-08T10:05:39.082890Z","shell.execute_reply":"2025-05-08T10:05:56.011836Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Clustered Heatmap of Outlier Counts\n\nThis clustered heatmap provides a compact and informative visual comparison of the top 10 features with the highest number of outliers, as detected by both **IQR** and **Isolation Forest** methods.\n\n#### 🔍 Key Insights:\n- **Side-by-side method comparison**: Easily compare how aggressively each method identifies outliers for each feature.\n- **Color intensity**: Brighter hues indicate higher outlier counts, helping quickly spot anomalous features.\n- **Clustering**: Method-based grouping improves readability and pattern recognition across methods.\n\nThis visualization combines numeric annotations and color encoding to deliver both precision and interpretability, making it a powerful tool for highlighting discrepancies or consensus in outlier detection.\n","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Prepare data\nheatmap_data = combined_outliers.copy().head(10).T  # Top 10 features, methods as rows\n\n# Create a clustermap\nsns.set(font_scale=1.0)\nclustermap = sns.clustermap(\n    heatmap_data,\n    annot=True,\n    fmt=\"d\",\n    cmap=\"coolwarm\",\n    linewidths=0.5,\n    figsize=(12, 6),\n    col_cluster=False,  # Only cluster rows (methods)\n    row_cluster=False   # Set True if you want method clustering\n)\n\nclustermap.ax_heatmap.set_title(\"Clustered Heatmap of Outlier Counts (Top 10 Features)\", pad=20)\nplt.show()\n\n\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T11:32:11.045396Z","iopub.execute_input":"2025-05-08T11:32:11.046986Z","iopub.status.idle":"2025-05-08T11:32:12.097018Z","shell.execute_reply.started":"2025-05-08T11:32:11.046914Z","shell.execute_reply":"2025-05-08T11:32:12.095723Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Comparison of IQR-Based and Isolation Forest Outlier Counts\n\nTo better understand the consistency and behavior of both methods, we now combine the number of outliers detected by:\n\n- **IQR-based method**: A rule-based statistical approach  \n- **Isolation Forest**: A model-based anomaly detection algorithm\n\nThis comparison highlights which features are consistently identified as containing outliers, and where the methods diverge. \n","metadata":{}},{"cell_type":"code","source":"# IQR results (from previous step)\niqr_df = pd.Series(outlier_summary).sort_values(ascending=False)\niqr_df.name = \"IQR Outliers\"\n\n# Isolation Forest results (from previous step)\nisof_df = pd.Series(isof_results).sort_values(ascending=False)\nisof_df.name = \"Isolation Forest Outliers\"\n\n# Merge the two into a single DataFrame\ncombined_outliers = pd.concat([iqr_df, isof_df], axis=1)\ncombined_outliers = combined_outliers.dropna().astype(int)  # Drop any missing and convert to int\ncombined_outliers\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T10:09:52.804625Z","iopub.execute_input":"2025-05-08T10:09:52.805122Z","iopub.status.idle":"2025-05-08T10:09:52.825662Z","shell.execute_reply.started":"2025-05-08T10:09:52.805082Z","shell.execute_reply":"2025-05-08T10:09:52.824605Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Visual Comparison of Outlier Counts\n\nTo better understand how the two outlier detection methods differ across features, we create a horizontal bar chart showing the number of outliers identified by:\n\n- **IQR-based method**\n- **Isolation Forest**\n\nThis visualization helps us easily detect which features exhibit the largest disagreement or alignment between the two methods.  \n**The percentages help quantify the relative sensitivity of each method across different features.**\n","metadata":{}},{"cell_type":"code","source":"# Calculate outlier percentages for both methods\niqr_percentage = (iqr_df / len(train)) * 100\nisof_percentage = (isof_df / len(train)) * 100\n\n# Combine them into a new DataFrame\noutlier_percentages = pd.concat([iqr_percentage, isof_percentage], axis=1)\noutlier_percentages.columns = ['IQR Outlier %', 'Isolation Forest Outlier %']\noutlier_percentages = outlier_percentages.round(2)\noutlier_percentages\n\n# Align index to have consistent comparison\ncommon_index = iqr_df.index.intersection(isof_df.index)\niqr_percentage = iqr_percentage.loc[common_index]\nisof_percentage = isof_percentage.loc[common_index]\n\noutlier_percentages = pd.concat([iqr_percentage, isof_percentage], axis=1)\noutlier_percentages.columns = ['IQR Outlier %', 'Isolation Forest Outlier %']\noutlier_percentages = outlier_percentages.round(2)\noutlier_percentages.head(10)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T10:48:31.952811Z","iopub.execute_input":"2025-05-08T10:48:31.953231Z","iopub.status.idle":"2025-05-08T10:48:31.979184Z","shell.execute_reply.started":"2025-05-08T10:48:31.953194Z","shell.execute_reply":"2025-05-08T10:48:31.978050Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Outlier Percentage Comparison (IQR vs Isolation Forest)\n\nThis bar plot presents a comparative view of the outlier percentages detected by the IQR and Isolation Forest methods for key features.\n\n- **Blue bars** represent IQR-based detection.\n- **Orange bars** show Isolation Forest detection.\n- The height of each bar indicates the proportion of values flagged as outliers.\n- Labeled values on top of the bars enhance clarity and facilitate comparison.\n\nThis visual aids in identifying features where detection sensitivity varies significantly between the two methods, supporting more informed modeling decisions.\n\n","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Stil ayarı\nsns.set_style(\"whitegrid\")\n\n# Grafik ayarları\nplt.figure(figsize=(14, 6))\nax = outlier_percentages.plot(\n    kind='bar',\n    width=0.75,\n    color=['#4C72B0', '#DD8452'],\n    edgecolor='black',\n    figsize=(14, 6)\n)\n\n# Başlık ve etiketler\nplt.title(\"Outlier Percentages by Feature (IQR vs Isolation Forest)\", fontsize=16, weight='bold')\nplt.ylabel(\"Outlier Percentage (%)\", fontsize=12)\nplt.xlabel(\"Feature\", fontsize=12)\nplt.xticks(rotation=45, ha='right')\nplt.legend(title='Detection Method')\nplt.grid(axis='y', linestyle='--', alpha=0.7)\nplt.tight_layout()\n\n# Değer etiketleri (bar üstüne yazdırma)\nfor container in ax.containers:\n    ax.bar_label(container, fmt='%.1f', label_type='edge', fontsize=9, padding=3)\n\nplt.show()\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T11:42:00.754986Z","iopub.execute_input":"2025-05-08T11:42:00.755437Z","iopub.status.idle":"2025-05-08T11:42:01.269842Z","shell.execute_reply.started":"2025-05-08T11:42:00.755403Z","shell.execute_reply":"2025-05-08T11:42:01.268341Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔄 Outlier Overlap Ratio – IQR vs Isolation Forest\n\nTo evaluate how much the IQR and Isolation Forest methods agree on outlier detection, we compute the **overlap ratio** for each feature. This ratio tells us what portion of the detected outliers were flagged by both methods.\n\nSince the full training set is large (~5.5 million rows), we **sampled 100,000 rows** to reduce runtime and make the comparison more efficient.\n\nWe also used:\n- `n_estimators=50` to limit tree count in Isolation Forest,\n- `n_jobs=-1` to enable parallel computation.\n\nThe overlap ratio is calculated as:\n\n","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import IsolationForest\nimport pandas as pd\n\n# Use a random sample to improve performance\nsample_train = train.sample(n=100_000, random_state=42)\n\noverlap_ratio = {}\n\nfor col in combined_outliers.index:\n    # IQR outlier detection on sampled data\n    q1 = sample_train[col].quantile(0.25)\n    q3 = sample_train[col].quantile(0.75)\n    iqr = q3 - q1\n    lower = q1 - 1.5 * iqr\n    upper = q3 + 1.5 * iqr\n    iqr_outliers = ((sample_train[col] < lower) | (sample_train[col] > upper))\n\n    # Isolation Forest on the same sample\n    clf = IsolationForest(n_estimators=50, contamination='auto', random_state=42, n_jobs=-1)\n    preds = clf.fit_predict(sample_train[[col]])\n    isof_outliers = (preds == -1)\n\n    # Overlap ratio calculation\n    both_outliers = iqr_outliers & isof_outliers\n    union_outliers = iqr_outliers | isof_outliers\n\n    intersection_count = both_outliers.sum()\n    union_count = union_outliers.sum()\n\n    overlap_ratio[col] = intersection_count / union_count if union_count > 0 else 0\n\n# Convert to DataFrame\noverlap_df = pd.DataFrame.from_dict(overlap_ratio, orient='index', columns=['Overlap Ratio'])\noverlap_df = overlap_df.sort_values(by='Overlap Ratio', ascending=False)\noverlap_df.head(10)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T10:23:09.623935Z","iopub.execute_input":"2025-05-08T10:23:09.624480Z","iopub.status.idle":"2025-05-08T10:23:26.193064Z","shell.execute_reply.started":"2025-05-08T10:23:09.624437Z","shell.execute_reply":"2025-05-08T10:23:26.191976Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔍 Visualizing Outlier Overlap\n\nThe chart below visualizes the top 10 features with the highest agreement between IQR and Isolation Forest methods using a more vibrant color palette. Annotated values on each bar show the exact overlap ratio, making it easier to interpret consistency across methods.\n\nA ratio closer to **1.0** suggests that both techniques flagged similar observations as outliers. Lower ratios may indicate disagreement due to distributional differences or method sensitivity.\n\nThis visualization helps to:\n- Prioritize features for further inspection,\n- Decide which detection method may be more suitable per feature,\n- Understand consistency between statistical and model-based approaches.\n","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Use a colorful palette and add annotations\nplt.figure(figsize=(12, 6))\nbarplot = sns.barplot(\n    x=overlap_df.head(10).index,\n    y=overlap_df['Overlap Ratio'].head(10),\n    palette='cubehelix'\n)\n\n# Add value labels on top of bars\nfor i, val in enumerate(overlap_df['Overlap Ratio'].head(10)):\n    barplot.text(i, val + 0.02, f\"{val:.2f}\", ha='center', fontsize=11, fontweight='bold')\n\nplt.title('Top 10 Features by Outlier Overlap Ratio\\n(IQR vs Isolation Forest)', fontsize=16, weight='bold')\nplt.ylabel('Overlap Ratio')\nplt.xlabel('Feature')\nplt.ylim(0, 1.1)\nplt.grid(axis='y', linestyle='--', alpha=0.6)\nplt.tight_layout()\nplt.show()\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T10:56:01.636307Z","iopub.execute_input":"2025-05-08T10:56:01.636782Z","iopub.status.idle":"2025-05-08T10:56:02.031184Z","shell.execute_reply.started":"2025-05-08T10:56:01.636743Z","shell.execute_reply":"2025-05-08T10:56:02.028988Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Outlier Count Comparison Between Methods\n\nTo deepen the comparison between the IQR-based method and Isolation Forest, the following bar chart visualizes the **absolute number of outliers** identified for the **top 10 features**.\n\nThis enhanced visualization supports both technical and practical insights:\n\n- **Magnitude View:** Observe how many outliers are flagged per feature by each method.\n- **Method Agreement:** Spot where one method detects significantly more outliers than the other.\n- **Domain Insight Triggers:** Large gaps may indicate non-normal distributions, skewness, or data-specific irregularities requiring further inspection.\n\nThe use of a clean color palette and edge highlights improves clarity and contrast, making it easier to compare both techniques side by side.\n\n\n","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Set aesthetic style\nsns.set_style('whitegrid')\nplt.figure(figsize=(14, 7))\n\n# Plot with clear colors, spacing and annotation\nax = combined_outliers.head(10).plot(\n    kind='bar',\n    color=sns.color_palette(\"Set2\", 2),\n    edgecolor='black',\n    width=0.7\n)\n\n# Titles and labels\nplt.title('Top 10 Features by Outlier Counts\\n(IQR vs Isolation Forest)', fontsize=16, weight='bold', pad=20)\nplt.ylabel('Number of Outliers', fontsize=12)\nplt.xlabel('Feature', fontsize=12)\n\n# Axis ticks\nplt.xticks(rotation=45, ha='right', fontsize=11)\nplt.yticks(fontsize=11)\n\n# Grid and legend\nplt.grid(axis='y', linestyle='--', alpha=0.5)\nplt.legend(title='Method', fontsize=11, title_fontsize=12, loc='upper right')\nplt.tight_layout()\nplt.show()\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T11:52:43.978632Z","iopub.execute_input":"2025-05-08T11:52:43.979086Z","iopub.status.idle":"2025-05-08T11:52:44.353677Z","shell.execute_reply.started":"2025-05-08T11:52:43.979049Z","shell.execute_reply":"2025-05-08T11:52:44.351880Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Data Type Conversion\n\nProper data types are essential for memory optimization and correct data interpretation.  \nIn this section, we:\n\n- Cast categorical features to the `category` type  \n- Ensure numeric features are stored in appropriate `float` or `integer` formats\n","metadata":{}},{"cell_type":"code","source":"# All object-type columns excluding customer_ID\nobject_cols = [col for col in train.columns if train[col].dtype == 'object' and col != 'customer_ID']\n\nfor col in object_cols:\n    print(f\"\\nColumn: {col}\")\n    print(train[col].unique()[:10])  # Print the first 10 unique values\n\n# object tipinde customer_ID dışında hiç sütun yok","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T11:08:28.799458Z","iopub.execute_input":"2025-05-08T11:08:28.799910Z","iopub.status.idle":"2025-05-08T11:08:28.808537Z","shell.execute_reply.started":"2025-05-08T11:08:28.799873Z","shell.execute_reply":"2025-05-08T11:08:28.807304Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Columns with the data type 'category'\ncat_cols = [col for col in train.columns if train[col].dtype.name == 'category']\nprint(\"Categorical columns:\", cat_cols)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T11:08:31.886485Z","iopub.execute_input":"2025-05-08T11:08:31.886927Z","iopub.status.idle":"2025-05-08T11:08:31.895664Z","shell.execute_reply.started":"2025-05-08T11:08:31.886892Z","shell.execute_reply":"2025-05-08T11:08:31.894135Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Convert categorical columns if not already 'category' type\nfor col in cat_cols:\n    if train[col].dtype != 'category':\n        train[col] = train[col].astype('category')\n\n# Final check: display data types\nprint(train[cat_cols].dtypes)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-08T11:09:32.435953Z","iopub.execute_input":"2025-05-08T11:09:32.436377Z","iopub.status.idle":"2025-05-08T11:09:32.446591Z","shell.execute_reply.started":"2025-05-08T11:09:32.436342Z","shell.execute_reply":"2025-05-08T11:09:32.445018Z"}},"outputs":[],"execution_count":null}]}