{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div style=\"padding:20px;color:white;margin:0;font-size:175%;text-align:center;display:fill;border-radius:5px;background-color:#016CC9;overflow:hidden;font-weight:500\">D: The Default Rate captured at 4%</div>","metadata":{}},{"cell_type":"markdown","source":"As explained by the organizers [here](https://www.kaggle.com/competitions/amex-default-prediction/overview/evaluation), the competition's metric M is the average of two sub-metrics: G an D. We will focus here on the D part.\n","metadata":{}},{"cell_type":"markdown","source":"The organizers have provided us with the [code](https://www.kaggle.com/code/inversion/amex-competition-metric-python)  to calculate the metrics. For those who have been using the provided functions and would like to know more about how they work, this Notebook will gently go through the calculation details for the D part of the metric. \n","metadata":{}},{"cell_type":"markdown","source":"[AmbrosM](https://www.kaggle.com/ambrosm) has published a beautiful graphical \n[explanation](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327464) for the two components of the metric.","metadata":{}},{"cell_type":"markdown","source":"# <b><span style='color:#4B4B4B'>1 |</span><span style='color:#016CC9'> Submission file</span></b>","metadata":{}},{"cell_type":"markdown","source":"We are going to use a submission example and work on it. The following submission file is coming from the train customers. A model has been trained and applied to some of the train customers. We have added the column \"target\" with the true value coming from the file train_label.csv.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n\nsub = pd.read_csv('../input/xgbwfecublend/submission.csv')\nprint(sub.head(5).to_markdown())","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.079270Z","iopub.execute_input":"2022-07-01T06:30:09.079664Z","iopub.status.idle":"2022-07-01T06:30:09.248312Z","shell.execute_reply.started":"2022-07-01T06:30:09.079634Z","shell.execute_reply":"2022-07-01T06:30:09.247483Z"},"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, this submission file contains 3 columns: \n* customer_ID (these are some customers from the training data)\n* target (this comes from the train_label file provided by amex) \n* prediction (this is the results of a trained model prediction).\n","metadata":{}},{"cell_type":"markdown","source":"We actually have everything we need to calculate the amex score, so let's do it.","metadata":{}},{"cell_type":"markdown","source":"# <b><span style='color:#4B4B4B'>2 |</span><span style='color:#016CC9'> Score</span></b>","metadata":{}},{"cell_type":"markdown","source":"We are going to use the functions provided by the organizers. We simply extracted the two sub-metrics to make them callable independently.","metadata":{}},{"cell_type":"code","source":" def top_four_percent_captured(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n    df = (pd.concat([y_true, y_pred], axis='columns')\n             .sort_values('prediction', ascending=False))\n    df['weight'] = df['target'].apply(lambda x: 20 if x==0 else 1)\n    four_pct_cutoff = int(0.04 * df['weight'].sum())\n    df['weight_cumsum'] = df['weight'].cumsum()\n    df_cutoff = df.loc[df['weight_cumsum'] <= four_pct_cutoff]\n    return (df_cutoff['target'] == 1).sum() / (df['target'] == 1).sum()\n        \ndef weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n    df = (pd.concat([y_true, y_pred], axis='columns')\n            .sort_values('prediction', ascending=False))\n    df['weight'] = df['target'].apply(lambda x: 20 if x==0 else 1)\n    df['random'] = (df['weight'] / df['weight'].sum()).cumsum()\n    total_pos = (df['target'] * df['weight']).sum()\n    df['cum_pos_found'] = (df['target'] * df['weight']).cumsum()\n    df['lorentz'] = df['cum_pos_found'] / total_pos\n    df['gini'] = (df['lorentz'] - df['random']) * df['weight']\n    return df['gini'].sum()\n\ndef normalized_weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n    y_true_pred = y_true.rename(columns={'target': 'prediction'})\n    return weighted_gini(y_true, y_pred) / weighted_gini(y_true, y_true_pred)\n\ndef amex_metric(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n    g = normalized_weighted_gini(y_true, y_pred)\n    d = top_four_percent_captured(y_true, y_pred)\n\n    return 0.5 * (g + d)","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.249989Z","iopub.execute_input":"2022-07-01T06:30:09.250457Z","iopub.status.idle":"2022-07-01T06:30:09.266392Z","shell.execute_reply.started":"2022-07-01T06:30:09.250427Z","shell.execute_reply":"2022-07-01T06:30:09.265171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score = amex_metric(sub[['target']],sub[['prediction']])\nprint (f'The amex score is: {score:.3f}')","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.267650Z","iopub.execute_input":"2022-07-01T06:30:09.267991Z","iopub.status.idle":"2022-07-01T06:30:09.538550Z","shell.execute_reply.started":"2022-07-01T06:30:09.267940Z","shell.execute_reply":"2022-07-01T06:30:09.537433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have a score of **0.796**. Let's now look at both sub-scores separately.","metadata":{}},{"cell_type":"code","source":"G = normalized_weighted_gini(sub[['target']],sub[['prediction']])\nD= top_four_percent_captured(sub[['target']],sub[['prediction']])\n\nprint(f'The score split is as follow. G: {G:.3f} and D: {D:.3f}')","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.542955Z","iopub.execute_input":"2022-07-01T06:30:09.543480Z","iopub.status.idle":"2022-07-01T06:30:09.802817Z","shell.execute_reply.started":"2022-07-01T06:30:09.543450Z","shell.execute_reply":"2022-07-01T06:30:09.801556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is a big surprise. The score is not balanced, G is much higher than D. You can check it while CVing your models, on your validation data. In my case, **D is always worse than G**. D is in the 60s and G in the 90s. The barrier at 0.80 seems to be coming from D.","metadata":{}},{"cell_type":"markdown","source":"For this reason it is interesting to understand exactly how D is calculated, maybe this can give us some insight on improving the models or features selection. Maybe someone will crack D and break the 0.800 barrier!","metadata":{}},{"cell_type":"markdown","source":" # <b><span style='color:#4B4B4B'>3 |</span><span style='color:#016CC9'> Adjusting the subsampling</span></b>","metadata":{}},{"cell_type":"markdown","source":"To understand the metric calculation, we really need to understand the subsampling process. This was the most confusing part in my case. As stated by the organizers:\n\n**Note that the negative class has been subsampled for this dataset at 5%**\n\nIn the original data, the ratio of defaulting customers is vey low. Amex would need to provide us with files 20 times larger, in order to get the same number of defaulters (to provide us with enough signal to train our models), But the files are already gigantic.\nIn the train and test data, the ratio of non defaulting customers has been artificially decreased. To achieve this, amex has subsampled the negative class. In other words, **they have removed 19 out of 20** non-defaulting customers to increase the density of defaulting customers.\n\n **and thus receives a 20x weighting in the scoring metric**.\n But for the scoring, amex wants to reproduce the orginal conditions. To this effect, we need to apply the opposite of the subsampling. As we have lost the removed customers, we need to find an artificial adjustment. To do this, any non defaulting customer (target=0) will be artificially duplicated 20 times, or attributed a weight of 20, while a defaulting customer (target=1) keeps a weight of 1.\nIn other words, during the subsampling, 19 out of 20 non-defaulting customers are removed. During the scoring, to artificially recreate these customers, each non-defaulting customer is multiplied 20 times.\n \n So this is the thing to keep in mind, **during the scoring, a non defaulting customer (target=0) is \"multiplied\" 20 times** (through a weight).","metadata":{}},{"cell_type":"code","source":"sub['weight']=20-sub['target']*19\nprint(sub.head().to_markdown())","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.804525Z","iopub.execute_input":"2022-07-01T06:30:09.804855Z","iopub.status.idle":"2022-07-01T06:30:09.814394Z","shell.execute_reply.started":"2022-07-01T06:30:09.804811Z","shell.execute_reply":"2022-07-01T06:30:09.813253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have added a weight column. Think of the weight as a multiplication factor. Each non-defaulting customer (target=0) 'exists' 20 times. Each defaulting customers (target=1) 'exists' only once.","metadata":{}},{"cell_type":"code","source":"print(f'the raw length of the submission file is {len(sub)}')\nlength_after_adjustment = sub['weight'].sum()\nprint (f'the length of the adjusted submission file is {length_after_adjustment}')","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.815897Z","iopub.execute_input":"2022-07-01T06:30:09.816423Z","iopub.status.idle":"2022-07-01T06:30:09.827456Z","shell.execute_reply.started":"2022-07-01T06:30:09.816392Z","shell.execute_reply":"2022-07-01T06:30:09.826323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The raw submission files contain 91,782 customers, but the adjusted file, reproducing the original conditions, contains 1,386,879 customers (most of them are artificially created by multiplication)","metadata":{}},{"cell_type":"code","source":"rr=sub['target'].sum()/len(sub)\nprint(f'The raw ratio of defaulting customers is: {rr*100:.1f}%')\nar=(sub['target'].sum()/length_after_adjustment)\nprint(f'The adjusted ratio of defaulting customers is: {ar*100:.1f}%')","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.828680Z","iopub.execute_input":"2022-07-01T06:30:09.829557Z","iopub.status.idle":"2022-07-01T06:30:09.843731Z","shell.execute_reply.started":"2022-07-01T06:30:09.829526Z","shell.execute_reply":"2022-07-01T06:30:09.842739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfig, ax = plt.subplots(ncols=2,figsize=(15,10))\nsizes=[sub['target'].sum(),len(sub)-sub['target'].sum()]\nlabels=['Default','Non Default']\nax[0].pie(sizes, labels=labels, autopct='%1.1f%%',\n        shadow=True, startangle=90)\nax[0].axis('equal') \nax[0].set_title('subsampled')\nsizes=[sub['target'].sum(),length_after_adjustment-sub['target'].sum()]\nlabels=['Default','Non Default']\nax[1].pie(sizes, labels=labels, autopct='%1.1f%%',\n        shadow=True, startangle=90)\nax[1].axis('equal') \nax[1].set_title('adjusted for subsampling')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:09.845406Z","iopub.execute_input":"2022-07-01T06:30:09.846301Z","iopub.status.idle":"2022-07-01T06:30:10.072939Z","shell.execute_reply.started":"2022-07-01T06:30:09.846253Z","shell.execute_reply":"2022-07-01T06:30:10.070532Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the train and test files we are working on, the defaulting customers represent around 25% of the cutomers. But in the original files at amex, it is much lower (1.70% in this example). By adjusting the submission file, we get back to this original ratio. As a rough idea, in the train/test datasets, the ratio is 1 Default to 3 Non-Defaults. After adjustment, the ratio becomes 1 Default to 60 Non-Defaults.","metadata":{}},{"cell_type":"markdown","source":" # <b><span style='color:#4B4B4B'>4 |</span><span style='color:#016CC9'> 4% cut-off</span></b>","metadata":{}},{"cell_type":"markdown","source":"We will first sort the customers by prediction (probability of default) from highest to lowest.","metadata":{}},{"cell_type":"code","source":"sub_sorted=sub.sort_values('prediction',ascending=False)\nprint(sub_sorted.head(5).to_markdown())","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:10.075247Z","iopub.execute_input":"2022-07-01T06:30:10.076360Z","iopub.status.idle":"2022-07-01T06:30:10.129699Z","shell.execute_reply.started":"2022-07-01T06:30:10.076300Z","shell.execute_reply":"2022-07-01T06:30:10.128530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We add a column weight_cumsum. This is a counter of the adjusted number of customers up to this row. ","metadata":{}},{"cell_type":"code","source":"sub_sorted['weight_cumsum']=sub_sorted['weight'].cumsum()\nprint(sub_sorted.head(5).to_markdown())\nprint(sub_sorted.tail(5).to_markdown())\n","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:10.132794Z","iopub.execute_input":"2022-07-01T06:30:10.133680Z","iopub.status.idle":"2022-07-01T06:30:10.145254Z","shell.execute_reply.started":"2022-07-01T06:30:10.133649Z","shell.execute_reply":"2022-07-01T06:30:10.144333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will now select the first 4% of the sorted and adjusted customers. ","metadata":{}},{"cell_type":"code","source":"sub_cutoff=sub_sorted.loc[sub_sorted['weight_cumsum'] <= (0.04*length_after_adjustment)]","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:10.146310Z","iopub.execute_input":"2022-07-01T06:30:10.147010Z","iopub.status.idle":"2022-07-01T06:30:10.158414Z","shell.execute_reply.started":"2022-07-01T06:30:10.146981Z","shell.execute_reply":"2022-07-01T06:30:10.157291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The idea here is to select the 4% customers with the highest probability of defaulting according to the model.","metadata":{}},{"cell_type":"code","source":"print(f'cut_off at : {len(sub_cutoff)/len(sub)*100:.1f}%')","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:10.159817Z","iopub.execute_input":"2022-07-01T06:30:10.160352Z","iopub.status.idle":"2022-07-01T06:30:10.165338Z","shell.execute_reply.started":"2022-07-01T06:30:10.160324Z","shell.execute_reply":"2022-07-01T06:30:10.164318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we have actually selected much more than 4% of the submission file, we have kept 19.3% of the submission file. This is explained by the adjustment process. To make it clearer, we will draw the cutoff post and pre adjustment.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport seaborn as sns\nfig, ax = plt.subplots(figsize=(15,10))\nl=len(sub_sorted)\nax.scatter(sub_sorted['weight_cumsum'],sub_sorted['prediction'],s=1)\nax.set_title('sorted prediction. cut-off after adjustment',fontsize = 12)\nplt.xlabel(\"prediction rank\",fontsize = 12)\nplt.ylabel(\"default prediction\",fontsize = 12)\nplt.ticklabel_format(style='plain')\nplt.xlim(left=0)\nplt.axvline(x=0.04*length_after_adjustment,color='r')\nplt.text(100000,0.8,'The red line is the 4% cutoff (after adjustment). The customers on the left of this line are selected.', fontsize = 12)\n \n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:10.166704Z","iopub.execute_input":"2022-07-01T06:30:10.167033Z","iopub.status.idle":"2022-07-01T06:30:10.411613Z","shell.execute_reply.started":"2022-07-01T06:30:10.167006Z","shell.execute_reply":"2022-07-01T06:30:10.410874Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"However, if we represent the raw submission file, the 4% cutoff looks like this.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport seaborn as sns\nfig, ax = plt.subplots(figsize=(15,10))\nl=len(sub_sorted)\nax.scatter(np.arange(l),sub_sorted['prediction'],s=1)\nax.set_title('sorted prediction, how the cuttoff looks like on the raw data',fontsize = 12)\nplt.xlabel(\"prediction rank\",fontsize = 12)\nplt.xlim(left=0)\nplt.ylabel(\"default prediction\",fontsize = 12)\nplt.axvline(x=len(sub_cutoff),color='r')\nplt.text(20000,0.8,'The red line is the 4% cutoff. The customers on the left of this line are selected.', fontsize = 12)\n \n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:10.412784Z","iopub.execute_input":"2022-07-01T06:30:10.413215Z","iopub.status.idle":"2022-07-01T06:30:10.649459Z","shell.execute_reply.started":"2022-07-01T06:30:10.413179Z","shell.execute_reply":"2022-07-01T06:30:10.648270Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wait a minute, what happened there? the red line has shifted! instead of selecting 4% of the submission file, we are selecting 19.3%! This is all due to the adjustment process. It is not a proportional dilatation of the curve. The customers on the right have far more chance to be non-defaulters and getting multiplied. So the dilatation mostly happens on the right side of the red line and the fraction selected goes from 19.3% to 4% post-adjustment.","metadata":{}},{"cell_type":"markdown","source":" # <b><span style='color:#4B4B4B'>5 |</span><span style='color:#016CC9'> D calculation</span></b>","metadata":{}},{"cell_type":"markdown","source":"Now that we have selected these 4% customers, most likely to default according to the model, we want to know if we captured most of the true defaulters. This is called the True Positive Rate. It is the ratio of True Positive captured within the 4% threshold to the total number of True Positives.\nIt's not really clear where this 4% comes from, it may be a fixed number that amex is using (maybe regulatory). So amex wants us to produce a model that captures as many True Positives as possible within this 4% limit. This is exactly what D measures.","metadata":{}},{"cell_type":"markdown","source":"First we calculate the number of True Positive (target=1) within this 4% cutoff. Then we simply divide it by the total number of Positive in the submission file.","metadata":{}},{"cell_type":"code","source":"TP=(sub_cutoff['target']==1).sum()# Number of True Positives within the 4% cuttoff\nP=(sub['target']==1).sum() #Total number of True Positives\n\nTPR=TP/P # Ratio to get the True Positive Rate\nprint(f'True Positive Rate with a cutoff at 4%: {TPR:.3f}')","metadata":{"execution":{"iopub.status.busy":"2022-07-01T06:30:10.650769Z","iopub.execute_input":"2022-07-01T06:30:10.651094Z","iopub.status.idle":"2022-07-01T06:30:10.660006Z","shell.execute_reply.started":"2022-07-01T06:30:10.651064Z","shell.execute_reply":"2022-07-01T06:30:10.658870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We get **0.667**. We have recalculated D in the same way as the function provided by amex !","metadata":{}},{"cell_type":"markdown","source":"Now the description of D in the [Evaluation part](https://www.kaggle.com/competitions/amex-default-prediction/overview/evaluation)  of the competition is clear:\n\n**The default rate captured at 4% is the percentage of the positive labels (defaults) captured within the highest-ranked 4% of the predictions**\n\nThis is done after adjustment for subsampling.\nThe wider the cutoff (5%, 6%...), the more True Positives are being captured, but unwanted False Positives will also be captured.","metadata":{}},{"cell_type":"markdown","source":" # <b><span style='color:#4B4B4B'>6 |</span><span style='color:#016CC9'> Conclusion</span></b>","metadata":{}},{"cell_type":"markdown","source":"Note that D=0.667 is not great, this means that the model missed a third of the defaulters when a 4% cuttoff was applied. It seems that this is where there is room for progress. \nIt would really be interesting to see each part of the score, G and D, on the LeaderBoard. Which sub-metric explains the gap from 0.795 and 0.800? Are we improving our models on the G or D part? You can see an example of the learning curves for both sub-metrics [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/334157)","metadata":{}}]}