{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":28755,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"##### Stage 1 — Data understanding & memory engineering. ✅ Done. You worked through the customer/statement structure, the class imbalance and the 20× downsampling correction, the metric's business meaning, memory-safe loading (the whole CSV-to-Parquet saga), and then EDA: missingness by family, statement-count distribution, and categorical cardinality.\n\n##### Stage 2 — Implement the competition metric (M) yourself. Build M as a Python function — the normalized Gini plus top-4% capture, with the 20× negative weighting — and unit-test it against toy inputs where you know the right answer, before any modeling. The point: you must trust your own scoring before you can trust any model's validation score.\n\n##### Stage 3 — A dead-simple baseline. Last statement per customer + LightGBM, end to end, just to get a working pipeline and a first CV/leaderboard sanity check. Deliberately crude — the goal is a complete loop, not a good score.\n\n##### Stage 4 — Real feature engineering. Per-customer aggregations across the 13 statements: mean/std/min/max/last, deltas, lag features. This is where most of the actual score lives in this competition, more than model choice — and where all your EDA findings (missingness handling, the −1 sentinels, categorical vs numeric treatment, statement count as a feature) get used.\n\n##### Stage 5 — Model tuning. LightGBM/XGBoost specifics for this metric — custom eval function, categorical handling, early stopping, hyperparameters.\n\n##### Stage 6 — Cross-validation strategy. A validation scheme that doesn't leak and actually correlates with the leaderboard. Critical, and easy to get subtly wrong.\n\n##### Stage 7 — (Optional) Sequence modeling. A GRU or Transformer over the raw 13-month history — plays directly to your PyTorch/CV background, and is what several top solutions did alongside the gradient-boosted trees.\n\n##### Stage 8 — Ensembling. Combine models, and only at the very end, read the top write-ups to see what you missed.","metadata":{}},{"cell_type":"markdown","source":"# American Express Default Prediction\n\n## Stage 1 — Data Understanding & EDA","metadata":{}},{"cell_type":"markdown","source":"### Import Libraries","metadata":{}},{"cell_type":"code","source":"import os\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nimport pyarrow.parquet as pq\n\npd.set_option(\"display.max_columns\", 120)\npd.set_option(\"display.max_rows\", 120)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2026-06-29T11:16:46.682419Z","iopub.execute_input":"2026-06-29T11:16:46.682732Z","iopub.status.idle":"2026-06-29T11:16:47.091884Z","shell.execute_reply.started":"2026-06-29T11:16:46.682702Z","shell.execute_reply":"2026-06-29T11:16:47.090695Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Load Dataset","metadata":{}},{"cell_type":"code","source":"BASE_PATH = \"/kaggle/input/datasets/raddar/amex-data-integer-dtypes-parquet-format\"\n\nTRAIN_PATH = f\"{BASE_PATH}/train.parquet\"\nTEST_PATH = f\"{BASE_PATH}/test.parquet\"\n\ntrain = pd.read_parquet(TRAIN_PATH)\n\nprint(f\"Train Shape : {train.shape}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T11:17:18.716752Z","iopub.execute_input":"2026-06-29T11:17:18.717236Z","iopub.status.idle":"2026-06-29T11:17:36.046494Z","shell.execute_reply.started":"2026-06-29T11:17:18.717177Z","shell.execute_reply":"2026-06-29T11:17:36.045494Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Dataset Informations","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T09:58:23.774580Z","iopub.execute_input":"2026-06-29T09:58:23.775213Z","iopub.status.idle":"2026-06-29T09:58:23.849010Z","shell.execute_reply.started":"2026-06-29T09:58:23.775172Z","shell.execute_reply":"2026-06-29T09:58:23.848048Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T09:58:37.786839Z","iopub.execute_input":"2026-06-29T09:58:37.787237Z","iopub.status.idle":"2026-06-29T09:58:37.802318Z","shell.execute_reply.started":"2026-06-29T09:58:37.787203Z","shell.execute_reply":"2026-06-29T09:58:37.801504Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.describe().T","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T09:58:53.478902Z","iopub.execute_input":"2026-06-29T09:58:53.479290Z","iopub.status.idle":"2026-06-29T09:59:09.361711Z","shell.execute_reply.started":"2026-06-29T09:58:53.479256Z","shell.execute_reply":"2026-06-29T09:59:09.360776Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Missing Values","metadata":{}},{"cell_type":"code","source":"missing = (\n    train\n    .isnull()\n    .sum()\n    .sort_values(ascending=False)\n    .to_frame(\"Missing Count\")\n)\n\nmissing[\"Missing %\"] = (\n    missing[\"Missing Count\"] / len(train)\n) * 100\n\nmissing.head(20)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T09:59:31.160844Z","iopub.execute_input":"2026-06-29T09:59:31.161189Z","iopub.status.idle":"2026-06-29T09:59:32.172199Z","shell.execute_reply.started":"2026-06-29T09:59:31.161163Z","shell.execute_reply":"2026-06-29T09:59:32.171292Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.dtypes.value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T10:00:20.565702Z","iopub.execute_input":"2026-06-29T10:00:20.566013Z","iopub.status.idle":"2026-06-29T10:00:20.574065Z","shell.execute_reply.started":"2026-06-29T10:00:20.565988Z","shell.execute_reply":"2026-06-29T10:00:20.573337Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"int_features = train.select_dtypes(include=\"integer\").columns.tolist()\nfloat_features = train.select_dtypes(include=\"float\").columns.tolist()\nobject_features = train.select_dtypes(include=\"object\").columns.tolist()\n\nprint(f\"Total Integer Features : {len(int_features)}\")\n\nprint(f\"Total Float Features : {len(float_features)}\")\n\nprint(f\"Total Object Features : {len(object_features)}\")\nobject_features","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T10:01:53.457014Z","iopub.execute_input":"2026-06-29T10:01:53.457288Z","iopub.status.idle":"2026-06-29T10:01:54.578175Z","shell.execute_reply.started":"2026-06-29T10:01:53.457269Z","shell.execute_reply":"2026-06-29T10:01:54.577481Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"memory_mb = train.memory_usage(deep=True).sum() / 1024**2\n\nprint(f\"Memory Usage : {memory_mb:.2f} MB\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T10:03:00.517849Z","iopub.execute_input":"2026-06-29T10:03:00.518154Z","iopub.status.idle":"2026-06-29T10:03:02.005865Z","shell.execute_reply.started":"2026-06-29T10:03:00.518127Z","shell.execute_reply":"2026-06-29T10:03:02.004760Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Customer's data","metadata":{}},{"cell_type":"code","source":"n_customers = train[\"customer_ID\"].nunique()\n\nprint(f\"Total Unique Customers : {n_customers:,}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T10:04:12.660063Z","iopub.execute_input":"2026-06-29T10:04:12.660358Z","iopub.status.idle":"2026-06-29T10:04:13.238379Z","shell.execute_reply.started":"2026-06-29T10:04:12.660336Z","shell.execute_reply":"2026-06-29T10:04:13.237629Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"statement_counts = train.groupby(\"customer_ID\").size()\n\nstatement_counts.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T10:04:26.679780Z","iopub.execute_input":"2026-06-29T10:04:26.680059Z","iopub.status.idle":"2026-06-29T10:04:27.442831Z","shell.execute_reply.started":"2026-06-29T10:04:26.680039Z","shell.execute_reply":"2026-06-29T10:04:27.441975Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(8,4))\n\nstatement_counts.value_counts().sort_index().plot(kind=\"bar\")\n\nplt.xlabel(\"Number of Statements\")\nplt.ylabel(\"Customers\")\nplt.title(\"Statements per Customer\")\n\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T10:04:55.381805Z","iopub.execute_input":"2026-06-29T10:04:55.382090Z","iopub.status.idle":"2026-06-29T10:04:55.626030Z","shell.execute_reply.started":"2026-06-29T10:04:55.382062Z","shell.execute_reply":"2026-06-29T10:04:55.625436Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Target Distribution","metadata":{}},{"cell_type":"code","source":"train[\"target\"].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-06-29T10:05:46.974899Z","iopub.execute_input":"2026-06-29T10:05:46.975226Z","iopub.status.idle":"2026-06-29T10:05:47.108927Z","shell.execute_reply.started":"2026-06-29T10:05:46.975205Z","shell.execute_reply":"2026-06-29T10:05:47.108095Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}