{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceType":"competition","sourceId":105399,"databundleVersionId":12733338,"isSourceIdPinned":false}],"dockerImageVersionId":31042,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Competition overview","metadata":{}},{"cell_type":"markdown","source":"**Overview**\n\nThis academic project summarizes a machine learning competition focused on building a flight recommendation system for business travelers. The core problem is to predict which flight a business traveler will choose from a list of search results, a task that differs from leisure travel due to the complexity of a business traveler's decision-making process. These decisions are influenced by a combination of corporate policies, meeting schedules, expense compliance, and personal convenience.\n\n**Competition Goal and Technical Challenge**\n\nThe primary goal is to solve a group-wise ranking problem. The model must analyze a set of flight options for a single user search session and rank them according to the likelihood of being chosen. The key technical challenge is the varying number of flight options per search session, which can range from a handful to thousands. This requires a model capable of effectively ranking a diverse and dynamic set of alternatives to accurately identify the most suitable option for each traveler.\n\n**Evaluation and Submission**\n\nThe model's performance is evaluated using the HitRate@3 metric. This metric measures the percentage of search queries where the model's top three ranked predictions include the flight that was actually selected by the user. Importantly, this metric is only calculated on search groups containing more than 10 flight options, focusing the evaluation on the more challenging ranking scenarios.\n\nThe training data is provided as a binary classification problem, where a selected=1 flag identifies the chosen flight within each search session. For submission, the model's output (typically a score or probability) must be converted into a rank (1, 2, 3, etc.) for every flight option within each search session. These ranks must be a valid permutation of integers, ensuring each flight option receives a unique rank.","metadata":{}},{"cell_type":"markdown","source":"**[Competition Link](https://www.kaggle.com/competitions/aeroclub-recsys-2025/overview)**","metadata":{}},{"cell_type":"markdown","source":"# Dataset Description","metadata":{}},{"cell_type":"markdown","source":"This dataset provides information on flight booking options for business travelers, including user and company details, pricing, flight timing, and service characteristics. The primary goal is to predict user flight selection preferences by ranking flight options within a given search session.\n\n## Data Structure\n\nThe dataset is structured around flight search sessions, where each session is identified by `ranker_id`. The main data is provided in `.parquet` format for training and testing, with additional raw JSON files available for feature engineering. The target variable, selected, is a binary indicator in the training data, marking the single flight option chosen by the user. The submission requires ranking all flight options for each session in the test set, with a rank of 1 indicating the most likely choice.\n\n**Main Data**\n\n- `train.parquet` - train data\n- `test.parquet` - test data\n- `sample_submission.parquet` - submission example\n\n**JSONs Raw Additional Data**\n\n- `jsons_raw.tar.kaggle` - Archived raw data in JSONs files (150K files, ~50GB). To use the file as a regular `.gz` archive you should manually change extension to '`.gz`'. Example `jsons_raw.tar.kaggle` -> `jsons_raw.tar.gz`\n- `jsons_structure.md` - JSONs raw data structure description","metadata":{}},{"cell_type":"markdown","source":"## Data Dictionary","metadata":{}},{"cell_type":"markdown","source":"This data dictionary describes the columns found in the main dataset files (`train.parquet` and `test.parquet`). The data is organized into logical sections for clarity.","metadata":{}},{"cell_type":"markdown","source":"**Training Data Target**\n\nIn the training data, the selected column is binary:\n- 1 = This flight was chosen by the traveler\n- 0 = This flight was not chosen\n\n**Important:** There is exactly one row with selected=1 per user search request (`ranker_id`). Each row within a `ranker_id` group represents a different flight option returned by the search system for that specific route and date.","metadata":{}},{"cell_type":"markdown","source":"**Training data example:**","metadata":{}},{"cell_type":"markdown","source":"
\nId,ranker_id,selected\n100,abc123,0     # Flight option 1 - not chosen\n101,abc123,0     # Flight option 2 - not chosen  \n102,abc123,1,    # Flight option 3 - SELECTED by user\n103,abc123,0     # Flight option 4 - not chosen\n
","metadata":{}},{"cell_type":"markdown","source":"### Column Descriptions","metadata":{}},{"cell_type":"markdown","source":"**Identifiers and Metadata**","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
Column NameData TypeDescription
IdintA unique identifier for each individual flight option.
ranker_idobjectA group identifier for each flight search session. This is the key for grouping flight options.
profileIdobjectA unique identifier for the user.
companyIDobjectA unique identifier for the user's company.
","metadata":{}},{"cell_type":"markdown","source":"**User and Company Information**","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Column NameData TypeDescription
sexobjectThe user's gender.
nationalityobjectThe user's nationality or citizenship.
frequentFlyerobjectThe user's frequent flyer program status.
isVipboolA binary indicator of whether the user has VIP status.
bySelfboolA binary indicator of whether the user booked the flight independently.
isAccess3DboolA binary marker for an internal feature.
corporateTariffCodeobjectThe corporate tariff code associated with company travel policies.
","metadata":{}},{"cell_type":"markdown","source":"**Search and Pricing Information**","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Column NameData TypeDescription
searchRouteobjectThe flight route, either a single direction or a round trip (separated by \"/\").
requestDatedatetimeThe date and time when the flight search was performed.
totalPricefloatThe total cost of the flight ticket.
taxesfloatThe taxes and fees component of the ticket price.
pricingInfo_isAccessTPboolA binary indicator of compliance with corporate travel policy.
pricingInfo_passengerCountintThe number of passengers for this booking option.
","metadata":{}},{"cell_type":"markdown","source":"**Flight Timing and Duration**","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Column NameData TypeDescription
legs0_departureAtdatetimeThe departure time for the outbound flight.
legs0_arrivalAtdatetimeThe arrival time for the outbound flight.
legs0_durationintThe total duration of the outbound flight in minutes.
legs1_departureAtdatetimeThe departure time for the return flight.
legs1_arrivalAtdatetimeThe arrival time for the return flight.
legs1_durationintThe total duration of the return flight in minutes.
","metadata":{}},{"cell_type":"markdown","source":"**Flight Segments (Example for Leg 0, Segment 0)**","metadata":{}},{"cell_type":"markdown","source":"Note: This structure is repeated for each leg (`legs0, legs1`) and each segment (`segments0 through segments3`).","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Column NameData TypeDescription
legs0_segments0_departureFrom_airport_iataobjectThe IATA code for the departure airport of the first segment.
legs0_segments0_arrivalTo_airport_iataobjectThe IATA code for the arrival airport of the first segment.
legs0_segments0_arrivalTo_airport_city_iataobjectThe IATA code for the arrival city of the first segment.
legs0_segments0_marketingCarrier_codeobjectThe marketing airline code.
legs0_segments0_operatingCarrier_codeobjectThe operating airline code (the actual carrier).
legs0_segments0_aircraft_codeobjectThe code for the aircraft type.
legs0_segments0_flightNumberintThe flight number.
legs0_segments0_durationintThe duration of this specific segment in minutes.
","metadata":{}},{"cell_type":"markdown","source":"**Service Characteristics (Example for Leg 0, Segment 0)**","metadata":{}},{"cell_type":"markdown","source":"Note: This structure is repeated for each leg and segment.","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Column NameData TypeDescription
legs0_segments0_baggageAllowance_quantityfloatThe baggage allowance, which is a piece count for small numbers or weight in kg for large numbers.
legs0_segments0_baggageAllowance_weightMeasurementTypeobjectThe type of baggage weight measurement.
legs0_segments0_cabinClassfloatThe cabin class of service (e.g., 1.0 = economy, 2.0 = business).
legs0_segments0_seatsAvailableintThe number of seats available on the flight.
","metadata":{}},{"cell_type":"markdown","source":"**Cancellation and Exchange Rules**","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Column NameData TypeDescription
miniRules0_monetaryAmountfloatThe monetary penalty for cancellation (Rule 0).
miniRules0_percentagefloatThe percentage penalty for cancellation.
miniRules0_statusInfosobjectThe cancellation rule status (e.g., \"0\" = no cancellation allowed).
miniRules1_monetaryAmountfloatThe monetary penalty for exchange (Rule 1).
miniRules1_percentagefloatThe percentage penalty for exchange.
miniRules1_statusInfosobjectThe exchange rule status.
","metadata":{}},{"cell_type":"markdown","source":"**Target Variable**","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n\n
Column NameData TypeDescription
selectedintIn training data: A binary variable indicating the selected flight option (1 = selected, 0 = not selected).
In submission: An integer rank for each flight option (1 = best, 2 = second best, etc.).
","metadata":{}},{"cell_type":"markdown","source":"**[Dataset Link](https://www.kaggle.com/competitions/aeroclub-recsys-2025/data)**","metadata":{}},{"cell_type":"markdown","source":"# Submission Requirements","metadata":{}},{"cell_type":"markdown","source":"The submission file for this project must contain a ranked list of flight options for each search session. Instead of the binary selected variable from the training data, the file must include an integer rank for every flight option within a `ranker_id` group. The best option is ranked 1, the second best is 2, and so on. The submission must preserve the exact row order of the `test.csv` file and include all flight options. Ranks within each `ranker_id` group must be unique, form a complete permutation from 1 to N (where N is the number of options in the group), and be integer values greater than or equal to 1. The submission is validated on these requirements.","metadata":{}},{"cell_type":"markdown","source":"**Submission File Format**","metadata":{}},{"cell_type":"markdown","source":"The submission file must contain ranks (not probabilities) for each flight option:","metadata":{}},{"cell_type":"markdown","source":"
\nId,ranker_id,selected\n100,abc123,4\n101,abc123,2\n102,abc123,1\n103,abc123,3\n
","metadata":{}},{"cell_type":"markdown","source":"Where:\n- `Id` matches the row identifier from the test set\n- `ranker_id` is the search session identifier (same as in `test.csv`)\n- `selected` is the rank you assign (1 = best option, 2 = second best, etc.)\n\n**Important:** Maintain the exact same row order as in test.csv\n\nIn this example above, the model predicts that:\n\n- Row 102 (Id=102) is the best option → Rank 1\n- Row 101 (Id=101) is second best → Rank 2\n- Row 103 (Id=103) is third best → Rank 3\n- Row 100 (Id=100) is the worst option → Rank 4","metadata":{}},{"cell_type":"markdown","source":"The submission file is a simple table with three columns that link a specific flight option to its predicted rank within a search session.","metadata":{}},{"cell_type":"markdown","source":"\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
Column NameData TypeDescription
IdintThe unique identifier for each flight option, matching the Id from the test.parquet file.
ranker_idobjectThe identifier for the flight search session, which groups flight options together.
selectedintThe predicted rank of the flight option within its ranker_id group. A rank of 1 indicates the most preferred option.
","metadata":{}},{"cell_type":"markdown","source":"**Submission Requirements**\n\n- **Preserve row order:** Maintain the exact same row order as in `test.csv`\n- **Complete rankings:** For each user search request, you must rank ALL flight options returned by the search system\n- **Valid permutation:** Ranks within each `ranker_id` must be a valid permutation (1, 2, 3, …, N) where N is the number of rows in that group\n- **No duplicate ranks:** Each row within a `ranker_id` group must have a unique rank\n- **Integer values:** All ranks must be integers ≥ 1","metadata":{}},{"cell_type":"markdown","source":"**Example for one user search request:**","metadata":{}},{"cell_type":"markdown","source":"
\nTraining data shows:\nranker_id: abc123 → Row 102 was chosen (selected=1)\n\nYour submission:\nranker_id: abc123\n├── Row 100 → Rank 4 (worst option)\n├── Row 101 → Rank 2 (second best)\n├── Row 102 → Rank 1 (best - correctly predicted!)\n└── Row 103 → Rank 3 (third best)\n
","metadata":{}},{"cell_type":"markdown","source":"# Pipeline Overview","metadata":{}},{"cell_type":"markdown","source":"**Overview of the Ranking Pipeline**\n\nThis pipeline uses an XGBoost model to address the flight recommendation ranking challenge. The pipeline is built using the Polars library for efficient data manipulation and feature engineering, and it is specifically designed to solve the group-wise ranking problem described in the competition brief.\n\nThe pipeline follows a standard machine learning workflow, which includes data loading, extensive feature engineering, model training, evaluation, and final prediction generation. The core concept behind using XGBoost for this task is gradient boosting, an ensemble learning method where new models are added sequentially to correct the errors of previous models, resulting in a highly accurate and robust predictive model.\n\n**Data Handling and Feature Engineering**\n\nThe code uses the Polars library, which is optimized for fast data processing, to handle the large-scale dataset. The training and test data are loaded and concatenated into a single DataFrame to ensure consistency in feature engineering across both datasets.\n\nThe most significant part of the pipeline is feature engineering. This process involves creating new, informative variables from the raw data to give the model better signals for distinguishing between chosen and unchosen flights. Key features engineered in this pipeline include:\n\n- **Price and Duration Features:** Creating new metrics like `price_per_tax, tax_rate, and log_price` to capture different aspects of cost. The `legs0_duration` and `legs1_duration` are converted to minutes, and a duration_ratio is calculated to compare legs.\n- **Trip and Route Features:** Identifying whether a flight is a one-way trip (`is_one_way`) and whether it belongs to a known popular route (`is_popular_route`).\n- **Ranking Features:** The pipeline also generates features relative to a given search session. For example, `price_rank` and `duration_rank` provide a direct ranking of each flight option within its group, while `is_cheapest` and `is_min_segments` are binary features that highlight key attributes of a flight relative to its peers.\n- **Categorical Encoding**: Important categorical features, such as `marketingCarrier_code` and `aircraft_code`, are converted into integer representations using a dense rank, which is a common practice for tree-based models like XGBoost.\n\n**Model Training and Evaluation**\n\nThe model is trained using XGBoost's pairwise ranking objective (`rank:pairwise`). This objective is ideal for this problem because it optimizes the model to learn the relative order of items within each search group, rather than just predicting an absolute probability.\n\nThe training process involves:\n\n- **Data Preparation:** The data is split into training, validation, and test sets. Crucially, the data is grouped by `ranker_id` to maintain the session-wise structure, with group sizes passed to XGBoost to ensure the ranking objective functions correctly.\n- **Hyperparameter Tuning:** The `xgb_params` dictionary specifies the model's configuration, including `max_depth`, `learning_rate`, and regularization terms. The `eval_metric: ndcg@3` is used to monitor performance during training, optimizing the model to place the correct flight in the top positions of each ranked list.\n- **Feature Importance:** After training, the `get_score` method is used to calculate the importance of each feature based on how much it contributes to the model's performance. This provides valuable insights into which factors - such as `is_min_segments` and `legs0_segments0_cabinClass` - are most influential in a business traveler's decision-making process.\n\nThe model's final performance is evaluated using the HitRate@3 metric on the validation set, which confirms its ability to correctly predict the chosen flight within the top three options for a given search query.\n\n**Final Prediction and Submission**\n\nOnce the model is trained and validated, it is used to make predictions on the unseen test data. The raw output from the XGBoost model (a continuous score) is then transformed into a rank. This transformation involves sorting the flight options by their predicted scores in descending order within each `ranker_id` group. The `rank(method='ordinal', descending=True)` function assigns a unique integer rank to each flight, with the highest score receiving rank 1. Finally, these ranks are saved to a CSV file in the required format for submission.","metadata":{}},{"cell_type":"markdown","source":"# Setup Environment","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install -U xgboost\n!pip install -U polars\n!pip install optuna","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:56:48.303590Z","iopub.status.busy":"2025-09-21T11:56:48.303385Z","iopub.status.idle":"2025-09-21T11:57:06.887833Z","shell.execute_reply":"2025-09-21T11:57:06.884451Z","shell.execute_reply.started":"2025-09-21T11:56:48.303572Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import polars as pl\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport time\nimport xgboost as xgb\nimport optuna\nimport joblib\nimport os\nfrom sklearn.metrics import ndcg_score\n\nRANDOM_STATE = 42\nnp.random.seed(RANDOM_STATE)","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:57:06.890714Z","iopub.status.busy":"2025-09-21T11:57:06.890535Z","iopub.status.idle":"2025-09-21T11:57:10.860603Z","shell.execute_reply":"2025-09-21T11:57:10.855092Z","shell.execute_reply.started":"2025-09-21T11:57:06.890694Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Dataset Loading","metadata":{}},{"cell_type":"markdown","source":"**Data Loading and Validation**\n\nThe code begins by loading the training and test datasets, which are stored in the efficient Parquet format, into Polars DataFrames. Polars is a high-performance library chosen for its ability to handle large datasets and complex operations. The training and test data are combined into a single raw DataFrame to ensure that all feature transformations are applied consistently, preventing data leakage and ensuring the model sees the same data structure for both training and prediction.","metadata":{}},{"cell_type":"code","source":"# Load data\ntrain = pl.read_parquet('/kaggle/input/aeroclub-recsys-2025/train.parquet').drop('__index_level_0__')\ntest = pl.read_parquet('/kaggle/input/aeroclub-recsys-2025/test.parquet').drop('__index_level_0__').with_columns(pl.lit(0, dtype=pl.Int64).alias(\"selected\"))\n\ndata_raw = pl.concat((train, test))","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:57:10.862026Z","iopub.status.busy":"2025-09-21T11:57:10.861732Z","iopub.status.idle":"2025-09-21T11:57:12.900041Z","shell.execute_reply":"2025-09-21T11:57:12.896935Z","shell.execute_reply.started":"2025-09-21T11:57:10.862007Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"In this step, its goal is to create new features that provide the XGBoost model with better context and more predictive power. The code generates a diverse set of features based on the following these concepts and steps:\n\n- **Data Transformation:** The raw data contains complex string-based features, such as flight durations. These are parsed and converted into a numerical format (minutes), which is a mandatory step for most machine learning models. Other transformations include creating binary flags (`is_one_way, has_corporate_tariff`) from categorical columns to highlight key attributes.\n- **Relative Features:** A core concept in learning to rank is providing the model with features that describe how a flight option compares to others within the same search session. The code achieves this by creating features like `price_rank and is_cheapest` for each flight. These relative features allow the model to learn not just about a flight's absolute price, but whether it is a good deal compared to other available options in that specific query. This approach is highly effective in solving group-wise ranking problems.\n- **User-Item Interaction Features:** The pipeline also incorporates a common recommender system technique by creating popularity features. It calculates the mean selected rate for each `marketingCarrier_code` in the training data, then joins these aggregated values back to the main DataFrame. This provides the model with a feature that represents a carrier's historical popularity, giving it a signal about which airlines are generally preferred by travelers. This is an example of an ensemble-based learning approach where insights from one part of the data are used to enrich another.\n- **Contextual Features:** Features such as `group_size` and `searchRoute` are added to provide the model with a better understanding of the search context. For example, the model can learn that a traveler's decision-making process may be different in a search with 500 flight options compared to one with just 15.\n\nFinally, the pipeline handles missing values by filling them with a default value (e.g., 0 for numerical features and \"missing\" for categorical features), ensuring the dataset is clean and ready for model training.","metadata":{}},{"cell_type":"code","source":"df = data_raw.clone()\n\n# More efficient duration to minutes converter\ndef dur_to_min(col):\n # Extract days and time parts in one pass\n days = col.str.extract(r\"^(\\d+)\\.\", 1).cast(pl.Int64).fill_null(0) * 1440\n time_str = pl.when(col.str.contains(r\"^\\d+\\.\")).then(col.str.replace(r\"^\\d+\\.\", \"\")).otherwise(col)\n hours = time_str.str.extract(r\"^(\\d+):\", 1).cast(pl.Int64).fill_null(0) * 60\n minutes = time_str.str.extract(r\":(\\d+):\", 1).cast(pl.Int64).fill_null(0)\n return (days + hours + minutes).fill_null(0)\n\n# Process duration columns\ndur_cols = [\"legs0_duration\", \"legs1_duration\"] + [f\"legs{l}_segments{s}_duration\" for l in (0, 1) for s in (0, 1)]\ndur_exprs = [dur_to_min(pl.col(c)).alias(c) for c in dur_cols if c in df.columns]\n\n# Apply duration transformations first\nif dur_exprs:\n df = df.with_columns(dur_exprs)\n\n# Precompute marketing carrier columns check\nmc_cols = [f'legs{l}_segments{s}_marketingCarrier_code' for l in (0, 1) for s in range(4)]\nmc_exists = [col for col in mc_cols if col in df.columns]\n\n# Combine all initial transformations\ndf = df.with_columns([\n # Price features\n (pl.col(\"totalPrice\") / (pl.col(\"taxes\") + 1)).alias(\"price_per_tax\"),\n (pl.col(\"taxes\") / (pl.col(\"totalPrice\") + 1)).alias(\"tax_rate\"),\n pl.col(\"totalPrice\").log1p().alias(\"log_price\"),\n \n # Duration features\n (pl.col(\"legs0_duration\").fill_null(0) + pl.col(\"legs1_duration\").fill_null(0)).alias(\"total_duration\"),\n pl.when(pl.col(\"legs1_duration\").fill_null(0) > 0)\n .then(pl.col(\"legs0_duration\") / (pl.col(\"legs1_duration\") + 1))\n .otherwise(1.0).alias(\"duration_ratio\"),\n \n # Trip type\n (pl.col(\"legs1_duration\").is_null() | \n (pl.col(\"legs1_duration\") == 0) | \n pl.col(\"legs1_segments0_departureFrom_airport_iata\").is_null()).cast(pl.Int32).alias(\"is_one_way\"),\n \n # Total segments count\n (pl.sum_horizontal(pl.col(col).is_not_null().cast(pl.UInt8) for col in mc_exists) \n if mc_exists else pl.lit(0)).alias(\"l0_seg\"),\n \n # FF features\n (pl.col(\"frequentFlyer\").fill_null(\"\").str.count_matches(\"/\") + \n (pl.col(\"frequentFlyer\").fill_null(\"\") != \"\").cast(pl.Int32)).alias(\"n_ff_programs\"),\n \n # Binary features\n pl.col(\"corporateTariffCode\").is_not_null().cast(pl.Int32).alias(\"has_corporate_tariff\"),\n (pl.col(\"pricingInfo_isAccessTP\") == 1).cast(pl.Int32).alias(\"has_access_tp\"),\n \n # Baggage & fees\n (pl.col(\"legs0_segments0_baggageAllowance_quantity\").fill_null(0) + \n pl.col(\"legs1_segments0_baggageAllowance_quantity\").fill_null(0)).alias(\"baggage_total\"),\n (pl.col(\"miniRules0_monetaryAmount\").fill_null(0) + \n pl.col(\"miniRules1_monetaryAmount\").fill_null(0)).alias(\"total_fees\"),\n \n # Routes & carriers\n pl.col(\"searchRoute\").is_in([\"MOWLED/LEDMOW\", \"LEDMOW/MOWLED\", \"MOWLED\", \"LEDMOW\", \"MOWAER/AERMOW\"])\n .cast(pl.Int32).alias(\"is_popular_route\"),\n \n # Cabin\n pl.mean_horizontal([\"legs0_segments0_cabinClass\", \"legs1_segments0_cabinClass\"]).alias(\"avg_cabin_class\"),\n (pl.col(\"legs0_segments0_cabinClass\").fill_null(0) - \n pl.col(\"legs1_segments0_cabinClass\").fill_null(0)).alias(\"cabin_class_diff\"),\n])\n\n# Segment counts - more efficient\nseg_exprs = []\nfor leg in (0, 1):\n seg_cols = [f\"legs{leg}_segments{s}_duration\" for s in range(4) if f\"legs{leg}_segments{s}_duration\" in df.columns]\n if seg_cols:\n seg_exprs.append(\n pl.sum_horizontal(pl.col(c).is_not_null() for c in seg_cols)\n .cast(pl.Int32).alias(f\"n_segments_leg{leg}\")\n )\n else:\n seg_exprs.append(pl.lit(0).cast(pl.Int32).alias(f\"n_segments_leg{leg}\"))\n\n# Add segment-based features\n# First create segment counts\ndf = df.with_columns(seg_exprs)\n\n# Then use them for derived features\ndf = df.with_columns([\n (pl.col(\"n_segments_leg0\") + pl.col(\"n_segments_leg1\")).alias(\"total_segments\"),\n (pl.col(\"n_segments_leg0\") == 1).cast(pl.Int32).alias(\"is_direct_leg0\"),\n pl.when(pl.col(\"is_one_way\") == 1).then(0)\n .otherwise((pl.col(\"n_segments_leg1\") == 1).cast(pl.Int32)).alias(\"is_direct_leg1\"),\n])\n\n# More derived features\ndf = df.with_columns([\n (pl.col(\"is_direct_leg0\") & pl.col(\"is_direct_leg1\")).cast(pl.Int32).alias(\"both_direct\"),\n ((pl.col(\"isVip\") == 1) | (pl.col(\"n_ff_programs\") > 0)).cast(pl.Int32).alias(\"is_vip_freq\"),\n (pl.col(\"baggage_total\") > 0).cast(pl.Int32).alias(\"has_baggage\"),\n (pl.col(\"total_fees\") > 0).cast(pl.Int32).alias(\"has_fees\"),\n (pl.col(\"total_fees\") / (pl.col(\"totalPrice\") + 1)).alias(\"fee_rate\"),\n pl.col(\"Id\").count().over(\"ranker_id\").alias(\"group_size\"),\n])\n\n# Add major carrier flag if column exists\nif \"legs0_segments0_marketingCarrier_code\" in df.columns:\n df = df.with_columns(\n pl.col(\"legs0_segments0_marketingCarrier_code\").is_in([\"SU\", \"S7\", \"U6\"])\n .cast(pl.Int32).alias(\"is_major_carrier\")\n )\nelse:\n df = df.with_columns(pl.lit(0).alias(\"is_major_carrier\"))\n\ndf = df.with_columns(pl.col(\"group_size\").log1p().alias(\"group_size_log\"))\n\n# Time features - batch process\ntime_exprs = []\nfor col in (\"legs0_departureAt\", \"legs0_arrivalAt\", \"legs1_departureAt\", \"legs1_arrivalAt\"):\n if col in df.columns:\n dt = pl.col(col).str.to_datetime(strict=False)\n h = dt.dt.hour().fill_null(12)\n time_exprs.extend([\n h.alias(f\"{col}_hour\"),\n dt.dt.weekday().fill_null(0).alias(f\"{col}_weekday\"),\n (((h >= 6) & (h <= 9)) | ((h >= 17) & (h <= 20))).cast(pl.Int32).alias(f\"{col}_business_time\")\n ])\nif time_exprs:\n df = df.with_columns(time_exprs)\n\n# Batch rank computations - more efficient with single pass\n# First apply the columns that will be used for ranking\ndf = df.with_columns([\n pl.col(\"group_size\").log1p().alias(\"group_size_log\"),\n])\n\n# Price and duration basic ranks\nrank_exprs = []\nfor col, alias in [(\"totalPrice\", \"price\"), (\"total_duration\", \"duration\")]:\n rank_exprs.append(pl.col(col).rank().over(\"ranker_id\").alias(f\"{alias}_rank\"))\n\n# Price-specific features\nprice_exprs = [\n (pl.col(\"totalPrice\").rank(\"average\").over(\"ranker_id\") / \n pl.col(\"totalPrice\").count().over(\"ranker_id\")).alias(\"price_pct_rank\"),\n (pl.col(\"totalPrice\") == pl.col(\"totalPrice\").min().over(\"ranker_id\")).cast(pl.Int32).alias(\"is_cheapest\"),\n ((pl.col(\"totalPrice\") - pl.col(\"totalPrice\").median().over(\"ranker_id\")) / \n (pl.col(\"totalPrice\").std().over(\"ranker_id\") + 1)).alias(\"price_from_median\"),\n (pl.col(\"l0_seg\") == pl.col(\"l0_seg\").min().over(\"ranker_id\")).cast(pl.Int32).alias(\"is_min_segments\"),\n]\n\n# Apply initial ranks\ndf = df.with_columns(rank_exprs + price_exprs)\n\n# Cheapest direct - more efficient\ndirect_cheapest = (\n df.filter(pl.col(\"is_direct_leg0\") == 1)\n .group_by(\"ranker_id\")\n .agg(pl.col(\"totalPrice\").min().alias(\"min_direct\"))\n)\n\ndf = df.join(direct_cheapest, on=\"ranker_id\", how=\"left\").with_columns(\n ((pl.col(\"is_direct_leg0\") == 1) & \n (pl.col(\"totalPrice\") == pl.col(\"min_direct\"))).cast(pl.Int32).fill_null(0).alias(\"is_direct_cheapest\")\n).drop(\"min_direct\")\n\n# Popularity features - efficient join\ndf = (\n df.join(\n train.group_by('legs0_segments0_marketingCarrier_code').agg(pl.mean('selected').alias('carrier0_pop')),\n on='legs0_segments0_marketingCarrier_code', \n how='left'\n )\n .join(\n train.group_by('legs1_segments0_marketingCarrier_code').agg(pl.mean('selected').alias('carrier1_pop')),\n on='legs1_segments0_marketingCarrier_code', \n how='left'\n )\n .with_columns([\n pl.col('carrier0_pop').fill_null(0.0),\n pl.col('carrier1_pop').fill_null(0.0),\n ])\n)\n\n# Final features including popularity\ndf = df.with_columns([\n (pl.col('carrier0_pop') * pl.col('carrier1_pop')).alias('carrier_pop_product'),\n])\n\n\n# Fill nulls\ndata = df.with_columns(\n [pl.col(c).fill_null(0) for c in df.select(pl.selectors.numeric()).columns] +\n [pl.col(c).fill_null(\"missing\") for c in df.select(pl.selectors.string()).columns]\n)","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:57:12.903839Z","iopub.status.busy":"2025-09-21T11:57:12.903539Z","iopub.status.idle":"2025-09-21T11:58:04.341059Z","shell.execute_reply":"2025-09-21T11:58:04.338162Z","shell.execute_reply.started":"2025-09-21T11:57:12.903819Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Feature Selection","metadata":{}},{"cell_type":"markdown","source":"This section of the code prepares the final dataset for the model training stage, where the features will be used to train a machine learning model. The goal of this step is to reduce the number of features by removing those that are uninformative, redundant, or could lead to data leakage. This is a core practice in machine learning to improve model performance and efficiency.\n\nThe code implements a systematic approach to feature selection:\n\n- **Categorical Feature:** A list of all categorical features is explicitly defined. These are important to track for later steps in the machine learning process, such as encoding.\n- **Exclusion of Uninformative Columns:** The code creates a list of columns to be excluded. These are features that are not useful for prediction, such as identifiers (`Id, profileId`), the target variable itself (`selected`), or timestamp data that has already been processed into more useful features (`requestDate`).\n- **Handling Sparse Data:** Columns with a high percentage of missing values (e.g., the `miniRules` columns with over 90% missing values) are also excluded. Keeping such features would add noise without providing significant predictive power.\n- **Final Feature Set Creation:** The code then generates the final feature list by including all columns that were not on the exclusion list. This ensures that the model only receives a clean, relevant, and well-curated set of features.\n\nThis process directly addresses the curse of dimensionality, a scenario where a model's performance can degrade as the number of features increases, especially with a fixed amount of training data. By carefully selecting features, the model is less likely to overfit to noise in the training data, resulting in a more robust and generalized model.","metadata":{}},{"cell_type":"code","source":"# Categorical features\ncat_features = [\n 'nationality', 'searchRoute', 'corporateTariffCode',\n 'bySelf', 'sex', 'companyID',\n # Leg 0 segments 0-1\n 'legs0_segments0_aircraft_code', 'legs0_segments0_arrivalTo_airport_city_iata',\n 'legs0_segments0_arrivalTo_airport_iata', 'legs0_segments0_departureFrom_airport_iata',\n 'legs0_segments0_marketingCarrier_code', 'legs0_segments0_operatingCarrier_code',\n 'legs0_segments0_flightNumber',\n 'legs0_segments1_aircraft_code', 'legs0_segments1_arrivalTo_airport_city_iata',\n 'legs0_segments1_arrivalTo_airport_iata', 'legs0_segments1_departureFrom_airport_iata',\n 'legs0_segments1_marketingCarrier_code', 'legs0_segments1_operatingCarrier_code',\n 'legs0_segments1_flightNumber',\n # Leg 1 segments 0-1\n 'legs1_segments0_aircraft_code', 'legs1_segments0_arrivalTo_airport_city_iata',\n 'legs1_segments0_arrivalTo_airport_iata', 'legs1_segments0_departureFrom_airport_iata',\n 'legs1_segments0_marketingCarrier_code', 'legs1_segments0_operatingCarrier_code',\n 'legs1_segments0_flightNumber',\n 'legs1_segments1_aircraft_code', 'legs1_segments1_arrivalTo_airport_city_iata',\n 'legs1_segments1_arrivalTo_airport_iata', 'legs1_segments1_departureFrom_airport_iata',\n 'legs1_segments1_marketingCarrier_code', 'legs1_segments1_operatingCarrier_code',\n 'legs1_segments1_flightNumber',\n]\n\n# Columns to exclude (uninformative or problematic)\nexclude_cols = [\n 'Id', 'ranker_id', 'selected', 'profileId', 'requestDate',\n 'legs0_departureAt', 'legs0_arrivalAt', 'legs1_departureAt', 'legs1_arrivalAt',\n 'miniRules0_percentage', 'miniRules1_percentage', # >90% missing\n 'frequentFlyer', # Already processed\n # Exclude constant columns\n 'pricingInfo_passengerCount'\n]\n\n\n# Exclude segment 2-3 columns (>98% missing)\nfor leg in [0, 1]:\n for seg in [2, 3]:\n for suffix in ['aircraft_code', 'arrivalTo_airport_city_iata', 'arrivalTo_airport_iata',\n 'baggageAllowance_quantity', 'baggageAllowance_weightMeasurementType',\n 'cabinClass', 'departureFrom_airport_iata', 'duration', 'flightNumber',\n 'marketingCarrier_code', 'operatingCarrier_code', 'seatsAvailable']:\n exclude_cols.append(f'legs{leg}_segments{seg}_{suffix}')\n\nfeature_cols = [col for col in data.columns if col not in exclude_cols]\ncat_features_final = [col for col in cat_features if col in feature_cols]\n\nprint(f\"Using {len(feature_cols)} features ({len(cat_features_final)} categorical)\")\n\nX = data.select(feature_cols)\ny = data.select('selected')\ngroups = data.select('ranker_id')","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:58:04.341914Z","iopub.status.busy":"2025-09-21T11:58:04.341757Z","iopub.status.idle":"2025-09-21T11:58:04.355188Z","shell.execute_reply":"2025-09-21T11:58:04.351117Z","shell.execute_reply.started":"2025-09-21T11:58:04.341898Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The process begins by preparing the data in the specialized format required by XGBoost. This involves converting the Polars DataFrame into an XGBoost `DMatrix` object, which is an optimized data structure for training. A key step here is to group the flights by `ranker_id` and pass these `group_sizes` to the `DMatrix`. This is fundamental for learning to rank, as it tells the model which rows belong to the same search session and should be ranked together.","metadata":{}},{"cell_type":"code","source":"data_xgb = X.with_columns([(pl.col(c).rank(\"dense\") - 1).fill_null(-1).cast(pl.Int32) for c in cat_features_final])\n\nn1 = 16487352 # split train to train and val (10%) in time\nn2 = train.height\ndata_xgb_tr, data_xgb_va, data_xgb_te = data_xgb[:n1], data_xgb[n1:n2], data_xgb[n2:]\ny_tr, y_va, y_te = y[:n1], y[n1:n2], y[n2:]\ngroups_tr, groups_va, groups_te = groups[:n1], groups[n1:n2], groups[n2:]\n\ngroup_sizes_tr = groups_tr.group_by('ranker_id', maintain_order=True).agg(pl.len())['len'].to_numpy()\ngroup_sizes_va = groups_va.group_by('ranker_id', maintain_order=True).agg(pl.len())['len'].to_numpy()\ngroup_sizes_te = groups_te.group_by('ranker_id', maintain_order=True).agg(pl.len())['len'].to_numpy()\ndtrain = xgb.DMatrix(data_xgb_tr, label=y_tr, group=group_sizes_tr, feature_names=data_xgb.columns)\ndval = xgb.DMatrix(data_xgb_va, label=y_va, group=group_sizes_va, feature_names=data_xgb.columns)\ndtest = xgb.DMatrix(data_xgb_te, label=y_te, group=group_sizes_te, feature_names=data_xgb.columns)","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:58:04.356692Z","iopub.status.busy":"2025-09-21T11:58:04.356528Z","iopub.status.idle":"2025-09-21T11:58:25.031063Z","shell.execute_reply":"2025-09-21T11:58:25.027267Z","shell.execute_reply.started":"2025-09-21T11:58:04.356676Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model Training","metadata":{}},{"cell_type":"markdown","source":"This part of the pipeline focuses on the core of the machine learning pipeline: training and evaluating an XGBoost model for a Learning to Rank problem.","metadata":{}},{"cell_type":"markdown","source":"### Initial Model Training","metadata":{}},{"cell_type":"markdown","source":"The XGBoost parameters are then set to configure the model's behavior. The most important parameter is `'objective': 'rank:pairwise'`, which instructs XGBoost to learn a ranking function by optimizing the pairwise relationships between items in a group. Instead of simply predicting whether a flight is selected, the model learns to score a selected flight higher than an unselected one within the same search session. This is a crucial concept in supervised learning to rank.","metadata":{}},{"cell_type":"code","source":"# XGBoost parameters\nxgb_params = {\n 'objective': 'rank:pairwise',\n 'eval_metric': 'ndcg@3',\n 'max_depth': 10,\n 'min_child_weight': 10,\n 'subsample': 0.8,\n 'colsample_bytree': 0.8,\n 'lambda': 10.0,\n 'learning_rate': 0.05,\n 'seed': RANDOM_STATE,\n 'n_jobs': -1,\n # 'device': 'cuda'\n}","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:58:25.033627Z","iopub.status.busy":"2025-09-21T11:58:25.033423Z","iopub.status.idle":"2025-09-21T11:58:25.042667Z","shell.execute_reply":"2025-09-21T11:58:25.038367Z","shell.execute_reply.started":"2025-09-21T11:58:25.033606Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The model is trained using the `xgb.train()` function with a validation set to monitor performance and prevent overfitting. The evaluation metric, `eval_metric`, is set to `'ndcg@3'`. Normalized Discounted Cumulative Gain (NDCG) is a widely used metric for ranking quality that measures the usefulness of a document based on its position in the result list. `NDCG@3` specifically considers the top three ranked items.","metadata":{}},{"cell_type":"code","source":"# Train XGBoost model\nprint(\"Training XGBoost model...\")\nxgb_model = xgb.train(\n xgb_params,\n dtrain,\n num_boost_round=1000,\n evals=[(dtrain, 'train'), (dval, 'val')],\n# early_stopping_rounds=100,\n verbose_eval=50\n)","metadata":{"execution":{"iopub.execute_input":"2025-09-21T11:58:25.044649Z","iopub.status.busy":"2025-09-21T11:58:25.044439Z","iopub.status.idle":"2025-09-21T12:05:10.302763Z","shell.execute_reply":"2025-09-21T12:05:10.298086Z","shell.execute_reply.started":"2025-09-21T11:58:25.044631Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Helper function","metadata":{}},{"cell_type":"markdown","source":"Helper function for validation follow competition objective.","metadata":{}},{"cell_type":"code","source":"def hitrate_at_3(y_true, y_pred, groups):\n df = pl.DataFrame({\n 'group': groups,\n 'pred': y_pred,\n 'true': y_true\n })\n \n return (\n df.filter(pl.col(\"group\").count().over(\"group\") > 10)\n .sort([\"group\", \"pred\"], descending=[False, True])\n .group_by(\"group\", maintain_order=True)\n .head(3)\n .group_by(\"group\")\n .agg(pl.col(\"true\").max())\n .select(pl.col(\"true\").mean())\n .item()\n )","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:05:10.304348Z","iopub.status.busy":"2025-09-21T12:05:10.303425Z","iopub.status.idle":"2025-09-21T12:05:10.314133Z","shell.execute_reply":"2025-09-21T12:05:10.309143Z","shell.execute_reply.started":"2025-09-21T12:05:10.304325Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Hyperparameters Tuning","metadata":{}},{"cell_type":"markdown","source":"This section is a script for hyperparameter tuning of a ranking model using a library called Optuna. This process automatically finds the best combination of settings (hyperparameters) for the model to achieve the highest possible score on a given metric.\n\n**Step-by-step code breakdown**\n\n**1.Defining the Objective Function**\n\nThe `objective(trial)` function is the core of the hyperparameter optimization. Optuna repeatedly calls this function with different sets of hyperparameters. Its goal is to train a model with those parameters and then return a single value that Optuna will try to minimize.\n - **Suggesting Hyperparameters:** The `trial.suggest_*` methods within the function define the search space. For example, `trial.suggest_int('max_depth', 3, 15)` tells Optuna to try different integer values for the max_depth parameter between 3 and 15. The search for other parameters, like `learning_rate and reg_lambda`, is done on a logarithmic scale (`suggest_loguniform`).\n - **Training the Model:** Inside the function, an XGBoost model is trained using the suggested parameters. The code includes `early_stopping_rounds=50`, which is a key concept in machine learning. It tells the model to stop training if the performance on the validation set doesn't improve for 50 consecutive rounds, preventing overfitting and saving time.\n - **Calculating the Metric:** After training, the model makes predictions on the validation set. The `hitrate_at_3` function is then called to calculate a performance score. Hit Rate is a ranking metric that checks if a relevant item is present in the top-k results. In this case, `k=3`.\n- **Returning the Value:** Optuna is designed to minimize an objective function. Since a higher Hit Rate is better, the function returns the negative of the calculated hr3 value (`-hr3`). This allows Optuna to find the parameters that result in the highest possible Hit Rate.\n\n**2. Setting up and Running the Optimization Study**\n\nThis section manages the optimization process and allows it to be resumed if it is interrupted.\n\n- **Creating or Loading the Study:** The `create_or_load_study()` function checks for an existing Optuna study database (`.db file`). If it finds one, it resumes the optimization from where it left off (a process known as checkpointing). If not, it creates a new study. This is an essential practice for long-running optimizations.\n- **Pruning:** The `MedianPruner` is set up to automatically stop unpromising trials early. The pruner analyzes the performance of a trial at different steps and, if it falls below the median performance of other trials at that same step, the trial is terminated. This saves significant computational resources by not completing trials that are likely to fail.\n- **Executing the Optimization:** The `study.optimize()` function starts the process. It is configured to run for a set number of trials (`n_trials=10`), with a time limit (`timeout=3600 seconds`), and to save a checkpoint every 10 trials using the `save_checkpoint_callback`.\n\n**3. Displaying the Best Result**\n\nOnce the optimization is complete, the final block of code retrieves and prints the best-performing set of hyperparameters found during the study.\n\n- **`study.best_trial`:** This retrieves the Trial object that achieved the best objective value.\n- **`trial.value`:** This is the objective value of the best trial. Since we minimized the negative Hit Rate, the printed value will also be negative.\n- **`trial.params`:** This is a dictionary containing all the hyperparameters from the best trial. It shows the optimal combination of values that the optimizer found.","metadata":{}},{"cell_type":"code","source":"# Define the objective function for Optuna using hitrate_at_3\ndef objective(trial):\n # Suggest hyperparameters\n params = {\n 'objective': 'rank:pairwise',\n 'eval_metric': 'ndcg@3',\n 'max_depth': trial.suggest_int('max_depth', 3, 15),\n 'min_child_weight': trial.suggest_loguniform('min_child_weight', 1e-5, 1e2),\n 'subsample': trial.suggest_float('subsample', 0.5, 1.0),\n 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),\n 'colsample_bylevel': trial.suggest_float('colsample_bylevel', 0.5, 1.0),\n 'colsample_bynode': trial.suggest_float('colsample_bynode', 0.5, 1.0),\n 'reg_lambda': trial.suggest_loguniform('reg_lambda', 1e-5, 1e2),\n 'reg_alpha': trial.suggest_loguniform('reg_alpha', 1e-5, 1e2),\n 'learning_rate': trial.suggest_loguniform('learning_rate', 1e-3, 0.3),\n 'gamma': trial.suggest_loguniform('gamma', 1e-5, 1e2),\n 'seed': RANDOM_STATE,\n 'n_jobs': -1,\n # 'device': 'cuda' if you want to use GPU\n }\n \n # Suggest number of boosting rounds\n num_boost_round = trial.suggest_int('num_boost_round', 100, 1000)\n \n # Train the model with early stopping\n model = xgb.train(\n params,\n dtrain,\n num_boost_round=num_boost_round,\n evals=[(dtrain, 'train'), (dval, 'val')],\n early_stopping_rounds=50,\n verbose_eval=False\n )\n \n # Make predictions on validation set\n preds = model.predict(dval)\n \n # Calculate HitRate@3 using your helper function\n hr3 = hitrate_at_3(y_va, preds, groups_va)\n \n # Return negative hitrate (since Optuna minimizes)\n return -hr3\n\n# Create study with checkpointing functionality\nstudy_name = \"xgb_ranking_study_V0\"\nstorage_name = f\"sqlite:////kaggle/working/{study_name}.db\" # Fixed storage URL\n\ndef create_or_load_study():\n try:\n # Try to load existing study\n study = optuna.load_study(\n study_name=study_name,\n storage=storage_name,\n sampler=optuna.samplers.TPESampler(seed=RANDOM_STATE)\n )\n print(f\"Loaded existing study with {len(study.trials)} trials\")\n print(f\"Best trial value (negative HR@3): {study.best_value:.4f}\")\n print(f\"Best trial HR@3: {study.best_value:.4f}\")\n except:\n # Create new study if it doesn't exist\n study = optuna.create_study(\n study_name=study_name,\n storage=storage_name,\n sampler=optuna.samplers.TPESampler(seed=RANDOM_STATE),\n direction=\"minimize\" # We minimize negative HR@3\n )\n print(\"Created new study\")\n return study\n\n\n# Callback function to save checkpoints\ndef save_checkpoint_callback(study, trial):\n # Save study periodically\n if trial.number % 10 == 0:\n joblib.dump(study, f\"{study_name}_checkpoint.pkl\")\n print(f\"Checkpoint saved at trial {trial.number}\")\n print(f\"Current best HR@3: {study.best_value:.4f}\")\n\n# Create or load study\nstudy = create_or_load_study()\n\n# Set up pruning\npruner = optuna.pruners.MedianPruner(n_warmup_steps=10)\n\n# Optimize\nprint(\"Starting hyperparameter optimization...\")\nstudy.optimize(\n objective, \n n_trials=10, # Have to adjust this, set as 10 for demostratetion\n timeout=3600, # 1 hour timeout\n catch=(Exception,), # Catch any exceptions\n callbacks=[save_checkpoint_callback],\n show_progress_bar=True\n)\n\n# Save study for later use\njoblib.dump(study, f\"{study_name}.pkl\")\n\n# Print best trial\nprint(\"\\nBest trial:\")\ntrial = study.best_trial\nprint(f\" HitRate@3: {trial.value:.4f}\")\nprint(\" Params: \")\nfor key, value in trial.params.items():\n print(f\" {key}: {value}\")","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:05:10.314796Z","iopub.status.busy":"2025-09-21T12:05:10.314637Z","iopub.status.idle":"2025-09-21T12:19:34.073750Z","shell.execute_reply":"2025-09-21T12:19:34.070249Z","shell.execute_reply.started":"2025-09-21T12:05:10.314780Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Continue tuning**","metadata":{}},{"cell_type":"markdown","source":"Continue tuning due to it take long time to tuning and my cloud competer also have time-out every 6 hours, so save tuning result and continue tuning from the previous hyperparameters search.","metadata":{}},{"cell_type":"code","source":"# Create study with checkpointing functionality\nstudy_name = \"xgb_ranking_study_V0\"\nstorage_name = f\"sqlite:////kaggle/working/{study_name}.db\" # Fixed storage URL\n\n# Function to continue tuning later\ndef continue_tuning(n_additional_trials=10): # Have to adjust this, set as 10 for demostratetion\n \"\"\"Continue hyperparameter tuning from where it left off\"\"\"\n study = create_or_load_study()\n print(f\"Continuing tuning with {n_additional_trials} additional trials...\")\n \n study.optimize(\n objective, \n n_trials=n_additional_trials,\n timeout=3600,\n catch=(Exception,),\n callbacks=[save_checkpoint_callback],\n show_progress_bar=True\n )\n \n joblib.dump(study, f\"{study_name}.pkl\")\n \n # Print updated results\n print(f\"Updated best HR@3: {study.best_value:.4f}\")\n return study\n\n# Continue tuning later:\nstudy = continue_tuning(n_additional_trials=10) # Have to adjust this, set as 1 for demostratetion\nbest_trial = study.best_trial\nprint(f\"New best HR@3: {best_trial.value:.4f}\")","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:19:34.077701Z","iopub.status.busy":"2025-09-21T12:19:34.077495Z","iopub.status.idle":"2025-09-21T12:42:53.615952Z","shell.execute_reply":"2025-09-21T12:42:53.610759Z","shell.execute_reply.started":"2025-09-21T12:19:34.077682Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Final Model Training","metadata":{}},{"cell_type":"markdown","source":"Train the model with the best hyperparameters, according to previous hyperparameters tuning section.","metadata":{}},{"cell_type":"code","source":"# Print best trial\nprint(\"\\nBest trial:\")\ntrial = study.best_trial\nprint(f\" HitRate@3: {-trial.value:.4f}\")\nprint(\" Params: \")\nfor key, value in trial.params.items():\n print(f\" {key}: {value}\")\n\n# Train final model with best parameters\nbest_params = {\n 'objective': 'rank:pairwise',\n 'eval_metric': 'ndcg@3',\n 'seed': RANDOM_STATE,\n 'n_jobs': -1,\n # 'device': 'cuda'\n}\n# Update with best parameters, excluding num_boost_round which is for training\nbest_training_params = {k: v for k, v in trial.params.items() if k != 'num_boost_round'}\nbest_params.update(best_training_params)\n\nprint(\"\\nTraining final model with best parameters...\")\nxgb_model = xgb.train(\n best_params,\n dtrain,\n num_boost_round=1000, # Use fixed number or optimize separately\n evals=[(dtrain, 'train'), (dval, 'val')],\n early_stopping_rounds=100,\n verbose_eval=50\n)","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:42:53.616875Z","iopub.status.busy":"2025-09-21T12:42:53.616678Z","iopub.status.idle":"2025-09-21T12:49:07.465720Z","shell.execute_reply":"2025-09-21T12:49:07.462715Z","shell.execute_reply.started":"2025-09-21T12:42:53.616855Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Validation","metadata":{}},{"cell_type":"markdown","source":"After training, the model's performance is evaluated on the validation set. The custom `hitrate_at_3` function is used to calculate the final score, which is a straightforward and interpretable measure of the model's success in placing the correct flight in the top three.","metadata":{}},{"cell_type":"code","source":"# Evaluate XGBoost\nxgb_va_preds = xgb_model.predict(dval)\nxgb_hr3 = hitrate_at_3(y_va, xgb_va_preds, groups_va)\nprint(f\"HitRate@3: {xgb_hr3:.3f}\")","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:49:07.467790Z","iopub.status.busy":"2025-09-21T12:49:07.467383Z","iopub.status.idle":"2025-09-21T12:49:08.458217Z","shell.execute_reply":"2025-09-21T12:49:08.453030Z","shell.execute_reply.started":"2025-09-21T12:49:07.467767Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Feature Important","metadata":{}},{"cell_type":"markdown","source":"Finally, input features of the model is analyzed by using feature importance using `xgb_model.get_score(importance_type='gain')`. The gain metric is a measure of how much a feature contributes to the model's performance by splitting nodes in the decision trees. Features with higher gain values are considered more important. The output shows that relative features like `is_min_segments` (identifying the flight with the fewest segments in a group) and `is_major_carrier` are among the most influential, which makes intuitive sense for a flight recommender system.","metadata":{}},{"cell_type":"code","source":"xgb_importance = xgb_model.get_score(importance_type='gain')\nxgb_importance_df = pl.DataFrame(\n [{'feature': k, 'importance': v} for k, v in xgb_importance.items()]\n).sort('importance', descending=bool(1))\nprint(xgb_importance_df.head(20).to_pandas().to_string())","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:49:08.460512Z","iopub.status.busy":"2025-09-21T12:49:08.460226Z","iopub.status.idle":"2025-09-21T12:49:08.521767Z","shell.execute_reply":"2025-09-21T12:49:08.517332Z","shell.execute_reply.started":"2025-09-21T12:49:08.460485Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model Evaluation","metadata":{}},{"cell_type":"markdown","source":"## Error analysis and visualization","metadata":{}},{"cell_type":"markdown","source":"This section of the pipeline goes beyond a single-number performance metric to conduct a deeper error analysis of the model. The primary goal is to understand how and where the model is performing well or poorly across different types of search queries.\n\nThe analysis is performed using two key visualization techniques:\n\n- **HitRate@k Curves:** The first visualization generates a series of HitRate@k curves, which plot the HitRate (the proportion of successful recommendations) as the number of recommended items (`k`) increases. This provides a detailed view of the model's recall at different ranks. By generating separate curves for queries with \"Small,\" \"Medium,\" and \"Large\" group sizes (quantiles are used to define these groups), the analysis reveals how the model's performance varies with the complexity of the user's search. This allows for a more nuanced understanding than a single HitRate@3 score alone.\n\n- **HitRate@3 vs. Group Size Scatter Plot:** The second visualization is a scatter plot that shows the relationship between the group size and the final HitRate@3 score for each individual search session. The x-axis uses a log scale to handle the wide range of group sizes, from very small to very large. This plot helps to identify any patterns or anomalies in the model's performance. For example, it might reveal that the model consistently struggles with queries that have an unusually large number of flight options. This form of analysis is critical for identifying potential weaknesses and guiding future model improvements.\n\nBy creating these visualizations and analyzing the results, the project shifts from a simple performance report to a deeper diagnostic analysis of the model's behavior. This is a fundamental practice in machine learning for building more robust and reliable systems.","metadata":{}},{"cell_type":"code","source":"# Color palette\nred = (0.86, 0.08, 0.24)\nblue = (0.12, 0.56, 1.0)\n\n# Prepare data for analysis\nva_df = pl.DataFrame({\n 'ranker_id': groups_va.to_numpy().flatten(),\n 'pred_score': xgb_va_preds,\n 'selected': y_va.to_numpy().flatten()\n})\n\n# Add group size and filter\nva_df = va_df.join(\n va_df.group_by('ranker_id').agg(pl.len().alias('group_size')), \n on='ranker_id'\n).filter(pl.col('group_size') > 10)\n\n# Calculate group size quantiles\nsize_quantiles = va_df.select('ranker_id', 'group_size').unique().select(\n pl.col('group_size').quantile(0.25).alias('q25'),\n pl.col('group_size').quantile(0.50).alias('q50'),\n pl.col('group_size').quantile(0.75).alias('q75')\n).to_dicts()[0]\n\n# Function to calculate hitrate curve efficiently\ndef calculate_hitrate_curve(df, k_values):\n # Sort once and calculate all k values\n sorted_df = df.sort([\"ranker_id\", \"pred_score\"], descending=[False, True])\n return [\n sorted_df.group_by(\"ranker_id\", maintain_order=True)\n .head(k)\n .group_by(\"ranker_id\")\n .agg(pl.col(\"selected\").max().alias(\"hit\"))\n .select(pl.col(\"hit\").mean())\n .item()\n for k in k_values\n ]\n\n# Calculate curves\nk_values = list(range(1, 21))\ncurves = {\n 'All groups (>10)': calculate_hitrate_curve(va_df, k_values),\n f'Small (11-{int(size_quantiles[\"q25\"])})': calculate_hitrate_curve(\n va_df.filter(pl.col('group_size') <= size_quantiles['q25']), k_values\n ),\n f'Medium ({int(size_quantiles[\"q25\"]+1)}-{int(size_quantiles[\"q75\"])})': calculate_hitrate_curve(\n va_df.filter((pl.col('group_size') > size_quantiles['q25']) & \n (pl.col('group_size') <= size_quantiles['q75'])), k_values\n ),\n f'Large (>{int(size_quantiles[\"q75\"])})': calculate_hitrate_curve(\n va_df.filter(pl.col('group_size') > size_quantiles['q75']), k_values\n )\n}\n\n# Calculate hitrate@3 by group size using log-scale bins\n# Create log-scale bins\nmin_size = va_df['group_size'].min()\nmax_size = va_df['group_size'].max()\nbins = np.logspace(np.log10(min_size), np.log10(max_size), 51) # 51 edges = 50 bins\n\n# Calculate hitrate@3 for each ranker_id\nranker_hr3 = (\n va_df.sort([\"ranker_id\", \"pred_score\"], descending=[False, True])\n .group_by(\"ranker_id\", maintain_order=True)\n .agg([\n pl.col(\"selected\").head(3).max().alias(\"hit_top3\"),\n pl.col(\"group_size\").first()\n ])\n)\n\n# Assign bins and calculate hitrate per bin\nbin_centers = (bins[:-1] + bins[1:]) / 2 # Geometric mean would be more accurate for log scale\nbin_indices = np.digitize(ranker_hr3['group_size'].to_numpy(), bins) - 1\n\nsize_analysis = pl.DataFrame({\n 'bin_idx': bin_indices,\n 'bin_center': bin_centers[np.clip(bin_indices, 0, len(bin_centers)-1)],\n 'hit_top3': ranker_hr3['hit_top3']\n}).group_by(['bin_idx', 'bin_center']).agg([\n pl.col('hit_top3').mean().alias('hitrate3'),\n pl.len().alias('n_groups')\n]).filter(pl.col('n_groups') >= 3).sort('bin_center') # At least 3 groups per bin\n\n# Create combined figure\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 4), dpi=400)\n\n# Left plot: HitRate@k curves\n# Create color gradient from blue to red for size groups\ncolors = ['black'] # All groups is black\nfor i in range(3): # 3 size groups\n t = i / 2 # 0, 0.5, 1\n color = tuple(blue[j] * (1 - t) + red[j] * t for j in range(3))\n colors.append(color)\n\nfor (label, hitrates), color in zip(curves.items(), colors):\n ax1.plot(k_values, hitrates, marker='o', label=label, color=color, markersize=3)\nax1.set_xlabel('k (top-k predictions)')\nax1.set_ylabel('HitRate@k')\nax1.set_title('HitRate@k by Group Size')\nax1.legend(fontsize=8)\nax1.grid(True, alpha=0.3)\nax1.set_xlim(0, 21)\nax1.set_ylim(-0.025, 1.025)\n\n# Right plot: HitRate@3 vs Group Size (log scale)\nax2.scatter(size_analysis['bin_center'], size_analysis['hitrate3'], s=30, alpha=0.6, color=blue)\nax2.set_xlabel('Group Size')\nax2.set_ylabel('HitRate@3')\nax2.set_title('HitRate@3 vs Group Size')\nax2.set_xscale('log')\nax2.grid(True, alpha=0.3)\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:49:08.523801Z","iopub.status.busy":"2025-09-21T12:49:08.523541Z","iopub.status.idle":"2025-09-21T12:49:11.428276Z","shell.execute_reply":"2025-09-21T12:49:11.424302Z","shell.execute_reply.started":"2025-09-21T12:49:08.523773Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**HitRate@k Curves**\n\nThis graph, on the left, shows the model's recall at different ranks. Each line represents the model's performance on a specific subset of the validation data, categorized by the number of flights in each search session (the \"group size\"). The y-axis (HitRate@k) tells us the percentage of sessions where the correct flight was found within the top k predictions.\n\n**Here's a breakdown of the results:**\n\n- **Small Groups (blue line):** The model performs exceptionally well on searches with a small number of flights (11-27). The hit rate climbs very quickly, reaching a high of nearly 100% by the time we consider the top 20 results. This is expected, as there are fewer choices, making it easier for the model to find the correct one.\n\n- **Medium Groups (purple line):** The performance on medium-sized groups (28-160) is still strong, but not as high as with small groups. The curve rises steadily, showing that as the complexity of the search increases, the model's task becomes more difficult.\n\n- **All Groups (black line):** This line represents the overall performance, including all group sizes. It's a good summary of the model's average performance. The fact that it's positioned between the small and medium groups makes sense, as it's an average of these and other group sizes.\n\n- **Large Groups (red line):** The model's performance on large groups (over 160 flights) is the weakest. The curve rises slowly, indicating that it struggles to place the correct flight in the top ranks when there are a huge number of options. This is a common challenge for ranking models, as the signal can get lost in the noise of many irrelevant options.\n\nThis visualization highlights a key finding: the model's performance is highly dependent on the complexity of the search query, as measured by group size.","metadata":{}},{"cell_type":"markdown","source":"**The HitRate@3 vs. Group Size Scatter Plot**\n\nThis graph, on the right, provides a more granular view of the model's performance. Each dot represents a bin of search sessions, showing the average HitRate@3 for that group size.\n\n- **Group Size and Performance:** The plot shows a clear negative correlation: as the group size (x-axis) increases, the HitRate@3 (y-axis) generally decreases. This confirms the conclusion from the HitRate@k curves - the model is less accurate when faced with a larger number of choices.\n\n- **Logarithmic Scale:** The use of a log scale on the x-axis is critical here. It allows us to visualize a wide range of group sizes, from 10 to over 1000, without the smaller values being squashed together. This reveals that the most significant drop in performance occurs as the group size moves from small to medium.\n\nThis analysis is valuable because it moves beyond a single, aggregated metric and provides actionable insights. It shows that future efforts to improve the model should focus on enhancing its ability to handle very large and complex search queries, possibly by creating new features or fine-tuning the model's parameters for those specific cases.","metadata":{}},{"cell_type":"code","source":"# Summary\nprint(f\"HitRate@1: {curves['All groups (>10)'][0]:.3f}\")\nprint(f\"HitRate@3: {curves['All groups (>10)'][2]:.3f}\")\nprint(f\"HitRate@5: {curves['All groups (>10)'][4]:.3f}\")\nprint(f\"HitRate@10: {curves['All groups (>10)'][9]:.3f}\")","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:49:11.429935Z","iopub.status.busy":"2025-09-21T12:49:11.429773Z","iopub.status.idle":"2025-09-21T12:49:11.437784Z","shell.execute_reply":"2025-09-21T12:49:11.433771Z","shell.execute_reply.started":"2025-09-21T12:49:11.429918Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The printed above shows the model's HitRate@k for all search groups with more than 10 flights. This directly relates to the HitRate@k Curves visualization described in the Canvas, giving us a quantitative summary of the black \"All groups\" line in that graph.\n\nHere's a breakdown of what each value means:\n\n- **HitRate@1: 0.277:** This indicates that in approximately 28% of all flight searches, the model successfully ranked the chosen flight as the top result. This is a strong starting point for a recommender system.\n- **HitRate@3: 0.510:** This is the key metric for the project. A value of over 0.5 means the model placed the correct flight within the top three recommendations in just over half of the search sessions.\n- **HitRate@5: 0.623:** By expanding the recommendation list to the top five, the model's success rate increases to over 62%, showing its ability to keep the correct flight close to the top.\n- **HitRate@10: 0.753:** When we look at the top ten recommendations, the model finds the correct flight over 75% of the time.\n\nIn summary, these results demonstrate that the model's recall steadily improves as the list of recommendations gets longer. The increasing hit rate confirms that even if the chosen flight isn't at the very top, it's likely to be found in the first few recommendations. These numbers validate the model's effectiveness and provide clear metrics to track future improvements.","metadata":{}},{"cell_type":"markdown","source":"# Submission Generation","metadata":{}},{"cell_type":"markdown","source":"This final part of the pipeline performs the inference and submission process for the machine learning competition follow requirement of the competitions It takes the trained XGBoost model and applies it to the unseen test data to generate the final predictions.\n\nThe process is as follows:\n\n- **Prediction:** The `xgb_model.predict(dtest)` function uses the trained model to generate a numerical prediction score for each flight in the test set.\n- **Group-wise Ranking:** The code then uses the Polars library to rank the predictions within each unique search session. The `rank(method='ordinal', descending=True).over('ranker_id')` operation is critical. It ensures that the highest-scoring flight in a search session gets a rank of 1, the second-highest gets a rank of 2, and so on. This is a direct application of the learning to rank concept, where the model's output scores are converted into a meaningful rank for each item within its group.\n- **Output Generation:** Finally, the code creates a new DataFrame with the required columns (`Id, ranker_id`, and the newly calculated selected rank) and saves it as `submission.csv`. This file adheres to the specific format required for submission to the competition, allowing the model's performance on the hidden test set to be evaluated.\n\nThis step demonstrates the end-to-end functionality of the machine learning pipeline, from data preparation and model training to generating the final, ranked predictions.","metadata":{}},{"cell_type":"code","source":"submission_xgb = (\n test.select(['Id', 'ranker_id'])\n .with_columns(pl.Series('pred_score', xgb_model.predict(dtest)))\n .with_columns(\n pl.col('pred_score')\n .rank(method='ordinal', descending=True)\n .over('ranker_id')\n .cast(pl.Int32)\n .alias('selected')\n )\n .select(['Id', 'ranker_id', 'selected'])\n)\nsubmission_xgb.write_csv('submission.csv')\nprint(\"Done\")","metadata":{"execution":{"iopub.execute_input":"2025-09-21T12:49:41.471976Z","iopub.status.busy":"2025-09-21T12:49:41.471710Z","iopub.status.idle":"2025-09-21T12:49:42.181160Z","shell.execute_reply":"2025-09-21T12:49:42.176924Z","shell.execute_reply.started":"2025-09-21T12:49:41.471958Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Reference and Related Notebook\n\n**[Inspiration notebook](https://www.kaggle.com/code/ka1242/xgboost-ranker-with-polars)**","metadata":{}}]}