Before diving into the solution description itself, I would first like to thank the competition organizers for an interesting task and for providing real, authentic data.
\nBaseline. I began with the solution by Kirill Khoruzhii (@ka1242): 'AeroClub RecSys 2025 - XGBoost Ranking Baseline' (https://www.kaggle.com/code/ka1242/xgboost-ranker-with-polars). I would like to thank him for an excellent set of features.
\nModels Used. I also experimented with using LightGBM both as a replacement for and in addition to XGBoost. However, it yielded roughly the same results, so I eventually discontinued its use by the end of the competition.
\nImportant Features. As expected, the most important features turned out to be ticket prices and their derivatives. This includes metrics like the ratio to the median price within a group, among others. Features related to flight duration and the number of connections were also significant. Additionally, features indicating the possibility of ticket cancellation or exchange, as well as the ticket class, proved to be substantial contributors.
\nFeature Selection: Features were selected primarily based on intuition, driven by what we expected to be important for a person when choosing a flight ticket. This initial selection was then validated, and if the assumption about a feature's usefulness was confirmed, it was kept for the final model.
\nIn several cases, however, adding certain features—particularly categorical ones related to the customers' past choice history—led to severe overfitting. Although using such features seemed intuitively beneficial, they proved to be impractical in practice.
\nFor example, a factor like the price at which a customer/company previously booked a ticket turned out to be useful. Conversely, trying to incorporate the type of aircraft or the airline carrier a customer/company preferred in the past led to overfitting.
\nOverfitting: Features based on customer choice history led to overfitting. In fact, the first-place competition winner, @goldenlock, stated in his solution https://www.kaggle.com/competitions/aeroclub-recsys-2025/writeups/1st-solution-using-count-based-features that he avoided using them entirely until the competition ended. However, after the competition concluded, he tested the approach I proposed for utilizing a subset of these features and managed to significantly improve the score of his final solution.
\nMitigating Overfitting: The approach was based on a windowed method for generating customer history features. Specifically, for each group, features were constructed using data only from groups where a ticket was purchased by the same customers prior to the current ticket selection group
\nHistorical Window Features: For the features used to calculate the history of option selection within a group, a specific company, or a user, the following metrics were generated: mean, median, and the number of groups in which a choice was previously made and this feature was present. The 25% and 75% quantiles were also generated. The historical features were based on the following features:
\n'legs0_departureAt_hour', 'legs1_departureAt_hour', 'legs0_arrivalAt_hour', 'legs1_arrivalAt_hour'
'price_rank', 'price_from_median', 'duration_rank', 'duration_from_median',
'avg_cabin_class', 'baggage_total', 'l0_seg',
'legs0_segments0_seatsAvailable', 'legs0_segments0_baggageAllowance_quantity', 'legs0_segments0_cabinClass',
'miniRules1_statusInfos', 'miniRules0_statusInfos'
Validation: When selecting features, the data was split into a training and test part, which was then combined with validation. As I now understand, this was probably a mistake, and validation and test should have been separated.
\nAt the same time, the score obtained on validation was almost the same as the one on the Public and Private Leaderboards. Moreover, improvements in local validation usually correlated with improvements on the LB.
\nLocal validation yielded about 0.535. For local validation, the last month of training data was used.
Training: Training was performed on the full training dataset without any use of test data. This included not using the test set for computing statistics or encoding categorical features.
\nHandling New Data in Test: To prevent the model from being significantly distorted by the appearance of new categories in categorical features, about 1% of categorical values in the training data were randomly replaced with missing values. If the proportion of such missing values was less than 1%, additional ones were added. Then, in the test set, unknown categories in categorical features were also marked as missing values (-1) during categorical encoding.
\nReducing Group Size: While considering ways to preselect the most promising candidates for the final choice of the three most likely options a client would choose, I tried simply randomly reducing all groups to a maximum of 50 options. This involved downsampling negative examples (those not chosen by the client). Unexpectedly, this improved the score. I hypothesized that ranking became easier for the models, and the class imbalance was reduced. This is because the probability of correctly guessing the user's choice in a large group was low anyway. However, without reducing group size, models might have struggled more to learn anything useful from such a large group.
\nA beneficial side effect was the reduction in memory usage during training and an increase in training speed. This, in turn, allowed for the use of more features. It's important to note that the group size reduction was performed after all features were calculated and statistics were collected. That is, the features were generated using the full groups. Reducing the group size before feature engineering led to worse model performance.
\nIt would be beneficial to find a way to train the model without overfitting, not only on numerical data but also on historical categorical data. This was not achieved within the scope of this solution.
\nApproaches that generate additional features using language models, or other transformers seem very promising. The idea is to use an attention mechanism to extract meaningful information from the full description of the offered options. @sergeyqt20244 mentioned attempts to apply Bert2rec, though so far without significant results: https://www.kaggle.com/competitions/aeroclub-recsys-2025/discussion/599559#3271192
\nBelow is a notebook with my single model solution:
\nhttps://www.kaggle.com/code/mikhailgolubchik/sm-xgboost-single
Before diving into the solution description itself, I would first like to thank the competition organizers for an interesting task and for providing real, authentic data.
\nBaseline. I began with the solution by Kirill Khoruzhii (@ka1242): 'AeroClub RecSys 2025 - XGBoost Ranking Baseline' (https://www.kaggle.com/code/ka1242/xgboost-ranker-with-polars). I would like to thank him for an excellent set of features.
\nModels Used. I also experimented with using LightGBM both as a replacement for and in addition to XGBoost. However, it yielded roughly the same results, so I eventually discontinued its use by the end of the competition.
\nImportant Features. As expected, the most important features turned out to be ticket prices and their derivatives. This includes metrics like the ratio to the median price within a group, among others. Features related to flight duration and the number of connections were also significant. Additionally, features indicating the possibility of ticket cancellation or exchange, as well as the ticket class, proved to be substantial contributors.
\nFeature Selection: Features were selected primarily based on intuition, driven by what we expected to be important for a person when choosing a flight ticket. This initial selection was then validated, and if the assumption about a feature's usefulness was confirmed, it was kept for the final model.
\nIn several cases, however, adding certain features—particularly categorical ones related to the customers' past choice history—led to severe overfitting. Although using such features seemed intuitively beneficial, they proved to be impractical in practice.
\nFor example, a factor like the price at which a customer/company previously booked a ticket turned out to be useful. Conversely, trying to incorporate the type of aircraft or the airline carrier a customer/company preferred in the past led to overfitting.
\nOverfitting: Features based on customer choice history led to overfitting. In fact, the first-place competition winner, @goldenlock, stated in his solution https://www.kaggle.com/competitions/aeroclub-recsys-2025/writeups/1st-solution-using-count-based-features that he avoided using them entirely until the competition ended. However, after the competition concluded, he tested the approach I proposed for utilizing a subset of these features and managed to significantly improve the score of his final solution.
\nMitigating Overfitting: The approach was based on a windowed method for generating customer history features. Specifically, for each group, features were constructed using data only from groups where a ticket was purchased by the same customers prior to the current ticket selection group
\nHistorical Window Features: For the features used to calculate the history of option selection within a group, a specific company, or a user, the following metrics were generated: mean, median, and the number of groups in which a choice was previously made and this feature was present. The 25% and 75% quantiles were also generated. The historical features were based on the following features:
\n'legs0_departureAt_hour', 'legs1_departureAt_hour', 'legs0_arrivalAt_hour', 'legs1_arrivalAt_hour'
'price_rank', 'price_from_median', 'duration_rank', 'duration_from_median',
'avg_cabin_class', 'baggage_total', 'l0_seg',
'legs0_segments0_seatsAvailable', 'legs0_segments0_baggageAllowance_quantity', 'legs0_segments0_cabinClass',
'miniRules1_statusInfos', 'miniRules0_statusInfos'
Validation: When selecting features, the data was split into a training and test part, which was then combined with validation. As I now understand, this was probably a mistake, and validation and test should have been separated.
\nAt the same time, the score obtained on validation was almost the same as the one on the Public and Private Leaderboards. Moreover, improvements in local validation usually correlated with improvements on the LB.
\nLocal validation yielded about 0.535. For local validation, the last month of training data was used.
Training: Training was performed on the full training dataset without any use of test data. This included not using the test set for computing statistics or encoding categorical features.
\nHandling New Data in Test: To prevent the model from being significantly distorted by the appearance of new categories in categorical features, about 1% of categorical values in the training data were randomly replaced with missing values. If the proportion of such missing values was less than 1%, additional ones were added. Then, in the test set, unknown categories in categorical features were also marked as missing values (-1) during categorical encoding.
\nReducing Group Size: While considering ways to preselect the most promising candidates for the final choice of the three most likely options a client would choose, I tried simply randomly reducing all groups to a maximum of 50 options. This involved downsampling negative examples (those not chosen by the client). Unexpectedly, this improved the score. I hypothesized that ranking became easier for the models, and the class imbalance was reduced. This is because the probability of correctly guessing the user's choice in a large group was low anyway. However, without reducing group size, models might have struggled more to learn anything useful from such a large group.
\nA beneficial side effect was the reduction in memory usage during training and an increase in training speed. This, in turn, allowed for the use of more features. It's important to note that the group size reduction was performed after all features were calculated and statistics were collected. That is, the features were generated using the full groups. Reducing the group size before feature engineering led to worse model performance.
\nIt would be beneficial to find a way to train the model without overfitting, not only on numerical data but also on historical categorical data. This was not achieved within the scope of this solution.
\nApproaches that generate additional features using language models, or other transformers seem very promising. The idea is to use an attention mechanism to extract meaningful information from the full description of the offered options. @sergeyqt20244 mentioned attempts to apply Bert2rec, though so far without significant results: https://www.kaggle.com/competitions/aeroclub-recsys-2025/discussion/599559#3271192
\nBelow is a notebook with my single model solution:
\nhttps://www.kaggle.com/code/mikhailgolubchik/sm-xgboost-single