{ "id": 587744, "title": "legs1_segments0_flightNumber", "url": "/competitions/aeroclub-recsys-2025/discussion/587744", "author_name": "", "post_date": "2025-07-02T13:59:25.441543300Z", "votes": 2, "comment_count": 1, "views": 0, "content": "

Hi,

\n

I’m looking for some suggestions on how to handle missing values in the \"legs1_segments0_flightNumber\" feature.

\n

📊 Dataset: Train
\nColumn: legs1_segments0_flightNumber
\nData type: float64
\nTotal rows: 18,145,372
\nMissing values: 4,387,201 (24.178071%)
\nUnique values (non-null): 5,376

\n

🔹 Descriptive statistics:
\ncount 1.375817e+07
\nmean 3.516597e+03
\nstd 2.702681e+03
\nmin 1.000000e+00
\n25% 1.127000e+03
\n50% 2.428000e+03
\n75% 6.189000e+03
\nmax 9.981000e+03
\nName: legs1_segments0_flightNumber, dtype: float64

\n

============================================================

\n

📊 Dataset: Test
\nColumn: legs1_segments0_flightNumber
\nData type: float64
\nTotal rows: 6,897,776
\nMissing values: 1,115,840 (16.176808%)
\nUnique values (non-null): 3,962

\n

🔹 Descriptive statistics:
\ncount 5.781936e+06
\nmean 3.908375e+03
\nstd 2.750874e+03
\nmin 2.000000e+00
\n25% 1.133000e+03
\n50% 6.008000e+03
\n75% 6.216000e+03
\nmax 9.980000e+03
\nName: legs1_segments0_flightNumber, dtype: float64

\n

For other columns with missing values, I used imputation based on an XGBRegressor or XGBClassifier, trained on the most informative and correlated features relative to the target.
\nBefore applying imputation, I validated the approach using test sets of 2,000,000 masked values (originally non-missing) to ensure the method's reliability.
\nThe performance metrics indicated strong confidence in the approach , accuracy between 0.96–0.99, F1 scores between 0.95–0.99, and very clean confusion matrices.

\n

Now I’d like to address the imputation of missing values in \"legs1_segments0_flightNumber\", knowing that the most informative features for this target are:

\n

🔹 TOP 15 FEATURES BY PEARSON TRAIN
\nlegs0_segments0_flightNumber 0.633726
\nlegs0_segments0_seatsAvailable -0.209847
\ntaxes -0.190560
\nlegs0_segments0_baggageAllowance_quantity -0.188609
\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.185692
\ntotalPrice -0.178710
\nminiRules0_statusInfos 0.154665
\nminiRules0_statusInfos_was_missing 0.117455
\nminiRules1_statusInfos_was_missing 0.103811
\nminiRules1_statusInfos -0.089303
\nminiRules1_monetaryAmount -0.075431
\npricingInfo_isAccessTP 0.054396
\nminiRules0_monetaryAmount -0.044281
\ncompanyID -0.039586
\nisVip -0.035204

\n

🔹 TOP 15 FEATURES BY MI TRAIN
\ntaxes 3.067493
\ntotalPrice 2.623157
\nlegs0_segments0_flightNumber 2.523073
\ncompanyID 1.069480
\nminiRules0_monetaryAmount 0.685185
\nminiRules1_monetaryAmount 0.490049
\nlegs0_segments0_baggageAllowance_quantity 0.249210
\nminiRules1_statusInfos_was_missing 0.154973
\nminiRules0_statusInfos_was_missing 0.149661
\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.149345
\nlegs0_segments0_seatsAvailable 0.148829
\nlegs0_segments0_cabinClass 0.111277
\nminiRules0_statusInfos 0.091255
\nisAccess3D 0.076807
\nnationality 0.075790

\n

🔹 TOP 15 FEATURES BY PEARSON TEST
\nlegs0_segments0_flightNumber 0.595955
\nlegs0_segments0_seatsAvailable -0.318228
\ntaxes -0.235064
\ntotalPrice -0.226184
\nlegs0_segments0_baggageAllowance_quantity -0.216154
\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.208696
\nminiRules0_statusInfos 0.167368
\nminiRules0_statusInfos_was_missing 0.162495
\nminiRules1_statusInfos_was_missing 0.151309
\nminiRules1_monetaryAmount -0.135436
\nminiRules0_monetaryAmount -0.099024
\nminiRules1_statusInfos -0.082233
\nlegs0_segments0_cabinClass -0.078870
\npricingInfo_isAccessTP 0.055643
\nbySelf -0.054677

\n

🔹 TOP 15 FEATURES BY MI TEST
\ntaxes 2.842257
\nlegs0_segments0_flightNumber 2.515544
\ntotalPrice 2.505068
\ncompanyID 1.085874
\nminiRules0_monetaryAmount 0.727438
\nminiRules1_monetaryAmount 0.526809
\nlegs0_segments0_baggageAllowance_quantity 0.245211
\nlegs0_segments0_seatsAvailable 0.184847
\nminiRules1_statusInfos_was_missing 0.164401
\nminiRules0_statusInfos_was_missing 0.157333
\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.142728
\nisAccess3D 0.123774
\nminiRules0_statusInfos 0.086724
\npricingInfo_isAccessTP 0.071179
\nminiRules1_statusInfos 0.068968

\n

Another important detail is the large number of classes (5,376 in train.csv, 3,962 in test.csv) and their highly imbalanced distribution.
\nI’d like to approach the imputation using XGBClassifier as well, but I’m running into memory issues due to the high number of classes and the overall data volume.

\n

How did you approach imputing missing values for \"legs1_segments0_flightNumber\"? What steps did you take to ensure that your method was solid and robust?

", "messages": [ { "id": "3239118", "postDate": "07/02/2025 13:59:25", "content": "

Hi,

\n

I’m looking for some suggestions on how to handle missing values in the \"legs1_segments0_flightNumber\" feature.

\n

📊 Dataset: Train
\nColumn: legs1_segments0_flightNumber
\nData type: float64
\nTotal rows: 18,145,372
\nMissing values: 4,387,201 (24.178071%)
\nUnique values (non-null): 5,376

\n

🔹 Descriptive statistics:
\ncount 1.375817e+07
\nmean 3.516597e+03
\nstd 2.702681e+03
\nmin 1.000000e+00
\n25% 1.127000e+03
\n50% 2.428000e+03
\n75% 6.189000e+03
\nmax 9.981000e+03
\nName: legs1_segments0_flightNumber, dtype: float64

\n

============================================================

\n

📊 Dataset: Test
\nColumn: legs1_segments0_flightNumber
\nData type: float64
\nTotal rows: 6,897,776
\nMissing values: 1,115,840 (16.176808%)
\nUnique values (non-null): 3,962

\n

🔹 Descriptive statistics:
\ncount 5.781936e+06
\nmean 3.908375e+03
\nstd 2.750874e+03
\nmin 2.000000e+00
\n25% 1.133000e+03
\n50% 6.008000e+03
\n75% 6.216000e+03
\nmax 9.980000e+03
\nName: legs1_segments0_flightNumber, dtype: float64

\n

For other columns with missing values, I used imputation based on an XGBRegressor or XGBClassifier, trained on the most informative and correlated features relative to the target.
\nBefore applying imputation, I validated the approach using test sets of 2,000,000 masked values (originally non-missing) to ensure the method's reliability.
\nThe performance metrics indicated strong confidence in the approach , accuracy between 0.96–0.99, F1 scores between 0.95–0.99, and very clean confusion matrices.

\n

Now I’d like to address the imputation of missing values in \"legs1_segments0_flightNumber\", knowing that the most informative features for this target are:

\n

🔹 TOP 15 FEATURES BY PEARSON TRAIN
\nlegs0_segments0_flightNumber 0.633726
\nlegs0_segments0_seatsAvailable -0.209847
\ntaxes -0.190560
\nlegs0_segments0_baggageAllowance_quantity -0.188609
\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.185692
\ntotalPrice -0.178710
\nminiRules0_statusInfos 0.154665
\nminiRules0_statusInfos_was_missing 0.117455
\nminiRules1_statusInfos_was_missing 0.103811
\nminiRules1_statusInfos -0.089303
\nminiRules1_monetaryAmount -0.075431
\npricingInfo_isAccessTP 0.054396
\nminiRules0_monetaryAmount -0.044281
\ncompanyID -0.039586
\nisVip -0.035204

\n

🔹 TOP 15 FEATURES BY MI TRAIN
\ntaxes 3.067493
\ntotalPrice 2.623157
\nlegs0_segments0_flightNumber 2.523073
\ncompanyID 1.069480
\nminiRules0_monetaryAmount 0.685185
\nminiRules1_monetaryAmount 0.490049
\nlegs0_segments0_baggageAllowance_quantity 0.249210
\nminiRules1_statusInfos_was_missing 0.154973
\nminiRules0_statusInfos_was_missing 0.149661
\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.149345
\nlegs0_segments0_seatsAvailable 0.148829
\nlegs0_segments0_cabinClass 0.111277
\nminiRules0_statusInfos 0.091255
\nisAccess3D 0.076807
\nnationality 0.075790

\n

🔹 TOP 15 FEATURES BY PEARSON TEST
\nlegs0_segments0_flightNumber 0.595955
\nlegs0_segments0_seatsAvailable -0.318228
\ntaxes -0.235064
\ntotalPrice -0.226184
\nlegs0_segments0_baggageAllowance_quantity -0.216154
\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.208696
\nminiRules0_statusInfos 0.167368
\nminiRules0_statusInfos_was_missing 0.162495
\nminiRules1_statusInfos_was_missing 0.151309
\nminiRules1_monetaryAmount -0.135436
\nminiRules0_monetaryAmount -0.099024
\nminiRules1_statusInfos -0.082233
\nlegs0_segments0_cabinClass -0.078870
\npricingInfo_isAccessTP 0.055643
\nbySelf -0.054677

\n

🔹 TOP 15 FEATURES BY MI TEST
\ntaxes 2.842257
\nlegs0_segments0_flightNumber 2.515544
\ntotalPrice 2.505068
\ncompanyID 1.085874
\nminiRules0_monetaryAmount 0.727438
\nminiRules1_monetaryAmount 0.526809
\nlegs0_segments0_baggageAllowance_quantity 0.245211
\nlegs0_segments0_seatsAvailable 0.184847
\nminiRules1_statusInfos_was_missing 0.164401
\nminiRules0_statusInfos_was_missing 0.157333
\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.142728
\nisAccess3D 0.123774
\nminiRules0_statusInfos 0.086724
\npricingInfo_isAccessTP 0.071179
\nminiRules1_statusInfos 0.068968

\n

Another important detail is the large number of classes (5,376 in train.csv, 3,962 in test.csv) and their highly imbalanced distribution.
\nI’d like to approach the imputation using XGBClassifier as well, but I’m running into memory issues due to the high number of classes and the overall data volume.

\n

How did you approach imputing missing values for \"legs1_segments0_flightNumber\"? What steps did you take to ensure that your method was solid and robust?

", "rawMarkdown": "Hi,\n\nI’m looking for some suggestions on how to handle missing values in the \"legs1_segments0_flightNumber\" feature.\n\n📊 Dataset: Train\nColumn: legs1_segments0_flightNumber\nData type: float64\nTotal rows: 18,145,372\nMissing values: 4,387,201 (24.178071%)\nUnique values (non-null): 5,376\n\n🔹 Descriptive statistics:\ncount 1.375817e+07\nmean 3.516597e+03\nstd 2.702681e+03\nmin 1.000000e+00\n25% 1.127000e+03\n50% 2.428000e+03\n75% 6.189000e+03\nmax 9.981000e+03\nName: legs1_segments0_flightNumber, dtype: float64\n\n============================================================\n\n📊 Dataset: Test\nColumn: legs1_segments0_flightNumber\nData type: float64\nTotal rows: 6,897,776\nMissing values: 1,115,840 (16.176808%)\nUnique values (non-null): 3,962\n\n🔹 Descriptive statistics:\ncount 5.781936e+06\nmean 3.908375e+03\nstd 2.750874e+03\nmin 2.000000e+00\n25% 1.133000e+03\n50% 6.008000e+03\n75% 6.216000e+03\nmax 9.980000e+03\nName: legs1_segments0_flightNumber, dtype: float64\n\nFor other columns with missing values, I used imputation based on an XGBRegressor or XGBClassifier, trained on the most informative and correlated features relative to the target.\nBefore applying imputation, I validated the approach using test sets of 2,000,000 masked values (originally non-missing) to ensure the method's reliability.\nThe performance metrics indicated strong confidence in the approach , accuracy between 0.96–0.99, F1 scores between 0.95–0.99, and very clean confusion matrices.\n\nNow I’d like to address the imputation of missing values in \"legs1_segments0_flightNumber\", knowing that the most informative features for this target are:\n\n🔹 TOP 15 FEATURES BY PEARSON TRAIN\nlegs0_segments0_flightNumber 0.633726\nlegs0_segments0_seatsAvailable -0.209847\ntaxes -0.190560\nlegs0_segments0_baggageAllowance_quantity -0.188609\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.185692\ntotalPrice -0.178710\nminiRules0_statusInfos 0.154665\nminiRules0_statusInfos_was_missing 0.117455\nminiRules1_statusInfos_was_missing 0.103811\nminiRules1_statusInfos -0.089303\nminiRules1_monetaryAmount -0.075431\npricingInfo_isAccessTP 0.054396\nminiRules0_monetaryAmount -0.044281\ncompanyID -0.039586\nisVip -0.035204\n\n🔹 TOP 15 FEATURES BY MI TRAIN\ntaxes 3.067493\ntotalPrice 2.623157\nlegs0_segments0_flightNumber 2.523073\ncompanyID 1.069480\nminiRules0_monetaryAmount 0.685185\nminiRules1_monetaryAmount 0.490049\nlegs0_segments0_baggageAllowance_quantity 0.249210\nminiRules1_statusInfos_was_missing 0.154973\nminiRules0_statusInfos_was_missing 0.149661\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.149345\nlegs0_segments0_seatsAvailable 0.148829\nlegs0_segments0_cabinClass 0.111277\nminiRules0_statusInfos 0.091255\nisAccess3D 0.076807\nnationality 0.075790\n\n🔹 TOP 15 FEATURES BY PEARSON TEST\nlegs0_segments0_flightNumber 0.595955\nlegs0_segments0_seatsAvailable -0.318228\ntaxes -0.235064\ntotalPrice -0.226184\nlegs0_segments0_baggageAllowance_quantity -0.216154\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.208696\nminiRules0_statusInfos 0.167368\nminiRules0_statusInfos_was_missing 0.162495\nminiRules1_statusInfos_was_missing 0.151309\nminiRules1_monetaryAmount -0.135436\nminiRules0_monetaryAmount -0.099024\nminiRules1_statusInfos -0.082233\nlegs0_segments0_cabinClass -0.078870\npricingInfo_isAccessTP 0.055643\nbySelf -0.054677\n\n🔹 TOP 15 FEATURES BY MI TEST\ntaxes 2.842257\nlegs0_segments0_flightNumber 2.515544\ntotalPrice 2.505068\ncompanyID 1.085874\nminiRules0_monetaryAmount 0.727438\nminiRules1_monetaryAmount 0.526809\nlegs0_segments0_baggageAllowance_quantity 0.245211\nlegs0_segments0_seatsAvailable 0.184847\nminiRules1_statusInfos_was_missing 0.164401\nminiRules0_statusInfos_was_missing 0.157333\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.142728\nisAccess3D 0.123774\nminiRules0_statusInfos 0.086724\npricingInfo_isAccessTP 0.071179\nminiRules1_statusInfos 0.068968\n\nAnother important detail is the large number of classes (5,376 in train.csv, 3,962 in test.csv) and their highly imbalanced distribution.\nI’d like to approach the imputation using XGBClassifier as well, but I’m running into memory issues due to the high number of classes and the overall data volume.\n\nHow did you approach imputing missing values for \"legs1_segments0_flightNumber\"? What steps did you take to ensure that your method was solid and robust?", "votes": null }, { "id": "3239138", "postDate": "07/02/2025 14:27:32", "content": "

Delete the entire column ) \"Seek and destroy\"

", "rawMarkdown": "Delete the entire column ) \"Seek and destroy\"", "votes": null } ], "comments": [ { "id": 3239138, "author_name": "sergeyqt2024", "author_url": "", "post_date": "07/02/2025 14:27:32", "content": "

Delete the entire column ) \"Seek and destroy\"

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3239118": "Hi,\n\nI’m looking for some suggestions on how to handle missing values in the \"legs1_segments0_flightNumber\" feature.\n\n📊 Dataset: Train\nColumn: legs1_segments0_flightNumber\nData type: float64\nTotal rows: 18,145,372\nMissing values: 4,387,201 (24.178071%)\nUnique values (non-null): 5,376\n\n🔹 Descriptive statistics:\ncount 1.375817e+07\nmean 3.516597e+03\nstd 2.702681e+03\nmin 1.000000e+00\n25% 1.127000e+03\n50% 2.428000e+03\n75% 6.189000e+03\nmax 9.981000e+03\nName: legs1_segments0_flightNumber, dtype: float64\n\n============================================================\n\n📊 Dataset: Test\nColumn: legs1_segments0_flightNumber\nData type: float64\nTotal rows: 6,897,776\nMissing values: 1,115,840 (16.176808%)\nUnique values (non-null): 3,962\n\n🔹 Descriptive statistics:\ncount 5.781936e+06\nmean 3.908375e+03\nstd 2.750874e+03\nmin 2.000000e+00\n25% 1.133000e+03\n50% 6.008000e+03\n75% 6.216000e+03\nmax 9.980000e+03\nName: legs1_segments0_flightNumber, dtype: float64\n\nFor other columns with missing values, I used imputation based on an XGBRegressor or XGBClassifier, trained on the most informative and correlated features relative to the target.\nBefore applying imputation, I validated the approach using test sets of 2,000,000 masked values (originally non-missing) to ensure the method's reliability.\nThe performance metrics indicated strong confidence in the approach , accuracy between 0.96–0.99, F1 scores between 0.95–0.99, and very clean confusion matrices.\n\nNow I’d like to address the imputation of missing values in \"legs1_segments0_flightNumber\", knowing that the most informative features for this target are:\n\n🔹 TOP 15 FEATURES BY PEARSON TRAIN\nlegs0_segments0_flightNumber 0.633726\nlegs0_segments0_seatsAvailable -0.209847\ntaxes -0.190560\nlegs0_segments0_baggageAllowance_quantity -0.188609\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.185692\ntotalPrice -0.178710\nminiRules0_statusInfos 0.154665\nminiRules0_statusInfos_was_missing 0.117455\nminiRules1_statusInfos_was_missing 0.103811\nminiRules1_statusInfos -0.089303\nminiRules1_monetaryAmount -0.075431\npricingInfo_isAccessTP 0.054396\nminiRules0_monetaryAmount -0.044281\ncompanyID -0.039586\nisVip -0.035204\n\n🔹 TOP 15 FEATURES BY MI TRAIN\ntaxes 3.067493\ntotalPrice 2.623157\nlegs0_segments0_flightNumber 2.523073\ncompanyID 1.069480\nminiRules0_monetaryAmount 0.685185\nminiRules1_monetaryAmount 0.490049\nlegs0_segments0_baggageAllowance_quantity 0.249210\nminiRules1_statusInfos_was_missing 0.154973\nminiRules0_statusInfos_was_missing 0.149661\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.149345\nlegs0_segments0_seatsAvailable 0.148829\nlegs0_segments0_cabinClass 0.111277\nminiRules0_statusInfos 0.091255\nisAccess3D 0.076807\nnationality 0.075790\n\n🔹 TOP 15 FEATURES BY PEARSON TEST\nlegs0_segments0_flightNumber 0.595955\nlegs0_segments0_seatsAvailable -0.318228\ntaxes -0.235064\ntotalPrice -0.226184\nlegs0_segments0_baggageAllowance_quantity -0.216154\nlegs0_segments0_baggageAllowance_weightMeasurementType -0.208696\nminiRules0_statusInfos 0.167368\nminiRules0_statusInfos_was_missing 0.162495\nminiRules1_statusInfos_was_missing 0.151309\nminiRules1_monetaryAmount -0.135436\nminiRules0_monetaryAmount -0.099024\nminiRules1_statusInfos -0.082233\nlegs0_segments0_cabinClass -0.078870\npricingInfo_isAccessTP 0.055643\nbySelf -0.054677\n\n🔹 TOP 15 FEATURES BY MI TEST\ntaxes 2.842257\nlegs0_segments0_flightNumber 2.515544\ntotalPrice 2.505068\ncompanyID 1.085874\nminiRules0_monetaryAmount 0.727438\nminiRules1_monetaryAmount 0.526809\nlegs0_segments0_baggageAllowance_quantity 0.245211\nlegs0_segments0_seatsAvailable 0.184847\nminiRules1_statusInfos_was_missing 0.164401\nminiRules0_statusInfos_was_missing 0.157333\nlegs0_segments0_baggageAllowance_weightMeasurementType 0.142728\nisAccess3D 0.123774\nminiRules0_statusInfos 0.086724\npricingInfo_isAccessTP 0.071179\nminiRules1_statusInfos 0.068968\n\nAnother important detail is the large number of classes (5,376 in train.csv, 3,962 in test.csv) and their highly imbalanced distribution.\nI’d like to approach the imputation using XGBClassifier as well, but I’m running into memory issues due to the high number of classes and the overall data volume.\n\nHow did you approach imputing missing values for \"legs1_segments0_flightNumber\"? What steps did you take to ensure that your method was solid and robust?", "3239138": "Delete the entire column ) \"Seek and destroy\"" }, "source": "meta" }