{ "id": 590264, "title": "How was train.parquet generated from raw data?", "url": "/competitions/aeroclub-recsys-2025/discussion/590264", "author_name": "", "post_date": "2025-07-19T07:29:47.095522600Z", "votes": 3, "comment_count": 10, "views": 0, "content": "

Hi organizers,

\n

Thanks for providing both the raw data and the preprocessed file train.parquet.
\nFor better understanding and to ensure consistency when I extract additional fields from the raw data,
\ncould you share the code (or a simplified version) used to generate train.parquet?

\n

Having the preprocessing script would make it easier for participants to extend or reproduce the dataset
\nwhile maintaining the same logic.

\n

Thanks in advance!

", "messages": [ { "id": "3250805", "postDate": "07/19/2025 07:29:47", "content": "

Hi organizers,

\n

Thanks for providing both the raw data and the preprocessed file train.parquet.
\nFor better understanding and to ensure consistency when I extract additional fields from the raw data,
\ncould you share the code (or a simplified version) used to generate train.parquet?

\n

Having the preprocessing script would make it easier for participants to extend or reproduce the dataset
\nwhile maintaining the same logic.

\n

Thanks in advance!

", "rawMarkdown": "Hi organizers,\n\nThanks for providing both the raw data and the preprocessed file `train.parquet`. \nFor better understanding and to ensure consistency when I extract additional fields from the raw data, \ncould you share the code (or a simplified version) used to generate `train.parquet`?\n\nHaving the preprocessing script would make it easier for participants to extend or reproduce the dataset \nwhile maintaining the same logic.\n\nThanks in advance!", "votes": null }, { "id": "3251008", "postDate": "07/19/2025 17:07:36", "content": "

Hi @mango789. Thanks for asking. We've decided to share our json_converters as a reference. Hope that helps. But please don't forget it couldn't be applied directly to the competition jsons since some ETL and sensitive data truncations.

", "rawMarkdown": "Hi @mango789. Thanks for asking. We've decided to share our [json_converters](https://www.kaggle.com/code/samvelkoch/reference-json-converters ) as a reference. Hope that helps. But please don't forget it couldn't be applied directly to the competition jsons since some ETL and sensitive data truncations.", "votes": null }, { "id": "3251096", "postDate": "07/20/2025 00:01:12", "content": "

Thanks a lot for sharing the json_converters and the note about ETL/data truncations. That’s very helpful!🥺

", "rawMarkdown": "Thanks a lot for sharing the json_converters and the note about ETL/data truncations. That’s very helpful!🥺", "votes": null }, { "id": "3253340", "postDate": "07/24/2025 13:45:56", "content": "

Hi @samvelkoch , I have a another quick question: within the JSON files, is the order of the flights preserved exactly in the parquet files for the same ranker_id group? In other words, can I assume the flights’ order in json matches the order in the parquet rows for that ranker_id?
\nThanks in advance for your help!

", "rawMarkdown": "Hi @samvelkoch , I have a another quick question: within the JSON files, is the order of the flights preserved exactly in the parquet files for the same `ranker_id` group? In other words, can I assume the flights’ order in json matches the order in the parquet rows for that `ranker_id`?\nThanks in advance for your help!", "votes": null }, { "id": "3253358", "postDate": "07/24/2025 14:16:12", "content": "

Hi @mango789. We just can't guarantee that. This assumption could not be 100% truth and I have no possibility to check it for sure now. So the answer I have to give you is NO.

", "rawMarkdown": "Hi @mango789. We just can't guarantee that. This assumption could not be 100% truth and I have no possibility to check it for sure now. So the answer I have to give you is NO.", "votes": null }, { "id": "3253765", "postDate": "07/25/2025 08:09:36", "content": "

Hi @samvelkoch, thanks for the clarification.

\n

Is there any recommended way to reliably align the option order between the json file and the parquet file?
\nToday I tried joining them on the following keys:

\n
on = [\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    \n]\n
\n

But I still found about 300,000 rows couldn’t be matched correctly.
\nWould appreciate any advice on how to better align them. Thanks!

", "rawMarkdown": "Hi @samvelkoch, thanks for the clarification.\n\nIs there any recommended way to reliably align the option order between the json file and the parquet file?\nToday I tried joining them on the following keys:\n\n```python\non = [\n \"ranker_id\",\n \"legs0_departureAt\",\n \"legs0_arrivalAt\",\n \"legs1_departureAt\",\n \"legs1_arrivalAt\",\n \"legs0_segments0_flightNumber\",\n \"legs1_segments0_flightNumber\",\n \"totalPrice\"\n]\n```\n\nBut I still found about 300,000 rows couldn’t be matched correctly.\nWould appreciate any advice on how to better align them. Thanks!", "votes": null }, { "id": "3253770", "postDate": "07/25/2025 08:15:07", "content": "

And I also tried grouping both the train and test datasets using:

\n
train = pl.read_parquet(os.path.join(, )).drop()\ntest = (\n    pl.read_parquet(os.path.join(, ))\n    .drop()\n    .with_columns(pl.lit(, dtype=pl.Int64).alias())\n)\n\ntrain_test = pl.concat((train, test))\n\ntrain_test_df = train_test.group_by([\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n]).agg(pl.())\n\n(train_test_df)\n
\n

This gives me a grouped dataframe of shape (8,165,696, 6).

\n

However, when I group the same data using the raw json files, I get only (7,489,609, 3) groups.

\n

Is there any known inconsistency or reason why the number of groups extracted from the parquet files differs significantly from the JSON source?

", "rawMarkdown": "And I also tried grouping both the train and test datasets using:\n\n```python\ntrain = pl.read_parquet(os.path.join(\"data\", \"train.parquet\")).drop(\"__index_level_0__\")\ntest = (\n pl.read_parquet(os.path.join(\"data\", \"test.parquet\"))\n .drop(\"__index_level_0__\")\n .with_columns(pl.lit(0, dtype=pl.Int64).alias(\"selected\"))\n)\n\ntrain_test = pl.concat((train, test))\n\ntrain_test_df = train_test.group_by([\n \"ranker_id\",\n \"legs0_duration\",\n \"legs1_duration\",\n \"legs0_departureAt\",\n \"legs0_arrivalAt\",\n \"legs1_departureAt\",\n \"legs1_arrivalAt\",\n \"legs0_segments0_flightNumber\",\n \"legs1_segments0_flightNumber\",\n]).agg(pl.len())\n\nprint(train_test_df)\n```\n\nThis gives me a grouped dataframe of shape (8,165,696, 6).\n\nHowever, when I group the same data using the raw json files, I get only (7,489,609, 3) groups.\n\nIs there any known inconsistency or reason why the number of groups extracted from the parquet files differs significantly from the JSON source?", "votes": null }, { "id": "3253793", "postDate": "07/25/2025 09:16:34", "content": "

Thanks for diving so deep into the data @mango789. I really would like to be helpful in both of your questions. We'll see if I found quick answer without hard reverse engineering research.

", "rawMarkdown": "Thanks for diving so deep into the data @mango789. I really would like to be helpful in both of your questions. We'll see if I found quick answer without hard reverse engineering research.", "votes": null }, { "id": "3253804", "postDate": "07/25/2025 09:41:54", "content": "

Thanks for the reply, @samvelkoch. Appreciate you taking the time to look into it.

", "rawMarkdown": "Thanks for the reply, @samvelkoch. Appreciate you taking the time to look into it.", "votes": null }, { "id": "3253916", "postDate": "07/25/2025 13:48:46", "content": "

@mango789 it seems the issue roots are in the logic of data deduplication and cleaning in the ETL json -> parquet pipeline. My suggestion is to focus on parquets as the primary source. It doesn't mean that jsons are useless.

", "rawMarkdown": "mango789 it seems the issue roots are in the logic of data deduplication and cleaning in the ETL json -> parquet pipeline. My suggestion is to focus on parquets as the primary source. It doesn't mean that jsons are useless.", "votes": null }, { "id": "3269513", "postDate": "08/14/2025 15:22:01", "content": "

I also found some mismatch within these features from json to parquet file

\n

Also, I found the total data is different between json and parquet file although they have the same ranker_id.

\n

you can check with

\n
, , , ,\n\n\ntarget_rid = \n\ntarget_rows = (\n    train_filled\n    .(pl.col() == target_rid)\n    .(pl.col().is_in([]))\n)\n\n\ntarget_rid = \n\ntarget_rows = (\n    train_filled\n    .(pl.col() == target_rid)\n    .(pl.col().is_in([, , ]))\n    .(pl.col() == )\n\n)\n\n\n\n\ntarget_rid = \n\ntarget_rows = (\n    train_filled\n    .(pl.col() == target_rid)\n    .(pl.col().is_in([, ]))\n    .(pl.col() == )\n\n)\n
", "rawMarkdown": "I also found some mismatch within these features from json to parquet file\n\nAlso, I found the total data is different between json and parquet file although they have the same ranker_id.\n\n you can check with \n\n```python\n'legs0_segments1_baggageAllowance_quantity','legs0_segments1_seatsAvailable' , \"miniRules0_monetaryAmount\", \"miniRules1_monetaryAmount\",\n\n\ntarget_rid = \"0f4a3e1286794696a08882bbb7813497\"\n\ntarget_rows = (\n train_filled\n .filter(pl.col(\"ranker_id\") == target_rid)\n .filter(pl.col(\"legs0_segments0_flightNumber\").is_in([\"208\"]))\n)\n\n\ntarget_rid = \"f7d878ee3ac944b38d084cf09e15f210\"\n\ntarget_rows = (\n train_filled\n .filter(pl.col(\"ranker_id\") == target_rid)#'legs0_segments2_baggageAllowance_quantity',\n .filter(pl.col(\"legs0_segments1_flightNumber\").is_in([\"5309\", \"5305\", \"6371\"]))\n .filter(pl.col(\"legs0_segments0_seatsAvailable\") == 2)\n\n)\n\n\n\n# 'legs0_segments2_seatsAvailable',#'legs0_segments2_baggageAllowance_quantity',\ntarget_rid = \"938b94a6d6584757a857ca180c9a060a\"\n\ntarget_rows = (\n train_filled\n .filter(pl.col(\"ranker_id\") == target_rid)\n .filter(pl.col(\"legs0_segments0_flightNumber\").is_in([\"1008\", \"1004\"]))\n .filter(pl.col(\"totalPrice\") == 262862.0)\n\n)\n\n```", "votes": null } ], "comments": [ { "id": 3251008, "author_name": "samvelkoch", "author_url": "", "post_date": "07/19/2025 17:07:36", "content": "

Hi @mango789. Thanks for asking. We've decided to share our json_converters as a reference. Hope that helps. But please don't forget it couldn't be applied directly to the competition jsons since some ETL and sensitive data truncations.

", "votes": null, "replies": [ { "id": 3251096, "author_name": "mango789", "author_url": "", "post_date": "07/20/2025 00:01:12", "content": "

Thanks a lot for sharing the json_converters and the note about ETL/data truncations. That’s very helpful!🥺

", "votes": null, "replies": [] }, { "id": 3253340, "author_name": "mango789", "author_url": "", "post_date": "07/24/2025 13:45:56", "content": "

Hi @samvelkoch , I have a another quick question: within the JSON files, is the order of the flights preserved exactly in the parquet files for the same ranker_id group? In other words, can I assume the flights’ order in json matches the order in the parquet rows for that ranker_id?
\nThanks in advance for your help!

", "votes": null, "replies": [ { "id": 3253358, "author_name": "samvelkoch", "author_url": "", "post_date": "07/24/2025 14:16:12", "content": "

Hi @mango789. We just can't guarantee that. This assumption could not be 100% truth and I have no possibility to check it for sure now. So the answer I have to give you is NO.

", "votes": null, "replies": [ { "id": 3253765, "author_name": "mango789", "author_url": "", "post_date": "07/25/2025 08:09:36", "content": "

Hi @samvelkoch, thanks for the clarification.

\n

Is there any recommended way to reliably align the option order between the json file and the parquet file?
\nToday I tried joining them on the following keys:

\n
on = [\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    \n]\n
\n

But I still found about 300,000 rows couldn’t be matched correctly.
\nWould appreciate any advice on how to better align them. Thanks!

", "votes": null, "replies": [] }, { "id": 3253770, "author_name": "mango789", "author_url": "", "post_date": "07/25/2025 08:15:07", "content": "

And I also tried grouping both the train and test datasets using:

\n
train = pl.read_parquet(os.path.join(, )).drop()\ntest = (\n    pl.read_parquet(os.path.join(, ))\n    .drop()\n    .with_columns(pl.lit(, dtype=pl.Int64).alias())\n)\n\ntrain_test = pl.concat((train, test))\n\ntrain_test_df = train_test.group_by([\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n    ,\n]).agg(pl.())\n\n(train_test_df)\n
\n

This gives me a grouped dataframe of shape (8,165,696, 6).

\n

However, when I group the same data using the raw json files, I get only (7,489,609, 3) groups.

\n

Is there any known inconsistency or reason why the number of groups extracted from the parquet files differs significantly from the JSON source?

", "votes": null, "replies": [ { "id": 3253793, "author_name": "samvelkoch", "author_url": "", "post_date": "07/25/2025 09:16:34", "content": "

Thanks for diving so deep into the data @mango789. I really would like to be helpful in both of your questions. We'll see if I found quick answer without hard reverse engineering research.

", "votes": null, "replies": [ { "id": 3253804, "author_name": "mango789", "author_url": "", "post_date": "07/25/2025 09:41:54", "content": "

Thanks for the reply, @samvelkoch. Appreciate you taking the time to look into it.

", "votes": null, "replies": [ { "id": 3253916, "author_name": "samvelkoch", "author_url": "", "post_date": "07/25/2025 13:48:46", "content": "

@mango789 it seems the issue roots are in the logic of data deduplication and cleaning in the ETL json -> parquet pipeline. My suggestion is to focus on parquets as the primary source. It doesn't mean that jsons are useless.

", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 3269513, "author_name": "dingyangwang", "author_url": "", "post_date": "08/14/2025 15:22:01", "content": "

I also found some mismatch within these features from json to parquet file

\n

Also, I found the total data is different between json and parquet file although they have the same ranker_id.

\n

you can check with

\n
, , , ,\n\n\ntarget_rid = \n\ntarget_rows = (\n    train_filled\n    .(pl.col() == target_rid)\n    .(pl.col().is_in([]))\n)\n\n\ntarget_rid = \n\ntarget_rows = (\n    train_filled\n    .(pl.col() == target_rid)\n    .(pl.col().is_in([, , ]))\n    .(pl.col() == )\n\n)\n\n\n\n\ntarget_rid = \n\ntarget_rows = (\n    train_filled\n    .(pl.col() == target_rid)\n    .(pl.col().is_in([, ]))\n    .(pl.col() == )\n\n)\n
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3250805": "Hi organizers,\n\nThanks for providing both the raw data and the preprocessed file `train.parquet`. \nFor better understanding and to ensure consistency when I extract additional fields from the raw data, \ncould you share the code (or a simplified version) used to generate `train.parquet`?\n\nHaving the preprocessing script would make it easier for participants to extend or reproduce the dataset \nwhile maintaining the same logic.\n\nThanks in advance!", "3251008": "Hi @mango789. Thanks for asking. We've decided to share our [json_converters](https://www.kaggle.com/code/samvelkoch/reference-json-converters ) as a reference. Hope that helps. But please don't forget it couldn't be applied directly to the competition jsons since some ETL and sensitive data truncations.", "3251096": "Thanks a lot for sharing the json_converters and the note about ETL/data truncations. That’s very helpful!🥺", "3253340": "Hi @samvelkoch , I have a another quick question: within the JSON files, is the order of the flights preserved exactly in the parquet files for the same `ranker_id` group? In other words, can I assume the flights’ order in json matches the order in the parquet rows for that `ranker_id`?\nThanks in advance for your help!", "3253358": "Hi @mango789. We just can't guarantee that. This assumption could not be 100% truth and I have no possibility to check it for sure now. So the answer I have to give you is NO.", "3253765": "Hi @samvelkoch, thanks for the clarification.\n\nIs there any recommended way to reliably align the option order between the json file and the parquet file?\nToday I tried joining them on the following keys:\n\n```python\non = [\n \"ranker_id\",\n \"legs0_departureAt\",\n \"legs0_arrivalAt\",\n \"legs1_departureAt\",\n \"legs1_arrivalAt\",\n \"legs0_segments0_flightNumber\",\n \"legs1_segments0_flightNumber\",\n \"totalPrice\"\n]\n```\n\nBut I still found about 300,000 rows couldn’t be matched correctly.\nWould appreciate any advice on how to better align them. Thanks!", "3253770": "And I also tried grouping both the train and test datasets using:\n\n```python\ntrain = pl.read_parquet(os.path.join(\"data\", \"train.parquet\")).drop(\"__index_level_0__\")\ntest = (\n pl.read_parquet(os.path.join(\"data\", \"test.parquet\"))\n .drop(\"__index_level_0__\")\n .with_columns(pl.lit(0, dtype=pl.Int64).alias(\"selected\"))\n)\n\ntrain_test = pl.concat((train, test))\n\ntrain_test_df = train_test.group_by([\n \"ranker_id\",\n \"legs0_duration\",\n \"legs1_duration\",\n \"legs0_departureAt\",\n \"legs0_arrivalAt\",\n \"legs1_departureAt\",\n \"legs1_arrivalAt\",\n \"legs0_segments0_flightNumber\",\n \"legs1_segments0_flightNumber\",\n]).agg(pl.len())\n\nprint(train_test_df)\n```\n\nThis gives me a grouped dataframe of shape (8,165,696, 6).\n\nHowever, when I group the same data using the raw json files, I get only (7,489,609, 3) groups.\n\nIs there any known inconsistency or reason why the number of groups extracted from the parquet files differs significantly from the JSON source?", "3253793": "Thanks for diving so deep into the data @mango789. I really would like to be helpful in both of your questions. We'll see if I found quick answer without hard reverse engineering research.", "3253804": "Thanks for the reply, @samvelkoch. Appreciate you taking the time to look into it.", "3253916": "mango789 it seems the issue roots are in the logic of data deduplication and cleaning in the ETL json -> parquet pipeline. My suggestion is to focus on parquets as the primary source. It doesn't mean that jsons are useless.", "3269513": "I also found some mismatch within these features from json to parquet file\n\nAlso, I found the total data is different between json and parquet file although they have the same ranker_id.\n\n you can check with \n\n```python\n'legs0_segments1_baggageAllowance_quantity','legs0_segments1_seatsAvailable' , \"miniRules0_monetaryAmount\", \"miniRules1_monetaryAmount\",\n\n\ntarget_rid = \"0f4a3e1286794696a08882bbb7813497\"\n\ntarget_rows = (\n train_filled\n .filter(pl.col(\"ranker_id\") == target_rid)\n .filter(pl.col(\"legs0_segments0_flightNumber\").is_in([\"208\"]))\n)\n\n\ntarget_rid = \"f7d878ee3ac944b38d084cf09e15f210\"\n\ntarget_rows = (\n train_filled\n .filter(pl.col(\"ranker_id\") == target_rid)#'legs0_segments2_baggageAllowance_quantity',\n .filter(pl.col(\"legs0_segments1_flightNumber\").is_in([\"5309\", \"5305\", \"6371\"]))\n .filter(pl.col(\"legs0_segments0_seatsAvailable\") == 2)\n\n)\n\n\n\n# 'legs0_segments2_seatsAvailable',#'legs0_segments2_baggageAllowance_quantity',\ntarget_rid = \"938b94a6d6584757a857ca180c9a060a\"\n\ntarget_rows = (\n train_filled\n .filter(pl.col(\"ranker_id\") == target_rid)\n .filter(pl.col(\"legs0_segments0_flightNumber\").is_in([\"1008\", \"1004\"]))\n .filter(pl.col(\"totalPrice\") == 262862.0)\n\n)\n\n```" }, "source": "meta" }