{"metadata":{"kernelspec":{"display_name":"Python (FlightRec)","language":"python","name":"flightrec-env"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.16"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":105399,"databundleVersionId":12733338,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"a31f8be3-a904-45fc-9c27-c665d50bca51","cell_type":"markdown","source":"# ✈️Flight Selection Analysis and Prediction — *FlightRank 2025*\n\nThis notebook tackles a **ranking-based recommendation system** challenge applied to the context of **corporate travel**. The goal is to predict which flight a user (business traveler) is most likely to select among multiple available options in a search session.\n\n> 📎 The complete project is available on [GitHub — FlightRank 2025](https://github.com/LeopoldoZanellato/FlightRank-2025)\n\n---\n\n## 🎯Problem and Objective\n\nThis is a **supervised learning problem with a ranking focus**, where observations are grouped by search sessions (`ranker_id`).\n\n- Each `ranker_id` represents a real flight search.\n- Each group contains multiple flight options.\n- Exactly **one** of these options was selected (`selected = 1`).\n\nOur goal is to **train a model that ranks the options correctly** so that the flight chosen by the user appears among the top-ranked options.\n\n---\n\n## 📏Evaluation Metric: HitRate@3\n\nThe official competition metric is **HitRate@3**, which checks whether the flight actually chosen appears among the **top 3 ranked options** for each search group.\n\nThe formula is:\n\n`HitRate@3 = (1 / |Q|) * ∑ 𝟙(rank_i ≤ 3)`\n\nWhere:\n\n- `|Q|` is the number of evaluated search sessions (with more than 10 flights).\n- `rank_i` is the rank assigned by the model to the correct flight in session `i`.\n- `𝟙(rank_i ≤ 3)` equals 1 if the flight is in the top 3, and 0 otherwise.\n\n> **Important:** only sessions with **more than 10 flight options** are considered in the final metric.\n\n","metadata":{}},{"id":"5e818c6b-6436-48ff-a5b3-de7839371ce4","cell_type":"code","source":"import pandas as pd\nimport os\nimport subprocess\nimport zipfile\nimport matplotlib.pyplot as plt\nimport lightgbm as lgb\nimport numpy as np\n\nfrom itertools import product\nfrom sklearn.model_selection import GroupShuffleSplit\nfrom itertools import chain\nfrom sklearn.model_selection import GroupKFold","metadata":{},"outputs":[],"execution_count":null},{"id":"02ac1ef6-094e-4b16-bbe1-1b97481f37ed","cell_type":"code","source":"# resetando as configurações \npd.reset_option('display.max_columns')\n\n# setando as configurações de coluna maxima\npd.set_option('display.max_columns', None)","metadata":{},"outputs":[],"execution_count":null},{"id":"7bc862a0-a756-4ead-b8d1-83c57855a0fc","cell_type":"markdown","source":"## ⚙️Download and Extraction of Data\n\nBefore starting the analysis, we need to ensure that the competition data is available locally.\n\nThis step performs:\n\n1. **Verification**: checks whether the files have already been downloaded.\n2. **Automatic download** via the Kaggle API (if necessary).\n3. **Extraction** of the `.zip` files into the `data/aeroclub/` folder.\n\n> The Kaggle API was configured using the `kaggle.json` file directly on Windows, without using the project directory.\n\n","metadata":{}},{"id":"a33d7fd1-d2de-4d9e-b223-12c08dea8ac2","cell_type":"code","source":"def download_files():\n    # Define caminhos\n    zip_path = \"data/aeroclub-recsys-2025.zip\"\n    extract_path = \"data/aeroclub\"\n    \n    # Cria a pasta base se necessário\n    os.makedirs(\"data\", exist_ok=True)\n\n    # Verifica se o arquivo .zip já foi baixado\n    if not os.path.exists(zip_path):\n        print(\"🔽 Baixando arquivos da competição...\")\n        subprocess.run([\n            \"kaggle\", \"competitions\", \"download\",\n            \"-c\", \"aeroclub-recsys-2025\",\n            \"-p\", \"data\"\n        ])\n    else:\n        print(\"✅ Arquivo ZIP já existe. Pulando download.\")\n\n    # Verifica se os arquivos já foram extraídos\n    if not os.path.exists(extract_path) or not os.listdir(extract_path):\n        print(\"📦 Extraindo arquivos...\")\n        with zipfile.ZipFile(zip_path, 'r') as zip_ref:\n            zip_ref.extractall(extract_path)\n    else:\n        print(\"✅ Arquivos já extraídos. Pulando extração.\")","metadata":{},"outputs":[],"execution_count":null},{"id":"af2958ae-c6ea-47fc-884c-d67edd9483eb","cell_type":"code","source":"# Executa\ndownload_files()","metadata":{},"outputs":[],"execution_count":null},{"id":"cc1bcffb-081b-42e2-87f7-6393d60b614b","cell_type":"markdown","source":"## 📚 3. Data Loading\n\nIn this step, we load the **training dataset**, available in the `train.parquet` file.\n\nThis dataset contains complete information about **flight search sessions**, including:\n\n- Flight and user identifiers  \n- Company-related data  \n- Route and schedule information  \n- Total price and taxes  \n- Cancellation and rebooking rules  \n- Indication of which flight was selected (`selected = 1`)\n\n> This will be the **main dataset** used for building, training, and validating the recommendation model.\n\n","metadata":{}},{"id":"076ee0c7-6aa1-404d-9285-b3dd597542cd","cell_type":"code","source":"train = pd.read_parquet(\"data/aeroclub/train.parquet\")","metadata":{},"outputs":[],"execution_count":null},{"id":"f36308db-3700-4225-9285-9c833ce76a32","cell_type":"code","source":"def reduce_memory_usage(df):\n    for col in df.columns:\n        col_type = df[col].dtypes\n        \n        if col_type == 'float64':\n            df[col] = pd.to_numeric(df[col], downcast='float')\n        elif col_type == 'int64':\n            df[col] = pd.to_numeric(df[col], downcast='integer')\n        elif col_type == 'object':\n            num_unique = df[col].nunique()\n            num_total = len(df[col])\n            if num_unique / num_total < 0.5:\n                df[col] = df[col].astype('category')\n    \n    return df\ntrain = reduce_memory_usage(train)","metadata":{},"outputs":[],"execution_count":null},{"id":"de897686-07fa-4cbc-887b-d680cbbfa9d4","cell_type":"code","source":"df_train_raw = train.copy()","metadata":{},"outputs":[],"execution_count":null},{"id":"3cfd46f3-15fa-4932-93e8-135b0e52e7d6","cell_type":"markdown","source":"## 🧹 4. Selection of Relevant Columns\n\nWith the dataset loaded, the next step is to **select only the most relevant columns** for the baseline model.\n\nThe focus is on keeping variables that provide **useful information for flight recommendation**, including:\n\n- Identifiers (`Id`, `ranker_id`, `profileId`, etc.)\n- Passenger and company information\n- Route details, timings, and connections\n- Price data, taxes, and refund/exchange policies\n- Flight segment details (airline, seat, baggage)\n- Target variable: `selected` (indicates the flight chosen by the user)\n\n> This step reduces dimensionality and improves model performance by eliminating irrelevant or redundant columns.\n\n---\n\n### Optional Sampling for Prototyping\n\nDuring early experiments, it is common to work with a **reduced sample of the data** to speed up iteration. For that, we created a helper function that allows:\n\n- Selecting only the desired columns  \n- Optionally limiting the number of rows loaded\n\n### Examples:\n\n```python\n# Load only a sample (1 million rows)\ndf_train = load_subset(df_train_raw, columns_to_keep, max_rows=1_000_000)\n\n# Load the full dataset (all rows)\ndf_train = load_subset(df_train_raw, columns_to_keep)\n\n","metadata":{}},{"id":"63def090-b4c1-4a4e-964e-7f6cfdbfea7c","cell_type":"code","source":"# Define as colunas que você quer manter\ncolumns_to_keep = [\n    # Identifiers\n    'Id',  # num\n    'ranker_id', \n    'profileId', \n    'companyID',\n    \n    # User info\n    'sex', 'nationality', 'frequentFlyer', 'isVip', 'bySelf', 'isAccess3D',\n\n    # Company info\n    'corporateTariffCode',\n\n    # Search & route\n    'searchRoute', 'requestDate',\n\n    # Pricing\n    'totalPrice', 'taxes',\n\n    # Flight timing\n    'legs0_departureAt', 'legs0_arrivalAt', 'legs0_duration',\n    'legs1_departureAt', 'legs1_arrivalAt', 'legs1_duration',\n\n    # Segment-level info (só do segmento 0 da ida para simplificar no baseline)\n    'legs0_segments0_departureFrom_airport_iata',\n    'legs0_segments0_arrivalTo_airport_iata',\n    'legs0_segments0_arrivalTo_airport_city_iata',\n    'legs0_segments0_marketingCarrier_code',\n    'legs0_segments0_operatingCarrier_code',\n    'legs0_segments0_aircraft_code',\n    'legs0_segments0_flightNumber',\n    'legs0_segments0_duration',\n    'legs0_segments0_baggageAllowance_quantity',\n    'legs0_segments0_baggageAllowance_weightMeasurementType',\n    'legs0_segments0_cabinClass',\n    'legs0_segments0_seatsAvailable', \n    'legs0_segments1_departureFrom_airport_iata',\n    'legs0_segments2_departureFrom_airport_iata',\n    'legs0_segments3_departureFrom_airport_iata',\n\n    # Cancellation & exchange rules\n    'miniRules0_monetaryAmount', 'miniRules0_percentage', 'miniRules0_statusInfos',\n    'miniRules1_monetaryAmount', 'miniRules1_percentage', 'miniRules1_statusInfos',\n\n    # Pricing policy\n    'pricingInfo_isAccessTP', 'pricingInfo_passengerCount',\n\n    # Target\n    'selected'\n]\n\n# Filtra os dados para o baseline\ndef load_subset(df, columns,  max_rows=None):\n    if max_rows:\n        return df[columns].iloc[:max_rows].copy()\n    else:\n        return df[columns].copy()\n\n# Exemplo de uso\ndf_train = load_subset(df_train_raw, columns_to_keep, max_rows=1_000_000) # ONLY 1M\n\n#############################          IMPORTANT      ########################################\n#############################          IMPORTANT      ########################################\n#############################          IMPORTANT      ########################################\n#############################          IMPORTANT      ########################################\n#df_train = load_subset(df_train_raw, columns_to_keep) # ALL REGISTERS","metadata":{},"outputs":[],"execution_count":null},{"id":"68857a98-3798-4c23-ae77-ae195801b902","cell_type":"markdown","source":"## 🛠️ 5. Feature Engineering\n\nIn this step, we transform raw columns into more informative, consistent, and suitable variables for use in machine learning models.\n\nThe features will be built in subtopics, organized by type of transformation.\n\n---\n\n### 🧾 5.1 Data Type Correction (`dtypes`)\n\nThe first step is to ensure that data types are correct and optimized.\n\n- Categorical columns initially encoded as `category` are analyzed and converted to:\n  - `int` or `float`, when possible  \n  - `bool`, if they contain only logical values  \n  - `category`, in all other cases\n\n- The `nationality` column, which arrives as an integer, is converted to `string` to preserve its categorical meaning.\n\n> This standardization is essential to avoid errors and ensure that the model interprets variables correctly.","metadata":{}},{"id":"485ce5ef-673e-4fdd-a7ee-075056c752c9","cell_type":"code","source":"def fix_column_types(df):\n    df_fixed = df.copy()\n    for col in df.columns:\n        if isinstance(df[col].dtype, pd.CategoricalDtype):\n            # Tenta converter para tipo numérico\n            try:\n                df_fixed[col] = pd.to_numeric(df[col])\n            except:\n                # Se não for numérico, tenta bool\n                unique_vals = df[col].dropna().unique()\n                if set(unique_vals) <= {True, False}:\n                    df_fixed[col] = df[col].astype(bool)\n                else:\n                    df_fixed[col] = df[col].astype(str)\n    return df_fixed\ndf_train = fix_column_types(df_train)\n\n# Ajusta a nacionalidade (está em Int)\ndf_train[\"nationality\"] = df_train[\"nationality\"].astype(\"str\")\ndf_train['companyID'] = df_train['companyID'].astype('category')\n\ndf_train.dtypes  # Checar resultado","metadata":{},"outputs":[],"execution_count":null},{"id":"06fd9eb3-0645-483c-95f4-52c95ba551f3","cell_type":"markdown","source":"### 🛫 5.2 Number of Segments in the Outbound Flight\n\nIn this step, we create variables related to the **structure of the outbound flight**, focusing on the number of connections.\n\n#### What is being done:\n\n- **`n_segments_ida`**: calculates the total number of segments in the outbound leg.\n  - By definition, every flight has at least **segment 0** (origin to first destination).\n  - If there are connections (segments 1, 2, or 3), this number is incremented.\n  \n- **`has_connections_ida`**: a boolean variable indicating whether the outbound flight has one or more connections.\n\n- After extracting this information, the auxiliary columns for segments 1 to 3 are dropped, as they are no longer needed directly.\n\n> These features help the model distinguish between direct and connecting flights, which may influence the corporate traveler's decision.","metadata":{}},{"id":"61244de5-f0be-4e43-a5ae-ca1592b87820","cell_type":"code","source":"# 1. Cria a feature 'n_segments_ida' baseada na presença de segmentos 1, 2, 3\nsegments_ida_cols = {\n    1: 'legs0_segments1_departureFrom_airport_iata',\n    2: 'legs0_segments2_departureFrom_airport_iata',\n    3: 'legs0_segments3_departureFrom_airport_iata'\n}\n\n# Começa com 1 porque o segmento 0 está sempre presente\ndf_train['n_segments_ida'] = 1\n\nfor seg, col in segments_ida_cols.items():\n    df_train['n_segments_ida'] += df_train[col].notnull().astype(int)\n\n# 2. Cria a flag booleana se há conexões (mais de 1 segmento na ida)\ndf_train['has_connections_ida'] = (df_train['n_segments_ida'] > 1).astype('boolean')\n\ndf_train.drop('legs0_segments1_departureFrom_airport_iata', inplace=True, axis=1)\ndf_train.drop('legs0_segments2_departureFrom_airport_iata', inplace=True, axis=1)\ndf_train.drop('legs0_segments3_departureFrom_airport_iata', inplace=True, axis=1)","metadata":{},"outputs":[],"execution_count":null},{"id":"6f97c101-156d-4623-9a36-ea22e8fc69e3","cell_type":"markdown","source":"### 🎫 5.3 Frequent Flyer Programs (`frequentFlyer`)\n\nThe `frequentFlyer` column indicates which loyalty programs the passenger is associated with. Since it contains multiple codes concatenated with \"/\", it needs to be processed into more informative and model-friendly features.\n\n#### 🧠 Applied transformations:\n\n- **`frequentFlyer_count`**: indicates how many frequent flyer programs the passenger participates in (number of codes separated by \"/\").\n\n- **`hasFrequentFlyer`**: a binary variable indicating whether the passenger participates in at least one program.\n\n- **`ff_XXX`**: individual boolean columns for each airline, representing whether the passenger is a member of that specific loyalty program.\n\n- After feature extraction, the original `frequentFlyer` column is removed from the dataset, as its information is now decomposed into more specific variables.\n\n> These features help capture the user's affinity with specific airlines, which can strongly influence their flight choice.","metadata":{}},{"id":"8921672f-f88c-4855-86fe-c1a1449e0609","cell_type":"code","source":"def count_frequent_flyers(value):\n    if pd.isna(value):\n        return 0\n    return len(str(value).split('/'))\n\ndf_train['frequentFlyer_count'] = df_train['frequentFlyer'].apply(count_frequent_flyers)\n\n# Cria flag binária para frequent flyer\ndf_train['hasFrequentFlyer'] = df_train['frequentFlyer'].notnull().astype(int)\n\n# Substituir valores NaN por string vazia\nff_series = df_train['frequentFlyer'].fillna('').astype(str)\n\n# Dividir por '/' para obter lista\nff_lists = ff_series.str.split('/')\n\nall_programs = set(chain.from_iterable(ff_lists))\nprint(f\"Total de companhias únicas: {len(all_programs)}\")\n\ndf_train.drop('frequentFlyer', axis=1, inplace=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"6c0d499e-6dfb-47f5-a6ba-ca75407f577d","cell_type":"markdown","source":"### ⏰ 5.4 Processing Dates, Times, and Durations\n\nIn this step, we extract temporal information from date and time columns, and convert duration columns into numerical formats.\n\n#### 📅 Applied transformations:\n\n- **Datetime conversion**: the columns `requestDate`, `legs0_departureAt`, `legs0_arrivalAt`, `legs1_departureAt`, and `legs1_arrivalAt` are converted to `datetime` type.\n\n- **Creation of new temporal variables**:\n  - `legs0_dep_hour` / `legs1_dep_hour`: departure hour (outbound and return).\n  - `legs0_dep_dayofweek` / `legs1_dep_dayofweek`: day of the week of departure.\n  - `trip_days`: trip duration in days (return - outbound).\n  - `booking_to_trip_days`: number of days between the search date and the departure.\n\n- **Binary flags**:\n  - `ida_fds` / `volta_fds`: indicates whether the flight occurs on a weekend.\n  - `ida_comercial` / `volta_comercial`: indicates whether the flight occurs during business hours (between 07:00 and 19:00).\n\n#### ⏱️ Conversion of durations to minutes\n\n- The columns `legs0_duration` and `legs1_duration` (in text format) are converted into **total duration in minutes**, becoming numeric variables.\n\n- The `legs0_segments0_duration` column, corresponding to the first segment of the outbound flight, is also converted into minutes in a new variable: `legs0_duration_minutes`.\n\n> Handling temporal variables is essential for capturing behavioral patterns, such as preferences for daytime flights, short trips, or early booking.","metadata":{}},{"id":"cf13fb0a-6db7-475e-ab4b-8bf5bd8d9d25","cell_type":"code","source":"# 🗓️ Colunas de datas e horários\ncols_datetime = [\n    'requestDate',\n    'legs0_departureAt', 'legs0_arrivalAt',\n    'legs1_departureAt', 'legs1_arrivalAt'\n]\ndef process_datetime_and_duration(df):\n    df_processed = df.copy()\n\n    # Datas para datetime\n    for col in cols_datetime:\n        df_processed[col] = pd.to_datetime(df_processed[col], errors='coerce')\n\n    # Features de hora e dia da semana\n    df_processed['legs0_dep_hour'] = df_processed['legs0_departureAt'].dt.hour\n    df_processed['legs0_dep_dayofweek'] = df_processed['legs0_departureAt'].dt.dayofweek\n    df_processed['legs1_dep_hour'] = df_processed['legs1_departureAt'].dt.hour\n    df_processed['legs1_dep_dayofweek'] = df_processed['legs1_departureAt'].dt.dayofweek\n\n    # Dias entre ida e volta (duração da viagem)\n    df_processed['trip_days'] = (df_processed['legs1_departureAt'] - df_processed['legs0_departureAt']).dt.days\n\n    # Dias de antecedência (request → ida)\n    df_processed['booking_to_trip_days'] = (df_processed['legs0_departureAt'] - df_processed['requestDate']).dt.days\n\n    # Final de semana (ida/volta)\n    df_processed['ida_fds'] = df_processed['legs0_dep_dayofweek'].isin([5, 6]).astype(int)\n    df_processed['volta_fds'] = df_processed['legs1_dep_dayofweek'].isin([5, 6]).astype(int)\n\n    # Horário comercial (7h às 19h)\n    def is_business_hour(hour):\n        return int(7 <= hour <= 19)\n\n    df_processed['ida_comercial'] = df_processed['legs0_dep_hour'].apply(is_business_hour)\n    df_processed['volta_comercial'] = df_processed['legs1_dep_hour'].apply(is_business_hour)\n\n    # ⏱️ Converter colunas de duração para minutos\n    def clean_and_convert_duration(col):\n        return (\n            col\n            .fillna(\"00:00:00\")\n            .astype(str)\n            .str.strip()\n            .str.replace(\"nan\", \"00:00:00\")\n            .pipe(pd.to_timedelta, errors='coerce')\n            .dt.total_seconds() / 60  # minutos\n        )\n\n    cols_duration = ['legs0_duration', 'legs1_duration']\n    for col in cols_duration:\n        df_processed[col] = clean_and_convert_duration(df_processed[col])\n\n    return df_processed","metadata":{},"outputs":[],"execution_count":null},{"id":"d1fbbbfc-0d31-4da9-89d9-944415d9a23b","cell_type":"code","source":"df_train['legs0_duration_minutes'] = (\n    pd.to_timedelta(\n        df_train['legs0_segments0_duration'].fillna(\"00:00:00\").astype(str).str.strip(),\n        errors='coerce'\n    ).dt.total_seconds() / 60  # em minutos\n)\n\ndf_train.drop('legs0_segments0_duration', axis=1, inplace=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"c7b38f93-10e2-4af6-a911-957e0c470153","cell_type":"code","source":"# ✅ Applicação\ndf_train = process_datetime_and_duration(df_train)\ndf_train.drop(columns=cols_datetime, inplace=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"404c1d77-7a61-4caf-827b-ba82772b63b8","cell_type":"markdown","source":"### 📍 5.5 Processing the Search Route (`searchRoute`)\n\nThe `searchRoute` column represents the full searched travel route, including both outbound and return legs, and is encoded as a string in the format:\n\nOUTBOUND/RETURN → e.g., \"GRUFOR/FORGRU\" or \"GRUCGHFOR/FORGRUCGH\"\n\n    \n#### 🛠️ Applied transformations:\n\n- The `searchRoute` column is converted to `string` type to ensure consistency.\n\n- The string is split into two parts:\n  - **`route_ida`**: outbound leg of the trip\n  - **`route_volta`**: return leg of the trip (if present)\n\n- From each leg, the following are extracted:\n  - **`ida_from`** and **`ida_to`**: origin and destination of the outbound leg\n  - **`volta_from`** and **`volta_to`**: origin and destination of the return leg\n\n- An auxiliary variable `searchRoute_count` was also created to indicate the number of legs in the route (via splitting by \"/\"), used only for validation and later removed.\n\n- The original `searchRoute` column was dropped after decomposing the information.\n\n> This decomposition allows the model to capture origin-destination patterns, as well as distinct behaviors in multi-leg routes — valuable insights for the recommendation task.","metadata":{}},{"id":"7c9ef241-fb6b-439d-b6a1-0e55c58e2d96","cell_type":"code","source":"df_train['searchRoute'] = df_train['searchRoute'].astype(str)\ndf_train['searchRoute_count'] = df_train['searchRoute'].apply(lambda x: x.split(\"/\"))\ndf_train['searchRoute_count'] = df_train['searchRoute_count'].apply(lambda x: len(x))\nprint(f\" min {min(df_train['searchRoute_count'])}\")\nprint(f\" max {max(df_train['searchRoute_count'])}\")\ndf_train.drop('searchRoute_count', axis=1, inplace=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"3118a7d4-2852-4735-a93d-522c3c72dcbb","cell_type":"code","source":"# Garante que searchRoute está como string\ndf_train['searchRoute'] = df_train['searchRoute'].astype(str)\n\n# Separa ida e volta\ndf_train[['route_ida', 'route_volta']] = df_train['searchRoute'].str.split('/', expand=True)\n\n# Extrai origem e destino da ida\ndf_train['ida_from'] = df_train['route_ida'].str[:3]\ndf_train['ida_to'] = df_train['route_ida'].str[3:]\n\n# Extrai origem e destino da volta (se existir)\ndf_train['volta_from'] = df_train['route_volta'].str[:3]\ndf_train['volta_to'] = df_train['route_volta'].str[3:]\n\ndf_train.drop('searchRoute', axis=1, inplace=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"0ba8dd5c-bac8-4d26-a55d-dbd7762cf4ac","cell_type":"code","source":"df_train.head()","metadata":{},"outputs":[],"execution_count":null},{"id":"b15dee5d-84c1-4a76-8510-4aa76cc0d2dc","cell_type":"markdown","source":"## 🧪 6. Training Preparation\n\nWith all features processed, the next step is to prepare the data for training the ranking model with LightGBM.\n\n---\n\n### 🎯 6.1 Target and Group Definition\n\n- **`target_col`**: target variable indicating whether the flight was selected (`selected = 1`).\n- **`group_col`**: identifies each flight search session (`ranker_id`), used to properly group options in the ranking model.\n\n---\n\n### 🧮 6.2 Feature Organization\n\nFeatures are divided into three types:\n\n- 🔢 **Numerical (`numeric_cols`)**: continuous values like price, duration, baggage count, etc.\n- 🏷️ **Categorical (`categorical_cols`)**: variables representing codes, airports, airlines, etc.\n- ✅ **Boolean (`boolean_cols`)**: indicator variables (e.g., `isVip`, `ida_fds`, `hasFrequentFlyer`, etc.)\n\nThese lists are combined into the final `features` variable, which will be used as input for the model.\n\n---\n\n### 🧪 6.3 Train/Validation Split\n\n`GroupShuffleSplit` is used to perform the **split while respecting groups (`ranker_id`)**, ensuring that all options from the same search session appear **either in training or in validation**, but not both.\n\n---\n\n### 📦 6.4 Dataset Construction for LightGBM\n\nThe `train_dataset` and `val_dataset` objects are created, which are optimized LightGBM structures for ranking:\n\n- Include the data (`X_train`, `X_val`) and targets (`y_train`, `y_val`)\n- Receive the list of categorical columns\n- Incorporate the groups (`group=...`) required for **supervised ranking**\n- Define the `max_bin` parameter, which controls discretization of continuous variables (used to speed up training and allow GPU usage)\n\n> This structure is essential for using the **`lambdarank` objective**, as the model needs to understand the comparison groups.\n","metadata":{}},{"id":"f0e8a0d3-05e2-484d-a5b0-4628207cfe56","cell_type":"code","source":"# --- Target e grupo\ntarget_col = \"selected\"\ngroup_col = \"ranker_id\"\n\n# --- Categóricas para LightGBM\ncategorical_cols = [\n    'companyID',\n    'nationality',\n    'legs0_segments0_departureFrom_airport_iata',\n    'legs0_segments0_arrivalTo_airport_iata',\n    'legs0_segments0_arrivalTo_airport_city_iata',\n    'legs0_segments0_marketingCarrier_code',\n    'legs0_segments0_operatingCarrier_code',\n    'legs0_segments0_aircraft_code',\n    'corporateTariffCode',\n    \n    # novas features categóricas da searchRoute\n    'route_ida',\n    'route_volta',\n    'ida_from',\n    'ida_to',\n    'volta_from',\n    'volta_to'\n]\n\n# --- Booleanas e numéricas\nboolean_cols = [\n    'sex', 'isVip', 'bySelf',\n    'pricingInfo_isAccessTP', 'hasFrequentFlyer',\n    'ida_fds', 'volta_fds',\n    'ida_comercial', 'volta_comercial',\n    'isAccess3D',\n    'has_connections_ida'\n] + [col for col in df_train.columns if col.startswith(\"ff_\")]\n\nnumeric_cols = [\n    'totalPrice', 'taxes',\n    'legs0_duration', 'legs1_duration',\n    'legs0_segments0_baggageAllowance_quantity',\n    'legs0_segments0_baggageAllowance_weightMeasurementType',\n    'legs0_segments0_cabinClass',\n    'legs0_segments0_seatsAvailable',\n    'miniRules0_monetaryAmount', 'miniRules0_percentage',\n    'miniRules1_monetaryAmount', 'miniRules1_percentage',\n    'booking_to_trip_days', 'trip_days',\n    'legs0_dep_hour', 'legs0_dep_dayofweek',\n    'legs1_dep_hour', 'legs1_dep_dayofweek',\n    'frequentFlyer_count', 'legs0_duration_minutes'\n]\nfeatures = numeric_cols + categorical_cols + boolean_cols\n\n# --- Converte categóricas para category\nfor col in categorical_cols:\n    df_train[col] = df_train[col].astype(\"category\")","metadata":{},"outputs":[],"execution_count":null},{"id":"14470131-2fd4-47e1-ac22-ad73229e2f18","cell_type":"code","source":"# --- Separação por grupo (ranker_id)\ngss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)\ntrain_idx, val_idx = next(gss.split(df_train, groups=df_train[\"ranker_id\"]))\n\ndf_train_split = df_train.iloc[train_idx].copy()\ndf_val = df_train.iloc[val_idx].copy()\n\n# --- Features e targets\nX_train = df_train_split[features]\ny_train = df_train_split[target_col]\ngroups_train = df_train_split[group_col].value_counts().sort_index().values\n\nX_val = df_val[features]\ny_val = df_val[target_col]\ngroups_val = df_val[group_col].value_counts().sort_index().values\n\ndataset_params = {\n    \"max_bin\": 63 \n}\n\n# --- Criação dos Datasets\ntrain_dataset = lgb.Dataset(\n    X_train,\n    label=y_train,\n    group=groups_train,\n    categorical_feature=categorical_cols,\n    params=dataset_params  # 💡 AQUI é onde max_bin deve ir também!\n)\n\nval_dataset = lgb.Dataset(\n    X_val,\n    label=y_val,\n    group=groups_val,\n    categorical_feature=categorical_cols,\n    reference=train_dataset,\n    params=dataset_params\n)\n\n","metadata":{},"outputs":[],"execution_count":null},{"id":"4fb6ad76-e353-4fd6-8e15-3e1d192fcc54","cell_type":"markdown","source":"## 🧮 7. Hyperparameter Search with LightGBM\n\nBefore training the final model, we perform a **manual Grid Search** to identify the best combination of hyperparameters for the `Lambdarank` model.\n\n---\n\n### 🔧 7.1 Parameters Tested\n\nThe search is conducted over the following hyperparameters:\n\n- `learning_rate`: learning rate (e.g., 0.05)\n- `num_leaves`: tree complexity (e.g., 63, 127)\n- `min_data_in_leaf`: regularization via minimum samples per leaf (e.g., 50, 70, 100)\n\nAll possible combinations of these values are tested using `itertools.product`.\n\n---\n\n### 🧪 7.2 Training and Validation\n\nFor each parameter combination:\n\n1. The LightGBM model is trained with:\n   - Objective: `lambdarank`\n   - Metric: `ndcg@3`\n   - Early stopping after 50 rounds without improvement\n\n2. The model's performance is evaluated based on the **best NDCG@3** score achieved on the validation set.\n\n3. The best model and parameter set are stored.\n\n---\n\n### ✅ 7.3 Search Result\n\nAt the end of the search:\n\n- The **best parameter combination** is displayed\n- The **NDCG@3 score** is reported\n- A **validation prediction** is performed using the best model\n- The **top-1 accuracy** is calculated, i.e., the fraction of sessions where the correct flight was ranked first by the model\n\n> This evaluation serves as a practical check of the recommendation quality before the final training on the full dataset.","metadata":{}},{"id":"99321909-9f95-4133-8ba6-077aaf287795","cell_type":"code","source":"param_grid = {\n    'learning_rate': [0.05],\n    'num_leaves': [63, 127],        \n    'min_data_in_leaf': [50, 70, 100]     \n}\n# Gera todas combinações de parâmetros\nparam_combinations = list(product(*param_grid.values()))\nparam_keys = list(param_grid.keys())\n\nbest_score = -1\nbest_model = None\nbest_params = None\n\nfor combo in param_combinations:\n    param_set = dict(zip(param_keys, combo))\n    print(f\"Treinando com: {param_set}\")\n\n    params = {\n        \"objective\": \"lambdarank\",\n        \"metric\": \"ndcg\",\n        \"ndcg_eval_at\": [3],\n        \"boosting_type\": \"gbdt\",\n        \"feature_fraction\": 0.8,\n        \"bagging_fraction\": 0.8,\n        \"bagging_freq\": 1,\n        \"seed\": 42,\n        \"verbosity\": -1,\n        \"num_threads\": 8,\n        **param_set\n    }\n\n    model = lgb.train(\n        params,\n        train_dataset,\n        valid_sets=[val_dataset],\n        valid_names=[\"valid\"],\n        num_boost_round=1000,\n        callbacks=[lgb.early_stopping(stopping_rounds=50)],\n    )\n\n    score = model.best_score[\"valid\"][\"ndcg@3\"]\n\n    if score > best_score:\n        best_score = score\n        best_model = model\n        best_params = param_set\n\nprint(\"\\n✅ Melhor combinação:\")\nprint(best_params)\nprint(f\"NDCG@3: {best_score:.5f}\")\n","metadata":{},"outputs":[],"execution_count":null},{"id":"77f7205f-9d0f-4ccb-9257-78c85e977ecf","cell_type":"code","source":"# --- Predição\ny_pred = model.predict(X_val)\n\n# --- Avaliação Top-1\ndf_pred = df_val.copy()\ndf_pred['y_true'] = y_val\ndf_pred['y_pred'] = y_pred\n\ndf_pred_sorted = df_pred.sort_values(['ranker_id', 'y_pred'], ascending=[True, False])\ndf_top1 = df_pred_sorted.groupby('ranker_id').head(1)\n\nacertos = df_top1['y_true'].sum()\ntotal = df_top1.shape[0]\n\nprint(f\"Voos escolhidos corretamente (top1): {acertos} de {total} sessões\")\nprint(f\"Acurácia top1: {acertos / total:.4f}\")","metadata":{},"outputs":[],"execution_count":null},{"id":"9c480004-ad47-49f8-9e32-6108f5afb0a1","cell_type":"markdown","source":"## 🧠 8. Final Training with Validation\n\nWith the best hyperparameters defined, we proceed with the full training of the `LightGBM` model using the `lambdarank` objective.\n\n---\n\n### ⚙️ 8.1 Model Configuration\n\nThe model is configured with:\n\n- `objective = \"lambdarank\"`: supervised ranking model\n- `metric = \"ndcg\"` with `ndcg_eval_at = [3]`: directly optimizes the evaluation metric\n- Hyperparameters:\n  - `\"learning_rate\": best_params['learning_rate']`\n  - `\"num_leaves\": best_params['num_leaves']`\n  - `\"min_data_in_leaf\": best_params['min_data_in_leaf']`\n  - `feature_fraction = 0.8`\n  - `bagging_fraction = 0.8`\n  - `bagging_freq = 1`\n- `early_stopping`: stops training after 120 rounds without improvement\n- Training runs for a maximum of `num_boost_round = 1000`\n\n---\n\n### 📊 8.2 Model Evaluation\n\nAfter training, a prediction is performed on the validation set (`X_val`). The evaluation considers:\n\n- Sorting flight options within each group (`ranker_id`) based on the predicted score (`y_pred`)\n- Computing **top-1 accuracy**, i.e., the fraction of sessions where the correct flight is ranked in the **first position**\n\n> This metric serves as a more direct proxy for assessing the model's practical effectiveness, complementing NDCG@3.\n","metadata":{}},{"id":"2bdb8d85-0bd2-477e-8a25-77b2927d37a3","cell_type":"code","source":"params = {\n    \"objective\": \"lambdarank\",\n    \"metric\": \"ndcg\",\n    \"ndcg_eval_at\": [3],\n    \"learning_rate\": best_params['learning_rate'],\n    \"num_leaves\": best_params['num_leaves'],\n    \"min_data_in_leaf\": best_params['min_data_in_leaf'],\n    \"boosting_type\": \"gbdt\",\n    \"feature_fraction\": 0.8,\n    \"bagging_fraction\": 0.8,\n    \"bagging_freq\": 1,\n    \"seed\": 42,\n    \"verbosity\": -1,\n    \"num_threads\": 8  # ✅ Aqui está correto\n}\n\nmodel = lgb.train(\n    params,\n    train_dataset,\n    valid_sets=[train_dataset, val_dataset],\n    valid_names=[\"train\", \"valid\"],\n    num_boost_round=1000, \n    callbacks=[lgb.early_stopping(stopping_rounds=120)],\n)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"483983be-eaaa-4845-964f-f231cc193b79","cell_type":"code","source":"# --- Predição\ny_pred = model.predict(X_val)\n\n# --- Avaliação Top-1\ndf_pred = df_val.copy()\ndf_pred['y_true'] = y_val\ndf_pred['y_pred'] = y_pred\n\ndf_pred_sorted = df_pred.sort_values(['ranker_id', 'y_pred'], ascending=[True, False])\ndf_top1 = df_pred_sorted.groupby('ranker_id').head(1)\n\nacertos = df_top1['y_true'].sum()\ntotal = df_top1.shape[0]\n\nprint(f\"Voos escolhidos corretamente (top1): {acertos} de {total} sessões\")\nprint(f\"Acurácia top1: {acertos / total:.4f}\")\n","metadata":{},"outputs":[],"execution_count":null},{"id":"fe3a8973-2440-45ed-97db-f10560ae97b1","cell_type":"markdown","source":"## 🏁 9. Final Training with the Entire Dataset\n\nAfter validating the model and finding the best hyperparameter configuration, we perform the **final training using 100% of the training data**.\n\n---\n\n### 📦 9.1 Full Dataset\n\nIn this stage, we use all available observations:\n\n- `X_full`: all features from the `df_train` base\n- `y_full`: target variable (`selected`)\n- `groups_full`: grouping structure (`ranker_id`) with all groups\n\nThese data are converted into a LightGBM `Dataset` optimized for ranking.\n\n---\n\n### 🧠 9.2 Use of `best_iteration`\n\nDuring validation, the model trained with `early_stopping` identified an optimal number of boosting rounds — stored in `model.best_iteration`.\n\nThis value represents the point where the model:\n\n- Achieved **best validation performance**\n- Before starting to **overfit**\n\n⚠️ Therefore, when training on the full dataset, we use:\n\n```python\nnum_boost_round = model.best_iteration\n\n","metadata":{}},{"id":"775a3e1d-0ef4-4b2d-ad67-128a41516edc","cell_type":"code","source":"\n# ✅ Treinamento final com TODO o dataset de treino\n#     usando best_iteration encontrado na validação\n# ============================================================\n\nX_full = df_train[features]\ny_full = df_train[target_col]\ngroups_full = df_train[group_col].value_counts().sort_index().values\n\nfull_dataset = lgb.Dataset(X_full, y_full, group=groups_full, categorical_feature=categorical_cols)\n\n# ⚠️ Usa o número ideal de iterações do treino anterior\nfinal_model = lgb.train(\n    params,\n    full_dataset,\n    num_boost_round=model.best_iteration \n)","metadata":{},"outputs":[],"execution_count":null},{"id":"4b2d12e5-12c3-49c1-a181-2dc94f901ce5","cell_type":"code","source":"final_model.save_model('modelo_final.txt')","metadata":{},"outputs":[],"execution_count":null},{"id":"910a1fa2-ae5e-4af8-b663-143be030d205","cell_type":"markdown","source":"## 📤 10. Submission Generation\n\n- Load the trained model from `modelo_final.txt`.\n- Apply the same preprocessing steps used during training.\n- Perform predictions with the final model and sort by `y_pred`.\n- Generate the `submission.csv` file with the ranked flight selections (`selected`).","metadata":{}},{"id":"872670e0-ef39-4aad-8be6-d9989242ced0","cell_type":"code","source":"model = lgb.Booster(model_file='modelo_final.txt')","metadata":{},"outputs":[],"execution_count":null},{"id":"42276c5b-db73-42bd-8b6e-440e29cb850e","cell_type":"code","source":"# ============================================================\n# ## 6. Geração de Submissão\n# ============================================================\n\n# 1. Ler test.parquet\ndf_test = pd.read_parquet(\"data/aeroclub/test.parquet\")\n\n# 2. Aplicar transformações mínimas necessárias\ndf_test['ranker_id'] = df_test['ranker_id'].astype(str)\ndf_test['nationality'] = df_test['nationality'].astype(str)\ndf_test['searchRoute'] = df_test['searchRoute'].astype(str)\n\n# --- Frequent Flyer (mesmos one-hot do treino)\ndf_test['frequentFlyer'] = df_test['frequentFlyer'].fillna('').astype(str)\nff_lists_test = df_test['frequentFlyer'].str.split('/')","metadata":{},"outputs":[],"execution_count":null},{"id":"2b09d897-bf48-475b-ad77-ed5374fcd168","cell_type":"code","source":"all_programs = set(chain.from_iterable(ff_lists))\nprint(f\"Total de companhias únicas: {len(all_programs)}\")\n\n\nfor program in all_programs:\n    if program == '':\n        continue\n    df_test[f'ff_{program}'] = ff_lists_test.apply(lambda x: int(program in x))\n\nfor col in [col for col in df_test.columns if col.startswith(\"ff_\")]:\n    df_test[col] = df_test[col].astype(pd.BooleanDtype())\n\ndf_test['frequentFlyer_count'] = df_test['frequentFlyer'].apply(count_frequent_flyers)\ndf_test['hasFrequentFlyer'] = df_test['frequentFlyer'].notnull().astype(int)\ndf_test.drop(columns=['frequentFlyer'], inplace=True)\n\n","metadata":{},"outputs":[],"execution_count":null},{"id":"659ef606-27d3-4e6a-b937-f24e1cf8d84a","cell_type":"code","source":"# --- Datas\ncols_datetime = [\n    'requestDate',\n    'legs0_departureAt', 'legs0_arrivalAt',\n    'legs1_departureAt', 'legs1_arrivalAt'\n]\nfor col in cols_datetime:\n    df_test[col] = pd.to_datetime(df_test[col], errors='coerce')\n\ndf_test['legs0_dep_hour'] = df_test['legs0_departureAt'].dt.hour\ndf_test['legs0_dep_dayofweek'] = df_test['legs0_departureAt'].dt.dayofweek\ndf_test['legs1_dep_hour'] = df_test['legs1_departureAt'].dt.hour\ndf_test['legs1_dep_dayofweek'] = df_test['legs1_departureAt'].dt.dayofweek\ndf_test['trip_days'] = (df_test['legs1_departureAt'] - df_test['legs0_departureAt']).dt.days\ndf_test['booking_to_trip_days'] = (df_test['legs0_departureAt'] - df_test['requestDate']).dt.days\ndf_test['ida_fds'] = df_test['legs0_dep_dayofweek'].isin([5, 6]).astype(int)\ndf_test['volta_fds'] = df_test['legs1_dep_dayofweek'].isin([5, 6]).astype(int)\n\ndf_test['ida_comercial'] = df_test['legs0_dep_hour'].apply(lambda x: int(7 <= x <= 19))\ndf_test['volta_comercial'] = df_test['legs1_dep_hour'].apply(lambda x: int(7 <= x <= 19))\n\ndf_test.drop(columns=cols_datetime, inplace=True)","metadata":{},"outputs":[],"execution_count":null},{"id":"786f71db-80e7-4497-a07d-3d6a5185f00f","cell_type":"code","source":"######################################\n# Cria a feature 'n_segments_ida' baseada na presença de segmentos 1, 2, 3\nsegments_ida_cols = {\n    1: 'legs0_segments1_departureFrom_airport_iata',\n    2: 'legs0_segments2_departureFrom_airport_iata',\n    3: 'legs0_segments3_departureFrom_airport_iata'\n}\n\n# Começa com 1 porque o segmento 0 está sempre presente\ndf_test['n_segments_ida'] = 1\n\nfor seg, col in segments_ida_cols.items():\n    df_test['n_segments_ida'] += df_test[col].notnull().astype(int)\n\n# Cria a flag booleana se há conexões (mais de 1 segmento na ida)\ndf_test['has_connections_ida'] = (df_test['n_segments_ida'] > 1).astype('boolean')\n\n# companyID como categoria\ndf_test['companyID'] = df_test['companyID'].astype('category')\n\n# isAccess3D como boolean\ndf_test['isAccess3D'] = df_test['isAccess3D'].astype('boolean')\n\ndf_test.drop('legs0_segments1_departureFrom_airport_iata', inplace=True, axis=1)\ndf_test.drop('legs0_segments2_departureFrom_airport_iata', inplace=True, axis=1)\ndf_test.drop('legs0_segments3_departureFrom_airport_iata', inplace=True, axis=1)\n########################\n\n\n# --- Duração\ndef clean_and_convert_duration(col):\n    return (\n        col\n        .fillna(\"00:00:00\")\n        .astype(str)\n        .str.strip()\n        .str.replace(\"nan\", \"00:00:00\")\n        .pipe(pd.to_timedelta, errors='coerce')\n        .dt.total_seconds() / 60\n    )\n\ndf_test['legs0_duration'] = clean_and_convert_duration(df_test['legs0_duration'])\ndf_test['legs1_duration'] = clean_and_convert_duration(df_test['legs1_duration'])\ndf_test['legs0_segments0_duration'] = clean_and_convert_duration(df_test['legs0_segments0_duration'])\ndf_test['legs0_duration_minutes'] = df_test['legs0_duration']\ndf_test.drop(columns=['legs0_segments0_duration'], inplace=True)\n\n# --- SearchRoute features\ndf_test[['route_ida', 'route_volta']] = df_test['searchRoute'].str.split('/', expand=True)\ndf_test['ida_from'] = df_test['route_ida'].str[:3]\ndf_test['ida_to'] = df_test['route_ida'].str[3:]\ndf_test['volta_from'] = df_test['route_volta'].str[:3]\ndf_test['volta_to'] = df_test['route_volta'].str[3:]\ndf_test.drop('searchRoute', axis=1, inplace=True)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"7141f052-110b-4740-b430-4c8eb1651203","cell_type":"code","source":"df_train['companyID'] = df_train['companyID'].astype('category')\n# Converte para booleano (caso ainda não esteja)\ndf_train['isAccess3D'] = df_train['isAccess3D'].astype('boolean')\n# Cria flag indicando se há conexões na ida\ndf_train['has_connections_ida'] = (df_train['n_segments_ida'] > 1).astype('boolean')\n\n# --- Tipagem\nfor col in categorical_cols:\n    df_test[col] = df_test[col].astype(\"category\")\n\nfor col in boolean_cols:\n    if col in df_test.columns:\n        df_test[col] = df_test[col].astype('boolean')","metadata":{},"outputs":[],"execution_count":null},{"id":"a02987c5-818f-4ba9-91e8-070d6c4846cc","cell_type":"code","source":"# Prever com o modelo\nX_test = df_test[features]\ndf_test['y_pred'] = model.predict(X_test)\n\n# 4. Gerar submissão\ndf_test_sorted = df_test.sort_values(['ranker_id', 'y_pred'], ascending=[True, False])\ndf_test_sorted['selected'] = df_test_sorted.groupby('ranker_id').cumcount() + 1\n\nsubmission = df_test_sorted[['Id', 'ranker_id', 'selected']]\nsubmission.to_csv(\"submission.csv\", index=False)\nprint(\"✅ Arquivo de submissão salvo como 'submission.csv'\")\n","metadata":{},"outputs":[],"execution_count":null}]}