{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84493,"databundleVersionId":9871156,"sourceType":"competition"}],"dockerImageVersionId":30787,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Outline\n\nAt the beginning, this note discussed the possibility that the **integer data** part is **categorical data**, but there is no need to do any additional processing here.\n\nThe main work of this note is as follows:\n\n> **1. Memory management: Convert <span style=\"color:red\">float32</span> to <span style=\"color:red\">float16</span>**\n> > To ensure the feasibility of the method, the **standard deviation** is used to measure whether the significance of the data can be maintained after reducing the precision.\n\n> > Cut down the data for the **first 247 date** because there are too many missing values  <span style=\"color:green\">-->datafile\"lower\"</span>\n\n> **2. Missing value processing: <span style=\"color:red\">Systematic Missing Values</span> and <span style=\"color:red\">Random Missing Values</span> are distinguished**\n\n> > <span style=\"color:red\">Systematic missing values:</span> at each date, data of some feature is generated from a specific time_id rather than from the beginning.\n\n> > > For these missing values we <span style=\"color:blue\">fill them with \"0\"</span>  <span style=\"color:green\">-->datafile\"deal_sys_miss\"</span>\n\n> ><span style=\"color:red\">Random missing values:</span> Normally all date_id and time_id should be generated, but there is missing data.\n\n> > > There are two categories:\n> > > > **1. The feature is 100% missing on a specific date**\n\n> > > > > * Absence of some feature in a continuous and long duration date period\n\n> > > > > > For such missing values, we <span style=\"color:blue\">fill them with \"1\"</span> (in order to avoid a large amount of data loss, we cannot directly delete the samples under these dates).\n\n> > > > > * Absence of some feature in a discrete and small number of dates\n\n> > > > > > For these missing values, we can <span style=\"color:blue\">simply delete</span> (a small number of samples).\n\n> > > > **2. The feature is partial missing on a specific date**\n\n> > > > > For such missing values, we chose a combination of <span style=\"color:blue\">forward and backward filling</span> (since it is time series data, this method is more suitable than using statistical characteristics such as mean).\n\n> > > > > > There is an interesting error **\"No matching signature found\"** : we need to convert the data of float16 back to float32 to fill, and then convert to float16 to reduce memory. <span style=\"color:green\">-->datafile\"deal_random_miss\"</span>\n\nFor the processing of key steps, we retained data sets. In the next step, we verified the effectiveness of this note work by running these three data sets against the same baseline to compare the scores of cv and lb.","metadata":{}},{"cell_type":"code","source":"import os\nfrom pathlib import Path\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm\nimport pyarrow.parquet as pq\nimport shutil\nimport time\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:28:44.618301Z","iopub.execute_input":"2024-12-11T07:28:44.618726Z","iopub.status.idle":"2024-12-11T07:28:45.907811Z","shell.execute_reply.started":"2024-12-11T07:28:44.618675Z","shell.execute_reply":"2024-12-11T07:28:45.906613Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data Type Overview\n\nHere we spot check the two files with ***partition_id=0 and 9*** to observe:","metadata":{}},{"cell_type":"code","source":"part_0_path = \"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=0/part-0.parquet\"\npart_9_path = \"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=9/part-0.parquet\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:28:45.910008Z","iopub.execute_input":"2024-12-11T07:28:45.911254Z","iopub.status.idle":"2024-12-11T07:28:45.916804Z","shell.execute_reply.started":"2024-12-11T07:28:45.911154Z","shell.execute_reply":"2024-12-11T07:28:45.915253Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"part_0 = pd.read_parquet(part_0_path)\npart_9 = pd.read_parquet(part_9_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:28:45.918159Z","iopub.execute_input":"2024-12-11T07:28:45.918577Z","iopub.status.idle":"2024-12-11T07:29:00.965781Z","shell.execute_reply.started":"2024-12-11T07:28:45.918532Z","shell.execute_reply":"2024-12-11T07:29:00.964569Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def analyze_and_plot(df):\n    selected_columns = df[['feature_09', 'feature_10', 'feature_11']]\n    \n    for col in selected_columns.columns:\n        \n        print(f\"Unique values count for {col}: {selected_columns[col].nunique()}\")\n        \n        print(f\"Max value for {col}: {selected_columns[col].max()}\")\n        print(f\"Min value for {col}: {selected_columns[col].min()}\")\n        \n        value_counts = selected_columns[col].value_counts(normalize=True)\n        print(f\"Frequency of unique values for {col}:\")\n        #print(value_counts)\n        \n        value_counts.plot(kind='bar', figsize=(5, 3), alpha=0.7)\n        plt.title(f\"Frequency of unique values in {col}\")\n        plt.xlabel('Unique Value')\n        plt.ylabel('Frequency')\n        plt.show()\n        \nanalyze_and_plot(part_0)\nanalyze_and_plot(part_9)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:29:00.969266Z","iopub.execute_input":"2024-12-11T07:29:00.969773Z","iopub.status.idle":"2024-12-11T07:29:02.988789Z","shell.execute_reply.started":"2024-12-11T07:29:00.969721Z","shell.execute_reply":"2024-12-11T07:29:02.987560Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Here ***feature_10*** may be **classified data**, because Unique values count and Max value are relatively small, such as financial indicator: Credit Rating Code. And the other two characteristics may be numeric meaning of integer financial indicators, such as: Volume, Number of Trades, Shares Outstanding.","metadata":{}},{"cell_type":"code","source":"def analyze_partition_data(base_folder):\n    \n    for partition_id in range(10):\n        partition_folder = os.path.join(base_folder, f'partition_id={partition_id}')\n        \n        part_file = os.path.join(partition_folder, 'part-0.parquet')\n        if os.path.exists(part_file):\n            print(f\"Processing partition_id={partition_id}\")\n            # read\n            df = pd.read_parquet(part_file)\n\n            # Groupby date_id\n            grouped = df.groupby('date_id')\n\n            # date number\n            num_days = grouped.size().shape[0]\n            print(f\"Number of days: {num_days}\")\n\n            # Calculate the number of time_id for each date_id (count of time_id per date_id)\n            time_id_counts = grouped['time_id'].count()\n\n            # Calculate the max, min, and mean of the time_id counts\n            time_id_stats = time_id_counts.agg(['max', 'min', 'mean'])\n\n            # Print the statistics\n            print(f\"Time_id count stats (max, min, mean):\")\n            print(time_id_stats)\n            print(\"\\n\" + \"-\"*50)\n\n# Run the function\nbase_folder = '/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet'\nanalyze_partition_data(base_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:29:02.990168Z","iopub.execute_input":"2024-12-11T07:29:02.990571Z","iopub.status.idle":"2024-12-11T07:30:17.213281Z","shell.execute_reply.started":"2024-12-11T07:29:02.990535Z","shell.execute_reply":"2024-12-11T07:30:17.212159Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Memory Management\nIt is indeed a more logical move to decide whether to convert a column to **float16** based on the standard deviation, which reflects the volatility and precision requirements of the data. For columns with a small standard deviation, converting to a lower precision float8 (or float16) may not affect the data much, while for columns with a large standard deviation, keeping a higher precision (such as float32) will help reduce the loss of data.\n\n* Smaller standard deviation (e.g. std < 0.0001) : For data of this standard deviation, it means that the data changes very little and there is almost no fluctuation. This kind of data will lose more information if converted to float8, so it is appropriate to keep float32.\n* Larger standard deviation (e.g. 0.0001 <= std) : These columns may be relatively stable, but still have some volatility. float16 conversions in this range may effectively reduce storage footprint while maintaining adequate precision.","metadata":{}},{"cell_type":"code","source":"# List of columns to convert\ncolumns_to_convert = [\n    'feature_00', 'feature_01', 'feature_02', 'feature_03', 'feature_04', 'feature_05', 'feature_06',\n    'feature_07', 'feature_08', 'feature_12', 'feature_13', 'feature_14', 'feature_15', 'feature_16', 'feature_17',\n    'feature_18', 'feature_19', 'feature_20', 'feature_21', 'feature_22', 'feature_23', 'feature_24', 'feature_25',\n    'feature_26', 'feature_27', 'feature_28', 'feature_29', 'feature_30', 'feature_31', 'feature_32', 'feature_33',\n    'feature_34', 'feature_35', 'feature_36', 'feature_37', 'feature_38', 'feature_39', 'feature_40', 'feature_41',\n    'feature_42', 'feature_43', 'feature_44', 'feature_45', 'feature_46', 'feature_47', 'feature_48', 'feature_49',\n    'feature_50', 'feature_51', 'feature_52', 'feature_53', 'feature_54', 'feature_55', 'feature_56', 'feature_57',\n    'feature_58', 'feature_59', 'feature_60', 'feature_61', 'feature_62', 'feature_63', 'feature_64', 'feature_65',\n    'feature_66', 'feature_67', 'feature_68', 'feature_69', 'feature_70', 'feature_71', 'feature_72', 'feature_73',\n    'feature_74', 'feature_75', 'feature_76', 'feature_77', 'feature_78', 'responder_0', 'responder_1', 'responder_2',\n    'responder_3', 'responder_4', 'responder_5', 'responder_7', 'responder_8'\n]  # 'weight', responder_6: Because it is the final target, the highest precision is retained...\n\n# Directories\ninput_directory = '/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet'\n\n# Set a single threshold for classification\nstd_threshold = 0.01\n\n# List of partition directories (partition_id=0 to partition_id=9)\npartition_dirs = [f\"partition_id={i}\" for i in range(10)]\n\nfor partition_dir in partition_dirs:\n    partition_path = os.path.join(input_directory, partition_dir, \"part-0.parquet\")\n    \n    # Check if partition exists\n    if os.path.exists(partition_path):\n        df = pd.read_parquet(partition_path)\n        \n        # Memory usage before conversion\n        #memory_before = df.memory_usage(deep=True).sum() / (1024 ** 2)  # Convert to MB\n        #print(f\"Partition {partition_dir} - Before Conversion: {memory_before:.2f} MB\")\n\n        # Initialize lists to store column names for each precision level\n        columns_to_convert32 = []\n        columns_to_convert16 = []\n\n        # Classify columns based on standard deviation\n        for col in columns_to_convert:\n            if col in df.columns:\n                std_dev = df[col].std()  # Calculate standard deviation\n                \n                if std_dev < std_threshold:\n                    columns_to_convert32.append(col)  # Retain as float32\n                else:\n                    columns_to_convert16.append(col)  # Convert to float16\n\n        # Print out the classified columns\n        print(f\"Partition {partition_dir} - Columns to keep as float32:\")\n        print(columns_to_convert32)\n\n        print(f\"Partition {partition_dir} - Columns to convert to float16:\")\n        #print(columns_to_convert16)\n        print(\"All remaining columns\")\n        print(\"-\" * 50)\n    else:\n        print(f\"Partition {partition_dir} not found.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:30:17.214993Z","iopub.execute_input":"2024-12-11T07:30:17.215368Z","iopub.status.idle":"2024-12-11T07:31:14.537986Z","shell.execute_reply.started":"2024-12-11T07:30:17.215334Z","shell.execute_reply":"2024-12-11T07:31:14.536761Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def convert_float32_to_float16(input_dir, output_dir, columns_to_convert):\n    partition_dirs = [f\"partition_id={i}\" for i in range(10)]  # partition_id=0 to partition_id=9\n\n    os.makedirs(output_dir, exist_ok=True)\n\n    for partition_dir in partition_dirs:\n        partition_path = os.path.join(input_dir, partition_dir, \"part-0.parquet\")\n        \n        df = pd.read_parquet(partition_path)\n        \n        memory_before = df.memory_usage(deep=True).sum() / (1024 ** 2)  # MB\n        print(f\"Partition {partition_dir} - Before Conversion: {memory_before:.2f} MB\")\n\n        for col in columns_to_convert:\n            if col in df.columns:\n                df[col] = df[col].astype(np.float16) \n        \n        memory_after = df.memory_usage(deep=True).sum() / (1024 ** 2)  # MB\n        print(f\"Partition {partition_dir} - After Conversion: {memory_after:.2f} MB\")\n        \n        output_file = os.path.join(output_dir, f\"{partition_dir}.parquet\")\n        df.to_parquet(output_file, index=False)\n        \n        tqdm.write(f\"Completed conversion for {partition_dir}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:31:14.540289Z","iopub.execute_input":"2024-12-11T07:31:14.540607Z","iopub.status.idle":"2024-12-11T07:31:14.548600Z","shell.execute_reply.started":"2024-12-11T07:31:14.540576Z","shell.execute_reply":"2024-12-11T07:31:14.547510Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n\ncolumns_to_convert = [\n    'feature_00', 'feature_01', 'feature_02', 'feature_03', 'feature_04', 'feature_05', 'feature_06',\n    'feature_07', 'feature_08', 'feature_12', 'feature_13', 'feature_14', 'feature_15', 'feature_16', 'feature_17',\n    'feature_18', 'feature_19', 'feature_20', 'feature_21', 'feature_22', 'feature_23', 'feature_24', 'feature_25',\n    'feature_26', 'feature_27', 'feature_28', 'feature_29', 'feature_30', 'feature_31', 'feature_32', 'feature_33',\n    'feature_34', 'feature_35', 'feature_36', 'feature_37', 'feature_38', 'feature_39', 'feature_40', 'feature_41',\n    'feature_42', 'feature_43', 'feature_44', 'feature_45', 'feature_46', 'feature_47', 'feature_48', 'feature_49',\n    'feature_50', 'feature_51', 'feature_52', 'feature_53', 'feature_54', 'feature_55', 'feature_56', 'feature_57',\n    'feature_58', 'feature_59', 'feature_60', 'feature_61', 'feature_62', 'feature_63', 'feature_64', 'feature_65',\n    'feature_66', 'feature_67', 'feature_68', 'feature_69', 'feature_70', 'feature_71', 'feature_72', 'feature_73',\n    'feature_74', 'feature_75', 'feature_76', 'feature_77', 'feature_78', 'responder_0', 'responder_1', 'responder_2',\n    'responder_3', 'responder_4', 'responder_5', 'responder_7', 'responder_8'\n] # 'weight', 'responder_6'\n\n\ninput_directory = '/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet'\noutput_directory = '/kaggle/working/lower'\n\n\nconvert_float32_to_float16(input_directory, output_directory, columns_to_convert)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:31:14.549916Z","iopub.execute_input":"2024-12-11T07:31:14.550275Z","iopub.status.idle":"2024-12-11T07:37:03.780609Z","shell.execute_reply.started":"2024-12-11T07:31:14.550241Z","shell.execute_reply":"2024-12-11T07:37:03.779295Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def analyze_partition_data_out(base_folder):\n    \n    for partition_id in range(10):\n        part_file = os.path.join(base_folder, f'partition_id={partition_id}.parquet')\n        if os.path.exists(part_file):\n            print(f\"Processing partition_id={partition_id}\")\n            # read\n            df = pd.read_parquet(part_file)\n\n            # Groupby date_id\n            grouped = df.groupby('date_id')\n\n            # date number\n            num_days = grouped.size().shape[0]\n            print(f\"Number of days: {num_days}\")\n\n            # Calculate the number of time_id for each date_id (count of time_id per date_id)\n            time_id_counts = grouped['time_id'].count()\n\n            # Calculate the max, min, and mean of the time_id counts\n            time_id_stats = time_id_counts.agg(['max', 'min', 'mean'])\n\n            # Print the statistics\n            print(f\"Time_id count stats (max, min, mean):\")\n            print(time_id_stats)\n            print(\"\\n\" + \"-\"*50)\n\n# Run the function\nbase_folder = '/kaggle/working/lower'\nanalyze_partition_data_out(base_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:37:03.782131Z","iopub.execute_input":"2024-12-11T07:37:03.782550Z","iopub.status.idle":"2024-12-11T07:38:01.644689Z","shell.execute_reply.started":"2024-12-11T07:37:03.782513Z","shell.execute_reply":"2024-12-11T07:38:01.643538Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Missing Value Analysis\n\n## Groupby(date_id)","metadata":{}},{"cell_type":"code","source":"def calculate_missing_values_by_date(folder_path):\n    for filename in os.listdir(folder_path):\n        if filename.endswith('.parquet'):\n            file_path = os.path.join(folder_path, filename)\n            \n            df = pd.read_parquet(file_path)\n            \n            missing_ratio_by_date = df.groupby('date_id').apply(lambda x: x.isnull().sum().sum() / x.size)\n            \n            print(f\"Missing values in {filename} (overall ratio by date_id):\")\n            print(missing_ratio_by_date)\n            print(\"-\" * 50)\n\ncalculate_missing_values_by_date(output_directory)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:38:01.648392Z","iopub.execute_input":"2024-12-11T07:38:01.648765Z","iopub.status.idle":"2024-12-11T07:39:18.447967Z","shell.execute_reply.started":"2024-12-11T07:38:01.648730Z","shell.execute_reply":"2024-12-11T07:39:18.446795Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def print_date_time_ids(output_dir):\n    \n    partition_dirs = [f\"partition_id={i}.parquet\" for i in range(10)]  # partition_id=0 to partition_id=9\n    \n    for partition_file in partition_dirs:\n        file_path = os.path.join(output_dir, partition_file)\n        \n        df = pd.read_parquet(file_path)\n        \n        first_row = df.iloc[0]\n        last_row = df.iloc[-1]\n        \n        print(f\"File: {partition_file}\")\n        print(f\"  First row - date_id: {first_row['date_id']}, time_id: {first_row['time_id']}\")\n        print(f\"  Last row  - date_id: {last_row['date_id']}, time_id: {last_row['time_id']}\")\n        print(\"-\" * 50)\n\noutput_directory = '/kaggle/working/lower'\n\nprint_date_time_ids(output_directory)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:39:18.449594Z","iopub.execute_input":"2024-12-11T07:39:18.450045Z","iopub.status.idle":"2024-12-11T07:39:47.883709Z","shell.execute_reply.started":"2024-12-11T07:39:18.449994Z","shell.execute_reply":"2024-12-11T07:39:47.882616Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can observe that ***partition_id=0 and 1*** May have the longest time distance, so the proportion of missing values is the highest, more than **10%** almost every day. In ***partition_id= 1,2 and 3***, the proportion of missing values is about 5%. It gradually falls below 1% and then stabilizes at 0.4%. Therefore, when training the model, we can delete early data with missing values accounting for > 10%.\n\nIn addition, there are missing values in almost every day, so in the next step we need to look at the distribution of missing values in different time periods of each day. in different time periods of each day.","metadata":{}},{"cell_type":"code","source":"def find_dates_with_low_missing_values(file_path, threshold=0.1):\n    \n    df = pd.read_parquet(file_path)\n    \n    missing_ratio_by_date = df.groupby('date_id').apply(lambda x: x.isnull().sum().sum() / x.size)\n    \n    low_missing_dates = missing_ratio_by_date[missing_ratio_by_date < threshold]\n    \n    print(f\"Date IDs with missing values less than {threshold*100}% in {file_path}:\")\n    print(low_missing_dates)\n\nfile_path = '/kaggle/working/lower/partition_id=1.parquet'\n\nfind_dates_with_low_missing_values(file_path, threshold=0.1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:39:47.885233Z","iopub.execute_input":"2024-12-11T07:39:47.885596Z","iopub.status.idle":"2024-12-11T07:39:52.396900Z","shell.execute_reply.started":"2024-12-11T07:39:47.885562Z","shell.execute_reply":"2024-12-11T07:39:52.395696Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"<span style=\"color:red\">So for training, we don't want data from the first 247 days.</span>","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Delete the partition_id= 0_convert. parquet file\nfile_to_delete = '/kaggle/working/lower/partition_id=0.parquet'\nif os.path.exists(file_to_delete):\n    os.remove(file_to_delete)\n    print(f\"File {file_to_delete} has been deleted.\")\nelse:\n    print(f\"File {file_to_delete} does not exist.\")\n\n\n# Update the partition_id= 1_render.parquet file, keeping only data with date_id > 247\nfile_to_update = '/kaggle/working/lower/partition_id=1.parquet'\n\ndf = pd.read_parquet(file_to_update)\ndf_filtered = df[df['date_id'] > 247]\ndf_filtered.to_parquet(file_to_update, index=False)\n\nprint(f\"File {file_to_update} has been updated to keep only rows with date_id > 247.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:39:52.398627Z","iopub.execute_input":"2024-12-11T07:39:52.398997Z","iopub.status.idle":"2024-12-11T07:40:04.460435Z","shell.execute_reply.started":"2024-12-11T07:39:52.398959Z","shell.execute_reply":"2024-12-11T07:40:04.459273Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lower_folder = '/kaggle/working/lower'\nanalyze_partition_data_out(lower_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:41:00.088373Z","iopub.execute_input":"2024-12-11T07:41:00.088819Z","iopub.status.idle":"2024-12-11T07:41:28.544632Z","shell.execute_reply.started":"2024-12-11T07:41:00.088779Z","shell.execute_reply":"2024-12-11T07:41:28.543476Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # Directory to save updated files\n# output_directory = '/kaggle/working/deal_sys_miss'\n# os.makedirs(output_directory, exist_ok=True)\n\n# # Update the partition_id=1_converted.parquet file, keeping only data with date_id > 247\n# file_to_update = '/kaggle/working/lower/partition_id=1_converted.parquet'\n# new_file_path = os.path.join(output_directory, 'partition_id=1_converted.parquet')\n\n# df = pd.read_parquet(file_to_update)\n# df_filtered = df[df['date_id'] > 247]\n# df_filtered.to_parquet(new_file_path, index=False)\n\n# print(f\"Filtered data has been saved to {new_file_path}.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:40:04.777556Z","iopub.status.idle":"2024-12-11T07:40:04.777976Z","shell.execute_reply.started":"2024-12-11T07:40:04.777780Z","shell.execute_reply":"2024-12-11T07:40:04.777801Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Groupby(time_id)","metadata":{}},{"cell_type":"code","source":"def calculate_missing_ratio_by_time_group(folder_path, group_size=60):\n    \n    for filename in os.listdir(folder_path):\n        if filename.endswith('.parquet'):\n            file_path = os.path.join(folder_path, filename)\n            \n            df = pd.read_parquet(file_path)\n            \n            df_sorted = df.sort_values(by='time_id')\n            \n            total_data = df_sorted.size\n            \n            print(f\"Missing values in {filename}:\")\n            \n            for start_time_id in range(0, df_sorted['time_id'].max() + 1, group_size):\n                end_time_id = start_time_id + group_size - 1\n                group = df_sorted[(df_sorted['time_id'] >= start_time_id) & (df_sorted['time_id'] <= end_time_id)]\n                \n                missing_count = group.isnull().sum().sum() \n                \n                missing_ratio = missing_count / total_data * 100\n\n                print(f\"{missing_ratio:.2f}% of missing values are under time_id {start_time_id}-{end_time_id}\")\n            \n            print(\"-\" * 50)\n\ncalculate_missing_ratio_by_time_group(lower_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:41:28.546524Z","iopub.execute_input":"2024-12-11T07:41:28.546861Z","iopub.status.idle":"2024-12-11T07:43:00.557953Z","shell.execute_reply.started":"2024-12-11T07:41:28.546827Z","shell.execute_reply":"2024-12-11T07:43:00.556741Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"For ***partition_id=1 and 2***, we can observe that the proportion of missing values in the interval of **time_id 0-59** is as high as **85%-90%**, and the proportion of missing values in each subsequent set of times is also more than 30%.\n\nFor ***partition_id=3***, the proportion of missing values in the interval of **time_id 0-59** is **56%**, after that it is lower.\n\nFor ***partition_id=4 to 9***, we can observe that the main missing values of each day are distributed in the **time_id 0-59** interval, and the missing values account for 44%.\n\nSubsequently, we should observe whether there are feature columns with high frequency in the missing values of these parts.","metadata":{}},{"cell_type":"code","source":"def print_missing_values_for_time_group(folder_path, time_id_start=0, time_id_end=59, partition_range=(4, 9)):\n    \n    for filename in os.listdir(folder_path):\n        if filename.endswith('.parquet'):\n            partition_id = int(filename.split('=')[1].split('.')[0])  # exact partition_id\n            if partition_range[0] <= partition_id <= partition_range[1]:\n                file_path = os.path.join(folder_path, filename)\n                \n                df = pd.read_parquet(file_path)                \n                \n                df_filtered = df[(df['time_id'] >= time_id_start) & (df['time_id'] <= time_id_end)]                \n                \n                missing_data = df_filtered.isnull().sum()\n                \n                missing_columns = missing_data[missing_data > 0] \n                \n                if not missing_columns.empty:\n                    print(f\"Missing values in partition_id={partition_id} (time_id {time_id_start}-{time_id_end}):\")\n                    total_rows = len(df_filtered)\n                    \n\n                    missing_columns_sorted = missing_columns.sort_values(ascending=False)\n                    \n                    for column, missing_count in missing_columns_sorted.items():\n                        missing_ratio = missing_count / total_rows * 100  \n                        print(f\"Column: {column}, Missing count: {missing_count}, Missing ratio: {missing_ratio:.2f}%\")\n                    print(\"-\" * 50)\n\nprint_missing_values_for_time_group(lower_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:43:00.559710Z","iopub.execute_input":"2024-12-11T07:43:00.560223Z","iopub.status.idle":"2024-12-11T07:43:44.769606Z","shell.execute_reply.started":"2024-12-11T07:43:00.560146Z","shell.execute_reply":"2024-12-11T07:43:44.768458Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see that ***feature_39,42,53,50*** are **100%** missing at **time_id 0-59**, which means that this part of the data should not have been generated earlier in the day, so we should not treat this part of the data as Nan. Instead, special values such as 0 or -1 should be filled in to facilitate model learning.\n\nSimilarly, the proportion of missing values for ***feature_15*** is always 40%, that for ***feature_41, 44,52,55*** is always 30%, and that for ***feature_73,74,32,33,58,17*** has a similar possibility because the proportion of missing values is similar in each partition.\n\n\nNext we need to find out what time_id these features are generated from each day to verify the conjecture.","metadata":{}},{"cell_type":"markdown","source":"## Systematic Missing Value\n\nSystematic Missing Value are data columns in a dataset where missing values occur consistently and predictably under specific conditions, such as during certain time periods or operational scenarios. These absences are not the result of data errors or quality issues but are inherent to the data generation process. As such, these features should not be treated as traditional missing values (NaN). Instead, they can be replaced with special placeholder values (e.g., 0 or -1) to facilitate model training and interpretation.","metadata":{}},{"cell_type":"code","source":"def get_first_non_missing_time_id(folder_path, features, output_file=None):\n    \n    all_results = []\n\n    for filename in os.listdir(folder_path):\n        if filename.endswith('.parquet'):\n            file_path = os.path.join(folder_path, filename)\n            print(f\"Processing file: {filename}\")\n\n            df = pd.read_parquet(file_path)\n\n            file_results = []\n\n            grouped = df.groupby('date_id')\n\n            for date_id, group in grouped:\n \n                result_row = {'date_id': date_id}\n\n                for feature in features:\n                    # Gets the time_id of the first non-missing value of the current feature\n                    non_missing_time_ids = group.loc[group[feature].notnull(), 'time_id']\n                    result_row[feature] = non_missing_time_ids.iloc[0] if not non_missing_time_ids.empty else None\n\n                file_results.append(result_row)\n\n            all_results.extend(file_results)\n\n    final_df = pd.DataFrame(all_results)\n    final_df = final_df.sort_values(by='date_id').reset_index(drop=True)\n\n    if output_file:\n        final_df.to_csv(output_file, index=False)\n        print(f\"Results saved to {output_file}\")\n\n    return final_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:43:44.772147Z","iopub.execute_input":"2024-12-11T07:43:44.773357Z","iopub.status.idle":"2024-12-11T07:43:44.781457Z","shell.execute_reply.started":"2024-12-11T07:43:44.773310Z","shell.execute_reply":"2024-12-11T07:43:44.780289Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"selected_features = ['feature_15', 'feature_17', 'feature_32', 'feature_33', 'feature_39', \n                     'feature_41', 'feature_42', 'feature_44', 'feature_50', \n                     'feature_52', 'feature_53', 'feature_55', 'feature_58', \n                     'feature_73', 'feature_74']\n\nresult_df = get_first_non_missing_time_id(output_directory, selected_features)\n\nresult_df.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:43:44.782791Z","iopub.execute_input":"2024-12-11T07:43:44.783225Z","iopub.status.idle":"2024-12-11T07:44:55.728471Z","shell.execute_reply.started":"2024-12-11T07:43:44.783157Z","shell.execute_reply":"2024-12-11T07:44:55.727296Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"result_df.nunique()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:44:55.729879Z","iopub.execute_input":"2024-12-11T07:44:55.730261Z","iopub.status.idle":"2024-12-11T07:44:55.746885Z","shell.execute_reply.started":"2024-12-11T07:44:55.730165Z","shell.execute_reply":"2024-12-11T07:44:55.745651Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"features_to_check = ['feature_33', 'feature_58', 'feature_73', 'feature_74']\n\nfor feature in features_to_check:\n    value_counts = result_df[feature].value_counts(dropna=False) \n    print(f\"Value frequencies for {feature}:\")\n    print(value_counts)\n    print(\"-\" * 50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:44:55.748668Z","iopub.execute_input":"2024-12-11T07:44:55.749501Z","iopub.status.idle":"2024-12-11T07:44:55.765951Z","shell.execute_reply.started":"2024-12-11T07:44:55.749433Z","shell.execute_reply":"2024-12-11T07:44:55.764859Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"<span style=\"color:red\">We may safely draw the conclusion, because ***feature_15, 17,32,39,41,42,44,50,52,53,55*** are from the same time_id appears in every day, so it can be regarded as is a special moment before the data produced. ***feature_58,73,74*** are slightly biased but should be the same. **We can fill with 0**</span>, they are Systematic Missing Value, Not Missing at Random.","metadata":{}},{"cell_type":"code","source":"# def fill_Systematic_missing_value(folder_path):\n#     # Features and their respective thresholds\n#     feature_time_thresholds = {\n#         'feature_15': 24, 'feature_17': 4, 'feature_32': 10, 'feature_33': 10,\n#         'feature_39': 68, 'feature_41': 18, 'feature_42': 68,\n#         'feature_44': 18, 'feature_50': 68, 'feature_52': 18,\n#         'feature_53': 68, 'feature_55': 18, 'feature_58': 10,\n#         'feature_73': 10, 'feature_74': 10\n#     }\n\n#     for filename in os.listdir(folder_path):\n#         if filename.endswith('.parquet'):\n#             file_path = os.path.join(folder_path, filename)\n#             print(f\"Processing file: {filename}\")\n\n#             # Load the parquet file\n#             df = pd.read_parquet(file_path)\n\n#             for feature, threshold in feature_time_thresholds.items():\n#                 if feature in df.columns:\n#                     # Fill NaN values with 0 for time_id < threshold\n#                     df.loc[df['time_id'] < threshold, feature] = df.loc[df['time_id'] < threshold, feature].fillna(0)\n\n#             # Save the updated file\n#             df.to_parquet(file_path, index=False)\n#             print(f\"Updated file saved: {filename}\")\n\n# fill_Systematic_missing_value(output_directory)  #/kaggle/working/deal_sys_miss","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:40:04.792983Z","iopub.status.idle":"2024-12-11T07:40:04.793366Z","shell.execute_reply.started":"2024-12-11T07:40:04.793149Z","shell.execute_reply":"2024-12-11T07:40:04.793166Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n\ndef fill_Systematic_missing_value(folder_path, output_folder_path):\n    # Create output folder if it doesn't exist\n    os.makedirs(output_folder_path, exist_ok=True)\n\n    # Features and their respective thresholds\n    feature_time_thresholds = {\n        'feature_15': 24, 'feature_17': 4, 'feature_32': 10, 'feature_33': 10,\n        'feature_39': 68, 'feature_41': 18, 'feature_42': 68,\n        'feature_44': 18, 'feature_50': 68, 'feature_52': 18,\n        'feature_53': 68, 'feature_55': 18, 'feature_58': 10,\n        'feature_73': 10, 'feature_74': 10\n    }\n\n    for filename in os.listdir(folder_path):\n        if filename.endswith('.parquet'):\n            file_path = os.path.join(folder_path, filename)\n            print(f\"Processing file: {filename}\")\n\n            # Load the parquet file\n            df = pd.read_parquet(file_path)\n\n            for feature, threshold in feature_time_thresholds.items():\n                if feature in df.columns:\n                    # Fill NaN values with 0 for time_id < threshold\n                    df.loc[df['time_id'] < threshold, feature] = df.loc[df['time_id'] < threshold, feature].fillna(0)\n\n            # Define output path\n            output_file_path = os.path.join(output_folder_path, filename)\n\n            # Save the updated file\n            df.to_parquet(output_file_path, index=False)\n            print(f\"Updated file saved: {output_file_path}\")\n\n# Input and output folders\n# input_folder = '/kaggle/working/lower' ----output_directory----\nsys_folder = '/kaggle/working/deal_sys_miss'\n\nfill_Systematic_missing_value(lower_folder, sys_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:44:55.767485Z","iopub.execute_input":"2024-12-11T07:44:55.768278Z","iopub.status.idle":"2024-12-11T07:49:35.897519Z","shell.execute_reply.started":"2024-12-11T07:44:55.768225Z","shell.execute_reply":"2024-12-11T07:49:35.896290Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"analyze_partition_data_out(sys_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:50:16.725568Z","iopub.execute_input":"2024-12-11T07:50:16.727398Z","iopub.status.idle":"2024-12-11T07:51:00.160417Z","shell.execute_reply.started":"2024-12-11T07:50:16.727188Z","shell.execute_reply":"2024-12-11T07:51:00.159021Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Random Missing Value","metadata":{}},{"cell_type":"code","source":"print_missing_values_for_time_group(sys_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:51:11.283610Z","iopub.execute_input":"2024-12-11T07:51:11.284013Z","iopub.status.idle":"2024-12-11T07:51:44.284013Z","shell.execute_reply.started":"2024-12-11T07:51:11.283981Z","shell.execute_reply":"2024-12-11T07:51:44.282797Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see that the proportion of the remaining missing values is relatively low, and it is difficult to directly observe the rule, so it can be judged as random missing values. Deleting missing values may lead to data not being aligned at the same time scale. In order to ensure that the timing of data is not lost, we try to fill in these missing values. We can first observe the distribution of the remaining missing values and summarize the data in all folders:\n* The column name of the missing value.\n* date_id of the column where the missing value appears.\n* The proportion of missing values in this column under each date_id, considering the time_id dimension.\n\n### 100% missing","metadata":{}},{"cell_type":"code","source":"def print_missing_value_details(folder_path, k):\n\n    files = os.listdir(folder_path)\n\n    for filename in tqdm(files, desc=\"Processing files\", unit=\"file\"):\n        file_path = os.path.join(folder_path, filename)\n\n        if file_path.endswith('.parquet'):\n            df = pd.read_parquet(file_path)\n\n            if 'date_id' in df.columns:\n                print(f\"\\nProcessing file: {filename}\")\n                \n                for column in df.columns:\n                    if df[column].isnull().any():  # Columns are checked for missing values\n                        \n                        # By `date_id` calculate the proportion of missing values\n                        missing_stats = df.groupby('date_id').apply(\n                            lambda group: group[column].isnull().sum() / len(group) * 100\n                        )\n                        \n                        # Screen for missing values `date_id`\n                        missing_stats = missing_stats[missing_stats > k]  # 0 -> Larger than 10%\n                        \n                        if not missing_stats.empty:\n                            print(f\"\\nColumn '{column}' has missing values:\")\n                            for date_id, ratio in missing_stats.items():\n                                print(f\"- date_id={date_id}: Missing ratio={ratio:.2f}%\")\n                        else:\n                            #print(f\"No missing values > 10% in column '{column}'.\")\n                            continue\n            else:\n                print(f\"Warning: 'date_id' column not found in {filename}\")\n\nprint_missing_value_details(sys_folder, 10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T07:51:44.286567Z","iopub.execute_input":"2024-12-11T07:51:44.287556Z","iopub.status.idle":"2024-12-11T08:02:34.966883Z","shell.execute_reply.started":"2024-12-11T07:51:44.287513Z","shell.execute_reply":"2024-12-11T08:02:34.965728Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see that the feature statistics that **are completely missing** in one day are as follows:\n\n**------Parquet------|----Number of Missing Days----|------feature------**\n\n---partition_id=1---|-----------------all-----------------|----21,26,27,31----\n\n---partition_id=2---|-----------------all-----------------|----21,26,27,31----\n\n---partition_id=3---|---------------first_17--------------|----21,26,27,31----\n\n---partition_id=7---|--------------random_2------------|---------08---------\n\n---partition_id=8---|--------------random_5------------|---------08---------\n\n---partition_id=9---|--------------random_1------------|---------08---------\n\n\nHere, we can directly delete all samples under the corresponding date when ***feature_08*** is missing, because there are only seven days, as well as the dates are discrete and random, so the impact is not significant.\n\nFor ***feature_21,26,27,31*** we cannot do this, otherwise the data loss is too large, so we consider filling with 1. Moreover, the missing date of these four features is a long and continuous period, so we can do this filling.\n\n<span style=\"color:red\">**That is, systematic missing values are filled with 0, and there is a large number of random missing values for a specific time period filled with 1.**</span>","metadata":{}},{"cell_type":"code","source":"dest_folder = '/kaggle/working/deal_random_miss'  # Where to save the new file\n\n\nif not os.path.exists(dest_folder):\n    os.makedirs(dest_folder)\n\n\nfor filename in os.listdir('/kaggle/working/deal_sys_miss'):\n    if filename.endswith('.parquet'):\n        file_path = os.path.join('/kaggle/working/deal_sys_miss', filename)\n        partition_id = int(filename.split('=')[1].split('.')[0])  # get partition_id\n\n        print(f\"Processing file: {filename}\")\n\n        # For partition_id = 1 and 2, the missing value for filling four columns is 1\n        if partition_id in [1, 2]:\n            df = pd.read_parquet(file_path)\n            for feature in ['feature_21', 'feature_26', 'feature_27', 'feature_31']:\n                if feature in df.columns:\n                    df[feature] = df[feature].fillna(1)\n            df.to_parquet(os.path.join(dest_folder, filename), index=False)\n            print(f\"File updated and saved: {filename}\")\n\n        # For partition_id = 3, the missing value for the 17 days before date_id is filled is 1\n        elif partition_id == 3:\n            df = pd.read_parquet(file_path)\n            for feature in ['feature_21', 'feature_26', 'feature_27', 'feature_31']:\n                if feature in df.columns:\n                    # Fill in missing values with date_id ranging from 510 to 527\n                    df.loc[df['date_id'].between(510, 527), feature] = df.loc[df['date_id'].between(510, 527), feature].fillna(1)\n            df.to_parquet(os.path.join(dest_folder, filename), index=False)\n            print(f\"File updated and saved: {filename}\")\n\n        # For partition_id = 7, 8, and 9, delete the rows with NaN for feature_08, and only delete the rows with all missing values for feature_08 columns under a date_id\n        elif partition_id in [7, 8, 9]:\n            df = pd.read_parquet(file_path)\n            if 'feature_08' in df.columns:\n                # Find those date_ids for which the feature_08 column has all missing values under a date_id\n                nan_date_ids = df.groupby('date_id')['feature_08'].apply(lambda x: x.isnull().all())\n                # Delete all rows under these date_ids\n                df = df[~df['date_id'].isin(nan_date_ids[nan_date_ids].index)]\n            df.to_parquet(os.path.join(dest_folder, filename), index=False)\n            print(f\"File updated and saved: {filename}\")\n\n        # For partition_id = 4, 5, 6, copy directly from the original folder to the target folder\n        elif partition_id in [4, 5, 6]:\n            shutil.copy(file_path, os.path.join(dest_folder, filename))\n            print(f\"File copied: {filename}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T08:02:34.968162Z","iopub.execute_input":"2024-12-11T08:02:34.968536Z","iopub.status.idle":"2024-12-11T08:05:55.814772Z","shell.execute_reply.started":"2024-12-11T08:02:34.968501Z","shell.execute_reply":"2024-12-11T08:05:55.813296Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"analyze_partition_data_out(dest_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T08:05:55.818081Z","iopub.execute_input":"2024-12-11T08:05:55.818889Z","iopub.status.idle":"2024-12-11T08:06:43.496957Z","shell.execute_reply.started":"2024-12-11T08:05:55.818835Z","shell.execute_reply":"2024-12-11T08:06:43.495656Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This function lets you check that we deleted feature_08 data correctly and did not cause high data loss during the whole process.\n\n### Partial missing","metadata":{}},{"cell_type":"code","source":"print_missing_value_details(dest_folder, 10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T08:06:43.498877Z","iopub.execute_input":"2024-12-11T08:06:43.500281Z","iopub.status.idle":"2024-12-11T08:17:02.680556Z","shell.execute_reply.started":"2024-12-11T08:06:43.500235Z","shell.execute_reply":"2024-12-11T08:17:02.679423Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"After checking, \"The feature is 100% missing on a specific date\" has all been processed, leaving us with \"The feature is partial missing on a specific date\".\n\nHere, choose to **fill with the previous data first, if the previous data is not, then fill with the latter data**.\n\n\nHere's an interesting thing:\n<span style=\"color:red\">if read the data as float16, you would get the TypeError: No matching signature found.</span> Therefore, for missing values to be filled, if the data type was previously converted to float16, we first convert back to float32, fill and then convert to float16 to reduce memory.","metadata":{}},{"cell_type":"code","source":"def fill_missing_and_check(folder_path):\n    # Iterate over all files in the folder\n    for filename in os.listdir(folder_path):\n        file_path = os.path.join(folder_path, filename)\n        \n        if not file_path.endswith('.parquet'):\n            continue \n        \n        try:\n            print(f\"\\nProcessing file: {filename}\")\n            \n            # Load the parquet file\n            df = pd.read_parquet(file_path)\n            \n            # Show column processing progress\n            columns = df.columns\n            for col in tqdm(columns, desc=f\"Processing columns in {filename}\"):\n                try:\n                    if pd.api.types.is_numeric_dtype(df[col]):\n                        if df[col].dtype == 'float16':\n                            # If the column is float16, convert to float32, fill missing values, then convert back to float16\n                            df[col] = df[col].astype('float32')\n                            df[col] = df[col].fillna(method='ffill').fillna(method='bfill')\n                            df[col] = df[col].astype('float16')\n                        else:\n                            # If the column is not float16, directly fill missing values\n                            df[col] = df[col].fillna(method='ffill').fillna(method='bfill')\n                except Exception as e:\n                    print(f\"Error while processing column '{col}' in file '{filename}': {e}\")\n            \n            # Check if there are still missing values after filling\n            if df.isnull().sum().sum() > 0:\n                print(f\"Warning: There are still missing values in {filename} after filling.\")\n            else:\n                print(f\"All missing values have been filled in {filename}.\")\n            \n            # Save the modified DataFrame back to the original parquet file\n            df.to_parquet(file_path)\n            print(f\"File {filename} has been updated.\")\n        \n        except Exception as e:\n            print(f\"Error while processing file '{filename}': {e}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T08:17:02.682159Z","iopub.execute_input":"2024-12-11T08:17:02.682547Z","iopub.status.idle":"2024-12-11T08:17:02.696846Z","shell.execute_reply.started":"2024-12-11T08:17:02.682513Z","shell.execute_reply":"2024-12-11T08:17:02.695413Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n\nfill_missing_and_check(dest_folder)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T08:17:02.698500Z","iopub.execute_input":"2024-12-11T08:17:02.699001Z","iopub.status.idle":"2024-12-11T08:23:05.349817Z","shell.execute_reply.started":"2024-12-11T08:17:02.698938Z","shell.execute_reply":"2024-12-11T08:23:05.348473Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print_missing_value_details(dest_folder, 0.01)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T08:23:05.351602Z","iopub.execute_input":"2024-12-11T08:23:05.352571Z","iopub.status.idle":"2024-12-11T08:23:47.621929Z","shell.execute_reply.started":"2024-12-11T08:23:05.352530Z","shell.execute_reply":"2024-12-11T08:23:47.620652Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Next\nFinally, let's do some comparative experiments to confirm the effectiveness of each link:\n\n* Experiment 1----lgb baseline+ raw data (Delete partition_id=0)\n* Experiment 2----lgb baseline+ compressed data (Delete earlier data)\n* Experiment 3----lgb baseline+ compressed data (Delete earlier data) + data after imputation of systematically missing values\n* Experiment 4----lgb baseline+ compressed data (Delete earlier data) + data after imputation of systematic missing values + data after imputation of random missing values\n\n**To facilitate subsequent cross-validation, the parquet under each folder are now redivided into five.**","metadata":{}},{"cell_type":"code","source":"# folder_path = Path(\"/kaggle/working/\")  # Replace with the actual folder path\n\n# # Traverse through all parquet files in the folder and subfolders\n# parquet_files = list(folder_path.rglob(\"*.parquet\"))\n\n# if not parquet_files:\n#     print(\"No parquet files found in the specified folder.\")\n# else:\n#     date_counts = pd.Series(dtype=int)\n\n#     for parquet_file in parquet_files:\n#         try:\n#             df = pd.read_parquet(parquet_file)\n\n#             # Check if 'date_id' column exists\n#             if 'date_id' not in df.columns:\n#                 print(f\"Skipping {parquet_file}: 'date_id' column not found.\")\n#                 continue\n\n#             date_counts = date_counts.add(df['date_id'].value_counts(), fill_value=0)\n\n#         except Exception as e:\n#             print(f\"Error reading {parquet_file}: {e}\")\n\n#     if not date_counts.empty:\n#         # Sort date_counts by date_id and determine date ranges for 5 parts\n#         date_counts = date_counts.sort_index()\n#         total_days = len(date_counts)\n#         split_indices = np.array_split(np.arange(total_days), 5)\n\n#         date_ranges = [(date_counts.index[indices[0]], date_counts.index[indices[-1]]) for indices in split_indices]\n\n#         print(\"Determined date ranges for splitting:\", date_ranges)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-11T09:02:54.180992Z","iopub.execute_input":"2024-12-11T09:02:54.181455Z","iopub.status.idle":"2024-12-11T09:03:20.224855Z","shell.execute_reply.started":"2024-12-11T09:02:54.181415Z","shell.execute_reply":"2024-12-11T09:03:20.223508Z"}},"outputs":[],"execution_count":null}]}