{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84493,"databundleVersionId":9871156,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Background\n\nWhen we have 968 time_id's per day, which could roughly map to 16 hours x 60 minutes = 960 minutes. If we ignore the 8 extra minutes, it nicely aligns with 16 regular trading hours plus pre/post market. Anything can be traded during this timeframe (equities, futures, forex, crypto, etc)\n\nOutside of this window, Futures trade 23 hours a day (23 x 60 = 1,380 / 968 = 1.43), Forex trade 24 hours a day (24 x 60 = 1,440 = 1.49). Regular Trading Hours is 6.5 hours (6.5 x 60 = 390 / 968 = 0.40) What's nice about 16 hours is that 16 x 60 = 960 / 968 = 0.99, which is pretty close to 1 (for 1-minute interval data). The other time ranges give odd partial ratios. And I don't think this would be tick or volume data as it's always the same number of time_id's (after it settles) per date_id with no missing values. So that suggests it's a clock-based sub-division.\n\n| **Trading Period**      | **Start Time** | **End Time**   | **Hours** | **Fraction** | **Expected IDs** | **Actual IDs** | **ID Range** |\n|--------------------------|----------------|----------------|-----------|--------------|------------------------|---------------------|--------------------|\n| Pre-Market              | 4:00 am ET     | 9:30 am ET     | 5.5 hrs   | 34.38%       | 330                    | 332                 | 0–331             |\n| Regular Trading Hours    | 9:30 am ET     | 4:00 pm ET     | 6.5 hrs   | 40.62%       | 390                    | 393                 | 332–724           |\n| After-Hours             | 4:00 pm ET     | 8:00 pm ET     | 4 hrs     | 25.0%        | 240                    | 243                 | 725–967           |\n| **Total**               | **4:00 am ET** | **8:00 pm ET** | **16 hrs** | **100%**     | **960**                | **968**             | **0–967**         |\n\n**Note:** The data description says \"the actual time intervals between time_id values may vary.\" That could be why there is a discrepancy, but it could also be that this rough mapping of 1 time-id to 1 minute is invalid.\n\nThe purpose of this exploration was to see if there appear to be correlations in feature activity from pre-market, to RTH, to after hours. If we can see clear shifts in the behavior of the signals (ex: changes in volume, volatility, trending vs. consolidation, etc.) that map to these different trading periods, it would reinforce this hypothesis.\n\nSo far, some features like 'feature_01' seem to have distinct shifts pre/post market vs. RTH, but it's not consistent every day. Other features do not show any discernable change between these periods. I haven't done a full analysis across every feature or responder. Please comment if you see patterns that might align with this idea (or contradict it).\n\n## Imports","metadata":{}},{"cell_type":"code","source":"# Data processing imports\nimport numpy as np\nimport pandas as pd\nimport os\nimport polars as pl\nfrom pathlib import Path\n\n# Visualization imports\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:13:20.952252Z","iopub.execute_input":"2024-12-02T05:13:20.952638Z","iopub.status.idle":"2024-12-02T05:13:24.614955Z","shell.execute_reply.started":"2024-12-02T05:13:20.952607Z","shell.execute_reply":"2024-12-02T05:13:24.613807Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Processing","metadata":{}},{"cell_type":"code","source":"# Gather the train data partitions\nDATA_DIR = Path('/kaggle/input/jane-street-real-time-market-data-forecasting')\nN_PARTITION = len(os.listdir(DATA_DIR / 'train.parquet'))\ntrain_parquets = [f\"{DATA_DIR}/train.parquet/partition_id={i}/part-0.parquet\" for i in range(N_PARTITION)]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:13:26.567955Z","iopub.execute_input":"2024-12-02T05:13:26.568493Z","iopub.status.idle":"2024-12-02T05:13:26.585664Z","shell.execute_reply.started":"2024-12-02T05:13:26.568454Z","shell.execute_reply":"2024-12-02T05:13:26.584601Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load a subset of the data, you can choose any partition or multiple\ndf = pd.read_parquet(train_parquets[6])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:13:27.437781Z","iopub.execute_input":"2024-12-02T05:13:27.438248Z","iopub.status.idle":"2024-12-02T05:13:39.182615Z","shell.execute_reply.started":"2024-12-02T05:13:27.438157Z","shell.execute_reply":"2024-12-02T05:13:39.181374Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Function to Plot by Trading Period","metadata":{}},{"cell_type":"code","source":"def plot_by_period(df, days, column, symbols=None, time_id_ranges=None, color_by='period'):\n    \"\"\"\n    Plots a specified feature or responder against time_id for given days, grouped by trading periods or symbol.\n    \n    Parameters:\n        df (DataFrame): The dataframe containing the data to plot.\n        days (str, int, list or tuple): The days (date_id) to visualize. Tuple will be start and end of a range.\n        column (str): The column (feature or responder) to plot.\n        symbols (str, int, list, or tuple, optional): The symbol(s) to filter the data by. Tuple will be start\n            and end of a range. Default is None (no filtering).\n        time_id_ranges (dict, optional): Custom time_id boundaries for trading periods. \n            Default is Pre-Market, Regular Trading Hours, and After-Hours.\n        color_by (str, optional): Determines the color coding ('period', 'symbol', or None). Default is 'period'.\n    \"\"\"\n    # Default trading periods\n    if time_id_ranges is None:\n        time_id_ranges = {\n            'Pre-Market': (0, 332),\n            'Regular Trading Hours': (333, 725),\n            'After-Hours': (726, 968)\n        }\n\n    # Function to identify trading period\n    def assign_period(time_id):\n        for period, (start, end) in time_id_ranges.items():\n            if start <= time_id <= end:\n                return period\n        return 'Unknown'\n\n    # Function to map hours to time_id ranges\n    def hour_to_time_id(hour):\n        total_minutes = (hour - 4) * 60\n        return int((total_minutes / (16 * 60)) * 968)\n\n    # Prepare hour ticks and labels\n    hour_ticks = [hour_to_time_id(hour) for hour in range(4, 21)]\n    hour_labels = [f\"{hour:02d}:00\" for hour in range(4, 21)]\n\n    # Handle days parameter\n    if isinstance(days, (str, int)):\n        days = [days]\n    elif isinstance(days, tuple) and len(days) == 2:\n        start, end = days\n        days = list(range(start, end + 1))\n    \n    # Filter by symbol if provided\n    if symbols is not None:\n        if isinstance(symbols, tuple) and len(symbols) == 2:\n            start, end = symbols\n            symbols = list(range(start, end + 1))\n            suffix = f\" for Symbols {start} - {end}\"\n        elif isinstance(symbols, (str, int)):\n            symbols = [symbols]\n            suffix = f\" for Symbol {symbols[0]}\"\n        elif isinstance(symbols, list):\n            suffix = f\" for Symbols {', '.join(map(str, symbols))}\"\n        df = df[df['symbol_id'].isin(symbols)]\n    elif color_by == 'period':\n        suffix = ' by Time Period'\n    elif color_by == 'symbol':\n        suffix = ' by Symbol'\n    else:\n        suffix = ''\n        \n    # Loop through each day and create a plot\n    for day in days:\n        df_day = df[df['date_id'] == day].copy()\n\n        # Assign coloring based on the `color_by` parameter\n        if color_by == 'period':\n            df_day['color_group'] = df_day['time_id'].apply(assign_period)\n            color_labels = list(time_id_ranges.keys())\n        elif color_by == 'symbol':\n            # Assign prefixed values directly to color_group\n            df_day['color_group'] = df_day['symbol_id'].apply(lambda symbol: f\"Symbol {symbol}\")\n            color_labels = df_day['color_group'].unique()\n        else:\n            df_day['color_group'] = 'All Data'\n            color_labels = ['All Data']\n        \n        fig, ax1 = plt.subplots(figsize=(14, 7))\n        \n        # Set vertical grid aligned with trading hours\n        ax1.set_xticks(hour_ticks)\n        ax1.grid(color='lightgrey', linestyle='-', linewidth=0.5, alpha=1, zorder=0)\n\n        # Plot color-coded data for each group\n        for group in color_labels:\n            group_data = df_day[df_day['color_group'] == group]\n            ax1.plot(group_data['time_id'], group_data[column], label=group, alpha=0.8, zorder=3)\n\n        ax1.set_title(f\"{column} on Day {day}{suffix}\", fontsize=16, pad=14)\n        ax1.set_xlabel(\"Time ID\", fontsize=14, labelpad=10)\n        ax1.set_ylabel(column, fontsize=14, labelpad=10)\n        ax1.legend()\n\n        # Add secondary x-axis for trading hours\n        ax2 = ax1.secondary_xaxis('top')\n        ax2.set_xticks(hour_ticks)\n        ax2.set_xticklabels(hour_labels)\n        ax2.set_xlabel(\"Trading Hour\", fontsize=14, labelpad=10)\n        \n        plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:25:21.934743Z","iopub.execute_input":"2024-12-02T05:25:21.935103Z","iopub.status.idle":"2024-12-02T05:25:21.952577Z","shell.execute_reply.started":"2024-12-02T05:25:21.935072Z","shell.execute_reply":"2024-12-02T05:25:21.951232Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Feature Plots by Time Period\n\nHere are plots for various feature, day and symbol combinations. You can call the function with any combination of `days`, the `column` you want to plot, and optionally any `symbols` you want to filter by (by default it's all).","metadata":{}},{"cell_type":"code","source":"# Plot 'feature_01' on day 1020\nplot_by_period(df, days=1020, column='feature_01')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:25:22.801628Z","iopub.execute_input":"2024-12-02T05:25:22.802020Z","iopub.status.idle":"2024-12-02T05:25:23.647197Z","shell.execute_reply.started":"2024-12-02T05:25:22.801985Z","shell.execute_reply":"2024-12-02T05:25:23.646023Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'feature_01' on day 1020 only for symbol_id 38\nplot_by_period(df, days=1020, symbols=38, column='feature_01')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:25:25.831363Z","iopub.execute_input":"2024-12-02T05:25:25.831815Z","iopub.status.idle":"2024-12-02T05:25:27.230904Z","shell.execute_reply.started":"2024-12-02T05:25:25.831777Z","shell.execute_reply":"2024-12-02T05:25:27.229294Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'feature_05' on days 1020-1021 only for symbol_id 30-40\nplot_by_period(df, days=(1020, 1021), symbols=(30,40), column='feature_05')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:25:31.229737Z","iopub.execute_input":"2024-12-02T05:25:31.230189Z","iopub.status.idle":"2024-12-02T05:25:33.773155Z","shell.execute_reply.started":"2024-12-02T05:25:31.230125Z","shell.execute_reply":"2024-12-02T05:25:33.771978Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'feature_01' on days 1020-1024 (5 days)\nplot_by_period(df, days=(1020, 1024), column='feature_01')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:25:37.792467Z","iopub.execute_input":"2024-12-02T05:25:37.792915Z","iopub.status.idle":"2024-12-02T05:25:43.130152Z","shell.execute_reply.started":"2024-12-02T05:25:37.792876Z","shell.execute_reply":"2024-12-02T05:25:43.129061Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'responder_6' on day 1020 for all symbols\nplot_by_period(df, days=1020, column='responder_6')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:25:46.668775Z","iopub.execute_input":"2024-12-02T05:25:46.669148Z","iopub.status.idle":"2024-12-02T05:25:48.158854Z","shell.execute_reply.started":"2024-12-02T05:25:46.669116Z","shell.execute_reply":"2024-12-02T05:25:48.157508Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'responder_6' on day 1020 only for symbol_id 38\nplot_by_period(df, days=1020, symbols=38, column='responder_6')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:25:53.333091Z","iopub.execute_input":"2024-12-02T05:25:53.333549Z","iopub.status.idle":"2024-12-02T05:25:54.508490Z","shell.execute_reply.started":"2024-12-02T05:25:53.333512Z","shell.execute_reply":"2024-12-02T05:25:54.507253Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'responder_6' on day 1020 only for symbol_id 38 without coloring\nplot_by_period(df, days=1020, symbols=38, column='responder_6', color_by=None)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:26:20.064781Z","iopub.execute_input":"2024-12-02T05:26:20.065247Z","iopub.status.idle":"2024-12-02T05:26:21.245521Z","shell.execute_reply.started":"2024-12-02T05:26:20.065202Z","shell.execute_reply":"2024-12-02T05:26:21.244211Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'feature_01' on day 1020 without coloring\nplot_by_period(df, days=1020, column='feature_01', color_by=None)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:26:26.807439Z","iopub.execute_input":"2024-12-02T05:26:26.807966Z","iopub.status.idle":"2024-12-02T05:26:27.574902Z","shell.execute_reply.started":"2024-12-02T05:26:26.807931Z","shell.execute_reply":"2024-12-02T05:26:27.573634Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'responder_6' on day 1020 and color by symbol\nplot_by_period(df, days=1020, column='responder_6', symbols=[1,2,3], color_by='symbol')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:26:34.578744Z","iopub.execute_input":"2024-12-02T05:26:34.579139Z","iopub.status.idle":"2024-12-02T05:26:35.570137Z","shell.execute_reply.started":"2024-12-02T05:26:34.579102Z","shell.execute_reply":"2024-12-02T05:26:35.568903Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot 'feature_12' on day 1020 and color by symbol\nplot_by_period(df, days=1020, column='feature_12', symbols=[10,20,30], color_by='symbol')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T05:29:52.169514Z","iopub.execute_input":"2024-12-02T05:29:52.170057Z","iopub.status.idle":"2024-12-02T05:29:53.713652Z","shell.execute_reply.started":"2024-12-02T05:29:52.170017Z","shell.execute_reply":"2024-12-02T05:29:53.712259Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}