{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84493,"databundleVersionId":9871156,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport pyarrow.parquet as pq\nimport glob\n\n# Set path to training data partitions\ntrain_path = \"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/*\"\n\n# Load one partition to explore data structure and memory footprint\nfiles = glob.glob(f\"{train_path}/*.parquet\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Visualizing Time Series #\n## I. Raw Time Series ##\nWe are looking for obvious patterns like trends (increasing or decreasing), seasonality (repeated cycles), and noise (random fluctuations).\n\nFor example any sudden jumps, dips, or gaps in the data (e.g., outliers or missing values).\n\n### Key Insights ###\n* If the data shows a trend (upward or downward), models like ARIMA, SARIMA, or linear regression with time as a feature might be appropriate.\n* If the data shows seasonality, models like SARIMA, Holt-Winters, or deep learning models (LSTM, Transformers) can be considered.\n* If there is random noise without clear structure, simpler models like ARMA might be effective.\n\n","metadata":{}},{"cell_type":"code","source":"# Load and inspect the first partition\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n\n    unique_symbols = data['symbol_id'].unique()\n\n    # Set up the figure\n    plt.figure(figsize=(24, 4))\n\n    # Select a sample of symbols to plot (e.g., first 5 unique symbols)\n    sample_symbols = unique_symbols[:5]  # Adjust the number as needed for clarity\n\n    for symbol in sample_symbols:\n        for column in list(data.columns.values):\n            if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n                symbol_data = data[data['symbol_id'] == symbol]\n                plt.plot(symbol_data['date_id'], symbol_data[f'{column}'], label=f'Symbol ID {symbol}')\n\n                plt.title(f'{column} over time by symbol id: {symbol}')\n                plt.xlabel('Date ID')\n                plt.ylabel(f'{column}')\n                plt.legend()\n                plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-15T00:29:15.880816Z","iopub.execute_input":"2024-11-15T00:29:15.881245Z","iopub.status.idle":"2024-11-15T00:48:41.073558Z","shell.execute_reply.started":"2024-11-15T00:29:15.881206Z","shell.execute_reply":"2024-11-15T00:48:41.072026Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-15T00:28:51.63961Z","iopub.status.idle":"2024-11-15T00:28:51.640052Z","shell.execute_reply.started":"2024-11-15T00:28:51.639843Z","shell.execute_reply":"2024-11-15T00:28:51.639865Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-15T00:28:51.64126Z","iopub.status.idle":"2024-11-15T00:28:51.641628Z","shell.execute_reply.started":"2024-11-15T00:28:51.641445Z","shell.execute_reply":"2024-11-15T00:28:51.641464Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-15T00:28:51.642953Z","iopub.status.idle":"2024-11-15T00:28:51.643423Z","shell.execute_reply.started":"2024-11-15T00:28:51.643213Z","shell.execute_reply":"2024-11-15T00:28:51.643235Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## II. Decompose the Time Series (Trend, Seasonality, Residuals) ##\nLet's use some decomposition methods like additive or multiplicative decomposition to separate the time series into trend, seasonal, and residual components.\n\n### Key Insights: ##\n* If the seasonal component is strong, consider models that can explicitly model seasonality (e.g., SARIMA, Prophet, or XGBoost with seasonal features).\n* If the trend is strong, models like ARIMA or models that can capture linear trends, such as linear regression, may be useful.\nResiduals should ideally resemble white noise (random noise without patterns).\n* If residuals exhibit structure, reconsider the model or try additional features.\n","metadata":{}},{"cell_type":"code","source":"# Iterate through each partition\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n    unique_symbols = data['symbol_id'].unique()\n\n    # Sample of symbols to plot\n    sample_symbols = unique_symbols[:5]  # Adjust the number as needed\n\n    # Loop through each symbol_id\n    for symbol in sample_symbols:\n        symbol_data = data[data['symbol_id'] == symbol].set_index('date_id').sort_index()\n\n        # Loop through each feature column\n        for column in data.columns:\n            if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n                # Ensure regular intervals and interpolate missing data\n                feature_data = symbol_data[column].asfreq('D').interpolate()\n\n                try:\n                    # Perform decomposition\n                    decomposition = seasonal_decompose(feature_data, model='additive', period=30)\n\n                    # Plot the decomposition\n                    plt.figure(figsize=(18, 10))\n\n                    plt.subplot(4, 1, 1)\n                    plt.plot(decomposition.observed)\n                    plt.title(f'Observed: {column} (Symbol ID: {symbol})')\n\n                    plt.subplot(4, 1, 2)\n                    plt.plot(decomposition.trend)\n                    plt.title('Trend')\n\n                    plt.subplot(4, 1, 3)\n                    plt.plot(decomposition.seasonal)\n                    plt.title('Seasonal')\n\n                    plt.subplot(4, 1, 4)\n                    plt.plot(decomposition.resid)\n                    plt.title('Residual')\n\n                    plt.tight_layout()\n                    plt.show()\n\n                except ValueError as e:\n                    print(f\"Decomposition failed for {column} (Symbol ID: {symbol}): {e}\")\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Plot ACF (Auto-Correlation Function) and PACF (Partial Auto-Correlation Function)\nACF shows the correlation of the time series with its lags.\nPACF shows the partial correlation after removing the effects of intervening lags.\n\n### Key Insights: ###\n\n* If the ACF decays exponentially and the PACF cuts off after a few lags, an AR (Auto-Regressive) model might be appropriate.\n* If the ACF cuts off and PACF decays exponentially, an MA (Moving Average) model might be suitable.\n* If both ACF and PACF decay, a combination of AR and MA (ARMA) or ARIMA could be a good choice.\n\n","metadata":{}},{"cell_type":"code","source":"for i in range(10):\n    data = pd.read_parquet(files[i])\n    unique_symbols = data['symbol_id'].unique()\n\n    # Sample of symbols to plot\n    sample_symbols = unique_symbols[:5]  # Adjust the number as needed\n\n    # Loop through each symbol_id\n    for symbol in sample_symbols:\n        symbol_data = data[data['symbol_id'] == symbol].set_index('date_id').sort_index()\n\n        # Loop through each feature column\n        for column in data.columns:\n            if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n                # Ensure regular intervals and interpolate missing data\n                feature_data = symbol_data[column].asfreq('D').interpolate()\n\n                try:\n                    # Set up the figure for ACF and PACF\n                    fig, axes = plt.subplots(1, 2, figsize=(16, 4))\n                    fig.suptitle(f'ACF and PACF for {column} (Symbol ID: {symbol})')\n\n                    # Plot ACF\n                    plot_acf(feature_data.dropna(), ax=axes[0], lags=30)\n                    axes[0].set_title(f'ACF: {column}')\n\n                    # Plot PACF\n                    plot_pacf(feature_data.dropna(), ax=axes[1], lags=30)\n                    axes[1].set_title(f'PACF: {column}')\n\n                    plt.tight_layout(rect=[0, 0, 1, 0.95])\n                    plt.show()\n\n                except ValueError as e:\n                    print(f\"ACF/PACF plot failed for {column} (Symbol ID: {symbol}): {e}\")\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Stationarity Testing #\n**Goal**: Most time series models (e.g., ARIMA) require the data to be stationary. \n\nStationarity means the statistical properties of the series, like mean and variance, do not change over time.\n\n## Dickey-Fuller Test (ADF Test): ##\nThis test checks for stationarity. \n\n### Key Insights: ###\n\n* If the p-value is less than a chosen threshold (e.g., 0.05), the series is stationary.\n* If the p-value is high, the series is non-stationary, and you may need to difference the series (or use models that can handle non-stationary data like SARIMA).","metadata":{}},{"cell_type":"code","source":"# Function to perform ADF test and print the results\ndef adf_test(series, column_name, symbol_id):\n    result = adfuller(series.dropna())\n    print(f'ADF Test for {column_name} (Symbol ID: {symbol_id}):')\n    print(f'ADF Statistic: {result[0]}')\n    print(f'p-value: {result[1]}')\n    print('Critical Values:')\n    for key, value in result[4].items():\n        print(f'   {key}, {value}')\n    print('\\n')\n\n# Iterate through each partition\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n    unique_symbols = data['symbol_id'].unique()\n\n    # Sample of symbols to test\n    sample_symbols = unique_symbols[:5]  # Adjust the number as needed\n\n    # Loop through each symbol_id\n    for symbol in sample_symbols:\n        symbol_data = data[data['symbol_id'] == symbol].set_index('date_id').sort_index()\n\n        # Loop through each feature column\n        for column in data.columns:\n            if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n                # Ensure regular intervals and interpolate missing data\n                feature_data = symbol_data[column].asfreq('D').interpolate()\n\n                try:\n                    # Perform ADF test\n                    adf_test(feature_data, column, symbol)\n\n                except ValueError as e:\n                    print(f\"ADF test failed for {column} (Symbol ID: {symbol}): {e}\")\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualizing Rolling Statistics: ##\nCalculate and plot the rolling mean and rolling standard deviation. A stationary series will have constant mean and variance over time.\n\n### Key Insights: ###\n* If these statistics change over time, consider differencing or using models that can handle non-stationary data.","metadata":{}},{"cell_type":"code","source":"# Parameters for rolling window\nwindow_size = 175  # Adjust as needed (e.g., 30 for monthly if data is daily)\n\n# Iterate through each partition\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n    unique_symbols = data['symbol_id'].unique()\n\n    # Sample of symbols to plot\n    sample_symbols = unique_symbols[:5]  # Adjust the number as needed\n\n    # Loop through each symbol_id\n    for symbol in sample_symbols:\n        symbol_data = data[data['symbol_id'] == symbol].set_index('date_id').sort_index()\n\n        # Loop through each feature column\n        for column in data.columns:\n            if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n                # Ensure regular intervals and interpolate missing data\n                feature_data = symbol_data[column].asfreq('D').interpolate()\n\n                # Calculate rolling mean and standard deviation\n                rolling_mean = feature_data.rolling(window=window_size).mean()\n                rolling_std = feature_data.rolling(window=window_size).std()\n\n                # Plot the original data, rolling mean, and rolling standard deviation\n                plt.figure(figsize=(12, 6))\n                plt.plot(feature_data, label=f'Original {column}')\n                plt.plot(rolling_mean, color='orange', label='Rolling Mean')\n                plt.plot(rolling_std, color='green', label='Rolling Std Dev')\n                \n                plt.title(f'Rolling Mean & Standard Deviation of {column} (Symbol ID: {symbol})')\n                plt.xlabel('Date ID')\n                plt.ylabel(f'{column}')\n                plt.legend()\n                plt.show()\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Missing Values and Outliers #\n**Goal:** Identifying missing values or outliers is important for model robustness.\n\n## Handling Missing Values: ##\nVisualize any missing values using a heatmap or line plot.\n\n### Key Insights: ###\n\n* For missing values, imputation methods (e.g., forward filling, backward filling, interpolation) can be used.\n* If a large portion of data is missing or outliers are frequent, consider whether imputation or data removal is the best approach.\n","metadata":{}},{"cell_type":"code","source":"data = pd.read_parquet(files[0])\n\n# Heatmap for missing values\nplt.figure(figsize=(12, 6))\nsns.heatmap(data.isnull(), cbar=False, cmap='viridis', yticklabels=False)\nplt.title('Missing Values Heatmap')\nplt.xlabel('Columns')\nplt.ylabel('Rows')\nplt.show()\n\n# Line plots to show missing data in each column\n# Adjust number of columns for clearer visualization if needed\nfor column in data.columns:\n    if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n        plt.figure(figsize=(12, 4))\n        plt.plot(data['date_id'], data[column], label=column)\n        plt.title(f'Missing Values in {column}')\n        plt.xlabel('Date ID')\n        plt.ylabel(column)\n        plt.legend()\n        plt.show()\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Detecting Outliers: ##\nVisualize data points that deviate significantly from the general pattern.\nUse statistical tests like IQR or z-scores to detect outliers.\n\n### Insights: ###\n\n* If the dataset contains a few outliers, these can be handled through robust models or by trimming/removing them. Some models, like Random Forest or XGBoost, can handle outliers naturally.\n* If outliers are systematic (e.g., seasonal spikes), consider explicitly modeling them as additional features or using models like Prophet.","metadata":{}},{"cell_type":"code","source":"# Parameters for z-score threshold and IQR multiplier\nz_threshold = 3\niqr_multiplier = 1.5\n\n# Load one partition to check for outliers\ndata = pd.read_parquet(files[0])\n\n# Loop through each feature column\nfor column in data.columns:\n    if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n        # Select the column data\n        column_data = data[column]\n\n        # Detect outliers using the IQR method\n        Q1 = column_data.quantile(0.25)\n        Q3 = column_data.quantile(0.75)\n        IQR = Q3 - Q1\n        iqr_outliers = (column_data < (Q1 - iqr_multiplier * IQR)) | (column_data > (Q3 + iqr_multiplier * IQR))\n\n        # Detect outliers using the z-score method\n        z_scores = zscore(column_data.fillna(column_data.mean()))  # fill NA for zscore calculation\n        z_outliers = (np.abs(z_scores) > z_threshold)\n\n        # Plot the data with outliers\n        plt.figure(figsize=(12, 6))\n        plt.plot(data['date_id'], column_data, label=column)\n        \n        # Plot IQR outliers\n        plt.scatter(data['date_id'][iqr_outliers], column_data[iqr_outliers], color='red', label='IQR Outliers')\n\n        # Plot z-score outliers\n        plt.scatter(data['date_id'][z_outliers], column_data[z_outliers], color='orange', label='Z-score Outliers')\n\n        # Labels and title\n        plt.title(f'Outliers in {column} using IQR and Z-scores')\n        plt.xlabel('Date ID')\n        plt.ylabel(column)\n        plt.legend()\n        plt.show()\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Feature Engineering #\n**Goal:** Create additional features from the time series that may improve model performance.\n\n## Time-based Features ##\nAdd features like month, day of the week, day of the year, or hour of the day to capture seasonality.\n\n\n","metadata":{}},{"cell_type":"code","source":"\n# Assuming day 0 corresponds to a specific start date\nstart_date = pd.to_datetime(\"2000-01-01\")  # Change as appropriate\n\n# Iterate through each partition to add seasonal features\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n\n    # Convert 'date_id' to a datetime format\n    data['date'] = start_date + pd.to_timedelta(data['date_id'], unit='D')\n    \n    # Extract seasonality features\n    data['month'] = data['date'].dt.month\n    data['day_of_week'] = data['date'].dt.dayofweek  # Monday=0, Sunday=6\n    data['day_of_year'] = data['date'].dt.dayofyear\n    data['week_of_year'] = data['date'].dt.isocalendar().week\n    data['quarter'] = data['date'].dt.quarter\n    \n    # Print or save modified data to verify new columns\n    print(data[['date_id', 'date', 'month', 'day_of_week', 'day_of_year', 'week_of_year', 'quarter']].head())\n    \n    # Save or process the data as needed for further analysis\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## Lag Features ##\nCreate lag features (e.g., values from previous time steps) to incorporate autocorrelations.\n","metadata":{}},{"cell_type":"code","source":"# Define the number of lags to create\nnum_lags = 5  # Adjust as needed to create more or fewer lags\n\n# Iterate through each partition to add lag features\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n\n    # Loop through each feature column to create lagged features\n    for column in data.columns:\n        if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n            # Create lagged columns for each feature\n            for lag in range(1, num_lags + 1):\n                data[f'{column}_lag_{lag}'] = data[column].shift(lag)\n    \n    # Drop rows with NaN values created by shifting (optional)\n    data = data.dropna().reset_index(drop=True)\n\n    # Print or save modified data to verify new lag features\n    print(data.head())\n\n    # Save or process the data as needed for further analysis\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## Rolling Window Features: ##\nUse rolling statistics (mean, median, standard deviation) for time windows (e.g., 7 days, 30 days).\n\n### Key Insights: ###\n\n* Time-based features can help models like XGBoost or deep learning models that do not explicitly handle seasonality or time-dependent features.\n* Lag and rolling features can help AR models like ARIMA or machine learning models capture temporal dependencies.","metadata":{}},{"cell_type":"code","source":"# Define the rolling windows\nwindows = [7, 30]  # Adjust window sizes as needed\n\n# Iterate through each partition to add rolling statistics\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n\n    # Loop through each feature column to create rolling statistics\n    for column in data.columns:\n        if column not in [\"date_id\", \"time_id\", \"symbol_id\"]:\n            # Create rolling statistics for each window\n            for window in windows:\n                data[f'{column}_rolling_mean_{window}'] = data[column].rolling(window=window).mean()\n                data[f'{column}_rolling_median_{window}'] = data[column].rolling(window=window).median()\n                data[f'{column}_rolling_std_{window}'] = data[column].rolling(window=window).std()\n    \n    # Drop rows with NaN values created by rolling (optional)\n    data = data.dropna().reset_index(drop=True)\n\n    # Print or save modified data to verify new rolling features\n    print(data.head())\n\n    # Save or process the data as needed for further analysis\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Correlation Analysis #\n**Goal:** Explore relationships between variables (if available) to help improve model predictions.\n\n## Correlation with Target Variable ##\nUse correlation matrices to identify how features correlate with the target variable (e.g., responder_6).\n\n### Key Insights: ###\n\n* Features that are highly correlated with the target variable might be useful predictors.\n* If multiple features are correlated with each other, consider dimensionality reduction techniques like PCA or feature selection to prevent multicollinearity.\n","metadata":{}},{"cell_type":"code","source":"# Iterate through each partition to calculate and visualize correlations\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n\n    # Drop non-numeric columns and rows with NaN values for correlation calculation\n    numeric_data = data.drop(columns=[\"date_id\", \"time_id\", \"symbol_id\"]).dropna()\n\n    # Calculate correlation matrix\n    correlation_matrix = numeric_data.corr()\n\n    # Extract correlations with responder_6\n    responder_corr = correlation_matrix['responder_6'].sort_values(ascending=False)\n    \n    # Display the top 10 features with highest correlation with responder_6\n    print(f\"\\nTop 10 correlations with responder_6 in Partition {i+1}:\\n\")\n    print(responder_corr.head(10))\n\n    # Plot heatmap of correlations with responder_6\n    plt.figure(figsize=(8, 12))\n    sns.heatmap(responder_corr.to_frame(), annot=True, cmap=\"coolwarm\", vmin=-1, vmax=1)\n    plt.title(f'Correlation of Features with responder_6 - Partition {i+1}')\n    plt.show()\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Cross-Correlation ##\nIf you have multiple time series (e.g., multiple responders), cross-correlation can reveal lagged relationships between the series.\n\n### Key Insights: ##\n* If there are cross-correlations between time series, multi-output models or multi-variate mod\n","metadata":{}},{"cell_type":"code","source":"\n# Define the range of lags to check for cross-correlation\nlags = range(-10, 11)  # Check lagged correlations from -10 to +10 days\n\n# Iterate through each partition to calculate and visualize cross-correlations\nfor i in range(10):\n    data = pd.read_parquet(files[i])\n\n    # Drop rows with NaN values for cleaner cross-correlation calculation\n    data = data.dropna()\n\n    # Loop through responder columns other than responder_6\n    for responder in [col for col in data.columns if col.startswith(\"responder_\") and col != \"responder_6\"]:\n        cross_corrs = []  # Store cross-correlation values for each lag\n\n        # Calculate cross-correlation at each lag\n        for lag in lags:\n            shifted_responder = data[responder].shift(lag)  # Shift the other responder\n            cross_corr = data['responder_6'].corr(shifted_responder)\n            cross_corrs.append(cross_corr)\n\n        # Plot the cross-correlation for each lag\n        plt.figure(figsize=(10, 5))\n        plt.plot(lags, cross_corrs, marker='o')\n        plt.title(f'Cross-Correlation between responder_6 and {responder} (Partition {i+1})')\n        plt.xlabel('Lag (days)')\n        plt.ylabel('Cross-Correlation')\n        plt.axhline(0, color='gray', linestyle='--')\n        plt.show()\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"","metadata":{}}]}