Sharing the code I use to debug my submissions. I used to run this locally but have since dropped it in a notebook to run before submitting to ensure I test in notebook conditions (performance is much worse ~4x slower for online training in notebook than on my machine). Just remember to comment out the code before submission!
\nThis will log out the average time to make predictions as well as daily R2 scores to ensure your model actually returns values and doesn't throw etc. It's more useful than the test set as it will span multiple time steps and days and pick up bugs around missing symbols or null handling.
\nJust tweak the date filter to more or less days to suit your needs
\n () -> :\n sum_square_residuals = (\n ys.get_column()\n * ((ys.get_column() - y_preds.get_column()).())\n ).()\n total_sum_squares = (\n ys.get_column() * (ys.get_column().())\n ).()\n - (sum_square_residuals / total_sum_squares)\n\n () -> :\n tqdm.auto tqdm\n ()\n\n r2 = \n :\n \n test_data = pl.scan_parquet(\n \n ).with_row_index()\n test_data = test_data.(\n pl.col() > from_day - \n ) \n last_day = test_data.last().select().collect().item()\n first_day = (\n test_data.first().select().collect().item() + \n )\n total_days = last_day - first_day\n\n tqdm(total=total_days, desc=, position=) day_pbar:\n day (first_day, first_day + total_days):\n day < from_day:\n day_pbar.update()\n \n\n preds = pl.DataFrame([])\n\n day_start = time.perf_counter()\n next_step = test_data.(pl.col() == day).with_columns(\n pl.col().alias()\n )\n lags = (\n test_data.(pl.col() == day - )\n .with_columns(pl.col() + )\n .with_columns(pl.col().alias())\n .with_columns(pl.col().alias())\n .collect()\n )\n time_partitions = next_step.collect().partition_by(\n , maintain_order=\n )\n t0 = \n tqdm(\n total=(time_partitions), desc=, leave=\n ) time_pbar:\n timestep_data time_partitions:\n timestep_preds = predict(\n timestep_data, lags t0 \n )\n t0 = \n preds = pl.concat([preds, timestep_preds])\n time_pbar.update()\n\n elapsed = time.perf_counter() - day_start\n\n ys = next_step.select(\n pl.col().cast(pl.Int64),\n pl.col().cast(pl.Float32),\n pl.col().cast(pl.Float32),\n ).collect()\n\n r2 = calc_r2(ys, preds)\n day_pbar.update()\n day_pbar.set_postfix(\n {\n : ,\n : day,\n : day - first_day,\n : ,\n }\n )\n\n Exception e:\n ()\n \n\nsimulate_competition()\nSharing the code I use to debug my submissions. I used to run this locally but have since dropped it in a notebook to run before submitting to ensure I test in notebook conditions (performance is much worse ~4x slower for online training in notebook than on my machine). Just remember to comment out the code before submission!
\nThis will log out the average time to make predictions as well as daily R2 scores to ensure your model actually returns values and doesn't throw etc. It's more useful than the test set as it will span multiple time steps and days and pick up bugs around missing symbols or null handling.
\nJust tweak the date filter to more or less days to suit your needs
\n () -> :\n sum_square_residuals = (\n ys.get_column()\n * ((ys.get_column() - y_preds.get_column()).())\n ).()\n total_sum_squares = (\n ys.get_column() * (ys.get_column().())\n ).()\n - (sum_square_residuals / total_sum_squares)\n\n () -> :\n tqdm.auto tqdm\n ()\n\n r2 = \n :\n \n test_data = pl.scan_parquet(\n \n ).with_row_index()\n test_data = test_data.(\n pl.col() > from_day - \n ) \n last_day = test_data.last().select().collect().item()\n first_day = (\n test_data.first().select().collect().item() + \n )\n total_days = last_day - first_day\n\n tqdm(total=total_days, desc=, position=) day_pbar:\n day (first_day, first_day + total_days):\n day < from_day:\n day_pbar.update()\n \n\n preds = pl.DataFrame([])\n\n day_start = time.perf_counter()\n next_step = test_data.(pl.col() == day).with_columns(\n pl.col().alias()\n )\n lags = (\n test_data.(pl.col() == day - )\n .with_columns(pl.col() + )\n .with_columns(pl.col().alias())\n .with_columns(pl.col().alias())\n .collect()\n )\n time_partitions = next_step.collect().partition_by(\n , maintain_order=\n )\n t0 = \n tqdm(\n total=(time_partitions), desc=, leave=\n ) time_pbar:\n timestep_data time_partitions:\n timestep_preds = predict(\n timestep_data, lags t0 \n )\n t0 = \n preds = pl.concat([preds, timestep_preds])\n time_pbar.update()\n\n elapsed = time.perf_counter() - day_start\n\n ys = next_step.select(\n pl.col().cast(pl.Int64),\n pl.col().cast(pl.Float32),\n pl.col().cast(pl.Float32),\n ).collect()\n\n r2 = calc_r2(ys, preds)\n day_pbar.update()\n day_pbar.set_postfix(\n {\n : ,\n : day,\n : day - first_day,\n : ,\n }\n )\n\n Exception e:\n ()\n \n\nsimulate_competition()\nThank you for sharing Michael! I'm going to try this before the end of the competition.
", "rawMarkdown": "Thank you for sharing Michael! I'm going to try this before the end of the competition.", "votes": null }, { "id": "3091326", "postDate": "01/08/2025 09:57:27", "content": "I noticed a few people use other lagged responders too. So you’ll need to tweak it to add those if you use them. I’ve only experimented with using responder_6 for online training so it’s the only one I’m passing through
", "rawMarkdown": "I noticed a few people use other lagged responders too. So you’ll need to tweak it to add those if you use them. I’ve only experimented with using responder_6 for online training so it’s the only one I’m passing through", "votes": null } ], "comments": [ { "id": 3090683, "author_name": "barbosajaf", "author_url": "", "post_date": "01/07/2025 14:49:25", "content": "Thank you for sharing Michael! I'm going to try this before the end of the competition.
", "votes": null, "replies": [] }, { "id": 3091326, "author_name": "michaeltimbs", "author_url": "", "post_date": "01/08/2025 09:57:27", "content": "I noticed a few people use other lagged responders too. So you’ll need to tweak it to add those if you use them. I’ve only experimented with using responder_6 for online training so it’s the only one I’m passing through
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3089474": "Sharing the code I use to debug my submissions. I used to run this locally but have since dropped it in a notebook to run before submitting to ensure I test in notebook conditions (performance is much worse ~4x slower for online training in notebook than on my machine). Just remember to comment out the code before submission!\n\nThis will log out the average time to make predictions as well as daily R2 scores to ensure your model actually returns values and doesn't throw etc. It's more useful than the test set as it will span multiple time steps and days and pick up bugs around missing symbols or null handling.\n\nJust tweak the date filter to more or less days to suit your needs\n\n```python\ndef calc_r2(ys: pl.DataFrame, y_preds: pl.DataFrame) -> float:\n sum_square_residuals = (\n ys.get_column(\"weight\")\n * ((ys.get_column(\"responder_6\") - y_preds.get_column(\"responder_6\")).pow(2))\n ).sum()\n total_sum_squares = (\n ys.get_column(\"weight\") * (ys.get_column(\"responder_6\").pow(2))\n ).sum()\n return 1 - (sum_square_residuals / total_sum_squares)\n \ndef simulate_competition(from_day: int = 1) -> None:\n from tqdm.auto import tqdm\n print(\"Starting gateway simulation\")\n\n r2 = 0.00\n try:\n # Initialise dataframe to hold predictions\n test_data = pl.scan_parquet(\n \"/kaggle/input/jane-street-real-time-market-data-forecasting/train.parquet/partition_id=9/part-0.parquet\"\n ).with_row_index()\n test_data = test_data.filter(\n pl.col(\"date_id\") > from_day - 1\n ) \n last_day = test_data.last().select(\"date_id\").collect().item()\n first_day = (\n test_data.first().select(\"date_id\").collect().item() + 1\n )\n total_days = last_day - first_day\n\n with tqdm(total=total_days, desc=\"Processing days\", position=0) as day_pbar:\n for day in range(first_day, first_day + total_days):\n if day < from_day:\n day_pbar.update(1)\n continue\n\n preds = pl.DataFrame([])\n\n day_start = time.perf_counter()\n next_step = test_data.filter(pl.col(\"date_id\") == day).with_columns(\n pl.col(\"index\").alias(\"row_id\")\n )\n lags = (\n test_data.filter(pl.col(\"date_id\") == day - 1)\n .with_columns(pl.col(\"date_id\") + 1)\n .with_columns(pl.col(\"responder_6\").alias(\"responder_6_lag_1\"))\n .with_columns(pl.col(\"index\").alias(\"row_id\"))\n .collect()\n )\n time_partitions = next_step.collect().partition_by(\n \"time_id\", maintain_order=True\n )\n t0 = True\n with tqdm(\n total=len(time_partitions), desc=f\"Day {day} timesteps\", leave=False\n ) as time_pbar:\n for timestep_data in time_partitions:\n timestep_preds = predict(\n timestep_data, lags if t0 is True else None\n )\n t0 = False\n preds = pl.concat([preds, timestep_preds])\n time_pbar.update(1)\n\n elapsed = time.perf_counter() - day_start\n\n ys = next_step.select(\n pl.col(\"row_id\").cast(pl.Int64),\n pl.col(\"weight\").cast(pl.Float32),\n pl.col(\"responder_6\").cast(pl.Float32),\n ).collect()\n\n r2 = calc_r2(ys, preds)\n day_pbar.update(1)\n day_pbar.set_postfix(\n {\n \"t\": f\"{elapsed:.2f}s\",\n \"date_id\": day,\n \"day\": day - first_day,\n \"R²\": f\"{r2:.4f}\",\n }\n )\n\n except Exception as e:\n print(f\"Gateway simulation failed: {str(e)}\")\n raise\n\nsimulate_competition(1680)\n```", "3090683": "Thank you for sharing Michael! I'm going to try this before the end of the competition.", "3091326": "I noticed a few people use other lagged responders too. So you’ll need to tweak it to add those if you use them. I’ve only experimented with using responder_6 for online training so it’s the only one I’m passing through" }, "source": "meta" }