{ "id": 543726, "title": "Unable to submit via given api", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/543726", "author_name": "Dear_Luna", "post_date": "2024-11-01T05:43:05.344000", "votes": 0, "comment_count": 8, "views": 0, "content": "

I submitted according to https://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission,

\n

`inference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)

\n

if os.getenv('KAGGLE_IS_COMPETITION_RERUN'):
\n inference_server.serve()
\nelse:
\n inference_server.run_local_gateway(
\n (
\n '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',
\n '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',
\n )
\n )`

\n

But an error was reported
\n`GatewayRuntimeError Traceback (most recent call last)
\nCell In[27], line 52
\n 50 inference_server.serve()
\n 51 else:
\n---> 52 inference_server.run_local_gateway(
\n 53 (
\n 54 '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',
\n 55 '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',
\n 56 )
\n 57 )

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:141, in InferenceServer.run_local_gateway(self, data_paths)
\n 139 self.gateway.run()
\n 140 except Exception as err:
\n--> 141 raise err from None
\n 142 finally:
\n 143 self.server.stop(0)

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:139, in InferenceServer.run_local_gateway(self, data_paths)
\n 137 try:
\n 138 self.gateway = self._get_gateway_for_test(data_paths)
\n--> 139 self.gateway.run()
\n 140 except Exception as err:
\n 141 raise err from None

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:98, in Gateway.run(self)
\n 95 self.write_result(error)
\n 96 elif error:
\n 97 # For local testing
\n---> 98 raise error

\n

GatewayRuntimeError: (, 'Traceback (most recent call last):\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py\", line 77, in run\\n self.write_submission(predictions)\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py\", line 159, in write_submission\\n predictions.to_parquet(\\'submission.parquet\\', index=False)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/util/_decorators.py\", line 333, in wrapper\\n return func(*args, **kwargs)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/core/frame.py\", line 3113, in to_parquet\\n return to_parquet(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 480, in to_parquet\\n impl.write(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 198, in write\\n path_or_handle, handles, filesystem = _get_path_or_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 140, in _get_path_or_handle\\n handles = get_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/common.py\", line 882, in get_handle\\n handle = open(handle, ioargs.mode)\\nOSError: [Errno 30] Read-only file system: \\'submission.parquet\\'\\n')`

\n

This means it's trying to write to a read-only folder, but my predicate function doesn't have any code trying to write，What should I do, can anyone help me?

", "messages": [ { "id": 3033852, "postDate": "2024-11-01T15:12:51.403Z", "content": "

Even when I run the official demo, I will get a similar error: trying to write a file to a read-only system.

", "rawMarkdown": "Even when I run the official demo, I will get a similar error: trying to write a file to a read-only system." }, { "id": 3033582, "postDate": "2024-11-01T10:25:38.607Z", "content": "

Share your prediction function that where error occurred

", "rawMarkdown": "Share your prediction function that where error occurred", "replies": [ { "id": 3033786, "postDate": "2024-11-01T14:11:24.423Z", "content": "

here

\n

`def predict(test: pl.DataFrame, _) -> pd.DataFrame:
\n print(1)
\n \"\"\"Make a prediction for responder_6 based on the predictions of other responders.\"\"\"

\n
\ntest_df = test.to_pandas()\n\n\n responder  responders:\n     responder != :\n        predictor = predictors[responder]\n        test_df[responder] = predictor.predict(test_df)\n\n\npredictor_6 = predictors[]\nresponder_6_predictions = predictor_6.predict(test_df)\n\n\npredictions = pd.DataFrame({\n    : test_df[],\n    : responder_6_predictions\n})\n\n\n predictions.columns.tolist() == [, ], \n (predictions, pd.DataFrame), \n()\n\n predictions`\n
\n

I haven't used lags in my training process because I don't know how to use it yet. I first supplement the corresponding responder through other predictors, and then predict responder6 on the data set with other responders.

", "rawMarkdown": "here\n\n`def predict(test: pl.DataFrame, _) -> pd.DataFrame:\n print(1)\n \"\"\"Make a prediction for responder_6 based on the predictions of other responders.\"\"\"\n \n # Convert to Pandas DataFrame for compatibility\n test_df = test.to_pandas()\n \n # Predict for each responder except responder_6\n for responder in responders:\n if responder != 'responder_6':\n predictor = predictors[responder]\n test_df[responder] = predictor.predict(test_df)\n \n # Now predict responder_6 using the other responders' predictions\n predictor_6 = predictors['responder_6']\n responder_6_predictions = predictor_6.predict(test_df)\n\n # Create the output DataFrame with 'row_id' and 'responder_6' predictions\n predictions = pd.DataFrame({\n 'row_id': test_df['row_id'],\n 'responder_6': responder_6_predictions\n })\n \n # Ensure predictions DataFrame has only ['row_id', 'responder_6'] columns\n assert predictions.columns.tolist() == ['row_id', 'responder_6'], \"Prediction DataFrame does not have expected columns\"\n assert isinstance(predictions, pd.DataFrame), \"Predictions must be a Pandas DataFrame\"\n print(2)\n\n return predictions`\n\nI haven't used lags in my training process because I don't know how to use it yet. I first supplement the corresponding responder through other predictors, and then predict responder6 on the data set with other responders.", "replies": [ { "id": 3034049, "postDate": "2024-11-01T18:05:50.370Z", "content": "

Try this

\n

def predict_10(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:
\n global 10_lags
\n if lags is not None:
\n 10_lags = lags

\n
    _10_predictions = test.(\n        ,\n        pl.lit().(),\n    )\n    symbol_ids = test.().to_numpy()[:, ]\n\n      lags  None:\n        lags = lags.group_by([, ], maintain_order=True).last() \n        test = test.(lags, =[, ],  how=)\n    :\n        test = test.with_columns(\n            ( pl.lit().(f)  = np.zeros((test.shape[],))\n     i, = model.predict(test[CONFIG.feature_cols].to_pandas()) / len(_10_models)\n    print(f, _10_preds.shape)\n\n    _10_predictions = \\\n    test.().\\\n    with_columns(\n        pl.Series(\n            name   = , \n            values = np.clip(_10_preds, a_min = , a_max = ),\n            dtype  = pl.Float64,\n        )\n    )\n
\n

# The predict function must return a DataFrame

\n

assert isinstance(_10_predictions, pl.DataFrame | pd.DataFrame)

\n

# with columns 'row_id', 'responer_6'

\n

assert list(_10_predictions.columns) == ['row_id', 'responder_6']

\n

# and as many rows as the test data.

\n

assert len(_10_predictions) == len(test)

\n
     _10_predictions\n
", "rawMarkdown": "Try this\n\n def predict_10(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:\n global _10_lags_\n if lags is not None:\n _10_lags_ = lags\n\n _10_predictions = test.select(\n 'row_id',\n pl.lit(0.0).alias('responder_6'),\n )\n symbol_ids = test.select('symbol_id').to_numpy()[:, 0]\n\n if not lags is None:\n lags = lags.group_by([\"date_id\", \"symbol_id\"], maintain_order=True).last() # pick up last record of previous date\n test = test.join(lags, on=[\"date_id\", \"symbol_id\"], how=\"left\")\n else:\n test = test.with_columns(\n ( pl.lit(0.0).alias(f'responder_{idx}_lag_1') for idx in range(9) )\n )\n\n _10_preds = np.zeros((test.shape[0],))\n for i, model in enumerate(tqdm(_10_models)):\n _10_preds += model.predict(test[CONFIG.feature_cols].to_pandas()) / len(_10_models)\n print(f\"predict> preds.shape =\", _10_preds.shape)\n\n _10_predictions = \\\n test.select('row_id').\\\n with_columns(\n pl.Series(\n name = 'responder_6', \n values = np.clip(_10_preds, a_min = -5, a_max = 5),\n dtype = pl.Float64,\n )\n )\n\n# # The predict function must return a DataFrame\n# assert isinstance(_10_predictions, pl.DataFrame | pd.DataFrame)\n# # with columns 'row_id', 'responer_6'\n# assert list(_10_predictions.columns) == ['row_id', 'responder_6']\n# # and as many rows as the test data.\n# assert len(_10_predictions) == len(test)\n\n return _10_predictions" }, { "id": 3034329, "postDate": "2024-11-02T02:51:08.647Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3034333, "postDate": "2024-11-02T02:56:00.003Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3034335, "postDate": "2024-11-02T02:56:55.560Z", "content": "

*First of all thank you for your help!

\n

I tried it and modified it according to my own prediction logic: predict responders0-5 and 7-8 first, then predict responder6, and added debugging information to see whether the model predicts smoothly:*

\n

`
\ndef predict(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:
\nglobal lags_10
\nif lags is not None:
\nlags_10 = lags

\n

\n

print(\"Initial test columns:\", test.columns)

\n

print(\"First few rows of test data:\\n\", test.head())

\n

for responder in range(9):
\n if responder != 6: # 跳过 responder_6
\n predictor = models[f\"responder_{responder}\"]

\n
    \n    \n    \n\n    test = test.with_columns(\n        pl.Series(name=f'responder_{responder}', values=predictor.predict(test.to_pandas()[feature_cols]))\n    )\n\n\n    \n
\n

predictor_6 = models[\"responder_6\"]
\nfeature_cols_with_responders = feature_cols + [f'responder_{i}' for i in range(9) if i != 6]

\n

print(\"Columns used for responder_6 prediction:\", feature_cols_with_responders)
\nprint(\"First few rows of test data for responder_6 prediction:\\n\", test.to_pandas()[feature_cols_with_responders].head())

\n

responder_6_predictions = predictor_6.predict(test.to_pandas()[feature_cols_with_responders])

\n

predictions = test.select(\"row_id\").with_columns(
\n pl.Series(
\n name=\"responder_6\",
\n values=np.clip(responder_6_predictions, a_min=-5, a_max=5),
\n dtype=pl.Float64
\n )
\n)

\n

assert isinstance(predictions, (pl.DataFrame, pd.DataFrame))
\nassert list(predictions.columns) == [\"row_id\", \"responder_6\"]
\nassert len(predictions) == len(test)

\n

print(\"Final predictions columns:\", predictions.columns)
\nprint(\"First few rows of predictions:\\n\", predictions.head())

\n

return predictions

\n

`
\nFinal predictions columns: ['row_id', 'responder_6']
\nFirst few rows of predictions:
\nshape: (5, 2)
\n┌────────┬─────────────┐
\n│ row_id ┆ responder_6 │
\n│ --- ┆ --- │
\n│ i64 ┆ f64 │
\n╞════════╪═════════════╡
\n│ 0 ┆ -0.165136 │
\n│ 1 ┆ -0.165721 │
\n│ 2 ┆ -0.166134 │
\n│ 3 ┆ -0.166615 │
\n│ 4 ┆ -0.167089 │
\n└────────┴─────────────┘
\nBut it still reports an error:

\n

GatewayRuntimeError Traceback (most recent call last)
\nCell In[44], line 75
\n73 inference_server.serve()
\n74 else:
\n---> 75 inference_server.run_local_gateway(
\n76 (
\n77 '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',
\n78 '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',
\n79 )
\n80 )

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:141, in InferenceServer.run_local_gateway(self, data_paths)
\n139 self.gateway.run()
\n140 except Exception as err:
\n--> 141 raise err from None
\n142 finally:
\n143 self.server.stop(0)

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:139, in InferenceServer.run_local_gateway(self, data_paths)
\n137 try:
\n138 self.gateway = self._get_gateway_for_test(data_paths)
\n--> 139 self.gateway.run()
\n140 except Exception as err:
\n141 raise err from None

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:98, in Gateway.run(self)
\n95 self.write_result(error)
\n96 elif error:
\n97 # For local testing
\n---> 98 raise error

\n

GatewayRuntimeError: (, 'Traceback (most recent call last):\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py\", line 77, in run\\n self.write_submission(predictions)\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py\", line 161, in write_submission\\n pl.DataFrame(predictions).write_parquet(\\'submission.parquet\\')\\n File \"/opt/conda/lib/python3.10/site-packages/polars/dataframe/frame.py\", line 3837, in write_parquet\\n self._df.write_parquet(\\nOSError: Read-only file system (os error 30)\\n')

\n

Still trying to write to a file on a read-only system, but no line in my prediction function attempts to write to a file

", "rawMarkdown": "*First of all thank you for your help!\n\nI tried it and modified it according to my own prediction logic: predict responders0-5 and 7-8 first, then predict responder6, and added debugging information to see whether the model predicts smoothly:*\n\n`\ndef predict(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:\nglobal lags_10\nif lags is not None:\nlags_10 = lags\n\n# \n#print(\"Initial test columns:\", test.columns)\n#print(\"First few rows of test data:\\n\", test.head())\n\n\nfor responder in range(9):\n if responder != 6: # 跳过 responder_6\n predictor = models[f\"responder_{responder}\"]\n\n #print(f\"Using predictor for responder_{responder}\")\n #print(\"Columns used for prediction:\", feature_cols)\n #print(\"First few rows of test data for prediction:\\n\", test.to_pandas()[feature_cols].head())\n\n test = test.with_columns(\n pl.Series(name=f'responder_{responder}', values=predictor.predict(test.to_pandas()[feature_cols]))\n )\n\n\n #print(f\"After adding responder_{responder} predictions, test columns:\", test.columns)\n\n\npredictor_6 = models[\"responder_6\"]\nfeature_cols_with_responders = feature_cols + [f'responder_{i}' for i in range(9) if i != 6]\n\n\nprint(\"Columns used for responder_6 prediction:\", feature_cols_with_responders)\nprint(\"First few rows of test data for responder_6 prediction:\\n\", test.to_pandas()[feature_cols_with_responders].head())\n\nresponder_6_predictions = predictor_6.predict(test.to_pandas()[feature_cols_with_responders])\n\npredictions = test.select(\"row_id\").with_columns(\n pl.Series(\n name=\"responder_6\", \n values=np.clip(responder_6_predictions, a_min=-5, a_max=5), \n dtype=pl.Float64\n )\n)\n\n\nassert isinstance(predictions, (pl.DataFrame, pd.DataFrame))\nassert list(predictions.columns) == [\"row_id\", \"responder_6\"]\nassert len(predictions) == len(test)\n\n\nprint(\"Final predictions columns:\", predictions.columns)\nprint(\"First few rows of predictions:\\n\", predictions.head())\n\nreturn predictions\n\n`\nFinal predictions columns: ['row_id', 'responder_6']\nFirst few rows of predictions:\nshape: (5, 2)\n┌────────┬─────────────┐\n│ row_id ┆ responder_6 │\n│ --- ┆ --- │\n│ i64 ┆ f64 │\n╞════════╪═════════════╡\n│ 0 ┆ -0.165136 │\n│ 1 ┆ -0.165721 │\n│ 2 ┆ -0.166134 │\n│ 3 ┆ -0.166615 │\n│ 4 ┆ -0.167089 │\n└────────┴─────────────┘\n*But it still reports an error:*\n\nGatewayRuntimeError Traceback (most recent call last)\nCell In[44], line 75\n73 inference_server.serve()\n74 else:\n---> 75 inference_server.run_local_gateway(\n76 (\n77 '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n78 '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n79 )\n80 )\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:141, in InferenceServer.run_local_gateway(self, data_paths)\n139 self.gateway.run()\n140 except Exception as err:\n--> 141 raise err from None\n142 finally:\n143 self.server.stop(0)\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:139, in InferenceServer.run_local_gateway(self, data_paths)\n137 try:\n138 self.gateway = self._get_gateway_for_test(data_paths)\n--> 139 self.gateway.run()\n140 except Exception as err:\n141 raise err from None\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:98, in Gateway.run(self)\n95 self.write_result(error)\n96 elif error:\n97 # For local testing\n---> 98 raise error\n\nGatewayRuntimeError: (, 'Traceback (most recent call last):\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py\", line 77, in run\\n self.write_submission(predictions)\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py\", line 161, in write_submission\\n pl.DataFrame(predictions).write_parquet(\\'submission.parquet\\')\\n File \"/opt/conda/lib/python3.10/site-packages/polars/dataframe/frame.py\", line 3837, in write_parquet\\n self._df.write_parquet(\\nOSError: Read-only file system (os error 30)\\n')\n\n*Still trying to write to a file on a read-only system, but no line in my prediction function attempts to write to a file*" }, { "id": 3038851, "postDate": "2024-11-07T12:40:47.573Z", "content": "

Hi, have you found an answer to this? Facing same issue. Thanks

", "rawMarkdown": "Hi, have you found an answer to this? Facing same issue. Thanks" } ] } ] }, { "id": 3033433, "postDate": "2024-11-01T05:43:05.343Z", "content": "

I submitted according to https://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission,

\n

`inference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)

\n

if os.getenv('KAGGLE_IS_COMPETITION_RERUN'):
\n inference_server.serve()
\nelse:
\n inference_server.run_local_gateway(
\n (
\n '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',
\n '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',
\n )
\n )`

\n

But an error was reported
\n`GatewayRuntimeError Traceback (most recent call last)
\nCell In[27], line 52
\n 50 inference_server.serve()
\n 51 else:
\n---> 52 inference_server.run_local_gateway(
\n 53 (
\n 54 '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',
\n 55 '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',
\n 56 )
\n 57 )

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:141, in InferenceServer.run_local_gateway(self, data_paths)
\n 139 self.gateway.run()
\n 140 except Exception as err:
\n--> 141 raise err from None
\n 142 finally:
\n 143 self.server.stop(0)

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:139, in InferenceServer.run_local_gateway(self, data_paths)
\n 137 try:
\n 138 self.gateway = self._get_gateway_for_test(data_paths)
\n--> 139 self.gateway.run()
\n 140 except Exception as err:
\n 141 raise err from None

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:98, in Gateway.run(self)
\n 95 self.write_result(error)
\n 96 elif error:
\n 97 # For local testing
\n---> 98 raise error

\n

GatewayRuntimeError: (, 'Traceback (most recent call last):\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py\", line 77, in run\\n self.write_submission(predictions)\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py\", line 159, in write_submission\\n predictions.to_parquet(\\'submission.parquet\\', index=False)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/util/_decorators.py\", line 333, in wrapper\\n return func(*args, **kwargs)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/core/frame.py\", line 3113, in to_parquet\\n return to_parquet(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 480, in to_parquet\\n impl.write(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 198, in write\\n path_or_handle, handles, filesystem = _get_path_or_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 140, in _get_path_or_handle\\n handles = get_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/common.py\", line 882, in get_handle\\n handle = open(handle, ioargs.mode)\\nOSError: [Errno 30] Read-only file system: \\'submission.parquet\\'\\n')`

\n

This means it's trying to write to a read-only folder, but my predicate function doesn't have any code trying to write，What should I do, can anyone help me?

", "rawMarkdown": "I submitted according to [https://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission](url),\n\n`inference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)\n\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n inference_server.serve()\nelse:\n inference_server.run_local_gateway(\n (\n '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n )\n )`\n\n\n\nBut an error was reported\n`GatewayRuntimeError Traceback (most recent call last)\nCell In[27], line 52\n 50 inference_server.serve()\n 51 else:\n---> 52 inference_server.run_local_gateway(\n 53 (\n 54 '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n 55 '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n 56 )\n 57 )\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:141, in InferenceServer.run_local_gateway(self, data_paths)\n 139 self.gateway.run()\n 140 except Exception as err:\n--> 141 raise err from None\n 142 finally:\n 143 self.server.stop(0)\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:139, in InferenceServer.run_local_gateway(self, data_paths)\n 137 try:\n 138 self.gateway = self._get_gateway_for_test(data_paths)\n--> 139 self.gateway.run()\n 140 except Exception as err:\n 141 raise err from None\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:98, in Gateway.run(self)\n 95 self.write_result(error)\n 96 elif error:\n 97 # For local testing\n---> 98 raise error\n\nGatewayRuntimeError: (, 'Traceback (most recent call last):\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py\", line 77, in run\\n self.write_submission(predictions)\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py\", line 159, in write_submission\\n predictions.to_parquet(\\'submission.parquet\\', index=False)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/util/_decorators.py\", line 333, in wrapper\\n return func(*args, **kwargs)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/core/frame.py\", line 3113, in to_parquet\\n return to_parquet(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 480, in to_parquet\\n impl.write(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 198, in write\\n path_or_handle, handles, filesystem = _get_path_or_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 140, in _get_path_or_handle\\n handles = get_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/common.py\", line 882, in get_handle\\n handle = open(handle, ioargs.mode)\\nOSError: [Errno 30] Read-only file system: \\'submission.parquet\\'\\n')`\n\nThis means it's trying to write to a read-only folder, but my predicate function doesn't have any code trying to write，What should I do, can anyone help me?" } ], "comments": [ { "id": 3033852, "author_name": "Dear_Luna", "author_url": "", "post_date": "2024-11-01T15:12:51.403000", "content": "

Even when I run the official demo, I will get a similar error: trying to write a file to a read-only system.

", "votes": 0, "replies": [] }, { "id": 3033582, "author_name": "Farhan Kardan", "author_url": "", "post_date": "2024-11-01T10:25:38.607000", "content": "

Share your prediction function that where error occurred

", "votes": 0, "replies": [ { "id": 3033786, "author_name": "Dear_Luna", "author_url": "", "post_date": "2024-11-01T14:11:24.423000", "content": "

here

\n

`def predict(test: pl.DataFrame, _) -> pd.DataFrame:
\n print(1)
\n \"\"\"Make a prediction for responder_6 based on the predictions of other responders.\"\"\"

\n
\ntest_df = test.to_pandas()\n\n\n responder  responders:\n     responder != :\n        predictor = predictors[responder]\n        test_df[responder] = predictor.predict(test_df)\n\n\npredictor_6 = predictors[]\nresponder_6_predictions = predictor_6.predict(test_df)\n\n\npredictions = pd.DataFrame({\n    : test_df[],\n    : responder_6_predictions\n})\n\n\n predictions.columns.tolist() == [, ], \n (predictions, pd.DataFrame), \n()\n\n predictions`\n
\n

I haven't used lags in my training process because I don't know how to use it yet. I first supplement the corresponding responder through other predictors, and then predict responder6 on the data set with other responders.

", "votes": 0, "replies": [ { "id": 3034049, "author_name": "Farhan Kardan", "author_url": "", "post_date": "2024-11-01T18:05:50.370000", "content": "

Try this

\n

def predict_10(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:
\n global 10_lags
\n if lags is not None:
\n 10_lags = lags

\n
    _10_predictions = test.(\n        ,\n        pl.lit().(),\n    )\n    symbol_ids = test.().to_numpy()[:, ]\n\n      lags  None:\n        lags = lags.group_by([, ], maintain_order=True).last() \n        test = test.(lags, =[, ],  how=)\n    :\n        test = test.with_columns(\n            ( pl.lit().(f)  = np.zeros((test.shape[],))\n     i, = model.predict(test[CONFIG.feature_cols].to_pandas()) / len(_10_models)\n    print(f, _10_preds.shape)\n\n    _10_predictions = \\\n    test.().\\\n    with_columns(\n        pl.Series(\n            name   = , \n            values = np.clip(_10_preds, a_min = , a_max = ),\n            dtype  = pl.Float64,\n        )\n    )\n
\n

# The predict function must return a DataFrame

\n

assert isinstance(_10_predictions, pl.DataFrame | pd.DataFrame)

\n

# with columns 'row_id', 'responer_6'

\n

assert list(_10_predictions.columns) == ['row_id', 'responder_6']

\n

# and as many rows as the test data.

\n

assert len(_10_predictions) == len(test)

\n
     _10_predictions\n
", "votes": 0, "replies": [] }, { "id": 3034329, "author_name": "", "author_url": "", "post_date": "2024-11-02T02:51:08.647000", "content": "", "votes": 0, "replies": [] }, { "id": 3034333, "author_name": "", "author_url": "", "post_date": "2024-11-02T02:56:00.003000", "content": "", "votes": 0, "replies": [] }, { "id": 3034335, "author_name": "Dear_Luna", "author_url": "", "post_date": "2024-11-02T02:56:55.560000", "content": "

*First of all thank you for your help!

\n

I tried it and modified it according to my own prediction logic: predict responders0-5 and 7-8 first, then predict responder6, and added debugging information to see whether the model predicts smoothly:*

\n

`
\ndef predict(test: pl.DataFrame, lags: pl.DataFrame | None) -> pl.DataFrame | pd.DataFrame:
\nglobal lags_10
\nif lags is not None:
\nlags_10 = lags

\n

\n

print(\"Initial test columns:\", test.columns)

\n

print(\"First few rows of test data:\\n\", test.head())

\n

for responder in range(9):
\n if responder != 6: # 跳过 responder_6
\n predictor = models[f\"responder_{responder}\"]

\n
    \n    \n    \n\n    test = test.with_columns(\n        pl.Series(name=f'responder_{responder}', values=predictor.predict(test.to_pandas()[feature_cols]))\n    )\n\n\n    \n
\n

predictor_6 = models[\"responder_6\"]
\nfeature_cols_with_responders = feature_cols + [f'responder_{i}' for i in range(9) if i != 6]

\n

print(\"Columns used for responder_6 prediction:\", feature_cols_with_responders)
\nprint(\"First few rows of test data for responder_6 prediction:\\n\", test.to_pandas()[feature_cols_with_responders].head())

\n

responder_6_predictions = predictor_6.predict(test.to_pandas()[feature_cols_with_responders])

\n

predictions = test.select(\"row_id\").with_columns(
\n pl.Series(
\n name=\"responder_6\",
\n values=np.clip(responder_6_predictions, a_min=-5, a_max=5),
\n dtype=pl.Float64
\n )
\n)

\n

assert isinstance(predictions, (pl.DataFrame, pd.DataFrame))
\nassert list(predictions.columns) == [\"row_id\", \"responder_6\"]
\nassert len(predictions) == len(test)

\n

print(\"Final predictions columns:\", predictions.columns)
\nprint(\"First few rows of predictions:\\n\", predictions.head())

\n

return predictions

\n

`
\nFinal predictions columns: ['row_id', 'responder_6']
\nFirst few rows of predictions:
\nshape: (5, 2)
\n┌────────┬─────────────┐
\n│ row_id ┆ responder_6 │
\n│ --- ┆ --- │
\n│ i64 ┆ f64 │
\n╞════════╪═════════════╡
\n│ 0 ┆ -0.165136 │
\n│ 1 ┆ -0.165721 │
\n│ 2 ┆ -0.166134 │
\n│ 3 ┆ -0.166615 │
\n│ 4 ┆ -0.167089 │
\n└────────┴─────────────┘
\nBut it still reports an error:

\n

GatewayRuntimeError Traceback (most recent call last)
\nCell In[44], line 75
\n73 inference_server.serve()
\n74 else:
\n---> 75 inference_server.run_local_gateway(
\n76 (
\n77 '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',
\n78 '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',
\n79 )
\n80 )

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:141, in InferenceServer.run_local_gateway(self, data_paths)
\n139 self.gateway.run()
\n140 except Exception as err:
\n--> 141 raise err from None
\n142 finally:
\n143 self.server.stop(0)

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:139, in InferenceServer.run_local_gateway(self, data_paths)
\n137 try:
\n138 self.gateway = self._get_gateway_for_test(data_paths)
\n--> 139 self.gateway.run()
\n140 except Exception as err:
\n141 raise err from None

\n

File /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:98, in Gateway.run(self)
\n95 self.write_result(error)
\n96 elif error:
\n97 # For local testing
\n---> 98 raise error

\n

GatewayRuntimeError: (, 'Traceback (most recent call last):\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py\", line 77, in run\\n self.write_submission(predictions)\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py\", line 161, in write_submission\\n pl.DataFrame(predictions).write_parquet(\\'submission.parquet\\')\\n File \"/opt/conda/lib/python3.10/site-packages/polars/dataframe/frame.py\", line 3837, in write_parquet\\n self._df.write_parquet(\\nOSError: Read-only file system (os error 30)\\n')

\n

Still trying to write to a file on a read-only system, but no line in my prediction function attempts to write to a file

", "votes": 0, "replies": [] }, { "id": 3038851, "author_name": "Alexander Hemingway", "author_url": "", "post_date": "2024-11-07T12:40:47.573000", "content": "

Hi, have you found an answer to this? Facing same issue. Thanks

", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3033852": "Even when I run the official demo, I will get a similar error: trying to write a file to a read-only system.", "3033582": "Share your prediction function that where error occurred", "3033433": "I submitted according to [https://www.kaggle.com/code/ryanholbrook/jane-street-rmf-demo-submission](url),\n\n`inference_server = kaggle_evaluation.jane_street_inference_server.JSInferenceServer(predict)\n\nif os.getenv('KAGGLE_IS_COMPETITION_RERUN'):\n inference_server.serve()\nelse:\n inference_server.run_local_gateway(\n (\n '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n )\n )`\n\n\n\nBut an error was reported\n`GatewayRuntimeError Traceback (most recent call last)\nCell In[27], line 52\n 50 inference_server.serve()\n 51 else:\n---> 52 inference_server.run_local_gateway(\n 53 (\n 54 '/kaggle/input/jane-street-real-time-market-data-forecasting/test.parquet',\n 55 '/kaggle/input/jane-street-real-time-market-data-forecasting/lags.parquet',\n 56 )\n 57 )\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:141, in InferenceServer.run_local_gateway(self, data_paths)\n 139 self.gateway.run()\n 140 except Exception as err:\n--> 141 raise err from None\n 142 finally:\n 143 self.server.stop(0)\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:139, in InferenceServer.run_local_gateway(self, data_paths)\n 137 try:\n 138 self.gateway = self._get_gateway_for_test(data_paths)\n--> 139 self.gateway.run()\n 140 except Exception as err:\n 141 raise err from None\n\nFile /kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py:98, in Gateway.run(self)\n 95 self.write_result(error)\n 96 elif error:\n 97 # For local testing\n---> 98 raise error\n\nGatewayRuntimeError: (, 'Traceback (most recent call last):\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py\", line 77, in run\\n self.write_submission(predictions)\\n File \"/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py\", line 159, in write_submission\\n predictions.to_parquet(\\'submission.parquet\\', index=False)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/util/_decorators.py\", line 333, in wrapper\\n return func(*args, **kwargs)\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/core/frame.py\", line 3113, in to_parquet\\n return to_parquet(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 480, in to_parquet\\n impl.write(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 198, in write\\n path_or_handle, handles, filesystem = _get_path_or_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/parquet.py\", line 140, in _get_path_or_handle\\n handles = get_handle(\\n File \"/opt/conda/lib/python3.10/site-packages/pandas/io/common.py\", line 882, in get_handle\\n handle = open(handle, ioargs.mode)\\nOSError: [Errno 30] Read-only file system: \\'submission.parquet\\'\\n')`\n\nThis means it's trying to write to a read-only folder, but my predicate function doesn't have any code trying to write，What should I do, can anyone help me?" } }