I've been getting a constant stream of 137 exit codes on my Competition Kaggle Kernels. The dataset was sufficently large that it couldn't all fit inside 16GB RAM, so I had to read the files individually do the submit in a loop, using model.predict()
I hope the root cause is this bug, because the Bengali AI deadline is in 3 hours and after two weeks of work I only managed to get a single commit to successfully complete (by accident - it was set to train for only 1 epoch and got a 0.20 score). I have coded a last-minute fix and have two notebooks compiling at the moment (with three hours left to go). I am really hoping to get a formal score on my first prize-pool competition.
\n\nTensorflow/Keras BUG: Repeatedly calling model.predict(...) results in memory leak #13118\n- https://github.com/keras-team/keras/issues/13118
\n\nAnyways, my solution was to simply call model.predict_on_batch() in a loop:\n```
def submission_df(model, output_shape):\n gc.collect()
\n\nsubmission = pd.DataFrame(columns=output_shape.keys())\n# large datasets on submit, so loop\nfor data_id in range(0,4):\n test_dataset = DatasetDF(test_train='test', data_id=data_id, transform_X_args = { \"normalize\": True } )\n test_dataset_rows = test_dataset.X['train'].shape[0]\n batch_size = 32\n for index in range(0, test_dataset_rows, 32):\n X_batch = test_dataset.X['train'][index : index+batch_size]\n predictions = model.predict_on_batch(X_batch)\n submission = submission.append(\n pd.DataFrame({\n key: np.argmax( predictions[index], axis=-1 )\n for index, key in enumerate(output_shape.keys())\n }, index=test_dataset.ID['train'])\n )\nreturn submission\n```
", "messages": [ { "id": "775604", "postDate": "03/16/2020 21:20:42", "content": "I've been getting a constant stream of 137 exit codes on my Competition Kaggle Kernels. The dataset was sufficently large that it couldn't all fit inside 16GB RAM, so I had to read the files individually do the submit in a loop, using model.predict()
I hope the root cause is this bug, because the Bengali AI deadline is in 3 hours and after two weeks of work I only managed to get a single commit to successfully complete (by accident - it was set to train for only 1 epoch and got a 0.20 score). I have coded a last-minute fix and have two notebooks compiling at the moment (with three hours left to go). I am really hoping to get a formal score on my first prize-pool competition.
\n\nTensorflow/Keras BUG: Repeatedly calling model.predict(...) results in memory leak #13118\n- https://github.com/keras-team/keras/issues/13118
\n\nAnyways, my solution was to simply call model.predict_on_batch() in a loop:\n```
def submission_df(model, output_shape):\n gc.collect()
\n\nsubmission = pd.DataFrame(columns=output_shape.keys())\n# large datasets on submit, so loop\nfor data_id in range(0,4):\n test_dataset = DatasetDF(test_train='test', data_id=data_id, transform_X_args = { \"normalize\": True } )\n test_dataset_rows = test_dataset.X['train'].shape[0]\n batch_size = 32\n for index in range(0, test_dataset_rows, 32):\n X_batch = test_dataset.X['train'][index : index+batch_size]\n predictions = model.predict_on_batch(X_batch)\n submission = submission.append(\n pd.DataFrame({\n key: np.argmax( predictions[index], axis=-1 )\n for index, key in enumerate(output_shape.keys())\n }, index=test_dataset.ID['train'])\n )\nreturn submission\n```
", "rawMarkdown": "I've been getting a constant stream of 137 exit codes on my Competition Kaggle Kernels. The dataset was sufficently large that it couldn't all fit inside 16GB RAM, so I had to read the files individually do the submit in a loop, using `model.predict()`\n\nI hope the root cause is this bug, because the Bengali AI deadline is in 3 hours and after two weeks of work I only managed to get a single commit to successfully complete (by accident - it was set to train for only 1 epoch and got a 0.20 score). I have coded a last-minute fix and have two notebooks compiling at the moment (with three hours left to go). I am really hoping to get a formal score on my first prize-pool competition.\n\nTensorflow/Keras BUG: Repeatedly calling model.predict(...) results in memory leak #13118\n- https://github.com/keras-team/keras/issues/13118\n\n\nAnyways, my solution was to simply call `model.predict_on_batch()` in a loop:\n```\n### BUGFIX: Repeatedly calling model.predict(...) results in memory leak - https://github.com/keras-team/keras/issues/13118\ndef submission_df(model, output_shape):\n gc.collect()\n\n submission = pd.DataFrame(columns=output_shape.keys())\n # large datasets on submit, so loop\n for data_id in range(0,4):\n test_dataset = DatasetDF(test_train='test', data_id=data_id, transform_X_args = { \"normalize\": True } )\n test_dataset_rows = test_dataset.X['train'].shape[0]\n batch_size = 32\n for index in range(0, test_dataset_rows, 32):\n X_batch = test_dataset.X['train'][index : index+batch_size]\n predictions = model.predict_on_batch(X_batch)\n submission = submission.append(\n pd.DataFrame({\n key: np.argmax( predictions[index], axis=-1 )\n for index, key in enumerate(output_shape.keys())\n }, index=test_dataset.ID['train'])\n )\n return submission\n```", "votes": null }, { "id": "775664", "postDate": "03/16/2020 23:24:24", "content": "Yep... that was exactlt the issue!
\n\nI finally haave the ability to commit and submit notebooks! This feels so good.
", "rawMarkdown": "Yep... that was exactlt the issue!\n\nI finally haave the ability to commit and submit notebooks! This feels so good.", "votes": null } ], "comments": [ { "id": 775664, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/16/2020 23:24:24", "content": "Yep... that was exactlt the issue!
\n\nI finally haave the ability to commit and submit notebooks! This feels so good.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "775604": "I've been getting a constant stream of 137 exit codes on my Competition Kaggle Kernels. The dataset was sufficently large that it couldn't all fit inside 16GB RAM, so I had to read the files individually do the submit in a loop, using `model.predict()`\n\nI hope the root cause is this bug, because the Bengali AI deadline is in 3 hours and after two weeks of work I only managed to get a single commit to successfully complete (by accident - it was set to train for only 1 epoch and got a 0.20 score). I have coded a last-minute fix and have two notebooks compiling at the moment (with three hours left to go). I am really hoping to get a formal score on my first prize-pool competition.\n\nTensorflow/Keras BUG: Repeatedly calling model.predict(...) results in memory leak #13118\n- https://github.com/keras-team/keras/issues/13118\n\n\nAnyways, my solution was to simply call `model.predict_on_batch()` in a loop:\n```\n### BUGFIX: Repeatedly calling model.predict(...) results in memory leak - https://github.com/keras-team/keras/issues/13118\ndef submission_df(model, output_shape):\n gc.collect()\n\n submission = pd.DataFrame(columns=output_shape.keys())\n # large datasets on submit, so loop\n for data_id in range(0,4):\n test_dataset = DatasetDF(test_train='test', data_id=data_id, transform_X_args = { \"normalize\": True } )\n test_dataset_rows = test_dataset.X['train'].shape[0]\n batch_size = 32\n for index in range(0, test_dataset_rows, 32):\n X_batch = test_dataset.X['train'][index : index+batch_size]\n predictions = model.predict_on_batch(X_batch)\n submission = submission.append(\n pd.DataFrame({\n key: np.argmax( predictions[index], axis=-1 )\n for index, key in enumerate(output_shape.keys())\n }, index=test_dataset.ID['train'])\n )\n return submission\n```", "775664": "Yep... that was exactlt the issue!\n\nI finally haave the ability to commit and submit notebooks! This feels so good." }, "source": "meta" }