Provided that almost half of the competitors saw their final submission fail on LB, I'm sharing some tips that helped us have a reliable pipeline. I would've shared such ideas earlier but I had no idea such a mess would happen on private. Hopefully this helps for future code competitions.
\nsubmission.csv file is generated if and only if your inference is done successfully.When comitting, if a notebook cell fails, the following ones will still be executed. I believe the same happens when submitting. To counter this issue, I advise generating the submission.csv in the same cell you do computations. See our submission notebook for reference.
\nThis way an unsuccessful run will get a submission error, instead of a silent 0 score.
Our inference notebooks were run on 4ef6695ce, the biggest training image, which is bigger than every image in the private test set (?). Having no memory errors on it meant our pipeline will not run out of memory during the submission. Setting DEBUG = True in our inference notebook does so.
Using code written by others is extremelly risky. It is tempting to fork the deepflash2 notebook because it performs very well and is simple to use, but it's a huge black box which makes it hard to understand what's going on when there are issues.
", "messages": [ { "id": "1302017", "postDate": "05/11/2021 11:05:44", "content": "Provided that almost half of the competitors saw their final submission fail on LB, I'm sharing some tips that helped us have a reliable pipeline. I would've shared such ideas earlier but I had no idea such a mess would happen on private. Hopefully this helps for future code competitions.
\nsubmission.csv file is generated if and only if your inference is done successfully.When comitting, if a notebook cell fails, the following ones will still be executed. I believe the same happens when submitting. To counter this issue, I advise generating the submission.csv in the same cell you do computations. See our submission notebook for reference.
\nThis way an unsuccessful run will get a submission error, instead of a silent 0 score.
Our inference notebooks were run on 4ef6695ce, the biggest training image, which is bigger than every image in the private test set (?). Having no memory errors on it meant our pipeline will not run out of memory during the submission. Setting DEBUG = True in our inference notebook does so.
Using code written by others is extremelly risky. It is tempting to fork the deepflash2 notebook because it performs very well and is simple to use, but it's a huge black box which makes it hard to understand what's going on when there are issues.
", "rawMarkdown": "Provided that almost half of the competitors saw their final submission fail on LB, I'm sharing some tips that helped us have a reliable pipeline. I would've shared such ideas earlier but I had no idea such a mess would happen on private. Hopefully this helps for future code competitions.\n\n- Make sure the `submission.csv` file is generated if and only if your inference is done successfully.\n\nWhen comitting, if a notebook cell fails, the following ones will still be executed. I believe the same happens when submitting. To counter this issue, I advise generating the `submission.csv` in the same cell you do computations. See [our submission notebook](https://www.kaggle.com/optimo/hubmap-inference-th-o) for reference.\nThis way an unsuccessful run will get a submission error, instead of a silent 0 score.\n\n- Test your pipeline on bigger images. \n\nOur inference notebooks were run on `4ef6695ce`, the biggest training image, which is bigger than every image in the private test set (?). Having no memory errors on it meant our pipeline will not run out of memory during the submission. Setting `DEBUG = True` in our inference notebook does so.\n\n- Build your own pipeline\n\nUsing code written by others is extremelly risky. It is tempting to fork the [deepflash2 notebook](https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub) because it performs very well and is simple to use, but it's a huge black box which makes it hard to understand what's going on when there are issues.", "votes": null }, { "id": "1302380", "postDate": "05/11/2021 14:11:11", "content": "yeah, for testing moving forward, I will raise an exception from inside the loop and see whether the csv still gets generated with empty rows. Another way to address it is to simply check for empty rows as the last step.Being still a newbie, I will probably use to great extent shared pipelines, for some time to come.
", "rawMarkdown": "yeah, for testing moving forward, I will raise an exception from inside the loop and see whether the csv still gets generated with empty rows. Another way to address it is to simply check for empty rows as the last step.Being still a newbie, I will probably use to great extent shared pipelines, for some time to come.", "votes": null }, { "id": "1302399", "postDate": "05/11/2021 14:24:27", "content": "\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\"
\nIt's really useful.
", "rawMarkdown": "\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\"\n\nIt's really useful.", "votes": null }, { "id": "1302583", "postDate": "05/11/2021 15:59:36", "content": "A lesson well learnt for some like me … who were surprised by the final score…
\nThanks for sharing @theoviel … very helpful!!
\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\" - yes this is what I missed. Thanks for the suggestions
", "rawMarkdown": "\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\" - yes this is what I missed. Thanks for the suggestions", "votes": null }, { "id": "1303989", "postDate": "05/12/2021 11:19:23", "content": "All these are really good practice for future code competitions. However, unfortunately \"Test your pipeline on bigger images\" alone doesn't really work for this deepflash2 v14 case. The pipeline is working for any training image 😂. I'm sure about that as I did my 5 fold validation on Kaggle.
", "rawMarkdown": "All these are really good practice for future code competitions. However, unfortunately \"Test your pipeline on bigger images\" alone doesn't really work for this deepflash2 v14 case. The pipeline is working for any training image 😂. I'm sure about that as I did my 5 fold validation on Kaggle.", "votes": null }, { "id": "1304653", "postDate": "05/12/2021 18:56:35", "content": "Great ideas Theo. Since Kaggle submit notebook keep running after an error occurs, these suggestions are very important to confirm that submission.csv file contains what we expect.
This is especially important regarding post process. Many times we modify a submission dataframe by sequentially running code cells to modify a prediction column. If one of these post process updates does not occur (due to error), the submission still gets outputted but without the PP. Then we falsely assume that the PP doesn't work.
", "rawMarkdown": "Great ideas Theo. Since Kaggle submit notebook keep running after an error occurs, these suggestions are very important to confirm that `submission.csv` file contains what we expect. \n\nThis is especially important regarding post process. Many times we modify a submission dataframe by sequentially running code cells to modify a prediction column. If one of these post process updates does not occur (due to error), the submission still gets outputted but without the PP. Then we falsely assume that the PP doesn't work.", "votes": null }, { "id": "1304889", "postDate": "05/13/2021 00:57:03", "content": "Another idea is to initialize SUCCESS=0 variable in the first code cell. And add as the last line to every code cell SUCCESS += 1. Then in your final code cell use
if SUCCESS == NUM_OF_CODE_CELLS:\n sub.to_csv('submission.csv',index=False)\nFYI -
\nHow to know if your private submissions will fail when public scored and succeeded
Hi @theoviel
\nWhat do you think is the rationale behind running notebook cells after an unhandled exception?
I think that's just how it works, and I'm not sure the Kaggle team even made that decision.
\nOverall, I think it hurts code robustness which is bad when running code on unseen data.
\nHowever, when I leave an unwanted cell of code in my notebooks i'm happy that the commit still finishes.
I cannot imagine any use for code that just ignores all the exceptions and continues to run as if nothing happened. This is not just a bad practice this is also a very dangerous one. Also, this is not about some legacy pieces of code. This is about error handling in general. I added the try-except clause and guess what. Raising error does not halt code execution. I simply don't get the rational behind such a behavior.
", "rawMarkdown": "I cannot imagine any use for code that just ignores all the exceptions and continues to run as if nothing happened. This is not just a bad practice this is also a very dangerous one. Also, this is not about some legacy pieces of code. This is about error handling in general. I added the try-except clause and guess what. Raising error does not halt code execution. I simply don't get the rational behind such a behavior.", "votes": null }, { "id": "1307141", "postDate": "05/14/2021 09:24:04", "content": "There's more! This happens ONLY on the private test run! If you run this code from the notebook on public test the exception handling is done correctly and halts script execution. Sorry, but no way this is \"normal\".
", "rawMarkdown": "There's more! This happens ONLY on the private test run! If you run this code from the notebook on public test the exception handling is done correctly and halts script execution. Sorry, but no way this is \"normal\".", "votes": null } ], "comments": [ { "id": 1302380, "author_name": "andrasferenczi", "author_url": "", "post_date": "05/11/2021 14:11:11", "content": "yeah, for testing moving forward, I will raise an exception from inside the loop and see whether the csv still gets generated with empty rows. Another way to address it is to simply check for empty rows as the last step.Being still a newbie, I will probably use to great extent shared pipelines, for some time to come.
", "votes": null, "replies": [] }, { "id": 1302399, "author_name": "tikutiku", "author_url": "", "post_date": "05/11/2021 14:24:27", "content": "\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\"
\nIt's really useful.
", "votes": null, "replies": [] }, { "id": 1302583, "author_name": "kmldas", "author_url": "", "post_date": "05/11/2021 15:59:36", "content": "A lesson well learnt for some like me … who were surprised by the final score…
\nThanks for sharing @theoviel … very helpful!!
\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\" - yes this is what I missed. Thanks for the suggestions
", "votes": null, "replies": [] }, { "id": 1303989, "author_name": "css919", "author_url": "", "post_date": "05/12/2021 11:19:23", "content": "All these are really good practice for future code competitions. However, unfortunately \"Test your pipeline on bigger images\" alone doesn't really work for this deepflash2 v14 case. The pipeline is working for any training image 😂. I'm sure about that as I did my 5 fold validation on Kaggle.
", "votes": null, "replies": [] }, { "id": 1304653, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 18:56:35", "content": "Great ideas Theo. Since Kaggle submit notebook keep running after an error occurs, these suggestions are very important to confirm that submission.csv file contains what we expect.
This is especially important regarding post process. Many times we modify a submission dataframe by sequentially running code cells to modify a prediction column. If one of these post process updates does not occur (due to error), the submission still gets outputted but without the PP. Then we falsely assume that the PP doesn't work.
", "votes": null, "replies": [] }, { "id": 1304889, "author_name": "cdeotte", "author_url": "", "post_date": "05/13/2021 00:57:03", "content": "Another idea is to initialize SUCCESS=0 variable in the first code cell. And add as the last line to every code cell SUCCESS += 1. Then in your final code cell use
if SUCCESS == NUM_OF_CODE_CELLS:\n sub.to_csv('submission.csv',index=False)\nFYI -
\nHow to know if your private submissions will fail when public scored and succeeded
Hi @theoviel
\nWhat do you think is the rationale behind running notebook cells after an unhandled exception?
I think that's just how it works, and I'm not sure the Kaggle team even made that decision.
\nOverall, I think it hurts code robustness which is bad when running code on unseen data.
\nHowever, when I leave an unwanted cell of code in my notebooks i'm happy that the commit still finishes.
I cannot imagine any use for code that just ignores all the exceptions and continues to run as if nothing happened. This is not just a bad practice this is also a very dangerous one. Also, this is not about some legacy pieces of code. This is about error handling in general. I added the try-except clause and guess what. Raising error does not halt code execution. I simply don't get the rational behind such a behavior.
", "votes": null, "replies": [] }, { "id": 1307141, "author_name": "sakvaua", "author_url": "", "post_date": "05/14/2021 09:24:04", "content": "There's more! This happens ONLY on the private test run! If you run this code from the notebook on public test the exception handling is done correctly and halts script execution. Sorry, but no way this is \"normal\".
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1302017": "Provided that almost half of the competitors saw their final submission fail on LB, I'm sharing some tips that helped us have a reliable pipeline. I would've shared such ideas earlier but I had no idea such a mess would happen on private. Hopefully this helps for future code competitions.\n\n- Make sure the `submission.csv` file is generated if and only if your inference is done successfully.\n\nWhen comitting, if a notebook cell fails, the following ones will still be executed. I believe the same happens when submitting. To counter this issue, I advise generating the `submission.csv` in the same cell you do computations. See [our submission notebook](https://www.kaggle.com/optimo/hubmap-inference-th-o) for reference.\nThis way an unsuccessful run will get a submission error, instead of a silent 0 score.\n\n- Test your pipeline on bigger images. \n\nOur inference notebooks were run on `4ef6695ce`, the biggest training image, which is bigger than every image in the private test set (?). Having no memory errors on it meant our pipeline will not run out of memory during the submission. Setting `DEBUG = True` in our inference notebook does so.\n\n- Build your own pipeline\n\nUsing code written by others is extremelly risky. It is tempting to fork the [deepflash2 notebook](https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub) because it performs very well and is simple to use, but it's a huge black box which makes it hard to understand what's going on when there are issues.", "1302380": "yeah, for testing moving forward, I will raise an exception from inside the loop and see whether the csv still gets generated with empty rows. Another way to address it is to simply check for empty rows as the last step.Being still a newbie, I will probably use to great extent shared pipelines, for some time to come.", "1302399": "\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\"\n\nIt's really useful.", "1302583": "A lesson well learnt for some like me ... who were surprised by the final score...\nThanks for sharing @theoviel ... very helpful!!", "1302733": "\"Make sure the submission.csv file is generated if and only if your inference is done successfully.\" - yes this is what I missed. Thanks for the suggestions", "1303989": "All these are really good practice for future code competitions. However, unfortunately \"Test your pipeline on bigger images\" alone doesn't really work for this deepflash2 v14 case. The pipeline is working for any training image 😂. I'm sure about that as I did my 5 fold validation on Kaggle.", "1304653": "Great ideas Theo. Since Kaggle submit notebook keep running after an error occurs, these suggestions are very important to confirm that `submission.csv` file contains what we expect. \n\nThis is especially important regarding post process. Many times we modify a submission dataframe by sequentially running code cells to modify a prediction column. If one of these post process updates does not occur (due to error), the submission still gets outputted but without the PP. Then we falsely assume that the PP doesn't work.", "1304889": "Another idea is to initialize `SUCCESS=0` variable in the first code cell. And add as the last line to every code cell `SUCCESS += 1`. Then in your final code cell use\n\n if SUCCESS == NUM_OF_CODE_CELLS:\n sub.to_csv('submission.csv',index=False)", "1306851": "FYI -\n[How to know if your private submissions will fail when public scored and succeeded](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238751)", "1307082": "Hi @theoviel \nWhat do you think is the rationale behind running notebook cells after an unhandled exception?", "1307105": "I think that's just how it works, and I'm not sure the Kaggle team even made that decision.\n\nOverall, I think it hurts code robustness which is bad when running code on unseen data.\nHowever, when I leave an unwanted cell of code in my notebooks i'm happy that the commit still finishes.", "1307114": "I cannot imagine any use for code that just ignores all the exceptions and continues to run as if nothing happened. This is not just a bad practice this is also a very dangerous one. Also, this is not about some legacy pieces of code. This is about error handling in general. I added the try-except clause and guess what. Raising error does not halt code execution. I simply don't get the rational behind such a behavior.", "1307141": "There's more! This happens ONLY on the private test run! If you run this code from the notebook on public test the exception handling is done correctly and halts script execution. Sorry, but no way this is \"normal\"." }, "source": "meta" }