I wonder if notebook running time includes scoring time? I 'm sure my notebook running time is about 7 hours, but it‘s time out.
", "messages": [ { "id": 1224956, "postDate": "2021-03-03T07:26:55.960Z", "content": "I wonder if notebook running time includes scoring time? I 'm sure my notebook running time is about 7 hours, but it‘s time out.
", "rawMarkdown": "I wonder if notebook running time includes scoring time? I 'm sure my notebook running time is about 7 hours, but it‘s time out.", "votes": 4 }, { "id": 1225248, "postDate": "2021-03-03T13:19:20.037Z", "content": "@daishu - Is your notebook running time 7 hours on the public test data?
\nI have submitted many times now and have changed the way I submit many times, and I have never noticed the scoring time to be included. That being said I'm not an expert, perhaps a Kaggle admin might better answer. I did hope to share some observations with you if that's alright.
\nMy observations are the following:
\nsubmission.csvIS_DEMO=len(ss_df)==559 and then use the flag IS_DEMO to control whether you are predicting on the entire public test set or only a few images. This works because during the Save&Commit part, your ss_df will have a length of 559… however, when the notebook is run on the backend, the length of ss_df will probably be closer to 1250 or something.My public notebooks and those by other talented Kagglers show these observations in action… I hope this helps!
", "rawMarkdown": "@daishu - Is your notebook running time 7 hours on the public test data?\n\nI have submitted many times now and have changed the way I submit many times, and I have never noticed the scoring time to be included. That being said I'm not an expert, perhaps a Kaggle admin might better answer. I did hope to share some observations with you if that's alright.\n\n---\n\n**My observations are the following:**\n\n- The private test set is 2-3 times larger than the public test set\n- If your notebook takes 2 hours to run on the public test set, it will take 4-6 hours to run when you submit your **`submission.csv`**\n- To see your LB score (and get 0 on the private LB), you only need to predict on the public test images and leave the default prediction string for the private test set (this will save you 2/3rds of the submission time). Although, obviously you will need to be submitting predictions on both public and private data when it comes time for your final submissions.\n- You only need to generate a submission.csv file on a few predictions when submitting (this will save you >90% of the time it takes to run your notebook on the public test set). i.e. **`IS_DEMO=len(ss_df)==559`** and then use the flag **`IS_DEMO`** to control whether you are predicting on the entire public test set or only a few images. This works because during the Save&Commit part, your **`ss_df`** will have a length of 559... however, when the notebook is run on the backend, the length of **`ss_df`** will probably be closer to 1250 or something.\n\n---\n\n*My public notebooks and those by other talented Kagglers show these observations in action... **I hope this helps!***", "votes": 1, "replies": [ { "id": 1225309, "postDate": "2021-03-03T14:25:03.010Z", "content": "Thanks, my notebook takes 2.5 hours on public dataset. So I guess it's about 7 hours on private dataset.
", "rawMarkdown": "Thanks, my notebook takes 2.5 hours on public dataset. So I guess it's about 7 hours on private dataset." }, { "id": 1225704, "postDate": "2021-03-03T20:52:52.367Z", "content": "Hmm, it seems to take 150*100/31=>8 hrs for all test data by simple calculation.
\nAlthough this may depend on your model, it will take more time if private includes more 3072 images or…
What do you mean by this?
\nYou only need to generate a submission.csv file on a few predictions when submitting
I don't get it, you are supposed to generate a submission file with all cell predictions
", "rawMarkdown": "Hi @dschettler8845 \n\nWhat do you mean by this? \n\n`You only need to generate a submission.csv file on a few predictions when submitting`\n\nI don't get it, you are supposed to generate a submission file with all cell predictions" }, { "id": 1251055, "postDate": "2021-03-24T13:17:22.530Z", "content": "@glopezzz (nice avatar) - There are two submission events that occur when you want to submit.
\nsubmission.csv file.submission.csv file and Kaggle reruns the notebook on the backend with the Private data swapped in for the Public data (3-4 times more data).My comment …
\n\n\n\"You only need to generate a submission.csv file on a few predictions when submitting\"
\n
… refers to the first type of submission event (generating the submission.csv on public data). If you have a catch in your notebook that identifies whether or not you are using the public data, …
IS_PUBLIC == len(ss_df)==559 # I think this is what it is... I can't remember exactly\n… which essentially checks that the length of the sample_submission.csv is the length of the known public sample_submission.csv, you only need to generate predictions on a couple of the rows. In fact, you don't really need to generate any predictions, I simply do that to make sure that everything looks good and so that I can demo what the predictions look like as my kernel is public.
All Kaggle is looking for is a submission.csv file in the output directory. If it finds that, it will then rerun the notebook. When it reruns the notebook the previously defined catch will be False (as the length of the sample_submission.csv will be greater than 559) and then you can use that flag to make sure you evaluate ALL of the data prior to submission (or just the public part if you want to save time and probe the LB).
I hope this makes sense! If not feel free to reply and I can clarify. My submission flow looks like this…
\nIS_PUBLIC evaluates to True and I only infer on a few rows from the public data. (3-5 minutes to complete)submission.csv file generated in step 1 knowing that IS_PUBLIC evaluates to False due to the presence of the private data. As such I infer on all of the data. (3-6 hours depending)Ok, I totally get it now, thanks a lot for the detailed explanation, I really appreciate it :)
\nI've having trouble with the computing time, my notebook takes around 5h with the public test data :c
\nAt least with your great idea, I can save a lot of time in the 'save and commit' process.
Thanks again
", "rawMarkdown": "Ok, I totally get it now, thanks a lot for the detailed explanation, I really appreciate it :)\n\nI've having trouble with the computing time, my notebook takes around 5h with the public test data :c\nAt least with your great idea, I can save a lot of time in the 'save and commit' process.\n\nThanks again", "votes": 2 }, { "id": 1251177, "postDate": "2021-03-24T14:46:27.727Z", "content": "Glad it helped!
\nIf you aren't already, I would recommend using an approach similar to the notebooks/methodology created by linshokaku and refined by Raman for the actual Cell Segmentation… as that is often the bottleneck in speed (it was for me).
\nPlease see these notebooks for more details.
\nThat's awesome!! Thanks for the information, I'll be working with that from now on :)
", "rawMarkdown": "That's awesome!! Thanks for the information, I'll be working with that from now on :)", "votes": 1 } ] } ], "comments": [ { "id": 1225248, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-03T13:19:20.037000", "content": "@daishu - Is your notebook running time 7 hours on the public test data?
\nI have submitted many times now and have changed the way I submit many times, and I have never noticed the scoring time to be included. That being said I'm not an expert, perhaps a Kaggle admin might better answer. I did hope to share some observations with you if that's alright.
\nMy observations are the following:
\nsubmission.csvIS_DEMO=len(ss_df)==559 and then use the flag IS_DEMO to control whether you are predicting on the entire public test set or only a few images. This works because during the Save&Commit part, your ss_df will have a length of 559… however, when the notebook is run on the backend, the length of ss_df will probably be closer to 1250 or something.My public notebooks and those by other talented Kagglers show these observations in action… I hope this helps!
", "votes": 1, "replies": [ { "id": 1225309, "author_name": "Correlation", "author_url": "", "post_date": "2021-03-03T14:25:03.010000", "content": "Thanks, my notebook takes 2.5 hours on public dataset. So I guess it's about 7 hours on private dataset.
", "votes": 0, "replies": [] }, { "id": 1225704, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-03-03T20:52:52.367000", "content": "Hmm, it seems to take 150*100/31=>8 hrs for all test data by simple calculation.
\nAlthough this may depend on your model, it will take more time if private includes more 3072 images or…
What do you mean by this?
\nYou only need to generate a submission.csv file on a few predictions when submitting
I don't get it, you are supposed to generate a submission file with all cell predictions
", "votes": 0, "replies": [] }, { "id": 1251055, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-24T13:17:22.530000", "content": "@glopezzz (nice avatar) - There are two submission events that occur when you want to submit.
\nsubmission.csv file.submission.csv file and Kaggle reruns the notebook on the backend with the Private data swapped in for the Public data (3-4 times more data).My comment …
\n\n\n\"You only need to generate a submission.csv file on a few predictions when submitting\"
\n
… refers to the first type of submission event (generating the submission.csv on public data). If you have a catch in your notebook that identifies whether or not you are using the public data, …
IS_PUBLIC == len(ss_df)==559 # I think this is what it is... I can't remember exactly\n… which essentially checks that the length of the sample_submission.csv is the length of the known public sample_submission.csv, you only need to generate predictions on a couple of the rows. In fact, you don't really need to generate any predictions, I simply do that to make sure that everything looks good and so that I can demo what the predictions look like as my kernel is public.
All Kaggle is looking for is a submission.csv file in the output directory. If it finds that, it will then rerun the notebook. When it reruns the notebook the previously defined catch will be False (as the length of the sample_submission.csv will be greater than 559) and then you can use that flag to make sure you evaluate ALL of the data prior to submission (or just the public part if you want to save time and probe the LB).
I hope this makes sense! If not feel free to reply and I can clarify. My submission flow looks like this…
\nIS_PUBLIC evaluates to True and I only infer on a few rows from the public data. (3-5 minutes to complete)submission.csv file generated in step 1 knowing that IS_PUBLIC evaluates to False due to the presence of the private data. As such I infer on all of the data. (3-6 hours depending)Ok, I totally get it now, thanks a lot for the detailed explanation, I really appreciate it :)
\nI've having trouble with the computing time, my notebook takes around 5h with the public test data :c
\nAt least with your great idea, I can save a lot of time in the 'save and commit' process.
Thanks again
", "votes": 2, "replies": [] }, { "id": 1251177, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-24T14:46:27.727000", "content": "Glad it helped!
\nIf you aren't already, I would recommend using an approach similar to the notebooks/methodology created by linshokaku and refined by Raman for the actual Cell Segmentation… as that is often the bottleneck in speed (it was for me).
\nPlease see these notebooks for more details.
\nThat's awesome!! Thanks for the information, I'll be working with that from now on :)
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1224956": "I wonder if notebook running time includes scoring time? I 'm sure my notebook running time is about 7 hours, but it‘s time out.", "1225248": "@daishu - Is your notebook running time 7 hours on the public test data?\n\nI have submitted many times now and have changed the way I submit many times, and I have never noticed the scoring time to be included. That being said I'm not an expert, perhaps a Kaggle admin might better answer. I did hope to share some observations with you if that's alright.\n\n---\n\n**My observations are the following:**\n\n- The private test set is 2-3 times larger than the public test set\n- If your notebook takes 2 hours to run on the public test set, it will take 4-6 hours to run when you submit your **`submission.csv`**\n- To see your LB score (and get 0 on the private LB), you only need to predict on the public test images and leave the default prediction string for the private test set (this will save you 2/3rds of the submission time). Although, obviously you will need to be submitting predictions on both public and private data when it comes time for your final submissions.\n- You only need to generate a submission.csv file on a few predictions when submitting (this will save you >90% of the time it takes to run your notebook on the public test set). i.e. **`IS_DEMO=len(ss_df)==559`** and then use the flag **`IS_DEMO`** to control whether you are predicting on the entire public test set or only a few images. This works because during the Save&Commit part, your **`ss_df`** will have a length of 559... however, when the notebook is run on the backend, the length of **`ss_df`** will probably be closer to 1250 or something.\n\n---\n\n*My public notebooks and those by other talented Kagglers show these observations in action... **I hope this helps!***" } }