Am I allowed to create a dataset with precomputed segmentation masks along with cropped cell images and use that for faster submission?
", "messages": [ { "id": 1221061, "postDate": "2021-02-28T16:33:51.960Z", "content": "Since the test set used for scoring is private it is not possible to precompute the masks as your kernel only sees them when you hit Submit.
", "rawMarkdown": "Since the test set used for scoring is private it is not possible to precompute the masks as your kernel only sees them when you hit Submit.\n\n", "votes": 1, "replies": [ { "id": 1221098, "postDate": "2021-02-28T17:07:36.023Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1221097, "postDate": "2021-02-28T17:07:36.023Z", "content": "How about just using them just for the public test set? Still you get a lot of time boost…
", "rawMarkdown": "How about just using them just for the public test set? Still you get a lot of time boost...", "votes": 1 }, { "id": 1221123, "postDate": "2021-02-28T17:25:46.013Z", "content": "I don't bother to run many public test images - I only do enough public test images when I save/commit to make sure that my batch generation is working and once I am sure of that I change the code to only do 3 images. Several shared kernels for prediction show methods to do this.
\nThat way when I do the submit/save than after a couple of minutes that is completed and I can do the Submit. That saves on your GPU/TPU quota. I not sure there is any value in running the full public test set unless you have some error that needs to be debugged.
", "rawMarkdown": "I don't bother to run many public test images - I only do enough public test images when I save/commit to make sure that my batch generation is working and once I am sure of that I change the code to only do 3 images. Several shared kernels for prediction show methods to do this. \n\nThat way when I do the submit/save than after a couple of minutes that is completed and I can do the Submit. That saves on your GPU/TPU quota. I not sure there is any value in running the full public test set unless you have some error that needs to be debugged." } ] }, { "id": 1221068, "postDate": "2021-02-28T16:41:10.353Z", "content": "I have done this here: https://www.kaggle.com/thedrcat/hpa-cell-tiles-test-with-enc-dataset
\nThis will only work for the public test set, not for private. I believe this is allowed, if not please let me know.
This is exactly what I was asking about :)
", "rawMarkdown": "This is exactly what I was asking about :)" }, { "id": 1221250, "postDate": "2021-02-28T19:42:55.117Z", "content": "@philculliton are you able to confirm this?
\nCan I compute outputs & masks for the public test set locally and use that in the notebook I submit as as a solution for the contest?
\nThis would mean saving GPU time on the public test set and only using it on the private test set… Also it would mean saving GPU time when I run the notebook before submitting …
This is my first contest and I was not able to confirm this.
\nThanks you for your help.
@bestfitting is this how you are able submit so much? or are you just using CPU?
", "rawMarkdown": "@bestfitting is this how you are able submit so much? or are you just using CPU?" }, { "id": 1222010, "postDate": "2021-03-01T13:19:20.970Z", "content": "After seeing @thedrcat notebooks and discussion posts, I reworked my inference notebook (it still performs multi-label classification during submission), to use pre-computed CellSegmentator masks.
\nNot to boost my own notebooks, but I have a really simple notebook here showing how it works. I have also included in this notebook a link to my dataset containing the precomputed RLEs (normal and submission style) and bounding boxes.
", "rawMarkdown": "After seeing @thedrcat notebooks and discussion posts, I reworked [**my inference notebook**](https://www.kaggle.com/dschettler8845/hpa-cellwise-classification-inference) (it still performs multi-label classification during submission), to use pre-computed CellSegmentator masks.\n* I went from **8-9 hour** submissions (sometimes with timeout) to **30-45 minute** submissions.\n* I strongly recommend this method to allow for much faster prototyping.\n\nThis means I can use locally computer submission file for the public test set? And just compute the private test set in the notebook?
\n", "rawMarkdown": "This means I can use locally computer submission file for the public test set? And just compute the private test set in the notebook?\n\n@maggiemd ?" }, { "id": 1222338, "postDate": "2021-03-01T17:56:37.510Z", "content": "I'm not @maggiemd, however, I can tell you that the answer is yes. That is exactly what @thedrcat does in his notebook, except instead of calculating the submission file for the public test set on his local, he does it in a different notebook than the submission notebook.
", "rawMarkdown": "I'm not @maggiemd, however, I can tell you that the answer is yes. That is exactly what @thedrcat does in his notebook, except instead of calculating the submission file for the public test set on his local, he does it in a different notebook than the submission notebook.", "votes": 1 }, { "id": 1222344, "postDate": "2021-03-01T17:59:55.767Z", "content": "\nPlease clarify - you indicate that you went from 8-9 hours to 45 minutes on \"submissions\".
\nDid you really mean on Save/Commit?
\nSomeone please - share with me the value of running the full public test set of images with or without precomputed RLE's. It seems to me to be only a waste of GPU quota to run the full public test when you Save/Commit. At present I am running only 3 images from the public test when I save/commit. That costs me a couple of minutes of GPU quota.
", "rawMarkdown": "[Darien](https://www.kaggle.com/dschettler8845)\n\nPlease clarify - you indicate that you went from 8-9 hours to 45 minutes on \"submissions\".\n\nDid you really mean on Save/Commit?\n\nSomeone please - share with me the value of running the full public test set of images with or without precomputed RLE's. It seems to me to be only a waste of GPU quota to run the full public test when you Save/Commit. At present I am running only 3 images from the public test when I save/commit. That costs me a couple of minutes of GPU quota." }, { "id": 1222387, "postDate": "2021-03-01T18:34:35.373Z", "content": "@pcjimmmy - No I meant on full public test set submission.
\nThe general value add is that you can submit a notebook and see the Public LB score within the hour… whereas before it would take 8-9 hours
\nYou can look at the versioning of my inference notebook and see that the last 4 versions (Version 37 through Version 40 were all within a 3-4 hour window and all 4 of them have scores generated within that timeframe.
\nTo iteratively adjust my confidence threshold and rerun each time like this would have taken 3-4 days previously. Or I would have had to submit all 4 blindly at the same time and I still would have been waiting 8-9 hours.
", "rawMarkdown": "@pcjimmmy - No I meant on full public test set submission.\n\n---\n\n- My Save/Commit takes 3-5 minutes. Similar to you, I also run on a subset of 3-6 images total instead of the full public test set for this. No one is suggesting that you do the Save/Commit on the full public test-set\n- Next, I submit the submission.csv file and it takes me to the submission page and reruns my notebook on the public and private data. \n- ~45 minutes later the submission is scored and I can see the \n\n\n---\n\n**The general value add is that you can submit a notebook and see the Public LB score within the hour... whereas before it would take 8-9 hours**\n\n---\n\nYou can look at the versioning of my [**inference notebook**](https://www.kaggle.com/dschettler8845/hpa-cellwise-classification-inference) and see that the last 4 versions (Version 37 through Version 40 were all within a 3-4 hour window and all 4 of them have scores generated within that timeframe. \n\n*To iteratively adjust my confidence threshold and rerun each time like this would have taken 3-4 days previously. Or I would have had to submit all 4 blindly at the same time and I still would have been waiting 8-9 hours.*\n", "votes": 1 }, { "id": 1235047, "postDate": "2021-03-11T19:22:14.093Z", "content": "Just to check that I understood your explanation: The Public LB score is calculated on the public test data and die Private LB score is calculated on the private test data?
", "rawMarkdown": "Just to check that I understood your explanation: The Public LB score is calculated on the public test data and die Private LB score is calculated on the private test data?", "votes": 1 }, { "id": 1235079, "postDate": "2021-03-11T19:51:53.920Z", "content": "From what I understand. Yes.
", "rawMarkdown": "From what I understand. Yes." } ] }, { "id": 1220908, "postDate": "2021-02-28T13:24:46.333Z", "content": "Am I allowed to create a dataset with precomputed segmentation masks along with cropped cell images and use that for faster submission?
", "rawMarkdown": "Am I allowed to create a dataset with precomputed segmentation masks along with cropped cell images and use that for faster submission?", "votes": 2 } ], "comments": [ { "id": 1221061, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-28T16:33:51.960000", "content": "Since the test set used for scoring is private it is not possible to precompute the masks as your kernel only sees them when you hit Submit.
", "votes": 1, "replies": [ { "id": 1221098, "author_name": "", "author_url": "", "post_date": "2021-02-28T17:07:36.023000", "content": "", "votes": 0, "replies": [] }, { "id": 1221097, "author_name": "CroDoc", "author_url": "", "post_date": "2021-02-28T17:07:36.023000", "content": "How about just using them just for the public test set? Still you get a lot of time boost…
", "votes": 1, "replies": [] }, { "id": 1221123, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-02-28T17:25:46.013000", "content": "I don't bother to run many public test images - I only do enough public test images when I save/commit to make sure that my batch generation is working and once I am sure of that I change the code to only do 3 images. Several shared kernels for prediction show methods to do this.
\nThat way when I do the submit/save than after a couple of minutes that is completed and I can do the Submit. That saves on your GPU/TPU quota. I not sure there is any value in running the full public test set unless you have some error that needs to be debugged.
", "votes": 0, "replies": [] } ] }, { "id": 1221068, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-02-28T16:41:10.353000", "content": "I have done this here: https://www.kaggle.com/thedrcat/hpa-cell-tiles-test-with-enc-dataset
\nThis will only work for the public test set, not for private. I believe this is allowed, if not please let me know.
This is exactly what I was asking about :)
", "votes": 0, "replies": [] }, { "id": 1221250, "author_name": "CroDoc", "author_url": "", "post_date": "2021-02-28T19:42:55.117000", "content": "@philculliton are you able to confirm this?
\nCan I compute outputs & masks for the public test set locally and use that in the notebook I submit as as a solution for the contest?
\nThis would mean saving GPU time on the public test set and only using it on the private test set… Also it would mean saving GPU time when I run the notebook before submitting …
This is my first contest and I was not able to confirm this.
\nThanks you for your help.
@bestfitting is this how you are able submit so much? or are you just using CPU?
", "votes": 0, "replies": [] }, { "id": 1222010, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-01T13:19:20.970000", "content": "After seeing @thedrcat notebooks and discussion posts, I reworked my inference notebook (it still performs multi-label classification during submission), to use pre-computed CellSegmentator masks.
\nNot to boost my own notebooks, but I have a really simple notebook here showing how it works. I have also included in this notebook a link to my dataset containing the precomputed RLEs (normal and submission style) and bounding boxes.
", "votes": 1, "replies": [] }, { "id": 1222218, "author_name": "CroDoc", "author_url": "", "post_date": "2021-03-01T16:30:08.857000", "content": "This means I can use locally computer submission file for the public test set? And just compute the private test set in the notebook?
\n", "votes": 0, "replies": [] }, { "id": 1222338, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-01T17:56:37.510000", "content": "I'm not @maggiemd, however, I can tell you that the answer is yes. That is exactly what @thedrcat does in his notebook, except instead of calculating the submission file for the public test set on his local, he does it in a different notebook than the submission notebook.
", "votes": 1, "replies": [] }, { "id": 1222344, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2021-03-01T17:59:55.767000", "content": "\nPlease clarify - you indicate that you went from 8-9 hours to 45 minutes on \"submissions\".
\nDid you really mean on Save/Commit?
\nSomeone please - share with me the value of running the full public test set of images with or without precomputed RLE's. It seems to me to be only a waste of GPU quota to run the full public test when you Save/Commit. At present I am running only 3 images from the public test when I save/commit. That costs me a couple of minutes of GPU quota.
", "votes": 0, "replies": [] }, { "id": 1222387, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-01T18:34:35.373000", "content": "@pcjimmmy - No I meant on full public test set submission.
\nThe general value add is that you can submit a notebook and see the Public LB score within the hour… whereas before it would take 8-9 hours
\nYou can look at the versioning of my inference notebook and see that the last 4 versions (Version 37 through Version 40 were all within a 3-4 hour window and all 4 of them have scores generated within that timeframe.
\nTo iteratively adjust my confidence threshold and rerun each time like this would have taken 3-4 days previously. Or I would have had to submit all 4 blindly at the same time and I still would have been waiting 8-9 hours.
", "votes": 1, "replies": [] }, { "id": 1235047, "author_name": "LucaMTB", "author_url": "", "post_date": "2021-03-11T19:22:14.093000", "content": "Just to check that I understood your explanation: The Public LB score is calculated on the public test data and die Private LB score is calculated on the private test data?
", "votes": 1, "replies": [] }, { "id": 1235079, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-03-11T19:51:53.920000", "content": "From what I understand. Yes.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1221061": "Since the test set used for scoring is private it is not possible to precompute the masks as your kernel only sees them when you hit Submit.\n\n", "1221068": "I have done this here: https://www.kaggle.com/thedrcat/hpa-cell-tiles-test-with-enc-dataset\nThis will only work for the public test set, not for private. I believe this is allowed, if not please let me know. ", "1220908": "Am I allowed to create a dataset with precomputed segmentation masks along with cropped cell images and use that for faster submission?" } }