Hi, newbie question, what is the submission.csv for? I took a look at this notebook and I read the comments at the bottom. I'm kinda confused. So, during the run and commit time, the fast_df is just for faster saving, but when you submit, it will pass a private test dataset to your code, making fast_sub = False. I have a few questions.
Thanks.
", "messages": [ { "id": "1364664", "postDate": "06/25/2021 06:07:12", "content": "Hi, newbie question, what is the submission.csv for? I took a look at this notebook and I read the comments at the bottom. I'm kinda confused. So, during the run and commit time, the fast_df is just for faster saving, but when you submit, it will pass a private test dataset to your code, making fast_sub = False. I have a few questions.
Thanks.
", "rawMarkdown": "Hi, newbie question, what is the submission.csv for? I took a look at [this](https://www.kaggle.com/h053473666/siim-cov19-efnb7-yolov5-infer/comments) notebook and I read the comments at the bottom. I'm kinda confused. So, during the run and commit time, the fast_df is just for faster saving, but when you submit, it will pass a private test dataset to your code, making `fast_sub = False`. I have a few questions. \n\n1. Is this private test dataset basically the test folder (I ask this question because the images in the test folder don't have CSVs with labels corresponding to them)? \n2. If the answer to the first question is no, then what are the test folder images for?\n3. I'm guessing the test folder images are for generating the submission.csv. And if that's the case, why are we submitting a submission.csv if it will be rerun on a private test dataset? Is it because they are testing to see if our notebooks are outputting a correct submission.csv format?\n4. Also, in [this](https://www.kaggle.com/h053473666/siim-cov19-efnb7-yolov5-infer/comments) notebook, why does @h053473666 import modules multiple times throughout the notebook? \n\nThanks.", "votes": null }, { "id": "1366105", "postDate": "06/26/2021 13:13:11", "content": "The fast_df sub in that notebook is just to save time when saving the notebook version. That way, it doesn't have to run through the entire test set to make predictions. It only predicts a few rows .. as the author stated in the notebook comments.
\nThe test set is the set of images you predict on to produce the submission file. They don't have labels because we have to predict the labels.
\nYou make submissions on the test set so you can track your score on the LB. Once the competition is over, they'll asses your model on the \"hidden\" test set and produce final scores. If they didn't do it this way, users could just have a radiologist label the test images and generate perfect submissions.
\nI don't know why there are multiple imports in that notebook .. but I suspect the author runs it locally with multiple 'include' files and then copy/pastes it into a Kaggle notebook .. likely an oversight.
", "rawMarkdown": "The fast_df sub in that notebook is just to save time when saving the notebook version. That way, it doesn't have to run through the entire test set to make predictions. It only predicts a few rows .. as the author stated in the notebook comments.\n\nThe test set is the set of images you predict on to produce the submission file. They don't have labels because we have to predict the labels.\n\nYou make submissions on the test set so you can track your score on the LB. Once the competition is over, they'll asses your model on the \"hidden\" test set and produce final scores. If they didn't do it this way, users could just have a radiologist label the test images and generate perfect submissions.\n\nI don't know why there are multiple imports in that notebook .. but I suspect the author runs it locally with multiple 'include' files and then copy/pastes it into a Kaggle notebook .. likely an oversight.", "votes": null }, { "id": "1366516", "postDate": "06/26/2021 21:24:09", "content": "Thank you. Alien's notebook generated a submission.csv with 4 rows: 2 study-level predictions and 2 image-level predictions. Is the public LB score calculated based on these 4 row predictions? Also, how are our models assessed by the \"hidden\" test set? Do they manually go through our code or is the \"hidden\" test set going to be located in \"/kaggle/input/\"?
\nI'm also a bit confused about how the submission.csv should look like. In the competition overview, it showed something like \"negative confidence_score 0 0 1 1\", but in Alien's submission, the submission.csv showed all the predictions and their corresponding confidence scores like
\n\"negative 0.07489524036645889 0 0 1 1 typical 0.5198757648468018 0 0 1 1 indeterminate 0.29435428977012634 0 0 1 1 atypical 0.11087466031312943 0 0 1 1\".
The four rows in Alien's notebook were from the fast_sub. It's not a complete submission .. just for demo purposes.
\nThe LB score is based off a complete submission file. If I understand it correctly ..
\nYou must submit one study level row and one image level row prediction for each test image.
\nThe study level row must contain at least one study level prediction (Negative, Atypical, Typical, Indeterminate)
\nThe image level row must contain at least one image level prediction (opacity, none and BB)
Some studies in the test set have multiple images. The host said they were the same patient/image with minor adjustments or processing. Theoretically, all images in a test study should score the same?.
\nThe test set has 1214 study directories.
\nThe sample_submission.csv file has 2477 rows
I have not checked, but I assume the sample submission file represents the entire test set since it's slightly more than twice the size of the test set .. which accounts for a few studies having more than one image.
\nWhat confuses me is, the host said .. chest radiographs are classified into one of four categories, which are mutually exclusive .. in this post -> https://www.kaggle.com/c/siim-covid19-detection/discussion/240250
But, the rules on the Evaluation page say .. For each study in the test set, you should predict at least one of the above labels .. and shows an example with more than one study level prediction.
None of the images in the train set have more than one of the four categories. So, they are mutually exclusive on the train set .. why aren't they also mutually exclusive in the test set?
", "rawMarkdown": "The four rows in Alien's notebook were from the fast_sub. It's not a complete submission .. just for demo purposes.\n\nThe LB score is based off a complete submission file. If I understand it correctly ..\n\nYou must submit one *study level* row and one *image level* row prediction for each test image.\n\nThe study level row must contain at least one study level prediction (*Negative, Atypical, Typical, Indeterminate*)\nThe image level row must contain at least one image level prediction (*opacity, none and BB*)\n\nSome studies in the test set have multiple images. The host said they were the same patient/image with minor adjustments or processing. Theoretically, all images in a test study should score the same?.\n\nThe test set has 1214 study directories.\nThe sample_submission.csv file has 2477 rows\n\nI have not checked, but I assume the sample submission file represents the entire test set since it's slightly more than twice the size of the test set .. which accounts for a few studies having more than one image.\n\nWhat confuses me is, the host said .. `chest radiographs are classified into one of four categories, which are mutually exclusive` .. in this post -> https://www.kaggle.com/c/siim-covid19-detection/discussion/240250\n\nBut, the rules on the Evaluation page say .. `For each study in the test set, you should predict at least one of the above labels` .. and shows an example with more than one study level prediction.\n\nNone of the images in the train set have more than one of the four categories. So, they are mutually exclusive on the train set .. why aren't they also mutually exclusive in the test set?", "votes": null }, { "id": "1385122", "postDate": "07/12/2021 13:22:41", "content": "Hi
\n1 & 2. The test folder are what we call public test set, it helps to generate public score. The organizer have another set of test set which is not disclosed yet, which we call private test set. So answer to your question 1 is those are public dataset.
\nThe fast_df sub in that notebook is just to save time when saving the notebook version. That way, it doesn't have to run through the entire test set to make predictions. It only predicts a few rows .. as the author stated in the notebook comments.
\nThe test set is the set of images you predict on to produce the submission file. They don't have labels because we have to predict the labels.
\nYou make submissions on the test set so you can track your score on the LB. Once the competition is over, they'll asses your model on the \"hidden\" test set and produce final scores. If they didn't do it this way, users could just have a radiologist label the test images and generate perfect submissions.
\nI don't know why there are multiple imports in that notebook .. but I suspect the author runs it locally with multiple 'include' files and then copy/pastes it into a Kaggle notebook .. likely an oversight.
", "votes": null, "replies": [ { "id": 1366516, "author_name": "vincenttu", "author_url": "", "post_date": "06/26/2021 21:24:09", "content": "Thank you. Alien's notebook generated a submission.csv with 4 rows: 2 study-level predictions and 2 image-level predictions. Is the public LB score calculated based on these 4 row predictions? Also, how are our models assessed by the \"hidden\" test set? Do they manually go through our code or is the \"hidden\" test set going to be located in \"/kaggle/input/\"?
\nI'm also a bit confused about how the submission.csv should look like. In the competition overview, it showed something like \"negative confidence_score 0 0 1 1\", but in Alien's submission, the submission.csv showed all the predictions and their corresponding confidence scores like
\n\"negative 0.07489524036645889 0 0 1 1 typical 0.5198757648468018 0 0 1 1 indeterminate 0.29435428977012634 0 0 1 1 atypical 0.11087466031312943 0 0 1 1\".
The four rows in Alien's notebook were from the fast_sub. It's not a complete submission .. just for demo purposes.
\nThe LB score is based off a complete submission file. If I understand it correctly ..
\nYou must submit one study level row and one image level row prediction for each test image.
\nThe study level row must contain at least one study level prediction (Negative, Atypical, Typical, Indeterminate)
\nThe image level row must contain at least one image level prediction (opacity, none and BB)
Some studies in the test set have multiple images. The host said they were the same patient/image with minor adjustments or processing. Theoretically, all images in a test study should score the same?.
\nThe test set has 1214 study directories.
\nThe sample_submission.csv file has 2477 rows
I have not checked, but I assume the sample submission file represents the entire test set since it's slightly more than twice the size of the test set .. which accounts for a few studies having more than one image.
\nWhat confuses me is, the host said .. chest radiographs are classified into one of four categories, which are mutually exclusive .. in this post -> https://www.kaggle.com/c/siim-covid19-detection/discussion/240250
But, the rules on the Evaluation page say .. For each study in the test set, you should predict at least one of the above labels .. and shows an example with more than one study level prediction.
None of the images in the train set have more than one of the four categories. So, they are mutually exclusive on the train set .. why aren't they also mutually exclusive in the test set?
", "votes": null, "replies": [] } ] }, { "id": 1385122, "author_name": "hawkeat", "author_url": "", "post_date": "07/12/2021 13:22:41", "content": "Hi
\n1 & 2. The test folder are what we call public test set, it helps to generate public score. The organizer have another set of test set which is not disclosed yet, which we call private test set. So answer to your question 1 is those are public dataset.
\n