\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4227277%2Fb73855bc19598e0c482a88898bb4cc4f%2F2020.08.23%20.17.35.17.png?generation=1598193352442435&alt=media\")
Hi guys,
\nMy private test set score is -24.7981 which is really weird cause my validation score is about -6.9.
\nDid someone have this score? someone have any ideas?
Hi guys,
\nMy private test set score is -24.7981 which is really weird cause my validation score is about -6.9.
\nDid someone have this score? someone have any ideas?
Hi! @omerhacohen make sure your submission file is populated in the right order.
", "rawMarkdown": "Hi! @omerhacohen make sure your submission file is populated in the right order.", "votes": null }, { "id": "981954", "postDate": "08/22/2020 22:07:59", "content": "Could you please tell us about your CV strategy? Cause it could be wrong (as in my case)
", "rawMarkdown": "Could you please tell us about your CV strategy? Cause it could be wrong (as in my case)", "votes": null }, { "id": "982014", "postDate": "08/23/2020 00:25:57", "content": "That's the score when your FVC predictions and sigma hit the bounds (1000 and 70).
", "rawMarkdown": "That's the score when your FVC predictions and sigma hit the bounds (1000 and 70).", "votes": null }, { "id": "982056", "postDate": "08/23/2020 02:30:22", "content": "How many rows in your submission file? Do a print of some rows for submission data frame and see if the values populated correctly.
", "rawMarkdown": "How many rows in your submission file? Do a print of some rows for submission data frame and see if the values populated correctly.", "votes": null }, { "id": "982621", "postDate": "08/23/2020 14:26:39", "content": "@cv13j0 What does it mean in the right order?
\nMust I populate it so every patient has patientX_1, patientY_1 and so on or can I do patientX_1 patientX_2 and so on..?
@omerhacohen Can you provide a snapshot of your Submission.csv file that you get when you commit your notebook? It would be easier for everyone else to identify your problem that way.
", "rawMarkdown": "omerhacohen Can you provide a snapshot of your Submission.csv file that you get when you commit your notebook? It would be easier for everyone else to identify your problem that way.", "votes": null }, { "id": "982638", "postDate": "08/23/2020 14:40:52", "content": "@jjinho That is correct!
\nBut why does that happen?
\nFirst I initiate an empty form with confidence and FVC set to zero, and it seems that my code for populating the table doesnt work on the private set.
\nBelow is the code snippet that populates the table.
\ntest_dirs are all of the directories in the test dataset (\"Patients\")
\npredict.predict_test opens the submission form, predicts FVC and confidence for one patient and then saves it on the same place.
*** This code runs just fine on the public test set, and populates the submission table with the right results. somehow on the private set this thing doesnt run. HELP! ***
\n
Thats a snapshot of my submission csv:
", "rawMarkdown": "Thats a snapshot of my submission csv:", "votes": null }, { "id": "982649", "postDate": "08/23/2020 14:51:48", "content": "This are the paths:
", "rawMarkdown": "This are the paths:", "votes": null }, { "id": "982657", "postDate": "08/23/2020 15:01:04", "content": "I see that from your post below that you are saving the dicom images as jpeg in a folder you create in /kaggle/working/. I think you are running out of disk space when running on the private test set and only part of your images are being saved. Since you are initializing your dataframe with zeros i think the scoring is done on this dataframe with zeros.
I would suggest you save your jpeg images in a folder in the kaggle root folder such as /kaggle/test_images/. You have access to about 2TB storage in the /kaggle/ directory compared to the 5GB in the /kaggle/working/ directory. The submission.csv file format that you posted looks fine.
Hi @omerhacohen, you are calling a funciton predict_test to create the csv file? Can you show that function?
Thats why I only save one patients photos, predict them on the csv file and then delete them. You can see the !rm -rf test at the end… Do you think its still the case?
", "rawMarkdown": "Thats why I only save one patients photos, predict them on the csv file and then delete them. You can see the !rm -rf test at the end... Do you think its still the case?", "votes": null }, { "id": "982835", "postDate": "08/23/2020 17:42:21", "content": "Yeah..
\nBut this one has some functions in it to… Its to messy to bring it all here
Tried saving it at the root folder… still doesnt work
", "rawMarkdown": "Tried saving it at the root folder... still doesnt work", "votes": null }, { "id": "983488", "postDate": "08/24/2020 10:56:17", "content": "@omerhacohen I didn't see the line where you delete the saved image. I have another question, why do you set your confidence values to 200 instead of predicting them. In my experience i found that the confidence value impacts the submission score significantly. If your model is overconfident or under-confident for a specific FVC value you will penalized harshly. Your problem may be due to you setting all confidence values to a certain value.
", "rawMarkdown": "omerhacohen I didn't see the line where you delete the saved image. I have another question, why do you set your confidence values to 200 instead of predicting them. In my experience i found that the confidence value impacts the submission score significantly. If your model is overconfident or under-confident for a specific FVC value you will penalized harshly. Your problem may be due to you setting all confidence values to a certain value.", "votes": null }, { "id": "983766", "postDate": "08/24/2020 15:24:20", "content": "I tried to put up a baseline model and then predict confidence… My validation score with a constant 200 is 6.9.. And -24.. Is the score for 0-0 submission. So theres a problem with populating the table.
\nDid you succeed with predicting the test images? Can I see the code?
Your validation score is that good because your model is probably biased and is overfitting. I'm afraid i can't share my code publicly and private sharing is prohibited. But there are many public kernels using the dicom images to predict and i would suggest you to go through them well and understand the confidence value prediction section.
", "rawMarkdown": "Your validation score is that good because your model is probably biased and is overfitting. I'm afraid i can't share my code publicly and private sharing is prohibited. But there are many public kernels using the dicom images to predict and i would suggest you to go through them well and understand the confidence value prediction section.", "votes": null }, { "id": "983924", "postDate": "08/24/2020 17:28:49", "content": "Can you link please link some of these notebooks?
\nI cant find a notebook that transforms CT scans to submission with public score…
https://www.kaggle.com/carlossouza/end-to-end-model-ct-scans-tabular
\nhttps://www.kaggle.com/jameschapman19/pytorch-tabular-qr-histogram
@omerhacohen Did you figure this out?
\nI am having the same issue and i am not sure what is causing it
Yea I did.
\nlike @jjinho said, -24.79 is the score you get when all predictions and sigma are set to 0.
\nBecause I initialized my form this way and there was a problem with my code, the form stayed that way and submitted.
\nThe problem was that I was accessing a saved copy of the test.csv table and not the original one. This prohibited my code from running correctly on the private set.
\nBut this might not be exactly your problem.. I encourage you the check whether theres a problem with the code populating your table so it stays in the 0/0 state.
Hi! @omerhacohen make sure your submission file is populated in the right order.
", "votes": null, "replies": [] }, { "id": 981954, "author_name": "eliasgreen", "author_url": "", "post_date": "08/22/2020 22:07:59", "content": "Could you please tell us about your CV strategy? Cause it could be wrong (as in my case)
", "votes": null, "replies": [] }, { "id": 982014, "author_name": "jjinho", "author_url": "", "post_date": "08/23/2020 00:25:57", "content": "That's the score when your FVC predictions and sigma hit the bounds (1000 and 70).
", "votes": null, "replies": [] }, { "id": 982056, "author_name": "srikanthpotukuchi", "author_url": "", "post_date": "08/23/2020 02:30:22", "content": "How many rows in your submission file? Do a print of some rows for submission data frame and see if the values populated correctly.
", "votes": null, "replies": [] }, { "id": 982621, "author_name": "omerhacohen", "author_url": "", "post_date": "08/23/2020 14:26:39", "content": "@cv13j0 What does it mean in the right order?
\nMust I populate it so every patient has patientX_1, patientY_1 and so on or can I do patientX_1 patientX_2 and so on..?
@omerhacohen Can you provide a snapshot of your Submission.csv file that you get when you commit your notebook? It would be easier for everyone else to identify your problem that way.
", "votes": null, "replies": [] }, { "id": 982641, "author_name": "omerhacohen", "author_url": "", "post_date": "08/23/2020 14:43:31", "content": "Thats a snapshot of my submission csv:
", "votes": null, "replies": [] }, { "id": 982657, "author_name": "yovinyahathugoda", "author_url": "", "post_date": "08/23/2020 15:01:04", "content": "I see that from your post below that you are saving the dicom images as jpeg in a folder you create in /kaggle/working/. I think you are running out of disk space when running on the private test set and only part of your images are being saved. Since you are initializing your dataframe with zeros i think the scoring is done on this dataframe with zeros.
I would suggest you save your jpeg images in a folder in the kaggle root folder such as /kaggle/test_images/. You have access to about 2TB storage in the /kaggle/ directory compared to the 5GB in the /kaggle/working/ directory. The submission.csv file format that you posted looks fine.
Thats why I only save one patients photos, predict them on the csv file and then delete them. You can see the !rm -rf test at the end… Do you think its still the case?
", "votes": null, "replies": [] }, { "id": 982853, "author_name": "omerhacohen", "author_url": "", "post_date": "08/23/2020 18:04:53", "content": "Tried saving it at the root folder… still doesnt work
", "votes": null, "replies": [] }, { "id": 983488, "author_name": "yovinyahathugoda", "author_url": "", "post_date": "08/24/2020 10:56:17", "content": "@omerhacohen I didn't see the line where you delete the saved image. I have another question, why do you set your confidence values to 200 instead of predicting them. In my experience i found that the confidence value impacts the submission score significantly. If your model is overconfident or under-confident for a specific FVC value you will penalized harshly. Your problem may be due to you setting all confidence values to a certain value.
", "votes": null, "replies": [] }, { "id": 983766, "author_name": "omerhacohen", "author_url": "", "post_date": "08/24/2020 15:24:20", "content": "I tried to put up a baseline model and then predict confidence… My validation score with a constant 200 is 6.9.. And -24.. Is the score for 0-0 submission. So theres a problem with populating the table.
\nDid you succeed with predicting the test images? Can I see the code?
Your validation score is that good because your model is probably biased and is overfitting. I'm afraid i can't share my code publicly and private sharing is prohibited. But there are many public kernels using the dicom images to predict and i would suggest you to go through them well and understand the confidence value prediction section.
", "votes": null, "replies": [] }, { "id": 983924, "author_name": "omerhacohen", "author_url": "", "post_date": "08/24/2020 17:28:49", "content": "Can you link please link some of these notebooks?
\nI cant find a notebook that transforms CT scans to submission with public score…
https://www.kaggle.com/carlossouza/end-to-end-model-ct-scans-tabular
\nhttps://www.kaggle.com/jameschapman19/pytorch-tabular-qr-histogram
@jjinho That is correct!
\nBut why does that happen?
\nFirst I initiate an empty form with confidence and FVC set to zero, and it seems that my code for populating the table doesnt work on the private set.
\nBelow is the code snippet that populates the table.
\ntest_dirs are all of the directories in the test dataset (\"Patients\")
\npredict.predict_test opens the submission form, predicts FVC and confidence for one patient and then saves it on the same place.
*** This code runs just fine on the public test set, and populates the submission table with the right results. somehow on the private set this thing doesnt run. HELP! ***
\n
This are the paths:
", "votes": null, "replies": [] }, { "id": 982761, "author_name": "jjinho", "author_url": "", "post_date": "08/23/2020 16:33:46", "content": "Hi @omerhacohen, you are calling a funciton predict_test to create the csv file? Can you show that function?
Yeah..
\nBut this one has some functions in it to… Its to messy to bring it all here
@omerhacohen Did you figure this out?
\nI am having the same issue and i am not sure what is causing it
Yea I did.
\nlike @jjinho said, -24.79 is the score you get when all predictions and sigma are set to 0.
\nBecause I initialized my form this way and there was a problem with my code, the form stayed that way and submitted.
\nThe problem was that I was accessing a saved copy of the test.csv table and not the original one. This prohibited my code from running correctly on the private set.
\nBut this might not be exactly your problem.. I encourage you the check whether theres a problem with the code populating your table so it stays in the 0/0 state.