Recently, some of notebooks scored very low. To find any bugs, I set all of the configurations of these notebooks the same as a previous high scoring notebook, which scored 0.481. The latest notebook and the high scoring notebook produced the same exact csv files. I even checked a couple of times with pandas .equals() function. But, while one scores 0.481, the other scores 0.136. Why would this even happen?
\nIf possible, can a competition host run the metric on the public test set to check if they give the same scores? @weiouyang @uaxelsson @cwinsnes @lnhtrang Sorry for the tags, but these last few submissions are really important for me. These are the two links to both of the submission.csv files I submitted:
\nHigh scoring: https://drive.google.com/file/d/1G5WDuJBD5MTh2-t4MOXB_uG0RTpf3X_Q/view?usp=sharing
\nLow scoring: https://drive.google.com/file/d/1mnsWdrkWwJYfvD_E_XEurdhtDX3hjlCa/view?usp=sharing
", "messages": [ { "id": 1302333, "postDate": "2021-05-11T13:50:52.383Z", "content": "I got the impression that when you submit, your code reruns on public + private test, produces one combined csv file, and then the public scoring happens on the public part of that one csv. If this is the case, you'd never know whether your submissions are the same, because you would never see that csv file. The files that you produce when committing the notebook only see public test, and therefore could be the same for 2 differently working in general notebooks, but the scoring never happens on the outputs you can actually see. That's my theory at least
", "rawMarkdown": "I got the impression that when you submit, your code reruns on public + private test, produces one combined csv file, and then the public scoring happens on the public part of that one csv. If this is the case, you'd never know whether your submissions are the same, because you would never see that csv file. The files that you produce when committing the notebook only see public test, and therefore could be the same for 2 differently working in general notebooks, but the scoring never happens on the outputs you can actually see. That's my theory at least", "votes": 1 }, { "id": 1300073, "postDate": "2021-05-10T08:51:55.457Z", "content": "Hi! @weiouyang and I have checked your last 2 submissions, one with 0.481 and one with 0.136, and they are different (your 2 files on drive are the same). So you might want to check your notebook again.
\nPlease also note that you haven't submitted anything on private leaderboard yet as far as we can see, so don't forget to do that before deadline :)
Hey @lnhtrang the latest submission is one that I made after a few hours of writing this post. My second latest submission (the submission before the latest one with 0.136 - version 46) and my 7th latest submission (submitted 2 days ago with 0.481 score - version 38) are the ones I was referring to in the post. I can assure you that they have the exact same submission files. Can you please test the outputs of these two notebooks? I can also let you know the exact names of the notebooks if that helps.
", "rawMarkdown": "Hey @lnhtrang the latest submission is one that I made after a few hours of writing this post. My second latest submission (the submission before the latest one with 0.136 - version 46) and my 7th latest submission (submitted 2 days ago with 0.481 score - version 38) are the ones I was referring to in the post. I can assure you that they have the exact same submission files. Can you please test the outputs of these two notebooks? I can also let you know the exact names of the notebooks if that helps." }, { "id": 1300083, "postDate": "2021-05-10T09:01:26.010Z", "content": "And yes, I will make sure to make submissions that will work for the private test set. I've been putting that off for a long time.
", "rawMarkdown": "And yes, I will make sure to make submissions that will work for the private test set. I've been putting that off for a long time." }, { "id": 1300091, "postDate": "2021-05-10T09:08:43.523Z", "content": "You should do that asap - I started with it yesterday , and my submissions are taking way too long to run in the time limit. After fixing that , got quite a few OOM issues. I think I have finally managed to do it. We will know only after 7 hours.
", "rawMarkdown": "You should do that asap - I started with it yesterday , and my submissions are taking way too long to run in the time limit. After fixing that , got quite a few OOM issues. I think I have finally managed to do it. We will know only after 7 hours. ", "votes": 1 }, { "id": 1300097, "postDate": "2021-05-10T09:11:59.773Z", "content": "Yup, will use most of my remaining submissions today and all my submissions tomorrow for submitting on the private set. I've also not used much of my GPU quota. So, I'm hoping I should be fine.
", "rawMarkdown": "Yup, will use most of my remaining submissions today and all my submissions tomorrow for submitting on the private set. I've also not used much of my GPU quota. So, I'm hoping I should be fine. " }, { "id": 1300109, "postDate": "2021-05-10T09:17:12.027Z", "content": "Yes, the only issue is debugging , and the amount of time a full submission takes. I have been super stressed out about it since yesterday. Lesson learnt!
", "rawMarkdown": "Yes, the only issue is debugging , and the amount of time a full submission takes. I have been super stressed out about it since yesterday. Lesson learnt!" }, { "id": 1300129, "postDate": "2021-05-10T09:33:35.677Z", "content": "I checked your 2nd (low, submitted 5/10/21 0:47) and 7th (high, submitted 5/08/21 20:06) latest submissions, and their contents are still different. One obvious difference is the size, your low score is 43MB, high score is 140MB. Could it be it got truncated when submitting?
", "rawMarkdown": "I checked your 2nd (low, submitted 5/10/21 0:47) and 7th (high, submitted 5/08/21 20:06) latest submissions, and their contents are still different. One obvious difference is the size, your low score is 43MB, high score is 140MB. Could it be it got truncated when submitting?", "votes": 1 }, { "id": 1300143, "postDate": "2021-05-10T09:44:20.127Z", "content": "That's weird. For me, they are the same csvs - same contents and same size. I'm starting to think this was a server side issue since you said that one is 43Mb. One reason could be that the kaggle servers are being overwhelmed since a lot of competitions are ending around this time. I hope something like this doesn't happen for the submissions of the private test set. Anyway, thank you so much for looking into this.
", "rawMarkdown": "That's weird. For me, they are the same csvs - same contents and same size. I'm starting to think this was a server side issue since you said that one is 43Mb. One reason could be that the kaggle servers are being overwhelmed since a lot of competitions are ending around this time. I hope something like this doesn't happen for the submissions of the private test set. Anyway, thank you so much for looking into this." }, { "id": 1300934, "postDate": "2021-05-10T21:00:20.357Z", "content": "@novice03 are you putting all the cropped images in the kaggle/working folder? There is a size limit there and it doesn't crash when you fill it up? This happened to me… I just used another folder when needed more space …
\nor are you creating this CSV locally?
", "rawMarkdown": "@novice03 are you putting all the cropped images in the kaggle/working folder? There is a size limit there and it doesn't crash when you fill it up? This happened to me... I just used another folder when needed more space ...\n\nor are you creating this CSV locally?" }, { "id": 1300942, "postDate": "2021-05-10T21:08:51.730Z", "content": "Yes I am storing all cropped images in a folder. I am running my notebook on private test set now, and so far there have been no errors. Will update if anything shows up. But, I think 20gb should be enough.
", "rawMarkdown": "Yes I am storing all cropped images in a folder. I am running my notebook on private test set now, and so far there have been no errors. Will update if anything shows up. But, I think 20gb should be enough." }, { "id": 1300944, "postDate": "2021-05-10T21:16:50.420Z", "content": "But, for the notebook I am talking about in this post, I only made submission for public test set. I didn't store cropped images at all. Just took a preds.pickle file and made it into csv.
", "rawMarkdown": "But, for the notebook I am talking about in this post, I only made submission for public test set. I didn't store cropped images at all. Just took a preds.pickle file and made it into csv." }, { "id": 1300947, "postDate": "2021-05-10T21:19:55.840Z", "content": "sounds strange … no other ideas from me unfortunately … hope you won't have a problem with the final submission process… good luck!
", "rawMarkdown": "sounds strange ... no other ideas from me unfortunately ... hope you won't have a problem with the final submission process... good luck!" }, { "id": 1300960, "postDate": "2021-05-10T21:31:42.423Z", "content": "Good luck!
", "rawMarkdown": "Good luck!", "votes": 1 } ] }, { "id": 1299869, "postDate": "2021-05-10T05:20:10.617Z", "content": "Recently, some of notebooks scored very low. To find any bugs, I set all of the configurations of these notebooks the same as a previous high scoring notebook, which scored 0.481. The latest notebook and the high scoring notebook produced the same exact csv files. I even checked a couple of times with pandas .equals() function. But, while one scores 0.481, the other scores 0.136. Why would this even happen?
\nIf possible, can a competition host run the metric on the public test set to check if they give the same scores? @weiouyang @uaxelsson @cwinsnes @lnhtrang Sorry for the tags, but these last few submissions are really important for me. These are the two links to both of the submission.csv files I submitted:
\nHigh scoring: https://drive.google.com/file/d/1G5WDuJBD5MTh2-t4MOXB_uG0RTpf3X_Q/view?usp=sharing
\nLow scoring: https://drive.google.com/file/d/1mnsWdrkWwJYfvD_E_XEurdhtDX3hjlCa/view?usp=sharing
", "rawMarkdown": "Recently, some of notebooks scored very low. To find any bugs, I set all of the configurations of these notebooks the same as a previous high scoring notebook, which scored 0.481. The latest notebook and the high scoring notebook produced the same exact csv files. I even checked a couple of times with pandas .equals() function. But, while one scores 0.481, the other scores 0.136. Why would this even happen?\n\nIf possible, can a competition host run the metric on the public test set to check if they give the same scores? @weiouyang @uaxelsson @cwinsnes @lnhtrang Sorry for the tags, but these last few submissions are really important for me. These are the two links to both of the submission.csv files I submitted:\n\nHigh scoring: https://drive.google.com/file/d/1G5WDuJBD5MTh2-t4MOXB_uG0RTpf3X_Q/view?usp=sharing\n\nLow scoring: https://drive.google.com/file/d/1mnsWdrkWwJYfvD_E_XEurdhtDX3hjlCa/view?usp=sharing\n", "votes": 2 }, { "id": 1300166, "postDate": "2021-05-10T10:05:37.577Z", "content": "Do your csv-s have predictions with exact same confidences for some of them? While scoring … the sorting of same-confidence prediction might differ… just a thought…
", "rawMarkdown": "Do your csv-s have predictions with exact same confidences for some of them? While scoring ... the sorting of same-confidence prediction might differ... just a thought...", "replies": [ { "id": 1300175, "postDate": "2021-05-10T10:09:24.810Z", "content": "Yup, the csvs are exactly the same. All the predictions strings in all the 559 is exactly the same. I think this is some server side issue.
", "rawMarkdown": "Yup, the csvs are exactly the same. All the predictions strings in all the 559 is exactly the same. I think this is some server side issue." }, { "id": 1300203, "postDate": "2021-05-10T10:27:42.343Z", "content": "I think we misunderstood eachother… do you round your predictions? So multiple predictions get the same score (e.g. 0.7)…? And not 0.700001, 0.700201, 0.70005…
", "rawMarkdown": "I think we misunderstood eachother... do you round your predictions? So multiple predictions get the same score (e.g. 0.7)...? And not 0.700001, 0.700201, 0.70005..." }, { "id": 1300206, "postDate": "2021-05-10T10:30:24.410Z", "content": "No, I did not round the predictions. All confidence scores are equal down to the last decimal place in both submissions. You can see the two csv files I attached in my post if you'd like.
", "rawMarkdown": "No, I did not round the predictions. All confidence scores are equal down to the last decimal place in both submissions. You can see the two csv files I attached in my post if you'd like." }, { "id": 1300212, "postDate": "2021-05-10T10:35:27.983Z", "content": "I am a bit in a rush at the moment … if I get some free time I will check it out in more depth … can't promise anything …
", "rawMarkdown": "I am a bit in a rush at the moment ... if I get some free time I will check it out in more depth ... can't promise anything ..." } ] }, { "id": 1300013, "postDate": "2021-05-10T07:54:54.683Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1300015, "postDate": "2021-05-10T07:55:43.263Z", "content": "Someone else in this forum also had different score and I also experienced for my last running notebook different scores with the same code…
", "rawMarkdown": "Someone else in this forum also had different score and I also experienced for my last running notebook different scores with the same code..." }, { "id": 1300044, "postDate": "2021-05-10T08:26:13.970Z", "content": "That is indeed a very crazy theory. Whatever it is, it is very strange. I hope we get to know what's happening. I submitted another notebook just now for which I thought I would get a high score, but I got the exact same score as my best submission (0.481). I don't know if these two are linked. But, I think its safe to not submit anything for some time.
", "rawMarkdown": "That is indeed a very crazy theory. Whatever it is, it is very strange. I hope we get to know what's happening. I submitted another notebook just now for which I thought I would get a high score, but I got the exact same score as my best submission (0.481). I don't know if these two are linked. But, I think its safe to not submit anything for some time." } ] }, { "id": 1299915, "postDate": "2021-05-10T06:29:15.150Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1302333, "author_name": "fumitoks", "author_url": "", "post_date": "2021-05-11T13:50:52.383000", "content": "I got the impression that when you submit, your code reruns on public + private test, produces one combined csv file, and then the public scoring happens on the public part of that one csv. If this is the case, you'd never know whether your submissions are the same, because you would never see that csv file. The files that you produce when committing the notebook only see public test, and therefore could be the same for 2 differently working in general notebooks, but the scoring never happens on the outputs you can actually see. That's my theory at least
", "votes": 1, "replies": [] }, { "id": 1300073, "author_name": "Trang Le", "author_url": "", "post_date": "2021-05-10T08:51:55.457000", "content": "Hi! @weiouyang and I have checked your last 2 submissions, one with 0.481 and one with 0.136, and they are different (your 2 files on drive are the same). So you might want to check your notebook again.
\nPlease also note that you haven't submitted anything on private leaderboard yet as far as we can see, so don't forget to do that before deadline :)
Hey @lnhtrang the latest submission is one that I made after a few hours of writing this post. My second latest submission (the submission before the latest one with 0.136 - version 46) and my 7th latest submission (submitted 2 days ago with 0.481 score - version 38) are the ones I was referring to in the post. I can assure you that they have the exact same submission files. Can you please test the outputs of these two notebooks? I can also let you know the exact names of the notebooks if that helps.
", "votes": 0, "replies": [] }, { "id": 1300083, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T09:01:26.010000", "content": "And yes, I will make sure to make submissions that will work for the private test set. I've been putting that off for a long time.
", "votes": 0, "replies": [] }, { "id": 1300091, "author_name": "Satwik", "author_url": "", "post_date": "2021-05-10T09:08:43.523000", "content": "You should do that asap - I started with it yesterday , and my submissions are taking way too long to run in the time limit. After fixing that , got quite a few OOM issues. I think I have finally managed to do it. We will know only after 7 hours.
", "votes": 1, "replies": [] }, { "id": 1300097, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T09:11:59.773000", "content": "Yup, will use most of my remaining submissions today and all my submissions tomorrow for submitting on the private set. I've also not used much of my GPU quota. So, I'm hoping I should be fine.
", "votes": 0, "replies": [] }, { "id": 1300109, "author_name": "Satwik", "author_url": "", "post_date": "2021-05-10T09:17:12.027000", "content": "Yes, the only issue is debugging , and the amount of time a full submission takes. I have been super stressed out about it since yesterday. Lesson learnt!
", "votes": 0, "replies": [] }, { "id": 1300129, "author_name": "Trang Le", "author_url": "", "post_date": "2021-05-10T09:33:35.677000", "content": "I checked your 2nd (low, submitted 5/10/21 0:47) and 7th (high, submitted 5/08/21 20:06) latest submissions, and their contents are still different. One obvious difference is the size, your low score is 43MB, high score is 140MB. Could it be it got truncated when submitting?
", "votes": 1, "replies": [] }, { "id": 1300143, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T09:44:20.127000", "content": "That's weird. For me, they are the same csvs - same contents and same size. I'm starting to think this was a server side issue since you said that one is 43Mb. One reason could be that the kaggle servers are being overwhelmed since a lot of competitions are ending around this time. I hope something like this doesn't happen for the submissions of the private test set. Anyway, thank you so much for looking into this.
", "votes": 0, "replies": [] }, { "id": 1300934, "author_name": "CroDoc", "author_url": "", "post_date": "2021-05-10T21:00:20.357000", "content": "@novice03 are you putting all the cropped images in the kaggle/working folder? There is a size limit there and it doesn't crash when you fill it up? This happened to me… I just used another folder when needed more space …
\nor are you creating this CSV locally?
", "votes": 0, "replies": [] }, { "id": 1300942, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T21:08:51.730000", "content": "Yes I am storing all cropped images in a folder. I am running my notebook on private test set now, and so far there have been no errors. Will update if anything shows up. But, I think 20gb should be enough.
", "votes": 0, "replies": [] }, { "id": 1300944, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T21:16:50.420000", "content": "But, for the notebook I am talking about in this post, I only made submission for public test set. I didn't store cropped images at all. Just took a preds.pickle file and made it into csv.
", "votes": 0, "replies": [] }, { "id": 1300947, "author_name": "CroDoc", "author_url": "", "post_date": "2021-05-10T21:19:55.840000", "content": "sounds strange … no other ideas from me unfortunately … hope you won't have a problem with the final submission process… good luck!
", "votes": 0, "replies": [] }, { "id": 1300960, "author_name": "Trang Le", "author_url": "", "post_date": "2021-05-10T21:31:42.423000", "content": "Good luck!
", "votes": 1, "replies": [] } ] }, { "id": 1300166, "author_name": "CroDoc", "author_url": "", "post_date": "2021-05-10T10:05:37.577000", "content": "Do your csv-s have predictions with exact same confidences for some of them? While scoring … the sorting of same-confidence prediction might differ… just a thought…
", "votes": 0, "replies": [ { "id": 1300175, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T10:09:24.810000", "content": "Yup, the csvs are exactly the same. All the predictions strings in all the 559 is exactly the same. I think this is some server side issue.
", "votes": 0, "replies": [] }, { "id": 1300203, "author_name": "CroDoc", "author_url": "", "post_date": "2021-05-10T10:27:42.343000", "content": "I think we misunderstood eachother… do you round your predictions? So multiple predictions get the same score (e.g. 0.7)…? And not 0.700001, 0.700201, 0.70005…
", "votes": 0, "replies": [] }, { "id": 1300206, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T10:30:24.410000", "content": "No, I did not round the predictions. All confidence scores are equal down to the last decimal place in both submissions. You can see the two csv files I attached in my post if you'd like.
", "votes": 0, "replies": [] }, { "id": 1300212, "author_name": "CroDoc", "author_url": "", "post_date": "2021-05-10T10:35:27.983000", "content": "I am a bit in a rush at the moment … if I get some free time I will check it out in more depth … can't promise anything …
", "votes": 0, "replies": [] } ] }, { "id": 1300013, "author_name": "", "author_url": "", "post_date": "2021-05-10T07:54:54.683000", "content": "", "votes": 1, "replies": [ { "id": 1300015, "author_name": "Alexander Riedel", "author_url": "", "post_date": "2021-05-10T07:55:43.263000", "content": "Someone else in this forum also had different score and I also experienced for my last running notebook different scores with the same code…
", "votes": 0, "replies": [] }, { "id": 1300044, "author_name": "novice03", "author_url": "", "post_date": "2021-05-10T08:26:13.970000", "content": "That is indeed a very crazy theory. Whatever it is, it is very strange. I hope we get to know what's happening. I submitted another notebook just now for which I thought I would get a high score, but I got the exact same score as my best submission (0.481). I don't know if these two are linked. But, I think its safe to not submit anything for some time.
", "votes": 0, "replies": [] } ] }, { "id": 1299915, "author_name": "", "author_url": "", "post_date": "2021-05-10T06:29:15.150000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1302333": "I got the impression that when you submit, your code reruns on public + private test, produces one combined csv file, and then the public scoring happens on the public part of that one csv. If this is the case, you'd never know whether your submissions are the same, because you would never see that csv file. The files that you produce when committing the notebook only see public test, and therefore could be the same for 2 differently working in general notebooks, but the scoring never happens on the outputs you can actually see. That's my theory at least", "1300073": "Hi! @weiouyang and I have checked your last 2 submissions, one with 0.481 and one with 0.136, and they are different (your 2 files on drive are the same). So you might want to check your notebook again.\nPlease also note that you haven't submitted anything on private leaderboard yet as far as we can see, so don't forget to do that before deadline :) ", "1299869": "Recently, some of notebooks scored very low. To find any bugs, I set all of the configurations of these notebooks the same as a previous high scoring notebook, which scored 0.481. The latest notebook and the high scoring notebook produced the same exact csv files. I even checked a couple of times with pandas .equals() function. But, while one scores 0.481, the other scores 0.136. Why would this even happen?\n\nIf possible, can a competition host run the metric on the public test set to check if they give the same scores? @weiouyang @uaxelsson @cwinsnes @lnhtrang Sorry for the tags, but these last few submissions are really important for me. These are the two links to both of the submission.csv files I submitted:\n\nHigh scoring: https://drive.google.com/file/d/1G5WDuJBD5MTh2-t4MOXB_uG0RTpf3X_Q/view?usp=sharing\n\nLow scoring: https://drive.google.com/file/d/1mnsWdrkWwJYfvD_E_XEurdhtDX3hjlCa/view?usp=sharing\n", "1300166": "Do your csv-s have predictions with exact same confidences for some of them? While scoring ... the sorting of same-confidence prediction might differ... just a thought...", "1300013": "", "1299915": "" } }