Seasoned Kagglers might already know that the order of Ids in the submission files matters. Not sure... but after I reordered the Ids according to sample_submission.csv my score jumped from 0.088 to 0.394!
I must have missed reading this in the rules/formatting guidelines???
", "messages": [ { "id": 408659, "postDate": "2018-10-23T08:51:30.153Z", "content": "Seasoned Kagglers might already know that the order of Ids in the submission files matters. Not sure... but after I reordered the Ids according to sample_submission.csv my score jumped from 0.088 to 0.394!
I must have missed reading this in the rules/formatting guidelines???
", "rawMarkdown": "Seasoned Kagglers might already know that the order of `Ids` in the submission files matters. Not sure... but after I reordered the `Ids` according to `sample_submission.csv` my score jumped from `0.088` to `0.394`!\n\nI must have missed reading this in the rules/formatting guidelines???", "votes": 37 }, { "id": 409225, "postDate": "2018-10-24T01:16:43.657Z", "content": "Looks like they loaded the submission to a list and expected it to have the exact same order as the sample one, instead of loading everything to a lookup table.
", "rawMarkdown": "Looks like they loaded the submission to a list and expected it to have the exact same order as the sample one, instead of loading everything to a lookup table.", "votes": 1 }, { "id": 409041, "postDate": "2018-10-23T18:48:15.380Z", "content": "I confirmed this too. see my kernel.
", "rawMarkdown": "I confirmed this too. see my [kernel](https://www.kaggle.com/zhugds/does-order-of-test-ids-matter).", "votes": 1 }, { "id": 413370, "postDate": "2018-10-31T20:04:38.087Z", "content": "Saved my day.
\n\nAfter creating the dataframe that is ready for submission
\n\nsubm_df= subm_df.sort_index()
\n\ncan give you the correct order.
", "rawMarkdown": "Saved my day.\n\nAfter creating the dataframe that is ready for submission \n\nsubm_df= subm_df.sort_index()\n\ncan give you the correct order.", "votes": 2 }, { "id": 408937, "postDate": "2018-10-23T16:15:17.943Z", "content": "This doesn't make sense, could the organizers please comment on this.
", "rawMarkdown": "This doesn't make sense, could the organizers please comment on this.", "votes": 2, "replies": [ { "id": 408948, "postDate": "2018-10-23T16:21:27.960Z", "content": "Well.... someone decided to calculate the metrics using an iterator instead of making a proper dictionary with ids.
", "rawMarkdown": "Well.... someone decided to calculate the metrics using an iterator instead of making a proper dictionary with ids. ", "votes": 3 } ] }, { "id": 452893, "postDate": "2019-01-09T10:17:05.890Z", "content": "Is kaggle have solved the problem?
", "rawMarkdown": "Is kaggle have solved the problem? ", "replies": [ { "id": 453384, "postDate": "2019-01-10T05:26:01.050Z", "content": "This shouldn't be a problem anymore
", "rawMarkdown": "This shouldn't be a problem anymore" } ] }, { "id": 411726, "postDate": "2018-10-28T22:12:51.513Z", "content": "Hmm... nice observation. I always thought that there is some procedure similar to merge by Ids (sample_submission.csv and valid_submission.csv), so order of test samples really doesn't matter. \nIn any case, it is written in the settings of InClass competitions when you are the host. Now it is necessary to clarify and make the official documentation.
@venkykrishna, Is there any feedback from Kaggle Team?
", "rawMarkdown": "Hmm... nice observation. I always thought that there is some procedure similar to merge by `Ids` (`sample_submission.csv` and `valid_submission.csv`), so order of test samples really doesn't matter. \nIn any case, it is written in the settings of *InClass* competitions when you are the host. Now it is necessary to clarify and make the official documentation.\n\n@venkykrishna, Is there any feedback from Kaggle Team?" }, { "id": 409988, "postDate": "2018-10-25T06:53:04.180Z", "content": "Keen eye!
", "rawMarkdown": "Keen eye!" }, { "id": 409637, "postDate": "2018-10-24T16:18:15.857Z", "content": "The expected order of Ids and predictions is specified in sample_submission.csv and that holds for all Kaggle competitions. In many instances - depending on metrics and the type of public LB calculation - it doesn't matter if the Id order is the same as in sample, as long as all Ids are there. However, I have always submitted them in the same order.
Out of curiosity: what would be the reason in the first place for submitting Ids out of order? Maybe that's laid out somehow in kernels, but I haven't looked closely.
", "rawMarkdown": "The expected order of Ids and predictions is specified in `sample_submission.csv` and that holds for all Kaggle competitions. In many instances - depending on metrics and the type of public LB calculation - it doesn't matter if the Id order is the same as in sample, as long as all Ids are there. However, I have always submitted them in the same order.\n\nOut of curiosity: what would be the reason in the first place for submitting Ids out of order? Maybe that's laid out somehow in kernels, but I haven't looked closely.", "replies": [ { "id": 409663, "postDate": "2018-10-24T16:56:20.340Z", "content": "Do you know if the above is mentioned in the competition rules or kaggle general rules? I cannot seem to find it.
\n\nIf you read files directly from the test directory, the order will obviously not be maintained. sample_submission.csv I assumed was just a guideline for formatting.
\n\n\nDo you know if the above is mentioned in the competition rules or kaggle general rules?
\n
It doesn't seem like it is spelled out for this competition, though sample_submission.csv is meant to be a template file for everything other than the contents of predicted columns.
I have posted a quick fix for this problem here.
", "rawMarkdown": "> Do you know if the above is mentioned in the competition rules or kaggle general rules?\n\nIt doesn't seem like it is spelled out for this competition, though `sample_submission.csv` is meant to be a template file for everything other than the contents of predicted columns.\n\nI have posted a quick fix for this problem [__here__](https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69537)." } ] }, { "id": 408959, "postDate": "2018-10-23T16:44:27.660Z", "content": "If it is the case, I would like to say to the organizers that I really think about them((( I spent a week trying to figure out why my validation score is 0.5+ https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-fast-ai (and even 0.7+ with some adjustment) while the test score is 0.06( I already gave up on the competition and wanted to check it out in a month or two...
", "rawMarkdown": "If it is the case, I would like to say to the organizers that I really think about them((( I spent a week trying to figure out why my validation score is 0.5+ https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-fast-ai (and even 0.7+ with some adjustment) while the test score is 0.06( I already gave up on the competition and wanted to check it out in a month or two...", "replies": [ { "id": 408963, "postDate": "2018-10-23T16:49:15.183Z", "content": "Yep... this is the case, just got a jump from .05 to .2xx (nothing great in LB, but definitely into the competition now, and going back to known practices instead of crazy trips I was trying because of this bug).
", "rawMarkdown": "Yep... this is the case, just got a jump from .05 to .2xx (nothing great in LB, but definitely into the competition now, and going back to known practices instead of crazy trips I was trying because of this bug). ", "votes": 2 }, { "id": 409000, "postDate": "2018-10-23T17:49:15.577Z", "content": "Got my 0.432 after reordering...
", "rawMarkdown": "Got my 0.432 after reordering... ", "votes": 3 } ] }, { "id": 408936, "postDate": "2018-10-23T16:14:16.077Z", "content": "Whoa... thank goodness.... I thougt I was getting crazy for thinking all my (little) knowledge was simply useless for this competition....
", "rawMarkdown": "Whoa... thank goodness.... I thougt I was getting crazy for thinking all my (little) knowledge was simply useless for this competition.... \n\n\n" }, { "id": 408803, "postDate": "2018-10-23T13:37:30.500Z", "content": "Well spottet!
", "rawMarkdown": "Well spottet!" }, { "id": 408767, "postDate": "2018-10-23T12:33:14.810Z", "content": "Can you point where it is defined in the rules/formatting guidelines?
", "rawMarkdown": "Can you point where it is defined in the rules/formatting guidelines?", "replies": [ { "id": 408919, "postDate": "2018-10-23T15:56:04.627Z", "content": "I couldn’t find it explicitly mentioned anywhere in the competition guidelines.
", "rawMarkdown": "I couldn’t find it explicitly mentioned anywhere in the competition guidelines. " }, { "id": 409066, "postDate": "2018-10-23T19:44:04.267Z", "rawMarkdown": "", "isDeleted": true }, { "id": 409151, "postDate": "2018-10-23T20:49:57.797Z", "content": "As Daniel Möller pointed out, the discussion you are referencing was talking about the order of target NOT the order of Id. This thread is a discussion about the order of Id in the submission file.
Looks like they loaded the submission to a list and expected it to have the exact same order as the sample one, instead of loading everything to a lookup table.
", "votes": 1, "replies": [] }, { "id": 409041, "author_name": "june", "author_url": "", "post_date": "2018-10-23T18:48:15.380000", "content": "I confirmed this too. see my kernel.
", "votes": 1, "replies": [] }, { "id": 413370, "author_name": "stumpen", "author_url": "", "post_date": "2018-10-31T20:04:38.087000", "content": "Saved my day.
\n\nAfter creating the dataframe that is ready for submission
\n\nsubm_df= subm_df.sort_index()
\n\ncan give you the correct order.
", "votes": 2, "replies": [] }, { "id": 408937, "author_name": "Robert", "author_url": "", "post_date": "2018-10-23T16:15:17.943000", "content": "This doesn't make sense, could the organizers please comment on this.
", "votes": 2, "replies": [ { "id": 408948, "author_name": "Daniel Möller", "author_url": "", "post_date": "2018-10-23T16:21:27.960000", "content": "Well.... someone decided to calculate the metrics using an iterator instead of making a proper dictionary with ids.
", "votes": 3, "replies": [] } ] }, { "id": 452893, "author_name": "BenChur", "author_url": "", "post_date": "2019-01-09T10:17:05.890000", "content": "Is kaggle have solved the problem?
", "votes": 0, "replies": [ { "id": 453384, "author_name": "William Horton", "author_url": "", "post_date": "2019-01-10T05:26:01.050000", "content": "This shouldn't be a problem anymore
", "votes": 0, "replies": [] } ] }, { "id": 411726, "author_name": "Sergey Bryansky", "author_url": "", "post_date": "2018-10-28T22:12:51.513000", "content": "Hmm... nice observation. I always thought that there is some procedure similar to merge by Ids (sample_submission.csv and valid_submission.csv), so order of test samples really doesn't matter. \nIn any case, it is written in the settings of InClass competitions when you are the host. Now it is necessary to clarify and make the official documentation.
@venkykrishna, Is there any feedback from Kaggle Team?
", "votes": 0, "replies": [] }, { "id": 409988, "author_name": "Denis Kanonik", "author_url": "", "post_date": "2018-10-25T06:53:04.180000", "content": "Keen eye!
", "votes": 0, "replies": [] }, { "id": 409637, "author_name": "Tilii", "author_url": "", "post_date": "2018-10-24T16:18:15.857000", "content": "The expected order of Ids and predictions is specified in sample_submission.csv and that holds for all Kaggle competitions. In many instances - depending on metrics and the type of public LB calculation - it doesn't matter if the Id order is the same as in sample, as long as all Ids are there. However, I have always submitted them in the same order.
Out of curiosity: what would be the reason in the first place for submitting Ids out of order? Maybe that's laid out somehow in kernels, but I haven't looked closely.
", "votes": 0, "replies": [ { "id": 409663, "author_name": "Venky Krishnamani", "author_url": "", "post_date": "2018-10-24T16:56:20.340000", "content": "Do you know if the above is mentioned in the competition rules or kaggle general rules? I cannot seem to find it.
\n\nIf you read files directly from the test directory, the order will obviously not be maintained. sample_submission.csv I assumed was just a guideline for formatting.
\n\n\nDo you know if the above is mentioned in the competition rules or kaggle general rules?
\n
It doesn't seem like it is spelled out for this competition, though sample_submission.csv is meant to be a template file for everything other than the contents of predicted columns.
I have posted a quick fix for this problem here.
", "votes": 0, "replies": [] } ] }, { "id": 408959, "author_name": "Iafoss", "author_url": "", "post_date": "2018-10-23T16:44:27.660000", "content": "If it is the case, I would like to say to the organizers that I really think about them((( I spent a week trying to figure out why my validation score is 0.5+ https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-fast-ai (and even 0.7+ with some adjustment) while the test score is 0.06( I already gave up on the competition and wanted to check it out in a month or two...
", "votes": 0, "replies": [ { "id": 408963, "author_name": "Daniel Möller", "author_url": "", "post_date": "2018-10-23T16:49:15.183000", "content": "Yep... this is the case, just got a jump from .05 to .2xx (nothing great in LB, but definitely into the competition now, and going back to known practices instead of crazy trips I was trying because of this bug).
", "votes": 2, "replies": [] }, { "id": 409000, "author_name": "Iafoss", "author_url": "", "post_date": "2018-10-23T17:49:15.577000", "content": "Got my 0.432 after reordering...
", "votes": 3, "replies": [] } ] }, { "id": 408936, "author_name": "Daniel Möller", "author_url": "", "post_date": "2018-10-23T16:14:16.077000", "content": "Whoa... thank goodness.... I thougt I was getting crazy for thinking all my (little) knowledge was simply useless for this competition....
", "votes": 0, "replies": [] }, { "id": 408803, "author_name": "WolfgangReuter", "author_url": "", "post_date": "2018-10-23T13:37:30.500000", "content": "Well spottet!
", "votes": 0, "replies": [] }, { "id": 408767, "author_name": "pi-null-mezon", "author_url": "", "post_date": "2018-10-23T12:33:14.810000", "content": "Can you point where it is defined in the rules/formatting guidelines?
", "votes": 0, "replies": [ { "id": 408919, "author_name": "Venky Krishnamani", "author_url": "", "post_date": "2018-10-23T15:56:04.627000", "content": "I couldn’t find it explicitly mentioned anywhere in the competition guidelines.
", "votes": 0, "replies": [] }, { "id": 409066, "author_name": "", "author_url": "", "post_date": "2018-10-23T19:44:04.267000", "content": "", "votes": 0, "replies": [] }, { "id": 409151, "author_name": "Venky Krishnamani", "author_url": "", "post_date": "2018-10-23T20:49:57.797000", "content": "As Daniel Möller pointed out, the discussion you are referencing was talking about the order of target NOT the order of Id. This thread is a discussion about the order of Id in the submission file.