{ "id": 44731, "title": "Stage 1 Test Set Labels ", "url": "/competitions/passenger-screening-algorithm-challenge/discussion/44731", "author_name": "", "post_date": "2017-12-01T18:16:46.216068Z", "votes": 1, "comment_count": 14, "views": 0, "content": "

Somewhere in the discussions I think that there was mention that the labels for the stage 1 test set would be released prior to stage 2 in order to allow people to include those data in the training of their models. Do I remember this correctly and, if so, when will this information be released?

", "messages": [ { "id": "251793", "postDate": "12/01/2017 18:16:46", "content": "

Somewhere in the discussions I think that there was mention that the labels for the stage 1 test set would be released prior to stage 2 in order to allow people to include those data in the training of their models. Do I remember this correctly and, if so, when will this information be released?

", "rawMarkdown": "Somewhere in the discussions I think that there was mention that the labels for the stage 1 test set would be released prior to stage 2 in order to allow people to include those data in the training of their models. Do I remember this correctly and, if so, when will this information be released?", "votes": null }, { "id": "251922", "postDate": "12/01/2017 22:38:32", "content": "

It will be released together with stage 2 data I think. I will be able to use it only partially as there is not enough time between the release and the end date to retrain the entire model. I guess I could have guessed those label by now if I only have more time to spare ...

", "rawMarkdown": "It will be released together with stage 2 data I think. I will be able to use it only partially as there is not enough time between the release and the end date to retrain the entire model. I guess I could have guessed those label by now if I only have more time to spare ...", "votes": null }, { "id": "252392", "postDate": "12/02/2017 20:46:37", "content": "

You might be able to fine tune on the full stage1 set maybe?

", "rawMarkdown": "You might be able to fine tune on the full stage1 set maybe?", "votes": null }, { "id": "253487", "postDate": "12/05/2017 02:45:18", "content": "

Are the labels not already given in the \"stage1_labels.csv\" file?

", "rawMarkdown": "Are the labels not already given in the \"stage1_labels.csv\" file?", "votes": null }, { "id": "253523", "postDate": "12/05/2017 05:36:48", "content": "

no, the test set is the sample submission file without labels that you are generating probabilities for submission with

", "rawMarkdown": "no, the test set is the sample submission file without labels that you are generating probabilities for submission with", "votes": null }, { "id": "253931", "postDate": "12/05/2017 21:51:32", "content": "

Could you please tell where is the test dataset as listed in the sample submission csv file?

", "rawMarkdown": "Could you please tell where is the test dataset as listed in the sample submission csv file?", "votes": null }, { "id": "253949", "postDate": "12/05/2017 22:34:24", "content": "

The train/test images are all in the same image folder. The test images are the images with ids that match up with the ids in the stage 1 sample submission file. The train image ids are in the stage 1 labels csv file.

\n\n
\n

Wensu wrote

\n \n
\n

Could you please tell where is the test dataset as listed in the sample submission csv file?

\n
\n
", "rawMarkdown": "The train/test images are all in the same image folder. The test images are the images with ids that match up with the ids in the stage 1 sample submission file. The train image ids are in the stage 1 labels csv file. \n\n\n> **Wensu wrote**\n> \n> > Could you please tell where is the test dataset as listed in the sample submission csv file?", "votes": null }, { "id": "254483", "postDate": "12/07/2017 02:48:31", "content": "

could any please tell in the test data submission file, do we need to give the actual probability values predicted from model or need to convert them to 1 or 0?

", "rawMarkdown": "could any please tell in the test data submission file, do we need to give the actual probability values predicted from model or need to convert them to 1 or 0?", "votes": null }, { "id": "254485", "postDate": "12/07/2017 03:01:00", "content": "

Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?

", "rawMarkdown": "Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?", "votes": null }, { "id": "254486", "postDate": "12/07/2017 03:04:06", "content": "

Its probably better to submit the probability itself, but if you prefer to submit thresholded predictions, you are welcome to do so. You won't break the evaluation by doing so. Be warned that thresholding doesn't typically reduce your log_loss score though. You are probably better off submitting the predictions themselves.

", "rawMarkdown": "Its probably better to submit the probability itself, but if you prefer to submit thresholded predictions, you are welcome to do so. You won't break the evaluation by doing so. Be warned that thresholding doesn't typically reduce your log_loss score though. You are probably better off submitting the predictions themselves.", "votes": null }, { "id": "254487", "postDate": "12/07/2017 03:04:38", "content": "

Wensu,

\n\n

The predicted probability values from the model are the values that you should be submitting. Hope this helps.

", "rawMarkdown": "Wensu,\n\nThe predicted probability values from the model are the values that you should be submitting. Hope this helps.", "votes": null }, { "id": "254515", "postDate": "12/07/2017 05:09:55", "content": "

if they are the predicted probability values, why several top teams in the leaderboard got log-loss scores are zero which seems impossible, I am confused a bit.

", "rawMarkdown": "if they are the predicted probability values, why several top teams in the leaderboard got log-loss scores are zero which seems impossible, I am confused a bit.", "votes": null }, { "id": "254516", "postDate": "12/07/2017 05:13:54", "content": "

Its possible they are thresholding and have models that are really good. Its also possible they are just using probing/hand labeling in order to give them an edge on the stage 2 phase by expanding their training set with test set labels prior to the start of stage 2.

", "rawMarkdown": "Its possible they are thresholding and have models that are really good. Its also possible they are just using probing/hand labeling in order to give them an edge on the stage 2 phase by expanding their training set with test set labels prior to the start of stage 2.", "votes": null }, { "id": "254518", "postDate": "12/07/2017 05:15:28", "content": "

I personally recommend submitting the probability values that were predicted from your model.

\n\n

While you could use a threshold to set all positive probabilities to 1 and negatives to 0, this is pretty risky because more than likely there will be \"false positives\" and \"false negatives\" in your predictions so doing this could end up generating a worse overall score than just submitting the probabilities directly.

\n\n
\n

Wensu wrote

\n \n
\n

Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?

\n
\n
", "rawMarkdown": "I personally recommend submitting the probability values that were predicted from your model. \n\nWhile you could use a threshold to set all positive probabilities to 1 and negatives to 0, this is pretty risky because more than likely there will be \"false positives\" and \"false negatives\" in your predictions so doing this could end up generating a worse overall score than just submitting the probabilities directly. \n\n> **Wensu wrote**\n> \n> > Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?", "votes": null }, { "id": "254529", "postDate": "12/07/2017 05:53:20", "content": "

You are right, the minimum theoretical cross-entropy one could get is -9.9920072216264148e-16, and this is when you have all predictions correct and thresholded it to 0 an 1. Maybe their score is far worse, in either case rounded to 5 decimals its zero.

", "rawMarkdown": "You are right, the minimum theoretical cross-entropy one could get is -9.9920072216264148e-16, and this is when you have all predictions correct and thresholded it to 0 an 1. Maybe their score is far worse, in either case rounded to 5 decimals its zero.", "votes": null } ], "comments": [ { "id": 251922, "author_name": "sarkadiu", "author_url": "", "post_date": "12/01/2017 22:38:32", "content": "

It will be released together with stage 2 data I think. I will be able to use it only partially as there is not enough time between the release and the end date to retrain the entire model. I guess I could have guessed those label by now if I only have more time to spare ...

", "votes": null, "replies": [ { "id": 252392, "author_name": "", "author_url": "", "post_date": "12/02/2017 20:46:37", "content": "

You might be able to fine tune on the full stage1 set maybe?

", "votes": null, "replies": [] } ] }, { "id": 253487, "author_name": "nikhilphatak", "author_url": "", "post_date": "12/05/2017 02:45:18", "content": "

Are the labels not already given in the \"stage1_labels.csv\" file?

", "votes": null, "replies": [ { "id": 253523, "author_name": "nickkimer", "author_url": "", "post_date": "12/05/2017 05:36:48", "content": "

no, the test set is the sample submission file without labels that you are generating probabilities for submission with

", "votes": null, "replies": [] } ] }, { "id": 253931, "author_name": "w9wang2", "author_url": "", "post_date": "12/05/2017 21:51:32", "content": "

Could you please tell where is the test dataset as listed in the sample submission csv file?

", "votes": null, "replies": [ { "id": 253949, "author_name": "jamesrequa", "author_url": "", "post_date": "12/05/2017 22:34:24", "content": "

The train/test images are all in the same image folder. The test images are the images with ids that match up with the ids in the stage 1 sample submission file. The train image ids are in the stage 1 labels csv file.

\n\n
\n

Wensu wrote

\n \n
\n

Could you please tell where is the test dataset as listed in the sample submission csv file?

\n
\n
", "votes": null, "replies": [] }, { "id": 254485, "author_name": "w9wang2", "author_url": "", "post_date": "12/07/2017 03:01:00", "content": "

Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?

", "votes": null, "replies": [] }, { "id": 254518, "author_name": "jamesrequa", "author_url": "", "post_date": "12/07/2017 05:15:28", "content": "

I personally recommend submitting the probability values that were predicted from your model.

\n\n

While you could use a threshold to set all positive probabilities to 1 and negatives to 0, this is pretty risky because more than likely there will be \"false positives\" and \"false negatives\" in your predictions so doing this could end up generating a worse overall score than just submitting the probabilities directly.

\n\n
\n

Wensu wrote

\n \n
\n

Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?

\n
\n
", "votes": null, "replies": [] } ] }, { "id": 254483, "author_name": "w9wang2", "author_url": "", "post_date": "12/07/2017 02:48:31", "content": "

could any please tell in the test data submission file, do we need to give the actual probability values predicted from model or need to convert them to 1 or 0?

", "votes": null, "replies": [ { "id": 254486, "author_name": "", "author_url": "", "post_date": "12/07/2017 03:04:06", "content": "

Its probably better to submit the probability itself, but if you prefer to submit thresholded predictions, you are welcome to do so. You won't break the evaluation by doing so. Be warned that thresholding doesn't typically reduce your log_loss score though. You are probably better off submitting the predictions themselves.

", "votes": null, "replies": [] }, { "id": 254487, "author_name": "nickkimer", "author_url": "", "post_date": "12/07/2017 03:04:38", "content": "

Wensu,

\n\n

The predicted probability values from the model are the values that you should be submitting. Hope this helps.

", "votes": null, "replies": [] }, { "id": 254515, "author_name": "w9wang2", "author_url": "", "post_date": "12/07/2017 05:09:55", "content": "

if they are the predicted probability values, why several top teams in the leaderboard got log-loss scores are zero which seems impossible, I am confused a bit.

", "votes": null, "replies": [] }, { "id": 254516, "author_name": "", "author_url": "", "post_date": "12/07/2017 05:13:54", "content": "

Its possible they are thresholding and have models that are really good. Its also possible they are just using probing/hand labeling in order to give them an edge on the stage 2 phase by expanding their training set with test set labels prior to the start of stage 2.

", "votes": null, "replies": [] }, { "id": 254529, "author_name": "bastiaanbergman", "author_url": "", "post_date": "12/07/2017 05:53:20", "content": "

You are right, the minimum theoretical cross-entropy one could get is -9.9920072216264148e-16, and this is when you have all predictions correct and thresholded it to 0 an 1. Maybe their score is far worse, in either case rounded to 5 decimals its zero.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "251793": "Somewhere in the discussions I think that there was mention that the labels for the stage 1 test set would be released prior to stage 2 in order to allow people to include those data in the training of their models. Do I remember this correctly and, if so, when will this information be released?", "251922": "It will be released together with stage 2 data I think. I will be able to use it only partially as there is not enough time between the release and the end date to retrain the entire model. I guess I could have guessed those label by now if I only have more time to spare ...", "252392": "You might be able to fine tune on the full stage1 set maybe?", "253487": "Are the labels not already given in the \"stage1_labels.csv\" file?", "253523": "no, the test set is the sample submission file without labels that you are generating probabilities for submission with", "253931": "Could you please tell where is the test dataset as listed in the sample submission csv file?", "253949": "The train/test images are all in the same image folder. The test images are the images with ids that match up with the ids in the stage 1 sample submission file. The train image ids are in the stage 1 labels csv file. \n\n\n> **Wensu wrote**\n> \n> > Could you please tell where is the test dataset as listed in the sample submission csv file?", "254483": "could any please tell in the test data submission file, do we need to give the actual probability values predicted from model or need to convert them to 1 or 0?", "254485": "Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?", "254486": "Its probably better to submit the probability itself, but if you prefer to submit thresholded predictions, you are welcome to do so. You won't break the evaluation by doing so. Be warned that thresholding doesn't typically reduce your log_loss score though. You are probably better off submitting the predictions themselves.", "254487": "Wensu,\n\nThe predicted probability values from the model are the values that you should be submitting. Hope this helps.", "254515": "if they are the predicted probability values, why several top teams in the leaderboard got log-loss scores are zero which seems impossible, I am confused a bit.", "254516": "Its possible they are thresholding and have models that are really good. Its also possible they are just using probing/hand labeling in order to give them an edge on the stage 2 phase by expanding their training set with test set labels prior to the start of stage 2.", "254518": "I personally recommend submitting the probability values that were predicted from your model. \n\nWhile you could use a threshold to set all positive probabilities to 1 and negatives to 0, this is pretty risky because more than likely there will be \"false positives\" and \"false negatives\" in your predictions so doing this could end up generating a worse overall score than just submitting the probabilities directly. \n\n> **Wensu wrote**\n> \n> > Thank you James. one more question, in the test data submission file, do we need to submit the actual probability values predicted from model or need to convert them to 1 or 0, then submit?", "254529": "You are right, the minimum theoretical cross-entropy one could get is -9.9920072216264148e-16, and this is when you have all predictions correct and thresholded it to 0 an 1. Maybe their score is far worse, in either case rounded to 5 decimals its zero." }, "source": "meta" }