{ "id": 13345, "title": "Is unsupervised pre-training on unlabeled (i.e. test data) permitted?", "url": "/competitions/diabetic-retinopathy-detection/discussion/13345", "author_name": "LAD", "post_date": "2015-04-13T23:09:26.693000", "votes": 4, "comment_count": 15, "views": 3655, "content": "

After reading the rules and searching the forums for any occurrence of "supervise" or "unlabeled," the only statement I could come up with related to the titular question was in the rules:

\n

"Semi-supervised learning is permitted."

\n

Just to make the matter explicitly clear so we all know the bounds in which we are operating, is it permissible to pre-train in an unsupervised fashion on the unlabeled data the test set represents? If I missed the already existing answer somewhere and any fellow Kaggler can point me to it, or if I can get an answer direct from competition management, I'd greatly appreciate it!

\n

Thanks so much, and sorry to be a pest! Just trying to avoid a lot of wasted effort if such actions would be frowned upon :)

", "messages": [ { "id": 72711, "postDate": "2015-04-20T18:30:14.540Z", "content": "

Milk and carrots are allowed, but will place you north of 200th place. You'll want to switch to Mountain Dew and Cheetos to have a chance at the top 10.

", "votes": 8 }, { "id": 72699, "postDate": "2015-04-20T18:20:29.750Z", "content": "

http://en.wikipedia.org/wiki/Semi-supervised_learning

\n

"Semi-supervised learning is a class of supervised learning tasks and techniques that also make use of unlabeled data for training" - in this case, he is asking if you can use the (unlabeled) test set for semi-supervised learning.

", "votes": 3 }, { "id": 71416, "postDate": "2015-04-13T23:09:26.693Z", "content": "

After reading the rules and searching the forums for any occurrence of "supervise" or "unlabeled," the only statement I could come up with related to the titular question was in the rules:

\n

"Semi-supervised learning is permitted."

\n

Just to make the matter explicitly clear so we all know the bounds in which we are operating, is it permissible to pre-train in an unsupervised fashion on the unlabeled data the test set represents? If I missed the already existing answer somewhere and any fellow Kaggler can point me to it, or if I can get an answer direct from competition management, I'd greatly appreciate it!

\n

Thanks so much, and sorry to be a pest! Just trying to avoid a lot of wasted effort if such actions would be frowned upon :)

", "votes": 4 }, { "id": 72715, "postDate": "2015-04-20T18:42:25.233Z", "content": "

[quote=Raffael Vogler;72703]

\n

Thanks, William. But, why would that have to be mentioned? This approach does not seem unfair or illicit by any standards.

\n

[/quote]

\n

I agree that it shouldn't have to be mentioned, but in one of the previous competitions I was a part of (one of the seizure challenges) a great deal of contention emerged over the wording of the rules and whether or not unlabeled test data would be permitted for helping to train the classifiers. Many people had labored through that point under the impression it couldn't be, and the decision to formally allow it part-way through got a lot of people up in arms.

\n

Given that other work in neural networks has shown that some performance gains can be had using e.g. unlabeled test data to drive weight initialization in a network as opposed to just random initialization, I thought it would be best to cover the question before it devolved into the mess it did last time.

\n

Forgive me for wanting to avoid yet another storm of whining and gnashing of teeth.

", "votes": 1 }, { "id": 72852, "postDate": "2015-04-21T05:35:01.967Z", "content": "

I confirm the hypothesis about Mountain Dew and Cheetos.

", "votes": 2 }, { "id": 72667, "postDate": "2015-04-20T16:46:33.783Z", "content": "

Hey LAD! Unless I am misunderstanding your question, what you are describing is what is being permitted by the rule you quoted.

", "votes": 2 }, { "id": 82372, "postDate": "2015-06-19T02:15:54.423Z", "content": "

"Who talks like that? It sounds like a crime boss talking to his henchmen over a telephone that the FBI is listening in on."

\n

Well I talk like that. I know exactly what it means.

", "votes": 1 }, { "id": 82343, "postDate": "2015-06-18T21:12:49.993Z", "content": "

[quote=George Solymosi;72766]

\n

So, once again, just to be sure:

\n

The unlabeled test data is NOT permitted for helping to train the classifiers in an unsupervised manner?

\n

Is it right?

\n

[/quote]

\n

It IS permitted. You can use the test data for training, as long as you do not hand-label it or otherwise look at it.

\n

The way William phrased it, however, leaves much to be desired. "Unless I am misunderstanding your question, what you are describing is what is being permitted by the rule you quoted." Who talks like that? It sounds like a crime boss talking to his henchmen over a telephone that the FBI is listening in on.

", "votes": 1 }, { "id": 82384, "postDate": "2015-06-19T07:34:29.407Z", "content": "

Yeah, this is like "I don't really have time for this now, anyway we are going to make this decision subsequently."

\n

That's why my question to be sure... :)

\n

[quote=James King;82372]

\n

"Who talks like that? It sounds like a crime boss talking to his henchmen over a telephone that the FBI is listening in on."

\n

Well I talk like that. I know exactly what it means.

\n

[/quote]

" }, { "id": 72766, "postDate": "2015-04-20T22:27:08.923Z", "content": "

So, once again, just to be sure:

\n

The unlabeled test data is NOT permitted for helping to train the classifiers in an unsupervised manner?

\n

Is it right?

\n

With Mountain Dew and Cheetos of course. :)

" }, { "id": 72705, "postDate": "2015-04-20T18:25:36.273Z", "content": "

We mention it because if we don't, somebody asks about it.

" }, { "id": 72691, "postDate": "2015-04-20T18:11:20.623Z", "content": "

I don't entirely understand the question?

\n

Are you asking if you can use someone elses pre-trained system?

\n

The I would think is quite unfair to everyone else if you did not do the work yourself and download the data and use the existing data to train your system by whatever method you desire

" }, { "id": 72678, "postDate": "2015-04-20T17:31:38.227Z", "content": "

Perfect, thank you for the clarification William!

" }, { "id": 72709, "postDate": "2015-04-20T18:28:08.867Z", "rawMarkdown": "", "isDeleted": true }, { "id": 72703, "postDate": "2015-04-20T18:24:40.607Z", "rawMarkdown": "", "isDeleted": true }, { "id": 72690, "postDate": "2015-04-20T18:11:13.830Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 72711, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-04-20T18:30:14.540000", "content": "

Milk and carrots are allowed, but will place you north of 200th place. You'll want to switch to Mountain Dew and Cheetos to have a chance at the top 10.

", "votes": 8, "replies": [] }, { "id": 72699, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-04-20T18:20:29.750000", "content": "

http://en.wikipedia.org/wiki/Semi-supervised_learning

\n

"Semi-supervised learning is a class of supervised learning tasks and techniques that also make use of unlabeled data for training" - in this case, he is asking if you can use the (unlabeled) test set for semi-supervised learning.

", "votes": 3, "replies": [] }, { "id": 72715, "author_name": "LAD", "author_url": "", "post_date": "2015-04-20T18:42:25.233000", "content": "

[quote=Raffael Vogler;72703]

\n

Thanks, William. But, why would that have to be mentioned? This approach does not seem unfair or illicit by any standards.

\n

[/quote]

\n

I agree that it shouldn't have to be mentioned, but in one of the previous competitions I was a part of (one of the seizure challenges) a great deal of contention emerged over the wording of the rules and whether or not unlabeled test data would be permitted for helping to train the classifiers. Many people had labored through that point under the impression it couldn't be, and the decision to formally allow it part-way through got a lot of people up in arms.

\n

Given that other work in neural networks has shown that some performance gains can be had using e.g. unlabeled test data to drive weight initialization in a network as opposed to just random initialization, I thought it would be best to cover the question before it devolved into the mess it did last time.

\n

Forgive me for wanting to avoid yet another storm of whining and gnashing of teeth.

", "votes": 1, "replies": [] }, { "id": 72852, "author_name": "Ionel Hosu", "author_url": "", "post_date": "2015-04-21T05:35:01.967000", "content": "

I confirm the hypothesis about Mountain Dew and Cheetos.

", "votes": 2, "replies": [] }, { "id": 72667, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-04-20T16:46:33.783000", "content": "

Hey LAD! Unless I am misunderstanding your question, what you are describing is what is being permitted by the rule you quoted.

", "votes": 2, "replies": [] }, { "id": 82372, "author_name": "James King", "author_url": "", "post_date": "2015-06-19T02:15:54.423000", "content": "

"Who talks like that? It sounds like a crime boss talking to his henchmen over a telephone that the FBI is listening in on."

\n

Well I talk like that. I know exactly what it means.

", "votes": 1, "replies": [] }, { "id": 82343, "author_name": "Aleksandr Dubinsky", "author_url": "", "post_date": "2015-06-18T21:12:49.993000", "content": "

[quote=George Solymosi;72766]

\n

So, once again, just to be sure:

\n

The unlabeled test data is NOT permitted for helping to train the classifiers in an unsupervised manner?

\n

Is it right?

\n

[/quote]

\n

It IS permitted. You can use the test data for training, as long as you do not hand-label it or otherwise look at it.

\n

The way William phrased it, however, leaves much to be desired. "Unless I am misunderstanding your question, what you are describing is what is being permitted by the rule you quoted." Who talks like that? It sounds like a crime boss talking to his henchmen over a telephone that the FBI is listening in on.

", "votes": 1, "replies": [] }, { "id": 82384, "author_name": "", "author_url": "", "post_date": "2015-06-19T07:34:29.407000", "content": "

Yeah, this is like "I don't really have time for this now, anyway we are going to make this decision subsequently."

\n

That's why my question to be sure... :)

\n

[quote=James King;82372]

\n

"Who talks like that? It sounds like a crime boss talking to his henchmen over a telephone that the FBI is listening in on."

\n

Well I talk like that. I know exactly what it means.

\n

[/quote]

", "votes": 0, "replies": [] }, { "id": 72766, "author_name": "", "author_url": "", "post_date": "2015-04-20T22:27:08.923000", "content": "

So, once again, just to be sure:

\n

The unlabeled test data is NOT permitted for helping to train the classifiers in an unsupervised manner?

\n

Is it right?

\n

With Mountain Dew and Cheetos of course. :)

", "votes": 0, "replies": [] }, { "id": 72705, "author_name": "Will Cukierski", "author_url": "", "post_date": "2015-04-20T18:25:36.273000", "content": "

We mention it because if we don't, somebody asks about it.

", "votes": 0, "replies": [] }, { "id": 72691, "author_name": "Spaceman", "author_url": "", "post_date": "2015-04-20T18:11:20.623000", "content": "

I don't entirely understand the question?

\n

Are you asking if you can use someone elses pre-trained system?

\n

The I would think is quite unfair to everyone else if you did not do the work yourself and download the data and use the existing data to train your system by whatever method you desire

", "votes": 0, "replies": [] }, { "id": 72678, "author_name": "LAD", "author_url": "", "post_date": "2015-04-20T17:31:38.227000", "content": "

Perfect, thank you for the clarification William!

", "votes": 0, "replies": [] }, { "id": 72709, "author_name": "", "author_url": "", "post_date": "2015-04-20T18:28:08.867000", "content": "", "votes": 0, "replies": [] }, { "id": 72703, "author_name": "", "author_url": "", "post_date": "2015-04-20T18:24:40.607000", "content": "", "votes": 0, "replies": [] }, { "id": 72690, "author_name": "", "author_url": "", "post_date": "2015-04-20T18:11:13.830000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "72711": "", "72699": "", "71416": "", "72715": "", "72852": "", "72667": "", "82372": "", "82343": "", "82384": "", "72766": "", "72705": "", "72691": "", "72678": "", "72709": "", "72703": "", "72690": "" } }