{ "id": 9412, "title": "Per Subject Classifiers or Universal Classifier?", "url": "/competitions/seizure-detection/discussion/9412", "author_name": "", "post_date": "2014-06-07T22:03:18.803Z", "votes": null, "comment_count": 14, "views": 3326, "content": "

Hi,

\n\n

I would like to know whether we are allowed to train our classifiers separately for each subject (i.e. dog 1, dog 2, human 1, etc.), or whether we should only train one universal classifier once and perform the predictions for all subjects using that one.

\n\n

Thanks,

\n

Anthony

", "messages": [ { "id": "48847", "postDate": "06/07/2014 22:03:18", "content": "

Hi,

\n\n

I would like to know whether we are allowed to train our classifiers separately for each subject (i.e. dog 1, dog 2, human 1, etc.), or whether we should only train one universal classifier once and perform the predictions for all subjects using that one.

\n\n

Thanks,

\n

Anthony

", "rawMarkdown": "", "votes": null }, { "id": "48854", "postDate": "06/08/2014 03:08:19", "content": "

I have the same question. I suppose either way is valid, insofar as this is not an "asking for too much information" question; how will the predictions be made in real life? Will lab techs be training models for each new patient, or classifying universally with the same models?

", "rawMarkdown": "", "votes": null }, { "id": "48867", "postDate": "06/08/2014 17:27:42", "content": "

As per the rules there is no restriction on subject specific models. Im sure its perfectly fine to train subject-specific models based on reapective subjects train data or pooled trainig data. As long as the model can be applied to a new subject given its trainig data.

\n

Cheers

", "rawMarkdown": "", "votes": null }, { "id": "48897", "postDate": "06/09/2014 12:42:46", "content": "

The rules page was fairly clear on this, IMHO:

\n\n

Any changes to the methodology across species/subjects must be done in an automated way, so that your approach will generalize to new subjects.

\n
# Not allowed:
if 'Dog_1' then foo()
if 'Patient_1' then bar()
...

# Allowed
if f(signal) < 2 then foo()
else bar()
", "rawMarkdown": "", "votes": null }, { "id": "48899", "postDate": "06/09/2014 12:49:44", "content": "

Thanks Will! Hoping you can clarify the evaluation criteria for me as well in this thread.

", "rawMarkdown": "", "votes": null }, { "id": "48916", "postDate": "06/09/2014 17:06:45", "content": "

Hi William!

\n\n

Does the automated way include a case where we consider each folder separately? For example, for each folder (i.e. subject folder) train using the training data within that folder and test using the test data within that folder - with no subject specific code (e.g. for dog 1 do that, etc.). That is what was not clear to me from the rules page.

\n\n

Thanks,

\n

Anthony

", "rawMarkdown": "", "votes": null }, { "id": "49021", "postDate": "06/11/2014 18:40:37", "content": "

This guideline is not a clear. The following function will generalize to new subjects, yet allows discrimination between Dogs and Patients in the current contest.

\n\n

if 'freq' <= 400 then foo()     # Dog case

\n

if 400 < 'freq' < 600 then bar()   # Patient 1 case

\n

if 'freq' > 600 then foobar()    # Other patients

\n\n

Not allowed?

\n\n

[quote=William Cukierski;48897]

\n

The rules page was fairly clear on this, IMHO:

\n

Any changes to the methodology across species/subjects must be done in an automated way, so that your approach will generalize to new subjects.

\n
# Not allowed:
if 'Dog_1' then foo()
if 'Patient_1' then bar()
...

# Allowed
if f(signal) < 2 then foo()
else bar()
\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "49065", "postDate": "06/12/2014 17:24:58", "content": "

The guideline requires using your judgement. In your example, frequency is acting as a proxy for the individual subjects. It would not generalize to future subjects or new data, so you should not use such a rule.

", "rawMarkdown": "", "votes": null }, { "id": "49097", "postDate": "06/13/2014 11:29:22", "content": "

[quote=Anthony Platanios;48916]

\n

Hi William!

\n

Does the automated way include a case where we consider each folder separately? For example, for each folder (i.e. subject folder) train using the training data within that folder and test using the test data within that folder - with no subject specific code (e.g. for dog 1 do that, etc.). That is what was not clear to me from the rules page.

\n

Thanks,

\n

Anthony

\n

[/quote]

\n

This is a good questions. It is also not clear to me. From an application point-of-view, I don't think it makes too much sense to enforce solutions that generalise over dogs and human patients. However, now that I read this thread, I am not sure anymore.

\n\n\n

Also: "Any changes to the methodology across species/subjects must be done in an automated way, so that your approach will generalize to new subjects." (from the rules) is a very broad statement, given that the data of each subject is so different (frequency, #channels, etc). For example, should our approaches generalise to subjects that are monitored by only one electrode? Or to snippets with a length of 2 seconds?

", "rawMarkdown": "", "votes": null }, { "id": "49129", "postDate": "06/13/2014 21:50:35", "content": "

Ah I am really confused now. I thought the reason the test and train for each subject are provided in their separate respective folder is to train subject specific models in a way that given training data for a subject the generalized model can predict the test trial. If this was not the case, then why all the test trials not provided in a general/common pooled directory!

\n\n

I think the way the data has been provided the following should be allowed:

\n\n

Predict label for subject n given the training data for that subject (n) using a generalized common model.

\n\n

i.e.

\n\n

if f(testSignaln | trainSignaln) < 2 then foo()
else bar()

\n

where n is the subject number

", "rawMarkdown": "", "votes": null }, { "id": "49247", "postDate": "06/17/2014 09:54:17", "content": "

This is important.

\n\n

William, is a per-subject classification scheme valid or not?

", "rawMarkdown": "", "votes": null }, { "id": "49252", "postDate": "06/17/2014 13:17:44", "content": "

Let me try to clarify-

\n\n

It is fine to train subject specific models. Chaos is right- this is why the folders were provided as they were. In the responsive neurostimulation application, we would expect to have to train an algorithm/approach for each patient. Also, as Ulo Gulo accuratly points out, electrode placement and configurations in this data set differ, so there's not a very good way to generalize an algorithm trained on pooled data.

\n\n

In real applications, data will be acquired in a continuous stream and processed in near real time, rather than as 1 second clips. The clips are a necessity for the contest. The requirement is to generalize over the range of data presented in this contest. You're right, there's an infinite range of possible electrode and data configurations, but we don't expect contestants to account for this.

\n\n

The sampling frequency case is interesting because you can separate dogs and humans by this metric - this was unintended, and please bear in mind that much clinical iEEG data currently is acquired sampled at 256-500 Hz. The fact that the contest data can be separated by sampling frequency is a peculiarity of us trying to provide the best and most novel iEEG data possible for the contest. So it would really not be appropriate to use the sampling frequency as a way to artificially use different approaches for dogs and humans, as described by Alexander VR. 

\n\n


I hope this helps. Keep the questions coming...

", "rawMarkdown": "", "votes": null }, { "id": "51105", "postDate": "07/23/2014 21:20:10", "content": "

Hi

\n

Following up on this thread..Did anyone try the  classification (subject specific and pooled subjects)..Any thoughts which seems to be better? :)

\n

Thanks

\n

Navin

", "rawMarkdown": "", "votes": null }, { "id": "51107", "postDate": "07/23/2014 23:08:06", "content": "

My bet is on the subject specific approach. That's why I made so many (unsuccessful) submissions! :)

", "rawMarkdown": "", "votes": null }, { "id": "51110", "postDate": "07/23/2014 23:13:39", "content": "

Thanks Jose for your thoughts !!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 48854, "author_name": "cavaunpeu", "author_url": "", "post_date": "06/08/2014 03:08:19", "content": "

I have the same question. I suppose either way is valid, insofar as this is not an "asking for too much information" question; how will the predictions be made in real life? Will lab techs be training models for each new patient, or classifying universally with the same models?

", "votes": null, "replies": [] }, { "id": 48867, "author_name": "chaotic", "author_url": "", "post_date": "06/08/2014 17:27:42", "content": "

As per the rules there is no restriction on subject specific models. Im sure its perfectly fine to train subject-specific models based on reapective subjects train data or pooled trainig data. As long as the model can be applied to a new subject given its trainig data.

\n

Cheers

", "votes": null, "replies": [] }, { "id": 48897, "author_name": "wcukierski", "author_url": "", "post_date": "06/09/2014 12:42:46", "content": "

The rules page was fairly clear on this, IMHO:

\n\n

Any changes to the methodology across species/subjects must be done in an automated way, so that your approach will generalize to new subjects.

\n
# Not allowed:
if 'Dog_1' then foo()
if 'Patient_1' then bar()
...

# Allowed
if f(signal) < 2 then foo()
else bar()
", "votes": null, "replies": [] }, { "id": 48899, "author_name": "cavaunpeu", "author_url": "", "post_date": "06/09/2014 12:49:44", "content": "

Thanks Will! Hoping you can clarify the evaluation criteria for me as well in this thread.

", "votes": null, "replies": [] }, { "id": 48916, "author_name": "eaplatanios", "author_url": "", "post_date": "06/09/2014 17:06:45", "content": "

Hi William!

\n\n

Does the automated way include a case where we consider each folder separately? For example, for each folder (i.e. subject folder) train using the training data within that folder and test using the test data within that folder - with no subject specific code (e.g. for dog 1 do that, etc.). That is what was not clear to me from the rules page.

\n\n

Thanks,

\n

Anthony

", "votes": null, "replies": [] }, { "id": 49021, "author_name": "alexvr", "author_url": "", "post_date": "06/11/2014 18:40:37", "content": "

This guideline is not a clear. The following function will generalize to new subjects, yet allows discrimination between Dogs and Patients in the current contest.

\n\n

if 'freq' <= 400 then foo()     # Dog case

\n

if 400 < 'freq' < 600 then bar()   # Patient 1 case

\n

if 'freq' > 600 then foobar()    # Other patients

\n\n

Not allowed?

\n\n

[quote=William Cukierski;48897]

\n

The rules page was fairly clear on this, IMHO:

\n

Any changes to the methodology across species/subjects must be done in an automated way, so that your approach will generalize to new subjects.

\n
# Not allowed:
if 'Dog_1' then foo()
if 'Patient_1' then bar()
...

# Allowed
if f(signal) < 2 then foo()
else bar()
\n

[/quote]

", "votes": null, "replies": [] }, { "id": 49065, "author_name": "wcukierski", "author_url": "", "post_date": "06/12/2014 17:24:58", "content": "

The guideline requires using your judgement. In your example, frequency is acting as a proxy for the individual subjects. It would not generalize to future subjects or new data, so you should not use such a rule.

", "votes": null, "replies": [] }, { "id": 49097, "author_name": "ugrossek", "author_url": "", "post_date": "06/13/2014 11:29:22", "content": "

[quote=Anthony Platanios;48916]

\n

Hi William!

\n

Does the automated way include a case where we consider each folder separately? For example, for each folder (i.e. subject folder) train using the training data within that folder and test using the test data within that folder - with no subject specific code (e.g. for dog 1 do that, etc.). That is what was not clear to me from the rules page.

\n

Thanks,

\n

Anthony

\n

[/quote]

\n

This is a good questions. It is also not clear to me. From an application point-of-view, I don't think it makes too much sense to enforce solutions that generalise over dogs and human patients. However, now that I read this thread, I am not sure anymore.

\n\n\n

Also: "Any changes to the methodology across species/subjects must be done in an automated way, so that your approach will generalize to new subjects." (from the rules) is a very broad statement, given that the data of each subject is so different (frequency, #channels, etc). For example, should our approaches generalise to subjects that are monitored by only one electrode? Or to snippets with a length of 2 seconds?

", "votes": null, "replies": [] }, { "id": 49129, "author_name": "chaotic", "author_url": "", "post_date": "06/13/2014 21:50:35", "content": "

Ah I am really confused now. I thought the reason the test and train for each subject are provided in their separate respective folder is to train subject specific models in a way that given training data for a subject the generalized model can predict the test trial. If this was not the case, then why all the test trials not provided in a general/common pooled directory!

\n\n

I think the way the data has been provided the following should be allowed:

\n\n

Predict label for subject n given the training data for that subject (n) using a generalized common model.

\n\n

i.e.

\n\n

if f(testSignaln | trainSignaln) < 2 then foo()
else bar()

\n

where n is the subject number

", "votes": null, "replies": [] }, { "id": 49247, "author_name": "joseleiva", "author_url": "", "post_date": "06/17/2014 09:54:17", "content": "

This is important.

\n\n

William, is a per-subject classification scheme valid or not?

", "votes": null, "replies": [] }, { "id": 49252, "author_name": "bbrinkm", "author_url": "", "post_date": "06/17/2014 13:17:44", "content": "

Let me try to clarify-

\n\n

It is fine to train subject specific models. Chaos is right- this is why the folders were provided as they were. In the responsive neurostimulation application, we would expect to have to train an algorithm/approach for each patient. Also, as Ulo Gulo accuratly points out, electrode placement and configurations in this data set differ, so there's not a very good way to generalize an algorithm trained on pooled data.

\n\n

In real applications, data will be acquired in a continuous stream and processed in near real time, rather than as 1 second clips. The clips are a necessity for the contest. The requirement is to generalize over the range of data presented in this contest. You're right, there's an infinite range of possible electrode and data configurations, but we don't expect contestants to account for this.

\n\n

The sampling frequency case is interesting because you can separate dogs and humans by this metric - this was unintended, and please bear in mind that much clinical iEEG data currently is acquired sampled at 256-500 Hz. The fact that the contest data can be separated by sampling frequency is a peculiarity of us trying to provide the best and most novel iEEG data possible for the contest. So it would really not be appropriate to use the sampling frequency as a way to artificially use different approaches for dogs and humans, as described by Alexander VR. 

\n\n


I hope this helps. Keep the questions coming...

", "votes": null, "replies": [] }, { "id": 51105, "author_name": "navincota", "author_url": "", "post_date": "07/23/2014 21:20:10", "content": "

Hi

\n

Following up on this thread..Did anyone try the  classification (subject specific and pooled subjects)..Any thoughts which seems to be better? :)

\n

Thanks

\n

Navin

", "votes": null, "replies": [] }, { "id": 51107, "author_name": "joseleiva", "author_url": "", "post_date": "07/23/2014 23:08:06", "content": "

My bet is on the subject specific approach. That's why I made so many (unsuccessful) submissions! :)

", "votes": null, "replies": [] }, { "id": 51110, "author_name": "navincota", "author_url": "", "post_date": "07/23/2014 23:13:39", "content": "

Thanks Jose for your thoughts !!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "48847": "", "48854": "", "48867": "", "48897": "", "48899": "", "48916": "", "49021": "", "49065": "", "49097": "", "49129": "", "49247": "", "49252": "", "51105": "", "51107": "", "51110": "" }, "source": "meta" }