I am coming into this competition quite late, most probably it's too late. Nevertheless, I find it an interesting competition to learn something new from. I've a question regarding the probabilistic classification being sought. If I understood it correctly, probabilistic classification is more useful for you here because deterministic classification would mean some information would be discarded in the process which will hamper/limit the usefulness of the results in turn.
\n\nI might be missing something but the question that arises in my mind is that given all the data has been simulated, am I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes? Would a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context?
\n\nThanks.
", "messages": [ { "id": "437176", "postDate": "12/11/2018 13:45:37", "content": "I am coming into this competition quite late, most probably it's too late. Nevertheless, I find it an interesting competition to learn something new from. I've a question regarding the probabilistic classification being sought. If I understood it correctly, probabilistic classification is more useful for you here because deterministic classification would mean some information would be discarded in the process which will hamper/limit the usefulness of the results in turn.
\n\nI might be missing something but the question that arises in my mind is that given all the data has been simulated, am I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes? Would a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context?
\n\nThanks.
", "rawMarkdown": "I am coming into this competition quite late, most probably it's too late. Nevertheless, I find it an interesting competition to learn something new from. I've a question regarding the probabilistic classification being sought. If I understood it correctly, probabilistic classification is more useful for you here because deterministic classification would mean some information would be discarded in the process which will hamper/limit the usefulness of the results in turn.\n\nI might be missing something but the question that arises in my mind is that given all the data has been simulated, am I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes? Would a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context? \n\nThanks.", "votes": null }, { "id": "437221", "postDate": "12/11/2018 15:11:05", "content": "\n\n\nam I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes?
\n
That distribution across classes is the output of your classifier model - check the sample submission csv under Data
\n\n\n\n\nWould a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context?
\n
Your models will always output probabilities - it is up to you to decide what to do with those probabilities. Normally you'd select the highest probability class as the \"right answer\"
", "rawMarkdown": "> am I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes?\n\nThat distribution across classes is the output of your classifier model - check the sample submission csv under Data\n\n> Would a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context?\n\nYour models will always output probabilities - it is up to you to decide what to do with those probabilities. Normally you'd select the highest probability class as the \"right answer\"", "votes": null }, { "id": "437262", "postDate": "12/11/2018 16:26:15", "content": "Having a probability rather than a single classification is important for a lot of science cases. For example, if we have a clean sample of Type Ia supernovae, then we can map out their measured brightnesses to measure properties of our universe. If other types of supernovae sneak into the sample, then they will bias your measurements. If you have a probability assigned to each class of supernova, you can incorporate that into your model to help account for the biases.
\n\nIf you really want to know what an object is, you can simply take more measurements of it. Getting a spectrum of a supernova will definitively tell you what kind of supernova it is. However, that requires a lot of telescope time, and won't be possible for every transient that we find with LSST.
", "rawMarkdown": "Having a probability rather than a single classification is important for a lot of science cases. For example, if we have a clean sample of Type Ia supernovae, then we can map out their measured brightnesses to measure properties of our universe. If other types of supernovae sneak into the sample, then they will bias your measurements. If you have a probability assigned to each class of supernova, you can incorporate that into your model to help account for the biases.\n\nIf you really want to know what an object is, you can simply take more measurements of it. Getting a spectrum of a supernova will definitively tell you what kind of supernova it is. However, that requires a lot of telescope time, and won't be possible for every transient that we find with LSST.", "votes": null }, { "id": "437276", "postDate": "12/11/2018 16:51:12", "content": "Hey Ganfear thanks for your response!
\n\n\n\n\nis the output of your classifier model
\n
Perhaps it wasn't very clear from my question, I am talking about the grounds the evaluation would be done on and the best way to match our results with the already generated label values which are obviously hidden for us right now.
\n\n\n\n\nthe \"right answer\"
\n
Again the question I am asking is, in this competition, whether a model should give priority and actively try to produce weighted values across classes for all rows or it should try the best to produce a single \"right answer\" (a single class pointed out with a value of 1) even for most rows if it can. Which one is more desirable as per the goals of the competiton if both are possible? That's the question!
\n\nI am asking this as I feel depending on the answer, the solution can be modified to have a positive effect on the score thereby producing the best tool for the scientific community.
", "rawMarkdown": "Hey Ganfear thanks for your response!\n\n>is the output of your classifier model\n\nPerhaps it wasn't very clear from my question, I am talking about the _grounds_ the evaluation would be done on and the _best way_ to match our results with the already generated label values which are obviously hidden for us right now. \n\n>the \"right answer\"\n\nAgain the question I am asking is, in this competition, whether a model _should_ give priority and actively try to produce weighted values across classes for all rows **or** it should try the best to produce a single \"right answer\" (a single class pointed out with a value of 1) even for most rows _if it can_. Which one is more _desirable_ as per the _goals of the competiton_ if both are possible? That's the question! \n\nI am asking this as I feel depending on the answer, the solution can be modified to have a positive effect on the score thereby producing the best tool for the scientific community.", "votes": null }, { "id": "437285", "postDate": "12/11/2018 17:13:58", "content": "The PLAsTiCC team put a lot of effort into choosing a metric that will be useful for science applications. They wrote a paper on it that you can find here: https://arxiv.org/abs/1809.11145. It is basically a weighted cross-entropy. Most people that I have spoken to in the astronomy community think that it was a good choice. So optimize the competition metric and you will be optimizing the impact on the science!
", "rawMarkdown": "The PLAsTiCC team put a lot of effort into choosing a metric that will be useful for science applications. They wrote a paper on it that you can find here: [https://arxiv.org/abs/1809.11145][1]. It is basically a weighted cross-entropy. Most people that I have spoken to in the astronomy community think that it was a good choice. So optimize the competition metric and you will be optimizing the impact on the science!\n\n\n [1]: https://arxiv.org/abs/1809.11145", "votes": null }, { "id": "437289", "postDate": "12/11/2018 17:24:31", "content": "Hey Kyle, thanks for the link and your answers! Please edit the link to exclude the dot at the end. Thanks.
", "rawMarkdown": "Hey Kyle, thanks for the link and your answers! Please edit the link to exclude the dot at the end. Thanks.", "votes": null }, { "id": "437327", "postDate": "12/11/2018 18:06:08", "content": "Thanks, should be fixed now.
", "rawMarkdown": "Thanks, should be fixed now.", "votes": null } ], "comments": [ { "id": 437221, "author_name": "ganfear", "author_url": "", "post_date": "12/11/2018 15:11:05", "content": "\n\n\nam I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes?
\n
That distribution across classes is the output of your classifier model - check the sample submission csv under Data
\n\n\n\n\nWould a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context?
\n
Your models will always output probabilities - it is up to you to decide what to do with those probabilities. Normally you'd select the highest probability class as the \"right answer\"
", "votes": null, "replies": [ { "id": 437276, "author_name": "kishupro", "author_url": "", "post_date": "12/11/2018 16:51:12", "content": "Hey Ganfear thanks for your response!
\n\n\n\n\nis the output of your classifier model
\n
Perhaps it wasn't very clear from my question, I am talking about the grounds the evaluation would be done on and the best way to match our results with the already generated label values which are obviously hidden for us right now.
\n\n\n\n\nthe \"right answer\"
\n
Again the question I am asking is, in this competition, whether a model should give priority and actively try to produce weighted values across classes for all rows or it should try the best to produce a single \"right answer\" (a single class pointed out with a value of 1) even for most rows if it can. Which one is more desirable as per the goals of the competiton if both are possible? That's the question!
\n\nI am asking this as I feel depending on the answer, the solution can be modified to have a positive effect on the score thereby producing the best tool for the scientific community.
", "votes": null, "replies": [] }, { "id": 437285, "author_name": "kyleboone", "author_url": "", "post_date": "12/11/2018 17:13:58", "content": "The PLAsTiCC team put a lot of effort into choosing a metric that will be useful for science applications. They wrote a paper on it that you can find here: https://arxiv.org/abs/1809.11145. It is basically a weighted cross-entropy. Most people that I have spoken to in the astronomy community think that it was a good choice. So optimize the competition metric and you will be optimizing the impact on the science!
", "votes": null, "replies": [] }, { "id": 437289, "author_name": "kishupro", "author_url": "", "post_date": "12/11/2018 17:24:31", "content": "Hey Kyle, thanks for the link and your answers! Please edit the link to exclude the dot at the end. Thanks.
", "votes": null, "replies": [] }, { "id": 437327, "author_name": "kyleboone", "author_url": "", "post_date": "12/11/2018 18:06:08", "content": "Thanks, should be fixed now.
", "votes": null, "replies": [] } ] }, { "id": 437262, "author_name": "kyleboone", "author_url": "", "post_date": "12/11/2018 16:26:15", "content": "Having a probability rather than a single classification is important for a lot of science cases. For example, if we have a clean sample of Type Ia supernovae, then we can map out their measured brightnesses to measure properties of our universe. If other types of supernovae sneak into the sample, then they will bias your measurements. If you have a probability assigned to each class of supernova, you can incorporate that into your model to help account for the biases.
\n\nIf you really want to know what an object is, you can simply take more measurements of it. Getting a spectrum of a supernova will definitively tell you what kind of supernova it is. However, that requires a lot of telescope time, and won't be possible for every transient that we find with LSST.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "437176": "I am coming into this competition quite late, most probably it's too late. Nevertheless, I find it an interesting competition to learn something new from. I've a question regarding the probabilistic classification being sought. If I understood it correctly, probabilistic classification is more useful for you here because deterministic classification would mean some information would be discarded in the process which will hamper/limit the usefulness of the results in turn.\n\nI might be missing something but the question that arises in my mind is that given all the data has been simulated, am I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes? Would a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context? \n\nThanks.", "437221": "> am I to understand that the test data has been intensionally generated such that for most of the rows there is a distribution of target values (i.e. values < 1) across classes?\n\nThat distribution across classes is the output of your classifier model - check the sample submission csv under Data\n\n> Would a model that is fine-tuned to somehow forcefully produce probabilistic results even where it actually can determine the classes clearly (i.e. value = 1 for a particular class) be more useful for the ultimate goal of the science involved in this context?\n\nYour models will always output probabilities - it is up to you to decide what to do with those probabilities. Normally you'd select the highest probability class as the \"right answer\"", "437262": "Having a probability rather than a single classification is important for a lot of science cases. For example, if we have a clean sample of Type Ia supernovae, then we can map out their measured brightnesses to measure properties of our universe. If other types of supernovae sneak into the sample, then they will bias your measurements. If you have a probability assigned to each class of supernova, you can incorporate that into your model to help account for the biases.\n\nIf you really want to know what an object is, you can simply take more measurements of it. Getting a spectrum of a supernova will definitively tell you what kind of supernova it is. However, that requires a lot of telescope time, and won't be possible for every transient that we find with LSST.", "437276": "Hey Ganfear thanks for your response!\n\n>is the output of your classifier model\n\nPerhaps it wasn't very clear from my question, I am talking about the _grounds_ the evaluation would be done on and the _best way_ to match our results with the already generated label values which are obviously hidden for us right now. \n\n>the \"right answer\"\n\nAgain the question I am asking is, in this competition, whether a model _should_ give priority and actively try to produce weighted values across classes for all rows **or** it should try the best to produce a single \"right answer\" (a single class pointed out with a value of 1) even for most rows _if it can_. Which one is more _desirable_ as per the _goals of the competiton_ if both are possible? That's the question! \n\nI am asking this as I feel depending on the answer, the solution can be modified to have a positive effect on the score thereby producing the best tool for the scientific community.", "437285": "The PLAsTiCC team put a lot of effort into choosing a metric that will be useful for science applications. They wrote a paper on it that you can find here: [https://arxiv.org/abs/1809.11145][1]. It is basically a weighted cross-entropy. Most people that I have spoken to in the astronomy community think that it was a good choice. So optimize the competition metric and you will be optimizing the impact on the science!\n\n\n [1]: https://arxiv.org/abs/1809.11145", "437289": "Hey Kyle, thanks for the link and your answers! Please edit the link to exclude the dot at the end. Thanks.", "437327": "Thanks, should be fixed now." }, "source": "meta" }