What level of accuracy are you guys getting during training? I'm talking about accuracy for the known classes, excluding class99.
\n\nFor instance, I currently get ~81% accuracy on my train and validation sets (haven't submitted this model yet). Just wondering if this is good/bad/average.
", "messages": [ { "id": "413455", "postDate": "11/01/2018 01:33:31", "content": "What level of accuracy are you guys getting during training? I'm talking about accuracy for the known classes, excluding class99.
\n\nFor instance, I currently get ~81% accuracy on my train and validation sets (haven't submitted this model yet). Just wondering if this is good/bad/average.
", "rawMarkdown": "What level of accuracy are you guys getting during training? I'm talking about accuracy for the known classes, excluding class99. \n\nFor instance, I currently get ~81% accuracy on my train and validation sets (haven't submitted this model yet). Just wondering if this is good/bad/average.", "votes": null }, { "id": "414432", "postDate": "11/02/2018 19:02:24", "content": "Hi you may wonder why you don't get any answer. The reason is simple IMHO: accuracy is not the metric used in this competition, hence no one is computing it probably.
", "rawMarkdown": "Hi you may wonder why you don't get any answer. The reason is simple IMHO: accuracy is not the metric used in this competition, hence no one is computing it probably.", "votes": null }, { "id": "414440", "postDate": "11/02/2018 19:11:55", "content": "I think you might be right. However, I believe accuracy is a useful metric to track. It helps you understand how good your model is.
", "rawMarkdown": "I think you might be right. However, I believe accuracy is a useful metric to track. It helps you understand how good your model is.", "votes": null }, { "id": "415610", "postDate": "11/05/2018 11:43:06", "content": "FYI, the accuracy for my latest single model is 0.782 on validation data, and 0.734 if I weight all classes according to the competition metric. The corresponding LB score is 0.944, and CV score is 0.506.
\n\nIt seems that you have a better accuracy, but I am puzzled by this:
\n\n\n\n\non my train and validation set
\n
Is your 81% based on a mix of train and validation accuracy? If you include train then you get a meaningless number IMHO. You should focus on the validation set accuracy, if you insist on using accuracy.
", "rawMarkdown": "FYI, the accuracy for my latest single model is 0.782 on validation data, and 0.734 if I weight all classes according to the competition metric. The corresponding LB score is 0.944, and CV score is 0.506.\n\nIt seems that you have a better accuracy, but I am puzzled by this:\n\n> on my train and validation set\n\nIs your 81% based on a mix of train and validation accuracy? If you include train then you get a meaningless number IMHO. You should focus on the validation set accuracy, if you insist on using accuracy.", "votes": null }, { "id": "415837", "postDate": "11/05/2018 19:24:03", "content": "Thanks for the reply, CPMP!
\n\nI do an old-school train/val split where a randomly selected 20% of the data is held out (i.e., the model doesn't get trained on it). I track the model's loss and accuracy on both the train set as well as the validation set during training. I have gotten up to ~81% accuracy on both - that is, including on the validation set.
\n\nSo I know my model is pretty good at classifying the 14 known classes. Unfortunately, even 100% accuracy on the 14 known classes can lead to a bad LB ranking if class 15 (the unknowns) are classified poorly on the test set.
", "rawMarkdown": "Thanks for the reply, CPMP! \n\nI do an old-school train/val split where a randomly selected 20% of the data is held out (i.e., the model doesn't get trained on it). I track the model's loss and accuracy on both the train set as well as the validation set during training. I have gotten up to ~81% accuracy on both - that is, including on the validation set. \n\nSo I know my model is pretty good at classifying the 14 known classes. Unfortunately, even 100% accuracy on the 14 known classes can lead to a bad LB ranking if class 15 (the unknowns) are classified poorly on the test set.", "votes": null }, { "id": "415838", "postDate": "11/05/2018 19:27:18", "content": "A great accuracy can lead to a poor LB rank because accuracy is NOT the metric used in this competition.
\n\nAlso, there is a risk of overfitting to training data distribution here.
", "rawMarkdown": "A great accuracy can lead to a poor LB rank because accuracy is NOT the metric used in this competition.\n\nAlso, there is a risk of overfitting to training data distribution here.", "votes": null }, { "id": "415842", "postDate": "11/05/2018 19:32:25", "content": "I get it. At the end of the day, the point of this competition is to minimize the loss. Accuracy and loss are correlated, but not perfectly. I just thought it would be interesting to also track accuracy since it's a more intuitive measure of classification performance.
", "rawMarkdown": "I get it. At the end of the day, the point of this competition is to minimize the loss. Accuracy and loss are correlated, but not perfectly. I just thought it would be interesting to also track accuracy since it's a more intuitive measure of classification performance.", "votes": null }, { "id": "416345", "postDate": "11/06/2018 15:13:54", "content": "It is possible to get good accuracy with high loss and bad accuracy with low loss because the loss does not depend on the classification part but on the confidence on each class.
", "rawMarkdown": "It is possible to get good accuracy with high loss and bad accuracy with low loss because the loss does not depend on the classification part but on the confidence on each class.", "votes": null }, { "id": "417703", "postDate": "11/08/2018 16:55:31", "content": "IMO, the class distributions make accuracy not very helpful here. You can probably get somewhere in the neighborhood of 60% accuracy with a model as simple as: class = (is in the milky way?) ? 65 : 90
Yea, overall accuracy might not be very helpful. Accuracy per class (i.e., confusion matrix) is extremely helpful. CPMP has a thread about this here: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70669
", "rawMarkdown": "Yea, overall accuracy might not be very helpful. Accuracy per class (i.e., confusion matrix) is extremely helpful. CPMP has a thread about this here: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70669", "votes": null }, { "id": "418919", "postDate": "11/10/2018 21:31:37", "content": "I used to work in the Variability Processing team on the preparation of the Gaia space telescope. We were really happy when we reached 90% accuracy of the variable star classification. OK, that was another dataset and we worked on it for 3 years :)
", "rawMarkdown": "I used to work in the Variability Processing team on the preparation of the Gaia space telescope. We were really happy when we reached 90% accuracy of the variable star classification. OK, that was another dataset and we worked on it for 3 years :)", "votes": null } ], "comments": [ { "id": 414432, "author_name": "cpmpml", "author_url": "", "post_date": "11/02/2018 19:02:24", "content": "Hi you may wonder why you don't get any answer. The reason is simple IMHO: accuracy is not the metric used in this competition, hence no one is computing it probably.
", "votes": null, "replies": [ { "id": 414440, "author_name": "borismarjanovic", "author_url": "", "post_date": "11/02/2018 19:11:55", "content": "I think you might be right. However, I believe accuracy is a useful metric to track. It helps you understand how good your model is.
", "votes": null, "replies": [] }, { "id": 415610, "author_name": "cpmpml", "author_url": "", "post_date": "11/05/2018 11:43:06", "content": "FYI, the accuracy for my latest single model is 0.782 on validation data, and 0.734 if I weight all classes according to the competition metric. The corresponding LB score is 0.944, and CV score is 0.506.
\n\nIt seems that you have a better accuracy, but I am puzzled by this:
\n\n\n\n\non my train and validation set
\n
Is your 81% based on a mix of train and validation accuracy? If you include train then you get a meaningless number IMHO. You should focus on the validation set accuracy, if you insist on using accuracy.
", "votes": null, "replies": [] }, { "id": 415837, "author_name": "borismarjanovic", "author_url": "", "post_date": "11/05/2018 19:24:03", "content": "Thanks for the reply, CPMP!
\n\nI do an old-school train/val split where a randomly selected 20% of the data is held out (i.e., the model doesn't get trained on it). I track the model's loss and accuracy on both the train set as well as the validation set during training. I have gotten up to ~81% accuracy on both - that is, including on the validation set.
\n\nSo I know my model is pretty good at classifying the 14 known classes. Unfortunately, even 100% accuracy on the 14 known classes can lead to a bad LB ranking if class 15 (the unknowns) are classified poorly on the test set.
", "votes": null, "replies": [] }, { "id": 415838, "author_name": "cpmpml", "author_url": "", "post_date": "11/05/2018 19:27:18", "content": "A great accuracy can lead to a poor LB rank because accuracy is NOT the metric used in this competition.
\n\nAlso, there is a risk of overfitting to training data distribution here.
", "votes": null, "replies": [] }, { "id": 415842, "author_name": "borismarjanovic", "author_url": "", "post_date": "11/05/2018 19:32:25", "content": "I get it. At the end of the day, the point of this competition is to minimize the loss. Accuracy and loss are correlated, but not perfectly. I just thought it would be interesting to also track accuracy since it's a more intuitive measure of classification performance.
", "votes": null, "replies": [] } ] }, { "id": 416345, "author_name": "bharat0", "author_url": "", "post_date": "11/06/2018 15:13:54", "content": "It is possible to get good accuracy with high loss and bad accuracy with low loss because the loss does not depend on the classification part but on the confidence on each class.
", "votes": null, "replies": [] }, { "id": 417703, "author_name": "mikeholcomb", "author_url": "", "post_date": "11/08/2018 16:55:31", "content": "IMO, the class distributions make accuracy not very helpful here. You can probably get somewhere in the neighborhood of 60% accuracy with a model as simple as: class = (is in the milky way?) ? 65 : 90
Yea, overall accuracy might not be very helpful. Accuracy per class (i.e., confusion matrix) is extremely helpful. CPMP has a thread about this here: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70669
", "votes": null, "replies": [] } ] }, { "id": 418919, "author_name": "klagyi", "author_url": "", "post_date": "11/10/2018 21:31:37", "content": "I used to work in the Variability Processing team on the preparation of the Gaia space telescope. We were really happy when we reached 90% accuracy of the variable star classification. OK, that was another dataset and we worked on it for 3 years :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "413455": "What level of accuracy are you guys getting during training? I'm talking about accuracy for the known classes, excluding class99. \n\nFor instance, I currently get ~81% accuracy on my train and validation sets (haven't submitted this model yet). Just wondering if this is good/bad/average.", "414432": "Hi you may wonder why you don't get any answer. The reason is simple IMHO: accuracy is not the metric used in this competition, hence no one is computing it probably.", "414440": "I think you might be right. However, I believe accuracy is a useful metric to track. It helps you understand how good your model is.", "415610": "FYI, the accuracy for my latest single model is 0.782 on validation data, and 0.734 if I weight all classes according to the competition metric. The corresponding LB score is 0.944, and CV score is 0.506.\n\nIt seems that you have a better accuracy, but I am puzzled by this:\n\n> on my train and validation set\n\nIs your 81% based on a mix of train and validation accuracy? If you include train then you get a meaningless number IMHO. You should focus on the validation set accuracy, if you insist on using accuracy.", "415837": "Thanks for the reply, CPMP! \n\nI do an old-school train/val split where a randomly selected 20% of the data is held out (i.e., the model doesn't get trained on it). I track the model's loss and accuracy on both the train set as well as the validation set during training. I have gotten up to ~81% accuracy on both - that is, including on the validation set. \n\nSo I know my model is pretty good at classifying the 14 known classes. Unfortunately, even 100% accuracy on the 14 known classes can lead to a bad LB ranking if class 15 (the unknowns) are classified poorly on the test set.", "415838": "A great accuracy can lead to a poor LB rank because accuracy is NOT the metric used in this competition.\n\nAlso, there is a risk of overfitting to training data distribution here.", "415842": "I get it. At the end of the day, the point of this competition is to minimize the loss. Accuracy and loss are correlated, but not perfectly. I just thought it would be interesting to also track accuracy since it's a more intuitive measure of classification performance.", "416345": "It is possible to get good accuracy with high loss and bad accuracy with low loss because the loss does not depend on the classification part but on the confidence on each class.", "417703": "IMO, the class distributions make accuracy not very helpful here. You can probably get somewhere in the neighborhood of 60% accuracy with a model as simple as: `class = (is in the milky way?) ? 65 : 90`", "417802": "Yea, overall accuracy might not be very helpful. Accuracy per class (i.e., confusion matrix) is extremely helpful. CPMP has a thread about this here: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70669", "418919": "I used to work in the Variability Processing team on the preparation of the Gaia space telescope. We were really happy when we reached 90% accuracy of the variable star classification. OK, that was another dataset and we worked on it for 3 years :)" }, "source": "meta" }