I've tried so many things here that I'm not sure I can list them all. I must admit I'm puzzled as to how to derive class 99 probability from the training set. It's the very first time I'm facing a similar problem.
\n\nHere are the things I've tried:
\n\nThe best I've found this far is to use the product of all opposite probabilities and adapting the mean but I'm not satisfied with that since it kills the other classes logloss.
\n\nIf anyone is willing to share his ideas I would be gratefull ;-)
", "messages": [ { "id": 406121, "postDate": "2018-10-18T17:21:01.407Z", "content": "I've tried so many things here that I'm not sure I can list them all. I must admit I'm puzzled as to how to derive class 99 probability from the training set. It's the very first time I'm facing a similar problem.
\n\nHere are the things I've tried:
\n\nThe best I've found this far is to use the product of all opposite probabilities and adapting the mean but I'm not satisfied with that since it kills the other classes logloss.
\n\nIf anyone is willing to share his ideas I would be gratefull ;-)
", "rawMarkdown": "I've tried so many things here that I'm not sure I can list them all. I must admit I'm puzzled as to how to derive class 99 probability from the training set. It's the very first time I'm facing a similar problem.\n\nHere are the things I've tried: \n\n - Use a constant\n - Going to logits, derive individual probabilities from that, compute p99 as the product of opposite class probabilities, go back to logit and use softmax to normalize (probably the worst result)\n - Computing the probability as the opposite of the max proba over all classes for each sample\n - Calculate p99 as the product of the other classes opposite proba, going to logits and using softmax to normalize all probanilities\n - Used [IsolationForest](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67205) but still have issues merging that with the other classes proba. I've found anomalies are way to high in my opinion.\n\nThe best I've found this far is to use the product of all opposite probabilities and adapting the mean but I'm not satisfied with that since it kills the other classes logloss.\n\nIf anyone is willing to share his ideas I would be gratefull ;-)\n ", "votes": 50 }, { "id": 410767, "postDate": "2018-10-26T16:15:57.050Z", "content": "I believe a prerequisite for predicting Class 99 is having a very good model for the other classes.\nThe good new is that a score of ~1.0 is achievable without really dealing with class 99. \nThe only thing I did up to this point with this prediction is:\nclass_99=np.where(other_classes.max>0.9 , 0.01, 0.1)\n[it is slightly better then uniform, If I use 0.8 the score degrades, and also other values of probabilities degrade the score]
", "rawMarkdown": "I believe a prerequisite for predicting Class 99 is having a very good model for the other classes.\nThe good new is that a score of ~1.0 is achievable without really dealing with class 99. \nThe only thing I did up to this point with this prediction is:\nclass_99=np.where(other_classes.max>0.9 , 0.01, 0.1)\n[it is slightly better then uniform, If I use 0.8 the score degrades, and also other values of probabilities degrade the score]", "votes": 11, "replies": [ { "id": 411329, "postDate": "2018-10-27T21:27:40.297Z", "content": "I tried your way, it is worse than Olivier's for my last sub, 1.069 vs 1.05.
\n\nI'll stick to Olivier's way for now ;)
", "rawMarkdown": "I tried your way, it is worse than Olivier's for my last sub, 1.069 vs 1.05.\n\nI'll stick to Olivier's way for now ;)" }, { "id": 411509, "postDate": "2018-10-28T10:46:43.497Z", "content": "It is probably model defendant. A small tweak on this calculation, just improved my LB by another 0.07.
", "rawMarkdown": "It is probably model defendant. A small tweak on this calculation, just improved my LB by another 0.07." }, { "id": 411512, "postDate": "2018-10-28T10:52:36.313Z", "content": "\n\n\nIt is probably model defendant.
\n
I'll focus on that when I'll be out of ideas for predicting the known classes. But yes, I agree this has to be tuned for each model/.
", "rawMarkdown": "> It is probably model defendant.\n\nI'll focus on that when I'll be out of ideas for predicting the known classes. But yes, I agree this has to be tuned for each model/.", "votes": 1 } ] }, { "id": 406122, "postDate": "2018-10-18T17:32:16.567Z", "content": "Hello Olivier,
\n\nBy \"inverse\" I assumed you're referring to 1 - x and not 1 / x. If so I think it's clearer to use the term \"opposite\".
Personally I'm using 1 - max(P) where P is the distribution of probabilities for the other classes. I guess this corresponds to the third approach you mentioned. I find this intuitive since essentially it produces high values for \"flat\" distributions where the model isn't sure on any of the known classes. Maybe this idea could be pushed further by looking at the skew/kurtosis of the predicted class distribution.
Anyway, just my 2 cents.
", "rawMarkdown": "Hello Olivier,\n\nBy \"inverse\" I assumed you're referring to `1 - x` and not `1 / x`. If so I think it's clearer to use the term \"opposite\".\n\nPersonally I'm using `1 - max(P)` where `P` is the distribution of probabilities for the other classes. I guess this corresponds to the third approach you mentioned. I find this intuitive since essentially it produces high values for \"flat\" distributions where the model isn't sure on any of the known classes. Maybe this idea could be pushed further by looking at the skew/kurtosis of the predicted class distribution.\n\nAnyway, just my 2 cents. ", "votes": 11, "replies": [ { "id": 406138, "postDate": "2018-10-18T18:06:07.493Z", "content": "Thanks for sharing Max.
\n\nYeah I thought 1 - max(P) would give better results as it was, as you said, very intuitive. Maybe I'm not good enough yet for other classes ;-)
", "rawMarkdown": "Thanks for sharing Max. \n\nYeah I thought 1 - max(P) would give better results as it was, as you said, very intuitive. Maybe I'm not good enough yet for other classes ;-)", "votes": 2 }, { "id": 406143, "postDate": "2018-10-18T18:16:20.473Z", "content": "Indeed, the predictions for class_99 will naturally get better with better predictions for the other classes.
Thanks for the insightful comments. Agreed, 1-max(P) sounds very promising. However, at the moment, I find the same as Olivier. For me, 1-max(p) was worse than the very crude p(class_99)=0.2.
\n\nI have found that using 1-max(p) when max(p) is high (say >0.8) works well to reduce p(class_99) when the classifier is confident about another class, but when max(p) is low this seems to give too much weight to class_99. I think I'll have to revisit this when my base classifier is better!
", "rawMarkdown": "Thanks for the insightful comments. Agreed, 1-max(P) sounds very promising. However, at the moment, I find the same as Olivier. For me, 1-max(p) was worse than the very crude p(class_99)=0.2.\n\nI have found that using 1-max(p) when max(p) is high (say >0.8) works well to reduce p(class_99) when the classifier is confident about another class, but when max(p) is low this seems to give too much weight to class_99. I think I'll have to revisit this when my base classifier is better!", "votes": 3 }, { "id": 408344, "postDate": "2018-10-22T17:57:38.090Z", "content": "Thanks Andy. I have not tried 1 - np.max(probas) lately with my new score. I may have a go at it and burn a submission to check if I can get anything better than my current process described in the public kernel.
My LB probing experiments, up to now, showed me that distribution of class_99 Galactic is around 10x smaller than class_99 ExtraGalactic. Hard coding class_99 of Galactic and ExtraGalactic to constants 0.017 and 0.17 improved my scores.
\n\nAnyone did any other probing experiments?
", "rawMarkdown": "My LB probing experiments, up to now, showed me that distribution of class_99 Galactic is around 10x smaller than class_99 ExtraGalactic. Hard coding class_99 of Galactic and ExtraGalactic to constants 0.017 and 0.17 improved my scores.\n\nAnyone did any other probing experiments?", "votes": 10, "replies": [ { "id": 411260, "postDate": "2018-10-27T18:10:26.903Z", "content": "I've got almost the same result. I took 0.015 and 0.15 - this gave me the best probe (2.081).
\n\nI also probed
\n\n0.020/0.147 -> 2.081
\n\n0.010/0.151 -> 2.081
\n\n0/0.154 -> 2.191
\n\n0.038/0.141 -> 2.084
\n\n0.106/0.113 - > 2.105
\n\n0.167/0.154 -> 2.118
\n\nI made a kernel about this here: https://www.kaggle.com/darbin/weighted-naive-benchmark-lb-2-081
", "rawMarkdown": "I've got almost the same result. I took 0.015 and 0.15 - this gave me the best probe (2.081).\n\nI also probed \n\n0.020/0.147 -> 2.081\n\n0.010/0.151 -> 2.081\n\n0/0.154 -> 2.191\n\n0.038/0.141 -> 2.084\n\n0.106/0.113 - > 2.105\n\n0.167/0.154 -> 2.118\n\nI made a kernel about this here: https://www.kaggle.com/darbin/weighted-naive-benchmark-lb-2-081", "votes": 4 }, { "id": 411287, "postDate": "2018-10-27T19:32:36.867Z", "content": "I tried your two constants, and it is a bit worse than Olivier's way for my latest sub, 1.057 vs 1.052. Maybe the constant should be tuned for a given sub.
", "rawMarkdown": "I tried your two constants, and it is a bit worse than Olivier's way for my latest sub, 1.057 vs 1.052. Maybe the constant should be tuned for a given sub.", "votes": 1 } ] }, { "id": 406406, "postDate": "2018-10-19T07:37:55.637Z", "content": "Most of the class_99 objects (~95% - 97.5%) are placed in the Extragalactic group (out of the Milky Way).
\n\nI tried public LB multiple times to detect this. Check the kernel: https://www.kaggle.com/darbin/weighted-naive-benchmark-lb-2-081
", "rawMarkdown": "Most of the class_99 objects (~95% - 97.5%) are placed in the Extragalactic group (out of the Milky Way).\n\nI tried public LB multiple times to detect this. Check the kernel: https://www.kaggle.com/darbin/weighted-naive-benchmark-lb-2-081", "votes": 9 }, { "id": 406189, "postDate": "2018-10-18T20:06:00.520Z", "content": "Hi,
\n\nI've implemented my own version of one vs all classifier, that trains any scikit model of my choice (and probably of other pkgs with similar interface). Once I get probabilities for all classes, I set P(99)=1-sum(P(.)) and adjust it to zero when it gets negative. Though I haven't yet submitted results and don't know how am I performing.
", "rawMarkdown": "Hi,\n\nI've implemented my own version of one vs all classifier, that trains any scikit model of my choice (and probably of other pkgs with similar interface). Once I get probabilities for all classes, I set P(99)=1-sum(P(.)) and adjust it to zero when it gets negative. Though I haven't yet submitted results and don't know how am I performing. ", "votes": 5, "replies": [ { "id": 406355, "postDate": "2018-10-19T05:38:18.890Z", "content": "Thanks for sharing Linards Kalvāns.
\n\nI had a look at that using lightgbm raw scores lgb.predict(raw_score=True) and sum of individual proba can go up to around 5.
Hope you'll hav more chance with that !
", "rawMarkdown": "Thanks for sharing Linards Kalvāns. \n\nI had a look at that using lightgbm raw scores `lgb.predict(raw_score=True)` and sum of individual proba can go up to around 5. \n\nHope you'll hav more chance with that !\n", "votes": 1 } ] }, { "id": 406165, "postDate": "2018-10-18T18:50:24.710Z", "content": "One more:
\n\nThanks Giba but I must be too silly for that :) I still don't get how to do that...
", "rawMarkdown": "Thanks Giba but I must be too silly for that :) I still don't get how to do that...", "votes": 2 }, { "id": 406410, "postDate": "2018-10-19T07:46:13.967Z", "content": "This way becomes universal for that competition :)
", "rawMarkdown": "This way becomes universal for that competition :)", "votes": 2 }, { "id": 408586, "postDate": "2018-10-23T05:51:38.323Z", "content": "It is consider as common practice now a days ; )
", "rawMarkdown": "It is consider as common practice now a days ; )", "votes": -1 } ] }, { "id": 409628, "postDate": "2018-10-24T16:06:13.033Z", "content": "Just another idea: add noise (random) rows to training data labeled as class 99.
\n\nI haven't had time to try it yet, so don't known if it would help or not. But as the aim of class 99 is to identify new objects not seen previously, it can be considered heterogeneous data, so random features from a correct distribution could help to emulate the heterogeneity.
", "rawMarkdown": "Just another idea: add noise (random) rows to training data labeled as class 99.\n\nI haven't had time to try it yet, so don't known if it would help or not. But as the aim of class 99 is to identify new objects not seen previously, it can be considered heterogeneous data, so random features from a correct distribution could help to emulate the heterogeneity.", "votes": 3, "replies": [ { "id": 410659, "postDate": "2018-10-26T12:18:08.077Z", "content": "Awesome idea! But it might be not as simple as it seems to be since we will probably need to guess what the noise really is and how the noise should look like... But the idea is brilliant and maybe there is a way to build an interesting model!
", "rawMarkdown": "Awesome idea! But it might be not as simple as it seems to be since we will probably need to guess what the noise really is and how the noise should look like... But the idea is brilliant and maybe there is a way to build an interesting model!", "votes": 2 } ] }, { "id": 408951, "postDate": "2018-10-23T16:32:14.813Z", "content": "Clustering with DBScan automatically labels rejects as label == -1. It may be worth using a lookup table to convert dbscan labels to probabilities
", "rawMarkdown": "Clustering with DBScan automatically labels rejects as label == -1. It may be worth using a lookup table to convert dbscan labels to probabilities", "votes": 4, "replies": [ { "id": 408977, "postDate": "2018-10-23T17:19:17.917Z", "content": "Thanks Scirpus, that's on my todo list...
", "rawMarkdown": "Thanks Scirpus, that's on my todo list... ", "votes": 1 }, { "id": 409318, "postDate": "2018-10-24T05:34:50.310Z", "content": "@Scirpus
\n\nI did some search with DBSAN with no real success... I will give it a new try . Thanks.
", "rawMarkdown": "@Scirpus\n\nI did some search with DBSAN with no real success... I will give it a new try . Thanks.", "votes": 1 } ] }, { "id": 408374, "postDate": "2018-10-22T18:42:53.553Z", "content": "To approach this unseen class issue, I think that I'm going to try to layer on top a One-Class SVM on the training data. When predicting, I'll first try to detect whether the test example is in the same \"class\" as the \"training class.\" If not, label it as class 99. If it is, use a full 14-class classifier trained on the seen classes in the training data.
", "rawMarkdown": "To approach this unseen class issue, I think that I'm going to try to layer on top a [One-Class SVM][1] on the training data. When predicting, I'll first try to detect whether the test example is in the same \"class\" as the \"training class.\" If not, label it as class 99. If it is, use a full 14-class classifier trained on the seen classes in the training data.\n\n [1]: http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py \"One-Class SVM\"", "votes": 4, "replies": [ { "id": 408380, "postDate": "2018-10-22T18:49:58.147Z", "content": "Interesting step!
", "rawMarkdown": "Interesting step!" }, { "id": 408636, "postDate": "2018-10-23T07:41:57.620Z", "content": "I tried that with the IsolationForest and I found very weird stuff; moreover: a t-SNE of the training set and another using a large sample of the test set (roughly 6000 rows taken at random) suggest that these two groups come from distinct distributions... I also tried a t-SNE using a concatenation of both the training and the test sets, and this anomaly still appear in the data... In conclusion: it may happen that one-class SVM \"fails\" at finding the \"true\" class_99 objects due to this inconsistency; as if a preprocessing (e.g., maybe a kind of a renormalization in the training or test sets) was necessary prior to doing anything else.
\n\nNote that both the train and test sets used in my analysis included the time series data, conveniently characterized using about 950 features per time series (for example: skewness per band, entropy, median, and so on).
", "rawMarkdown": "I tried that with the *IsolationForest* and I found very weird stuff; moreover: a t-SNE of the training set and another using a large sample of the test set (roughly 6000 rows taken at random) suggest that these two groups come from distinct distributions... I also tried a t-SNE using a concatenation of both the training and the test sets, and this anomaly still appear in the data... In conclusion: it may happen that one-class SVM \"fails\" at finding the \"true\" class_99 objects due to this inconsistency; as if a preprocessing (e.g., maybe a kind of a *renormalization* in the training or test sets) was necessary prior to doing anything else.\n\nNote that both the train and test sets used in my analysis included the time series data, conveniently characterized using about 950 features per time series (for example: skewness per band, entropy, median, and so on).", "votes": 4 }, { "id": 408793, "postDate": "2018-10-23T13:17:40.933Z", "content": "Interesting find. After you mentioned that, this part of the data note sticks out to me.
\n\n\n\n\n3.2. Training and test data
\n \nMoreover, the training data properties are non-representative of distributions of the the test[sic] data set. The training data are mostly composed of nearby, low-redshift, brighter objects while the test data contain more distant (higher redshift) and fainter objects.
\n
Your findings and Ilya Khristoforov's kernel are consistent with this. Maybe developing a One-Class SVM for each category (Milky Way vs. Extragalactic) can help...
", "rawMarkdown": "Interesting find. After you mentioned that, this part of the data note sticks out to me.\n\n> **3.2. Training and test data**\n\n> Moreover, the training data properties are non-representative of distributions of the the test[sic] data set. The training data are mostly composed of nearby, low-redshift, brighter objects while the test data contain more distant (higher redshift) and fainter objects.\n\nYour findings and Ilya Khristoforov's kernel are consistent with this. Maybe developing a One-Class SVM for each category (Milky Way vs. Extragalactic) can help...", "votes": 5 }, { "id": 408807, "postDate": "2018-10-23T13:41:48.957Z", "content": "That's a great idea actually; I'll try the same analysis with the Milky Way vs. Extragalactic and see what happens. I'll post here the findings.
", "rawMarkdown": "That's a great idea actually; I'll try the same analysis with the Milky Way vs. Extragalactic and see what happens. I'll post here the findings.", "votes": 4 } ] }, { "id": 410609, "postDate": "2018-10-26T10:43:29.417Z", "content": "Maybe one class randomforest ? (https://github.com/ngoix/OCRF)\nI try my own version (some kind of mixed lightgbm in this algorithm)\nbut it didn't work , LB is terrible :)
", "rawMarkdown": "Maybe one class randomforest ? (https://github.com/ngoix/OCRF)\nI try my own version (some kind of mixed lightgbm in this algorithm)\nbut it didn't work , LB is terrible :)", "votes": 1 }, { "id": 410576, "postDate": "2018-10-26T09:39:38.080Z", "content": "Olivier,
\n\nIn my limited experience, the way you compute it in your kernel is a bit better than using a constant 1/9 prediction for class 99. LB gain is about 0.02 for me.
", "rawMarkdown": "Olivier, \n\nIn my limited experience, the way you compute it in your kernel is a bit better than using a constant 1/9 prediction for class 99. LB gain is about 0.02 for me.", "votes": 1, "replies": [ { "id": 410693, "postDate": "2018-10-26T13:32:33.493Z", "content": "@CPMP, yes that's what I found. Using different averages for galactic and extragalactic and prod of opposite proba gives the best results for me.
", "rawMarkdown": "@CPMP, yes that's what I found. Using different averages for galactic and extragalactic and prod of opposite proba gives the best results for me.", "votes": 1 } ] }, { "id": 410092, "postDate": "2018-10-25T11:53:01.113Z", "content": "How about connecting 99 to flux rows where detected==0\nIf a flux row is detected then it must be one of the training targets if it isn't then it must be 99 by definition.
", "rawMarkdown": "How about connecting 99 to flux rows where detected==0\nIf a flux row is detected then it must be one of the training targets if it isn't then it must be 99 by definition.", "votes": 1 }, { "id": 409344, "postDate": "2018-10-24T06:37:02.297Z", "content": "Seems LB probing will play a role here.
", "rawMarkdown": "Seems LB probing will play a role here.", "votes": 1, "replies": [ { "id": 409360, "postDate": "2018-10-24T07:17:28.470Z", "content": "Unfortunately I wholeheartedly agree with you\nOne other gaming approach would be to make sure that the predictions are not too confident. A 1 will be punished heavily so I am going to reduce the predictions between a min and a max before row normalizing
", "rawMarkdown": "Unfortunately I wholeheartedly agree with you\nOne other gaming approach would be to make sure that the predictions are not too confident. A 1 will be punished heavily so I am going to reduce the predictions between a min and a max before row normalizing", "votes": 2 } ] }, { "id": 406439, "postDate": "2018-10-19T08:44:25.033Z", "content": "I'm still very far from it, I have also tried auto encoders and distance error but it did not do significant improvement to my score.
", "rawMarkdown": "I'm still very far from it, I have also tried auto encoders and distance error but it did not do significant improvement to my score.", "votes": 1 }, { "id": 406187, "postDate": "2018-10-18T20:02:08.690Z", "content": "@ogreiller
\n\nThanks for raising this issue I have too, and thanks to all for the insightfull replies.
", "rawMarkdown": "@ogreiller\n\nThanks for raising this issue I have too, and thanks to all for the insightfull replies.", "votes": 1 }, { "id": 408351, "postDate": "2018-10-22T18:07:41.940Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 410767, "author_name": "yuval reina", "author_url": "", "post_date": "2018-10-26T16:15:57.050000", "content": "I believe a prerequisite for predicting Class 99 is having a very good model for the other classes.\nThe good new is that a score of ~1.0 is achievable without really dealing with class 99. \nThe only thing I did up to this point with this prediction is:\nclass_99=np.where(other_classes.max>0.9 , 0.01, 0.1)\n[it is slightly better then uniform, If I use 0.8 the score degrades, and also other values of probabilities degrade the score]
", "votes": 11, "replies": [ { "id": 411329, "author_name": "CPMP", "author_url": "", "post_date": "2018-10-27T21:27:40.297000", "content": "I tried your way, it is worse than Olivier's for my last sub, 1.069 vs 1.05.
\n\nI'll stick to Olivier's way for now ;)
", "votes": 0, "replies": [] }, { "id": 411509, "author_name": "yuval reina", "author_url": "", "post_date": "2018-10-28T10:46:43.497000", "content": "It is probably model defendant. A small tweak on this calculation, just improved my LB by another 0.07.
", "votes": 0, "replies": [] }, { "id": 411512, "author_name": "CPMP", "author_url": "", "post_date": "2018-10-28T10:52:36.313000", "content": "\n\n\nIt is probably model defendant.
\n
I'll focus on that when I'll be out of ideas for predicting the known classes. But yes, I agree this has to be tuned for each model/.
", "votes": 1, "replies": [] } ] }, { "id": 406122, "author_name": "Max Halford", "author_url": "", "post_date": "2018-10-18T17:32:16.567000", "content": "Hello Olivier,
\n\nBy \"inverse\" I assumed you're referring to 1 - x and not 1 / x. If so I think it's clearer to use the term \"opposite\".
Personally I'm using 1 - max(P) where P is the distribution of probabilities for the other classes. I guess this corresponds to the third approach you mentioned. I find this intuitive since essentially it produces high values for \"flat\" distributions where the model isn't sure on any of the known classes. Maybe this idea could be pushed further by looking at the skew/kurtosis of the predicted class distribution.
Anyway, just my 2 cents.
", "votes": 11, "replies": [ { "id": 406138, "author_name": "olivier", "author_url": "", "post_date": "2018-10-18T18:06:07.493000", "content": "Thanks for sharing Max.
\n\nYeah I thought 1 - max(P) would give better results as it was, as you said, very intuitive. Maybe I'm not good enough yet for other classes ;-)
", "votes": 2, "replies": [] }, { "id": 406143, "author_name": "Max Halford", "author_url": "", "post_date": "2018-10-18T18:16:20.473000", "content": "Indeed, the predictions for class_99 will naturally get better with better predictions for the other classes.
Thanks for the insightful comments. Agreed, 1-max(P) sounds very promising. However, at the moment, I find the same as Olivier. For me, 1-max(p) was worse than the very crude p(class_99)=0.2.
\n\nI have found that using 1-max(p) when max(p) is high (say >0.8) works well to reduce p(class_99) when the classifier is confident about another class, but when max(p) is low this seems to give too much weight to class_99. I think I'll have to revisit this when my base classifier is better!
", "votes": 3, "replies": [] }, { "id": 408344, "author_name": "olivier", "author_url": "", "post_date": "2018-10-22T17:57:38.090000", "content": "Thanks Andy. I have not tried 1 - np.max(probas) lately with my new score. I may have a go at it and burn a submission to check if I can get anything better than my current process described in the public kernel.
My LB probing experiments, up to now, showed me that distribution of class_99 Galactic is around 10x smaller than class_99 ExtraGalactic. Hard coding class_99 of Galactic and ExtraGalactic to constants 0.017 and 0.17 improved my scores.
\n\nAnyone did any other probing experiments?
", "votes": 10, "replies": [ { "id": 411260, "author_name": "Ilya Khristoforov", "author_url": "", "post_date": "2018-10-27T18:10:26.903000", "content": "I've got almost the same result. I took 0.015 and 0.15 - this gave me the best probe (2.081).
\n\nI also probed
\n\n0.020/0.147 -> 2.081
\n\n0.010/0.151 -> 2.081
\n\n0/0.154 -> 2.191
\n\n0.038/0.141 -> 2.084
\n\n0.106/0.113 - > 2.105
\n\n0.167/0.154 -> 2.118
\n\nI made a kernel about this here: https://www.kaggle.com/darbin/weighted-naive-benchmark-lb-2-081
", "votes": 4, "replies": [] }, { "id": 411287, "author_name": "CPMP", "author_url": "", "post_date": "2018-10-27T19:32:36.867000", "content": "I tried your two constants, and it is a bit worse than Olivier's way for my latest sub, 1.057 vs 1.052. Maybe the constant should be tuned for a given sub.
", "votes": 1, "replies": [] } ] }, { "id": 406406, "author_name": "Ilya Khristoforov", "author_url": "", "post_date": "2018-10-19T07:37:55.637000", "content": "Most of the class_99 objects (~95% - 97.5%) are placed in the Extragalactic group (out of the Milky Way).
\n\nI tried public LB multiple times to detect this. Check the kernel: https://www.kaggle.com/darbin/weighted-naive-benchmark-lb-2-081
", "votes": 9, "replies": [] }, { "id": 406189, "author_name": "Linards Kalvāns", "author_url": "", "post_date": "2018-10-18T20:06:00.520000", "content": "Hi,
\n\nI've implemented my own version of one vs all classifier, that trains any scikit model of my choice (and probably of other pkgs with similar interface). Once I get probabilities for all classes, I set P(99)=1-sum(P(.)) and adjust it to zero when it gets negative. Though I haven't yet submitted results and don't know how am I performing.
", "votes": 5, "replies": [ { "id": 406355, "author_name": "olivier", "author_url": "", "post_date": "2018-10-19T05:38:18.890000", "content": "Thanks for sharing Linards Kalvāns.
\n\nI had a look at that using lightgbm raw scores lgb.predict(raw_score=True) and sum of individual proba can go up to around 5.
Hope you'll hav more chance with that !
", "votes": 1, "replies": [] } ] }, { "id": 406165, "author_name": "Giba", "author_url": "", "post_date": "2018-10-18T18:50:24.710000", "content": "One more:
\n\nThanks Giba but I must be too silly for that :) I still don't get how to do that...
", "votes": 2, "replies": [] }, { "id": 406410, "author_name": "Ilya Khristoforov", "author_url": "", "post_date": "2018-10-19T07:46:13.967000", "content": "This way becomes universal for that competition :)
", "votes": 2, "replies": [] }, { "id": 408586, "author_name": "Harsh", "author_url": "", "post_date": "2018-10-23T05:51:38.323000", "content": "It is consider as common practice now a days ; )
", "votes": -1, "replies": [] } ] }, { "id": 409628, "author_name": "JaimeF", "author_url": "", "post_date": "2018-10-24T16:06:13.033000", "content": "Just another idea: add noise (random) rows to training data labeled as class 99.
\n\nI haven't had time to try it yet, so don't known if it would help or not. But as the aim of class 99 is to identify new objects not seen previously, it can be considered heterogeneous data, so random features from a correct distribution could help to emulate the heterogeneity.
", "votes": 3, "replies": [ { "id": 410659, "author_name": "Ilya Khristoforov", "author_url": "", "post_date": "2018-10-26T12:18:08.077000", "content": "Awesome idea! But it might be not as simple as it seems to be since we will probably need to guess what the noise really is and how the noise should look like... But the idea is brilliant and maybe there is a way to build an interesting model!
", "votes": 2, "replies": [] } ] }, { "id": 408951, "author_name": "Scirpus", "author_url": "", "post_date": "2018-10-23T16:32:14.813000", "content": "Clustering with DBScan automatically labels rejects as label == -1. It may be worth using a lookup table to convert dbscan labels to probabilities
", "votes": 4, "replies": [ { "id": 408977, "author_name": "olivier", "author_url": "", "post_date": "2018-10-23T17:19:17.917000", "content": "Thanks Scirpus, that's on my todo list...
", "votes": 1, "replies": [] }, { "id": 409318, "author_name": "mezoganet", "author_url": "", "post_date": "2018-10-24T05:34:50.310000", "content": "@Scirpus
\n\nI did some search with DBSAN with no real success... I will give it a new try . Thanks.
", "votes": 1, "replies": [] } ] }, { "id": 408374, "author_name": "Mike Holcomb", "author_url": "", "post_date": "2018-10-22T18:42:53.553000", "content": "To approach this unseen class issue, I think that I'm going to try to layer on top a One-Class SVM on the training data. When predicting, I'll first try to detect whether the test example is in the same \"class\" as the \"training class.\" If not, label it as class 99. If it is, use a full 14-class classifier trained on the seen classes in the training data.
", "votes": 4, "replies": [ { "id": 408380, "author_name": "Ilya Khristoforov", "author_url": "", "post_date": "2018-10-22T18:49:58.147000", "content": "Interesting step!
", "votes": 0, "replies": [] }, { "id": 408636, "author_name": "Andreu Sancho-Asensio", "author_url": "", "post_date": "2018-10-23T07:41:57.620000", "content": "I tried that with the IsolationForest and I found very weird stuff; moreover: a t-SNE of the training set and another using a large sample of the test set (roughly 6000 rows taken at random) suggest that these two groups come from distinct distributions... I also tried a t-SNE using a concatenation of both the training and the test sets, and this anomaly still appear in the data... In conclusion: it may happen that one-class SVM \"fails\" at finding the \"true\" class_99 objects due to this inconsistency; as if a preprocessing (e.g., maybe a kind of a renormalization in the training or test sets) was necessary prior to doing anything else.
\n\nNote that both the train and test sets used in my analysis included the time series data, conveniently characterized using about 950 features per time series (for example: skewness per band, entropy, median, and so on).
", "votes": 4, "replies": [] }, { "id": 408793, "author_name": "Mike Holcomb", "author_url": "", "post_date": "2018-10-23T13:17:40.933000", "content": "Interesting find. After you mentioned that, this part of the data note sticks out to me.
\n\n\n\n\n3.2. Training and test data
\n \nMoreover, the training data properties are non-representative of distributions of the the test[sic] data set. The training data are mostly composed of nearby, low-redshift, brighter objects while the test data contain more distant (higher redshift) and fainter objects.
\n
Your findings and Ilya Khristoforov's kernel are consistent with this. Maybe developing a One-Class SVM for each category (Milky Way vs. Extragalactic) can help...
", "votes": 5, "replies": [] }, { "id": 408807, "author_name": "Andreu Sancho-Asensio", "author_url": "", "post_date": "2018-10-23T13:41:48.957000", "content": "That's a great idea actually; I'll try the same analysis with the Milky Way vs. Extragalactic and see what happens. I'll post here the findings.
", "votes": 4, "replies": [] } ] }, { "id": 410609, "author_name": "yentianbao", "author_url": "", "post_date": "2018-10-26T10:43:29.417000", "content": "Maybe one class randomforest ? (https://github.com/ngoix/OCRF)\nI try my own version (some kind of mixed lightgbm in this algorithm)\nbut it didn't work , LB is terrible :)
", "votes": 1, "replies": [] }, { "id": 410576, "author_name": "CPMP", "author_url": "", "post_date": "2018-10-26T09:39:38.080000", "content": "Olivier,
\n\nIn my limited experience, the way you compute it in your kernel is a bit better than using a constant 1/9 prediction for class 99. LB gain is about 0.02 for me.
", "votes": 1, "replies": [ { "id": 410693, "author_name": "olivier", "author_url": "", "post_date": "2018-10-26T13:32:33.493000", "content": "@CPMP, yes that's what I found. Using different averages for galactic and extragalactic and prod of opposite proba gives the best results for me.
", "votes": 1, "replies": [] } ] }, { "id": 410092, "author_name": "Scirpus", "author_url": "", "post_date": "2018-10-25T11:53:01.113000", "content": "How about connecting 99 to flux rows where detected==0\nIf a flux row is detected then it must be one of the training targets if it isn't then it must be 99 by definition.
", "votes": 1, "replies": [] }, { "id": 409344, "author_name": "CPMP", "author_url": "", "post_date": "2018-10-24T06:37:02.297000", "content": "Seems LB probing will play a role here.
", "votes": 1, "replies": [ { "id": 409360, "author_name": "Scirpus", "author_url": "", "post_date": "2018-10-24T07:17:28.470000", "content": "Unfortunately I wholeheartedly agree with you\nOne other gaming approach would be to make sure that the predictions are not too confident. A 1 will be punished heavily so I am going to reduce the predictions between a min and a max before row normalizing
", "votes": 2, "replies": [] } ] }, { "id": 406439, "author_name": "dylonLL", "author_url": "", "post_date": "2018-10-19T08:44:25.033000", "content": "I'm still very far from it, I have also tried auto encoders and distance error but it did not do significant improvement to my score.
", "votes": 1, "replies": [] }, { "id": 406187, "author_name": "mezoganet", "author_url": "", "post_date": "2018-10-18T20:02:08.690000", "content": "@ogreiller
\n\nThanks for raising this issue I have too, and thanks to all for the insightfull replies.
", "votes": 1, "replies": [] }, { "id": 408351, "author_name": "", "author_url": "", "post_date": "2018-10-22T18:07:41.940000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "406121": "I've tried so many things here that I'm not sure I can list them all. I must admit I'm puzzled as to how to derive class 99 probability from the training set. It's the very first time I'm facing a similar problem.\n\nHere are the things I've tried: \n\n - Use a constant\n - Going to logits, derive individual probabilities from that, compute p99 as the product of opposite class probabilities, go back to logit and use softmax to normalize (probably the worst result)\n - Computing the probability as the opposite of the max proba over all classes for each sample\n - Calculate p99 as the product of the other classes opposite proba, going to logits and using softmax to normalize all probanilities\n - Used [IsolationForest](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67205) but still have issues merging that with the other classes proba. I've found anomalies are way to high in my opinion.\n\nThe best I've found this far is to use the product of all opposite probabilities and adapting the mean but I'm not satisfied with that since it kills the other classes logloss.\n\nIf anyone is willing to share his ideas I would be gratefull ;-)\n ", "410767": "I believe a prerequisite for predicting Class 99 is having a very good model for the other classes.\nThe good new is that a score of ~1.0 is achievable without really dealing with class 99. \nThe only thing I did up to this point with this prediction is:\nclass_99=np.where(other_classes.max>0.9 , 0.01, 0.1)\n[it is slightly better then uniform, If I use 0.8 the score degrades, and also other values of probabilities degrade the score]", "406122": "Hello Olivier,\n\nBy \"inverse\" I assumed you're referring to `1 - x` and not `1 / x`. If so I think it's clearer to use the term \"opposite\".\n\nPersonally I'm using `1 - max(P)` where `P` is the distribution of probabilities for the other classes. I guess this corresponds to the third approach you mentioned. I find this intuitive since essentially it produces high values for \"flat\" distributions where the model isn't sure on any of the known classes. Maybe this idea could be pushed further by looking at the skew/kurtosis of the predicted class distribution.\n\nAnyway, just my 2 cents. ", "411143": "My LB probing experiments, up to now, showed me that distribution of class_99 Galactic is around 10x smaller than class_99 ExtraGalactic. Hard coding class_99 of Galactic and ExtraGalactic to constants 0.017 and 0.17 improved my scores.\n\nAnyone did any other probing experiments?", "406406": "Most of the class_99 objects (~95% - 97.5%) are placed in the Extragalactic group (out of the Milky Way).\n\nI tried public LB multiple times to detect this. Check the kernel: https://www.kaggle.com/darbin/weighted-naive-benchmark-lb-2-081", "406189": "Hi,\n\nI've implemented my own version of one vs all classifier, that trains any scikit model of my choice (and probably of other pkgs with similar interface). Once I get probabilities for all classes, I set P(99)=1-sum(P(.)) and adjust it to zero when it gets negative. Though I haven't yet submitted results and don't know how am I performing. ", "406165": "One more:\n\n - probe the Public LB :-)", "409628": "Just another idea: add noise (random) rows to training data labeled as class 99.\n\nI haven't had time to try it yet, so don't known if it would help or not. But as the aim of class 99 is to identify new objects not seen previously, it can be considered heterogeneous data, so random features from a correct distribution could help to emulate the heterogeneity.", "408951": "Clustering with DBScan automatically labels rejects as label == -1. It may be worth using a lookup table to convert dbscan labels to probabilities", "408374": "To approach this unseen class issue, I think that I'm going to try to layer on top a [One-Class SVM][1] on the training data. When predicting, I'll first try to detect whether the test example is in the same \"class\" as the \"training class.\" If not, label it as class 99. If it is, use a full 14-class classifier trained on the seen classes in the training data.\n\n [1]: http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py \"One-Class SVM\"", "410609": "Maybe one class randomforest ? (https://github.com/ngoix/OCRF)\nI try my own version (some kind of mixed lightgbm in this algorithm)\nbut it didn't work , LB is terrible :)", "410576": "Olivier, \n\nIn my limited experience, the way you compute it in your kernel is a bit better than using a constant 1/9 prediction for class 99. LB gain is about 0.02 for me.", "410092": "How about connecting 99 to flux rows where detected==0\nIf a flux row is detected then it must be one of the training targets if it isn't then it must be 99 by definition.", "409344": "Seems LB probing will play a role here.", "406439": "I'm still very far from it, I have also tried auto encoders and distance error but it did not do significant improvement to my score.", "406187": "@ogreiller\n\nThanks for raising this issue I have too, and thanks to all for the insightfull replies.", "408351": "" } }