First of all, congrats to the prize winners and all medal winners. During the competition, I have used the name Sisyphus because Kaggle Leaderboard was normally a daily routine for me to go up and down. But in this competition, I was the most stable Sisyphus ever till the last 3 days, staying on my comfortable 5th place. Then I saw the post from CPMP about their single model scoring 0.750 and I gave up because my blend was barely scoring that much. In the weekend, I went more experimental with this recklessness and it made me 3rd. Then I landed in 4th position in private LB.
\n\nMy final score is a blend of LGB, NN and several stacking models. I will summarize my solution shortly. Later I will edit them with more details most probably. I will put (G) on the things that you can find on my Github repo. For now, my repo doesn't have the full solution but I will complete it soon:\nhttps://github.com/aerdem4/kaggle-plasticc
\n\nWhat didn't work for me?
\n\nAutoencoders: Tried to encode the light curves. Generated vectors didn't help neither as features nor for clustering.
\n\nData augmentation using test set: Tried to augment the data with the samples from the test set using the pseudo-labels. Improved CV, worsened LB.
\n\nRemove background effect: It seems background subtraction was noising the light curves a bit. Tried to normalize them as if they all have black background, couldn't find a way to do it.
\n\nWhat kinda worked?
\n\nProbing class99: I assumed that class99 are similar to certain classes. By probing, I ended up with the information that it is not similar to 15, 64, 67, 88, 90 classes predicted by my model. So I gave different constant to class99 depending on my prediction's highest class.
\n\nAdversarial Validation & Weighting: Since train/test sets were obviously very different, I tried Adversarial Validation. Using all the features, it was very easy to distinguish train and test, so it didn't work for me. Therefore, I only used hostgal_photoz and ddf for weighting the samples. With these sample weights, my score improved around 0.02 and my CV was always higher (worse) than the ones reported on the forum. (G)
\n\nWhat really worked?
\n\nRatio features: I guess most of you did this. Instead of using raw features from each passband, I used their ratio over all passbands.
\n\nStacking: Even a simple Logistic Regression model on top of my LGB model's predictions (confusion matrix) improved my score around 0.04. (G) Later, I did more stacking.
\n\nHostgal_specz model: I trained a model to predict hostgal specz using training set+ test set with hostgal_specz. Then used this model's predictions as a feature.
\n\nUsing normal values and log transformed values together on Neural Net: Having both gives you the opportunity to do all four operations between the features (+, -, / *) because you can write log(xy) as log(x) + log(y). I guess hostgal_photoz was interacting with most of the features that I have.
\n\nBazin: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement. (G)
\n\nLog Ensemble: Instead of averaging predictions, I have averaged the logarithm of the predictions because in the end we try to regress the log values. This worked better for me. (G)
", "messages": [ { "id": 440749, "postDate": "2018-12-18T00:31:26.630Z", "content": "First of all, congrats to the prize winners and all medal winners. During the competition, I have used the name Sisyphus because Kaggle Leaderboard was normally a daily routine for me to go up and down. But in this competition, I was the most stable Sisyphus ever till the last 3 days, staying on my comfortable 5th place. Then I saw the post from CPMP about their single model scoring 0.750 and I gave up because my blend was barely scoring that much. In the weekend, I went more experimental with this recklessness and it made me 3rd. Then I landed in 4th position in private LB.
\n\nMy final score is a blend of LGB, NN and several stacking models. I will summarize my solution shortly. Later I will edit them with more details most probably. I will put (G) on the things that you can find on my Github repo. For now, my repo doesn't have the full solution but I will complete it soon:\nhttps://github.com/aerdem4/kaggle-plasticc
\n\nWhat didn't work for me?
\n\nAutoencoders: Tried to encode the light curves. Generated vectors didn't help neither as features nor for clustering.
\n\nData augmentation using test set: Tried to augment the data with the samples from the test set using the pseudo-labels. Improved CV, worsened LB.
\n\nRemove background effect: It seems background subtraction was noising the light curves a bit. Tried to normalize them as if they all have black background, couldn't find a way to do it.
\n\nWhat kinda worked?
\n\nProbing class99: I assumed that class99 are similar to certain classes. By probing, I ended up with the information that it is not similar to 15, 64, 67, 88, 90 classes predicted by my model. So I gave different constant to class99 depending on my prediction's highest class.
\n\nAdversarial Validation & Weighting: Since train/test sets were obviously very different, I tried Adversarial Validation. Using all the features, it was very easy to distinguish train and test, so it didn't work for me. Therefore, I only used hostgal_photoz and ddf for weighting the samples. With these sample weights, my score improved around 0.02 and my CV was always higher (worse) than the ones reported on the forum. (G)
\n\nWhat really worked?
\n\nRatio features: I guess most of you did this. Instead of using raw features from each passband, I used their ratio over all passbands.
\n\nStacking: Even a simple Logistic Regression model on top of my LGB model's predictions (confusion matrix) improved my score around 0.04. (G) Later, I did more stacking.
\n\nHostgal_specz model: I trained a model to predict hostgal specz using training set+ test set with hostgal_specz. Then used this model's predictions as a feature.
\n\nUsing normal values and log transformed values together on Neural Net: Having both gives you the opportunity to do all four operations between the features (+, -, / *) because you can write log(xy) as log(x) + log(y). I guess hostgal_photoz was interacting with most of the features that I have.
\n\nBazin: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement. (G)
\n\nLog Ensemble: Instead of averaging predictions, I have averaged the logarithm of the predictions because in the end we try to regress the log values. This worked better for me. (G)
", "rawMarkdown": "First of all, congrats to the prize winners and all medal winners. During the competition, I have used the name Sisyphus because Kaggle Leaderboard was normally a daily routine for me to go up and down. But in this competition, I was the most stable Sisyphus ever till the last 3 days, staying on my comfortable 5th place. Then I saw the post from CPMP about their single model scoring 0.750 and I gave up because my blend was barely scoring that much. In the weekend, I went more experimental with this recklessness and it made me 3rd. Then I landed in 4th position in private LB.\n\n\nMy final score is a blend of LGB, NN and several stacking models. I will summarize my solution shortly. Later I will edit them with more details most probably. I will put **(G)** on the things that you can find on my Github repo. For now, my repo doesn't have the full solution but I will complete it soon:\nhttps://github.com/aerdem4/kaggle-plasticc\n\n\n**What didn't work for me?**\n\n\n**Autoencoders**: Tried to encode the light curves. Generated vectors didn't help neither as features nor for clustering.\n\n\n**Data augmentation using test set**: Tried to augment the data with the samples from the test set using the pseudo-labels. Improved CV, worsened LB.\n\n\n**Remove background effect**: It seems background subtraction was noising the light curves a bit. Tried to normalize them as if they all have black background, couldn't find a way to do it.\n\n**What kinda worked?**\n\n\n**Probing class99**: I assumed that class99 are similar to certain classes. By probing, I ended up with the information that it is not similar to 15, 64, 67, 88, 90 classes predicted by my model. So I gave different constant to class99 depending on my prediction's highest class.\n\n\n**Adversarial Validation & Weighting**: Since train/test sets were obviously very different, I tried Adversarial Validation. Using all the features, it was very easy to distinguish train and test, so it didn't work for me. Therefore, I only used hostgal_photoz and ddf for weighting the samples. With these sample weights, my score improved around 0.02 and my CV was always higher (worse) than the ones reported on the forum. **(G)**\n\n\n**What really worked?**\n\n\n**Ratio features**: I guess most of you did this. Instead of using raw features from each passband, I used their ratio over all passbands.\n\n\n**Stacking**: Even a simple Logistic Regression model on top of my LGB model's predictions (confusion matrix) improved my score around 0.04. **(G)** Later, I did more stacking.\n\n\n**Hostgal_specz model**: I trained a model to predict hostgal specz using training set+ test set with hostgal_specz. Then used this model's predictions as a feature.\n\n\n**Using normal values and log transformed values together on Neural Net**: Having both gives you the opportunity to do all four operations between the features (+, -, / *) because you can write log(xy) as log(x) + log(y). I guess hostgal_photoz was interacting with most of the features that I have.\n\n\n**Bazin**: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement. **(G)**\n\n\n**Log Ensemble**: Instead of averaging predictions, I have averaged the logarithm of the predictions because in the end we try to regress the log values. This worked better for me. **(G)**\n", "votes": 105 }, { "id": 440778, "postDate": "2018-12-18T01:15:23.970Z", "content": "Big congrats Ahmed for the #4 place solo! You're on the way to become a GM. Great solution. I also tried to model Hostgalspecz but in my case it didn't helped.
", "rawMarkdown": "Big congrats Ahmed for the #4 place solo! You're on the way to become a GM. Great solution. I also tried to model Hostgalspecz but in my case it didn't helped.", "votes": 6, "replies": [ { "id": 440963, "postDate": "2018-12-18T05:59:21.030Z", "content": "I tried but it didn't help me too.
", "rawMarkdown": "I tried but it didn't help me too." }, { "id": 441021, "postDate": "2018-12-18T07:28:29.173Z", "content": "Thanks a lot! My hostgal specz model improved my CV around 0.005 but LB around 0.015.
", "rawMarkdown": "Thanks a lot! My hostgal specz model improved my CV around 0.005 but LB around 0.015." } ] }, { "id": 442328, "postDate": "2018-12-19T19:58:17.697Z", "content": "Congrats AhmetErdem and thanks for sharing your solution, I'm really impressed with your incredible solo performance!\nI really enjoyed competing with you again, and would like to compete with you in my next competition!
", "rawMarkdown": "Congrats AhmetErdem and thanks for sharing your solution, I'm really impressed with your incredible solo performance!\nI really enjoyed competing with you again, and would like to compete with you in my next competition!\n", "votes": 1, "replies": [ { "id": 442341, "postDate": "2018-12-19T20:23:07.013Z", "content": "Congrats mamas, I really enjoyed competing again with you too. Next time, let's team up instead of competing against each other, otherwise we may become archenemies:)
", "rawMarkdown": "Congrats mamas, I really enjoyed competing again with you too. Next time, let's team up instead of competing against each other, otherwise we may become archenemies:) ", "votes": 1 } ] }, { "id": 442098, "postDate": "2018-12-19T13:40:21.570Z", "content": "Congratulations and thank you for sharing all the code!
", "rawMarkdown": "Congratulations and thank you for sharing all the code!", "votes": 1 }, { "id": 441998, "postDate": "2018-12-19T10:41:19.060Z", "content": "Congrats
", "rawMarkdown": "Congrats", "votes": 1 }, { "id": 441843, "postDate": "2018-12-19T06:39:18.933Z", "content": "Congratulations to you , your work is excellent in the competition ! And thank you for your kind sharing of code and solution. It will help us a lot !
", "rawMarkdown": "Congratulations to you , your work is excellent in the competition ! And thank you for your kind sharing of code and solution. It will help us a lot !", "votes": 1 }, { "id": 441382, "postDate": "2018-12-18T15:51:03.633Z", "content": "congrats!!
", "rawMarkdown": "congrats!!", "votes": 1 }, { "id": 441360, "postDate": "2018-12-18T15:30:16.443Z", "content": "Congratulations :)
", "rawMarkdown": "Congratulations :)", "votes": 1 }, { "id": 441105, "postDate": "2018-12-18T09:13:43.873Z", "content": "Congrats!! By the way, what is the connection between adversarial validation and weighting? And do you use the weighted model to predict directly, or you have some other post process before prediction? Thanks!
", "rawMarkdown": "Congrats!! By the way, what is the connection between adversarial validation and weighting? And do you use the weighted model to predict directly, or you have some other post process before prediction? Thanks!", "votes": 1, "replies": [ { "id": 441532, "postDate": "2018-12-18T18:53:54.480Z", "content": "Thanks and congrats too. So Adversarial Validation is done to understand how different train and test are. And the predictions of Adversarial Model can be interpreted as likelihood of being in the test set, therefore setting sample weights using these likelihoods will make your train set more representative of test set. In that case my adversarial model was extremely simple, just hostgalphotoz ratios, and I weighted train samples accordingly. Then trained my models and directly used the predictions. If you worry about imbalancing the classes while weighting, you may check how I do it without ruining the balance on my repo.
", "rawMarkdown": "Thanks and congrats too. So Adversarial Validation is done to understand how different train and test are. And the predictions of Adversarial Model can be interpreted as likelihood of being in the test set, therefore setting sample weights using these likelihoods will make your train set more representative of test set. In that case my adversarial model was extremely simple, just hostgalphotoz ratios, and I weighted train samples accordingly. Then trained my models and directly used the predictions. If you worry about imbalancing the classes while weighting, you may check how I do it without ruining the balance on my repo." }, { "id": 441543, "postDate": "2018-12-18T19:14:16.353Z", "content": "Thanks a lot for the detailed answer!
", "rawMarkdown": "Thanks a lot for the detailed answer!" } ] }, { "id": 441075, "postDate": "2018-12-18T08:28:57.143Z", "content": "congratulations Ahmet and thanks for sharing your solution.
", "rawMarkdown": "congratulations Ahmet and thanks for sharing your solution.", "votes": 1 }, { "id": 441010, "postDate": "2018-12-18T07:16:28.343Z", "content": "Congratulations Ahmet ! This is an amazing achievement and you're now a few medals away from Competition GM !
\n\nThanks for sharing your code and solution.
", "rawMarkdown": "Congratulations Ahmet ! This is an amazing achievement and you're now a few medals away from Competition GM !\n\nThanks for sharing your code and solution.", "votes": 1, "replies": [ { "id": 441025, "postDate": "2018-12-18T07:33:55.437Z", "content": "Thanks and you are welcome. Let's see if I can manage to be a GM:)
", "rawMarkdown": "Thanks and you are welcome. Let's see if I can manage to be a GM:)" } ] }, { "id": 440984, "postDate": "2018-12-18T06:33:16.690Z", "content": "Congrats again, you are on your way to become a competition grandmaster! Do you remember how much you get from class_99 probing? And from your specz prediction? There are two pieces we lack entirely.
", "rawMarkdown": "Congrats again, you are on your way to become a competition grandmaster! Do you remember how much you get from class_99 probing? And from your specz prediction? There are two pieces we lack entirely.", "votes": 1, "replies": [ { "id": 441024, "postDate": "2018-12-18T07:32:49.780Z", "content": "Thanks and congrats to you too. Compared to giving constant (anything over 0.9 prob -> 0.02, gal -> 0.02, exgal -> 0.018), class99 probing improved my score around 0.03. Hostgalspecz model improved my NN CV by 0.01 but resulted on 0.02 improvement on LB. For LGB, it didn't improve my CV but LB is improved by 0.01. Maybe my models were overfitting to hostgalphotoz and it just prevented it.
", "rawMarkdown": "Thanks and congrats to you too. Compared to giving constant (anything over 0.9 prob -> 0.02, gal -> 0.02, exgal -> 0.018), class99 probing improved my score around 0.03. Hostgalspecz model improved my NN CV by 0.01 but resulted on 0.02 improvement on LB. For LGB, it didn't improve my CV but LB is improved by 0.01. Maybe my models were overfitting to hostgalphotoz and it just prevented it.", "votes": 1 } ] }, { "id": 440938, "postDate": "2018-12-18T05:38:55.800Z", "content": "Congrats! And amazing last week performance! At some point, I realized I really need to log transform the flux and redo everything but it was too late or I was too lazy :P
", "rawMarkdown": "Congrats! And amazing last week performance! At some point, I realized I really need to log transform the flux and redo everything but it was too late or I was too lazy :P", "votes": 1, "replies": [ { "id": 441016, "postDate": "2018-12-18T07:27:01.123Z", "content": "Thanks, congrats to you too. I have just read your solution. Really amazed by how you trained your NN on pseudo-labeled test set. Really good idea!
", "rawMarkdown": "Thanks, congrats to you too. I have just read your solution. Really amazed by how you trained your NN on pseudo-labeled test set. Really good idea!" } ] }, { "id": 440899, "postDate": "2018-12-18T04:31:38.873Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 1 }, { "id": 440893, "postDate": "2018-12-18T04:23:41.317Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 1 }, { "id": 440890, "postDate": "2018-12-18T04:18:24.087Z", "content": "Congrats!
", "rawMarkdown": "Congrats! ", "votes": 1 }, { "id": 440874, "postDate": "2018-12-18T03:51:02.220Z", "content": "congrats!!!
", "rawMarkdown": "congrats!!! ", "votes": 1 }, { "id": 440845, "postDate": "2018-12-18T02:52:14.413Z", "content": "Congratulations! It is interesting that you made some modifications to the light curve fitting. It has always been one of our most important features. We also tried many different types of curve fits but none of them work better than the vanilla version, probably due to overfitting. It appear that a more restrictive rather than expressive model actually works better in this case. We do have some ways to drastically speed up curve fitting time we would like to share later.
\n\nAlso, it appears I really suck at standard Kaggle operations... I had a lot of the problem-specific ideas like you do but can't really effectively verify if they work or not because I had a pretty messy validation workflow and zero stacking / ensembling for most of the competition...
\n\nEDIT: made an attempt to optimise the curve fitting code so someone else interested can try it out.\nhttps://www.kaggle.com/mithrillion/ahmeterdem-s-curve-fit-speed-up
", "rawMarkdown": "Congratulations! It is interesting that you made some modifications to the light curve fitting. It has always been one of our most important features. We also tried many different types of curve fits but none of them work better than the vanilla version, probably due to overfitting. It appear that a more restrictive rather than expressive model actually works better in this case. We do have some ways to drastically speed up curve fitting time we would like to share later.\n\nAlso, it appears I really suck at standard Kaggle operations... I had a lot of the problem-specific ideas like you do but can't really effectively verify if they work or not because I had a pretty messy validation workflow and zero stacking / ensembling for most of the competition...\n\nEDIT: made an attempt to optimise the curve fitting code so someone else interested can try it out.\nhttps://www.kaggle.com/mithrillion/ahmeterdem-s-curve-fit-speed-up", "votes": 1, "replies": [ { "id": 441036, "postDate": "2018-12-18T07:39:42.193Z", "content": "Thanks and congrats to you too. Impressive to see that numba makes it 2 times faster, I will update it on my repo, thanks a lot.
", "rawMarkdown": "Thanks and congrats to you too. Impressive to see that numba makes it 2 times faster, I will update it on my repo, thanks a lot.", "votes": 1 } ] }, { "id": 440836, "postDate": "2018-12-18T02:40:23.080Z", "content": "Congratulations! And very interesting and informative read.
", "rawMarkdown": "Congratulations! And very interesting and informative read. ", "votes": 1 }, { "id": 440814, "postDate": "2018-12-18T01:50:55.090Z", "content": "Congrats!
\n\nBazin: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement.
\n\n-- i have a question. how to use fit method results for model . use predictions for mjd or use period ?
", "rawMarkdown": "Congrats! \n\nBazin: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement.\n\n-- i have a question. how to use fit method results for model . use predictions for mjd or use period ? ", "votes": 1, "replies": [ { "id": 441047, "postDate": "2018-12-18T08:00:17.380Z", "content": "Thanks! I have used the optimal fit parameters and fit_error as features on my models.
", "rawMarkdown": "Thanks! I have used the optimal fit parameters and fit_error as features on my models." } ] }, { "id": 440805, "postDate": "2018-12-18T01:31:09.103Z", "content": "Congrats for the interesting solution and thanks for sharing your clean code
", "rawMarkdown": "Congrats for the interesting solution and thanks for sharing your clean code", "votes": 1 }, { "id": 440793, "postDate": "2018-12-18T01:23:42.417Z", "content": "Big congrats
", "rawMarkdown": "Big congrats", "votes": 1 }, { "id": 440760, "postDate": "2018-12-18T00:48:08.090Z", "content": "Thanks for sharing! \nVery excited about your Hostgalspecz model. Do you know how much it helped?
", "rawMarkdown": "Thanks for sharing! \nVery excited about your Hostgalspecz model. Do you know how much it helped?", "votes": 1 }, { "id": 442883, "postDate": "2018-12-20T16:31:09.490Z", "content": "Now my repo has the full solution. The things that make the code longer but give little improvement are discarded for better readibility. This refactored code is not tested yet, but will be tested soon. Please don't make the assumption that it is mistake free.
", "rawMarkdown": "Now my repo has the full solution. The things that make the code longer but give little improvement are discarded for better readibility. This refactored code is not tested yet, but will be tested soon. Please don't make the assumption that it is mistake free.", "votes": 2 }, { "id": 442836, "postDate": "2018-12-20T15:33:51.053Z", "content": "could you upload your best sub here? \nwe already got to 0.617 on Public LB, and I think it's possible to get to 0.5x if you share your best sub :)\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179
", "rawMarkdown": "could you upload your best sub here? \nwe already got to 0.617 on Public LB, and I think it's possible to get to 0.5x if you share your best sub :)\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179" }, { "id": 442686, "postDate": "2018-12-20T10:30:23.650Z", "content": "Congrats, nice work
", "rawMarkdown": "Congrats, nice work" }, { "id": 441201, "postDate": "2018-12-18T11:58:21.877Z", "content": "could you please explain why stacking helped so much?
", "rawMarkdown": "could you please explain why stacking helped so much?", "replies": [ { "id": 441307, "postDate": "2018-12-18T14:28:05.217Z", "content": "because it always works :)
", "rawMarkdown": "because it always works :)", "votes": 2 }, { "id": 441312, "postDate": "2018-12-18T14:34:40.383Z", "content": "Thanks, it should be in my pipeline for the next competition then
", "rawMarkdown": "Thanks, it should be in my pipeline for the next competition then" } ] }, { "id": 441359, "postDate": "2018-12-18T15:28:49.820Z", "rawMarkdown": "", "isDeleted": true }, { "id": 441034, "postDate": "2018-12-18T07:38:16.073Z", "content": "Congratulations and thanks for sharing.
", "rawMarkdown": "Congratulations and thanks for sharing.", "votes": 1 }, { "id": 440903, "postDate": "2018-12-18T04:36:27.573Z", "content": "Congratulations! Thanks for sharing.
", "rawMarkdown": "Congratulations! Thanks for sharing.", "votes": 1 }, { "id": 440752, "postDate": "2018-12-18T00:35:09.140Z", "content": "Congrats and thanks for sharing.
", "rawMarkdown": "Congrats and thanks for sharing. ", "votes": 1 }, { "id": 444878, "postDate": "2018-12-25T03:43:28.703Z", "content": "thank you.
", "rawMarkdown": "thank you." }, { "id": 443086, "postDate": "2018-12-21T02:03:49.793Z", "content": "congrats and thanks for sharing!
", "rawMarkdown": "congrats and thanks for sharing!" }, { "id": 443084, "postDate": "2018-12-21T01:56:02.187Z", "content": "thanks for sharing.
", "rawMarkdown": "thanks for sharing." } ], "comments": [ { "id": 440778, "author_name": "Giba", "author_url": "", "post_date": "2018-12-18T01:15:23.970000", "content": "Big congrats Ahmed for the #4 place solo! You're on the way to become a GM. Great solution. I also tried to model Hostgalspecz but in my case it didn't helped.
", "votes": 6, "replies": [ { "id": 440963, "author_name": "MuhammedBuyukkinaci", "author_url": "", "post_date": "2018-12-18T05:59:21.030000", "content": "I tried but it didn't help me too.
", "votes": 0, "replies": [] }, { "id": 441021, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-18T07:28:29.173000", "content": "Thanks a lot! My hostgal specz model improved my CV around 0.005 but LB around 0.015.
", "votes": 0, "replies": [] } ] }, { "id": 442328, "author_name": "mamas", "author_url": "", "post_date": "2018-12-19T19:58:17.697000", "content": "Congrats AhmetErdem and thanks for sharing your solution, I'm really impressed with your incredible solo performance!\nI really enjoyed competing with you again, and would like to compete with you in my next competition!
", "votes": 1, "replies": [ { "id": 442341, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-19T20:23:07.013000", "content": "Congrats mamas, I really enjoyed competing again with you too. Next time, let's team up instead of competing against each other, otherwise we may become archenemies:)
", "votes": 1, "replies": [] } ] }, { "id": 442098, "author_name": "Tania J", "author_url": "", "post_date": "2018-12-19T13:40:21.570000", "content": "Congratulations and thank you for sharing all the code!
", "votes": 1, "replies": [] }, { "id": 441998, "author_name": "Dhanashree Shinde", "author_url": "", "post_date": "2018-12-19T10:41:19.060000", "content": "Congrats
", "votes": 1, "replies": [] }, { "id": 441843, "author_name": "yangDDD", "author_url": "", "post_date": "2018-12-19T06:39:18.933000", "content": "Congratulations to you , your work is excellent in the competition ! And thank you for your kind sharing of code and solution. It will help us a lot !
", "votes": 1, "replies": [] }, { "id": 441382, "author_name": "Mayank", "author_url": "", "post_date": "2018-12-18T15:51:03.633000", "content": "congrats!!
", "votes": 1, "replies": [] }, { "id": 441360, "author_name": "Manoj", "author_url": "", "post_date": "2018-12-18T15:30:16.443000", "content": "Congratulations :)
", "votes": 1, "replies": [] }, { "id": 441105, "author_name": "lucaskg", "author_url": "", "post_date": "2018-12-18T09:13:43.873000", "content": "Congrats!! By the way, what is the connection between adversarial validation and weighting? And do you use the weighted model to predict directly, or you have some other post process before prediction? Thanks!
", "votes": 1, "replies": [ { "id": 441532, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-18T18:53:54.480000", "content": "Thanks and congrats too. So Adversarial Validation is done to understand how different train and test are. And the predictions of Adversarial Model can be interpreted as likelihood of being in the test set, therefore setting sample weights using these likelihoods will make your train set more representative of test set. In that case my adversarial model was extremely simple, just hostgalphotoz ratios, and I weighted train samples accordingly. Then trained my models and directly used the predictions. If you worry about imbalancing the classes while weighting, you may check how I do it without ruining the balance on my repo.
", "votes": 0, "replies": [] }, { "id": 441543, "author_name": "lucaskg", "author_url": "", "post_date": "2018-12-18T19:14:16.353000", "content": "Thanks a lot for the detailed answer!
", "votes": 0, "replies": [] } ] }, { "id": 441075, "author_name": "aashish malik", "author_url": "", "post_date": "2018-12-18T08:28:57.143000", "content": "congratulations Ahmet and thanks for sharing your solution.
", "votes": 1, "replies": [] }, { "id": 441010, "author_name": "olivier", "author_url": "", "post_date": "2018-12-18T07:16:28.343000", "content": "Congratulations Ahmet ! This is an amazing achievement and you're now a few medals away from Competition GM !
\n\nThanks for sharing your code and solution.
", "votes": 1, "replies": [ { "id": 441025, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-18T07:33:55.437000", "content": "Thanks and you are welcome. Let's see if I can manage to be a GM:)
", "votes": 0, "replies": [] } ] }, { "id": 440984, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-18T06:33:16.690000", "content": "Congrats again, you are on your way to become a competition grandmaster! Do you remember how much you get from class_99 probing? And from your specz prediction? There are two pieces we lack entirely.
", "votes": 1, "replies": [ { "id": 441024, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-18T07:32:49.780000", "content": "Thanks and congrats to you too. Compared to giving constant (anything over 0.9 prob -> 0.02, gal -> 0.02, exgal -> 0.018), class99 probing improved my score around 0.03. Hostgalspecz model improved my NN CV by 0.01 but resulted on 0.02 improvement on LB. For LGB, it didn't improve my CV but LB is improved by 0.01. Maybe my models were overfitting to hostgalphotoz and it just prevented it.
", "votes": 1, "replies": [] } ] }, { "id": 440938, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2018-12-18T05:38:55.800000", "content": "Congrats! And amazing last week performance! At some point, I realized I really need to log transform the flux and redo everything but it was too late or I was too lazy :P
", "votes": 1, "replies": [ { "id": 441016, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-18T07:27:01.123000", "content": "Thanks, congrats to you too. I have just read your solution. Really amazed by how you trained your NN on pseudo-labeled test set. Really good idea!
", "votes": 0, "replies": [] } ] }, { "id": 440899, "author_name": "Kyle Boone", "author_url": "", "post_date": "2018-12-18T04:31:38.873000", "content": "Congrats!
", "votes": 1, "replies": [] }, { "id": 440893, "author_name": "khyeh", "author_url": "", "post_date": "2018-12-18T04:23:41.317000", "content": "Congrats!
", "votes": 1, "replies": [] }, { "id": 440890, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-18T04:18:24.087000", "content": "Congrats!
", "votes": 1, "replies": [] }, { "id": 440874, "author_name": "Marcus Lin", "author_url": "", "post_date": "2018-12-18T03:51:02.220000", "content": "congrats!!!
", "votes": 1, "replies": [] }, { "id": 440845, "author_name": "Mithrillion", "author_url": "", "post_date": "2018-12-18T02:52:14.413000", "content": "Congratulations! It is interesting that you made some modifications to the light curve fitting. It has always been one of our most important features. We also tried many different types of curve fits but none of them work better than the vanilla version, probably due to overfitting. It appear that a more restrictive rather than expressive model actually works better in this case. We do have some ways to drastically speed up curve fitting time we would like to share later.
\n\nAlso, it appears I really suck at standard Kaggle operations... I had a lot of the problem-specific ideas like you do but can't really effectively verify if they work or not because I had a pretty messy validation workflow and zero stacking / ensembling for most of the competition...
\n\nEDIT: made an attempt to optimise the curve fitting code so someone else interested can try it out.\nhttps://www.kaggle.com/mithrillion/ahmeterdem-s-curve-fit-speed-up
", "votes": 1, "replies": [ { "id": 441036, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-18T07:39:42.193000", "content": "Thanks and congrats to you too. Impressive to see that numba makes it 2 times faster, I will update it on my repo, thanks a lot.
", "votes": 1, "replies": [] } ] }, { "id": 440836, "author_name": "PeterSorensen", "author_url": "", "post_date": "2018-12-18T02:40:23.080000", "content": "Congratulations! And very interesting and informative read.
", "votes": 1, "replies": [] }, { "id": 440814, "author_name": "HanLi", "author_url": "", "post_date": "2018-12-18T01:50:55.090000", "content": "Congrats!
\n\nBazin: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement.
\n\n-- i have a question. how to use fit method results for model . use predictions for mjd or use period ?
", "votes": 1, "replies": [ { "id": 441047, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-18T08:00:17.380000", "content": "Thanks! I have used the optimal fit parameters and fit_error as features on my models.
", "votes": 0, "replies": [] } ] }, { "id": 440805, "author_name": "iprapas", "author_url": "", "post_date": "2018-12-18T01:31:09.103000", "content": "Congrats for the interesting solution and thanks for sharing your clean code
", "votes": 1, "replies": [] }, { "id": 440793, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2018-12-18T01:23:42.417000", "content": "Big congrats
", "votes": 1, "replies": [] }, { "id": 440760, "author_name": "S D", "author_url": "", "post_date": "2018-12-18T00:48:08.090000", "content": "Thanks for sharing! \nVery excited about your Hostgalspecz model. Do you know how much it helped?
", "votes": 1, "replies": [] }, { "id": 442883, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-12-20T16:31:09.490000", "content": "Now my repo has the full solution. The things that make the code longer but give little improvement are discarded for better readibility. This refactored code is not tested yet, but will be tested soon. Please don't make the assumption that it is mistake free.
", "votes": 2, "replies": [] }, { "id": 442836, "author_name": "mamas", "author_url": "", "post_date": "2018-12-20T15:33:51.053000", "content": "could you upload your best sub here? \nwe already got to 0.617 on Public LB, and I think it's possible to get to 0.5x if you share your best sub :)\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179
", "votes": 0, "replies": [] }, { "id": 442686, "author_name": "Koara", "author_url": "", "post_date": "2018-12-20T10:30:23.650000", "content": "Congrats, nice work
", "votes": 0, "replies": [] }, { "id": 441201, "author_name": "iprapas", "author_url": "", "post_date": "2018-12-18T11:58:21.877000", "content": "could you please explain why stacking helped so much?
", "votes": 0, "replies": [ { "id": 441307, "author_name": "", "author_url": "", "post_date": "2018-12-18T14:28:05.217000", "content": "because it always works :)
", "votes": 2, "replies": [] }, { "id": 441312, "author_name": "iprapas", "author_url": "", "post_date": "2018-12-18T14:34:40.383000", "content": "Thanks, it should be in my pipeline for the next competition then
", "votes": 0, "replies": [] } ] }, { "id": 441359, "author_name": "", "author_url": "", "post_date": "2018-12-18T15:28:49.820000", "content": "", "votes": 0, "replies": [] }, { "id": 441034, "author_name": "Eric Vos", "author_url": "", "post_date": "2018-12-18T07:38:16.073000", "content": "Congratulations and thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 440903, "author_name": "yuval reina", "author_url": "", "post_date": "2018-12-18T04:36:27.573000", "content": "Congratulations! Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 440752, "author_name": "", "author_url": "", "post_date": "2018-12-18T00:35:09.140000", "content": "", "votes": 1, "replies": [] }, { "id": 444878, "author_name": "", "author_url": "", "post_date": "2018-12-25T03:43:28.703000", "content": "", "votes": 0, "replies": [] }, { "id": 443086, "author_name": "", "author_url": "", "post_date": "2018-12-21T02:03:49.793000", "content": "", "votes": 0, "replies": [] }, { "id": 443084, "author_name": "", "author_url": "", "post_date": "2018-12-21T01:56:02.187000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "440749": "First of all, congrats to the prize winners and all medal winners. During the competition, I have used the name Sisyphus because Kaggle Leaderboard was normally a daily routine for me to go up and down. But in this competition, I was the most stable Sisyphus ever till the last 3 days, staying on my comfortable 5th place. Then I saw the post from CPMP about their single model scoring 0.750 and I gave up because my blend was barely scoring that much. In the weekend, I went more experimental with this recklessness and it made me 3rd. Then I landed in 4th position in private LB.\n\n\nMy final score is a blend of LGB, NN and several stacking models. I will summarize my solution shortly. Later I will edit them with more details most probably. I will put **(G)** on the things that you can find on my Github repo. For now, my repo doesn't have the full solution but I will complete it soon:\nhttps://github.com/aerdem4/kaggle-plasticc\n\n\n**What didn't work for me?**\n\n\n**Autoencoders**: Tried to encode the light curves. Generated vectors didn't help neither as features nor for clustering.\n\n\n**Data augmentation using test set**: Tried to augment the data with the samples from the test set using the pseudo-labels. Improved CV, worsened LB.\n\n\n**Remove background effect**: It seems background subtraction was noising the light curves a bit. Tried to normalize them as if they all have black background, couldn't find a way to do it.\n\n**What kinda worked?**\n\n\n**Probing class99**: I assumed that class99 are similar to certain classes. By probing, I ended up with the information that it is not similar to 15, 64, 67, 88, 90 classes predicted by my model. So I gave different constant to class99 depending on my prediction's highest class.\n\n\n**Adversarial Validation & Weighting**: Since train/test sets were obviously very different, I tried Adversarial Validation. Using all the features, it was very easy to distinguish train and test, so it didn't work for me. Therefore, I only used hostgal_photoz and ddf for weighting the samples. With these sample weights, my score improved around 0.02 and my CV was always higher (worse) than the ones reported on the forum. **(G)**\n\n\n**What really worked?**\n\n\n**Ratio features**: I guess most of you did this. Instead of using raw features from each passband, I used their ratio over all passbands.\n\n\n**Stacking**: Even a simple Logistic Regression model on top of my LGB model's predictions (confusion matrix) improved my score around 0.04. **(G)** Later, I did more stacking.\n\n\n**Hostgal_specz model**: I trained a model to predict hostgal specz using training set+ test set with hostgal_specz. Then used this model's predictions as a feature.\n\n\n**Using normal values and log transformed values together on Neural Net**: Having both gives you the opportunity to do all four operations between the features (+, -, / *) because you can write log(xy) as log(x) + log(y). I guess hostgal_photoz was interacting with most of the features that I have.\n\n\n**Bazin**: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement. **(G)**\n\n\n**Log Ensemble**: Instead of averaging predictions, I have averaged the logarithm of the predictions because in the end we try to regress the log values. This worked better for me. **(G)**\n", "440778": "Big congrats Ahmed for the #4 place solo! You're on the way to become a GM. Great solution. I also tried to model Hostgalspecz but in my case it didn't helped.", "442328": "Congrats AhmetErdem and thanks for sharing your solution, I'm really impressed with your incredible solo performance!\nI really enjoyed competing with you again, and would like to compete with you in my next competition!\n", "442098": "Congratulations and thank you for sharing all the code!", "441998": "Congrats", "441843": "Congratulations to you , your work is excellent in the competition ! And thank you for your kind sharing of code and solution. It will help us a lot !", "441382": "congrats!!", "441360": "Congratulations :)", "441105": "Congrats!! By the way, what is the connection between adversarial validation and weighting? And do you use the weighted model to predict directly, or you have some other post process before prediction? Thanks!", "441075": "congratulations Ahmet and thanks for sharing your solution.", "441010": "Congratulations Ahmet ! This is an amazing achievement and you're now a few medals away from Competition GM !\n\nThanks for sharing your code and solution.", "440984": "Congrats again, you are on your way to become a competition grandmaster! Do you remember how much you get from class_99 probing? And from your specz prediction? There are two pieces we lack entirely.", "440938": "Congrats! And amazing last week performance! At some point, I realized I really need to log transform the flux and redo everything but it was too late or I was too lazy :P", "440899": "Congrats!", "440893": "Congrats!", "440890": "Congrats! ", "440874": "congrats!!! ", "440845": "Congratulations! It is interesting that you made some modifications to the light curve fitting. It has always been one of our most important features. We also tried many different types of curve fits but none of them work better than the vanilla version, probably due to overfitting. It appear that a more restrictive rather than expressive model actually works better in this case. We do have some ways to drastically speed up curve fitting time we would like to share later.\n\nAlso, it appears I really suck at standard Kaggle operations... I had a lot of the problem-specific ideas like you do but can't really effectively verify if they work or not because I had a pretty messy validation workflow and zero stacking / ensembling for most of the competition...\n\nEDIT: made an attempt to optimise the curve fitting code so someone else interested can try it out.\nhttps://www.kaggle.com/mithrillion/ahmeterdem-s-curve-fit-speed-up", "440836": "Congratulations! And very interesting and informative read. ", "440814": "Congrats! \n\nBazin: This is a light curve fit method that can be found on: https://github.com/COINtoolbox/ActSNClass/blob/master/examples/1_fit_LC/fit_lc_parametric.py I have changed it significantly and it gave me very big improvement.\n\n-- i have a question. how to use fit method results for model . use predictions for mjd or use period ? ", "440805": "Congrats for the interesting solution and thanks for sharing your clean code", "440793": "Big congrats", "440760": "Thanks for sharing! \nVery excited about your Hostgalspecz model. Do you know how much it helped?", "442883": "Now my repo has the full solution. The things that make the code longer but give little improvement are discarded for better readibility. This refactored code is not tested yet, but will be tested soon. Please don't make the assumption that it is mistake free.", "442836": "could you upload your best sub here? \nwe already got to 0.617 on Public LB, and I think it's possible to get to 0.5x if you share your best sub :)\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179", "442686": "Congrats, nice work", "441201": "could you please explain why stacking helped so much?", "441359": "", "441034": "Congratulations and thanks for sharing.", "440903": "Congratulations! Thanks for sharing.", "440752": "Congrats and thanks for sharing. ", "444878": "thank you.", "443086": "congrats and thanks for sharing!", "443084": "thanks for sharing." } }