![\"gp](\"https://storage.googleapis.com/kaggle-forum-message-attachments/440847/10890/gp_example.png\")
EDIT: code is now available on my github page at https://github.com/kboone/avocado
\n\nFirst of all, thanks to everyone who participated in this competition! I learned a lot doing it, and I have enjoyed all of the discussions that I had with you. Here is an overview of my model that took 1st place in this competition. I will be releasing the code with a full writeup shortly.
\n\nI am an astronomer studying supernova cosmology, so my work mainly focused on trying to tell the different supernova types apart. This ended up working out well because everything else was fairly easy to tell apart. Here is a summary of my solution:
\n\nI first use Gaussian process (GP) regression to extract features. I trained a GP on each object using a Matern Kernel with a fixed length scale in the wavelength direction and a variable length scale in the time direction. My machine could do ~10 fits per second so it took around 3 days of computation time to do all the fits. Gaussian processes produce very nice models for well-sampled lightcurves, and are able to get a nice model even when the measurements are in different bands. They also handle measurements with large uncertainties very gracefully. For poorly sampled lightcurves, the GP fits the available data well, but doesn't always do great for extrapolation. Here is an example of what comes out:
\n\n
Using the GP predictions, I calculated lots of different features. The distinguishing features of supernovae are their peak brightnesses and the width of their lightcurves, so I put several measures of those into the model. For poorly sampled lightcurves, the GP doesn't always give great results, so I added features that let the model know how well the GP is doing. This basically boils down to counting the number of observations in different windows around maximum light. I also added features related to the signal-to-noise in each band, and some simple peak detection and counting to help with the non-supernova classes.
\n\nNow the training set is very different from the test set. To deal with this, I took every lightcurve in the training set and degraded it up to 40 times to get something that looks like the less well-sampled lightcurves in the test set. The degradation includes:
\n\nThis degradation was all tuned to the training/test datasets, and no external data was used. After this procedure, I end up with a training set of ~270000 objects that is much more representative of the test set than the original training set. I trained a LightGBM model on this training set using 5-fold cross-validation and making sure that I kept the up to 40 degradations of each object in the same folds. After tuning this model, I get a CV of around 0.4 on the original training set. Here is the confusion matrix:
\n\n![\"confusion](\"https://storage.googleapis.com/kaggle-forum-message-attachments/438639/10876/confusion_matrix.png\")
There are some interesting differences compared to CPMP's confusion matrix, such as the fact that I have much lower accuracy on class 6 objects than CPMP. This appears to be due to the fact that my degradation procedure produces more low signal-to-noise class 65 than class 6 objects. It will be interesting to see if my method really does reproduce the test set better.
\n\nI played a lot with how to identify the class 99 targets. I found that the tree based models that I used are not very good for outlier detection. My best results came from choosing a flat score to give to the class 99 objects, and then using that in the soft-max to get the final probabilities. Using this, I got what I consider my best real score of 0.726 on the public leaderboard.
\n\nAfter trying to improve this score for a long time and getting nowhere, with a week to go I realized that I could figure out what the class 99 objects look like/don't look like by probing the leaderboard. This defeats the purpose of the class 99 prediction, and unfortunately ends up doing much better than any real estimate of the class 99 objects. I contacted the organizers about it, and was told that it was within the Kaggle rules to do this. In the end, I found that my best prediction of the class 99 objects was a weighted average of the predictions for classes 42, 52, 62 and 95. This trick boosted my final score to 0.670 on the public leaderboard. It will be interesting to see what other competitors did here.
\n\nOverall, I really enjoyed this competition and I learned a lot! I am working on cleaning up my code and making it friendly for others to play with. I think that there is quite a bit of room to improve the tuning of my model, and I did not try doing any kind of ensembling or using classifiers other than LGBM. Thanks to everyone who participated, especially @cpmpml who initiated a lot of great discussions and @ogrellier whose kernel I started with for my work!
\n\nFor any astronomers who participated, I'll be at AAS in a couple weeks. I'd love to meet up and discuss the competition!
", "messages": [ { "id": 440847, "postDate": "2018-12-18T02:53:12.237Z", "content": "EDIT: code is now available on my github page at https://github.com/kboone/avocado
\n\nFirst of all, thanks to everyone who participated in this competition! I learned a lot doing it, and I have enjoyed all of the discussions that I had with you. Here is an overview of my model that took 1st place in this competition. I will be releasing the code with a full writeup shortly.
\n\nI am an astronomer studying supernova cosmology, so my work mainly focused on trying to tell the different supernova types apart. This ended up working out well because everything else was fairly easy to tell apart. Here is a summary of my solution:
\n\nI first use Gaussian process (GP) regression to extract features. I trained a GP on each object using a Matern Kernel with a fixed length scale in the wavelength direction and a variable length scale in the time direction. My machine could do ~10 fits per second so it took around 3 days of computation time to do all the fits. Gaussian processes produce very nice models for well-sampled lightcurves, and are able to get a nice model even when the measurements are in different bands. They also handle measurements with large uncertainties very gracefully. For poorly sampled lightcurves, the GP fits the available data well, but doesn't always do great for extrapolation. Here is an example of what comes out:
\n\n
Using the GP predictions, I calculated lots of different features. The distinguishing features of supernovae are their peak brightnesses and the width of their lightcurves, so I put several measures of those into the model. For poorly sampled lightcurves, the GP doesn't always give great results, so I added features that let the model know how well the GP is doing. This basically boils down to counting the number of observations in different windows around maximum light. I also added features related to the signal-to-noise in each band, and some simple peak detection and counting to help with the non-supernova classes.
\n\nNow the training set is very different from the test set. To deal with this, I took every lightcurve in the training set and degraded it up to 40 times to get something that looks like the less well-sampled lightcurves in the test set. The degradation includes:
\n\nThis degradation was all tuned to the training/test datasets, and no external data was used. After this procedure, I end up with a training set of ~270000 objects that is much more representative of the test set than the original training set. I trained a LightGBM model on this training set using 5-fold cross-validation and making sure that I kept the up to 40 degradations of each object in the same folds. After tuning this model, I get a CV of around 0.4 on the original training set. Here is the confusion matrix:
\n\n
There are some interesting differences compared to CPMP's confusion matrix, such as the fact that I have much lower accuracy on class 6 objects than CPMP. This appears to be due to the fact that my degradation procedure produces more low signal-to-noise class 65 than class 6 objects. It will be interesting to see if my method really does reproduce the test set better.
\n\nI played a lot with how to identify the class 99 targets. I found that the tree based models that I used are not very good for outlier detection. My best results came from choosing a flat score to give to the class 99 objects, and then using that in the soft-max to get the final probabilities. Using this, I got what I consider my best real score of 0.726 on the public leaderboard.
\n\nAfter trying to improve this score for a long time and getting nowhere, with a week to go I realized that I could figure out what the class 99 objects look like/don't look like by probing the leaderboard. This defeats the purpose of the class 99 prediction, and unfortunately ends up doing much better than any real estimate of the class 99 objects. I contacted the organizers about it, and was told that it was within the Kaggle rules to do this. In the end, I found that my best prediction of the class 99 objects was a weighted average of the predictions for classes 42, 52, 62 and 95. This trick boosted my final score to 0.670 on the public leaderboard. It will be interesting to see what other competitors did here.
\n\nOverall, I really enjoyed this competition and I learned a lot! I am working on cleaning up my code and making it friendly for others to play with. I think that there is quite a bit of room to improve the tuning of my model, and I did not try doing any kind of ensembling or using classifiers other than LGBM. Thanks to everyone who participated, especially @cpmpml who initiated a lot of great discussions and @ogrellier whose kernel I started with for my work!
\n\nFor any astronomers who participated, I'll be at AAS in a couple weeks. I'd love to meet up and discuss the competition!
", "rawMarkdown": "EDIT: code is now available on my github page at https://github.com/kboone/avocado\n\nFirst of all, thanks to everyone who participated in this competition! I learned a lot doing it, and I have enjoyed all of the discussions that I had with you. Here is an overview of my model that took 1st place in this competition. I will be releasing the code with a full writeup shortly.\n\nI am an astronomer studying supernova cosmology, so my work mainly focused on trying to tell the different supernova types apart. This ended up working out well because everything else was fairly easy to tell apart. Here is a summary of my solution:\n\n- Augmented the training set by degrading the well-observed lightcurves in the training set to match the properties of the test set.\n- Use Gaussian processes to predict the lightcurves.\n- Measured 200 features on the raw data and Gaussian process predictions.\n- Trained a single LGBM model with 5-fold cross-validation.\n\nI first use Gaussian process (GP) regression to extract features. I trained a GP on each object using a Matern Kernel with a fixed length scale in the wavelength direction and a variable length scale in the time direction. My machine could do ~10 fits per second so it took around 3 days of computation time to do all the fits. Gaussian processes produce very nice models for well-sampled lightcurves, and are able to get a nice model even when the measurements are in different bands. They also handle measurements with large uncertainties very gracefully. For poorly sampled lightcurves, the GP fits the available data well, but doesn't always do great for extrapolation. Here is an example of what comes out:\n\n![gp example][1]\n\nUsing the GP predictions, I calculated lots of different features. The distinguishing features of supernovae are their peak brightnesses and the width of their lightcurves, so I put several measures of those into the model. For poorly sampled lightcurves, the GP doesn't always give great results, so I added features that let the model know how well the GP is doing. This basically boils down to counting the number of observations in different windows around maximum light. I also added features related to the signal-to-noise in each band, and some simple peak detection and counting to help with the non-supernova classes.\n\nNow the training set is very different from the test set. To deal with this, I took every lightcurve in the training set and degraded it up to 40 times to get something that looks like the less well-sampled lightcurves in the test set. The degradation includes:\n\n- Modifying the brightness of galactic objects.\n- Modifying the redshift of extragalactic objects (including dilating time and changing the brightness).\n- Adding in large \"gaps\" like the ones in the real data that show up because of the time of year.\n- Choosing a new photo-z and photo-z error for the observation based on a model of how the spec-zs in the data turn into photo-zs.\n- Simulating the detection to choose which objects would be included in the dataset that we were given.\n\nThis degradation was all tuned to the training/test datasets, and no external data was used. After this procedure, I end up with a training set of ~270000 objects that is much more representative of the test set than the original training set. I trained a LightGBM model on this training set using 5-fold cross-validation and making sure that I kept the up to 40 degradations of each object in the same folds. After tuning this model, I get a CV of around 0.4 on the original training set. Here is the confusion matrix:\n\n![confusion matrix][2]\n\nThere are some interesting differences compared to [CPMP's confusion matrix][3], such as the fact that I have much lower accuracy on class 6 objects than CPMP. This appears to be due to the fact that my degradation procedure produces more low signal-to-noise class 65 than class 6 objects. It will be interesting to see if my method really does reproduce the test set better.\n\nI played a lot with how to identify the class 99 targets. I found that the tree based models that I used are not very good for outlier detection. My best results came from choosing a flat score to give to the class 99 objects, and then using that in the soft-max to get the final probabilities. Using this, I got what I consider my best real score of 0.726 on the public leaderboard.\n\nAfter trying to improve this score for a long time and getting nowhere, with a week to go I realized that I could figure out what the class 99 objects look like/don't look like by probing the leaderboard. This defeats the purpose of the class 99 prediction, and unfortunately ends up doing much better than any real estimate of the class 99 objects. I contacted the organizers about it, and was told that it was within the Kaggle rules to do this. In the end, I found that my best prediction of the class 99 objects was a weighted average of the predictions for classes 42, 52, 62 and 95. This trick boosted my final score to 0.670 on the public leaderboard. It will be interesting to see what other competitors did here.\n\nOverall, I really enjoyed this competition and I learned a lot! I am working on cleaning up my code and making it friendly for others to play with. I think that there is quite a bit of room to improve the tuning of my model, and I did not try doing any kind of ensembling or using classifiers other than LGBM. Thanks to everyone who participated, especially @cpmpml who initiated a lot of great discussions and @ogrellier whose kernel I started with for my work!\n\nFor any astronomers who participated, I'll be at AAS in a couple weeks. I'd love to meet up and discuss the competition!\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/440847/10890/gp_example.png\n [2]: https://storage.googleapis.com/kaggle-forum-message-attachments/438639/10876/confusion_matrix.png\n [3]: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74564", "votes": 206 }, { "id": 445181, "postDate": "2018-12-25T20:58:15.600Z", "content": "The code is now available on my github page at https://github.com/kboone/plasticc
", "rawMarkdown": "The code is now available on my github page at https://github.com/kboone/plasticc", "votes": 6 }, { "id": 441007, "postDate": "2018-12-18T07:13:19.880Z", "content": "Congratulations Kyle ! Awesome work.
\n\nAfter reading your write up I believe I now understand the dataset better ;-)
", "rawMarkdown": "Congratulations Kyle ! Awesome work. \n\nAfter reading your write up I believe I now understand the dataset better ;-)", "votes": 3 }, { "id": 440856, "postDate": "2018-12-18T03:08:19.143Z", "content": "Thanks for sharing! It's amazing! @cpmpml is indeed a discussion GM...\nIf we had 270000 objects, things would be very much different for all of us, lol\nWe did some augmentation in training, but far from your level...
\n\nCongratulation!
", "rawMarkdown": "Thanks for sharing! It's amazing! @cpmpml is indeed a discussion GM...\nIf we had 270000 objects, things would be very much different for all of us, lol\nWe did some augmentation in training, but far from your level...\n\nCongratulation!", "votes": 4 }, { "id": 443230, "postDate": "2018-12-21T08:55:18.790Z", "content": "Congrats Kyle Boone, thank you for detailed explanation.
", "rawMarkdown": "Congrats Kyle Boone, thank you for detailed explanation.", "votes": 1 }, { "id": 442329, "postDate": "2018-12-19T20:00:54.677Z", "content": "Congrats Kyle Boone, you are the real super man in this competition :)\nI hope you continue kaggle and if possible I would like to compete with you again!
", "rawMarkdown": "Congrats Kyle Boone, you are the real super man in this competition :)\nI hope you continue kaggle and if possible I would like to compete with you again!", "votes": 1 }, { "id": 441202, "postDate": "2018-12-18T11:59:12.083Z", "content": "Congratulations Kyle ! I'm so happy so see you win the race. This is also a good lesson that domain knowledge is useful to design the solution, and I'm pretty confident that mixing the solutions of the top rankers can still improve a lot the solution. In other words, you made a clear case for a collaborative approach to the solution...
\n\nRegarding the GP, I think the way data are extrapolated is one of the key issues of this challenge. Have you considered at some point to build templates for the lightcuves ? This was the approach I was trying, as it alleviates the problem of interpolation over long range of missing data by using the template. Of course, interpolation is then dependent on the hypothesis on the class. See for instance what this approach gives below. It probably beats what GP can do (erro band represent class variability, not object interpolation uncertainty)
\n\n![\"Template](\"https://storage.googleapis.com/kaggle-forum-message-attachments/441202/10896/Unknown.png\")
Hi Manu, good to hear from you. It's been a while! I definitely think that l could mix my solution with other teams' solutions and get a boost. Since I had a lot to learn, I decided to focus on one thing and try to get it right. It seems like in the end lgbm models outperformed everything else by quite a bit, so I'm not sure how much of a gain I'd get.
\n\nWhat did you use to build your models? I thought about doing that, but decided that I didn't have the computational resources or time to do fits to everything. I'm guessing that you did something like GP interpolation of well sampled timeseries + factor analysis?
", "rawMarkdown": "Hi Manu, good to hear from you. It's been a while! I definitely think that l could mix my solution with other teams' solutions and get a boost. Since I had a lot to learn, I decided to focus on one thing and try to get it right. It seems like in the end lgbm models outperformed everything else by quite a bit, so I'm not sure how much of a gain I'd get.\n\nWhat did you use to build your models? I thought about doing that, but decided that I didn't have the computational resources or time to do fits to everything. I'm guessing that you did something like GP interpolation of well sampled timeseries + factor analysis?" }, { "id": 442002, "postDate": "2018-12-19T10:47:08.590Z", "content": "You are right about this: fitting a model per light-curve takes time to understand the data, plus time to converge the model, so you got the right hint about going directly to some more generic method, while I'm still struggling with doing the fits ;-) . I was first looking at GP, then switched to kernel methods, then getting back to good old splines to fit the ensemble of time series (not fitting one by one, but rather building the model with everything at once)... This is a variation of the trick we used to greatly improve GP performance by deriving an average value from the data (an often overlooked problem). For the next step I'm still undecided between EM algorithms to provide a model with linear variations (better for porting the model to the test set) and K-means inspired methods which may provide better hint for interpreting the data.\nMy main driver in this is to understand how statistics-based models with a likelihood (supposedly an optimal method) can be compared with all novelties in ML which seem to be able to outperform it !
", "rawMarkdown": "You are right about this: fitting a model per light-curve takes time to understand the data, plus time to converge the model, so you got the right hint about going directly to some more generic method, while I'm still struggling with doing the fits ;-) . I was first looking at GP, then switched to kernel methods, then getting back to good old splines to fit the ensemble of time series (not fitting one by one, but rather building the model with everything at once)... This is a variation of the trick we used to greatly improve GP performance by deriving an average value from the data (an often overlooked problem). For the next step I'm still undecided between EM algorithms to provide a model with linear variations (better for porting the model to the test set) and K-means inspired methods which may provide better hint for interpreting the data.\nMy main driver in this is to understand how statistics-based models with a likelihood (supposedly an optimal method) can be compared with all novelties in ML which seem to be able to outperform it !" } ] }, { "id": 440973, "postDate": "2018-12-18T06:08:18.200Z", "content": "Congrats! You are a great specialist in your \n domain ( astronomy) and an awesome ML practicioner . You have shown us how to combine talent and work in an extraordinary and optimized way. Thanks to the organizers for allowing us to struggle and enjoy in this challenge with data, code,algorithms and patience , beatiful ingredients for a stellar soup, and to our families and friends for giving us time to cook all this.
", "rawMarkdown": "Congrats! You are a great specialist in your \n domain ( astronomy) and an awesome ML practicioner . You have shown us how to combine talent and work in an extraordinary and optimized way. Thanks to the organizers for allowing us to struggle and enjoy in this challenge with data, code,algorithms and patience , beatiful ingredients for a stellar soup, and to our families and friends for giving us time to cook all this.", "votes": 1 }, { "id": 441046, "postDate": "2018-12-18T07:55:49.953Z", "content": "Awesome work Kyle! I hope you include more about how you fit the GPs in your full write-up. Even basic (GP 101) stuff would be useful for many people I think, who have come across them for the first time in this competition. For instance, why did you choose the Matern kernel over others (I guess its not by chance that both you and CPMP used it) and how you choose the parameters. Also any links to good resources for learning about GPs would be appreciated. I tried GPs towards the end but never managed to get good fits.
", "rawMarkdown": "Awesome work Kyle! I hope you include more about how you fit the GPs in your full write-up. Even basic (GP 101) stuff would be useful for many people I think, who have come across them for the first time in this competition. For instance, why did you choose the Matern kernel over others (I guess its not by chance that both you and CPMP used it) and how you choose the parameters. Also any links to good resources for learning about GPs would be appreciated. I tried GPs towards the end but never managed to get good fits.", "votes": 2 }, { "id": 440863, "postDate": "2018-12-18T03:16:27.900Z", "content": "Congrats Kyle. Would you like to give more explanations about \"Measured 200 features on the raw data and Gaussian process predictions.\" Especially how to use GP to extract important features, this would be quite helpful for many other related problems.
", "rawMarkdown": "Congrats Kyle. Would you like to give more explanations about \"Measured 200 features on the raw data and Gaussian process predictions.\" Especially how to use GP to extract important features, this would be quite helpful for many other related problems. ", "votes": 2, "replies": [ { "id": 440875, "postDate": "2018-12-18T03:52:00.963Z", "content": "I'll release a full write-up when I get the code all cleaned up that details all of these features.
", "rawMarkdown": "I'll release a full write-up when I get the code all cleaned up that details all of these features.", "votes": 2 } ] }, { "id": 457546, "postDate": "2019-01-17T17:02:17.610Z", "content": "Hi guys I am new to this competition and to the field of machine learning, (4th year computer science undergraduate), congratulations on the 1st place solution, If it would be possible for anyone to help me understand the approach to this challenge, regarding the dimensionality reduction, lightcurve fitting(why it is important and how is it different to fitting a classifier), and the importance of timeseries and how they are produced and what are their uses. I am very eager to learn these methods and techniques but without a clear idea of i would appreciate anyones help to get me comfortable to explore on my own.
", "rawMarkdown": "Hi guys I am new to this competition and to the field of machine learning, (4th year computer science undergraduate), congratulations on the 1st place solution, If it would be possible for anyone to help me understand the approach to this challenge, regarding the dimensionality reduction, lightcurve fitting(why it is important and how is it different to fitting a classifier), and the importance of timeseries and how they are produced and what are their uses. I am very eager to learn these methods and techniques but without a clear idea of i would appreciate anyones help to get me comfortable to explore on my own. " }, { "id": 448219, "postDate": "2018-12-31T13:56:48.133Z", "content": "Hi Kyle,
\n\n Great work! and thanks for the nice post and code. I am currently working on a classifier and would like to use your feature set for experimentation and testing. I kindly request for simple features (200) and class labels file, for both train and test set. It would be great if I have these two files, as I dont have the environment/facility to use your python code on github.\nThank you in advance.\nRegards\nRajesh
", "rawMarkdown": "Hi Kyle,\n\n Great work! and thanks for the nice post and code. I am currently working on a classifier and would like to use your feature set for experimentation and testing. I kindly request for simple features (200) and class labels file, for both train and test set. It would be great if I have these two files, as I dont have the environment/facility to use your python code on github.\n\nThank you in advance.\nRegards\nRajesh\n" }, { "id": 445580, "postDate": "2018-12-26T19:10:14.920Z", "content": "Congratulations and thank you for detailed explanation!
", "rawMarkdown": "Congratulations and thank you for detailed explanation!" }, { "id": 445328, "postDate": "2018-12-26T08:15:21.013Z", "content": "Congrats, Kyle !
", "rawMarkdown": "Congrats, Kyle !" }, { "id": 445209, "postDate": "2018-12-26T00:20:15.253Z", "content": "congrats
", "rawMarkdown": "congrats" }, { "id": 444751, "postDate": "2018-12-24T18:14:32.983Z", "content": "Gratz, great approach)
", "rawMarkdown": "Gratz, great approach)" }, { "id": 444565, "postDate": "2018-12-24T09:36:53.887Z", "content": "Awesome work.
", "rawMarkdown": "Awesome work. " }, { "id": 444557, "postDate": "2018-12-24T09:03:51.507Z", "content": "Very nice solution Kyle Boone!! Congratulations!
", "rawMarkdown": "Very nice solution Kyle Boone!! Congratulations!" }, { "id": 444501, "postDate": "2018-12-24T06:26:08.353Z", "content": "Congrats! I love simple solution. Interesting to see how you leverage your domain expertise in physics (cosmology to be specific?) and apply to the understand the problem (data part of ML). Then, you focus on doing that one thing well. And keep believing that ensembling is the last option. A recipe of success. Brilliant and inspiring.
", "rawMarkdown": "Congrats! I love simple solution. Interesting to see how you leverage your domain expertise in physics (cosmology to be specific?) and apply to the understand the problem (data part of ML). Then, you focus on doing that one thing well. And keep believing that ensembling is the last option. A recipe of success. Brilliant and inspiring." }, { "id": 444425, "postDate": "2018-12-24T02:33:21.650Z", "content": "Brilliant. Thanks for sharing, really helps alot.
", "rawMarkdown": "Brilliant. Thanks for sharing, really helps alot." }, { "id": 444416, "postDate": "2018-12-24T02:21:32.557Z", "content": "Congratulations !
\n\nThanks for the explanations!
", "rawMarkdown": "Congratulations !\n\nThanks for the explanations!" }, { "id": 444338, "postDate": "2018-12-23T20:20:07.560Z", "content": "congrats
", "rawMarkdown": "congrats" }, { "id": 443835, "postDate": "2018-12-22T14:23:29.970Z", "content": "great
", "rawMarkdown": "great" }, { "id": 443300, "postDate": "2018-12-21T11:44:09.990Z", "content": "Congrats!
\n\nWas really nice to hear about the data augmentation method you used! Also I found particularly interesting how you fitted the redshift to align with the test set redshifts and adjusted the fluxes according to the relativistic effects you described.
", "rawMarkdown": "Congrats! \n\nWas really nice to hear about the data augmentation method you used! Also I found particularly interesting how you fitted the redshift to align with the test set redshifts and adjusted the fluxes according to the relativistic effects you described." }, { "id": 443182, "postDate": "2018-12-21T07:00:54.540Z", "content": "Congratulations!\nIt's a Amazing
", "rawMarkdown": "Congratulations!\nIt's a Amazing" }, { "id": 443051, "postDate": "2018-12-20T23:57:42.303Z", "content": "Congratulations!\nI could not think of this solution.
", "rawMarkdown": "Congratulations!\nI could not think of this solution." }, { "id": 442598, "postDate": "2018-12-20T07:15:13.950Z", "content": "Congrats Kyle..
", "rawMarkdown": "Congrats Kyle.." }, { "id": 442507, "postDate": "2018-12-20T03:26:18.773Z", "content": "Congratulations Kyle ..Amazing writeup ..
", "rawMarkdown": "Congratulations Kyle ..Amazing writeup .." }, { "id": 442154, "postDate": "2018-12-19T14:46:56.160Z", "content": "Congratulations and thank you for sharing your solution!
", "rawMarkdown": "Congratulations and thank you for sharing your solution!" }, { "id": 441653, "postDate": "2018-12-18T22:33:26.950Z", "content": "Congratulations on a stunning first place! Thanks for sharing this and thanks for your comments in the discussions along the way.
", "rawMarkdown": "Congratulations on a stunning first place! Thanks for sharing this and thanks for your comments in the discussions along the way." }, { "id": 441647, "postDate": "2018-12-18T22:16:47.737Z", "content": "Congrats! What a cool solution!
", "rawMarkdown": "Congrats! What a cool solution!" }, { "id": 441618, "postDate": "2018-12-18T21:00:20.137Z", "content": "Congratulations! Brilliant solution!
", "rawMarkdown": "Congratulations! Brilliant solution!" }, { "id": 441368, "postDate": "2018-12-18T15:34:49.377Z", "content": "Congrats !
", "rawMarkdown": "Congrats !" }, { "id": 441328, "postDate": "2018-12-18T14:50:21.093Z", "content": "Congratulations @kyleboone and thanks for sharing your solution. Lot to learn from it.
", "rawMarkdown": "Congratulations @kyleboone and thanks for sharing your solution. Lot to learn from it." }, { "id": 441319, "postDate": "2018-12-18T14:42:44.847Z", "content": "Congrats and thanks for sharing the solution...
", "rawMarkdown": "Congrats and thanks for sharing the solution..." }, { "id": 441259, "postDate": "2018-12-18T13:35:55.650Z", "content": "Congratulations and thank you for sharing your solution!
", "rawMarkdown": "Congratulations and thank you for sharing your solution! " }, { "id": 441242, "postDate": "2018-12-18T12:59:46.307Z", "content": "Ming blowing solution @kyleboone, you have focused more on data part rather than modeling part.
\n\nGenerally 1st place team have large ensembles of 2-3 layers of NNs & boosting algorithms & some linear model as meta model. But you haven't done that still you managed to be 1st! That's outstanding..
", "rawMarkdown": "Ming blowing solution @kyleboone, you have focused more on data part rather than modeling part.\n\nGenerally 1st place team have large ensembles of 2-3 layers of NNs & boosting algorithms & some linear model as meta model. But you haven't done that still you managed to be 1st! That's outstanding.. ", "replies": [ { "id": 441392, "postDate": "2018-12-18T15:59:26.500Z", "content": "I'm definitely less experienced with ML than a lot of people on here, so I decided to focus on doing one thing well. I'm very interested to see how well we can do if we blend everyone's models together.
", "rawMarkdown": "I'm definitely less experienced with ML than a lot of people on here, so I decided to focus on doing one thing well. I'm very interested to see how well we can do if we blend everyone's models together.", "votes": 2 }, { "id": 441473, "postDate": "2018-12-18T17:30:14.777Z", "content": "If we blend everyone's model together & do stacking of 2-3 layers, with your features only, then also score can be close to 0.65 or 0.64, may be 0.60. \nFeature engineering is the key.
", "rawMarkdown": "If we blend everyone's model together & do stacking of 2-3 layers, **with your features only**, then also score can be close to 0.65 or 0.64, may be 0.60. \nFeature engineering is the key." }, { "id": 441558, "postDate": "2018-12-18T19:37:20.803Z", "content": "I'll release my code along with the augmented training set and GP features after I finish cleaning it up. It will be very interesting to see how the different models complement each other.
", "rawMarkdown": "I'll release my code along with the augmented training set and GP features after I finish cleaning it up. It will be very interesting to see how the different models complement each other.", "votes": 2 } ] }, { "id": 441203, "postDate": "2018-12-18T11:59:30.130Z", "content": "Great Kyle. Congratulations and Well deserved First place solution.
", "rawMarkdown": "Great Kyle. Congratulations and Well deserved First place solution. " }, { "id": 441195, "postDate": "2018-12-18T11:49:13.913Z", "content": "Big congrats Kyle! Very impressive solution, specially the GP, augmentation and class 99 probing.
", "rawMarkdown": "Big congrats Kyle! Very impressive solution, specially the GP, augmentation and class 99 probing." }, { "id": 441139, "postDate": "2018-12-18T10:08:53.210Z", "content": "Congrats for the great work, and the highly deserved win! Admirable performance! It would be very interesting to see how your data-oriented method could gain from other solutions, which look more ml-oriented.
", "rawMarkdown": "Congrats for the great work, and the highly deserved win! Admirable performance! It would be very interesting to see how your data-oriented method could gain from other solutions, which look more ml-oriented.\n\n" }, { "id": 441118, "postDate": "2018-12-18T09:28:03.140Z", "content": "Congrats!! My GP fitting is still running... Can you explain more about the two dimensional GP? The passband in this case will be continuous or discrete?
", "rawMarkdown": "Congrats!! My GP fitting is still running... Can you explain more about the two dimensional GP? The passband in this case will be continuous or discrete?", "replies": [ { "id": 441390, "postDate": "2018-12-18T15:57:19.227Z", "content": "I used a continuous GP for the passband, just using the passband labels as the coordinates in the second dimension.
", "rawMarkdown": "I used a continuous GP for the passband, just using the passband labels as the coordinates in the second dimension.", "votes": 1 }, { "id": 441426, "postDate": "2018-12-18T16:47:16.687Z", "content": "Thanks for your answer. When you augment train set by degrading lightcurves, do you do GP again based on the degraded lightcurve? Or you just add noise to some of your features?
", "rawMarkdown": "Thanks for your answer. When you augment train set by degrading lightcurves, do you do GP again based on the degraded lightcurve? Or you just add noise to some of your features?" } ] }, { "id": 441095, "postDate": "2018-12-18T08:58:08.460Z", "content": "Thanks for sharing!
\n\nI also augmented train set, but doesn’t work. More specific, I augmented ddf object by degrading 200 times to convert ddf into wfd, coz I thought test set has much more wfd. Each wfd has about 120 samples, and each ddf has about 320 samples.
\n\nI have not tested yet, but I believe I degraded too much.
\n\nAnyway, good for you 1st place! Well deserved!
", "rawMarkdown": "Thanks for sharing!\n\nI also augmented train set, but doesn’t work. More specific, I augmented ddf object by degrading 200 times to convert ddf into wfd, coz I thought test set has much more wfd. Each wfd has about 120 samples, and each ddf has about 320 samples.\n\nI have not tested yet, but I believe I degraded too much.\n\nAnyway, good for you 1st place! Well deserved!", "replies": [ { "id": 441387, "postDate": "2018-12-18T15:55:17.380Z", "content": "Throwing out observations to get lightcurves that look more like the WFD was essential to my degradation. I came up with a model of how many lightcurve points there are for a typical WFD object, and I threw out data to get to that.
", "rawMarkdown": "Throwing out observations to get lightcurves that look more like the WFD was essential to my degradation. I came up with a model of how many lightcurve points there are for a typical WFD object, and I threw out data to get to that." } ] }, { "id": 441065, "postDate": "2018-12-18T08:20:41.173Z", "content": "Congratulations Kyle!! That's amazing work! \nI was so looking forward to this post... I had no idea how to treat this difference distributions between train and test and I don't think I would have come up with the GP idea.
\n\nThank you for sharing and for all the discussions!
", "rawMarkdown": " Congratulations Kyle!! That's amazing work! \nI was so looking forward to this post... I had no idea how to treat this difference distributions between train and test and I don't think I would have come up with the GP idea.\n\nThank you for sharing and for all the discussions!" }, { "id": 441064, "postDate": "2018-12-18T08:20:02.623Z", "content": "Thank you Kyle !\nIt was definitely the most interesting competition I participated in.\nI would like to give a special thank to the active kagglers who have shared a lot of ideas and good mood!
", "rawMarkdown": "Thank you Kyle !\nIt was definitely the most interesting competition I participated in.\nI would like to give a special thank to the active kagglers who have shared a lot of ideas and good mood!" }, { "id": 441041, "postDate": "2018-12-18T07:47:09.910Z", "content": "Congratulations !
\n\nThanks for the explanations too.
", "rawMarkdown": "Congratulations !\n\nThanks for the explanations too." }, { "id": 440950, "postDate": "2018-12-18T05:45:38.430Z", "content": "Congrats! Gaussian process gave me a headache at the stats class. I really need to pick it up again. :D
", "rawMarkdown": "Congrats! Gaussian process gave me a headache at the stats class. I really need to pick it up again. :D" }, { "id": 440887, "postDate": "2018-12-18T04:09:36.503Z", "content": "Congrats. Thanks for sharing that fast.
\n\nI wish I had started earlier on gaussian process modeling. That's the code I finished the last day of the competition, and I had too few submisssions to start using it properly. I also used Matern kernel :) I guess you used celerite, right?
\n\nYour data augmentation/degradation is awesome. I vaguely thought it could be done, but not to base the whole solution on it.
\n\nI am surprised by how much you gained by probing for class 99. We haven't done it at all.
", "rawMarkdown": "Congrats. Thanks for sharing that fast.\n\nI wish I had started earlier on gaussian process modeling. That's the code I finished the last day of the competition, and I had too few submisssions to start using it properly. I also used Matern kernel :) I guess you used celerite, right?\n\nYour data augmentation/degradation is awesome. I vaguely thought it could be done, but not to base the whole solution on it.\n\nI am surprised by how much you gained by probing for class 99. We haven't done it at all. ", "replies": [ { "id": 440891, "postDate": "2018-12-18T04:19:23.790Z", "content": "I actually used george instead of celerite because I have used that in the past. celerite is a newer package by the same author, and is presumably better although I haven't tried it myself.
\n\nI had the idea for the data augmentation and it coded up all in one night. When I finished it, I immediately jumped from 1.0X on the leaderboard to 0.8X and was very excited. Before that point I thought that I had no chance in this competition! I'll make sure to share the augmented training set that I produced with everyone. I'm sure that your ML models are much much better than mine, and I think that the augmented training set was what won me the competition.
\n\nThanks for all of the posts that you made on the discussion board. They were very useful to me, and helped guide me along in the competition.
", "rawMarkdown": "I actually used george instead of celerite because I have used that in the past. celerite is a newer package by the same author, and is presumably better although I haven't tried it myself.\n\nI had the idea for the data augmentation and it coded up all in one night. When I finished it, I immediately jumped from 1.0X on the leaderboard to 0.8X and was very excited. Before that point I thought that I had no chance in this competition! I'll make sure to share the augmented training set that I produced with everyone. I'm sure that your ML models are much much better than mine, and I think that the augmented training set was what won me the competition.\n\nThanks for all of the posts that you made on the discussion board. They were very useful to me, and helped guide me along in the competition.", "votes": 8 }, { "id": 440895, "postDate": "2018-12-18T04:28:38.440Z", "content": "Also, I think that what happened with class 99 was very unfortunate. If there is a PLAsTiCC 2.0, the private test should have different kinds of class 99 objects than the public one so that you can't cheat by probing the leaderboard. I only discovered the probing method in the last week. I contacted the Kaggle staff, and I was advised not to share it because it would really have messed up the final placements.
\n\nIt would be very interesting if the contest organizers could recalculate our scores with class 99 left out!
", "rawMarkdown": "Also, I think that what happened with class 99 was very unfortunate. If there is a PLAsTiCC 2.0, the private test should have different kinds of class 99 objects than the public one so that you can't cheat by probing the leaderboard. I only discovered the probing method in the last week. I contacted the Kaggle staff, and I was advised not to share it because it would really have messed up the final placements.\n\nIt would be very interesting if the contest organizers could recalculate our scores with class 99 left out!", "votes": 5 }, { "id": 441054, "postDate": "2018-12-18T08:12:41.620Z", "content": "Right, a different class 99 between public and private would have been much safer to check models ability to predict it. Not sure organizers will get a reliable assessment of how effective solutions are. We all did some probing (some more than others ;) ) to calibrate it.
", "rawMarkdown": "Right, a different class 99 between public and private would have been much safer to check models ability to predict it. Not sure organizers will get a reliable assessment of how effective solutions are. We all did some probing (some more than others ;) ) to calibrate it." }, { "id": 441253, "postDate": "2018-12-18T13:28:39.197Z", "content": "Another question: you said that it helped you to see my confusion matrix. I was surprised, as I didn't think it could help others. How did it help you?
", "rawMarkdown": "Another question: you said that it helped you to see my confusion matrix. I was surprised, as I didn't think it could help others. How did it help you? " } ] }, { "id": 440862, "postDate": "2018-12-18T03:16:12.940Z", "content": "Wow, that's impressive !! Now I understand better why my score is so far from yours ;-)
", "rawMarkdown": "Wow, that's impressive !! Now I understand better why my score is so far from yours ;-)" }, { "id": 440860, "postDate": "2018-12-18T03:13:24.570Z", "content": "Congratulations Kyle!\nThis is very interesting! You managed to pull off exactly what we have been trying and failing! It's very insightful hearing how you managed to apply Gaussian Process for feature extraction!
\n\nWe also tried GP, but are unable to properly create a model that combines multiple passbands while dealing with missingness. As a result, the GP model is not really able to cope with low sample rate objects without significant over/underfitting.
\n\nWe did the downsampling of DDF objects during the last phase of the competition and it really helped greatly. We also had this idea to combine GP and downsampling, using GP to fit DDF objects in the training set then downsample them to non-DDF resolution using a random observation timestamps from an arbitrary object. We had issues separating bad generated examples from informative ones, mainly due to the inability to replicate accurate observation and 'detected' status estimates for the resampled points. However, I still believe this approach should work well if done properly, and can potentially lead to a model with very good uncertainty estimates without the need to fit Gaussian Processes on every object during inference. The idea is to build a model P(class|observed) by creating more realistic 'observed' objects rather than building a model P(class|full_curve)P(full_curve|observed) by trying to reconstruct every light curve. If anyone is interested I would like to develop this idea further and see if it can help the astronomers in performance-critical situations.
\n\nBTW, how can I learn about Gaussian Process that deals with missingness and/or 'observed' measurement errors? Would appreciate it if anyone can point the way!
", "rawMarkdown": "Congratulations Kyle!\nThis is very interesting! You managed to pull off exactly what we have been trying and failing! It's very insightful hearing how you managed to apply Gaussian Process for feature extraction!\n\nWe also tried GP, but are unable to properly create a model that combines multiple passbands while dealing with missingness. As a result, the GP model is not really able to cope with low sample rate objects without significant over/underfitting.\n\nWe did the downsampling of DDF objects during the last phase of the competition and it really helped greatly. We also had this idea to combine GP and downsampling, using GP to fit DDF objects in the training set then downsample them to non-DDF resolution using a random observation timestamps from an arbitrary object. We had issues separating bad generated examples from informative ones, mainly due to the inability to replicate accurate observation and 'detected' status estimates for the resampled points. However, I still believe this approach should work well if done properly, and can potentially lead to a model with very good uncertainty estimates without the need to fit Gaussian Processes on every object during inference. The idea is to build a model P(class|observed) by creating more realistic 'observed' objects rather than building a model P(class|full_curve)P(full_curve|observed) by trying to reconstruct every light curve. If anyone is interested I would like to develop this idea further and see if it can help the astronomers in performance-critical situations.\n\nBTW, how can I learn about Gaussian Process that deals with missingness and/or 'observed' measurement errors? Would appreciate it if anyone can point the way!", "replies": [ { "id": 440882, "postDate": "2018-12-18T04:03:15.780Z", "content": "I used a 2-dimensional Gaussian process where one dimension is the time and the other is the passband. Both dimensions used a Matern kernel. I fixed the passband kernel length scale to be a fixed, relatively large value since the fits for it weren't great. The GP definitely messes up if it doesn't have all of the information, eg this GP for a SN Ia where the rise time should be < 50 days:
\n\n![\"bad](\"https://i.imgur.com/NYbb8bo.png\")
It would be nice if the GP could be trained to match a SN Ia more accurately, but that is left for future work. Instead, I dealt with this in two ways. First, I added features to my LGBM model that let it figure out when the GP fit was bad. In this case, there are several features counting how many observations there were before maximum, so it knows not to trust the rise time. Second, I augmented the training set by throwing out blocks of observations. That created lots of lightcurves like this for the LGBM model to train on.
\n\nI did not use the Gaussian Process predictions for generating new objects. I thought about doing that, but decided that the errors on the GP model were too large, and I would end up generating non-realistic models. Instead, I simply noised up, scaled and threw out points from the training set spectra.
", "rawMarkdown": "I used a 2-dimensional Gaussian process where one dimension is the time and the other is the passband. Both dimensions used a Matern kernel. I fixed the passband kernel length scale to be a fixed, relatively large value since the fits for it weren't great. The GP definitely messes up if it doesn't have all of the information, eg this GP for a SN Ia where the rise time should be < 50 days:\n\n\n\nIt would be nice if the GP could be trained to match a SN Ia more accurately, but that is left for future work. Instead, I dealt with this in two ways. First, I added features to my LGBM model that let it figure out when the GP fit was bad. In this case, there are several features counting how many observations there were before maximum, so it knows not to trust the rise time. Second, I augmented the training set by throwing out blocks of observations. That created lots of lightcurves like this for the LGBM model to train on.\n\nI did not use the Gaussian Process predictions for generating new objects. I thought about doing that, but decided that the errors on the GP model were too large, and I would end up generating non-realistic models. Instead, I simply noised up, scaled and threw out points from the training set spectra.", "votes": 4 }, { "id": 440897, "postDate": "2018-12-18T04:29:45.223Z", "content": "Yeah, that is an excellent example of where I think GP could be improved. It only works great if the SN curve contains a peak, and okay-ish if the ubobserved cutoff point is not in the middle of the rising/falling phase. I tried using an autoencoder to combat this, but autoencoder has an opposite issue: it is more conservative about where the missing values should be, but too much so that all interpolated portions of light curves look alike even if the observed portion is only remotely similar.
\n\nI think the major weakness of GP is that it only ever learns from one object, so it cannot infer how the missing portion of the data should look like based on what a typical object in this class looks like. This is actually the original reason I wanted to do GP downsampling - the classifier should know that a perfectly looking DDF sample could end up looking like something very different if captured in the non-DDF portion, so it should not be overconfident when seeing a partial curve. However, since GP itself suffers from this issue, I am unable to use GP to remedy it... I guess the resampling trick would be really useful if you had a dataset of more complete light curves like those from some of the earlier researches.
", "rawMarkdown": "Yeah, that is an excellent example of where I think GP could be improved. It only works great if the SN curve contains a peak, and okay-ish if the ubobserved cutoff point is not in the middle of the rising/falling phase. I tried using an autoencoder to combat this, but autoencoder has an opposite issue: it is more conservative about where the missing values should be, but too much so that all interpolated portions of light curves look alike even if the observed portion is only remotely similar.\n\nI think the major weakness of GP is that it only ever learns from one object, so it cannot infer how the missing portion of the data should look like based on what a typical object in this class looks like. This is actually the original reason I wanted to do GP downsampling - the classifier should know that a perfectly looking DDF sample could end up looking like something very different if captured in the non-DDF portion, so it should not be overconfident when seeing a partial curve. However, since GP itself suffers from this issue, I am unable to use GP to remedy it... I guess the resampling trick would be really useful if you had a dataset of more complete light curves like those from some of the earlier researches.", "votes": 1 }, { "id": 441143, "postDate": "2018-12-18T10:12:57.360Z", "content": "Hi Guys, Congratulations on great work, and awesome job Kyle. I did not compete but wanted to pitch in a couple of connections that may help moving forward:
\n\nGaussian Processes go by the name \"Kriging\" in the geospatial statistics literature, and there is a lot of theory and software out there. For example, SAS Proc Mixed can fit the models, including Matern and simpler kernels.
Using likelihood-based methods one can fit data from multiple objects at once, thereby borrowing strength from each other and shrinking uncertain curves towards a centrally-estimated one.
Wondering if it might be possible to devise something like a Matern-kernel layer to be used in NN architectures. With regularization this would have similarities to #2.
\n\n\nUsing likelihood-based methods one can fit data from multiple objects at once
\n
I am a beginner here, and I was wondering about this precisely, as fitting all light curves for a given source at once seems quite logical to me. Your remark makes me think about how I could have fit on several curves at once. I guess I could have concatenated the gradient and mean function for all curves... Let me try. But if you have great resources about it then I'd be interested in seeing them.
", "rawMarkdown": "> Using likelihood-based methods one can fit data from multiple objects at once\n\nI am a beginner here, and I was wondering about this precisely, as fitting all light curves for a given source at once seems quite logical to me. Your remark makes me think about how I could have fit on several curves at once. I guess I could have concatenated the gradient and mean function for all curves... Let me try. But if you have great resources about it then I'd be interested in seeing them." }, { "id": 441497, "postDate": "2018-12-18T18:01:27.590Z", "content": "If you have access to SAS/STAT, then I think it should be relatively straightforward to set up. The input data needs to be stacked into a single very tall 2D array, with columns for flux, curve id, x-coord, y-coord (typically rescaled for numerical stability). Forgive me, I'm unfamiliar with these data, but it looks like there may also be higher-level ways to group the curves. If so, the model can be expanded to estimate additional covariance parameters; for example, the six curves shown above by Kyle might be a single group. This is handled by adding more columns in the array indicating group levels. There are then around 5-6 lines of Proc Mixed syntax to set up the model, and it subsequently does the optimization using Newton-Raphson. A BY statement can optionally be used to perform completely separate fits.
\n\nBeyond this, I would guess you could make good headway using any decent kriging package. Was just glancing at the PyKriging web page--have never used it but it appears capable.
\n\nP.S. The same kind of model is often used in Bayesian Optimization of hyperparameters.
", "rawMarkdown": "If you have access to SAS/STAT, then I think it should be relatively straightforward to set up. The input data needs to be stacked into a single very tall 2D array, with columns for flux, curve id, x-coord, y-coord (typically rescaled for numerical stability). Forgive me, I'm unfamiliar with these data, but it looks like there may also be higher-level ways to group the curves. If so, the model can be expanded to estimate additional covariance parameters; for example, the six curves shown above by Kyle might be a single group. This is handled by adding more columns in the array indicating group levels. There are then around 5-6 lines of Proc Mixed syntax to set up the model, and it subsequently does the optimization using Newton-Raphson. A BY statement can optionally be used to perform completely separate fits.\n\nBeyond this, I would guess you could make good headway using any decent kriging package. Was just glancing at the PyKriging web page--have never used it but it appears capable.\n\nP.S. The same kind of model is often used in Bayesian Optimization of hyperparameters.", "votes": 1 }, { "id": 442123, "postDate": "2018-12-19T14:08:24.860Z", "content": "Russ, thanks. I don' t have access to SAS as I work for a company that competes with it. But I'll have a look at Kriging packages.
\n\nMay I ask why you didn't enter this competition given you seem to know quite a bit here, It may have resulted in our rank being one higher but interesting for all still ;)
", "rawMarkdown": "Russ, thanks. I don' t have access to SAS as I work for a company that competes with it. But I'll have a look at Kriging packages. \n\nMay I ask why you didn't enter this competition given you seem to know quite a bit here, It may have resulted in our rank being one higher but interesting for all still ;)" } ] }, { "id": 440854, "postDate": "2018-12-18T03:05:18.740Z", "content": "Brilliant stuff! How did you figure out which parameters to degrade in the train set? Was there any sort of algorithmic process or was it more investigative trial-and-error?
", "rawMarkdown": "Brilliant stuff! How did you figure out which parameters to degrade in the train set? Was there any sort of algorithmic process or was it more investigative trial-and-error? ", "replies": [ { "id": 440873, "postDate": "2018-12-18T03:50:03.800Z", "content": "I basically made histograms of everything that you could measure in the test and validation sets, and added degradations based off of that.
\n\nFor example, I found that objects that have speczs (like the training set) tend to be at lower redshifts than objects in the test set. Looking at the log(redshift), there is a shift of about 0.2 between the two:
\n\n![\"specz](\"https://i.imgur.com/xOOxGfa.png\")
To pick redshifts for my new sample, I added the shift along with a gaussian with a spread of 0.2 in log(redshift). That makes the new training set more representative of the test set. I did something similar for other variables, and I'll discuss that in my full writeup.
", "rawMarkdown": "I basically made histograms of everything that you could measure in the test and validation sets, and added degradations based off of that.\n\nFor example, I found that objects that have speczs (like the training set) tend to be at lower redshifts than objects in the test set. Looking at the log(redshift), there is a shift of about 0.2 between the two:\n\n![specz comparison][1]\n\nTo pick redshifts for my new sample, I added the shift along with a gaussian with a spread of 0.2 in log(redshift). That makes the new training set more representative of the test set. I did something similar for other variables, and I'll discuss that in my full writeup.\n\n[1]: https://i.imgur.com/xOOxGfa.png", "votes": 7 }, { "id": 440878, "postDate": "2018-12-18T03:54:53.180Z", "content": "Intuitively, since we do have more data on lower z objects it does make sense that more of those have spectroscopic estimates. And that's a very clean process that you just outlined, very very nicely done!
\n\nP.S. Though it probably doesn't matter, I have never used log(z) in my research. We typically only use log representations for quantities which can be described by a power law (Ngals, X--ray luminosity etc). But as far as a statistical comparison of 2 distributions goes, I guess it works fine!
", "rawMarkdown": "Intuitively, since we do have more data on lower z objects it does make sense that more of those have spectroscopic estimates. And that's a very clean process that you just outlined, very very nicely done! \n\nP.S. Though it probably doesn't matter, I have never used log(z) in my research. We typically only use log representations for quantities which can be described by a power law (Ngals, X--ray luminosity etc). But as far as a statistical comparison of 2 distributions goes, I guess it works fine! " }, { "id": 440884, "postDate": "2018-12-18T04:07:39.320Z", "content": "I used log(z) because redshift is always greater than 0. The log transformation preserves that.
\n\nWe have more spectra of low redshift objects because they are brighter. With a big telescope, you can measure the redshift of a Type Ia supernova at redshift 0.1 in seconds, but it will take hours to measure the redshift of one at redshift 1. That naturally leads to a bias in which objects we end up having redshifts for.
", "rawMarkdown": "I used log(z) because redshift is always greater than 0. The log transformation preserves that.\n\nWe have more spectra of low redshift objects because they are brighter. With a big telescope, you can measure the redshift of a Type Ia supernova at redshift 0.1 in seconds, but it will take hours to measure the redshift of one at redshift 1. That naturally leads to a bias in which objects we end up having redshifts for.", "votes": 1 }, { "id": 440888, "postDate": "2018-12-18T04:09:51.777Z", "content": "Makes sense!
", "rawMarkdown": "Makes sense! " }, { "id": 442367, "postDate": "2018-12-19T21:39:09.163Z", "content": "@kyleboone, nice trick with z augmentation! But I think if you change z, you should do appropriate change in flux. Have you done corresponding flux correction in your augmentation? I'm quite interested how to do it correctly.
", "rawMarkdown": "@kyleboone, nice trick with z augmentation! But I think if you change z, you should do appropriate change in flux. Have you done corresponding flux correction in your augmentation? I'm quite interested how to do it correctly." }, { "id": 442378, "postDate": "2018-12-19T21:56:15.417Z", "content": "When I change the redshift, I adjust the flux to account for the change in distance (the general relativistic version of 1/r^2) and for the time dilation of photons. This is actually quite straightforward: the \"distance modulus\" is the combination of all of these effects. You just need to apply a correction of 10**(0.4*(old distmod - new distmod)) to your observations.
\n\nYou should also take into account the fact that the redshift shifts what part of the spectrum the bands fall in. I did not incorporate that into my solution because I didn't have time to evaluate how GP modeling errors would introduce biases into the data. In principle, it should be pretty straightforward to do with the output of my GP.
\n\nI also adjusted the observed times to stretch out the lightcurve. eg: at redshift 1 a supernova takes twice as long to decline as at redshift 0.
", "rawMarkdown": "When I change the redshift, I adjust the flux to account for the change in distance (the general relativistic version of 1/r^2) and for the time dilation of photons. This is actually quite straightforward: the \"distance modulus\" is the combination of all of these effects. You just need to apply a correction of 10\\*\\*(0.4\\*(old distmod - new distmod)) to your observations.\n\nYou should also take into account the fact that the redshift shifts what part of the spectrum the bands fall in. I did not incorporate that into my solution because I didn't have time to evaluate how GP modeling errors would introduce biases into the data. In principle, it should be pretty straightforward to do with the output of my GP.\n\nI also adjusted the observed times to stretch out the lightcurve. eg: at redshift 1 a supernova takes twice as long to decline as at redshift 0.", "votes": 2 }, { "id": 442393, "postDate": "2018-12-19T22:45:27.597Z", "content": "Kyle, thank you for your answer! Did you mean 10**(0.4(old distmod - new distmod))? But this correction is multiplicative, so it's not clear (at least for me)) what to do with negative flux?
\n\nIt would be straightforward if we had real flux, not difference with background. For example, if new distmod is greater than old, we have correction<1, absolute value of flux will decrease. Decreased value of positive flux corresponds to less bright object (which is true), but decrease of negative flux corresponds to more bright object (which is false).
", "rawMarkdown": "Kyle, thank you for your answer! Did you mean 10**(0.4(old distmod - new distmod))? But this correction is multiplicative, so it's not clear (at least for me)) what to do with negative flux? \n\nIt would be straightforward if we had real flux, not difference with background. For example, if new distmod is greater than old, we have correction<1, absolute value of flux will decrease. Decreased value of positive flux corresponds to less bright object (which is true), but decrease of negative flux corresponds to more bright object (which is false)." }, { "id": 442438, "postDate": "2018-12-20T00:52:35.857Z", "content": "Yes, the markdown formatter ate my asterisks. I fixed it now.
\n\nGetting the background right doesn't really matter for the scaling. After you take the exponential above (turning from magnitudes into a scaling of the flux), you end up with a positive number that you are multiplying everything by. If I move an object farther away, I end up with a scale that is less than one, so a large positive number gets transformed into a small positive number, and a large negative number gets transformed into a small negative number. Subtracting the background before or after gives the same result, regardless of what method you use.
", "rawMarkdown": "Yes, the markdown formatter ate my asterisks. I fixed it now.\n\nGetting the background right doesn't really matter for the scaling. After you take the exponential above (turning from magnitudes into a scaling of the flux), you end up with a positive number that you are multiplying everything by. If I move an object farther away, I end up with a scale that is less than one, so a large positive number gets transformed into a small positive number, and a large negative number gets transformed into a small negative number. Subtracting the background before or after gives the same result, regardless of what method you use." }, { "id": 442731, "postDate": "2018-12-20T11:56:17.823Z", "content": "Hmm, (flux-background) * correction and (flux * correction-background) look like different operations for me, am I missing something?) When you move an object farther away, background stays the same, right?
\n\nAs for negative numbers, I mean that when moving object farther away, all flux values should be shifted in negative direction, but making negative number smaller shifts it in positive direction.
\n\nAnd I thought that real flux before background subtraction is always positive, number of received photons coudn't be negative :)
", "rawMarkdown": "Hmm, (flux-background) * correction and (flux * correction-background) look like different operations for me, am I missing something?) When you move an object farther away, background stays the same, right?\n\nAs for negative numbers, I mean that when moving object farther away, all flux values should be shifted in negative direction, but making negative number smaller shifts it in positive direction. \n\nAnd I thought that real flux before background subtraction is always positive, number of received photons coudn't be negative :) " } ] }, { "id": 2354208, "postDate": "2023-07-22T09:39:21.017Z", "content": "Congratulations Kyle !
", "rawMarkdown": "Congratulations Kyle ! ", "isDeleted": true }, { "id": 444935, "postDate": "2018-12-25T07:19:08.093Z", "content": "Thanks for the detailed explanation :)
", "rawMarkdown": "Thanks for the detailed explanation :)", "votes": -1 }, { "id": 448152, "postDate": "2018-12-31T10:59:40.117Z", "content": "Congrats! And thank you for sharing.
", "rawMarkdown": "Congrats! And thank you for sharing." }, { "id": 446173, "postDate": "2018-12-27T16:40:20.947Z", "content": "Thanks for all this info! Very Useful
", "rawMarkdown": "Thanks for all this info! Very Useful" }, { "id": 444413, "postDate": "2018-12-24T02:06:27.107Z", "content": "This is perfect! Thank you for sharing!
", "rawMarkdown": "This is perfect! Thank you for sharing! " }, { "id": 441676, "postDate": "2018-12-18T23:25:43.040Z", "content": "Congraturations! Thank you for sharing.
", "rawMarkdown": "Congraturations! Thank you for sharing. " } ], "comments": [ { "id": 445181, "author_name": "Kyle Boone", "author_url": "", "post_date": "2018-12-25T20:58:15.600000", "content": "The code is now available on my github page at https://github.com/kboone/plasticc
", "votes": 6, "replies": [] }, { "id": 441007, "author_name": "olivier", "author_url": "", "post_date": "2018-12-18T07:13:19.880000", "content": "Congratulations Kyle ! Awesome work.
\n\nAfter reading your write up I believe I now understand the dataset better ;-)
", "votes": 3, "replies": [] }, { "id": 440856, "author_name": "khyeh", "author_url": "", "post_date": "2018-12-18T03:08:19.143000", "content": "Thanks for sharing! It's amazing! @cpmpml is indeed a discussion GM...\nIf we had 270000 objects, things would be very much different for all of us, lol\nWe did some augmentation in training, but far from your level...
\n\nCongratulation!
", "votes": 4, "replies": [] }, { "id": 443230, "author_name": "Ilhan Aytutuldu", "author_url": "", "post_date": "2018-12-21T08:55:18.790000", "content": "Congrats Kyle Boone, thank you for detailed explanation.
", "votes": 1, "replies": [] }, { "id": 442329, "author_name": "mamas", "author_url": "", "post_date": "2018-12-19T20:00:54.677000", "content": "Congrats Kyle Boone, you are the real super man in this competition :)\nI hope you continue kaggle and if possible I would like to compete with you again!
", "votes": 1, "replies": [] }, { "id": 441202, "author_name": "Manu Gangler", "author_url": "", "post_date": "2018-12-18T11:59:12.083000", "content": "Congratulations Kyle ! I'm so happy so see you win the race. This is also a good lesson that domain knowledge is useful to design the solution, and I'm pretty confident that mixing the solutions of the top rankers can still improve a lot the solution. In other words, you made a clear case for a collaborative approach to the solution...
\n\nRegarding the GP, I think the way data are extrapolated is one of the key issues of this challenge. Have you considered at some point to build templates for the lightcuves ? This was the approach I was trying, as it alleviates the problem of interpolation over long range of missing data by using the template. Of course, interpolation is then dependent on the hypothesis on the class. See for instance what this approach gives below. It probably beats what GP can do (erro band represent class variability, not object interpolation uncertainty)
\n\n
Hi Manu, good to hear from you. It's been a while! I definitely think that l could mix my solution with other teams' solutions and get a boost. Since I had a lot to learn, I decided to focus on one thing and try to get it right. It seems like in the end lgbm models outperformed everything else by quite a bit, so I'm not sure how much of a gain I'd get.
\n\nWhat did you use to build your models? I thought about doing that, but decided that I didn't have the computational resources or time to do fits to everything. I'm guessing that you did something like GP interpolation of well sampled timeseries + factor analysis?
", "votes": 0, "replies": [] }, { "id": 442002, "author_name": "Manu Gangler", "author_url": "", "post_date": "2018-12-19T10:47:08.590000", "content": "You are right about this: fitting a model per light-curve takes time to understand the data, plus time to converge the model, so you got the right hint about going directly to some more generic method, while I'm still struggling with doing the fits ;-) . I was first looking at GP, then switched to kernel methods, then getting back to good old splines to fit the ensemble of time series (not fitting one by one, but rather building the model with everything at once)... This is a variation of the trick we used to greatly improve GP performance by deriving an average value from the data (an often overlooked problem). For the next step I'm still undecided between EM algorithms to provide a model with linear variations (better for porting the model to the test set) and K-means inspired methods which may provide better hint for interpreting the data.\nMy main driver in this is to understand how statistics-based models with a likelihood (supposedly an optimal method) can be compared with all novelties in ML which seem to be able to outperform it !
", "votes": 0, "replies": [] } ] }, { "id": 440973, "author_name": "joxemi", "author_url": "", "post_date": "2018-12-18T06:08:18.200000", "content": "Congrats! You are a great specialist in your \n domain ( astronomy) and an awesome ML practicioner . You have shown us how to combine talent and work in an extraordinary and optimized way. Thanks to the organizers for allowing us to struggle and enjoy in this challenge with data, code,algorithms and patience , beatiful ingredients for a stellar soup, and to our families and friends for giving us time to cook all this.
", "votes": 1, "replies": [] }, { "id": 441046, "author_name": "Tania J", "author_url": "", "post_date": "2018-12-18T07:55:49.953000", "content": "Awesome work Kyle! I hope you include more about how you fit the GPs in your full write-up. Even basic (GP 101) stuff would be useful for many people I think, who have come across them for the first time in this competition. For instance, why did you choose the Matern kernel over others (I guess its not by chance that both you and CPMP used it) and how you choose the parameters. Also any links to good resources for learning about GPs would be appreciated. I tried GPs towards the end but never managed to get good fits.
", "votes": 2, "replies": [] }, { "id": 440863, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2018-12-18T03:16:27.900000", "content": "Congrats Kyle. Would you like to give more explanations about \"Measured 200 features on the raw data and Gaussian process predictions.\" Especially how to use GP to extract important features, this would be quite helpful for many other related problems.
", "votes": 2, "replies": [ { "id": 440875, "author_name": "Kyle Boone", "author_url": "", "post_date": "2018-12-18T03:52:00.963000", "content": "I'll release a full write-up when I get the code all cleaned up that details all of these features.
", "votes": 2, "replies": [] } ] }, { "id": 457546, "author_name": "axk477", "author_url": "", "post_date": "2019-01-17T17:02:17.610000", "content": "Hi guys I am new to this competition and to the field of machine learning, (4th year computer science undergraduate), congratulations on the 1st place solution, If it would be possible for anyone to help me understand the approach to this challenge, regarding the dimensionality reduction, lightcurve fitting(why it is important and how is it different to fitting a classifier), and the importance of timeseries and how they are produced and what are their uses. I am very eager to learn these methods and techniques but without a clear idea of i would appreciate anyones help to get me comfortable to explore on my own.
", "votes": 0, "replies": [] }, { "id": 448219, "author_name": "Rajesh D", "author_url": "", "post_date": "2018-12-31T13:56:48.133000", "content": "Hi Kyle,
\n\n Great work! and thanks for the nice post and code. I am currently working on a classifier and would like to use your feature set for experimentation and testing. I kindly request for simple features (200) and class labels file, for both train and test set. It would be great if I have these two files, as I dont have the environment/facility to use your python code on github.\nThank you in advance.\nRegards\nRajesh
", "votes": 0, "replies": [] }, { "id": 445580, "author_name": "Dmitry Vanyagin", "author_url": "", "post_date": "2018-12-26T19:10:14.920000", "content": "Congratulations and thank you for detailed explanation!
", "votes": 0, "replies": [] }, { "id": 445328, "author_name": "DW", "author_url": "", "post_date": "2018-12-26T08:15:21.013000", "content": "Congrats, Kyle !
", "votes": 0, "replies": [] }, { "id": 445209, "author_name": "Diogo Veloso", "author_url": "", "post_date": "2018-12-26T00:20:15.253000", "content": "congrats
", "votes": 0, "replies": [] }, { "id": 444751, "author_name": "Yersain Makazhanov", "author_url": "", "post_date": "2018-12-24T18:14:32.983000", "content": "Gratz, great approach)
", "votes": 0, "replies": [] }, { "id": 444565, "author_name": "sarthak rawat", "author_url": "", "post_date": "2018-12-24T09:36:53.887000", "content": "Awesome work.
", "votes": 0, "replies": [] }, { "id": 444557, "author_name": "Tom Rupp", "author_url": "", "post_date": "2018-12-24T09:03:51.507000", "content": "Very nice solution Kyle Boone!! Congratulations!
", "votes": 0, "replies": [] }, { "id": 444501, "author_name": "Cedric Chee", "author_url": "", "post_date": "2018-12-24T06:26:08.353000", "content": "Congrats! I love simple solution. Interesting to see how you leverage your domain expertise in physics (cosmology to be specific?) and apply to the understand the problem (data part of ML). Then, you focus on doing that one thing well. And keep believing that ensembling is the last option. A recipe of success. Brilliant and inspiring.
", "votes": 0, "replies": [] }, { "id": 444425, "author_name": "YupengYao", "author_url": "", "post_date": "2018-12-24T02:33:21.650000", "content": "Brilliant. Thanks for sharing, really helps alot.
", "votes": 0, "replies": [] }, { "id": 444416, "author_name": "Tommy Tang", "author_url": "", "post_date": "2018-12-24T02:21:32.557000", "content": "Congratulations !
\n\nThanks for the explanations!
", "votes": 0, "replies": [] }, { "id": 444338, "author_name": "kuldip chauhan", "author_url": "", "post_date": "2018-12-23T20:20:07.560000", "content": "congrats
", "votes": 0, "replies": [] }, { "id": 443835, "author_name": "huangzhiyong", "author_url": "", "post_date": "2018-12-22T14:23:29.970000", "content": "great
", "votes": 0, "replies": [] }, { "id": 443300, "author_name": "Alex Schiaucu", "author_url": "", "post_date": "2018-12-21T11:44:09.990000", "content": "Congrats!
\n\nWas really nice to hear about the data augmentation method you used! Also I found particularly interesting how you fitted the redshift to align with the test set redshifts and adjusted the fluxes according to the relativistic effects you described.
", "votes": 0, "replies": [] }, { "id": 443182, "author_name": "HelloGurney", "author_url": "", "post_date": "2018-12-21T07:00:54.540000", "content": "Congratulations!\nIt's a Amazing
", "votes": 0, "replies": [] }, { "id": 443051, "author_name": "hisashi_h", "author_url": "", "post_date": "2018-12-20T23:57:42.303000", "content": "Congratulations!\nI could not think of this solution.
", "votes": 0, "replies": [] }, { "id": 442598, "author_name": "Sennanaicker", "author_url": "", "post_date": "2018-12-20T07:15:13.950000", "content": "Congrats Kyle..
", "votes": 0, "replies": [] }, { "id": 442507, "author_name": "GSD", "author_url": "", "post_date": "2018-12-20T03:26:18.773000", "content": "Congratulations Kyle ..Amazing writeup ..
", "votes": 0, "replies": [] }, { "id": 442154, "author_name": "NamApolo", "author_url": "", "post_date": "2018-12-19T14:46:56.160000", "content": "Congratulations and thank you for sharing your solution!
", "votes": 0, "replies": [] }, { "id": 441653, "author_name": "Andy Penrose", "author_url": "", "post_date": "2018-12-18T22:33:26.950000", "content": "Congratulations on a stunning first place! Thanks for sharing this and thanks for your comments in the discussions along the way.
", "votes": 0, "replies": [] }, { "id": 441647, "author_name": "", "author_url": "", "post_date": "2018-12-18T22:16:47.737000", "content": "", "votes": 0, "replies": [] }, { "id": 441618, "author_name": "", "author_url": "", "post_date": "2018-12-18T21:00:20.137000", "content": "", "votes": 0, "replies": [] }, { "id": 441368, "author_name": "", "author_url": "", "post_date": "2018-12-18T15:34:49.377000", "content": "", "votes": 0, "replies": [] }, { "id": 441328, "author_name": "", "author_url": "", "post_date": "2018-12-18T14:50:21.093000", "content": "", "votes": 0, "replies": [] }, { "id": 441319, "author_name": "", "author_url": "", "post_date": "2018-12-18T14:42:44.847000", "content": "", "votes": 0, "replies": [] }, { "id": 441259, "author_name": "", "author_url": "", "post_date": "2018-12-18T13:35:55.650000", "content": "", "votes": 0, "replies": [] }, { "id": 441242, "author_name": "", "author_url": "", "post_date": "2018-12-18T12:59:46.307000", "content": "", "votes": 0, "replies": [ { "id": 441392, "author_name": "", "author_url": "", "post_date": "2018-12-18T15:59:26.500000", "content": "", "votes": 2, "replies": [] }, { "id": 441473, "author_name": "", "author_url": "", "post_date": "2018-12-18T17:30:14.777000", "content": "", "votes": 0, "replies": [] }, { "id": 441558, "author_name": "", "author_url": "", "post_date": "2018-12-18T19:37:20.803000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 441203, "author_name": "", "author_url": "", "post_date": "2018-12-18T11:59:30.130000", "content": "", "votes": 0, "replies": [] }, { "id": 441195, "author_name": "", "author_url": "", "post_date": "2018-12-18T11:49:13.913000", "content": "", "votes": 0, "replies": [] }, { "id": 441139, "author_name": "", "author_url": "", "post_date": "2018-12-18T10:08:53.210000", "content": "", "votes": 0, "replies": [] }, { "id": 441118, "author_name": "", "author_url": "", "post_date": "2018-12-18T09:28:03.140000", "content": "", "votes": 0, "replies": [ { "id": 441390, "author_name": "", "author_url": "", "post_date": "2018-12-18T15:57:19.227000", "content": "", "votes": 1, "replies": [] }, { "id": 441426, "author_name": "", "author_url": "", "post_date": "2018-12-18T16:47:16.687000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 441095, "author_name": "", "author_url": "", "post_date": "2018-12-18T08:58:08.460000", "content": "", "votes": 0, "replies": [ { "id": 441387, "author_name": "", "author_url": "", "post_date": "2018-12-18T15:55:17.380000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 441065, "author_name": "", "author_url": "", "post_date": "2018-12-18T08:20:41.173000", "content": "", "votes": 0, "replies": [] }, { "id": 441064, "author_name": "", "author_url": "", "post_date": "2018-12-18T08:20:02.623000", "content": "", "votes": 0, "replies": [] }, { "id": 441041, "author_name": "", "author_url": "", "post_date": "2018-12-18T07:47:09.910000", "content": "", "votes": 0, "replies": [] }, { "id": 440950, "author_name": "", "author_url": "", "post_date": "2018-12-18T05:45:38.430000", "content": "", "votes": 0, "replies": [] }, { "id": 440887, "author_name": "", "author_url": "", "post_date": "2018-12-18T04:09:36.503000", "content": "", "votes": 0, "replies": [ { "id": 440891, "author_name": "", "author_url": "", "post_date": "2018-12-18T04:19:23.790000", "content": "", "votes": 8, "replies": [] }, { "id": 440895, "author_name": "", "author_url": "", "post_date": "2018-12-18T04:28:38.440000", "content": "", "votes": 5, "replies": [] }, { "id": 441054, "author_name": "", "author_url": "", "post_date": "2018-12-18T08:12:41.620000", "content": "", "votes": 0, "replies": [] }, { "id": 441253, "author_name": "", "author_url": "", "post_date": "2018-12-18T13:28:39.197000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 440862, "author_name": "", "author_url": "", "post_date": "2018-12-18T03:16:12.940000", "content": "", "votes": 0, "replies": [] }, { "id": 440860, "author_name": "", "author_url": "", "post_date": "2018-12-18T03:13:24.570000", "content": "", "votes": 0, "replies": [ { "id": 440882, "author_name": "", "author_url": "", "post_date": "2018-12-18T04:03:15.780000", "content": "", "votes": 4, "replies": [] }, { "id": 440897, "author_name": "", "author_url": "", "post_date": "2018-12-18T04:29:45.223000", "content": "", "votes": 1, "replies": [] }, { "id": 441143, "author_name": "", "author_url": "", "post_date": "2018-12-18T10:12:57.360000", "content": "", "votes": 1, "replies": [] }, { "id": 441438, "author_name": "", "author_url": "", "post_date": "2018-12-18T16:59:18.407000", "content": "", "votes": 0, "replies": [] }, { "id": 441497, "author_name": "", "author_url": "", "post_date": "2018-12-18T18:01:27.590000", "content": "", "votes": 1, "replies": [] }, { "id": 442123, "author_name": "", "author_url": "", "post_date": "2018-12-19T14:08:24.860000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 440854, "author_name": "", "author_url": "", "post_date": "2018-12-18T03:05:18.740000", "content": "", "votes": 0, "replies": [ { "id": 440873, "author_name": "", "author_url": "", "post_date": "2018-12-18T03:50:03.800000", "content": "", "votes": 7, "replies": [] }, { "id": 440878, "author_name": "", "author_url": "", "post_date": "2018-12-18T03:54:53.180000", "content": "", "votes": 0, "replies": [] }, { "id": 440884, "author_name": "", "author_url": "", "post_date": "2018-12-18T04:07:39.320000", "content": "", "votes": 1, "replies": [] }, { "id": 440888, "author_name": "", "author_url": "", "post_date": "2018-12-18T04:09:51.777000", "content": "", "votes": 0, "replies": [] }, { "id": 442367, "author_name": "", "author_url": "", "post_date": "2018-12-19T21:39:09.163000", "content": "", "votes": 0, "replies": [] }, { "id": 442378, "author_name": "", "author_url": "", "post_date": "2018-12-19T21:56:15.417000", "content": "", "votes": 2, "replies": [] }, { "id": 442393, "author_name": "", "author_url": "", "post_date": "2018-12-19T22:45:27.597000", "content": "", "votes": 0, "replies": [] }, { "id": 442438, "author_name": "", "author_url": "", "post_date": "2018-12-20T00:52:35.857000", "content": "", "votes": 0, "replies": [] }, { "id": 442731, "author_name": "", "author_url": "", "post_date": "2018-12-20T11:56:17.823000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2354208, "author_name": "", "author_url": "", "post_date": "2023-07-22T09:39:21.017000", "content": "", "votes": 0, "replies": [] }, { "id": 444935, "author_name": "", "author_url": "", "post_date": "2018-12-25T07:19:08.093000", "content": "", "votes": -1, "replies": [] }, { "id": 448152, "author_name": "", "author_url": "", "post_date": "2018-12-31T10:59:40.117000", "content": "", "votes": 0, "replies": [] }, { "id": 446173, "author_name": "", "author_url": "", "post_date": "2018-12-27T16:40:20.947000", "content": "", "votes": 0, "replies": [] }, { "id": 444413, "author_name": "", "author_url": "", "post_date": "2018-12-24T02:06:27.107000", "content": "", "votes": 0, "replies": [] }, { "id": 441676, "author_name": "", "author_url": "", "post_date": "2018-12-18T23:25:43.040000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "440847": "EDIT: code is now available on my github page at https://github.com/kboone/avocado\n\nFirst of all, thanks to everyone who participated in this competition! I learned a lot doing it, and I have enjoyed all of the discussions that I had with you. Here is an overview of my model that took 1st place in this competition. I will be releasing the code with a full writeup shortly.\n\nI am an astronomer studying supernova cosmology, so my work mainly focused on trying to tell the different supernova types apart. This ended up working out well because everything else was fairly easy to tell apart. Here is a summary of my solution:\n\n- Augmented the training set by degrading the well-observed lightcurves in the training set to match the properties of the test set.\n- Use Gaussian processes to predict the lightcurves.\n- Measured 200 features on the raw data and Gaussian process predictions.\n- Trained a single LGBM model with 5-fold cross-validation.\n\nI first use Gaussian process (GP) regression to extract features. I trained a GP on each object using a Matern Kernel with a fixed length scale in the wavelength direction and a variable length scale in the time direction. My machine could do ~10 fits per second so it took around 3 days of computation time to do all the fits. Gaussian processes produce very nice models for well-sampled lightcurves, and are able to get a nice model even when the measurements are in different bands. They also handle measurements with large uncertainties very gracefully. For poorly sampled lightcurves, the GP fits the available data well, but doesn't always do great for extrapolation. Here is an example of what comes out:\n\n![gp example][1]\n\nUsing the GP predictions, I calculated lots of different features. The distinguishing features of supernovae are their peak brightnesses and the width of their lightcurves, so I put several measures of those into the model. For poorly sampled lightcurves, the GP doesn't always give great results, so I added features that let the model know how well the GP is doing. This basically boils down to counting the number of observations in different windows around maximum light. I also added features related to the signal-to-noise in each band, and some simple peak detection and counting to help with the non-supernova classes.\n\nNow the training set is very different from the test set. To deal with this, I took every lightcurve in the training set and degraded it up to 40 times to get something that looks like the less well-sampled lightcurves in the test set. The degradation includes:\n\n- Modifying the brightness of galactic objects.\n- Modifying the redshift of extragalactic objects (including dilating time and changing the brightness).\n- Adding in large \"gaps\" like the ones in the real data that show up because of the time of year.\n- Choosing a new photo-z and photo-z error for the observation based on a model of how the spec-zs in the data turn into photo-zs.\n- Simulating the detection to choose which objects would be included in the dataset that we were given.\n\nThis degradation was all tuned to the training/test datasets, and no external data was used. After this procedure, I end up with a training set of ~270000 objects that is much more representative of the test set than the original training set. I trained a LightGBM model on this training set using 5-fold cross-validation and making sure that I kept the up to 40 degradations of each object in the same folds. After tuning this model, I get a CV of around 0.4 on the original training set. Here is the confusion matrix:\n\n![confusion matrix][2]\n\nThere are some interesting differences compared to [CPMP's confusion matrix][3], such as the fact that I have much lower accuracy on class 6 objects than CPMP. This appears to be due to the fact that my degradation procedure produces more low signal-to-noise class 65 than class 6 objects. It will be interesting to see if my method really does reproduce the test set better.\n\nI played a lot with how to identify the class 99 targets. I found that the tree based models that I used are not very good for outlier detection. My best results came from choosing a flat score to give to the class 99 objects, and then using that in the soft-max to get the final probabilities. Using this, I got what I consider my best real score of 0.726 on the public leaderboard.\n\nAfter trying to improve this score for a long time and getting nowhere, with a week to go I realized that I could figure out what the class 99 objects look like/don't look like by probing the leaderboard. This defeats the purpose of the class 99 prediction, and unfortunately ends up doing much better than any real estimate of the class 99 objects. I contacted the organizers about it, and was told that it was within the Kaggle rules to do this. In the end, I found that my best prediction of the class 99 objects was a weighted average of the predictions for classes 42, 52, 62 and 95. This trick boosted my final score to 0.670 on the public leaderboard. It will be interesting to see what other competitors did here.\n\nOverall, I really enjoyed this competition and I learned a lot! I am working on cleaning up my code and making it friendly for others to play with. I think that there is quite a bit of room to improve the tuning of my model, and I did not try doing any kind of ensembling or using classifiers other than LGBM. Thanks to everyone who participated, especially @cpmpml who initiated a lot of great discussions and @ogrellier whose kernel I started with for my work!\n\nFor any astronomers who participated, I'll be at AAS in a couple weeks. I'd love to meet up and discuss the competition!\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/440847/10890/gp_example.png\n [2]: https://storage.googleapis.com/kaggle-forum-message-attachments/438639/10876/confusion_matrix.png\n [3]: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/74564", "445181": "The code is now available on my github page at https://github.com/kboone/plasticc", "441007": "Congratulations Kyle ! Awesome work. \n\nAfter reading your write up I believe I now understand the dataset better ;-)", "440856": "Thanks for sharing! It's amazing! @cpmpml is indeed a discussion GM...\nIf we had 270000 objects, things would be very much different for all of us, lol\nWe did some augmentation in training, but far from your level...\n\nCongratulation!", "443230": "Congrats Kyle Boone, thank you for detailed explanation.", "442329": "Congrats Kyle Boone, you are the real super man in this competition :)\nI hope you continue kaggle and if possible I would like to compete with you again!", "441202": "Congratulations Kyle ! I'm so happy so see you win the race. This is also a good lesson that domain knowledge is useful to design the solution, and I'm pretty confident that mixing the solutions of the top rankers can still improve a lot the solution. In other words, you made a clear case for a collaborative approach to the solution...\n\nRegarding the GP, I think the way data are extrapolated is one of the key issues of this challenge. Have you considered at some point to build templates for the lightcuves ? This was the approach I was trying, as it alleviates the problem of interpolation over long range of missing data by using the template. Of course, interpolation is then dependent on the hypothesis on the class. See for instance what this approach gives below. It probably beats what GP can do (erro band represent class variability, not object interpolation uncertainty)\n\n![Template interpolation on class 90][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/441202/10896/Unknown.png", "440973": "Congrats! You are a great specialist in your \n domain ( astronomy) and an awesome ML practicioner . You have shown us how to combine talent and work in an extraordinary and optimized way. Thanks to the organizers for allowing us to struggle and enjoy in this challenge with data, code,algorithms and patience , beatiful ingredients for a stellar soup, and to our families and friends for giving us time to cook all this.", "441046": "Awesome work Kyle! I hope you include more about how you fit the GPs in your full write-up. Even basic (GP 101) stuff would be useful for many people I think, who have come across them for the first time in this competition. For instance, why did you choose the Matern kernel over others (I guess its not by chance that both you and CPMP used it) and how you choose the parameters. Also any links to good resources for learning about GPs would be appreciated. I tried GPs towards the end but never managed to get good fits.", "440863": "Congrats Kyle. Would you like to give more explanations about \"Measured 200 features on the raw data and Gaussian process predictions.\" Especially how to use GP to extract important features, this would be quite helpful for many other related problems. ", "457546": "Hi guys I am new to this competition and to the field of machine learning, (4th year computer science undergraduate), congratulations on the 1st place solution, If it would be possible for anyone to help me understand the approach to this challenge, regarding the dimensionality reduction, lightcurve fitting(why it is important and how is it different to fitting a classifier), and the importance of timeseries and how they are produced and what are their uses. I am very eager to learn these methods and techniques but without a clear idea of i would appreciate anyones help to get me comfortable to explore on my own. ", "448219": "Hi Kyle,\n\n Great work! and thanks for the nice post and code. I am currently working on a classifier and would like to use your feature set for experimentation and testing. I kindly request for simple features (200) and class labels file, for both train and test set. It would be great if I have these two files, as I dont have the environment/facility to use your python code on github.\n\nThank you in advance.\nRegards\nRajesh\n", "445580": "Congratulations and thank you for detailed explanation!", "445328": "Congrats, Kyle !", "445209": "congrats", "444751": "Gratz, great approach)", "444565": "Awesome work. ", "444557": "Very nice solution Kyle Boone!! Congratulations!", "444501": "Congrats! I love simple solution. Interesting to see how you leverage your domain expertise in physics (cosmology to be specific?) and apply to the understand the problem (data part of ML). Then, you focus on doing that one thing well. And keep believing that ensembling is the last option. A recipe of success. Brilliant and inspiring.", "444425": "Brilliant. Thanks for sharing, really helps alot.", "444416": "Congratulations !\n\nThanks for the explanations!", "444338": "congrats", "443835": "great", "443300": "Congrats! \n\nWas really nice to hear about the data augmentation method you used! Also I found particularly interesting how you fitted the redshift to align with the test set redshifts and adjusted the fluxes according to the relativistic effects you described.", "443182": "Congratulations!\nIt's a Amazing", "443051": "Congratulations!\nI could not think of this solution.", "442598": "Congrats Kyle..", "442507": "Congratulations Kyle ..Amazing writeup ..", "442154": "Congratulations and thank you for sharing your solution!", "441653": "Congratulations on a stunning first place! Thanks for sharing this and thanks for your comments in the discussions along the way.", "441647": "Congrats! What a cool solution!", "441618": "Congratulations! Brilliant solution!", "441368": "Congrats !", "441328": "Congratulations @kyleboone and thanks for sharing your solution. Lot to learn from it.", "441319": "Congrats and thanks for sharing the solution...", "441259": "Congratulations and thank you for sharing your solution! ", "441242": "Ming blowing solution @kyleboone, you have focused more on data part rather than modeling part.\n\nGenerally 1st place team have large ensembles of 2-3 layers of NNs & boosting algorithms & some linear model as meta model. But you haven't done that still you managed to be 1st! That's outstanding.. ", "441203": "Great Kyle. Congratulations and Well deserved First place solution. ", "441195": "Big congrats Kyle! Very impressive solution, specially the GP, augmentation and class 99 probing.", "441139": "Congrats for the great work, and the highly deserved win! Admirable performance! It would be very interesting to see how your data-oriented method could gain from other solutions, which look more ml-oriented.\n\n", "441118": "Congrats!! My GP fitting is still running... Can you explain more about the two dimensional GP? The passband in this case will be continuous or discrete?", "441095": "Thanks for sharing!\n\nI also augmented train set, but doesn’t work. More specific, I augmented ddf object by degrading 200 times to convert ddf into wfd, coz I thought test set has much more wfd. Each wfd has about 120 samples, and each ddf has about 320 samples.\n\nI have not tested yet, but I believe I degraded too much.\n\nAnyway, good for you 1st place! Well deserved!", "441065": " Congratulations Kyle!! That's amazing work! \nI was so looking forward to this post... I had no idea how to treat this difference distributions between train and test and I don't think I would have come up with the GP idea.\n\nThank you for sharing and for all the discussions!", "441064": "Thank you Kyle !\nIt was definitely the most interesting competition I participated in.\nI would like to give a special thank to the active kagglers who have shared a lot of ideas and good mood!", "441041": "Congratulations !\n\nThanks for the explanations too.", "440950": "Congrats! Gaussian process gave me a headache at the stats class. I really need to pick it up again. :D", "440887": "Congrats. Thanks for sharing that fast.\n\nI wish I had started earlier on gaussian process modeling. That's the code I finished the last day of the competition, and I had too few submisssions to start using it properly. I also used Matern kernel :) I guess you used celerite, right?\n\nYour data augmentation/degradation is awesome. I vaguely thought it could be done, but not to base the whole solution on it.\n\nI am surprised by how much you gained by probing for class 99. We haven't done it at all. ", "440862": "Wow, that's impressive !! Now I understand better why my score is so far from yours ;-)", "440860": "Congratulations Kyle!\nThis is very interesting! You managed to pull off exactly what we have been trying and failing! It's very insightful hearing how you managed to apply Gaussian Process for feature extraction!\n\nWe also tried GP, but are unable to properly create a model that combines multiple passbands while dealing with missingness. As a result, the GP model is not really able to cope with low sample rate objects without significant over/underfitting.\n\nWe did the downsampling of DDF objects during the last phase of the competition and it really helped greatly. We also had this idea to combine GP and downsampling, using GP to fit DDF objects in the training set then downsample them to non-DDF resolution using a random observation timestamps from an arbitrary object. We had issues separating bad generated examples from informative ones, mainly due to the inability to replicate accurate observation and 'detected' status estimates for the resampled points. However, I still believe this approach should work well if done properly, and can potentially lead to a model with very good uncertainty estimates without the need to fit Gaussian Processes on every object during inference. The idea is to build a model P(class|observed) by creating more realistic 'observed' objects rather than building a model P(class|full_curve)P(full_curve|observed) by trying to reconstruct every light curve. If anyone is interested I would like to develop this idea further and see if it can help the astronomers in performance-critical situations.\n\nBTW, how can I learn about Gaussian Process that deals with missingness and/or 'observed' measurement errors? Would appreciate it if anyone can point the way!", "440854": "Brilliant stuff! How did you figure out which parameters to degrade in the train set? Was there any sort of algorithmic process or was it more investigative trial-and-error? ", "2354208": "Congratulations Kyle ! ", "444935": "Thanks for the detailed explanation :)", "448152": "Congrats! And thank you for sharing.", "446173": "Thanks for all this info! Very Useful", "444413": "This is perfect! Thank you for sharing! ", "441676": "Congraturations! Thank you for sharing. " } }