First of all, thanks to Kaggle and LSST teams for holding this fantastic competition. It is also very interesting and made me know a lot more about what mysterious things our astronomers are doing. I am very grateful for the community and learned very much from kagglers' generous kernels and discussions. Congratulations to the incredible @Kyle, all the medal winners, my teammates @lucaskg, @zuoweijian, @xietian6578, and especially my teammate @strideradu who became a Kaggle Competition Master.
\n\nOur final model is an ensemble of LightGBM, XGBoost, and binary classifiers. Final CV=0.3252(with specz)/0.3842(without specz), Public LB=0.814, Private LB=0.826.
\n\nHere is a summary of what we did during the last tough several weeks.
\n\nWe performed feature extraction manually one by one, inspired by the functions listed on FATS. I guess they are basically similar to those extracted from packages such as tsfresh and cesium. Around a total of 50-60 kinds of features are extracted to characterize each light curve. We did both passband-level and object-level feature extractions.
\n\nWe referred to this kernel published by @Scirpus. We evaluated the periodogram and grouped the result into ~20 frequency bins. The amplitude was summed in each frequency bin.
\n\nMy teammate @lucaskg did the curve fitting, using this paper for reference.
\n\nWe performed feature extractions on different datasets, including:
\n\noriginal flux features,
original flux features while detected == 1,
magnitude1 (flux * specz * specz) features,
magnitude1 features while detected == 1,
magnitude2 (flux * photoz * photoz) features,
magnitude2 features while detected == 1
Besides aggregating passband-level features using 'std' and 'mean', we constructed some features that reflect the difference among passbands or passband groups, such as:
\n\nPlease refer to Part 2.1 below.
\n\nWe built three models: galactic, extragalactic with specz, and extragalactic without specz.
\n\nWe ran a LightGBM on passband level. In this model, we only employed those features related to the shape of the light curve, such as flux skewness and frequency features. All features related to the magnitude of the flux, such as flux max and flux mean, are ignored. After we got the passband-level probability prediction, we flattened them as a group of meta-features.
\n\nIt may be noteworthy that in the galactic model, we removed all the data with passband == 0. We found the prediction can be more accurate without them. At least, CV improved with doing that.
\n\nWe applied LightGBM at most of the time during the competition and tried XGBoost on the last day. To avoid overfitting, feature numbers are restricted to 100 for galactic model and 140 for extragalactic model, based on the feature importance obtained by LightGBM. The performance of single models is as follows.
\n\nBesides the multi-class classification models, my teammate @lucaskg conducted a binary classification on each of extragalactic classes.
\n\nI tried to stack LightGBM and XGBoost by allocating them different weights, but CV did not improve at all. I guess the reason could be the high correlation between them. So I turned to use LightGBM to attempt an ensemble on LightGBM, XGBoost and binary classification predictions. It turned to be useful with CV improved from 0.3448 to 0.3252(with specz), 0.4143 to 0.3842(without specz), public LB score improved from 0.831 to 0.814, and private LB score improved from 0.847 to 0.826.
\n\nActually, what we did on class_99 is not worth mentioning because we only got a 0.001 boost on the leaderboard. However, considering we are finally just 0.0004 ahead of 13th place (who is the first of silver medal winners) on private LB, it was very crucial to help us achieve the gold medal.
\n\nWe applied @olivier's method and @Scirpus' method, and found the latter is always a little better than the former on public LB. So I tried to interpolate class_99 by setting it to:
\n\nclass99Prob = class99scirpus + (class99scirpus - class99olivier)
\n\nwhich boosted the LB score by 0.001. Here, we would like to express our sincere thanks and appreciation to @olivier and @Scirpus, helping us make our gold dream come true.
\n\nFirst of all, thanks to Kaggle and LSST teams for holding this fantastic competition. It is also very interesting and made me know a lot more about what mysterious things our astronomers are doing. I am very grateful for the community and learned very much from kagglers' generous kernels and discussions. Congratulations to the incredible @Kyle, all the medal winners, my teammates @lucaskg, @zuoweijian, @xietian6578, and especially my teammate @strideradu who became a Kaggle Competition Master.
\n\nOur final model is an ensemble of LightGBM, XGBoost, and binary classifiers. Final CV=0.3252(with specz)/0.3842(without specz), Public LB=0.814, Private LB=0.826.
\n\nHere is a summary of what we did during the last tough several weeks.
\n\nWe performed feature extraction manually one by one, inspired by the functions listed on FATS. I guess they are basically similar to those extracted from packages such as tsfresh and cesium. Around a total of 50-60 kinds of features are extracted to characterize each light curve. We did both passband-level and object-level feature extractions.
\n\nWe referred to this kernel published by @Scirpus. We evaluated the periodogram and grouped the result into ~20 frequency bins. The amplitude was summed in each frequency bin.
\n\nMy teammate @lucaskg did the curve fitting, using this paper for reference.
\n\nWe performed feature extractions on different datasets, including:
\n\noriginal flux features,
original flux features while detected == 1,
magnitude1 (flux * specz * specz) features,
magnitude1 features while detected == 1,
magnitude2 (flux * photoz * photoz) features,
magnitude2 features while detected == 1
Besides aggregating passband-level features using 'std' and 'mean', we constructed some features that reflect the difference among passbands or passband groups, such as:
\n\nPlease refer to Part 2.1 below.
\n\nWe built three models: galactic, extragalactic with specz, and extragalactic without specz.
\n\nWe ran a LightGBM on passband level. In this model, we only employed those features related to the shape of the light curve, such as flux skewness and frequency features. All features related to the magnitude of the flux, such as flux max and flux mean, are ignored. After we got the passband-level probability prediction, we flattened them as a group of meta-features.
\n\nIt may be noteworthy that in the galactic model, we removed all the data with passband == 0. We found the prediction can be more accurate without them. At least, CV improved with doing that.
\n\nWe applied LightGBM at most of the time during the competition and tried XGBoost on the last day. To avoid overfitting, feature numbers are restricted to 100 for galactic model and 140 for extragalactic model, based on the feature importance obtained by LightGBM. The performance of single models is as follows.
\n\nBesides the multi-class classification models, my teammate @lucaskg conducted a binary classification on each of extragalactic classes.
\n\nI tried to stack LightGBM and XGBoost by allocating them different weights, but CV did not improve at all. I guess the reason could be the high correlation between them. So I turned to use LightGBM to attempt an ensemble on LightGBM, XGBoost and binary classification predictions. It turned to be useful with CV improved from 0.3448 to 0.3252(with specz), 0.4143 to 0.3842(without specz), public LB score improved from 0.831 to 0.814, and private LB score improved from 0.847 to 0.826.
\n\nActually, what we did on class_99 is not worth mentioning because we only got a 0.001 boost on the leaderboard. However, considering we are finally just 0.0004 ahead of 13th place (who is the first of silver medal winners) on private LB, it was very crucial to help us achieve the gold medal.
\n\nWe applied @olivier's method and @Scirpus' method, and found the latter is always a little better than the former on public LB. So I tried to interpolate class_99 by setting it to:
\n\nclass99Prob = class99scirpus + (class99scirpus - class99olivier)
\n\nwhich boosted the LB score by 0.001. Here, we would like to express our sincere thanks and appreciation to @olivier and @Scirpus, helping us make our gold dream come true.
\n\nSeems feature engineering & ensemble are very important in this task, thanks for your sharing.
", "rawMarkdown": "Seems feature engineering & ensemble are very important in this task, thanks for your sharing.", "votes": null }, { "id": "446029", "postDate": "12/27/2018 11:18:14", "content": "Thank you for sharing!!
\n\n\n", "rawMarkdown": "Thank you for sharing!!\n> It may be noteworthy that in the galactic model, we removed all the data with passband == 0. \nHow did you find that interesting process?", "votes": null }, { "id": "446116", "postDate": "12/27/2018 14:29:36", "content": "It may be noteworthy that in the galactic model, we removed all the data with passband == 0. \n How did you find that interesting process?
\n
We checked the confusion matrix of our passband-level model and found that the accuracy is always better if only using data of higher passbands. But we didn't have time to try others.
", "rawMarkdown": "We checked the confusion matrix of our passband-level model and found that the accuracy is always better if only using data of higher passbands. But we didn't have time to try others.", "votes": null }, { "id": "446568", "postDate": "12/28/2018 09:42:53", "content": "Checking passband-level confusion matrix is great idea!!
\nI think this idea is useful in general.
\nThank you!!!
Seems feature engineering & ensemble are very important in this task, thanks for your sharing.
", "votes": null, "replies": [] }, { "id": 446029, "author_name": "go5kuramubon", "author_url": "", "post_date": "12/27/2018 11:18:14", "content": "Thank you for sharing!!
\n\n\n", "votes": null, "replies": [ { "id": 446116, "author_name": "lfcpeng17", "author_url": "", "post_date": "12/27/2018 14:29:36", "content": "It may be noteworthy that in the galactic model, we removed all the data with passband == 0. \n How did you find that interesting process?
\n
We checked the confusion matrix of our passband-level model and found that the accuracy is always better if only using data of higher passbands. But we didn't have time to try others.
", "votes": null, "replies": [] }, { "id": 446568, "author_name": "go5kuramubon", "author_url": "", "post_date": "12/28/2018 09:42:53", "content": "Checking passband-level confusion matrix is great idea!!
\nI think this idea is useful in general.
\nThank you!!!