![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F493138%2Fd817d546a501a2858b05da9f8b75b2e4%2Ffirst_iter_plasstic.png?generation=1561450779846289&alt=media\")
\nand after aggregation we have an updated representative light curve for the class:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F493138%2F2e9e3763208d178050a8d727cbdc1b72%2Fagg_1.png?generation=1561450855362832&alt=media\")
During the competition I got curious about possible ways how to reconstruct representative light curves of hard classes. I was lucky to somehow reconstruct those. Features based on it likely helped me to jump from the 335th place on public LB place to the 182th place on private LB, which seems to be one of the largest jumps in this competition.
\n\nIt was very exciting for me to see how “signatures of supernovae” emerge from noisy data! 😊 I’ve even created a kernel to share some of routines I tried out. However, I made it private almost immediately. Because when gratefully reading about all the great solutions by top-scoring teams I just felt humble and didn’t want to distract anyone from all those fantastic solutions. After half a year I’d like to briefly share a couple of things I did, just in case one day it might be of any practical use to someone in the astroinformatics community.
\n\nThe main idea was to match and then to robustly aggregate light curves of the same class. For instance, after some rescaling and shifting we match two instances of the class 62 like this:\n\nand after aggregation we have an updated representative light curve for the class:\n
and 80 iterations later we have:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F493138%2F4f3319866f3d10cb5c64e3341f6ac4ea%2Flater_iter_plasstic.png?generation=1561450954218460&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F493138%2F1941477896f468acbb5971366365af9b%2Fagg_later.png?generation=1561450974081452&alt=media\")
This way I match-aggregated light curves which likely had maximum. Next, I match-aggregated light curves without apparent maximum, where I had to be slightly more careful with rescaling:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F493138%2F8f15a4d1b39ab2b92ef31760ce5bd630%2Fpartial_match.png?generation=1561451250020125&alt=media\")
In the end I've got representative light curves for different classes, and for the first time I saw slight differences in the hard classes. E.g., we dealt with the class 62 in the previous figures. In the figure below we can compare representative light curves of two different classes with the mentioned 62-class light curve:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F493138%2Fade9b0bf26bfd81877a037f44cd69e86%2Fcomparison.png?generation=1561451697251362&alt=media\")
Finally, I computed some features based on the representative light curves. The main idea was to try to match a light curve with every class prototype, looking for the best rescaling (both time and flux scales). The matching was done using either squared error or cosine error (so, either forcing points to be close to each other, or forcing segments to have similar directions). As CPMP advised to write efficient code, I wrote an efficient implementation in Cython.
\n\nIf interested, you can find the implementation with an example of usage in this kernel.
\n\nP.S. I forgot to mention that before starting the match-aggregate thing, I combined passbands into a single light curve. The details can be also found in the kernel.
\n\nI'd like to thank the organizers and the community for fantastic opportunity to learn more about the exciting project and to see all the cool techniques used for the problem!
", "messages": [ { "id": 560293, "postDate": "2019-06-25T09:05:39.773Z", "content": "During the competition I got curious about possible ways how to reconstruct representative light curves of hard classes. I was lucky to somehow reconstruct those. Features based on it likely helped me to jump from the 335th place on public LB place to the 182th place on private LB, which seems to be one of the largest jumps in this competition.
\n\nIt was very exciting for me to see how “signatures of supernovae” emerge from noisy data! 😊 I’ve even created a kernel to share some of routines I tried out. However, I made it private almost immediately. Because when gratefully reading about all the great solutions by top-scoring teams I just felt humble and didn’t want to distract anyone from all those fantastic solutions. After half a year I’d like to briefly share a couple of things I did, just in case one day it might be of any practical use to someone in the astroinformatics community.
\n\nThe main idea was to match and then to robustly aggregate light curves of the same class. For instance, after some rescaling and shifting we match two instances of the class 62 like this:\n\nand after aggregation we have an updated representative light curve for the class:\n
and 80 iterations later we have:\n\n
This way I match-aggregated light curves which likely had maximum. Next, I match-aggregated light curves without apparent maximum, where I had to be slightly more careful with rescaling:\n
In the end I've got representative light curves for different classes, and for the first time I saw slight differences in the hard classes. E.g., we dealt with the class 62 in the previous figures. In the figure below we can compare representative light curves of two different classes with the mentioned 62-class light curve:\n
Finally, I computed some features based on the representative light curves. The main idea was to try to match a light curve with every class prototype, looking for the best rescaling (both time and flux scales). The matching was done using either squared error or cosine error (so, either forcing points to be close to each other, or forcing segments to have similar directions). As CPMP advised to write efficient code, I wrote an efficient implementation in Cython.
\n\nIf interested, you can find the implementation with an example of usage in this kernel.
\n\nP.S. I forgot to mention that before starting the match-aggregate thing, I combined passbands into a single light curve. The details can be also found in the kernel.
\n\nI'd like to thank the organizers and the community for fantastic opportunity to learn more about the exciting project and to see all the cool techniques used for the problem!
", "rawMarkdown": "During the competition I got curious about possible ways how to reconstruct representative light curves of hard classes. I was lucky to somehow reconstruct those. Features based on it likely helped me to jump from the 335th place on public LB place to the 182th place on private LB, which seems to be one of the largest jumps in this competition.\n\nIt was very exciting for me to see how “signatures of supernovae” emerge from noisy data! 😊 I’ve even created a kernel to share some of routines I tried out. However, I made it private almost immediately. Because when gratefully reading about all the great solutions by top-scoring teams I just felt humble and didn’t want to distract anyone from all those fantastic solutions. After half a year I’d like to briefly share a couple of things I did, just in case one day it might be of any practical use to someone in the astroinformatics community. \n\nThe main idea was to match and then to robustly aggregate light curves of the same class. For instance, after some rescaling and shifting we match two instances of the class 62 like this:\n\nand after aggregation we have an updated representative light curve for the class:\n\n\nand 80 iterations later we have:\n\n\n\nThis way I match-aggregated light curves which likely had maximum. Next, I match-aggregated light curves without apparent maximum, where I had to be slightly more careful with rescaling:\n\n\nIn the end I've got representative light curves for different classes, and for the first time I saw slight differences in the hard classes. E.g., we dealt with the class 62 in the previous figures. In the figure below we can compare representative light curves of two different classes with the mentioned 62-class light curve:\n\n\nFinally, I computed some features based on the representative light curves. The main idea was to try to match a light curve with every class prototype, looking for the best rescaling (both time and flux scales). The matching was done using either squared error or cosine error (so, either forcing points to be close to each other, or forcing segments to have similar directions). As CPMP advised to write efficient code, I wrote an efficient implementation in Cython.\n\nIf interested, you can find the implementation with an example of usage in this [kernel](https://www.kaggle.com/samusram/representative-light-curves-for-hard-classes).\n\nP.S. I forgot to mention that before starting the match-aggregate thing, I combined passbands into a single light curve. The details can be also found in the [kernel](https://www.kaggle.com/samusram/representative-light-curves-for-hard-classes).\n\n\nI'd like to thank the organizers and the community for fantastic opportunity to learn more about the exciting project and to see all the cool techniques used for the problem!", "votes": 1 } ], "comments": [], "raw_markdown_by_id": { "560293": "During the competition I got curious about possible ways how to reconstruct representative light curves of hard classes. I was lucky to somehow reconstruct those. Features based on it likely helped me to jump from the 335th place on public LB place to the 182th place on private LB, which seems to be one of the largest jumps in this competition.\n\nIt was very exciting for me to see how “signatures of supernovae” emerge from noisy data! 😊 I’ve even created a kernel to share some of routines I tried out. However, I made it private almost immediately. Because when gratefully reading about all the great solutions by top-scoring teams I just felt humble and didn’t want to distract anyone from all those fantastic solutions. After half a year I’d like to briefly share a couple of things I did, just in case one day it might be of any practical use to someone in the astroinformatics community. \n\nThe main idea was to match and then to robustly aggregate light curves of the same class. For instance, after some rescaling and shifting we match two instances of the class 62 like this:\n\nand after aggregation we have an updated representative light curve for the class:\n\n\nand 80 iterations later we have:\n\n\n\nThis way I match-aggregated light curves which likely had maximum. Next, I match-aggregated light curves without apparent maximum, where I had to be slightly more careful with rescaling:\n\n\nIn the end I've got representative light curves for different classes, and for the first time I saw slight differences in the hard classes. E.g., we dealt with the class 62 in the previous figures. In the figure below we can compare representative light curves of two different classes with the mentioned 62-class light curve:\n\n\nFinally, I computed some features based on the representative light curves. The main idea was to try to match a light curve with every class prototype, looking for the best rescaling (both time and flux scales). The matching was done using either squared error or cosine error (so, either forcing points to be close to each other, or forcing segments to have similar directions). As CPMP advised to write efficient code, I wrote an efficient implementation in Cython.\n\nIf interested, you can find the implementation with an example of usage in this [kernel](https://www.kaggle.com/samusram/representative-light-curves-for-hard-classes).\n\nP.S. I forgot to mention that before starting the match-aggregate thing, I combined passbands into a single light curve. The details can be also found in the [kernel](https://www.kaggle.com/samusram/representative-light-curves-for-hard-classes).\n\n\nI'd like to thank the organizers and the community for fantastic opportunity to learn more about the exciting project and to see all the cool techniques used for the problem!" } }