Really great competition & Congrats to everyone!!!
\n\nI just want to share a very simple rnn baseline (in pytorch), kernel is here(results are not stable and maybe lr schedule should be set carefully) -> https://www.kaggle.com/johnfarrell/plasticc-2018-emb-gru
\n\nwhich only uses 5 raw series: 'mjd', 'flux', 'flux_err', 'detected', 'passband'(embedding dim is 16) & meta features: 'ddf', 'hostgal_photoz', 'hostgal_photoz_err', 'distmod', 'mwebv'. The score result(training.ipynb source file) is oof0.650/pub0.938/pri0.970. And it has very low correlation with stats features so stacking works very well with other models using stats features.
\n\nFinally, I'm really eager to know other better RNN/CNN/?NN solutions, if you have some please share with me! :)
", "messages": [ { "id": "440838", "postDate": "12/18/2018 02:42:07", "content": "Really great competition & Congrats to everyone!!!
\n\nI just want to share a very simple rnn baseline (in pytorch), kernel is here(results are not stable and maybe lr schedule should be set carefully) -> https://www.kaggle.com/johnfarrell/plasticc-2018-emb-gru
\n\nwhich only uses 5 raw series: 'mjd', 'flux', 'flux_err', 'detected', 'passband'(embedding dim is 16) & meta features: 'ddf', 'hostgal_photoz', 'hostgal_photoz_err', 'distmod', 'mwebv'. The score result(training.ipynb source file) is oof0.650/pub0.938/pri0.970. And it has very low correlation with stats features so stacking works very well with other models using stats features.
\n\nFinally, I'm really eager to know other better RNN/CNN/?NN solutions, if you have some please share with me! :)
", "rawMarkdown": "Really great competition & Congrats to everyone!!! \n\nI just want to share a very simple rnn baseline (in pytorch), kernel is here(*results are not stable and maybe lr schedule should be set carefully*) -> https://www.kaggle.com/johnfarrell/plasticc-2018-emb-gru \n\nwhich only uses 5 raw series: 'mjd', 'flux', 'flux\\_err', 'detected', 'passband'(embedding dim is 16) & meta features: 'ddf', 'hostgal\\_photoz', 'hostgal\\_photoz\\_err', 'distmod', 'mwebv'. The score result([training.ipynb source file](https://drive.google.com/file/d/1GSkoyQ-kBU0ZpjQmSQ38X7roFl9gGi0m/view?usp=sharing)) is oof0.650/pub0.938/pri0.970. And it has very low correlation with stats features so stacking works very well with other models using stats features.\n\nFinally, I'm really eager to know other better RNN/CNN/?NN solutions, if you have some please share with me! :)", "votes": null }, { "id": "440867", "postDate": "12/18/2018 03:26:13", "content": "Nice. My LSTM solution performs similarly (1.014 LB). It uses BiDirectional LSTM followed by separable depth-wise convolution layer.
\n\nOne thing I did different was I wrote code that takes the fluxes, and linearly interpolates them so long as the mjd gap<80. This results in \"equispaced\" curves that look more like an 'image' or maybe more appropriately 'sound wave' since we're dealing with 1D convolutions here. The input is more dense though due to this and so it takes foreverrrrrr to run. I also augmented every iteration by adding a normal amount of flux_err to a copy of flux every epoch before performing the interpolation.
", "rawMarkdown": "Nice. My LSTM solution performs similarly (1.014 LB). It uses BiDirectional LSTM followed by separable depth-wise convolution layer.\n\nOne thing I did different was I wrote code that takes the fluxes, and linearly interpolates them so long as the mjd gap<80. This results in \"equispaced\" curves that look more like an 'image' or maybe more appropriately 'sound wave' since we're dealing with 1D convolutions here. The input is more dense though due to this and so it takes foreverrrrrr to run. I also augmented every iteration by adding a normal amount of flux_err to a copy of flux every epoch before performing the interpolation.", "votes": null }, { "id": "440872", "postDate": "12/18/2018 03:46:41", "content": "I also thought about more preprocessing methods like interpolatings but got no time to run in only last 2~3 days so I only applied standardization on 'mjd' (finally, with such setup, 150 epochs X 5 folds training + predicting took ~6.5 Hr with one 1080TI).
\nFor raw (non-equispaced) series, I used 'pack_padded_sequence' in pytorch to process variable lengths in input tensor.
\nI also used adding 'flux_err' random noise to 'flux' in other run but it seemed valid score got worse, maybe some params need to be tuning, though...
Thanks for sharing, and congrats for the result. Maybe this one will be what makes me start using Pytorch ;)
", "rawMarkdown": "Thanks for sharing, and congrats for the result. Maybe this one will be what makes me start using Pytorch ;)", "votes": null }, { "id": "440999", "postDate": "12/18/2018 07:01:44", "content": "Thanks for sharing. Very interesting kernel.i tried training an autoencoder to learn a better representation, but it didn't work out. I found MLP (inspired by densenet architecture) to give good results. You can find the kernel here https://www.kaggle.com/meaninglesslives/a-slightly-better-nn-arch-and-some-tricks
", "rawMarkdown": "Thanks for sharing. Very interesting kernel.i tried training an autoencoder to learn a better representation, but it didn't work out. I found MLP (inspired by densenet architecture) to give good results. You can find the kernel here https://www.kaggle.com/meaninglesslives/a-slightly-better-nn-arch-and-some-tricks", "votes": null }, { "id": "441000", "postDate": "12/18/2018 07:02:08", "content": "Congrats & thanks for your sharings in discussion!!
\nEveryone should try Pytorch ;p
Thanks and congrats!
\nIt's exactly your very early MLP kernel which made me to consider some NN based methods :)
\nThis densenet-ish MLP is really amazing, both accuracy(loss) and speed performance! Thanks for sharing!!
Love the pytorch api, but facebook '>.<
", "rawMarkdown": "Love the pytorch api, but facebook '>.<", "votes": null }, { "id": "441531", "postDate": "12/18/2018 18:52:56", "content": "Augmentation by randomly shifting along time axis improved my score by 0.1. I used cnn before gru so in my case it might be more useful than in the case of just rnn.
", "rawMarkdown": "Augmentation by randomly shifting along time axis improved my score by 0.1. I used cnn before gru so in my case it might be more useful than in the case of just rnn.", "votes": null }, { "id": "441719", "postDate": "12/19/2018 00:39:22", "content": "random shifting sounds interesting! with others' solutions, it seems augmentations in this comp are playing an important role to achieve better result.
", "rawMarkdown": "random shifting sounds interesting! with others' solutions, it seems augmentations in this comp are playing an important role to achieve better result.", "votes": null } ], "comments": [ { "id": 440867, "author_name": "authman", "author_url": "", "post_date": "12/18/2018 03:26:13", "content": "Nice. My LSTM solution performs similarly (1.014 LB). It uses BiDirectional LSTM followed by separable depth-wise convolution layer.
\n\nOne thing I did different was I wrote code that takes the fluxes, and linearly interpolates them so long as the mjd gap<80. This results in \"equispaced\" curves that look more like an 'image' or maybe more appropriately 'sound wave' since we're dealing with 1D convolutions here. The input is more dense though due to this and so it takes foreverrrrrr to run. I also augmented every iteration by adding a normal amount of flux_err to a copy of flux every epoch before performing the interpolation.
", "votes": null, "replies": [ { "id": 440872, "author_name": "johnfarrell", "author_url": "", "post_date": "12/18/2018 03:46:41", "content": "I also thought about more preprocessing methods like interpolatings but got no time to run in only last 2~3 days so I only applied standardization on 'mjd' (finally, with such setup, 150 epochs X 5 folds training + predicting took ~6.5 Hr with one 1080TI).
\nFor raw (non-equispaced) series, I used 'pack_padded_sequence' in pytorch to process variable lengths in input tensor.
\nI also used adding 'flux_err' random noise to 'flux' in other run but it seemed valid score got worse, maybe some params need to be tuning, though...
Thanks for sharing, and congrats for the result. Maybe this one will be what makes me start using Pytorch ;)
", "votes": null, "replies": [ { "id": 441000, "author_name": "johnfarrell", "author_url": "", "post_date": "12/18/2018 07:02:08", "content": "Congrats & thanks for your sharings in discussion!!
\nEveryone should try Pytorch ;p
Love the pytorch api, but facebook '>.<
", "votes": null, "replies": [] } ] }, { "id": 440999, "author_name": "meaninglesslives", "author_url": "", "post_date": "12/18/2018 07:01:44", "content": "Thanks for sharing. Very interesting kernel.i tried training an autoencoder to learn a better representation, but it didn't work out. I found MLP (inspired by densenet architecture) to give good results. You can find the kernel here https://www.kaggle.com/meaninglesslives/a-slightly-better-nn-arch-and-some-tricks
", "votes": null, "replies": [ { "id": 441003, "author_name": "johnfarrell", "author_url": "", "post_date": "12/18/2018 07:09:47", "content": "Thanks and congrats!
\nIt's exactly your very early MLP kernel which made me to consider some NN based methods :)
\nThis densenet-ish MLP is really amazing, both accuracy(loss) and speed performance! Thanks for sharing!!
Augmentation by randomly shifting along time axis improved my score by 0.1. I used cnn before gru so in my case it might be more useful than in the case of just rnn.
", "votes": null, "replies": [ { "id": 441719, "author_name": "johnfarrell", "author_url": "", "post_date": "12/19/2018 00:39:22", "content": "random shifting sounds interesting! with others' solutions, it seems augmentations in this comp are playing an important role to achieve better result.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "440838": "Really great competition & Congrats to everyone!!! \n\nI just want to share a very simple rnn baseline (in pytorch), kernel is here(*results are not stable and maybe lr schedule should be set carefully*) -> https://www.kaggle.com/johnfarrell/plasticc-2018-emb-gru \n\nwhich only uses 5 raw series: 'mjd', 'flux', 'flux\\_err', 'detected', 'passband'(embedding dim is 16) & meta features: 'ddf', 'hostgal\\_photoz', 'hostgal\\_photoz\\_err', 'distmod', 'mwebv'. The score result([training.ipynb source file](https://drive.google.com/file/d/1GSkoyQ-kBU0ZpjQmSQ38X7roFl9gGi0m/view?usp=sharing)) is oof0.650/pub0.938/pri0.970. And it has very low correlation with stats features so stacking works very well with other models using stats features.\n\nFinally, I'm really eager to know other better RNN/CNN/?NN solutions, if you have some please share with me! :)", "440867": "Nice. My LSTM solution performs similarly (1.014 LB). It uses BiDirectional LSTM followed by separable depth-wise convolution layer.\n\nOne thing I did different was I wrote code that takes the fluxes, and linearly interpolates them so long as the mjd gap<80. This results in \"equispaced\" curves that look more like an 'image' or maybe more appropriately 'sound wave' since we're dealing with 1D convolutions here. The input is more dense though due to this and so it takes foreverrrrrr to run. I also augmented every iteration by adding a normal amount of flux_err to a copy of flux every epoch before performing the interpolation.", "440872": "I also thought about more preprocessing methods like interpolatings but got no time to run in only last 2~3 days so I only applied standardization on 'mjd' (finally, with such setup, 150 epochs X 5 folds training + predicting took ~6.5 Hr with one 1080TI). \nFor raw (non-equispaced) series, I used 'pack\\_padded\\_sequence' in pytorch to process variable lengths in input tensor. \nI also used adding 'flux\\_err' random noise to 'flux' in other run but it seemed valid score got worse, maybe some params need to be tuning, though...", "440991": "Thanks for sharing, and congrats for the result. Maybe this one will be what makes me start using Pytorch ;)", "440999": "Thanks for sharing. Very interesting kernel.i tried training an autoencoder to learn a better representation, but it didn't work out. I found MLP (inspired by densenet architecture) to give good results. You can find the kernel here https://www.kaggle.com/meaninglesslives/a-slightly-better-nn-arch-and-some-tricks", "441000": "Congrats & thanks for your sharings in discussion!! \n[Everyone should try Pytorch ;p](https://twitter.com/karpathy/status/868178954032513024)", "441003": "Thanks and congrats! \nIt's exactly your very early MLP kernel which made me to consider some NN based methods :) \nThis densenet-ish MLP is really amazing, both accuracy(loss) and speed performance! Thanks for sharing!!", "441264": "Love the pytorch api, but facebook '>.<", "441531": "Augmentation by randomly shifting along time axis improved my score by 0.1. I used cnn before gru so in my case it might be more useful than in the case of just rnn.", "441719": "random shifting sounds interesting! with others' solutions, it seems augmentations in this comp are playing an important role to achieve better result." }, "source": "meta" }