I've wondered about using SIREN here to find a data representation - or rather the initial layers of their NN and the initialization method (the latter being one of the big tricks they came up with). It seems to me like this and similar ideas for getting cyclic activation functions to work could be a good idea here, with the motivation of course being that this should enable neural networks to learn something similar to traditional signal processing techniques, but with all the flexibility of neural networks. Of course, all that flexibility might require regularization.
\nAn alternative would be to do what is often done in audio data (the example I link uses a VGG type architecture, but resnet is likely better): convert to a spectogram and use a CNN on the spectrogram. That approach is of course already proven (at least in audio), while SIREN is very, very new. With a spectrogram approach, perhaps one could even pre-train on spectrograms generated by something else (perhaps even shifting frequencies to match the present data)?!
", "messages": [ { "id": "1064046", "postDate": "10/29/2020 16:23:24", "content": "I've wondered about using SIREN here to find a data representation - or rather the initial layers of their NN and the initialization method (the latter being one of the big tricks they came up with). It seems to me like this and similar ideas for getting cyclic activation functions to work could be a good idea here, with the motivation of course being that this should enable neural networks to learn something similar to traditional signal processing techniques, but with all the flexibility of neural networks. Of course, all that flexibility might require regularization.
\nAn alternative would be to do what is often done in audio data (the example I link uses a VGG type architecture, but resnet is likely better): convert to a spectogram and use a CNN on the spectrogram. That approach is of course already proven (at least in audio), while SIREN is very, very new. With a spectrogram approach, perhaps one could even pre-train on spectrograms generated by something else (perhaps even shifting frequencies to match the present data)?!
", "rawMarkdown": "I've wondered about using [SIREN](https://vsitzmann.github.io/siren/) here to find a data representation - or rather the initial layers of their NN and the initialization method (the latter being one of the big tricks they came up with). It seems to me like this and similar ideas for getting cyclic activation functions to work could be a good idea here, with the motivation of course being that this should enable neural networks to learn something similar to traditional signal processing techniques, but with all the flexibility of neural networks. Of course, all that flexibility might require regularization.\n\nAn alternative would be to do what is [often done](https://github.com/tensorflow/models/tree/master/research/audioset/vggish) in audio data (the example I link uses a VGG type architecture, but resnet is likely better): convert to a spectogram and use a CNN on the spectrogram. That approach is of course already proven (at least in audio), while SIREN is very, very new. With a spectrogram approach, perhaps one could even pre-train on spectrograms generated by something else (perhaps even shifting frequencies to match the present data)?!", "votes": null }, { "id": "1064234", "postDate": "10/29/2020 22:01:13", "content": "I like the idea of SIREN's ability to have priors as well as some of the architecture. It would certainly be interesting if you could use it effectively with all of the lost sensor data.
", "rawMarkdown": "I like the idea of SIREN's ability to have priors as well as some of the architecture. It would certainly be interesting if you could use it effectively with all of the lost sensor data.", "votes": null } ], "comments": [ { "id": 1064234, "author_name": "eladwar", "author_url": "", "post_date": "10/29/2020 22:01:13", "content": "I like the idea of SIREN's ability to have priors as well as some of the architecture. It would certainly be interesting if you could use it effectively with all of the lost sensor data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1064046": "I've wondered about using [SIREN](https://vsitzmann.github.io/siren/) here to find a data representation - or rather the initial layers of their NN and the initialization method (the latter being one of the big tricks they came up with). It seems to me like this and similar ideas for getting cyclic activation functions to work could be a good idea here, with the motivation of course being that this should enable neural networks to learn something similar to traditional signal processing techniques, but with all the flexibility of neural networks. Of course, all that flexibility might require regularization.\n\nAn alternative would be to do what is [often done](https://github.com/tensorflow/models/tree/master/research/audioset/vggish) in audio data (the example I link uses a VGG type architecture, but resnet is likely better): convert to a spectogram and use a CNN on the spectrogram. That approach is of course already proven (at least in audio), while SIREN is very, very new. With a spectrogram approach, perhaps one could even pre-train on spectrograms generated by something else (perhaps even shifting frequencies to match the present data)?!", "1064234": "I like the idea of SIREN's ability to have priors as well as some of the architecture. It would certainly be interesting if you could use it effectively with all of the lost sensor data." }, "source": "meta" }