First of all, my sincere thanks to the organizers for hosting such an interesting competition. I really enjoyed playing around with the dataset and experimenting with various models. I will describe one of my interesting experiments using semi-supervised learning below. The presented solution scores 0.851 on LB. I came up with this approach a few days ago, so further improvement is definitely possible.
\nContext:
\nI was really bored with fitting CNNs on CQT spectrograms using supervised learning. So, i decided to try a semi-supervised training approach. One of the main components of semi-supervised training is the augmentation scheme.
\nAs many have pointed out, it is difficult to come up with a good augmentation for this dataset. While thinking how to augment the data such that the signal characteristic is preserved, I realized that we already have an augmented dataset :-)
\nWhenever there is a gravitational wave, all three detectors must detect it. So, we can imagine that we have access to augmented versions of the same signal!!
Training details:
\nI used the recently proposed Barlow Twins method for semi-supervised training. I would highly recommend going through the paper to understand the details. The basic idea is to have two models which see different versions of data (in our case the gravitational wave signal) and use the barlow twin's objective function to learn embeddings. While training on imagenet, people generally use cropping, flipping, blurring, random contrast etc
\nto create different versions of the same image. In our case, we can feed the data from Hanford/Livingston into Net 1 and data from the Virgo detector into Net 2. Since the noise in Virgo detector is quite different from Hanford/Livingston, we can imagine it as an augmented sample of the same underlying signal. Most semi-supervised work is focused on images. So, usually Net 1 and Net 2 are 2D CNN like resnet and the inputs are images. However, for faster training I decided to train using waveforms. Since the input was 1D so as an initial experiment, I tried the publicly available 1D CNN. Outputs of both the networks are normalized and along the batch dimension and we calculate the invariance and redundancy reduction terms. Using these terms,
\nwe get the final loss used for training the networks. In the original paper, they used LARS optimizer but I found that AdamW also works. (I haven't tried LARS optimizer yet, so maybe it will work even better.)
Evaluation details:
\nAfter training both CNNs, we are able to get embeddings of the input waveform. We need to train a Fully connected (FC) layer to make final prediction. We obtain embeddings for all three detectors. Embeddings for Hanford/Livingston are obtained from Net 1 and Net 2 provides the
\nVirgo embedding. The concatenation of all the embeddings is used as input to the FC layer.
\nWe freeze the backbone layer weights (cnn1 and cnn2) and train the FC layer only on a subset of training dataset (i used 10%). It is sufficient to train FC layer for 4-5 epochs. Once the FC training is complete, we can use the model to detect GW.
Things to try:
\nI am planning to share the training & evaluation code later.
", "messages": [ { "id": "1529186", "postDate": "09/30/2021 08:02:55", "content": "First of all, my sincere thanks to the organizers for hosting such an interesting competition. I really enjoyed playing around with the dataset and experimenting with various models. I will describe one of my interesting experiments using semi-supervised learning below. The presented solution scores 0.851 on LB. I came up with this approach a few days ago, so further improvement is definitely possible.
\nContext:
\nI was really bored with fitting CNNs on CQT spectrograms using supervised learning. So, i decided to try a semi-supervised training approach. One of the main components of semi-supervised training is the augmentation scheme.
\nAs many have pointed out, it is difficult to come up with a good augmentation for this dataset. While thinking how to augment the data such that the signal characteristic is preserved, I realized that we already have an augmented dataset :-)
\nWhenever there is a gravitational wave, all three detectors must detect it. So, we can imagine that we have access to augmented versions of the same signal!!
Training details:
\nI used the recently proposed Barlow Twins method for semi-supervised training. I would highly recommend going through the paper to understand the details. The basic idea is to have two models which see different versions of data (in our case the gravitational wave signal) and use the barlow twin's objective function to learn embeddings. While training on imagenet, people generally use cropping, flipping, blurring, random contrast etc
\nto create different versions of the same image. In our case, we can feed the data from Hanford/Livingston into Net 1 and data from the Virgo detector into Net 2. Since the noise in Virgo detector is quite different from Hanford/Livingston, we can imagine it as an augmented sample of the same underlying signal. Most semi-supervised work is focused on images. So, usually Net 1 and Net 2 are 2D CNN like resnet and the inputs are images. However, for faster training I decided to train using waveforms. Since the input was 1D so as an initial experiment, I tried the publicly available 1D CNN. Outputs of both the networks are normalized and along the batch dimension and we calculate the invariance and redundancy reduction terms. Using these terms,
\nwe get the final loss used for training the networks. In the original paper, they used LARS optimizer but I found that AdamW also works. (I haven't tried LARS optimizer yet, so maybe it will work even better.)
Evaluation details:
\nAfter training both CNNs, we are able to get embeddings of the input waveform. We need to train a Fully connected (FC) layer to make final prediction. We obtain embeddings for all three detectors. Embeddings for Hanford/Livingston are obtained from Net 1 and Net 2 provides the
\nVirgo embedding. The concatenation of all the embeddings is used as input to the FC layer.
\nWe freeze the backbone layer weights (cnn1 and cnn2) and train the FC layer only on a subset of training dataset (i used 10%). It is sufficient to train FC layer for 4-5 epochs. Once the FC training is complete, we can use the model to detect GW.
Things to try:
\nI am planning to share the training & evaluation code later.
", "rawMarkdown": "First of all, my sincere thanks to the organizers for hosting such an interesting competition. I really enjoyed playing around with the dataset and experimenting with various models. I will describe one of my interesting experiments using semi-supervised learning below. The presented solution scores 0.851 on LB. I came up with this approach a few days ago, so further improvement is definitely possible.\n\n**Context:**\nI was really bored with fitting CNNs on CQT spectrograms using supervised learning. So, i decided to try a semi-supervised training approach. One of the main components of semi-supervised training is the augmentation scheme. \nAs many have pointed out, it is difficult to come up with a good augmentation for this dataset. While thinking how to augment the data such that the signal characteristic is preserved, I realized that we already have an augmented dataset :-)\nWhenever there is a gravitational wave, all three detectors must detect it. So, we can imagine that we have access to augmented versions of the same signal!!\n\n**Training details:**\nI used the recently proposed Barlow Twins method for semi-supervised training. I would highly recommend going through the [paper](https://arxiv.org/pdf/2103.03230.pdf) to understand the details. The basic idea is to have two models which see different versions of data (in our case the gravitational wave signal) and use the barlow twin's objective function to learn embeddings. While training on imagenet, people generally use cropping, flipping, blurring, random contrast etc\nto create different versions of the same image. In our case, we can feed the data from Hanford/Livingston into Net 1 and data from the Virgo detector into Net 2. Since the noise in Virgo detector is quite different from Hanford/Livingston, we can imagine it as an augmented sample of the same underlying signal. Most semi-supervised work is focused on images. So, usually Net 1 and Net 2 are 2D CNN like resnet and the inputs are images. However, for faster training I decided to train using waveforms. Since the input was 1D so as an initial experiment, I tried the [publicly available 1D CNN](https://www.kaggle.com/scaomath/g2net-1d-cnn-gem-pool-pytorch-train-inference). Outputs of both the networks are normalized and along the batch dimension and we calculate the invariance and redundancy reduction terms. Using these terms,\nwe get the final loss used for training the networks. In the original paper, they used LARS optimizer but I found that AdamW also works. (I haven't tried LARS optimizer yet, so maybe it will work even better.)\n\n**Evaluation details:**\nAfter training both CNNs, we are able to get embeddings of the input waveform. We need to train a Fully connected (FC) layer to make final prediction. We obtain embeddings for all three detectors. Embeddings for Hanford/Livingston are obtained from Net 1 and Net 2 provides the\nVirgo embedding. The concatenation of all the embeddings is used as input to the FC layer. \nWe freeze the backbone layer weights (cnn1 and cnn2) and train the FC layer only on a subset of training dataset (i used 10%). It is sufficient to train FC layer for 4-5 epochs. Once the FC training is complete, we can use the model to detect GW.\n\n**Things to try:**\n- Try this semi-supervised approach with 2D CNN model. In this case, instead of using direct waveforms we can use the extracted CQT features.\n- Different 1d CNN architectures.\n- Augmentations\n\nI am planning to share the training & evaluation code later.", "votes": null }, { "id": "1529257", "postDate": "09/30/2021 08:54:38", "content": "Nice !! Thank you for sharing. Looking forward to the training and evaluation code as these are new approach for me 👍
", "rawMarkdown": "Nice !! Thank you for sharing. Looking forward to the training and evaluation code as these are new approach for me 👍", "votes": null }, { "id": "1529687", "postDate": "09/30/2021 15:42:04", "content": "Wow, quite interesting approach…nice try!!
\nDid you also try other SSL approaches? Also, did you try putting two augmented samples into Siamese architecture instead of comparing the embeddings of Virgo vs Hanford/Livingston?
Thanks. I haven't tried other approaches like SimCLR, BYOL yet. It would be interesting to try them!
\nBarlow twins performs very similar to these approaches on ImageNet (refer Table 1 of paper), so I am expecting them to perform similarly here. But it will be interesting to try!
\nI tried adding noise from pycbc. I am not sure to what extent it's helpful. I haven't tried any adding any other augmentation yet. The performance may improve with suitable augmentation.
UPDATE:
\nIf anyone is interested they can find the notebook here https://www.kaggle.com/meaninglesslives/self-supervised-method-for-gravitation-wave-det
\nThe notebook is complete in itself. It first trains the backbone model which takes the waveforms as input and outputs embedding for the waveforms. Next the backbone weights are frozen and we train a FC layer. Finally we do test set predictions.
Github Repo: https://github.com/sidml/Self-Supervised-Learning-for-Gravitational-Waves
", "rawMarkdown": "UPDATE:\nIf anyone is interested they can find the notebook here https://www.kaggle.com/meaninglesslives/self-supervised-method-for-gravitation-wave-det\nThe notebook is complete in itself. It first trains the backbone model which takes the waveforms as input and outputs embedding for the waveforms. Next the backbone weights are frozen and we train a FC layer. Finally we do test set predictions.\n\nGithub Repo: https://github.com/sidml/Self-Supervised-Learning-for-Gravitational-Waves", "votes": null }, { "id": "1559904", "postDate": "10/27/2021 08:07:38", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Nice !! Thank you for sharing. Looking forward to the training and evaluation code as these are new approach for me 👍
", "votes": null, "replies": [] }, { "id": 1529687, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "09/30/2021 15:42:04", "content": "Wow, quite interesting approach…nice try!!
\nDid you also try other SSL approaches? Also, did you try putting two augmented samples into Siamese architecture instead of comparing the embeddings of Virgo vs Hanford/Livingston?
Thanks. I haven't tried other approaches like SimCLR, BYOL yet. It would be interesting to try them!
\nBarlow twins performs very similar to these approaches on ImageNet (refer Table 1 of paper), so I am expecting them to perform similarly here. But it will be interesting to try!
\nI tried adding noise from pycbc. I am not sure to what extent it's helpful. I haven't tried any adding any other augmentation yet. The performance may improve with suitable augmentation.
UPDATE:
\nIf anyone is interested they can find the notebook here https://www.kaggle.com/meaninglesslives/self-supervised-method-for-gravitation-wave-det
\nThe notebook is complete in itself. It first trains the backbone model which takes the waveforms as input and outputs embedding for the waveforms. Next the backbone weights are frozen and we train a FC layer. Finally we do test set predictions.
Github Repo: https://github.com/sidml/Self-Supervised-Learning-for-Gravitational-Waves
", "votes": null, "replies": [] }, { "id": 1559904, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:07:38", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris