I prepared 2 datasets based on the code by @mistag (https://www.kaggle.com/mistag/data-preprocessing-with-gwpy) but so far wasn't able to train a working model on those images. Has anyone had success with it?
\nhttps://www.kaggle.com/thedrcat/g2net-train-images-with-gpwy-sample/
\nhttps://www.kaggle.com/thedrcat/g2net-test-imgs-with-gwpy/
Here's an example notebook showing how I prepared the datasets:
\nhttps://www.kaggle.com/thedrcat/data-preprocessing-with-gwpy-1/
I prepared 2 datasets based on the code by @mistag (https://www.kaggle.com/mistag/data-preprocessing-with-gwpy) but so far wasn't able to train a working model on those images. Has anyone had success with it?
\nhttps://www.kaggle.com/thedrcat/g2net-train-images-with-gpwy-sample/
\nhttps://www.kaggle.com/thedrcat/g2net-test-imgs-with-gwpy/
Here's an example notebook showing how I prepared the datasets:
\nhttps://www.kaggle.com/thedrcat/data-preprocessing-with-gwpy-1/
I started by using GWPy and the qtransforms, however I found this took too long to transform all the signals and the images were too large to store.
\nI then tried nnAudio which was 100x faster at the qtransfrom. I found I could generate the qtranfrom on the fly for the model.
\nI this notebook was a very helpful starter
\nhttps://www.kaggle.com/mrigendraagrawal/tf-g2net-eda-and-starter
It shows how to generate qtransfroms using nnAudio and the Keras model is very easy to understand.
", "rawMarkdown": "I started by using GWPy and the qtransforms, however I found this took too long to transform all the signals and the images were too large to store. \n\nI then tried nnAudio which was 100x faster at the qtransfrom. I found I could generate the qtranfrom on the fly for the model. \n\nI this notebook was a very helpful starter\nhttps://www.kaggle.com/mrigendraagrawal/tf-g2net-eda-and-starter\n\nIt shows how to generate qtransfroms using nnAudio and the Keras model is very easy to understand. ", "votes": 5, "replies": [ { "id": 1383416, "postDate": "2021-07-10T20:04:41.997Z", "content": "Good point to mention nnAudio and its fast computation, thank you.
", "rawMarkdown": "Good point to mention nnAudio and its fast computation, thank you.", "votes": 1 }, { "id": 1471576, "postDate": "2021-08-14T09:58:23.953Z", "content": "Indeed, nnAudio is a great option here. I guess once you find a good enough transformation, you can save it and make a dataset as suggested here.
", "rawMarkdown": "Indeed, nnAudio is a great option here. I guess once you find a good enough transformation, you can save it and make a dataset as suggested here." } ] }, { "id": 1380986, "postDate": "2021-07-08T13:47:49.443Z", "content": "My images did not come out like that. Maybe there's a mistake in the processing?
", "rawMarkdown": "My images did not come out like that. Maybe there's a mistake in the processing?", "votes": 1 }, { "id": 1375393, "postDate": "2021-07-04T07:16:25.060Z", "content": "I didn't use the mentioned data set but I trained resnet18 (just to check) on spectrogram after applying the Q-Transform (using PyCBC) on randomly selected 80K training signals from data set and got 0.809 LB.
", "rawMarkdown": "I didn't use the mentioned data set but I trained resnet18 (just to check) on spectrogram after applying the Q-Transform (using PyCBC) on randomly selected 80K training signals from data set and got 0.809 LB.", "votes": 2 }, { "id": 1561179, "postDate": "2021-10-27T12:15:02.433Z", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
My quick approach of using this dataset for training. I did this to have a quick experiment to learn about signal processing using images. The training notebook is here.
\nI didn't get a good LB score though but Learned a lot of things while doing this experiment(I never knew that deep learning on time series data can be done using time series as images.)
\nAny suggestion on approaches which can be experimented with to increase the performance ?
\nAlso, I found that training on the entire dataset vs training only on a subset of 20,000 images yields similar performance while using resnet34.
", "rawMarkdown": "My quick approach of using this dataset for training. I did this to have a quick experiment to learn about signal processing using images. The training notebook is [here](https://www.kaggle.com/sapal6/g2gwd-base-model-fastai-resnet).\n\nI didn't get a good LB score though but Learned a lot of things while doing this experiment(I never knew that deep learning on time series data can be done using time series as images.) \n\nAny suggestion on approaches which can be experimented with to increase the performance ?\n\nAlso, I found that training on the entire dataset vs training only on a subset of 20,000 images yields similar performance while using resnet34." }, { "id": 1478795, "postDate": "2021-08-18T06:32:03.853Z", "content": "This is a great work. I tried to do something similar i.e. converting the q transforms into images(only a sample of data). However, I converted each q transform into a separate image. I was not able to test my trained model on test data as my kernel was taking too long to convert the the entire test set into images(even with parallel processing) . Due to this the kernel was getting terminated by kaggle.
\nI will try with these dataset and post my code soon.
", "rawMarkdown": "This is a great work. I tried to do something similar i.e. converting the q transforms into images(only a sample of data). However, I converted each q transform into a separate image. I was not able to test my trained model on test data as my kernel was taking too long to convert the the entire test set into images(even with parallel processing) . Due to this the kernel was getting terminated by kaggle.\n\nI will try with these dataset and post my code soon." }, { "id": 1381509, "postDate": "2021-07-09T03:59:54.593Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1374620, "postDate": "2021-07-03T13:12:05.767Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1381506, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-07-09T03:57:57.040000", "content": "I started by using GWPy and the qtransforms, however I found this took too long to transform all the signals and the images were too large to store.
\nI then tried nnAudio which was 100x faster at the qtransfrom. I found I could generate the qtranfrom on the fly for the model.
\nI this notebook was a very helpful starter
\nhttps://www.kaggle.com/mrigendraagrawal/tf-g2net-eda-and-starter
It shows how to generate qtransfroms using nnAudio and the Keras model is very easy to understand.
", "votes": 5, "replies": [ { "id": 1383416, "author_name": "Ali Abdin", "author_url": "", "post_date": "2021-07-10T20:04:41.997000", "content": "Good point to mention nnAudio and its fast computation, thank you.
", "votes": 1, "replies": [] }, { "id": 1471576, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-08-14T09:58:23.953000", "content": "Indeed, nnAudio is a great option here. I guess once you find a good enough transformation, you can save it and make a dataset as suggested here.
", "votes": 0, "replies": [] } ] }, { "id": 1380986, "author_name": "Henry Zhang", "author_url": "", "post_date": "2021-07-08T13:47:49.443000", "content": "My images did not come out like that. Maybe there's a mistake in the processing?
", "votes": 1, "replies": [] }, { "id": 1375393, "author_name": "BaAis", "author_url": "", "post_date": "2021-07-04T07:16:25.060000", "content": "I didn't use the mentioned data set but I trained resnet18 (just to check) on spectrogram after applying the Q-Transform (using PyCBC) on randomly selected 80K training signals from data set and got 0.809 LB.
", "votes": 2, "replies": [] }, { "id": 1561179, "author_name": "ChristopherZerafa", "author_url": "", "post_date": "2021-10-27T12:15:02.433000", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
My quick approach of using this dataset for training. I did this to have a quick experiment to learn about signal processing using images. The training notebook is here.
\nI didn't get a good LB score though but Learned a lot of things while doing this experiment(I never knew that deep learning on time series data can be done using time series as images.)
\nAny suggestion on approaches which can be experimented with to increase the performance ?
\nAlso, I found that training on the entire dataset vs training only on a subset of 20,000 images yields similar performance while using resnet34.
", "votes": 0, "replies": [] }, { "id": 1478795, "author_name": "sapal6", "author_url": "", "post_date": "2021-08-18T06:32:03.853000", "content": "This is a great work. I tried to do something similar i.e. converting the q transforms into images(only a sample of data). However, I converted each q transform into a separate image. I was not able to test my trained model on test data as my kernel was taking too long to convert the the entire test set into images(even with parallel processing) . Due to this the kernel was getting terminated by kaggle.
\nI will try with these dataset and post my code soon.
", "votes": 0, "replies": [] }, { "id": 1381509, "author_name": "", "author_url": "", "post_date": "2021-07-09T03:59:54.593000", "content": "", "votes": 0, "replies": [] }, { "id": 1374620, "author_name": "", "author_url": "", "post_date": "2021-07-03T13:12:05.767000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1374264": "I prepared 2 datasets based on the code by @mistag (https://www.kaggle.com/mistag/data-preprocessing-with-gwpy) but so far wasn't able to train a working model on those images. Has anyone had success with it? \n\nhttps://www.kaggle.com/thedrcat/g2net-train-images-with-gpwy-sample/\nhttps://www.kaggle.com/thedrcat/g2net-test-imgs-with-gwpy/\n\nHere's an example notebook showing how I prepared the datasets: \nhttps://www.kaggle.com/thedrcat/data-preprocessing-with-gwpy-1/", "1381506": "I started by using GWPy and the qtransforms, however I found this took too long to transform all the signals and the images were too large to store. \n\nI then tried nnAudio which was 100x faster at the qtransfrom. I found I could generate the qtranfrom on the fly for the model. \n\nI this notebook was a very helpful starter\nhttps://www.kaggle.com/mrigendraagrawal/tf-g2net-eda-and-starter\n\nIt shows how to generate qtransfroms using nnAudio and the Keras model is very easy to understand. ", "1380986": "My images did not come out like that. Maybe there's a mistake in the processing?", "1375393": "I didn't use the mentioned data set but I trained resnet18 (just to check) on spectrogram after applying the Q-Transform (using PyCBC) on randomly selected 80K training signals from data set and got 0.809 LB.", "1561179": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "1484066": "My quick approach of using this dataset for training. I did this to have a quick experiment to learn about signal processing using images. The training notebook is [here](https://www.kaggle.com/sapal6/g2gwd-base-model-fastai-resnet).\n\nI didn't get a good LB score though but Learned a lot of things while doing this experiment(I never knew that deep learning on time series data can be done using time series as images.) \n\nAny suggestion on approaches which can be experimented with to increase the performance ?\n\nAlso, I found that training on the entire dataset vs training only on a subset of 20,000 images yields similar performance while using resnet34.", "1478795": "This is a great work. I tried to do something similar i.e. converting the q transforms into images(only a sample of data). However, I converted each q transform into a separate image. I was not able to test my trained model on test data as my kernel was taking too long to convert the the entire test set into images(even with parallel processing) . Due to this the kernel was getting terminated by kaggle.\n\nI will try with these dataset and post my code soon.", "1381509": "", "1374620": "" } }