{ "id": 275331, "title": "4th Place Solution Brief Summary : Magic of 1D CNN", "url": "/competitions/g2net-gravitational-wave-detection/writeups/ogogog-jarvislabs-ai-4th-place-solution-brief-summ", "author_name": "", "post_date": "2021-09-30T07:17:40.920Z", "votes": 67, "comment_count": 47, "views": 0, "content": "

Hi all ,
\nFirst of all I want to thank the organisers and kaggle team for organising such a wonderful and interesting competition , we learned a lot . I would like to thank JarvisLabs.ai (a GPU cloud based platform offering modern and extremely easy to launch GPU instances) for helping us during the competition by providing modern GPU cards. The platform enabled us to do multiple experiments rapidly with instant GPU instances. All our models were trained on cloud.jarvislabs.ai GPU instances and this could not have been achieved without them.

\n

It was really a tough fight , we would have liked to have finished in the money but nevertheless we are really happy with our finish. It was lovely to once again team up with @nischaydnk @benihime91
\n@pheadrus and @proletheus

\n

Similar to other teams we also started with constant Q Transforms and 2D CNN architectures , we did a lot of experiments with different preprocessing techniques (bandpass, whitening , denoising AE's , CWT ,etc details to be shared later) , this allowed us to reach top 15x .

\n

Thanks to heng and other kaggler's experimentation we realized how well conv1d and sequence models are working on this data and it also intuitively made sense to us , so we started working on that and a custom 1d cnn architecture forms the main backbone of our solution

\n

1D/Sequence Model Magic

\n

We are really happy to announce that our single custom 1d architecture model scores 0.8838 public LB / 0.8823 private LB and is in the gold zone alone . (Hoping to write a paper around it)

\n

We started with a rather simple Conv1d architecture with just 8 conv1d layers along with a normal 2x linear head , to our surprise it scored really well cv 0.8766 Lb 0.8788 . This encouraged us to experiment more with sequence models , we tried a mix of LSTM's , GRU's , transformers ,etc but were not able to beat the normal conv1d model .

\n

Finally we decided to train deep conv1d model with residuals (similar to resnet) and it worked like a charm , we then changed the head from linear to LSTM and got further boost . Our final model architecture has the following flow :

\n

GW waves numpy array --> horizontal stacking all three to get (1,4096*3) array --> band pass filtering ---> Deep conv1d backbone with residuals --->LSTM head ---> Prediction

\n

We tried GRU , transformer and bert like heads but LSTM worked best

\n

2D Model

\n

Most of our strong 2d models came from @proletheus who was at 15th position back then when we merged . He used a mix of augmentations and good normalization technique that gave us a good amount of boost in 2d models . We used both CQT and CWT based models in our final ensemble .

\n

We mainly used nnAudio CQT1992v2 and CQT2010 during the preprocessing with
\nconfig.

\n
qtransform_params={\"sr\": 2048, \"fmin\": 30, \"fmax\": 400, \"hop_length\": 4, \n\"bins_per_octave\": 12, \"filter_scale\" : 0.3}\n
\n

Sequence of Preprocessing is as follows:
\nNumpy ---> signal tukey ---> band pass filter ---> normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] --->CQT--> Augmentations[coloredNoise and shift]

\n

Torch audiomentations were used to apply augmentations. Colored noise augmentation was done channel wise while shift was applied sample wise.

\n

Please note that this is a small gist of our solution/journey .
\n@benihime91 will be publishing a detailed solution/explanation with code by tomorrow

\n

Thanks for reading

", "messages": [ { "id": "1528791", "postDate": "09/30/2021 00:57:13", "content": "

Hi all ,
\nFirst of all I want to thank the organisers and kaggle team for organising such a wonderful and interesting competition , we learned a lot . I would like to thank JarvisLabs.ai (a GPU cloud based platform offering modern and extremely easy to launch GPU instances) for helping us during the competition by providing modern GPU cards. The platform enabled us to do multiple experiments rapidly with instant GPU instances. All our models were trained on cloud.jarvislabs.ai GPU instances and this could not have been achieved without them.

\n

It was really a tough fight , we would have liked to have finished in the money but nevertheless we are really happy with our finish. It was lovely to once again team up with @nischaydnk @benihime91
\n@pheadrus and @proletheus

\n

Similar to other teams we also started with constant Q Transforms and 2D CNN architectures , we did a lot of experiments with different preprocessing techniques (bandpass, whitening , denoising AE's , CWT ,etc details to be shared later) , this allowed us to reach top 15x .

\n

Thanks to heng and other kaggler's experimentation we realized how well conv1d and sequence models are working on this data and it also intuitively made sense to us , so we started working on that and a custom 1d cnn architecture forms the main backbone of our solution

\n

1D/Sequence Model Magic

\n

We are really happy to announce that our single custom 1d architecture model scores 0.8838 public LB / 0.8823 private LB and is in the gold zone alone . (Hoping to write a paper around it)

\n

We started with a rather simple Conv1d architecture with just 8 conv1d layers along with a normal 2x linear head , to our surprise it scored really well cv 0.8766 Lb 0.8788 . This encouraged us to experiment more with sequence models , we tried a mix of LSTM's , GRU's , transformers ,etc but were not able to beat the normal conv1d model .

\n

Finally we decided to train deep conv1d model with residuals (similar to resnet) and it worked like a charm , we then changed the head from linear to LSTM and got further boost . Our final model architecture has the following flow :

\n

GW waves numpy array --> horizontal stacking all three to get (1,4096*3) array --> band pass filtering ---> Deep conv1d backbone with residuals --->LSTM head ---> Prediction

\n

We tried GRU , transformer and bert like heads but LSTM worked best

\n

2D Model

\n

Most of our strong 2d models came from @proletheus who was at 15th position back then when we merged . He used a mix of augmentations and good normalization technique that gave us a good amount of boost in 2d models . We used both CQT and CWT based models in our final ensemble .

\n

We mainly used nnAudio CQT1992v2 and CQT2010 during the preprocessing with
\nconfig.

\n
qtransform_params={\"sr\": 2048, \"fmin\": 30, \"fmax\": 400, \"hop_length\": 4, \n\"bins_per_octave\": 12, \"filter_scale\" : 0.3}\n
\n

Sequence of Preprocessing is as follows:
\nNumpy ---> signal tukey ---> band pass filter ---> normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] --->CQT--> Augmentations[coloredNoise and shift]

\n

Torch audiomentations were used to apply augmentations. Colored noise augmentation was done channel wise while shift was applied sample wise.

\n

Please note that this is a small gist of our solution/journey .
\n@benihime91 will be publishing a detailed solution/explanation with code by tomorrow

\n

Thanks for reading

", "rawMarkdown": "Hi all ,\nFirst of all I want to thank the organisers and kaggle team for organising such a wonderful and interesting competition , we learned a lot . I would like to thank JarvisLabs.ai (a GPU cloud based platform offering modern and extremely easy to launch GPU instances) for helping us during the competition by providing modern GPU cards. The platform enabled us to do multiple experiments rapidly with instant GPU instances. All our models were trained on cloud.jarvislabs.ai GPU instances and this could not have been achieved without them. \n\nIt was really a tough fight , we would have liked to have finished in the money but nevertheless we are really happy with our finish. It was lovely to once again team up with @nischaydnk @benihime91\n@pheadrus and @proletheus\n\nSimilar to other teams we also started with constant Q Transforms and 2D CNN architectures , we did a lot of experiments with different preprocessing techniques (bandpass, whitening , denoising AE's , CWT ,etc details to be shared later) , this allowed us to reach top 15x .\n\nThanks to heng and other kaggler's experimentation we realized how well conv1d and sequence models are working on this data and it also intuitively made sense to us , so we started working on that and a custom 1d cnn architecture forms the main backbone of our solution\n\n# 1D/Sequence Model Magic\n\nWe are really happy to announce that our single custom 1d architecture model scores 0.8838 public LB / 0.8823 private LB and is in the gold zone alone . (Hoping to write a paper around it)\n\nWe started with a rather simple Conv1d architecture with just 8 conv1d layers along with a normal 2x linear head , to our surprise it scored really well cv 0.8766 Lb 0.8788 . This encouraged us to experiment more with sequence models , we tried a mix of LSTM's , GRU's , transformers ,etc but were not able to beat the normal conv1d model . \n\nFinally we decided to train deep conv1d model with residuals (similar to resnet) and it worked like a charm , we then changed the head from linear to LSTM and got further boost . Our final model architecture has the following flow :\n\nGW waves numpy array --> horizontal stacking all three to get (1,4096*3) array --> band pass filtering ---> Deep conv1d backbone with residuals --->LSTM head ---> Prediction\n\nWe tried GRU , transformer and bert like heads but LSTM worked best\n\n# 2D Model \n\nMost of our strong 2d models came from @proletheus who was at 15th position back then when we merged . He used a mix of augmentations and good normalization technique that gave us a good amount of boost in 2d models . We used both CQT and CWT based models in our final ensemble . \n\nWe mainly used nnAudio CQT1992v2 and CQT2010 during the preprocessing with \nconfig. \n\n```\nqtransform_params={\"sr\": 2048, \"fmin\": 30, \"fmax\": 400, \"hop_length\": 4, \n\"bins_per_octave\": 12, \"filter_scale\" : 0.3}\n```\n\nSequence of Preprocessing is as follows:\nNumpy ---> signal tukey ---> band pass filter ---> normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] --->CQT--> Augmentations[coloredNoise and shift]\n\nTorch audiomentations were used to apply augmentations. Colored noise augmentation was done channel wise while shift was applied sample wise.\n\n\nPlease note that this is a small gist of our solution/journey .\n@benihime91 will be publishing a detailed solution/explanation with code by tomorrow \n\nThanks for reading", "votes": null }, { "id": "1528797", "postDate": "09/30/2021 01:06:56", "content": "

I once thought about using 1DCNN, which is similar to RESNET model, but I think it is unlikely to surpass my existing score, so I didn't put it into practice. As a result, I only got a bronze medal.This result tells me to try all the ideas in the discussion more

", "rawMarkdown": "I once thought about using 1DCNN, which is similar to RESNET model, but I think it is unlikely to surpass my existing score, so I didn't put it into practice. As a result, I only got a bronze medal.This result tells me to try all the ideas in the discussion more", "votes": null }, { "id": "1528805", "postDate": "09/30/2021 01:12:51", "content": "

Big congratulations on getting a gold medal! We were neighbors at the LB almost all the time during the last week or two.

", "rawMarkdown": "Big congratulations on getting a gold medal! We were neighbors at the LB almost all the time during the last week or two.", "votes": null }, { "id": "1528808", "postDate": "09/30/2021 01:13:44", "content": "

You beat us right at the end 😂.

", "rawMarkdown": "You beat us right at the end 😂.", "votes": null }, { "id": "1528810", "postDate": "09/30/2021 01:14:19", "content": "

Congrats @tanulsingh077 + team!

", "rawMarkdown": "Congrats @tanulsingh077 + team!", "votes": null }, { "id": "1528816", "postDate": "09/30/2021 01:23:33", "content": "

It was an honour for us to have competed with you this close and also fun .

", "rawMarkdown": "It was an honour for us to have competed with you this close and also fun .", "votes": null }, { "id": "1528817", "postDate": "09/30/2021 01:23:52", "content": "

Thanks my friend

", "rawMarkdown": "Thanks my friend", "votes": null }, { "id": "1528819", "postDate": "09/30/2021 01:24:33", "content": "

Congrats! Learned a lot from the way you think in the development of Conv1d model.
\nAbout the 2D Model, I wonder how you guys figure out the norm_by=[7.729773e-21,8.228142e-21, 8.750003e-21]? I can only come up with trial & error search.

", "rawMarkdown": "Congrats! Learned a lot from the way you think in the development of Conv1d model.\nAbout the 2D Model, I wonder how you guys figure out the norm_by=[7.729773e-21,8.228142e-21, 8.750003e-21]? I can only come up with trial & error search.", "votes": null }, { "id": "1528823", "postDate": "09/30/2021 01:28:58", "content": "

Congrats, interested in seeing the architecture of your conv1d network. Do you plan to upload to github/publish a kernel?

", "rawMarkdown": "Congrats, interested in seeing the architecture of your conv1d network. Do you plan to upload to github/publish a kernel?", "votes": null }, { "id": "1528824", "postDate": "09/30/2021 01:29:03", "content": "

just run all the images through the bandpass once, then store the highest values.

", "rawMarkdown": "just run all the images through the bandpass once, then store the highest values.", "votes": null }, { "id": "1528838", "postDate": "09/30/2021 01:48:27", "content": "

We might once we've consolidated and clean codes. Might take a few days.

", "rawMarkdown": "We might once we've consolidated and clean codes. Might take a few days.", "votes": null }, { "id": "1529018", "postDate": "09/30/2021 05:12:06", "content": "

i wonder when does skip connections work well. I tried on the top of 1dcnn skip connections across the layers but dint work. This was only modification i thought of making. to 1dcnn..

", "rawMarkdown": "i wonder when does skip connections work well. I tried on the top of 1dcnn skip connections across the layers but dint work. This was only modification i thought of making. to 1dcnn..", "votes": null }, { "id": "1529040", "postDate": "09/30/2021 05:37:18", "content": "

Congratulations on the strong finish, especially the custom 1Dcnn. Looking forward to the paper or code if you ever publish it 👍

", "rawMarkdown": "Congratulations on the strong finish, especially the custom 1Dcnn. Looking forward to the paper or code if you ever publish it 👍", "votes": null }, { "id": "1529174", "postDate": "09/30/2021 07:44:43", "content": "

Congratz !
\nGreat to see you in 4th place @pheadrus @tanulsingh077 !

", "rawMarkdown": "Congratz !\nGreat to see you in 4th place @pheadrus @tanulsingh077 !", "votes": null }, { "id": "1529206", "postDate": "09/30/2021 08:22:14", "content": "

Congrats! I was rooting you guys @pheadrus @tanulsingh077

", "rawMarkdown": "Congrats! I was rooting you guys @pheadrus @tanulsingh077", "votes": null }, { "id": "1529215", "postDate": "09/30/2021 08:27:19", "content": "

Thankyou @onodera . This is very kind of you! :)

", "rawMarkdown": "Thankyou @onodera . This is very kind of you! :)", "votes": null }, { "id": "1529217", "postDate": "09/30/2021 08:28:06", "content": "

Thankyou @theoviel ; your dedication in the last comp rubbed off on us! :)

", "rawMarkdown": "Thankyou @theoviel ; your dedication in the last comp rubbed off on us! :)", "votes": null }, { "id": "1529249", "postDate": "09/30/2021 08:49:17", "content": "

Thanks @theoviel ❤️

", "rawMarkdown": "Thanks @theoviel ❤️", "votes": null }, { "id": "1529250", "postDate": "09/30/2021 08:49:40", "content": "

We will share the code and full solution very soon @nyleve

", "rawMarkdown": "We will share the code and full solution very soon @nyleve", "votes": null }, { "id": "1529251", "postDate": "09/30/2021 08:49:55", "content": "

Huge congrats to you guys! It was really fun exchanging positions over the last few weeks. Looking forward to more of it in the future! 🍻

", "rawMarkdown": "Huge congrats to you guys! It was really fun exchanging positions over the last few weeks. Looking forward to more of it in the future! 🍻", "votes": null }, { "id": "1529253", "postDate": "09/30/2021 08:50:45", "content": "

Haha that's very sweet of you also loved your zebra augs 🔥

", "rawMarkdown": "Haha that's very sweet of you also loved your zebra augs 🔥", "votes": null }, { "id": "1529255", "postDate": "09/30/2021 08:53:52", "content": "

Haha ,congrats to you as well , it was fun to see GOGOGO and OGOGOG together @anjum48 😋

", "rawMarkdown": "Haha ,congrats to you as well , it was fun to see GOGOGO and OGOGOG together @anjum48 😋", "votes": null }, { "id": "1529258", "postDate": "09/30/2021 08:54:54", "content": "

The skip connection idea came from MoA competition , there @nischaydnk has used skip connections with DNNS to get good boost when we tried it here as well

", "rawMarkdown": "The skip connection idea came from MoA competition , there @nischaydnk has used skip connections with DNNS to get good boost when we tried it here as well", "votes": null }, { "id": "1529272", "postDate": "09/30/2021 09:03:09", "content": "

Congrats GOGOGO (>‿◠)✌

", "rawMarkdown": "Congrats GOGOGO (>‿◠)✌", "votes": null }, { "id": "1529275", "postDate": "09/30/2021 09:04:05", "content": "

@tanulsingh077 how u guys computed stats.

", "rawMarkdown": "tanulsingh077 how u guys computed stats.", "votes": null }, { "id": "1529293", "postDate": "09/30/2021 09:17:18", "content": "

Which stats? Normalization stats were calculated based on entire dataset properties, while params were found via experimentation.

", "rawMarkdown": "Which stats? Normalization stats were calculated based on entire dataset properties, while params were found via experimentation.", "votes": null }, { "id": "1529294", "postDate": "09/30/2021 09:18:58", "content": "

Are you talking about normalization values?? We simply took the maximum values of each channel after applying preprocessing until the bandpass filter.

", "rawMarkdown": "Are you talking about normalization values?? We simply took the maximum values of each channel after applying preprocessing until the bandpass filter.", "votes": null }, { "id": "1529298", "postDate": "09/30/2021 09:26:32", "content": "

@anjum48 loved the LB symmetry in our names , and the fact that everytime it was disturbed you guys came back to restore! That was until the Private was disclosed where you turned the tables. :) Well done to you guys.

", "rawMarkdown": "anjum48 loved the LB symmetry in our names , and the fact that everytime it was disturbed you guys came back to restore! That was until the Private was disclosed where you turned the tables. :) Well done to you guys.", "votes": null }, { "id": "1529318", "postDate": "09/30/2021 09:41:30", "content": "

for bandpass you did butter worth filter or passed min and max to cqt

", "rawMarkdown": "for bandpass you did butter worth filter or passed min and max to cqt", "votes": null }, { "id": "1529353", "postDate": "09/30/2021 10:14:21", "content": "

at one time.. i even thought of mixing Squeeze net and 1dcnn.. but dint spend time and efforts :)… good your guys did perfect modelling that worked well together with 2dcnns . Norm was offcourse one trick that worked well to bring people in silver zone there after these differentiated
\ncongrats @benihime91 @nischaydnk @pheadrus @tanulsingh077

", "rawMarkdown": "at one time.. i even thought of mixing Squeeze net and 1dcnn.. but dint spend time and efforts :)... good your guys did perfect modelling that worked well together with 2dcnns . Norm was offcourse one trick that worked well to bring people in silver zone there after these differentiated \ncongrats @benihime91 @nischaydnk @pheadrus @tanulsingh077", "votes": null }, { "id": "1529370", "postDate": "09/30/2021 10:34:43", "content": "

Congrats!

\n

I tried 1d-cnn, but cv is up to around 0.867.

\n

How did you set kernel size ? I found kernel size 65 is more effective than stacking many small size conv1d.
\nAnd, did you implement using tensorflow?

", "rawMarkdown": "Congrats!\n\nI tried 1d-cnn, but cv is up to around ~~0.873~~ 0.867.\n\nHow did you set kernel size ? I found kernel size 65 is more effective than stacking many small size conv1d.\nAnd, did you implement using tensorflow?", "votes": null }, { "id": "1529374", "postDate": "09/30/2021 10:36:09", "content": "

Thanks. We computed multi scale outputs using varying kernel sizes. (1,2,4,8 and so on).

", "rawMarkdown": "Thanks. We computed multi scale outputs using varying kernel sizes. (1,2,4,8 and so on).", "votes": null }, { "id": "1529394", "postDate": "09/30/2021 11:00:04", "content": "

@pheadrus Thank you for replying!

\n
\n

multi scale outputs

\n
\n

Do you mean multi scale is like inception module in GoogLeNet?
\nI want to know how did you define module if you like.

", "rawMarkdown": "pheadrus Thank you for replying!\n\n> multi scale outputs\n\nDo you mean multi scale is like inception module in GoogLeNet?\nI want to know how did you define module if you like.", "votes": null }, { "id": "1529475", "postDate": "09/30/2021 12:40:37", "content": "

Congratulations! When I first started this competition I jumped right into modeling and the first thing I did was try to create a conv1D model as intuitively it made the most sense to me but I achieved poor results. It goes to show that properly understanding the data and utilizing good processing/bandwidth filters will lead to the best results.

\n

Congratulations again.

", "rawMarkdown": "Congratulations! When I first started this competition I jumped right into modeling and the first thing I did was try to create a conv1D model as intuitively it made the most sense to me but I achieved poor results. It goes to show that properly understanding the data and utilizing good processing/bandwidth filters will lead to the best results. \n\nCongratulations again.", "votes": null }, { "id": "1529520", "postDate": "09/30/2021 13:11:50", "content": "

@jaideepvalani Yeah, it was indeed painful for us as well in terms of finding the right architecture. We struggled to improve 1D above 0.878x until the very last week but continued to do experiments as it was giving good results at ensemble.

", "rawMarkdown": "jaideepvalani Yeah, it was indeed painful for us as well in terms of finding the right architecture. We struggled to improve 1D above 0.878x until the very last week but continued to do experiments as it was giving good results at ensemble.", "votes": null }, { "id": "1529592", "postDate": "09/30/2021 14:02:52", "content": "

Huge congratulations on a successful finish and 4th place in this competition!! :)
\n@tanulsingh077 @nischaydnk @pheadrus @benihime91 @proletheus

", "rawMarkdown": "Huge congratulations on a successful finish and 4th place in this competition!! :)\n@tanulsingh077 @nischaydnk @pheadrus @benihime91 @proletheus", "votes": null }, { "id": "1529633", "postDate": "09/30/2021 14:40:00", "content": "

Thankyou @piantic

", "rawMarkdown": "Thankyou @piantic", "votes": null }, { "id": "1530345", "postDate": "10/01/2021 05:37:42", "content": "

Congrats OGOGOG! Thank you for the inspiration for the latter part of our team name

", "rawMarkdown": "Congrats OGOGOG! Thank you for the inspiration for the latter part of our team name", "votes": null }, { "id": "1530353", "postDate": "10/01/2021 05:43:39", "content": "

Thanks @nanthennguyen , I think being persistent was the key here haha

", "rawMarkdown": "Thanks @nanthennguyen , I think being persistent was the key here haha", "votes": null }, { "id": "1530529", "postDate": "10/01/2021 08:01:03", "content": "

Yes,something similar. We are not releasing code for 1D, as we might pursue publication later. A high level diagram is posted here on my LinkedIn post https://www.linkedin.com/posts/rajneesh-tiwari-693894122_kaggle-machinelearning-artificialintelligence-activity-6849269401263386624-jEL4

", "rawMarkdown": "Yes,something similar. We are not releasing code for 1D, as we might pursue publication later. A high level diagram is posted here on my LinkedIn post https://www.linkedin.com/posts/rajneesh-tiwari-693894122_kaggle-machinelearning-artificialintelligence-activity-6849269401263386624-jEL4", "votes": null }, { "id": "1531444", "postDate": "10/02/2021 02:37:44", "content": "

I'm curious what is the boost you got from the LSTM head vs a regular one (pooling+several layers)? I tried to put a transformer layer before pooling in 1D model but didn't get any improvement with that(

", "rawMarkdown": "I'm curious what is the boost you got from the LSTM head vs a regular one (pooling+several layers)? I tried to put a transformer layer before pooling in 1D model but didn't get any improvement with that(", "votes": null }, { "id": "1531649", "postDate": "10/02/2021 08:13:06", "content": "

@iafoss I just calculated fold 0 results with multiple linear layers head and jumped onto Lstm, so couldn't compare LB results. There wasn't a significant boost with LSTM (around 0.0003) in cv compared to a linear head.

", "rawMarkdown": "iafoss I just calculated fold 0 results with multiple linear layers head and jumped onto Lstm, so couldn't compare LB results. There wasn't a significant boost with LSTM (around 0.0003) in cv compared to a linear head.", "votes": null }, { "id": "1532671", "postDate": "10/03/2021 08:48:02", "content": "

Many thanks!

", "rawMarkdown": "Many thanks!", "votes": null }, { "id": "1533141", "postDate": "10/03/2021 17:35:33", "content": "

Thanks

", "rawMarkdown": "Thanks", "votes": null }, { "id": "1534513", "postDate": "10/05/2021 01:54:39", "content": "

1D CNN is all you need!👍
\nCongratulations!

", "rawMarkdown": "1D CNN is all you need!👍\nCongratulations!", "votes": null }, { "id": "1540061", "postDate": "10/10/2021 05:34:13", "content": "

Thanks for sharing.

", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "1559913", "postDate": "10/27/2021 08:08:31", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "votes": null }, { "id": "1617675", "postDate": "12/14/2021 09:04:52", "content": "

Hi, Thanks for the explanation! It is a great work!
\nI am looking forward to see the codes. Is it public now? Thank you so much.

", "rawMarkdown": "Hi, Thanks for the explanation! It is a great work!\nI am looking forward to see the codes. Is it public now? Thank you so much.", "votes": null } ], "comments": [ { "id": 1528797, "author_name": "zhangeng", "author_url": "", "post_date": "09/30/2021 01:06:56", "content": "

I once thought about using 1DCNN, which is similar to RESNET model, but I think it is unlikely to surpass my existing score, so I didn't put it into practice. As a result, I only got a bronze medal.This result tells me to try all the ideas in the discussion more

", "votes": null, "replies": [] }, { "id": 1528805, "author_name": "iafoss", "author_url": "", "post_date": "09/30/2021 01:12:51", "content": "

Big congratulations on getting a gold medal! We were neighbors at the LB almost all the time during the last week or two.

", "votes": null, "replies": [ { "id": 1528808, "author_name": "benihime91", "author_url": "", "post_date": "09/30/2021 01:13:44", "content": "

You beat us right at the end 😂.

", "votes": null, "replies": [] }, { "id": 1528816, "author_name": "tanulsingh077", "author_url": "", "post_date": "09/30/2021 01:23:33", "content": "

It was an honour for us to have competed with you this close and also fun .

", "votes": null, "replies": [] } ] }, { "id": 1528810, "author_name": "authman", "author_url": "", "post_date": "09/30/2021 01:14:19", "content": "

Congrats @tanulsingh077 + team!

", "votes": null, "replies": [ { "id": 1528817, "author_name": "tanulsingh077", "author_url": "", "post_date": "09/30/2021 01:23:52", "content": "

Thanks my friend

", "votes": null, "replies": [] } ] }, { "id": 1528819, "author_name": "playif1", "author_url": "", "post_date": "09/30/2021 01:24:33", "content": "

Congrats! Learned a lot from the way you think in the development of Conv1d model.
\nAbout the 2D Model, I wonder how you guys figure out the norm_by=[7.729773e-21,8.228142e-21, 8.750003e-21]? I can only come up with trial & error search.

", "votes": null, "replies": [ { "id": 1528824, "author_name": "proletheus", "author_url": "", "post_date": "09/30/2021 01:29:03", "content": "

just run all the images through the bandpass once, then store the highest values.

", "votes": null, "replies": [] }, { "id": 1529318, "author_name": "jaideepvalani", "author_url": "", "post_date": "09/30/2021 09:41:30", "content": "

for bandpass you did butter worth filter or passed min and max to cqt

", "votes": null, "replies": [] } ] }, { "id": 1528823, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/30/2021 01:28:58", "content": "

Congrats, interested in seeing the architecture of your conv1d network. Do you plan to upload to github/publish a kernel?

", "votes": null, "replies": [ { "id": 1528838, "author_name": "pheadrus", "author_url": "", "post_date": "09/30/2021 01:48:27", "content": "

We might once we've consolidated and clean codes. Might take a few days.

", "votes": null, "replies": [] } ] }, { "id": 1529018, "author_name": "jaideepvalani", "author_url": "", "post_date": "09/30/2021 05:12:06", "content": "

i wonder when does skip connections work well. I tried on the top of 1dcnn skip connections across the layers but dint work. This was only modification i thought of making. to 1dcnn..

", "votes": null, "replies": [ { "id": 1529258, "author_name": "tanulsingh077", "author_url": "", "post_date": "09/30/2021 08:54:54", "content": "

The skip connection idea came from MoA competition , there @nischaydnk has used skip connections with DNNS to get good boost when we tried it here as well

", "votes": null, "replies": [] }, { "id": 1529353, "author_name": "jaideepvalani", "author_url": "", "post_date": "09/30/2021 10:14:21", "content": "

at one time.. i even thought of mixing Squeeze net and 1dcnn.. but dint spend time and efforts :)… good your guys did perfect modelling that worked well together with 2dcnns . Norm was offcourse one trick that worked well to bring people in silver zone there after these differentiated
\ncongrats @benihime91 @nischaydnk @pheadrus @tanulsingh077

", "votes": null, "replies": [] }, { "id": 1529520, "author_name": "nischaydnk", "author_url": "", "post_date": "09/30/2021 13:11:50", "content": "

@jaideepvalani Yeah, it was indeed painful for us as well in terms of finding the right architecture. We struggled to improve 1D above 0.878x until the very last week but continued to do experiments as it was giving good results at ensemble.

", "votes": null, "replies": [] } ] }, { "id": 1529040, "author_name": "nyleve", "author_url": "", "post_date": "09/30/2021 05:37:18", "content": "

Congratulations on the strong finish, especially the custom 1Dcnn. Looking forward to the paper or code if you ever publish it 👍

", "votes": null, "replies": [ { "id": 1529250, "author_name": "tanulsingh077", "author_url": "", "post_date": "09/30/2021 08:49:40", "content": "

We will share the code and full solution very soon @nyleve

", "votes": null, "replies": [] } ] }, { "id": 1529174, "author_name": "theoviel", "author_url": "", "post_date": "09/30/2021 07:44:43", "content": "

Congratz !
\nGreat to see you in 4th place @pheadrus @tanulsingh077 !

", "votes": null, "replies": [ { "id": 1529217, "author_name": "pheadrus", "author_url": "", "post_date": "09/30/2021 08:28:06", "content": "

Thankyou @theoviel ; your dedication in the last comp rubbed off on us! :)

", "votes": null, "replies": [] }, { "id": 1529249, "author_name": "tanulsingh077", "author_url": "", "post_date": "09/30/2021 08:49:17", "content": "

Thanks @theoviel ❤️

", "votes": null, "replies": [] } ] }, { "id": 1529206, "author_name": "onodera", "author_url": "", "post_date": "09/30/2021 08:22:14", "content": "

Congrats! I was rooting you guys @pheadrus @tanulsingh077

", "votes": null, "replies": [ { "id": 1529215, "author_name": "pheadrus", "author_url": "", "post_date": "09/30/2021 08:27:19", "content": "

Thankyou @onodera . This is very kind of you! :)

", "votes": null, "replies": [] }, { "id": 1529253, "author_name": "tanulsingh077", "author_url": "", "post_date": "09/30/2021 08:50:45", "content": "

Haha that's very sweet of you also loved your zebra augs 🔥

", "votes": null, "replies": [] } ] }, { "id": 1529251, "author_name": "anjum48", "author_url": "", "post_date": "09/30/2021 08:49:55", "content": "

Huge congrats to you guys! It was really fun exchanging positions over the last few weeks. Looking forward to more of it in the future! 🍻

", "votes": null, "replies": [ { "id": 1529255, "author_name": "tanulsingh077", "author_url": "", "post_date": "09/30/2021 08:53:52", "content": "

Haha ,congrats to you as well , it was fun to see GOGOGO and OGOGOG together @anjum48 😋

", "votes": null, "replies": [] }, { "id": 1529272, "author_name": "nischaydnk", "author_url": "", "post_date": "09/30/2021 09:03:09", "content": "

Congrats GOGOGO (>‿◠)✌

", "votes": null, "replies": [] }, { "id": 1529298, "author_name": "pheadrus", "author_url": "", "post_date": "09/30/2021 09:26:32", "content": "

@anjum48 loved the LB symmetry in our names , and the fact that everytime it was disturbed you guys came back to restore! That was until the Private was disclosed where you turned the tables. :) Well done to you guys.

", "votes": null, "replies": [] }, { "id": 1530345, "author_name": "richx86", "author_url": "", "post_date": "10/01/2021 05:37:42", "content": "

Congrats OGOGOG! Thank you for the inspiration for the latter part of our team name

", "votes": null, "replies": [] } ] }, { "id": 1529275, "author_name": "jaideepvalani", "author_url": "", "post_date": "09/30/2021 09:04:05", "content": "

@tanulsingh077 how u guys computed stats.

", "votes": null, "replies": [ { "id": 1529293, "author_name": "pheadrus", "author_url": "", "post_date": "09/30/2021 09:17:18", "content": "

Which stats? Normalization stats were calculated based on entire dataset properties, while params were found via experimentation.

", "votes": null, "replies": [] }, { "id": 1529294, "author_name": "nischaydnk", "author_url": "", "post_date": "09/30/2021 09:18:58", "content": "

Are you talking about normalization values?? We simply took the maximum values of each channel after applying preprocessing until the bandpass filter.

", "votes": null, "replies": [] } ] }, { "id": 1529370, "author_name": "yoshito", "author_url": "", "post_date": "09/30/2021 10:34:43", "content": "

Congrats!

\n

I tried 1d-cnn, but cv is up to around 0.867.

\n

How did you set kernel size ? I found kernel size 65 is more effective than stacking many small size conv1d.
\nAnd, did you implement using tensorflow?

", "votes": null, "replies": [ { "id": 1529374, "author_name": "pheadrus", "author_url": "", "post_date": "09/30/2021 10:36:09", "content": "

Thanks. We computed multi scale outputs using varying kernel sizes. (1,2,4,8 and so on).

", "votes": null, "replies": [] }, { "id": 1529394, "author_name": "yoshito", "author_url": "", "post_date": "09/30/2021 11:00:04", "content": "

@pheadrus Thank you for replying!

\n
\n

multi scale outputs

\n
\n

Do you mean multi scale is like inception module in GoogLeNet?
\nI want to know how did you define module if you like.

", "votes": null, "replies": [] }, { "id": 1530529, "author_name": "pheadrus", "author_url": "", "post_date": "10/01/2021 08:01:03", "content": "

Yes,something similar. We are not releasing code for 1D, as we might pursue publication later. A high level diagram is posted here on my LinkedIn post https://www.linkedin.com/posts/rajneesh-tiwari-693894122_kaggle-machinelearning-artificialintelligence-activity-6849269401263386624-jEL4

", "votes": null, "replies": [] }, { "id": 1532671, "author_name": "yoshito", "author_url": "", "post_date": "10/03/2021 08:48:02", "content": "

Many thanks!

", "votes": null, "replies": [] } ] }, { "id": 1529475, "author_name": "nathennguyen", "author_url": "", "post_date": "09/30/2021 12:40:37", "content": "

Congratulations! When I first started this competition I jumped right into modeling and the first thing I did was try to create a conv1D model as intuitively it made the most sense to me but I achieved poor results. It goes to show that properly understanding the data and utilizing good processing/bandwidth filters will lead to the best results.

\n

Congratulations again.

", "votes": null, "replies": [ { "id": 1530353, "author_name": "tanulsingh077", "author_url": "", "post_date": "10/01/2021 05:43:39", "content": "

Thanks @nanthennguyen , I think being persistent was the key here haha

", "votes": null, "replies": [] } ] }, { "id": 1529592, "author_name": "piantic", "author_url": "", "post_date": "09/30/2021 14:02:52", "content": "

Huge congratulations on a successful finish and 4th place in this competition!! :)
\n@tanulsingh077 @nischaydnk @pheadrus @benihime91 @proletheus

", "votes": null, "replies": [ { "id": 1529633, "author_name": "pheadrus", "author_url": "", "post_date": "09/30/2021 14:40:00", "content": "

Thankyou @piantic

", "votes": null, "replies": [] } ] }, { "id": 1531444, "author_name": "iafoss", "author_url": "", "post_date": "10/02/2021 02:37:44", "content": "

I'm curious what is the boost you got from the LSTM head vs a regular one (pooling+several layers)? I tried to put a transformer layer before pooling in 1D model but didn't get any improvement with that(

", "votes": null, "replies": [ { "id": 1531649, "author_name": "nischaydnk", "author_url": "", "post_date": "10/02/2021 08:13:06", "content": "

@iafoss I just calculated fold 0 results with multiple linear layers head and jumped onto Lstm, so couldn't compare LB results. There wasn't a significant boost with LSTM (around 0.0003) in cv compared to a linear head.

", "votes": null, "replies": [] }, { "id": 1533141, "author_name": "iafoss", "author_url": "", "post_date": "10/03/2021 17:35:33", "content": "

Thanks

", "votes": null, "replies": [] } ] }, { "id": 1534513, "author_name": "ohseokkim", "author_url": "", "post_date": "10/05/2021 01:54:39", "content": "

1D CNN is all you need!👍
\nCongratulations!

", "votes": null, "replies": [] }, { "id": 1540061, "author_name": "saleesh", "author_url": "", "post_date": "10/10/2021 05:34:13", "content": "

Thanks for sharing.

", "votes": null, "replies": [] }, { "id": 1559913, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 08:08:31", "content": "

Hey All,

\n

Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.

\n

Thank you all,

\n

Regards,
\nChris

", "votes": null, "replies": [] }, { "id": 1617675, "author_name": "wuyunan2168", "author_url": "", "post_date": "12/14/2021 09:04:52", "content": "

Hi, Thanks for the explanation! It is a great work!
\nI am looking forward to see the codes. Is it public now? Thank you so much.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1528791": "Hi all ,\nFirst of all I want to thank the organisers and kaggle team for organising such a wonderful and interesting competition , we learned a lot . I would like to thank JarvisLabs.ai (a GPU cloud based platform offering modern and extremely easy to launch GPU instances) for helping us during the competition by providing modern GPU cards. The platform enabled us to do multiple experiments rapidly with instant GPU instances. All our models were trained on cloud.jarvislabs.ai GPU instances and this could not have been achieved without them. \n\nIt was really a tough fight , we would have liked to have finished in the money but nevertheless we are really happy with our finish. It was lovely to once again team up with @nischaydnk @benihime91\n@pheadrus and @proletheus\n\nSimilar to other teams we also started with constant Q Transforms and 2D CNN architectures , we did a lot of experiments with different preprocessing techniques (bandpass, whitening , denoising AE's , CWT ,etc details to be shared later) , this allowed us to reach top 15x .\n\nThanks to heng and other kaggler's experimentation we realized how well conv1d and sequence models are working on this data and it also intuitively made sense to us , so we started working on that and a custom 1d cnn architecture forms the main backbone of our solution\n\n# 1D/Sequence Model Magic\n\nWe are really happy to announce that our single custom 1d architecture model scores 0.8838 public LB / 0.8823 private LB and is in the gold zone alone . (Hoping to write a paper around it)\n\nWe started with a rather simple Conv1d architecture with just 8 conv1d layers along with a normal 2x linear head , to our surprise it scored really well cv 0.8766 Lb 0.8788 . This encouraged us to experiment more with sequence models , we tried a mix of LSTM's , GRU's , transformers ,etc but were not able to beat the normal conv1d model . \n\nFinally we decided to train deep conv1d model with residuals (similar to resnet) and it worked like a charm , we then changed the head from linear to LSTM and got further boost . Our final model architecture has the following flow :\n\nGW waves numpy array --> horizontal stacking all three to get (1,4096*3) array --> band pass filtering ---> Deep conv1d backbone with residuals --->LSTM head ---> Prediction\n\nWe tried GRU , transformer and bert like heads but LSTM worked best\n\n# 2D Model \n\nMost of our strong 2d models came from @proletheus who was at 15th position back then when we merged . He used a mix of augmentations and good normalization technique that gave us a good amount of boost in 2d models . We used both CQT and CWT based models in our final ensemble . \n\nWe mainly used nnAudio CQT1992v2 and CQT2010 during the preprocessing with \nconfig. \n\n```\nqtransform_params={\"sr\": 2048, \"fmin\": 30, \"fmax\": 400, \"hop_length\": 4, \n\"bins_per_octave\": 12, \"filter_scale\" : 0.3}\n```\n\nSequence of Preprocessing is as follows:\nNumpy ---> signal tukey ---> band pass filter ---> normalized by norm_by =[7.729773e-21,8.228142e-21, 8.750003e-21] --->CQT--> Augmentations[coloredNoise and shift]\n\nTorch audiomentations were used to apply augmentations. Colored noise augmentation was done channel wise while shift was applied sample wise.\n\n\nPlease note that this is a small gist of our solution/journey .\n@benihime91 will be publishing a detailed solution/explanation with code by tomorrow \n\nThanks for reading", "1528797": "I once thought about using 1DCNN, which is similar to RESNET model, but I think it is unlikely to surpass my existing score, so I didn't put it into practice. As a result, I only got a bronze medal.This result tells me to try all the ideas in the discussion more", "1528805": "Big congratulations on getting a gold medal! We were neighbors at the LB almost all the time during the last week or two.", "1528808": "You beat us right at the end 😂.", "1528810": "Congrats @tanulsingh077 + team!", "1528816": "It was an honour for us to have competed with you this close and also fun .", "1528817": "Thanks my friend", "1528819": "Congrats! Learned a lot from the way you think in the development of Conv1d model.\nAbout the 2D Model, I wonder how you guys figure out the norm_by=[7.729773e-21,8.228142e-21, 8.750003e-21]? I can only come up with trial & error search.", "1528823": "Congrats, interested in seeing the architecture of your conv1d network. Do you plan to upload to github/publish a kernel?", "1528824": "just run all the images through the bandpass once, then store the highest values.", "1528838": "We might once we've consolidated and clean codes. Might take a few days.", "1529018": "i wonder when does skip connections work well. I tried on the top of 1dcnn skip connections across the layers but dint work. This was only modification i thought of making. to 1dcnn..", "1529040": "Congratulations on the strong finish, especially the custom 1Dcnn. Looking forward to the paper or code if you ever publish it 👍", "1529174": "Congratz !\nGreat to see you in 4th place @pheadrus @tanulsingh077 !", "1529206": "Congrats! I was rooting you guys @pheadrus @tanulsingh077", "1529215": "Thankyou @onodera . This is very kind of you! :)", "1529217": "Thankyou @theoviel ; your dedication in the last comp rubbed off on us! :)", "1529249": "Thanks @theoviel ❤️", "1529250": "We will share the code and full solution very soon @nyleve", "1529251": "Huge congrats to you guys! It was really fun exchanging positions over the last few weeks. Looking forward to more of it in the future! 🍻", "1529253": "Haha that's very sweet of you also loved your zebra augs 🔥", "1529255": "Haha ,congrats to you as well , it was fun to see GOGOGO and OGOGOG together @anjum48 😋", "1529258": "The skip connection idea came from MoA competition , there @nischaydnk has used skip connections with DNNS to get good boost when we tried it here as well", "1529272": "Congrats GOGOGO (>‿◠)✌", "1529275": "tanulsingh077 how u guys computed stats.", "1529293": "Which stats? Normalization stats were calculated based on entire dataset properties, while params were found via experimentation.", "1529294": "Are you talking about normalization values?? We simply took the maximum values of each channel after applying preprocessing until the bandpass filter.", "1529298": "anjum48 loved the LB symmetry in our names , and the fact that everytime it was disturbed you guys came back to restore! That was until the Private was disclosed where you turned the tables. :) Well done to you guys.", "1529318": "for bandpass you did butter worth filter or passed min and max to cqt", "1529353": "at one time.. i even thought of mixing Squeeze net and 1dcnn.. but dint spend time and efforts :)... good your guys did perfect modelling that worked well together with 2dcnns . Norm was offcourse one trick that worked well to bring people in silver zone there after these differentiated \ncongrats @benihime91 @nischaydnk @pheadrus @tanulsingh077", "1529370": "Congrats!\n\nI tried 1d-cnn, but cv is up to around ~~0.873~~ 0.867.\n\nHow did you set kernel size ? I found kernel size 65 is more effective than stacking many small size conv1d.\nAnd, did you implement using tensorflow?", "1529374": "Thanks. We computed multi scale outputs using varying kernel sizes. (1,2,4,8 and so on).", "1529394": "pheadrus Thank you for replying!\n\n> multi scale outputs\n\nDo you mean multi scale is like inception module in GoogLeNet?\nI want to know how did you define module if you like.", "1529475": "Congratulations! When I first started this competition I jumped right into modeling and the first thing I did was try to create a conv1D model as intuitively it made the most sense to me but I achieved poor results. It goes to show that properly understanding the data and utilizing good processing/bandwidth filters will lead to the best results. \n\nCongratulations again.", "1529520": "jaideepvalani Yeah, it was indeed painful for us as well in terms of finding the right architecture. We struggled to improve 1D above 0.878x until the very last week but continued to do experiments as it was giving good results at ensemble.", "1529592": "Huge congratulations on a successful finish and 4th place in this competition!! :)\n@tanulsingh077 @nischaydnk @pheadrus @benihime91 @proletheus", "1529633": "Thankyou @piantic", "1530345": "Congrats OGOGOG! Thank you for the inspiration for the latter part of our team name", "1530353": "Thanks @nanthennguyen , I think being persistent was the key here haha", "1530529": "Yes,something similar. We are not releasing code for 1D, as we might pursue publication later. A high level diagram is posted here on my LinkedIn post https://www.linkedin.com/posts/rajneesh-tiwari-693894122_kaggle-machinelearning-artificialintelligence-activity-6849269401263386624-jEL4", "1531444": "I'm curious what is the boost you got from the LSTM head vs a regular one (pooling+several layers)? I tried to put a transformer layer before pooling in 1D model but didn't get any improvement with that(", "1531649": "iafoss I just calculated fold 0 results with multiple linear layers head and jumped onto Lstm, so couldn't compare LB results. There wasn't a significant boost with LSTM (around 0.0003) in cv compared to a linear head.", "1532671": "Many thanks!", "1533141": "Thanks", "1534513": "1D CNN is all you need!👍\nCongratulations!", "1540061": "Thanks for sharing.", "1559913": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "1617675": "Hi, Thanks for the explanation! It is a great work!\nI am looking forward to see the codes. Is it public now? Thank you so much." }, "source": "meta" }