![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/0a47af6a917c_original.png?raw=true\")
Hi fellow Kagglers,
\nAs I am sure most of us would have seen examples as shown below where there is a constant band across a particular frequency in both the ON and OFF scenarios.
\n
So technically that is not adding any additional useful information to the signal and is most likely caused by a consistent source of Radio Frequency Interference (RFI). It also looks similar to some of the positive example as illustrated below, it might be a source of confusion/contradiction for the model.
\n![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/0b57c2f1504d.png?raw=true\")
How to get rid of such bands?
\nWe already know from the Data Description that:-
\n\n\nOne method we use to isolate candidate techno signatures from RFI is to look for signals that appear to be coming from particular positions on the sky. Typically we do this by alternating observations of our primary target star with observations of three nearby stars: 5 minutes on star “A”, then 5 minutes on star “B”, then back to star “A” for 5 minutes, then “C”, then back to “A”, then finishing with 5 minutes on star “D”.
\n
This gives us two possibilities:-
\n1. Taking difference between even and odd channels
\nTaking scalar difference between subsequent even and odd channels will help to eliminate bands which are exactly same in both the ON and OFF channels.
\nExample:-
\n![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/difference_only.png?raw=true\")
2. Comparing NMF of ON and OFF images
\nIn this method we decompose both the ON and OFF images into their base matrices W and H whose product approximates the ON/OFF matrix respectively. Then we reconstruct the noise matrix by multiplying the constituent matrix W of ON snippet with the matrix H of OFF snippet. This noise matrix is then subtracted from the ON snippet to get a clean representation of the signal.
\nThis basically recreates the features which are unique to the ON and OFF snippets while eliminating anything in common.
\nExample:-
\n![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/nmf_example_3.png?raw=true\")
The function used to create these Images can be found in this Notebook
\nAssumption:-
\nWe can do this as it is clarified by @stevecroft in this discussion thread that the observations are taken sequentially one after the other and I am assuming the external sources of noise remain the same within that period.
Some more examples can be found below:-
\n![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/nmf_example_2.png?raw=true\")
![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/nmf_example_4.png?raw=true\")
![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/nmf_example_6.png?raw=true\")
![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/nmf_example_9.png?raw=true\")
![\"\"](\"https://github.com/mrout94/Corona_Virus_Spread_Timelapse/blob/master/img_temp/nmf_example_8.png?raw=true\")
The function used to create these Images can be found in this Notebook
\nThings that did not work:-
\nHope you find this useful. 😊
\nUPDATE:-
\nAs clarified by @yuhongc in the comments below:-
\n", "messages": [ { "id": 1347466, "postDate": "2021-06-13T08:22:00.113Z", "content": "The vertical bright line is what we call a DC spike, and is an artifact from the way we convert the raw signal from the telescope into spectrograms. We left them in because we wanted the data to be as realistic as possible, and it happens to be one of the simplest forms of interference (although technically not RFI).
\n
Hi fellow Kagglers,
\nAs I am sure most of us would have seen examples as shown below where there is a constant band across a particular frequency in both the ON and OFF scenarios.
\n
So technically that is not adding any additional useful information to the signal and is most likely caused by a consistent source of Radio Frequency Interference (RFI). It also looks similar to some of the positive example as illustrated below, it might be a source of confusion/contradiction for the model.
\n
How to get rid of such bands?
\nWe already know from the Data Description that:-
\n\n\nOne method we use to isolate candidate techno signatures from RFI is to look for signals that appear to be coming from particular positions on the sky. Typically we do this by alternating observations of our primary target star with observations of three nearby stars: 5 minutes on star “A”, then 5 minutes on star “B”, then back to star “A” for 5 minutes, then “C”, then back to “A”, then finishing with 5 minutes on star “D”.
\n
This gives us two possibilities:-
\n1. Taking difference between even and odd channels
\nTaking scalar difference between subsequent even and odd channels will help to eliminate bands which are exactly same in both the ON and OFF channels.
\nExample:-
\n
2. Comparing NMF of ON and OFF images
\nIn this method we decompose both the ON and OFF images into their base matrices W and H whose product approximates the ON/OFF matrix respectively. Then we reconstruct the noise matrix by multiplying the constituent matrix W of ON snippet with the matrix H of OFF snippet. This noise matrix is then subtracted from the ON snippet to get a clean representation of the signal.
\nThis basically recreates the features which are unique to the ON and OFF snippets while eliminating anything in common.
\nExample:-
\n
The function used to create these Images can be found in this Notebook
\nAssumption:-
\nWe can do this as it is clarified by @stevecroft in this discussion thread that the observations are taken sequentially one after the other and I am assuming the external sources of noise remain the same within that period.
Some more examples can be found below:-
\nThe function used to create these Images can be found in this Notebook
\nThings that did not work:-
\nHope you find this useful. 😊
\nUPDATE:-
\nAs clarified by @yuhongc in the comments below:-
\n", "rawMarkdown": "Hi fellow Kagglers,\n\nAs I am sure most of us would have seen examples as shown below where there is a constant band across a particular frequency in both the ON and OFF scenarios.\n\n\n\nSo technically that is not adding any additional useful information to the signal and is most likely caused by a consistent source of **Radio Frequency Interference (RFI)**. It also looks similar to some of the positive example as illustrated below, it might be a source of confusion/contradiction for the model.\n\n\n\n**How to get rid of such bands?**\n\nWe already know from the Data Description that:-\n>One method we use to isolate candidate techno signatures from RFI is to look for signals that appear to be coming from particular positions on the sky. Typically we do this by alternating observations of our primary target star with observations of three nearby stars: 5 minutes on star “A”, then 5 minutes on star “B”, then back to star “A” for 5 minutes, then “C”, then back to “A”, then finishing with 5 minutes on star “D”.\n\nThis gives us two possibilities:-\n\n**1. Taking difference between even and odd channels**\nTaking scalar difference between subsequent even and odd channels will help to eliminate bands which are exactly same in both the ON and OFF channels.\nExample:-\n\n\n**2. Comparing NMF of ON and OFF images**\nIn this method we decompose both the ON and OFF images into their base matrices W and H whose product approximates the ON/OFF matrix respectively. Then we reconstruct the noise matrix by multiplying the constituent matrix W of ON snippet with the matrix H of OFF snippet. This noise matrix is then subtracted from the ON snippet to get a clean representation of the signal. \nThis basically recreates the features which are unique to the ON and OFF snippets while eliminating anything in common.\nExample:-\n\n\n**The function used to create these Images can be found in [this Notebook](https://www.kaggle.com/manabendrarout/rfi-reduction-ideas-examples-seti)**\n\n**Assumption:-**\nWe can do this as it is clarified by @stevecroft in [this](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/241475) discussion thread that the observations are taken sequentially one after the other and I am assuming the external sources of noise remain the same within that period.\n\nSome more examples can be found below:-\n1. \n\n2. \n\n3. \n\n4. \n\n5. \n\n\n**The function used to create these Images can be found in [this Notebook](https://www.kaggle.com/manabendrarout/rfi-reduction-ideas-examples-seti)**\n\nThings that did not work:-\n1. PCA for Noise reduction\n2. SVD for matrix decomposition and noise re-construction\n\nHope you find this useful. 😊\n\n**UPDATE:-**\nAs clarified by @yuhongc in the comments below:- \n> **The vertical bright line is what we call a DC spike**, and is an artifact from the way we convert the raw signal from the telescope into spectrograms. We left them in because we wanted the data to be as realistic as possible, and it happens to be one of the simplest forms of interference (**although technically not RFI**).", "votes": 26 }, { "id": 1348157, "postDate": "2021-06-13T19:17:31.620Z", "content": "The vertical bright line is what we call a DC spike, and is an artifact from the way we convert the raw signal from the telescope into spectrograms. We left them in because we wanted the data to be as realistic as possible, and it happens to be one of the simplest forms of interference (although technically not RFI).
\n
This isn't really related to the Non Negative Matrix factorization, but the vertical bright line is what we call a DC spike, and is an artifact from the way we convert the raw signal from the telescope into spectrograms. We left them in because we wanted the data to be as realistic as possible, and it happens to be one of the simplest forms of interference (although technically not RFI).
", "rawMarkdown": "This isn't really related to the Non Negative Matrix factorization, but the vertical bright line is what we call a DC spike, and is an artifact from the way we convert the raw signal from the telescope into spectrograms. We left them in because we wanted the data to be as realistic as possible, and it happens to be one of the simplest forms of interference (although technically not RFI).", "votes": 5, "replies": [ { "id": 1348162, "postDate": "2021-06-13T19:27:54.390Z", "content": "Thanks for the clarification @yuhongc
\nIt really makes a lot of sense considering the frequency is perfectly constant and magnitude is a lot higher.
\nI will add it as an update to my topic. 😊
Perfect.
\nI am training network by using similar preprocessing function.
\nJust waiting for it to converge.
Good to know. I am also waiting for it to converge… If it is no trouble, kindly update how it went after it is done! I am also having a similar approach but limited by my compute capabilities 😓
", "rawMarkdown": "Good to know. I am also waiting for it to converge... If it is no trouble, kindly update how it went after it is done! I am also having a similar approach but limited by my compute capabilities 😓", "votes": 1 }, { "id": 1347483, "postDate": "2021-06-13T08:39:21.520Z", "content": "Sure.
", "rawMarkdown": "Sure. ", "votes": 2 }, { "id": 1348011, "postDate": "2021-06-13T16:32:00.370Z", "content": "Update:-
\nMy model converges at ~25 epochs. I am getting 0.991 CV|0.97x LB with 1xResnet18d vanilla model. No k-fold ensemble.
\nProbably need to run with a slightly bigger model. I am thinking Resnet34d or Efficientnet B0 with 5-fold ensemble.
Wow. I am getting 0.97 CV at 21 Epochs. :(
", "rawMarkdown": "Wow. I am getting 0.97 CV at 21 Epochs. :( " } ] }, { "id": 1348888, "postDate": "2021-06-14T11:07:44.983Z", "content": "Thanks for sharing this @manabendrarout. I have a quick question - when you put in an image X there are max_iter times the model tries to optimize the distance between X and W, H (non-negative matrices). However, for many images, the default max_iter=200 is not sufficient to converge and throws a ConvergenceWarning. How are you ensuring that each image and the non-negative matrix are converged properly?
Good question @ayuraj
\nI was also facing a similar problem when I started with this idea. The trick is that you need to reduce the number of components. You can find the working code here. This does not throw any error and works much faster as well.
Thanks for sharing!!
\nwhen you use this, you save denoised image before run training code?
Yes @abebe9849
\nI also tried to implement this in the pipeline directly, but that was making the iterations vert very slow.
This isn't really related to the Non Negative Matrix factorization, but the vertical bright line is what we call a DC spike, and is an artifact from the way we convert the raw signal from the telescope into spectrograms. We left them in because we wanted the data to be as realistic as possible, and it happens to be one of the simplest forms of interference (although technically not RFI).
", "votes": 5, "replies": [ { "id": 1348162, "author_name": "Manav", "author_url": "", "post_date": "2021-06-13T19:27:54.390000", "content": "Thanks for the clarification @yuhongc
\nIt really makes a lot of sense considering the frequency is perfectly constant and magnitude is a lot higher.
\nI will add it as an update to my topic. 😊
Perfect.
\nI am training network by using similar preprocessing function.
\nJust waiting for it to converge.
Good to know. I am also waiting for it to converge… If it is no trouble, kindly update how it went after it is done! I am also having a similar approach but limited by my compute capabilities 😓
", "votes": 1, "replies": [] }, { "id": 1347483, "author_name": "Salman", "author_url": "", "post_date": "2021-06-13T08:39:21.520000", "content": "Sure.
", "votes": 2, "replies": [] }, { "id": 1348011, "author_name": "Manav", "author_url": "", "post_date": "2021-06-13T16:32:00.370000", "content": "Update:-
\nMy model converges at ~25 epochs. I am getting 0.991 CV|0.97x LB with 1xResnet18d vanilla model. No k-fold ensemble.
\nProbably need to run with a slightly bigger model. I am thinking Resnet34d or Efficientnet B0 with 5-fold ensemble.
Wow. I am getting 0.97 CV at 21 Epochs. :(
", "votes": 0, "replies": [] } ] }, { "id": 1348888, "author_name": "Ayush Thakur", "author_url": "", "post_date": "2021-06-14T11:07:44.983000", "content": "Thanks for sharing this @manabendrarout. I have a quick question - when you put in an image X there are max_iter times the model tries to optimize the distance between X and W, H (non-negative matrices). However, for many images, the default max_iter=200 is not sufficient to converge and throws a ConvergenceWarning. How are you ensuring that each image and the non-negative matrix are converged properly?
Good question @ayuraj
\nI was also facing a similar problem when I started with this idea. The trick is that you need to reduce the number of components. You can find the working code here. This does not throw any error and works much faster as well.
Thanks for sharing!!
\nwhen you use this, you save denoised image before run training code?
Yes @abebe9849
\nI also tried to implement this in the pipeline directly, but that was making the iterations vert very slow.