![\"spectrogram_question\"](\"http://dragonaur.io/wp-content/uploads/2021/05/spectrogram_q.png\")
The data description details the following about the dimensions of an individual data file:
\n\n\nEach file has dimension (6, 273, 256), with the 1st dimension representing the 6 positions of the cadence, and the 2nd and 3rd dimensions representing the 2D spectrogram.
\n
The first dimension is well explained, but I'm seeking clarification the the 2nd and 3rd dimension.
\nFirst, which dimension represents time and which represents frequency?
\nFrom my initial exploration of the data, I believe I can identify radio frequency interference (RFI) sources at a constant frequency through time, leading me to believe the 2nd dimension in the data (with length 273) is time, and the 3rd dimension (with length 256) is frequency.
\nAn example of perceived RFI observed at constant frequency through time:
Again, this is my interpretation of the data, confirmation from the competition organizers would be welcome.
\nSecondly, a slew of question about the data units: What are the frequency and time spans in the data? What is the discrete time and frequency step per bin? Are these spans and step sizes consistent throughout all data files?
\nThe data information tab discusses 5 minute long observations but neither dimension seems to cleanly divide into 5 minutes.
\nI'm assuming everything I'm asking for is in the meta-data the competition organizers intentionally excluded. Maybe they thought it was extraneous or maybe it contains data that would allow competitors to game the competition, but I believe getting clarification on the above is essential to enabling exploration of digital signal processing techniques that may be used to pre-condition the data. Without clarification, I think this competition is limited to any techniques applied to any other image/pattern recognition task.
", "messages": [ { "id": 1309350, "postDate": "2021-05-15T20:57:46.143Z", "content": "The data description details the following about the dimensions of an individual data file:
\n\n\nEach file has dimension (6, 273, 256), with the 1st dimension representing the 6 positions of the cadence, and the 2nd and 3rd dimensions representing the 2D spectrogram.
\n
The first dimension is well explained, but I'm seeking clarification the the 2nd and 3rd dimension.
\nFirst, which dimension represents time and which represents frequency?
\nFrom my initial exploration of the data, I believe I can identify radio frequency interference (RFI) sources at a constant frequency through time, leading me to believe the 2nd dimension in the data (with length 273) is time, and the 3rd dimension (with length 256) is frequency.
\nAn example of perceived RFI observed at constant frequency through time:
Again, this is my interpretation of the data, confirmation from the competition organizers would be welcome.
\nSecondly, a slew of question about the data units: What are the frequency and time spans in the data? What is the discrete time and frequency step per bin? Are these spans and step sizes consistent throughout all data files?
\nThe data information tab discusses 5 minute long observations but neither dimension seems to cleanly divide into 5 minutes.
\nI'm assuming everything I'm asking for is in the meta-data the competition organizers intentionally excluded. Maybe they thought it was extraneous or maybe it contains data that would allow competitors to game the competition, but I believe getting clarification on the above is essential to enabling exploration of digital signal processing techniques that may be used to pre-condition the data. Without clarification, I think this competition is limited to any techniques applied to any other image/pattern recognition task.
", "rawMarkdown": "The [data description](https://www.kaggle.com/c/seti-breakthrough-listen/data) details the following about the dimensions of an individual data file:\n> Each file has dimension (6, 273, 256), with the 1st dimension representing the 6 positions of the cadence, and the 2nd and 3rd dimensions representing the 2D spectrogram.\n\nThe first dimension is well explained, but I'm seeking clarification the the 2nd and 3rd dimension. \n\n**First, which dimension represents time and which represents frequency?**\n\nFrom my initial exploration of the data, I believe I can identify radio frequency interference (RFI) sources at a constant frequency through time, leading me to believe the 2nd dimension in the data (with length 273) is time, and the 3rd dimension (with length 256) is frequency. \n\nAn example of perceived RFI observed at constant frequency through time: \n\nAgain, this is my interpretation of the data, confirmation from the competition organizers would be welcome.\n\n**Secondly, a slew of question about the data units: What are the frequency and time spans in the data? What is the discrete time and frequency step per bin? Are these spans and step sizes consistent throughout all data files?**\n\nThe [data information](https://www.kaggle.com/c/seti-breakthrough-listen/overview/data-information) tab discusses 5 minute long observations but neither dimension seems to cleanly divide into 5 minutes.\n\nI'm assuming everything I'm asking for is in the meta-data the competition organizers intentionally excluded. Maybe they thought it was extraneous or maybe it contains data that would allow competitors to game the competition, but I believe getting clarification on the above is essential to enabling exploration of digital signal processing techniques that may be used to pre-condition the data. Without clarification, I think this competition is limited to any techniques applied to any other image/pattern recognition task. ", "votes": 21 }, { "id": 1310349, "postDate": "2021-05-16T15:43:55.840Z", "content": "273 is indeed the time axis, and it's the full 5 minutes, across ~0.7 MHz of frequency. Snippets are extracted from the mid-resolution files (~1.07 second time resolution, 2.86 kHz frequency resolution - see table 4 of Lebofsky et al.).
", "rawMarkdown": "273 is indeed the time axis, and it's the full 5 minutes, across ~0.7 MHz of frequency. Snippets are extracted from the mid-resolution files (~1.07 second time resolution, 2.86 kHz frequency resolution - see table 4 of [Lebofsky et al.](https://arxiv.org/pdf/1906.07391.pdf)).", "votes": 12, "replies": [ { "id": 1310411, "postDate": "2021-05-16T16:32:13.257Z", "content": "Thank you for your response confirming the axes and resolutions along with the reference!
", "rawMarkdown": "Thank you for your response confirming the axes and resolutions along with the reference! ", "votes": 1 }, { "id": 1310589, "postDate": "2021-05-16T18:37:17.137Z", "content": "Thanks for the clarification. I've found, in many years of working with multi-dimensional arrays, that a significant part of the challenge may just be figuring out the order of the indices. For example, with regard to image data, different languages and different packages often have different conventions for whether the color channel comes first or last, what order the horizontal and vertical dimensions are in, etc. Sometimes I have to \"twist my poor brain into a pretzel\" to get it right.
", "rawMarkdown": "Thanks for the clarification. I've found, in many years of working with multi-dimensional arrays, that a significant part of the challenge may just be figuring out the order of the indices. For example, with regard to image data, different languages and different packages often have different conventions for whether the color channel comes first or last, what order the horizontal and vertical dimensions are in, etc. Sometimes I have to \"twist my poor brain into a pretzel\" to get it right.", "votes": 3 } ] }, { "id": 1310018, "postDate": "2021-05-16T12:41:51.430Z", "content": "These are techniques of Signal processing right?
\nI think competition host want from us to use Computer vision techniques so they didn't provide us the whole spectrogram of 5 min. But let the organizers clarify this.
From the competition's overview page:
\n\n\nThe data consist of two-dimensional arrays, so there may be approaches from computer vision that are promising, as well as digital signal processing, anomaly detection, and more. The algorithm that’s successful at identifying the most needles will win a cash prize, but also has the potential to help answer one of the biggest questions in science.
\n
So the competition description is encouraging many domains and algorithmic approaches but I'll concur that the organizer's data descriptions and much of the community's discussions and approaches shared to date are centered around computer vision. I'm personally inclined to explore what gains, if any, can be achieved with signal processing techniques first though, if only to satisfy my own curiosity.
", "rawMarkdown": "From the competition's [overview](https://www.kaggle.com/c/seti-breakthrough-listen/overview) page:\n> The data consist of two-dimensional arrays, so there may be approaches from computer vision that are promising, **as well as digital signal processing**, anomaly detection, and more. The algorithm that’s successful at identifying the most needles will win a cash prize, but also has the potential to help answer one of the biggest questions in science.\n\nSo the competition description is encouraging many domains and algorithmic approaches but I'll concur that the organizer's data descriptions and much of the community's discussions and approaches shared to date are centered around computer vision. I'm personally inclined to explore what gains, if any, can be achieved with signal processing techniques first though, if only to satisfy my own curiosity.", "votes": 3 } ] } ], "comments": [ { "id": 1310349, "author_name": "Steve Croft", "author_url": "", "post_date": "2021-05-16T15:43:55.840000", "content": "273 is indeed the time axis, and it's the full 5 minutes, across ~0.7 MHz of frequency. Snippets are extracted from the mid-resolution files (~1.07 second time resolution, 2.86 kHz frequency resolution - see table 4 of Lebofsky et al.).
", "votes": 12, "replies": [ { "id": 1310411, "author_name": "Jeffrey Egan", "author_url": "", "post_date": "2021-05-16T16:32:13.257000", "content": "Thank you for your response confirming the axes and resolutions along with the reference!
", "votes": 1, "replies": [] }, { "id": 1310589, "author_name": "David J. Slate", "author_url": "", "post_date": "2021-05-16T18:37:17.137000", "content": "Thanks for the clarification. I've found, in many years of working with multi-dimensional arrays, that a significant part of the challenge may just be figuring out the order of the indices. For example, with regard to image data, different languages and different packages often have different conventions for whether the color channel comes first or last, what order the horizontal and vertical dimensions are in, etc. Sometimes I have to \"twist my poor brain into a pretzel\" to get it right.
", "votes": 3, "replies": [] } ] }, { "id": 1310018, "author_name": "Aman Deep Gupta", "author_url": "", "post_date": "2021-05-16T12:41:51.430000", "content": "These are techniques of Signal processing right?
\nI think competition host want from us to use Computer vision techniques so they didn't provide us the whole spectrogram of 5 min. But let the organizers clarify this.
From the competition's overview page:
\n\n\nThe data consist of two-dimensional arrays, so there may be approaches from computer vision that are promising, as well as digital signal processing, anomaly detection, and more. The algorithm that’s successful at identifying the most needles will win a cash prize, but also has the potential to help answer one of the biggest questions in science.
\n
So the competition description is encouraging many domains and algorithmic approaches but I'll concur that the organizer's data descriptions and much of the community's discussions and approaches shared to date are centered around computer vision. I'm personally inclined to explore what gains, if any, can be achieved with signal processing techniques first though, if only to satisfy my own curiosity.
", "votes": 3, "replies": [] } ] } ], "raw_markdown_by_id": { "1309350": "The [data description](https://www.kaggle.com/c/seti-breakthrough-listen/data) details the following about the dimensions of an individual data file:\n> Each file has dimension (6, 273, 256), with the 1st dimension representing the 6 positions of the cadence, and the 2nd and 3rd dimensions representing the 2D spectrogram.\n\nThe first dimension is well explained, but I'm seeking clarification the the 2nd and 3rd dimension. \n\n**First, which dimension represents time and which represents frequency?**\n\nFrom my initial exploration of the data, I believe I can identify radio frequency interference (RFI) sources at a constant frequency through time, leading me to believe the 2nd dimension in the data (with length 273) is time, and the 3rd dimension (with length 256) is frequency. \n\nAn example of perceived RFI observed at constant frequency through time: \n\nAgain, this is my interpretation of the data, confirmation from the competition organizers would be welcome.\n\n**Secondly, a slew of question about the data units: What are the frequency and time spans in the data? What is the discrete time and frequency step per bin? Are these spans and step sizes consistent throughout all data files?**\n\nThe [data information](https://www.kaggle.com/c/seti-breakthrough-listen/overview/data-information) tab discusses 5 minute long observations but neither dimension seems to cleanly divide into 5 minutes.\n\nI'm assuming everything I'm asking for is in the meta-data the competition organizers intentionally excluded. Maybe they thought it was extraneous or maybe it contains data that would allow competitors to game the competition, but I believe getting clarification on the above is essential to enabling exploration of digital signal processing techniques that may be used to pre-condition the data. Without clarification, I think this competition is limited to any techniques applied to any other image/pattern recognition task. ", "1310349": "273 is indeed the time axis, and it's the full 5 minutes, across ~0.7 MHz of frequency. Snippets are extracted from the mid-resolution files (~1.07 second time resolution, 2.86 kHz frequency resolution - see table 4 of [Lebofsky et al.](https://arxiv.org/pdf/1906.07391.pdf)).", "1310018": "These are techniques of Signal processing right?\nI think competition host want from us to use Computer vision techniques so they didn't provide us the whole spectrogram of 5 min. But let the organizers clarify this." } }