I am a little bit confused by the kernels. And I think it is important for some type of models. As far as I understood the time-step dimension is 273 in (6, 273, 256). First of all, am I right?
\nSecondly, If I am right, then why some kernels use the 256 dimensions for the stacking, or if they want to use channel-wise models they transpose the model to (6, 256, 273) (e.g. the biGRU kernel https://www.kaggle.com/xhlulu/seti-bidirectional-gru-in-keras)?
\nIn total, how important do you think it is to mind which sequence of dimensions one uses?
I am a little bit confused by the kernels. And I think it is important for some type of models. As far as I understood the time-step dimension is 273 in (6, 273, 256). First of all, am I right?
\nSecondly, If I am right, then why some kernels use the 256 dimensions for the stacking, or if they want to use channel-wise models they transpose the model to (6, 256, 273) (e.g. the biGRU kernel https://www.kaggle.com/xhlulu/seti-bidirectional-gru-in-keras)?
\nIn total, how important do you think it is to mind which sequence of dimensions one uses?
Guys, I encountered sth very strange here. And apparently, I am not the only one.
\nWhen I use any RNN unit on top of either the image itself or its embedding, and I use 273 or 1638 as the time-step dimension (seq dim as in Pytorch), I get a very bad result (CV 50+). However, when I use 256 as the time-step dimension, the results are much better (CV 90+). I really don't understand why.
It looks quite logical, that you have better results when using 256 as time-step dimension. Take two slices of image containing target: one 1638x16 and other 16x256. It's much easies to check the slice contains target if you see full time picture even in narrow bandwidth range.
", "rawMarkdown": "It looks quite logical, that you have better results when using 256 as time-step dimension. Take two slices of image containing target: one 1638x16 and other 16x256. It's much easies to check the slice contains target if you see full time picture even in narrow bandwidth range.\n", "votes": 1, "replies": [ { "id": 1319016, "postDate": "2021-05-22T19:21:56.747Z", "content": "Hmm, sorry but i don't still get it. You say \"It's much easies to check the slice contains target if you see full time picture even in narrow bandwidth range.\" But when i state 1638 as the time dim., the RNN unit is looking at the full time picture. It is like the RNN is looking at each slice of time with freq. Vector as its feature vector. Just to make sure we are in the same page, 1638 ( or 273) is the time dim and 256 is the freq. dim.
", "rawMarkdown": "Hmm, sorry but i don't still get it. You say \"*It's much easies to check the slice contains target if you see full time picture even in narrow bandwidth range*.\" But when i state 1638 as the time dim., the RNN unit is looking at the full time picture. It is like the RNN is looking at each slice of time with freq. Vector as its feature vector. Just to make sure we are in the same page, 1638 ( or 273) is the time dim and 256 is the freq. dim." }, { "id": 1319237, "postDate": "2021-05-23T03:45:23.840Z", "content": "Yes, we are on the same page (1638 = 273*6 - it is time dimension). RNN have full access to information during training, it can actually (in theory) use memory to store what is important. But… training an RNN (to use information that used to be 273 time steps before) is not an easy task. On the other hand (using 256 as a time step) RNN has almost all the necessary information on one page and does not need to strain.
", "rawMarkdown": "Yes, we are on the same page (1638 = 273*6 - it is time dimension). RNN have full access to information during training, it can actually (in theory) use memory to store what is important. But... training an RNN (to use information that used to be 273 time steps before) is not an easy task. On the other hand (using 256 as a time step) RNN has almost all the necessary information on one page and does not need to strain." }, { "id": 1319423, "postDate": "2021-05-23T07:14:10.340Z", "content": "Tnx for clarification. Let me see if I understood correctly. You are basically saying that 1638 is too long for RNN, and since if one uses the feature vectors as the time steps, theoretically there should be no problem, hence RNN works better.
\nBut, even when one uses 273 as the time step and run an RNN for each channel, one gets a poor result (and if one does it other way around 256 as the time step and 273 as feature vectors the performance gets high again)
\nBtw, do you have any references for your argument regarding RNN's compatibility when using transposed information?
My two cents on this: When you use the time dimension (273) as the time dimension in the RNN (sorry for the wording 😊) the model \"sees\" (=gets as input) all the frequencies (256) at one point in time, on the other hand if you use the frequencies as the time dimension, the model's input is one frequence over the whole time span (273).
\nIt may just be that the latter incorporates more useful information for the model to learn from (carrying this information through the network).
Hmmm, yeah it seems you are right. Oddly, I have seen no paper using freq. as the sequence dim. in any similar model in any context.
\n@hannes82 @achikin Do you know of any references or examples that people used word embeddings instead of word sequences (or any other example in any other context) as the actual sequence?
\n@xhlulu what is your opinion on this?
@rythian47 No I don't know any. Not sure if this would work for NLP as it is everything about context there.
", "rawMarkdown": "@rythian47 No I don't know any. Not sure if this would work for NLP as it is everything about context there." } ] } ], "comments": [ { "id": 1318346, "author_name": "Baran Hashemi", "author_url": "", "post_date": "2021-05-22T08:32:43.237000", "content": "Guys, I encountered sth very strange here. And apparently, I am not the only one.
\nWhen I use any RNN unit on top of either the image itself or its embedding, and I use 273 or 1638 as the time-step dimension (seq dim as in Pytorch), I get a very bad result (CV 50+). However, when I use 256 as the time-step dimension, the results are much better (CV 90+). I really don't understand why.
It looks quite logical, that you have better results when using 256 as time-step dimension. Take two slices of image containing target: one 1638x16 and other 16x256. It's much easies to check the slice contains target if you see full time picture even in narrow bandwidth range.
", "votes": 1, "replies": [ { "id": 1319016, "author_name": "Baran Hashemi", "author_url": "", "post_date": "2021-05-22T19:21:56.747000", "content": "Hmm, sorry but i don't still get it. You say \"It's much easies to check the slice contains target if you see full time picture even in narrow bandwidth range.\" But when i state 1638 as the time dim., the RNN unit is looking at the full time picture. It is like the RNN is looking at each slice of time with freq. Vector as its feature vector. Just to make sure we are in the same page, 1638 ( or 273) is the time dim and 256 is the freq. dim.
", "votes": 0, "replies": [] }, { "id": 1319237, "author_name": "Anton Chikin", "author_url": "", "post_date": "2021-05-23T03:45:23.840000", "content": "Yes, we are on the same page (1638 = 273*6 - it is time dimension). RNN have full access to information during training, it can actually (in theory) use memory to store what is important. But… training an RNN (to use information that used to be 273 time steps before) is not an easy task. On the other hand (using 256 as a time step) RNN has almost all the necessary information on one page and does not need to strain.
", "votes": 0, "replies": [] }, { "id": 1319423, "author_name": "Baran Hashemi", "author_url": "", "post_date": "2021-05-23T07:14:10.340000", "content": "Tnx for clarification. Let me see if I understood correctly. You are basically saying that 1638 is too long for RNN, and since if one uses the feature vectors as the time steps, theoretically there should be no problem, hence RNN works better.
\nBut, even when one uses 273 as the time step and run an RNN for each channel, one gets a poor result (and if one does it other way around 256 as the time step and 273 as feature vectors the performance gets high again)
\nBtw, do you have any references for your argument regarding RNN's compatibility when using transposed information?
My two cents on this: When you use the time dimension (273) as the time dimension in the RNN (sorry for the wording 😊) the model \"sees\" (=gets as input) all the frequencies (256) at one point in time, on the other hand if you use the frequencies as the time dimension, the model's input is one frequence over the whole time span (273).
\nIt may just be that the latter incorporates more useful information for the model to learn from (carrying this information through the network).
Hmmm, yeah it seems you are right. Oddly, I have seen no paper using freq. as the sequence dim. in any similar model in any context.
\n@hannes82 @achikin Do you know of any references or examples that people used word embeddings instead of word sequences (or any other example in any other context) as the actual sequence?
\n@xhlulu what is your opinion on this?
@rythian47 No I don't know any. Not sure if this would work for NLP as it is everything about context there.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1314373": "I am a little bit confused by the kernels. And I think it is important for some type of models. As far as I understood the time-step dimension is 273 in (6, 273, 256). First of all, am I right?\nSecondly, If I am right, then why some kernels use the 256 dimensions for the stacking, or if they want to use channel-wise models they transpose the model to (6, 256, 273) (e.g. the biGRU kernel [https://www.kaggle.com/xhlulu/seti-bidirectional-gru-in-keras](url))?\nIn total, how important do you think it is to mind which sequence of dimensions one uses?", "1318346": "Guys, I encountered sth very strange here. And apparently, I am not the only one. \nWhen I use any RNN unit on top of either the image itself or its embedding, and I use 273 or 1638 as the time-step dimension (seq dim as in Pytorch), I get a very bad result (CV 50+). However, when I use 256 as the time-step dimension, the results are much better (CV 90+). I really don't understand why.\n", "1318769": "It looks quite logical, that you have better results when using 256 as time-step dimension. Take two slices of image containing target: one 1638x16 and other 16x256. It's much easies to check the slice contains target if you see full time picture even in narrow bandwidth range.\n" } }