{ "id": 444653, "title": "How to check if your model generalizes to long sequences", "url": "/competitions/stanford-ribonanza-rna-folding/discussion/444653", "author_name": "Shujun", "post_date": "2023-10-03T02:51:06.598000", "votes": 60, "comment_count": 78, "views": 0, "content": "

An important task in this competition is for models to generalize to long RNA sequences, because many RNAs of interest can be much longer than what is available in the training set. Therefore, we have included sequences that are much longer (as long as 457 nt) in the private test set. A good way to check your model's performance on these long sequences is to visually inspect a subset of sequences which we call mutate and map (used to infer base pairings by point mutations). We have an example of these using predictions from a model that gives reasonable predictions on a subset of 457nt sequences with known structure. Note that the red arrow points to a region of known long range pseudo-knot (the darker rectangle) which have been impossible to predict with prior modeling methods. When exposed following point mutations, the long range pseudo-knot region has increased reactivity.
\n\"\"
\nTo plot these:

\n
import polars as pl\nimport matplotlib.pyplot as plt\n\n=pl.read_csv(\"sub.csv\")\n\n=6\n=269545321\n=269724007\n=391\n=457\n\n=df[id1:id2+1][].to_numpy().reshape(reshape1,reshape2)\n=df[id1:id2+1][].to_numpy().reshape(reshape1,reshape2)\n\nfig = plt.figure()\nplt.subplot(121)\nplt.title(f, =font_size)\nplt.imshow(pred_DMS,=0,vmax=1, =)\nplt.subplot(122)\nplt.title(f, =font_size)\nplt.imshow(pred_2A3,=0,vmax=1, =)\nplt.tight_layout()\nplt.savefig(f,=500)\nplt.clf()\nplt.close()\n
\n

Additionally, you also want to make sure that your model is outputting reasonable predictions for the beginning and end regions where there is no training data. Currently, due to some technical reasons, we cannot measure these positions but we might be able to resolve these limitations in the final private test set. We have noticed internally that some of our models under certain conditions like to output entirely 0s for the beginning and end regions, which should not be happening.
\nFor anyone confused about how these mutations work, I have added another plot visualizing where these mutations occur. From position 27 to 416, the mutations at each position is one of {'A->U', 'C->G', 'G->C', 'U->A'}. Therefore, you can see a diagonal line in the middle. Note that the beginning and end regions are constant, while right before the end, there's also a bar code region unique to each sequence.
\n\"\"

", "messages": [ { "id": 2465490, "postDate": "2023-10-03T02:51:06.600Z", "content": "

An important task in this competition is for models to generalize to long RNA sequences, because many RNAs of interest can be much longer than what is available in the training set. Therefore, we have included sequences that are much longer (as long as 457 nt) in the private test set. A good way to check your model's performance on these long sequences is to visually inspect a subset of sequences which we call mutate and map (used to infer base pairings by point mutations). We have an example of these using predictions from a model that gives reasonable predictions on a subset of 457nt sequences with known structure. Note that the red arrow points to a region of known long range pseudo-knot (the darker rectangle) which have been impossible to predict with prior modeling methods. When exposed following point mutations, the long range pseudo-knot region has increased reactivity.
\n\"\"
\nTo plot these:

\n
import polars as pl\nimport matplotlib.pyplot as plt\n\n=pl.read_csv(\"sub.csv\")\n\n=6\n=269545321\n=269724007\n=391\n=457\n\n=df[id1:id2+1][].to_numpy().reshape(reshape1,reshape2)\n=df[id1:id2+1][].to_numpy().reshape(reshape1,reshape2)\n\nfig = plt.figure()\nplt.subplot(121)\nplt.title(f, =font_size)\nplt.imshow(pred_DMS,=0,vmax=1, =)\nplt.subplot(122)\nplt.title(f, =font_size)\nplt.imshow(pred_2A3,=0,vmax=1, =)\nplt.tight_layout()\nplt.savefig(f,=500)\nplt.clf()\nplt.close()\n
\n

Additionally, you also want to make sure that your model is outputting reasonable predictions for the beginning and end regions where there is no training data. Currently, due to some technical reasons, we cannot measure these positions but we might be able to resolve these limitations in the final private test set. We have noticed internally that some of our models under certain conditions like to output entirely 0s for the beginning and end regions, which should not be happening.
\nFor anyone confused about how these mutations work, I have added another plot visualizing where these mutations occur. From position 27 to 416, the mutations at each position is one of {'A->U', 'C->G', 'G->C', 'U->A'}. Therefore, you can see a diagonal line in the middle. Note that the beginning and end regions are constant, while right before the end, there's also a bar code region unique to each sequence.
\n\"\"

", "rawMarkdown": "\nAn important task in this competition is for models to generalize to long RNA sequences, because many RNAs of interest can be much longer than what is available in the training set. Therefore, we have included sequences that are much longer (as long as 457 nt) in the private test set. A good way to check your model's performance on these long sequences is to visually inspect a subset of sequences which we call [mutate and map](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4080707/) (used to infer base pairings by point mutations). We have an example of these using predictions from a model that gives reasonable predictions on a subset of 457nt sequences with known structure. Note that the red arrow points to a region of known long range pseudo-knot (the darker rectangle) which have been impossible to predict with prior modeling methods. When exposed following point mutations, the long range pseudo-knot region has increased reactivity.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2Fbda49dc7b259bcec466b575c5a75d97c%2Fexample.png?generation=1696301353614738&alt=media)\n\nTo plot these:\n```\nimport polars as pl\nimport matplotlib.pyplot as plt\n\n#read your sub here\ndf=pl.read_csv(\"sub.csv\")\n\n#some parameters\nfont_size=6\nid1=269545321\nid2=269724007\nreshape1=391\nreshape2=457\n\n#get predictions\npred_DMS=df[id1:id2+1]['reactivity_DMS_MaP'].to_numpy().reshape(reshape1,reshape2)\npred_2A3=df[id1:id2+1]['reactivity_2A3_MaP'].to_numpy().reshape(reshape1,reshape2)\n\n#plot mutate and map\nfig = plt.figure()\nplt.subplot(121)\nplt.title(f'reactivity_DMS_MaP', fontsize=font_size)\nplt.imshow(pred_DMS,vmin=0,vmax=1, cmap='gray_r')\nplt.subplot(122)\nplt.title(f'reactivity_2A3_MaP', fontsize=font_size)\nplt.imshow(pred_2A3,vmin=0,vmax=1, cmap='gray_r')\n\nplt.tight_layout()\nplt.savefig(f\"plot.png\",dpi=500)\nplt.clf()\nplt.close()\n```\n\nAdditionally, you also want to make sure that your model is outputting reasonable predictions for the beginning and end regions where there is no training data. Currently, due to some technical reasons, we cannot measure these positions but we might be able to resolve these limitations in the final private test set. We have noticed internally that some of our models under certain conditions like to output entirely 0s for the beginning and end regions, which should not be happening.\n\nFor anyone confused about how these mutations work, I have added another plot visualizing where these mutations occur. From position 27 to 416, the mutations at each position is one of {'A->U', 'C->G', 'G->C', 'U->A'}. Therefore, you can see a diagonal line in the middle. Note that the beginning and end regions are constant, while right before the end, there's also a bar code region unique to each sequence. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F633626f460d2a563a3757e5e003529e8%2FTribozyme_point_mutations.png?generation=1697823756980699&alt=media)\n\n", "votes": 60 }, { "id": 2544569, "postDate": "2023-11-30T23:42:47.880Z", "content": "

Hi Kagglers, we have another interesting test of generality with a sequence that was in the last CASP competition: R1138v1 or 7PTL in protein data bank (https://www.rcsb.org/structure/7PTL ). We have extracted the secondary structure from the experimentally resolved 3D structure:\"\"

\n

and looked at our model’s predictions using the mutate-and-map technique with our model:
\n\"\"

\n

\n

While we definitely see the main secondary structure, the kissing loops above the diagonal between the 5' and 3' 'halves' are not clearly present (circled in cyan) and there are also some short stems missing (circled in red)

\n

\"\"

\n

You can check your predictions by making predictions for the sequences in the following CSV file), where the sequence ids are in the format of V1138v1_m2_{mutated_position}. And I have prepared bpp files for these. And after making predictions you can visualize with the following code:

\n
m2_preds=(m2_sequences) #x720x2\n\nplt()\nplt()\nplt(m2_preds,vmin=,vmax=,cmap=)\nplt()\nplt()\nplt(m2_preds,vmin=,vmax=,cmap=)\n
\n

We’re excited to see what you have and if your model is generalizing better!

", "rawMarkdown": "Hi Kagglers, we have another interesting test of generality with a sequence that was in the last CASP competition: R1138v1 or 7PTL in protein data bank (https://www.rcsb.org/structure/7PTL ). We have extracted the secondary structure from the experimentally resolved 3D structure:![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F592eb93db2a8a6be01f8f4daaf0ba369%2F7PTL.png?generation=1701387560513990&alt=media)\n\nand looked at our model’s predictions using the mutate-and-map technique with our model:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F1bd6d46561d3f9c3b7869e2f28fcf9f6%2FR1138v1.png?generation=1701387579567164&alt=media)\n\n~~While we definitely see the main secondary structure, the kissing loops above the diagonal (in cyan) between the 5' and 3' 'halves' are not clearly present (circled in cyan)~~\n\nWhile we definitely see the main secondary structure, the kissing loops above the diagonal between the 5' and 3' 'halves' are not clearly present (circled in cyan) and there are also some short stems missing (circled in red)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F929d70a5ec679273b52565cbf37ee7c7%2F7PTL_annotated.png?generation=1701392197627840&alt=media)\n\n\nYou can check your predictions by making predictions for the sequences in the following [CSV file](https://www.kaggle.com/datasets/shujun717/r1138v1-m2)), where the sequence ids are in the format of V1138v1_m2_{mutated_position}. And I have prepared [bpp files](https://www.kaggle.com/datasets/shujun717/r1138-bpp) for these. And after making predictions you can visualize with the following code:\n\n```\nm2_preds=model(m2_sequences) #721x720x2\n\nplt.subplot(121)\nplt.title('2A3')\nplt.imshow(m2_preds[:,:,0],vmin=0,vmax=1,cmap='gray_r')\nplt.subplot(122)\nplt.title('DMS')\nplt.imshow(m2_preds[:,:,1],vmin=0,vmax=1,cmap='gray_r')\n```\n\nWe’re excited to see what you have and if your model is generalizing better!\n\n", "votes": 9, "replies": [ { "id": 2544572, "postDate": "2023-11-30T23:47:57.213Z", "content": "

The link directs nowhere for now, can you make dataset public? Excited to try our models on these sequences c:

", "rawMarkdown": "The link directs nowhere for now, can you make dataset public? Excited to try our models on these sequences c:", "votes": 2 }, { "id": 2544573, "postDate": "2023-11-30T23:48:05.633Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2544582, "postDate": "2023-12-01T00:04:02.757Z", "content": "

@martynoveduard fixed. Can you see it now?

", "rawMarkdown": "@martynoveduard fixed. Can you see it now?", "votes": 1, "replies": [ { "id": 2544583, "postDate": "2023-12-01T00:05:12.310Z", "content": "

yup, thanks

", "rawMarkdown": "yup, thanks", "votes": 1 } ] }, { "id": 2544608, "postDate": "2023-12-01T01:09:26.333Z", "content": "

There was a mistake in my post regarding the kissing loops and I have updated the post and annotated secondary structure plot

", "rawMarkdown": "There was a mistake in my post regarding the kissing loops and I have updated the post and annotated secondary structure plot", "votes": 1 }, { "id": 2544622, "postDate": "2023-12-01T01:31:33.877Z", "content": "

Could you please provide the bpp files of the sequences ?

", "rawMarkdown": "Could you please provide the bpp files of the sequences ?", "replies": [ { "id": 2544632, "postDate": "2023-12-01T01:48:08.160Z", "content": "

Yeah see here: https://www.kaggle.com/datasets/shujun717/r1138-bpp

", "rawMarkdown": "Yeah see here: https://www.kaggle.com/datasets/shujun717/r1138-bpp", "votes": 2 }, { "id": 2544646, "postDate": "2023-12-01T02:07:15.123Z", "content": "

Thanks a lot !

", "rawMarkdown": "Thanks a lot !" } ] }, { "id": 2545058, "postDate": "2023-12-01T09:04:55.683Z", "content": "

\"This

", "rawMarkdown": "![This one is really hard...](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6832115%2F2cf362b4bdcb279336a4d2b81659dcff%2FR1138v1-m2.png?generation=1701421412590124&alt=media)", "replies": [ { "id": 2545116, "postDate": "2023-12-01T10:00:20.753Z", "content": "

I tried to take the mean of 2A3 and DMS for a better picture. I can actually see hints of this middle kissing loop. But the side ones are too difficult… perhaps the top LB models would do better there.\"\"

", "rawMarkdown": "I tried to take the mean of 2A3 and DMS for a better picture. I can actually see hints of this middle kissing loop. But the side ones are too difficult... perhaps the top LB models would do better there.![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6832115%2F4a067628c581254f866c329c4a4f84ca%2FR1138v1-m2%20mean.png?generation=1701424745768217&alt=media)" } ] }, { "id": 2545614, "postDate": "2023-12-01T15:56:03.477Z", "content": "

Thanks for another example. It seems my model gets signs of all lines except the ones in the middle you outline with cyan(
\n\"\"

", "rawMarkdown": "Thanks for another example. It seems my model gets signs of all lines except the ones in the middle you outline with cyan(\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1212661%2F713d4a75f27a4c1c64317e9ead70d19a%2Fg1.png?generation=1701445917913290&alt=media)", "votes": 1 } ] }, { "id": 2468976, "postDate": "2023-10-06T03:32:47.713Z", "content": "

Nice post
\n\"\"

", "rawMarkdown": "Nice post\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1212661%2F5c3b1d7781037565087d814379e80f37%2Fplot.png?generation=1696563122851918&alt=media)", "votes": 7 }, { "id": 2497319, "postDate": "2023-10-24T15:13:38.733Z", "content": "

Hopefully similar enough
\n\"\"

", "rawMarkdown": "Hopefully similar enough\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3586013%2Ff0719545c5889b67ddba45f7da00a3bf%2Fplot0.png?generation=1698160337092019&alt=media)", "votes": 6, "replies": [ { "id": 2499257, "postDate": "2023-10-25T22:27:41.210Z", "content": "

Your's looks really similar!

", "rawMarkdown": "Your's looks really similar!" }, { "id": 2513861, "postDate": "2023-11-05T19:47:20.967Z", "content": "

It's very impressive! Do you mind sharing the LB score of this model?

", "rawMarkdown": "It's very impressive! Do you mind sharing the LB score of this model?", "replies": [ { "id": 2514136, "postDate": "2023-11-06T04:54:33.823Z", "content": "

The LB score is actually pretty bad ~0.17 but it was because of a bug

", "rawMarkdown": "The LB score is actually pretty bad ~0.17 but it was because of a bug", "votes": 1 } ] } ] }, { "id": 2544552, "postDate": "2023-11-30T22:50:01.543Z", "content": "

Plot by current best submission
\n\"\"

\n

I still don't know how to interpret the plots.

", "rawMarkdown": "Plot by current best submission\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1745801%2F685f90a707c237413455e9757b3a8355%2Fplot.png?generation=1701385653066069&alt=media)\n\nI still don't know how to interpret the plots.", "votes": 4 }, { "id": 2467044, "postDate": "2023-10-04T09:04:43.060Z", "content": "

I think this topic deserves a pin! I was wondering if the model you used was trained on the same data we have.

", "rawMarkdown": "I think this topic deserves a pin! I was wondering if the model you used was trained on the same data we have.", "votes": 3, "replies": [ { "id": 2467846, "postDate": "2023-10-04T23:01:57.430Z", "content": "

Yes it was!

", "rawMarkdown": "Yes it was! ", "votes": 3 }, { "id": 2471884, "postDate": "2023-10-06T17:58:03.313Z", "content": "

Topic is pinned! Thanks for suggesting.

", "rawMarkdown": "Topic is pinned! Thanks for suggesting.", "votes": 2 } ] }, { "id": 2551957, "postDate": "2023-12-07T05:03:40.100Z", "content": "

@shujun717 I'm just curious in case if you are allowed to share this information, were you able to resolve the technical difficulties with measuring the activity at the sequence ends, or do you still plan to evaluate only the middle parts of the sequences? Thanks.

", "rawMarkdown": "@shujun717 I'm just curious in case if you are allowed to share this information, were you able to resolve the technical difficulties with measuring the activity at the sequence ends, or do you still plan to evaluate only the middle parts of the sequences? Thanks.", "votes": 2, "replies": [ { "id": 2553843, "postDate": "2023-12-08T14:47:27.777Z", "content": "

I think sequence ends were not used for scoring

", "rawMarkdown": "I think sequence ends were not used for scoring", "votes": 1 } ] }, { "id": 2473265, "postDate": "2023-10-08T06:01:29.310Z", "content": "

Great Post!
\n\"plot\"

", "rawMarkdown": "Great Post!\n![plot](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6491363%2F13f80753ed9d37e3528d7b3a93fc0bc0%2Fplot.png?generation=1696744827715873&alt=media)", "votes": 1 }, { "id": 2489571, "postDate": "2023-10-20T04:46:23.360Z", "content": "

I am still confused. Not sure what the arrangement is. Are these just lists of reactivity by nt position?

", "rawMarkdown": "I am still confused. Not sure what the arrangement is. Are these just lists of reactivity by nt position?", "replies": [ { "id": 2490474, "postDate": "2023-10-20T17:54:43.980Z", "content": "

Yes I have added an explanation for this! See the added plot and last paragraph

", "rawMarkdown": "Yes I have added an explanation for this! See the added plot and last paragraph", "replies": [ { "id": 2490680, "postDate": "2023-10-20T23:21:33.423Z", "content": "

Thank you. How did you determine the mutation?

", "rawMarkdown": "Thank you. How did you determine the mutation?", "votes": 1 } ] } ] }, { "id": 2465687, "postDate": "2023-10-03T07:36:09.713Z", "content": "

Thank you for providing this valuable example!

\n

I'll give plots of two of my models here, - not generalizible and somewhat generalizable

", "rawMarkdown": "Thank you for providing this valuable example!\n\nI'll give plots of two of my models here, - not generalizible and somewhat generalizable", "votes": 2, "replies": [ { "id": 2465690, "postDate": "2023-10-03T07:38:42.970Z", "content": "

This one I thought won't generalize to private LB

\n

\"\"

", "rawMarkdown": "This one I thought won't generalize to private LB\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5487737%2F33511d686900e64bb2a2b80d38fa71fc%2FScreenshot%202023-10-03%20at%2010.36.42.png?generation=1696318705717511&alt=media)", "votes": 2 }, { "id": 2465692, "postDate": "2023-10-03T07:42:11.820Z", "content": "

I don't see the black rectangle here :D But the plot overall resembles yours

\n

\"\"

", "rawMarkdown": "I don't see the black rectangle here :D But the plot overall resembles yours \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5487737%2F52d785fe5fb11ed872c9d7d7223487c4%2FScreenshot%202023-10-03%20at%2010.40.47.png?generation=1696318881885098&alt=media)", "votes": 4, "replies": [ { "id": 2467875, "postDate": "2023-10-05T00:57:05.717Z", "content": "

Thanks for posting. Any idea why the first model does not generalize well? It's fine if you don't want to share though.

", "rawMarkdown": "Thanks for posting. Any idea why the first model does not generalize well? It's fine if you don't want to share though. ", "votes": 2 }, { "id": 2468063, "postDate": "2023-10-05T06:41:32.447Z", "content": "

My early models were GRUs and LSTMs. I had two issue with those models.
\n1) The outputs were preferentially zeros at start and end of the sequences.
\n2) I was struggling to generalize for long sequences.
\nThen, I implemented the model proposed by @IAFOSS https://www.kaggle.com/code/iafoss/rna-starter-submission-0-186-lb and the issues disappeared

", "rawMarkdown": "My early models were GRUs and LSTMs. I had two issue with those models. \n1) The outputs were preferentially zeros at start and end of the sequences.\n2) I was struggling to generalize for long sequences. \nThen, I implemented the model proposed by @IAFOSS https://www.kaggle.com/code/iafoss/rna-starter-submission-0-186-lb and the issues disappeared", "votes": 3 }, { "id": 2468100, "postDate": "2023-10-05T07:14:51.937Z", "content": "
\n

Any idea why the first model does not generalize well?

\n
\n

Sure! First model is using absolute positional encoding and was not trained on longer sequences.

\n

We've got to figure out how to align feature spaces of train and test distributions so that we can predict something like you shared in the post

", "rawMarkdown": "> Any idea why the first model does not generalize well?\n\nSure! First model is using absolute positional encoding and was not trained on longer sequences.\n\nWe've got to figure out how to align feature spaces of train and test distributions so that we can predict something like you shared in the post", "votes": 6 }, { "id": 2474180, "postDate": "2023-10-09T03:57:04.610Z", "content": "

@martynoveduard what do you mean training on longer sequences? Most of the sequences in train are 177

", "rawMarkdown": "@martynoveduard what do you mean training on longer sequences? Most of the sequences in train are 177", "votes": 1 }, { "id": 2474197, "postDate": "2023-10-09T04:20:21.983Z", "content": "

When I talk about longer sequences, I'm referring to sequences with a length greater than 206. In the private test, it's apparent that the model hasn't learned positional encoding for tokens beyond the 206th position. Because of this, I consider these tokens as out-of-distribution (OOD) samples for the model and as can be seen from the first plot, predictions for these positions are different from the previous positions.

", "rawMarkdown": "When I talk about longer sequences, I'm referring to sequences with a length greater than 206. In the private test, it's apparent that the model hasn't learned positional encoding for tokens beyond the 206th position. Because of this, I consider these tokens as out-of-distribution (OOD) samples for the model and as can be seen from the first plot, predictions for these positions are different from the previous positions.", "votes": 1 }, { "id": 2502078, "postDate": "2023-10-27T23:47:58.583Z", "content": "

Forgive my stupidity but, how can you know that in the private test your model doesn't work on beyond 206th. The private test data has not been released.

", "rawMarkdown": "Forgive my stupidity but, how can you know that in the private test your model doesn't work on beyond 206th. The private test data has not been released. " }, { "id": 2502338, "postDate": "2023-10-28T06:00:25.833Z", "content": "

I just compared (visually) the plots I obtained from the script provided by Shujun for my models and hosts model, as you can see there are sequences of length >400, therefore its a part of the private data, and it's provided to us for convinience so we can sanity check our models on sequences with greater lengths

", "rawMarkdown": "I just compared (visually) the plots I obtained from the script provided by Shujun for my models and hosts model, as you can see there are sequences of length >400, therefore its a part of the private data, and it's provided to us for convinience so we can sanity check our models on sequences with greater lengths", "votes": 1 }, { "id": 2539283, "postDate": "2023-11-26T23:57:05.847Z", "content": "

Hello!
\nthe model does actually generalize quite worse than yours. Did you do anything specific to improve generalization or does it generalize better with accuracy improvement?
\nIf your willing to share i'll be glad to hear what you did to improve generalization…
\nI've tried implementing sliding windows and similar embedding stuff on top but haven't been able to generalize as well

\n

\"@iafoss\"

", "rawMarkdown": "Hello! \nthe model does actually generalize quite worse than yours. Did you do anything specific to improve generalization or does it generalize better with accuracy improvement?\nIf your willing to share i'll be glad to hear what you did to improve generalization... \nI've tried implementing sliding windows and similar embedding stuff on top but haven't been able to generalize as well\n\n![@iafoss](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12000969%2Fc96903cd3bbd05b126094a7f4321bb22%2Fplot3%20(3).png?generation=1701042395156105&alt=media)" }, { "id": 2539286, "postDate": "2023-11-26T23:59:45.200Z", "content": "

implementing sliding window on top
\n\"\"

", "rawMarkdown": "implementing sliding window on top\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12000969%2F414a53cf6db6ca39ec1df130a5932aa5%2Fplot3.png?generation=1701043156096329&alt=media)\n" }, { "id": 2540819, "postDate": "2023-11-28T03:25:10.050Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2465562, "postDate": "2023-10-03T05:14:30.540Z", "content": "

It is not easy.

", "rawMarkdown": "It is not easy.", "votes": 2 }, { "id": 2552073, "postDate": "2023-12-07T07:06:08.943Z", "content": "

Did anybody succeed in predicting long-range pseudoknot?

", "rawMarkdown": "Did anybody succeed in predicting long-range pseudoknot?", "replies": [ { "id": 2552173, "postDate": "2023-12-07T09:06:40.060Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 2552270, "postDate": "2023-12-07T10:49:21.667Z", "content": "

Black rectangle

", "rawMarkdown": "Black rectangle" }, { "id": 2552309, "postDate": "2023-12-07T11:23:36.517Z", "content": "

@sroger got it; he posted a picture below.
\nI got it, too, but my accuracy is really low, so unless there is a massive shakeup, I'm not a competition hehe.

", "rawMarkdown": "@sroger got it; he posted a picture below.\nI got it, too, but my accuracy is really low, so unless there is a massive shakeup, I'm not a competition hehe.", "votes": 1 }, { "id": 2552409, "postDate": "2023-12-07T12:39:06.940Z", "content": "

Unfortunately I was not able to maintain that knot as I lowered cv/lb.

", "rawMarkdown": "Unfortunately I was not able to maintain that knot as I lowered cv/lb. ", "votes": 2 }, { "id": 2552411, "postDate": "2023-12-07T12:43:49.903Z", "content": "

Yes, is it true that you were able to maintain it untill you started to use bpp?

", "rawMarkdown": "Yes, is it true that you were able to maintain it untill you started to use bpp?\n" }, { "id": 2552413, "postDate": "2023-12-07T12:43:58.987Z", "content": "

Hmm, this is interesting. Do you have an idea why? (it's ok if you postpone your answer to the end of the competition)

", "rawMarkdown": "Hmm, this is interesting. Do you have an idea why? (it's ok if you postpone your answer to the end of the competition)" }, { "id": 2552449, "postDate": "2023-12-07T13:23:26.133Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 2552468, "postDate": "2023-12-07T13:46:48.930Z", "content": "

Very surprising…

", "rawMarkdown": "Very surprising..." }, { "id": 2552511, "postDate": "2023-12-07T14:17:11.630Z", "content": "

About software — it seems not as simple because bpp from eterna/contrafold can be used to predict pseudoknots. But yes, I think that they are anyway terrible at that.
\nUnfortunately, other matrices-producing algorithms haven't shown better score.

", "rawMarkdown": "About software — it seems not as simple because bpp from eterna/contrafold can be used to predict pseudoknots. But yes, I think that they are anyway terrible at that. \nUnfortunately, other matrices-producing algorithms haven't shown better score. " }, { "id": 2552528, "postDate": "2023-12-07T14:31:08.607Z", "content": "

Have you tried also the last-generation deep-learning bpp models like UFold/RFold?

", "rawMarkdown": "Have you tried also the last-generation deep-learning bpp models like UFold/RFold?" }, { "id": 2552538, "postDate": "2023-12-07T14:37:45.367Z", "content": "

Yes and its a huge disappointment. Maybe, more can be achieved by using model finetuning.

", "rawMarkdown": "Yes and its a huge disappointment. Maybe, more can be achieved by using model finetuning. \n", "votes": 1 } ] } ] }, { "id": 2542027, "postDate": "2023-11-29T02:36:39.497Z", "content": "

What's the issue with absolute positional encodings? Some people here have mentioned they don't think their model will generalize because they've used them. Are relative or rotary embeddings better in some way?

", "rawMarkdown": "What's the issue with absolute positional encodings? Some people here have mentioned they don't think their model will generalize because they've used them. Are relative or rotary embeddings better in some way?", "replies": [ { "id": 2542496, "postDate": "2023-11-29T09:59:23.170Z", "content": "

Since the test set contains longer sequences than the train set, the model doesn't 'know' how to deal with unseen-before positions and does not generalize, i.e., the generated pictures are not similar to the ones posted by @shujun717 above. As for other embeddings, I advise you to experiment and find out what works best.

", "rawMarkdown": "Since the test set contains longer sequences than the train set, the model doesn't 'know' how to deal with unseen-before positions and does not generalize, i.e., the generated pictures are not similar to the ones posted by @shujun717 above. As for other embeddings, I advise you to experiment and find out what works best." } ] }, { "id": 2541974, "postDate": "2023-11-29T01:16:23.643Z", "content": "

I am unable to follow this point:

\n
Additionally, you also want  make sure  your model  outputting reasonable predictions      regions  there  no training data. Currently, due   technical reasons, we cannot measure these positions  we might be able  resolve these limitations   final private test . We have noticed internally    our models under certain conditions like  output entirely s      regions, which should  be happening.\n
\n

E.g. if at the beginning the model has just nan from the data, why do we expect it to output a valid set of floats?

\n

I have two models, and just with change of depth in transformer I am seeing one model is outputting all zeros in beginning and another some constant float. The latter does have better LB and my val set accuracy. But it seems just just a chance.

", "rawMarkdown": "I am unable to follow this point:\n```\nAdditionally, you also want to make sure that your model is outputting reasonable predictions for the beginning and end regions where there is no training data. Currently, due to some technical reasons, we cannot measure these positions but we might be able to resolve these limitations in the final private test set. We have noticed internally that some of our models under certain conditions like to output entirely 0s for the beginning and end regions, which should not be happening.\n```\n\nE.g. if at the beginning the model has just nan from the data, why do we expect it to output a valid set of floats? \n\nI have two models, and just with change of depth in transformer I am seeing one model is outputting all zeros in beginning and another some constant float. The latter does have better LB and my val set accuracy. But it seems just just a chance. ", "replies": [ { "id": 2541976, "postDate": "2023-11-29T01:17:48.960Z", "content": "

\"\"

\n

\"\"

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6760%2F916b055e92a0457b367d14186654bdac%2F116_plot.png?generation=1701220652522549&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6760%2F38dc3df65fdb197bb532f83a2ae84320%2F124_plot.png?generation=1701220666076154&alt=media)", "replies": [ { "id": 2541979, "postDate": "2023-11-29T01:20:06.953Z", "content": "

Seeking some opinions on the plots - what aspect looks bad. It seems in plot-set-1, 2A3 looks smeared.
\nPlot set 2 looks better in some aspects.
\nDmS in plot-set-1 looks much crispier.

\n

Any other pointers?
\n(thanks in advance)

", "rawMarkdown": "Seeking some opinions on the plots - what aspect looks bad. It seems in plot-set-1, 2A3 looks smeared. \nPlot set 2 looks better in some aspects. \nDmS in plot-set-1 looks much crispier. \n\nAny other pointers? \n(thanks in advance)" }, { "id": 2541980, "postDate": "2023-11-29T01:21:20.257Z", "content": "

(I am struggling to understand the plot tbh)

", "rawMarkdown": "(I am struggling to understand the plot tbh)" } ] } ] }, { "id": 2538258, "postDate": "2023-11-26T00:33:30.490Z", "content": "

Anyone having sucess to convert this into a numerical metric?
\nafter a degree of generalization its very hard to compare the images

", "rawMarkdown": "Anyone having sucess to convert this into a numerical metric?\nafter a degree of generalization its very hard to compare the images", "replies": [ { "id": 2538470, "postDate": "2023-11-26T06:46:13.473Z", "content": "

It will be useless anyway, since this picture is also a prediction, not ground truth. So it is possible that your model will do better than theirs.

", "rawMarkdown": "It will be useless anyway, since this picture is also a prediction, not ground truth. So it is possible that your model will do better than theirs.", "votes": 2, "replies": [ { "id": 2538545, "postDate": "2023-11-26T08:48:59.073Z", "content": "

Ooh thanks!

", "rawMarkdown": "Ooh thanks!" } ] } ] }, { "id": 2488102, "postDate": "2023-10-19T03:25:30.650Z", "content": "

What do the x and y axis represent?

", "rawMarkdown": "What do the x and y axis represent?", "replies": [ { "id": 2490475, "postDate": "2023-10-20T17:54:55.957Z", "content": "

x axis represents position, while y axis represents sequence number. I have also added a plot in the main post visualizing sequence (AUGC) and therefore where the point mutations happen

", "rawMarkdown": "x axis represents position, while y axis represents sequence number. I have also added a plot in the main post visualizing sequence (AUGC) and therefore where the point mutations happen" } ] }, { "id": 2474942, "postDate": "2023-10-09T14:53:55.930Z", "content": "

Very interesting, thanks for sharing this and the plots!

", "rawMarkdown": "Very interesting, thanks for sharing this and the plots!" }, { "id": 2472724, "postDate": "2023-10-07T14:38:04.070Z", "content": "

\"\"

\n

Too blurry!

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14573416%2Fc171dba0da0fd0fbb23777bce343f8ab%2Fplot.png?generation=1696689349360396&alt=media)\n\nToo blurry!", "replies": [ { "id": 2529639, "postDate": "2023-11-18T12:16:35.493Z", "content": "

\"\"

\n

A bit of progress for me during this month, still far away from the ground truth.

\n

The score for this prediction is 0.15387 in the public lb, without training on the shared sequences in case anyone is wondering.

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14573416%2F9de73bab64e1da812cb4766ea85eca25%2Fprogress.png?generation=1700309493366166&alt=media)\n\nA bit of progress for me during this month, still far away from the ground truth.\n\nThe score for this prediction is 0.15387 in the public lb, without training on the shared sequences in case anyone is wondering.", "replies": [ { "id": 2541250, "postDate": "2023-11-28T10:29:58.207Z", "content": "

Would you mind sharing what helped you improve generalization?

\n

I'll share first: Sliding window surprisingly dosen't help too much for generalization, using rotary embeddings helped a bit but still far away… The biggest catch for generalization is how we're dealing with the data/embeddings though trying various augmentation techniques shifting the sequences randomly eg GAB -> ABG ->BGA , etc seemingly helps, not sure though.
\nhaven't implemented bpps yet not sure if it helps with generalization, i know that it should help with the score

", "rawMarkdown": "Would you mind sharing what helped you improve generalization?\n\nI'll share first: Sliding window surprisingly dosen't help too much for generalization, using rotary embeddings helped a bit but still far away... The biggest catch for generalization is how we're dealing with the data/embeddings though trying various augmentation techniques shifting the sequences randomly eg GAB -> ABG ->BGA , etc seemingly helps, not sure though.\nhaven't implemented bpps yet not sure if it helps with generalization, i know that it should help with the score" }, { "id": 2541411, "postDate": "2023-11-28T13:24:02.050Z", "content": "

I'm just new and learning as I go, will share it at end of the competition if I think it can add anything.

\n

About the sliding window, I'm not sure what is the benefit of using it with such short sequences, the main idea behind sliding window is to reduce the complexity from O(n^2) to O(n * w) where w is smaller than n but the RNA sequences of this competition are <= 457, so the normal attention can be computed without much trouble.

\n

\"\"

\n

\"\"

\n

Source: https://arxiv.org/pdf/2004.05150.pdf

\n

No benefit at 512 seqlen and n^2, I could be wrong, so please if anyone knows better correct me.

", "rawMarkdown": "I'm just new and learning as I go, will share it at end of the competition if I think it can add anything.\n\nAbout the sliding window, I'm not sure what is the benefit of using it with such short sequences, the main idea behind sliding window is to reduce the complexity from O(n^2) to O(n * w) where w is smaller than n but the RNA sequences of this competition are <= 457, so the normal attention can be computed without much trouble.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14573416%2Fc527f175c09c1b82b9e2424467ee8fd6%2FScreenshot%20from%202023-11-28%2014-19-09.png?generation=1701177743958059&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14573416%2Ffd303593fde0bfe1a4fb80da56e0888b%2FScreenshot%20from%202023-11-28%2014-12-33.png?generation=1701177168201962&alt=media)\n\nSource: https://arxiv.org/pdf/2004.05150.pdf\n\nNo benefit at 512 seqlen and n^2, I could be wrong, so please if anyone knows better correct me." }, { "id": 2541427, "postDate": "2023-11-28T13:38:17.120Z", "content": "

Sliding window can be seen as a form of restricting the model to see more than W tokens at a time, there are papers that claim that transformers with casual mask learn positional information from the amount of tokens being processed in attention (unmasked tokens) [e.g. when computing single-query/multiple-keys attention, depending on the amount of keys we get different smoothness of attention scores, I guess]

\n

It can be useful in this problem, since in train set Lmax=206, and in test Lmax=457

\n

My guess was if I am to use sliding window attention, my generalization plots would become closer to the ones Shujun shared, but it was not the case, they are still \"too sharp\"

", "rawMarkdown": "Sliding window can be seen as a form of restricting the model to see more than W tokens at a time, there are papers that claim that transformers with casual mask learn positional information from the amount of tokens being processed in attention (unmasked tokens) [e.g. when computing single-query/multiple-keys attention, depending on the amount of keys we get different smoothness of attention scores, I guess]\n\nIt can be useful in this problem, since in train set Lmax=206, and in test Lmax=457\n\nMy guess was if I am to use sliding window attention, my generalization plots would become closer to the ones Shujun shared, but it was not the case, they are still \"too sharp\"", "votes": 2 }, { "id": 2541455, "postDate": "2023-11-28T14:02:45.123Z", "content": "

Just to clarify if I'm understanding your point, the idea is to reduce the attention to smaller patchs so it can learn from smaller sequences and hope it will generalize in the longer sequence by attending to smaller patchs and sliding it.

\n

Isn't that the reverse idea of chopping 'very' long sequences and attending to smaller patches hoping to capture most of the important information?

\n

Is a cool idea slime.

", "rawMarkdown": "Just to clarify if I'm understanding your point, the idea is to reduce the attention to smaller patchs so it can learn from smaller sequences and hope it will generalize in the longer sequence by attending to smaller patchs and sliding it.\n\nIsn't that the reverse idea of chopping 'very' long sequences and attending to smaller patches hoping to capture most of the important information?\n\nIs a cool idea slime.", "votes": 1 }, { "id": 2541864, "postDate": "2023-11-28T21:27:41.623Z", "content": "

I think, there was a very valid point that the picture from the original post is also a **prediction **. So it is not a gold standard. But sliding window seems to be useless, yes.

", "rawMarkdown": "I think, there was a very valid point that the picture from the original post is also a **prediction **. So it is not a gold standard. But sliding window seems to be useless, yes. ", "votes": 1 }, { "id": 2542483, "postDate": "2023-11-29T09:47:26.417Z", "content": "

Oh true, is a prediction, thanks for pointing it out.

", "rawMarkdown": "Oh true, is a prediction, thanks for pointing it out." } ] } ] }, { "id": 2547182, "postDate": "2023-12-03T08:58:20.580Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2544767, "postDate": "2023-12-01T04:28:09.573Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2544764, "postDate": "2023-12-01T04:27:28.890Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2534742, "postDate": "2023-11-22T20:58:55.103Z", "content": "

that's great! thank you

", "rawMarkdown": "that's great! thank you" }, { "id": 2529727, "postDate": "2023-11-18T13:54:56.587Z", "content": "

Thanks for this good work

", "rawMarkdown": "Thanks for this good work" } ], "comments": [ { "id": 2544569, "author_name": "Shujun", "author_url": "", "post_date": "2023-11-30T23:42:47.880000", "content": "

Hi Kagglers, we have another interesting test of generality with a sequence that was in the last CASP competition: R1138v1 or 7PTL in protein data bank (https://www.rcsb.org/structure/7PTL ). We have extracted the secondary structure from the experimentally resolved 3D structure:\"\"

\n

and looked at our model’s predictions using the mutate-and-map technique with our model:
\n\"\"

\n

\n

While we definitely see the main secondary structure, the kissing loops above the diagonal between the 5' and 3' 'halves' are not clearly present (circled in cyan) and there are also some short stems missing (circled in red)

\n

\"\"

\n

You can check your predictions by making predictions for the sequences in the following CSV file), where the sequence ids are in the format of V1138v1_m2_{mutated_position}. And I have prepared bpp files for these. And after making predictions you can visualize with the following code:

\n
m2_preds=(m2_sequences) #x720x2\n\nplt()\nplt()\nplt(m2_preds,vmin=,vmax=,cmap=)\nplt()\nplt()\nplt(m2_preds,vmin=,vmax=,cmap=)\n
\n

We’re excited to see what you have and if your model is generalizing better!

", "votes": 9, "replies": [ { "id": 2544572, "author_name": "slime", "author_url": "", "post_date": "2023-11-30T23:47:57.213000", "content": "

The link directs nowhere for now, can you make dataset public? Excited to try our models on these sequences c:

", "votes": 2, "replies": [] }, { "id": 2544573, "author_name": "", "author_url": "", "post_date": "2023-11-30T23:48:05.633000", "content": "", "votes": 0, "replies": [] }, { "id": 2544582, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-01T00:04:02.757000", "content": "

@martynoveduard fixed. Can you see it now?

", "votes": 1, "replies": [ { "id": 2544583, "author_name": "slime", "author_url": "", "post_date": "2023-12-01T00:05:12.310000", "content": "

yup, thanks

", "votes": 1, "replies": [] } ] }, { "id": 2544608, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-01T01:09:26.333000", "content": "

There was a mistake in my post regarding the kissing loops and I have updated the post and annotated secondary structure plot

", "votes": 1, "replies": [] }, { "id": 2544622, "author_name": "Laura Romar", "author_url": "", "post_date": "2023-12-01T01:31:33.877000", "content": "

Could you please provide the bpp files of the sequences ?

", "votes": 0, "replies": [ { "id": 2544632, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-01T01:48:08.160000", "content": "

Yeah see here: https://www.kaggle.com/datasets/shujun717/r1138-bpp

", "votes": 2, "replies": [] }, { "id": 2544646, "author_name": "Laura Romar", "author_url": "", "post_date": "2023-12-01T02:07:15.123000", "content": "

Thanks a lot !

", "votes": 0, "replies": [] } ] }, { "id": 2545058, "author_name": "greySnow", "author_url": "", "post_date": "2023-12-01T09:04:55.683000", "content": "

\"This

", "votes": 0, "replies": [ { "id": 2545116, "author_name": "greySnow", "author_url": "", "post_date": "2023-12-01T10:00:20.753000", "content": "

I tried to take the mean of 2A3 and DMS for a better picture. I can actually see hints of this middle kissing loop. But the side ones are too difficult… perhaps the top LB models would do better there.\"\"

", "votes": 0, "replies": [] } ] }, { "id": 2545614, "author_name": "Iafoss", "author_url": "", "post_date": "2023-12-01T15:56:03.477000", "content": "

Thanks for another example. It seems my model gets signs of all lines except the ones in the middle you outline with cyan(
\n\"\"

", "votes": 1, "replies": [] } ] }, { "id": 2468976, "author_name": "Iafoss", "author_url": "", "post_date": "2023-10-06T03:32:47.713000", "content": "

Nice post
\n\"\"

", "votes": 7, "replies": [] }, { "id": 2497319, "author_name": "sroger", "author_url": "", "post_date": "2023-10-24T15:13:38.733000", "content": "

Hopefully similar enough
\n\"\"

", "votes": 6, "replies": [ { "id": 2499257, "author_name": "junseonglee11", "author_url": "", "post_date": "2023-10-25T22:27:41.210000", "content": "

Your's looks really similar!

", "votes": 0, "replies": [] }, { "id": 2513861, "author_name": "greySnow", "author_url": "", "post_date": "2023-11-05T19:47:20.967000", "content": "

It's very impressive! Do you mind sharing the LB score of this model?

", "votes": 0, "replies": [ { "id": 2514136, "author_name": "sroger", "author_url": "", "post_date": "2023-11-06T04:54:33.823000", "content": "

The LB score is actually pretty bad ~0.17 but it was because of a bug

", "votes": 1, "replies": [] } ] } ] }, { "id": 2544552, "author_name": "tattaka", "author_url": "", "post_date": "2023-11-30T22:50:01.543000", "content": "

Plot by current best submission
\n\"\"

\n

I still don't know how to interpret the plots.

", "votes": 4, "replies": [] }, { "id": 2467044, "author_name": "MT", "author_url": "", "post_date": "2023-10-04T09:04:43.060000", "content": "

I think this topic deserves a pin! I was wondering if the model you used was trained on the same data we have.

", "votes": 3, "replies": [ { "id": 2467846, "author_name": "Shujun", "author_url": "", "post_date": "2023-10-04T23:01:57.430000", "content": "

Yes it was!

", "votes": 3, "replies": [] }, { "id": 2471884, "author_name": "Rhiju Das", "author_url": "", "post_date": "2023-10-06T17:58:03.313000", "content": "

Topic is pinned! Thanks for suggesting.

", "votes": 2, "replies": [] } ] }, { "id": 2551957, "author_name": "Iafoss", "author_url": "", "post_date": "2023-12-07T05:03:40.100000", "content": "

@shujun717 I'm just curious in case if you are allowed to share this information, were you able to resolve the technical difficulties with measuring the activity at the sequence ends, or do you still plan to evaluate only the middle parts of the sequences? Thanks.

", "votes": 2, "replies": [ { "id": 2553843, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-08T14:47:27.777000", "content": "

I think sequence ends were not used for scoring

", "votes": 1, "replies": [] } ] }, { "id": 2473265, "author_name": "DECEM", "author_url": "", "post_date": "2023-10-08T06:01:29.310000", "content": "

Great Post!
\n\"plot\"

", "votes": 1, "replies": [] }, { "id": 2489571, "author_name": "NathanTuttle", "author_url": "", "post_date": "2023-10-20T04:46:23.360000", "content": "

I am still confused. Not sure what the arrangement is. Are these just lists of reactivity by nt position?

", "votes": 0, "replies": [ { "id": 2490474, "author_name": "Shujun", "author_url": "", "post_date": "2023-10-20T17:54:43.980000", "content": "

Yes I have added an explanation for this! See the added plot and last paragraph

", "votes": 0, "replies": [ { "id": 2490680, "author_name": "NathanTuttle", "author_url": "", "post_date": "2023-10-20T23:21:33.423000", "content": "

Thank you. How did you determine the mutation?

", "votes": 1, "replies": [] } ] } ] }, { "id": 2465687, "author_name": "slime", "author_url": "", "post_date": "2023-10-03T07:36:09.713000", "content": "

Thank you for providing this valuable example!

\n

I'll give plots of two of my models here, - not generalizible and somewhat generalizable

", "votes": 2, "replies": [ { "id": 2465690, "author_name": "slime", "author_url": "", "post_date": "2023-10-03T07:38:42.970000", "content": "

This one I thought won't generalize to private LB

\n

\"\"

", "votes": 2, "replies": [] }, { "id": 2465692, "author_name": "slime", "author_url": "", "post_date": "2023-10-03T07:42:11.820000", "content": "

I don't see the black rectangle here :D But the plot overall resembles yours

\n

\"\"

", "votes": 4, "replies": [ { "id": 2467875, "author_name": "Shujun", "author_url": "", "post_date": "2023-10-05T00:57:05.717000", "content": "

Thanks for posting. Any idea why the first model does not generalize well? It's fine if you don't want to share though.

", "votes": 2, "replies": [] }, { "id": 2468063, "author_name": "MT", "author_url": "", "post_date": "2023-10-05T06:41:32.447000", "content": "

My early models were GRUs and LSTMs. I had two issue with those models.
\n1) The outputs were preferentially zeros at start and end of the sequences.
\n2) I was struggling to generalize for long sequences.
\nThen, I implemented the model proposed by @IAFOSS https://www.kaggle.com/code/iafoss/rna-starter-submission-0-186-lb and the issues disappeared

", "votes": 3, "replies": [] }, { "id": 2468100, "author_name": "slime", "author_url": "", "post_date": "2023-10-05T07:14:51.937000", "content": "
\n

Any idea why the first model does not generalize well?

\n
\n

Sure! First model is using absolute positional encoding and was not trained on longer sequences.

\n

We've got to figure out how to align feature spaces of train and test distributions so that we can predict something like you shared in the post

", "votes": 6, "replies": [] }, { "id": 2474180, "author_name": "Shujun", "author_url": "", "post_date": "2023-10-09T03:57:04.610000", "content": "

@martynoveduard what do you mean training on longer sequences? Most of the sequences in train are 177

", "votes": 1, "replies": [] }, { "id": 2474197, "author_name": "slime", "author_url": "", "post_date": "2023-10-09T04:20:21.983000", "content": "

When I talk about longer sequences, I'm referring to sequences with a length greater than 206. In the private test, it's apparent that the model hasn't learned positional encoding for tokens beyond the 206th position. Because of this, I consider these tokens as out-of-distribution (OOD) samples for the model and as can be seen from the first plot, predictions for these positions are different from the previous positions.

", "votes": 1, "replies": [] }, { "id": 2502078, "author_name": "Frankly", "author_url": "", "post_date": "2023-10-27T23:47:58.583000", "content": "

Forgive my stupidity but, how can you know that in the private test your model doesn't work on beyond 206th. The private test data has not been released.

", "votes": 0, "replies": [] }, { "id": 2502338, "author_name": "slime", "author_url": "", "post_date": "2023-10-28T06:00:25.833000", "content": "

I just compared (visually) the plots I obtained from the script provided by Shujun for my models and hosts model, as you can see there are sequences of length >400, therefore its a part of the private data, and it's provided to us for convinience so we can sanity check our models on sequences with greater lengths

", "votes": 1, "replies": [] }, { "id": 2539283, "author_name": "blurrylogic", "author_url": "", "post_date": "2023-11-26T23:57:05.847000", "content": "

Hello!
\nthe model does actually generalize quite worse than yours. Did you do anything specific to improve generalization or does it generalize better with accuracy improvement?
\nIf your willing to share i'll be glad to hear what you did to improve generalization…
\nI've tried implementing sliding windows and similar embedding stuff on top but haven't been able to generalize as well

\n

\"@iafoss\"

", "votes": 0, "replies": [] }, { "id": 2539286, "author_name": "blurrylogic", "author_url": "", "post_date": "2023-11-26T23:59:45.200000", "content": "

implementing sliding window on top
\n\"\"

", "votes": 0, "replies": [] }, { "id": 2540819, "author_name": "", "author_url": "", "post_date": "2023-11-28T03:25:10.050000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2465562, "author_name": "Gary", "author_url": "", "post_date": "2023-10-03T05:14:30.540000", "content": "

It is not easy.

", "votes": 2, "replies": [] }, { "id": 2552073, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-07T07:06:08.943000", "content": "

Did anybody succeed in predicting long-range pseudoknot?

", "votes": 0, "replies": [ { "id": 2552173, "author_name": "", "author_url": "", "post_date": "2023-12-07T09:06:40.060000", "content": "", "votes": 0, "replies": [ { "id": 2552270, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-07T10:49:21.667000", "content": "

Black rectangle

", "votes": 0, "replies": [] }, { "id": 2552309, "author_name": "greySnow", "author_url": "", "post_date": "2023-12-07T11:23:36.517000", "content": "

@sroger got it; he posted a picture below.
\nI got it, too, but my accuracy is really low, so unless there is a massive shakeup, I'm not a competition hehe.

", "votes": 1, "replies": [] }, { "id": 2552409, "author_name": "sroger", "author_url": "", "post_date": "2023-12-07T12:39:06.940000", "content": "

Unfortunately I was not able to maintain that knot as I lowered cv/lb.

", "votes": 2, "replies": [] }, { "id": 2552411, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-07T12:43:49.903000", "content": "

Yes, is it true that you were able to maintain it untill you started to use bpp?

", "votes": 0, "replies": [] }, { "id": 2552413, "author_name": "greySnow", "author_url": "", "post_date": "2023-12-07T12:43:58.987000", "content": "

Hmm, this is interesting. Do you have an idea why? (it's ok if you postpone your answer to the end of the competition)

", "votes": 0, "replies": [] }, { "id": 2552449, "author_name": "", "author_url": "", "post_date": "2023-12-07T13:23:26.133000", "content": "", "votes": 2, "replies": [] }, { "id": 2552468, "author_name": "junseonglee11", "author_url": "", "post_date": "2023-12-07T13:46:48.930000", "content": "

Very surprising…

", "votes": 0, "replies": [] }, { "id": 2552511, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-07T14:17:11.630000", "content": "

About software — it seems not as simple because bpp from eterna/contrafold can be used to predict pseudoknots. But yes, I think that they are anyway terrible at that.
\nUnfortunately, other matrices-producing algorithms haven't shown better score.

", "votes": 0, "replies": [] }, { "id": 2552528, "author_name": "greySnow", "author_url": "", "post_date": "2023-12-07T14:31:08.607000", "content": "

Have you tried also the last-generation deep-learning bpp models like UFold/RFold?

", "votes": 0, "replies": [] }, { "id": 2552538, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-07T14:37:45.367000", "content": "

Yes and its a huge disappointment. Maybe, more can be achieved by using model finetuning.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2542027, "author_name": "Abhishek Shah", "author_url": "", "post_date": "2023-11-29T02:36:39.497000", "content": "

What's the issue with absolute positional encodings? Some people here have mentioned they don't think their model will generalize because they've used them. Are relative or rotary embeddings better in some way?

", "votes": 0, "replies": [ { "id": 2542496, "author_name": "greySnow", "author_url": "", "post_date": "2023-11-29T09:59:23.170000", "content": "

Since the test set contains longer sequences than the train set, the model doesn't 'know' how to deal with unseen-before positions and does not generalize, i.e., the generated pictures are not similar to the ones posted by @shujun717 above. As for other embeddings, I advise you to experiment and find out what works best.

", "votes": 0, "replies": [] } ] }, { "id": 2541974, "author_name": "FdotRK", "author_url": "", "post_date": "2023-11-29T01:16:23.643000", "content": "

I am unable to follow this point:

\n
Additionally, you also want  make sure  your model  outputting reasonable predictions      regions  there  no training data. Currently, due   technical reasons, we cannot measure these positions  we might be able  resolve these limitations   final private test . We have noticed internally    our models under certain conditions like  output entirely s      regions, which should  be happening.\n
\n

E.g. if at the beginning the model has just nan from the data, why do we expect it to output a valid set of floats?

\n

I have two models, and just with change of depth in transformer I am seeing one model is outputting all zeros in beginning and another some constant float. The latter does have better LB and my val set accuracy. But it seems just just a chance.

", "votes": 0, "replies": [ { "id": 2541976, "author_name": "FdotRK", "author_url": "", "post_date": "2023-11-29T01:17:48.960000", "content": "

\"\"

\n

\"\"

", "votes": 0, "replies": [ { "id": 2541979, "author_name": "FdotRK", "author_url": "", "post_date": "2023-11-29T01:20:06.953000", "content": "

Seeking some opinions on the plots - what aspect looks bad. It seems in plot-set-1, 2A3 looks smeared.
\nPlot set 2 looks better in some aspects.
\nDmS in plot-set-1 looks much crispier.

\n

Any other pointers?
\n(thanks in advance)

", "votes": 0, "replies": [] }, { "id": 2541980, "author_name": "FdotRK", "author_url": "", "post_date": "2023-11-29T01:21:20.257000", "content": "

(I am struggling to understand the plot tbh)

", "votes": 0, "replies": [] } ] } ] }, { "id": 2538258, "author_name": "blurrylogic", "author_url": "", "post_date": "2023-11-26T00:33:30.490000", "content": "

Anyone having sucess to convert this into a numerical metric?
\nafter a degree of generalization its very hard to compare the images

", "votes": 0, "replies": [ { "id": 2538470, "author_name": "greySnow", "author_url": "", "post_date": "2023-11-26T06:46:13.473000", "content": "

It will be useless anyway, since this picture is also a prediction, not ground truth. So it is possible that your model will do better than theirs.

", "votes": 2, "replies": [ { "id": 2538545, "author_name": "blurrylogic", "author_url": "", "post_date": "2023-11-26T08:48:59.073000", "content": "

Ooh thanks!

", "votes": 0, "replies": [] } ] } ] }, { "id": 2488102, "author_name": "NathanTuttle", "author_url": "", "post_date": "2023-10-19T03:25:30.650000", "content": "

What do the x and y axis represent?

", "votes": 0, "replies": [ { "id": 2490475, "author_name": "Shujun", "author_url": "", "post_date": "2023-10-20T17:54:55.957000", "content": "

x axis represents position, while y axis represents sequence number. I have also added a plot in the main post visualizing sequence (AUGC) and therefore where the point mutations happen

", "votes": 0, "replies": [] } ] }, { "id": 2474942, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2023-10-09T14:53:55.930000", "content": "

Very interesting, thanks for sharing this and the plots!

", "votes": 0, "replies": [] }, { "id": 2472724, "author_name": "Antonio Félix", "author_url": "", "post_date": "2023-10-07T14:38:04.070000", "content": "

\"\"

\n

Too blurry!

", "votes": 0, "replies": [ { "id": 2529639, "author_name": "Antonio Félix", "author_url": "", "post_date": "2023-11-18T12:16:35.493000", "content": "

\"\"

\n

A bit of progress for me during this month, still far away from the ground truth.

\n

The score for this prediction is 0.15387 in the public lb, without training on the shared sequences in case anyone is wondering.

", "votes": 0, "replies": [ { "id": 2541250, "author_name": "Dhruv Dhilla", "author_url": "", "post_date": "2023-11-28T10:29:58.207000", "content": "

Would you mind sharing what helped you improve generalization?

\n

I'll share first: Sliding window surprisingly dosen't help too much for generalization, using rotary embeddings helped a bit but still far away… The biggest catch for generalization is how we're dealing with the data/embeddings though trying various augmentation techniques shifting the sequences randomly eg GAB -> ABG ->BGA , etc seemingly helps, not sure though.
\nhaven't implemented bpps yet not sure if it helps with generalization, i know that it should help with the score

", "votes": 0, "replies": [] }, { "id": 2541411, "author_name": "Antonio Félix", "author_url": "", "post_date": "2023-11-28T13:24:02.050000", "content": "

I'm just new and learning as I go, will share it at end of the competition if I think it can add anything.

\n

About the sliding window, I'm not sure what is the benefit of using it with such short sequences, the main idea behind sliding window is to reduce the complexity from O(n^2) to O(n * w) where w is smaller than n but the RNA sequences of this competition are <= 457, so the normal attention can be computed without much trouble.

\n

\"\"

\n

\"\"

\n

Source: https://arxiv.org/pdf/2004.05150.pdf

\n

No benefit at 512 seqlen and n^2, I could be wrong, so please if anyone knows better correct me.

", "votes": 0, "replies": [] }, { "id": 2541427, "author_name": "slime", "author_url": "", "post_date": "2023-11-28T13:38:17.120000", "content": "

Sliding window can be seen as a form of restricting the model to see more than W tokens at a time, there are papers that claim that transformers with casual mask learn positional information from the amount of tokens being processed in attention (unmasked tokens) [e.g. when computing single-query/multiple-keys attention, depending on the amount of keys we get different smoothness of attention scores, I guess]

\n

It can be useful in this problem, since in train set Lmax=206, and in test Lmax=457

\n

My guess was if I am to use sliding window attention, my generalization plots would become closer to the ones Shujun shared, but it was not the case, they are still \"too sharp\"

", "votes": 2, "replies": [] }, { "id": 2541455, "author_name": "Antonio Félix", "author_url": "", "post_date": "2023-11-28T14:02:45.123000", "content": "

Just to clarify if I'm understanding your point, the idea is to reduce the attention to smaller patchs so it can learn from smaller sequences and hope it will generalize in the longer sequence by attending to smaller patchs and sliding it.

\n

Isn't that the reverse idea of chopping 'very' long sequences and attending to smaller patches hoping to capture most of the important information?

\n

Is a cool idea slime.

", "votes": 1, "replies": [] }, { "id": 2541864, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-11-28T21:27:41.623000", "content": "

I think, there was a very valid point that the picture from the original post is also a **prediction **. So it is not a gold standard. But sliding window seems to be useless, yes.

", "votes": 1, "replies": [] }, { "id": 2542483, "author_name": "Antonio Félix", "author_url": "", "post_date": "2023-11-29T09:47:26.417000", "content": "

Oh true, is a prediction, thanks for pointing it out.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2547182, "author_name": "", "author_url": "", "post_date": "2023-12-03T08:58:20.580000", "content": "", "votes": 0, "replies": [] }, { "id": 2544767, "author_name": "", "author_url": "", "post_date": "2023-12-01T04:28:09.573000", "content": "", "votes": 0, "replies": [] }, { "id": 2544764, "author_name": "", "author_url": "", "post_date": "2023-12-01T04:27:28.890000", "content": "", "votes": 0, "replies": [] }, { "id": 2534742, "author_name": "max regis", "author_url": "", "post_date": "2023-11-22T20:58:55.103000", "content": "

that's great! thank you

", "votes": 0, "replies": [] }, { "id": 2529727, "author_name": "Rewida shabaan mohamed", "author_url": "", "post_date": "2023-11-18T13:54:56.587000", "content": "

Thanks for this good work

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2465490": "\nAn important task in this competition is for models to generalize to long RNA sequences, because many RNAs of interest can be much longer than what is available in the training set. Therefore, we have included sequences that are much longer (as long as 457 nt) in the private test set. A good way to check your model's performance on these long sequences is to visually inspect a subset of sequences which we call [mutate and map](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4080707/) (used to infer base pairings by point mutations). We have an example of these using predictions from a model that gives reasonable predictions on a subset of 457nt sequences with known structure. Note that the red arrow points to a region of known long range pseudo-knot (the darker rectangle) which have been impossible to predict with prior modeling methods. When exposed following point mutations, the long range pseudo-knot region has increased reactivity.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2Fbda49dc7b259bcec466b575c5a75d97c%2Fexample.png?generation=1696301353614738&alt=media)\n\nTo plot these:\n```\nimport polars as pl\nimport matplotlib.pyplot as plt\n\n#read your sub here\ndf=pl.read_csv(\"sub.csv\")\n\n#some parameters\nfont_size=6\nid1=269545321\nid2=269724007\nreshape1=391\nreshape2=457\n\n#get predictions\npred_DMS=df[id1:id2+1]['reactivity_DMS_MaP'].to_numpy().reshape(reshape1,reshape2)\npred_2A3=df[id1:id2+1]['reactivity_2A3_MaP'].to_numpy().reshape(reshape1,reshape2)\n\n#plot mutate and map\nfig = plt.figure()\nplt.subplot(121)\nplt.title(f'reactivity_DMS_MaP', fontsize=font_size)\nplt.imshow(pred_DMS,vmin=0,vmax=1, cmap='gray_r')\nplt.subplot(122)\nplt.title(f'reactivity_2A3_MaP', fontsize=font_size)\nplt.imshow(pred_2A3,vmin=0,vmax=1, cmap='gray_r')\n\nplt.tight_layout()\nplt.savefig(f\"plot.png\",dpi=500)\nplt.clf()\nplt.close()\n```\n\nAdditionally, you also want to make sure that your model is outputting reasonable predictions for the beginning and end regions where there is no training data. Currently, due to some technical reasons, we cannot measure these positions but we might be able to resolve these limitations in the final private test set. We have noticed internally that some of our models under certain conditions like to output entirely 0s for the beginning and end regions, which should not be happening.\n\nFor anyone confused about how these mutations work, I have added another plot visualizing where these mutations occur. From position 27 to 416, the mutations at each position is one of {'A->U', 'C->G', 'G->C', 'U->A'}. Therefore, you can see a diagonal line in the middle. Note that the beginning and end regions are constant, while right before the end, there's also a bar code region unique to each sequence. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F633626f460d2a563a3757e5e003529e8%2FTribozyme_point_mutations.png?generation=1697823756980699&alt=media)\n\n", "2544569": "Hi Kagglers, we have another interesting test of generality with a sequence that was in the last CASP competition: R1138v1 or 7PTL in protein data bank (https://www.rcsb.org/structure/7PTL ). We have extracted the secondary structure from the experimentally resolved 3D structure:![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F592eb93db2a8a6be01f8f4daaf0ba369%2F7PTL.png?generation=1701387560513990&alt=media)\n\nand looked at our model’s predictions using the mutate-and-map technique with our model:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F1bd6d46561d3f9c3b7869e2f28fcf9f6%2FR1138v1.png?generation=1701387579567164&alt=media)\n\n~~While we definitely see the main secondary structure, the kissing loops above the diagonal (in cyan) between the 5' and 3' 'halves' are not clearly present (circled in cyan)~~\n\nWhile we definitely see the main secondary structure, the kissing loops above the diagonal between the 5' and 3' 'halves' are not clearly present (circled in cyan) and there are also some short stems missing (circled in red)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3355848%2F929d70a5ec679273b52565cbf37ee7c7%2F7PTL_annotated.png?generation=1701392197627840&alt=media)\n\n\nYou can check your predictions by making predictions for the sequences in the following [CSV file](https://www.kaggle.com/datasets/shujun717/r1138v1-m2)), where the sequence ids are in the format of V1138v1_m2_{mutated_position}. And I have prepared [bpp files](https://www.kaggle.com/datasets/shujun717/r1138-bpp) for these. And after making predictions you can visualize with the following code:\n\n```\nm2_preds=model(m2_sequences) #721x720x2\n\nplt.subplot(121)\nplt.title('2A3')\nplt.imshow(m2_preds[:,:,0],vmin=0,vmax=1,cmap='gray_r')\nplt.subplot(122)\nplt.title('DMS')\nplt.imshow(m2_preds[:,:,1],vmin=0,vmax=1,cmap='gray_r')\n```\n\nWe’re excited to see what you have and if your model is generalizing better!\n\n", "2468976": "Nice post\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1212661%2F5c3b1d7781037565087d814379e80f37%2Fplot.png?generation=1696563122851918&alt=media)", "2497319": "Hopefully similar enough\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3586013%2Ff0719545c5889b67ddba45f7da00a3bf%2Fplot0.png?generation=1698160337092019&alt=media)", "2544552": "Plot by current best submission\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1745801%2F685f90a707c237413455e9757b3a8355%2Fplot.png?generation=1701385653066069&alt=media)\n\nI still don't know how to interpret the plots.", "2467044": "I think this topic deserves a pin! I was wondering if the model you used was trained on the same data we have.", "2551957": "@shujun717 I'm just curious in case if you are allowed to share this information, were you able to resolve the technical difficulties with measuring the activity at the sequence ends, or do you still plan to evaluate only the middle parts of the sequences? Thanks.", "2473265": "Great Post!\n![plot](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6491363%2F13f80753ed9d37e3528d7b3a93fc0bc0%2Fplot.png?generation=1696744827715873&alt=media)", "2489571": "I am still confused. Not sure what the arrangement is. Are these just lists of reactivity by nt position?", "2465687": "Thank you for providing this valuable example!\n\nI'll give plots of two of my models here, - not generalizible and somewhat generalizable", "2465562": "It is not easy.", "2552073": "Did anybody succeed in predicting long-range pseudoknot?", "2542027": "What's the issue with absolute positional encodings? Some people here have mentioned they don't think their model will generalize because they've used them. Are relative or rotary embeddings better in some way?", "2541974": "I am unable to follow this point:\n```\nAdditionally, you also want to make sure that your model is outputting reasonable predictions for the beginning and end regions where there is no training data. Currently, due to some technical reasons, we cannot measure these positions but we might be able to resolve these limitations in the final private test set. We have noticed internally that some of our models under certain conditions like to output entirely 0s for the beginning and end regions, which should not be happening.\n```\n\nE.g. if at the beginning the model has just nan from the data, why do we expect it to output a valid set of floats? \n\nI have two models, and just with change of depth in transformer I am seeing one model is outputting all zeros in beginning and another some constant float. The latter does have better LB and my val set accuracy. But it seems just just a chance. ", "2538258": "Anyone having sucess to convert this into a numerical metric?\nafter a degree of generalization its very hard to compare the images", "2488102": "What do the x and y axis represent?", "2474942": "Very interesting, thanks for sharing this and the plots!", "2472724": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14573416%2Fc171dba0da0fd0fbb23777bce343f8ab%2Fplot.png?generation=1696689349360396&alt=media)\n\nToo blurry!", "2547182": "", "2544767": "", "2544764": "", "2534742": "that's great! thank you", "2529727": "Thanks for this good work" } }