![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1233385%2Fef5a1e8baadc56507bee658eb624c633%2FScreenshot%20from%202023-12-08%2023-14-44.png?generation=1702046810306490&alt=media\")
er. It just cut a certain length sequence from the original sequence.First of all, I would like to thank all the participants in the competition. My solution is a cheap transformer that does not require bpp-matrix and can learn and predict with poor computational resources named \"Cheap Transformers\"
\nMy starting point is Iafoss's amazing work where the reactivities are predicted by simple transformers.
\nIt works very well in cross validation in random fold splitting but gives poor performance when train and validation data is split by length of sequence like following.
\nshort_indices = df[df[]..() < ].index.tolist()\nlong_indices = df[df[]..() > ].index.tolist()\nTherefore, I need another architecture to generalize to long sequences.
\nThis is an architecture of cheap transformer
\ner. It just cut a certain length sequence from the original sequence.
When regression, it cuts the sequence in multiple ways and ensemble the prediction results for each character. The ensemble is important. It improves cv-score at about 0.005.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1233385%2Fbbaf5eee4d3b85a8c3ce62c7bc6cf34f%2FScreenshot%20from%202023-12-08%2023-14-52.png?generation=1702046847620018&alt=media\")
This architecture uses short sequences, reduces training time, and requires less GPU memory during execution. Therefore, it runs fast enough even with the GPU of a gaming laptop.
\nI prepared multiple models with cut sequence lengths of 64, 96, and 128 and with various transformer's parameters and created an ensemble. Pseudo labeling was also performed, but it had little effect on LB. As you can see the ensemble contributed LB but its effect is not so big, therefore, using single model would be the most practival.
\n| Method | \nPublic | \nPrivate | \n
|---|---|---|
| Iafoss's architecture | \n0.15669 | \n0.20204 | \n
| Cheap transformers (len=64) | \n0.15635 | \n0.15661 | \n
| Cheap transformers (len=96) | \n0.15458 | \n0.15445 | \n
| Cheap transformers (len=128) | \n0.15518 | \n0.15625 | \n
| Public best (ensemble with Iafoss's arch) | \n0.14716 | \n0.18222 | \n
| ensemble | \n0.15271 | \n0.15346 | \n
First of all, I would like to thank all the participants in the competition. My solution is a cheap transformer that does not require bpp-matrix and can learn and predict with poor computational resources named \"Cheap Transformers\"
\nMy starting point is Iafoss's amazing work where the reactivities are predicted by simple transformers.
\nIt works very well in cross validation in random fold splitting but gives poor performance when train and validation data is split by length of sequence like following.
\nshort_indices = df[df[]..() < ].index.tolist()\nlong_indices = df[df[]..() > ].index.tolist()\nTherefore, I need another architecture to generalize to long sequences.
\nThis is an architecture of cheap transformer
\ner. It just cut a certain length sequence from the original sequence.
When regression, it cuts the sequence in multiple ways and ensemble the prediction results for each character. The ensemble is important. It improves cv-score at about 0.005.
\n
This architecture uses short sequences, reduces training time, and requires less GPU memory during execution. Therefore, it runs fast enough even with the GPU of a gaming laptop.
\nI prepared multiple models with cut sequence lengths of 64, 96, and 128 and with various transformer's parameters and created an ensemble. Pseudo labeling was also performed, but it had little effect on LB. As you can see the ensemble contributed LB but its effect is not so big, therefore, using single model would be the most practival.
\n| Method | \nPublic | \nPrivate | \n
|---|---|---|
| Iafoss's architecture | \n0.15669 | \n0.20204 | \n
| Cheap transformers (len=64) | \n0.15635 | \n0.15661 | \n
| Cheap transformers (len=96) | \n0.15458 | \n0.15445 | \n
| Cheap transformers (len=128) | \n0.15518 | \n0.15625 | \n
| Public best (ensemble with Iafoss's arch) | \n0.14716 | \n0.18222 | \n
| ensemble | \n0.15271 | \n0.15346 | \n
Very similar with the approach I discarted but that obtained best private scoring. With a small difference. I'm giving it a second chance while trying to remember the particular details of that submission. I will prepare the results and so we can compare them.
", "rawMarkdown": "Very similar with the approach I discarted but that obtained best private scoring. With a small difference. I'm giving it a second chance while trying to remember the particular details of that submission. I will prepare the results and so we can compare them.", "votes": null }, { "id": "2554219", "postDate": "12/08/2023 23:12:28", "content": "Thank you for your comment. As far as I know, the following point is important to use this architecture.
\nIt improves CV score at approximately 0.01. I tried other libraries such as U-Fold, R-fold, Vienna RNA etc but none of them worked well.
\nToo long/short length of sequence makes LB score worse.
", "rawMarkdown": "Thank you for your comment. As far as I know, the following point is important to use this architecture.\n\n## Using Eternafold to get structure\nIt improves CV score at approximately 0.01. I tried other libraries such as U-Fold, R-fold, Vienna RNA etc but none of them worked well.\n\n## Keeping sequence length 64-128\nToo long/short length of sequence makes LB score worse.", "votes": null }, { "id": "2559485", "postDate": "12/12/2023 21:14:56", "content": "It's really refreshing to see a well-placing solution that pays attention to hardware requirements. Great job!
", "rawMarkdown": "It's really refreshing to see a well-placing solution that pays attention to hardware requirements. Great job!", "votes": null } ], "comments": [ { "id": 2553970, "author_name": "sacuscreed", "author_url": "", "post_date": "12/08/2023 17:00:40", "content": "Very similar with the approach I discarted but that obtained best private scoring. With a small difference. I'm giving it a second chance while trying to remember the particular details of that submission. I will prepare the results and so we can compare them.
", "votes": null, "replies": [ { "id": 2554219, "author_name": "qhapaq49", "author_url": "", "post_date": "12/08/2023 23:12:28", "content": "Thank you for your comment. As far as I know, the following point is important to use this architecture.
\nIt improves CV score at approximately 0.01. I tried other libraries such as U-Fold, R-fold, Vienna RNA etc but none of them worked well.
\nToo long/short length of sequence makes LB score worse.
", "votes": null, "replies": [] } ] }, { "id": 2559485, "author_name": "alberteinsten", "author_url": "", "post_date": "12/12/2023 21:14:56", "content": "It's really refreshing to see a well-placing solution that pays attention to hardware requirements. Great job!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2553860": "First of all, I would like to thank all the participants in the competition. My solution is a cheap transformer that does not require bpp-matrix and can learn and predict with poor computational resources named \"Cheap Transformers\"\n\n# Generalization to long sequences\nMy starting point is [Iafoss's amazing work](https://www.kaggle.com/code/iafoss/rna-starter-0-186-lb) where the reactivities are predicted by simple transformers.\n\nIt works very well in cross validation in random fold splitting but gives poor performance when train and validation data is split by length of sequence like following. \n\n```python\nshort_indices = df[df['sequence'].str.len() < 177].index.tolist()\nlong_indices = df[df['sequence'].str.len() > 177].index.tolist()\n```\n\nTherefore, I need another architecture to generalize to long sequences.\n\n# Cheap Transformer\n\nThis is an architecture of cheap transformer\n\ner. It just cut a certain length sequence from the original sequence.\n\nWhen regression, it cuts the sequence in multiple ways and ensemble the prediction results for each character. The ensemble is important. It improves cv-score at about 0.005.\n\n\n\nThis architecture uses short sequences, reduces training time, and requires less GPU memory during execution. Therefore, it runs fast enough even with the GPU of a gaming laptop.\n\n# Final submission\nI prepared multiple models with cut sequence lengths of 64, 96, and 128 and with various transformer's parameters and created an ensemble. Pseudo labeling was also performed, but it had little effect on LB. As you can see the ensemble contributed LB but its effect is not so big, therefore, using single model would be the most practival.\n\n\n\n# Scores\n| Method | Public | Private |\n| --- | --- | --- |\n| Iafoss's architecture | 0.15669 | 0.20204 |\n| Cheap transformers (len=64) | 0.15635 | 0.15661 |\n| Cheap transformers (len=96) | 0.15458 | 0.15445 |\n| Cheap transformers (len=128) | 0.15518 | 0.15625 |\n|Public best (ensemble with Iafoss's arch) | 0.14716 | 0.18222 |\n| ensemble | 0.15271 | 0.15346 |", "2553970": "Very similar with the approach I discarted but that obtained best private scoring. With a small difference. I'm giving it a second chance while trying to remember the particular details of that submission. I will prepare the results and so we can compare them.", "2554219": "Thank you for your comment. As far as I know, the following point is important to use this architecture.\n\n## Using Eternafold to get structure\nIt improves CV score at approximately 0.01. I tried other libraries such as U-Fold, R-fold, Vienna RNA etc but none of them worked well.\n\n## Keeping sequence length 64-128\nToo long/short length of sequence makes LB score worse.", "2559485": "It's really refreshing to see a well-placing solution that pays attention to hardware requirements. Great job!" }, "source": "meta" }