![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2311404%2Fb39fa96937be0505cbe1b8480910784e%2Fsplit.png?generation=1702029388253122&alt=media\")
First of all, I would like to thank the competition hosts for their exceptional organization and the challenging yet enriching environment they created. Their dedication to fostering a space for learning and innovation is deeply appreciated. I also express my profound gratitude to my teammates, @onodera and @christofhenkel. Their collaboration, expertise, and unwavering commitment throughout this project were invaluable.
\nHere I will try to summarize some of the main points of our solution.
\nIn the field of RNA secondary structure estimation, various studies including Sato et al., 2023, have highlighted the inappropriateness of using random splits for evaluation.This stems from the concept of 'Families' in RNA experimental data - groups of RNA molecules sharing specific functions or structures, resulting in high structural similarity within the same family. For instance, Fu et al., 2022 noted that the E2EFold model was overestimated due to this issue.
\nTo address this, we adopted clustering based on RNA sequence edit distances, using cluster IDs for GroupKFold. Additionally, we aggregated all the limited samples with seqlen=206 into fold=0. This strategy allowed for continuous evaluation of the model's performance on longer sequences.
\n
As part of my models, I experimented with alternating training using both the RMDB dataset and the competition dataset. This approach was adopted with the anticipation that it would enhance the model's ability to handle longer RNA sequences, a critical aspect in RNA secondary structure prediction.
\nWe processed input sequences using the 'arnie' package, mainly utilizing eternafold, rnasoft, and rnastructure. The model inputs included embedding layers for structure and loop_type, and a Graph Neural Network (GNN) adjacency matrix for bpp (base pair probability).
\nHowever, bpp alone, representing the probability of forming a pair, requires many layers when combined with CNNs to view the entire graph. As an alternative to using Transformers for a global view, we utilized “structure” (renamed to 'chunk' due to naming conflicts) and “segment”, as defined in bpRNA [Danaee et al., 2018].
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2311404%2Fd2a22732d565f34536b3ad5c9af99220%2Fgraph_features.png?generation=1702027001721359&alt=media\")
My architecture is inspired by LegNet from @dmitrypenzar1996 and OpenVaccine's 6th place solution [nyanp]. It consists solely of CNN and GNN layers. The adjacency matrices for bpp, structure, chunk, and segment significantly differ, so I prepared independent CNN+GNN blocks for each type. Group Convolution and einsum enabled this without for loops.
\nFollowing LegNet, prediction head outputs a 100d vector instead of a 1d scalar, with weighted summation over bin values for the final output. This approach stabilized learning by controlling the output range in regression tasks.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2311404%2Feed37f263a4a1321dbd91ccaf915d66c%2Fmodel.png?generation=1702027231710653&alt=media\")
We opted for an MAE + MSE weighted by signal-to-noise ratio as my optimization function. MSE provides gradients similar to MAE in the early stages of training but approaches zero later, aiding in convergence.
\nWe conducted pseudo label training using predictions on the test set created collaboratively with teammates @onodera and @christofhenkel.
\n| Name | \nCV | \nCV (seqlen=206) | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|
| single best (scratch training) | \n0.1336 | \n0.1133 | \n0.14012 | \n0.14299 | \n
| single best (+pseudo label 1st) | \n0.1313 | \n0.1152 | \n0.13828 | \n0.14222 | \n
| single best (+pseudo label 2nd) | \n0.1306 | \n0.1221 | \n0.13739 | \n0.14186 | \n
| blending w/ all models | \n0.127645 | \n0.109341 | \n0.13626 | \n0.14263 | \n
First of all, I would like to thank the competition hosts for their exceptional organization and the challenging yet enriching environment they created. Their dedication to fostering a space for learning and innovation is deeply appreciated. I also express my profound gratitude to my teammates, @onodera and @christofhenkel. Their collaboration, expertise, and unwavering commitment throughout this project were invaluable.
\nHere I will try to summarize some of the main points of our solution.
\nIn the field of RNA secondary structure estimation, various studies including Sato et al., 2023, have highlighted the inappropriateness of using random splits for evaluation.This stems from the concept of 'Families' in RNA experimental data - groups of RNA molecules sharing specific functions or structures, resulting in high structural similarity within the same family. For instance, Fu et al., 2022 noted that the E2EFold model was overestimated due to this issue.
\nTo address this, we adopted clustering based on RNA sequence edit distances, using cluster IDs for GroupKFold. Additionally, we aggregated all the limited samples with seqlen=206 into fold=0. This strategy allowed for continuous evaluation of the model's performance on longer sequences.
\n
As part of my models, I experimented with alternating training using both the RMDB dataset and the competition dataset. This approach was adopted with the anticipation that it would enhance the model's ability to handle longer RNA sequences, a critical aspect in RNA secondary structure prediction.
\nWe processed input sequences using the 'arnie' package, mainly utilizing eternafold, rnasoft, and rnastructure. The model inputs included embedding layers for structure and loop_type, and a Graph Neural Network (GNN) adjacency matrix for bpp (base pair probability).
\nHowever, bpp alone, representing the probability of forming a pair, requires many layers when combined with CNNs to view the entire graph. As an alternative to using Transformers for a global view, we utilized “structure” (renamed to 'chunk' due to naming conflicts) and “segment”, as defined in bpRNA [Danaee et al., 2018].
\n
My architecture is inspired by LegNet from @dmitrypenzar1996 and OpenVaccine's 6th place solution [nyanp]. It consists solely of CNN and GNN layers. The adjacency matrices for bpp, structure, chunk, and segment significantly differ, so I prepared independent CNN+GNN blocks for each type. Group Convolution and einsum enabled this without for loops.
\nFollowing LegNet, prediction head outputs a 100d vector instead of a 1d scalar, with weighted summation over bin values for the final output. This approach stabilized learning by controlling the output range in regression tasks.
\n
We opted for an MAE + MSE weighted by signal-to-noise ratio as my optimization function. MSE provides gradients similar to MAE in the early stages of training but approaches zero later, aiding in convergence.
\nWe conducted pseudo label training using predictions on the test set created collaboratively with teammates @onodera and @christofhenkel.
\n| Name | \nCV | \nCV (seqlen=206) | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|
| single best (scratch training) | \n0.1336 | \n0.1133 | \n0.14012 | \n0.14299 | \n
| single best (+pseudo label 1st) | \n0.1313 | \n0.1152 | \n0.13828 | \n0.14222 | \n
| single best (+pseudo label 2nd) | \n0.1306 | \n0.1221 | \n0.13739 | \n0.14186 | \n
| blending w/ all models | \n0.127645 | \n0.109341 | \n0.13626 | \n0.14263 | \n
Oh, you made LegNet work for this task! Can't wait to see the code if you are going to publish it!
", "rawMarkdown": "Oh, you made LegNet work for this task! Can't wait to see the code if you are going to publish it!", "votes": null }, { "id": "2556082", "postDate": "12/10/2023 13:02:16", "content": "Thanks for your comment.
\nMy code is dirty so I don't have a plan to publish yet, but the main differences between my approach and LegNet are the presence of GNN layers and the objective function.
\nGNN layers based on secondary structures mentioned above enable the model to attend to long-range contacts.
Oh, you made LegNet work for this task! Can't wait to see the code if you are going to publish it!
", "votes": null, "replies": [ { "id": 2556082, "author_name": "kfujikawa", "author_url": "", "post_date": "12/10/2023 13:02:16", "content": "Thanks for your comment.
\nMy code is dirty so I don't have a plan to publish yet, but the main differences between my approach and LegNet are the presence of GNN layers and the objective function.
\nGNN layers based on secondary structures mentioned above enable the model to attend to long-range contacts.