![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16725112%2F482c2d20fce57fa513021909a515e98b%2Fpreprocessing.png?generation=1701994647204619&alt=media\")
The Ribonanza RNA Folding competition has been an amazing opportunity and we are really glad to have participated in it. Kudos to the hosts for their substantial effort in keeping the high organizational level and giving the contestants plenty of valuable insight and support. We would also like to thank the community for the fruitful public discussions and coding initiatives. Congratulations to everyone involved! đ
\nOpen source code is available on github.
\nOur solution is based on transformer architecture. We predicted base pair probability matrix(BPPM) for each RNA sequence using EternaFold and added convolutional blocks to our architecture to process BPPM features. For some models we added Squeeze-and-Excitation layer in convolutional blocks Also we add these features with attention values before softmax operation in Self-Attention block. To allow better generalization for longer input we implemented Dynamic Positional Bias. Then we ensembled models with slight differences in architecture and training process.
\nAt first, for each sequence in train and test datasets we calculated a base pair probability matrix (BPPM) using EternaFold. During training and inference phases we passed to our model the RNA sequences encoded with a learnable embedding layer. Each nucleotide was considered a token and special < start > and < end > tokens were placed at both ends as well. To provide the model with the information about whether the sequence comes from a âcleanâ subset of training dataset (which SN_filter values show - 1 corresponds to âcleanâ sequences with high signal-to-noise ratio and 0 is otherwise) we encoded SN_filter values with a learnable embedding layer and added corresponding embeddings to sequence embeddings. The embedding dimension was chosen to be of size 192. BPPMs were padded with zeros at their margins to account for adding < start > and < end > tokens. The figure below summarizes the data preprocessing part.
\n
The model idea is partially inspired by @shujun717 solution to the Open Vaccine challenge, link
\nThe model takes a sequence of tokens and BPPM as input and outputs DMS_MaP and 2A3_MaP reactivities for each input nucleotide. Its architecture comprises 12 consecutive Transformer Encoder Layers and an output projection linear layer. Each Transformer block takes a sequence of tokens and BPPM features from the previous layer and outputs updated feature maps as shown below. The Transformer Encoder block adopts common transformer encoder architecture, except we modified the Self-Attention block to ensure interaction between BPPM features and sequence features.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16725112%2F0c70879344b972d64ba13945e163a126%2Fmodel_general_view.png?generation=1701994737102906&alt=media\")
In the Self-Attention (SA) block we implemented an attention mechanism with the following modifications: after attention values for each head are calculated, we add BPPM features updated by the âConvolutional blockâ, which outputs BPPM features with the number of channels corresponding to the number of heads in the SA block. We set the number of heads in SA and corresponding channels of BPPM features to be 6. Thus, the hidden dimension size of Q, K, V matrices is 32. The overall structure of the SA block is shown below.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16725112%2F5a7bf6e6f893938ced8bdd2e9a1fb08c%2Fselfattention.png?generation=1701994715778085&alt=media\")
The sequence length is distributed differently in train and test datasets (test sequences are generally longer). To allow for the better generalization of longer inputs we implemented a positional encoding to be added to attention values. We found Dynamic Positional Bias to be of more use compared to other relative positional encoding methods we tried, such as xPos (rotary positional encoding) and ALiBi. Dynamic Positional Bias calculates for each head a relative positional bias map, which is learnable and depends on sequence length. Relative positional bias doesnât allow to leverage distance from start and end of sequence, so tokens < start > and < end > were added to fix that.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16725112%2F8d8468983d7a7d057dcf74d7511b3ac8%2Fdynpos%20(1).png?generation=1701994757603986&alt=media\")
The models in the final ensemble come in two versions that are slightly different in the structure of the Convolutional block. The basic Convolutional block consists of 2D convolutional layer, batchnorm layer, activation and learnable gamma parameters that scale the output feature channels, whereas the modified version of this block (SE-Convolutional block) also contains Squeeze-and-Excitation layer, hence the name. The SE layer applies input-dependent rescaling of values along the channels as shown below. Thus, the only difference between models in the ensemble is the presence of the SE layer.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16725112%2F4b8f15f5aa950ba64ecbd55df123cad4%2Fconv_se.png?generation=1701994778649467&alt=media\")
The training process has been adapted from the notebook by @IAFOSS
\nWe used a one-cycle learning rate schedule (pct_start=0.05, lr_max=2.5e-3,) coupled with AdamW optimizer (wd=0.05), and batch size set to 128.
\nThe number of epochs for model training was determined from the dataset size.
\nFor final models (trained almost on the whole dataset) we used 270 epochs.
\nIn each epoch, 1791 batches were processed (we kept this number due to historical reasons), elements of each batch were sampled from dataset with the weight = 0.5 * torch.clamp_min(torch.log(sn + 1.01),0.01)
We have also found that training model with a simple SGD optimizer for ~15 epochs of 500 batches each additionally improves model performance (the number of epochs varies so we used a small validation set to determine the exact number of epochs)
\nAdditionally, one model was trained to predict 2A3 given DMS, RNA sequence and BPPM (dms-to-2a3 model).
\nAt the inference phase for all inputs we set the SN filter value to be 1 as if they come from a âcleanâ dataset.
\nWe ensembled 15 models with SE-Convolutional block, 10 models with plain Convolutional block, 2 with plain Convolutional block trained on split by sequence lengths (one of these two models accepts bracket features). We took the average of their predictions. Then we predicted 2a3 reactivities based on averaged dms reactivities using dms-to-2a3 model and added these predictions to averaged 2a3 reactivities in the following way: (27/28)averaged_2a3 + (1/28)predicted_2a3.
\nThe clear problem with a simple KFold split is the high sequence similarity across the training dataset and the fact that test sequences are very distinct from the training data. This might hamper the model development, because the increase in quality on the validation set could be due to overfitting rather than actual improvement.
\nWe have calculated a hamming distance matrix for all the sequences present in the training set and performed a modified DBSCAN clustering procedure with distance threshold set at 0.2. In the following picture we show the clusters mapped to their respective cluster identifier (a cluster ID was assigned as a number of the smallest sequence within that cluster in the train dataset).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16725112%2F55dc21823386d878fb212d681cae593e%2Fcluster.jpg?generation=1701994798203967&alt=media\")
We tested several splits based on sequence identity (the easiest one is just to split data into folds without shuffling) and found out that while the model final validation performance degrades as we choose more and more stricter distance threshold, its relative value behave the same way as for a simple KFold split. So, we decided to use simple KFold split than training final models
\nAlso, we have conducted a test of the model performance on length-based split. For that, we trained a model on the short sequences (length < 206) and validated it on sequences of size = 206. The behavior of the validation metric was slightly noisier but still highly correlated with a metric for a simple KFold split.
\nApproximately 13% of public test sequences are identical to the ones present in the train dataset (by sequence). To avoid selecting a model that is memorizing more of these sequences rather than learning RNA-related stuff, we zeroed out the predictions for these sequences in most of our submissions, while sometimes sending non-zeroed out submissions in order to compare our performance to other participants.
\nWe donât have conclusive results for that feature. It seems like it is not beneficial for the model on average but sometimes it resulted in a better model and sometimes â in worse. We decided to not use this feature.
\nWe have tried to use brackets generated by EternaFold, ContraFold, ViennaRNA, etc., as well as programs for pseudoknots prediction (like IPknot). However, after adding EternaFold BPPMs to the model, adding other features yields no significant increase in model performance. For some models we used brackets just to augment model
\nAll other BPPMs (ContraFold, ViennaRNA, RNAsoft, RNAstructure) result in a suboptimal model.
\nAveraging BPPMs doesnât result in better performance.
\nBPPMs from RFold yield the same quality as ViennaRNA BPPMs.
\nThe RNA-FM model produces both per-nucleotide embeddings and BPPM-like matrix. Still, those donât help the model at all, and using them results only in a slight increase in model performance then compared to sequence only model
SQUARNA (github) outputs a matrix different from BPPM, but can be used in the same manner. Unfortunately, this feature also didnât yield any additional performance increase.
\nThe initial reason we decided to take part in the competition was to test our model LegNet link, which shows SOTA results on DNA sequences and worked well on some RNA-related tasks (not yet published).
\nUnfortunately, any modifications of this architecture resulted in subpar performance when compared with properly tuned transformer models. This can be explained by the fact that predicting RNA secondary structure requires attending to long-range contacts. The transformer architecture suits better for such cases.
Subsetting data (filtering by different thresholds on SN ratio) resulted in a performance boost for all models. However, this technique was superseded by weight sampling, which proved itself to be more effective.
\nWe tried to fine-tune our model on a public dataset, gathered by the organizers (link) by training model to predict the results of the public experiments (excluding ones with a small number of samples) and for Ribonanza data simultaneously. Unfortunately, this also gave no boost to the model performance.
\nWe tried to use data about predicted 3D structure of 100k sequences from the train dataset but gave up on that once we had visually analyzed them:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16725112%2F2d054d1679176e0664130f6c46ccb0ae%2Ftrashfold.jpg?generation=1701994817511350&alt=media\")
Using absolute positional embedding leads to unsolvable issues when generalizing upon longer sequences.
\nThe simplest approach is to augment absolute positional encoding during the training phase to shift randomly from 0 to (Lmax - seqlen) position. This indeed solves the issue with extrapolation, but works worse than other methods.
\nRotational positional embedding, unfortunately, doesnât help the model to generalize on larger lengths.
\nALiBi positional embedding solves the issue with extrapolation but even after keeping it only for a part of heads (as suggested in https://github.com/lucidrains/x-transformers) still behaves worse than dynamic positional bias.
First, we tried to use reverse augmentation. This can be done in three ways:
\nreversing sequence before any modifications,
\nreversing sequence before padding, but after adding and tokens,
\nreversing sequence after padding.
The first two ways yield no gain for all variants of models we tested. Yet, the third one (upon coupling with additional finetuning) gave us a good result for a model with xpos positional encoding. The resulting single-model performance was 0.13937 on the public leaderboard. Unfortunately, xpos shows rather poor performance on long sequences so we abstained from using this model in the final submission.
\nWe have also tried shift augmentation and different sequence padding approaches. This didnât improve our model performance as well.
\nOne of the possible ways to generalize for larger sequences is to predict reactivities using the sliding window. However, the idea is somewhat wrong in a biological sense, and it results in performance degradation when testing on train dataset sequences.
\nOnce we obtained ensembles of best-performing models we tried to use them to pseudolabel test dataset and use predictions with the highest confidence to train new models. While this indeed results in a better single-performing model, adding such a model to ensemble doesnât improve ensemble performance.
\nInstead of filtering sequences with low SN we tried to mask positions with high reactivity error as it was done by @nullrecurrent in Open Vaccine challenge (link)
\nThis resulted in poor performance.
\nWe tried to weight loss for each sequence by its SN â it didnât result in any improvement
Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141).
\nWang, W., Chen, W., Qiu, Q., Chen, L., Wu, B., Lin, B., ⊠& Liu, W. (2023). Crossformer++: A versatile vision transformer hinging on cross-scale attention. arXiv preprint arXiv:2303.06908.
", "messages": [ { "id": 2553003, "postDate": "2023-12-08T00:33:45.897Z", "content": "The Ribonanza RNA Folding competition has been an amazing opportunity and we are really glad to have participated in it. Kudos to the hosts for their substantial effort in keeping the high organizational level and giving the contestants plenty of valuable insight and support. We would also like to thank the community for the fruitful public discussions and coding initiatives. Congratulations to everyone involved! đ
\nOpen source code is available on github.
\nOur solution is based on transformer architecture. We predicted base pair probability matrix(BPPM) for each RNA sequence using EternaFold and added convolutional blocks to our architecture to process BPPM features. For some models we added Squeeze-and-Excitation layer in convolutional blocks Also we add these features with attention values before softmax operation in Self-Attention block. To allow better generalization for longer input we implemented Dynamic Positional Bias. Then we ensembled models with slight differences in architecture and training process.
\nAt first, for each sequence in train and test datasets we calculated a base pair probability matrix (BPPM) using EternaFold. During training and inference phases we passed to our model the RNA sequences encoded with a learnable embedding layer. Each nucleotide was considered a token and special < start > and < end > tokens were placed at both ends as well. To provide the model with the information about whether the sequence comes from a âcleanâ subset of training dataset (which SN_filter values show - 1 corresponds to âcleanâ sequences with high signal-to-noise ratio and 0 is otherwise) we encoded SN_filter values with a learnable embedding layer and added corresponding embeddings to sequence embeddings. The embedding dimension was chosen to be of size 192. BPPMs were padded with zeros at their margins to account for adding < start > and < end > tokens. The figure below summarizes the data preprocessing part.
\n
The model idea is partially inspired by @shujun717 solution to the Open Vaccine challenge, link
\nThe model takes a sequence of tokens and BPPM as input and outputs DMS_MaP and 2A3_MaP reactivities for each input nucleotide. Its architecture comprises 12 consecutive Transformer Encoder Layers and an output projection linear layer. Each Transformer block takes a sequence of tokens and BPPM features from the previous layer and outputs updated feature maps as shown below. The Transformer Encoder block adopts common transformer encoder architecture, except we modified the Self-Attention block to ensure interaction between BPPM features and sequence features.
\n
In the Self-Attention (SA) block we implemented an attention mechanism with the following modifications: after attention values for each head are calculated, we add BPPM features updated by the âConvolutional blockâ, which outputs BPPM features with the number of channels corresponding to the number of heads in the SA block. We set the number of heads in SA and corresponding channels of BPPM features to be 6. Thus, the hidden dimension size of Q, K, V matrices is 32. The overall structure of the SA block is shown below.
\n
The sequence length is distributed differently in train and test datasets (test sequences are generally longer). To allow for the better generalization of longer inputs we implemented a positional encoding to be added to attention values. We found Dynamic Positional Bias to be of more use compared to other relative positional encoding methods we tried, such as xPos (rotary positional encoding) and ALiBi. Dynamic Positional Bias calculates for each head a relative positional bias map, which is learnable and depends on sequence length. Relative positional bias doesnât allow to leverage distance from start and end of sequence, so tokens < start > and < end > were added to fix that.
\n
The models in the final ensemble come in two versions that are slightly different in the structure of the Convolutional block. The basic Convolutional block consists of 2D convolutional layer, batchnorm layer, activation and learnable gamma parameters that scale the output feature channels, whereas the modified version of this block (SE-Convolutional block) also contains Squeeze-and-Excitation layer, hence the name. The SE layer applies input-dependent rescaling of values along the channels as shown below. Thus, the only difference between models in the ensemble is the presence of the SE layer.
\n
The training process has been adapted from the notebook by @IAFOSS
\nWe used a one-cycle learning rate schedule (pct_start=0.05, lr_max=2.5e-3,) coupled with AdamW optimizer (wd=0.05), and batch size set to 128.
\nThe number of epochs for model training was determined from the dataset size.
\nFor final models (trained almost on the whole dataset) we used 270 epochs.
\nIn each epoch, 1791 batches were processed (we kept this number due to historical reasons), elements of each batch were sampled from dataset with the weight = 0.5 * torch.clamp_min(torch.log(sn + 1.01),0.01)
We have also found that training model with a simple SGD optimizer for ~15 epochs of 500 batches each additionally improves model performance (the number of epochs varies so we used a small validation set to determine the exact number of epochs)
\nAdditionally, one model was trained to predict 2A3 given DMS, RNA sequence and BPPM (dms-to-2a3 model).
\nAt the inference phase for all inputs we set the SN filter value to be 1 as if they come from a âcleanâ dataset.
\nWe ensembled 15 models with SE-Convolutional block, 10 models with plain Convolutional block, 2 with plain Convolutional block trained on split by sequence lengths (one of these two models accepts bracket features). We took the average of their predictions. Then we predicted 2a3 reactivities based on averaged dms reactivities using dms-to-2a3 model and added these predictions to averaged 2a3 reactivities in the following way: (27/28)averaged_2a3 + (1/28)predicted_2a3.
\nThe clear problem with a simple KFold split is the high sequence similarity across the training dataset and the fact that test sequences are very distinct from the training data. This might hamper the model development, because the increase in quality on the validation set could be due to overfitting rather than actual improvement.
\nWe have calculated a hamming distance matrix for all the sequences present in the training set and performed a modified DBSCAN clustering procedure with distance threshold set at 0.2. In the following picture we show the clusters mapped to their respective cluster identifier (a cluster ID was assigned as a number of the smallest sequence within that cluster in the train dataset).
\n
We tested several splits based on sequence identity (the easiest one is just to split data into folds without shuffling) and found out that while the model final validation performance degrades as we choose more and more stricter distance threshold, its relative value behave the same way as for a simple KFold split. So, we decided to use simple KFold split than training final models
\nAlso, we have conducted a test of the model performance on length-based split. For that, we trained a model on the short sequences (length < 206) and validated it on sequences of size = 206. The behavior of the validation metric was slightly noisier but still highly correlated with a metric for a simple KFold split.
\nApproximately 13% of public test sequences are identical to the ones present in the train dataset (by sequence). To avoid selecting a model that is memorizing more of these sequences rather than learning RNA-related stuff, we zeroed out the predictions for these sequences in most of our submissions, while sometimes sending non-zeroed out submissions in order to compare our performance to other participants.
\nWe donât have conclusive results for that feature. It seems like it is not beneficial for the model on average but sometimes it resulted in a better model and sometimes â in worse. We decided to not use this feature.
\nWe have tried to use brackets generated by EternaFold, ContraFold, ViennaRNA, etc., as well as programs for pseudoknots prediction (like IPknot). However, after adding EternaFold BPPMs to the model, adding other features yields no significant increase in model performance. For some models we used brackets just to augment model
\nAll other BPPMs (ContraFold, ViennaRNA, RNAsoft, RNAstructure) result in a suboptimal model.
\nAveraging BPPMs doesnât result in better performance.
\nBPPMs from RFold yield the same quality as ViennaRNA BPPMs.
\nThe RNA-FM model produces both per-nucleotide embeddings and BPPM-like matrix. Still, those donât help the model at all, and using them results only in a slight increase in model performance then compared to sequence only model
SQUARNA (github) outputs a matrix different from BPPM, but can be used in the same manner. Unfortunately, this feature also didnât yield any additional performance increase.
\nThe initial reason we decided to take part in the competition was to test our model LegNet link, which shows SOTA results on DNA sequences and worked well on some RNA-related tasks (not yet published).
\nUnfortunately, any modifications of this architecture resulted in subpar performance when compared with properly tuned transformer models. This can be explained by the fact that predicting RNA secondary structure requires attending to long-range contacts. The transformer architecture suits better for such cases.
Subsetting data (filtering by different thresholds on SN ratio) resulted in a performance boost for all models. However, this technique was superseded by weight sampling, which proved itself to be more effective.
\nWe tried to fine-tune our model on a public dataset, gathered by the organizers (link) by training model to predict the results of the public experiments (excluding ones with a small number of samples) and for Ribonanza data simultaneously. Unfortunately, this also gave no boost to the model performance.
\nWe tried to use data about predicted 3D structure of 100k sequences from the train dataset but gave up on that once we had visually analyzed them:
\n
Using absolute positional embedding leads to unsolvable issues when generalizing upon longer sequences.
\nThe simplest approach is to augment absolute positional encoding during the training phase to shift randomly from 0 to (Lmax - seqlen) position. This indeed solves the issue with extrapolation, but works worse than other methods.
\nRotational positional embedding, unfortunately, doesnât help the model to generalize on larger lengths.
\nALiBi positional embedding solves the issue with extrapolation but even after keeping it only for a part of heads (as suggested in https://github.com/lucidrains/x-transformers) still behaves worse than dynamic positional bias.
First, we tried to use reverse augmentation. This can be done in three ways:
\nreversing sequence before any modifications,
\nreversing sequence before padding, but after adding and tokens,
\nreversing sequence after padding.
The first two ways yield no gain for all variants of models we tested. Yet, the third one (upon coupling with additional finetuning) gave us a good result for a model with xpos positional encoding. The resulting single-model performance was 0.13937 on the public leaderboard. Unfortunately, xpos shows rather poor performance on long sequences so we abstained from using this model in the final submission.
\nWe have also tried shift augmentation and different sequence padding approaches. This didnât improve our model performance as well.
\nOne of the possible ways to generalize for larger sequences is to predict reactivities using the sliding window. However, the idea is somewhat wrong in a biological sense, and it results in performance degradation when testing on train dataset sequences.
\nOnce we obtained ensembles of best-performing models we tried to use them to pseudolabel test dataset and use predictions with the highest confidence to train new models. While this indeed results in a better single-performing model, adding such a model to ensemble doesnât improve ensemble performance.
\nInstead of filtering sequences with low SN we tried to mask positions with high reactivity error as it was done by @nullrecurrent in Open Vaccine challenge (link)
\nThis resulted in poor performance.
\nWe tried to weight loss for each sequence by its SN â it didnât result in any improvement
Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141).
\nWang, W., Chen, W., Qiu, Q., Chen, L., Wu, B., Lin, B., ⊠& Liu, W. (2023). Crossformer++: A versatile vision transformer hinging on cross-scale attention. arXiv preprint arXiv:2303.06908.
", "rawMarkdown": "The Ribonanza RNA Folding competition has been an amazing opportunity and we are really glad to have participated in it. Kudos to the hosts for their substantial effort in keeping the high organizational level and giving the contestants plenty of valuable insight and support. We would also like to thank the community for the fruitful public discussions and coding initiatives. Congratulations to everyone involved! đ\n\n# Code\nOpen source code is available on [github](https://github.com/autosome-ru/vigg_ribonanza/).\n\n# TLDR\n\nOur solution is based on transformer architecture. We predicted base pair probability matrix(BPPM) for each RNA sequence using EternaFold and added convolutional blocks to our architecture to process BPPM features. For some models we added Squeeze-and-Excitation layer in convolutional blocks Also we add these features with attention values before softmax operation in Self-Attention block. To allow better generalization for longer input we implemented Dynamic Positional Bias. Then we ensembled models with slight differences in architecture and training process.\n\n# Data Preprocessing\n\nAt first, for each sequence in train and test datasets we calculated a base pair probability matrix (BPPM) using EternaFold. During training and inference phases we passed to our model the RNA sequences encoded with a learnable embedding layer. Each nucleotide was considered a token and special **< start >** and **< end >** tokens were placed at both ends as well. To provide the model with the information about whether the sequence comes from a âcleanâ subset of training dataset (which SN_filter values show - 1 corresponds to âcleanâ sequences with high signal-to-noise ratio and 0 is otherwise) we encoded SN_filter values with a learnable embedding layer and added corresponding embeddings to sequence embeddings. The embedding dimension was chosen to be of size **192**. BPPMs were padded with zeros at their margins to account for adding **< start >** and **< end >** tokens. The figure below summarizes the data preprocessing part.\n\n\n\n# Model\n\nThe model idea is partially inspired by @shujun717 solution to the Open Vaccine challenge, [link](https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189564)\n\nThe model takes a sequence of tokens and BPPM as input and outputs DMS_MaP and 2A3_MaP reactivities for each input nucleotide. Its architecture comprises 12 consecutive Transformer Encoder Layers and an output projection linear layer. Each Transformer block takes a sequence of tokens and BPPM features from the previous layer and outputs updated feature maps as shown below. The Transformer Encoder block adopts common transformer encoder architecture, except we modified the Self-Attention block to ensure interaction between BPPM features and sequence features.\n\n\n\n## Self-Attention Block\n\nIn the Self-Attention (SA) block we implemented an attention mechanism with the following modifications: after attention values for each head are calculated, we add BPPM features updated by the âConvolutional blockâ, which outputs BPPM features with the number of channels corresponding to the number of heads in the SA block. We set the number of heads in SA and corresponding channels of BPPM features to be 6. Thus, the hidden dimension size of Q, K, V matrices is 32. The overall structure of the SA block is shown below.\n\n\n\n## Dynamic Positional Bias\n\nThe sequence length is distributed differently in train and test datasets (test sequences are generally longer). To allow for the better generalization of longer inputs we implemented a positional encoding to be added to attention values. We found Dynamic Positional Bias to be of more use compared to other relative positional encoding methods we tried, such as xPos (rotary positional encoding) and ALiBi. Dynamic Positional Bias calculates for each head a relative positional bias map, which is learnable and depends on sequence length. Relative positional bias doesnât allow to leverage distance from start and end of sequence, so tokens **< start >** and **< end >** were added to fix that.\n\n.png?generation=1701994757603986&alt=media)\n\n## Convolutional Block and SE block\n\nThe models in the final ensemble come in two versions that are slightly different in the structure of the Convolutional block. The basic Convolutional block consists of 2D convolutional layer, batchnorm layer, activation and learnable gamma parameters that scale the output feature channels, whereas the modified version of this block (SE-Convolutional block) also contains Squeeze-and-Excitation layer, hence the name. The SE layer applies input-dependent rescaling of values along the channels as shown below. Thus, the only difference between models in the ensemble is the presence of the SE layer.\n\n\n\n# Training Process\n\nThe training process has been adapted from the [notebook](https://www.kaggle.com/code/iafoss/rna-starter-0-186-lb/notebook) by @IAFOSS\nWe used a one-cycle learning rate schedule (pct_start=0.05, lr_max=2.5e-3,) coupled with AdamW optimizer (wd=0.05), and batch size set to 128.\nThe number of epochs for model training was determined from the dataset size.\nFor final models (trained almost on the whole dataset) we used 270 epochs. \nIn each epoch, 1791 batches were processed (we kept this number due to historical reasons), elements of each batch were sampled from dataset with the weight = 0.5 * torch.clamp_min(torch.log(sn + 1.01),0.01)\n\nWe have also found that training model with a simple SGD optimizer for ~15 epochs of 500 batches each additionally improves model performance (the number of epochs varies so we used a small validation set to determine the exact number of epochs)\n\nAdditionally, one model was trained to predict 2A3 given DMS, RNA sequence and BPPM (dms-to-2a3 model).\n\n# Inference\nAt the inference phase for all inputs we set the SN filter value to be 1 as if they come from a âcleanâ dataset.\n\n# Ensembling\n\nWe ensembled 15 models with SE-Convolutional block, 10 models with plain Convolutional block, 2 with plain Convolutional block trained on split by sequence lengths (one of these two models accepts bracket features). We took the average of their predictions. Then we predicted 2a3 reactivities based on averaged dms reactivities using dms-to-2a3 model and added these predictions to averaged 2a3 reactivities in the following way: (27/28)*averaged_2a3 + (1/28)*predicted_2a3.\n\n# Other splits \n\nThe clear problem with a simple KFold split is the high sequence similarity across the training dataset and the fact that test sequences are very distinct from the training data. This might hamper the model development, because the increase in quality on the validation set could be due to overfitting rather than actual improvement.\n\nWe have calculated a hamming distance matrix for all the sequences present in the training set and performed a modified DBSCAN clustering procedure with distance threshold set at 0.2. In the following picture we show the clusters mapped to their respective cluster identifier (a cluster ID was assigned as a number of the smallest sequence within that cluster in the train dataset).\n\n\n\nWe tested several splits based on sequence identity (the easiest one is just to split data into folds without shuffling) and found out that while the model final validation performance degrades as we choose more and more stricter distance threshold, its relative value behave the same way as for a simple KFold split. So, we decided to use simple KFold split than training final models \n\nAlso, we have conducted a test of the model performance on length-based split. For that, we trained a model on the short sequences (length < 206) and validated it on sequences of size = 206. The behavior of the validation metric was slightly noisier but still highly correlated with a metric for a simple KFold split.\n\n# Public data leakage \n\nApproximately 13% of public test sequences are identical to the ones present in the train dataset (by sequence). To avoid selecting a model that is memorizing more of these sequences rather than learning RNA-related stuff, we zeroed out the predictions for these sequences in most of our submissions, while sometimes sending non-zeroed out submissions in order to compare our performance to other participants.\n\n# Features we also tried\n\n## capR\n\nWe donât have conclusive results for that feature. It seems like it is not beneficial for the model on average but sometimes it resulted in a better model and sometimes â in worse. We decided to not use this feature.\n\n## Brackets\n\nWe have tried to use brackets generated by EternaFold, ContraFold, ViennaRNA, etc., as well as programs for pseudoknots prediction (like IPknot). However, after adding EternaFold BPPMs to the model, adding other features yields no significant increase in model performance. For some models we used brackets just to augment model\n\n## Different BPPMs\n\nAll other BPPMs (ContraFold, ViennaRNA, RNAsoft, RNAstructure) result in a suboptimal model.\nAveraging BPPMs doesnât result in better performance.\nBPPMs from RFold yield the same quality as ViennaRNA BPPMs.\nThe RNA-FM model produces both per-nucleotide embeddings and BPPM-like matrix. Still, those donât help the model at all, and using them results only in a slight increase in model performance then compared to sequence only model\n \n## SQUARNA matrix\n\nSQUARNA ([github](https://github.com/febos/SQUARNA)) outputs a matrix different from BPPM, but can be used in the same manner. Unfortunately, this feature also didnât yield any additional performance increase.\n\n# What we also tried\n\n## Fully-Convolutional architecture\n\nThe initial reason we decided to take part in the competition was to test our model LegNet [link](https://academic.oup.com/bioinformatics/article/39/8/btad457/7230784), which shows SOTA results on DNA sequences and worked well on some RNA-related tasks (not yet published).\nUnfortunately, any modifications of this architecture resulted in subpar performance when compared with properly tuned transformer models. This can be explained by the fact that predicting RNA secondary structure requires attending to long-range contacts. The transformer architecture suits better for such cases.\n\n## Subsetting data \n\nSubsetting data (filtering by different thresholds on SN ratio) resulted in a performance boost for all models. However, this technique was superseded by weight sampling, which proved itself to be more effective.\n\n## Fine-tuning on public datasets \n\nWe tried to fine-tune our model on a public dataset, gathered by the organizers ([link](https://www.kaggle.com/datasets/rhijudas/rmdb-rna-mapping-database-2023-data/data)) by training model to predict the results of the public experiments (excluding ones with a small number of samples) and for Ribonanza data simultaneously. Unfortunately, this also gave no boost to the model performance.\n\n## Using 3D data \n\nWe tried to use data about predicted 3D structure of 100k sequences from the train dataset but gave up on that once we had visually analyzed them:\n\n \n\n## Absolute positional embedding\n\nUsing absolute positional embedding leads to unsolvable issues when generalizing upon longer sequences.\n\n## Relative positional embedding \n\nThe simplest approach is to augment absolute positional encoding during the training phase to shift randomly from 0 to (Lmax - seqlen) position. This indeed solves the issue with extrapolation, but works worse than other methods.\nRotational positional embedding, unfortunately, doesnât help the model to generalize on larger lengths.\nALiBi positional embedding solves the issue with extrapolation but even after keeping it only for a part of heads (as suggested in https://github.com/lucidrains/x-transformers) still behaves worse than dynamic positional bias.\n\n## Augmentation \n\nFirst, we tried to use reverse augmentation. This can be done in three ways:\nreversing sequence before any modifications,\nreversing sequence before padding, but after addingCongratulations! @vyaltsevvaleriy đđđ This is a very, very great method. May I ask what software was used to draw this picture? This is so beautiful
", "rawMarkdown": "Congratulations! @vyaltsevvaleriy đđđ This is a very, very great method. May I ask what software was used to draw this picture? This is so beautiful", "votes": 9, "replies": [ { "id": 2553451, "postDate": "2023-12-08T08:56:55.440Z", "content": "Haha, I also want to know
", "rawMarkdown": "Haha, I also want to know", "votes": 2 }, { "id": 2553696, "postDate": "2023-12-08T12:56:38.877Z", "content": "Hi! Thank you for your appreciation! I used Inkscape - it is a vector graphics editor allowing to flexibly implement any ideas.
", "rawMarkdown": "Hi! Thank you for your appreciation! I used Inkscape - it is a vector graphics editor allowing to flexibly implement any ideas.", "votes": 10, "replies": [ { "id": 2553829, "postDate": "2023-12-08T14:32:42.767Z", "content": "Thank you very much! vyaltsevvaleriyđ„°
", "rawMarkdown": "Thank you very much! vyaltsevvaleriyđ„°", "votes": 2 } ] } ] }, { "id": 2553004, "postDate": "2023-12-08T00:36:09.530Z", "content": "Wonderful work and presentation! Congratulations! Would you mind providing more improvement details (improvement points) with each operation?
", "rawMarkdown": "Wonderful work and presentation! Congratulations! Would you mind providing more improvement details (improvement points) with each operation?", "votes": 5, "replies": [ { "id": 2553022, "postDate": "2023-12-08T00:52:28.640Z", "content": "Yes, sure, I hope we will be able to explain this observation - https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/458478
", "rawMarkdown": "Yes, sure, I hope we will be able to explain this observation - https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/458478", "votes": 4 } ] }, { "id": 2553008, "postDate": "2023-12-08T00:39:09.843Z", "content": "Nice! Actually my model has also been published here: https://academic.oup.com/bib/article/24/1/bbac581/6986359. Congrats!
", "rawMarkdown": "Nice! Actually my model has also been published here: https://academic.oup.com/bib/article/24/1/bbac581/6986359. Congrats!", "votes": 6, "replies": [ { "id": 2553023, "postDate": "2023-12-08T00:53:19.323Z", "content": "We failed to google your publication initially, thanks!)
", "rawMarkdown": "We failed to google your publication initially, thanks!) ", "votes": 1, "replies": [ { "id": 2553026, "postDate": "2023-12-08T00:58:28.277Z", "content": "No problem! I actually didn't share my model because we realized models using bpp are a bit conservative on pseudo-knots and often miss them, so I didn't want to influence competitors to use bpp by sharing my paper, but I guess it ended up happening anyway haha
", "rawMarkdown": "No problem! I actually didn't share my model because we realized models using bpp are a bit conservative on pseudo-knots and often miss them, so I didn't want to influence competitors to use bpp by sharing my paper, but I guess it ended up happening anyway haha", "votes": 6 } ] } ] }, { "id": 2553032, "postDate": "2023-12-08T01:07:23.680Z", "content": "Great solution. Iâm very surprised by how similar it is to mine. congrats on winning!
", "rawMarkdown": "Great solution. Iâm very surprised by how similar it is to mine. congrats on winning!", "votes": 3 }, { "id": 2553018, "postDate": "2023-12-08T00:50:21.310Z", "content": "Congratulations, your solution is so stable!
", "rawMarkdown": "Congratulations, your solution is so stable!", "votes": 3 }, { "id": 2553173, "postDate": "2023-12-08T03:56:25.133Z", "content": "Congratulations on topping the leaderboard. Thanks for sharing the details of your solution with colorful diagrams and charts.
", "rawMarkdown": "Congratulations on topping the leaderboard. Thanks for sharing the details of your solution with colorful diagrams and charts. ", "votes": 4 }, { "id": 2575203, "postDate": "2023-12-26T16:52:02.887Z", "content": "Wow! Amazing⊠and congratulations
", "rawMarkdown": "Wow! Amazing... and congratulations", "votes": 1 }, { "id": 2566435, "postDate": "2023-12-18T19:47:51.347Z", "content": "Congratulations
", "rawMarkdown": "Congratulations\n", "votes": 1 }, { "id": 2558443, "postDate": "2023-12-12T07:01:45.770Z", "content": "Interesting read, congrats!
", "rawMarkdown": "Interesting read, congrats!", "votes": 1 }, { "id": 2557661, "postDate": "2023-12-11T15:46:04.120Z", "content": "congratulations, this is really nice!
", "rawMarkdown": "congratulations, this is really nice!", "votes": 1 }, { "id": 2557425, "postDate": "2023-12-11T13:33:40.570Z", "content": "For Dynamic Positional Bias: is there a relative length limit?
", "rawMarkdown": "For Dynamic Positional Bias: is there a relative length limit? \n", "votes": 1, "replies": [ { "id": 2557913, "postDate": "2023-12-11T18:06:37.723Z", "content": "We did not impose any explicit relative distance constraint; however, this layer may perform differently at relative distances greater than those in the training data set.
", "rawMarkdown": "We did not impose any explicit relative distance constraint; however, this layer may perform differently at relative distances greater than those in the training data set.", "votes": 1, "replies": [ { "id": 2557965, "postDate": "2023-12-11T19:08:54.850Z", "content": "interesting as this may lead to unexpected behavior at long distances. surprised you didn't cap it
", "rawMarkdown": "interesting as this may lead to unexpected behavior at long distances. surprised you didn't cap it" }, { "id": 2558064, "postDate": "2023-12-11T20:55:49.773Z", "content": "We followed the realization from https://github.com/bob80333/investigating_extrapolation and lucidrains x-transformers, where clipping is not done. And it's shows a good (https://github.com/bob80333/investigating_extrapolation/blob/master/experiments.txt) extrapolation performance
\nIf you look at Swin-Transformer realization - they also do no capping (although they do some scaling for distances). Also they use log-distances but in already mentioned experiments log-distances shown to behave worse than linear
\nWe have plans to test AlphaFold positional embedding used by the 3st place. This embedding clips positions at 32 but we are not sure if transferring the choice from protein modelling is the best one.
", "rawMarkdown": "We followed the realization from https://github.com/bob80333/investigating_extrapolation and lucidrains x-transformers, where clipping is not done. And it's shows a good (https://github.com/bob80333/investigating_extrapolation/blob/master/experiments.txt) extrapolation performance\n\nIf you look at Swin-Transformer realization - they also do no capping (although they do some scaling for distances). Also they use log-distances but in already mentioned experiments log-distances shown to behave worse than linear\n \nWe have plans to test AlphaFold positional embedding used by the 3st place. This embedding clips positions at 32 but we are not sure if transferring the choice from protein modelling is the best one. \n\n\n" } ] } ] }, { "id": 2555740, "postDate": "2023-12-10T06:21:00.163Z", "content": "Congratulations this is truly amazing . great idea and implementation
", "rawMarkdown": "Congratulations this is truly amazing . great idea and implementation ", "votes": 1 }, { "id": 2555205, "postDate": "2023-12-09T18:17:58.893Z", "content": "Wow, congratulations on the win! Your write up looks so professional too. Itâs not easy to maintain first place from public to private, and especially not in this competition!
", "rawMarkdown": "Wow, congratulations on the win! Your write up looks so professional too. Itâs not easy to maintain first place from public to private, and especially not in this competition!", "votes": 1 }, { "id": 2554352, "postDate": "2023-12-09T04:47:15.327Z", "content": "It is truly amazing
", "rawMarkdown": "It is truly amazing", "votes": 1 }, { "id": 2553452, "postDate": "2023-12-08T08:56:58.873Z", "content": "Congratulationsđ @vyaltsevvaleriy on your victory in the competition! Your work was not only impressive but also visually beautiful. I'm curious, what software did you use to create those visuals?
", "rawMarkdown": "Congratulationsđ @vyaltsevvaleriy on your victory in the competition! Your work was not only impressive but also visually beautiful. I'm curious, what software did you use to create those visuals?", "votes": 1, "replies": [ { "id": 2553711, "postDate": "2023-12-08T13:08:15.627Z", "content": "Hi! I'm very glad to hear that! I used a vector graphics editor called Inkscape.
", "rawMarkdown": "Hi! I'm very glad to hear that! I used a vector graphics editor called Inkscape.", "votes": 2 } ] }, { "id": 2553407, "postDate": "2023-12-08T08:23:42.537Z", "content": "Congratulations well done. Nice explanations and charts đ
", "rawMarkdown": "Congratulations well done. Nice explanations and charts đ", "votes": 1 }, { "id": 2553326, "postDate": "2023-12-08T06:39:13.563Z", "content": "A few assorted thoughts that came to mind that could be interesting to explore/think about
\nI think it would still be possible to use NUPACK. Just apply it only for some sequences. For example, we could select these sequences using active learning.
", "rawMarkdown": "I think it would still be possible to use NUPACK. Just apply it only for some sequences. For example, we could select these sequences using active learning.", "replies": [ { "id": 2553841, "postDate": "2023-12-08T14:47:00.250Z", "content": "If you're doing that, you would still presumably need to run nupack during inference though. For our use case on Eterna, even ~155 base strands were a problem in both runtime and memory usage just for MFE prediction (especially on lower-end hardware we target), not even BPPs (which has worse computational complexity). The time/memory complexity as it gets longer is pretty bad, so trying to use it on strands that are, say, a few thousand bases long (think SARS-CoV-2 spike protein mRNA or the larger parts of ribosomal RNA) is not viable (even presumably for higher-end hardware).
", "rawMarkdown": "If you're doing that, you would still presumably need to run nupack during inference though. For our use case on Eterna, even ~155 base strands were a problem in both runtime and memory usage _just for MFE prediction_ (especially on lower-end hardware we target), not even BPPs (which has worse computational complexity). The time/memory complexity as it gets longer is pretty bad, so trying to use it on strands that are, say, a few thousand bases long (think SARS-CoV-2 spike protein mRNA or the larger parts of ribosomal RNA) is not viable (even presumably for higher-end hardware).", "votes": 1 }, { "id": 2553894, "postDate": "2023-12-08T15:33:53.010Z", "content": "Exactly, I apologize, I didn't think about it in the middle of the night. But it can still be used, we just need to slightly change the approach. In theory, there is nothing stopping the Transformer from learning the \"real\" BPP in the attention layers simply from the sequence, if we train it on a large amount of data. We have limited data, but we could use distillation. First, create an accurate model on sequences and BPP, and then distill the knowledge from it into a model based only on sequences. In other domains, this usually leads to a significant improvement in metrics, and I believe this task will not be an exception.
", "rawMarkdown": "Exactly, I apologize, I didn't think about it in the middle of the night. But it can still be used, we just need to slightly change the approach. In theory, there is nothing stopping the Transformer from learning the \"real\" BPP in the attention layers simply from the sequence, if we train it on a large amount of data. We have limited data, but we could use distillation. First, create an accurate model on sequences and BPP, and then distill the knowledge from it into a model based only on sequences. In other domains, this usually leads to a significant improvement in metrics, and I believe this task will not be an exception.", "votes": 2 }, { "id": 2553897, "postDate": "2023-12-08T15:35:35.060Z", "content": "I'd love to see that approach!
", "rawMarkdown": "I'd love to see that approach!" }, { "id": 2554335, "postDate": "2023-12-09T04:17:33.890Z", "content": "Yeap, we thought about that. But it seems @dankrstev has a solution that is able to achieve that without distillation. Lets wait.
\nIt seems, there are multiple ways to integrate bpps knowledge in neural network (maybe, just forcing model to predict instead of providing model with bpps). Or somehow integrate the data used by eternafold during deducing parameters for their bpp-predicting model (it's not so complex so definitely can be learned by the network itself)
\nUnfortunately, we have no time during the competition to test this approaches. Actually, we tested so many things the post still lacking 10-20% of them and we will try to add it as soon as we will have time for that. E.g, we tried to allow the model to modify bpps given attention matricies. This lead to faster convergence (in number of epochs), but resulted in the same model performance.
\nAnyway we a not going to stop at this, maybe the further model development will be a part of @vyaltsevvaleriy diploma project. Especially given the fact organizers are going to publish raw data -- a whole room for testing and additional improvements.
", "rawMarkdown": "Yeap, we thought about that. But it seems @dankrstev has a solution that is able to achieve that without distillation. Lets wait.\n\nIt seems, there are multiple ways to integrate bpps knowledge in neural network (maybe, just forcing model to predict instead of providing model with bpps). Or somehow integrate the data used by eternafold during deducing parameters for their bpp-predicting model (it's not so complex so definitely can be learned by the network itself)\n\nUnfortunately, we have no time during the competition to test this approaches. Actually, we tested so many things the post still lacking 10-20% of them and we will try to add it as soon as we will have time for that. E.g, we tried to allow the model to modify bpps given attention matricies. This lead to faster convergence (in number of epochs), but resulted in the same model performance. \n\nAnyway we a not going to stop at this, maybe the further model development will be a part of @vyaltsevvaleriy diploma project. Especially given the fact organizers are going to publish raw data -- a whole room for testing and additional improvements.", "votes": 1 }, { "id": 2555084, "postDate": "2023-12-09T17:02:32.180Z", "content": "Yes, I think you can just embed information about BPP into the transformer. Although, I would take a Graph Transformer for such a case, it could be taught to simply predict BPP for edges, it seems more intuitive.
\nIn addition, there is a NodeFormer for graphs, where, with the same global attention, the complexity is reduced to O(N).
\nThere were thoughts of crossing NodeFormer with BPP, but we joined the competition only 2 weeks before the end, so I did not try.
@jonathanromano These bpps are all generated by LinearPartition with Eternafold's parameters actually, though locally it doesn't seem that much faster.
", "rawMarkdown": "@jonathanromano These bpps are all generated by LinearPartition with Eternafold's parameters actually, though locally it doesn't seem that much faster. ", "votes": 2, "replies": [ { "id": 2555431, "postDate": "2023-12-09T22:10:06.383Z", "content": "Interesting - admittedly I haven't gotten the chance to really sit down and play with it much. I assume linearpartition is being run with default parameters? IIRC depending on how you set beam size it changes how close to the original complexity it is (changing how much of the solution space gets thrown out)
", "rawMarkdown": "Interesting - admittedly I haven't gotten the chance to really sit down and play with it much. I assume linearpartition is being run with default parameters? IIRC depending on how you set beam size it changes how close to the original complexity it is (changing how much of the solution space gets thrown out)", "votes": 1 }, { "id": 2555441, "postDate": "2023-12-09T22:20:44.210Z", "content": "Yes. I used the default beam size which is kind of big (100), so for 200 long sequences, not much faster
", "rawMarkdown": "Yes. I used the default beam size which is kind of big (100), so for 200 long sequences, not much faster" }, { "id": 2555446, "postDate": "2023-12-09T22:28:31.573Z", "content": "Oh yeah, we are working on relatively small sequences - this would matter more if we were looking at longer sequences (going into the thousands of bases), which we do have use cases for
", "rawMarkdown": "Oh yeah, we are working on relatively small sequences - this would matter more if we were looking at longer sequences (going into the thousands of bases), which we do have use cases for" } ] } ] }, { "id": 2553246, "postDate": "2023-12-08T05:36:49.340Z", "content": "Not to rush on as you said you would release the code.
\nI am trying to understand the split of dataset and would want to understand:
Congratulations on winning. Excellent explanation of the methods used.
", "rawMarkdown": "Congratulations on winning. Excellent explanation of the methods used.", "votes": 1 }, { "id": 2553208, "postDate": "2023-12-08T04:46:37.470Z", "content": "Congrats! Definitely see a lot of similarities, but also some differences.
\nFor us relative worked better than dynamic positional bias and we were able to utilize pseudolabeling for a sizable boost.
Congratulations! Would you consider sharing the code?
", "rawMarkdown": "Congratulations! Would you consider sharing the code? ", "votes": 1, "replies": [ { "id": 2553019, "postDate": "2023-12-08T00:50:48.200Z", "content": "Yes, sure. We need some time to wrap it up
", "rawMarkdown": "Yes, sure. We need some time to wrap it up", "votes": 6 } ] }, { "id": 2553055, "postDate": "2023-12-08T01:45:17.623Z", "content": "Congratulations!!
\nWonderful work and great to learn.
Thanks for the incredibly detailed write up and congratulations!
", "rawMarkdown": "Thanks for the incredibly detailed write up and congratulations!", "votes": 2 }, { "id": 2553036, "postDate": "2023-12-08T01:14:17.540Z", "content": "Congratulations!
\nReally a wonderful work. learn a lot!
Congratulations on your first place! Our solution is very similar to yours! Great solutuion!
", "rawMarkdown": "Congratulations on your first place! Our solution is very similar to yours! Great solutuion!", "votes": 2 }, { "id": 2553020, "postDate": "2023-12-08T00:51:20.427Z", "content": "Congrats!
\nThanks for the great explanation.
", "rawMarkdown": "Congrats!\n\nThanks for the great explanation.", "votes": 2 }, { "id": 2602045, "postDate": "2024-01-14T20:58:22.943Z", "content": "@vyaltsevvaleriy @dmitrypenzar1996 I read your codes, and I have a question.
\nYou prepared
But you actually used only EternaFold BPP? Because this option doesn't use --brackets.
\npython3 python_scripts/train_uni_adjnet_se.py --bpp_path /projects_nvme/deepbeer/eterna/ --train_path /projects/deepbeer/ribonanza/train_data/train_data.parquet --out_path outmodel_dir --device 0 --num_workers 20 --wd 0.05 --epoch 270 --lr_max 5e-3 --pct_start 0.05 --batch_cnt 1791 --sgd_lr 5e-5 --sgd_epochs 25 --sgd_batch_cnt 500 --sgd_wd 0.05 --fold 0 --nfolds 1000 --pos_embedding dyn --adj_ks 3 --seed 42 --use_se
We used 2 and 3 for one model in our ensemble. In general, this features are completely useless, but they were used in final submission so we have to include them to git
", "rawMarkdown": "We used 2 and 3 for one model in our ensemble. In general, this features are completely useless, but they were used in final submission so we have to include them to git", "votes": 1, "replies": [ { "id": 2603110, "postDate": "2024-01-15T15:25:48.123Z", "content": "\n\nthis features are completely useless
\n
Unbelievable! I used bpp, structure, loop_type, chunk and segment.
\nYour model is so much simpler than I thought.
Did you tune the number of 12 transformer layers and 192 embedding?
", "rawMarkdown": ">this features are completely useless\n\nUnbelievable! I used bpp, structure, loop_type, chunk and segment.\nYour model is so much simpler than I thought.\n\nDid you tune the number of 12 transformer layers and 192 embedding?" }, { "id": 2618348, "postDate": "2024-01-24T17:20:59.567Z", "content": "We tried but there was no significan improvement.
", "rawMarkdown": "We tried but there was no significan improvement." } ] } ] }, { "id": 2557147, "postDate": "2023-12-11T08:46:28.860Z", "content": "May you share your CV-LB correlation?
", "rawMarkdown": "May you share your CV-LB correlation?" }, { "id": 2554943, "postDate": "2023-12-09T14:49:15.010Z", "content": "nice. congratulations
", "rawMarkdown": "nice. congratulations" }, { "id": 2553780, "postDate": "2023-12-08T14:06:06.373Z", "content": "Was any data pre-processing done on reactivity values, wherein the reactivity values are adjusted based on Signal to Noise and reactivity error?
", "rawMarkdown": "Was any data pre-processing done on reactivity values, wherein the reactivity values are adjusted based on Signal to Noise and reactivity error?", "replies": [ { "id": 2554322, "postDate": "2023-12-09T04:00:24.640Z", "content": "No, only preprocessing done by the organizers. We tried to use additional losses based on reactivities errors but they haven't resulted in increased performance
", "rawMarkdown": "No, only preprocessing done by the organizers. We tried to use additional losses based on reactivities errors but they haven't resulted in increased performance" } ] }, { "id": 2553759, "postDate": "2023-12-08T13:52:35.540Z", "content": "congratulations !
\npredicting reactivity of DMS given 2A3 and vice-versa is smart.
\nIs there some relation between DMS and 2A3 ? What was the intuition behind this thought?
\nIf you can share that will be great.
Both dms and 2a3 are reactivities and somehow show the RNA secondary structure. So, if we know one of them it should be easier to predict the second. We are really good at predicting dms values while 2a3 seems to be much more noisier and harder to predict. So predicting dms and then using it to predict 2a3 works (while 2a3->dms conversion doens't)
", "rawMarkdown": "Both dms and 2a3 are reactivities and somehow show the RNA secondary structure. So, if we know one of them it should be easier to predict the second. We are really good at predicting dms values while 2a3 seems to be much more noisier and harder to predict. So predicting dms and then using it to predict 2a3 works (while 2a3->dms conversion doens't)", "votes": 2 } ] }, { "id": 2553051, "postDate": "2023-12-08T01:38:00.170Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 2553074, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-12-08T01:56:43.433000", "content": "Congratulations! @vyaltsevvaleriy đđđ This is a very, very great method. May I ask what software was used to draw this picture? This is so beautiful
", "votes": 9, "replies": [ { "id": 2553451, "author_name": "try", "author_url": "", "post_date": "2023-12-08T08:56:55.440000", "content": "Haha, I also want to know
", "votes": 2, "replies": [] }, { "id": 2553696, "author_name": "vyaltsevvaleriy", "author_url": "", "post_date": "2023-12-08T12:56:38.877000", "content": "Hi! Thank you for your appreciation! I used Inkscape - it is a vector graphics editor allowing to flexibly implement any ideas.
", "votes": 10, "replies": [ { "id": 2553829, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-12-08T14:32:42.767000", "content": "Thank you very much! vyaltsevvaleriyđ„°
", "votes": 2, "replies": [] } ] } ] }, { "id": 2553004, "author_name": "medicine-wave", "author_url": "", "post_date": "2023-12-08T00:36:09.530000", "content": "Wonderful work and presentation! Congratulations! Would you mind providing more improvement details (improvement points) with each operation?
", "votes": 5, "replies": [ { "id": 2553022, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-08T00:52:28.640000", "content": "Yes, sure, I hope we will be able to explain this observation - https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/458478
", "votes": 4, "replies": [] } ] }, { "id": 2553008, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-08T00:39:09.843000", "content": "Nice! Actually my model has also been published here: https://academic.oup.com/bib/article/24/1/bbac581/6986359. Congrats!
", "votes": 6, "replies": [ { "id": 2553023, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-08T00:53:19.323000", "content": "We failed to google your publication initially, thanks!)
", "votes": 1, "replies": [ { "id": 2553026, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-08T00:58:28.277000", "content": "No problem! I actually didn't share my model because we realized models using bpp are a bit conservative on pseudo-knots and often miss them, so I didn't want to influence competitors to use bpp by sharing my paper, but I guess it ended up happening anyway haha
", "votes": 6, "replies": [] } ] } ] }, { "id": 2553032, "author_name": "hoyso48", "author_url": "", "post_date": "2023-12-08T01:07:23.680000", "content": "Great solution. Iâm very surprised by how similar it is to mine. congrats on winning!
", "votes": 3, "replies": [] }, { "id": 2553018, "author_name": "Xiang Huang", "author_url": "", "post_date": "2023-12-08T00:50:21.310000", "content": "Congratulations, your solution is so stable!
", "votes": 3, "replies": [] }, { "id": 2553173, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-12-08T03:56:25.133000", "content": "Congratulations on topping the leaderboard. Thanks for sharing the details of your solution with colorful diagrams and charts.
", "votes": 4, "replies": [] }, { "id": 2575203, "author_name": "Shivansh", "author_url": "", "post_date": "2023-12-26T16:52:02.887000", "content": "Wow! Amazing⊠and congratulations
", "votes": 1, "replies": [] }, { "id": 2566435, "author_name": "Md Nazrul Islam", "author_url": "", "post_date": "2023-12-18T19:47:51.347000", "content": "Congratulations
", "votes": 1, "replies": [] }, { "id": 2558443, "author_name": "Borna Jamali", "author_url": "", "post_date": "2023-12-12T07:01:45.770000", "content": "Interesting read, congrats!
", "votes": 1, "replies": [] }, { "id": 2557661, "author_name": "Zafir Khalid", "author_url": "", "post_date": "2023-12-11T15:46:04.120000", "content": "congratulations, this is really nice!
", "votes": 1, "replies": [] }, { "id": 2557425, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-11T13:33:40.570000", "content": "For Dynamic Positional Bias: is there a relative length limit?
", "votes": 1, "replies": [ { "id": 2557913, "author_name": "vyaltsevvaleriy", "author_url": "", "post_date": "2023-12-11T18:06:37.723000", "content": "We did not impose any explicit relative distance constraint; however, this layer may perform differently at relative distances greater than those in the training data set.
", "votes": 1, "replies": [ { "id": 2557965, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-11T19:08:54.850000", "content": "interesting as this may lead to unexpected behavior at long distances. surprised you didn't cap it
", "votes": 0, "replies": [] }, { "id": 2558064, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-11T20:55:49.773000", "content": "We followed the realization from https://github.com/bob80333/investigating_extrapolation and lucidrains x-transformers, where clipping is not done. And it's shows a good (https://github.com/bob80333/investigating_extrapolation/blob/master/experiments.txt) extrapolation performance
\nIf you look at Swin-Transformer realization - they also do no capping (although they do some scaling for distances). Also they use log-distances but in already mentioned experiments log-distances shown to behave worse than linear
\nWe have plans to test AlphaFold positional embedding used by the 3st place. This embedding clips positions at 32 but we are not sure if transferring the choice from protein modelling is the best one.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2555740, "author_name": "mhdaw", "author_url": "", "post_date": "2023-12-10T06:21:00.163000", "content": "Congratulations this is truly amazing . great idea and implementation
", "votes": 1, "replies": [] }, { "id": 2555205, "author_name": "Abhishek Shah", "author_url": "", "post_date": "2023-12-09T18:17:58.893000", "content": "Wow, congratulations on the win! Your write up looks so professional too. Itâs not easy to maintain first place from public to private, and especially not in this competition!
", "votes": 1, "replies": [] }, { "id": 2554352, "author_name": "Izam Mohammed", "author_url": "", "post_date": "2023-12-09T04:47:15.327000", "content": "It is truly amazing
", "votes": 1, "replies": [] }, { "id": 2553452, "author_name": "Choekyel", "author_url": "", "post_date": "2023-12-08T08:56:58.873000", "content": "Congratulationsđ @vyaltsevvaleriy on your victory in the competition! Your work was not only impressive but also visually beautiful. I'm curious, what software did you use to create those visuals?
", "votes": 1, "replies": [ { "id": 2553711, "author_name": "vyaltsevvaleriy", "author_url": "", "post_date": "2023-12-08T13:08:15.627000", "content": "Hi! I'm very glad to hear that! I used a vector graphics editor called Inkscape.
", "votes": 2, "replies": [] } ] }, { "id": 2553407, "author_name": "B@rro_07", "author_url": "", "post_date": "2023-12-08T08:23:42.537000", "content": "Congratulations well done. Nice explanations and charts đ
", "votes": 1, "replies": [] }, { "id": 2553326, "author_name": "Jonathan Romano", "author_url": "", "post_date": "2023-12-08T06:39:13.563000", "content": "A few assorted thoughts that came to mind that could be interesting to explore/think about
\nI think it would still be possible to use NUPACK. Just apply it only for some sequences. For example, we could select these sequences using active learning.
", "votes": 0, "replies": [ { "id": 2553841, "author_name": "Jonathan Romano", "author_url": "", "post_date": "2023-12-08T14:47:00.250000", "content": "If you're doing that, you would still presumably need to run nupack during inference though. For our use case on Eterna, even ~155 base strands were a problem in both runtime and memory usage just for MFE prediction (especially on lower-end hardware we target), not even BPPs (which has worse computational complexity). The time/memory complexity as it gets longer is pretty bad, so trying to use it on strands that are, say, a few thousand bases long (think SARS-CoV-2 spike protein mRNA or the larger parts of ribosomal RNA) is not viable (even presumably for higher-end hardware).
", "votes": 1, "replies": [] }, { "id": 2553894, "author_name": "Aleksey Trepetsky", "author_url": "", "post_date": "2023-12-08T15:33:53.010000", "content": "Exactly, I apologize, I didn't think about it in the middle of the night. But it can still be used, we just need to slightly change the approach. In theory, there is nothing stopping the Transformer from learning the \"real\" BPP in the attention layers simply from the sequence, if we train it on a large amount of data. We have limited data, but we could use distillation. First, create an accurate model on sequences and BPP, and then distill the knowledge from it into a model based only on sequences. In other domains, this usually leads to a significant improvement in metrics, and I believe this task will not be an exception.
", "votes": 2, "replies": [] }, { "id": 2553897, "author_name": "Jonathan Romano", "author_url": "", "post_date": "2023-12-08T15:35:35.060000", "content": "I'd love to see that approach!
", "votes": 0, "replies": [] }, { "id": 2554335, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-09T04:17:33.890000", "content": "Yeap, we thought about that. But it seems @dankrstev has a solution that is able to achieve that without distillation. Lets wait.
\nIt seems, there are multiple ways to integrate bpps knowledge in neural network (maybe, just forcing model to predict instead of providing model with bpps). Or somehow integrate the data used by eternafold during deducing parameters for their bpp-predicting model (it's not so complex so definitely can be learned by the network itself)
\nUnfortunately, we have no time during the competition to test this approaches. Actually, we tested so many things the post still lacking 10-20% of them and we will try to add it as soon as we will have time for that. E.g, we tried to allow the model to modify bpps given attention matricies. This lead to faster convergence (in number of epochs), but resulted in the same model performance.
\nAnyway we a not going to stop at this, maybe the further model development will be a part of @vyaltsevvaleriy diploma project. Especially given the fact organizers are going to publish raw data -- a whole room for testing and additional improvements.
", "votes": 1, "replies": [] }, { "id": 2555084, "author_name": "Aleksey Trepetsky", "author_url": "", "post_date": "2023-12-09T17:02:32.180000", "content": "Yes, I think you can just embed information about BPP into the transformer. Although, I would take a Graph Transformer for such a case, it could be taught to simply predict BPP for edges, it seems more intuitive.
\nIn addition, there is a NodeFormer for graphs, where, with the same global attention, the complexity is reduced to O(N).
\nThere were thoughts of crossing NodeFormer with BPP, but we joined the competition only 2 weeks before the end, so I did not try.
@jonathanromano These bpps are all generated by LinearPartition with Eternafold's parameters actually, though locally it doesn't seem that much faster.
", "votes": 2, "replies": [ { "id": 2555431, "author_name": "Jonathan Romano", "author_url": "", "post_date": "2023-12-09T22:10:06.383000", "content": "Interesting - admittedly I haven't gotten the chance to really sit down and play with it much. I assume linearpartition is being run with default parameters? IIRC depending on how you set beam size it changes how close to the original complexity it is (changing how much of the solution space gets thrown out)
", "votes": 1, "replies": [] }, { "id": 2555441, "author_name": "Shujun", "author_url": "", "post_date": "2023-12-09T22:20:44.210000", "content": "Yes. I used the default beam size which is kind of big (100), so for 200 long sequences, not much faster
", "votes": 0, "replies": [] }, { "id": 2555446, "author_name": "Jonathan Romano", "author_url": "", "post_date": "2023-12-09T22:28:31.573000", "content": "Oh yeah, we are working on relatively small sequences - this would matter more if we were looking at longer sequences (going into the thousands of bases), which we do have use cases for
", "votes": 0, "replies": [] } ] } ] }, { "id": 2553246, "author_name": "FdotRK", "author_url": "", "post_date": "2023-12-08T05:36:49.340000", "content": "Not to rush on as you said you would release the code.
\nI am trying to understand the split of dataset and would want to understand:
Congratulations on winning. Excellent explanation of the methods used.
", "votes": 1, "replies": [] }, { "id": 2553208, "author_name": "sroger", "author_url": "", "post_date": "2023-12-08T04:46:37.470000", "content": "Congrats! Definitely see a lot of similarities, but also some differences.
\nFor us relative worked better than dynamic positional bias and we were able to utilize pseudolabeling for a sizable boost.
Congratulations! Would you consider sharing the code?
", "votes": 1, "replies": [ { "id": 2553019, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2023-12-08T00:50:48.200000", "content": "Yes, sure. We need some time to wrap it up
", "votes": 6, "replies": [] } ] }, { "id": 2553055, "author_name": "FdotRK", "author_url": "", "post_date": "2023-12-08T01:45:17.623000", "content": "Congratulations!!
\nWonderful work and great to learn.
Thanks for the incredibly detailed write up and congratulations!
", "votes": 2, "replies": [] }, { "id": 2553036, "author_name": "Horikita Saku", "author_url": "", "post_date": "2023-12-08T01:14:17.540000", "content": "Congratulations!
\nReally a wonderful work. learn a lot!
Congratulations on your first place! Our solution is very similar to yours! Great solutuion!
", "votes": 2, "replies": [] }, { "id": 2553020, "author_name": "Antonio FĂ©lix", "author_url": "", "post_date": "2023-12-08T00:51:20.427000", "content": "Congrats!
\nThanks for the great explanation.
", "votes": 2, "replies": [] }, { "id": 2602045, "author_name": "ONODERA", "author_url": "", "post_date": "2024-01-14T20:58:22.943000", "content": "@vyaltsevvaleriy @dmitrypenzar1996 I read your codes, and I have a question.
\nYou prepared
But you actually used only EternaFold BPP? Because this option doesn't use --brackets.
\npython3 python_scripts/train_uni_adjnet_se.py --bpp_path /projects_nvme/deepbeer/eterna/ --train_path /projects/deepbeer/ribonanza/train_data/train_data.parquet --out_path outmodel_dir --device 0 --num_workers 20 --wd 0.05 --epoch 270 --lr_max 5e-3 --pct_start 0.05 --batch_cnt 1791 --sgd_lr 5e-5 --sgd_epochs 25 --sgd_batch_cnt 500 --sgd_wd 0.05 --fold 0 --nfolds 1000 --pos_embedding dyn --adj_ks 3 --seed 42 --use_se
We used 2 and 3 for one model in our ensemble. In general, this features are completely useless, but they were used in final submission so we have to include them to git
", "votes": 1, "replies": [ { "id": 2603110, "author_name": "ONODERA", "author_url": "", "post_date": "2024-01-15T15:25:48.123000", "content": "\n\nthis features are completely useless
\n
Unbelievable! I used bpp, structure, loop_type, chunk and segment.
\nYour model is so much simpler than I thought.
Did you tune the number of 12 transformer layers and 192 embedding?
", "votes": 0, "replies": [] }, { "id": 2618348, "author_name": "Penzar Dmitry", "author_url": "", "post_date": "2024-01-24T17:20:59.567000", "content": "We tried but there was no significan improvement.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2557147, "author_name": "Correlation", "author_url": "", "post_date": "2023-12-11T08:46:28.860000", "content": "May you share your CV-LB correlation?
", "votes": 0, "replies": [] }, { "id": 2554943, "author_name": "Durlav", "author_url": "", "post_date": "2023-12-09T14:49:15.010000", "content": "nice. congratulations
", "votes": 0, "replies": [] }, { "id": 2553780, "author_name": "", "author_url": "", "post_date": "2023-12-08T14:06:06.373000", "content": "", "votes": 0, "replies": [ { "id": 2554322, "author_name": "", "author_url": "", "post_date": "2023-12-09T04:00:24.640000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2553759, "author_name": "", "author_url": "", "post_date": "2023-12-08T13:52:35.540000", "content": "", "votes": 0, "replies": [ { "id": 2554325, "author_name": "", "author_url": "", "post_date": "2023-12-09T04:02:22.970000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 2553051, "author_name": "", "author_url": "", "post_date": "2023-12-08T01:38:00.170000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2553003": "The Ribonanza RNA Folding competition has been an amazing opportunity and we are really glad to have participated in it. Kudos to the hosts for their substantial effort in keeping the high organizational level and giving the contestants plenty of valuable insight and support. We would also like to thank the community for the fruitful public discussions and coding initiatives. Congratulations to everyone involved! đ\n\n# Code\nOpen source code is available on [github](https://github.com/autosome-ru/vigg_ribonanza/).\n\n# TLDR\n\nOur solution is based on transformer architecture. We predicted base pair probability matrix(BPPM) for each RNA sequence using EternaFold and added convolutional blocks to our architecture to process BPPM features. For some models we added Squeeze-and-Excitation layer in convolutional blocks Also we add these features with attention values before softmax operation in Self-Attention block. To allow better generalization for longer input we implemented Dynamic Positional Bias. Then we ensembled models with slight differences in architecture and training process.\n\n# Data Preprocessing\n\nAt first, for each sequence in train and test datasets we calculated a base pair probability matrix (BPPM) using EternaFold. During training and inference phases we passed to our model the RNA sequences encoded with a learnable embedding layer. Each nucleotide was considered a token and special **< start >** and **< end >** tokens were placed at both ends as well. To provide the model with the information about whether the sequence comes from a âcleanâ subset of training dataset (which SN_filter values show - 1 corresponds to âcleanâ sequences with high signal-to-noise ratio and 0 is otherwise) we encoded SN_filter values with a learnable embedding layer and added corresponding embeddings to sequence embeddings. The embedding dimension was chosen to be of size **192**. BPPMs were padded with zeros at their margins to account for adding **< start >** and **< end >** tokens. The figure below summarizes the data preprocessing part.\n\n\n\n# Model\n\nThe model idea is partially inspired by @shujun717 solution to the Open Vaccine challenge, [link](https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189564)\n\nThe model takes a sequence of tokens and BPPM as input and outputs DMS_MaP and 2A3_MaP reactivities for each input nucleotide. Its architecture comprises 12 consecutive Transformer Encoder Layers and an output projection linear layer. Each Transformer block takes a sequence of tokens and BPPM features from the previous layer and outputs updated feature maps as shown below. The Transformer Encoder block adopts common transformer encoder architecture, except we modified the Self-Attention block to ensure interaction between BPPM features and sequence features.\n\n\n\n## Self-Attention Block\n\nIn the Self-Attention (SA) block we implemented an attention mechanism with the following modifications: after attention values for each head are calculated, we add BPPM features updated by the âConvolutional blockâ, which outputs BPPM features with the number of channels corresponding to the number of heads in the SA block. We set the number of heads in SA and corresponding channels of BPPM features to be 6. Thus, the hidden dimension size of Q, K, V matrices is 32. The overall structure of the SA block is shown below.\n\n\n\n## Dynamic Positional Bias\n\nThe sequence length is distributed differently in train and test datasets (test sequences are generally longer). To allow for the better generalization of longer inputs we implemented a positional encoding to be added to attention values. We found Dynamic Positional Bias to be of more use compared to other relative positional encoding methods we tried, such as xPos (rotary positional encoding) and ALiBi. Dynamic Positional Bias calculates for each head a relative positional bias map, which is learnable and depends on sequence length. Relative positional bias doesnât allow to leverage distance from start and end of sequence, so tokens **< start >** and **< end >** were added to fix that.\n\n.png?generation=1701994757603986&alt=media)\n\n## Convolutional Block and SE block\n\nThe models in the final ensemble come in two versions that are slightly different in the structure of the Convolutional block. The basic Convolutional block consists of 2D convolutional layer, batchnorm layer, activation and learnable gamma parameters that scale the output feature channels, whereas the modified version of this block (SE-Convolutional block) also contains Squeeze-and-Excitation layer, hence the name. The SE layer applies input-dependent rescaling of values along the channels as shown below. Thus, the only difference between models in the ensemble is the presence of the SE layer.\n\n\n\n# Training Process\n\nThe training process has been adapted from the [notebook](https://www.kaggle.com/code/iafoss/rna-starter-0-186-lb/notebook) by @IAFOSS\nWe used a one-cycle learning rate schedule (pct_start=0.05, lr_max=2.5e-3,) coupled with AdamW optimizer (wd=0.05), and batch size set to 128.\nThe number of epochs for model training was determined from the dataset size.\nFor final models (trained almost on the whole dataset) we used 270 epochs. \nIn each epoch, 1791 batches were processed (we kept this number due to historical reasons), elements of each batch were sampled from dataset with the weight = 0.5 * torch.clamp_min(torch.log(sn + 1.01),0.01)\n\nWe have also found that training model with a simple SGD optimizer for ~15 epochs of 500 batches each additionally improves model performance (the number of epochs varies so we used a small validation set to determine the exact number of epochs)\n\nAdditionally, one model was trained to predict 2A3 given DMS, RNA sequence and BPPM (dms-to-2a3 model).\n\n# Inference\nAt the inference phase for all inputs we set the SN filter value to be 1 as if they come from a âcleanâ dataset.\n\n# Ensembling\n\nWe ensembled 15 models with SE-Convolutional block, 10 models with plain Convolutional block, 2 with plain Convolutional block trained on split by sequence lengths (one of these two models accepts bracket features). We took the average of their predictions. Then we predicted 2a3 reactivities based on averaged dms reactivities using dms-to-2a3 model and added these predictions to averaged 2a3 reactivities in the following way: (27/28)*averaged_2a3 + (1/28)*predicted_2a3.\n\n# Other splits \n\nThe clear problem with a simple KFold split is the high sequence similarity across the training dataset and the fact that test sequences are very distinct from the training data. This might hamper the model development, because the increase in quality on the validation set could be due to overfitting rather than actual improvement.\n\nWe have calculated a hamming distance matrix for all the sequences present in the training set and performed a modified DBSCAN clustering procedure with distance threshold set at 0.2. In the following picture we show the clusters mapped to their respective cluster identifier (a cluster ID was assigned as a number of the smallest sequence within that cluster in the train dataset).\n\n\n\nWe tested several splits based on sequence identity (the easiest one is just to split data into folds without shuffling) and found out that while the model final validation performance degrades as we choose more and more stricter distance threshold, its relative value behave the same way as for a simple KFold split. So, we decided to use simple KFold split than training final models \n\nAlso, we have conducted a test of the model performance on length-based split. For that, we trained a model on the short sequences (length < 206) and validated it on sequences of size = 206. The behavior of the validation metric was slightly noisier but still highly correlated with a metric for a simple KFold split.\n\n# Public data leakage \n\nApproximately 13% of public test sequences are identical to the ones present in the train dataset (by sequence). To avoid selecting a model that is memorizing more of these sequences rather than learning RNA-related stuff, we zeroed out the predictions for these sequences in most of our submissions, while sometimes sending non-zeroed out submissions in order to compare our performance to other participants.\n\n# Features we also tried\n\n## capR\n\nWe donât have conclusive results for that feature. It seems like it is not beneficial for the model on average but sometimes it resulted in a better model and sometimes â in worse. We decided to not use this feature.\n\n## Brackets\n\nWe have tried to use brackets generated by EternaFold, ContraFold, ViennaRNA, etc., as well as programs for pseudoknots prediction (like IPknot). However, after adding EternaFold BPPMs to the model, adding other features yields no significant increase in model performance. For some models we used brackets just to augment model\n\n## Different BPPMs\n\nAll other BPPMs (ContraFold, ViennaRNA, RNAsoft, RNAstructure) result in a suboptimal model.\nAveraging BPPMs doesnât result in better performance.\nBPPMs from RFold yield the same quality as ViennaRNA BPPMs.\nThe RNA-FM model produces both per-nucleotide embeddings and BPPM-like matrix. Still, those donât help the model at all, and using them results only in a slight increase in model performance then compared to sequence only model\n \n## SQUARNA matrix\n\nSQUARNA ([github](https://github.com/febos/SQUARNA)) outputs a matrix different from BPPM, but can be used in the same manner. Unfortunately, this feature also didnât yield any additional performance increase.\n\n# What we also tried\n\n## Fully-Convolutional architecture\n\nThe initial reason we decided to take part in the competition was to test our model LegNet [link](https://academic.oup.com/bioinformatics/article/39/8/btad457/7230784), which shows SOTA results on DNA sequences and worked well on some RNA-related tasks (not yet published).\nUnfortunately, any modifications of this architecture resulted in subpar performance when compared with properly tuned transformer models. This can be explained by the fact that predicting RNA secondary structure requires attending to long-range contacts. The transformer architecture suits better for such cases.\n\n## Subsetting data \n\nSubsetting data (filtering by different thresholds on SN ratio) resulted in a performance boost for all models. However, this technique was superseded by weight sampling, which proved itself to be more effective.\n\n## Fine-tuning on public datasets \n\nWe tried to fine-tune our model on a public dataset, gathered by the organizers ([link](https://www.kaggle.com/datasets/rhijudas/rmdb-rna-mapping-database-2023-data/data)) by training model to predict the results of the public experiments (excluding ones with a small number of samples) and for Ribonanza data simultaneously. Unfortunately, this also gave no boost to the model performance.\n\n## Using 3D data \n\nWe tried to use data about predicted 3D structure of 100k sequences from the train dataset but gave up on that once we had visually analyzed them:\n\n \n\n## Absolute positional embedding\n\nUsing absolute positional embedding leads to unsolvable issues when generalizing upon longer sequences.\n\n## Relative positional embedding \n\nThe simplest approach is to augment absolute positional encoding during the training phase to shift randomly from 0 to (Lmax - seqlen) position. This indeed solves the issue with extrapolation, but works worse than other methods.\nRotational positional embedding, unfortunately, doesnât help the model to generalize on larger lengths.\nALiBi positional embedding solves the issue with extrapolation but even after keeping it only for a part of heads (as suggested in https://github.com/lucidrains/x-transformers) still behaves worse than dynamic positional bias.\n\n## Augmentation \n\nFirst, we tried to use reverse augmentation. This can be done in three ways:\nreversing sequence before any modifications,\nreversing sequence before padding, but after adding