First off, I wanted to thank the competition hosts for putting together this competition as well as the community for all of the discussion.
\nThe input features were:
\nI was hesitant to include the BPPM for a few reasons that I thought would result in hindering the generalized performance for longer sequences:
\nMy compromise was to consider the global probability of pairing for each base. This paper from the Das lab assessed how Pr(paired) as calculated by various RNA structure packages compared to chemical mapping assay results. They found CONTRAfold 2, ViennaRNA 2 at 60 degrees C, and RNAsoft with BLStar parameters to be the most effective approaches. Therefore, I generated Pr(paired) vectors for all sequences from these three methods as well as Eternafold to use as features.
\nAs an aside, I used Arnie to run these packages and spent more time than I care to admit getting it setup for multiprocessing, although overall I think it ended up saving time. I did try to expand to additional methods, however, I pretty quickly abandoned it due to bugs / run time constraints.
\nEarlier on in the competition, I did also try various structure features generated from ViennaRNA (MFE, centroid, ensemble). Compared to sequence alone, I was able to get improved performance, however, it added no benefit once the Pr(base paired) feature was included.
\nMy approach ended up being a single transformer architecture separately trained to predict 2A3 or DMS.
\nSeq
\nPr(base paired)
\nTransformer
\nThis was my first time working with a transformer architecture. Luckily, I found the library x-transformers by @lucidrains and used that to try out / learn about numerous variations of transformers.
\nWhat helped:
\nDynamic Position Bias
\nOne of my first goals was to understand how to generalize a transformer to longer sequences. Initially I found the ALiBi paper and thought that would be a useful approach. However, based on the x-transformers documentation it appeared as if dynamic position bias would be more likely to be effective in this context. I did not attempt any direct comparisons between ALiBi (or other relative position biases) and dynamic position bias since I did not trust performance differences on length extrapolation from the train set alone. Dynamic position bias did show the largest bump of any modification on the public vs. private sets (public - 0.15400 vs 0.15002; private - 0.18476 vs 0.14976). Note: this result was before the change to using Pr(base paired) data.
Intra-attention Gating on Values
\nThis is an attention variant from Alphafold2 that gates the aggregated values with the input. This was one of the feature variations available on x-transformers that appeared to have a small improvement on performance. Note: I added this one a while back and just kept it in, so it is entirely possible it adds nothing of value in the final model.
Postnorm
\nOne of the first changes I assessed with the transformer was to switch from the generally more preferred prenorm to the original postnorm approach. This resulted in improvements to convergence rate and performance. I rechecked this multiple times over the course of the competition as the architecture and input data approaches shift and consistently got the same type of result. I also attempted the various other norm approaches found in x-transformers, but none resulted in any noticeable improvement.
18 Layers
\n18 layers seemed to be the sweet spot of performance and training time. The main depths I tested were 12, 18, 24. 12 vs 18 showed a small performance improvement on the leader boards (public - 0.14383 vs 0.14330; private - 0.14595 vs 0.14517). 18 vs 24 I did not notice any difference in the CV score, so I did not submit any 24 layer models.
Sum embeds
\nMy approach to integrating the embedded sequence and the Pr(base paired) after passing through a linear layer (both of dimension 512) was to simply sum them elementwise. I tried other approaches, but they all appeared to have negative impacts to convergence and/or performance.
I used the splits used in @iafoss starter transformer notebook and all of my experimental work was done using fold 0. Because I ended up using individual models for 2A3 and DMS prediction, the set of sequences satisfying SN >= 1.0 and reads >= 100 was higher for each group since there was a subset of sequences where only one of 2A3/DMS satisfied the constraint.
\nAdditionally, I incorporated the react_error values as a means of weighting the loss so that I could use more of the lower quality sequences with greater confidence (luckily this happened right after the time the issue with react_error was identified and corrected). The react_error values were strictly greater than 0 and were usually fairly low (I believe the median on at least the high quality samples was ~0.125). I decided to try weighting the loss of each base using an exponential function:
\nWeighting v1:
\nexp(-react_error)
Weighting v2:
\nexp(-(react_error^2))
This would result in a weight value between 0 and 1 for each base in the sequence. While I don’t have a good mathematical rationale for this approach and I am sure there may be better options, it did give a noticeable boost to CV performance (~0.0005 - 0.001), including when using only the higher quality sequences.
\nMy approach to training ended up being somewhat erratic for the final model.
\nAs I mentioned above, I used fold 0 of my CV splits for all experimentation. Unfortunately, I lost time testing out other architectural approaches in the final 2 weeks and only settled on coming back to the split models due to the slight performance improvement within the last day, so I was unable to run the full set of CV folds. However, in a last minute effort to at least capture some of the fold 0 validation data in my model I ended up running 1 epoch on this data for my second final submission, which ended up giving me a nice little last minute boost on the leaderboard (public: 0.14330 vs 0.14199; private: 0.14517 vs 0.14458).
", "messages": [ { "id": 2555320, "postDate": "2023-12-09T20:45:44.187Z", "content": "First off, I wanted to thank the competition hosts for putting together this competition as well as the community for all of the discussion.
\nThe input features were:
\nI was hesitant to include the BPPM for a few reasons that I thought would result in hindering the generalized performance for longer sequences:
\nMy compromise was to consider the global probability of pairing for each base. This paper from the Das lab assessed how Pr(paired) as calculated by various RNA structure packages compared to chemical mapping assay results. They found CONTRAfold 2, ViennaRNA 2 at 60 degrees C, and RNAsoft with BLStar parameters to be the most effective approaches. Therefore, I generated Pr(paired) vectors for all sequences from these three methods as well as Eternafold to use as features.
\nAs an aside, I used Arnie to run these packages and spent more time than I care to admit getting it setup for multiprocessing, although overall I think it ended up saving time. I did try to expand to additional methods, however, I pretty quickly abandoned it due to bugs / run time constraints.
\nEarlier on in the competition, I did also try various structure features generated from ViennaRNA (MFE, centroid, ensemble). Compared to sequence alone, I was able to get improved performance, however, it added no benefit once the Pr(base paired) feature was included.
\nMy approach ended up being a single transformer architecture separately trained to predict 2A3 or DMS.
\nSeq
\nPr(base paired)
\nTransformer
\nThis was my first time working with a transformer architecture. Luckily, I found the library x-transformers by @lucidrains and used that to try out / learn about numerous variations of transformers.
\nWhat helped:
\nDynamic Position Bias
\nOne of my first goals was to understand how to generalize a transformer to longer sequences. Initially I found the ALiBi paper and thought that would be a useful approach. However, based on the x-transformers documentation it appeared as if dynamic position bias would be more likely to be effective in this context. I did not attempt any direct comparisons between ALiBi (or other relative position biases) and dynamic position bias since I did not trust performance differences on length extrapolation from the train set alone. Dynamic position bias did show the largest bump of any modification on the public vs. private sets (public - 0.15400 vs 0.15002; private - 0.18476 vs 0.14976). Note: this result was before the change to using Pr(base paired) data.
Intra-attention Gating on Values
\nThis is an attention variant from Alphafold2 that gates the aggregated values with the input. This was one of the feature variations available on x-transformers that appeared to have a small improvement on performance. Note: I added this one a while back and just kept it in, so it is entirely possible it adds nothing of value in the final model.
Postnorm
\nOne of the first changes I assessed with the transformer was to switch from the generally more preferred prenorm to the original postnorm approach. This resulted in improvements to convergence rate and performance. I rechecked this multiple times over the course of the competition as the architecture and input data approaches shift and consistently got the same type of result. I also attempted the various other norm approaches found in x-transformers, but none resulted in any noticeable improvement.
18 Layers
\n18 layers seemed to be the sweet spot of performance and training time. The main depths I tested were 12, 18, 24. 12 vs 18 showed a small performance improvement on the leader boards (public - 0.14383 vs 0.14330; private - 0.14595 vs 0.14517). 18 vs 24 I did not notice any difference in the CV score, so I did not submit any 24 layer models.
Sum embeds
\nMy approach to integrating the embedded sequence and the Pr(base paired) after passing through a linear layer (both of dimension 512) was to simply sum them elementwise. I tried other approaches, but they all appeared to have negative impacts to convergence and/or performance.
I used the splits used in @iafoss starter transformer notebook and all of my experimental work was done using fold 0. Because I ended up using individual models for 2A3 and DMS prediction, the set of sequences satisfying SN >= 1.0 and reads >= 100 was higher for each group since there was a subset of sequences where only one of 2A3/DMS satisfied the constraint.
\nAdditionally, I incorporated the react_error values as a means of weighting the loss so that I could use more of the lower quality sequences with greater confidence (luckily this happened right after the time the issue with react_error was identified and corrected). The react_error values were strictly greater than 0 and were usually fairly low (I believe the median on at least the high quality samples was ~0.125). I decided to try weighting the loss of each base using an exponential function:
\nWeighting v1:
\nexp(-react_error)
Weighting v2:
\nexp(-(react_error^2))
This would result in a weight value between 0 and 1 for each base in the sequence. While I don’t have a good mathematical rationale for this approach and I am sure there may be better options, it did give a noticeable boost to CV performance (~0.0005 - 0.001), including when using only the higher quality sequences.
\nMy approach to training ended up being somewhat erratic for the final model.
\nAs I mentioned above, I used fold 0 of my CV splits for all experimentation. Unfortunately, I lost time testing out other architectural approaches in the final 2 weeks and only settled on coming back to the split models due to the slight performance improvement within the last day, so I was unable to run the full set of CV folds. However, in a last minute effort to at least capture some of the fold 0 validation data in my model I ended up running 1 epoch on this data for my second final submission, which ended up giving me a nice little last minute boost on the leaderboard (public: 0.14330 vs 0.14199; private: 0.14517 vs 0.14458).
", "rawMarkdown": "First off, I wanted to thank the competition hosts for putting together this competition as well as the community for all of the discussion.\n\n# Summary\n- Transformer architecture\n - Separate models for 2A3 and DMS\n- Used Pr(base paired) derived from BPPM\n- Dynamic position bias\n\n# Approach\n\n## Data:\nThe input features were:\n- Sequence\n- Pr(base paired) (i.e. sum along one axis of the BPPM) from 4 RNA structure packages\n\nI was hesitant to include the BPPM for a few reasons that I thought would result in hindering the generalized performance for longer sequences:\n1. Base-base pairing probability is overly specific ‘wrong’ information\n2. Lack of consideration of pseudoknots\n3. Dilution of individual probabilities for longer test sequences compared to the shorter train sequences\n\nMy compromise was to consider the global probability of pairing for each base. [This paper from the Das lab](https://www.nature.com/articles/s41592-022-01605-0) assessed how Pr(paired) as calculated by various RNA structure packages compared to chemical mapping assay results. They found CONTRAfold 2, ViennaRNA 2 at 60 degrees C, and RNAsoft with BLStar parameters to be the most effective approaches. Therefore, I generated Pr(paired) vectors for all sequences from these three methods as well as Eternafold to use as features.\n\nAs an aside, I used Arnie to run these packages and spent more time than I care to admit getting it setup for multiprocessing, although overall I think it ended up saving time. I did try to expand to additional methods, however, I pretty quickly abandoned it due to bugs / run time constraints.\n\nEarlier on in the competition, I did also try various structure features generated from ViennaRNA (MFE, centroid, ensemble). Compared to sequence alone, I was able to get improved performance, however, it added no benefit once the Pr(base paired) feature was included.\n\n## Architecture:\nMy approach ended up being a single transformer architecture separately trained to predict 2A3 or DMS.\n\n**Seq**\n* 4 token embedding -> 512 output dimension\n\n**Pr(base paired)**\n* Linear layer 4 input dimension -> 512 output dimension\n\n**Transformer**\n* 18 layers, 512 dimension, 8 heads\n* Postnorm\n* Dynamics position bias\n* Intra-attention Gating on Values\n* Input: Seq + Pr(base paired)\n\nThis was my first time working with a transformer architecture. Luckily, I found the library [x-transformers](https://github.com/lucidrains/x-transformers) by @lucidrains and used that to try out / learn about numerous variations of transformers.\n\n**What helped:**\n*Dynamic Position Bias*\nOne of my first goals was to understand how to generalize a transformer to longer sequences. Initially I found the ALiBi paper and thought that would be a useful approach. However, based on the x-transformers documentation it appeared as if dynamic position bias would be more likely to be effective in this context. I did not attempt any direct comparisons between ALiBi (or other relative position biases) and dynamic position bias since I did not trust performance differences on length extrapolation from the train set alone. Dynamic position bias did show the largest bump of any modification on the public vs. private sets (public - 0.15400 vs 0.15002; private - 0.18476 vs 0.14976). Note: this result was before the change to using Pr(base paired) data.\n\n*Intra-attention Gating on Values*\nThis is an attention variant from Alphafold2 that gates the aggregated values with the input. This was one of the feature variations available on x-transformers that appeared to have a small improvement on performance. Note: I added this one a while back and just kept it in, so it is entirely possible it adds nothing of value in the final model.\n\n*Postnorm*\nOne of the first changes I assessed with the transformer was to switch from the generally more preferred prenorm to the original postnorm approach. This resulted in improvements to convergence rate and performance. I rechecked this multiple times over the course of the competition as the architecture and input data approaches shift and consistently got the same type of result. I also attempted the various other norm approaches found in x-transformers, but none resulted in any noticeable improvement.\n\n*18 Layers*\n18 layers seemed to be the sweet spot of performance and training time. The main depths I tested were 12, 18, 24. 12 vs 18 showed a small performance improvement on the leader boards (public - 0.14383 vs 0.14330; private - 0.14595 vs 0.14517). 18 vs 24 I did not notice any difference in the CV score, so I did not submit any 24 layer models.\n\n*Sum embeds*\nMy approach to integrating the embedded sequence and the Pr(base paired) after passing through a linear layer (both of dimension 512) was to simply sum them elementwise. I tried other approaches, but they all appeared to have negative impacts to convergence and/or performance.\n\n## Training:\n\nI used the splits used in @iafoss starter transformer notebook and all of my experimental work was done using fold 0. Because I ended up using individual models for 2A3 and DMS prediction, the set of sequences satisfying SN >= 1.0 and reads >= 100 was higher for each group since there was a subset of sequences where only one of 2A3/DMS satisfied the constraint.\n\nAdditionally, I incorporated the react_error values as a means of weighting the loss so that I could use more of the lower quality sequences with greater confidence (luckily this happened right after the time the issue with react_error was identified and corrected). The react_error values were strictly greater than 0 and were usually fairly low (I believe the median on at least the high quality samples was ~0.125). I decided to try weighting the loss of each base using an exponential function:\n\nWeighting v1:\nexp(-react_error)\n\nWeighting v2:\nexp(-(react_error^2))\n\nThis would result in a weight value between 0 and 1 for each base in the sequence. While I don’t have a good mathematical rationale for this approach and I am sure there may be better options, it did give a noticeable boost to CV performance (~0.0005 - 0.001), including when using only the higher quality sequences.\n\nMy approach to training ended up being somewhat erratic for the final model.\n\n- 2A3 model\n - fit_one_cycle(epochs=256, lr=5e-4, pctstart=0.01)\n - SN >= 1.0, reads >= 100\n - exp(-(react_error^2)) weighting\n - Early stopping at epoch 45\n- 2A3 model part 2\n - Fine tune 2A3 model, lr=5e-5\n - SN >= 0.2, reads >= 75\n - exp(-react_error) weighting\n- DMS model\n - Fine tune 2A3 model part 2, lr=5e-5\n - SN >= 1.0, reads >= 100\n - exp(-(react_error^2)) weighting\n- DMS model part 2\n - Fine tune DMS model, lr=5e-5\n - SN >= 0.2, reads >= 75\n - exp(-react_error) weighting\n\nAs I mentioned above, I used fold 0 of my CV splits for all experimentation. Unfortunately, I lost time testing out other architectural approaches in the final 2 weeks and only settled on coming back to the split models due to the slight performance improvement within the last day, so I was unable to run the full set of CV folds. However, in a last minute effort to at least capture some of the fold 0 validation data in my model I ended up running 1 epoch on this data for my second final submission, which ended up giving me a nice little last minute boost on the leaderboard (public: 0.14330 vs 0.14199; private: 0.14517 vs 0.14458).", "votes": 12 } ], "comments": [], "raw_markdown_by_id": { "2555320": "First off, I wanted to thank the competition hosts for putting together this competition as well as the community for all of the discussion.\n\n# Summary\n- Transformer architecture\n - Separate models for 2A3 and DMS\n- Used Pr(base paired) derived from BPPM\n- Dynamic position bias\n\n# Approach\n\n## Data:\nThe input features were:\n- Sequence\n- Pr(base paired) (i.e. sum along one axis of the BPPM) from 4 RNA structure packages\n\nI was hesitant to include the BPPM for a few reasons that I thought would result in hindering the generalized performance for longer sequences:\n1. Base-base pairing probability is overly specific ‘wrong’ information\n2. Lack of consideration of pseudoknots\n3. Dilution of individual probabilities for longer test sequences compared to the shorter train sequences\n\nMy compromise was to consider the global probability of pairing for each base. [This paper from the Das lab](https://www.nature.com/articles/s41592-022-01605-0) assessed how Pr(paired) as calculated by various RNA structure packages compared to chemical mapping assay results. They found CONTRAfold 2, ViennaRNA 2 at 60 degrees C, and RNAsoft with BLStar parameters to be the most effective approaches. Therefore, I generated Pr(paired) vectors for all sequences from these three methods as well as Eternafold to use as features.\n\nAs an aside, I used Arnie to run these packages and spent more time than I care to admit getting it setup for multiprocessing, although overall I think it ended up saving time. I did try to expand to additional methods, however, I pretty quickly abandoned it due to bugs / run time constraints.\n\nEarlier on in the competition, I did also try various structure features generated from ViennaRNA (MFE, centroid, ensemble). Compared to sequence alone, I was able to get improved performance, however, it added no benefit once the Pr(base paired) feature was included.\n\n## Architecture:\nMy approach ended up being a single transformer architecture separately trained to predict 2A3 or DMS.\n\n**Seq**\n* 4 token embedding -> 512 output dimension\n\n**Pr(base paired)**\n* Linear layer 4 input dimension -> 512 output dimension\n\n**Transformer**\n* 18 layers, 512 dimension, 8 heads\n* Postnorm\n* Dynamics position bias\n* Intra-attention Gating on Values\n* Input: Seq + Pr(base paired)\n\nThis was my first time working with a transformer architecture. Luckily, I found the library [x-transformers](https://github.com/lucidrains/x-transformers) by @lucidrains and used that to try out / learn about numerous variations of transformers.\n\n**What helped:**\n*Dynamic Position Bias*\nOne of my first goals was to understand how to generalize a transformer to longer sequences. Initially I found the ALiBi paper and thought that would be a useful approach. However, based on the x-transformers documentation it appeared as if dynamic position bias would be more likely to be effective in this context. I did not attempt any direct comparisons between ALiBi (or other relative position biases) and dynamic position bias since I did not trust performance differences on length extrapolation from the train set alone. Dynamic position bias did show the largest bump of any modification on the public vs. private sets (public - 0.15400 vs 0.15002; private - 0.18476 vs 0.14976). Note: this result was before the change to using Pr(base paired) data.\n\n*Intra-attention Gating on Values*\nThis is an attention variant from Alphafold2 that gates the aggregated values with the input. This was one of the feature variations available on x-transformers that appeared to have a small improvement on performance. Note: I added this one a while back and just kept it in, so it is entirely possible it adds nothing of value in the final model.\n\n*Postnorm*\nOne of the first changes I assessed with the transformer was to switch from the generally more preferred prenorm to the original postnorm approach. This resulted in improvements to convergence rate and performance. I rechecked this multiple times over the course of the competition as the architecture and input data approaches shift and consistently got the same type of result. I also attempted the various other norm approaches found in x-transformers, but none resulted in any noticeable improvement.\n\n*18 Layers*\n18 layers seemed to be the sweet spot of performance and training time. The main depths I tested were 12, 18, 24. 12 vs 18 showed a small performance improvement on the leader boards (public - 0.14383 vs 0.14330; private - 0.14595 vs 0.14517). 18 vs 24 I did not notice any difference in the CV score, so I did not submit any 24 layer models.\n\n*Sum embeds*\nMy approach to integrating the embedded sequence and the Pr(base paired) after passing through a linear layer (both of dimension 512) was to simply sum them elementwise. I tried other approaches, but they all appeared to have negative impacts to convergence and/or performance.\n\n## Training:\n\nI used the splits used in @iafoss starter transformer notebook and all of my experimental work was done using fold 0. Because I ended up using individual models for 2A3 and DMS prediction, the set of sequences satisfying SN >= 1.0 and reads >= 100 was higher for each group since there was a subset of sequences where only one of 2A3/DMS satisfied the constraint.\n\nAdditionally, I incorporated the react_error values as a means of weighting the loss so that I could use more of the lower quality sequences with greater confidence (luckily this happened right after the time the issue with react_error was identified and corrected). The react_error values were strictly greater than 0 and were usually fairly low (I believe the median on at least the high quality samples was ~0.125). I decided to try weighting the loss of each base using an exponential function:\n\nWeighting v1:\nexp(-react_error)\n\nWeighting v2:\nexp(-(react_error^2))\n\nThis would result in a weight value between 0 and 1 for each base in the sequence. While I don’t have a good mathematical rationale for this approach and I am sure there may be better options, it did give a noticeable boost to CV performance (~0.0005 - 0.001), including when using only the higher quality sequences.\n\nMy approach to training ended up being somewhat erratic for the final model.\n\n- 2A3 model\n - fit_one_cycle(epochs=256, lr=5e-4, pctstart=0.01)\n - SN >= 1.0, reads >= 100\n - exp(-(react_error^2)) weighting\n - Early stopping at epoch 45\n- 2A3 model part 2\n - Fine tune 2A3 model, lr=5e-5\n - SN >= 0.2, reads >= 75\n - exp(-react_error) weighting\n- DMS model\n - Fine tune 2A3 model part 2, lr=5e-5\n - SN >= 1.0, reads >= 100\n - exp(-(react_error^2)) weighting\n- DMS model part 2\n - Fine tune DMS model, lr=5e-5\n - SN >= 0.2, reads >= 75\n - exp(-react_error) weighting\n\nAs I mentioned above, I used fold 0 of my CV splits for all experimentation. Unfortunately, I lost time testing out other architectural approaches in the final 2 weeks and only settled on coming back to the split models due to the slight performance improvement within the last day, so I was unable to run the full set of CV folds. However, in a last minute effort to at least capture some of the fold 0 validation data in my model I ended up running 1 epoch on this data for my second final submission, which ended up giving me a nice little last minute boost on the leaderboard (public: 0.14330 vs 0.14199; private: 0.14517 vs 0.14458)." } }