![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2Feb2f9716c901ef7021c64e644a16d2a6%2Fesm2.png?generation=1702046716073336&alt=media\")
TLDR
\nWe entered the competition quite late, our first sub was from 10 days ago. Given the limited time, we decided to bet on a low development path and leverage the excellent ESM2 model family from META.
\nWe pretrained ESM2 backbones on competition data and RNA Central long sequences, then added custom folding heads and fine tuned on DMS and 2A3 labels.
\nWe honestly are a bit surprised by our end result, we expected worse when we started.
\nESM2 Pretraining
\nWe have some experience pretraining ESM2 protein language models, so we thought they would be a good fit for this competition. We pretrained three versions of ESM2 models (35M, 150M, 650M) on both train and test sequences using MLM (masked language modeling) loss.
\nIn particular, we used huggingface’s EsmForMaskedLM model class and DataCollatorForLanguageModeling(tokenizer, mlm=True) data collator which supports MLM loss.
Since there are no train sequences and only 8000 test sequences longer than 208, we downloaded RNA Central data and selected 12 million samples with length between 115 and 457 (the range in this competition data) to complement the competition data.
\nWe pretrained 5-20 epochs depending on how many RNA central data used, for a total of 25M to 70M total samples seen.
\n
Folding Head
\nWe adapted the model CPMP used in the Covid Vaccine competition three years ago to devise a folding head on top of ESM2 token embeddings. That model was presented here.
\nThe folding head takes as input both the ESM2 token embeddings and the bpp files. Its main component is an attention/convolution layer, repeated twice, then a linear classification head.
\nThe Attention/Convolution layers are similar to the transformer encoder structure: an attention layer followed by a convolution bloc with skip connection.
\nThe attention layer is a bpp attention layer followed by a bidirectional RNN (GRU or LSTM yields two model variants). Sure, GRU is not attention, but here it plays the same role as an attention layer where each node attends its two neighbors in the sequence.
\nThe bpp attention is a standard attention layer where attention weights are the bpp values.
\nThe convolution layer was designed after the Efficientnet convolution bloc. We then stack these as in this notebook.
We bet on this RNN/Convolution architecture rather than a transformer for two reasons:
\nFinetuning
\nOur best models are the ones with the custom folding head on top of the ESM2 backbone. We also trained simple models with just a linear head. The final ensemble is a mix of both, most being with a folding head.
\nWe used four variants of training data:
\nQuick start
\nQuick start + SN >= 0.3
\nQuick start + SN >= 0.3, and SN as sample weight
\nQuick start + SN >= 0.1, and SN as sample weight
\nSN is signal_to_noise clipped to (0, 1). We used it as sample noise in some variants. Model quality increased from top variant to bottom one.
We used a 5 fold CV and submitted the average of each fold model predictions.
\nScores
\nOur best single model is a 650M LSTM model with SN weight. CV = 0.12847, Public LB = 0.14308 (5 fold ensemble)
\nOur best ensemble is a 15 model ensemble (75 folds in total), CV = 0.12478, Public LB = 0.14174
What didn’t work
\nPretrained models from RNA-FM were not as good as our ESM2 pretrained models.
\nESM2 can also output contact predictions. To do so it stacks all attention activations, then it applies a regression head. We tried to use that to predict bpp values via an auxiliary loss. This improved the competition metric score a bit. But this comes at the expense of a 4x space and a 2x time increase, which prevented us from using it with 150M and 650M variants of ESM2.
\nBo & CPMP
", "messages": [ { "id": "2553840", "postDate": "12/08/2023 14:45:46", "content": "TLDR
\nWe entered the competition quite late, our first sub was from 10 days ago. Given the limited time, we decided to bet on a low development path and leverage the excellent ESM2 model family from META.
\nWe pretrained ESM2 backbones on competition data and RNA Central long sequences, then added custom folding heads and fine tuned on DMS and 2A3 labels.
\nWe honestly are a bit surprised by our end result, we expected worse when we started.
\nESM2 Pretraining
\nWe have some experience pretraining ESM2 protein language models, so we thought they would be a good fit for this competition. We pretrained three versions of ESM2 models (35M, 150M, 650M) on both train and test sequences using MLM (masked language modeling) loss.
\nIn particular, we used huggingface’s EsmForMaskedLM model class and DataCollatorForLanguageModeling(tokenizer, mlm=True) data collator which supports MLM loss.
Since there are no train sequences and only 8000 test sequences longer than 208, we downloaded RNA Central data and selected 12 million samples with length between 115 and 457 (the range in this competition data) to complement the competition data.
\nWe pretrained 5-20 epochs depending on how many RNA central data used, for a total of 25M to 70M total samples seen.
\n
Folding Head
\nWe adapted the model CPMP used in the Covid Vaccine competition three years ago to devise a folding head on top of ESM2 token embeddings. That model was presented here.
\nThe folding head takes as input both the ESM2 token embeddings and the bpp files. Its main component is an attention/convolution layer, repeated twice, then a linear classification head.
\nThe Attention/Convolution layers are similar to the transformer encoder structure: an attention layer followed by a convolution bloc with skip connection.
\nThe attention layer is a bpp attention layer followed by a bidirectional RNN (GRU or LSTM yields two model variants). Sure, GRU is not attention, but here it plays the same role as an attention layer where each node attends its two neighbors in the sequence.
\nThe bpp attention is a standard attention layer where attention weights are the bpp values.
\nThe convolution layer was designed after the Efficientnet convolution bloc. We then stack these as in this notebook.
We bet on this RNN/Convolution architecture rather than a transformer for two reasons:
\nFinetuning
\nOur best models are the ones with the custom folding head on top of the ESM2 backbone. We also trained simple models with just a linear head. The final ensemble is a mix of both, most being with a folding head.
\nWe used four variants of training data:
\nQuick start
\nQuick start + SN >= 0.3
\nQuick start + SN >= 0.3, and SN as sample weight
\nQuick start + SN >= 0.1, and SN as sample weight
\nSN is signal_to_noise clipped to (0, 1). We used it as sample noise in some variants. Model quality increased from top variant to bottom one.
We used a 5 fold CV and submitted the average of each fold model predictions.
\nScores
\nOur best single model is a 650M LSTM model with SN weight. CV = 0.12847, Public LB = 0.14308 (5 fold ensemble)
\nOur best ensemble is a 15 model ensemble (75 folds in total), CV = 0.12478, Public LB = 0.14174
What didn’t work
\nPretrained models from RNA-FM were not as good as our ESM2 pretrained models.
\nESM2 can also output contact predictions. To do so it stacks all attention activations, then it applies a regression head. We tried to use that to predict bpp values via an auxiliary loss. This improved the competition metric score a bit. But this comes at the expense of a 4x space and a 2x time increase, which prevented us from using it with 150M and 650M variants of ESM2.
\nBo & CPMP
", "rawMarkdown": "**TLDR**\n\nWe entered the competition quite late, our first sub was from 10 days ago. Given the limited time, we decided to bet on a low development path and leverage the excellent ESM2 model family from META.\n\nWe pretrained ESM2 backbones on competition data and RNA Central long sequences, then added custom folding heads and fine tuned on DMS and 2A3 labels.\n\nWe honestly are a bit surprised by our end result, we expected worse when we started. \n\n\n**ESM2 Pretraining**\n\nWe have some experience pretraining [ESM2](https://github.com/facebookresearch/esm) protein language models, so we thought they would be a good fit for this competition. We pretrained three versions of ESM2 models (35M, 150M, 650M) on both train and test sequences using MLM (masked language modeling) loss. \n\nIn particular, we used huggingface’s [`EsmForMaskedLM`](https://huggingface.co/docs/transformers/model_doc/esm#transformers.EsmForMaskedLM) model class and [`DataCollatorForLanguageModeling(tokenizer, mlm=True)` ](https://huggingface.co/docs/transformers/main/main_classes/data_collator#transformers.DataCollatorForLanguageModeling) data collator which supports MLM loss.\n\nSince there are no train sequences and only 8000 test sequences longer than 208, we downloaded [RNA Central ](https://rnacentral.org/) data and selected 12 million samples with length between 115 and 457 (the range in this competition data) to complement the competition data.\n\nWe pretrained 5-20 epochs depending on how many RNA central data used, for a total of 25M to 70M total samples seen.\n\n\n\n**Folding Head**\n\nWe adapted the model CPMP used in the Covid Vaccine competition three years ago to devise a folding head on top of ESM2 token embeddings. That model was presented [here](https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189723).\n\nThe folding head takes as input both the ESM2 token embeddings and the bpp files. Its main component is an attention/convolution layer, repeated twice, then a linear classification head. \n\nThe Attention/Convolution layers are similar to the transformer encoder structure: an attention layer followed by a convolution bloc with skip connection.\nThe attention layer is a bpp attention layer followed by a bidirectional RNN (GRU or LSTM yields two model variants). Sure, GRU is not attention, but here it plays the same role as an attention layer where each node attends its two neighbors in the sequence.\nThe bpp attention is a standard attention layer where attention weights are the bpp values.\nThe convolution layer was designed after the Efficientnet convolution bloc. We then stack these as in this [notebook](https://www.kaggle.com/thebigd8ta/open-vaccine-pytorch-v).\n\nWe bet on this RNN/Convolution architecture rather than a transformer for two reasons:\n- We literally started working on it 5 days before competition end, and we did not have time to tune a new architecture\n- we hoped that ESM2 attention would have already learned what a transformer head attention would learn.\n\n**Finetuning**\n\nOur best models are the ones with the custom folding head on top of the ESM2 backbone. We also trained simple models with just a linear head. The final ensemble is a mix of both, most being with a folding head.\n\nWe used four variants of training data:\nQuick start\nQuick start + SN >= 0.3\nQuick start + SN >= 0.3, and SN as sample weight\nQuick start + SN >= 0.1, and SN as sample weight\nSN is signal_to_noise clipped to (0, 1). We used it as sample noise in some variants. Model quality increased from top variant to bottom one.\n\nWe used a 5 fold CV and submitted the average of each fold model predictions.\n\n**Scores**\n\nOur best single model is a 650M LSTM model with SN weight. CV = 0.12847, Public LB = 0.14308 (5 fold ensemble)\nOur best ensemble is a 15 model ensemble (75 folds in total), CV = 0.12478, Public LB = 0.14174\n\n**What didn’t work**\n\nPretrained models from [RNA-FM](https://github.com/ml4bio/RNA-FM) were not as good as our ESM2 pretrained models.\n\nESM2 can also output contact predictions. To do so it stacks all attention activations, then it applies a regression head. We tried to use that to predict bpp values via an auxiliary loss. This improved the competition metric score a bit. But this comes at the expense of a 4x space and a 2x time increase, which prevented us from using it with 150M and 650M variants of ESM2.\n\nBo & CPMP", "votes": null }, { "id": "2554204", "postDate": "12/08/2023 22:29:07", "content": "Congrats! I also joined late and tried similar ideas, but wasn't so successful, lots of room to learn :) Do you know how much value you get from pre-training ESM with MLM? I also saw poor performance of RNA-FM and concluded MLM might not be helpful here, because we're dealing with a lot of synthetic sequences, as opposed to ESM that is trained on evolutionary designed proteins.
", "rawMarkdown": "Congrats! I also joined late and tried similar ideas, but wasn't so successful, lots of room to learn :) Do you know how much value you get from pre-training ESM with MLM? I also saw poor performance of RNA-FM and concluded MLM might not be helpful here, because we're dealing with a lot of synthetic sequences, as opposed to ESM that is trained on evolutionary designed proteins.", "votes": null }, { "id": "2554223", "postDate": "12/08/2023 23:36:29", "content": "Pretraining was key. We saw a performance improvement with more pretraining. I'm sorry I can't say more because I have not tried without a pretrained ESM2.
\nYour score is not that bad!
", "rawMarkdown": "Pretraining was key. We saw a performance improvement with more pretraining. I'm sorry I can't say more because I have not tried without a pretrained ESM2.\n\nYour score is not that bad!", "votes": null }, { "id": "2554905", "postDate": "12/09/2023 14:05:20", "content": "Did you convert the RNA to proteins/AA?
", "rawMarkdown": "Did you convert the RNA to proteins/AA?", "votes": null }, { "id": "2554955", "postDate": "12/09/2023 14:57:50", "content": "no, we replaced the vocab.txt files used by the ESM tokenizer. We used this as rna_vocab.txt:
<cls>\n<pad>\n<eos>\n<unk>\nA\nC\nG\nU\n<null_1>\n<mask>\nThen create the tokenizer with:
\ntokenizer = EsmTokenizer('../rna_vocab.txt', model_max_length=context_length)
@cpmpml Did you try not using bpp? Curious how that would work
", "rawMarkdown": "cpmpml Did you try not using bpp? Curious how that would work", "votes": null }, { "id": "2555708", "postDate": "12/10/2023 05:37:46", "content": "Yes, we have submitted ESM2 alone. Best one has 0.14781 private / 0.14359 public. But it looks like an outlier, others have much worse scores.
\n@boliu0 did the ESM2 part, maybe he can say more.
", "rawMarkdown": "Yes, we have submitted ESM2 alone. Best one has 0.14781 private / 0.14359 public. But it looks like an outlier, others have much worse scores. \n\n@boliu0 did the ESM2 part, maybe he can say more.", "votes": null }, { "id": "2556239", "postDate": "12/10/2023 15:01:31", "content": "\n\nDo you know how much value you get from pre-training ESM with MLM?
\n
A lot. In my very first baseline, I compared non-pretrained ESM 35M vs a pretrained ESM 35M (pre-trained on competition train set only, so it’s not the best pretrained version) on the downstream supervised task, the MAE is not close. 0.01-0.02 gap if I remember correctly. So I never tried non-pretrained ESM anymore.
", "rawMarkdown": "> Do you know how much value you get from pre-training ESM with MLM?\n\nA lot. In my very first baseline, I compared non-pretrained ESM 35M vs a pretrained ESM 35M (pre-trained on competition train set only, so it’s not the best pretrained version) on the downstream supervised task, the MAE is not close. 0.01-0.02 gap if I remember correctly. So I never tried non-pretrained ESM anymore.", "votes": null }, { "id": "2556245", "postDate": "12/10/2023 15:07:19", "content": "The one with 0.14781 private / 0.14359 public is not ESM2 alone actually. My incorrect naming of that sub confused you.
\nBest ESM alone models:
\nESM 35M: public 0.15095, private 0.16162
\nESM 150M: public 0.15110, private 0.15836
\nESM 650M: public 0.14789, private 0.15331
They are obviously not as good as bpp models, but they add diversity and boost cv scores, so we included them in the final ensemble.
", "rawMarkdown": "The one with 0.14781 private / 0.14359 public is not ESM2 alone actually. My incorrect naming of that sub confused you. \n\nBest ESM alone models:\nESM 35M: public 0.15095, private 0.16162\nESM 150M: public 0.15110, private 0.15836\nESM 650M: public 0.14789, private 0.15331\n\nThey are obviously not as good as bpp models, but they add diversity and boost cv scores, so we included them in the final ensemble.", "votes": null }, { "id": "2559478", "postDate": "12/12/2023 21:10:30", "content": "Wow, 21st in just 10 days is insane! Congratulations to you and your team!
", "rawMarkdown": "Wow, 21st in just 10 days is insane! Congratulations to you and your team!", "votes": null } ], "comments": [ { "id": 2554204, "author_name": "thedrcat", "author_url": "", "post_date": "12/08/2023 22:29:07", "content": "Congrats! I also joined late and tried similar ideas, but wasn't so successful, lots of room to learn :) Do you know how much value you get from pre-training ESM with MLM? I also saw poor performance of RNA-FM and concluded MLM might not be helpful here, because we're dealing with a lot of synthetic sequences, as opposed to ESM that is trained on evolutionary designed proteins.
", "votes": null, "replies": [ { "id": 2554223, "author_name": "cpmpml", "author_url": "", "post_date": "12/08/2023 23:36:29", "content": "Pretraining was key. We saw a performance improvement with more pretraining. I'm sorry I can't say more because I have not tried without a pretrained ESM2.
\nYour score is not that bad!
", "votes": null, "replies": [ { "id": 2556239, "author_name": "boliu0", "author_url": "", "post_date": "12/10/2023 15:01:31", "content": "\n\nDo you know how much value you get from pre-training ESM with MLM?
\n
A lot. In my very first baseline, I compared non-pretrained ESM 35M vs a pretrained ESM 35M (pre-trained on competition train set only, so it’s not the best pretrained version) on the downstream supervised task, the MAE is not close. 0.01-0.02 gap if I remember correctly. So I never tried non-pretrained ESM anymore.
", "votes": null, "replies": [] } ] } ] }, { "id": 2554905, "author_name": "danofer", "author_url": "", "post_date": "12/09/2023 14:05:20", "content": "Did you convert the RNA to proteins/AA?
", "votes": null, "replies": [ { "id": 2554955, "author_name": "cpmpml", "author_url": "", "post_date": "12/09/2023 14:57:50", "content": "no, we replaced the vocab.txt files used by the ESM tokenizer. We used this as rna_vocab.txt:
<cls>\n<pad>\n<eos>\n<unk>\nA\nC\nG\nU\n<null_1>\n<mask>\nThen create the tokenizer with:
\ntokenizer = EsmTokenizer('../rna_vocab.txt', model_max_length=context_length)
@cpmpml Did you try not using bpp? Curious how that would work
", "votes": null, "replies": [ { "id": 2555708, "author_name": "cpmpml", "author_url": "", "post_date": "12/10/2023 05:37:46", "content": "Yes, we have submitted ESM2 alone. Best one has 0.14781 private / 0.14359 public. But it looks like an outlier, others have much worse scores.
\n@boliu0 did the ESM2 part, maybe he can say more.
", "votes": null, "replies": [ { "id": 2556245, "author_name": "boliu0", "author_url": "", "post_date": "12/10/2023 15:07:19", "content": "The one with 0.14781 private / 0.14359 public is not ESM2 alone actually. My incorrect naming of that sub confused you.
\nBest ESM alone models:
\nESM 35M: public 0.15095, private 0.16162
\nESM 150M: public 0.15110, private 0.15836
\nESM 650M: public 0.14789, private 0.15331
They are obviously not as good as bpp models, but they add diversity and boost cv scores, so we included them in the final ensemble.
", "votes": null, "replies": [] } ] } ] }, { "id": 2559478, "author_name": "alberteinsten", "author_url": "", "post_date": "12/12/2023 21:10:30", "content": "Wow, 21st in just 10 days is insane! Congratulations to you and your team!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2553840": "**TLDR**\n\nWe entered the competition quite late, our first sub was from 10 days ago. Given the limited time, we decided to bet on a low development path and leverage the excellent ESM2 model family from META.\n\nWe pretrained ESM2 backbones on competition data and RNA Central long sequences, then added custom folding heads and fine tuned on DMS and 2A3 labels.\n\nWe honestly are a bit surprised by our end result, we expected worse when we started. \n\n\n**ESM2 Pretraining**\n\nWe have some experience pretraining [ESM2](https://github.com/facebookresearch/esm) protein language models, so we thought they would be a good fit for this competition. We pretrained three versions of ESM2 models (35M, 150M, 650M) on both train and test sequences using MLM (masked language modeling) loss. \n\nIn particular, we used huggingface’s [`EsmForMaskedLM`](https://huggingface.co/docs/transformers/model_doc/esm#transformers.EsmForMaskedLM) model class and [`DataCollatorForLanguageModeling(tokenizer, mlm=True)` ](https://huggingface.co/docs/transformers/main/main_classes/data_collator#transformers.DataCollatorForLanguageModeling) data collator which supports MLM loss.\n\nSince there are no train sequences and only 8000 test sequences longer than 208, we downloaded [RNA Central ](https://rnacentral.org/) data and selected 12 million samples with length between 115 and 457 (the range in this competition data) to complement the competition data.\n\nWe pretrained 5-20 epochs depending on how many RNA central data used, for a total of 25M to 70M total samples seen.\n\n\n\n**Folding Head**\n\nWe adapted the model CPMP used in the Covid Vaccine competition three years ago to devise a folding head on top of ESM2 token embeddings. That model was presented [here](https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189723).\n\nThe folding head takes as input both the ESM2 token embeddings and the bpp files. Its main component is an attention/convolution layer, repeated twice, then a linear classification head. \n\nThe Attention/Convolution layers are similar to the transformer encoder structure: an attention layer followed by a convolution bloc with skip connection.\nThe attention layer is a bpp attention layer followed by a bidirectional RNN (GRU or LSTM yields two model variants). Sure, GRU is not attention, but here it plays the same role as an attention layer where each node attends its two neighbors in the sequence.\nThe bpp attention is a standard attention layer where attention weights are the bpp values.\nThe convolution layer was designed after the Efficientnet convolution bloc. We then stack these as in this [notebook](https://www.kaggle.com/thebigd8ta/open-vaccine-pytorch-v).\n\nWe bet on this RNN/Convolution architecture rather than a transformer for two reasons:\n- We literally started working on it 5 days before competition end, and we did not have time to tune a new architecture\n- we hoped that ESM2 attention would have already learned what a transformer head attention would learn.\n\n**Finetuning**\n\nOur best models are the ones with the custom folding head on top of the ESM2 backbone. We also trained simple models with just a linear head. The final ensemble is a mix of both, most being with a folding head.\n\nWe used four variants of training data:\nQuick start\nQuick start + SN >= 0.3\nQuick start + SN >= 0.3, and SN as sample weight\nQuick start + SN >= 0.1, and SN as sample weight\nSN is signal_to_noise clipped to (0, 1). We used it as sample noise in some variants. Model quality increased from top variant to bottom one.\n\nWe used a 5 fold CV and submitted the average of each fold model predictions.\n\n**Scores**\n\nOur best single model is a 650M LSTM model with SN weight. CV = 0.12847, Public LB = 0.14308 (5 fold ensemble)\nOur best ensemble is a 15 model ensemble (75 folds in total), CV = 0.12478, Public LB = 0.14174\n\n**What didn’t work**\n\nPretrained models from [RNA-FM](https://github.com/ml4bio/RNA-FM) were not as good as our ESM2 pretrained models.\n\nESM2 can also output contact predictions. To do so it stacks all attention activations, then it applies a regression head. We tried to use that to predict bpp values via an auxiliary loss. This improved the competition metric score a bit. But this comes at the expense of a 4x space and a 2x time increase, which prevented us from using it with 150M and 650M variants of ESM2.\n\nBo & CPMP", "2554204": "Congrats! I also joined late and tried similar ideas, but wasn't so successful, lots of room to learn :) Do you know how much value you get from pre-training ESM with MLM? I also saw poor performance of RNA-FM and concluded MLM might not be helpful here, because we're dealing with a lot of synthetic sequences, as opposed to ESM that is trained on evolutionary designed proteins.", "2554223": "Pretraining was key. We saw a performance improvement with more pretraining. I'm sorry I can't say more because I have not tried without a pretrained ESM2.\n\nYour score is not that bad!", "2554905": "Did you convert the RNA to proteins/AA?", "2554955": "no, we replaced the vocab.txt files used by the ESM tokenizer. We used this as `rna_vocab.txt`:\n\n```python\n