{ "id": 460277, "title": "81st Place Solution for the Stanford Ribonanza RNA Folding Competition", "url": "/competitions/stanford-ribonanza-rna-folding/discussion/460277", "author_name": "neilus", "post_date": "2023-12-08T14:12:44.919000", "votes": 4, "comment_count": 0, "views": 0, "content": "

Thanks to the organizers and Kaggle staff for holding the competition, and congratulations to the winners!

\n

Context

\n\n

Overview of the Approach

\n

I referenced the model used in the ASL Fingerspelling Competition 2nd place solution.
\nI employed a model consisting of Conv1D (x3) + Transformer.

\n

Details of the Submission

\n

Model

\n

I initially used a model with 12 Transformer layers, based on the excellent baseline provided by Iafoss.
\nHowever, due to stagnation in LB performance, I investigated other models.
\nInfluenced by the model used by the runner-up in a ASL Fingerspelling Competition, which incorporated Conv1D,
\nI saw improved performance and thus switched to the following structure:

\n

pseudo model code:

\n
 _  range(5):\n     _  range(3):\n        x = Conv1DBlock(=192, =17, =4, =0.2)(x)\n    x = TransformerBlock(=192, =2, =4, =0.2, =0.2)(x)\n
\n

Dataset

\n

Modifications applied to train_data.csv included:

\n\n

This approach was inspired by the Stanford COVID Vaccine Competition winner's solution

\n

My Mistake

\n

For the final submission, I used an ensemble approach different from my main score.
\nWhile it improved my Public LB score to 0.14381, it detrimentally lowered my Private LB score to 0.18093.

\n

The mistake was including a model with an extremely poor Private LB score of 0.32353 in the ensemble.
\n(comparable to the performance of sample_submission.csv at Private 0.33290)

\n

\"\"

\n

This flawed model was created by merely changing the seed during training, with almost no predictive performance for lengths not present in the private dataset (207, 357, 457).
\nThe reason is unclear, but I suspect overfitting to a length of 177. Attempting to address potential leakage in the public LB by setting the MaP of leaked sequences to 0 made no difference.

\n

I regret not validating with test data lengths like 206, which could have potentially averted this issue.

\n

Source

\n", "messages": [ { "id": 2553793, "postDate": "2023-12-08T14:12:44.920Z", "content": "

Thanks to the organizers and Kaggle staff for holding the competition, and congratulations to the winners!

\n

Context

\n\n

Overview of the Approach

\n

I referenced the model used in the ASL Fingerspelling Competition 2nd place solution.
\nI employed a model consisting of Conv1D (x3) + Transformer.

\n

Details of the Submission

\n

Model

\n

I initially used a model with 12 Transformer layers, based on the excellent baseline provided by Iafoss.
\nHowever, due to stagnation in LB performance, I investigated other models.
\nInfluenced by the model used by the runner-up in a ASL Fingerspelling Competition, which incorporated Conv1D,
\nI saw improved performance and thus switched to the following structure:

\n

pseudo model code:

\n
 _  range(5):\n     _  range(3):\n        x = Conv1DBlock(=192, =17, =4, =0.2)(x)\n    x = TransformerBlock(=192, =2, =4, =0.2, =0.2)(x)\n
\n

Dataset

\n

Modifications applied to train_data.csv included:

\n\n

This approach was inspired by the Stanford COVID Vaccine Competition winner's solution

\n

My Mistake

\n

For the final submission, I used an ensemble approach different from my main score.
\nWhile it improved my Public LB score to 0.14381, it detrimentally lowered my Private LB score to 0.18093.

\n

The mistake was including a model with an extremely poor Private LB score of 0.32353 in the ensemble.
\n(comparable to the performance of sample_submission.csv at Private 0.33290)

\n

\"\"

\n

This flawed model was created by merely changing the seed during training, with almost no predictive performance for lengths not present in the private dataset (207, 357, 457).
\nThe reason is unclear, but I suspect overfitting to a length of 177. Attempting to address potential leakage in the public LB by setting the MaP of leaked sequences to 0 made no difference.

\n

I regret not validating with test data lengths like 206, which could have potentially averted this issue.

\n

Source

\n", "rawMarkdown": "Thanks to the organizers and Kaggle staff for holding the competition, and congratulations to the winners!\n\n# Context\n- Business context: https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/overview\n- Data context: https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/data\n\n# Overview of the Approach\nI referenced the model used in the [ASL Fingerspelling Competition 2nd place solution](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434588).\nI employed a model consisting of Conv1D (x3) + Transformer.\n\n# Details of the Submission\n\n## Model\nI initially used a model with 12 Transformer layers, based on the [excellent baseline provided by Iafoss](https://www.kaggle.com/code/iafoss/rna-starter-0-186-lb).\nHowever, due to stagnation in LB performance, I investigated other models. \nInfluenced by the model used by the runner-up in a ASL Fingerspelling Competition, which incorporated Conv1D,\nI saw improved performance and thus switched to the following structure:\n\npseudo model code:\n\n```\nfor _ in range(5):\n for _ in range(3):\n x = Conv1DBlock(channel_size=192, kernel_size=17, expand_ratio=4, drop_rate=0.2)(x)\n x = TransformerBlock(channel_size=192, expand=2, num_heads=4, drop_rate=0.2, attn_dropout=0.2)(x)\n```\n\n\n\n## Dataset\nModifications applied to `train_data.csv` included:\n\n- Filtering for reads >= 100.\n- Adjusting for large errors in reactivity and reactivity_error by setting them to NaN, excluding them from loss calculations.\nThis involved computing signal-to-noise for each reactivity value and replacing reactivity with NaN under the following conditions:\n - reactivity_error > 1.5\n - reactivity / reactivity_error > 1.5\n\nThis approach was inspired by the [Stanford COVID Vaccine Competition winner's solution](https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189620)\n\n## My Mistake\nFor the final submission, I used an ensemble approach different from my main score.\nWhile it improved my Public LB score to 0.14381, it detrimentally lowered my Private LB score to 0.18093.\n\nThe mistake was including a model with an extremely poor Private LB score of 0.32353 in the ensemble.\n(comparable to the performance of sample_submission.csv at Private 0.33290)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8577629%2F2ba2f5941a271b75c9111a6675669677%2F.PNG?generation=1702044697974444&alt=media)\n\nThis flawed model was created by merely changing the seed during training, with almost no predictive performance for lengths not present in the private dataset (207, 357, 457). \nThe reason is unclear, but I suspect overfitting to a length of 177. Attempting to address potential leakage in the public LB by setting the MaP of leaked sequences to 0 made no difference.\n\nI regret not validating with test data lengths like 206, which could have potentially averted this issue.\n\n## Source\n- https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434588\n- https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189620\n- https://www.kaggle.com/code/iafoss/rna-starter-0-186-lb", "votes": 4 } ], "comments": [], "raw_markdown_by_id": { "2553793": "Thanks to the organizers and Kaggle staff for holding the competition, and congratulations to the winners!\n\n# Context\n- Business context: https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/overview\n- Data context: https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/data\n\n# Overview of the Approach\nI referenced the model used in the [ASL Fingerspelling Competition 2nd place solution](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434588).\nI employed a model consisting of Conv1D (x3) + Transformer.\n\n# Details of the Submission\n\n## Model\nI initially used a model with 12 Transformer layers, based on the [excellent baseline provided by Iafoss](https://www.kaggle.com/code/iafoss/rna-starter-0-186-lb).\nHowever, due to stagnation in LB performance, I investigated other models. \nInfluenced by the model used by the runner-up in a ASL Fingerspelling Competition, which incorporated Conv1D,\nI saw improved performance and thus switched to the following structure:\n\npseudo model code:\n\n```\nfor _ in range(5):\n for _ in range(3):\n x = Conv1DBlock(channel_size=192, kernel_size=17, expand_ratio=4, drop_rate=0.2)(x)\n x = TransformerBlock(channel_size=192, expand=2, num_heads=4, drop_rate=0.2, attn_dropout=0.2)(x)\n```\n\n\n\n## Dataset\nModifications applied to `train_data.csv` included:\n\n- Filtering for reads >= 100.\n- Adjusting for large errors in reactivity and reactivity_error by setting them to NaN, excluding them from loss calculations.\nThis involved computing signal-to-noise for each reactivity value and replacing reactivity with NaN under the following conditions:\n - reactivity_error > 1.5\n - reactivity / reactivity_error > 1.5\n\nThis approach was inspired by the [Stanford COVID Vaccine Competition winner's solution](https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189620)\n\n## My Mistake\nFor the final submission, I used an ensemble approach different from my main score.\nWhile it improved my Public LB score to 0.14381, it detrimentally lowered my Private LB score to 0.18093.\n\nThe mistake was including a model with an extremely poor Private LB score of 0.32353 in the ensemble.\n(comparable to the performance of sample_submission.csv at Private 0.33290)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8577629%2F2ba2f5941a271b75c9111a6675669677%2F.PNG?generation=1702044697974444&alt=media)\n\nThis flawed model was created by merely changing the seed during training, with almost no predictive performance for lengths not present in the private dataset (207, 357, 457). \nThe reason is unclear, but I suspect overfitting to a length of 177. Attempting to address potential leakage in the public LB by setting the MaP of leaked sequences to 0 made no difference.\n\nI regret not validating with test data lengths like 206, which could have potentially averted this issue.\n\n## Source\n- https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434588\n- https://www.kaggle.com/competitions/stanford-covid-vaccine/discussion/189620\n- https://www.kaggle.com/code/iafoss/rna-starter-0-186-lb" } }