{ "id": 460145, "title": "19th Solution", "url": "/competitions/stanford-ribonanza-rna-folding/discussion/460145", "author_name": "tereka", "post_date": "2023-12-08T02:31:59.917000", "votes": 17, "comment_count": 2, "views": 0, "content": "

Thank you for host! it's very interesting competition.
\nI share about my solution.

\n

TL;DR

\n
    \n
  1. Conv + Transformer + LSTM Architecture
    2. \n
  2. Pseudo Labeling
    3. \n
  3. Weighted Average
    4. \n
\n

important point is that I use test data while training process for robustness of Private data.

\n

Solution

\n

Architecture

\n

basic architecture

\n
    \n
  1. Conv + Transformer + LSTM
    2. \n
  2. Conv + LSTM + Graph Attetion
    3. \n
\n

Training Strategy

\n
    \n
  1. MLM
    2. \n
  2. Traning
    3. \n
  3. Pseudo Labeling
    4. \n
\n

MLM

\n

I use first pretreind Masked Language Model(Masked RNA Model?)
\nI delete some RNA sequence sometimes(0.3), and predict deleted sequence.

\n

Training

\n

Feature

\n
    \n
  1. sequence
    2. \n
  2. predicted_loop(eternafold/contrafold_2/vienna_2)
    3. \n
  3. structure(eternafold/contrafold_2/vienna_2)
    4. \n
\n

I use one hot encoding with 2/3 feature, also I express probability(eternafold: (, contrafold_2: (, vienna_2:) ->(:0.66, ):0.33)

\n

Training Strategy

\n

Training is very simple

\n
    \n
  1. use Filtered Dataset
    2. \n
  2. weighted loss(sequence)
    3. \n
  3. remove some sequence(structure/predicted_loop)
    4. \n
\n

Pseudo Labeling

\n

Private dataset is longer than Public dataset. I decide to get robustness from test data.
\nbut this competiton is regression, I cannot get confidence.
\nI use pretrain, then train only training dataset.

\n

I apply iterative that pseudo labeling.
\nalso few step later, I relabel noisy training data(=filter0), and train this process with pseudo labeling data.

\n

Public: 0.141->0.139->0.1384->0.1383

\n

Ensemble

\n

use weighted average
\nI check only public because I didn't believe cv score when I used pseudo labeling.

\n

Note

\n

I forgot exhanging old and new dataset….
\nit's very important…(Single model achieve Private 0.143..)

", "messages": [ { "id": 2553098, "postDate": "2023-12-08T02:31:59.917Z", "content": "

Thank you for host! it's very interesting competition.
\nI share about my solution.

\n

TL;DR

\n
    \n
  1. Conv + Transformer + LSTM Architecture
    2. \n
  2. Pseudo Labeling
    3. \n
  3. Weighted Average
    4. \n
\n

important point is that I use test data while training process for robustness of Private data.

\n

Solution

\n

Architecture

\n

basic architecture

\n
    \n
  1. Conv + Transformer + LSTM
    2. \n
  2. Conv + LSTM + Graph Attetion
    3. \n
\n

Training Strategy

\n
    \n
  1. MLM
    2. \n
  2. Traning
    3. \n
  3. Pseudo Labeling
    4. \n
\n

MLM

\n

I use first pretreind Masked Language Model(Masked RNA Model?)
\nI delete some RNA sequence sometimes(0.3), and predict deleted sequence.

\n

Training

\n

Feature

\n
    \n
  1. sequence
    2. \n
  2. predicted_loop(eternafold/contrafold_2/vienna_2)
    3. \n
  3. structure(eternafold/contrafold_2/vienna_2)
    4. \n
\n

I use one hot encoding with 2/3 feature, also I express probability(eternafold: (, contrafold_2: (, vienna_2:) ->(:0.66, ):0.33)

\n

Training Strategy

\n

Training is very simple

\n
    \n
  1. use Filtered Dataset
    2. \n
  2. weighted loss(sequence)
    3. \n
  3. remove some sequence(structure/predicted_loop)
    4. \n
\n

Pseudo Labeling

\n

Private dataset is longer than Public dataset. I decide to get robustness from test data.
\nbut this competiton is regression, I cannot get confidence.
\nI use pretrain, then train only training dataset.

\n

I apply iterative that pseudo labeling.
\nalso few step later, I relabel noisy training data(=filter0), and train this process with pseudo labeling data.

\n

Public: 0.141->0.139->0.1384->0.1383

\n

Ensemble

\n

use weighted average
\nI check only public because I didn't believe cv score when I used pseudo labeling.

\n

Note

\n

I forgot exhanging old and new dataset….
\nit's very important…(Single model achieve Private 0.143..)

", "rawMarkdown": "Thank you for host! it's very interesting competition.\nI share about my solution.\n \n# TL;DR\n1. Conv + Transformer + LSTM Architecture\n2. Pseudo Labeling\n3. Weighted Average\n\nimportant point is that I use test data while training process for robustness of Private data.\n\n# Solution\n## Architecture\nbasic architecture\n\n1. Conv + Transformer + LSTM\n2. Conv + LSTM + Graph Attetion\n\n## Training Strategy\n1. MLM\n2. Traning\n3. Pseudo Labeling\n\n### MLM\nI use first pretreind Masked Language Model(Masked RNA Model?)\nI delete some RNA sequence sometimes(0.3), and predict deleted sequence.\n\n### Training\n#### Feature\n1. sequence\n2. predicted_loop(eternafold/contrafold_2/vienna_2)\n3. structure(eternafold/contrafold_2/vienna_2)\n\nI use one hot encoding with 2/3 feature, also I express probability(eternafold: (, contrafold_2: (, vienna_2:) ->(:0.66, ):0.33)\n\n#### Training Strategy\nTraining is very simple\n1. use Filtered Dataset\n2. weighted loss(sequence)\n3. remove some sequence(structure/predicted_loop)\n\n### Pseudo Labeling\nPrivate dataset is longer than Public dataset. I decide to get robustness from test data.\nbut this competiton is regression, I cannot get confidence.\nI use pretrain, then train only training dataset.\n\nI apply iterative that pseudo labeling.\nalso few step later, I relabel noisy training data(=filter0), and train this process with pseudo labeling data.\n\nPublic: 0.141->0.139->0.1384->0.1383\n\n## Ensemble\nuse weighted average\nI check only public because I didn't believe cv score when I used pseudo labeling.\n\n## Note\nI forgot exhanging old and new dataset....\nit's very important...(Single model achieve Private 0.143..)", "votes": 17 }, { "id": 2553114, "postDate": "2023-12-08T02:56:34.770Z", "content": "

Single model tsuyo
\nAlthough some regret

", "rawMarkdown": "Single model tsuyo\nAlthough some regret" }, { "id": 2553239, "postDate": "2023-12-08T05:32:59.680Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 2553114, "author_name": "Horikita Saku", "author_url": "", "post_date": "2023-12-08T02:56:34.770000", "content": "

Single model tsuyo
\nAlthough some regret

", "votes": 0, "replies": [] }, { "id": 2553239, "author_name": "", "author_url": "", "post_date": "2023-12-08T05:32:59.680000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2553098": "Thank you for host! it's very interesting competition.\nI share about my solution.\n \n# TL;DR\n1. Conv + Transformer + LSTM Architecture\n2. Pseudo Labeling\n3. Weighted Average\n\nimportant point is that I use test data while training process for robustness of Private data.\n\n# Solution\n## Architecture\nbasic architecture\n\n1. Conv + Transformer + LSTM\n2. Conv + LSTM + Graph Attetion\n\n## Training Strategy\n1. MLM\n2. Traning\n3. Pseudo Labeling\n\n### MLM\nI use first pretreind Masked Language Model(Masked RNA Model?)\nI delete some RNA sequence sometimes(0.3), and predict deleted sequence.\n\n### Training\n#### Feature\n1. sequence\n2. predicted_loop(eternafold/contrafold_2/vienna_2)\n3. structure(eternafold/contrafold_2/vienna_2)\n\nI use one hot encoding with 2/3 feature, also I express probability(eternafold: (, contrafold_2: (, vienna_2:) ->(:0.66, ):0.33)\n\n#### Training Strategy\nTraining is very simple\n1. use Filtered Dataset\n2. weighted loss(sequence)\n3. remove some sequence(structure/predicted_loop)\n\n### Pseudo Labeling\nPrivate dataset is longer than Public dataset. I decide to get robustness from test data.\nbut this competiton is regression, I cannot get confidence.\nI use pretrain, then train only training dataset.\n\nI apply iterative that pseudo labeling.\nalso few step later, I relabel noisy training data(=filter0), and train this process with pseudo labeling data.\n\nPublic: 0.141->0.139->0.1384->0.1383\n\n## Ensemble\nuse weighted average\nI check only public because I didn't believe cv score when I used pseudo labeling.\n\n## Note\nI forgot exhanging old and new dataset....\nit's very important...(Single model achieve Private 0.143..)", "2553114": "Single model tsuyo\nAlthough some regret", "2553239": "" } }