![\"adv](\"https://ucaa060899a3c8e0931bf9d91ade.previews.dropboxusercontent.com/p/thumb/ABLhfY6YZmPuwcl_I1J-i3lYgYixcNXnWlOyDMBIigA4YS88S8InmWe-QhG8Y2MoClDpnXxH96ia6Cy38-rarws-WL8pB59Q_DYay85J19n8R-jp-ucGx5cnkI4rMpbyGcmSO3aQGaPHoxMrdtEQuDchv_nJO8MoZRDJYXADTLCbDcptwZ806rIePExIe6-klx1th-v3kSfz267TdsYhG-YeJziUYy8FzTXr9xVn7gvaEDu40-REAbdxA_CqnyulEUq-8OS8kzBXv3rUs3kFqP9exr3AZ_0LRmFrJsN0Adbj4DJnBm_WUuZPWoXITU2OoG4E7sh1fekri7_ei0DSctsjzIuC_pJkOL7XouqLmmUd2NtWBflkqG5B6JLwK12yGzthW35ZOKL8R2hGZMJwezHG/p.png?fv_content=true&size_mode=5\")
First of all, I would like to say thanks to Kaggle and organizers hosts this exciting competition. Congrats winners who won this tough competition!
\nI have tried adversarial validation in early stage. Here is the result.
\n
This figure by itself tells how this test set was made. Test set can be split into two subsets. I call them as …
\nThen, I was pretty sure that pseudo labeling will work well for test set. I also thought that it's necessary to create models which fits both subset well. It's possible to estimate how much current model fits for each samples, if erase all half predictions.
\nI used 3 models, Deit tiny, small and base.
\nMy model mainly predicts smiles instead of inchi. smiles is more compact than inchi. It helps reducing training time. smiles is converted inchi before submission. I also created a model to predict inchi, but it's for fallback when smiles is invalid. As a summary, I predicted inchi from 3 ways. The priority order is followings.
\nTraining steps consist from 3 steps.
\nAt step1, I train transformer to predict canonicalized inchi and smiles from randomized smiles. Next, self attention and heads on decoder were transferred and combined with Deit. So, only cross attention on decoder is trained from scratch. As I don't have so many GPUs, 224 images are used at this step.
\nFinally, I have trained 6 models at step 3.
\nPosition embedding was expanded for image size 384 by interpolation at step 3.
\nI used TTA (rot90) and logits averaging per token.
\nFor Test subset A, I used averaging weight 9:9:9:1:1:1 for models a1, a2, a3, b1, b2 and b3. On the other hand, for Test subset B, I used weight 1:1:1:9:9:9. It ends up with 0.64 on LB.
", "messages": [ { "id": "1336017", "postDate": "06/04/2021 15:30:40", "content": "First of all, I would like to say thanks to Kaggle and organizers hosts this exciting competition. Congrats winners who won this tough competition!
\nI have tried adversarial validation in early stage. Here is the result.
\n
This figure by itself tells how this test set was made. Test set can be split into two subsets. I call them as …
\nThen, I was pretty sure that pseudo labeling will work well for test set. I also thought that it's necessary to create models which fits both subset well. It's possible to estimate how much current model fits for each samples, if erase all half predictions.
\nI used 3 models, Deit tiny, small and base.
\nMy model mainly predicts smiles instead of inchi. smiles is more compact than inchi. It helps reducing training time. smiles is converted inchi before submission. I also created a model to predict inchi, but it's for fallback when smiles is invalid. As a summary, I predicted inchi from 3 ways. The priority order is followings.
\nTraining steps consist from 3 steps.
\nAt step1, I train transformer to predict canonicalized inchi and smiles from randomized smiles. Next, self attention and heads on decoder were transferred and combined with Deit. So, only cross attention on decoder is trained from scratch. As I don't have so many GPUs, 224 images are used at this step.
\nFinally, I have trained 6 models at step 3.
\nPosition embedding was expanded for image size 384 by interpolation at step 3.
\nI used TTA (rot90) and logits averaging per token.
\nFor Test subset A, I used averaging weight 9:9:9:1:1:1 for models a1, a2, a3, b1, b2 and b3. On the other hand, for Test subset B, I used weight 1:1:1:9:9:9. It ends up with 0.64 on LB.
", "rawMarkdown": "First of all, I would like to say thanks to Kaggle and organizers hosts this exciting competition. Congrats winners who won this tough competition! \n\n## Findings\n\nI have tried adversarial validation in early stage. Here is the result.\n\n\n\nThis figure by itself tells how this test set was made. Test set can be split into two subsets. I call them as ...\n\n* Test subset A (left samples on figure which is not similar with train set)\n* Test subset B (right samples on figure which is similar with train set)\n\nThen, I was pretty sure that pseudo labeling will work well for test set. I also thought that it's necessary to create models which fits both subset well. It's possible to estimate how much current model fits for each samples, if erase all half predictions.\n\n\n## Training and modeling\n\nI used 3 models, Deit tiny, small and base.\n\nMy model mainly predicts smiles instead of inchi. smiles is more compact than inchi. It helps reducing training time. smiles is converted inchi before submission. I also created a model to predict inchi, but it's for fallback when smiles is invalid. As a summary, I predicted inchi from 3 ways. The priority order is followings.\n\n* norm inchi from predicted smiles\n* norm inchi from predicted inchi\n* raw predicted inchi\n\nTraining steps consist from 3 steps.\n\n1. Pre-train only decoder by using randomized smiles canonicalization.\n2. Combine Deit with the decoder trained on step 1 and fine tune by extra synthesized images (size 224).\n3. Fine tune with train set (only hard samples) and pseudo labels with image size=384.\n\nAt step1, I train transformer to predict canonicalized inchi and smiles from randomized smiles. Next, self attention and heads on decoder were transferred and combined with Deit. So, only cross attention on decoder is trained from scratch. As I don't have so many GPUs, 224 images are used at this step.\n\nFinally, I have trained 6 models at step 3.\n\n* a1) Deit tiny trained by pseudo labels from Test subset A + train set (only hard samples).\n* a2) Deit small trained by pseudo labels from Test subset A + train set (only hard samples).\n* a3) Deit base trained by pseudo labels from Test subset A + train set (only hard samples).\n* b1) Deit tiny trained by pseudo labels from Test subset B + train set (only hard samples).\n* b2) Deit small trained by pseudo labels from Test subset B + train set (only hard samples).\n* b3) Deit base trained by pseudo labels from Test subset B + train set (only hard samples).\n\nPosition embedding was expanded for image size 384 by interpolation at step 3.\n\n## Inference\n\nI used TTA (rot90) and logits averaging per token.\n\nFor Test subset A, I used averaging weight 9:9:9:1:1:1 for models a1, a2, a3, b1, b2 and b3. On the other hand, for Test subset B, I used weight 1:1:1:9:9:9. It ends up with 0.64 on LB.", "votes": null }, { "id": "1336051", "postDate": "06/04/2021 15:51:35", "content": "Great work! LB 0.65 as a solo competitor is amazing.
\nLet's fight together next time :)
Deit. learned new staff.
", "rawMarkdown": "Deit. learned new staff.", "votes": null } ], "comments": [ { "id": 1336051, "author_name": "kfujikawa", "author_url": "", "post_date": "06/04/2021 15:51:35", "content": "Great work! LB 0.65 as a solo competitor is amazing.
\nLet's fight together next time :)
Deit. learned new staff.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1336017": "First of all, I would like to say thanks to Kaggle and organizers hosts this exciting competition. Congrats winners who won this tough competition! \n\n## Findings\n\nI have tried adversarial validation in early stage. Here is the result.\n\n\n\nThis figure by itself tells how this test set was made. Test set can be split into two subsets. I call them as ...\n\n* Test subset A (left samples on figure which is not similar with train set)\n* Test subset B (right samples on figure which is similar with train set)\n\nThen, I was pretty sure that pseudo labeling will work well for test set. I also thought that it's necessary to create models which fits both subset well. It's possible to estimate how much current model fits for each samples, if erase all half predictions.\n\n\n## Training and modeling\n\nI used 3 models, Deit tiny, small and base.\n\nMy model mainly predicts smiles instead of inchi. smiles is more compact than inchi. It helps reducing training time. smiles is converted inchi before submission. I also created a model to predict inchi, but it's for fallback when smiles is invalid. As a summary, I predicted inchi from 3 ways. The priority order is followings.\n\n* norm inchi from predicted smiles\n* norm inchi from predicted inchi\n* raw predicted inchi\n\nTraining steps consist from 3 steps.\n\n1. Pre-train only decoder by using randomized smiles canonicalization.\n2. Combine Deit with the decoder trained on step 1 and fine tune by extra synthesized images (size 224).\n3. Fine tune with train set (only hard samples) and pseudo labels with image size=384.\n\nAt step1, I train transformer to predict canonicalized inchi and smiles from randomized smiles. Next, self attention and heads on decoder were transferred and combined with Deit. So, only cross attention on decoder is trained from scratch. As I don't have so many GPUs, 224 images are used at this step.\n\nFinally, I have trained 6 models at step 3.\n\n* a1) Deit tiny trained by pseudo labels from Test subset A + train set (only hard samples).\n* a2) Deit small trained by pseudo labels from Test subset A + train set (only hard samples).\n* a3) Deit base trained by pseudo labels from Test subset A + train set (only hard samples).\n* b1) Deit tiny trained by pseudo labels from Test subset B + train set (only hard samples).\n* b2) Deit small trained by pseudo labels from Test subset B + train set (only hard samples).\n* b3) Deit base trained by pseudo labels from Test subset B + train set (only hard samples).\n\nPosition embedding was expanded for image size 384 by interpolation at step 3.\n\n## Inference\n\nI used TTA (rot90) and logits averaging per token.\n\nFor Test subset A, I used averaging weight 9:9:9:1:1:1 for models a1, a2, a3, b1, b2 and b3. On the other hand, for Test subset B, I used weight 1:1:1:9:9:9. It ends up with 0.64 on LB.", "1336051": "Great work! LB 0.65 as a solo competitor is amazing.\nLet's fight together next time :)", "1336497": "Deit. learned new staff." }, "source": "meta" }