Thanks to @grohith327 who posted an interesting paper url in Tweet competition here.
\n\nThe paper notes: \"... fine-tuning a model multiple times on the same dataset, varying only the random seed, leads to a large standard deviation of the fine-tuning accuracy (Devlin et al., 2019;Dodge et al., 2020)\" . They propose a baseline to deal with the spurious pattern.
\n\nFrom the paper:\"\n- Use small learning rates combined with bias correction to avoid vanishing gradients early intraining.\n- Increase the number of iterations considerably and train to (almost) zero training loss whilemaking use of early stopping.\"
\n\nIt would be interesting to know who does what to deal with the scores variability. I personally couldn't find any remedy to this.
", "messages": [ { "id": "880204", "postDate": "06/10/2020 05:15:30", "content": "Thanks to @grohith327 who posted an interesting paper url in Tweet competition here.
\n\nThe paper notes: \"... fine-tuning a model multiple times on the same dataset, varying only the random seed, leads to a large standard deviation of the fine-tuning accuracy (Devlin et al., 2019;Dodge et al., 2020)\" . They propose a baseline to deal with the spurious pattern.
\n\nFrom the paper:\"\n- Use small learning rates combined with bias correction to avoid vanishing gradients early intraining.\n- Increase the number of iterations considerably and train to (almost) zero training loss whilemaking use of early stopping.\"
\n\nIt would be interesting to know who does what to deal with the scores variability. I personally couldn't find any remedy to this.
", "rawMarkdown": "Thanks to @grohith327 who posted an interesting paper url in Tweet competition [here](https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/157262).\n\nThe paper notes: \"... fine-tuning a model multiple times on the same dataset, varying only the random seed, leads to a large standard deviation of the fine-tuning accuracy (Devlin et al., 2019;Dodge et al., 2020)\" . They propose a baseline to deal with the spurious pattern.\n\nFrom the paper:\"\n- Use small learning rates combined with bias correction to avoid vanishing gradients early intraining.\n- Increase the number of iterations considerably and train to (almost) zero training loss whilemaking use of early stopping.\"\n\nIt would be interesting to know who does what to deal with the scores variability. I personally couldn't find any remedy to this.", "votes": null }, { "id": "880329", "postDate": "06/10/2020 07:20:00", "content": "hello, @isakev , This problem also bothers me. My keras based model cannot reproduce the results using same random seed. My pytorch based model can reproduce well on kaggle TPU, but Shonenkov's notebook cannot reproduce on Colab TPU， I don' know why. My previous pytorch based model on kaggle TPU changes a lot(from 0.91 to 0.72 at public leaderboard) when I using different random seed.
", "rawMarkdown": "hello, @isakev , This problem also bothers me. My keras based model cannot reproduce the results using same random seed. My pytorch based model can reproduce well on kaggle TPU, but [Shonenkov's notebook](https://www.kaggle.com/shonenkov/tpu-training-super-fast-xlmroberta) cannot reproduce on Colab TPU， I don' know why. My previous pytorch based model on kaggle TPU changes a lot(from 0.91 to 0.72 at public leaderboard) when I using different random seed.", "votes": null }, { "id": "880535", "postDate": "06/10/2020 11:24:31", "content": "With the same (for python, tensorflow, and numpy) random seed my scores changed once by 0.003 (0.9390-0.9420). I did not test it again, though. Explanation might be that the TPU brings randomness when splitting batches across its 8 units. I have no idea how to deal with this problem except for treating the scores with a grain of doubt, or maybe manually managing distribution of batches at a low level programmatically.
", "rawMarkdown": "With the same (for python, tensorflow, and numpy) random seed my scores changed once by 0.003 (0.9390-0.9420). I did not test it again, though. Explanation might be that the TPU brings randomness when splitting batches across its 8 units. I have no idea how to deal with this problem except for treating the scores with a grain of doubt, or maybe manually managing distribution of batches at a low level programmatically.", "votes": null }, { "id": "881266", "postDate": "06/10/2020 21:44:42", "content": "That is something thats been baffling me in this competition. And it makes experimenting really hard. i mostly worked with image data before this competition and I'm not used to this level of instability and scatter with CNNs. Honestly, I have no idea why does this happen.
\n\nThe best I could come up with was first training with MLM, and after that using a different learning rate for the head and the transformer layers. While the scatter is not completely gone this way, but it is somewhat smaller.\nI posted these here: https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/156622
", "rawMarkdown": "That is something thats been baffling me in this competition. And it makes experimenting really hard. i mostly worked with image data before this competition and I'm not used to this level of instability and scatter with CNNs. Honestly, I have no idea why does this happen.\n\nThe best I could come up with was first training with MLM, and after that using a different learning rate for the head and the transformer layers. While the scatter is not completely gone this way, but it is somewhat smaller.\nI posted these here: https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/156622", "votes": null } ], "comments": [ { "id": 880329, "author_name": "shangweichen", "author_url": "", "post_date": "06/10/2020 07:20:00", "content": "hello, @isakev , This problem also bothers me. My keras based model cannot reproduce the results using same random seed. My pytorch based model can reproduce well on kaggle TPU, but Shonenkov's notebook cannot reproduce on Colab TPU， I don' know why. My previous pytorch based model on kaggle TPU changes a lot(from 0.91 to 0.72 at public leaderboard) when I using different random seed.
", "votes": null, "replies": [ { "id": 880535, "author_name": "isakev", "author_url": "", "post_date": "06/10/2020 11:24:31", "content": "With the same (for python, tensorflow, and numpy) random seed my scores changed once by 0.003 (0.9390-0.9420). I did not test it again, though. Explanation might be that the TPU brings randomness when splitting batches across its 8 units. I have no idea how to deal with this problem except for treating the scores with a grain of doubt, or maybe manually managing distribution of batches at a low level programmatically.
", "votes": null, "replies": [] } ] }, { "id": 881266, "author_name": "riblidezso", "author_url": "", "post_date": "06/10/2020 21:44:42", "content": "That is something thats been baffling me in this competition. And it makes experimenting really hard. i mostly worked with image data before this competition and I'm not used to this level of instability and scatter with CNNs. Honestly, I have no idea why does this happen.
\n\nThe best I could come up with was first training with MLM, and after that using a different learning rate for the head and the transformer layers. While the scatter is not completely gone this way, but it is somewhat smaller.\nI posted these here: https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/156622
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "880204": "Thanks to @grohith327 who posted an interesting paper url in Tweet competition [here](https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/157262).\n\nThe paper notes: \"... fine-tuning a model multiple times on the same dataset, varying only the random seed, leads to a large standard deviation of the fine-tuning accuracy (Devlin et al., 2019;Dodge et al., 2020)\" . They propose a baseline to deal with the spurious pattern.\n\nFrom the paper:\"\n- Use small learning rates combined with bias correction to avoid vanishing gradients early intraining.\n- Increase the number of iterations considerably and train to (almost) zero training loss whilemaking use of early stopping.\"\n\nIt would be interesting to know who does what to deal with the scores variability. I personally couldn't find any remedy to this.", "880329": "hello, @isakev , This problem also bothers me. My keras based model cannot reproduce the results using same random seed. My pytorch based model can reproduce well on kaggle TPU, but [Shonenkov's notebook](https://www.kaggle.com/shonenkov/tpu-training-super-fast-xlmroberta) cannot reproduce on Colab TPU， I don' know why. My previous pytorch based model on kaggle TPU changes a lot(from 0.91 to 0.72 at public leaderboard) when I using different random seed.", "880535": "With the same (for python, tensorflow, and numpy) random seed my scores changed once by 0.003 (0.9390-0.9420). I did not test it again, though. Explanation might be that the TPU brings randomness when splitting batches across its 8 units. I have no idea how to deal with this problem except for treating the scores with a grain of doubt, or maybe manually managing distribution of batches at a low level programmatically.", "881266": "That is something thats been baffling me in this competition. And it makes experimenting really hard. i mostly worked with image data before this competition and I'm not used to this level of instability and scatter with CNNs. Honestly, I have no idea why does this happen.\n\nThe best I could come up with was first training with MLM, and after that using a different learning rate for the head and the transformer layers. While the scatter is not completely gone this way, but it is somewhat smaller.\nI posted these here: https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/156622" }, "source": "meta" }