\nhttps://www.groundai.com/project/training-large-neural-networks-with-constant-memory-using-a-new-execution-algorithm/1
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F58279%2F9fbd04f415ad9585c71302b7469fccb0%2FScreenshot%202020-05-24%20at%2011.52.23%20AM.png?generation=1590292404851167&alt=media\")
Dear Kagglers,
\nThe authors need your interest level in order to release the open source as per the paper below:
\nhttps://www.groundai.com/project/training-large-neural-networks-with-constant-memory-using-a-new-execution-algorithm/1
\n
L2L can run a gigantic 96 layer BERT on a single GPU with only 11GB. Think this is what everyone wants to try bigger network with limited resources.
\nPlease show your interest by upvotes. I will communicate to them.
\nThanks.
Dr.Patrick
\nUpdate: Microsoft is slow to the open source release. Another author has done the release at the following url :
\nhttps://github.com/TezRomacH/layer-to-layer-pytorch
\nPlease try loading a large transformer and let us know your example. Thanks. Dr.
", "messages": [ { "id": "859009", "postDate": "05/24/2020 03:50:12", "content": "Dear Kagglers,
\nThe authors need your interest level in order to release the open source as per the paper below:
\nhttps://www.groundai.com/project/training-large-neural-networks-with-constant-memory-using-a-new-execution-algorithm/1
\n
L2L can run a gigantic 96 layer BERT on a single GPU with only 11GB. Think this is what everyone wants to try bigger network with limited resources.
\nPlease show your interest by upvotes. I will communicate to them.
\nThanks.
Dr.Patrick
\nUpdate: Microsoft is slow to the open source release. Another author has done the release at the following url :
\nhttps://github.com/TezRomacH/layer-to-layer-pytorch
\nPlease try loading a large transformer and let us know your example. Thanks. Dr.
", "rawMarkdown": "Dear Kagglers,\n\nThe authors need your interest level in order to release the open source as per the paper below:\nhttps://www.groundai.com/project/training-large-neural-networks-with-constant-memory-using-a-new-execution-algorithm/1\n\n\nL2L can run a gigantic 96 layer BERT on a single GPU with only 11GB. Think this is what everyone wants to try bigger network with limited resources.\n\nPlease show your interest by upvotes. I will communicate to them.\nThanks.\n\nDr.Patrick\n\nUpdate: Microsoft is slow to the open source release. Another author has done the release at the following url : \n\nhttps://github.com/TezRomacH/layer-to-layer-pytorch\n\nPlease try loading a large transformer and let us know your example. Thanks. Dr.", "votes": null }, { "id": "859027", "postDate": "05/24/2020 04:21:46", "content": "This is precisely what I needed right now! I'm stuck trying to find a way to fine-tune XLM-R (MLM) on Kaggle and Colab Pro. I've tried everything and yet still OOM :(
", "rawMarkdown": "This is precisely what I needed right now! I'm stuck trying to find a way to fine-tune XLM-R (MLM) on Kaggle and Colab Pro. I've tried everything and yet still OOM :(", "votes": null }, { "id": "859212", "postDate": "05/24/2020 08:37:32", "content": "Yes, all of us need to be able to explore without resources limitation and to work on finetuning existing large models. That way, NLP research can improve!
", "rawMarkdown": "Yes, all of us need to be able to explore without resources limitation and to work on finetuning existing large models. That way, NLP research can improve!", "votes": null }, { "id": "860271", "postDate": "05/25/2020 06:54:35", "content": "It's a time-demands solution. I'm super excited to see how it will go.
", "rawMarkdown": "It's a time-demands solution. I'm super excited to see how it will go.", "votes": null }, { "id": "860484", "postDate": "05/25/2020 10:46:59", "content": "Yes, I am very interested to see their code released as well. Thanks!
", "rawMarkdown": "Yes, I am very interested to see their code released as well. Thanks!", "votes": null }, { "id": "880270", "postDate": "06/10/2020 06:40:43", "content": "Any idea when the codebase will be open sourced?
", "rawMarkdown": "Any idea when the codebase will be open sourced?", "votes": null }, { "id": "890589", "postDate": "06/17/2020 15:35:40", "content": "The author just responded to us saying that they are releasing the source code soon. Thanks to all the responses from the kagglers. Let us look forward to it. Cheers Dr.
", "rawMarkdown": "The author just responded to us saying that they are releasing the source code soon. Thanks to all the responses from the kagglers. Let us look forward to it. Cheers Dr.", "votes": null } ], "comments": [ { "id": 859027, "author_name": "ilhamfp31", "author_url": "", "post_date": "05/24/2020 04:21:46", "content": "This is precisely what I needed right now! I'm stuck trying to find a way to fine-tune XLM-R (MLM) on Kaggle and Colab Pro. I've tried everything and yet still OOM :(
", "votes": null, "replies": [ { "id": 859212, "author_name": "drpatrickchan", "author_url": "", "post_date": "05/24/2020 08:37:32", "content": "Yes, all of us need to be able to explore without resources limitation and to work on finetuning existing large models. That way, NLP research can improve!
", "votes": null, "replies": [] } ] }, { "id": 860271, "author_name": "ipythonx", "author_url": "", "post_date": "05/25/2020 06:54:35", "content": "It's a time-demands solution. I'm super excited to see how it will go.
", "votes": null, "replies": [] }, { "id": 860484, "author_name": "learndo9999", "author_url": "", "post_date": "05/25/2020 10:46:59", "content": "Yes, I am very interested to see their code released as well. Thanks!
", "votes": null, "replies": [] }, { "id": 880270, "author_name": "abhilashjain1993", "author_url": "", "post_date": "06/10/2020 06:40:43", "content": "Any idea when the codebase will be open sourced?
", "votes": null, "replies": [] }, { "id": 890589, "author_name": "drpatrickchan", "author_url": "", "post_date": "06/17/2020 15:35:40", "content": "The author just responded to us saying that they are releasing the source code soon. Thanks to all the responses from the kagglers. Let us look forward to it. Cheers Dr.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "859009": "Dear Kagglers,\n\nThe authors need your interest level in order to release the open source as per the paper below:\nhttps://www.groundai.com/project/training-large-neural-networks-with-constant-memory-using-a-new-execution-algorithm/1\n\n\nL2L can run a gigantic 96 layer BERT on a single GPU with only 11GB. Think this is what everyone wants to try bigger network with limited resources.\n\nPlease show your interest by upvotes. I will communicate to them.\nThanks.\n\nDr.Patrick\n\nUpdate: Microsoft is slow to the open source release. Another author has done the release at the following url : \n\nhttps://github.com/TezRomacH/layer-to-layer-pytorch\n\nPlease try loading a large transformer and let us know your example. Thanks. Dr.", "859027": "This is precisely what I needed right now! I'm stuck trying to find a way to fine-tune XLM-R (MLM) on Kaggle and Colab Pro. I've tried everything and yet still OOM :(", "859212": "Yes, all of us need to be able to explore without resources limitation and to work on finetuning existing large models. That way, NLP research can improve!", "860271": "It's a time-demands solution. I'm super excited to see how it will go.", "860484": "Yes, I am very interested to see their code released as well. Thanks!", "880270": "Any idea when the codebase will be open sourced?", "890589": "The author just responded to us saying that they are releasing the source code soon. Thanks to all the responses from the kagglers. Let us look forward to it. Cheers Dr." }, "source": "meta" }