Hi! I want to finetune a wav2vec2 model. I am seeing all the models people use to fine-tune. I am curious are they using Kaggle notebooks to fine-tune or locally and how many rows of data are they to fine-tune? Because I am trying to fine-tune and get out of memory when using GPU. Is there any way people are using to fine-tune any efficient way? Any insights would be helpful.
", "messages": [ { "id": "2429679", "postDate": "09/08/2023 18:16:00", "content": "Hi! I want to finetune a wav2vec2 model. I am seeing all the models people use to fine-tune. I am curious are they using Kaggle notebooks to fine-tune or locally and how many rows of data are they to fine-tune? Because I am trying to fine-tune and get out of memory when using GPU. Is there any way people are using to fine-tune any efficient way? Any insights would be helpful.
", "rawMarkdown": "Hi! I want to finetune a wav2vec2 model. I am seeing all the models people use to fine-tune. I am curious are they using Kaggle notebooks to fine-tune or locally and how many rows of data are they to fine-tune? Because I am trying to fine-tune and get out of memory when using GPU. Is there any way people are using to fine-tune any efficient way? Any insights would be helpful.", "votes": null }, { "id": "2429788", "postDate": "09/08/2023 20:00:07", "content": "You do not have much RAM on Kaggle notebooks. There is not much RAM in GoogleCollab neither. How much RAM do you have on your computer ? Did you try TPU here on Kaggle? It is more performant thant GPU.
", "rawMarkdown": "You do not have much RAM on Kaggle notebooks. There is not much RAM in GoogleCollab neither. How much RAM do you have on your computer ? Did you try TPU here on Kaggle? It is more performant thant GPU.", "votes": null }, { "id": "2430253", "postDate": "09/09/2023 06:59:08", "content": "Thanks for your response. My laptop has 8 GB RAM. It's very little. Personally, I haven't used TPU before to train. So, I don't know how to set up TPU in Pytorch for training the model. If you have, can you share codes, steps, or information? Thanks.
", "rawMarkdown": "Thanks for your response. My laptop has 8 GB RAM. It's very little. Personally, I haven't used TPU before to train. So, I don't know how to set up TPU in Pytorch for training the model. If you have, can you share codes, steps, or information? Thanks.", "votes": null }, { "id": "2430497", "postDate": "09/09/2023 10:21:48", "content": "I am amazed at your ranking. Did you achieve that without model finetuning ?
\nCurious to know what contributed to your ranking ?
\nI am using a GPU with 64GB RAM.
It is possible to work in TPU with pytorch, but it is not trivial. Since you don't have TPU experience, I recommend starting with TensorFlow, which is natively compatible with TPU, and trying to move to pytorch on TPU only after gaining some understanding and confidence. In particular, your code must be XLA-compatible. In TensorFlow, it is as straightforward as making your code compatible with jit_compile = True; however, in pytorch you will have to use torch_xla, which might add another level of complexity. (And believe me, when working with TPU there is always some complexity lol)
", "rawMarkdown": "It is possible to work in TPU with pytorch, but it is not trivial. Since you don't have TPU experience, I recommend starting with TensorFlow, which is natively compatible with TPU, and trying to move to pytorch on TPU only after gaining some understanding and confidence. In particular, your code must be XLA-compatible. In TensorFlow, it is as straightforward as making your code compatible with jit_compile = True; however, in pytorch you will have to use torch_xla, which might add another level of complexity. (And believe me, when working with TPU there is always some complexity lol)", "votes": null }, { "id": "2430765", "postDate": "09/09/2023 15:09:05", "content": "I used a pre-trained model. Since I do not have a high-configuration system.
", "rawMarkdown": "I used a pre-trained model. Since I do not have a high-configuration system.", "votes": null } ], "comments": [ { "id": 2429788, "author_name": "catadanna", "author_url": "", "post_date": "09/08/2023 20:00:07", "content": "You do not have much RAM on Kaggle notebooks. There is not much RAM in GoogleCollab neither. How much RAM do you have on your computer ? Did you try TPU here on Kaggle? It is more performant thant GPU.
", "votes": null, "replies": [ { "id": 2430253, "author_name": "pritamsinha23", "author_url": "", "post_date": "09/09/2023 06:59:08", "content": "Thanks for your response. My laptop has 8 GB RAM. It's very little. Personally, I haven't used TPU before to train. So, I don't know how to set up TPU in Pytorch for training the model. If you have, can you share codes, steps, or information? Thanks.
", "votes": null, "replies": [ { "id": 2430687, "author_name": "shlomoron", "author_url": "", "post_date": "09/09/2023 13:40:01", "content": "It is possible to work in TPU with pytorch, but it is not trivial. Since you don't have TPU experience, I recommend starting with TensorFlow, which is natively compatible with TPU, and trying to move to pytorch on TPU only after gaining some understanding and confidence. In particular, your code must be XLA-compatible. In TensorFlow, it is as straightforward as making your code compatible with jit_compile = True; however, in pytorch you will have to use torch_xla, which might add another level of complexity. (And believe me, when working with TPU there is always some complexity lol)
", "votes": null, "replies": [] } ] } ] }, { "id": 2430497, "author_name": "dhakshiin1601", "author_url": "", "post_date": "09/09/2023 10:21:48", "content": "I am amazed at your ranking. Did you achieve that without model finetuning ?
\nCurious to know what contributed to your ranking ?
\nI am using a GPU with 64GB RAM.
I used a pre-trained model. Since I do not have a high-configuration system.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2429679": "Hi! I want to finetune a wav2vec2 model. I am seeing all the models people use to fine-tune. I am curious are they using Kaggle notebooks to fine-tune or locally and how many rows of data are they to fine-tune? Because I am trying to fine-tune and get out of memory when using GPU. Is there any way people are using to fine-tune any efficient way? Any insights would be helpful.", "2429788": "You do not have much RAM on Kaggle notebooks. There is not much RAM in GoogleCollab neither. How much RAM do you have on your computer ? Did you try TPU here on Kaggle? It is more performant thant GPU.", "2430253": "Thanks for your response. My laptop has 8 GB RAM. It's very little. Personally, I haven't used TPU before to train. So, I don't know how to set up TPU in Pytorch for training the model. If you have, can you share codes, steps, or information? Thanks.", "2430497": "I am amazed at your ranking. Did you achieve that without model finetuning ?\nCurious to know what contributed to your ranking ?\nI am using a GPU with 64GB RAM.", "2430687": "It is possible to work in TPU with pytorch, but it is not trivial. Since you don't have TPU experience, I recommend starting with TensorFlow, which is natively compatible with TPU, and trying to move to pytorch on TPU only after gaining some understanding and confidence. In particular, your code must be XLA-compatible. In TensorFlow, it is as straightforward as making your code compatible with jit_compile = True; however, in pytorch you will have to use torch_xla, which might add another level of complexity. (And believe me, when working with TPU there is always some complexity lol)", "2430765": "I used a pre-trained model. Since I do not have a high-configuration system." }, "source": "meta" }