Hello!
\nI just released some resources on how to conduct a distributed training on TPU. I started by converting the entire dataset into blocks of tfrecord files(around 100MB each block). The conversion notebook is at https://www.kaggle.com/tchaye59/mt-tfrecord-custom-vocab and the resulting dataset at https://www.kaggle.com/tchaye59/mtcustomvocabimg. The Encoder, Decoder with Attention Mechanism was used for the training: https://www.kaggle.com/tchaye59/mt-fast-distributed-training-tpu. The training pipeline is only tested on TPU, and the global training bach size is 512*8=4096. (It cloud run on CPU or GPU with reduced batch size)
Hello!
\nI just released some resources on how to conduct a distributed training on TPU. I started by converting the entire dataset into blocks of tfrecord files(around 100MB each block). The conversion notebook is at https://www.kaggle.com/tchaye59/mt-tfrecord-custom-vocab and the resulting dataset at https://www.kaggle.com/tchaye59/mtcustomvocabimg. The Encoder, Decoder with Attention Mechanism was used for the training: https://www.kaggle.com/tchaye59/mt-fast-distributed-training-tpu. The training pipeline is only tested on TPU, and the global training bach size is 512*8=4096. (It cloud run on CPU or GPU with reduced batch size)
How long did it take to convert everything (train and test) to tfrecord?
", "rawMarkdown": "How long did it take to convert everything (train and test) to tfrecord?", "votes": 1, "replies": [ { "id": 1281477, "postDate": "2021-04-23T01:57:54.663Z", "content": "It took longtime around 7 hours for train and 5 hours for test. I first converted the train set. After I run the notebook a second time to only convert the test set and copy back the train dataset
", "rawMarkdown": "It took longtime around 7 hours for train and 5 hours for test. I first converted the train set. After I run the notebook a second time to only convert the test set and copy back the train dataset", "votes": 1 }, { "id": 1289421, "postDate": "2021-05-01T00:41:36.303Z", "content": "Did you have any trouble loading 9GB back to Kaggle??
", "rawMarkdown": "Did you have any trouble loading 9GB back to Kaggle?? ", "votes": 1 }, { "id": 1289445, "postDate": "2021-05-01T02:08:05.443Z", "content": "No check training notebook
", "rawMarkdown": "No check training notebook" } ] }, { "id": 1277604, "postDate": "2021-04-19T01:28:09.463Z", "content": "good work!
\ni think one day, there will be a transformer encoder+decoder TPU public notebook …. that will be killer
\nin the end, the minimum score for bronze could be below 1.5 to 1.7
I'm working on transformer / TPU, but there are still some problems. :(
", "rawMarkdown": "I'm working on transformer / TPU, but there are still some problems. :(" }, { "id": 1281883, "postDate": "2021-04-23T11:37:58.677Z", "content": "We can work together if you want 🧐
", "rawMarkdown": "We can work together if you want 🧐", "votes": 1 }, { "id": 1283663, "postDate": "2021-04-25T06:38:13.863Z", "content": "i open-source the input=patch+coord version in https://www.kaggle.com/c/bms-molecular-translation/discussion/231190
\ni think this can run very fast in TPU
\nbasically, there is no image. It is pure seq-to-seq.
\nthe input token is image patch of dim=16x16 pixel
\nif it can be run very fast, maybe one can train from scratch
", "rawMarkdown": "i open-source the input=patch+coord version in https://www.kaggle.com/c/bms-molecular-translation/discussion/231190\n\ni think this can run very fast in TPU\n\nbasically, there is no image. It is pure seq-to-seq.\n\nthe input token is image patch of dim=16x16 pixel\n\nif it can be run very fast, maybe one can train from scratch", "votes": 1 } ] }, { "id": 1283628, "postDate": "2021-04-25T05:38:47.420Z", "content": "Looks very neat. I would like to use them as my study references, thank you @tchaye59.
", "rawMarkdown": "Looks very neat. I would like to use them as my study references, thank you @tchaye59.", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1281432, "author_name": "Matt Yates", "author_url": "", "post_date": "2021-04-22T23:51:36.840000", "content": "How long did it take to convert everything (train and test) to tfrecord?
", "votes": 1, "replies": [ { "id": 1281477, "author_name": "Jude TCHAYE", "author_url": "", "post_date": "2021-04-23T01:57:54.663000", "content": "It took longtime around 7 hours for train and 5 hours for test. I first converted the train set. After I run the notebook a second time to only convert the test set and copy back the train dataset
", "votes": 1, "replies": [] }, { "id": 1289421, "author_name": "Matt Yates", "author_url": "", "post_date": "2021-05-01T00:41:36.303000", "content": "Did you have any trouble loading 9GB back to Kaggle??
", "votes": 1, "replies": [] }, { "id": 1289445, "author_name": "Jude TCHAYE", "author_url": "", "post_date": "2021-05-01T02:08:05.443000", "content": "No check training notebook
", "votes": 0, "replies": [] } ] }, { "id": 1277604, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-19T01:28:09.463000", "content": "good work!
\ni think one day, there will be a transformer encoder+decoder TPU public notebook …. that will be killer
\nin the end, the minimum score for bronze could be below 1.5 to 1.7
I'm working on transformer / TPU, but there are still some problems. :(
", "votes": 0, "replies": [] }, { "id": 1281883, "author_name": "Jude TCHAYE", "author_url": "", "post_date": "2021-04-23T11:37:58.677000", "content": "We can work together if you want 🧐
", "votes": 1, "replies": [] }, { "id": 1283663, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-25T06:38:13.863000", "content": "i open-source the input=patch+coord version in https://www.kaggle.com/c/bms-molecular-translation/discussion/231190
\ni think this can run very fast in TPU
\nbasically, there is no image. It is pure seq-to-seq.
\nthe input token is image patch of dim=16x16 pixel
\nif it can be run very fast, maybe one can train from scratch
", "votes": 1, "replies": [] } ] }, { "id": 1283628, "author_name": "", "author_url": "", "post_date": "2021-04-25T05:38:47.420000", "content": "Looks very neat. I would like to use them as my study references, thank you @tchaye59.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1277594": "Hello!\nI just released some resources on how to conduct a distributed training on TPU. I started by converting the entire dataset into blocks of tfrecord files(around 100MB each block). The conversion notebook is at https://www.kaggle.com/tchaye59/mt-tfrecord-custom-vocab and the resulting dataset at https://www.kaggle.com/tchaye59/mtcustomvocabimg. The Encoder, Decoder with Attention Mechanism was used for the training: https://www.kaggle.com/tchaye59/mt-fast-distributed-training-tpu. The training pipeline is only tested on TPU, and the global training bach size is 512*8=4096. (It cloud run on CPU or GPU with reduced batch size)", "1281432": "How long did it take to convert everything (train and test) to tfrecord?", "1277604": "good work!\n\ni think one day, there will be a transformer encoder+decoder TPU public notebook .... that will be killer\nin the end, the minimum score for bronze could be below 1.5 to 1.7", "1283628": "Looks very neat. I would like to use them as my study references, thank you @tchaye59." } }