Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created a video on the topic, and would love to hear more about what resonated with you, and what questions you might have about TPUs!
\n\nYour forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.
", "messages": [ { "id": "800851", "postDate": "04/07/2020 19:47:17", "content": "Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created a video on the topic, and would love to hear more about what resonated with you, and what questions you might have about TPUs!
\n\nYour forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.
", "rawMarkdown": "Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created [a video on the topic](https://www.youtube.com/watch?v=JC84GCU7zqA), and would love to hear more about what resonated with you, and what questions you might have about TPUs!\n\nYour forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.", "votes": null }, { "id": "801511", "postDate": "04/08/2020 15:00:38", "content": "Hi Jesse, Glad to have you here in Kaggle. I don't have much experience in TPU or DL. In fact, I have none (zero). However, I watched the video and all I can say that you have great communications skills. And I hope that this talented community, full of brilliant, privileged minds can say anything about your starting video. Welcome!
", "rawMarkdown": "Hi Jesse, Glad to have you here in Kaggle. I don't have much experience in TPU or DL. In fact, I have none (zero). However, I watched the video and all I can say that you have great communications skills. And I hope that this talented community, full of brilliant, privileged minds can say anything about your starting video. Welcome!", "votes": null }, { "id": "801522", "postDate": "04/08/2020 15:04:52", "content": "Thank you so much for your kind words, Marília! I'm just getting started with TPUs and deep learning myself, and excited to learn with y'all!
", "rawMarkdown": "Thank you so much for your kind words, Marília! I'm just getting started with TPUs and deep learning myself, and excited to learn with y'all!", "votes": null }, { "id": "801559", "postDate": "04/08/2020 15:32:33", "content": "That's good because in another post the subject is about the TPU costs. At this moment new Kagglers have more concern about costs than in B float 16/32, systolic arrays, less debugging. So it would be useful if Kaggle can add information for those news that are arriving in this community daily. Maybe another topic writing about that could be better (since this topic is about the opinion about video)
", "rawMarkdown": "That's good because in another post the subject is about the TPU costs. At this moment new Kagglers have more concern about costs than in B float 16/32, systolic arrays, less debugging. So it would be useful if Kaggle can add information for those news that are arriving in this community daily. Maybe another topic writing about that could be better (since this topic is about the opinion about video)", "votes": null }, { "id": "801596", "postDate": "04/08/2020 15:58:21", "content": "Thank you for flagging that! When you get a moment, could you link the post? I've done a quick look and can't seem to find it.
", "rawMarkdown": "Thank you for flagging that! When you get a moment, could you link the post? I've done a quick look and can't seem to find it.", "votes": null }, { "id": "801604", "postDate": "04/08/2020 16:06:42", "content": "https://www.kaggle.com/c/flower-classification-with-tpus/discussion/141981 All doubts about TPUs are in TPU flowers Competition. Martin (Görner) had answered some. That topic above has been posted 4 hours ago.
", "rawMarkdown": "https://www.kaggle.com/c/flower-classification-with-tpus/discussion/141981 All doubts about TPUs are in TPU flowers Competition. Martin (Görner) had answered some. That topic above has been posted 4 hours ago.", "votes": null }, { "id": "801671", "postDate": "04/08/2020 17:18:29", "content": "perfect, thank you! it looks like someone has answered them :) and I'll keep this in mind as we update TPU documentation for beginners!
", "rawMarkdown": "perfect, thank you! it looks like someone has answered them :) and I'll keep this in mind as we update TPU documentation for beginners!", "votes": null }, { "id": "801676", "postDate": "04/08/2020 17:22:33", "content": "Kaggle never ceases to Amaze Me 💚! ; Welcome to the Family :)
", "rawMarkdown": "Kaggle never ceases to Amaze Me 💚! ; Welcome to the Family :)", "votes": null }, { "id": "807417", "postDate": "04/14/2020 16:21:06", "content": "@jessemostipak Thanks for the educative video! I have a question. In this tutorial, it's still recommended to use mixed_precision policy, even if bfloat16 is already used in TPU computation. Can we still get benefit by using it? If so, why not it's default?
\n\nThanks in advance,
", "rawMarkdown": "jessemostipak Thanks for the educative video! I have a question. In [this tutorial](https://www.tensorflow.org/guide/keras/mixed_precision), it's still recommended to use mixed_precision policy, even if bfloat16 is already used in TPU computation. Can we still get benefit by using it? If so, why not it's default?\n\nThanks in advance,", "votes": null }, { "id": "807506", "postDate": "04/14/2020 17:30:13", "content": "bfloat16 conversion happens automatically when you're using TPUs, and you don't need to write any code to accomplish the conversion. this article on bfloat16 in TPUs does a good job of breaking things down. the section on mixed-precision training in particular talks about the conversion done by the XLA compiler.
", "rawMarkdown": "bfloat16 conversion happens automatically when you're using TPUs, and you don't need to write any code to accomplish the conversion. [this article on bfloat16 in TPUs](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus) does a good job of breaking things down. the section on mixed-precision training in particular talks about the conversion done by the XLA compiler.", "votes": null }, { "id": "807537", "postDate": "04/14/2020 17:50:29", "content": "@jessemostipak Thanks, that means we can still get benefit by covnersion by XLA compiler, am I undestanding correct?\nthis looks easy to use! Do you know it also work with tf.keras model?\n```\ndef build_model(features):\n # write your model like normal\n return logits
\n\ndef model_fn(features):\n with tf.contrib.tpu.bfloat16_scope():\n logits = build_model(features)\n logits = tf.cast(logits, dtype=tf.float32)\n```
", "rawMarkdown": "jessemostipak Thanks, that means we can still get benefit by covnersion by XLA compiler, am I undestanding correct?\nthis looks easy to use! Do you know it also work with tf.keras model?\n```\ndef build_model(features):\n # write your model like normal\n return logits\n\ndef model_fn(features):\n with tf.contrib.tpu.bfloat16_scope():\n logits = build_model(features)\n logits = tf.cast(logits, dtype=tf.float32)\n```", "votes": null }, { "id": "807578", "postDate": "04/14/2020 18:11:18", "content": "Nice video. I am scared of using TPUs, but after this video I will.\nI am used to see Rachael Tatman videos. I hope to see more great videos of yours. Thanks a lot @jessemostipak.
", "rawMarkdown": "Nice video. I am scared of using TPUs, but after this video I will.\nI am used to see Rachael Tatman videos. I hope to see more great videos of yours. Thanks a lot @jessemostipak.", "votes": null }, { "id": "808746", "postDate": "04/15/2020 15:30:49", "content": "Thank you! And if you're looking to get started with TPUs, we have a great Getting Started notebook that sets you up to do the Flower Classification Playground Competition!
", "rawMarkdown": "Thank you! And if you're looking to get started with TPUs, we have a great [Getting Started notebook](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/) that sets you up to do the [Flower Classification Playground Competition](https://www.kaggle.com/c/flower-classification-with-tpus)!", "votes": null }, { "id": "808749", "postDate": "04/15/2020 15:34:07", "content": "I definitely recommend checking out some of our TPU resources for getting started! I really like our TPU documentation as well as the TPU Getting Started notebook :) These will help walk you through the basics of using TPUs!
", "rawMarkdown": "I definitely recommend checking out some of our TPU resources for getting started! I really like our [TPU documentation](https://www.kaggle.com/docs/tpu) as well as the [TPU Getting Started notebook ](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu):) These will help walk you through the basics of using TPUs!", "votes": null }, { "id": "808875", "postDate": "04/15/2020 17:29:51", "content": "You can use the new Keras mixed-precision API tf.keras.mixed_precision.Policy('mixed_bfloat16') on TPUs. It has no effect on computations which happen in mixed precision at the hardware level no matter what. However, it can be used as a memory optimization. Using this policy, some tensors will be stored in memory in 16-bit format which saves space. This in turn can lead to faster training if the memory optimization allows you to increase the batch size and the increased batch size leads to better hardware utilization.
", "rawMarkdown": "You can use the new Keras mixed-precision API tf.keras.mixed_precision.Policy('mixed_bfloat16') on TPUs. It has no effect on computations which happen in mixed precision at the hardware level no matter what. However, it can be used as a memory optimization. Using this policy, some tensors will be stored in memory in 16-bit format which saves space. This in turn can lead to faster training if the memory optimization allows you to increase the batch size and the increased batch size leads to better hardware utilization.", "votes": null }, { "id": "808941", "postDate": "04/15/2020 18:17:21", "content": "@mgornergoogle @jessemostipak Thanks, I'll try it!
", "rawMarkdown": "mgornergoogle @jessemostipak Thanks, I'll try it!", "votes": null }, { "id": "1926948", "postDate": "09/05/2022 09:16:55", "content": "WOW! I am new but very excited to learn so many things.
", "rawMarkdown": "WOW! I am new but very excited to learn so many things.", "votes": null } ], "comments": [ { "id": 1926948, "author_name": "ayushgupta3005", "author_url": "", "post_date": "09/05/2022 09:16:55", "content": "WOW! I am new but very excited to learn so many things.
", "votes": null, "replies": [] }, { "id": 801511, "author_name": "mpwolke", "author_url": "", "post_date": "04/08/2020 15:00:38", "content": "Hi Jesse, Glad to have you here in Kaggle. I don't have much experience in TPU or DL. In fact, I have none (zero). However, I watched the video and all I can say that you have great communications skills. And I hope that this talented community, full of brilliant, privileged minds can say anything about your starting video. Welcome!
", "votes": null, "replies": [ { "id": 801522, "author_name": "jessemostipak", "author_url": "", "post_date": "04/08/2020 15:04:52", "content": "Thank you so much for your kind words, Marília! I'm just getting started with TPUs and deep learning myself, and excited to learn with y'all!
", "votes": null, "replies": [] }, { "id": 801559, "author_name": "mpwolke", "author_url": "", "post_date": "04/08/2020 15:32:33", "content": "That's good because in another post the subject is about the TPU costs. At this moment new Kagglers have more concern about costs than in B float 16/32, systolic arrays, less debugging. So it would be useful if Kaggle can add information for those news that are arriving in this community daily. Maybe another topic writing about that could be better (since this topic is about the opinion about video)
", "votes": null, "replies": [] }, { "id": 801596, "author_name": "jessemostipak", "author_url": "", "post_date": "04/08/2020 15:58:21", "content": "Thank you for flagging that! When you get a moment, could you link the post? I've done a quick look and can't seem to find it.
", "votes": null, "replies": [] } ] }, { "id": 801604, "author_name": "mpwolke", "author_url": "", "post_date": "04/08/2020 16:06:42", "content": "https://www.kaggle.com/c/flower-classification-with-tpus/discussion/141981 All doubts about TPUs are in TPU flowers Competition. Martin (Görner) had answered some. That topic above has been posted 4 hours ago.
", "votes": null, "replies": [ { "id": 801671, "author_name": "jessemostipak", "author_url": "", "post_date": "04/08/2020 17:18:29", "content": "perfect, thank you! it looks like someone has answered them :) and I'll keep this in mind as we update TPU documentation for beginners!
", "votes": null, "replies": [] } ] }, { "id": 801676, "author_name": "adityaecdrid", "author_url": "", "post_date": "04/08/2020 17:22:33", "content": "Kaggle never ceases to Amaze Me 💚! ; Welcome to the Family :)
", "votes": null, "replies": [] }, { "id": 807417, "author_name": "bamps53", "author_url": "", "post_date": "04/14/2020 16:21:06", "content": "@jessemostipak Thanks for the educative video! I have a question. In this tutorial, it's still recommended to use mixed_precision policy, even if bfloat16 is already used in TPU computation. Can we still get benefit by using it? If so, why not it's default?
\n\nThanks in advance,
", "votes": null, "replies": [ { "id": 807506, "author_name": "jessemostipak", "author_url": "", "post_date": "04/14/2020 17:30:13", "content": "bfloat16 conversion happens automatically when you're using TPUs, and you don't need to write any code to accomplish the conversion. this article on bfloat16 in TPUs does a good job of breaking things down. the section on mixed-precision training in particular talks about the conversion done by the XLA compiler.
", "votes": null, "replies": [] }, { "id": 807537, "author_name": "bamps53", "author_url": "", "post_date": "04/14/2020 17:50:29", "content": "@jessemostipak Thanks, that means we can still get benefit by covnersion by XLA compiler, am I undestanding correct?\nthis looks easy to use! Do you know it also work with tf.keras model?\n```\ndef build_model(features):\n # write your model like normal\n return logits
\n\ndef model_fn(features):\n with tf.contrib.tpu.bfloat16_scope():\n logits = build_model(features)\n logits = tf.cast(logits, dtype=tf.float32)\n```
", "votes": null, "replies": [] }, { "id": 808749, "author_name": "jessemostipak", "author_url": "", "post_date": "04/15/2020 15:34:07", "content": "I definitely recommend checking out some of our TPU resources for getting started! I really like our TPU documentation as well as the TPU Getting Started notebook :) These will help walk you through the basics of using TPUs!
", "votes": null, "replies": [] }, { "id": 808875, "author_name": "mgorner", "author_url": "", "post_date": "04/15/2020 17:29:51", "content": "You can use the new Keras mixed-precision API tf.keras.mixed_precision.Policy('mixed_bfloat16') on TPUs. It has no effect on computations which happen in mixed precision at the hardware level no matter what. However, it can be used as a memory optimization. Using this policy, some tensors will be stored in memory in 16-bit format which saves space. This in turn can lead to faster training if the memory optimization allows you to increase the batch size and the increased batch size leads to better hardware utilization.
", "votes": null, "replies": [] }, { "id": 808941, "author_name": "bamps53", "author_url": "", "post_date": "04/15/2020 18:17:21", "content": "@mgornergoogle @jessemostipak Thanks, I'll try it!
", "votes": null, "replies": [] } ] }, { "id": 807578, "author_name": "masteronepiece", "author_url": "", "post_date": "04/14/2020 18:11:18", "content": "Nice video. I am scared of using TPUs, but after this video I will.\nI am used to see Rachael Tatman videos. I hope to see more great videos of yours. Thanks a lot @jessemostipak.
", "votes": null, "replies": [ { "id": 808746, "author_name": "jessemostipak", "author_url": "", "post_date": "04/15/2020 15:30:49", "content": "Thank you! And if you're looking to get started with TPUs, we have a great Getting Started notebook that sets you up to do the Flower Classification Playground Competition!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "800851": "Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created [a video on the topic](https://www.youtube.com/watch?v=JC84GCU7zqA), and would love to hear more about what resonated with you, and what questions you might have about TPUs!\n\nYour forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.", "801511": "Hi Jesse, Glad to have you here in Kaggle. I don't have much experience in TPU or DL. In fact, I have none (zero). However, I watched the video and all I can say that you have great communications skills. And I hope that this talented community, full of brilliant, privileged minds can say anything about your starting video. Welcome!", "801522": "Thank you so much for your kind words, Marília! I'm just getting started with TPUs and deep learning myself, and excited to learn with y'all!", "801559": "That's good because in another post the subject is about the TPU costs. At this moment new Kagglers have more concern about costs than in B float 16/32, systolic arrays, less debugging. So it would be useful if Kaggle can add information for those news that are arriving in this community daily. Maybe another topic writing about that could be better (since this topic is about the opinion about video)", "801596": "Thank you for flagging that! When you get a moment, could you link the post? I've done a quick look and can't seem to find it.", "801604": "https://www.kaggle.com/c/flower-classification-with-tpus/discussion/141981 All doubts about TPUs are in TPU flowers Competition. Martin (Görner) had answered some. That topic above has been posted 4 hours ago.", "801671": "perfect, thank you! it looks like someone has answered them :) and I'll keep this in mind as we update TPU documentation for beginners!", "801676": "Kaggle never ceases to Amaze Me 💚! ; Welcome to the Family :)", "807417": "jessemostipak Thanks for the educative video! I have a question. In [this tutorial](https://www.tensorflow.org/guide/keras/mixed_precision), it's still recommended to use mixed_precision policy, even if bfloat16 is already used in TPU computation. Can we still get benefit by using it? If so, why not it's default?\n\nThanks in advance,", "807506": "bfloat16 conversion happens automatically when you're using TPUs, and you don't need to write any code to accomplish the conversion. [this article on bfloat16 in TPUs](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus) does a good job of breaking things down. the section on mixed-precision training in particular talks about the conversion done by the XLA compiler.", "807537": "jessemostipak Thanks, that means we can still get benefit by covnersion by XLA compiler, am I undestanding correct?\nthis looks easy to use! Do you know it also work with tf.keras model?\n```\ndef build_model(features):\n # write your model like normal\n return logits\n\ndef model_fn(features):\n with tf.contrib.tpu.bfloat16_scope():\n logits = build_model(features)\n logits = tf.cast(logits, dtype=tf.float32)\n```", "807578": "Nice video. I am scared of using TPUs, but after this video I will.\nI am used to see Rachael Tatman videos. I hope to see more great videos of yours. Thanks a lot @jessemostipak.", "808746": "Thank you! And if you're looking to get started with TPUs, we have a great [Getting Started notebook](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/) that sets you up to do the [Flower Classification Playground Competition](https://www.kaggle.com/c/flower-classification-with-tpus)!", "808749": "I definitely recommend checking out some of our TPU resources for getting started! I really like our [TPU documentation](https://www.kaggle.com/docs/tpu) as well as the [TPU Getting Started notebook ](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu):) These will help walk you through the basics of using TPUs!", "808875": "You can use the new Keras mixed-precision API tf.keras.mixed_precision.Policy('mixed_bfloat16') on TPUs. It has no effect on computations which happen in mixed precision at the hardware level no matter what. However, it can be used as a memory optimization. Using this policy, some tensors will be stored in memory in 16-bit format which saves space. This in turn can lead to faster training if the memory optimization allows you to increase the batch size and the increased batch size leads to better hardware utilization.", "808941": "mgornergoogle @jessemostipak Thanks, I'll try it!", "1926948": "WOW! I am new but very excited to learn so many things." }, "source": "meta" }