\"Both techniques used in parallel training of machine learning models, but they serve different purposes.\"
\n\"Tensor tiling is a technique used to optimize the performance of tensor operations by partitioning the tensor into smaller, fixed-size tiles that can be loaded into memory and processed more efficiently.\"
\n\"Tensor sharding, on the other hand, is a technique used to distribute the computation of large tensors across multiple devices or machines in a distributed system. The tensor is divided into smaller pieces, or shards, and each shard is processed independently on different devices.\"
\n\"Both techniques can be used in conjunction with XLA (Accelerated Linear Algebra) and Hlo (High-Level Optimizer) technologies to optimize the computation graph used in deep learning training. GSPMD (gated synchronous parallelism data parallelism) is a specific parallel training approach that leverages these technologies and techniques to efficiently distribute the data and computations required for training across multiple devices or machines.\"
\nAnswered by Joe Apr 12 at 10:38
\n\n\"Simulated annealing is a method for solving unconstrained and bound-constrained optimization problems. The method models the physical process of heating a material and then slowly lowering the temperature to decrease defects, thus minimizing the system energy.\"
\n\"At each iteration of the simulated annealing algorithm, a new point is randomly generated. The distance of the new point from the current point, or the extent of the search, is based on a probability distribution with a scale proportional to the temperature.\"
\n\"The algorithm accepts all new points that lower the objective, but also, with a certain probability, points that raise the objective. By accepting points that raise the objective, the algorithm avoids being trapped in local minima, and is able to explore globally for more possible solutions. An annealing schedule is selected to systematically decrease the temperature as the algorithm proceeds. As the temperature decreases, the algorithm reduces the extent of its search to converge to a minimum.\"
\nhttps://www.mathworks.com/help/gads/what-is-simulated-annealing.html
\nBayesian Learning via Stochastic Gradient Langevin Dynamics
\nAuthors: Max Welling and Yee Whye Teh
\n\"In that paper the authors proposed a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize.\"
\n\"This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. The authors also proposed a practical method for Monte Carlo estimates of posterior statistics which monitors a “sampling threshold” and collects samples after it has been surpassed. They applied the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients.\"
\n\"Given the similarities between stochastic gradient algorithms and Langevin dynamics, it is natural to consider combining ideas from the two approaches. This allows efficient use of large datasets while allowing for parameter uncertainty to be captured in a Bayesian manner.
\n\"The approach is straightforward: use Robbins-Monro stochastic gradients, add an amount of Gaussian noise balanced with the step size used, and allow step sizes to go to zero.\"
\nhttps://www.stats.ox.ac.uk/~teh/research/compstats/WelTeh2011a.pdf
", "messages": [ { "id": "2414992", "postDate": "08/30/2023 04:00:35", "content": "\"Both techniques used in parallel training of machine learning models, but they serve different purposes.\"
\n\"Tensor tiling is a technique used to optimize the performance of tensor operations by partitioning the tensor into smaller, fixed-size tiles that can be loaded into memory and processed more efficiently.\"
\n\"Tensor sharding, on the other hand, is a technique used to distribute the computation of large tensors across multiple devices or machines in a distributed system. The tensor is divided into smaller pieces, or shards, and each shard is processed independently on different devices.\"
\n\"Both techniques can be used in conjunction with XLA (Accelerated Linear Algebra) and Hlo (High-Level Optimizer) technologies to optimize the computation graph used in deep learning training. GSPMD (gated synchronous parallelism data parallelism) is a specific parallel training approach that leverages these technologies and techniques to efficiently distribute the data and computations required for training across multiple devices or machines.\"
\nAnswered by Joe Apr 12 at 10:38
\n\n\"Simulated annealing is a method for solving unconstrained and bound-constrained optimization problems. The method models the physical process of heating a material and then slowly lowering the temperature to decrease defects, thus minimizing the system energy.\"
\n\"At each iteration of the simulated annealing algorithm, a new point is randomly generated. The distance of the new point from the current point, or the extent of the search, is based on a probability distribution with a scale proportional to the temperature.\"
\n\"The algorithm accepts all new points that lower the objective, but also, with a certain probability, points that raise the objective. By accepting points that raise the objective, the algorithm avoids being trapped in local minima, and is able to explore globally for more possible solutions. An annealing schedule is selected to systematically decrease the temperature as the algorithm proceeds. As the temperature decreases, the algorithm reduces the extent of its search to converge to a minimum.\"
\nhttps://www.mathworks.com/help/gads/what-is-simulated-annealing.html
\nBayesian Learning via Stochastic Gradient Langevin Dynamics
\nAuthors: Max Welling and Yee Whye Teh
\n\"In that paper the authors proposed a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize.\"
\n\"This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. The authors also proposed a practical method for Monte Carlo estimates of posterior statistics which monitors a “sampling threshold” and collects samples after it has been surpassed. They applied the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients.\"
\n\"Given the similarities between stochastic gradient algorithms and Langevin dynamics, it is natural to consider combining ideas from the two approaches. This allows efficient use of large datasets while allowing for parameter uncertainty to be captured in a Bayesian manner.
\n\"The approach is straightforward: use Robbins-Monro stochastic gradients, add an amount of Gaussian noise balanced with the step size used, and allow step sizes to go to zero.\"
\nhttps://www.stats.ox.ac.uk/~teh/research/compstats/WelTeh2011a.pdf
", "rawMarkdown": "#Tensor Tiling and Tensor Sharding\n\n\"Both techniques used in parallel training of machine learning models, but they serve different purposes.\"\n\n\"Tensor tiling is a technique used to optimize the performance of tensor operations by partitioning the tensor into smaller, fixed-size tiles that can be loaded into memory and processed more efficiently.\"\n\n\"Tensor sharding, on the other hand, is a technique used to distribute the computation of large tensors across multiple devices or machines in a distributed system. The tensor is divided into smaller pieces, or shards, and each shard is processed independently on different devices.\"\n\n\"Both techniques can be used in conjunction with XLA (Accelerated Linear Algebra) and Hlo (High-Level Optimizer) technologies to optimize the computation graph used in deep learning training. GSPMD (gated synchronous parallelism data parallelism) is a specific parallel training approach that leverages these technologies and techniques to efficiently distribute the data and computations required for training across multiple devices or machines.\"\n\nAnswered by Joe Apr 12 at 10:38\n\nhttps://stackoverflow.com/questions/75994203/are-tensor-sharding-and-tensor-tilting-the-same-implementation\n\n#Simulated Annealing\n\n\"Simulated annealing is a method for solving unconstrained and bound-constrained optimization problems. The method models the physical process of heating a material and then slowly lowering the temperature to decrease defects, thus minimizing the system energy.\"\n\n\"At each iteration of the simulated annealing algorithm, a new point is randomly generated. The distance of the new point from the current point, or the extent of the search, is based on a probability distribution with a scale proportional to the temperature.\"\n\n\"The algorithm accepts all new points that lower the objective, but also, with a certain probability, points that raise the objective. By accepting points that raise the objective, the algorithm avoids being trapped in local minima, and is able to explore globally for more possible solutions. An annealing schedule is selected to systematically decrease the temperature as the algorithm proceeds. As the temperature decreases, the algorithm reduces the extent of its search to converge to a minimum.\"\n\nhttps://www.mathworks.com/help/gads/what-is-simulated-annealing.html\n\n#Stochastic Gradient Langevin Dynamics\n\nBayesian Learning via Stochastic Gradient Langevin Dynamics\n\nAuthors: Max Welling and Yee Whye Teh\n\n\"In that paper the authors proposed a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize.\"\n\n\"This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. The authors also proposed a practical method for Monte Carlo estimates of posterior statistics which monitors a “sampling threshold” and collects samples after it has been surpassed. They applied the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients.\"\n\n\"Given the similarities between stochastic gradient algorithms and Langevin dynamics, it is natural to consider combining ideas from the two approaches. This allows efficient use of large datasets while allowing for parameter uncertainty to be captured in a Bayesian manner.\n\n\"The approach is straightforward: use Robbins-Monro stochastic gradients, add an amount of Gaussian noise balanced with the step size used, and allow step sizes to go to zero.\"\n\nhttps://www.stats.ox.ac.uk/~teh/research/compstats/WelTeh2011a.pdf", "votes": null }, { "id": "2415185", "postDate": "08/30/2023 06:57:06", "content": "Great content to start this competition.Thank You !
", "rawMarkdown": "Great content to start this competition.Thank You !", "votes": null }, { "id": "2415576", "postDate": "08/30/2023 13:07:51", "content": "We're going to learn a lot on this competition since everything is different with npz files, Tensor tiling, XLA and much more.
", "rawMarkdown": "We're going to learn a lot on this competition since everything is different with npz files, Tensor tiling, XLA and much more.", "votes": null }, { "id": "2415900", "postDate": "08/30/2023 17:23:24", "content": "Thank you for the good educational summary about these topics!
", "rawMarkdown": "Thank you for the good educational summary about these topics!", "votes": null }, { "id": "2415915", "postDate": "08/30/2023 17:30:45", "content": "Since I'm a beginner, it was the first time that I read about such subjects Samihaija.
", "rawMarkdown": "Since I'm a beginner, it was the first time that I read about such subjects Samihaija.", "votes": null }, { "id": "2416504", "postDate": "08/31/2023 05:15:47", "content": "@mpwolke Thank you for the details analysis.
", "rawMarkdown": "mpwolke Thank you for the details analysis.", "votes": null }, { "id": "2420074", "postDate": "09/02/2023 11:54:59", "content": "Thanks for sharing the information on Tensor Tiling and Tensor Sharding.
\nWe can also observe the multi-core computing with many CPUs.
I've only learned those concepts now CR S Kumar.
\nBy the way, you're doing great on this competition. Good luck!
Thank Miah. I'm glad to hear your words.
", "rawMarkdown": "Thank Miah. I'm glad to hear your words.", "votes": null }, { "id": "2420547", "postDate": "09/02/2023 17:23:04", "content": "Do tensor tiles or tensor shards play a role in improving the efficiency or performance of deep learning models?
", "rawMarkdown": "Do tensor tiles or tensor shards play a role in improving the efficiency or performance of deep learning models?", "votes": null }, { "id": "2420559", "postDate": "09/02/2023 17:35:14", "content": "Hi Miah,
\nAs I wrote (copied from StackOverflow on the 1st §):
\n\"Both techniques can be used in conjunction with XLA and HLO technologies to optimize the computation graph used in deep learning training.\"
\n", "rawMarkdown": "Hi Miah,\n\nAs I wrote (copied from StackOverflow on the 1st §):\n\n\"Both techniques can be used in conjunction with XLA and HLO technologies to optimize the computation graph used in deep learning training.\"\n\nhttps://stackoverflow.com/questions/75994203/are-tensor-sharding-and-tensor-tilting-the-same-implementation", "votes": null }, { "id": "2420577", "postDate": "09/02/2023 17:51:08", "content": "Thanks for replying, I heard those terms first time in your article and tried to learn. Keep writing more articles I learned a lot from you.
", "rawMarkdown": "Thanks for replying, I heard those terms first time in your article and tried to learn. Keep writing more articles I learned a lot from you.", "votes": null }, { "id": "2423238", "postDate": "09/04/2023 14:23:05", "content": "Thanks for sharing this information @mpwolke . I am currently participating in this competition and this information will definitely help.
", "rawMarkdown": "Thanks for sharing this information @mpwolke . I am currently participating in this competition and this information will definitely help.", "votes": null }, { "id": "2423286", "postDate": "09/04/2023 14:43:39", "content": "You're doing great on this competition. I'm glad that you found it useful.
", "rawMarkdown": "You're doing great on this competition. I'm glad that you found it useful.", "votes": null }, { "id": "2425162", "postDate": "09/05/2023 17:30:51", "content": "Very clean content. Thank you for the explanations @mpwolke !
", "rawMarkdown": "Very clean content. Thank you for the explanations @mpwolke !", "votes": null }, { "id": "2425180", "postDate": "09/05/2023 17:36:25", "content": "Merci beaucoup Thibaud.
\nI'm glad that you found it clean.
Great content to start this competition.Thank You !
", "votes": null, "replies": [ { "id": 2415576, "author_name": "mpwolke", "author_url": "", "post_date": "08/30/2023 13:07:51", "content": "We're going to learn a lot on this competition since everything is different with npz files, Tensor tiling, XLA and much more.
", "votes": null, "replies": [] } ] }, { "id": 2415900, "author_name": "samihaija", "author_url": "", "post_date": "08/30/2023 17:23:24", "content": "Thank you for the good educational summary about these topics!
", "votes": null, "replies": [ { "id": 2415915, "author_name": "mpwolke", "author_url": "", "post_date": "08/30/2023 17:30:45", "content": "Since I'm a beginner, it was the first time that I read about such subjects Samihaija.
", "votes": null, "replies": [] } ] }, { "id": 2416504, "author_name": "faysalmiah1721758", "author_url": "", "post_date": "08/31/2023 05:15:47", "content": "@mpwolke Thank you for the details analysis.
", "votes": null, "replies": [ { "id": 2420331, "author_name": "mpwolke", "author_url": "", "post_date": "09/02/2023 14:51:57", "content": "Thank Miah. I'm glad to hear your words.
", "votes": null, "replies": [] } ] }, { "id": 2420074, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "09/02/2023 11:54:59", "content": "Thanks for sharing the information on Tensor Tiling and Tensor Sharding.
\nWe can also observe the multi-core computing with many CPUs.
I've only learned those concepts now CR S Kumar.
\nBy the way, you're doing great on this competition. Good luck!
Do tensor tiles or tensor shards play a role in improving the efficiency or performance of deep learning models?
", "votes": null, "replies": [ { "id": 2420559, "author_name": "mpwolke", "author_url": "", "post_date": "09/02/2023 17:35:14", "content": "Hi Miah,
\nAs I wrote (copied from StackOverflow on the 1st §):
\n\"Both techniques can be used in conjunction with XLA and HLO technologies to optimize the computation graph used in deep learning training.\"
\n", "votes": null, "replies": [ { "id": 2420577, "author_name": "faysalmiah1721758", "author_url": "", "post_date": "09/02/2023 17:51:08", "content": "Thanks for replying, I heard those terms first time in your article and tried to learn. Keep writing more articles I learned a lot from you.
", "votes": null, "replies": [] } ] } ] }, { "id": 2423238, "author_name": "rishabh15virgo", "author_url": "", "post_date": "09/04/2023 14:23:05", "content": "Thanks for sharing this information @mpwolke . I am currently participating in this competition and this information will definitely help.
", "votes": null, "replies": [ { "id": 2423286, "author_name": "mpwolke", "author_url": "", "post_date": "09/04/2023 14:43:39", "content": "You're doing great on this competition. I'm glad that you found it useful.
", "votes": null, "replies": [] } ] }, { "id": 2425162, "author_name": "theudbald", "author_url": "", "post_date": "09/05/2023 17:30:51", "content": "Very clean content. Thank you for the explanations @mpwolke !
", "votes": null, "replies": [ { "id": 2425180, "author_name": "mpwolke", "author_url": "", "post_date": "09/05/2023 17:36:25", "content": "Merci beaucoup Thibaud.
\nI'm glad that you found it clean.