{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on August 29, 2023. By Marília Prata, mpwolke.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn import feature_extraction, linear_model, model_selection, preprocessing\nimport plotly.graph_objs as go\nimport plotly.offline as py\nimport plotly.express as px\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-08-30T01:33:05.971354Z","iopub.execute_input":"2023-08-30T01:33:05.971996Z","iopub.status.idle":"2023-08-30T01:33:16.381983Z","shell.execute_reply.started":"2023-08-30T01:33:05.971954Z","shell.execute_reply":"2023-08-30T01:33:16.380933Z"},"_kg_hide-output":true,"_kg_hide-input":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#TpuGraphs\n\nTpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs\n\nAuthors: Phitchaya Mangpo Phothilimthana, Sami Abu-El-Haija, Kaidi Cao, Bahare Fatemi, Charith Mendis, Bryan Perozzi\n\n\"Precise hardware performance models play a crucial role in code optimizations. They can assist compilers in making heuristic decisions or aid autotuners in identifying the optimal configuration for a given program.\"\n\n\"For example, the autotuner for XLA, a machine learning compiler, discovered 10-20% speedup on state-of-the-art models serving substantial production traffic at Google. Although there exist a few datasets for program performance prediction, they target small sub-programs such as basic blocks or kernels. This paper introduces TpuGraphs, a performance prediction dataset on full tensor programs, represented as computational graphs, running on Tensor Processing Units (TPUs).\"\n\n\"Each graph in the dataset represents the main computation of a machine learning workload, e.g., a training epoch or an inference step. Each data sample contains a computational graph, a compilation configuration, and the execution time of the graph when compiled with the configuration.\"\n\n\"The graphs in the dataset are collected from open-source machine learning programs, featuring popular model architectures, e.g., ResNet, EfficientNet, Mask R-CNN, and Transformer. TpuGraphs provides 25x more graphs than the largest graph property prediction dataset (with comparable graph sizes), and 770x larger graphs on average compared to existing performance prediction datasets on machine learning programs.\"\n\n\"This graph-level prediction task on large graphs introduces new challenges in learning, ranging from scalability, training efficiency, to model quality.\"\n\nhttps://arxiv.org/abs/2308.13490","metadata":{}},{"cell_type":"markdown","source":"#Fast or Slow? Predict AI Model Runtime\n\n![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSerjIWkN9y7SJNs4G3dopqNGgMKq_g0ENyV0ml5TzmQXGBk4AdUg4RxwSod9wECQsSQHY&usqp=CAU)boomf","metadata":{}},{"cell_type":"markdown","source":"#Download - You have two options to download the dataset:\n\n\"(Recommended) From this page. Please download the npz_all zip file (scroll towards bottom of this page, on the right, look for \"Data Explorer\", click the npz_all directory, then click the download icon left of the \"Data Explorer\" pane. Then, unzip this file to the path ~/data/tpugraphs. Or GitHub.\"\n\n\"If a tensor has N dimensions, feature values of _i are set to -1 if i >= N (-1 padding). A layout determines the order of minor-to-major tensor dimensions. For example, the layout of {1, 0, 2, -1, -1, -1} of a 3D tensor indicates that dimension 1 is the most minor (elements of the most minor dimension are consecutive in the physical space), and dimension 2 is the most major.\" \n\nhttps://www.kaggle.com/competitions/predict-ai-model-runtime/data\n\n#Oh Boy! Only to read the instructions I felt groggy.","metadata":{}},{"cell_type":"markdown","source":"#Open and view .npz file in Python","metadata":{}},{"cell_type":"code","source":"#https://stackoverflow.com/questions/48429408/open-and-view-npz-file-in-python\n\n#npz_test = np.load(\"../input/predict-ai-model-runtime/npz_all/npz/tile/xla/valid/resnet_v1_50_official_batch_128_bf16_2bea628b72fd4080.npz\")\n\nimport numpy\nb = numpy.load('../input/predict-ai-model-runtime/npz_all/npz/tile/xla/valid/resnet_v1_50_official_batch_128_bf16_2bea628b72fd4080.npz')\nprint(b.files)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T02:06:16.343164Z","iopub.execute_input":"2023-08-30T02:06:16.343604Z","iopub.status.idle":"2023-08-30T02:06:16.353451Z","shell.execute_reply.started":"2023-08-30T02:06:16.343568Z","shell.execute_reply":"2023-08-30T02:06:16.352285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Keys -TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs\n\n\nSuppose a .npz file stores a graph (representing a kernel) with n nodes and m edges. In addition, suppose we compile the graph with c different configurations, and run each on a TPU. Crucially, the configuration is at the graph-level. Then, the .npz file stores the following dictionary (can be loaded with d = dict(np.load(\"npz/tile/xla/train/<pick 1>.npz\"))):\"\n\n\"Key \"node_feat\": contains float32 matrix with shape (n, 140). The uth row contains the feature vector for node u < n (please see Subsection \"Node Features\", below). Nodes are ordered topologically.\"\n\n\"Key \"node_opcode\" contains int32 vector with shape (n, ). The uth entry stores the op-code for node u (please see the mapping of opcode to instruction name here).\"\n\n\"Key \"edge_index\" contains int32 matrix with shape (m, 2). If entry i is = [u, v] (where 0 <= u, v < n), then there is a directed edge from node u to node v, where u consumes the output of v.\"\n\n\"Key \"config_feat\" contains float32 matrix with shape (c, 24) with row j containing the (graph-level) configuration feature vector (please see Subsection \"Tile Config Features\").\"\n\n\"Keys \"config_runtime\" and \"config_runtime_normalizers\": both are int64 vectors of length c. Entry j stores the runtime (in nanoseconds) of the given graph compiled with configuration j and a default configuration, respectively. Samples from the same graph may have slightly different \"config_runtime_normalizers\" because they are measured from different runs on multiple machines.\"\n\n\"Finally, for the tile collection, your job is to predict the indices of the best configurations (i.e., ones leading to the smallest d[\"config_runtime\"] / d[\"config_runtime_normalizers\"]).\"","metadata":{}},{"cell_type":"markdown","source":"#Open one .npz file","metadata":{}},{"cell_type":"code","source":"#https://stackoverflow.com/questions/31368710/how-to-open-an-npz-file/71183327#71183327\n\nfrom numpy import load\n\ndata = load('../input/predict-ai-model-runtime/npz_all/npz/tile/xla/valid/resnet_v1_50_official_batch_128_bf16_2bea628b72fd4080.npz')\nlst = data.files\nfor item in lst:\n    print(item)\n    print(data[item])","metadata":{"execution":{"iopub.status.busy":"2023-08-30T02:12:17.795729Z","iopub.execute_input":"2023-08-30T02:12:17.796124Z","iopub.status.idle":"2023-08-30T02:12:17.815464Z","shell.execute_reply.started":"2023-08-30T02:12:17.796091Z","shell.execute_reply":"2023-08-30T02:12:17.814279Z"},"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Tensor Tiling and Tensor Sharding\n\n\"Both techniques used in parallel training of machine learning models, but they serve different purposes.\"\n\n\"Tensor tilting is a technique used to optimize the performance of tensor operations by partitioning the tensor into smaller, fixed-size tiles that can be loaded into memory and processed more efficiently.\"\n\n\"Tensor sharding, on the other hand, is a technique used to distribute the computation of large tensors across multiple devices or machines in a distributed system. The tensor is divided into smaller pieces, or shards, and each shard is processed independently on different devices.\"\n\n\"Both techniques can be used in conjunction with XLA (Accelerated Linear Algebra) and Hlo (High-Level Optimizer) technologies to optimize the computation graph used in deep learning training. GSPMD (gated synchronous parallelism data parallelism) is a specific parallel training approach that leverages these technologies and techniques to efficiently distribute the data and computations required for training across multiple devices or machines.\"\n\nhttps://stackoverflow.com/questions/75994203/are-tensor-sharding-and-tensor-tilting-the-same-implementation\n\nBy Joe Apr 12 at 10:38","metadata":{}},{"cell_type":"code","source":"from numpy import asarray\nfrom numpy import exp\nfrom numpy.random import randn\nfrom numpy.random import rand\nfrom numpy.random import seed\nfrom matplotlib import pyplot","metadata":{"execution":{"iopub.status.busy":"2023-08-30T02:55:37.884167Z","iopub.execute_input":"2023-08-30T02:55:37.884660Z","iopub.status.idle":"2023-08-30T02:55:37.891294Z","shell.execute_reply.started":"2023-08-30T02:55:37.884619Z","shell.execute_reply":"2023-08-30T02:55:37.890056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# objective function\ndef objective(x):\n    return x[0]**2.0","metadata":{"execution":{"iopub.status.busy":"2023-08-30T02:55:54.627856Z","iopub.execute_input":"2023-08-30T02:55:54.628309Z","iopub.status.idle":"2023-08-30T02:55:54.634009Z","shell.execute_reply.started":"2023-08-30T02:55:54.628276Z","shell.execute_reply":"2023-08-30T02:55:54.632789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Simulated Annealing\n\n\"Simulated annealing is a method for solving unconstrained and bound-constrained optimization problems. The method models the physical process of heating a material and then slowly lowering the temperature to decrease defects, thus minimizing the system energy.\"\n\n\"At each iteration of the simulated annealing algorithm, a new point is randomly generated. The distance of the new point from the current point, or the extent of the search, is based on a probability distribution with a scale proportional to the temperature.\"\n\n\"The algorithm accepts all new points that lower the objective, but also, with a certain probability, points that raise the objective. By accepting points that raise the objective, the algorithm avoids being trapped in local minima, and is able to explore globally for more possible solutions. An annealing schedule is selected to systematically decrease the temperature as the algorithm proceeds. As the temperature decreases, the algorithm reduces the extent of its search to converge to a minimum.\"\n\nhttps://www.mathworks.com/help/gads/what-is-simulated-annealing.html","metadata":{}},{"cell_type":"code","source":"#By Celia Ryleigh https://www.kaggle.com/code/celiaryleigh/simulated-annealing\n\n# simulated annealing algorithm\ndef simulated_annealing(objective, bounds, n_iterations, step_size, temp):\n    # generate an initial point\n    best = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])\n    # evaluate the initial point\n    best_eval = objective(best)\n    # current working solution\n    curr, curr_eval = best, best_eval\n    scores = list()\n    # run the algorithm\n    for i in range(n_iterations):\n        # take a step\n        candidate = curr + randn(len(bounds)) * step_size\n        # evaluate candidate point\n        candidate_eval = objective(candidate)\n        # check for new best solution\n        if candidate_eval < best_eval:\n            # store new best point\n            best, best_eval = candidate, candidate_eval\n            # keep track of scores\n            scores.append(best_eval)\n            # report progress\n            print('>%d f(%s) = %.5f' % (i, best, best_eval))\n        # difference between candidate and current point evaluation\n        diff = candidate_eval - curr_eval\n        # calculate temperature for current epoch\n        t = temp / float(i + 1)\n        # calculate metropolis acceptance criterion\n        metropolis = exp(-diff / t)\n        # check if we should keep the new point\n        if diff < 0 or rand() < metropolis:\n            # store the new current point\n            curr, curr_eval = candidate, candidate_eval\n    return [best, best_eval, scores]","metadata":{"execution":{"iopub.status.busy":"2023-08-30T02:56:30.796071Z","iopub.execute_input":"2023-08-30T02:56:30.796476Z","iopub.status.idle":"2023-08-30T02:56:30.805485Z","shell.execute_reply.started":"2023-08-30T02:56:30.796444Z","shell.execute_reply":"2023-08-30T02:56:30.804282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Checking a tf2_bert_pretrain_dynamic_batch_size NPZ file","metadata":{}},{"cell_type":"code","source":"data1 = load('../input/predict-ai-model-runtime/npz_all/npz/tile/xla/valid/tf2_bert_pretrain_dynamic_batch_size_4235885105a43ec2.npz')\nlst1 = data1.files\nfor item1 in lst1:\n    print(item1)\n    print(data1[item1])","metadata":{"execution":{"iopub.status.busy":"2023-08-30T02:25:36.870374Z","iopub.execute_input":"2023-08-30T02:25:36.870833Z","iopub.status.idle":"2023-08-30T02:25:36.895293Z","shell.execute_reply.started":"2023-08-30T02:25:36.870800Z","shell.execute_reply":"2023-08-30T02:25:36.893462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#What should I do with those configurations above?\n\nFrom fastest (smallest runtime) to slowest (largest runtime).  ","metadata":{}},{"cell_type":"markdown","source":"#Stochastic Gradient Langevin Dynamics\n\nBayesian Learning via Stochastic Gradient Langevin Dynamics\n\nAuthors: Max Welling and Yee Whye Teh\n\n\"In that paper the authors proposed a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize.\"\n\n\"This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. The authors also proposed a practical method for Monte Carlo estimates of posterior statistics which monitors a “sampling threshold” and collects samples after it has been surpassed. They applied the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients.\"\n\n\"Given the similarities between stochastic gradient algorithms and Langevin dynamics, it is natural to consider combining ideas from the two approaches. This allows efficient use of large datasets while allowing for parameter uncertainty to be captured in a Bayesian manner. \n\n\"The approach is straightforward: use Robbins-Monro stochastic gradients, add\nan amount of Gaussian noise balanced with the step size used, and allow step sizes to go to zero.\"\n\nhttps://www.stats.ox.ac.uk/~teh/research/compstats/WelTeh2011a.pdf","metadata":{}},{"cell_type":"markdown","source":"#Preserve the order of the configurations (from fastest to slowest).","metadata":{}},{"cell_type":"markdown","source":"Tensorflow XLA: The Fusion Compiler for Tensorflow\n\n![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTe5b2qMT5WHIbj8a9_eseysc8Gp_0zaiRQE1qVIYKMfkrYHa9swpJRHSql1Higg89xeg&usqp=CAU)AnalyticsVidhya","metadata":{}},{"cell_type":"markdown","source":"#Competition Citation\n\n@misc{predict-ai-model-runtime,\n\n    author = {Ashley Chow, Bryan Perozzi, HCL-Jevster, inversion, Mangpo Phothilimthana, Sami Abu-El-Haija},\n    \n    title = {Google - Fast or Slow? Predict AI Model Runtime},\n    \n    publisher = {Kaggle},\n    \n    year = {2023},\n    \n    url = {https://kaggle.com/competitions/predict-ai-model-runtime}\n}","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}