{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Implement Evaluation Metrics | Google Fast or Slow? Competition","metadata":{}},{"cell_type":"markdown","source":"As described in the [Competition Overview: Evaluation Section](www.kaggle.com/competitions/predict-ai-model-runtime/overview/evaluation) there are 2 evaluation metrics used in this competition. The final score is the average of the scores across all collections.\n\nThe 2 metrics used are based on the data collection to be evaluated:\n1. Metric for the collection `title:xla`\n2. Metric for the collections `layout:*`\n\n","metadata":{}},{"cell_type":"markdown","source":"## 1. Metric for the collection `title:xla`\n- This metric is used specifically for the collection `title:xla`.\n\n- `(1-slowdown)` inccured of the top-K predictions is used to reflect how much slower the top-K configurations predicted by the model is from the actual fastest configurations.\n\n- The metric can be formulated as follows:\n$$1 - \\left( \\frac{\\text{The best runtime of the top-k predictions}}{\\text{The best runtime of all configurations}} - 1 \\right) = 2 - \\frac{\\min_{i \\in K} y_i}{\\min_{i \\in A} y_i}$$\n Where K is the top-K predictions, A is all configurations of the given graph from the dataset collection, and y is the measured execution time.\n \n- Reasoning: \nSince the number of possibilities is relatively small, one can enumerate all posibilities and invoke a model on each, then choose the best few (=5, here) configurations as suggested by the model, compile with each of them, then measure the runtime of each and commit to the best.","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\ndef runtime(config):\n    \"\"\"\n    Given a config runs it and returns runtime.\n    (generates a random runtime value for now)\n    \"\"\"\n    return np.random.randint(1, 5000)","metadata":{"execution":{"iopub.status.busy":"2023-08-31T15:22:31.957188Z","iopub.execute_input":"2023-08-31T15:22:31.957858Z","iopub.status.idle":"2023-08-31T15:22:31.994584Z","shell.execute_reply.started":"2023-08-31T15:22:31.957816Z","shell.execute_reply":"2023-08-31T15:22:31.993573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def metric_for_title_collections(\npredicted_rankings: list, actual_rankings:list) -> float:\n    \"\"\"\n    Calculates how much slower the top-K configurations predicted \n    by the model is from the actual fastest configurations.\n    \n    Args:\n        predicted_rankings (list): Performance rankings predicted by model \n        actual_rankings (list): Actual performance rankings\n        \n    Returns:\n        Metric values for title collections\n    \"\"\"\n    # predicted_rankings are sliced down to the first 5 predictions\n    predicted_rankings = predicted_rankings[:5]\n    \n    runtime_dict = {config: runtime(config) for config in actual_rankings}\n    \n    best_runtime_top_k_pred = min([runtime(config) for config in predicted_rankings])\n    best_runtime_all_config = min(runtime_dict.values())\n    \n    result = 2 - (best_runtime_top_k_pred / best_runtime_all_config)\n    \n    return result\n    ","metadata":{"execution":{"iopub.status.busy":"2023-08-31T15:22:31.996166Z","iopub.execute_input":"2023-08-31T15:22:31.996632Z","iopub.status.idle":"2023-08-31T15:22:32.003761Z","shell.execute_reply.started":"2023-08-31T15:22:31.996600Z","shell.execute_reply":"2023-08-31T15:22:32.002227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run for dummy data\nactual_ranking = list(np.random.randint(low = 0,high=35,size=35))\npredicted_ranking = list(np.random.randint(low = 0,high=35,size=35))\n\nmetric_for_title_collections(predicted_ranking, actual_ranking)","metadata":{"execution":{"iopub.status.busy":"2023-08-31T15:22:32.005174Z","iopub.execute_input":"2023-08-31T15:22:32.005576Z","iopub.status.idle":"2023-08-31T15:22:32.025803Z","shell.execute_reply.started":"2023-08-31T15:22:32.005543Z","shell.execute_reply":"2023-08-31T15:22:32.024663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Metric for the collections `layout:*`\n- This metric is used for the collections `layout:xla:random`, `layout:xla:default`, `layout:nlp:random`, and `layout:nlp:default`.\n\n- The metric used is the [Kendal Tau Correlation](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient) (a ranking metric: how well does your model-predicted ranking, correspond to the real ranking of runtimes).\n\n- The metric can be formulated as follows:\n\n Let $(x_1, y_1), \\ldots, (x_n, y_n)$ be a set of obserbations of the joint random variables $X$ and $Y$, such that all the values of $(x_i)$ and $(y_i)$ are unique (ties are neglected for simplicity). Any pair of observations $(x_i, y_i)$ and $(x_j, y_j)$, where $i<j$, are said to be **concordant** if the sort order of $(x_i, x_j)$ and $(y_i, y_j)$ agrees: that is, if either both $x_i > x_j$ and $y_i > y_j$ holds or both $x_i < x_j$ and $y_i < y_j$; otherwise they are said to be **discordant**.\n    $$ \\text{Kendall's} \\; \\tau = \\frac{\\text{(number of concordant pairs)} - \\text{(number of discordant pairs)}}{\\text{(number of pairs)}} \\\\ = 1 - \\frac{2 \\text{(number of discordant pairs)}}{\\frac{n(n-1)}{2}}$$\n    \n    where $-1 \\leq \\tau \\leq 1$.\n\n- Kendall's tau measures the correlation between the predicted rankingings and the actual rankings. A higher value indicates higher correlation.\n\n- Reasoning:\nSince the search space is quite large. Therefore, common search strategies, such as Genetic Algorithm, Simulated Annealing, and Langevin Dynamics, need access to a fitness/utility function (which can be your model). Therefore, it is important that the model can well-preserve the order of the configurations (from fastest to slowest).","metadata":{}},{"cell_type":"code","source":"from scipy.stats import kendalltau\n\ndef metric_for_layout_collections(\npredicted_rankings: list, actual_rankings:list) -> float:\n    \"\"\"\n    Calcuates the kendal tau correaltion between the \n    predicted and actual performance rankings.\n    \n    Args:\n        predicted_rankings (list): Performance rankings predicted by model\n        actual_rankings (list): Actual performance rankings\n        \n    Returns:\n        Kendall's Tau correlation coefficent value of the two lists\n    \n    \"\"\"\n    if len(predicted_rankings) != len(actual_rankings):\n        raise ValueError(f\"\"\"\n        Length of predicted rankings (len = {len(predicted_rankings)}) and actual rankings (len = {len(actual_rankings)}) must be equal.\"\"\")\n    \n    corr, _ = kendalltau(predicted_rankings, actual_rankings)\n    return corr","metadata":{"execution":{"iopub.status.busy":"2023-08-31T15:22:32.028270Z","iopub.execute_input":"2023-08-31T15:22:32.029524Z","iopub.status.idle":"2023-08-31T15:22:33.048955Z","shell.execute_reply.started":"2023-08-31T15:22:32.029438Z","shell.execute_reply":"2023-08-31T15:22:33.047606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run for dummy data\nactual_ranking = list(np.random.randint(low = 0,high=35,size=35))\npredicted_ranking = list(np.random.randint(low = 0,high=35,size=35))\n\nmetric_for_layout_collections(predicted_ranking, actual_ranking)","metadata":{"execution":{"iopub.status.busy":"2023-08-31T15:22:33.050661Z","iopub.execute_input":"2023-08-31T15:22:33.051513Z","iopub.status.idle":"2023-08-31T15:22:33.066319Z","shell.execute_reply.started":"2023-08-31T15:22:33.051468Z","shell.execute_reply":"2023-08-31T15:22:33.064755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## References\n- https://arxiv.org/pdf/2308.13490.pdf\n- https://arxiv.org/pdf/2008.01040.pdf\n- https://arxiv.org/pdf/1712.03351.pdf","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}