Hello everyone, you can visit the notebook Implement Evaluation Metrics | Google Fast or Slow to understand and use the evaluation metrics for this competition.
\nAs described in the Competition Overview: Evaluation Section there are 2 evaluation metrics used in this competition. The final score is the average of the scores across all collections.
\nThe 2 metrics used are based on the data collection to be evaluated:
\ntitle:xlalayout:*title:xlaThis metric is used specifically for the collection title:xla.
(1-slowdown) inccured of the top-K predictions is used to reflect how much slower the top-K configurations predicted by the model is from the actual fastest configurations.
The metric can be formulated as follows:
\n$$1 - \\left( \\frac{\\text{The best runtime of the top-k predictions}}{\\text{The best runtime of all configurations}} - 1 \\right) = 2 - \\frac{\\min_{i \\in K} y_i}{\\min_{i \\in A} y_i}$$
\nWhere K is the top-K predictions, A is all configurations of the given graph from the dataset collection, and y is the measured execution time.
Reasoning:
\nSince the number of possibilities is relatively small, one can enumerate all possibilities and invoke a model on each, then choose the best few (=5, here) configurations as suggested by the model, compile with each of them, then measure the runtime of each and commit to the best.
layout:*This metric is used for the collections layout:xla:random, layout:xla:default, layout:nlp:random, and layout:nlp:default.
The metric used is the Kendal Tau Correlation (a ranking metric: how well does your model-predicted ranking, correspond to the real ranking of runtimes).
The metric can be formulated as follows:
\nLet $(x_1, y_1), \\ldots, (x_n, y_n)$ be a set of obserbations of the joint random variables $X$ and $Y$, such that all the values of $(x_i)$ and $(y_i)$ are unique (ties are neglected for simplicity). Any pair of observations $(x_i, y_i)$ and $(x_j, y_j)$, where $iconcordant if the sort order of $(x_i, x_j)$ and $(y_i, y_j)$ agrees: that is, if either both $x_i > x_j$ and $y_i > y_j$ holds or both $x_i < x_j$ and $y_i < y_j$; otherwise they are said to be discordant.
\n$$ \\text{Kendall's} \\; \\tau = \\frac{\\text{(number of concordant pairs)} - \\text{(number of discordant pairs)}}{\\text{(number of pairs)}} \\ = 1 - \\frac{2 \\text{(number of discordant pairs)}}{\\frac{n(n-1)}{2}}$$
where $-1 \\leq \\tau \\leq 1$.
Kendall's tau measures the correlation between the predicted rankings and the actual rankings. A higher value indicates hihigher correlation.
Reasoning:
\nSince the search space is quite large. Therefore, common search strategies, such as Genetic Algorithm, Simulated Annealing, and Langevin Dynamics, need access to a fitness/utility function (which can be your model). Therefore, it is important that the model can well-preserve the oderd of the configurations (from fastest to slowest).
Hello everyone, you can visit the notebook Implement Evaluation Metrics | Google Fast or Slow to understand and use the evaluation metrics for this competition.
\nAs described in the Competition Overview: Evaluation Section there are 2 evaluation metrics used in this competition. The final score is the average of the scores across all collections.
\nThe 2 metrics used are based on the data collection to be evaluated:
\ntitle:xlalayout:*title:xlaThis metric is used specifically for the collection title:xla.
(1-slowdown) inccured of the top-K predictions is used to reflect how much slower the top-K configurations predicted by the model is from the actual fastest configurations.
The metric can be formulated as follows:
\n$$1 - \\left( \\frac{\\text{The best runtime of the top-k predictions}}{\\text{The best runtime of all configurations}} - 1 \\right) = 2 - \\frac{\\min_{i \\in K} y_i}{\\min_{i \\in A} y_i}$$
\nWhere K is the top-K predictions, A is all configurations of the given graph from the dataset collection, and y is the measured execution time.
Reasoning:
\nSince the number of possibilities is relatively small, one can enumerate all possibilities and invoke a model on each, then choose the best few (=5, here) configurations as suggested by the model, compile with each of them, then measure the runtime of each and commit to the best.
layout:*This metric is used for the collections layout:xla:random, layout:xla:default, layout:nlp:random, and layout:nlp:default.
The metric used is the Kendal Tau Correlation (a ranking metric: how well does your model-predicted ranking, correspond to the real ranking of runtimes).
The metric can be formulated as follows:
\nLet $(x_1, y_1), \\ldots, (x_n, y_n)$ be a set of obserbations of the joint random variables $X$ and $Y$, such that all the values of $(x_i)$ and $(y_i)$ are unique (ties are neglected for simplicity). Any pair of observations $(x_i, y_i)$ and $(x_j, y_j)$, where $iconcordant if the sort order of $(x_i, x_j)$ and $(y_i, y_j)$ agrees: that is, if either both $x_i > x_j$ and $y_i > y_j$ holds or both $x_i < x_j$ and $y_i < y_j$; otherwise they are said to be discordant.
\n$$ \\text{Kendall's} \\; \\tau = \\frac{\\text{(number of concordant pairs)} - \\text{(number of discordant pairs)}}{\\text{(number of pairs)}} \\ = 1 - \\frac{2 \\text{(number of discordant pairs)}}{\\frac{n(n-1)}{2}}$$
where $-1 \\leq \\tau \\leq 1$.
Kendall's tau measures the correlation between the predicted rankings and the actual rankings. A higher value indicates hihigher correlation.
Reasoning:
\nSince the search space is quite large. Therefore, common search strategies, such as Genetic Algorithm, Simulated Annealing, and Langevin Dynamics, need access to a fitness/utility function (which can be your model). Therefore, it is important that the model can well-preserve the oderd of the configurations (from fastest to slowest).
Thank you so much for providing the detailed explanation and the python notebook! One minor error in the notebook: the arguments to metric_for_*_collections are swapped. It doesn't affect to the result thought, but to avoid confusion.
", "rawMarkdown": "Thank you so much for providing the detailed explanation and the python notebook! One minor error in the notebook: the arguments to metric_for_*_collections are swapped. It doesn't affect to the result thought, but to avoid confusion.", "votes": null }, { "id": "2416950", "postDate": "08/31/2023 10:10:49", "content": "Competition evaluation metrics are used to assess the performance of participants in various competitive settings, such as machine learning competitions or sports tournaments. These metrics help determine who has achieved the best results or solutions. Common metrics include:
\nAccuracy: Measures the proportion of correctly classified instances. It's commonly used in classification tasks.
Precision and Recall: Used in binary classification, precision measures the proportion of true positives among predicted positives, while recall measures the proportion of true positives among actual positives.
F1 Score: A balance between precision and recall, it's the harmonic mean of the two. It's useful when there's an uneven class distribution.
Mean Absolute Error (MAE): Measures the average absolute difference between predicted and actual values in regression problems.
Root Mean Squared Error (RMSE): Similar to MAE but penalizes large errors more. Commonly used in regression tasks.
Area Under the Curve (AUC): Used for evaluating binary classification models, it represents the model's ability to distinguish between positive and negative classes.
Log-Loss (Logarithmic Loss): Commonly used in probabilistic classification tasks, it quantifies the accuracy of predicted probabilities.
R-squared (R²): Measures the proportion of variance in the dependent variable that's predictable from the independent variables in regression analysis.
IoU (Intersection over Union): Used in object detection tasks, it measures the overlap between predicted and actual bounding boxes or regions.
BLEU Score: Evaluates the quality of machine-generated text in natural language processing tasks, like machine translation.
Cohen's Kappa: Measures the agreement between annotators in classification tasks, correcting for chance agreement.
Spearman Rank Correlation: Assesses the strength and direction of the monotonic relationship between two variables.
Mean Average Precision (mAP): Commonly used in object detection and information retrieval, it calculates the average precision across multiple classes or queries.
Concordance Index (C-Index): Used in survival analysis to measure the discriminatory power of a survival model.
Gini Coefficient: Measures inequality in a dataset, often used in economics and credit risk assessment.
The choice of metric depends on the specific competition or task and what aspect of performance is most important. It's also important to consider the characteristics of your data and the potential biases these metrics might have. In many cases, a combination of metrics is used to provide a more comprehensive evaluation of performance.
", "rawMarkdown": "Competition evaluation metrics are used to assess the performance of participants in various competitive settings, such as machine learning competitions or sports tournaments. These metrics help determine who has achieved the best results or solutions. Common metrics include:\n\n1. **Accuracy**: Measures the proportion of correctly classified instances. It's commonly used in classification tasks.\n\n2. **Precision and Recall**: Used in binary classification, precision measures the proportion of true positives among predicted positives, while recall measures the proportion of true positives among actual positives.\n\n3. **F1 Score**: A balance between precision and recall, it's the harmonic mean of the two. It's useful when there's an uneven class distribution.\n\n4. **Mean Absolute Error (MAE)**: Measures the average absolute difference between predicted and actual values in regression problems.\n\n5. **Root Mean Squared Error (RMSE)**: Similar to MAE but penalizes large errors more. Commonly used in regression tasks.\n\n6. **Area Under the Curve (AUC)**: Used for evaluating binary classification models, it represents the model's ability to distinguish between positive and negative classes.\n\n7. **Log-Loss (Logarithmic Loss)**: Commonly used in probabilistic classification tasks, it quantifies the accuracy of predicted probabilities.\n\n8. **R-squared (R²)**: Measures the proportion of variance in the dependent variable that's predictable from the independent variables in regression analysis.\n\n9. **IoU (Intersection over Union)**: Used in object detection tasks, it measures the overlap between predicted and actual bounding boxes or regions.\n\n10. **BLEU Score**: Evaluates the quality of machine-generated text in natural language processing tasks, like machine translation.\n\n11. **Cohen's Kappa**: Measures the agreement between annotators in classification tasks, correcting for chance agreement.\n\n12. **Spearman Rank Correlation**: Assesses the strength and direction of the monotonic relationship between two variables.\n\n13. **Mean Average Precision (mAP)**: Commonly used in object detection and information retrieval, it calculates the average precision across multiple classes or queries.\n\n14. **Concordance Index (C-Index)**: Used in survival analysis to measure the discriminatory power of a survival model.\n\n15. **Gini Coefficient**: Measures inequality in a dataset, often used in economics and credit risk assessment.\n\nThe choice of metric depends on the specific competition or task and what aspect of performance is most important. It's also important to consider the characteristics of your data and the potential biases these metrics might have. In many cases, a combination of metrics is used to provide a more comprehensive evaluation of performance.", "votes": null }, { "id": "2417389", "postDate": "08/31/2023 15:23:57", "content": "Thanks, I have fixed the error.
", "rawMarkdown": "Thanks, I have fixed the error.", "votes": null }, { "id": "2420377", "postDate": "09/02/2023 15:12:35", "content": "Thanks for sharing the valuable information and also the notebook.
\nCan we get some useful visualization for these metrics?
Thank you so much for providing the detailed explanation and the python notebook! One minor error in the notebook: the arguments to metric_for_*_collections are swapped. It doesn't affect to the result thought, but to avoid confusion.
", "votes": null, "replies": [ { "id": 2417389, "author_name": "dariussingh", "author_url": "", "post_date": "08/31/2023 15:23:57", "content": "Thanks, I have fixed the error.
", "votes": null, "replies": [] } ] }, { "id": 2416950, "author_name": "bhanupratapbiswas", "author_url": "", "post_date": "08/31/2023 10:10:49", "content": "Competition evaluation metrics are used to assess the performance of participants in various competitive settings, such as machine learning competitions or sports tournaments. These metrics help determine who has achieved the best results or solutions. Common metrics include:
\nAccuracy: Measures the proportion of correctly classified instances. It's commonly used in classification tasks.
Precision and Recall: Used in binary classification, precision measures the proportion of true positives among predicted positives, while recall measures the proportion of true positives among actual positives.
F1 Score: A balance between precision and recall, it's the harmonic mean of the two. It's useful when there's an uneven class distribution.
Mean Absolute Error (MAE): Measures the average absolute difference between predicted and actual values in regression problems.
Root Mean Squared Error (RMSE): Similar to MAE but penalizes large errors more. Commonly used in regression tasks.
Area Under the Curve (AUC): Used for evaluating binary classification models, it represents the model's ability to distinguish between positive and negative classes.
Log-Loss (Logarithmic Loss): Commonly used in probabilistic classification tasks, it quantifies the accuracy of predicted probabilities.
R-squared (R²): Measures the proportion of variance in the dependent variable that's predictable from the independent variables in regression analysis.
IoU (Intersection over Union): Used in object detection tasks, it measures the overlap between predicted and actual bounding boxes or regions.
BLEU Score: Evaluates the quality of machine-generated text in natural language processing tasks, like machine translation.
Cohen's Kappa: Measures the agreement between annotators in classification tasks, correcting for chance agreement.
Spearman Rank Correlation: Assesses the strength and direction of the monotonic relationship between two variables.
Mean Average Precision (mAP): Commonly used in object detection and information retrieval, it calculates the average precision across multiple classes or queries.
Concordance Index (C-Index): Used in survival analysis to measure the discriminatory power of a survival model.
Gini Coefficient: Measures inequality in a dataset, often used in economics and credit risk assessment.
The choice of metric depends on the specific competition or task and what aspect of performance is most important. It's also important to consider the characteristics of your data and the potential biases these metrics might have. In many cases, a combination of metrics is used to provide a more comprehensive evaluation of performance.
", "votes": null, "replies": [] }, { "id": 2420377, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "09/02/2023 15:12:35", "content": "Thanks for sharing the valuable information and also the notebook.
\nCan we get some useful visualization for these metrics?