![\"Quadratic](\"https://i.postimg.cc/DyRTy1mB/9-A1013-F6-6-E10-4-D3-C-8-E20-9399-D5-A6-AEA3.png\")
\n
![\"Cross](\"https://i.postimg.cc/xC8xzF4H/E68-F60-B8-FBFE-4-FA2-B29-D-7-B6-BFB07-FF72.png\")
Use test set get a reliable estimate of each models' performance.
\nRegression :
\nMean Squared Error (MSE) or Mean Absolute Error (MAE). (Lower values are better)
\nMean Absolute Error.
\nMAE = sum( abs(predicted_i - actual_i) ) / total predictions
\ngives an idea of the magnitude of the error, but no idea of the direction
\n0 indicates no error or perfect predictions
\nMean Squared Error.
\nRoot Mean Squared Error (or RMSE)
\nRMSE = sqrt( sum( (predicted_i - actual_i)^2 ) / total predictions)
\nTaking the square root of the mean squared error converts the units back to the original units of the output variable and can be meaningful for description and presentation
\nR^2.
\naka coefficient of determination
\nprovides an indication of the goodness of fit of a set of predictions to the actual values
\n0 and 1 for no-fit and perfect fit respectively
\nClassification
\nClassification Accuracy.
\naccuracy = correct predictions / total predictions * 100
\nnumber of correct predictions made as a ratio of all predictions made
\nthe most misused
\nonly suitable when there are an equal number of observations in each class and all predictions and prediction errors are equally important
\nLogarithmic Loss.
\nevaluating the predictions of probabilities of membership to a given class
\nscalar probability between 0 and 1
\nPredictions that are correct or incorrect are rewarded or punished proportionally to the confidence of the prediction
\nArea Under ROC Curve (AUROC). (Higher values are better)
\nBinary classification problems
\nrepresents a model’s ability to discriminate between positive and negative classes
\narea of 1.0 represents a model that made all predictions perfectly
\nAn area of 0.5 represents a model as good as random>
\ncan be broken down into sensitivity and specificity
\nSensitivity
\ntrue positive rate also called the recall.
\nnumber instances from the positive (first) class that actually predicted correctly.
\nSpecificity
\nConfusion Matrix\n[](https://postimg.cc/DJJszGvp)
\nAccuracy = (TP+TN)/(P+N)
\nPrecision = TP/(TP+FP)
\nRecall/TP rate = TP/P
\nFP Rate = FP/N
\nUnsupervised Learning - Evaluation
\nMuch harder to evaluate, depends on overall goal of the task
\nNever had “Correct Labels” to compare to
\nCluster Homogeneity, Rand Index
\nReinforcement Learning - Evaluation
\nUsually more obvious, since the “evaluation” is built into the actual training of the model.
\nHow well the model performs the task its assigned.
\nquestions to help you pick the winning model:
\nWhich model had the best performance on the test set? (performance)
\nDoes it perform well across various performance metrics? (robustness)
\nDid it also have (one of) the best cross-validated scores from the training set? (consistency)
\nDoes it solve the original business problem? (win condition)
\n
\n
Use test set get a reliable estimate of each models' performance.
\nRegression :
\nMean Squared Error (MSE) or Mean Absolute Error (MAE). (Lower values are better)
\nMean Absolute Error.
\nMAE = sum( abs(predicted_i - actual_i) ) / total predictions
\ngives an idea of the magnitude of the error, but no idea of the direction
\n0 indicates no error or perfect predictions
\nMean Squared Error.
\nRoot Mean Squared Error (or RMSE)
\nRMSE = sqrt( sum( (predicted_i - actual_i)^2 ) / total predictions)
\nTaking the square root of the mean squared error converts the units back to the original units of the output variable and can be meaningful for description and presentation
\nR^2.
\naka coefficient of determination
\nprovides an indication of the goodness of fit of a set of predictions to the actual values
\n0 and 1 for no-fit and perfect fit respectively
\nClassification
\nClassification Accuracy.
\naccuracy = correct predictions / total predictions * 100
\nnumber of correct predictions made as a ratio of all predictions made
\nthe most misused
\nonly suitable when there are an equal number of observations in each class and all predictions and prediction errors are equally important
\nLogarithmic Loss.
\nevaluating the predictions of probabilities of membership to a given class
\nscalar probability between 0 and 1
\nPredictions that are correct or incorrect are rewarded or punished proportionally to the confidence of the prediction
\nArea Under ROC Curve (AUROC). (Higher values are better)
\nBinary classification problems
\nrepresents a model’s ability to discriminate between positive and negative classes
\narea of 1.0 represents a model that made all predictions perfectly
\nAn area of 0.5 represents a model as good as random>
\ncan be broken down into sensitivity and specificity
\nSensitivity
\ntrue positive rate also called the recall.
\nnumber instances from the positive (first) class that actually predicted correctly.
\nSpecificity
\nConfusion Matrix\n[](https://postimg.cc/DJJszGvp)
\nAccuracy = (TP+TN)/(P+N)
\nPrecision = TP/(TP+FP)
\nRecall/TP rate = TP/P
\nFP Rate = FP/N
\nUnsupervised Learning - Evaluation
\nMuch harder to evaluate, depends on overall goal of the task
\nNever had “Correct Labels” to compare to
\nCluster Homogeneity, Rand Index
\nReinforcement Learning - Evaluation
\nUsually more obvious, since the “evaluation” is built into the actual training of the model.
\nHow well the model performs the task its assigned.
\nquestions to help you pick the winning model:
\nWhich model had the best performance on the test set? (performance)
\nDoes it perform well across various performance metrics? (robustness)
\nDid it also have (one of) the best cross-validated scores from the training set? (consistency)
\nDoes it solve the original business problem? (win condition)
\n
\n
Use test set get a reliable estimate of each models' performance.
\nRegression :
\nMean Squared Error (MSE) or Mean Absolute Error (MAE). (Lower values are better)
\nMean Absolute Error.
\nMAE = sum( abs(predicted_i - actual_i) ) / total predictions
\ngives an idea of the magnitude of the error, but no idea of the direction
\n0 indicates no error or perfect predictions
\nMean Squared Error.
\nRoot Mean Squared Error (or RMSE)
\nRMSE = sqrt( sum( (predicted_i - actual_i)^2 ) / total predictions)
\nTaking the square root of the mean squared error converts the units back to the original units of the output variable and can be meaningful for description and presentation
\nR^2.
\naka coefficient of determination
\nprovides an indication of the goodness of fit of a set of predictions to the actual values
\n0 and 1 for no-fit and perfect fit respectively
\nClassification
\nClassification Accuracy.
\naccuracy = correct predictions / total predictions * 100
\nnumber of correct predictions made as a ratio of all predictions made
\nthe most misused
\nonly suitable when there are an equal number of observations in each class and all predictions and prediction errors are equally important
\nLogarithmic Loss.
\nevaluating the predictions of probabilities of membership to a given class
\nscalar probability between 0 and 1
\nPredictions that are correct or incorrect are rewarded or punished proportionally to the confidence of the prediction
\nArea Under ROC Curve (AUROC). (Higher values are better)
\nBinary classification problems
\nrepresents a model’s ability to discriminate between positive and negative classes
\narea of 1.0 represents a model that made all predictions perfectly
\nAn area of 0.5 represents a model as good as random>
\ncan be broken down into sensitivity and specificity
\nSensitivity
\ntrue positive rate also called the recall.
\nnumber instances from the positive (first) class that actually predicted correctly.
\nSpecificity
\nConfusion Matrix\n[](https://postimg.cc/DJJszGvp)
\nAccuracy = (TP+TN)/(P+N)
\nPrecision = TP/(TP+FP)
\nRecall/TP rate = TP/P
\nFP Rate = FP/N
\nUnsupervised Learning - Evaluation
\nMuch harder to evaluate, depends on overall goal of the task
\nNever had “Correct Labels” to compare to
\nCluster Homogeneity, Rand Index
\nReinforcement Learning - Evaluation
\nUsually more obvious, since the “evaluation” is built into the actual training of the model.
\nHow well the model performs the task its assigned.
\nquestions to help you pick the winning model:
\nWhich model had the best performance on the test set? (performance)
\nDoes it perform well across various performance metrics? (robustness)
\nDid it also have (one of) the best cross-validated scores from the training set? (consistency)
\nDoes it solve the original business problem? (win condition)
\nVery useful information, thanks
", "rawMarkdown": "Very useful information, thanks", "votes": null }, { "id": "1718220", "postDate": "03/10/2022 15:56:46", "content": "Thank you @djacon
", "rawMarkdown": "Thank you @djacon", "votes": null }, { "id": "1718258", "postDate": "03/10/2022 16:59:25", "content": "Compiled version of the knowledge accompanied by the alignment of the concepts as per their hierarchy really helps.
\nThanks for sharing ^_^
Thanks @surajjha101
", "rawMarkdown": "Thanks @surajjha101", "votes": null }, { "id": "1721410", "postDate": "03/13/2022 16:58:01", "content": "Thanks for sharing :D @kalilurrahman
", "rawMarkdown": "Thanks for sharing :D @kalilurrahman", "votes": null }, { "id": "1724237", "postDate": "03/16/2022 05:37:27", "content": "Thank you @andriyuru
", "rawMarkdown": "Thank you @andriyuru", "votes": null } ], "comments": [ { "id": 1718216, "author_name": "djacon", "author_url": "", "post_date": "03/10/2022 15:54:29", "content": "Very useful information, thanks
", "votes": null, "replies": [ { "id": 1718220, "author_name": "kalilurrahman", "author_url": "", "post_date": "03/10/2022 15:56:46", "content": "Thank you @djacon
", "votes": null, "replies": [] } ] }, { "id": 1718258, "author_name": "surajjha101", "author_url": "", "post_date": "03/10/2022 16:59:25", "content": "Compiled version of the knowledge accompanied by the alignment of the concepts as per their hierarchy really helps.
\nThanks for sharing ^_^
Thanks @surajjha101
", "votes": null, "replies": [] } ] }, { "id": 1721410, "author_name": "", "author_url": "", "post_date": "03/13/2022 16:58:01", "content": "Thanks for sharing :D @kalilurrahman
", "votes": null, "replies": [ { "id": 1724237, "author_name": "kalilurrahman", "author_url": "", "post_date": "03/16/2022 05:37:27", "content": "Thank you @andriyuru
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1718073": "###### [Reference Source :]( https://github.com/ankschoubey/Huge-Data-Science-Mind-Map/blob/master/Data%20Science%20Markdown%20with%20Images.textbundle/text.markdown)Github\n \n\n## Some good Content on Evaluation metrics (link above)\nUse test set get a reliable estimate of each models' performance.
\nRegression :
\nMean Squared Error (MSE) or Mean Absolute Error (MAE). (Lower values are better)
\nMean Absolute Error.
\nMAE = sum( abs(predicted_i - actual_i) ) / total predictions
\ngives an idea of the magnitude of the error, but no idea of the direction
\n0 indicates no error or perfect predictions
\nMean Squared Error.
\nRoot Mean Squared Error (or RMSE)
\nRMSE = sqrt( sum( (predicted_i - actual_i)^2 ) / total predictions)
\nTaking the square root of the mean squared error converts the units back to the original units of the output variable and can be meaningful for description and presentation
\nR^2.
\naka coefficient of determination
\nprovides an indication of the goodness of fit of a set of predictions to the actual values
\n0 and 1 for no-fit and perfect fit respectively
\nClassification
\nClassification Accuracy.
\naccuracy = correct predictions / total predictions * 100
\nnumber of correct predictions made as a ratio of all predictions made
\nthe most misused
\nonly suitable when there are an equal number of observations in each class and all predictions and prediction errors are equally important
\nLogarithmic Loss.
\nevaluating the predictions of probabilities of membership to a given class
\nscalar probability between 0 and 1
\nPredictions that are correct or incorrect are rewarded or punished proportionally to the confidence of the prediction
\nArea Under ROC Curve (AUROC). (Higher values are better)
\nBinary classification problems
\nrepresents a model’s ability to discriminate between positive and negative classes
\narea of 1.0 represents a model that made all predictions perfectly
\nAn area of 0.5 represents a model as good as random>
\ncan be broken down into sensitivity and specificity
\nSensitivity
\ntrue positive rate also called the recall.
\nnumber instances from the positive (first) class that actually predicted correctly.
\nSpecificity
\nConfusion Matrix\n[](https://postimg.cc/DJJszGvp)
\nAccuracy = (TP+TN)/(P+N)
\nPrecision = TP/(TP+FP)
\nRecall/TP rate = TP/P
\nFP Rate = FP/N
\nUnsupervised Learning - Evaluation
\nMuch harder to evaluate, depends on overall goal of the task
\nNever had “Correct Labels” to compare to
\nCluster Homogeneity, Rand Index
\nReinforcement Learning - Evaluation
\nUsually more obvious, since the “evaluation” is built into the actual training of the model.
\nHow well the model performs the task its assigned.
\nquestions to help you pick the winning model:
\nWhich model had the best performance on the test set? (performance)
\nDoes it perform well across various performance metrics? (robustness)
\nDid it also have (one of) the best cross-validated scores from the training set? (consistency)
\nDoes it solve the original business problem? (win condition)
\n