{"cells":[{"metadata":{},"cell_type":"markdown","source":"# How To Implement Baseline Machine Learning Algorithms From Scratch With Python\nA baseline provides a point of comparison for the more advanced methods that you evaluate later.\n\nIn this tutorial, you will discover how to implement baseline machine learning algorithms from scratch in Python.\n\nAfter completing this tutorial, you will know:\n\nHow to implement the random prediction algorithm.\nHow to implement the zero rule prediction algorithm.\n\n\n![img](https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2016/10/How-To-Implement-Baseline-Machine-Learning-Algorithms-From-Scratch-With-Python.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"You must know whether the predictions for a given algorithm are good or not. But how do you know?\n\nThe answer is to use a baseline prediction algorithm. A baseline prediction algorithm provides a set of predictions that you can evaluate as you would any predictions for your problem, such as classification accuracy or RMSE.\n\nThe scores from these algorithms provide the required point of comparison when evaluating all other machine learning algorithms on your problem.\n\nOnce established, you can comment on how much better a given algorithm is as compared to the naive baseline algorithm, providing context on just how good a given method actually is.\n\nThe two most commonly used baseline algorithms are:\n\n* Random Prediction Algorithm.\n* Zero Rule Algorithm.\n\nWhen starting on a new problem that is more sticky than a conventional classification or regression problem, it is a good idea to first devise a random prediction algorithm that is specific to your prediction problem. Later you can improve upon this and devise a zero rule algorithm.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1. Random Prediction Algorithm\nThe random prediction algorithm predicts a random outcome as observed in the training data.\n\nIt is perhaps the simplest algorithm to implement.\n\nIt requires that you store all of the distinct outcome values in the training data, which could be large on regression problems with lots of distinct values.\n\nBecause [random numbers](https://machinelearningmastery.com/how-to-generate-random-numbers-in-python/) are used to make decisions, it is a good idea to fix the random number seed prior to using the algorithm. This is to ensure that we get the same set of random numbers, and in turn the same decisions each time the algorithm is run.\n\nBelow is an implementation of the Random Prediction Algorithm in a function named random_algorithm().\n\nThe function takes both a training dataset that includes output values and a test dataset for which output values must be predicted.\n\nThe function will work for both classification and regression problems. It assumes that the output value in the training data is the final column for each row.\n\nFirst, the set of unique output values is collected from the training data. Then a randomly selected output value from the set is selected for each row in the test set.\n","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"\nfrom random import seed\nfrom random import randrange\n\n# Generate random predictions\ndef random_algorithm(train, test):\n\toutput_values = [row[-1] for row in train]\n\tunique = list(set(output_values))\n\tpredicted = list()\n\tfor row in test:\n\t\tindex = randrange(len(unique))\n\t\tpredicted.append(unique[index])\n\treturn predicted\n\nseed(1)\ntrain = [[0], [1], [0], [1], [0], [1]]\ntest = [[None], [None], [None], [None]]\npredictions = random_algorithm(train, test)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can test this function with a small dataset that only contains the output column for simplicity.\n\nThe output values in the training dataset are either “0” or “1”, meaning that the set of predictions the algorithm will choose from is {0, 1}. The test set also contains a single column, with no data as the predictions are not known.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from random import seed\nfrom random import randrange\n \n# Generate random predictions\ndef random_algorithm(train, test):\n\toutput_values = [row[-1] for row in train]\n\tunique = list(set(output_values))\n\tpredicted = list()\n\tfor row in test:\n\t\tindex = randrange(len(unique))\n\t\tpredicted.append(unique[index])\n\treturn predicted\n \nseed(1)\ntrain = [[0], [1], [0], [1], [0], [1]]\ntest = [[None], [None], [None], [None]]\npredictions = random_algorithm(train, test)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Zero Rule Algorithm\n The Zero Rule Algorithm is a better baseline than the random algorithm.\n\nIt uses more information about a given problem to create one rule in order to make predictions. This rule is different depending on the problem type.\n\nLet’s start with classification problems, predicting a class label.\n### classification\nFor classification problems, the one rule is to predict the class value that is most common in the training dataset. This means that if a training dataset has 90 instances of class “0” and 10 instances of class “1” that it will predict “0” and achieve a baseline accuracy of 90/100 or 90%.\n\nThis is much better than the random prediction algorithm that would only achieve 82% accuracy on average. For details on how this is estimate for random search is calculated, see below:\n\n= ((0.9 * 0.9) + (0.1 * 0.1)) * 100\n\n= 82%","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Below is a function named zero_rule_algorithm_classification() that implements this for the classification case.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# zero rule algorithm for classification\ndef zero_rule_algorithm_classification(train, test):\n\toutput_values = [row[-1] for row in train]\n\tprediction = max(set(output_values), key=output_values.count)\n\tpredicted = [prediction for i in range(len(test))]\n\treturn predicted","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The function makes use of the max() function with the key attribute, which is a little clever.\n\nGiven a list of class values observed in the training data, the max() function takes a set of unique class values and calls the count on the list of class values for each class value in the set.\n\nThe result is that it returns the class value that has the highest count of observed values in the list of class values observed in the training dataset.\n\nIf all class values have the same count, then we will choose the first class value observed in the dataset.\n\nOnce we select a class value, it is used to make a prediction for each row in the test dataset.\n\nBelow is a worked example with a contrived dataset that contains 4 examples of class “0” and 2 examples of class “1”. We would expect the algorithm to choose the class value “0” as the prediction for each row in the test dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from random import seed\nfrom random import randrange\n \n# zero rule algorithm for classification\ndef zero_rule_algorithm_classification(train, test):\n\toutput_values = [row[-1] for row in train]\n\tprediction = max(set(output_values), key=output_values.count)\n\tpredicted = [prediction for i in range(len(test))]\n\treturn predicted\n \nseed(1)\ntrain = [['0'], ['0'], ['0'], ['0'], ['1'], ['1']]\ntest = [[None], [None], [None], [None]]\npredictions = zero_rule_algorithm_classification(train, test)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Regression\nRegression problems require the prediction of a real value.\n\nA good default prediction for real values is to predict the central tendency. This could be the mean or the median.\n\nA good default is to use the mean (also called the average) of the output value observed in the training data.\n\nThis is likely to have a lower error than random prediction which will return any observed output value.\n\nBelow is a function to do that named zero_rule_algorithm_regression(). It works by calculating the mean value for the observed output values.\n\nmean = sum(value) / total_values\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Once calculated, the mean is then predicted for each row in the training data.\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom random import randrange\n\n# zero rule algorithm for regression\ndef zero_rule_algorithm_regression(train, test):\n\toutput_values = [row[-1] for row in train]\n\tprediction = sum(output_values) / float(len(output_values))\n\tpredicted = [prediction for i in range(len(test))]\n\treturn predicted","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### We can contrive a small dataset where the mean value is known to be 15.\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n10\n15\n12\n15\n18\n20\n\nmean = (10 + 15 + 12 + 15 + 18 + 20) / 6\nmean = 90 / 6\nmean = 15\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Below is the complete example. We would expect that the mean value of 15 will be predicted for each of the 4 rows in the test dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from random import seed\nfrom random import randrange\n \n# zero rule algorithm for regression\ndef zero_rule_algorithm_regression(train, test):\n\toutput_values = [row[-1] for row in train]\n\tprediction = sum(output_values) / float(len(output_values))\n\tpredicted = [prediction for i in range(len(test))]\n\treturn predicted\n \nseed(1)\ntrain = [[10], [15], [12], [15], [18], [20]]\ntest = [[None], [None], [None], [None]]\npredictions = zero_rule_algorithm_regression(train, test)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extensions\nBelow are a few extensions to the baseline algorithms that you may wish to investigate an implement as an extension to this tutorial.\n\n* Alternate Central Tendency where the median, mode or other central tendency calculations are predicted instead of the mean.\n* Moving Average for time series problems where the mean of the last n records is predicted.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}