{"cells":[{"metadata":{"_uuid":"79fdcf5952a4438bd6ef522f933d6b70ecab38f1","_cell_guid":"4fd8b437-1a3e-4254-9853-91ac67906b1b"},"cell_type":"markdown","source":"Recently I was looking for an easy way to calculate/plot the ROC-AUC score of my validation predictions and this is the little script I found.\n\nHere I show a sample use case. We will take the first million lines of the training dataset and generate random predictions for them. Then we will calculate the ROC-AUC score between the \"predictions\" and the real class of the data. Let's get started!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","collapsed":true,"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nn = 1000000\ntrain = pd.read_csv('../input/train.csv', nrows=n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c3965229cc02050156e5d2a1f2d0b56b1029a9c","_cell_guid":"79902019-c922-47a9-9ea5-a1eaa75a1257"},"cell_type":"markdown","source":"First we will read the target values of the dataset. These are the correct classifications we will test against."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"4d2fcdac054a0e3b183da3da47770d5789b2a307"},"cell_type":"code","source":"y_train = train['is_attributed']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"566a74cac40ca924eb71328049f4091875285a93"},"cell_type":"markdown","source":"Next we need to make our predictions. These will usually come from a model. For the purposes of this tutorial though, I will simply create a `Series` of random values that will serve as our 'predictions'."},{"metadata":{"_uuid":"9d5e43bf394632b67ad7e1378b624b449f143282","_cell_guid":"6cd20ab5-e678-43ee-b5eb-0d0ab30c0beb","collapsed":true,"trusted":true},"cell_type":"code","source":"predictions = pd.Series(np.random.rand(y_train.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c0cfa9158afccb3f3226140ff38fb4dc8717dc7","_cell_guid":"cda50512-ac0a-4a4a-8c47-7e1a9bd53797"},"cell_type":"markdown","source":"And now we will calculate and plot the AUC score of `y_train` and `predictions`:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom sklearn.metrics import (confusion_matrix, precision_recall_curve, auc,\n                             roc_curve, recall_score, classification_report, f1_score,\n                             precision_recall_fscore_support)\n\nfpr, tpr, thresholds = roc_curve(y_train, predictions)\nroc_auc = auc(fpr, tpr)\n\nplt.title('Receiver Operating Characteristic')\nplt.plot(fpr, tpr, label='AUC = %0.4f'% roc_auc)\nplt.legend(loc='lower right')\nplt.plot([0,1],[0,1],'r--')\nplt.xlim([-0.001, 1])\nplt.ylim([0, 1.001])\nplt.ylabel('True Positive Rate')\nplt.xlabel('False Positive Rate')\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d04b3a60c7bc035dda7aa0d02074adef46fc4dc","_cell_guid":"aa0999f1-6f39-4303-a15b-1c39687ee26a"},"cell_type":"markdown","source":"Unsurprisingly, the AUC score is terrible for the above, since I merely used randomly generated numbers.\n\nYou can use this snippet by replacing the `y_train` data with your validation set data and `predictions` with the predictions your model generated.\n\nHope this helps!\n\n*Disclaimer: I did not create this script, I only moved it from [an article online](https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd) for ease of use.*"}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}