{"cells":[{"metadata":{},"cell_type":"markdown","source":"This competion uses kappa (specifically quadratic weighted kappa) for evaluation. Unlike accuracy score, this metric takes **chance** into account. [Here](https://en.wikipedia.org/wiki/Cohen%27s_kappa#Example) is nice example."},{"metadata":{},"cell_type":"markdown","source":"## So what is (Cohen's) Kappa?"},{"metadata":{},"cell_type":"markdown","source":"Kappa is *Chance adjusted index* for reliability of categorical measurements. This means that, it accounts for amount of agreement between raters that can be expected to have occurred due to chance (i.e., random guessing). This is unlike accuracy where you can get relatively fair score by intelligently random guessing.\nHere 3 matrices are involved:\n* N&times;N(N is number of categories) histogram matrix **O**, where each element e<sub>ij</sub> of O corresponds to the number of observations that received a category i by A and a category j by B. In our case, N=5 so O is 5&times;5 matrix. Each element e<sub>ij</sub> will represent count of images that recieved category i by A(say human) and category j by B(our models). So greater the number in diagonal, greater good.\n* N&times;N weights matrix **W**, where each element is calculated using distance between ratings. More on this later.\n* N&times;N histogram matrix of expected ratings **E**, which is calculated as the outer product between each rater's histogram *vector* of ratings.\n**E** is normalized so that **E** and **O** have the same sum.\nNow, each cell in **O** is multiplied by corresponding cell in **W** and sum the results across all the cells. Call this P<sub>o</sub>. Same is done for **E**. Call this P<sub>e</sub>.\nThen kappa is calculated as below:$$Kappa = \\frac{P_o - P_e}{1 - p_e}$$\nYou can find good example [here](http://vassarstats.net/kappaexp.html)"},{"metadata":{},"cell_type":"markdown","source":"## What is \"quadratic weighted\"?"},{"metadata":{},"cell_type":"markdown","source":"When no weight matrix is involved, its called unweighted kappa. This means that there is no progression between categories. They are nominal.But when categories are ordinal, i.e., they have some kind of progression relationship, for example: sad, ok, happy, very happy or No DR, Mild, Moderate, Severe, Proliferative DR, then weighted kappa is used.\nConcept of **distance** is used to to calculate each element of **W**. Distance between category 2 and 0 is 2, between 3, 0 is 3, between 4 and 1 is 3 and so on. \nLinear weight is calculated as: $$weight = 1 - \\frac{|\\text{distance}|}{\\text{Maximum Possible Distance}}$$\nQuadratic weight is calculated as: $$weight = 1 - \\frac{|\\text{distance}|^2 }{\\text{(Maximum Possible Distance)}^2}$$\nIn this competition, maximum possible distance will be 4."},{"metadata":{},"cell_type":"markdown","source":"## So, what this means for us?"},{"metadata":{},"cell_type":"markdown","source":"* 1st thing 1st, any random guess will be penalized.\n* *If you use ensemble of model by averaging their prediction*, you will get floating values between 0 and 4, which you will then have to convert to integer using round, clip or some other method. But keep in mind that you will be penalized more if distance is more. So if your average is 1.6, and true label is 3, then if you predict to 1, distance would be 2, hence less weight: $$1-\\frac{2^2}{4^2} = 0.75$$. But if you predict 2, then distance will be 1 and thus, comparitively more weight: $$1-\\frac{1^2}{4^2} = 0.9375$$.\n* This means we need to come with threshold for every category. Here's where, [OptimizedRounder class for Quadratic Weighted Kappa (QWK)](https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa) by Abhishek, comes into play."},{"metadata":{},"cell_type":"markdown","source":"## Using Dummy Classifier"},{"metadata":{},"cell_type":"markdown","source":"Let's now try to predict using random guess but stratified, using dummy classifier. Note difference between score and QWK."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.metrics import cohen_kappa_score\n\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's use stratified strategy. So, this classifier will generate predictions by respecting the training set’s class distribution."},{"metadata":{"trusted":true},"cell_type":"code","source":"dummy_clf = DummyClassifier() # Default strategy is 'stratified'\ndummy_clf.fit(train.id_code, train.diagnosis) # Inputs doesn't matter, it's dummy\ntrain_predictions = dummy_clf.predict(train.id_code)\nprint(f\"Score: {dummy_clf.score(train.id_code, train.diagnosis)}\")\nprint(f\"Cohen kappa score: {cohen_kappa_score(train_predictions, train.diagnosis, weights='quadratic')}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv('../input/test.csv')\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = dummy_clf.predict(test.id_code)\nsubmissions = pd.read_csv('../input/sample_submission.csv')\nsubmissions['diagnosis'] = predictions\nsubmissions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submissions.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check out dummy's super high LB score. 1st, if you look bottom-up :D"},{"metadata":{},"cell_type":"markdown","source":"Please let me know if you have any questions, corrections or any other comments.\nAlso bonus point is that you can use this metric as loss ;). Happy Kaggling!!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}