{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nfrom statsmodels.stats.weightstats import zconfint\nfrom statsmodels.stats.weightstats import _zconfint_generic, _tconfint_generic\nfrom math import sqrt\nfrom scipy import stats\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets assume that we have two models that has been trained with 5 fold cross validation.\nHow to decide which model is better? \nDue to randomness in augumentation usually it hard to do.\nIn this case statistica could help us. \n\n### First thing we could do:\nCalculate 95% confidential intervals for mean score and compare intervals for first and the second model. If they do not cross with each other - choose the model with more scewed interval (for example - for kappa we need to choose model with maximum skew to right due to maximization).\n\n### Second:\nif confidential inetrvals is crossing with each other then we cant choose model with maxium skew because there is a chance that true mean kappa (or score), laying in confidential inetrval could be less than we expecting. So in this reason we need to calculate p-value.\n"},{"metadata":{},"cell_type":"markdown","source":"### Lets see in practice"},{"metadata":{"trusted":true},"cell_type":"code","source":"# CV for first model\nkappabefore = np.array([0.8765, 0.8711, 0.8476, 0.8471, 0.9164]) #v12\nscorebefore = np.array([4.4126, 4.0255, 5.0050, 4.4838, 3.9395]) #v12\n\n# CV for second model\nkappaafter = np.array([0.9028, 0.8792, 0.8715, 0.8756, 0.9123]) #v13\nscoreafter = np.array([3.8182, 3.9054, 4.0769, 4.3103, 3.7365]) #v13","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some fuctions for premutation test\n\ndef permutation_t_stat_1sample(sample, mean):\n    t_stat = sum(list(map(lambda x: x - mean, sample)))\n    return t_stat\n\ndef permutation_zero_distr_1sample(sample, mean, max_permutations = None):\n    centered_sample = list(map(lambda x: x - mean, sample))\n    if max_permutations:\n        signs_array = set([tuple(x) for x in 2 * np.random.randint(2, size = (max_permutations, \n                                                                              len(sample))) - 1 ])\n    else:\n        signs_array =  itertools.product([-1, 1], repeat = len(sample))\n    distr = [sum(centered_sample * np.array(signs)) for signs in signs_array] #####\n    return distr\n\ndef permutation_test(sample, mean, max_permutations = None, alternative = 'two-sided'):\n    if alternative not in ('two-sided', 'less', 'greater'):\n        raise ValueError(\"alternative not recognized\\n\"\n                         \"should be 'two-sided', 'less' or 'greater'\")\n    \n    t_stat = permutation_t_stat_1sample(sample, mean)\n    \n    zero_distr = permutation_zero_distr_1sample(sample, mean, max_permutations)\n    \n    if alternative == 'two-sided':\n        return sum([1. if abs(x) >= abs(t_stat) else 0. for x in zero_distr]) / len(zero_distr)\n    \n    if alternative == 'less':\n        return sum([1. if x <= t_stat else 0. for x in zero_distr]) / len(zero_distr)\n\n    if alternative == 'greater':\n        return sum([1. if x >= t_stat else 0. for x in zero_distr]) / len(zero_distr)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### Calculating confidential intervals"},{"metadata":{"trusted":true},"cell_type":"code","source":"before_mean_std = kappabefore.std(ddof=1)/sqrt(len(kappabefore))\nafter_mean_std = kappaafter.std(ddof=1)/sqrt(len(kappaafter))\nbefore_mean_std_score = scorebefore.std(ddof=1)/sqrt(len(scorebefore))\nafter_mean_std_score = scoreafter.std(ddof=1)/sqrt(len(scoreafter))\nprint('======================== KAPPA ========================')\nprint('mean kappa before {:.4f}'.format(kappabefore.mean()))\nprint('mean kappa after {:.4f}'.format(kappaafter.mean()))\nprint(\"model before mean kappa 95%% confidence interval\", _tconfint_generic(kappabefore.mean(), before_mean_std,\n                                                                       len(kappabefore) - 1,\n                                                                       0.05, 'two-sided'))\nprint(\"model after mean kappa 95%% confidence interval\", _tconfint_generic(kappaafter.mean(), after_mean_std,\n                                                                       len(kappaafter) - 1,\n                                                                       0.05, 'two-sided'))\nprint('======================== LOSS ========================')\nprint('mean score before {:.4f}'.format(scorebefore.mean()))\nprint('mean score after {:.4f}'.format(scoreafter.mean()))\nprint(\"model before mean loss 95%% confidence interval\", _tconfint_generic(scorebefore.mean(), before_mean_std_score,\n                                                                       len(scorebefore) - 1,\n                                                                       0.05, 'two-sided'))\nprint(\"model after mean loss 95%% confidence interval\", _tconfint_generic(scoreafter.mean(), after_mean_std_score,\n                                                                       len(scoreafter) - 1,\n                                                                       0.05, 'two-sided'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we see that both intervals for kappa and score is crossing with each other so there is a chance that TRUE mean kappa for second model is LOWER than TRUE kappa for first model.\nIn our toy example we have only five obseravtions for kappa and loss (5 fold CV) so we have a chance that we observed skewed results that not close to the true values.\n\nSo now we need to calculate a p-value to decide is second model better for true?\ni will use two test: premutation test and Wiloxon test.\nPermutation test is non parametric test that we could use for variables with unknown distributions.\nWilcoxon test is for variables with normal distribution.\nWe dont know what distribution is exactly we have (we could use special tests to check if distribution is normal) so i show both methods. If it will show different results i would prefer to use permutation test results."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('======================== p-test KAPPA ========================')\n_, p = stats.wilcoxon(kappabefore, kappaafter)\nprint('p-value WilcoxonResult test: %f' % p)\nprint(\"p-value permutation test: %f\" % permutation_test(kappabefore - kappaafter, 0., max_permutations = 50000))\n\nprint('======================== p-test LOSS ========================')\n_, p = stats.wilcoxon(scorebefore, scoreafter)\nprint('p-value WilcoxonResult test: %f' % p)\nprint(\"p-value permutation test: %f\" % permutation_test(scorebefore - scoreafter, 0., max_permutations = 50000))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Usually 5% trashold is choosing to decide. In case of 5% we could see >5% p-value for KAPPA and around 5% p-value for SCORE. So what does it mean?\nIt means that we could not say that new model make better KAPPA. If we use KAPPA metric to choose model - we have no any evidence that second model is better so we need to go futher in research.\nIf we use loss as measure - we have chance that second model is better but not for shure here."},{"metadata":{},"cell_type":"markdown","source":"### this type of model selecting we could use not only with CV but also with TTA"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}