{"cells":[{"metadata":{"trusted":false},"cell_type":"code","source":"import numpy as np\nimport scipy.stats as stats\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This notebook corresponds [this](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/183086) topic and I use it for drawing distributions and samples from it. Steps:\n- get hyperparameters $n_{cv}$, $n_{test}$ (amount of samples in the train and test set respectively) and $\\sigma$ (domain shifting factor).\n- generate some prior losses $l_{CV}^0, l_{pub}^0, l_{priv}^0$. We dont use some particular solution *S* for getting losses (in this case we would get only posterior losses), instead we just generate solution *S* by generating prior losses for each sample.\n- count mean losses and drawing distributions with some samples."},{"metadata":{},"cell_type":"markdown","source":"# Hyperparameters"},{"metadata":{"trusted":false},"cell_type":"code","source":"# hyperparameters for prior losses generating (some estimation from leaderboard)\nL0, SIGMA0 = 6.8, 2\n\n# small noise deviation\nEPS = 0.1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code"},{"metadata":{"trusted":false},"cell_type":"code","source":"def plot_solution(n_cv, n_test, sigma, ax):\n    ## hyperparameters\n    n_pub, n_priv = 0.15 * n_test, 0.85 * n_test\n\n    ## prior losses\n    l0_CV_samples = np.random.normal(loc=L0, scale=SIGMA0, size=int(n_cv))\n    l0_pub_samples = np.random.normal(loc=L0, scale=SIGMA0, size=int(n_pub))\n    l0_priv_samples = np.random.normal(loc=L0, scale=SIGMA0, size=int(n_priv))\n\n    ## mean losess \n    l0_CV = l0_CV_samples.mean()\n    l0_pub = l0_pub_samples.mean()\n    l0_priv = l0_priv_samples.mean()\n    \n    ## distributions\n    l_CV_samples = np.random.normal(loc=l0_CV, scale=EPS, size=5)\n    l_pub_samples = np.random.normal(loc=l0_pub, scale=sigma, size=5)\n    l_priv_samples = np.random.normal(loc=l0_priv, scale=sigma, size=5)\n    \n    ## ## distributions distributions & samples\n    x = np.linspace(L0 - 1*SIGMA0, L0 + 1*SIGMA0, 100)\n\n    ax.plot(x, stats.norm.pdf(x, l0_CV, EPS), label=\"CV\")\n    _ = ax.scatter(l_CV_samples, 0.2 * np.random.rand(5), marker='x')\n\n    ax.plot(x, stats.norm.pdf(x, l0_pub, sigma), label=\"public\")\n    _ = ax.scatter(l_pub_samples, 0.2 * np.random.rand(5), marker='x')\n\n    ax.plot(x, stats.norm.pdf(x, l0_priv, sigma), label=\"private\")\n    _ = ax.scatter(l_priv_samples, 0.2 * np.random.rand(5), marker='x')\n\n    _ = ax.legend()\n    \ndef plot_solutions(n_cv, n_test, sigma):\n    # repeat plot_solution\n    nrows, ncols = 3, 2\n    fig, ax = plt.subplots(nrows=nrows, ncols=ncols, figsize=(16, 12))\n\n    for i in range(nrows):\n        for j in range(ncols):\n            plot_solution(n_cv=n_cv, n_test=n_test, sigma=sigma, ax=ax[i][j])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Results"},{"metadata":{},"cell_type":"markdown","source":"## 1. Big Data & No Domain Shifting"},{"metadata":{"trusted":false},"cell_type":"code","source":"plot_solutions(n_cv=1e3, n_test=1e3, sigma=EPS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Big Data & Domain Shifting"},{"metadata":{"trusted":false},"cell_type":"code","source":"plot_solutions(n_cv=1e3, n_test=1e3, sigma=0.3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plot_solutions(n_cv=1e3, n_test=1e3, sigma=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Small Data & Domain Shifting"},{"metadata":{"trusted":false},"cell_type":"code","source":"plot_solutions(n_cv=2 * 176, n_test=200, sigma=0.3)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}