{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Description of the Competetion**\n\nImagine one day, your breathing became consistently labored and shallow. Months later you were finally diagnosed with pulmonary fibrosis, a disorder with no known cause and no known cure, created by scarring of the lungs. If that happened to you, you would want to know your prognosis. That’s where a troubling disease becomes frightening for the patient: outcomes can range from long-term stability to rapid deterioration, but doctors aren’t easily able to tell where an individual may fall on that spectrum. Your help, and data science, may be able to aid in this prediction, which would dramatically help both patients and clinicians.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom scipy import stats\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load Data\ndf_train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ndf_test = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training Set FVC Measurements Per Patient\ntraining_sample_counts = df_train.rename(columns={'Weeks': 'Samples'}).groupby('Patient').agg('count')['Samples']\n#print(f'Training Set FVC Measurements Per Patient \\n{(\"-\") * 41}\\n{training_sample_counts}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv( '../input/osic-pulmonary-fibrosis-progression/sample_submission.csv' )\ndf_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'FVC Statistical Summary\\n{\"-\" * 23}')\n\nprint(f'Mean: {df_train[\"FVC\"].mean():.6}  -  Median: {df_train[\"FVC\"].median():.6}  -  Std: {df_train[\"FVC\"].std():.6}')\nprint(f'Min: {df_train[\"FVC\"].min()}  -  25%: {df_train[\"FVC\"].quantile(0.25)}  -  50%: {df_train[\"FVC\"].quantile(0.5)}  -  75%: {df_train[\"FVC\"].quantile(0.75)}  -  Max: {df_train[\"FVC\"].max()}')\nprint(f'Skew: {df_train[\"FVC\"].skew():.6}  -  Kurtosis: {df_train[\"FVC\"].kurtosis():.6}')\nmissing_values_count = df_train[df_train[\"FVC\"].isnull()].shape[0]\ntraining_samples_count = df_train.shape[0]\nprint(f'Missing Values: {missing_values_count}/{training_samples_count} ({missing_values_count * 100 / training_samples_count:.4}%)')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(ncols=2, figsize=(18, 6), dpi=150)\nsns.distplot(df_train['FVC'], label='FVC', ax=axes[0])\nstats.probplot(df_train['FVC'], plot=axes[1])\naxes[0].set_title(f'FVC Distribution in Training Set', size=15, pad=15)\naxes[1].set_title(f'FVC Probability Plot', size=15, pad=15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Tabular Data**\n\nThere are four continuous features along with `FVC` in tabular data. Those features are:\n\n * `Weeks`: The relative number of weeks pre/post the baseline CT (may be negative). It doesn't have any significant relationship with other features because patients got both better or worse over the course of time regardless of their `Age`.\n * `Percent`: A computed field which approximates the patient's `FVC` as a percent of the typical `FVC` for a person of similar characteristics. This feature has a strong relationship with `FVC` because it is derived from it, but it doesn't have any significant relationship with other features.\n * `Age`: Age of the patient. `Age` has a slight relationship with `FVC` and `Percent` since younger patients have higher lung capacity.\n\nDistributions of `FVC`, `Percent` and `Age` are very similar but `Weeks` is different than those features.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.pairplot(df_train[['FVC', 'Weeks', 'Percent', 'Age', 'Sex']], aspect=1.4, hue='Sex', height=5, diag_kind='kde', kind='reg')\n\ng.axes[3, 0].set_xlabel('FVC', fontsize=20)\ng.axes[3, 1].set_xlabel('Weeks', fontsize=20)\ng.axes[3, 2].set_xlabel('Percent', fontsize=20)\ng.axes[3, 3].set_xlabel('Age', fontsize=20)\ng.axes[0, 0].set_ylabel('FVC', fontsize=20)\ng.axes[1, 0].set_ylabel('Weeks', fontsize=20)\ng.axes[2, 0].set_ylabel('Percent', fontsize=20)\ng.axes[3, 0].set_ylabel('Age', fontsize=20)\n\ng.axes[3, 0].tick_params(axis='x', labelsize=15)\ng.axes[3, 1].tick_params(axis='x', labelsize=15)\ng.axes[3, 2].tick_params(axis='x', labelsize=15)\ng.axes[3, 3].tick_params(axis='x', labelsize=15)\ng.axes[0, 0].tick_params(axis='y', labelsize=15)\ng.axes[1, 0].tick_params(axis='y', labelsize=15)\ng.axes[2, 0].tick_params(axis='y', labelsize=15)\ng.axes[3, 0].tick_params(axis='y', labelsize=15)\n\ng.fig.suptitle('Tabular Data Feature Distributions and Interactions', fontsize=25, y=1.08)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# As seen from the plots above, the only strong correlation is between FVC and Percent. The other features' correlations are between -0.1 and 0.1.\nfig = plt.figure(figsize=(10, 10), dpi=100)\n\nsns.heatmap(df_train.corr(), annot=True, square=True, cmap='coolwarm', annot_kws={'size': 15},  fmt='.2f')   \n\nplt.tick_params(axis='x', labelsize=18, rotation=75)\nplt.tick_params(axis='y', labelsize=18, rotation=0)\nplt.title('Tabular Data Feature Correlations', size=20, pad=20)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport random\nimport math\n\nfrom tqdm.notebook import tqdm\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn.model_selection import StratifiedKFold, GroupKFold, KFold\nfrom sklearn.metrics import mean_squared_error\nimport category_encoders as ce\n\nfrom sklearn.linear_model import Ridge, ElasticNet\nfrom functools import partial\nimport scipy as sp\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed=777):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"OUTPUT_DICT = './'\n\nID = 'Patient_Week'\nTARGET = 'FVC'\nSEED = 777\nseed_everything(seed=SEED)\n\nN_FOLD = 7","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\notest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"otest","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# construct train input\ntrain = pd.concat([train,otest])\noutput = pd.DataFrame()\ngb = train.groupby('Patient')\ntk0 = tqdm(gb, total=len(gb))\nfor _, usr_df in tk0:\n    usr_output = pd.DataFrame()\n    for week, tmp in usr_df.groupby('Weeks'):\n        rename_cols = {'Weeks': 'base_Week', 'FVC': 'base_FVC', 'Age': 'base_Age'}\n        tmp = tmp.rename(columns=rename_cols)\n        drop_cols = ['Age', 'Sex', 'SmokingStatus', 'Percent']\n        _usr_output = usr_df.drop(columns=drop_cols).rename(columns={'Weeks': 'predict_Week'}).merge(tmp, on='Patient')\n        _usr_output['Week_passed'] = _usr_output['predict_Week'] - _usr_output['base_Week']\n        usr_output = pd.concat([usr_output, _usr_output])\n        print(usr_output)\n    output = pd.concat([output, usr_output])\n    \ntrain = output[output['Week_passed']!=0].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# construct test input\ntest = otest.rename(columns={'Weeks': 'base_Week', 'FVC': 'base_FVC', 'Age': 'base_Age'})\nsubmission = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/sample_submission.csv')\nsubmission['Patient'] = submission['Patient_Week'].apply(lambda x: x.split('_')[0])\nsubmission['predict_Week'] = submission['Patient_Week'].apply(lambda x: x.split('_')[1]).astype(int)\ntest = submission.drop(columns=['FVC', 'Confidence']).merge(test, on='Patient')\ntest['Week_passed'] = test['predict_Week'] - test['base_Week']\ntest.set_index('Patient_Week', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"OUTPUT_DICT = './'\n\nID = 'Patient_Week'\nTARGET = 'FVC'\nSEED = 777\nseed_everything(seed=SEED)\n\nN_FOLD = 7","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"folds = train[['Patient', TARGET]].copy()\nFold = GroupKFold(n_splits=N_FOLD)\ngroups = folds['Patient'].values\nfor n, (train_index, val_index) in enumerate(Fold.split(folds, folds[TARGET], groups)):\n    folds.loc[val_index, 'fold'] = int(n)\nfolds['fold'] = folds['fold'].astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#===========================================================\n# model\n#===========================================================\ndef run_single_model(clf, train_df, test_df, folds, features, target, fold_num=0):\n    \n    trn_idx = folds[folds.fold!=fold_num].index\n    val_idx = folds[folds.fold==fold_num].index\n    \n    y_tr = target.iloc[trn_idx].values\n    X_tr = train_df.iloc[trn_idx][features].values\n    y_val = target.iloc[val_idx].values\n    X_val = train_df.iloc[val_idx][features].values\n    \n    oof = np.zeros(len(train_df))\n    predictions = np.zeros(len(test_df))\n    clf.fit(X_tr, y_tr)\n    \n    oof[val_idx] = clf.predict(X_val)\n    predictions += clf.predict(test_df[features])\n    return oof, predictions\n\n\ndef run_kfold_model(clf, train, test, folds, features, target, n_fold=7):\n    \n    oof = np.zeros(len(train))\n    predictions = np.zeros(len(test))\n    feature_importance_df = pd.DataFrame()\n\n    for fold_ in range(n_fold):\n\n        _oof, _predictions = run_single_model(clf,\n                                              train, \n                                              test,\n                                              folds,  \n                                              features,\n                                              target, \n                                              fold_num=fold_)\n        oof += _oof\n        predictions += _predictions/n_fold\n    return oof, predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Predict Dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = train[TARGET]\ntest[TARGET] = np.nan\n\n# features\ncat_features = ['Sex', 'SmokingStatus']\nnum_features = [c for c in test.columns if (test.dtypes[c] != 'object') & (c not in cat_features)]\nfeatures = num_features + cat_features\ndrop_features = [TARGET, 'predict_Week', 'Percent', 'base_Week']\nfeatures = [c for c in features if c not in drop_features]\n\nif cat_features:\n    ce_oe = ce.OrdinalEncoder(cols=cat_features, handle_unknown='impute')\n    ce_oe.fit(train)\n    train = ce_oe.transform(train)\n    test = ce_oe.transform(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scorelist = []\nfor alpha1 in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]:\n    for l1s in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]:\n        \n        print(\" For alpha:\",alpha1,\"& l1_ratio:\",l1s)\n        clf = ElasticNet(alpha=alpha1, l1_ratio = l1s)\n        oof, predictions = run_kfold_model(clf, train, test, folds, features, target, n_fold=N_FOLD)\n\n        train['FVC_pred'] = oof\n        test['FVC_pred'] = predictions\n\n        # baseline score\n        train['Confidence'] = 100\n        train['sigma_clipped'] = train['Confidence'].apply(lambda x: max(x, 70))\n        train['diff'] = abs(train['FVC'] - train['FVC_pred'])\n        train['delta'] = train['diff'].apply(lambda x: min(x, 1000))\n        train['score'] = -math.sqrt(2)*train['delta']/train['sigma_clipped'] - np.log(math.sqrt(2)*train['sigma_clipped'])\n        score = train['score'].mean()\n        print(score)\n\n        def loss_func(weight, row):\n            confidence = weight\n            sigma_clipped = max(confidence, 70)\n            diff = abs(row['FVC'] - row['FVC_pred'])\n            delta = min(diff, 1000)\n            score = -math.sqrt(2)*delta/sigma_clipped - np.log(math.sqrt(2)*sigma_clipped)\n            return -score\n\n        results = []\n        tk0 = tqdm(train.iterrows(), total=len(train))\n        for _, row in tk0:\n            loss_partial = partial(loss_func, row=row)\n            weight = [100]\n            result = sp.optimize.minimize(loss_partial, weight, method='SLSQP')\n            x = result['x']\n            results.append(x[0])\n\n        # optimized score\n        train['Confidence'] = results\n        train['sigma_clipped'] = train['Confidence'].apply(lambda x: max(x, 70))\n        train['diff'] = abs(train['FVC'] - train['FVC_pred'])\n        train['delta'] = train['diff'].apply(lambda x: min(x, 1000))\n        train['score'] = -math.sqrt(2)*train['delta']/train['sigma_clipped'] - np.log(math.sqrt(2)*train['sigma_clipped'])\n        score = train['score'].mean()\n        scorelist.append(score)\n        print(score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scorelist","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Predict Confidence","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TARGET = 'Confidence'\n\ntarget = train[TARGET]\ntest[TARGET] = np.nan\n\n# features\ncat_features = ['Sex', 'SmokingStatus']\nnum_features = [c for c in test.columns if (test.dtypes[c] != 'object') & (c not in cat_features)]\nfeatures = num_features + cat_features\ndrop_features = [ID, TARGET, 'predict_Week', 'base_Week', 'FVC', 'FVC_pred']\nfeatures = [c for c in features if c not in drop_features]\n\noof, predictions = run_kfold_model(clf, train, test, folds, features, target, n_fold=N_FOLD)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['Confidence'] = oof\ntrain['sigma_clipped'] = train['Confidence'].apply(lambda x: max(x, 70))\ntrain['diff'] = abs(train['FVC'] - train['FVC_pred'])\ntrain['delta'] = train['diff'].apply(lambda x: min(x, 1000))\ntrain['score'] = -math.sqrt(2)*train['delta']/train['sigma_clipped'] - np.log(math.sqrt(2)*train['sigma_clipped'])\nscore = train['score'].mean()\nprint(score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test['Confidence'] = predictions\ntest = test.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = submission[['Patient_Week']].merge(test[['Patient_Week', 'FVC_pred', 'Confidence']], on='Patient_Week')\nsub = sub.rename(columns={'FVC_pred': 'FVC'})\n\nfor i in range(len(otest)):\n    sub.loc[sub['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'FVC'] = otest.FVC[i]\n    sub.loc[sub['Patient_Week']==otest.Patient[i]+'_'+str(otest.Weeks[i]), 'Confidence'] = 0.1\n    \nsub[sub.Confidence<1]\n\nsub.to_csv('submission.csv', index=False, float_format='%.1f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### -------------------END FIRST OSIC Pulmonary Fibrosis Progression-----------------------","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}