{"cells":[{"metadata":{"papermill":{"duration":0.012261,"end_time":"2020-08-20T13:10:33.841122","exception":false,"start_time":"2020-08-20T13:10:33.828861","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Overview\n\nBased on Wei Hao's awesome training pipeline [3]\n\n# Features\n \n- GPU with optional Mixed precision support\n- TPU support *IN PROGRESS*\n- Wide range of supported backend models via Keras Applications models\n- Wide range of supported optimizers in Tensorflow optimizers and Tensorflow Addons\n- Wide range of supported learning rate scheduling strategies via Tensorflow, Tensorflow experimental and Tensorflow Addons\n- RandAug augmentation support with UNetAug added as a custom aug strategy *IN PROGRESS*\n- Overall cleanup and parameterization of the pipeline\n\n\n\nBelow you can train efnB7 relatively fast ~18 seconds per epoch with 600 image size which makes this notebook able to train large models for more epochs in the 9 hour limit.\n\nDon't forget to turn on the GPU or TPU :)\n","execution_count":null},{"metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2020-08-20T13:10:33.870556Z","iopub.status.busy":"2020-08-20T13:10:33.869709Z","iopub.status.idle":"2020-08-20T13:10:54.378664Z","shell.execute_reply":"2020-08-20T13:10:54.377677Z"},"papermill":{"duration":20.529455,"end_time":"2020-08-20T13:10:54.378916","exception":false,"start_time":"2020-08-20T13:10:33.849461","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"!pip install ../input/kerasapplications/keras-team-keras-applications-3b180cb -f ./ --no-index >/dev/null","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-08-20T13:10:54.404282Z","iopub.status.busy":"2020-08-20T13:10:54.403401Z","iopub.status.idle":"2020-08-20T13:11:01.208783Z","shell.execute_reply":"2020-08-20T13:11:01.209418Z"},"papermill":{"duration":6.82276,"end_time":"2020-08-20T13:11:01.20963","exception":false,"start_time":"2020-08-20T13:10:54.38687","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport pydicom\nimport pandas as pd\nimport numpy as np \nimport tensorflow as tf \nimport matplotlib.pyplot as plt \nfrom tqdm.notebook import tqdm \nfrom tensorflow.keras.layers import (\n    Dense, Dropout, Activation, Flatten, Input, BatchNormalization, GlobalAveragePooling2D, Add, Conv2D, AveragePooling2D, \n    LeakyReLU, Concatenate \n)\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.utils import Sequence\nimport tensorflow.keras.backend as K\nimport tensorflow.keras.applications as tfka\n\nfrom sklearn.model_selection import train_test_split, KFold\nimport seaborn as sns\nfrom tensorflow.keras.mixed_precision import experimental as mixed_precision\nimport tensorflow_addons as tfa\nfrom tensorflow_addons.optimizers import AdamW","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Future TPU support\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection\n    print(\"Running on TPU \", tpu.cluster_spec().as_dict()[\"worker\"])\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n    GPU=False\nexcept ValueError:\n    print(\"Not connected to a TPU runtime. Using CPU/GPU strategy\")\n    strategy = tf.distribute.MirroredStrategy()\n    GPU=True","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.008417,"end_time":"2020-08-20T13:11:03.751195","exception":false,"start_time":"2020-08-20T13:11:03.742778","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Training Parameters\n\n","execution_count":null},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:03.770223Z","iopub.status.busy":"2020-08-20T13:11:03.769612Z","iopub.status.idle":"2020-08-20T13:11:03.773847Z","shell.execute_reply":"2020-08-20T13:11:03.773374Z"},"papermill":{"duration":0.015392,"end_time":"2020-08-20T13:11:03.773943","exception":false,"start_time":"2020-08-20T13:11:03.758551","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"EPOCHS = 10 # number of epochs to train for in each fold\n\nBATCH_SIZE = 4 * strategy.num_replicas_in_sync # batch size of images during training\n\nprint('Batch size = '+str(BATCH_SIZE))\n\nIM_SIZE = 512 #If you are training efficientnet please read https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/\n\n\n\nBASE_MODEL='EfficientNetB1' # supported models are listed here https://www.tensorflow.org/api_docs/python/tf/keras/applications\n\n# args to pass to base model e.g. for EfficientNet {drop_connect_rate=0.4} default is 0.2\nBASE_MODEL_KWARGS = dict(drop_connect_rate=0.2,input_shape=(IM_SIZE,IM_SIZE,1),include_top=False,weights=None)\n\nNFOLD = 5 # number of folds in K-fold cross-validation (CV)\n\n\n\nSTART_LR = 1e-2 * strategy.num_replicas_in_sync # the initial learning rate \nFIRST_DECAY_STEPS = 1e1\n\nLR_STRATEGY='CosineDecayRestarts' # check for options in the cell below\nLR_KWARGS=dict(initial_learning_rate=START_LR, first_decay_steps=FIRST_DECAY_STEPS)\n\nWD = 1e-2 #Weight decay\nOPTIMIZER='AdamW' # check for options in the cell below\nOPTIMIZER_KWARGS=dict(weight_decay=WD)\n\nSAVE_BEST = True # default is True to save best weights on validation loss\n\nMIXED_PRECISION = True\n\n\nEALRY_STOPPING_PATIENCE = 40\n\nTRAIN_STEPS = int(( 8 / BATCH_SIZE) * 32) # number of steps per epoch in training\n\nVAL_STEPS = int((8 / BATCH_SIZE  ) * 16) # number of steps per epoch in validation\n\n\n\nDATA_DIR=[\"../input/osic-pulmonary-fibrosis-progression\",\"../input/osic-pulmonary-fibrosis-progression-lungs-mask\"]\n\n\n\nimport datetime\n\ndef get_now_string():\n  x = str(datetime.datetime.now())\n  date=x[:10].split('-')\n  time=x[11:].split(':')\n  time[2]=time[2].split('.')[0]\n  dates=\"\".join(date)+\"_\"+\"\".join(time)\n  return dates\n\nOUTPUT_DIR=get_now_string()+\"/\"\nos.mkdir(OUTPUT_DIR)\nprint(\"saving models to :\",OUTPUT_DIR)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Supported optimizers\n\noptimizers_mapper={\n    \"AdamW\":tfa.optimizers,\n    \"ConditionalGradient\":tfa.optimizers,\n    \"LAMB\":tfa.optimizers,\n    \"LazyAdam\":tfa.optimizers,\n    \"NovoGrad\":tfa.optimizers,\n    \"RectifiedAdam\":tfa.optimizers,\n    \"SGDW\":tfa.optimizers,\n    \"SWA\":tfa.optimizers,\n    \"Yogi\":tfa.optimizers,\n    \"Adadelta\":tf.keras.optimizers,\n    \"Adagrad\":tf.keras.optimizers,\n    \"Adam\":tf.keras.optimizers,\n    \"Adamax\":tf.keras.optimizers,\n    \"Ftrl\":tf.keras.optimizers,\n    \"Nadam\":tf.keras.optimizers,\n    \"RMSprop\":tf.keras.optimizers,\n    \"SGD\":tf.keras.optimizers\n       }\n\n# Supported Learning rates schedules\nschedules_mapper={\n    \"CyclicalLearningRate\":tfa.optimizers,\n    \"ExponentialCyclicalLearningRate\":tfa.optimizers,\n    \"Triangular2CyclicalLearningRate\":tfa.optimizers,\n    \"TriangularCyclicalLearningRate\":tfa.optimizers,\n    \"ExponentialDecay\":tf.keras.optimizers.schedules,\n    \"InverseTimeDecay\":tf.keras.optimizers.schedules,\n    \"PiecewiseConstantDecay\":tf.keras.optimizers.schedules,\n    \"PolynomialDecay\":tf.keras.optimizers.schedules,\n    \"CosineDecay\":tf.keras.experimental,\n    \"CosineDecayRestarts\":tf.keras.experimental,\n    \"LinearCosineDecay\":tf.keras.experimental,\n    \"NoisyLinearCosineDecay\":tf.keras.experimental,\n    \n       }","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:03.797636Z","iopub.status.busy":"2020-08-20T13:11:03.796851Z","iopub.status.idle":"2020-08-20T13:11:03.809048Z","shell.execute_reply":"2020-08-20T13:11:03.80851Z"},"papermill":{"duration":0.027299,"end_time":"2020-08-20T13:11:03.809146","exception":false,"start_time":"2020-08-20T13:11:03.781847","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv') ","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:03.839331Z","iopub.status.busy":"2020-08-20T13:11:03.838573Z","iopub.status.idle":"2020-08-20T13:11:03.851608Z","shell.execute_reply":"2020-08-20T13:11:03.850982Z"},"papermill":{"duration":0.035288,"end_time":"2020-08-20T13:11:03.851713","exception":false,"start_time":"2020-08-20T13:11:03.816425","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:03.873351Z","iopub.status.busy":"2020-08-20T13:11:03.872738Z","iopub.status.idle":"2020-08-20T13:11:03.879439Z","shell.execute_reply":"2020-08-20T13:11:03.878952Z"},"papermill":{"duration":0.020138,"end_time":"2020-08-20T13:11:03.879533","exception":false,"start_time":"2020-08-20T13:11:03.859395","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train.SmokingStatus.unique()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:03.905438Z","iopub.status.busy":"2020-08-20T13:11:03.903861Z","iopub.status.idle":"2020-08-20T13:11:03.906306Z","shell.execute_reply":"2020-08-20T13:11:03.906759Z"},"papermill":{"duration":0.01946,"end_time":"2020-08-20T13:11:03.906908","exception":false,"start_time":"2020-08-20T13:11:03.887448","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_tab(df):\n    vector = [(df.Age.values[0] - 30) / 30] \n    \n    if df.Sex.values[0] == 'male':\n       vector.append(0)\n    else:\n       vector.append(1)\n    \n    if df.SmokingStatus.values[0] == 'Never smoked':\n        vector.extend([0,0])\n    elif df.SmokingStatus.values[0] == 'Ex-smoker':\n        vector.extend([1,1])\n    elif df.SmokingStatus.values[0] == 'Currently smokes':\n        vector.extend([0,1])\n    else:\n        vector.extend([1,0])\n    return np.array(vector) ","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:03.9316Z","iopub.status.busy":"2020-08-20T13:11:03.930755Z","iopub.status.idle":"2020-08-20T13:11:04.243675Z","shell.execute_reply":"2020-08-20T13:11:04.243093Z"},"papermill":{"duration":0.329148,"end_time":"2020-08-20T13:11:04.243842","exception":false,"start_time":"2020-08-20T13:11:03.914694","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"A = {} \nTAB = {} \nP = [] \nfor i, p in tqdm(enumerate(train.Patient.unique())):\n    sub = train.loc[train.Patient == p, :] \n    fvc = sub.FVC.values\n    weeks = sub.Weeks.values\n    c = np.vstack([weeks, np.ones(len(weeks))]).T\n    a, b = np.linalg.lstsq(c, fvc)[0]\n    \n    A[p] = a\n    TAB[p] = get_tab(sub)\n    P.append(p)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:04.266426Z","iopub.status.busy":"2020-08-20T13:11:04.265843Z","iopub.status.idle":"2020-08-20T13:11:04.269515Z","shell.execute_reply":"2020-08-20T13:11:04.26999Z"},"papermill":{"duration":0.017061,"end_time":"2020-08-20T13:11:04.270109","exception":false,"start_time":"2020-08-20T13:11:04.253048","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_img(path):\n    d = pydicom.dcmread(path)\n    return cv2.resize((d.pixel_array - d.RescaleIntercept) / (d.RescaleSlope * 1000), (IM_SIZE, IM_SIZE))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:11:04.312319Z","iopub.status.busy":"2020-08-20T13:11:04.292846Z","iopub.status.idle":"2020-08-20T13:15:13.311411Z","shell.execute_reply":"2020-08-20T13:15:13.308024Z"},"papermill":{"duration":249.033369,"end_time":"2020-08-20T13:15:13.311573","exception":false,"start_time":"2020-08-20T13:11:04.278204","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"x, y = [], []\nfor p in tqdm(train.Patient.unique()):\n    try:\n        ldir = os.listdir(f'../input/osic-pulmonary-fibrosis-progression-lungs-mask/mask_noise/mask_noise/{p}/')\n        numb = [float(i[:-4]) for i in ldir]\n        for i in ldir:\n            x.append(cv2.imread(f'../input/osic-pulmonary-fibrosis-progression-lungs-mask/mask_noise/mask_noise/{p}/{i}', 0).mean())\n            y.append(float(i[:-4]) / max(numb))\n    except:\n        pass","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:15:13.345343Z","iopub.status.busy":"2020-08-20T13:15:13.341014Z","iopub.status.idle":"2020-08-20T13:15:13.347919Z","shell.execute_reply":"2020-08-20T13:15:13.348389Z"},"papermill":{"duration":0.028408,"end_time":"2020-08-20T13:15:13.348505","exception":false,"start_time":"2020-08-20T13:15:13.320097","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class IGenerator(Sequence):\n    BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618']\n    def __init__(self, keys, a, tab, batch_size=BATCH_SIZE):\n        self.keys = [k for k in keys if k not in self.BAD_ID]\n        self.a = a\n        self.tab = tab\n        self.batch_size = batch_size\n        \n        self.train_data = {}\n        for p in train.Patient.values:\n            self.train_data[p] = os.listdir(f'../input/osic-pulmonary-fibrosis-progression/train/{p}/')\n    \n    def __len__(self):\n        return 1000\n    \n    def __getitem__(self, idx):\n        x = []\n        a, tab = [], [] \n        keys = np.random.choice(self.keys, size = self.batch_size)\n        for k in keys:\n            try:\n                i = np.random.choice(self.train_data[k], size=1)[0]\n                img = get_img(f'../input/osic-pulmonary-fibrosis-progression/train/{k}/{i}')\n                x.append(img)\n                a.append(self.a[k])\n                tab.append(self.tab[k])\n            except:\n                print(k, i)\n       \n        x,a,tab = np.array(x), np.array(a), np.array(tab)\n        x = np.expand_dims(x, axis=-1)\n        return [x, tab] , a","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:15:13.374062Z","iopub.status.busy":"2020-08-20T13:15:13.373386Z","iopub.status.idle":"2020-08-20T13:15:13.377306Z","shell.execute_reply":"2020-08-20T13:15:13.376812Z"},"papermill":{"duration":0.020256,"end_time":"2020-08-20T13:15:13.377421","exception":false,"start_time":"2020-08-20T13:15:13.357165","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def build_model(base_model=BASE_MODEL,**kwargs):\n\n    inp = Input(shape=kwargs['input_shape'])\n    del kwargs['input_shape']\n    base =  getattr(tfka, base_model)(**kwargs) \n    x = base(inp)\n    x = GlobalAveragePooling2D()(x)\n    inp2 = Input(shape=(4,))\n    x2 = tf.keras.layers.GaussianNoise(0.2)(inp2)\n    x = Concatenate()([x, x2]) \n    x = Dropout(0.5)(x)\n    \n    # Explicit f32 because of mixed precision \n    x = Dense(1,dtype='float32')(x)\n    \n    model = Model([inp, inp2] , x)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training","execution_count":null},{"metadata":{"execution":{"iopub.execute_input":"2020-08-20T13:15:13.408437Z","iopub.status.busy":"2020-08-20T13:15:13.407625Z","iopub.status.idle":"2020-08-20T14:38:47.301073Z","shell.execute_reply":"2020-08-20T14:38:47.300505Z"},"papermill":{"duration":5013.916312,"end_time":"2020-08-20T14:38:47.302146","exception":false,"start_time":"2020-08-20T13:15:13.385834","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"\nif GPU:\n    config = tf.compat.v1.ConfigProto()\n    config.gpu_options.allow_growth = True\n    session = tf.compat.v1.Session(config=config)\n    if MIXED_PRECISION:\n      policy = mixed_precision.Policy('mixed_float16')\n      \n      mixed_precision.set_policy(policy)\n\n        \nkf = KFold(n_splits=NFOLD, random_state=42,shuffle=False)\nP = np.array(P)\nsubs = []\nfolds_history = []\nwith strategy.scope():\n\n    for fold, (tr_idx, val_idx) in enumerate(kf.split(P)):\n        print('#####################')\n        print('####### Fold %i ######'%fold)\n        print('#####################')\n        \n        print('Training...')\n\n        er = tf.keras.callbacks.EarlyStopping(\n            monitor=\"val_loss\",\n            min_delta=1e-3,\n            patience=EALRY_STOPPING_PATIENCE,\n            verbose=1,\n            mode=\"auto\",\n            baseline=None,\n            restore_best_weights=True,\n        )\n\n        cpt = tf.keras.callbacks.ModelCheckpoint(\n            filepath=f'{OUTPUT_DIR}fold-%i.h5'%fold,\n            monitor='val_loss', \n            verbose=1, \n            save_best_only=SAVE_BEST,\n            mode='auto'\n        )\n        \n        lrs=getattr(schedules_mapper[LR_STRATEGY], LR_STRATEGY)(**LR_KWARGS)\n        lrs_call_back=tf.keras.callbacks.LearningRateScheduler(lrs)\n        callbacks =[lrs_call_back,cpt,er]\n\n        optimizer=getattr(optimizers_mapper[OPTIMIZER], OPTIMIZER)(**OPTIMIZER_KWARGS)\n        #Loss scaling for GPU\n        if MIXED_PRECISION:\n            optimizer = mixed_precision.LossScaleOptimizer(optimizer, loss_scale='dynamic')\n\n        model = build_model(**BASE_MODEL_KWARGS)\n        model.compile(optimizer=optimizer, loss=\"mae\") \n        history = model.fit(IGenerator(keys=P[tr_idx], \n                                       a = A, \n                                       tab = TAB), \n                            steps_per_epoch = TRAIN_STEPS,\n                            validation_data=IGenerator(keys=P[val_idx], \n                                       a = A, \n                                       tab = TAB),\n                            validation_steps = VAL_STEPS, \n                            callbacks = callbacks, \n                            epochs=EPOCHS)\n        folds_history.append(history.history)\n        print('Training done!')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CV Evaluation","execution_count":null},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-08-20T14:38:48.311214Z","iopub.status.busy":"2020-08-20T14:38:48.310434Z","iopub.status.idle":"2020-08-20T14:38:48.315566Z","shell.execute_reply":"2020-08-20T14:38:48.315096Z"},"papermill":{"duration":0.548975,"end_time":"2020-08-20T14:38:48.315673","exception":false,"start_time":"2020-08-20T14:38:47.766698","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"if SAVE_BEST:\n    mean_val_loss = np.mean([np.min(h['val_loss']) for h in folds_history])\nelse:\n    mean_val_loss = np.mean([h['val_loss'][-1] for h in folds_history])\nprint('Our mean CV MAE is: ' + str(mean_val_loss))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.465078,"end_time":"2020-08-20T14:38:49.245316","exception":false,"start_time":"2020-08-20T14:38:48.780238","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n# Future Work\n\nPlease suggest in the comments\n\n# References\n\n[1] Michael Kazachok's Linear Decay (based on ResNet CNN)\n     Model that uses images can be found at: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn\n     \n[2] Ulrich GOUE's Osic-Multiple-Quantile-Regression-Starter\n     Model that uses tabular data can be found at: https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter\n\n[3] Wei Hao Khoong's K-Fold TF-EfficientNet Models (Training) https://www.kaggle.com/khoongweihao/k-fold-tf-efficientnet-models-training\n\n[4] Jeremy Howard's AMAZING Fast.ai course https://docs.fast.ai/\n\n[5] Yixing Fu's awesome guide \"Image classification via fine-tuning with EfficientNet\" https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}