{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Hello!\n\n**[NIH Chest X-rays](https://www.kaggle.com/nih-chest-xrays/data)**, one of the largest X-ray images dataset available, is currently in the spotlight of the ongoing **[RANZCR CLiP Competition](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification)** as its entire train and test data is 100% relabeled data from the former. Therefore one should be careful to avoid overfitting thus making more harm than good from using this external data either for pretraining models or as an additional data source.\n\nThe point of my concern, however, is time and resources, as training single EfficientNetB4 on TPU with `TFRecordDataset` of `112,120` samples (and images downscaled to `600x600`) for 20 epochs must take over 2.5 hours. Skipping serialization and going with `from_tensor_slices` must be at least 3-4 times slower.","metadata":{}},{"cell_type":"markdown","source":"## Contents\n\n1. Preprocess the initial `Data_Entry_2017.csv` DataFrame to make it suitable for training a CNN (take the output `preprocessed_data.csv` for training if you decide to go with `from_tensor_slices`)\n2. Serialize the entire `112,120` samples to 256 `.tfrec` files (place a filter here, e.g. this **[duplicate reference](https://www.kaggle.com/mohamed3abdelrazik/paths-for-duplicated-images-on-chestx-and-ranczr/settings)** to delete duplicates before serialization)\n3. Test whether everything works fine by placing a train placeholder at the end (or train in this notebook by adding your train pipeline)","metadata":{}},{"cell_type":"markdown","source":"### Imports\nI do not allocate TPU here as its usage requires results to be published to a dataset. To run on TPU, save your custom results as a dataset.","metadata":{}},{"cell_type":"code","source":"from sklearn.utils import shuffle\nfrom tqdm.notebook import tqdm\nimport tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport os","metadata":{"execution":{"iopub.status.busy":"2023-04-27T22:46:00.428292Z","iopub.execute_input":"2023-04-27T22:46:00.428677Z","iopub.status.idle":"2023-04-27T22:46:05.599531Z","shell.execute_reply.started":"2023-04-27T22:46:00.428591Z","shell.execute_reply":"2023-04-27T22:46:05.598726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STRATEGY = tf.distribute.get_strategy()    \nBATCH_SIZE = 16\nIMG_SIZE = 600\nSEED = 42\n    \nprint('Using tensorflow %s' % tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2023-04-27T22:46:47.427269Z","iopub.execute_input":"2023-04-27T22:46:47.427621Z","iopub.status.idle":"2023-04-27T22:46:47.438289Z","shell.execute_reply.started":"2023-04-27T22:46:47.427586Z","shell.execute_reply":"2023-04-27T22:46:47.437207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Preprocess the initial dataframe\nHere I completely disregard all the additional information provided in the initial DataFrame (e.g. age, sex, etc.), leaving only filename and diagnosis. Add those columns to the preprocessed DataFrame if you decide to keep them.","metadata":{}},{"cell_type":"code","source":"init_df = pd.read_csv('../input/data/Data_Entry_2017.csv', index_col='Image Index')\n\ndiseases = [\n    'No Finding',\n    'Atelectasis',\n    'Consolidation',\n    'Infiltration',\n    'Pneumothorax',\n    'Edema',\n    'Emphysema',\n    'Fibrosis',\n    'Effusion',\n    'Pneumonia',\n    'Pleural_Thickening',\n    'Cardiomegaly',\n    'Nodule',\n    'Mass',\n    'Hernia']\n\n\nfilenames = []\n\nfor i in range(12):\n    filenames.append(os.listdir('../input/data/images_%.3i/images' % (i + 1)))\n\n\ndef _parse_findings(raw):\n    raw = raw.split('|')\n    parsed = np.zeros((len(diseases),), dtype=np.bool)\n    for i in range(len(diseases)):\n        if diseases[i] in raw:\n            parsed[i] = 1\n    return parsed\n\n\ndef _find_image_path(image_id):\n    for i in range(12):\n        if image_id in filenames[i]:\n            path = os.path.join('../input/data/images_%.3i/images' % (i + 1), image_id)\n            return path\n\n\ndf = pd.DataFrame(\n    columns=diseases, \n    data=[_parse_findings(x) for x in init_df['Finding Labels']],\n    index=[_find_image_path(x) for x in tqdm(init_df.index, total=len(init_df))])\n\ndf.to_csv('preprocessed_data.csv')\ndisplay(df.head())","metadata":{"execution":{"iopub.status.busy":"2023-04-27T22:46:52.328279Z","iopub.execute_input":"2023-04-27T22:46:52.328847Z","iopub.status.idle":"2023-04-27T22:48:29.918398Z","shell.execute_reply.started":"2023-04-27T22:46:52.328791Z","shell.execute_reply":"2023-04-27T22:48:29.917574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Serialization\n### Serialization functions","metadata":{}},{"cell_type":"code","source":"def _serialize_image(path):\n    image = tf.io.read_file(path)\n    image = tf.image.decode_png(image, channels=1)\n    image = tf.image.resize(image, [IMG_SIZE, IMG_SIZE])\n    image = tf.cast(image, tf.uint8)\n    return tf.image.encode_jpeg(image).numpy()\n\n\ndef _serialize_sample(image_id, image, proba):\n    feature = {\n        'image': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image])),\n        'image_id': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image_id])),\n        'No Finding': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[0]])),\n        'Atelectasis': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[1]])),\n        'Consolidation': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[2]])),\n        'Infiltration': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[3]])),\n        'Pneumothorax': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[4]])),\n        'Edema': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[5]])),\n        'Emphysema': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[6]])),\n        'Fibrosis': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[7]])),\n        'Effusion': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[8]])),\n        'Pneumonia': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[9]])),\n        'Pleural_Thickening': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[10]])),\n        'Cardiomegaly': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[11]])),\n        'Nodule': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[12]])),\n        'Mass': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[13]])),\n        'Hernia': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[14]]))}\n    sample = tf.train.Example(features=tf.train.Features(feature=feature))\n    return sample.SerializeToString()\n        \n        \ndef serialize_fold(fold, name):\n    samples = []\n    \n    for index, proba in fold.iterrows():\n        samples.append(_serialize_sample(\n            index.split('/')[-1].encode(), \n            _serialize_image(index), \n            proba))\n    \n    with tf.io.TFRecordWriter(name + '.tfrec') as writer:\n        [writer.write(x) for x in samples]","metadata":{"execution":{"iopub.status.busy":"2023-04-27T23:08:26.938446Z","iopub.execute_input":"2023-04-27T23:08:26.938837Z","iopub.status.idle":"2023-04-27T23:08:26.954428Z","shell.execute_reply.started":"2023-04-27T23:08:26.938803Z","shell.execute_reply":"2023-04-27T23:08:26.95357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run serialization","metadata":{}},{"cell_type":"code","source":"df = shuffle(df, random_state=SEED)\nfolds = 256\n\nos.mkdir('./data')\n\nfor i, fold in tqdm(enumerate(np.array_split(df, folds)), total=folds):\n    serialize_fold(fold, name='./data/%.3i-%.3i' % (i, len(fold)))","metadata":{"execution":{"iopub.status.busy":"2023-04-27T23:08:36.328582Z","iopub.execute_input":"2023-04-27T23:08:36.328917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Train placeholder\nChange this to your train pipeline if you decide to train in this notebook.\n### Data workflow functions","metadata":{}},{"cell_type":"code","source":"feature_map = {\n    'image': tf.io.FixedLenFeature([], tf.string),\n    'image_id': tf.io.FixedLenFeature([], tf.string),\n    'No Finding': tf.io.FixedLenFeature([], tf.int64),\n    'Atelectasis': tf.io.FixedLenFeature([], tf.int64),\n    'Consolidation': tf.io.FixedLenFeature([], tf.int64),\n    'Infiltration': tf.io.FixedLenFeature([], tf.int64),\n    'Pneumothorax': tf.io.FixedLenFeature([], tf.int64),\n    'Edema': tf.io.FixedLenFeature([], tf.int64),\n    'Emphysema': tf.io.FixedLenFeature([], tf.int64),\n    'Fibrosis': tf.io.FixedLenFeature([], tf.int64),\n    'Effusion': tf.io.FixedLenFeature([], tf.int64),\n    'Pneumonia': tf.io.FixedLenFeature([], tf.int64),\n    'Pleural_Thickening': tf.io.FixedLenFeature([], tf.int64),\n    'Cardiomegaly': tf.io.FixedLenFeature([], tf.int64),\n    'Nodule': tf.io.FixedLenFeature([], tf.int64),\n    'Mass': tf.io.FixedLenFeature([], tf.int64),\n    'Hernia': tf.io.FixedLenFeature([], tf.int64)}\n\n\ndef count_data_items(filenames):\n    return np.sum([int(x[:-6].split('-')[-1]) for x in filenames])\n\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=1)\n    image = tf.reshape(image, [IMG_SIZE, IMG_SIZE, 1])\n    return image\n\n\ndef scale_image(image, target):\n    image = tf.cast(image, tf.float32) / 255.\n    return image, target\n\n\ndef read_tfrecord(example):\n    example = tf.io.parse_single_example(example, feature_map)\n    image = decode_image(example['image'])\n    target = [\n        example['No Finding'],\n        example['Atelectasis'],\n        example['Consolidation'],\n        example['Infiltration'],\n        example['Pneumothorax'],\n        example['Edema'],\n        example['Emphysema'],\n        example['Fibrosis'],\n        example['Effusion'],\n        example['Pneumonia'],\n        example['Pleural_Thickening'],\n        example['Cardiomegaly'],\n        example['Nodule'],\n        example['Mass'],\n        example['Hernia']]\n    return image, target\n\n\ndef data_augment(image, target):\n    image = tf.image.random_flip_left_right(image, seed=SEED)\n    image = tf.image.random_flip_up_down(image, seed=SEED)\n    return image, target\n\n\ndef get_dataset(filenames, shuffled=False, repeated=False, \n                cached=False, augmented=False, distributed=True):\n    auto = tf.data.experimental.AUTOTUNE\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=auto)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=auto)\n    if augmented:\n        dataset = dataset.map(data_augment, num_parallel_calls=auto)\n    dataset = dataset.map(scale_image, num_parallel_calls=auto)\n    if shuffled:\n        dataset = dataset.shuffle(2048, seed=SEED)\n    if repeated:\n        dataset = dataset.repeat()\n    dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)\n    if cached:\n        dataset = dataset.cache()\n    dataset = dataset.prefetch(auto)\n    if distributed:\n        dataset = STRATEGY.experimental_distribute_dataset(dataset)\n    return dataset\n\n\ndef get_model():\n    model = tf.keras.models.Sequential([\n        tf.keras.applications.EfficientNetB0(\n            include_top=False,\n            input_shape=(None, None, 1),\n            weights=None,\n            pooling='avg'),\n        tf.keras.layers.Dense(15, activation='sigmoid')\n    ])\n    model.compile(\n        optimizer='adam',\n        loss='binary_crossentropy',\n        metrics=tf.keras.metrics.AUC(multi_label=True))\n\n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run test","metadata":{}},{"cell_type":"code","source":"filenames = tf.io.gfile.glob('./data/*.tfrec')\n\ntrain_filenames = [filenames[0]]\nval_filenames = [filenames[1]]\n\nsteps_per_epoch = count_data_items(train_filenames) // BATCH_SIZE\nvalidation_steps = count_data_items(val_filenames) // BATCH_SIZE\n\ntrain_dataset = get_dataset(train_filenames, shuffled=True, repeated=True, augmented=True)\nval_dataset = get_dataset(val_filenames, cached=True)\n\nwith STRATEGY.scope():\n    model = get_model()\n    \nhistory = model.fit(\n    train_dataset,\n    steps_per_epoch=steps_per_epoch,\n    epochs=2,\n    validation_data=val_dataset,\n    validation_steps=validation_steps,\n    verbose=2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}