{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# SIIM Covid-19 Detection Basic Exploratory Data Analysis (EDA)\nThis notebook provides a basic exploratory data analysis of the SIIM Covid-19 Detection data set. \n\n# Summary of Findings\n## Training Set\n- The dataset contains 6054 annotated training studies.\n- Each study has one of 4 labels.\n- Three labels indicates Covid-19 positive, 1 label Covid-10 negative.\n- Study labels are not balanced.\n- Each study contains between 1 and 9 training images with most studies constisting of 1 or 2 training images.\n- The dataset contains 6334 annotated training images.\n- Each training images has between 0 and 8 bounding boxes with most images having 0 to 2 bounding boxes.\n- **The dataset contains several duplicate images with different boudning boxes marked.**\n- The image height and width varies roughly between 1500 and 4000 pixels.\n- The width to height ratio of the images is approximately 1.2\n\n## Test Set\n- The dataset contains 1214 training studies.\n- Each study contains between 1 and 7 images with most studies containing 1 or 2 images.\n- The dataset contains 1263 images.\n- **The dataset contains several duplicate images.**\n- The image height and width varies roughly between 1500 and 4000 pixels.\n- The width to height ratio of the images is approximately 1.2\n\n# References\nThe following references were used in this notebook.\n- Notebook with all duplicate images: https://www.kaggle.com/kwk100/siim-covid-19-duplicate-training-images\n- Reading dicom images: https://www.kaggle.com/trungthanhnguyen0502/eda-vinbigdata-chest-x-ray-abnormalities\n ","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"!conda install gdcm -c conda-forge -y","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-06-07T03:14:36.58243Z","iopub.execute_input":"2021-06-07T03:14:36.582855Z","iopub.status.idle":"2021-06-07T03:15:39.850398Z","shell.execute_reply.started":"2021-06-07T03:14:36.582771Z","shell.execute_reply":"2021-06-07T03:15:39.849366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2\nimport datetime\nimport gc\nimport glob\nimport imagehash\nimport math\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport os\nimport pandas as pd\nimport PIL\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport sys\nimport tqdm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-07T03:15:39.852069Z","iopub.execute_input":"2021-06-07T03:15:39.85235Z","iopub.status.idle":"2021-06-07T03:15:40.293921Z","shell.execute_reply.started":"2021-06-07T03:15:39.852315Z","shell.execute_reply":"2021-06-07T03:15:40.292805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parameters","metadata":{}},{"cell_type":"code","source":"base_path = '../input/siim-covid19-detection'","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:15:40.295543Z","iopub.execute_input":"2021-06-07T03:15:40.295843Z","iopub.status.idle":"2021-06-07T03:15:40.300832Z","shell.execute_reply.started":"2021-06-07T03:15:40.295815Z","shell.execute_reply":"2021-06-07T03:15:40.299574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utility Functions","metadata":{}},{"cell_type":"code","source":"def read_dicom_image(image_file, voi_lut=True, fix_monochrome=True):\n    \"\"\"\n    Reads a dicom image from a file an returns a numpy array.\n    References: https://www.kaggle.com/trungthanhnguyen0502/eda-vinbigdata-chest-x-ray-abnormalities\n    Args:\n        image_file:\n        voi_lut:\n        fix_monochrome:\n\n    Returns:\n\n    \"\"\"\n    dicom = pydicom.read_file(image_file)\n    # VOI LUT (if available by DICOM device) is used to\n    # transform raw DICOM data to \"human-friendly\" view\n    if voi_lut:\n        data = apply_voi_lut(dicom.pixel_array, dicom)\n    else:\n        data = dicom.pixel_array\n    # depending on this value, X-ray may look inverted - fix that:\n    if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        data = np.amax(data) - data\n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data\n\n\ndef string2boxes(string):\n    strings = string.split()\n    if strings[0].lower() == 'none':\n        return []\n    else:\n        return [{'class': strings[idx],\n                 'conf': float(strings[idx+1]),\n                 'x1': float(strings[idx+2]),\n                 'y1': float(strings[idx+3]),\n                 'x2': float(strings[idx+4]),\n                 'y2': float(strings[idx+5]),\n                 } for idx in range(0, len(strings), 6)]\n\n    \ndef plot_image(image, boxes=None, size=(5,5), title=None, columns=4):\n    def plot_img(image, boxes=None, title=None):\n        if isinstance(image, str):\n            image_id = os.path.splitext(os.path.split(image)[1])[0]\n            df = df_image.loc[df_image['id'] == image_id + '_image']\n            boxes = string2boxes(df['label'].iloc[0]) if len(df) > 0 else None\n            image = read_dicom_image(image)\n        image = np.stack([image] * 3, axis=-1)\n        if boxes is not None:\n            for box in boxes:\n                image = cv2.rectangle(image, (int(box['x1']), int(box['y1'])), (int(box['x2']), int(box['y2'])), [0, 255, 0], 10)\n        plt.axis('on')\n        plt.imshow(image, cmap='gray')\n        if title is not None:\n            plt.title(title)\n\n    plt.figure(figsize=size)\n    if isinstance(image, list):\n        num = len(image)\n        columns = min(columns, num)\n        rows = math.ceil(num / columns)\n\n        for index, single_image in enumerate(image):\n            plt.subplot(rows, columns, index + 1)\n            plot_img(single_image, boxes=boxes, title=None if title is None else title[index])\n    else:\n        plot_img(image, boxes=boxes, title=title)\n    plt.show()\n\n\ndef images_find_duplicates(image_files, threshold=0.9):\n    \"\"\"\n    Function to find duplicates in images.\n    References: https://www.kaggle.com/appian/let-s-find-out-duplicate-images-with-imagehash\n    Args:\n        image_files:\n        threshold:\n\n    Returns:\n\n    \"\"\"\n    funcs = [imagehash.average_hash, imagehash.phash, imagehash.dhash, imagehash.whash]\n    image_ids = image_files\n    hashes = []\n    for file in tqdm.tqdm(image_files):\n        image = PIL.Image.fromarray(read_dicom_image(file))\n        hashes.append(np.array([f(image).hash for f in funcs]).reshape(256))\n    hashes_all = np.array(hashes)\n\n    # Comparisons without Pytorch\n    sim_list = []\n    for i in tqdm.tqdm(range(hashes_all.shape[0])):\n        sim_list.append(np.sum(hashes_all[i] == hashes_all, axis=1)/256)\n\n    # nxn-matrix of similarities (n = # of images), upper triangular matrix\n    similarities = np.triu(np.array(sim_list), 1)\n\n    idx_pair = np.where(similarities > threshold)\n    df_pairs = pd.DataFrame({'image1': [image_ids[i] for i in list(idx_pair[0])],\n                             'image2': [image_ids[i] for i in list(idx_pair[1])],\n                             'similarity': [similarities[i1, i2] for i1, i2 in zip(idx_pair[0], idx_pair[1])]})\n\n    idx_group = np.zeros(len(image_files))\n    group_id = 1\n    for i1, i2 in zip(idx_pair[0], idx_pair[1]):\n        if idx_group[i1] == 0 and idx_group[i2] == 0:\n            idx_group[i1] = group_id\n            idx_group[i2] = group_id\n            group_id += 1\n        elif idx_group[i1] != 0 and idx_group[i2] == 0:\n            idx_group[i2] = idx_group[i1]\n        elif idx_group[i1] == 0 and idx_group[i2] != 0:\n            idx_group[i1] = idx_group[i2]\n        elif idx_group[i1] != 0 and idx_group[i2] != 0 and idx_group[i1] != idx_group[i2]:\n            common_id = min(idx_group[i1], idx_group[i2])\n            idx_group[idx_group == idx_group[i1]] = common_id\n            idx_group[idx_group == idx_group[i2]] = common_id\n\n    group_list = []\n    for i in range(1, group_id + 1):\n        group_ids = list(np.where(idx_group == i)[0])\n        if len(group_ids) > 0:\n            group_list.append([image_ids[j] for j in group_ids])\n\n    return df_pairs, group_list","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-06-07T03:15:40.303117Z","iopub.execute_input":"2021-06-07T03:15:40.303571Z","iopub.status.idle":"2021-06-07T03:15:40.336545Z","shell.execute_reply.started":"2021-06-07T03:15:40.303527Z","shell.execute_reply":"2021-06-07T03:15:40.335494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# File Structure\nThe files in the root of the dataset are shown below.\nThe dataset consists of 2 directories that contain training and test images and 3 csv-files with additional information about the images and the competition.\n## Directory Contents","metadata":{}},{"cell_type":"code","source":"print('Directories:')\nprint('\\n'.join([dir for dir in os.listdir(base_path) if os.path.isdir(os.path.join(base_path, dir))]))\nprint('\\nFiles:')\nprint('\\n'.join([dir for dir in os.listdir(base_path) if not os.path.isdir(os.path.join(base_path, dir))]))","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:15:40.337888Z","iopub.execute_input":"2021-06-07T03:15:40.338293Z","iopub.status.idle":"2021-06-07T03:15:40.358572Z","shell.execute_reply.started":"2021-06-07T03:15:40.338249Z","shell.execute_reply":"2021-06-07T03:15:40.357597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CSV Files\n# Train_Study_Level.csv\nLets take a look at the contents of the study training file.","metadata":{}},{"cell_type":"code","source":"df_study = pd.read_csv(os.path.join(base_path, 'train_study_level.csv'))\nprint(f'Number of rows: {len(df_study)}')\ndisplay(df_study)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:15:40.35969Z","iopub.execute_input":"2021-06-07T03:15:40.360126Z","iopub.status.idle":"2021-06-07T03:15:40.407236Z","shell.execute_reply.started":"2021-06-07T03:15:40.360091Z","shell.execute_reply":"2021-06-07T03:15:40.406239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_study['Num Labels'] = df_study.iloc[:,1:].sum(axis=1)\nprint(f'Minimum number of labels per row: {min(df_study[\"Num Labels\"])}')\nprint(f'Maximum number of labels per row: {max(df_study[\"Num Labels\"])}')\nprint(f'Number of unique ids: {len(df_study[\"id\"].unique())}')","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:15:40.408346Z","iopub.execute_input":"2021-06-07T03:15:40.408647Z","iopub.status.idle":"2021-06-07T03:15:40.42336Z","shell.execute_reply.started":"2021-06-07T03:15:40.408607Z","shell.execute_reply":"2021-06-07T03:15:40.422173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In each row, exactly one of the 4 class is associated with the study id.\n\nSince the number of unique studies is the same as the number of rows, so each study occurs only once in the table and each study is associated with only 1 class.\n\nThe frequency of the 4 classes in the training set is shown below.","metadata":{"execution":{"iopub.status.busy":"2021-05-20T01:43:53.526429Z","iopub.execute_input":"2021-05-20T01:43:53.526819Z","iopub.status.idle":"2021-05-20T01:43:53.536937Z","shell.execute_reply.started":"2021-05-20T01:43:53.526782Z","shell.execute_reply":"2021-05-20T01:43:53.53562Z"}}},{"cell_type":"code","source":"plt.figure(figsize=(10,5))\nplt.bar([1,2,3,4], df_study.iloc[:,1:5].sum(axis=0), tick_label=df_study.columns[1:5])\nplt.xlabel('Class')\nplt.ylabel('Frequency')\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:15:40.425339Z","iopub.execute_input":"2021-06-07T03:15:40.42587Z","iopub.status.idle":"2021-06-07T03:15:40.592879Z","shell.execute_reply.started":"2021-06-07T03:15:40.425824Z","shell.execute_reply":"2021-06-07T03:15:40.592122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train_Image_Level.csv\nLets take a look at the contents of the image training file.","metadata":{}},{"cell_type":"code","source":"df_image = pd.read_csv(os.path.join(base_path, 'train_image_level.csv'))\nprint(f'Number of rows: {len(df_image)}')\ndisplay(df_image)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:15:40.594384Z","iopub.execute_input":"2021-06-07T03:15:40.594978Z","iopub.status.idle":"2021-06-07T03:15:40.654199Z","shell.execute_reply.started":"2021-06-07T03:15:40.594926Z","shell.execute_reply":"2021-06-07T03:15:40.653267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The `label` column contains the bounding boxes for the image `bounding box 1`, `bounding box 2`, ...\n  - Each bounding box is given in the format `class` `confidence score` `x1` `y1` `x2` `y2`\n- The `boxes` column contains the x, y, width, and height information only in dictionary format.","metadata":{}},{"cell_type":"markdown","source":"# Training Images\nThe train images are arranged in directories and sub-directories with the structure `study`/`series`/`image`.\n\n## Number of Studies\n\nEach study contains between 1 and 9 series, with most studies containing 1 series.","metadata":{}},{"cell_type":"code","source":"dir_studies = glob.glob(os.path.join(base_path, 'train/*'))\nprint(f'Number of studies: {len(dir_studies)}')\ndir_series = [glob.glob(dir_study + '/*') for dir_study in dir_studies]\ncount_series = pd.Series([len(dir_ser) for dir_ser in dir_series])\n_ = plt.hist(count_series)\nplt.title('Number of series per study')\nplt.xlabel('Number of series per study')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()\nprint(count_series.value_counts().sort_index().to_string())","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:15:40.655491Z","iopub.execute_input":"2021-06-07T03:15:40.655792Z","iopub.status.idle":"2021-06-07T03:15:50.186722Z","shell.execute_reply.started":"2021-06-07T03:15:40.655762Z","shell.execute_reply":"2021-06-07T03:15:50.185666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Number of Images\nSimilarly, each study contains between 1 and 9 images with most studies consisting of 1 image.","metadata":{}},{"cell_type":"code","source":"dir_images = [glob.glob(dir_study + '/*/*.dcm') for dir_study in dir_studies]\ndf_study_train = pd.DataFrame({'study_id': [s.split('/')[-1] for s in dir_studies]})\ndf_study_train['num_images'] = [len(dir_img) for dir_img in dir_images]\n_ = plt.hist(df_study_train['num_images'])\nplt.title('Number of images per study')\nplt.xlabel('Number of images per study')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()\nprint(df_study_train['num_images'].value_counts().sort_index().to_string() + '\\n')\nprint('Studies with the most images')\ndisplay(df_study_train.sort_values('num_images', ascending=False)[0:10])\ndf_study_train.to_csv('study_train.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:47:35.548382Z","iopub.execute_input":"2021-06-07T03:47:35.548801Z","iopub.status.idle":"2021-06-07T03:47:36.447693Z","shell.execute_reply.started":"2021-06-07T03:47:35.548765Z","shell.execute_reply":"2021-06-07T03:47:36.446894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image Display\nThe next step is to load the list of training files.","metadata":{}},{"cell_type":"code","source":"train_files = sorted(glob.glob(os.path.join(base_path, 'train/*/*/*.dcm')))\nprint(f'Number of training files: {len(train_files)}')","metadata":{"execution":{"iopub.status.busy":"2021-06-01T02:26:57.482192Z","iopub.execute_input":"2021-06-01T02:26:57.482479Z","iopub.status.idle":"2021-06-01T02:27:03.29352Z","shell.execute_reply.started":"2021-06-01T02:26:57.482454Z","shell.execute_reply":"2021-06-01T02:27:03.292576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that the list is loaded, lets take a look at one image full-size and several scaled down images with their bounding boxes.","metadata":{}},{"cell_type":"code","source":"plot_image(train_files[7], size=(20,20))\nplot_image(train_files[0:16], size=(20,20))","metadata":{"execution":{"iopub.status.busy":"2021-06-01T02:27:03.294549Z","iopub.execute_input":"2021-06-01T02:27:03.29483Z","iopub.status.idle":"2021-06-01T02:27:26.925557Z","shell.execute_reply.started":"2021-06-01T02:27:03.294803Z","shell.execute_reply":"2021-06-01T02:27:26.924563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Bounding Boxes\nThe number of bounding boxes in an image varies. From the data below, we can see that each image can contain between 0 and 8 bounding with most of the images having between 0 and 2 boxes.","metadata":{}},{"cell_type":"code","source":"df_image['box_dict'] = df_image['label'].apply(lambda x: string2boxes(x))\ndf_image['num_boxes'] = df_image['box_dict'].apply(lambda x: len(x))\noutput = plt.hist(df_image['num_boxes'])\nplt.xlabel('Number of bounding boxes')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()\nprint(df_image['num_boxes'].value_counts().sort_index().to_string())","metadata":{"execution":{"iopub.status.busy":"2021-06-01T02:27:26.926785Z","iopub.execute_input":"2021-06-01T02:27:26.927054Z","iopub.status.idle":"2021-06-01T02:27:27.13118Z","shell.execute_reply.started":"2021-06-01T02:27:26.927028Z","shell.execute_reply":"2021-06-01T02:27:27.130259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image Size","metadata":{}},{"cell_type":"code","source":"img_list = []\nfor file in tqdm.tqdm(train_files):\n    img = read_dicom_image(file)\n    img_list.append({'file': file, 'width': img.shape[1], 'height': img.shape[0]})\ndf_images = pd.DataFrame(img_list)\ndf_images['ratio'] = df_images['width'] / df_images['height']\nplt.figure(figsize=(18,5))\nplt.subplot(1,2,1)\nplt.hist(df_images['width'])\nplt.title('Width Distribution of the Images')\nplt.xlabel('Width')\nplt.ylabel('Frequency')\nplt.grid()\nplt.subplot(1,2,2)\nplt.hist(df_images['height'])\nplt.title('Height Distribution of the Images')\nplt.xlabel('Height')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()\nplt.figure(figsize=(18,5))\nplt.subplot(1,2,1)\nplt.scatter(df_images['width'], df_images['height'])\nplt.title('Width and Height Distribution of the Images')\nplt.xlabel('Width')\nplt.ylabel('Height')\nplt.grid()\nplt.subplot(1,2,2)\nplt.hist(df_images['ratio'], bins = 20)\nplt.title('Width to Height Ratio of the Images')\nplt.xlabel('Width to Height Ratio')\nplt.ylabel('Frequency')\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T02:27:27.132206Z","iopub.execute_input":"2021-06-01T02:27:27.132484Z","iopub.status.idle":"2021-06-01T03:00:38.211136Z","shell.execute_reply.started":"2021-06-01T02:27:27.132459Z","shell.execute_reply":"2021-06-01T03:00:38.209981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Duplicate Images\nIt is worthwile to check the dataset for duplicate images. For speed reasons, we are only checking the first 200 files for duplicates.","metadata":{}},{"cell_type":"code","source":"df_pairs, group_list = images_find_duplicates(train_files[0:200], threshold=0.95)\nprint(f'\\nNumber of duplicate pairs: {len(df_pairs)}')\nprint(f'Number of duplicate groups: {len(group_list)}')","metadata":{"execution":{"iopub.status.busy":"2021-06-01T03:17:29.108332Z","iopub.execute_input":"2021-06-01T03:17:29.108933Z","iopub.status.idle":"2021-06-01T03:21:32.564483Z","shell.execute_reply.started":"2021-06-01T03:17:29.108874Z","shell.execute_reply":"2021-06-01T03:21:32.563418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since several duplicate images were found in the dataset, we plot some of them. It turns out that in the duplicate images, the bounding boxes are different!","metadata":{}},{"cell_type":"code","source":"for i, group in enumerate(group_list):\n    group_ids = [os.path.basename(file) for file in group]\n    print(f'\\nGroup {i+1}')\n    plot_image(group, size=(20, 10), title=group_ids, columns=8)","metadata":{"execution":{"iopub.status.busy":"2021-06-01T03:21:32.56599Z","iopub.execute_input":"2021-06-01T03:21:32.566274Z","iopub.status.idle":"2021-06-01T03:21:41.028841Z","shell.execute_reply.started":"2021-06-01T03:21:32.566245Z","shell.execute_reply":"2021-06-01T03:21:41.028015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test Images\nThe test images are arranged in directories and sub-directories with the structure `study`/`series`/`image`.\n\n## Number of Studies\n\nEach study contains between 1 and 4 series, with most studies containing 1 series.","metadata":{}},{"cell_type":"code","source":"dir_studies = glob.glob(os.path.join(base_path, 'test/*'))\nprint(f'Number of studies: {len(dir_studies)}')\ndir_series = [glob.glob(dir_study + '/*') for dir_study in dir_studies]\ncount_series = pd.Series([len(dir_ser) for dir_ser in dir_series])\n_ = plt.hist(count_series)\nplt.title('Number of series per study')\nplt.xlabel('Number of series per study')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()\nprint(count_series.value_counts().sort_index().to_string())","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:45:32.177532Z","iopub.execute_input":"2021-06-07T03:45:32.178106Z","iopub.status.idle":"2021-06-07T03:45:33.889916Z","shell.execute_reply.started":"2021-06-07T03:45:32.178069Z","shell.execute_reply":"2021-06-07T03:45:33.888909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Number of Images\nSimilarly, each study contains between 1 and 7 images with most studies consisting of 1 image.","metadata":{}},{"cell_type":"code","source":"dir_images = [glob.glob(dir_study + '/*/*.dcm') for dir_study in dir_studies]\ndf_study_test = pd.DataFrame({'study_id': [s.split('/')[-1] for s in dir_studies]})\ndf_study_test['num_images'] = [len(dir_img) for dir_img in dir_images]\n_ = plt.hist(df_study_test['num_images'])\nplt.title('Number of images per study')\nplt.xlabel('Number of images per study')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()\nprint(df_study_test['num_images'].value_counts().sort_index().to_string() + '\\n')\nprint('Studies with the most images')\ndisplay(df_study_test.sort_values('num_images', ascending=False)[0:10])\ndf_study_test.to_csv('study_test.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T03:48:53.478661Z","iopub.execute_input":"2021-06-07T03:48:53.479158Z","iopub.status.idle":"2021-06-07T03:48:54.32576Z","shell.execute_reply.started":"2021-06-07T03:48:53.479127Z","shell.execute_reply":"2021-06-07T03:48:54.325012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image Display\nThe next step is to load the list of test files.","metadata":{}},{"cell_type":"code","source":"test_files = sorted(glob.glob(os.path.join(base_path, 'test/*/*/*.dcm')))\nprint(f'Number of test files: {len(test_files)}')","metadata":{"execution":{"iopub.status.busy":"2021-06-01T03:21:46.946623Z","iopub.execute_input":"2021-06-01T03:21:46.946911Z","iopub.status.idle":"2021-06-01T03:21:48.221835Z","shell.execute_reply.started":"2021-06-01T03:21:46.946884Z","shell.execute_reply":"2021-06-01T03:21:48.220835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that the list is loaded, lets take a look at one image full-size and several scaled down images.","metadata":{}},{"cell_type":"code","source":"plot_image(test_files[7], size=(20,20))\nplot_image(test_files[0:16], size=(20,20))","metadata":{"execution":{"iopub.status.busy":"2021-06-01T03:21:48.223048Z","iopub.execute_input":"2021-06-01T03:21:48.223346Z","iopub.status.idle":"2021-06-01T03:22:12.920655Z","shell.execute_reply.started":"2021-06-01T03:21:48.223316Z","shell.execute_reply":"2021-06-01T03:22:12.919556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image Size","metadata":{}},{"cell_type":"code","source":"img_list = []\nfor file in tqdm.tqdm(test_files):\n    img = read_dicom_image(file)\n    img_list.append({'file': file, 'width': img.shape[1], 'height': img.shape[0]})\ndf_images = pd.DataFrame(img_list)\ndf_images['ratio'] = df_images['width'] / df_images['height']\nplt.figure(figsize=(18,5))\nplt.subplot(1,2,1)\nplt.hist(df_images['width'])\nplt.title('Width Distribution of the Images')\nplt.xlabel('Width')\nplt.ylabel('Frequency')\nplt.grid()\nplt.subplot(1,2,2)\nplt.hist(df_images['height'])\nplt.title('Height Distribution of the Images')\nplt.xlabel('Height')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()\nplt.figure(figsize=(18,5))\nplt.subplot(1,2,1)\nplt.scatter(df_images['width'], df_images['height'])\nplt.title('Width and Height Distribution of the Images')\nplt.xlabel('Width')\nplt.ylabel('Height')\nplt.grid()\nplt.subplot(1,2,2)\nplt.hist(df_images['ratio'], bins = 20)\nplt.title('Width to Height Ratio of the Images')\nplt.xlabel('Width to Height Ratio')\nplt.ylabel('Frequency')\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-01T03:22:12.922306Z","iopub.execute_input":"2021-06-01T03:22:12.922721Z","iopub.status.idle":"2021-06-01T03:29:14.461114Z","shell.execute_reply.started":"2021-06-01T03:22:12.92266Z","shell.execute_reply":"2021-06-01T03:29:14.460418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Duplicate Images\nWe also check the test images for duplicates. For speed reasons, we are only checking the first 200 files for duplicates.","metadata":{}},{"cell_type":"code","source":"df_pairs, group_list = images_find_duplicates(test_files[0:200], threshold=0.95)\nprint(f'\\nNumber of duplicate pairs: {len(df_pairs)}')\nprint(f'Number of duplicate groups: {len(group_list)}')","metadata":{"execution":{"iopub.status.busy":"2021-06-01T03:29:14.462838Z","iopub.execute_input":"2021-06-01T03:29:14.463284Z","iopub.status.idle":"2021-06-01T03:33:25.323363Z","shell.execute_reply.started":"2021-06-01T03:29:14.463243Z","shell.execute_reply":"2021-06-01T03:33:25.322319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since several duplicate images were found in the dataset, we plot some of them.","metadata":{}},{"cell_type":"code","source":"for i, group in enumerate(group_list):\n    group_ids = [os.path.basename(file) for file in group]\n    print(f'\\nGroup {i+1}')\n    plot_image(group, size=(20, 10), title=group_ids, columns=8)","metadata":{"execution":{"iopub.status.busy":"2021-06-01T03:33:25.32511Z","iopub.execute_input":"2021-06-01T03:33:25.325517Z","iopub.status.idle":"2021-06-01T03:33:28.469895Z","shell.execute_reply.started":"2021-06-01T03:33:25.325473Z","shell.execute_reply":"2021-06-01T03:33:28.468878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sample_Submission.csv\nThe `sample_submission.csv` file shows the format of the submissions file. consisting of the test image id and an rle encoded masks. The submission file contains both the study predictions and the image predictions using a common format with two columns.\n- `id`: Id of the study/image followed by '_study' or '_image'\n- `PredictionString`: A single string for the prediction for the study/image.\n  - for studies: This is the predicted class followed by a confidence score and a one-pixel bounding box '0 0 1 1'\n  - for images: class ID ('opacity'/'none'), confidence score, bounding box for each detected object.","metadata":{}},{"cell_type":"code","source":"df_submission = pd.read_csv(os.path.join(base_path,'sample_submission.csv'))\ndisplay(df_submission)","metadata":{"execution":{"iopub.status.busy":"2021-05-25T03:49:35.111069Z","iopub.execute_input":"2021-05-25T03:49:35.111364Z","iopub.status.idle":"2021-05-25T03:49:35.132502Z","shell.execute_reply.started":"2021-05-25T03:49:35.111334Z","shell.execute_reply":"2021-05-25T03:49:35.131323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}