{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div class='alert alert-info' style='text-align: center'><h1>Standardizing chest x-ray dataset exports</h1>\n    - yet another chest x-ray processing notebook -\n</div>\n\n#### In this competition, the DICOMs come from multiple modalities, with different bit depths and various post-processing filters applied.\n#### This causes equalization/normalization anomalies to appear. Some images are much brighter .. some more blurred etc.\n\n- The goal here is to make the images look a little more similar across image types.\n- The important thing is to perform this before crunching the pixels down to 8 bit.\n\nHere's a notebook that exports the entire SIIM-FISABIO COVID-19 DICOM dataset with this processing applied -> https://www.kaggle.com/davidbroberts/export-processed-jpg-512\n\nThis is the dataset I created using the above notebook -> https://www.kaggle.com/davidbroberts/siimcovid-jpg-512-processed","metadata":{}},{"cell_type":"code","source":"# You may need to uncomment and run the conda install. Then restart the notebook if GDCM pukes.\n#!conda install gdcm -c conda-forge -y","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nfrom os import path\nimport pydicom\nfrom skimage.filters import unsharp_mask\nfrom skimage import exposure","metadata":{"execution":{"iopub.status.busy":"2021-07-01T12:48:14.76292Z","iopub.execute_input":"2021-07-01T12:48:14.76323Z","iopub.status.idle":"2021-07-01T12:48:14.772101Z","shell.execute_reply.started":"2021-07-01T12:48:14.76319Z","shell.execute_reply":"2021-07-01T12:48:14.771265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to get some test images\ndef get_some_images(process):\n    \n    count = 0\n    base_path = \"../input/siim-covid19-detection/\"\n    fig = plt.figure(figsize=(15, 6))\n\n    for study_dir in sorted(os.listdir(base_path + \"/test\")):\n        for series_dir in os.listdir(base_path + \"/test/\" + study_dir):\n            for image in os.listdir(base_path + \"/test/\" + study_dir + \"/\" + series_dir):\n                im = image.split(\".\")\n                file_in = base_path + \"/test/\" + study_dir + \"/\" + series_dir + \"/\" + image\n            \n                img = pydicom.dcmread(file_in)\n                pixels = img.pixel_array\n                \n                if img.PhotometricInterpretation == \"MONOCHROME1\":\n                    pixels = np.amax(pixels) - pixels\n                    \n                if process:\n                    pixels = remove_borders(pixels)      \n                    pixels = process_image(pixels)\n                 \n                pixels = pixels - np.min(pixels)\n                pixels = pixels / np.max(pixels)\n                pixels = (pixels * 255).astype(np.uint8)\n                \n                count += 1         \n                fig.add_subplot(3, 5, count)\n                plt.imshow(pixels, cmap='gray')\n\n        if count > 14:\n            break\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-01T12:48:14.773544Z","iopub.execute_input":"2021-07-01T12:48:14.773821Z","iopub.status.idle":"2021-07-01T12:48:14.783953Z","shell.execute_reply.started":"2021-07-01T12:48:14.773794Z","shell.execute_reply":"2021-07-01T12:48:14.783017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Apply unsharp mask and hist equalization\ndef process_image(pixels):\n\n    # Tweak the radius and amount for more/less sharpening\n    unsharp = unsharp_mask(pixels, radius=5, amount=2)\n    equalized = exposure.equalize_hist(unsharp)\n    \n    return equalized","metadata":{"execution":{"iopub.status.busy":"2021-07-01T12:48:14.78556Z","iopub.execute_input":"2021-07-01T12:48:14.785982Z","iopub.status.idle":"2021-07-01T12:48:14.80037Z","shell.execute_reply.started":"2021-07-01T12:48:14.785954Z","shell.execute_reply":"2021-07-01T12:48:14.799452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### The next function is designed to crop pixel rows off image edges that are all the same color.\n\n- More information can be found about this technique on this notebook I made -> https://www.kaggle.com/davidbroberts/cropping-chest-x-rays\n- This notebook shows how to retain BB info when cropping -> https://www.kaggle.com/davidbroberts/bounding-boxes-on-cropped-images\n\n*note - This notebook does not save the BB info, you'll need to add it to your export notebook, or this one","metadata":{}},{"cell_type":"code","source":"# Try to remove borders\ndef remove_borders(pixels):\n    x = 0\n    y = 0\n    img_orig = pixels\n    w = pixels.shape[1]\n    h = pixels.shape[0]\n\n    for i in range(h):\n        if not np.all(pixels[i] == pixels[i][0]):\n            y = i\n            break\n              \n    for i in range(h-1, 0, -1):\n        if not np.all(pixels[i] == pixels[i][0]):\n            h = i\n            break\n            \n    pixels = pixels[y:h,x:w] \n    pixels = np.rot90(pixels)\n    \n    w = pixels.shape[1]\n    h = pixels.shape[0]\n\n    for i in range(h):\n        if not np.all(pixels[i] == pixels[i][0]):\n            y = i\n            break\n              \n    for i in range(h-1, 0, -1):\n        if not np.all(pixels[i] == pixels[i][0]):\n            h = i\n            break\n            \n    pixels = pixels[y:h,x:w]\n    img_cropped = np.rot90(pixels, 3)\n    return img_cropped","metadata":{"execution":{"iopub.status.busy":"2021-07-01T12:48:14.80191Z","iopub.execute_input":"2021-07-01T12:48:14.802205Z","iopub.status.idle":"2021-07-01T12:48:14.81227Z","shell.execute_reply.started":"2021-07-01T12:48:14.802175Z","shell.execute_reply":"2021-07-01T12:48:14.811144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Take a look at a sample of images without and then with processing\n- The first set will be exported by pydicom .. using *some* algoritm with default settings. This is ugly!\n- We'll add processing to the same images using the `process=True` argument.","metadata":{}},{"cell_type":"code","source":"get_some_images(process=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-01T12:48:14.813521Z","iopub.execute_input":"2021-07-01T12:48:14.813962Z","iopub.status.idle":"2021-07-01T12:48:24.956039Z","shell.execute_reply.started":"2021-07-01T12:48:14.813932Z","shell.execute_reply":"2021-07-01T12:48:24.952848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_some_images(process=True)","metadata":{"execution":{"iopub.status.busy":"2021-07-01T12:48:24.957184Z","iopub.execute_input":"2021-07-01T12:48:24.957617Z","iopub.status.idle":"2021-07-01T12:48:48.78669Z","shell.execute_reply.started":"2021-07-01T12:48:24.957577Z","shell.execute_reply":"2021-07-01T12:48:48.785772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div class=\"alert alert-info\">\n    <h2> - Now that's a good looking dataset!</h2>\n</div>\n\n- By setting the process=True flag, we enable unsharp masking and histogram leveling.\n- If you look at the sets as a single image, you can see more standard colors and some cropping has happened in the second set.\n- Notice the image in the lower right corner, it was useless before we leveled it.\n- This works well on most of the images in the comp dataset. Some are already over processed though.","metadata":{}},{"cell_type":"code","source":"#Load specific a image manually to verify and tweak processing settings.\nimg = pydicom.dcmread('../input/siim-covid19-detection/test/00d63957bc3a/07919a1b758c/dbae9b9b9500.dcm')\n\npixels = img.pixel_array\n\nif img.PhotometricInterpretation == \"MONOCHROME1\":\n    pixels = np.amax(pixels) - pixels\n\ncropped = remove_borders(pixels)\ncropped = process_image(cropped)\n\nplt.figure(figsize=(15,5))\nplt.subplot(1, 2, 1)\nplt.title('Before')\nplt.imshow(pixels,cmap=\"gray\");\n\nplt.subplot(1, 2, 2)\nplt.title('After')\nplt.imshow(cropped,cmap=\"gray\");","metadata":{"execution":{"iopub.status.busy":"2021-07-01T12:50:59.748811Z","iopub.execute_input":"2021-07-01T12:50:59.749355Z","iopub.status.idle":"2021-07-01T12:51:00.80387Z","shell.execute_reply.started":"2021-07-01T12:50:59.749324Z","shell.execute_reply":"2021-07-01T12:51:00.803247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion\n\n- By making a simple processing pipeline, we can create more cohesive image sets.\n\n**Here are some other processing notebooks I made:**\n- Applying filters to x-rays -> https://www.kaggle.com/davidbroberts/applying-filters-to-chest-x-rays\n- Rib supression on Chest X-Rays -> https://www.kaggle.com/davidbroberts/rib-suppression-poc\n- Manual DICOM VOI LUT -> https://www.kaggle.com/davidbroberts/manual-dicom-voi-lut\n- Apply Unsharp Mask to Chest X-Rays -> https://www.kaggle.com/davidbroberts/unsharp-masking-chest-x-rays\n- Cropping Chest X-Rays -> https://www.kaggle.com/davidbroberts/cropping-chest-x-rays\n- Bounding Boxes on Cropped Images -> https://www.kaggle.com/davidbroberts/bounding-boxes-on-cropped-images\n- Visualizing Chest X-Ray bit planes -> https://www.kaggle.com/davidbroberts/visualizing-chest-x-ray-bitplanes\n- DICOM full range pixels as CNN input -> https://www.kaggle.com/davidbroberts/dicom-full-range-pixels-as-cnn-input","metadata":{}}]}