{"cells":[{"metadata":{"_uuid":"ff3a64da83c425b351fb218ddf5bc719fd39dd37"},"cell_type":"markdown","source":"Kaggle has put forward another image challenge! I love these because convolutional networks are pretty nifty, and they are the perfect too for this job. Let's begin by getting to know the data and what format it is stored in! to do this we are going to need to import some libraries.\n\n**A wild Image Format appeared!** This is the first time i have dealt with DICOM images personally. I was vaguely aware of their existence working as a technical analyst for an EHR system however I've never had the opportunity to get down and dirty with them! Another reason I love Kaggle!\n\nThe new library I am referring to is the pydicom library. Reading through the documentation it appears that a DCM image is actually an archive with a lot of interesting meta data and an image. The meta data in these images isnt quite as interesting as i hoped since the data has been deidentified however its still awesome to have such a fun storage format to play with.\n\nLets load some DICOM files, view some metadata and plot the images."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pydicom\nimport os\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import cm\nfrom matplotlib import pyplot as plt\nimport cv2\nimport seaborn as sns\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"173d7c492ff1ddf974d70f023e80b06f8f86f206"},"cell_type":"markdown","source":"No we can define a function to view some interesting metadata fields, this is taken almost directly from the [pydicom website](https://pydicom.github.io/pydicom/stable/auto_examples/input_output/plot_read_dicom.html), I have just added a few fields I found interesting and turned it into a function since I beleive that code is easier to manage like this, even in Jupyter Notebooks."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"339f5f11b9fc4b9955871f92d69a096f0cfcac84"},"cell_type":"code","source":"def show_dcm_info(dataset):\n    print(\"Filename.........:\", file_path)\n    print(\"Storage type.....:\", dataset.SOPClassUID)\n    print()\n\n    pat_name = dataset.PatientName\n    display_name = pat_name.family_name + \", \" + pat_name.given_name\n    print(\"Patient's name......:\", display_name)\n    print(\"Patient id..........:\", dataset.PatientID)\n    print(\"Patient's Age.......:\", dataset.PatientAge)\n    print(\"Patient's Sex.......:\", dataset.PatientSex)\n    print(\"Modality............:\", dataset.Modality)\n    print(\"Body Part Examined..:\", dataset.BodyPartExamined)\n    print(\"View Position.......:\", dataset.ViewPosition)\n    \n    if 'PixelData' in dataset:\n        rows = int(dataset.Rows)\n        cols = int(dataset.Columns)\n        print(\"Image size.......: {rows:d} x {cols:d}, {size:d} bytes\".format(\n            rows=rows, cols=cols, size=len(dataset.PixelData)))\n        if 'PixelSpacing' in dataset:\n            print(\"Pixel spacing....:\", dataset.PixelSpacing)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0aa26d359ae41e46a89d3927e816cf4d147ea33"},"cell_type":"markdown","source":"We will also need a function to plot the images (we will be using matplotlib.pyplot for this). I've included the option to control the figure size, you can fork this Notebook and adjust that default argument for the function if you have a smaller/bigger screen."},{"metadata":{"trusted":true,"_uuid":"a1d5747542fa0e773d99840e4ffd7186425e4f13"},"cell_type":"code","source":"def plot_pixel_array(dataset, figsize=(10,10)):\n    plt.figure(figsize=figsize)\n    plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3536d3a860cbe0223ecd1c8df364f600745322a0"},"cell_type":"markdown","source":"Now we have all our ground work in place, lets look at what is required to read and load a DICOM image from disk! We will only visualise a few images here."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"440f2f6de3c5e3bc58aeb9d6fc4981f386260435"},"cell_type":"code","source":"i = 1\nnum_to_plot = 5\nfor file_name in os.listdir('../input/stage_1_train_images/'):\n    file_path = os.path.join('../input/stage_1_train_images/', file_name)\n    dataset = pydicom.dcmread(file_path)\n    show_dcm_info(dataset)\n    plot_pixel_array(dataset)\n    \n    if i >= num_to_plot:\n        break\n    \n    i += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ad69a2fe7d7de60007435d64d1a82843ebe0704"},"cell_type":"markdown","source":"Wow! These images are fairly large! they are 1024x1024 pixels, and they look wonderful when plotted. This is going to be a spectacular project/competition to work on! The bone colour map is a beutiful touch, they give the plots that authentic X-ray feeling."},{"metadata":{"_uuid":"4635282cccd39a53e7660df2d3d00d405cdfe8a4"},"cell_type":"markdown","source":"Looking at the images we can see that the Female and Male bodies actually do differ in shape and size a little bit. Lets do some EDA on the gender and ages of clients in this dataset and find out what sort of distributions we are dealing with!"},{"metadata":{"trusted":true,"_uuid":"d4714f5ef95021c816791bbfe843d97b5787e5a8"},"cell_type":"code","source":"train_demo_df = pd.DataFrame()\nids = []\nages = []\nsexs = []\nimg_avg_lums = []\nimg_max_lums = []\nimg_min_lums = []\n\nfrom multiprocessing.pool import Pool, ThreadPool\n\npool = ThreadPool(4)\n\ndef process_image(dataset):\n    _id = dataset.PatientID\n    _age = dataset.PatientAge\n    _sex = dataset.PatientSex\n    _mean = np.mean(dataset.pixel_array)\n    _min = np.max(dataset.pixel_array)\n    _max = np.min(dataset.pixel_array)\n    return _id, _age, _sex, _min, _max, _mean\n\nresponses = []\nfor file_name in tqdm(os.listdir('../input/stage_1_train_images/')):\n    \n    file_path = os.path.join('../input/stage_1_train_images/', file_name)\n    dataset = pydicom.dcmread(file_path)\n\n    responses.append(pool.apply_async(process_image, (dataset,)))\n\n\npool.close()\npool.join()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d5eb5b6bc6f6f9b0d5fe23ac85a81406fa6fae8"},"cell_type":"code","source":"for response in tqdm(responses):\n    _id, _age, _sex, _min, _max, _mean = response.get()\n    ids.append(_id)\n    ages.append(_age)\n    sexs.append(_sex)\n    img_min_lums.append(_min)\n    img_max_lums.append(_max)\n    img_avg_lums.append(_mean)\n\n\ntrain_demo_df['patientId'] = pd.Series(ids)\ntrain_demo_df['patientAge'] = pd.Series(ages, dtype='int')\ntrain_demo_df['patientSex'] = pd.Series(sexs)\n\ntrain_demo_df['imageMin'] = pd.Series(img_max_lums)\ntrain_demo_df['imageMax'] = pd.Series(img_min_lums)\ntrain_demo_df['imageMean'] = pd.Series(img_avg_lums)\n\nsex_map = {'F': 0, 'M': 1}\ntrain_demo_df['patientSex'] = train_demo_df['patientSex'].replace(sex_map).astype('int')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81b3cefc0e0d970f88bc102d9ba96fa4f5a358e1"},"cell_type":"markdown","source":"We will also need to load the class data and append it to the dataframe, for this we will use Pandas."},{"metadata":{"trusted":true,"_uuid":"057f090cae83600b212448ac4afd0be9bc9ac47d"},"cell_type":"code","source":"class_df = pd.read_csv('../input/stage_1_detailed_class_info.csv')\n\ntrain_demo_df = pd.merge(left=train_demo_df, right=class_df, left_on='patientId', right_on='patientId')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92de6b4260482da8dda3cd9c50e865a48f74d87a"},"cell_type":"markdown","source":"Let's quickly inspect the integer columns to see if there are any outliers."},{"metadata":{"trusted":true,"_uuid":"6ab67ea26138ee28e45fe74bf3af36b6f555fcf8"},"cell_type":"code","source":"print(train_demo_df.describe())\ntrain_demo_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a5827823d396608265b87c059f1cf23b708e7f3"},"cell_type":"markdown","source":"Ok since only a handful of people make it past 110 years of age lets assume that the max patient age of 155 is an error, lets view all the images where the age is over 100"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"893e1bd362b567e9112b5ce4673b3e46f84adc42"},"cell_type":"code","source":"for file_name in tqdm(os.listdir('../input/stage_1_train_images/')):\n    file_path = os.path.join('../input/stage_1_train_images/', file_name)\n    dataset = pydicom.dcmread(file_path)\n    if int(dataset.PatientAge) >= 100:\n        show_dcm_info(dataset)\n        plot_pixel_array(dataset)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24c651bd03a5ef12c0d4c3a6bf7f66d929085675"},"cell_type":"markdown","source":"I was hoping these images would show smaller people and we could assume they were children and that the age was in months, it would a simple way to fix it. After doing a few comparisons to different age groups I will assume that the 1 has been placed there in error. They appear to be too old to by under 13 and they appear to young to be over 80. To fix this we will clean the data and replace any values over 100 with the last two digits, making the 155 example 55 years old. "},{"metadata":{"trusted":true,"_uuid":"e4b24548e11ae90c625aec7441edf4c83b9cdc63"},"cell_type":"code","source":"train_demo_df = train_demo_df.where(train_demo_df['patientAge'] <= 100, train_demo_df['patientAge']-100, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9bd0eb72963e6e805bcf12417d4bc39a78583a87"},"cell_type":"markdown","source":"Ok now we've decided and implemented out data cleansing, lets graph the distributions for Age and Sex using a Seaborn pairplot!"},{"metadata":{"trusted":true,"_uuid":"28c0c0d7f828a75e77362f72475c9e39325b92c6"},"cell_type":"code","source":"sns.pairplot(train_demo_df, hue='class', height=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"faa4076990abfc5de5d2e4d81f5f8a37ddbce521"},"cell_type":"markdown","source":"From these charts we can see that there are more images for males than females, both genders have the most classes of \"No Lung Opacity/Not Normal\", however besides this fact the men are more likely to have a class of \"Lung Opacity\" where as women are by proportion less likely.\n\nto explain what I mean a little clearer see the list below:\n\n  * Ranking of class probability in females:\n      - No Lung Opacity/Not Normal\n      - Normal\n      - Lung Opacity\n  * Ranking of class probability in males:\n     - No Lung Opacity/Not Normal\n     - Lung Opacity\n     - Normal\n\nLung opacity seems to also be distributed with a slight twin peaks shape, from the graph you can see that people around the age of 30 are the slightly more likely to have Lung Opacity for the age group below 40.\n\nAnother interesting note from these charts, is that some of those statistic values we extracted fro the images show clear and distinctive seperations between the classes. It would appear as if having a higher mean, and a higher minimum that the images are more likely to be classed as either \"Normal\" or \"No Lung Opacity/Not Normal\"."},{"metadata":{"trusted":true,"_uuid":"640c757b6ee43f339934c7f19038e098d5bc3d3a"},"cell_type":"markdown","source":"**TO BE CONTINUED**"},{"metadata":{"trusted":true,"_uuid":"b0e9d61d9742ef0fa88223e2e39f2ed1da5a0fa0"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}