{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Project dataset information:\nIn the process of taking the image, an X-ray passes through the body and reaches a detector on the other side. Tissues with sparse material, such as lungs which are full of air, do not absorb the X-rays and appear black in the image. Dense tissues such as bones absorb the X-rays and appear white in the image. In short -\n\n* Black = Air\n* White = Bone\n* Grey = Tissue or Fluid\n\nThe left side of the subject is on the right side of the screen by convention. You can also see the small L at the top of the right corner. In a normal image we see the lungs as black, but they have different projections on them - mainly the rib cage bones, main airways, blood vessels and the heart."},{"metadata":{},"cell_type":"markdown","source":"# Importing necessary packages"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport pylab\nimport numpy as np\nimport pydicom\nimport pandas as pd\nfrom glob import glob\nimport os\nfrom matplotlib.patches import Rectangle\n\ndatapath = '../input/rsna-pneumonia-detection-challenge/'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!ls ../input/rsna-pneumonia-detection-challenge/stage_2_train_images/ | wc -l","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Utility Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def parse_data(df):\n    extract_box = lambda row: [row['y'], row['x'], row['height'], row['width']]\n\n    parsed = {}\n    for n, row in df.iterrows():\n        # --- Initialize patient entry into parsed \n        pid = row['patientId']\n        if pid not in parsed:\n            parsed[pid] = {\n                'dicom': datapath + 'stage_2_train_images/%s.dcm' % pid,\n                'label': row['Target'],\n                'boxes': []}\n\n        # --- Add box if opacity is present\n        if parsed[pid]['label'] == 1:\n            parsed[pid]['boxes'].append(extract_box(row))\n\n    return parsed\n\ndef draw(data):\n    \"\"\"\n    Method to draw single patient with bounding box(es) if present \n\n    \"\"\"\n    # --- Open DICOM file\n    d = pydicom.read_file(data['dicom'])\n    im = d.pixel_array\n\n    # --- Convert from single-channel grayscale to 3-channel RGB\n    im = np.stack([im] * 3, axis=2)\n\n    # --- Add boxes with random color if present\n    for box in data['boxes']:\n        rgb = np.floor(np.random.rand(3) * 256).astype('int')\n        im = overlay_box(im=im, box=box, rgb=rgb, stroke=6)\n\n    plt.imshow(im, cmap=plt.cm.gist_gray)\n    \n    \ndef overlay_box(im, box, rgb, stroke=1):\n    \"\"\"\n    Method to overlay single box on image\n\n    \"\"\"\n    # --- Convert coordinates to integers\n    box = [int(b) for b in box]\n    \n    # --- Extract coordinates\n    y1, x1, height, width = box\n    y2 = y1 + height\n    x2 = x1 + width\n\n    im[y1:y1 + stroke, x1:x2] = rgb\n    im[y2:y2 + stroke, x1:x2] = rgb\n    im[y1:y2, x1:x1 + stroke] = rgb\n    im[y1:y2, x2:x2 + stroke] = rgb\n\n    return im","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis and Visualizations"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_box = pd.read_csv(datapath+'stage_2_train_labels.csv')\nprint('Number of rows (unique boxes per patient) in main train dataset:', df_box.shape[0])\nprint('Number of unique patient IDs:', df_box['patientId'].nunique())\ndf_box.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_box.groupby('Target').size().plot.bar()\nprint(df_box.groupby('Target').size() / df_box.shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Detailed Class Info Data**\n\nThe file stage_2_detailed_class_info.csv contains detailed information about the positive and negative classes in the training set, and may be used to build more nuanced models. As in the main training dataset, this auxiliary dataset contains 28989 rows and 25684 unique patient IDs. There's 3 classes:\n\n1. Normal (29%)\n2. No Lung Opacity / Not Normal (40%)\n3. Lung Opacity (31%)\n\nThe first two classes correspond to Target = 0, whereas the third class correspond to Target = 1."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_det = pd.read_csv(datapath+'stage_2_detailed_class_info.csv')\nprint('Number of rows in auxiliary dataset:', df_det.shape[0])\nprint('Number of unique patient IDs:', df_det['patientId'].nunique())\ndf_det.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_det.groupby('class').size().plot.bar()\nprint(df_det.groupby('class').size() / df_det.shape[0])\nassert df_det.loc[df_box['Target']==0].shape[0] == df_det.loc[df_det['class'].isin(['Normal', \\\n    'No Lung Opacity / Not Normal'])].shape[0], 'Number of negative targets does not match between main and detailed dataset.'\nassert df_box.loc[df_box['Target']==1].shape[0] == df_det.loc[df_det['class'] == 'Lung Opacity'].shape[0], \\\n    'Number of positive targets does not match between main and detailed dataset.'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert df_box['patientId'].values.tolist() == df_det['patientId'].values.tolist(), 'PatientId columns are different.'\ndf_train = pd.concat([df_box, df_det.drop(labels=['patientId'], axis=1)], axis=1)\ndf_train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pId = \"003d8fa0-6bf1-40ed-b54c-ac657f8495c5\"    \ndcmdata = pydicom.read_file(datapath+'stage_2_train_images/'+pId+'.dcm')\nprint(dcmdata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dcmimg = dcmdata.pixel_array\nplt.figure(figsize=(7,7))\nplt.imshow(dcmimg, cmap=pylab.cm.binary)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Lung Opacity Visualization**"},{"metadata":{"trusted":true},"cell_type":"code","source":"parsed = parse_data(df_box)\n\npatientId = df_box['patientId'][8]\n#print(df_det.loc[patientId])\n\nplt.figure(figsize=(7,7))\nplt.title(\"Sample Patient - Lung Opacity\")\n\ndraw(parsed[patientId])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Side by Side Comparision of classes**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10))\n\nplt.subplot(131)\nplt.title(\"Normal Image\")\ndraw(parsed[df_box['patientId'][3]])\n\nplt.subplot(132)\nplt.title(\"Lung Opacity\")\ndraw(parsed[df_box['patientId'][16]])\n\nplt.subplot(133)\nplt.title(\"No Lung Opacity / Not Normal\")\ndraw(parsed[df_box['patientId'][1]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}