{"cells":[{"metadata":{"_uuid":"236e3c5f006b673e26cc8bb16a03708cd655e38d"},"cell_type":"markdown","source":"# RSNA 2018 EDA, Part 1"},{"metadata":{"trusted":true,"_uuid":"57508a269aa8b3e14945e39f501f6ffcdb22471d"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport pydicom\nimport pylab\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42ee8af8ccea3566b97b1632be16524168e0dc4b"},"cell_type":"markdown","source":"## 1. Exploring the Given Files"},{"metadata":{"_uuid":"c439e828616769c1d2ea7c6d6e7ec8954e18d4bf"},"cell_type":"markdown","source":"The directory listing below shows the files we are given, which we will explore one by one.  (Except for the credit request link, which I believe has expired,)"},{"metadata":{"trusted":true,"_uuid":"1b6c13edf09c7d80145173edad847c34db1ba5f2"},"cell_type":"code","source":"!ls -rtl ../input","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0fd23837d9c41adde5dc4f8eefac0df5a4aefac3"},"cell_type":"markdown","source":"### 1.1 Sample Submission"},{"metadata":{"_uuid":"fec32353595a5c033e32c5876fcba787cd25f052"},"cell_type":"markdown","source":"A sample submission file is show below.  (There is more information on this in the \"Evaluation\" section of the contest rules.)  There will be one row in the submission file for each observation in the test set, and each row will consist of two strings separated by a comma (the patient ID and our prediction for that patient.)  The prediction string will contain 5 numbers (separated by spaces) for EACH spot of pneumonia we find for that patient (a confidence level, the x,y location of the box (I believe this is upper left corner and not center, based on the sample submission data and the fact that the training labels are that way ,) and the width and heigth of the box.  For a patient for whom we predict NO pneumonia, the prediction string is enpty.\n\nThe information required in our submission will affect our choice of model.  We need something that will output bounding boxes and confidence levels.  I think YOLO will do this, but maybe there are other algorithms that will as well."},{"metadata":{"trusted":true,"_uuid":"22d70ebc7c6f15f9479688ab6e6bb1ab2f14d5eb"},"cell_type":"code","source":"df = pd.read_csv('../input/stage_1_sample_submission.csv')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ffd9a01b72e5d61e656f17934198df9e9facb9f"},"cell_type":"markdown","source":"### 1.2 Training Labels"},{"metadata":{"_uuid":"04b0623e96295b353b5d871e9829c49a7a00acb7"},"cell_type":"markdown","source":"The training labels file contains the \"right answers\" for the training set, so we can use some kind of supervised learning algorithm.  The contest information states that each row in this table contains information for just one spot of pneumonia, and there may be more than one row for a given patient.  Also, it looks like rows in which Target=0 (meaning the patient has NO pneumonia) also have null values for the bounding box, but we should verify this.\n\nWe see that there are 28,989 rows in the labels information, and 25,684 unique patients represented.  We also see that there are no values in the Target column other than 0 or 1, which is good, and that there are 20,025 patients having NO pneumonia."},{"metadata":{"trusted":true,"_uuid":"9793a87b8e6c58131dc0bfa719a16a00a8d7eab8"},"cell_type":"code","source":"df_lbl = pd.read_csv('../input/stage_1_train_labels.csv')\ndf_lbl.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0e33179171f893fe5846d2e80028d35a72a22d4"},"cell_type":"code","source":"df_lbl.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3b063cdf5fba83f7ef36fd638bd2e8d1aa671b25"},"cell_type":"code","source":"n_pat = df_lbl.patientId.nunique()\nprint('Number of unique patients in training data: ', n_pat)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2e85fbad71954697a4141e9a38ea2e9f62951b67"},"cell_type":"code","source":"df_lbl.Target.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b152bab233a76a442626ec66e6e0ab7105a791c"},"cell_type":"markdown","source":"To make it more convenient to check things in the labels, let's split it into separate tables of positive and negative results.\n\nWe see first that all observations for patients with NO pneumonia have null values for box information.  I'm not sure it would matter much if this was not the case, but it is one way in which the data is consistent, which is always good.  More importantly, we see that all positive observations DO have bounding box information.  We also see that we have an unbalanced training set (only 22% of patients diagnosed positive,) and that, on average, we can expect to find multiple spots on each X-ray."},{"metadata":{"trusted":true,"_uuid":"4b851a5eff708d23232ce4844905c7778b43a8f3"},"cell_type":"code","source":"df_lbl_neg = df_lbl[df_lbl.Target == 0]\ndf_lbl_pos = df_lbl[df_lbl.Target == 1]\ndf_lbl_neg.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f6b62a2c2c1e02a5f17507ac2884f84949261ed5"},"cell_type":"code","source":"df_lbl_pos.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"27675fa46fd8c9660de248d125f253c4b47b76d6"},"cell_type":"code","source":"n_pat_pos = df_lbl_pos.patientId.nunique()\nprint('Number of unique patients having pneumonia: ', n_pat_pos)\nprint('Fracion of unique patients having pneumonia: ', n_pat_pos / n_pat)\nprint('Avg number of anomalies per patient having pneumonia: ', len(df_lbl_pos) / n_pat_pos)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23d1514def72b67f8180bbbb42cde6ca0f32463e"},"cell_type":"markdown","source":"Below we take a closer look at \"spots per patient\", and see that the most spots any patient in the training data has is 4, and the vast majority of patients with pneumonia have 1 or 2."},{"metadata":{"trusted":true,"_uuid":"5a4309d1c443db032d125924703820f4a2d66d83"},"cell_type":"code","source":"df_lbl_pos.patientId.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0dbaee6f0fd1a0d9cc1babee9cfb3aba557bddb3"},"cell_type":"code","source":"df_lbl_pos.patientId.value_counts().hist()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0129466b235f0d6d9cf47832ea0575bf0ca107a9"},"cell_type":"markdown","source":"Although we have not chosen an algorithm yet, we know that we will have to detect bounding boxes, so let's get more feel for what the set of boxes in the training data looks like, because it is reasonable to expect that the box population in the test data will be similar. This information might be useful if we choose an algorithm that uses anchor boxes.\n\nI have learned from inspecting the DICOM images (see below) and from other sources such as contest information, discussions, sample kernels, etc, that the units of x, y, width, and heigth are pixels, that the X-ray images are 1024x1024, and than pixel 0,0 is in the upper left corner of the image. \n\nLet's add columns for the box center coords, because if we have to defined anchor boxes that is how we will do it, and because if we want to visualize the distribution of box locations across the lungs, that is more intuitive.  (I think we can ignore the \"setting on a copy of a slice\" warnings here, because we really do just want these new fields on the positive subset of the data.)"},{"metadata":{"trusted":true,"_uuid":"a1384b2962adf4fc40cac450ca4965c7d41aaa6c"},"cell_type":"code","source":"df_lbl_pos['xc'] = df_lbl_pos.x + df_lbl_pos.width / 2\ndf_lbl_pos['yc'] = df_lbl_pos.y + df_lbl_pos.height / 2\ndf_lbl_pos.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e5724e168eb41e1f1283b491063daf3e675349a9"},"cell_type":"code","source":"df_lbl_pos.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57bf700207f870b43654be32248e19a70f727ff4"},"cell_type":"markdown","source":"Let's try to do some visualization.  Keep in mind that the y-axis of the plots is flipped with respect to the X-ray images.\n\nThe first plot below examines the distribution of box centers.  We see that the mean positions of spots are more or less in the center of each lung.  Surprisingly, there appears to be more slightly spots in one lung than the other.  (Which lung this is depends on which way the X-ray is shot, which information can be extracted from the image files, but I have not examined that yet.)  The plot suggests that if we have to choose anchor boxes, we might want to define two sets (one for each lung), or possibly divide each image in half vertically and examine the halves separately.\n\nThe second plot examines the distribution of box widths and heights.  We can see from the scales of the plot that spots tend to be taller than they are wide, which makes sense because that's the way the lungs are shaped.  Despite this, however, we also see that the mean in both dimensions is roughly 200 pixels."},{"metadata":{"trusted":true,"_uuid":"6a8116de70428f350ae64eb700b8f298ed33f81b"},"cell_type":"code","source":"sns.jointplot(x='xc', y='yc', data=df_lbl_pos, kind='kde')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98e0ee80926246d3224d14ca83730b28dab45bda"},"cell_type":"code","source":"sns.jointplot(x='width', y='height', data=df_lbl_pos, kind='kde')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a8007d7ae48552dc64920b45069d0ee2cd4279d"},"cell_type":"markdown","source":"### 1.3 Detailed Class Info"},{"metadata":{"_uuid":"53f23d0406a6f9b6d92f1deb9b8503d936467b57"},"cell_type":"markdown","source":"The only remaining file in the main level of the input directory is the detailed class file, which we see below has just 2 columns, the same number of rows as the labels file, and no missing values.  The only added information in this file is in the \"class\" column, which for each observation contains one of three values.  One of these values is \"Lung Opacity\", which corresponds to a pneumonia diagnosis, and there are the same number of occurences of this value as there are Target=1 in the lables file, which is good.  The other two classes designate either lungs that are normal or those with spots that are not pneumonia, and the sum of the occurrences of these two values matches the number of Target=0 observations in the labels file.\n\nIt may be that the information in this file will help the model discriminate against \"spots\" that are not pneumonia.  However, we do not have boxes in the training labels for any non-pneumonia spots, so it will be hard to tell if the model is on the right track in detecting such boxes."},{"metadata":{"trusted":true,"_uuid":"0a646212ef9482f8da048a2e0c516bed6629a8a4"},"cell_type":"code","source":"df_class = pd.read_csv('../input/stage_1_detailed_class_info.csv')\ndf_class.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"547775c6a3e7f73e6d622e5a502874be052dd763"},"cell_type":"code","source":"df_class.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bc29ac68dd9f81a6317c75da9faad07a2aa6169b"},"cell_type":"code","source":"df_class['class'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32203edec8d826ca3f5f849a91e84943574517a2"},"cell_type":"markdown","source":"### 1.4 Image Files"},{"metadata":{"_uuid":"12433fcd191e99b1ab2347c52bc1142de9c40d15"},"cell_type":"markdown","source":"The final part of the input data to examine is the X-ray image files that correspond to the training labels, and there are the same number of files as there are unique patients represented in the training data, which is good."},{"metadata":{"trusted":true,"_uuid":"b5b624e8fd25558385a2c3e5762c6e5572862ef6"},"cell_type":"code","source":"print('Number of training images:')\n!ls ../input/stage_1_train_images | wc -l","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65fdd15392da7c10c94787bdf46d2643bc30765e"},"cell_type":"markdown","source":"The X-ray files are in DICOM format -- they contain not only the X-ray image itself but also related information (example below.)  The files have been sanitized of all patient-related information, but there are still a few fields that might be useful, such as age, sex, and X-ray orientation."},{"metadata":{"trusted":true,"_uuid":"2734601e3697e9a4b501a006e535c402ee233a7b"},"cell_type":"code","source":"patientId = df_lbl_pos.iloc[0,0]\ndcm_file = '../input/stage_1_train_images/%s.dcm' % patientId\ndcm_data = pydicom.read_file(dcm_file)\nprint(dcm_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e406cf834a10d773e6a08131d9af07afbf007ee"},"cell_type":"code","source":"# extracting single field from DICOM file\nage = dcm_data[0x10,0x1010].value\nprint(age)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c19315e4ba91c800b4e6319d5781bb586ff942de"},"cell_type":"code","source":"# displaying image from DICOM file\nimg = dcm_data.pixel_array\npylab.imshow(img, cmap=pylab.cm.gist_gray)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1410a2bc91956ffda0eb26c1c8708ae422b16f31"},"cell_type":"markdown","source":"Here we will put together some code (mostly copied or adapted from sample kernels) that will not only help us explore the x-rays a little bit but also might be helpful for evaluating the output of our model.\n\nI want to see negative as well as positive x-rays, so I will go back to using the complete table of training labels.  For convenience in plotting the boxes, I will transform the x, y, w, h data and make sure the table is sorted by patient ID so that all boxes for a given patient are contiguous.\n\nWe may at some point want to show boxes in different colors, so we will convert the images from single-channel grayscale to 3-channel RGB."},{"metadata":{"trusted":true,"_uuid":"72b26c9d234b9106106f693dad3d262066bca84e"},"cell_type":"code","source":"df2 = pd.read_csv('../input/stage_1_train_labels.csv')\ndf2.fillna(0, inplace=True)\ndf2.iloc[:, 1:5] = df2.iloc[:, 1:5].astype('int')\ndf2.sort_values('patientId', inplace=True)\ndf2.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21b834f2265173873cd1d6c51e4b207158887f19"},"cell_type":"code","source":"df2.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bc99dcd1c2035f6114f92035a18cb4c863684d5b"},"cell_type":"code","source":"def add_box(img, x1, y1, w, h, rgb, lw=1):\n    x2 = x1 + w\n    y2 = y1 + h\n    img[y1:y1 + lw, x1:x2] = rgb\n    img[y2:y2 + lw, x1:x2] = rgb\n    img[y1:y2, x1:x1 + lw] = rgb\n    img[y1:y2, x2:x2 + lw] = rgb\n    return img\n\ndef find_pat_rows(idx):\n    pid = df2.iloc[idx,0]\n    start = idx\n    stop = idx + 1\n    while (df2.iloc[start - 1, 0] == pid) and (start > 0):\n        start -= 1\n    while (df2.iloc[stop + 1, 0] == pid) and (stop < len(df2) - 1):\n        stop += 1\n    return start, stop\n\ndef show_xray(pat_idx):\n    # get observation data from table index\n    pid, x, y, w, h, t = df2.iloc[pat_idx,:]\n    # load image, convert from grayscale to RGB\n    dcm_file = '../input/stage_1_train_images/%s.dcm' % pid\n    dcm_data = pydicom.read_file(dcm_file)\n    img = np.stack([dcm_data.pixel_array] * 3, axis=2)\n    # if this row has Target=1, add all relevant boxes\n    if t == 1:\n        start, stop = find_pat_rows(pat_idx)\n        for n in range(start, stop):\n            pid, x, y, w, h, t = df2.iloc[n,:]\n            rgb = np.array([255, 0, 0])\n            img = add_box(img, x, y, w, h, rgb, 5)\n    # display image\n    pylab.imshow(img, cmap=pylab.cm.gist_gray)\n\n# for playing at this point, just choose a random x-ray each time it runs\nidx = np.random.randint(0, len(df2), 1)\nshow_xray(idx[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f4da05fb65ba9b4ce1b14b7a9c85140fcd037d5"},"cell_type":"markdown","source":"# Notes on Next Steps (from Starter Kernel)\n\nNow that you understand the data structures, imaging file formats and label types, it's time to make an algorithm! Keep in mind that the primary endpoint is the detection of bounding boxes, thus you will likely be considering various **object localization** algorithms. An alternative strategy is to consider the related family of **segmentation** algorithms with the acknowledgement that bounding boxes will only be a coarse approximation to true pixel-by-pixel image segmentation masks.\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}