{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport sys\nimport re\n\nimport numpy as np \nimport pandas as pd \nimport cv2\nimport pydicom\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm\nimport lightgbm as lgb\nfrom IPython.display import Image, display\nfrom ipywidgets import interact\nfrom skimage.filters import threshold_otsu, threshold_local","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"<img src=\"https://universityhealthnews.com/wp-content/uploads/pneumonia.jpg\" width=\"500px\"/>\n\nLung consolidation occurs when the air that usually fills the small airways in your lungs is replaced with something else. Depending on the cause, the air may be replaced with:\n\n* a fluid, such as pus (in pneumonia) , blood, or water\n* a solid, such as stomach contents or cells\n\nPresence of such substances in your lungs appears as white patches "},{"metadata":{"_uuid":"d763b95a3a5196175f39644a553922e1007d715e"},"cell_type":"markdown","source":"<div class=\"row\">\n    <div class=\"column\">\n        <img src = \"http://www.startradiology.com/uploads/images/class-x-thorax-fig28b-alveolar-consolidations-heart-failure-version2-blanco.jpg\" width=\"200px\">\n    </div>\n    <div class=\"column\">\n        <img src=\"http://www.chestx-ray.com/images/igallery/resized/1-100/32-41-500-500-100.jpg\" width=\"200px\">\n    </div>\n</div>\nInfections put added stress on your heart, forcing it to work harder. Hence it may appear enlarged in some cases."},{"metadata":{"trusted":true,"_uuid":"802d200db133d65ec0f85bf92aa72bc97ed55a8b"},"cell_type":"code","source":"train_df = pd.read_csv('../input/stage_1_train_labels.csv')    \ntrain_info = pd.read_csv(\"../input/stage_1_detailed_class_info.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"d9a376b1541cfa45607ef26d212c6de7e1daae5d"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"016f48fb90e5fa51f2a2e05a3bd4797e19ab941f"},"cell_type":"code","source":"train_info.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"617448533614feb24c0b2bfcd305fc3ddf7b0f55"},"cell_type":"code","source":"def get_dcm_data(pId):\n    dicom_data = pydicom.read_file('../input/stage_1_train_images/'+pId+'.dcm')\n    return dicom_data\n\ndef get_metadata(pId, attribute):\n    dcmdata = get_dcm_data(pId)\n    # extract attribute values\n    attribute_value = getattr(dcmdata, attribute)\n    return attribute_value\n\ndef get_image(pId, size=(256, 256)):\n    dcmdata = get_dcm_data(pId)\n    img = cv2.resize(dcmdata.pixel_array, size)\n    return img\n\ndef get_otsu_image(img):\n    global_thresh = threshold_otsu(img)\n    binary_global = img > global_thresh\n    return binary_global.astype('int')\n    \ndef get_adaptive_image(img, block_size=31):\n    local_thresh = threshold_local(image, block_size, offset=10)\n    binary_local = image > local_thresh\n    return binary_local.astype('int')\n\n# Crop out the window and calculate the histogram\ndef get_window_avg(test_image, windowsize_r=32, windowsize_c=32):\n    ls = []\n    for r in range(0,test_image.shape[0] - windowsize_r, windowsize_r):\n        for c in range(0,test_image.shape[1] - windowsize_c, windowsize_c):\n            window = test_image[r:r+windowsize_r,c:c+windowsize_c]\n            ls.append(np.mean(window))\n            \n    return ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f776e736b0f15d65ba4fdcb7c8fe470c873d3ce4"},"cell_type":"code","source":"attributes = ['PatientSex', 'PatientAge', 'ViewPosition']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a05981c37dc4371f04b8a3e89b381b4c83984273"},"cell_type":"code","source":"for attr in tqdm(attributes):\n    train_df[attr] = train_df[\"patientId\"].apply(lambda x: get_metadata(x, attr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15b3bd0752fddd7857e57dcb8d035acde3a9f1c2"},"cell_type":"code","source":"train_df[\"images\"] = train_df[\"patientId\"].apply(lambda x: get_image(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"91b8c3b54fc523a018528902cba443021bac8e00","scrolled":false},"cell_type":"code","source":"print(\"Total number of observations: {}\".format(train_df.shape[0]))\nprint(\"Number of unique patients: {}\".format(train_df.patientId.nunique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ad716d9c81b443129c00ae7134bd21fbe3dca33"},"cell_type":"code","source":"train_df['class'] = train_info['class']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1960823acc7b3d3ede3b3f5d2c9d794062b53e29"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b111dcf519f61e3d4723351bab1d3f0335c1e6f8"},"cell_type":"markdown","source":"## Posteroanterior (PA) vs Anteroposterior (AP)\n\nThe PA (posterioranterior) film is obtained with the patient facing the cassette and the x-ray tube 6 feet away. This distance diminishes the effect of beam divergence and magnification of structures closer to the x-ray tube.  The heart shadow is magnified because it is an anterior structure. The pulmonary vasculature is also altered when patients are examined in the AP position.\n\n\n<img src=\"https://preview.ibb.co/e21u5f/PAvsAP.jpg\">\n"},{"metadata":{"_uuid":"bdc90ef862e61083f43ba2475a91b0c42d8b5011"},"cell_type":"markdown","source":"## Distribution of target variable"},{"metadata":{"_uuid":"c59707286e0881bf8a628f350664610b30f68c4e"},"cell_type":"markdown","source":"* Pnuemonia is equally likely in both males/females (though resent studies show otherwise)\n*  Equally likely to among all age groups[](http://)"},{"metadata":{"trusted":true,"_uuid":"b18257d6c408b71541a0b20dece26a0e55c6c60c","scrolled":true},"cell_type":"code","source":"select = [\"PatientSex\", \"ViewPosition\"]\n@interact(select=select)\ndef plot_bar(select):\n    fig, axs = plt.subplots(1, 2, figsize=(20, 10))\n    df = train_df[train_df['Target'] == 1]\n    df = df[[\"patientId\", select]].drop_duplicates().drop([\"patientId\"], axis=1)[select]\n    freq = df.value_counts().sort_index()\n    axs[0].bar(x=freq.index, height=freq.values)\n    axs[0].set_title(\"Abnormal\")\n    \n    df2 = train_df[train_df['Target'] == 0]\n    df2 = df2[[\"patientId\", select]].drop_duplicates().drop([\"patientId\"], axis=1)[select]\n    freq2 = df2.value_counts().sort_index()\n    axs[1].bar(x=freq2.index, height=freq2.values, color='orange')\n    axs[1].set_title(\"Normal\")\n    fig.suptitle(\"Distribution of Target Variable\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"79ed9daa1e87a858cb3a51ffd7a1c617c209b69b"},"cell_type":"markdown","source":"## Feature Engineering"},{"metadata":{"_uuid":"cabc2e1e4d9ae54342d5f500e4b0b9d424215cc0"},"cell_type":"markdown","source":"#### Adaptive Thresholding\n\nIn adaptive threshold, the threshold value at each pixel location depends on the neighboring pixel intensities. To calculate the threshold T(x, y) i.e. the threshold value at pixel location (x, y) in the image, we perform the following steps \n- A bxb region around the pixel location is selected. b is selected by the user.\n- The next step is to calculate the weighted average of the bxb region. OpenCV provides 2 methods to calculate this weighted average. We can either use the average (mean) of all the pixel location that lie in the bxb box or we can use a Gaussian weighted average of the pixel values that lie in the box. In the latter case, the pixel values that are near to the center of the box, will have higher weight. We will represent this value by WA(x, y).\n- The next step is to find the Threshold value T(x, y) by subtracting a constant parameter, let’s name it param1 from the weighted average value WA(x, y) calculated for each pixel in the previous step. The threshold valueT(x, y) at pixel location (x, y) is then calculated using the formula given below \n\n#### Otsu Thresholding\n\n"},{"metadata":{"_uuid":"b25baeda13f191f18942b211aea6b5f1fef86618"},"cell_type":"markdown","source":"<img src = \"http://scikit-image.org/docs/dev/_images/sphx_glr_plot_thresholding_003.png\" width='700px'></img>"},{"metadata":{"trusted":true,"_uuid":"1888b7d70dc914c6e2df75e09a932903aa96313b"},"cell_type":"code","source":"image = train_df.iloc[3,9]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90ea31540ae6852d1ef302896d5ecb5330298952"},"cell_type":"code","source":"plt.hist(image.ravel(), bins=256, fc='k', ec='k')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1888b7d70dc914c6e2df75e09a932903aa96313b"},"cell_type":"code","source":"global_thresh = threshold_otsu(image)\nbinary_global = image > global_thresh","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1888b7d70dc914c6e2df75e09a932903aa96313b"},"cell_type":"code","source":"block_size = 31\nlocal_thresh = threshold_local(image, block_size, offset=10)\nbinary_local = image > local_thresh","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0d9af269e43639b53a4e72ba4562ee0686c5d1bb"},"cell_type":"code","source":"block_size2 = 121\nlocal_thresh2 = threshold_local(image, block_size2, offset=10)\nbinary_local2 = image > local_thresh2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f1baf88cedfc18078a01b3a4fab114c5d73085e2","scrolled":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(20, 10))\nax = axes.ravel()\nplt.gray()\n\nax[0].imshow(image)\nax[0].set_title('Original')\n\nax[1].imshow(binary_global)\nax[1].set_title(\"Global thresholding - Otsu\")\n\nax[2].imshow(binary_local)\nax[2].set_title('Local thresholding (block size=30)')\n\nax[3].imshow(binary_local2)\nax[3].set_title('Local thresholding (block size=120)')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21fa27b6df9e25e361f9bb6a6d98afae88702bf0"},"cell_type":"code","source":"train_df[\"images_otsu\"] = train_df[\"images\"].apply(lambda x : get_otsu_image(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c9f800a491db140df649174e85038e4c256e8ae9"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"007c99fb5aefde108be572ec464bc2e8c0476ae8"},"cell_type":"code","source":"get_window_avg(binary_local)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}