{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# OSIC Pulmonary Fibrosis Progression\n<img src=\"https://health.clevelandclinic.org/wp-content/uploads/sites/3/2017/05/Lungs3D.jpg\" width=400, height=400>\n\n\n### what is Pulmonary Fibrosis?\n#### Pulmonary fibrosis is a progressive disease that naturally gets worse over time. This worsening is related to the amount of fibrosis (scarring) in the lungs. As this occurs, a person's breathing becomes more difficult, eventually resulting in shortness of breath, even at rest\n### Types Pulmonary Fibrosis\n<img src=\"https://www.nationaljewish.org/getattachment/conditions/Familial-Pulmonary-Fibrosis/Forms/FPF_forms_800v2.png\" width=600, height=300>\n\n### what causes pulmonary fibrosis?\n\n* Some cases of pulmonary fibrosis occur without known cause (this is called idiopathic pulmonary fibrosis). Other cases are caused by exposure to environmental hazards (such as asbestos) and autoimmune diseases (such as rheumatoid arthritis)\n\n* Idiopathic pulmonary fibrosis (IPF) is a disease of unknown etiology with considerable morbidity and mortality. Cigarette smoking is one of the most recognized risk factors for development of IPF. [Furthermore](https://www.hindawi.com/journals/pm/2012/808260/)\n\n### Death rate\n**Idiopathic pulmonary fibrosis (IPF) portends a poor prognosis. With regard to idiopathic pulmonary fibrosis life expectancy, the estimated mean survival is 2-5 years from the time of diagnosis. Estimated mortality rates are 64.3 deaths per million in men and 58.4 deaths per million in women.**\n\nDeath rates in patients with idiopathic pulmonary fibrosis increase with increasing age, are consistently higher in men than women, and experience seasonal variation, with the highest death rates occurring in the winter, even when infectious causes are excluded.\n\n### Treatment\nThere's currently no cure for idiopathic pulmonary fibrosis (IPF). The main aim of treatment is to relieve the symptoms as much as possible and slow down its progression. As the condition becomes more advanced, end of life (palliative) care will be offered.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Let's get into the data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport warnings\nimport seaborn as sns\nimport pandas as pd\nimport numpy as np\nfrom colorama import Fore, Back, Style \nfrom sklearn.model_selection import train_test_split\nfrom sklearn import svm, datasets\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport xgboost\nfrom plotly.offline import plot, iplot, init_notebook_mode\nimport plotly.graph_objs as go\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nfrom statsmodels.formula.api import ols\nimport plotly.graph_objs as gobj\nimport argparse\nimport cv2\nimport cv2 as cv\nimport pydicom as dicom\nfrom pydicom.filereader import dcmread\nimport pydicom\nimport re\nfrom PIL import Image\nfrom IPython.display import Image as show_gif\nfrom PIL import Image\nfrom IPython.display import Image as show_gif\nimport scipy.misc\nimport matplotlib\nfrom skimage import exposure\nimport numpy as np\nimport os\nimport matplotlib.pyplot as plt\nfrom glob import glob\nfrom mpl_toolkits.mplot3d.art3d import Poly3DCollection\nimport scipy.ndimage\nfrom skimage import morphology\nfrom skimage import measure\nfrom skimage.transform import resize\nfrom sklearn.cluster import KMeans\nfrom plotly import __version__\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nfrom plotly.tools import FigureFactory as FF\nimport plotly.figure_factory as ff\nfrom plotly.graph_objs import *\ninit_notebook_mode(connected=True) \nwarnings.filterwarnings(\"ignore\")\n\npd.set_option('display.max_rows', 10000)\npd.set_option('display.max_columns', 10000)\npd.set_option('display.width', 10000)\n\n%matplotlib inline\n\ntrain=pd.read_csv(\"../input/osic-pulmonary-fibrosis-progression/train.csv\")\ntrain.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is the train dataset. there is no missing values\n","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.isnull().mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Age Distribution ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hist_data =[train[\"Age\"].values]\ngroup_labels = ['Age'] \n\nfig = ff.create_distplot(hist_data, group_labels,)\n#fig.update_layout(title_text='Age Distribution plot')\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We can clearly see that patients are above 50 age. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Age distribution Gender wise","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.box(train, x=\"Sex\", y=\"Age\", points=\"all\",)\n#fig.update_layout(\n  #  title_text=\"Gender wise Age Spread - Male = 1224 Female =325\")\nprint(train[\"Sex\"].value_counts())\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Male's count is much higher than Female\n* However age spread is looks similar by gencer","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Analysis in FVC on Smoking Status","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train[\"SmokingStatus\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"smok=train[train[\"SmokingStatus\"]==\"Ex-smoker\"][\"FVC\"]\nnot_smok=train[train[\"SmokingStatus\"]==\"Never smoked\"][\"FVC\"]\ncr_smoke=train[train[\"SmokingStatus\"]==\"Currently smokes\"][\"FVC\"]\nhist_data = [smok,not_smok,cr_smoke]\n\ngroup_labels = ['EX-Smoke', 'Never Smoke',\"Currently Smoke\"]\n\nfig = ff.create_distplot(hist_data, group_labels, bin_size=.2)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Never Smoke - Their spread is not increasing more than 4400-FVC and also more spread in between 1200 to 3400\n* Ex Smoke - the spread is all over the FVC , mostly between 1200 to 4700 \n* Currently smoke - the data is low for this category , so we can not be sure , but it is going higher than the never smoke spread ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Analysis in Percent - Gender and Smoking Status wise ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.violin(train, y=\"Percent\", x=\"Sex\", color=\"SmokingStatus\", box=True,\n          hover_data=train.columns)\n\nfig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Maturation of the airways and lungs continues through childhood and into adolescence during which time, for the most part, males continue to have larger lungs than females\n*  Fibrosis stiffens the lungs, reducing their size and capacity\n* Never Smoke - their Percent is not increased more than 149 both Male an Female wise\n* Those who are currently smoking is increasing their Percent ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Patient wise FVC on weeks","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"dt = train.groupby(by=\"Patient\")[\"Weeks\"].count().reset_index()\ntrain[\"time\"] = 0\n\nfor patient, times in zip(dt[\"Patient\"], dt[\"Weeks\"]):\n    train.loc[train[\"Patient\"] == patient, 'time'] = range(1, times+1)\ndf = px.data.gapminder().query(\"continent != 'Asia'\") # remove Asia for visibility\nfig = px.line(train, x=\"Weeks\", y=\"FVC\", color=\"SmokingStatus\",\n              line_group=\"Patient\",hover_name=\"time\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We can see that Currently smokes value are really changing up and down, even they are taking treatments but still smoking it makes the case much worse\n* If you hover the mouse around the lines you can see that count of the week they have attended.\n* Highest FVC value case is EX-smoke, Lowest is Never Smoke.\n* But we can see most of that differences between Never smoke and Ex-smoker case's FVC values.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## What is Dicom images?\n\nDigital Imaging and Communications in Medicine (DICOM) is the standard for the communication and management of medical imaging information and related data.\n\n#### Let's look into one of the CT-Scan \n\nSample CT-scan of ID00007637202177411956430","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"dataset = dcmread(\"../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/23.dcm\")\nfig = px.imshow(dataset.pixel_array, color_continuous_scale='plasma')\nfig.update_layout(coloraxis_showscale=False)\nfig.update_xaxes(showticklabels=False)\nfig.update_yaxes(showticklabels=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Pulmonary fibrosis is a lung disease that occurs when lung tissue becomes damaged and scarred. This thickened, stiff tissue makes it more difficult for your lungs to work properly\n* So if we focus on slicing the portion of lung tissue with image processing  may the ML model can learn and can give much more accuracy","execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"from PIL import Image\n\ndef img2gif(id_num):\n    tr=train.iloc[id_num,:]\n    d=tr.Patient\n    smoke=tr.SmokingStatus\n    age=tr.Age\n    gender=tr.Sex\n    inputdir = '../input/osic-pulmonary-fibrosis-progression/train/'+ d\n    outdir = './'\n\n    test_list = [ f for f in  os.listdir(inputdir)]\n    tt=[]\n\n    for f in test_list[:]: \n        ds = pydicom.read_file(inputdir +\"/\"+ f) \n        img = ds.pixel_array\n        img=exposure.equalize_adapthist(img)\n\n        plt.axis('off')\n\n        plt.imsave(outdir + f.replace('.dcm','.png'),img,cmap=\"plasma\")\n\n        #cv2.imwrite(outdir + f.replace('.dcm','.png'),img) \n        tt.append(outdir + f.replace('.dcm','.png'))\n    tt.sort(key=lambda f: int(re.sub('\\D', '', f)))\n    im_cnt=len(tt)\n    for i in tt:\n        im_gray = cv2.imread(i)\n        kernel = np.ones((1,1), np.uint8)\n        erosion = cv2.erode(im_gray, kernel, iterations = 1)\n\n        dilation = cv2.dilate(erosion, kernel, iterations = 1)\n        cv2.imwrite(i,dilation)\n\n    new_im=[]\n    for file in tt:\n        new_frame = Image.open(file)\n        new_im.append(new_frame)\n    new_im[0].save(\"./\"+'gif_ok.gif', format='GIF',append_images=new_im[:],save_all=True,duration=400, loop=0)\n    return im_cnt,d,smoke,age,gender\nim_cnt,id_num,smoke,age, gender=img2gif(1)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### DICOM images to Gif","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nprint(\"Image count : \",im_cnt,\"\\nPatient id : \",id_num,\"\\nSmokingStatus : \",smoke, \"\\nAge : \",age,\"\\nGender : \",gender)\nshow_gif(filename=\"gif_ok.gif\", format='png', width=400, height=400)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dilation and Erosion, Morphology","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* Dilation and erosion are two fundamental morphological operations. Dilation adds pixels to the boundaries of objects in an image, while erosion removes pixels on object boundaries.\n* Opening is just another name of erosion followed by dilation. It is useful in removing noise\n* Closing is reverse of Opening, Dilation followed by Erosion. It is useful in closing small holes inside the foreground objects, or small black points on the object","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Let's define our kernel size\nkernel = np.ones((5,5), np.uint8)\nimage=dataset.pixel_array\nimage=exposure.equalize_adapthist(image)\nplt.figure(figsize = (65,35))\nplt.axis('off')\n\nplt.subplot(341)\n\n# Now we erode\nerosion = cv2.erode(image, kernel, iterations = 1)\nplt.axis('off')\nplt.title(\"Erosion\", fontsize=50)\n\nplt.imshow(erosion)\n\nplt.subplot(342, frameon=False)\n\nkernel = np.ones((5,5), np.uint8)\ndilation = cv2.dilate(image, kernel, iterations = 1)\nplt.axis('off')\nplt.title(\"Dilation\", fontsize=50)\n\nplt.imshow(dilation)\n\nplt.subplot(343, frameon=False)\n\n# Opening - Good for removing noise\nopening = cv2.morphologyEx(image, cv2.MORPH_OPEN, kernel)\nplt.axis('off')\nplt.title(\"Opening\", fontsize=50)\n\nplt.imshow(opening)\n\nplt.subplot(344, frameon=False)\n\n# Closing - Good for removing noise\nclosing = cv2.morphologyEx(image, cv2.MORPH_CLOSE, kernel)\nplt.title(\"Closing\", fontsize=50)\nplt.axis('off')\n\nplt.imshow(closing)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Hounsfield unit (HU) \n\n#### Further analysis I'm going to use the case id's which is having much FVC in the data.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"srted=train.sort_values(by=\"FVC\")\nhr=srted.iloc[-1,0]\n\ndata_path = \"../input/osic-pulmonary-fibrosis-progression/train/\"+hr\noutput_path = working_path = \"../input/output/\"\ng = glob(data_path + '/*.dcm')\n\n# Print out the first 5 file names to verify we're in the right folder.\nprint (\"Total of %d DICOM images.\\nFirst 5 filenames:\" % len(g))\nprint ('\\n'.join(g[:5]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### The Hounsfield unit (HU) is a relative quantitative measurement of radio density used by radiologists in the interpretation of computed tomography (CT) images. The absorption/attenuation coefficient of radiation within a tissue is used during CT reconstruction to produce a grayscale image.\n\n<table style=\"width:50%\">\n  <tr>\n    <th>Substance</th>\n    <th>HU</th>\n  </tr>\n  \n \n  \n   <tr>\n <td> Air\t</td>\n  <td>−1000</td>\n  </tr>\n  \n   <tr>\n<td>Lung</td>\n<td>−500</td>\n </tr>\n  <tr>\n<td>Fat</td>\n<td>−100 to −50</td>\n </tr>\n  <tr>\n<td>Water</td>\n<td>0</td>\n </tr>\n  <tr>\n<td>Blood</td>\n<td>+30 to +70</td>\n </tr>\n  <tr>\n<td>Muscle</td>\n<td>+10 to +40</td>\n </tr>\n <tr>\n<td>Liver</td>\n<td>+40 to +60</td>\n <tr>\n  <tr>\n<td>Bone\t</td>\n\n<td>+700 (cancellous bone) to +3000 (cortical bone)</td>\n </tr>\n\n  \n</table>\n\n\n### How to see this histogram ?\n\nBased on the above table we have plotted that in x-axis , to underastand this x axis are the table values how much y - value it has that much of that overall CT-Scan's of that person is covered.\n\n<br>\n\nFor exmaple air value is -1000<br>\nIn the given below histogram you can see that air value is high by y -axis , so There is lots of air\n","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Loop over the image files and store everything into a list.\n\ndef load_scan(path):\n    slices = [dicom.read_file(path + '/' + s) for s in os.listdir(path)]\n    slices.sort(key = lambda x: int(x.InstanceNumber))\n    try:\n        slice_thickness = np.abs(slices[0].ImagePositionPatient[2] - slices[1].ImagePositionPatient[2])\n    except:\n        slice_thickness = np.abs(slices[0].SliceLocation - slices[1].SliceLocation)\n        \n    for s in slices:\n        s.SliceThickness = slice_thickness\n        \n    return slices\n\ndef get_pixels_hu(scans):\n    image = np.stack([s.pixel_array for s in scans])\n\n    image = image.astype(np.int16)\n\n    image[image == -2000] = 0\n    \n    # Convert to Hounsfield units (HU)\n    intercept = scans[0].RescaleIntercept\n    slope = scans[0].RescaleSlope\n    \n    if slope != 1:\n        image = slope * image.astype(np.float64)\n        image = image.astype(np.int16)\n        \n    image += np.int16(intercept)\n    \n    return np.array(image, dtype=np.int16)\n\nid=0\npatient = load_scan(data_path)\nimgs = get_pixels_hu(patient)\nnp.save(\"fullimages_0.npy\", imgs)\nfile_used=\"fullimages_0.npy\"\nimgs_to_process = np.load(file_used).astype(np.float64) \nplt.figure(figsize=(20,6))\nplt.hist(imgs_to_process.flatten(), bins=50, color='c')\nplt.xlabel(\"Hounsfield Units (HU)\")\nplt.ylabel(\"Frequency\")\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"HU's is standardized across all CT-scan , so it can be useful.\n\n* There is lots of air\n* There is some lung\n* Lots of Fat and Water, Blood ,Muscle ,Liver\n* The bone level is low. Not even reached the 3000","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Image Stack","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"imgs_to_process = np.load('fullimages_0.npy')\n\ndef sample_stack(stack, rows=6, cols=6, start_with=10, show_every=6):\n    fig,ax = plt.subplots(rows,cols,figsize=[12,12])\n    for i in range(rows*cols):\n        ind = start_with + i*show_every\n        ax[int(i/rows),int(i % rows)].set_title('slice %d' % ind)\n        ax[int(i/rows),int(i % rows)].imshow(stack[ind],cmap='gray')\n        ax[int(i/rows),int(i % rows)].axis('off')\n    plt.axis('off')\n    plt.show()\n\nsample_stack(imgs_to_process)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Slice Thickness","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print (\"Slice Thickness: %f\" % patient[0].SliceThickness)\nprint (\"Pixel Spacing (row, col): (%f, %f) \" % (patient[0].PixelSpacing[0], patient[0].PixelSpacing[1]))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"###### This means we have 1.0 mm slices, and each voxel represents 0.79 mm.\n\nSlices means?\n\n* Slice thickness and slice increment are central concepts that surround CT/MRI imaging. Slice thickness refers to the (often axial) resolution of the scan (2 mm in the illustration). Slice Increment refers to the movement of the table/scanner for scanning the next slice (varying from 1 mm to 4 mm in the illustration).\n\nVoxle means?\n\n* A voxel is a combination of “volume” and “pixel”, and represents a value on a regular grid in three dimensional space. \n\n* All the slices of ct are only constructed as the 512 x 512\n\n* We can use the DICOM to know about the size of each voxel as the slice thickness\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 3D plotting\n\n#### In order to plot it in 3d , we have to resample each slice into 1x1x1 mm pixels and slices","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"imgs_to_process = np.load('fullimages_0.npy')\ndef resample(image, scan, new_spacing=[1,1,1]):\n    # Determine current pixel spacing\n    spacing = map(float, ([scan[0].SliceThickness] + list(scan[0].PixelSpacing)))\n    spacing = np.array(list(spacing))\n\n    resize_factor = spacing / new_spacing\n    new_real_shape = image.shape * resize_factor\n    new_shape = np.round(new_real_shape)\n    real_resize_factor = new_shape / image.shape\n    new_spacing = spacing / real_resize_factor\n    \n    image = scipy.ndimage.interpolation.zoom(image, real_resize_factor)\n    \n    return image, new_spacing\n\nprint (\"Shape before resampling\\t\", imgs_to_process.shape)\nimgs_after_resamp, spacing = resample(imgs_to_process, patient, [1,1,1])\nprint (\"Shape after resampling\\t\", imgs_after_resamp.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def make_mesh(image, threshold=-300, step_size=1):\n\n    print (\"Transposing surface\")\n    p = image.transpose(2,1,0)\n    \n    print (\"Calculating surface\")\n    verts, faces, norm, val = measure.marching_cubes_lewiner(p, threshold, step_size=step_size, allow_degenerate=True) \n    return verts, faces\n\ndef plotly_3d(verts, faces):\n    x,y,z = zip(*verts) \n    \n    print (\"Drawing\")\n    \n    # Make the colormap single color since the axes are positional not intensity. \n#    colormap=['rgb(255,105,180)','rgb(255,255,51)','rgb(0,191,255)']\n    colormap=['rgb(236, 236, 212)','rgb(236, 236, 212)']\n    \n    fig = FF.create_trisurf(x=x,\n                        y=y, \n                        z=z, \n                        plot_edges=False,\n                        colormap=colormap,\n                        simplices=faces,\n                        backgroundcolor='rgb(64, 64, 64)',\n                        title=\"Interactive Visualization\")\n    iplot(fig)\n\ndef plt_3d(verts, faces):\n    print(\"Drawing\")\n    x,y,z = zip(*verts) \n    fig = plt.figure(figsize=(10, 10))\n    ax = fig.add_subplot(111, projection='3d')\n\n    # Fancy indexing: `verts[faces]` to generate a collection of triangles\n    mesh = Poly3DCollection(verts[faces], linewidths=0.05, alpha=1)\n    face_color = [1, 1, 0.9]\n    mesh.set_facecolor(face_color)\n    ax.add_collection3d(mesh)\n\n    ax.set_xlim(0, max(x))\n    ax.set_ylim(0, max(y))\n    ax.set_zlim(0, max(z))\n    ax.set_facecolor((0.7, 0.7, 0.7))\n    plt.show()\nv, f = make_mesh(imgs_after_resamp, 730, 2)\nplotly_3d(v, f)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Here we have set the Threshold higher , because we are rendering Bones , as we seen in the Hounsfield unit's analysis bones were too low and also slice 1.0 mm slices.\n\n* The marching cubes algorithm is used to generate a 3D mesh from the dataset. The plotly model will utilize a higher step_size with lower voxel threshold to avoid overwhelming the web browser.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Apply Mask\n\nMasking Images. Using an image as a mask. A mask image is simply an image where some of the pixel intensity values are zero, and others are non-zero. Wherever the pixel intensity value is zero in the mask image, then the pixel intensity of the resulting masked image will be set to the background value (normally zero).[know more](http://www.xinapse.com/Manual/masking.html)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def make_lungmask(img, display=False):\n    row_size= img.shape[0]\n    col_size = img.shape[1]\n    \n    mean = np.mean(img)\n    std = np.std(img)\n    img = img-mean\n    img = img/std\n    # Find the average pixel value near the lungs\n    # to renormalize washed out images\n    middle = img[int(col_size/5):int(col_size/5*4),int(row_size/5):int(row_size/5*4)] \n    mean = np.mean(middle)  \n    max = np.max(img)\n    min = np.min(img)\n    # To improve threshold finding, I'm moving the \n    # underflow and overflow on the pixel spectrum\n    img[img==max]=mean\n    img[img==min]=mean\n    #\n    # Using Kmeans to separate foreground (soft tissue / bone) and background (lung/air)\n    #\n    kmeans = KMeans(n_clusters=2).fit(np.reshape(middle,[np.prod(middle.shape),1]))\n    centers = sorted(kmeans.cluster_centers_.flatten())\n    threshold = np.mean(centers)\n    thresh_img = np.where(img<threshold,1.0,0.0)  # threshold the image\n\n    # First erode away the finer elements, then dilate to include some of the pixels surrounding the lung.  \n    # We don't want to accidentally clip the lung.\n\n    eroded = morphology.erosion(thresh_img,np.ones([3,3]))\n    dilation = morphology.dilation(eroded,np.ones([8,8]))\n\n    labels = measure.label(dilation) # Different labels are displayed in different colors\n    label_vals = np.unique(labels)\n    regions = measure.regionprops(labels)\n    good_labels = []\n    for prop in regions:\n        B = prop.bbox\n        if B[2]-B[0]<row_size/10*9 and B[3]-B[1]<col_size/10*9 and B[0]>row_size/5 and B[2]<col_size/5*4:\n            good_labels.append(prop.label)\n    mask = np.ndarray([row_size,col_size],dtype=np.int8)\n    mask[:] = 0\n\n    #\n    #  After just the lungs are left, we do another large dilation\n    #  in order to fill in and out the lung mask \n    #\n    for N in good_labels:\n        mask = mask + np.where(labels==N,1,0)\n    mask = morphology.dilation(mask,np.ones([10,10])) # one last dilation\n\n    if (display):\n        fig, ax = plt.subplots(3, 2, figsize=[12, 12])\n        ax[0, 0].set_title(\"Original\")\n        ax[0, 0].imshow(img, cmap='gray')\n        ax[0, 0].axis('off')\n        ax[0, 1].set_title(\"Threshold\")\n        ax[0, 1].imshow(thresh_img, cmap='gray')\n        ax[0, 1].axis('off')\n        ax[1, 0].set_title(\"After Erosion and Dilation\")\n        ax[1, 0].imshow(dilation, cmap='gray')\n        ax[1, 0].axis('off')\n        ax[1, 1].set_title(\"Color Labels\")\n        ax[1, 1].imshow(labels)\n        ax[1, 1].axis('off')\n        ax[2, 0].set_title(\"Final Mask\")\n        ax[2, 0].imshow(mask, cmap='gray')\n        ax[2, 0].axis('off')\n        ax[2, 1].set_title(\"Apply Mask on Original\")\n        ax[2, 1].imshow(mask*img, cmap='gray')\n        ax[2, 1].axis('off')\n        \n        plt.show()\n    return mask*img\nimg = imgs_after_resamp[230]\nmake_lungmask(img, display=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Here we applied mask to single image\n* It is more often used to reveal some portions\n* In the final image Mask in original , clearly shows it , if we fine tune that abd create an ml model would give somwhat good accuracy.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Mask to all slices","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"masked_lung = []\n\nfor img in imgs_after_resamp:\n    masked_lung.append(make_lungmask(img))\n\nsample_stack(masked_lung, show_every=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"* Pulmonary fibrosis is damages and scarres the lung tissue. So by this mask if we change it little bit we can clearly see the lung tissue. Now we can see little bit of lung tissue.\n* If we do much research we can slice the portion of lung tissue cleaned and contrast. so that can increase ML model's accuracy\n","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"#to remove all png siles in working dir\nimport os\n\nfilelist = [ f for f in os.listdir(\"./\") if f.endswith(\".png\") ]\nfor f in filelist:\n    os.remove(os.path.join(\"./\", f))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n<img src=\"https://quotefancy.com/media/wallpaper/3840x2160/1753004-Desiderius-Erasmus-Quote-Prevention-is-better-than-cure.jpg\" width=400>\n\nThere are currently no established ways to prevent pulmonary fibrosis, particularly since in most cases the cause of the disease cannot be identified.\n\nwhilst\n\n### Factors that Impact PF Prevention\n\n* One of the most common and avoidable risk factors for IPF is smoking, and everyone should quit smoking to avoid not only pulmonary fibrosis, but also many other respiratory diseases. \n* Those who are above 50 can undergo regular medical examinations.\n* In cases where people need to work in contact with toxins and pollutants like silica dust, asbestos fibers, grain dust, and bird and animal droppings, it is important to reduce exposure by using a mask, and make sure regulations regarding these materials are being followed.\n* Age and genetics are also risk factors,  Patients who have family members who are suffering from or have had pulmonary fibrosis can undergo genetic testes, which can help in disease prevention.\n* Open Source Imaging Consortium (OSIC) can also make much more aware of these all.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## would continue.....","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}