{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"\nThe word “pulmonary” means lung and the word “fibrosis” means scar tissue— similar to scars that you may have on your skin from an old injury or surgery. So, in its simplest sense, pulmonary fibrosis (PF) means scarring in the lungs. Over time, the scar tissue can destroy the normal lung and make it hard for oxygen to get into your blood. Low oxygen levels (and the stiff scar tissue itself) can cause you to feel short of breath, particularly when walking and exercising. Pulmonary fibrosis isn’t just one disease. It is a family of more than 200 different lung diseases that all look very much alike. The PF family of lung diseases falls into an even larger group of diseases called the interstitial lung diseases (also known as ILD), which includes all of the diseases that have inflammation and/or scarring in the lung. Some interstitial lung diseases don’t include scar tissue. When an interstitial lung disease does include scar tissue in the lung, we call it pulmonary fibrosis.\n\nNo one is certain how many people are affected by PF. One recent study estimated that idiopathic pulmonary fibrosis (or IPF, which is just one of more than 200 types of PF) affects 1 out of 200 adults over the age of 60 in the United States—that translates to more than 200,000 people living with PF today. Approximately 50,000 new cases are diagnosed each year and as many as 40,000 Americans die from IPF each year.\n\nRef\n\n
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"My goal in this notebook is to explore CSVs as well as dicom files to get a general idea of what kind of data we are dealing with and to have a hunch about their behavior. enjoy =)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Contents\n\n* [Libraries For Fun](#1)\n* [Dicom Files](#2)\n * [1. Image Type](#2.1)\n * [2. Manufacturer / Manufacturer's model name](#2.2)\n * [3. Slice Thickness](#2.3)\n * [4. KVP](#2.4)\n * [5. Spacing Between Slices](#2.5)\n * [6. Table Height](#2.6)\n * [7. Convolution Kernel](#2.7)\n * [8. Patient Position](#2.8)\n * [9. Instance Number](#2.9)\n * [10. Image Position & Image Orientation (Patient) ](#2.10)\n * [11. Position Reference Indicator ](#2.11)\n * [12. Slice Location Attribute](#2.12)\n * [13. Rows & Columns](#2.13)\n * [14. Pixel Spacing Attribute](#2.14)\n * [15. Bits Stored & High Bit](#2.15)\n * [16. Pixel Representation Attribute](#2.16)\n * [17. Window Center & Window Width ](#2.17)\n * [18. Rescale Intercept & Rescale Slope](#2.18)\n * [19. Images](#2.19)\n* [Train and Test](#3)\n * [1. Smoking Status ](#3.1)\n * [2. Sex ](#3.2)\n * [3. Age](#3.3)\n * [4. FVC & Percentage](#3.4)\n * [5. Heatmap](#3.5)\n* [Conclusion](#4)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Libraries For Fun ","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport pydicom\nfrom pydicom import dcmread\nimport glob, os\nfrom collections import defaultdict\nimport tqdm\nimport gc\nimport seaborn as sns\nimport ast\nimport plotly.express as px\nfrom pandas_profiling import ProfileReport \npd.options.display.max_columns = None\nimport cv2\n\nfrom collections import defaultdict\nimport collections\nimport imageio\nfrom IPython.display import HTML\n\nimport plotly.offline as pyo\nfrom scipy import ndimage, misc\nimport warnings\nwarnings.filterwarnings('ignore')\n\npyo.init_notebook_mode()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dicom Files ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First we gonna explore dicom files.\n\n

Digital Imaging and Communications in Medicine (DICOM) is the standard for the communication and management of medical imaging information and related data.[1] DICOM is most commonly used for storing and transmitting medical images enabling the integration of medical imaging devices such as scanners, servers, workstations, printers, network hardware, and picture archiving and communication systems (PACS) from multiple manufacturers. It has been widely adopted by hospitals and is making inroads into smaller applications like dentists' and doctors' offices.\n

\n\n

DICOM files can be exchanged between two entities that are capable of receiving image and patient data in DICOM format. The different devices come with DICOM Conformance Statements which state which DICOM classes they support. The standard includes a file format definition and a network communications protocol that uses TCP/IP to communicate between systems.\n

\n\n

The National Electrical Manufacturers Association (NEMA) holds the copyright to the published standard[2] which was developed by the DICOM Standards Committee, whose members[3] are also partly members of NEMA.[4] It is also known as NEMA standard PS3, and as ISO standard 12052:2017 \"Health informatics -- Digital imaging and communication in medicine (DICOM) including workflow and data management\".\n

\n\nRef","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"A preview of a random dicom file :","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pydicom.read_file(\"../input/osic-pulmonary-fibrosis-progression/train/ID00213637202257692916109/29.dcm\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\nTo handle dicom files well I decided to insert them in a pandas dataframe where each column represents Attribute stored in the dicom files except the images and the given array like 'ImageType' or 'PixelSpacing' will be split into several columns and each dataframe line represents a dicom file.\n\nAs shown in the image on the right.\n\nI merged the result with the csv train.\n\nI did this whole process on another notebook because it takes a long time to run.\n\nMy notebook to prepare the data: OSIC : Transform DICOM into DataFrame\n\n\n
","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\nData = pd.read_pickle('../input/osic-transform-dicom-into-dataframe/output_data.pkl')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train and test data","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Head :')\n\ndisplay(Data.head())\n\nprint('Info :')\n\nData.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Image Type ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n
\n
\n
\n
\nImage Type
\n
\n
\n
\n

\nImage Type (0008,0008) identifies important image identification characteristics. These characteristics are:

\n
\n
    \n
  1. \n

    \nPixel Data Characteristics

    \n
    \n
      \n
    1. \n

      \nis the image an ORIGINAL Image; an image whose pixel values are based on original or source data

      \n
      2. \n
    2. \n

      \nis the image a DERIVED Image; an image whose pixel values have been derived in some manner from the pixel value of one or more other images

      \n
      3. \n
    \n
    \n
    2. \n
  2. \n

    \nPatient Examination Characteristics

    \n
    \n
      \n
    1. \n

      \nis the image a PRIMARY Image; an image created as a direct result of the patient examination

      \n
      2. \n
    2. \n

      \nis the image a SECONDARY Image; an image created after the initial patient examination

      \n
      3. \n
    \n
    \n
    3. \n
  3. \n

    \nModality Specific Characteristics

    \n
    4. \n
  4. \n

    \nImplementation specific identifiers; other implementation specific identifiers shall be documented in an implementation's conformance statement.

    \n
    5. \n
\n
\n

\nThe Image Type Attribute is multi-valued and shall be provided in the following manner:

\n
\n
    \n
  1. \n

    \nValue 1 shall identify the Pixel Data Characteristics

    \n
    \n

    \nEnumerated Values:\n

    \n
    \n
    \nORIGINAL\n
    \n
    \n

    \nidentifies an Original Image

    \n
    \n
    \nDERIVED\n
    \n
    \n

    \nidentifies a Derived Image

    \n
    \n
    \n
    \n
    2. \n
  2. \n

    \nValue 2 shall identify the Patient Examination Characteristics

    \n
    \n

    \nEnumerated Values:\n

    \n
    \n
    \nPRIMARY\n
    \n
    \n

    \nidentifies a Primary Image

    \n
    \n
    \nSECONDARY\n
    \n
    \n

    \nidentifies a Secondary Image

    \n
    \n
    \n
    \n
    3. \n
  3. \n

    \nValue 3 shall identify any Image IOD specific specialization (optional)

    \n
    4. \n
  4. \n

    \nOther Values that are implementation specific (optional)

    \n
    5. \n
\n
\n

\nAny of the optional values (value 3 and beyond) may be encoded either with a value or zero-length, independent of other optional values, unless otherwise specified by a specialization of this Attribute in an IOD.

\n

\nIf the pixel data of the derived Image is different from the pixel data of the source images and this difference is expected to affect professional interpretation of the image, the Derived Image shall have a UID different than all the source images.

\n
\n\nImage Type","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"type_dict_all = ['ORIGINAL', 'PRIMARY', 'AXIAL', 'CT_SOM5 SPI', 'HELIX', 'CT_SOM5 SEQ', 'SECONDARY', 'DERIVED', 'JP2K LOSSY 6:1', 'VOLUME', 'OTHER', 'CSA MPR', 'CSAPARALLEL', \n 'CSA RESAMPLED', 'REFORMATTED', 'AVERAGE', 'CT_SOM7 SPI DUAL', 'STD', 'SNRG', 'DET_AB']\nsns.set(rc={'figure.figsize':(15,7.5)})\nplt.xticks(rotation=45)\nax = sns.barplot(y=0, x = Data[type_dict_all].sum().to_frame().sort_values(0, ascending=False).index, data=Data[type_dict_all].sum().to_frame().sort_values(0, ascending=False))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data.groupby('Manufacturer')[type_dict_all].sum()\ntmp = pd.melt(tmp.reset_index(), id_vars=['Manufacturer'])\ntmp.columns = ['Manufacturer','ImageType', 'Value']\nsns.factorplot(x='Manufacturer', y='Value', data=tmp, kind='bar' , hue = 'ImageType', size=10, aspect=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The three most common types of images are axial, original and primary and they are generated by Siemence, Toshiba, Philips and GE Medical Systems.\n\nHowever, it is important to specify that Philips generates images of type CT_SOMS SPI and Siemence generates images of type HELIX.\n\nWe will see the marks in the next section.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2. Manufacturer / Manufacturer's model name ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Manufacturer of the equipment that produced the Composite Instances.\nManufacturer's model name of the equipment that is to be used for beam delivery.\n\nManufacturer / Manufacturer's model name ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30,10))\n\nplt.subplot(1,2,1)\n\nsns.set(rc={'figure.figsize':(15,7.5)})\ntmp = Data['Manufacturer'].value_counts(ascending=False).to_frame().reset_index()\ntmp\nax1 = sns.barplot(y='index', x = 'Manufacturer', data=tmp)\n\nplt.subplot(1,2,2)\n\nsns.set(rc={'figure.figsize':(15,7.5)})\ntmp = Data['ManufacturerModelName'].value_counts(ascending=False).to_frame().reset_index()\ntmp\nax2 = sns.barplot(y='index', x = 'ManufacturerModelName', data=tmp)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data.groupby(['Manufacturer','ManufacturerModelName']).count()['PatientID'].to_frame().reset_index()\nsns.factorplot(x='Manufacturer', y='PatientID', data=tmp, kind='bar' , hue = 'ManufacturerModelName', size=10 , aspect=3 )#, palette=tmp['ManufacturerModelName'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nMost popular Manufacturer ModelName:\n\n1. TOSHIBA\tAquilion\n2. GE MEDICAL SYSTEMS\tLightSpeed VCT\t\n3. GE MEDICAL SYSTEMS\tOsiriX\n4. SIEMENS\tSensation 16\n5. SIEMENS\tOsiriX\t\n\nMost popular Model Name for each Manufacturer :\n\n* TOSHIBA Aquilion (Image in the left)\n* GE MEDICAL SYSTEMS LightSpeed VCT\n* SIEMENS Sensation 16\n* Philips Brilliance 64\n* Hitachi Medical Corporation\tECLOS\t\n* PACSGEAR\tLightSpeed VCT\n* PACSMATT\tOsiriX\t\n
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3. Slice Thickness ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Nominal slice thickness, in mm.\n\nSlice Thickness","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data['SliceThickness'].astype('float').value_counts().to_frame().reset_index().sort_values(by = 'index')\ntmp.columns = ['SliceThickness','Count']\nax = sns.barplot(y='Count', x = 'SliceThickness', data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mostly 1 mm and sometimes 1.25, 0.625 and 0.5","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4. KVP ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Peak kilo voltage output of the X-Ray generator used.\n\nKVP","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data['KVP'].astype('float').value_counts().to_frame().reset_index().sort_values(by = 'index')\ntmp.columns = ['KVP','Count']\nax = sns.barplot(y='Count', x = 'KVP', data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mostly 120 KVP","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 5. Spacing Between Slices ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Spacing between slices, in mm, measured from center-to-center of each slice along the normal to the first image. The sign of the Spacing Between Slices (0018,0088) determines the direction of stacking. The normal is determined by the cross product of the direction cosines of the first row and first column of the first frame, such that a positive spacing indicates slices are stacked behind the first slice and a negative spacing indicates slices are stacked in front of the first slice. See Image Orientation (0020,0037) in the NM Detector Module.\n\nSpacing Between Slices","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data['SpacingBetweenSlices'].astype('float').value_counts().to_frame().reset_index().sort_values(by = 'index')\ntmp.columns = ['SpacingBetweenSlices','Count']\nax = sns.barplot(y='Count', x = 'SpacingBetweenSlices', data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mostly 0 mm","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 6. Table Height ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The distance in mm of the top of the patient table to the center of rotation; below the center is positive.\n\nTable Height","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ig, ax = plt.subplots()\n\ntmp = Data['TableHeight'].astype('float').value_counts().to_frame().reset_index().sort_values(by = 'index')\nsns.set(rc={'figure.figsize':(20,10)})\nsns.distplot(tmp[\"TableHeight\"])\n\nax2 = plt.axes([0.7, 0.5, .15, .3], facecolor='y')\nax2 = sns.violinplot(y=tmp[\"TableHeight\"], ax=ax2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distance mainly between 0 and 1000 mm and there are rare cases where this distance is greater.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 7. X-Ray Tube Current ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"X-Ray Tube Current in mA.\n\nX-Ray Tube Current","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ig, ax = plt.subplots()\n\ntmp = Data['XRayTubeCurrent'].astype('float').value_counts().to_frame().reset_index().sort_values(by = 'index')\nsns.set(rc={'figure.figsize':(20,10)})\nsns.distplot(tmp[\"XRayTubeCurrent\"])\n\nax2 = plt.axes([0.7, 0.5, .15, .3], facecolor='y')\nax2 = sns.violinplot(y=tmp[\"XRayTubeCurrent\"], ax=ax2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mostly between 0 and 500 mA","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 7. Convolution Kernel ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"A label describing the convolution kernel or algorithm used to reconstruct the data\n\nConvolution Kernel","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data['ConvolutionKernel'].value_counts().to_frame().reset_index().sort_values(by = 'ConvolutionKernel',ascending = False)\ntmp.columns = ['Convolution Kernel','Count']\nplt.xticks(rotation=45)\nax = sns.barplot(y='Count', x = 'Convolution Kernel', data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Top 5 most used Convolution Kernel :\n\n* LUNG\n* C\n* B70f\n* BONEPLUS \n* FC01","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 8. Patient Position ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n
\n
\n
\n
\n Patient Position
\n
\n
\n
\n

\nPatient Position (0018,5100) specifies the position of the patient relative to the imaging equipment space. This Attribute is intended for annotation purposes only. It does not provide an exact mathematical relationship of the patient to the imaging equipment.

\n

\nWhen multiple subjects are present in the same image, and arranged with different positions, then the Patient Position (0018,5100) in the General Series Module is nominal, does not apply to each subject, but does define the relationship of the nominal Patient-Based Coordinate System to the machine.

\n
\n

Note

\n

\nIn conjunction with the Patient Position (0018,5100) in each Item of the Group of Patients Identification Sequence (0010,0027), Patient Position (0018,5100) in the General Series Module may be helpful to compute patient-relative spatial information for each subject from the Attributes of the Image Plane Module.

\n
\n

\nWhen facing the front of the imaging equipment, Head First is defined as the patient's head being positioned toward the front of the imaging equipment (i.e., head entering the front of the equipment). Feet First is defined as the patient's feet being positioned toward the front of the imaging equipment (i.e., feet entering the front of the equipment). Left First is defined as the patient's left side being positioned towards the front of the imaging equipment (i.e., patient's left side entering the front of the equipment). Right First is defined as the patient's right being positioned towards the front of the imaging equipment (i.e., patient's right side entering the front of the equipment). Prone is defined as the patient's face being positioned in a downward (gravity) direction. Supine is defined as the patient's face being in an upward direction. Decubitus Right is defined as the patient's right side\n being in a downward direction. Decubitus Left is defined as the patient's left side being in a downward direction.

\n
\n

\nDefined Terms:\n

\n
\n
\nHFP\n
\n
\n

\nHead First-Prone

\n
\n
\nHFS\n
\n
\n

\nHead First-Supine

\n
\n
\nHFDR\n
\n
\n

\nHead First-Decubitus Right

\n
\n
\nHFDL\n
\n
\n

\nHead First-Decubitus Left

\n
\n
\nFFDR\n
\n
\n

\nFeet First-Decubitus Right

\n
\n
\nFFDL\n
\n
\n

\nFeet First-Decubitus Left

\n
\n
\nFFP\n
\n
\n

\nFeet First-Prone

\n
\n
\nFFS\n
\n
\n

\nFeet First-Supine

\n
\n
\nLFP\n
\n
\n

\nLeft First-Prone

\n
\n
\nLFS\n
\n
\n

\nLeft First-Supine

\n
\n
\nRFP\n
\n
\n

\nRight First-Prone

\n
\n
\nRFS\n
\n
\n

\nRight First-Supine

\n
\n
\nAFDR\n
\n
\n

\nAnterior First-Decubitus Right

\n
\n
\nAFDL\n
\n
\n

\nAnterior First-Decubitus Left

\n
\n
\nPFDR\n
\n
\n

\nPosterior First-Decubitus Right

\n
\n
\nPFDL\n
\n
\n

\nPosterior First-Decubitus Left

\n
\n
\n
\n
\n

Note

\n
\n
    \n
  1. \n

    \nFor quadrupeds, separate concepts for ventral and dorsal are not introduced, rather it is expected that anterior and posterior will be considered synonymous as they are when applied to the trunk.

    \n
    2. \n
  2. \n

    \nThere are no decubitus variants of left or right first, since for imaging equipment that is aligned horizontally with respect to gravity the patient cannot be both decubitus and have the left or right side towards the front of the imaging equipment.

    \n
    3. \n
  3. \n

    \nThere are no prone or supine variants of anterior or posterior first, since for imaging equipment that is aligned horizontally with respect to gravity the patient cannot be prone or supine and have the anterior or posterior side towards the front of the imaging equipment.

    \n
    4. \n
\n
\n
\n

\nThe Figure C.7.3.1.1.2-1 illustrates some of these Defined Terms for imaging equipment with a table, such as in X-Ray Angiography. The orientation of the patient related to gravity is always recumbent.

\n
\n
\n
\n\n
\n
\n

\nFigure C.7.3.1.1.2-1. Representation of the Eight Different Patient Positions on the X-Ray Table\n

\n
\n
\n
\n
\n
\n\n
\n
\n

\nFigure C.7.3.1.1.2-2. Example of Right First-Prone (RFP) Patient Position Relative to the Gantry and Table for a Small Animal\n

\n
\n
\n
\n\nPatient Position","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data['PatientPosition'].value_counts().to_frame().reset_index().sort_values(by = 'index')\ntmp.columns = ['PatientPosition','Count']\nplt.xticks(rotation=45)\nax = sns.barplot(y='Count', x = 'PatientPosition', data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most used positions :\n \n* HFS : Head First-Supine \n\n* FFS : Feet First-Supine\n
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 9. Instance Number ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"number that identifies this image, This Attribute was named Image Number in earlier versions of this Standard.\n\nInstance Number","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ig, ax = plt.subplots()\n\ntmp = Data['InstanceNumber'].astype('float').value_counts().to_frame().reset_index().sort_values(by = 'index')\nsns.set(rc={'figure.figsize':(20,10)})\nsns.distplot(tmp[\"InstanceNumber\"])\n\nax2 = plt.axes([0.7, 0.5, .15, .3], facecolor='y')\nax2 = sns.violinplot(y=tmp[\"InstanceNumber\"], ax=ax2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mostly between 0 and 100 but whatever number doesn't matter it just gives the number of the picture it doesn't give any information about pulmonary fibrosis.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 10. Image Position & Image Orientation (Patient) ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Image Position is the x, y, and z coordinates of the upper left hand corner (center of the first voxel transmitted) of the image, in mm. See Section C.7.6.2.1.1 for further explanation.\n\nImage Orientation is the direction cosines of the first row and the first column with respect to the patient. See Section C.7.6.2.1.1 for further explanation.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"

Section 

\n
\n
\n
\n
\n Image Position and Image Orientation
\n
\n
\n
\n

\nImage Position (0020,0032) specifies the x, y, and z coordinates of the upper left hand corner of the image; it is the center of the first voxel transmitted. Image Orientation (0020,0037) specifies the direction cosines of the first row and the first column with respect to the patient. These Attributes shall be provide as a pair. Row value for the x, y, and z axes respectively followed by the Column value for the x, y, and z axes respectively.

\n

\nThe direction of the axes is defined fully by the patient's orientation.

\n

\nIf Anatomical Orientation Type (0010,2210) is absent or has a value of BIPED, the x-axis is increasing to the left hand side of the patient. The y-axis is increasing to the posterior side of the patient. The z-axis is increasing toward the head of the patient.

\n

\nIf Anatomical Orientation Type (0010,2210) has a value of QUADRUPED, the

\n
\n
    \n
  • \n

    \nx-axis is increasing to the left (as opposed to right) side of the patient

    \n
    • \n
  • \n

    \nthe y-axis is increasing towards

    \n
    \n
      \n
    • \n

      \nthe dorsal (as opposed to ventral) side of the patient for the neck, trunk and tail,

      \n
      • \n
    • \n

      \nthe dorsal (as opposed to ventral) side of the patient for the head,

      \n
      • \n
    • \n

      \nthe dorsal (as opposed to plantar or palmar) side of the distal limbs,

      \n
      • \n
    • \n

      \nthe cranial (as opposed caudal) side of the proximal limbs, and

      \n
      • \n
    \n
    \n
    • \n
  • \n

    \nthe z-axis is increasing towards

    \n
    \n
      \n
    • \n

      \nthe cranial (as opposed to caudal) end of the patient for the neck, trunk and tail,

      \n
      • \n
    • \n

      \nthe rostral (as opposed to caudal) end of the patient for the head, and

      \n
      • \n
    • \n

      \nthe proximal (as opposed to distal) end of the limbs

      \n
      • \n
    \n
    \n
    • \n
\n
\n
\n

Note

\n
\n
    \n
  1. \n

    \nThe axes for quadrupeds are those defined and illustrated in Smallwood et al for proper anatomic directional terms as they apply to various parts of the body.

    \n
    2. \n
  2. \n

    \nIt should be anticipated that when quadrupeds are imaged on human equipment, and particularly when they are position in a manner different from the traditional human prone and supine head or feet first longitudinal position, then the equipment may well not indicate the correct orientation, though it will remain an orthogonal Cartesian right-handed system that could be corrected subsequently.

    \n
    3. \n
\n
\n
\n

\nThe Patient-Based Coordinate System is a right handed system, i.e., the vector cross product of a unit vector along the positive x-axis and a unit vector along the positive y-axis is equal to a unit vector along the positive z-axis.

\n
\n

Note

\n

\nIf a patient is positioned parallel to the ground, in dorsal recumbency (i.e., for humans, face-up on the table), with the caudo-cranial (i.e., for humans, feet-to-head) direction the same as the front-to-back direction of the imaging equipment, the direction of the axes of this Patient-Based Coordinate System and the Equipment-Based Coordinate System in previous versions of this Standard will coincide.

\n
\n

\nThe Image Plane Attributes, in conjunction with the Pixel Spacing Attribute, describe the position and orientation of the image slices relative to the Patient-Based Coordinate System. In each image frame Image Position (Patient) (0020,0032) specifies the origin of the image with respect to the Patient-Based Coordinate System. RCS and Image Orientation (Patient) (0020,0037) values specify the orientation of the image frame rows and columns. The mapping of pixel location (i,j) to the RCS is calculated as follows:

\n

\n

\n
\n

\nEquation . \n

\n
\n\n
\n
\n

\n
\n

\n

\nWhere:

\n
\n
    \n
  • \n

    \nPxyz The coordinates of the voxel (i,j) in the frame's image plane in units of mm.

    \n
    • \n
  • \n

    \nSxyz The three values of Image Position (Patient) (0020,0032). It is the location in mm from the origin of the RCS.

    \n
    • \n
  • \n

    \nXxyz The values from the row (X) direction cosine of Image Orientation (Patient) (0020,0037).

    \n
    • \n
  • \n

    \nYxyz The values from the column (Y) direction cosine of Image Orientation (Patient) (0020,0037).

    \n
    • \n
  • \n

    \ni Column index to the image plane. The first column is index zero.

    \n
    • \n
  • \n

    \nΔi\n Column pixel resolution of Pixel Spacing (0028,0030) in units of mm.

    \n
    • \n
  • \n

    \nj Row index to the image plane. The first row index is zero.

    \n
    • \n
  • \n

    \nΔj\n Row pixel resolution of Pixel Spacing (0028,0030) in units of mm.

    \n
    • \n
\n
\n

\nAdditional constraints apply:

\n
\n
    \n
  1. \n

    \nThe row and column direction cosine vectors shall be orthogonal, i.e., their dot product shall be zero.

    \n
    2. \n
  2. \n

    \nThe row and column direction cosine vectors shall be normal, i.e., the dot product of each direction cosine vector with itself shall be unity.

    \n
    3. \n
\n
\n
\n\n\nImage Position & Image Orientation (Patient)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":" \n\nThen the 3 parameters which give the position of the upper left voxel of the image vary as shown in the image on the right.\ncorrect me if im wrong.\n\nlet's see the position of the coordinates on the position of the image in a 3d space:\n
","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.scatter_3d(Data, x='ImagePositionPatient_x', y='ImagePositionPatient_y', z='ImagePositionPatient_z', color='PatientID')\n# fig.update_layout(autosize=False,\n# scene_camera_eye=dict(x=1.87, y=0.88, z=-0.64),\n# width=500, height=500,\n# margin=dict(l=65, r=50, b=65, t=90)\n# )\nfig.update_traces(marker=dict(size=5,\n line=dict(width=0,\n color='DarkSlateGrey')),\n selector=dict(mode='markers'))\nfig.update_layout(showlegend=False) \nfig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each color represents a distinct patient.\n\nFirst we can clearly see that the scanner images were taken one below the other successively with the aim of scanning up and down or vice versa a presize area of the lungs.\n\nSo the position of the images relative to the body are as follows:\n\n\n\nForgive me for this abominable drawing but the idea is there ^_^'.\n\nnow that we know the position of the images in relation to the human body, let's look at the direction.\n\nIf we display the axes of the director cosine on each position of the image we get this.","execution_count":null},{"metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp1 = Data[['ImagePositionPatient_x','ImagePositionPatient_y', 'ImagePositionPatient_z', 'ImageOrientationPatient_a','ImageOrientationPatient_b', 'ImageOrientationPatient_c']]\ntmp1.columns = ['x','y','z','a','b','c']\n\ntmp1['Cos'] = 'red'\ntmp2 = Data[['ImagePositionPatient_x','ImagePositionPatient_y', 'ImagePositionPatient_z', 'ImageOrientationPatient_d','ImageOrientationPatient_e', 'ImageOrientationPatient_f']]\ntmp2.columns = ['x','y','z','a','b','c']\ntmp2['Cos'] = 'blue'\n\ncos = pd.concat([tmp1, tmp2], ignore_index = True)\ncos['width'] = 10\n\ncos[['a','b','c']] = cos[['a','b','c']] * 200\n\nfig = plt.figure()\nax = fig.gca(projection='3d')\nax.view_init(60, 35)\nax.quiver(cos['x'], cos['y'], cos['z'], cos['a'], cos['b'], cos['c'], length=0.1, colors = cos['Cos'])\n\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure()\nax = fig.gca(projection='3d')\n\nax.quiver(cos['x'], cos['y'], cos['z'], cos['a'], cos['b'], cos['c'], length=0.1, colors = cos['Cos'])\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We notice very well that all the images go in a direction anterior to the chest towards the lungs of the patient which is logical.\n\n \n\nLet's take a look at the vectors that indicate the direction of the image.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=3)\nData['ImageOrientationPatient_a'].plot.hist(title = 'Alpha 1', ax=axes[0,0])\nData['ImageOrientationPatient_b'].plot.hist(title = 'Beta 1', ax=axes[0,1])\nData['ImageOrientationPatient_c'].plot.hist(title = 'Gamma 1', ax=axes[0,2])\nData['ImageOrientationPatient_d'].plot.hist(title = 'Alpha 2', ax=axes[1,0])\nData['ImageOrientationPatient_e'].plot.hist(title = 'Beta 2', ax=axes[1,1])\nData['ImageOrientationPatient_f'].plot.hist(title = 'Gamma 2', ax=axes[1,2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can clearly see that the only two axes which influence the direction of the image are alpha 1 and beta 2 which explains the converging direction of the images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 11. Position Reference Indicator ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n
\n
\n
\n
\nC.7.4.1.1.2 Position Reference Indicator
\n
\n
\n
\n

\nThe Position Reference Indicator (0020,1040) specifies the part of the imaging target that was used as a reference point associated with a specific Frame of Reference UID. The Position Reference Indicator may or may not coincide with the origin of the fixed Frame of Reference related to the Frame of Reference UID.

\n

\nFor a Patient-related Frame of Reference, this is an anatomical reference point such as the iliac crest, orbital-medial, sternal notch, symphysis pubis, xiphoid, lower costal margin, or external auditory meatus, or a fiducial marker placed on the patient. The Patient-Based Coordinate System is described in Section C.7.6.2.1.1.

\n

\nFor a slide-related Frame of Reference, this is the slide corner as specified in Section C.8.12.2.1 and shall be identified in this Attribute with the value \"SLIDE_CORNER\". The slide-based coordinate system is described in Section C.8.12.2.1.

\n

\nFor an Ophthalmic Coordinate System, the Frame of Reference is based upon the corneal vertex. The corneal vertex is determined by the measuring instrument and shall be identified in this Attribute with the value CORNEAL_VERTEX_R (for the right eye) or CORNEAL_VERTEX_L (for the left eye). The Ophthalmic Coordinate System is described in Section C.8.30.3.1.4.

\n

\nThe Position Reference Indicator shall be used only for annotation purposes and is not intended to be used as a mathematical spatial reference.

\n
\n

Note

\n

\nThe Position Reference Indicator may be encoded as zero length when it has no meaning, for example, when the Frame of Reference Module is required to relate mammographic images of the breast acquired without releasing breast compression, but where there is no meaningful anatomical reference point as such.

\n
\n
\n\nPosition Reference Indicator","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data.PositionReferenceIndicator.value_counts().to_frame().reset_index()\ntmp.columns = ['Position Reference Indicator','Count']\n\nsns.barplot(x=\"Position Reference Indicator\", y=\"Count\", data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The majority of Position Reference Indicator are of no specific value.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 12. Slice Location Attribute ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Slice Location is defined as the relative position of the image plane expressed in mm. This information is relative to an unspecified implementation specific reference point.\n\nSlice Location Attribute","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Data.SliceLocation.astype(float).value_counts()\n\ntmp = Data['SliceLocation'].astype('float').value_counts().to_frame().reset_index().sort_values(by = 'index')\nsns.set(rc={'figure.figsize':(20,10)})\nsns.distplot(tmp[\"SliceLocation\"])\n\nax2 = plt.axes([0.7, 0.5, .15, .3], facecolor='y')\nax2 = sns.violinplot(y=tmp[\"SliceLocation\"], ax=ax2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The image does not deviate too much from its origin, it is between 0 mm and 200 mm","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 13. Rows & Columns ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Rows :\n\nNumber of rows in the image.\n\nShall be an exact multiple of the vertical downsampling factor if any of the samples (planes) are encoded downsampled in the vertical direction for pixel data encoded in a Native (uncompressed) format. E.g., required to be an even value for a Photometric Interpretation (0028,0004) of YBR_FULL_422.\n\nColumns :\n\nNumber of columns in the image.\n\nShall be an exact multiple of the horizontal downsampling factor if any of the samples (planes) are encoded downsampled in the horizontal direction for pixel data encoded in a Native (uncompressed) format. E.g., required to be an even value for a Photometric Interpretation (0028,0004) of YBR_FULL_422.\n\n\nRows & Columns","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.jointplot(x=\"Rows\", y=\"Columns\", data=Data[['Rows','Columns']].astype('int'), kind='reg',joint_kws={'color':'green'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a strong correlation between columns and rows except in a few cases where the images are rectangular","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 14. Pixel Spacing Attribute ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Physical distance in the patient between the center of each pixel, specified by a numeric pair - adjacent row spacing (delimiter) adjacent column spacing in mm.\n\n
\n
\n
\n
\n

\n10.7.1.3 Pixel Spacing Value Order and Valid Values

\n
\n
\n
\n

\nAll pixel spacing related Attributes are encoded as the physical distance between the centers of each two-dimensional pixel, specified by two numeric values.

\n

\nThe first value is the row spacing in mm, that is the spacing between the centers of adjacent rows, or vertical spacing.

\n

\nThe second value is the column spacing in mm, that is the spacing between the centers of adjacent columns, or horizontal spacing.

\n

\nTo illustrate, consider the example shown in Figure 10.7.1.3-1.

\n

\n

\n
\n
\n
\n\n
\n
\n

\nFigure 10.7.1.3-1. Example of Pixel Spacing Value Order\n

\n
\n

\n
\n

\n

\nPixel Spacing = Row Spacing \\ Column Spacing = 0.30\\0.25.

\n

\nAll pixel spacing related Attributes shall have positive non-zero values, except when there is only a single row or column or pixel of data present, in which case the corresponding value may be zero.

\n
\n

Note

\n

\nA single row or column or \"pixel\" may occur in MR Spectroscopy Instances.

\n
\n

\nThis description applies to:

\n
\n
    \n
  • \n

    \nPixel Spacing (0028,0030)

    \n
    • \n
  • \n

    \nImager Pixel Spacing (0018,1164)

    \n
    • \n
  • \n

    \nNominal Scanned Pixel Spacing (0018,2010)

    \n
    • \n
  • \n

    \nImage Plane Pixel Spacing (3002,0011)

    \n
    • \n
  • \n

    \nCompensator Pixel Spacing (300A,00E9)

    \n
    • \n
  • \n

    \nDetector Element Spacing (0018,7022)

    \n
    • \n
  • \n

    \nPresentation Pixel Spacing (0070,0101)

    \n
    • \n
  • \n

    \nPrinter Pixel Spacing (2010,0376)

    \n
    • \n
  • \n

    \nObject Pixel Spacing in Center of Beam (0018,9404)

    \n
    • \n
\n
\n
\n\nPixel Spacing Attribute","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.jointplot(x=\"PixelSpacing_row\", y=\"PixelSpacing_column\", data=Data[['PixelSpacing_row','PixelSpacing_column']].astype('float'),kind='reg',joint_kws={'color':'green'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Row spacing and column spacing are same","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 15. Bits Stored & High Bit ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Bits Stored is the number of bits stored for each pixel sample. Each sample shall have the same number of bits stored.\n\nBits Stored Attribute\n\nHigh Bit is the most significant bit for pixel sample data. Each sample shall have the same high bit. High Bit (0028,0102) shall be one less than Bits Stored (0028,0101).\n\nHigh Bit","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30,10))\n\nplt.subplot(1,2,1)\ntmp = Data.BitsStored.value_counts().to_frame().reset_index()\ntmp.columns = ['Bits Stored','Count']\nsns.barplot(x=\"Bits Stored\", y=\"Count\", data=tmp)\n\nplt.subplot(1,2,2)\ntmp = Data.HighBit.value_counts().to_frame().reset_index()\ntmp.columns = ['High Bit','Count']\nsns.barplot(x=\"High Bit\", y=\"Count\", data=tmp)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.jointplot(x=\"HighBit\", y=\"BitsStored\", data=Data[['HighBit','BitsStored']].astype('float'), kind='reg',joint_kws={'color':'green'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"BitsStored = BitsStored +1","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 16. Pixel Representation Attribute ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Data representation of the pixel samples. Each sample shall have the same pixel representation.\n\nPixel Representation Attribute","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp = Data.PixelRepresentation.value_counts().to_frame().reset_index()\ntmp.columns = ['Pixel Representation','Count']\nsns.barplot(x=\"Pixel Representation\", y=\"Count\", data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 17. Window Center & Window Width ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Widow Center attribute Defines a Window Center for display.\n\nSee Section C.8.11.3.1.5 for further explanation.\n\nRequired if Presentation Intent Type (0008,0068) is FOR PRESENTATION and VOI LUT Sequence (0028,3010) is not present. May also be present if VOI LUT Sequence (0028,3010) is present.\n\nWindow Width attribute Defines the Window Width for display. See Section C.8.11.3.1.5 for further explanation.\n\nRequired if Window Center (0028,1050) is present.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* Window Center \n* Window Width ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n
\n
\n
\n
\nC.8.11.3.1.5 VOI Attributes
\n
\n
\n
\n

\nThe Attributes of the VOI LUT Module are specialized in the DX Image Module.

\n

\nWindow Center (0028,1050) and Window Width (0028,1051) specify a linear conversion (unless otherwise specified by the value of VOI LUT Function (0028,1056); See Section C.11.2.1.3) from the output of the (conceptual) Modality LUT values to the input to the (conceptual) Presentation LUT. Window Center contains the value that is the center of the window. Window Width contains the width of the window.

\n

\nThe application of Window Center (0028,1050) and Window Width (0028,1051) shall not produce a signed result.

\n
\n

Note

\n

\nIf the Presentation LUT Shape (2050,0020) is IDENTITY, then the result of applying Window Center (0028,1050) and Window Width (0028,1051) is P-Values.

\n
\n

\nIf multiple values are present, both Attributes shall have the same number of values and shall be considered as pairs. Multiple values indicate that multiple alternative views should be presented.

\n

\nThe VOI LUT Sequence specifies a (potentially non-linear) conversion from the output of the (conceptual) Modality LUT values to the input to the (conceptual) Presentation LUT.

\n

\nIf multiple Items are present in VOI LUT Sequence (0028,3010), only one shall be applied. Multiple Items indicate that multiple alternative views should be presented.

\n

\nIf any VOI LUT Attributes are included by an Image, a Window Width and Window Center or the VOI LUT Table, but not both, shall be applied to the Image for display. Inclusion of both indicates that multiple alternative views should be presented.

\n

\nThe three values of LUT Descriptor (0028,3002) describe the format of LUT Data (0028,3006).

\n

\nThe first value is the number of entries in the lookup table.

\n

\nThe second value is the first stored pixel value mapped. This pixel value is mapped to the first entry in the LUT. All image pixel values less than the first value mapped are also mapped to the first entry in the LUT Data. An image pixel value one greater than the first value mapped is mapped to the second entry in the LUT Data. Subsequent image pixel values are mapped to the subsequent entries in the LUT Data up to an image pixel value equal to number of entries + first value mapped - 1 that is mapped to the last entry in the LUT Data. Image pixel values greater than number of entries + first value mapped are also mapped to the last entry in the LUT Data.

\n

\nThe third value specifies the number of bits for each entry in the LUT Data (analogous to \"bits stored\"). It shall be between 10-16. The LUT Data shall be stored in a format equivalent to 16 \"bits allocated\" and \"high bit\" equal to \"bits stored\" - 1. The third value conveys the range of LUT entry values. These unsigned LUT entry values shall range between 0 and 2n-1, where n is the third value of the LUT Descriptor.

\n
\n

Note

\n
\n
    \n
  1. \n

    \nThe third value is restricted in the VOI LUT Module to 8 or 16 but is specialized here.

    \n
    2. \n
  2. \n

    \nThe first and second values are not specialized and are the same as in the VOI LUT Module.

    \n
    3. \n
\n
\n
\n

\nLUT Data (0028,3006) contains the LUT entry values.

\n
\n\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30,10))\n\nplt.subplot(1,2,1)\n\ntmp = Data.WindowCenter.value_counts().to_frame().reset_index()\ntmp.columns = ['Window Center','Count']\nsns.barplot(x=\"Window Center\", y=\"Count\", data=tmp)\n\nplt.subplot(1,2,2)\n\ntmp = Data.WindowWidth.value_counts().to_frame().reset_index()\ntmp.columns = ['Window Width','Count']\nsns.barplot(x=\"Window Width\", y=\"Count\", data=tmp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.jointplot(x=\"WindowCenter\", y=\"WindowWidth\", data=Data[Data['WindowCenter'] != '[-500, 40]'][['WindowCenter','WindowWidth']].astype('float'), )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = Data.groupby(['WindowCenter','WindowWidth']).count().reset_index()[['WindowCenter','WindowWidth','ImageType']]\ntmp.columns = ['WindowCenter','WindowWidth','Count']\ntmp['Window Center and Width'] = tmp['WindowCenter'] + ' | ' + tmp['WindowWidth']\nsns.barplot(x=\"Window Center and Width\", y=\"Count\", data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 18. Rescale Intercept & Rescale Slope ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Rescale Intercept is the value b in relationship between stored values (SV) and the output units.\n\nRescale Slope is m in the equation specified in Rescale Intercept (0028,1052).\n\nOutput units = m*SV+b","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* Rescale Intercept \n* Rescale Slope ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30,10))\n\nplt.subplot(1,2,1)\n\ntmp = Data.RescaleIntercept.astype('float').value_counts().to_frame().reset_index()\ntmp.columns = ['Rescale Intercept','Count']\nsns.barplot(x=\"Rescale Intercept\", y=\"Count\", data=tmp)\n\nplt.subplot(1,2,2)\n\ntmp = Data.RescaleSlope.astype('float').value_counts().to_frame().reset_index()\ntmp.columns = ['Rescale Slope','Count']\nsns.barplot(x=\"Rescale Slope\", y=\"Count\", data=tmp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the time the value b of the equation is negative which reduces the tail of the output","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 19. Images ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"let's take a look at the images","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"img_array = []\n\nfor filename in glob.glob('../input/osic-pulmonary-fibrosis-progression/train/ID00061637202188184085559/*.dcm'):\n \n img = pydicom.dcmread(filename)\n img_array.append(img.pixel_array)\nimageio.mimsave('movie.gif', img_array)\n\nHTML('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look of all images from random train patient.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,15))\ni=1\nfor filename in glob.glob('../input/osic-pulmonary-fibrosis-progression/train/ID00048637202185016727717/*.dcm'): \n plt.subplot(5,6,i)\n plt.grid(False)\n plt.imshow(pydicom.dcmread(filename).pixel_array, cmap=plt.cm.bone)\n i = i + 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Do the same thing with test dicoms","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30,15))\ni=1\nfor filename in glob.glob('../input/osic-pulmonary-fibrosis-progression/test/ID00421637202311550012437/*.dcm'): \n plt.subplot(6,11,i)\n plt.grid(False)\n plt.imshow(pydicom.dcmread(filename).pixel_array, cmap=plt.cm.bone)\n i = i + 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's apply some effects and see what happen","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,10))\nimg = '../input/osic-pulmonary-fibrosis-progression/train/ID00007637202177411956430/18.dcm'\n\nplt.subplot(1,2,1)\nplt.grid(False)\nplt.imshow(pydicom.dcmread(img).pixel_array, cmap=plt.cm.bone)\nplt.title(\"Original\")\n\nplt.subplot(1,2,2)\nplt.grid(False)\ntest = cv2.bitwise_not(pydicom.dcmread(img).pixel_array)\nplt.title(\"invert the image\")\n\nplt.imshow(test, cmap=plt.cm.bone)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image = pydicom.dcmread(img).pixel_array\n\n\nimageio.imwrite('img.jpg', image)\nimage = imageio.imread('./img.jpg')\n\nplt.figure(figsize=(30, 30))\nplt.subplot(3, 2, 1)\nplt.grid(False)\nplt.title(\"Original\")\nplt.imshow(image, cmap=plt.cm.bone)\n\nret,thresh1 = cv2.threshold(image, 127, 255, cv2.THRESH_BINARY)\n\nplt.subplot(3, 2, 2)\nplt.grid(False)\nplt.title(\"Threshold Binary\")\nplt.imshow(thresh1, cmap=plt.cm.bone)\n\n# image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n# image = np.array(image, dtype=np.uint8)\n# image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\nimage = cv2.GaussianBlur(image, (3, 3), 0)\n# print(image)\n# image = image.reshape(768, 768, 1)\nthresh = cv2.adaptiveThreshold(image, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 3, 5) \n\nplt.subplot(3, 2, 3)\nplt.grid(False)\nplt.title(\"Adaptive Mean Thresholding\")\nplt.imshow(thresh, cmap=plt.cm.bone)\n\n\n_, th2 = cv2.threshold(image, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)\n\nplt.subplot(3, 2, 4)\nplt.grid(False)\nplt.title(\"Otsu's Thresholding\")\nplt.imshow(th2, cmap=plt.cm.bone)\n\n\nplt.subplot(3, 2, 5)\nplt.grid(False)\nblur = cv2.GaussianBlur(image, (5,5), 0)\n_, th3 = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)\nplt.title(\"Guassian Otsu's Thresholding\")\nplt.imshow(th3, cmap=plt.cm.bone)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image = pydicom.dcmread(img).pixel_array\n\nplt.figure(figsize=(20, 20))\nplt.subplot(3, 2, 1)\nplt.grid(False)\nplt.title(\"Original\")\nplt.imshow(image, cmap=plt.cm.bone)\n\n\n# Let's define our kernel size\nkernel = np.ones((5,5), np.uint8)\n\n# Now we erode\nerosion = cv2.erode(image, kernel, iterations = 1)\n\nplt.subplot(3, 2, 2)\nplt.grid(False)\nplt.title(\"Erosion\")\nplt.imshow(erosion, cmap=plt.cm.bone)\n\n# \ndilation = cv2.dilate(image, kernel, iterations = 1)\nplt.subplot(3, 2, 3)\nplt.grid(False)\nplt.title(\"Dilation\")\nplt.imshow(dilation, cmap=plt.cm.bone)\n\n\n# Opening - Good for removing noise\nopening = cv2.morphologyEx(image, cv2.MORPH_OPEN, kernel)\nplt.subplot(3, 2, 4)\nplt.grid(False)\nplt.title(\"Opening\")\nplt.imshow(opening, cmap=plt.cm.bone)\n\n\n# Closing - Good for removing noise\nclosing = cv2.morphologyEx(image, cv2.MORPH_CLOSE, kernel)\nplt.subplot(3, 2, 5)\nplt.grid(False)\nplt.title(\"Closing\")\nplt.imshow(closing, cmap=plt.cm.bone)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# image = pydicom.dcmread(img).pixel_array\nimage = imageio.imread('./img.jpg')\n\nheight, width = image.shape\n\n# Extract Sobel Edges\nsobel_x = cv2.Sobel(image, cv2.CV_64F, 0, 1, ksize=5)\nsobel_y = cv2.Sobel(image, cv2.CV_64F, 1, 0, ksize=5)\n\nplt.figure(figsize=(20, 20))\n\nplt.subplot(3, 2, 1)\nplt.grid(False)\nplt.title(\"Original\")\nplt.imshow(image, cmap=plt.cm.bone)\n\nplt.subplot(3, 2, 2)\nplt.grid(False)\nplt.title(\"Sobel X\")\nplt.imshow(sobel_x, cmap=plt.cm.bone)\n\n\nplt.subplot(3, 2, 3)\nplt.grid(False)\nplt.title(\"Sobel Y\")\nplt.imshow(sobel_y, cmap=plt.cm.bone)\n\nsobel_OR = cv2.bitwise_or(sobel_x, sobel_y)\n\nplt.subplot(3, 2, 4)\nplt.grid(False)\nplt.title(\"sobel_OR\")\nplt.imshow(sobel_OR, cmap=plt.cm.bone)\n\nlaplacian = cv2.Laplacian(image, cv2.CV_64F)\n\nplt.subplot(3, 2, 5)\nplt.grid(False)\nplt.title(\"Laplacian\")\nplt.imshow(laplacian, cmap=plt.cm.bone)\n\n# image = np.array(image*255, dtype=np.uint8)\ncanny = cv2.Canny(image, 50, 120)\n\nplt.subplot(3, 2, 6)\nplt.grid(False)\nplt.title(\"Canny\")\nplt.imshow(canny, cmap=plt.cm.bone)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# image = pydicom.dcmread(img).pixel_array\nimage = imageio.imread('./img.jpg')\n\nplt.figure(figsize=(20, 20))\n\nplt.subplot(2, 2, 1)\nplt.grid(False)\nplt.title(\"Original\")\nplt.imshow(image, cmap=plt.cm.bone)\n\n\n# Grayscale\n# gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)\n\n# Find Canny edges\nedged = cv2.Canny(image, 30, 200)\n\nplt.subplot(2, 2, 2)\nplt.grid(False)\nplt.title(\"Canny Edges\")\nplt.imshow(edged, cmap=plt.cm.bone)\n\n\n# Finding Contours\n# Use a copy of your image e.g. edged.copy(), since findContours alters the image\ncontours, hierarchy = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)\n\nplt.subplot(2, 2, 3)\nplt.grid(False)\nplt.title(\"Canny Edges After Contouring\")\nplt.imshow(edged, cmap=plt.cm.bone)\n\nprint(\"Number of Contours found = \" + str(len(contours)))\n\n# Draw all contours\n# Use '-1' as the 3rd parameter to draw all\ncv2.drawContours(image, contours, -1, (0,255,0), 3)\n\nplt.subplot(2, 2, 4)\nplt.grid(False)\nplt.title(\"Contours\")\nplt.imshow(image, cmap=plt.cm.bone)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train and Test ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/train.csv')\ntest = pd.read_csv('../input/osic-pulmonary-fibrosis-progression/test.csv')\ndata = pd.concat([train, test], ignore_index = True)\n\nProfileReport(data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Smoking Status ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f, axes = plt.subplots(2 ,figsize=(30, 10), sharex=True)\n# plt.subplot(1,2,1);\nsns.lineplot(hue=\"SmokingStatus\", x=\"Weeks\", y = 'FVC', data = data, ax=axes[0])\n# subplot(1,2,2);\nsns.lineplot(hue=\"SmokingStatus\", x=\"Weeks\", y = 'Percent', data = data, ax=axes[1])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Smokers clearly have more FVC and percent, however ex-smokers are slightly more affected by FVC but the first few weeks only until week 70 or so and after that they chauvinate with non-smokers.\n\nThe percentage of ex-smokers and non-smokers is equivalent.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f, axes = plt.subplots(3, figsize=(10, 20), sharex=True)\nsns.violinplot(x=\"SmokingStatus\", y=\"Age\", data=data, ax=axes[0])\n\n\ntmp = data.groupby(['SmokingStatus', 'Sex']).count()['Patient'].reset_index()\ntmp.columns= ['Smoking Status', 'Sex', 'Count']\n\nsns.barplot(x=\"Smoking Status\", y=\"Count\", hue=\"Sex\", data=tmp, ax=axes[1])\n\ntmp = data.groupby('SmokingStatus').count()['Patient'].reset_index()\ntmp.columns= ['Smoking Status', 'Count']\nsns.barplot(x=\"Smoking Status\", y=\"Count\", data= tmp, ax=axes[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Smokers are generally elderly, non-smokers are less so, and ex-smokers are among the least aged.\n* there are more male smokers than female smokers.\n* Ex-smokers are many followed by non-smokers and finally ex-smokers are minority.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2. Sex ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f, axes = plt.subplots(2 ,figsize=(30, 10), sharex=True)\n# plt.subplot(1,2,1);\nsns.lineplot(hue=\"Sex\", x=\"Weeks\", y = 'FVC', data = data, ax=axes[0])\n# subplot(1,2,2);\nsns.lineplot(hue=\"Sex\", x=\"Weeks\", y = 'Percent', data = data, ax=axes[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Women clearly have more FVC than men but the percentage remains broadly equivalent.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, axes = plt.subplots(3, figsize=(10, 20), sharex=True)\nsns.violinplot(x=\"Sex\", y=\"Age\", data=data, ax=axes[0])\n\n\ntmp = data.groupby(['Sex', 'SmokingStatus']).count()['Patient'].reset_index()\ntmp.columns= ['Sex', 'Smoking Status', 'Count']\n\nsns.barplot(x=\"Sex\", y=\"Count\", hue=\"Smoking Status\", data=tmp, ax=axes[1])\n\ntmp = data.groupby('Sex').count()['Patient'].reset_index()\ntmp.columns= ['Sex', 'Count']\nsns.barplot(x=\"Sex\", y=\"Count\", data= tmp, ax=axes[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The majority of women are older than men.\n* There are more smokers and ex-smokers among men than women.\n* There are alsom more men then womens.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3. Age ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tmp = pd.cut(data.Age, 3).to_frame().merge(data,left_index=True, right_index=True)\ntmp.columns = ['Age_Range', 'Patient', 'Weeks', 'FVC', 'Percent','Age', 'Sex', 'SmokingStatus']\n\nf, axes = plt.subplots(2 ,figsize=(30, 10), sharex=True)\n# plt.subplot(1,2,1);\nsns.lineplot(hue=\"Age_Range\", x=\"Weeks\", y = 'FVC', data = tmp, ax=axes[0])\n# subplot(1,2,2);\nsns.lineplot(hue=\"Age_Range\", x=\"Weeks\", y = 'Percent', data = tmp, ax=axes[1])\n\n# sns.distplot(data.Age)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we look closely we notice that people aged between 62 and 75 years are more affected by CVF but the percentage is equivalent for all.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.distplot(data.Age)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The majority are between 65 and 75 years old.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4. FVC & Percentage ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# sns.lineplot( x=\"FVC\", y = 'Percent', data = tmp)\n\nf, axes = plt.subplots(2 ,figsize=(30, 10), sharex=True)\n# plt.subplot(1,2,1);\nsns.lineplot( x=\"Weeks\", y = 'FVC', data = tmp, ax=axes[0])\n# subplot(1,2,2);\nsns.lineplot( x=\"Weeks\", y = 'Percent', data = tmp, ax=axes[1])\n\n# sns.distplot(data.Age)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hum, High correlation between FVC and Percent.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.jointplot(x=\"FVC\", y=\"Percent\", data=data, kind='reg',\n joint_kws={'line_kws':{'color':'green'}})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Indeed a positive correlation exists.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 5. Heatmap ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"corr = data[['Weeks','FVC','Percent','Age','Sex','SmokingStatus']].corr()\n\nmask = np.triu(np.ones_like(corr, dtype=np.bool))\n\nf, ax = plt.subplots(figsize=(11, 9))\n\ncmap = sns.diverging_palette(220, 10, as_cmap=True)\n\nsns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,\n square=True, linewidths=.5, cbar_kws={\"shrink\": .5}, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.pairplot(data[['Weeks','FVC','Percent','Age','Sex','SmokingStatus']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"It was my analysis on the csv data as well as the dicom files, it allowed me to have an intuition on the data to start the modeling.\n\nIf this is the case for you please drop an upvote it will help me a lot.\n\nThe next step is to train an LGBM and see if the performance is there before starting the CNN.\n\nSee you soon =)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}