{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"'''Import basic modules.'''\nimport pandas as pd\nimport numpy as np\nimport os\nimport pydicom as dcm\nimport cv2\nimport time\n\n'''Customize visualization\nSeaborn and matplotlib visualization.'''\nimport altair as alt\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_style(\"whitegrid\")\n%matplotlib inline\nimport folium \nfrom IPython.core.display import HTML\nimport urllib.request\nfrom PIL import Image\nimport imageio\n\n\n\n'''Plotly visualization .'''\nimport plotly.express as px\nimport plotly.offline as py\nfrom plotly.offline import iplot, init_notebook_mode\nimport plotly.graph_objs as go\npy.init_notebook_mode(connected = True) # Required to use plotly offline in jupyter notebook\n\n\n\n'''Display markdown formatted output like bold, italic bold etc.'''\nfrom IPython.display import Markdown\ndef bold(string):\n display(Markdown(string))\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

SEER & SIIM-ISIC Melanoma: Facts, Statistics, & EDA

","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n\n[Image source](https://www.healthline.com/health/skin-cancer/facts-and-stats#1)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# About Melanoma\nMelanoma, also known as malignant melanoma, is a type of skin cancer that develops from the pigment-producing cells known as melanocytes.Melanomas typically occur in the skin but may rarely occur in the mouth, intestines or eye (uveal melanoma). In women, they most commonly occur on the legs, while in men they most commonly occur on the back.About 25% of melanomas develop from moles. Changes in a mole that can indicate melanoma include an increase in size, irregular edges, change in color, itchiness or skin breakdown.\n\n### Cause\n**Melanomas are usually caused by DNA damage** resulting from exposure to ultraviolet light from the sun. **Genetics also plays a role.** Melanoma can also occur in skin areas with little sun exposure (i.e. mouth, soles of feet, palms of hands, genital areas). People with dysplastic nevus syndrome, also known as familial atypical multiple mole melanoma (FAMMM), are at increased risk for the development of melanoma.\n\nHaving more than fifty moles indicates an increased risk melanoma might arise. A weakened immune system makes it easier for cancer to arise due to the body's weakened ability to fight cancer cells.\n\n### Where melanoma is most likely to develop\n\n\n### Signs and symptoms\nEarly signs of melanoma are summarized by the mnemonic \"ABCDEF\":\n\n* Asymmetry\n* Borders (irregular with edges and corners)\n* Colour (variegated)\n* Diameter (greater than 6 mm (0.24 in), about the size of a pencil eraser)\n* Evolving over time\n* Funny looking\n\nThis classification does not apply to nodular melanoma, which has its own classifications:\n\n* Elevated above the skin surface\n* Firm to the touch\n* Growing\n\n### Prevention\n* Minimizing exposure to sources of ultraviolet radiation (the sun and sunbeds), following sun protection measures and wearing sun protective clothing (long-sleeved shirts, long trousers, and broad-brimmed hats) can offer protection.\n\n* Sunscreen appears to be effective in preventing melanoma.\n\n* [Statin](https://en.wikipedia.org/wiki/Statin) and [Fibrate](https://en.wikipedia.org/wiki/Fibrate) medication may decrease the risk of melanoma.\n\n**Reference:** \n* http://www.cancer.gov/types/skin/hp/melanoma-treatment-pdq\n* https://www.iarc.fr/en/publications/pdfs-online/wcr/2003/WorldCancerReport.pdf","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"

U.S. Melanoma Skin Cancer: Facts & Statistics

","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"incidence_rates = pd.read_csv('../input/melanoma-skin-cancer-dataset/annual_incidence_rates.csv')\ndeath_rates = pd.read_csv('../input/melanoma-skin-cancer-dataset/annual_death_rates.csv')\nage_incidence_rates = pd.read_csv('../input/melanoma-skin-cancer-dataset/age_specific_incidence_rate.csv')\nage_death_rates = pd.read_csv('../input/melanoma-skin-cancer-dataset/age_specific_death_rate.csv')\nstate_death_rates = pd.read_csv('../input/melanoma-skin-cancer-dataset/state_death_rates.csv')\ncode = ['AL','AK','AZ','AR','CA','CO','CT','DE','DC','FL','GA','HI','ID','IL','IN','IA','KS','KY','LA','ME','MD','MA','MI','MN','MS','MO','MT','NE','NV','NH','NJ','NM','NY','NC','ND','OH','OK','OR','PA','RI','SC','SD','TN','TX','UT','VT','VA','WA','WV','WI','WY']\nstate_death_rates['code'] = code","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The incidence of both non-melanoma and melanoma skin cancers has been increasing over the past decades. Currently, between 2 and 3 million non-melanoma skin cancers and 132,000 melanoma skin cancers occur globally each year. One in every three cancers diagnosed is a skin cancer and, according to Skin Cancer Foundation Statistics, one in every five Americans will develop skin cancer in their lifetime. [Source](https://www.who.int/)\n\n**Dataset of U.S. Melanoma Skin Cancer Facts & Statistics Collected from:\n* https://seer.cancer.gov/ \n* https://gis.cdc.gov/Cancer/USCS/DataViz.html\n* https://seer.cancer.gov/csr/1975_2017/results_figure/sect_01_intro2_25pgs.pdf","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trace1 = go.Scatter(\n x=incidence_rates['Year of Diagnosis'],\n y=incidence_rates['All Races,Males'],\n name=\"All Races,Males\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='dodgerblue')\n\ntrace2 = go.Scatter(\n x=incidence_rates['Year of Diagnosis'],\n y=incidence_rates['All Races,Females'],\n name=\"All Races,Females\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='fuchsia')\n\nlayout = go.Layout(template = 'plotly_white', width=700, height=500, title_text = 'Incidencea Rates by Year, All Race and Sex ',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace1,trace2], layout = layout)\nfig.add_shape(\n # Line Vertical\n dict(\n type=\"line\",\n x0='2015',\n y0=6.5,\n x1='2015',\n y1=35,\n line=dict(\n color=\"black\",\n width=1,\n dash=\"dashdot\"\n )))\nfig.add_annotation( # add a text callout with arrow\n text=\" Slightly Decreasing\", x='2015', y=23, arrowhead=1, showarrow=True\n)\nfig.show()\n\ntrace1 = go.Scatter(\n x=incidence_rates['Year of Diagnosis'],\n y=incidence_rates['Whites,Both Sexes'],\n name=\"Whites,Both Sexes\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='lightsalmon')\n\ntrace2 = go.Scatter(\n x=incidence_rates['Year of Diagnosis'],\n y=incidence_rates['Blacks,Both Sexes'],\n name=\"Blacks,Both Sexes\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='maroon')\n\nlayout = go.Layout(template = 'plotly_white', width=700, height=500, title_text = 'Incidencea Rates by Year, Between Race',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace1,trace2], layout = layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"πŸ“Œ**Inference**: Men are two times more likely to develop melanoma than women as shown in above chart. We can see that continuous rise in the incidence rate until 2015, we can obverse that slightly decrease in rates. White american skin are 20 times more likely to be diagnosed with cancer that african American.\n\nπŸ“**Footnotes:** SEER 9 areas (San Francisco, Connecticut, Detroit, Hawaii, Iowa, New Mexico, Seattle, Utah, and Atlanta). Rates are per 100,000 and are age-adjusted to the 2000 US Std Population (19 age groups - Census P25-1130).","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trace1 = go.Scatter(\n x=death_rates['Year of Death'],\n y=death_rates['All Races,Males'],\n name=\"All Races,Males\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='dodgerblue')\n\ntrace2 = go.Scatter(\n x=death_rates['Year of Death'],\n y=death_rates['All Races,Females'],\n name=\"All Races,Females\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='fuchsia')\n\nlayout = go.Layout(template = 'plotly_white', width=700, height=500, title_text = 'Death Rates by Year, All Race and Sex ',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace1,trace2], layout = layout)\nfig.add_shape(\n # Line Vertical\n dict(\n type=\"line\",\n x0='2009',\n y0=1.3,\n x1='2009',\n y1=4.5,\n line=dict(\n color=\"black\",\n width=1,\n dash=\"dashdot\"\n )))\nfig.add_annotation( # add a text callout with arrow\n text=\"Start Decreasing\", x='2009', y=2.5, arrowhead=1, showarrow=True\n)\nfig.show()\n\ntrace1 = go.Scatter(\n x=death_rates['Year of Death'],\n y=death_rates['Whites,Both Sexes'],\n name=\"Whites,Both Sexes\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='lightsalmon')\n\ntrace2 = go.Scatter(\n x=death_rates['Year of Death'],\n y=death_rates['Blacks,Both Sexes'],\n name=\"Blacks,Both Sexes\",\n marker=dict(size=3.5),\n mode='lines+markers',\n line_color='maroon')\n\nlayout = go.Layout(template = 'plotly_white', width=700, height=500, title_text = 'Death Rates by Year, Between Race',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace1,trace2], layout = layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"πŸ“Œ**Inference**: Death rate is also higher in males, charts shows increase trend but in females death rate is flate trend over the year. After 2019, we can see that death rate in both sex is significatly decreasing.\n\nπŸ“**Footnotes:** US Mortality Files, National Center for Health Statistics, Centers for Disease Control and Prevention. Rates are per 100,000 and are age-adjusted to the 2000 US Std Population (19 age groups - Census P25-1130).","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\ntrace1 = go.Pie(\n labels=age_incidence_rates[' Age at Diagnosis'],\n values=age_incidence_rates['All Races, Both Sexes'],\n hoverinfo='label+percent', \n textfont_size=12,\n marker=dict(colors=px.colors.sequential.RdBu, \n line=dict(color='#000000', width=2)))\nlayout = go.Layout(width=600, height=500,title_text = 'Age-Specific Incidence Rates By All Race and Sex , 2013-2017',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace1], layout = layout)\nfig.show()\n\ntrace2 = go.Bar(\n x=age_incidence_rates[' Age at Diagnosis'], \n y=age_incidence_rates['All Races,Males'],\n text=age_incidence_rates['All Races,Males'],\n name = 'All Races,Males',\n textposition='auto',\n marker_color='dodgerblue')\ntrace3 = go.Bar(\n x=age_incidence_rates[' Age at Diagnosis'], \n y=age_incidence_rates['All Races,Females'],\n text=age_incidence_rates['All Races,Females'],\n name = 'All Races,Females',\n textposition='auto',\n marker_color='fuchsia')\nlayout = go.Layout(barmode='group', template = 'plotly_white',width=700, height=500, \n title_text = 'Age-Specific Incidence Rates Between Sex, 2013-2017',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace2, trace3], layout = layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"πŸ“Œ**Inference:** The age of 50, men are more likely to develop melanoma than women. Over an entire lifetime, 1 in 34 men will develop melanoma. Only 1 in 53 women will be.\n\nMoreover, men are two times more likely to develop melanoma than women after the age of 60. After age 80, men are three times more likely to develop melanoma. Before 60, though, the statistic is reversed. Women are nearly twice as likely to develop melanoma before their 60th birthday.\n\nπŸ“**Footnotes:** Rates are per 100,000 and are age-adjusted to the 2000 US Std Population (19 age groups - Census P25-1130)\n","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"trace1 = go.Pie(\n labels=age_death_rates[' Age at Death'],\n values=age_death_rates['All Races, Both Sexes'],\n hoverinfo='label+percent', \n textfont_size=12,\n marker=dict(colors=px.colors.sequential.RdBu, \n line=dict(color='#000000', width=2)))\nlayout = go.Layout(width=600, height=500,title_text = 'Age-Specific Death Rates By All Race and Sex , 2013-2017',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace1], layout = layout)\nfig.show()\n\ntrace2 = go.Bar(\n x=age_death_rates[' Age at Death'], \n y=age_death_rates['All Races,Males'],\n text=age_death_rates['All Races,Males'],\n name = 'All Races,Males',\n textposition='auto',\n marker_color='dodgerblue')\ntrace3 = go.Bar(\n x=age_death_rates[' Age at Death'], \n y=age_death_rates['All Races,Females'],\n text=age_death_rates['All Races,Females'],\n name = 'All Races,Females',\n textposition='auto',\n marker_color='fuchsia')\nlayout = go.Layout(barmode='group', template = 'plotly_white',width=650, height=500, \n title_text = 'Age-Specific Death Rates Between Sex, 2013-2017',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\nfig = go.Figure(data = [trace2, trace3], layout = layout)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"πŸ“Œ**Inference:** Deaths are singificantly increase in the age above 50. High death rates recorded in the age of above 70 in the both males and females.\n\nπŸ“**Footnotes:** Rates are per 100,000 and are age-adjusted to the 2000 US Std Population (19 age groups - Census P25-1130)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(data=go.Choropleth(\n locations=state_death_rates['code'], \n z = state_death_rates['Both Sex '], \n locationmode = 'USA-states',\n colorscale = 'brbg',\n colorbar_title = \"Death Rate\",\n text=state_death_rates['State'],\n))\n\nfig.update_layout(width=650, height=600, \n title_text = 'Individual State Death Rates',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'),\n geo = dict(\n scope='usa',\n projection=go.layout.geo.Projection(type = 'albers usa'), \n))\n\nfig.show()\n\nfig = px.bar(state_death_rates.sort_values(by='Both Sex ', ascending=False).head(10).sort_values('Both Sex ', ascending=True), \n x='Both Sex ', y=\"State\", \n title='Top 10 States by Death Rates',\n text='Both Sex ', \n orientation='h', \n width=700, height=500)\nfig.update_traces(marker_color='khaki', opacity=0.8, textposition='inside',)\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"Female = np.array([-1.50,-1.21,-1.67,-1.35,-1.26,-1.97,-1.31,-1.69,-0.00,-1.48,-1.39,-0.77,-2.15,-1.45,\n -1.72,-1.88,-1.82,-1.87,-1.07,-1.93,-1.22,-1.73,-1.41,-1.60,-1.24,-1.59,-1.50,-1.52,-1.35,\n -1.84,-1.44,-1.44,-1.17,-1.54,-1.26,-1.71,-1.61,-1.73,-1.72,-1.62,-1.49,-1.51,-1.71,-1.19,\n -1.64,-1.67,-1.51,-1.74,-2.00,-1.46,-1.69])\n\ndata = [go.Bar(y=state_death_rates['State'],\n x=state_death_rates['Male'],\n orientation='h',\n name='Male',\n hoverinfo='x',\n marker=dict(color='dodgerblue')\n ),\n go.Bar(y=state_death_rates['State'],\n x=Female,\n orientation='h',\n name='Female',\n text=-1 * Female,\n hoverinfo='text',\n marker=dict(color='fuchsia')\n )]\n\nlayout = go.Layout(width=650, height=700,\n template = 'plotly_white',\n yaxis=go.layout.YAxis(title='State'),\n xaxis=go.layout.XAxis(\n range=[-6, 6],\n tickvals=[-5.5,-4.5,-3.5, -2.5, -1.5, 0, 1.5, 2.5, 3.5, 4.5, 5.5],\n ticktext=[5.5,4.5,3.5, 2.5, 1.5, 0, 1.5, 2.5, 3.5, 4.5, 5.5],\n title='Death Rate'),\n barmode='overlay',\n bargap=0.1,\n title_text = 'Pyramid Chart of States Death Rates Between Sex',\n font=dict(family=\"Arial, Balto, Courier New, Droid Sans\",color='black'))\n\npy.iplot(dict(data=data, layout=layout))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"πŸ“Œ**Inference:** You may think places with sunnier, hotter weather have more cases of skin cancer. This isn’t necessarily the case. In fact, the Centers for Disease Control and Prevention notes California and Florida had fewer cases per 100,000 people than states with cooler climates, like Wyoming, Montana, and Idaho. \n
The states with the fewest cases of skin cancer are:
\n* D.C.\t\n* Hawaii\n* Louisiana\n* Florida\n* New York\t\n* Texas\n\nπŸ“**Footnotes:** US Mortality Files, National Center for Health Statistics, Centers for Disease Control and Prevention. Rates are per 100,000 and are age-adjusted to the 2000 US Std Population (19 age groups - Census P25-1130).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"

","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"

Competition: SIIM-ISIC Melanoma Classification -EDA

\n\n### About the competition\nIn this competition, you’ll identify melanoma in images of skin lesions. In particular, you’ll use images within the same patient and determine which are likely to represent a melanoma. Using patient-level contextual information may help the development of image analysis tools, which could better support clinical dermatologists.\n\n### What am I predicting?\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant.\n\n### Dataset\nThe dataset consists of images in :\n\n* DIOCOM format\n* JPEG format in JPEG directory\n* TFRecord format in tfrecords directory\n* Additionally, there is a metadata comprising of train, test and submission file in CSV format.\n\n### Columns\n* image_name - unique identifier, points to filename of related DICOM image\n* patient_id - unique patient identifier\n* sex - the sex of the patient (when unknown, will be blank)\n* age_approx - approximate patient age at time of imaging\n* anatom_site_general_challenge - location of imaged site\n* diagnosis - detailed diagnosis information (train only)\n* benign_malignant - indicator of malignancy of imaged lesion\n* target - binarized version of the target variable\n\n### Evaluation Metric\nSubmissions are evaluated on [area under the ROC curve](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc) between the predicted probability and the observed target.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1. Reading The Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_PATH = \"../input/siim-isic-melanoma-classification/\"\n\ntrain_df = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest_df = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\n\n\n#Training data\nprint('Training data shape: ', train_df.shape)\nprint(' ')\nprint('Test data shape: ', test_df.shape)\n\nbold('**TRAINING DATA**')\ndisplay(train_df.head(5))\nbold('**TEST DATA**')\ndisplay(test_df.head(5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Data Exploration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**MISSING DATA AND DATA TYPES**')\nprint('===== Train Set =====')\nprint(train_df.info())\nprint('\\n')\nprint('===== Train Set =====')\nprint(train_df.info())\n\nbold('**TOTAL NUMBER OF IMAGES**')\nprint(\"Total images in Train set: \",train_df['image_name'].count())\nprint(\"Total images in Test set: \",test_df['image_name'].count())\n\nbold('**UNIQUE IDs**')\nprint('Total nuber of patient ids in train set ', train_df['patient_id'].count())\nprint('Total nuber of patient ids test set ', test_df['patient_id'].count())\nprint('Unique ids in train set ',train_df['patient_id'].nunique())\nprint('Unique ids in test set ', test_df['patient_id'].nunique())\n\nids_train = train_df.patient_id.values\nids_test = test_df.patient_id.values\npatient_overlap = list(set(ids_train).intersection(set(ids_test)))\nprint('Patient IDs in both the training and test sets ', len(patient_overlap))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*There are some missing values in both the train and text set.*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3. EDA\n\n### 3.1 Target (binarized version)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"target_count = train_df['target'].value_counts().reset_index()\ntarget_count['percent']=np.round(train_df['target'].value_counts(normalize=True), 2)\ntarget_count.rename(columns={'index': 'target', 'target':'count'}, inplace=True)\n\nfig = px.bar(target_count, \n x='target', y=\"count\", \n title='Distribution of the Target (binarized version)',\n text='percent', \n orientation='v', \n width=500, height=600)\nfig.update_traces(opacity=0.8, marker=dict(color='yellowgreen',\n line=dict(width=2, color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Target variable is highly imbalanced*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.2 Gender distribution","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sex_count = train_df['sex'].value_counts().reset_index()\nsex_count['percent']=np.round(train_df['sex'].value_counts(normalize=True).reset_index()['sex'], 2)\nsex_count.rename(columns={'index': 'sex', 'sex':'count'}, inplace=True)\n\nfig = px.bar(sex_count, \n x='sex', y=\"count\", \n title='Distribution of the gender',\n text='percent', \n orientation='v', \n width=500, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(color='lightsalmon',\n line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.3 Gender by target","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp = train_df.groupby(['target','sex'])['target'].count().to_frame('count').reset_index()\nfig = px.bar(temp, \n x=\"target\", y=\"count\", \n color='sex', \n barmode='group',\n title='Distribution of the gender by target',\n text='count', \n orientation='v', \n width=500, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*It seem Gender variable is equally distributed between target.*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.4 Distribution of patients age","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.histogram(train_df, \n x=\"age_approx\", \n nbins=30,\n barmode='group',\n title='Distribution of the gender by target',\n marginal=\"box\",\n width=600, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(color='palegoldenrod',\n line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Age variable is normally distributed* ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.5 Distribution of patients age by gender","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp = train_df[['age_approx', 'sex']].dropna()\nfig = px.histogram(temp, \n x=\"age_approx\",\n color = 'sex',\n nbins=30,\n barmode='group',\n title='Distribution of patients age by gender',\n marginal=\"box\",\n width=600, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(\n line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.6 Distribution of benign malignant (target variable non binarized version)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"benign_count = train_df['benign_malignant'].value_counts().reset_index()\nbenign_count['percent']=np.round(train_df['benign_malignant'].value_counts(normalize=True).reset_index()['benign_malignant'], 2)\nbenign_count.rename(columns={'index': 'benign_malignant', 'benign_malignant':'count'}, inplace=True)\n\nfig = px.bar(benign_count, \n x='benign_malignant', y=\"count\", \n title='Distribution of the benign malignant',\n text='percent', \n orientation='v', \n width=500, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(color='moccasin',\n line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*It seem dataset have more of beign tumor than malignant tumor. There are two main classifications of tumors. One is known as benign and the other as malignant. A benign tumor is a tumor that does not invade its surrounding tissue or spread around the body. A malignant tumor is a tumor that may invade its surrounding tissue or spread around the body.[source](https://study.com/academy/lesson/benign-vs-malignant-definition-characteristics-differences.html).*\n![](https://charlottedermatologypa.com/new/wp-content/uploads/2016/08/melanoma.jpeg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.7 Distribution of patients age by benign malignant","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp = train_df[['age_approx', 'benign_malignant']].dropna()\nfig = px.histogram(temp, \n x=\"age_approx\",\n color = 'benign_malignant',\n nbins=30,\n barmode='group',\n title='Distribution of patients age by benign malignant',\n marginal=\"box\",\n width=600, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(\n line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.8 Location of imaged site","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"anatom_count = train_df['anatom_site_general_challenge'].value_counts().reset_index()\nanatom_count['percent']=np.round(train_df['anatom_site_general_challenge'].value_counts(normalize=True).reset_index()['anatom_site_general_challenge'], 2)\nanatom_count.rename(columns={'index': 'anatom_site_general_challenge', 'anatom_site_general_challenge':'count'}, inplace=True)\n\nfig = px.bar(anatom_count, \n x='anatom_site_general_challenge', y=\"count\", \n title='Distribution of the Location of imaged site',\n text='percent', \n orientation='v', \n width=600, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(color='lightcoral',\n line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*It seem torso (the trunk of the human body) is most affected area by melanoma, after the extremities of the body (upper/lower) and Very less cases in palms/ soles and oral/genital areas*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.9 Location of imaged site by gender","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp = train_df.groupby(['anatom_site_general_challenge','sex'])['anatom_site_general_challenge'].count().to_frame('count').reset_index()\nfig = px.bar(temp, \n x=\"anatom_site_general_challenge\", y=\"count\", \n color='sex', \n barmode='group',\n title='Distribution of the Location of imaged site by gender',\n text='count', \n orientation='v', \n width=700, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*In men, melanoma is most commonly found on the back and other places on the trunk (from the shoulders to the hips) or the head and neck. The most common sites in women are the arms and the legs.*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.10 Distribution of Diagnosis","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"diagnosis_count = train_df['diagnosis'].value_counts().reset_index()\ndiagnosis_count['percent']=np.round(train_df['diagnosis'].value_counts(normalize=True).reset_index()['diagnosis'], 2)\ndiagnosis_count.rename(columns={'index': 'diagnosis', 'diagnosis':'count'}, inplace=True)\n\nfig = px.bar(diagnosis_count, \n x='diagnosis', y=\"count\", \n title='Distribution of the diagnosis',\n text='count', \n orientation='v', \n width=600, height=600,\n )\nfig.update_traces(opacity=0.8, marker=dict(color='rebeccapurple',\n line=dict(width=2,\n color='DarkSlateGrey')))\n\nfig.update_layout(template = 'plotly_white')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*These are the different form of molanoma skin cancer. Most of the unknown but nevus is a type of mole that looks different from a common mole, and it is most likely to convert into melonama shin cancer. Lentigo has a lower rate of transformation to invasive melanoma than the other form of melanoma*\n* Unknown refers to metastatic melanoma in lymph nodes, subcutaneous tissue, or visceral sites in the absence of a detectable primary tumour despite detailed examination\n\n[Source](https://www.cancer.gov/types/skin/moles-fact-sheet)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4. Visualising Images\nLooking at the images we can observe the moles that are irregular in color or shape are typically treated as candidates. To detect melanomas (and increase survival rates), it is recommended to learn to recognize them (see \"ABCDE\" mnemonic), to regularly examine moles for changes (shape, size, color, itching or bleeding) and to consult a qualified physician when a candidate appears.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 4.1 Exploration of Images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**Let’s load an image and observe its various properties in general**')\n\nimages = train_df['image_name'].values\nimg_dir = IMAGE_PATH+'/jpeg/train'\n\nimg = imageio.imread(os.path.join(img_dir, (images+'.jpg')[1]))\nplt.figure(figsize = (5,5))\nplt.imshow(img)\nplt.show()\n\nbold('**Observe Basic Properties of Image**')\nprint('Type of the image : ' , type(img)) \nprint('Shape of the image : {}'.format(img.shape)) \nprint('Image Hight {}'.format(img.shape[0])) \nprint('Image Width {}'.format(img.shape[1])) \nprint('Dimension of Image {}'.format(img.ndim))\n\nbold('**calculate the size of an RGB image**')\nprint('Image size {}'.format(img.size)) \nprint('Maximum RGB value in this image {}'.format(img.max())) \nprint('Minimum RGB value in this image {}'.format(img.min()))\nprint('Value of only R channel {}'.format(img[ 100, 50, 0])) \nprint('Value of only G channel {}'.format(img[ 100, 50, 1])) \nprint('Value of only B channel {}'.format(img[ 100, 50, 2]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"bold('**view of random images in single channel**')\nrandom_images = [np.random.choice(images+'.jpg') for i in range(9)]\nplt.figure(figsize=(10,8))\n\n# Iterate and plot random images\nfor i in range(9):\n plt.subplot(3, 3, i + 1)\n img = imageio.imread(os.path.join(img_dir, random_images[i]))\n plt.imshow(img[ : , : , 0], cmap='gray')\n \n# Adjust subplot parameters to give specified padding\nplt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Images corresponding to target","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/gpreda/siim-isic-melanoma-classification-eda\ndef show_dicom_images(data):\n img_data = list(data.T.to_dict().values())\n f, ax = plt.subplots(3,3, figsize=(16,18))\n for i,data_row in enumerate(img_data):\n patientImage = data_row['image_name']+'.dcm'\n imagePath = os.path.join(IMAGE_PATH,\"train/\",patientImage)\n data_row_img_data = dcm.read_file(imagePath)\n modality = data_row_img_data.Modality\n age = data_row_img_data.PatientAge\n sex = data_row_img_data.PatientSex\n data_row_img = dcm.dcmread(imagePath)\n ax[i//3, i%3].imshow(data_row_img.pixel_array, cmap=plt.cm.gray) \n ax[i//3, i%3].axis('off')\n ax[i//3, i%3].set_title(f\"ID: {data_row['image_name']}\\nModality: {modality} Age: {age} Sex: {sex}\\nDiagnosis: {data_row['diagnosis']}\")\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**Images with Malignant lesions**')\nshow_dicom_images(train_df[train_df['target']==1].sample(9))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**Images with benign lesions**')\nshow_dicom_images(train_df[train_df['target']==0].sample(9))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.3 Histograms\nHistograms are a graphical representation showing how frequently various color values occur in the image i.e frequency of pixels intensity values. In a RGB color space, pixel values range from 0 to 255 where 0 stands for black and 255 stands for white. Analysis of a histogram can help us understand thee brightness, contrast and intensity distribution of an image. Now let's look at the histogram of a random selected sample from each category.\n\n![](https://opencv-python-tutroals.readthedocs.io/en/latest/_images/histogram_sample.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Histogram of Grayscale Image","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"benign = train_df[train_df['benign_malignant']=='benign']\nmalignant = train_df[train_df['benign_malignant']=='malignant']\n\n\nf = plt.figure(figsize=(16,8))\nf.add_subplot(2,2, 1)\n\nbenign_img = benign['image_name'][1]+'.jpg'\nbenign_img = plt.imread(os.path.join(img_dir, benign_img))\nplt.imshow(benign_img[ : , : , 0], cmap='gray')\nplt.colorbar()\nplt.title('Benign Image', fontsize=15)\n\nf.add_subplot(2,2, 2)\n_ = plt.hist(benign_img.ravel(),256,[0,256])\n\nf.add_subplot(2,2, 3)\nmalignant_img = malignant['image_name'][235]+'.jpg'\nmalignant_img = plt.imread(os.path.join(img_dir, malignant_img))\nplt.imshow(malignant_img[ : , : , 0], cmap='gray')\nplt.colorbar()\nplt.title('Malignant Image', fontsize=15)\n\nf.add_subplot(2,2, 4)\n_ = plt.hist(malignant_img.ravel(),256,[0,256])\n\nplt.tight_layout() \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Images are stored as pixels values, each pixel value represents a color intensity value. Histograms are frequency distribution of these intensity values that occur in an image.* ***Here we can differentiate the tine gray spot distribution of benign image between 150 to 200 gray scale and spread of gray scale distribution of malignant image between 100 to 250 gray scale.*** *We can more classify this by histogram of masked image.*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Histogram of Mask image\nWhat if you want to find histograms of some regions of an image? Just create a mask image with white color on the region you want to find histogram and black otherwise. ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(10,5))\nf.add_subplot(2,2, 1)\n\nbenign_img = benign['image_name'][2]+'.jpg'\nimg = plt.imread(os.path.join(img_dir, benign_img))\n\n# create a mask\nmask = np.zeros(img.shape[:2], np.uint8)\nmask[200:800, 500:1400] = 255\nmasked_img = cv2.bitwise_and(img,img,mask = mask)\n\n# Calculate histogram with mask and without mask\n# Check third argument for mask\nhist_full = cv2.calcHist([img],[0],None,[256],[0,256])\nhist_mask = cv2.calcHist([img],[0],mask,[256],[0,256])\n\nraw_image = plt.imread(os.path.join(img_dir, benign_img))\nplt.imshow(raw_image[ : , : , 0], cmap='gray')\nplt.colorbar()\nplt.title('Benign Image', fontsize=15)\n\nf.add_subplot(2,2, 2)\nplt.imshow(mask, cmap='gray')\nplt.colorbar()\nplt.title('Mask', fontsize=15)\n\nf.add_subplot(2,2, 3)\nplt.imshow(masked_img[ : , : , 0], cmap='gray')\nplt.colorbar()\nplt.title('Mask Image', fontsize=15)\n\nf.add_subplot(2,2, 4)\nplt.plot(hist_full)\nplt.plot(hist_mask)\nplt.title('Histogram', fontsize=15)\n\nplt.tight_layout() \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(10,5))\nf.add_subplot(2,2, 1)\n\nmalignant_img = malignant['image_name'][235]+'.jpg'\nimg = plt.imread(os.path.join(img_dir, malignant_img))\n\n# create a mask\nmask = np.zeros(img.shape[:2], np.uint8)\nmask[500:2000, 500:2500] = 255\nmasked_img = cv2.bitwise_and(img,img,mask = mask)\n\n# Calculate histogram with mask and without mask\n# Check third argument for mask\nhist_full = cv2.calcHist([img],[0],None,[256],[0,256])\nhist_mask = cv2.calcHist([img],[0],mask,[256],[0,256])\n\nraw_image = plt.imread(os.path.join(img_dir, malignant_img))\nplt.imshow(raw_image[ : , : , 0], cmap='gray')\nplt.colorbar()\nplt.title('Malignant Image', fontsize=15)\n\nf.add_subplot(2,2, 2)\nplt.imshow(mask, cmap='gray')\nplt.colorbar()\nplt.title('Mask', fontsize=15)\n\nf.add_subplot(2,2, 3)\nplt.imshow(masked_img[ : , : , 0], cmap='gray')\nplt.colorbar()\nplt.title('Mask Image', fontsize=15)\n\nf.add_subplot(2,2, 4)\nplt.plot(hist_full)\nplt.plot(hist_mask)\nplt.title('Histogram', fontsize=15)\n\nplt.tight_layout() \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***In the histogram plot, blue line shows histogram of full image while orange line shows histogram of masked region.*** *Here we can clearly distribution of benign image between 180 to 220 gray scale and spread of gray scale distribution of malignant image between 150 to 250 gray scale.*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Histogram of Color Image","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/parulpandey/melanoma-classification-eda-starter\nf = plt.figure(figsize=(16,8))\nf.add_subplot(2,2, 1)\n\nbenign_img = benign['image_name'][1]+'.jpg'\nbenign_img = plt.imread(os.path.join(img_dir, benign_img))\nplt.imshow(benign_img, cmap='gray')\nplt.colorbar()\nplt.title('Benign Image', fontsize=15)\n\nf.add_subplot(2,2, 2)\n_ = plt.hist(benign_img.ravel(),bins = 256, color = 'orange', alpha=0.3)\n_ = plt.hist(benign_img[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n_ = plt.hist(benign_img[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n_ = plt.hist(benign_img[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Value')\n_ = plt.ylabel('Count')\n_ = plt.legend(['Total', 'Red_Channel', 'Green_Channel', 'Blue_Channel'])\n\nf.add_subplot(2,2, 3)\nmalignant_img = malignant['image_name'][235]+'.jpg'\nmalignant_img = plt.imread(os.path.join(img_dir, malignant_img))\nplt.imshow(malignant_img, cmap='gray')\nplt.colorbar()\nplt.title('Malignant Image', fontsize=15)\n\nf.add_subplot(2,2, 4)\n_ = plt.hist(malignant_img.ravel(),bins = 256, color = 'orange', alpha = 0.3)\n_ = plt.hist(malignant_img[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n_ = plt.hist(malignant_img[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n_ = plt.hist(malignant_img[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n_ = plt.xlabel('Intensity Value')\n_ = plt.ylabel('Count')\n_ = plt.legend(['Total', 'Red_Channel', 'Green_Channel', 'Blue_Channel'])\n\nplt.tight_layout() \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Explore DICOM files\nA DICOM file is an image saved in the Digital Imaging and Communications in Medicine (DICOM) format. It contains an image from a medical scan, such as an ultrasound or MRI. DICOM files may also include identification data for patients so that the image is linked to a specific individual. [source](https://fileinfo.com/extension/dicom#:~:text=A%20DICOM%20file%20is%20an,linked%20to%20a%20specific%20individual.)\n\n
[1] https://pydicom.github.io/pydicom/stable/auto_examples/input_output/plot_read_dicom.html
\n
[2] https://www.kaggle.com/gpreda/siim-isic-melanoma-classification-eda
\n
[3] https://www.kaggle.com/schlerp/getting-to-know-dicom-and-the-data
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_dcm_info(dataset):\n print(\"Filename.........:\", file_path)\n print(\"Storage type.....:\", dataset.SOPClassUID)\n print()\n\n pat_name = dataset.PatientName\n display_name = pat_name.family_name + \", \" + pat_name.given_name\n print(\"Patient's name......:\", display_name)\n print(\"Patient id..........:\", dataset.PatientID)\n print(\"Patient's Age.......:\", dataset.PatientAge)\n print(\"Patient's Sex.......:\", dataset.PatientSex)\n print(\"Modality............:\", dataset.Modality)\n print(\"Body Part Examined..:\", dataset.BodyPartExamined)\n \n \n if 'PixelData' in dataset:\n rows = int(dataset.Rows)\n cols = int(dataset.Columns)\n print(\"Image size.......: {rows:d} x {cols:d}, {size:d} bytes\".format(\n rows=rows, cols=cols, size=len(dataset.PixelData)))\n if 'PixelSpacing' in dataset:\n print(\"Pixel spacing....:\", dataset.PixelSpacing)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_pixel_array(dataset, figsize=(5,5)):\n plt.figure(figsize=figsize)\n plt.grid(False)\n plt.imshow(dataset.pixel_array)\n plt.show()\n \ni = 1\nnum_to_plot = 5\nfor file_name in os.listdir('../input/siim-isic-melanoma-classification/train/'):\n file_path = os.path.join('../input/siim-isic-melanoma-classification/train/',file_name)\n dataset = dcm.dcmread(file_path)\n show_dcm_info(dataset)\n plot_pixel_array(dataset)\n \n if i >= num_to_plot:\n break\n \n i += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. tSNE Embeddings \n* t-SNE stands for t-distributed stochastic neighbor embedding. It is a technique for dimensionality reduction that is best suited for the visualization of high dimensional data-set. t-SNE is a randomized algorithm, i;e every time we run the algorithm it returns slightly different results on the same data-set. To control this we set a random state with some arbitrary value. Random state is used here to seed the cost function of the algorithm.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n# https://www.kaggle.com/tunguz/melanoma-tsne-and-umap-embeddings-with-rapids/?\n\n# INSTALL RAPIDS OFFLINE (FROM KAGGLE DATASET). TAKES 1 MINUTE :-)\nimport sys\n!cp ../input/rapids/rapids.0.13.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + [\"/opt/conda/envs/rapids/lib/python3.6\"] + [\"/opt/conda/envs/rapids/lib/python3.6/site-packages\"] + sys.path\n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load resized image numpy arry\nfrom cuml.manifold import TSNE\nfrom cuml.decomposition import PCA\n\ntrain = np.load('../input/siimisic-melanoma-resized-images/x_train_32.npy')\ntrain = train.reshape((train.shape[0], 32*32*3))\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"time_start = time.time()\npca = PCA(n_components=2)\npca_2D = pca.fit_transform(train.astype(np.float32))\nprint(pca.explained_variance_ratio_)\nprint(' ')\nprint('PCA done! Time elapsed: {} seconds'.format(time.time()-time_start))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca_2D_one = pca_2D[:,0]\npca_2D_two = pca_2D[:,1]\nplt.figure(figsize=(16,10))\nsns.scatterplot(\n x=pca_2D_one, y=pca_2D_two,\n hue=train_df['target'].values,\n palette=sns.color_palette(\"Paired\", 2),\n legend=\"full\",\n alpha=0.7\n)\nplt.xlabel('Principal Component 1', fontsize = 15)\nplt.ylabel('Principal Component 2', fontsize = 15)\nplt.title('2 component PCA', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By using the attribute explained_variance_ratio_, you can see that the first principal component contains 70.27% of the variance and the second principal component contains 0.07% of the variance. Together, the two components contain 70.34% of the information. From graph we can say that image pixel data not clearly have some clustering structure. \n\nLet's now take a look at what TSNE can discern.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"time_start = time.time()\ntsne = TSNE(n_components=2)\ntsne_2D = tsne.fit_transform(train)\nprint('t-SNE done! Time elapsed: {} seconds'.format(time.time()-time_start))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne_2D_one = tsne_2D[:,0]\ntsne_2D_two = tsne_2D[:,1]\nplt.figure(figsize=(16,10))\nsns.scatterplot(\n x=tsne_2D_one, y=tsne_2D_two,\n hue=train_df['target'].values,\n palette=sns.color_palette(\"Set2\", 2),\n legend=\"full\",\n alpha=0.7\n)\nplt.xlabel('t-SNE 1', fontsize = 15)\nplt.ylabel('t-SNE 2', fontsize = 15)\nplt.title('2 component t-SNE' ,fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is already a significant improvement over the PCA visualisation we used earlier. We see that the image pixel data definitley has some clustering structure. At first glance it doesn't seem that it's easy to clearly separate the target cases, partly becasue they comprise less than 2% of all the points. However, it seems like the most of them are concentrated in the lower areas.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## --Thank you for reading my kernel\n## Give me your feedback and if you like my kernel please UPVOTE will be appreciated.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}