{"cells":[{"metadata":{},"cell_type":"markdown","source":"*This is a work in progress*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Melanoma is a type of skin cancer that can quickly spread to other organs if not treated, which means that it is very dangerous. However, if caught early, it can be treated with minor surgery.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this contest, you take an image of a lesion, predict the probability of the image being a malignant tumor.  Then, you give the image a label.\n\n0 is benign; 1 is malign.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The evaluation is with the receiver operating characteristic curve.\n\nThe true positive rate (the probability of proper detection) is plotted against the false-positive rate (the probability of false alarm).\n\nThe true positive rate (TPR) is equal to: **1 - misses**\n\nThe false positive rate (FPR) is equal to: **1 - correct rejections**\n","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfrom os import listdir\nfrom os.path import isfile, join\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n%matplotlib inline\nimport pydicom\nfrom sklearn.impute import SimpleImputer\nprint(\"Complete\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. The Data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"File Paths:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_jpeg_dir = '../input/siim-isic-melanoma-classification/jpeg/train/'\ntrain_jpeg = [f for f in listdir(train_jpeg_dir) if isfile(join(train_jpeg_dir, f))]\n\ntest_jpeg_dir = '../input/siim-isic-melanoma-classification/jpeg/test/'\ntest_jpeg = [f for f in listdir(test_jpeg_dir) if isfile(join(test_jpeg_dir, f))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dcm_dir = '../input/siim-isic-melanoma-classification/train/'\ntrain_dcm = [f for f in listdir(train_dcm_dir) if isfile(join(train_dcm_dir, f))]\n\ntest_dcm_dir = '../input/siim-isic-melanoma-classification/test/'\ntest_dcm = [f for f in listdir(test_dcm_dir) if isfile(join(test_dcm_dir, f))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train and Test Information","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train: jpeg and dcm images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(15, 10))\ncolumns = 4\nrows = 3\nfor i in range(1, columns*rows +1):\n    path = train_jpeg_dir + train_jpeg[i]\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(mpimg.imread(path))\n    fig.add_subplot","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(15, 10))\ncolumns = 4\nrows = 3\nfor i in range(1, columns*rows +1):\n    ds = pydicom.dcmread(train_dcm_dir + train_dcm[i])\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(ds.pixel_array, cmap=plt.cm.bone)\n    fig.add_subplot","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Test: jpeg and dcm images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(15, 10))\ncolumns = 4\nrows = 3\nfor i in range(1, columns*rows +1):\n    path = test_jpeg_dir + test_jpeg[i]\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(mpimg.imread(path))\n    fig.add_subplot","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(15, 10))\ncolumns = 4\nrows = 3\nfor i in range(1, columns*rows +1):\n    ds = pydicom.dcmread(test_dcm_dir + test_dcm[i])\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(ds.pixel_array, cmap=plt.cm.bone)\n    fig.add_subplot","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Basic Graphs Representing Distribution of Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"features_first = [\"sex\", \"age_approx\", \"anatom_site_general_challenge\"]\nfeatures_train = [\"diagnosis\", \"benign_malignant\", \"target\"]\nfeatures = [\"sex\", \"age_approx\", \"anatom_site_general_challenge\", \"diagnosis\", \"benign_malignant\", \"target\"]\n\nsns.set(style=\"ticks\", color_codes=True)\nfig = plt.gcf()\nfig.set_size_inches(15, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in features_first:\n    sns.set(font_scale=0.6)\n    plt.title(\"Count of \" + i + \" train\")\n    sns.catplot(x = i, kind=\"count\", palette=\"ch:.25\", data=train)\n    \n    sns.set(font_scale=0.6)\n    plt.title(\"Count of \" + i + \" test\")\n    sns.catplot(x = i, kind=\"count\", palette=\"ch:.25\", data=test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in features_train:\n    sns.set(font_scale=0.6)\n    plt.title(\"Count of \" + i + \" train\")\n    sns.catplot(x = i, kind=\"count\", palette=\"ch:.25\", data=train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From these graphs, we can see several things:\n- the distribution based on sex is different in the train group than the test group\n- the age distribution is more similar between the train group and the test group but slightly different\n- the train group has most of the lesions in the torso\n- the test group has the same distribution of lesions in train and test group","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in features_first:\n    sns.set(font_scale=0.7)\n    plt.title(\"belign_malignant for \" + i)\n    sns.catplot(x=i,kind='count', hue = \"benign_malignant\", palette=\"ch:.25\", data=train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in features_first:\n    sns.set(font_scale=0.7)\n    plt.title(\"target for \" + i)\n    sns.catplot(x=i,kind='count', hue = \"target\", palette=\"ch:.25\", data = train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in features_first:\n    sns.set(font_scale=0.7)\n    plt.title(\"melanoma for \" + i)\n    sns.catplot(x=i,kind= 'count', hue= \"diagnosis\", palette=\"ch:.25\", data = train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Missing Values:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train Set')\nprint(train.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Test Set')\nprint(test.info())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Filling null values with simple imputer (https://gist.github.com/wmlba/07a36758096b9462431b3e7daca3ad41)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"imp_mean_train = SimpleImputer( strategy='most_frequent')\ntrain_no_null = pd.DataFrame(imp_mean_train.fit_transform(train))\ntrain_no_null.columns=train.columns\ntrain_no_null.index=train.index\ntrain_no_null.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imp_mean_test = SimpleImputer( strategy='most_frequent')\ntest_no_null = pd.DataFrame(imp_mean_train.fit_transform(test))\ntest_no_null.columns=test.columns\ntest_no_null.index=test.index\ntest_no_null.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Graphs of Columns with Missing Values Filled Compared To Those Without","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in features:\n    sns.set(font_scale=0.6)\n    plt.title(\"Count of \" + i + \" without filling missing values\")\n    sns.catplot(x = i, kind=\"count\", palette=\"ch:.25\", data=train)\n    \n    sns.set(font_scale=0.6)\n    plt.title(\"Count of \" + i + \" filling missing values\")\n    sns.catplot(x = i, kind=\"count\", palette=\"ch:.25\", data=train_no_null)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in features_first:\n    sns.set(font_scale=0.6)\n    plt.title(\"Count of \" + i + \" without filling missing values\")\n    sns.catplot(x = i, kind=\"count\", palette=\"ch:.25\", data=test)\n    \n    sns.set(font_scale=0.6)\n    plt.title(\"Count of \" + i + \" filling missing values\")\n    sns.catplot(x = i, kind=\"count\", palette=\"ch:.25\", data=test_no_null)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n**Attribution:**\n\nThe ISIC 2020 Challenge Dataset https://doi.org/10.34970/2020-ds01 (c) by ISDIS, 2020\n\nCreative Commons Attribution-Non Commercial 4.0 International License.\n\nThe dataset was generated by the International Skin Imaging Collaboration (ISIC) and images are from the following sources: Hospital Clínic de Barcelona, Medical University of Vienna, Memorial Sloan Kettering Cancer Center, Melanoma Institute Australia, The University of Queensland, and the University of Athens Medical School.\n\nYou should have received a copy of the license along with this work.\n\nIf not, see https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Information about Melanoma:\nhttps://www.skincancer.org/skin-cancer-information/melanoma/","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Inspiration**:\n\nhttps://www.kaggle.com/nxrprime/siim-eda-augmentations-model-seresnet-unet/?#six\n\nhttps://www.kaggle.com/parulpandey/melanoma-classification-eda-starter","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}