{"cells":[{"metadata":{},"cell_type":"markdown","source":"# SIIM Playground EDA\n\nIn this notebook, I will focus on exploring through EDA the tabular data joined with the images and compare properties between the training and testing set to see if they have similarities.\n\n## 1. Introduction: Overview of the data\n\n### 1.1 Imports","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport cv2\nimport os\nimport shutil\nimport matplotlib.pyplot as plt\n\ninput_dir = \"/kaggle/input/siim-isic-melanoma-classification\"\ntrain_dir = os.path.join(input_dir, \"jpeg/train\")\ntest_dir = os.path.join(input_dir, \"jpeg/test\")\n\n\ntrain_csv = pd.read_csv(os.path.join(input_dir,\"train.csv\"))\ntest_csv = pd.read_csv(os.path.join(input_dir,\"test.csv\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"| Column                        | Type        | Description                                                                                    |\n|-------------------------------|-------------|------------------------------------------------------------------------------------------------|\n| image_name                    | String      | Identifies the name of the image file corresponding to the given row. Should not be redundant. |\n| patient_id                    | String      | Identifier of the patient from whom the skin picture has been taken. Could be redundant.       |\n| sex                           | Categorical | Identifies the sex of the patient.                                                             |\n| age_approx                    | Integer     | Age rounded to the nearest multiple of 5.                                                      |\n| anatom_site_general_challenge | Categorical | Defines the location of the picture on the patient's body.                                     |\n| diagnosis                     | Categorical | Provide details about the diagnosis. Is only found in the train dataset.                       |\n| benign_malignant              | Categorical | Target classes to be predicted (benign=0, malignant=1)                                         |","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 1.2 Features analysis\n\nIn this part, I will work on the previously described features and try to gain insights on data properties.\n\n#### 1.2.a Patient Id\n---\nThe '*patient_id*' column is a patient identifier. Using it, we could know if a same single patient has been taken care of multiple times and try to see how redundant this feature is within the dataset.\n\nWe will first split it to see if the nomenclature always stays the same (i.e '*IP_XXXX*' where '*XXXX*' consists of 7 numbers). ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[\"preffix_id\"] = train_csv[\"patient_id\"].apply(lambda x: x.split(\"_\")[0])\ntrain_csv[\"suffix_id\"] = train_csv[\"patient_id\"].apply(lambda x: x.split(\"_\")[-1])\n\ntest_csv[\"preffix_id\"] = test_csv[\"patient_id\"].apply(lambda x: x.split(\"_\")[0])\ntest_csv[\"suffix_id\"] = test_csv[\"patient_id\"].apply(lambda x: x.split(\"_\")[-1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[\"preffix_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_csv[\"preffix_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see here that there only exists a single preffix in the *id* field being *IP*.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"\\tTRAIN SUFFIX ID VALUE COUNTS\\n---\\n\")\nprint(train_csv[\"suffix_id\"].value_counts())\nprint(\"-----------------------------------\")\nprint(\"\\tTEST SUFFIX ID VALUE COUNTS\\n---\\n\")\nprint(test_csv[\"suffix_id\"].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ids are redundant: same patients seem to be present multiple times. Are the ids always the same size (7 characters) ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[\"suffix_id\"].apply(lambda x: len(x)).value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Are some patient both in the test **and** in the train set ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"list(set(train_csv[\"suffix_id\"].unique()) & set(test_csv[\"suffix_id\"].unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems like none are shared between the two sets.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### 1.2.b Patient age \n---\nType: Integer\n\nDescription: Age rounded to the nearest multiple of 5.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(25,10))\nplt.ylabel(\"Age count (in # of entry)\")\nplt.xlabel(\"Age\")\n\nax_tr = train_csv[\"age_approx\"].plot(kind='hist')\nax_te = test_csv[\"age_approx\"].plot(kind='hist')\n\nax_tr.set_label(\"Train set\")\nax_te.set_label(\"Test set\")\n\nL=plt.legend()\nL.get_texts()[0].set_text('Train set')\nL.get_texts()[1].set_text('Test set')\n\nplt.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"\\tTRAIN AGE DESCRIPTION\\n---\\n\")\nprint(train_csv[\"age_approx\"].describe())\nprint('-----------------------------')\nprint(\"\\tTEST AGE DESCRIPTION\\n---\\n\")\nprint(test_csv[\"age_approx\"].describe())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While having very similar distribution, the only noticeable difference we can find between these two descriptions are:\n\n- mean value being slightly higher for the test set (but it is almost not noticeable);\n- the value is **not skewed** as the mean and the median are almost equal for both sets;\n- the training set has very young patients, while the test sets only have patients older than 10.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Train age unique values: \",np.sort(train_csv[\"age_approx\"].unique()))\nprint(\"Test age unique values: \",np.sort(test_csv[\"age_approx\"].unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training set seems to suffer from nan ages. How many are there ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"{train_csv['age_approx'].isnull().sum()} rows have a 'nan' age out of {len(train_csv)} total rows\").","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Which patients have 'nan' age ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[train_csv['age_approx'].isnull()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We seem to notice that the sex column is also empty (nan here).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[train_csv[\"age_approx\"].isnull()][\"suffix_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3 patients have null age. Let's check their sex column now.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[train_csv[\"age_approx\"].isnull() & train_csv[\"sex\"].isnull()][\"suffix_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These two patients neither have gender nor age. Are all entries of these patients without an age ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"nan_age = ['5205991', '9835712', '0550106']\ntrain_csv[train_csv[\"suffix_id\"].apply(lambda x: x in nan_age)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Dataframes are of the **same size** => patient without age are never labelled as having an age.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### **1.2.c Patient sex**\n---\nType: Categorical\n\nDescription: Identifies the sex of the patient","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We noticed Nan values before, how often are they found ? Also, what is the distribution of men/women in both sets ? Is there any missing sex in test set ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv_cp = train_csv.copy()\ntrain_csv_cp[\"sex\"] = train_csv_cp[\"sex\"].fillna(\"null\")\ntest_csv_cp = test_csv.copy()\ntest_csv_cp[\"sex\"] = test_csv_cp[\"sex\"].fillna(\"null\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"\\tTRAIN SEX VALUE COUNTS\\n---\\n\")\nprint(train_csv_cp[\"sex\"].value_counts())\nprint(\"-----------------------------------\")\nprint(\"\\tTEST SEX VALUE COUNTS\\n---\\n\")\nprint(test_csv_cp[\"sex\"].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Which individuals have missing sex ?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[train_csv[\"sex\"].isnull()][\"suffix_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The individuals with missing sex are the same than for missing age.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"#### 1.2.d anatom_site_general_challenge\n---\nType: Categorical\n\nDescription: Defines the location of the picture on the patient's body.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"\\tTRAIN ANATOM SITE VALUE COUNTS:\\n---\\n\")\nprint(train_csv[\"anatom_site_general_challenge\"].fillna(\"null\").value_counts())\nprint('--------------------------------------')\nprint(\"\\tTEST ANATOM SITE VALUE COUNTS:\\n---\\n\")\nprint(test_csv[\"anatom_site_general_challenge\"].fillna(\"null\").value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both train and test sets have missing values. Distributions look similar, let's plot frequency of each value for both sets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(25,10))\nplt.ylabel(\"Anatomy site count (in frequency)\")\nplt.xlabel(\"Anatomy site\")\n\nax_tr = train_csv[\"anatom_site_general_challenge\"].fillna(\"null\").value_counts(normalize=True).plot()\nax_te = test_csv[\"anatom_site_general_challenge\"].fillna(\"null\").value_counts(normalize=True).plot()\n\nax_tr.set_label(\"Train set\")\nax_te.set_label(\"Test set\")\n\nL=plt.legend()\nL.get_texts()[0].set_text('Train set')\nL.get_texts()[1].set_text('Test set')\n\nplt.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Distributions are indeed very similar.\nAre the null values coming from the same patients (as for age or sex ?). Here it would seem unlikely except if multiple spots have been photographed for the same patients at the same body parts.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"\\tTRAIN NULL VALUE COUNTS:\\n---\\n\")\nprint(train_csv[train_csv[\"anatom_site_general_challenge\"].isnull()][\"suffix_id\"].value_counts())\nprint(\"-----------------------\")\nprint(\"\\tTEST NULL VALUE COUNTS:\\n---\\n\")\nprint(test_csv[test_csv[\"anatom_site_general_challenge\"].isnull()][\"suffix_id\"].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Train missing values come from 220 different patients, with 102 rows coming from a same single patient ‘3057277’.\n- Test missing values come from 61 different patients, with 240 rows coming from a same single patient ‘3579794’.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_csv[train_csv[\"anatom_site_general_challenge\"].isnull()][\"benign_malignant\"].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 1.2.e Diagnosis\n---\nType: Categorical\n\nDescription: Provide details about the diagnosis. Is only found in the train dataset.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"\\tBENIGN DIAGNOSIS VALUE COUNT:\\n---\\n\")\nprint(train_csv[train_csv[\"benign_malignant\"] == \"benign\"][\"diagnosis\"].value_counts())\nprint('------------------------')\nprint(\"\\tMALIGNANT DIAGNOSIS VALUE COUNT:\\n---\\n\")\nprint(train_csv[train_csv[\"benign_malignant\"] != \"benign\"][\"diagnosis\"].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This seems to indicate that benign diagnosis are left unknown, while at least being sure that what is observed is **NOT** a melanoma.\n\n#### 1.2.f Benign/Malignant\n---\nType: Categorical\n\nDescription: Target variable, describes if the spot is benign or not.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv[\"benign_malignant\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Using ML algorithms to find correlations between variables.\n\nWe will now try to develop a predictive algorithm that will only use the table data to predict benign/malignant in order to figure out eventual correlations. The diagnosis column will be removed from this procedure and new features will be created.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"new_train_csv = train_csv.copy()\nto_drop = [\"image_name\", \"diagnosis\", \"benign_malignant\", \"preffix_id\", \"patient_id\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2.a Sex\n---","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"new_train_csv[\"is_female\"] = new_train_csv[\"sex\"].apply(lambda x: 0 if type(x) != float and x.lower() == 'male' else 1 if type(x) != float else -1)\nto_drop.append(\"sex\")\nnew_train_csv.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2.b Anatom Site General Challenge\n---","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"anatom_classes = {\n    val: i for i, val in enumerate(new_train_csv[\"anatom_site_general_challenge\"].unique())\n}\nprint(\"String to class: \"+str(anatom_classes))\nnew_train_csv[\"anatom_classes\"] = new_train_csv[\"anatom_site_general_challenge\"].apply(lambda x: anatom_classes[x])\nto_drop.append(\"anatom_site_general_challenge\")\nnew_train_csv.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2.c Age Approx\n---","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"new_train_csv[\"reduced_age\"] = new_train_csv[\"age_approx\"].apply(lambda x: int(x/5) if np.isnan(x) == False else -1)\nto_drop.append(\"age_approx\")\nnew_train_csv.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2.d Patient id\n---","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"suffix_classes = {\n\n    val: i for i,val in enumerate(new_train_csv[\"suffix_id\"].unique())\n\n}\n\nnew_train_csv[\"suffix_classes\"] = new_train_csv[\"suffix_id\"].apply(lambda x: suffix_classes[x])\nto_drop.append(\"suffix_id\")\nto_drop.append(\"suffix_classes\")\nnew_train_csv.head()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2.e Cleaning/splitting data and creating classifier","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y = new_train_csv.drop(to_drop, axis=1).iloc[:, 0].to_numpy()\nX = new_train_csv.drop(to_drop, axis=1).iloc[:, 1:].to_numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, roc_auc_score\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.cluster import KMeans\nN_CLUSTERS = 200","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf = KMeans(N_CLUSTERS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"kmeans = clf.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_per_cluster = {}\npreds = kmeans.predict(X_train)\n\nfor i in range(N_CLUSTERS):\n    classes = y_train[np.where(preds == i)]\n    c, counts = np.unique(classes, return_counts=True)\n    s = sum(counts)\n    \n    counts = counts/s\n        \n    class_per_cluster[i] = counts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_preds = kmeans.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_preds_classes = []\nfor pred in y_preds:\n    probs = class_per_cluster[pred]\n    \n    c = np.random.choice(list(range(len(probs))), p=probs)\n    \n    y_preds_classes.append(c)\n    \nprint(roc_auc_score(y_preds_classes, y_test))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}