{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MELANOMA CLASSIFICATION","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This kernel demostrates the steps followed in building a classifier for the various skin images shared under the SIIM-ISIC Melanoma Classification competition. The final model predicts the probabilities of malignancy of the lesions in the images. Let's start!!!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# About Melanoma","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Skin cancer is the most prevalent type of cancer. Melanoma, specifically, is responsible for 75% of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It's also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective.\n\nCurrently, dermatologists evaluate every one of a patient's moles to identify outlier lesions or “ugly ducklings” that are most likely to be melanoma. Existing AI approaches have not adequately considered this clinical frame of reference. Dermatologists could enhance their diagnostic accuracy if detection algorithms take into account “contextual” images within the same patient to determine which images represent a melanoma. If successful, classifiers would be more accurate and could better support dermatological clinic work.\n\nAs the leading healthcare organization for informatics in medical imaging, the Society for Imaging Informatics in Medicine (SIIM)'s mission is to advance medical imaging informatics through education, research, and innovation in a multi-disciplinary community. SIIM is joined by the International Skin Imaging Collaboration (ISIC), an international effort to improve melanoma diagnosis. The ISIC Archive contains the largest publicly available collection of quality-controlled dermoscopic images of skin lesions.\n\nMelanoma is a deadly disease, but if caught early, most melanomas can be cured with minor surgery. Image analysis tools that automate the diagnosis of melanoma will improve dermatologists' diagnostic accuracy. Better detection of melanoma has the opportunity to positively impact millions of people.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# About the Data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The dataset used is under CC BY-NC 4.0 with the following attribution:\n\nThe ISIC 2020 Challenge Dataset https://doi.org/10.34970/2020-ds01 (c) by ISDIS, 2020\n\nCreative Commons Attribution-Non Commercial 4.0 International License.\n\nThe dataset was generated by the International Skin Imaging Collaboration (ISIC) and images are from the following sources: Hospital Clínic de Barcelona, Medical University of Vienna, Memorial Sloan Kettering Cancer Center, Melanoma Institute Australia, The University of Queensland, and the University of Athens Medical School.\n\nYou should have received a copy of the license along with this work.\n\nIf not, see https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# EXPLORATORY DATA ANALYSIS(EDA)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I started by making some imports.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport tempfile\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n    #for filename in filenames:\n        #print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib.image import imread\nimport cv2\n# Technically not necessary in newest versions of jupyter\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I read the contents of the train and test csv into two dataframes train and test. Here I have worked only with the images under the jpeg folders. As a start I got the image features of the images in both train and test folders extracted using a DenseNet121 architecture. The reference to the kernel is attached below:-\n\nhttps://www.kaggle.com/siddhartamukherjee/melanoma-classification-image-feature-extraction\n\nThe image features csv files were included here under the folder image features. These image features were copied in two dataframes train_features and test_features as shown below:-","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/siim-isic-melanoma-classification/train.csv\")\ntest = pd.read_csv(\"../input/siim-isic-melanoma-classification/test.csv\")\nsubmission = pd.read_csv(\"../input/siim-isic-melanoma-classification/sample_submission.csv\")\ntrain_features = pd.read_csv(\"../input/siimisicimagefeaturesextracted/Image Features/train_img_features.csv\")\ntest_features = pd.read_csv(\"../input/siimisicimagefeaturesextracted/Image Features/test_img_features.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I changed the column name under which the image names in the train_features as well as the test_features dataframes appeared to **\"image_name\"** as in both the train_features and test_features dataframe they appeared with a name **\"Unnamed: 0\"**.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_features = train_features.rename(columns={\"Unnamed: 0\" : \"image_name\"})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_features = test_features.rename(columns={\"Unnamed: 0\" : \"image_name\"})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To check if the change reflected we displayed the first five columns of both the train_features and the test_features dataframes. From the below results it can be also seen that both train_features and the test_features dataframes contains 256 features per image.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_features.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_features.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I displayed the first five rows of the train set and also checked the individual column information as shown below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The train set contains dimension is 33126 row and 8 columns.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above results it can be seen that the columns sex,age_approx and anatom_site_general_challenge under the train dataframe have some null values. It would be interesting to see how we can deal with these missing values.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Just like the train set I then displayed the first rows of the test set as shown below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The test set is of dimension 10982 rows and 5 columns as shown below. The test set don't have the columns diagnosis, beningn_malignant and target.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From, the test dataframe's info column it seems that the anatom_Site_general_challenge column has some null values.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I checked the count of images that we have in train and test set. The train set has 33126 images and test set has 10982 images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"path, dirs, files = next(os.walk(\"/kaggle/input/siim-isic-melanoma-classification/jpeg/train\"))\nfile_count = len(files)\nfile_count","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path, dirs, files = next(os.walk(\"/kaggle/input/siim-isic-melanoma-classification/jpeg/test\"))\nfile_count = len(files)\nfile_count","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After this I analyzed how many unique patient ids is present. From the below results it can be found that in the train set we have 2056 unique patient ids.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train['patient_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I made a data analysis of both the train and test set based on gender as shown below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(x='sex',data=train,ax=ax[0])\nax[0].set_xlabel(\" \")\nax[0].set_title(\"Gender counts in train set\")\n\nsns.countplot(x='sex',data=test,ax=ax[1])\nax[1].set_xlabel(\" \")\nax[1].set_title(\"Gender counts in test set\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"People belonging to male gender is more in both train and test set. In test set male gender count is even more than in train set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['sex'].isnull() == True]['target'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also, as shown above for 65 records in the training set we have sex as null. Also, the target values for those records are 0. Since, we have maximum images belonging to target 0 so we may drop these 65 records during our model building.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Next I made a count plot to check the age distribution in both train and test set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(x='age_approx',data=train,ax=ax[0])\nax[0].set_xlabel(\" \")\nax[0].set_title(\"Age distribution in train set\")\n\nsns.countplot(x='age_approx',data=test,ax=ax[1])\nax[1].set_xlabel(\" \")\nax[1].set_title(\"Age distribution in test set\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems the age distribution is uneven in test set.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Next I made a graphical display of the location of the images belonging to both train and test set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_train = train.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\ntemp_test = test.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\n\nfig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.barplot(x=temp_train.index.values, y=temp_train.values,ax=ax[0])\nax[0].set_xlabel(\" \")\nlabels = ax[0].get_xticklabels()\nax[0].set_xticklabels(labels, rotation=90)\nax[0].set_title(\"Image location in train set\")\n\nsns.barplot(x=temp_test.index.values, y=temp_test.values,ax=ax[1])\nax[1].set_xlabel(\" \")\nlabels = ax[1].get_xticklabels()\nax[1].set_xticklabels(labels, rotation=90)\nax[1].set_title(\"Image location in test set\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of image locations is same in both train and test set.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Next, lets see what the diagnosis column of the train set has to tell us.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"chart = sns.countplot(x='diagnosis', data = train)\nchart.set_xticklabels(chart.get_xticklabels(), rotation=45, horizontalalignment='right')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems for most images diagnosis is unknown. So, we may drop this column during our model building.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Next, let's analyze the columns **target** and **benign_malignant**. Both,the columns are providing the same message. Hence, during model building I will only use target. The column **benign_malignant** will be dropped.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(x='target',data=train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(x='benign_malignant',data=train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From both the count plots we can see that the dataset is highly imbalanced i.e we have very minimal number of images belonging to class malignant.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Since, there is comparitively less number of malignat images so we will try resampling to increase the amount of malignat images during our data preprocessing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Paths to train and test images\ntrain_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train/'\ntest_img_path = '/kaggle/input/siim-isic-melanoma-classification/jpeg/test/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I will display some images in the train and test set. The code for this was referred from the kernel given below:-\n\nhttps://www.kaggle.com/siddhartamukherjee/siim-isic-melanoma-analysis-eda-prediction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Let's take a look at some benign tumours from the train set.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(50, 50))\nfor i,idx in enumerate(np.random.choice(train[train['benign_malignant']=='benign'].index,8)):\n    img = cv2.imread(train_img_path+str(train.loc[idx,'image_name'])+'.jpg')\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    fig.add_subplot(2, 4, i+1)\n    plt.imshow(img)\n    plt.title('Patient_id: '+train.loc[idx,'patient_id']+'\\n'\\\n              +'Site: '+str(train.loc[idx,'anatom_site_general_challenge'])+'\\n'\\\n              +'Sex: '+str(train.loc[idx,'sex'])+'\\n'\\\n              +'Approximate Age: '+str(train.loc[idx,'age_approx'])+'\\n'\\\n              +'Diagnosis: '+str(train.loc[idx,'diagnosis']),fontsize=30)\n    plt.axis(\"off\")\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now let's take a look at some malignant tumours from the train set.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(50, 50))\nfor i,idx in enumerate(np.random.choice(train[train['benign_malignant']=='malignant'].index,8)):\n    img = cv2.imread(train_img_path+str(train.loc[idx,'image_name'])+'.jpg')\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    fig.add_subplot(2, 4, i+1)\n    plt.imshow(img)\n    plt.title('Patient_id: '+train.loc[idx,'patient_id']+'\\n'\\\n              +'Site: '+str(train.loc[idx,'anatom_site_general_challenge'])+'\\n'\\\n              +'Sex: '+str(train.loc[idx,'sex'])+'\\n'\\\n              +'Approximate Age: '+str(train.loc[idx,'age_approx'])+'\\n'\\\n              +'Diagnosis: '+str(train.loc[idx,'diagnosis']),fontsize=30)\n    plt.axis(\"off\")\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Finally,let's check what we have to predict.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(50, 50))\nfor i,idx in enumerate(np.random.choice(test.index,8)):\n    img = cv2.imread(test_img_path+str(test.loc[idx,'image_name'])+'.jpg')\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    fig.add_subplot(2, 4, i+1)\n    plt.imshow(img)\n    plt.title('Patient_id: '+test.loc[idx,'patient_id']+'\\n'\\\n              +'Site: '+str(test.loc[idx,'anatom_site_general_challenge'])+'\\n'\\\n              +'Sex: '+str(test.loc[idx,'sex'])+'\\n'\\\n              +'Approximate Age: '+str(test.loc[idx,'age_approx']),fontsize=30)\n    plt.axis(\"off\")\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATA PREPROCESSING","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"From the train set I took the columns image_name, sex, age_approx, anatom_site_general_challenge and traget and created a new dataframe df. The columns diagnosis , bening_malignant won't be of any use. Hence, I didnot consider them for model building.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train[['image_name','sex', 'age_approx','anatom_site_general_challenge','target']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I merged the dataframe df with the train features by image_name.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.merge(df, train_features, on='image_name')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After merging the dataframe looks as shown below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, the same set of steps were repeated for the test set and a dataset df1 was created. Note, that df1 has no target column as this would be added later after model's prediction.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df1 = test[['image_name','sex','age_approx','anatom_site_general_challenge']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df1 = pd.merge(df1, test_features, on='image_name')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df1.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As already highlighted in my section EDA I dropped the 65 records which has null value in the sex column of the dataset df. This won't impact our model's performance much as the records having null values under the sex column belonged to target class 0. And as highlighted the taget class is highly imbalanced with maximum number of images belonging to target 0, so removing 65 records belonging to target 0 won't cause much impact.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.dropna(axis=0, subset=['sex'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The sex column has 2 unique values **\"male\"** and **\"female\"**. Since, it is hard to make a model with text data so I converted the values to numeric with male replaced by 0 and female replaced by 1. This was done in both the dataframes df and df1.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sex = {\"male\":0, \"female\":1}\ndf['sex'] = df['sex'].map(sex)\ndf1['sex'] = df1['sex'].map(sex)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next coming to the age_approx column I already highlighted during EDA that it has some null values so best thing to do would be to replace the null values with the mean age. This was done in both the dataframes df and df1.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df['age_approx'].fillna(df['age_approx'].mean(), inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df1['age_approx'].fillna(df1['age_approx'].mean(), inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I replaced the null values appearing in anatom_site_general_challenge with value unknown and then assigned a class to each values of this column as shown below. This was done for both dataframes df and df1.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df['anatom_site_general_challenge'].fillna('unknown', inplace=True)\ndf1['anatom_site_general_challenge'].fillna('unknown', inplace=True)\nimg_loc = {'head/neck':1, 'upper extremity':2, 'lower extremity':3, 'torso':4, 'palms/soles':5, 'oral/genital':6, 'unknown':7}\ndf['anatom_site_general_challenge'] = df['anatom_site_general_challenge'].map(img_loc)\ndf1['anatom_site_general_challenge'] = df1['anatom_site_general_challenge'].map(img_loc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I one hot encoded the columns sex and anatom_site_general_challenge as shown below for both the datasets df and df1.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.get_dummies(df, columns=[\"sex\"])\ndf1 = pd.get_dummies(df1, columns=[\"sex\"])\ndf = pd.get_dummies(df, columns=[\"anatom_site_general_challenge\"])\ndf1 = pd.get_dummies(df1, columns=[\"anatom_site_general_challenge\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The results after one hot encoding is shown below for both the dataframes df and df1.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df1.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I calculated the percentage of target 1 records in the dataframe. It seems the target 1 records only forms 1.77% of the entire dataset. This hints that we have to resmaple and increase the target 1 records to prevent overfitting of the model to the target class 0.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"neg, pos = np.bincount(df['target'])\ntotal = neg + pos\nprint('Examples:\\n    Total: {}\\n    Positive: {} ({:.2f}% of total)\\n'.format(\n    total, pos, 100 * pos / total))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, lets make the X and y set. So, all the columns except the image_name and target will go to X. The target column would be our y set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X = df.drop(columns = ['target','image_name'], axis=1)\ny = df['target']\ndf_test = df1.drop(columns = ['image_name'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I made a train test split of our X and y dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.33, random_state=101)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I will use **Adaptive Synthetic Sampling (ADASYN)** to increase the minority class. It is a type of **Synthetic Minority Oversampling Technique(SMOTE)** which generates more synthetic examples in regions of the feature space where the density of minority examples is low, and fewer or none where the density is high. More details of different types of SMOTE approaches can be found from the below link:-\n\nhttps://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/\n\nAfter applying ADASYN we can see that the samples belonging to target class 1 increased from 382 to 21754 in the X_train dataset. Also, SMOTE is applied only on the training set and not on the test set because we want to keep real data in test set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from imblearn.over_sampling import SMOTE, SVMSMOTE, ADASYN,BorderlineSMOTE\nfrom imblearn.under_sampling import RandomUnderSampler\nfrom imblearn.under_sampling import NearMiss \nfrom numpy import mean\nfrom imblearn.pipeline import Pipeline\nfrom collections import Counter\n# summarize class distribution\ncounter = Counter(y_train)\nprint(counter)\n# Oversample with SMOTE and random undersample for imbalanced dataset\nover = ADASYN()\n#under = RandomUnderSampler(sampling_strategy=0.5)\n#steps = [('o', over), ('u', under)]\n#pipeline = Pipeline(steps=steps)\nX_train, y_train= over.fit_resample(X_train, y_train)\n# summarize the new class distribution\ncounter = Counter(y_train)\nprint(counter)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next I displayed the shape of X_train and X_test to check how many records we now have in train and test set. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above results it seems the train set has 43522 samples and test set has 10911 samples.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# MODEL BUILDING\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now, its time build my model. I chose Light GBM Classifier to build my model.Light GBM is prefixed as ‘Light’ because of its high speed. Light GBM can handle the large size of data and takes lower memory to run. Another reason of why Light GBM is popular is because it focuses on accuracy of results. LGBM also supports GPU learning. \n\nIn this model I will also do hyper parameter tuning of LGBMClassifier with Random search followed by Grid Search. All of these are shown below. The implementation was referred from the below kernel:-\n\nhttps://www.kaggle.com/mlisovyi/lightgbm-hyperparameter-optimisation-lb-0-761\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* # Prepare learning rate shrinkage","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def learning_rate_010_decay_power_099(current_iter):\n    base_learning_rate = 0.1\n    lr = base_learning_rate  * np.power(.99, current_iter)\n    return lr if lr > 1e-3 else 1e-3\n\ndef learning_rate_010_decay_power_0995(current_iter):\n    base_learning_rate = 0.1\n    lr = base_learning_rate  * np.power(.995, current_iter)\n    return lr if lr > 1e-3 else 1e-3\n\ndef learning_rate_005_decay_power_099(current_iter):\n    base_learning_rate = 0.05\n    lr = base_learning_rate  * np.power(.99, current_iter)\n    return lr if lr > 1e-3 else 1e-3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* # Set up HyperParameter search\nWe use random search, which is more flexible and more efficient than a grid search","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.stats import randint as sp_randint\nfrom scipy.stats import uniform as sp_uniform\nparam_test ={'num_leaves': sp_randint(6, 50), \n             'min_child_samples': sp_randint(100, 500), \n             'min_child_weight': [1e-5, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3, 1e4],\n             'subsample': sp_uniform(loc=0.2, scale=0.8), \n             'colsample_bytree': sp_uniform(loc=0.4, scale=0.6),\n             'reg_alpha': [0, 1e-1, 1, 2, 5, 7, 10, 50, 100],\n             'reg_lambda': [0, 1e-1, 1, 5, 10, 20, 50, 100]}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#This parameter defines the number of HP points to be tested\nn_HP_points_to_test = 100\n\nimport lightgbm as lgb\nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\n\n#n_estimators is set to a \"large value\". The actual number of trees build will depend on early stopping and 5000 define only the absolute maximum\nclf = lgb.LGBMClassifier(max_depth=-1, random_state=314, silent=True, metric='None', n_jobs=4, n_estimators=5000)\ngs = RandomizedSearchCV(\n    estimator=clf, param_distributions=param_test, \n    n_iter=n_HP_points_to_test,\n    scoring='roc_auc',\n    cv=3,\n    refit=True,\n    random_state=314,\n    verbose=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* # Use test subset for early stopping criterion\nThis allows us to avoid overtraining and we do not need to optimise the number of trees","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fit_params={\"early_stopping_rounds\":30, \n            \"eval_metric\" : 'auc', \n            \"eval_set\" : [(X_test,y_test)],\n            'eval_names': ['valid'],\n            #'callbacks': [lgb.reset_parameter(learning_rate=learning_rate_010_decay_power_099)],\n            'verbose': 100,\n            'categorical_feature': 'auto'}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Hyper parameter optimization using random search was run as shown below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"gs.fit(X_train, y_train, **fit_params)\nprint('Best score reached: {} with params: {} '.format(gs.best_score_, gs.best_params_))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The optimal parameters from the search was saved in the dictionary opt_parameters.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"opt_parameters = {'colsample_bytree': 0.9023523372315546, \n                  'min_child_samples': 237, \n                  'min_child_weight': 0.01, \n                  'num_leaves': 39, \n                  'reg_alpha': 10, \n                  'reg_lambda': 0.1, \n                  'subsample': 0.7187028219151861}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* # Some more tuning","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"clf_sw = lgb.LGBMClassifier(**clf.get_params())\n#set optimal parameters\nclf_sw.set_params(**opt_parameters)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gs_sample_weight = GridSearchCV(estimator=clf_sw, \n                                param_grid={'scale_pos_weight':[1,2,6,12]},\n                                scoring='roc_auc',\n                                cv=5,\n                                refit=True,\n                                verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gs_sample_weight.fit(X_train, y_train, **fit_params)\nprint('Best score reached: {} with params: {} '.format(gs_sample_weight.best_score_, gs_sample_weight.best_params_))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* # Build the final model\nI used the tuned parameter values but a smaller learning rate to allow smoother convergence to the minimum.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Configure from the HP optimisation\nclf_final = lgb.LGBMClassifier(**gs.best_estimator_.get_params())\n\n#Configure locally from hardcoded values\n#clf_final = lgb.LGBMClassifier(**clf.get_params())\n#set optimal parameters\nclf_final.set_params(**opt_parameters)\n\n#Train the final model with learning rate decay\nclf_final.fit(X_train, y_train, **fit_params, callbacks=[lgb.reset_parameter(learning_rate=learning_rate_010_decay_power_0995)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot feature importance","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"feat_imp = pd.Series(clf_final.feature_importances_, index=X.columns)\nfeat_imp.nlargest(20).plot(kind='barh', figsize=(8,10))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PREDICTIONS","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Next the predictions were made on the df_test dataset which contains the unknown images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import classification_report,confusion_matrix,roc_auc_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Prediction\ny_pred=clf_final.predict_proba(df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = y_pred[:,1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SUBMISSIONS ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({\n    \"image_name\": df1.image_name, \n    \"target\": y_pred\n})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}