{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":20270,"databundleVersionId":1222630,"sourceType":"competition"},{"sourceId":1193409,"sourceType":"datasetVersion","datasetId":679322},{"sourceId":8650850,"sourceType":"datasetVersion","datasetId":5181809},{"sourceId":8674069,"sourceType":"datasetVersion","datasetId":5198975},{"sourceId":8842548,"sourceType":"datasetVersion","datasetId":5321960},{"sourceId":8842632,"sourceType":"datasetVersion","datasetId":5322008},{"sourceId":8852976,"sourceType":"datasetVersion","datasetId":5328949},{"sourceId":8856787,"sourceType":"datasetVersion","datasetId":5331698}],"dockerImageVersionId":30733,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# Melanoma Classification","metadata":{}},{"cell_type":"markdown","source":"\nThe \"Skin Lesion Images for Melanoma Classification\" dataset on Kaggle includes several key variables and data types essential for developing machine learning models. Here is a detailed breakdown of these variables and their data types:\n\nImage Data:\n\nFile Name: Each image is stored with a unique file name, typically in .jpg or .png format. The file name often includes an identifier that corresponds to the patient or the specific lesion.\nImage Pixel Data: The images themselves are represented as pixel data, typically in RGB format, which can be processed as multi-dimensional arrays (e.g., 224x224x3 for 224x224 pixel images with 3 color channels).\nMetadata:\n\nPatient ID: A unique identifier for each patient, usually a string or integer.\nAge: The age of the patient, represented as an integer.\nSex: The gender of the patient, usually represented as a categorical variable (e.g., 'male', 'female').\nAnatom Site General Challenge: The anatomical site of the lesion, represented as a categorical variable (e.g., 'lower extremity', 'upper extremity', 'torso', 'head/neck').\nLesion Information:\n\nDiagnosis: The diagnosis for each lesion, categorized into multiple classes such as 'melanoma', 'nevus', 'seborrheic keratosis', etc. This is a categorical variable.\nDiagnosis Confirm Type: The method used to confirm the diagnosis, which might include biopsy, clinical, follow-up, or consensus.\nImage Metadata:\n\nImage Width: The width of the image in pixels, represented as an integer.\nImage Height: The height of the image in pixels, represented as an integer.\nResolution: The resolution of the image, usually given in DPI (dots per inch), which can be useful for scaling and preprocessing steps.","metadata":{}},{"cell_type":"markdown","source":"This dataset contains the training data for the ISIC 2019 challenge, note that it already includes data from previous years (2018 and 2017).\n\nThe dataset for ISIC 2019 contains 25,331 images available for the classification of dermoscopic images among nine different diagnostic categories:\n\nMelanoma\nMelanocytic nevus\nBasal cell carcinoma\nActinic keratosis\nBenign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like keratosis)\nDermatofibroma\nVascular lesion\nSquamous cell carcinoma\nNone of the above","metadata":{}},{"cell_type":"code","source":"pip install pandas scikit-learn tensorflow matplotlib","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\n\n# Limiting GPU memory growth\ngpus = tf.config.experimental.list_physical_devices('GPU')\nif gpus:\n    try:\n        for gpu in gpus:\n            tf.config.experimental.set_memory_growth(gpu, True)\n        logical_gpus = tf.config.experimental.list_logical_devices('GPU')\n        print(len(gpus), \"Physical GPUs,\", len(logical_gpus), \"Logical GPUs\")\n    except RuntimeError as e:\n        print(e)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:51:45.859011Z","iopub.execute_input":"2024-07-05T20:51:45.859334Z","iopub.status.idle":"2024-07-05T20:51:58.853359Z","shell.execute_reply.started":"2024-07-05T20:51:45.859288Z","shell.execute_reply":"2024-07-05T20:51:58.852351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"code","source":"import pandas as pd ","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:45:01.606889Z","iopub.execute_input":"2024-07-05T11:45:01.607592Z","iopub.status.idle":"2024-07-05T11:45:01.613302Z","shell.execute_reply.started":"2024-07-05T11:45:01.607559Z","shell.execute_reply":"2024-07-05T11:45:01.612019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metadata = pd.read_csv(r'/kaggle/input/isic-2019/ISIC_2019_Training_Metadata.csv')\nground_truth = pd.read_csv(r'/kaggle/input/isic-2019/ISIC_2019_Training_GroundTruth.csv')\n# Concatenate the two DataFrames based on the 'image' column\ndf = pd.merge(ground_truth, metadata, on='image').drop('lesion_id', axis=1)\ndf['target'] = df[['MEL', 'NV', 'BCC', 'AK', 'DF', 'VASC', 'SCC', 'UNK']].sum(axis=1) # 1 for Malignant || 0 for Benign\ndiagnosis_map = {0: 'benign', 1: 'malignant'}\ndf['diagnosis'] = df['target'].map(diagnosis_map)\ndata = df.loc[:, ['image', 'age_approx', 'anatom_site_general', 'sex', 'target', 'diagnosis']]\ndata['image'] = '/kaggle/input/isic-2019/ISIC_2019_Training_Input/ISIC_2019_Training_Input/' + data['image'] + '.jpg'\n# data['anatom_site_general'] = data['anatom_site_general'].replace(['lateral torso', 'anterior torso', 'posterior torso'], 'torso')\ndata","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:45:03.559057Z","iopub.execute_input":"2024-07-05T11:45:03.559473Z","iopub.status.idle":"2024-07-05T11:45:03.802180Z","shell.execute_reply.started":"2024-07-05T11:45:03.559442Z","shell.execute_reply":"2024-07-05T11:45:03.801018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"isic_data = pd.read_csv(r'/kaggle/input/siim-isic-melanoma-classification/train.csv').drop(['patient_id', 'diagnosis'],axis=1)\n\n# Selecting specific columns using .loc[]\nisic_Data = isic_data.loc[:, ['image_name', 'age_approx', 'anatom_site_general_challenge', 'sex', 'target', 'benign_malignant']]\nisic_data = isic_data.rename(columns={\n    'image_name' : 'image',\n    'anatom_site_general_challenge': 'anatom_site_general',\n    'benign_malignant': 'diagnosis',\n})\n\ncolumn_order = ['image', 'age_approx', 'anatom_site_general', 'sex', 'target', 'diagnosis']\n\nisic_data = isic_data[column_order]\n\n# Update the image column with the path to the image\nisic_data['image'] = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train/' + isic_data['image'] + '.jpg'\nisic_data","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:45:06.201531Z","iopub.execute_input":"2024-07-05T11:45:06.201950Z","iopub.status.idle":"2024-07-05T11:45:06.383414Z","shell.execute_reply.started":"2024-07-05T11:45:06.201900Z","shell.execute_reply":"2024-07-05T11:45:06.382265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Concatenating the two DataFrames\ncombined_data = pd.concat([isic_data, data], ignore_index=True).reset_index(drop=True)\n\ncombined_data","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:45:09.122508Z","iopub.execute_input":"2024-07-05T11:45:09.122953Z","iopub.status.idle":"2024-07-05T11:45:09.164073Z","shell.execute_reply.started":"2024-07-05T11:45:09.122898Z","shell.execute_reply":"2024-07-05T11:45:09.162962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Summarizing the combined data\nsummary = combined_data.describe(include='all')\n\n# Display the summary\nprint(summary)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:45:11.069782Z","iopub.execute_input":"2024-07-05T11:45:11.070856Z","iopub.status.idle":"2024-07-05T11:45:11.217289Z","shell.execute_reply.started":"2024-07-05T11:45:11.070809Z","shell.execute_reply":"2024-07-05T11:45:11.216159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset comprises 58,457 entries, each corresponding to an image with various associated attributes. Upon examining the summary statistics, we observe that all entries have non-null values for the 'image', 'sex', 'target', and 'diagnosis' columns, while the 'age_approx' and 'anatom_site_general' columns have slightly fewer entries (57,952 and 55,299 non-null values, respectively). The 'age_approx' column provides the approximate age of patients, with an average age of approximately 51.8 years and a standard deviation of 16.3 years, indicating a diverse age range from 0 to 90 years.\n\nThe 'anatom_site_general' column lists nine unique anatomical sites, with the most common site being the torso (16,845 occurrences). The 'sex' column is binary, consisting of 30,366 males and 28,092 females. The 'target' column, representing whether a lesion is malignant or benign, has a mean value of approximately 0.393, indicating that around 39.3% of the lesions are malignant.\n\nThe 'diagnosis' column, which further classifies the lesions into benign and malignant categories, shows that out of the total entries, 35,166 are benign, and 23,291 are malignant. This results in a class distribution where benign cases make up approximately 60.1% of the dataset, while malignant cases constitute the remaining 39.9%. This class distribution is crucial for training machine learning models, as it highlights the potential for class imbalance, which may need to be addressed to ensure accurate and unbiased model performance.\n\nOverall, the dataset is comprehensive, with well-distributed age and sex attributes, as well as a balanced representation of anatomical sites. The benign to malignant ratio is slightly imbalanced, which should be considered during model training to ensure robust predictive performance.","metadata":{}},{"cell_type":"code","source":"# Count of each diagnosis in the combined data\ndiagnosis_counts = combined_data['diagnosis'].value_counts().reset_index()\ndiagnosis_counts.columns = ['Diagnosis', 'Count']\n\n# Display the counts\nprint(diagnosis_counts)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:46:18.809174Z","iopub.execute_input":"2024-07-05T11:46:18.809582Z","iopub.status.idle":"2024-07-05T11:46:18.829676Z","shell.execute_reply.started":"2024-07-05T11:46:18.809552Z","shell.execute_reply":"2024-07-05T11:46:18.828190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\n# Ensure 'target' is treated as a categorical variable\ncombined_data['target'] = combined_data['target'].astype('category')\n\n# Define categorical and numerical variables\ncategorical_vars = ['anatom_site_general', 'sex', 'diagnosis', 'target']\nnumerical_vars = ['age_approx']\n\n# Summary of categorical variables (count and proportion)\ncategorical_summary = combined_data[categorical_vars].apply(lambda x: x.value_counts()).transpose()\ncategorical_proportions = combined_data[categorical_vars].apply(lambda x: x.value_counts(normalize=True)).transpose()\n\n# Summary of numerical variables (mean and standard deviation)\nnumerical_summary = combined_data[numerical_vars].describe().loc[['mean', 'std']]\n\n# Summary by diagnosis for categorical variables\ncategorical_by_diagnosis = combined_data.groupby('diagnosis')[categorical_vars].apply(lambda x: x.describe()).unstack()\n\n# Summary by diagnosis for numerical variables\nnumerical_by_diagnosis = combined_data.groupby('diagnosis')[numerical_vars].describe().unstack()\n\n# Display the summaries\nprint(\"Categorical Summary:\")\nprint(categorical_summary)\nprint(\"\\nCategorical Proportions:\")\nprint(categorical_proportions)\nprint(\"\\nNumerical Summary:\")\nprint(numerical_summary)\nprint(\"\\nCategorical Summary by Diagnosis:\")\nprint(categorical_by_diagnosis)\nprint(\"\\nNumerical Summary by Diagnosis:\")\nprint(numerical_by_diagnosis)\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:46:21.754345Z","iopub.execute_input":"2024-07-05T11:46:21.754759Z","iopub.status.idle":"2024-07-05T11:46:21.966442Z","shell.execute_reply.started":"2024-07-05T11:46:21.754728Z","shell.execute_reply":"2024-07-05T11:46:21.965358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For benign cases:\n- The torso is the most frequent site (47.22%), followed by the anterior torso (18.42%).\n- Other significant sites include the posterior torso (9.52%) and upper extremity (14.43%).\n\nFor malignant cases:\n- The torso also dominates (28.53%), followed by the anterior torso (9.83%).\n- The upper extremity is another notable site (33.11%).\n\n#### Sex Proportions by Diagnosis:\n\nFor benign cases:\n- Males make up 51.72% of the cases, while females account for 48.28%.\n\nFor malignant cases:\n- There is a higher proportion of males (56.01%) compared to females (43.99%).\n\n#### Age Group Proportions by Diagnosis:\n\nFor benign cases:\n- The most represented age group is 41-60 years (44.24%), followed by 21-40 years (32.98%).\n- The 61-80 years group accounts for 18.24%, and the least represented group is 0-20 years (3.15%).\n\nFor malignant cases:\n- The 41-60 years group is again the most represented (41.87%), followed by 61-80 years (25.11%).\n- The 21-40 years group accounts for 23.12%, and the 0-20 years group is the least represented (9.90%).","metadata":{}},{"cell_type":"markdown","source":"## check for nulls and types","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\ndef null_analysis(data):\n    df_null_info = pd.DataFrame(columns=['Feature Name', 'Number of Nulls'])\n    for data_label in data.columns.tolist():\n        num_nulls = data[data_label].isnull().sum()\n        df_null_info = pd.concat([df_null_info, pd.DataFrame({'Feature Name': [data_label], 'Number of Nulls': [num_nulls]})], ignore_index=True)\n    display(df_null_info)\n    print(data.info())\n\n    plt.figure(figsize=(10, 6))\n    plt.bar(df_null_info['Feature Name'], df_null_info['Number of Nulls'], color='midnightblue')\n    plt.xlabel('Feature Name')\n    plt.ylabel('Number of Nulls')\n    plt.title('Number of Nulls per Feature')\n\n    # Show the plot\n    plt.tight_layout()\n    plt.show()\n\nnull_analysis(combined_data)\ncombined_data = combined_data.dropna()\nnull_analysis(combined_data)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:46:53.569630Z","iopub.execute_input":"2024-07-05T11:46:53.570181Z","iopub.status.idle":"2024-07-05T11:46:54.491674Z","shell.execute_reply.started":"2024-07-05T11:46:53.570136Z","shell.execute_reply":"2024-07-05T11:46:54.490395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Some graphs","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:47:35.607538Z","iopub.execute_input":"2024-07-05T11:47:35.607941Z","iopub.status.idle":"2024-07-05T11:47:35.612912Z","shell.execute_reply.started":"2024-07-05T11:47:35.607891Z","shell.execute_reply":"2024-07-05T11:47:35.611640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_distribution(dataframe, target_column):\n    # Calculate value counts and sort them\n    value_counts = dataframe[target_column].value_counts().sort_index()\n\n    # Create a figure with two subplots\n    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\n\n    # Bar plot on the first subplot\n    bar_width = 0.35\n    ind = np.arange(len(value_counts))\n    rects = ax1.bar(ind, value_counts.values, width=bar_width, color='midnightblue')\n    ax1.set_xlabel(target_column, fontsize=12)\n    ax1.set_ylabel('Count', fontsize=12)\n    ax1.set_xticks(ind)\n    ax1.set_xticklabels(value_counts.index, rotation=45, ha='right', fontsize=10)\n\n    # Add values on top of bars\n    for index, value in enumerate(value_counts.values):\n        ax1.text(index, value + 0.1, str(value), ha='center', fontsize=10)\n\n    # Pie plot \n    explode = [0.1 for i in range(len(value_counts))]\n    patches, texts, autotexts = ax2.pie(value_counts, labels=value_counts.index, autopct='%1.1f%%', colors=[\"blue\",\"midnightblue\"], explode=explode, startangle=140)\n\n    # Adjust the distance of the labels from the center\n    for text in texts:\n        text.set_color('black')\n        text.set_fontsize(10)\n    for autotext in autotexts:\n        autotext.set_fontsize(10)\n        autotext.set_color('white')\n    # Main title for the figure\n    fig.suptitle(f'Distribution in {target_column} column', fontsize=18)\n    \n    # Adjust layout and display the figure\n    plt.tight_layout()\n    plt.show()\n    \nfor column in data.columns:\n    if column == 'image':\n        continue\n    plot_distribution(combined_data, column)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:47:41.414983Z","iopub.execute_input":"2024-07-05T11:47:41.415890Z","iopub.status.idle":"2024-07-05T11:47:44.102108Z","shell.execute_reply.started":"2024-07-05T11:47:41.415845Z","shell.execute_reply":"2024-07-05T11:47:44.100771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n\n# Load the data\nisic_data = pd.read_csv(r'/kaggle/input/siim-isic-melanoma-classification/train.csv').drop(['patient_id', 'diagnosis'],axis=1)\n\n# Selecting specific columns\nisic_data = isic_data.loc[:, ['image_name', 'age_approx', 'anatom_site_general_challenge', 'sex', 'target', 'benign_malignant']]\nisic_data = isic_data.rename(columns={\n    'image_name' : 'image',\n    'anatom_site_general_challenge': 'anatom_site_general',\n    'benign_malignant': 'diagnosis',\n})\n\ncolumn_order = ['image', 'age_approx', 'anatom_site_general', 'sex', 'target', 'diagnosis']\nisic_data = isic_data[column_order]\n\n# Update the image column with the path to the image\nisic_data['image'] = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train/' + isic_data['image'] + '.jpg'\n\n# Define the dataset class\nclass CustomDataset(Dataset):\n    def __init__(self, dataframe, transform=None):\n        self.dataframe = dataframe\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.dataframe)\n\n    def __getitem__(self, idx):\n        img_name = self.dataframe.iloc[idx, 0]\n        image = Image.open(img_name)\n\n        age = torch.tensor(self.dataframe.iloc[idx, 1], dtype=torch.float32)\n        anatom_site = self.dataframe.iloc[idx, 2]\n        sex = self.dataframe.iloc[idx, 3]\n        target = torch.tensor(self.dataframe.iloc[idx, 4], dtype=torch.float32)\n        label = self.dataframe.iloc[idx, 5]\n\n        if self.transform:\n            image = self.transform(image)\n\n        return image, age, anatom_site, sex, target, label\n\n# Define transforms for the training data\ntrain_transform = transforms.Compose([\n    transforms.Resize((224, 224)),\n    transforms.RandomHorizontalFlip(p=0.5),\n    transforms.RandomRotation(degrees=30),\n    transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1),\n    transforms.RandomApply([transforms.GaussianBlur(kernel_size=3)], p=0.5),\n    transforms.RandomApply([transforms.GaussianBlur(kernel_size=5)], p=0.5),\n    transforms.RandomApply([transforms.GaussianBlur(kernel_size=7)], p=0.5),\n    transforms.RandomApply([transforms.RandomAffine(degrees=30, translate=(0.1, 0.1), scale=(0.9, 1.1), shear=10)], p=0.5),\n    transforms.RandomApply([transforms.RandomPerspective(distortion_scale=0.5)], p=0.5),\n    transforms.ToTensor(),  \n])\n\n# Define transforms for the test data\ntest_transform = transforms.Compose([\n    transforms.Resize((224, 224)),\n    transforms.ToTensor(), \n])\n\n# Create the dataset and dataloader\ntrain_dataset = CustomDataset(isic_data, transform=train_transform)\ntrain_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, drop_last=True)\n\n# Visualize the images\n# Get one batch\nfor batch in train_loader:\n    images, ages, anatom_sites, sexs, targets, labels = batch\n    break\n\n# Create a figure and a grid of subplots\nfig, axs = plt.subplots(4, 8, figsize=(16, 8))\n\n# Flatten the axs array so that we can iterate over subplots easily\naxs = axs.flatten()\n\n# Plot each image\nfor i, ax in enumerate(axs):\n    ax.imshow(images[i].permute(1, 2, 0).cpu().numpy())  # Convert tensor to numpy array and permute dimensions\n    ax.axis('off')  # Turn off axis\n    ax.set_title(f'Label: {labels[i]}')  # Set title for each subplot\n\nplt.tight_layout()  # Adjust layout to make subplots fit\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:47:49.932023Z","iopub.execute_input":"2024-07-05T11:47:49.932527Z","iopub.status.idle":"2024-07-05T11:48:03.807596Z","shell.execute_reply.started":"2024-07-05T11:47:49.932491Z","shell.execute_reply":"2024-07-05T11:48:03.806106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filter the dataset to get an equal number of benign and malignant samples\nbenign_data2 = isic_data[isic_data['diagnosis'] == 'benign'].sample(16)\nmalignant_data2 = isic_data[isic_data['diagnosis'] == 'malignant'].sample(16)\nbalanced_data2 = pd.concat([benign_data2, malignant_data2]).reset_index(drop=True)\n\n# Create the dataset and dataloader\nbalanced_dataset2 = CustomDataset(balanced_data2, transform=train_transform)\nbalanced_loader2 = DataLoader(balanced_dataset2, batch_size=32, shuffle=True, drop_last=True)\n\n# Visualize the images\n# Get one batch\nfor batch in balanced_loader2:\n    images, ages, anatom_sites, sexs, targets, labels = batch\n    break\n\n# Create a figure and a grid of subplots\nfig, axs = plt.subplots(4, 8, figsize=(16, 8))\n\n# Flatten the axs array so that we can iterate over subplots easily\naxs = axs.flatten()\n\n# Plot each image\nfor i, ax in enumerate(axs):\n    ax.imshow(images[i].permute(1, 2, 0).cpu().numpy())  # Convert tensor to numpy array and permute dimensions\n    ax.axis('off')  # Turn off axis\n    ax.set_title(f'Label: {labels[i]}')  # Set title for each subplot\n\nplt.tight_layout()  # Adjust layout to make subplots fit\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:48:08.145404Z","iopub.execute_input":"2024-07-05T11:48:08.145807Z","iopub.status.idle":"2024-07-05T11:48:16.129649Z","shell.execute_reply.started":"2024-07-05T11:48:08.145773Z","shell.execute_reply":"2024-07-05T11:48:16.128246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Given that the dataset already contains a split for training (train.csv) and testing (test.csv), but not for validation, you can create a validation set by further splitting the training data.  ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# Load the training and testing metadata\ntrain_metadata = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')\ntest_metadata = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/test.csv')\n\n# Print columns to verify\nprint(train_metadata.columns)\nprint(test_metadata.columns)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:56:15.089404Z","iopub.execute_input":"2024-07-05T20:56:15.090123Z","iopub.status.idle":"2024-07-05T20:56:15.176684Z","shell.execute_reply.started":"2024-07-05T20:56:15.090088Z","shell.execute_reply":"2024-07-05T20:56:15.175822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The dataset already provides a train and a test dataset","metadata":{}},{"cell_type":"markdown","source":"test.csv does not contain the target column, which makes sense since the test set usually doesn't have labels (these are typically provided for model evaluation after submission in competitions). Therefore, you can only split the train.csv into training and validation sets, while using test.csv purely for making predictions without evaluation until the test labels are provided.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\n# Paths to the CSV files\ntrain_csv_path = '/kaggle/input/siim-isic-melanoma-classification/train.csv'\ntest_csv_path = '/kaggle/input/siim-isic-melanoma-classification/test.csv'\n\n# Load metadata\ntrain_metadata = pd.read_csv(train_csv_path)\ntest_metadata = pd.read_csv(test_csv_path)\n\n# Add file paths to the metadata\ntrain_metadata['file_path'] = train_metadata['image_name'].apply(lambda x: f'/kaggle/input/siim-isic-melanoma-classification/jpeg/train/{x}.jpg')\ntest_metadata['file_path'] = test_metadata['image_name'].apply(lambda x: f'/kaggle/input/siim-isic-melanoma-classification/jpeg/test/{x}.jpg')\n\n# Define the proportions for the split\nvalidation_proportion = 0.15\n\n# Split the training data to create a validation set\ntrain_data, validation_data = train_test_split(train_metadata, test_size=validation_proportion, stratify=train_metadata['target'], random_state=42)\n\n# Save the splits\ntrain_data.to_csv('/kaggle/working/train_split.csv', index=False)\nvalidation_data.to_csv('/kaggle/working/validation_split.csv', index=False)\ntest_metadata.to_csv('/kaggle/working/test_metadata.csv', index=False)\n\nprint(\"Datasets saved successfully:\")\nprint(\"Train dataset: /kaggle/working/train_split.csv\")\nprint(\"Validation dataset: /kaggle/working/validation_split.csv\")\nprint(\"Test dataset: /kaggle/working/test_metadata.csv\")\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:56:17.396690Z","iopub.execute_input":"2024-07-05T20:56:17.397051Z","iopub.status.idle":"2024-07-05T20:56:17.877381Z","shell.execute_reply.started":"2024-07-05T20:56:17.397022Z","shell.execute_reply":"2024-07-05T20:56:17.876431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test data","metadata":{}},{"cell_type":"code","source":"import os\n\n# Function to check if files exist\ndef check_file_existence(metadata, metadata_name):\n    print(f\"\\nChecking file paths in {metadata_name} dataset...\")\n    metadata['file_exists'] = metadata['file_path'].apply(os.path.exists)\n    non_existent_files = metadata[~metadata['file_exists']]\n    if not non_existent_files.empty:\n        print(f\"Some files are missing in the {metadata_name} dataset:\")\n        print(non_existent_files[['image_name', 'file_path']])\n    else:\n        print(f\"All files exist in the {metadata_name} dataset.\")\n\n# Check file existence in train, validation, and test datasets\ncheck_file_existence(train_data, 'train')\ncheck_file_existence(validation_data, 'validation')\ncheck_file_existence(test_metadata, 'test')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:56:21.805638Z","iopub.execute_input":"2024-07-05T20:56:21.806292Z","iopub.status.idle":"2024-07-05T20:56:39.972982Z","shell.execute_reply.started":"2024-07-05T20:56:21.806256Z","shell.execute_reply":"2024-07-05T20:56:39.971872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Count the outputs","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\n# Paths to the CSV files\ntrain_csv_path = '/kaggle/input/siim-isic-melanoma-classification/train.csv'\ntest_csv_path = '/kaggle/input/siim-isic-melanoma-classification/test.csv'\n\n# Load metadata\ntrain_metadata = pd.read_csv(train_csv_path)\ntest_metadata = pd.read_csv(test_csv_path)\n\n# Add file paths to the metadata\ntrain_metadata['file_path'] = train_metadata['image_name'].apply(lambda x: f'/kaggle/input/siim-isic-melanoma-classification/jpeg/train/{x}.jpg')\ntest_metadata['file_path'] = test_metadata['image_name'].apply(lambda x: f'/kaggle/input/siim-isic-melanoma-classification/jpeg/test/{x}.jpg')\n\n# Define the proportions for the split\nvalidation_proportion = 0.15\n\n# Split the training data to create a validation set\ntrain_data, validation_data = train_test_split(train_metadata, test_size=validation_proportion, stratify=train_metadata['target'], random_state=42)\n\n# Save the splits\ntrain_data.to_csv('/kaggle/working/train_split.csv', index=False)\nvalidation_data.to_csv('/kaggle/working/validation_split.csv', index=False)\ntest_metadata.to_csv('/kaggle/working/test_metadata.csv', index=False)\n\n# Print the counts of each target class in the training and validation sets\nprint(\"Train dataset class distribution:\")\nprint(train_data['target'].value_counts())\nprint(\"\\nValidation dataset class distribution:\")\nprint(validation_data['target'].value_counts())\n\nprint(\"Datasets saved successfully:\")\nprint(\"Train dataset: /kaggle/working/train_split.csv\")\nprint(\"Validation dataset: /kaggle/working/validation_split.csv\")\nprint(\"Test dataset: /kaggle/working/test_metadata.csv\")\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:57:10.135109Z","iopub.execute_input":"2024-07-05T20:57:10.135514Z","iopub.status.idle":"2024-07-05T20:57:10.644430Z","shell.execute_reply.started":"2024-07-05T20:57:10.135480Z","shell.execute_reply":"2024-07-05T20:57:10.643510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Re-do the numbers to achieve balance\n\n# while keeping the original proportion (38 vs 62%) try to keep these minimum numbers (3500 training; 1500 validation)","metadata":{}},{"cell_type":"markdown","source":"Balancing training data","metadata":{}},{"cell_type":"code","source":"from sklearn.utils import resample\n\n# Desired sample size\nmin_samples_train = 3500\nproportion_malignant = 0.38\n\n# Calculate the number of samples for each class based on the desired proportion\nn_samples_malignant_train = int(min_samples_train * proportion_malignant)\nn_samples_benign_train = min_samples_train - n_samples_malignant_train\n\n# Separate majority and minority classes\ndf_majority = train_data[train_data['target'] == 0]\ndf_minority = train_data[train_data['target'] == 1]\n\n# Downsample majority class and upsample minority class\ndf_majority_sampled = resample(df_majority, \n                               replace=False, \n                               n_samples=n_samples_benign_train, \n                               random_state=42)\n\ndf_minority_sampled = resample(df_minority, \n                               replace=True,  # upsample with replacement\n                               n_samples=n_samples_malignant_train, \n                               random_state=42)\n\n# Combine minority class with sampled majority class\ntrain_data_balanced = pd.concat([df_majority_sampled, df_minority_sampled])\n\nprint(\"Training set class distribution after balancing:\")\nprint(train_data_balanced['target'].value_counts())\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:57:28.033920Z","iopub.execute_input":"2024-07-05T20:57:28.034321Z","iopub.status.idle":"2024-07-05T20:57:28.056133Z","shell.execute_reply.started":"2024-07-05T20:57:28.034288Z","shell.execute_reply":"2024-07-05T20:57:28.055198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Desired sample size\nmin_samples_val = 1500\nproportion_malignant = 0.38\n\n# Calculate the number of samples for each class based on the desired proportion\nn_samples_malignant_val = int(min_samples_val * proportion_malignant)\nn_samples_benign_val = min_samples_val - n_samples_malignant_val\n\n# Separate majority and minority classes\ndf_majority_val = validation_data[validation_data['target'] == 0]\ndf_minority_val = validation_data[validation_data['target'] == 1]\n\n# Downsample majority class and upsample minority class\ndf_majority_sampled_val = resample(df_majority_val, \n                                   replace=False, \n                                   n_samples=n_samples_benign_val, \n                                   random_state=42)\n\ndf_minority_sampled_val = resample(df_minority_val, \n                                   replace=True,  # upsample with replacement\n                                   n_samples=n_samples_malignant_val, \n                                   random_state=42)\n\n# Combine minority class with sampled majority class\nvalidation_data_balanced = pd.concat([df_majority_sampled_val, df_minority_sampled_val])\n\nprint(\"Validation set class distribution after balancing:\")\nprint(validation_data_balanced['target'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:57:31.198662Z","iopub.execute_input":"2024-07-05T20:57:31.199067Z","iopub.status.idle":"2024-07-05T20:57:31.215993Z","shell.execute_reply.started":"2024-07-05T20:57:31.199032Z","shell.execute_reply":"2024-07-05T20:57:31.214940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the balanced splits\ntrain_data_balanced.to_csv('/kaggle/working/train_split_balanced.csv', index=False)\nvalidation_data_balanced.to_csv('/kaggle/working/validation_split_balanced.csv', index=False)\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:57:34.573718Z","iopub.execute_input":"2024-07-05T20:57:34.574096Z","iopub.status.idle":"2024-07-05T20:57:34.628136Z","shell.execute_reply.started":"2024-07-05T20:57:34.574066Z","shell.execute_reply":"2024-07-05T20:57:34.627241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model S (3500 training; 1500 validation)","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n\n# Load the balanced data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:51:38.429140Z","iopub.execute_input":"2024-07-05T11:51:38.430115Z","iopub.status.idle":"2024-07-05T11:51:40.452694Z","shell.execute_reply.started":"2024-07-05T11:51:38.430073Z","shell.execute_reply":"2024-07-05T11:51:40.451442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the model architecture\ndef create_model_s_lite():\n    model = Sequential([\n        Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)),\n        MaxPooling2D((2, 2)),\n        Conv2D(64, (3, 3), activation='relu'),\n        MaxPooling2D((2, 2)),\n        Flatten(),\n        Dense(128, activation='relu'),\n        Dropout(0.5),\n        Dense(1, activation='sigmoid')\n    ])\n    return model\n\n# Instantiate the model\nmodel_s_lite = create_model_s_lite()\n\n# Compile the model\nmodel_s_lite.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\n# Print the model summary\nmodel_s_lite.summary()\n\n# Calculate steps_per_epoch and validation_steps\nsteps_per_epoch = len(train_metadata) // 32\nvalidation_steps = len(validation_metadata) // 32\n\n# Train the model\nhistory_s_lite = model_s_lite.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,  # Adjust the number of epochs if needed\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model on the validation set\nval_loss_s_lite, val_accuracy_s_lite = model_s_lite.evaluate(validation_generator)\nprint(f'Validation loss (Model S Lite): {val_loss_s_lite}')\nprint(f'Validation accuracy (Model S Lite): {val_accuracy_s_lite}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T11:51:42.660841Z","iopub.execute_input":"2024-07-05T11:51:42.661253Z","iopub.status.idle":"2024-07-05T12:35:22.511684Z","shell.execute_reply.started":"2024-07-05T11:51:42.661224Z","shell.execute_reply":"2024-07-05T12:35:22.510050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the trained model\nmodel_filename = 'model_s_lite.h5'\nif os.path.exists(model_filename):\n    import time\n    timestamp = int(time.time())\n    new_model_filename = f'model_s_lite_{timestamp}.h5'\n    model_s_lite.save(new_model_filename)\n    print(f'Model saved as {new_model_filename}')\nelse:\n    model_s_lite.save(model_filename)\n    print(f'Model saved as {model_filename}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T12:37:07.791113Z","iopub.execute_input":"2024-07-05T12:37:07.791605Z","iopub.status.idle":"2024-07-05T12:37:08.409275Z","shell.execute_reply.started":"2024-07-05T12:37:07.791565Z","shell.execute_reply":"2024-07-05T12:37:08.408076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Confusion matrix","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import confusion_matrix, classification_report\n\n# Generate confusion matrix and classification report\n# Make predictions on the validation data\nvalidation_generator.reset()  # Reset the generator before predictions\npredictions = model_s_lite.predict(validation_generator)\npredicted_classes = np.where(predictions > 0.5, 1, 0).flatten()\n\n# Get true labels\ntrue_classes = validation_generator.classes\nclass_labels = list(validation_generator.class_indices.keys())\n\n# Calculate the confusion matrix\ncm = confusion_matrix(true_classes, predicted_classes)\nprint(\"Confusion Matrix:\")\nprint(cm)\n\n# Plot the confusion matrix\nplt.figure(figsize=(10, 7))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=class_labels, yticklabels=class_labels)\nplt.xlabel('Predicted')\nplt.ylabel('True')\nplt.title('Confusion Matrix for Model S Lite')\nplt.show()\n\n# Print the classification report\nreport = classification_report(true_classes, predicted_classes, target_names=class_labels)\nprint(\"Classification Report:\")\nprint(report)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T12:37:13.636460Z","iopub.execute_input":"2024-07-05T12:37:13.636876Z","iopub.status.idle":"2024-07-05T12:39:01.386977Z","shell.execute_reply.started":"2024-07-05T12:37:13.636846Z","shell.execute_reply":"2024-07-05T12:39:01.385763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model S WITHOUT data augmentation","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n\n# Load the balanced data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen_aug = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\n\n# Create ImageDataGenerator instances - No Data Augmentation\ntrain_datagen_no_aug = ImageDataGenerator(rescale=1./255)\n\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators with augmentation\ntrain_generator_aug = train_datagen_aug.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\n# Create data generators without augmentation\ntrain_generator_no_aug = train_datagen_no_aug.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\n# Define the model architecture\ndef create_model_s():\n    model = Sequential([\n        Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)),\n        MaxPooling2D((2, 2)),\n        Conv2D(64, (3, 3), activation='relu'),\n        MaxPooling2D((2, 2)),\n        Flatten(),\n        Dense(128, activation='relu'),\n        Dropout(0.5),\n        Dense(1, activation='sigmoid')\n    ])\n    return model\n\n# Instantiate the model\nmodel_s_without_augmentation = create_model_s()\n\n# Compile the model\nmodel_s_without_augmentation.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\n# Print the model summary\nmodel_s_without_augmentation.summary()\n\n# Calculate steps_per_epoch and validation_steps\nsteps_per_epoch = len(train_metadata) // 32\nvalidation_steps = len(validation_metadata) // 32\n\n# Train the model without data augmentation\nhistory_s_without_augmentation = model_s_without_augmentation.fit(\n    train_generator_no_aug,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model on the validation set\nval_loss_s_without_augmentation, val_accuracy_s_without_augmentation = model_s_without_augmentation.evaluate(validation_generator)\nprint(f'Validation loss (Model S without Augmentation): {val_loss_s_without_augmentation}')\nprint(f'Validation accuracy (Model S without Augmentation): {val_accuracy_s_without_augmentation}')\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T12:42:11.290799Z","iopub.execute_input":"2024-07-05T12:42:11.291757Z","iopub.status.idle":"2024-07-05T13:19:24.726882Z","shell.execute_reply.started":"2024-07-05T12:42:11.291716Z","shell.execute_reply":"2024-07-05T13:19:24.724501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the model\nmodel_save_path = '/kaggle/working/model_s_without_augmentation.h5'\nmodel_s_without_augmentation.save(model_save_path)\nprint(f'Model saved to {model_save_path}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T13:20:20.520129Z","iopub.execute_input":"2024-07-05T13:20:20.520648Z","iopub.status.idle":"2024-07-05T13:20:21.108251Z","shell.execute_reply.started":"2024-07-05T13:20:20.520617Z","shell.execute_reply":"2024-07-05T13:20:21.106879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate predictions\nY_pred = model_s_without_augmentation.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred = (Y_pred > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix\ncm = confusion_matrix(validation_generator.classes, y_pred)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix for Model S without Augmentation')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T13:20:25.973082Z","iopub.execute_input":"2024-07-05T13:20:25.974140Z","iopub.status.idle":"2024-07-05T13:22:11.846582Z","shell.execute_reply.started":"2024-07-05T13:20:25.974101Z","shell.execute_reply":"2024-07-05T13:22:11.845301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Different loss options for Model S (load architecture)","metadata":{}},{"cell_type":"markdown","source":"## binary cross-entropy loss","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n\n# Load the balanced data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\n# Define the model architecture\ndef create_model_s_lite():\n    model = Sequential([\n        Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)),\n        MaxPooling2D((2, 2)),\n        Conv2D(64, (3, 3), activation='relu'),\n        MaxPooling2D((2, 2)),\n        Flatten(),\n        Dense(128, activation='relu'),\n        Dropout(0.5),\n        Dense(1, activation='sigmoid')\n    ])\n    return model\n\n# Train and evaluate the model with binary cross-entropy loss\nmodel_s_bce = create_model_s_lite()\nmodel_s_bce.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\n# Calculate steps_per_epoch and validation_steps\nsteps_per_epoch = len(train_metadata) // 32\nvalidation_steps = len(validation_metadata) // 32\n\n# Train the model\nhistory_s_bce = model_s_bce.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,  # Adjust the number of epochs if needed\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model on the validation set\nval_loss_s_bce, val_accuracy_s_bce = model_s_bce.evaluate(validation_generator)\nprint(f'Validation loss (Model S Lite with Binary Cross-Entropy): {val_loss_s_bce}')\nprint(f'Validation accuracy (Model S Lite with Binary Cross-Entropy): {val_accuracy_s_bce}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T13:48:58.068844Z","iopub.execute_input":"2024-07-05T13:48:58.069813Z","iopub.status.idle":"2024-07-05T14:29:15.255002Z","shell.execute_reply.started":"2024-07-05T13:48:58.069775Z","shell.execute_reply":"2024-07-05T14:29:15.253308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the model\nmodel_s_bce.save('model_s_bce.h5')\nprint('Model saved as model_s_bce.h5')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T14:33:42.575075Z","iopub.execute_input":"2024-07-05T14:33:42.575510Z","iopub.status.idle":"2024-07-05T14:33:43.201211Z","shell.execute_reply.started":"2024-07-05T14:33:42.575479Z","shell.execute_reply":"2024-07-05T14:33:43.199995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate confusion matrix\ncm = confusion_matrix(validation_generator.classes, y_pred)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix for Model S Lite')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred, target_names=validation_generator.class_indices.keys()))\n\n# Ensure steps_per_epoch and validation_steps are defined\nsteps_per_epoch = train_generator.samples // 32\nvalidation_steps = validation_generator.samples // 32","metadata":{"execution":{"iopub.status.busy":"2024-07-05T14:32:26.620515Z","iopub.execute_input":"2024-07-05T14:32:26.621067Z","iopub.status.idle":"2024-07-05T14:32:26.984273Z","shell.execute_reply.started":"2024-07-05T14:32:26.621009Z","shell.execute_reply":"2024-07-05T14:32:26.982986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Mean Squared Error Loss","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n\n# Load the balanced data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\n# Define the model architecture\ndef create_model_s_lite():\n    model = Sequential([\n        Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)),\n        MaxPooling2D((2, 2)),\n        Conv2D(64, (3, 3), activation='relu'),\n        MaxPooling2D((2, 2)),\n        Flatten(),\n        Dense(128, activation='relu'),\n        Dropout(0.5),\n        Dense(1, activation='sigmoid')\n    ])\n    return model\n\n# Mean Squared Error Loss\nmodel_s_mse = create_model_s_lite()\nmodel_s_mse.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])\nmodel_s_mse.summary()\n\nhistory_s_mse = model_s_mse.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,  # Adjust the number of epochs if needed\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\nval_loss_s_mse, val_accuracy_s_mse = model_s_mse.evaluate(validation_generator)\nprint(f'Validation loss (Model S Lite with Mean Squared Error): {val_loss_s_mse}')\nprint(f'Validation accuracy (Model S Lite with Mean Squared Error): {val_accuracy_s_mse}')\n\nmodel_s_mse.save('model_s_mse.h5')\nprint('Model saved as model_s_mse.h5')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T14:43:15.890644Z","iopub.execute_input":"2024-07-05T14:43:15.891114Z","iopub.status.idle":"2024-07-05T15:23:27.025188Z","shell.execute_reply.started":"2024-07-05T14:43:15.891077Z","shell.execute_reply":"2024-07-05T15:23:27.023718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the model\nmodel_s_mse.save('model_s_mse.h5')\nprint('Model saved as model_s_mse.h5')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate predictions\nY_pred = model_s_mse.predict(validation_generator, validation_steps)\ny_pred = (Y_pred > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix\ncm = confusion_matrix(validation_generator.classes, y_pred)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix for Model S Lite with Mean Squared Error')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T15:26:36.503834Z","iopub.execute_input":"2024-07-05T15:26:36.505203Z","iopub.status.idle":"2024-07-05T15:28:26.996804Z","shell.execute_reply.started":"2024-07-05T15:26:36.505153Z","shell.execute_reply":"2024-07-05T15:28:26.995669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Different Optimizers for Model S","metadata":{}},{"cell_type":"markdown","source":"## We already had Adam","metadata":{}},{"cell_type":"markdown","source":"## SGD","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import load_model\nfrom tensorflow.keras.optimizers import SGD\nfrom sklearn.metrics import confusion_matrix, classification_report\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Load the balanced data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary',\n    shuffle=False  # Important: shuffle=False for validation data to match predictions\n)\n\n# Load the pre-trained model and call it model_s_lite\nmodel_s_path = '/kaggle/input/model-s-dl/model_s_lite.h5'\nmodel_s_lite = load_model(model_s_path)\nprint(f'Model loaded from {model_s_path}')\n\n# Ensure steps_per_epoch and validation_steps are defined\nsteps_per_epoch = train_generator.samples // 32\nvalidation_steps = validation_generator.samples // 32\n\n# Recompile and train Model S with SGD optimizer\nmodel_s_lite.compile(optimizer=SGD(), loss='binary_crossentropy', metrics=['accuracy'])\n\n# Train the model with SGD optimizer\nhistory_s_lite_sgd = model_s_lite.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model with SGD optimizer\nval_loss_s_lite_sgd, val_accuracy_s_lite_sgd = model_s_lite.evaluate(validation_generator)\nprint(f'Validation loss (Model S Lite with SGD Optimizer): {val_loss_s_lite_sgd}')\nprint(f'Validation accuracy (Model S Lite with SGD Optimizer): {val_accuracy_s_lite_sgd}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T15:28:57.556976Z","iopub.execute_input":"2024-07-05T15:28:57.557416Z","iopub.status.idle":"2024-07-05T16:09:42.969633Z","shell.execute_reply.started":"2024-07-05T15:28:57.557381Z","shell.execute_reply":"2024-07-05T16:09:42.968451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the model\nmodel_save_path_sgd = '/kaggle/working/model_s_sgd.h5'\nmodel_s_lite.save(model_save_path_sgd)\nprint(f'Model saved to {model_save_path_sgd}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T16:11:13.458345Z","iopub.execute_input":"2024-07-05T16:11:13.458748Z","iopub.status.idle":"2024-07-05T16:11:13.692767Z","shell.execute_reply.started":"2024-07-05T16:11:13.458719Z","shell.execute_reply":"2024-07-05T16:11:13.691444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate predictions with SGD optimizer\nY_pred_sgd = model_s_lite.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred_sgd = (Y_pred_sgd > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix for SGD optimizer\ncm_sgd = confusion_matrix(validation_generator.classes, y_pred_sgd)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm_sgd, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix for Model S Lite with SGD Optimizer')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred_sgd, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T16:11:26.527077Z","iopub.execute_input":"2024-07-05T16:11:26.527527Z","iopub.status.idle":"2024-07-05T16:13:12.838961Z","shell.execute_reply.started":"2024-07-05T16:11:26.527496Z","shell.execute_reply":"2024-07-05T16:13:12.837715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### RMSprop","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.optimizers import RMSprop\n\n# Recompile and train Model S with RMSprop optimizer\nmodel_s_lite.compile(optimizer=RMSprop(), loss='binary_crossentropy', metrics=['accuracy'])\n\n# Train the model with RMSprop optimizer\nhistory_s_lite_rmsprop = model_s_lite.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model with RMSprop optimizer\nval_loss_s_lite_rmsprop, val_accuracy_s_lite_rmsprop = model_s_lite.evaluate(validation_generator)\nprint(f'Validation loss (Model S Lite with RMSprop Optimizer): {val_loss_s_lite_rmsprop}')\nprint(f'Validation accuracy (Model S Lite with RMSprop Optimizer): {val_accuracy_s_lite_rmsprop}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T16:15:50.139053Z","iopub.execute_input":"2024-07-05T16:15:50.140266Z","iopub.status.idle":"2024-07-05T16:56:20.099054Z","shell.execute_reply.started":"2024-07-05T16:15:50.140195Z","shell.execute_reply":"2024-07-05T16:56:20.097800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the model\nmodel_save_path = '/kaggle/working/model_s_rmsprop.h5'\nmodel_s_lite.save(model_save_path)\nprint(f'Model saved to {model_save_path}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T16:56:43.759963Z","iopub.execute_input":"2024-07-05T16:56:43.760378Z","iopub.status.idle":"2024-07-05T16:56:44.062425Z","shell.execute_reply.started":"2024-07-05T16:56:43.760349Z","shell.execute_reply":"2024-07-05T16:56:44.061197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate predictions with RMSprop optimizer\nY_pred_rmsprop = model_s_lite.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred_rmsprop = (Y_pred_rmsprop > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix for RMSprop optimizer\ncm_rmsprop = confusion_matrix(validation_generator.classes, y_pred_rmsprop)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm_rmsprop, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix for Model S Lite with RMSprop Optimizer')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred_rmsprop, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T16:57:02.585447Z","iopub.execute_input":"2024-07-05T16:57:02.585867Z","iopub.status.idle":"2024-07-05T16:58:48.618972Z","shell.execute_reply.started":"2024-07-05T16:57:02.585835Z","shell.execute_reply":"2024-07-05T16:58:48.617773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Different regularization techniques - Model S","metadata":{}},{"cell_type":"markdown","source":"## L2 dropout","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.regularizers import l2\n\n# Load the balanced data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\n# Define Model S with L2 Regularization and Dropout\ndef create_model_s_l2_dropout():\n    model = Sequential([\n        Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3), kernel_regularizer=l2(0.01)),\n        MaxPooling2D((2, 2)),\n        Conv2D(64, (3, 3), activation='relu', kernel_regularizer=l2(0.01)),\n        MaxPooling2D((2, 2)),\n        Flatten(),\n        Dense(128, activation='relu', kernel_regularizer=l2(0.01)),\n        Dropout(0.5),\n        Dense(1, activation='sigmoid', kernel_regularizer=l2(0.01))\n    ])\n    return model\n\nmodel_s_l2_dropout = create_model_s_l2_dropout()\n\n# Compile and train Model S with L2 regularization and Dropout\nmodel_s_l2_dropout.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\nsteps_per_epoch = train_generator.samples // 32\nvalidation_steps = validation_generator.samples // 32\n\nhistory_s_l2_dropout = model_s_l2_dropout.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model on the validation set\nval_loss_s_l2_dropout, val_accuracy_s_l2_dropout = model_s_l2_dropout.evaluate(validation_generator)\nprint(f'Validation loss (Model S Lite with L2 Regularization and Dropout): {val_loss_s_l2_dropout}')\nprint(f'Validation accuracy (Model S Lite with L2 Regularization and Dropout): {val_accuracy_s_l2_dropout}')\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T16:59:01.708706Z","iopub.execute_input":"2024-07-05T16:59:01.709125Z","iopub.status.idle":"2024-07-05T17:42:29.849206Z","shell.execute_reply.started":"2024-07-05T16:59:01.709092Z","shell.execute_reply":"2024-07-05T17:42:29.847257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the model\nmodel_save_path_l2_dropout = '/kaggle/working/model_s_l2_dropout.h5'\nmodel_s_l2_dropout.save(model_save_path_l2_dropout)\nprint(f'Model saved to {model_save_path_l2_dropout}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T17:46:01.457698Z","iopub.execute_input":"2024-07-05T17:46:01.458680Z","iopub.status.idle":"2024-07-05T17:46:01.898704Z","shell.execute_reply.started":"2024-07-05T17:46:01.458634Z","shell.execute_reply":"2024-07-05T17:46:01.897432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate predictions\nY_pred_l2_dropout = model_s_l2_dropout.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred_l2_dropout = (Y_pred_l2_dropout > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix for L2 regularization and Dropout model\ncm_l2_dropout = confusion_matrix(validation_generator.classes, y_pred_l2_dropout)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm_l2_dropout, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix for Model S Lite with L2 Regularization and Dropout')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred_l2_dropout, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T17:46:17.703937Z","iopub.execute_input":"2024-07-05T17:46:17.704343Z","iopub.status.idle":"2024-07-05T17:48:01.321523Z","shell.execute_reply.started":"2024-07-05T17:46:17.704314Z","shell.execute_reply":"2024-07-05T17:48:01.320380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## L1 (lasso) + L2","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential, load_model\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.regularizers import l1, l2\nfrom sklearn.metrics import confusion_matrix, classification_report\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Load the balanced data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary',\n    shuffle=False  # Important: shuffle=False for validation data to match predictions\n)\n\n# Define Model S with L1 and L2 Regularization and Dropout\ndef create_model_s_l1_l2_dropout():\n    model = Sequential([\n        Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3), kernel_regularizer=l1(0.01)),\n        MaxPooling2D((2, 2)),\n        Conv2D(64, (3, 3), activation='relu', kernel_regularizer=l2(0.01)),\n        MaxPooling2D((2, 2)),\n        Flatten(),\n        Dense(128, activation='relu', kernel_regularizer=l2(0.01)),\n        Dropout(0.5),\n        Dense(1, activation='sigmoid', kernel_regularizer=l1(0.01))\n    ])\n    return model\n\nmodel_s_l1_l2_dropout = create_model_s_l1_l2_dropout()\n\n# Compile and train Model S with L1 and L2 regularization and Dropout\nmodel_s_l1_l2_dropout.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\nsteps_per_epoch = train_generator.samples // 32\nvalidation_steps = validation_generator.samples // 32\n\nhistory_s_l1_l2_dropout = model_s_l1_l2_dropout.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model on the validation set\nval_loss_s_l1_l2_dropout, val_accuracy_s_l1_l2_dropout = model_s_l1_l2_dropout.evaluate(validation_generator)\nprint(f'Validation loss (Model S Lite with L1 and L2 Regularization and Dropout): {val_loss_s_l1_l2_dropout}')\nprint(f'Validation accuracy (Model S Lite with L1 and L2 Regularization and Dropout): {val_accuracy_s_l1_l2_dropout}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T17:53:31.132313Z","iopub.execute_input":"2024-07-05T17:53:31.132765Z","iopub.status.idle":"2024-07-05T18:35:34.287491Z","shell.execute_reply.started":"2024-07-05T17:53:31.132735Z","shell.execute_reply":"2024-07-05T18:35:34.286269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the model\nmodel_save_path_l1_l2_dropout = '/kaggle/working/model_s_l1_l2_dropout.h5'\nmodel_s_l1_l2_dropout.save(model_save_path_l1_l2_dropout)\nprint(f'Model saved to {model_save_path_l1_l2_dropout}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T18:37:20.292421Z","iopub.execute_input":"2024-07-05T18:37:20.292878Z","iopub.status.idle":"2024-07-05T18:37:20.728183Z","shell.execute_reply.started":"2024-07-05T18:37:20.292834Z","shell.execute_reply":"2024-07-05T18:37:20.726980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate predictions\nY_pred_l1_l2_dropout = model_s_l1_l2_dropout.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred_l1_l2_dropout = (Y_pred_l1_l2_dropout > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix for L1 and L2 regularization and Dropout model\ncm_l1_l2_dropout = confusion_matrix(validation_generator.classes, y_pred_l1_l2_dropout)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm_l1_l2_dropout, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix for Model S Lite with L1 and L2 Regularization and Dropout')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred_l1_l2_dropout, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T18:37:30.066607Z","iopub.execute_input":"2024-07-05T18:37:30.067069Z","iopub.status.idle":"2024-07-05T18:39:15.760692Z","shell.execute_reply.started":"2024-07-05T18:37:30.067036Z","shell.execute_reply":"2024-07-05T18:39:15.759444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- - - -","metadata":{}},{"cell_type":"markdown","source":"# Model T transfer learning","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications import VGG16\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Flatten, Dense, Dropout\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nimport os\nimport pandas as pd\n\n# Load the data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nprint(f\"Training samples: {train_generator.samples}\")\nprint(f\"Validation samples: {validation_generator.samples}\")\nprint(f\"Training steps per epoch: {train_generator.samples // 32}\")\nprint(f\"Validation steps per epoch: {validation_generator.samples // 32}\")\n\n# Load the VGG16 model without the top classification layer\nbase_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\n\n# Freeze the base model\nbase_model.trainable = False\n\n# Add new classification layers on top of the base model\nx = base_model.output\nx = Flatten()(x)\nx = Dense(128, activation='relu')(x)\nx = Dropout(0.5)(x)\npredictions = Dense(1, activation='sigmoid')(x)\n\n# Create the new model\nmodel = Model(inputs=base_model.input, outputs=predictions)\n\n# Compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\n# Print the model summary\nmodel.summary()\n\n# Calculate steps_per_epoch and validation_steps\nsteps_per_epoch = train_generator.samples // 32\nvalidation_steps = validation_generator.samples // 32\n\n# Train the model\nhistory = model.fit(\n    train_generator,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T20:57:42.200812Z","iopub.execute_input":"2024-07-05T20:57:42.201720Z","iopub.status.idle":"2024-07-05T21:24:57.240523Z","shell.execute_reply.started":"2024-07-05T20:57:42.201684Z","shell.execute_reply":"2024-07-05T21:24:57.239498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the trained model\nmodel_filename = 'model_T.h5'\nif os.path.exists(model_filename):\n    import time\n    timestamp = int(time.time())\n    new_model_filename = f'model_T_{timestamp}.h5'\n    model.save(new_model_filename)\n    print(f'Model saved as {new_model_filename}')\nelse:\n    model.save(model_filename)\n    print(f'Model saved as {model_filename}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T21:32:12.351763Z","iopub.execute_input":"2024-07-05T21:32:12.352548Z","iopub.status.idle":"2024-07-05T21:32:12.531729Z","shell.execute_reply.started":"2024-07-05T21:32:12.352515Z","shell.execute_reply":"2024-07-05T21:32:12.530758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import confusion_matrix, classification_report\n\n# Make predictions on the validation data\nvalidation_generator.reset()  # Reset the generator before predictions\npredictions = model.predict(validation_generator)\npredicted_classes = np.where(predictions > 0.5, 1, 0).flatten()\n\n# Get true labels\ntrue_classes = validation_generator.classes\nclass_labels = list(validation_generator.class_indices.keys())\n\n# Calculate the confusion matrix\ncm = confusion_matrix(true_classes, predicted_classes)\nprint(\"Confusion Matrix:\")\nprint(cm)\n\n# Plot the confusion matrix\nplt.figure(figsize=(10, 7))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=class_labels, yticklabels=class_labels)\nplt.xlabel('Predicted')\nplt.ylabel('True')\nplt.title('Confusion Matrix for Model T')\nplt.show()\n\n# Print the classification report\nreport = classification_report(true_classes, predicted_classes, target_names=class_labels)\nprint(\"Classification Report:\")\nprint(report)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T21:32:18.410293Z","iopub.execute_input":"2024-07-05T21:32:18.410908Z","iopub.status.idle":"2024-07-05T21:33:57.491076Z","shell.execute_reply.started":"2024-07-05T21:32:18.410878Z","shell.execute_reply":"2024-07-05T21:33:57.490127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model T without data Augmentation","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications import VGG16\nfrom tensorflow.keras.models import Model, Sequential\nfrom tensorflow.keras.layers import Flatten, Dense, Dropout\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.optimizers import SGD\nfrom tensorflow.keras.losses import CategoricalCrossentropy\nimport os\nimport pandas as pd\n\n# Load the data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Without Data Augmentation\ntrain_datagen_no_aug = ImageDataGenerator(rescale=1./255)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators without data augmentation\ntrain_generator_no_aug = train_datagen_no_aug.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nprint(f\"Training samples: {train_generator_no_aug.samples}\")\nprint(f\"Validation samples: {validation_generator.samples}\")\nprint(f\"Training steps per epoch: {train_generator_no_aug.samples // 32}\")\nprint(f\"Validation steps per epoch: {validation_generator.samples // 32}\")\n\n# Load the VGG16 model without the top classification layer\nbase_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\n\n# Freeze the base model\nbase_model.trainable = False\n\n# Add new classification layers on top of the base model\ndef create_model_t():\n    model = Sequential([\n        base_model,\n        Flatten(),\n        Dense(128, activation='relu'),\n        Dropout(0.5),  # Regularization\n        Dense(1, activation='sigmoid')\n    ])\n    return model\n\n# Compile and train Model T without data augmentation\nmodel_t_no_aug = create_model_t()\nmodel_t_no_aug.compile(optimizer=SGD(), loss='binary_crossentropy', metrics=['accuracy'])\n\nsteps_per_epoch = train_generator_no_aug.samples // 32\nvalidation_steps = validation_generator.samples // 32\n\nhistory_t_no_aug = model_t_no_aug.fit(\n    train_generator_no_aug,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model\nresults_t_no_aug = model_t_no_aug.evaluate(validation_generator)\n\nprint(\"Model T without Augmentation Test Loss, Test Accuracy:\", results_t_no_aug)\n\n# Save the model\nmodel_t_no_aug.save('model_t_no_aug.h5')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T21:38:30.315708Z","iopub.execute_input":"2024-07-05T21:38:30.316404Z","iopub.status.idle":"2024-07-05T22:02:01.525004Z","shell.execute_reply.started":"2024-07-05T21:38:30.316359Z","shell.execute_reply":"2024-07-05T22:02:01.524055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport time \n\n# Save the trained model\nmodel_filename = '/kaggle/working/model_t_no_aug.h5'\nif os.path.exists(model_filename):\n    timestamp = int(time.time())\n    new_model_filename = f'/kaggle/working/model_t_no_aug_{timestamp}.h5'\n    model_t_no_aug.save(new_model_filename)\n    print(f'Model saved as {new_model_filename}')\nelse:\n    model_t_no_aug.save(model_filename)\n    print(f'Model saved as {model_filename}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T22:02:06.079992Z","iopub.execute_input":"2024-07-05T22:02:06.080472Z","iopub.status.idle":"2024-07-05T22:02:06.207648Z","shell.execute_reply.started":"2024-07-05T22:02:06.080438Z","shell.execute_reply":"2024-07-05T22:02:06.206690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Generate predictions\nY_pred = model_t_no_aug.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred = (Y_pred > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix\ncm = confusion_matrix(validation_generator.classes, y_pred)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T22:02:16.945330Z","iopub.execute_input":"2024-07-05T22:02:16.945672Z","iopub.status.idle":"2024-07-05T22:03:42.711501Z","shell.execute_reply.started":"2024-07-05T22:02:16.945646Z","shell.execute_reply":"2024-07-05T22:03:42.710462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Implementation for Model T with Fine Tuning and Data Augmentation","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications import VGG16\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Flatten, Dense, Dropout\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.optimizers import SGD, Adam\nfrom tensorflow.keras.losses import CategoricalCrossentropy\nimport pandas as pd\nimport os\nimport time\nfrom sklearn.metrics import confusion_matrix, classification_report\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Load the data\ntrain_metadata = pd.read_csv('/kaggle/working/train_split_balanced.csv')\nvalidation_metadata = pd.read_csv('/kaggle/working/validation_split_balanced.csv')\n\n# Convert 'target' column to strings\ntrain_metadata['target'] = train_metadata['target'].astype(str)\nvalidation_metadata['target'] = validation_metadata['target'].astype(str)\n\n# Ensure the image_name column contains only the image filename\ntrain_metadata['image_name'] = train_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\nvalidation_metadata['image_name'] = validation_metadata['image_name'].apply(lambda x: f\"{x}.jpg\")\n\n# Directory paths\nbase_dir_train = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\n# Create ImageDataGenerator instances - Data Augmentation\ntrain_datagen_aug = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    shear_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\nvalidation_datagen = ImageDataGenerator(rescale=1./255)\n\n# Create data generators with data augmentation\ntrain_generator_aug = train_datagen_aug.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nvalidation_generator = validation_datagen.flow_from_dataframe(\n    dataframe=validation_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\nprint(f\"Training samples: {train_generator_aug.samples}\")\nprint(f\"Validation samples: {validation_generator.samples}\")\nprint(f\"Training steps per epoch: {train_generator_aug.samples // 32}\")\nprint(f\"Validation steps per epoch: {validation_generator.samples // 32}\")\n\n# Load the VGG16 model without the top classification layer\nbase_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\n\n# Unfreeze some layers for fine-tuning\nfor layer in base_model.layers[-4:]:\n    layer.trainable = True\n\n# Add new classification layers on top of the base model\ndef create_model_t_fine_tune():\n    model = Sequential([\n        base_model,\n        Flatten(),\n        Dense(128, activation='relu'),\n        Dropout(0.5),  # Regularization\n        Dense(1, activation='sigmoid')\n    ])\n    return model\n\n# Compile and train Model T with fine-tuning and data augmentation\nmodel_t_fine_tune_aug = create_model_t_fine_tune()\nmodel_t_fine_tune_aug.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy'])\n\nsteps_per_epoch = train_generator_aug.samples // 32\nvalidation_steps = validation_generator.samples // 32\n\nhistory_t_fine_tune_aug = model_t_fine_tune_aug.fit(\n    train_generator_aug,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)\n\n# Evaluate the model\nresults_t_fine_tune_aug = model_t_fine_tune_aug.evaluate(validation_generator)\n\nprint(\"Model T with Fine-Tuning and Augmentation Test Loss, Test Accuracy:\", results_t_fine_tune_aug)\n","metadata":{"execution":{"iopub.status.busy":"2024-07-05T22:04:06.475939Z","iopub.execute_input":"2024-07-05T22:04:06.476314Z","iopub.status.idle":"2024-07-05T22:33:51.219000Z","shell.execute_reply.started":"2024-07-05T22:04:06.476279Z","shell.execute_reply":"2024-07-05T22:33:51.217890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the trained model\nmodel_filename = '/kaggle/working/model_t_fine_tune_aug.h5'\nif os.path.exists(model_filename):\n    timestamp = int(time.time())\n    new_model_filename = f'/kaggle/working/model_t_fine_tune_aug_{timestamp}.h5'\n    model_t_fine_tune_aug.save(new_model_filename)\n    print(f'Model saved as {new_model_filename}')\nelse:\n    model_t_fine_tune_aug.save(model_filename)\n    print(f'Model saved as {model_filename}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T22:34:44.100847Z","iopub.execute_input":"2024-07-05T22:34:44.101699Z","iopub.status.idle":"2024-07-05T22:34:44.483588Z","shell.execute_reply.started":"2024-07-05T22:34:44.101662Z","shell.execute_reply":"2024-07-05T22:34:44.482457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate and save confusion matrices, accuracy, precision, recall, and F1 score\n# Generate predictions\nY_pred = model_t_fine_tune_aug.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred = (Y_pred > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix\ncm = confusion_matrix(validation_generator.classes, y_pred)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T22:34:55.778070Z","iopub.execute_input":"2024-07-05T22:34:55.779024Z","iopub.status.idle":"2024-07-05T22:36:38.134979Z","shell.execute_reply.started":"2024-07-05T22:34:55.778985Z","shell.execute_reply":"2024-07-05T22:36:38.133723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Implementation for Model T Fine Tuning and WITHOUT Data Augmentation","metadata":{}},{"cell_type":"code","source":"# Create data generators without data augmentation\ntrain_generator_no_aug = train_datagen_no_aug.flow_from_dataframe(\n    dataframe=train_metadata,\n    directory=base_dir_train,\n    x_col='image_name',\n    y_col='target',\n    target_size=(224, 224),\n    batch_size=32,\n    class_mode='binary'\n)\n\n# Compile and train Model T with fine-tuning without data augmentation\nmodel_t_fine_tune_no_aug = create_model_t_fine_tune()\nmodel_t_fine_tune_no_aug.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy'])\n\nsteps_per_epoch = train_generator_no_aug.samples // 32\nvalidation_steps = validation_generator.samples // 32\n\nhistory_t_fine_tune_no_aug = model_t_fine_tune_no_aug.fit(\n    train_generator_no_aug,\n    validation_data=validation_generator,\n    epochs=10,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps\n)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T22:36:46.963059Z","iopub.execute_input":"2024-07-05T22:36:46.964052Z","iopub.status.idle":"2024-07-05T23:01:12.227654Z","shell.execute_reply.started":"2024-07-05T22:36:46.964012Z","shell.execute_reply":"2024-07-05T23:01:12.226745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate the model\nresults_t_fine_tune_no_aug = model_t_fine_tune_no_aug.evaluate(validation_generator)\n\nprint(\"Model T with Fine-Tuning without Augmentation Test Loss, Test Accuracy:\", results_t_fine_tune_no_aug)","metadata":{"execution":{"iopub.status.busy":"2024-07-05T23:14:02.445374Z","iopub.execute_input":"2024-07-05T23:14:02.445819Z","iopub.status.idle":"2024-07-05T23:15:34.604494Z","shell.execute_reply.started":"2024-07-05T23:14:02.445783Z","shell.execute_reply":"2024-07-05T23:15:34.603368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the trained model\nmodel_filename = '/kaggle/working/model_t_fine_tune_no_aug.h5'\nif os.path.exists(model_filename):\n    timestamp = int(time.time())\n    new_model_filename = f'/kaggle/working/model_t_fine_tune_no_aug_{timestamp}.h5'\n    model_t_fine_tune_no_aug.save(new_model_filename)\n    print(f'Model saved as {new_model_filename}')\nelse:\n    model_t_fine_tune_no_aug.save(model_filename)\n    print(f'Model saved as {model_filename}')","metadata":{"execution":{"iopub.status.busy":"2024-07-05T23:16:12.935724Z","iopub.execute_input":"2024-07-05T23:16:12.936542Z","iopub.status.idle":"2024-07-05T23:16:13.300552Z","shell.execute_reply.started":"2024-07-05T23:16:12.936507Z","shell.execute_reply":"2024-07-05T23:16:13.299409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate and save confusion matrices, accuracy, precision, recall, and F1 score\n# Generate predictions\nY_pred = model_t_fine_tune_no_aug.predict(validation_generator, validation_generator.samples // 32 + 1)\ny_pred = (Y_pred > 0.5).astype(int)  # Convert probabilities to binary predictions\n\n# Generate confusion matrix\ncm = confusion_matrix(validation_generator.classes, y_pred)\n\n# Plot confusion matrix\nplt.figure(figsize=(10, 8))\nsns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=validation_generator.class_indices.keys(), yticklabels=validation_generator.class_indices.keys())\nplt.ylabel('Actual')\nplt.xlabel('Predicted')\nplt.title('Confusion Matrix')\nplt.show()\n\n# Print classification report\nprint(classification_report(validation_generator.classes, y_pred, target_names=validation_generator.class_indices.keys()))","metadata":{"execution":{"iopub.status.busy":"2024-07-05T23:16:22.866086Z","iopub.execute_input":"2024-07-05T23:16:22.867024Z","iopub.status.idle":"2024-07-05T23:17:53.507459Z","shell.execute_reply.started":"2024-07-05T23:16:22.866990Z","shell.execute_reply":"2024-07-05T23:17:53.506284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- - - ","metadata":{}}]}