\n \n## Transfer Learning\n\n#### Why transfer learning and how to use it?\n \nIf dataset is very large, it can easily take more than a few weeks to train. It's good to use transfer learning in this case. A few ways to use it:\n- **As fixed feature extractor**: froze the weights in the convoluntion layers. Can add/train fully connected layers.\n- **Fine-tuning the CNN**: you data is big, similar to the transfer learning dataset but not exactly the same. You can 'de-forze' and train some CNN layers weights too.\n- **Use only part of CNN layer**: add a few of your own CNN layers also.\n - A typical example for medical images.\n - **Medical images** are harder to collect and even harder to label since they need to labeled by doctors.\n - Trained by big ImageNet data, the first few layers can find some primitive features that can be used for medical images. We can use the earlier layers and add some later layers to train specifically for medical images.\n \n \n \n#### When to use a pre-trained network?\n- **โNew dataset is small (ex: 20,000 with 100 categories) and similar to the original dataset (ex: 2,000,000 with 1000 categories)**\n - Since the network is huge, you have froze the weight, which means you aren't training the model. Even not consider computation resource that you can actually re-train the model, the model is going to overfit since it's too big and more suitable for much bigger data.\n \n- **โ
New dataset is large and similar to the original dataset**\n - can gain training time by using either fixed or fresh weights\n- **โNew dataset is small but very different from the original dataset**\n - Overfit as in case in. Additionally, if very different, the transfer learning efficiency would be low.\n- **โ
New dataset is large and very different from the original dataset**\n - Can still use part of feature extractor of earlier layers. Still can save lots of training time.\n \nVarious [models](https://keras.io/applications/) pre-trained on ImageNet.","metadata":{}},{"cell_type":"markdown","source":"
\n \n## InceptionV3 \n#### Insights from InceptionV3 Architecture:\n- Lower Layers:\n - Basic convolutional Operations: such as Conv2d_1a_3x3, Conv2d_2a_3x3, and Conv2d_2b_3x3 \n - Fundamental local features extractraction: such as edges, textures, colors, and gradients\n - Simple Shapes and Contrast: In addition to low-level features, identify simple shapes and capturing contrast variations in the images.\n- Intermediate Layers:\n - Mixed Operations: blocks such as Mixed_3a, Mixed_4a, and Mixed_5a, combining different filter sizes and operations in parallel.\n - Repetitive Structures capturing: such as repetitive structures within the images, recognizing patterns that repeat across the dataset.\n - Color Variations and Relationships: further understand color variations and establish hierarchical relationships between different parts of objects in the images.\n - Object Boundaries: delineating object boundaries, enhancing the model's ability to recognize distinct objects and their spatial relationships.\n- Deeper Inception Blocks:\n - Mixed Operations: Blocks such as Mixed_5b, Mixed_6a, Mixed_7a\n - Complex Shapes and Structures: improved understanding of complex shapes and intricate structures present in the images.\n - Discrimination of Patterns: enhanced discrimination of complex patterns, distinguish fine-grained details and textures.\n\n#### My Adaptation Plan:\n- Feature Extraction:\n - Utilize earlier blocks for low-level features \n- Intermediate Layers Modification:\n - Introduce extra convolution layers for medical image nuances.\n - Post 'mixed2' layer in InceptionV3, the feature map is roughly (9 * 9). Add padding to prevent rapid size reduction (retain spatial information)\n- Enhancements with 1x1 Filters:\n - Incorporate 1x1 filters for improved learning.\n- Handle Overfitting:\n - Batch normalization\n - Dropout layers.\n- Image Augmentations for better generalization.","metadata":{}},{"cell_type":"markdown","source":"## Model 3 - transfer learning with Inception V3","metadata":{}},{"cell_type":"code","source":"# Build model 3\n# Note for model 1 to model 3, I used copied numpy data. I later found this method consumes too much memory.\n\n# Load the InceptionV3 model\nincept_model = InceptionV3(weights='imagenet', include_top=False, input_shape=(96, 96, 3))\n\n# Freeze the weights of the layers.\nfor layer in incept_model.layers:\n layer.trainable = False\n\n# get the last_layer of inception we want to use\nlast_layer = incept_model.get_layer('mixed2')\nprint('last layer output shape: ', last_layer.output_shape)\n\n# Create a new model using the functional API\nbase_model = Model(inputs=incept_model.input, outputs=last_layer.output)\n\n# Add layers to the model\nmymodel_3 = Sequential([\n base_model,\n \n Conv2D(64, (3, 3), padding='same', activation='relu'), # keep the width and height\n BatchNormalization(),\n Conv2D(64, (1, 1),activation='relu'),\n BatchNormalization(),\n Dropout(0.2),\n \n Conv2D(128, (3, 3), activation='relu'),\n BatchNormalization(), \n Conv2D(64, (1, 1), activation='relu'),\n BatchNormalization(),\n Dropout(0.2),\n\n Flatten(),\n Dense(128, activation='relu'),\n BatchNormalization(),\n Dropout(0.2),\n \n Dense(128, activation='relu'),\n BatchNormalization(),\n Dropout(0.2),\n \n Dense(1, activation='sigmoid')\n])\n\n# Print the model summary\nmymodel_3.build(input_shape=(None, 96, 96, 3))\nmymodel_3.summary()","metadata":{"execution":{"iopub.status.busy":"2024-02-05T22:29:22.011125Z","iopub.execute_input":"2024-02-05T22:29:22.011511Z","iopub.status.idle":"2024-02-05T22:29:29.937632Z","shell.execute_reply.started":"2024-02-05T22:29:22.011475Z","shell.execute_reply":"2024-02-05T22:29:29.936077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compile model 3\n\n#Define the learning rate schedule (exponential decay)\ninitial_learning_rate = 0.001\ndecay_steps = 1000\ndecay_rate = 0.9 \nlr_schedule = ExponentialDecay(initial_learning_rate, decay_steps, decay_rate, staircase=True)\n\n# Compile the model\nmymodel_3.compile(\n optimizer=Adam(learning_rate=lr_schedule),\n loss='binary_crossentropy',\n metrics=[AUC()]\n)","metadata":{"execution":{"iopub.status.busy":"2024-02-05T22:29:29.939580Z","iopub.execute_input":"2024-02-05T22:29:29.939985Z","iopub.status.idle":"2024-02-05T22:29:30.003582Z","shell.execute_reply.started":"2024-02-05T22:29:29.939948Z","shell.execute_reply":"2024-02-05T22:29:30.002428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit model 3\nBATCH_SIZE = 32\nEPOCHS = 10\n\n# data generator\ndatagen = ImageDataGenerator(\n rescale=1./255,\n rotation_range=20,\n width_shift_range=0.2,\n height_shift_range=0.2,\n vertical_flip=True,\n horizontal_flip=True,\n)\n\ndatagen_val = ImageDataGenerator(\n rescale=1./255,\n)\n\ntrain_generator = datagen.flow_from_directory(directory=train_dir,\n batch_size=BATCH_SIZE,\n class_mode='binary',\n target_size=(96, 96))\n\nval_generator = datagen_val.flow_from_directory(directory=val_dir, \n batch_size=BATCH_SIZE, \n class_mode='binary',\n target_size=(96,96))\n\nsteps_per_epoch = len(train_generator) \nvalidation_steps = len(val_generator)\n\n# Fit model 3\nhistory_3 = mymodel_3.fit(\n train_generator,\n validation_data=val_generator,\n epochs=EPOCHS,\n steps_per_epoch=steps_per_epoch,\n validation_steps=validation_steps,\n verbose=1\n)","metadata":{"execution":{"iopub.status.busy":"2024-02-05T22:29:30.005488Z","iopub.execute_input":"2024-02-05T22:29:30.005963Z","iopub.status.idle":"2024-02-06T02:57:39.976310Z","shell.execute_reply.started":"2024-02-05T22:29:30.005921Z","shell.execute_reply":"2024-02-06T02:57:39.967553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot metrics\nepochs = np.arange(1, EPOCHS+1)\nplt.figure(figsize=(8, 3))\n\n# AUC\nplt.subplot(1, 2, 1)\nplt.plot(epochs, history_3.history['auc'], label='Train')\nplt.plot(epochs, history_3.history['val_auc'], label='Validation')\nplt.xlabel('Epochs')\nplt.ylabel('AUC')\nplt.legend()\nplt.title('AUC Over Epochs')\n\n# Loss\nplt.subplot(1, 2, 2)\nplt.plot(epochs, history_3.history['loss'], label='Train')\nplt.plot(epochs, history_3.history['val_loss'], label='Validation')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\nplt.legend()\nplt.title('Loss Over Epochs')\n\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:08:07.118520Z","iopub.execute_input":"2024-02-06T03:08:07.119013Z","iopub.status.idle":"2024-02-06T03:08:07.678826Z","shell.execute_reply.started":"2024-02-06T03:08:07.118975Z","shell.execute_reply":"2024-02-06T03:08:07.677475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 3 Performance:\n- Achieved satisfactory training AUC of around 0.96 and a commendable validation AUC of 0.97 in the last epoch.\n- validation AUC plateaued after 8 epochs.\n\n### Next Steps:\n- Model evaluation, focusing on metrics such as precision, recall, F1 score, and confusion matrix to gain a holistic understanding of the model's performance. This approach ensures a balanced assessment of the model's predictive capabilities beyond AUC, providing insights into its precision, recall, and overall classification effectiveness.","metadata":{}},{"cell_type":"code","source":"# Predict validation set \nsamples_val = len(os.listdir(os.path.join(val_dir, '1'))) + len(os.listdir(os.path.join(val_dir, '0')))\nval_generator_for_pred = datagen_val.flow_from_directory(directory=val_dir, \n batch_size=1, \n class_mode='binary',\n target_size=(96,96),\n shuffle=False)\n\n\nval_predictions = mymodel_3.predict_generator(val_generator_for_pred,steps=samples_val, verbose=1)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:08:07.681489Z","iopub.execute_input":"2024-02-06T03:08:07.682002Z","iopub.status.idle":"2024-02-06T03:12:45.634359Z","shell.execute_reply.started":"2024-02-06T03:08:07.681953Z","shell.execute_reply":"2024-02-06T03:12:45.632480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Actual vs. predicted classes for validation set\n\n# actual\nval_labels = val_generator_for_pred.classes\n\n# convert probability to classes using 0.5 shreshold\nbinary_val_predictions = (val_predictions > 0.5).astype(int)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:12:45.636862Z","iopub.execute_input":"2024-02-06T03:12:45.637278Z","iopub.status.idle":"2024-02-06T03:12:45.645082Z","shell.execute_reply.started":"2024-02-06T03:12:45.637243Z","shell.execute_reply":"2024-02-06T03:12:45.643860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot ROC curve\n\n# Calculate AUC\nfpr, tpr, thresholds = roc_curve(val_labels, val_predictions)\nroc_auc = auc(fpr, tpr)\n\n# Plot \nplt.figure(figsize=(4, 4))\nplt.plot(fpr, tpr, color='orange', lw=2, label='ROC curve (AUC = {:.2f})'.format(roc_auc))\nplt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', label='Random Guessing')\nplt.xlabel('FPR')\nplt.ylabel('TPR')\nplt.title('ROC Curve')\nplt.legend(loc='lower right')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:12:45.648578Z","iopub.execute_input":"2024-02-06T03:12:45.649111Z","iopub.status.idle":"2024-02-06T03:12:45.922113Z","shell.execute_reply.started":"2024-02-06T03:12:45.649069Z","shell.execute_reply":"2024-02-06T03:12:45.920710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get recall, predicision, F1, accuracy scores for validation set\nrecall = recall_score(val_labels, binary_val_predictions)\nprecision = precision_score(val_labels, binary_val_predictions)\nF1 = f1_score(val_labels, binary_val_predictions)\nacc= accuracy_score(val_labels, binary_val_predictions)\n\nprint(\"Recall on validation set:\", recall)\nprint(\"Precision on validation set:\", precision)\nprint(\"F1 score n on validation set:\", F1)\nprint(\"Accuracy score n on validation set:\", acc)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:12:45.924145Z","iopub.execute_input":"2024-02-06T03:12:45.925080Z","iopub.status.idle":"2024-02-06T03:12:45.966357Z","shell.execute_reply.started":"2024-02-06T03:12:45.925024Z","shell.execute_reply":"2024-02-06T03:12:45.965028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot confusion matrix\ncm = confusion_matrix(val_labels, binary_val_predictions)\ndisp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=['Non-cancer', 'Cancer'])\ndisp.plot(cmap='Blues', values_format='d')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:12:45.968510Z","iopub.execute_input":"2024-02-06T03:12:45.969154Z","iopub.status.idle":"2024-02-06T03:12:46.289572Z","shell.execute_reply.started":"2024-02-06T03:12:45.969115Z","shell.execute_reply":"2024-02-06T03:12:46.288476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Final Model Reflection:\n\n- Recall Concerns:\n - Recall around 83% is suboptimal for cancer detection.\n - 1350+ cancer images misclassified as non-cancer signals a notable false-negative rate.\n- Importance of Recall:\n - High recall crucial in cancer detection.\n\n- Future Considerations:\n - Boosting recall through class weight adjustment or model fine-tuning. Emphasizes the need to strike a balance between overall model performance and sensitivity in critical scenarios like cancer detection.","metadata":{}},{"cell_type":"markdown","source":"#
\n
\n
Part 3: Test Results
\n","metadata":{}},{"cell_type":"code","source":"# Copy files to test_dir\n\n# Directories\ntest_source_path = os.path.join(base_dir,'test')\ntest_images = os.listdir(test_path)\ntest_dir=os.path.join(work_dir,'test')\ntest_dir_images=os.path.join(test_dir,'Pos and neg images') # put all images directly to work_dir won't work! Need to put images under a folder in test_dir \n\n# remove the entire directory and its contents if already exist\nif os.path.exists(test_dir):\n shutil.rmtree(test_dir)\n \n# Make directories \nos.makedirs(test_dir)\nos.makedirs(test_dir_images)\n\n# Copy images\nfor fn in test_images:\n source_fns = os.path.join(test_source_path, fn)\n test_fns = os.path.join(test_dir_images, fn)\n shutil.copyfile(source_fns,test_fns)\n\n","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:12:46.291227Z","iopub.execute_input":"2024-02-06T03:12:46.292305Z","iopub.status.idle":"2024-02-06T03:22:35.687067Z","shell.execute_reply.started":"2024-02-06T03:12:46.292249Z","shell.execute_reply":"2024-02-06T03:22:35.685629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check number of images under test dir\nprint(len(os.listdir(test_dir)))\nprint(len(os.listdir(test_source_path)))","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:22:35.688685Z","iopub.execute_input":"2024-02-06T03:22:35.689087Z","iopub.status.idle":"2024-02-06T03:22:35.739848Z","shell.execute_reply.started":"2024-02-06T03:22:35.689052Z","shell.execute_reply":"2024-02-06T03:22:35.738852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Test data generator\ndatagen_test = ImageDataGenerator(\n rescale=1./255)\n\ntest_generator = datagen_test.flow_from_directory(directory=test_dir, \n batch_size= BATCH_SIZE, \n class_mode=None,\n target_size=(96,96),\n shuffle=False)\n\ntest_predictions = mymodel_3.predict(test_generator, verbose=1)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:22:35.741227Z","iopub.execute_input":"2024-02-06T03:22:35.741804Z","iopub.status.idle":"2024-02-06T03:27:08.008894Z","shell.execute_reply.started":"2024-02-06T03:22:35.741746Z","shell.execute_reply":"2024-02-06T03:27:08.007439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create test df\ntest_df= pd.DataFrame({'id': test_generator.filenames,'label': test_predictions.flatten()})\ntest_df.head(1)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:27:08.013056Z","iopub.execute_input":"2024-02-06T03:27:08.013477Z","iopub.status.idle":"2024-02-06T03:27:08.065451Z","shell.execute_reply.started":"2024-02-06T03:27:08.013440Z","shell.execute_reply":"2024-02-06T03:27:08.064196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### The id for submission should not have '.tif'","metadata":{}},{"cell_type":"code","source":"# Source https://www.kaggle.com/code/vbookshelf/cnn-how-to-use-160-000-images-without-crashing\n# Get rid of '.tif' in 'id' column\ndef extract_id(x):\n \n # split into a list\n a = x.split('/')\n # split into a list\n b = a[1].split('.')\n extracted_id = b[0]\n \n return extracted_id\n\ntest_df['id'] = test_df['id'].apply(extract_id)\n\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:27:08.067371Z","iopub.execute_input":"2024-02-06T03:27:08.067753Z","iopub.status.idle":"2024-02-06T03:27:08.154079Z","shell.execute_reply.started":"2024-02-06T03:27:08.067718Z","shell.execute_reply":"2024-02-06T03:27:08.152813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submition\ntest_df.to_csv('CNN submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:27:08.155570Z","iopub.execute_input":"2024-02-06T03:27:08.156478Z","iopub.status.idle":"2024-02-06T03:27:08.434369Z","shell.execute_reply.started":"2024-02-06T03:27:08.156437Z","shell.execute_reply":"2024-02-06T03:27:08.433034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# remove files in test_dir to prevent error\nshutil.rmtree(test_dir)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T03:27:08.435744Z","iopub.execute_input":"2024-02-06T03:27:08.436275Z","iopub.status.idle":"2024-02-06T03:27:11.108602Z","shell.execute_reply.started":"2024-02-06T03:27:08.436225Z","shell.execute_reply":"2024-02-06T03:27:11.107215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Testing AUC after submission๐:\n- Score: 0.9334\n- Private score: 0.9361\n\n### Thoughts on testing AUC:\n - There is a notable gap exists between testing AUC (around 0.93) and validation AUC (around 0.97). Possible Explanations and improvement tips: \n - Overfitting Issues: The discrepancy may indicate lingering overfitting concerns. Increasing augmentation and dropout might help.\n - Validation Set Size: The validation set size at 10% might be limiting. Expanding it could provide a more representative evaluation of model generalization.\n - AUC as Metric: AUC, while a powerful metric, introduces complexities during model compilation, as it considers performance across all thresholds. In contrast, practical evaluation metrics like recall, precision, and the confusion matrix often rely on a fixed threshold, typically set at 0.5. Using another metric might help.\n\n \n","metadata":{}},{"cell_type":"markdown","source":"### Model Tuning Progress and Performance Comparison summary:\n\n- Model 1:\n\n - Utilized 3 blocks of 2 convolutions followed by 1 max pooling structure.\n - Achieved a training AUC of 0.90 and a validation AUC of 0.86.\n - Training for 5 epochs revealed potential for further improvement.\n\n- Model 2:\n\n - Retained Model 1 structure but extended training duration.\n - Training AUC remained unchanged, while validation AUC increased to 0.88.\n - Indicated that the initial model structure was too simplistic.\n- Model 3 (Transfer Learning):\n - Employed Transfer Learning, significantly increasing model complexity and addressed overfitting problem at the same time.\n - Demonstrated substantial improvement, elevating validation AUC from 0.88 to an impressive 0.97, and training AUC from 0.90 to 0.96.","metadata":{}},{"cell_type":"markdown","source":"#
\n
\n
Part 3: Project Conclusion
\n","metadata":{}},{"cell_type":"markdown","source":"### Conclusion and take aways\nIn the course of this project, I gained valuable insights into leveraging Convolutional Neural Networks (CNNs) for medical image cancer detection. Several noteworthy takeaways:\n- Data Handling Challenges: Managing large datasets can be challenging. I explored various methods for efficient data loading, such as converting images to int8 numpy arrays. However, this approach proved memory-intensive, impacting model training.\n- Transfer Learning Impact: The adoption of transfer learning significantly elevated the model's complexity. Also, transfer learning model is usually trained on a more diverse and larger dataset,this infusion of external knowledge could empower the model to discern intricate patterns in medical images. Moreover, transfer Learning could address overfitting problems at the same time.\n- Considerations for Model Evaluation: AUC, while a powerful metric, introduces complexities during model compilation, as it considers performance across all thresholds. In contrast, practical evaluation metrics like recall, precision, and the confusion matrix often rely on a fixed threshold, typically set at 0.5.\n\n### Future Improvements:\n\n- Exploring Alternative Metrics: Experimenting with alternative metrics, such as accuracy, could provide a different perspective on model performance. However, achieving a balanced dataset would be crucial in this context.\n\n- Optimizing Region of Interest: Focusing on the central 32x32 region, where cancer detection is most relevant, may involve cropping images to enhance model efficiency and effectiveness.\n- Addressing testing AUC discrepancy:\n - Experimenting with increased augmentation and dropout techniques to reduce overfitting.\n - Consider expanding the validation set size for a more representative evaluation of model generalization.\n\n","metadata":{}},{"cell_type":"markdown","source":"#
\n\n## References (really helpful notebooks๐๐ฝ)\nhttps://www.kaggle.com/code/jieshends2020/baseline-keras-cnn-roc-fast-10min-0-925-lb/edit
\nhttps://www.kaggle.com/code/akarshu121/cancer-detection-with-cnn-for-beginners
\nhttps://www.kaggle.com/code/qitvision/a-complete-ml-pipeline-fast-ai
\nhttps://www.kaggle.com/code/CVxTz/cnn-starter-nasnet-mobile-0-9709-lb
\nhttps://www.kaggle.com/code/suicaokhoailang/wip-densenet121-baseline-with-fastai
\nhttps://www.kaggle.com/code/vbookshelf/cnn-how-to-use-160-000-images-without-crashing
\n","metadata":{}}]}