{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":44527,"databundleVersionId":5167437,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Final Assignment: aim to be in the top 25% of the leaderboard of a competition.\nBy Rubén Lucendo\n\nThis notebook documents the full process I did to solve the final assignament in the course \"HBKUx HBKU011 Applications of AI in Healthcare\".\n\nBelow you will find the requeriments of this assignament:\n\n- Metrics Description: Provide a detailed description of the metrics used to evaluate your model. Appropriate metrics may include accuracy, precision, recall, F1 score, and the Dice coefficient, as relevant to your selected task.\n- Kaggle Rank Screenshot: Include a screenshot showing your rank on the Kaggle leaderboard.\n- Visualization: Use a visualization technique such as Grad-CAM to illustrate what your model has learned. This should demonstrate the interpretive process of your model.","metadata":{}},{"cell_type":"markdown","source":"# Data selection and setup of the training process\n\nI have decided to participate in the competition \"SPR X-Ray Gender Prediction Challenge\", because I found it in a range of complexity very challenging but achivable, as the other two competitions are, in my opinion, \"too simple\" and the other one, too complex for my knoledge and the time I have to resolve it.\n\nIn this case, to aim to be in the top 25% of the competition, I need a AUC score > 0.98475.\n\nThis dataset is different than the others I used in this course. This time, the tags are listed in a separate file (.csv). Therefore, I will list boths (images IDs and tags) in a list togheter and check that they listed in the right order.","metadata":{}},{"cell_type":"markdown","source":"# First step - Setup\n\nImporting the necessary libraries and defining paths.","metadata":{}},{"cell_type":"code","source":"from pathlib import Path\nimport pandas as pd\nfrom fastai.vision.all import *\nfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score\n\n# Rutas\ntrain_dir = Path('/kaggle/input/spr-x-ray-gender/kaggle/kaggle/train/')  # 000000.png, 000001.png, ...\nlabels_csv = Path('/kaggle/input/spr-x-ray-gender/train_gender.csv')     # columnas: imageId, gender\n\n# Cargar CSV y normalizar columnas\ndf = pd.read_csv(labels_csv)\ndf.columns = [c.strip() for c in df.columns]\ndf = df.rename(columns={'imageId': 'Image', 'gender': 'Gender'})\n\n# Detectar automáticamente el ancho de los nombres\nwidth = max(len(p.stem) for p in train_dir.glob('*.png'))\n\n# Normalizar los IDs: string, sin extensión, zero-padding al ancho detectado\ndf['Image'] = (df['Image'].astype(str)\n                         .str.strip()\n                         .str.replace(r'\\.(png|jpg|jpeg)$', '', regex=True)\n                         .str.zfill(width))\n\n# Diccionario id -> gender (0/1)\ndf['Gender'] = df['Gender'].astype(int)\nid2gender = df.set_index('Image')['Gender'].to_dict()\n\n# Mapeo opcional de 0/1 a clases de texto\nint2class = {0: 'mujer', 1: 'hombre'}\n\ndef label_func(o: Path):\n    g = id2gender[o.stem]           # lanzará KeyError si falta en el CSV\n    return int2class[int(g)]\n\n# DataBlock\nsz = 224\ndb = DataBlock(\n    blocks=(ImageBlock, CategoryBlock(vocab=list(int2class.values()))),\n    get_items=get_image_files,\n    splitter=RandomSplitter(valid_pct=0.2, seed=42),\n    get_y=label_func,\n    item_tfms=Resize(sz*2, method='pad'),\n    batch_tfms=[*aug_transforms(size=sz), Normalize.from_stats(*imagenet_stats)]\n)\n\ndls = db.dataloaders(train_dir, bs = 64)\ndls.show_batch(max_n=9)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:34:48.942793Z","iopub.execute_input":"2025-08-13T21:34:48.943053Z","iopub.status.idle":"2025-08-13T21:35:14.105430Z","shell.execute_reply.started":"2025-08-13T21:34:48.943029Z","shell.execute_reply":"2025-08-13T21:35:14.104640Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Checking the structure of the items list before the training, to be sure that it was correctly sorted.","metadata":{}},{"cell_type":"code","source":"# DEBUG: taking a look to the data before we load the trainer\nfrom pathlib import Path\n\n# Items de cada split\ntrain_items = list(dls.train_ds.items)\nvalid_items = list(dls.valid_ds.items)\n\n# Construye DataFrames con imageId y etiqueta (int y texto)\ntrain_df_dbg = pd.DataFrame({\n    'split': 'train',\n    'relpath': [str(Path(o).relative_to(train_dir)) for o in train_items],\n    'imageId': [Path(o).stem for o in train_items],\n})\ntrain_df_dbg['label_int'] = train_df_dbg['imageId'].map(id2gender)\ntrain_df_dbg['label'] = train_df_dbg['label_int'].map(int2class)\n\nvalid_df_dbg = pd.DataFrame({\n    'split': 'valid',\n    'relpath': [str(Path(o).relative_to(train_dir)) for o in valid_items],\n    'imageId': [Path(o).stem for o in valid_items],\n})\nvalid_df_dbg['label_int'] = valid_df_dbg['imageId'].map(id2gender)\nvalid_df_dbg['label'] = valid_df_dbg['label_int'].map(int2class)\n\ndbg = pd.concat([train_df_dbg, valid_df_dbg], ignore_index=True)\n\n# Muestra una muestra de la tabla y recuentos\nprint(\"\\n Preview de la tabla que entra al loader \")\nprint(dbg.head(20))  # cambia 20 por lo que quieras\n\nprint(\"\\n Recuento por split y clase \")\nprint(dbg.groupby(['split','label']).size())\n\n# Comprueba si hay imágenes sin etiqueta en el CSV\nmissing = dbg[dbg['label_int'].isna()]\nif not missing.empty:\n    print(\"\\n Imágenes sin etiqueta en el CSV (primeras 10):\")\n    print(missing.head(10))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:35:14.106118Z","iopub.execute_input":"2025-08-13T21:35:14.106307Z","iopub.status.idle":"2025-08-13T21:35:14.278881Z","shell.execute_reply.started":"2025-08-13T21:35:14.106291Z","shell.execute_reply":"2025-08-13T21:35:14.278094Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Just creating a folder to save the trained model.","metadata":{}},{"cell_type":"code","source":"model_dir = Path('/') / 'kaggle' / 'working' / 'models'\nmodel_dir","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:35:14.280468Z","iopub.execute_input":"2025-08-13T21:35:14.280670Z","iopub.status.idle":"2025-08-13T21:35:14.285370Z","shell.execute_reply.started":"2025-08-13T21:35:14.280655Z","shell.execute_reply":"2025-08-13T21:35:14.284786Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Loading the model to be trained.","metadata":{}},{"cell_type":"code","source":"learn = vision_learner(dls,\n                      resnet34,\n                      metrics=error_rate,\n                      loss_func=LabelSmoothingCrossEntropy(),\n                      cbs=[BnFreeze,\n                          SaveModelCallback(monitor='error_rate'),\n                          ShowGraphCallback,\n                          ],\n                      model_dir=model_dir,\n                      ).to_fp16()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:35:14.286085Z","iopub.execute_input":"2025-08-13T21:35:14.286324Z","iopub.status.idle":"2025-08-13T21:35:15.273767Z","shell.execute_reply.started":"2025-08-13T21:35:14.286301Z","shell.execute_reply":"2025-08-13T21:35:15.273199Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"I launch a \"Scores check\" before the training, to have a reference of the learning from different perspectives/parameters.","metadata":{}},{"cell_type":"code","source":"# Scores check\n\npreds, targs = learn.get_preds()\n\nif preds.shape[1] == 2:\n    pred_labels = preds.argmax(dim=1)\n    probas = preds[:, 1]\nelse:\n    pred_labels = preds.argmax(dim=1)\n    probas = None\n\nacc = accuracy_score(targs, pred_labels)\nprec = precision_score(targs, pred_labels, average='weighted')\nrec = recall_score(targs, pred_labels, average='weighted')\nf1 = f1_score(targs, pred_labels, average='weighted')\n\nauc = roc_auc_score(targs, probas) if probas is not None else None\n\nprint(f\"Accuracy:  {acc:.4f}\")\nprint(f\"Precision: {prec:.4f}\")\nprint(f\"Recall:    {rec:.4f}\")\nprint(f\"F1-score:  {f1:.4f}\")\nif auc is not None:\n    print(f\"AUC-ROC:   {auc:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:35:15.274532Z","iopub.execute_input":"2025-08-13T21:35:15.274712Z","iopub.status.idle":"2025-08-13T21:35:54.472555Z","shell.execute_reply.started":"2025-08-13T21:35:15.274697Z","shell.execute_reply":"2025-08-13T21:35:54.471620Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As shown in the reasult, mainly in the AUC score, the accuracy is very low. The chance to get the right gender is almost random.","metadata":{}},{"cell_type":"code","source":"@delegates(learn.fit_one_cycle)\ndef train(learn, name, lr, n_epochs=5, **kwargs):\n    learn.fit_one_cycle(n_epochs, lr, **kwargs)\n    learn.save(name)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:35:54.473644Z","iopub.execute_input":"2025-08-13T21:35:54.473922Z","iopub.status.idle":"2025-08-13T21:35:54.478179Z","shell.execute_reply.started":"2025-08-13T21:35:54.473898Z","shell.execute_reply":"2025-08-13T21:35:54.477449Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"For the 1st training stage (just one epoch) I will use the default lr value.","metadata":{}},{"cell_type":"code","source":"lr = defaults.lr","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:35:54.478998Z","iopub.execute_input":"2025-08-13T21:35:54.479251Z","iopub.status.idle":"2025-08-13T21:35:54.494758Z","shell.execute_reply.started":"2025-08-13T21:35:54.479228Z","shell.execute_reply":"2025-08-13T21:35:54.494091Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 1st stage training","metadata":{}},{"cell_type":"code","source":"train(learn, 'stage_1', lr, n_epochs=1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:35:54.495434Z","iopub.execute_input":"2025-08-13T21:35:54.495705Z","iopub.status.idle":"2025-08-13T21:39:03.779871Z","shell.execute_reply.started":"2025-08-13T21:35:54.495689Z","shell.execute_reply":"2025-08-13T21:39:03.778970Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:39:03.782291Z","iopub.execute_input":"2025-08-13T21:39:03.782546Z","iopub.status.idle":"2025-08-13T21:40:14.627682Z","shell.execute_reply.started":"2025-08-13T21:39:03.782523Z","shell.execute_reply":"2025-08-13T21:40:14.626925Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Scores check\n\npreds, targs = learn.get_preds()\n\nif preds.shape[1] == 2:\n    pred_labels = preds.argmax(dim=1)\n    probas = preds[:, 1]\nelse:\n    pred_labels = preds.argmax(dim=1)\n    probas = None\n\nacc = accuracy_score(targs, pred_labels)\nprec = precision_score(targs, pred_labels, average='weighted')\nrec = recall_score(targs, pred_labels, average='weighted')\nf1 = f1_score(targs, pred_labels, average='weighted')\n\nauc = roc_auc_score(targs, probas) if probas is not None else None\n\nprint(f\"Accuracy:  {acc:.4f}\")\nprint(f\"Precision: {prec:.4f}\")\nprint(f\"Recall:    {rec:.4f}\")\nprint(f\"F1-score:  {f1:.4f}\")\nif auc is not None:\n    print(f\"AUC-ROC:   {auc:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:40:14.628589Z","iopub.execute_input":"2025-08-13T21:40:14.628830Z","iopub.status.idle":"2025-08-13T21:40:49.826996Z","shell.execute_reply.started":"2025-08-13T21:40:14.628797Z","shell.execute_reply":"2025-08-13T21:40:49.826208Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"After just 1 epoch, we have now a much better global view of the status of the model.\n- In the plotted confussion matrix, is shown almost a 10% of faliures predicting the gender.\n- On the other hand, just after 1 epoch, the rest of the scores have been notably improved.","metadata":{}},{"cell_type":"markdown","source":"Now, after the 1st stage training, I would like to check the better lr value before starting the 10 epochs training.","metadata":{}},{"cell_type":"code","source":"learn.lr_find()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:40:49.827888Z","iopub.execute_input":"2025-08-13T21:40:49.828130Z","iopub.status.idle":"2025-08-13T21:42:29.584441Z","shell.execute_reply.started":"2025-08-13T21:40:49.828106Z","shell.execute_reply":"2025-08-13T21:42:29.583689Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"While the function lf_find is recommending a value of 0.0012, I will it too big, what could add a bit desviation during the training, so I decided to be a bit more conservating taking a 3/4 of the value: 0.0008.","metadata":{}},{"cell_type":"markdown","source":"# 2nd stage training","metadata":{}},{"cell_type":"code","source":"train(learn, 'stage_2', lr=0.0008022644514217973, n_epochs=10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T21:43:12.759460Z","iopub.execute_input":"2025-08-13T21:43:12.759773Z","iopub.status.idle":"2025-08-13T22:12:08.341265Z","shell.execute_reply.started":"2025-08-13T21:43:12.759747Z","shell.execute_reply":"2025-08-13T22:12:08.339491Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T22:12:08.343863Z","iopub.execute_input":"2025-08-13T22:12:08.344260Z","iopub.status.idle":"2025-08-13T22:13:17.929539Z","shell.execute_reply.started":"2025-08-13T22:12:08.344212Z","shell.execute_reply":"2025-08-13T22:13:17.928742Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Scores check\n\npreds, targs = learn.get_preds()\n\nif preds.shape[1] == 2:\n    pred_labels = preds.argmax(dim=1)\n    probas = preds[:, 1]\nelse:\n    pred_labels = preds.argmax(dim=1)\n    probas = None\n\nacc = accuracy_score(targs, pred_labels)\nprec = precision_score(targs, pred_labels, average='weighted')\nrec = recall_score(targs, pred_labels, average='weighted')\nf1 = f1_score(targs, pred_labels, average='weighted')\n\nauc = roc_auc_score(targs, probas) if probas is not None else None\n\nprint(f\"Accuracy:  {acc:.4f}\")\nprint(f\"Precision: {prec:.4f}\")\nprint(f\"Recall:    {rec:.4f}\")\nprint(f\"F1-score:  {f1:.4f}\")\nif auc is not None:\n    print(f\"AUC-ROC:   {auc:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T22:13:17.930445Z","iopub.execute_input":"2025-08-13T22:13:17.931017Z","iopub.status.idle":"2025-08-13T22:13:52.117536Z","shell.execute_reply.started":"2025-08-13T22:13:17.930991Z","shell.execute_reply":"2025-08-13T22:13:52.116754Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"After the 2nd stage of the training, the confussion and scres check has shown a relevant improve and a very decent value specifically for the AUC score.","metadata":{}},{"cell_type":"markdown","source":"We will safe the model, before continuing with some tests.","metadata":{}},{"cell_type":"code","source":"learn.export('/kaggle/working/models/complete_model.pkl')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T22:13:52.119146Z","iopub.execute_input":"2025-08-13T22:13:52.119662Z","iopub.status.idle":"2025-08-13T22:13:52.386894Z","shell.execute_reply.started":"2025-08-13T22:13:52.119637Z","shell.execute_reply":"2025-08-13T22:13:52.386104Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.predict('/kaggle/input/spr-x-ray-gender/kaggle/kaggle/test/000000.png')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T22:13:52.387726Z","iopub.execute_input":"2025-08-13T22:13:52.388045Z","iopub.status.idle":"2025-08-13T22:13:52.570030Z","shell.execute_reply.started":"2025-08-13T22:13:52.388024Z","shell.execute_reply":"2025-08-13T22:13:52.569431Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.predict('/kaggle/input/spr-x-ray-gender/kaggle/kaggle/test/000001.png')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T22:13:52.570814Z","iopub.execute_input":"2025-08-13T22:13:52.571100Z","iopub.status.idle":"2025-08-13T22:13:52.681758Z","shell.execute_reply.started":"2025-08-13T22:13:52.571075Z","shell.execute_reply":"2025-08-13T22:13:52.681049Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Final test and submission","metadata":{}},{"cell_type":"code","source":"# Archivos de test en orden determinista\ntest_dir = Path('/kaggle/input/spr-x-ray-gender/kaggle/kaggle/test')\ntest_files = get_image_files(test_dir).sorted()  # '000000.png', '000001.png', ...\n\n# DataLoader de test con la MISMA preproc que train/valid\ntest_dl = dls.test_dl(test_files, bs=dls.bs)\n\n# Probabilidades por clase\nprobs, _ = learn.get_preds(dl=test_dl)  # shape: [N, 2] en tu caso\n\n# Índice de la clase positiva (como i[1] en tu ejemplo TF)\npos_idx = dls.vocab.o2i['hombre'] if hasattr(dls.vocab, 'o2i') else list(dls.vocab).index('hombre')\n\np_pos = probs[:, pos_idx].detach().cpu().numpy()  # prob. de 'hombre'\n\n# imageId sin ceros a la izquierda (para coincidir con sample_submission)\nimage_ids = [int(p.stem) for p in test_files]  # '000123' -> 123\n\n# DataFrame y guardado\nsubmission = pd.DataFrame({'imageId': image_ids, 'gender': p_pos})\nsubmission = submission.sort_values('imageId').reset_index(drop=True)\nsubmission.to_csv('/kaggle/working/submission.csv', index=False)\n\nprint(submission.head())\nprint('Guardado en /kaggle/working/submission.csv  -> filas:', len(submission))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-13T22:13:52.682577Z","iopub.execute_input":"2025-08-13T22:13:52.682817Z","iopub.status.idle":"2025-08-13T22:17:21.045150Z","shell.execute_reply.started":"2025-08-13T22:13:52.682793Z","shell.execute_reply":"2025-08-13T22:17:21.044297Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Conclusion\n\nAfter many trials, I have reached a decent value, that I consider that could match very well the goal of the assignment.\nSpecifically, I mean the AUC-ROC score, shown in the last check with a value of 0.9924.\n\n# Parameters and changes made during the trials and the conclusions after the tests\n\nI have made a combination of different parameters: sz value on the Resize, lr value, quantity of epochs and separating this quantity in several stages correcting the lr value between the stages, the type of loss_func, etc.\n\n- sz value: increasing the size value to 256 affected negatively in the final result. In the 1st stage training the result was much better, promissing a potential improvement during the 2nd stage trianing, but during the 2nd stage the loss decresed very slow and reached its limite during the first 5 epochs of training.\n- lr value: naturally, too small values have slowed the improvement process limiting also the possibility to reach low values of loss. On the other hand, big values, has entered the training in a fluctuating decrease of the loss, ramdomizing the success of the training. \n- epochs: after trying to separate the training in more stages with less epochs or less stages with more epochs, doens't improved aparently the results. The best middle point was to do a 1st stage to find the best valley value and then use it for the 2nd stage of the training with 10 epochs.\n- loss_func: even it could be confussing for me at the beginning, that the gender value was not so extremly tagged as I expected, the results of the training were much accurate, probably because the value of the tags during the training were much \"smooth\" than using the standard CrossEntropyLoss.\n\n\nFinally, I have found a combination of these value that works as best as I could reach using the maximal computing hours kaggle offered me.\nI see anyway much place to improve. I would try using probably another model (for example ResNet50 or EfficientNet) and probably TensorFlow instead of FastAI. It will be a very interesting challenge to do in my future practices.","metadata":{}}]}