{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Versión 4 Empezado a las 15:00 07/02/2022","metadata":{}},{"cell_type":"markdown","source":"# ⬜ Importamos las librerías que vamos a utilizar \n* Por lo que he visto en apuntes se utilizan estas, ya veremos si acaba siendo así o prescindiremos de algunas","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport glob\nimport shutil\nimport torch\nimport ast\nimport sys\nsys.path.append('../input/tensorflow-great-barrier-reef')\n\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\n\nfrom PIL import Image\n\nfrom joblib import Parallel, delayed\n\nfrom IPython.display import display\n\ntorch.__version__","metadata":{"execution":{"iopub.status.busy":"2022-02-08T14:50:56.808038Z","iopub.execute_input":"2022-02-08T14:50:56.808684Z","iopub.status.idle":"2022-02-08T14:50:58.486486Z","shell.execute_reply.started":"2022-02-08T14:50:56.808552Z","shell.execute_reply":"2022-02-08T14:50:58.485675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🟦 Definición de parámetros y funciones\n\n### 🟦 Parámetros\n\n#### He modificado los siguientes parámetros del entrenamiento respecto a los entrenamientos anteriores, de manera que:\n\n* `BATCH` :  **10 (autobatching)** Dejaremos al propio programa que decida el tamaño de batch más adecuado dado el hardware del que disponemos\n* `MODEL` :  **yolov5s6** Utilizaremos una versión más extensa del modelo, más profunda y precisa pero más lenta realizando detecciones\n* `EPOCHS` :  **20** Ahora ue utilizaremos la versión m, vamos a realizar 30 epochs, habra que establecer también un early stop por si nos encontramos con overfitting\n* `PATIENCE` : **3** Epochs, pasadas 6 epochs sin mejoras, el entrenamiento finalizará\n* `OPTIMIZER` :  **Adam**\n* `CONF` : **0.275** El modelo tendrá en cuenta las predicciones con una confianza (score) de CONF o superior\n* `IOU` : **0.25** Queremos generar una bbox que se ajuste perfectamente alrededor del objeto a detectar. Las mmuestras de entrenamiento, tienen boxes que hemos considerado óptimas, mientras el modelo predice boxes que podrían coinicidir en mejor o peor medida con la box real. El objetivo es ir mejorando, hasta que la box de entrenamiento y la box del modelo se superpongan perfectamente, el IOU entre las dos cajas sea igual a 1, nosotros estableceremos un IOU 0.5 para mejorar la precisión en la detección \n* `IMG_SIZE (anteriormente DIM)` : **1400** Aumentaremos el tamaño de las imagenes de validación","metadata":{}},{"cell_type":"code","source":"FOLD      = 1 # which fold to train\nIMG_SIZE  = 1400\nMODEL     = 'yolov5s6'\nBATCH     = 10\nEPOCHS    = 20\nOPTMIZER  = 'Adam'\nAUGMENT   = True\nCONF      = 0.275   \nIOU       = 0.2\nPATIENCE  = 5\n\nPROJECT   = 'great-barrier-reef'\nNAME      = f'{MODEL}-dim{IMG_SIZE}-fold{FOLD}'\n\nREMOVE_NOBBOX = True # remove images with no bbox\nROOT_DIR  = '/kaggle/input/tensorflow-great-barrier-reef/'\nIMAGE_DIR = '/kaggle/images' # directory to save images\nLABEL_DIR = '/kaggle/labels' # directory to save labels\n\n\nweights_path = '../input/yolov5-1400-1/'\nCKPT_PATH = weights_path + '/best.pt'\n\nTRACKING  = False\nFDA_aug   = False\nWORKERS   = 0\n\n#AUGMENT  = False # Set True again     # TTA will run for an hour, will gain improvement","metadata":{"execution":{"iopub.status.busy":"2022-02-08T14:50:58.488452Z","iopub.execute_input":"2022-02-08T14:50:58.489012Z","iopub.status.idle":"2022-02-08T14:50:58.496025Z","shell.execute_reply.started":"2022-02-08T14:50:58.488944Z","shell.execute_reply":"2022-02-08T14:50:58.495343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 🟦 Importamos las funciones para crear las cajas de las imágenes","metadata":{}},{"cell_type":"code","source":"!pip install -qU bbox-utility # check https://github.com/awsaf49/bbox for source code","metadata":{"execution":{"iopub.status.busy":"2022-02-08T14:51:05.650766Z","iopub.execute_input":"2022-02-08T14:51:05.651211Z","iopub.status.idle":"2022-02-08T14:51:15.191072Z","shell.execute_reply.started":"2022-02-08T14:51:05.651173Z","shell.execute_reply":"2022-02-08T14:51:15.190032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check https://github.com/awsaf49/bbox for source code of following utility functions\n\nfrom bbox.utils import coco2yolo, coco2voc, voc2yolo\nfrom bbox.utils import draw_bboxes, load_image\nfrom bbox.utils import clip_bbox, str2annot, annot2str\n\ndef get_bbox(annots):\n    bboxes = [list(annot.values()) for annot in annots]\n    return bboxes\n\ndef get_imgsize(row):\n    row['width'], row['height'] = imagesize.get(row['image_path'])\n    return row\n\nnp.random.seed(32)\ncolors = [(np.random.randint(255), np.random.randint(255), np.random.randint(255))\\\n          for idx in range(1)]\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-02-08T14:51:15.192898Z","iopub.execute_input":"2022-02-08T14:51:15.193165Z","iopub.status.idle":"2022-02-08T14:51:15.954913Z","shell.execute_reply.started":"2022-02-08T14:51:15.193136Z","shell.execute_reply":"2022-02-08T14:51:15.954172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🟧 Preparación del dataset\n\n#### Nuestro dataset `train.csv` se compone de \n* `video_id` El ID del video al que pertenece el frame\n* `sequence` La secuencia (irrelevante)\n* `video_frame` El número del frame dentro del video **(importante)** \n* `sequence_frame` El número del frame dentro de su secuencia (irrelevante)\n* `image_id` El ID del frame respecto a al ID del video al que pertenece \n* `annotations` Las boxes del frame si las tiene, con el siguiente formato;  [{'x': 559, 'y': 213, 'width': 50, 'height': 32}] **(importante)**\n\n### La entrada de YOLO requerirá de:\n* `image_path` - Las direcciones de las imagenes que utilizaremos para entrenar\n* `num_bbox` - La cantidad de boxes que contendrá el frame\n* `bboxes` - Arrays de 4 números para las boxes que contendrán los targets de las imágenes, con el siguiente formato;  [559,213,50,32}]","metadata":{}},{"cell_type":"code","source":"# Creamos los directorios básicos\n\n!mkdir -p {IMAGE_DIR}\n!mkdir -p {LABEL_DIR}","metadata":{"execution":{"iopub.status.busy":"2022-02-08T14:51:27.297495Z","iopub.execute_input":"2022-02-08T14:51:27.297931Z","iopub.status.idle":"2022-02-08T14:51:28.877546Z","shell.execute_reply.started":"2022-02-08T14:51:27.297891Z","shell.execute_reply":"2022-02-08T14:51:28.876403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train Data\n\ndframe = pd.read_csv(f'{ROOT_DIR}/train.csv')\n\ndframe['old_image_path'] = f'{ROOT_DIR}/train_images/video_'+dframe.video_id.astype(str)+'/'+dframe.video_frame.astype(str)+'.jpg'\n\ndframe['image_path']  = f'{IMAGE_DIR}/'+dframe.image_id+'.jpg'\ndframe['label_path']  = f'{LABEL_DIR}/'+dframe.image_id+'.txt'\n\ndframe['annotations'] = dframe['annotations'].progress_apply(eval)\ndframe.head(-1)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:20:24.030464Z","iopub.execute_input":"2022-02-08T15:20:24.031175Z","iopub.status.idle":"2022-02-08T15:20:24.445983Z","shell.execute_reply.started":"2022-02-08T15:20:24.031127Z","shell.execute_reply":"2022-02-08T15:20:24.445347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Duplicamos nuestro set de entrenamiento para poder añadir imagenes sin bboxes","metadata":{}},{"cell_type":"code","source":"copy_of_frame = dframe","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:20:27.340586Z","iopub.execute_input":"2022-02-08T15:20:27.341683Z","iopub.status.idle":"2022-02-08T15:20:27.346385Z","shell.execute_reply.started":"2022-02-08T15:20:27.341627Z","shell.execute_reply":"2022-02-08T15:20:27.344935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 🟧 Observamos que tan sólo el 20% de la imagenes cuentan con boxes, vamos a quedarnos con un subset que contenga únicamente las imágenes con boxes","metadata":{}},{"cell_type":"code","source":"dframe['num_bbox'] = dframe['annotations'].progress_apply(lambda x: len(x))\ndata = (dframe.num_bbox>0).value_counts()/len(dframe)*100\nprint(f\"No BBox: {data[0]:0.2f}% | With BBox: {data[1]:0.2f}%\")\ndframe.head(-1)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:20:30.031819Z","iopub.execute_input":"2022-02-08T15:20:30.032367Z","iopub.status.idle":"2022-02-08T15:20:30.119650Z","shell.execute_reply.started":"2022-02-08T15:20:30.032315Z","shell.execute_reply":"2022-02-08T15:20:30.118820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dframe = dframe.query(\"num_bbox>0\")","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:21:54.486250Z","iopub.execute_input":"2022-02-08T15:21:54.486543Z","iopub.status.idle":"2022-02-08T15:21:54.498112Z","shell.execute_reply.started":"2022-02-08T15:21:54.486515Z","shell.execute_reply":"2022-02-08T15:21:54.497290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 🟧 Añadiremos un subset de imágenes sin boxes para que el modelo se encuentre con cualquier caso, incluso cuando no tiene que buscar nada","metadata":{}},{"cell_type":"code","source":"copy_of_frame.query(\"num_bbox<1\").progress_apply(lambda x: len(x))\ncopy_of_frame = copy_of_frame.sample(n = 6000 - len(dframe))","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:22:23.071940Z","iopub.execute_input":"2022-02-08T15:22:23.072804Z","iopub.status.idle":"2022-02-08T15:22:23.110199Z","shell.execute_reply.started":"2022-02-08T15:22:23.072765Z","shell.execute_reply":"2022-02-08T15:22:23.109312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.concat([dframe, copy_of_frame])\ndf","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:22:25.211709Z","iopub.execute_input":"2022-02-08T15:22:25.212003Z","iopub.status.idle":"2022-02-08T15:22:25.236496Z","shell.execute_reply.started":"2022-02-08T15:22:25.211971Z","shell.execute_reply":"2022-02-08T15:22:25.235705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 🟧 Copiamos las imagenes a nuestro directorio de trabajo para que YOLO pueda acceder a ellas","metadata":{}},{"cell_type":"code","source":"def make_copy(row):\n    shutil.copyfile(row.old_image_path, row.image_path)\n    return\n\nimage_paths = df.old_image_path.tolist()\n_ = Parallel(n_jobs=-1, backend='threading')(delayed(make_copy)(row) for _, row in tqdm(df.iterrows(), total=len(df)))","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:22:33.753974Z","iopub.execute_input":"2022-02-08T15:22:33.754242Z","iopub.status.idle":"2022-02-08T15:23:20.225671Z","shell.execute_reply.started":"2022-02-08T15:22:33.754216Z","shell.execute_reply":"2022-02-08T15:23:20.225023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 🟧 Añadimos las boxes en el formato adecuado al dataset ","metadata":{}},{"cell_type":"code","source":"df['bboxes'] = df.annotations.progress_apply(get_bbox)\ndf.head(-1)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:23:22.229327Z","iopub.execute_input":"2022-02-08T15:23:22.229611Z","iopub.status.idle":"2022-02-08T15:23:22.307360Z","shell.execute_reply.started":"2022-02-08T15:23:22.229582Z","shell.execute_reply":"2022-02-08T15:23:22.306796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 🟧 Todas nuestras imágenes tienen un tamaño de [Ancho, Alto] de `[1280, 720]` , estas dimensiones las añadimos a nuestro DataFrame","metadata":{}},{"cell_type":"code","source":"df['width']  = 1280\ndf['height'] = 720\ndisplay(df.head(2))","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:23:26.107927Z","iopub.execute_input":"2022-02-08T15:23:26.108321Z","iopub.status.idle":"2022-02-08T15:23:26.126182Z","shell.execute_reply.started":"2022-02-08T15:23:26.108274Z","shell.execute_reply":"2022-02-08T15:23:26.125552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🟧 Creamos las etiquetas\n\n### Necesitamos exportar nuestras etiquetas en el formato que necesita YOLO, `con un archivo *.txt por imagen` (si no hay bboxes en la imagen, no será necesario). Los requisitos del texto serán:\n\n#### - Una fila por objeto a detectar.\n#### - Cada fila seguirá el siguiente formato: `x_center, y_center, width, height]`.\n#### - Las coordenadas de las bboxes tienen que ser normalizadas (de 0 a 1): \n##### Cómo la unidad de medida en nuestra imagen es el píxel, dividiremos `x_center` y `width` entre `image width`, lo mismo para el eje y, `y_center` y `height` por `image height`.\n#### - Las números de clase se indexarán igual que en python (empezando desde el 0).\n\n### El formato de las bboxes en esta competición es COCO [x_min, y_min, width, height]. Por lo que deberemos de convertir del formato COCO a YOLO.\n","metadata":{}},{"cell_type":"code","source":"cnt = 0\nall_bboxes = []\nbboxes_info = []\n\nfor row_idx in tqdm(range(df.shape[0])):\n    row = df.iloc[row_idx]\n    image_height = row.height\n    image_width  = row.width\n    bboxes_coco  = np.array(row.bboxes).astype(np.float32).copy()\n    num_bbox     = len(bboxes_coco)\n    names        = ['cots']*num_bbox\n    labels       = np.array([0]*num_bbox)[..., None].astype(str)\n    ## Create Annotation(YOLO)\n    \n    with open(row.label_path, 'w') as f:\n        if num_bbox<1:\n            annot = ''\n            f.write(annot)\n            cnt+=1\n            continue\n        bboxes_voc  = coco2voc(bboxes_coco, image_height, image_width)\n        bboxes_voc  = clip_bbox(bboxes_voc, image_height, image_width)\n        bboxes_yolo = voc2yolo(bboxes_voc, image_height, image_width).astype(str)\n        all_bboxes.extend(bboxes_yolo.astype(float))\n        bboxes_info.extend([[row.image_id, row.video_id, row.sequence]]*len(bboxes_yolo))\n        annots = np.concatenate([labels, bboxes_yolo], axis=1)\n        string = annot2str(annots)\n        f.write(string)\n\nprint('Missing:',cnt)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:23:45.412500Z","iopub.execute_input":"2022-02-08T15:23:45.412917Z","iopub.status.idle":"2022-02-08T15:23:51.524845Z","shell.execute_reply.started":"2022-02-08T15:23:45.412887Z","shell.execute_reply":"2022-02-08T15:23:51.524216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🟧 Creamos las carpetas\n\n#### No todos los conjuntos de imágenes tienen el mismo tamaño, cosa que podrá dar algo de problema en la validación cruzada \"cross_validation\"\n\n","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GroupKFold\nkf = GroupKFold(n_splits = 3)\ndf = df.reset_index(drop=True)\ndf['fold'] = -1\nfor fold, (train_idx, val_idx) in enumerate(kf.split(df, groups=df.video_id.tolist())):\n    df.loc[val_idx, 'fold'] = fold\ndisplay(df.fold.value_counts())","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:24:37.979087Z","iopub.execute_input":"2022-02-08T15:24:37.979371Z","iopub.status.idle":"2022-02-08T15:24:38.703994Z","shell.execute_reply.started":"2022-02-08T15:24:37.979340Z","shell.execute_reply":"2022-02-08T15:24:38.703165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🟧 Vista de algunas imagenes de entrenamiento con sus bboxes.\n\n#### El objetivo de YOLO será obtener los resultados más similares posibles a los de éstas imagenes","metadata":{}},{"cell_type":"code","source":"df2 = df[(df.num_bbox>0)].sample(100) # takes samples with bbox\ny = 2; x = 2\n\nplt.figure(figsize=(12.8*x, 7.2*y))\n\nfor idx in range(x*y):\n    row = df2.iloc[idx]\n    img           = load_image(row.image_path)\n    image_height  = row.height\n    image_width   = row.width\n    \n    with open(row.label_path) as f:\n        annot = str2annot(f.read())\n    \n    bboxes_yolo = annot[...,1:]\n    labels      = annot[..., 0].astype(int).tolist()\n    names         = ['cots']*len(bboxes_yolo)\n    \n    plt.subplot(y, x, idx+1)\n    plt.imshow(draw_bboxes(img = img,\n                           bboxes = bboxes_yolo, \n                           classes = names,\n                           class_ids = labels,\n                           class_name = True, \n                           colors = colors, \n                           bbox_format = 'yolo',\n                           line_thickness = 3))\n    plt.axis('OFF')\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:24:48.989142Z","iopub.execute_input":"2022-02-08T15:24:48.989587Z","iopub.status.idle":"2022-02-08T15:24:51.374923Z","shell.execute_reply.started":"2022-02-08T15:24:48.989558Z","shell.execute_reply":"2022-02-08T15:24:51.374223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🟧 Finalmente, veamos nuestro set de imágenes de entrenamiento y validación","metadata":{}},{"cell_type":"code","source":"train_files = []\nval_files   = []\ntrain_df = df.query(\"fold!=@FOLD\")\nvalid_df = df.query(\"fold==@FOLD\")\ntrain_files += list(train_df.image_path.unique())\nval_files += list(valid_df.image_path.unique())\nlen(train_files), len(val_files)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:25:14.820740Z","iopub.execute_input":"2022-02-08T15:25:14.821039Z","iopub.status.idle":"2022-02-08T15:25:14.842127Z","shell.execute_reply.started":"2022-02-08T15:25:14.821008Z","shell.execute_reply":"2022-02-08T15:25:14.841251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🟥 Configuración\n### \n### 🟥 Directorios de las imagenes de entrenamiento y validación","metadata":{}},{"cell_type":"code","source":"import yaml\n\ncwd = '/kaggle/working/'\n\nwith open(os.path.join( cwd , 'train.txt'), 'w') as f:\n    for path in train_df.image_path.tolist():\n        f.write(path+'\\n')\n            \nwith open(os.path.join(cwd , 'val.txt'), 'w') as f:\n    for path in valid_df.image_path.tolist():\n        f.write(path+'\\n')\n\ndata = dict(\n    path  = '/kaggle/working',\n    train =  os.path.join( cwd , 'train.txt') ,\n    val   =  os.path.join( cwd , 'val.txt' ),\n    nc    = 1,\n    names = ['cots'],\n    )\n\nwith open(os.path.join( cwd , 'gbr.yaml'), 'w') as outfile:\n    yaml.dump(data, outfile, default_flow_style=False)\n\nf = open(os.path.join( cwd , 'gbr.yaml'), 'r')\nprint('\\nyaml:')\nprint(f.read())","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:25:19.790745Z","iopub.execute_input":"2022-02-08T15:25:19.791562Z","iopub.status.idle":"2022-02-08T15:25:19.806109Z","shell.execute_reply.started":"2022-02-08T15:25:19.791526Z","shell.execute_reply":"2022-02-08T15:25:19.805062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 🟥 Parámetros de YOLO, podemos ajustarlos a nuestro gusto, aunque los resultados pueden variar mucho\n\nYOLOv5 has about 30 hyperparameters used for various training settings. These are defined in yaml files in the /data directory. Better initial guesses will produce better final results, so it is important to initialize these values properly before evolving. If in doubt, simply use the default values, which are optimized for YOLOv5 COCO training from scratch.\n\n#### Train\n* `lr0` : **0.001**   (initial learning rate (SGD=1E-2, Adam=1E-3))\n* `lrf` : **0.05**   (final OneCycleLR learning rate (lr0 * lrf))\n* `momentum` : **0.937**   (SGD momentum/Adam beta1)\n* `weight_decay` : **0.0005**   (optimizer weight decay 5e-4)\n* `warmup_epochs` : **2.0**   (warmup epochs (fractions ok))\n* `warmup_momentum` : **0.8**   (warmup initial momentum)\n* `warmup_bias_lr` : **0.1**   (warmup initial bias lr)\n* `box` : **0.04**   (box loss gain)\n* `cls` : **0.5**   (cls loss gain)\n* `cls_pw` : **1.0**   (cls BCELoss positive_weight)\n* `obj` : **1.0**   (obj loss gain (scale with pixels))\n* `obj_pw` : **1.0**   (obj BCELoss positive_weight)\n* `iou_t` : **0.250**   (IoU training threshold)\n* `anchor_t` : **4.0**   (anchor-multiple threshold)\n* `anchors` : **3**   (achors per output layer (0 to ignore))\n* `fl_gamma` : **0.0**   (focal loss gamma (efficientDet default gamma=1.5))\n* `copy_paste` : **0.0**   (segment copy-paste (probability))\n\n#### Augmentation\n\n##### Cambios en la vista\n* `degrees` : **0.0**   (image rotation (+/- deg))\n* `translate` : **0.10**   (image translation (+/- fraction))\n* `scale` : **0.6**   (image scale (+/- gain))\n* `shear` : **0.0**   (image shear (+/- deg))\n* `perspective` : **0.0**   (image perspective (+/- fraction), range 0-0.001)\n* `flipud` : **0.75**   (image flip up-down (probability))\n* `fliplr` : **0.75**   (image flip left-right (probability))\n* `mosaic` : **0.75**   (image mosaic (probability))\n\n##### Mezcla de imagenes para confundiar al modelo\n* `mixup` : **0.75**   (image mixup (probability))\n\n##### Edición de color\n* `hsv_h` : **0.015**   (image HSV-Hue augmentation (fraction))\n* `hsv_s` : **0.7**   (image HSV-Saturation augmentation (fraction))\n* `hsv_v` : **0.5**   (image HSV-Value augmentation (fraction))","metadata":{}},{"cell_type":"code","source":"%%writefile /kaggle/working/hyp.yaml\nlr0: 0.001  # initial learning rate (SGD=1E-2, Adam=1E-3)\nlrf: 0.05  # final OneCycleLR learning rate (lr0 * lrf)\nmomentum: 0.937  # SGD momentum/Adam beta1\nweight_decay: 0.0005  # optimizer weight decay 5e-4\nwarmup_epochs: 2.0  # warmup epochs (fractions ok)\nwarmup_momentum: 0.8  # warmup initial momentum\nwarmup_bias_lr: 0.1  # warmup initial bias lr\nbox: 0.04  # box loss gain\ncls: 0.5  # cls loss gain\ncls_pw: 1.0  # cls BCELoss positive_weight\nobj: 1.0  # obj loss gain (scale with pixels)\nobj_pw: 1.0  # obj BCELoss positive_weight\niou_t: 0.250  # IoU training threshold\nanchor_t: 4.0  # anchor-multiple threshold\n# anchors: 3  # anchors per output layer (0 to ignore)\nfl_gamma: 0.0  # focal loss gamma (efficientDet default gamma=1.5)\nhsv_h: 0.015  # image HSV-Hue augmentation (fraction)\nhsv_s: 0.7  # image HSV-Saturation augmentation (fraction)\nhsv_v: 0.4  # image HSV-Value augmentation (fraction)\ndegrees: 0.0  # image rotation (+/- deg)\ntranslate: 0.10  # image translation (+/- fraction)\nscale: 0.5  # image scale (+/- gain)\nshear: 0.0  # image shear (+/- deg)\nperspective: 0.0  # image perspective (+/- fraction), range 0-0.001\nflipud: 0.7  # image flip up-down (probability)\nfliplr: 0.7  # image flip left-right (probability)\nmosaic: 0.7  # image mosaic (probability)\nmixup: 0.7 # image mixup (probability)\ncopy_paste: 0.0  # segment copy-paste (probability)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T15:25:24.427191Z","iopub.execute_input":"2022-02-08T15:25:24.427840Z","iopub.status.idle":"2022-02-08T15:25:24.434548Z","shell.execute_reply.started":"2022-02-08T15:25:24.427808Z","shell.execute_reply":"2022-02-08T15:25:24.433673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ⬛ Entrenamiento de YOLOv5\n\n### ⬛ Preparamos todo lo necesario para poder utilizar el modelo","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n!rm -r /kaggle/working/yolov5\n!git clone https://github.com/ultralytics/yolov5 # clone\n!cp -r /kaggle/input/yolov5-lib-ds /kaggle/working/yolov5\n%cd yolov5\n%pip install -qr requirements.txt  # install","metadata":{"execution":{"iopub.status.busy":"2022-02-03T16:11:17.675729Z","iopub.execute_input":"2022-02-03T16:11:17.676168Z","iopub.status.idle":"2022-02-03T16:11:29.921624Z","shell.execute_reply.started":"2022-02-03T16:11:17.676132Z","shell.execute_reply":"2022-02-03T16:11:29.920725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Comprobamos que todo funciona correctamente","metadata":{}},{"cell_type":"code","source":"from yolov5 import utils\ndisplay = utils.notebook_init()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T16:11:29.92333Z","iopub.execute_input":"2022-02-03T16:11:29.92504Z","iopub.status.idle":"2022-02-03T16:11:30.307908Z","shell.execute_reply.started":"2022-02-03T16:11:29.924996Z","shell.execute_reply":"2022-02-03T16:11:30.307038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ⬛ Entrenamiento\n\n### Llamamos al script `train.py` al que podemos especificar los siguientes argumentos:\n\n#### Parámetros del entrenamiento\n* `--weights`,  **type**=str, default=ROOT / 'yolov5s.pt', *help='initial weights path'*\n* `--epochs`,  **type**=int, default=300\n* `--data`,  **type**=str, default=ROOT / 'data/coco128.yaml', *help='dataset.yaml path'*\n* `--hyp`,  **type**=str, default=ROOT / 'data/hyps/hyp.scratch.yaml', *help='hyperparameters path'*\n* `--batch-size`, **type**=int, default=16, *help='total batch size for all GPUs, -1 for autobatch'*\n* `--optimizer`,  **type**=str, default='SGD', choices=['SGD', 'Adam', 'AdamW'], *help='optimizer'*\n* `--cfg`,  **type**=str, default='', *help='model.yaml path'*\n* `--evolve`,  **type**=int, nargs='?', const=300, *help='evolve hyperparameters for x generations'*\n* `--bucket`,  **type**=str, default='', *help='gsutil bucket'*\n* `--cache`,  **type**=str, nargs='?', const='ram', *help='--cache images in \"ram\" (default) or \"disk\"'*\n* `--imgsz` o `--img` o `--img-size`, **type**=int, default=640, *help='train, val image size (pixels)'*\n* `--workers`,  **type**=int, default=8, *help='max dataloader workers (per RANK in DDP mode)'*\n* `--label-smoothing`, **type**=float, default=0.0, *help='Label smoothing epsilon'*\n* `--patience`,  **type**=int, default=100, *help='EarlyStopping patience (epochs without improvement)'*\n* `--save-period`, **type**=int, default=-1, *help='Save checkpoint every x epochs (disabled if < 1)'*\n* `--local_rank`,  **type**=int, default=-1, *help='DDP parameter, do not modify'*\n* `--freeze`,  **nargs**='+', type=int, default=[0], *help='Freeze layers: backbone=10, first3=0 1 2'*\n* `--resume`,  **nargs**='?', **const**=True, default=False, *help='resume most recent training'*\n* `--device`,  **default**='', *help='cuda device, i.e. 0 or 0,1,2,3 or cpu'*\n\n#### Parámetros activados *por defecto*\n* `--rect`,  **action**='store_true', *help='rectangular training'*\n* `--nosave`,  **action**='store_true', *help='only save final checkpoint'*\n* `--noval`,  **action**='store_true', *help='only validate final epoch'*\n* `--noautoanchor`,  **action**='store_true', *help='disable AutoAnchor'*\n* `--image-weights`, **action**='store_true', *help='use weighted image selection for training'*\n* `--multi-scale`, **action**='store_true', *help='vary img-size +/- 50%%'*\n* `--single-cls`, **action**='store_true', *help='train multi-class data as single-class'*\n* `--sync-bn`, **action**='store_true', *help='use SyncBatchNorm, only available in DDP mode'*\n* `--exist-ok`, **action**='store_true', *help='existing project/name ok, do not increment'*\n* `--quad`,  **action**='store_true', *help='quad dataloader'*\n* `--linear-lr`, **action**='store_true', *help='linear LR' *\n\n### Argumentos que forman en nombre y ubicación del proyecto\n* `--project`,  **default**=ROOT / 'runs/train', *help='save to project/name'*\n* `--name`,  **default**='exp', *help='save to project/name'*\n","metadata":{}},{"cell_type":"code","source":"!python train.py --img {IMG_SIZE}\\\n--batch {BATCH}\\\n--epochs {EPOCHS}\\\n--optimizer {OPTMIZER}\\\n--data /kaggle/working/gbr.yaml\\\n--hyp /kaggle/working/hyp.yaml\\\n--weights {MODEL}.pt\\\n--project {PROJECT}\\\n--name {NAME}\\\n--save-period 5\\\n--workers {WORKERS}\\\n--exist-ok ","metadata":{"execution":{"iopub.status.busy":"2022-02-03T16:11:30.309841Z","iopub.execute_input":"2022-02-03T16:11:30.310888Z","iopub.status.idle":"2022-02-03T16:53:40.198282Z","shell.execute_reply.started":"2022-02-03T16:11:30.310834Z","shell.execute_reply":"2022-02-03T16:53:40.19727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -r {IMAGE_DIR}\n!rm -r {LABEL_DIR}","metadata":{"execution":{"iopub.status.busy":"2022-02-03T16:53:40.208069Z","iopub.execute_input":"2022-02-03T16:53:40.208493Z","iopub.status.idle":"2022-02-03T16:53:42.079294Z","shell.execute_reply.started":"2022-02-03T16:53:40.208445Z","shell.execute_reply":"2022-02-03T16:53:42.07836Z"},"trusted":true},"execution_count":null,"outputs":[]}]}