{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# [Tensorflow - Help Protect the Great Barrier Reef](https://www.kaggle.com/c/tensorflow-great-barrier-reef)\n> Detect crown-of-thorns starfish in underwater image data\n\n![](\"https://storage.googleapis.com/kaggle-competitions/kaggle/31703/logos/header.png?t=2021-10-29-00-30-04\")
","metadata":{}},{"cell_type":"markdown","source":"## ๐ Notebooks:\n* Train: [Great-Barrier-Reef: YOLOv5 [train] ๐](https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-train)\n* Infer: [Great-Barrier-Reef: YOLOv5 [infer] ๐](https://www.kaggle.com/awsaf49/great-barrier-reef-yolov5-infer)","metadata":{}},{"cell_type":"markdown","source":"# ๐ Install Libraries","metadata":{}},{"cell_type":"code","source":"!pip install -qU wandb\n!pip install -qU bbox-utility # check https://github.com/awsaf49/bbox for source code","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-06T20:28:25.757940Z","iopub.execute_input":"2023-07-06T20:28:25.758632Z","iopub.status.idle":"2023-07-06T20:28:54.864942Z","shell.execute_reply.started":"2023-07-06T20:28:25.758596Z","shell.execute_reply":"2023-07-06T20:28:54.863752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ Import Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport glob\n\nimport shutil\nimport sys\nsys.path.append('../input/tensorflow-great-barrier-reef')\n\nfrom joblib import Parallel, delayed\n\nfrom IPython.display import display","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-07-06T20:28:54.867827Z","iopub.execute_input":"2023-07-06T20:28:54.868231Z","iopub.status.idle":"2023-07-06T20:28:55.228606Z","shell.execute_reply.started":"2023-07-06T20:28:54.868186Z","shell.execute_reply":"2023-07-06T20:28:55.227659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ Key-Points\n* One have to submit prediction using the provided **python time-series API**, which makes this competition different from previous Object Detection Competitions.\n* Each prediction row needs to include all bounding boxes for the image. Submission is format seems also **COCO** which means `[x_min, y_min, width, height]`\n* Copmetition metric `F2` tolerates some false positives(FP) in order to ensure very few starfish are missed. Which means tackling **false negatives(FN)** is more important than false positives(FP). \n$$F2 = 5 \\cdot \\frac{precision \\cdot recall}{4\\cdot precision + recall}$$","metadata":{}},{"cell_type":"markdown","source":"# โญ WandB\n![](\"https://camo.githubusercontent.com/dd842f7b0be57140e68b2ab9cb007992acd131c48284eaf6b1aca758bfea358b/68747470733a2f2f692e696d6775722e636f6d2f52557469567a482e706e67\")
\n\nWeights & Biases (W&B) is MLOps platform for tracking our experiemnts. We can use it to Build better models faster with experiment tracking, dataset versioning, and model management. Some of the cool features of W&B:\n\n* Track, compare, and visualize ML experiments\n* Get live metrics, terminal logs, and system stats streamed to the centralized dashboard.\n* Explain how your model works, show graphs of how model versions improved, discuss bugs, and demonstrate progress towards milestones.\n","metadata":{}},{"cell_type":"code","source":"import wandb\n\ntry:\n from kaggle_secrets import UserSecretsClient\n user_secrets = UserSecretsClient()\n api_key = user_secrets.get_secret(\"WANDB\")\n wandb.login(key=api_key)\n anonymous = None\nexcept:\n wandb.login(anonymous='must')\n print('To use your W&B account,\\nGo to Add-ons -> Secrets and provide your W&B access token. Use the Label name as WANDB. \\nGet your W&B access token from here: https://wandb.ai/authorize')","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:29:02.076758Z","iopub.execute_input":"2023-07-06T20:29:02.077125Z","iopub.status.idle":"2023-07-06T20:29:06.905623Z","shell.execute_reply.started":"2023-07-06T20:29:02.077095Z","shell.execute_reply":"2023-07-06T20:29:06.904450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ Meta Data\n* `train_images/` - Folder containing training set photos of the form `video_{video_id}/{video_frame}.jpg`.\n\n* `[train/test].csv` - Metadata for the images. As with other test files, most of the test metadata data is only available to your notebook upon submission. Just the first few rows available for download.\n\n* `video_id` - ID number of the video the image was part of. The video ids are not meaningfully ordered.\n* `video_frame` - The frame number of the image within the video. Expect to see occasional gaps in the frame number from when the diver surfaced.\n* `sequence` - ID of a gap-free subset of a given video. The sequence ids are not meaningfully ordered.\n* `sequence_frame` - The frame number within a given sequence.\n* `image_id` - ID code for the image, in the format `{video_id}-{video_frame}`\n* `annotations` - The bounding boxes of any starfish detections in a string format that can be evaluated directly with Python. Does not use the same format as the predictions you will submit. Not available in test.csv. A bounding box is described by the pixel coordinate `(x_min, y_min)` of its lower left corner within the image together with its `width` and `height` in pixels --> (COCO format).","metadata":{}},{"cell_type":"code","source":"FOLD = 1 # which fold to train\nDIM = 3000 \nMODEL = 'yolov5s'\nBATCH = 4\nEPOCHS = 15\nOPTMIZER = 'Adam'\n\nPROJECT = 'great-barrier-reef' # w&b in yolov5\nNAME = f'{MODEL}-dim{DIM}-fold{FOLD}' # w&b for yolov5\n\nREMOVE_NOBBOX = True # remove images with no bbox\nROOT_DIR = '/kaggle/input/tensorflow-great-barrier-reef/'\nIMAGE_DIR = '/kaggle/images' # directory to save images\nLABEL_DIR = '/kaggle/labels' # directory to save labels","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:41:32.554739Z","iopub.execute_input":"2023-07-06T20:41:32.555162Z","iopub.status.idle":"2023-07-06T20:41:32.564120Z","shell.execute_reply.started":"2023-07-06T20:41:32.555124Z","shell.execute_reply":"2023-07-06T20:41:32.562791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create Directories","metadata":{}},{"cell_type":"code","source":"!mkdir -p {IMAGE_DIR}\n!mkdir -p {LABEL_DIR}","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:29:25.653032Z","iopub.execute_input":"2023-07-06T20:29:25.653421Z","iopub.status.idle":"2023-07-06T20:29:27.535201Z","shell.execute_reply.started":"2023-07-06T20:29:25.653391Z","shell.execute_reply":"2023-07-06T20:29:27.533924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get Paths","metadata":{}},{"cell_type":"code","source":"# Train Data\ndf = pd.read_csv(f'{ROOT_DIR}/train.csv')\ndf['old_image_path'] = f'{ROOT_DIR}/train_images/video_'+df.video_id.astype(str)+'/'+df.video_frame.astype(str)+'.jpg'\ndf['image_path'] = f'{IMAGE_DIR}/'+df.image_id+'.jpg'\ndf['label_path'] = f'{LABEL_DIR}/'+df.image_id+'.txt'\ndf['annotations'] = df['annotations'].progress_apply(eval)\ndisplay(df.head(2))","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:29:27.537442Z","iopub.execute_input":"2023-07-06T20:29:27.538123Z","iopub.status.idle":"2023-07-06T20:29:28.201111Z","shell.execute_reply.started":"2023-07-06T20:29:27.538083Z","shell.execute_reply":"2023-07-06T20:29:28.200305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Number of BBoxes\n> Nearly 80% images are without any bbox.","metadata":{}},{"cell_type":"code","source":"df['num_bbox'] = df['annotations'].progress_apply(lambda x: len(x))\ndata = (df.num_bbox>0).value_counts(normalize=True)*100\nprint(f\"No BBox: {data[0]:0.2f}% | With BBox: {data[1]:0.2f}%\")","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:29:33.097235Z","iopub.execute_input":"2023-07-06T20:29:33.097593Z","iopub.status.idle":"2023-07-06T20:29:33.194082Z","shell.execute_reply.started":"2023-07-06T20:29:33.097564Z","shell.execute_reply":"2023-07-06T20:29:33.193104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐งน Clean Data\n* In this notebook, we use only **bboxed-images** (`~5k`). We can use all `~23K` images for train but most of them don't have any labels. So it would be easier to carry out experiments using only **bboxed images**.","metadata":{}},{"cell_type":"code","source":"if REMOVE_NOBBOX:\n df = df.query(\"num_bbox>0\")","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:29:33.609883Z","iopub.execute_input":"2023-07-06T20:29:33.611353Z","iopub.status.idle":"2023-07-06T20:29:33.631031Z","shell.execute_reply.started":"2023-07-06T20:29:33.611310Z","shell.execute_reply":"2023-07-06T20:29:33.630087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# โ๏ธ Write Images\n* We need to copy the Images to Current Directory(`/kaggle/working`) as `/kaggle/input` doesn't have **write access** which is needed for **YOLOv5**.\n* We can make this process faster using **Joblib** which uses **Parallel** computing.","metadata":{}},{"cell_type":"code","source":"def make_copy(row):\n shutil.copyfile(row.old_image_path, row.image_path)\n return","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:29:34.221635Z","iopub.execute_input":"2023-07-06T20:29:34.221999Z","iopub.status.idle":"2023-07-06T20:29:34.227574Z","shell.execute_reply.started":"2023-07-06T20:29:34.221971Z","shell.execute_reply":"2023-07-06T20:29:34.226444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_paths = df.old_image_path.tolist()\n_ = Parallel(n_jobs=-1, backend='threading')(delayed(make_copy)(row) for _, row in tqdm(df.iterrows(), total=len(df)))","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:29:34.914611Z","iopub.execute_input":"2023-07-06T20:29:34.914969Z","iopub.status.idle":"2023-07-06T20:30:12.952968Z","shell.execute_reply.started":"2023-07-06T20:29:34.914940Z","shell.execute_reply":"2023-07-06T20:30:12.951964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐จ Helper","metadata":{}},{"cell_type":"code","source":"# check https://github.com/awsaf49/bbox for source code of following utility functions\nfrom bbox.utils import coco2yolo, coco2voc, voc2yolo\nfrom bbox.utils import draw_bboxes, load_image\nfrom bbox.utils import clip_bbox, str2annot, annot2str\n\ndef get_bbox(annots):\n bboxes = [list(annot.values()) for annot in annots]\n return bboxes\n\ndef get_imgsize(row):\n row['width'], row['height'] = imagesize.get(row['image_path'])\n return row\n\nnp.random.seed(32)\ncolors = [(np.random.randint(255), np.random.randint(255), np.random.randint(255))\\\n for idx in range(1)]","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-07-06T20:30:12.955210Z","iopub.execute_input":"2023-07-06T20:30:12.955571Z","iopub.status.idle":"2023-07-06T20:30:13.549310Z","shell.execute_reply.started":"2023-07-06T20:30:12.955536Z","shell.execute_reply":"2023-07-06T20:30:13.547775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create BBox","metadata":{}},{"cell_type":"code","source":"df['bboxes'] = df.annotations.progress_apply(get_bbox)\ndf.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:30:13.550857Z","iopub.execute_input":"2023-07-06T20:30:13.551537Z","iopub.status.idle":"2023-07-06T20:30:13.609655Z","shell.execute_reply.started":"2023-07-06T20:30:13.551501Z","shell.execute_reply":"2023-07-06T20:30:13.608609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get Image-Size\n> All Images have same dimension, [Width, Height] = `[1280, 720]`","metadata":{}},{"cell_type":"code","source":"df['width'] = 1280\ndf['height'] = 720\ndisplay(df.head(2))","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:30:13.612616Z","iopub.execute_input":"2023-07-06T20:30:13.612947Z","iopub.status.idle":"2023-07-06T20:30:13.634309Z","shell.execute_reply.started":"2023-07-06T20:30:13.612916Z","shell.execute_reply":"2023-07-06T20:30:13.633455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ท๏ธ Create Labels\nWe need to export our labels to **YOLO** format, with one `*.txt` file per image (if no objects in image, no `*.txt` file is required). The *.txt file specifications are:\n\n* One row per object\n* Each row is class `[x_center, y_center, width, height]` format.\n* Box coordinates must be in **normalized** `xywh` format (from `0 - 1`). If your boxes are in pixels, divide `x_center` and `width` by `image width`, and `y_center` and `height` by `image height`.\n* Class numbers are **zero-indexed** (start from `0`).\n\n> Competition bbox format is **COCO** hence `[x_min, y_min, width, height]`. So, we need to convert form **COCO** to **YOLO** format.\n","metadata":{}},{"cell_type":"code","source":"cnt = 0\nall_bboxes = []\nbboxes_info = []\nfor row_idx in tqdm(range(df.shape[0])):\n row = df.iloc[row_idx]\n image_height = row.height\n image_width = row.width\n bboxes_coco = np.array(row.bboxes).astype(np.float32).copy()\n num_bbox = len(bboxes_coco)\n names = ['cots']*num_bbox\n labels = np.array([0]*num_bbox)[..., None].astype(str)\n ## Create Annotation(YOLO)\n with open(row.label_path, 'w') as f:\n if num_bbox<1:\n annot = ''\n f.write(annot)\n cnt+=1\n continue\n bboxes_voc = coco2voc(bboxes_coco, image_height, image_width)\n bboxes_voc = clip_bbox(bboxes_voc, image_height, image_width)\n bboxes_yolo = voc2yolo(bboxes_voc, image_height, image_width).astype(str)\n all_bboxes.extend(bboxes_yolo.astype(float))\n bboxes_info.extend([[row.image_id, row.video_id, row.sequence]]*len(bboxes_yolo))\n annots = np.concatenate([labels, bboxes_yolo], axis=1)\n string = annot2str(annots)\n f.write(string)\nprint('Missing:',cnt)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-07-06T20:30:13.635546Z","iopub.execute_input":"2023-07-06T20:30:13.635873Z","iopub.status.idle":"2023-07-06T20:30:18.447797Z","shell.execute_reply.started":"2023-07-06T20:30:13.635848Z","shell.execute_reply":"2023-07-06T20:30:18.446763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ Create Folds\n> Number of samples aren't same in each fold which can create large variance in **Cross-Validation**.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GroupKFold\nkf = GroupKFold(n_splits = 3)\ndf = df.reset_index(drop=True)\ndf['fold'] = -1\nfor fold, (train_idx, val_idx) in enumerate(kf.split(df, groups=df.video_id.tolist())):\n df.loc[val_idx, 'fold'] = fold\ndisplay(df.fold.value_counts())","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:30:18.449570Z","iopub.execute_input":"2023-07-06T20:30:18.450247Z","iopub.status.idle":"2023-07-06T20:30:19.122735Z","shell.execute_reply.started":"2023-07-06T20:30:18.450211Z","shell.execute_reply":"2023-07-06T20:30:19.121759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# โญ BBox Distribution","metadata":{}},{"cell_type":"code","source":"bbox_df = pd.DataFrame(np.concatenate([bboxes_info, all_bboxes], axis=1),\n columns=['image_id','video_id','sequence',\n 'xmid','ymid','w','h'])\nbbox_df[['xmid','ymid','w','h']] = bbox_df[['xmid','ymid','w','h']].astype(float)\nbbox_df['area'] = bbox_df.w * bbox_df.h * 1280 * 720\nbbox_df = bbox_df.merge(df[['image_id','fold']], on='image_id', how='left')\nbbox_df.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:30:19.124401Z","iopub.execute_input":"2023-07-06T20:30:19.124783Z","iopub.status.idle":"2023-07-06T20:30:20.560984Z","shell.execute_reply.started":"2023-07-06T20:30:19.124748Z","shell.execute_reply":"2023-07-06T20:30:20.560013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `x_center` Vs `y_center`","metadata":{}},{"cell_type":"code","source":"from scipy.stats import gaussian_kde\n\nall_bboxes = np.array(all_bboxes)\n\nx_val = all_bboxes[...,0]\ny_val = all_bboxes[...,1]\n\n# Calculate the point density\nxy = np.vstack([x_val,y_val])\nz = gaussian_kde(xy)(xy)\n\nfig, ax = plt.subplots(figsize = (10, 10))\n# ax.axis('off')\nax.scatter(x_val, y_val, c=z, s=100, cmap='viridis')\n# ax.set_xlabel('x_mid')\n# ax.set_ylabel('y_mid')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-06T20:30:20.563092Z","iopub.execute_input":"2023-07-06T20:30:20.563742Z","iopub.status.idle":"2023-07-06T20:30:23.159629Z","shell.execute_reply.started":"2023-07-06T20:30:20.563707Z","shell.execute_reply":"2023-07-06T20:30:23.158358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `width` Vs `height`","metadata":{}},{"cell_type":"code","source":"x_val = all_bboxes[...,2]\ny_val = all_bboxes[...,3]\n\n# Calculate the point density\nxy = np.vstack([x_val,y_val])\nz = gaussian_kde(xy)(xy)\n\nfig, ax = plt.subplots(figsize = (10, 10))\n# ax.axis('off')\nax.scatter(x_val, y_val, c=z, s=100, cmap='viridis')\n# ax.set_xlabel('bbox_width')\n# ax.set_ylabel('bbox_height')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-06T20:30:23.160940Z","iopub.execute_input":"2023-07-06T20:30:23.161382Z","iopub.status.idle":"2023-07-06T20:30:26.625906Z","shell.execute_reply.started":"2023-07-06T20:30:23.161338Z","shell.execute_reply":"2023-07-06T20:30:26.624976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Area","metadata":{}},{"cell_type":"code","source":"import matplotlib as mpl\nimport seaborn as sns\n\nf, ax = plt.subplots(figsize=(12, 6))\nsns.despine(f)\n\nsns.histplot(\n bbox_df,\n x=\"area\", hue=\"fold\",\n multiple=\"stack\",\n palette=\"viridis\",\n edgecolor=\".3\",\n linewidth=.5,\n log_scale=True,\n)\nax.xaxis.set_major_formatter(mpl.ticker.ScalarFormatter())\nax.set_xticks([500, 1000, 2000, 5000, 10000]);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-06T20:30:26.629499Z","iopub.execute_input":"2023-07-06T20:30:26.630186Z","iopub.status.idle":"2023-07-06T20:30:27.850089Z","shell.execute_reply.started":"2023-07-06T20:30:26.630151Z","shell.execute_reply":"2023-07-06T20:30:27.849045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ Visualization","metadata":{}},{"cell_type":"code","source":"df2 = df[(df.num_bbox>0)].sample(100) # takes samples with bbox\ny = 3; x = 2\nplt.figure(figsize=(12.8*x, 7.2*y))\nfor idx in range(x*y):\n row = df2.iloc[idx]\n img = load_image(row.image_path)\n image_height = row.height\n image_width = row.width\n with open(row.label_path) as f:\n annot = str2annot(f.read())\n bboxes_yolo = annot[...,1:]\n labels = annot[..., 0].astype(int).tolist()\n names = ['cots']*len(bboxes_yolo)\n plt.subplot(y, x, idx+1)\n plt.imshow(draw_bboxes(img = img,\n bboxes = bboxes_yolo, \n classes = names,\n class_ids = labels,\n class_name = True, \n colors = colors, \n bbox_format = 'yolo',\n line_thickness = 2))\n plt.axis('OFF')\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-06T20:30:27.851477Z","iopub.execute_input":"2023-07-06T20:30:27.851908Z","iopub.status.idle":"2023-07-06T20:30:32.457226Z","shell.execute_reply.started":"2023-07-06T20:30:27.851873Z","shell.execute_reply":"2023-07-06T20:30:32.452171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ Dataset","metadata":{}},{"cell_type":"code","source":"train_files = []\nval_files = []\ntrain_df = df.query(\"fold!=@FOLD\")\nvalid_df = df.query(\"fold==@FOLD\")\ntrain_files += list(train_df.image_path.unique())\nval_files += list(valid_df.image_path.unique())\nlen(train_files), len(val_files)","metadata":{"execution":{"iopub.status.busy":"2023-07-06T20:30:32.458973Z","iopub.execute_input":"2023-07-06T20:30:32.459731Z","iopub.status.idle":"2023-07-06T20:30:32.490256Z","shell.execute_reply.started":"2023-07-06T20:30:32.459685Z","shell.execute_reply":"2023-07-06T20:30:32.489473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# โ๏ธ Configuration\nThe dataset config file requires\n1. The dataset root directory path and relative paths to `train / val / test` image directories (or *.txt files with image paths)\n2. The number of classes `nc` and \n3. A list of class `names`:`['cots']`","metadata":{}},{"cell_type":"code","source":"import yaml\n\ncwd = '/kaggle/working/'\n\nwith open(os.path.join( cwd , 'train.txt'), 'w') as f:\n for path in train_df.image_path.tolist():\n f.write(path+'\\n')\n \nwith open(os.path.join(cwd , 'val.txt'), 'w') as f:\n for path in valid_df.image_path.tolist():\n f.write(path+'\\n')\n\ndata = dict(\n path = '/kaggle/working',\n train = os.path.join( cwd , 'train.txt') ,\n val = os.path.join( cwd , 'val.txt' ),\n nc = 1,\n names = ['cots'],\n )\n\nwith open(os.path.join( cwd , 'gbr.yaml'), 'w') as outfile:\n yaml.dump(data, outfile, default_flow_style=False)\n\nf = open(os.path.join( cwd , 'gbr.yaml'), 'r')\nprint('\\nyaml:')\nprint(f.read())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-06T20:41:45.755523Z","iopub.execute_input":"2023-07-06T20:41:45.756330Z","iopub.status.idle":"2023-07-06T20:41:45.771879Z","shell.execute_reply.started":"2023-07-06T20:41:45.756295Z","shell.execute_reply":"2023-07-06T20:41:45.770684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%writefile /kaggle/working/hyp.yaml\nlr0: 0.01 # initial learning rate (SGD=1E-2, Adam=1E-3)\nlrf: 0.1 # final OneCycleLR learning rate (lr0 * lrf)\nmomentum: 0.937 # SGD momentum/Adam beta1\nweight_decay: 0.0005 # optimizer weight decay 5e-4\nwarmup_epochs: 3.0 # warmup epochs (fractions ok)\nwarmup_momentum: 0.8 # warmup initial momentum\nwarmup_bias_lr: 0.1 # warmup initial bias lr\nbox: 0.05 # box loss gain\ncls: 0.5 # cls loss gain\ncls_pw: 1.0 # cls BCELoss positive_weight\nobj: 1.0 # obj loss gain (scale with pixels)\nobj_pw: 1.0 # obj BCELoss positive_weight\niou_t: 0.20 # IoU training threshold\nanchor_t: 4.0 # anchor-multiple threshold\n# anchors: 3 # anchors per output layer (0 to ignore)\nfl_gamma: 0.0 # focal loss gamma (efficientDet default gamma=1.5)\nhsv_h: 0.015 # image HSV-Hue augmentation (fraction)\nhsv_s: 0.7 # image HSV-Saturation augmentation (fraction)\nhsv_v: 0.4 # image HSV-Value augmentation (fraction)\ndegrees: 0.0 # image rotation (+/- deg)\ntranslate: 0.10 # image translation (+/- fraction)\nscale: 0.5 # image scale (+/- gain)\nshear: 0.0 # image shear (+/- deg)\nperspective: 0.0 # image perspective (+/- fraction), range 0-0.001\nflipud: 0.5 # image flip up-down (probability)\nfliplr: 0.5 # image flip left-right (probability)\nmosaic: 0.5 # image mosaic (probability)\nmixup: 0.5 # image mixup (probability)\ncopy_paste: 0.0 # segment copy-paste (probability)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-06T20:30:32.508024Z","iopub.execute_input":"2023-07-06T20:30:32.508678Z","iopub.status.idle":"2023-07-06T20:30:32.516023Z","shell.execute_reply.started":"2023-07-06T20:30:32.508646Z","shell.execute_reply":"2023-07-06T20:30:32.515123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ฆ YOLOv5 with ๐ช WandB Integration\n\n\n\n
\n ![](\"https://raw.githubusercontent.com/ultralytics/assets/master/yolov5/v70/splash.png\")
\n\n
![](\"https://www.pngall.com/wp-content/uploads/2018/04/Under-Construction-PNG-File.png\")
","metadata":{}}]}