{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# NFL Player Contact Detection - DAMO-YOLO for object detection on Person and Helmet","metadata":{}},{"cell_type":"markdown","source":"## Overview\nIn this notebook, We use DAMO-YOLO repository for Person and Helmet Detection. In order to make it more convenient for you to use, we provide the trained model and the inference code in **Detect Person and Helmet** section. \n\nMeanwhile, if you have more training data or finetuning ideas, you can follow **Customize Training** section to customize model training.\n\nHope DAMO-YOLO can help you complete NFL Player Contact Detection competition and achieve good results.\n\n
","metadata":{}},{"cell_type":"markdown","source":"# Usage of DAMO-YOLO","metadata":{}},{"cell_type":"markdown","source":"## Install","metadata":{}},{"cell_type":"code","source":"git clone https://github.com/tinyvision/damo-yolo.git\ncd DAMO-YOLO/\nconda create -n DAMO-YOLO python=3.7 -y\nconda activate DAMO-YOLO\nconda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=10.2 -c pytorch\npip install -r requirements.txt\nexport PYTHONPATH=$PWD:$PYTHONPATH","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## detect Person and Helmet","metadata":{}},{"cell_type":"code","source":"# download pretrained Person Detection models\nwget 'http://idstcv.oss-cn-zhangjiakou.aliyuncs.com/DAMO-YOLO/Kaggle_data/damoyolo_tinynasL25_S_Person.pth'\n# detect Person\npython -m pdb tools/demo.py -f ./configs/damoyolo_tinynasL25_S.py \\\n --engine damoyolo_tinynasL25_S_Person.pth \\\n --engine_type torch \n --conf 0.4 --infer_size 640 640 --device cuda --path 58102_002798_Endzone.mp4\n \n\n# download pretrained Helmet Detection models\nwget 'http://idstcv.oss-cn-zhangjiakou.aliyuncs.com/DAMO-YOLO/Kaggle_data/damoyolo_tinynasL25_S_Helmet.pth'\n# detect Helmet\npython -m pdb tools/demo.py -f ./configs/damoyolo_tinynasL25_S_helmet.py \\\n --engine damoyolo_tinynasL25_S_Helmet.pth \\\n --engine_type torch \n --conf 0.4 --infer_size 640 640 --device cuda --path 58102_002798_Endzone.mp4","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Detection Results on Person and Helmet Detection","metadata":{}},{"cell_type":"markdown","source":"
\n \n \n \n
","metadata":{}},{"cell_type":"markdown","source":"## Customized Training ","metadata":{}},{"cell_type":"markdown","source":"If you want to try training the DAMO-YOLO yourself to further improve the accuracy of the model, follow these steps.","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"### Transfer data into COCO Format","metadata":{}},{"cell_type":"code","source":"import os\nimport datetime\nimport random\nimport numpy as np\nfrom pathlib import Path\nimport datetime\nimport pandas as pd\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\nimport cv2\nimport json\nimport matplotlib.pyplot as plt\nfrom IPython.core.display import Video, display\nimport subprocess\nimport gc\nimport shutil\n\ndef create_ann_file(df, category_id):\n\n now = datetime.datetime.now()\n\n data = dict(\n info=dict(\n description='NFL-Helmet-Assignment',\n url=None,\n version=None,\n year=now.year,\n contributor=None,\n date_created=now.strftime('%Y-%m-%d %H:%M:%S.%f'),\n ),\n licenses=[dict(\n url=None,\n id=0,\n name=None,\n )],\n images=[\n # license, url, file_name, height, width, date_captured, id\n ],\n type='instances',\n annotations=[\n # segmentation, area, iscrowd, image_id, bbox, category_id, id\n ],\n categories=[\n # supercategory, id, name\n ],\n )\n\n class_name_to_id = {}\n # labels = [\"__ignore__\",\n # 'Helmet',\n # 'Helmet-Blurred',\n # 'Helmet-Difficult',\n # 'Helmet-Sideline',\n # 'Helmet-Partial']\n labels = [\"__ignore__\",\n 'Helmet']\n\n\n for i, each_label in enumerate(labels):\n class_id = i - 1 # starts with -1\n class_name = each_label\n if class_id == -1:\n assert class_name == '__ignore__'\n continue\n class_name_to_id[class_name] = class_id\n data['categories'].append(dict(\n supercategory=None,\n id=class_id,\n name=class_name,\n ))\n\n box_id = 0\n for i, image in tqdm(enumerate(os.listdir(TRAIN_PATH))):\n\n img = cv2.imread(TRAIN_PATH+'/'+image)\n height, width, _ = img.shape\n\n data['images'].append({\n 'license':0,\n 'url': None,\n 'file_name': image,\n 'height': height,\n 'width': width,\n 'date_camputured': None,\n 'id': i\n })\n\n df_temp = df[df.image == image]\n for index, row in df_temp.iterrows():\n\n area = round(row.width*row.height, 1)\n bbox =[row.left, row.top, row.width, row.height]\n\n data['annotations'].append({\n 'id': box_id,\n 'image_id': i,\n 'category_id': category_id[row.label],\n 'area': area,\n 'bbox':bbox,\n 'iscrowd':0\n })\n box_id+=1\n\n return data\n\nTRAIN_PATH = '/home/yiqi.jyq/data/kaggle_data/nfl-health-and-safety-helmet-assignment/images'\nextra_df = pd.read_csv('/home/yiqi.jyq/data/kaggle_data/nfl-health-and-safety-helmet-assignment/image_labels.csv')\n\n# category_id = {'Helmet':0, 'Helmet-Blurred':1,\n# 'Helmet-Difficult':2, 'Helmet-Sideline':3,\n# 'Helmet-Partial':4}\n# all helmet type to helmet\ncategory_id = {'Helmet':0, 'Helmet-Blurred':0,\n 'Helmet-Difficult':0, 'Helmet-Sideline':0,\n 'Helmet-Partial':0}\n\n\ndf_train, df_val = train_test_split(extra_df, test_size=0.2, random_state=42)\nann_file_train = create_ann_file(df_train, category_id)\nann_file_val = create_ann_file(df_val, category_id)\n\n# save data sets\nos.makedirs('/home/yiqi.jyq/data/kaggle_data/coco-format-helmet-assignment', exist_ok=True)\n\nwith open('/home/yiqi.jyq/data/kaggle_data/coco-format-helmet-assignment/ann_file_train.json', 'w') as f:\n json.dump(ann_file_train, f, indent=4)\n\nwith open('/home/yiqi.jyq/data/kaggle_data/coco-format-helmet-assignment/ann_file_val.json', 'w') as f:\n json.dump(ann_file_val, f, indent=4)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Correlate training data to the DAMO-YOLO","metadata":{}},{"cell_type":"code","source":"ln -s /home/yiqi.jyq/data/kaggle_data/coco-format-helmet-assignment/ datasets/coco-helmet","metadata":{"execution":{"iopub.status.busy":"2023-01-25T09:27:57.017192Z","iopub.execute_input":"2023-01-25T09:27:57.01801Z","iopub.status.idle":"2023-01-25T09:27:57.024431Z","shell.execute_reply.started":"2023-01-25T09:27:57.017972Z","shell.execute_reply":"2023-01-25T09:27:57.023659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Edit damo/config/paths_catalog.py file and add path to our own data","metadata":{}},{"cell_type":"code","source":"class DatasetCatalog(object):\n DATA_DIR = 'datasets'\n DATASETS = {\n 'coco_2017_train': {\n 'img_dir': 'coco/train2017',\n 'ann_file': 'coco/annotations/instances_train2017.json'\n },\n 'coco_2017_val': {\n 'img_dir': 'coco/val2017',\n 'ann_file': 'coco/annotations/instances_val2017.json'\n },\n 'coco_2017_test_dev': {\n 'img_dir': 'coco/test2017',\n 'ann_file': 'coco/annotations/image_info_test-dev2017.json'\n },\n 'coco_helmet_train': {\n 'img_dir': 'coco_helmet/images',\n 'ann_file': 'coco_helmet/ann_file_train.json'\n },\n 'coco_helmet_val': {\n 'img_dir': 'coco_helmet/images',\n 'ann_file': 'coco_helmet/ann_file_val.json'\n },\n\n }\n\n @staticmethod\n def get(name):\n if 'coco' in name:\n data_dir = DatasetCatalog.DATA_DIR\n attrs = DatasetCatalog.DATASETS[name]\n args = dict(\n root=os.path.join(data_dir, attrs['img_dir']),\n ann_file=os.path.join(data_dir, attrs['ann_file']),\n )\n return dict(\n factory='COCODataset',\n args=args,\n )\n else:\n raise RuntimeError('Only support coco format now!')\n return None","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create training config file","metadata":{}},{"cell_type":"markdown","source":"With the data ready, we can modify the training profile as needed.\n\nFor finetune, you'll need to download the trained model from the DAMO-YOLO website. Here we are going to use DAMO-YOLO-S as the pretrained model for finetune. Directly download the damoyolo_tinynasL20_S.pth stored under the DAMO-YOLO directory.\n\nOpen the DAMO-YOLO-S training config file configs/damoyolo_tinynasL25_S_helmet.py. We need to add the pretrained model to the config file. Meanwhile, the number of categories should be modified, for helmet detection, we change it to 1 as there are only one categories: ['Helmet'].","metadata":{}},{"cell_type":"code","source":"#!/usr/bin/env python3\n\nimport os\n\nfrom damo.config import Config as MyConfig\n\n\nclass Config(MyConfig):\n def __init__(self):\n super(Config, self).__init__()\n\n self.miscs.exp_name = os.path.split(\n os.path.realpath(__file__))[1].split('.')[0]\n self.miscs.eval_interval_epochs = 10\n self.miscs.ckpt_interval_epochs = 10\n # optimizer\n self.train.batch_size = 256\n self.train.base_lr_per_img = 0.01 / 64\n self.train.min_lr_ratio = 0.05\n self.train.weight_decay = 5e-4\n self.train.momentum = 0.9\n self.train.no_aug_epochs = 16\n self.train.warmup_epochs = 5\n\n # augment\n self.train.augment.transform.image_max_range = (640, 640)\n self.train.augment.mosaic_mixup.mixup_prob = 0.15\n self.train.augment.mosaic_mixup.degrees = 10.0\n self.train.augment.mosaic_mixup.translate = 0.2\n self.train.augment.mosaic_mixup.shear = 2.0\n self.train.augment.mosaic_mixup.mosaic_scale = (0.1, 2.0)\n\n # finetune path\n self.train.finetune_path = './damoyolo_tinynasL25_S.pth'\n\n self.dataset.train_ann = ('coco_single_helmet_train', )\n self.dataset.val_ann = ('coco_single_helmet_val', )\n\n # backbone\n structure = self.read_structure(\n './damo/base_models/backbones/nas_backbones/tinynas_L25_k1kx.txt')\n TinyNAS = {\n 'name': 'TinyNAS_res',\n 'net_structure_str': structure,\n 'out_indices': (2, 4, 5),\n 'with_spp': True,\n 'use_focus': True,\n 'act': 'relu',\n 'reparam': True,\n }\n\n self.model.backbone = TinyNAS\n\n GiraffeNeckV2 = {\n 'name': 'GiraffeNeckV2',\n 'depth': 1.0,\n 'hidden_ratio': 0.75,\n 'in_channels': [128, 256, 512],\n 'out_channels': [128, 256, 512],\n 'act': 'relu',\n 'spp': False,\n 'block_name': 'BasicBlock_3x3_Reverse',\n }\n\n self.model.neck = GiraffeNeckV2\n\n ZeroHead = {\n 'name': 'ZeroHead',\n 'num_classes': 1,\n 'in_channels': [128, 256, 512],\n 'stacked_convs': 0,\n 'reg_max': 16,\n 'act': 'silu',\n 'nms_conf_thre': 0.4,\n 'nms_iou_thre': 0.7\n }\n self.model.head = ZeroHead\n\n self.dataset.class_names = ['Helmet']","metadata":{"execution":{"iopub.status.busy":"2023-01-25T09:27:57.025593Z","iopub.execute_input":"2023-01-25T09:27:57.026154Z","iopub.status.idle":"2023-01-25T09:27:57.036763Z","shell.execute_reply.started":"2023-01-25T09:27:57.026117Z","shell.execute_reply":"2023-01-25T09:27:57.03599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Training","metadata":{}},{"cell_type":"code","source":"python -m torch.distributed.launch --nproc_per_node=8 tools/train.py -f configs/damoyolo_tinynasL25_S_single_helmet.py","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# What can DAMO-YOLO do for You\n- DAMO-YOLO provides precise inference on Person and Helmet for subsequent NFL Player-Contact-Detection.\n- With DAMO-YOLO, You can train your model by yourself so as to achieve higher accuracy.\n- Detection results could also be used for the isolation of the pair of players and helmets.\n- DAMO-YOLO provides a large scale range of models for the needs of different latency-precision trade-offs.","metadata":{}},{"cell_type":"markdown","source":"# References\n- https://github.com/tinyvision/DAMO-YOLO\n- https://www.kaggle.com/code/robikscube/nfl-player-contact-detection-getting-started","metadata":{}}]}