{"cells":[{"metadata":{"_uuid":"cc075cd8b1f3912c3e65651871a4ce263f772750"},"cell_type":"markdown","source":"## Contents\n### YOLO v3 for image detection\n0. <a href='#0.-Introduction'>Introduction</a>\n1. <a href='#1.-Clone-and-Build-YOLOv3'>Clone and Build YOLOv3</a>\n2. <a href='#2.-Data-Migration-for-YOLOv3'>Data Migration</a>\n3. <a href='#3.-Prepare-Configuration-Files-for-Using-YOLOv3'>Prepare Configuration Files for training</a>\n4. <a href='#4.-Training-YOLOv3'>Training model</a>\n5. <a href='#5.-How-to-use-trainined-YOLOv3-for-test-images-(command-line)'>How to use trained model for test images (command line)</a>\n6. <a href='#6.-Generate-Submission-Files-with-YOLOv3-Python-Wrapper'>Generate Submission Files Using YOLOv3 Python Wrapper</a>\n7. <a href='#7.-Future-works-&-Etc'>Future works & Etc</a>"},{"metadata":{"_uuid":"c828e6dd0b8dc2e80806ef2d3f4137ae4a3ea670"},"cell_type":"markdown","source":"## 0. Introduction\n* I'll introduce super easy and quick way to train [YOLOv3](https://pjreddie.com/darknet/yolo/) on RSNA and to generate submission file (to be honest, not super easy ...!).\n\n\n* The purpose of this competition is 'object detection'. Generally, object detection algorithms with deep learning take a long time to train model and require a lot of gpu resources. Most individual participants use one or two gpu (... or zero). Therefore, there is a need for **algorithms that works quickly with less gpu resources.**\n\n\n* I tried to use Mask R-CNN, UNet, Fast R-CNN and FCN algorithms, But eventually switched to YOLOv3.\n\n\n* In comparison to YOLOv3, Other algorithms(Mask R-CNN, UNet, FCN, ...) which contain instance/sementic segmentation tasks are very slow, and require more gpu resources, redundant parameter tunning and post-processes. Therefore, if you try to use these algorithms, you may experience difficulties in terms of training time and gpu resources. (Please see [YOLOv3 paper](https://pjreddie.com/media/files/papers/YOLOv3.pdf) for details)\n\n\n* In addition, **YOLOv3 was able to obtain high score (LB: 0.141) without additional processes(data augmentation, parameter tunning, etc...)** compared to other algorithms. So I think YOLOv3 has sufficient potential for this competition.\n\n\n* In this notebook, I'll introduce how to simply apply YOLOv3 on RSNA data. I hope this notebook would be helpful for everyone."},{"metadata":{"trusted":true,"_uuid":"ce28f29df7b9f64725998f8133be50068c53cb69"},"cell_type":"code","source":"import math\nimport os\nimport shutil\nimport sys\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport glob\nimport pydicom\nimport cv2\nfrom sklearn.model_selection import train_test_split\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c4b33affdc049734daa58852c4ed06e37c344063"},"cell_type":"code","source":"random_stat = 123\nnp.random.seed(random_stat)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97d22b2e1c828e655e7df04474169cd0d7a5a031"},"cell_type":"markdown","source":"## 1. Clone and Build YOLOv3"},{"metadata":{"trusted":true,"_uuid":"558964006de8b4e454d76c91a5d0a605e397a93d"},"cell_type":"code","source":"!git clone https://github.com/pjreddie/darknet.git\n\n# Build gpu version darknet\n!cd darknet && sed '1 s/^.*$/GPU=1/; 2 s/^.*$/CUDNN=1/' -i Makefile\n\n# -j <The # of cpu cores to use>. Chang 999 to fit your environment. Actually i used '-j 50'.\n!cd darknet && make -j 50 -s\n!cp darknet/darknet darknet_gpu","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f77f087a15c73fae7a710fd560244b8f7884197"},"cell_type":"markdown","source":"## 2. Data Migration for YOLOv3\nIt might take a while."},{"metadata":{"_uuid":"5702330ac23995099c3f184ef0424731d91a303c"},"cell_type":"markdown","source":"### 2.0. Make subdirectories"},{"metadata":{"trusted":true,"_uuid":"55f5d67ac085afcc445db23227ba60748eb269fa"},"cell_type":"code","source":"DATA_DIR = \"../input\"\n\ntrain_dcm_dir = os.path.join(DATA_DIR, \"stage_2_train_images\")\ntest_dcm_dir = os.path.join(DATA_DIR, \"stage_2_test_images\")\n\nimg_dir = os.path.join(os.getcwd(), \"images\")  # .jpg\nlabel_dir = os.path.join(os.getcwd(), \"labels\")  # .txt\nmetadata_dir = os.path.join(os.getcwd(), \"metadata\") # .txt\n\n# YOLOv3 config file directory\ncfg_dir = os.path.join(os.getcwd(), \"cfg\")\n# YOLOv3 training checkpoints will be saved here\nbackup_dir = os.path.join(os.getcwd(), \"backup\")\n\nfor directory in [img_dir, label_dir, metadata_dir, cfg_dir, backup_dir]:\n    if os.path.isdir(directory):\n        continue\n    os.mkdir(directory)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0b2173fa0ff6a1bfdf304310466d01052920fe6"},"cell_type":"code","source":"!ls -shtl","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba6a35b97d0a42060e6fdd68a6da23454aa40568"},"cell_type":"markdown","source":"### 2.1. Load stage_1_train_labels.csv"},{"metadata":{"trusted":true,"_uuid":"e242b29961c22433703e442a84572783b9459de9"},"cell_type":"code","source":"annots = pd.read_csv(os.path.join(DATA_DIR, \"stage_2_train_labels.csv\"))\nannots.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b26d929bb119deca558ac983a5ce4e95fe9b7cc"},"cell_type":"markdown","source":"### 2.2. Generate images and labels for training YOLOv3\n* YOLOv3 needs .txt file for each image, which contains ground truth object in the image that looks like:\n```\n<object-class_1> <x_1> <y_1> <width_1> <height_1>\n<object-class_2> <x_2> <y_2> <width_2> <height_2>\n```\n* <object-class\\>: Since RSNA task is binary classification basically, <object-class\\> is 0.\n* <x\\>, <y\\>: Those are float values of bbox center coordinate, divided by image width and height respectively.\n* <w\\>, <h\\>: Those are width and height of bbox, divided by image width and height respectively.\n\n* So it is different from the format of label data provided by kaggle. We should change it."},{"metadata":{"trusted":true,"_uuid":"5204d3b142e17927c06e9e992dd4e4f4a3d0c8ab"},"cell_type":"code","source":"def save_img_from_dcm(dcm_dir, img_dir, patient_id):\n    img_fp = os.path.join(img_dir, \"{}.jpg\".format(patient_id))\n    if os.path.exists(img_fp):\n        return\n    dcm_fp = os.path.join(dcm_dir, \"{}.dcm\".format(patient_id))\n    img_1ch = pydicom.read_file(dcm_fp).pixel_array\n    img_3ch = np.stack([img_1ch]*3, -1)\n\n    img_fp = os.path.join(img_dir, \"{}.jpg\".format(patient_id))\n    cv2.imwrite(img_fp, img_3ch)\n    \ndef save_label_from_dcm(label_dir, patient_id, row=None):\n    # rsna defualt image size\n    img_size = 1024\n    label_fp = os.path.join(label_dir, \"{}.txt\".format(patient_id))\n    \n    f = open(label_fp, \"a\")\n    if row is None:\n        f.close()\n        return\n\n    top_left_x = row[1]\n    top_left_y = row[2]\n    w = row[3]\n    h = row[4]\n    \n    # 'r' means relative. 'c' means center.\n    rx = top_left_x/img_size\n    ry = top_left_y/img_size\n    rw = w/img_size\n    rh = h/img_size\n    rcx = rx+rw/2\n    rcy = ry+rh/2\n    \n    line = \"{} {} {} {} {}\\n\".format(0, rcx, rcy, rw, rh)\n    \n    f.write(line)\n    f.close()\n        \ndef save_yolov3_data_from_rsna(dcm_dir, img_dir, label_dir, annots):\n    for row in tqdm(annots.values):\n        patient_id = row[0]\n\n        img_fp = os.path.join(img_dir, \"{}.jpg\".format(patient_id))\n        if os.path.exists(img_fp):\n            save_label_from_dcm(label_dir, patient_id, row)\n            continue\n\n        target = row[5]\n        # Since kaggle kernel have samll volume (5GB ?), I didn't contain files with no bbox here.\n        if target == 0:\n            continue\n        save_label_from_dcm(label_dir, patient_id, row)\n        save_img_from_dcm(dcm_dir, img_dir, patient_id)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e907f30b11bf6956884061138d615bee60c2a15"},"cell_type":"code","source":"save_yolov3_data_from_rsna(train_dcm_dir, img_dir, label_dir, annots)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19d2c4ab8448c0572ac57b49eff52c2230c9f079"},"cell_type":"code","source":"!du -sh images labels","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"17868a4d327c2e22adae76cc7ce163c5546e1743"},"cell_type":"markdown","source":"### 2.3. Plot a sample train image and label"},{"metadata":{"trusted":true,"_uuid":"8ebf845a4f3b51555991292d10ddc272b226bf7c"},"cell_type":"code","source":"ex_patient_id = annots[annots.Target == 1].patientId.values[10]\nex_img_path = os.path.join(img_dir, \"{}.jpg\".format(ex_patient_id))\nex_label_path = os.path.join(label_dir, \"{}.txt\".format(ex_patient_id))\nprint(ex_patient_id)\n\nplt.imshow(cv2.imread(ex_img_path))\n\nimg_size = 1014\nwith open(ex_label_path, \"r\") as f:\n    for line in f:\n        print(line)\n        class_id, rcx, rcy, rw, rh = list(map(float, line.strip().split()))\n        x = (rcx-rw/2)*img_size\n        y = (rcy-rh/2)*img_size\n        w = rw*img_size\n        h = rh*img_size\n        plt.plot([x, x, x+w, x+w, x], [y, y+h, y+h, y, y])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42551c1e11208ee88f9b93bea40c23dcad295b8a"},"cell_type":"markdown","source":"### 2.4. Generate train/val file path list (.txt)\n* We should give the list of image paths to YOLO. two seperate list textfiles for training images and validation images."},{"metadata":{"trusted":true,"_uuid":"d62f98b287e73a66c234c4c9f9e41503fc0d4bca"},"cell_type":"code","source":"def write_train_list(metadata_dir, img_dir, name, series):\n    list_fp = os.path.join(metadata_dir, name)\n    with open(list_fp, \"w\") as f:\n        for patient_id in series:\n            line = \"{}\\n\".format(os.path.join(img_dir, \"{}.jpg\".format(patient_id)))\n            f.write(line)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c3982b42711fe4a20600b90ed3d123384da7096"},"cell_type":"code","source":"# Following lines do not contain data with no bbox\npatient_id_series = annots[annots.Target == 1].patientId.drop_duplicates()\n\ntr_series, val_series = train_test_split(patient_id_series, test_size=0.1, random_state=random_stat)\nprint(\"The # of train set: {}, The # of validation set: {}\".format(tr_series.shape[0], val_series.shape[0]))\n\n# train image path list\nwrite_train_list(metadata_dir, img_dir, \"tr_list.txt\", tr_series)\n# validation image path list\nwrite_train_list(metadata_dir, img_dir, \"val_list.txt\", val_series)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"482fc4b015279312504c910b6c910a5d58960159"},"cell_type":"markdown","source":"### 2.5. Create test image and labels for YOLOv3"},{"metadata":{"trusted":true,"_uuid":"7394207f5f23778aee12d5f4dadae632b340ce62"},"cell_type":"code","source":"def save_yolov3_test_data(test_dcm_dir, img_dir, metadata_dir, name, series):\n    list_fp = os.path.join(metadata_dir, name)\n    with open(list_fp, \"w\") as f:\n        for patient_id in series:\n            save_img_from_dcm(test_dcm_dir, img_dir, patient_id)\n            line = \"{}\\n\".format(os.path.join(img_dir, \"{}.jpg\".format(patient_id)))\n            f.write(line)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e99463093b202e2aabfcd3979701b7e47ba91c3e"},"cell_type":"code","source":"test_dcm_fps = list(set(glob.glob(os.path.join(test_dcm_dir, '*.dcm'))))\ntest_dcm_fps = pd.Series(test_dcm_fps).apply(lambda dcm_fp: dcm_fp.strip().split(\"/\")[-1].replace(\".dcm\",\"\"))\n\nsave_yolov3_test_data(test_dcm_dir, img_dir, metadata_dir, \"te_list.txt\", test_dcm_fps)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42c007da9fd2009310dcf697bca943c4dd7a33f7"},"cell_type":"markdown","source":"### 2.6. Plot a sample test Image"},{"metadata":{"trusted":true,"_uuid":"f057a0fc9938308f69a5d402cf51ba340af592c0"},"cell_type":"code","source":"#ex_patient_id = test_dcm_fps[0]\nex_patient_id = \"0100515c-5204-4f31-98e0-f35e4b00004a\"\nex_img_path = os.path.join(img_dir, \"{}.jpg\".format(ex_patient_id))\n\nplt.imshow(cv2.imread(ex_img_path))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ac8f43aae2bc82599fd1bb4a66d1dd444b66a28"},"cell_type":"markdown","source":"## 3. Prepare Configuration Files for Using YOLOv3\nWe should prepare and modify config files, and bring pre-trained weights necessary for training. This proceeds with following four steps.\n```\n cfg/rsna.data\n cfg/rsna.names\n darknet53.conv.74\n cfg/rsna_yolov3.cfg_train\n```"},{"metadata":{"_uuid":"72fb38e755495fca3bf456c4494c05b5300d3411"},"cell_type":"markdown","source":"### - cfg/rsna.data\nThis file point to RSNA data path\n  * train: Path to training image list textfile\n  * val: Path to validation image list textfile\n  * names: RSNA class name list (see <a href='#3.1.-cfg/rsna.names'>3.1</a>)\n  * backup: A directory where trained weights(checkpoints) will be stored as training progresses."},{"metadata":{"trusted":true,"_uuid":"f78dcf8c107c67e49f8bba8f483ab6a5b409ec0b"},"cell_type":"code","source":"data_extention_file_path = os.path.join(cfg_dir, 'rsna.data')\nwith open(data_extention_file_path, 'w') as f:\n    contents = \"\"\"classes= 1\ntrain  = {}\nvalid  = {}\nnames  = {}\nbackup = {}\n    \"\"\".format(os.path.join(metadata_dir, \"tr_list.txt\"),\n               os.path.join(metadata_dir, \"val_list.txt\"),\n               os.path.join(cfg_dir, 'rsna.names'),\n               backup_dir)\n    f.write(contents)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"684b3e9dde58f56a746952d2934661712e417161"},"cell_type":"code","source":"!cat cfg/rsna.data","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"46cd3f02af89917b9715f51f188a47d6030bd758"},"cell_type":"markdown","source":"### - cfg/rsna.names"},{"metadata":{"trusted":true,"_uuid":"917144c1cffd160a997db563cf9506db13e69414"},"cell_type":"code","source":"# Label list of bounding box.\n!echo \"pneumonia\" > cfg/rsna.names","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6415f6efcae6fca1812cb402af668428d2918e46"},"cell_type":"markdown","source":"### - darknet53.conv.74  (Download Pre-trained Model)\nFor training, we would download the pre-trained model weights(darknet53.conv.74) using following wget command. I recommend you to use this pre-trained weight too. Author of darknet also uses this pre-trained weights in different fields of image recognition."},{"metadata":{"trusted":true,"_uuid":"ec2d5fbdfe6f3eebd553495d10f9adcdae29f16f"},"cell_type":"code","source":"!wget -q https://pjreddie.com/media/files/darknet53.conv.74","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"978655a61edee55f635c8bf8c5befae03c838652"},"cell_type":"markdown","source":"### - cfg/rsna_yolov3.cfg_train\n* Basically, you can use darknet/cfg/yolov3.cfg files. However it won't work for RSNA. you need to edit for RSNA.\n* You can just download a cfg file I edited for RSNA with following wget command.\n\n\n* I refer to the following articles for editing cfg files.\n  * [YOLOv3 blog](https://pjreddie.com/darknet/yolo/)\n  * [YOLOv3 paper](https://pjreddie.com/media/files/papers/YOLOv3.pdf)\n  * [how to train yolov2 blog](https://medium.com/@manivannan_data/how-to-train-yolov2-to-detect-custom-objects-9010df784f36)\n  * [darknet github issues/236](https://github.com/pjreddie/darknet/issues/236)"},{"metadata":{"trusted":true,"_uuid":"f64517332ee89f69af1b013ef6376b19d639f0a6"},"cell_type":"code","source":"!wget --no-check-certificate -q \"https://docs.google.com/uc?export=download&id=18ptTK4Vbeokqpux8Onr0OmwUP9ipmcYO\" -O cfg/rsna_yolov3.cfg_train","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f03592ee228f9ae16edac192071955d01233a147"},"cell_type":"markdown","source":"## 4. Training YOLOv3"},{"metadata":{"_uuid":"69f52f31dea66f3da1952b537c63dc5376d64492"},"cell_type":"markdown","source":"### 4.0. Command for training with Pre-trained CNN Weights (darknet53.conv.74)\n* I didn't run following command on kaggle kernel becuase of the long output.\n* If you crash with  'CUDA Error: out of memory', Solve it by Editing 'batch' and 'subdivisions' in 'cfg/rsna_yolov3.cfg_train'\n* If 'batch' and 'subdivisions' are 64 and 64 respectively, for every iteration only one image will be loaded on GPU memory. So it will use less GPU memory."},{"metadata":{"trusted":true,"_uuid":"0be53a581ab12776c1c276f72acbf08b2db16d77"},"cell_type":"code","source":"# !./darknet_gpu detector train cfg/rsna.data cfg/rsna_yolov3.cfg_train darknet53.conv.74 -i 0 | tee train_log.txt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c43f25ac894192e9c0e03cbcb4c36072210be12"},"cell_type":"markdown","source":"### 4.1. Command for training with Multi-gpu after 1000 iteration\n\nIf you are trying to train with multi-gpu, there are three things to watch out.\n* (The # of gpus)x('learning rate' in 'cfg/rsna_yolov3.cfg_train') is the real learning rate for training\n* I don't recommend you to use multi-gpu for first 1000 iterations. with multi-gpu, training would not be stable. Use single gpu before 1000 and after 1000, continue with more gpus.\n* By the way, If the # of gpus is over 5, training is not stable.\n\n```\nAbove things will depend on your environment. The best way to find the most appropriate method is to just give it a try :)\n```"},{"metadata":{"trusted":true,"_uuid":"a3694016ec0d7372b131837b8e6279a6f59a12a2"},"cell_type":"code","source":"# !./darknet_gpu detector train cfg/rsna.data cfg/rsna_yolov3.cfg_train backup/rsna_yolov3_1000.weights -gpus 0,1,2,3 | tee train_log.txt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef7409909b9403f43b2c92760220b23a38d42029"},"cell_type":"markdown","source":"## 5. How to use trainined YOLOv3 for test images (command line)"},{"metadata":{"_uuid":"c97da5c5f52e0257a3b1a2568e6260a9560700a0"},"cell_type":"markdown","source":"### 5.0. Copy sample test image"},{"metadata":{"trusted":true,"_uuid":"688344d91c7f7fb87418486b9b6b5f91a8b26e9a"},"cell_type":"code","source":"ex_patient_id = annots[annots.Target == 1].patientId.values[10]\nshutil.copy(ex_img_path, \"test.jpg\")\nprint(ex_patient_id)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"245c607d7753b947a91fe4d7ce8daf983afdfd0e"},"cell_type":"markdown","source":"### 5.1. Load trained model (at 15300 iteration)\nSince i uploaded the weights file (large big file) on my google drive, the command is very very long ...\n* It's a weight file at 15300 iteration, which I made submission file with. If you use this weight, you'll get a score of 0.141LB.\n  * Up to 15300 iteration, It takes about 8 hours.\n    * In .cfg file, I set 'batch' and 'subdivisions' as 64 and 8 respectively.\n    * Up to 1000 iteration from 0, it takes about 1h with **one** Tesla P100 GPU.      **(1000 iter/h)**\n    * Up to 15300 iteration from 1000, it takes about 7h with **four** Tesla P100 GPU. **(2043 iter/h)**"},{"metadata":{"trusted":true,"_uuid":"91d60e964d4cbd5397f6094fd811d115977abc3e"},"cell_type":"code","source":"!wget --load-cookies /tmp/cookies.txt -q \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1FDzMN-kGVYCvBeDKwemAazldSVkAEFyd' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1FDzMN-kGVYCvBeDKwemAazldSVkAEFyd\" -O backup/rsna_yolov3_15300.weights && rm -rf /tmp/cookies.txt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d745c3d07102f9f1f27bf57a0c009311f9e0fd3b"},"cell_type":"code","source":"!ls -alsth backup","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41b4ca297869af6808b758824960e9dad12d00c0"},"cell_type":"markdown","source":"### 5.2. cfg file for test (not for training)"},{"metadata":{"trusted":true,"_uuid":"17708ffdc9b834060ccb2b8cb4aefaa3c00b96b6"},"cell_type":"code","source":"!wget --no-check-certificate -q \"https://docs.google.com/uc?export=download&id=10Yk6ZMAKGz5LeBbikciALy82aK3lX-57\" -O cfg/rsna_yolov3.cfg_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de9f7f2f30feeeaa3417ebd338c402e57a9b1f38"},"cell_type":"code","source":"!cd darknet && ./darknet detector test ../cfg/rsna.data ../cfg/rsna_yolov3.cfg_train ../backup/rsna_yolov3_15300.weights ../test.jpg -thresh 0.005","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0dec1cc6f5bd1f8c854697128091ec7bd69d6e3e"},"cell_type":"code","source":"# ![](predictions.jpg)\nplt.imshow(cv2.imread(\"./darknet/predictions.jpg\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"064029258018a3aa0ee22c59d74e6f321b767bf8"},"cell_type":"markdown","source":"## 6. Generate Submission Files with YOLOv3 Python Wrapper"},{"metadata":{"_uuid":"a4848803d5f41f1082de0b0a1e348a26f5d38d58"},"cell_type":"markdown","source":"### 6.0. Download darknet python wrapper (darknet.py)\n* Basically, you can use darknet/python/darknet.py files. However it'll show error.\n* So, I edited the darknet.py. There are two main modifications.\n  * Change print statement to print function for python3\n  * Edit dynamic library('libdarknet.so') file path\n* I leaved '# ===' marks where i edited in darknet.py. For example,\n```\n# ==============================================================================\n#lib = CDLL(\"/home/pjreddie/documents/darknet/libdarknet.so\", RTLD_GLOBAL)\ndarknet_lib_path = os.path.join(os.getcwd(), \"darknet\", \"libdarknet.so\")\nlib = CDLL(darknet_lib_path, RTLD_GLOBAL)\n# ==============================================================================\n```"},{"metadata":{"trusted":true,"_uuid":"6cef00c59783cf8ff052b3c8b79903a9dfa24fc5"},"cell_type":"code","source":"!wget --no-check-certificate -q \"https://docs.google.com/uc?export=download&id=1-KTV7K9G1bl3SmnLnzmpkDyNt6tDmH7j\" -O darknet.py","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91eafd163f9e78557c7acb1be7176845cc15d4c1"},"cell_type":"markdown","source":"### 6.1. Load darknet python wrapper module"},{"metadata":{"trusted":true,"_uuid":"238a70858c290db54f72700fe57215bd7934f6d8"},"cell_type":"code","source":"from darknet import *","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f6193268a5bd5b27d3ba855d6d36715a63d4ee0"},"cell_type":"markdown","source":"### 6.2. Generate submission files\n* When making submission files, be aware of label format which is different in yolo."},{"metadata":{"trusted":true,"_uuid":"877c1dc79f4a4cb0dc3c322a05282d3d902ed615"},"cell_type":"code","source":"threshold = 0.2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b776bab5a37d180d2e5185555b47cafb1a964a71"},"cell_type":"code","source":"submit_file_path = \"submission_stage_train_2.csv\"\ncfg_path = os.path.join(cfg_dir, \"rsna_yolov3.cfg_train\")\nweight_path = os.path.join(backup_dir, \"rsna_yolov3_15300.weights\")\n\ntrain_img_list_path = os.path.join(metadata_dir, \"te_list.txt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d82a944502827338091976c4912f5eb6af6e886e"},"cell_type":"code","source":"gpu_index = 0\nnet = load_net(cfg_path.encode(),\n               weight_path.encode(), \n               gpu_index)\nmeta = load_meta(data_extention_file_path.encode())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c11fa319a577beff0a0d53f107489ee79771bc22"},"cell_type":"code","source":"submit_dict = {\"patientId\": [], \"PredictionString\": []}\n\nwith open(train_img_list_path, \"r\") as train_img_list_f:\n    # tqdm run up to 1000(The # of test set)\n    for line in tqdm(train_img_list_f):\n        patient_id = line.strip().split('/')[-1].strip().split('.')[0]\n\n        infer_result = detect(net, meta, line.strip().encode(), thresh=threshold)\n\n        submit_line = \"\"\n        for e in infer_result:\n            confi = e[1]\n            w = e[2][2]\n            h = e[2][3]\n            x = e[2][0]-w/2\n            y = e[2][1]-h/2\n            submit_line += \"{} {} {} {} {} \".format(confi, x, y, w, h)\n\n        submit_dict[\"patientId\"].append(patient_id)\n        submit_dict[\"PredictionString\"].append(submit_line)\n\npd.DataFrame(submit_dict).to_csv(submit_file_path, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ffbb3e3205f76f5b8f216266539469c65d56cc1"},"cell_type":"code","source":"# !ls -lsht\n!rm -rf darknet images labels metadata backup cfg\n!rm -rf train_log.txt darknet53.conv.74 darknet.py darknet_gpu\n!rm -rf test.jpg\n!rm -rf __pycache__ .ipynb_checkpoints","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"74cd76e2b853568a7f2962f0c220a64fe87e67c1"},"cell_type":"code","source":"!ls -alsht","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2fbf050ee377907eb544c04382d07e1449ef577a"},"cell_type":"markdown","source":"## 7. Future works & Etc\n\n### Future works (Things to try)\n* Image augmentation\n* More training\n* Utilizing the not labeled images because we got rid of not labeled images above\n\n### ETC\n* For a private matter, i can not proceed RSNA task after 09/27. If you have any ideas, questions and problems with this kernel after 09/27, Please leave those things anyway~! Collaborator '@John Byun' will reply to your comments."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}