{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# TFOD instalation and setup","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:38:58.787274Z","iopub.execute_input":"2021-07-24T03:38:58.787585Z","iopub.status.idle":"2021-07-24T03:38:58.791575Z","shell.execute_reply.started":"2021-07-24T03:38:58.787556Z","shell.execute_reply":"2021-07-24T03:38:58.790279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check NVIDIA System Management Interface\n!nvidia-smi\n\n# If you'd like to see your GPU (maybe to see if it's CUDA compatible)\n!lspci | grep -i nvidia\n\n# If you's like to see the location of CUDA\n!whereis cuda\n\n# If you'd like to see CUDA version\n!nvcc --version\n\n# Check presence of CuDNN \n!whereis cudnn\n\n# Check version of cudnn\n!cat /usr/include/cudnn_version.h","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:39:00.115395Z","iopub.execute_input":"2021-07-24T03:39:00.115728Z","iopub.status.idle":"2021-07-24T03:39:04.319997Z","shell.execute_reply.started":"2021-07-24T03:39:00.115697Z","shell.execute_reply":"2021-07-24T03:39:04.318963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check is TensorFlow can access GPU\nfrom tensorflow.python.client import device_lib\ndevice_lib.list_local_devices()\n\n\n\n# Alternatively, check device name\nimport tensorflow as tf\ntf.test.gpu_device_name()\n\n# Check for CUDA\ntf.test.is_built_with_cuda()","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:39:08.745643Z","iopub.execute_input":"2021-07-24T03:39:08.746029Z","iopub.status.idle":"2021-07-24T03:39:14.801445Z","shell.execute_reply.started":"2021-07-24T03:39:08.745994Z","shell.execute_reply":"2021-07-24T03:39:14.800431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clearly, TensorFlow is able access GPU and everything seems normal. What we need to ensure is that the **TensorFlow 2 Object Detection API** is able to access the **GPU**.","metadata":{}},{"cell_type":"markdown","source":"**What options do we have?**\n1. We could upgrade to CUDA v11.2. I found this <a href=\"https://medium.com/analytics-vidhya/installing-any-version-of-cuda-on-ubuntu-and-using-tensorflow-and-torch-on-gpu-b1a954500786\">blog</a> useful for doing so. But it involves installing 2-3GB CUDA Toolkit from <a href=\"https://developer.nvidia.com/cuda-11.2.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal\">here</a> and other complicated software engineering that's especially difficult on Kaggle.\n2. We can work with an older version of TensorFlow.\n\n**How do we work with an older version?**\n1. Install the latest TF2 OD API (v2.5). You'll also find a directory structure with everything you need.\n2. Uninstall tf-models-official v2.5\n3. Install tensorflow v2.4\n4. Install tf-models-official v2.4 (it is compatible with the hardware/software configuration)","metadata":{}},{"cell_type":"markdown","source":"### TensorFlow Object Detection API Installation\n#### 1. Downloading the TensorFlow Model Garden\n>workspace/\n\n>└─ models/\n\n>        ├─ community/\n\n>        ├─ official/\n\n>        ├─ orbit/\n\n>        ├─ research/\n\n>        └── ...\n#### 2. Protobuf Installation/Compilation\n#### 3. COCO API installation\n#### 4. Install the Object Detection API","metadata":{}},{"cell_type":"code","source":"# Download models\n!git clone --depth 1 https://github.com/tensorflow/models\n\n# Compile proto files \n%cd models/research\n!pip install protobuf\n!protoc object_detection/protos/*.proto --python_out=.\n%cd ..\n%cd ..\n\n# Install cocoapi\n!pip install cython \n!git clone https://github.com/cocodataset/cocoapi.git\n%cd cocoapi/PythonAPI\n!make\n%cd ..\n%cd ..\n!cp -r cocoapi/PythonAPI/pycocotools models/research/\n\n# Install object detection api\n%cd models/research\n!cp object_detection/packages/tf2/setup.py .\n!python -m pip install .\n%cd ..\n%cd ..","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-24T03:39:14.802946Z","iopub.execute_input":"2021-07-24T03:39:14.803349Z","iopub.status.idle":"2021-07-24T03:41:22.385411Z","shell.execute_reply.started":"2021-07-24T03:39:14.803312Z","shell.execute_reply":"2021-07-24T03:41:22.384335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# View tensorflow installation\n!pip freeze | grep tensorflow\n\n# Uninstall version 2.5\n!pip uninstall tf-models-official --yes\n# Install version 2.4\n!pip install tensorflow==2.4\n!pip install tf-models-official==2.4","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2021-07-24T03:41:22.391988Z","iopub.execute_input":"2021-07-24T03:41:22.392302Z","iopub.status.idle":"2021-07-24T03:42:23.404768Z","shell.execute_reply.started":"2021-07-24T03:41:22.392269Z","shell.execute_reply":"2021-07-24T03:42:23.403727Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# View tensorflow installation\n!pip freeze | grep tensorflow","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2021-07-24T03:42:23.408415Z","iopub.execute_input":"2021-07-24T03:42:23.408692Z","iopub.status.idle":"2021-07-24T03:42:25.172331Z","shell.execute_reply.started":"2021-07-24T03:42:23.408652Z","shell.execute_reply":"2021-07-24T03:42:25.17141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Verify TensorFlow 2 Object Detection API installation (optional)\n%cd models/research\n!python object_detection/builders/model_builder_tf2_test.py\n%cd ..\n%cd ..","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2021-07-24T03:42:25.173899Z","iopub.execute_input":"2021-07-24T03:42:25.174267Z","iopub.status.idle":"2021-07-24T03:43:01.406534Z","shell.execute_reply.started":"2021-07-24T03:42:25.174228Z","shell.execute_reply":"2021-07-24T03:43:01.405107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Making Directories \n!mkdir workspace\n%cd workspace\n!mkdir annotations\n!mkdir script\n!mkdir csv_files\n!mkdir pre_trained_models\n!mkdir models","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:43:01.411709Z","iopub.execute_input":"2021-07-24T03:43:01.411986Z","iopub.status.idle":"2021-07-24T03:43:05.464697Z","shell.execute_reply.started":"2021-07-24T03:43:01.411957Z","shell.execute_reply":"2021-07-24T03:43:05.463473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Skip this..........if you already having tf.records generated and uploaded to kaggle/input/**tf_records folder\n\n# or else Run this following code\n# make sure you have the necessary kaggle/input/folders are uploaded properly\n\n# copy the scripts file uploaded to kaggle/input/scripts to kaggle/.../workspace/scripts/\n%cd script\n!cp /kaggle/input/opacity-files/generate_tfrecord_csv_to_tfrecord.py .\n!ls  \n%cd ..\n\n# Copy the csv files uploaded to kaggle/input/**csv_files to kaggle/working/workspace/csv_files folders\n%cd csv_files\n!cp /kaggle/input/opacity-files/opacity_train.csv  .\n!cp /kaggle/input/opacity-files/opacity_test.csv  .\n!ls\n%cd ..\n\n# run this code to generate tfrecords --- if tf records are not present in kaggle input or kaggle/working/annotations folders\n%cd script\n! python generate_tfrecord_csv_to_tfrecord.py --csv_input=/kaggle/working/workspace/csv_files/opacity_train.csv  --output_path=/kaggle/working/workspace/annotations/train.record  --image_dir=\"\"\n! python generate_tfrecord_csv_to_tfrecord.py --csv_input=/kaggle/working/workspace/csv_files/opacity_test.csv  --output_path=/kaggle/working/workspace/annotations/test.record  --image_dir=\"\"\n\n%cd ..\n%cd annotations\n# copy and move the label_map_file\n!cp /kaggle/input/opacity-files/label_map.pbtxt   .\n!ls\n%cd ..","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:43:05.46948Z","iopub.execute_input":"2021-07-24T03:43:05.471632Z","iopub.status.idle":"2021-07-24T03:43:53.083491Z","shell.execute_reply.started":"2021-07-24T03:43:05.471587Z","shell.execute_reply":"2021-07-24T03:43:53.081938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Don't Run this cell\n\nLook at the instruction belowe before running it","metadata":{}},{"cell_type":"code","source":"# Run this only if tf.record file are already uploaded to kaggle/input/**tfrecords/folder\n# copy the available file to appropriate kaggle/working/***   folders\n\n# skip this if you are generating tf.records here on the run.....\n%cd annotations\n!cp /kaggle/input/covid-scripts/covid_label_map.pbtxt . \n!cp /kaggle/input/****/train.record . \n!cp /kaggle/input/****/test.record . \n\n!ls\n%cd ..","metadata":{"execution":{"iopub.status.busy":"2021-07-23T11:46:21.042819Z","iopub.execute_input":"2021-07-23T11:46:21.043212Z","iopub.status.idle":"2021-07-23T11:46:23.69032Z","shell.execute_reply.started":"2021-07-23T11:46:21.043159Z","shell.execute_reply":"2021-07-23T11:46:23.689029Z"},"jupyter":{"source_hidden":true,"outputs_hidden":true},"collapsed":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd pre_trained_models\n# Choose model of your choice and dpaste the download link kere \n!wget http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d1_coco17_tpu-32.tar.gz\n!tar -xf efficientdet_d1_coco17_tpu-32.tar.gz\n%cd efficientdet_d1_coco17_tpu-32\n!ls\n%cd ..\n%cd ..","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:43:53.086228Z","iopub.execute_input":"2021-07-24T03:43:53.086609Z","iopub.status.idle":"2021-07-24T03:43:56.507615Z","shell.execute_reply.started":"2021-07-24T03:43:53.086568Z","shell.execute_reply":"2021-07-24T03:43:56.506491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# copying uploaded config file from kaggle/input/**config folder\n%cd models\n!mkdir efficientdet_d1_coco17_tpu-32\n%cd efficientdet_d1_coco17_tpu-32\n!cp /kaggle/input/ednetd1-config/pipeline.config  . \n!ls\n%cd ..\n%cd ..","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:43:56.510705Z","iopub.execute_input":"2021-07-24T03:43:56.510981Z","iopub.status.idle":"2021-07-24T03:43:58.557633Z","shell.execute_reply.started":"2021-07-24T03:43:56.510952Z","shell.execute_reply":"2021-07-24T03:43:58.556217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make sure that your are in kaggle/working/workspace\n# execute the following commands","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!cp  /kaggle/working/models/research/object_detection/model_main_tf2.py  .","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:44:55.787571Z","iopub.execute_input":"2021-07-24T03:44:55.788027Z","iopub.status.idle":"2021-07-24T03:44:56.487168Z","shell.execute_reply.started":"2021-07-24T03:44:55.787986Z","shell.execute_reply":"2021-07-24T03:44:56.48594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Command line to start Training","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:38:50.554015Z","iopub.execute_input":"2021-07-24T03:38:50.554415Z","iopub.status.idle":"2021-07-24T03:38:50.558166Z","shell.execute_reply.started":"2021-07-24T03:38:50.554324Z","shell.execute_reply":"2021-07-24T03:38:50.557308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python model_main_tf2.py --model_dir=models/efficientdet_d1_coco17_tpu-32 --pipeline_config_path=models/efficientdet_d1_coco17_tpu-32/pipeline.config","metadata":{"execution":{"iopub.status.busy":"2021-07-24T03:44:59.460452Z","iopub.execute_input":"2021-07-24T03:44:59.460791Z","iopub.status.idle":"2021-07-24T11:05:24.775053Z","shell.execute_reply.started":"2021-07-24T03:44:59.460756Z","shell.execute_reply":"2021-07-24T11:05:24.773967Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exporting the trained model","metadata":{"execution":{"iopub.status.busy":"2021-07-23T17:10:37.700956Z","iopub.execute_input":"2021-07-23T17:10:37.701377Z","iopub.status.idle":"2021-07-23T17:10:37.71058Z","shell.execute_reply.started":"2021-07-23T17:10:37.701344Z","shell.execute_reply":"2021-07-23T17:10:37.70933Z"}}},{"cell_type":"code","source":"pwd","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:06:07.204904Z","iopub.execute_input":"2021-07-24T11:06:07.205233Z","iopub.status.idle":"2021-07-24T11:06:07.216093Z","shell.execute_reply.started":"2021-07-24T11:06:07.205198Z","shell.execute_reply":"2021-07-24T11:06:07.215164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!cp /kaggle/working/models/research/object_detection/exporter_main_v2.py  .","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:06:18.711957Z","iopub.execute_input":"2021-07-24T11:06:18.712366Z","iopub.status.idle":"2021-07-24T11:06:19.538032Z","shell.execute_reply.started":"2021-07-24T11:06:18.712328Z","shell.execute_reply":"2021-07-24T11:06:19.53691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir trained_models","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:06:35.16086Z","iopub.execute_input":"2021-07-24T11:06:35.161225Z","iopub.status.idle":"2021-07-24T11:06:35.841839Z","shell.execute_reply.started":"2021-07-24T11:06:35.161191Z","shell.execute_reply":"2021-07-24T11:06:35.840764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python exporter_main_v2.py --input_type image_tensor --pipeline_config_path  models/efficientdet_d1_coco17_tpu-32/pipeline.config --trained_checkpoint_dir models/efficientdet_d1_coco17_tpu-32 --output_directory trained_models","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:07:45.667807Z","iopub.execute_input":"2021-07-24T11:07:45.66815Z","iopub.status.idle":"2021-07-24T11:10:21.985594Z","shell.execute_reply.started":"2021-07-24T11:07:45.668119Z","shell.execute_reply":"2021-07-24T11:10:21.984576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pwd","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:12:11.041872Z","iopub.execute_input":"2021-07-24T11:12:11.042245Z","iopub.status.idle":"2021-07-24T11:12:11.048999Z","shell.execute_reply.started":"2021-07-24T11:12:11.042209Z","shell.execute_reply":"2021-07-24T11:12:11.048009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ls","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:12:20.062431Z","iopub.execute_input":"2021-07-24T11:12:20.062762Z","iopub.status.idle":"2021-07-24T11:12:20.712898Z","shell.execute_reply.started":"2021-07-24T11:12:20.062731Z","shell.execute_reply":"2021-07-24T11:12:20.712017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cd trained_models","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:12:48.362888Z","iopub.execute_input":"2021-07-24T11:12:48.363227Z","iopub.status.idle":"2021-07-24T11:12:48.370494Z","shell.execute_reply.started":"2021-07-24T11:12:48.363195Z","shell.execute_reply":"2021-07-24T11:12:48.369297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ls","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:12:57.070623Z","iopub.execute_input":"2021-07-24T11:12:57.070988Z","iopub.status.idle":"2021-07-24T11:12:57.753236Z","shell.execute_reply.started":"2021-07-24T11:12:57.070957Z","shell.execute_reply":"2021-07-24T11:12:57.752264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cd trained_models\nls","metadata":{"execution":{"iopub.status.busy":"2021-07-23T17:30:31.138066Z","iopub.execute_input":"2021-07-23T17:30:31.138479Z","iopub.status.idle":"2021-07-23T17:30:31.146002Z","shell.execute_reply.started":"2021-07-23T17:30:31.138446Z","shell.execute_reply":"2021-07-23T17:30:31.144735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pwd","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:19:02.435872Z","iopub.execute_input":"2021-07-24T11:19:02.436272Z","iopub.status.idle":"2021-07-24T11:19:02.443998Z","shell.execute_reply.started":"2021-07-24T11:19:02.436237Z","shell.execute_reply":"2021-07-24T11:19:02.44294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cd ..","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:19:07.779911Z","iopub.execute_input":"2021-07-24T11:19:07.780249Z","iopub.status.idle":"2021-07-24T11:19:07.787008Z","shell.execute_reply.started":"2021-07-24T11:19:07.780219Z","shell.execute_reply":"2021-07-24T11:19:07.786171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pwd","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:19:10.642192Z","iopub.execute_input":"2021-07-24T11:19:10.642603Z","iopub.status.idle":"2021-07-24T11:19:10.654411Z","shell.execute_reply.started":"2021-07-24T11:19:10.642569Z","shell.execute_reply":"2021-07-24T11:19:10.653404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cd ..","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:19:16.51526Z","iopub.execute_input":"2021-07-24T11:19:16.51558Z","iopub.status.idle":"2021-07-24T11:19:16.520684Z","shell.execute_reply.started":"2021-07-24T11:19:16.515549Z","shell.execute_reply":"2021-07-24T11:19:16.519782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir sheri","metadata":{"execution":{"iopub.status.busy":"2021-07-24T11:20:04.466806Z","iopub.execute_input":"2021-07-24T11:20:04.467129Z","iopub.status.idle":"2021-07-24T11:20:05.107262Z","shell.execute_reply.started":"2021-07-24T11:20:04.467098Z","shell.execute_reply":"2021-07-24T11:20:05.106135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}