{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":6799,"databundleVersionId":4225553,"sourceType":"competition"}],"dockerImageVersionId":30627,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Searching for Hardware Aware Neural Networks on ImageNet for FPGA in MobilenetV3 Search Space\n\n#### If you liked notebook, please upvote, gives a lot of motivation to learn more and work  \nGithub Repo to the work: [FONAS](https://github.com/FPGA-Vision/FONAS.git)\n.........\n\nWe will be utilizing **[Once-for-All (OFA)](https://github.com/mit-han-lab/once-for-all)** network trained on MobilenetV3 search space as a our supernetwork to retrieve the weights and to estimate the accuracy of subnets in our search space because it requires significant amount of time to train each subnet for evaluating its perfomace. This slows down the searching process. Utilizing the OFA would speed up the search process as we won't have to train the network at all. The search is perfomed on the Imagenet Dataset. \n\n![Image](https://i.ytimg.com/vi/a2wOjxRf_xg/hq720.jpg?sqp=-oaymwEhCK4FEIIDSFryq4qpAxMIARUAAAAAGAElAADIQj0AgKJD&rs=AOn4CLCjHqPbKYJ57Nn3LPKLIe184e7YWA)\n\n#### #To Do - Construct Latency Table and Add latency Constraint\n","metadata":{}},{"cell_type":"markdown","source":"## 1. Preparation\nLet's first install all the required packages:","metadata":{}},{"cell_type":"code","source":"%%script echo skipping  \n\nprint('Installing PyTorch...')\n! pip install torch \nprint('Installing torchvision...')\n! pip install torchvision\nprint('Installing numpy...')\n! pip install numpy \n# thop is a package for FLOPs computing.\nprint('Installing thop (FLOPs counter) ...')\n! pip install thop \n# ofa is a package containing training code, pretrained specialized models and inference code for the once-for-all networks.\n# print('Installing OFA...')\n# ! pip install ofa \n# tqdm is a package for displaying a progress bar.\nprint('Installing tqdm (progress bar) ...')\n! pip install tqdm \nprint('Installing matplotlib...')\n! pip install matplotlib \n!pip install graphviz\n! pip install torch-summary \n\nprint('All required packages have been successfully installed!')\n","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:17:19.352529Z","iopub.execute_input":"2024-01-18T11:17:19.353262Z","iopub.status.idle":"2024-01-18T11:17:19.375012Z","shell.execute_reply.started":"2024-01-18T11:17:19.353225Z","shell.execute_reply":"2024-01-18T11:17:19.374052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For kaggle\n# !rm -r /kaggle/working/Evolutionary-Neural-Architectural-Search-for-FPGAs /kaggle/working/ofa /kaggle/working/viz /kaggle/working/search_space_blocks /kaggle/working/blocks /kaggle/working/figures\n!pip install thop \n! pip install gdown\n!pip install shutil\n!pip install graphviz\n! pip install torch-summary \n! git clone --branch for_kaggle https://github.com/amitpant7/Evolutionary-Neural-Architectural-Search-for-FPGAs.git\n! mv -f /kaggle/working/Evolutionary-Neural-Architectural-Search-for-FPGAs/* /kaggle/working\n! rm -r Evolutionary-Neural-Architectural-Search-for-FPGAs\n","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:17:26.55194Z","iopub.execute_input":"2024-01-18T11:17:26.55232Z","iopub.status.idle":"2024-01-18T11:18:51.644641Z","shell.execute_reply.started":"2024-01-18T11:17:26.552288Z","shell.execute_reply":"2024-01-18T11:18:51.643147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, we can import the packages used in this tutorial:","metadata":{}},{"cell_type":"code","source":"import os\nimport torch\nimport torch.nn as nn\nfrom torchvision import transforms, datasets\nimport numpy as np\nimport time\nimport random\nimport shutil\nimport math\nfrom PIL import Image\nimport copy\nfrom matplotlib import pyplot as plt\nfrom torchsummary import summary\n\nfrom ofa.model_zoo import ofa_net\nfrom ofa.utils import download_url\n\nfrom ofa.accuracy_predictor import AccuracyPredictor\nfrom ofa.flops_table import ArthIntTable\n\nfrom ofa.latency_table import LatencyTable\nfrom ofa.evolution_finder import EvolutionFinder\nfrom ofa.imagenet_eval_helper import evaluate_ofa_subnet, evaluate_ofa_specialized\nfrom ofa.imagenet_classification.elastic_nn.networks.ofa_mbv3 import OFAMobileNetV3\n\nfrom ofa.utils.arch_visualization_helper import draw_arch\n# from ofa.tutorial import AccuracyPredictor, FLOPsTable, LatencyTable, EvolutionFinder\n# from ofa.tutorial import evaluate_ofa_subnet, evaluate_ofa_specialized\n\nfrom tqdm import tqdm\n\n# set random seed\nrandom_seed = 1\nrandom.seed(random_seed)\nnp.random.seed(random_seed)\ntorch.manual_seed(random_seed)\nprint('Successfully imported all packages and configured random seed to %d!'%random_seed)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:25:46.868737Z","iopub.execute_input":"2024-01-18T11:25:46.869173Z","iopub.status.idle":"2024-01-18T11:25:48.892782Z","shell.execute_reply.started":"2024-01-18T11:25:46.869139Z","shell.execute_reply":"2024-01-18T11:25:48.891614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now it's time to determine which device to use for neural network inference in the rest of this tutorial. If your machine is equipped with GPU(s), we will use the GPU by default. Otherwise, we will use the CPU.","metadata":{}},{"cell_type":"code","source":"#os.environ['CUDA_VISIBLE_DEVICES'] = '0'\ncuda_available = torch.cuda.is_available()\nif cuda_available:\n    torch.backends.cudnn.enabled = True\n    torch.backends.cudnn.benchmark = True\n    torch.cuda.manual_seed(random_seed)\n    print('Using GPU.')\nelse:\n    print('Using CPU.')","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2024-01-18T11:25:48.894851Z","iopub.execute_input":"2024-01-18T11:25:48.895315Z","iopub.status.idle":"2024-01-18T11:25:48.91929Z","shell.execute_reply.started":"2024-01-18T11:25:48.895285Z","shell.execute_reply":"2024-01-18T11:25:48.918097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  2. Architecutre Visualization & Encoding: Exploring the OFA network ","metadata":{}},{"cell_type":"markdown","source":"Good! Now you have successfully configured the environment! It's time to import the **OFA network** for the following experiments.\nThe OFA network used in this tutorial is built upon MobileNetV3 with width multiplier 1.2, supporting elastic depth (2, 3, 4) per stage, elastic expand ratio (3, 4, 6), and elastic kernel size (3, 5 7) per block.","metadata":{}},{"cell_type":"code","source":"net_id  = 'ofa_mbv3_d234_e346_k357_w1.2'\nurl_base = \"https://raw.githubusercontent.com/han-cai/files/master/ofa/ofa_nets/\"\n\nofa_network = OFAMobileNetV3(\n            dropout_rate=0,\n            width_mult=1.2,\n            ks_list=[3, 5, 7],\n            expand_ratio_list=[3, 4, 6],\n            depth_list=[2, 3, 4],\n        )\n\npt_path = download_url(url_base + net_id, model_dir=\".torch/ofa_nets\")\ninit = torch.load(pt_path, map_location=\"cpu\")[\"state_dict\"]\nofa_network.load_state_dict(init)\nprint('Supernetwork Ready')","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:25:48.920676Z","iopub.execute_input":"2024-01-18T11:25:48.921143Z","iopub.status.idle":"2024-01-18T11:25:49.135443Z","shell.execute_reply.started":"2024-01-18T11:25:48.921085Z","shell.execute_reply":"2024-01-18T11:25:49.13444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# super_net_name = \"ofa_supernet_mbv3_w12\"\n# super_net = torch.hub.load('mit-han-lab/once-for-all', super_net_name, pretrained=True)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:25:49.136776Z","iopub.execute_input":"2024-01-18T11:25:49.137147Z","iopub.status.idle":"2024-01-18T11:25:49.14122Z","shell.execute_reply.started":"2024-01-18T11:25:49.137112Z","shell.execute_reply":"2024-01-18T11:25:49.140409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets visualize a randomly sampled network from our supernet.\n\nIn the architecture visualization, the legend of each block MBConv{e}-{k}x{k} means that the current block is a mobile inverted block with expand ratio e and the kernel size of the depthwise convolution layer is k. Different colors of the blocks indicate different kernel sizes, and gray blocks are network stage dividers. Different widths for the blocks indicate different expand ratios. We also annotate the output resolution close to each block.","metadata":{}},{"cell_type":"code","source":"# Randomly sample sub-networks from OFA network\nimage_size = 224\n\ncfg1 = ofa_network.sample_active_subnet()\nsubnet = ofa_network.get_active_subnet(preserve_weight=True)\n\n#manualy set the subnet \ncfg = ofa_network.set_active_subnet(ks=3, e=6, d=4)\n\ncfg = ofa_network.set_max_net()\nsubnet2 = ofa_network.get_active_subnet(preserve_weight=True)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:25:49.144586Z","iopub.execute_input":"2024-01-18T11:25:49.145134Z","iopub.status.idle":"2024-01-18T11:25:49.355751Z","shell.execute_reply.started":"2024-01-18T11:25:49.145097Z","shell.execute_reply":"2024-01-18T11:25:49.35464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_subnet(cfg):\n    draw_arch(cfg[\"ks\"], cfg[\"e\"], cfg[\"d\"], image_size, out_name=\"viz/subnet\")\n    im = Image.open(\"viz/subnet.png\")\n    im = im.rotate(90, expand=1)\n    fig = plt.figure(figsize=(im.size[0] / 250, im.size[1] / 250))\n    plt.axis(\"off\")\n    plt.imshow(im)\n    plt.show()\n\nvisualize_subnet(cfg)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:25:49.357012Z","iopub.execute_input":"2024-01-18T11:25:49.357316Z","iopub.status.idle":"2024-01-18T11:25:49.935141Z","shell.execute_reply.started":"2024-01-18T11:25:49.35729Z","shell.execute_reply":"2024-01-18T11:25:49.934092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \nEvery subnet in the ofa can be represented in the form of python dictionary. This encoding helps us to represent entire netowk with just few numbers.\n\nHere is one expample of encoding","metadata":{}},{"cell_type":"code","source":"print('The architecture encoding of random subnetwork', cfg)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:25:49.936498Z","iopub.execute_input":"2024-01-18T11:25:49.936893Z","iopub.status.idle":"2024-01-18T11:25:49.9426Z","shell.execute_reply.started":"2024-01-18T11:25:49.936855Z","shell.execute_reply":"2024-01-18T11:25:49.941676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Dataset Preperation","metadata":{}},{"cell_type":"markdown","source":"Now, let's build the ImageNet dataset and the corresponding dataloader. Notice that **if you are not in kaggle it will be skipped** since it will be very slow.\n\n\nWe will only use subset of ImageNet validation set which will contains 10,000 images for testing.\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"**I will utilize subsets of imagenet for both validation and retraining.**","metadata":{}},{"cell_type":"code","source":"# # This was prepared with the help of imagenet dataset.\n# %%script echo skipping\n\n# def make_subset(old_path, new_path, number_per_class=10):\n#     # Remove existing new_path if it exists\n#     if os.path.exists(new_path):\n#         shutil.rmtree(new_path)\n\n#     os.makedirs(new_path)\n\n#     print('Creating subset...')\n#     dirs = [d for d in os.listdir(old_path) if os.path.isdir(os.path.join(old_path, d))]\n\n#     for i, directory in enumerate(dirs):\n#         directory_path = os.path.join(old_path, directory)\n#         filenames_in_dir = [filename for filename in os.listdir(directory_path) if os.path.isfile(os.path.join(directory_path, filename))]\n#         sampled_filenames = random.sample(filenames_in_dir, min(number_per_class, len(filenames_in_dir)))\n\n#         new_directory_path = os.path.join(new_path, directory)\n#         os.makedirs(new_directory_path)\n\n#         for filename in sampled_filenames:\n#             src_file_path = os.path.join(directory_path, filename)\n#             tgt_file_path = os.path.join(new_directory_path, filename)\n\n#             try:\n#                 shutil.copy(src_file_path, tgt_file_path)\n#             except Exception as e:\n#                 print(f\"Error copying {src_file_path} to {tgt_file_path}: {e}\")\n                \n#     print('Subset creation complete.')\n\n# # Example usage\n# old_val = '/kaggle/input/imagenet1kvalidation/val'\n# new_val = '/kaggle/working/imagenet_sub_val'\n\n# make_subset(old_val, new_val, number_per_class=10)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:26:36.244226Z","iopub.execute_input":"2024-01-18T11:26:36.245213Z","iopub.status.idle":"2024-01-18T11:26:36.250866Z","shell.execute_reply.started":"2024-01-18T11:26:36.245172Z","shell.execute_reply":"2024-01-18T11:26:36.249686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size=32\n\n#I will use a susbset of imagenetval of 10k images \nif cuda_available:\n    # path to the ImageNet dataset\n    # link --> https://www.kaggle.com/datasets/titericz/imagenet1k-val\n    \n    imagenet_data_path = '/kaggle/working/imagenet_sub_val'\n\n    # if 'imagenet_data_path' is empty, download a subset of ImageNet containing 2000 images (~250M) for test\n    if not os.path.isdir(imagenet_data_path):\n        print('%s is empty. Download a subset of ImageNet for test.' % imagenet_data_path)\n\n    print('The ImageNet dataset files are ready.')\nelse:\n    print('Since GPU is not found in the environment, we skip all scripts related to ImageNet evaluation.')\n    \n    \n  \nif cuda_available:\n    # The following function build the data transforms for test\n    def build_val_transform(size):\n        return transforms.Compose([\n            transforms.Resize(int(math.ceil(size / 0.875))),\n            transforms.CenterCrop(size),\n            transforms.ToTensor(),\n            transforms.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225]\n            ),\n        ])\n    \n    val_data = datasets.ImageFolder(\n            root=os.path.join(imagenet_data_path),\n            transform=build_val_transform(224)\n        )\n    \n\n    val_loader = torch.utils.data.DataLoader(\n        val_data,\n        batch_size=batch_size,  \n        shuffle = True,\n        num_workers=4,  \n        pin_memory=True,\n        drop_last=False,\n    )\n    print('The ImageNet dataloader is ready. Size : {}'.format(len(val_loader)*batch_size))\nelse:\n    data_loader = None\n    print('Since GPU is not found in the environment, we skip all scripts related to ImageNet evaluation.')","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2024-01-18T11:26:36.252798Z","iopub.execute_input":"2024-01-18T11:26:36.253106Z","iopub.status.idle":"2024-01-18T11:26:36.341948Z","shell.execute_reply.started":"2024-01-18T11:26:36.253078Z","shell.execute_reply":"2024-01-18T11:26:36.340994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now you have configured the dataset. Let's build the dataloader for evaluation.\nAgain, this will be skipped if you are in a CPU environment.","metadata":{}},{"cell_type":"markdown","source":"Lets evaluate our randomly sampled network on imagenet validation set ","metadata":{}},{"cell_type":"code","source":"from ofa.utils.common_tools import *\ndef evaluate_sub(net, data_loader=val_loader ,device=\"cuda:0\"):\n    if \"cuda\" in device:\n        net = torch.nn.DataParallel(net).to(device)\n    else:\n        net = net.to(device)\n\n    criterion = nn.CrossEntropyLoss().to(device)\n\n    net.eval()\n    net = net.to(device)\n    losses = AverageMeter()\n    top1 = AverageMeter()\n    top5 = AverageMeter()\n\n    with torch.no_grad():\n        with tqdm(total=len(data_loader), desc=\"Validate\") as t:\n            for i, (images, labels) in enumerate(data_loader):\n                images, labels = images.to(device), labels.to(device)\n                # compute output\n                output = net(images)\n                loss = criterion(output, labels)\n                # measure accuracy and record loss\n                acc1, acc5 = accuracy(output, labels, topk=(1, 5))\n\n                losses.update(loss.item(), images.size(0))\n                top1.update(acc1[0].item(), images.size(0))\n                top5.update(acc5[0].item(), images.size(0))\n                t.set_postfix(\n                    {\n                        \"loss\": losses.avg,\n                        \"top1\": top1.avg,\n                        \"top5\": top5.avg,\n                        \"img_size\": images.size(2),\n                    }\n                )\n                t.update(1)\n\n    print(\n        \"Results: loss=%.5f,\\t top1=%.1f,\\t top5=%.1f\"\n        % (losses.avg, top1.avg, top5.avg)\n    )\n    return top1.avg","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:26:36.342996Z","iopub.execute_input":"2024-01-18T11:26:36.343276Z","iopub.status.idle":"2024-01-18T11:26:36.354187Z","shell.execute_reply.started":"2024-01-18T11:26:36.343252Z","shell.execute_reply":"2024-01-18T11:26:36.353202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%script echo skipping\nif cuda_available:\n    top1 = evaluate_sub(subnet2)\n\n    print('Finished evaluating the pretrained sub-network: %s!' % top1)\nelse:\n    print('Since GPU is not found in the environment, we skip all scripts related to ImageNet evaluation.')","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:26:36.355559Z","iopub.execute_input":"2024-01-18T11:26:36.355901Z","iopub.status.idle":"2024-01-18T11:27:11.4819Z","shell.execute_reply.started":"2024-01-18T11:26:36.355867Z","shell.execute_reply":"2024-01-18T11:27:11.480786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Accuracy Predictor \n\nThe key components of very fast neural network deployment are **accuracy predictors** and **efficiency predictors**.\nFor the accuracy predictor, it predicts the Top-1 accuracy of a given sub-network on a **holdout validation set**\n(different from the official 50K validation set) so that we do **NOT** need to run very costly inference on ImageNet\nwhile searching for specialized models. Such an accuracy predictor is trained using an accuracy dataset built with the OFA network.","metadata":{}},{"cell_type":"code","source":"# accuracy predictor\naccuracy_predictor = AccuracyPredictor(\n    pretrained=True,\n    device='cuda:0' if cuda_available else 'cpu'\n)\n\nprint('The accuracy predictor is ready!')\nprint(accuracy_predictor.model)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2024-01-18T11:27:11.484588Z","iopub.execute_input":"2024-01-18T11:27:11.484917Z","iopub.status.idle":"2024-01-18T11:27:11.826426Z","shell.execute_reply.started":"2024-01-18T11:27:11.484888Z","shell.execute_reply":"2024-01-18T11:27:11.825484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets try predicting accuracy of our randomly sampled subnet","metadata":{}},{"cell_type":"code","source":"cfg['r']= [224]\n\nacc = accuracy_predictor.predict_accuracy([cfg])\nprint(acc*100)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:27:11.827587Z","iopub.execute_input":"2024-01-18T11:27:11.827884Z","iopub.status.idle":"2024-01-18T11:27:11.865188Z","shell.execute_reply.started":"2024-01-18T11:27:11.827858Z","shell.execute_reply":"2024-01-18T11:27:11.864222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we have the powerful **accuracy predictor**. We then introduce two types of **efficiency predictors**: the latency predictor and the FLOPs predictor. \n\nThe intuition of having efficiency predictors, especially the latency predictor, is that measuring the latency of a sub-network on-the-fly is also costly, especially for FPGA devices becuase it takes hours of work to implement even a single network and then measure latency.\n\nThe latency predictor is designed to eliminate this cost.","metadata":{}},{"cell_type":"markdown","source":"## 5. Searching with Arthemetic Intensity Constraint\nFor edge deployement like FPGAs computation is cheap where as memory movement is expensive. That's why we need to do as much as computation with fewer number of data movement. This can be defined by arthemetic constraint which is the ratio of MACs to size in bytes. \n\nNow, let's proceed towards searching for efficient network under arthemetic intensity constraint. We use the same accuracy predictor since accuracy predictors are agnostic to the types of efficiency constraint . For the efficiency predictor, we a arthametic intensity table. You can run the code below to setup it in a few seconds.","metadata":{}},{"cell_type":"code","source":"arthemetic_intensity_lookup = ArthIntTable(pred_type='arthemetic_intensity', \n                                  device='cuda:0' if cuda_available else 'cpu',batch_size=1, \n                                  )\n\nprint('The Arthemetic Intensity lookup table is ready!')","metadata":{"pycharm":{"is_executing":true,"name":"#%%\n"},"execution":{"iopub.status.busy":"2024-01-18T11:27:11.866873Z","iopub.execute_input":"2024-01-18T11:27:11.867255Z","iopub.status.idle":"2024-01-18T11:27:14.054631Z","shell.execute_reply.started":"2024-01-18T11:27:11.86722Z","shell.execute_reply":"2024-01-18T11:27:14.053804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets try to find the Arthemetic intensity of our random sampled subnet","metadata":{}},{"cell_type":"code","source":"arth = 1/arthemetic_intensity_lookup.predict_efficiency(cfg)   #as it returns bytes/ops\nprint('arthemetic intensity = {} ops/byte'.format(arth))","metadata":{"execution":{"iopub.status.busy":"2024-01-18T11:27:14.055931Z","iopub.execute_input":"2024-01-18T11:27:14.05628Z","iopub.status.idle":"2024-01-18T11:27:14.061687Z","shell.execute_reply.started":"2024-01-18T11:27:14.056249Z","shell.execute_reply":"2024-01-18T11:27:14.060761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run Evolutionary search","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#  Hyper-parameters for the evolutionary search process\n\nP = 2500  # The size of population in each generation\nN = 1000  # How many generations of population to be searched\nr = 0.25  # The ratio of networks that are used as parents for next generation\n\nparams = {\n    'constraint_type': 'arthemetic_intensity', # Let's do FLOPs-constrained search\n    'efficiency_constraint': 13,  # ai constraint , suggested range [10, 25]\n    'mutate_prob': 0.1, # The probability of mutation in evolutionary search\n    'mutation_ratio': 0.5, # The ratio of networks that are generated through mutation in generation n >= 2.\n    'efficiency_predictor': arthemetic_intensity_lookup, # To use a predefined efficiency predictor.\n    'accuracy_predictor': accuracy_predictor, # To use a predefined accuracy_predictor predictor.\n    'population_size': P,\n    'max_time_budget': N,\n    'parent_ratio': r,\n}\n\n\nfinder = EvolutionFinder(**params)\n\n\nresult_lis = []\nresult_valids = []\ninfo = []\nfor arth_intensity in [15,18, 20]:\n    st = time.time()\n    finder.set_efficiency_constraint(arth_intensity)\n    best_valids, best_info = finder.run_evolution_search()\n    ed = time.time()\n    \n    print('Found best architecture at arthemetic intensity >= %.2f M in %.2f seconds! It achieves %.2f%s predicted accuracy with arthemetic intensity of %.2f ops/byte./n' % (arth_intensity, ed-st, best_info[0] * 100, '%',1/ best_info[-1]))\n    result_lis.append(best_info)\n    result_valids.append(best_valids)\n    info.append(ed-st)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2024-01-18T11:27:14.062865Z","iopub.execute_input":"2024-01-18T11:27:14.06316Z","iopub.status.idle":"2024-01-18T12:13:07.898127Z","shell.execute_reply.started":"2024-01-18T11:27:14.063134Z","shell.execute_reply":"2024-01-18T12:13:07.896911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets visualize and evalute the found architecutre","metadata":{}},{"cell_type":"markdown","source":"Evaluation of the searched networks on imagnet10k","metadata":{}},{"cell_type":"code","source":"for arch in result_lis:\n    cfg = arch[1]\n    visualize_subnet(cfg)  \n    ofa_network.set_active_subnet(cfg['ks'], cfg['e'], cfg['d'])\n    network = ofa_network.get_active_subnet()\n    acc = evaluate_sub(network)\n    \n    print('The evaluated accuracy is : {}'.format(acc))\n    print('--'*50)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:13:07.899416Z","iopub.execute_input":"2024-01-18T12:13:07.899722Z","iopub.status.idle":"2024-01-18T12:14:44.33633Z","shell.execute_reply.started":"2024-01-18T12:13:07.899696Z","shell.execute_reply":"2024-01-18T12:14:44.335157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"archs = [\n['Efficientnet B0', 77.8, 19.5],\n['Mobilenet V3_large', 77, 11.5],\n['nvidia_efficientnet_b4',82, 20.3]\n]","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:14:44.338222Z","iopub.execute_input":"2024-01-18T12:14:44.338685Z","iopub.status.idle":"2024-01-18T12:14:44.344457Z","shell.execute_reply.started":"2024-01-18T12:14:44.33864Z","shell.execute_reply":"2024-01-18T12:14:44.343425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ny = [res[0]*100 for res in result_lis]+ [79.5] #added evaulated accuracy after one epcoh train\nx = [1/res[2] for res in result_lis]+ [20.1]\n\n\n\nplt.figure(figsize=(12, 8))\nplt.xlabel('Arithmetic Intensity (ops/byte)', fontsize=14)\nplt.ylabel('Predicted Accuracy', fontsize=14)\n\n# Plot the line and scatter plot with 'X' marker for each point in the same color\nplt.plot(x, y, marker='X', linestyle='-', color='blue', markersize=8, label = 'Ours')\nplt.plot(archs[0][2], archs[0][1], marker='*', color='red', label ='other', markersize=10 )\nplt.plot(archs[1][2],archs[1][1], marker = '*', color = 'red', markersize = 10)\nplt.plot(archs[2][2],archs[2][1], marker = '*', color = 'red', markersize = 10)\n\n\n# Annotate the middle point\nplt.text(20.2,79, f' Evaluated Accuracy \\n after 1 epoch', fontsize = 10)\nplt.text(17, 85.2, f'      FPGA specialized networks', fontsize=12, verticalalignment='bottom', horizontalalignment='left', color='blue')\nplt.text(archs[0][2], archs[0][1], f'  Efficientnet_b0', fontsize=12 )\nplt.text(archs[1][2],archs[1][1], f' {archs[1][0]}')\nplt.text(archs[2][2]+0.1,archs[2][1], f' {archs[2][0]}', fontsize=12)\n\nplt.ylim(75, 89)\nplt.xlim(10, 24)\n# Add legend\nplt.legend(fontsize=12)\nplt.title('Searching for FPGAs')\n# Show the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:14:44.3457Z","iopub.execute_input":"2024-01-18T12:14:44.345991Z","iopub.status.idle":"2024-01-18T12:14:44.734848Z","shell.execute_reply.started":"2024-01-18T12:14:44.345966Z","shell.execute_reply":"2024-01-18T12:14:44.733888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Retrain the network\nLets fine tune the network on a subset of training set of imagenet.  \nWe will retrain the third searched model with ops/byte>20","metadata":{}},{"cell_type":"code","source":"#extracting the model, \ncfg = result_lis[2][1]\nofa_network.set_active_subnet(cfg['ks'], cfg['e'], cfg['d'])\nnetwork = ofa_network.get_active_subnet()\nprint('Predicted accuracy: {} \\nArthemetic Intensity of searched Model:{}'.format(result_lis[2][0]*100,1/result_lis[2][2]) )","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:14:44.73822Z","iopub.execute_input":"2024-01-18T12:14:44.73855Z","iopub.status.idle":"2024-01-18T12:14:44.85641Z","shell.execute_reply.started":"2024-01-18T12:14:44.738521Z","shell.execute_reply":"2024-01-18T12:14:44.855396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prepartion for training datase","metadata":{}},{"cell_type":"code","source":"#lets train model in Imagenet 100k\n\nbatch_size = 32\n\ntrain_path = '/kaggle/working/imagenet_subtrain'\n\ntrain_data = datasets.ImageFolder(\n            root= train_path,\n            transform=build_val_transform(224)\n        )\n\ntrain_loader = torch.utils.data.DataLoader(\n        train_data,\n        batch_size=batch_size, \n        shuffle = True,\n        num_workers=4,  \n        pin_memory=True,\n        drop_last=False,\n    )\n\nprint('The ImageNet train subset is ready. Size : {}'.format(len(train_loader)*batch_size))\n","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:14:44.857834Z","iopub.execute_input":"2024-01-18T12:14:44.858221Z","iopub.status.idle":"2024-01-18T12:14:45.340618Z","shell.execute_reply.started":"2024-01-18T12:14:44.858184Z","shell.execute_reply":"2024-01-18T12:14:45.339604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training Code to train a network **","metadata":{}},{"cell_type":"code","source":"\n#Function for training the model, forware prop and backward prop \n\ndef train_model(model, criterion, optimizer, scheduler, num_epochs=1):\n    since = time.time()\n\n    #storing epoch data\n    epoch_data =     {\n        'epoch': [],\n        'train': {'loss': [], 'acc': []},\n        'val': {'loss': [], 'acc': [] }\n    }\n    \n    # Create a temporary directory in Kaggle's temp directory\n    tempdir = '/kaggle/working/temp'\n    os.makedirs(tempdir, exist_ok=True)\n    best_model_params_path = os.path.join(tempdir, 'best_model_params.pt')\n\n    torch.save(model.state_dict(), best_model_params_path)\n    best_acc = 0.0\n\n    for epoch in range(num_epochs):\n        print(f'Epoch {epoch+1}/{num_epochs}')\n        print('-' * 10)\n        epoch_data['epoch'].append(epoch+1)\n        \n        for phase in ['train', 'val']:\n            if phase == 'train':\n                model.train()\n            else:\n                model.eval()\n            running_loss = 0.0\n            running_corrects = 0\n\n            \n\n            for inputs, labels in tqdm(dataloaders[phase], leave=False):\n                inputs = inputs.to(device)\n                labels = labels.to(device)\n\n                optimizer.zero_grad()\n\n                with torch.set_grad_enabled(phase == 'train'):\n                    outputs = model(inputs)\n                    _, preds = torch.max(outputs, 1)\n                    loss = criterion(outputs, labels)\n\n                    if phase == 'train':\n                        loss.backward()\n                        optimizer.step()\n\n                running_loss += loss.item() * inputs.size(0)\n                running_corrects += torch.sum(preds == labels.data)\n\n            if phase == 'train':\n                scheduler.step()\n\n            epoch_loss = running_loss / dataset_sizes[phase]\n            epoch_acc = running_corrects.double() / dataset_sizes[phase]\n            epoch_data[phase]['loss'].append(epoch_loss)\n            epoch_data[phase]['acc'].append(epoch_acc)\n\n            print(f'{phase} Loss: {epoch_loss:.4f} Acc: {epoch_acc:.4f}')\n\n            if phase == 'val' and epoch_acc > best_acc:\n                best_acc = epoch_acc\n                torch.save(model.state_dict(), best_model_params_path)\n\n        print()\n\n    time_elapsed = time.time() - since\n    print(f'Training complete in {time_elapsed // 60:.0f}m {time_elapsed % 60:.0f}s')\n    print(f'Best val Acc: {best_acc:4f}')\n\n    model.load_state_dict(torch.load(best_model_params_path))\n\n    # Clean up the temporary directory\n    shutil.rmtree(tempdir)\n\n    return model, epoch_data\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2024-01-18T12:14:45.342055Z","iopub.execute_input":"2024-01-18T12:14:45.342383Z","iopub.status.idle":"2024-01-18T12:14:45.357225Z","shell.execute_reply.started":"2024-01-18T12:14:45.342336Z","shell.execute_reply":"2024-01-18T12:14:45.356153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataloaders = {}\ndataloaders['train'] = train_loader\ndataloaders['val'] = val_loader\n\ndataset_sizes = {'train': len(train_loader)*32,\n                'val': len(val_loader)*32}","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:14:45.358392Z","iopub.execute_input":"2024-01-18T12:14:45.358677Z","iopub.status.idle":"2024-01-18T12:14:45.37225Z","shell.execute_reply.started":"2024-01-18T12:14:45.358653Z","shell.execute_reply":"2024-01-18T12:14:45.37141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Lets try retraining the model with ops/byte > 20**","metadata":{}},{"cell_type":"code","source":"device = 'cuda' if cuda_available else 'cpu'\n\nmodel = network.to(device)\ncriterion = nn.CrossEntropyLoss()\n\n# Observe that all parameters are being optimized\noptimizer_ft = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)\n\n# Decay LR by a factor of 0.1 every 7 epochs\nexp_lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:14:45.373491Z","iopub.execute_input":"2024-01-18T12:14:45.373792Z","iopub.status.idle":"2024-01-18T12:14:45.408389Z","shell.execute_reply.started":"2024-01-18T12:14:45.373761Z","shell.execute_reply":"2024-01-18T12:14:45.40773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.empty_cache()\nmodel, epoch_data = train_model(model, criterion, optimizer_ft, exp_lr_scheduler,\n                       num_epochs=1)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:14:45.409577Z","iopub.execute_input":"2024-01-18T12:14:45.410285Z","iopub.status.idle":"2024-01-18T12:29:11.393951Z","shell.execute_reply.started":"2024-01-18T12:14:45.410251Z","shell.execute_reply":"2024-01-18T12:29:11.392717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### It is clear that accuracy is recoverable and we can obtain comparable accuracy with our efficiency contraints by retraining for few epochs ","metadata":{}},{"cell_type":"markdown","source":"## 6. Model Conversion and Saving","metadata":{}},{"cell_type":"code","source":"\ntorch.save(model, '20.08ops_byte_79.pth')","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:29:11.395812Z","iopub.execute_input":"2024-01-18T12:29:11.396731Z","iopub.status.idle":"2024-01-18T12:29:11.545Z","shell.execute_reply.started":"2024-01-18T12:29:11.396684Z","shell.execute_reply":"2024-01-18T12:29:11.543814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%script echo skipping\nimport torch.onnx \n\n#Function to Convert to ONNX \ndef Convert_ONNX(model, in_shape, name): \n\n    # set the model to inference mode \n    model.eval() \n\n    # Let's create a dummy input tensor  \n\n    dummy_input = torch.randn(in_shape, requires_grad=True)  \n\n    # Export the model   \n    torch.onnx.export(model,         # model being run \n         dummy_input,       # model input (or a tuple for multiple inputs) \n         f\"{name}.onnx\",       # where to save the model  \n         export_params=True,  # store the trained parameter weights inside the model file \n         opset_version=10,    # the ONNX version to export the model to \n         do_constant_folding=True,  # whether to execute constant folding for optimization \n         input_names = ['modelInput'],   # the model's input names \n         output_names = ['modelOutput'], # the model's output names \n         dynamic_axes={'modelInput' : {0 : 'batch_size'},    # variable length axes \n                                'modelOutput' : {0 : 'batch_size'}}) \n    print(\" \") \n    print('Model has been converted to ONNX')","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:29:11.54655Z","iopub.execute_input":"2024-01-18T12:29:11.546923Z","iopub.status.idle":"2024-01-18T12:29:11.558823Z","shell.execute_reply.started":"2024-01-18T12:29:11.546888Z","shell.execute_reply":"2024-01-18T12:29:11.557924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"{'ResNet-152':81.3,\n'Efficientnet V2 Small': 83.6,}","metadata":{"execution":{"iopub.status.busy":"2024-01-18T12:36:30.251917Z","iopub.execute_input":"2024-01-18T12:36:30.252309Z","iopub.status.idle":"2024-01-18T12:36:30.259175Z","shell.execute_reply.started":"2024-01-18T12:36:30.252275Z","shell.execute_reply":"2024-01-18T12:36:30.258219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}}]}