{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this kernel goal is to simply extract video batches to be fed to a model. For this we will compare vanilla open-cv. FileVideoStream from imutils and excellent latest contribution: Decord from https://www.kaggle.com/leighplt/decord-videoreader/notebook.\n\n### Steps\n\n- Read frames with sample rate 10 (every 10th starting from 0th index) ~ batch of 30 frames per video\n- Resize frames\n- Convert to pytorch batch "},{"metadata":{},"cell_type":"markdown","source":"TODO: Add DALI as a dataset, or if it's available let me know :)"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/cuda/10.0 nvidia-dali\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Intro"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from fastai.vision import *\nimport cv2 as cv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_sample_metadata = pd.read_json('../input/deepfake-detection-challenge/train_sample_videos/metadata.json').T.reset_index()\ntrain_sample_metadata.columns = ['fname','label','split','original']\ntrain_sample_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fake_sample_df = train_sample_metadata[train_sample_metadata.label == 'FAKE']\nreal_sample_df = train_sample_metadata[train_sample_metadata.label == 'REAL']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dir = Path('/kaggle/input/deepfake-detection-challenge/train_sample_videos/')\ntest_dir = Path('/kaggle/input/deepfake-detection-challenge/test_videos/')\ntrain_video_files = get_files(train_dir, extensions=['.mp4'])\ntest_video_files = get_files(test_dir, extensions=['.mp4'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train_video_files), len(test_video_files)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Nvidia DALI"},{"metadata":{"trusted":true},"cell_type":"code","source":"from nvidia.dali.pipeline import Pipeline\nfrom nvidia.dali import ops\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fname = train_video_files[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size=1\nsequence_length=30\ninitial_prefetch_size=16\n\nclass VideoPipe(Pipeline):\n    \"video pipeline for a single video with 30 frames\"\n    def __init__(self, batch_size, num_threads, device_id, data, shuffle):\n        super(VideoPipe, self).__init__(batch_size, num_threads, device_id, seed=16)\n        self.input = ops.VideoReader(device=\"gpu\", filenames=data, sequence_length=sequence_length,\n                                     shard_id=0, num_shards=1,\n                                     random_shuffle=shuffle, initial_fill=initial_prefetch_size)\n    def define_graph(self):\n        output = self.input(name=\"Reader\")\n        return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def dali_batch(fname):\n    pipe = VideoPipe(batch_size=batch_size, num_threads=defaults.cpus, device_id=0, data=[fname], shuffle=False)\n    pipe.build()\n    pipe_out = pipe.run()\n    sequences_out = pipe_out[0].as_cpu().as_array()\n    data = torch.from_numpy(sequences_out[0])\n    data = data.permute(0,3,1,2).cuda()\n    return F.interpolate(data.to(torch.float32), (640,640))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndata = dali_batch(train_video_files[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img0 = Image(data[0]/255)\nimg0.show(figsize=(10,10))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Vanilla opencv"},{"metadata":{"trusted":true},"cell_type":"code","source":"def frame_img_generator(path, freq=None):\n    \"frame generator for a given video file\"\n    vidcap = cv.VideoCapture(str(path))\n    n_frames = 0\n    while True:\n        success = vidcap.grab()\n        if not success: \n            vidcap.release()\n            break   \n            \n        if (freq is None) or (n_frames % freq == 0):\n            _, frame = vidcap.retrieve()\n            frame = cv.cvtColor(frame, cv.COLOR_BGR2RGB)\n#             frame = cv.resize(frame, (640,640))\n            yield frame    \n        \n        n_frames += 1\n        \n    vidcap.release()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# CPU warm up\nframes = list(frame_img_generator(train_video_files[0], 10)); len(frames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nframes = list(frame_img_generator(train_video_files[0], freq=10)); len(frames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nframes = frame_img_generator(train_video_files[0], 10)\ndata = torch.from_numpy(array(frames))\ndata = data.permute(0,3,1,2).cuda()\ndata = F.interpolate(data.to(torch.float32), (640,640))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img1 = Image(data[0]/255)\nimg1.show(figsize=(10,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del frames\ndel data; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### imutils: FileVideoStream"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q /kaggle/input/imutils/imutils-0.5.3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from imutils.video import FileVideoStream","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fvs_img_generator(path, freq=None):\n    \"frame generator for a given video file\"\n    fvs = FileVideoStream(str(path)).start()\n    n_frames = 0\n    while fvs.more():\n        frame = fvs.read()\n        if frame is None: break # https://github.com/jrosebr1/imutils/pull/119\n        frame = cv.cvtColor(frame, cv.COLOR_BGR2RGB)\n        \n        if (freq is None) or (n_frames % freq == 0):\n            yield frame\n        \n        n_frames += 1\n    fvs.stop()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nframes = list(fvs_img_generator(str(train_video_files[0]), 10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nframes = list(fvs_img_generator(str(train_video_files[0]), 10))\ndata = torch.from_numpy(array(frames))\ndata = data.permute(0,3,1,2).cuda()\ndata = F.interpolate(data.to(torch.float32), (640,640))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img2 = Image(data[0]/255)\nimg2.show(figsize=(10,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del frames\ndel data; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert torch.all(img1.data == img2.data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Decord Reader GPU\n\nThanks to: https://www.kaggle.com/leighplt/decord-videoreader/data"},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp /kaggle/input/decord/install.sh . && chmod  +x install.sh && ./install.sh ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sys.path.insert(0,'/kaggle/working/reader/python')\n\nfrom decord import VideoReader\nfrom decord import cpu, gpu\nfrom decord.bridge import set_bridge\nset_bridge('torch')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# GPU warm up\nvideo = VideoReader(str(train_video_files[0]), ctx=gpu())\ndel video; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nvideo = VideoReader(str(train_video_files[0]), ctx=gpu())\ndata = video.get_batch(range(0, len(video), 10))\ndata = F.interpolate(data.to(torch.float32), (640,640))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img3 = Image(data[0]/255)\nimg3.show(figsize=(10,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del video\ndel data; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One thing we can notice is that Decord GPU is not given exactly same results whereas previous 2 methods give exact same pixel level results. Let's check how close both results are.\n\n98% of the pixels are within -+0.01 difference."},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.mean(torch.isclose(img1.data, img3.data, atol=0.01).float())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Decord Reader CPU\n\nOn CPU we don't see any pixel level difference but it's slower."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nvideo = VideoReader(str(train_video_files[0]), ctx=cpu())\ndata = video.get_batch(range(0, len(video), 10)).cuda()\ndata = F.interpolate(data.to(torch.float32), (640,640))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img4 = Image(data[0]/255)\nimg4.show(figsize=(10,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del video\ndel data; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert torch.all(img1.data == img4.data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"NVIDIA Dali seems to be more accurate"},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.mean(torch.isclose(img0.data, img1.data, atol=0.01).float())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Conclusion\n\n- Decoder GPU gives a really good boost with litle cost of ~%3 of deviated pixels within (-0.01, 0.01) range. That's a risk I am willing to take :)\n- FileVideoStream isn't much different than open-cv probably we do resizing on a GPU and don't have much CPU bound processing to get full power of threading\n- Let me know if there are anything I am mising\n- As discussed on https://www.kaggle.com/leighplt/decord-videoreader/notebook memory leaks can occur, so garbage collection is important. I also recommend using https://github.com/stas00/ipyexperiments/blob/master/docs/ipyexperiments.md\n- Feedback are welcome!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}