{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# This notebook\nThis notebook is based in several other notebooks. I want to thank the authors of these notebooks for sharing their thoughts and code.\n\n- https://www.kaggle.com/zaharch/train-set-metadata-for-dfdc\n- https://www.kaggle.com/basharallabadi/dfdc-video-audio-labels\n- https://www.kaggle.com/rakibilly/extract-audio-starter\n\nThis notebook outputs deepfake labels by separating audio and video deepfakes. In addition, at the end of the notebook I've included code for visualizing a sample frame from a video and code for creating an audio listening interface.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport random\nimport subprocess\nfrom pathlib import Path\nimport IPython\n\nimport seaborn as sns\nimport cv2\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\ntqdm.pandas()\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-06-29T06:01:38.590770Z","iopub.execute_input":"2022-06-29T06:01:38.591069Z","iopub.status.idle":"2022-06-29T06:01:41.033988Z","shell.execute_reply.started":"2022-06-29T06:01:38.591022Z","shell.execute_reply":"2022-06-29T06:01:41.033033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Separating video from audio labels\nNow, the original competition metadata only contains either the label \"FAKE\" or \"REAL\" for a given video. Therefore, in this notebook we will categorize the type of deepfake, i.e. either **audio deepfake** or **video deepfake**. \n\n- Add this to your dataset: https://www.kaggle.com/zaharch/train-set-metadata-for-dfdc\n- The functions used are based on: https://www.kaggle.com/basharallabadi/dfdc-video-audio-labels\n","metadata":{}},{"cell_type":"code","source":"metadata = pd.read_csv('../input/train-set-metadata-for-dfdc/metadata', low_memory=False)\nmetadata.head()","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2022-06-29T06:01:41.035522Z","iopub.execute_input":"2022-06-29T06:01:41.035728Z","iopub.status.idle":"2022-06-29T06:01:42.880456Z","shell.execute_reply.started":"2022-06-29T06:01:41.035683Z","shell.execute_reply":"2022-06-29T06:01:42.879770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def audio_label(row): \n    if row['label'] == 'REAL':\n        return 'REAL'\n    if row['wav.hash'] != row['wav.hash.orig'] and row['audio.@codec_time_base'] != '1/16000':\n        return 'FAKE'\n    return 'REAL'\n\ndef video_label(row):\n    if row['label'] == 'REAL':\n        return 'REAL'\n    if row['pxl.hash'] != row['pxl.hash.orig']:\n        return 'FAKE'\n    return 'REAL'","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:42.882192Z","iopub.execute_input":"2022-06-29T06:01:42.882696Z","iopub.status.idle":"2022-06-29T06:01:42.890121Z","shell.execute_reply.started":"2022-06-29T06:01:42.882647Z","shell.execute_reply":"2022-06-29T06:01:42.889131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metadata[\"video_label\"] = metadata.progress_apply(video_label, axis=1)\nmetadata[\"audio_label\"] = metadata.progress_apply(audio_label, axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:42.891563Z","iopub.execute_input":"2022-06-29T06:01:42.891791Z","iopub.status.idle":"2022-06-29T06:01:51.771402Z","shell.execute_reply.started":"2022-06-29T06:01:42.891746Z","shell.execute_reply":"2022-06-29T06:01:51.770177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clean_labels = metadata[[\"filename\", \"video_label\", \"audio_label\"]]","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:51.774403Z","iopub.execute_input":"2022-06-29T06:01:51.774682Z","iopub.status.idle":"2022-06-29T06:01:51.849497Z","shell.execute_reply.started":"2022-06-29T06:01:51.774621Z","shell.execute_reply":"2022-06-29T06:01:51.848597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis\n\nLet's understand this data a little! We start by looking at the **distribution of labels**.","metadata":{}},{"cell_type":"code","source":"sns.set_style('darkgrid')\n\nplt.figure(figsize=(12,5))\nplt.title('Label Distribution')\n\nplt.subplot(1, 3, 1)\nax1 = sns.countplot(metadata[\"video_label\"], order=[\"REAL\", \"FAKE\"])\nplt.subplot(1, 3, 2)\nax2 = sns.countplot(metadata[\"audio_label\"], order=[\"REAL\", \"FAKE\"])\n\nunion_label = metadata[\"video_label\"].str.cat(metadata[\"audio_label\"], sep=\"_\")\n\nplt.subplot(1, 3, 3)\nax3 = sns.countplot(union_label)\n\nax1.set_ylim(0, 120000)\nax2.set_ylim(0, 120000)\nax3.set_ylim(0, 120000)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:51.850707Z","iopub.execute_input":"2022-06-29T06:01:51.850991Z","iopub.status.idle":"2022-06-29T06:01:52.217184Z","shell.execute_reply.started":"2022-06-29T06:01:51.850933Z","shell.execute_reply":"2022-06-29T06:01:52.216224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems that all audio deepfakes are also image deepfakes. There are no audio deepfakes that are not image deepfakes.","metadata":{}},{"cell_type":"code","source":"print(f\"Number of both FAKE video and FAKE audio: {len(union_label[union_label == 'FAKE_FAKE'])}\")\nprint(f\"Number of only FAKE audio: {len(metadata[metadata['audio_label']=='FAKE'])}\")","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:52.218373Z","iopub.execute_input":"2022-06-29T06:01:52.218619Z","iopub.status.idle":"2022-06-29T06:01:52.247460Z","shell.execute_reply.started":"2022-06-29T06:01:52.218576Z","shell.execute_reply":"2022-06-29T06:01:52.246943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_audio_fakes = metadata[\"audio_label\"].value_counts()[\"FAKE\"]\nprint(f\"We only have {num_audio_fakes} fake audio samples. It is undersampled in comparison to other labels.\")","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:52.248837Z","iopub.execute_input":"2022-06-29T06:01:52.249120Z","iopub.status.idle":"2022-06-29T06:01:52.263570Z","shell.execute_reply.started":"2022-06-29T06:01:52.249056Z","shell.execute_reply":"2022-06-29T06:01:52.262938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = \"../input/deepfake-detection-challenge/train_sample_videos/\"\nvideos = [os.path.join(path, video) for video in os.listdir(path)]","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:52.264827Z","iopub.execute_input":"2022-06-29T06:01:52.265101Z","iopub.status.idle":"2022-06-29T06:01:52.413680Z","shell.execute_reply.started":"2022-06-29T06:01:52.265038Z","shell.execute_reply":"2022-06-29T06:01:52.413118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_video_label(path, metadata):\n    filename = os.path.basename(path)\n    data = metadata[metadata[\"filename\"] == filename]\n    return data[\"video_label\"]\n\ndef get_audio_label(path, metadata):\n    filename = os.path.basename(path)\n    data = metadata[metadata[\"filename\"] == filename]\n    return data[\"audio_label\"]","metadata":{"_cell_guid":"","_uuid":"","execution":{"iopub.status.busy":"2022-06-29T06:01:52.414774Z","iopub.execute_input":"2022-06-29T06:01:52.415204Z","iopub.status.idle":"2022-06-29T06:01:52.421165Z","shell.execute_reply.started":"2022-06-29T06:01:52.415159Z","shell.execute_reply":"2022-06-29T06:01:52.420667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Audio reading","metadata":{}},{"cell_type":"markdown","source":"Static Build of ffmpeg: https://johnvansickle.com/ffmpeg/ <- internet is not available.\nThe public data set: https://www.kaggle.com/rakibilly/ffmpeg-static-build\n\nThis kernel helped me alot https://www.kaggle.com/rakibilly/extract-audio-starter","metadata":{}},{"cell_type":"code","source":"! tar xvf ../input/ffmpeg-static-build/ffmpeg-git-amd64-static.tar.xz","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:52.422595Z","iopub.execute_input":"2022-06-29T06:01:52.422903Z","iopub.status.idle":"2022-06-29T06:01:56.444208Z","shell.execute_reply.started":"2022-06-29T06:01:52.422847Z","shell.execute_reply":"2022-06-29T06:01:56.443481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_audio(file, save_path):\n    command = f\"../working/ffmpeg-git-20191209-amd64-static/ffmpeg -i {file} -ab 192000 -ac 2 -ar 44100 -vn {save_path}\"\n    subprocess.call(command, shell=True)\n    \noutput_format = \"mp3\"\noutput_dir = Path(f\"mp3_files\")\nPath(output_dir).mkdir(exist_ok=True, parents=True)","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:56.445846Z","iopub.execute_input":"2022-06-29T06:01:56.446096Z","iopub.status.idle":"2022-06-29T06:01:56.452234Z","shell.execute_reply.started":"2022-06-29T06:01:56.446049Z","shell.execute_reply":"2022-06-29T06:01:56.451122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Video reading","metadata":{}},{"cell_type":"code","source":"def get_random_frame(path):\n    cap = cv2.VideoCapture(path)\n    cap.set(cv2.CAP_PROP_POS_FRAMES, random.uniform(0, 1))\n    _, img = cap.read()\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\ndef visualize_sample(sample):\n    video_label = get_video_label(sample, clean_labels)\n    audio_label = get_audio_label(sample, clean_labels)\n\n    # Read random image\n    img = get_random_frame(sample)\n    plt.imshow(img)\n    plt.title(f\"Video: {video_label.item()}, Audio: {audio_label.item()}\") # .item() works as of now 16/03/2020, but will be removed in the future\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:56.453583Z","iopub.execute_input":"2022-06-29T06:01:56.453909Z","iopub.status.idle":"2022-06-29T06:01:56.461243Z","shell.execute_reply.started":"2022-06-29T06:01:56.453746Z","shell.execute_reply":"2022-06-29T06:01:56.460223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualization","metadata":{}},{"cell_type":"code","source":"sample = videos[100]\n\n# Visualize random frame\nvisualize_sample(sample)\n\n# Read audio\naudio_file = f\"{output_dir/sample[-14:-4]}.{output_format}\"\ncreate_audio(sample, audio_file)\nIPython.display.Audio(audio_file)","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:02:49.474722Z","iopub.execute_input":"2022-06-29T06:02:49.474992Z","iopub.status.idle":"2022-06-29T06:02:50.281871Z","shell.execute_reply.started":"2022-06-29T06:02:49.474960Z","shell.execute_reply":"2022-06-29T06:02:50.281046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = videos[203]\n\n# Visualize random frame\nvisualize_sample(sample)\n\n# Read audio\naudio_file = f\"{output_dir/sample[-14:-4]}.{output_format}\"\ncreate_audio(sample, audio_file)\nIPython.display.Audio(audio_file)","metadata":{"execution":{"iopub.status.busy":"2022-06-29T06:01:57.435499Z","iopub.execute_input":"2022-06-29T06:01:57.435754Z","iopub.status.idle":"2022-06-29T06:01:57.920746Z","shell.execute_reply.started":"2022-06-29T06:01:57.435708Z","shell.execute_reply":"2022-06-29T06:01:57.920182Z"},"trusted":true},"execution_count":null,"outputs":[]}]}