{"cells":[{"metadata":{},"cell_type":"markdown","source":"# This notebook\nThis notebook is based in several other notebooks. I want to thank the authors of these notebooks for sharing their thoughts and code.\n\n- https://www.kaggle.com/zaharch/train-set-metadata-for-dfdc\n- https://www.kaggle.com/basharallabadi/dfdc-video-audio-labels\n- https://www.kaggle.com/rakibilly/extract-audio-starter\n\nThis notebook outputs deepfake labels by separating audio and video deepfakes. In addition, at the end of the notebook I've included code for visualizing a sample frame from a video and code for creating an audio listening interface."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport random\nimport subprocess\nfrom pathlib import Path\nimport IPython\n\nimport seaborn as sns\nimport cv2\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\ntqdm.pandas()\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Separating video from audio labels\nNow, the original competition metadata only contains either the label \"FAKE\" or \"REAL\" for a given video. Therefore, in this notebook we will categorize the type of deepfake, i.e. either **audio deepfake** or **video deepfake**. \n\n- Add this to your dataset: https://www.kaggle.com/zaharch/train-set-metadata-for-dfdc\n- The functions used are based on: https://www.kaggle.com/basharallabadi/dfdc-video-audio-labels\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"metadata = pd.read_csv('../input/train-set-metadata-for-dfdc/metadata', low_memory=False)\nmetadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def audio_label(row): \n    if row['label'] == 'REAL':\n        return 'REAL'\n    if row['wav.hash'] != row['wav.hash.orig'] and row['audio.@codec_time_base'] != '1/16000':\n        return 'FAKE'\n    return 'REAL'\n\ndef video_label(row):\n    if row['label'] == 'REAL':\n        return 'REAL'\n    if row['pxl.hash'] != row['pxl.hash.orig']:\n        return 'FAKE'\n    return 'REAL'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata[\"video_label\"] = metadata.progress_apply(video_label, axis=1)\nmetadata[\"audio_label\"] = metadata.progress_apply(audio_label, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clean_labels = metadata[[\"filename\", \"video_label\", \"audio_label\"]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploratory Data Analysis\n\nLet's understand this data a little! We start by looking at the **distribution of labels**."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set_style('darkgrid')\n\nplt.figure(figsize=(12,5))\nplt.title('Label Distribution')\n\nplt.subplot(1, 3, 1)\nax1 = sns.countplot(metadata[\"video_label\"], order=[\"REAL\", \"FAKE\"])\nplt.subplot(1, 3, 2)\nax2 = sns.countplot(metadata[\"audio_label\"], order=[\"REAL\", \"FAKE\"])\n\nunion_label = metadata[\"video_label\"].str.cat(metadata[\"audio_label\"], sep=\"_\")\n\nplt.subplot(1, 3, 3)\nax3 = sns.countplot(union_label)\n\nax1.set_ylim(0, 120000)\nax2.set_ylim(0, 120000)\nax3.set_ylim(0, 120000)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that all audio deepfakes are also image deepfakes. There are no audio deepfakes that are not image deepfakes."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Number of both FAKE video and FAKE audio: {len(union_label[union_label == 'FAKE_FAKE'])}\")\nprint(f\"Number of only FAKE audio: {len(metadata[metadata['audio_label']=='FAKE'])}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_audio_fakes = metadata[\"audio_label\"].value_counts()[\"FAKE\"]\nprint(f\"We only have {num_audio_fakes} fake audio samples. It is undersampled in comparison to other labels.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/deepfake-detection-challenge/train_sample_videos/\"\nvideos = [os.path.join(path, video) for video in os.listdir(path)]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"def get_video_label(path, metadata):\n    filename = os.path.basename(path)\n    data = metadata[metadata[\"filename\"] == filename]\n    return data[\"video_label\"]\n\ndef get_audio_label(path, metadata):\n    filename = os.path.basename(path)\n    data = metadata[metadata[\"filename\"] == filename]\n    return data[\"audio_label\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Audio reading"},{"metadata":{},"cell_type":"markdown","source":"Static Build of ffmpeg: https://johnvansickle.com/ffmpeg/ <- internet is not available.\nThe public data set: https://www.kaggle.com/rakibilly/ffmpeg-static-build\n\nThis kernel helped me alot https://www.kaggle.com/rakibilly/extract-audio-starter"},{"metadata":{"trusted":true},"cell_type":"code","source":"! tar xvf ../input/ffmpeg-static-build/ffmpeg-git-amd64-static.tar.xz","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_audio(file, save_path):\n    command = f\"../working/ffmpeg-git-20191209-amd64-static/ffmpeg -i {file} -ab 192000 -ac 2 -ar 44100 -vn {save_path}\"\n    subprocess.call(command, shell=True)\n    \noutput_format = \"mp3\"\noutput_dir = Path(f\"mp3_files\")\nPath(output_dir).mkdir(exist_ok=True, parents=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Video reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_random_frame(path):\n    cap = cv2.VideoCapture(path)\n    cap.set(cv2.CAP_PROP_POS_FRAMES, random.uniform(0, 1))\n    _, img = cap.read()\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\ndef visualize_sample(sample):\n    video_label = get_video_label(sample, clean_labels)\n    audio_label = get_audio_label(sample, clean_labels)\n\n    # Read random image\n    img = get_random_frame(sample)\n    plt.imshow(img)\n    plt.title(f\"Video: {video_label.item()}, Audio: {audio_label.item()}\") # .item() works as of now 16/03/2020, but will be removed in the future\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = videos[143]\n\n# Visualize random frame\nvisualize_sample(sample)\n\n# Read audio\naudio_file = f\"{output_dir/sample[-14:-4]}.{output_format}\"\ncreate_audio(sample, audio_file)\nIPython.display.Audio(audio_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = videos[203]\n\n# Visualize random frame\nvisualize_sample(sample)\n\n# Read audio\naudio_file = f\"{output_dir/sample[-14:-4]}.{output_format}\"\ncreate_audio(sample, audio_file)\nIPython.display.Audio(audio_file)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}