{"cells":[{"metadata":{},"cell_type":"markdown","source":"### This notebook explores what we have in the DFDC full train set metadata and json files."},{"metadata":{},"cell_type":"markdown","source":"The dataset for this notebook  is at\nhttps://www.kaggle.com/zaharch/train-set-metadata-for-dfdc\n\nThe train data for this competition is big, almost 500Gb, so I hope it can be useful to have all the json files and the metadata in one dataframe.\n\nThe dataset includes, for each video file\n1. Info from the json files: **filename**, **folder**, **label**, **original**\n2. **split**: train (118346 videos), public validation test (400 videos) or train sample (400 videos). 119146 videos in total. Note that the public validation and the train sample are subsets of the full train, so it is enough to mark them in this dataframe.\n3. Full file **md5** column\n4. Hash on audio file sequence **wav.hash** and on subset of pixels **pxl.hash**\n5. The rest are metadata fields from the files, obtained with ffprobe. Note that I removed many columns, which didn't give new information."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\npd.set_option('display.max_columns', 100)\npd.set_option('display.max_rows', 100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/train-set-metadata-for-dfdc/metadata', low_memory=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Hashes"},{"metadata":{},"cell_type":"markdown","source":"Fakes always have at least some pixel-level changes. That means that all audio fakes are also video fakes."},{"metadata":{"trusted":true},"cell_type":"code","source":"(data['pxl.hash'] == data['pxl.hash.orig']).value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are duplicated for both **md5**, **pxl.hash** and **wav.hash**. Duplicates for **wav.hash** are OK, but duplicates for **md5** mean that there are identical files in the dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"data['md5'].value_counts().value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['wav.hash'].value_counts().value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['pxl.hash'].value_counts().value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Other fields"},{"metadata":{},"cell_type":"markdown","source":"This is how the data looks like"},{"metadata":{"trusted":true},"cell_type":"code","source":"data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.split.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set(data.original) - set(data.filename)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set(data.loc[data.original == 'NAN', 'filename']) - set(data.original)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.loc[data.original != 'NAN', 'original'].value_counts().hist(bins=40)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.loc[data.original != 'NAN', 'original'].value_counts().value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"for col in data.columns:\n    print(pd.crosstab(data[col],data['label']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.crosstab(data['video.@display_aspect_ratio'],data['label'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.crosstab([data['video.@display_aspect_ratio'], data['audio.@codec_time_base']],data['label'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}