I wanted to use @phunghieu 's datasets for this comp but he wasn't able to finish it and it is logistically to large (11.8+ million images, 800+ GB for just the first 15 parts of the data set). So I went ahead and finished the data set:
\n\nAll datasets contain jpg images to conserve space
\n\nPart 16: https://www.kaggle.com/greatgamedota/dfdc-part-16 (121 MB)
\n\nPart 17: https://www.kaggle.com/greatgamedota/dfdc-part-17 (1GB)
\n\nPart 18: https://www.kaggle.com/greatgamedota/dfdc-part-18 (158 MB)\n...\nPart 32: https://www.kaggle.com/greatgamedota/dfdc-part-32 (150 MB)
\n\nPart 33: https://www.kaggle.com/greatgamedota/dfdc-part-33 (139 MB)
\n\nPart 34: https://www.kaggle.com/greatgamedota/dfdc-part-34 (147 MB)\n...\nPart 49: https://www.kaggle.com/greatgamedota/dfdc-part-49 (189 MB)
\n\n```
\n\nmeta = glob.glob('../input/deepfake-detection-faces-/.csv')\nmeta.sort(key=lambda f: int(re.sub('\\D', '', f)))
\n\ndfs = []\nfor path in meta:\n df = pd.read_csv(path)\n df['path'] = ''\n path = path.split(\"/\")[:-1]\n path = path[0] + '/' + path[1] + '/' + path[2] + '/'\n for i in range(len(df)):\n df.loc[i]['path'] = f'{path}{df.loc[i][\"filename\"][:-4]}'\n dfs.append(df)
\n\ntrain_df = pd.concat(dfs)\ntrain_df = train_df.reset_index(drop=True)\nprint(len(train_df))\n```
\n\n```
\n\npart = 16\nfor j in range((49-16)+1):\n if part+j != 17:\n meta = pd.read_csv(f'../input/dfdc-part-{part+j}/images/metadata{part+j}.csv')\n else:\n meta = pd.read_csv(f'../input/dfdc-part-{part+j}/images/metadata{part+j}.json', index_col=0)\n meta['path'] = ''\n print(part+j)\n del_idxs = []\n for i in range(len(meta)):\n if os.path.isdir(f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}'):\n if len(os.listdir(f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}')) < 5:\n del_idxs.append(i)\n else:\n meta.loc[i]['path'] = f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}'\n else:\n del_idxs.append(i)\n print(del_idxs) # Print out removed video ids\n for idx in del_idxs:\n meta = meta.drop(idx)\n train_df = pd.concat([train_df,meta])\n train_df = train_df.reset_index(drop=True)\nprint(len(train_df)) # Should be ~120k if everything is successfully loaded\n```\nThe above snippet adds data that has at least 5 images for that video. (Fun fact: mobilenet >> facenet just based on the amount of videos facenet failed on and need to be removed)
\n\nI wanted to use @phunghieu 's datasets for this comp but he wasn't able to finish it and it is logistically to large (11.8+ million images, 800+ GB for just the first 15 parts of the data set). So I went ahead and finished the data set:
\n\nAll datasets contain jpg images to conserve space
\n\nPart 16: https://www.kaggle.com/greatgamedota/dfdc-part-16 (121 MB)
\n\nPart 17: https://www.kaggle.com/greatgamedota/dfdc-part-17 (1GB)
\n\nPart 18: https://www.kaggle.com/greatgamedota/dfdc-part-18 (158 MB)\n...\nPart 32: https://www.kaggle.com/greatgamedota/dfdc-part-32 (150 MB)
\n\nPart 33: https://www.kaggle.com/greatgamedota/dfdc-part-33 (139 MB)
\n\nPart 34: https://www.kaggle.com/greatgamedota/dfdc-part-34 (147 MB)\n...\nPart 49: https://www.kaggle.com/greatgamedota/dfdc-part-49 (189 MB)
\n\n```
\n\nmeta = glob.glob('../input/deepfake-detection-faces-/.csv')\nmeta.sort(key=lambda f: int(re.sub('\\D', '', f)))
\n\ndfs = []\nfor path in meta:\n df = pd.read_csv(path)\n df['path'] = ''\n path = path.split(\"/\")[:-1]\n path = path[0] + '/' + path[1] + '/' + path[2] + '/'\n for i in range(len(df)):\n df.loc[i]['path'] = f'{path}{df.loc[i][\"filename\"][:-4]}'\n dfs.append(df)
\n\ntrain_df = pd.concat(dfs)\ntrain_df = train_df.reset_index(drop=True)\nprint(len(train_df))\n```
\n\n```
\n\npart = 16\nfor j in range((49-16)+1):\n if part+j != 17:\n meta = pd.read_csv(f'../input/dfdc-part-{part+j}/images/metadata{part+j}.csv')\n else:\n meta = pd.read_csv(f'../input/dfdc-part-{part+j}/images/metadata{part+j}.json', index_col=0)\n meta['path'] = ''\n print(part+j)\n del_idxs = []\n for i in range(len(meta)):\n if os.path.isdir(f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}'):\n if len(os.listdir(f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}')) < 5:\n del_idxs.append(i)\n else:\n meta.loc[i]['path'] = f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}'\n else:\n del_idxs.append(i)\n print(del_idxs) # Print out removed video ids\n for idx in del_idxs:\n meta = meta.drop(idx)\n train_df = pd.concat([train_df,meta])\n train_df = train_df.reset_index(drop=True)\nprint(len(train_df)) # Should be ~120k if everything is successfully loaded\n```\nThe above snippet adds data that has at least 5 images for that video. (Fun fact: mobilenet >> facenet just based on the amount of videos facenet failed on and need to be removed)
\n\nGreat job, @greatgamedota 👍. I hope that other competitors can benefit from these datasets to continue to push their models' performance in this competition.
", "rawMarkdown": "Great job, @greatgamedota 👍. I hope that other competitors can benefit from these datasets to continue to push their models' performance in this competition.", "votes": null }, { "id": "766456", "postDate": "03/08/2020 06:52:22", "content": "This is amazing work @greatgamedota. Thanks for putting all the hard work to create and upload this dataset for people who are using kaggle only for this comp. You and @phunghieu are the chief reason, Kaggle community rocks. Appreciated.
\n\n+10 if you can upload a kernal with all the data preloaded :). I know, its a lot to ask, However it will great for people to quickly try various training approaches on this dataset.
", "rawMarkdown": "This is amazing work @greatgamedota. Thanks for putting all the hard work to create and upload this dataset for people who are using kaggle only for this comp. You and @phunghieu are the chief reason, Kaggle community rocks. Appreciated. \n\n+10 if you can upload a kernal with all the data preloaded :). I know, its a lot to ask, However it will great for people to quickly try various training approaches on this dataset.", "votes": null }, { "id": "766651", "postDate": "03/08/2020 14:15:29", "content": "Here is a kaggle kernal with all the datasets preloaded: https://www.kaggle.com/greatgamedota/deepfake-detection-full-data (WARNING: Kaggle attempts to load a preview of ALL the datasets which causes Kaggle to tank)
", "rawMarkdown": "Here is a kaggle kernal with all the datasets preloaded: https://www.kaggle.com/greatgamedota/deepfake-detection-full-data (WARNING: Kaggle attempts to load a preview of ALL the datasets which causes Kaggle to tank)", "votes": null }, { "id": "766652", "postDate": "03/08/2020 14:15:46", "content": "Thank you! I have shared a kernal.
", "rawMarkdown": "Thank you! I have shared a kernal.", "votes": null }, { "id": "780959", "postDate": "03/20/2020 19:22:53", "content": "UPDATE: I updated datasets 16, and 18-32 with images using mobilenet as the detector instead of facenet_pytorch due to its increased accuracy.
", "rawMarkdown": "UPDATE: I updated datasets 16, and 18-32 with images using mobilenet as the detector instead of facenet_pytorch due to its increased accuracy.", "votes": null } ], "comments": [ { "id": 766341, "author_name": "phunghieu", "author_url": "", "post_date": "03/08/2020 03:08:32", "content": "Great job, @greatgamedota 👍. I hope that other competitors can benefit from these datasets to continue to push their models' performance in this competition.
", "votes": null, "replies": [] }, { "id": 766456, "author_name": "pankymathur", "author_url": "", "post_date": "03/08/2020 06:52:22", "content": "This is amazing work @greatgamedota. Thanks for putting all the hard work to create and upload this dataset for people who are using kaggle only for this comp. You and @phunghieu are the chief reason, Kaggle community rocks. Appreciated.
\n\n+10 if you can upload a kernal with all the data preloaded :). I know, its a lot to ask, However it will great for people to quickly try various training approaches on this dataset.
", "votes": null, "replies": [ { "id": 766652, "author_name": "greatgamedota", "author_url": "", "post_date": "03/08/2020 14:15:46", "content": "Thank you! I have shared a kernal.
", "votes": null, "replies": [] } ] }, { "id": 766651, "author_name": "greatgamedota", "author_url": "", "post_date": "03/08/2020 14:15:29", "content": "Here is a kaggle kernal with all the datasets preloaded: https://www.kaggle.com/greatgamedota/deepfake-detection-full-data (WARNING: Kaggle attempts to load a preview of ALL the datasets which causes Kaggle to tank)
", "votes": null, "replies": [] }, { "id": 780959, "author_name": "greatgamedota", "author_url": "", "post_date": "03/20/2020 19:22:53", "content": "UPDATE: I updated datasets 16, and 18-32 with images using mobilenet as the detector instead of facenet_pytorch due to its increased accuracy.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "766323": "I wanted to use @phunghieu 's datasets for this comp but he wasn't able to finish it and it is logistically to large (**11.8+** million images, **800+** GB for just the first **15** parts of the data set). So I went ahead and finished the data set:\n\n### First 15 parts: [Link](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/128954)\n\n### I wasn't consistent in how I made the datasets so I'll label each part with what it contains\n\nAll datasets contain jpg images to conserve space\n\n### 10 frames all 150x150 images, mobilenet used for detection\nPart 16: https://www.kaggle.com/greatgamedota/dfdc-part-16 (121 MB)\n\n### 150 frames all 150x150 images, facenet pytorch used for detection\nPart 17: https://www.kaggle.com/greatgamedota/dfdc-part-17 (1GB)\n\n### 10 frames, 150x150 images, mobilenet detector\nPart 18: https://www.kaggle.com/greatgamedota/dfdc-part-18 (158 MB)\n**...**\nPart 32: https://www.kaggle.com/greatgamedota/dfdc-part-32 (150 MB)\n\n### 10 frames, 160x160, [mobilenet detector](https://www.kaggle.com/unkownhihi/mobilenet-face-extractor-helper-code)\nPart 33: https://www.kaggle.com/greatgamedota/dfdc-part-33 (139 MB)\n\n### 10 frames, 150x150, mobilenet detector\nPart 34: https://www.kaggle.com/greatgamedota/dfdc-part-34 (147 MB)\n**...**\nPart 49: https://www.kaggle.com/greatgamedota/dfdc-part-49 (189 MB)\n\n### Here are some code snippets to load all of these datasets into a kaggle kernal (after you spend 10 minutes just adding them lol)\n\n```\n# First 15 parts:\nmeta = glob.glob('../input/deepfake-detection-faces-*/*.csv')\nmeta.sort(key=lambda f: int(re.sub('\\D', '', f)))\n\ndfs = []\nfor path in meta:\n df = pd.read_csv(path)\n df['path'] = ''\n path = path.split(\"/\")[:-1]\n path = path[0] + '/' + path[1] + '/' + path[2] + '/'\n for i in range(len(df)):\n df.loc[i]['path'] = f'{path}{df.loc[i][\"filename\"][:-4]}'\n dfs.append(df)\n\ntrain_df = pd.concat(dfs)\ntrain_df = train_df.reset_index(drop=True)\nprint(len(train_df))\n```\n### My datasets require more cleaning:\n```\n# @GreatGameDota's datasets\npart = 16\nfor j in range((49-16)+1):\n if part+j != 17:\n meta = pd.read_csv(f'../input/dfdc-part-{part+j}/images/metadata{part+j}.csv')\n else:\n meta = pd.read_csv(f'../input/dfdc-part-{part+j}/images/metadata{part+j}.json', index_col=0)\n meta['path'] = ''\n print(part+j)\n del_idxs = []\n for i in range(len(meta)):\n if os.path.isdir(f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}'):\n if len(os.listdir(f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}')) < 5:\n del_idxs.append(i)\n else:\n meta.loc[i]['path'] = f'../input/dfdc-part-{part+j}/images/{meta.loc[i][\"filename\"][:-4]}'\n else:\n del_idxs.append(i)\n print(del_idxs) # Print out removed video ids\n for idx in del_idxs:\n meta = meta.drop(idx)\n train_df = pd.concat([train_df,meta])\n train_df = train_df.reset_index(drop=True)\nprint(len(train_df)) # Should be ~120k if everything is successfully loaded\n```\nThe above snippet adds data that has at least 5 images for that video. (Fun fact: mobilenet >> facenet just based on the amount of videos facenet failed on and need to be removed)\n\n### Hope this helps anyone using/wants to use kaggle for this comp :)", "766341": "Great job, @greatgamedota 👍. I hope that other competitors can benefit from these datasets to continue to push their models' performance in this competition.", "766456": "This is amazing work @greatgamedota. Thanks for putting all the hard work to create and upload this dataset for people who are using kaggle only for this comp. You and @phunghieu are the chief reason, Kaggle community rocks. Appreciated. \n\n+10 if you can upload a kernal with all the data preloaded :). I know, its a lot to ask, However it will great for people to quickly try various training approaches on this dataset.", "766651": "Here is a kaggle kernal with all the datasets preloaded: https://www.kaggle.com/greatgamedota/deepfake-detection-full-data (WARNING: Kaggle attempts to load a preview of ALL the datasets which causes Kaggle to tank)", "766652": "Thank you! I have shared a kernal.", "780959": "UPDATE: I updated datasets 16, and 18-32 with images using mobilenet as the detector instead of facenet_pytorch due to its increased accuracy." }, "source": "meta" }