{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Analysis of the Submission.csv \nBy running this script you can visualize how your model performed in the ```test_videos``` folder.\n\nThe resources needed:\n\n- A dataset that contains ```submission.csv```. I have included a sample submission here.\n\n- A metadata file to get the ground truth of the videos in the ```test_videos``` folder. It is possible because the videos inside this folder is a subset of the training folders provided in the competition. I am using metadata dataset provided in https://www.kaggle.com/calebeverett/metadata-dataframe.\n\nPlease upvote if you like it! Thanks :)\n\n### <span style=\"color:red\">PLEASE NOTE:</span> \nOf course, the model is overfitting in the test_videos folder, as it is a subset of full training data. But, this can help in comparing performance of two models (in subtle ways), not for a single model. This script is mere a tool for having more insights, not a sure-shot \"metric\" to determine a good model :)\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Import Packages"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os, sys, random\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read Data"},{"metadata":{},"cell_type":"markdown","source":"### Metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_MetaData = pd.read_csv('../input/train-set-metadata-for-dfdc/metadata')\ndf_MetaData.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Sample submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv(\"../input/sampledeepfakesubmissioncsv/sample_submission.csv\")\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate Dataframe with Predictions and Ground Truth Labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"def label_convert(label):\n    if label==\"REAL\":\n        return 0\n    else:\n        return 1\n\ntest_dir = \"/kaggle/input/deepfake-detection-challenge/test_videos/\"\ntest_videos = sorted([x for x in os.listdir(test_dir) if x[-4:] == \".mp4\"])\n\ndf_TestData = pd.DataFrame(columns=['label', 'prediction'])\nfor file_id in tqdm(test_videos):\n    df_TestData.loc[file_id] = [label_convert(list(df_MetaData[df_MetaData.filename==file_id].label)[0]),list(sample_submission[sample_submission.filename==file_id].label) [0]\n]\n    \ndf_TestData.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize How Well Your Model Performed"},{"metadata":{},"cell_type":"markdown","source":"Now, you can visualize your predictions in a histogram. \n\nThe blue bars show your REAL predictions, and orange bars show FAKE predictions. For a perfect prediction, all blue bars should be in '0' and all orange bars should be in '1'."},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"df_Real =df_TestData[df_TestData.label==0]\ndf_Fake =df_TestData[df_TestData.label==1]\n\ndata = list(df_Real.prediction)\ncount = np.histogram(data)[0]\nplt.hist(data,50)\n\ndata = list(df_Fake.prediction)\ncount = np.histogram(data)[0]\nplt.hist(data,50)\n\nplt.legend(['REAL', 'FAKE'])\n\nplt.axis([0,1,0,140])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Estimate log_loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import log_loss\nLOG_LOSS = log_loss(list(df_TestData.label),list(df_TestData.prediction))\nprint(\"Log loss in the test folder is: \" + str(LOG_LOSS))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Estimate Confusion Matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\ndf_CM = df_TestData.copy()\ndf_CM.loc[df_CM.prediction>0.5,'prediction']=1\ndf_CM.loc[df_CM.prediction<=0.5,'prediction']=0\n\nCONFUSION_MATRIX = confusion_matrix(list(df_TestData.label),list(df_CM.prediction))\nprint(\"Confusion Matrix is:\\n\" + str(CONFUSION_MATRIX))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}