{"cells":[{"metadata":{},"cell_type":"markdown","source":"# An issue to Submission CSV Not Found\nAs @Giba wrote in https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121484, the \"Submission CSV Not Found\"  message comes when there is a bug in your notebook. But that bug could not have been seen on the train dataset or on the first test dataset (the one with only 400 videos). But the bug comes only on the big test dataset (the one with 4000 videos).<br>\n**In this notebook, I will explain a way to find where is your bug, without put exceptions handling every where.**<br>\nFirst, **let's suppose you have a very long notebook like the next cell**, which is doing a lot of things until your submission file to competition :"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\n# First part of your notebook\n \n# Part2\n\n# Part 3\n\n# Part 4\ndef prob(): # Your very complex model to do predictions\n    return .5\n\n# Part 5\nsf = pd.read_csv('../input/deepfake-detection-challenge/sample_submission.csv', index_col='filename')\nsf[\"label\"] = prob()\nsf[\"label\"].fillna(0.5, inplace = True)\nsf.reset_index(inplace=True)\nsf[['filename','label']].to_csv('submission.csv', index=False)\n\n# Part 6\nprint(\"Shape {}\\nmin {}\\nmax {}\\nNA {}\".format(sf.shape\n                , sf[\"label\"].min(), sf[\"label\"].max(), sf[\"label\"].isna().sum()))\nsf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's suppose you have worked the best as it is possible to do, let's suppose you don't have any bug with this notebook, neither on the train dataset nor the first test dataset. **Let's suppose you're ready to click on \"Submit to competition\"**.<br>\nAnd a few hours after submitting, you read the score of your submission : **\"Submission CSV not found\"**.<br>\nARRRRRGR !!! What could have happen ?\n\nThis message told us that the fifth part of the code in the previous cell had not been executed. Why it did not ? Because the notebook had stopped before of course !<br>\nBut where ?  In part 1, part 2 part 3 ? Let's use public score to find the place in your code where the bug is.\n\nWe know that there is 4000 videos in the big data set and that there 2000 are fake. So :"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import log_loss\n\ny = np.concatenate([np.ones(2000), np.zeros(2000)], axis=0)\n\n# If you submit a submission file with those values\nfor i in [0, 0.05, 0.1, .15, .2, .25, .3, .35, .4, .45, .5, 1]:\n    y_pred = np.full(4000, i)\n    print(\"{:.2f} : {:.5f}\".format(i, log_loss(y, y_pred)))\n    \n# Then you will have those public log loss :","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You don't believe me ? Look at the end of public leaderboard. And look with some small differences : "},{"metadata":{"trusted":true},"cell_type":"code","source":"# with small differences \ny = np.concatenate([np.ones(2010), np.zeros(1990)], axis=0)\n\n# the log loss is not the same (except for 0.5)\nfor i in [0, 0.05, 0.1, .15, .2, .25, .3, .35, .4, .45, .5, 1]:\n    y_pred = np.full(4000, i)\n    print(\"{:.2f} : {:.5f}\".format(i, log_loss(y, y_pred)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Hence a way to find where is your bug, is to submit many files** in your code, and **next, read your public score to guess where the code had stopped**. Let's modify your work from first cell by submit many files with different constants as prediction :"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\n# read the sample submission file\n_temp = pd.read_csv('../input/deepfake-detection-challenge/sample_submission.csv', index_col='filename')\n_cte = len(_temp)\n\n# Create a submssion file with i as a constant prediction\ndef submission_to_find_bug(i, verbose=False):\n    ts = _temp.copy()\n    y_pred = np.full(_cte, i)\n    ts[\"label\"] = y_pred\n    ts.reset_index(inplace=True)\n    ts[['filename','label']].to_csv('submission.csv', index=False)\n    if verbose:\n        print(\"Debug with value {}\".format(i))\n        print(ts.head(3))\n        print(ts.tail(3))\n\n# First part of the notebook\n# ...\n# Create the submission file with 0 as predcition for all videos \nsubmission_to_find_bug(0)\n\n# Part2\n# ...\n# Create the submission file with 0.05 as predcition for all videos \nsubmission_to_find_bug(0.05)\n\n# Part 3\n# ...\nsubmission_to_find_bug(0.1)\n\n# Part 4\ndef prob(): # Imagine your very complex model making your predictions\n    submission_to_find_bug(0.15)\n    return .5\nsubmission_to_find_bug(0.2, verbose=True)\n\n# Part 5\nsf = pd.read_csv('../input/deepfake-detection-challenge/sample_submission.csv', index_col='filename')\nsubmission_to_find_bug(0.25)\nsf[\"label\"] = prob()\nsf[\"label\"].fillna(0.5, inplace = True)\nsf.reset_index(inplace=True)\nsf[['filename','label']].to_csv('submission.csv', index=False)\n\n# Part 6\nprint(\"Shape {}\\nmin {}\\nmax {}\\nNA {}\".format(sf.shape\n                , sf[\"label\"].min(), sf[\"label\"].max(), sf[\"label\"].isna().sum()))\nsf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Run your notebook and submit to competition !"},{"metadata":{},"cell_type":"markdown","source":"Now, you have a score (not a brillant one) on the public leaderboard.<br>\nIf your score is 17.26939, then you have a bug in the part 2 of your notebook.<br>\nIf your score is 1.20397, then you have a bug in part 4 of your notebook.<br>\nIf your score is 0.15, you will probably be the winner and you did not need to read my notebook (but thank you) !"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}