The title says it all.\nI've been spending the whole week trying to figure out what may have gone wrong, only to be frustrated by another submission scoring error.\nIt runs perfectly well on 400 public test set, and finishes at about 2000 seconds, so it's not a timing issue. (submission scoring error occurs at about 4 hours every time)\nI've tried almost everything that's been suggested on Discussion with regards to submission scorrng error.
\n\nCould someone please help me figure out what the problem is?
\n\nThe main script looks like this:
\n\n```\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport time
\n\nimport os\nimport sys\nsys.path.insert(0, \"/kaggle/input\")\nsys.path.insert(0, \"/kaggle/input/nnabla-wheels\")\nsys.path.insert(0, \"/kaggle/input/adamtest\")
\n\n!pip install /kaggle/input/nnabla-wheels/configparser-4.0.2-py2.py3-none-any.whl\n!pip install /kaggle/input/nnabla-wheels/nnabla-1.6.0-cp36-cp36m-manylinux1_x86_64.whl\n!pip install /kaggle/input/adamtest/dlib-19.19.0/dlib-19.19.0\n!pip install /kaggle/input/nnabla-wheels/nnabla_ext_cuda100-1.6.0-cp36-cp36m-manylinux1_x86_64.whl
\n\nimport nnabla as nn\nimport nnabla.functions as F\nimport nnabla.parametric_functions as PF\nfrom nnabla.ext_utils import get_extension_context\nimport functools\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True\nfrom data_iterator import load_imgs
\n\nimport resnet
\n\nnn.load_parameters('/kaggle/input/adamtest/params_adam_055.h5')
\n\nctx = get_extension_context('cudnn', device_id=0)\nnn.set_default_context(ctx)
\n\nimg = nn.Variable((1,3,5,224,224), need_grad=False)\npred, hidden = resnet.resnet_imagenet(img, num_classes=2, num_layers=50, shortcut_type='b', test = True)\nprob = F.softmax(pred)
\n\ntest_frame_folder = '/kaggle/input/deepfake-detection-challenge/test_videos'\nlist_of_test_data = [os.path.join(test_frame_folder,f) for f in os.listdir(test_frame_folder) if f.endswith('.mp4')]#.sort()
\n\npred_list = [0.5]*len(list_of_test_data)
\n\nfor i,v in enumerate(list_of_test_data):\n try:\n img.d = load_imgs(v)\n prob.forward()\n print(i, prob.shape, prob.d[0][0], prob.d[0][1])\n pred_list[i]=np.clip(prob.d[0][0],0.1,0.9)\n except:\n pred_list[i]=0.5
\n\nres = pd.DataFrame({\n 'filename': list_of_test_data,\n 'label': pred_list,\n})
\n\nres.sort_values(by='filename', ascending=True, inplace=True)\nwith open(\"submission.csv\",mode=\"w\",errors=\"ignore\")as f:\n res.to_csv('submission.csv', index=False)\n```
\n\nand the data_iterator looks like this:\n(dlib looks for 25 consecutive faces, and if not found, the script returns random values)
\n\n```\nfrom contextlib import contextmanager\nimport numpy as np\nimport struct\nimport tarfile\nimport zlib\nimport time\nimport os\nimport errno
\n\nfrom nnabla.logger import logger\nfrom nnabla.utils.data_iterator import data_iterator\nfrom nnabla.utils.data_source import DataSource\nfrom nnabla.utils.data_source_loader import download, get_data_home
\n\nfrom nnabla.utils.image_utils import imread, imresize
\n\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True
\n\ndef load_imgs(image_path, input_image_shape=(224, 224)):\n cv2.setNumThreads(1)
\n\ndetector = dlib.get_frontal_face_detector()\n#face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')\ncap = cv2.VideoCapture(image_path)\n\nimgs = []\nsuccessive_frames = 0\nx1 = 0\nx2 = 100\ny1 = 0\ny2 = 100\n\nprint(image_path)\nstart = time.time()\n\nwhile cap.isOpened() and successive_frames < 5 and time.time()<start+8:\n frameId = cap.get(1)\n ret, frame = cap.read()\n if ret !=True: break\n gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\n face_rects, scores, idx = detector.run(gray, 0)\n #print(successive_frames, len(face_rects))\n if len(face_rects) >= 1:\n successive_frames += 1\n if successive_frames == 1:\n x1 = face_rects[0].left()\n y1 = face_rects[0].top()\n x2 = face_rects[0].right()\n y2 = face_rects[0].bottom()\n face = frame[y1:y2, x1:x2]\n try:\n face = cv2.resize(face, (224,224))\n except cv2.error:\n face = cv2.resize(frame, (224,224)) # fix later\n imgs.append(face)\n else:\n imgs = []\n successive_frames = 0\n\nif len(imgs) == 0:\n print(\"NO FACE FOUND\")\n imgs.append(np.random.randint(255, size=(224,224,3),dtype=np.uint8))\n\nif len(imgs)<5:\n imgs = imgs[:5] + [imgs[-1] for _ in range(5-len(imgs))]\n\nimgs = np.stack(imgs, axis=0) # NHWC\n\nreturn imgs.transpose((3, 0, 1, 2)) # NHWC -> CNHW\n```
", "messages": [ { "id": "791625", "postDate": "03/30/2020 14:57:33", "content": "The title says it all.\nI've been spending the whole week trying to figure out what may have gone wrong, only to be frustrated by another submission scoring error.\nIt runs perfectly well on 400 public test set, and finishes at about 2000 seconds, so it's not a timing issue. (submission scoring error occurs at about 4 hours every time)\nI've tried almost everything that's been suggested on Discussion with regards to submission scorrng error.
\n\nCould someone please help me figure out what the problem is?
\n\nThe main script looks like this:
\n\n```\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport time
\n\nimport os\nimport sys\nsys.path.insert(0, \"/kaggle/input\")\nsys.path.insert(0, \"/kaggle/input/nnabla-wheels\")\nsys.path.insert(0, \"/kaggle/input/adamtest\")
\n\n!pip install /kaggle/input/nnabla-wheels/configparser-4.0.2-py2.py3-none-any.whl\n!pip install /kaggle/input/nnabla-wheels/nnabla-1.6.0-cp36-cp36m-manylinux1_x86_64.whl\n!pip install /kaggle/input/adamtest/dlib-19.19.0/dlib-19.19.0\n!pip install /kaggle/input/nnabla-wheels/nnabla_ext_cuda100-1.6.0-cp36-cp36m-manylinux1_x86_64.whl
\n\nimport nnabla as nn\nimport nnabla.functions as F\nimport nnabla.parametric_functions as PF\nfrom nnabla.ext_utils import get_extension_context\nimport functools\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True\nfrom data_iterator import load_imgs
\n\nimport resnet
\n\nnn.load_parameters('/kaggle/input/adamtest/params_adam_055.h5')
\n\nctx = get_extension_context('cudnn', device_id=0)\nnn.set_default_context(ctx)
\n\nimg = nn.Variable((1,3,5,224,224), need_grad=False)\npred, hidden = resnet.resnet_imagenet(img, num_classes=2, num_layers=50, shortcut_type='b', test = True)\nprob = F.softmax(pred)
\n\ntest_frame_folder = '/kaggle/input/deepfake-detection-challenge/test_videos'\nlist_of_test_data = [os.path.join(test_frame_folder,f) for f in os.listdir(test_frame_folder) if f.endswith('.mp4')]#.sort()
\n\npred_list = [0.5]*len(list_of_test_data)
\n\nfor i,v in enumerate(list_of_test_data):\n try:\n img.d = load_imgs(v)\n prob.forward()\n print(i, prob.shape, prob.d[0][0], prob.d[0][1])\n pred_list[i]=np.clip(prob.d[0][0],0.1,0.9)\n except:\n pred_list[i]=0.5
\n\nres = pd.DataFrame({\n 'filename': list_of_test_data,\n 'label': pred_list,\n})
\n\nres.sort_values(by='filename', ascending=True, inplace=True)\nwith open(\"submission.csv\",mode=\"w\",errors=\"ignore\")as f:\n res.to_csv('submission.csv', index=False)\n```
\n\nand the data_iterator looks like this:\n(dlib looks for 25 consecutive faces, and if not found, the script returns random values)
\n\n```\nfrom contextlib import contextmanager\nimport numpy as np\nimport struct\nimport tarfile\nimport zlib\nimport time\nimport os\nimport errno
\n\nfrom nnabla.logger import logger\nfrom nnabla.utils.data_iterator import data_iterator\nfrom nnabla.utils.data_source import DataSource\nfrom nnabla.utils.data_source_loader import download, get_data_home
\n\nfrom nnabla.utils.image_utils import imread, imresize
\n\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True
\n\ndef load_imgs(image_path, input_image_shape=(224, 224)):\n cv2.setNumThreads(1)
\n\ndetector = dlib.get_frontal_face_detector()\n#face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')\ncap = cv2.VideoCapture(image_path)\n\nimgs = []\nsuccessive_frames = 0\nx1 = 0\nx2 = 100\ny1 = 0\ny2 = 100\n\nprint(image_path)\nstart = time.time()\n\nwhile cap.isOpened() and successive_frames < 5 and time.time()<start+8:\n frameId = cap.get(1)\n ret, frame = cap.read()\n if ret !=True: break\n gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\n face_rects, scores, idx = detector.run(gray, 0)\n #print(successive_frames, len(face_rects))\n if len(face_rects) >= 1:\n successive_frames += 1\n if successive_frames == 1:\n x1 = face_rects[0].left()\n y1 = face_rects[0].top()\n x2 = face_rects[0].right()\n y2 = face_rects[0].bottom()\n face = frame[y1:y2, x1:x2]\n try:\n face = cv2.resize(face, (224,224))\n except cv2.error:\n face = cv2.resize(frame, (224,224)) # fix later\n imgs.append(face)\n else:\n imgs = []\n successive_frames = 0\n\nif len(imgs) == 0:\n print(\"NO FACE FOUND\")\n imgs.append(np.random.randint(255, size=(224,224,3),dtype=np.uint8))\n\nif len(imgs)<5:\n imgs = imgs[:5] + [imgs[-1] for _ in range(5-len(imgs))]\n\nimgs = np.stack(imgs, axis=0) # NHWC\n\nreturn imgs.transpose((3, 0, 1, 2)) # NHWC -> CNHW\n```
", "rawMarkdown": "The title says it all.\nI've been spending the whole week trying to figure out what may have gone wrong, only to be frustrated by another submission scoring error.\nIt runs perfectly well on 400 public test set, and finishes at about 2000 seconds, so it's not a timing issue. (submission scoring error occurs at about 4 hours every time)\nI've tried almost everything that's been suggested on Discussion with regards to submission scorrng error.\n\nCould someone please help me figure out what the problem is?\n\nThe main script looks like this:\n\n```\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport time\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport sys\nsys.path.insert(0, \"/kaggle/input\")\nsys.path.insert(0, \"/kaggle/input/nnabla-wheels\")\nsys.path.insert(0, \"/kaggle/input/adamtest\")\n\n#pip install nnabla\n!pip install /kaggle/input/nnabla-wheels/configparser-4.0.2-py2.py3-none-any.whl\n!pip install /kaggle/input/nnabla-wheels/nnabla-1.6.0-cp36-cp36m-manylinux1_x86_64.whl\n!pip install /kaggle/input/adamtest/dlib-19.19.0/dlib-19.19.0\n!pip install /kaggle/input/nnabla-wheels/nnabla_ext_cuda100-1.6.0-cp36-cp36m-manylinux1_x86_64.whl\n\nimport nnabla as nn\nimport nnabla.functions as F\nimport nnabla.parametric_functions as PF\nfrom nnabla.ext_utils import get_extension_context\nimport functools\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True\nfrom data_iterator import load_imgs\n\nimport resnet\n\nnn.load_parameters('/kaggle/input/adamtest/params_adam_055.h5')\n\nctx = get_extension_context('cudnn', device_id=0)\nnn.set_default_context(ctx)\n\n\nimg = nn.Variable((1,3,5,224,224), need_grad=False)\npred, hidden = resnet.resnet_imagenet(img, num_classes=2, num_layers=50, shortcut_type='b', test = True)\nprob = F.softmax(pred)\n\n\ntest_frame_folder = '/kaggle/input/deepfake-detection-challenge/test_videos'\nlist_of_test_data = [os.path.join(test_frame_folder,f) for f in os.listdir(test_frame_folder) if f.endswith('.mp4')]#.sort()\n\npred_list = [0.5]*len(list_of_test_data)\n\n\nfor i,v in enumerate(list_of_test_data):\n try:\n img.d = load_imgs(v)\n prob.forward()\n print(i, prob.shape, prob.d[0][0], prob.d[0][1])\n pred_list[i]=np.clip(prob.d[0][0],0.1,0.9)\n except:\n pred_list[i]=0.5\n\nres = pd.DataFrame({\n 'filename': list_of_test_data,\n 'label': pred_list,\n})\n\nres.sort_values(by='filename', ascending=True, inplace=True)\nwith open(\"submission.csv\",mode=\"w\",errors=\"ignore\")as f:\n res.to_csv('submission.csv', index=False)\n```\n\nand the data_iterator looks like this:\n(dlib looks for 25 consecutive faces, and if not found, the script returns random values)\n\n```\nfrom contextlib import contextmanager\nimport numpy as np\nimport struct\nimport tarfile\nimport zlib\nimport time\nimport os\nimport errno\n\nfrom nnabla.logger import logger\nfrom nnabla.utils.data_iterator import data_iterator\nfrom nnabla.utils.data_source import DataSource\nfrom nnabla.utils.data_source_loader import download, get_data_home\n\nfrom nnabla.utils.image_utils import imread, imresize\n\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True\n\ndef load_imgs(image_path, input_image_shape=(224, 224)):\n cv2.setNumThreads(1)\n\n detector = dlib.get_frontal_face_detector()\n #face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')\n cap = cv2.VideoCapture(image_path)\n\n imgs = []\n successive_frames = 0\n x1 = 0\n x2 = 100\n y1 = 0\n y2 = 100\n\n print(image_path)\n start = time.time()\n\n while cap.isOpened() and successive_frames < 5 and time.time()", "votes": null }, { "id": "791638", "postDate": "03/30/2020 15:08:30", "content": "I don't see how it is a problem, but I think you don't need this row
\n\nwith open(\"submission.csv\",mode=\"w\",errors=\"ignore\")as f:
may be in some cases prob.d[0][0] is nan?
Make sure you handle the corrupted videos, go through this interface help to make the submission run below 4 hours
\n\nhttps://www.kaggle.com/unkownhihi/dfdc-lrcn-inference
", "rawMarkdown": "Make sure you handle the corrupted videos, go through this interface help to make the submission run below 4 hours\n\nhttps://www.kaggle.com/unkownhihi/dfdc-lrcn-inference", "votes": null }, { "id": "791726", "postDate": "03/30/2020 16:05:29", "content": "I am having issues as well and I don't know what to do....I have removed all corrupted videos and am not getting any nan values when working on the test set to make a submission. It seems like there has to be an issue. Are you submitting via firefox in a linux OS by any chance?
", "rawMarkdown": "I am having issues as well and I don't know what to do....I have removed all corrupted videos and am not getting any nan values when working on the test set to make a submission. It seems like there has to be an issue. Are you submitting via firefox in a linux OS by any chance?", "votes": null }, { "id": "791942", "postDate": "03/30/2020 19:43:07", "content": "Are you certain you're not leaking memory?
", "rawMarkdown": "Are you certain you're not leaking memory?", "votes": null }, { "id": "792235", "postDate": "03/31/2020 02:14:40", "content": "My team is having the same issue, we havent been able to submit a new submission in over a week. Strange.
", "rawMarkdown": "My team is having the same issue, we havent been able to submit a new submission in over a week. Strange.", "votes": null } ], "comments": [ { "id": 791638, "author_name": "zaharch", "author_url": "", "post_date": "03/30/2020 15:08:30", "content": "I don't see how it is a problem, but I think you don't need this row
\n\nwith open(\"submission.csv\",mode=\"w\",errors=\"ignore\")as f:
may be in some cases prob.d[0][0] is nan?
Make sure you handle the corrupted videos, go through this interface help to make the submission run below 4 hours
\n\nhttps://www.kaggle.com/unkownhihi/dfdc-lrcn-inference
", "votes": null, "replies": [] }, { "id": 791726, "author_name": "goalieperson", "author_url": "", "post_date": "03/30/2020 16:05:29", "content": "I am having issues as well and I don't know what to do....I have removed all corrupted videos and am not getting any nan values when working on the test set to make a submission. It seems like there has to be an issue. Are you submitting via firefox in a linux OS by any chance?
", "votes": null, "replies": [] }, { "id": 791942, "author_name": "humananalog", "author_url": "", "post_date": "03/30/2020 19:43:07", "content": "Are you certain you're not leaking memory?
", "votes": null, "replies": [] }, { "id": 792235, "author_name": "sherkt1", "author_url": "", "post_date": "03/31/2020 02:14:40", "content": "My team is having the same issue, we havent been able to submit a new submission in over a week. Strange.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "791625": "The title says it all.\nI've been spending the whole week trying to figure out what may have gone wrong, only to be frustrated by another submission scoring error.\nIt runs perfectly well on 400 public test set, and finishes at about 2000 seconds, so it's not a timing issue. (submission scoring error occurs at about 4 hours every time)\nI've tried almost everything that's been suggested on Discussion with regards to submission scorrng error.\n\nCould someone please help me figure out what the problem is?\n\nThe main script looks like this:\n\n```\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport time\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport sys\nsys.path.insert(0, \"/kaggle/input\")\nsys.path.insert(0, \"/kaggle/input/nnabla-wheels\")\nsys.path.insert(0, \"/kaggle/input/adamtest\")\n\n#pip install nnabla\n!pip install /kaggle/input/nnabla-wheels/configparser-4.0.2-py2.py3-none-any.whl\n!pip install /kaggle/input/nnabla-wheels/nnabla-1.6.0-cp36-cp36m-manylinux1_x86_64.whl\n!pip install /kaggle/input/adamtest/dlib-19.19.0/dlib-19.19.0\n!pip install /kaggle/input/nnabla-wheels/nnabla_ext_cuda100-1.6.0-cp36-cp36m-manylinux1_x86_64.whl\n\nimport nnabla as nn\nimport nnabla.functions as F\nimport nnabla.parametric_functions as PF\nfrom nnabla.ext_utils import get_extension_context\nimport functools\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True\nfrom data_iterator import load_imgs\n\nimport resnet\n\nnn.load_parameters('/kaggle/input/adamtest/params_adam_055.h5')\n\nctx = get_extension_context('cudnn', device_id=0)\nnn.set_default_context(ctx)\n\n\nimg = nn.Variable((1,3,5,224,224), need_grad=False)\npred, hidden = resnet.resnet_imagenet(img, num_classes=2, num_layers=50, shortcut_type='b', test = True)\nprob = F.softmax(pred)\n\n\ntest_frame_folder = '/kaggle/input/deepfake-detection-challenge/test_videos'\nlist_of_test_data = [os.path.join(test_frame_folder,f) for f in os.listdir(test_frame_folder) if f.endswith('.mp4')]#.sort()\n\npred_list = [0.5]*len(list_of_test_data)\n\n\nfor i,v in enumerate(list_of_test_data):\n try:\n img.d = load_imgs(v)\n prob.forward()\n print(i, prob.shape, prob.d[0][0], prob.d[0][1])\n pred_list[i]=np.clip(prob.d[0][0],0.1,0.9)\n except:\n pred_list[i]=0.5\n\nres = pd.DataFrame({\n 'filename': list_of_test_data,\n 'label': pred_list,\n})\n\nres.sort_values(by='filename', ascending=True, inplace=True)\nwith open(\"submission.csv\",mode=\"w\",errors=\"ignore\")as f:\n res.to_csv('submission.csv', index=False)\n```\n\nand the data_iterator looks like this:\n(dlib looks for 25 consecutive faces, and if not found, the script returns random values)\n\n```\nfrom contextlib import contextmanager\nimport numpy as np\nimport struct\nimport tarfile\nimport zlib\nimport time\nimport os\nimport errno\n\nfrom nnabla.logger import logger\nfrom nnabla.utils.data_iterator import data_iterator\nfrom nnabla.utils.data_source import DataSource\nfrom nnabla.utils.data_source_loader import download, get_data_home\n\nfrom nnabla.utils.image_utils import imread, imresize\n\nimport cv2\nimport dlib\ndlib.DLIB_USE_CUDA = True\n\ndef load_imgs(image_path, input_image_shape=(224, 224)):\n cv2.setNumThreads(1)\n\n detector = dlib.get_frontal_face_detector()\n #face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')\n cap = cv2.VideoCapture(image_path)\n\n imgs = []\n successive_frames = 0\n x1 = 0\n x2 = 100\n y1 = 0\n y2 = 100\n\n print(image_path)\n start = time.time()\n\n while cap.isOpened() and successive_frames < 5 and time.time()", "791638": "I don't see how it is a problem, but I think you don't need this row\n\n`with open(\"submission.csv\",mode=\"w\",errors=\"ignore\")as f:`\n\nmay be in some cases `prob.d[0][0]` is nan?", "791639": "Make sure you handle the corrupted videos, go through this interface help to make the submission run below 4 hours\n\nhttps://www.kaggle.com/unkownhihi/dfdc-lrcn-inference", "791726": "I am having issues as well and I don't know what to do....I have removed all corrupted videos and am not getting any nan values when working on the test set to make a submission. It seems like there has to be an issue. Are you submitting via firefox in a linux OS by any chance?", "791942": "Are you certain you're not leaking memory?", "792235": "My team is having the same issue, we havent been able to submit a new submission in over a week. Strange." }, "source": "meta" }