![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2Ff1cb5a298387f7c826ee0009446285c0%2F__results___33_0.png?generation=1584135632083106&alt=media\")
The basic intuition of this algorithm is that the model must have used some pixels to identify the image class. This is based on the following paper - Grad-CAM visualizations
\n\nYou can browse through the notebook over here - Link to kernel
\n\nAfter going over a few samples, it looks like the model is using facial features like nose, ears, eyes etc. to classify the faces. So is the model identifying the deepfake artifacts or just noise?
\n\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F8354c97a9fe69f3b785508d23e8531bb%2Fdownload.png?generation=1584135719614810&alt=media\")
.png?generation=1584135760984220&alt=media)
.png?generation=1584135851598595&alt=media)
.png?generation=1584135889179734&alt=media)
.png?generation=1584135929242126&alt=media)
.png?generation=1584135958664344&alt=media)
The basic intuition of this algorithm is that the model must have used some pixels to identify the image class. This is based on the following paper - Grad-CAM visualizations
\n\nYou can browse through the notebook over here - Link to kernel
\n\nAfter going over a few samples, it looks like the model is using facial features like nose, ears, eyes etc. to classify the faces. So is the model identifying the deepfake artifacts or just noise?
\n\n
.png?generation=1584135760984220&alt=media)
.png?generation=1584135851598595&alt=media)
.png?generation=1584135889179734&alt=media)
.png?generation=1584135929242126&alt=media)
.png?generation=1584135958664344&alt=media)
Did a few more runs on the videos where the loss is high. The class activations for the top k mis-classified videos is as follows (k=10):
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F30b5d4b25e5c90da8efb0a48834d0ca4%2FHIghDiff_04.png?generation=1584173469546915&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2Fb6b39cfa5283365cb06e221136c95083%2FHIghDiff_03.png?generation=1584173469367546&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2Fdb90df81a971e16bb354d38bba301d50%2FHighReal_0.png?generation=1584173215785085&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F4da7996109a4fdebc9edd2c261a6f7ff%2FHIghDiff_00.png?generation=1584173265337593&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2Fe0a13c7fd52b9f7891837718d93f0b05%2FHIghDiff_02.png?generation=1584173307222628&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F368499f96929a96212712239b0e2bde5%2FHighReal_12.png?generation=1584173340636721&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F60a23357b01a1b3ccd9795d2182ea164%2FHighReal_10.png?generation=1584173376716114&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F651dffdd5bdcf393c7b010c641fdd43b%2FHIghDiff_01.png?generation=1584173412740079&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F8235f52380e849c4d1464983c43327a1%2FHighReal_09.png?generation=1584173512865681&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F252874%2F57407660c162d40f2a59f4ebc15728e4%2FHighReal_16.png?generation=1584173515119630&alt=media\")
Nice. I am thinking of passing the heat map output to another CNN. It works similar to attention.
", "rawMarkdown": "Nice. I am thinking of passing the heat map output to another CNN. It works similar to attention.", "votes": null } ], "comments": [ { "id": 771481, "author_name": "skylord", "author_url": "", "post_date": "03/14/2020 08:11:59", "content": "Did a few more runs on the videos where the loss is high. The class activations for the top k mis-classified videos is as follows (k=10):
\n\n\n
\n
Nice. I am thinking of passing the heat map output to another CNN. It works similar to attention.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "771213": "The basic intuition of this algorithm is that the model must have used some pixels to identify the image class. This is based on the following paper - [Grad-CAM visualizations](https://arxiv.org/pdf/1610.02391.pdf) \n\nYou can browse through the notebook over here - [Link to kernel](https://www.kaggle.com/skylord/grad-cam-on-resnext) \n\nAfter going over a few samples, it looks like the model is using facial features like nose, ears, eyes etc. to classify the faces. So is the model identifying the deepfake artifacts or just noise? \n\n\n\n\n\n.png?generation=1584135760984220&alt=media)\n\n.png?generation=1584135851598595&alt=media)\n\n.png?generation=1584135889179734&alt=media)\n\n.png?generation=1584135929242126&alt=media)\n\n.png?generation=1584135958664344&alt=media)", "771481": "Did a few more runs on the videos where the loss is high. The class activations for the top k mis-classified videos is as follows (k=10): \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "771910": "Nice. I am thinking of passing the heat map output to another CNN. It works similar to attention." }, "source": "meta" }