![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F59a224f82717c86f75429810090e84a2%2FSelection_027.png?generation=1594470581684448&alt=media\")
This competition reminds me of IEEE's Signal Processing Society - Camera Model Identification competition where at the end winning models used off-the-shelf architectures which managed to get ~0.99ish accuracy in classifying which camera took a picture (10-class classification problem).
\n\nIn this competition off-the-shelf Efficientnets achieve 0.92 w-AUC w/o much effort treating this as 4-class classification problem.
\n\nDoes anybody have a clue as to what the net is looking at to make the right predictions?
", "messages": [ { "id": "904562", "postDate": "06/27/2020 18:00:09", "content": "This competition reminds me of IEEE's Signal Processing Society - Camera Model Identification competition where at the end winning models used off-the-shelf architectures which managed to get ~0.99ish accuracy in classifying which camera took a picture (10-class classification problem).
\n\nIn this competition off-the-shelf Efficientnets achieve 0.92 w-AUC w/o much effort treating this as 4-class classification problem.
\n\nDoes anybody have a clue as to what the net is looking at to make the right predictions?
", "rawMarkdown": "This competition reminds me of [IEEE's Signal Processing Society - Camera Model Identification](https://www.kaggle.com/c/sp-society-camera-model-identification) competition where at the end winning models used off-the-shelf architectures which managed to get ~0.99ish accuracy in classifying which camera took a picture (10-class classification problem).\n\nIn this competition off-the-shelf Efficientnets achieve 0.92 w-AUC w/o much effort treating this as 4-class classification problem.\n\nDoes anybody have a clue as to what the net is looking at to make the right predictions?", "votes": null }, { "id": "906201", "postDate": "06/29/2020 05:58:10", "content": "Maybe it is possible that these are pretrained with natural images and this competition is also using natural images so transfer learning is easier to achieve, they share similarity in features, etc. compared to something like medical imaging or cells.
", "rawMarkdown": "Maybe it is possible that these are pretrained with natural images and this competition is also using natural images so transfer learning is easier to achieve, they share similarity in features, etc. compared to something like medical imaging or cells.", "votes": null }, { "id": "906341", "postDate": "06/29/2020 07:59:03", "content": "Don't be fooled by the metric, accuracy here is still c80%.
\n\nWhat is interesting is that much prior research has centred on removing the natural image through a bettery of filters: https://www.google.com/search?q=\"steganalysis\"+\"remove+the+image+content\"
\n\nAnd yet, using networks pretrained on natural images appears to work far better than any of the preceding feature-crafted networks that seek to remove the natural content, and are published as sota. I'm waiting to see if ABBA's solution at end turns out a new method for these carefully processed RGB images.
", "rawMarkdown": "Don't be fooled by the metric, accuracy here is still c80%.\n\nWhat is interesting is that much prior research has centred on removing the natural image through a bettery of filters: [https://www.google.com/search?q=\"steganalysis\"+\"remove+the+image+content\"](https://www.google.com/search?ei=var5XuG7FJWV1fAPgdqsmAU&q=%22steganalysis%22+%22remove+the+image+content%22&oq=%22steganalysis%22+%22remove+the+image+content%22&gs_lcp=CgZwc3ktYWIQAzIFCCEQoAFQAFgAYPo2aABwAHgAgAFtiAFtkgEDMC4xmAEAqgEHZ3dzLXdpeg&sclient=psy-ab&ved=0ahUKEwihjp7s0abqAhWVShUIHQEtC1MQ4dUDCAs&uact=5)\n\nAnd yet, using networks pretrained on natural images appears to work far better than any of the preceding feature-crafted networks that seek to remove the natural content, and are published as sota. I'm waiting to see if ABBA's solution at end turns out a new method for these carefully processed RGB images.", "votes": null }, { "id": "918008", "postDate": "07/06/2020 22:02:12", "content": "Good question. The nets that seem to converge to reasonably high scores for me are MobileNet, MixNet, and EfficientNet. Depthwise separable convolutions are common between these backbones.
", "rawMarkdown": "Good question. The nets that seem to converge to reasonably high scores for me are MobileNet, MixNet, and EfficientNet. Depthwise separable convolutions are common between these backbones.", "votes": null }, { "id": "918013", "postDate": "07/06/2020 22:09:39", "content": "Nice to see you competing, @tivfrvqhs5 =]
", "rawMarkdown": "Nice to see you competing, @tivfrvqhs5 =]", "votes": null }, { "id": "918482", "postDate": "07/07/2020 09:20:39", "content": "Great insight. I was wondering architectural element made efficientnets work but not eg resnets.
", "rawMarkdown": "Great insight. I was wondering architectural element made efficientnets work but not eg resnets.", "votes": null }, { "id": "919088", "postDate": "07/07/2020 17:45:31", "content": "Expanding on this idea... FBNet, MNASNet, SinglePath NAS also converge, but don't achieve as high an accuracy as the bigger Efficientnets (they tend to level off in the Efficientnet B0-B1 range). So I'd say more generally that backbones based on the MobileNet V1/V2 block sequence tend to work. More work to do on understanding why this is the case.\nThis observation can also form the basis for custom network design.
", "rawMarkdown": "Expanding on this idea... FBNet, MNASNet, SinglePath NAS also converge, but don't achieve as high an accuracy as the bigger Efficientnets (they tend to level off in the Efficientnet B0-B1 range). So I'd say more generally that backbones based on the MobileNet V1/V2 block sequence tend to work. More work to do on understanding why this is the case.\nThis observation can also form the basis for custom network design.", "votes": null }, { "id": "919536", "postDate": "07/08/2020 00:33:38", "content": "what is the score of MixNet? is it in the range of B1 (this is what i got) ?
", "rawMarkdown": "what is the score of MixNet? is it in the range of B1 (this is what i got) ?", "votes": null }, { "id": "919542", "postDate": "07/08/2020 00:43:52", "content": "MixNet-XL w/TTA got 0.917 on the LB for me
", "rawMarkdown": "MixNet-XL w/TTA got 0.917 on the LB for me", "votes": null }, { "id": "919569", "postDate": "07/08/2020 01:36:30", "content": "0.917 is what i have with B1 without TTA. Was Mobilenet any better ?
", "rawMarkdown": "0.917 is what i have with B1 without TTA. Was Mobilenet any better ?", "votes": null }, { "id": "919595", "postDate": "07/08/2020 02:07:05", "content": "Mobilenet V3 was 0.914, could have gotten a little more out of it but I've been characterizing performance before pushing the models as far as they can go.
", "rawMarkdown": "Mobilenet V3 was 0.914, could have gotten a little more out of it but I've been characterizing performance before pushing the models as far as they can go.", "votes": null }, { "id": "924448", "postDate": "07/11/2020 12:30:12", "content": "results for mixnet-L
\n\n
related: there is something better than mixnet : mux conv
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F0858fc672def3e38dd92acc3131289fd%2FSelection_026.png?generation=1594470583701074&alt=media\")
@tivfrvqhs5 \nensembling mixnet with efficientnet tends to lower lb score! Even with TTA! Any clue why so?
Adding in mixnet to one of my ensembles gave a very slight LB score increase for me of 0.001. Usually when this happens it's due to lack of diversity between predictions but I haven't tested it in this case. I don't think mixnet is going to take you into efficientnet territory, it's more the observation of similar blocks in the networks that converge that may yield clues.
", "rawMarkdown": "Adding in mixnet to one of my ensembles gave a very slight LB score increase for me of 0.001. Usually when this happens it's due to lack of diversity between predictions but I haven't tested it in this case. I don't think mixnet is going to take you into efficientnet territory, it's more the observation of similar blocks in the networks that converge that may yield clues.", "votes": null }, { "id": "924692", "postDate": "07/11/2020 15:25:05", "content": "\n\n\nM.Innat wrote:
\n \n@tivfrvqhs5 \n ensembling
\nmixnetwithefficientnettends to lower lb score! Even with TTA! Any clue why so?
one likely reason is public LB is not representative of the actual quality of over test prediction.\nI would be more interesting to check if such ensembling improve your CV
", "rawMarkdown": "> **M.Innat wrote:**\n> \n> @tivfrvqhs5 \n> ensembling `mixnet` with `efficientnet` tends to lower lb score! Even with TTA! Any clue why so? \n> \n\none likely reason is public LB is not representative of the actual quality of over test prediction.\nI would be more interesting to check if such ensembling improve your CV", "votes": null } ], "comments": [ { "id": 906201, "author_name": "something4kag", "author_url": "", "post_date": "06/29/2020 05:58:10", "content": "Maybe it is possible that these are pretrained with natural images and this competition is also using natural images so transfer learning is easier to achieve, they share similarity in features, etc. compared to something like medical imaging or cells.
", "votes": null, "replies": [] }, { "id": 906341, "author_name": "robga", "author_url": "", "post_date": "06/29/2020 07:59:03", "content": "Don't be fooled by the metric, accuracy here is still c80%.
\n\nWhat is interesting is that much prior research has centred on removing the natural image through a bettery of filters: https://www.google.com/search?q=\"steganalysis\"+\"remove+the+image+content\"
\n\nAnd yet, using networks pretrained on natural images appears to work far better than any of the preceding feature-crafted networks that seek to remove the natural content, and are published as sota. I'm waiting to see if ABBA's solution at end turns out a new method for these carefully processed RGB images.
", "votes": null, "replies": [] }, { "id": 918008, "author_name": "tivfrvqhs5", "author_url": "", "post_date": "07/06/2020 22:02:12", "content": "Good question. The nets that seem to converge to reasonably high scores for me are MobileNet, MixNet, and EfficientNet. Depthwise separable convolutions are common between these backbones.
", "votes": null, "replies": [ { "id": 918013, "author_name": "authman", "author_url": "", "post_date": "07/06/2020 22:09:39", "content": "Nice to see you competing, @tivfrvqhs5 =]
", "votes": null, "replies": [] }, { "id": 918482, "author_name": "antorsae", "author_url": "", "post_date": "07/07/2020 09:20:39", "content": "Great insight. I was wondering architectural element made efficientnets work but not eg resnets.
", "votes": null, "replies": [] }, { "id": 919088, "author_name": "tivfrvqhs5", "author_url": "", "post_date": "07/07/2020 17:45:31", "content": "Expanding on this idea... FBNet, MNASNet, SinglePath NAS also converge, but don't achieve as high an accuracy as the bigger Efficientnets (they tend to level off in the Efficientnet B0-B1 range). So I'd say more generally that backbones based on the MobileNet V1/V2 block sequence tend to work. More work to do on understanding why this is the case.\nThis observation can also form the basis for custom network design.
", "votes": null, "replies": [] }, { "id": 919536, "author_name": "hengck23", "author_url": "", "post_date": "07/08/2020 00:33:38", "content": "what is the score of MixNet? is it in the range of B1 (this is what i got) ?
", "votes": null, "replies": [] }, { "id": 919542, "author_name": "tivfrvqhs5", "author_url": "", "post_date": "07/08/2020 00:43:52", "content": "MixNet-XL w/TTA got 0.917 on the LB for me
", "votes": null, "replies": [] }, { "id": 919569, "author_name": "nyleve", "author_url": "", "post_date": "07/08/2020 01:36:30", "content": "0.917 is what i have with B1 without TTA. Was Mobilenet any better ?
", "votes": null, "replies": [] }, { "id": 919595, "author_name": "tivfrvqhs5", "author_url": "", "post_date": "07/08/2020 02:07:05", "content": "Mobilenet V3 was 0.914, could have gotten a little more out of it but I've been characterizing performance before pushing the models as far as they can go.
", "votes": null, "replies": [] }, { "id": 924448, "author_name": "hengck23", "author_url": "", "post_date": "07/11/2020 12:30:12", "content": "results for mixnet-L
\n\n
related: there is something better than mixnet : mux conv
\n\n
@tivfrvqhs5 \nensembling mixnet with efficientnet tends to lower lb score! Even with TTA! Any clue why so?
Adding in mixnet to one of my ensembles gave a very slight LB score increase for me of 0.001. Usually when this happens it's due to lack of diversity between predictions but I haven't tested it in this case. I don't think mixnet is going to take you into efficientnet territory, it's more the observation of similar blocks in the networks that converge that may yield clues.
", "votes": null, "replies": [] }, { "id": 924692, "author_name": "yifanxie", "author_url": "", "post_date": "07/11/2020 15:25:05", "content": "\n\n\nM.Innat wrote:
\n \n@tivfrvqhs5 \n ensembling
\nmixnetwithefficientnettends to lower lb score! Even with TTA! Any clue why so?
one likely reason is public LB is not representative of the actual quality of over test prediction.\nI would be more interesting to check if such ensembling improve your CV
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "904562": "This competition reminds me of [IEEE's Signal Processing Society - Camera Model Identification](https://www.kaggle.com/c/sp-society-camera-model-identification) competition where at the end winning models used off-the-shelf architectures which managed to get ~0.99ish accuracy in classifying which camera took a picture (10-class classification problem).\n\nIn this competition off-the-shelf Efficientnets achieve 0.92 w-AUC w/o much effort treating this as 4-class classification problem.\n\nDoes anybody have a clue as to what the net is looking at to make the right predictions?", "906201": "Maybe it is possible that these are pretrained with natural images and this competition is also using natural images so transfer learning is easier to achieve, they share similarity in features, etc. compared to something like medical imaging or cells.", "906341": "Don't be fooled by the metric, accuracy here is still c80%.\n\nWhat is interesting is that much prior research has centred on removing the natural image through a bettery of filters: [https://www.google.com/search?q=\"steganalysis\"+\"remove+the+image+content\"](https://www.google.com/search?ei=var5XuG7FJWV1fAPgdqsmAU&q=%22steganalysis%22+%22remove+the+image+content%22&oq=%22steganalysis%22+%22remove+the+image+content%22&gs_lcp=CgZwc3ktYWIQAzIFCCEQoAFQAFgAYPo2aABwAHgAgAFtiAFtkgEDMC4xmAEAqgEHZ3dzLXdpeg&sclient=psy-ab&ved=0ahUKEwihjp7s0abqAhWVShUIHQEtC1MQ4dUDCAs&uact=5)\n\nAnd yet, using networks pretrained on natural images appears to work far better than any of the preceding feature-crafted networks that seek to remove the natural content, and are published as sota. I'm waiting to see if ABBA's solution at end turns out a new method for these carefully processed RGB images.", "918008": "Good question. The nets that seem to converge to reasonably high scores for me are MobileNet, MixNet, and EfficientNet. Depthwise separable convolutions are common between these backbones.", "918013": "Nice to see you competing, @tivfrvqhs5 =]", "918482": "Great insight. I was wondering architectural element made efficientnets work but not eg resnets.", "919088": "Expanding on this idea... FBNet, MNASNet, SinglePath NAS also converge, but don't achieve as high an accuracy as the bigger Efficientnets (they tend to level off in the Efficientnet B0-B1 range). So I'd say more generally that backbones based on the MobileNet V1/V2 block sequence tend to work. More work to do on understanding why this is the case.\nThis observation can also form the basis for custom network design.", "919536": "what is the score of MixNet? is it in the range of B1 (this is what i got) ?", "919542": "MixNet-XL w/TTA got 0.917 on the LB for me", "919569": "0.917 is what i have with B1 without TTA. Was Mobilenet any better ?", "919595": "Mobilenet V3 was 0.914, could have gotten a little more out of it but I've been characterizing performance before pushing the models as far as they can go.", "924448": "results for mixnet-L\n\n\n\n\n\nrelated: there is something better than mixnet : mux conv\n\n\n\n", "924482": "tivfrvqhs5 \nensembling `mixnet` with `efficientnet` tends to lower lb score! Even with TTA! Any clue why so?", "924617": "Adding in mixnet to one of my ensembles gave a very slight LB score increase for me of 0.001. Usually when this happens it's due to lack of diversity between predictions but I haven't tested it in this case. I don't think mixnet is going to take you into efficientnet territory, it's more the observation of similar blocks in the networks that converge that may yield clues.", "924692": "> **M.Innat wrote:**\n> \n> @tivfrvqhs5 \n> ensembling `mixnet` with `efficientnet` tends to lower lb score! Even with TTA! Any clue why so? \n> \n\none likely reason is public LB is not representative of the actual quality of over test prediction.\nI would be more interesting to check if such ensembling improve your CV" }, "source": "meta" }