{
  "id": 39223,
  "title": "Another dead end for Kaggle!",
  "url": "/competitions/carvana-image-masking-challenge/discussion/39223",
  "author_name": "",
  "post_date": "2017-09-09T17:56:23.142299700Z",
  "votes": -5,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Unfortunately, in this competition there is another dead end.\nAs in Amazon Planet and many others.</p>\n\n<p>Again, everything is decided by brute force (image resolution) and a combination of ensembles.\nHow do you imagine this in real work?\nWill the customer arrange such a decision?\nThis has a very negative impact on the reputation of the whole Kaagle community and the direction of Deep lerning in general.</p>\n\n<p>We need something more cardinal and more intelligent.\nI get 0.9966 LB on one UNET512\n1 epoch at NVIDIA 1080 takes 4 minutes.\nUsing a simple trick with Transfer Learning. </p>\n\n<p>I also get 0.9988 on the training set.\nUsing a combination of two UNET128 and UNET512 and some ideas from GAN.\nUnfortunately, with this huge overfitting LB 0.995. =(</p>\n\n<p>In my opinion, the decision to be here.\n\"Unsupervised Domain Adaptation\"\n<a href=\"https://arxiv.org/pdf/1409.7495.pdf\">https://arxiv.org/pdf/1409.7495.pdf</a></p>\n\n<p>The key to real success lies in the organization of the learning process, and not in primitive methods like ensembles.</p>\n\n<p>I welcome any non-standard ideas.\nI'm ready to publish technical details.</p>",
  "messages": [
    {
      "id": "219792",
      "postDate": "09/09/2017 17:56:23",
      "content": "<p>Unfortunately, in this competition there is another dead end.\nAs in Amazon Planet and many others.</p>\n\n<p>Again, everything is decided by brute force (image resolution) and a combination of ensembles.\nHow do you imagine this in real work?\nWill the customer arrange such a decision?\nThis has a very negative impact on the reputation of the whole Kaagle community and the direction of Deep lerning in general.</p>\n\n<p>We need something more cardinal and more intelligent.\nI get 0.9966 LB on one UNET512\n1 epoch at NVIDIA 1080 takes 4 minutes.\nUsing a simple trick with Transfer Learning. </p>\n\n<p>I also get 0.9988 on the training set.\nUsing a combination of two UNET128 and UNET512 and some ideas from GAN.\nUnfortunately, with this huge overfitting LB 0.995. =(</p>\n\n<p>In my opinion, the decision to be here.\n\"Unsupervised Domain Adaptation\"\n<a href=\"https://arxiv.org/pdf/1409.7495.pdf\">https://arxiv.org/pdf/1409.7495.pdf</a></p>\n\n<p>The key to real success lies in the organization of the learning process, and not in primitive methods like ensembles.</p>\n\n<p>I welcome any non-standard ideas.\nI'm ready to publish technical details.</p>",
      "rawMarkdown": "Unfortunately, in this competition there is another dead end.\nAs in Amazon Planet and many others.\n\nAgain, everything is decided by brute force (image resolution) and a combination of ensembles.\nHow do you imagine this in real work?\nWill the customer arrange such a decision?\nThis has a very negative impact on the reputation of the whole Kaagle community and the direction of Deep lerning in general.\n\n\nWe need something more cardinal and more intelligent.\nI get 0.9966 LB on one UNET512\n1 epoch at NVIDIA 1080 takes 4 minutes.\nUsing a simple trick with Transfer Learning. \n\nI also get 0.9988 on the training set.\nUsing a combination of two UNET128 and UNET512 and some ideas from GAN.\nUnfortunately, with this huge overfitting LB 0.995. =(\n\nIn my opinion, the decision to be here.\n\"Unsupervised Domain Adaptation\"\nhttps://arxiv.org/pdf/1409.7495.pdf\n\nThe key to real success lies in the organization of the learning process, and not in primitive methods like ensembles.\n\nI welcome any non-standard ideas.\nI'm ready to publish technical details.",
      "votes": null
    },
    {
      "id": "219814",
      "postDate": "09/09/2017 19:00:32",
      "content": "<p>The top-ranking participants usually have the best ensembles. You are right in this regard. But you forgot they usually also have one if not the best single classifier in their ensemble.\nAlso usually every solution in the top-20 (and in this competition probably even more) are normally totally sufficient for the customer. I do not see how this casts a poor light onto the community...</p>",
      "rawMarkdown": "The top-ranking participants usually have the best ensembles. You are right in this regard. But you forgot they usually also have one if not the best single classifier in their ensemble.\nAlso usually every solution in the top-20 (and in this competition probably even more) are normally totally sufficient for the customer. I do not see how this casts a poor light onto the community...",
      "votes": null
    },
    {
      "id": "219822",
      "postDate": "09/09/2017 19:25:38",
      "content": "<p>Tim,\nI have nothing against the ensemble as a concept.\nI am against attempts to achieve an improvement in the result solely in this way.</p>\n\n<p>Here is an example - how it can be done differently and improve the generalization of models.</p>\n\n<p>\"Unsupervised domain adaptation in the brain lesion segmentation with adversarial networks\"\n<a href=\"https://arxiv.org/abs/1612.08894\">https://arxiv.org/abs/1612.08894</a></p>\n\n<p>Organization of the learning process, work with losses and new architectural solutions.\nCan give a qualitative leap for the entire direction of Deep lerning</p>",
      "rawMarkdown": "Tim,\nI have nothing against the ensemble as a concept.\nI am against attempts to achieve an improvement in the result solely in this way.\n\nHere is an example - how it can be done differently and improve the generalization of models.\n\n\"Unsupervised domain adaptation in the brain lesion segmentation with adversarial networks\"\nhttps://arxiv.org/abs/1612.08894\n\nOrganization of the learning process, work with losses and new architectural solutions.\nCan give a qualitative leap for the entire direction of Deep lerning",
      "votes": null
    },
    {
      "id": "219827",
      "postDate": "09/09/2017 19:40:52",
      "content": "<p>I see, I just wanted clear up, that there is nothing wrong with doing ensembles for these competitions.\nAnd I see your point. But you have to understand. Most of the people here are not researches I think. So you have either students or professionals. I think creating new architectures and evaluating them is not feasible within the deadline. If you look at the recent SOTA achievements I think we can agree you need to be a top researcher with a lot of time to do your research and often (especially for this competition) you will need a lot of compute power. So instead of going for completely new approaches people try already existing SOTA architectures. Which makes sense because for students like me it is a great way to even get to know the existing architectures before I would be able to create something new. And for everyone else I think they would rather get the price money with 90% chance and only 1% improvement over the 2nd place vs 1% chance and 10% improvement over the second place.</p>",
      "rawMarkdown": "I see, I just wanted clear up, that there is nothing wrong with doing ensembles for these competitions.\nAnd I see your point. But you have to understand. Most of the people here are not researches I think. So you have either students or professionals. I think creating new architectures and evaluating them is not feasible within the deadline. If you look at the recent SOTA achievements I think we can agree you need to be a top researcher with a lot of time to do your research and often (especially for this competition) you will need a lot of compute power. So instead of going for completely new approaches people try already existing SOTA architectures. Which makes sense because for students like me it is a great way to even get to know the existing architectures before I would be able to create something new. And for everyone else I think they would rather get the price money with 90% chance and only 1% improvement over the 2nd place vs 1% chance and 10% improvement over the second place.",
      "votes": null
    },
    {
      "id": "219843",
      "postDate": "09/09/2017 22:10:29",
      "content": "<p>There is no right or wrong answer. It is a just a matter of choice on how you want to get your results. It also depend if you want to be \"results orientated\" or \"method orientated\".\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/219843/7273/methods.png\" alt=\"enter image description here\" title=\"\"></p>\n\n<p>In real product development for customer, it is usually:</p>\n\n<ol>\n<li><p>show that it works (proof of concept)! ... together with the limits and problems ...</p></li>\n<li><p>make it work! (scale-up) ... e.g. work within certain memory/computing power/data</p></li>\n</ol>\n\n<p>I don't think kaggle solution is for phase 2. Someone is going to take solution from kaggle to refine it for real commerical applications later, and there are many ways to do it. (e.g. it is possible to reduce ensemble classifiers to single classifer)</p>",
      "rawMarkdown": "There is no right or wrong answer. It is a just a matter of choice on how you want to get your results. It also depend if you want to be \"results orientated\" or \"method orientated\".\n  ![enter image description here][1]\n\n\n\nIn real product development for customer, it is usually:\n\n1.  show that it works (proof of concept)! ... together with the limits and problems ...\n\n2. make it work! (scale-up) ... e.g. work within certain memory/computing power/data\n\nI don't think kaggle solution is for phase 2. Someone is going to take solution from kaggle to refine it for real commerical applications later, and there are many ways to do it. (e.g. it is possible to reduce ensemble classifiers to single classifer)\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/219843/7273/methods.png",
      "votes": null
    },
    {
      "id": "220134",
      "postDate": "09/11/2017 10:10:59",
      "content": "<p>You bring up a good point, but:</p>\n\n<p>a) How do you know what approaches competitors are taking? Speak for yourself.</p>\n\n<p>b) Why the negativity?</p>\n\n<p>Engineering is about balancing tradeoffs, in this competition the score is LB, not efficiency, novelty, etc. You can always come up with a novel network and write a paper, if you haven't done so already.</p>",
      "rawMarkdown": "You bring up a good point, but:\n\na) How do you know what approaches competitors are taking? Speak for yourself.\n\nb) Why the negativity?\n\nEngineering is about balancing tradeoffs, in this competition the score is LB, not efficiency, novelty, etc. You can always come up with a novel network and write a paper, if you haven't done so already.",
      "votes": null
    },
    {
      "id": "220172",
      "postDate": "09/11/2017 12:00:31",
      "content": "<p>Andreas,</p>\n\n<p>For me, Kaggle is very personal.\nIf there was something on your street in your favorite city that you really do not like.\nWould you not go out and express your opinion?</p>\n\n<p>Methods - I judge the situation based on the messages on the forum.</p>\n\n<p>At the same time I understand that \"Unsupervised Domain Adaptation\" and creation of a new method of searching for invariants is not a task for a single Kaagle competition.\nBut for sure there are many engineering ideas that can bring us closer to solving the problem. Not this time, so the next.\nThe accumulated experience and knowledge of the whole community will ultimately lead us to a common success.</p>",
      "rawMarkdown": "Andreas,\n \nFor me, Kaggle is very personal.\nIf there was something on your street in your favorite city that you really do not like.\nWould you not go out and express your opinion?\n \nMethods - I judge the situation based on the messages on the forum.\n\nAt the same time I understand that \"Unsupervised Domain Adaptation\" and creation of a new method of searching for invariants is not a task for a single Kaagle competition.\nBut for sure there are many engineering ideas that can bring us closer to solving the problem. Not this time, so the next.\nThe accumulated experience and knowledge of the whole community will ultimately lead us to a common success.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 219814,
      "author_name": "timjoseph",
      "author_url": "",
      "post_date": "09/09/2017 19:00:32",
      "content": "<p>The top-ranking participants usually have the best ensembles. You are right in this regard. But you forgot they usually also have one if not the best single classifier in their ensemble.\nAlso usually every solution in the top-20 (and in this competition probably even more) are normally totally sufficient for the customer. I do not see how this casts a poor light onto the community...</p>",
      "votes": null,
      "replies": [
        {
          "id": 219822,
          "author_name": "markpopov",
          "author_url": "",
          "post_date": "09/09/2017 19:25:38",
          "content": "<p>Tim,\nI have nothing against the ensemble as a concept.\nI am against attempts to achieve an improvement in the result solely in this way.</p>\n\n<p>Here is an example - how it can be done differently and improve the generalization of models.</p>\n\n<p>\"Unsupervised domain adaptation in the brain lesion segmentation with adversarial networks\"\n<a href=\"https://arxiv.org/abs/1612.08894\">https://arxiv.org/abs/1612.08894</a></p>\n\n<p>Organization of the learning process, work with losses and new architectural solutions.\nCan give a qualitative leap for the entire direction of Deep lerning</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 219827,
          "author_name": "timjoseph",
          "author_url": "",
          "post_date": "09/09/2017 19:40:52",
          "content": "<p>I see, I just wanted clear up, that there is nothing wrong with doing ensembles for these competitions.\nAnd I see your point. But you have to understand. Most of the people here are not researches I think. So you have either students or professionals. I think creating new architectures and evaluating them is not feasible within the deadline. If you look at the recent SOTA achievements I think we can agree you need to be a top researcher with a lot of time to do your research and often (especially for this competition) you will need a lot of compute power. So instead of going for completely new approaches people try already existing SOTA architectures. Which makes sense because for students like me it is a great way to even get to know the existing architectures before I would be able to create something new. And for everyone else I think they would rather get the price money with 90% chance and only 1% improvement over the 2nd place vs 1% chance and 10% improvement over the second place.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 219843,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "09/09/2017 22:10:29",
      "content": "<p>There is no right or wrong answer. It is a just a matter of choice on how you want to get your results. It also depend if you want to be \"results orientated\" or \"method orientated\".\n  <img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/219843/7273/methods.png\" alt=\"enter image description here\" title=\"\"></p>\n\n<p>In real product development for customer, it is usually:</p>\n\n<ol>\n<li><p>show that it works (proof of concept)! ... together with the limits and problems ...</p></li>\n<li><p>make it work! (scale-up) ... e.g. work within certain memory/computing power/data</p></li>\n</ol>\n\n<p>I don't think kaggle solution is for phase 2. Someone is going to take solution from kaggle to refine it for real commerical applications later, and there are many ways to do it. (e.g. it is possible to reduce ensemble classifiers to single classifer)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 220134,
      "author_name": "antorsae",
      "author_url": "",
      "post_date": "09/11/2017 10:10:59",
      "content": "<p>You bring up a good point, but:</p>\n\n<p>a) How do you know what approaches competitors are taking? Speak for yourself.</p>\n\n<p>b) Why the negativity?</p>\n\n<p>Engineering is about balancing tradeoffs, in this competition the score is LB, not efficiency, novelty, etc. You can always come up with a novel network and write a paper, if you haven't done so already.</p>",
      "votes": null,
      "replies": [
        {
          "id": 220172,
          "author_name": "markpopov",
          "author_url": "",
          "post_date": "09/11/2017 12:00:31",
          "content": "<p>Andreas,</p>\n\n<p>For me, Kaggle is very personal.\nIf there was something on your street in your favorite city that you really do not like.\nWould you not go out and express your opinion?</p>\n\n<p>Methods - I judge the situation based on the messages on the forum.</p>\n\n<p>At the same time I understand that \"Unsupervised Domain Adaptation\" and creation of a new method of searching for invariants is not a task for a single Kaagle competition.\nBut for sure there are many engineering ideas that can bring us closer to solving the problem. Not this time, so the next.\nThe accumulated experience and knowledge of the whole community will ultimately lead us to a common success.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "219792": "Unfortunately, in this competition there is another dead end.\nAs in Amazon Planet and many others.\n\nAgain, everything is decided by brute force (image resolution) and a combination of ensembles.\nHow do you imagine this in real work?\nWill the customer arrange such a decision?\nThis has a very negative impact on the reputation of the whole Kaagle community and the direction of Deep lerning in general.\n\n\nWe need something more cardinal and more intelligent.\nI get 0.9966 LB on one UNET512\n1 epoch at NVIDIA 1080 takes 4 minutes.\nUsing a simple trick with Transfer Learning. \n\nI also get 0.9988 on the training set.\nUsing a combination of two UNET128 and UNET512 and some ideas from GAN.\nUnfortunately, with this huge overfitting LB 0.995. =(\n\nIn my opinion, the decision to be here.\n\"Unsupervised Domain Adaptation\"\nhttps://arxiv.org/pdf/1409.7495.pdf\n\nThe key to real success lies in the organization of the learning process, and not in primitive methods like ensembles.\n\nI welcome any non-standard ideas.\nI'm ready to publish technical details.",
    "219814": "The top-ranking participants usually have the best ensembles. You are right in this regard. But you forgot they usually also have one if not the best single classifier in their ensemble.\nAlso usually every solution in the top-20 (and in this competition probably even more) are normally totally sufficient for the customer. I do not see how this casts a poor light onto the community...",
    "219822": "Tim,\nI have nothing against the ensemble as a concept.\nI am against attempts to achieve an improvement in the result solely in this way.\n\nHere is an example - how it can be done differently and improve the generalization of models.\n\n\"Unsupervised domain adaptation in the brain lesion segmentation with adversarial networks\"\nhttps://arxiv.org/abs/1612.08894\n\nOrganization of the learning process, work with losses and new architectural solutions.\nCan give a qualitative leap for the entire direction of Deep lerning",
    "219827": "I see, I just wanted clear up, that there is nothing wrong with doing ensembles for these competitions.\nAnd I see your point. But you have to understand. Most of the people here are not researches I think. So you have either students or professionals. I think creating new architectures and evaluating them is not feasible within the deadline. If you look at the recent SOTA achievements I think we can agree you need to be a top researcher with a lot of time to do your research and often (especially for this competition) you will need a lot of compute power. So instead of going for completely new approaches people try already existing SOTA architectures. Which makes sense because for students like me it is a great way to even get to know the existing architectures before I would be able to create something new. And for everyone else I think they would rather get the price money with 90% chance and only 1% improvement over the 2nd place vs 1% chance and 10% improvement over the second place.",
    "219843": "There is no right or wrong answer. It is a just a matter of choice on how you want to get your results. It also depend if you want to be \"results orientated\" or \"method orientated\".\n  ![enter image description here][1]\n\n\n\nIn real product development for customer, it is usually:\n\n1.  show that it works (proof of concept)! ... together with the limits and problems ...\n\n2. make it work! (scale-up) ... e.g. work within certain memory/computing power/data\n\nI don't think kaggle solution is for phase 2. Someone is going to take solution from kaggle to refine it for real commerical applications later, and there are many ways to do it. (e.g. it is possible to reduce ensemble classifiers to single classifer)\n\n\n  [1]: https://kaggle2.blob.core.windows.net/forum-message-attachments/219843/7273/methods.png",
    "220134": "You bring up a good point, but:\n\na) How do you know what approaches competitors are taking? Speak for yourself.\n\nb) Why the negativity?\n\nEngineering is about balancing tradeoffs, in this competition the score is LB, not efficiency, novelty, etc. You can always come up with a novel network and write a paper, if you haven't done so already.",
    "220172": "Andreas,\n \nFor me, Kaggle is very personal.\nIf there was something on your street in your favorite city that you really do not like.\nWould you not go out and express your opinion?\n \nMethods - I judge the situation based on the messages on the forum.\n\nAt the same time I understand that \"Unsupervised Domain Adaptation\" and creation of a new method of searching for invariants is not a task for a single Kaagle competition.\nBut for sure there are many engineering ideas that can bring us closer to solving the problem. Not this time, so the next.\nThe accumulated experience and knowledge of the whole community will ultimately lead us to a common success."
  },
  "source": "meta"
}