{
  "id": 80113,
  "title": "Optimizing for Long-Tailed Dataset",
  "url": "/competitions/humpback-whale-identification/discussion/80113",
  "author_name": "",
  "post_date": "2019-02-10T18:04:44.469225500Z",
  "votes": 15,
  "comment_count": 4,
  "views": 0,
  "content": "<p><a href=\"https://papers.nips.cc/paper/7278-learning-to-model-the-tail\">https://papers.nips.cc/paper/7278-learning-to-model-the-tail</a></p>\n\n<p>\"We describe an approach to learning from long-tailed, imbalanced datasets that are prevalent in real-world settings. Here, the challenge is to learn accurate \"few-shot'' models for classes in the tail of the class distribution, for which little data is available. We cast this problem as transfer learning, where knowledge from the data-rich classes in the head of the distribution is transferred to the data-poor classes in the tail. Our key insights are as follows. First, we propose to transfer meta-knowledge about learning-to-learn from the head classes. This knowledge is encoded with a meta-network that operates on the space of model parameters, that is trained to predict many-shot model parameters from few-shot model parameters. Second, we transfer this meta-knowledge in a progressive manner, from classes in the head to the \"body'', and from the \"body'' to the tail. That is, we transfer knowledge in a gradual fashion, regularizing meta-networks for few-shot regression with those trained with more training data. This allows our final network to capture a notion of model dynamics, that predicts how model parameters are likely to change as more training data is gradually added. We demonstrate results on image classification datasets (SUN, Places, and ImageNet) tuned for the long-tailed setting, that significantly outperform common heuristics, such as data resampling or reweighting.\"</p>\n\n<p><img src=\"https://i.ibb.co/vmrMQpw/687300980.jpg\" alt=\"\"></p>",
  "messages": [
    {
      "id": "469212",
      "postDate": "02/10/2019 18:04:44",
      "content": "<p><a href=\"https://papers.nips.cc/paper/7278-learning-to-model-the-tail\">https://papers.nips.cc/paper/7278-learning-to-model-the-tail</a></p>\n\n<p>\"We describe an approach to learning from long-tailed, imbalanced datasets that are prevalent in real-world settings. Here, the challenge is to learn accurate \"few-shot'' models for classes in the tail of the class distribution, for which little data is available. We cast this problem as transfer learning, where knowledge from the data-rich classes in the head of the distribution is transferred to the data-poor classes in the tail. Our key insights are as follows. First, we propose to transfer meta-knowledge about learning-to-learn from the head classes. This knowledge is encoded with a meta-network that operates on the space of model parameters, that is trained to predict many-shot model parameters from few-shot model parameters. Second, we transfer this meta-knowledge in a progressive manner, from classes in the head to the \"body'', and from the \"body'' to the tail. That is, we transfer knowledge in a gradual fashion, regularizing meta-networks for few-shot regression with those trained with more training data. This allows our final network to capture a notion of model dynamics, that predicts how model parameters are likely to change as more training data is gradually added. We demonstrate results on image classification datasets (SUN, Places, and ImageNet) tuned for the long-tailed setting, that significantly outperform common heuristics, such as data resampling or reweighting.\"</p>\n\n<p><img src=\"https://i.ibb.co/vmrMQpw/687300980.jpg\" alt=\"\"></p>",
      "rawMarkdown": "[https://papers.nips.cc/paper/7278-learning-to-model-the-tail][1]\n\n\"We describe an approach to learning from long-tailed, imbalanced datasets that are prevalent in real-world settings. Here, the challenge is to learn accurate \"few-shot'' models for classes in the tail of the class distribution, for which little data is available. We cast this problem as transfer learning, where knowledge from the data-rich classes in the head of the distribution is transferred to the data-poor classes in the tail. Our key insights are as follows. First, we propose to transfer meta-knowledge about learning-to-learn from the head classes. This knowledge is encoded with a meta-network that operates on the space of model parameters, that is trained to predict many-shot model parameters from few-shot model parameters. Second, we transfer this meta-knowledge in a progressive manner, from classes in the head to the \"body'', and from the \"body'' to the tail. That is, we transfer knowledge in a gradual fashion, regularizing meta-networks for few-shot regression with those trained with more training data. This allows our final network to capture a notion of model dynamics, that predicts how model parameters are likely to change as more training data is gradually added. We demonstrate results on image classification datasets (SUN, Places, and ImageNet) tuned for the long-tailed setting, that significantly outperform common heuristics, such as data resampling or reweighting.\"\n\n![][2]\n\n\n  [1]: https://papers.nips.cc/paper/7278-learning-to-model-the-tail\n  [2]: https://i.ibb.co/vmrMQpw/687300980.jpg",
      "votes": null
    },
    {
      "id": "480910",
      "postDate": "02/28/2019 20:56:16",
      "content": "<p>It is a really nice paper, never seen something like it. Did you guys try it?</p>",
      "rawMarkdown": "It is a really nice paper, never seen something like it. Did you guys try it?",
      "votes": null
    },
    {
      "id": "480919",
      "postDate": "02/28/2019 21:29:24",
      "content": "<p>I think approach like that was crucial for competition. In general, I have the problem that the minority classes have very 'general' feature (described by Net) so it was similar to many others classes. In contrast, the majority classes have 'strict' features and they were so so similar to 'new_whale'. I tried 'Cluster-Mining' mean creating the batch based on similarity between classes but unfortuantelly model just overfitt. </p>\n\n<p>Now I'm just waiting to read the final solutions :)</p>",
      "rawMarkdown": "I think approach like that was crucial for competition. In general, I have the problem that the minority classes have very 'general' feature (described by Net) so it was similar to many others classes. In contrast, the majority classes have 'strict' features and they were so so similar to 'new_whale'. I tried 'Cluster-Mining' mean creating the batch based on similarity between classes but unfortuantelly model just overfitt. \n\nNow I'm just waiting to read the final solutions :)",
      "votes": null
    },
    {
      "id": "480984",
      "postDate": "03/01/2019 00:21:46",
      "content": "<p>Actually, we didn't have enough time to try this out :(</p>",
      "rawMarkdown": "Actually, we didn't have enough time to try this out :(",
      "votes": null
    },
    {
      "id": "480991",
      "postDate": "03/01/2019 00:26:16",
      "content": "<p>I know the feeling but you guys won anyway! Congratulations :))</p>",
      "rawMarkdown": "I know the feeling but you guys won anyway! Congratulations :))",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 480910,
      "author_name": "arc144",
      "author_url": "",
      "post_date": "02/28/2019 20:56:16",
      "content": "<p>It is a really nice paper, never seen something like it. Did you guys try it?</p>",
      "votes": null,
      "replies": [
        {
          "id": 480984,
          "author_name": "alexanderliao",
          "author_url": "",
          "post_date": "03/01/2019 00:21:46",
          "content": "<p>Actually, we didn't have enough time to try this out :(</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480991,
          "author_name": "arc144",
          "author_url": "",
          "post_date": "03/01/2019 00:26:16",
          "content": "<p>I know the feeling but you guys won anyway! Congratulations :))</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480919,
      "author_name": "melgor",
      "author_url": "",
      "post_date": "02/28/2019 21:29:24",
      "content": "<p>I think approach like that was crucial for competition. In general, I have the problem that the minority classes have very 'general' feature (described by Net) so it was similar to many others classes. In contrast, the majority classes have 'strict' features and they were so so similar to 'new_whale'. I tried 'Cluster-Mining' mean creating the batch based on similarity between classes but unfortuantelly model just overfitt. </p>\n\n<p>Now I'm just waiting to read the final solutions :)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "469212": "[https://papers.nips.cc/paper/7278-learning-to-model-the-tail][1]\n\n\"We describe an approach to learning from long-tailed, imbalanced datasets that are prevalent in real-world settings. Here, the challenge is to learn accurate \"few-shot'' models for classes in the tail of the class distribution, for which little data is available. We cast this problem as transfer learning, where knowledge from the data-rich classes in the head of the distribution is transferred to the data-poor classes in the tail. Our key insights are as follows. First, we propose to transfer meta-knowledge about learning-to-learn from the head classes. This knowledge is encoded with a meta-network that operates on the space of model parameters, that is trained to predict many-shot model parameters from few-shot model parameters. Second, we transfer this meta-knowledge in a progressive manner, from classes in the head to the \"body'', and from the \"body'' to the tail. That is, we transfer knowledge in a gradual fashion, regularizing meta-networks for few-shot regression with those trained with more training data. This allows our final network to capture a notion of model dynamics, that predicts how model parameters are likely to change as more training data is gradually added. We demonstrate results on image classification datasets (SUN, Places, and ImageNet) tuned for the long-tailed setting, that significantly outperform common heuristics, such as data resampling or reweighting.\"\n\n![][2]\n\n\n  [1]: https://papers.nips.cc/paper/7278-learning-to-model-the-tail\n  [2]: https://i.ibb.co/vmrMQpw/687300980.jpg",
    "480910": "It is a really nice paper, never seen something like it. Did you guys try it?",
    "480919": "I think approach like that was crucial for competition. In general, I have the problem that the minority classes have very 'general' feature (described by Net) so it was similar to many others classes. In contrast, the majority classes have 'strict' features and they were so so similar to 'new_whale'. I tried 'Cluster-Mining' mean creating the batch based on similarity between classes but unfortuantelly model just overfitt. \n\nNow I'm just waiting to read the final solutions :)",
    "480984": "Actually, we didn't have enough time to try this out :(",
    "480991": "I know the feeling but you guys won anyway! Congratulations :))"
  },
  "source": "meta"
}