{
  "id": 306579,
  "title": "The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning",
  "url": "/competitions/h-and-m-personalized-fashion-recommendations/discussion/306579",
  "author_name": "",
  "post_date": "2022-02-10T02:21:16.990953900Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p><strong>Research paper excerpt:</strong></p>\n<p>Decision-making AI agents are often faced with two important challenges: the depth of the planning horizon, and the branching factor due to having many choices. Hierarchical reinforcement learning methods aim to solve the first problem, by providing shortcuts that skip over multiple time steps. To cope with the breadth, it is desirable to restrict the agent's attention at each step to a reasonable number of possible choices. The concept of affordances (Gibson, 1977) suggests that only certain actions are feasible in certain states. In this work, we model \"affordances\" through an attention mechanism that limits the available choices of temporally extended options. We present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. We investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices. We identify and empirically illustrate the settings in which the paradox of choice arises, i.e. when having fewer but more meaningful choices improves the learning speed and performance of a reinforcement learning agent.</p>\n<p><strong><a href=\"https://arxiv.org/pdf/2201.09653.pdf\" target=\"_blank\">Read the paper</a></strong></p>\n<p><strong><a href=\"https://github.com/andreicnica/hrl_attention\" target=\"_blank\">Code on Github</a></strong></p>",
  "messages": [
    {
      "id": "1683718",
      "postDate": "02/10/2022 02:21:16",
      "content": "<p><strong>Research paper excerpt:</strong></p>\n<p>Decision-making AI agents are often faced with two important challenges: the depth of the planning horizon, and the branching factor due to having many choices. Hierarchical reinforcement learning methods aim to solve the first problem, by providing shortcuts that skip over multiple time steps. To cope with the breadth, it is desirable to restrict the agent's attention at each step to a reasonable number of possible choices. The concept of affordances (Gibson, 1977) suggests that only certain actions are feasible in certain states. In this work, we model \"affordances\" through an attention mechanism that limits the available choices of temporally extended options. We present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. We investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices. We identify and empirically illustrate the settings in which the paradox of choice arises, i.e. when having fewer but more meaningful choices improves the learning speed and performance of a reinforcement learning agent.</p>\n<p><strong><a href=\"https://arxiv.org/pdf/2201.09653.pdf\" target=\"_blank\">Read the paper</a></strong></p>\n<p><strong><a href=\"https://github.com/andreicnica/hrl_attention\" target=\"_blank\">Code on Github</a></strong></p>",
      "rawMarkdown": "**Research paper excerpt:**\n\nDecision-making AI agents are often faced with two important challenges: the depth of the planning horizon, and the branching factor due to having many choices. Hierarchical reinforcement learning methods aim to solve the first problem, by providing shortcuts that skip over multiple time steps. To cope with the breadth, it is desirable to restrict the agent's attention at each step to a reasonable number of possible choices. The concept of affordances (Gibson, 1977) suggests that only certain actions are feasible in certain states. In this work, we model \"affordances\" through an attention mechanism that limits the available choices of temporally extended options. We present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. We investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices. We identify and empirically illustrate the settings in which the paradox of choice arises, i.e. when having fewer but more meaningful choices improves the learning speed and performance of a reinforcement learning agent.\n\n**[Read the paper](https://arxiv.org/pdf/2201.09653.pdf)**\n\n**[Code on Github](https://github.com/andreicnica/hrl_attention)**",
      "votes": null
    },
    {
      "id": "1684728",
      "postDate": "02/10/2022 16:59:46",
      "content": "<p>Sorry if I'm struggling to understand this-how do you imagine applying RL in this competition? </p>\n<p>Or is there a special trick in the paper? (I just skimmed through so I don't have a thorough understanding)</p>",
      "rawMarkdown": "Sorry if I'm struggling to understand this-how do you imagine applying RL in this competition? \n\nOr is there a special trick in the paper? (I just skimmed through so I don't have a thorough understanding)",
      "votes": null
    },
    {
      "id": "1684736",
      "postDate": "02/10/2022 17:05:40",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/init27\" target=\"_blank\">@init27</a> I wasn't intending this to be the solution vs. are there things that could be used in association with other methods. This line is what caught my attention: \"investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices\"</p>",
      "rawMarkdown": "Hi @init27 I wasn't intending this to be the solution vs. are there things that could be used in association with other methods. This line is what caught my attention: \"investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices\"",
      "votes": null
    },
    {
      "id": "1686050",
      "postDate": "02/11/2022 18:10:40",
      "content": "<p>Thanks for clarifying! :) </p>\n<p>My intuition suggests since RL is distant from CV this might not apply here 😅</p>",
      "rawMarkdown": "Thanks for clarifying! :) \n\nMy intuition suggests since RL is distant from CV this might not apply here 😅",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1684728,
      "author_name": "init27",
      "author_url": "",
      "post_date": "02/10/2022 16:59:46",
      "content": "<p>Sorry if I'm struggling to understand this-how do you imagine applying RL in this competition? </p>\n<p>Or is there a special trick in the paper? (I just skimmed through so I don't have a thorough understanding)</p>",
      "votes": null,
      "replies": [
        {
          "id": 1684736,
          "author_name": "crained",
          "author_url": "",
          "post_date": "02/10/2022 17:05:40",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/init27\" target=\"_blank\">@init27</a> I wasn't intending this to be the solution vs. are there things that could be used in association with other methods. This line is what caught my attention: \"investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices\"</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1686050,
          "author_name": "init27",
          "author_url": "",
          "post_date": "02/11/2022 18:10:40",
          "content": "<p>Thanks for clarifying! :) </p>\n<p>My intuition suggests since RL is distant from CV this might not apply here 😅</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1683718": "**Research paper excerpt:**\n\nDecision-making AI agents are often faced with two important challenges: the depth of the planning horizon, and the branching factor due to having many choices. Hierarchical reinforcement learning methods aim to solve the first problem, by providing shortcuts that skip over multiple time steps. To cope with the breadth, it is desirable to restrict the agent's attention at each step to a reasonable number of possible choices. The concept of affordances (Gibson, 1977) suggests that only certain actions are feasible in certain states. In this work, we model \"affordances\" through an attention mechanism that limits the available choices of temporally extended options. We present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. We investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices. We identify and empirically illustrate the settings in which the paradox of choice arises, i.e. when having fewer but more meaningful choices improves the learning speed and performance of a reinforcement learning agent.\n\n**[Read the paper](https://arxiv.org/pdf/2201.09653.pdf)**\n\n**[Code on Github](https://github.com/andreicnica/hrl_attention)**",
    "1684728": "Sorry if I'm struggling to understand this-how do you imagine applying RL in this competition? \n\nOr is there a special trick in the paper? (I just skimmed through so I don't have a thorough understanding)",
    "1684736": "Hi @init27 I wasn't intending this to be the solution vs. are there things that could be used in association with other methods. This line is what caught my attention: \"investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices\"",
    "1686050": "Thanks for clarifying! :) \n\nMy intuition suggests since RL is distant from CV this might not apply here 😅"
  },
  "source": "meta"
}