{
  "id": 403494,
  "title": "Dropout layer with variable rate in TensorFlow",
  "url": "/competitions/asl-signs/discussion/403494",
  "author_name": "",
  "post_date": "2023-04-23T12:47:26.730796600Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Is it possible to make dropout layer in TF with variable rate depending on current epoch? Like rate A until epoch X, and changing after it to rate B.</p>",
  "messages": [
    {
      "id": "2231570",
      "postDate": "04/23/2023 12:47:26",
      "content": "<p>Is it possible to make dropout layer in TF with variable rate depending on current epoch? Like rate A until epoch X, and changing after it to rate B.</p>",
      "rawMarkdown": "Is it possible to make dropout layer in TF with variable rate depending on current epoch? Like rate A until epoch X, and changing after it to rate B.",
      "votes": null
    },
    {
      "id": "2231694",
      "postDate": "04/23/2023 15:03:33",
      "content": "<pre><code> (tf.keras.layers.Layer):\n     ():\n        ().__init__(**kwargs)\n        self.rate = rate\n        self.start_step = start_step\n        self.dropout = tf.keras.layers.Dropout(rate, noise_shape=noise_shape)\n\n     ():\n        ().build(input_shape)\n        agg = tf.VariableAggregation.ONLY_FIRST_REPLICA\n        self._train_counter = tf.Variable(, dtype=, aggregation=agg, trainable=)\n\n     ():\n         training:\n            x = tf.cond(self._train_counter &lt; self.start_step, :inputs,  :self.dropout(inputs,training=training))\n            self._train_counter.assign_add()\n       :\n            x = inputs\n         x\n</code></pre>\n<p>if epoch = X, you should pass argument start_step = X * steps_per_epoch</p>",
      "rawMarkdown": "```python\nclass LateDropout(tf.keras.layers.Layer):\n    def __init__(self, rate, noise_shape=None, start_step=0, **kwargs):\n        super().__init__(**kwargs)\n        self.rate = rate\n        self.start_step = start_step\n        self.dropout = tf.keras.layers.Dropout(rate, noise_shape=noise_shape)\n      \n    def build(self, input_shape):\n        super().build(input_shape)\n        agg = tf.VariableAggregation.ONLY_FIRST_REPLICA\n        self._train_counter = tf.Variable(0, dtype=\"int64\", aggregation=agg, trainable=False)\n\n    def call(self, inputs, training=False):\n        if training:\n            x = tf.cond(self._train_counter < self.start_step, lambda:inputs,  lambda:self.dropout(inputs,training=training))\n            self._train_counter.assign_add(1)\n       else:\n            x = inputs\n        return x\n```\n\nif epoch = X, you should pass argument start_step = X * steps_per_epoch",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2231694,
      "author_name": "hoyso48",
      "author_url": "",
      "post_date": "04/23/2023 15:03:33",
      "content": "<pre><code> (tf.keras.layers.Layer):\n     ():\n        ().__init__(**kwargs)\n        self.rate = rate\n        self.start_step = start_step\n        self.dropout = tf.keras.layers.Dropout(rate, noise_shape=noise_shape)\n\n     ():\n        ().build(input_shape)\n        agg = tf.VariableAggregation.ONLY_FIRST_REPLICA\n        self._train_counter = tf.Variable(, dtype=, aggregation=agg, trainable=)\n\n     ():\n         training:\n            x = tf.cond(self._train_counter &lt; self.start_step, :inputs,  :self.dropout(inputs,training=training))\n            self._train_counter.assign_add()\n       :\n            x = inputs\n         x\n</code></pre>\n<p>if epoch = X, you should pass argument start_step = X * steps_per_epoch</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2231570": "Is it possible to make dropout layer in TF with variable rate depending on current epoch? Like rate A until epoch X, and changing after it to rate B.",
    "2231694": "```python\nclass LateDropout(tf.keras.layers.Layer):\n    def __init__(self, rate, noise_shape=None, start_step=0, **kwargs):\n        super().__init__(**kwargs)\n        self.rate = rate\n        self.start_step = start_step\n        self.dropout = tf.keras.layers.Dropout(rate, noise_shape=noise_shape)\n      \n    def build(self, input_shape):\n        super().build(input_shape)\n        agg = tf.VariableAggregation.ONLY_FIRST_REPLICA\n        self._train_counter = tf.Variable(0, dtype=\"int64\", aggregation=agg, trainable=False)\n\n    def call(self, inputs, training=False):\n        if training:\n            x = tf.cond(self._train_counter < self.start_step, lambda:inputs,  lambda:self.dropout(inputs,training=training))\n            self._train_counter.assign_add(1)\n       else:\n            x = inputs\n        return x\n```\n\nif epoch = X, you should pass argument start_step = X * steps_per_epoch"
  },
  "source": "meta"
}