![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1984321%2Fde889b3780cca3ee13c8505e6798b7e6%2F1.png?generation=1608595823213095&alt=media\")
A recent work, quite interesting. In case you're interested, check it out: ARELU:, code.
\n
class AReLU(nn.Module):\n def __init__(self, alpha=0.90, beta=2.0):\n super().__init__()\n self.alpha = nn.Parameter(torch.tensor([alpha]))\n self.beta = nn.Parameter(torch.tensor([beta]))\n\n def forward(self, input):\n alpha = torch.clamp(self.alpha, min=0.01, max=0.99)\n beta = 1 + torch.sigmoid(self.beta)\n\n return F.relu(input) * beta - F.relu(-input) * alpha\n# by a function \ndef ARelu(x, alpha=0.90, beta=2.0):\n alpha = tf.clip_by_value(alpha, clip_value_min=0.01, \n clip_value_max=0.99)\n beta = 1 + tf.math.sigmoid(beta)\n return tf.nn.relu(x) * beta - tf.nn.relu(-x) * alpha\n# or by a custom layer\nclass ARelu(tf.keras.layers.Layer):\n def __init__(self, alpha=0.90, beta=2.0, **kwargs):\n super(ARelu, self).__init__(**kwargs)\n self.alpha = 0.90\n self.beta = 2.0\n\n def call(self, inputs, training=None):\n alpha = tf.clip_by_value(self.alpha, clip_value_min=0.01, \n clip_value_max=0.99)\n beta = 1 + tf.math.sigmoid(self.beta)\n return tf.nn.relu(inputs) * beta - tf.nn.relu(-inputs) * alpha\nA recent work, quite interesting. In case you're interested, check it out: ARELU:, code.
\n
class AReLU(nn.Module):\n def __init__(self, alpha=0.90, beta=2.0):\n super().__init__()\n self.alpha = nn.Parameter(torch.tensor([alpha]))\n self.beta = nn.Parameter(torch.tensor([beta]))\n\n def forward(self, input):\n alpha = torch.clamp(self.alpha, min=0.01, max=0.99)\n beta = 1 + torch.sigmoid(self.beta)\n\n return F.relu(input) * beta - F.relu(-input) * alpha\n# by a function \ndef ARelu(x, alpha=0.90, beta=2.0):\n alpha = tf.clip_by_value(alpha, clip_value_min=0.01, \n clip_value_max=0.99)\n beta = 1 + tf.math.sigmoid(beta)\n return tf.nn.relu(x) * beta - tf.nn.relu(-x) * alpha\n# or by a custom layer\nclass ARelu(tf.keras.layers.Layer):\n def __init__(self, alpha=0.90, beta=2.0, **kwargs):\n super(ARelu, self).__init__(**kwargs)\n self.alpha = 0.90\n self.beta = 2.0\n\n def call(self, inputs, training=None):\n alpha = tf.clip_by_value(self.alpha, clip_value_min=0.01, \n clip_value_max=0.99)\n beta = 1 + tf.math.sigmoid(self.beta)\n return tf.nn.relu(inputs) * beta - tf.nn.relu(-inputs) * alpha\nThank you for sharing, have to try it out.
\nFun fact : pytorch uses the name clamp and tensorflow uses the name clip.
\nclamp, clip 👍
Thanks for sharing.
\nTo readers who wonder what the function looks like:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1700650%2F5427cb8f8d7e08f57c4255b94e912816%2Farelu.png?generation=1608804119069252&alt=media\")
Very good! What is it's key idea? Why they benefits come from?
", "rawMarkdown": "Very good! What is it's key idea? Why they benefits come from?", "votes": null }, { "id": "1129934", "postDate": "12/28/2020 16:53:16", "content": "From the abstraction
\n\n\nWe propose a new perspective of learnable activation function through formulating
\n
\n them with element-wise attention mechanism. In each network layer, we devise
\n an attention module which learns an element-wise, sign-based attention map for
\n the pre-activation feature map. The attention map scales an element based on its
\n sign. Adding the attention module with a rectified linear unit (ReLU) results in
\n an amplification of positive elements and a suppression of negative ones, both
\n with learned, data-adaptive parameters.
You can read the paper, they provide comprehensive studies.
", "rawMarkdown": "From the abstraction\n\n> We propose a new perspective of **learnable activation function** through formulating\nthem with **element-wise** attention mechanism. In each network layer, we devise\nan attention module which learns an element-wise, **sign-based attention map** for\nthe pre-activation feature map. The attention map scales an element based on its\nsign. Adding the attention module with a rectified linear unit (ReLU) results in\nan **amplification of positive elements and a suppression of negative ones**, both\nwith learned, data-adaptive parameters. \n\nYou can read the [paper](https://arxiv.org/pdf/2006.13858.pdf), they provide comprehensive studies.", "votes": null } ], "comments": [ { "id": 1124340, "author_name": "alincijov", "author_url": "", "post_date": "12/23/2020 20:51:32", "content": "Thank you for sharing, have to try it out.
\nFun fact : pytorch uses the name clamp and tensorflow uses the name clip.
\nclamp, clip 👍
Thanks for sharing.
\nTo readers who wonder what the function looks like:
\n
Very good! What is it's key idea? Why they benefits come from?
", "votes": null, "replies": [ { "id": 1129934, "author_name": "ipythonx", "author_url": "", "post_date": "12/28/2020 16:53:16", "content": "From the abstraction
\n\n\nWe propose a new perspective of learnable activation function through formulating
\n
\n them with element-wise attention mechanism. In each network layer, we devise
\n an attention module which learns an element-wise, sign-based attention map for
\n the pre-activation feature map. The attention map scales an element based on its
\n sign. Adding the attention module with a rectified linear unit (ReLU) results in
\n an amplification of positive elements and a suppression of negative ones, both
\n with learned, data-adaptive parameters.
You can read the paper, they provide comprehensive studies.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1121831": "A recent work, quite interesting. In case you're interested, check it out: [**ARELU:**](https://arxiv.org/pdf/2006.13858.pdf), [code](https://github.com/densechen/AReLU). \n\n\n\n### Pytorch Implementation [Official]\n\n```\nclass AReLU(nn.Module):\n def __init__(self, alpha=0.90, beta=2.0):\n super().__init__()\n self.alpha = nn.Parameter(torch.tensor([alpha]))\n self.beta = nn.Parameter(torch.tensor([beta]))\n\n def forward(self, input):\n alpha = torch.clamp(self.alpha, min=0.01, max=0.99)\n beta = 1 + torch.sigmoid(self.beta)\n\n return F.relu(input) * beta - F.relu(-input) * alpha\n```\n\n### TensorFlow/Keras Implementation [Un-official]\n\n```\n# by a function \ndef ARelu(x, alpha=0.90, beta=2.0):\n alpha = tf.clip_by_value(alpha, clip_value_min=0.01, \n clip_value_max=0.99)\n beta = 1 + tf.math.sigmoid(beta)\n return tf.nn.relu(x) * beta - tf.nn.relu(-x) * alpha\n```\n\n```\n# or by a custom layer\nclass ARelu(tf.keras.layers.Layer):\n def __init__(self, alpha=0.90, beta=2.0, **kwargs):\n super(ARelu, self).__init__(**kwargs)\n self.alpha = 0.90\n self.beta = 2.0\n\n def call(self, inputs, training=None):\n alpha = tf.clip_by_value(self.alpha, clip_value_min=0.01, \n clip_value_max=0.99)\n beta = 1 + tf.math.sigmoid(self.beta)\n return tf.nn.relu(inputs) * beta - tf.nn.relu(-inputs) * alpha\n```", "1124340": "Thank you for sharing, have to try it out.\nFun fact : pytorch uses the name clamp and tensorflow uses the name clip. \nclamp, clip 👍", "1124983": "Thanks for sharing. \nTo readers who wonder what the function looks like: \n", "1128278": "Very good! What is it's key idea? Why they benefits come from?", "1129934": "From the abstraction\n\n> We propose a new perspective of **learnable activation function** through formulating\nthem with **element-wise** attention mechanism. In each network layer, we devise\nan attention module which learns an element-wise, **sign-based attention map** for\nthe pre-activation feature map. The attention map scales an element based on its\nsign. Adding the attention module with a rectified linear unit (ReLU) results in\nan **amplification of positive elements and a suppression of negative ones**, both\nwith learned, data-adaptive parameters. \n\nYou can read the [paper](https://arxiv.org/pdf/2006.13858.pdf), they provide comprehensive studies." }, "source": "meta" }