{ "id": 312631, "title": "The Neural Addition Unit (NAU)", "url": "/competitions/ultra-mnist/discussion/312631", "author_name": "Carl McBride Ellis", "post_date": "2022-03-13T06:53:01.543000", "votes": 14, "comment_count": 0, "views": 0, "content": "

In this competition our task is to predict the sum of the digits per image. Given the recent Rules Clarification with the belated prohibition of the obvious workflow of the extraction and subsequent classification and a posteriori summation of the individual digits now not being eligible (as it is \"not considered to be innovation\"), it seems that an alternative way forward is to perform the summation 'on-board' the neural network. However, getting a neural network to learn how to add numbers is no mean feat in itself. To that end I have happened across this potentially interesting paper, which introduces the Neural Addition Unit (NAU).

\n

From the abstract:

\n
\n

\"Neural networks can approximate complex functions, but they struggle to perform\nexact arithmetic operations over real numbers. The lack of inductive bias for\narithmetic operations leaves neural networks without the underlying logic necessary\nto extrapolate on tasks such as addition, subtraction, and multiplication. We present\ntwo new neural network components: the Neural Addition Unit (NAU)…\"

\n
\n\n

This paper builds upon the work presented in the paper:

\n\n

where they lay the foundations for the \"MNIST Digit Addition task\" (see § 4.2), and which rather helpfully comes with a basic pytorch implementation of a NAC/NALU, and accessible via

\n\n

with the neural accumulator (NAC) achieving the best results for addition tasks (MAE of 1.42 for a series of 10 digits, see Table 2).

\n

For more details see GitHub repository Neural Arithmetic Logic Units (NALU).

\n

All the best,
\ncarl

", "messages": [ { "id": 1720809, "postDate": "2022-03-13T06:53:01.543Z", "content": "

In this competition our task is to predict the sum of the digits per image. Given the recent Rules Clarification with the belated prohibition of the obvious workflow of the extraction and subsequent classification and a posteriori summation of the individual digits now not being eligible (as it is \"not considered to be innovation\"), it seems that an alternative way forward is to perform the summation 'on-board' the neural network. However, getting a neural network to learn how to add numbers is no mean feat in itself. To that end I have happened across this potentially interesting paper, which introduces the Neural Addition Unit (NAU).

\n

From the abstract:

\n
\n

\"Neural networks can approximate complex functions, but they struggle to perform\nexact arithmetic operations over real numbers. The lack of inductive bias for\narithmetic operations leaves neural networks without the underlying logic necessary\nto extrapolate on tasks such as addition, subtraction, and multiplication. We present\ntwo new neural network components: the Neural Addition Unit (NAU)…\"

\n
\n\n

This paper builds upon the work presented in the paper:

\n\n

where they lay the foundations for the \"MNIST Digit Addition task\" (see § 4.2), and which rather helpfully comes with a basic pytorch implementation of a NAC/NALU, and accessible via

\n\n

with the neural accumulator (NAC) achieving the best results for addition tasks (MAE of 1.42 for a series of 10 digits, see Table 2).

\n

For more details see GitHub repository Neural Arithmetic Logic Units (NALU).

\n

All the best,
\ncarl

", "rawMarkdown": "In this competition our task is to predict the sum of the digits per image. Given the recent [Rules Clarification](https://www.kaggle.com/c/ultra-mnist/discussion/312470) with the belated prohibition of the obvious workflow of the extraction and subsequent classification and *a posteriori* summation of the individual digits now not being eligible (as it is \"*not considered to be innovation*\"), it seems that an alternative way forward is to perform the summation 'on-board' the neural network. However, getting a neural network to learn how to add numbers is no mean feat in itself. To that end I have happened across this potentially interesting [paper](https://arxiv.org/pdf/2001.05016.pdf), which introduces the ***Neural Addition Unit (NAU)***. \n\nFrom the abstract:\n\n> \"***Neural networks can approximate complex functions, but they struggle to perform\nexact arithmetic operations over real numbers. The lack of inductive bias for\narithmetic operations leaves neural networks without the underlying logic necessary\nto extrapolate on tasks such as addition, subtraction, and multiplication. We present\ntwo new neural network components: the Neural Addition Unit (NAU)...***\"\n\n* [Andreas Madsen and Alexander Rosenberg Johansen \"*Neural Arithmetic Units*\", arXiv:2001.05016 (2020)](https://arxiv.org/pdf/2001.05016.pdf)\n* [Neural Arithmetic Units](https://github.com/AndreasMadsen/stable-nalu) GitHub code (they even perform a **MNIST add experiment**)\n\nThis paper builds upon the work presented in the paper: \n\n* [Andrew Trask *et al. \"Neural Arithmetic Logic Units*\", arXiv:1808.00508 (2018)](https://arxiv.org/pdf/1808.00508.pdf)\n\nwhere they lay the foundations for the \"MNIST Digit Addition task\" (see § 4.2), and which rather helpfully comes with a basic pytorch implementation of a NAC/NALU, and accessible via \n\n* `pip install NALU`\n\nwith the neural accumulator (NAC) achieving the best results for addition tasks (MAE of 1.42 for a series of 10 digits, see Table 2). \n\nFor more details see GitHub repository [Neural Arithmetic Logic Units (NALU)](https://github.com/bharathgs/NALU).\n\nAll the best,\ncarl", "votes": 14 } ], "comments": [], "raw_markdown_by_id": { "1720809": "In this competition our task is to predict the sum of the digits per image. Given the recent [Rules Clarification](https://www.kaggle.com/c/ultra-mnist/discussion/312470) with the belated prohibition of the obvious workflow of the extraction and subsequent classification and *a posteriori* summation of the individual digits now not being eligible (as it is \"*not considered to be innovation*\"), it seems that an alternative way forward is to perform the summation 'on-board' the neural network. However, getting a neural network to learn how to add numbers is no mean feat in itself. To that end I have happened across this potentially interesting [paper](https://arxiv.org/pdf/2001.05016.pdf), which introduces the ***Neural Addition Unit (NAU)***. \n\nFrom the abstract:\n\n> \"***Neural networks can approximate complex functions, but they struggle to perform\nexact arithmetic operations over real numbers. The lack of inductive bias for\narithmetic operations leaves neural networks without the underlying logic necessary\nto extrapolate on tasks such as addition, subtraction, and multiplication. We present\ntwo new neural network components: the Neural Addition Unit (NAU)...***\"\n\n* [Andreas Madsen and Alexander Rosenberg Johansen \"*Neural Arithmetic Units*\", arXiv:2001.05016 (2020)](https://arxiv.org/pdf/2001.05016.pdf)\n* [Neural Arithmetic Units](https://github.com/AndreasMadsen/stable-nalu) GitHub code (they even perform a **MNIST add experiment**)\n\nThis paper builds upon the work presented in the paper: \n\n* [Andrew Trask *et al. \"Neural Arithmetic Logic Units*\", arXiv:1808.00508 (2018)](https://arxiv.org/pdf/1808.00508.pdf)\n\nwhere they lay the foundations for the \"MNIST Digit Addition task\" (see § 4.2), and which rather helpfully comes with a basic pytorch implementation of a NAC/NALU, and accessible via \n\n* `pip install NALU`\n\nwith the neural accumulator (NAC) achieving the best results for addition tasks (MAE of 1.42 for a series of 10 digits, see Table 2). \n\nFor more details see GitHub repository [Neural Arithmetic Logic Units (NALU)](https://github.com/bharathgs/NALU).\n\nAll the best,\ncarl" } }