{
  "id": 393062,
  "title": "Torch Preprocessing ≠ Tensorflow Preprocessing",
  "url": "/competitions/asl-signs/discussion/393062",
  "author_name": "",
  "post_date": "2023-03-07T21:12:59.633703500Z",
  "votes": 7,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Be careful, if you are using someone else's preprocessing you may encounter differences in precision that can actually result in some decent drift. </p>\n<p>This is mostly because it appears that torch caps the precision of mean (and maybe std?) to 4 significant figures? While TF is full precision.</p>\n<p>I discovered this because np.allclose kept failing. <br>\nHere's the code I used to test it.</p>\n<pre><code> n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_mean(a))\n    (, torch.mean(torch.from_numpy(a), dtype=torch.float64))\n</code></pre>\n<p><b>Which outputs:</b></p>\n<pre><code> --- RANDOM ARRAY OF SIZE: 1 ---\n    TF    --&gt;  tf.Tensor(0.9095935277196137, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.9096, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    TF    --&gt;  tf.Tensor(0.49933835994177134, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.4993, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    TF    --&gt;  tf.Tensor(0.5080983223254402, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.5081, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    TF    --&gt;  tf.Tensor(0.4930732425105786, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.4931, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    TF    --&gt;  tf.Tensor(0.49728703517617723, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.4973, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    TF    --&gt;  tf.Tensor(0.5003284489028175, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.5003, dtype=torch.float64)\n</code></pre>\n<p>Any comments on this? I think for now I will just round my tf.reduce_sum to 4 decimal places (as I'm using a torch preprocessed dataset), but it's pretty clear I need to use TF in the future.</p>\n<p>Let me know if I mucked something up and I hope this helps!</p>\n<hr>\n<p><strong>I checked <code>torch.std</code> and <code>tf.reduce_std</code> and the same problem occurs. Here's the outputs:</strong></p>\n<pre><code> --- RANDOM ARRAY OF SIZE: 1 ---\n    TF    --&gt;  tf.Tensor(0.0, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    TF    --&gt;  tf.Tensor(0.3390631844757934, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.3574, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    TF    --&gt;  tf.Tensor(0.28735353435525773, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2888, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    TF    --&gt;  tf.Tensor(0.2898271651938271, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2900, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    TF    --&gt;  tf.Tensor(0.28905290508071924, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2891, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    TF    --&gt;  tf.Tensor(0.288857244804416, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2889, dtype=torch.float64)\n</code></pre>",
  "messages": [
    {
      "id": "2172846",
      "postDate": "03/07/2023 21:12:59",
      "content": "<p>Be careful, if you are using someone else's preprocessing you may encounter differences in precision that can actually result in some decent drift. </p>\n<p>This is mostly because it appears that torch caps the precision of mean (and maybe std?) to 4 significant figures? While TF is full precision.</p>\n<p>I discovered this because np.allclose kept failing. <br>\nHere's the code I used to test it.</p>\n<pre><code> n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_mean(a))\n    (, torch.mean(torch.from_numpy(a), dtype=torch.float64))\n</code></pre>\n<p><b>Which outputs:</b></p>\n<pre><code> --- RANDOM ARRAY OF SIZE: 1 ---\n    TF    --&gt;  tf.Tensor(0.9095935277196137, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.9096, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    TF    --&gt;  tf.Tensor(0.49933835994177134, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.4993, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    TF    --&gt;  tf.Tensor(0.5080983223254402, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.5081, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    TF    --&gt;  tf.Tensor(0.4930732425105786, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.4931, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    TF    --&gt;  tf.Tensor(0.49728703517617723, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.4973, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    TF    --&gt;  tf.Tensor(0.5003284489028175, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.5003, dtype=torch.float64)\n</code></pre>\n<p>Any comments on this? I think for now I will just round my tf.reduce_sum to 4 decimal places (as I'm using a torch preprocessed dataset), but it's pretty clear I need to use TF in the future.</p>\n<p>Let me know if I mucked something up and I hope this helps!</p>\n<hr>\n<p><strong>I checked <code>torch.std</code> and <code>tf.reduce_std</code> and the same problem occurs. Here's the outputs:</strong></p>\n<pre><code> --- RANDOM ARRAY OF SIZE: 1 ---\n    TF    --&gt;  tf.Tensor(0.0, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    TF    --&gt;  tf.Tensor(0.3390631844757934, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.3574, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    TF    --&gt;  tf.Tensor(0.28735353435525773, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2888, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    TF    --&gt;  tf.Tensor(0.2898271651938271, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2900, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    TF    --&gt;  tf.Tensor(0.28905290508071924, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2891, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    TF    --&gt;  tf.Tensor(0.288857244804416, shape=(), dtype=float64)\n    TORCH --&gt;  tensor(0.2889, dtype=torch.float64)\n</code></pre>",
      "rawMarkdown": "Be careful, if you are using someone else's preprocessing you may encounter differences in precision that can actually result in some decent drift. \n\nThis is mostly because it appears that torch caps the precision of mean (and maybe std?) to 4 significant figures? While TF is full precision.\n\nI discovered this because np.allclose kept failing. \nHere's the code I used to test it.\n\n```Python\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tTF    --> \", tf.math.reduce_mean(a))\n    print(\"\\tTORCH --> \", torch.mean(torch.from_numpy(a), dtype=torch.float64))\n```\n\n<b>Which outputs:</b>\n\n```\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tTF    -->  tf.Tensor(0.9095935277196137, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.9096, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tTF    -->  tf.Tensor(0.49933835994177134, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.4993, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tTF    -->  tf.Tensor(0.5080983223254402, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.5081, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tTF    -->  tf.Tensor(0.4930732425105786, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.4931, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tTF    -->  tf.Tensor(0.49728703517617723, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.4973, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tTF    -->  tf.Tensor(0.5003284489028175, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.5003, dtype=torch.float64)\n```\n\nAny comments on this? I think for now I will just round my tf.reduce_sum to 4 decimal places (as I'm using a torch preprocessed dataset), but it's pretty clear I need to use TF in the future.\n\nLet me know if I mucked something up and I hope this helps!\n\n\n---\n\n\n**I checked `torch.std` and `tf.reduce_std` and the same problem occurs. Here's the outputs:**\n\n```\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tTF    -->  tf.Tensor(0.0, shape=(), dtype=float64)\n\tTORCH -->  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tTF    -->  tf.Tensor(0.3390631844757934, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.3574, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tTF    -->  tf.Tensor(0.28735353435525773, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2888, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tTF    -->  tf.Tensor(0.2898271651938271, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2900, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tTF    -->  tf.Tensor(0.28905290508071924, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2891, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tTF    -->  tf.Tensor(0.288857244804416, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2889, dtype=torch.float64)\n\n```",
      "votes": null
    },
    {
      "id": "2172865",
      "postDate": "03/07/2023 21:29:44",
      "content": "<p>I fixed it:</p>\n<pre><code> ():\n     tf.(tf.math.reduce_mean(x, axis=axis) * **n_decimal_places) / **n_decimal_places\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_mean(a))\n    (, dumb_tf_mean(a))\n    (, torch.mean(torch.from_numpy(a)))\n\n ():\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=)\n     tf.(tf.experimental.numpy.sqrt(x) * **n_decimal_places) / **n_decimal_places\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_std(a))\n    (, dumb_tf_std(a))\n    (, torch.std(torch.from_numpy(a)))\n</code></pre>\n<p><strong>gives the now same results:</strong></p>\n<pre><code>MEANS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n    NORMAL TF --&gt;  tf.Tensor(0.030653538975419004, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.0307, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.0307, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5083839486843929, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.50838, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5084, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5151898293050522, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.51519, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5152, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5039305329992863, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.5039, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5039, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.4982654741047983, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.4983, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.4983, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5005517251925236, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.5006, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5006, dtype=torch.float64)\n\n\nSTANDARD DEVIATIONS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n    NORMAL TF --&gt;  tf.Tensor(0.0, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(nan, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    NORMAL TF --&gt;  tf.Tensor(0.26266802218192437, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2769, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2769, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    NORMAL TF --&gt;  tf.Tensor(0.2975046424468065, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.299, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2990, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.29141487053521814, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2916, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2916, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.2880588939299123, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2881, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2881, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.28778107280272164, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2878, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2878, dtype=torch.float64)\n</code></pre>",
      "rawMarkdown": "I fixed it:\n\n```Python\ndef dumb_tf_mean(x, n_decimal_places=4, axis=None):\n    return tf.round(tf.math.reduce_mean(x, axis=axis) * 10**n_decimal_places) / 10**n_decimal_places\n\nprint(\"\\nMEANS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_mean(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_mean(a))\n    print(\"\\tTORCH     --> \", torch.mean(torch.from_numpy(a)))\n\ndef dumb_tf_std(x, n_decimal_places=4, axis=None):\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=1)\n    return tf.round(tf.experimental.numpy.sqrt(x) * 10**n_decimal_places) / 10**n_decimal_places\n\nprint(\"\\n\\nSTANDARD DEVIATIONS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_std(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_std(a))\n    print(\"\\tTORCH     --> \", torch.std(torch.from_numpy(a)))\n```\n\n**gives the now same results:**\n\n```\nMEANS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tNORMAL TF -->  tf.Tensor(0.030653538975419004, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.0307, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.0307, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tNORMAL TF -->  tf.Tensor(0.5083839486843929, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.50838, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5084, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tNORMAL TF -->  tf.Tensor(0.5151898293050522, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.51519, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5152, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5039305329992863, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.5039, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5039, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tNORMAL TF -->  tf.Tensor(0.4982654741047983, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.4983, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.4983, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5005517251925236, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.5006, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5006, dtype=torch.float64)\n\n\nSTANDARD DEVIATIONS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tNORMAL TF -->  tf.Tensor(0.0, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(nan, shape=(), dtype=float32)\n\tTORCH     -->  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tNORMAL TF -->  tf.Tensor(0.26266802218192437, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2769, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2769, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tNORMAL TF -->  tf.Tensor(0.2975046424468065, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.299, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2990, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tNORMAL TF -->  tf.Tensor(0.29141487053521814, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2916, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2916, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tNORMAL TF -->  tf.Tensor(0.2880588939299123, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2881, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2881, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tNORMAL TF -->  tf.Tensor(0.28778107280272164, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2878, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2878, dtype=torch.float64)\n\n```",
      "votes": null
    },
    {
      "id": "2172935",
      "postDate": "03/08/2023 00:15:32",
      "content": "<p>You should just use .item() for torch output<br>\n<code>print(\"\\tTORCH     --&gt; \", torch.mean(torch.from_numpy(a)).item())</code><br>\ninstead of round TF</p>\n<pre><code>MEANS\n\n\n --- RANDOM ARRAY OF SIZE:  ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE:  ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE:  ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE: , ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE: , ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE: , ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n</code></pre>",
      "rawMarkdown": "You should just use .item() for torch output\n`print(\"\\tTORCH     --> \", torch.mean(torch.from_numpy(a)).item())`\ninstead of round TF\n\n```python\nMEANS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tNORMAL TF -->  tf.Tensor(0.7714272444681669, shape=(), dtype=float64)\n\tDUMB TF   -->  0.7714272444681669\n\tTORCH     -->  0.7714272444681669\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tNORMAL TF -->  tf.Tensor(0.4988981206635745, shape=(), dtype=float64)\n\tDUMB TF   -->  0.4988981206635744\n\tTORCH     -->  0.4988981206635744\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tNORMAL TF -->  tf.Tensor(0.5118923313890587, shape=(), dtype=float64)\n\tDUMB TF   -->  0.5118923313890587\n\tTORCH     -->  0.5118923313890585\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5034262655306663, shape=(), dtype=float64)\n\tDUMB TF   -->  0.5034262655306663\n\tTORCH     -->  0.5034262655306663\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tNORMAL TF -->  tf.Tensor(0.49799206150197367, shape=(), dtype=float64)\n\tDUMB TF   -->  0.49799206150197367\n\tTORCH     -->  0.49799206150197367\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5008496029683892, shape=(), dtype=float64)\n\tDUMB TF   -->  0.5008496029683892\n\tTORCH     -->  0.5008496029683892\n```",
      "votes": null
    },
    {
      "id": "2172952",
      "postDate": "03/08/2023 00:52:54",
      "content": "<p>That’s awesome. I need to do it in TF in my version hence the round and whatnot. And the version that I’m loading from (one of the public numpy datasets) clearly didn’t use that .item thing on their torch arrays prior to conversion to Numpy.</p>\n<p>Definitely good to note for torch users going forward that they should use .item() to ensure full precision.</p>",
      "rawMarkdown": "That’s awesome. I need to do it in TF in my version hence the round and whatnot. And the version that I’m loading from (one of the public numpy datasets) clearly didn’t use that .item thing on their torch arrays prior to conversion to Numpy.\n\nDefinitely good to note for torch users going forward that they should use .item() to ensure full precision.",
      "votes": null
    },
    {
      "id": "2172954",
      "postDate": "03/08/2023 00:54:08",
      "content": "<p>Does that work for std too? I noticed more numerical differences in that when compared with mean?</p>",
      "rawMarkdown": "Does that work for std too? I noticed more numerical differences in that when compared with mean?",
      "votes": null
    },
    {
      "id": "2172958",
      "postDate": "03/08/2023 01:08:43",
      "content": "<pre><code> ():\n     tf.math.reduce_mean(x, axis=axis).numpy().item()\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_mean(a))\n    (, dumb_tf_mean(a))\n    (, torch.mean(torch.from_numpy(a)).item())\n\n ():\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=)\n     tf.experimental.numpy.sqrt(x).numpy().item()\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_std(a))\n    (, dumb_tf_std(a))\n    (, torch.std(torch.from_numpy(a)).item())\n</code></pre>\n<p>For tf code you could use .numpy().item() and here DUMB_TF would be the same with torch as fp32</p>\n<p>For the message above: it not necessary to use .item() in some operations. Torch tensor already has this data, but torch prints has only 4 decimals (idk why). You also will lost gradient if you will use item() </p>",
      "rawMarkdown": "```python\ndef dumb_tf_mean(x, n_decimal_places=4, axis=None):\n    return tf.math.reduce_mean(x, axis=axis).numpy().item()\n\nprint(\"\\nMEANS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_mean(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_mean(a))\n    print(\"\\tTORCH     --> \", torch.mean(torch.from_numpy(a)).item())\n\ndef dumb_tf_std(x, n_decimal_places=4, axis=None):\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=1)\n    return tf.experimental.numpy.sqrt(x).numpy().item()\n\nprint(\"\\n\\nSTANDARD DEVIATIONS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_std(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_std(a))\n    print(\"\\tTORCH     --> \", torch.std(torch.from_numpy(a)).item())\n```\n\nFor tf code you could use .numpy().item() and here DUMB_TF would be the same with torch as fp32\n\nFor the message above: it not necessary to use .item() in some operations. Torch tensor already has this data, but torch prints has only 4 decimals (idk why). You also will lost gradient if you will use item()",
      "votes": null
    },
    {
      "id": "2173008",
      "postDate": "03/08/2023 02:22:19",
      "content": "<p>That makes a lot of sense.</p>\n<p>One note, I’m trying to write a TF preprocessing function that can be included in a tflite model that matches the preprocessing that was done via torch and saved as numpy for training… </p>\n<p>It’s mental. Just writing it all indicates the obvious solution: <strong>keep it all consistent.</strong></p>\n<p>Thanks for teaching me about the above! It was really helpful!</p>",
      "rawMarkdown": "That makes a lot of sense.\n\nOne note, I’m trying to write a TF preprocessing function that can be included in a tflite model that matches the preprocessing that was done via torch and saved as numpy for training… \n\nIt’s mental. Just writing it all indicates the obvious solution: **keep it all consistent.**\n\nThanks for teaching me about the above! It was really helpful!",
      "votes": null
    },
    {
      "id": "2765354",
      "postDate": "04/21/2024 05:34:08",
      "content": "<p>That's Awesome !!</p>",
      "rawMarkdown": "That's Awesome !!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2172865,
      "author_name": "dschettler8845",
      "author_url": "",
      "post_date": "03/07/2023 21:29:44",
      "content": "<p>I fixed it:</p>\n<pre><code> ():\n     tf.(tf.math.reduce_mean(x, axis=axis) * **n_decimal_places) / **n_decimal_places\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_mean(a))\n    (, dumb_tf_mean(a))\n    (, torch.mean(torch.from_numpy(a)))\n\n ():\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=)\n     tf.(tf.experimental.numpy.sqrt(x) * **n_decimal_places) / **n_decimal_places\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_std(a))\n    (, dumb_tf_std(a))\n    (, torch.std(torch.from_numpy(a)))\n</code></pre>\n<p><strong>gives the now same results:</strong></p>\n<pre><code>MEANS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n    NORMAL TF --&gt;  tf.Tensor(0.030653538975419004, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.0307, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.0307, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5083839486843929, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.50838, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5084, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5151898293050522, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.51519, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5152, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5039305329992863, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.5039, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5039, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.4982654741047983, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.4983, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.4983, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.5005517251925236, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.5006, shape=(), dtype=float64)\n    TORCH     --&gt;  tensor(0.5006, dtype=torch.float64)\n\n\nSTANDARD DEVIATIONS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n    NORMAL TF --&gt;  tf.Tensor(0.0, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(nan, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n    NORMAL TF --&gt;  tf.Tensor(0.26266802218192437, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2769, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2769, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n    NORMAL TF --&gt;  tf.Tensor(0.2975046424468065, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.299, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2990, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.29141487053521814, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2916, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2916, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.2880588939299123, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2881, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2881, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n    NORMAL TF --&gt;  tf.Tensor(0.28778107280272164, shape=(), dtype=float64)\n    DUMB TF   --&gt;  tf.Tensor(0.2878, shape=(), dtype=float32)\n    TORCH     --&gt;  tensor(0.2878, dtype=torch.float64)\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 2172935,
          "author_name": "kolyaforrat",
          "author_url": "",
          "post_date": "03/08/2023 00:15:32",
          "content": "<p>You should just use .item() for torch output<br>\n<code>print(\"\\tTORCH     --&gt; \", torch.mean(torch.from_numpy(a)).item())</code><br>\ninstead of round TF</p>\n<pre><code>MEANS\n\n\n --- RANDOM ARRAY OF SIZE:  ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE:  ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE:  ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE: , ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE: , ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n\n --- RANDOM ARRAY OF SIZE: , ---\n    NORMAL TF --&gt;  tf.Tensor(, shape=(), dtype=float64)\n    DUMB TF   --&gt;  \n    TORCH     --&gt;  \n</code></pre>",
          "votes": null,
          "replies": [
            {
              "id": 2172952,
              "author_name": "dschettler8845",
              "author_url": "",
              "post_date": "03/08/2023 00:52:54",
              "content": "<p>That’s awesome. I need to do it in TF in my version hence the round and whatnot. And the version that I’m loading from (one of the public numpy datasets) clearly didn’t use that .item thing on their torch arrays prior to conversion to Numpy.</p>\n<p>Definitely good to note for torch users going forward that they should use .item() to ensure full precision.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2172954,
                  "author_name": "dschettler8845",
                  "author_url": "",
                  "post_date": "03/08/2023 00:54:08",
                  "content": "<p>Does that work for std too? I noticed more numerical differences in that when compared with mean?</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2172958,
                      "author_name": "kolyaforrat",
                      "author_url": "",
                      "post_date": "03/08/2023 01:08:43",
                      "content": "<pre><code> ():\n     tf.math.reduce_mean(x, axis=axis).numpy().item()\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_mean(a))\n    (, dumb_tf_mean(a))\n    (, torch.mean(torch.from_numpy(a)).item())\n\n ():\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=)\n     tf.experimental.numpy.sqrt(x).numpy().item()\n\n()\n n  ():\n    ()\n    a = np.random.random(**n)\n    (, tf.math.reduce_std(a))\n    (, dumb_tf_std(a))\n    (, torch.std(torch.from_numpy(a)).item())\n</code></pre>\n<p>For tf code you could use .numpy().item() and here DUMB_TF would be the same with torch as fp32</p>\n<p>For the message above: it not necessary to use .item() in some operations. Torch tensor already has this data, but torch prints has only 4 decimals (idk why). You also will lost gradient if you will use item() </p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 2173008,
                          "author_name": "dschettler8845",
                          "author_url": "",
                          "post_date": "03/08/2023 02:22:19",
                          "content": "<p>That makes a lot of sense.</p>\n<p>One note, I’m trying to write a TF preprocessing function that can be included in a tflite model that matches the preprocessing that was done via torch and saved as numpy for training… </p>\n<p>It’s mental. Just writing it all indicates the obvious solution: <strong>keep it all consistent.</strong></p>\n<p>Thanks for teaching me about the above! It was really helpful!</p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2765354,
      "author_name": "sonialikhan",
      "author_url": "",
      "post_date": "04/21/2024 05:34:08",
      "content": "<p>That's Awesome !!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2172846": "Be careful, if you are using someone else's preprocessing you may encounter differences in precision that can actually result in some decent drift. \n\nThis is mostly because it appears that torch caps the precision of mean (and maybe std?) to 4 significant figures? While TF is full precision.\n\nI discovered this because np.allclose kept failing. \nHere's the code I used to test it.\n\n```Python\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tTF    --> \", tf.math.reduce_mean(a))\n    print(\"\\tTORCH --> \", torch.mean(torch.from_numpy(a), dtype=torch.float64))\n```\n\n<b>Which outputs:</b>\n\n```\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tTF    -->  tf.Tensor(0.9095935277196137, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.9096, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tTF    -->  tf.Tensor(0.49933835994177134, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.4993, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tTF    -->  tf.Tensor(0.5080983223254402, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.5081, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tTF    -->  tf.Tensor(0.4930732425105786, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.4931, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tTF    -->  tf.Tensor(0.49728703517617723, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.4973, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tTF    -->  tf.Tensor(0.5003284489028175, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.5003, dtype=torch.float64)\n```\n\nAny comments on this? I think for now I will just round my tf.reduce_sum to 4 decimal places (as I'm using a torch preprocessed dataset), but it's pretty clear I need to use TF in the future.\n\nLet me know if I mucked something up and I hope this helps!\n\n\n---\n\n\n**I checked `torch.std` and `tf.reduce_std` and the same problem occurs. Here's the outputs:**\n\n```\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tTF    -->  tf.Tensor(0.0, shape=(), dtype=float64)\n\tTORCH -->  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tTF    -->  tf.Tensor(0.3390631844757934, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.3574, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tTF    -->  tf.Tensor(0.28735353435525773, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2888, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tTF    -->  tf.Tensor(0.2898271651938271, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2900, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tTF    -->  tf.Tensor(0.28905290508071924, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2891, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tTF    -->  tf.Tensor(0.288857244804416, shape=(), dtype=float64)\n\tTORCH -->  tensor(0.2889, dtype=torch.float64)\n\n```",
    "2172865": "I fixed it:\n\n```Python\ndef dumb_tf_mean(x, n_decimal_places=4, axis=None):\n    return tf.round(tf.math.reduce_mean(x, axis=axis) * 10**n_decimal_places) / 10**n_decimal_places\n\nprint(\"\\nMEANS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_mean(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_mean(a))\n    print(\"\\tTORCH     --> \", torch.mean(torch.from_numpy(a)))\n\ndef dumb_tf_std(x, n_decimal_places=4, axis=None):\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=1)\n    return tf.round(tf.experimental.numpy.sqrt(x) * 10**n_decimal_places) / 10**n_decimal_places\n\nprint(\"\\n\\nSTANDARD DEVIATIONS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_std(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_std(a))\n    print(\"\\tTORCH     --> \", torch.std(torch.from_numpy(a)))\n```\n\n**gives the now same results:**\n\n```\nMEANS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tNORMAL TF -->  tf.Tensor(0.030653538975419004, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.0307, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.0307, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tNORMAL TF -->  tf.Tensor(0.5083839486843929, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.50838, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5084, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tNORMAL TF -->  tf.Tensor(0.5151898293050522, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.51519, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5152, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5039305329992863, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.5039, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5039, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tNORMAL TF -->  tf.Tensor(0.4982654741047983, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.4983, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.4983, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5005517251925236, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.5006, shape=(), dtype=float64)\n\tTORCH     -->  tensor(0.5006, dtype=torch.float64)\n\n\nSTANDARD DEVIATIONS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tNORMAL TF -->  tf.Tensor(0.0, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(nan, shape=(), dtype=float32)\n\tTORCH     -->  tensor(nan, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tNORMAL TF -->  tf.Tensor(0.26266802218192437, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2769, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2769, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tNORMAL TF -->  tf.Tensor(0.2975046424468065, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.299, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2990, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tNORMAL TF -->  tf.Tensor(0.29141487053521814, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2916, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2916, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tNORMAL TF -->  tf.Tensor(0.2880588939299123, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2881, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2881, dtype=torch.float64)\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tNORMAL TF -->  tf.Tensor(0.28778107280272164, shape=(), dtype=float64)\n\tDUMB TF   -->  tf.Tensor(0.2878, shape=(), dtype=float32)\n\tTORCH     -->  tensor(0.2878, dtype=torch.float64)\n\n```",
    "2172935": "You should just use .item() for torch output\n`print(\"\\tTORCH     --> \", torch.mean(torch.from_numpy(a)).item())`\ninstead of round TF\n\n```python\nMEANS\n\n\n --- RANDOM ARRAY OF SIZE: 1 ---\n\tNORMAL TF -->  tf.Tensor(0.7714272444681669, shape=(), dtype=float64)\n\tDUMB TF   -->  0.7714272444681669\n\tTORCH     -->  0.7714272444681669\n\n --- RANDOM ARRAY OF SIZE: 10 ---\n\tNORMAL TF -->  tf.Tensor(0.4988981206635745, shape=(), dtype=float64)\n\tDUMB TF   -->  0.4988981206635744\n\tTORCH     -->  0.4988981206635744\n\n --- RANDOM ARRAY OF SIZE: 100 ---\n\tNORMAL TF -->  tf.Tensor(0.5118923313890587, shape=(), dtype=float64)\n\tDUMB TF   -->  0.5118923313890587\n\tTORCH     -->  0.5118923313890585\n\n --- RANDOM ARRAY OF SIZE: 1,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5034262655306663, shape=(), dtype=float64)\n\tDUMB TF   -->  0.5034262655306663\n\tTORCH     -->  0.5034262655306663\n\n --- RANDOM ARRAY OF SIZE: 10,000 ---\n\tNORMAL TF -->  tf.Tensor(0.49799206150197367, shape=(), dtype=float64)\n\tDUMB TF   -->  0.49799206150197367\n\tTORCH     -->  0.49799206150197367\n\n --- RANDOM ARRAY OF SIZE: 100,000 ---\n\tNORMAL TF -->  tf.Tensor(0.5008496029683892, shape=(), dtype=float64)\n\tDUMB TF   -->  0.5008496029683892\n\tTORCH     -->  0.5008496029683892\n```",
    "2172952": "That’s awesome. I need to do it in TF in my version hence the round and whatnot. And the version that I’m loading from (one of the public numpy datasets) clearly didn’t use that .item thing on their torch arrays prior to conversion to Numpy.\n\nDefinitely good to note for torch users going forward that they should use .item() to ensure full precision.",
    "2172954": "Does that work for std too? I noticed more numerical differences in that when compared with mean?",
    "2172958": "```python\ndef dumb_tf_mean(x, n_decimal_places=4, axis=None):\n    return tf.math.reduce_mean(x, axis=axis).numpy().item()\n\nprint(\"\\nMEANS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_mean(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_mean(a))\n    print(\"\\tTORCH     --> \", torch.mean(torch.from_numpy(a)).item())\n\ndef dumb_tf_std(x, n_decimal_places=4, axis=None):\n    x = tf.experimental.numpy.var(x, axis=axis, dtype=tf.float32, ddof=1)\n    return tf.experimental.numpy.sqrt(x).numpy().item()\n\nprint(\"\\n\\nSTANDARD DEVIATIONS\\n\")\nfor n in range(6):\n    print(f\"\\n --- RANDOM ARRAY OF SIZE: {10**n:,} ---\")\n    a = np.random.random(10**n)\n    print(\"\\tNORMAL TF --> \", tf.math.reduce_std(a))\n    print(\"\\tDUMB TF   --> \", dumb_tf_std(a))\n    print(\"\\tTORCH     --> \", torch.std(torch.from_numpy(a)).item())\n```\n\nFor tf code you could use .numpy().item() and here DUMB_TF would be the same with torch as fp32\n\nFor the message above: it not necessary to use .item() in some operations. Torch tensor already has this data, but torch prints has only 4 decimals (idk why). You also will lost gradient if you will use item()",
    "2173008": "That makes a lot of sense.\n\nOne note, I’m trying to write a TF preprocessing function that can be included in a tflite model that matches the preprocessing that was done via torch and saved as numpy for training… \n\nIt’s mental. Just writing it all indicates the obvious solution: **keep it all consistent.**\n\nThanks for teaching me about the above! It was really helpful!",
    "2765354": "That's Awesome !!"
  },
  "source": "meta"
}