{
  "id": 451798,
  "title": "Is this a supposed leak? Some models in the test data seem to be able to identify",
  "url": "/competitions/predict-ai-model-runtime/discussion/451798",
  "author_name": "",
  "post_date": "2023-10-30T14:33:42.802438800Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I checked the number of nodes and configurable nodes in test data. <br>\nWhen I compared these with the training data, I noticed that there were some cases that matched perfectly.<br>\nFor example, two efficientnet_b7 seem to exist in the test data.</p>\n<p>train: </p>\n<pre><code>num_node:, num_config_node:, file:xla/default/train/efficientnet_b7_eval_batch_1`\n</code></pre>\n<p>test:</p>\n<pre><code>num_node:, num_config_node:, file:xla/default/test/05ae41e26dd3c4c06390371a0423233c\nnum_node:, num_config_node:, file:xla/default/test/3e7156ac468dfb75cf5c9615e1e5887d\nnum_node:, num_config_node:, file:xla/default/test/5335ed13823b0a518ee3c79ba4425f34\nnum_node:, num_config_node:, file:xla/default/test/937ee0eb0d5d6151b7b8252933b5c1c9\nnum_node:, num_config_node:, file:xla/default/test/cd708819d3f5103afd6460b15e74eaf3\nnum_node:, num_config_node:, file:xla/default/test/db59a991b7c607634f13570d52ce885f\nnum_node:, num_config_node:, file:xla/default/test/e8a3a1401b5e79f66d7037e424f3b6df\nnum_node:, num_config_node:, file:xla/default/test/fbaa8bb6a1aed9988281085c91065c05\n</code></pre>",
  "messages": [
    {
      "id": "2505329",
      "postDate": "10/30/2023 14:33:42",
      "content": "<p>I checked the number of nodes and configurable nodes in test data. <br>\nWhen I compared these with the training data, I noticed that there were some cases that matched perfectly.<br>\nFor example, two efficientnet_b7 seem to exist in the test data.</p>\n<p>train: </p>\n<pre><code>num_node:, num_config_node:, file:xla/default/train/efficientnet_b7_eval_batch_1`\n</code></pre>\n<p>test:</p>\n<pre><code>num_node:, num_config_node:, file:xla/default/test/05ae41e26dd3c4c06390371a0423233c\nnum_node:, num_config_node:, file:xla/default/test/3e7156ac468dfb75cf5c9615e1e5887d\nnum_node:, num_config_node:, file:xla/default/test/5335ed13823b0a518ee3c79ba4425f34\nnum_node:, num_config_node:, file:xla/default/test/937ee0eb0d5d6151b7b8252933b5c1c9\nnum_node:, num_config_node:, file:xla/default/test/cd708819d3f5103afd6460b15e74eaf3\nnum_node:, num_config_node:, file:xla/default/test/db59a991b7c607634f13570d52ce885f\nnum_node:, num_config_node:, file:xla/default/test/e8a3a1401b5e79f66d7037e424f3b6df\nnum_node:, num_config_node:, file:xla/default/test/fbaa8bb6a1aed9988281085c91065c05\n</code></pre>",
      "rawMarkdown": "I checked the number of nodes and configurable nodes in test data. \nWhen I compared these with the training data, I noticed that there were some cases that matched perfectly.\nFor example, two efficientnet_b7 seem to exist in the test data.\n\ntrain: \n```python\nnum_node:43615, num_config_node:1158, file:xla/default/train/efficientnet_b7_eval_batch_1`\n```\n\ntest:\n```python\nnum_node:43615, num_config_node:1158, file:xla/default/test/05ae41e26dd3c4c06390371a0423233c\nnum_node:41522, num_config_node:2244, file:xla/default/test/3e7156ac468dfb75cf5c9615e1e5887d\nnum_node:43615, num_config_node:1158, file:xla/default/test/5335ed13823b0a518ee3c79ba4425f34\nnum_node:5279, num_config_node:161, file:xla/default/test/937ee0eb0d5d6151b7b8252933b5c1c9\nnum_node:490, num_config_node:38, file:xla/default/test/cd708819d3f5103afd6460b15e74eaf3\nnum_node:5810, num_config_node:166, file:xla/default/test/db59a991b7c607634f13570d52ce885f\nnum_node:23363, num_config_node:2301, file:xla/default/test/e8a3a1401b5e79f66d7037e424f3b6df\nnum_node:24790, num_config_node:2002, file:xla/default/test/fbaa8bb6a1aed9988281085c91065c05\n```",
      "votes": null
    },
    {
      "id": "2505890",
      "postDate": "10/31/2023 00:04:07",
      "content": "<p>I wrote a script the other week to marry the edge_index and node_opcode shapes; its possible to find training \"equivalents\" for each shape in the layout:nlp sets. However, I did not find it useful to use as a means to shrink the training set and increase the number of configs and/or nodes.</p>\n<p>For example, with layout:nlp:random… (not deduplicated)</p>\n<pre><code>VALID\n\n\\layout\\nlp\\random\\valid\\albert_en_xlarge_batch_size_16_test.npz -&gt; \\layout\\nlp\\random\\train\\albert_en_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\bert_en_cased_L-12_H-768_A-12_batch_size_16_test.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\bert_multi_cased_L-12_H-768_A-12_batch_size_16_train.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_32_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-512_A-8_batch_size_64_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_16_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_32_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-12_H-768_A-12_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n(, )(,) \\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-512_A-8_batch_size_64_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_16_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\talking-heads_large_batch_size_16_train.npz -&gt; \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\n\nTEST\n\n\n\\layout\\nlp\\random\\test\\016ac66a44a906a695afd2228509046a.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\171b0513d8874a427ccfa46d136fbadc.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\23559853d9702baaaacbb0c83fd32266.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\29886a50d55cfe77a9497bc906c76ce9.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\32531d07a084b319dce484f53a4cf3fc.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\38524e2ff135ded55b5286407e7af6b7.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\3a0c5517a87df8d82fd637b83298a3ba.npz -&gt; \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\492c7a94d559aa4a88769142d2a68362.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\test\\58cc2e418c3a8a19b871e15964b534ad.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\60880ed76de53f4d7a1b960b24f20f7d.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\6c1101f6231f4d1722c3b9f6d1e25026.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\1001e119f65b66570d170b94a8.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\71b79ca6db513e7979c3702c595150c2.npz -&gt; \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\7f6284ebe027b1e9a3850fc703858a59.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\b2fdde3b72980907578648774101543e.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\d15316c12eefdef1ba549eb433797f77.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\f6c146fc5cf10be4f3accbaca9897311.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_16_train.npz\n</code></pre>",
      "rawMarkdown": "I wrote a script the other week to marry the edge_index and node_opcode shapes; its possible to find training \"equivalents\" for each shape in the layout:nlp sets. However, I did not find it useful to use as a means to shrink the training set and increase the number of configs and/or nodes.\n\nFor example, with layout:nlp:random... (not deduplicated)\n\n```python\nVALID\n\n\\layout\\nlp\\random\\valid\\albert_en_xlarge_batch_size_16_test.npz -> \\layout\\nlp\\random\\train\\albert_en_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\bert_en_cased_L-12_H-768_A-12_batch_size_16_test.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\bert_multi_cased_L-12_H-768_A-12_batch_size_16_train.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_32_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-512_A-8_batch_size_64_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_16_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_32_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-12_H-768_A-12_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n(9657, 2)(5875,) \\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-512_A-8_batch_size_64_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_16_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\talking-heads_large_batch_size_16_train.npz -> \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\n\nTEST\n\n\n\\layout\\nlp\\random\\test\\016ac66a44a906a695afd2228509046a.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\171b0513d8874a427ccfa46d136fbadc.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\23559853d9702baaaacbb0c83fd32266.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\29886a50d55cfe77a9497bc906c76ce9.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\32531d07a084b319dce484f53a4cf3fc.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\38524e2ff135ded55b5286407e7af6b7.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\3a0c5517a87df8d82fd637b83298a3ba.npz -> \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\492c7a94d559aa4a88769142d2a68362.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\test\\58cc2e418c3a8a19b871e15964b534ad.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\60880ed76de53f4d7a1b960b24f20f7d.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\6c1101f6231f4d1722c3b9f6d1e25026.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\7105451001e119f65b66570d170b94a8.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\71b79ca6db513e7979c3702c595150c2.npz -> \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\7f6284ebe027b1e9a3850fc703858a59.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\b2fdde3b72980907578648774101543e.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\d15316c12eefdef1ba549eb433797f77.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\f6c146fc5cf10be4f3accbaca9897311.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_16_train.npz\n```",
      "votes": null
    },
    {
      "id": "2505909",
      "postDate": "10/31/2023 00:50:38",
      "content": "<p>I think that that one graph in the test set is similar to efficientnet_b7_eval_batch_1, but it is not exactly the same.</p>",
      "rawMarkdown": "I think that that one graph in the test set is similar to efficientnet_b7_eval_batch_1, but it is not exactly the same.",
      "votes": null
    },
    {
      "id": "2506169",
      "postDate": "10/31/2023 06:23:03",
      "content": "<p>Thank you for sharing.<br>\nIn my case, the CV and LB in the XLA layout data do not match very well, so I think I can at least use this information to create a better CV.</p>",
      "rawMarkdown": "Thank you for sharing.\nIn my case, the CV and LB in the XLA layout data do not match very well, so I think I can at least use this information to create a better CV.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2505890,
      "author_name": "rob1080ti",
      "author_url": "",
      "post_date": "10/31/2023 00:04:07",
      "content": "<p>I wrote a script the other week to marry the edge_index and node_opcode shapes; its possible to find training \"equivalents\" for each shape in the layout:nlp sets. However, I did not find it useful to use as a means to shrink the training set and increase the number of configs and/or nodes.</p>\n<p>For example, with layout:nlp:random… (not deduplicated)</p>\n<pre><code>VALID\n\n\\layout\\nlp\\random\\valid\\albert_en_xlarge_batch_size_16_test.npz -&gt; \\layout\\nlp\\random\\train\\albert_en_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\bert_en_cased_L-12_H-768_A-12_batch_size_16_test.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\bert_multi_cased_L-12_H-768_A-12_batch_size_16_train.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_32_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-512_A-8_batch_size_64_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_16_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_32_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-12_H-768_A-12_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n(, )(,) \\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_64_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-512_A-8_batch_size_64_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_16_test.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_32_train.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\talking-heads_large_batch_size_16_train.npz -&gt; \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\n\nTEST\n\n\n\\layout\\nlp\\random\\test\\016ac66a44a906a695afd2228509046a.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\171b0513d8874a427ccfa46d136fbadc.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\23559853d9702baaaacbb0c83fd32266.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\29886a50d55cfe77a9497bc906c76ce9.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\32531d07a084b319dce484f53a4cf3fc.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\38524e2ff135ded55b5286407e7af6b7.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\3a0c5517a87df8d82fd637b83298a3ba.npz -&gt; \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\492c7a94d559aa4a88769142d2a68362.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\test\\58cc2e418c3a8a19b871e15964b534ad.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\60880ed76de53f4d7a1b960b24f20f7d.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\6c1101f6231f4d1722c3b9f6d1e25026.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\1001e119f65b66570d170b94a8.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\71b79ca6db513e7979c3702c595150c2.npz -&gt; \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\7f6284ebe027b1e9a3850fc703858a59.npz -&gt; \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\b2fdde3b72980907578648774101543e.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\d15316c12eefdef1ba549eb433797f77.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\f6c146fc5cf10be4f3accbaca9897311.npz -&gt; \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_16_train.npz\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 2506169,
          "author_name": "ebinan92",
          "author_url": "",
          "post_date": "10/31/2023 06:23:03",
          "content": "<p>Thank you for sharing.<br>\nIn my case, the CV and LB in the XLA layout data do not match very well, so I think I can at least use this information to create a better CV.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2505909,
      "author_name": "mangpophothilimthana",
      "author_url": "",
      "post_date": "10/31/2023 00:50:38",
      "content": "<p>I think that that one graph in the test set is similar to efficientnet_b7_eval_batch_1, but it is not exactly the same.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2505329": "I checked the number of nodes and configurable nodes in test data. \nWhen I compared these with the training data, I noticed that there were some cases that matched perfectly.\nFor example, two efficientnet_b7 seem to exist in the test data.\n\ntrain: \n```python\nnum_node:43615, num_config_node:1158, file:xla/default/train/efficientnet_b7_eval_batch_1`\n```\n\ntest:\n```python\nnum_node:43615, num_config_node:1158, file:xla/default/test/05ae41e26dd3c4c06390371a0423233c\nnum_node:41522, num_config_node:2244, file:xla/default/test/3e7156ac468dfb75cf5c9615e1e5887d\nnum_node:43615, num_config_node:1158, file:xla/default/test/5335ed13823b0a518ee3c79ba4425f34\nnum_node:5279, num_config_node:161, file:xla/default/test/937ee0eb0d5d6151b7b8252933b5c1c9\nnum_node:490, num_config_node:38, file:xla/default/test/cd708819d3f5103afd6460b15e74eaf3\nnum_node:5810, num_config_node:166, file:xla/default/test/db59a991b7c607634f13570d52ce885f\nnum_node:23363, num_config_node:2301, file:xla/default/test/e8a3a1401b5e79f66d7037e424f3b6df\nnum_node:24790, num_config_node:2002, file:xla/default/test/fbaa8bb6a1aed9988281085c91065c05\n```",
    "2505890": "I wrote a script the other week to marry the edge_index and node_opcode shapes; its possible to find training \"equivalents\" for each shape in the layout:nlp sets. However, I did not find it useful to use as a means to shrink the training set and increase the number of configs and/or nodes.\n\nFor example, with layout:nlp:random... (not deduplicated)\n\n```python\nVALID\n\n\\layout\\nlp\\random\\valid\\albert_en_xlarge_batch_size_16_test.npz -> \\layout\\nlp\\random\\train\\albert_en_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\bert_en_cased_L-12_H-768_A-12_batch_size_16_test.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\bert_multi_cased_L-12_H-768_A-12_batch_size_16_train.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_32_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-512_A-8_batch_size_64_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_16_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-10_H-768_A-12_batch_size_32_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-10_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-12_H-768_A-12_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n(9657, 2)(5875,) \\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_16_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_64_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-512_A-8_batch_size_64_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_16_test.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\valid\\small_bert_bert_en_uncased_L-6_H-768_A-12_batch_size_32_train.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\valid\\talking-heads_large_batch_size_16_train.npz -> \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\n\nTEST\n\n\n\\layout\\nlp\\random\\test\\016ac66a44a906a695afd2228509046a.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\171b0513d8874a427ccfa46d136fbadc.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\23559853d9702baaaacbb0c83fd32266.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\29886a50d55cfe77a9497bc906c76ce9.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\32531d07a084b319dce484f53a4cf3fc.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-4_H-512_A-8_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\38524e2ff135ded55b5286407e7af6b7.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\3a0c5517a87df8d82fd637b83298a3ba.npz -> \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\492c7a94d559aa4a88769142d2a68362.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-256_A-4_batch_size_32_train.npz\n\\layout\\nlp\\random\\test\\58cc2e418c3a8a19b871e15964b534ad.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\60880ed76de53f4d7a1b960b24f20f7d.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\6c1101f6231f4d1722c3b9f6d1e25026.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-128_A-2_batch_size_16_train.npz\n\\layout\\nlp\\random\\test\\7105451001e119f65b66570d170b94a8.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-2_H-256_A-4_batch_size_64_train.npz\n\\layout\\nlp\\random\\test\\71b79ca6db513e7979c3702c595150c2.npz -> \\layout\\nlp\\random\\train\\talking-heads_large_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\7f6284ebe027b1e9a3850fc703858a59.npz -> \\layout\\nlp\\random\\train\\bert_en_cased_L-12_H-768_A-12_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\b2fdde3b72980907578648774101543e.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_32_test.npz\n\\layout\\nlp\\random\\test\\d15316c12eefdef1ba549eb433797f77.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-8_H-128_A-2_batch_size_16_test.npz\n\\layout\\nlp\\random\\test\\f6c146fc5cf10be4f3accbaca9897311.npz -> \\layout\\nlp\\random\\train\\small_bert_bert_en_uncased_L-6_H-128_A-2_batch_size_16_train.npz\n```",
    "2505909": "I think that that one graph in the test set is similar to efficientnet_b7_eval_batch_1, but it is not exactly the same.",
    "2506169": "Thank you for sharing.\nIn my case, the CV and LB in the XLA layout data do not match very well, so I think I can at least use this information to create a better CV."
  },
  "source": "meta"
}