What I am doing is rather simple: when I load data I normalize each of the 6 snippets separately. And I don't use any file metadata ofc.
\nThis is good enough to remove known leaks, and I hope there are none left…
\nLoading function:
\ndef load_image(filename, data_path=train_path):\n data = np.load(data_path / filename[0] / (filename + '.npy')).astype(np.float32)\n for i in range(data.shape[0]):\n data[i] -= data[i].mean()\n data[i] /= data[i].std()\n return data\nEdit. The above was for the old dataset. The new dataset is properly normalized.
", "messages": [ { "id": 1361282, "postDate": "2021-06-22T17:33:33.550Z", "content": "What I am doing is rather simple: when I load data I normalize each of the 6 snippets separately. And I don't use any file metadata ofc.
\nThis is good enough to remove known leaks, and I hope there are none left…
\nLoading function:
\ndef load_image(filename, data_path=train_path):\n data = np.load(data_path / filename[0] / (filename + '.npy')).astype(np.float32)\n for i in range(data.shape[0]):\n data[i] -= data[i].mean()\n data[i] /= data[i].std()\n return data\nEdit. The above was for the old dataset. The new dataset is properly normalized.
", "rawMarkdown": "What I am doing is rather simple: when I load data I normalize each of the 6 snippets separately. And I don't use any file metadata ofc.\n\nThis is good enough to remove known leaks, and I hope there are none left...\n\nLoading function:\n\n```\ndef load_image(filename, data_path=train_path):\n data = np.load(data_path / filename[0] / (filename + '.npy')).astype(np.float32)\n for i in range(data.shape[0]):\n data[i] -= data[i].mean()\n data[i] /= data[i].std()\n return data\n```\n\nEdit. The above was for the old dataset. The new dataset is properly normalized.", "votes": 58 }, { "id": 1362366, "postDate": "2021-06-23T11:57:27.500Z", "content": "Thanks for sharing, utilizing this in my notebook :)
", "rawMarkdown": "Thanks for sharing, utilizing this in my notebook :)", "votes": 1 }, { "id": 1361337, "postDate": "2021-06-22T18:18:15.590Z", "content": "@cpmpml Can you tell a bit about your current LB score, is it based on a single model? what image size have you used?
", "rawMarkdown": "@cpmpml Can you tell a bit about your current LB score, is it based on a single model? what image size have you used?", "votes": 1, "replies": [ { "id": 1361847, "postDate": "2021-06-23T05:08:58.617Z", "content": "It is a blend. One of my best non leak model has LB 0.983, CV 0.987.
", "rawMarkdown": "It is a blend. One of my best non leak model has LB 0.983, CV 0.987.", "votes": 1 } ] }, { "id": 1363036, "postDate": "2021-06-23T21:28:19.257Z", "content": "What about using the axis param instead of the for loop ?
\ndata = (data - data.mean( (1, 2), keepdims=True) ) / data.std( (1, 2), keepdims=True)
Edits
\nInclude second axis into the mean as data is 3D. Thanks Nikita.
@kneroma makes sense, but I think this would yield a different result because you set axis to 1. If you import data in the same way, you should rather use axes (1, 2) to get channel-specific stats:
\ndata = (data - np.mean(data, axis = (1, 2), keepdims = True)) / np.std(data, axis = (1, 2), keepdims = True)\n@kneroma because what you do is not the standardization host said they performed.
\n@kozodoi this is neat indeed. Not sure it changes running time though. Have you measured it?
", "rawMarkdown": "@kneroma because what you do is not the standardization host said they performed.\n\n@kozodoi this is neat indeed. Not sure it changes running time though. Have you measured it?" }, { "id": 1363690, "postDate": "2021-06-24T09:17:43.337Z", "content": "@cpmpml I did a quick check. There might be a difference, but it is too small to significantly affect the overall running time (~4 seconds over one full epoch). Also not sure how stable it is given the magnitude.
", "rawMarkdown": "@cpmpml I did a quick check. There might be a difference, but it is too small to significantly affect the overall running time (~4 seconds over one full epoch). Also not sure how stable it is given the magnitude.\n", "votes": 1 }, { "id": 1363830, "postDate": "2021-06-24T11:42:34.370Z", "content": "Nikita, thanks for checking.
", "rawMarkdown": "Nikita, thanks for checking." }, { "id": 1363864, "postDate": "2021-06-24T12:15:58.480Z", "content": "Curiously enough, I also get ~3sec / epoch (82sec vs 85sec). I'll take what I can get.
", "rawMarkdown": "Curiously enough, I also get ~3sec / epoch (82sec vs 85sec). I'll take what I can get.", "votes": 1 }, { "id": 1374276, "postDate": "2021-07-03T07:11:05.750Z", "content": "The difference is neglectable due to small amount of dimensions (6).
\nIf you will have lets say 1000 dimensions, then vectorized version will run much much faster than loop
Hello, did you use this on new data?
", "rawMarkdown": "Hello, did you use this on new data?", "replies": [ { "id": 1464480, "postDate": "2021-08-10T15:32:24.760Z", "content": "new data is already normalized that way.
", "rawMarkdown": "new data is already normalized that way." }, { "id": 1465364, "postDate": "2021-08-11T03:14:16.100Z", "content": "Thanks ! !!!
", "rawMarkdown": "Thanks ! !!!" } ] }, { "id": 1371352, "postDate": "2021-06-30T22:55:27.730Z", "content": "thanks for sharing!!
\nWhen I did this to remove leak, LB score has dropped..
\nCV:0.99086 ←0.990752
\nLB:0.983←0.984
Yes, I saw a drop too.
", "rawMarkdown": "Yes, I saw a drop too.", "votes": 1 } ] }, { "id": 1363842, "postDate": "2021-06-24T11:55:36.930Z", "content": "Excuse me, do you think snippet-wise normalization is better than image-wise normalization (i.e., data = (data- np.mean(data)) / np.std(data))?
\nFor images with target=0, the differences between raw images and snippet-wise normalized images are larger than those between raw images and image-wise normalized images.
\nThis is the code I used.
\ndf_tmp = df_train[df_train[\"target\"] == 0].sample(10)\nfor ind, row in df_tmp.iterrows():\n filename = get_train_filename_by_id(row[\"id\"])\n data = np.load(filename).astype(np.float32)\n data_ns = (data - np.mean(data, axis = (1, 2), keepdims = True)) / np.std(data, axis = (1, 2), keepdims = True)\n data_ni = (data - np.mean(data)) / np.std(data)\n ns_error = ((data_ns - data)**2).sum()\n ni_error = ((data_ni - data)**2).sum()\n print('ns_err: '+str(ns_error)+', ni_err: '+str(ni_error)+', diff: '+str(ns_error-ni_error))\nThis is the output.
\nns_err: 3.0059648e-06, ni_err: 2.3881663e-07, diff: 2.767148e-06\nns_err: 0.0013199919, ni_err: 0.0005787969, diff: 0.000741195\nns_err: 1.4696176e-05, ni_err: 3.3954695e-06, diff: 1.1300706e-05\nns_err: 0.0014001788, ni_err: 0.00010142397, diff: 0.0012987548\nns_err: 8.7956134e-05, ni_err: 4.4935976e-05, diff: 4.3020158e-05\nns_err: 8.891458e-05, ni_err: 2.2894217e-05, diff: 6.6020366e-05\nns_err: 7.280553e-05, ni_err: 1.0672206e-06, diff: 7.173831e-05\nns_err: 0.0029661339, ni_err: 3.1767113e-06, diff: 0.0029629571\nns_err: 3.1827344e-06, ni_err: 6.338249e-07, diff: 2.5489094e-06\nns_err: 0.0004897718, ni_err: 4.8265196e-05, diff: 0.00044150656\nThe diffs were always positive. Correct me if I am wrong.
", "rawMarkdown": "Excuse me, do you think snippet-wise normalization is better than image-wise normalization (i.e., data = (data- np.mean(data)) / np.std(data))?\n\nFor images with target=0, the differences between raw images and snippet-wise normalized images are larger than those between raw images and image-wise normalized images.\n\nThis is the code I used.\n```\ndf_tmp = df_train[df_train[\"target\"] == 0].sample(10)\nfor ind, row in df_tmp.iterrows():\n filename = get_train_filename_by_id(row[\"id\"])\n data = np.load(filename).astype(np.float32)\n data_ns = (data - np.mean(data, axis = (1, 2), keepdims = True)) / np.std(data, axis = (1, 2), keepdims = True)\n data_ni = (data - np.mean(data)) / np.std(data)\n ns_error = ((data_ns - data)**2).sum()\n ni_error = ((data_ni - data)**2).sum()\n print('ns_err: '+str(ns_error)+', ni_err: '+str(ni_error)+', diff: '+str(ns_error-ni_error))\n```\n\nThis is the output.\n```\nns_err: 3.0059648e-06, ni_err: 2.3881663e-07, diff: 2.767148e-06\nns_err: 0.0013199919, ni_err: 0.0005787969, diff: 0.000741195\nns_err: 1.4696176e-05, ni_err: 3.3954695e-06, diff: 1.1300706e-05\nns_err: 0.0014001788, ni_err: 0.00010142397, diff: 0.0012987548\nns_err: 8.7956134e-05, ni_err: 4.4935976e-05, diff: 4.3020158e-05\nns_err: 8.891458e-05, ni_err: 2.2894217e-05, diff: 6.6020366e-05\nns_err: 7.280553e-05, ni_err: 1.0672206e-06, diff: 7.173831e-05\nns_err: 0.0029661339, ni_err: 3.1767113e-06, diff: 0.0029629571\nns_err: 3.1827344e-06, ni_err: 6.338249e-07, diff: 2.5489094e-06\nns_err: 0.0004897718, ni_err: 4.8265196e-05, diff: 0.00044150656\n```\n\nThe diffs were always positive. Correct me if I am wrong.", "replies": [ { "id": 1364004, "postDate": "2021-06-24T14:29:08.130Z", "content": "\n\ndo you think snippet-wise normalization is better than image-wise normalization
\n
Yes because it removes any distribution shift your global image standardization removes, and it also removes any distribution shift between snippets with target 1 and snippets with target 0.
", "rawMarkdown": "> do you think snippet-wise normalization is better than image-wise normalization \n\nYes because it removes any distribution shift your global image standardization removes, and it also removes any distribution shift between snippets with target 1 and snippets with target 0. ", "votes": 2 }, { "id": 1364402, "postDate": "2021-06-25T00:11:52.050Z", "content": "I see. Thank you for your reply and sharing valuable information.
", "rawMarkdown": "I see. Thank you for your reply and sharing valuable information.", "votes": 2 }, { "id": 1364998, "postDate": "2021-06-25T11:30:45.253Z", "content": "Tx. I hope host will confirm it is good enough or provide a better way.
", "rawMarkdown": "Tx. I hope host will confirm it is good enough or provide a better way.", "votes": 1 } ] }, { "id": 1371508, "postDate": "2021-07-01T04:09:29.353Z", "content": "Thanks for sharing ..
", "rawMarkdown": "Thanks for sharing .." } ], "comments": [ { "id": 1362366, "author_name": "Old Monk", "author_url": "", "post_date": "2021-06-23T11:57:27.500000", "content": "Thanks for sharing, utilizing this in my notebook :)
", "votes": 1, "replies": [] }, { "id": 1361337, "author_name": "sajwankit", "author_url": "", "post_date": "2021-06-22T18:18:15.590000", "content": "@cpmpml Can you tell a bit about your current LB score, is it based on a single model? what image size have you used?
", "votes": 1, "replies": [ { "id": 1361847, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-23T05:08:58.617000", "content": "It is a blend. One of my best non leak model has LB 0.983, CV 0.987.
", "votes": 1, "replies": [] } ] }, { "id": 1363036, "author_name": "kkiller", "author_url": "", "post_date": "2021-06-23T21:28:19.257000", "content": "What about using the axis param instead of the for loop ?
\ndata = (data - data.mean( (1, 2), keepdims=True) ) / data.std( (1, 2), keepdims=True)
Edits
\nInclude second axis into the mean as data is 3D. Thanks Nikita.
@kneroma makes sense, but I think this would yield a different result because you set axis to 1. If you import data in the same way, you should rather use axes (1, 2) to get channel-specific stats:
\ndata = (data - np.mean(data, axis = (1, 2), keepdims = True)) / np.std(data, axis = (1, 2), keepdims = True)\n@kneroma because what you do is not the standardization host said they performed.
\n@kozodoi this is neat indeed. Not sure it changes running time though. Have you measured it?
", "votes": 0, "replies": [] }, { "id": 1363690, "author_name": "Nikita Kozodoi", "author_url": "", "post_date": "2021-06-24T09:17:43.337000", "content": "@cpmpml I did a quick check. There might be a difference, but it is too small to significantly affect the overall running time (~4 seconds over one full epoch). Also not sure how stable it is given the magnitude.
", "votes": 1, "replies": [] }, { "id": 1363830, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-24T11:42:34.370000", "content": "Nikita, thanks for checking.
", "votes": 0, "replies": [] }, { "id": 1363864, "author_name": "عثمان", "author_url": "", "post_date": "2021-06-24T12:15:58.480000", "content": "Curiously enough, I also get ~3sec / epoch (82sec vs 85sec). I'll take what I can get.
", "votes": 1, "replies": [] }, { "id": 1374276, "author_name": "Pavel Vodolazov", "author_url": "", "post_date": "2021-07-03T07:11:05.750000", "content": "The difference is neglectable due to small amount of dimensions (6).
\nIf you will have lets say 1000 dimensions, then vectorized version will run much much faster than loop
Hello, did you use this on new data?
", "votes": 0, "replies": [ { "id": 1464480, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-10T15:32:24.760000", "content": "new data is already normalized that way.
", "votes": 0, "replies": [] }, { "id": 1465364, "author_name": "Zekun", "author_url": "", "post_date": "2021-08-11T03:14:16.100000", "content": "Thanks ! !!!
", "votes": 0, "replies": [] } ] }, { "id": 1371352, "author_name": "patriot", "author_url": "", "post_date": "2021-06-30T22:55:27.730000", "content": "thanks for sharing!!
\nWhen I did this to remove leak, LB score has dropped..
\nCV:0.99086 ←0.990752
\nLB:0.983←0.984
Yes, I saw a drop too.
", "votes": 1, "replies": [] } ] }, { "id": 1363842, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-06-24T11:55:36.930000", "content": "Excuse me, do you think snippet-wise normalization is better than image-wise normalization (i.e., data = (data- np.mean(data)) / np.std(data))?
\nFor images with target=0, the differences between raw images and snippet-wise normalized images are larger than those between raw images and image-wise normalized images.
\nThis is the code I used.
\ndf_tmp = df_train[df_train[\"target\"] == 0].sample(10)\nfor ind, row in df_tmp.iterrows():\n filename = get_train_filename_by_id(row[\"id\"])\n data = np.load(filename).astype(np.float32)\n data_ns = (data - np.mean(data, axis = (1, 2), keepdims = True)) / np.std(data, axis = (1, 2), keepdims = True)\n data_ni = (data - np.mean(data)) / np.std(data)\n ns_error = ((data_ns - data)**2).sum()\n ni_error = ((data_ni - data)**2).sum()\n print('ns_err: '+str(ns_error)+', ni_err: '+str(ni_error)+', diff: '+str(ns_error-ni_error))\nThis is the output.
\nns_err: 3.0059648e-06, ni_err: 2.3881663e-07, diff: 2.767148e-06\nns_err: 0.0013199919, ni_err: 0.0005787969, diff: 0.000741195\nns_err: 1.4696176e-05, ni_err: 3.3954695e-06, diff: 1.1300706e-05\nns_err: 0.0014001788, ni_err: 0.00010142397, diff: 0.0012987548\nns_err: 8.7956134e-05, ni_err: 4.4935976e-05, diff: 4.3020158e-05\nns_err: 8.891458e-05, ni_err: 2.2894217e-05, diff: 6.6020366e-05\nns_err: 7.280553e-05, ni_err: 1.0672206e-06, diff: 7.173831e-05\nns_err: 0.0029661339, ni_err: 3.1767113e-06, diff: 0.0029629571\nns_err: 3.1827344e-06, ni_err: 6.338249e-07, diff: 2.5489094e-06\nns_err: 0.0004897718, ni_err: 4.8265196e-05, diff: 0.00044150656\nThe diffs were always positive. Correct me if I am wrong.
", "votes": 0, "replies": [ { "id": 1364004, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-24T14:29:08.130000", "content": "\n\ndo you think snippet-wise normalization is better than image-wise normalization
\n
Yes because it removes any distribution shift your global image standardization removes, and it also removes any distribution shift between snippets with target 1 and snippets with target 0.
", "votes": 2, "replies": [] }, { "id": 1364402, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-06-25T00:11:52.050000", "content": "I see. Thank you for your reply and sharing valuable information.
", "votes": 2, "replies": [] }, { "id": 1364998, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-25T11:30:45.253000", "content": "Tx. I hope host will confirm it is good enough or provide a better way.
", "votes": 1, "replies": [] } ] }, { "id": 1371508, "author_name": "laxman kusuma", "author_url": "", "post_date": "2021-07-01T04:09:29.353000", "content": "Thanks for sharing ..
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1361282": "What I am doing is rather simple: when I load data I normalize each of the 6 snippets separately. And I don't use any file metadata ofc.\n\nThis is good enough to remove known leaks, and I hope there are none left...\n\nLoading function:\n\n```\ndef load_image(filename, data_path=train_path):\n data = np.load(data_path / filename[0] / (filename + '.npy')).astype(np.float32)\n for i in range(data.shape[0]):\n data[i] -= data[i].mean()\n data[i] /= data[i].std()\n return data\n```\n\nEdit. The above was for the old dataset. The new dataset is properly normalized.", "1362366": "Thanks for sharing, utilizing this in my notebook :)", "1361337": "@cpmpml Can you tell a bit about your current LB score, is it based on a single model? what image size have you used?", "1363036": "What about using the **axis** param instead of the **for loop** ?\n\n`data = (data - data.mean( (1, 2), keepdims=True) ) / data.std( (1, 2), keepdims=True)`\n\n**Edits**\n*Include second axis into the **mean** as data is 3D. Thanks Nikita.*", "1464331": "Hello, did you use this on new data?", "1371352": "thanks for sharing!!\nWhen I did this to remove leak, LB score has dropped..\nCV:0.99086 ←0.990752\nLB:0.983←0.984 ", "1363842": "Excuse me, do you think snippet-wise normalization is better than image-wise normalization (i.e., data = (data- np.mean(data)) / np.std(data))?\n\nFor images with target=0, the differences between raw images and snippet-wise normalized images are larger than those between raw images and image-wise normalized images.\n\nThis is the code I used.\n```\ndf_tmp = df_train[df_train[\"target\"] == 0].sample(10)\nfor ind, row in df_tmp.iterrows():\n filename = get_train_filename_by_id(row[\"id\"])\n data = np.load(filename).astype(np.float32)\n data_ns = (data - np.mean(data, axis = (1, 2), keepdims = True)) / np.std(data, axis = (1, 2), keepdims = True)\n data_ni = (data - np.mean(data)) / np.std(data)\n ns_error = ((data_ns - data)**2).sum()\n ni_error = ((data_ni - data)**2).sum()\n print('ns_err: '+str(ns_error)+', ni_err: '+str(ni_error)+', diff: '+str(ns_error-ni_error))\n```\n\nThis is the output.\n```\nns_err: 3.0059648e-06, ni_err: 2.3881663e-07, diff: 2.767148e-06\nns_err: 0.0013199919, ni_err: 0.0005787969, diff: 0.000741195\nns_err: 1.4696176e-05, ni_err: 3.3954695e-06, diff: 1.1300706e-05\nns_err: 0.0014001788, ni_err: 0.00010142397, diff: 0.0012987548\nns_err: 8.7956134e-05, ni_err: 4.4935976e-05, diff: 4.3020158e-05\nns_err: 8.891458e-05, ni_err: 2.2894217e-05, diff: 6.6020366e-05\nns_err: 7.280553e-05, ni_err: 1.0672206e-06, diff: 7.173831e-05\nns_err: 0.0029661339, ni_err: 3.1767113e-06, diff: 0.0029629571\nns_err: 3.1827344e-06, ni_err: 6.338249e-07, diff: 2.5489094e-06\nns_err: 0.0004897718, ni_err: 4.8265196e-05, diff: 0.00044150656\n```\n\nThe diffs were always positive. Correct me if I am wrong.", "1371508": "Thanks for sharing .." } }