Hi, When I ran this in tf notebook:
\nfrom sklearn.neighbors import NearestNeighbors\nneigh = NearestNeighbors(n_neighbors=config.KNN,metric='cosine')\nneigh.fit(train_embeddings)\nI got an error,that is, nan value is in train_embeddings.
\nHow can I deal with it ?
", "messages": [ { "id": 1731446, "postDate": "2022-03-22T11:28:41.623Z", "content": "Hi, When I ran this in tf notebook:
\nfrom sklearn.neighbors import NearestNeighbors\nneigh = NearestNeighbors(n_neighbors=config.KNN,metric='cosine')\nneigh.fit(train_embeddings)\nI got an error,that is, nan value is in train_embeddings.
\nHow can I deal with it ?
", "rawMarkdown": "Hi, When I ran this in tf notebook:\n```\nfrom sklearn.neighbors import NearestNeighbors\nneigh = NearestNeighbors(n_neighbors=config.KNN,metric='cosine')\nneigh.fit(train_embeddings)\n```\nI got an error,that is, nan value is in train_embeddings.\n\nHow can I deal with it ?", "votes": 7 }, { "id": 1735038, "postDate": "2022-03-25T19:52:38.333Z", "content": "The best solution is to investigate why NAN exist and correct the source of the problem. However, if you want a quick fix, you can use NumPy's NAN to num function:
\n import numpy as np\n train_embeddings = np.nan_to_num( train_embeddings )\nIn my case, NAN is in the model predictions. When I run
\nemb = model.predict(ds)\nSome value in the emb is NAN. Is this the problem of input (images) or the model?
", "rawMarkdown": "In my case, NAN is in the model predictions. When I run \n\n```\nemb = model.predict(ds)\n```\n\nSome value in the emb is NAN. Is this the problem of input (images) or the model?" }, { "id": 1737164, "postDate": "2022-03-28T06:58:51.357Z", "content": "@ptran1203 I'm having the same issue. How did you solve this? I use now the suggestion by Chris Deotte:
\nembedding = np.nan_to_num( embedding )
Hi there! I also encountered this problem. I solved it by simply installing another version of tensorflow.
", "rawMarkdown": "Hi there! I also encountered this problem. I solved it by simply installing another version of tensorflow.", "votes": 1 }, { "id": 1735090, "postDate": "2022-03-25T20:59:23.467Z", "content": "I think I also had a NaN error at that point in one particular fold. I just reduced the batch size from 4 to 2 to fix it.
", "rawMarkdown": "I think I also had a NaN error at that point in one particular fold. I just reduced the batch size from 4 to 2 to fix it. ", "votes": 1 }, { "id": 1735708, "postDate": "2022-03-26T14:39:04.763Z", "content": "nan_to_num should work but I've found SimpleImputer is a bit better probably
\nfrom sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ntrain_embeddings = my_imputer.fit_transform(train_embeddings)\nyou can check this for more details
", "rawMarkdown": "nan_to_num should work but I've found SimpleImputer is a bit better probably\n\n```\nfrom sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ntrain_embeddings = my_imputer.fit_transform(train_embeddings)\n```\nyou can check [this](https://www.kaggle.com/code/dansbecker/handling-missing-values/notebook) for more details" }, { "id": 1735614, "postDate": "2022-03-26T12:48:52.230Z", "content": "my solution is add this line
\ntrain_embeddings[np.isnan(train_embeddings)] = 1e-9
", "rawMarkdown": "my solution is add this line\n\ntrain_embeddings[np.isnan(train_embeddings)] = 1e-9" }, { "id": 1731488, "postDate": "2022-03-22T12:16:57.753Z", "content": "If I run on Colab it's ok but when I copy the notebook to Kaggle, It produces nan value in test embeddings
", "rawMarkdown": "If I run on Colab it's ok but when I copy the notebook to Kaggle, It produces nan value in test embeddings", "replies": [ { "id": 1731515, "postDate": "2022-03-22T12:42:33.067Z", "content": "Me,too. How do you deal with this?
", "rawMarkdown": "Me,too. How do you deal with this?" }, { "id": 1732594, "postDate": "2022-03-23T14:41:23.920Z", "content": "I did not solve it yet, just manually copy the weight and run on colab :v
", "rawMarkdown": "I did not solve it yet, just manually copy the weight and run on colab :v", "votes": 1 }, { "id": 1734522, "postDate": "2022-03-25T12:16:26.437Z", "content": "@rainfalllove ,对这些出现nan的数值取均值，或者赋值一个极小的数：1e-9，顺便问一句，你还需要队友吗，我们可以一起组队往前冲吗？我目前single fold 0.773💪💪💪
", "rawMarkdown": "@rainfalllove ,对这些出现nan的数值取均值，或者赋值一个极小的数：1e-9，顺便问一句，你还需要队友吗，我们可以一起组队往前冲吗？我目前single fold 0.773💪💪💪" } ] }, { "id": 1731462, "postDate": "2022-03-22T11:44:43.423Z", "content": "I'd recommend to check the train_embeddings source (where you got it), there are a lot of potential reasons - from bad model prediction to bugs with numpy/torch processing
problem solved, thank everyone!
", "rawMarkdown": "problem solved, thank everyone!" } ], "comments": [ { "id": 1735038, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-25T19:52:38.333000", "content": "The best solution is to investigate why NAN exist and correct the source of the problem. However, if you want a quick fix, you can use NumPy's NAN to num function:
\n import numpy as np\n train_embeddings = np.nan_to_num( train_embeddings )\nIn my case, NAN is in the model predictions. When I run
\nemb = model.predict(ds)\nSome value in the emb is NAN. Is this the problem of input (images) or the model?
", "votes": 0, "replies": [] }, { "id": 1737164, "author_name": "Meesz9", "author_url": "", "post_date": "2022-03-28T06:58:51.357000", "content": "@ptran1203 I'm having the same issue. How did you solve this? I use now the suggestion by Chris Deotte:
\nembedding = np.nan_to_num( embedding )
Hi there! I also encountered this problem. I solved it by simply installing another version of tensorflow.
", "votes": 1, "replies": [] }, { "id": 1735090, "author_name": "Bruce Young", "author_url": "", "post_date": "2022-03-25T20:59:23.467000", "content": "I think I also had a NaN error at that point in one particular fold. I just reduced the batch size from 4 to 2 to fix it.
", "votes": 1, "replies": [] }, { "id": 1735708, "author_name": "Hannah B", "author_url": "", "post_date": "2022-03-26T14:39:04.763000", "content": "nan_to_num should work but I've found SimpleImputer is a bit better probably
\nfrom sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ntrain_embeddings = my_imputer.fit_transform(train_embeddings)\nyou can check this for more details
", "votes": 0, "replies": [] }, { "id": 1735614, "author_name": "liuzhangzhen", "author_url": "", "post_date": "2022-03-26T12:48:52.230000", "content": "my solution is add this line
\ntrain_embeddings[np.isnan(train_embeddings)] = 1e-9
", "votes": 0, "replies": [] }, { "id": 1731488, "author_name": "Phat Tran", "author_url": "", "post_date": "2022-03-22T12:16:57.753000", "content": "If I run on Colab it's ok but when I copy the notebook to Kaggle, It produces nan value in test embeddings
", "votes": 0, "replies": [ { "id": 1731515, "author_name": "Time Master", "author_url": "", "post_date": "2022-03-22T12:42:33.067000", "content": "Me,too. How do you deal with this?
", "votes": 0, "replies": [] }, { "id": 1732594, "author_name": "Phat Tran", "author_url": "", "post_date": "2022-03-23T14:41:23.920000", "content": "I did not solve it yet, just manually copy the weight and run on colab :v
", "votes": 1, "replies": [] }, { "id": 1734522, "author_name": "huyahuya", "author_url": "", "post_date": "2022-03-25T12:16:26.437000", "content": "@rainfalllove ,对这些出现nan的数值取均值，或者赋值一个极小的数：1e-9，顺便问一句，你还需要队友吗，我们可以一起组队往前冲吗？我目前single fold 0.773💪💪💪
", "votes": 0, "replies": [] } ] }, { "id": 1731462, "author_name": "Aleksey Alekseev", "author_url": "", "post_date": "2022-03-22T11:44:43.423000", "content": "I'd recommend to check the train_embeddings source (where you got it), there are a lot of potential reasons - from bad model prediction to bugs with numpy/torch processing
problem solved, thank everyone!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1731446": "Hi, When I ran this in tf notebook:\n```\nfrom sklearn.neighbors import NearestNeighbors\nneigh = NearestNeighbors(n_neighbors=config.KNN,metric='cosine')\nneigh.fit(train_embeddings)\n```\nI got an error,that is, nan value is in train_embeddings.\n\nHow can I deal with it ?", "1735038": "The best solution is to investigate why NAN exist and correct the source of the problem. However, if you want a quick fix, you can use NumPy's NAN to num function:\n\n import numpy as np\n train_embeddings = np.nan_to_num( train_embeddings )", "1736779": "Hi there! I also encountered this problem. I solved it by simply installing another version of tensorflow.", "1735090": "I think I also had a NaN error at that point in one particular fold. I just reduced the batch size from 4 to 2 to fix it. ", "1735708": "nan_to_num should work but I've found SimpleImputer is a bit better probably\n\n```\nfrom sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ntrain_embeddings = my_imputer.fit_transform(train_embeddings)\n```\nyou can check [this](https://www.kaggle.com/code/dansbecker/handling-missing-values/notebook) for more details", "1735614": "my solution is add this line\n\ntrain_embeddings[np.isnan(train_embeddings)] = 1e-9", "1731488": "If I run on Colab it's ok but when I copy the notebook to Kaggle, It produces nan value in test embeddings", "1731462": "I'd recommend to check the `train_embeddings` source (where you got it), there are a lot of potential reasons - from bad model prediction to bugs with numpy/torch processing ", "1739573": "", "1739571": "", "1734932": "", "1737240": "problem solved, thank everyone!" } }