{ "id": 182811, "title": "Results differ in Interactive and Commit session", "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/182811", "author_name": "Jaideep", "post_date": "2020-09-14T12:10:01.967000", "votes": 2, "comment_count": 13, "views": 0, "content": "

I find quite different results for same notebook when run in interactive mode and when i commit that same notebook. I fix the seed also.
\nWhat could be the issue ?

", "messages": [ { "id": 1032431, "postDate": "2020-09-30T07:43:37.993Z", "content": "

@jaideepvalani @alexj21 I faced the same issue, but i seem to have resolved it so far. I do the following steps in the exact order for the interactive session. After making any changes to your code in the interactive session;

\n
    \n
  1. Restart the session
    2. \n
  2. Power off the session
    3. \n
  3. Close the chrome or firefox tab that contains the interactive session completely
    4. \n
  4. Wait a few minutes then reopen the interactive session and run your notebook.
    5. \n
\n

After i do this both commit and interactive session results match.

\n

P.S I don't know why this works but it worked for me so far

", "rawMarkdown": "@jaideepvalani @alexj21 I faced the same issue, but i seem to have resolved it so far. I do the following steps in the exact order for the interactive session. After making any changes to your code in the interactive session; \n1. Restart the session\n2. Power off the session\n3. Close the chrome or firefox tab that contains the interactive session completely\n4. Wait a few minutes then reopen the interactive session and run your notebook.\n\nAfter i do this both commit and interactive session results match.\n\nP.S I don't know why this works but it worked for me so far", "votes": 1, "replies": [ { "id": 1032522, "postDate": "2020-09-30T09:05:33.683Z", "content": "

Thank you, I'll give it a try. Pretty weird indeed. My kernel is an ensemble of two models, the first one is giving consistent results but the second one is giving different results in interactive and commit session.
\nI'm wondering if this happens during submission and how it can potentially change the LB score

", "rawMarkdown": "Thank you, I'll give it a try. Pretty weird indeed. My kernel is an ensemble of two models, the first one is giving consistent results but the second one is giving different results in interactive and commit session. \nI'm wondering if this happens during submission and how it can potentially change the LB score " }, { "id": 1034346, "postDate": "2020-10-01T16:53:10.087Z", "content": "

@alexj21 @yovinyahathugoda @jaideepvalani this type of issue is typically not caused by Kaggle, but by the underlying implementation of the code you are calling (some stuff just isn't \"deterministic\", some methods don't accept/respect global seeds, etc.).

\n

If you have a clear, small, reproducible example of code you are certain should be stable but isn't, we're happy to take a look! Beyond that, we are not equipped to dive into third-party codebases to find sources of variability.

", "rawMarkdown": "@alexj21 @yovinyahathugoda @jaideepvalani this type of issue is typically not caused by Kaggle, but by the underlying implementation of the code you are calling (some stuff just isn't \"deterministic\", some methods don't accept/respect global seeds, etc.).\n\nIf you have a clear, small, reproducible example of code you are certain should be stable but isn't, we're happy to take a look! Beyond that, we are not equipped to dive into third-party codebases to find sources of variability." }, { "id": 1034748, "postDate": "2020-10-02T06:21:31.573Z", "content": "

@wcukierski I have set a fixed seed and PyTorch deterministic=True along with the other flags needed for reproducibility. I can replicate the results of an experiment accurately with the same hyperparameters as shown in the 1st image below. Problem is when the draft mode and commit mode results do not match even though the code is exactly the same, and the 2nd image below shows an example.
\n\"\"
\n\"\"
\nThe 3 different experiments shown above contain no changes in any part of the code. If I share a link to my private notebook will you be able to take a look at it? Also I don't have this problem when I carry out the following steps every time I run the notebook.

\n
\n
    \n
  1. Restart the session
    2. \n
  2. Power off the session
    3. \n
  3. Close the chrome or firefox tab that contains the interactive session completely
    4. \n
  4. Wait a few minutes then reopen the interactive session and run your notebook.
    5. \n
\n
", "rawMarkdown": "@wcukierski I have set a fixed seed and `PyTorch deterministic=True` along with the other flags needed for reproducibility. I can replicate the results of an experiment accurately with the same hyperparameters as shown in the 1st image below. Problem is when the draft mode and commit mode results do not match even though the code is exactly the same, and the 2nd image below shows an example.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1554871%2F22581c9fffd76baed74ac125d460fd10%2Fmatch.png?generation=1601619342940037&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1554871%2Fda29060c17f8d30e6fd026fd8fafb39a%2Fnon-match.png?generation=1601619392887980&alt=media)\nThe 3 different experiments shown above contain no changes in any part of the code. If I share a link to my private notebook will you be able to take a look at it? Also I don't have this problem when I carry out the following steps every time I run the notebook.\n> \n1. Restart the session\n2. Power off the session\n3. Close the chrome or firefox tab that contains the interactive session completely\n4. Wait a few minutes then reopen the interactive session and run your notebook." }, { "id": 1035140, "postDate": "2020-10-02T13:33:42.927Z", "content": "

@yovinyahathugoda I'm not a pytorch expert and likely wouldn't be able to quickly spot the issue, but Googling \"pytorch deterministic\" turns up quite a few discussions and caveats (including https://pytorch.org/docs/stable/notes/randomness.html ). I assume that you also get different results if you run the same code interactive twice in a row (without restarting the session)? I agree it would be strange if the code is reproducible under fresh interactive sessions but not in a commit (which runs on the same hardware, same docker env, etc), but given all the possible factors affecting this, I'm afraid I wouldn't be much help in diagnosing.

", "rawMarkdown": "@yovinyahathugoda I'm not a pytorch expert and likely wouldn't be able to quickly spot the issue, but Googling \"pytorch deterministic\" turns up quite a few discussions and caveats (including https://pytorch.org/docs/stable/notes/randomness.html ). I assume that you also get different results if you run the same code interactive twice in a row (without restarting the session)? I agree it would be strange if the code is reproducible under fresh interactive sessions but not in a commit (which runs on the same hardware, same docker env, etc), but given all the possible factors affecting this, I'm afraid I wouldn't be much help in diagnosing.", "votes": 1 }, { "id": 1035308, "postDate": "2020-10-02T16:13:32.757Z", "content": "

@wcukierski To answer your question, yes I get different results running the same code twice in interactive mode without restarting the session. Thanks anyway for trying to help.

", "rawMarkdown": "@wcukierski To answer your question, yes I get different results running the same code twice in interactive mode without restarting the session. Thanks anyway for trying to help.", "votes": 1 } ] }, { "id": 1010546, "postDate": "2020-09-14T21:08:44.580Z", "content": "

Depends on the model that you are using. For example, if I'm not mistaken, NNs aren't 100% reproducible.

", "rawMarkdown": "Depends on the model that you are using. For example, if I'm not mistaken, NNs aren't 100% reproducible.", "votes": 1 }, { "id": 1009967, "postDate": "2020-09-14T12:10:01.967Z", "content": "

I find quite different results for same notebook when run in interactive mode and when i commit that same notebook. I fix the seed also.
\nWhat could be the issue ?

", "rawMarkdown": "I find quite different results for same notebook when run in interactive mode and when i commit that same notebook. I fix the seed also. \nWhat could be the issue ?", "votes": 2 }, { "id": 1031891, "postDate": "2020-09-29T18:52:51.887Z", "content": "

@jaideepvalani did you figure it out ? I'm facing the same issue

", "rawMarkdown": "@jaideepvalani did you figure it out ? I'm facing the same issue", "replies": [ { "id": 1032474, "postDate": "2020-09-30T08:33:58.783Z", "content": "

no not really.. slight difference always there.. if huge diff then may b u changed some thing and dont remember :)

", "rawMarkdown": "no not really.. slight difference always there.. if huge diff then may b u changed some thing and dont remember :)" }, { "id": 1032515, "postDate": "2020-09-30T09:00:10.067Z", "content": "

No same as you, slight differences

", "rawMarkdown": "No same as you, slight differences" } ] }, { "id": 1010571, "postDate": "2020-09-14T22:05:46.943Z", "content": "

The interactive test set is a sample of 5 studies (actually from the train set). When you commit the model it is run against the hidden data. So the results will always differ.

", "rawMarkdown": "The interactive test set is a sample of 5 studies (actually from the train set). When you commit the model it is run against the hidden data. So the results will always differ.", "replies": [ { "id": 1010606, "postDate": "2020-09-14T23:01:18.547Z", "content": "
\n

The interactive test set is a sample of 5 studies (actually from the train set). When you commit the model it is run against the hidden data. So the results will always differ.

\n
\n

This is when you make a submission, not when you commit a notebook version! If that was the case, the data wouldn't be considered private!

", "rawMarkdown": "> The interactive test set is a sample of 5 studies (actually from the train set). When you commit the model it is run against the hidden data. So the results will always differ.\n\nThis is when you make a **submission**, not when you commit a notebook version! If that was the case, the data wouldn't be considered private!\n" } ] }, { "id": 1014022, "postDate": "2020-09-17T06:38:42.890Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 1032431, "author_name": "Yovin Yahathugoda", "author_url": "", "post_date": "2020-09-30T07:43:37.993000", "content": "

@jaideepvalani @alexj21 I faced the same issue, but i seem to have resolved it so far. I do the following steps in the exact order for the interactive session. After making any changes to your code in the interactive session;

\n
    \n
  1. Restart the session
    2. \n
  2. Power off the session
    3. \n
  3. Close the chrome or firefox tab that contains the interactive session completely
    4. \n
  4. Wait a few minutes then reopen the interactive session and run your notebook.
    5. \n
\n

After i do this both commit and interactive session results match.

\n

P.S I don't know why this works but it worked for me so far

", "votes": 1, "replies": [ { "id": 1032522, "author_name": "Alex", "author_url": "", "post_date": "2020-09-30T09:05:33.683000", "content": "

Thank you, I'll give it a try. Pretty weird indeed. My kernel is an ensemble of two models, the first one is giving consistent results but the second one is giving different results in interactive and commit session.
\nI'm wondering if this happens during submission and how it can potentially change the LB score

", "votes": 0, "replies": [] }, { "id": 1034346, "author_name": "Will Cukierski", "author_url": "", "post_date": "2020-10-01T16:53:10.087000", "content": "

@alexj21 @yovinyahathugoda @jaideepvalani this type of issue is typically not caused by Kaggle, but by the underlying implementation of the code you are calling (some stuff just isn't \"deterministic\", some methods don't accept/respect global seeds, etc.).

\n

If you have a clear, small, reproducible example of code you are certain should be stable but isn't, we're happy to take a look! Beyond that, we are not equipped to dive into third-party codebases to find sources of variability.

", "votes": 0, "replies": [] }, { "id": 1034748, "author_name": "Yovin Yahathugoda", "author_url": "", "post_date": "2020-10-02T06:21:31.573000", "content": "

@wcukierski I have set a fixed seed and PyTorch deterministic=True along with the other flags needed for reproducibility. I can replicate the results of an experiment accurately with the same hyperparameters as shown in the 1st image below. Problem is when the draft mode and commit mode results do not match even though the code is exactly the same, and the 2nd image below shows an example.
\n\"\"
\n\"\"
\nThe 3 different experiments shown above contain no changes in any part of the code. If I share a link to my private notebook will you be able to take a look at it? Also I don't have this problem when I carry out the following steps every time I run the notebook.

\n
\n
    \n
  1. Restart the session
    2. \n
  2. Power off the session
    3. \n
  3. Close the chrome or firefox tab that contains the interactive session completely
    4. \n
  4. Wait a few minutes then reopen the interactive session and run your notebook.
    5. \n
\n
", "votes": 0, "replies": [] }, { "id": 1035140, "author_name": "Will Cukierski", "author_url": "", "post_date": "2020-10-02T13:33:42.927000", "content": "

@yovinyahathugoda I'm not a pytorch expert and likely wouldn't be able to quickly spot the issue, but Googling \"pytorch deterministic\" turns up quite a few discussions and caveats (including https://pytorch.org/docs/stable/notes/randomness.html ). I assume that you also get different results if you run the same code interactive twice in a row (without restarting the session)? I agree it would be strange if the code is reproducible under fresh interactive sessions but not in a commit (which runs on the same hardware, same docker env, etc), but given all the possible factors affecting this, I'm afraid I wouldn't be much help in diagnosing.

", "votes": 1, "replies": [] }, { "id": 1035308, "author_name": "Yovin Yahathugoda", "author_url": "", "post_date": "2020-10-02T16:13:32.757000", "content": "

@wcukierski To answer your question, yes I get different results running the same code twice in interactive mode without restarting the session. Thanks anyway for trying to help.

", "votes": 1, "replies": [] } ] }, { "id": 1010546, "author_name": "Vasileios Kagklis", "author_url": "", "post_date": "2020-09-14T21:08:44.580000", "content": "

Depends on the model that you are using. For example, if I'm not mistaken, NNs aren't 100% reproducible.

", "votes": 1, "replies": [] }, { "id": 1031891, "author_name": "Alex", "author_url": "", "post_date": "2020-09-29T18:52:51.887000", "content": "

@jaideepvalani did you figure it out ? I'm facing the same issue

", "votes": 0, "replies": [ { "id": 1032474, "author_name": "Jaideep", "author_url": "", "post_date": "2020-09-30T08:33:58.783000", "content": "

no not really.. slight difference always there.. if huge diff then may b u changed some thing and dont remember :)

", "votes": 0, "replies": [] }, { "id": 1032515, "author_name": "Alex", "author_url": "", "post_date": "2020-09-30T09:00:10.067000", "content": "

No same as you, slight differences

", "votes": 0, "replies": [] } ] }, { "id": 1010571, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-09-14T22:05:46.943000", "content": "

The interactive test set is a sample of 5 studies (actually from the train set). When you commit the model it is run against the hidden data. So the results will always differ.

", "votes": 0, "replies": [ { "id": 1010606, "author_name": "Vasileios Kagklis", "author_url": "", "post_date": "2020-09-14T23:01:18.547000", "content": "
\n

The interactive test set is a sample of 5 studies (actually from the train set). When you commit the model it is run against the hidden data. So the results will always differ.

\n
\n

This is when you make a submission, not when you commit a notebook version! If that was the case, the data wouldn't be considered private!

", "votes": 0, "replies": [] } ] }, { "id": 1014022, "author_name": "", "author_url": "", "post_date": "2020-09-17T06:38:42.890000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "1032431": "@jaideepvalani @alexj21 I faced the same issue, but i seem to have resolved it so far. I do the following steps in the exact order for the interactive session. After making any changes to your code in the interactive session; \n1. Restart the session\n2. Power off the session\n3. Close the chrome or firefox tab that contains the interactive session completely\n4. Wait a few minutes then reopen the interactive session and run your notebook.\n\nAfter i do this both commit and interactive session results match.\n\nP.S I don't know why this works but it worked for me so far", "1010546": "Depends on the model that you are using. For example, if I'm not mistaken, NNs aren't 100% reproducible.", "1009967": "I find quite different results for same notebook when run in interactive mode and when i commit that same notebook. I fix the seed also. \nWhat could be the issue ?", "1031891": "@jaideepvalani did you figure it out ? I'm facing the same issue", "1010571": "The interactive test set is a sample of 5 studies (actually from the train set). When you commit the model it is run against the hidden data. So the results will always differ.", "1014022": "" } }