I am trying to use this public kernel and modify it to do k-fold training on TPU. But I am getting the following error after first fold finishes. Thanks for the kernel @riblidezso .
\n\n\nNotFoundError: {{function_node __inference_distributed_train_step_139198}} Resource worker/Adam/iter/replica_6_57590/N10tensorflow3VarE does not exist.\n [[{{node cluster_distributed_train_step/_execute_6_0}}]]\n\nI made changes like calling tf.tpu.experimental.initialize_tpu_system(tpu) before each of the fold. I also added drop_reminder=True where needed. But I am getting above error when starting the second fold. Not sure why yet. Seems like there is an issue in creating a distributed dataset in the second fold. So how can I fix this? I am not able to paste the full traceback somehow here not sure why.
My dumb fix is to have K-kernels for K-folds (as a last resort :D )
", "rawMarkdown": "My dumb fix is to have K-kernels for K-folds (as a last resort :D )", "votes": 3, "replies": [ { "id": 885259, "postDate": "2020-06-14T04:14:23.053Z", "content": "yeah but that is not going to help much as now I am not going to have enough time to do so. I found some kernels from Flower classification challenge on TPU which might be helpful.
Just simply don't do this, as this causes the problem of the deleted variable.\n\ntf.tpu.experimental.initialize_tpu_system(tpu)\n\nthis is a brute force way to fix issues with Keras taking and leaving around too much memory.
With my kernel you can have repeated train/predict calls (10+) without any memory issue.
", "rawMarkdown": "Just simply don't do this, as this causes the problem of the deleted variable.\n```\ntf.tpu.experimental.initialize_tpu_system(tpu)\n```\nthis is a brute force way to fix issues with Keras taking and leaving around too much memory.\n\nWith my kernel you can have repeated train/predict calls (10+) without any memory issue.", "votes": 3, "replies": [ { "id": 885260, "postDate": "2020-06-14T04:17:01.680Z", "content": "I did not get the first point Just do not do it, what do you mean? Your kernel is great and I think I will make it work for me for k-folds. I used the above-suggested way but then the data is not getting distributed somehow even if I call the function. Not sure why.
I think I misunderstood your first point. You want to say that tf.tpu.experimental.initialize_tpu_system(tpu) do not use this at the first line of each fold? Am I right?
I removed this line from the beginning of the fold but then got the resource exhausted error as mentioned in other discussions. I am doing following steps while training \n1. Re-initialize the strategy, tpu and global batch size.\n2. Make the distributed data.\n3. Load losses and metrics\n4. Load model and optimizer \n5. Train and fine-tune model\n6. Predict on test data.\nNow if I remove tf.tpu.experimental.initialize_tpu_system(tpu), then I got resource exhausted error and if not then above not found error. What steps should I remove from the loop in order to make it work?
Sorry If I was not totally clear. Yes I meant do no reinitizalize the TPU.
\n\nYou get resource exhausted error because you reinitialize stuff for each fold. You don't need to reinitialize the TPU or the model or optimizers.
\n\nJust reset the model weights, and redefine the training and validation datasets.
\n\nMy pseudo code for cross validation:
\n\n```\nfor tr_idx, test_idx in KFold(K).split(X):\n ### Splitting data ### \n ...
\n\n### Creating Datasets ###\ntrain_dist_dataset = create_dist_dataset(X_train, y_train, training = True)\nval_dist_dataset = create_dist_dataset(X_val)\n\nmodel.set_weights(initial_w)\ntrain()\npredict()\n```
\n\nI hope this helps. Good luck.
", "rawMarkdown": "Sorry If I was not totally clear. Yes I meant do no reinitizalize the TPU.\n\nYou get resource exhausted error because you reinitialize stuff for each fold. You don't need to reinitialize the TPU or the model or optimizers. \n\nJust reset the model weights, and redefine the training and validation datasets.\n\nMy pseudo code for cross validation:\n\n```\nfor tr_idx, test_idx in KFold(K).split(X):\n ### Splitting data ### \n ...\n\n ### Creating Datasets ###\n train_dist_dataset = create_dist_dataset(X_train, y_train, training = True)\n val_dist_dataset = create_dist_dataset(X_val)\n\n model.set_weights(initial_w)\n train()\n predict()\n```\n \nI hope this helps. Good luck.\n", "votes": 3 }, { "id": 886728, "postDate": "2020-06-15T08:38:35.537Z", "content": "Thanks for the help. I got your point. I will do this quickly and will see if I can get more boost in score. Thanks.
", "rawMarkdown": "Thanks for the help. I got your point. I will do this quickly and will see if I can get more boost in score. Thanks." } ] }, { "id": 885384, "postDate": "2020-06-14T06:45:11.950Z", "content": "I solved it finally. I realized how much silly the mistake was. The fix was that I had to define all the functions inside the for loop which iterates over each fold. For example, define the load model, create a dataset, train step etc inside the loop and it worked. Some useful resources were this and this
", "rawMarkdown": "I solved it finally. I realized how much silly the mistake was. The fix was that I had to define all the functions inside the for loop which iterates over each fold. For example, define the load model, create a dataset, train step etc inside the loop and it worked. Some useful resources were [this](https://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu#Optimized-custom-training-loop) and [this](https://www.kaggle.com/dimitreoliveira/flower-with-tpus-k-fold-optimized-training-loops)", "votes": 1 }, { "id": 885114, "postDate": "2020-06-13T22:25:08.047Z", "content": "Beware of 5GB limit for output. You can save only 2 models as one is 2+gb
", "rawMarkdown": "Beware of 5GB limit for output. You can save only 2 models as one is 2+gb", "votes": 1, "replies": [ { "id": 885261, "postDate": "2020-06-14T04:17:52.097Z", "content": "I do not want to save all models, at this stage I need better predictions only so that I can later make ensemble out of it.
", "rawMarkdown": "I do not want to save all models, at this stage I need better predictions only so that I can later make ensemble out of it. " } ] }, { "id": 884497, "postDate": "2020-06-13T12:20:53.067Z", "content": "I am trying to use this public kernel and modify it to do k-fold training on TPU. But I am getting the following error after first fold finishes. Thanks for the kernel @riblidezso .
\n\n\nNotFoundError: {{function_node __inference_distributed_train_step_139198}} Resource worker/Adam/iter/replica_6_57590/N10tensorflow3VarE does not exist.\n [[{{node cluster_distributed_train_step/_execute_6_0}}]]\n\nI made changes like calling tf.tpu.experimental.initialize_tpu_system(tpu) before each of the fold. I also added drop_reminder=True where needed. But I am getting above error when starting the second fold. Not sure why yet. Seems like there is an issue in creating a distributed dataset in the second fold. So how can I fix this? I am not able to paste the full traceback somehow here not sure why.
My dumb fix is to have K-kernels for K-folds (as a last resort :D )
", "votes": 3, "replies": [ { "id": 885259, "author_name": "Urvish", "author_url": "", "post_date": "2020-06-14T04:14:23.053000", "content": "yeah but that is not going to help much as now I am not going to have enough time to do so. I found some kernels from Flower classification challenge on TPU which might be helpful.
Just simply don't do this, as this causes the problem of the deleted variable.\n\ntf.tpu.experimental.initialize_tpu_system(tpu)\n\nthis is a brute force way to fix issues with Keras taking and leaving around too much memory.
With my kernel you can have repeated train/predict calls (10+) without any memory issue.
", "votes": 3, "replies": [ { "id": 885260, "author_name": "Urvish", "author_url": "", "post_date": "2020-06-14T04:17:01.680000", "content": "I did not get the first point Just do not do it, what do you mean? Your kernel is great and I think I will make it work for me for k-folds. I used the above-suggested way but then the data is not getting distributed somehow even if I call the function. Not sure why.
I think I misunderstood your first point. You want to say that tf.tpu.experimental.initialize_tpu_system(tpu) do not use this at the first line of each fold? Am I right?
I removed this line from the beginning of the fold but then got the resource exhausted error as mentioned in other discussions. I am doing following steps while training \n1. Re-initialize the strategy, tpu and global batch size.\n2. Make the distributed data.\n3. Load losses and metrics\n4. Load model and optimizer \n5. Train and fine-tune model\n6. Predict on test data.\nNow if I remove tf.tpu.experimental.initialize_tpu_system(tpu), then I got resource exhausted error and if not then above not found error. What steps should I remove from the loop in order to make it work?
Sorry If I was not totally clear. Yes I meant do no reinitizalize the TPU.
\n\nYou get resource exhausted error because you reinitialize stuff for each fold. You don't need to reinitialize the TPU or the model or optimizers.
\n\nJust reset the model weights, and redefine the training and validation datasets.
\n\nMy pseudo code for cross validation:
\n\n```\nfor tr_idx, test_idx in KFold(K).split(X):\n ### Splitting data ### \n ...
\n\n### Creating Datasets ###\ntrain_dist_dataset = create_dist_dataset(X_train, y_train, training = True)\nval_dist_dataset = create_dist_dataset(X_val)\n\nmodel.set_weights(initial_w)\ntrain()\npredict()\n```
\n\nI hope this helps. Good luck.
", "votes": 3, "replies": [] }, { "id": 886728, "author_name": "Urvish", "author_url": "", "post_date": "2020-06-15T08:38:35.537000", "content": "Thanks for the help. I got your point. I will do this quickly and will see if I can get more boost in score. Thanks.
", "votes": 0, "replies": [] } ] }, { "id": 885384, "author_name": "Urvish", "author_url": "", "post_date": "2020-06-14T06:45:11.950000", "content": "I solved it finally. I realized how much silly the mistake was. The fix was that I had to define all the functions inside the for loop which iterates over each fold. For example, define the load model, create a dataset, train step etc inside the loop and it worked. Some useful resources were this and this
", "votes": 1, "replies": [] }, { "id": 885114, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-06-13T22:25:08.047000", "content": "Beware of 5GB limit for output. You can save only 2 models as one is 2+gb
", "votes": 1, "replies": [ { "id": 885261, "author_name": "Urvish", "author_url": "", "post_date": "2020-06-14T04:17:52.097000", "content": "I do not want to save all models, at this stage I need better predictions only so that I can later make ensemble out of it.
", "votes": 0, "replies": [] } ] }, { "id": 884509, "author_name": "", "author_url": "", "post_date": "2020-06-13T12:25:28.017000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "885144": "My dumb fix is to have K-kernels for K-folds (as a last resort :D )", "885119": "Just simply don't do this, as this causes the problem of the deleted variable.\n```\ntf.tpu.experimental.initialize_tpu_system(tpu)\n```\nthis is a brute force way to fix issues with Keras taking and leaving around too much memory.\n\nWith my kernel you can have repeated train/predict calls (10+) without any memory issue.", "885384": "I solved it finally. I realized how much silly the mistake was. The fix was that I had to define all the functions inside the for loop which iterates over each fold. For example, define the load model, create a dataset, train step etc inside the loop and it worked. Some useful resources were [this](https://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu#Optimized-custom-training-loop) and [this](https://www.kaggle.com/dimitreoliveira/flower-with-tpus-k-fold-optimized-training-loops)", "885114": "Beware of 5GB limit for output. You can save only 2 models as one is 2+gb", "884497": "I am trying to use [this](https://www.kaggle.com/riblidezso/train-from-mlm-finetuned-xlm-roberta-large) public kernel and modify it to do k-fold training on TPU. But I am getting the following error after first fold finishes. Thanks for the kernel @riblidezso .\n\n```\nNotFoundError: {{function_node __inference_distributed_train_step_139198}} Resource worker/Adam/iter/replica_6_57590/N10tensorflow3VarE does not exist.\n\t [[{{node cluster_distributed_train_step/_execute_6_0}}]]\n```\nI made changes like calling `tf.tpu.experimental.initialize_tpu_system(tpu)` before each of the fold. I also added `drop_reminder=True` where needed. But I am getting above error when starting the second fold. Not sure why yet. Seems like there is an issue in creating a distributed dataset in the second fold. So how can I fix this? I am not able to paste the full traceback somehow here not sure why. ", "884509": "" } }