Hello all!
\n\nThis is not my first rodeo with OOM problems...
\n\nKernel is available here.
\n\nIf I keep everything as it is (batch sizes etc) the model finishes the one epoch and the kernel restarts. I tried dividing the steps per epoch by 2, to debug, but it still resets. My training seems to be blowing up the memory. I'm using a TFRecord input pipeline.
\n\nWhat should we do in these cases? I even tried two networks to see if the problem was the size of the network.
", "messages": [ { "id": "936092", "postDate": "07/20/2020 01:05:40", "content": "Hello all!
\n\nThis is not my first rodeo with OOM problems...
\n\nKernel is available here.
\n\nIf I keep everything as it is (batch sizes etc) the model finishes the one epoch and the kernel restarts. I tried dividing the steps per epoch by 2, to debug, but it still resets. My training seems to be blowing up the memory. I'm using a TFRecord input pipeline.
\n\nWhat should we do in these cases? I even tried two networks to see if the problem was the size of the network.
", "rawMarkdown": "Hello all!\n\nThis is not my first rodeo with OOM problems...\n\nKernel is available [here](https://www.kaggle.com/amneves/mobilenet-inception-v3-transfer-learning).\n\nIf I keep everything as it is (batch sizes etc) the model finishes the one epoch and the kernel restarts. I tried dividing the steps per epoch by 2, to debug, but it still resets. My training seems to be blowing up the memory. I'm using a TFRecord input pipeline. \n\nWhat should we do in these cases? I even tried two networks to see if the problem was the size of the network.", "votes": null }, { "id": "936157", "postDate": "07/20/2020 03:01:14", "content": "I would say try reducing the batch size to a really small number (maybe 4 or 8) and see if it trains properly. In most situations having a lower batch size will fix your problem
", "rawMarkdown": "I would say try reducing the batch size to a really small number (maybe 4 or 8) and see if it trains properly. In most situations having a lower batch size will fix your problem", "votes": null }, { "id": "936213", "postDate": "07/20/2020 04:37:58", "content": "Seems like you are resizing the images in the input pipeline.You shouldn't do that for 2 reasons\n1) Your data loading becomes very slow as it have resize every images and then feed into the network\n2) Maybe when resizing every image concurrently in GPU your GPU VRAM gets blown up before training\nSolution:\n1) Before feeding into the network use already resized images
", "rawMarkdown": "Seems like you are resizing the images in the input pipeline.You shouldn't do that for 2 reasons\n1) Your data loading becomes very slow as it have resize every images and then feed into the network\n2) Maybe when resizing every image concurrently in GPU your GPU VRAM gets blown up before training\nSolution:\n1) Before feeding into the network use already resized images", "votes": null }, { "id": "936269", "postDate": "07/20/2020 05:42:23", "content": "I see the same situation you describe on my local Ubuntu system when I hit borderline memory for tensorflow models with this data set.
\n\nI am resizing in the input pipeline in the same fashion as your code. (I played with elimination of the pipeline resize and it gave me such a small improvement that I felt it was in the measurement error band rather than real - but that's on a system with a lot more punch than a kaggle kernel provides.)
\n\nI play with batch size to get fastest possible execution of the code for a given image size. OOM errors are of course the clue that I made the batch too big.
\n\nOn occasion I can get a batch/image size that works for training but kills the PC after completion of the first epoch. When I reboot the 1st epoch is done and the tensor board file was created for both train and validation.
\n\nThis occurs most often when I am trying large image sizes - like 512 and I have needed to reduce batch size to 4 or less per GPU. My assumption is that I tweaked the memory usage to the border and something in the very last part of the epoch takes the memory ugly. Not sure why I don't get an OOM message in these cases, instead it kills the PC - only a machine reboot can bring it back. I have a big RAM memory of 64GB and another 200GB swap so assuming that the issue is in the GPU memory.
\n\nI also can get just normal kernel death in these situations play with sizes, but the kernel deaths seem to occur almost always before the end of the epoch and no tensor board file for the validation generated.
\n\nI run 4 machines that vary based on time I built them and all four have hit the ugly spot. But the fastest and biggest is the most frequent machine only because that is the one I most often use for large image sizes.
\n\nFixes are (assuming I don't want to change my model)\n1. Reduce batch size.\n2. Reduce the image size.
\n\nI have a model version that I will likely run tomorrow with large image size and will try MhdSharuk suggestion and not resize in pipeline IF I get a crash.
", "rawMarkdown": "I see the same situation you describe on my local Ubuntu system when I hit borderline memory for tensorflow models with this data set. \n\nI am resizing in the input pipeline in the same fashion as your code. (I played with elimination of the pipeline resize and it gave me such a small improvement that I felt it was in the measurement error band rather than real - but that's on a system with a lot more punch than a kaggle kernel provides.)\n\nI play with batch size to get fastest possible execution of the code for a given image size. OOM errors are of course the clue that I made the batch too big. \n\nOn occasion I can get a batch/image size that works for training but kills the PC after completion of the first epoch. When I reboot the 1st epoch is done and the tensor board file was created for both train and validation. \n\nThis occurs most often when I am trying large image sizes - like 512 and I have needed to reduce batch size to 4 or less per GPU. My assumption is that I tweaked the memory usage to the border and something in the very last part of the epoch takes the memory ugly. Not sure why I don't get an OOM message in these cases, instead it kills the PC - only a machine reboot can bring it back. I have a big RAM memory of 64GB and another 200GB swap so assuming that the issue is in the GPU memory. \n\nI also can get just normal kernel death in these situations play with sizes, but the kernel deaths seem to occur almost always before the end of the epoch and no tensor board file for the validation generated.\n\nI run 4 machines that vary based on time I built them and all four have hit the ugly spot. But the fastest and biggest is the most frequent machine only because that is the one I most often use for large image sizes.\n\nFixes are (assuming I don't want to change my model)\n1. Reduce batch size.\n2. Reduce the image size.\n\nI have a model version that I will likely run tomorrow with large image size and will try MhdSharuk suggestion and not resize in pipeline IF I get a crash.", "votes": null }, { "id": "936520", "postDate": "07/20/2020 09:38:16", "content": "Are there any datasets you recommend with already resized images? I thought this could be the problem too.
\n\nI'll probably just download the images and preprocess them locally, I might even just run everything locally!
", "rawMarkdown": "Are there any datasets you recommend with already resized images? I thought this could be the problem too.\n\nI'll probably just download the images and preprocess them locally, I might even just run everything locally!", "votes": null }, { "id": "937704", "postDate": "07/21/2020 06:10:43", "content": "Got to running my model with large image size on my local system. It crashed and died near the end of epoch 1.
\n\nContrary to my belief that I had enough RAM - 64 in chips and 200 in an ssd swap file - the issue was RAM memory. Death occurs as the system attempts to put stuff in the swap file after the hardware chips are full. With a 200GB swap I NEVER have RAM memory crashes - until now. I assume tf is responsible for the bug that prevents a smooth swap - almost looks like tf assumed the more standard swap of 2GB vs 200gb.
\n\nObservation indicated that the RAM overload was occurring during the validation. Each batch appeared to be adding to the RAM.
\n\nThe Fix - thank goodness I found one for my system :)\n Here was my validation dataset def (once again Kaggle craps all over any attempt to make this look like code!!!!
\n\n`def get_validation_dataset(dataset):
\n\ndataset = dataset.batch(BATCH_SIZE)\n\ndataset = dataset.cache()\n\n# prefetch next batch while training (autotune prefetch buffer size)\n\ndataset = dataset.prefetch(AUTO)\n\nreturn dataset`\nRemoval of the dataset = dataset.cache() line is the winner. My model built upon Alex kernel - cache works with small images but appears to fill the bucket on larger.
\n\nLet me know if you have cache in your dataset build def.
", "rawMarkdown": "Got to running my model with large image size on my local system. It crashed and died near the end of epoch 1.\n\nContrary to my belief that I had enough RAM - 64 in chips and 200 in an ssd swap file - the issue was RAM memory. Death occurs as the system attempts to put stuff in the swap file after the hardware chips are full. With a 200GB swap I NEVER have RAM memory crashes - until now. I assume tf is responsible for the bug that prevents a smooth swap - almost looks like tf assumed the more standard swap of 2GB vs 200gb.\n\nObservation indicated that the RAM overload was occurring during the validation. Each batch appeared to be adding to the RAM.\n\nThe Fix - thank goodness I found one for my system :)\n Here was my validation dataset def (once again Kaggle craps all over any attempt to make this look like code!!!!\n\n`def get_validation_dataset(dataset):\n\n dataset = dataset.batch(BATCH_SIZE)\n\n dataset = dataset.cache()\n\n # prefetch next batch while training (autotune prefetch buffer size)\n\n dataset = dataset.prefetch(AUTO)\n\n return dataset`\n\nRemoval of the dataset = dataset.cache() line is the winner. My model built upon Alex [kernel](https://www.kaggle.com/graf10a/efficientnet-bn-tabular-features-tf-cv5-512x512) - cache works with small images but appears to fill the bucket on larger.\n\nLet me know if you have cache in your dataset build def.", "votes": null }, { "id": "937708", "postDate": "07/21/2020 06:12:00", "content": "Realized you linked your kernel. You do have cache line in your validation def.
", "rawMarkdown": "Realized you linked your kernel. You do have cache line in your validation def.", "votes": null }, { "id": "937725", "postDate": "07/21/2020 06:21:19", "content": "Most public notebooks are using the resized images here. You can download either JPEGs or TFRecords.
", "rawMarkdown": "Most public notebooks are using the resized images [here][1]. You can download either JPEGs or TFRecords.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092", "votes": null } ], "comments": [ { "id": 936157, "author_name": "aryanpandey1109", "author_url": "", "post_date": "07/20/2020 03:01:14", "content": "I would say try reducing the batch size to a really small number (maybe 4 or 8) and see if it trains properly. In most situations having a lower batch size will fix your problem
", "votes": null, "replies": [] }, { "id": 936213, "author_name": "msharuk589", "author_url": "", "post_date": "07/20/2020 04:37:58", "content": "Seems like you are resizing the images in the input pipeline.You shouldn't do that for 2 reasons\n1) Your data loading becomes very slow as it have resize every images and then feed into the network\n2) Maybe when resizing every image concurrently in GPU your GPU VRAM gets blown up before training\nSolution:\n1) Before feeding into the network use already resized images
", "votes": null, "replies": [ { "id": 936520, "author_name": "amneves", "author_url": "", "post_date": "07/20/2020 09:38:16", "content": "Are there any datasets you recommend with already resized images? I thought this could be the problem too.
\n\nI'll probably just download the images and preprocess them locally, I might even just run everything locally!
", "votes": null, "replies": [] }, { "id": 937725, "author_name": "cdeotte", "author_url": "", "post_date": "07/21/2020 06:21:19", "content": "Most public notebooks are using the resized images here. You can download either JPEGs or TFRecords.
", "votes": null, "replies": [] } ] }, { "id": 936269, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/20/2020 05:42:23", "content": "I see the same situation you describe on my local Ubuntu system when I hit borderline memory for tensorflow models with this data set.
\n\nI am resizing in the input pipeline in the same fashion as your code. (I played with elimination of the pipeline resize and it gave me such a small improvement that I felt it was in the measurement error band rather than real - but that's on a system with a lot more punch than a kaggle kernel provides.)
\n\nI play with batch size to get fastest possible execution of the code for a given image size. OOM errors are of course the clue that I made the batch too big.
\n\nOn occasion I can get a batch/image size that works for training but kills the PC after completion of the first epoch. When I reboot the 1st epoch is done and the tensor board file was created for both train and validation.
\n\nThis occurs most often when I am trying large image sizes - like 512 and I have needed to reduce batch size to 4 or less per GPU. My assumption is that I tweaked the memory usage to the border and something in the very last part of the epoch takes the memory ugly. Not sure why I don't get an OOM message in these cases, instead it kills the PC - only a machine reboot can bring it back. I have a big RAM memory of 64GB and another 200GB swap so assuming that the issue is in the GPU memory.
\n\nI also can get just normal kernel death in these situations play with sizes, but the kernel deaths seem to occur almost always before the end of the epoch and no tensor board file for the validation generated.
\n\nI run 4 machines that vary based on time I built them and all four have hit the ugly spot. But the fastest and biggest is the most frequent machine only because that is the one I most often use for large image sizes.
\n\nFixes are (assuming I don't want to change my model)\n1. Reduce batch size.\n2. Reduce the image size.
\n\nI have a model version that I will likely run tomorrow with large image size and will try MhdSharuk suggestion and not resize in pipeline IF I get a crash.
", "votes": null, "replies": [] }, { "id": 937704, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/21/2020 06:10:43", "content": "Got to running my model with large image size on my local system. It crashed and died near the end of epoch 1.
\n\nContrary to my belief that I had enough RAM - 64 in chips and 200 in an ssd swap file - the issue was RAM memory. Death occurs as the system attempts to put stuff in the swap file after the hardware chips are full. With a 200GB swap I NEVER have RAM memory crashes - until now. I assume tf is responsible for the bug that prevents a smooth swap - almost looks like tf assumed the more standard swap of 2GB vs 200gb.
\n\nObservation indicated that the RAM overload was occurring during the validation. Each batch appeared to be adding to the RAM.
\n\nThe Fix - thank goodness I found one for my system :)\n Here was my validation dataset def (once again Kaggle craps all over any attempt to make this look like code!!!!
\n\n`def get_validation_dataset(dataset):
\n\ndataset = dataset.batch(BATCH_SIZE)\n\ndataset = dataset.cache()\n\n# prefetch next batch while training (autotune prefetch buffer size)\n\ndataset = dataset.prefetch(AUTO)\n\nreturn dataset`\nRemoval of the dataset = dataset.cache() line is the winner. My model built upon Alex kernel - cache works with small images but appears to fill the bucket on larger.
\n\nLet me know if you have cache in your dataset build def.
", "votes": null, "replies": [ { "id": 937708, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/21/2020 06:12:00", "content": "Realized you linked your kernel. You do have cache line in your validation def.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "936092": "Hello all!\n\nThis is not my first rodeo with OOM problems...\n\nKernel is available [here](https://www.kaggle.com/amneves/mobilenet-inception-v3-transfer-learning).\n\nIf I keep everything as it is (batch sizes etc) the model finishes the one epoch and the kernel restarts. I tried dividing the steps per epoch by 2, to debug, but it still resets. My training seems to be blowing up the memory. I'm using a TFRecord input pipeline. \n\nWhat should we do in these cases? I even tried two networks to see if the problem was the size of the network.", "936157": "I would say try reducing the batch size to a really small number (maybe 4 or 8) and see if it trains properly. In most situations having a lower batch size will fix your problem", "936213": "Seems like you are resizing the images in the input pipeline.You shouldn't do that for 2 reasons\n1) Your data loading becomes very slow as it have resize every images and then feed into the network\n2) Maybe when resizing every image concurrently in GPU your GPU VRAM gets blown up before training\nSolution:\n1) Before feeding into the network use already resized images", "936269": "I see the same situation you describe on my local Ubuntu system when I hit borderline memory for tensorflow models with this data set. \n\nI am resizing in the input pipeline in the same fashion as your code. (I played with elimination of the pipeline resize and it gave me such a small improvement that I felt it was in the measurement error band rather than real - but that's on a system with a lot more punch than a kaggle kernel provides.)\n\nI play with batch size to get fastest possible execution of the code for a given image size. OOM errors are of course the clue that I made the batch too big. \n\nOn occasion I can get a batch/image size that works for training but kills the PC after completion of the first epoch. When I reboot the 1st epoch is done and the tensor board file was created for both train and validation. \n\nThis occurs most often when I am trying large image sizes - like 512 and I have needed to reduce batch size to 4 or less per GPU. My assumption is that I tweaked the memory usage to the border and something in the very last part of the epoch takes the memory ugly. Not sure why I don't get an OOM message in these cases, instead it kills the PC - only a machine reboot can bring it back. I have a big RAM memory of 64GB and another 200GB swap so assuming that the issue is in the GPU memory. \n\nI also can get just normal kernel death in these situations play with sizes, but the kernel deaths seem to occur almost always before the end of the epoch and no tensor board file for the validation generated.\n\nI run 4 machines that vary based on time I built them and all four have hit the ugly spot. But the fastest and biggest is the most frequent machine only because that is the one I most often use for large image sizes.\n\nFixes are (assuming I don't want to change my model)\n1. Reduce batch size.\n2. Reduce the image size.\n\nI have a model version that I will likely run tomorrow with large image size and will try MhdSharuk suggestion and not resize in pipeline IF I get a crash.", "936520": "Are there any datasets you recommend with already resized images? I thought this could be the problem too.\n\nI'll probably just download the images and preprocess them locally, I might even just run everything locally!", "937704": "Got to running my model with large image size on my local system. It crashed and died near the end of epoch 1.\n\nContrary to my belief that I had enough RAM - 64 in chips and 200 in an ssd swap file - the issue was RAM memory. Death occurs as the system attempts to put stuff in the swap file after the hardware chips are full. With a 200GB swap I NEVER have RAM memory crashes - until now. I assume tf is responsible for the bug that prevents a smooth swap - almost looks like tf assumed the more standard swap of 2GB vs 200gb.\n\nObservation indicated that the RAM overload was occurring during the validation. Each batch appeared to be adding to the RAM.\n\nThe Fix - thank goodness I found one for my system :)\n Here was my validation dataset def (once again Kaggle craps all over any attempt to make this look like code!!!!\n\n`def get_validation_dataset(dataset):\n\n dataset = dataset.batch(BATCH_SIZE)\n\n dataset = dataset.cache()\n\n # prefetch next batch while training (autotune prefetch buffer size)\n\n dataset = dataset.prefetch(AUTO)\n\n return dataset`\n\nRemoval of the dataset = dataset.cache() line is the winner. My model built upon Alex [kernel](https://www.kaggle.com/graf10a/efficientnet-bn-tabular-features-tf-cv5-512x512) - cache works with small images but appears to fill the bucket on larger.\n\nLet me know if you have cache in your dataset build def.", "937708": "Realized you linked your kernel. You do have cache line in your validation def.", "937725": "Most public notebooks are using the resized images [here][1]. You can download either JPEGs or TFRecords.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092" }, "source": "meta" }