![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2F7c2b9f0aaa0b042e6f04a6aae13ead38%2F_2024-12-29_171636_504.png?generation=1735463813123187&alt=media\")
Tensorflow taking a lot of GPU RAM while torch only take a little?
\nI tried to load only 2 day of data by tensorflow:
\n ():\n data = pl.scan_parquet(path).select(\n pl.(),).(\n pl.col().gt(start_dt),\n pl.col().le(end_dt),\n ).fill_null().fill_null()\n\n data = data.collect().to_pandas()\n\n data.replace([np.inf, -np.inf], , inplace=)\n data\nX_train = load_data(training_resp_lag_path, start_dt=, end_dt=)\n
and after running tensorflow:
\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2F3a93bc58373e85f56f2bba8b8c3c9220%2Fwechat_2024-12-29_171842_716.png?generation=1735463938077122&alt=media\")
only two day of data already taking 14 gig data? how is that. and if I load more days, like 200
\nX_train = load_data(training_resp_lag_path, start_dt=, end_dt=)\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2Fdb6e7436adefff1cf156a8ce1a1d7746%2Fwechat_2024-12-29_172010_446.png?generation=1735464033383761&alt=media\")
the memory usage did not seem to change much.
\nbut if I use even more like from 600-1690, it will blow up the GPU RAM.
X_train = load_data(training_resp_lag_path, start_dt=, end_dt=)\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n\nW external/local_xla/xla/tsl/framework/bfc_allocator.cc:] Allocator (GPU_0_bfc) ran out of memory trying to allocate GiB (rounded to )requested by op _EagerConst\nIf the cause memory fragmentation maybe the environment variable will improve the situation. ...\nso this does really limit my ability to use tensorflow to train model. Batching the datasize did not really help with gpu memory, using batch size =1000 or batch size =8000 did not seemed to make much difference, but how much data I was loading was really matters. How to solve this?
\nwhile using torch, refering to the public code that got 0.0077:
\nhttps://www.kaggle.com/code/voix97/jane-street-rmf-training-nn
while training with this code, the GPU usage was only 0.3/15 gig space.
\nhow to config tensorflow to not taking so much of space?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2Fa337c65052a14a3ea0ff0c16e90a37fd%2Fwechat_2024-12-29_173649_987.png?generation=1735465073974414&alt=media\")
Tensorflow taking a lot of GPU RAM while torch only take a little?
\nI tried to load only 2 day of data by tensorflow:
\n ():\n data = pl.scan_parquet(path).select(\n pl.(),).(\n pl.col().gt(start_dt),\n pl.col().le(end_dt),\n ).fill_null().fill_null()\n\n data = data.collect().to_pandas()\n\n data.replace([np.inf, -np.inf], , inplace=)\n data\nX_train = load_data(training_resp_lag_path, start_dt=, end_dt=)\n
and after running tensorflow:
\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n
only two day of data already taking 14 gig data? how is that. and if I load more days, like 200
\nX_train = load_data(training_resp_lag_path, start_dt=, end_dt=)\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n
the memory usage did not seem to change much.
\nbut if I use even more like from 600-1690, it will blow up the GPU RAM.
X_train = load_data(training_resp_lag_path, start_dt=, end_dt=)\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n\nW external/local_xla/xla/tsl/framework/bfc_allocator.cc:] Allocator (GPU_0_bfc) ran out of memory trying to allocate GiB (rounded to )requested by op _EagerConst\nIf the cause memory fragmentation maybe the environment variable will improve the situation. ...\nso this does really limit my ability to use tensorflow to train model. Batching the datasize did not really help with gpu memory, using batch size =1000 or batch size =8000 did not seemed to make much difference, but how much data I was loading was really matters. How to solve this?
\nwhile using torch, refering to the public code that got 0.0077:
\nhttps://www.kaggle.com/code/voix97/jane-street-rmf-training-nn
while training with this code, the GPU usage was only 0.3/15 gig space.
\nhow to config tensorflow to not taking so much of space?
\n
Tensorflow always allocates the maximum available memory on the GPU. There are probably ways to force it to allocate only a part, but the total allocation is the usual behavior.
\nWhat is strange here is that test = tf.data.Dataset.from_tensor_slices(X_train) tried to save the data on the GPU. The behavior I am familiar with is saving the dataset on the system RAM. Only the model itself is supposed to be saved on the GPU RAM. Maybe you forced it somehow to save the data on the GPU?
Specify the device when creating tensor slices and model fitting
\nwith tf():\n train_dataset = tf(\n (features_batch, labels_batch, weights_batch)\n )\n.....\n\nwith tf():\n model(\n train_dataset,\n epochs=epochs,\n validation_data=valid_dataset,\n callbacks=,\n )\nmy question is, using tensorflow cannot really control how many gpu is using and easy to blow the VRAM, how to use a large dataset while not blow up the gpu ram?
", "rawMarkdown": "my question is, using tensorflow cannot really control how many gpu is using and easy to blow the VRAM, how to use a large dataset while not blow up the gpu ram?", "votes": null }, { "id": "3083465", "postDate": "12/29/2024 14:16:14", "content": "Per Tensorflow documentation, would this help?
\n\n", "rawMarkdown": "Per Tensorflow documentation, would this help?\n> If memory growth is enabled for a PhysicalDevice, the runtime initialization will not allocate all memory on the device.\n\n`physical_devices = tf.config.list_physical_devices('GPU')\ntry:\n tf.config.experimental.set_memory_growth(physical_devices[0], True)\nexcept:\n # Invalid device or cannot modify virtual devices once initialized.\n pass`", "votes": null } ], "comments": [ { "id": 3083373, "author_name": "shlomoron", "author_url": "", "post_date": "12/29/2024 11:54:25", "content": "If memory growth is enabled for a PhysicalDevice, the runtime initialization will not allocate all memory on the device.
\n
\nphysical_devices = tf.config.list_physical_devices('GPU')\ntry:\n tf.config.experimental.set_memory_growth(physical_devices[0], True)\nexcept:\n # Invalid device or cannot modify virtual devices once initialized.\n pass
Tensorflow always allocates the maximum available memory on the GPU. There are probably ways to force it to allocate only a part, but the total allocation is the usual behavior.
\nWhat is strange here is that test = tf.data.Dataset.from_tensor_slices(X_train) tried to save the data on the GPU. The behavior I am familiar with is saving the dataset on the system RAM. Only the model itself is supposed to be saved on the GPU RAM. Maybe you forced it somehow to save the data on the GPU?
Specify the device when creating tensor slices and model fitting
\nwith tf():\n train_dataset = tf(\n (features_batch, labels_batch, weights_batch)\n )\n.....\n\nwith tf():\n model(\n train_dataset,\n epochs=epochs,\n validation_data=valid_dataset,\n callbacks=,\n )\nmy question is, using tensorflow cannot really control how many gpu is using and easy to blow the VRAM, how to use a large dataset while not blow up the gpu ram?
", "votes": null, "replies": [ { "id": 3083465, "author_name": "edmund870", "author_url": "", "post_date": "12/29/2024 14:16:14", "content": "Per Tensorflow documentation, would this help?
\n\n", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3083313": "Tensorflow taking a lot of GPU RAM while torch only take a little? \n\nI tried to load only 2 day of data by tensorflow:\n```python\ndef load_data(path, start_dt, end_dt):\n data = pl.scan_parquet(path).select(\n pl.all(),).filter(\n pl.col(\"date_id\").gt(start_dt),\n pl.col(\"date_id\").le(end_dt),\n ).fill_null(0).fill_null(0)\n\n data = data.collect().to_pandas()\n\n data.replace([np.inf, -np.inf], 0, inplace=True)\n return data\n```\n\n```python\nX_train = load_data(training_resp_lag_path, start_dt=1688, end_dt=1690)\n```\n\n\nand after running tensorflow:\n\n```python\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n\n```\n\n\n\n\nonly two day of data already taking 14 gig data? how is that. and if I load more days, like 200\n```python\nX_train = load_data(training_resp_lag_path, start_dt=1500, end_dt=1690)\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n```\n\n\n\n\nthe memory usage did not seem to change much.\nbut if I use even more like from 600-1690, it will blow up the GPU RAM.\n\n```python\nX_train = load_data(training_resp_lag_path, start_dt=600, end_dt=1690)\ntest = tf.data.Dataset.from_tensor_slices(X_train)\n\nW external/local_xla/xla/tsl/framework/bfc_allocator.cc:497] Allocator (GPU_0_bfc) ran out of memory trying to allocate 28.57GiB (rounded to 30673070336)requested by op _EagerConst\nIf the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. ...\n```\n\nso this does really limit my ability to use tensorflow to train model. Batching the datasize did not really help with gpu memory, using batch size =1000 or batch size =8000 did not seemed to make much difference, but how much data I was loading was really matters. How to solve this?\n\nwhile using torch, refering to the public code that got 0.0077:\nhttps://www.kaggle.com/code/voix97/jane-street-rmf-training-nn\n\nwhile training with this code, the GPU usage was only 0.3/15 gig space.\nhow to config tensorflow to not taking so much of space?\n", "3083373": "Tensorflow always allocates the maximum available memory on the GPU. There are probably ways to force it to allocate only a part, but the total allocation is the usual behavior. \nWhat is strange here is that test = tf.data.Dataset.from_tensor_slices(X_train) tried to save the data on the GPU. The behavior I am familiar with is saving the dataset on the system RAM. Only the model itself is supposed to be saved on the GPU RAM. Maybe you forced it somehow to save the data on the GPU?", "3083377": "Specify the device when creating tensor slices and model fitting\n\n```\nwith tf.device(\"/CPU:0\"):\n train_dataset = tf.data.Dataset.from_tensor_slices(\n (features_batch, labels_batch, weights_batch)\n )\n.....\n\nwith tf.device(\"/GPU:0\"):\n model.fit(\n train_dataset,\n epochs=epochs,\n validation_data=valid_dataset,\n callbacks=[callback],\n )\n```", "3083447": "my question is, using tensorflow cannot really control how many gpu is using and easy to blow the VRAM, how to use a large dataset while not blow up the gpu ram?", "3083465": "Per Tensorflow documentation, would this help?\n> If memory growth is enabled for a PhysicalDevice, the runtime initialization will not allocate all memory on the device.\n\n`physical_devices = tf.config.list_physical_devices('GPU')\ntry:\n tf.config.experimental.set_memory_growth(physical_devices[0], True)\nexcept:\n # Invalid device or cannot modify virtual devices once initialized.\n pass`" }, "source": "meta" }If memory growth is enabled for a PhysicalDevice, the runtime initialization will not allocate all memory on the device.
\n
\nphysical_devices = tf.config.list_physical_devices('GPU')\ntry:\n tf.config.experimental.set_memory_growth(physical_devices[0], True)\nexcept:\n # Invalid device or cannot modify virtual devices once initialized.\n pass