I would like to combine models that use spectrograms as seen in some notebooks and the wavenet model. Whenever I try this however I exceed the maximum amount of ram that I can use. This due to the fact that both models use different kinds of data and thus both need to be loaded in one notebook. To counterfeit these issues I use GC.collect() and %reset -f. However even after using these tricks the model isn't able to compile. Is there anyone who made this mix possible, or which other tricks can you use besides GC.collect() and %reset -f?
", "messages": [ { "id": "2660638", "postDate": "02/20/2024 18:22:18", "content": "I would like to combine models that use spectrograms as seen in some notebooks and the wavenet model. Whenever I try this however I exceed the maximum amount of ram that I can use. This due to the fact that both models use different kinds of data and thus both need to be loaded in one notebook. To counterfeit these issues I use GC.collect() and %reset -f. However even after using these tricks the model isn't able to compile. Is there anyone who made this mix possible, or which other tricks can you use besides GC.collect() and %reset -f?
", "rawMarkdown": "I would like to combine models that use spectrograms as seen in some notebooks and the wavenet model. Whenever I try this however I exceed the maximum amount of ram that I can use. This due to the fact that both models use different kinds of data and thus both need to be loaded in one notebook. To counterfeit these issues I use GC.collect() and %reset -f. However even after using these tricks the model isn't able to compile. Is there anyone who made this mix possible, or which other tricks can you use besides GC.collect() and %reset -f?", "votes": null }, { "id": "2660677", "postDate": "02/20/2024 18:55:25", "content": "\"This due to the fact that both models use different kinds of data\"
\nYou mean eegs and spectograms? There is a notebook that uses both of them:
\nhttps://www.kaggle.com/code/nischaydnk/training-multimodal-1d-2d-approach-eegs
I'll have a look I'm using TensorFlow instead of Pytorch Lightning. I can investigate further if it happens before or after the declaration of the dataloaders. Although I think it is the model.
", "rawMarkdown": "I'll have a look I'm using TensorFlow instead of Pytorch Lightning. I can investigate further if it happens before or after the declaration of the dataloaders. Although I think it is the model.", "votes": null }, { "id": "2660845", "postDate": "02/20/2024 20:39:43", "content": "Thank you so much. I have been looking for such a notebook that combines two model.
", "rawMarkdown": "Thank you so much. I have been looking for such a notebook that combines two model.", "votes": null }, { "id": "2660986", "postDate": "02/20/2024 23:32:22", "content": "I recommend that you learn how to use TPU. Especially if you are using TensorFlow, the transition should be seamless. Once you train in TPU, all your memory problems will be solved (and the training will be a lot faster, too)
", "rawMarkdown": "I recommend that you learn how to use TPU. Especially if you are using TensorFlow, the transition should be seamless. Once you train in TPU, all your memory problems will be solved (and the training will be a lot faster, too)", "votes": null }, { "id": "2661379", "postDate": "02/21/2024 07:44:37", "content": "Can you use the TPU in this competition? Because indeed this would solve my problem.
", "rawMarkdown": "Can you use the TPU in this competition? Because indeed this would solve my problem.", "votes": null }, { "id": "2661613", "postDate": "02/21/2024 11:24:36", "content": "Use TPU to train and GPU for inference
", "rawMarkdown": "Use TPU to train and GPU for inference", "votes": null }, { "id": "2661828", "postDate": "02/21/2024 14:28:19", "content": "I also train a combination of specs and EEG signals. Try to incorporate gradient accumulation into the training process, I know that PyTorch Lightning offers this functionality, and also try to load each sample individually on your dataloader instead of loading all into the memory.
", "rawMarkdown": "I also train a combination of specs and EEG signals. Try to incorporate gradient accumulation into the training process, I know that PyTorch Lightning offers this functionality, and also try to load each sample individually on your dataloader instead of loading all into the memory.", "votes": null }, { "id": "2664074", "postDate": "02/22/2024 19:13:47", "content": "I'm also having the same problem.
", "rawMarkdown": "I'm also having the same problem.", "votes": null }, { "id": "2664077", "postDate": "02/22/2024 19:14:54", "content": "This notebook also run out of memory
", "rawMarkdown": "This notebook also run out of memory", "votes": null }, { "id": "2665925", "postDate": "02/24/2024 01:36:09", "content": "If I get a high rank in this competition I'll just look for someone very good at memory management.
", "rawMarkdown": "If I get a high rank in this competition I'll just look for someone very good at memory management.", "votes": null }, { "id": "2665926", "postDate": "02/24/2024 01:36:50", "content": "Currently using tensorflow but yes indeed, I should have a look into pl lightning.
", "rawMarkdown": "Currently using tensorflow but yes indeed, I should have a look into pl lightning.", "votes": null }, { "id": "2665944", "postDate": "02/24/2024 02:12:47", "content": "hi, what's the result of trying this notebook? I've been trying for a long time but the LB is only 0.41
", "rawMarkdown": "hi, what's the result of trying this notebook? I've been trying for a long time but the LB is only 0.41", "votes": null } ], "comments": [ { "id": 2660677, "author_name": "sacuscreed", "author_url": "", "post_date": "02/20/2024 18:55:25", "content": "\"This due to the fact that both models use different kinds of data\"
\nYou mean eegs and spectograms? There is a notebook that uses both of them:
\nhttps://www.kaggle.com/code/nischaydnk/training-multimodal-1d-2d-approach-eegs
I'll have a look I'm using TensorFlow instead of Pytorch Lightning. I can investigate further if it happens before or after the declaration of the dataloaders. Although I think it is the model.
", "votes": null, "replies": [] }, { "id": 2660845, "author_name": "minhsienweng", "author_url": "", "post_date": "02/20/2024 20:39:43", "content": "Thank you so much. I have been looking for such a notebook that combines two model.
", "votes": null, "replies": [ { "id": 2665944, "author_name": "chenboluo", "author_url": "", "post_date": "02/24/2024 02:12:47", "content": "hi, what's the result of trying this notebook? I've been trying for a long time but the LB is only 0.41
", "votes": null, "replies": [] } ] }, { "id": 2664077, "author_name": "yantxx", "author_url": "", "post_date": "02/22/2024 19:14:54", "content": "This notebook also run out of memory
", "votes": null, "replies": [] } ] }, { "id": 2660986, "author_name": "shlomoron", "author_url": "", "post_date": "02/20/2024 23:32:22", "content": "I recommend that you learn how to use TPU. Especially if you are using TensorFlow, the transition should be seamless. Once you train in TPU, all your memory problems will be solved (and the training will be a lot faster, too)
", "votes": null, "replies": [ { "id": 2661379, "author_name": "stefanoclss", "author_url": "", "post_date": "02/21/2024 07:44:37", "content": "Can you use the TPU in this competition? Because indeed this would solve my problem.
", "votes": null, "replies": [ { "id": 2661613, "author_name": "shlomoron", "author_url": "", "post_date": "02/21/2024 11:24:36", "content": "Use TPU to train and GPU for inference
", "votes": null, "replies": [] } ] } ] }, { "id": 2661828, "author_name": "eliork", "author_url": "", "post_date": "02/21/2024 14:28:19", "content": "I also train a combination of specs and EEG signals. Try to incorporate gradient accumulation into the training process, I know that PyTorch Lightning offers this functionality, and also try to load each sample individually on your dataloader instead of loading all into the memory.
", "votes": null, "replies": [ { "id": 2665926, "author_name": "stefanoclss", "author_url": "", "post_date": "02/24/2024 01:36:50", "content": "Currently using tensorflow but yes indeed, I should have a look into pl lightning.
", "votes": null, "replies": [] } ] }, { "id": 2664074, "author_name": "yantxx", "author_url": "", "post_date": "02/22/2024 19:13:47", "content": "I'm also having the same problem.
", "votes": null, "replies": [ { "id": 2665925, "author_name": "stefanoclss", "author_url": "", "post_date": "02/24/2024 01:36:09", "content": "If I get a high rank in this competition I'll just look for someone very good at memory management.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2660638": "I would like to combine models that use spectrograms as seen in some notebooks and the wavenet model. Whenever I try this however I exceed the maximum amount of ram that I can use. This due to the fact that both models use different kinds of data and thus both need to be loaded in one notebook. To counterfeit these issues I use GC.collect() and %reset -f. However even after using these tricks the model isn't able to compile. Is there anyone who made this mix possible, or which other tricks can you use besides GC.collect() and %reset -f?", "2660677": "\"This due to the fact that both models use different kinds of data\"\nYou mean eegs and spectograms? There is a notebook that uses both of them:\nhttps://www.kaggle.com/code/nischaydnk/training-multimodal-1d-2d-approach-eegs", "2660699": "I'll have a look I'm using TensorFlow instead of Pytorch Lightning. I can investigate further if it happens before or after the declaration of the dataloaders. Although I think it is the model.", "2660845": "Thank you so much. I have been looking for such a notebook that combines two model.", "2660986": "I recommend that you learn how to use TPU. Especially if you are using TensorFlow, the transition should be seamless. Once you train in TPU, all your memory problems will be solved (and the training will be a lot faster, too)", "2661379": "Can you use the TPU in this competition? Because indeed this would solve my problem.", "2661613": "Use TPU to train and GPU for inference", "2661828": "I also train a combination of specs and EEG signals. Try to incorporate gradient accumulation into the training process, I know that PyTorch Lightning offers this functionality, and also try to load each sample individually on your dataloader instead of loading all into the memory.", "2664074": "I'm also having the same problem.", "2664077": "This notebook also run out of memory", "2665925": "If I get a high rank in this competition I'll just look for someone very good at memory management.", "2665926": "Currently using tensorflow but yes indeed, I should have a look into pl lightning.", "2665944": "hi, what's the result of trying this notebook? I've been trying for a long time but the LB is only 0.41" }, "source": "meta" }