Hello friends!
\nI'm currently trying to add my pipeline to the best public one, but my notebook just crashes without any errors on the T4x2 GPU. I use tensorflow and at the model.predict stage the notebook just flies out. What's interesting: I can build a maximum of two of my models and with conveyors for the third, the problem repeats itself, individual models work well.
\nHas anyone experienced this, I can't post my code right now for obvious reasons.
Thank you.
", "messages": [ { "id": "2728186", "postDate": "04/02/2024 06:12:05", "content": "Hello friends!
\nI'm currently trying to add my pipeline to the best public one, but my notebook just crashes without any errors on the T4x2 GPU. I use tensorflow and at the model.predict stage the notebook just flies out. What's interesting: I can build a maximum of two of my models and with conveyors for the third, the problem repeats itself, individual models work well.
\nHas anyone experienced this, I can't post my code right now for obvious reasons.
Thank you.
", "rawMarkdown": "Hello friends! \nI'm currently trying to add my pipeline to the best public one, but my notebook just crashes without any errors on the T4x2 GPU. I use tensorflow and at the model.predict stage the notebook just flies out. What's interesting: I can build a maximum of two of my models and with conveyors for the third, the problem repeats itself, individual models work well. \nHas anyone experienced this, I can't post my code right now for obvious reasons.\n\nThank you.", "votes": null }, { "id": "2728392", "postDate": "04/02/2024 08:05:37", "content": "maybe you could try the following code, to make sure that they are running completely individual.
\n!python /kaggle/input/infer1.py
\n!python /kaggle/input/infer2.py
\n!python /kaggle/input/infer3.py
Good idea! I will try
", "rawMarkdown": "Good idea! I will try", "votes": null }, { "id": "2730414", "postDate": "04/02/2024 17:44:33", "content": "flying out - almost always needs a smaller batch size cause of memory exceeding limit for me.
", "rawMarkdown": "flying out - almost always needs a smaller batch size cause of memory exceeding limit for me.", "votes": null }, { "id": "2731418", "postDate": "04/02/2024 17:54:04", "content": "Yes, but not in this case, we are talking about one test sample, the size of the previous models is small and I collect garbage after each predict. I think the problem is something else.
", "rawMarkdown": "Yes, but not in this case, we are talking about one test sample, the size of the previous models is small and I collect garbage after each predict. I think the problem is something else.", "votes": null }, { "id": "2732166", "postDate": "04/03/2024 03:05:41", "content": "I had a similar issue. Don't know exactly what the problem was but I fixed it by creating predictions from my pipeline and then deleting all variables in my code except for predictions array and then running public notebook code after that. Hope it works for you
", "rawMarkdown": "I had a similar issue. Don't know exactly what the problem was but I fixed it by creating predictions from my pipeline and then deleting all variables in my code except for predictions array and then running public notebook code after that. Hope it works for you", "votes": null }, { "id": "2732382", "postDate": "04/03/2024 05:33:00", "content": "Thank you, the problem was that in order to save resources, I used create_spectrogram_with_cusignal when preparing the dataset for tensorflows, if I first process the data, let's say in the dictionary, then everything works, but then I didn't have enough memory, I bypassed it according to the advice above
", "rawMarkdown": "Thank you, the problem was that in order to save resources, I used create_spectrogram_with_cusignal when preparing the dataset for tensorflows, if I first process the data, let's say in the dictionary, then everything works, but then I didn't have enough memory, I bypassed it according to the advice above", "votes": null }, { "id": "2733269", "postDate": "04/03/2024 15:04:03", "content": "Thank you, it solved our problem.
", "rawMarkdown": "Thank you, it solved our problem.", "votes": null } ], "comments": [ { "id": 2728392, "author_name": "jimmyisme1", "author_url": "", "post_date": "04/02/2024 08:05:37", "content": "maybe you could try the following code, to make sure that they are running completely individual.
\n!python /kaggle/input/infer1.py
\n!python /kaggle/input/infer2.py
\n!python /kaggle/input/infer3.py
Good idea! I will try
", "votes": null, "replies": [] }, { "id": 2733269, "author_name": "yunxiaoliemory", "author_url": "", "post_date": "04/03/2024 15:04:03", "content": "Thank you, it solved our problem.
", "votes": null, "replies": [] } ] }, { "id": 2730414, "author_name": "pcjimmmy", "author_url": "", "post_date": "04/02/2024 17:44:33", "content": "flying out - almost always needs a smaller batch size cause of memory exceeding limit for me.
", "votes": null, "replies": [ { "id": 2731418, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "04/02/2024 17:54:04", "content": "Yes, but not in this case, we are talking about one test sample, the size of the previous models is small and I collect garbage after each predict. I think the problem is something else.
", "votes": null, "replies": [] } ] }, { "id": 2732166, "author_name": "snehalverma10", "author_url": "", "post_date": "04/03/2024 03:05:41", "content": "I had a similar issue. Don't know exactly what the problem was but I fixed it by creating predictions from my pipeline and then deleting all variables in my code except for predictions array and then running public notebook code after that. Hope it works for you
", "votes": null, "replies": [ { "id": 2732382, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "04/03/2024 05:33:00", "content": "Thank you, the problem was that in order to save resources, I used create_spectrogram_with_cusignal when preparing the dataset for tensorflows, if I first process the data, let's say in the dictionary, then everything works, but then I didn't have enough memory, I bypassed it according to the advice above
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2728186": "Hello friends! \nI'm currently trying to add my pipeline to the best public one, but my notebook just crashes without any errors on the T4x2 GPU. I use tensorflow and at the model.predict stage the notebook just flies out. What's interesting: I can build a maximum of two of my models and with conveyors for the third, the problem repeats itself, individual models work well. \nHas anyone experienced this, I can't post my code right now for obvious reasons.\n\nThank you.", "2728392": "maybe you could try the following code, to make sure that they are running completely individual.\n!python /kaggle/input/infer1.py\n!python /kaggle/input/infer2.py\n!python /kaggle/input/infer3.py", "2728409": "Good idea! I will try", "2730414": "flying out - almost always needs a smaller batch size cause of memory exceeding limit for me.", "2731418": "Yes, but not in this case, we are talking about one test sample, the size of the previous models is small and I collect garbage after each predict. I think the problem is something else.", "2732166": "I had a similar issue. Don't know exactly what the problem was but I fixed it by creating predictions from my pipeline and then deleting all variables in my code except for predictions array and then running public notebook code after that. Hope it works for you", "2732382": "Thank you, the problem was that in order to save resources, I used create_spectrogram_with_cusignal when preparing the dataset for tensorflows, if I first process the data, let's say in the dictionary, then everything works, but then I didn't have enough memory, I bypassed it according to the advice above", "2733269": "Thank you, it solved our problem." }, "source": "meta" }