![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F85156%2Fab3e3d6a75959d59e7b43a6d19b35603%2F38qdsy.jpg?generation=1566586491283350&alt=media\")
We have recently had high demand for GPUs in Kaggle Notebooks. To better allocate the resources we have, we will be slightly limiting concurrent GPU sessions to allow 1 interactive GPU session and 2 commit sessions at one time. This change is effective immediately. We also anticipate adding user-level GPU runtime quotas to accounts within the coming weeks (but not before the close of this competition). Thanks for your understanding!
", "messages": [ { "id": 606578, "postDate": "2019-08-23T18:55:32.693Z", "content": "
We have to admit this was funny. In every joke is a sliver of truth...
\n\nI wanted to provide candor in this situation. What's happening here isn't just launching lots of image competitions and being surprised that we have high GPU use.
\n\nOur GPU situation:
\n\nThanks for reading and for the feedback. We promise to stay on top of these limits to be sure Kernels offers as much as it can offer.
", "rawMarkdown": "We have to admit this was funny. In every joke is a sliver of truth...\n\nI wanted to provide candor in this situation. What's happening here isn't just launching lots of image competitions and being surprised that we have high GPU use.\n\nOur GPU situation:\n\n - We have a budget for Kernels compute. Recent faster-than-expected growth has pushed us too close to this budget, with GPU use being the primary driver. Budget is the primary reason for this and forthcoming GPU quota changes.\n - We are trying to limit waste and abuse. Google's cloud abuse team has sophisticated low-level detection mechanisms to spot abuse, be we must supplement them with our own user-level guardrails to limit it. We’re also working on some plans to autoscale the kernels fleet more rapidly. Our current scaling abilities lag behind the real demand, so we are forced to pay for reserve instances in case of usage spikes.\n - The resources we provide may change over time. We know the effect changes have on competitions and are trying to be as steady as possible. We tried to do this here by waiting on launching quotas until this competition is over. Note that there is no right time to slot in sitewide changes; we always have something closing and something else launching.\n - In spite of the reduction, we believe access to several GPUs is a pretty fortunate thing. You are lucky to have access to them, and we are even luckier to be able to provide them gratis.\n - We’re working on some expanded plans for TPUs (subject to different budgetary constraints) that we hope will turbocharge our capacity.\n\nThanks for reading and for the feedback. We promise to stay on top of these limits to be sure Kernels offers as much as it can offer.\n", "votes": 26 }, { "id": 608566, "postDate": "2019-08-27T00:26:38.380Z", "content": "This should be posted across the platform, if it has not been already. It will at least provide more context on the situation to many Kagglers. :)
", "rawMarkdown": "This should be posted across the platform, if it has not been already. It will at least provide more context on the situation to many Kagglers. :)", "votes": 1 }, { "id": 608579, "postDate": "2019-08-27T01:01:05.370Z", "content": "One way to decrease GPU usage:\nI use only scripts, not notebooks.\nUntil now, selecting GPU in \"interactive\" session, I always implied using GPU for \"commit\" version. There is only one button for that. By default, \"interactive\" version gets GPU too and future \"commit\" has the same setting as \"interactive session\". I guess it came from notebook design. \"interactive\" session for creating and editing script is just editor from which I can \"submit\" script. It does not need GPU. \nDuring \"interactive\" session I mostly do small editing if script failed (even it worked locally, some errors like file names or paths names happen). I can do it without using GPU in \"interactive\" session. I only need to check that script started without errors. When I \"commit\" I thought my work is done and interactive session ended.\nNow I see that interactive session is still active. Maybe, it is not available for use by others for 9 hours (even Kaggle reports \"inactive\" state in 60 min). When \"commit\" is successful, I do \"submit\". If I close interactive version after I \"submit\", it is OK. If I close interactive session when \"commit\" session is running, \"commit\" session is being canceled too.\nI don't know whether it is an error from my side, but I tried to do so several times last days, and I did not have any success. So I need to wait until my \"commit\" is finished, start \"submit\", and only after that I can stop interactive session.\nIt means that for script kernels during \"commit\" session \"interactive\" session should be active which effectively means that we only can run one \"commit\" session and keep \"interactive\" session as balance.\nIt is double usage of GPU instances instead of one even for prediction and much more for training commit.
\n\nQuestions:\n1. Is it possible to have two buttons: use GPU for interactive session and for future \"commit\" or \"submit\"?\n2. Is it possible to stop \"interactive\" session automatically (without stopping \"'commit\" session) when \"commit\" or \"submit\" button is pressed? \nI think it will free a lot of \"interactive\" sessions with GPU hanging out.\nPlus, given the restriction \"one interactive session\", it will mean \"one commit session\".\n3. If after finishing python script GPU instance is not freed it would be good to do so. I start training script before going to sleep, for example, and hope that GPU instance will stop automatically after script is finished (not after 9 hours). Otherwise, it is a waste of resources (and our future quote).
", "rawMarkdown": "One way to decrease GPU usage:\nI use only scripts, not notebooks.\nUntil now, selecting GPU in \"interactive\" session, I always implied using GPU for \"commit\" version. There is only one button for that. By default, \"interactive\" version gets GPU too and future \"commit\" has the same setting as \"interactive session\". I guess it came from notebook design. \"interactive\" session for creating and editing script is just editor from which I can \"submit\" script. It does not need GPU. \nDuring \"interactive\" session I mostly do small editing if script failed (even it worked locally, some errors like file names or paths names happen). I can do it without using GPU in \"interactive\" session. I only need to check that script started without errors. When I \"commit\" I thought my work is done and interactive session ended.\nNow I see that interactive session is still active. Maybe, it is not available for use by others for 9 hours (even Kaggle reports \"inactive\" state in 60 min). When \"commit\" is successful, I do \"submit\". If I close interactive version after I \"submit\", it is OK. If I close interactive session when \"commit\" session is running, \"commit\" session is being canceled too.\nI don't know whether it is an error from my side, but I tried to do so several times last days, and I did not have any success. So I need to wait until my \"commit\" is finished, start \"submit\", and only after that I can stop interactive session.\nIt means that for script kernels during \"commit\" session \"interactive\" session should be active which effectively means that we only can run **one** \"commit\" session and keep \"interactive\" session as balance.\nIt is **double** usage of GPU instances instead of one even for prediction and much more for training commit.\n\n**Questions**:\n1. Is it possible to have two buttons: use GPU for interactive session and for future \"commit\" or \"submit\"?\n2. Is it possible to stop \"interactive\" session automatically (without stopping \"'commit\" session) when \"commit\" or \"submit\" button is pressed? \nI think it will free a lot of \"interactive\" sessions with GPU hanging out.\nPlus, given the restriction \"one interactive session\", it will mean \"one commit session\".\n3. If after finishing python script GPU instance is not freed it would be good to do so. I start training script before going to sleep, for example, and hope that GPU instance will stop automatically after script is finished (not after 9 hours). Otherwise, it is a waste of resources (and our future quote).", "votes": 5 }, { "id": 608966, "postDate": "2019-08-27T09:51:08.200Z", "content": "Perfectly summarized! For interactive development and testing I rarely need the GPU or can break a look after a few iterations when I know it works. But I want to commit with a GPU because the commit otherwise takes forever. A \"commit as GPU kernel\" would be ideal!
", "rawMarkdown": "Perfectly summarized! For interactive development and testing I rarely need the GPU or can break a look after a few iterations when I know it works. But I want to commit with a GPU because the commit otherwise takes forever. A \"commit as GPU kernel\" would be ideal!", "votes": 1 } ] }, { "id": 607985, "postDate": "2019-08-26T07:17:23.180Z", "content": "\n\n\nWe have recently had high demand for GPUs in Kaggle Notebooks.
\n
You hosted 10 competetitions on Image data at the same time, what did you expect?
", "rawMarkdown": "> We have recently had high demand for GPUs in Kaggle Notebooks. \n\nYou hosted 10 competetitions on Image data at the same time, what did you expect?", "votes": 28 }, { "id": 607076, "postDate": "2019-08-24T15:18:31.803Z", "content": "This kind of restrictions should not be imposed 2 weeks before the end of a competition, not even in the middle of a competition. Once a competition starts the rules should be kept until the end. \nProbably I would not participated in this competition if I would have known about this GPU session limit.
", "rawMarkdown": "This kind of restrictions should not be imposed 2 weeks before the end of a competition, not even in the middle of a competition. Once a competition starts the rules should be kept until the end. \nProbably I would not participated in this competition if I would have known about this GPU session limit.", "votes": 27, "replies": [ { "id": 607078, "postDate": "2019-08-24T15:22:24.800Z", "content": "Fully agree
", "rawMarkdown": "Fully agree", "votes": 3 }, { "id": 608347, "postDate": "2019-08-26T16:46:43.333Z", "content": "Me too. I spent weeks for competition and now I'm almost blocked because I haven't possibility to continue. I never start competition with those limitations.
", "rawMarkdown": "Me too. I spent weeks for competition and now I'm almost blocked because I haven't possibility to continue. I never start competition with those limitations.", "votes": 1 } ] }, { "id": 606543, "postDate": "2019-08-23T18:04:59.133Z", "content": "What I am really curious about is why we have this kernel system of needing to submit twice to make a submission. This takes away so many resources that could be allocated better.
", "rawMarkdown": "What I am really curious about is why we have this kernel system of needing to submit twice to make a submission. This takes away so many resources that could be allocated better.", "votes": 23, "replies": [ { "id": 606577, "postDate": "2019-08-23T18:54:26.480Z", "content": "It's because they run it using private images and test.csv. I think that was the point of Synchroneous K.O. competitions in the first place - so that they wouldn't have to run it afterwards and that people can be sure their submissions won't suddenly fail on private data after competition is over.
", "rawMarkdown": "It's because they run it using private images and test.csv. I think that was the point of Synchroneous K.O. competitions in the first place - so that they wouldn't have to run it afterwards and that people can be sure their submissions won't suddenly fail on private data after competition is over." }, { "id": 606588, "postDate": "2019-08-23T19:08:34.040Z", "content": "I understand that but it is such a resource waste. There is a difference between making 200+ submissions and rerunning it once in the end.
", "rawMarkdown": "I understand that but it is such a resource waste. There is a difference between making 200+ submissions and rerunning it once in the end.", "votes": 7 }, { "id": 607605, "postDate": "2019-08-25T15:22:06.697Z", "content": "What would be interesting would be a \"test\" button where you can test a kernel vs the private dataset to see if it goes with no errors. it would count as a kernel being committed but you would only need to do it for what you would want to submit
", "rawMarkdown": "What would be interesting would be a \"test\" button where you can test a kernel vs the private dataset to see if it goes with no errors. it would count as a kernel being committed but you would only need to do it for what you would want to submit", "votes": 1 } ] }, { "id": 606535, "postDate": "2019-08-23T18:00:19.823Z", "content": "Does it use more Kaggle resources if Kaggle allows 3 GPUs which we can allocate however we want to \"commit\" or \"interactive\"?
", "rawMarkdown": "Does it use more Kaggle resources if Kaggle allows 3 GPUs which we can allocate however we want to \"commit\" or \"interactive\"?", "votes": 21 }, { "id": 607204, "postDate": "2019-08-24T19:40:58.237Z", "content": "What a surprise. You add more Kernels only competitions. You make my submission run again against a very private data set - and than you discover more resources are needed/being used.
\n\nUse that thing on top of body for more than a hat rack - of course high demand - surely that was on your list of concerns when you implemented the changes that led to the high demand.
\n\nYour telling me one of two things 1) Google does not own enough servers to meet the demand or 2) Google does not care for Kaggler's - its too hard to advertise here.
\n\nI own 4 PC's with decent GPU's and I am still PO with you. Can hardly guess how folks with no GPU resources are feeling.
", "rawMarkdown": "What a surprise. You add more Kernels only competitions. You make my submission run again against a very private data set - and than you discover more resources are needed/being used.\n\nUse that thing on top of body for more than a hat rack - of course high demand - surely that was on your list of concerns when you implemented the changes that led to the high demand. \n\nYour telling me one of two things 1) Google does not own enough servers to meet the demand or 2) Google does not care for Kaggler's - its too hard to advertise here.\n\nI own 4 PC's with decent GPU's and I am still PO with you. Can hardly guess how folks with no GPU resources are feeling.\n", "votes": 20 }, { "id": 606533, "postDate": "2019-08-23T17:58:54.047Z", "content": "It seems like it's gonna become even harder for people without GPU's to keep up.\nGiven the amount of image competitions running at the same time, it's genuinely disappointing.\nI wonder if people will start making more fake accounts because of that.
", "rawMarkdown": "It seems like it's gonna become even harder for people without GPU's to keep up.\nGiven the amount of image competitions running at the same time, it's genuinely disappointing.\nI wonder if people will start making more fake accounts because of that.", "votes": 16 }, { "id": 606522, "postDate": "2019-08-23T17:44:42.173Z", "content": "Please increase the interactive limit to at least 2, it's really hard for people like me who don't have local GPU/cloud.
", "rawMarkdown": "Please increase the interactive limit to at least 2, it's really hard for people like me who don't have local GPU/cloud.", "votes": 13 }, { "id": 606519, "postDate": "2019-08-23T17:41:46.757Z", "content": "You are kidding me...
", "rawMarkdown": "You are kidding me...", "votes": 11 }, { "id": 606807, "postDate": "2019-08-24T05:43:36.170Z", "content": "You should cross post this in EVERY competition. People complain in Molecular that they can't get 2 interactive notebooks without knowing it is by design.
", "rawMarkdown": "You should cross post this in EVERY competition. People complain in Molecular that they can't get 2 interactive notebooks without knowing it is by design.", "votes": 12, "replies": [ { "id": 606830, "postDate": "2019-08-24T06:30:19.957Z", "content": "Should be posted as bigger news since it affects many competitions now and going forward (imo).
", "rawMarkdown": "Should be posted as bigger news since it affects many competitions now and going forward (imo).", "votes": 1 } ] }, { "id": 607891, "postDate": "2019-08-26T03:52:16.297Z", "content": "Let me try to summarize the overall feeback in short\nThis current limit imposed is not welcomed by almost all participants as it severely restricts the no of experiments that one can do before committing some thing which may or may not be useful but at the same time it is also essential to well balance the resource usage to avoid overloading of the server which causes outages
\n\nI believe Allowing 2+ 2 should still be balanced approach of addressing the overloading issues on Kaggle server and at the same time participants also will have adequate kernels available to perform experiments. \nCurrent limit favors more those participants who can afford the paid GPUs outside the kaggle so i request let the competition be fair enough so that it dsnt gets sided to those who have more resources.
\n\nWhosoever agrees just give one word \nAgreed !
", "rawMarkdown": "Let me try to summarize the overall feeback in short\nThis current limit imposed is not welcomed by almost all participants as it severely restricts the no of experiments that one can do before committing some thing which may or may not be useful but at the same time it is also essential to well balance the resource usage to avoid overloading of the server which causes outages \n\nI believe Allowing 2+ 2 should still be balanced approach of addressing the overloading issues on Kaggle server and at the same time participants also will have adequate kernels available to perform experiments. \nCurrent limit favors more those participants who can afford the paid GPUs outside the kaggle so i request let the competition be fair enough so that it dsnt gets sided to those who have more resources. \n\nWhosoever agrees just give one word \nAgreed ! \n", "votes": 10 }, { "id": 607424, "postDate": "2019-08-25T08:18:32.167Z", "content": "How about not starting all 8-9 vision-based challenges at the same time? I refuse to believe that they didn't see this coming and perhaps this was pre-planned for quite some time.
", "rawMarkdown": "How about not starting all 8-9 vision-based challenges at the same time? I refuse to believe that they didn't see this coming and perhaps this was pre-planned for quite some time.", "votes": 10 }, { "id": 606680, "postDate": "2019-08-23T22:44:15.547Z", "content": "That is pretty much killing the idea of Kernels only DL competitions (and even Kernels competitions in general). What is the point since these will be again restricted only to people that have enough local resources? Can you rely (when you are starting a new competition) that you will have the resources to finish it? Apart from that - your restriction mechanism is not even working properly. At the moment I have one commit and no interactive sessions working - and it is not allowing me to start a new kernel saying \"Maximum interactive GPU session count of 1 reached.\". ... :(
", "rawMarkdown": "That is pretty much killing the idea of Kernels only DL competitions (and even Kernels competitions in general). What is the point since these will be again restricted only to people that have enough local resources? Can you rely (when you are starting a new competition) that you will have the resources to finish it? Apart from that - your restriction mechanism is not even working properly. At the moment I have one commit and no interactive sessions working - and it is not allowing me to start a new kernel saying \"Maximum interactive GPU session count of 1 reached.\". ... :(", "votes": 10, "replies": [ { "id": 607021, "postDate": "2019-08-24T13:08:17.797Z", "content": "same concern and incident.
", "rawMarkdown": "same concern and incident.", "votes": 1 } ] }, { "id": 606676, "postDate": "2019-08-23T22:28:00.430Z", "content": "Some members of the community (I included) were contacted a while ago to get some feedback on the idea to restrict GPU usage. Needless to say that my answer was not really taken into account.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2062758%2F1b2f3551dca67b8998c58bdbd5251ef5%2Fmel.png?generation=1566598987705800&alt=media\")
Your concerns mirror what I had shared as well. Thank you for also having tried.
", "rawMarkdown": "Your concerns mirror what I had shared as well. Thank you for also having tried. ", "votes": 2 } ] }, { "id": 608866, "postDate": "2019-08-27T08:24:22.677Z", "content": "Hi\nI have newly encountered with a problem due to submission... even-tough the program produces expected outputs including submission file in correct format and directory, committing does not produce any outputs nor completes running and gives no errors...\n (I checked with my already submitted versions having same work-space the same problem occurs)...\nBest
", "rawMarkdown": "Hi\nI have newly encountered with a problem due to submission... even-tough the program produces expected outputs including submission file in correct format and directory, committing does not produce any outputs nor completes running and gives no errors...\n (I checked with my already submitted versions having same work-space the same problem occurs)...\nBest", "votes": 7, "replies": [ { "id": 608913, "postDate": "2019-08-27T09:07:19.507Z", "content": "Similar issue with my commits - they looked like running for couple of hours - and then silently died without error. Perfectly valid and tested code. Now running again (after loosing couple of precious hours). Lets see what will happen...
", "rawMarkdown": "Similar issue with my commits - they looked like running for couple of hours - and then silently died without error. Perfectly valid and tested code. Now running again (after loosing couple of precious hours). Lets see what will happen..." }, { "id": 608915, "postDate": "2019-08-27T09:08:53.780Z", "content": "me too
", "rawMarkdown": "me too" }, { "id": 609007, "postDate": "2019-08-27T10:27:37.280Z", "content": "Same issue. I tried to run in iteractive session and works correctly till the end and generate output. But in commit, no output and no log.
", "rawMarkdown": "Same issue. I tried to run in iteractive session and works correctly till the end and generate output. But in commit, no output and no log." }, { "id": 609119, "postDate": "2019-08-27T12:28:30.730Z", "content": "Same for me since yesterday; commit runs to end of kernel and then times out. Commit sessions is deemed \"Completed\" (no Code(0)) and no outputs are written. The relevant commit logs follows:
\n\n216.26\n18\n[NbConvertApp] WARNING | Timeout waiting for IOPub output\n24128.49\n19\nError in atexit._run_exitfuncs: Traceback (most recent call last): File \"/opt/conda/lib/python3.6/logging/__init__.py\", line 1944, in shutdown h.flush() File \"/opt/conda/lib/python3.6/site-packages/absl/logging/__init__.py\", line 882, in flush self._current_handler.flush() File \"/opt/conda/lib/python3.6/site-packages/absl/logging/__init__.py\", line 776, in flush self.stream.flush() File \"/opt/conda/lib/python3.6/site-packages/ipykernel/iostream.py\", line 341, in flush if self.pub_thread.thread.is_alive(): AttributeError: 'NoneType' object has no attribute 'thread'\n24135.68\n20\n[NbConvertApp] Writing 71718 bytes to __notebook__.ipynb\n24138.15\n21\n[NbConvertApp] Converting notebook __notebook__.ipynb to html
Exactly the same problem. If you try to commit again the same Kernel it will result in same problem. Try to re-create from scratch (create new with the same code) - think that helps.
", "rawMarkdown": "Exactly the same problem. If you try to commit again the same Kernel it will result in same problem. Try to re-create from scratch (create new with the same code) - think that helps." }, { "id": 609191, "postDate": "2019-08-27T13:35:57.293Z", "content": "same issue @wcukierski
", "rawMarkdown": "same issue @wcukierski" }, { "id": 609282, "postDate": "2019-08-27T15:01:42.490Z", "content": "Try this folks:\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105968#latest-609131
", "rawMarkdown": "Try this folks:\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105968#latest-609131" } ] }, { "id": 606971, "postDate": "2019-08-24T11:44:36.993Z", "content": "I'm wondering what the ratio is between people actually running a model on a GPU or just clicking on \"GPU\" because they assume it's faster somehow.
\nI'm under the impression that many public notebooks I see spend most of their time doing feature engineering on a GPU instance which has only half the cores of a CPU instance. They just don't notice it because they don't use more than one core anyway.
This is a great point Arigion. Kaggle should calculate statistics to see what percentage of GPU enabled kernels are actually using the GPU. I suspect many have GPU turned on but do not use GPU
", "rawMarkdown": "This is a great point Arigion. Kaggle should calculate statistics to see what percentage of GPU enabled kernels are actually using the GPU. I suspect many have GPU turned on but do not use GPU", "votes": 1 }, { "id": 607068, "postDate": "2019-08-24T14:50:33.770Z", "content": "Maybe many GPUs are opened because of bitcoin mining.. lol
", "rawMarkdown": "Maybe many GPUs are opened because of bitcoin mining.. lol", "votes": 1 }, { "id": 607160, "postDate": "2019-08-24T17:28:15.493Z", "content": "I heard that too, bitcoin mining on Kaggle. It's a joke I assume. It should be pretty easy to prevent that or at least detect it, I think. Since I'm not doing cryptomining at all, does anyone have information about the possibility to mine on Kaggle or how to block mining?
", "rawMarkdown": "I heard that too, bitcoin mining on Kaggle. It's a joke I assume. It should be pretty easy to prevent that or at least detect it, I think. Since I'm not doing cryptomining at all, does anyone have information about the possibility to mine on Kaggle or how to block mining? ", "votes": 2 }, { "id": 607164, "postDate": "2019-08-24T17:37:04.917Z", "content": "It wasn't a joke when GPUs were first released a year ago because you could run unlimited GPUs at once. It was possible to run 100+ commit GPUs simultaneously and have them communicate via internet.
", "rawMarkdown": "It wasn't a joke when GPUs were first released a year ago because you could run unlimited GPUs at once. It was possible to run 100+ commit GPUs simultaneously and have them communicate via internet. ", "votes": 2 }, { "id": 607411, "postDate": "2019-08-25T07:28:45.717Z", "content": "@ppleskov can tell you a cool story about mining on kaggle kernels :)
", "rawMarkdown": "@ppleskov can tell you a cool story about mining on kaggle kernels :)", "votes": 2 }, { "id": 858048, "postDate": "2020-05-23T07:22:06.937Z", "content": "Is there a real story about crypto mining on Kaggle kernels?
", "rawMarkdown": "Is there a real story about crypto mining on Kaggle kernels?" } ] }, { "id": 624458, "postDate": "2019-09-12T05:27:55.730Z", "content": "30 hours per week would seem to be the unofficial end to individuals in computer vision challenges unless they have their own local GPU or can afford to buy GPU time from Google cloud. \"Company teams\" using company resources would seem to be the future Kaggle wants to encourage.
\n\nPerhaps because this topic is buried within another challenge it only has less than 200 down votes.
", "rawMarkdown": "30 hours per week would seem to be the unofficial end to individuals in computer vision challenges unless they have their own local GPU or can afford to buy GPU time from Google cloud. \"Company teams\" using company resources would seem to be the future Kaggle wants to encourage.\n\nPerhaps because this topic is buried within another challenge it only has less than 200 down votes. ", "votes": 6, "replies": [ { "id": 626089, "postDate": "2019-09-13T19:31:08.437Z", "content": "I've trained 1 model for Severstal and I am already at 11hrs of GPU usage(with some interactive session usage). I am not even sure 30hrs/week is adequate enough to run inference-only kernels if there are 2-3 active competitions which are kernel-only/kernel-inference-only.
\n\nFeature request: Please allow global account level settings to automatically have the option of turning GPU off/on as default when Forking a notebook. Right now, interactive GPU session already starts, even if a EDA notebook is forked(and I forget to disable the GPU everytime I fork).
", "rawMarkdown": "I've trained 1 model for Severstal and I am already at 11hrs of GPU usage(with some interactive session usage). I am not even sure 30hrs/week is adequate enough to run inference-only kernels if there are 2-3 active competitions which are kernel-only/kernel-inference-only. \n\nFeature request: Please allow global account level settings to automatically have the option of turning GPU off/on as default when Forking a notebook. Right now, interactive GPU session already starts, even if a EDA notebook is forked(and I forget to disable the GPU everytime I fork)." } ] }, { "id": 614356, "postDate": "2019-08-31T11:52:19.550Z", "content": "would it be possible to have an interactive kernel with GPU-OFF with options Commit and Commit with GPU
So you're telling me it's not possible to actively compete in Deep Learning competitions just because I don't have a GPU and I cannot afford one. Great! What's the point in competing in Kaggle (such an open and great community) if only people with beefy machines can make progress quickly. Truly gutted! Please understand that Kaggle Kernels and their GPUs are integral to this community and Kaggle's success over the years. I'll take this one for now :(
", "rawMarkdown": "So you're telling me it's not possible to actively compete in Deep Learning competitions just because I don't have a GPU and I cannot afford one. Great! What's the point in competing in Kaggle (such an open and great community) if only people with beefy machines can make progress quickly. Truly gutted! Please understand that Kaggle Kernels and their GPUs are integral to this community and Kaggle's success over the years. I'll take this one for now :( ", "votes": 5 }, { "id": 606928, "postDate": "2019-08-24T10:16:30.087Z", "content": "A couple of more image competitions might solve this problem. Oh wait...
", "rawMarkdown": "A couple of more image competitions might solve this problem. Oh wait...", "votes": 6 }, { "id": 606532, "postDate": "2019-08-23T17:58:39.623Z", "content": "Currently the system will not let me run my first interactive GPU kernel. Does anyone know a solution to this?
", "rawMarkdown": "Currently the system will not let me run my first interactive GPU kernel. Does anyone know a solution to this?", "votes": 6, "replies": [ { "id": 606534, "postDate": "2019-08-23T17:59:37.400Z", "content": "same
", "rawMarkdown": "same" }, { "id": 606553, "postDate": "2019-08-23T18:18:32.910Z", "content": "Might be because you have more than 2 commit session running, submitted before this change.\nHad the same problem with 3 commit sessions. After I stopped one of them, I was able to run my interactive session.\nHope this helps.
", "rawMarkdown": "Might be because you have more than 2 commit session running, submitted before this change.\nHad the same problem with 3 commit sessions. After I stopped one of them, I was able to run my interactive session.\nHope this helps.", "votes": 1 }, { "id": 606574, "postDate": "2019-08-23T18:49:51.490Z", "content": "same, I stopped all the kernels that might interfere and still have this issue
", "rawMarkdown": "same, I stopped all the kernels that might interfere and still have this issue" }, { "id": 606591, "postDate": "2019-08-23T19:12:33.947Z", "content": "I had one \"commit\" running. It stopped and now I can use my one \"interactive\". The problem was probably something to do with having a \"commit\" running before Kaggle made the change.
", "rawMarkdown": "I had one \"commit\" running. It stopped and now I can use my one \"interactive\". The problem was probably something to do with having a \"commit\" running before Kaggle made the change.", "votes": 1 }, { "id": 606935, "postDate": "2019-08-24T10:26:08.187Z", "content": "The same.I had a ''commit' running,I can't edit any kernel , only one \"interactive\" kernel.
", "rawMarkdown": "The same.I had a ''commit' running,I can't edit any kernel , only one \"interactive\" kernel." }, { "id": 607152, "postDate": "2019-08-24T17:09:08.640Z", "content": "I had the same issue but found that I had to re-edit the kernel I had just committed and then use the \"Run - Power off\" option to fully close the kernel. I could then open another kernel in edit mode.
", "rawMarkdown": "I had the same issue but found that I had to re-edit the kernel I had just committed and then use the \"Run - Power off\" option to fully close the kernel. I could then open another kernel in edit mode.", "votes": 2 } ] }, { "id": 606652, "postDate": "2019-08-23T20:48:37.863Z", "content": "I have no local resources, I've almost gave up this competition....🙄 Good luck everyone!
", "rawMarkdown": "I have no local resources, I've almost gave up this competition....🙄 Good luck everyone! ", "votes": 4, "replies": [ { "id": 606719, "postDate": "2019-08-24T01:29:04.933Z", "content": "Try vast.ai. It's cheap compares to AWS or GG Clouds but a little bit less reliable.
", "rawMarkdown": "Try vast.ai. It's cheap compares to AWS or GG Clouds but a little bit less reliable.", "votes": 2 } ] }, { "id": 613693, "postDate": "2019-08-30T18:34:39.927Z", "content": "I do agree that there are constraints in using kernel resources. Since, there are many kernel only competitions and the number of iterations each Kaggler tries to get the best fit for the data, 2 GPU kernels were actually good for someone who wants to compete on Image related competitions. It will be easier for many Kaggler's who depend on Kaggle's GPU kernels to stay away from competitions and it might become the competition for those who have GPU's for their personal use. Neverthless, if it can be increased to 2, it will be very useful.
", "rawMarkdown": "I do agree that there are constraints in using kernel resources. Since, there are many kernel only competitions and the number of iterations each Kaggler tries to get the best fit for the data, 2 GPU kernels were actually good for someone who wants to compete on Image related competitions. It will be easier for many Kaggler's who depend on Kaggle's GPU kernels to stay away from competitions and it might become the competition for those who have GPU's for their personal use. Neverthless, if it can be increased to 2, it will be very useful. ", "votes": 1 }, { "id": 608339, "postDate": "2019-08-26T16:30:56.983Z", "content": "Any chance that this limit is updated after the Prediction Molecules Competition ends? It would be important to have more GPU sessions in this final period of the competition. Thanks
", "rawMarkdown": "Any chance that this limit is updated after the Prediction Molecules Competition ends? It would be important to have more GPU sessions in this final period of the competition. Thanks", "votes": 1 }, { "id": 607692, "postDate": "2019-08-25T18:22:27.473Z", "content": "It is definitive ? Or we will retrieve more of \"interactive GPU session\" after the end of these competitions ?
", "rawMarkdown": "It is definitive ? Or we will retrieve more of \"interactive GPU session\" after the end of these competitions ?", "votes": 1 }, { "id": 607307, "postDate": "2019-08-25T01:55:28.100Z", "content": "I'm confused by sudden and big changes.\nI almost give up my score improvement.\nIsn't it possible to limit only new competitions that started recently?
", "rawMarkdown": "I'm confused by sudden and big changes.\nI almost give up my score improvement.\nIsn't it possible to limit only new competitions that started recently?", "votes": 1, "replies": [ { "id": 607356, "postDate": "2019-08-25T04:42:51.013Z", "content": "@octpath0302 Don't give up. Wish you can find a great team that help you continue strongly in this competition.
", "rawMarkdown": "@octpath0302 Don't give up. Wish you can find a great team that help you continue strongly in this competition.", "votes": 2 } ] }, { "id": 607048, "postDate": "2019-08-24T14:12:50.030Z", "content": "So you mean this competition is for who have gpu and AWS instances.Instead of this shit remove sync KO thing.
", "rawMarkdown": "So you mean this competition is for who have gpu and AWS instances.Instead of this shit remove sync KO thing.", "votes": 1 }, { "id": 606904, "postDate": "2019-08-24T09:27:30.977Z", "content": "How do you manage folks to do 2 commits? When I do one commit I cannot fork/edit another kernel (to be able to commit it). So the limitation is actually 1 commit and 0 interactive sessions until the committed kernel runs... Am I missing something?
", "rawMarkdown": "How do you manage folks to do 2 commits? When I do one commit I cannot fork/edit another kernel (to be able to commit it). So the limitation is actually 1 commit and 0 interactive sessions until the committed kernel runs... Am I missing something?", "votes": 1, "replies": [ { "id": 606924, "postDate": "2019-08-24T10:07:28.840Z", "content": "In your fork screen, where the limit exceed, there should show the previous session (top-right corner) where you can press 'stop' to kill that session.
\n\n(or in cases of edit-mainly switch GPU=Off)
", "rawMarkdown": "In your fork screen, where the limit exceed, there should show the previous session (top-right corner) where you can press 'stop' to kill that session.\n\n(or in cases of edit-mainly switch GPU=Off)", "votes": 2 }, { "id": 606942, "postDate": "2019-08-24T10:47:32.397Z", "content": "Thanks mate! It got fixed now.
", "rawMarkdown": "Thanks mate! It got fixed now.", "votes": 1 } ] }, { "id": 614681, "postDate": "2019-08-31T22:09:22.893Z", "content": "What is even worse about this is that kernels are not even stable enough...\nMy one interactive session disconnected after 1 hour, it even had a bug of looking like plain jupyter notebook and not displaying output... You should work on stability and not add some lesser features like ram/cpu gauge GUI... What was wrong with previous cpu/ram info?\nAnd ofcourse 1+2 doesn't make any sense, give us at least 3 gpus to use how we want, or at least make it 2+2.\nWhat you did here is a huge downgrade especially in interactive sessions and for people that are still learning cause they need to experiment more! And doing it just before competition end is also a very bad move...
", "rawMarkdown": "What is even worse about this is that kernels are not even stable enough...\nMy one interactive session disconnected after 1 hour, it even had a bug of looking like plain jupyter notebook and not displaying output... You should work on stability and not add some lesser features like ram/cpu gauge GUI... What was wrong with previous cpu/ram info?\nAnd ofcourse 1+2 doesn't make any sense, give us at least 3 gpus to use how we want, or at least make it 2+2.\nWhat you did here is a huge downgrade especially in interactive sessions and for people that are still learning cause they need to experiment more! And doing it just before competition end is also a very bad move...", "votes": 2 }, { "id": 610800, "postDate": "2019-08-29T01:27:15.767Z", "content": "Due to a new bug, I'm now unable to use even one session, or read any kaggle kernel. The competition is ending and hope that this will get fixed soon.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1607546%2F870ffa6bdad71b6ae217a7d729312830%2FZECLGAUM22T7ZFLGXVB6.png?generation=1567042021126344&alt=media\")
Hi JianWen Yang. Did the issue get resolved??..
", "rawMarkdown": "Hi JianWen Yang. Did the issue get resolved??.. ", "votes": 1 }, { "id": 626091, "postDate": "2019-09-13T19:37:18.440Z", "content": "I think clicking on the \"stop\" button just below 1/1 GPU (top right) should solve the issue by closing previous kernel and allowing new one to be launched.
", "rawMarkdown": "I think clicking on the \"stop\" button just below 1/1 GPU (top right) should solve the issue by closing previous kernel and allowing new one to be launched." } ] }, { "id": 608539, "postDate": "2019-08-26T22:40:21.903Z", "content": "It's possible find a way to work with single session without a pain? Any time I try to use one, I found occupied. And often stopping the others, the pending commits are cancelled.
", "rawMarkdown": "It's possible find a way to work with single session without a pain? Any time I try to use one, I found occupied. And often stopping the others, the pending commits are cancelled.", "votes": 2 }, { "id": 607999, "postDate": "2019-08-26T07:36:02.653Z", "content": "@wcukierski Is there any chance we could get more cpu cores in gpu kernels? As you didn't limit the cpu kernels, I guess you have spare ones.
", "rawMarkdown": "@wcukierski Is there any chance we could get more cpu cores in gpu kernels? As you didn't limit the cpu kernels, I guess you have spare ones.", "votes": 2 }, { "id": 607024, "postDate": "2019-08-24T13:20:59.567Z", "content": "I will agree on 1 GPU per user but make it 32GB instead of 16 GB (and also increase RAM too). I think we can fast experimentation and also fast training! :D
", "rawMarkdown": "I will agree on 1 GPU per user but make it 32GB instead of 16 GB (and also increase RAM too). I think we can fast experimentation and also fast training! :D", "votes": 2 }, { "id": 616284, "postDate": "2019-09-03T00:37:53.543Z", "content": "We should all be grateful that Kaggle allows us any free GPU time at all. I don't understand what all this entitlement is about.
\n\nAnyway, Google colab is a nice alternative when the kaggle kernels are giving you trouble. You can even link directly to any datasets on kaggle.
", "rawMarkdown": "We should all be grateful that Kaggle allows us any free GPU time at all. I don't understand what all this entitlement is about. \n\nAnyway, Google colab is a nice alternative when the kaggle kernels are giving you trouble. You can even link directly to any datasets on kaggle. \n\n", "replies": [ { "id": 616441, "postDate": "2019-09-03T06:17:23.483Z", "content": "When you have tried to participate in a number of Kaggle challenges I think you learn that Time Is Not Your Friend.
\n\nEven an old guy like myself can come up with many ideas and paths that I want to follow on a challenge. If your only taking two steps per day (limited by the number of GPU's) its very difficult to get all your ideas tested and the good ones put into your submission.
\n\nHaving a reduce number of available GPU along with the time restriction of their run length makes it a serious problem to try and get a submission that puts you in the gold ring. This restriction will give folks who do not have access to GPU a serious handicap vs those folks who have outside GPU resources. This makes the nature of Kaggle seem to be heading to \"professional\" teams vs hard working individuals who are here to learn and perhaps build a career.
", "rawMarkdown": "When you have tried to participate in a number of Kaggle challenges I think you learn that Time Is Not Your Friend. \n\nEven an old guy like myself can come up with many ideas and paths that I want to follow on a challenge. If your only taking two steps per day (limited by the number of GPU's) its very difficult to get all your ideas tested and the good ones put into your submission.\n\nHaving a reduce number of available GPU along with the time restriction of their run length makes it a serious problem to try and get a submission that puts you in the gold ring. This restriction will give folks who do not have access to GPU a serious handicap vs those folks who have outside GPU resources. This makes the nature of Kaggle seem to be heading to \"professional\" teams vs hard working individuals who are here to learn and perhaps build a career.\n\n", "votes": 6 }, { "id": 617386, "postDate": "2019-09-04T04:31:59.177Z", "content": "SterlingRamroach,\n1. People are grateful for having a possibility to use Kaggle resource. \n People have concerns that the rules changed very close to the end of the competition. When choosing the competition, people take into account time line and the resources they can use. Any change affects the successful finishing the competition.\n The number of negative votes approximately reflect the number of people that were affected by this decision.
\n\nSo, occasional and not computationally intense use of GC (not every day and less than 12 hours per day) would help, but current Kaggle kernel is much more stable, faster, and has more memory.
", "rawMarkdown": "SterlingRamroach,\n1. People are grateful for having a possibility to use Kaggle resource. \n People have concerns that the rules changed very close to the end of the competition. When choosing the competition, people take into account time line and the resources they can use. Any change affects the successful finishing the competition.\n The number of negative votes approximately reflect the number of people that were affected by this decision.\n\n2. If you finished at least one competition with heavy use of GPU and big volume of data and used google collaborate for that competition, you would not mislead people recommending use of Google Collaborate (GC) for that.\n GC is a terrific tool to get first experience working in google cloud and use GPU before starting working in paid instance of Google Cloud, using Google Cloud Storage (paid). \n It is not for long training.\n Why? I used it for several competitions after I learned about GC.\nAfter 4-5 sessions with GPU (12 hours limit, one 12-hour session during the day and one 12-hour session at night - in about 2-3 days ) you can not get next GPU session immediately:\n\"No available GPU, try later ...\" (forgot exact phrase)\nMy guess is that the user gets priority approximately as (M - total-time-used-GPU-recently), where M is some constant. \nIn 1-2-3 hours, if you are lucky, you would get this instance. Next available session would be in 1-2-3 days. You get the picture...\n I would not even mention the shared GPU - my 'typical' GPU instance was 560 KB (instead of 11 Gb). I spent a lot of time to figure why my successful scripts crashes because of OOM.\nFinally, before running any script, I wrote and ran small script \"show-available-gpu-memory\" that was showing gpu memory available and repeated getting a new instance until I got a new instance with 11 Gb. \n\nSo, occasional and not computationally intense use of GC (not every day and less than 12 hours per day) would help, but current Kaggle kernel is much more stable, faster, and has more memory. ", "votes": 2 }, { "id": 617763, "postDate": "2019-09-04T12:53:36.623Z", "content": "Oh thanks for clearing that up. I've used google colab almost every day for the past year and never had an issue getting GPU time (although I only needed it a few times per week). I assumed everyone got ~12 hours per GPU session like me with 100% availability.
\n\nI guess I must have been lucky those days I used their GPU consistently 😄.
", "rawMarkdown": "Oh thanks for clearing that up. I've used google colab almost every day for the past year and never had an issue getting GPU time (although I only needed it a few times per week). I assumed everyone got ~12 hours per GPU session like me with 100% availability. \n\nI guess I must have been lucky those days I used their GPU consistently 😄. " } ] }, { "id": 606492, "postDate": "2019-08-23T17:01:08.457Z", "content": "We have recently had high demand for GPUs in Kaggle Notebooks. To better allocate the resources we have, we will be slightly limiting concurrent GPU sessions to allow 1 interactive GPU session and 2 commit sessions at one time. This change is effective immediately. We also anticipate adding user-level GPU runtime quotas to accounts within the coming weeks (but not before the close of this competition). Thanks for your understanding!
", "rawMarkdown": "We have recently had high demand for GPUs in Kaggle Notebooks. To better allocate the resources we have, we will be slightly limiting concurrent GPU sessions to allow 1 interactive GPU session and 2 commit sessions at one time. This change is effective immediately. We also anticipate adding user-level GPU runtime quotas to accounts within the coming weeks (but not before the close of this competition). Thanks for your understanding!", "votes": -198 }, { "id": 1213840, "postDate": "2021-02-22T11:27:35.723Z", "content": "I am getting the error
\nMaximum interactive GPU session count of 1 reached.
\nHow do i solve it plz help
Bottom left corner, \"Active Events\" look for a running interactive GPU notebook and stop it.
", "rawMarkdown": "Bottom left corner, \"Active Events\" look for a running interactive GPU notebook and stop it." } ] }, { "id": 631919, "postDate": "2019-09-23T00:30:23.700Z", "content": "@wcukierski @addisonhoward @inversion @wendykan I feel that the GPU limit could hinder people trying to become Kaggle Kernel expert/master/grandmaster because they how have a limit on submitting kernels, i.e. updating their existing kernels. Just brainstorming, maybe there could be some sort of GPU reward given for each new bronze/silver/gold public kernel created to give the resources for people making helpful kernel content on kaggle.
", "rawMarkdown": "@wcukierski @addisonhoward @inversion @wendykan I feel that the GPU limit could hinder people trying to become Kaggle Kernel expert/master/grandmaster because they how have a limit on submitting kernels, i.e. updating their existing kernels. Just brainstorming, maybe there could be some sort of GPU reward given for each new bronze/silver/gold public kernel created to give the resources for people making helpful kernel content on kaggle. " }, { "id": 625429, "postDate": "2019-09-13T04:42:10.343Z", "content": "While the exact value of the quota is questionable, I would say this is a fair way to restrict excessive usage of GPU, and is an alright way to make sure people who don't need GPU aren't running GPU instances (I personally like that update, though the commit/interactive restriction is indeed a real shame). However, as others pointed out I think it is important for Kaggle to mass-post this update to kagglers while fully explaining the reasoning as to people know exactly what is available to them before they join a competition. Otherwise, thanks for being on the most-affordable option to gain GPU instance Kaggle! I hope the following updates will help reduce wastage while giving back more use-time to people who actually will make those instances worthwhile. 😄
", "rawMarkdown": "While the exact value of the quota is questionable, I would say this is a fair way to restrict excessive usage of GPU, and is an alright way to make sure people who don't need GPU aren't running GPU instances (I personally like that update, though the commit/interactive restriction is indeed a real shame). However, as others pointed out I think it is important for Kaggle to mass-post this update to kagglers while fully explaining the reasoning as to people know exactly what is available to them before they join a competition. Otherwise, thanks for being on the most-affordable option to gain GPU instance Kaggle! I hope the following updates will help reduce wastage while giving back more use-time to people who actually will make those instances worthwhile. 😄 " }, { "id": 624358, "postDate": "2019-09-12T02:42:13.493Z", "content": "Things are getting interesting. Now we have GPU Quota 30 hrs per week. In my opinion, I will accept these changes as this is out of my control. My request, aside from competition top prizes, could you please grant an additional prize, which is an extended number of hours of GPU Quota per week (maybe a additional 20 or 30 GPU hours) for all competition medalists).
", "rawMarkdown": "Things are getting interesting. Now we have GPU Quota 30 hrs per week. In my opinion, I will accept these changes as this is out of my control. My request, aside from competition top prizes, could you please grant an additional prize, which is an extended number of hours of GPU Quota per week (maybe a additional 20 or 30 GPU hours) for all competition medalists)." }, { "id": 609722, "postDate": "2019-08-28T03:20:45.687Z", "content": "Must be doing something wrong - I can get 1 and 1 at one time.
\n\nUpdate: Things changed - Now I can get 1 and ZERO.
\n\nLove it
", "rawMarkdown": "Must be doing something wrong - I can get 1 and 1 at one time.\n\nUpdate: Things changed - Now I can get 1 and ZERO.\n\nLove it" }, { "id": 608937, "postDate": "2019-08-27T09:25:37.517Z", "content": "Although I understand your difficulty and feel OK with one GPU，I would still get this downvoted.
", "rawMarkdown": "Although I understand your difficulty and feel OK with one GPU，I would still get this downvoted." }, { "id": 606930, "postDate": "2019-08-24T10:17:05.167Z", "content": "Boo. Good way to chase away users though, if that's your intention.
", "rawMarkdown": "Boo. Good way to chase away users though, if that's your intention." }, { "id": 606837, "postDate": "2019-08-24T06:44:12.030Z", "content": "just a small note to many of us who have a problem with the maximum \"interactive=1\" session.
\n\nIn case that we just want to program and save (commit), we can switch GPU=OFF, the current limit of \"cpu-only\" is 10. Therefore, we can still edit many kernels at the same time.
", "rawMarkdown": "just a small note to many of us who have a problem with the **maximum \"interactive=1\" session**.\n\nIn case that we just want to program and save (commit), **we can switch GPU=OFF, the current limit of \"cpu-only\" is 10**. Therefore, we can still edit many kernels at the same time.", "replies": [ { "id": 607047, "postDate": "2019-08-24T14:08:48.323Z", "content": "Congratulations for your second place on LB.\nAbout your suggestion of doing commit with GPU=OFF; I think it will run for much longer time or it may even timeout. My commits last as long as if the execute and save.
", "rawMarkdown": "Congratulations for your second place on LB.\nAbout your suggestion of doing commit with GPU=OFF; I think it will run for much longer time or it may even timeout. My commits last as long as if the execute and save.", "votes": 1 } ] }, { "id": 606829, "postDate": "2019-08-24T06:27:09.923Z", "content": "I understand why you have done this, but has this also changed what GPU is used or % is allocated ? Some of my kernels ran fine before with plenty of head room but now they timeout.
", "rawMarkdown": "I understand why you have done this, but has this also changed what GPU is used or % is allocated ? Some of my kernels ran fine before with plenty of head room but now they timeout.", "replies": [ { "id": 608310, "postDate": "2019-08-26T16:01:01.467Z", "content": "This change only affected number of concurrent kernels, individual kernel resources were not changed.
", "rawMarkdown": "This change only affected number of concurrent kernels, individual kernel resources were not changed." } ] }, { "id": 606779, "postDate": "2019-08-24T04:22:14.363Z", "content": "Hi, anyone know what is \"interactive\" means?
\n\nI committed one session, then closed the page, but I could not even open another kernel for editing?\nI thought closing the page means it not interactive anymore?
\n\nOr any idea how to commit two sessions? Thanks.
", "rawMarkdown": "Hi, anyone know what is \"interactive\" means?\n\nI committed one session, then closed the page, but I could not even open another kernel for editing?\nI thought closing the page means it not interactive anymore?\n\nOr any idea how to commit two sessions? Thanks.", "replies": [ { "id": 606780, "postDate": "2019-08-24T04:26:04.063Z", "content": "Oh I figured it out. Have to stop the interactive session at the top-right corner...
", "rawMarkdown": "Oh I figured it out. Have to stop the interactive session at the top-right corner...", "votes": 1 } ] }, { "id": 606770, "postDate": "2019-08-24T03:51:56.473Z", "content": "too strict limits!
", "rawMarkdown": "too strict limits! " }, { "id": 606769, "postDate": "2019-08-24T03:51:02.433Z", "content": "I agree there is a need but I think this one is severe restriction and should be relaxed a bit..\nto say atleast 2 GPU session . Many people do some hit n trial before commiting and this severe restriction would affect that..\nPlease revisit this limitation... will not welcomed by many participants
", "rawMarkdown": "I agree there is a need but I think this one is severe restriction and should be relaxed a bit..\nto say atleast 2 GPU session . Many people do some hit n trial before commiting and this severe restriction would affect that..\nPlease revisit this limitation... will not welcomed by many participants", "replies": [ { "id": 607353, "postDate": "2019-08-25T04:33:43.350Z", "content": "i think 2+2 is still fine.. kaggle team must send this feedback to higher management to relax unwelcomed restriction..\nThis severe restriction may force people to go for paid resources like GCp ( not sure if this move is done to push people towards gcp and hence more billing there ) which is not good
", "rawMarkdown": "i think 2+2 is still fine.. kaggle team must send this feedback to higher management to relax unwelcomed restriction..\nThis severe restriction may force people to go for paid resources like GCp ( not sure if this move is done to push people towards gcp and hence more billing there ) which is not good" } ] }, { "id": 606707, "postDate": "2019-08-23T23:56:02.250Z", "content": "oof, as someone who only uses kernels this is really gonna affect what I can do(thankfully I have teammates with local machines). I guess i can start chipping in to some free GCP credits/using Colab
", "rawMarkdown": "oof, as someone who only uses kernels this is really gonna affect what I can do(thankfully I have teammates with local machines). I guess i can start chipping in to some free GCP credits/using Colab" }, { "id": 606640, "postDate": "2019-08-23T20:23:14.317Z", "content": "Since Kaggle is with Google these days, how about joining forces with Colab and enable use some of those TPU's (and GPU's) with the much nicer interface of Kaggle..?
", "rawMarkdown": "Since Kaggle is with Google these days, how about joining forces with Colab and enable use some of those TPU's (and GPU's) with the much nicer interface of Kaggle..?" }, { "id": 606579, "postDate": "2019-08-23T18:57:39.647Z", "content": "@wcukierski Demand is high though, can you at least keep commit sessions? it is mostly used for different experiments. or increase GPU memory from 16 to 32 if possible?
", "rawMarkdown": "@wcukierski Demand is high though, can you at least keep commit sessions? it is mostly used for different experiments. or increase GPU memory from 16 to 32 if possible?" }, { "id": 606514, "postDate": "2019-08-23T17:35:08.977Z", "content": "pity
", "rawMarkdown": "pity" }, { "id": 606502, "postDate": "2019-08-23T17:15:53.073Z", "content": "-1
", "rawMarkdown": "-1" }, { "id": 632273, "postDate": "2019-09-23T12:21:23.610Z", "rawMarkdown": "", "isDeleted": true }, { "id": 615303, "postDate": "2019-09-01T18:45:21.220Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 609038, "postDate": "2019-08-27T11:18:59.203Z", "rawMarkdown": "", "isDeleted": true }, { "id": 608299, "postDate": "2019-08-26T15:49:45.550Z", "rawMarkdown": "", "isDeleted": true }, { "id": 607351, "postDate": "2019-08-25T04:26:52.013Z", "content": "I wonder anyone thinks about training on Google Colab(which ads tpu) and upload data to Google drive?
", "rawMarkdown": "I wonder anyone thinks about training on Google Colab(which ads tpu) and upload data to Google drive?", "isDeleted": true, "replies": [ { "id": 607445, "postDate": "2019-08-25T09:05:19.547Z", "content": "Not working for me - tried this Yesterday and my session died in an hour or so. Totally unreliable. Apart from that GPUs are shared there - so you might get let's say 200 mb of its memory which results in memory error. You can give it a try of course - for me it was lost time...
", "rawMarkdown": "Not working for me - tried this Yesterday and my session died in an hour or so. Totally unreliable. Apart from that GPUs are shared there - so you might get let's say 200 mb of its memory which results in memory error. You can give it a try of course - for me it was lost time...", "votes": 1 } ] } ], "comments": [ { "id": 606578, "author_name": "dott", "author_url": "", "post_date": "2019-08-23T18:55:32.693000", "content": "
We have to admit this was funny. In every joke is a sliver of truth...
\n\nI wanted to provide candor in this situation. What's happening here isn't just launching lots of image competitions and being surprised that we have high GPU use.
\n\nOur GPU situation:
\n\nThanks for reading and for the feedback. We promise to stay on top of these limits to be sure Kernels offers as much as it can offer.
", "votes": 26, "replies": [] }, { "id": 608566, "author_name": "Brian Lee", "author_url": "", "post_date": "2019-08-27T00:26:38.380000", "content": "This should be posted across the platform, if it has not been already. It will at least provide more context on the situation to many Kagglers. :)
", "votes": 1, "replies": [] }, { "id": 608579, "author_name": "Nat Bel ML Fun", "author_url": "", "post_date": "2019-08-27T01:01:05.370000", "content": "One way to decrease GPU usage:\nI use only scripts, not notebooks.\nUntil now, selecting GPU in \"interactive\" session, I always implied using GPU for \"commit\" version. There is only one button for that. By default, \"interactive\" version gets GPU too and future \"commit\" has the same setting as \"interactive session\". I guess it came from notebook design. \"interactive\" session for creating and editing script is just editor from which I can \"submit\" script. It does not need GPU. \nDuring \"interactive\" session I mostly do small editing if script failed (even it worked locally, some errors like file names or paths names happen). I can do it without using GPU in \"interactive\" session. I only need to check that script started without errors. When I \"commit\" I thought my work is done and interactive session ended.\nNow I see that interactive session is still active. Maybe, it is not available for use by others for 9 hours (even Kaggle reports \"inactive\" state in 60 min). When \"commit\" is successful, I do \"submit\". If I close interactive version after I \"submit\", it is OK. If I close interactive session when \"commit\" session is running, \"commit\" session is being canceled too.\nI don't know whether it is an error from my side, but I tried to do so several times last days, and I did not have any success. So I need to wait until my \"commit\" is finished, start \"submit\", and only after that I can stop interactive session.\nIt means that for script kernels during \"commit\" session \"interactive\" session should be active which effectively means that we only can run one \"commit\" session and keep \"interactive\" session as balance.\nIt is double usage of GPU instances instead of one even for prediction and much more for training commit.
\n\nQuestions:\n1. Is it possible to have two buttons: use GPU for interactive session and for future \"commit\" or \"submit\"?\n2. Is it possible to stop \"interactive\" session automatically (without stopping \"'commit\" session) when \"commit\" or \"submit\" button is pressed? \nI think it will free a lot of \"interactive\" sessions with GPU hanging out.\nPlus, given the restriction \"one interactive session\", it will mean \"one commit session\".\n3. If after finishing python script GPU instance is not freed it would be good to do so. I start training script before going to sleep, for example, and hope that GPU instance will stop automatically after script is finished (not after 9 hours). Otherwise, it is a waste of resources (and our future quote).
", "votes": 5, "replies": [] }, { "id": 608966, "author_name": "Phil", "author_url": "", "post_date": "2019-08-27T09:51:08.200000", "content": "Perfectly summarized! For interactive development and testing I rarely need the GPU or can break a look after a few iterations when I know it works. But I want to commit with a GPU because the commit otherwise takes forever. A \"commit as GPU kernel\" would be ideal!
", "votes": 1, "replies": [] } ] }, { "id": 607985, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2019-08-26T07:17:23.180000", "content": "\n\n\nWe have recently had high demand for GPUs in Kaggle Notebooks.
\n
You hosted 10 competetitions on Image data at the same time, what did you expect?
", "votes": 28, "replies": [] }, { "id": 607076, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-08-24T15:18:31.803000", "content": "This kind of restrictions should not be imposed 2 weeks before the end of a competition, not even in the middle of a competition. Once a competition starts the rules should be kept until the end. \nProbably I would not participated in this competition if I would have known about this GPU session limit.
", "votes": 27, "replies": [ { "id": 607078, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-24T15:22:24.800000", "content": "Fully agree
", "votes": 3, "replies": [] }, { "id": 608347, "author_name": "Enrico Beltramo", "author_url": "", "post_date": "2019-08-26T16:46:43.333000", "content": "Me too. I spent weeks for competition and now I'm almost blocked because I haven't possibility to continue. I never start competition with those limitations.
", "votes": 1, "replies": [] } ] }, { "id": 606543, "author_name": "Psi", "author_url": "", "post_date": "2019-08-23T18:04:59.133000", "content": "What I am really curious about is why we have this kernel system of needing to submit twice to make a submission. This takes away so many resources that could be allocated better.
", "votes": 23, "replies": [ { "id": 606577, "author_name": "Daniil Barysevich", "author_url": "", "post_date": "2019-08-23T18:54:26.480000", "content": "It's because they run it using private images and test.csv. I think that was the point of Synchroneous K.O. competitions in the first place - so that they wouldn't have to run it afterwards and that people can be sure their submissions won't suddenly fail on private data after competition is over.
", "votes": 0, "replies": [] }, { "id": 606588, "author_name": "Psi", "author_url": "", "post_date": "2019-08-23T19:08:34.040000", "content": "I understand that but it is such a resource waste. There is a difference between making 200+ submissions and rerunning it once in the end.
", "votes": 7, "replies": [] }, { "id": 607605, "author_name": "sh", "author_url": "", "post_date": "2019-08-25T15:22:06.697000", "content": "What would be interesting would be a \"test\" button where you can test a kernel vs the private dataset to see if it goes with no errors. it would count as a kernel being committed but you would only need to do it for what you would want to submit
", "votes": 1, "replies": [] } ] }, { "id": 606535, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-23T18:00:19.823000", "content": "Does it use more Kaggle resources if Kaggle allows 3 GPUs which we can allocate however we want to \"commit\" or \"interactive\"?
", "votes": 21, "replies": [] }, { "id": 607204, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-08-24T19:40:58.237000", "content": "What a surprise. You add more Kernels only competitions. You make my submission run again against a very private data set - and than you discover more resources are needed/being used.
\n\nUse that thing on top of body for more than a hat rack - of course high demand - surely that was on your list of concerns when you implemented the changes that led to the high demand.
\n\nYour telling me one of two things 1) Google does not own enough servers to meet the demand or 2) Google does not care for Kaggler's - its too hard to advertise here.
\n\nI own 4 PC's with decent GPU's and I am still PO with you. Can hardly guess how folks with no GPU resources are feeling.
", "votes": 20, "replies": [] }, { "id": 606533, "author_name": "Daniil Barysevich", "author_url": "", "post_date": "2019-08-23T17:58:54.047000", "content": "It seems like it's gonna become even harder for people without GPU's to keep up.\nGiven the amount of image competitions running at the same time, it's genuinely disappointing.\nI wonder if people will start making more fake accounts because of that.
", "votes": 16, "replies": [] }, { "id": 606522, "author_name": "Aptha K S", "author_url": "", "post_date": "2019-08-23T17:44:42.173000", "content": "Please increase the interactive limit to at least 2, it's really hard for people like me who don't have local GPU/cloud.
", "votes": 13, "replies": [] }, { "id": 606519, "author_name": "Psi", "author_url": "", "post_date": "2019-08-23T17:41:46.757000", "content": "You are kidding me...
", "votes": 11, "replies": [] }, { "id": 606807, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-24T05:43:36.170000", "content": "You should cross post this in EVERY competition. People complain in Molecular that they can't get 2 interactive notebooks without knowing it is by design.
", "votes": 12, "replies": [ { "id": 606830, "author_name": "JM", "author_url": "", "post_date": "2019-08-24T06:30:19.957000", "content": "Should be posted as bigger news since it affects many competitions now and going forward (imo).
", "votes": 1, "replies": [] } ] }, { "id": 607891, "author_name": "Jaideep", "author_url": "", "post_date": "2019-08-26T03:52:16.297000", "content": "Let me try to summarize the overall feeback in short\nThis current limit imposed is not welcomed by almost all participants as it severely restricts the no of experiments that one can do before committing some thing which may or may not be useful but at the same time it is also essential to well balance the resource usage to avoid overloading of the server which causes outages
\n\nI believe Allowing 2+ 2 should still be balanced approach of addressing the overloading issues on Kaggle server and at the same time participants also will have adequate kernels available to perform experiments. \nCurrent limit favors more those participants who can afford the paid GPUs outside the kaggle so i request let the competition be fair enough so that it dsnt gets sided to those who have more resources.
\n\nWhosoever agrees just give one word \nAgreed !
", "votes": 10, "replies": [] }, { "id": 607424, "author_name": "Mayukh Bhattacharyya", "author_url": "", "post_date": "2019-08-25T08:18:32.167000", "content": "How about not starting all 8-9 vision-based challenges at the same time? I refuse to believe that they didn't see this coming and perhaps this was pre-planned for quite some time.
", "votes": 10, "replies": [] }, { "id": 606680, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-23T22:44:15.547000", "content": "That is pretty much killing the idea of Kernels only DL competitions (and even Kernels competitions in general). What is the point since these will be again restricted only to people that have enough local resources? Can you rely (when you are starting a new competition) that you will have the resources to finish it? Apart from that - your restriction mechanism is not even working properly. At the moment I have one commit and no interactive sessions working - and it is not allowing me to start a new kernel saying \"Maximum interactive GPU session count of 1 reached.\". ... :(
", "votes": 10, "replies": [ { "id": 607021, "author_name": "FGPC", "author_url": "", "post_date": "2019-08-24T13:08:17.797000", "content": "same concern and incident.
", "votes": 1, "replies": [] } ] }, { "id": 606676, "author_name": "Theo Viel", "author_url": "", "post_date": "2019-08-23T22:28:00.430000", "content": "Some members of the community (I included) were contacted a while ago to get some feedback on the idea to restrict GPU usage. Needless to say that my answer was not really taken into account.
\n\n
Your concerns mirror what I had shared as well. Thank you for also having tried.
", "votes": 2, "replies": [] } ] }, { "id": 608866, "author_name": "Güner ALPAYDIN", "author_url": "", "post_date": "2019-08-27T08:24:22.677000", "content": "Hi\nI have newly encountered with a problem due to submission... even-tough the program produces expected outputs including submission file in correct format and directory, committing does not produce any outputs nor completes running and gives no errors...\n (I checked with my already submitted versions having same work-space the same problem occurs)...\nBest
", "votes": 7, "replies": [ { "id": 608913, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-27T09:07:19.507000", "content": "Similar issue with my commits - they looked like running for couple of hours - and then silently died without error. Perfectly valid and tested code. Now running again (after loosing couple of precious hours). Lets see what will happen...
", "votes": 0, "replies": [] }, { "id": 608915, "author_name": "Yangfan", "author_url": "", "post_date": "2019-08-27T09:08:53.780000", "content": "me too
", "votes": 0, "replies": [] }, { "id": 609007, "author_name": "Enrico Beltramo", "author_url": "", "post_date": "2019-08-27T10:27:37.280000", "content": "Same issue. I tried to run in iteractive session and works correctly till the end and generate output. But in commit, no output and no log.
", "votes": 0, "replies": [] }, { "id": 609119, "author_name": "Ramiro Debbe", "author_url": "", "post_date": "2019-08-27T12:28:30.730000", "content": "Same for me since yesterday; commit runs to end of kernel and then times out. Commit sessions is deemed \"Completed\" (no Code(0)) and no outputs are written. The relevant commit logs follows:
\n\n216.26\n18\n[NbConvertApp] WARNING | Timeout waiting for IOPub output\n24128.49\n19\nError in atexit._run_exitfuncs: Traceback (most recent call last): File \"/opt/conda/lib/python3.6/logging/__init__.py\", line 1944, in shutdown h.flush() File \"/opt/conda/lib/python3.6/site-packages/absl/logging/__init__.py\", line 882, in flush self._current_handler.flush() File \"/opt/conda/lib/python3.6/site-packages/absl/logging/__init__.py\", line 776, in flush self.stream.flush() File \"/opt/conda/lib/python3.6/site-packages/ipykernel/iostream.py\", line 341, in flush if self.pub_thread.thread.is_alive(): AttributeError: 'NoneType' object has no attribute 'thread'\n24135.68\n20\n[NbConvertApp] Writing 71718 bytes to __notebook__.ipynb\n24138.15\n21\n[NbConvertApp] Converting notebook __notebook__.ipynb to html
Exactly the same problem. If you try to commit again the same Kernel it will result in same problem. Try to re-create from scratch (create new with the same code) - think that helps.
", "votes": 0, "replies": [] }, { "id": 609191, "author_name": "ZeroWen", "author_url": "", "post_date": "2019-08-27T13:35:57.293000", "content": "same issue @wcukierski
", "votes": 0, "replies": [] }, { "id": 609282, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-27T15:01:42.490000", "content": "Try this folks:\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105968#latest-609131
", "votes": 0, "replies": [] } ] }, { "id": 606971, "author_name": "Arigion", "author_url": "", "post_date": "2019-08-24T11:44:36.993000", "content": "I'm wondering what the ratio is between people actually running a model on a GPU or just clicking on \"GPU\" because they assume it's faster somehow.
\nI'm under the impression that many public notebooks I see spend most of their time doing feature engineering on a GPU instance which has only half the cores of a CPU instance. They just don't notice it because they don't use more than one core anyway.
This is a great point Arigion. Kaggle should calculate statistics to see what percentage of GPU enabled kernels are actually using the GPU. I suspect many have GPU turned on but do not use GPU
", "votes": 1, "replies": [] }, { "id": 607068, "author_name": "FGPC", "author_url": "", "post_date": "2019-08-24T14:50:33.770000", "content": "Maybe many GPUs are opened because of bitcoin mining.. lol
", "votes": 1, "replies": [] }, { "id": 607160, "author_name": "Arigion", "author_url": "", "post_date": "2019-08-24T17:28:15.493000", "content": "I heard that too, bitcoin mining on Kaggle. It's a joke I assume. It should be pretty easy to prevent that or at least detect it, I think. Since I'm not doing cryptomining at all, does anyone have information about the possibility to mine on Kaggle or how to block mining?
", "votes": 2, "replies": [] }, { "id": 607164, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-24T17:37:04.917000", "content": "It wasn't a joke when GPUs were first released a year ago because you could run unlimited GPUs at once. It was possible to run 100+ commit GPUs simultaneously and have them communicate via internet.
", "votes": 2, "replies": [] }, { "id": 607411, "author_name": "Roman", "author_url": "", "post_date": "2019-08-25T07:28:45.717000", "content": "@ppleskov can tell you a cool story about mining on kaggle kernels :)
", "votes": 2, "replies": [] }, { "id": 858048, "author_name": "mzha2019", "author_url": "", "post_date": "2020-05-23T07:22:06.937000", "content": "Is there a real story about crypto mining on Kaggle kernels?
", "votes": 0, "replies": [] } ] }, { "id": 624458, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-09-12T05:27:55.730000", "content": "30 hours per week would seem to be the unofficial end to individuals in computer vision challenges unless they have their own local GPU or can afford to buy GPU time from Google cloud. \"Company teams\" using company resources would seem to be the future Kaggle wants to encourage.
\n\nPerhaps because this topic is buried within another challenge it only has less than 200 down votes.
", "votes": 6, "replies": [ { "id": 626089, "author_name": "Kartik Godawat", "author_url": "", "post_date": "2019-09-13T19:31:08.437000", "content": "I've trained 1 model for Severstal and I am already at 11hrs of GPU usage(with some interactive session usage). I am not even sure 30hrs/week is adequate enough to run inference-only kernels if there are 2-3 active competitions which are kernel-only/kernel-inference-only.
\n\nFeature request: Please allow global account level settings to automatically have the option of turning GPU off/on as default when Forking a notebook. Right now, interactive GPU session already starts, even if a EDA notebook is forked(and I forget to disable the GPU everytime I fork).
", "votes": 0, "replies": [] } ] }, { "id": 614356, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2019-08-31T11:52:19.550000", "content": "would it be possible to have an interactive kernel with GPU-OFF with options Commit and Commit with GPU
So you're telling me it's not possible to actively compete in Deep Learning competitions just because I don't have a GPU and I cannot afford one. Great! What's the point in competing in Kaggle (such an open and great community) if only people with beefy machines can make progress quickly. Truly gutted! Please understand that Kaggle Kernels and their GPUs are integral to this community and Kaggle's success over the years. I'll take this one for now :(
", "votes": 5, "replies": [] }, { "id": 606928, "author_name": "Roman", "author_url": "", "post_date": "2019-08-24T10:16:30.087000", "content": "A couple of more image competitions might solve this problem. Oh wait...
", "votes": 6, "replies": [] }, { "id": 606532, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-23T17:58:39.623000", "content": "Currently the system will not let me run my first interactive GPU kernel. Does anyone know a solution to this?
", "votes": 6, "replies": [ { "id": 606534, "author_name": "Kupchanski", "author_url": "", "post_date": "2019-08-23T17:59:37.400000", "content": "same
", "votes": 0, "replies": [] }, { "id": 606553, "author_name": "MD", "author_url": "", "post_date": "2019-08-23T18:18:32.910000", "content": "Might be because you have more than 2 commit session running, submitted before this change.\nHad the same problem with 3 commit sessions. After I stopped one of them, I was able to run my interactive session.\nHope this helps.
", "votes": 1, "replies": [] }, { "id": 606574, "author_name": "Daniil Barysevich", "author_url": "", "post_date": "2019-08-23T18:49:51.490000", "content": "same, I stopped all the kernels that might interfere and still have this issue
", "votes": 0, "replies": [] }, { "id": 606591, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-08-23T19:12:33.947000", "content": "I had one \"commit\" running. It stopped and now I can use my one \"interactive\". The problem was probably something to do with having a \"commit\" running before Kaggle made the change.
", "votes": 1, "replies": [] }, { "id": 606935, "author_name": "ynhuhu", "author_url": "", "post_date": "2019-08-24T10:26:08.187000", "content": "The same.I had a ''commit' running,I can't edit any kernel , only one \"interactive\" kernel.
", "votes": 0, "replies": [] }, { "id": 607152, "author_name": "Bon Intelli", "author_url": "", "post_date": "2019-08-24T17:09:08.640000", "content": "I had the same issue but found that I had to re-edit the kernel I had just committed and then use the \"Run - Power off\" option to fully close the kernel. I could then open another kernel in edit mode.
", "votes": 2, "replies": [] } ] }, { "id": 606652, "author_name": "Camaro", "author_url": "", "post_date": "2019-08-23T20:48:37.863000", "content": "I have no local resources, I've almost gave up this competition....🙄 Good luck everyone!
", "votes": 4, "replies": [ { "id": 606719, "author_name": "Quan", "author_url": "", "post_date": "2019-08-24T01:29:04.933000", "content": "Try vast.ai. It's cheap compares to AWS or GG Clouds but a little bit less reliable.
", "votes": 2, "replies": [] } ] }, { "id": 613693, "author_name": "Manoj Prabhakar", "author_url": "", "post_date": "2019-08-30T18:34:39.927000", "content": "I do agree that there are constraints in using kernel resources. Since, there are many kernel only competitions and the number of iterations each Kaggler tries to get the best fit for the data, 2 GPU kernels were actually good for someone who wants to compete on Image related competitions. It will be easier for many Kaggler's who depend on Kaggle's GPU kernels to stay away from competitions and it might become the competition for those who have GPU's for their personal use. Neverthless, if it can be increased to 2, it will be very useful.
", "votes": 1, "replies": [] }, { "id": 608339, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-08-26T16:30:56.983000", "content": "Any chance that this limit is updated after the Prediction Molecules Competition ends? It would be important to have more GPU sessions in this final period of the competition. Thanks
", "votes": 1, "replies": [] }, { "id": 607692, "author_name": "Arnaud ROUGETET", "author_url": "", "post_date": "2019-08-25T18:22:27.473000", "content": "It is definitive ? Or we will retrieve more of \"interactive GPU session\" after the end of these competitions ?
", "votes": 1, "replies": [] }, { "id": 607307, "author_name": "Filemide", "author_url": "", "post_date": "2019-08-25T01:55:28.100000", "content": "I'm confused by sudden and big changes.\nI almost give up my score improvement.\nIsn't it possible to limit only new competitions that started recently?
", "votes": 1, "replies": [ { "id": 607356, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-08-25T04:42:51.013000", "content": "@octpath0302 Don't give up. Wish you can find a great team that help you continue strongly in this competition.
", "votes": 2, "replies": [] } ] }, { "id": 607048, "author_name": "Mayur Newase", "author_url": "", "post_date": "2019-08-24T14:12:50.030000", "content": "So you mean this competition is for who have gpu and AWS instances.Instead of this shit remove sync KO thing.
", "votes": 1, "replies": [] }, { "id": 606904, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-24T09:27:30.977000", "content": "How do you manage folks to do 2 commits? When I do one commit I cannot fork/edit another kernel (to be able to commit it). So the limitation is actually 1 commit and 0 interactive sessions until the committed kernel runs... Am I missing something?
", "votes": 1, "replies": [ { "id": 606924, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-08-24T10:07:28.840000", "content": "In your fork screen, where the limit exceed, there should show the previous session (top-right corner) where you can press 'stop' to kill that session.
\n\n(or in cases of edit-mainly switch GPU=Off)
", "votes": 2, "replies": [] }, { "id": 606942, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-24T10:47:32.397000", "content": "Thanks mate! It got fixed now.
", "votes": 1, "replies": [] } ] }, { "id": 614681, "author_name": "MasterM", "author_url": "", "post_date": "2019-08-31T22:09:22.893000", "content": "What is even worse about this is that kernels are not even stable enough...\nMy one interactive session disconnected after 1 hour, it even had a bug of looking like plain jupyter notebook and not displaying output... You should work on stability and not add some lesser features like ram/cpu gauge GUI... What was wrong with previous cpu/ram info?\nAnd ofcourse 1+2 doesn't make any sense, give us at least 3 gpus to use how we want, or at least make it 2+2.\nWhat you did here is a huge downgrade especially in interactive sessions and for people that are still learning cause they need to experiment more! And doing it just before competition end is also a very bad move...
", "votes": 2, "replies": [] }, { "id": 610800, "author_name": "", "author_url": "", "post_date": "2019-08-29T01:27:15.767000", "content": "", "votes": 2, "replies": [ { "id": 613695, "author_name": "", "author_url": "", "post_date": "2019-08-30T18:36:19.923000", "content": "", "votes": 1, "replies": [] }, { "id": 626091, "author_name": "", "author_url": "", "post_date": "2019-09-13T19:37:18.440000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 608539, "author_name": "", "author_url": "", "post_date": "2019-08-26T22:40:21.903000", "content": "", "votes": 2, "replies": [] }, { "id": 607999, "author_name": "", "author_url": "", "post_date": "2019-08-26T07:36:02.653000", "content": "", "votes": 2, "replies": [] }, { "id": 607024, "author_name": "", "author_url": "", "post_date": "2019-08-24T13:20:59.567000", "content": "", "votes": 2, "replies": [] }, { "id": 616284, "author_name": "", "author_url": "", "post_date": "2019-09-03T00:37:53.543000", "content": "", "votes": 0, "replies": [ { "id": 616441, "author_name": "", "author_url": "", "post_date": "2019-09-03T06:17:23.483000", "content": "", "votes": 6, "replies": [] }, { "id": 617386, "author_name": "", "author_url": "", "post_date": "2019-09-04T04:31:59.177000", "content": "", "votes": 2, "replies": [] }, { "id": 617763, "author_name": "", "author_url": "", "post_date": "2019-09-04T12:53:36.623000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1213840, "author_name": "", "author_url": "", "post_date": "2021-02-22T11:27:35.723000", "content": "", "votes": 0, "replies": [ { "id": 1214122, "author_name": "", "author_url": "", "post_date": "2021-02-22T15:52:44.710000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 631919, "author_name": "", "author_url": "", "post_date": "2019-09-23T00:30:23.700000", "content": "", "votes": 0, "replies": [] }, { "id": 625429, "author_name": "", "author_url": "", "post_date": "2019-09-13T04:42:10.343000", "content": "", "votes": 0, "replies": [] }, { "id": 624358, "author_name": "", "author_url": "", "post_date": "2019-09-12T02:42:13.493000", "content": "", "votes": 0, "replies": [] }, { "id": 609722, "author_name": "", "author_url": "", "post_date": "2019-08-28T03:20:45.687000", "content": "", "votes": 0, "replies": [] }, { "id": 608937, "author_name": "", "author_url": "", "post_date": "2019-08-27T09:25:37.517000", "content": "", "votes": 0, "replies": [] }, { "id": 606930, "author_name": "", "author_url": "", "post_date": "2019-08-24T10:17:05.167000", "content": "", "votes": 0, "replies": [] }, { "id": 606837, "author_name": "", "author_url": "", "post_date": "2019-08-24T06:44:12.030000", "content": "", "votes": 0, "replies": [ { "id": 607047, "author_name": "", "author_url": "", "post_date": "2019-08-24T14:08:48.323000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 606829, "author_name": "", "author_url": "", "post_date": "2019-08-24T06:27:09.923000", "content": "", "votes": 0, "replies": [ { "id": 608310, "author_name": "", "author_url": "", "post_date": "2019-08-26T16:01:01.467000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 606779, "author_name": "", "author_url": "", "post_date": "2019-08-24T04:22:14.363000", "content": "", "votes": 0, "replies": [ { "id": 606780, "author_name": "", "author_url": "", "post_date": "2019-08-24T04:26:04.063000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 606770, "author_name": "", "author_url": "", "post_date": "2019-08-24T03:51:56.473000", "content": "", "votes": 0, "replies": [] }, { "id": 606769, "author_name": "", "author_url": "", "post_date": "2019-08-24T03:51:02.433000", "content": "", "votes": 0, "replies": [ { "id": 607353, "author_name": "", "author_url": "", "post_date": "2019-08-25T04:33:43.350000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 606707, "author_name": "", "author_url": "", "post_date": "2019-08-23T23:56:02.250000", "content": "", "votes": 0, "replies": [] }, { "id": 606640, "author_name": "", "author_url": "", "post_date": "2019-08-23T20:23:14.317000", "content": "", "votes": 0, "replies": [] }, { "id": 606579, "author_name": "", "author_url": "", "post_date": "2019-08-23T18:57:39.647000", "content": "", "votes": 0, "replies": [] }, { "id": 606514, "author_name": "", "author_url": "", "post_date": "2019-08-23T17:35:08.977000", "content": "", "votes": 0, "replies": [] }, { "id": 606502, "author_name": "", "author_url": "", "post_date": "2019-08-23T17:15:53.073000", "content": "", "votes": 0, "replies": [] }, { "id": 632273, "author_name": "", "author_url": "", "post_date": "2019-09-23T12:21:23.610000", "content": "", "votes": 0, "replies": [] }, { "id": 615303, "author_name": "", "author_url": "", "post_date": "2019-09-01T18:45:21.220000", "content": "", "votes": -1, "replies": [] }, { "id": 609038, "author_name": "", "author_url": "", "post_date": "2019-08-27T11:18:59.203000", "content": "", "votes": 0, "replies": [] }, { "id": 608299, "author_name": "", "author_url": "", "post_date": "2019-08-26T15:49:45.550000", "content": "", "votes": 0, "replies": [] }, { "id": 607351, "author_name": "", "author_url": "", "post_date": "2019-08-25T04:26:52.013000", "content": "", "votes": 0, "replies": [ { "id": 607445, "author_name": "", "author_url": "", "post_date": "2019-08-25T09:05:19.547000", "content": "", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "606578": "\n", "607985": "> We have recently had high demand for GPUs in Kaggle Notebooks. \n\nYou hosted 10 competetitions on Image data at the same time, what did you expect?", "607076": "This kind of restrictions should not be imposed 2 weeks before the end of a competition, not even in the middle of a competition. Once a competition starts the rules should be kept until the end. \nProbably I would not participated in this competition if I would have known about this GPU session limit.", "606543": "What I am really curious about is why we have this kernel system of needing to submit twice to make a submission. This takes away so many resources that could be allocated better.", "606535": "Does it use more Kaggle resources if Kaggle allows 3 GPUs which we can allocate however we want to \"commit\" or \"interactive\"?", "607204": "What a surprise. You add more Kernels only competitions. You make my submission run again against a very private data set - and than you discover more resources are needed/being used.\n\nUse that thing on top of body for more than a hat rack - of course high demand - surely that was on your list of concerns when you implemented the changes that led to the high demand. \n\nYour telling me one of two things 1) Google does not own enough servers to meet the demand or 2) Google does not care for Kaggler's - its too hard to advertise here.\n\nI own 4 PC's with decent GPU's and I am still PO with you. Can hardly guess how folks with no GPU resources are feeling.\n", "606533": "It seems like it's gonna become even harder for people without GPU's to keep up.\nGiven the amount of image competitions running at the same time, it's genuinely disappointing.\nI wonder if people will start making more fake accounts because of that.", "606522": "Please increase the interactive limit to at least 2, it's really hard for people like me who don't have local GPU/cloud.", "606519": "You are kidding me...", "606807": "You should cross post this in EVERY competition. People complain in Molecular that they can't get 2 interactive notebooks without knowing it is by design.", "607891": "Let me try to summarize the overall feeback in short\nThis current limit imposed is not welcomed by almost all participants as it severely restricts the no of experiments that one can do before committing some thing which may or may not be useful but at the same time it is also essential to well balance the resource usage to avoid overloading of the server which causes outages \n\nI believe Allowing 2+ 2 should still be balanced approach of addressing the overloading issues on Kaggle server and at the same time participants also will have adequate kernels available to perform experiments. \nCurrent limit favors more those participants who can afford the paid GPUs outside the kaggle so i request let the competition be fair enough so that it dsnt gets sided to those who have more resources. \n\nWhosoever agrees just give one word \nAgreed ! \n", "607424": "How about not starting all 8-9 vision-based challenges at the same time? I refuse to believe that they didn't see this coming and perhaps this was pre-planned for quite some time.", "606680": "That is pretty much killing the idea of Kernels only DL competitions (and even Kernels competitions in general). What is the point since these will be again restricted only to people that have enough local resources? Can you rely (when you are starting a new competition) that you will have the resources to finish it? Apart from that - your restriction mechanism is not even working properly. At the moment I have one commit and no interactive sessions working - and it is not allowing me to start a new kernel saying \"Maximum interactive GPU session count of 1 reached.\". ... :(", "606676": "Some members of the community (I included) were contacted a while ago to get some feedback on the idea to restrict GPU usage. Needless to say that my answer was not really taken into account. \n\n\n", "608866": "Hi\nI have newly encountered with a problem due to submission... even-tough the program produces expected outputs including submission file in correct format and directory, committing does not produce any outputs nor completes running and gives no errors...\n (I checked with my already submitted versions having same work-space the same problem occurs)...\nBest", "606971": "I'm wondering what the ratio is between people actually running a model on a GPU or just clicking on \"GPU\" because they assume it's faster somehow. \nI'm under the impression that many public notebooks I see spend most of their time doing feature engineering on a GPU instance which has [only half the cores](https://www.kaggle.com/docs/kernels#technical-specifications) of a CPU instance. They just don't notice it because they don't use more than one core anyway.", "624458": "30 hours per week would seem to be the unofficial end to individuals in computer vision challenges unless they have their own local GPU or can afford to buy GPU time from Google cloud. \"Company teams\" using company resources would seem to be the future Kaggle wants to encourage.\n\nPerhaps because this topic is buried within another challenge it only has less than 200 down votes. ", "614356": "would it be possible to have an interactive kernel with GPU-OFF with options `Commit` and `Commit with GPU` ", "607016": "So you're telling me it's not possible to actively compete in Deep Learning competitions just because I don't have a GPU and I cannot afford one. Great! What's the point in competing in Kaggle (such an open and great community) if only people with beefy machines can make progress quickly. Truly gutted! Please understand that Kaggle Kernels and their GPUs are integral to this community and Kaggle's success over the years. I'll take this one for now :( ", "606928": "A couple of more image competitions might solve this problem. Oh wait...", "606532": "Currently the system will not let me run my first interactive GPU kernel. Does anyone know a solution to this?", "606652": "I have no local resources, I've almost gave up this competition....🙄 Good luck everyone! ", "613693": "I do agree that there are constraints in using kernel resources. Since, there are many kernel only competitions and the number of iterations each Kaggler tries to get the best fit for the data, 2 GPU kernels were actually good for someone who wants to compete on Image related competitions. It will be easier for many Kaggler's who depend on Kaggle's GPU kernels to stay away from competitions and it might become the competition for those who have GPU's for their personal use. Neverthless, if it can be increased to 2, it will be very useful. ", "608339": "Any chance that this limit is updated after the Prediction Molecules Competition ends? It would be important to have more GPU sessions in this final period of the competition. Thanks", "607692": "It is definitive ? Or we will retrieve more of \"interactive GPU session\" after the end of these competitions ?", "607307": "\n", "607159": "I'm confused by sudden and big changes.\nI almost give up my score improvement.\nIsn't it possible to limit only new competitions that started recently?", "607048": "So you mean this competition is for who have gpu and AWS instances.Instead of this shit remove sync KO thing.", "606904": "How do you manage folks to do 2 commits? When I do one commit I cannot fork/edit another kernel (to be able to commit it). So the limitation is actually 1 commit and 0 interactive sessions until the committed kernel runs... Am I missing something?", "614681": "What is even worse about this is that kernels are not even stable enough...\nMy one interactive session disconnected after 1 hour, it even had a bug of looking like plain jupyter notebook and not displaying output... You should work on stability and not add some lesser features like ram/cpu gauge GUI... What was wrong with previous cpu/ram info?\nAnd ofcourse 1+2 doesn't make any sense, give us at least 3 gpus to use how we want, or at least make it 2+2.\nWhat you did here is a huge downgrade especially in interactive sessions and for people that are still learning cause they need to experiment more! And doing it just before competition end is also a very bad move...", "610800": "Due to a new bug, I'm now unable to use even one session, or read any kaggle kernel. The competition is ending and hope that this will get fixed soon.\n\n", "608539": "It's possible find a way to work with single session without a pain? Any time I try to use one, I found occupied. And often stopping the others, the pending commits are cancelled.", "607999": "@wcukierski Is there any chance we could get more cpu cores in gpu kernels? As you didn't limit the cpu kernels, I guess you have spare ones.", "607024": "I will agree on 1 GPU per user but make it 32GB instead of 16 GB (and also increase RAM too). I think we can fast experimentation and also fast training! :D", "616284": "We should all be grateful that Kaggle allows us any free GPU time at all. I don't understand what all this entitlement is about. \n\nAnyway, Google colab is a nice alternative when the kaggle kernels are giving you trouble. You can even link directly to any datasets on kaggle. \n\n", "606492": "We have recently had high demand for GPUs in Kaggle Notebooks. To better allocate the resources we have, we will be slightly limiting concurrent GPU sessions to allow 1 interactive GPU session and 2 commit sessions at one time. This change is effective immediately. We also anticipate adding user-level GPU runtime quotas to accounts within the coming weeks (but not before the close of this competition). Thanks for your understanding!", "1213840": "I am getting the error \nMaximum interactive GPU session count of 1 reached.\nHow do i solve it plz help", "631919": "@wcukierski @addisonhoward @inversion @wendykan I feel that the GPU limit could hinder people trying to become Kaggle Kernel expert/master/grandmaster because they how have a limit on submitting kernels, i.e. updating their existing kernels. Just brainstorming, maybe there could be some sort of GPU reward given for each new bronze/silver/gold public kernel created to give the resources for people making helpful kernel content on kaggle. ", "625429": "While the exact value of the quota is questionable, I would say this is a fair way to restrict excessive usage of GPU, and is an alright way to make sure people who don't need GPU aren't running GPU instances (I personally like that update, though the commit/interactive restriction is indeed a real shame). However, as others pointed out I think it is important for Kaggle to mass-post this update to kagglers while fully explaining the reasoning as to people know exactly what is available to them before they join a competition. Otherwise, thanks for being on the most-affordable option to gain GPU instance Kaggle! I hope the following updates will help reduce wastage while giving back more use-time to people who actually will make those instances worthwhile. 😄 ", "624358": "Things are getting interesting. Now we have GPU Quota 30 hrs per week. In my opinion, I will accept these changes as this is out of my control. My request, aside from competition top prizes, could you please grant an additional prize, which is an extended number of hours of GPU Quota per week (maybe a additional 20 or 30 GPU hours) for all competition medalists).", "609722": "Must be doing something wrong - I can get 1 and 1 at one time.\n\nUpdate: Things changed - Now I can get 1 and ZERO.\n\nLove it", "608937": "Although I understand your difficulty and feel OK with one GPU，I would still get this downvoted.", "606930": "Boo. Good way to chase away users though, if that's your intention.", "606837": "just a small note to many of us who have a problem with the **maximum \"interactive=1\" session**.\n\nIn case that we just want to program and save (commit), **we can switch GPU=OFF, the current limit of \"cpu-only\" is 10**. Therefore, we can still edit many kernels at the same time.", "606829": "I understand why you have done this, but has this also changed what GPU is used or % is allocated ? Some of my kernels ran fine before with plenty of head room but now they timeout.", "606779": "Hi, anyone know what is \"interactive\" means?\n\nI committed one session, then closed the page, but I could not even open another kernel for editing?\nI thought closing the page means it not interactive anymore?\n\nOr any idea how to commit two sessions? Thanks.", "606770": "too strict limits! ", "606769": "I agree there is a need but I think this one is severe restriction and should be relaxed a bit..\nto say atleast 2 GPU session . Many people do some hit n trial before commiting and this severe restriction would affect that..\nPlease revisit this limitation... will not welcomed by many participants", "606707": "oof, as someone who only uses kernels this is really gonna affect what I can do(thankfully I have teammates with local machines). I guess i can start chipping in to some free GCP credits/using Colab", "606640": "Since Kaggle is with Google these days, how about joining forces with Colab and enable use some of those TPU's (and GPU's) with the much nicer interface of Kaggle..?", "606579": "@wcukierski Demand is high though, can you at least keep commit sessions? it is mostly used for different experiments. or increase GPU memory from 16 to 32 if possible?", "606514": "pity", "606502": "-1", "632273": "", "615303": "", "609038": "", "608299": "", "607351": "I wonder anyone thinks about training on Google Colab(which ads tpu) and upload data to Google drive?" } }