I was going to do this competition but 84 GB of data is simply too much. Is there a way around it or a similar kaggle with much, much lower data requirement? Thanks
", "messages": [ { "id": "672156", "postDate": "11/13/2019 15:08:41", "content": "I was going to do this competition but 84 GB of data is simply too much. Is there a way around it or a similar kaggle with much, much lower data requirement? Thanks
", "rawMarkdown": "I was going to do this competition but 84 GB of data is simply too much. Is there a way around it or a similar kaggle with much, much lower data requirement? Thanks", "votes": null }, { "id": "672475", "postDate": "11/13/2019 23:43:51", "content": "and sadly they use zip file instead of tar :( with tar file, you can decompress stream during download and filter files if you want
", "rawMarkdown": "and sadly they use zip file instead of tar :( with tar file, you can decompress stream during download and filter files if you want", "votes": null }, { "id": "673062", "postDate": "11/14/2019 13:31:06", "content": "for those that give negative points can they tell something intelligent and useful like you can run the data on their server, Please stop being in love with yourselves.
", "rawMarkdown": "for those that give negative points can they tell something intelligent and useful like you can run the data on their server, Please stop being in love with yourselves.", "votes": null }, { "id": "673069", "postDate": "11/14/2019 13:34:48", "content": "Is it possible to run it on kaggle server? you know what are these negative points for?
", "rawMarkdown": "Is it possible to run it on kaggle server? you know what are these negative points for?", "votes": null }, { "id": "673138", "postDate": "11/14/2019 15:03:53", "content": "For this kind of competition you really need to have a dedicated Deep Learning PC. A typical setup is Ubuntu, multicore cpu, SSD drive, lots of ram 32gb+ preferably, and a GPU like a GTX 1080ti, RTX 2070 etc...
\n\nUsing Kaggle notebooks is also a great option and they have GPU access but it's limited to 30 hrs per week.
", "rawMarkdown": "For this kind of competition you really need to have a dedicated Deep Learning PC. A typical setup is Ubuntu, multicore cpu, SSD drive, lots of ram 32gb+ preferably, and a GPU like a GTX 1080ti, RTX 2070 etc...\n\nUsing Kaggle notebooks is also a great option and they have GPU access but it's limited to 30 hrs per week.", "votes": null }, { "id": "673148", "postDate": "11/14/2019 15:27:36", "content": "Thank you very much Jack, this is what I call useful information and it would be a delight to get a negative mark from you. I have never hear about kaggle notebooks, I am very new to this. I take it, is a sight I sign into and I do not have to download the data any more. Learning is what is important to me. Thanks again.
", "rawMarkdown": "Thank you very much Jack, this is what I call useful information and it would be a delight to get a negative mark from you. I have never hear about kaggle notebooks, I am very new to this. I take it, is a sight I sign into and I do not have to download the data any more. Learning is what is important to me. Thanks again.", "votes": null }, { "id": "673153", "postDate": "11/14/2019 15:32:57", "content": "Try not be discouraged by the downvotes or take it too personally. If your questions is something you could have easily googled or spent a few minutes clicking around and exploring the interface to figure out then probably not a great idea to ask it here.
\n\nAlso, mentioning mac is probably going to guarantee a few downvotes! 😛 I use a mac for my day job but there is not much love for macs on here, the price being a big factor and the lack of support for CUDA and Nvidia GPUs.
", "rawMarkdown": "Try not be discouraged by the downvotes or take it too personally. If your questions is something you could have easily googled or spent a few minutes clicking around and exploring the interface to figure out then probably not a great idea to ask it here.\n\nAlso, mentioning mac is probably going to guarantee a few downvotes! 😛 I use a mac for my day job but there is not much love for macs on here, the price being a big factor and the lack of support for CUDA and Nvidia GPUs.", "votes": null }, { "id": "673155", "postDate": "11/14/2019 15:38:22", "content": "Welcome, if you look at the tab bar with overview, data, notebooks and click on notebooks you can create a Jupyter Notebook that runs on Kaggle's servers, the competition dataset will be available to you within this environment.
", "rawMarkdown": "Welcome, if you look at the tab bar with overview, data, notebooks and click on notebooks you can create a Jupyter Notebook that runs on Kaggle's servers, the competition dataset will be available to you within this environment.", "votes": null }, { "id": "673196", "postDate": "11/14/2019 16:46:32", "content": "I bought mac thinking it would support the latest technologies and I was shocked when I wanted to use it for GPU, but I am stuck with it as I paid so much but never again. Thanks again
", "rawMarkdown": "I bought mac thinking it would support the latest technologies and I was shocked when I wanted to use it for GPU, but I am stuck with it as I paid so much but never again. Thanks again", "votes": null }, { "id": "673249", "postDate": "11/14/2019 18:05:35", "content": "You need to throw more money at it and buy a thunderbolt-pcie-gpu-enclosure like razer core-x or similar then put in a decent pcie-nvidia-gpu and boot linux anyways for deeplearning frameworks.
", "rawMarkdown": "You need to throw more money at it and buy a thunderbolt-pcie-gpu-enclosure like razer core-x or similar then put in a decent pcie-nvidia-gpu and boot linux anyways for deeplearning frameworks.", "votes": null }, { "id": "673260", "postDate": "11/14/2019 18:16:22", "content": "What kind of money are we talking about? One big problem I have is that I travel a lot and relocate completely so it has to be transportable.
", "rawMarkdown": "What kind of money are we talking about? One big problem I have is that I travel a lot and relocate completely so it has to be transportable.", "votes": null }, { "id": "673307", "postDate": "11/14/2019 19:50:11", "content": "You using a mac has nothing to do with the downvotes. I use a macbook everyday, I switched about a year ago because developing IOS apps on a hackintosh VM was a nightmare. For coding IMO it's the best notebook money can buy,\nbut for deep learning, it's only good to SSH into a workstation :)
\n\nTo answer your original question, the starter kernel creates 2D images from the data, you could create a dataset from that, and download it. Only about a few GB in size.
", "rawMarkdown": "You using a mac has nothing to do with the downvotes. I use a macbook everyday, I switched about a year ago because developing IOS apps on a hackintosh VM was a nightmare. For coding IMO it's the best notebook money can buy,\nbut for deep learning, it's only good to SSH into a workstation :)\n\nTo answer your original question, the starter kernel creates 2D images from the data, you could create a dataset from that, and download it. Only about a few GB in size.", "votes": null }, { "id": "673347", "postDate": "11/14/2019 21:44:27", "content": "Thanks for the info. So there is a kernel called starter that you run for that and you do not need to convert the 2D images on your computer to full size data? I have not come across that kernel.
", "rawMarkdown": "Thanks for the info. So there is a kernel called starter that you run for that and you do not need to convert the 2D images on your computer to full size data? I have not come across that kernel.", "votes": null }, { "id": "673358", "postDate": "11/14/2019 22:17:20", "content": "84 GB of data is simply too much.\nI just don't understand this. You bought a PC that doesn't have 100 GB HDD or SSD? For reference, I bought a 3 TB HDD in 2012 and I'm still using it. This year I bought a couple of 1 TB NVMe SSD disks, they really do help a lot when I'm dealing with Computer Vision challenges.
And yes, you absolutely need a computer with GPU. It's typically a Linux PC, Ubuntu or something similar. Ideally, you need more than one GPU, 1 TB SSD and (32 Gb * number of GPUs) of memory.
", "rawMarkdown": "`84 GB of data is simply too much.`\nI just don't understand this. You bought a PC that doesn't have 100 GB HDD or SSD? For reference, I bought a 3 TB HDD in 2012 and I'm still using it. This year I bought a couple of 1 TB NVMe SSD disks, they really do help a lot when I'm dealing with Computer Vision challenges.\n\nAnd yes, you absolutely need a computer with GPU. It's typically a Linux PC, Ubuntu or something similar. Ideally, you need more than one GPU, 1 TB SSD and (32 Gb * number of GPUs) of memory.", "votes": null }, { "id": "673592", "postDate": "11/15/2019 07:45:05", "content": "Sorry it's called reference model, here you go: https://www.kaggle.com/gzuidhof/reference-model
", "rawMarkdown": "Sorry it's called reference model, here you go: https://www.kaggle.com/gzuidhof/reference-model", "votes": null }, { "id": "673775", "postDate": "11/15/2019 13:25:30", "content": "I have a mac with 250 SSD for everything, unfortunately I travel a lot and impossible to have pc, but I am dying to buy one. For now I have to be content on running code on clouds and other people servers
", "rawMarkdown": "I have a mac with 250 SSD for everything, unfortunately I travel a lot and impossible to have pc, but I am dying to buy one. For now I have to be content on running code on clouds and other people servers", "votes": null } ], "comments": [ { "id": 672475, "author_name": "oarphme", "author_url": "", "post_date": "11/13/2019 23:43:51", "content": "and sadly they use zip file instead of tar :( with tar file, you can decompress stream during download and filter files if you want
", "votes": null, "replies": [ { "id": 673069, "author_name": "kevinehsani", "author_url": "", "post_date": "11/14/2019 13:34:48", "content": "Is it possible to run it on kaggle server? you know what are these negative points for?
", "votes": null, "replies": [] } ] }, { "id": 673062, "author_name": "kevinehsani", "author_url": "", "post_date": "11/14/2019 13:31:06", "content": "for those that give negative points can they tell something intelligent and useful like you can run the data on their server, Please stop being in love with yourselves.
", "votes": null, "replies": [] }, { "id": 673138, "author_name": "jackvial", "author_url": "", "post_date": "11/14/2019 15:03:53", "content": "For this kind of competition you really need to have a dedicated Deep Learning PC. A typical setup is Ubuntu, multicore cpu, SSD drive, lots of ram 32gb+ preferably, and a GPU like a GTX 1080ti, RTX 2070 etc...
\n\nUsing Kaggle notebooks is also a great option and they have GPU access but it's limited to 30 hrs per week.
", "votes": null, "replies": [ { "id": 673148, "author_name": "kevinehsani", "author_url": "", "post_date": "11/14/2019 15:27:36", "content": "Thank you very much Jack, this is what I call useful information and it would be a delight to get a negative mark from you. I have never hear about kaggle notebooks, I am very new to this. I take it, is a sight I sign into and I do not have to download the data any more. Learning is what is important to me. Thanks again.
", "votes": null, "replies": [] }, { "id": 673155, "author_name": "jackvial", "author_url": "", "post_date": "11/14/2019 15:38:22", "content": "Welcome, if you look at the tab bar with overview, data, notebooks and click on notebooks you can create a Jupyter Notebook that runs on Kaggle's servers, the competition dataset will be available to you within this environment.
", "votes": null, "replies": [] } ] }, { "id": 673153, "author_name": "jackvial", "author_url": "", "post_date": "11/14/2019 15:32:57", "content": "Try not be discouraged by the downvotes or take it too personally. If your questions is something you could have easily googled or spent a few minutes clicking around and exploring the interface to figure out then probably not a great idea to ask it here.
\n\nAlso, mentioning mac is probably going to guarantee a few downvotes! 😛 I use a mac for my day job but there is not much love for macs on here, the price being a big factor and the lack of support for CUDA and Nvidia GPUs.
", "votes": null, "replies": [ { "id": 673196, "author_name": "kevinehsani", "author_url": "", "post_date": "11/14/2019 16:46:32", "content": "I bought mac thinking it would support the latest technologies and I was shocked when I wanted to use it for GPU, but I am stuck with it as I paid so much but never again. Thanks again
", "votes": null, "replies": [] }, { "id": 673249, "author_name": "marekwyborski", "author_url": "", "post_date": "11/14/2019 18:05:35", "content": "You need to throw more money at it and buy a thunderbolt-pcie-gpu-enclosure like razer core-x or similar then put in a decent pcie-nvidia-gpu and boot linux anyways for deeplearning frameworks.
", "votes": null, "replies": [] }, { "id": 673260, "author_name": "kevinehsani", "author_url": "", "post_date": "11/14/2019 18:16:22", "content": "What kind of money are we talking about? One big problem I have is that I travel a lot and relocate completely so it has to be transportable.
", "votes": null, "replies": [] } ] }, { "id": 673307, "author_name": "doncalculator", "author_url": "", "post_date": "11/14/2019 19:50:11", "content": "You using a mac has nothing to do with the downvotes. I use a macbook everyday, I switched about a year ago because developing IOS apps on a hackintosh VM was a nightmare. For coding IMO it's the best notebook money can buy,\nbut for deep learning, it's only good to SSH into a workstation :)
\n\nTo answer your original question, the starter kernel creates 2D images from the data, you could create a dataset from that, and download it. Only about a few GB in size.
", "votes": null, "replies": [ { "id": 673347, "author_name": "kevinehsani", "author_url": "", "post_date": "11/14/2019 21:44:27", "content": "Thanks for the info. So there is a kernel called starter that you run for that and you do not need to convert the 2D images on your computer to full size data? I have not come across that kernel.
", "votes": null, "replies": [] }, { "id": 673592, "author_name": "doncalculator", "author_url": "", "post_date": "11/15/2019 07:45:05", "content": "Sorry it's called reference model, here you go: https://www.kaggle.com/gzuidhof/reference-model
", "votes": null, "replies": [] } ] }, { "id": 673358, "author_name": "artyomp", "author_url": "", "post_date": "11/14/2019 22:17:20", "content": "84 GB of data is simply too much.\nI just don't understand this. You bought a PC that doesn't have 100 GB HDD or SSD? For reference, I bought a 3 TB HDD in 2012 and I'm still using it. This year I bought a couple of 1 TB NVMe SSD disks, they really do help a lot when I'm dealing with Computer Vision challenges.
And yes, you absolutely need a computer with GPU. It's typically a Linux PC, Ubuntu or something similar. Ideally, you need more than one GPU, 1 TB SSD and (32 Gb * number of GPUs) of memory.
", "votes": null, "replies": [ { "id": 673775, "author_name": "kevinehsani", "author_url": "", "post_date": "11/15/2019 13:25:30", "content": "I have a mac with 250 SSD for everything, unfortunately I travel a lot and impossible to have pc, but I am dying to buy one. For now I have to be content on running code on clouds and other people servers
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "672156": "I was going to do this competition but 84 GB of data is simply too much. Is there a way around it or a similar kaggle with much, much lower data requirement? Thanks", "672475": "and sadly they use zip file instead of tar :( with tar file, you can decompress stream during download and filter files if you want", "673062": "for those that give negative points can they tell something intelligent and useful like you can run the data on their server, Please stop being in love with yourselves.", "673069": "Is it possible to run it on kaggle server? you know what are these negative points for?", "673138": "For this kind of competition you really need to have a dedicated Deep Learning PC. A typical setup is Ubuntu, multicore cpu, SSD drive, lots of ram 32gb+ preferably, and a GPU like a GTX 1080ti, RTX 2070 etc...\n\nUsing Kaggle notebooks is also a great option and they have GPU access but it's limited to 30 hrs per week.", "673148": "Thank you very much Jack, this is what I call useful information and it would be a delight to get a negative mark from you. I have never hear about kaggle notebooks, I am very new to this. I take it, is a sight I sign into and I do not have to download the data any more. Learning is what is important to me. Thanks again.", "673153": "Try not be discouraged by the downvotes or take it too personally. If your questions is something you could have easily googled or spent a few minutes clicking around and exploring the interface to figure out then probably not a great idea to ask it here.\n\nAlso, mentioning mac is probably going to guarantee a few downvotes! 😛 I use a mac for my day job but there is not much love for macs on here, the price being a big factor and the lack of support for CUDA and Nvidia GPUs.", "673155": "Welcome, if you look at the tab bar with overview, data, notebooks and click on notebooks you can create a Jupyter Notebook that runs on Kaggle's servers, the competition dataset will be available to you within this environment.", "673196": "I bought mac thinking it would support the latest technologies and I was shocked when I wanted to use it for GPU, but I am stuck with it as I paid so much but never again. Thanks again", "673249": "You need to throw more money at it and buy a thunderbolt-pcie-gpu-enclosure like razer core-x or similar then put in a decent pcie-nvidia-gpu and boot linux anyways for deeplearning frameworks.", "673260": "What kind of money are we talking about? One big problem I have is that I travel a lot and relocate completely so it has to be transportable.", "673307": "You using a mac has nothing to do with the downvotes. I use a macbook everyday, I switched about a year ago because developing IOS apps on a hackintosh VM was a nightmare. For coding IMO it's the best notebook money can buy,\nbut for deep learning, it's only good to SSH into a workstation :)\n\nTo answer your original question, the starter kernel creates 2D images from the data, you could create a dataset from that, and download it. Only about a few GB in size.", "673347": "Thanks for the info. So there is a kernel called starter that you run for that and you do not need to convert the 2D images on your computer to full size data? I have not come across that kernel.", "673358": "`84 GB of data is simply too much.`\nI just don't understand this. You bought a PC that doesn't have 100 GB HDD or SSD? For reference, I bought a 3 TB HDD in 2012 and I'm still using it. This year I bought a couple of 1 TB NVMe SSD disks, they really do help a lot when I'm dealing with Computer Vision challenges.\n\nAnd yes, you absolutely need a computer with GPU. It's typically a Linux PC, Ubuntu or something similar. Ideally, you need more than one GPU, 1 TB SSD and (32 Gb * number of GPUs) of memory.", "673592": "Sorry it's called reference model, here you go: https://www.kaggle.com/gzuidhof/reference-model", "673775": "I have a mac with 250 SSD for everything, unfortunately I travel a lot and impossible to have pc, but I am dying to buy one. For now I have to be content on running code on clouds and other people servers" }, "source": "meta" }