When I run the kernel with code:\ntrain_data = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_file': np.float64})，I get the message \"The kernel appears to have died. It will restart automatically\", and the kernel died immediately.\nEven though I change the code like this:\ntrain_data = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_file': np.float32}), the same message occurs too.\nI have tried a lot of times, anyone can tell me how to solve the problem?\nBy the way: I didn't get the trouble in the past days until yesterday.
When I run the kernel with code:\ntrain_data = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_file': np.float64})，I get the message \"The kernel appears to have died. It will restart automatically\", and the kernel died immediately.\nEven though I change the code like this:\ntrain_data = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_file': np.float32}), the same message occurs too.\nI have tried a lot of times, anyone can tell me how to solve the problem?\nBy the way: I didn't get the trouble in the past days until yesterday.
I ran it again, and this time I got the correct result. But I just want to know why the problem occurs and how I can sove it if I get trouble with it again.
", "rawMarkdown": "I ran it again, and this time I got the correct result. But I just want to know why the problem occurs and how I can sove it if I get trouble with it again.", "votes": null }, { "id": "531556", "postDate": "05/15/2019 06:08:57", "content": "Please!!! I get the trouble again...
", "rawMarkdown": "Please!!! I get the trouble again...", "votes": null }, { "id": "531559", "postDate": "05/15/2019 06:10:02", "content": "I find that it is because the RAM up to 16G. what's wrong with it?
", "rawMarkdown": "I find that it is because the RAM up to 16G. what's wrong with it?", "votes": null }, { "id": "531614", "postDate": "05/15/2019 07:45:35", "content": "I could not load the LANL dataset in my kernel. I think we may get the same problem.
", "rawMarkdown": "I could not load the LANL dataset in my kernel. I think we may get the same problem.", "votes": null }, { "id": "531628", "postDate": "05/15/2019 08:27:58", "content": "The other column is called time_to_failure not time_to_file
Thank you very much!
", "rawMarkdown": "Thank you very much!", "votes": null }, { "id": "531651", "postDate": "05/15/2019 09:22:01", "content": "Maybe you can tell us about your problem.
", "rawMarkdown": "Maybe you can tell us about your problem.", "votes": null }, { "id": "531710", "postDate": "05/15/2019 11:38:09", "content": "16GB of RAM is not enough to fruitfully process the entire train.csv at once, you should consider using the chunksize parameter of the pandas.read_csv method. If needed, use the partial_fit method of a suitable model.
Thank you. I thought anthoer method to read the original data with float64:\n```\n%%time
\n\ntrain_data_ad = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.int16, usecols=['acoustic_data'])\ntrain_data_ttf = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.float64, usecols=['time_to_failure'])\ntrain_data = pd.concat([train_data_ad, train_data_ttf], axis=1)\ndel train_data_ad\ndel train_data_ttf\n```
", "rawMarkdown": "Thank you. I thought anthoer method to read the original data with float64:\n```\n%%time\n\ntrain_data_ad = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.int16, usecols=['acoustic_data'])\ntrain_data_ttf = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.float64, usecols=['time_to_failure'])\ntrain_data = pd.concat([train_data_ad, train_data_ttf], axis=1)\ndel train_data_ad\ndel train_data_ttf\n```", "votes": null }, { "id": "531764", "postDate": "05/15/2019 13:58:00", "content": "I got the error message: Failed to load input data, this could result in the Kernel failing to load.
", "rawMarkdown": "I got the error message: Failed to load input data, this could result in the Kernel failing to load.", "votes": null }, { "id": "531765", "postDate": "05/15/2019 14:03:17", "content": "I also got the error from time to time. But it didn't influence the runing of the kernel.
", "rawMarkdown": "I also got the error from time to time. But it didn't influence the runing of the kernel.", "votes": null }, { "id": "531770", "postDate": "05/15/2019 14:15:50", "content": "This doesn't make it easier, if anything it gets worse because you are reading in the whole amount of original data and after that you are making another copy of it just before deleting! The memory requirements increase here actually rather than decreasing. See, bifurcating the data into two dataframes does not lower the memory requirement, all the data is being loaded anyway. This is why reading in chunks is the way to go if memory is low.
", "rawMarkdown": "This doesn't make it easier, if anything it gets worse because you are reading in the whole amount of original data and after that you are making another copy of it just before deleting! The memory requirements increase here actually rather than decreasing. See, bifurcating the data into two dataframes does not lower the memory requirement, all the data is being loaded anyway. This is why reading in chunks is the way to go if memory is low.", "votes": null }, { "id": "531784", "postDate": "05/15/2019 14:36:25", "content": "Yeah, but I found a fact that the function pd.read_csv() occupy higher RAM when we running it. When the function finished, the RAM of occupation will be lower. So I bifurcating the data into two dataframes. In fact, it works. Anyway, thank you very much!
I checked it out and it does give some leg-room for sure, but hits the limits after that. Perhaps you could try reading in chunks if you are getting errors often. This is just the memory management part though, I wish you all the best for the competition!
", "rawMarkdown": "I checked it out and it does give some leg-room for sure, but hits the limits after that. Perhaps you could try reading in chunks if you are getting errors often. This is just the memory management part though, I wish you all the best for the competition!", "votes": null }, { "id": "532055", "postDate": "05/16/2019 05:28:28", "content": "It’s probably because of the huge size of the train data. Try to load a small portion just for checking.
", "rawMarkdown": "It’s probably because of the huge size of the train data. Try to load a small portion just for checking.", "votes": null }, { "id": "532224", "postDate": "05/16/2019 12:55:49", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": null }, { "id": "532225", "postDate": "05/16/2019 12:57:48", "content": "Thank you. I wish you all the best for the competition too!
", "rawMarkdown": "Thank you. I wish you all the best for the competition too!", "votes": null } ], "comments": [ { "id": 531481, "author_name": "", "author_url": "", "post_date": "05/15/2019 01:57:43", "content": "I ran it again, and this time I got the correct result. But I just want to know why the problem occurs and how I can sove it if I get trouble with it again.
", "votes": null, "replies": [] }, { "id": 531556, "author_name": "", "author_url": "", "post_date": "05/15/2019 06:08:57", "content": "Please!!! I get the trouble again...
", "votes": null, "replies": [] }, { "id": 531559, "author_name": "", "author_url": "", "post_date": "05/15/2019 06:10:02", "content": "I find that it is because the RAM up to 16G. what's wrong with it?
", "votes": null, "replies": [ { "id": 531710, "author_name": "kishupro", "author_url": "", "post_date": "05/15/2019 11:38:09", "content": "16GB of RAM is not enough to fruitfully process the entire train.csv at once, you should consider using the chunksize parameter of the pandas.read_csv method. If needed, use the partial_fit method of a suitable model.
Thank you. I thought anthoer method to read the original data with float64:\n```\n%%time
\n\ntrain_data_ad = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.int16, usecols=['acoustic_data'])\ntrain_data_ttf = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.float64, usecols=['time_to_failure'])\ntrain_data = pd.concat([train_data_ad, train_data_ttf], axis=1)\ndel train_data_ad\ndel train_data_ttf\n```
", "votes": null, "replies": [] }, { "id": 531770, "author_name": "kishupro", "author_url": "", "post_date": "05/15/2019 14:15:50", "content": "This doesn't make it easier, if anything it gets worse because you are reading in the whole amount of original data and after that you are making another copy of it just before deleting! The memory requirements increase here actually rather than decreasing. See, bifurcating the data into two dataframes does not lower the memory requirement, all the data is being loaded anyway. This is why reading in chunks is the way to go if memory is low.
", "votes": null, "replies": [] }, { "id": 531784, "author_name": "", "author_url": "", "post_date": "05/15/2019 14:36:25", "content": "Yeah, but I found a fact that the function pd.read_csv() occupy higher RAM when we running it. When the function finished, the RAM of occupation will be lower. So I bifurcating the data into two dataframes. In fact, it works. Anyway, thank you very much!
I checked it out and it does give some leg-room for sure, but hits the limits after that. Perhaps you could try reading in chunks if you are getting errors often. This is just the memory management part though, I wish you all the best for the competition!
", "votes": null, "replies": [] }, { "id": 532225, "author_name": "", "author_url": "", "post_date": "05/16/2019 12:57:48", "content": "Thank you. I wish you all the best for the competition too!
", "votes": null, "replies": [] } ] }, { "id": 531614, "author_name": "dysonlin", "author_url": "", "post_date": "05/15/2019 07:45:35", "content": "I could not load the LANL dataset in my kernel. I think we may get the same problem.
", "votes": null, "replies": [ { "id": 531651, "author_name": "", "author_url": "", "post_date": "05/15/2019 09:22:01", "content": "Maybe you can tell us about your problem.
", "votes": null, "replies": [] }, { "id": 531764, "author_name": "dysonlin", "author_url": "", "post_date": "05/15/2019 13:58:00", "content": "I got the error message: Failed to load input data, this could result in the Kernel failing to load.
", "votes": null, "replies": [] }, { "id": 531765, "author_name": "", "author_url": "", "post_date": "05/15/2019 14:03:17", "content": "I also got the error from time to time. But it didn't influence the runing of the kernel.
", "votes": null, "replies": [] } ] }, { "id": 531628, "author_name": "returnofsputnik", "author_url": "", "post_date": "05/15/2019 08:27:58", "content": "The other column is called time_to_failure not time_to_file
Thank you very much!
", "votes": null, "replies": [] } ] }, { "id": 532055, "author_name": "yunlongzhang", "author_url": "", "post_date": "05/16/2019 05:28:28", "content": "It’s probably because of the huge size of the train data. Try to load a small portion just for checking.
", "votes": null, "replies": [ { "id": 532224, "author_name": "", "author_url": "", "post_date": "05/16/2019 12:55:49", "content": "Thank you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "531478": "When I run the kernel with code:\n`train_data = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_file': np.float64})`，I get the message \"The kernel appears to have died. It will restart automatically\", and the kernel died immediately.\nEven though I change the code like this:\n`train_data = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype={'acoustic_data': np.int16, 'time_to_file': np.float32}),` the same message occurs too.\nI have tried a lot of times, anyone can tell me how to solve the problem?\nBy the way: I didn't get the trouble in the past days until yesterday.", "531481": "I ran it again, and this time I got the correct result. But I just want to know why the problem occurs and how I can sove it if I get trouble with it again.", "531556": "Please!!! I get the trouble again...", "531559": "I find that it is because the RAM up to 16G. what's wrong with it?", "531614": "I could not load the LANL dataset in my kernel. I think we may get the same problem.", "531628": "The other column is called `time_to_failure` not `time_to_file`", "531650": "Thank you very much!", "531651": "Maybe you can tell us about your problem.", "531710": "`16GB` of `RAM` is not enough to fruitfully process the entire `train.csv` at once, you should consider using the `chunksize` parameter of the `pandas.read_csv` method. If needed, use the `partial_fit` method of a suitable model.", "531741": "Thank you. I thought anthoer method to read the original data with float64:\n```\n%%time\n\ntrain_data_ad = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.int16, usecols=['acoustic_data'])\ntrain_data_ttf = pd.read_csv(os.path.join(PATH, 'train.csv'), dtype=np.float64, usecols=['time_to_failure'])\ntrain_data = pd.concat([train_data_ad, train_data_ttf], axis=1)\ndel train_data_ad\ndel train_data_ttf\n```", "531764": "I got the error message: Failed to load input data, this could result in the Kernel failing to load.", "531765": "I also got the error from time to time. But it didn't influence the runing of the kernel.", "531770": "This doesn't make it easier, if anything it gets worse because you are reading in the whole amount of original data and after that you are making another copy of it just before deleting! The memory requirements increase here actually rather than decreasing. See, bifurcating the data into two dataframes does not lower the memory requirement, all the data is being loaded anyway. This is why reading in chunks is the way to go if memory is low.", "531784": "Yeah, but I found a fact that the function `pd.read_csv()` occupy higher RAM when we running it. When the function finished, the RAM of occupation will be lower. So I bifurcating the data into two dataframes. In fact, it works. Anyway, thank you very much!", "531811": "I checked it out and it does give some leg-room for sure, but hits the limits after that. Perhaps you could try reading in chunks if you are getting errors often. This is just the memory management part though, I wish you all the best for the competition!", "532055": "It’s probably because of the huge size of the train data. Try to load a small portion just for checking.", "532224": "Thank you.", "532225": "Thank you. I wish you all the best for the competition too!" }, "source": "meta" }