When working with high datasets and common libraries such as Pandas, and trying to plot or visualize parts of the data, we sometimes encounter memory issues simply by trying to plot a simple plot.
\nThis is of high importance when doing EDA since fast iteration means everything to data science.
Here is a short list of tricks and code examples for performing EDA on HUGE datasets.
\ndf = pd.read_csv(\"data.csv\", header=0, nrows=10000) \nimport gzip\nf = gzip.open('file.txt.gz', 'r')\nfile_content = f.read()\nf.close()\nimport pandas as pd\ndf.to_csv('./file_output.gz', index=False, compression='gzip') \nloaded_data = pd.read_csv('./file_output.gz') \nimport numpy as np\ndata = np.memmap('memmap', dtype='float32', mode='w+', shape=(10000, 1000000)\nchunksize argument of pandas, without loading it all into memory at oncechunksize = 10**6 # for example\nfor chunk in pd.read_csv('train.csv', chunksize=chunksize):\n #EDA\nIt’s often a possibility to perform what is needed with much fewer data. For example, a histogram plot can be constructed with a small evenly-spaced sample from the data rather than the entire dataset. Often times it’s unlikely that a plot of every element of the dataset would add much insight.
Example: Aggregating dataset into a smaller number of summary statistics that can be fitted into memory by chunks
\n\nNote bessel's correction for the streaming variance
\n
def get_summary_statistics(path):\n N = 0\n M2 = 0\n mean = None\n variance = None\n for chunk in pd.read_csv(path):\n N += len(chunk)\n for idx, col in enumerate(chunk.columns):\n data[idx] += chunk[col].values\n delta = data[idx] - mean\n mean += (delta/N)\n M2 += delta*(data[idx] - mean)\n variance = M2/(N-1) \n return mean, variance\ndf_subset = df_subset.sample(frac=0.5, random_state=1)\nchunk = pd.read_csv(csvfile, usecols=[0,1,2,3,4,5,6,7,8,9])\ndtype = {'col1': np.int32, 'col2':'float16', 'col3':'float64'}\nchunk = pd.read_csv(csvfile, dtype=dtype)\nfor chunk in pd.read_csv(csvfile, chunksize=chunksize):\n #Calculate the hash of the chunks\n chunk['hash'] = chunk.apply(lambda x: hash(tuple(x)), axis=1) # any non ram consuming hash function will work here\n all_chunks_hash.append(chunk)\n #Keep only unique rows indices\n unique_rows_idx = all_chunks_hash[all_chunks_hash['hash'].duplicated()==False].index\n udf = all_chunks_hash.loc[unique_rows_idx]\n\n\nThe largest hurdle to the use of Python for high-performance computing is that it is interpreted, not compiled. This fact is at the heart of every debate about the suitability
\n
df_binary_encoded = pd.get_dummies(binary_cols)\ndf_binary_cr_matrix = csr_matrix(df_binary_encoded.values)\ndf_binary_cr_matrix.to_pickle('binary_encoded.pkl')\nFor Category columns Converting categories to numeric and compressing the numeric representation by converting it to the smallest numeric type possible
For Numeric columns Compressing numeric columns
\n\nfor example for integers convert to smaller integers
\n
df_numeric['col'] = df_numeric['col'].astype('uint8')\n\n\nSee the famous \"reduce_mem_usage\" for more on downsampled data types
\n
Use memory-efficient built-in data structures
Discard prior history as soon as we can and always use del to delete old objects and then collect memory with gc.collect()
del old_object\ngc.collect()\nDebugging Memory Issues
\nmemory_profiler library to see where memory is getting eaten upfrom memory_profiler import memory_usage\nres = memory_usage(proc())\nindex = res.index(max(res))\nwith pd.open_csv('myfile.csv') as df:\n for dataset in df:\n proc(dataset)\nBonus Section: Bypassing Memory Limits with swapfiles
\nCommon limits that we encounter:
\nmemory limits set by the OS (such as those used with virtual machines).
Some applications limit how much memory can be used by a given process.
Python’s multiprocessing module will fail to fail with an OSError error if one attempts to launch a process that exceeds the amount of memory available.
Memory profile typically shows that most of the memory is used by object allocations and only a fraction is used for allocated data. Traditionally languages such as C have placed memory usage limits on the size of data that can be allocated, but this is receding as allocations may be performed out of the physical address space
Swapfiles on Linux-based systems (swapfiles are files that can act as a backup memory for our RAM memory)
To set up swapfile and extend our system's ram we simply can use the following commands (Example is 4GB swapfile on path: \"/swapfile\":
\nsudo fallocate -l 4G /swapfile \nsudo chmod 600 /swapfile\nsudo mkswap /swapfile\nsudo swapon /swapfile\nBonus: Multiprocessing with limited memory
\nNote: Here is a comparison between the two approaches:
\nimport joblib vs import multiprocessing: The multiprocessing module relies on spawning processes and passing messages between them, while the joblib module uses functions provided by the operating system. Both enable the parallel execution of tasks within a single computer and on a single shared-memory system, but they support different patterns of use. The choice of which to use often breaks down to the pattern of use (batch or online) and whether the data is shared between tasks.Bonus: GPU memory tricks
\nGPUs are not created equal and so have different capabilities in terms of the amount of memory they can access.
\nTo save up on GPU ram footprint we can use one of the following approaches:
To improve GPU ram usage, we use wrappers that replace CPU-based training routines with GPU-based equivalents, for example: cupy vs numpy:
\nimport cupy as cp\n\nx = cp.asarray(x)\ny = cp.asarray(y)\nOther Tricks
\nFancy Plots: For other kinds of plots that require loading all the data, downsampling can be useful to both allow plots to be generated faster than if the entire dataset were used and can also provide a means to view all of the datasets at once. Generating the best-looking plots will require some experimentation but some general tips include:
Resampling to a lower resolution on images and time series can reduce the dataset size by orders of magnitude.
Visualizing/Loading Top/Bottom Results By a Model If loading the entire dataset at once won’t fit in memory, we can draw plots of just the top results and then color/shade the other data points in a way that indicates that the rest of the data are less important.
Plotting Library: Changing to a different plotting library more suitable for large datasets some libraries provide plotting methods that are more memory efficient than others. If generating the plots takes too long and/or won’t fit in memory, a different plotting library might simply \"just work\" making the same type of plot.
Streaming numeric data from files does have some disadvantages in the event that we need a sorted copy of the data, for example. To address this need, it is possible to store the data in memory but use a different strategy for storing the data in order to reduce the amount of memory required. Python’s built-in sorted and heapq modules can be used to sort data in a streaming fashion that requires less memory overhead.
Distribution of the dataset can be summarized by histograms without first loading the entire dataset.
\ngenerate_histograms_from_data_stream() provides an example of the code, which generates the histograms one chunk at a time without requiring any data to be loaded into memory at once.
Just reduce_mem_usage it: Losing precision by storing numbers as bytes or even bits can dramatically reduce memory requirements at the expense of precision, in terms of EDA: this \"might\" be OK. still: be careful.
When working with high datasets and common libraries such as Pandas, and trying to plot or visualize parts of the data, we sometimes encounter memory issues simply by trying to plot a simple plot.
\nThis is of high importance when doing EDA since fast iteration means everything to data science.
Here is a short list of tricks and code examples for performing EDA on HUGE datasets.
\ndf = pd.read_csv(\"data.csv\", header=0, nrows=10000) \nimport gzip\nf = gzip.open('file.txt.gz', 'r')\nfile_content = f.read()\nf.close()\nimport pandas as pd\ndf.to_csv('./file_output.gz', index=False, compression='gzip') \nloaded_data = pd.read_csv('./file_output.gz') \nimport numpy as np\ndata = np.memmap('memmap', dtype='float32', mode='w+', shape=(10000, 1000000)\nchunksize argument of pandas, without loading it all into memory at oncechunksize = 10**6 # for example\nfor chunk in pd.read_csv('train.csv', chunksize=chunksize):\n #EDA\nIt’s often a possibility to perform what is needed with much fewer data. For example, a histogram plot can be constructed with a small evenly-spaced sample from the data rather than the entire dataset. Often times it’s unlikely that a plot of every element of the dataset would add much insight.
Example: Aggregating dataset into a smaller number of summary statistics that can be fitted into memory by chunks
\n\nNote bessel's correction for the streaming variance
\n
def get_summary_statistics(path):\n N = 0\n M2 = 0\n mean = None\n variance = None\n for chunk in pd.read_csv(path):\n N += len(chunk)\n for idx, col in enumerate(chunk.columns):\n data[idx] += chunk[col].values\n delta = data[idx] - mean\n mean += (delta/N)\n M2 += delta*(data[idx] - mean)\n variance = M2/(N-1) \n return mean, variance\ndf_subset = df_subset.sample(frac=0.5, random_state=1)\nchunk = pd.read_csv(csvfile, usecols=[0,1,2,3,4,5,6,7,8,9])\ndtype = {'col1': np.int32, 'col2':'float16', 'col3':'float64'}\nchunk = pd.read_csv(csvfile, dtype=dtype)\nfor chunk in pd.read_csv(csvfile, chunksize=chunksize):\n #Calculate the hash of the chunks\n chunk['hash'] = chunk.apply(lambda x: hash(tuple(x)), axis=1) # any non ram consuming hash function will work here\n all_chunks_hash.append(chunk)\n #Keep only unique rows indices\n unique_rows_idx = all_chunks_hash[all_chunks_hash['hash'].duplicated()==False].index\n udf = all_chunks_hash.loc[unique_rows_idx]\n\n\nThe largest hurdle to the use of Python for high-performance computing is that it is interpreted, not compiled. This fact is at the heart of every debate about the suitability
\n
df_binary_encoded = pd.get_dummies(binary_cols)\ndf_binary_cr_matrix = csr_matrix(df_binary_encoded.values)\ndf_binary_cr_matrix.to_pickle('binary_encoded.pkl')\nFor Category columns Converting categories to numeric and compressing the numeric representation by converting it to the smallest numeric type possible
For Numeric columns Compressing numeric columns
\n\nfor example for integers convert to smaller integers
\n
df_numeric['col'] = df_numeric['col'].astype('uint8')\n\n\nSee the famous \"reduce_mem_usage\" for more on downsampled data types
\n
Use memory-efficient built-in data structures
Discard prior history as soon as we can and always use del to delete old objects and then collect memory with gc.collect()
del old_object\ngc.collect()\nDebugging Memory Issues
\nmemory_profiler library to see where memory is getting eaten upfrom memory_profiler import memory_usage\nres = memory_usage(proc())\nindex = res.index(max(res))\nwith pd.open_csv('myfile.csv') as df:\n for dataset in df:\n proc(dataset)\nBonus Section: Bypassing Memory Limits with swapfiles
\nCommon limits that we encounter:
\nmemory limits set by the OS (such as those used with virtual machines).
Some applications limit how much memory can be used by a given process.
Python’s multiprocessing module will fail to fail with an OSError error if one attempts to launch a process that exceeds the amount of memory available.
Memory profile typically shows that most of the memory is used by object allocations and only a fraction is used for allocated data. Traditionally languages such as C have placed memory usage limits on the size of data that can be allocated, but this is receding as allocations may be performed out of the physical address space
Swapfiles on Linux-based systems (swapfiles are files that can act as a backup memory for our RAM memory)
To set up swapfile and extend our system's ram we simply can use the following commands (Example is 4GB swapfile on path: \"/swapfile\":
\nsudo fallocate -l 4G /swapfile \nsudo chmod 600 /swapfile\nsudo mkswap /swapfile\nsudo swapon /swapfile\nBonus: Multiprocessing with limited memory
\nNote: Here is a comparison between the two approaches:
\nimport joblib vs import multiprocessing: The multiprocessing module relies on spawning processes and passing messages between them, while the joblib module uses functions provided by the operating system. Both enable the parallel execution of tasks within a single computer and on a single shared-memory system, but they support different patterns of use. The choice of which to use often breaks down to the pattern of use (batch or online) and whether the data is shared between tasks.Bonus: GPU memory tricks
\nGPUs are not created equal and so have different capabilities in terms of the amount of memory they can access.
\nTo save up on GPU ram footprint we can use one of the following approaches:
To improve GPU ram usage, we use wrappers that replace CPU-based training routines with GPU-based equivalents, for example: cupy vs numpy:
\nimport cupy as cp\n\nx = cp.asarray(x)\ny = cp.asarray(y)\nOther Tricks
\nFancy Plots: For other kinds of plots that require loading all the data, downsampling can be useful to both allow plots to be generated faster than if the entire dataset were used and can also provide a means to view all of the datasets at once. Generating the best-looking plots will require some experimentation but some general tips include:
Resampling to a lower resolution on images and time series can reduce the dataset size by orders of magnitude.
Visualizing/Loading Top/Bottom Results By a Model If loading the entire dataset at once won’t fit in memory, we can draw plots of just the top results and then color/shade the other data points in a way that indicates that the rest of the data are less important.
Plotting Library: Changing to a different plotting library more suitable for large datasets some libraries provide plotting methods that are more memory efficient than others. If generating the plots takes too long and/or won’t fit in memory, a different plotting library might simply \"just work\" making the same type of plot.
Streaming numeric data from files does have some disadvantages in the event that we need a sorted copy of the data, for example. To address this need, it is possible to store the data in memory but use a different strategy for storing the data in order to reduce the amount of memory required. Python’s built-in sorted and heapq modules can be used to sort data in a streaming fashion that requires less memory overhead.
Distribution of the dataset can be summarized by histograms without first loading the entire dataset.
\ngenerate_histograms_from_data_stream() provides an example of the code, which generates the histograms one chunk at a time without requiring any data to be loaded into memory at once.
Just reduce_mem_usage it: Losing precision by storing numbers as bytes or even bits can dramatically reduce memory requirements at the expense of precision, in terms of EDA: this \"might\" be OK. still: be careful.
Very useful tricks. Thanks for sharing
", "rawMarkdown": "Very useful tricks. Thanks for sharing", "votes": null }, { "id": "1817783", "postDate": "06/11/2022 19:37:28", "content": "This is one of the best and most informative posts I have seen on this platform lately.
", "rawMarkdown": "This is one of the best and most informative posts I have seen on this platform lately.", "votes": null }, { "id": "1817788", "postDate": "06/11/2022 19:51:32", "content": "Thank you! I'm glad I was able to help :)
", "rawMarkdown": "Thank you! I'm glad I was able to help :)", "votes": null }, { "id": "1817790", "postDate": "06/11/2022 19:52:10", "content": "Thanks! If you got any more you know, I will be more then happy to know about them!
", "rawMarkdown": "Thanks! If you got any more you know, I will be more then happy to know about them!", "votes": null }, { "id": "1819329", "postDate": "06/13/2022 16:43:06", "content": "Nice! Didn't understand half of them, but now I have something to work on ;)
", "rawMarkdown": "Nice! Didn't understand half of them, but now I have something to work on ;)", "votes": null }, { "id": "1819860", "postDate": "06/14/2022 07:31:36", "content": "This should be a blog post @thedevastator ✨
\nSuper useful for competitions that have large datasets!
Great insights on the challenges that working with large datasets can present. Really liked the ideas around optimal memory usage techniques. A grasp of the message provided by a sample of the data can help with the overall story.
", "rawMarkdown": "Great insights on the challenges that working with large datasets can present. Really liked the ideas around optimal memory usage techniques. A grasp of the message provided by a sample of the data can help with the overall story.", "votes": null }, { "id": "1820288", "postDate": "06/14/2022 13:57:12", "content": "Amazing, thanks for sharing !
", "rawMarkdown": "Amazing, thanks for sharing !", "votes": null }, { "id": "1820364", "postDate": "06/14/2022 15:05:35", "content": "Thanks for sharing. It is so useful! Have a nice Data Trip!
", "rawMarkdown": "Thanks for sharing. It is so useful! Have a nice Data Trip!", "votes": null }, { "id": "1820556", "postDate": "06/14/2022 18:31:40", "content": "Amazing explanations to handle big datasets. Love it.
\nMaybe you can also use datatable for loading the data faster. I made a notebook for this one.
\nLoading your data Faster
Very Helpful Tips !! Thanks for sharing :)
", "rawMarkdown": "Very Helpful Tips !! Thanks for sharing :)", "votes": null }, { "id": "1821443", "postDate": "06/15/2022 14:38:19", "content": "Nice tricks @thedevastator and congratulations on becoming kaggle expert. Very impressive the way your going forward
", "rawMarkdown": "Nice tricks @thedevastator and congratulations on becoming **kaggle expert**. Very impressive the way your going forward", "votes": null }, { "id": "1821481", "postDate": "06/15/2022 15:10:37", "content": "Amazing tip, thanks!
", "rawMarkdown": "Amazing tip, thanks!", "votes": null }, { "id": "1906573", "postDate": "08/20/2022 02:16:35", "content": "Most helpful tips!! I'm faced Memory issue, but it might solve my error, I think.😃👍
", "rawMarkdown": "Most helpful tips!! I'm faced Memory issue, but it might solve my error, I think.😃👍", "votes": null } ], "comments": [ { "id": 1817776, "author_name": "nymfree", "author_url": "", "post_date": "06/11/2022 19:22:39", "content": "Very useful tricks. Thanks for sharing
", "votes": null, "replies": [ { "id": 1817790, "author_name": "thedevastator", "author_url": "", "post_date": "06/11/2022 19:52:10", "content": "Thanks! If you got any more you know, I will be more then happy to know about them!
", "votes": null, "replies": [] } ] }, { "id": 1817783, "author_name": "ravi20076", "author_url": "", "post_date": "06/11/2022 19:37:28", "content": "This is one of the best and most informative posts I have seen on this platform lately.
", "votes": null, "replies": [ { "id": 1817788, "author_name": "thedevastator", "author_url": "", "post_date": "06/11/2022 19:51:32", "content": "Thank you! I'm glad I was able to help :)
", "votes": null, "replies": [] } ] }, { "id": 1819329, "author_name": "alonsourbano", "author_url": "", "post_date": "06/13/2022 16:43:06", "content": "Nice! Didn't understand half of them, but now I have something to work on ;)
", "votes": null, "replies": [] }, { "id": 1819860, "author_name": "ruchi798", "author_url": "", "post_date": "06/14/2022 07:31:36", "content": "This should be a blog post @thedevastator ✨
\nSuper useful for competitions that have large datasets!
Great insights on the challenges that working with large datasets can present. Really liked the ideas around optimal memory usage techniques. A grasp of the message provided by a sample of the data can help with the overall story.
", "votes": null, "replies": [] }, { "id": 1820288, "author_name": "paulojunqueira", "author_url": "", "post_date": "06/14/2022 13:57:12", "content": "Amazing, thanks for sharing !
", "votes": null, "replies": [] }, { "id": 1820364, "author_name": "sikymddeberias", "author_url": "", "post_date": "06/14/2022 15:05:35", "content": "Thanks for sharing. It is so useful! Have a nice Data Trip!
", "votes": null, "replies": [] }, { "id": 1820556, "author_name": "naiborhujosua", "author_url": "", "post_date": "06/14/2022 18:31:40", "content": "Amazing explanations to handle big datasets. Love it.
\nMaybe you can also use datatable for loading the data faster. I made a notebook for this one.
\nLoading your data Faster
Very Helpful Tips !! Thanks for sharing :)
", "votes": null, "replies": [] }, { "id": 1821443, "author_name": "thamotharan", "author_url": "", "post_date": "06/15/2022 14:38:19", "content": "Nice tricks @thedevastator and congratulations on becoming kaggle expert. Very impressive the way your going forward
", "votes": null, "replies": [] }, { "id": 1821481, "author_name": "knedliky", "author_url": "", "post_date": "06/15/2022 15:10:37", "content": "Amazing tip, thanks!
", "votes": null, "replies": [] }, { "id": 1906573, "author_name": "jackwilliams3rd", "author_url": "", "post_date": "08/20/2022 02:16:35", "content": "Most helpful tips!! I'm faced Memory issue, but it might solve my error, I think.😃👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1817773": "# A List of EDA tricks when memory is limited\n\nWhen working with high datasets and common libraries such as Pandas, and trying to plot or visualize parts of the data, we sometimes encounter memory issues simply by trying to plot a simple plot. \nThis is of high importance when doing EDA since fast iteration means everything to data science.\n\nHere is a short list of tricks and code examples for performing EDA on HUGE datasets. \n\n- **Load only a subsample of the data into memory**\nWe need to be careful when using subsamples of the data since we are taking the risk of sampling a biased sample. But it's pretty useful early on the EDA round to avoid choking the computer. I have chosen to load the first 10k data points of the dataset.\n\n```python\ndf = pd.read_csv(\"data.csv\", header=0, nrows=10000) \n```\n- **Keep your data in compressed format, and decompress on the fly as needed**\n\n```python\nimport gzip\nf = gzip.open('file.txt.gz', 'r')\nfile_content = f.read()\nf.close()\n```\n- **Store data as text with a compression scheme**\nSaving the data as text instead of its original format can be a good choice since we don't work like GP us but with libraries instead. We can opt for saving the data as text files. \nIn the case of large datasets, it's recommended to use a compression algorithm to reduce space.\n\n```python\nimport pandas as pd\ndf.to_csv('./file_output.gz', index=False, compression='gzip') \nloaded_data = pd.read_csv('./file_output.gz') \n```\n\n- **Use numpy's builtin tricks**\nUse features supported by numpy to work with dataset that doesn’t fit into memory.\nYou can create an array or matrix that is too big for main memory as an array that is physically stored outside the memory. Being stored outside the main memory, the Numpy array object is an efficient way to limit the memory usage from being filled with the contents of the physical file in its entirety.\n\n```python\nimport numpy as np\ndata = np.memmap('memmap', dtype='float32', mode='w+', shape=(10000, 1000000)\n```\n\n- **Use streaming** methods to do EDA one chank at a time by using `chunksize` argument of pandas, without loading it all into memory at once\n\n```python\nchunksize = 10**6 # for example\nfor chunk in pd.read_csv('train.csv', chunksize=chunksize):\n #EDA\n```\n\n- It’s often a possibility to perform what is needed with much fewer data. For example, a histogram plot can be constructed with a small evenly-spaced sample from the data rather than the entire dataset. Often times it’s unlikely that a plot of every element of the dataset would add much insight.\n\n\n- **Example:** Aggregating dataset into a smaller number of summary statistics that can be fitted into memory by chunks \n\n> Note bessel's correction for the streaming variance \n\n```python\ndef get_summary_statistics(path):\n N = 0\n M2 = 0\n mean = None\n variance = None\n for chunk in pd.read_csv(path):\n N += len(chunk)\n for idx, col in enumerate(chunk.columns):\n data[idx] += chunk[col].values\n delta = data[idx] - mean\n mean += (delta/N)\n M2 += delta*(data[idx] - mean)\n variance = M2/(N-1) \n return mean, variance\n```\n\n- **Randomly sample** subsets of rows and columns and then run your EDA and plot on this subset to save on memory footprint:\n\n```python\ndf_subset = df_subset.sample(frac=0.5, random_state=1)\n```\n\n- **Load only a subset** of columns each time and run your EDA and plot on this subset to save on memory footprint\n\n```python\nchunk = pd.read_csv(csvfile, usecols=[0,1,2,3,4,5,6,7,8,9])\n```\n\n- **Load into lighter** numeric formats \n```python\ndtype = {'col1': np.int32, 'col2':'float16', 'col3':'float64'}\nchunk = pd.read_csv(csvfile, dtype=dtype)\n```\n\n- **Don't load duplicates** by first hashing rows chunk by chunk\n\n```python\nfor chunk in pd.read_csv(csvfile, chunksize=chunksize):\n #Calculate the hash of the chunks\n chunk['hash'] = chunk.apply(lambda x: hash(tuple(x)), axis=1) # any non ram consuming hash function will work here\n all_chunks_hash.append(chunk)\n #Keep only unique rows indices\n unique_rows_idx = all_chunks_hash[all_chunks_hash['hash'].duplicated()==False].index\n udf = all_chunks_hash.loc[unique_rows_idx]\n```\n\n> The largest hurdle to the use of Python for high-performance computing is that it is interpreted, not compiled. This fact is at the heart of every debate about the suitability\n\n- **For Binary columns** If a dataset has many binary columns, use csr_matrix for a more compact representation\n\n```python\ndf_binary_encoded = pd.get_dummies(binary_cols)\ndf_binary_cr_matrix = csr_matrix(df_binary_encoded.values)\ndf_binary_cr_matrix.to_pickle('binary_encoded.pkl')\n```\n\n- **For Category columns** Converting categories to numeric and compressing the numeric representation by converting it to the smallest numeric type possible\n\n- **For Numeric columns** Compressing numeric columns\n\n> for example for integers convert to smaller integers\n\n```python\ndf_numeric['col'] = df_numeric['col'].astype('uint8')\n```\n> See the famous \"reduce_mem_usage\" for more on downsampled data types\n\n- **Use memory-efficient built-in data structures**\n\n- Discard prior history as soon as we can and always use `del` to delete old objects and then collect memory with `gc.collect()`\n\n```python\ndel old_object\ngc.collect()\n```\n\n- In General, try not to use visualization methods that require the sequence length to be known beforehand or visualization methods that are \"fancy\" in such cases that they are not \"a must\". \n\n\n**Debugging Memory Issues**\n\n- Use the Python `memory_profiler` library to see where memory is getting eaten up\n```python\nfrom memory_profiler import memory_usage\nres = memory_usage(proc())\nindex = res.index(max(res))\nwith pd.open_csv('myfile.csv') as df:\n for dataset in df:\n proc(dataset)\n```\n\n- For example, is it eating memory up with your EDA and visualization code, or is it being eaten up in the loading process? Or is the file being stored on disk in an inefficient format (sparsed areas in the data take up just as much space as sparse areas in the data)\n\n\n\n**Bonus Section: Bypassing Memory Limits with swapfiles**\n\nCommon limits that we encounter:\n- memory limits set by the OS (such as those used with virtual machines).\n- Some applications limit how much memory can be used by a given process.\n- Python’s multiprocessing module will fail to fail with an OSError error if one attempts to launch a process that exceeds the amount of memory available.\n\n- Memory profile typically shows that most of the memory is used by object allocations and only a fraction is used for allocated data. Traditionally languages such as C have placed memory usage limits on the size of data that can be allocated, but this is receding as allocations may be performed out of the physical address space\n\n- Swapfiles on Linux-based systems (swapfiles are files that can act as a backup memory for our RAM memory)\n\nTo set up swapfile and extend our system's ram we simply can use the following commands (Example is 4GB swapfile on path: \"/swapfile\":\n```\nsudo fallocate -l 4G /swapfile \nsudo chmod 600 /swapfile\nsudo mkswap /swapfile\nsudo swapon /swapfile\n```\n\n**Bonus: Multiprocessing with limited memory**\n\n- Use joblib to use multiple cores while the dataset is still loaded on disk.\n- Use **multiprocessing.Pool** to load the data and distribute it between cores. \n\n**Note:** Here is a comparison between the two approaches:\n\n- **`import joblib`** vs **`import multiprocessing`**: The multiprocessing module relies on spawning processes and passing messages between them, while the joblib module uses functions provided by the operating system. Both enable the parallel execution of tasks within a single computer and on a single shared-memory system, but they support different patterns of use. The choice of which to use often breaks down to the pattern of use (batch or online) and whether the data is shared between tasks.\n\n**Bonus: GPU memory tricks**\nGPUs are not created equal and so have different capabilities in terms of the amount of memory they can access. \nTo save up on GPU ram footprint we can use one of the following approaches:\n- **Batching** loading data in smaller (cpu) batches to the gpu and in parallel loading next batch from disk to avoid waiting on the gpu with no data to process.\n- **Downsample** the data to a smaller size\n- **Compress** the data to a size that will fit on the gpu.\n\nTo improve GPU ram usage, we use wrappers that replace CPU-based training routines with GPU-based equivalents, for example: **cupy vs numpy**:\n```python\nimport cupy as cp\n\nx = cp.asarray(x)\ny = cp.asarray(y)\n\n```\n\n**Other Tricks**\n\n- **Fancy Plots:** For other kinds of plots that require loading all the data, downsampling can be useful to both allow plots to be generated faster than if the entire dataset were used and can also provide a means to view all of the datasets at once. Generating the best-looking plots will require some experimentation but some general tips include:\n\n- **Resampling** to a lower resolution on images and time series can reduce the dataset size by orders of magnitude.\n\n- **Visualizing/Loading Top/Bottom Results By a Model** If loading the entire dataset at once won’t fit in memory, we can draw plots of just the top results and then color/shade the other data points in a way that indicates that the rest of the data are less important.\n\n- **Plotting Library:** Changing to a different plotting library more suitable for large datasets some libraries provide plotting methods that are more memory efficient than others. If generating the plots takes too long and/or won’t fit in memory, a different plotting library might simply \"just work\" making the same type of plot.\n\n- **Streaming numeric** data from files does have some disadvantages in the event that we need a sorted copy of the data, for example. To address this need, it is possible to store the data in memory but use a different strategy for storing the data in order to reduce the amount of memory required. Python’s built-in sorted and `heapq` modules can be used to sort data in a streaming fashion that requires less memory overhead.\n\n- **Distribution** of the dataset can be summarized by histograms without first loading the entire dataset.\ngenerate_histograms_from_data_stream() provides an example of the code, which generates the histograms one chunk at a time without requiring any data to be loaded into memory at once.\n\n- **Just reduce_mem_usage it:** Losing precision by storing numbers as bytes or even bits can dramatically reduce memory requirements at the expense of precision, in terms of EDA: this \"might\" be OK. still: be careful.", "1817776": "Very useful tricks. Thanks for sharing", "1817783": "This is one of the best and most informative posts I have seen on this platform lately.", "1817788": "Thank you! I'm glad I was able to help :)", "1817790": "Thanks! If you got any more you know, I will be more then happy to know about them!", "1819329": "Nice! Didn't understand half of them, but now I have something to work on ;)", "1819860": "This should be a blog post @thedevastator ✨\nSuper useful for competitions that have large datasets!", "1819876": "Great insights on the challenges that working with large datasets can present. Really liked the ideas around optimal memory usage techniques. A grasp of the message provided by a sample of the data can help with the overall story.", "1820288": "Amazing, thanks for sharing !", "1820364": "Thanks for sharing. It is so useful! Have a nice Data Trip!", "1820556": "Amazing explanations to handle big datasets. Love it.\nMaybe you can also use datatable for loading the data faster. I made a notebook for this one. \n[Loading your data Faster](https://www.kaggle.com/code/naiborhujosua/loading-your-data-faster)", "1821425": "Very Helpful Tips !! Thanks for sharing :)", "1821443": "Nice tricks @thedevastator and congratulations on becoming **kaggle expert**. Very impressive the way your going forward", "1821481": "Amazing tip, thanks!", "1906573": "Most helpful tips!! I'm faced Memory issue, but it might solve my error, I think.😃👍" }, "source": "meta" }