{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3136,"databundleVersionId":26502,"sourceType":"competition"},{"sourceId":5407,"databundleVersionId":868283,"sourceType":"competition"},{"sourceId":19018,"databundleVersionId":2703900,"sourceType":"competition"}],"dockerImageVersionId":29908,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n![](https://translucentcomputing.com/wp-content/uploads/2019/12/performance.png)\nWhile working in python jupyter notebooks using pandas with small datasets (100MB), performance degradation is not seen however when we work on large datasets (1 GB or above), performance issue of the notebook is clearly seen as it takes longer time for the cell to execute resulting in either longer execution time of the cell or the notebook failing due to insufficient memory.\n\nTools like Spark can handle large datasets (100's of GB to TB), taking full advantage of their capabilities usually requires more expensive hardware which is practically not possible while we try to work POC's or doing research. And unlike pandas, they lack rich feature sets for high quality data cleaning, exploration, and analysis. For medium-sized data, we’re better off trying to get more out of pandas, rather than switching to a different tool.\n\n### Dataset\n\nI have choosen a existing kaggle dataset from competition data namely **Jigsaw Multilingual Toxic Comment Classification** which has sizable data to demonstrate what I am trying to convey in this notebook.\n\n#### Brief about the data:\n\nJigsaw's API, Perspective, serves toxicity models and others in a growing set of languages (see our documentation for the full list). Over the past year, the field has seen impressive multilingual capabilities from the latest model innovations, including few- and zero-shot learning. We're excited to learn whether these results \"translate\" (pun intended!) to toxicity classification. Your training data will be the English data provided for our previous two competitions and your test data will be Wikipedia talk page comments in several different languages.\n\nAs our computing resources and modeling capabilities grow, so does our potential to support healthy conversations across the globe. Develop strategies to build effective multilingual models and you'll help Conversation AI and the entire industry realize that potential.\n\n#### Files\n\n* jigsaw-toxic-comment-train.csv - data from our first competition. The dataset is made up of English comments from Wikipedia’s talk page edits.\n* jigsaw-unintended-bias-train.csv - data from our second competition. This is an expanded version of the Civil Comments dataset with a range of additional labels.\n* sample_submission.csv - a sample submission file in the correct format\n* test.csv - comments from Wikipedia talk pages in different non-English languages.\n* validation.csv - comments from Wikipedia talk pages in different non-English languages.\n* jigsaw-toxic-comment-train-processed-seqlen128.csv - training data preprocessed for BERT\n* jigsaw-unintended-bias-train-processed-seqlen128.csv - training data preprocessed for BERT\n* validation-processed-seqlen128.csv - validation data preprocessed for BERT\n* test-processed-seqlen128.csv - test data preprocessed for BERT\n\nLet’s start by importing both pandas and our data in Python and taking a look at the first five rows.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport warnings\nwarnings.simplefilter('ignore')\nimport gc\nimport subprocess","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jigsaw_seqlen128_df1 = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train-processed-seqlen128.csv\")","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jigsaw_seqlen128_df1.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this notebook,I will demonstrating Python’s efficient RAM memory usage **techniques** with pandas, how to reduce a dataframe’s memory footprint by almost 90%!!!!\n\n## Technique 1: Free Memory using gc.collect() \n\nIn python notebook once a dataset loads into RAM it does not free on its own.So if you load a huge dataframe like above into pandas, and then make a copy of it and never use it again, that original dataframe will still be in RAM consuming memory which is same as other variables.Therefore we should adopt the habit of deleting dataframe when it is no longer in use to save memory.  \n\nFor example, suppose we create a dataframe by name `jigsaw_seqlen128_df1`, extract/explore some features and got the insights of the data but later on never used it ,then  `jigsaw_seqlen128_df1` will be consuming space.So we need to explicitely delete it using 'del' command.Note that there is no variable dependency on this dataframe before deleting it. So even after doing so there may be some residual memory usage.\n\nThat's where the garbage collection module comes into effect. `import gc` at the beginning of your project, and then each time you want to clear up space put command `gc.collect()` .  \n\nIt also helps to run `gc.collect()` after multiple transformations/functions/copying etc...  as all the little references/values accumulate.\n\nNow we loaded the dataframe it is evident that it is consuming memory.Now let us delete the dataframe and collect residual garbage using gc.collect() method","metadata":{}},{"cell_type":"code","source":"del jigsaw_seqlen128_df1\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Technique 2: Datatype Conversions\n\nWe can use the info() method to fetch high level information about our dataframe, including its size, information about data types and memory usage.\n\nBy default, pandas approximates of the memory usage of the dataframe to save time. Because we’re interested in accuracy, we’ll set the memory_usage parameter to 'deep' to get an accurate number.","metadata":{}},{"cell_type":"code","source":"jigsaw_seqlen128_df = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train-processed-seqlen128.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jigsaw_seqlen128_df.info(memory_usage='deep')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we have 1902194  rows and 47 columns. Pandas has automatically detected types for us, with 41 numeric columns and 6 object columns. Object columns are used for strings or where a column contains mixed data types.\n\nSo we can get a better understanding of where we can reduce this memory usage, let’s take a look into how Python and pandas store data in memory.\n\n\nEach type has a specialized class in the pandas.core.internals module. Pandas uses the ObjectBlock class to represent the block containing string columns, and the FloatBlock class to represent the block containing float columns. For blocks representing numeric values like integers and floats, pandas combines the columns and stores them as a NumPy ndarray. The NumPy ndarray is built around a C array, and the values are stored in a contiguous block of memory. Due to this storage scheme, accessing a slice of values is incredibly fast.\n\nBecause each data type is stored separately, we are going to examine the memory usage by data type. Let’s start by looking at the average memory usage for data type.","metadata":{}},{"cell_type":"code","source":"for dtype in ['float','int','object']:\n    selected_dtype = jigsaw_seqlen128_df.select_dtypes(include=[dtype])\n    mean_usage_b = selected_dtype.memory_usage(deep=True).mean()\n    mean_usage_mb = mean_usage_b / 1024 ** 2\n    print(\"Average memory usage for {} columns: {:03.2f} MB\".format(dtype,mean_usage_mb))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Immediately we can see that most of our memory is used by our 6 object columns. We’ll look at those later, but first lets see if we can improve on the memory usage for our numeric columns.\n\n### Deeper understanding of subtypes\n\nUnder the hood pandas represents numeric values as NumPy ndarrays and stores them in a continuous block of memory. This storage model consumes less space and allows us to access the values themselves quickly. Because pandas represents each value of the same type using the same number of bytes, and a NumPy ndarray stores the number of values, pandas can return the number of bytes a numeric column consumes quickly and accurately.\n\nMany types in pandas have multiple subtypes that can use fewer bytes to represent each value. For example, the float type has the float16, float32, and float64 subtypes. The number portion of a type’s name indicates the number of bits that type uses to represent values. For example, the subtypes we just listed use 2, 4, 8 and 16 bytes, respectively.\n\nAn int8 value uses 1 byte (or 8 bits) to store a value, and can represent 256 values (2^8) in binary. This means that we can use this subtype to represent values ranging from -128 to 127 (including 0).\n\nWe can use the numpy.iinfo class to verify the minimum and maximum values for each integer subtype. Let’s look at an example:","metadata":{}},{"cell_type":"code","source":"int_types = [\"uint8\", \"int8\", \"int16\"]\nfor it in int_types:\n    print(np.iinfo(it))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see here the difference between uint (unsigned integers) and int (signed integers). Both types have the same capacity for storage, but by only storing positive values, unsigned integers allow us to be more efficient with our storage of columns that only contain positive values.\n\n### Numeric Column Optimization using Subtypes\nWe can use the function pd.to_numeric() to downcast our numeric types. We’ll use DataFrame.select_dtypes to select only the integer columns, then we’ll optimize the types and compare the memory usage.","metadata":{}},{"cell_type":"code","source":"# We will be calculating memory usage a lot,so will create a function save some resource.\ndef mem_usage(pandas_obj):\n    if isinstance(pandas_obj,pd.DataFrame):\n        usage_b = pandas_obj.memory_usage(deep=True).sum()\n    else: # we assume if not a df it's a series\n        usage_b = pandas_obj.memory_usage(deep=True)\n    usage_mb = usage_b / 1024 ** 2 # convert bytes to megabytes\n    return \"{:03.2f} MB\".format(usage_mb)\ngl_int = jigsaw_seqlen128_df.select_dtypes(include=['int'])\nconverted_int = gl_int.apply(pd.to_numeric,downcast='unsigned')\nprint(mem_usage(gl_int))\nprint(mem_usage(converted_int))\ncompare_ints = pd.concat([gl_int.dtypes,converted_int.dtypes],axis=1)\ncompare_ints.columns = ['Before','After']\ncompare_ints.apply(pd.Series.value_counts)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see a drop from 145.13 to 34.47 megabytes in memory usage, which is a more than 75% reduction. The overall impact on our original dataframe isn’t massive though, because there are so few integer columns.\n\nLets do the same thing with  float columns.","metadata":{}},{"cell_type":"code","source":"gl_float = jigsaw_seqlen128_df.select_dtypes(include=['float'])\nconverted_float = gl_float.apply(pd.to_numeric,downcast='float')\nprint(mem_usage(gl_float))\nprint(mem_usage(converted_float))\ncompare_floats = pd.concat([gl_float.dtypes,converted_float.dtypes],axis=1)\ncompare_floats.columns = ['Before','After']\ncompare_floats.apply(pd.Series.value_counts)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that all our float columns were converted from float64 to float32, giving us a 50% reduction in memory usage.\n\nLet’s create a copy of our original dataframe, assign these optimized numeric columns in place of the originals, and see what our overall memory usage is now.","metadata":{}},{"cell_type":"code","source":"optimized_jigsaw_seqlen128_df = jigsaw_seqlen128_df.copy()\noptimized_jigsaw_seqlen128_df[converted_int.columns] = converted_int\noptimized_jigsaw_seqlen128_df[converted_float.columns] = converted_float\nprint(mem_usage(jigsaw_seqlen128_df))\nprint(mem_usage(optimized_jigsaw_seqlen128_df))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While we have dramatically reduced the memory usage of the numeric columns, overall see a 6-7 % reduction in the memory usage of our dataframe. Most of the gain will be observed while optimizing the object types.\n\nBefore we do, let’s take a closer look at how strings are stored in pandas compared to the numeric types\n### Comparing String to Numeric storage\nThe object type represents values using Python string objects, partly due to the lack of support for missing string values in NumPy. Because Python is a high-level, interpreted language, it doesn’t have fine grained-control over how values in memory are stored.\n\nThis limitation causes strings to be stored in a fragmented way that consumes more memory and is slower to access. Each element in an object column is really a pointer that contains the “address” for the actual value’s location in memory.\n\nBelow is a diagram showing how numeric data is stored in NumPy data types vs how strings are stored using Python’s inbuilt types.\n\n![](https://www.dataquest.io/wp-content/uploads/2019/01/numpy_vs_python.png)\nAs you might have noticed by now that object types consume considerable amount of memory. While each pointer takes up 1 byte of memory, each actual string value uses the same amount of memory that string would use if stored individually in Python. Let’s use sys.getsizeof() to prove that out, first by looking at individual strings, and then items in a pandas series.","metadata":{}},{"cell_type":"code","source":"from sys import getsizeof\ns1 = 'In progress'\ns2 = 'memory consumption'\ns3 = 'Objects in python is fun!'\ns4 = 'Numerics consume less memory!'\nfor s in [s1, s2, s3, s4]:\n    print(getsizeof(s))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"obj_ser = pd.Series(['In progress',\n    'memory consumption',\n    'Objects in python is fun!',\n    'Numerics consume less memory!'])\nobj_ser.apply(getsizeof)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above analogy it is observed that the size of strings when stored in a pandas series are identical to their usage as separate strings in Python.\n\n### Object types optimization using categoricals\n\nPandas introduced Categoricals in version 0.15. The category type uses integer values under the hood to represent the values in a column, rather than the raw values. Pandas uses a separate mapping dictionary that maps the integer values to the raw ones. This arrangement is useful whenever a column contains a limited set of values. When we convert a column to the category dtype, pandas uses the most space efficient int subtype that can represent all of the unique values in a column.\n\nTo get an overview of where we might be able to use this type to reduce memory, let’s take a look at the number of unique values of each of our object types.","metadata":{}},{"cell_type":"code","source":"jigsaw_seqlen128_df_obj = jigsaw_seqlen128_df.select_dtypes(include=['object']).copy()\njigsaw_seqlen128_df_obj.describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A quick glance reveals many columns where there are few unique values relative to the overall ~1902194 records in our data set.\n\nBefore we dive too far in, we’ll start by selecting just one of our object columns, and looking at what happens behind the scenes when we convert it to the categorical type. We will use the second column of our data set, rating.\n\nLooking at the table above. we can see that it only contains two unique values. We’ll convert it to categorical by using the .astype() method.","metadata":{}},{"cell_type":"code","source":"rating = jigsaw_seqlen128_df_obj.rating\nprint(rating.head())\nrating_cat = rating.astype('category')\nprint(rating_cat.head())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, apart from the fact that the type of the column has changed, the data looks exactly the same. Let’s take a look under the hood at what’s happening.\n\nIn the following code, we use the Series.cat.codes attribute to return the integer values the category type uses to represent each value.","metadata":{}},{"cell_type":"code","source":"rating_cat.head().cat.codes","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can see that each unique value has been assigned an integer, and that the underlying datatype for the column is now int8. This column doesn’t have any missing values, but if it did, the category subtype handles missing values by setting them to -1.\n\nLastly, let’s look at the memory usage for this column before and after converting to the category type.","metadata":{}},{"cell_type":"code","source":"print(mem_usage(rating))\nprint(mem_usage(rating_cat))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We’ve gone from 117.91MB of memory usage to 1.81MB of memory usage, or a 99% reduction! Note that this particular column rating represents one of our best-case scenarios – a column with ~1902194 items of which there only 2 unique values.\n\nWhile converting all of the columns to this type sounds appealing, it’s important to be aware of the trade-offs. The biggest one is the inability to perform numerical computations. We can’t do arithmetic with category columns or use methods like Series.min() and Series.max() without converting to a true numeric dtype first.\n\n**We should stick to using the category type primarily for object columns where less than 50% of the values are unique.** If all of the values in a column are unique, the category type will end up using more memory. That’s because the column is storing all of the raw string values in addition to the integer category codes. You can read more about the limitations of the category type in the pandas documentation.\n\nWe’ll write a loop to iterate over each object column, check if the number of unique values is less than 50%, and if so, convert it to the category type.","metadata":{}},{"cell_type":"code","source":"jigsaw_seqlen128_df_converted_obj = pd.DataFrame()\nfor col in jigsaw_seqlen128_df_obj.columns:\n    num_unique_values = len(jigsaw_seqlen128_df_obj[col].unique())\n    num_total_values = len(jigsaw_seqlen128_df_obj[col])\n    if num_unique_values / num_total_values < 0.5:\n        jigsaw_seqlen128_df_converted_obj.loc[:,col] = jigsaw_seqlen128_df_obj[col].astype('category')\n    else:\n        jigsaw_seqlen128_df_converted_obj.loc[:,col] = jigsaw_seqlen128_df_obj[col]\nprint(mem_usage(jigsaw_seqlen128_df_obj))\nprint(mem_usage(jigsaw_seqlen128_df_converted_obj))\ncompare_obj = pd.concat([jigsaw_seqlen128_df_obj.dtypes,jigsaw_seqlen128_df_converted_obj.dtypes],axis=1)\ncompare_obj.columns = ['Before','After']\ncompare_obj.apply(pd.Series.value_counts)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case, 50% of object columns were converted to the category type, however this won’t be the case with all data sets, so you should be sure to use the process above to check.But to demonstrate a better results I have used this technique in my latest kernel https://www.kaggle.com/pavansanagapati/ad-click-prediction-deep-interest-network-model.\n\nWhat’s more, our memory usage for our object columns in this kernel example has gone from 3806.64MB to 2094.21MB, or a reduction of 45%. Let’s combine this with the rest of our dataframe and see where we sit in relation to the 4300MB memory usage we started with.","metadata":{}},{"cell_type":"code","source":"optimized_jigsaw_seqlen128_df[jigsaw_seqlen128_df_converted_obj.columns] = jigsaw_seqlen128_df_converted_obj\nmem_usage(optimized_jigsaw_seqlen128_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wow, we have really made some progress by memory reduction of around 50% !!! \n\nSo far, we have explored ways to reduce the memory footprint of an existing dataframe. By reading the dataframe in first and then iterating on ways to save memory, we were able to understand the amount of memory we can expect to save from each optimization better.\n\nHowever in practice we often won’t have enough memory to represent all the values in a data set.\n\nSo the question is how do we apply memory-saving techniques when we can’t even create the dataframe in the first place? Let's explore and find out.\n\nWe can specify the optimal column types when we read the data set in. The pandas.read_csv() function has a few different parameters that allow us to do this. The dtype parameter accepts a dictionary that has (string) column names as the keys and NumPy type objects as the values.\n\nFirst, we’ll store the final types of every column in a dictionary with keys for column names","metadata":{}},{"cell_type":"code","source":"dtypes = optimized_jigsaw_seqlen128_df.dtypes\ndtypes_col = dtypes.index\ndtypes_type = [i.name for i in dtypes.values]\ncolumn_types = dict(zip(dtypes_col, dtypes_type))\n# rather than print all 161 items, we'll\n# sample 10 key/value pairs from the dict\n# and print it nicely using prettyprint\npreview = first2pairs = {key:value for key,value in list(column_types.items())[:10]}\nprint(preview)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can use the dictionary, along with a few parameters for the date to read in the data with the correct types in a few lines:","metadata":{}},{"cell_type":"code","source":"read_and_optimized = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train-processed-seqlen128.csv',dtype=column_types)\nprint(mem_usage(read_and_optimized))\nread_and_optimized.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### There we go finally by optimizing the columns, we’ve managed to reduce the memory usage in pandas from 4300 MB to 2320.70 MB – an impressive 55% reduction!\n\n# Technique 3: Import selective no of rows in data\n\nUsually by default we read a csv file as a whole but there is an option called nrows where we can specify no of rows to import .For example if we want to read a csv file with the first 2000 rows only then syntax is as follows ","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\", nrows=20000)\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Technique 4: Random Row Selection\nThis is nothing but performing a **random sampling** of the data and make best use of it.\nFor this we can build a list of random rows by writing a function ","metadata":{}},{"cell_type":"code","source":"def csv_file_length(fname):\n    process = subprocess.Popen(['wc', '-l', fname], stdout=subprocess.PIPE, \n                                              stderr=subprocess.PIPE)\n    result, error = process.communicate()\n    if process.returncode != 0:\n        raise IOError(error)\n    return int(result.strip().split()[0])\n\nrandom_rows_selection = csv_file_length('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv')\nprint('Number of random rows in \"jigsaw-toxic-comment-train.csv\" is:', random_rows_selection)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Suppose we want to fetch a random sample of 10000 lines out of the total dataset. i.e we need to fetch a list of `lines - 1 - 10000` random numbers ranging from 1 to 796373. \n\nNote that while we are generating such a long list also takes a lot of space and  some time. So let us use the Technique 1 mentioned above and make sure to use del and gc.collect() when complete!!!!\n\nNow let us generate list of lines to skip","metadata":{}},{"cell_type":"code","source":"skip_rows = np.random.choice(np.arange(1, random_rows_selection), size=random_rows_selection-1-10000, replace=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us sort the above list and print to check .....","metadata":{}},{"cell_type":"code","source":"skip_rows=np.sort(skip_rows)\nprint('Rows to skip:', len(skip_rows))\nprint('Remaining rows in the random sample:', random_rows_selection-len(skip_rows))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we got the random sample rows , let us fetch them from the csv file","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv', skiprows=skip_rows)\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now time for cleanup of unused list generated above and garbage collect.","metadata":{}},{"cell_type":"code","source":"del skip_rows\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Technique 5: Skip No of Rows\nWe can mention no of rows to skip by using the option of **\"skiprows\"** as shown below without heading.Note that plain skipping looses heading info. ","metadata":{}},{"cell_type":"code","source":"train_df1 = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\",skiprows=2000 ,header=None,nrows=20000)\ntrain_df1.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can mention no of rows to skip by using the option of **\"skiprows\"** as shown below with heading.Note that if we  want to import the headings from the csv file we need to skip first rows but use the first row for heading:","metadata":{}},{"cell_type":"code","source":"train_df2 = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\",skiprows=range(1,2000) ,nrows=20000)\ntrain_df2.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Technique 6: Use of Generators\nGenerators allow you to create a function that returns one item at a time rather than all the items at once. This means that if you have a large dataset, you don’t have to wait for the entire dataset to be accessible.\n\nGenerator functions allow you to declare a function that behaves like an iterator. It allows us to make an iterator in a fast, easy, and clean way. An iterator is an object that can be iterated (looped) upon. It is used to abstract a container of data to make it behave like an iterable object. Some examples of common iterable objects are strings, lists and dictionaries.\n\nA generator looks a lot like a function, but uses the keyword **yield** instead of **return**. Let us take an example for better understanding. Suppose we try to generate numbers less than 10 using a generator by name \"number_generator\" as follows","metadata":{}},{"cell_type":"code","source":"def number_generator():\n    n = 0\n    while n < 10:\n        yield n\n        n += 1\nnumbers = number_generator()\ntype(numbers)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That’s a generator function. So when we call it, it returns a generator object as shown above.\n\nAn important note here is how the state is encapsulated within the body of the generator function.To observe one by one value we can use the built-in **next()** function as follows","metadata":{}},{"cell_type":"code","source":"next_no = number_generator()\nnext(next_no)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"next(next_no)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So you may wonder what will happen if we call **next()** past the end. For this we have a built in exception type called **StopIteration** that automatically raises once the generator stops yielding.\n\nIn summary when we call a **generator** function or use a generator expression, we return a special iterator called a generator. We can assign this generator to a variable in order to use it. When we call special methods on the generator, such as next(), the code within the function is executed up to yield.\n\nLet us compare a normal approach vs using **generators** with respect to memory usage and time taken for the code to execute.\n\nSuppose we iterate through a large list of numbers 500000 and store the square of all the numbers which are even in a seperate list.\n\n### Normal Approach","metadata":{}},{"cell_type":"code","source":"import memory_profiler\nimport time\ndef even_numbers(numbers):\n    even = []\n    for num in numbers:\n        if num % 2 == 0: \n            even.append(num*num)\n            \n    return even\nif __name__ == '__main__':\n    m1 = memory_profiler.memory_usage()\n    t1 = time.clock()\n    cubes = even_numbers(range(500000))\n    t2 = time.clock()\n    m2 = memory_profiler.memory_usage()\n    time_diff = t2 - t1\n    mem_diff = m2[0] - m1[0]\n    print(f\"It took {time_diff} Secs and {mem_diff} Mb memory to execute Normal method\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generator Approach:\nNow let us look at how efficient generator approach ","metadata":{}},{"cell_type":"code","source":"import memory_profiler\nimport time\ndef even_numbers(numbers):\n    for num in numbers:\n        if num % 2 == 0:\n            yield num * num \n    \nif __name__ == '__main__':\n    m1 = memory_profiler.memory_usage()\n    t1 = time.clock()\n    cubes = even_numbers(range(500000))\n    t2 = time.clock()\n    m2 = memory_profiler.memory_usage()\n    time_diff = t2 - t1\n    mem_diff = m2[0] - m1[0]\n    print(f\"It took {time_diff} Secs and {mem_diff} Mb memory to execute Generator method\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The advantage lies in the fact that generators don’t store all results in memory, rather they generate them on the fly, hence the memory is only used when we ask for the result. Also generators abstract away much of the boilerplate code needed when writing iterators, hence also helps in reducing the size of code.\n# Technique 7: Eliminate unneccessary loops by using itertools\n**itertools** saves you a lot of time on loops. It also gets rid of the complexity of the code.This module implements a number of iterator building blocks.\n\n### Itertool functions\n\nI will demostrating some of the itertool functions in this notebook .They provide streams of infinite length, so they should only be accessed by functions or loops that truncate the stream.\n\n**itertools.accumulate(iterable[, func, *, initial=None])**\n\nThis iterator returns accumulated sums, or accumulated results of other binary functions (specified via the optional func argument).If func is supplied, it should be a function of two arguments. Elements of the input iterable may be any type that can be accepted as arguments to func.\n\nUsually, the number of elements output matches the input iterable. However, if the keyword argument initial is provided, the accumulation leads off with the initial value so that the output has one more element than the input iterable.\n\nLet us see an example to see how it works","metadata":{}},{"cell_type":"code","source":"import operator\nfrom itertools import product, chain\ndef accumulate(iterable, func=operator.add, *, initial=None):\n    'Return running totals'\n    # Example 1: accumulate([4,5,6,7,8,9]) --> 4 9 15 22 30 39\n    # Example 2: accumulate([6,7,8,9], initial=1000) --> 1000 1006 1013 1021 \n    # Example 3: accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120\n    it = iter(iterable)\n    total = initial\n    if initial is None:\n        try:\n            total = next(it)\n        except StopIteration:\n            return\n    yield total\n    for element in it:\n        total = func(total, element)\n        yield total","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us see how we can multiply the values using the accumulate()","metadata":{}},{"cell_type":"code","source":"data = [3, 4, 6, 2, 1, 9, 0, 7, 5, 8]\nlist(accumulate(data, operator.mul))     # running multiplication","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us see how we can maximum values using the accumulate()","metadata":{}},{"cell_type":"code","source":"list(accumulate(data, max))              # running maximum","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Technique 8: Do not use + operator for strings\n\nStart avoiding + operator for concatenation for strings as they are immutable,i.e every time we add an element to a string, Python creates a new string and a new address which means** new memory** needs to be allocated each time the string is altered.\n\nLet us see with a simple example as shown below \n\n**Normal approach:**","metadata":{}},{"cell_type":"code","source":"mymsg = \"Micheal\"\nmsg=\"My name is \"+mymsg+\". I live in US\"\nprint(msg)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**New approach**","metadata":{}},{"cell_type":"code","source":"mymsg = \"Micheal\"\nmsg=\"My name is %s . I live in US\"% mymsg\nprint(msg)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* # Technique 9: Memory profiling\n\nJuggling with large data sets involves having a clear sight of memory consumption and allocation processes going on in the background. There are tools to monitor the memory usage of your notebook.\n\nUse **%memit** in familiar fashion to **%timeit**\n\nLet us see this with an example .Suppose we want to caalculate PI value in a monte carlo simulation","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom random import random\ndef pi_calculation(n=50000000) -> \"area\":\n    \"\"\"Estimate pi with monte carlo simulation.\n    \n    Arguments:\n        n: Number of Simulations\n    \"\"\"\n    return np.sum(np.random.random(n)**2 + np.random.random(n)**2 <= 1) / n * 4","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use this command to run the above function \n**%memit pi_calculation()**","metadata":{}},{"cell_type":"markdown","source":" # Technique 10: Memory leaks\n \n We can avoid memory leaks in Jupyter Notebook by removing the temporary variables when we no longer need them.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndata = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')\n\nno_nans = data.dropna()\none_hot_encoded = pd.get_dummies(no_nans)\n# some other temp variables\n#processed_data = ...\n\ndel no_nans\ndel one_hot_encoded","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That method is not recommended, because we can easily overlook a variable that should be removed or remove a used variable by mistake.\n\nThe better method of avoiding memory leaks is doing data processing inside a function. It creates a new scope for the intermediate variables and removes them automatically when the interpreter exits the function:","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\ndef data_preprocessing(raw_data):\n    no_nans = data.dropna()\n    one_hot_encoded = pd.get_dummies(no_nans)\n    # some other temp variables\n    processed_data = 10\n    return processed_data\n\ndata = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')\nprocessed_data = data_preprocessing(data)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Technique 11: Line profiler\n\n**%lprun** command yields the time spent on each line of code giving us a line by line report. Since not shipped by default, install the library with pip","metadata":{}},{"cell_type":"code","source":"!pip install line_profiler","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"and load the extension manually in the notebook.","metadata":{}},{"cell_type":"code","source":"%load_ext line_profiler","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To demonstrate the capabilities of this feature let us define method which evaluates pi using random generated data points and then look for ways to optimize. Remember that the area covered by a circle with radius 1 inscribed in a square, equals exactly to a quarter of pi.\n\nWe get the value of pi by taking the ratio of area of circle to area of the square,","metadata":{}},{"cell_type":"code","source":"from random import random\n\ndef estimate_pi(n=1e7) -> \"area\":\n    \"\"\"Estimate pi with monte carlo simulation.\n    \n    Arguments:\n        n: number of simulations\n    \"\"\"\n    in_circle = 0\n    total = n\n    \n    while n != 0:\n        prec_x = random()\n        prec_y = random()\n        if pow(prec_x, 2) + pow(prec_y, 2) <= 1:\n            in_circle += 1 # inside the circle\n        n -= 1\n        \n    return 4 * in_circle / total","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" It is easy as pi(e) to locate hotspots in the code with only difference that functions need to be explicitly defined.","metadata":{}},{"cell_type":"code","source":"%lprun -f estimate_pi estimate_pi()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Timer unit: 1e-06 s\n\nTotal time: 62.5456 s\nFile: <ipython-input-58-0f6e1f5ac99b>\nFunction: estimate_pi at line 3\n\nLine #      Hits         Time  Per Hit   % Time  Line Contents\n==============================================================\n     3                                           def estimate_pi(n=1e7) -> \"area\":\n     4                                               \"\"\"Estimate pi with monte carlo simulation.\n     5                                               \n     6                                               Arguments:\n     7                                                   n: number of simulations\n     8                                               \"\"\"\n     9         1          3.0      3.0      0.0      in_circle = 0\n    10         1          1.0      1.0      0.0      total = n\n    11                                               \n    12  10000001    9739317.0      1.0     15.6      while n != 0:\n    13  10000000   10079187.0      1.0     16.1          prec_x = random()\n    14  10000000    9900919.0      1.0     15.8          prec_y = random()\n    15  10000000   15220356.0      1.5     24.3          if pow(prec_x, 2) + pow(prec_y, 2) <= 1:\n    16   7855684    7664704.0      1.0     12.3              in_circle += 1 # inside the circle\n    17  10000000    9941136.0      1.0     15.9          n -= 1\n    18                                                   \n    19         1          3.0      3.0      0.0      return 4 * in_circle / total\n\nNotice the extensive time (24.3%) spent on the if statement on line 15. Take in mind that we can even colorize the report above, giving us a more intuitive way to see **hot-spots**.\n \n All we have to do is insert the %%heat command at the top of the cell (to use load the extension%load_ext heat after installing with !pip install py-heat-magic) and it allows us to see the completely red while loop obliquing high cost of CPU-time, clearly showing room for optimization.\n \n# Technique 12: Memory Profiler\nIn this technique, we will look at a simple **memory profiler** called memory_profiler. It is very similar to **line_profiler**, and it can be conveniently be used from IPython.The **memory_profiler** package checks the memory usage of the interpreter at every line. The increment column allows us to spot those places in the code where large amounts of memory are allocated. This is especially important when working with arrays. Unnecessary array creations and copies can considerably slow down a program. We will tackle this issue in the next few recipes.\n\nLet us install it first","metadata":{}},{"cell_type":"code","source":"!pip install memory_profiler","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we load the memory_profiler IPython extension","metadata":{}},{"cell_type":"code","source":"%load_ext memory_profiler","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let us define a function that allocates large objects:","metadata":{}},{"cell_type":"code","source":"%%writefile memscript.py\ndef myfunc():\n    a = [1] * 5000000\n    b = [2] * 7000000\n    del b\n    return a","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now  let's run the code under the control of the memory profiler.**","metadata":{}},{"cell_type":"code","source":"from memscript import myfunc\n%mprun -T mprof0 -f myfunc myfunc()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Finally let us see the outcome**","metadata":{}},{"cell_type":"code","source":"print(open('mprof0', 'r').read())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Technique 13: autoreload\nThe autoreload module is there to help with code structure. The module reloads the code before each execution. Once you get locked into a TDD loop and you start refactoring code from the notebook into additional files, this module will reload the code in the additional files.","metadata":{}},{"cell_type":"code","source":"# Reload modules before executing user code\n%load_ext autoreload\n%autoreload 2\n\n# setup backend for matplotlibs plots\n%matplotlib inline","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Technique 14: Increase default memory limit size\n\nJupyter notebook has a default memory limit size. You can try to increase the memory limit by following the steps:\n\n**1) Generate Config file using command:**\n\n*jupyter notebook --generate-config*\n\n**2) Open jupyter_notebook_config.py file situated inside 'jupyter' folder and edit the following property:**\n\n*NotebookApp.max_buffer_size = your desired value*\n\nRemember to remove the '#' before the property value.\n\n**3) Save and run the jupyter notebook. It should now utilize the set memory value. Also, don't forget to run the notebook from inside the jupyter folder.**\n\nAlternatively, you can simply run the Notebook using below command:\n\n*jupyter notebook --NotebookApp.max_buffer_size=your_value*","metadata":{}},{"cell_type":"markdown","source":"\n\n### I hope you had a good insight on memory optimisation techniques mentioned above while handling large datasets(1GB or more) in this notebook and I am hopeful that these techniques comes handy in your day to day work. \n\n# Greatly appreciate to leave your comments /suggestions .\n# If you like this notebook please do <font color='red'>UPVOTE .","metadata":{}}]}