{"cells":[{"metadata":{"_uuid":"6e0d90e1f1a0d3d94e05e9b14232a47524492d77"},"cell_type":"markdown","source":"Preface: This is a notebook testing the performance of str methods in pandas and how they compare to other ways to achieve the same results of the methods."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport re\nfrom textblob import TextBlob\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ff18e37c7173061eefa0f5963bf62202fb8a92f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3cbd11517ce28b3de770e417b487b439b327ef3d"},"cell_type":"markdown","source":"# Example Set 1 - concatenating and acessing"},{"metadata":{"trusted":true,"_uuid":"55562fbeb487e8c4b7beea654d2e1de2ffdfbe30"},"cell_type":"code","source":"# Create pools\nmonths = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\n# months = [str(i).zfill(2) for i in range(1, 13)]\ndays = [str(i).zfill(2) for i in range(1, 29)]\nyears = [str(i) for i in range(1992, 2016)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a1a023ef19c6b50d8c389d681aa615973bdbe48"},"cell_type":"code","source":"# Randomly generate from pools\nrand_months = np.random.choice(months, size=10000)\nrand_days = np.random.choice(days, size=10000)\nrand_years = np.random.choice(years, size=10000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d04079908dacb8d47e73a8a06e040ae13651463f"},"cell_type":"code","source":"# Place into df\n%time rand_dates = pd.DataFrame({'yr':rand_years, 'mo':rand_months, 'da': rand_days})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dbc22926a31862636a2c5bba15c8b6cfcd25134c"},"cell_type":"code","source":"rand_dates.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5da2842e03a1eadabc45ae949fe796ba82d940d"},"cell_type":"markdown","source":"Pandas str.cat method vs simple concat - 3x slower"},{"metadata":{"trusted":true,"_uuid":"66b99d11ce1655a17a2da5f823bbe68f41884f65"},"cell_type":"code","source":"%timeit rand_dates['mo_da']  = rand_dates['mo'].str.cat(rand_dates['da'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6cf4b98bff7612310841dc8c12d6a10b7768bf52"},"cell_type":"code","source":"%timeit rand_dates['mo_da2']  = rand_dates['mo'] + rand_dates['da']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef9d500b72d3c077593acdf90790a6086284ebd1"},"cell_type":"markdown","source":"Str.get() or Str[idx] vs Str.slice(i, j) - 2x slower"},{"metadata":{"trusted":true,"_uuid":"ab389edec6dc2af09fd31fbeadfa7180b155fac4"},"cell_type":"code","source":"%timeit rand_dates['mo'].str.slice(0,1) + rand_dates['da'].str.slice(0,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e65f89646aad546f28352a0df2aaa6065a1ac9f1"},"cell_type":"code","source":"%timeit rand_dates['mo'].str[0] + rand_dates['da'].str[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e2ff69b6c3ed3029ae7dea3e5b25c189ef8470e"},"cell_type":"code","source":"%timeit rand_dates['mo'].str.get(0) + rand_dates['da'].str.get(0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ed7e9a7c6276636ff0ef9e5352a1f27f71daf1f"},"cell_type":"markdown","source":"Adding an empty string - stays the same"},{"metadata":{"trusted":true,"_uuid":"4af608ec01892c395ce6e5074f0d8dad6ec09e76"},"cell_type":"code","source":"rand_dates.loc[23, 'mo'] = ''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1f19138562ef366da896d35b7fccc2c9d5a08c8f"},"cell_type":"code","source":"%timeit rand_dates['mo'].str.slice(0,1) + rand_dates['da'].str.slice(0,1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a94d637800ba87a4b0cb3e9b33a87d2309abf7a7"},"cell_type":"markdown","source":"Adding NaN, a different data type - takes 1.2x longer"},{"metadata":{"trusted":true,"_uuid":"54b359c009890ba3d54e3dc0c3ce124de01539f5"},"cell_type":"code","source":"rand_dates.loc[23, 'mo'] = np.NaN","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c146b59acdfe6b240c728027174d271ecfa4f882"},"cell_type":"code","source":"%timeit rand_dates['mo'].str.slice(0,1) + rand_dates['da'].str.slice(0,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f39471fc3a1ea42f74b60a65cfc5de3fe462f611"},"cell_type":"code","source":"%timeit rand_dates['mo'] + rand_dates['da']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc7c5464563def92b5447975796056c2fa0caacf"},"cell_type":"markdown","source":"There may be some extra functionality in the str methods but for the simple case where you know the dtypes of your columns, you can get an extra boost in performance by opting for the simpler calls. "},{"metadata":{"_uuid":"095894fa16c1b65a248d229da72daf49bd0b9ec5"},"cell_type":"markdown","source":"# Example 2:  Operating on the same string"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv', nrows=10000)\ntrain2 = train.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0f37462dd559e4557146c42f9560b0d8e9ff702","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"%timeit d = train['question_text'].str.slice(0,1) # same trend as earlier","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f1c83ce9b8b18c24f0f84e2e1f8f0233a16cbafd","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"%timeit a = train['question_text'].str[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20c4b877d1802b29f21a0d2eceaec91ed87e9f64"},"cell_type":"code","source":"%timeit b = train['question_text'].str.count('e')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5dc19d68924f0b75b5b022416daa1cc9b4f00c17"},"cell_type":"code","source":"%timeit c = train['question_text'].str.capitalize()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57f1ad7c564ffa531cd3b4218cf9230056baafc4"},"cell_type":"markdown","source":"### Using pandas str method once to get create each new column"},{"metadata":{"trusted":true,"_uuid":"7533b070bfe1c3ca0c798bf7133687884889e4ee"},"cell_type":"code","source":"%%timeit\ntrain['first'] = train['question_text'].str[0]\ntrain['count_e'] = train['question_text'].str.count('e')\ntrain['cap'] = train['question_text'].str.capitalize()\n# Just the individual values added together","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc00546914a2d9a09b7a0f87259fd94e07f9e08d"},"cell_type":"markdown","source":"### Returning a tuple with apply and zip(*) - ~15% faster "},{"metadata":{"trusted":true,"_uuid":"3a3399f4379c9f030bc879f0711190ba1baa5a2b"},"cell_type":"code","source":"def extract_text_features(x):\n    return x[0], x.count('e'), x.capitalize()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"556ea330d2b47dcd03f555d73a9aa4103fd1fb02"},"cell_type":"code","source":"%timeit train['first'], train['count_e'], train['cap'] = zip(*train['question_text'].apply(extract_text_features))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7576a06fa63e8c8b15fa9963e2d9a518b8feead4"},"cell_type":"markdown","source":"### Basic python loop and then assign to new columns - Another 25% faster "},{"metadata":{"trusted":true,"_uuid":"e5433273e703c55bdecbee17ee5f3a82e89cb209"},"cell_type":"code","source":"%%timeit\na,b,c = [], [], []\nfor s in train['question_text']:\n    a.append(s[0]), b.append(s.count('e')), c.append(s.capitalize())\ntrain['first'] = a\ntrain['count_e'] = b\ntrain['cap'] = c\n# assigning to new column takes about the same time in either method","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f802658310720f16a04753eab67ee83fa28a2e6a"},"cell_type":"markdown","source":"### Back to str methods:  \n    Str.len() vs apply lambda - ~25% faster - has some optimization done"},{"metadata":{"trusted":true,"_uuid":"dfe911e10af3ff2eabb5ca1d0bd35208e11bca57"},"cell_type":"code","source":"%timeit x = train['question_text'].str.len()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0eb2c0648e896a89a3ccf700cdbeb27ec00b8ab7"},"cell_type":"code","source":"%timeit b = train['question_text'].apply(lambda x:len(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a3282e49247d62de93be50751db84675a1265db"},"cell_type":"code","source":"# bonus - getting memory of your array\ntrain['question_text'].values.nbytes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"feff782bc019ead5ac12d120c98f70ce6faec7d7"},"cell_type":"markdown","source":"### More string methods"},{"metadata":{"trusted":true,"_uuid":"a23e0b02906e25eb2b85f8d30df926b97ba38824"},"cell_type":"markdown","source":"### Individually vs Tuples vs Series -  \n Function calls are expensive "},{"metadata":{"trusted":true,"_uuid":"9b03bec3842c258481e0c8db938cc143c91e0c89"},"cell_type":"code","source":"%%timeit \ntrain2['num_chars'] = train2['question_text'].str.len()\ntrain2['is_titlecase'] = train2['question_text'].str.istitle().astype('int')\ntrain2['has_*'] = train2['question_text'].str.contains(r'[A-Za-z]\\*.|.\\*[A-Za-z]', regex=True).astype('int')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"51149246ec2a41c0d86dbcab3bac11d407782853"},"cell_type":"code","source":"def srs_funcs(srs):\n    a = len(srs)\n    b = int(srs.istitle())\n    c = int(bool(re.search(r'[A-Za-z]\\*.|.\\*[A-Za-z]', srs)))\n    return a, b, c\n# would have expected this to be faster than creating three new columns individually but maybe the type conversion calls slowed it down","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92b6aa0c74f1096f6991f100dfc8e9f54edbd7f0"},"cell_type":"code","source":"%timeit  train2['num_chars'] , train2['is_titlecase'], train2['has_*'] = zip(*train2['question_text'].apply(srs_funcs))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4cde04b7cfb6b6f4089dddac52cd0ac3245217ff"},"cell_type":"code","source":"def srs_funcs2(srs):\n    a = len(srs)\n    b = int(srs.istitle())\n    c = int(bool(re.search(r'[A-Za-z]\\*.|.\\*[A-Za-z]', srs)))\n    return pd.Series([a, b, c])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8cd65a01df013c3848e64850bff80fb08cd58466"},"cell_type":"code","source":"%timeit  train2[['num_chars','is_titlecase','has_*']] = train2['question_text'].apply(srs_funcs2)\n# calling pd.series each time through loop kills performance","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb6dd6ce3c3a07ae499579ff376e7fec2415d9b3"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f402fbf297b028b24e3ff47ec168cbc442b7682"},"cell_type":"markdown","source":"### Example 3: Same string, more complicated function"},{"metadata":{"trusted":true,"_uuid":"2e42c79e44d031268d35018df63d7b88b24bb884"},"cell_type":"code","source":"def textblob_methods(blob):\n    '''Access Textblob methods and returns as tuple\n    '''\n    # convert to python list of tokens\n    return blob.polarity, blob.subjectivity, int(blob.ends_with('?'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bea3b7e1bed285b02084bcd585bee082837226c8"},"cell_type":"code","source":"train3 = pd.read_csv('../input/train.csv', nrows=10000)\ntrain3.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4401115709342c6da3469beae07b823fc5e1f3e"},"cell_type":"code","source":"# Convert  - any ways to make this faster? \n%timeit train3['blobs'] = train3['question_text'].map(lambda x: TextBlob(x))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2797c2a6efd08210b664a7db03a0181204268991"},"cell_type":"markdown","source":"# Suggestions: \n1) parallelize with joblib, multiprocessing\n2) SpaCy parallel processing\n3) Pandas extension arrays"},{"metadata":{"_uuid":"9d0b05a38ac2126ac679a4acb5b23d26ff241677"},"cell_type":"markdown","source":"###  Locate both index and column at the same time s faster"},{"metadata":{"trusted":true,"_uuid":"46e5162ee7524a24f8ba0a45a2bb26b5c47c17d8"},"cell_type":"code","source":"%timeit zsamp = train3.loc[5006,'blobs']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"246780aa6e35ae2c478401b7b3c26b96af6d15ca"},"cell_type":"code","source":"%timeit zsamp = train3.loc[5006]['blobs']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b0ab5c0dd558222d4cf0ac8dc03d9a7766ab6675"},"cell_type":"code","source":"zsamp = train3.loc[5006]['blobs']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7340d13b1689577fc37a9a98013539e8a2f43bca"},"cell_type":"code","source":"%timeit textblob_methods(zsamp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c12668b99e4765888a38d29ea6ae15f7ab8226f2"},"cell_type":"code","source":"%timeit  train3['polarity'], train3['subjectivity'], train3['ends_with_?'] = zip(*train3['blobs'].map(textblob_methods))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"72a3265436f5bb032ae4417e5f8937b270ea05eb"},"cell_type":"code","source":"%%timeit\na, b, c = [], [], []\nfor s in train3['blobs']:\n    a.append(s.polarity), b.append(s.subjectivity), c.append(int(s[-1] in '?'))\ntrain3['polarity'], train3['subjectivity'], train3['ends_with_?'] = a, b, c","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"acd82665c7d732a8d227a12a507d40a0ea41ab82"},"cell_type":"code","source":"%%timeit\n# Doing it separately - takes longer\ntrain3['polarity'] = train3['blobs'].apply(lambda x: x.polarity)\ntrain3['subjectivity'] = train3['blobs'].apply(lambda x: x.subjectivity)\ntrain3['ends_with_?'] = train3['blobs'].apply(lambda x: x.endswith('?'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe8884b8f6d85b8496e8b02513ba0a5f054ab49a"},"cell_type":"code","source":"def textblob_methods2(blob):\n    '''Access Textblob methods and returns as tuple\n    '''\n    # convert to python list of tokens\n    return blob.polarity, blob.subjectivity","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eea9375d0a0ba36086ee7c6d1e2dae67f354c02b"},"cell_type":"code","source":"%timeit  train3['polarity'], train3['subjectivity'] = zip(*train3['blobs'].map(textblob_methods2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67fc007b7f523605cdfebbce76127e9db593a5a7"},"cell_type":"code","source":"%%timeit\na, b = [], []\nfor s in train3['blobs']:\n    a.append(s.polarity), b.append(s.subjectivity)\ntrain3['polarity'], train3['subjectivity'] = a, b","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2582e62913f8ab52e6866f2a7d67a4858fcd1419"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68a1a19e014a5636169028450720932843072559"},"cell_type":"markdown","source":"# Takeaways\n1. Str methods are better than apply but still nowhere close to the performance increase from vectorization like with numeric columns.   \n2. Apply is still a loop. Try to do multiple things within the same loop if you can.\n3. Unzipping tuples can be a way to output multiple columns, but using lists and python loops can be surprisingly fast for str series."},{"metadata":{"trusted":true,"_uuid":"f890d80e4c8169d9c69c0fa6787510d63ec3ea72"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e4a840999c645ead0da5223d773a9e0309100f1"},"cell_type":"markdown","source":"# Resources:\n1. [**Pandas docs on working with text**](https://pandas.pydata.org/pandas-docs/stable/user_guide/text.html)\n2. [General order of precedence for performance of various operations from pandas maintainer](https://stackoverflow.com/questions/24870953/does-iterrows-have-performance-issues/24871316#24871316)\n3. [Numeric Vectorization](https://stackoverflow.com/questions/52673285/performance-of-pandas-apply-vs-np-vectorize-to-create-new-column-from-existing-c/52674448#52674448)\n4. [Article overview, focused more on numeric columns](https://realpython.com/fast-flexible-pandas/)\n5. [**Python loop outperforming apply**](https://stackoverflow.com/questions/16236684/apply-pandas-function-to-column-to-create-multiple-new-columns/47097625#47097625)\n\nThis notebook was based primarily on the two bolded links.\n    "},{"metadata":{"trusted":true,"_uuid":"29b5e15cb3431aad37a6321e3fe295b8ef9836ea"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}