{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import datatable as dt\nfrom datatable import f\nimport pandas as pd\n\nimport numpy as np\nfrom sklearn.metrics import roc_auc_score,roc_curve, auc, log_loss\nfrom sklearn.linear_model import LogisticRegression\n\nimport gc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Maybe it's a bit late for most people, but I just came up with this trick and it's already signinficantly improving my life.\n\nIf this is helpful, feel free to either pull in the output directly, or tweak the code and rerun to generate what you need.\n\nThe training dataset is rather unwieldy because it's so large. What makes it even more unwieldy is that Pandas tends to take quite a few liberties with how it uses memory when processing data, seemingly making lots of copies of stuff in RAM during various intermediate states of doing what you ask. This is problematic because the training dataset is around 5 GB, and we only get 16GB of memory. So the training dataset would only possibly fit into memory at most 3 times. And apparently Pandas tries for more than 3, since my notebooks constantly run out of RAM **and restart, losing any intermediate state** when I try to perform some calculation on the whole training dataset. Most of the things I'm trying to do ought to be O(N) time-wise and O(N) or constant space-wise, but again, Pandas seems to have other ideas.\n\nAnyway, the solution is to **split the dataset into mostly-independent chunks**. And because most of the calculation and feature engineering I do is on a per-user basis (and I imagine this is true for most  features you might want to engineer for this dataset), it makes sense to split it up such that **each chunk contains all the data  for some subset of users**.\n\nThis way, I can perform my calculations independently on each chunk, Pandas will create its temporary overhead on just a small fraction of the data, and I can then manually aggregate the per-chunk results however I need to. *(in a way, I'm manually replicating a lot of how Dask does things, except Dask can't split on anything but an index... and can't have multi-index... and makes some assumptions about groupby)*"},{"metadata":{},"cell_type":"markdown","source":"# Load the training data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%time\n# load data via datatable: \n# much faster to ask datatable to load it, and then to translate to Pandas, than trying to get Pandas to load the .csv\ntrain_dt = dt.fread(\"../input/riiid-test-answer-prediction/train.csv\")\ntrain_df = train_dt.to_pandas()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculate per-user running total of rows\n\nI sort the user ids in ascending order, so that I can select a range of users with `<` and `>=`\n\nI then get the cumulative number of training data rows for all users so far."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# get cumulative counts of users\nuser_counts = train_df.groupby('user_id').count()\nuser_counts.sort_index(inplace=True)\nuser_counts['total_rows'] = user_counts['timestamp'].cumsum()\nuser_counts[['timestamp','total_rows']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Find \"break points\"\n\nI semi-arbitrarily decided I want roughly 500000 rows per chunk - this makes for about 200 chunks.\n\nFor each \"break point\" (multiple of 500000), I find the user whose `total_rows` is closest to that break point. \nAll users between this user and the previous \"break point\" user go into the current chunk.\n\n(and don't forget the last chunk - all users after the last break point)"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n\nrows_per_df = 500000\ndf_split = []\nprev_break_user = 0\nfor row_break in range(rows_per_df, 101230332, rows_per_df):\n    print(row_break)\n    break_i = user_counts['total_rows'].searchsorted(row_break)\n    break_user = user_counts.iloc[break_i].name\n    %time part = train_df.loc[(train_df['user_id']<=break_user) & (train_df['user_id']>prev_break_user)]\n    prev_break_user = break_user\n    df_split.append(part)\n    gc.collect()\n# last bit\nprint('>',row_break)\npart = train_df.loc[train_df['user_id']>break_user]\ndf_split.append(part)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfor i, df in enumerate(df_split):\n    df.to_csv(f'train_{i}.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Example use case\n\nThis example is based loosely on the idea of Learning Factor Analysis: that students get better at a skill as they practice it, and that this learning curve follows a specific power curve. See [this notebook](https://www.kaggle.com/yanamal/learning-factor-analysis-are-tags-skills) for details if you're interested.\n\nFor simplicity, in this example I'll just pretend that \"answering all kinds of TOEIC questions\" is a learnable skill. This is not particularly true in practice, but this is just an illustration of how I might deal with the split-out dataframes."},{"metadata":{},"cell_type":"markdown","source":"First, add a column to each dataframe to represent per-user \"encounters\" with the \"skill\" of answering questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndef add_peruser_encounters(df):\n    df['encounters'] = df.groupby('user_id').cumcount()+1\n    \n\n[add_peruser_encounters(df) for df in df_split]\nprint()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now I want to do logistic rergression to predict the `answered_correctly` column, using the `encounters` column as input.\n\nThe simpler way to do this would be to pull out the relevant columns from each chunk and then stick them together them using `np.concatenate`, e.g. `all_encounters = np.concatenate([df['encounters'] for df in df_split])`. \n\nBut in this case, the logistic regression actually runs out of memory! Also it's actually needlessly slow, since what we really have is lots of copies of the same few combinations of `answered_correctly` and `encounters` (there are only so many combinations possible!)\n\nSo instead, I count the number of occurrences of each combination; and then use those counts as weights for the regression."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# sum up all the weights\n\ndef count_encounter_and_label_combinations(df):\n    qs = df[df['answered_correctly']!=-1]  # skip lectures (I want to classify question answers)\n    return qs.groupby(['encounters', 'answered_correctly'])['timestamp'].count()\n\nweights = [count_encounter_and_label_combinations(df) for df in df_split]\n\n# add up all the weights (I could also probably use the fold function, but I'm trying to keep it readable)\nsum_weights = weights[0]\nfor i in range(1, len(weights)):\n    sum_weights.add(weights[i], fill_value=0)\nsum_weights","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# delete intermediate variable, call garbage collection\ndel weights\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can extract the relevant fields from this multi-index series (it's a bit finicky but doable) and the actual regression takes no time at all:"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = np.array(sum_weights.index.get_level_values('encounters')).reshape(-1,1)\ny = sum_weights.index.get_level_values('answered_correctly')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nl = LogisticRegression().fit(X,y,sample_weight=sum_weights)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some sample results of the regresssion:"},{"metadata":{"trusted":true},"cell_type":"code","source":"probabilities = l.predict_proba(np.array(sum_weights.index.get_level_values('encounters')).reshape(-1,1))\nprobabilities","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# regression score (more is better)\nl.score(X, y, sample_weight=sum_weights)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# another scoring metric (less is better)\nlog_loss(y, probabilities[:,0], sample_weight=sum_weights)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr, thresholds = roc_curve(y, probabilities[:,0])\nroc_auc = auc(fpr, tpr)\nroc_auc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well, I mean... it's better than random?.."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}