{
  "id": 200483,
  "title": "Dict+List comprehension is >10x faster than pandas filter",
  "url": "/competitions/riiid-test-answer-prediction/discussion/200483",
  "author_name": "Anurag Trivedi",
  "post_date": "2020-11-30T19:29:15.380000",
  "votes": 7,
  "comment_count": 0,
  "views": 0,
  "content": "<p>For same train set, a code like this runs in 0.13 hours:</p>\n<pre><code>    for row in train.itertuples():\n        try:\n            dict_mini = {}\n            dict_mini = dict_in[row.user_id]\n            train['l_incor_a_ts'][row.Index] = [x for x in dict_mini if x&lt;row.timestamp][-1]\n            ...\n</code></pre>\n<p>compared to a code like this which runs in 1.8 hours:</p>\n<pre><code>    for row in train.itertuples():\n        try:\n            temp_df = df_incor_a.loc[df_incor_a['user_id']==row.user_id]\n            train['l_incor_a_ts'][row.Index] = temp_df.loc[(temp_df['incor_a_ts']&lt;row.timestamp)].tail(1)['incor_a_ts']\n            ...\n</code></pre>",
  "messages": [
    {
      "id": 1096793,
      "postDate": "2020-11-30T19:29:15.380Z",
      "content": "<p>For same train set, a code like this runs in 0.13 hours:</p>\n<pre><code>    for row in train.itertuples():\n        try:\n            dict_mini = {}\n            dict_mini = dict_in[row.user_id]\n            train['l_incor_a_ts'][row.Index] = [x for x in dict_mini if x&lt;row.timestamp][-1]\n            ...\n</code></pre>\n<p>compared to a code like this which runs in 1.8 hours:</p>\n<pre><code>    for row in train.itertuples():\n        try:\n            temp_df = df_incor_a.loc[df_incor_a['user_id']==row.user_id]\n            train['l_incor_a_ts'][row.Index] = temp_df.loc[(temp_df['incor_a_ts']&lt;row.timestamp)].tail(1)['incor_a_ts']\n            ...\n</code></pre>",
      "rawMarkdown": "For same train set, a code like this runs in 0.13 hours:\n```\n    for row in train.itertuples():\n        try:\n            dict_mini = {}\n            dict_mini = dict_in[row.user_id]\n            train['l_incor_a_ts'][row.Index] = [x for x in dict_mini if x<row.timestamp][-1]\n            ...\n```\ncompared to a code like this which runs in 1.8 hours:\n```\n    for row in train.itertuples():\n        try:\n            temp_df = df_incor_a.loc[df_incor_a['user_id']==row.user_id]\n            train['l_incor_a_ts'][row.Index] = temp_df.loc[(temp_df['incor_a_ts']<row.timestamp)].tail(1)['incor_a_ts']\n            ...\n```",
      "votes": 7
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1096793": "For same train set, a code like this runs in 0.13 hours:\n```\n    for row in train.itertuples():\n        try:\n            dict_mini = {}\n            dict_mini = dict_in[row.user_id]\n            train['l_incor_a_ts'][row.Index] = [x for x in dict_mini if x<row.timestamp][-1]\n            ...\n```\ncompared to a code like this which runs in 1.8 hours:\n```\n    for row in train.itertuples():\n        try:\n            temp_df = df_incor_a.loc[df_incor_a['user_id']==row.user_id]\n            train['l_incor_a_ts'][row.Index] = temp_df.loc[(temp_df['incor_a_ts']<row.timestamp)].tail(1)['incor_a_ts']\n            ...\n```"
  }
}