{ "id": 206141, "title": "Why using merge with Questions_df fails", "url": "/competitions/riiid-test-answer-prediction/discussion/206141", "author_name": "Jaideep", "post_date": "2020-12-23T11:56:02.270000", "votes": 1, "comment_count": 11, "views": 0, "content": "

Hi all,
\ni am having very tough time to understand some weird things..
\nbelow from my Test inference loop. It fails with in 2 minutes run of submission

\n
for (test_df, sample_prediction_df) in iter_test:\n\n        test_df=pd.merge(test_df,questions_df[['question_id','part']],\n                         left_on='content_id',right_on='question_id',copy=False)\n
\n

This below thing works

\n
for (test_df, sample_prediction_df) in iter_test:\n\n        test_df=pd.merge(test_df,questions_df[['question_id','part']],\n                         left_on='content_id',right_on='question_id',copy=False,how='left)\n       test_df[test_df.part.isnull(),part]=1 \n
\n

The below also works

\n

test_df['part]=test_df.map(dict)

", "messages": [ { "id": 1123653, "postDate": "2020-12-23T11:56:02.270Z", "content": "

Hi all,
\ni am having very tough time to understand some weird things..
\nbelow from my Test inference loop. It fails with in 2 minutes run of submission

\n
for (test_df, sample_prediction_df) in iter_test:\n\n        test_df=pd.merge(test_df,questions_df[['question_id','part']],\n                         left_on='content_id',right_on='question_id',copy=False)\n
\n

This below thing works

\n
for (test_df, sample_prediction_df) in iter_test:\n\n        test_df=pd.merge(test_df,questions_df[['question_id','part']],\n                         left_on='content_id',right_on='question_id',copy=False,how='left)\n       test_df[test_df.part.isnull(),part]=1 \n
\n

The below also works

\n

test_df['part]=test_df.map(dict)

", "rawMarkdown": "Hi all,\ni am having very tough time to understand some weird things..\nbelow from my Test inference loop. It fails with in 2 minutes run of submission\n\n ```\nfor (test_df, sample_prediction_df) in iter_test:\n \n test_df=pd.merge(test_df,questions_df[['question_id','part']],\n left_on='content_id',right_on='question_id',copy=False)\n```\nThis below thing works\n ```\nfor (test_df, sample_prediction_df) in iter_test:\n \n test_df=pd.merge(test_df,questions_df[['question_id','part']],\n left_on='content_id',right_on='question_id',copy=False,how='left)\n test_df[test_df.part.isnull(),part]=1 \n```\n\nThe below also works\n\n`test_df['part]=test_df.map(dict) `", "votes": 1 }, { "id": 1124461, "postDate": "2020-12-24T01:02:37.387Z", "content": "

By default merge is having a inner join, if you don't change that, you will miss on rows basically; And that's what breaks your code as row counts doesn't match.

", "rawMarkdown": "By default merge is having a inner join, if you don't change that, you will miss on rows basically; And that's what breaks your code as row counts doesn't match.", "replies": [ { "id": 1124524, "postDate": "2020-12-24T02:47:33.333Z", "content": "

@adityaecdrid see my response below

", "rawMarkdown": "@adityaecdrid see my response below " } ] }, { "id": 1124428, "postDate": "2020-12-23T23:36:32.703Z", "content": "

how=left is required to preserve the original row order.

", "rawMarkdown": "`how=left` is required to preserve the original row order.", "replies": [ { "id": 1124523, "postDate": "2020-12-24T02:46:54.297Z", "content": "

@nyanpn inner join will lead to only content id = False rows as we joining with questions, ,which any ways we have to do, before submitting preds. So that is what making me scratch head

", "rawMarkdown": "@nyanpn inner join will lead to only content id = False rows as we joining with questions, ,which any ways we have to do, before submitting preds. So that is what making me scratch head \n" }, { "id": 1124546, "postDate": "2020-12-24T03:32:54.280Z", "content": "
\n

inner join will lead to only content id = False rows as we joining with questions

\n
\n

But the answers have values for lecture rows as well; you will have a shape mismatch when you will try updating your cache; Look into common failures tips!

", "rawMarkdown": ">inner join will lead to only content id = False rows as we joining with questions\n\nBut the answers have values for lecture rows as well; you will have a shape mismatch when you will try updating your cache; Look into common failures tips!" }, { "id": 1124559, "postDate": "2020-12-24T03:53:25.383Z", "content": "

Hmm ,I get a bit i did thought about it but here comes a conflict
\nIf I do
\ntest [col]= test.content I'd. Map (dict) this works as it preserves rows goes with explanation.

\n

test [col]= test.content I'd. Map (lambda x : df.loc[x,col][0])
\nFails

", "rawMarkdown": "Hmm ,I get a bit i did thought about it but here comes a conflict \nIf I do\n` test [col]= test.content I'd. Map (dict) ` this works as it preserves rows goes with explanation. \n\n` test [col]= test.content I'd. Map (lambda x : df.loc[x,col][0]) ` \nFails \n\n\n" }, { "id": 1124560, "postDate": "2020-12-24T03:53:25.383Z", "content": "

Hmm ,I get a bit i did thought about it but here comes a conflict
\nIf I do
\ntest [col]= test.content I'd. Map (dict) this works as it preserves rows goes with explanation.

\n

test [col]= test.content I'd. Map (lambda x : df.loc[x,col][0])
\nFails

\n

Moreover I do prev df with content id =False before updating answers . If there is size mismatch issue then I should get the same during commit itself

", "rawMarkdown": "Hmm ,I get a bit i did thought about it but here comes a conflict \nIf I do\n` test [col]= test.content I'd. Map (dict) ` this works as it preserves rows goes with explanation. \n\n` test [col]= test.content I'd. Map (lambda x : df.loc[x,col][0]) ` \nFails \n\nMoreover I do prev df with content id =False before updating answers . If there is size mismatch issue then I should get the same during commit itself \n" } ] }, { "id": 1124342, "postDate": "2020-12-23T20:53:32.987Z", "content": "

I think we need to see the whole loop to understand.

", "rawMarkdown": "I think we need to see the whole loop to understand.", "replies": [ { "id": 1124530, "postDate": "2020-12-24T02:57:31.013Z", "content": "

@ahmeterdem

\n
for (test_df, sample_prediction_df) in iter_test:\ntest_df=pd.merge(test_df,questions_df[['question_id','part']],  left_on='content_id',right_on='question_id',copy=False)\nIf Prev df is not none:\n......\nPrev df = copy test df \nTest df = testdf content type =0\n
", "rawMarkdown": "\n\n@ahmeterdem\n```\nfor (test_df, sample_prediction_df) in iter_test:\ntest_df=pd.merge(test_df,questions_df[['question_id','part']], left_on='content_id',right_on='question_id',copy=False)\n\nIf Prev df is not none:\n......\n\nPrev df = copy test df \n\nTest df = testdf content type =0\n```" }, { "id": 1124873, "postDate": "2020-12-24T08:44:46.897Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1124892, "postDate": "2020-12-24T09:03:02.430Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1124461, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-24T01:02:37.387000", "content": "

By default merge is having a inner join, if you don't change that, you will miss on rows basically; And that's what breaks your code as row counts doesn't match.

", "votes": 0, "replies": [ { "id": 1124524, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-24T02:47:33.333000", "content": "

@adityaecdrid see my response below

", "votes": 0, "replies": [] } ] }, { "id": 1124428, "author_name": "nyanp", "author_url": "", "post_date": "2020-12-23T23:36:32.703000", "content": "

how=left is required to preserve the original row order.

", "votes": 0, "replies": [ { "id": 1124523, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-24T02:46:54.297000", "content": "

@nyanpn inner join will lead to only content id = False rows as we joining with questions, ,which any ways we have to do, before submitting preds. So that is what making me scratch head

", "votes": 0, "replies": [] }, { "id": 1124546, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-24T03:32:54.280000", "content": "
\n

inner join will lead to only content id = False rows as we joining with questions

\n
\n

But the answers have values for lecture rows as well; you will have a shape mismatch when you will try updating your cache; Look into common failures tips!

", "votes": 0, "replies": [] }, { "id": 1124559, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-24T03:53:25.383000", "content": "

Hmm ,I get a bit i did thought about it but here comes a conflict
\nIf I do
\ntest [col]= test.content I'd. Map (dict) this works as it preserves rows goes with explanation.

\n

test [col]= test.content I'd. Map (lambda x : df.loc[x,col][0])
\nFails

", "votes": 0, "replies": [] }, { "id": 1124560, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-24T03:53:25.383000", "content": "

Hmm ,I get a bit i did thought about it but here comes a conflict
\nIf I do
\ntest [col]= test.content I'd. Map (dict) this works as it preserves rows goes with explanation.

\n

test [col]= test.content I'd. Map (lambda x : df.loc[x,col][0])
\nFails

\n

Moreover I do prev df with content id =False before updating answers . If there is size mismatch issue then I should get the same during commit itself

", "votes": 0, "replies": [] } ] }, { "id": 1124342, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-12-23T20:53:32.987000", "content": "

I think we need to see the whole loop to understand.

", "votes": 0, "replies": [ { "id": 1124530, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-24T02:57:31.013000", "content": "

@ahmeterdem

\n
for (test_df, sample_prediction_df) in iter_test:\ntest_df=pd.merge(test_df,questions_df[['question_id','part']],  left_on='content_id',right_on='question_id',copy=False)\nIf Prev df is not none:\n......\nPrev df = copy test df \nTest df = testdf content type =0\n
", "votes": 0, "replies": [] }, { "id": 1124873, "author_name": "", "author_url": "", "post_date": "2020-12-24T08:44:46.897000", "content": "", "votes": 0, "replies": [] }, { "id": 1124892, "author_name": "", "author_url": "", "post_date": "2020-12-24T09:03:02.430000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1123653": "Hi all,\ni am having very tough time to understand some weird things..\nbelow from my Test inference loop. It fails with in 2 minutes run of submission\n\n ```\nfor (test_df, sample_prediction_df) in iter_test:\n \n test_df=pd.merge(test_df,questions_df[['question_id','part']],\n left_on='content_id',right_on='question_id',copy=False)\n```\nThis below thing works\n ```\nfor (test_df, sample_prediction_df) in iter_test:\n \n test_df=pd.merge(test_df,questions_df[['question_id','part']],\n left_on='content_id',right_on='question_id',copy=False,how='left)\n test_df[test_df.part.isnull(),part]=1 \n```\n\nThe below also works\n\n`test_df['part]=test_df.map(dict) `", "1124461": "By default merge is having a inner join, if you don't change that, you will miss on rows basically; And that's what breaks your code as row counts doesn't match.", "1124428": "`how=left` is required to preserve the original row order.", "1124342": "I think we need to see the whole loop to understand." } }