When running certain queries in Whoosh, the program becomes unresponsive and fails to proceed, regardless of the waiting time.
\nI'm not sure about the condition that is causing this problem.
\nCould someone please help investigate?
\nBelow is the code that reproduces the problem:
import whoosh_utils\ntrain_idx = whoosh_utils.load\nsearcher = whoosh_utils.get\nqp = whoosh_utils.get\n\nquery_1 = 'cpc:G03G15/ OR cpc:B33Y10/'\nwhoosh_utils.execute # this line works without freezing\n\nquery_2 = 'cpc:G03G15/ OR cpc:B33Y10/ OR cpc:B33Y30/ OR ti:device'\nwhoosh_utils.execute # this line also works without freezing\n\nquery_3 = 'cpc:G03G15/ OR cpc:B33Y10/ OR cpc:B33Y30/'\nwhoosh_utils.execute # this line not works causes freezing\nWhen running certain queries in Whoosh, the program becomes unresponsive and fails to proceed, regardless of the waiting time.
\nI'm not sure about the condition that is causing this problem.
\nCould someone please help investigate?
\nBelow is the code that reproduces the problem:
import whoosh_utils\ntrain_idx = whoosh_utils.load\nsearcher = whoosh_utils.get\nqp = whoosh_utils.get\n\nquery_1 = 'cpc:G03G15/ OR cpc:B33Y10/'\nwhoosh_utils.execute # this line works without freezing\n\nquery_2 = 'cpc:G03G15/ OR cpc:B33Y10/ OR cpc:B33Y30/ OR ti:device'\nwhoosh_utils.execute # this line also works without freezing\n\nquery_3 = 'cpc:G03G15/ OR cpc:B33Y10/ OR cpc:B33Y30/'\nwhoosh_utils.execute # this line not works causes freezing\nIt appears that floating-point errors caused an infinite loop in the skip_to_quality while loop in whoosh/matching/binary.py.
\n/opt/conda/lib/python3.10/site-packages/whoosh/matching/binary.py
\n...\n ():\n a = self.a\n b = self.b\n minquality = minquality\n\n a.is_active():\n mcore.ReadTooFar\n b.is_active():\n a.skip_to_quality(minquality)\n\n skipped = \n aq = a.block_quality()\n bq = b.block_quality()\n \n \n \n \n \n a.is_active() b.is_active() aq + bq <= minquality:\n aq < bq:\n skipped += a.skip_to_quality(minquality - bq)\n aq = a.block_quality()\n :\n \n skipped += b.skip_to_quality(minquality - aq)\n bq = b.block_quality()\n\n skipped\n...\n/opt/conda/lib/python3.10/site-packages/whoosh/codec/whoosh3.py
\n...\n ():\n \n \n\n block_quality = self.block_quality\n\n \n \n\n \n \n \n block_quality() > minquality:\n \n\n \n \n self._skip_to_block(: block_quality() <= minquality)\n...\nAs a workaround, setting a tolerance for absolute errors has now allowed results to return quickly even for query_3.
\n\n \n\n \n \n\n \n\n \n\n \n\n \n\n\n \ncode: https://www.kaggle.com/code/sash2104/debug-uspto-discussion-498283
", "rawMarkdown": "It appears that floating-point errors caused an infinite loop in the skip_to_quality while loop in whoosh/matching/binary.py.\n\n/opt/conda/lib/python3.10/site-packages/whoosh/matching/binary.py\n```py\n...\n def skip_to_quality(self, minquality):\n a = self.a\n b = self.b\n minquality = minquality\n\n if not a.is_active():\n raise mcore.ReadTooFar\n if not b.is_active():\n return a.skip_to_quality(minquality)\n\n skipped = 0\n aq = a.block_quality()\n bq = b.block_quality()\n # aq = 7.781122628048771\n # bq = 7.362528228498814\n # minquality = 15.143650856547584\n # `aq < bq` is False\n # `aq + bq <= minquarity` is True\n while a.is_active() and b.is_active() and aq + bq <= minquality:\n if aq < bq:\n skipped += a.skip_to_quality(minquality - bq)\n aq = a.block_quality()\n else:\n # Goes to skip_to_quality in W3LeafMatcher of codec/whoosh3.py\n skipped += b.skip_to_quality(minquality - aq)\n bq = b.block_quality()\n\n return skipped\n...\n```\n\n/opt/conda/lib/python3.10/site-packages/whoosh/codec/whoosh3.py\n```py\n...\n def skip_to_quality(self, minquality):\n # Skip blocks until we find one that might exceed the given minimum\n # quality\n\n block_quality = self.block_quality\n\n # If the quality of this block is already higher than the minimum,\n # do nothing\n\n # block_quality() = 7.362528228498814\n # minquality = 7.362528228498813\n # `block_quality() > minquality` is True\n if block_quality() > minquality:\n return 0\n\n # Skip blocks as long as the block quality is not greater than the\n # minimum\n return self._skip_to_block(lambda: block_quality() <= minquality)\n...\n```\n\nAs a workaround, setting a tolerance for absolute errors has now allowed results to return quickly even for query_3.\n```\n%%bash\ncat << EOS > whoosh3.patch\n1039a1040,1041\n> tolerance = 1e-10\n> \n1042c1044\n< if block_quality() > minquality:\n---\n> if block_quality() > minquality + tolerance:\n1047c1049\n< return self._skip_to_block(lambda: block_quality() <= minquality)\n---\n> return self._skip_to_block(lambda: block_quality() <= minquality + tolerance)\nEOS\n\npatch /opt/conda/lib/python3.10/site-packages/whoosh/codec/whoosh3.py whoosh3.patch\n```\ncode: https://www.kaggle.com/code/sash2104/debug-uspto-discussion-498283", "votes": null }, { "id": "2780171", "postDate": "04/28/2024 04:55:59", "content": "Wow, thank you so much for the speedy and detailed investigation!
\nReally appreciate it!
I had a submission time out just now. The queries were the same complexity as a submission that finished in 35 minutes, so I am thinking it is likely this bug is to fault. When testing some of the queries did trigger the bug on the index I am using. Hopefully the patch can be applied (or some other work around found) so we don't have to worry about solutions timing out from this bug.
", "rawMarkdown": "I had a submission time out just now. The queries were the same complexity as a submission that finished in 35 minutes, so I am thinking it is likely this bug is to fault. When testing some of the queries did trigger the bug on the index I am using. Hopefully the patch can be applied (or some other work around found) so we don't have to worry about solutions timing out from this bug.", "votes": null }, { "id": "2781844", "postDate": "04/29/2024 03:17:05", "content": "i'm facing scoring error too, didn't found solution how to fix it. is it take more then 1 hour, do you got scoring error? @devinanzelmo
", "rawMarkdown": "i'm facing scoring error too, didn't found solution how to fix it. is it take more then 1 hour, do you got scoring error? @devinanzelmo", "votes": null }, { "id": "2782101", "postDate": "04/29/2024 06:01:08", "content": "Thanks to @sash2104 's investigation, many queries that were freezing no longer do so.
\nHowever, freezing still occurs with some queries as described below.
= 'cpc:C07B2200/ OR cpc:A61P35/ OR cpc:C07D487/ OR cpc:C07D471/ OR cpc:A61P43/ OR cpc:C07D401/ OR cpc:A61P31/ OR cpc:A61K31/ OR cpc:A61P31/ OR cpc:C07D513/ OR cpc:C07D403/ OR cpc:C07D417/ OR cpc:C07D405/'\n.execute_query(query_4, qp, searcher) # this line not works and causes freezing\nI also suspect that this approach is not working during the score calculation of submissions.
\nIf the score calculation of submissions runs in a separate environment from our notebook, we cannot implement any freeze prevention from our side.
\nI'm concerned that due to freezing, it might reach the 1 hour limit and result in a Submission Scoring Error.
As you suspected, this approach is not working for the metric notebook because it is out of our control. It is merely a workaround that is better than not being able to experiment at all, even when running locally. Ultimately, we have to hope that the host will address the issue, so all we can do is wait.
\nBy the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue.
", "rawMarkdown": "As you suspected, this approach is not working for the metric notebook because it is out of our control. It is merely a workaround that is better than not being able to experiment at all, even when running locally. Ultimately, we have to hope that the host will address the issue, so all we can do is wait.\n\nBy the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue.", "votes": null }, { "id": "2782584", "postDate": "04/29/2024 11:23:44", "content": "Yeah it says we only get one hour to score, and I got a scoring error after running 1+ hours. If there is a problem with the queries it usually gives an error in under 15 minutes. No solution yet. If it is a bug with Whoosh we will have to wait for the hosts to fix it.
", "rawMarkdown": "Yeah it says we only get one hour to score, and I got a scoring error after running 1+ hours. If there is a problem with the queries it usually gives an error in under 15 minutes. No solution yet. If it is a bug with Whoosh we will have to wait for the hosts to fix it.", "votes": null }, { "id": "2782599", "postDate": "04/29/2024 11:31:57", "content": "I calculated the notebook run time for over 2500 patents and some additional tasks. When the extra tasks extended beyond one hour, there was a score error, which I cannot replicate in Kaggle without knowing the specifics of the Kaggle environment or the maximum size for the training. @devinanzelmo
", "rawMarkdown": "I calculated the notebook run time for over 2500 patents and some additional tasks. When the extra tasks extended beyond one hour, there was a score error, which I cannot replicate in Kaggle without knowing the specifics of the Kaggle environment or the maximum size for the training. @devinanzelmo", "votes": null }, { "id": "2783203", "postDate": "04/29/2024 16:09:15", "content": "I can pretty much confirm the metric is suffering from bug. I took a submission that timed out and replaced a OR b with (a AND a) OR (b AND b). The more complex one finished in 24 minutes while the simpler one timed out.
Thanks, i will submit similar way for my failed case @devinanzelmo
", "rawMarkdown": "Thanks, i will submit similar way for my failed case @devinanzelmo", "votes": null }, { "id": "2783214", "postDate": "04/29/2024 16:13:46", "content": "It can still run into the bug even doing (a AND a) but maybe its less frequent not sure yet.
A similar issue was reported on Whoosh's GitHub page, and it appears to be already addressed in the forked repository whoosh-reloaded.
\nIssue: https://github.com/whoosh-community/whoosh/issues/461
\nFix PRs:
I tested with whoosh-reloaded version 2.7.5, and it seems that the issue has been resolved, at least for query_3 and query_4.
\nCode: https://www.kaggle.com/code/sash2104/debug-ver-2-uspto-discussion-498283
Thank you all for flagging this and for the detailed follow up. I will investigate the feasibility of incorporating the patch @sash2104 outlined, though I may not be able to get to it until later this week.
", "rawMarkdown": "Thank you all for flagging this and for the detailed follow up. I will investigate the feasibility of incorporating the patch @sash2104 outlined, though I may not be able to get to it until later this week.", "votes": null }, { "id": "2785419", "postDate": "04/30/2024 18:56:50", "content": "Thank you @sohier, is patch from @sash2104 ( still have few cases freeze )
\n\n", "rawMarkdown": "Thank you @sohier, is patch from @sash2104 ( still have few cases freeze )\n> By the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue. by @sash2104", "votes": null }, { "id": "2793837", "postDate": "05/05/2024 02:41:11", "content": "By the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue. by @sash2104
\n
Hi @sohier, is there any update on the issue?
", "rawMarkdown": "Hi @sohier, is there any update on the issue?", "votes": null }, { "id": "2797208", "postDate": "05/06/2024 15:41:46", "content": "I took a look at whoosh-reloaded, and while it looks great it's unfortunately too broad for me to properly test and incorporate with the competition already live. I'll need to instead make a custom patch that applies only the single fix @sash2104 noted. My availability is pretty limited this week due to travel but I will try to get to this soon.
", "rawMarkdown": "I took a look at whoosh-reloaded, and while it looks great it's unfortunately too broad for me to properly test and incorporate with the competition already live. I'll need to instead make a custom patch that applies only the single fix @sash2104 noted. My availability is pretty limited this week due to travel but I will try to get to this soon.", "votes": null }, { "id": "2797236", "postDate": "05/06/2024 15:52:04", "content": "Thanks @sohier, is possible to move cpc schema as TEXT instead KEYWORD
\n\n\nschema = whoosh.fields.Schema(
\n
\n id=whoosh.fields.ID(stored=True),
\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n cpc=whoosh.fields.KEYWORD(stored=False, scorable=True)
\n )
=======>
\n\n", "rawMarkdown": "Thanks @sohier, is possible to move cpc schema as TEXT instead KEYWORD\n> schema = whoosh.fields.Schema(\n id=whoosh.fields.ID(stored=True),\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n **cpc=whoosh.fields.KEYWORD(stored=False, scorable=True)**\n )\n\n=======>\n> schema = whoosh.fields.Schema(\n id=whoosh.fields.ID(stored=True),\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n **cpc=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False, scorable=True)**\n )", "votes": null }, { "id": "2808635", "postDate": "05/12/2024 10:26:29", "content": "schema = whoosh.fields.Schema(
\n
\n id=whoosh.fields.ID(stored=True),
\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n cpc=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False, scorable=True)
\n )
Can someone confirm if this is fixed (or at least partially fixed)?
", "rawMarkdown": "Can someone confirm if this is fixed (or at least partially fixed)?", "votes": null }, { "id": "2828059", "postDate": "05/21/2024 20:42:52", "content": "The patch is live. Sorry for the slow response and thank you all for the valuable discussion!
", "rawMarkdown": "[The patch is live](https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/506428). Sorry for the slow response and thank you all for the valuable discussion!", "votes": null }, { "id": "2828076", "postDate": "05/21/2024 21:00:17", "content": "Thank you @sohier
", "rawMarkdown": "Thank you @sohier", "votes": null } ], "comments": [ { "id": 2780133, "author_name": "sash2104", "author_url": "", "post_date": "04/28/2024 04:25:12", "content": "It appears that floating-point errors caused an infinite loop in the skip_to_quality while loop in whoosh/matching/binary.py.
\n/opt/conda/lib/python3.10/site-packages/whoosh/matching/binary.py
\n...\n ():\n a = self.a\n b = self.b\n minquality = minquality\n\n a.is_active():\n mcore.ReadTooFar\n b.is_active():\n a.skip_to_quality(minquality)\n\n skipped = \n aq = a.block_quality()\n bq = b.block_quality()\n \n \n \n \n \n a.is_active() b.is_active() aq + bq <= minquality:\n aq < bq:\n skipped += a.skip_to_quality(minquality - bq)\n aq = a.block_quality()\n :\n \n skipped += b.skip_to_quality(minquality - aq)\n bq = b.block_quality()\n\n skipped\n...\n/opt/conda/lib/python3.10/site-packages/whoosh/codec/whoosh3.py
\n...\n ():\n \n \n\n block_quality = self.block_quality\n\n \n \n\n \n \n \n block_quality() > minquality:\n \n\n \n \n self._skip_to_block(: block_quality() <= minquality)\n...\nAs a workaround, setting a tolerance for absolute errors has now allowed results to return quickly even for query_3.
\n\n \n\n \n \n\n \n\n \n\n \n\n \n\n\n \ncode: https://www.kaggle.com/code/sash2104/debug-uspto-discussion-498283
", "votes": null, "replies": [ { "id": 2780171, "author_name": "shunrcn", "author_url": "", "post_date": "04/28/2024 04:55:59", "content": "Wow, thank you so much for the speedy and detailed investigation!
\nReally appreciate it!
I had a submission time out just now. The queries were the same complexity as a submission that finished in 35 minutes, so I am thinking it is likely this bug is to fault. When testing some of the queries did trigger the bug on the index I am using. Hopefully the patch can be applied (or some other work around found) so we don't have to worry about solutions timing out from this bug.
", "votes": null, "replies": [ { "id": 2781844, "author_name": "seshurajup", "author_url": "", "post_date": "04/29/2024 03:17:05", "content": "i'm facing scoring error too, didn't found solution how to fix it. is it take more then 1 hour, do you got scoring error? @devinanzelmo
", "votes": null, "replies": [] }, { "id": 2782584, "author_name": "devinanzelmo", "author_url": "", "post_date": "04/29/2024 11:23:44", "content": "Yeah it says we only get one hour to score, and I got a scoring error after running 1+ hours. If there is a problem with the queries it usually gives an error in under 15 minutes. No solution yet. If it is a bug with Whoosh we will have to wait for the hosts to fix it.
", "votes": null, "replies": [ { "id": 2782599, "author_name": "seshurajup", "author_url": "", "post_date": "04/29/2024 11:31:57", "content": "I calculated the notebook run time for over 2500 patents and some additional tasks. When the extra tasks extended beyond one hour, there was a score error, which I cannot replicate in Kaggle without knowing the specifics of the Kaggle environment or the maximum size for the training. @devinanzelmo
", "votes": null, "replies": [] } ] }, { "id": 2783203, "author_name": "devinanzelmo", "author_url": "", "post_date": "04/29/2024 16:09:15", "content": "I can pretty much confirm the metric is suffering from bug. I took a submission that timed out and replaced a OR b with (a AND a) OR (b AND b). The more complex one finished in 24 minutes while the simpler one timed out.
Thanks, i will submit similar way for my failed case @devinanzelmo
", "votes": null, "replies": [] }, { "id": 2783214, "author_name": "devinanzelmo", "author_url": "", "post_date": "04/29/2024 16:13:46", "content": "It can still run into the bug even doing (a AND a) but maybe its less frequent not sure yet.
Thanks to @sash2104 's investigation, many queries that were freezing no longer do so.
\nHowever, freezing still occurs with some queries as described below.
= 'cpc:C07B2200/ OR cpc:A61P35/ OR cpc:C07D487/ OR cpc:C07D471/ OR cpc:A61P43/ OR cpc:C07D401/ OR cpc:A61P31/ OR cpc:A61K31/ OR cpc:A61P31/ OR cpc:C07D513/ OR cpc:C07D403/ OR cpc:C07D417/ OR cpc:C07D405/'\n.execute_query(query_4, qp, searcher) # this line not works and causes freezing\nI also suspect that this approach is not working during the score calculation of submissions.
\nIf the score calculation of submissions runs in a separate environment from our notebook, we cannot implement any freeze prevention from our side.
\nI'm concerned that due to freezing, it might reach the 1 hour limit and result in a Submission Scoring Error.
As you suspected, this approach is not working for the metric notebook because it is out of our control. It is merely a workaround that is better than not being able to experiment at all, even when running locally. Ultimately, we have to hope that the host will address the issue, so all we can do is wait.
\nBy the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue.
", "votes": null, "replies": [] }, { "id": 2784358, "author_name": "sash2104", "author_url": "", "post_date": "04/30/2024 08:14:02", "content": "A similar issue was reported on Whoosh's GitHub page, and it appears to be already addressed in the forked repository whoosh-reloaded.
\nIssue: https://github.com/whoosh-community/whoosh/issues/461
\nFix PRs:
I tested with whoosh-reloaded version 2.7.5, and it seems that the issue has been resolved, at least for query_3 and query_4.
\nCode: https://www.kaggle.com/code/sash2104/debug-ver-2-uspto-discussion-498283
Thank you all for flagging this and for the detailed follow up. I will investigate the feasibility of incorporating the patch @sash2104 outlined, though I may not be able to get to it until later this week.
", "votes": null, "replies": [ { "id": 2785419, "author_name": "seshurajup", "author_url": "", "post_date": "04/30/2024 18:56:50", "content": "Thank you @sohier, is patch from @sash2104 ( still have few cases freeze )
\n\n", "votes": null, "replies": [] }, { "id": 2793837, "author_name": "suicaokhoailang", "author_url": "", "post_date": "05/05/2024 02:41:11", "content": "By the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue. by @sash2104
\n
Hi @sohier, is there any update on the issue?
", "votes": null, "replies": [] }, { "id": 2797208, "author_name": "sohier", "author_url": "", "post_date": "05/06/2024 15:41:46", "content": "I took a look at whoosh-reloaded, and while it looks great it's unfortunately too broad for me to properly test and incorporate with the competition already live. I'll need to instead make a custom patch that applies only the single fix @sash2104 noted. My availability is pretty limited this week due to travel but I will try to get to this soon.
", "votes": null, "replies": [ { "id": 2797236, "author_name": "seshurajup", "author_url": "", "post_date": "05/06/2024 15:52:04", "content": "Thanks @sohier, is possible to move cpc schema as TEXT instead KEYWORD
\n\n\nschema = whoosh.fields.Schema(
\n
\n id=whoosh.fields.ID(stored=True),
\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n cpc=whoosh.fields.KEYWORD(stored=False, scorable=True)
\n )
=======>
\n\n", "votes": null, "replies": [] } ] } ] }, { "id": 2808635, "author_name": "suicaokhoailang", "author_url": "", "post_date": "05/12/2024 10:26:29", "content": "schema = whoosh.fields.Schema(
\n
\n id=whoosh.fields.ID(stored=True),
\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),
\n cpc=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False, scorable=True)
\n )
Can someone confirm if this is fixed (or at least partially fixed)?
", "votes": null, "replies": [] }, { "id": 2828059, "author_name": "sohier", "author_url": "", "post_date": "05/21/2024 20:42:52", "content": "The patch is live. Sorry for the slow response and thank you all for the valuable discussion!
", "votes": null, "replies": [ { "id": 2828076, "author_name": "seshurajup", "author_url": "", "post_date": "05/21/2024 21:00:17", "content": "Thank you @sohier
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2779359": "When running certain queries in Whoosh, the program becomes unresponsive and fails to proceed, regardless of the waiting time.\nI'm not sure about the condition that is causing this problem.\nCould someone please help investigate?\nBelow is the code that reproduces the problem:\n\n```\nimport whoosh_utils\ntrain_idx = whoosh_utils.load_index('/kaggle/input/uspto-boolean-search-optimization/train_index')\nsearcher = whoosh_utils.get_searcher(train_idx)\nqp = whoosh_utils.get_query_parser()\n\nquery_1 = 'cpc:G03G15/224 OR cpc:B33Y10/00'\nwhoosh_utils.execute_query(query_1, qp, searcher) # this line works without freezing\n\nquery_2 = 'cpc:G03G15/224 OR cpc:B33Y10/00 OR cpc:B33Y30/00 OR ti:device'\nwhoosh_utils.execute_query(query_2, qp, searcher) # this line also works without freezing\n\nquery_3 = 'cpc:G03G15/224 OR cpc:B33Y10/00 OR cpc:B33Y30/00'\nwhoosh_utils.execute_query(query_3, qp, searcher) # this line not works and causes freezing\n```", "2780133": "It appears that floating-point errors caused an infinite loop in the skip_to_quality while loop in whoosh/matching/binary.py.\n\n/opt/conda/lib/python3.10/site-packages/whoosh/matching/binary.py\n```py\n...\n def skip_to_quality(self, minquality):\n a = self.a\n b = self.b\n minquality = minquality\n\n if not a.is_active():\n raise mcore.ReadTooFar\n if not b.is_active():\n return a.skip_to_quality(minquality)\n\n skipped = 0\n aq = a.block_quality()\n bq = b.block_quality()\n # aq = 7.781122628048771\n # bq = 7.362528228498814\n # minquality = 15.143650856547584\n # `aq < bq` is False\n # `aq + bq <= minquarity` is True\n while a.is_active() and b.is_active() and aq + bq <= minquality:\n if aq < bq:\n skipped += a.skip_to_quality(minquality - bq)\n aq = a.block_quality()\n else:\n # Goes to skip_to_quality in W3LeafMatcher of codec/whoosh3.py\n skipped += b.skip_to_quality(minquality - aq)\n bq = b.block_quality()\n\n return skipped\n...\n```\n\n/opt/conda/lib/python3.10/site-packages/whoosh/codec/whoosh3.py\n```py\n...\n def skip_to_quality(self, minquality):\n # Skip blocks until we find one that might exceed the given minimum\n # quality\n\n block_quality = self.block_quality\n\n # If the quality of this block is already higher than the minimum,\n # do nothing\n\n # block_quality() = 7.362528228498814\n # minquality = 7.362528228498813\n # `block_quality() > minquality` is True\n if block_quality() > minquality:\n return 0\n\n # Skip blocks as long as the block quality is not greater than the\n # minimum\n return self._skip_to_block(lambda: block_quality() <= minquality)\n...\n```\n\nAs a workaround, setting a tolerance for absolute errors has now allowed results to return quickly even for query_3.\n```\n%%bash\ncat << EOS > whoosh3.patch\n1039a1040,1041\n> tolerance = 1e-10\n> \n1042c1044\n< if block_quality() > minquality:\n---\n> if block_quality() > minquality + tolerance:\n1047c1049\n< return self._skip_to_block(lambda: block_quality() <= minquality)\n---\n> return self._skip_to_block(lambda: block_quality() <= minquality + tolerance)\nEOS\n\npatch /opt/conda/lib/python3.10/site-packages/whoosh/codec/whoosh3.py whoosh3.patch\n```\ncode: https://www.kaggle.com/code/sash2104/debug-uspto-discussion-498283", "2780171": "Wow, thank you so much for the speedy and detailed investigation!\nReally appreciate it!", "2781312": "I had a submission time out just now. The queries were the same complexity as a submission that finished in 35 minutes, so I am thinking it is likely this bug is to fault. When testing some of the queries did trigger the bug on the index I am using. Hopefully the patch can be applied (or some other work around found) so we don't have to worry about solutions timing out from this bug.", "2781844": "i'm facing scoring error too, didn't found solution how to fix it. is it take more then 1 hour, do you got scoring error? @devinanzelmo", "2782101": "Thanks to @sash2104 's investigation, many queries that were freezing no longer do so. \nHowever, freezing still occurs with some queries as described below.\n\n```\nquery_4 = 'cpc:C07B2200/13 OR cpc:A61P35/00 OR cpc:C07D487/04 OR cpc:C07D471/04 OR cpc:A61P43/00 OR cpc:C07D401/14 OR cpc:A61P31/12 OR cpc:A61K31/519 OR cpc:A61P31/20 OR cpc:C07D513/04 OR cpc:C07D403/06 OR cpc:C07D417/14 OR cpc:C07D405/12'\nwhoosh_utils.execute_query(query_4, qp, searcher) # this line not works and causes freezing\n```\n\nI also suspect that this approach is not working during the score calculation of submissions.\nIf the score calculation of submissions runs in a separate environment from our notebook, we cannot implement any freeze prevention from our side. \nI'm concerned that due to freezing, it might reach the 1 hour limit and result in a Submission Scoring Error.", "2782202": "As you suspected, this approach is not working for the metric notebook because it is out of our control. It is merely a workaround that is better than not being able to experiment at all, even when running locally. Ultimately, we have to hope that the host will address the issue, so all we can do is wait.\n\nBy the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue.", "2782584": "Yeah it says we only get one hour to score, and I got a scoring error after running 1+ hours. If there is a problem with the queries it usually gives an error in under 15 minutes. No solution yet. If it is a bug with Whoosh we will have to wait for the hosts to fix it.", "2782599": "I calculated the notebook run time for over 2500 patents and some additional tasks. When the extra tasks extended beyond one hour, there was a score error, which I cannot replicate in Kaggle without knowing the specifics of the Kaggle environment or the maximum size for the training. @devinanzelmo", "2783203": "I can pretty much confirm the metric is suffering from bug. I took a submission that timed out and replaced `a OR b` with `(a AND a) OR (b AND b)`. The more complex one finished in 24 minutes while the simpler one timed out.", "2783209": "Thanks, i will submit similar way for my failed case @devinanzelmo", "2783214": "It can still run into the bug even doing `(a AND a)` but maybe its less frequent not sure yet.", "2784358": "A similar issue was reported on Whoosh's GitHub page, and it appears to be already addressed in the forked repository whoosh-reloaded.\n\nIssue: https://github.com/whoosh-community/whoosh/issues/461\nFix PRs:\n- https://github.com/whoosh-community/whoosh/pull/519\n- https://github.com/Sygil-Dev/whoosh-reloaded/pull/16\n\nI tested with whoosh-reloaded version 2.7.5, and it seems that the issue has been resolved, at least for query_3 and query_4.\nCode: https://www.kaggle.com/code/sash2104/debug-ver-2-uspto-discussion-498283", "2785354": "Thank you all for flagging this and for the detailed follow up. I will investigate the feasibility of incorporating the patch @sash2104 outlined, though I may not be able to get to it until later this week.", "2785419": "Thank you @sohier, is patch from @sash2104 ( still have few cases freeze )\n> By the way, it seems that query_4 was causing an infinite loop due to the similar issue as query_3, at skip_to_quality (AndMaybeMatcher in whoosh/matching/binary.py). I couldn't figure out how to fix it, so I'm hoping someone else will be able to address the issue. by @sash2104", "2793837": "Hi @sohier, is there any update on the issue?", "2797208": "I took a look at whoosh-reloaded, and while it looks great it's unfortunately too broad for me to properly test and incorporate with the competition already live. I'll need to instead make a custom patch that applies only the single fix @sash2104 noted. My availability is pretty limited this week due to travel but I will try to get to this soon.", "2797236": "Thanks @sohier, is possible to move cpc schema as TEXT instead KEYWORD\n> schema = whoosh.fields.Schema(\n id=whoosh.fields.ID(stored=True),\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n **cpc=whoosh.fields.KEYWORD(stored=False, scorable=True)**\n )\n\n=======>\n> schema = whoosh.fields.Schema(\n id=whoosh.fields.ID(stored=True),\n ti=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n ab=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n clm=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n detd=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False),\n **cpc=whoosh.fields.TEXT(analyzer=custom_analyzer, stored=False, scorable=True)**\n )", "2808635": "Can someone confirm if this is fixed (or at least partially fixed)?", "2828059": "[The patch is live](https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/506428). Sorry for the slow response and thank you all for the valuable discussion!", "2828076": "Thank you @sohier" }, "source": "meta" }