As discussed in this thread, certain queries can leave Whoosh stuck in an endless loop. I've updated the Whoosh wheel used by the metric based on recommendations in that thread and confirmed that the most severe crash is no longer an issue.
\nThis patch took me longer to deploy than I had hoped; apologies for the delay. Thanks to everyone who contributed to the discussion and @sash2104 in particular.
\nThe specific changes relative to the git copy of Whoosh are as follows:
\ngit diff\ndiff --git awhooshwhoosh3.py bwhooshwhoosh3.py\nindex a6b982a..c72 \n--- awhooshwhoosh3.py\n+++ bwhooshwhoosh3.py\n@@ -, +, @@ W3LeafMatcher(LeafMatcher):\n\n # the quality of block is already higher than the minimum,\n # nothing\n- block_quality() > minquality:\n+ tolerance = e-\n+ block_quality() > minquality + tolerance:\n \n\n # Skip blocks as as the block quality is not greater than the\n # minimum\n- self._skip_to_block(lambda: block_quality() <= minquality)\n+ self._skip_to_block(lambda: block_quality() <= minquality + tolerance)\n\n block_min_id(self):\n self._ids is None:\ndiff --git awhooshbinary.py bwhooshbinary.py\nindex d2aff26..bd23af4 \n--- awhooshbinary.py\n+++ bwhooshbinary.py\n@@ -, +, @@ DisjunctionMaxMatcher(UnionMatcher):\n skipped = \n aq = a.block_quality()\n bq = b.block_quality()\n- a.is_active() and b.is_active() and max(aq, bq) <= minquality:\n- aq <= minquality:\n+ a.is_active() and b.is_active() and max(aq, bq) < minquality:\n+ aq < minquality:\n skipped += a.skip_to_quality(minquality)\n aq = a.block_quality()\n- bq <= minquality:\n+ bq < minquality:\n skipped += b.skip_to_quality(minquality)\n bq = b.block_quality()\n skipped\nAs discussed in this thread, certain queries can leave Whoosh stuck in an endless loop. I've updated the Whoosh wheel used by the metric based on recommendations in that thread and confirmed that the most severe crash is no longer an issue.
\nThis patch took me longer to deploy than I had hoped; apologies for the delay. Thanks to everyone who contributed to the discussion and @sash2104 in particular.
\nThe specific changes relative to the git copy of Whoosh are as follows:
\ngit diff\ndiff --git awhooshwhoosh3.py bwhooshwhoosh3.py\nindex a6b982a..c72 \n--- awhooshwhoosh3.py\n+++ bwhooshwhoosh3.py\n@@ -, +, @@ W3LeafMatcher(LeafMatcher):\n\n # the quality of block is already higher than the minimum,\n # nothing\n- block_quality() > minquality:\n+ tolerance = e-\n+ block_quality() > minquality + tolerance:\n \n\n # Skip blocks as as the block quality is not greater than the\n # minimum\n- self._skip_to_block(lambda: block_quality() <= minquality)\n+ self._skip_to_block(lambda: block_quality() <= minquality + tolerance)\n\n block_min_id(self):\n self._ids is None:\ndiff --git awhooshbinary.py bwhooshbinary.py\nindex d2aff26..bd23af4 \n--- awhooshbinary.py\n+++ bwhooshbinary.py\n@@ -, +, @@ DisjunctionMaxMatcher(UnionMatcher):\n skipped = \n aq = a.block_quality()\n bq = b.block_quality()\n- a.is_active() and b.is_active() and max(aq, bq) <= minquality:\n- aq <= minquality:\n+ a.is_active() and b.is_active() and max(aq, bq) < minquality:\n+ aq < minquality:\n skipped += a.skip_to_quality(minquality)\n aq = a.block_quality()\n- bq <= minquality:\n+ bq < minquality:\n skipped += b.skip_to_quality(minquality)\n bq = b.block_quality()\n skipped\n@sohier is it possible to re-run our Submission Scoring Error submissions?
", "rawMarkdown": "sohier is it possible to re-run our Submission Scoring Error submissions?", "votes": null }, { "id": "2832811", "postDate": "05/23/2024 21:06:21", "content": "That's a good idea. I've started the bulk rescore process for error submissions only. It'll take until tomorrow to finish, at the earliest.
", "rawMarkdown": "That's a good idea. I've started the bulk rescore process for error submissions only. It'll take until tomorrow to finish, at the earliest.", "votes": null }, { "id": "2832993", "postDate": "05/24/2024 02:52:26", "content": "Thank you @sohier for the update
", "rawMarkdown": "Thank you @sohier for the update", "votes": null }, { "id": "2834425", "postDate": "05/24/2024 18:11:40", "content": "Okay: another ~120 submissions succeeded, 160 still failed, and a handful got stuck in the rescore. I won't be able to make time for clearing out those last few any time soon, unfortunately.
", "rawMarkdown": "Okay: another ~120 submissions succeeded, 160 still failed, and a handful got stuck in the rescore. I won't be able to make time for clearing out those last few any time soon, unfortunately.", "votes": null }, { "id": "2834441", "postDate": "05/24/2024 18:23:22", "content": "Thank you for the update @sohier, is it possible score failures are not consider as submission limit? ( since its different environment scoring is calculated )
", "rawMarkdown": "Thank you for the update @sohier, is it possible score failures are not consider as submission limit? ( since its different environment scoring is calculated )", "votes": null } ], "comments": [ { "id": 2829779, "author_name": "seshurajup", "author_url": "", "post_date": "05/22/2024 19:25:39", "content": "@sohier is it possible to re-run our Submission Scoring Error submissions?
", "votes": null, "replies": [ { "id": 2832811, "author_name": "sohier", "author_url": "", "post_date": "05/23/2024 21:06:21", "content": "That's a good idea. I've started the bulk rescore process for error submissions only. It'll take until tomorrow to finish, at the earliest.
", "votes": null, "replies": [ { "id": 2832993, "author_name": "seshurajup", "author_url": "", "post_date": "05/24/2024 02:52:26", "content": "Thank you @sohier for the update
", "votes": null, "replies": [] }, { "id": 2834425, "author_name": "sohier", "author_url": "", "post_date": "05/24/2024 18:11:40", "content": "Okay: another ~120 submissions succeeded, 160 still failed, and a handful got stuck in the rescore. I won't be able to make time for clearing out those last few any time soon, unfortunately.
", "votes": null, "replies": [ { "id": 2834441, "author_name": "seshurajup", "author_url": "", "post_date": "05/24/2024 18:23:22", "content": "Thank you for the update @sohier, is it possible score failures are not consider as submission limit? ( since its different environment scoring is calculated )
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2828055": "As discussed [in this thread](https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/498283), certain queries can leave Whoosh stuck in an endless loop. I've updated the Whoosh wheel used by the metric based on recommendations in that thread and confirmed that [the most severe crash](https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/498283#2782101) is no longer an issue.\n\nThis patch took me longer to deploy than I had hoped; apologies for the delay. Thanks to everyone who contributed to the discussion and @sash2104 in particular.\n\nThe specific changes relative to [the git copy of Whoosh](https://github.com/mchaput/whoosh) are as follows:\n\n```\ngit diff\ndiff --git a/src/whoosh/codec/whoosh3.py b/src/whoosh/codec/whoosh3.py\nindex a6b982a..3390c72 100644\n--- a/src/whoosh/codec/whoosh3.py\n+++ b/src/whoosh/codec/whoosh3.py\n@@ -1043,12 +1043,13 @@ class W3LeafMatcher(LeafMatcher):\n \n # If the quality of this block is already higher than the minimum,\n # do nothing\n- if block_quality() > minquality:\n+ tolerance = 1e-10\n+ if block_quality() > minquality + tolerance:\n return 0\n \n # Skip blocks as long as the block quality is not greater than the\n # minimum\n- return self._skip_to_block(lambda: block_quality() <= minquality)\n+ return self._skip_to_block(lambda: block_quality() <= minquality + tolerance)\n \n def block_min_id(self):\n if self._ids is None:\ndiff --git a/src/whoosh/matching/binary.py b/src/whoosh/matching/binary.py\nindex d2aff26..bd23af4 100644\n--- a/src/whoosh/matching/binary.py\n+++ b/src/whoosh/matching/binary.py\n@@ -395,11 +395,11 @@ class DisjunctionMaxMatcher(UnionMatcher):\n skipped = 0\n aq = a.block_quality()\n bq = b.block_quality()\n- while a.is_active() and b.is_active() and max(aq, bq) <= minquality:\n- if aq <= minquality:\n+ while a.is_active() and b.is_active() and max(aq, bq) < minquality:\n+ if aq < minquality:\n skipped += a.skip_to_quality(minquality)\n aq = a.block_quality()\n- if bq <= minquality:\n+ if bq < minquality:\n skipped += b.skip_to_quality(minquality)\n bq = b.block_quality()\n return skipped\n```", "2829779": "sohier is it possible to re-run our Submission Scoring Error submissions?", "2832811": "That's a good idea. I've started the bulk rescore process for error submissions only. It'll take until tomorrow to finish, at the earliest.", "2832993": "Thank you @sohier for the update", "2834425": "Okay: another ~120 submissions succeeded, 160 still failed, and a handful got stuck in the rescore. I won't be able to make time for clearing out those last few any time soon, unfortunately.", "2834441": "Thank you for the update @sohier, is it possible score failures are not consider as submission limit? ( since its different environment scoring is calculated )" }, "source": "meta" }