EDIT: I tried a wider variety of queries and I was still able to get a freeze even after following the work around below. It worked for me but might not for you.
\nBy applying the custom_analyzer from whoosh_utils to text and only using its output for searches I have been able to avoid Whoosh freezing and make successful submissions. This has worked ti:, ab:, clm:, and detd: fields but not for the cpc: field (which has no preprocessing).
Use the following code for set up.
\n\n re\n whoosh\n\nNUMBER_REGEX = re.()\n\n (whoosh.analysis.Filter):\n ():\n t tokens:\n NUMBER_REGEX.(t.text):\n t\n\nBRS_STOPWORDS = [, , , , , , , , , , , ,\n , , , , , , , , , , ]\ncustom_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\nThen to use it
\n = \n = [token.text for token in custom_analyzer(some_text)]\nThen use the contents of tokens as input to your searches. After doing this my local searches where sped up and there were no more freezes.
Again this doesn't work for CPC_CODEs to fix that the competition admins will have update Whoosh.
", "messages": [ { "id": "2788631", "postDate": "05/02/2024 10:16:41", "content": "EDIT: I tried a wider variety of queries and I was still able to get a freeze even after following the work around below. It worked for me but might not for you.
\nBy applying the custom_analyzer from whoosh_utils to text and only using its output for searches I have been able to avoid Whoosh freezing and make successful submissions. This has worked ti:, ab:, clm:, and detd: fields but not for the cpc: field (which has no preprocessing).
Use the following code for set up.
\n\n re\n whoosh\n\nNUMBER_REGEX = re.()\n\n (whoosh.analysis.Filter):\n ():\n t tokens:\n NUMBER_REGEX.(t.text):\n t\n\nBRS_STOPWORDS = [, , , , , , , , , , , ,\n , , , , , , , , , , ]\ncustom_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\nThen to use it
\n = \n = [token.text for token in custom_analyzer(some_text)]\nThen use the contents of tokens as input to your searches. After doing this my local searches where sped up and there were no more freezes.
Again this doesn't work for CPC_CODEs to fix that the competition admins will have update Whoosh.
", "rawMarkdown": "EDIT: I tried a wider variety of queries and I was still able to get a freeze even after following the work around below. It worked for me but might not for you.\n\nBy applying the `custom_analyzer` from `whoosh_utils` to text and only using its output for searches I have been able to avoid Whoosh freezing and make successful submissions. This has worked `ti:`, `ab:`, `clm:`, and `detd:` fields but not for the `cpc:` field (which has no preprocessing). \n\nUse the following code for set up. \n\n```\n# thanks to https://www.kaggle.com/code/seshurajup/uspto-whoosh-utils-eda for pointing this out\nimport re\nimport whoosh\n\nNUMBER_REGEX = re.compile(r'^(\\d+|\\d{1,3}(,\\d{3})*)(\\.\\d+)?$')\n\nclass NumberFilter(whoosh.analysis.Filter):\n def __call__(self, tokens):\n for t in tokens:\n if not NUMBER_REGEX.match(t.text):\n yield t\n \nBRS_STOPWORDS = ['an', 'are', 'by', 'for', 'if', 'into', 'is', 'no', 'not', 'of', 'on', 'such',\n 'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'to', 'was', 'will']\ncustom_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\n```\n\nThen to use it\n\n\n```\nsome_text = \"Some text you want to query part of\"\ntokens = [token.text for token in custom_analyzer(some_text)]\n```\n\nThen use the contents of `tokens` as input to your searches. After doing this my local searches where sped up and there were no more freezes. \n\nAgain this doesn't work for CPC_CODEs to fix that the competition admins will have update Whoosh.", "votes": null }, { "id": "2788724", "postDate": "05/02/2024 11:39:53", "content": "This does not fix the existing problem for submissions but only for us as developers, right?
", "rawMarkdown": "This does not fix the existing problem for submissions but only for us as developers, right?", "votes": null }, { "id": "2788727", "postDate": "05/02/2024 11:43:01", "content": "Works for submissions since we can do before generating our queries which then get passed to the metric. I am not sure whats going on but doing this gives a 3-4x speed up in searches locally, and I haven't observed any freezing.
", "rawMarkdown": "Works for submissions since we can do before generating our queries which then get passed to the metric. I am not sure whats going on but doing this gives a 3-4x speed up in searches locally, and I haven't observed any freezing.", "votes": null }, { "id": "2788946", "postDate": "05/02/2024 13:40:56", "content": "Great, trying it now too - Looks like the data maybe contains some special characters that are not parseable?
", "rawMarkdown": "Great, trying it now too - Looks like the data maybe contains some special characters that are not parseable?", "votes": null }, { "id": "2789414", "postDate": "05/02/2024 16:57:03", "content": "Did not fix it for me, still getting the same submission error.
", "rawMarkdown": "Did not fix it for me, still getting the same submission error.", "votes": null }, { "id": "2789444", "postDate": "05/02/2024 17:09:34", "content": "@devinanzelmo Thanks for my eda helped it. for CPC is not using StandardAnalyzer.
\n\n", "rawMarkdown": "devinanzelmo Thanks for my eda helped it. for CPC is not using StandardAnalyzer. \n\n> is OR query working? * and ? tested", "votes": null }, { "id": "2789452", "postDate": "05/02/2024 17:15:42", "content": "is OR query working? * and ? tested
\n
Yeah the OR query is working. I am only using OR and AND so far and have not had any errors for last 10 submissions.
Thanks for sharing @devinanzelmo, i will also try.
", "rawMarkdown": "Thanks for sharing @devinanzelmo, i will also try.", "votes": null }, { "id": "2789565", "postDate": "05/02/2024 18:05:10", "content": "What I did when getting submission errors was run 100k rows from nearest neighbor through my solution and check all the queries with QueryValidator and count_query_tokens. If no errors showed up I would run 1k (or however many you have patience for) queries through the search, after each search I would print something so I could see how long each search took. I was able to observe all errors, and see the search freeze doing this.
", "rawMarkdown": "What I did when getting submission errors was run 100k rows from nearest neighbor through my solution and check all the queries with QueryValidator and count_query_tokens. If no errors showed up I would run 1k (or however many you have patience for) queries through the search, after each search I would print something so I could see how long each search took. I was able to observe all errors, and see the search freeze doing this.", "votes": null }, { "id": "2789638", "postDate": "05/02/2024 18:51:20", "content": "@valentinwerner I would also time your queries see how long they take. I just ran some searches with queries similar to yours and they were taking a long time(over 3 hours for a submission). Also I noticed a freeze which I hadn't noticed with queries generated by my solution. So the type of query matters also I guess.
", "rawMarkdown": "valentinwerner I would also time your queries see how long they take. I just ran some searches with queries similar to yours and they were taking a long time(over 3 hours for a submission). Also I noticed a freeze which I hadn't noticed with queries generated by my solution. So the type of query matters also I guess.", "votes": null }, { "id": "2790941", "postDate": "05/03/2024 11:50:42", "content": "@devinanzelmo is OR query working fine with 50 tokens with operator?
", "rawMarkdown": "devinanzelmo is OR query working fine with 50 tokens with operator?", "votes": null } ], "comments": [ { "id": 2788724, "author_name": "valentinwerner", "author_url": "", "post_date": "05/02/2024 11:39:53", "content": "This does not fix the existing problem for submissions but only for us as developers, right?
", "votes": null, "replies": [ { "id": 2788727, "author_name": "devinanzelmo", "author_url": "", "post_date": "05/02/2024 11:43:01", "content": "Works for submissions since we can do before generating our queries which then get passed to the metric. I am not sure whats going on but doing this gives a 3-4x speed up in searches locally, and I haven't observed any freezing.
", "votes": null, "replies": [ { "id": 2788946, "author_name": "valentinwerner", "author_url": "", "post_date": "05/02/2024 13:40:56", "content": "Great, trying it now too - Looks like the data maybe contains some special characters that are not parseable?
", "votes": null, "replies": [ { "id": 2789414, "author_name": "valentinwerner", "author_url": "", "post_date": "05/02/2024 16:57:03", "content": "Did not fix it for me, still getting the same submission error.
", "votes": null, "replies": [] } ] }, { "id": 2789565, "author_name": "devinanzelmo", "author_url": "", "post_date": "05/02/2024 18:05:10", "content": "What I did when getting submission errors was run 100k rows from nearest neighbor through my solution and check all the queries with QueryValidator and count_query_tokens. If no errors showed up I would run 1k (or however many you have patience for) queries through the search, after each search I would print something so I could see how long each search took. I was able to observe all errors, and see the search freeze doing this.
", "votes": null, "replies": [] }, { "id": 2789638, "author_name": "devinanzelmo", "author_url": "", "post_date": "05/02/2024 18:51:20", "content": "@valentinwerner I would also time your queries see how long they take. I just ran some searches with queries similar to yours and they were taking a long time(over 3 hours for a submission). Also I noticed a freeze which I hadn't noticed with queries generated by my solution. So the type of query matters also I guess.
", "votes": null, "replies": [] } ] } ] }, { "id": 2789444, "author_name": "seshurajup", "author_url": "", "post_date": "05/02/2024 17:09:34", "content": "@devinanzelmo Thanks for my eda helped it. for CPC is not using StandardAnalyzer.
\n\n", "votes": null, "replies": [ { "id": 2789452, "author_name": "devinanzelmo", "author_url": "", "post_date": "05/02/2024 17:15:42", "content": "is OR query working? * and ? tested
\n
Yeah the OR query is working. I am only using OR and AND so far and have not had any errors for last 10 submissions.
Thanks for sharing @devinanzelmo, i will also try.
", "votes": null, "replies": [] }, { "id": 2790941, "author_name": "seshurajup", "author_url": "", "post_date": "05/03/2024 11:50:42", "content": "@devinanzelmo is OR query working fine with 50 tokens with operator?
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2788631": "EDIT: I tried a wider variety of queries and I was still able to get a freeze even after following the work around below. It worked for me but might not for you.\n\nBy applying the `custom_analyzer` from `whoosh_utils` to text and only using its output for searches I have been able to avoid Whoosh freezing and make successful submissions. This has worked `ti:`, `ab:`, `clm:`, and `detd:` fields but not for the `cpc:` field (which has no preprocessing). \n\nUse the following code for set up. \n\n```\n# thanks to https://www.kaggle.com/code/seshurajup/uspto-whoosh-utils-eda for pointing this out\nimport re\nimport whoosh\n\nNUMBER_REGEX = re.compile(r'^(\\d+|\\d{1,3}(,\\d{3})*)(\\.\\d+)?$')\n\nclass NumberFilter(whoosh.analysis.Filter):\n def __call__(self, tokens):\n for t in tokens:\n if not NUMBER_REGEX.match(t.text):\n yield t\n \nBRS_STOPWORDS = ['an', 'are', 'by', 'for', 'if', 'into', 'is', 'no', 'not', 'of', 'on', 'such',\n 'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'to', 'was', 'will']\ncustom_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\n```\n\nThen to use it\n\n\n```\nsome_text = \"Some text you want to query part of\"\ntokens = [token.text for token in custom_analyzer(some_text)]\n```\n\nThen use the contents of `tokens` as input to your searches. After doing this my local searches where sped up and there were no more freezes. \n\nAgain this doesn't work for CPC_CODEs to fix that the competition admins will have update Whoosh.", "2788724": "This does not fix the existing problem for submissions but only for us as developers, right?", "2788727": "Works for submissions since we can do before generating our queries which then get passed to the metric. I am not sure whats going on but doing this gives a 3-4x speed up in searches locally, and I haven't observed any freezing.", "2788946": "Great, trying it now too - Looks like the data maybe contains some special characters that are not parseable?", "2789414": "Did not fix it for me, still getting the same submission error.", "2789444": "devinanzelmo Thanks for my eda helped it. for CPC is not using StandardAnalyzer. \n\n> is OR query working? * and ? tested", "2789452": "Yeah the `OR` query is working. I am only using `OR` and `AND` so far and have not had any errors for last 10 submissions.", "2789490": "Thanks for sharing @devinanzelmo, i will also try.", "2789565": "What I did when getting submission errors was run 100k rows from nearest neighbor through my solution and check all the queries with QueryValidator and count_query_tokens. If no errors showed up I would run 1k (or however many you have patience for) queries through the search, after each search I would print something so I could see how long each search took. I was able to observe all errors, and see the search freeze doing this.", "2789638": "valentinwerner I would also time your queries see how long they take. I just ran some searches with queries similar to yours and they were taking a long time(over 3 hours for a submission). Also I noticed a freeze which I hadn't noticed with queries generated by my solution. So the type of query matters also I guess.", "2790941": "devinanzelmo is OR query working fine with 50 tokens with operator?" }, "source": "meta" }