\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4269349%2Fe3fb7fd0773b64a9427524c8c016abc4%2FScreenshot%202024-07-25%20at%209.11.39PM.png?generation=1721913150365151&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4269349%2F3a50c6e77cd7e85f3ee5b8bfbe99c027%2FScreenshot%202024-07-25%20at%209.12.01PM.png?generation=1721913166903252&alt=media\")
2 weeks before the deadline I started having submission scoring errors again after updating my dataset. After extensive checks I was sure that it was due to query time, and I rushed to fix this without reverting my data. Here’s my findings and I hope it will help all those who face this issue to finally resolve the problem or at least get submissions with not too degraded results.
\nMy queries are all disjunctions of bigrams/trigrams from title, abstract, claims and description fields (the approach is detailed here). Parsing is fast and there are no illegal characters (I strip special characters in preprocessing).
\nIn Whoosh, I believe all phrase terms are actually converted to adjacent queries under the hood. Majority of the slowness I think comes from querying phrases with very high frequency individual words in very large fields (description!), as there are a very large number of positions to iterate through. Not all phrases with high frequency words are slow though (in fact the majority are not).
\nWhoosh doesn’t have a query profiler like Elasticsearch, and I didn’t investigate the actual query execution so I don’t have an answer on how to check whether a query is slow or not without running it. Phrases in title, abstract, claims are safe regardless of word frequency. I think Whoosh also has some caching for example a field cache that can distort the raw query time of a term (in general all search engines have multiple caching layers). Overall I think if you’re using expensive operators (positional, wildcard) on dense fields you may need to drop them.
I found that my slow phrases are consistently slow across different sampled indexes of size 2500, which gave me confidence on dropping bad phrases just based on single query execution. If I had a lot of time and resources I could test the query time of each phrase, but in the end I settled for only checking phrases that only consist of the top 250 words from my phrase vocabulary. This was sufficient to cut about a second off my mean query time, a second off the standard deviation, and bring 97%-99% of my test queries to below 8s.
\nThe following information have been mentioned in the competition overview:
\n\n\nThe metric notebook must finish running your queries in 60 minutes, not including the time required for loading the whoosh index. Note that the metric uses four Whoosh searchers in parallel.
\n
The environment is unknown and it’s hard to evaluate the performance of Whoosh parallel search, but I believe the limitations are closer to 60 mins for all sequential rather than 4x60. I’m also not sure if it's a shared environment but I suspect so as my running times were slightly longer near to the competition deadline.
\nIn practice I found these were some guideline numbers that were ‘safe’ and ‘not safe’ for me, for various validation indexes with 2500 targets:
\nsafe: mean query time of 2.1s, std dev of 1.4s, max of 10s, total time of 87 mins
\nunsafe: mean query time of 3.5s, std dev of 2.7s, max of 20s, total time of 145 mins
In whoosh_utils execute_query() we can implement a searcher with a TimeLimitCollector. This forces the searcher to raise an exception and return once the given time limit is reached.
\n () -> :\n (query) > :\n ValueError()\n query:\n ValueError()\n\n tc = TopCollector(limit=results_limit)\n tec = TermsCollector(tc)\n tlc = TimeLimitCollector(tec, timelimit=time_limit)\n\n to_search = qp.parse(query)\n results = (, )\n :\n searcher.search_with_collector(to_search, tlc)\n results = ([x[] x tlc.results()], [k k tec.termdocs.keys()])\n results\n TimeLimit:\n results = ([x[] x tlc.results()], [k k tec.termdocs.keys()])\n TimeLimit(, results)\n Exception e:\n e\nNote the TermsCollector. This enables us to get partial results even if time limit is reached. With these partial results we can cross reference the original query to find the ‘safe’ words.
\nNote however the partial results are given as unigrams, which makes it hard to correlate exactly which phrases are safe. Here I just give a naive approach of checking if all the phrase words are contained within the partial results. Also, if the validation index does not contain that word, it gives a false negative. But at least in this way, we can have some partial query instead of dropping the query completely when time limit is reached (or using some random heuristic)
\nWith the combination of dropping individual phrases with bad query time and implementing time limit with partial results, I was able to climb back slowly from the scoring error black hole to a stable state.
\n
\n
2 weeks before the deadline I started having submission scoring errors again after updating my dataset. After extensive checks I was sure that it was due to query time, and I rushed to fix this without reverting my data. Here’s my findings and I hope it will help all those who face this issue to finally resolve the problem or at least get submissions with not too degraded results.
\nMy queries are all disjunctions of bigrams/trigrams from title, abstract, claims and description fields (the approach is detailed here). Parsing is fast and there are no illegal characters (I strip special characters in preprocessing).
\nIn Whoosh, I believe all phrase terms are actually converted to adjacent queries under the hood. Majority of the slowness I think comes from querying phrases with very high frequency individual words in very large fields (description!), as there are a very large number of positions to iterate through. Not all phrases with high frequency words are slow though (in fact the majority are not).
\nWhoosh doesn’t have a query profiler like Elasticsearch, and I didn’t investigate the actual query execution so I don’t have an answer on how to check whether a query is slow or not without running it. Phrases in title, abstract, claims are safe regardless of word frequency. I think Whoosh also has some caching for example a field cache that can distort the raw query time of a term (in general all search engines have multiple caching layers). Overall I think if you’re using expensive operators (positional, wildcard) on dense fields you may need to drop them.
I found that my slow phrases are consistently slow across different sampled indexes of size 2500, which gave me confidence on dropping bad phrases just based on single query execution. If I had a lot of time and resources I could test the query time of each phrase, but in the end I settled for only checking phrases that only consist of the top 250 words from my phrase vocabulary. This was sufficient to cut about a second off my mean query time, a second off the standard deviation, and bring 97%-99% of my test queries to below 8s.
\nThe following information have been mentioned in the competition overview:
\n\n\nThe metric notebook must finish running your queries in 60 minutes, not including the time required for loading the whoosh index. Note that the metric uses four Whoosh searchers in parallel.
\n
The environment is unknown and it’s hard to evaluate the performance of Whoosh parallel search, but I believe the limitations are closer to 60 mins for all sequential rather than 4x60. I’m also not sure if it's a shared environment but I suspect so as my running times were slightly longer near to the competition deadline.
\nIn practice I found these were some guideline numbers that were ‘safe’ and ‘not safe’ for me, for various validation indexes with 2500 targets:
\nsafe: mean query time of 2.1s, std dev of 1.4s, max of 10s, total time of 87 mins
\nunsafe: mean query time of 3.5s, std dev of 2.7s, max of 20s, total time of 145 mins
In whoosh_utils execute_query() we can implement a searcher with a TimeLimitCollector. This forces the searcher to raise an exception and return once the given time limit is reached.
\n () -> :\n (query) > :\n ValueError()\n query:\n ValueError()\n\n tc = TopCollector(limit=results_limit)\n tec = TermsCollector(tc)\n tlc = TimeLimitCollector(tec, timelimit=time_limit)\n\n to_search = qp.parse(query)\n results = (, )\n :\n searcher.search_with_collector(to_search, tlc)\n results = ([x[] x tlc.results()], [k k tec.termdocs.keys()])\n results\n TimeLimit:\n results = ([x[] x tlc.results()], [k k tec.termdocs.keys()])\n TimeLimit(, results)\n Exception e:\n e\nNote the TermsCollector. This enables us to get partial results even if time limit is reached. With these partial results we can cross reference the original query to find the ‘safe’ words.
\nNote however the partial results are given as unigrams, which makes it hard to correlate exactly which phrases are safe. Here I just give a naive approach of checking if all the phrase words are contained within the partial results. Also, if the validation index does not contain that word, it gives a false negative. But at least in this way, we can have some partial query instead of dropping the query completely when time limit is reached (or using some random heuristic)
\nWith the combination of dropping individual phrases with bad query time and implementing time limit with partial results, I was able to climb back slowly from the scoring error black hole to a stable state.
\n
\n