The current design of train_index does not allow for validation of queries. The issue is that not enough neighbors of any given publication_number are included in the index. The maximum is 10 as shown in this notebook.
As described in the linked notebook this could be fixed by choosing ~4k publication numbers and including all their neighbors in the train_index. This way the index will remain the same size but we will be able much more accurately validate our queries.
Please fix the submission error first, but afterwords it would be greatly appreciated if the train_index is updated to something that makes validation of queries feasible. Thanks! @sohier
", "messages": [ { "id": "2776880", "postDate": "04/26/2024 11:43:27", "content": "The current design of train_index does not allow for validation of queries. The issue is that not enough neighbors of any given publication_number are included in the index. The maximum is 10 as shown in this notebook.
As described in the linked notebook this could be fixed by choosing ~4k publication numbers and including all their neighbors in the train_index. This way the index will remain the same size but we will be able much more accurately validate our queries.
Please fix the submission error first, but afterwords it would be greatly appreciated if the train_index is updated to something that makes validation of queries feasible. Thanks! @sohier
", "rawMarkdown": "The current design of `train_index` does not allow for validation of queries. The issue is that not enough neighbors of any given `publication_number` are included in the index. The maximum is 10 as shown in [this notebook](https://www.kaggle.com/code/devinanzelmo/uspto-train-index-problem). \n\nAs described in the linked notebook this could be fixed by choosing ~4k publication numbers and including all their neighbors in the `train_index`. This way the index will remain the same size but we will be able much more accurately validate our queries. \n\nPlease fix the submission error first, but afterwords it would be greatly appreciated if the train_index is updated to something that makes validation of queries feasible. Thanks! @sohier", "votes": null }, { "id": "2777576", "postDate": "04/26/2024 18:02:24", "content": "You are welcome to create your own validation index. See the create_index function in the whoosh utils script.
Thanks I didn't realize there was a utility to do this and got kind of overwhelmed by the whoosh docs.
", "rawMarkdown": "Thanks I didn't realize there was a utility to do this and got kind of overwhelmed by the whoosh docs.", "votes": null }, { "id": "2777744", "postDate": "04/26/2024 19:35:05", "content": "No problem! My hope is that you can't mostly avoid having to deal with the whoosh docs by taking advantage of whoosh_utils.
I created a better validation index which can be found here
", "rawMarkdown": "I created a better validation index which can be found [here](https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index/)", "votes": null }, { "id": "2779447", "postDate": "04/27/2024 16:54:08", "content": "Thanks for sharing @devinanzelmo, is it correlate to LB ?
", "rawMarkdown": "Thanks for sharing @devinanzelmo, is it correlate to LB ?", "votes": null }, { "id": "2779468", "postDate": "04/27/2024 16:59:45", "content": "I haven't got that far just started exploring it. Definitely much more informative for testing queries. Still a bit of work to get to a local validation score.
", "rawMarkdown": "I haven't got that far just started exploring it. Definitely much more informative for testing queries. Still a bit of work to get to a local validation score.", "votes": null }, { "id": "2779564", "postDate": "04/27/2024 17:45:52", "content": "@devinanzelmo your validation help to fix my notebook submission issue.
", "rawMarkdown": "devinanzelmo your validation help to fix my notebook submission issue.", "votes": null }, { "id": "2779581", "postDate": "04/27/2024 17:55:10", "content": "If your having issues with submission I recommend the QueryValidator and count_query_tokens from whoosh_utils I wouldn't have been able to make a submission without them.
its logical mistake, sure i will use QueryValidator and count_query_tokens
", "rawMarkdown": "its logical mistake, sure i will use QueryValidator and count_query_tokens", "votes": null } ], "comments": [ { "id": 2777576, "author_name": "sohier", "author_url": "", "post_date": "04/26/2024 18:02:24", "content": "You are welcome to create your own validation index. See the create_index function in the whoosh utils script.
Thanks I didn't realize there was a utility to do this and got kind of overwhelmed by the whoosh docs.
", "votes": null, "replies": [ { "id": 2777744, "author_name": "sohier", "author_url": "", "post_date": "04/26/2024 19:35:05", "content": "No problem! My hope is that you can't mostly avoid having to deal with the whoosh docs by taking advantage of whoosh_utils.
I created a better validation index which can be found here
", "votes": null, "replies": [ { "id": 2779447, "author_name": "seshurajup", "author_url": "", "post_date": "04/27/2024 16:54:08", "content": "Thanks for sharing @devinanzelmo, is it correlate to LB ?
", "votes": null, "replies": [] }, { "id": 2779468, "author_name": "devinanzelmo", "author_url": "", "post_date": "04/27/2024 16:59:45", "content": "I haven't got that far just started exploring it. Definitely much more informative for testing queries. Still a bit of work to get to a local validation score.
", "votes": null, "replies": [ { "id": 2779564, "author_name": "seshurajup", "author_url": "", "post_date": "04/27/2024 17:45:52", "content": "@devinanzelmo your validation help to fix my notebook submission issue.
", "votes": null, "replies": [] }, { "id": 2779581, "author_name": "devinanzelmo", "author_url": "", "post_date": "04/27/2024 17:55:10", "content": "If your having issues with submission I recommend the QueryValidator and count_query_tokens from whoosh_utils I wouldn't have been able to make a submission without them.
its logical mistake, sure i will use QueryValidator and count_query_tokens
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2776880": "The current design of `train_index` does not allow for validation of queries. The issue is that not enough neighbors of any given `publication_number` are included in the index. The maximum is 10 as shown in [this notebook](https://www.kaggle.com/code/devinanzelmo/uspto-train-index-problem). \n\nAs described in the linked notebook this could be fixed by choosing ~4k publication numbers and including all their neighbors in the `train_index`. This way the index will remain the same size but we will be able much more accurately validate our queries. \n\nPlease fix the submission error first, but afterwords it would be greatly appreciated if the train_index is updated to something that makes validation of queries feasible. Thanks! @sohier", "2777576": "You are welcome to create your own validation index. See the `create_index` function in [the whoosh utils script](https://www.kaggle.com/code/metric/whoosh-utils).", "2777593": "Thanks I didn't realize there was a utility to do this and got kind of overwhelmed by the whoosh docs.", "2777744": "No problem! My hope is that you can't mostly avoid having to deal with the whoosh docs by taking advantage of `whoosh_utils`.", "2779402": "I created a better validation index which can be found [here](https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index/)", "2779447": "Thanks for sharing @devinanzelmo, is it correlate to LB ?", "2779468": "I haven't got that far just started exploring it. Definitely much more informative for testing queries. Still a bit of work to get to a local validation score.", "2779564": "devinanzelmo your validation help to fix my notebook submission issue.", "2779581": "If your having issues with submission I recommend the `QueryValidator` and `count_query_tokens` from `whoosh_utils` I wouldn't have been able to make a submission without them.", "2779583": "its logical mistake, sure i will use QueryValidator and count_query_tokens" }, "source": "meta" }