Our final approach employed a preprocessing and filtering pipeline designed to minimize overfitting. Initially, we utilized a complex method involving CPC code searches with linear programming, multiple algorithms including decision trees, and logical chains (NOT, OR). Ultimately, we simplified this to a straightforward OR-chain based on single word matches. We created an SQLite database containing filtered information for all publications, implemented in this notebook using only a single OR-chain to achieve a 0.72+ score (our best submission was only 0.66, this notebook just changed the target score of our approach from precision to mAP, we explain the confusion below).
\nSQL Database Creation:
\nwhoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter(), followed by removing duplicates to save memory and because single word search was initially prioritized.Neighbors at Inference:
\nGeneration of Feature DataFrame:
\nOR-Chain Generation:
\nA significant oversight was the misinterpretation of the scoring metric. Initially, we assumed the metric could be treated as precision since false matches filled up to 50 without considering order. Later, we realized that tf-idf and ordering were used but did not reevaluate our precision-based approach. This led to opting for a 35/15 split instead of a 25/0 split, resulting in lower scores (0.7 vs 0.84).
\nPotential improvements included:
\nWe extend our gratitude to the competition organizers, participants who shared their approaches and ideas, and congratulations to the winners!
", "messages": [ { "id": 2936167, "postDate": "2024-07-25T21:43:30.077Z", "content": "Our final approach employed a preprocessing and filtering pipeline designed to minimize overfitting. Initially, we utilized a complex method involving CPC code searches with linear programming, multiple algorithms including decision trees, and logical chains (NOT, OR). Ultimately, we simplified this to a straightforward OR-chain based on single word matches. We created an SQLite database containing filtered information for all publications, implemented in this notebook using only a single OR-chain to achieve a 0.72+ score (our best submission was only 0.66, this notebook just changed the target score of our approach from precision to mAP, we explain the confusion below).
\nSQL Database Creation:
\nwhoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter(), followed by removing duplicates to save memory and because single word search was initially prioritized.Neighbors at Inference:
\nGeneration of Feature DataFrame:
\nOR-Chain Generation:
\nA significant oversight was the misinterpretation of the scoring metric. Initially, we assumed the metric could be treated as precision since false matches filled up to 50 without considering order. Later, we realized that tf-idf and ordering were used but did not reevaluate our precision-based approach. This led to opting for a 35/15 split instead of a 25/0 split, resulting in lower scores (0.7 vs 0.84).
\nPotential improvements included:
\nWe extend our gratitude to the competition organizers, participants who shared their approaches and ideas, and congratulations to the winners!
", "rawMarkdown": "# Overview of the Approach\n\nOur final approach employed a preprocessing and filtering pipeline designed to minimize overfitting. Initially, we utilized a complex method involving CPC code searches with linear programming, multiple algorithms including decision trees, and logical chains (NOT, OR). Ultimately, we simplified this to a straightforward OR-chain based on single word matches. We created an [SQLite database](https://www.kaggle.com/datasets/raki21/uspto-full-error-corrected) containing filtered information for all publications, implemented in this [notebook using only a single OR-chain](https://www.kaggle.com/code/raki21/simple-or-chain-sqlite-magic-free) to achieve a 0.72+ score (our best submission was only 0.66, this notebook just changed the target score of our approach from precision to mAP, we explain the confusion below).\n\n### Details\n\n0. **SQL Database Creation**:\n - Data was split by words to create an SQL database for inference.\n - Each column was processed using `whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()`, followed by removing duplicates to save memory and because single word search was initially prioritized.\n\n1. **Neighbors at Inference**:\n - We loaded the 125k main publications and their neighbors into a dictionary for O(1) retrieval during inference. Additional neighbors-neighbors contributed to a word occurrence dictionary but were not used directly.\n\n2. **Generation of Feature DataFrame**:\n - We identified all words occurring in any neighbor and added them to the feature list. A dataframe was created for each patent with all features as columns and 50 rows indicating occurrence in neighbors, resulting in approximately 10,000 unique words.\n\n3. **OR-Chain Generation**:\n - Features were iteratively selected based on a function penalizing false positives and rewarding true matches. Parameters for penalizing false positives were optimized to select the best scoring OR-chain, focusing on precision rather than mean Average Precision (mAP).\n\n### Problem: Metric Misinterpretation\nA significant oversight was the misinterpretation of the scoring metric. Initially, we assumed the metric could be treated as precision since false matches filled up to 50 without considering order. Later, we realized that tf-idf and ordering were used but did not reevaluate our precision-based approach. This led to opting for a 35/15 split instead of a 25/0 split, resulting in lower scores (0.7 vs 0.84).\n\n### Missing Additions\nPotential improvements included:\n- Faster inference with Ray/GPU to utilize more neighbors.\n- Usage of AND for multiple OR-chains.\n- A semantically independent NOT-OR chain, leveraging a global word count dictionary to omit highly frequent words missing from all positive cases. This would also need some way to identify omissions that have lots of internal overlap, maybe some semantic similarity measure, as often you would have top 3 omissions being something like [material, materials, metal].\n\n### Final Remarks\nWe extend our gratitude to the competition organizers, participants who shared their approaches and ideas, and congratulations to the winners!\n", "votes": 9 }, { "id": 2936361, "postDate": "2024-07-26T05:01:07.700Z", "content": "Thank you for sharing your detailed approach and insights; it's fascinating to see how you tackled the challenge and addressed metric misinterpretation—congratulations on your innovative solution! @raki21
", "rawMarkdown": "Thank you for sharing your detailed approach and insights; it's fascinating to see how you tackled the challenge and addressed metric misinterpretation—congratulations on your innovative solution! @raki21 \n\n", "votes": 1 }, { "id": 2936286, "postDate": "2024-07-26T03:32:52.440Z", "content": "Congrats on your achievement as a non-magician, @raki21, and thanks for sharing all the details. Can you elaborate on what you mean by \"opting for a 35/15 split instead of a 25/0 split\"?
", "rawMarkdown": "Congrats on your achievement as a non-magician, @raki21, and thanks for sharing all the details. Can you elaborate on what you mean by \"opting for a 35/15 split instead of a 25/0 split\"?", "votes": 1, "replies": [ { "id": 2936560, "postDate": "2024-07-26T09:08:40.983Z", "content": "Basically we optimized in a way that would choose a query yielding 35 true positives (part of the 50 real neighbors) and 15 false positives (mismatched patents) as better then the 25 true positive, 0 false positive match case, as the 50 matches are filled up anyway leading to 25 true positives and 25 false positives. Because there is an ordering though and the fill-up of mismatches happens to the end of the prediction list, the 25/0 case is better. If ordering was random mAP ~ precision and 35/15 vs 25/0 -> 25/25 gets 0.7 vs 0.5 score. With actual ordering we get 0.7 vs 0.84 though, making the query avoiding false positives much stronger.
", "rawMarkdown": "Basically we optimized in a way that would choose a query yielding 35 true positives (part of the 50 real neighbors) and 15 false positives (mismatched patents) as better then the 25 true positive, 0 false positive match case, as the 50 matches are filled up anyway leading to 25 true positives and 25 false positives. Because there is an ordering though and the fill-up of mismatches happens to the end of the prediction list, the 25/0 case is better. If ordering was random mAP ~ precision and 35/15 vs 25/0 -> 25/25 gets 0.7 vs 0.5 score. With actual ordering we get 0.7 vs 0.84 though, making the query avoiding false positives much stronger. ", "votes": 1, "replies": [ { "id": 2936657, "postDate": "2024-07-26T11:16:40.897Z", "content": "@raki21 I tried the 30/0 approach, but it didn't work for me, possibly due to some missing logic. Thanks for your explanation!
", "rawMarkdown": "@raki21 I tried the 30/0 approach, but it didn't work for me, possibly due to some missing logic. Thanks for your explanation!" } ] } ] }, { "id": 2936285, "postDate": "2024-07-26T03:31:14.863Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2946950, "postDate": "2024-08-04T19:54:43.823Z", "content": "Thanks for suggesting to do this :)
", "rawMarkdown": "Thanks for suggesting to do this :)" } ], "comments": [ { "id": 2936361, "author_name": "Aadit Shukla", "author_url": "", "post_date": "2024-07-26T05:01:07.700000", "content": "Thank you for sharing your detailed approach and insights; it's fascinating to see how you tackled the challenge and addressed metric misinterpretation—congratulations on your innovative solution! @raki21
", "votes": 1, "replies": [] }, { "id": 2936286, "author_name": "FullEmpty", "author_url": "", "post_date": "2024-07-26T03:32:52.440000", "content": "Congrats on your achievement as a non-magician, @raki21, and thanks for sharing all the details. Can you elaborate on what you mean by \"opting for a 35/15 split instead of a 25/0 split\"?
", "votes": 1, "replies": [ { "id": 2936560, "author_name": "Raki", "author_url": "", "post_date": "2024-07-26T09:08:40.983000", "content": "Basically we optimized in a way that would choose a query yielding 35 true positives (part of the 50 real neighbors) and 15 false positives (mismatched patents) as better then the 25 true positive, 0 false positive match case, as the 50 matches are filled up anyway leading to 25 true positives and 25 false positives. Because there is an ordering though and the fill-up of mismatches happens to the end of the prediction list, the 25/0 case is better. If ordering was random mAP ~ precision and 35/15 vs 25/0 -> 25/25 gets 0.7 vs 0.5 score. With actual ordering we get 0.7 vs 0.84 though, making the query avoiding false positives much stronger.
", "votes": 1, "replies": [ { "id": 2936657, "author_name": "FullEmpty", "author_url": "", "post_date": "2024-07-26T11:16:40.897000", "content": "@raki21 I tried the 30/0 approach, but it didn't work for me, possibly due to some missing logic. Thanks for your explanation!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2936285, "author_name": "", "author_url": "", "post_date": "2024-07-26T03:31:14.863000", "content": "", "votes": 0, "replies": [] }, { "id": 2946950, "author_name": "Lukas Bogenrieder", "author_url": "", "post_date": "2024-08-04T19:54:43.823000", "content": "Thanks for suggesting to do this :)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2936167": "# Overview of the Approach\n\nOur final approach employed a preprocessing and filtering pipeline designed to minimize overfitting. Initially, we utilized a complex method involving CPC code searches with linear programming, multiple algorithms including decision trees, and logical chains (NOT, OR). Ultimately, we simplified this to a straightforward OR-chain based on single word matches. We created an [SQLite database](https://www.kaggle.com/datasets/raki21/uspto-full-error-corrected) containing filtered information for all publications, implemented in this [notebook using only a single OR-chain](https://www.kaggle.com/code/raki21/simple-or-chain-sqlite-magic-free) to achieve a 0.72+ score (our best submission was only 0.66, this notebook just changed the target score of our approach from precision to mAP, we explain the confusion below).\n\n### Details\n\n0. **SQL Database Creation**:\n - Data was split by words to create an SQL database for inference.\n - Each column was processed using `whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()`, followed by removing duplicates to save memory and because single word search was initially prioritized.\n\n1. **Neighbors at Inference**:\n - We loaded the 125k main publications and their neighbors into a dictionary for O(1) retrieval during inference. Additional neighbors-neighbors contributed to a word occurrence dictionary but were not used directly.\n\n2. **Generation of Feature DataFrame**:\n - We identified all words occurring in any neighbor and added them to the feature list. A dataframe was created for each patent with all features as columns and 50 rows indicating occurrence in neighbors, resulting in approximately 10,000 unique words.\n\n3. **OR-Chain Generation**:\n - Features were iteratively selected based on a function penalizing false positives and rewarding true matches. Parameters for penalizing false positives were optimized to select the best scoring OR-chain, focusing on precision rather than mean Average Precision (mAP).\n\n### Problem: Metric Misinterpretation\nA significant oversight was the misinterpretation of the scoring metric. Initially, we assumed the metric could be treated as precision since false matches filled up to 50 without considering order. Later, we realized that tf-idf and ordering were used but did not reevaluate our precision-based approach. This led to opting for a 35/15 split instead of a 25/0 split, resulting in lower scores (0.7 vs 0.84).\n\n### Missing Additions\nPotential improvements included:\n- Faster inference with Ray/GPU to utilize more neighbors.\n- Usage of AND for multiple OR-chains.\n- A semantically independent NOT-OR chain, leveraging a global word count dictionary to omit highly frequent words missing from all positive cases. This would also need some way to identify omissions that have lots of internal overlap, maybe some semantic similarity measure, as often you would have top 3 omissions being something like [material, materials, metal].\n\n### Final Remarks\nWe extend our gratitude to the competition organizers, participants who shared their approaches and ideas, and congratulations to the winners!\n", "2936361": "Thank you for sharing your detailed approach and insights; it's fascinating to see how you tackled the challenge and addressed metric misinterpretation—congratulations on your innovative solution! @raki21 \n\n", "2936286": "Congrats on your achievement as a non-magician, @raki21, and thanks for sharing all the details. Can you elaborate on what you mean by \"opting for a 35/15 split instead of a 25/0 split\"?", "2936285": "", "2946950": "Thanks for suggesting to do this :)" } }