{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59575,"databundleVersionId":8060720,"sourceType":"competition"},{"sourceId":8413600,"sourceType":"datasetVersion","datasetId":5007812},{"sourceId":9005061,"sourceType":"datasetVersion","datasetId":5163044}],"dockerImageVersionId":30716,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Speeding up everything\n\n[RAPIDS cuDF](https://rapids.ai/cudf-pandas/) is now able to accelerate pandas without any code change !\n\nOnly two things are needed:\n- A GPU \n- This one line before importing pandas :","metadata":{}},{"cell_type":"code","source":"%load_ext cudf.pandas","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:03:28.491373Z","iopub.execute_input":"2024-07-24T14:03:28.491757Z","iopub.status.idle":"2024-07-24T14:03:28.497122Z","shell.execute_reply.started":"2024-07-24T14:03:28.491720Z","shell.execute_reply":"2024-07-24T14:03:28.495785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"| Benchmark - GPU T4         | Without cudf-pandas | With cudf-pandas | Speed Up     |\n|----------------------------|---------------------|------------------|--------------|\n| load_titles                |    36s              |   **4s**         |  **9x !**   | \n| build 2500 queries         |    2min30s          |   **30s**        |  **5x !**  | \n\nThe speed-up is huge, the entire code runs in less than a minute on GPU.\n\nGPU acceleration allows to greatly speed-up computations, it plays a key role in our final solution.\n\nWe optimize our queries in a big search space, such computations would've never fit in the 9h runtime limit if done on CPU.","metadata":{}},{"cell_type":"code","source":"!pip install -qqq /kaggle/input/uspto-whoosh-reloaded-2-7-5-patched/Whoosh_Reloaded-2.7.5-py2.py3-none-any.whl\n\nimport re\nimport time\nimport whoosh\nimport numpy as np\nimport pandas as pd\nimport whoosh.analysis\nfrom tqdm.notebook import tqdm","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-07-24T14:03:28.498684Z","iopub.execute_input":"2024-07-24T14:03:28.499094Z","iopub.status.idle":"2024-07-24T14:04:03.102873Z","shell.execute_reply.started":"2024-07-24T14:03:28.499057Z","shell.execute_reply":"2024-07-24T14:04:03.101642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data","metadata":{}},{"cell_type":"code","source":"DATA_PATH = \"/kaggle/input/uspto-explainable-ai/\"","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:04:03.104766Z","iopub.execute_input":"2024-07-24T14:04:03.105092Z","iopub.status.idle":"2024-07-24T14:04:03.110705Z","shell.execute_reply.started":"2024-07-24T14:04:03.105059Z","shell.execute_reply":"2024-07-24T14:04:03.109421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = pd.read_csv(DATA_PATH + \"test.csv\")\n\nif len(df_test) < 100:  # Replace with a 2500 rows dataframe\n    df_test = pd.read_csv(\"/kaggle/input/uspto-patent-metadata/nearest_neighbors.csv\")\n\npatents = df_test.values[:, 1:].flatten()","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:04:03.112371Z","iopub.execute_input":"2024-07-24T14:04:03.112841Z","iopub.status.idle":"2024-07-24T14:04:03.226033Z","shell.execute_reply.started":"2024-07-24T14:04:03.112803Z","shell.execute_reply":"2024-07-24T14:04:03.224902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_titles(patents, data_path=\"../input/\"):\n    df = pd.DataFrame({\"publication_number\": patents})\n\n    df_meta = pd.read_parquet(\n        \"/kaggle/input/uspto-patent-metadata/all_patents.parquet\", columns=[\"publication_number\", \"title\"]\n    )\n    df = df.merge(df_meta, how=\"left\", on=\"publication_number\")\n    df[\"title\"] = df[\"title\"].fillna(\"\")\n    return df","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:04:03.229028Z","iopub.execute_input":"2024-07-24T14:04:03.229357Z","iopub.status.idle":"2024-07-24T14:04:03.235674Z","shell.execute_reply.started":"2024-07-24T14:04:03.229329Z","shell.execute_reply":"2024-07-24T14:04:03.234623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndf = load_titles(patents, data_path=DATA_PATH)","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:04:03.237095Z","iopub.execute_input":"2024-07-24T14:04:03.237486Z","iopub.status.idle":"2024-07-24T14:04:39.815496Z","shell.execute_reply.started":"2024-07-24T14:04:03.237428Z","shell.execute_reply":"2024-07-24T14:04:39.814403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df.head())","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:26.075722Z","iopub.execute_input":"2024-07-24T14:05:26.076479Z","iopub.status.idle":"2024-07-24T14:05:26.092522Z","shell.execute_reply.started":"2024-07-24T14:05:26.076422Z","shell.execute_reply":"2024-07-24T14:05:26.091420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Number of patents:', len(patents))\nprint('Test size:', len(df_test))","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:26.511085Z","iopub.execute_input":"2024-07-24T14:05:26.511444Z","iopub.status.idle":"2024-07-24T14:05:26.517445Z","shell.execute_reply.started":"2024-07-24T14:05:26.511415Z","shell.execute_reply":"2024-07-24T14:05:26.516337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The Magic\n\nThe idea is to query exact titles, by using queries of the form `ti:title_1 OR ti:title_2`.\nThis will return `publication_1` and `publication_2`. \n\nThe issue with this approach is that the number of tokens quickly goes up ! Titles are 7 words long on average.","metadata":{}},{"cell_type":"code","source":"def count_query_tokens(query: str):\n    return len([i for i in re.split('[\\s+()]', query) if i])","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:27.272763Z","iopub.execute_input":"2024-07-24T14:05:27.273156Z","iopub.status.idle":"2024-07-24T14:05:27.280178Z","shell.execute_reply.started":"2024-07-24T14:05:27.273122Z","shell.execute_reply":"2024-07-24T14:05:27.279013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"query = 'ti:\"Autofocusing apparatus of a sighting telescope\" OR ti:\"Auto-focusing apparatus\"'\nprint(f'Number of tokens:', count_query_tokens(query))","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:27.841229Z","iopub.execute_input":"2024-07-24T14:05:27.841607Z","iopub.status.idle":"2024-07-24T14:05:27.847175Z","shell.execute_reply.started":"2024-07-24T14:05:27.841577Z","shell.execute_reply":"2024-07-24T14:05:27.846126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"However, whoosh applies preprocessing on its side to filter texts. See the function below:\n\nThis is applied to both texts and queries.","metadata":{}},{"cell_type":"code","source":"NUMBER_REGEX = re.compile(r'^(\\d+|\\d{1,3}(,\\d{3})*)(\\.\\d+)?$')\n\nclass NumberFilter(whoosh.analysis.Filter):\n    def __call__(self, tokens):\n        for t in tokens:\n            if not NUMBER_REGEX.match(t.text):\n                yield t\n\nBRS_STOPWORDS = ['an', 'are', 'by', 'for', 'if', 'into', 'is', 'no', 'not', 'of', 'on', 'such',\n        'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'to', 'was', 'will']\n\ncustom_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\n\ndef identity(doc):\n    return doc\n\ndef tokenizer(doc):\n    return [token.text for token in custom_analyzer(doc)]","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:28.617923Z","iopub.execute_input":"2024-07-24T14:05:28.618295Z","iopub.status.idle":"2024-07-24T14:05:28.628079Z","shell.execute_reply.started":"2024-07-24T14:05:28.618268Z","shell.execute_reply":"2024-07-24T14:05:28.626728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"t1 = \" \".join(tokenizer(\"Autofocusing apparatus of a sighting telescope\"))\nt2 = \" \".join(tokenizer(\"Auto-focusing apparatus\"))\nquery = f'ti:\"{t1}\" OR ti:\"{t2}\"'\n\nprint(query)\nprint(f'Number of tokens:', count_query_tokens(query))","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:29.101758Z","iopub.execute_input":"2024-07-24T14:05:29.102144Z","iopub.status.idle":"2024-07-24T14:05:29.108759Z","shell.execute_reply.started":"2024-07-24T14:05:29.102112Z","shell.execute_reply":"2024-07-24T14:05:29.107700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We've already gained one token, but this is not enough. Here's the trick ","metadata":{}},{"cell_type":"code","source":"example = 'The~magic~happens~here'\n\nprint(\" \".join(tokenizer(example)))\nprint(f'Number of tokens:', count_query_tokens(example))","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:29.825282Z","iopub.execute_input":"2024-07-24T14:05:29.825984Z","iopub.status.idle":"2024-07-24T14:05:29.831528Z","shell.execute_reply.started":"2024-07-24T14:05:29.825949Z","shell.execute_reply":"2024-07-24T14:05:29.830404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The whoosh analyser splits words separated by the `~` character, but the `count_query_tokens` them as one token only !\n\nThis also works with other special characters such as `-` and `^` and many more.","metadata":{}},{"cell_type":"code","source":"t1 = \"~\".join(tokenizer(\"Autofocusing apparatus of a sighting telescope\"))\nt2 = \"~\".join(tokenizer(\"Auto-focusing apparatus\"))\nquery = f'ti:\"{t1}\" OR ti:\"{t2}\"'\n\nprint(query)\nprint(f'Number of tokens:', count_query_tokens(query))","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:30.142594Z","iopub.execute_input":"2024-07-24T14:05:30.143345Z","iopub.status.idle":"2024-07-24T14:05:30.149526Z","shell.execute_reply.started":"2024-07-24T14:05:30.143295Z","shell.execute_reply":"2024-07-24T14:05:30.148498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Boom ! We have a 3-tokens query that matches 2 publications.\n\nWe can therefore build a 50 tokens one that will match 25 publications !\n\nFor 25 correct matches, the metric should score ~0.84 !","metadata":{}},{"cell_type":"code","source":"%%time\n\nMAX_WORDS = 20\nSPACE_TOKEN = \"~\"\n\ndf[\"processed_title\"] = df[\"title\"].apply(lambda x: SPACE_TOKEN.join(tokenizer(x)[:MAX_WORDS]))\ndf[\"length\"] = df[\"processed_title\"].apply(len)","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:40.594614Z","iopub.execute_input":"2024-07-24T14:05:40.595608Z","iopub.status.idle":"2024-07-24T14:05:44.435990Z","shell.execute_reply.started":"2024-07-24T14:05:40.595573Z","shell.execute_reply":"2024-07-24T14:05:44.434718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df.head())","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:45.135255Z","iopub.execute_input":"2024-07-24T14:05:45.135655Z","iopub.status.idle":"2024-07-24T14:05:45.148278Z","shell.execute_reply.started":"2024-07-24T14:05:45.135622Z","shell.execute_reply":"2024-07-24T14:05:45.147112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Main\n\nWe build our queries using the 25 longest titles, applying the magic preprocessing.","metadata":{}},{"cell_type":"code","source":"all_neighbors = df_test.values[:, 1:]\npublication_ids = df_test.values[:, 0]","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:45.920048Z","iopub.execute_input":"2024-07-24T14:05:45.920810Z","iopub.status.idle":"2024-07-24T14:05:45.925664Z","shell.execute_reply.started":"2024-07-24T14:05:45.920773Z","shell.execute_reply":"2024-07-24T14:05:45.924579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nqueries = []\nfor idx in tqdm(range(len(df_test))):\n    query = \"ti:device\"\n\n    try:\n        # Build neighbors dataframe\n        neighbors = all_neighbors[idx]\n        df_n = pd.DataFrame({'publication_number': neighbors})\n\n        # Retrieve metadata\n        df_n = df_n.merge(df, how=\"left\")\n\n        # Keep the 25 longest processed titles - those are less likely to give FPs\n        df_n = df_n.drop_duplicates(subset=\"processed_title\", keep=\"first\")\n        df_n = df_n.sort_values('length', ascending=False)\n        df_n = df_n.head(25)\n\n        # Build query\n        query = []\n        for title in df_n['processed_title']:\n            q = 'ti:\"' + title + '\"'\n            if len(title) and len(q) < 500:  # title sanity check\n                query.append(q)\n        query = \" OR \".join(query)\n\n    except:\n        query = \"ti:device\"\n        pass\n\n    # Catch errors\n    query = query if len(query) else \"ti:device\"\n    query = query if len(query) < 10000 else \"ti:device\"\n            \n    queries.append({\"publication_number\": publication_ids[idx], \"query\": query})","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:05:46.249105Z","iopub.execute_input":"2024-07-24T14:05:46.249518Z","iopub.status.idle":"2024-07-24T14:08:18.425181Z","shell.execute_reply.started":"2024-07-24T14:05:46.249485Z","shell.execute_reply":"2024-07-24T14:08:18.423988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.DataFrame(queries)\nsub.to_csv('submission.csv', index=False)\ndisplay(sub.head(10))","metadata":{"execution":{"iopub.status.busy":"2024-07-24T14:08:18.427029Z","iopub.execute_input":"2024-07-24T14:08:18.427369Z","iopub.status.idle":"2024-07-24T14:08:18.666977Z","shell.execute_reply.started":"2024-07-24T14:08:18.427340Z","shell.execute_reply":"2024-07-24T14:08:18.665842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Thanks for reading!*","metadata":{}}]}