{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59575,"databundleVersionId":8060720,"sourceType":"competition"}],"dockerImageVersionId":30698,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ How dataset is build ◀️</div>","metadata":{}},{"cell_type":"markdown","source":"# My guess how dataset is build ( from my job knowledge as i work in patent domain )\n\n> **Step 1 by Kaggle :  Select Publication Numbers** => Select query for unique patent number reduce by family_id ( one publication number for each family_id - from EDA is verified ) from Google BigQuery Patent\n\n---\n\n> **Step 2 by Kaggle : Find Nearest Neighbours** => For each publication find top 50 nearest neighbours embedding_v1 of size 64  using vector similarity of Google BigQuery Patent => costly so not verified with BigQuery Patent\n\n---\n\n> **Step 3 by Kaggle :  Build Queries for test set** => Building search for each top results need more manual resource. So, i guess query build only happen for hidden dataset of those 2,500 nearest neighbours using boolean queries.\n\n---\n\n> **for Kaggler** => We need to focus first to solve training Query part then training models.\n\n--- ","metadata":{}},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Patent Data: 113GB ◀️</div>","metadata":{}},{"cell_type":"code","source":"! du -sk /kaggle/input/uspto-explainable-ai/* | sort -rn","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:09:40.738382Z","iopub.execute_input":"2024-04-26T06:09:40.738959Z","iopub.status.idle":"2024-04-26T06:09:43.154450Z","shell.execute_reply.started":"2024-04-26T06:09:40.738913Z","shell.execute_reply":"2024-04-26T06:09:43.152886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Nearest Neighbors ◀️</div>","metadata":{}},{"cell_type":"code","source":"%%time\nimport dask.dataframe\nnn_df = dask.dataframe.read_csv(\"/kaggle/input/uspto-explainable-ai/nearest_neighbors.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:18.648068Z","iopub.execute_input":"2024-04-26T06:10:18.648475Z","iopub.status.idle":"2024-04-26T06:10:20.289315Z","shell.execute_reply.started":"2024-04-26T06:10:18.648442Z","shell.execute_reply":"2024-04-26T06:10:20.287519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Use Dash instead DataFrame (1msec)","metadata":{}},{"cell_type":"code","source":"nn_df.head(5)","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:20.726222Z","iopub.execute_input":"2024-04-26T06:10:20.727351Z","iopub.status.idle":"2024-04-26T06:10:25.479897Z","shell.execute_reply.started":"2024-04-26T06:10:20.727304Z","shell.execute_reply":"2024-04-26T06:10:25.478810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Patent Metadata ◀️</div>","metadata":{}},{"cell_type":"code","source":"%%time\nimport pandas as pd\npatent_meta = pd.read_parquet(\"/kaggle/input/uspto-explainable-ai/patent_metadata.parquet\")","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:25.481698Z","iopub.execute_input":"2024-04-26T06:10:25.482234Z","iopub.status.idle":"2024-04-26T06:10:53.483087Z","shell.execute_reply.started":"2024-04-26T06:10:25.482205Z","shell.execute_reply":"2024-04-26T06:10:53.482257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patent_meta","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:53.484188Z","iopub.execute_input":"2024-04-26T06:10:53.484487Z","iopub.status.idle":"2024-04-26T06:10:53.509151Z","shell.execute_reply.started":"2024-04-26T06:10:53.484462Z","shell.execute_reply":"2024-04-26T06:10:53.507798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patent_meta[0:1].info()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:53.512336Z","iopub.execute_input":"2024-04-26T06:10:53.512907Z","iopub.status.idle":"2024-04-26T06:10:53.534722Z","shell.execute_reply.started":"2024-04-26T06:10:53.512864Z","shell.execute_reply":"2024-04-26T06:10:53.533500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **publication_number** : Publication number\n* **publication_date** : Published dated\n* **file_date** : Filed Date\n* **family_id** : Unique Family ID for the same patent published in multiple countries and continution of the patent\n* **cpc_codes** : CPC Classification Codes => patent categorised into multiple cpc class codes","metadata":{}},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Publication Trends ◀️</div>","metadata":{}},{"cell_type":"code","source":"from tqdm import tqdm\ntqdm.pandas()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:53.536353Z","iopub.execute_input":"2024-04-26T06:10:53.536824Z","iopub.status.idle":"2024-04-26T06:10:53.551901Z","shell.execute_reply.started":"2024-04-26T06:10:53.536782Z","shell.execute_reply":"2024-04-26T06:10:53.550781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\npatent_meta['publication_year'] = patent_meta['publication_date'].dt.year\npatent_meta['filing_year'] = patent_meta['filing_date'].dt.year\n\npublication_year_count = patent_meta['publication_year'].value_counts().sort_index()\nfiling_year_count = patent_meta['filing_year'].value_counts().sort_index()\n\nplt.figure(figsize=(14, 7))\nplt.plot(publication_year_count.index, publication_year_count.values, marker='o', label='Publications')\nplt.plot(filing_year_count.index, filing_year_count.values, marker='x', label='Filings')\nplt.title('# of Publications and Filings per Year')\nplt.xlabel('Year')\nplt.ylabel('Total Patents')\nplt.legend()\nplt.grid(True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:53.553432Z","iopub.execute_input":"2024-04-26T06:10:53.554354Z","iopub.status.idle":"2024-04-26T06:10:56.104861Z","shell.execute_reply.started":"2024-04-26T06:10:53.554311Z","shell.execute_reply":"2024-04-26T06:10:56.103805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ CPC Codes ◀️</div>","metadata":{}},{"cell_type":"code","source":"import ast\npatent_meta['total_cpc_codes'] = patent_meta['cpc_codes'].progress_apply(len)\nplt.figure(figsize=(14, 7))\nplt.subplot(1, 2, 1)\nsns.histplot(patent_meta['total_cpc_codes'], bins=20, kde=False)\nplt.title('# of CPC Codes per Patent')\nplt.xlabel('Total CPC Codes')\nplt.ylabel('Frequency')\n\n\nplt.subplot(1, 2, 2)\nsns.boxplot(y=patent_meta['total_cpc_codes'])\nplt.title('Total CPC Codes per Patent')\nplt.ylabel('Total CPC Codes')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:10:56.106196Z","iopub.execute_input":"2024-04-26T06:10:56.106861Z","iopub.status.idle":"2024-04-26T06:11:25.016513Z","shell.execute_reply.started":"2024-04-26T06:10:56.106828Z","shell.execute_reply":"2024-04-26T06:11:25.015217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filtered_patent_meta = patent_meta[patent_meta['total_cpc_codes'] < 25]\nplt.figure(figsize=(14, 7))\nplt.subplot(1, 2, 1)\nsns.histplot(filtered_patent_meta['total_cpc_codes'], bins=20, kde=False)\nplt.title('# of CPC Codes per Patent')\nplt.xlabel('Total CPC Codes')\nplt.ylabel('Frequency')\n\n\nplt.subplot(1, 2, 2)\nsns.boxplot(y=filtered_patent_meta['total_cpc_codes'])\nplt.title('Total CPC Codes per Patent')\nplt.ylabel('Total CPC Codes')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:11:25.018221Z","iopub.execute_input":"2024-04-26T06:11:25.018580Z","iopub.status.idle":"2024-04-26T06:11:38.721849Z","shell.execute_reply.started":"2024-04-26T06:11:25.018549Z","shell.execute_reply":"2024-04-26T06:11:38.720743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patent_meta['primary_cpc_class'] = patent_meta['cpc_codes'].progress_apply(lambda x: list(set([y[0] for y in x if len(y) > 0])))","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:11:38.723469Z","iopub.execute_input":"2024-04-26T06:11:38.724211Z","iopub.status.idle":"2024-04-26T06:12:39.244605Z","shell.execute_reply.started":"2024-04-26T06:11:38.724172Z","shell.execute_reply":"2024-04-26T06:12:39.243394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patent_meta['total_primary_cpc_codes'] = patent_meta['primary_cpc_class'].progress_apply(len)","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:12:39.249461Z","iopub.execute_input":"2024-04-26T06:12:39.249939Z","iopub.status.idle":"2024-04-26T06:12:55.467937Z","shell.execute_reply.started":"2024-04-26T06:12:39.249907Z","shell.execute_reply":"2024-04-26T06:12:55.466769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filtered_patent_meta = patent_meta\nplt.figure(figsize=(14, 7))\nplt.subplot(1, 2, 1)\nsns.histplot(filtered_patent_meta['total_primary_cpc_codes'], bins=20, kde=False)\nplt.title('# of Priary CPC Codes per Patent')\nplt.xlabel('Total CPC Codes')\nplt.ylabel('Frequency')\n\n\nplt.subplot(1, 2, 2)\nsns.boxplot(y=filtered_patent_meta['total_primary_cpc_codes'])\nplt.title('Total Primary CPC Codes per Patent')\nplt.ylabel('Total CPC Codes')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:12:55.469445Z","iopub.execute_input":"2024-04-26T06:12:55.469893Z","iopub.status.idle":"2024-04-26T06:13:06.705983Z","shell.execute_reply.started":"2024-04-26T06:12:55.469854Z","shell.execute_reply":"2024-04-26T06:13:06.704743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patent_meta['first_primary_cpc_codes'] = patent_meta['primary_cpc_class'].progress_apply(lambda x: x[0] if len(x) > 0 else None)\nprimary_group_counts = patent_meta['first_primary_cpc_codes'].value_counts().sort_index()\nprint(primary_group_counts)\nplt.figure(figsize=(10, 5))\nsns.barplot(x=primary_group_counts.index, y=primary_group_counts.values, palette=\"viridis\")\nplt.title('# of First Primary CPC Groups')\nplt.xlabel('First Primary Group')\nplt.ylabel('Total Patents')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:13:06.707626Z","iopub.execute_input":"2024-04-26T06:13:06.708097Z","iopub.status.idle":"2024-04-26T06:13:22.616375Z","shell.execute_reply.started":"2024-04-26T06:13:06.708057Z","shell.execute_reply":"2024-04-26T06:13:22.615146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patent_meta['second_primary_cpc_codes'] = patent_meta['primary_cpc_class'].progress_apply(lambda x: x[1] if len(x) > 1 else None)\nsecond_group_counts = patent_meta['second_primary_cpc_codes'].value_counts().sort_index()\nprint(second_group_counts)\nplt.figure(figsize=(10, 5))\nsns.barplot(x=second_group_counts.index, y=second_group_counts.values, palette=\"viridis\")\nplt.title('# of First Primary CPC Groups')\nplt.xlabel('Second Primary Group')\nplt.ylabel('Total Patents')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:13:22.617746Z","iopub.execute_input":"2024-04-26T06:13:22.618156Z","iopub.status.idle":"2024-04-26T06:13:38.459301Z","shell.execute_reply.started":"2024-04-26T06:13:22.618129Z","shell.execute_reply":"2024-04-26T06:13:38.458084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"publication_group_year = patent_meta.groupby(['first_primary_cpc_codes', 'publication_year']).size().unstack(fill_value=0)\npublication_group_year = publication_group_year.sort_index(axis=1)\n\nplt.figure(figsize=(12, 8))\nsns.heatmap(publication_group_year, annot=False, cmap=\"viridis\")\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:16:50.952407Z","iopub.execute_input":"2024-04-26T06:16:50.954034Z","iopub.status.idle":"2024-04-26T06:16:53.708023Z","shell.execute_reply.started":"2024-04-26T06:16:50.953977Z","shell.execute_reply":"2024-04-26T06:16:53.706887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patent_meta.columns\npatent_meta['missing_cpc_codes'] = patent_meta['total_primary_cpc_codes'].progress_apply(lambda x: None if x == 0 else x )","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:13:48.945704Z","iopub.execute_input":"2024-04-26T06:13:48.946163Z","iopub.status.idle":"2024-04-26T06:14:04.985385Z","shell.execute_reply.started":"2024-04-26T06:13:48.946130Z","shell.execute_reply":"2024-04-26T06:14:04.984130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Publication Country/Kind Code ◀️</div>","metadata":{}},{"cell_type":"code","source":"patent_meta['publication_country'] = patent_meta['publication_number'].progress_apply(lambda x: x.split('-')[0] if '-' in x else None)\npatent_meta['publication_kind_code'] = patent_meta['publication_number'].progress_apply(lambda x: x.split('-')[-1] if '-' in x else None)","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:14:36.823613Z","iopub.execute_input":"2024-04-26T06:14:36.824035Z","iopub.status.idle":"2024-04-26T06:15:12.854694Z","shell.execute_reply.started":"2024-04-26T06:14:36.824003Z","shell.execute_reply":"2024-04-26T06:15:12.853527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"publication_country_count = patent_meta['publication_country'].value_counts().sort_index()\nplt.figure(figsize=(10, 5))\nsns.barplot(x=publication_country_count.index, y=publication_country_count.values, palette=\"viridis\")\nplt.title('# of Patents per Country')\nplt.xlabel('Country')\nplt.ylabel('Total Patents')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:18:06.203406Z","iopub.execute_input":"2024-04-26T06:18:06.203880Z","iopub.status.idle":"2024-04-26T06:18:07.690420Z","shell.execute_reply.started":"2024-04-26T06:18:06.203834Z","shell.execute_reply":"2024-04-26T06:18:07.688345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"publication_kind_code_count = patent_meta['publication_kind_code'].value_counts().sort_index()\nplt.figure(figsize=(10, 5))\nsns.barplot(x=publication_kind_code_count.index, y=publication_kind_code_count.values, palette=\"viridis\")\nplt.title('# of Patents per Kind Codes')\nplt.xlabel('Kind Codes')\nplt.ylabel('Total Patents')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:18:40.250208Z","iopub.execute_input":"2024-04-26T06:18:40.250603Z","iopub.status.idle":"2024-04-26T06:18:41.707391Z","shell.execute_reply.started":"2024-04-26T06:18:40.250575Z","shell.execute_reply":"2024-04-26T06:18:41.706173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Missing Values ◀️</div>","metadata":{}},{"cell_type":"code","source":"missing_values = patent_meta[['publication_number','publication_date','filing_date','family_id','missing_cpc_codes','publication_country','publication_kind_code']].isnull().sum()\nplt.figure(figsize=(10, 6))\nsns.barplot(x=missing_values.values, y=missing_values.index, palette='viridis')\nplt.title('Missing Values')\nplt.xlabel('# of Missing Values')\nplt.ylabel('Columns')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:15:20.738192Z","iopub.execute_input":"2024-04-26T06:15:20.738721Z","iopub.status.idle":"2024-04-26T06:15:24.856030Z","shell.execute_reply.started":"2024-04-26T06:15:20.738683Z","shell.execute_reply":"2024-04-26T06:15:24.854598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Family Ids ◀️</div>","metadata":{}},{"cell_type":"code","source":"filtered_patent_meta = patent_meta[~patent_meta['family_id'].isnull()].reset_index(drop=True)\nfiltered_patent_meta = filtered_patent_meta[~filtered_patent_meta['publication_date'].isnull()].reset_index(drop=True)\nfiltered_patent_meta = filtered_patent_meta.loc[filtered_patent_meta.groupby('family_id')['publication_date'].idxmin()]","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:13:41.176054Z","iopub.status.idle":"2024-04-26T06:13:41.176892Z","shell.execute_reply.started":"2024-04-26T06:13:41.176686Z","shell.execute_reply":"2024-04-26T06:13:41.176703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filtered_patent_meta.shape, patent_meta.shape","metadata":{"execution":{"iopub.status.busy":"2024-04-26T06:13:41.178186Z","iopub.status.idle":"2024-04-26T06:13:41.178531Z","shell.execute_reply.started":"2024-04-26T06:13:41.178362Z","shell.execute_reply":"2024-04-26T06:13:41.178376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"color:white;display:fill;border-radius:5px;background-color:seaGreen;text-align:center;letter-spacing:0.1px;overflow:hidden;padding:20px;color:white;overflow:hidden;margin:0;font-size:100%\">▶️ Upvote if it useful ◀️</div>","metadata":{}},{"cell_type":"code","source":"! du -sk /kaggle/input/uspto-explainable-ai/patent_data/* | sort -rn","metadata":{"execution":{"iopub.status.busy":"2024-04-26T09:01:56.136569Z","iopub.execute_input":"2024-04-26T09:01:56.137105Z","iopub.status.idle":"2024-04-26T09:01:58.518902Z","shell.execute_reply.started":"2024-04-26T09:01:56.137062Z","shell.execute_reply":"2024-04-26T09:01:58.517792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}