{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":59575,"databundleVersionId":8060720,"sourceType":"competition"},{"sourceId":33551,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":28083}],"dockerImageVersionId":30699,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on April 27, 2024. By Marília Prata, mpwolke","metadata":{"_kg_hide-output":false}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport plotly.graph_objs as go\nimport plotly.offline as py\nimport plotly.express as px\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-28T00:35:19.135650Z","iopub.execute_input":"2024-04-28T00:35:19.136450Z","iopub.status.idle":"2024-04-28T00:35:23.812839Z","shell.execute_reply.started":"2024-04-28T00:35:19.136417Z","shell.execute_reply":"2024-04-28T00:35:23.811898Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Competition Goal: Generate Boolean search queries.\n\n\"The goal of this competition is to generate Boolean search queries that effectively characterize collections of patent documents. You are challenged to create a query generation model that, given an input set of related patents, outputs a Boolean query that returns the same set of patent documents.\"\n\nUSPTO Default Operators: OR, AND, ADJ, NEAR, SAME, WITH.\n\nhttps://ppubs.uspto.gov/pubwebapp/\n\nhttps://www.kaggle.com/competitions/uspto-explainable-ai","metadata":{}},{"cell_type":"markdown","source":"#George Boole - The standard Boolean operators (AND, OR, NOT, or NEAR)\n\n![](https://study.com/cimages/multimages/16/george-boole-boolean-operators.jpg)https://study.com/academy/lesson/search-engines-keywords-web-portals.html","metadata":{}},{"cell_type":"markdown","source":"#George Boole AND Boolean Search\n\nBy Wendy Boswell\n\nWHERE DOES THE TERM BOOLEAN ORIGINATE?\n\n\"George Boole, an English mathematician in the 19th century, developed \"Boolean Logic\" in order to combine certain concepts and exclude certain concepts when searching databases.\"\n\nHOW DO WE DO A BOOLEAN SEARCH?\n\n\"You have two choices: you can use the standard Boolean operators (AND, OR, \nNOT, or NEAR), or you can use their math equivalents.\"\n\nBoolean Search Operators\n\n• \"The Boolean search operator AND is equal to the \"+\" symbol.\"\n\n• \"The Boolean search operator NOT is equal to the \"-\" symbol.\"\n\n• \"The Boolean search operator OR is the default setting of any search \nengine; meaning, all search engines will return all the words you type in, \nautomatically.\"\n\n• \"The Boolean search operator NEAR is equal to putting a search query in \nquotes, i.e., \"sponge bob squarepants\". You're essentially telling \nthe search engine that you want all of these words, in this specific order, or this specific phrase.\"\n\nhttps://www.hplct.org/assets/uploads/files/Immigrant%20Youth/What%20Does%20Boolean%20Search%20Really%20Mean.pdf","metadata":{}},{"cell_type":"markdown","source":"#Import packages","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nfrom time import time\nimport torch\nimport transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nfrom IPython.display import display, Markdown","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:36:41.266581Z","iopub.execute_input":"2024-04-28T00:36:41.267164Z","iopub.status.idle":"2024-04-28T00:36:46.842360Z","shell.execute_reply.started":"2024-04-28T00:36:41.267117Z","shell.execute_reply":"2024-04-28T00:36:46.841399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By SeshuRajup https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda/notebook\n\npatent_details = pd.read_parquet(\"/kaggle/input/uspto-explainable-ai/patent_data/2013_12.parquet\")","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:36:57.099796Z","iopub.execute_input":"2024-04-28T00:36:57.101003Z","iopub.status.idle":"2024-04-28T00:37:08.522426Z","shell.execute_reply.started":"2024-04-28T00:36:57.100960Z","shell.execute_reply":"2024-04-28T00:37:08.521631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Whoosh","metadata":{}},{"cell_type":"code","source":"!pip install whoosh","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-04-28T00:37:13.051568Z","iopub.execute_input":"2024-04-28T00:37:13.051944Z","iopub.status.idle":"2024-04-28T00:37:27.358449Z","shell.execute_reply.started":"2024-04-28T00:37:13.051915Z","shell.execute_reply":"2024-04-28T00:37:27.356996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By SeshuRajup https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda/notebook\n\nimport re\nimport whoosh\nimport whoosh.analysis\n\nNUMBER_REGEX = re.compile(r'^(\\d+|\\d{1,3}(,\\d{3})*)(\\.\\d+)?$')\n\nclass NumberFilter(whoosh.analysis.Filter):\n    def __call__(self, tokens):\n        for t in tokens:\n            if not NUMBER_REGEX.match(t.text):\n                yield t\n                \nBRS_STOPWORDS = ['an', 'are', 'by', 'for', 'if', 'into', 'is', 'no', 'not', 'of', 'on', 'such',\n        'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'to', 'was', 'will']\ntext_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:37:30.676934Z","iopub.execute_input":"2024-04-28T00:37:30.677338Z","iopub.status.idle":"2024-04-28T00:37:30.701423Z","shell.execute_reply.started":"2024-04-28T00:37:30.677303Z","shell.execute_reply":"2024-04-28T00:37:30.700419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Orthodontic Patents Abstracts on December 2013\n\nNumber 1284 was taken from the previous Code: USPTO Finding Patents Whoosh & Non-Whoosh ways","metadata":{}},{"cell_type":"code","source":"\n#By SeshuRajup https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda/notebook\n\nfrom IPython.core.display import display, HTML\ndef parse_data(section, text):\n    display(HTML(f\"<b style='color:blue'>{section.title()}</b>: <span style='color:blue'>{text}<span>\"))\n    tokens = text_analyzer(text)\n    display(HTML(f\"<b style='color:green'>{section.title()} Tokens</b>: <span style='color:green'>{str([token.text for token in tokens])}</span>\"))\n    \nfor i in range(len(patent_details['publication_number'])):\n    if i > 0:  #Original was higher number\n        break\n    \n    pubnum = patent_details['publication_number'][1284]\n    title = patent_details['title'][1284]\n    abstract = patent_details['abstract'][1284]\n    #claims = patent_details['claims'][1284]\n    #description = patent_details['description'][1284]\n    display(HTML(f\"<br/>Publication number: <b style='color:red'>{pubnum}</b>\"))\n    parse_data('title', title)\n    parse_data('abstract', abstract)\n    #parse_data('claims', claims)\n    #parse_data('description', description)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-28T00:37:35.972059Z","iopub.execute_input":"2024-04-28T00:37:35.972445Z","iopub.status.idle":"2024-04-28T00:37:35.993756Z","shell.execute_reply.started":"2024-04-28T00:37:35.972417Z","shell.execute_reply":"2024-04-28T00:37:35.992735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Cryptografic Patent December 2013\n\nNumber 20195 was taken from the previous Code: [USPTO Finding Patents Whoosh & Non-Whoosh ways](https://www.kaggle.com/code/mpwolke/uspto-finding-patents-whoosh-non-whoosh-ways)","metadata":{}},{"cell_type":"code","source":"for i in range(len(patent_details['publication_number'])):\n    if i > 0:  #Original was higher number\n        break\n    \n    pubnum = patent_details['publication_number'][20195]\n    title = patent_details['title'][20195]\n    abstract = patent_details['abstract'][20195]\n    #claims = patent_details['claims'][20195]\n    #description = patent_details['description'][20195]\n    display(HTML(f\"<br/>Publication number: <b style='color:red'>{pubnum}</b>\"))\n    parse_data('title', title)\n    parse_data('abstract', abstract)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-28T00:37:45.554591Z","iopub.execute_input":"2024-04-28T00:37:45.555232Z","iopub.status.idle":"2024-04-28T00:37:45.570853Z","shell.execute_reply.started":"2024-04-28T00:37:45.555202Z","shell.execute_reply":"2024-04-28T00:37:45.569855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nmodel = \"/kaggle/input/llama-3/transformers/8b-chat-hf/1\"\n\npipeline = transformers.pipeline(\n    \"text-generation\",\n    model=model,\n    torch_dtype=torch.float16,\n    device_map=\"auto\",\n)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-04-28T00:38:52.194536Z","iopub.execute_input":"2024-04-28T00:38:52.195352Z","iopub.status.idle":"2024-04-28T00:40:54.279153Z","shell.execute_reply.started":"2024-04-28T00:38:52.195315Z","shell.execute_reply":"2024-04-28T00:40:54.278287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Define model","metadata":{}},{"cell_type":"markdown","source":"#Query Function","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\ndef query_model(\n    prompt, \n    temperature=0.7,\n    max_length=512\n    ):\n    start_time = time()\n    sequences = pipeline(\n        prompt,\n        do_sample=True,\n        top_k=10,\n        temperature=temperature,\n        num_return_sequences=1,\n        eos_token_id=pipeline.tokenizer.eos_token_id,\n        max_length=max_length,\n    )\n    answer = f\"{sequences[0]['generated_text'][len(prompt):]}\\n\"\n    end_time = time()\n    ttime = f\"Total time: {round(end_time-start_time, 2)} sec.\"\n\n    return prompt + \" \" + answer  + \" \" +  ttime","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:40:59.896777Z","iopub.execute_input":"2024-04-28T00:40:59.898002Z","iopub.status.idle":"2024-04-28T00:40:59.904524Z","shell.execute_reply.started":"2024-04-28T00:40:59.897969Z","shell.execute_reply":"2024-04-28T00:40:59.903514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Utility Function","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\ndef colorize_text(text):\n    for word, color in zip([\"Reasoning\", \"Question\", \"Answer\", \"Total time\"], [\"blue\", \"red\", \"green\", \"magenta\"]):\n        text = text.replace(f\"{word}:\", f\"\\n\\n**<font color='{color}'>{word}:</font>**\")\n    return text","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:41:09.037437Z","iopub.execute_input":"2024-04-28T00:41:09.038253Z","iopub.status.idle":"2024-04-28T00:41:09.043494Z","shell.execute_reply.started":"2024-04-28T00:41:09.038218Z","shell.execute_reply":"2024-04-28T00:41:09.042539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Test with few simple questions","metadata":{}},{"cell_type":"markdown","source":"#Turn on GPU to avoid error\n\nRuntimeError: \"addmm_implcpu\" not implemented for 'Half'","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nprompt = \"\"\"\nYou are an AI assistant designed to answer simple questions.\nPlease restrict your answer to the exact question asked.\nPlease limit your answer to less than {size} tokens.\nQuestion: {question}\nAnswer:\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:41:16.443455Z","iopub.execute_input":"2024-04-28T00:41:16.444173Z","iopub.status.idle":"2024-04-28T00:41:16.448762Z","shell.execute_reply.started":"2024-04-28T00:41:16.444142Z","shell.execute_reply":"2024-04-28T00:41:16.447638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Orthodontic Patents Abstracts on December 2013\n\nNumber 1284 was taken from the previous Code: [USPTO Finding Patents Whoosh & Non-Whoosh ways](https://www.kaggle.com/code/mpwolke/uspto-finding-patents-whoosh-non-whoosh-ways)","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\n    prompt.format(question=\"Were Orthodontic OR Cryptographic US Patents published?\",\n                 size=32), \n    max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:41:30.944191Z","iopub.execute_input":"2024-04-28T00:41:30.944563Z","iopub.status.idle":"2024-04-28T00:43:41.766828Z","shell.execute_reply.started":"2024-04-28T00:41:30.944532Z","shell.execute_reply":"2024-04-28T00:43:41.765811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#I expected just True or False, though Llama provided more info than it was on the query.\n\n\"Dr. Angle is known as the father of Orthodontics. Angle died on August 11, 1930, in Santa Monica at the age of 75 from heart failure saying, \"I have finished my work and I did my best.\"\n\nUnfortunately, the Model changed its answer.","metadata":{}},{"cell_type":"markdown","source":"#The Boolean search operator AND is equal to the \"+\" symbol.\"","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic AND Cryptographic Patents were published?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:45:15.391238Z","iopub.execute_input":"2024-04-28T00:45:15.391630Z","iopub.status.idle":"2024-04-28T00:47:47.531747Z","shell.execute_reply.started":"2024-04-28T00:45:15.391600Z","shell.execute_reply":"2024-04-28T00:47:47.530743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#The model changed its answer and wrote: \"I am sure there are many more interesting cases.\"\n\nI'm not interested in what you think Llama. For the record, DO Not share your opinion. Just Answer!","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic Patents + Cryptographic Patents were published?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#\"The Boolean search operator NEAR\n\nNEAR is equal to putting a search query in quotes, i.e., \"sponge bob squarepants\". You're essentially telling the search engine that you want all of these words, in this specific order, or this specific phrase.\"\n\nhttps://www.hplct.org/assets/uploads/files/Immigrant%20Youth/What%20Does%20Boolean%20Search%20Really%20Mean.pdf","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic Patents NEAR Aligners were published?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{"execution":{"iopub.status.busy":"2024-04-28T00:50:30.438588Z","iopub.execute_input":"2024-04-28T00:50:30.439045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#I'm satisfied with \"Orthodontic Patents NEAR Aligners\" Llama3 answer","metadata":{}},{"cell_type":"markdown","source":"#The Boolean search operator NOT is equal to the \"-\" symbol.\"","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic Patents NOT Publication number: US-2013323665-A1?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_parquet(\"/kaggle/input/uspto-explainable-ai/patent_data/2013_12.parquet\")\ndf.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['abstract'][1284]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta = pd.read_parquet(\"/kaggle/input/uspto-explainable-ai/patent_metadata.parquet\")\nmeta.head(2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"arch = meta[(meta['publication_number']=='US-2013323665-A1')].reset_index(drop=True)\narch.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#The model wrote US-2013323665-A1 Abstract. Publication_date isn't correct ","metadata":{}},{"cell_type":"markdown","source":"#The Boolean search operator NOT is equal to the \"-\" symbol.\"","metadata":{}},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic Patents - US Publication number: US-RE44668-E?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Cryptographic Patents - US Publication number: US-8612761-B2?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic Patent ADJ Publication number: US-2013323697-A1?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic Patent Publication number: US-2013323697-A1 SAME title?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Gabriel Preda https://www.kaggle.com/code/gpreda/prompting-llama-3-like-a-pro\n\nresponse = query_model(\"Orthodontic Patent WITH Publication number: US-2013323697-A1?\",\n     max_length=256)\ndisplay(Markdown(colorize_text(response)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Shakespearean Boolean Search': To Whoosh or Not to Whoosh?\n\n![](https://media.licdn.com/dms/image/C5612AQE6dlLeldxQpw/article-cover_image-shrink_600_2000/0/1520205437016?e=2147483647&v=beta&t=g-wAG3jIiiqhaFYq9NgMNnSTLxVUfVTGwYlAF00b2Cw)","metadata":{}},{"cell_type":"markdown","source":"#Why I applied Llama3:\n\nI intended to use Gemma/Langchain though it doesn't have ParquetLoaders to load the parquet files.\n\nThen, I tried Mistral which gave me wrong answers, took a lot of time and when I ran the \"Save and run all commit\" version, it returned: \"RuntimeError: cutlassF: no kernel found to launch!","metadata":{}},{"cell_type":"markdown","source":"#Dr. Angle \"father of Orthodontics\" last quote: \"I have finished my work and I did my best.\"\n\n![](https://image.slidesharecdn.com/copyofbiographicalaccountof-140211023917-phpapp02/85/copy-of-biographical-account-of-certified-fixed-orthodontic-courses-by-indian-dental-academy-74-320.jpg)https://pt.slideshare.net/indiandentalacademy/copy-of-biographical-account-of","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nGabriel Preda https://www.kaggle.com/code/gpreda/multiple-task-chain-with-mistral-model\n\nSeshuRajup https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda/notebook","metadata":{}}]}