{
  "id": 498102,
  "title": "All we need TF IDF instead BM25 score for boolean query",
  "url": "/competitions/uspto-explainable-ai/discussion/498102",
  "author_name": "SeshuRaju 🧘‍♂️",
  "post_date": "2024-04-26T21:38:54.835000",
  "votes": 7,
  "comment_count": 0,
  "views": 0,
  "content": "<blockquote>\n  <p>In information retrieval, tf–idf (also TF*IDF, TFIDF, TF–IDF, or Tf–idf), short for term frequency–inverse document frequency, is a measure of importance of a word to a document in a collection or corpus, adjusted for the fact that some words appear more frequently in general. <a href=\"https://en.wikipedia.org/wiki/Tf%E2%80%93idf\" target=\"_blank\">Source - Wikipedia</a></p>\n</blockquote>\n<hr>\n<h1>Score by TF IDF</h1>\n<pre><code> ():\n   whoosh_index.searcher(weighting=whoosh.scoring.TF_IDF())\n</code></pre>\n<h1>Max Tokens 50 =&gt; Max Operators -&gt; 24 and Key words -&gt; 26</h1>\n<pre><code> re\n\n ():\n    tokens = [i  i  re.split(, query)  i]\n     (tokens), (tokens)\n</code></pre>\n<pre><code>       \n  \n</code></pre>\n<h1>Text =&gt; Tokens</h1>\n<pre><code> re\n whoosh\n\nNUMBER_REGEX = re.()\n\n (whoosh.analysis.Filter):\n     ():\n         t  tokens:\n              NUMBER_REGEX.(t.text):\n                 t\n\nBRS_STOPWORDS = [, , , , , , , , , , , ,\n        , , , , , , , , , , ]\ntext_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\n\ntokens = text_analyzer(text)\n</code></pre>\n<pre><code>                                                                                                                                                                          \n\n [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ]\n</code></pre>\n<h2><a href=\"https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda?scriptVersionId=174207354\" target=\"_blank\">Notebook - Patent Professionals USPTO: whoosh utils EDA</a></h2>",
  "messages": [
    {
      "id": 2777943,
      "postDate": "2024-04-26T21:38:54.837Z",
      "content": "<blockquote>\n  <p>In information retrieval, tf–idf (also TF*IDF, TFIDF, TF–IDF, or Tf–idf), short for term frequency–inverse document frequency, is a measure of importance of a word to a document in a collection or corpus, adjusted for the fact that some words appear more frequently in general. <a href=\"https://en.wikipedia.org/wiki/Tf%E2%80%93idf\" target=\"_blank\">Source - Wikipedia</a></p>\n</blockquote>\n<hr>\n<h1>Score by TF IDF</h1>\n<pre><code> ():\n   whoosh_index.searcher(weighting=whoosh.scoring.TF_IDF())\n</code></pre>\n<h1>Max Tokens 50 =&gt; Max Operators -&gt; 24 and Key words -&gt; 26</h1>\n<pre><code> re\n\n ():\n    tokens = [i  i  re.split(, query)  i]\n     (tokens), (tokens)\n</code></pre>\n<pre><code>       \n  \n</code></pre>\n<h1>Text =&gt; Tokens</h1>\n<pre><code> re\n whoosh\n\nNUMBER_REGEX = re.()\n\n (whoosh.analysis.Filter):\n     ():\n         t  tokens:\n              NUMBER_REGEX.(t.text):\n                 t\n\nBRS_STOPWORDS = [, , , , , , , , , , , ,\n        , , , , , , , , , , ]\ntext_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\n\ntokens = text_analyzer(text)\n</code></pre>\n<pre><code>                                                                                                                                                                          \n\n [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ]\n</code></pre>\n<h2><a href=\"https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda?scriptVersionId=174207354\" target=\"_blank\">Notebook - Patent Professionals USPTO: whoosh utils EDA</a></h2>",
      "rawMarkdown": "> In information retrieval, tf–idf (also TF*IDF, TFIDF, TF–IDF, or Tf–idf), short for term frequency–inverse document frequency, is a measure of importance of a word to a document in a collection or corpus, adjusted for the fact that some words appear more frequently in general. [Source - Wikipedia](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)\n\n---\n\n\n# Score by TF IDF\n```python\ndef get_searcher(whoosh_index):\n  return whoosh_index.searcher(weighting=whoosh.scoring.TF_IDF())\n```\n\n# Max Tokens 50 => Max Operators -> 24 and Key words -> 26\n```python\nimport re\n\ndef count_query_tokens(query: str):\n    tokens = [i for i in re.split('[\\s+()]', query) if i]\n    return len(tokens), str(tokens)\n```\n\n```yaml\nQuery: (cpc:AO1B33/00 AND cpc:B60H1/00885) OR (ti:bread ADJ5 ti:cheese)\nTokens: (7, \"['cpc:AO1B33/00', 'AND', 'cpc:B60H1/00885', 'OR', 'ti:bread', 'ADJ5', 'ti:cheese']\")\n```\n\n# Text => Tokens\n```python\nimport re\nimport whoosh\n\nNUMBER_REGEX = re.compile(r'^(\\d+|\\d{1,3}(,\\d{3})*)(\\.\\d+)?$')\n\nclass NumberFilter(whoosh.analysis.Filter):\n    def __call__(self, tokens):\n        for t in tokens:\n            if not NUMBER_REGEX.match(t.text):\n                yield t\n                \nBRS_STOPWORDS = ['an', 'are', 'by', 'for', 'if', 'into', 'is', 'no', 'not', 'of', 'on', 'such',\n        'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'to', 'was', 'will']\ntext_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\n\ntokens = text_analyzer(text)\n\n```\n\n```yaml\nAbstract: A method is provided for determining a weight of a payload carried by a support structure of an agricultural utility vehicle via a hitch. The hitch includes at least one upper link and at least one lower link. The method includes determining the weight based on at least one of (1) an angle between the upper link and a vehicle horizontal, and (2) a holding force that arises at a connection between the upper link and the payload and is effective along the upper link., A method is provided for determining a weight of a payload carried by a support structure of an agricultural utility vehicle via a hitch. The hitch includes at least one upper link and at least one lower link. The method includes determining the weight based on at least one of (1) an angle between the upper link and a vehicle horizontal, and (2) a holding force that arises at a connection between the upper link and the payload and is effective along the upper link.\n\nAbstract Tokens: ['method', 'provided', 'determining', 'weight', 'payload', 'carried', 'support', 'structure', 'agricultural', 'utility', 'vehicle', 'via', 'hitch', 'hitch', 'includes', 'at', 'least', 'one', 'upper', 'link', 'and', 'at', 'least', 'one', 'lower', 'link', 'method', 'includes', 'determining', 'weight', 'based', 'at', 'least', 'one', 'angle', 'between', 'upper', 'link', 'and', 'vehicle', 'horizontal', 'and', 'holding', 'force', 'arises', 'at', 'connection', 'between', 'upper', 'link', 'and', 'payload', 'and', 'effective', 'along', 'upper', 'link', 'method', 'provided', 'determining', 'weight', 'payload', 'carried', 'support', 'structure', 'agricultural', 'utility', 'vehicle', 'via', 'hitch', 'hitch', 'includes', 'at', 'least', 'one', 'upper', 'link', 'and', 'at', 'least', 'one', 'lower', 'link', 'method', 'includes', 'determining', 'weight', 'based', 'at', 'least', 'one', 'angle', 'between', 'upper', 'link', 'and', 'vehicle', 'horizontal', 'and', 'holding', 'force', 'arises', 'at', 'connection', 'between', 'upper', 'link', 'and', 'payload', 'and', 'effective', 'along', 'upper', 'link']\n```\n\n## [Notebook - Patent Professionals USPTO: whoosh utils EDA](https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda?scriptVersionId=174207354)",
      "votes": 7
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2777943": "> In information retrieval, tf–idf (also TF*IDF, TFIDF, TF–IDF, or Tf–idf), short for term frequency–inverse document frequency, is a measure of importance of a word to a document in a collection or corpus, adjusted for the fact that some words appear more frequently in general. [Source - Wikipedia](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)\n\n---\n\n\n# Score by TF IDF\n```python\ndef get_searcher(whoosh_index):\n  return whoosh_index.searcher(weighting=whoosh.scoring.TF_IDF())\n```\n\n# Max Tokens 50 => Max Operators -> 24 and Key words -> 26\n```python\nimport re\n\ndef count_query_tokens(query: str):\n    tokens = [i for i in re.split('[\\s+()]', query) if i]\n    return len(tokens), str(tokens)\n```\n\n```yaml\nQuery: (cpc:AO1B33/00 AND cpc:B60H1/00885) OR (ti:bread ADJ5 ti:cheese)\nTokens: (7, \"['cpc:AO1B33/00', 'AND', 'cpc:B60H1/00885', 'OR', 'ti:bread', 'ADJ5', 'ti:cheese']\")\n```\n\n# Text => Tokens\n```python\nimport re\nimport whoosh\n\nNUMBER_REGEX = re.compile(r'^(\\d+|\\d{1,3}(,\\d{3})*)(\\.\\d+)?$')\n\nclass NumberFilter(whoosh.analysis.Filter):\n    def __call__(self, tokens):\n        for t in tokens:\n            if not NUMBER_REGEX.match(t.text):\n                yield t\n                \nBRS_STOPWORDS = ['an', 'are', 'by', 'for', 'if', 'into', 'is', 'no', 'not', 'of', 'on', 'such',\n        'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'to', 'was', 'will']\ntext_analyzer = whoosh.analysis.StandardAnalyzer(stoplist=BRS_STOPWORDS) | NumberFilter()\n\ntokens = text_analyzer(text)\n\n```\n\n```yaml\nAbstract: A method is provided for determining a weight of a payload carried by a support structure of an agricultural utility vehicle via a hitch. The hitch includes at least one upper link and at least one lower link. The method includes determining the weight based on at least one of (1) an angle between the upper link and a vehicle horizontal, and (2) a holding force that arises at a connection between the upper link and the payload and is effective along the upper link., A method is provided for determining a weight of a payload carried by a support structure of an agricultural utility vehicle via a hitch. The hitch includes at least one upper link and at least one lower link. The method includes determining the weight based on at least one of (1) an angle between the upper link and a vehicle horizontal, and (2) a holding force that arises at a connection between the upper link and the payload and is effective along the upper link.\n\nAbstract Tokens: ['method', 'provided', 'determining', 'weight', 'payload', 'carried', 'support', 'structure', 'agricultural', 'utility', 'vehicle', 'via', 'hitch', 'hitch', 'includes', 'at', 'least', 'one', 'upper', 'link', 'and', 'at', 'least', 'one', 'lower', 'link', 'method', 'includes', 'determining', 'weight', 'based', 'at', 'least', 'one', 'angle', 'between', 'upper', 'link', 'and', 'vehicle', 'horizontal', 'and', 'holding', 'force', 'arises', 'at', 'connection', 'between', 'upper', 'link', 'and', 'payload', 'and', 'effective', 'along', 'upper', 'link', 'method', 'provided', 'determining', 'weight', 'payload', 'carried', 'support', 'structure', 'agricultural', 'utility', 'vehicle', 'via', 'hitch', 'hitch', 'includes', 'at', 'least', 'one', 'upper', 'link', 'and', 'at', 'least', 'one', 'lower', 'link', 'method', 'includes', 'determining', 'weight', 'based', 'at', 'least', 'one', 'angle', 'between', 'upper', 'link', 'and', 'vehicle', 'horizontal', 'and', 'holding', 'force', 'arises', 'at', 'connection', 'between', 'upper', 'link', 'and', 'payload', 'and', 'effective', 'along', 'upper', 'link']\n```\n\n## [Notebook - Patent Professionals USPTO: whoosh utils EDA](https://www.kaggle.com/code/seshurajup/patent-professionals-uspto-whoosh-utils-eda?scriptVersionId=174207354)"
  }
}