{
  "id": 522201,
  "title": "5th Place Solution",
  "url": "/competitions/uspto-explainable-ai/writeups/mcd-5th-place-solution",
  "author_name": "",
  "post_date": "2024-08-06T09:24:58.303Z",
  "votes": 26,
  "comment_count": 2,
  "views": 0,
  "content": "<p>First of all, thanks to the organizers for hosting this competition, and to my teammate <a href=\"https://www.kaggle.com/sega1031\" target=\"_blank\">@sega1031</a>.</p>\n<h2>Overview</h2>\n<ul>\n<li>Create many query candidates with minimal false positives.</li>\n<li>Select up to 25 candidates using solver, then concatenate using OR.</li>\n<li>Token count: you can reduce the token count of an AND query to one token (see below).</li>\n</ul>\n<h2>Validation Strategy</h2>\n<ul>\n<li>We use the published validation index.  <br>\n<a href=\"https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index\" target=\"_blank\">https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index</a></li>\n<li>The leaderboard score is slightly lower than the validation score but is well correlated.</li>\n</ul>\n<h2>The Metric and Basic Strategy</h2>\n<ul>\n<li>We believe assumption from <a href=\"https://www.kaggle.com/devinanzelmo\" target=\"_blank\">@devinanzelmo</a> is correct: <a href=\"https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/499981#2791642\" target=\"_blank\">https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/499981#2791642</a></li>\n<li>We confirm this through two submissions. The theoretical and actual leaderboard values are very close.<ol>\n<li>Select one patent from the 50 neighbors and build a query from first 20 words of its abstract with phrase search. This procedure gives a query with 20 tokens, TP1, and FP0 (= one true positive and zero false positive).</li>\n<li>Select two patents from the 50 neighbors and build a query from first 20 words of each abstract for a phrase search. The two queries are concatenated with OR. This results in 41 tokens, TP2, and FP0.</li></ol></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>exp</th>\n<th>TP</th>\n<th>FP</th>\n<th>Theoretical LB</th>\n<th>Actual LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>1</td>\n<td>0</td>\n<td>0.089</td>\n<td>0.08</td>\n</tr>\n<tr>\n<td>2</td>\n<td>2</td>\n<td>0</td>\n<td>0.159</td>\n<td>0.15</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>In this evaluation metric, it is crucial for TP to rank highly and FP NOT to rank highly.<ul>\n<li>If 10 TPs are followed by 40 FPs -&gt; score: 0.514</li>\n<li>If 40 FPs are followed by 10 TPs -&gt; score: 0.023</li></ul></li>\n<li>Therefore, our basic strategy is to collect as many TPs as possible while keeping FPs as close to zero as possible.</li>\n<li>This plot shows the score for N TPs followed by (50-N) FPs.<ul>\n<li>if 41 TPs are followed by 9 FPs -&gt; score: 0.980</li>\n<li>if 44 TPs are followed by 6 FPs -&gt; score: 0.991</li></ul></li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2304617%2Ff0d3b8a7adc696a2971955b617e550eb%2F__results___5_0.png?generation=1721866334202765&amp;alt=media\" alt=\"\"></p>\n<h2>Creating Candidates</h2>\n<h3>Global Counter Candidates</h3>\n<ul>\n<li>Preparation<ul>\n<li>Count how many times a specific word or n-gram appears in all given patents. For example, \"ti:device\" may appear 20,000 times in 200,000 patents.</li>\n<li>We call this \"global counter\".</li>\n<li>This global counter is prepared in advance.</li></ul></li>\n<li>Creating candidates<ul>\n<li>Check the global counter for all words and n-grams appearing in the 50 neighbors we want to retrieve.</li>\n<li>If \"ti:device\" appears in 3 out of the 50 neighbors, and the global counter's count is 4, then \"ti:device\" query results with TP3 and FP1.</li>\n<li>Create these candidates for ti, ab, clm, detd, and cpc, using unigrams, bigrams, and trigrams.</li></ul></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>candidates</th>\n<th>tp_cover</th>\n<th>tp</th>\n<th>global_count</th>\n<th>fp</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ti:device</td>\n<td>{0,1,2}</td>\n<td>len({0,1,2}) = 3</td>\n<td>4</td>\n<td>4 - 3 = 1</td>\n</tr>\n<tr>\n<td>clm:invention</td>\n<td>{0,2,3,4,5}</td>\n<td>5</td>\n<td>7</td>\n<td>2</td>\n</tr>\n<tr>\n<td>detd:method detd:device</td>\n<td>{6}</td>\n<td>1</td>\n<td>1</td>\n<td>0</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>tp_cover = {0,1,2} indicates that this query candidate can retrieve neighbors with indices 0, 1, and 2.</li>\n<li>Example:<ul>\n<li>Concatenate three candidates in this table with OR: <code>ti:device OR clm:invention OR (detd:method detd:device)</code>.</li>\n<li>Since they are concatenated with OR, tp_cover becomes {0,1,2,3,4,5,6}.  </li>\n<li>Therefore, this query will be 6 tokens, TP7, and FP3.</li></ul></li>\n</ul>\n<h3>50c2 Candidates</h3>\n<p>Select two patents from the 50 neighbors.</p>\n<table>\n<thead>\n<tr>\n<th>publication_number</th>\n<th>ti</th>\n<th>abst</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>US-0000-A</td>\n<td>dog cat fox</td>\n<td>pig cow</td>\n</tr>\n<tr>\n<td>US-0001-A</td>\n<td>cat fox</td>\n<td>duck frog cow</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>Extract the common words from these two patents and create an AND query.<ul>\n<li><code>ti:cat AND ti:fox AND ab:cow</code></li></ul></li>\n<li>This query will retrieve these two patents, but we need to calculate/estimate how many FPs this query will retrieve.</li>\n<li>We use IDF (inverse document frequency).<ul>\n<li>Since IDF is the logarithm of the inverse of the occurrence probability, summed-up IDF values give the occurrence probability of ANDed candidates.</li>\n<li>Calculate the IDF sum of the created query, and if it is above a certain threshold, estimate that FP is zero.</li>\n<li>We set this threshold to 80 using validation data. In this example, we are checking if idf(ti:cat) + idf(ti:fox) + idf(ab:cow) &gt; 80.</li></ul></li>\n<li>Calculate this for all combinations of 50 choose 2 (= 1225).<ul>\n<li>Each candidate must be 2 TPs and 0 FP.</li></ul></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>candidates</th>\n<th>tp_cover</th>\n<th>tp</th>\n<th>IDF_sum</th>\n<th>fp</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ti:cat ti:fox ab:cow</td>\n<td>{0,1}</td>\n<td>2</td>\n<td>103.24</td>\n<td>0</td>\n</tr>\n</tbody>\n</table>\n<h2>Token Count</h2>\n<pre><code> ():\n     ([i  i  re.split(, query)  i])    \n</code></pre>\n<ul>\n<li>The number of tokens is counted by this function,</li>\n<li>Through a particular approach to constructing the query, it is possible to keep the parse result the same while reducing the token count to one.</li>\n</ul>\n<h3>Example</h3>\n<p>Normal AND</p>\n<pre><code>query = \nnum_tokens = whoosh_utils.count_query_tokens(query)\n(, num_tokens)\nqp = whoosh_utils.get_query_parser()\nqp.parse(query)\n</code></pre>\n<blockquote>\n  <p>num_tokens: 2  <br>\n  And([Term('ti', 'dog'), Term('ti', 'cat')])</p>\n</blockquote>\n<p>Special AND</p>\n<pre><code>query = \n</code></pre>\n<blockquote>\n  <p>num_tokens: 1  <br>\n  And([Term('ti', 'dog'), Term('ti', 'cat')])</p>\n</blockquote>\n<p>Normal phrase</p>\n<pre><code>query = \n</code></pre>\n<blockquote>\n  <p>num_tokens: 9  <br>\n  Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)</p>\n</blockquote>\n<p>Special phrase</p>\n<pre><code>query = \n</code></pre>\n<blockquote>\n  <p>num_tokens: 1  <br>\n  Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)</p>\n</blockquote>\n<h3>Note</h3>\n<ul>\n<li>We would like to apply this technique to OR as well, but we couldn't find a way.</li>\n<li>Even without this technique, we were able to achieve a LB score of 0.85.</li>\n</ul>\n<h2>Set Cover Problem</h2>\n<ul>\n<li>Up to this point, we have many query candidates created by global counter and 50c2.</li>\n<li>For example, the following candidates may have been obtained.</li>\n<li>Note: With the above token count technique, each token count is always 1.</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>candidates</th>\n<th>tp_covers</th>\n<th>FP</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ti:device</td>\n<td>{0, 1, 2, 3, 4, 5, 6}</td>\n<td>3</td>\n</tr>\n<tr>\n<td>ti:\"dog\"ti:\"cat\"</td>\n<td>{0, 1, 2, 7}</td>\n<td>1</td>\n</tr>\n<tr>\n<td>clm:\"dissociation\"ab:\"proteins\"</td>\n<td>{3, 4, 10}</td>\n<td>0</td>\n</tr>\n<tr>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>Selecting some of these candidates and concatenate them with OR optimally is difficult, because this is a <a href=\"https://en.wikipedia.org/wiki/Set_cover_problem\" target=\"_blank\">set cover problem</a>, which is known to be NP-hard.</li>\n<li>We solved it as a integer linear programming problem using a python solver.</li>\n<li>Maximize TP and minimize FP with a token count of 50 or less. In practice, we maximized <code>10xTP − FP</code>.</li>\n<li>It can also be solved using some kind of greedy algorithm, but the solver provided a slightly better score (+0.01).</li>\n</ul>\n<h2>Query Example</h2>\n<p></p>Here is a query example with 49 token.<p></p>\n<blockquote>\n  <p>(cpc:\"B60P1/165\"cpc:\"E02F3/3483\"cpc:\"B60P1/165\"cpc:\"E02F3/3486\") OR (cpc:\"B60D1/26\"cpc:\"B62D49/04\"cpc:\"B60D1/26\"cpc:\"E02F3/6472\"cpc:\"B60D1/26\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F3/64\"cpc:\"B62D49/04\"cpc:\"E02F3/6472\"cpc:\"B62D49/04\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F9/2016\"cpc:\"E02F3/6472\"cpc:\"E02F9/2016\"cpc:\"E02F3/655\"cpc:\"E02F9/2016\"detd:\"courli\"detd:\"lgayea\") OR (cpc:\"B65F2003/0283\"cpc:\"E02F3/3486\"cpc:\"B65F3/046\"cpc:\"E02F3/3486\"ti:\"end@loader@actuating\"ti:\"loader@actuating\"ti:\"loader@actuating@mechanism\"ti:\"actuating@mechanism@dump\"cpc:\"B65F2003/0283\"cpc:\"B65F3/046\"detd:\"horbe\") OR (detd:\"wharfpthe\"ti:\"dumping@scoop\"ti:\"side@dumping@scoop\"detd:\"hatchof\") OR (detd:\"powerswung\") OR (detd:\"shlppee\") OR (detd:\"0rneypearce\"detd:\"schaepcrklaus\"ti:\"as@asphalt@or\"ti:\"improvement@therein@laying\"ti:\"laying@surfacing\"ti:\"laying@surfacing@material\"ti:\"machine@and@improvement\"ti:\"surfacing@material@as\"ti:\"therein@laying\"ti:\"therein@laying@surfacing\"detd:\"comprlsesr\") OR (detd:\"disswingable\"detd:\"rdlyonto\"detd:\"sitionedsymmetrically\"detd:\"understandingflof\"detd:\"opjusting\") OR (ti:\"mechanical@shoveling@machine\"ti:\"mechanical@shoveling\") OR ((detd:\"tractors\"detd:\"ravity\"detd:\"scrapers\"detd:\"kick\"cpc:\"E02F3/6472\"ti:\"scraper\"cpc:\"E02F3/656\"detd:\"apron\"detd:\"scraper\"detd:\"hingedly\"detd:\"tractor\"detd:\"sheave\"detd:\"sheaves\"detd:\"cooperates\"detd:\"dead\"detd:\"axle\")) OR ((detd:\"oor\"detd:\"exible\"detd:\"retracting\"detd:\"trough\"detd:\"retracted\"detd:\"turntable\"detd:\"underground\"detd:\"jacks\"detd:\"therealong\"detd:\"rectilinear\"detd:\"propelling\"detd:\"conveyor\"detd:\"extensible\"detd:\"slip\"detd:\"clutch\"detd:\"mines\"detd:\"adjustably\"detd:\"engageable\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"shovelling\"detd:\"tom\"detd:\"cushioned\"detd:\"compel\"detd:\"swiveled\"detd:\"abruptly\"detd:\"swivelly\"detd:\"wardly\"detd:\"hose\"detd:\"guideway\"detd:\"swivelled\"detd:\"undesired\"detd:\"sup\"detd:\"ported\"detd:\"swivel\"detd:\"osgood\"detd:\"pile\"detd:\"muck\"detd:\"guideways\"detd:\"dig\")) OR ((detd:\"planetaries\"detd:\"payed\"detd:\"planetary\"detd:\"mosier\"detd:\"fiexible\"detd:\"2li\"detd:\"conveyer\"detd:\"tractive\"detd:\"lil\"detd:\"ior\"detd:\"exible\"detd:\"2s\"detd:\"yieldable\"detd:\"tunnels\"detd:\"compensating\"detd:\"excepting\"detd:\"lll\"detd:\"chute\"detd:\"illinois\"detd:\"geared\")) OR ((detd:\"fioor\"detd:\"simmons\"detd:\"overlie\"detd:\"hydraulically\"detd:\"attachable\"detd:\"swivelly\"detd:\"coal\"detd:\"jacks\"detd:\"swivel\"detd:\"anchor\"detd:\"joy\"detd:\"guideways\"detd:\"unison\"detd:\"propelling\"detd:\"pennsylvania\"detd:\"mining\"detd:\"extensible\"detd:\"transporting\"detd:\"tilted\"detd:\"elevating\")) OR ((detd:\"vfiled\"detd:\"communicable\"detd:\"eiect\"detd:\"hydraulically\"detd:\"sullivan\"ti:\"material\"detd:\"propulsion\"detd:\"machinery\"detd:\"claremont\"detd:\"bores\"detd:\"uid\"detd:\"pile\"detd:\"muck\"detd:\"lin\"detd:\"dig\"detd:\"trackway\"detd:\"conduits\"detd:\"5i\"detd:\"massachusetts\"detd:\"hydraulic\")) OR ((detd:\"i23\"detd:\"movementof\"detd:\"i26\"detd:\"insides\"detd:\"encircles\"detd:\"compactness\"detd:\"h2\"detd:\"cushioning\"detd:\"yieldably\"detd:\"reversely\"detd:\"pivoting\"detd:\"cams\"detd:\"mucking\"detd:\"abuts\"detd:\"lug\"detd:\"therealong\"detd:\"teeth\"detd:\"axles\"detd:\"scoop\"detd:\"nuts\")) OR ((detd:\"foolproof\"detd:\"selfcentering\"detd:\"impetus\"detd:\"i85\"detd:\"coaction\"detd:\"rockers\"detd:\"pinned\"detd:\"maxson\"detd:\"i00\"detd:\"pivotable\"detd:\"i05\"detd:\"fork\"detd:\"inadequate\"detd:\"dipper\"detd:\"compelling\"detd:\"plungers\"detd:\"incapable\"detd:\"h5\"detd:\"abrupt\"detd:\"tang\")) OR ((detd:\"evidently\"detd:\"shank\"detd:\"receivable\"detd:\"hoses\"detd:\"venting\"detd:\"urges\"detd:\"interrupting\"detd:\"vented\"detd:\"hose\"detd:\"rolls\"detd:\"inactive\"detd:\"claremont\"detd:\"interrupted\"detd:\"3i\"detd:\"joy\"detd:\"guideways\"detd:\"trackway\"detd:\"pennsylvania\"detd:\"embodies\"detd:\"assumes\")) OR ((detd:\"seam\"detd:\"bevel\"detd:\"brake\"detd:\"rocked\"detd:\"wheeled\"detd:\"propulsion\"detd:\"spur\"detd:\"rst\"detd:\"coal\"detd:\"pinion\"detd:\"withdrawal\"detd:\"gearing\"detd:\"shafts\"detd:\"extensible\"detd:\"clutch\"detd:\"keyed\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"draulic\"detd:\"hy\"detd:\"4s\"detd:\"vfor\"detd:\"oor\"detd:\"zontal\"detd:\"withdrawing\"detd:\"ie\"detd:\"andv\"detd:\"progresses\"detd:\"mw\"detd:\"lo\"detd:\"anchor\"detd:\"teeth\"detd:\"conveyor\"detd:\"extremity\"detd:\"3l\"detd:\"hydraulic\")) OR ((detd:\"isv\"detd:\"isprovided\"detd:\"sion\"detd:\"ropes\"detd:\"rope\"detd:\"suddenly\"detd:\"vof\"detd:\"relied\"detd:\"thel\"detd:\"urge\"detd:\"anda\"detd:\"4l\"detd:\"0f\"detd:\"gearing\"detd:\"anchored\"detd:\"transportation\"detd:\"mining\"detd:\"lthe\"detd:\"transporting\")) OR ((detd:\"posltion\"detd:\"motive\"detd:\"unwind\"detd:\"drifts\"detd:\"rearmost\"detd:\"tunnels\"detd:\"cated\"detd:\"segmental\"detd:\"injury\"detd:\"imparting\"detd:\"pile\"detd:\"ward\"detd:\"ap\"detd:\"scoop\"detd:\"swings\"detd:\"mines\"detd:\"reversible\")) OR ((detd:\"adaptedv\"detd:\"grooved\"detd:\"ore\"detd:\"chine\"detd:\"vhen\"detd:\"t0\"detd:\"opera\"detd:\"vand\"detd:\"sidewise\"detd:\"wheeled\"detd:\"thev\"detd:\"meshes\"detd:\"andv\"cpc:\"E02F9/022\"detd:\"movably\"detd:\"idler\"detd:\"coal\"detd:\"pinion\"detd:\"casting\"detd:\"bears\")) OR ((detd:\"compel\"detd:\"abruptly\"detd:\"tensioned\"detd:\"swivelled\"detd:\"coincident\"detd:\"assured\"detd:\"turntable\"detd:\"claremont\"detd:\"swivel\"detd:\"osgood\"detd:\"fulcrum\"detd:\"loader\"detd:\"guideways\"cpc:\"E02F3/3486\"detd:\"trackway\"detd:\"propelling\"detd:\"rolling\"detd:\"assumes\"detd:\"alinement\"detd:\"swings\")) OR ((detd:\"hoists\"cpc:\"E02F3/657\"detd:\"hoist\"cpc:\"E02F3/656\"detd:\"medial\"detd:\"bumper\"detd:\"trunnions\"detd:\"trunnion\"detd:\"grading\"detd:\"brake\"detd:\"steering\"detd:\"spreading\"detd:\"tractor\"detd:\"propulsion\"detd:\"coacting\"detd:\"extremities\"detd:\"gearing\"detd:\"rail\"detd:\"dump\"detd:\"clutch\"))</p>\n</blockquote>\n<hr>\n<p>EDIT: 2024/08/06</p>\n<p>We pulished our codes.</p>\n<ul>\n<li>with magic<ul>\n<li><a href=\"https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-with-magic?scriptVersionId=190687232\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-with-magic?scriptVersionId=190687232</a></li></ul></li>\n<li>without magic<ul>\n<li><a href=\"https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-without-magic?scriptVersionId=190688084\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-without-magic?scriptVersionId=190688084</a></li></ul></li>\n<li>global counter(title)<ul>\n<li><a href=\"https://www.kaggle.com/code/sega1031/uspto-global-title-word-counter\" target=\"_blank\">https://www.kaggle.com/code/sega1031/uspto-global-title-word-counter</a></li></ul></li>\n<li>global counter as one <ul>\n<li><a href=\"https://www.kaggle.com/code/sega1031/uspto-global-counters-limit30\" target=\"_blank\">https://www.kaggle.com/code/sega1031/uspto-global-counters-limit30</a></li></ul></li>\n</ul>",
  "messages": [
    {
      "id": "2935058",
      "postDate": "07/25/2024 00:13:46",
      "content": "<p>First of all, thanks to the organizers for hosting this competition, and to my teammate <a href=\"https://www.kaggle.com/sega1031\" target=\"_blank\">@sega1031</a>.</p>\n<h2>Overview</h2>\n<ul>\n<li>Create many query candidates with minimal false positives.</li>\n<li>Select up to 25 candidates using solver, then concatenate using OR.</li>\n<li>Token count: you can reduce the token count of an AND query to one token (see below).</li>\n</ul>\n<h2>Validation Strategy</h2>\n<ul>\n<li>We use the published validation index.  <br>\n<a href=\"https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index\" target=\"_blank\">https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index</a></li>\n<li>The leaderboard score is slightly lower than the validation score but is well correlated.</li>\n</ul>\n<h2>The Metric and Basic Strategy</h2>\n<ul>\n<li>We believe assumption from <a href=\"https://www.kaggle.com/devinanzelmo\" target=\"_blank\">@devinanzelmo</a> is correct: <a href=\"https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/499981#2791642\" target=\"_blank\">https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/499981#2791642</a></li>\n<li>We confirm this through two submissions. The theoretical and actual leaderboard values are very close.<ol>\n<li>Select one patent from the 50 neighbors and build a query from first 20 words of its abstract with phrase search. This procedure gives a query with 20 tokens, TP1, and FP0 (= one true positive and zero false positive).</li>\n<li>Select two patents from the 50 neighbors and build a query from first 20 words of each abstract for a phrase search. The two queries are concatenated with OR. This results in 41 tokens, TP2, and FP0.</li></ol></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>exp</th>\n<th>TP</th>\n<th>FP</th>\n<th>Theoretical LB</th>\n<th>Actual LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>1</td>\n<td>0</td>\n<td>0.089</td>\n<td>0.08</td>\n</tr>\n<tr>\n<td>2</td>\n<td>2</td>\n<td>0</td>\n<td>0.159</td>\n<td>0.15</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>In this evaluation metric, it is crucial for TP to rank highly and FP NOT to rank highly.<ul>\n<li>If 10 TPs are followed by 40 FPs -&gt; score: 0.514</li>\n<li>If 40 FPs are followed by 10 TPs -&gt; score: 0.023</li></ul></li>\n<li>Therefore, our basic strategy is to collect as many TPs as possible while keeping FPs as close to zero as possible.</li>\n<li>This plot shows the score for N TPs followed by (50-N) FPs.<ul>\n<li>if 41 TPs are followed by 9 FPs -&gt; score: 0.980</li>\n<li>if 44 TPs are followed by 6 FPs -&gt; score: 0.991</li></ul></li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2304617%2Ff0d3b8a7adc696a2971955b617e550eb%2F__results___5_0.png?generation=1721866334202765&amp;alt=media\" alt=\"\"></p>\n<h2>Creating Candidates</h2>\n<h3>Global Counter Candidates</h3>\n<ul>\n<li>Preparation<ul>\n<li>Count how many times a specific word or n-gram appears in all given patents. For example, \"ti:device\" may appear 20,000 times in 200,000 patents.</li>\n<li>We call this \"global counter\".</li>\n<li>This global counter is prepared in advance.</li></ul></li>\n<li>Creating candidates<ul>\n<li>Check the global counter for all words and n-grams appearing in the 50 neighbors we want to retrieve.</li>\n<li>If \"ti:device\" appears in 3 out of the 50 neighbors, and the global counter's count is 4, then \"ti:device\" query results with TP3 and FP1.</li>\n<li>Create these candidates for ti, ab, clm, detd, and cpc, using unigrams, bigrams, and trigrams.</li></ul></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>candidates</th>\n<th>tp_cover</th>\n<th>tp</th>\n<th>global_count</th>\n<th>fp</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ti:device</td>\n<td>{0,1,2}</td>\n<td>len({0,1,2}) = 3</td>\n<td>4</td>\n<td>4 - 3 = 1</td>\n</tr>\n<tr>\n<td>clm:invention</td>\n<td>{0,2,3,4,5}</td>\n<td>5</td>\n<td>7</td>\n<td>2</td>\n</tr>\n<tr>\n<td>detd:method detd:device</td>\n<td>{6}</td>\n<td>1</td>\n<td>1</td>\n<td>0</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>tp_cover = {0,1,2} indicates that this query candidate can retrieve neighbors with indices 0, 1, and 2.</li>\n<li>Example:<ul>\n<li>Concatenate three candidates in this table with OR: <code>ti:device OR clm:invention OR (detd:method detd:device)</code>.</li>\n<li>Since they are concatenated with OR, tp_cover becomes {0,1,2,3,4,5,6}.  </li>\n<li>Therefore, this query will be 6 tokens, TP7, and FP3.</li></ul></li>\n</ul>\n<h3>50c2 Candidates</h3>\n<p>Select two patents from the 50 neighbors.</p>\n<table>\n<thead>\n<tr>\n<th>publication_number</th>\n<th>ti</th>\n<th>abst</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>US-0000-A</td>\n<td>dog cat fox</td>\n<td>pig cow</td>\n</tr>\n<tr>\n<td>US-0001-A</td>\n<td>cat fox</td>\n<td>duck frog cow</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>Extract the common words from these two patents and create an AND query.<ul>\n<li><code>ti:cat AND ti:fox AND ab:cow</code></li></ul></li>\n<li>This query will retrieve these two patents, but we need to calculate/estimate how many FPs this query will retrieve.</li>\n<li>We use IDF (inverse document frequency).<ul>\n<li>Since IDF is the logarithm of the inverse of the occurrence probability, summed-up IDF values give the occurrence probability of ANDed candidates.</li>\n<li>Calculate the IDF sum of the created query, and if it is above a certain threshold, estimate that FP is zero.</li>\n<li>We set this threshold to 80 using validation data. In this example, we are checking if idf(ti:cat) + idf(ti:fox) + idf(ab:cow) &gt; 80.</li></ul></li>\n<li>Calculate this for all combinations of 50 choose 2 (= 1225).<ul>\n<li>Each candidate must be 2 TPs and 0 FP.</li></ul></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>candidates</th>\n<th>tp_cover</th>\n<th>tp</th>\n<th>IDF_sum</th>\n<th>fp</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ti:cat ti:fox ab:cow</td>\n<td>{0,1}</td>\n<td>2</td>\n<td>103.24</td>\n<td>0</td>\n</tr>\n</tbody>\n</table>\n<h2>Token Count</h2>\n<pre><code> ():\n     ([i  i  re.split(, query)  i])    \n</code></pre>\n<ul>\n<li>The number of tokens is counted by this function,</li>\n<li>Through a particular approach to constructing the query, it is possible to keep the parse result the same while reducing the token count to one.</li>\n</ul>\n<h3>Example</h3>\n<p>Normal AND</p>\n<pre><code>query = \nnum_tokens = whoosh_utils.count_query_tokens(query)\n(, num_tokens)\nqp = whoosh_utils.get_query_parser()\nqp.parse(query)\n</code></pre>\n<blockquote>\n  <p>num_tokens: 2  <br>\n  And([Term('ti', 'dog'), Term('ti', 'cat')])</p>\n</blockquote>\n<p>Special AND</p>\n<pre><code>query = \n</code></pre>\n<blockquote>\n  <p>num_tokens: 1  <br>\n  And([Term('ti', 'dog'), Term('ti', 'cat')])</p>\n</blockquote>\n<p>Normal phrase</p>\n<pre><code>query = \n</code></pre>\n<blockquote>\n  <p>num_tokens: 9  <br>\n  Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)</p>\n</blockquote>\n<p>Special phrase</p>\n<pre><code>query = \n</code></pre>\n<blockquote>\n  <p>num_tokens: 1  <br>\n  Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)</p>\n</blockquote>\n<h3>Note</h3>\n<ul>\n<li>We would like to apply this technique to OR as well, but we couldn't find a way.</li>\n<li>Even without this technique, we were able to achieve a LB score of 0.85.</li>\n</ul>\n<h2>Set Cover Problem</h2>\n<ul>\n<li>Up to this point, we have many query candidates created by global counter and 50c2.</li>\n<li>For example, the following candidates may have been obtained.</li>\n<li>Note: With the above token count technique, each token count is always 1.</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>candidates</th>\n<th>tp_covers</th>\n<th>FP</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>ti:device</td>\n<td>{0, 1, 2, 3, 4, 5, 6}</td>\n<td>3</td>\n</tr>\n<tr>\n<td>ti:\"dog\"ti:\"cat\"</td>\n<td>{0, 1, 2, 7}</td>\n<td>1</td>\n</tr>\n<tr>\n<td>clm:\"dissociation\"ab:\"proteins\"</td>\n<td>{3, 4, 10}</td>\n<td>0</td>\n</tr>\n<tr>\n<td>…</td>\n<td>…</td>\n<td>…</td>\n</tr>\n</tbody>\n</table>\n<ul>\n<li>Selecting some of these candidates and concatenate them with OR optimally is difficult, because this is a <a href=\"https://en.wikipedia.org/wiki/Set_cover_problem\" target=\"_blank\">set cover problem</a>, which is known to be NP-hard.</li>\n<li>We solved it as a integer linear programming problem using a python solver.</li>\n<li>Maximize TP and minimize FP with a token count of 50 or less. In practice, we maximized <code>10xTP − FP</code>.</li>\n<li>It can also be solved using some kind of greedy algorithm, but the solver provided a slightly better score (+0.01).</li>\n</ul>\n<h2>Query Example</h2>\n<p></p>Here is a query example with 49 token.<p></p>\n<blockquote>\n  <p>(cpc:\"B60P1/165\"cpc:\"E02F3/3483\"cpc:\"B60P1/165\"cpc:\"E02F3/3486\") OR (cpc:\"B60D1/26\"cpc:\"B62D49/04\"cpc:\"B60D1/26\"cpc:\"E02F3/6472\"cpc:\"B60D1/26\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F3/64\"cpc:\"B62D49/04\"cpc:\"E02F3/6472\"cpc:\"B62D49/04\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F9/2016\"cpc:\"E02F3/6472\"cpc:\"E02F9/2016\"cpc:\"E02F3/655\"cpc:\"E02F9/2016\"detd:\"courli\"detd:\"lgayea\") OR (cpc:\"B65F2003/0283\"cpc:\"E02F3/3486\"cpc:\"B65F3/046\"cpc:\"E02F3/3486\"ti:\"end@loader@actuating\"ti:\"loader@actuating\"ti:\"loader@actuating@mechanism\"ti:\"actuating@mechanism@dump\"cpc:\"B65F2003/0283\"cpc:\"B65F3/046\"detd:\"horbe\") OR (detd:\"wharfpthe\"ti:\"dumping@scoop\"ti:\"side@dumping@scoop\"detd:\"hatchof\") OR (detd:\"powerswung\") OR (detd:\"shlppee\") OR (detd:\"0rneypearce\"detd:\"schaepcrklaus\"ti:\"as@asphalt@or\"ti:\"improvement@therein@laying\"ti:\"laying@surfacing\"ti:\"laying@surfacing@material\"ti:\"machine@and@improvement\"ti:\"surfacing@material@as\"ti:\"therein@laying\"ti:\"therein@laying@surfacing\"detd:\"comprlsesr\") OR (detd:\"disswingable\"detd:\"rdlyonto\"detd:\"sitionedsymmetrically\"detd:\"understandingflof\"detd:\"opjusting\") OR (ti:\"mechanical@shoveling@machine\"ti:\"mechanical@shoveling\") OR ((detd:\"tractors\"detd:\"ravity\"detd:\"scrapers\"detd:\"kick\"cpc:\"E02F3/6472\"ti:\"scraper\"cpc:\"E02F3/656\"detd:\"apron\"detd:\"scraper\"detd:\"hingedly\"detd:\"tractor\"detd:\"sheave\"detd:\"sheaves\"detd:\"cooperates\"detd:\"dead\"detd:\"axle\")) OR ((detd:\"oor\"detd:\"exible\"detd:\"retracting\"detd:\"trough\"detd:\"retracted\"detd:\"turntable\"detd:\"underground\"detd:\"jacks\"detd:\"therealong\"detd:\"rectilinear\"detd:\"propelling\"detd:\"conveyor\"detd:\"extensible\"detd:\"slip\"detd:\"clutch\"detd:\"mines\"detd:\"adjustably\"detd:\"engageable\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"shovelling\"detd:\"tom\"detd:\"cushioned\"detd:\"compel\"detd:\"swiveled\"detd:\"abruptly\"detd:\"swivelly\"detd:\"wardly\"detd:\"hose\"detd:\"guideway\"detd:\"swivelled\"detd:\"undesired\"detd:\"sup\"detd:\"ported\"detd:\"swivel\"detd:\"osgood\"detd:\"pile\"detd:\"muck\"detd:\"guideways\"detd:\"dig\")) OR ((detd:\"planetaries\"detd:\"payed\"detd:\"planetary\"detd:\"mosier\"detd:\"fiexible\"detd:\"2li\"detd:\"conveyer\"detd:\"tractive\"detd:\"lil\"detd:\"ior\"detd:\"exible\"detd:\"2s\"detd:\"yieldable\"detd:\"tunnels\"detd:\"compensating\"detd:\"excepting\"detd:\"lll\"detd:\"chute\"detd:\"illinois\"detd:\"geared\")) OR ((detd:\"fioor\"detd:\"simmons\"detd:\"overlie\"detd:\"hydraulically\"detd:\"attachable\"detd:\"swivelly\"detd:\"coal\"detd:\"jacks\"detd:\"swivel\"detd:\"anchor\"detd:\"joy\"detd:\"guideways\"detd:\"unison\"detd:\"propelling\"detd:\"pennsylvania\"detd:\"mining\"detd:\"extensible\"detd:\"transporting\"detd:\"tilted\"detd:\"elevating\")) OR ((detd:\"vfiled\"detd:\"communicable\"detd:\"eiect\"detd:\"hydraulically\"detd:\"sullivan\"ti:\"material\"detd:\"propulsion\"detd:\"machinery\"detd:\"claremont\"detd:\"bores\"detd:\"uid\"detd:\"pile\"detd:\"muck\"detd:\"lin\"detd:\"dig\"detd:\"trackway\"detd:\"conduits\"detd:\"5i\"detd:\"massachusetts\"detd:\"hydraulic\")) OR ((detd:\"i23\"detd:\"movementof\"detd:\"i26\"detd:\"insides\"detd:\"encircles\"detd:\"compactness\"detd:\"h2\"detd:\"cushioning\"detd:\"yieldably\"detd:\"reversely\"detd:\"pivoting\"detd:\"cams\"detd:\"mucking\"detd:\"abuts\"detd:\"lug\"detd:\"therealong\"detd:\"teeth\"detd:\"axles\"detd:\"scoop\"detd:\"nuts\")) OR ((detd:\"foolproof\"detd:\"selfcentering\"detd:\"impetus\"detd:\"i85\"detd:\"coaction\"detd:\"rockers\"detd:\"pinned\"detd:\"maxson\"detd:\"i00\"detd:\"pivotable\"detd:\"i05\"detd:\"fork\"detd:\"inadequate\"detd:\"dipper\"detd:\"compelling\"detd:\"plungers\"detd:\"incapable\"detd:\"h5\"detd:\"abrupt\"detd:\"tang\")) OR ((detd:\"evidently\"detd:\"shank\"detd:\"receivable\"detd:\"hoses\"detd:\"venting\"detd:\"urges\"detd:\"interrupting\"detd:\"vented\"detd:\"hose\"detd:\"rolls\"detd:\"inactive\"detd:\"claremont\"detd:\"interrupted\"detd:\"3i\"detd:\"joy\"detd:\"guideways\"detd:\"trackway\"detd:\"pennsylvania\"detd:\"embodies\"detd:\"assumes\")) OR ((detd:\"seam\"detd:\"bevel\"detd:\"brake\"detd:\"rocked\"detd:\"wheeled\"detd:\"propulsion\"detd:\"spur\"detd:\"rst\"detd:\"coal\"detd:\"pinion\"detd:\"withdrawal\"detd:\"gearing\"detd:\"shafts\"detd:\"extensible\"detd:\"clutch\"detd:\"keyed\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"draulic\"detd:\"hy\"detd:\"4s\"detd:\"vfor\"detd:\"oor\"detd:\"zontal\"detd:\"withdrawing\"detd:\"ie\"detd:\"andv\"detd:\"progresses\"detd:\"mw\"detd:\"lo\"detd:\"anchor\"detd:\"teeth\"detd:\"conveyor\"detd:\"extremity\"detd:\"3l\"detd:\"hydraulic\")) OR ((detd:\"isv\"detd:\"isprovided\"detd:\"sion\"detd:\"ropes\"detd:\"rope\"detd:\"suddenly\"detd:\"vof\"detd:\"relied\"detd:\"thel\"detd:\"urge\"detd:\"anda\"detd:\"4l\"detd:\"0f\"detd:\"gearing\"detd:\"anchored\"detd:\"transportation\"detd:\"mining\"detd:\"lthe\"detd:\"transporting\")) OR ((detd:\"posltion\"detd:\"motive\"detd:\"unwind\"detd:\"drifts\"detd:\"rearmost\"detd:\"tunnels\"detd:\"cated\"detd:\"segmental\"detd:\"injury\"detd:\"imparting\"detd:\"pile\"detd:\"ward\"detd:\"ap\"detd:\"scoop\"detd:\"swings\"detd:\"mines\"detd:\"reversible\")) OR ((detd:\"adaptedv\"detd:\"grooved\"detd:\"ore\"detd:\"chine\"detd:\"vhen\"detd:\"t0\"detd:\"opera\"detd:\"vand\"detd:\"sidewise\"detd:\"wheeled\"detd:\"thev\"detd:\"meshes\"detd:\"andv\"cpc:\"E02F9/022\"detd:\"movably\"detd:\"idler\"detd:\"coal\"detd:\"pinion\"detd:\"casting\"detd:\"bears\")) OR ((detd:\"compel\"detd:\"abruptly\"detd:\"tensioned\"detd:\"swivelled\"detd:\"coincident\"detd:\"assured\"detd:\"turntable\"detd:\"claremont\"detd:\"swivel\"detd:\"osgood\"detd:\"fulcrum\"detd:\"loader\"detd:\"guideways\"cpc:\"E02F3/3486\"detd:\"trackway\"detd:\"propelling\"detd:\"rolling\"detd:\"assumes\"detd:\"alinement\"detd:\"swings\")) OR ((detd:\"hoists\"cpc:\"E02F3/657\"detd:\"hoist\"cpc:\"E02F3/656\"detd:\"medial\"detd:\"bumper\"detd:\"trunnions\"detd:\"trunnion\"detd:\"grading\"detd:\"brake\"detd:\"steering\"detd:\"spreading\"detd:\"tractor\"detd:\"propulsion\"detd:\"coacting\"detd:\"extremities\"detd:\"gearing\"detd:\"rail\"detd:\"dump\"detd:\"clutch\"))</p>\n</blockquote>\n<hr>\n<p>EDIT: 2024/08/06</p>\n<p>We pulished our codes.</p>\n<ul>\n<li>with magic<ul>\n<li><a href=\"https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-with-magic?scriptVersionId=190687232\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-with-magic?scriptVersionId=190687232</a></li></ul></li>\n<li>without magic<ul>\n<li><a href=\"https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-without-magic?scriptVersionId=190688084\" target=\"_blank\">https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-without-magic?scriptVersionId=190688084</a></li></ul></li>\n<li>global counter(title)<ul>\n<li><a href=\"https://www.kaggle.com/code/sega1031/uspto-global-title-word-counter\" target=\"_blank\">https://www.kaggle.com/code/sega1031/uspto-global-title-word-counter</a></li></ul></li>\n<li>global counter as one <ul>\n<li><a href=\"https://www.kaggle.com/code/sega1031/uspto-global-counters-limit30\" target=\"_blank\">https://www.kaggle.com/code/sega1031/uspto-global-counters-limit30</a></li></ul></li>\n</ul>",
      "rawMarkdown": "First of all, thanks to the organizers for hosting this competition, and to my teammate @sega1031.\n\n## Overview\n* Create many query candidates with minimal false positives.\n* Select up to 25 candidates using solver, then concatenate using OR.\n* Token count: you can reduce the token count of an AND query to one token (see below).\n\n## Validation Strategy\n* We use the published validation index.  \n  https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index\n* The leaderboard score is slightly lower than the validation score but is well correlated.\n\n## The Metric and Basic Strategy\n* We believe assumption from @devinanzelmo is correct: https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/499981#2791642\n* We confirm this through two submissions. The theoretical and actual leaderboard values are very close.\n  1. Select one patent from the 50 neighbors and build a query from first 20 words of its abstract with phrase search. This procedure gives a query with 20 tokens, TP1, and FP0 (= one true positive and zero false positive).\n  2. Select two patents from the 50 neighbors and build a query from first 20 words of each abstract for a phrase search. The two queries are concatenated with OR. This results in 41 tokens, TP2, and FP0.\n\n| exp | TP | FP | Theoretical LB | Actual LB |\n|-----|----|----|----------------|-----------|\n|  1  |  1 |  0 |      0.089     |    0.08   |\n|  2  |  2 |  0 |      0.159     |    0.15   |\n\n* In this evaluation metric, it is crucial for TP to rank highly and FP NOT to rank highly.\n  * If 10 TPs are followed by 40 FPs -> score: 0.514\n  * If 40 FPs are followed by 10 TPs -> score: 0.023\n* Therefore, our basic strategy is to collect as many TPs as possible while keeping FPs as close to zero as possible.\n* This plot shows the score for N TPs followed by (50-N) FPs.\n  * if 41 TPs are followed by 9 FPs -> score: 0.980\n  * if 44 TPs are followed by 6 FPs -> score: 0.991\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2304617%2Ff0d3b8a7adc696a2971955b617e550eb%2F__results___5_0.png?generation=1721866334202765&alt=media)\n\n## Creating Candidates\n### Global Counter Candidates\n* Preparation\n  * Count how many times a specific word or n-gram appears in all given patents. For example, \"ti:device\" may appear 20,000 times in 200,000 patents.\n  * We call this \"global counter\".\n  * This global counter is prepared in advance.\n* Creating candidates\n  * Check the global counter for all words and n-grams appearing in the 50 neighbors we want to retrieve.\n  * If \"ti:device\" appears in 3 out of the 50 neighbors, and the global counter's count is 4, then \"ti:device\" query results with TP3 and FP1.\n  * Create these candidates for ti, ab, clm, detd, and cpc, using unigrams, bigrams, and trigrams.\n\n| candidates    | tp_cover     | tp                   | global_count | fp         |\n|------------|--------------|----------------------|--------------|------------|\n| ti:device  | {0,1,2}      | len({0,1,2}) = 3     | 4            | 4 - 3 = 1  |\n| clm:invention    | {0,2,3,4,5}  | 5                    | 7            | 2          |\n| detd:method detd:device   | {6}          | 1                    | 1            | 0          |\n\n\n* tp_cover = {0,1,2} indicates that this query candidate can retrieve neighbors with indices 0, 1, and 2.\n* Example:\n  * Concatenate three candidates in this table with OR: `ti:device OR clm:invention OR (detd:method detd:device)`.\n  * Since they are concatenated with OR, tp_cover becomes {0,1,2,3,4,5,6}.  \n  * Therefore, this query will be 6 tokens, TP7, and FP3.\n\n### 50c2 Candidates\n\nSelect two patents from the 50 neighbors.\n\n| publication_number | ti         | abst            |\n|--------|---------------|-----------------|\n| US-0000-A    | dog cat fox   | pig cow         |\n| US-0001-A    | cat fox       | duck frog cow   |\n\n* Extract the common words from these two patents and create an AND query.\n  * `ti:cat AND ti:fox AND ab:cow`\n* This query will retrieve these two patents, but we need to calculate/estimate how many FPs this query will retrieve.\n* We use IDF (inverse document frequency).\n  * Since IDF is the logarithm of the inverse of the occurrence probability, summed-up IDF values give the occurrence probability of ANDed candidates.\n  * Calculate the IDF sum of the created query, and if it is above a certain threshold, estimate that FP is zero.\n  * We set this threshold to 80 using validation data. In this example, we are checking if idf(ti:cat) + idf(ti:fox) + idf(ab:cow) > 80.\n* Calculate this for all combinations of 50 choose 2 (= 1225).\n  * Each candidate must be 2 TPs and 0 FP.\n\n| candidates    | tp_cover     | tp                   |  IDF_sum | fp         |\n|------------|--------------|----------------------|------------|------------|\n| ti:cat ti:fox ab:cow  | {0,1}      |  2            | 103.24 |0  |\n\n\n## Token Count\n```python\ndef count_query_tokens(query: str):\n    return len([i for i in re.split('[\\s+()]', query) if i])\t\n```\n\n* The number of tokens is counted by this function,\n* Through a particular approach to constructing the query, it is possible to keep the parse result the same while reducing the token count to one.\n\n### Example\nNormal AND\n```python\nquery = \"ti:dog ti:cat\"\nnum_tokens = whoosh_utils.count_query_tokens(query)\nprint(\"num_tokens:\", num_tokens)\nqp = whoosh_utils.get_query_parser()\nqp.parse(query)\n```\n> num_tokens: 2  \n> And([Term('ti', 'dog'), Term('ti', 'cat')])\n\nSpecial AND\n```python\nquery = 'ti:\"dog\"ti:\"cat\"'\n```\n\n> num_tokens: 1  \n> And([Term('ti', 'dog'), Term('ti', 'cat')])\n\nNormal phrase\n```python\nquery = 'ti:\"The quick brown fox jumps over the lazy dog\"'\n```\n\n> num_tokens: 9  \n> Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)\n\nSpecial phrase\n```python\nquery = 'ti:\"The@quick@brown@fox@jumps@over@the@lazy@dog\"'\n```\n\n> num_tokens: 1  \n> Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)\n\n### Note\n* We would like to apply this technique to OR as well, but we couldn't find a way.\n* Even without this technique, we were able to achieve a LB score of 0.85.\n\n## Set Cover Problem\n* Up to this point, we have many query candidates created by global counter and 50c2.\n* For example, the following candidates may have been obtained.\n* Note: With the above token count technique, each token count is always 1.\n\n| candidates                           | tp_covers                 | FP | \n|---------------------------------|---------------------------|----|\n| ti:device                       | {0, 1, 2, 3, 4, 5, 6}     | 3 |\n| ti:\"dog\"ti:\"cat\"                   | {0, 1, 2, 7}                 |  1 |\n| clm:\"dissociation\"ab:\"proteins\" | {3, 4, 10}                |  0 | \n| ...                             | ...                       | ...| \n\n\n* Selecting some of these candidates and concatenate them with OR optimally is difficult, because this is a [set cover problem](https://en.wikipedia.org/wiki/Set_cover_problem), which is known to be NP-hard.\n* We solved it as a integer linear programming problem using a python solver.\n* Maximize TP and minimize FP with a token count of 50 or less. In practice, we maximized `10xTP − FP`.\n* It can also be solved using some kind of greedy algorithm, but the solver provided a slightly better score (+0.01).\n\n## Query Example\n\n<details><summary>Here is a query example with 49 token.</summary>\n\n> (cpc:\"B60P1/165\"cpc:\"E02F3/3483\"cpc:\"B60P1/165\"cpc:\"E02F3/3486\") OR (cpc:\"B60D1/26\"cpc:\"B62D49/04\"cpc:\"B60D1/26\"cpc:\"E02F3/6472\"cpc:\"B60D1/26\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F3/64\"cpc:\"B62D49/04\"cpc:\"E02F3/6472\"cpc:\"B62D49/04\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F9/2016\"cpc:\"E02F3/6472\"cpc:\"E02F9/2016\"cpc:\"E02F3/655\"cpc:\"E02F9/2016\"detd:\"courli\"detd:\"lgayea\") OR (cpc:\"B65F2003/0283\"cpc:\"E02F3/3486\"cpc:\"B65F3/046\"cpc:\"E02F3/3486\"ti:\"end@loader@actuating\"ti:\"loader@actuating\"ti:\"loader@actuating@mechanism\"ti:\"actuating@mechanism@dump\"cpc:\"B65F2003/0283\"cpc:\"B65F3/046\"detd:\"horbe\") OR (detd:\"wharfpthe\"ti:\"dumping@scoop\"ti:\"side@dumping@scoop\"detd:\"hatchof\") OR (detd:\"powerswung\") OR (detd:\"shlppee\") OR (detd:\"0rneypearce\"detd:\"schaepcrklaus\"ti:\"as@asphalt@or\"ti:\"improvement@therein@laying\"ti:\"laying@surfacing\"ti:\"laying@surfacing@material\"ti:\"machine@and@improvement\"ti:\"surfacing@material@as\"ti:\"therein@laying\"ti:\"therein@laying@surfacing\"detd:\"comprlsesr\") OR (detd:\"disswingable\"detd:\"rdlyonto\"detd:\"sitionedsymmetrically\"detd:\"understandingflof\"detd:\"opjusting\") OR (ti:\"mechanical@shoveling@machine\"ti:\"mechanical@shoveling\") OR ((detd:\"tractors\"detd:\"ravity\"detd:\"scrapers\"detd:\"kick\"cpc:\"E02F3/6472\"ti:\"scraper\"cpc:\"E02F3/656\"detd:\"apron\"detd:\"scraper\"detd:\"hingedly\"detd:\"tractor\"detd:\"sheave\"detd:\"sheaves\"detd:\"cooperates\"detd:\"dead\"detd:\"axle\")) OR ((detd:\"oor\"detd:\"exible\"detd:\"retracting\"detd:\"trough\"detd:\"retracted\"detd:\"turntable\"detd:\"underground\"detd:\"jacks\"detd:\"therealong\"detd:\"rectilinear\"detd:\"propelling\"detd:\"conveyor\"detd:\"extensible\"detd:\"slip\"detd:\"clutch\"detd:\"mines\"detd:\"adjustably\"detd:\"engageable\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"shovelling\"detd:\"tom\"detd:\"cushioned\"detd:\"compel\"detd:\"swiveled\"detd:\"abruptly\"detd:\"swivelly\"detd:\"wardly\"detd:\"hose\"detd:\"guideway\"detd:\"swivelled\"detd:\"undesired\"detd:\"sup\"detd:\"ported\"detd:\"swivel\"detd:\"osgood\"detd:\"pile\"detd:\"muck\"detd:\"guideways\"detd:\"dig\")) OR ((detd:\"planetaries\"detd:\"payed\"detd:\"planetary\"detd:\"mosier\"detd:\"fiexible\"detd:\"2li\"detd:\"conveyer\"detd:\"tractive\"detd:\"lil\"detd:\"ior\"detd:\"exible\"detd:\"2s\"detd:\"yieldable\"detd:\"tunnels\"detd:\"compensating\"detd:\"excepting\"detd:\"lll\"detd:\"chute\"detd:\"illinois\"detd:\"geared\")) OR ((detd:\"fioor\"detd:\"simmons\"detd:\"overlie\"detd:\"hydraulically\"detd:\"attachable\"detd:\"swivelly\"detd:\"coal\"detd:\"jacks\"detd:\"swivel\"detd:\"anchor\"detd:\"joy\"detd:\"guideways\"detd:\"unison\"detd:\"propelling\"detd:\"pennsylvania\"detd:\"mining\"detd:\"extensible\"detd:\"transporting\"detd:\"tilted\"detd:\"elevating\")) OR ((detd:\"vfiled\"detd:\"communicable\"detd:\"eiect\"detd:\"hydraulically\"detd:\"sullivan\"ti:\"material\"detd:\"propulsion\"detd:\"machinery\"detd:\"claremont\"detd:\"bores\"detd:\"uid\"detd:\"pile\"detd:\"muck\"detd:\"lin\"detd:\"dig\"detd:\"trackway\"detd:\"conduits\"detd:\"5i\"detd:\"massachusetts\"detd:\"hydraulic\")) OR ((detd:\"i23\"detd:\"movementof\"detd:\"i26\"detd:\"insides\"detd:\"encircles\"detd:\"compactness\"detd:\"h2\"detd:\"cushioning\"detd:\"yieldably\"detd:\"reversely\"detd:\"pivoting\"detd:\"cams\"detd:\"mucking\"detd:\"abuts\"detd:\"lug\"detd:\"therealong\"detd:\"teeth\"detd:\"axles\"detd:\"scoop\"detd:\"nuts\")) OR ((detd:\"foolproof\"detd:\"selfcentering\"detd:\"impetus\"detd:\"i85\"detd:\"coaction\"detd:\"rockers\"detd:\"pinned\"detd:\"maxson\"detd:\"i00\"detd:\"pivotable\"detd:\"i05\"detd:\"fork\"detd:\"inadequate\"detd:\"dipper\"detd:\"compelling\"detd:\"plungers\"detd:\"incapable\"detd:\"h5\"detd:\"abrupt\"detd:\"tang\")) OR ((detd:\"evidently\"detd:\"shank\"detd:\"receivable\"detd:\"hoses\"detd:\"venting\"detd:\"urges\"detd:\"interrupting\"detd:\"vented\"detd:\"hose\"detd:\"rolls\"detd:\"inactive\"detd:\"claremont\"detd:\"interrupted\"detd:\"3i\"detd:\"joy\"detd:\"guideways\"detd:\"trackway\"detd:\"pennsylvania\"detd:\"embodies\"detd:\"assumes\")) OR ((detd:\"seam\"detd:\"bevel\"detd:\"brake\"detd:\"rocked\"detd:\"wheeled\"detd:\"propulsion\"detd:\"spur\"detd:\"rst\"detd:\"coal\"detd:\"pinion\"detd:\"withdrawal\"detd:\"gearing\"detd:\"shafts\"detd:\"extensible\"detd:\"clutch\"detd:\"keyed\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"draulic\"detd:\"hy\"detd:\"4s\"detd:\"vfor\"detd:\"oor\"detd:\"zontal\"detd:\"withdrawing\"detd:\"ie\"detd:\"andv\"detd:\"progresses\"detd:\"mw\"detd:\"lo\"detd:\"anchor\"detd:\"teeth\"detd:\"conveyor\"detd:\"extremity\"detd:\"3l\"detd:\"hydraulic\")) OR ((detd:\"isv\"detd:\"isprovided\"detd:\"sion\"detd:\"ropes\"detd:\"rope\"detd:\"suddenly\"detd:\"vof\"detd:\"relied\"detd:\"thel\"detd:\"urge\"detd:\"anda\"detd:\"4l\"detd:\"0f\"detd:\"gearing\"detd:\"anchored\"detd:\"transportation\"detd:\"mining\"detd:\"lthe\"detd:\"transporting\")) OR ((detd:\"posltion\"detd:\"motive\"detd:\"unwind\"detd:\"drifts\"detd:\"rearmost\"detd:\"tunnels\"detd:\"cated\"detd:\"segmental\"detd:\"injury\"detd:\"imparting\"detd:\"pile\"detd:\"ward\"detd:\"ap\"detd:\"scoop\"detd:\"swings\"detd:\"mines\"detd:\"reversible\")) OR ((detd:\"adaptedv\"detd:\"grooved\"detd:\"ore\"detd:\"chine\"detd:\"vhen\"detd:\"t0\"detd:\"opera\"detd:\"vand\"detd:\"sidewise\"detd:\"wheeled\"detd:\"thev\"detd:\"meshes\"detd:\"andv\"cpc:\"E02F9/022\"detd:\"movably\"detd:\"idler\"detd:\"coal\"detd:\"pinion\"detd:\"casting\"detd:\"bears\")) OR ((detd:\"compel\"detd:\"abruptly\"detd:\"tensioned\"detd:\"swivelled\"detd:\"coincident\"detd:\"assured\"detd:\"turntable\"detd:\"claremont\"detd:\"swivel\"detd:\"osgood\"detd:\"fulcrum\"detd:\"loader\"detd:\"guideways\"cpc:\"E02F3/3486\"detd:\"trackway\"detd:\"propelling\"detd:\"rolling\"detd:\"assumes\"detd:\"alinement\"detd:\"swings\")) OR ((detd:\"hoists\"cpc:\"E02F3/657\"detd:\"hoist\"cpc:\"E02F3/656\"detd:\"medial\"detd:\"bumper\"detd:\"trunnions\"detd:\"trunnion\"detd:\"grading\"detd:\"brake\"detd:\"steering\"detd:\"spreading\"detd:\"tractor\"detd:\"propulsion\"detd:\"coacting\"detd:\"extremities\"detd:\"gearing\"detd:\"rail\"detd:\"dump\"detd:\"clutch\"))\n\n\n---------------------------\nEDIT: 2024/08/06\n\nWe pulished our codes.\n\n* with magic\n  * https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-with-magic?scriptVersionId=190687232\n* without magic\n  * https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-without-magic?scriptVersionId=190688084\n* global counter(title)\n  * https://www.kaggle.com/code/sega1031/uspto-global-title-word-counter\n* global counter as one \n  * https://www.kaggle.com/code/sega1031/uspto-global-counters-limit30",
      "votes": null
    },
    {
      "id": "2935189",
      "postDate": "07/25/2024 03:52:37",
      "content": "<p>Congratulations!<br>\nAwesome request with 49 tokens!😂</p>",
      "rawMarkdown": "Congratulations!\nAwesome request with 49 tokens!😂",
      "votes": null
    },
    {
      "id": "2935300",
      "postDate": "07/25/2024 06:31:31",
      "content": "<p>Congratulations on winning the 5th place in this competition. Thanks for sharing details of your solution with charts and code. </p>",
      "rawMarkdown": "Congratulations on winning the 5th place in this competition. Thanks for sharing details of your solution with charts and code.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2935189,
      "author_name": "qurusx",
      "author_url": "",
      "post_date": "07/25/2024 03:52:37",
      "content": "<p>Congratulations!<br>\nAwesome request with 49 tokens!😂</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2935300,
      "author_name": "crsuthikshnkumar",
      "author_url": "",
      "post_date": "07/25/2024 06:31:31",
      "content": "<p>Congratulations on winning the 5th place in this competition. Thanks for sharing details of your solution with charts and code. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2935058": "First of all, thanks to the organizers for hosting this competition, and to my teammate @sega1031.\n\n## Overview\n* Create many query candidates with minimal false positives.\n* Select up to 25 candidates using solver, then concatenate using OR.\n* Token count: you can reduce the token count of an AND query to one token (see below).\n\n## Validation Strategy\n* We use the published validation index.  \n  https://www.kaggle.com/datasets/devinanzelmo/uspto-explainable-ai-validation-index\n* The leaderboard score is slightly lower than the validation score but is well correlated.\n\n## The Metric and Basic Strategy\n* We believe assumption from @devinanzelmo is correct: https://www.kaggle.com/competitions/uspto-explainable-ai/discussion/499981#2791642\n* We confirm this through two submissions. The theoretical and actual leaderboard values are very close.\n  1. Select one patent from the 50 neighbors and build a query from first 20 words of its abstract with phrase search. This procedure gives a query with 20 tokens, TP1, and FP0 (= one true positive and zero false positive).\n  2. Select two patents from the 50 neighbors and build a query from first 20 words of each abstract for a phrase search. The two queries are concatenated with OR. This results in 41 tokens, TP2, and FP0.\n\n| exp | TP | FP | Theoretical LB | Actual LB |\n|-----|----|----|----------------|-----------|\n|  1  |  1 |  0 |      0.089     |    0.08   |\n|  2  |  2 |  0 |      0.159     |    0.15   |\n\n* In this evaluation metric, it is crucial for TP to rank highly and FP NOT to rank highly.\n  * If 10 TPs are followed by 40 FPs -> score: 0.514\n  * If 40 FPs are followed by 10 TPs -> score: 0.023\n* Therefore, our basic strategy is to collect as many TPs as possible while keeping FPs as close to zero as possible.\n* This plot shows the score for N TPs followed by (50-N) FPs.\n  * if 41 TPs are followed by 9 FPs -> score: 0.980\n  * if 44 TPs are followed by 6 FPs -> score: 0.991\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2304617%2Ff0d3b8a7adc696a2971955b617e550eb%2F__results___5_0.png?generation=1721866334202765&alt=media)\n\n## Creating Candidates\n### Global Counter Candidates\n* Preparation\n  * Count how many times a specific word or n-gram appears in all given patents. For example, \"ti:device\" may appear 20,000 times in 200,000 patents.\n  * We call this \"global counter\".\n  * This global counter is prepared in advance.\n* Creating candidates\n  * Check the global counter for all words and n-grams appearing in the 50 neighbors we want to retrieve.\n  * If \"ti:device\" appears in 3 out of the 50 neighbors, and the global counter's count is 4, then \"ti:device\" query results with TP3 and FP1.\n  * Create these candidates for ti, ab, clm, detd, and cpc, using unigrams, bigrams, and trigrams.\n\n| candidates    | tp_cover     | tp                   | global_count | fp         |\n|------------|--------------|----------------------|--------------|------------|\n| ti:device  | {0,1,2}      | len({0,1,2}) = 3     | 4            | 4 - 3 = 1  |\n| clm:invention    | {0,2,3,4,5}  | 5                    | 7            | 2          |\n| detd:method detd:device   | {6}          | 1                    | 1            | 0          |\n\n\n* tp_cover = {0,1,2} indicates that this query candidate can retrieve neighbors with indices 0, 1, and 2.\n* Example:\n  * Concatenate three candidates in this table with OR: `ti:device OR clm:invention OR (detd:method detd:device)`.\n  * Since they are concatenated with OR, tp_cover becomes {0,1,2,3,4,5,6}.  \n  * Therefore, this query will be 6 tokens, TP7, and FP3.\n\n### 50c2 Candidates\n\nSelect two patents from the 50 neighbors.\n\n| publication_number | ti         | abst            |\n|--------|---------------|-----------------|\n| US-0000-A    | dog cat fox   | pig cow         |\n| US-0001-A    | cat fox       | duck frog cow   |\n\n* Extract the common words from these two patents and create an AND query.\n  * `ti:cat AND ti:fox AND ab:cow`\n* This query will retrieve these two patents, but we need to calculate/estimate how many FPs this query will retrieve.\n* We use IDF (inverse document frequency).\n  * Since IDF is the logarithm of the inverse of the occurrence probability, summed-up IDF values give the occurrence probability of ANDed candidates.\n  * Calculate the IDF sum of the created query, and if it is above a certain threshold, estimate that FP is zero.\n  * We set this threshold to 80 using validation data. In this example, we are checking if idf(ti:cat) + idf(ti:fox) + idf(ab:cow) > 80.\n* Calculate this for all combinations of 50 choose 2 (= 1225).\n  * Each candidate must be 2 TPs and 0 FP.\n\n| candidates    | tp_cover     | tp                   |  IDF_sum | fp         |\n|------------|--------------|----------------------|------------|------------|\n| ti:cat ti:fox ab:cow  | {0,1}      |  2            | 103.24 |0  |\n\n\n## Token Count\n```python\ndef count_query_tokens(query: str):\n    return len([i for i in re.split('[\\s+()]', query) if i])\t\n```\n\n* The number of tokens is counted by this function,\n* Through a particular approach to constructing the query, it is possible to keep the parse result the same while reducing the token count to one.\n\n### Example\nNormal AND\n```python\nquery = \"ti:dog ti:cat\"\nnum_tokens = whoosh_utils.count_query_tokens(query)\nprint(\"num_tokens:\", num_tokens)\nqp = whoosh_utils.get_query_parser()\nqp.parse(query)\n```\n> num_tokens: 2  \n> And([Term('ti', 'dog'), Term('ti', 'cat')])\n\nSpecial AND\n```python\nquery = 'ti:\"dog\"ti:\"cat\"'\n```\n\n> num_tokens: 1  \n> And([Term('ti', 'dog'), Term('ti', 'cat')])\n\nNormal phrase\n```python\nquery = 'ti:\"The quick brown fox jumps over the lazy dog\"'\n```\n\n> num_tokens: 9  \n> Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)\n\nSpecial phrase\n```python\nquery = 'ti:\"The@quick@brown@fox@jumps@over@the@lazy@dog\"'\n```\n\n> num_tokens: 1  \n> Phrase('ti', ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog'], slop=1, boost=1.000000)\n\n### Note\n* We would like to apply this technique to OR as well, but we couldn't find a way.\n* Even without this technique, we were able to achieve a LB score of 0.85.\n\n## Set Cover Problem\n* Up to this point, we have many query candidates created by global counter and 50c2.\n* For example, the following candidates may have been obtained.\n* Note: With the above token count technique, each token count is always 1.\n\n| candidates                           | tp_covers                 | FP | \n|---------------------------------|---------------------------|----|\n| ti:device                       | {0, 1, 2, 3, 4, 5, 6}     | 3 |\n| ti:\"dog\"ti:\"cat\"                   | {0, 1, 2, 7}                 |  1 |\n| clm:\"dissociation\"ab:\"proteins\" | {3, 4, 10}                |  0 | \n| ...                             | ...                       | ...| \n\n\n* Selecting some of these candidates and concatenate them with OR optimally is difficult, because this is a [set cover problem](https://en.wikipedia.org/wiki/Set_cover_problem), which is known to be NP-hard.\n* We solved it as a integer linear programming problem using a python solver.\n* Maximize TP and minimize FP with a token count of 50 or less. In practice, we maximized `10xTP − FP`.\n* It can also be solved using some kind of greedy algorithm, but the solver provided a slightly better score (+0.01).\n\n## Query Example\n\n<details><summary>Here is a query example with 49 token.</summary>\n\n> (cpc:\"B60P1/165\"cpc:\"E02F3/3483\"cpc:\"B60P1/165\"cpc:\"E02F3/3486\") OR (cpc:\"B60D1/26\"cpc:\"B62D49/04\"cpc:\"B60D1/26\"cpc:\"E02F3/6472\"cpc:\"B60D1/26\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F3/64\"cpc:\"B62D49/04\"cpc:\"E02F3/6472\"cpc:\"B62D49/04\"cpc:\"E02F3/655\"cpc:\"B62D49/04\"cpc:\"E02F9/2016\"cpc:\"E02F3/6472\"cpc:\"E02F9/2016\"cpc:\"E02F3/655\"cpc:\"E02F9/2016\"detd:\"courli\"detd:\"lgayea\") OR (cpc:\"B65F2003/0283\"cpc:\"E02F3/3486\"cpc:\"B65F3/046\"cpc:\"E02F3/3486\"ti:\"end@loader@actuating\"ti:\"loader@actuating\"ti:\"loader@actuating@mechanism\"ti:\"actuating@mechanism@dump\"cpc:\"B65F2003/0283\"cpc:\"B65F3/046\"detd:\"horbe\") OR (detd:\"wharfpthe\"ti:\"dumping@scoop\"ti:\"side@dumping@scoop\"detd:\"hatchof\") OR (detd:\"powerswung\") OR (detd:\"shlppee\") OR (detd:\"0rneypearce\"detd:\"schaepcrklaus\"ti:\"as@asphalt@or\"ti:\"improvement@therein@laying\"ti:\"laying@surfacing\"ti:\"laying@surfacing@material\"ti:\"machine@and@improvement\"ti:\"surfacing@material@as\"ti:\"therein@laying\"ti:\"therein@laying@surfacing\"detd:\"comprlsesr\") OR (detd:\"disswingable\"detd:\"rdlyonto\"detd:\"sitionedsymmetrically\"detd:\"understandingflof\"detd:\"opjusting\") OR (ti:\"mechanical@shoveling@machine\"ti:\"mechanical@shoveling\") OR ((detd:\"tractors\"detd:\"ravity\"detd:\"scrapers\"detd:\"kick\"cpc:\"E02F3/6472\"ti:\"scraper\"cpc:\"E02F3/656\"detd:\"apron\"detd:\"scraper\"detd:\"hingedly\"detd:\"tractor\"detd:\"sheave\"detd:\"sheaves\"detd:\"cooperates\"detd:\"dead\"detd:\"axle\")) OR ((detd:\"oor\"detd:\"exible\"detd:\"retracting\"detd:\"trough\"detd:\"retracted\"detd:\"turntable\"detd:\"underground\"detd:\"jacks\"detd:\"therealong\"detd:\"rectilinear\"detd:\"propelling\"detd:\"conveyor\"detd:\"extensible\"detd:\"slip\"detd:\"clutch\"detd:\"mines\"detd:\"adjustably\"detd:\"engageable\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"shovelling\"detd:\"tom\"detd:\"cushioned\"detd:\"compel\"detd:\"swiveled\"detd:\"abruptly\"detd:\"swivelly\"detd:\"wardly\"detd:\"hose\"detd:\"guideway\"detd:\"swivelled\"detd:\"undesired\"detd:\"sup\"detd:\"ported\"detd:\"swivel\"detd:\"osgood\"detd:\"pile\"detd:\"muck\"detd:\"guideways\"detd:\"dig\")) OR ((detd:\"planetaries\"detd:\"payed\"detd:\"planetary\"detd:\"mosier\"detd:\"fiexible\"detd:\"2li\"detd:\"conveyer\"detd:\"tractive\"detd:\"lil\"detd:\"ior\"detd:\"exible\"detd:\"2s\"detd:\"yieldable\"detd:\"tunnels\"detd:\"compensating\"detd:\"excepting\"detd:\"lll\"detd:\"chute\"detd:\"illinois\"detd:\"geared\")) OR ((detd:\"fioor\"detd:\"simmons\"detd:\"overlie\"detd:\"hydraulically\"detd:\"attachable\"detd:\"swivelly\"detd:\"coal\"detd:\"jacks\"detd:\"swivel\"detd:\"anchor\"detd:\"joy\"detd:\"guideways\"detd:\"unison\"detd:\"propelling\"detd:\"pennsylvania\"detd:\"mining\"detd:\"extensible\"detd:\"transporting\"detd:\"tilted\"detd:\"elevating\")) OR ((detd:\"vfiled\"detd:\"communicable\"detd:\"eiect\"detd:\"hydraulically\"detd:\"sullivan\"ti:\"material\"detd:\"propulsion\"detd:\"machinery\"detd:\"claremont\"detd:\"bores\"detd:\"uid\"detd:\"pile\"detd:\"muck\"detd:\"lin\"detd:\"dig\"detd:\"trackway\"detd:\"conduits\"detd:\"5i\"detd:\"massachusetts\"detd:\"hydraulic\")) OR ((detd:\"i23\"detd:\"movementof\"detd:\"i26\"detd:\"insides\"detd:\"encircles\"detd:\"compactness\"detd:\"h2\"detd:\"cushioning\"detd:\"yieldably\"detd:\"reversely\"detd:\"pivoting\"detd:\"cams\"detd:\"mucking\"detd:\"abuts\"detd:\"lug\"detd:\"therealong\"detd:\"teeth\"detd:\"axles\"detd:\"scoop\"detd:\"nuts\")) OR ((detd:\"foolproof\"detd:\"selfcentering\"detd:\"impetus\"detd:\"i85\"detd:\"coaction\"detd:\"rockers\"detd:\"pinned\"detd:\"maxson\"detd:\"i00\"detd:\"pivotable\"detd:\"i05\"detd:\"fork\"detd:\"inadequate\"detd:\"dipper\"detd:\"compelling\"detd:\"plungers\"detd:\"incapable\"detd:\"h5\"detd:\"abrupt\"detd:\"tang\")) OR ((detd:\"evidently\"detd:\"shank\"detd:\"receivable\"detd:\"hoses\"detd:\"venting\"detd:\"urges\"detd:\"interrupting\"detd:\"vented\"detd:\"hose\"detd:\"rolls\"detd:\"inactive\"detd:\"claremont\"detd:\"interrupted\"detd:\"3i\"detd:\"joy\"detd:\"guideways\"detd:\"trackway\"detd:\"pennsylvania\"detd:\"embodies\"detd:\"assumes\")) OR ((detd:\"seam\"detd:\"bevel\"detd:\"brake\"detd:\"rocked\"detd:\"wheeled\"detd:\"propulsion\"detd:\"spur\"detd:\"rst\"detd:\"coal\"detd:\"pinion\"detd:\"withdrawal\"detd:\"gearing\"detd:\"shafts\"detd:\"extensible\"detd:\"clutch\"detd:\"keyed\"detd:\"elevating\"detd:\"hydraulic\")) OR ((detd:\"draulic\"detd:\"hy\"detd:\"4s\"detd:\"vfor\"detd:\"oor\"detd:\"zontal\"detd:\"withdrawing\"detd:\"ie\"detd:\"andv\"detd:\"progresses\"detd:\"mw\"detd:\"lo\"detd:\"anchor\"detd:\"teeth\"detd:\"conveyor\"detd:\"extremity\"detd:\"3l\"detd:\"hydraulic\")) OR ((detd:\"isv\"detd:\"isprovided\"detd:\"sion\"detd:\"ropes\"detd:\"rope\"detd:\"suddenly\"detd:\"vof\"detd:\"relied\"detd:\"thel\"detd:\"urge\"detd:\"anda\"detd:\"4l\"detd:\"0f\"detd:\"gearing\"detd:\"anchored\"detd:\"transportation\"detd:\"mining\"detd:\"lthe\"detd:\"transporting\")) OR ((detd:\"posltion\"detd:\"motive\"detd:\"unwind\"detd:\"drifts\"detd:\"rearmost\"detd:\"tunnels\"detd:\"cated\"detd:\"segmental\"detd:\"injury\"detd:\"imparting\"detd:\"pile\"detd:\"ward\"detd:\"ap\"detd:\"scoop\"detd:\"swings\"detd:\"mines\"detd:\"reversible\")) OR ((detd:\"adaptedv\"detd:\"grooved\"detd:\"ore\"detd:\"chine\"detd:\"vhen\"detd:\"t0\"detd:\"opera\"detd:\"vand\"detd:\"sidewise\"detd:\"wheeled\"detd:\"thev\"detd:\"meshes\"detd:\"andv\"cpc:\"E02F9/022\"detd:\"movably\"detd:\"idler\"detd:\"coal\"detd:\"pinion\"detd:\"casting\"detd:\"bears\")) OR ((detd:\"compel\"detd:\"abruptly\"detd:\"tensioned\"detd:\"swivelled\"detd:\"coincident\"detd:\"assured\"detd:\"turntable\"detd:\"claremont\"detd:\"swivel\"detd:\"osgood\"detd:\"fulcrum\"detd:\"loader\"detd:\"guideways\"cpc:\"E02F3/3486\"detd:\"trackway\"detd:\"propelling\"detd:\"rolling\"detd:\"assumes\"detd:\"alinement\"detd:\"swings\")) OR ((detd:\"hoists\"cpc:\"E02F3/657\"detd:\"hoist\"cpc:\"E02F3/656\"detd:\"medial\"detd:\"bumper\"detd:\"trunnions\"detd:\"trunnion\"detd:\"grading\"detd:\"brake\"detd:\"steering\"detd:\"spreading\"detd:\"tractor\"detd:\"propulsion\"detd:\"coacting\"detd:\"extremities\"detd:\"gearing\"detd:\"rail\"detd:\"dump\"detd:\"clutch\"))\n\n\n---------------------------\nEDIT: 2024/08/06\n\nWe pulished our codes.\n\n* with magic\n  * https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-with-magic?scriptVersionId=190687232\n* without magic\n  * https://www.kaggle.com/code/iiyamaiiyama/uspto-5th-place-submission-without-magic?scriptVersionId=190688084\n* global counter(title)\n  * https://www.kaggle.com/code/sega1031/uspto-global-title-word-counter\n* global counter as one \n  * https://www.kaggle.com/code/sega1031/uspto-global-counters-limit30",
    "2935189": "Congratulations!\nAwesome request with 49 tokens!😂",
    "2935300": "Congratulations on winning the 5th place in this competition. Thanks for sharing details of your solution with charts and code."
  },
  "source": "meta"
}