{ "id": 499109, "title": "Keep getting submission scoring error", "url": "/competitions/uspto-explainable-ai/discussion/499109", "author_name": "", "post_date": "2024-04-30T17:10:16.195777800Z", "votes": 4, "comment_count": 9, "views": 0, "content": "

Hello everybody, I am struggling to get my first submission out. I learned from prior challenges that I don't just want to copy the currently available notebooks and am trying to get my own version going.

\n

My \"hello world\" attempt is using key word from the title:

\n
queries = []\n target in test.publication_number.to_list():\n    patent_data = metadata[metadata.publication_number == target].iloc[].parquet\n\n    title = get_title(patent_data, target)\n    cleaned = title.translate(str.maketrans(, , .punctuation))\n    cleaned = cleaned.translate(str.maketrans(, , .digits))\n\n    #  query with unique words, excluding  words\n     = ((cleaned.())-stops)\n     () > : # reduce     words ->  tokens\n         = [:]\n    query =  + .()\n\n    queryValidator = whoosh_utils.QueryValidator()\n\n    : #  seen in https://www.kaggle./code/seshurajup/---uspto-single-cpc-query\n        queryValidator.validate_query(query)\n    except:\n        query = \n     whoosh_utils.count_query_tokens(query) > :\n        query = \n\n    queries.(query)\n    gc.collect()    \n
\n

I am getting a DataFrame that looks appropriate:
\n

\n

However, after 2-3 hours of running I am getting the error \"Submission scoring error\"

\n

Am I missing something obvious? All the queries done on the fake test set are working if I query them on the training index..

", "messages": [ { "id": "2785222", "postDate": "04/30/2024 17:10:16", "content": "

Hello everybody, I am struggling to get my first submission out. I learned from prior challenges that I don't just want to copy the currently available notebooks and am trying to get my own version going.

\n

My \"hello world\" attempt is using key word from the title:

\n
queries = []\n target in test.publication_number.to_list():\n    patent_data = metadata[metadata.publication_number == target].iloc[].parquet\n\n    title = get_title(patent_data, target)\n    cleaned = title.translate(str.maketrans(, , .punctuation))\n    cleaned = cleaned.translate(str.maketrans(, , .digits))\n\n    #  query with unique words, excluding  words\n     = ((cleaned.())-stops)\n     () > : # reduce     words ->  tokens\n         = [:]\n    query =  + .()\n\n    queryValidator = whoosh_utils.QueryValidator()\n\n    : #  seen in https://www.kaggle./code/seshurajup/---uspto-single-cpc-query\n        queryValidator.validate_query(query)\n    except:\n        query = \n     whoosh_utils.count_query_tokens(query) > :\n        query = \n\n    queries.(query)\n    gc.collect()    \n
\n

I am getting a DataFrame that looks appropriate:
\n

\n

However, after 2-3 hours of running I am getting the error \"Submission scoring error\"

\n

Am I missing something obvious? All the queries done on the fake test set are working if I query them on the training index..

", "rawMarkdown": "Hello everybody, I am struggling to get my first submission out. I learned from prior challenges that I don't just want to copy the currently available notebooks and am trying to get my own version going.\n\nMy \"hello world\" attempt is using key word from the title:\n\n queries = []\n for target in test.publication_number.to_list():\n patent_data = metadata[metadata.publication_number == target].iloc[0].parquet\n\n title = get_title(patent_data, target)\n cleaned = title.translate(str.maketrans('', '', string.punctuation))\n cleaned = cleaned.translate(str.maketrans('', '', string.digits))\n\n # make query with unique words, excluding stop words\n res = list(set(cleaned.split())-stops)\n if len(res) > 25: # reduce list to only 25 words -> 49 tokens\n res = res[:25]\n query = \"ti:\" + \" OR ti:\".join(res)\n\n queryValidator = whoosh_utils.QueryValidator()\n\n try: # as seen in https://www.kaggle.com/code/seshurajup/lb-0-11-uspto-single-cpc-query\n queryValidator.validate_query(query)\n except:\n query = \"ti:device\"\n if whoosh_utils.count_query_tokens(query) > 50:\n query = \"ti:mobile\"\n\n queries.append(query)\n gc.collect() \n\nI am getting a DataFrame that looks appropriate: \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8504824%2Fd66295b0bfd0aeb099f63e46795b1a46%2Fquery%20result.png?generation=1714497061379059&alt=media)\n\nHowever, after 2-3 hours of running I am getting the error \"Submission scoring error\"\n\nAm I missing something obvious? All the queries done on the fake test set are working if I query them on the training index..", "votes": null }, { "id": "2785225", "postDate": "04/30/2024 17:12:30", "content": "

@valentinwerner, check with my verify notebook, with your title OR queries, if u keep one key. it won't freeze, as you add OR for some queries it will freeze.

", "rawMarkdown": "valentinwerner, check with my verify notebook, with your title OR queries, if u keep one key. it won't freeze, as you add OR for some queries it will freeze.", "votes": null }, { "id": "2785284", "postDate": "04/30/2024 17:46:02", "content": "

Ah I didnt fully read it before… this makes no sense but explains my issue. It is extremely frustrating and will block query generation enormously.

\n

This basically means we have less effective tokens per query and models have to learn to write these new/weird query with (a and a) OR (b and b)

", "rawMarkdown": "Ah I didnt fully read it before... this makes no sense but explains my issue. It is extremely frustrating and will block query generation enormously.\n\nThis basically means we have less effective tokens per query and models have to learn to write these new/weird query with (a and a) OR (b and b)", "votes": null }, { "id": "2785353", "postDate": "04/30/2024 18:17:51", "content": "

i mean @valentinwerner

\n
\n

for cpc:A - we will get results for all queries
\n for cpc:A OR cpc:B - sometimes it will freeze ( is my observation )

\n
", "rawMarkdown": "i mean @valentinwerner \n> for cpc:A - we will get results for all queries \n> for cpc:A OR cpc:B - sometimes it will freeze ( is my observation )", "votes": null }, { "id": "2785982", "postDate": "05/01/2024 05:24:41", "content": "

I understood that, but that means our query builder will not be able to include OR statements, which is a heavy limitation..

\n

// just saw that they will fix it soon :)

", "rawMarkdown": "I understood that, but that means our query builder will not be able to include OR statements, which is a heavy limitation..\n\n// just saw that they will fix it soon :)", "votes": null }, { "id": "2791944", "postDate": "05/04/2024 00:24:08", "content": "

In the competition overview, it mentioned \"The metric notebook must finish running your queries in 60 minutes\". Is it possible that the statement with \"OR\" generated too much results, slowing down the queries and cause it to exceed the time limit?

", "rawMarkdown": "In the competition overview, it mentioned \"The metric notebook must finish running your queries in 60 minutes\". Is it possible that the statement with \"OR\" generated too much results, slowing down the queries and cause it to exceed the time limit?", "votes": null }, { "id": "2792365", "postDate": "05/04/2024 07:23:13", "content": "

Running them on the train index takes 0.3 seconds for the 10 test queries.

\n

This means the 2500 queries run in 75 seconds (given same time).

", "rawMarkdown": "Running them on the train index takes 0.3 seconds for the 10 test queries. \n\nThis means the 2500 queries run in 75 seconds (given same time).", "votes": null }, { "id": "2793860", "postDate": "05/05/2024 03:21:28", "content": "

The scoring for this competition seems flawed or they just want us to fix their whoosh issues because there are only a few valid submissions from a big community and there are little to no responses from competition admins on submission issues.

", "rawMarkdown": "The scoring for this competition seems flawed or they just want us to fix their whoosh issues because there are only a few valid submissions from a big community and there are little to no responses from competition admins on submission issues.", "votes": null }, { "id": "2793918", "postDate": "05/05/2024 04:10:14", "content": "

@jazivxt already Kaggle Staff answered about it, in other discussion i.e will be fixed in this week

", "rawMarkdown": "jazivxt already Kaggle Staff answered about it, in other discussion i.e will be fixed in this week", "votes": null }, { "id": "2794605", "postDate": "05/05/2024 12:04:35", "content": "

Did it work for you @valentinwerner and @seshurajup ?
\nI can just get the single cpc query working right now.

", "rawMarkdown": "Did it work for you @valentinwerner and @seshurajup ?\nI can just get the single cpc query working right now.", "votes": null } ], "comments": [ { "id": 2785225, "author_name": "seshurajup", "author_url": "", "post_date": "04/30/2024 17:12:30", "content": "

@valentinwerner, check with my verify notebook, with your title OR queries, if u keep one key. it won't freeze, as you add OR for some queries it will freeze.

", "votes": null, "replies": [ { "id": 2785284, "author_name": "valentinwerner", "author_url": "", "post_date": "04/30/2024 17:46:02", "content": "

Ah I didnt fully read it before… this makes no sense but explains my issue. It is extremely frustrating and will block query generation enormously.

\n

This basically means we have less effective tokens per query and models have to learn to write these new/weird query with (a and a) OR (b and b)

", "votes": null, "replies": [ { "id": 2785353, "author_name": "seshurajup", "author_url": "", "post_date": "04/30/2024 18:17:51", "content": "

i mean @valentinwerner

\n
\n

for cpc:A - we will get results for all queries
\n for cpc:A OR cpc:B - sometimes it will freeze ( is my observation )

\n
", "votes": null, "replies": [ { "id": 2785982, "author_name": "valentinwerner", "author_url": "", "post_date": "05/01/2024 05:24:41", "content": "

I understood that, but that means our query builder will not be able to include OR statements, which is a heavy limitation..

\n

// just saw that they will fix it soon :)

", "votes": null, "replies": [ { "id": 2791944, "author_name": "tanghung64", "author_url": "", "post_date": "05/04/2024 00:24:08", "content": "

In the competition overview, it mentioned \"The metric notebook must finish running your queries in 60 minutes\". Is it possible that the statement with \"OR\" generated too much results, slowing down the queries and cause it to exceed the time limit?

", "votes": null, "replies": [ { "id": 2792365, "author_name": "valentinwerner", "author_url": "", "post_date": "05/04/2024 07:23:13", "content": "

Running them on the train index takes 0.3 seconds for the 10 test queries.

\n

This means the 2500 queries run in 75 seconds (given same time).

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2794605, "author_name": "benbla", "author_url": "", "post_date": "05/05/2024 12:04:35", "content": "

Did it work for you @valentinwerner and @seshurajup ?
\nI can just get the single cpc query working right now.

", "votes": null, "replies": [] } ] } ] }, { "id": 2793860, "author_name": "jazivxt", "author_url": "", "post_date": "05/05/2024 03:21:28", "content": "

The scoring for this competition seems flawed or they just want us to fix their whoosh issues because there are only a few valid submissions from a big community and there are little to no responses from competition admins on submission issues.

", "votes": null, "replies": [ { "id": 2793918, "author_name": "seshurajup", "author_url": "", "post_date": "05/05/2024 04:10:14", "content": "

@jazivxt already Kaggle Staff answered about it, in other discussion i.e will be fixed in this week

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2785222": "Hello everybody, I am struggling to get my first submission out. I learned from prior challenges that I don't just want to copy the currently available notebooks and am trying to get my own version going.\n\nMy \"hello world\" attempt is using key word from the title:\n\n queries = []\n for target in test.publication_number.to_list():\n patent_data = metadata[metadata.publication_number == target].iloc[0].parquet\n\n title = get_title(patent_data, target)\n cleaned = title.translate(str.maketrans('', '', string.punctuation))\n cleaned = cleaned.translate(str.maketrans('', '', string.digits))\n\n # make query with unique words, excluding stop words\n res = list(set(cleaned.split())-stops)\n if len(res) > 25: # reduce list to only 25 words -> 49 tokens\n res = res[:25]\n query = \"ti:\" + \" OR ti:\".join(res)\n\n queryValidator = whoosh_utils.QueryValidator()\n\n try: # as seen in https://www.kaggle.com/code/seshurajup/lb-0-11-uspto-single-cpc-query\n queryValidator.validate_query(query)\n except:\n query = \"ti:device\"\n if whoosh_utils.count_query_tokens(query) > 50:\n query = \"ti:mobile\"\n\n queries.append(query)\n gc.collect() \n\nI am getting a DataFrame that looks appropriate: \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8504824%2Fd66295b0bfd0aeb099f63e46795b1a46%2Fquery%20result.png?generation=1714497061379059&alt=media)\n\nHowever, after 2-3 hours of running I am getting the error \"Submission scoring error\"\n\nAm I missing something obvious? All the queries done on the fake test set are working if I query them on the training index..", "2785225": "valentinwerner, check with my verify notebook, with your title OR queries, if u keep one key. it won't freeze, as you add OR for some queries it will freeze.", "2785284": "Ah I didnt fully read it before... this makes no sense but explains my issue. It is extremely frustrating and will block query generation enormously.\n\nThis basically means we have less effective tokens per query and models have to learn to write these new/weird query with (a and a) OR (b and b)", "2785353": "i mean @valentinwerner \n> for cpc:A - we will get results for all queries \n> for cpc:A OR cpc:B - sometimes it will freeze ( is my observation )", "2785982": "I understood that, but that means our query builder will not be able to include OR statements, which is a heavy limitation..\n\n// just saw that they will fix it soon :)", "2791944": "In the competition overview, it mentioned \"The metric notebook must finish running your queries in 60 minutes\". Is it possible that the statement with \"OR\" generated too much results, slowing down the queries and cause it to exceed the time limit?", "2792365": "Running them on the train index takes 0.3 seconds for the 10 test queries. \n\nThis means the 2500 queries run in 75 seconds (given same time).", "2793860": "The scoring for this competition seems flawed or they just want us to fix their whoosh issues because there are only a few valid submissions from a big community and there are little to no responses from competition admins on submission issues.", "2793918": "jazivxt already Kaggle Staff answered about it, in other discussion i.e will be fixed in this week", "2794605": "Did it work for you @valentinwerner and @seshurajup ?\nI can just get the single cpc query working right now." }, "source": "meta" }