Hi Team,\n i have a question related to data preprocessing during inference. the nature of the competition is to submit the trained model alone. in this case, how to specify or carryout data preprocessing, normalization if any, applied while training.
\n\nthanks\nyuvaram
", "messages": [ { "id": "912657", "postDate": "07/02/2020 16:21:18", "content": "Hi Team,\n i have a question related to data preprocessing during inference. the nature of the competition is to submit the trained model alone. in this case, how to specify or carryout data preprocessing, normalization if any, applied while training.
\n\nthanks\nyuvaram
", "rawMarkdown": "Hi Team,\n i have a question related to data preprocessing during inference. the nature of the competition is to submit the trained model alone. in this case, how to specify or carryout data preprocessing, normalization if any, applied while training. \n\nthanks\nyuvaram", "votes": null }, { "id": "914073", "postDate": "07/03/2020 15:58:03", "content": "If your model needs any type of data preprocessing, you should export the preprocessing operations within the submitted SavedModel, together with the rest of your model.
\n\nThe SavedModel should take a [H,W,3] uint8 tensor as input (see here for an input example), and the output should be a dict containing key 'global_descriptor' mapped to a [D] float tensor (see here for an output example).
\n\nAnother concrete example is the baseline model we released. You can also try out the SavedModel command-line interface to inspect this model, which can show you the above-mentioned input and output structure.
", "rawMarkdown": "If your model needs any type of data preprocessing, you should export the preprocessing operations within the submitted SavedModel, together with the rest of your model.\n\nThe SavedModel should take a [H,W,3] uint8 tensor as input (see [here](https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/model/export_global_model.py#L112) for an input example), and the output should be a dict containing key 'global_descriptor' mapped to a [D] float tensor (see [here](https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/model/export_global_model.py#L106) for an output example).\n\nAnother concrete example is the [baseline model](https://www.kaggle.com/camaskew/baseline-landmark-retrieval-model) we released. You can also try out the [SavedModel command-line interface](https://www.tensorflow.org/guide/saved_model#details_of_the_savedmodel_command_line_interface) to inspect this model, which can show you the above-mentioned input and output structure.", "votes": null }, { "id": "914188", "postDate": "07/03/2020 17:05:43", "content": "hi @andrefaraujo . can you provide a simpler code snippet on how to map a model input and output into the required format . the github code is too huge to follow :P
", "rawMarkdown": "hi @andrefaraujo . can you provide a simpler code snippet on how to map a model input and output into the required format . the github code is too huge to follow :P", "votes": null }, { "id": "914399", "postDate": "07/03/2020 20:44:16", "content": "@andrefaraujo I think this should be put up in a place where everyone can see it. I have spent 6-7 hours trying to work out this. No one is going to adopt TF2.0 if it isn't made user friendly and if there aren't any documentations available.
", "rawMarkdown": "andrefaraujo I think this should be put up in a place where everyone can see it. I have spent 6-7 hours trying to work out this. No one is going to adopt TF2.0 if it isn't made user friendly and if there aren't any documentations available.", "votes": null }, { "id": "914421", "postDate": "07/03/2020 21:07:55", "content": "@mayukh18 do you have a solution for mapping the layers . is it possible to share code snippet.
", "rawMarkdown": "mayukh18 do you have a solution for mapping the layers . is it possible to share code snippet.", "votes": null }, { "id": "914429", "postDate": "07/03/2020 21:25:25", "content": "Currently working on it. Haven't got anything concrete yet.
", "rawMarkdown": "Currently working on it. Haven't got anything concrete yet.", "votes": null }, { "id": "916081", "postDate": "07/05/2020 10:39:05", "content": "@andrefaraujo can we have a batch dimension in the input (None,None,None,3) . is this acceptable?
", "rawMarkdown": "andrefaraujo can we have a batch dimension in the input (None,None,None,3) . is this acceptable?", "votes": null }, { "id": "916489", "postDate": "07/05/2020 17:42:45", "content": "No, the input must be of shape [None,None,3].
\n\nTo export a model that is trained with [None,None,None,3] shapes, you can follow a recipe very similar to the one used in the code linked in the \"second submission\" post. Essentially, just define the exported function to have an input [None,None,3] then expand the dimensions. Something like this:
\n\npython\n@tf.function(input_signature=[\n tf.TensorSpec(shape=[None, None, 3], dtype=tf.uint8, name='input_image')\n ])\ndef YourModelExportFunction(self, input_image):\n image_tensor = tf.expand_dims(image_tensor, 0, name='image/expand_dims')\n extracted_feature = your_function_that_computes_embedding(image_tensor)\n # Note: you may need to reshape \"extracted_feature\" in order to make it have \n # shape (D,).\n named_output_tensors[] = {}\n named_output_tensors['global_descriptor'] = tf.identity(\n extracted_feature, name='global_descriptor')\n return named_output_tensors\n
For the codebase I mentioned, you can see that this is done here.
", "rawMarkdown": "No, the input must be of shape [None,None,3].\n\nTo export a model that is trained with [None,None,None,3] shapes, you can follow a recipe very similar to the one used in the code linked in the [\"second submission\" post](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350). Essentially, just define the exported function to have an input [None,None,3] then expand the dimensions. Something like this:\n\n```python\n@tf.function(input_signature=[\n tf.TensorSpec(shape=[None, None, 3], dtype=tf.uint8, name='input_image')\n ])\ndef YourModelExportFunction(self, input_image):\n image_tensor = tf.expand_dims(image_tensor, 0, name='image/expand_dims')\n extracted_feature = your_function_that_computes_embedding(image_tensor)\n # Note: you may need to reshape \"extracted_feature\" in order to make it have \n # shape (D,).\n named_output_tensors[] = {}\n named_output_tensors['global_descriptor'] = tf.identity(\n extracted_feature, name='global_descriptor')\n return named_output_tensors\n```\n\nFor the codebase I mentioned, you can see that this is done [here](https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/model/export_model_utils.py#L213).", "votes": null }, { "id": "916979", "postDate": "07/06/2020 06:51:02", "content": "soo.. there is no requirement of \"named_output_tensors\" dict now, we can return the output directly..
", "rawMarkdown": "soo.. there is no requirement of \"named_output_tensors\" dict now, we can return the output directly..", "votes": null }, { "id": "917548", "postDate": "07/06/2020 15:53:26", "content": "Sorry, I forgot to include that part -- I was emphasizing the input handling. Just updated it.
", "rawMarkdown": "Sorry, I forgot to include that part -- I was emphasizing the input handling. Just updated it.", "votes": null }, { "id": "918209", "postDate": "07/07/2020 04:44:58", "content": "why even the failed submissions takes a long time. Could you please look into it?
", "rawMarkdown": "why even the failed submissions takes a long time. Could you please look into it?", "votes": null }, { "id": "918667", "postDate": "07/07/2020 11:45:08", "content": "I am trying to implement this, but I run into a problem:\nYou can't save a tf2 model with subclassed keras model.
\n\nSource:\nhttps://stackoverflow.com/questions/51806852/cant-save-custom-subclassed-model
\n\nIs there any way to use Keras for this competition?
", "rawMarkdown": "I am trying to implement this, but I run into a problem:\nYou can't save a tf2 model with subclassed keras model.\n\nSource:\nhttps://stackoverflow.com/questions/51806852/cant-save-custom-subclassed-model\n\nIs there any way to use Keras for this competition?", "votes": null }, { "id": "919028", "postDate": "07/07/2020 17:04:48", "content": "Yes, absolutely, Keras can be directly used. As a matter of fact, the example in our second submission post is using a Keras model.
", "rawMarkdown": "Yes, absolutely, Keras can be directly used. As a matter of fact, the example in our [second submission post](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350) is using a Keras model.", "votes": null }, { "id": "919067", "postDate": "07/07/2020 17:31:08", "content": "Sorry, you are right. Managed to get it to work. I'll make a notebook for others.
", "rawMarkdown": "Sorry, you are right. Managed to get it to work. I'll make a notebook for others.", "votes": null } ], "comments": [ { "id": 914073, "author_name": "andrefaraujo", "author_url": "", "post_date": "07/03/2020 15:58:03", "content": "If your model needs any type of data preprocessing, you should export the preprocessing operations within the submitted SavedModel, together with the rest of your model.
\n\nThe SavedModel should take a [H,W,3] uint8 tensor as input (see here for an input example), and the output should be a dict containing key 'global_descriptor' mapped to a [D] float tensor (see here for an output example).
\n\nAnother concrete example is the baseline model we released. You can also try out the SavedModel command-line interface to inspect this model, which can show you the above-mentioned input and output structure.
", "votes": null, "replies": [ { "id": 914188, "author_name": "yuvaramsingh", "author_url": "", "post_date": "07/03/2020 17:05:43", "content": "hi @andrefaraujo . can you provide a simpler code snippet on how to map a model input and output into the required format . the github code is too huge to follow :P
", "votes": null, "replies": [] }, { "id": 914399, "author_name": "mayukh18", "author_url": "", "post_date": "07/03/2020 20:44:16", "content": "@andrefaraujo I think this should be put up in a place where everyone can see it. I have spent 6-7 hours trying to work out this. No one is going to adopt TF2.0 if it isn't made user friendly and if there aren't any documentations available.
", "votes": null, "replies": [] }, { "id": 914421, "author_name": "yuvaramsingh", "author_url": "", "post_date": "07/03/2020 21:07:55", "content": "@mayukh18 do you have a solution for mapping the layers . is it possible to share code snippet.
", "votes": null, "replies": [] }, { "id": 914429, "author_name": "mayukh18", "author_url": "", "post_date": "07/03/2020 21:25:25", "content": "Currently working on it. Haven't got anything concrete yet.
", "votes": null, "replies": [] }, { "id": 916081, "author_name": "yuvaramsingh", "author_url": "", "post_date": "07/05/2020 10:39:05", "content": "@andrefaraujo can we have a batch dimension in the input (None,None,None,3) . is this acceptable?
", "votes": null, "replies": [] }, { "id": 916489, "author_name": "andrefaraujo", "author_url": "", "post_date": "07/05/2020 17:42:45", "content": "No, the input must be of shape [None,None,3].
\n\nTo export a model that is trained with [None,None,None,3] shapes, you can follow a recipe very similar to the one used in the code linked in the \"second submission\" post. Essentially, just define the exported function to have an input [None,None,3] then expand the dimensions. Something like this:
\n\npython\n@tf.function(input_signature=[\n tf.TensorSpec(shape=[None, None, 3], dtype=tf.uint8, name='input_image')\n ])\ndef YourModelExportFunction(self, input_image):\n image_tensor = tf.expand_dims(image_tensor, 0, name='image/expand_dims')\n extracted_feature = your_function_that_computes_embedding(image_tensor)\n # Note: you may need to reshape \"extracted_feature\" in order to make it have \n # shape (D,).\n named_output_tensors[] = {}\n named_output_tensors['global_descriptor'] = tf.identity(\n extracted_feature, name='global_descriptor')\n return named_output_tensors\n
For the codebase I mentioned, you can see that this is done here.
", "votes": null, "replies": [] }, { "id": 916979, "author_name": "udaygurugubelli", "author_url": "", "post_date": "07/06/2020 06:51:02", "content": "soo.. there is no requirement of \"named_output_tensors\" dict now, we can return the output directly..
", "votes": null, "replies": [] }, { "id": 917548, "author_name": "andrefaraujo", "author_url": "", "post_date": "07/06/2020 15:53:26", "content": "Sorry, I forgot to include that part -- I was emphasizing the input handling. Just updated it.
", "votes": null, "replies": [] }, { "id": 918209, "author_name": "udaygurugubelli", "author_url": "", "post_date": "07/07/2020 04:44:58", "content": "why even the failed submissions takes a long time. Could you please look into it?
", "votes": null, "replies": [] }, { "id": 918667, "author_name": "reflexion", "author_url": "", "post_date": "07/07/2020 11:45:08", "content": "I am trying to implement this, but I run into a problem:\nYou can't save a tf2 model with subclassed keras model.
\n\nSource:\nhttps://stackoverflow.com/questions/51806852/cant-save-custom-subclassed-model
\n\nIs there any way to use Keras for this competition?
", "votes": null, "replies": [] }, { "id": 919028, "author_name": "andrefaraujo", "author_url": "", "post_date": "07/07/2020 17:04:48", "content": "Yes, absolutely, Keras can be directly used. As a matter of fact, the example in our second submission post is using a Keras model.
", "votes": null, "replies": [] }, { "id": 919067, "author_name": "reflexion", "author_url": "", "post_date": "07/07/2020 17:31:08", "content": "Sorry, you are right. Managed to get it to work. I'll make a notebook for others.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "912657": "Hi Team,\n i have a question related to data preprocessing during inference. the nature of the competition is to submit the trained model alone. in this case, how to specify or carryout data preprocessing, normalization if any, applied while training. \n\nthanks\nyuvaram", "914073": "If your model needs any type of data preprocessing, you should export the preprocessing operations within the submitted SavedModel, together with the rest of your model.\n\nThe SavedModel should take a [H,W,3] uint8 tensor as input (see [here](https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/model/export_global_model.py#L112) for an input example), and the output should be a dict containing key 'global_descriptor' mapped to a [D] float tensor (see [here](https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/model/export_global_model.py#L106) for an output example).\n\nAnother concrete example is the [baseline model](https://www.kaggle.com/camaskew/baseline-landmark-retrieval-model) we released. You can also try out the [SavedModel command-line interface](https://www.tensorflow.org/guide/saved_model#details_of_the_savedmodel_command_line_interface) to inspect this model, which can show you the above-mentioned input and output structure.", "914188": "hi @andrefaraujo . can you provide a simpler code snippet on how to map a model input and output into the required format . the github code is too huge to follow :P", "914399": "andrefaraujo I think this should be put up in a place where everyone can see it. I have spent 6-7 hours trying to work out this. No one is going to adopt TF2.0 if it isn't made user friendly and if there aren't any documentations available.", "914421": "mayukh18 do you have a solution for mapping the layers . is it possible to share code snippet.", "914429": "Currently working on it. Haven't got anything concrete yet.", "916081": "andrefaraujo can we have a batch dimension in the input (None,None,None,3) . is this acceptable?", "916489": "No, the input must be of shape [None,None,3].\n\nTo export a model that is trained with [None,None,None,3] shapes, you can follow a recipe very similar to the one used in the code linked in the [\"second submission\" post](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350). Essentially, just define the exported function to have an input [None,None,3] then expand the dimensions. Something like this:\n\n```python\n@tf.function(input_signature=[\n tf.TensorSpec(shape=[None, None, 3], dtype=tf.uint8, name='input_image')\n ])\ndef YourModelExportFunction(self, input_image):\n image_tensor = tf.expand_dims(image_tensor, 0, name='image/expand_dims')\n extracted_feature = your_function_that_computes_embedding(image_tensor)\n # Note: you may need to reshape \"extracted_feature\" in order to make it have \n # shape (D,).\n named_output_tensors[] = {}\n named_output_tensors['global_descriptor'] = tf.identity(\n extracted_feature, name='global_descriptor')\n return named_output_tensors\n```\n\nFor the codebase I mentioned, you can see that this is done [here](https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/model/export_model_utils.py#L213).", "916979": "soo.. there is no requirement of \"named_output_tensors\" dict now, we can return the output directly..", "917548": "Sorry, I forgot to include that part -- I was emphasizing the input handling. Just updated it.", "918209": "why even the failed submissions takes a long time. Could you please look into it?", "918667": "I am trying to implement this, but I run into a problem:\nYou can't save a tf2 model with subclassed keras model.\n\nSource:\nhttps://stackoverflow.com/questions/51806852/cant-save-custom-subclassed-model\n\nIs there any way to use Keras for this competition?", "919028": "Yes, absolutely, Keras can be directly used. As a matter of fact, the example in our [second submission post](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163350) is using a Keras model.", "919067": "Sorry, you are right. Managed to get it to work. I'll make a notebook for others." }, "source": "meta" }