\n
![\"Selection-999-1248\"](\"https://i.ibb.co/g6nThdQ/Selection-999-1248.png\")
i am interested in this competition.
\ni am familiar with pytorch and onnx.
[updated post]
\nthis thread is for conversion problems and solution.
\n[1] https://www.kaggle.com/code/myso1987/gislr-pytorch-tflite-baseline
\n[2] https://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission
[3] working tutorial
\n[old post]
\n
\n
\n
\n
i am interested in this competition.
\ni am familiar with pytorch and onnx.
[updated post]
\nthis thread is for conversion problems and solution.
\n[1] https://www.kaggle.com/code/myso1987/gislr-pytorch-tflite-baseline
\n[2] https://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission
[3] working tutorial
\n[old post]
\n
\n
\n
\n
Hi, you can easily do the conversion from pytorch to onnx and
\nthen from onnx to tensorflow
\nand finally tensorflow to tensorflow lite.
Please find the sample notebooks from other kagglers:
\n@jarvisai7 : https://www.kaggle.com/code/jarvisai7/tensorflow-pytorch-to-tflite
\n@mhetrerajat : https://www.kaggle.com/code/mhetrerajat/pytorch-tflite
hope this helps you..
", "rawMarkdown": "Hi, you can easily do the conversion from pytorch to onnx and\nthen from onnx to tensorflow \nand finally tensorflow to tensorflow lite. \n\nPlease find the sample notebooks from other kagglers: \n@jarvisai7 : https://www.kaggle.com/code/jarvisai7/tensorflow-pytorch-to-tflite\n@mhetrerajat : https://www.kaggle.com/code/mhetrerajat/pytorch-tflite\n\nhope this helps you..", "votes": null }, { "id": "2163269", "postDate": "02/28/2023 17:38:37", "content": "this should discussion should be pinned :D
", "rawMarkdown": "this should discussion should be pinned :D", "votes": null }, { "id": "2163305", "postDate": "02/28/2023 17:55:45", "content": "It would be better to post here https://www.kaggle.com/competitions/asl-signs/discussion/390008
", "rawMarkdown": "It would be better to post here https://www.kaggle.com/competitions/asl-signs/discussion/390008", "votes": null }, { "id": "2163887", "postDate": "03/01/2023 06:32:56", "content": "[error] tf.erf op error
\nif you are using pytorch mutihead attention, you may encounter this error.
\nbecuase onnx don't have gelu, it implement gelu with erf
\nsee : https://user-images.githubusercontent.com/103593150/220924683-5da1ca1b-0864-43fd-9ec3-804d8be8b2a1.png
\nhttps://github.com/onnx/onnx/issues/4933
tf has tf.erf
\ntflite don't have erf.
\nbut tflite does have gelu.
i haven't figure out how to solve this. if you know, please suggest. thanks!
\ntemproarily solution is to use relu
[error] tflite output numerical differences
\n
\n
\n
\n
i note that the cause is due:
\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]\nafter a day of work, my submission still failed.
\ni wonder if anyone can help:
https://www.kaggle.com/code/hengck23/failed-pytorch-to-onnx-to-tflite
", "rawMarkdown": "after a day of work, my submission still failed.\ni wonder if anyone can help:\n\nhttps://www.kaggle.com/code/hengck23/failed-pytorch-to-onnx-to-tflite", "votes": null }, { "id": "2164226", "postDate": "03/01/2023 12:28:37", "content": "[error] runtime error due to dynamic shape
\ni think the issue is dynamic input shape.
\ni used up my submission, i will report it tmr
1 follow the notebook:
\nhttps://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission
(i do not know why the combined code failed although the code is the same)
\nwith this i can make pytorch transformer conversion on tflite to work.
", "rawMarkdown": "[error] runtime error due to dynamic shape\ni think the issue is dynamic input shape.\ni used up my submission, i will report it tmr\n\n1 follow the notebook:\nhttps://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission\n\n- split your model into two. \n- one for just dynamic input size \n- another for prediction\n\n(i do not know why the combined code failed although the code is the same)\n\nwith this i can make pytorch transformer conversion on tflite to work.", "votes": null }, { "id": "2166040", "postDate": "03/02/2023 15:36:45", "content": "Have you made transformer submission successfully ？I have an error :\"TF Select ops: Range\".
\nAfter I added following setting, it looks good to run in the notebook, but still failed when submitting.
\nconverter.target_spec.supported_ops = [\n tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.\n tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n]
my suggestion is:
\ni have made the pytorch transformer sucessfully
", "rawMarkdown": "my suggestion is:\n- try not to use converter.target_spec.supported_ops\n- run a few thousands of parquet file at development in notebook (not just one or two)\n\ni have made the pytorch transformer sucessfully", "votes": null }, { "id": "2166720", "postDate": "03/03/2023 01:10:41", "content": "Thanks, I removed ops tflite not supported and reran the code without converter.target_spec.supported_ops. It works.
", "rawMarkdown": "Thanks, I removed ops tflite not supported and reran the code without converter.target_spec.supported_ops. It works.", "votes": null }, { "id": "2166726", "postDate": "03/03/2023 01:21:37", "content": "i find that if onnx works, onnx-to-tflite should work.
\nhere are some magic tools to do network surgery on onnx:
\nhttps://github.com/PINTO0309/simple-onnx-processing-tools
\nhttps://github.com/PINTO0309/onnx2tf
many other onnx, tflite tools:
\nhttps://github.com/PINTO0309?tab=repositories
[error] \"Can't reduce on dim with value of 0 if 'keepdims' is false\" in onnx runtime
\nfor computation od mean, std as feature input, you may end up with no points after you remove the nan values.
\n[solution] see: https://github.com/microsoft/onnxruntime/issues/7563
better solution
\ndef pre_process(xyz):\n idx_range_face = (0, 468)\n idx_range_hand_left = (468, 489)\n idx_range_pose = (489, 522)\n idx_range_hand_right = (522, 543)\n\n x_face = get_flat_features(xyz, idx_range_face)\n x_hand_left = get_flat_features(xyz, idx_range_hand_left)\n x_pose = get_flat_features(xyz, idx_range_pose)\n x_hand_right = get_flat_features(xyz, idx_range_hand_right)\n\n x_hand_left = x_hand_left[~torch.any(torch.isnan(x_hand_left), dim=1), :]\n x_hand_right = x_hand_right[~torch.any(torch.isnan(x_hand_right), dim=1), :]\n\n if len(x_hand_left)==0:\n x_hand_left_mean = torch.zeros(63)\n x_hand_left_std = torch.zeros(63)\n else:\n x_hand_left_mean = torch.mean(x_hand_left, 0)\n x_hand_left_std = torch.std(x_hand_left, 0)\n\n if len(x_hand_right)==0:\n x_hand_right_mean = torch.zeros(63)\n x_hand_right_std = torch.zeros(63)\n else:\n x_hand_right_mean = torch.mean(x_hand_right, 0)\n x_hand_right_std = torch.std(x_hand_right, 0)\n\n x_face_mean = torch.mean(x_face, 0)#, keepdims=True).squeeze(0)\n x_pose_mean = torch.mean(x_pose, 0)\n\n x_face_std = torch.std(x_face, 0)\n x_pose_std = torch.std(x_pose, 0)\n\n x_features = torch.cat(\n [\n x_face_mean,\n x_hand_left_mean,\n x_pose_mean,\n x_hand_right_mean,\n x_face_std,\n x_hand_left_std,\n x_pose_std,\n x_hand_right_std,\n ],\n dim=0,\n )\n\n x_features = torch.where(\n torch.isnan(x_features), torch.tensor(0.0, dtype=torch.float32), x_features\n )\n #x_features = x_features.unsqueeze(0)\n\n return x_features\nthen use:
\n torch.onnx.export(\n torch.jit.script(input_net),\n...\njit.script will explicitly unrolled all if else conditions
", "rawMarkdown": "better solution\n\n```\n\ndef pre_process(xyz):\n idx_range_face = (0, 468)\n idx_range_hand_left = (468, 489)\n idx_range_pose = (489, 522)\n idx_range_hand_right = (522, 543)\n\n x_face = get_flat_features(xyz, idx_range_face)\n x_hand_left = get_flat_features(xyz, idx_range_hand_left)\n x_pose = get_flat_features(xyz, idx_range_pose)\n x_hand_right = get_flat_features(xyz, idx_range_hand_right)\n\n x_hand_left = x_hand_left[~torch.any(torch.isnan(x_hand_left), dim=1), :]\n x_hand_right = x_hand_right[~torch.any(torch.isnan(x_hand_right), dim=1), :]\n\n if len(x_hand_left)==0:\n x_hand_left_mean = torch.zeros(63)\n x_hand_left_std = torch.zeros(63)\n else:\n x_hand_left_mean = torch.mean(x_hand_left, 0)\n x_hand_left_std = torch.std(x_hand_left, 0)\n\n if len(x_hand_right)==0:\n x_hand_right_mean = torch.zeros(63)\n x_hand_right_std = torch.zeros(63)\n else:\n x_hand_right_mean = torch.mean(x_hand_right, 0)\n x_hand_right_std = torch.std(x_hand_right, 0)\n\n x_face_mean = torch.mean(x_face, 0)#, keepdims=True).squeeze(0)\n x_pose_mean = torch.mean(x_pose, 0)\n\n x_face_std = torch.std(x_face, 0)\n x_pose_std = torch.std(x_pose, 0)\n\n x_features = torch.cat(\n [\n x_face_mean,\n x_hand_left_mean,\n x_pose_mean,\n x_hand_right_mean,\n x_face_std,\n x_hand_left_std,\n x_pose_std,\n x_hand_right_std,\n ],\n dim=0,\n )\n\n x_features = torch.where(\n torch.isnan(x_features), torch.tensor(0.0, dtype=torch.float32), x_features\n )\n #x_features = x_features.unsqueeze(0)\n\n return x_features\n```\n\nthen use:\n\n```\n torch.onnx.export(\n torch.jit.script(input_net),\n...\n\n```\njit.script will explicitly unrolled all if else conditions", "votes": null }, { "id": "2171172", "postDate": "03/06/2023 15:03:33", "content": "[error] onnx to tflite for nn.multiHeadAttention
\nSome ops are not supported by the native TFLite runtime, you can enable TF kernels fallback using TF Select. See instructions: https://www.tensorflow.org/lite/guide/ops_select \nTF Select ops: Cast, RealDiv\nDetails:\n tf.Cast(tensor<f64>) -> (tensor<i64>) : {Truncate = false, device = \"\"}\n tf.Cast(tensor<i64>) -> (tensor<f64>) : {Truncate = false, device = \"\"}\n tf.RealDiv(tensor<f64>, tensor<f64>) -> (tensor<f64>) : {device = \"\"}\nin computation of the head dim, we need to divide the embed dim by the num of head.
\n
onnx graph
\n![\"Selection-999-1247\"](\"https://i.ibb.co/SwpK0p5/Selection-999-1247.png\")
tflite graph
\nthis division should be constant, since we know what is the embed dim and num of head. For unknown reason, tflite convert failed to detect this and uses real division.
\n[solution]
\nedit the onnx graph and use fix const values when reshape
\nhttps://github.com/ZhangGe6/onnx-modifier
or rewrite the forward function nn.multiHeadAttention with constant values in reshape
e.g.\nhead_dim=128\nnum_head=8 \n\n def __init__(self,...):\n ...\n self.mha = nn.MultiheadAttention(...)\n\n def forward(self,x):\n ...\n k = F.linear(x, self.mha.in_proj_weight[1024:2048], self.mha.in_proj_bias[1024:2048])\n v = F.linear(x, self.mha.in_proj_weight[2048:], self.mha.in_proj_bias[2048:])\n\n q = q.reshape(-1, 8, 128).permute(1, 0, 2)\n k = k.reshape(-1, 8, 128).permute(1, 0, 2)\n v = v.reshape(-1, 8, 128).permute(1, 0, 2)\n dot = torch.matmul(q, k.transpose(-1, -2)) * (1/128**0.5) # H L L\nI am the author of onnx2tf.
\nonnx2tf can convert almost any model except RNNs. There is also an option to make all Flex operations as harmless as possible. GELU conversions are also available.
\nonnx2tf -i xxxx.onnx -rtpo Erf -cotof -cotoa 1e-3\nI hope this will be of help to all of you.
\nBy the way, I am a hobby programmer not doing Kaggle.
Thanks for the comments.
\nYour tools has been a great help in the competition! 😊
[error] onnx tp tflite conversion fail for using F.pad
\n[solution]
\nsee https://stackoverflow.com/questions/75016155/converting-onnx-model-to-tensorflow-fails
you need to specify fully for mode and value
\n reduced_xyz = F.pad(reduced_xyz,[0,0,0,0,0,1], mode='constant', value=0.)\n[error] tfRealDiv and tfCast are not naive tflite ops,e.g. if i want to do center crop, i need the division to compute the offset
\n[solution]
offset = (np.arange(1000)-max_length)//2 #assume 1000 is the longest video\noffset = np.clip(offset,0, 1000).tolist()\n\nclass InputNet(nn.Module):\n\n def __init__(self, ):\n super().__init__()\n self.max_length = max_length\n self.offset = nn.Parameter(torch.LongTensor(offset),requires_grad=False)\n\n def forward(self, xyz):\n L = len(xyz)\n if L>self.max_length:\n #xyz = xyz[:self.max_length] #first\n #xyz = xyz[-self.max_length:] #last\n\n i = self.offset[L]\n xyz = xyz[i:i+self.max_length] #center\ni have 2 sub-network. one for pre-processing (e.g. shape nomalisation), another for prediction (transformer).
\ni analyse the tflite runtime memory for the sub-network one by one.
\n(i was very baffled why my transformer network with disk size of just 2 mb would consume 200 mb ram at peak)
for unknown reason, it seems that the problem i have is the pre-processing network.
\nthis is unexpected for me.
i comment and uncoment line by line to find out the issue.
\nfor example:
from tflite benchmark tool\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 222.941\n\n#####\n\n not_nan_xyz = xyz[~torch.isnan(xyz)]\n if len(not_nan_xyz) != 0: \n xyz = xyz- torch.mean(xyz[~torch.isnan(xyz)],0, keepdim=True)\njust this few lines brew out my memory by 200MB???
\n(input is xyz = 500,543,3)
i check the graph of pre-processing network. it is quite messy
\nit seems that while pytorch-to-onnx-to-tflite conversion works good for the conv, linear, transformer, etc layer,
\nthe conversion is very bad for simple operation like \"custom tensor manipulation\" ????
next i would check if i write my preprossing in keras would do a better job.
", "rawMarkdown": "i have 2 sub-network. one for pre-processing (e.g. shape nomalisation), another for prediction (transformer). \ni analyse the tflite runtime memory for the sub-network one by one. \n(i was very baffled why my transformer network with disk size of just 2 mb would consume 200 mb ram at peak)\n\n---\n\nfor unknown reason, it seems that the problem i have is the pre-processing network.\nthis is unexpected for me.\n\n\ni comment and uncoment line by line to find out the issue.\nfor example:\n\n```\nfrom tflite benchmark tool\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 222.941\n\n#####\n\n not_nan_xyz = xyz[~torch.isnan(xyz)]\n if len(not_nan_xyz) != 0: \n xyz = xyz- torch.mean(xyz[~torch.isnan(xyz)],0, keepdim=True)\n```\n\njust this few lines brew out my memory by 200MB???\n(input is xyz = 500,543,3)\n\ni check the graph of pre-processing network. it is quite messy\n\nit seems that while pytorch-to-onnx-to-tflite conversion works good for the conv, linear, transformer, etc layer,\nthe conversion is very bad for simple operation like \"custom tensor manipulation\" ????\n\n---\n\nnext i would check if i write my preprossing in keras would do a better job.", "votes": null }, { "id": "2176766", "postDate": "03/10/2023 22:15:04", "content": "Is there a problem with consuming 200MB ram? That just sounds like it's copying around a bit of memory while processing data chunks. I don't think it would make your submission fail?
\nWhat problem does it cause? Or you are just trying to optimize for faster operation?
", "rawMarkdown": "Is there a problem with consuming 200MB ram? That just sounds like it's copying around a bit of memory while processing data chunks. I don't think it would make your submission fail?\n\nWhat problem does it cause? Or you are just trying to optimize for faster operation?", "votes": null }, { "id": "2176801", "postDate": "03/10/2023 23:22:29", "content": "here is the comparsion.
\none has to be careful with pytorch to tflite conversion.
![\"https://i.ibb.co/PCS6NsJ/Selection-999-1336.png\"](\"https://i.ibb.co/PCS6NsJ/Selection-999-1336.png\")
@roberthatch
\n\"Is there a problem with consuming 200MB ram?\"
your submission will fail with out-of-memory submission error.
\ni think although the restiction is simplified to \"40 mb on disk\",
\nbut 40mb on memory still apply at submission script (which totally make sense)
if i can solve the issue, it means that trasnformer can take in input length (num of frames) up to 512 under memory and time constraint.
\nThis improve accuracy (estimated LB 0.67 for one fold) and no need for complicated pre-processing.
memory size of xyz array of varying length:
\n''' num_frame, file, mem (mb)\n 0, train_landmark_files/16069/1558159851.parquet , 537, 6.87, 3.34\n 1, train_landmark_files/16069/2399841238.parquet , 299, 4.32, 1.86\n 2, train_landmark_files/61333/117070402.parquet , 271, 4.00, 1.68\n 3, train_landmark_files/27610/3762968283.parquet , 255, 3.75, 1.58\n 4, train_landmark_files/26734/2959425762.parquet , 246, 3.63, 1.53\n 5, train_landmark_files/18796/2947969142.parquet , 238, 3.57, 1.48\n 6, train_landmark_files/28656/2467637816.parquet , 232, 3.43, 1.44\n\n 150, train_landmark_files/32319/2814895419.parquet , 62, 0.94, 0.39\n 151, train_landmark_files/27610/1173378699.parquet , 62, 0.93, 0.39\n 152, train_landmark_files/49445/195491904.parquet , 61, 0.90, 0.38\n 153, train_landmark_files/49445/1303464708.parquet , 61, 0.92, 0.38\nsurprising, a dummy identity keras model consume quite a lot of memory
\n class InputNet(tf.keras.layers.Layer):\n def __init__(self, ):\n super(InputNet, self).__init__()\n def call(self, xyz):\n x = xyz\n return x\n #dummy identity\n#cmd_str += f'./benchmark_model \\\\'\n# ...\n#cmd_str += f'--input_layer_shape={\"512,543,3\"} \\\\' #543\n\n\"doing nothing model\"\n...\nINFO: The input model file size (MB): 0.00066\n...\nINFO: Memory footprint delta from the start of the tool (MB): init=2.43359 overall=8.67578\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 13.1797\nINFO: Memory status at the end of exeution: #catch one spelling error bug!!!\nINFO: - VmRSS : 13 MB\nINFO: + RssAnnon : 7 MB\nINFO: + RssFile + RssShmem : 6 MB\nif i replace input net with keras version,
\ni can run pytorch transformer at input length 256 at 20 msec per video!
tflite benchmark tool gives:
\nINFO: Memory footprint delta from the start of the tool (MB): init=3.59375 overall=30.9961\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 35.3594\nvideo length=512 seems to be borderline case with
\nINFO: Memory footprint delta from the start of the tool (MB): init=3.64062 overall=43.9102\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.332\na smarter way is to use e.g. 256 for normalization, 512 for prediction. with that it would be possible to predict up to 512 length within the 40 mb limit
", "rawMarkdown": "if i replace input net with keras version,\ni can run pytorch transformer at input length 256 at 20 msec per video!\n\ntflite benchmark tool gives:\n\n```\nINFO: Memory footprint delta from the start of the tool (MB): init=3.59375 overall=30.9961\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 35.3594\n```\n\nvideo length=512 seems to be borderline case with \n```\nINFO: Memory footprint delta from the start of the tool (MB): init=3.64062 overall=43.9102\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.332\n```\n\na smarter way is to use e.g. 256 for normalization, 512 for prediction. with that it would be possible to predict up to 512 length within the 40 mb limit", "votes": null }, { "id": "2177148", "postDate": "03/11/2023 08:44:17", "content": "I'm not convinced that the peak memory footprint needs to be within 40MB.
\nIf I benchmark our model with a sequence length of 512 it has a peak memory consumption of 61.9 MB and we used full sequence lengths so it will have reached that a few times.
thanks for feedback. the limit is probably not 40mb (eg need to subtract off other setup) or they are using other tools, etc.
\nfor my case submission failed with memory error with peak reach more than 100 mb.
\ni will try the 512 48 mb and let u know the results
", "rawMarkdown": "thanks for feedback. the limit is probably not 40mb (eg need to subtract off other setup) or they are using other tools, etc. \n\nfor my case submission failed with memory error with peak reach more than 100 mb.\n\ni will try the 512 48 mb and let u know the results", "votes": null }, { "id": "2177364", "postDate": "03/11/2023 12:03:47", "content": "i confirm length 512 with 48 mb peak also passed the submission
", "rawMarkdown": "i confirm length 512 with 48 mb peak also passed the submission", "votes": null }, { "id": "2178698", "postDate": "03/12/2023 16:21:37", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "2179938", "postDate": "03/13/2023 13:46:45", "content": "After reading this discussion, I decide to use TensorFlow directly😪
", "rawMarkdown": "After reading this discussion, I decide to use TensorFlow directly😪", "votes": null }, { "id": "2180243", "postDate": "03/13/2023 17:27:38", "content": "We now just run os.path.getsizeof on the unzipped model checkpoint file. You can probably get much higher memory use if you're including the impact of loading the data.
i have been translating code from pytorch to keras and vice versa ….
\nwe need a chatgpt for that !!!!!
\n(or language model code pytorch to tf translator)
Have you found the solution by pytorch?
", "rawMarkdown": "Have you found the solution by pytorch?", "votes": null }, { "id": "2184604", "postDate": "03/16/2023 13:38:07", "content": "\"Have you found the solution by pytorch?\"
\neven if you use pytorch, you still need to convert to tflite.
\nhence just measure using os.path.getsizeof on the unzipped model checkpoint file as stated above
Hi! I guess I'm still missing something.
\nMy model size is around 4 MB, but it still gets a RAM error. So I rewrote the Input Net with the following way:
\n (nn.Module):\n ():\n ().__init__()\n self.max_length = \n\n ():\n xyz = xyz[:,:,:]\n xyz = xyz[:self.max_length]\n xyz = xyz - xyz[~torch.isnan(xyz)].mean(,keepdim=) \n xyz = xyz / xyz[~torch.isnan(xyz)].std(, keepdim=)\n\n LIP = [\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n ]\n\n lip = xyz[:, LIP]\n lhand = xyz[:, :]\n rhand = xyz[:, :]\n xyz = torch.cat([ \n lip,\n lhand,\n rhand,\n ], )\n xyz[torch.isnan(xyz)] = \n xyz \nAs you can see I'm using small max length and I also moved the string xyz = xyz[:self.max_length] from the end to the beginning. However I do still get the same error after around 25 minutes of submission time. I would be grateful for any advice.
this is due to inefficiency using the pytorch-onnx-tflite conversion pipeline.
\nif you use the tflite benchmark tool, this tflite uses 200MB runtime memory (that is the cause of out of memory)
the soslution is to write your InputNet() in keras. Then use the keras-tflite conversion pipeline.
", "rawMarkdown": "this is due to inefficiency using the pytorch-onnx-tflite conversion pipeline.\nif you use the tflite benchmark tool, this tflite uses 200MB runtime memory (that is the cause of out of memory)\n\nthe soslution is to write your InputNet() in keras. Then use the keras-tflite conversion pipeline.", "votes": null }, { "id": "2190079", "postDate": "03/21/2023 02:45:22", "content": "https://qiita.com/PINTO/items/ed06e03eb5c007c2e102
\nhere is another pytorch to onnx pipeline
\n(i haven;t tried it though)
\n PyTorch -> ONNX -> OpenVINO -> TensorFlow / Tensorflow Lite
the advantages:
\nI am the author of openvino2tensorflow.
\nThe article you are referring to was written by me but is very old and openvino2tensorflow has not been maintained for some time. I now allocate almost all of my private development time to onnx2tf.
\nIf you want to modify the model, for example, it would be easier for many people to modify the PyTorch source code directly. However, if you become proficient in this pipeline, you can modify the model to avoid bugs in the inference framework and take advantage of the high-performance optimization capabilities of OpenVINO's model optimizer.
\nSince openvino2tensorflow was an early tool when I first started writing programs, the source code should be very messy and often behave in a way that is difficult to understand.
", "rawMarkdown": "I am the author of openvino2tensorflow.\n\nThe article you are referring to was written by me but is very old and openvino2tensorflow has not been maintained for some time. I now allocate almost all of my private development time to onnx2tf.\n\nIf you want to modify the model, for example, it would be easier for many people to modify the PyTorch source code directly. However, if you become proficient in this pipeline, you can modify the model to avoid bugs in the inference framework and take advantage of the high-performance optimization capabilities of OpenVINO's model optimizer.\n\nSince openvino2tensorflow was an early tool when I first started writing programs, the source code should be very messy and often behave in a way that is difficult to understand.", "votes": null }, { "id": "2212711", "postDate": "04/07/2023 02:45:25", "content": "Hi OctOpus, how did you solve tf.Range issue? You just removed the function/method that is causing the problem?
", "rawMarkdown": "Hi OctOpus, how did you solve tf.Range issue? You just removed the function/method that is causing the problem?", "votes": null } ], "comments": [ { "id": 2162242, "author_name": "sairamarajupenmatsa", "author_url": "", "post_date": "02/28/2023 04:29:44", "content": "Hi, you can easily do the conversion from pytorch to onnx and
\nthen from onnx to tensorflow
\nand finally tensorflow to tensorflow lite.
Please find the sample notebooks from other kagglers:
\n@jarvisai7 : https://www.kaggle.com/code/jarvisai7/tensorflow-pytorch-to-tflite
\n@mhetrerajat : https://www.kaggle.com/code/mhetrerajat/pytorch-tflite
hope this helps you..
", "votes": null, "replies": [ { "id": 2178698, "author_name": "papapapaz", "author_url": "", "post_date": "03/12/2023 16:21:37", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 2163269, "author_name": "christofhenkel", "author_url": "", "post_date": "02/28/2023 17:38:37", "content": "this should discussion should be pinned :D
", "votes": null, "replies": [] }, { "id": 2163305, "author_name": "ipythonx", "author_url": "", "post_date": "02/28/2023 17:55:45", "content": "It would be better to post here https://www.kaggle.com/competitions/asl-signs/discussion/390008
", "votes": null, "replies": [] }, { "id": 2163887, "author_name": "hengck23", "author_url": "", "post_date": "03/01/2023 06:32:56", "content": "[error] tf.erf op error
\nif you are using pytorch mutihead attention, you may encounter this error.
\nbecuase onnx don't have gelu, it implement gelu with erf
\nsee : https://user-images.githubusercontent.com/103593150/220924683-5da1ca1b-0864-43fd-9ec3-804d8be8b2a1.png
\nhttps://github.com/onnx/onnx/issues/4933
tf has tf.erf
\ntflite don't have erf.
\nbut tflite does have gelu.
i haven't figure out how to solve this. if you know, please suggest. thanks!
\ntemproarily solution is to use relu
[error] tflite output numerical differences
\n
\n
\n
\n
i note that the cause is due:
\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]\ni find that if onnx works, onnx-to-tflite should work.
\nhere are some magic tools to do network surgery on onnx:
\nhttps://github.com/PINTO0309/simple-onnx-processing-tools
\nhttps://github.com/PINTO0309/onnx2tf
many other onnx, tflite tools:
\nhttps://github.com/PINTO0309?tab=repositories
I am the author of onnx2tf.
\nonnx2tf can convert almost any model except RNNs. There is also an option to make all Flex operations as harmless as possible. GELU conversions are also available.
\nonnx2tf -i xxxx.onnx -rtpo Erf -cotof -cotoa 1e-3\nI hope this will be of help to all of you.
\nBy the way, I am a hobby programmer not doing Kaggle.
Thanks for the comments.
\nYour tools has been a great help in the competition! 😊
after a day of work, my submission still failed.
\ni wonder if anyone can help:
https://www.kaggle.com/code/hengck23/failed-pytorch-to-onnx-to-tflite
", "votes": null, "replies": [ { "id": 2164226, "author_name": "hengck23", "author_url": "", "post_date": "03/01/2023 12:28:37", "content": "[error] runtime error due to dynamic shape
\ni think the issue is dynamic input shape.
\ni used up my submission, i will report it tmr
1 follow the notebook:
\nhttps://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission
(i do not know why the combined code failed although the code is the same)
\nwith this i can make pytorch transformer conversion on tflite to work.
", "votes": null, "replies": [] } ] }, { "id": 2166040, "author_name": "qiaoshiji", "author_url": "", "post_date": "03/02/2023 15:36:45", "content": "Have you made transformer submission successfully ？I have an error :\"TF Select ops: Range\".
\nAfter I added following setting, it looks good to run in the notebook, but still failed when submitting.
\nconverter.target_spec.supported_ops = [\n tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.\n tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n]
my suggestion is:
\ni have made the pytorch transformer sucessfully
", "votes": null, "replies": [ { "id": 2166720, "author_name": "qiaoshiji", "author_url": "", "post_date": "03/03/2023 01:10:41", "content": "Thanks, I removed ops tflite not supported and reran the code without converter.target_spec.supported_ops. It works.
", "votes": null, "replies": [ { "id": 2212711, "author_name": "abdulkadirguner", "author_url": "", "post_date": "04/07/2023 02:45:25", "content": "Hi OctOpus, how did you solve tf.Range issue? You just removed the function/method that is causing the problem?
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2167285, "author_name": "hengck23", "author_url": "", "post_date": "03/03/2023 11:45:11", "content": "[error] \"Can't reduce on dim with value of 0 if 'keepdims' is false\" in onnx runtime
\nfor computation od mean, std as feature input, you may end up with no points after you remove the nan values.
\n[solution] see: https://github.com/microsoft/onnxruntime/issues/7563
better solution
\ndef pre_process(xyz):\n idx_range_face = (0, 468)\n idx_range_hand_left = (468, 489)\n idx_range_pose = (489, 522)\n idx_range_hand_right = (522, 543)\n\n x_face = get_flat_features(xyz, idx_range_face)\n x_hand_left = get_flat_features(xyz, idx_range_hand_left)\n x_pose = get_flat_features(xyz, idx_range_pose)\n x_hand_right = get_flat_features(xyz, idx_range_hand_right)\n\n x_hand_left = x_hand_left[~torch.any(torch.isnan(x_hand_left), dim=1), :]\n x_hand_right = x_hand_right[~torch.any(torch.isnan(x_hand_right), dim=1), :]\n\n if len(x_hand_left)==0:\n x_hand_left_mean = torch.zeros(63)\n x_hand_left_std = torch.zeros(63)\n else:\n x_hand_left_mean = torch.mean(x_hand_left, 0)\n x_hand_left_std = torch.std(x_hand_left, 0)\n\n if len(x_hand_right)==0:\n x_hand_right_mean = torch.zeros(63)\n x_hand_right_std = torch.zeros(63)\n else:\n x_hand_right_mean = torch.mean(x_hand_right, 0)\n x_hand_right_std = torch.std(x_hand_right, 0)\n\n x_face_mean = torch.mean(x_face, 0)#, keepdims=True).squeeze(0)\n x_pose_mean = torch.mean(x_pose, 0)\n\n x_face_std = torch.std(x_face, 0)\n x_pose_std = torch.std(x_pose, 0)\n\n x_features = torch.cat(\n [\n x_face_mean,\n x_hand_left_mean,\n x_pose_mean,\n x_hand_right_mean,\n x_face_std,\n x_hand_left_std,\n x_pose_std,\n x_hand_right_std,\n ],\n dim=0,\n )\n\n x_features = torch.where(\n torch.isnan(x_features), torch.tensor(0.0, dtype=torch.float32), x_features\n )\n #x_features = x_features.unsqueeze(0)\n\n return x_features\nthen use:
\n torch.onnx.export(\n torch.jit.script(input_net),\n...\njit.script will explicitly unrolled all if else conditions
", "votes": null, "replies": [] } ] }, { "id": 2171172, "author_name": "hengck23", "author_url": "", "post_date": "03/06/2023 15:03:33", "content": "[error] onnx to tflite for nn.multiHeadAttention
\nSome ops are not supported by the native TFLite runtime, you can enable TF kernels fallback using TF Select. See instructions: https://www.tensorflow.org/lite/guide/ops_select \nTF Select ops: Cast, RealDiv\nDetails:\n tf.Cast(tensor<f64>) -> (tensor<i64>) : {Truncate = false, device = \"\"}\n tf.Cast(tensor<i64>) -> (tensor<f64>) : {Truncate = false, device = \"\"}\n tf.RealDiv(tensor<f64>, tensor<f64>) -> (tensor<f64>) : {device = \"\"}\nin computation of the head dim, we need to divide the embed dim by the num of head.
\n
onnx graph
\n
tflite graph
\nthis division should be constant, since we know what is the embed dim and num of head. For unknown reason, tflite convert failed to detect this and uses real division.
\n[solution]
\nedit the onnx graph and use fix const values when reshape
\nhttps://github.com/ZhangGe6/onnx-modifier
or rewrite the forward function nn.multiHeadAttention with constant values in reshape
e.g.\nhead_dim=128\nnum_head=8 \n\n def __init__(self,...):\n ...\n self.mha = nn.MultiheadAttention(...)\n\n def forward(self,x):\n ...\n k = F.linear(x, self.mha.in_proj_weight[1024:2048], self.mha.in_proj_bias[1024:2048])\n v = F.linear(x, self.mha.in_proj_weight[2048:], self.mha.in_proj_bias[2048:])\n\n q = q.reshape(-1, 8, 128).permute(1, 0, 2)\n k = k.reshape(-1, 8, 128).permute(1, 0, 2)\n v = v.reshape(-1, 8, 128).permute(1, 0, 2)\n dot = torch.matmul(q, k.transpose(-1, -2)) * (1/128**0.5) # H L L\n[error] onnx tp tflite conversion fail for using F.pad
\n[solution]
\nsee https://stackoverflow.com/questions/75016155/converting-onnx-model-to-tensorflow-fails
you need to specify fully for mode and value
\n reduced_xyz = F.pad(reduced_xyz,[0,0,0,0,0,1], mode='constant', value=0.)\n[error] tfRealDiv and tfCast are not naive tflite ops,e.g. if i want to do center crop, i need the division to compute the offset
\n[solution]
offset = (np.arange(1000)-max_length)//2 #assume 1000 is the longest video\noffset = np.clip(offset,0, 1000).tolist()\n\nclass InputNet(nn.Module):\n\n def __init__(self, ):\n super().__init__()\n self.max_length = max_length\n self.offset = nn.Parameter(torch.LongTensor(offset),requires_grad=False)\n\n def forward(self, xyz):\n L = len(xyz)\n if L>self.max_length:\n #xyz = xyz[:self.max_length] #first\n #xyz = xyz[-self.max_length:] #last\n\n i = self.offset[L]\n xyz = xyz[i:i+self.max_length] #center\ni have 2 sub-network. one for pre-processing (e.g. shape nomalisation), another for prediction (transformer).
\ni analyse the tflite runtime memory for the sub-network one by one.
\n(i was very baffled why my transformer network with disk size of just 2 mb would consume 200 mb ram at peak)
for unknown reason, it seems that the problem i have is the pre-processing network.
\nthis is unexpected for me.
i comment and uncoment line by line to find out the issue.
\nfor example:
from tflite benchmark tool\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 222.941\n\n#####\n\n not_nan_xyz = xyz[~torch.isnan(xyz)]\n if len(not_nan_xyz) != 0: \n xyz = xyz- torch.mean(xyz[~torch.isnan(xyz)],0, keepdim=True)\njust this few lines brew out my memory by 200MB???
\n(input is xyz = 500,543,3)
i check the graph of pre-processing network. it is quite messy
\nit seems that while pytorch-to-onnx-to-tflite conversion works good for the conv, linear, transformer, etc layer,
\nthe conversion is very bad for simple operation like \"custom tensor manipulation\" ????
next i would check if i write my preprossing in keras would do a better job.
", "votes": null, "replies": [ { "id": 2176766, "author_name": "roberthatch", "author_url": "", "post_date": "03/10/2023 22:15:04", "content": "Is there a problem with consuming 200MB ram? That just sounds like it's copying around a bit of memory while processing data chunks. I don't think it would make your submission fail?
\nWhat problem does it cause? Or you are just trying to optimize for faster operation?
", "votes": null, "replies": [ { "id": 2176801, "author_name": "hengck23", "author_url": "", "post_date": "03/10/2023 23:22:29", "content": "here is the comparsion.
\none has to be careful with pytorch to tflite conversion.
@roberthatch
\n\"Is there a problem with consuming 200MB ram?\"
your submission will fail with out-of-memory submission error.
\ni think although the restiction is simplified to \"40 mb on disk\",
\nbut 40mb on memory still apply at submission script (which totally make sense)
if i can solve the issue, it means that trasnformer can take in input length (num of frames) up to 512 under memory and time constraint.
\nThis improve accuracy (estimated LB 0.67 for one fold) and no need for complicated pre-processing.
memory size of xyz array of varying length:
\n''' num_frame, file, mem (mb)\n 0, train_landmark_files/16069/1558159851.parquet , 537, 6.87, 3.34\n 1, train_landmark_files/16069/2399841238.parquet , 299, 4.32, 1.86\n 2, train_landmark_files/61333/117070402.parquet , 271, 4.00, 1.68\n 3, train_landmark_files/27610/3762968283.parquet , 255, 3.75, 1.58\n 4, train_landmark_files/26734/2959425762.parquet , 246, 3.63, 1.53\n 5, train_landmark_files/18796/2947969142.parquet , 238, 3.57, 1.48\n 6, train_landmark_files/28656/2467637816.parquet , 232, 3.43, 1.44\n\n 150, train_landmark_files/32319/2814895419.parquet , 62, 0.94, 0.39\n 151, train_landmark_files/27610/1173378699.parquet , 62, 0.93, 0.39\n 152, train_landmark_files/49445/195491904.parquet , 61, 0.90, 0.38\n 153, train_landmark_files/49445/1303464708.parquet , 61, 0.92, 0.38\nsurprising, a dummy identity keras model consume quite a lot of memory
\n class InputNet(tf.keras.layers.Layer):\n def __init__(self, ):\n super(InputNet, self).__init__()\n def call(self, xyz):\n x = xyz\n return x\n #dummy identity\n#cmd_str += f'./benchmark_model \\\\'\n# ...\n#cmd_str += f'--input_layer_shape={\"512,543,3\"} \\\\' #543\n\n\"doing nothing model\"\n...\nINFO: The input model file size (MB): 0.00066\n...\nINFO: Memory footprint delta from the start of the tool (MB): init=2.43359 overall=8.67578\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 13.1797\nINFO: Memory status at the end of exeution: #catch one spelling error bug!!!\nINFO: - VmRSS : 13 MB\nINFO: + RssAnnon : 7 MB\nINFO: + RssFile + RssShmem : 6 MB\nif i replace input net with keras version,
\ni can run pytorch transformer at input length 256 at 20 msec per video!
tflite benchmark tool gives:
\nINFO: Memory footprint delta from the start of the tool (MB): init=3.59375 overall=30.9961\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 35.3594\nvideo length=512 seems to be borderline case with
\nINFO: Memory footprint delta from the start of the tool (MB): init=3.64062 overall=43.9102\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.332\na smarter way is to use e.g. 256 for normalization, 512 for prediction. with that it would be possible to predict up to 512 length within the 40 mb limit
", "votes": null, "replies": [ { "id": 2177148, "author_name": "mxbonn", "author_url": "", "post_date": "03/11/2023 08:44:17", "content": "I'm not convinced that the peak memory footprint needs to be within 40MB.
\nIf I benchmark our model with a sequence length of 512 it has a peak memory consumption of 61.9 MB and we used full sequence lengths so it will have reached that a few times.
thanks for feedback. the limit is probably not 40mb (eg need to subtract off other setup) or they are using other tools, etc.
\nfor my case submission failed with memory error with peak reach more than 100 mb.
\ni will try the 512 48 mb and let u know the results
", "votes": null, "replies": [ { "id": 2177364, "author_name": "hengck23", "author_url": "", "post_date": "03/11/2023 12:03:47", "content": "i confirm length 512 with 48 mb peak also passed the submission
", "votes": null, "replies": [ { "id": 2180243, "author_name": "sohier", "author_url": "", "post_date": "03/13/2023 17:27:38", "content": "We now just run os.path.getsizeof on the unzipped model checkpoint file. You can probably get much higher memory use if you're including the impact of loading the data.
Have you found the solution by pytorch?
", "votes": null, "replies": [ { "id": 2184604, "author_name": "hengck23", "author_url": "", "post_date": "03/16/2023 13:38:07", "content": "\"Have you found the solution by pytorch?\"
\neven if you use pytorch, you still need to convert to tflite.
\nhence just measure using os.path.getsizeof on the unzipped model checkpoint file as stated above
Hi! I guess I'm still missing something.
\nMy model size is around 4 MB, but it still gets a RAM error. So I rewrote the Input Net with the following way:
\n (nn.Module):\n ():\n ().__init__()\n self.max_length = \n\n ():\n xyz = xyz[:,:,:]\n xyz = xyz[:self.max_length]\n xyz = xyz - xyz[~torch.isnan(xyz)].mean(,keepdim=) \n xyz = xyz / xyz[~torch.isnan(xyz)].std(, keepdim=)\n\n LIP = [\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n , , , , , , , , , ,\n ]\n\n lip = xyz[:, LIP]\n lhand = xyz[:, :]\n rhand = xyz[:, :]\n xyz = torch.cat([ \n lip,\n lhand,\n rhand,\n ], )\n xyz[torch.isnan(xyz)] = \n xyz \nAs you can see I'm using small max length and I also moved the string xyz = xyz[:self.max_length] from the end to the beginning. However I do still get the same error after around 25 minutes of submission time. I would be grateful for any advice.
this is due to inefficiency using the pytorch-onnx-tflite conversion pipeline.
\nif you use the tflite benchmark tool, this tflite uses 200MB runtime memory (that is the cause of out of memory)
the soslution is to write your InputNet() in keras. Then use the keras-tflite conversion pipeline.
", "votes": null, "replies": [] } ] } ] }, { "id": 2179938, "author_name": "wuwenmin", "author_url": "", "post_date": "03/13/2023 13:46:45", "content": "After reading this discussion, I decide to use TensorFlow directly😪
", "votes": null, "replies": [] }, { "id": 2182986, "author_name": "hengck23", "author_url": "", "post_date": "03/15/2023 12:18:45", "content": "i have been translating code from pytorch to keras and vice versa ….
\nwe need a chatgpt for that !!!!!
\n(or language model code pytorch to tf translator)
https://qiita.com/PINTO/items/ed06e03eb5c007c2e102
\nhere is another pytorch to onnx pipeline
\n(i haven;t tried it though)
\n PyTorch -> ONNX -> OpenVINO -> TensorFlow / Tensorflow Lite
the advantages:
\nI am the author of openvino2tensorflow.
\nThe article you are referring to was written by me but is very old and openvino2tensorflow has not been maintained for some time. I now allocate almost all of my private development time to onnx2tf.
\nIf you want to modify the model, for example, it would be easier for many people to modify the PyTorch source code directly. However, if you become proficient in this pipeline, you can modify the model to avoid bugs in the inference framework and take advantage of the high-performance optimization capabilities of OpenVINO's model optimizer.
\nSince openvino2tensorflow was an early tool when I first started writing programs, the source code should be very messy and often behave in a way that is difficult to understand.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2161948": "i am interested in this competition.\ni am familiar with pytorch and onnx.\n\n[updated post]\n- there are new post and code that shows conversion from pytorch to tflite.\n- i followed them[1],[2],[3] and there are quite successful.\n- however, there are new functions (e.g. like GELU from transformer) that may cause issues.\n\nthis thread is for conversion problems and solution.\n\n[1] https://www.kaggle.com/code/myso1987/gislr-pytorch-tflite-baseline\n[2] https://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission\n\n[3] working tutorial\n- https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html\n- https://zhuanlan.zhihu.com/p/363317178\n- https://towardsdatascience.com/my-journey-in-converting-pytorch-to-tensorflow-lite-d244376beed\n\n\n[old post]\n\n~~i read that i can convert onnx to tflite from the web, but i never did it before.~~\n~~It is a feasible way for this compeition?~~\n\n~~it also seems that onnx-tf is no longer active (i.e. new layers for conversion may not be available)~~\n~~https://github.com/onnx/onnx-tensorflow~~\n\n~~(part of my day job is model deployment, and there are usually many problems if your model \"is not of the standard type and has customized layers\")~~\n\n~~or do you suggest that i should just use keras?~~\n~~(the problem is that there are less model zoo for keras)~~\n\n~~but i think even if i use keras, the conversion to tflite might still have problem?~~\n\n~~Hope to get feedback from tf users. Thanks!~~", "2162242": "Hi, you can easily do the conversion from pytorch to onnx and\nthen from onnx to tensorflow \nand finally tensorflow to tensorflow lite. \n\nPlease find the sample notebooks from other kagglers: \n@jarvisai7 : https://www.kaggle.com/code/jarvisai7/tensorflow-pytorch-to-tflite\n@mhetrerajat : https://www.kaggle.com/code/mhetrerajat/pytorch-tflite\n\nhope this helps you..", "2163269": "this should discussion should be pinned :D", "2163305": "It would be better to post here https://www.kaggle.com/competitions/asl-signs/discussion/390008", "2163887": "[error] tf.erf op error\n\nif you are using pytorch mutihead attention, you may encounter this error.\nbecuase onnx don't have gelu, it implement gelu with erf\nsee : https://user-images.githubusercontent.com/103593150/220924683-5da1ca1b-0864-43fd-9ec3-804d8be8b2a1.png\nhttps://github.com/onnx/onnx/issues/4933\n\ntf has tf.erf \ntflite don't have erf.\nbut tflite does have gelu. \n\n---\n\ni haven't figure out how to solve this. if you know, please suggest. thanks!\ntemproarily solution is to use relu", "2164190": "[error] tflite output numerical differences\n~~i note that ~~\n~~- pytorch to onnx usually gives the same results (within numerical error)~~\n~~- but onnx to tf to tflite can gives different results.~~\n\n~~the argmax of the results are likely to be the same.~~\n~~but if you print out the raw logit values, they are greater than numerical error.~~\n\n~~i wonder what is the cause~~\n\ni note that the cause is due:\n\n```\nconverter.optimizations = [tf.lite.Optimize.DEFAULT]\n```", "2164214": "after a day of work, my submission still failed.\ni wonder if anyone can help:\n\nhttps://www.kaggle.com/code/hengck23/failed-pytorch-to-onnx-to-tflite", "2164226": "[error] runtime error due to dynamic shape\ni think the issue is dynamic input shape.\ni used up my submission, i will report it tmr\n\n1 follow the notebook:\nhttps://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission\n\n- split your model into two. \n- one for just dynamic input size \n- another for prediction\n\n(i do not know why the combined code failed although the code is the same)\n\nwith this i can make pytorch transformer conversion on tflite to work.", "2166040": "Have you made transformer submission successfully ？I have an error :\"TF Select ops: Range\". \nAfter I added following setting, it looks good to run in the notebook, but still failed when submitting.\n`converter.target_spec.supported_ops = [\n tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.\n tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n]`", "2166143": "my suggestion is:\n- try not to use converter.target_spec.supported_ops\n- run a few thousands of parquet file at development in notebook (not just one or two)\n\ni have made the pytorch transformer sucessfully", "2166720": "Thanks, I removed ops tflite not supported and reran the code without converter.target_spec.supported_ops. It works.", "2166726": "i find that if onnx works, onnx-to-tflite should work.\n\nhere are some magic tools to do network surgery on onnx:\nhttps://github.com/PINTO0309/simple-onnx-processing-tools\nhttps://github.com/PINTO0309/onnx2tf\n\nmany other onnx, tflite tools:\nhttps://github.com/PINTO0309?tab=repositories", "2167285": "[error] \"Can't reduce on dim with value of 0 if 'keepdims' is false\" in onnx runtime\n\nfor computation od mean, std as feature input, you may end up with no points after you remove the nan values.\n[solution] see: https://github.com/microsoft/onnxruntime/issues/7563", "2167447": "better solution\n\n```\n\ndef pre_process(xyz):\n idx_range_face = (0, 468)\n idx_range_hand_left = (468, 489)\n idx_range_pose = (489, 522)\n idx_range_hand_right = (522, 543)\n\n x_face = get_flat_features(xyz, idx_range_face)\n x_hand_left = get_flat_features(xyz, idx_range_hand_left)\n x_pose = get_flat_features(xyz, idx_range_pose)\n x_hand_right = get_flat_features(xyz, idx_range_hand_right)\n\n x_hand_left = x_hand_left[~torch.any(torch.isnan(x_hand_left), dim=1), :]\n x_hand_right = x_hand_right[~torch.any(torch.isnan(x_hand_right), dim=1), :]\n\n if len(x_hand_left)==0:\n x_hand_left_mean = torch.zeros(63)\n x_hand_left_std = torch.zeros(63)\n else:\n x_hand_left_mean = torch.mean(x_hand_left, 0)\n x_hand_left_std = torch.std(x_hand_left, 0)\n\n if len(x_hand_right)==0:\n x_hand_right_mean = torch.zeros(63)\n x_hand_right_std = torch.zeros(63)\n else:\n x_hand_right_mean = torch.mean(x_hand_right, 0)\n x_hand_right_std = torch.std(x_hand_right, 0)\n\n x_face_mean = torch.mean(x_face, 0)#, keepdims=True).squeeze(0)\n x_pose_mean = torch.mean(x_pose, 0)\n\n x_face_std = torch.std(x_face, 0)\n x_pose_std = torch.std(x_pose, 0)\n\n x_features = torch.cat(\n [\n x_face_mean,\n x_hand_left_mean,\n x_pose_mean,\n x_hand_right_mean,\n x_face_std,\n x_hand_left_std,\n x_pose_std,\n x_hand_right_std,\n ],\n dim=0,\n )\n\n x_features = torch.where(\n torch.isnan(x_features), torch.tensor(0.0, dtype=torch.float32), x_features\n )\n #x_features = x_features.unsqueeze(0)\n\n return x_features\n```\n\nthen use:\n\n```\n torch.onnx.export(\n torch.jit.script(input_net),\n...\n\n```\njit.script will explicitly unrolled all if else conditions", "2171172": "[error] onnx to tflite for nn.multiHeadAttention\n```\nSome ops are not supported by the native TFLite runtime, you can enable TF kernels fallback using TF Select. See instructions: https://www.tensorflow.org/lite/guide/ops_select \nTF Select ops: Cast, RealDiv\nDetails:\n\ttf.Cast(tensor\n\nonnx graph\n\n\n\ntflite graph\n\n\nthis division should be constant, since we know what is the embed dim and num of head. For unknown reason, tflite convert failed to detect this and uses real division.\n\n[solution]\n- edit the onnx graph and use fix const values when reshape\nhttps://github.com/ZhangGe6/onnx-modifier\n\n- or rewrite the forward function nn.multiHeadAttention with constant values in reshape\n\n```\ne.g.\nhead_dim=128\nnum_head=8 \n\n def __init__(self,...):\n ...\n self.mha = nn.MultiheadAttention(...)\n\n def forward(self,x):\n ...\n\t\tk = F.linear(x, self.mha.in_proj_weight[1024:2048], self.mha.in_proj_bias[1024:2048])\n\t\tv = F.linear(x, self.mha.in_proj_weight[2048:], self.mha.in_proj_bias[2048:])\n\n\t\tq = q.reshape(-1, 8, 128).permute(1, 0, 2)\n\t\tk = k.reshape(-1, 8, 128).permute(1, 0, 2)\n\t\tv = v.reshape(-1, 8, 128).permute(1, 0, 2)\n\t\tdot = torch.matmul(q, k.transpose(-1, -2)) * (1/128**0.5) # H L L\n\n```",
"2172194": "I am the author of onnx2tf.\n\nonnx2tf can convert almost any model except RNNs. There is also an option to make all Flex operations as harmless as possible. GELU conversions are also available.\n\n```\nonnx2tf -i xxxx.onnx -rtpo Erf -cotof -cotoa 1e-3\n```\n\nI hope this will be of help to all of you.\nBy the way, I am a hobby programmer not doing Kaggle.",
"2172798": "katsuyahyodo \n\nThanks for the comments.\nYour tools has been a great help in the competition! 😊",
"2173826": "[error] onnx tp tflite conversion fail for using F.pad\n[solution]\nsee https://stackoverflow.com/questions/75016155/converting-onnx-model-to-tensorflow-fails\n\nyou need to specify fully for mode and value\n```\n\treduced_xyz = F.pad(reduced_xyz,[0,0,0,0,0,1], mode='constant', value=0.)\n```",
"2174721": "[error] tfRealDiv and tfCast are not naive tflite ops,e.g. if i want to do center crop, i need the division to compute the offset\n[solution]\n\n```\n\noffset = (np.arange(1000)-max_length)//2 #assume 1000 is the longest video\noffset = np.clip(offset,0, 1000).tolist()\n\nclass InputNet(nn.Module):\n\n def __init__(self, ):\n super().__init__()\n self.max_length = max_length\n self.offset = nn.Parameter(torch.LongTensor(offset),requires_grad=False)\n\n def forward(self, xyz):\n L = len(xyz)\n if L>self.max_length:\n #xyz = xyz[:self.max_length] #first\n #xyz = xyz[-self.max_length:] #last\n\n i = self.offset[L]\n xyz = xyz[i:i+self.max_length] #center\n\n```",
"2176549": "i have 2 sub-network. one for pre-processing (e.g. shape nomalisation), another for prediction (transformer). \ni analyse the tflite runtime memory for the sub-network one by one. \n(i was very baffled why my transformer network with disk size of just 2 mb would consume 200 mb ram at peak)\n\n---\n\nfor unknown reason, it seems that the problem i have is the pre-processing network.\nthis is unexpected for me.\n\n\ni comment and uncoment line by line to find out the issue.\nfor example:\n\n```\nfrom tflite benchmark tool\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 222.941\n\n#####\n\n not_nan_xyz = xyz[~torch.isnan(xyz)]\n if len(not_nan_xyz) != 0: \n xyz = xyz- torch.mean(xyz[~torch.isnan(xyz)],0, keepdim=True)\n```\n\njust this few lines brew out my memory by 200MB???\n(input is xyz = 500,543,3)\n\ni check the graph of pre-processing network. it is quite messy\n\nit seems that while pytorch-to-onnx-to-tflite conversion works good for the conv, linear, transformer, etc layer,\nthe conversion is very bad for simple operation like \"custom tensor manipulation\" ????\n\n---\n\nnext i would check if i write my preprossing in keras would do a better job.",
"2176766": "Is there a problem with consuming 200MB ram? That just sounds like it's copying around a bit of memory while processing data chunks. I don't think it would make your submission fail?\n\nWhat problem does it cause? Or you are just trying to optimize for faster operation?",
"2176801": "here is the comparsion.\none has to be careful with pytorch to tflite conversion.\n\n",
"2176802": "roberthatch \n\"Is there a problem with consuming 200MB ram?\"\n\nyour submission will fail with out-of-memory submission error.\ni think although the restiction is simplified to \"40 mb on disk\",\nbut 40mb on memory still apply at submission script (which totally make sense)",
"2176808": "if i can solve the issue, it means that trasnformer can take in input length (num of frames) up to 512 under memory and time constraint.\nThis improve accuracy (estimated LB 0.67 for one fold) and no need for complicated pre-processing.",
"2176909": "memory size of xyz array of varying length:\n\n```\n''' num_frame, file, mem (mb)\n 0, train_landmark_files/16069/1558159851.parquet , 537, 6.87, 3.34\n 1, train_landmark_files/16069/2399841238.parquet , 299, 4.32, 1.86\n 2, train_landmark_files/61333/117070402.parquet , 271, 4.00, 1.68\n 3, train_landmark_files/27610/3762968283.parquet , 255, 3.75, 1.58\n 4, train_landmark_files/26734/2959425762.parquet , 246, 3.63, 1.53\n 5, train_landmark_files/18796/2947969142.parquet , 238, 3.57, 1.48\n 6, train_landmark_files/28656/2467637816.parquet , 232, 3.43, 1.44\n \n 150, train_landmark_files/32319/2814895419.parquet , 62, 0.94, 0.39\n 151, train_landmark_files/27610/1173378699.parquet , 62, 0.93, 0.39\n 152, train_landmark_files/49445/195491904.parquet , 61, 0.90, 0.38\n 153, train_landmark_files/49445/1303464708.parquet , 61, 0.92, 0.38\n \n```\n\nsurprising, a dummy identity keras model consume quite a lot of memory\n```\n\tclass InputNet(tf.keras.layers.Layer):\n\t\tdef __init__(self, ):\n\t\t\tsuper(InputNet, self).__init__()\n\t\tdef call(self, xyz):\n\t\t\tx = xyz\n\t\t\treturn x\n\t\t#dummy identity\n```\n```\n#cmd_str += f'./benchmark_model \\\\'\n# ...\n#cmd_str += f'--input_layer_shape={\"512,543,3\"} \\\\' #543\n\n\"doing nothing model\"\n...\nINFO: The input model file size (MB): 0.00066\n...\nINFO: Memory footprint delta from the start of the tool (MB): init=2.43359 overall=8.67578\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 13.1797\nINFO: Memory status at the end of exeution: #catch one spelling error bug!!!\nINFO: - VmRSS : 13 MB\nINFO: + RssAnnon : 7 MB\nINFO: + RssFile + RssShmem : 6 MB\n\n```",
"2177113": "if i replace input net with keras version,\ni can run pytorch transformer at input length 256 at 20 msec per video!\n\ntflite benchmark tool gives:\n\n```\nINFO: Memory footprint delta from the start of the tool (MB): init=3.59375 overall=30.9961\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 35.3594\n```\n\nvideo length=512 seems to be borderline case with \n```\nINFO: Memory footprint delta from the start of the tool (MB): init=3.64062 overall=43.9102\nINFO: Overall peak memory footprint (MB) via periodic monitoring: 48.332\n```\n\na smarter way is to use e.g. 256 for normalization, 512 for prediction. with that it would be possible to predict up to 512 length within the 40 mb limit",
"2177148": "I'm not convinced that the peak memory footprint needs to be within 40MB. \nIf I benchmark our model with a sequence length of 512 it has a peak memory consumption of 61.9 MB and we used full sequence lengths so it will have reached that a few times.",
"2177159": "thanks for feedback. the limit is probably not 40mb (eg need to subtract off other setup) or they are using other tools, etc. \n\nfor my case submission failed with memory error with peak reach more than 100 mb.\n\ni will try the 512 48 mb and let u know the results",
"2177364": "i confirm length 512 with 48 mb peak also passed the submission",
"2178698": "Thank you!",
"2179938": "After reading this discussion, I decide to use TensorFlow directly😪",
"2180243": "We now just run `os.path.getsizeof` on the unzipped model checkpoint file. You can probably get much higher memory use if you're including the impact of loading the data.",
"2182986": "i have been translating code from pytorch to keras and vice versa ....\nwe need a chatgpt for that !!!!!\n(or language model code pytorch to tf translator)",
"2184583": "Have you found the solution by pytorch?",
"2184604": "\"Have you found the solution by pytorch?\"\n\neven if you use pytorch, you still need to convert to tflite.\nhence just measure using os.path.getsizeof on the unzipped model checkpoint file as stated above",
"2187526": "Hi! I guess I'm still missing something.\n\nMy model size is around 4 MB, but it still gets a RAM error. So I rewrote the Input Net with the following way:\n\n```python\nclass InputNet(nn.Module):\n def __init__(self, ):\n super().__init__()\n self.max_length = 60 \n \n def forward(self, xyz):\n xyz = xyz[:,:,:2]\n xyz = xyz[:self.max_length]\n xyz = xyz - xyz[~torch.isnan(xyz)].mean(0,keepdim=True) #noramlisation to common maen\n xyz = xyz / xyz[~torch.isnan(xyz)].std(0, keepdim=True)\n\n LIP = [\n 61, 185, 40, 39, 37, 0, 267, 269, 270, 409,\n 291, 146, 91, 181, 84, 17, 314, 405, 321, 375,\n 78, 191, 80, 81, 82, 13, 312, 311, 310, 415,\n 95, 88, 178, 87, 14, 317, 402, 318, 324, 308,\n ]\n \n lip = xyz[:, LIP]\n lhand = xyz[:, 468:489]\n rhand = xyz[:, 522:543]\n xyz = torch.cat([ # (none, 82, 3)\n lip,\n lhand,\n rhand,\n ], 1)\n xyz[torch.isnan(xyz)] = 0\n return xyz \n```\n\nAs you can see I'm using small max length and I also moved the string `xyz = xyz[:self.max_length]` from the end to the beginning. However I do still get the same error after around 25 minutes of submission time. I would be grateful for any advice.",
"2187568": "this is due to inefficiency using the pytorch-onnx-tflite conversion pipeline.\nif you use the tflite benchmark tool, this tflite uses 200MB runtime memory (that is the cause of out of memory)\n\nthe soslution is to write your InputNet() in keras. Then use the keras-tflite conversion pipeline.",
"2190079": "https://qiita.com/PINTO/items/ed06e03eb5c007c2e102\n\nhere is another pytorch to onnx pipeline\n(i haven;t tried it though)\n PyTorch -> ONNX -> OpenVINO -> TensorFlow / Tensorflow Lite\n\nthe advantages:\n\n- The Model Optimizer of OpenVINO optimizes the model during the conversion process by itself\n- OpenVINO itself, as a dedicated inference framework, is specialized in the role of inference and inter-framework conversion, so it is sophisticated as a common format.\n- All operation information and connections between operations are output in a simple, human-readable XML file so that the structure of the trained model can be easily rewritten later using an editor.",
"2192485": "I am the author of openvino2tensorflow.\n\nThe article you are referring to was written by me but is very old and openvino2tensorflow has not been maintained for some time. I now allocate almost all of my private development time to onnx2tf.\n\nIf you want to modify the model, for example, it would be easier for many people to modify the PyTorch source code directly. However, if you become proficient in this pipeline, you can modify the model to avoid bugs in the inference framework and take advantage of the high-performance optimization capabilities of OpenVINO's model optimizer.\n\nSince openvino2tensorflow was an early tool when I first started writing programs, the source code should be very messy and often behave in a way that is difficult to understand.",
"2212711": "Hi OctOpus, how did you solve tf.Range issue? You just removed the function/method that is causing the problem?"
},
"source": "meta"
}