Inferencing on 1.5 million images is really slow. So i made some experiments with resnet34 to accelerate the inferencing. Here are results on the validation set (20% of the train set):
\nGPU: 1x 2080ti\nencoder: resnet34\ndecoder: lstm with attention\nbatch size: 256\nworker: 8\nimage size: 224x224\n\n20m14s: inferencing like Y.Nakamas public notebook\n16m50s: break if batch EOS\n12m39s: break if batch EOS + sort by InChi length\n12m42s: break if batch EOS + sort by InChi length + torch.jit.script(encoder)\nSo with sorting by InChi length, you can make your inferencing 35% faster if you have already a previous submission. If not, you can use batch EOS break which is still much faster than the naive inferencing.
\nTo use the batch EOS break, change the Y.Nakamas code this way:
\nfor t in range(decode_lengths):\n ...\n if np.argmax(preds.detach().cpu().numpy()) == tokenizer.stoi[\"<eos>\"]:\n break\nto
\nend_condition = torch.zeros(batch_size, dtype=torch.long).to(encoder_out.device)\nfor t in range(decode_lengths):\n ...\n end_condition |= (torch.argmax(preds, -1) == tokenizer.stoi[\"<eos>\"])\n if end_condition.sum() == batch_size:\n break\nUsing torch.jit on the resnet34 encoder seems to have no effect.
", "messages": [ { "id": 1267215, "postDate": "2021-04-08T11:49:42.297Z", "content": "Inferencing on 1.5 million images is really slow. So i made some experiments with resnet34 to accelerate the inferencing. Here are results on the validation set (20% of the train set):
\nGPU: 1x 2080ti\nencoder: resnet34\ndecoder: lstm with attention\nbatch size: 256\nworker: 8\nimage size: 224x224\n\n20m14s: inferencing like Y.Nakamas public notebook\n16m50s: break if batch EOS\n12m39s: break if batch EOS + sort by InChi length\n12m42s: break if batch EOS + sort by InChi length + torch.jit.script(encoder)\nSo with sorting by InChi length, you can make your inferencing 35% faster if you have already a previous submission. If not, you can use batch EOS break which is still much faster than the naive inferencing.
\nTo use the batch EOS break, change the Y.Nakamas code this way:
\nfor t in range(decode_lengths):\n ...\n if np.argmax(preds.detach().cpu().numpy()) == tokenizer.stoi[\"<eos>\"]:\n break\nto
\nend_condition = torch.zeros(batch_size, dtype=torch.long).to(encoder_out.device)\nfor t in range(decode_lengths):\n ...\n end_condition |= (torch.argmax(preds, -1) == tokenizer.stoi[\"<eos>\"])\n if end_condition.sum() == batch_size:\n break\nUsing torch.jit on the resnet34 encoder seems to have no effect.
", "rawMarkdown": "Inferencing on 1.5 million images is really slow. So i made some experiments with resnet34 to accelerate the inferencing. Here are results on the validation set (20% of the train set):\n```\nGPU: 1x 2080ti\nencoder: resnet34\ndecoder: lstm with attention\nbatch size: 256\nworker: 8\nimage size: 224x224\n\n20m14s: inferencing like Y.Nakamas public notebook\n16m50s: break if batch EOS\n12m39s: break if batch EOS + sort by InChi length\n12m42s: break if batch EOS + sort by InChi length + torch.jit.script(encoder)\n```\n\nSo with sorting by InChi length, you can make your inferencing 35% faster if you have already a previous submission. If not, you can use batch EOS break which is still much faster than the naive inferencing.\n\nTo use the batch EOS break, change the Y.Nakamas code this way:\n```\nfor t in range(decode_lengths):\n ...\n if np.argmax(preds.detach().cpu().numpy()) == tokenizer.stoi[\"if you are using vision transformer:
\nhttps://pytorch.org/tutorials/beginner/vt_tutorial.html
\noriginal model: 169.95ms\nscripted model: 134.65ms\nscripted & quantized model: 125.27ms\nscripted & quantized & optimized model: 120.05ms\nlite model: 113.12ms\nthanks for sharing，I don’t understand why sort by length can speed up inference? Could you give me some explanation about that, thanks.
", "rawMarkdown": "thanks for sharing，I don’t understand why sort by length can speed up inference? Could you give me some explanation about that, thanks.", "votes": 1, "replies": [ { "id": 1267503, "postDate": "2021-04-08T15:04:18.183Z", "content": "If you sort the data, the elements in your batch will have similar lengths and reach their EOS in similar time steps.
", "rawMarkdown": "If you sort the data, the elements in your batch will have similar lengths and reach their EOS in similar time steps.", "votes": 5 }, { "id": 1267517, "postDate": "2021-04-08T15:11:59.070Z", "content": "Ok, I’ll try, thanks
", "rawMarkdown": "Ok, I’ll try, thanks" } ] }, { "id": 1268260, "postDate": "2021-04-09T08:45:01.187Z", "content": "i recall someone use this trick before:
\nfor those interested in jit, refer to thsi: https://pytorch.org/tutorials/beginner/deploy_seq2seq_hybrid_frontend_tutorial.html
", "rawMarkdown": "for those interested in jit, refer to thsi: https://pytorch.org/tutorials/beginner/deploy_seq2seq_hybrid_frontend_tutorial.html", "votes": 2, "replies": [ { "id": 1268610, "postDate": "2021-04-09T15:27:39.730Z", "content": "Not sure if this is the same…. but for those interested in JIT within the context of TF and XLA acceleration please refer to this:
\n", "rawMarkdown": "Not sure if this is the same.... but for those interested in JIT within the context of TF and XLA acceleration please refer to this:\n- [**JIT TF Documentation**](https://www.tensorflow.org/api_docs/python/tf/config/optimizer/set_jit?version=nightly)\n- [**XLA Optimization Tutorial Which Utilizes JIT**](https://www.tensorflow.org/xla/tutorials/autoclustering_xla)" } ] }, { "id": 1267978, "postDate": "2021-04-09T03:06:52.720Z", "content": "How about using TPU? It takes about 1 Hr in my case. Too long?
", "rawMarkdown": "How about using TPU? It takes about 1 Hr in my case. Too long?", "replies": [ { "id": 1268159, "postDate": "2021-04-09T07:12:00.900Z", "content": "@wuliaokaola are you using TPUs for inference?
\n(I thought Kaggle allows using TPUs for training only, not inference.)
@sirishks This is not code competition. You can use TPU as you like.
\nBTW, I think code competition does not support TPU just because it need internet connection.
I’m using TPU.
\nIt takes 8-10 minutes for inference on 1.5 million images with an EfficientNetB3 and 384x384 images.
\nThis is the fastest I’ve seen. I’ll be publishing a notebook in a week or so showing my process for both training and testing using TPU in detail.
", "rawMarkdown": "I’m using TPU. \n\nIt takes 8-10 minutes for inference on 1.5 million images with an EfficientNetB3 and 384x384 images.\n\nThis is the fastest I’ve seen. I’ll be publishing a notebook in a week or so showing my process for both training and testing using TPU in detail." }, { "id": 1268488, "postDate": "2021-04-09T12:59:46.090Z", "content": "You mean TPU + TF with well sharded TFRecords ?
\nBecause you still have dataloader bottleneck with Pytorch/XLA which uses only local VM
", "rawMarkdown": "You mean TPU + TF with well sharded TFRecords ? \n\nBecause you still have dataloader bottleneck with Pytorch/XLA which uses only local VM\n" }, { "id": 1268608, "postDate": "2021-04-09T15:25:59.940Z", "content": "\n\nTPU+TF with sharded TFRecords
\n
Yep! I'm not sorting the data or doing any tricks like that… so I will hopefully be able to get it even faster.
", "rawMarkdown": "> TPU+TF with sharded TFRecords\n\nYep! I'm not sorting the data or doing any tricks like that... so I will hopefully be able to get it even faster.", "votes": 1 }, { "id": 1270660, "postDate": "2021-04-11T21:08:17.993Z", "content": "I tried to use distributed TPU ,however I got shared memory expired! error. can not find any distributed TPU inference examples.
", "rawMarkdown": "I tried to use distributed TPU ,however I got shared memory expired! error. can not find any distributed TPU inference examples." }, { "id": 1279058, "postDate": "2021-04-20T14:53:37.117Z", "content": "Here is my inference notebook - https://www.kaggle.com/dschettler8845/bms-cnn-attn-lstm-tpu-tutorial-infer
", "rawMarkdown": "Here is my inference notebook - https://www.kaggle.com/dschettler8845/bms-cnn-attn-lstm-tpu-tutorial-infer", "votes": 1 }, { "id": 1279181, "postDate": "2021-04-20T17:07:52.777Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1274092, "postDate": "2021-04-15T01:15:55.850Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1269888, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-11T03:07:10.657000", "content": "if you are using vision transformer:
\nhttps://pytorch.org/tutorials/beginner/vt_tutorial.html
\noriginal model: 169.95ms\nscripted model: 134.65ms\nscripted & quantized model: 125.27ms\nscripted & quantized & optimized model: 120.05ms\nlite model: 113.12ms\nthanks for sharing，I don’t understand why sort by length can speed up inference? Could you give me some explanation about that, thanks.
", "votes": 1, "replies": [ { "id": 1267503, "author_name": "tugstugi", "author_url": "", "post_date": "2021-04-08T15:04:18.183000", "content": "If you sort the data, the elements in your batch will have similar lengths and reach their EOS in similar time steps.
", "votes": 5, "replies": [] }, { "id": 1267517, "author_name": "Matrix", "author_url": "", "post_date": "2021-04-08T15:11:59.070000", "content": "Ok, I’ll try, thanks
", "votes": 0, "replies": [] } ] }, { "id": 1268260, "author_name": "hengck23", "author_url": "", "post_date": "2021-04-09T08:45:01.187000", "content": "i recall someone use this trick before:
\nfor those interested in jit, refer to thsi: https://pytorch.org/tutorials/beginner/deploy_seq2seq_hybrid_frontend_tutorial.html
", "votes": 2, "replies": [ { "id": 1268610, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-04-09T15:27:39.730000", "content": "Not sure if this is the same…. but for those interested in JIT within the context of TF and XLA acceleration please refer to this:
\n", "votes": 0, "replies": [] } ] }, { "id": 1267978, "author_name": "Johnny Lee", "author_url": "", "post_date": "2021-04-09T03:06:52.720000", "content": "How about using TPU? It takes about 1 Hr in my case. Too long?
", "votes": 0, "replies": [ { "id": 1268159, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2021-04-09T07:12:00.900000", "content": "@wuliaokaola are you using TPUs for inference?
\n(I thought Kaggle allows using TPUs for training only, not inference.)
@sirishks This is not code competition. You can use TPU as you like.
\nBTW, I think code competition does not support TPU just because it need internet connection.
I’m using TPU.
\nIt takes 8-10 minutes for inference on 1.5 million images with an EfficientNetB3 and 384x384 images.
\nThis is the fastest I’ve seen. I’ll be publishing a notebook in a week or so showing my process for both training and testing using TPU in detail.
", "votes": 0, "replies": [] }, { "id": 1268488, "author_name": "Serigne ", "author_url": "", "post_date": "2021-04-09T12:59:46.090000", "content": "You mean TPU + TF with well sharded TFRecords ?
\nBecause you still have dataloader bottleneck with Pytorch/XLA which uses only local VM
", "votes": 0, "replies": [] }, { "id": 1268608, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-04-09T15:25:59.940000", "content": "\n\nTPU+TF with sharded TFRecords
\n
Yep! I'm not sorting the data or doing any tricks like that… so I will hopefully be able to get it even faster.
", "votes": 1, "replies": [] }, { "id": 1270660, "author_name": "dragon zhang", "author_url": "", "post_date": "2021-04-11T21:08:17.993000", "content": "I tried to use distributed TPU ,however I got shared memory expired! error. can not find any distributed TPU inference examples.
", "votes": 0, "replies": [] }, { "id": 1279058, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-04-20T14:53:37.117000", "content": "Here is my inference notebook - https://www.kaggle.com/dschettler8845/bms-cnn-attn-lstm-tpu-tutorial-infer
", "votes": 1, "replies": [] }, { "id": 1279181, "author_name": "", "author_url": "", "post_date": "2021-04-20T17:07:52.777000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1274092, "author_name": "", "author_url": "", "post_date": "2021-04-15T01:15:55.850000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1267215": "Inferencing on 1.5 million images is really slow. So i made some experiments with resnet34 to accelerate the inferencing. Here are results on the validation set (20% of the train set):\n```\nGPU: 1x 2080ti\nencoder: resnet34\ndecoder: lstm with attention\nbatch size: 256\nworker: 8\nimage size: 224x224\n\n20m14s: inferencing like Y.Nakamas public notebook\n16m50s: break if batch EOS\n12m39s: break if batch EOS + sort by InChi length\n12m42s: break if batch EOS + sort by InChi length + torch.jit.script(encoder)\n```\n\nSo with sorting by InChi length, you can make your inferencing 35% faster if you have already a previous submission. If not, you can use batch EOS break which is still much faster than the naive inferencing.\n\nTo use the batch EOS break, change the Y.Nakamas code this way:\n```\nfor t in range(decode_lengths):\n ...\n if np.argmax(preds.detach().cpu().numpy()) == tokenizer.stoi[\"