\n
my best CV score is about 9.28 +-0.02 , LB is about 9.18
\nso I never reach CV score 9.33+
\nI read many efficientnet+unet notebook ,
\nmaybe you can take a look at my work
\nand tell me what I made mistake on architecture or params tuning ? to get a higher CV score?
\n\n
\n
Preproccess tfrecord:https://www.kaggle.com/drzhuzhe/hubmap-tf-with-tpu-efficientunet-256-tfrecord
\ntrainning: https://www.kaggle.com/drzhuzhe/hubmap-efficientnet-and-linknet-train thanks to wrrosa and vgarshin's work
\ninfer: https://www.kaggle.com/drzhuzhe/efficientnet-linknet-or-unet
\n
\n
\n
\n
\n
\n
\n
\n
\n
my best CV score is about 9.28 +-0.02 , LB is about 9.18
\nso I never reach CV score 9.33+
\nI read many efficientnet+unet notebook ,
\nmaybe you can take a look at my work
\nand tell me what I made mistake on architecture or params tuning ? to get a higher CV score?
\n\n
\n
Preproccess tfrecord:https://www.kaggle.com/drzhuzhe/hubmap-tf-with-tpu-efficientunet-256-tfrecord
\ntrainning: https://www.kaggle.com/drzhuzhe/hubmap-efficientnet-and-linknet-train thanks to wrrosa and vgarshin's work
\ninfer: https://www.kaggle.com/drzhuzhe/efficientnet-linknet-or-unet
\n
\n
\n
\n
\n
\n
\n
\n
Hey @drzhuzhe May'be try more and different things? Try out different losses and combinations of those. Try out different backbones. Try out may'be other frameworks than just Unet.
\nGood luck.
", "rawMarkdown": "Hey @drzhuzhe May'be try more and different things? Try out different losses and combinations of those. Try out different backbones. Try out may'be other frameworks than just Unet.\n\nGood luck.", "votes": null }, { "id": "1270794", "postDate": "04/12/2021 03:05:09", "content": "About batchsize: From my opinion, 1024 batchsize is too large. Usually 32 or 64 is enough.
\nAbout backbone: efficientb3 can also get 0.92x results on LB.
\nAbout epochs: Try to use pretrain Encode weights from imagenet. Usually it will converge in 30 epochs. In my case, I use AdamW optimizer. And set initial lr to 1e-4.
Big batch size requires high learning rate.
\nThere is an anology - imagine a pool table with 3 billiard balls randomly moving, and we will draw a center of mass of these balls. What trajectory these center of mass will have? With 3 balls that will be a trajectory with big oscillations. But what happends if there will be 1000 balls? Center of mass will not moving, because random directions of balls will mutually compensate each other.
\nSame situation with big batch size - gradients of different samples will mutually compensate each other, so final gradient will be smaller, so higher learning rate is required
", "rawMarkdown": "Big batch size requires high learning rate. \n\nThere is an anology - imagine a pool table with 3 billiard balls randomly moving, and we will draw a center of mass of these balls. What trajectory these center of mass will have? With 3 balls that will be a trajectory with big oscillations. But what happends if there will be 1000 balls? Center of mass will not moving, because random directions of balls will mutually compensate each other. \n\nSame situation with big batch size - gradients of different samples will mutually compensate each other, so final gradient will be smaller, so higher learning rate is required", "votes": null }, { "id": "1272188", "postDate": "04/13/2021 09:48:47", "content": "Really awesome explain, dude
\nBut I still have two question make me very confuse
\nI set my optimizer a learning rate reduce
\nwith factor 0.1 and minimal: 0.01 * my initial learning rate
\ndoes my minimal learning rate need scaler up with initial learning rate ?
Does 1024 such a large batch size really hurt metrics score?
\nsome paper said small batch make more Regularization to optimizer, so larger batch may
\nin disadvantage
but other paper said large batch make gradient change more slowly, so what I just need is only training more epoch till convergence ?
Due to my CV score is too close, I cannot figure out which change in my params make sense
", "rawMarkdown": "Really awesome explain, dude\n\nBut I still have two question make me very confuse \n\n1. I set my optimizer **a learning rate reduce**\n\n with factor 0.1 and minimal: 0.01 * my initial learning rate\n\n does my **minimal learning rate** need scaler up with initial learning rate ?\n\n\n2. Does 1024 such a large batch size really hurt metrics score? \n\n some paper said **small batch make more Regularization to optimizer**, so larger batch may \nin disadvantage\n\n but other paper said large batch make gradient change more slowly, so what I just need is only training more epoch till convergence ?\n\n\nDue to my CV score is too close, I cannot figure out which change in my params make sense", "votes": null }, { "id": "1272245", "postDate": "04/13/2021 10:44:27", "content": "Yes, small batch size can make regularization. With small batch size, network need to adapt to \"unseen\" samples on each new batch. So features, that badly generalize to unseen data, will not survive with small batch size (big gradient from new batch will destroy these features).
", "rawMarkdown": "Yes, small batch size can make regularization. With small batch size, network need to adapt to \"unseen\" samples on each new batch. So features, that badly generalize to unseen data, will not survive with small batch size (big gradient from new batch will destroy these features).", "votes": null } ], "comments": [ { "id": 1270590, "author_name": "rsmits", "author_url": "", "post_date": "04/11/2021 19:20:43", "content": "Hey @drzhuzhe May'be try more and different things? Try out different losses and combinations of those. Try out different backbones. Try out may'be other frameworks than just Unet.
\nGood luck.
", "votes": null, "replies": [ { "id": 1270794, "author_name": "drzhuzhe", "author_url": "", "post_date": "04/12/2021 03:05:09", "content": "About batchsize: From my opinion, 1024 batchsize is too large. Usually 32 or 64 is enough.
\nAbout backbone: efficientb3 can also get 0.92x results on LB.
\nAbout epochs: Try to use pretrain Encode weights from imagenet. Usually it will converge in 30 epochs. In my case, I use AdamW optimizer. And set initial lr to 1e-4.
Big batch size requires high learning rate.
\nThere is an anology - imagine a pool table with 3 billiard balls randomly moving, and we will draw a center of mass of these balls. What trajectory these center of mass will have? With 3 balls that will be a trajectory with big oscillations. But what happends if there will be 1000 balls? Center of mass will not moving, because random directions of balls will mutually compensate each other.
\nSame situation with big batch size - gradients of different samples will mutually compensate each other, so final gradient will be smaller, so higher learning rate is required
", "votes": null, "replies": [ { "id": 1272188, "author_name": "drzhuzhe", "author_url": "", "post_date": "04/13/2021 09:48:47", "content": "Really awesome explain, dude
\nBut I still have two question make me very confuse
\nI set my optimizer a learning rate reduce
\nwith factor 0.1 and minimal: 0.01 * my initial learning rate
\ndoes my minimal learning rate need scaler up with initial learning rate ?
Does 1024 such a large batch size really hurt metrics score?
\nsome paper said small batch make more Regularization to optimizer, so larger batch may
\nin disadvantage
but other paper said large batch make gradient change more slowly, so what I just need is only training more epoch till convergence ?
Due to my CV score is too close, I cannot figure out which change in my params make sense
", "votes": null, "replies": [] }, { "id": 1272245, "author_name": "zavodrobotov", "author_url": "", "post_date": "04/13/2021 10:44:27", "content": "Yes, small batch size can make regularization. With small batch size, network need to adapt to \"unseen\" samples on each new batch. So features, that badly generalize to unseen data, will not survive with small batch size (big gradient from new batch will destroy these features).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1270248": "# I made some config wrong ?\n\n \n\nmy best CV score is about 9.28 +-0.02 , LB is about 9.18 \n\nso **I never reach CV score 9.33+**\n\nI read many efficientnet+unet notebook ,\n\nmaybe you can take a look at my work \n\nand tell me what I made mistake on architecture or params tuning ? to get a higher CV score?\n\n \n\n# Here is my notebook: \n\n \n\n**Preproccess tfrecord**:https://www.kaggle.com/drzhuzhe/hubmap-tf-with-tpu-efficientunet-256-tfrecord\n**trainning**: https://www.kaggle.com/drzhuzhe/hubmap-efficientnet-and-linknet-train thanks to wrrosa and vgarshin's work\n**infer**: https://www.kaggle.com/drzhuzhe/efficientnet-linknet-or-unet\n\n \n\n \n\n## Preproccess tfrecord: \n\n \n\n1. reduce 1024 tile to 256 tiles\n2. shift every tile 512 \n3. create tfrecord for tpu\n\n \n\n \n\n## Training :\n\n \n\n- what works\n\n1. add some augments like **hue** **brightness** **saturation** **contrast**\n2. *larger **batchsize** my batchsize is **1024** will significant speed up TPU trainning, but I don't know whether I set learning rate 5e-4 enough?*\n3. **more epoch** some times training seem unstable suddenly more epoch may leads to an finally converge \n\n \n\n- what doesn't work \n\n1. optimize.lookahead seems not improve \n\n \n\n- what hurt \n\n1. jpeg_quality noise will hurt CV and LB score \n2. **smaller Learning rat** that 5e-4 like 5e-5 or **minimal Learning rate** after 1e-6 may hurt", "1270590": "Hey @drzhuzhe May'be try more and different things? Try out different losses and combinations of those. Try out different backbones. Try out may'be other frameworks than just Unet.\n\nGood luck.", "1270794": "1. [loss]: I try binaryCrossEntropy but seems not benefit\n\n2. [backbone]: I try efficientbet b4 but it make training 40% slower but not give an improve on CV too\n\n3. [framework]: I try linknet too, but I use as same optimize params as Unet seems as Unet seems not \nwork well...\n\nwhat I most suspect is my Batch Size and learning rate doesn't config right", "1270919": "About batchsize: From my opinion, 1024 batchsize is too large. Usually 32 or 64 is enough.\nAbout backbone: efficientb3 can also get 0.92x results on LB. \nAbout epochs: Try to use pretrain Encode weights from imagenet. Usually it will converge in 30 epochs. In my case, I use AdamW optimizer. And set initial lr to 1e-4.", "1272162": "Big batch size requires high learning rate. \n\nThere is an anology - imagine a pool table with 3 billiard balls randomly moving, and we will draw a center of mass of these balls. What trajectory these center of mass will have? With 3 balls that will be a trajectory with big oscillations. But what happends if there will be 1000 balls? Center of mass will not moving, because random directions of balls will mutually compensate each other. \n\nSame situation with big batch size - gradients of different samples will mutually compensate each other, so final gradient will be smaller, so higher learning rate is required", "1272188": "Really awesome explain, dude\n\nBut I still have two question make me very confuse \n\n1. I set my optimizer **a learning rate reduce**\n\n with factor 0.1 and minimal: 0.01 * my initial learning rate\n\n does my **minimal learning rate** need scaler up with initial learning rate ?\n\n\n2. Does 1024 such a large batch size really hurt metrics score? \n\n some paper said **small batch make more Regularization to optimizer**, so larger batch may \nin disadvantage\n\n but other paper said large batch make gradient change more slowly, so what I just need is only training more epoch till convergence ?\n\n\nDue to my CV score is too close, I cannot figure out which change in my params make sense", "1272245": "Yes, small batch size can make regularization. With small batch size, network need to adapt to \"unseen\" samples on each new batch. So features, that badly generalize to unseen data, will not survive with small batch size (big gradient from new batch will destroy these features)." }, "source": "meta" }