![\"\"](\"https://i.redd.it/8rm53y9puf141.png\")
![\"\"](\"https://neurohive.io/wp-content/uploads/2019/11/Screenshot-from-2019-11-26-23-34-49-570x419.png\")
version 20191230:\n- densenet 121 ensemble + swa without cyclic rate, without bn refinement\n- inference at 25 min
\n\nversion 20200111:\n- se-resnext50 + balanced sampler + mixup (please refer to readme.ppt at the google drive
\n\nversion 20200224:\n- modified se-resnext50 with augmentation that works. please see read file in folder. This version can gives CV 0.991 and LB 0.980
\n\nhttps://drive.google.com/open?id=1A5CygainZ4rO_rOs4mUjQk378WlF8MrN
\n\n... to be updated ...\ne.g. \n- next version mixup,cutout, manifold mixup, adversarial loss?\nsee https://github.com/rois-codh/kmnist
\n\nmetric learning loss ... large-margin softmax, cosine, center loss?
long tail, distribution, loss weighing, balance sampling, etc?
sub class, metric distance without triplet
attention-based (stroke based?), human in-the-loop
LSTM, RNN, CTC loss, stroke parsing?
spatial transformation net, bounding box normalisation
feature embedding (https://github.com/qychen13/DifficultyAwareEmbedding), metric learning (https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109)
fine grained, attribute learning
version 20191230:\n- densenet 121 ensemble + swa without cyclic rate, without bn refinement\n- inference at 25 min
\n\nversion 20200111:\n- se-resnext50 + balanced sampler + mixup (please refer to readme.ppt at the google drive
\n\nversion 20200224:\n- modified se-resnext50 with augmentation that works. please see read file in folder. This version can gives CV 0.991 and LB 0.980
\n\nhttps://drive.google.com/open?id=1A5CygainZ4rO_rOs4mUjQk378WlF8MrN
\n\n... to be updated ...\ne.g. \n- next version mixup,cutout, manifold mixup, adversarial loss?\nsee https://github.com/rois-codh/kmnist
\n\nmetric learning loss ... large-margin softmax, cosine, center loss?
long tail, distribution, loss weighing, balance sampling, etc?
sub class, metric distance without triplet
attention-based (stroke based?), human in-the-loop
LSTM, RNN, CTC loss, stroke parsing?
spatial transformation net, bounding box normalisation
feature embedding (https://github.com/qychen13/DifficultyAwareEmbedding), metric learning (https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109)
fine grained, attribute learning
this is one of the kaggle competitions that i did the most number of experiments. I notice something interesting.
\n\nfor several experiments, it is important to apply the method at the early start of training. e.g. you get different results if you apply ohem on trained models (i.e. end stage of training) versus you apply it right at the start of training.
\n\ni think that when data size is small, the number of good solutions (or generalized solutions) region is small. i.e. there are many local minimum. It is difficult to jump from one solution to another solution
", "rawMarkdown": "this is one of the kaggle competitions that i did the most number of experiments. I notice something interesting.\n\nfor several experiments, it is important to apply the method at the early start of training. e.g. you get different results if you apply ohem on trained models (i.e. end stage of training) versus you apply it right at the start of training.\n\ni think that when data size is small, the number of good solutions (or generalized solutions) region is small. i.e. there are many local minimum. It is difficult to jump from one solution to another solution \n", "votes": 8, "replies": [ { "id": 771347, "postDate": "2020-03-14T04:07:33.507Z", "content": "I do find out applying ohem at the end stage of training will save you from overfitting. It will shake a little bit at the beginning but it will increase your local cv at the end. It seems like applying ohem make it jumping out of its local minimum.
", "rawMarkdown": "I do find out applying ohem at the end stage of training will save you from overfitting. It will shake a little bit at the beginning but it will increase your local cv at the end. It seems like applying ohem make it jumping out of its local minimum.", "votes": 3 } ] }, { "id": 756402, "postDate": "2020-02-25T17:37:08.607Z", "content": "this is a reference version for those who are struggling:
\n\nversion 20200224:\n- modified se-resnext50 (64x112 small input) with augmentation that works. please see readme file in folder. \n This version can gives CV 0.991 and LB 0.980 after you optimized the augmentation hyper parameters yourselves.
\n\nit provides reference log file for you to compare the loss curve
it has train/validation split that has gap of CV/LB 0.011
code is not complete, but should have enough details to reproduce the above mentioned results.\n(do not request me for missing files or functions)
Thanks for sharing and for the best thread in the discussion!
", "rawMarkdown": "Thanks for sharing and for the best thread in the discussion!" }, { "id": 756562, "postDate": "2020-02-25T20:56:28.917Z", "content": "Hi Heng Thanks for the update. May I ask if the 64x112 is a simple downsize from the original 137x236 image or did you do some post processing?
\n\nAnd may I ask how long have you trained the model to reach cv ~.99. From that log file it seems it's been training for 500 epochs?!
", "rawMarkdown": "Hi Heng Thanks for the update. May I ask if the 64x112 is a simple downsize from the original 137x236 image or did you do some post processing?\n\nAnd may I ask how long have you trained the model to reach cv ~.99. From that log file it seems it's been training for 500 epochs?!" }, { "id": 756712, "postDate": "2020-02-26T01:43:17.457Z", "content": "please refer to the readme file
", "rawMarkdown": "please refer to the readme file", "votes": 2 }, { "id": 757358, "postDate": "2020-02-26T17:09:16.117Z", "content": "how to train a modified resenext model with pretrained imagenet model
\n\nuse the unmodified version resenext model first. initialised with pretrained imagenet model and train as usual.
assume resenext = block0, block1 ... block4. we modify block0.
load the previously trained model except for the the modified part (i.e block0).
freeze all modified part (including the classifier head). Train and the gradient will only back-propagated at the modified block0.
when the accuracy/loss for the modified model is about the same as the previously unmodified one, say 90% of the results, unfreeze all model and proceed training as usual.
this is how i do network surgery. apply this when there is structural change of the model, e.g. add new convolution layers, reduce number of channels, change kernel size, ...
\n\nif the old pretrained imagenet model can be loaded after modification (e.g. simple modification like changing strike), there is no need to freeze and unfreeze. In such minor modification, just train as usual.
", "rawMarkdown": "how to train a modified resenext model with pretrained imagenet model\n\n1. use the unmodified version resenext model first. initialised with pretrained imagenet model and train as usual.\n\n2. assume resenext = block0, block1 ... block4. we modify block0.\n\n3. load the previously trained model except for the the modified part (i.e block0).\n\n4. freeze all modified part (including the classifier head). Train and the gradient will only back-propagated at the modified block0.\n\n5. when the accuracy/loss for the modified model is about the same as the previously unmodified one, say 90% of the results, unfreeze all model and proceed training as usual.\n\nthis is how i do network surgery. apply this when there is structural change of the model, e.g. add new convolution layers, reduce number of channels, change kernel size, ...\n\n---\n\nif the old pretrained imagenet model can be loaded after modification (e.g. simple modification like changing strike), there is no need to freeze and unfreeze. In such minor modification, just train as usual.\n ", "votes": 3 }, { "id": 758076, "postDate": "2020-02-27T12:16:27.667Z", "content": "purely trained from scratch \"without surgery above\".\nno augmentation is used at all!
\n\nlocal cv = 0.972 (at rate 0.05)\nlocal cv = 0.972 (at rate 0.005)
\n\ntraining loss is almost zero, so there is no point to train further. expected LB should be around 0.960 (without augmentation)
\n\nsee attached file.\n```\ndef valid_augment(image, label, infor):\n image = to_64x112(image)\n return image, label, infor
\n\ndef train_augment(image, label, infor):\n original = image.copy()\n original = to_64x112(original)\n image = to_64x112(image)\n return original, image, label, infor
\n\ndef train_batch_augment(original, input, onehot):\n return input, onehot
\n\n0.05000 9.0 15.6 | 0.972 : 0.957 0.989 0.982 0.955 | 0.20, 0.06, 0.06, 0.210 : 0.96, 0.99, 0.99, 0.956 | 0.00, 0.00, 0.00, 0.009 | 1 hr 38 min\n0.05000 10.0 17.3 | 0.972 : 0.957 0.990 0.986 0.955 | 0.21, 0.07, 0.07, 0.217 : 0.96, 0.99, 0.99, 0.956 | 0.00, 0.00, 0.00, 0.007 | 1 hr 49 min\n0.05000 11.0 19.0 | 0.971 : 0.957 0.989 0.982 0.955 | 0.21, 0.06, 0.07, 0.214 : 0.96, 0.99, 0.99, 0.956 | 0.01, 0.00, 0.00, 0.008 | 2 hr 00 min\n0.05000 12.0* 20.8 | 0.970 : 0.955 0.990 0.982 0.953 | 0.22, 0.06, 0.07, 0.224 : 0.96, 0.99, 0.99, 0.954 | 0.00, 0.00, 0.00, 0.007 | 2 hr 11 min\n```\nnote: some kaggler reported about to get LB 0.974 without augmentation (which i think 0.984 for local CV). hence i need to investigate better model or regularization (like dropout, shakedrop, etc) or better optimiser
", "rawMarkdown": "purely trained from scratch \"without surgery above\".\nno augmentation is used at all!\n\nlocal cv = 0.972 (at rate 0.05)\nlocal cv = 0.972 (at rate 0.005)\n\ntraining loss is almost zero, so there is no point to train further. expected LB should be around 0.960 (without augmentation)\n\n\nsee attached file.\n```\ndef valid_augment(image, label, infor):\n image = to_64x112(image)\n return image, label, infor\n\n\ndef train_augment(image, label, infor):\n original = image.copy()\n original = to_64x112(original)\n image = to_64x112(image)\n return original, image, label, infor\n\ndef train_batch_augment(original, input, onehot):\n return input, onehot\n\n0.05000 9.0 15.6 | 0.972 : 0.957 0.989 0.982 0.955 | 0.20, 0.06, 0.06, 0.210 : 0.96, 0.99, 0.99, 0.956 | 0.00, 0.00, 0.00, 0.009 | 1 hr 38 min\n0.05000 10.0 17.3 | 0.972 : 0.957 0.990 0.986 0.955 | 0.21, 0.07, 0.07, 0.217 : 0.96, 0.99, 0.99, 0.956 | 0.00, 0.00, 0.00, 0.007 | 1 hr 49 min\n0.05000 11.0 19.0 | 0.971 : 0.957 0.989 0.982 0.955 | 0.21, 0.06, 0.07, 0.214 : 0.96, 0.99, 0.99, 0.956 | 0.01, 0.00, 0.00, 0.008 | 2 hr 00 min\n0.05000 12.0* 20.8 | 0.970 : 0.955 0.990 0.982 0.953 | 0.22, 0.06, 0.07, 0.224 : 0.96, 0.99, 0.99, 0.954 | 0.00, 0.00, 0.00, 0.007 | 2 hr 11 min\n```\nnote: some kaggler reported about to get LB 0.974 without augmentation (which i think 0.984 for local CV). hence i need to investigate better model or regularization (like dropout, shakedrop, etc) or better optimiser", "votes": 2 }, { "id": 758342, "postDate": "2020-02-27T16:49:19.633Z", "content": "additional information:
\n\nmodified se-resnext50 (32x56 mini input) with same augmentation that works gives CV 0.987\nthis may be useful for auto-augmentation parameters search?
", "rawMarkdown": "additional information:\n\nmodified se-resnext50 (32x56 mini input) with same augmentation that works gives CV 0.987\nthis may be useful for auto-augmentation parameters search?\n\n \n", "votes": 4 }, { "id": 759575, "postDate": "2020-02-29T07:58:51.023Z", "content": "Thanks for your sharing,but I have a question.\nIn your train_rand_aug2c.py,there is a func named INVERSE_COMPOSE,but you didn't assign it before using，I guess it is used to add information for first three predict form firth .\nCould you explain its meaning in detail? Thanks！
", "rawMarkdown": "Thanks for your sharing,but I have a question.\nIn your train_rand_aug2c.py,there is a func named INVERSE_COMPOSE,but you didn't assign it before using，I guess it is used to add information for first three predict form firth .\nCould you explain its meaning in detail? Thanks！" }, { "id": 759588, "postDate": "2020-02-29T08:24:02.713Z", "content": "we know that each of the 1295 grapheme is made up of grapheme=root, vowel, consonant
\n\nso if you have predicted the 1295 class, simply decode (root, vowel, consonant) = grapheme
", "rawMarkdown": "we know that each of the 1295 grapheme is made up of grapheme=root, vowel, consonant\n\n\nso if you have predicted the 1295 class, simply decode (root, vowel, consonant) = grapheme" }, { "id": 759960, "postDate": "2020-02-29T16:56:38.817Z", "content": "Oh,yes.I understand your idea.
\n\nBut I got another question when I use your code to train the model .Most of the time my GPU utilization ratio is 0 and I thought the reason for this is dataloder's speed limitation.When we load the data by dataloader,we do some augmentation,and before send data to the net,maybe do some mixup or cutout.These process are not handled by GPU but CPU.
\n\nThis problem made my training process very slow,but in your log.train,it shows the speed is not so slow,so I want to ask whether you use some tricks to speed up.
", "rawMarkdown": "Oh,yes.I understand your idea.\n\nBut I got another question when I use your code to train the model .Most of the time my GPU utilization ratio is 0 and I thought the reason for this is dataloder's speed limitation.When we load the data by dataloader,we do some augmentation,and before send data to the net,maybe do some mixup or cutout.These process are not handled by GPU but CPU.\n\nThis problem made my training process very slow,but in your log.train,it shows the speed is not so slow,so I want to ask whether you use some tricks to speed up.\n" }, { "id": 759969, "postDate": "2020-02-29T17:13:32.273Z", "content": "df['xxx'].values should be done once at initialisation and not everytime at getitem()
", "rawMarkdown": "df['xxx'].values should be done once at initialisation and not everytime at getitem()" }, { "id": 759981, "postDate": "2020-02-29T17:27:09.323Z", "content": "```
\n\nclass KaggleDataset(Dataset):\n def init(self, split, mode, csv, parquet, augment=None):\n global TRAIN_PARQUET\n ...\n self.df_values = df.values
\n\ndef __getitem__(self, index): \n i, image_id, grapheme_root, vowel_diacritic, consonant_diacritic, grapheme, grapheme_symbol = self.df_values[index]\n```
", "rawMarkdown": "```\n\n\nclass KaggleDataset(Dataset):\n def __init__(self, split, mode, csv, parquet, augment=None):\n global TRAIN_PARQUET\n ...\n self.df_values = df.values\n\n def __getitem__(self, index): \n i, image_id, grapheme_root, vowel_diacritic, consonant_diacritic, grapheme, grapheme_symbol = self.df_values[index]\n\n```" }, { "id": 760463, "postDate": "2020-03-01T10:33:43.803Z", "content": "Thank you,I will have a try.
", "rawMarkdown": "Thank you,I will have a try." }, { "id": 764147, "postDate": "2020-03-05T07:19:47Z", "content": "Thanks for sharing, very helpful! Quick question, when I'm using your model ResNext50, it seems that the model takes more than 11GB of GPU memory. Did you encounter similar issue?
Or does this model only work when you freeze part of the model and only train its head?
\n\nThanks.
", "rawMarkdown": "Thanks for sharing, very helpful! Quick question, when I'm using your model `ResNext50`, it seems that the model takes more than 11GB of GPU memory. Did you encounter similar issue? \n\nOr does this model only work when you freeze part of the model and only train its head?\n\nThanks. " }, { "id": 765755, "postDate": "2020-03-07T03:43:19.483Z", "content": "I am still trying to figure out how to write inverse_compose function. Any idea ?
", "rawMarkdown": "I am still trying to figure out how to write inverse_compose function. Any idea ?" }, { "id": 766504, "postDate": "2020-03-08T09:09:05.520Z", "content": "\n\n```\ndef run_prepare_data4():\n df_grapheme = pd.read_csv(DATA_DIR+'/grapheme_1295.csv')\n df_root = pd.read_csv(DATA_DIR+'/root_168.csv')\n df_vowel = pd.read_csv(DATA_DIR+'/vowel_11.csv')\n df_consonant = pd.read_csv(DATA_DIR+'/consonant_7.csv')
\n\ngrapheme_map = dict(df_grapheme[['grapheme','label']].values)\nroot_map = dict(df_root[['root','label']].values)\nvowel_map = dict(df_vowel[['vowel','label']].values)\nconsonant_map = dict(df_consonant[['consonant','label']].values)\n\ngrapheme_inverse_map = dict(df_grapheme[['label','grapheme']].values)\nroot_inverse_map = dict(df_root[['label','root']].values)\nvowel_inverse_map = dict(df_vowel[['label','vowel']].values)\nconsonant_inverse_map = dict(df_consonant[['label','consonant',]].values)\n\n#----\ndf_train = pd.read_csv(DATA_DIR+'/train.csv')\ndf_train['grapheme']=df_train['grapheme'].map(grapheme_map)\nassert(df_train['grapheme'].isnull().values.any() == False)\n\ncompose = {}\ninverse_compose = {}\ngb = df_train.groupby(['grapheme',])\nfor i in range(1295):\n gp = gb.get_group(i) #.index\n\n gp = gp.drop_duplicates(subset=['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'])\n assert(len(gp)==1)\n\n g = i\n r,v,c = gp[['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']].values[0]\n\n inverse_compose[g]= (r,v,c)\n compose[grapheme_inverse_map[g]]= (\n root_inverse_map[r],\n vowel_inverse_map[v],\n consonant_inverse_map[c]\n )\n\nprint(inverse_compose)\nprint(compose)\n```
", "rawMarkdown": "@phoenix9032 \n\n```\ndef run_prepare_data4():\n df_grapheme = pd.read_csv(DATA_DIR+'/grapheme_1295.csv')\n df_root = pd.read_csv(DATA_DIR+'/root_168.csv')\n df_vowel = pd.read_csv(DATA_DIR+'/vowel_11.csv')\n df_consonant = pd.read_csv(DATA_DIR+'/consonant_7.csv')\n\n\n grapheme_map = dict(df_grapheme[['grapheme','label']].values)\n root_map = dict(df_root[['root','label']].values)\n vowel_map = dict(df_vowel[['vowel','label']].values)\n consonant_map = dict(df_consonant[['consonant','label']].values)\n\n grapheme_inverse_map = dict(df_grapheme[['label','grapheme']].values)\n root_inverse_map = dict(df_root[['label','root']].values)\n vowel_inverse_map = dict(df_vowel[['label','vowel']].values)\n consonant_inverse_map = dict(df_consonant[['label','consonant',]].values)\n\n #----\n df_train = pd.read_csv(DATA_DIR+'/train.csv')\n df_train['grapheme']=df_train['grapheme'].map(grapheme_map)\n assert(df_train['grapheme'].isnull().values.any() == False)\n\n compose = {}\n inverse_compose = {}\n gb = df_train.groupby(['grapheme',])\n for i in range(1295):\n gp = gb.get_group(i) #.index\n\n gp = gp.drop_duplicates(subset=['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'])\n assert(len(gp)==1)\n\n g = i\n r,v,c = gp[['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']].values[0]\n\n inverse_compose[g]= (r,v,c)\n compose[grapheme_inverse_map[g]]= (\n root_inverse_map[r],\n vowel_inverse_map[v],\n consonant_inverse_map[c]\n )\n\n print(inverse_compose)\n print(compose)\n\n```", "votes": 1 }, { "id": 766512, "postDate": "2020-03-08T09:21:13.480Z", "content": "Thanks so much Heng :)
", "rawMarkdown": "Thanks so much Heng :) " } ] }, { "id": 737223, "postDate": "2020-02-05T04:15:29.610Z", "content": "i sometime have discussions with friends on their phd research work. recently, they give me a few pointers on network design:
\n\nit is possible that a single batch norm cannot normalized the feature well for multi-task. different task needs different normalization. Care needs to be taken at designing the head
you can force the network to learn high frequency features (e.g. edge, etc) over texture by adding loss related to high frequency feature extraction , together with the usual cross entropy classification loss.
related:\nusing Adversarial Examples to expand dataset
\n\n\"With an enhanced EfficientNet-B8, our method achieves the state-of-the-art 85.5% ImageNet\ntop-1 accuracy without extra data. This result even surpasses the best model in [20] which is trained with 3.5B Instagram images (∼3000× more than ImageNet) and ∼9.4× more parameters.\"
\n\nAdversarial Examples Improve Image Recognition\nhttps://arxiv.org/abs/1911.09665\nhttps://www.youtube.com/watch?v=KTCztkNJm50
\n\n
see also:\nCompounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network https://arxiv.org/pdf/2001.06268.pdf
", "rawMarkdown": "related:\nusing Adversarial Examples to expand dataset\n\n\"With an enhanced EfficientNet-B8, our method achieves the state-of-the-art 85.5% ImageNet\ntop-1 accuracy without extra data. This result even surpasses the best model in [20] which is trained with 3.5B Instagram images (∼3000× more than ImageNet) and ∼9.4× more parameters.\"\n\nAdversarial Examples Improve Image Recognition\nhttps://arxiv.org/abs/1911.09665\nhttps://www.youtube.com/watch?v=KTCztkNJm50\n\n \n\nsee also:\nCompounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network https://arxiv.org/pdf/2001.06268.pdf\n\n\n", "votes": 1 } ] }, { "id": 723886, "postDate": "2020-01-20T15:16:12.777Z", "content": "magic net
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fd43e2775c784672d47d6b37cc582cb90%2FSelection_073.png?generation=1579533369553774&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fd5ff591bb2d20c7c87753e6c2170189b%2FSelection_079.png?generation=1579594728840307&alt=media\")
single model, single fold performance:\n```\nLB = 0.9755\nCV = 0.983629\n grapheme_root 0.977137\n vowel_diacritic 0.993395\n consonant_diacritic 0.986848
\n\nmodel: SE-Resnext50 with stride replaced with max pooling + SWA\ninput: 137x236\naugmentation: mixup + cut-fade
\n\n```
\n\nafter more tunning:
\n\n```\nLB = 0.9760\nCV = 0.984155\n grapheme_root 0.978920\n vowel_diacritic 0.992851\n consonant_diacritic 0.985929
\n\n```
", "rawMarkdown": "single model, single fold performance:\n```\nLB = 0.9755\nCV = 0.983629\n grapheme_root 0.977137\n vowel_diacritic 0.993395\n consonant_diacritic 0.986848 \n\nmodel: SE-Resnext50 with stride replaced with max pooling + SWA\ninput: 137x236\naugmentation: mixup + cut-fade\n\n```\n\nafter more tunning:\n\n```\nLB = 0.9760\nCV = 0.984155\n grapheme_root 0.978920\n vowel_diacritic 0.992851\n consonant_diacritic 0.985929\n \n\n\n```" }, { "id": 726108, "postDate": "2020-01-22T20:27:47.137Z", "content": "Hi Heng! I am trying to automate the 'magic' net architecture. First I replaced de stride 2 to 1, saving the conv 'path':
\n\n\nmodules = {}\nfor name, module in model.named_modules():\n if(isinstance(module, nn.Conv2d)):\n stride = module.stride\n if stride == (2, 2) or stride == 2:\n module.stride = (1,1)\n modules[name] = module\n elif stride == 2:\n module.stride = 1\n modules[name] = module\n
Next I try to insert the maxpool as follows:
\n\n```\nfor name in modules:\n parent_module = model\n objs = name.split(\".\")\n if len(objs) == 1:\n #model.setattr(name, modules[name])\n model.setattr(\"magicMaxPool\", nn.MaxPool2d(kernel_size=2, stride=2))\n continue
\n\nfor obj in objs[:-1]:\n parent_module = parent_module.__getattr__(obj)\n\nparent_module.__setattr__(\"magicMaxPool\", nn.MaxPool2d(kernel_size=2, stride=2))\n```
\n\nIf I inspect the net/modules, it appears the maxpool but when I do the inference stride works properly but maxpool is not used (vector size before global avg pool is too big).
", "rawMarkdown": "Hi Heng! I am trying to automate the 'magic' net architecture. First I replaced de stride 2 to 1, saving the conv 'path':\n\n```\nmodules = {}\nfor name, module in model.named_modules():\n if(isinstance(module, nn.Conv2d)):\n stride = module.stride\n if stride == (2, 2) or stride == 2:\n module.stride = (1,1)\n modules[name] = module\n elif stride == 2:\n module.stride = 1\n modules[name] = module\n```\n\nNext I try to insert the maxpool as follows:\n\n```\nfor name in modules:\n parent_module = model\n objs = name.split(\".\")\n if len(objs) == 1:\n #model.__setattr__(name, modules[name])\n model.__setattr__(\"magicMaxPool\", nn.MaxPool2d(kernel_size=2, stride=2))\n continue\n\n for obj in objs[:-1]:\n parent_module = parent_module.__getattr__(obj)\n\n parent_module.__setattr__(\"magicMaxPool\", nn.MaxPool2d(kernel_size=2, stride=2))\n```\n\nIf I inspect the net/modules, it appears the maxpool but when I do the inference stride works properly but maxpool is not used (vector size before global avg pool is too big).", "votes": 1 }, { "id": 729048, "postDate": "2020-01-25T16:46:53.873Z", "content": "related: http://openaccess.thecvf.com/content_cvpr_2017/papers/Zhai_S3Pool_Pooling_With_CVPR_2017_paper.pdf
\n\nS3Pool: Pooling with Stochastic Spatial Sampling
", "rawMarkdown": "related: http://openaccess.thecvf.com/content_cvpr_2017/papers/Zhai_S3Pool_Pooling_With_CVPR_2017_paper.pdf\n\nS3Pool: Pooling with Stochastic Spatial Sampling", "votes": 2 }, { "id": 735731, "postDate": "2020-02-03T11:53:18.557Z", "content": "Hi @hengck23 \nI always look at the code you write and I am very grateful.\nI try to training Resnext50 with magic net,But cv value did not go above 0.96. Did you do anything other than change the layer?
", "rawMarkdown": "Hi @hengck23 \nI always look at the code you write and I am very grateful.\nI try to training Resnext50 with magic net,But cv value did not go above 0.96. Did you do anything other than change the layer?" } ] }, { "id": 707588, "postDate": "2020-01-01T05:35:10.807Z", "content": "after a series of experiments, how are my conclusion:
\n\nbaseline results:\n a. 0.9675~0.9715\n - standard method (joint 3 class) and ensemble standard model (e.g. densenet, resnet, efficientnet, etc) \n - to maximize performance, search for best input size (e.g. original size, enlarged size like 224x224,256x256, or even larger), try better augmentation, train longer with better learning rate (swa, cyclic annealing, ...), etc\n - size, scale normalization
\n\nb. 0.9700~0.9800\n... to be updated ... (better formulation, loss etc?)
\n\nc. 0.9800~0.9900\n... to be updated ... (some way to create more data?)
\n\ndon't forget Google Cloud AutoML Vision as part of solution !!!!
", "rawMarkdown": "after a series of experiments, how are my conclusion:\n\n1. final first LB ranking: 0.99+ after 3 months\n2. shakeup: +/- 0.002 (should be quite small)\n\nbaseline results:\n a. 0.9675~0.9715\n - standard method (joint 3 class) and ensemble standard model (e.g. densenet, resnet, efficientnet, etc) \n - to maximize performance, search for best input size (e.g. original size, enlarged size like 224x224,256x256, or even larger), try better augmentation, train longer with better learning rate (swa, cyclic annealing, ...), etc\n - size, scale normalization\n \n b. 0.9700~0.9800\n... to be updated ... (better formulation, loss etc?)\n\n c. 0.9800~0.9900\n... to be updated ... (some way to create more data?)\n\ndon't forget Google Cloud AutoML Vision as part of solution !!!!", "votes": 8, "replies": [ { "id": 708848, "postDate": "2020-01-02T19:00:30.557Z", "content": "Thanks for the pointers @hengck23 . I started playing with augmentations . I have got the cutout working here .
\n\nhttps://www.kaggle.com/phoenix9032/pytorch-efficientnet-starter-code
\n\nNow , need to see how MixUp and RICAP and other thing can work out along with the training for multi-heads . Need to make some modification from the original implementation.
\n\nWe can try dual cutout as well.
", "rawMarkdown": "Thanks for the pointers @hengck23 . I started playing with augmentations . I have got the cutout working here . \n\nhttps://www.kaggle.com/phoenix9032/pytorch-efficientnet-starter-code\n\nNow , need to see how MixUp and RICAP and other thing can work out along with the training for multi-heads . Need to make some modification from the original implementation.\n\nWe can try dual cutout as well. ", "votes": 3 }, { "id": 719435, "postDate": "2020-01-15T14:01:09.920Z", "content": "some important (?) experimental results
\n\nsampling affects results (e.g. balance sampling of \"root\" or \"grapheme\" affects \"root\", \"constant\", \"vowel\" differently)
augmentation affects results (e.g. some drop in accuracy when scaling and rotation is used)
arcface and large-margin like softamx does improve results but cannot be used with mixup?
Hi Heng @hengck23 (and other Kagglers), I see in your latest starting kit that you added a fourth target of the 1295 unique classes of Bengali letters as provided in the training dataset. If the testing dataset (which is not accessible to us) has letters other than those 1295 classes, would the training approach be prone to overfitting? Please correct me if I am wrong.
", "rawMarkdown": "Hi Heng @hengck23 (and other Kagglers), I see in your latest starting kit that you added a fourth target of the 1295 unique classes of Bengali letters as provided in the training dataset. If the testing dataset (which is not accessible to us) has letters other than those 1295 classes, would the training approach be prone to overfitting? Please correct me if I am wrong. " }, { "id": 721132, "postDate": "2020-01-17T04:53:19.677Z", "content": "you can use low weight like 0.1 for the fourth class, you can also use 1295+1 (background class for none of the above). to create background class, selectively flip the original train samples
", "rawMarkdown": "you can use low weight like 0.1 for the fourth class, you can also use 1295+1 (background class for none of the above). to create background class, selectively flip the original train samples" } ] }, { "id": 758757, "postDate": "2020-02-28T05:38:01.697Z", "content": "good for augmentation?
\n\n![\"\"](\"https://raw.githubusercontent.com/warbean/tps_stn_pytorch/master/demo/top_1.gif\")
https://github.com/WarBean/tps_stn_pytorch
", "rawMarkdown": "good for augmentation?\n\n\n\nhttps://github.com/WarBean/tps_stn_pytorch", "votes": 5, "replies": [ { "id": 759014, "postDate": "2020-02-28T13:02:09.523Z", "content": "very interesting augmemtation!
", "rawMarkdown": "very interesting augmemtation!" } ] }, { "id": 724848, "postDate": "2020-01-21T14:51:04.107Z", "content": "in google doddle, you can add and subtract drawing. I wonder if this can be done in bengali grapheme? e.g. add/remove or replace vowel, etc\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F249a956c438d0dd9ef6f618882dde10a%2FSelection_087.png?generation=1579618261743829&alt=media\")
new augmentation
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fb35fa0d1187ed81acb31b38fedbc6828%2FSelection_069.png?generation=1579532069051469&alt=media\")
related: augmentation on feature map:
\n\nthis attenuate feature values, preventing it from over dominating and lead to over fitting
", "rawMarkdown": "related: augmentation on feature map:\n\n1. choose some random feature map at training\n2. choose same max value\n3. modify: new value = alpha * old value, where alpha is between 0 to 1 (preferably near 0.5)\n\nthis attenuate feature values, preventing it from over dominating and lead to over fitting " } ] }, { "id": 775716, "postDate": "2020-03-17T00:23:49.243Z", "content": "private lb of coded posted here:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fda66de721f1d8955b9925e3642f5a0f0%2FSelection_074.png?generation=1584404626455936&alt=media\")
Amazing results but sorry to see they are not selected...This post is really helpful and I have learned a lot from your code for the first competition I decided to take part in seriously. Thank you!\nBy the way, did you forget to select final submission?😂 Or just because of the shake up?
", "rawMarkdown": "Amazing results but sorry to see they are not selected...This post is really helpful and I have learned a lot from your code for the first competition I decided to take part in seriously. Thank you!\nBy the way, did you forget to select final submission?😂 Or just because of the shake up?" } ] }, { "id": 765573, "postDate": "2020-03-06T20:08:44.370Z", "content": "there is one extra supervision signal.
\n\n\n list('ক্ট্রো')\n['ক', '্', 'ট', '্', 'র', 'ো']\n\na list command breaks the graheme into more component. you can use this the measure the amount of common parts (distance) between 2 graphemes
if this model is not correlated to image model, it is good for ensemble.\nthis work enables input of very large image
\n\nLearning in the Frequency Domain\nKai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-kuang Chen, Fengbo Ren
\n\n![\"\"](\"https://storage.googleapis.com/groundai-web-prod/media/users/user_12457/project_409932/images/x2.png\")
![\"\"](\"https://storage.googleapis.com/groundai-web-prod/media/users/user_12457/project_409932/images/x4.png\")
```
\n\nResNet-50 #Channels Size Per Channel Top-1 Top-5 Normalized Input Size
\nRGB 3 224x224 75.780 92.650 1.0
\nDCT-24 (ours) 24 56x56 77.196 93.504 0.5
```
", "rawMarkdown": "if this model is not correlated to image model, it is good for ensemble.\nthis work enables input of very large image\n\nLearning in the Frequency Domain\nKai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-kuang Chen, Fengbo Ren\n\n\n\n\n\n\n```\n\nResNet-50\t#Channels\tSize Per Channel\tTop-1\tTop-5\tNormalized Input Size\t\nRGB\t 3\t 224x224\t 75.780\t92.650\t1.0\t \t \nDCT-24 (ours) 24\t 56x56\t 77.196\t93.504\t0.5\t \t \n \n\n\n```", "votes": 3, "replies": [ { "id": 763297, "postDate": "2020-03-04T10:37:02.583Z", "content": "Thanks for sharing an interesting paper.\nThe code is available below.
\n\nhttps://github.com/calmevtime1990/supp/tree/master/classification
", "rawMarkdown": "Thanks for sharing an interesting paper.\nThe code is available below.\n\nhttps://github.com/calmevtime1990/supp/tree/master/classification", "votes": 1 } ] }, { "id": 728658, "postDate": "2020-01-25T03:28:10.440Z", "content": "top-1 and top-2 recall:
\n\n```\nlocal validation results
\n\navgerage recall (top-1): 0.982253\n grapheme_root 0.974616\n vowel_diacritic 0.992676\n consonant_diacritic 0.987103
\n\navgerage recall (top-2) : 0.995918\n grapheme_root 0.993451\n vowel_diacritic 0.998653\n consonant_diacritic 0.998116
\n\n```
\n\nif you make a mistake, the correct results is probably the top-2. if you can think of a good way to post-process, you can improve your score
", "rawMarkdown": "top-1 and top-2 recall:\n\n```\nlocal validation results\n\navgerage recall (top-1): 0.982253\n grapheme_root 0.974616\n vowel_diacritic 0.992676\n consonant_diacritic 0.987103\n \n\navgerage recall (top-2) : 0.995918\n grapheme_root 0.993451\n vowel_diacritic 0.998653\n consonant_diacritic 0.998116\n \n\n```\n\nif you make a mistake, the correct results is probably the top-2. if you can think of a good way to post-process, you can improve your score", "votes": 3, "replies": [ { "id": 729149, "postDate": "2020-01-25T20:10:12.520Z", "content": "this screams for an ensemble of models ;)
", "rawMarkdown": "this screams for an ensemble of models ;)" } ] }, { "id": 757812, "postDate": "2020-02-27T05:58:26.333Z", "content": "New paper today\nOn Feature Normalization and Data Augmentation
\n\nhttps://arxiv.org/pdf/2002.11102.pdf
", "rawMarkdown": "New paper today\nOn Feature Normalization and Data Augmentation\n\nhttps://arxiv.org/pdf/2002.11102.pdf", "votes": 4, "replies": [ { "id": 757817, "postDate": "2020-02-27T06:09:36.200Z", "content": "arXiv:2002.11022 (cross-list from cs.LG) [pdf, other]\nBeyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks\nYehui Tang, Yunhe Wang, Yixing Xu, Boxin Shi, Chao Xu, Chunjing Xu, Chang Xu\nSubjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
", "rawMarkdown": "arXiv:2002.11022 (cross-list from cs.LG) [pdf, other]\nBeyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks\nYehui Tang, Yunhe Wang, Yixing Xu, Boxin Shi, Chao Xu, Chunjing Xu, Chang Xu\nSubjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)", "votes": 1 } ] }, { "id": 706574, "postDate": "2019-12-30T15:27:50.050Z", "content": "What's the meaning of “without bn refinement”?
", "rawMarkdown": "What's the meaning of “without bn refinement”? ", "votes": 3, "replies": [ { "id": 706611, "postDate": "2019-12-30T16:07:09.213Z", "content": "I think Heng is talking about the recently proposed 'Full Normalization' technique, a better alternative (or you can say modification) to the conventional Batch Normalization. The idea is described in the following paper:\nhttps://arxiv.org/abs/1810.06177
", "rawMarkdown": "I think Heng is talking about the recently proposed 'Full Normalization' technique, a better alternative (or you can say modification) to the conventional Batch Normalization. The idea is described in the following paper:\nhttps://arxiv.org/abs/1810.06177" }, { "id": 706618, "postDate": "2019-12-30T16:11:02.760Z", "content": "https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/
\n\n```\nBATCH NORMALIZATION\nOne important detail to keep in mind is batch normalization. Batch normalization layers compute running statistics of activations during training. Note that the SWA averages of the weights are never used to make predictions during training, and so the batch normalization layers do not have the activation statistics computed after you reset the weights of your model with opt.swap_swa_sgd(). To compute the activation statistics you can just make a forward pass on your training data using the SWA model once the training is finished. In the SWA class we provide a helper function opt.bn_update(train_loader, model). It updates the activation statistics for every batch normalization layer in the model by making a forward pass on the train_loader data loader. You only need to call this function once in the end of training.
\n\n```
\n\nthe above step is not implemented
", "rawMarkdown": "https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/\n\n```\nBATCH NORMALIZATION\nOne important detail to keep in mind is batch normalization. Batch normalization layers compute running statistics of activations during training. Note that the SWA averages of the weights are never used to make predictions during training, and so the batch normalization layers do not have the activation statistics computed after you reset the weights of your model with opt.swap_swa_sgd(). To compute the activation statistics you can just make a forward pass on your training data using the SWA model once the training is finished. In the SWA class we provide a helper function opt.bn_update(train_loader, model). It updates the activation statistics for every batch normalization layer in the model by making a forward pass on the train_loader data loader. You only need to call this function once in the end of training.\n\n```\n\nthe above step is not implemented", "votes": 5 }, { "id": 706626, "postDate": "2019-12-30T16:19:48.630Z", "content": "@hengck23 Thanks for your clarification!
", "rawMarkdown": "@hengck23 Thanks for your clarification! " } ] }, { "id": 706492, "postDate": "2019-12-30T13:15:02.323Z", "content": "What's the meaning of swa ?
", "rawMarkdown": "What's the meaning of swa ?", "votes": 3, "replies": [ { "id": 706607, "postDate": "2019-12-30T16:03:32.357Z", "content": "Stochastic Weight Averaging, you can read this paper for more details:\nhttps://arxiv.org/abs/1803.05407
", "rawMarkdown": "Stochastic Weight Averaging, you can read this paper for more details:\nhttps://arxiv.org/abs/1803.05407", "votes": 4 }, { "id": 707056, "postDate": "2019-12-31T08:05:01.543Z", "content": "thx
", "rawMarkdown": "thx", "votes": 1 } ] }, { "id": 728668, "postDate": "2020-01-25T04:01:19.887Z", "content": "multi-task and task dependency
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fefb21bf6c5f708e84efb92a2a610f24f%2FClipboard05.png?generation=1579924874232793&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F0b65fbdbc95cf5730ca99eff92e7bf6f%2FClipboard04.png?generation=1579924877627631&alt=media\")
Im studying the correlation among tasks. What's this paper? Thanks!
", "rawMarkdown": "Im studying the correlation among tasks. What's this paper? Thanks!" } ] }, { "id": 726161, "postDate": "2020-01-22T21:58:56.863Z", "content": "loss formulation:
\n\nsome task (root, constant, or vowel) are more confidence than others.\nsimilarly, some classes are easier within a class.
\n\ne.g. given task 1 has class A,B,C,D,E and task 2 has class a,b,c and if we know that some class cannot co-exist, it may give better results if we apply the rules as follow:\n```\n1. naive method:\ntreat task 1 as 5 classes problem and task 2 as 3 classes. ignore any co-occurrence prior knowledge
\n\n```\nif you know the language, you may apply such rules
", "rawMarkdown": "loss formulation:\n\nsome task (root, constant, or vowel) are more confidence than others.\nsimilarly, some classes are easier within a class.\n\ne.g. given task 1 has class A,B,C,D,E and task 2 has class a,b,c and if we know that some class cannot co-exist, it may give better results if we apply the rules as follow:\n```\n1. naive method:\ntreat task 1 as 5 classes problem and task 2 as 3 classes. ignore any co-occurrence prior knowledge\n\n 2. use prior knowledge\nif input is a, then it can only be A,B,C\nif input is b, then it can only be A,B,E\nif input is c, then it can only be D,E\n\n```\nif you know the language, you may apply such rules\n", "votes": 4, "replies": [ { "id": 726925, "postDate": "2020-01-23T10:16:45.180Z", "content": "or we could change the architecture of model's head such that at the time of predicting task 2 we feed the probabilities(softmax) for task 1 predicted by our model.
", "rawMarkdown": "or we could change the architecture of model's head such that at the time of predicting task 2 we feed the probabilities(softmax) for task 1 predicted by our model.", "votes": 2 } ] }, { "id": 716656, "postDate": "2020-01-12T04:17:58.407Z", "content": "version 20200111:\nse-resnext50 + balanced sampler + mixup (please refer to readme.ppt at the google drive)
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F97673c14b01d4aff29fc1d688b036267%2FSelection_069.png?generation=1578830084131882&alt=media\")
@hengck23 thanks for your sharing again. Do you have the updated verison of compute_kaggle_metric in your kaggle.py?
```\ndef compute_kaggle_metric(probability, truth):
\n\ndef compute_recall(probability,truth):\n num_class = probability.shape[-1]\n y = probability.argmax(-1)\n t = truth\n correct = y==t\n\n recall = np.zeros(num_class)\n for c in range(num_class):\n e = correct[t==c]\n if len(e)>0:\n recall[c]=e.mean()\n return recall\n\ncomponet = []\nrecall = []\nfor p,t in zip(probability,truth):\n r = compute_recall(p,t)\n recall.append(r)\n componet.append(r.mean())\n\naverage = np.average(componet, weights=[2,1,1,0])\nreturn average, componet, recall\n```
", "rawMarkdown": "```\ndef compute_kaggle_metric(probability, truth):\n\n def compute_recall(probability,truth):\n num_class = probability.shape[-1]\n y = probability.argmax(-1)\n t = truth\n correct = y==t\n\n recall = np.zeros(num_class)\n for c in range(num_class):\n e = correct[t==c]\n if len(e)>0:\n recall[c]=e.mean()\n return recall\n\n componet = []\n recall = []\n for p,t in zip(probability,truth):\n r = compute_recall(p,t)\n recall.append(r)\n componet.append(r.mean())\n\n average = np.average(componet, weights=[2,1,1,0])\n return average, componet, recall\n```", "votes": 2 } ] }, { "id": 706712, "postDate": "2019-12-30T18:06:51.830Z", "content": "there are 168 grapheme_root, 11 vowel_diacritic, 7 consonant_diacritic and 1295 grapheme classes!\nyou can build a fourth classifier for 1295 grapheme classes and decode the component from it
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fbeafecfbcd56e0f8c5ee4ed6d98bd2b3%2FSelection_088.png?generation=1577729205397481&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F0e6843ab6fef11c4b7b478c45d0f07f2%2FSelection_087.png?generation=1577729209517703&alt=media\")
Find a way to balance per class accuracy. <- I thought this approach too and tried a bit although I don’t have convincing reason why it is helpful. Do you have it?
", "rawMarkdown": "Find a way to balance per class accuracy. <- I thought this approach too and tried a bit although I don’t have convincing reason why it is helpful. Do you have it?" }, { "id": 717364, "postDate": "2020-01-13T03:58:19.630Z", "rawMarkdown": "" }, { "id": 759126, "postDate": "2020-02-28T16:12:54.400Z", "content": "I was curious about how to build a fourth classifier for 1295 grapheme classes,is that just groupby grapheme and label it from 1 to 1295?\nHope you could clarify,thanks.
", "rawMarkdown": "I was curious about how to build a fourth classifier for 1295 grapheme classes,is that just groupby grapheme and label it from 1 to 1295?\nHope you could clarify,thanks." }, { "id": 759134, "postDate": "2020-02-28T16:19:22.130Z", "content": "\n\nyes, you are correct
", "rawMarkdown": "@thefatcat \n\nyes, you are correct" } ] }, { "id": 706395, "postDate": "2019-12-30T10:31:29.073Z", "content": "augmentation in the release 'version 20191230'
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F13e81e022c9b8ebf067517e15d62e938%2FSelection_081.png?generation=1577701870877750&alt=media\")
@hengck23 Hi. Can u tell what augmentations u used ? Thanks
", "rawMarkdown": "@hengck23 Hi. Can u tell what augmentations u used ? Thanks" } ] }, { "id": 716020, "postDate": "2020-01-11T06:47:03.763Z", "content": "I published a kernel based on this so that people can train it using kaggle GPUs
", "rawMarkdown": "I published a [kernel](https://www.kaggle.com/bibek777/heng-s-starter-training-kernel) based on this so that people can train it using kaggle GPUs", "votes": 2 }, { "id": 760615, "postDate": "2020-03-01T14:25:04.707Z", "content": "yet another paper:\nMaxUp: A Simple Way to Improve Generalization of Neural Network Training\n```
\n\nMethod Top-1 error Top-5 error\nVanilla (He et al., 2016a) 76.3 -\nDropout (Srivastava et al., 2014) 76.8 93.4\nDropPath (Larsson et al., 2017) 77.1 93.5\nManifold Mixup (Verma et al., 2019) 77.5 93.8\nAutoAugment (Cubuk et al., 2019a) 77.6 93.8\nMixup (Zhang et al., 2018) 77.9 93.9\nDropBlock (Ghiasi et al., 2018) 78.3 94.1\nCutMix (Yun et al., 2019) 78.6 94.0\nMaxUp+CutMix 78.9 94.2
\n\nTable 1: Summary of top1 and top5 accuracies on the validation set of ImageNet for ResNet-50.\n```
", "rawMarkdown": "yet another paper:\nMaxUp: A Simple Way to Improve Generalization of Neural Network Training\n```\n\nMethod\tTop-1 error\tTop-5 error\nVanilla (He et al., 2016a)\t76.3\t-\nDropout (Srivastava et al., 2014)\t76.8\t93.4\nDropPath (Larsson et al., 2017)\t77.1\t93.5\nManifold Mixup (Verma et al., 2019)\t77.5\t93.8\nAutoAugment (Cubuk et al., 2019a)\t77.6\t93.8\nMixup (Zhang et al., 2018)\t77.9\t93.9\nDropBlock (Ghiasi et al., 2018)\t78.3\t94.1\nCutMix (Yun et al., 2019)\t78.6\t94.0\nMaxUp+CutMix\t78.9\t94.2\n\nTable 1: Summary of top1 and top5 accuracies on the validation set of ImageNet for ResNet-50.\n```", "votes": 1 }, { "id": 758761, "postDate": "2020-02-28T05:43:32.973Z", "content": "it is possible to treat it as a detection problem\n![\"\"](\"https://raw.githubusercontent.com/RubanSeven/CRAFT_keras/master/images/CRAFT%E9%AB%98%E6%96%AF%E7%83%AD%E5%9B%BE.png\")
![\"\"](\"https://pythonawesome.com/content/images/2019/06/CRAFT.jpg\")
Didn't expect to see CRAFT here
", "rawMarkdown": "Didn't expect to see CRAFT here" }, { "id": 759082, "postDate": "2020-02-28T14:49:24.743Z", "content": "just an idea\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F2e6d456f54d997d4cfabfd1ad5a6af41%2FSelection_082.png?generation=1582901362622740&alt=media\")
https://github.com/sujatasaini/Kuzushiji-DropBlock
\n\n|Models | MNIST | Fashion-MNIST | Kuzushiji-MNIST | Kuzushiji-49 |\n|---------------------------------|-------|---------------|-----------------|--------------|\n|DCNN-DropBlock | 99.47% | 93.40% | 97.66% | 95.67% |\n|DCNN-Dropout | 97.99% | 85.47% | 86.43% | 95.34% |\n|DCNN-Spatial-Dropout | 97.17% | 84.44% | 81.08% | 58.18 |
", "rawMarkdown": "https://github.com/sujatasaini/Kuzushiji-DropBlock\n\n \n\n|Models | MNIST | Fashion-MNIST | Kuzushiji-MNIST | Kuzushiji-49 |\n|---------------------------------|-------|---------------|-----------------|--------------|\n|[DCNN-DropBlock](DropBlock/Kuzushiji-MNIST/train.py) | **99.47%** | **93.40%** | **97.66%** | **95.67%** |\n|[DCNN-Dropout](Dropout/train.py) | 97.99% | 85.47% | 86.43% | 95.34% |\n|[DCNN-Spatial-Dropout](SpatialDropout/train.py) | 97.17% | 84.44% | 81.08% | 58.18 |\n ", "votes": 1 }, { "id": 751629, "postDate": "2020-02-20T10:55:19.673Z", "content": "another augmentation that works for me:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F5800610b7d3b4f3f03d8e9675b889030%2FSelection_085.png?generation=1582196108850096&alt=media\")
![\"\"](\"http://yann.lecun.com/ex/images/invar.png\")
```\ndef train_augment(image, label, infor):\n original = image.copy()\n if 1:\n operation = [\n lambda image : do_random_erode(image),\n lambda image : do_random_dilate(image),\n lambda image : do_random_grid_distortion(image, distort=0.10, num_step = 5),\n lambda image : do_random_custom_distortion1(image, distort=0.10),\n lambda image : do_random_crop_rotate_rescale(image, mode={'rotate':10}),\n lambda image : do_random_block_fade(image, size=[0.1, 0.9], alpha=(0.1,0.5)),\n lambda image : do_random_line(image),\n ]\n num_op = np.random.choice(2)\n if num_op>0:\n for op in np.random.choice(operation,num_op):\n image = op(image)\n image = do_random_pad_crop(image, 3)
\n\n#----------------------------------------------------\nif 1:\n operation = [\n lambda image : image,\n lambda image : do_random_salt_pepper(image),\n lambda image : do_random_noise(image),\n ]\n for op in np.random.choice(operation,1):\n image = op(image)\n\n#----------------------------------------------------\nreturn original, image, label, infor\ndef train_batch_augment(original, input, onehot):\n operation = [\n lambda input, onehot : (input, onehot),\n lambda input, onehot : do_random_batch_cutout(input, onehot, fill=0),\n lambda input, onehot : do_random_batch_mixup(original, onehot),\n ]\n op = np.random.choice(operation)\n with torch.no_grad():\n input, onehot = op(input, onehot)
\n\nreturn input, onehot\n```
", "rawMarkdown": "```\ndef train_augment(image, label, infor):\n original = image.copy()\n if 1:\n operation = [\n lambda image : do_random_erode(image),\n lambda image : do_random_dilate(image),\n lambda image : do_random_grid_distortion(image, distort=0.10, num_step = 5),\n lambda image : do_random_custom_distortion1(image, distort=0.10),\n lambda image : do_random_crop_rotate_rescale(image, mode={'rotate':10}),\n lambda image : do_random_block_fade(image, size=[0.1, 0.9], alpha=(0.1,0.5)),\n lambda image : do_random_line(image),\n ]\n num_op = np.random.choice(2)\n if num_op>0:\n for op in np.random.choice(operation,num_op):\n image = op(image)\n image = do_random_pad_crop(image, 3)\n\n #----------------------------------------------------\n if 1:\n operation = [\n lambda image : image,\n lambda image : do_random_salt_pepper(image),\n lambda image : do_random_noise(image),\n ]\n for op in np.random.choice(operation,1):\n image = op(image)\n\n #----------------------------------------------------\n return original, image, label, infor\n\n\n\ndef train_batch_augment(original, input, onehot):\n operation = [\n lambda input, onehot : (input, onehot),\n lambda input, onehot : do_random_batch_cutout(input, onehot, fill=0),\n lambda input, onehot : do_random_batch_mixup(original, onehot),\n ]\n op = np.random.choice(operation)\n with torch.no_grad():\n input, onehot = op(input, onehot)\n\n return input, onehot\n\n```", "votes": 1 }, { "id": 761929, "postDate": "2020-03-03T03:36:04.637Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 737055, "postDate": "2020-02-04T21:26:15.080Z", "content": "This discussion forum is a gold-mine of ideas. Thanks a lot @hengck23 for initiating this and for sharing such wonderful ideas.
\n\nLots to do, lots of learn, not enough time.
", "rawMarkdown": "This discussion forum is a gold-mine of ideas. Thanks a lot @hengck23 for initiating this and for sharing such wonderful ideas. \n\nLots to do, lots of learn, not enough time.", "votes": 1 }, { "id": 733502, "postDate": "2020-01-31T08:15:44.307Z", "content": "don't forget that each of has has addition 36 unlabelled training images at the test parquet files
", "rawMarkdown": "don't forget that each of has has addition 36 unlabelled training images at the test parquet files", "votes": 1 }, { "id": 733430, "postDate": "2020-01-31T05:30:36.087Z", "content": "some tricks from ICDAR compeitions
\n\n", "rawMarkdown": "some tricks from ICDAR compeitions\n\nhttps://gateway.newton.ac.uk/sites/default/files/asset/doc/1811/20181127-Large%20Scale%20and%20Unconstrained%20OHCCR%20Based%20on%20Deep%20Learning%20and%20Path%20Signature.pdf", "votes": 1, "replies": [ { "id": 733436, "postDate": "2020-01-31T05:52:45.607Z", "content": "handwriting trajectory recovery\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F1001950b2fa0efdac57746685d495781%2FSelection_066.png?generation=1580484022666591&alt=media\")
https://arxiv.org/pdf/1606.05763.pdf
", "rawMarkdown": "\n\nhttps://arxiv.org/pdf/1606.05763.pdf" }, { "id": 733850, "postDate": "2020-01-31T15:24:05.603Z", "content": "dropsample --- remove outlier\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F631f809daaa2cdbbbe2938d1efd1b8ed%2FSelection_067.png?generation=1580484242967773&alt=media\")
drop-distortion\nhttps://arxiv.org/pdf/1702.07508.pdf
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fdbe0c609d2f6e881203780e458191c35%2FSelection_068.png?generation=1580484634022137&alt=media\")
artificial handwriting\nhttps://distill.pub/2016/handwriting/\nhttp://otoro.net/ml/\n![\"\"](\"http://blog.otoro.net/wp-content/uploads/sites/2/2015/12/cover_simple_full.svg\")
https://genekogan.com/works/a-book-from-the-sky/
\n\n![\"\"](\"https://genekogan.com/images/a-book-from-the-sky/radinterpolations/rad30is.gif\")
\n![\"\"](\"https://cdn.rawgit.com/hardmaru/resnet-cppn-gan-tensorflow/master/examples/example_sinusoid.gif\")
https://github.com/osmr/imgclsmob\nmega list of pretrain models
", "rawMarkdown": "https://github.com/osmr/imgclsmob\nmega list of pretrain models", "votes": 1 }, { "id": 722316, "postDate": "2020-01-18T11:44:21.563Z", "content": "something interesting ...\nhttps://github.com/Natsu6767/Generating-Devanagari-Using-DRAW\n![\"\"](\"https://github.com/Natsu6767/Generating-Devanagari-Using-DRAW/raw/master/images/devanagari_generate.gif\")
Is it like GAN with RNN ? I saw an encoder and decoder structure .This is extremely interesting .
", "rawMarkdown": "Is it like GAN with RNN ? I saw an encoder and decoder structure .This is extremely interesting .", "votes": 1 }, { "id": 724852, "postDate": "2020-01-21T14:55:07.983Z", "content": "font RNN\nhttps://xiazeqing.github.io/FontRNN/
", "rawMarkdown": "font RNN\nhttps://xiazeqing.github.io/FontRNN/" } ] }, { "id": 721139, "postDate": "2020-01-17T05:04:47.173Z", "content": "one weird idea to fight overfitting:\n```\n1. given N class train images\n2. flip the images to make 2 N class (if can do transpose, horizontal, vertical flip to give more class)\n3. pretrain a deep network with 2N class
\n\neither \n 4. finetune to original N class
\n\nor\n4. continue to 2N class and apply test-time augment (TTA) for test + flip images
\n\n```
\n\nsince the problem is now more complex, maybe the network is less likely to overfit
", "rawMarkdown": "one weird idea to fight overfitting:\n```\n1. given N class train images\n2. flip the images to make 2 N class (if can do transpose, horizontal, vertical flip to give more class)\n3. pretrain a deep network with 2N class\n\neither \n 4. finetune to original N class\n\nor\n4. continue to 2N class and apply test-time augment (TTA) for test + flip images \n\n```\n\nsince the problem is now more complex, maybe the network is less likely to overfit", "votes": 1 }, { "id": 706961, "postDate": "2019-12-31T04:48:04.067Z", "content": "In your code you load the dataset based on split=balance2/train_b_fold0_184855.npy. Could you clarify on this? Does it mean that the split fold contains enough samples from each class?
@bibek777 @hengck23 I am curious how this split is made too. In the previous competition steel and this competition, the split assignment has always been a puzzle. I think it does more than a multi-label stratification.
", "rawMarkdown": "@bibek777 @hengck23 I am curious how this split is made too. In the previous competition steel and this competition, the split assignment has always been a puzzle. I think it does more than a multi-label stratification. " }, { "id": 708607, "postDate": "2020-01-02T13:25:20.540Z", "content": "A random split is what i am using now
", "rawMarkdown": "A random split is what i am using now", "votes": 1 } ] }, { "id": 760537, "postDate": "2020-03-01T12:41:06.157Z", "content": "CamDrop: A New Explanation of Dropout and A Guided Regularization Method for Deep Neural Networks
\n\n![\"\"](\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/9f13cd4841fb050b345f3a56398871232ea5a58c/3-Figure1-1.png\")
\n![\"\"](\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/9f13cd4841fb050b345f3a56398871232ea5a58c/7-Table6-1.png\")
imageBERT\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F679ae3e3b92efdb848b8a7ef3c4bc81c%2FSelection_067.png?generation=1581094696419061&alt=media\")
to my surprise, the error are very subtle
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F910127fdfeb8365828a06159457427df%2FSelection_057.png?generation=1580196437091996&alt=media\")
error analysis\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fcc5686df09164b9bded1f56fa081a095%2FSelection_124.png?generation=1579969484208260&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F099b1bde79742f98fa45cba0c951d807%2FSelection_085.png?generation=1579599439688221&alt=media\")
https://openreview.net/forum?id=rJMw747l_4\n\"To that end, we train ResNet-50 classifiers using either purely BigGAN images or mixtures of ImageNet and BigGAN images, and test on the ImageNet validation set.Our preliminary results suggest both a measured view of state-of-the-art GAN quality and highlight limitations of current metrics. Using only BigGAN images, we find that Top-1 and Top-5 error increased by 120% and 384%, respectively, and furthermore, adding more BigGAN data to the ImageNet training set at best only marginally improves classifier performance.\"
", "rawMarkdown": "https://openreview.net/forum?id=rJMw747l_4\n\"To that end, we train ResNet-50 classifiers using either purely BigGAN images or mixtures of ImageNet and BigGAN images, and test on the ImageNet validation set.Our preliminary results suggest both a measured view of state-of-the-art GAN quality and highlight limitations of current metrics. Using only BigGAN images, we find that Top-1 and Top-5 error increased by 120% and 384%, respectively, and furthermore, adding more BigGAN data to the ImageNet training set at best only marginally improves classifier performance.\"", "votes": 2 } ] }, { "id": 719277, "postDate": "2020-01-15T10:35:04.510Z", "content": "just an idea\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2Fbcaa1cc21e04038d9e9dafec16ae41e8%2FSelection_052.png?generation=1579084501734131&alt=media\")
You are onto something. As an example আ should be classified as grapheme root: 3 with vowel diacritic: 0 and consonant diacritic: 0 whereas তা should be classified as grapheme root: 64 and vowel diacritic: 1. Though both have া, one has it as a part of the root and other don't. Similarly মু, ম, both are same root but the former one has a vowel diacritic. Though the character itself has a shape ু, it is important that the network can distinguish which is a part of root and which isn't.
", "rawMarkdown": "You are onto something. As an example আ should be classified as grapheme root: 3 with vowel diacritic: 0 and consonant diacritic: 0 whereas তা should be classified as grapheme root: 64 and vowel diacritic: 1. Though both have া, one has it as a part of the root and other don't. Similarly মু, ম, both are same root but the former one has a vowel diacritic. Though the character itself has a shape ু, it is important that the network can distinguish which is a part of root and which isn't.", "votes": 4 } ] }, { "id": 707041, "postDate": "2019-12-31T07:53:57.780Z", "content": "Extremely happy to see you in the competition . Considering ,I have joined this competition to try out some basics and some new stuff . Your sharing will surely uplift the purpose of this competition and my own . One question though LSTM would normally help if you have a word written to get something contextual . The few paper I read it is used for that purpose . There could be some that I missed . Here the context that it can derive is if some grapheme combination does not occur in real life and if we can fine-tune the final result based on that ?
", "rawMarkdown": "Extremely happy to see you in the competition . Considering ,I have joined this competition to try out some basics and some new stuff . Your sharing will surely uplift the purpose of this competition and my own . One question though LSTM would normally help if you have a word written to get something contextual . The few paper I read it is used for that purpose . There could be some that I missed . Here the context that it can derive is if some grapheme combination does not occur in real life and if we can fine-tune the final result based on that ? ", "votes": 2, "replies": [ { "id": 707078, "postDate": "2019-12-31T08:29:54.473Z", "content": "since you are familiar with the language, human-in-the loop attention method may be suitable for you.\nyou can further create your own training samples (hard sample mining) as you mentioned in https://www.kaggle.com/c/bengaliai-cv19/discussion/123002
\n\n![\"\"](\"https://d3i71xaburhd42.cloudfront.net/5393ccde31e38b1f278bc70af0c5b0a9cc758b96/3-Figure3-1.png\")
Embedding Human Knowledge in Deep Neural Network via Attention Map\nhttps://arxiv.org/abs/1905.03540
\n\nhttps://github.com/machine-perception-robotics-group/attention_branch_network
", "rawMarkdown": "since you are familiar with the language, human-in-the loop attention method may be suitable for you.\nyou can further create your own training samples (hard sample mining) as you mentioned in https://www.kaggle.com/c/bengaliai-cv19/discussion/123002\n\n\n\nEmbedding Human Knowledge in Deep Neural Network via Attention Map\nhttps://arxiv.org/abs/1905.03540\n\nhttps://github.com/machine-perception-robotics-group/attention_branch_network", "votes": 10 }, { "id": 707248, "postDate": "2019-12-31T13:29:34.917Z", "content": "![\"\"](\"https://ars.els-cdn.com/content/image/1-s2.0-S0167865519302181-gr2.jpg\")
How much did the swa help? any idea?
", "rawMarkdown": "How much did the swa help? any idea?", "votes": 2, "replies": [ { "id": 717410, "postDate": "2020-01-13T04:46:23.400Z", "content": "My score has been gone up 0.05
", "rawMarkdown": "My score has been gone up 0.05" }, { "id": 718243, "postDate": "2020-01-14T08:04:35.903Z", "content": "you mean 0.05 or 0.005? like 5 percent? that's a lot my god
", "rawMarkdown": "you mean 0.05 or 0.005? like 5 percent? that's a lot my god" } ] }, { "id": 706924, "postDate": "2019-12-31T02:31:43.057Z", "content": "Frog brother, See you again! Thx for your generous share; hope you get better rank
", "rawMarkdown": "Frog brother, See you again! Thx for your generous share; hope you get better rank", "votes": 1 }, { "id": 775962, "postDate": "2020-03-17T03:43:12.780Z", "content": "Thanks for sharing the ideas! Helped a lot and inspired during the competition
", "rawMarkdown": "Thanks for sharing the ideas! Helped a lot and inspired during the competition " }, { "id": 770235, "postDate": "2020-03-12T18:04:17.307Z", "content": "getting the handwriting strokes
\n\nhttps://www.youtube.com/watch?v=47CXR42_b2Y
", "rawMarkdown": "getting the handwriting strokes\n\nhttps://www.youtube.com/watch?v=47CXR42_b2Y" }, { "id": 770233, "postDate": "2020-03-12T18:02:41.877Z", "content": "just wonder if anyone uses Tesseract?
", "rawMarkdown": "just wonder if anyone uses Tesseract?" }, { "id": 770227, "postDate": "2020-03-12T17:51:34.303Z", "content": "this is a common trick to train image of larger size (or deep network) with limited GPU resource:
\n\n1) how to train large size:\n1. train image e.g. 360x360\n2. design fully convolutional net\n3. in training crop to smaller size, e.g. 224x224\n4. in final stage, fintunning at full image size (i.e. batch get smaller or freeze bottom layers)
\n\n2) how to train very deep network\nthis is how vgg16 and googlenett did it when the gpu are small in the early days\n- train few layers first. then freeze these layers. add new trainable layers.\n (or, implement deep network but alternate between freezing upper and bottom layers at training)
", "rawMarkdown": "this is a common trick to train image of larger size (or deep network) with limited GPU resource:\n\n1) how to train large size:\n1. train image e.g. 360x360\n2. design fully convolutional net\n3. in training crop to smaller size, e.g. 224x224\n4. in final stage, fintunning at full image size (i.e. batch get smaller or freeze bottom layers)\n\n\n2) how to train very deep network\nthis is how vgg16 and googlenett did it when the gpu are small in the early days\n- train few layers first. then freeze these layers. add new trainable layers.\n (or, implement deep network but alternate between freezing upper and bottom layers at training)\n\n\n" }, { "id": 737229, "postDate": "2020-02-05T04:23:30.307Z", "content": "encoder based augmentation?
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F0cc99af813bda0e3f1ac4680dd8618ec%2FClipboard09.png?generation=1580876607555657&alt=media\")
fine-grained classification\nhttps://arxiv.org/pdf/1903.06150v2.pdf
\n\nLooking for the Devil in the Details: Learning Trilinear Attention Sampling\nNetwork for Fine-grained Image Recognition
\n\n![\"\"](\"https://paperswithcode.com/media/thumbnails/task/task-0000000719-ccaa52fd.jpg\")
Thanks for sharing this.
\n\nYou use a file \"grapheme_1295.csv\" for creating the balanced sampler. I have not seen any indication of what the content is or how the content is created. Could you please give some explanations?
", "rawMarkdown": "Thanks for sharing this.\n\nYou use a file \"grapheme_1295.csv\" for creating the balanced sampler. I have not seen any indication of what the content is or how the content is created. Could you please give some explanations?", "replies": [ { "id": 718375, "postDate": "2020-01-14T10:38:05.637Z", "content": "You can do something like this -pd.DataFrame({'grapheme' :df['grapheme'].unique()}).to_csv('../grapheme_1295.csv') df - train.csv
sample_submission.csv : just map grapheme symbol to numerical label, e.g.
\n\n```\nlabel,grapheme\n0,ং\n1,ঃ\n2,অ\n3,অ্যা\n4,আ\n5,আঁ\n6,ই\n7,ইঁ\n8,ঈ\n9,উ\n10,উঁ\n11,ঊ
\n\n```
", "rawMarkdown": "sample_submission.csv : just map grapheme symbol to numerical label, e.g.\n\n```\nlabel,grapheme\n0,ং\n1,ঃ\n2,অ\n3,অ্যা\n4,আ\n5,আঁ\n6,ই\n7,ইঁ\n8,ঈ\n9,উ\n10,উঁ\n11,ঊ\n\n```", "votes": 1 }, { "id": 719045, "postDate": "2020-01-15T04:07:56.087Z", "content": "I have to thank you again, Heng. Your sharing always inspires me. @hengck23
", "rawMarkdown": "I have to thank you again, Heng. Your sharing always inspires me. @hengck23 " } ] }, { "id": 717306, "postDate": "2020-01-13T02:15:26.040Z", "content": "@hengck23 Thanks buddy and best of luck!!!
", "rawMarkdown": "@hengck23 Thanks buddy and best of luck!!!" }, { "id": 714390, "postDate": "2020-01-09T11:48:55.480Z", "content": "Thanks for sharing, I will try to follow your step in this competition. Learned a lot from you
", "rawMarkdown": "Thanks for sharing, I will try to follow your step in this competition. Learned a lot from you " }, { "id": 706664, "postDate": "2019-12-30T17:15:39.820Z", "content": "I will stay tuned on your post.
", "rawMarkdown": "I will stay tuned on your post." }, { "id": 761708, "postDate": "2020-03-02T21:19:23.920Z", "rawMarkdown": "", "isDeleted": true }, { "id": 759337, "postDate": "2020-02-28T22:49:45.423Z", "rawMarkdown": "", "isDeleted": true }, { "id": 722257, "postDate": "2020-01-18T10:21:43.973Z", "rawMarkdown": "", "isDeleted": true }, { "id": 706310, "postDate": "2019-12-30T08:30:20.873Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 771046, "author_name": "hengck23", "author_url": "", "post_date": "2020-03-13T17:42:27.323000", "content": "this is one of the kaggle competitions that i did the most number of experiments. I notice something interesting.
\n\nfor several experiments, it is important to apply the method at the early start of training. e.g. you get different results if you apply ohem on trained models (i.e. end stage of training) versus you apply it right at the start of training.
\n\ni think that when data size is small, the number of good solutions (or generalized solutions) region is small. i.e. there are many local minimum. It is difficult to jump from one solution to another solution
", "votes": 8, "replies": [ { "id": 771347, "author_name": "MadCoder", "author_url": "", "post_date": "2020-03-14T04:07:33.507000", "content": "I do find out applying ohem at the end stage of training will save you from overfitting. It will shake a little bit at the beginning but it will increase your local cv at the end. It seems like applying ohem make it jumping out of its local minimum.
", "votes": 3, "replies": [] } ] }, { "id": 756402, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-25T17:37:08.607000", "content": "this is a reference version for those who are struggling:
\n\nversion 20200224:\n- modified se-resnext50 (64x112 small input) with augmentation that works. please see readme file in folder. \n This version can gives CV 0.991 and LB 0.980 after you optimized the augmentation hyper parameters yourselves.
\n\nit provides reference log file for you to compare the loss curve
it has train/validation split that has gap of CV/LB 0.011
code is not complete, but should have enough details to reproduce the above mentioned results.\n(do not request me for missing files or functions)
Thanks for sharing and for the best thread in the discussion!
", "votes": 0, "replies": [] }, { "id": 756562, "author_name": "YL", "author_url": "", "post_date": "2020-02-25T20:56:28.917000", "content": "Hi Heng Thanks for the update. May I ask if the 64x112 is a simple downsize from the original 137x236 image or did you do some post processing?
\n\nAnd may I ask how long have you trained the model to reach cv ~.99. From that log file it seems it's been training for 500 epochs?!
", "votes": 0, "replies": [] }, { "id": 756712, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-26T01:43:17.457000", "content": "please refer to the readme file
", "votes": 2, "replies": [] }, { "id": 757358, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-26T17:09:16.117000", "content": "how to train a modified resenext model with pretrained imagenet model
\n\nuse the unmodified version resenext model first. initialised with pretrained imagenet model and train as usual.
assume resenext = block0, block1 ... block4. we modify block0.
load the previously trained model except for the the modified part (i.e block0).
freeze all modified part (including the classifier head). Train and the gradient will only back-propagated at the modified block0.
when the accuracy/loss for the modified model is about the same as the previously unmodified one, say 90% of the results, unfreeze all model and proceed training as usual.
this is how i do network surgery. apply this when there is structural change of the model, e.g. add new convolution layers, reduce number of channels, change kernel size, ...
\n\nif the old pretrained imagenet model can be loaded after modification (e.g. simple modification like changing strike), there is no need to freeze and unfreeze. In such minor modification, just train as usual.
", "votes": 3, "replies": [] }, { "id": 758076, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-27T12:16:27.667000", "content": "purely trained from scratch \"without surgery above\".\nno augmentation is used at all!
\n\nlocal cv = 0.972 (at rate 0.05)\nlocal cv = 0.972 (at rate 0.005)
\n\ntraining loss is almost zero, so there is no point to train further. expected LB should be around 0.960 (without augmentation)
\n\nsee attached file.\n```\ndef valid_augment(image, label, infor):\n image = to_64x112(image)\n return image, label, infor
\n\ndef train_augment(image, label, infor):\n original = image.copy()\n original = to_64x112(original)\n image = to_64x112(image)\n return original, image, label, infor
\n\ndef train_batch_augment(original, input, onehot):\n return input, onehot
\n\n0.05000 9.0 15.6 | 0.972 : 0.957 0.989 0.982 0.955 | 0.20, 0.06, 0.06, 0.210 : 0.96, 0.99, 0.99, 0.956 | 0.00, 0.00, 0.00, 0.009 | 1 hr 38 min\n0.05000 10.0 17.3 | 0.972 : 0.957 0.990 0.986 0.955 | 0.21, 0.07, 0.07, 0.217 : 0.96, 0.99, 0.99, 0.956 | 0.00, 0.00, 0.00, 0.007 | 1 hr 49 min\n0.05000 11.0 19.0 | 0.971 : 0.957 0.989 0.982 0.955 | 0.21, 0.06, 0.07, 0.214 : 0.96, 0.99, 0.99, 0.956 | 0.01, 0.00, 0.00, 0.008 | 2 hr 00 min\n0.05000 12.0* 20.8 | 0.970 : 0.955 0.990 0.982 0.953 | 0.22, 0.06, 0.07, 0.224 : 0.96, 0.99, 0.99, 0.954 | 0.00, 0.00, 0.00, 0.007 | 2 hr 11 min\n```\nnote: some kaggler reported about to get LB 0.974 without augmentation (which i think 0.984 for local CV). hence i need to investigate better model or regularization (like dropout, shakedrop, etc) or better optimiser
", "votes": 2, "replies": [] }, { "id": 758342, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-27T16:49:19.633000", "content": "additional information:
\n\nmodified se-resnext50 (32x56 mini input) with same augmentation that works gives CV 0.987\nthis may be useful for auto-augmentation parameters search?
", "votes": 4, "replies": [] }, { "id": 759575, "author_name": "thefatcat", "author_url": "", "post_date": "2020-02-29T07:58:51.023000", "content": "Thanks for your sharing,but I have a question.\nIn your train_rand_aug2c.py,there is a func named INVERSE_COMPOSE,but you didn't assign it before using，I guess it is used to add information for first three predict form firth .\nCould you explain its meaning in detail? Thanks！
", "votes": 0, "replies": [] }, { "id": 759588, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-29T08:24:02.713000", "content": "we know that each of the 1295 grapheme is made up of grapheme=root, vowel, consonant
\n\nso if you have predicted the 1295 class, simply decode (root, vowel, consonant) = grapheme
", "votes": 0, "replies": [] }, { "id": 759960, "author_name": "thefatcat", "author_url": "", "post_date": "2020-02-29T16:56:38.817000", "content": "Oh,yes.I understand your idea.
\n\nBut I got another question when I use your code to train the model .Most of the time my GPU utilization ratio is 0 and I thought the reason for this is dataloder's speed limitation.When we load the data by dataloader,we do some augmentation,and before send data to the net,maybe do some mixup or cutout.These process are not handled by GPU but CPU.
\n\nThis problem made my training process very slow,but in your log.train,it shows the speed is not so slow,so I want to ask whether you use some tricks to speed up.
", "votes": 0, "replies": [] }, { "id": 759969, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-29T17:13:32.273000", "content": "df['xxx'].values should be done once at initialisation and not everytime at getitem()
", "votes": 0, "replies": [] }, { "id": 759981, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-29T17:27:09.323000", "content": "```
\n\nclass KaggleDataset(Dataset):\n def init(self, split, mode, csv, parquet, augment=None):\n global TRAIN_PARQUET\n ...\n self.df_values = df.values
\n\ndef __getitem__(self, index): \n i, image_id, grapheme_root, vowel_diacritic, consonant_diacritic, grapheme, grapheme_symbol = self.df_values[index]\n```
", "votes": 0, "replies": [] }, { "id": 760463, "author_name": "thefatcat", "author_url": "", "post_date": "2020-03-01T10:33:43.803000", "content": "Thank you,I will have a try.
", "votes": 0, "replies": [] }, { "id": 764147, "author_name": "PY", "author_url": "", "post_date": "2020-03-05T07:19:47", "content": "Thanks for sharing, very helpful! Quick question, when I'm using your model ResNext50, it seems that the model takes more than 11GB of GPU memory. Did you encounter similar issue?
Or does this model only work when you freeze part of the model and only train its head?
\n\nThanks.
", "votes": 0, "replies": [] }, { "id": 765755, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-03-07T03:43:19.483000", "content": "I am still trying to figure out how to write inverse_compose function. Any idea ?
", "votes": 0, "replies": [] }, { "id": 766504, "author_name": "hengck23", "author_url": "", "post_date": "2020-03-08T09:09:05.520000", "content": "\n\n```\ndef run_prepare_data4():\n df_grapheme = pd.read_csv(DATA_DIR+'/grapheme_1295.csv')\n df_root = pd.read_csv(DATA_DIR+'/root_168.csv')\n df_vowel = pd.read_csv(DATA_DIR+'/vowel_11.csv')\n df_consonant = pd.read_csv(DATA_DIR+'/consonant_7.csv')
\n\ngrapheme_map = dict(df_grapheme[['grapheme','label']].values)\nroot_map = dict(df_root[['root','label']].values)\nvowel_map = dict(df_vowel[['vowel','label']].values)\nconsonant_map = dict(df_consonant[['consonant','label']].values)\n\ngrapheme_inverse_map = dict(df_grapheme[['label','grapheme']].values)\nroot_inverse_map = dict(df_root[['label','root']].values)\nvowel_inverse_map = dict(df_vowel[['label','vowel']].values)\nconsonant_inverse_map = dict(df_consonant[['label','consonant',]].values)\n\n#----\ndf_train = pd.read_csv(DATA_DIR+'/train.csv')\ndf_train['grapheme']=df_train['grapheme'].map(grapheme_map)\nassert(df_train['grapheme'].isnull().values.any() == False)\n\ncompose = {}\ninverse_compose = {}\ngb = df_train.groupby(['grapheme',])\nfor i in range(1295):\n gp = gb.get_group(i) #.index\n\n gp = gp.drop_duplicates(subset=['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'])\n assert(len(gp)==1)\n\n g = i\n r,v,c = gp[['grapheme_root', 'vowel_diacritic', 'consonant_diacritic']].values[0]\n\n inverse_compose[g]= (r,v,c)\n compose[grapheme_inverse_map[g]]= (\n root_inverse_map[r],\n vowel_inverse_map[v],\n consonant_inverse_map[c]\n )\n\nprint(inverse_compose)\nprint(compose)\n```
", "votes": 1, "replies": [] }, { "id": 766512, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-03-08T09:21:13.480000", "content": "Thanks so much Heng :)
", "votes": 0, "replies": [] } ] }, { "id": 737223, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-05T04:15:29.610000", "content": "i sometime have discussions with friends on their phd research work. recently, they give me a few pointers on network design:
\n\nit is possible that a single batch norm cannot normalized the feature well for multi-task. different task needs different normalization. Care needs to be taken at designing the head
you can force the network to learn high frequency features (e.g. edge, etc) over texture by adding loss related to high frequency feature extraction , together with the usual cross entropy classification loss.
related:\nusing Adversarial Examples to expand dataset
\n\n\"With an enhanced EfficientNet-B8, our method achieves the state-of-the-art 85.5% ImageNet\ntop-1 accuracy without extra data. This result even surpasses the best model in [20] which is trained with 3.5B Instagram images (∼3000× more than ImageNet) and ∼9.4× more parameters.\"
\n\nAdversarial Examples Improve Image Recognition\nhttps://arxiv.org/abs/1911.09665\nhttps://www.youtube.com/watch?v=KTCztkNJm50
\n\n
see also:\nCompounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network https://arxiv.org/pdf/2001.06268.pdf
", "votes": 1, "replies": [] } ] }, { "id": 723886, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-20T15:16:12.777000", "content": "magic net
\n\n
single model, single fold performance:\n```\nLB = 0.9755\nCV = 0.983629\n grapheme_root 0.977137\n vowel_diacritic 0.993395\n consonant_diacritic 0.986848
\n\nmodel: SE-Resnext50 with stride replaced with max pooling + SWA\ninput: 137x236\naugmentation: mixup + cut-fade
\n\n```
\n\nafter more tunning:
\n\n```\nLB = 0.9760\nCV = 0.984155\n grapheme_root 0.978920\n vowel_diacritic 0.992851\n consonant_diacritic 0.985929
\n\n```
", "votes": 0, "replies": [] }, { "id": 726108, "author_name": "Mario Parreño Lara", "author_url": "", "post_date": "2020-01-22T20:27:47.137000", "content": "Hi Heng! I am trying to automate the 'magic' net architecture. First I replaced de stride 2 to 1, saving the conv 'path':
\n\n\nmodules = {}\nfor name, module in model.named_modules():\n if(isinstance(module, nn.Conv2d)):\n stride = module.stride\n if stride == (2, 2) or stride == 2:\n module.stride = (1,1)\n modules[name] = module\n elif stride == 2:\n module.stride = 1\n modules[name] = module\n
Next I try to insert the maxpool as follows:
\n\n```\nfor name in modules:\n parent_module = model\n objs = name.split(\".\")\n if len(objs) == 1:\n #model.setattr(name, modules[name])\n model.setattr(\"magicMaxPool\", nn.MaxPool2d(kernel_size=2, stride=2))\n continue
\n\nfor obj in objs[:-1]:\n parent_module = parent_module.__getattr__(obj)\n\nparent_module.__setattr__(\"magicMaxPool\", nn.MaxPool2d(kernel_size=2, stride=2))\n```
\n\nIf I inspect the net/modules, it appears the maxpool but when I do the inference stride works properly but maxpool is not used (vector size before global avg pool is too big).
", "votes": 1, "replies": [] }, { "id": 729048, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-25T16:46:53.873000", "content": "related: http://openaccess.thecvf.com/content_cvpr_2017/papers/Zhai_S3Pool_Pooling_With_CVPR_2017_paper.pdf
\n\nS3Pool: Pooling with Stochastic Spatial Sampling
", "votes": 2, "replies": [] }, { "id": 735731, "author_name": "makogarei", "author_url": "", "post_date": "2020-02-03T11:53:18.557000", "content": "Hi @hengck23 \nI always look at the code you write and I am very grateful.\nI try to training Resnext50 with magic net,But cv value did not go above 0.96. Did you do anything other than change the layer?
", "votes": 0, "replies": [] } ] }, { "id": 707588, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-01T05:35:10.807000", "content": "after a series of experiments, how are my conclusion:
\n\nbaseline results:\n a. 0.9675~0.9715\n - standard method (joint 3 class) and ensemble standard model (e.g. densenet, resnet, efficientnet, etc) \n - to maximize performance, search for best input size (e.g. original size, enlarged size like 224x224,256x256, or even larger), try better augmentation, train longer with better learning rate (swa, cyclic annealing, ...), etc\n - size, scale normalization
\n\nb. 0.9700~0.9800\n... to be updated ... (better formulation, loss etc?)
\n\nc. 0.9800~0.9900\n... to be updated ... (some way to create more data?)
\n\ndon't forget Google Cloud AutoML Vision as part of solution !!!!
", "votes": 8, "replies": [ { "id": 708848, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-01-02T19:00:30.557000", "content": "Thanks for the pointers @hengck23 . I started playing with augmentations . I have got the cutout working here .
\n\nhttps://www.kaggle.com/phoenix9032/pytorch-efficientnet-starter-code
\n\nNow , need to see how MixUp and RICAP and other thing can work out along with the training for multi-heads . Need to make some modification from the original implementation.
\n\nWe can try dual cutout as well.
", "votes": 3, "replies": [] }, { "id": 719435, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-15T14:01:09.920000", "content": "some important (?) experimental results
\n\nsampling affects results (e.g. balance sampling of \"root\" or \"grapheme\" affects \"root\", \"constant\", \"vowel\" differently)
augmentation affects results (e.g. some drop in accuracy when scaling and rotation is used)
arcface and large-margin like softamx does improve results but cannot be used with mixup?
Hi Heng @hengck23 (and other Kagglers), I see in your latest starting kit that you added a fourth target of the 1295 unique classes of Bengali letters as provided in the training dataset. If the testing dataset (which is not accessible to us) has letters other than those 1295 classes, would the training approach be prone to overfitting? Please correct me if I am wrong.
", "votes": 0, "replies": [] }, { "id": 721132, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-17T04:53:19.677000", "content": "you can use low weight like 0.1 for the fourth class, you can also use 1295+1 (background class for none of the above). to create background class, selectively flip the original train samples
", "votes": 0, "replies": [] } ] }, { "id": 758757, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-28T05:38:01.697000", "content": "good for augmentation?
\n\n
https://github.com/WarBean/tps_stn_pytorch
", "votes": 5, "replies": [ { "id": 759014, "author_name": "Soonhwan Kwon", "author_url": "", "post_date": "2020-02-28T13:02:09.523000", "content": "very interesting augmemtation!
", "votes": 0, "replies": [] } ] }, { "id": 724848, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-21T14:51:04.107000", "content": "in google doddle, you can add and subtract drawing. I wonder if this can be done in bengali grapheme? e.g. add/remove or replace vowel, etc\n
new augmentation
\n\n
related: augmentation on feature map:
\n\nthis attenuate feature values, preventing it from over dominating and lead to over fitting
", "votes": 0, "replies": [] } ] }, { "id": 775716, "author_name": "hengck23", "author_url": "", "post_date": "2020-03-17T00:23:49.243000", "content": "private lb of coded posted here:\n
Amazing results but sorry to see they are not selected...This post is really helpful and I have learned a lot from your code for the first competition I decided to take part in seriously. Thank you!\nBy the way, did you forget to select final submission?😂 Or just because of the shake up?
", "votes": 0, "replies": [] } ] }, { "id": 765573, "author_name": "hengck23", "author_url": "", "post_date": "2020-03-06T20:08:44.370000", "content": "there is one extra supervision signal.
\n\n\n list('ক্ট্রো')\n['ক', '্', 'ট', '্', 'র', 'ো']\n\na list command breaks the graheme into more component. you can use this the measure the amount of common parts (distance) between 2 graphemes
if this model is not correlated to image model, it is good for ensemble.\nthis work enables input of very large image
\n\nLearning in the Frequency Domain\nKai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-kuang Chen, Fengbo Ren
\n\n
```
\n\nResNet-50 #Channels Size Per Channel Top-1 Top-5 Normalized Input Size
\nRGB 3 224x224 75.780 92.650 1.0
\nDCT-24 (ours) 24 56x56 77.196 93.504 0.5
```
", "votes": 3, "replies": [ { "id": 763297, "author_name": "momi64", "author_url": "", "post_date": "2020-03-04T10:37:02.583000", "content": "Thanks for sharing an interesting paper.\nThe code is available below.
\n\nhttps://github.com/calmevtime1990/supp/tree/master/classification
", "votes": 1, "replies": [] } ] }, { "id": 728658, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-25T03:28:10.440000", "content": "top-1 and top-2 recall:
\n\n```\nlocal validation results
\n\navgerage recall (top-1): 0.982253\n grapheme_root 0.974616\n vowel_diacritic 0.992676\n consonant_diacritic 0.987103
\n\navgerage recall (top-2) : 0.995918\n grapheme_root 0.993451\n vowel_diacritic 0.998653\n consonant_diacritic 0.998116
\n\n```
\n\nif you make a mistake, the correct results is probably the top-2. if you can think of a good way to post-process, you can improve your score
", "votes": 3, "replies": [ { "id": 729149, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-01-25T20:10:12.520000", "content": "this screams for an ensemble of models ;)
", "votes": 0, "replies": [] } ] }, { "id": 757812, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-27T05:58:26.333000", "content": "New paper today\nOn Feature Normalization and Data Augmentation
\n\nhttps://arxiv.org/pdf/2002.11102.pdf
", "votes": 4, "replies": [ { "id": 757817, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-27T06:09:36.200000", "content": "arXiv:2002.11022 (cross-list from cs.LG) [pdf, other]\nBeyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks\nYehui Tang, Yunhe Wang, Yixing Xu, Boxin Shi, Chao Xu, Chunjing Xu, Chang Xu\nSubjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
", "votes": 1, "replies": [] } ] }, { "id": 706574, "author_name": "seefun", "author_url": "", "post_date": "2019-12-30T15:27:50.050000", "content": "What's the meaning of “without bn refinement”?
", "votes": 3, "replies": [ { "id": 706611, "author_name": "Uday Kamal", "author_url": "", "post_date": "2019-12-30T16:07:09.213000", "content": "I think Heng is talking about the recently proposed 'Full Normalization' technique, a better alternative (or you can say modification) to the conventional Batch Normalization. The idea is described in the following paper:\nhttps://arxiv.org/abs/1810.06177
", "votes": 0, "replies": [] }, { "id": 706618, "author_name": "hengck23", "author_url": "", "post_date": "2019-12-30T16:11:02.760000", "content": "https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/
\n\n```\nBATCH NORMALIZATION\nOne important detail to keep in mind is batch normalization. Batch normalization layers compute running statistics of activations during training. Note that the SWA averages of the weights are never used to make predictions during training, and so the batch normalization layers do not have the activation statistics computed after you reset the weights of your model with opt.swap_swa_sgd(). To compute the activation statistics you can just make a forward pass on your training data using the SWA model once the training is finished. In the SWA class we provide a helper function opt.bn_update(train_loader, model). It updates the activation statistics for every batch normalization layer in the model by making a forward pass on the train_loader data loader. You only need to call this function once in the end of training.
\n\n```
\n\nthe above step is not implemented
", "votes": 5, "replies": [] }, { "id": 706626, "author_name": "Uday Kamal", "author_url": "", "post_date": "2019-12-30T16:19:48.630000", "content": "@hengck23 Thanks for your clarification!
", "votes": 0, "replies": [] } ] }, { "id": 706492, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2019-12-30T13:15:02.323000", "content": "What's the meaning of swa ?
", "votes": 3, "replies": [ { "id": 706607, "author_name": "Uday Kamal", "author_url": "", "post_date": "2019-12-30T16:03:32.357000", "content": "Stochastic Weight Averaging, you can read this paper for more details:\nhttps://arxiv.org/abs/1803.05407
", "votes": 4, "replies": [] }, { "id": 707056, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2019-12-31T08:05:01.543000", "content": "thx
", "votes": 1, "replies": [] } ] }, { "id": 728668, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-25T04:01:19.887000", "content": "multi-task and task dependency
\n\n\n
Im studying the correlation among tasks. What's this paper? Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 726161, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-22T21:58:56.863000", "content": "loss formulation:
\n\nsome task (root, constant, or vowel) are more confidence than others.\nsimilarly, some classes are easier within a class.
\n\ne.g. given task 1 has class A,B,C,D,E and task 2 has class a,b,c and if we know that some class cannot co-exist, it may give better results if we apply the rules as follow:\n```\n1. naive method:\ntreat task 1 as 5 classes problem and task 2 as 3 classes. ignore any co-occurrence prior knowledge
\n\n```\nif you know the language, you may apply such rules
", "votes": 4, "replies": [ { "id": 726925, "author_name": "Dhananjay Raut", "author_url": "", "post_date": "2020-01-23T10:16:45.180000", "content": "or we could change the architecture of model's head such that at the time of predicting task 2 we feed the probabilities(softmax) for task 1 predicted by our model.
", "votes": 2, "replies": [] } ] }, { "id": 716656, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-12T04:17:58.407000", "content": "version 20200111:\nse-resnext50 + balanced sampler + mixup (please refer to readme.ppt at the google drive)
\n\n
@hengck23 thanks for your sharing again. Do you have the updated verison of compute_kaggle_metric in your kaggle.py?
```\ndef compute_kaggle_metric(probability, truth):
\n\ndef compute_recall(probability,truth):\n num_class = probability.shape[-1]\n y = probability.argmax(-1)\n t = truth\n correct = y==t\n\n recall = np.zeros(num_class)\n for c in range(num_class):\n e = correct[t==c]\n if len(e)>0:\n recall[c]=e.mean()\n return recall\n\ncomponet = []\nrecall = []\nfor p,t in zip(probability,truth):\n r = compute_recall(p,t)\n recall.append(r)\n componet.append(r.mean())\n\naverage = np.average(componet, weights=[2,1,1,0])\nreturn average, componet, recall\n```
", "votes": 2, "replies": [] } ] }, { "id": 706712, "author_name": "hengck23", "author_url": "", "post_date": "2019-12-30T18:06:51.830000", "content": "there are 168 grapheme_root, 11 vowel_diacritic, 7 consonant_diacritic and 1295 grapheme classes!\nyou can build a fourth classifier for 1295 grapheme classes and decode the component from it
\n\n\n
Find a way to balance per class accuracy. <- I thought this approach too and tried a bit although I don’t have convincing reason why it is helpful. Do you have it?
", "votes": 0, "replies": [] }, { "id": 717364, "author_name": "Yangfan", "author_url": "", "post_date": "2020-01-13T03:58:19.630000", "content": "", "votes": 0, "replies": [] }, { "id": 759126, "author_name": "thefatcat", "author_url": "", "post_date": "2020-02-28T16:12:54.400000", "content": "I was curious about how to build a fourth classifier for 1295 grapheme classes,is that just groupby grapheme and label it from 1 to 1295?\nHope you could clarify,thanks.
", "votes": 0, "replies": [] }, { "id": 759134, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-28T16:19:22.130000", "content": "\n\nyes, you are correct
", "votes": 0, "replies": [] } ] }, { "id": 706395, "author_name": "hengck23", "author_url": "", "post_date": "2019-12-30T10:31:29.073000", "content": "augmentation in the release 'version 20191230'
\n\n
@hengck23 Hi. Can u tell what augmentations u used ? Thanks
", "votes": 0, "replies": [] } ] }, { "id": 716020, "author_name": "Bibek", "author_url": "", "post_date": "2020-01-11T06:47:03.763000", "content": "I published a kernel based on this so that people can train it using kaggle GPUs
", "votes": 2, "replies": [] }, { "id": 760615, "author_name": "hengck23", "author_url": "", "post_date": "2020-03-01T14:25:04.707000", "content": "yet another paper:\nMaxUp: A Simple Way to Improve Generalization of Neural Network Training\n```
\n\nMethod Top-1 error Top-5 error\nVanilla (He et al., 2016a) 76.3 -\nDropout (Srivastava et al., 2014) 76.8 93.4\nDropPath (Larsson et al., 2017) 77.1 93.5\nManifold Mixup (Verma et al., 2019) 77.5 93.8\nAutoAugment (Cubuk et al., 2019a) 77.6 93.8\nMixup (Zhang et al., 2018) 77.9 93.9\nDropBlock (Ghiasi et al., 2018) 78.3 94.1\nCutMix (Yun et al., 2019) 78.6 94.0\nMaxUp+CutMix 78.9 94.2
\n\nTable 1: Summary of top1 and top5 accuracies on the validation set of ImageNet for ResNet-50.\n```
", "votes": 1, "replies": [] }, { "id": 758761, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-28T05:43:32.973000", "content": "it is possible to treat it as a detection problem\n
Didn't expect to see CRAFT here
", "votes": 0, "replies": [] }, { "id": 759082, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-28T14:49:24.743000", "content": "just an idea\n
https://github.com/sujatasaini/Kuzushiji-DropBlock
\n\n|Models | MNIST | Fashion-MNIST | Kuzushiji-MNIST | Kuzushiji-49 |\n|---------------------------------|-------|---------------|-----------------|--------------|\n|DCNN-DropBlock | 99.47% | 93.40% | 97.66% | 95.67% |\n|DCNN-Dropout | 97.99% | 85.47% | 86.43% | 95.34% |\n|DCNN-Spatial-Dropout | 97.17% | 84.44% | 81.08% | 58.18 |
", "votes": 1, "replies": [] }, { "id": 751629, "author_name": "hengck23", "author_url": "", "post_date": "2020-02-20T10:55:19.673000", "content": "another augmentation that works for me:
\n\n
```\ndef train_augment(image, label, infor):\n original = image.copy()\n if 1:\n operation = [\n lambda image : do_random_erode(image),\n lambda image : do_random_dilate(image),\n lambda image : do_random_grid_distortion(image, distort=0.10, num_step = 5),\n lambda image : do_random_custom_distortion1(image, distort=0.10),\n lambda image : do_random_crop_rotate_rescale(image, mode={'rotate':10}),\n lambda image : do_random_block_fade(image, size=[0.1, 0.9], alpha=(0.1,0.5)),\n lambda image : do_random_line(image),\n ]\n num_op = np.random.choice(2)\n if num_op>0:\n for op in np.random.choice(operation,num_op):\n image = op(image)\n image = do_random_pad_crop(image, 3)
\n\n#----------------------------------------------------\nif 1:\n operation = [\n lambda image : image,\n lambda image : do_random_salt_pepper(image),\n lambda image : do_random_noise(image),\n ]\n for op in np.random.choice(operation,1):\n image = op(image)\n\n#----------------------------------------------------\nreturn original, image, label, infor\ndef train_batch_augment(original, input, onehot):\n operation = [\n lambda input, onehot : (input, onehot),\n lambda input, onehot : do_random_batch_cutout(input, onehot, fill=0),\n lambda input, onehot : do_random_batch_mixup(original, onehot),\n ]\n op = np.random.choice(operation)\n with torch.no_grad():\n input, onehot = op(input, onehot)
\n\nreturn input, onehot\n```
", "votes": 1, "replies": [] }, { "id": 761929, "author_name": "", "author_url": "", "post_date": "2020-03-03T03:36:04.637000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 737055, "author_name": "Rohit Agarwal", "author_url": "", "post_date": "2020-02-04T21:26:15.080000", "content": "This discussion forum is a gold-mine of ideas. Thanks a lot @hengck23 for initiating this and for sharing such wonderful ideas.
\n\nLots to do, lots of learn, not enough time.
", "votes": 1, "replies": [] }, { "id": 733502, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-31T08:15:44.307000", "content": "don't forget that each of has has addition 36 unlabelled training images at the test parquet files
", "votes": 1, "replies": [] }, { "id": 733430, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-31T05:30:36.087000", "content": "some tricks from ICDAR compeitions
\n\n", "votes": 1, "replies": [ { "id": 733436, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-31T05:52:45.607000", "content": "handwriting trajectory recovery\n
https://arxiv.org/pdf/1606.05763.pdf
", "votes": 0, "replies": [] }, { "id": 733850, "author_name": "hengck23", "author_url": "", "post_date": "2020-01-31T15:24:05.603000", "content": "dropsample --- remove outlier\n
drop-distortion\nhttps://arxiv.org/pdf/1702.07508.pdf
\n\n
artificial handwriting\nhttps://distill.pub/2016/handwriting/\nhttp://otoro.net/ml/\n
https://genekogan.com/works/a-book-from-the-sky/
\n\n\n