I'm having difficulty understanding about the training flow of the training process\nI can only train on the kernel because I have no local gpu.
\n\nThe question is, how can I submit mean of 5 different models(different fold) that are trained separately? can someone very kind can explain to me about the flow?.. everyone is doing it except me..
\n\nthanks to Lopuhin I made 5 different fold but I can only train with one fold because of the limitation of training time in the kernel :(
\n\nany comment would help me\nthanks!
", "messages": [ { "id": "530499", "postDate": "05/13/2019 02:55:25", "content": "I'm having difficulty understanding about the training flow of the training process\nI can only train on the kernel because I have no local gpu.
\n\nThe question is, how can I submit mean of 5 different models(different fold) that are trained separately? can someone very kind can explain to me about the flow?.. everyone is doing it except me..
\n\nthanks to Lopuhin I made 5 different fold but I can only train with one fold because of the limitation of training time in the kernel :(
\n\nany comment would help me\nthanks!
", "rawMarkdown": "I'm having difficulty understanding about the training flow of the training process\nI can only train on the kernel because I have no local gpu.\n\nThe question is, how can I submit mean of 5 different models(different fold) that are trained separately? can someone very kind can explain to me about the flow?.. everyone is doing it except me..\n\nthanks to Lopuhin I made 5 different fold but I can only train with one fold because of the limitation of training time in the kernel :(\n\nany comment would help me\nthanks!", "votes": null }, { "id": "530518", "postDate": "05/13/2019 05:01:19", "content": "@yangsaewon what you can do is run the same model in 5 different kernels for N number of epochs & save their weights. Now you can create a new kernel which uses the weights of all 5 models to generate 5 predictions on the test set & take the mean of the predictions.
Another approach would be to generate predictions separately for all 5 kernels & save the output of model in .npy files which you can use in a new kernel to do a mean operation & decide threshold.
Hope this helps, let me know if you have any doubts.
", "rawMarkdown": "yangsaewon what you can do is run the same model in 5 different kernels for `N` number of epochs & save their weights. Now you can create a new kernel which uses the weights of all 5 models to generate 5 predictions on the test set & take the mean of the predictions.\n\nAnother approach would be to generate predictions separately for all 5 kernels & save the output of model in `.npy` files which you can use in a new kernel to do a mean operation & decide threshold.\n\nHope this helps, let me know if you have any doubts.", "votes": null }, { "id": "530519", "postDate": "05/13/2019 05:02:16", "content": "Have fun.
", "rawMarkdown": "1. Run the kernel for each fold. Make sure each run saves the best weights.\n2. Create a dataset with previously saved weights. \n3. Write a new kernel which does\n3.1 load weights from dataset\n3.2 inference and keep its outputs\n3.3 repeat 3.1-3.2 for each fold\n3.4 take average of all outputs \n3.5 make submission.csv\n\nHave fun.", "votes": null }, { "id": "530647", "postDate": "05/13/2019 11:07:32", "content": "Could you please clarify your idea about .npy files, what is saved in them?
\n\nIn case of test set is changed, how this kernel based on your second approach with ready predictions work?
", "rawMarkdown": "Could you please clarify your idea about .npy files, what is saved in them? \n\nIn case of test set is changed, how this kernel based on your second approach with ready predictions work?", "votes": null }, { "id": "530836", "postDate": "05/13/2019 19:47:01", "content": "@appian @axel81 I'm relatively new to ensembling and averaging folds. Could you explain what you mean by averaging the outputs of the folds. I tried the steps above with 5 folds. Do you average the pre-sigmoid outputs of the folds or the post sigmoid outputs of the folds?
\n\nI took the mean of all of the pre-sigmoid outputs from the folds, applied sigmoid and thresholding to the mean and found that the score was really terrible upon submission. I could be missing something here or making a trivial mistake. Some insight would really help :)
\n\nThanks!
", "rawMarkdown": "appian @axel81 I'm relatively new to ensembling and averaging folds. Could you explain what you mean by averaging the outputs of the folds. I tried the steps above with 5 folds. Do you average the pre-sigmoid outputs of the folds or the post sigmoid outputs of the folds?\n\nI took the mean of all of the pre-sigmoid outputs from the folds, applied sigmoid and thresholding to the mean and found that the score was really terrible upon submission. I could be missing something here or making a trivial mistake. Some insight would really help :) \n\nThanks!", "votes": null }, { "id": "530890", "postDate": "05/13/2019 22:32:06", "content": "After sigmoid. NN outputs without sigmoid have different range of values. It needs to be scaled to 0-1 before averaging.
", "rawMarkdown": "After sigmoid. NN outputs without sigmoid have different range of values. It needs to be scaled to 0-1 before averaging.", "votes": null }, { "id": "530899", "postDate": "05/13/2019 23:05:08", "content": "Thanks a lot :)
", "rawMarkdown": "Thanks a lot :)", "votes": null }, { "id": "531010", "postDate": "05/14/2019 05:46:53", "content": "thanks so much!
", "rawMarkdown": "thanks so much!", "votes": null }, { "id": "531011", "postDate": "05/14/2019 05:47:05", "content": "helped me a lot!
", "rawMarkdown": "helped me a lot!", "votes": null }, { "id": "531243", "postDate": "05/14/2019 14:35:59", "content": "@demonplus you store the output from the model in .npy files, basically for a sample in test & validation set you store probability of the sample belonging to all attribute ids. So in short the shape of these numpy arrays will be (N_sample, N_attribute_ids) and each preds[sample_id][attribute_id] will have a probability of sample sample_id belonging to a class attribute_id
When making submission.csv, can I use the sample_submission.csv file to get the test images' id? Or, I must get all images' name in the test folder by using some library like glob? As a beginner, I'm not sure if there also is a sample_submission.csv file in the second stage test.
", "rawMarkdown": "When making submission.csv, can I use the sample_submission.csv file to get the test images' id? Or, I must get all images' name in the test folder by using some library like glob? As a beginner, I'm not sure if there also is a sample_submission.csv file in the second stage test.", "votes": null }, { "id": "533482", "postDate": "05/19/2019 11:08:50", "content": "sample_submission.csv will be replaced for 2nd stage and you can use that to get a list of test image ids for 2nd stage.
", "rawMarkdown": "sample_submission.csv will be replaced for 2nd stage and you can use that to get a list of test image ids for 2nd stage.", "votes": null }, { "id": "533488", "postDate": "05/19/2019 11:24:23", "content": "thanks very much!
", "rawMarkdown": "thanks very much!", "votes": null }, { "id": "535656", "postDate": "05/23/2019 09:42:27", "content": "I've encountered a problem while averaging sigmoid outputs of different folds like it is described in current thread. For me in different folds a number of out_features is a little bit different (from 1098 to 1101 for 5 fold model), probably this is because some attributes are not encountered in some folds.
\n\nThis is why I got an error while taking mean/average of sigmoids (predictions just have different size). I thought may be somebody had similar error and how have you solved it?
", "rawMarkdown": "I've encountered a problem while averaging sigmoid outputs of different folds like it is described in current thread. For me in different folds a number of out_features is a little bit different (from 1098 to 1101 for 5 fold model), probably this is because some attributes are not encountered in some folds. \n\nThis is why I got an error while taking mean/average of sigmoids (predictions just have different size). I thought may be somebody had similar error and how have you solved it?", "votes": null }, { "id": "536069", "postDate": "05/23/2019 22:11:11", "content": "The number of out_features should be the same among all folds. You can hardcode that number instead of dynamically determining on each run.
", "rawMarkdown": "The number of out_features should be the same among all folds. You can hardcode that number instead of dynamically determining on each run.", "votes": null } ], "comments": [ { "id": 530518, "author_name": "axel81", "author_url": "", "post_date": "05/13/2019 05:01:19", "content": "@yangsaewon what you can do is run the same model in 5 different kernels for N number of epochs & save their weights. Now you can create a new kernel which uses the weights of all 5 models to generate 5 predictions on the test set & take the mean of the predictions.
Another approach would be to generate predictions separately for all 5 kernels & save the output of model in .npy files which you can use in a new kernel to do a mean operation & decide threshold.
Hope this helps, let me know if you have any doubts.
", "votes": null, "replies": [ { "id": 530647, "author_name": "demonplus", "author_url": "", "post_date": "05/13/2019 11:07:32", "content": "Could you please clarify your idea about .npy files, what is saved in them?
\n\nIn case of test set is changed, how this kernel based on your second approach with ready predictions work?
", "votes": null, "replies": [] }, { "id": 531011, "author_name": "yangsaewon", "author_url": "", "post_date": "05/14/2019 05:47:05", "content": "helped me a lot!
", "votes": null, "replies": [] }, { "id": 531243, "author_name": "axel81", "author_url": "", "post_date": "05/14/2019 14:35:59", "content": "@demonplus you store the output from the model in .npy files, basically for a sample in test & validation set you store probability of the sample belonging to all attribute ids. So in short the shape of these numpy arrays will be (N_sample, N_attribute_ids) and each preds[sample_id][attribute_id] will have a probability of sample sample_id belonging to a class attribute_id
Have fun.
", "votes": null, "replies": [ { "id": 531010, "author_name": "yangsaewon", "author_url": "", "post_date": "05/14/2019 05:46:53", "content": "thanks so much!
", "votes": null, "replies": [] }, { "id": 533113, "author_name": "qunyang", "author_url": "", "post_date": "05/18/2019 13:26:06", "content": "When making submission.csv, can I use the sample_submission.csv file to get the test images' id? Or, I must get all images' name in the test folder by using some library like glob? As a beginner, I'm not sure if there also is a sample_submission.csv file in the second stage test.
", "votes": null, "replies": [] }, { "id": 533482, "author_name": "appian", "author_url": "", "post_date": "05/19/2019 11:08:50", "content": "sample_submission.csv will be replaced for 2nd stage and you can use that to get a list of test image ids for 2nd stage.
", "votes": null, "replies": [] }, { "id": 533488, "author_name": "qunyang", "author_url": "", "post_date": "05/19/2019 11:24:23", "content": "thanks very much!
", "votes": null, "replies": [] } ] }, { "id": 530836, "author_name": "sairam6087", "author_url": "", "post_date": "05/13/2019 19:47:01", "content": "@appian @axel81 I'm relatively new to ensembling and averaging folds. Could you explain what you mean by averaging the outputs of the folds. I tried the steps above with 5 folds. Do you average the pre-sigmoid outputs of the folds or the post sigmoid outputs of the folds?
\n\nI took the mean of all of the pre-sigmoid outputs from the folds, applied sigmoid and thresholding to the mean and found that the score was really terrible upon submission. I could be missing something here or making a trivial mistake. Some insight would really help :)
\n\nThanks!
", "votes": null, "replies": [ { "id": 530890, "author_name": "appian", "author_url": "", "post_date": "05/13/2019 22:32:06", "content": "After sigmoid. NN outputs without sigmoid have different range of values. It needs to be scaled to 0-1 before averaging.
", "votes": null, "replies": [] }, { "id": 530899, "author_name": "sairam6087", "author_url": "", "post_date": "05/13/2019 23:05:08", "content": "Thanks a lot :)
", "votes": null, "replies": [] } ] }, { "id": 535656, "author_name": "demonplus", "author_url": "", "post_date": "05/23/2019 09:42:27", "content": "I've encountered a problem while averaging sigmoid outputs of different folds like it is described in current thread. For me in different folds a number of out_features is a little bit different (from 1098 to 1101 for 5 fold model), probably this is because some attributes are not encountered in some folds.
\n\nThis is why I got an error while taking mean/average of sigmoids (predictions just have different size). I thought may be somebody had similar error and how have you solved it?
", "votes": null, "replies": [ { "id": 536069, "author_name": "appian", "author_url": "", "post_date": "05/23/2019 22:11:11", "content": "The number of out_features should be the same among all folds. You can hardcode that number instead of dynamically determining on each run.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "530499": "I'm having difficulty understanding about the training flow of the training process\nI can only train on the kernel because I have no local gpu.\n\nThe question is, how can I submit mean of 5 different models(different fold) that are trained separately? can someone very kind can explain to me about the flow?.. everyone is doing it except me..\n\nthanks to Lopuhin I made 5 different fold but I can only train with one fold because of the limitation of training time in the kernel :(\n\nany comment would help me\nthanks!", "530518": "yangsaewon what you can do is run the same model in 5 different kernels for `N` number of epochs & save their weights. Now you can create a new kernel which uses the weights of all 5 models to generate 5 predictions on the test set & take the mean of the predictions.\n\nAnother approach would be to generate predictions separately for all 5 kernels & save the output of model in `.npy` files which you can use in a new kernel to do a mean operation & decide threshold.\n\nHope this helps, let me know if you have any doubts.", "530519": "1. Run the kernel for each fold. Make sure each run saves the best weights.\n2. Create a dataset with previously saved weights. \n3. Write a new kernel which does\n3.1 load weights from dataset\n3.2 inference and keep its outputs\n3.3 repeat 3.1-3.2 for each fold\n3.4 take average of all outputs \n3.5 make submission.csv\n\nHave fun.", "530647": "Could you please clarify your idea about .npy files, what is saved in them? \n\nIn case of test set is changed, how this kernel based on your second approach with ready predictions work?", "530836": "appian @axel81 I'm relatively new to ensembling and averaging folds. Could you explain what you mean by averaging the outputs of the folds. I tried the steps above with 5 folds. Do you average the pre-sigmoid outputs of the folds or the post sigmoid outputs of the folds?\n\nI took the mean of all of the pre-sigmoid outputs from the folds, applied sigmoid and thresholding to the mean and found that the score was really terrible upon submission. I could be missing something here or making a trivial mistake. Some insight would really help :) \n\nThanks!", "530890": "After sigmoid. NN outputs without sigmoid have different range of values. It needs to be scaled to 0-1 before averaging.", "530899": "Thanks a lot :)", "531010": "thanks so much!", "531011": "helped me a lot!", "531243": "demonplus you store the output from the model in `.npy` files, basically for a sample in test & validation set you store probability of the sample belonging to all attribute ids. So in short the shape of these numpy arrays will be `(N_sample, N_attribute_ids)` and each `preds[sample_id][attribute_id]` will have a probability of sample `sample_id` belonging to a class `attribute_id`", "533113": "When making submission.csv, can I use the sample_submission.csv file to get the test images' id? Or, I must get all images' name in the test folder by using some library like glob? As a beginner, I'm not sure if there also is a sample_submission.csv file in the second stage test.", "533482": "sample_submission.csv will be replaced for 2nd stage and you can use that to get a list of test image ids for 2nd stage.", "533488": "thanks very much!", "535656": "I've encountered a problem while averaging sigmoid outputs of different folds like it is described in current thread. For me in different folds a number of out_features is a little bit different (from 1098 to 1101 for 5 fold model), probably this is because some attributes are not encountered in some folds. \n\nThis is why I got an error while taking mean/average of sigmoids (predictions just have different size). I thought may be somebody had similar error and how have you solved it?", "536069": "The number of out_features should be the same among all folds. You can hardcode that number instead of dynamically determining on each run." }, "source": "meta" }