![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2845696%2Fa8d12855e3686d61ba27b878dce222e1%2FScreenshot%202020-06-26%20at%2010.27.12%20PM.png?generation=1593191160675741&alt=media\")
I just started working on this competition and decided to start off by making my own model from scratch (No transfer learning yet) and see how it performs. I am using resized 128x128 images.\nThe problem is that I am not able to get my model to learn properly. Below is a screenshot of my model architecture.
\n\n
It starts off strong, improving its AUC and loss for the first few epochs, but after that it just suddenly dips down, and gets a validation AUC of 0.5. The screenshot below shows my problem. The model is learning fine till epoch 14, after which it starts to dip and eventually reaches AUC of 0.5.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2845696%2F5463d21bc76e734be8ba16b06b21505f%2FScreenshot%202020-06-26%20at%2010.35.28%20PM.png?generation=1593191292491730&alt=media\")
I have no idea on what's happening since my model seems to be learning in the beginning, and sort of forgetting everything towards the end.\nIs the problem the way I have handled my data? Or something with the architecture itself?
", "messages": [ { "id": "903241", "postDate": "06/26/2020 17:08:52", "content": "I just started working on this competition and decided to start off by making my own model from scratch (No transfer learning yet) and see how it performs. I am using resized 128x128 images.\nThe problem is that I am not able to get my model to learn properly. Below is a screenshot of my model architecture.
\n\n
It starts off strong, improving its AUC and loss for the first few epochs, but after that it just suddenly dips down, and gets a validation AUC of 0.5. The screenshot below shows my problem. The model is learning fine till epoch 14, after which it starts to dip and eventually reaches AUC of 0.5.
\n\n
I have no idea on what's happening since my model seems to be learning in the beginning, and sort of forgetting everything towards the end.\nIs the problem the way I have handled my data? Or something with the architecture itself?
", "rawMarkdown": "I just started working on this competition and decided to start off by making my own model from scratch (No transfer learning yet) and see how it performs. I am using resized 128x128 images.\nThe problem is that I am not able to get my model to learn properly. Below is a screenshot of my model architecture.\n\n\n\n\nIt starts off strong, improving its AUC and loss for the first few epochs, but after that it just suddenly dips down, and gets a validation AUC of 0.5. The screenshot below shows my problem. The model is learning fine till epoch 14, after which it starts to dip and eventually reaches AUC of 0.5.\n\n\n\n\nI have no idea on what's happening since my model seems to be learning in the beginning, and sort of forgetting everything towards the end.\nIs the problem the way I have handled my data? Or something with the architecture itself?", "votes": null }, { "id": "903417", "postDate": "06/26/2020 20:18:48", "content": "Do you have dropout in the model? If yes, the amount may be too high.
", "rawMarkdown": "Do you have dropout in the model? If yes, the amount may be too high.", "votes": null }, { "id": "903441", "postDate": "06/26/2020 20:35:49", "content": "It looks like something is up with your loss function. Maybe post your focal loss code. Usually when your provide the loss function as an argument to model.compile it is the function object, not a called version of the function. It looks like you are calling focal_loss() with parens. The loss functions in keras have two versions, a camel cased version that is a class that returns a function when you call it and a lower cased version, which is a callable function itself. You can provide the lower cased version as the loss argument, but in that case you would provide it without the parens to call it, so that it can be called with each training step.
", "rawMarkdown": "It looks like something is up with your loss function. Maybe post your focal loss code. Usually when your provide the loss function as an argument to model.compile it is the function object, not a called version of the function. It looks like you are calling focal_loss() with parens. The loss functions in keras have two versions, a camel cased version that is a class that returns a function when you call it and a lower cased version, which is a callable function itself. You can provide the lower cased version as the loss argument, but in that case you would provide it without the parens to call it, so that it can be called with each training step.", "votes": null }, { "id": "903454", "postDate": "06/26/2020 20:46:37", "content": "Dropout,loss function or Learning rate. Check these three.
", "rawMarkdown": "Dropout,loss function or Learning rate. Check these three.", "votes": null }, { "id": "903759", "postDate": "06/27/2020 05:12:02", "content": "I do have dropout, but it isn't high at all - only 0.1. Also, what could be causing the model to learn, and then drop?
", "rawMarkdown": "I do have dropout, but it isn't high at all - only 0.1. Also, what could be causing the model to learn, and then drop?", "votes": null }, { "id": "903769", "postDate": "06/27/2020 05:17:29", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2845696%2Fde333dd29e95df75f7be49e1ca6dc6a7%2FScreenshot%202020-06-27%20at%2010.44.50%20AM.png?generation=1593234920253229&alt=media\")
This is the code for my loss function. I think the function which I have called returns a non-paranthesis based function, like you mentioned. Is there something else I am missing? I plucked this out of a public notebook.
", "rawMarkdown": "\n\nThis is the code for my loss function. I think the function which I have called returns a non-paranthesis based function, like you mentioned. Is there something else I am missing? I plucked this out of a public notebook.", "votes": null }, { "id": "903781", "postDate": "06/27/2020 05:32:18", "content": "I guess your learning rate of 0.01 is pretty larger for the later epochs.
", "rawMarkdown": "I guess your learning rate of 0.01 is pretty larger for the later epochs.", "votes": null }, { "id": "903891", "postDate": "06/27/2020 07:35:36", "content": "I see. I guess I'll try with smaller learning rates and see how it goes.
", "rawMarkdown": "I see. I guess I'll try with smaller learning rates and see how it goes.", "votes": null }, { "id": "903892", "postDate": "06/27/2020 07:36:02", "content": "I think the Dropout and loss function is fine. I'll check the learning rate
", "rawMarkdown": "I think the Dropout and loss function is fine. I'll check the learning rate", "votes": null }, { "id": "903914", "postDate": "06/27/2020 07:57:11", "content": "Trying to train a model from scratch when the data is so imbalanced and the number of positive examples is so small is very very very difficult. For this data you really must use transfer learning. But, if your really want to try, you should:\n1. Do something about the imbalance (weighted loss / uneven sampling / etc.)\n2. Use more data\n3. Use extensive augmentation\n4. Think about what you are predicting
", "rawMarkdown": "Trying to train a model from scratch when the data is so imbalanced and the number of positive examples is so small is very very very difficult. For this data you really must use transfer learning. But, if your really want to try, you should:\n1. Do something about the imbalance (weighted loss / uneven sampling / etc.)\n2. Use more data\n3. Use extensive augmentation\n4. Think about what you are predicting", "votes": null }, { "id": "903943", "postDate": "06/27/2020 08:21:42", "content": "If your'e using focal loss, try to set alpha=0.75 and not 0.25.
", "rawMarkdown": "If your'e using focal loss, try to set alpha=0.75 and not 0.25.", "votes": null }, { "id": "904447", "postDate": "06/27/2020 16:24:08", "content": "I replaced my loss function with yours in my model - auc did the exact same thing - drove to 0.50. Using 0.75 for alpha. Will try the same model with 0.25.
", "rawMarkdown": "I replaced my loss function with yours in my model - auc did the exact same thing - drove to 0.50. Using 0.75 for alpha. Will try the same model with 0.25.", "votes": null }, { "id": "904494", "postDate": "06/27/2020 17:07:43", "content": "I see you're providing a callable function - so that's not the source of your issue.
", "rawMarkdown": "I see you're providing a callable function - so that's not the source of your issue.", "votes": null }, { "id": "904594", "postDate": "06/27/2020 18:39:20", "content": "I plan on using Transfer learning eventually but just wanted to start off by creating my own model from scratch. I think the imbalance might be hurting my model here, so I think I will work on that.
\n\nMy main issue here was not that the AUC was kind of low, but that the AUC dipped. I think I will try augmentation as well.
", "rawMarkdown": "I plan on using Transfer learning eventually but just wanted to start off by creating my own model from scratch. I think the imbalance might be hurting my model here, so I think I will work on that. \n\nMy main issue here was not that the AUC was kind of low, but that the AUC dipped. I think I will try augmentation as well.", "votes": null }, { "id": "904595", "postDate": "06/27/2020 18:40:07", "content": "I see. But how exactly do you find out which alpha to try?
", "rawMarkdown": "I see. But how exactly do you find out which alpha to try?", "votes": null }, { "id": "904632", "postDate": "06/27/2020 19:21:09", "content": "Thanks a lot @ishalgarg , @yash612 . The issue was the high learning rate. I changed that and the model runs as expected.
", "rawMarkdown": "Thanks a lot @ishalgarg , @yash612 . The issue was the high learning rate. I changed that and the model runs as expected.", "votes": null }, { "id": "904720", "postDate": "06/27/2020 20:52:50", "content": "Set alpha closer to 1 will give more importance to false negatives. \nThats a way of dealing with the imbalanced dataset.
", "rawMarkdown": "Set alpha closer to 1 will give more importance to false negatives. \nThats a way of dealing with the imbalanced dataset.", "votes": null }, { "id": "904840", "postDate": "06/28/2020 02:11:00", "content": "As noted by you - high learning rate was an issue for me with your loss function. Reduce LR and auc looking good after 20 epochs - have 4 machines running different alpha values with up to 200 epochs possible. Will update when those runs completed.
", "rawMarkdown": "As noted by you - high learning rate was an issue for me with your loss function. Reduce LR and auc looking good after 20 epochs - have 4 machines running different alpha values with up to 200 epochs possible. Will update when those runs completed.", "votes": null }, { "id": "905070", "postDate": "06/28/2020 08:17:10", "content": "you are welcome , any help just ping me.
", "rawMarkdown": "you are welcome , any help just ping me.", "votes": null }, { "id": "905071", "postDate": "06/28/2020 08:19:56", "content": "@yuval6967 Can we augment the class which is low in number more to obtain balance or have to augment all of them equally , any preference?
", "rawMarkdown": "yuval6967 Can we augment the class which is low in number more to obtain balance or have to augment all of them equally , any preference?", "votes": null }, { "id": "905723", "postDate": "06/28/2020 18:35:30", "content": "@yash612 No!!!! The model will learn how to detect the augmentation. \nYou must augment every class the same way.
", "rawMarkdown": "yash612 No!!!! The model will learn how to detect the augmentation. \nYou must augment every class the same way.", "votes": null }, { "id": "905755", "postDate": "06/28/2020 19:05:54", "content": "@benboren I see. Could you also explain what the gamma factor does? and did increasing alpha help you, or is the current value of 0.25 good enough?
", "rawMarkdown": "benboren I see. Could you also explain what the gamma factor does? and did increasing alpha help you, or is the current value of 0.25 good enough?", "votes": null } ], "comments": [ { "id": 903417, "author_name": "pcjimmmy", "author_url": "", "post_date": "06/26/2020 20:18:48", "content": "Do you have dropout in the model? If yes, the amount may be too high.
", "votes": null, "replies": [ { "id": 903759, "author_name": "yushg123", "author_url": "", "post_date": "06/27/2020 05:12:02", "content": "I do have dropout, but it isn't high at all - only 0.1. Also, what could be causing the model to learn, and then drop?
", "votes": null, "replies": [] }, { "id": 904447, "author_name": "pcjimmmy", "author_url": "", "post_date": "06/27/2020 16:24:08", "content": "I replaced my loss function with yours in my model - auc did the exact same thing - drove to 0.50. Using 0.75 for alpha. Will try the same model with 0.25.
", "votes": null, "replies": [] }, { "id": 904840, "author_name": "pcjimmmy", "author_url": "", "post_date": "06/28/2020 02:11:00", "content": "As noted by you - high learning rate was an issue for me with your loss function. Reduce LR and auc looking good after 20 epochs - have 4 machines running different alpha values with up to 200 epochs possible. Will update when those runs completed.
", "votes": null, "replies": [] } ] }, { "id": 903441, "author_name": "calebeverett", "author_url": "", "post_date": "06/26/2020 20:35:49", "content": "It looks like something is up with your loss function. Maybe post your focal loss code. Usually when your provide the loss function as an argument to model.compile it is the function object, not a called version of the function. It looks like you are calling focal_loss() with parens. The loss functions in keras have two versions, a camel cased version that is a class that returns a function when you call it and a lower cased version, which is a callable function itself. You can provide the lower cased version as the loss argument, but in that case you would provide it without the parens to call it, so that it can be called with each training step.
", "votes": null, "replies": [ { "id": 903769, "author_name": "yushg123", "author_url": "", "post_date": "06/27/2020 05:17:29", "content": "
This is the code for my loss function. I think the function which I have called returns a non-paranthesis based function, like you mentioned. Is there something else I am missing? I plucked this out of a public notebook.
", "votes": null, "replies": [] }, { "id": 904494, "author_name": "calebeverett", "author_url": "", "post_date": "06/27/2020 17:07:43", "content": "I see you're providing a callable function - so that's not the source of your issue.
", "votes": null, "replies": [] } ] }, { "id": 903454, "author_name": "yash612", "author_url": "", "post_date": "06/26/2020 20:46:37", "content": "Dropout,loss function or Learning rate. Check these three.
", "votes": null, "replies": [ { "id": 903892, "author_name": "yushg123", "author_url": "", "post_date": "06/27/2020 07:36:02", "content": "I think the Dropout and loss function is fine. I'll check the learning rate
", "votes": null, "replies": [] } ] }, { "id": 903781, "author_name": "ishalgarg", "author_url": "", "post_date": "06/27/2020 05:32:18", "content": "I guess your learning rate of 0.01 is pretty larger for the later epochs.
", "votes": null, "replies": [ { "id": 903891, "author_name": "yushg123", "author_url": "", "post_date": "06/27/2020 07:35:36", "content": "I see. I guess I'll try with smaller learning rates and see how it goes.
", "votes": null, "replies": [] } ] }, { "id": 903914, "author_name": "yuval6967", "author_url": "", "post_date": "06/27/2020 07:57:11", "content": "Trying to train a model from scratch when the data is so imbalanced and the number of positive examples is so small is very very very difficult. For this data you really must use transfer learning. But, if your really want to try, you should:\n1. Do something about the imbalance (weighted loss / uneven sampling / etc.)\n2. Use more data\n3. Use extensive augmentation\n4. Think about what you are predicting
", "votes": null, "replies": [ { "id": 904594, "author_name": "yushg123", "author_url": "", "post_date": "06/27/2020 18:39:20", "content": "I plan on using Transfer learning eventually but just wanted to start off by creating my own model from scratch. I think the imbalance might be hurting my model here, so I think I will work on that.
\n\nMy main issue here was not that the AUC was kind of low, but that the AUC dipped. I think I will try augmentation as well.
", "votes": null, "replies": [] }, { "id": 905071, "author_name": "yash612", "author_url": "", "post_date": "06/28/2020 08:19:56", "content": "@yuval6967 Can we augment the class which is low in number more to obtain balance or have to augment all of them equally , any preference?
", "votes": null, "replies": [] }, { "id": 905723, "author_name": "yuval6967", "author_url": "", "post_date": "06/28/2020 18:35:30", "content": "@yash612 No!!!! The model will learn how to detect the augmentation. \nYou must augment every class the same way.
", "votes": null, "replies": [] } ] }, { "id": 903943, "author_name": "benboren", "author_url": "", "post_date": "06/27/2020 08:21:42", "content": "If your'e using focal loss, try to set alpha=0.75 and not 0.25.
", "votes": null, "replies": [ { "id": 904595, "author_name": "yushg123", "author_url": "", "post_date": "06/27/2020 18:40:07", "content": "I see. But how exactly do you find out which alpha to try?
", "votes": null, "replies": [] }, { "id": 904720, "author_name": "benboren", "author_url": "", "post_date": "06/27/2020 20:52:50", "content": "Set alpha closer to 1 will give more importance to false negatives. \nThats a way of dealing with the imbalanced dataset.
", "votes": null, "replies": [] }, { "id": 905755, "author_name": "yushg123", "author_url": "", "post_date": "06/28/2020 19:05:54", "content": "@benboren I see. Could you also explain what the gamma factor does? and did increasing alpha help you, or is the current value of 0.25 good enough?
", "votes": null, "replies": [] } ] }, { "id": 904632, "author_name": "yushg123", "author_url": "", "post_date": "06/27/2020 19:21:09", "content": "Thanks a lot @ishalgarg , @yash612 . The issue was the high learning rate. I changed that and the model runs as expected.
", "votes": null, "replies": [ { "id": 905070, "author_name": "yash612", "author_url": "", "post_date": "06/28/2020 08:17:10", "content": "you are welcome , any help just ping me.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "903241": "I just started working on this competition and decided to start off by making my own model from scratch (No transfer learning yet) and see how it performs. I am using resized 128x128 images.\nThe problem is that I am not able to get my model to learn properly. Below is a screenshot of my model architecture.\n\n\n\n\nIt starts off strong, improving its AUC and loss for the first few epochs, but after that it just suddenly dips down, and gets a validation AUC of 0.5. The screenshot below shows my problem. The model is learning fine till epoch 14, after which it starts to dip and eventually reaches AUC of 0.5.\n\n\n\n\nI have no idea on what's happening since my model seems to be learning in the beginning, and sort of forgetting everything towards the end.\nIs the problem the way I have handled my data? Or something with the architecture itself?", "903417": "Do you have dropout in the model? If yes, the amount may be too high.", "903441": "It looks like something is up with your loss function. Maybe post your focal loss code. Usually when your provide the loss function as an argument to model.compile it is the function object, not a called version of the function. It looks like you are calling focal_loss() with parens. The loss functions in keras have two versions, a camel cased version that is a class that returns a function when you call it and a lower cased version, which is a callable function itself. You can provide the lower cased version as the loss argument, but in that case you would provide it without the parens to call it, so that it can be called with each training step.", "903454": "Dropout,loss function or Learning rate. Check these three.", "903759": "I do have dropout, but it isn't high at all - only 0.1. Also, what could be causing the model to learn, and then drop?", "903769": "\n\nThis is the code for my loss function. I think the function which I have called returns a non-paranthesis based function, like you mentioned. Is there something else I am missing? I plucked this out of a public notebook.", "903781": "I guess your learning rate of 0.01 is pretty larger for the later epochs.", "903891": "I see. I guess I'll try with smaller learning rates and see how it goes.", "903892": "I think the Dropout and loss function is fine. I'll check the learning rate", "903914": "Trying to train a model from scratch when the data is so imbalanced and the number of positive examples is so small is very very very difficult. For this data you really must use transfer learning. But, if your really want to try, you should:\n1. Do something about the imbalance (weighted loss / uneven sampling / etc.)\n2. Use more data\n3. Use extensive augmentation\n4. Think about what you are predicting", "903943": "If your'e using focal loss, try to set alpha=0.75 and not 0.25.", "904447": "I replaced my loss function with yours in my model - auc did the exact same thing - drove to 0.50. Using 0.75 for alpha. Will try the same model with 0.25.", "904494": "I see you're providing a callable function - so that's not the source of your issue.", "904594": "I plan on using Transfer learning eventually but just wanted to start off by creating my own model from scratch. I think the imbalance might be hurting my model here, so I think I will work on that. \n\nMy main issue here was not that the AUC was kind of low, but that the AUC dipped. I think I will try augmentation as well.", "904595": "I see. But how exactly do you find out which alpha to try?", "904632": "Thanks a lot @ishalgarg , @yash612 . The issue was the high learning rate. I changed that and the model runs as expected.", "904720": "Set alpha closer to 1 will give more importance to false negatives. \nThats a way of dealing with the imbalanced dataset.", "904840": "As noted by you - high learning rate was an issue for me with your loss function. Reduce LR and auc looking good after 20 epochs - have 4 machines running different alpha values with up to 200 epochs possible. Will update when those runs completed.", "905070": "you are welcome , any help just ping me.", "905071": "yuval6967 Can we augment the class which is low in number more to obtain balance or have to augment all of them equally , any preference?", "905723": "yash612 No!!!! The model will learn how to detect the augmentation. \nYou must augment every class the same way.", "905755": "benboren I see. Could you also explain what the gamma factor does? and did increasing alpha help you, or is the current value of 0.25 good enough?" }, "source": "meta" }