![\"https://i.ibb.co/W0c4fDL/Selection-774.png\"](\"https://i.ibb.co/W0c4fDL/Selection-774.png\")
did you encounter a black hole in your training?
\n
did you encounter a black hole in your training?
\n
Yes, it happened multiple times during my trainings.
\nIn some cases, by reducing the learning rate I managed to get rid of it. Not always though.
\nFor some runs, such \"blackholes\" happened after the first epoch.
\nMoreover it looks lihe the weight update was so brutal that after the \"blackhole\" the learning never recovers and get stuck at AUC=0.5.
It also happens to TUMOR competition. lol
", "rawMarkdown": "It also happens to TUMOR competition. lol", "votes": null }, { "id": "1496668", "postDate": "08/30/2021 14:47:01", "content": "\"weight update was so brutal that after the \"blackhole\"
\nthe probability score for the good +ve samples is almost 1.0. high score means zero loss. They have \"virtually disappeared\" from the training set
\n![\"https://i.ibb.co/qWMCWFv/Selection-775.png\"](\"https://i.ibb.co/qWMCWFv/Selection-775.png\")
It for this reason I thought focal loss would be a sword in this competition, but it ended up being a butter knife.
", "rawMarkdown": "It for this reason I thought focal loss would be a sword in this competition, but it ended up being a butter knife.", "votes": null }, { "id": "1496816", "postDate": "08/30/2021 16:48:32", "content": "It happens frequently when the batch size is large. Trying to survive from black hole :)
", "rawMarkdown": "It happens frequently when the batch size is large. Trying to survive from black hole :)", "votes": null }, { "id": "1497096", "postDate": "08/30/2021 23:46:23", "content": "For me, this has been very architecture-specific. Some architectures always converge, and some never do.
", "rawMarkdown": "For me, this has been very architecture-specific. Some architectures always converge, and some never do.", "votes": null }, { "id": "1497113", "postDate": "08/31/2021 00:52:36", "content": "Exactly. NFNet variants always cross the event horizon.
", "rawMarkdown": "Exactly. NFNet variants always cross the event horizon.", "votes": null }, { "id": "1497118", "postDate": "08/31/2021 01:00:24", "content": "Glad to hear someone else is experiencing this. I was going crazy trying to get the ECA variant of NFNet's working since it seemed to be successful in other spectrogram-type competitions.
", "rawMarkdown": "Glad to hear someone else is experiencing this. I was going crazy trying to get the ECA variant of NFNet's working since it seemed to be successful in other spectrogram-type competitions.", "votes": null }, { "id": "1497130", "postDate": "08/31/2021 01:28:13", "content": "I have been facing this as well. Specifically when training WaveNets
", "rawMarkdown": "I have been facing this as well. Specifically when training WaveNets", "votes": null }, { "id": "1497262", "postDate": "08/31/2021 04:54:43", "content": "Interesting plot! can you elaborate on what +ve label means? Thanks!
", "rawMarkdown": "Interesting plot! can you elaborate on what +ve label means? Thanks!", "votes": null }, { "id": "1497349", "postDate": "08/31/2021 06:42:22", "content": "could you please explain a little bit more? I found in tumor competition, some models works, some don't. not understanding the fundamental.
", "rawMarkdown": "could you please explain a little bit more? I found in tumor competition, some models works, some don't. not understanding the fundamental.", "votes": null }, { "id": "1497904", "postDate": "08/31/2021 13:51:05", "content": "Wow this is beyond my imagination
", "rawMarkdown": "Wow this is beyond my imagination", "votes": null }, { "id": "1498225", "postDate": "08/31/2021 19:06:25", "content": "good observation….
", "rawMarkdown": "good observation....", "votes": null }, { "id": "1498256", "postDate": "08/31/2021 19:41:46", "content": "positive and negative labels
", "rawMarkdown": "positive and negative labels", "votes": null }, { "id": "1498623", "postDate": "09/01/2021 05:19:20", "content": "Got it. Thanks!
", "rawMarkdown": "Got it. Thanks!", "votes": null }, { "id": "1501798", "postDate": "09/03/2021 14:48:58", "content": "Reduce mini batch from 32 to 16 and reduce learning rate by 1/10 works for me. Thank you for the suggestions
", "rawMarkdown": "Reduce mini batch from 32 to 16 and reduce learning rate by 1/10 works for me. Thank you for the suggestions", "votes": null }, { "id": "1501809", "postDate": "09/03/2021 14:58:41", "content": "Could that behavior of the networks come from an exploding gradient ? I plan to test some kind of gradient clipping to see if this would solve the issue: I wonder if somebody already checked that possibility ?
", "rawMarkdown": "Could that behavior of the networks come from an exploding gradient ? I plan to test some kind of gradient clipping to see if this would solve the issue: I wonder if somebody already checked that possibility ?", "votes": null }, { "id": "1502554", "postDate": "09/04/2021 11:46:23", "content": "Nice finding, considering the plots I saw in forum, there are around 120'000 (for 120'000 TPs typical model gives > 0.99 probability) good +ve samples in the train set, it means that there is a possibility that more than half of samples from batch give almost zero loss, hence it means that something I'd like to call \"hidden learning rate\" is being introduced (because suppose batch size is N, but in average we have N / 2 effective samples which bring new information to the network, yet still loss is being divided by N, hence learning rate approximately reduced by half)
\nWe should consider it when setting schedulers.
Also, should we try active learning approaches?
\n(exclude samples from the train set if they're already learned, i.e. have zero loss)
image by @authman
\n
Uh, why was I downvoted? Did I say something wrong? I had never even imagined that something like this could also occur
", "rawMarkdown": "Uh, why was I downvoted? Did I say something wrong? I had never even imagined that something like this could also occur", "votes": null }, { "id": "1502681", "postDate": "09/04/2021 14:34:17", "content": "@analokamus you mean nfnets are working well or poor?
", "rawMarkdown": "analokamus you mean nfnets are working well or poor?", "votes": null }, { "id": "1502699", "postDate": "09/04/2021 14:48:25", "content": "@pheadrus It does not work at all.
", "rawMarkdown": "pheadrus It does not work at all.", "votes": null }, { "id": "1502880", "postDate": "09/04/2021 18:09:43", "content": "Insane Observation!!
", "rawMarkdown": "Insane Observation!!", "votes": null }, { "id": "1502938", "postDate": "09/04/2021 19:08:03", "content": "Don't pay attention to haters, just keep modeling.
", "rawMarkdown": "Don't pay attention to haters, just keep modeling.", "votes": null }, { "id": "1506183", "postDate": "09/08/2021 01:43:41", "content": "update1:
\n![\"https://i.ibb.co/pXcbTc8/Selection-806.png\"](\"https://i.ibb.co/pXcbTc8/Selection-806.png\")
this model? What model?
@tuckerarrants exactly the same I'm facing with wevanet/1d variants - I'd be interested to know if you did find a breakthrough
", "rawMarkdown": "tuckerarrants exactly the same I'm facing with wevanet/1d variants - I'd be interested to know if you did find a breakthrough", "votes": null }, { "id": "1508069", "postDate": "09/09/2021 19:35:37", "content": "@kfk42kfk You don't think Heng shares enough? He is kind to share his ideas, but maybe he can kep some details hidden?
", "rawMarkdown": "kfk42kfk You don't think Heng shares enough? He is kind to share his ideas, but maybe he can kep some details hidden?", "votes": null }, { "id": "1559698", "postDate": "10/27/2021 07:07:11", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Yes, it happened multiple times during my trainings.
\nIn some cases, by reducing the learning rate I managed to get rid of it. Not always though.
\nFor some runs, such \"blackholes\" happened after the first epoch.
\nMoreover it looks lihe the weight update was so brutal that after the \"blackhole\" the learning never recovers and get stuck at AUC=0.5.
\"weight update was so brutal that after the \"blackhole\"
\nthe probability score for the good +ve samples is almost 1.0. high score means zero loss. They have \"virtually disappeared\" from the training set
\n
It for this reason I thought focal loss would be a sword in this competition, but it ended up being a butter knife.
", "votes": null, "replies": [] }, { "id": 1496816, "author_name": "seshurajup", "author_url": "", "post_date": "08/30/2021 16:48:32", "content": "It happens frequently when the batch size is large. Trying to survive from black hole :)
", "votes": null, "replies": [] }, { "id": 1497262, "author_name": "richx86", "author_url": "", "post_date": "08/31/2021 04:54:43", "content": "Interesting plot! can you elaborate on what +ve label means? Thanks!
", "votes": null, "replies": [] }, { "id": 1498256, "author_name": "mihtw1", "author_url": "", "post_date": "08/31/2021 19:41:46", "content": "positive and negative labels
", "votes": null, "replies": [] }, { "id": 1498623, "author_name": "richx86", "author_url": "", "post_date": "09/01/2021 05:19:20", "content": "Got it. Thanks!
", "votes": null, "replies": [] }, { "id": 1502554, "author_name": "martynoveduard", "author_url": "", "post_date": "09/04/2021 11:46:23", "content": "Nice finding, considering the plots I saw in forum, there are around 120'000 (for 120'000 TPs typical model gives > 0.99 probability) good +ve samples in the train set, it means that there is a possibility that more than half of samples from batch give almost zero loss, hence it means that something I'd like to call \"hidden learning rate\" is being introduced (because suppose batch size is N, but in average we have N / 2 effective samples which bring new information to the network, yet still loss is being divided by N, hence learning rate approximately reduced by half)
\nWe should consider it when setting schedulers.
Also, should we try active learning approaches?
\n(exclude samples from the train set if they're already learned, i.e. have zero loss)
image by @authman
\n
It also happens to TUMOR competition. lol
", "votes": null, "replies": [] }, { "id": 1497096, "author_name": "marktenenholtz", "author_url": "", "post_date": "08/30/2021 23:46:23", "content": "For me, this has been very architecture-specific. Some architectures always converge, and some never do.
", "votes": null, "replies": [ { "id": 1497113, "author_name": "analokamus", "author_url": "", "post_date": "08/31/2021 00:52:36", "content": "Exactly. NFNet variants always cross the event horizon.
", "votes": null, "replies": [] }, { "id": 1497118, "author_name": "marktenenholtz", "author_url": "", "post_date": "08/31/2021 01:00:24", "content": "Glad to hear someone else is experiencing this. I was going crazy trying to get the ECA variant of NFNet's working since it seemed to be successful in other spectrogram-type competitions.
", "votes": null, "replies": [] }, { "id": 1497130, "author_name": "tuckerarrants", "author_url": "", "post_date": "08/31/2021 01:28:13", "content": "I have been facing this as well. Specifically when training WaveNets
", "votes": null, "replies": [] }, { "id": 1497349, "author_name": "dragonzhang", "author_url": "", "post_date": "08/31/2021 06:42:22", "content": "could you please explain a little bit more? I found in tumor competition, some models works, some don't. not understanding the fundamental.
", "votes": null, "replies": [] }, { "id": 1502681, "author_name": "pheadrus", "author_url": "", "post_date": "09/04/2021 14:34:17", "content": "@analokamus you mean nfnets are working well or poor?
", "votes": null, "replies": [] }, { "id": 1502699, "author_name": "analokamus", "author_url": "", "post_date": "09/04/2021 14:48:25", "content": "@pheadrus It does not work at all.
", "votes": null, "replies": [] }, { "id": 1507839, "author_name": "imeintanis", "author_url": "", "post_date": "09/09/2021 15:32:38", "content": "@tuckerarrants exactly the same I'm facing with wevanet/1d variants - I'd be interested to know if you did find a breakthrough
", "votes": null, "replies": [] } ] }, { "id": 1497904, "author_name": "keagle", "author_url": "", "post_date": "08/31/2021 13:51:05", "content": "Wow this is beyond my imagination
", "votes": null, "replies": [ { "id": 1502616, "author_name": "keagle", "author_url": "", "post_date": "09/04/2021 13:23:02", "content": "Uh, why was I downvoted? Did I say something wrong? I had never even imagined that something like this could also occur
", "votes": null, "replies": [] }, { "id": 1502938, "author_name": "authman", "author_url": "", "post_date": "09/04/2021 19:08:03", "content": "Don't pay attention to haters, just keep modeling.
", "votes": null, "replies": [] } ] }, { "id": 1498225, "author_name": "nishanthaddagatla", "author_url": "", "post_date": "08/31/2021 19:06:25", "content": "good observation….
", "votes": null, "replies": [] }, { "id": 1501798, "author_name": "ivan3357", "author_url": "", "post_date": "09/03/2021 14:48:58", "content": "Reduce mini batch from 32 to 16 and reduce learning rate by 1/10 works for me. Thank you for the suggestions
", "votes": null, "replies": [] }, { "id": 1501809, "author_name": "fabiendaniel", "author_url": "", "post_date": "09/03/2021 14:58:41", "content": "Could that behavior of the networks come from an exploding gradient ? I plan to test some kind of gradient clipping to see if this would solve the issue: I wonder if somebody already checked that possibility ?
", "votes": null, "replies": [] }, { "id": 1502880, "author_name": "ayan007", "author_url": "", "post_date": "09/04/2021 18:09:43", "content": "Insane Observation!!
", "votes": null, "replies": [] }, { "id": 1506183, "author_name": "hengck23", "author_url": "", "post_date": "09/08/2021 01:43:41", "content": "update1:
\n
this model? What model?
@kfk42kfk You don't think Heng shares enough? He is kind to share his ideas, but maybe he can kep some details hidden?
", "votes": null, "replies": [] } ] }, { "id": 1559698, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 07:07:11", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris