![\"TSNE\"](\"https://i.imgur.com/FGNKvrP.png\")
First of all thanks to my teammates @tikutiku, @titericz , @cpmpml , @cooolz for this great collaboration on this interesting SETI competition. It was really nice for me to meet new Kagglers again and work together to reach 9th place. I’m impressed about the rising of every day new ideas.
\nOur solution is an ensemble of multiple models. The team tried many approaches to make relevant models and close the CV/LB gap. @titericz has found the S-signal (a.k.a. magic#1) by inspecting the predictions and we decided to try different approaches to have similar messages in train data:
\nWe quickly realized that it improved the score but it would never close the CV/LB gap. Another magic was required. We’ve tried to analyze the difference between train and test datasets and apply some preprocessings to make models learn better. We’ve trained some AE + clustered the embeddings and we discovered that many similar images/backgrounds were used for generation: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513
\nBelow is TSNE of images embeddings (train + test) after clustering, small islands have similar backgrounds.
\n
We did not go further in this way (even if it was promising) mainly because it looked more like some kind of reverse engineering of the generated data instead of Machine Learning.
\nAt the end, we’ve ensembled 6 groups of models that scored public LB=0.801:
\nWhat did work:
\nWhat did not work:
\nThanks for reading.
", "messages": [ { "id": 1483522, "postDate": "2021-08-20T17:26:32.587Z", "content": "First of all thanks to my teammates @tikutiku, @titericz , @cpmpml , @cooolz for this great collaboration on this interesting SETI competition. It was really nice for me to meet new Kagglers again and work together to reach 9th place. I’m impressed about the rising of every day new ideas.
\nOur solution is an ensemble of multiple models. The team tried many approaches to make relevant models and close the CV/LB gap. @titericz has found the S-signal (a.k.a. magic#1) by inspecting the predictions and we decided to try different approaches to have similar messages in train data:
\nWe quickly realized that it improved the score but it would never close the CV/LB gap. Another magic was required. We’ve tried to analyze the difference between train and test datasets and apply some preprocessings to make models learn better. We’ve trained some AE + clustered the embeddings and we discovered that many similar images/backgrounds were used for generation: https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513
\nBelow is TSNE of images embeddings (train + test) after clustering, small islands have similar backgrounds.
\n
We did not go further in this way (even if it was promising) mainly because it looked more like some kind of reverse engineering of the generated data instead of Machine Learning.
\nAt the end, we’ve ensembled 6 groups of models that scored public LB=0.801:
\nWhat did work:
\nWhat did not work:
\nThanks for reading.
", "rawMarkdown": "First of all thanks to my teammates @tikutiku, @titericz , @cpmpml , @cooolz for this great collaboration on this interesting SETI competition. It was really nice for me to meet new Kagglers again and work together to reach 9th place. I’m impressed about the rising of every day new ideas.\n\nOur solution is an ensemble of multiple models. The team tried many approaches to make relevant models and close the CV/LB gap. @titericz has found the S-signal (a.k.a. magic#1) by inspecting the predictions and we decided to try different approaches to have similar messages in train data:\n\n\n\n* Different custom augmentations to make existing narrow-band messages look similar to S-signal. The idea is that sine deformation will move a line to a S and all other properties (intensity, SNR, ..) would remain.\n* A generator to have much more control on S-signal (and create other signals too): [https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266805](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266805)\n\nWe quickly realized that it improved the score but it would never close the CV/LB gap. Another magic was required. We’ve tried to analyze the difference between train and test datasets and apply some preprocessings to make models learn better. We’ve trained some AE + clustered the embeddings and we discovered that many similar images/backgrounds were used for generation: [https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513)\n\nBelow is TSNE of images embeddings (train + test) after clustering, small islands have similar backgrounds.\n\n\n\nWe did not go further in this way (even if it was promising) mainly because it looked more like some kind of reverse engineering of the generated data instead of Machine Learning. \n\nAt the end, we’ve ensembled 6 groups of models that scored public LB=0.801:\n\n\n\n* Classification models with different backbone and input size: 512x512 for effnetb5-ns and 512x640 for effnetv2-s. Based only on [0,2,4] ON images. ShiftScaleRotate + H/V flip + Resize + Mixup augmentations. Checkpoint averaging. Pseudo labels included. 4 random TTA on inference.Public LB=0.795 \n* Classification models with different preprocessings, mixup with target multiplicator, additional custom sine augmentation. Pseudo labels included. TTA on inference. Public LB=0.792.\n* Classification models with more messages from our custom generator/simulator. TTA on inference. Public LB=0.788\n* Classification models with different input sizes (640x512, 480x384) and various preprocessings. Based only on [0,2,4]. ShiftScaleRotate + H/V + Resize + Mixup + sine augmentation. WeightedRandomSampler to combat imbalance, H/V TTA on inference. Some trained on all data (old_train, old_test, new_train). Public LB=0.781\n* Single classification model (tf_efficientnetv2_s with keeping high resolution - first conv, only freq axis - and L2 normalize). Based on 2 channels [0,2,4] + [1,3,5]. Mixup (simulating faint signal) + H/V augmentation. Pseudo labels included. H/V TTA on inference. Public LB=0.779\n* Segmentation model trained with our custom generator/simulator. This one is able to extract the E.T. signal which could be interesting in production.Public LB=0.771\n\n \n\nWhat did work:\n\n\n\n* Different preprocessings (normalizations on different axis, [difference_of_gaussians](https://scikit-image.org/docs/dev/api/skimage.filters.html#skimage.filters.difference_of_gaussians)) \n* Different image sizes, one or two channels\n* Custom sine augmentation\n* Additional simulator messages\n* Checkpoint averaging\n* Pseudo labels\n* Rank probabilities (a bit better than raw probabilities)\n\nWhat did not work:\n\n\n\n* Shuffle the [0,2,4] channel as augmentation\n* Cutout augmentation\n* Label smoothing\n* Pseudo labels (failed only for me but might be due to calibration issue or training mess-up with test backgrounds as explained by @philippsinger in his solution).\n\nThanks for reading.", "votes": 19 }, { "id": 1495850, "postDate": "2021-08-29T20:42:26.713Z", "content": "Good work =))
", "rawMarkdown": "Good work =))", "votes": 1 }, { "id": 1488092, "postDate": "2021-08-24T05:12:08.990Z", "content": "Well done to team, and congratulations on achieving rank of GM @mpware!
", "rawMarkdown": "Well done to team, and congratulations on achieving rank of GM @mpware!", "votes": 2, "replies": [ { "id": 1488210, "postDate": "2021-08-24T06:55:35.923Z", "content": "Thank you @rafiko1 !
", "rawMarkdown": "Thank you @rafiko1 !", "votes": 2 } ] }, { "id": 1485293, "postDate": "2021-08-22T01:03:15.553Z", "content": "Thanks for teaming, it was an interesting experience. Maybe we could have done better if we had not chased magic 2.
", "rawMarkdown": "Thanks for teaming, it was an interesting experience. Maybe we could have done better if we had not chased magic 2.", "votes": 2 }, { "id": 1483528, "postDate": "2021-08-20T17:29:03.320Z", "content": "Thanks a lot for sharing this, it was really helpful, upvoting it! ⬆️
", "rawMarkdown": "Thanks a lot for sharing this, it was really helpful, upvoting it! ⬆️", "votes": 2, "replies": [ { "id": 1483531, "postDate": "2021-08-20T17:30:26.010Z", "content": "Thanks for the reading!
", "rawMarkdown": "Thanks for the reading! ", "votes": 1 } ] } ], "comments": [ { "id": 1495850, "author_name": "Fuco", "author_url": "", "post_date": "2021-08-29T20:42:26.713000", "content": "Good work =))
", "votes": 1, "replies": [] }, { "id": 1488092, "author_name": "Rafi Hai", "author_url": "", "post_date": "2021-08-24T05:12:08.990000", "content": "Well done to team, and congratulations on achieving rank of GM @mpware!
", "votes": 2, "replies": [ { "id": 1488210, "author_name": "MPWARE", "author_url": "", "post_date": "2021-08-24T06:55:35.923000", "content": "Thank you @rafiko1 !
", "votes": 2, "replies": [] } ] }, { "id": 1485293, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-22T01:03:15.553000", "content": "Thanks for teaming, it was an interesting experience. Maybe we could have done better if we had not chased magic 2.
", "votes": 2, "replies": [] }, { "id": 1483528, "author_name": "Rishiraj Acharya", "author_url": "", "post_date": "2021-08-20T17:29:03.320000", "content": "Thanks a lot for sharing this, it was really helpful, upvoting it! ⬆️
", "votes": 2, "replies": [ { "id": 1483531, "author_name": "MPWARE", "author_url": "", "post_date": "2021-08-20T17:30:26.010000", "content": "Thanks for the reading!
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1483522": "First of all thanks to my teammates @tikutiku, @titericz , @cpmpml , @cooolz for this great collaboration on this interesting SETI competition. It was really nice for me to meet new Kagglers again and work together to reach 9th place. I’m impressed about the rising of every day new ideas.\n\nOur solution is an ensemble of multiple models. The team tried many approaches to make relevant models and close the CV/LB gap. @titericz has found the S-signal (a.k.a. magic#1) by inspecting the predictions and we decided to try different approaches to have similar messages in train data:\n\n\n\n* Different custom augmentations to make existing narrow-band messages look similar to S-signal. The idea is that sine deformation will move a line to a S and all other properties (intensity, SNR, ..) would remain.\n* A generator to have much more control on S-signal (and create other signals too): [https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266805](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266805)\n\nWe quickly realized that it improved the score but it would never close the CV/LB gap. Another magic was required. We’ve tried to analyze the difference between train and test datasets and apply some preprocessings to make models learn better. We’ve trained some AE + clustered the embeddings and we discovered that many similar images/backgrounds were used for generation: [https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513](https://www.kaggle.com/c/seti-breakthrough-listen/discussion/266513)\n\nBelow is TSNE of images embeddings (train + test) after clustering, small islands have similar backgrounds.\n\n\n\nWe did not go further in this way (even if it was promising) mainly because it looked more like some kind of reverse engineering of the generated data instead of Machine Learning. \n\nAt the end, we’ve ensembled 6 groups of models that scored public LB=0.801:\n\n\n\n* Classification models with different backbone and input size: 512x512 for effnetb5-ns and 512x640 for effnetv2-s. Based only on [0,2,4] ON images. ShiftScaleRotate + H/V flip + Resize + Mixup augmentations. Checkpoint averaging. Pseudo labels included. 4 random TTA on inference.Public LB=0.795 \n* Classification models with different preprocessings, mixup with target multiplicator, additional custom sine augmentation. Pseudo labels included. TTA on inference. Public LB=0.792.\n* Classification models with more messages from our custom generator/simulator. TTA on inference. Public LB=0.788\n* Classification models with different input sizes (640x512, 480x384) and various preprocessings. Based only on [0,2,4]. ShiftScaleRotate + H/V + Resize + Mixup + sine augmentation. WeightedRandomSampler to combat imbalance, H/V TTA on inference. Some trained on all data (old_train, old_test, new_train). Public LB=0.781\n* Single classification model (tf_efficientnetv2_s with keeping high resolution - first conv, only freq axis - and L2 normalize). Based on 2 channels [0,2,4] + [1,3,5]. Mixup (simulating faint signal) + H/V augmentation. Pseudo labels included. H/V TTA on inference. Public LB=0.779\n* Segmentation model trained with our custom generator/simulator. This one is able to extract the E.T. signal which could be interesting in production.Public LB=0.771\n\n \n\nWhat did work:\n\n\n\n* Different preprocessings (normalizations on different axis, [difference_of_gaussians](https://scikit-image.org/docs/dev/api/skimage.filters.html#skimage.filters.difference_of_gaussians)) \n* Different image sizes, one or two channels\n* Custom sine augmentation\n* Additional simulator messages\n* Checkpoint averaging\n* Pseudo labels\n* Rank probabilities (a bit better than raw probabilities)\n\nWhat did not work:\n\n\n\n* Shuffle the [0,2,4] channel as augmentation\n* Cutout augmentation\n* Label smoothing\n* Pseudo labels (failed only for me but might be due to calibration issue or training mess-up with test backgrounds as explained by @philippsinger in his solution).\n\nThanks for reading.", "1495850": "Good work =))", "1488092": "Well done to team, and congratulations on achieving rank of GM @mpware!", "1485293": "Thanks for teaming, it was an interesting experience. Maybe we could have done better if we had not chased magic 2.", "1483528": "Thanks a lot for sharing this, it was really helpful, upvoting it! ⬆️" } }