I am thrilled to finally unveil my solution to this competition!
\nMy optimal model emerged as a composite of four models, where three were employed as a normalized sum, and the fourth acted as an amplifier, showcasing discernible impact, potentially on labels with alternating signs:
\nweight_df1: 0.5 (utilizing std, mean, and clustering sampling, yielding 0.551)
weight_df2: 0.25 (excluding uncommon elements, resulting in 0.559)
weight_df3: 0.25 (leveraging clustering sampling, achieving 0.575)
weight_df4: 0.3 (incorporating mean, random sampling, and excluding std, attaining 0.554)
\nThe resultant model is expressed as the weighted sum of these components: resulted_model = weight_df1 * df1 + weight_df2 * df2 + weight_df3 * df3 + weight_df4 * df4. All models were trained using the same transformer architecture, albeit with varying dimensional models (with the optimal dimension being 128).
Data preprocessing and feature selection
\nTo train deep learning models on categorical labels, I employed one-hot encoding transformation. To address high and low bias labels, I utilized target encoding by calculating the mean and standard deviation for each cell type and SM name. During experimentation, I compared models with and without uncommon columns and discovered that including uncommon columns significantly improved the training of encoding layers for mean and standard deviation feature vectors, resulting in noticeable performance enhancement.
\nEDA
In my feature exploration, I performed truncated Singular Value Decomposition (SVD) specifically on the target variables. Despite experimenting with different SVD sizes, the models generated from this approach consistently fell short of replicating the performance observed in the full targets regression. Notably, as part of this analysis, I identified certain targets with significant standard deviation values.
\nTo leverage this insight, I strategically enriched the feature set by introducing the standard deviation as a dedicated feature for each cell type and SM name. This involved concatenating these standard deviation values into a unified vector format (std_cell_type, std_sm_name).
\nFurthermore, as I delved into the dataset, I conducted a comprehensive examination, revealing a remarkable cleanliness characterized by the absence of NaN values and duplicates. This meticulous exploration not only extended to model training but also encompassed a nuanced analysis of the target variables, contributing to the informed enhancement of the feature set for improved model performance.
\nSampling strategy
\nIn devising an effective sampling strategy for partitioning the data into training and validation sets, I employed a sophisticated approach rooted in the clusters derived from a K-Means clustering analysis on the target values. The rationale behind this strategy lies in the premise that data points within similar clusters share inherent patterns or characteristics, thus ensuring a more nuanced and representative split.
For each distinct cluster identified through K-Means, I executed the data partitioning using the train_test_split function from the scikit-learn library. This method facilitated a controlled allocation of data points to both the training and validation sets, ensuring that the models were exposed to a diverse yet stratified representation of the data.
\nThe validation percentage, a crucial parameter in this process, was meticulously chosen to strike a balance between model robustness and computational efficiency. By experimenting with validation percentages ranging from 0.1 to 0.2, I systematically assessed the impact on model performance. Ultimately, for the optimal model configuration, a validation percentage of 0.1 was deemed most effective. This decision was guided by a careful consideration of the trade-off between the need for a sufficiently large training set and the importance of a robust validation set to gauge model generalizability.
\nIn summary, this sampling strategy, grounded in cluster-based data splitting, not only acknowledges the underlying structure within the target values but also optimizes the partitioning process to enhance the model's ability to capture diverse patterns and generalize effectively to unseen data.
\nModeling
\nThis is my best architecture
\n (nn.Module): \n ():\n (CustomTransformer_v3, self).__init__()\n self.num_target_encodings = * \n self.num_sparse_features = num_features - self.num_target_encodings\n\n self.sparse_feature_embedding = nn.Linear(self.num_sparse_features, d_model)\n self.target_encoding_embedding = nn.Linear(self.num_target_encodings, d_model)\n self.norm = nn.LayerNorm(d_model)\n\n self.concatenation_layer = nn.Linear( * d_model, d_model)\n self.transformer = nn.TransformerEncoder(\n nn.TransformerEncoderLayer(d_model=d_model, nhead=num_heads, dropout=dropout, activation=nn.GELU(),\n batch_first=),\n num_layers=num_layers\n )\n self.fc = nn.Linear(d_model, num_labels)\n\n ():\n sparse_features = x[:, :self.num_sparse_features]\n target_encodings = x[:, self.num_sparse_features:]\n\n sparse_features = self.sparse_feature_embedding(sparse_features)\n target_encodings = self.target_encoding_embedding(target_encodings)\n\n combined_features = torch.cat((sparse_features, target_encodings), dim=)\n combined_features = self.concatenation_layer(combined_features)\n combined_features = self.norm(combined_features)\n\n x = self.transformer(combined_features)\n x = self.norm(x)\n\n x = self.fc(x)\n x\nI modularized my model into two distinct segments to efficiently handle the complexity of the input data:
\nSparse Feature Encoding:
\nFor encoding sparse features, I utilized an embedding layer tailored to handle the sparsity inherent in certain feature types.
\nConsidering the nature of sparse features, I initially explored utilizing the nn.Embedding layer. However, due to computational constraints on my laptop GPU, I opted for an alternative approach using a linear layer (nn.Linear) to convert sparse features into a dense representation.
\nTarget Encoding Feature Encoding (Dense Features):
Concurrently, I addressed the encoding of target encodings, which are inherently dense features.
\nTo accomplish this, I employed a separate linear layer (nn.Linear) designed specifically for target encodings, ensuring an effective transformation into a meaningful latent space.
\nFollowing these individual encodings, I concatenated the resulting embeddings into a unified latent vector, fostering a comprehensive representation of both sparse and dense feature information. To further enhance the model's capacity to capture nuanced patterns, I employed normalization (nn.LayerNorm) on the concatenated features.
Considering the computational challenges posed by sparse feature embedding using nn.Embedding, I strategically utilized nn.Linear for efficient processing on my laptop GPU.
\nIn addition, I introduced a nuanced approach for encoding dense features by breaking them into four distinct encoding layers, each dedicated to capturing nuanced patterns related to mean and standard deviation for both 'sm_name' and 'cell_type'.
\nFurthermore, the model architecture leveraged the state-of-the-art Lion optimizer, recognized for its efficacy in optimizing transformer-based models. The specific model implementation, as exemplified in the CustomTransformer_v3 class, showcases a transformer architecture with multiple layers, heads, and dropout for optimal learning.
\nIn summary, this modularized and intricately designed model not only optimizes computational efficiency but also demonstrates a keen understanding of the unique characteristics of sparse and dense features, contributing to the overall effectiveness of the learning process.
\nHyperparameters
\nThe model training regimen was characterized by a judicious selection of hyperparameters, featuring an initial learning rate of 1e-5 and a weight decay of 1e-4, the latter serving as an additional regularization mechanism, particularly for dropout layers within the model architecture. This meticulous choice of hyperparameters aimed at striking a balance between effective training and prevention of overfitting.
To dynamically adjust the learning rate during training, a learning rate scheduler of the ReduceLROnPlateau type was employed. The scheduler, configured with a mode of \"min\" and a reduction factor of 0.9999, adeptly adapted the learning rate in response to plateaus in model performance, ensuring an efficient convergence towards optimal results. The patience parameter, set to 500, determined the number of epochs with no improvement before triggering a reduction in the learning rate.
\nThe training process spanned 20,000 epochs, incorporating an early stopping mechanism triggered after 5,000 epochs of stagnation. This comprehensive training strategy underscored a thoughtful balance between fine-tuning the model's parameters and preventing overfitting, contributing to the robustness and efficiency of the learning process.
\nLoss function
\nWhile Mean Absolute Error (MAE) and Mean Squared Error (MSE) individually prove effective for gauging bias or variance, it's crucial to recognize their distinct roles in assessing model performance. To strike a balance between these metrics, I opted for Huber loss, amalgamating the strengths of both MAE and MSE. Despite utilizing the Mrrmse metric, the ultimate model selection hinged on the performance demonstrated by the optimal loss function on the validation set.
\nIn addressing the challenges posed by the limited size of the dataset, a critical consideration was ensuring the model's ability to effectively regularize. To achieve this, I incorporated Dropout layers within the transformer architecture and applied weight decay for L2 penalty on the model weights. Additionally, to mitigate the impact of exploding gradients, I implemented gradient norm clipping with a maximum norm of 1.
\nIn pursuit of an optimal model configuration, I adopted a robust validation strategy by training with different seeds and employing k-fold cross-validation. The diverse set of performance metrics for each fold included:
\nI am thrilled to finally unveil my solution to this competition!
\nMy optimal model emerged as a composite of four models, where three were employed as a normalized sum, and the fourth acted as an amplifier, showcasing discernible impact, potentially on labels with alternating signs:
\nweight_df1: 0.5 (utilizing std, mean, and clustering sampling, yielding 0.551)
weight_df2: 0.25 (excluding uncommon elements, resulting in 0.559)
weight_df3: 0.25 (leveraging clustering sampling, achieving 0.575)
weight_df4: 0.3 (incorporating mean, random sampling, and excluding std, attaining 0.554)
\nThe resultant model is expressed as the weighted sum of these components: resulted_model = weight_df1 * df1 + weight_df2 * df2 + weight_df3 * df3 + weight_df4 * df4. All models were trained using the same transformer architecture, albeit with varying dimensional models (with the optimal dimension being 128).
Data preprocessing and feature selection
\nTo train deep learning models on categorical labels, I employed one-hot encoding transformation. To address high and low bias labels, I utilized target encoding by calculating the mean and standard deviation for each cell type and SM name. During experimentation, I compared models with and without uncommon columns and discovered that including uncommon columns significantly improved the training of encoding layers for mean and standard deviation feature vectors, resulting in noticeable performance enhancement.
\nEDA
In my feature exploration, I performed truncated Singular Value Decomposition (SVD) specifically on the target variables. Despite experimenting with different SVD sizes, the models generated from this approach consistently fell short of replicating the performance observed in the full targets regression. Notably, as part of this analysis, I identified certain targets with significant standard deviation values.
\nTo leverage this insight, I strategically enriched the feature set by introducing the standard deviation as a dedicated feature for each cell type and SM name. This involved concatenating these standard deviation values into a unified vector format (std_cell_type, std_sm_name).
\nFurthermore, as I delved into the dataset, I conducted a comprehensive examination, revealing a remarkable cleanliness characterized by the absence of NaN values and duplicates. This meticulous exploration not only extended to model training but also encompassed a nuanced analysis of the target variables, contributing to the informed enhancement of the feature set for improved model performance.
\nSampling strategy
\nIn devising an effective sampling strategy for partitioning the data into training and validation sets, I employed a sophisticated approach rooted in the clusters derived from a K-Means clustering analysis on the target values. The rationale behind this strategy lies in the premise that data points within similar clusters share inherent patterns or characteristics, thus ensuring a more nuanced and representative split.
For each distinct cluster identified through K-Means, I executed the data partitioning using the train_test_split function from the scikit-learn library. This method facilitated a controlled allocation of data points to both the training and validation sets, ensuring that the models were exposed to a diverse yet stratified representation of the data.
\nThe validation percentage, a crucial parameter in this process, was meticulously chosen to strike a balance between model robustness and computational efficiency. By experimenting with validation percentages ranging from 0.1 to 0.2, I systematically assessed the impact on model performance. Ultimately, for the optimal model configuration, a validation percentage of 0.1 was deemed most effective. This decision was guided by a careful consideration of the trade-off between the need for a sufficiently large training set and the importance of a robust validation set to gauge model generalizability.
\nIn summary, this sampling strategy, grounded in cluster-based data splitting, not only acknowledges the underlying structure within the target values but also optimizes the partitioning process to enhance the model's ability to capture diverse patterns and generalize effectively to unseen data.
\nModeling
\nThis is my best architecture
\n (nn.Module): \n ():\n (CustomTransformer_v3, self).__init__()\n self.num_target_encodings = * \n self.num_sparse_features = num_features - self.num_target_encodings\n\n self.sparse_feature_embedding = nn.Linear(self.num_sparse_features, d_model)\n self.target_encoding_embedding = nn.Linear(self.num_target_encodings, d_model)\n self.norm = nn.LayerNorm(d_model)\n\n self.concatenation_layer = nn.Linear( * d_model, d_model)\n self.transformer = nn.TransformerEncoder(\n nn.TransformerEncoderLayer(d_model=d_model, nhead=num_heads, dropout=dropout, activation=nn.GELU(),\n batch_first=),\n num_layers=num_layers\n )\n self.fc = nn.Linear(d_model, num_labels)\n\n ():\n sparse_features = x[:, :self.num_sparse_features]\n target_encodings = x[:, self.num_sparse_features:]\n\n sparse_features = self.sparse_feature_embedding(sparse_features)\n target_encodings = self.target_encoding_embedding(target_encodings)\n\n combined_features = torch.cat((sparse_features, target_encodings), dim=)\n combined_features = self.concatenation_layer(combined_features)\n combined_features = self.norm(combined_features)\n\n x = self.transformer(combined_features)\n x = self.norm(x)\n\n x = self.fc(x)\n x\nI modularized my model into two distinct segments to efficiently handle the complexity of the input data:
\nSparse Feature Encoding:
\nFor encoding sparse features, I utilized an embedding layer tailored to handle the sparsity inherent in certain feature types.
\nConsidering the nature of sparse features, I initially explored utilizing the nn.Embedding layer. However, due to computational constraints on my laptop GPU, I opted for an alternative approach using a linear layer (nn.Linear) to convert sparse features into a dense representation.
\nTarget Encoding Feature Encoding (Dense Features):
Concurrently, I addressed the encoding of target encodings, which are inherently dense features.
\nTo accomplish this, I employed a separate linear layer (nn.Linear) designed specifically for target encodings, ensuring an effective transformation into a meaningful latent space.
\nFollowing these individual encodings, I concatenated the resulting embeddings into a unified latent vector, fostering a comprehensive representation of both sparse and dense feature information. To further enhance the model's capacity to capture nuanced patterns, I employed normalization (nn.LayerNorm) on the concatenated features.
Considering the computational challenges posed by sparse feature embedding using nn.Embedding, I strategically utilized nn.Linear for efficient processing on my laptop GPU.
\nIn addition, I introduced a nuanced approach for encoding dense features by breaking them into four distinct encoding layers, each dedicated to capturing nuanced patterns related to mean and standard deviation for both 'sm_name' and 'cell_type'.
\nFurthermore, the model architecture leveraged the state-of-the-art Lion optimizer, recognized for its efficacy in optimizing transformer-based models. The specific model implementation, as exemplified in the CustomTransformer_v3 class, showcases a transformer architecture with multiple layers, heads, and dropout for optimal learning.
\nIn summary, this modularized and intricately designed model not only optimizes computational efficiency but also demonstrates a keen understanding of the unique characteristics of sparse and dense features, contributing to the overall effectiveness of the learning process.
\nHyperparameters
\nThe model training regimen was characterized by a judicious selection of hyperparameters, featuring an initial learning rate of 1e-5 and a weight decay of 1e-4, the latter serving as an additional regularization mechanism, particularly for dropout layers within the model architecture. This meticulous choice of hyperparameters aimed at striking a balance between effective training and prevention of overfitting.
To dynamically adjust the learning rate during training, a learning rate scheduler of the ReduceLROnPlateau type was employed. The scheduler, configured with a mode of \"min\" and a reduction factor of 0.9999, adeptly adapted the learning rate in response to plateaus in model performance, ensuring an efficient convergence towards optimal results. The patience parameter, set to 500, determined the number of epochs with no improvement before triggering a reduction in the learning rate.
\nThe training process spanned 20,000 epochs, incorporating an early stopping mechanism triggered after 5,000 epochs of stagnation. This comprehensive training strategy underscored a thoughtful balance between fine-tuning the model's parameters and preventing overfitting, contributing to the robustness and efficiency of the learning process.
\nLoss function
\nWhile Mean Absolute Error (MAE) and Mean Squared Error (MSE) individually prove effective for gauging bias or variance, it's crucial to recognize their distinct roles in assessing model performance. To strike a balance between these metrics, I opted for Huber loss, amalgamating the strengths of both MAE and MSE. Despite utilizing the Mrrmse metric, the ultimate model selection hinged on the performance demonstrated by the optimal loss function on the validation set.
\nIn addressing the challenges posed by the limited size of the dataset, a critical consideration was ensuring the model's ability to effectively regularize. To achieve this, I incorporated Dropout layers within the transformer architecture and applied weight decay for L2 penalty on the model weights. Additionally, to mitigate the impact of exploding gradients, I implemented gradient norm clipping with a maximum norm of 1.
\nIn pursuit of an optimal model configuration, I adopted a robust validation strategy by training with different seeds and employing k-fold cross-validation. The diverse set of performance metrics for each fold included:
\nCongratulations! @eliork 🎉🎉🎉Thank you very much for sharing. Your work is so creative. But due to my limited knowledge, I still cannot understand some parts well. I am already following your code sharing. Looking forward to your future achievements!
", "rawMarkdown": "Congratulations! @eliork 🎉🎉🎉Thank you very much for sharing. Your work is so creative. But due to my limited knowledge, I still cannot understand some parts well. I am already following your code sharing. Looking forward to your future achievements!", "votes": 6, "replies": [ { "id": 2547275, "postDate": "2023-12-03T11:16:38.170Z", "content": "Hi @songqizhou, I've uploaded the code to the Git repository. Please feel free to explore it and don't hesitate to reach out if you have any questions.
", "rawMarkdown": "Hi @songqizhou, I've uploaded the code to the Git repository. Please feel free to explore it and don't hesitate to reach out if you have any questions.", "votes": 1, "replies": [ { "id": 2548184, "postDate": "2023-12-04T07:37:23.963Z", "content": "Many thanks to you @eliork! Your code is as elegant as poetry!🥰
", "rawMarkdown": "Many thanks to you @eliork! Your code is as elegant as poetry!🥰", "votes": 1 } ] } ] }, { "id": 2545111, "postDate": "2023-12-01T09:55:10.963Z", "content": "congrats !!
\ni thought none of the top solution would be transformer based
Thanks! I tried xgboost and simple mlp but the results were the best for this model.
", "rawMarkdown": "Thanks! I tried xgboost and simple mlp but the results were the best for this model. ", "votes": 2 }, { "id": 2546938, "postDate": "2023-12-03T02:41:45.023Z", "content": "This competition made me realize the power of the \"generative\" NN model, generating 18211 points from chem with 155 ranks and cell with 6 ranks, though the smart feature engineering makes the difference.
", "rawMarkdown": "This competition made me realize the power of the \"generative\" NN model, generating 18211 points from chem with 155 ranks and cell with 6 ranks, though the smart feature engineering makes the difference.", "votes": 1 } ] }, { "id": 2556380, "postDate": "2023-12-10T16:07:52.213Z", "content": "Congratulations with the great result !
\nWould you be so kind to share the submission files of your solutions - for further research analysis ?
Thanks! The files are in my GitHub repository under submissions folder also check seq.py for more info.
", "rawMarkdown": "Thanks! The files are in my GitHub repository under submissions folder also check seq.py for more info. ", "votes": 1 } ] }, { "id": 2549515, "postDate": "2023-12-05T10:05:19.620Z", "content": "I followed you during the competition and I was sure you would be recognized in the private LB. Superb job @eliork ! :)
", "rawMarkdown": "I followed you during the competition and I was sure you would be recognized in the private LB. Superb job @eliork ! :)", "votes": 1 }, { "id": 2547767, "postDate": "2023-12-03T20:05:00.767Z", "content": "Thank you a lot for sharing your approach! And congratulations on the excellent result!
", "rawMarkdown": "Thank you a lot for sharing your approach! And congratulations on the excellent result!", "votes": 1 }, { "id": 2547308, "postDate": "2023-12-03T12:08:35.827Z", "content": "Congratulations on achieving the 2nd place. Thanks for sharing details about CustomTransformer model. Very interesting indeed.
", "rawMarkdown": "Congratulations on achieving the 2nd place. Thanks for sharing details about CustomTransformer model. Very interesting indeed.", "votes": 1 }, { "id": 2546311, "postDate": "2023-12-02T09:42:50.793Z", "content": "Great Efforts Really!
", "rawMarkdown": "Great Efforts Really!", "votes": 1 }, { "id": 2545286, "postDate": "2023-12-01T12:08:27.960Z", "content": "Congratulations and thank you very much for sharing the solution! Seeing your solution is indeed an aha moment; it is unbiased and summarizing both cell and chem features; wow, simple and strong in prediction, it is quite a learning.
", "rawMarkdown": "Congratulations and thank you very much for sharing the solution! Seeing your solution is indeed an aha moment; it is unbiased and summarizing both cell and chem features; wow, simple and strong in prediction, it is quite a learning.", "votes": 2 }, { "id": 2545173, "postDate": "2023-12-01T10:48:32.767Z", "content": "I have been dreaming of seeing your solutions for a long time.
", "rawMarkdown": "I have been dreaming of seeing your solutions for a long time.", "votes": 2, "replies": [ { "id": 2545200, "postDate": "2023-12-01T11:00:59.393Z", "content": "I hope I didn’t disappoint :)
", "rawMarkdown": "I hope I didn’t disappoint :)", "votes": 3 } ] }, { "id": 2545580, "postDate": "2023-12-01T15:24:15.043Z", "content": "Your notebooks are truly commendable! I upvoted all of them. If you get a chance, I'd be grateful for your input on my latest work. Thanks
", "rawMarkdown": "Your notebooks are truly commendable! I upvoted all of them. If you get a chance, I'd be grateful for your input on my latest work. Thanks" }, { "id": 2608084, "postDate": "2024-01-18T14:56:00.903Z", "content": "congrats !!
\nI‘m confused about the discription : [My optimal model emerged as a composite of four models], aren't these models trained seperately?
\nHow can we conbine them together? Or just compute the weighted sum of their separate results?
\nThanks a lot for your contributions!
", "rawMarkdown": "congrats !!\n\nI‘m confused about the discription : [My optimal model emerged as a composite of four models], aren't these models trained seperately? \n\nHow can we conbine them together? Or just compute the weighted sum of their separate results?\n\nThanks a lot for your contributions!", "replies": [ { "id": 2608192, "postDate": "2024-01-18T15:46:33.143Z", "content": "The models were trained separately, with the same general architecture but with different features (mean,std,etc), I just ensembled the results using weighted sum, I hope it clarifies my approach.
", "rawMarkdown": "The models were trained separately, with the same general architecture but with different features (mean,std,etc), I just ensembled the results using weighted sum, I hope it clarifies my approach. ", "replies": [ { "id": 2608810, "postDate": "2024-01-19T05:41:58.703Z", "content": "Thanks for your reply!
\nI've checked the seq.py，but I still have some questions about the specific features of weight_df.
\nMy understands are as follows, please correct me if I'm wrong:
\nweight_df1: Utilizing std, mean, and clustering sampling
weight_df2: Same as weight_df1 but excluding uncommon elements.
weight_df3: I'm not quite understand how to ' leveraging ' clustering sampling, are you using std and mean both here?
\nWhat are the differences between Weight_ df1 and weight_df3 ?
weight_df4: Incorporating mean, random sampling, and excluding std
Thanks again for your reply!
", "rawMarkdown": "Thanks for your reply!\n\nI've checked the seq.py，but I still have some questions about the specific features of weight_df.\n\nMy understands are as follows, please correct me if I'm wrong:\n\n- **weight_df1**: Utilizing std, mean, and clustering sampling\n\n- **weight_df2**: Same as weight_df1 but excluding uncommon elements.\n\n- **weight_df3**: I'm not quite understand how to ' **leveraging** ' clustering sampling, are you using std and mean both here? \nWhat are the differences between Weight_ df1 and weight_df3 ?\n\n- **weight_df4**: Incorporating mean, random sampling, and excluding std\n\nThanks again for your reply!" }, { "id": 2609034, "postDate": "2024-01-19T08:10:14.097Z", "content": "The difference between df1 to df3 is the sampling method.
\nweight_df3: I clustered all of the data using k-means, and I sampled from each cluster instead of random sampling.
This is my brief summary on the work flow of the solution (correct me if I'm wrong🤣):
\ntrain_data and turned the train_data shape from (614,152) to (614,72996), y shape to (614,18211).I employed K-means to create samples from each cluster. A simple analogy is when dealing with significantly imbalanced data, it's crucial to ensure a representation of categories in both the training and validation sets. The optimal model was attained without truncated SVD. However, if you choose to apply this transformation, remember to also perform the inverse transform.
", "rawMarkdown": "I employed K-means to create samples from each cluster. A simple analogy is when dealing with significantly imbalanced data, it's crucial to ensure a representation of categories in both the training and validation sets. The optimal model was attained without truncated SVD. However, if you choose to apply this transformation, remember to also perform the inverse transform.", "votes": 1 } ] }, { "id": 2546667, "postDate": "2023-12-02T17:19:51.950Z", "content": "Can you give me an example of when you utilized the Lion-Python optimizer?
\nI have been waiting for the model.
\n\n", "rawMarkdown": "Can you give me an example of when you utilized the Lion-Python optimizer?\n\nI have been waiting for the model.\n>Deep learning models for the Kaggle's Open Problems – Single-Cell Perturbations competition. Coming soon. \n>https://github.com/Eliorkalfon/single_cell_pb)", "replies": [ { "id": 2547278, "postDate": "2023-12-03T11:17:31.653Z", "content": "Deep learning models for the Kaggle's Open Problems – Single-Cell Perturbations competition. Coming soon.
\n
\n https://github.com/Eliorkalfon/single_cell_pb)
Hi, I've uploaded the code to the Git repository, check it out :)
", "rawMarkdown": "Hi, I've uploaded the code to the Git repository, check it out :)", "votes": 1 } ] }, { "id": 2559121, "postDate": "2023-12-12T16:07:24.337Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2548329, "postDate": "2023-12-04T10:33:06.630Z", "content": "Congrats! Thanks for sharing
", "rawMarkdown": "Congrats! Thanks for sharing ", "votes": 1 } ], "comments": [ { "id": 2545964, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-12-02T01:32:45.897000", "content": "Congratulations! @eliork 🎉🎉🎉Thank you very much for sharing. Your work is so creative. But due to my limited knowledge, I still cannot understand some parts well. I am already following your code sharing. Looking forward to your future achievements!
", "votes": 6, "replies": [ { "id": 2547275, "author_name": "EliKal", "author_url": "", "post_date": "2023-12-03T11:16:38.170000", "content": "Hi @songqizhou, I've uploaded the code to the Git repository. Please feel free to explore it and don't hesitate to reach out if you have any questions.
", "votes": 1, "replies": [ { "id": 2548184, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-12-04T07:37:23.963000", "content": "Many thanks to you @eliork! Your code is as elegant as poetry!🥰
", "votes": 1, "replies": [] } ] } ] }, { "id": 2545111, "author_name": "Séraphin Lampion", "author_url": "", "post_date": "2023-12-01T09:55:10.963000", "content": "congrats !!
\ni thought none of the top solution would be transformer based
Thanks! I tried xgboost and simple mlp but the results were the best for this model.
", "votes": 2, "replies": [] }, { "id": 2546938, "author_name": "makio323", "author_url": "", "post_date": "2023-12-03T02:41:45.023000", "content": "This competition made me realize the power of the \"generative\" NN model, generating 18211 points from chem with 155 ranks and cell with 6 ranks, though the smart feature engineering makes the difference.
", "votes": 1, "replies": [] } ] }, { "id": 2556380, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2023-12-10T16:07:52.213000", "content": "Congratulations with the great result !
\nWould you be so kind to share the submission files of your solutions - for further research analysis ?
Thanks! The files are in my GitHub repository under submissions folder also check seq.py for more info.
", "votes": 1, "replies": [] } ] }, { "id": 2549515, "author_name": "Octavi Grau", "author_url": "", "post_date": "2023-12-05T10:05:19.620000", "content": "I followed you during the competition and I was sure you would be recognized in the private LB. Superb job @eliork ! :)
", "votes": 1, "replies": [] }, { "id": 2547767, "author_name": "Aya", "author_url": "", "post_date": "2023-12-03T20:05:00.767000", "content": "Thank you a lot for sharing your approach! And congratulations on the excellent result!
", "votes": 1, "replies": [] }, { "id": 2547308, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2023-12-03T12:08:35.827000", "content": "Congratulations on achieving the 2nd place. Thanks for sharing details about CustomTransformer model. Very interesting indeed.
", "votes": 1, "replies": [] }, { "id": 2546311, "author_name": "Abhas Malguri", "author_url": "", "post_date": "2023-12-02T09:42:50.793000", "content": "Great Efforts Really!
", "votes": 1, "replies": [] }, { "id": 2545286, "author_name": "makio323", "author_url": "", "post_date": "2023-12-01T12:08:27.960000", "content": "Congratulations and thank you very much for sharing the solution! Seeing your solution is indeed an aha moment; it is unbiased and summarizing both cell and chem features; wow, simple and strong in prediction, it is quite a learning.
", "votes": 2, "replies": [] }, { "id": 2545173, "author_name": "", "author_url": "", "post_date": "2023-12-01T10:48:32.767000", "content": "I have been dreaming of seeing your solutions for a long time.
", "votes": 2, "replies": [ { "id": 2545200, "author_name": "EliKal", "author_url": "", "post_date": "2023-12-01T11:00:59.393000", "content": "I hope I didn’t disappoint :)
", "votes": 3, "replies": [] } ] }, { "id": 2545580, "author_name": "VIKRAM MISHRA", "author_url": "", "post_date": "2023-12-01T15:24:15.043000", "content": "Your notebooks are truly commendable! I upvoted all of them. If you get a chance, I'd be grateful for your input on my latest work. Thanks
", "votes": 0, "replies": [] }, { "id": 2608084, "author_name": "yu3jun", "author_url": "", "post_date": "2024-01-18T14:56:00.903000", "content": "congrats !!
\nI‘m confused about the discription : [My optimal model emerged as a composite of four models], aren't these models trained seperately?
\nHow can we conbine them together? Or just compute the weighted sum of their separate results?
\nThanks a lot for your contributions!
", "votes": 0, "replies": [ { "id": 2608192, "author_name": "EliKal", "author_url": "", "post_date": "2024-01-18T15:46:33.143000", "content": "The models were trained separately, with the same general architecture but with different features (mean,std,etc), I just ensembled the results using weighted sum, I hope it clarifies my approach.
", "votes": 0, "replies": [ { "id": 2608810, "author_name": "yu3jun", "author_url": "", "post_date": "2024-01-19T05:41:58.703000", "content": "Thanks for your reply!
\nI've checked the seq.py，but I still have some questions about the specific features of weight_df.
\nMy understands are as follows, please correct me if I'm wrong:
\nweight_df1: Utilizing std, mean, and clustering sampling
weight_df2: Same as weight_df1 but excluding uncommon elements.
weight_df3: I'm not quite understand how to ' leveraging ' clustering sampling, are you using std and mean both here?
\nWhat are the differences between Weight_ df1 and weight_df3 ?
weight_df4: Incorporating mean, random sampling, and excluding std
Thanks again for your reply!
", "votes": 0, "replies": [] }, { "id": 2609034, "author_name": "EliKal", "author_url": "", "post_date": "2024-01-19T08:10:14.097000", "content": "The difference between df1 to df3 is the sampling method.
\nweight_df3: I clustered all of the data using k-means, and I sampled from each cluster instead of random sampling.
This is my brief summary on the work flow of the solution (correct me if I'm wrong🤣):
\ntrain_data and turned the train_data shape from (614,152) to (614,72996), y shape to (614,18211).I employed K-means to create samples from each cluster. A simple analogy is when dealing with significantly imbalanced data, it's crucial to ensure a representation of categories in both the training and validation sets. The optimal model was attained without truncated SVD. However, if you choose to apply this transformation, remember to also perform the inverse transform.
", "votes": 1, "replies": [] } ] }, { "id": 2546667, "author_name": "", "author_url": "", "post_date": "2023-12-02T17:19:51.950000", "content": "Can you give me an example of when you utilized the Lion-Python optimizer?
\nI have been waiting for the model.
\n\n", "votes": 0, "replies": [ { "id": 2547278, "author_name": "EliKal", "author_url": "", "post_date": "2023-12-03T11:17:31.653000", "content": "Deep learning models for the Kaggle's Open Problems – Single-Cell Perturbations competition. Coming soon.
\n
\n https://github.com/Eliorkalfon/single_cell_pb)
Hi, I've uploaded the code to the Git repository, check it out :)
", "votes": 1, "replies": [] } ] }, { "id": 2559121, "author_name": "", "author_url": "", "post_date": "2023-12-12T16:07:24.337000", "content": "", "votes": 1, "replies": [] }, { "id": 2548329, "author_name": "hannah142", "author_url": "", "post_date": "2023-12-04T10:33:06.630000", "content": "Congrats! Thanks for sharing
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2545091": "I am thrilled to finally unveil my solution to this competition!\n## Context\n- https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/overview\n- https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/data\n## Overview of the approach\nMy optimal model emerged as a composite of four models, where three were employed as a normalized sum, and the fourth acted as an amplifier, showcasing discernible impact, potentially on labels with alternating signs:\n\n- weight_df1: 0.5 (utilizing std, mean, and clustering sampling, yielding 0.551)\n- weight_df2: 0.25 (excluding uncommon elements, resulting in 0.559)\n- weight_df3: 0.25 (leveraging clustering sampling, achieving 0.575)\n\n- weight_df4: 0.3 (incorporating mean, random sampling, and excluding std, attaining 0.554)\nThe resultant model is expressed as the weighted sum of these components: resulted_model = weight_df1 * df1 + weight_df2 * df2 + weight_df3 * df3 + weight_df4 * df4. All models were trained using the same transformer architecture, albeit with varying dimensional models (with the optimal dimension being 128).\n\n**Data preprocessing and feature selection**\nTo train deep learning models on categorical labels, I employed one-hot encoding transformation. To address high and low bias labels, I utilized target encoding by calculating the mean and standard deviation for each cell type and SM name. During experimentation, I compared models with and without uncommon columns and discovered that including uncommon columns significantly improved the training of encoding layers for mean and standard deviation feature vectors, resulting in noticeable performance enhancement.\n**EDA**\n\nIn my feature exploration, I performed truncated Singular Value Decomposition (SVD) specifically on the target variables. Despite experimenting with different SVD sizes, the models generated from this approach consistently fell short of replicating the performance observed in the full targets regression. Notably, as part of this analysis, I identified certain targets with significant standard deviation values.\n\nTo leverage this insight, I strategically enriched the feature set by introducing the standard deviation as a dedicated feature for each cell type and SM name. This involved concatenating these standard deviation values into a unified vector format (std_cell_type, std_sm_name).\n\nFurthermore, as I delved into the dataset, I conducted a comprehensive examination, revealing a remarkable cleanliness characterized by the absence of NaN values and duplicates. This meticulous exploration not only extended to model training but also encompassed a nuanced analysis of the target variables, contributing to the informed enhancement of the feature set for improved model performance.\n\n\n**Sampling strategy**\nIn devising an effective sampling strategy for partitioning the data into training and validation sets, I employed a sophisticated approach rooted in the clusters derived from a K-Means clustering analysis on the target values. The rationale behind this strategy lies in the premise that data points within similar clusters share inherent patterns or characteristics, thus ensuring a more nuanced and representative split.\n\nFor each distinct cluster identified through K-Means, I executed the data partitioning using the train_test_split function from the scikit-learn library. This method facilitated a controlled allocation of data points to both the training and validation sets, ensuring that the models were exposed to a diverse yet stratified representation of the data.\n\nThe validation percentage, a crucial parameter in this process, was meticulously chosen to strike a balance between model robustness and computational efficiency. By experimenting with validation percentages ranging from 0.1 to 0.2, I systematically assessed the impact on model performance. Ultimately, for the optimal model configuration, a validation percentage of 0.1 was deemed most effective. This decision was guided by a careful consideration of the trade-off between the need for a sufficiently large training set and the importance of a robust validation set to gauge model generalizability.\n\nIn summary, this sampling strategy, grounded in cluster-based data splitting, not only acknowledges the underlying structure within the target values but also optimizes the partitioning process to enhance the model's ability to capture diverse patterns and generalize effectively to unseen data.\n\n**Modeling**\n\nThis is my best architecture\n```python\n class CustomTransformer_v3(nn.Module): # mean + std\n def __init__(self, num_features, num_labels, d_model=128, num_heads=8, num_layers=6, dropout=0.3):\n super(CustomTransformer_v3, self).__init__()\n self.num_target_encodings = 18211 * 4\n self.num_sparse_features = num_features - self.num_target_encodings\n\n self.sparse_feature_embedding = nn.Linear(self.num_sparse_features, d_model)\n self.target_encoding_embedding = nn.Linear(self.num_target_encodings, d_model)\n self.norm = nn.LayerNorm(d_model)\n\n self.concatenation_layer = nn.Linear(2 * d_model, d_model)\n self.transformer = nn.TransformerEncoder(\n nn.TransformerEncoderLayer(d_model=d_model, nhead=num_heads, dropout=dropout, activation=nn.GELU(),\n batch_first=True),\n num_layers=num_layers\n )\n self.fc = nn.Linear(d_model, num_labels)\n\n def forward(self, x):\n sparse_features = x[:, :self.num_sparse_features]\n target_encodings = x[:, self.num_sparse_features:]\n\n sparse_features = self.sparse_feature_embedding(sparse_features)\n target_encodings = self.target_encoding_embedding(target_encodings)\n\n combined_features = torch.cat((sparse_features, target_encodings), dim=1)\n combined_features = self.concatenation_layer(combined_features)\n combined_features = self.norm(combined_features)\n\n x = self.transformer(combined_features)\n x = self.norm(x)\n\n x = self.fc(x)\n return x\n\n```\nI modularized my model into two distinct segments to efficiently handle the complexity of the input data:\n\nSparse Feature Encoding:\n\nFor encoding sparse features, I utilized an embedding layer tailored to handle the sparsity inherent in certain feature types.\nConsidering the nature of sparse features, I initially explored utilizing the nn.Embedding layer. However, due to computational constraints on my laptop GPU, I opted for an alternative approach using a linear layer (nn.Linear) to convert sparse features into a dense representation.\nTarget Encoding Feature Encoding (Dense Features):\n\nConcurrently, I addressed the encoding of target encodings, which are inherently dense features.\nTo accomplish this, I employed a separate linear layer (nn.Linear) designed specifically for target encodings, ensuring an effective transformation into a meaningful latent space.\nFollowing these individual encodings, I concatenated the resulting embeddings into a unified latent vector, fostering a comprehensive representation of both sparse and dense feature information. To further enhance the model's capacity to capture nuanced patterns, I employed normalization (nn.LayerNorm) on the concatenated features.\n\nConsidering the computational challenges posed by sparse feature embedding using nn.Embedding, I strategically utilized nn.Linear for efficient processing on my laptop GPU.\n\nIn addition, I introduced a nuanced approach for encoding dense features by breaking them into four distinct encoding layers, each dedicated to capturing nuanced patterns related to mean and standard deviation for both 'sm_name' and 'cell_type'.\n\nFurthermore, the model architecture leveraged the state-of-the-art Lion optimizer, recognized for its efficacy in optimizing transformer-based models. The specific model implementation, as exemplified in the CustomTransformer_v3 class, showcases a transformer architecture with multiple layers, heads, and dropout for optimal learning.\n\nIn summary, this modularized and intricately designed model not only optimizes computational efficiency but also demonstrates a keen understanding of the unique characteristics of sparse and dense features, contributing to the overall effectiveness of the learning process.\n\n\n**Hyperparameters**\nThe model training regimen was characterized by a judicious selection of hyperparameters, featuring an initial learning rate of 1e-5 and a weight decay of 1e-4, the latter serving as an additional regularization mechanism, particularly for dropout layers within the model architecture. This meticulous choice of hyperparameters aimed at striking a balance between effective training and prevention of overfitting.\n\nTo dynamically adjust the learning rate during training, a learning rate scheduler of the ReduceLROnPlateau type was employed. The scheduler, configured with a mode of \"min\" and a reduction factor of 0.9999, adeptly adapted the learning rate in response to plateaus in model performance, ensuring an efficient convergence towards optimal results. The patience parameter, set to 500, determined the number of epochs with no improvement before triggering a reduction in the learning rate.\n\nThe training process spanned 20,000 epochs, incorporating an early stopping mechanism triggered after 5,000 epochs of stagnation. This comprehensive training strategy underscored a thoughtful balance between fine-tuning the model's parameters and preventing overfitting, contributing to the robustness and efficiency of the learning process.\n\n**Loss function**\n\nWhile Mean Absolute Error (MAE) and Mean Squared Error (MSE) individually prove effective for gauging bias or variance, it's crucial to recognize their distinct roles in assessing model performance. To strike a balance between these metrics, I opted for Huber loss, amalgamating the strengths of both MAE and MSE. Despite utilizing the Mrrmse metric, the ultimate model selection hinged on the performance demonstrated by the optimal loss function on the validation set.\n\n## Preventing overfitting\nIn addressing the challenges posed by the limited size of the dataset, a critical consideration was ensuring the model's ability to effectively regularize. To achieve this, I incorporated Dropout layers within the transformer architecture and applied weight decay for L2 penalty on the model weights. Additionally, to mitigate the impact of exploding gradients, I implemented gradient norm clipping with a maximum norm of 1.\n\n## Validation Strategy\nIn pursuit of an optimal model configuration, I adopted a robust validation strategy by training with different seeds and employing k-fold cross-validation. The diverse set of performance metrics for each fold included:\n\n- Achieving a score of 0.551 through the incorporation of standard deviation, mean, and clustering sampling.\n- Attaining a score of 0.559 by excluding uncommon elements from the dataset.\n- Realizing a score of 0.575 through clustering sampling.\n- Securing a score of 0.554 by integrating mean, random sampling, and excluding standard deviation from the features.\nThis validation approach not only facilitated the identification of the best-performing model but also guided the fine-tuning of hyperparameters to optimize overall model performance.\n## Sources\n- https://github.com/lucidrains/lion-pytorch\n- https://www.kaggle.com/code/ayushs9020/understanding-the-competition-open-problems\n- https://www.kaggle.com/code/alexandervc/op2-eda-baseline-s\n- https://github.com/Eliorkalfon/single_cell_pb\n", "2545964": "Congratulations! @eliork 🎉🎉🎉Thank you very much for sharing. Your work is so creative. But due to my limited knowledge, I still cannot understand some parts well. I am already following your code sharing. Looking forward to your future achievements!", "2545111": "congrats !! \ni thought none of the top solution would be transformer based\n", "2556380": "Congratulations with the great result ! \nWould you be so kind to share the submission files of your solutions - for further research analysis ? \n", "2549515": "I followed you during the competition and I was sure you would be recognized in the private LB. Superb job @eliork ! :)", "2547767": "Thank you a lot for sharing your approach! And congratulations on the excellent result!", "2547308": "Congratulations on achieving the 2nd place. Thanks for sharing details about CustomTransformer model. Very interesting indeed.", "2546311": "Great Efforts Really!", "2545286": "Congratulations and thank you very much for sharing the solution! Seeing your solution is indeed an aha moment; it is unbiased and summarizing both cell and chem features; wow, simple and strong in prediction, it is quite a learning.", "2545173": "I have been dreaming of seeing your solutions for a long time.", "2545580": "Your notebooks are truly commendable! I upvoted all of them. If you get a chance, I'd be grateful for your input on my latest work. Thanks", "2608084": "congrats !!\n\nI‘m confused about the discription : [My optimal model emerged as a composite of four models], aren't these models trained seperately? \n\nHow can we conbine them together? Or just compute the weighted sum of their separate results?\n\nThanks a lot for your contributions!", "2549326": "This is my brief summary on the work flow of the solution (correct me if I'm wrong🤣):\n- data augmentation: utilized std and mean of the ``train_data`` and turned the train_data shape from ``(614,152)`` to ``(614,72996)``, y shape to ``(614,18211)``.\n- then utilize K-means to clean the train data?\n- then (check the model) train the transformer model (with 2 sperated MLP for sparse and dense features then a concatenate) with Lion and HuberLoss, output a rather high dim predictions.\n- predict the test data using truancatedSVD to down sample the predictions to submission format.\n\n", "2546667": "Can you give me an example of when you utilized the Lion-Python optimizer?\n\nI have been waiting for the model.\n>Deep learning models for the Kaggle's Open Problems – Single-Cell Perturbations competition. Coming soon. \n>https://github.com/Eliorkalfon/single_cell_pb)", "2559121": "", "2548329": "Congrats! Thanks for sharing " } }