Our approach is principally based on trial and error. Our optimal model is derived from several different models, of which the first model, RAPIDS SVR, was used to provide pseudo-labels but did not participate in the final ensembling:
\n((0.3*Pyboost + 0.7*nn)*0.9 + open-source solution 0720*0.1)*0.95 + open-source solution 0531*0.05Different models employed different feature engineering strategies, which we will detail below.
\nWe believe that the method of cross-validation is a very important point for scoring improvement in this competition. A reasonable validation set allows us to trust our local CV instead of an overfitted LB.
\nInitially, we used random k-fold cross-validation. This method's CV maintained a consistent trend with the LB in the early stages. However, as we further incorporated nn models and added more training strategies, we noticed a discrepancy between the CV and LB. Consequently, we experimented with public two other forms of cross-validation which come from AmbrosM and MT. Thanks for their sharing. Ultimately, we chose MT's method of cross-validation, which showed excellent consistency between CV and LB in our models.
The first model we experimented with was RAPIDS SVR. It is frequently used in Kaggle competitions and has been part of multiple winning solutions. It is renowned for its fast training capabilities.
\nThe feature engineering for the RAPIDS SVR model primarily included generating embedding features from ChemBERTa-10M-MTR for SMILES and statistical features obtained by aggregating target data. During the competition, we noticed in the discussion forums that many mentioned the embedding features generated by ChemBERTa-10M-MTR did not yield positive results. In our trials, we found that these features had a negative impact on models like LGBM and CatBoost, but they improved performance in RAPIDS SVR. Therefore, we decided to use RAPIDS SVR to generate pseudo-labels as a basis for subsequent model training.
\nHere are the features we used in our RAPIDS SVR model:
\ndef mean_pooling(model_output, attention_mask):\n token_embeddings = model_output.last_hidden_state.detach().cpu()\n input_mask_expanded = (\n attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()\n )\n return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(\n input_mask_expanded.sum(1), =1e-9\n )\n\nclass EmbedDataset(torch.utils.data.Dataset):\n def __init__(self,df):\n self.df = df.reset_index(=)\n def __len__(self):\n return len(self.df)\n def __getitem__(self,idx):\n text = self.df.loc[idx,]\n tokens = tokenizer(\n text,\n None,\n =,\n =,\n =,\n =150,\n =)\n tokens = {k:v.squeeze(0) k,v tokens.items()}\n return tokens\n\n\ndef get_embeddings(de_train, =, =150, =32, =):\n global tokenizer\n # Extract unique texts\n unique_texts = de_train[].unique()\n\n # Create a dataset unique texts\n ds_unique = EmbedDataset(pd.DataFrame(unique_texts, columns=[]))\n embed_dataloader_unique = torch.utils.data.DataLoader(ds_unique, =BATCH_SIZE, =)\n\n =\n model = AutoModel.from_pretrained( MODEL_NM )\n tokenizer = AutoTokenizer.from_pretrained( MODEL_NM )\n\n model = model.(DEVICE)\n model.eval()\n unique_emb = []\n batch tqdm(embed_dataloader_unique,=len(embed_dataloader_unique)):\n input_ids = batch[].(DEVICE)\n attention_mask = batch[].(DEVICE)\n with torch.no_grad():\n model_output = model(=input_ids,attention_mask=attention_mask)\n sentence_embeddings = mean_pooling(model_output, attention_mask.detach().cpu())\n # Normalize the embeddings\n sentence_embeddings = F.normalize(sentence_embeddings, =2, =1)\n sentence_embeddings = sentence_embeddings.squeeze(0).detach().cpu().numpy()\n unique_emb.extend(sentence_embeddings)\n unique_emb = np.array(unique_emb)\n verbose:\n (,unique_emb.shape)\n\n text_to_embedding = {text: emb text, emb zip(unique_texts, unique_emb)}\n\n train_emb = np.array([text_to_embedding[text] text de_train[]])\n test_emb = np.array([text_to_embedding[text] text id_map[]])\n\n return train_emb, test_emb\n\nMODEL_NM = \nall_train_text_feats, te_text_feats = get_embeddings(df_de_train, MODEL_NM)\n3.Statistical features for cell_type and sm_name by aggregated from the target column. We retained features like ['mean', 'min', 'max', 'median', 'first', 'quantile_0.4'].
\nInterestingly, when experimenting with different features, we did not train on all targets but selected only the first target, A1BG, for training and validation. This approach of feature selection allowed us to screen all features in just a few minutes, recording the CV scores for different features.
\nOur Pyboost solution is based on an open-source approach from Alexander Chervov. Thanks for your sharing.
\nIn our Pyboost model, we used four types of features:
\nmodel = GradientBoosting(\n \n ,=1000\n ,=0.01\n ,=10\n ,=1\n ,=0.2\n ,=1\n ,=0\n ,=100)\nIn our nn model, we only used leaveoneout encoding features for cell_type and sm_name, as other features caused a decrease in CV scores. We also performed dimensionality reduction on the target data based on TruncatedSVD.
\nOur nn model consisted of a 3-layer 1D convolutional layer + 1 fully connected layer + a non-pretrained ResNet18 network. This structure allowed us to achieve a LB score of around 0.57 based solely on leaveoneout encoding features for cell_type and sm_name, which seems quite tricky.
\nOur initial nn model aimed to convert SMILES expressions into image data using the rdkit.Chem library and then input these images along with other features into the network for training, thus employing the ResNet network for image processing. However, we found that introducing SMILES image data did not improve training results. Despite this, we retained parts of the network structure and ended up with the aforementioned network.
class ResNetRegression(nn.Module):\n def __init__(self, input_size, output_size, =, =32, =16, =0.2):\n super(ResNetRegression, self).__init__()\n self.reshape_size = reshape_size\n\n self.conv1d_layers = nn.Sequential(\n nn.Conv1d(1, num_channels, =3, =1, =1),\n nn.ReLU(),\n nn.Dropout(dropout_rate),\n\n nn.Conv1d(num_channels, num_channels, =3, =1, =1),\n nn.ReLU(),\n nn.Dropout(dropout_rate),\n\n nn.Conv1d(num_channels, num_channels, =3, =1, =1),\n nn.ReLU(),\n nn.Dropout(dropout_rate),\n )\n\n self.fc_layers = nn.Sequential(\n nn.Linear(num_channels * 4 * input_size, self.reshape_size * self.reshape_size),\n )\n\n self.resnet = models.resnet18(=pretrained)\n self.resnet.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), =)\n self.resnet.fc = nn.Linear(self.resnet.fc.in_features, output_size)\n\n def forward(self, x):\n x = x.unsqueeze(1) # Reshape x Conv1d\n x = self.conv1d_layers(x)\n x = x.view(x.size(0), -1) # Flatten the linear layer\n x = self.fc_layers(x)\n x = x.view(x.size(0), 1, self.reshape_size, self.reshape_size)\n x = self.resnet(x)\n return x\nIn our final model, we assigned a weight of 0.05 to the 0720 open-source solution for ensembling. This is the result of a great notebook that uses the \"Autoencoder\" method. This integration resulted in an increase of 0.001 in our scores on both the Public LB and Private LB. We are grateful for the contribution shared by vendekagonlabs and discussion.
\nWe are not enthusiasts of parameter tuning, as, in our experience, tuning does not bring qualitative improvements to the model. In this competition, we only experimented with tuning towards the end, using Optuna to try out different settings for 'n_components' and 'n_iter' in TruncatedSVD, as well as 'sigma' in LeaveOneOutEncoder. Ultimately, we selected a few sets of parameters that yielded the best CV scores.
\nCode
\nhttps://github.com/paralyzed2023/4st-place-solution-single-cell-pbs.git
Our approach is principally based on trial and error. Our optimal model is derived from several different models, of which the first model, RAPIDS SVR, was used to provide pseudo-labels but did not participate in the final ensembling:
\n((0.3*Pyboost + 0.7*nn)*0.9 + open-source solution 0720*0.1)*0.95 + open-source solution 0531*0.05Different models employed different feature engineering strategies, which we will detail below.
\nWe believe that the method of cross-validation is a very important point for scoring improvement in this competition. A reasonable validation set allows us to trust our local CV instead of an overfitted LB.
\nInitially, we used random k-fold cross-validation. This method's CV maintained a consistent trend with the LB in the early stages. However, as we further incorporated nn models and added more training strategies, we noticed a discrepancy between the CV and LB. Consequently, we experimented with public two other forms of cross-validation which come from AmbrosM and MT. Thanks for their sharing. Ultimately, we chose MT's method of cross-validation, which showed excellent consistency between CV and LB in our models.
The first model we experimented with was RAPIDS SVR. It is frequently used in Kaggle competitions and has been part of multiple winning solutions. It is renowned for its fast training capabilities.
\nThe feature engineering for the RAPIDS SVR model primarily included generating embedding features from ChemBERTa-10M-MTR for SMILES and statistical features obtained by aggregating target data. During the competition, we noticed in the discussion forums that many mentioned the embedding features generated by ChemBERTa-10M-MTR did not yield positive results. In our trials, we found that these features had a negative impact on models like LGBM and CatBoost, but they improved performance in RAPIDS SVR. Therefore, we decided to use RAPIDS SVR to generate pseudo-labels as a basis for subsequent model training.
\nHere are the features we used in our RAPIDS SVR model:
\ndef mean_pooling(model_output, attention_mask):\n token_embeddings = model_output.last_hidden_state.detach().cpu()\n input_mask_expanded = (\n attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()\n )\n return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(\n input_mask_expanded.sum(1), =1e-9\n )\n\nclass EmbedDataset(torch.utils.data.Dataset):\n def __init__(self,df):\n self.df = df.reset_index(=)\n def __len__(self):\n return len(self.df)\n def __getitem__(self,idx):\n text = self.df.loc[idx,]\n tokens = tokenizer(\n text,\n None,\n =,\n =,\n =,\n =150,\n =)\n tokens = {k:v.squeeze(0) k,v tokens.items()}\n return tokens\n\n\ndef get_embeddings(de_train, =, =150, =32, =):\n global tokenizer\n # Extract unique texts\n unique_texts = de_train[].unique()\n\n # Create a dataset unique texts\n ds_unique = EmbedDataset(pd.DataFrame(unique_texts, columns=[]))\n embed_dataloader_unique = torch.utils.data.DataLoader(ds_unique, =BATCH_SIZE, =)\n\n =\n model = AutoModel.from_pretrained( MODEL_NM )\n tokenizer = AutoTokenizer.from_pretrained( MODEL_NM )\n\n model = model.(DEVICE)\n model.eval()\n unique_emb = []\n batch tqdm(embed_dataloader_unique,=len(embed_dataloader_unique)):\n input_ids = batch[].(DEVICE)\n attention_mask = batch[].(DEVICE)\n with torch.no_grad():\n model_output = model(=input_ids,attention_mask=attention_mask)\n sentence_embeddings = mean_pooling(model_output, attention_mask.detach().cpu())\n # Normalize the embeddings\n sentence_embeddings = F.normalize(sentence_embeddings, =2, =1)\n sentence_embeddings = sentence_embeddings.squeeze(0).detach().cpu().numpy()\n unique_emb.extend(sentence_embeddings)\n unique_emb = np.array(unique_emb)\n verbose:\n (,unique_emb.shape)\n\n text_to_embedding = {text: emb text, emb zip(unique_texts, unique_emb)}\n\n train_emb = np.array([text_to_embedding[text] text de_train[]])\n test_emb = np.array([text_to_embedding[text] text id_map[]])\n\n return train_emb, test_emb\n\nMODEL_NM = \nall_train_text_feats, te_text_feats = get_embeddings(df_de_train, MODEL_NM)\n3.Statistical features for cell_type and sm_name by aggregated from the target column. We retained features like ['mean', 'min', 'max', 'median', 'first', 'quantile_0.4'].
\nInterestingly, when experimenting with different features, we did not train on all targets but selected only the first target, A1BG, for training and validation. This approach of feature selection allowed us to screen all features in just a few minutes, recording the CV scores for different features.
\nOur Pyboost solution is based on an open-source approach from Alexander Chervov. Thanks for your sharing.
\nIn our Pyboost model, we used four types of features:
\nmodel = GradientBoosting(\n \n ,=1000\n ,=0.01\n ,=10\n ,=1\n ,=0.2\n ,=1\n ,=0\n ,=100)\nIn our nn model, we only used leaveoneout encoding features for cell_type and sm_name, as other features caused a decrease in CV scores. We also performed dimensionality reduction on the target data based on TruncatedSVD.
\nOur nn model consisted of a 3-layer 1D convolutional layer + 1 fully connected layer + a non-pretrained ResNet18 network. This structure allowed us to achieve a LB score of around 0.57 based solely on leaveoneout encoding features for cell_type and sm_name, which seems quite tricky.
\nOur initial nn model aimed to convert SMILES expressions into image data using the rdkit.Chem library and then input these images along with other features into the network for training, thus employing the ResNet network for image processing. However, we found that introducing SMILES image data did not improve training results. Despite this, we retained parts of the network structure and ended up with the aforementioned network.
class ResNetRegression(nn.Module):\n def __init__(self, input_size, output_size, =, =32, =16, =0.2):\n super(ResNetRegression, self).__init__()\n self.reshape_size = reshape_size\n\n self.conv1d_layers = nn.Sequential(\n nn.Conv1d(1, num_channels, =3, =1, =1),\n nn.ReLU(),\n nn.Dropout(dropout_rate),\n\n nn.Conv1d(num_channels, num_channels, =3, =1, =1),\n nn.ReLU(),\n nn.Dropout(dropout_rate),\n\n nn.Conv1d(num_channels, num_channels, =3, =1, =1),\n nn.ReLU(),\n nn.Dropout(dropout_rate),\n )\n\n self.fc_layers = nn.Sequential(\n nn.Linear(num_channels * 4 * input_size, self.reshape_size * self.reshape_size),\n )\n\n self.resnet = models.resnet18(=pretrained)\n self.resnet.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), =)\n self.resnet.fc = nn.Linear(self.resnet.fc.in_features, output_size)\n\n def forward(self, x):\n x = x.unsqueeze(1) # Reshape x Conv1d\n x = self.conv1d_layers(x)\n x = x.view(x.size(0), -1) # Flatten the linear layer\n x = self.fc_layers(x)\n x = x.view(x.size(0), 1, self.reshape_size, self.reshape_size)\n x = self.resnet(x)\n return x\nIn our final model, we assigned a weight of 0.05 to the 0720 open-source solution for ensembling. This is the result of a great notebook that uses the \"Autoencoder\" method. This integration resulted in an increase of 0.001 in our scores on both the Public LB and Private LB. We are grateful for the contribution shared by vendekagonlabs and discussion.
\nWe are not enthusiasts of parameter tuning, as, in our experience, tuning does not bring qualitative improvements to the model. In this competition, we only experimented with tuning towards the end, using Optuna to try out different settings for 'n_components' and 'n_iter' in TruncatedSVD, as well as 'sigma' in LeaveOneOutEncoder. Ultimately, we selected a few sets of parameters that yielded the best CV scores.
\nCode
\nhttps://github.com/paralyzed2023/4st-place-solution-single-cell-pbs.git
I'm scrolling down top solutions and let me tel you: I would never expect this competition be won with ResNet18
\n🤩
I'm scrolling down top solutions and let me tel you: I would never expect this competition be won with ResNet18
\n🤩