{ "id": 320026, "title": "DOLG Army: 7th Place Solution Summary", "url": "/competitions/happy-whale-and-dolphin/writeups/dolg-army-dolg-army-7th-place-solution-summary", "author_name": "", "post_date": "2022-04-21T03:22:58.023Z", "votes": 32, "comment_count": 9, "views": 0, "content": "

Hi all,

\n

Congratulations to all the winners. And thank you to kaggle host and all participants for this exciting competition.It was as always very tough last week with huge LB changing everybody , fortunately for us , our ideas worked and we ended on right side of the private LB.

\n

Congratulations to @ks2019 for becoming Kaggle Competitions GM. Thanks a lot to my teammates @nischaydnk @tanulsingh077 @navjotbansal. Such a great team effort 🤜.

\n

Our solution consists of the following major components:

\n
    \n
  1. Data Recipe
    2. \n
  2. Modelling
    3. \n
  3. Progressive Psuedo Labelling
    4. \n
  4. PostProcessing
    5. \n
  5. Ensemble
    6. \n
\n

All our models are trained on TPUs using more or less the same tensorflow pipeline that KS shared in the beginning of the competition.

\n

Data Recipe

\n

There were different datasets available publically, which we thought could help us in diversity. Inspired from @thedrcat 's solution in Chaii, we decided to make our own data recipe. We used the following combination of datasets to train our models:

\n\n

Fullbody Annotations
\n\"fullbody\"

\n

Fullbody Annotations + Backfins concatenated horizontally
\n\"fullbody-backfin\"

\n

Fullbody Annotations + Original Images concatenated horizontally
\n\"fullbody-original\"

\n

We used simple augmentations:

\n\n

Modelling

\n

We used a combination of DOLG (with EFFNet backbone) and normal EFFNets with CurricularFace loss. We used @christofhenkel 's implementation of DOLG model, implemented in PyTorch which we ported to TensorFlow because it worked better than the public implementation in this competition. We also used multiple heads in all the models: one for species classification and another for individual classification. Species classification head was trained with normal softmax loss while the individual classification head was trained with CurricularFace loss.

\n

Models in our final submission:

\n\n

All the models were trained on psuedo labelled data from our best ensemble. During inference we also use hflip as TTA.

\n

PostProcessing

\n

We used a second stage model for getting better confidence scores which is less susceptible to threshold changes. The idea was to use predictions/embeddings/features to get a better confidence score than just using nearest distances. After candidate generation (top100 neighbors based on knn distances) we engineered the following features for our 2nd stage model (features were generated from the models stated above):

\n\n

We trained a 5 folds XGB and LightGBM models on the above features and used a ensemble of their predicitons as final confidence scores.

\n

Ensemble

\n

We used simple weighted voting approach using the confidence scores obtained from above where the weights were optimized using 5 fold OOFs.

\n

Acknowledgements

\n", "messages": [ { "id": "1761241", "postDate": "04/19/2022 18:21:43", "content": "

Hi all,

\n

Congratulations to all the winners. And thank you to kaggle host and all participants for this exciting competition.It was as always very tough last week with huge LB changing everybody , fortunately for us , our ideas worked and we ended on right side of the private LB.

\n

Congratulations to @ks2019 for becoming Kaggle Competitions GM. Thanks a lot to my teammates @nischaydnk @tanulsingh077 @navjotbansal. Such a great team effort 🤜.

\n

Our solution consists of the following major components:

\n
    \n
  1. Data Recipe
    2. \n
  2. Modelling
    3. \n
  3. Progressive Psuedo Labelling
    4. \n
  4. PostProcessing
    5. \n
  5. Ensemble
    6. \n
\n

All our models are trained on TPUs using more or less the same tensorflow pipeline that KS shared in the beginning of the competition.

\n

Data Recipe

\n

There were different datasets available publically, which we thought could help us in diversity. Inspired from @thedrcat 's solution in Chaii, we decided to make our own data recipe. We used the following combination of datasets to train our models:

\n\n

Fullbody Annotations
\n\"fullbody\"

\n

Fullbody Annotations + Backfins concatenated horizontally
\n\"fullbody-backfin\"

\n

Fullbody Annotations + Original Images concatenated horizontally
\n\"fullbody-original\"

\n

We used simple augmentations:

\n\n

Modelling

\n

We used a combination of DOLG (with EFFNet backbone) and normal EFFNets with CurricularFace loss. We used @christofhenkel 's implementation of DOLG model, implemented in PyTorch which we ported to TensorFlow because it worked better than the public implementation in this competition. We also used multiple heads in all the models: one for species classification and another for individual classification. Species classification head was trained with normal softmax loss while the individual classification head was trained with CurricularFace loss.

\n

Models in our final submission:

\n\n

All the models were trained on psuedo labelled data from our best ensemble. During inference we also use hflip as TTA.

\n

PostProcessing

\n

We used a second stage model for getting better confidence scores which is less susceptible to threshold changes. The idea was to use predictions/embeddings/features to get a better confidence score than just using nearest distances. After candidate generation (top100 neighbors based on knn distances) we engineered the following features for our 2nd stage model (features were generated from the models stated above):

\n\n

We trained a 5 folds XGB and LightGBM models on the above features and used a ensemble of their predicitons as final confidence scores.

\n

Ensemble

\n

We used simple weighted voting approach using the confidence scores obtained from above where the weights were optimized using 5 fold OOFs.

\n

Acknowledgements

\n", "rawMarkdown": "Hi all, \n\nCongratulations to all the winners. And thank you to kaggle host and all participants for this exciting competition.It was as always very tough last week with huge LB changing everybody , fortunately for us , our ideas worked and we ended on right side of the private LB.\n\nCongratulations to @ks2019 for becoming Kaggle Competitions GM. Thanks a lot to my teammates @nischaydnk @tanulsingh077 @navjotbansal. Such a great team effort 🤜.\n\nOur solution consists of the following major components:\n1. Data Recipe\n2. Modelling\n3. Progressive Psuedo Labelling\n4. PostProcessing\n5. Ensemble\n\nAll our models are trained on TPUs using more or less the same tensorflow pipeline that KS shared in the beginning of the competition.\n\n## Data Recipe\nThere were different datasets available publically, which we thought could help us in diversity. Inspired from @thedrcat 's solution in Chaii, we decided to make our own data recipe. We used the following combination of datasets to train our models:\n- Fullbody Annotations\n- Fullbody Annotations + Backfins concatenated horizontally \n- Fullbody Annotations + Original Images concatenated horizontally\n\nFullbody Annotations\n\"fullbody\"

\n\nFullbody Annotations + Backfins concatenated horizontally \n\"fullbody-backfin\"

\n\nFullbody Annotations + Original Images concatenated horizontally\n\"fullbody-original\"

\n\nWe used simple augmentations:\n- horizontal flip\n- random pixel based augmentation (brightness, contrast, HSV)\n- cutout\n\n## Modelling\nWe used a combination of DOLG (with EFFNet backbone) and normal EFFNets with CurricularFace loss. We used @christofhenkel 's implementation of DOLG model, [implemented in PyTorch](https://www.kaggle.com/competitions/landmark-retrieval-2021/discussion/277099) which we ported to TensorFlow because it worked better than the public implementation in this competition. We also used multiple heads in all the models: one for species classification and another for individual classification. Species classification head was trained with normal softmax loss while the individual classification head was trained with CurricularFace loss.\n\nModels in our final submission:\n- DOLG B5/B6/B7, image_sizes: (786, 786x2), (896, 896x2), dataset: Fullbody Annotations + Backfins\n- DOLG B6/B7, image_sizes: (896, 896x2), dataset: Fullbody Annotations + Original Images\n- DOLG B5/B6/B7, image_sizes: (1024, 1024), dataset: Fullbody Annotations\n- EFFNet B5/B6/B7, image_sizes: (786, 786x2), (896, 896x2), dataset: Fullbody Annotations + Backfins\n- EFFNet B5/B6/B7, image_sizes: (1024, 1024), dataset: Fullbody Annotations\n\nAll the models were trained on psuedo labelled data from our best ensemble. During inference we also use hflip as TTA.\n\n## PostProcessing\nWe used a second stage model for getting better confidence scores which is less susceptible to threshold changes. The idea was to use predictions/embeddings/features to get a better confidence score than just using nearest distances. After candidate generation (top100 neighbors based on knn distances) we engineered the following features for our 2nd stage model (features were generated from the models stated above):\n- species probabilites for each image_id\n- top3 nearest distances for each (image_id, unique individual_id) pair present in the candidates\n- distance of each image_id from centroid of each unique individual_id present in the candidates\n- rank of each unique individual_id present in the candidates\n- sum of top3 neighbor distances \n- OOF predictions\n\nWe trained a 5 folds XGB and LightGBM models on the above features and used a ensemble of their predicitons as final confidence scores.\n\n## Ensemble\nWe used simple weighted voting approach using the confidence scores obtained from above where the weights were optimized using 5 fold OOFs.\n\n## Acknowledgements\n- Thanks a lot to @jpbremer for sharing the datasets (The real hero 🔥)\n- Special mention to Google TPU Research Program (https://sites.research.google/trc/) which helped us in training bigger models in the last few days of the competition by supporting us with V3 TPUs on GCP.", "votes": null }, { "id": "1761383", "postDate": "04/19/2022 21:10:12", "content": "

Thanks for your sharing. Can you share your source code? I'm really interested in stacking part

", "rawMarkdown": "Thanks for your sharing. Can you share your source code? I'm really interested in stacking part", "votes": null }, { "id": "1761429", "postDate": "04/19/2022 22:48:02", "content": "

Good job, I am also interested in PostProcessing part.

", "rawMarkdown": "Good job, I am also interested in PostProcessing part.", "votes": null }, { "id": "1763110", "postDate": "04/21/2022 09:25:12", "content": "

Hi, sorry for late reply. Taking some time off from kaggle after intense competition. We'll make the complete source code public in few days.

", "rawMarkdown": "Hi, sorry for late reply. Taking some time off from kaggle after intense competition. We'll make the complete source code public in few days.", "votes": null }, { "id": "1763118", "postDate": "04/21/2022 09:29:39", "content": "

Here is the code for feature engineering part. We trained the LGBM and XGBoost model on these features. Hyperparameters tuned with optuna to optimise competition metric. This gave us ~ 0.01-0.015 boost in public & private lb.

\n
KNN = 100\nnn_cols = [f'nn-{x}' for x in range(KNN)]\ndist_cols = [f'dist-{x}' for x in range(KNN)]\nreciprocals = 1/(np.arange(1,6))\n\ndef generate_preds(row,threshold=0.5):\n\n    nns = list(zip(row[dist_cols],row[nn_cols].values))\n    nns.append((threshold,-1))\n    preds = []\n    for dist,pred in sorted(nns):\n        if pred not in preds:\n            preds.append(pred)\n        if len(preds)==5:\n            break\n    if len(preds)<5:\n        preds = preds+[-1,-1,-1,-1,-1]\n        preds = preds[:5]\n    return np.array(preds)\n\n\ndef map_per_sample(row):\n    return ((row.target==row['preds'])*reciprocals).sum()\n\ndef get_sample_neighbours(data,MODE):\n\n    train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n    allowed_targets = set(train_targets)\n    neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n    neigh.fit(train_embeddings)\n    distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n    targets = train_targets[idxs]\n    ids = train_ids[idxs]\n\n    val_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n    val_predictions['nn_ids'] = train_ids[idxs].reshape(-1)\n    val_predictions['nn_distances'] = distances.reshape(-1)\n    val_predictions['nn_rank'] = val_predictions.index//KNN\n    val_predictions['image'] = val_predictions['nn_rank'].apply(lambda x: val_ids[x])\n    if MODE=='TRAIN':\n        val_predictions['target'] = val_predictions['nn_rank'].apply(lambda x: val_targets[x])\n    else:\n        val_predictions['target'] = -1\n    val_predictions['nn_rank'] = val_predictions.index%KNN\n    return val_predictions\n\ndef get_centroid_neighbours(data):\n\n    train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n    train_embeddings_df = pd.DataFrame(train_embeddings)\n    train_embeddings_df['target'] = train_targets\n    centroid_embeddings = train_embeddings_df.groupby('target').mean()\n    centroid_targets = centroid_embeddings.index.values\n    centroid_embeddings = centroid_embeddings.values\n\n    from sklearn.neighbors import NearestNeighbors\n    neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n    neigh.fit(centroid_embeddings)\n    distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n    targets = centroid_targets[idxs]\n\n    centroid_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n    centroid_predictions['nn_distances'] = distances.reshape(-1)\n    centroid_predictions['nn_rank'] = centroid_predictions.index//KNN\n    centroid_predictions['image'] = centroid_predictions['nn_rank'].apply(lambda x: val_ids[x])\n    centroid_predictions['nn_rank'] = centroid_predictions.index%KNN\n    centroid_predictions = centroid_predictions.set_index(['image','nn_target']).nn_distances.to_dict()\n\n    return centroid_predictions\n\ndef get_target_counts(data):\n    train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n    target_counts = pd.Series(train_targets).value_counts().to_dict()\n    return target_counts\n\ndef engineer_features(train_pattern,test_pattern,fold=0,MODE='TEST'):\n\n    print(\"Loading Data\")\n    val_species_ids,val_species,test_species_ids,test_species = load_species(fold=fold)\n    val_species = {x:y for x,y in zip(val_species_ids,val_species)}\n    test_species = {x:y for x,y in zip(test_species_ids,test_species)}\n\n    if MODE=='TEST':\n        data = load_test_embeds(train_pattern,test_pattern,fold=fold)\n        species = test_species\n    else:\n        data = load_embeds(train_pattern,test_pattern,fold=fold)\n        species = val_species\n\n    spec_df = pd.DataFrame(species).T.reset_index()\n    spec_df.columns = ['image']+[f'spec_{i}' for i in range(0,26)]\n\n    print(\"Getting Sample Neighbours\")\n    val_predictions = get_sample_neighbours(data,MODE)\n\n    print(\"Getting Centroid Neighbours\")\n    centroid_predictions = get_centroid_neighbours(data)\n\n    print(\"Getting Target Counts\")\n    target_counts = get_target_counts(data)\n\n    print(len(species),len(val_predictions))\n\n    train_df = val_predictions.groupby(['image','nn_target']).head(3)\n    train_df['nn_rank_target'] = train_df.groupby(['image','nn_target']).nn_distances.rank(method='first')\n    train_df = pd.pivot_table(train_df, values='nn_distances', index=['image','target', 'nn_target'],\n                        columns=['nn_rank_target'], aggfunc=np.sum).reset_index().fillna(1)\n    train_df = train_df.rename(columns={1:'nn1',2:'nn2',3:'nn3','nn_target':'individual'}).reset_index(drop=True)\n    train_df['centroid'] = train_df.apply(lambda row:centroid_predictions[(row.image,row.individual)]\n                                              if (row.image,row.individual) in centroid_predictions else 1\n                                          ,axis=1)\n    train_df['target_counts'] = train_df.apply(lambda row:target_counts[row.individual],axis=1)\n    train_df['individual_species'] = train_df.individual.map(species_individual_mappings)\n    train_df['species_conf'] = train_df.apply(lambda row:species[row.image][row.individual_species],axis=1)\n#     train_df[[f'species_{i}' for i in range(26)]] = train_df.apply(lambda row: species[row.image],axis=1)\n    return train_df,spec_df\n
", "rawMarkdown": "Here is the code for feature engineering part. We trained the LGBM and XGBoost model on these features. Hyperparameters tuned with optuna to optimise competition metric. This gave us ~ 0.01-0.015 boost in public & private lb.\n\n```\nKNN = 100\nnn_cols = [f'nn-{x}' for x in range(KNN)]\ndist_cols = [f'dist-{x}' for x in range(KNN)]\nreciprocals = 1/(np.arange(1,6))\n\ndef generate_preds(row,threshold=0.5):\n \n nns = list(zip(row[dist_cols],row[nn_cols].values))\n nns.append((threshold,-1))\n preds = []\n for dist,pred in sorted(nns):\n if pred not in preds:\n preds.append(pred)\n if len(preds)==5:\n break\n if len(preds)<5:\n preds = preds+[-1,-1,-1,-1,-1]\n preds = preds[:5]\n return np.array(preds)\n \n \ndef map_per_sample(row):\n return ((row.target==row['preds'])*reciprocals).sum()\n\ndef get_sample_neighbours(data,MODE):\n \n train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n allowed_targets = set(train_targets)\n neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n neigh.fit(train_embeddings)\n distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n targets = train_targets[idxs]\n ids = train_ids[idxs]\n\n val_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n val_predictions['nn_ids'] = train_ids[idxs].reshape(-1)\n val_predictions['nn_distances'] = distances.reshape(-1)\n val_predictions['nn_rank'] = val_predictions.index//KNN\n val_predictions['image'] = val_predictions['nn_rank'].apply(lambda x: val_ids[x])\n if MODE=='TRAIN':\n val_predictions['target'] = val_predictions['nn_rank'].apply(lambda x: val_targets[x])\n else:\n val_predictions['target'] = -1\n val_predictions['nn_rank'] = val_predictions.index%KNN\n return val_predictions\n\ndef get_centroid_neighbours(data):\n \n train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n train_embeddings_df = pd.DataFrame(train_embeddings)\n train_embeddings_df['target'] = train_targets\n centroid_embeddings = train_embeddings_df.groupby('target').mean()\n centroid_targets = centroid_embeddings.index.values\n centroid_embeddings = centroid_embeddings.values\n\n from sklearn.neighbors import NearestNeighbors\n neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n neigh.fit(centroid_embeddings)\n distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n targets = centroid_targets[idxs]\n\n centroid_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n centroid_predictions['nn_distances'] = distances.reshape(-1)\n centroid_predictions['nn_rank'] = centroid_predictions.index//KNN\n centroid_predictions['image'] = centroid_predictions['nn_rank'].apply(lambda x: val_ids[x])\n centroid_predictions['nn_rank'] = centroid_predictions.index%KNN\n centroid_predictions = centroid_predictions.set_index(['image','nn_target']).nn_distances.to_dict()\n \n return centroid_predictions\n\ndef get_target_counts(data):\n train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n target_counts = pd.Series(train_targets).value_counts().to_dict()\n return target_counts\n\ndef engineer_features(train_pattern,test_pattern,fold=0,MODE='TEST'):\n \n print(\"Loading Data\")\n val_species_ids,val_species,test_species_ids,test_species = load_species(fold=fold)\n val_species = {x:y for x,y in zip(val_species_ids,val_species)}\n test_species = {x:y for x,y in zip(test_species_ids,test_species)}\n \n if MODE=='TEST':\n data = load_test_embeds(train_pattern,test_pattern,fold=fold)\n species = test_species\n else:\n data = load_embeds(train_pattern,test_pattern,fold=fold)\n species = val_species\n \n spec_df = pd.DataFrame(species).T.reset_index()\n spec_df.columns = ['image']+[f'spec_{i}' for i in range(0,26)]\n \n print(\"Getting Sample Neighbours\")\n val_predictions = get_sample_neighbours(data,MODE)\n \n print(\"Getting Centroid Neighbours\")\n centroid_predictions = get_centroid_neighbours(data)\n \n print(\"Getting Target Counts\")\n target_counts = get_target_counts(data)\n \n print(len(species),len(val_predictions))\n \n train_df = val_predictions.groupby(['image','nn_target']).head(3)\n train_df['nn_rank_target'] = train_df.groupby(['image','nn_target']).nn_distances.rank(method='first')\n train_df = pd.pivot_table(train_df, values='nn_distances', index=['image','target', 'nn_target'],\n columns=['nn_rank_target'], aggfunc=np.sum).reset_index().fillna(1)\n train_df = train_df.rename(columns={1:'nn1',2:'nn2',3:'nn3','nn_target':'individual'}).reset_index(drop=True)\n train_df['centroid'] = train_df.apply(lambda row:centroid_predictions[(row.image,row.individual)]\n if (row.image,row.individual) in centroid_predictions else 1\n ,axis=1)\n train_df['target_counts'] = train_df.apply(lambda row:target_counts[row.individual],axis=1)\n train_df['individual_species'] = train_df.individual.map(species_individual_mappings)\n train_df['species_conf'] = train_df.apply(lambda row:species[row.image][row.individual_species],axis=1)\n# train_df[[f'species_{i}' for i in range(26)]] = train_df.apply(lambda row: species[row.image],axis=1)\n return train_df,spec_df\n\n```", "votes": null }, { "id": "1765975", "postDate": "04/24/2022 04:24:37", "content": "

Great job on the competition. I had heard that not very much computing resources went into your training pipeline. Is this true? How many iterations did your team go through to get such good results?

", "rawMarkdown": "Great job on the competition. I had heard that not very much computing resources went into your training pipeline. Is this true? How many iterations did your team go through to get such good results?", "votes": null }, { "id": "1766206", "postDate": "04/24/2022 09:47:04", "content": "

Thanks for your reply！

", "rawMarkdown": "Thanks for your reply！", "votes": null }, { "id": "1767557", "postDate": "04/25/2022 13:21:23", "content": "

No, It's not that. There is some miscommunication. We didn't have any personal hardware (GPUs). We used TPUs to train our models, as some other top teams did. But, TPUs are good compute resources, available on Kaggle and Colab. Almost all of my kaggle finishes are with the models trained on TPUs/GPUs provided by Kaggle and Colab, except few in which I teamed up with someone who has good hardware. This is what Tanul wanted to convey in his post.
\nWe have been competing actively in this competition for last 1.5 months. So, our results has been results of many iterations. At first, our postprocessing was fixed (the same which I released in first week of start of the competition), and we tuned the architecture/training strategy for both DOLG & Non-DOLG models. Then, a lot of time went in understanding which preprocessing works better and designing data recipe, experimenting with psuedolabelling. Last week went in mostly designing & optimising postprocessing.

", "rawMarkdown": "No, It's not that. There is some miscommunication. We didn't have any personal hardware (GPUs). We used TPUs to train our models, as some other top teams did. But, TPUs are good compute resources, available on Kaggle and Colab. Almost all of my kaggle finishes are with the models trained on TPUs/GPUs provided by Kaggle and Colab, except few in which I teamed up with someone who has good hardware. This is what Tanul wanted to convey in his post. \nWe have been competing actively in this competition for last 1.5 months. So, our results has been results of many iterations. At first, our postprocessing was fixed (the same which I released in first week of start of the competition), and we tuned the architecture/training strategy for both DOLG & Non-DOLG models. Then, a lot of time went in understanding which preprocessing works better and designing data recipe, experimenting with psuedolabelling. Last week went in mostly designing & optimising postprocessing.", "votes": null }, { "id": "1767615", "postDate": "04/25/2022 14:02:34", "content": "

@ks2019 thanks for your sharing. Can you share your code for training LGB/XGB. I'm not familiar with stacking and want to learn.

", "rawMarkdown": "ks2019 thanks for your sharing. Can you share your code for training LGB/XGB. I'm not familiar with stacking and want to learn.", "votes": null }, { "id": "1852856", "postDate": "07/12/2022 12:39:06", "content": "

I wanted to ask about the your 2nd stage model, I bit confused what was the target used for it
\nLike suppose you have 1 image and its 100 possible matches (founded via knn), then you generate features for that image using the matches, now here what target do you use, and is it the knn distance or the cosine similarity?

", "rawMarkdown": "I wanted to ask about the your 2nd stage model, I bit confused what was the target used for it\nLike suppose you have 1 image and its 100 possible matches (founded via knn), then you generate features for that image using the matches, now here what target do you use, and is it the knn distance or the cosine similarity?", "votes": null } ], "comments": [ { "id": 1761383, "author_name": "cuongnn218", "author_url": "", "post_date": "04/19/2022 21:10:12", "content": "

Thanks for your sharing. Can you share your source code? I'm really interested in stacking part

", "votes": null, "replies": [ { "id": 1761429, "author_name": "librauee", "author_url": "", "post_date": "04/19/2022 22:48:02", "content": "

Good job, I am also interested in PostProcessing part.

", "votes": null, "replies": [] }, { "id": 1763110, "author_name": "ks2019", "author_url": "", "post_date": "04/21/2022 09:25:12", "content": "

Hi, sorry for late reply. Taking some time off from kaggle after intense competition. We'll make the complete source code public in few days.

", "votes": null, "replies": [] }, { "id": 1763118, "author_name": "ks2019", "author_url": "", "post_date": "04/21/2022 09:29:39", "content": "

Here is the code for feature engineering part. We trained the LGBM and XGBoost model on these features. Hyperparameters tuned with optuna to optimise competition metric. This gave us ~ 0.01-0.015 boost in public & private lb.

\n
KNN = 100\nnn_cols = [f'nn-{x}' for x in range(KNN)]\ndist_cols = [f'dist-{x}' for x in range(KNN)]\nreciprocals = 1/(np.arange(1,6))\n\ndef generate_preds(row,threshold=0.5):\n\n    nns = list(zip(row[dist_cols],row[nn_cols].values))\n    nns.append((threshold,-1))\n    preds = []\n    for dist,pred in sorted(nns):\n        if pred not in preds:\n            preds.append(pred)\n        if len(preds)==5:\n            break\n    if len(preds)<5:\n        preds = preds+[-1,-1,-1,-1,-1]\n        preds = preds[:5]\n    return np.array(preds)\n\n\ndef map_per_sample(row):\n    return ((row.target==row['preds'])*reciprocals).sum()\n\ndef get_sample_neighbours(data,MODE):\n\n    train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n    allowed_targets = set(train_targets)\n    neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n    neigh.fit(train_embeddings)\n    distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n    targets = train_targets[idxs]\n    ids = train_ids[idxs]\n\n    val_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n    val_predictions['nn_ids'] = train_ids[idxs].reshape(-1)\n    val_predictions['nn_distances'] = distances.reshape(-1)\n    val_predictions['nn_rank'] = val_predictions.index//KNN\n    val_predictions['image'] = val_predictions['nn_rank'].apply(lambda x: val_ids[x])\n    if MODE=='TRAIN':\n        val_predictions['target'] = val_predictions['nn_rank'].apply(lambda x: val_targets[x])\n    else:\n        val_predictions['target'] = -1\n    val_predictions['nn_rank'] = val_predictions.index%KNN\n    return val_predictions\n\ndef get_centroid_neighbours(data):\n\n    train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n    train_embeddings_df = pd.DataFrame(train_embeddings)\n    train_embeddings_df['target'] = train_targets\n    centroid_embeddings = train_embeddings_df.groupby('target').mean()\n    centroid_targets = centroid_embeddings.index.values\n    centroid_embeddings = centroid_embeddings.values\n\n    from sklearn.neighbors import NearestNeighbors\n    neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n    neigh.fit(centroid_embeddings)\n    distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n    targets = centroid_targets[idxs]\n\n    centroid_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n    centroid_predictions['nn_distances'] = distances.reshape(-1)\n    centroid_predictions['nn_rank'] = centroid_predictions.index//KNN\n    centroid_predictions['image'] = centroid_predictions['nn_rank'].apply(lambda x: val_ids[x])\n    centroid_predictions['nn_rank'] = centroid_predictions.index%KNN\n    centroid_predictions = centroid_predictions.set_index(['image','nn_target']).nn_distances.to_dict()\n\n    return centroid_predictions\n\ndef get_target_counts(data):\n    train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n    target_counts = pd.Series(train_targets).value_counts().to_dict()\n    return target_counts\n\ndef engineer_features(train_pattern,test_pattern,fold=0,MODE='TEST'):\n\n    print(\"Loading Data\")\n    val_species_ids,val_species,test_species_ids,test_species = load_species(fold=fold)\n    val_species = {x:y for x,y in zip(val_species_ids,val_species)}\n    test_species = {x:y for x,y in zip(test_species_ids,test_species)}\n\n    if MODE=='TEST':\n        data = load_test_embeds(train_pattern,test_pattern,fold=fold)\n        species = test_species\n    else:\n        data = load_embeds(train_pattern,test_pattern,fold=fold)\n        species = val_species\n\n    spec_df = pd.DataFrame(species).T.reset_index()\n    spec_df.columns = ['image']+[f'spec_{i}' for i in range(0,26)]\n\n    print(\"Getting Sample Neighbours\")\n    val_predictions = get_sample_neighbours(data,MODE)\n\n    print(\"Getting Centroid Neighbours\")\n    centroid_predictions = get_centroid_neighbours(data)\n\n    print(\"Getting Target Counts\")\n    target_counts = get_target_counts(data)\n\n    print(len(species),len(val_predictions))\n\n    train_df = val_predictions.groupby(['image','nn_target']).head(3)\n    train_df['nn_rank_target'] = train_df.groupby(['image','nn_target']).nn_distances.rank(method='first')\n    train_df = pd.pivot_table(train_df, values='nn_distances', index=['image','target', 'nn_target'],\n                        columns=['nn_rank_target'], aggfunc=np.sum).reset_index().fillna(1)\n    train_df = train_df.rename(columns={1:'nn1',2:'nn2',3:'nn3','nn_target':'individual'}).reset_index(drop=True)\n    train_df['centroid'] = train_df.apply(lambda row:centroid_predictions[(row.image,row.individual)]\n                                              if (row.image,row.individual) in centroid_predictions else 1\n                                          ,axis=1)\n    train_df['target_counts'] = train_df.apply(lambda row:target_counts[row.individual],axis=1)\n    train_df['individual_species'] = train_df.individual.map(species_individual_mappings)\n    train_df['species_conf'] = train_df.apply(lambda row:species[row.image][row.individual_species],axis=1)\n#     train_df[[f'species_{i}' for i in range(26)]] = train_df.apply(lambda row: species[row.image],axis=1)\n    return train_df,spec_df\n
", "votes": null, "replies": [] }, { "id": 1766206, "author_name": "librauee", "author_url": "", "post_date": "04/24/2022 09:47:04", "content": "

Thanks for your reply！

", "votes": null, "replies": [] }, { "id": 1767615, "author_name": "cuongnn218", "author_url": "", "post_date": "04/25/2022 14:02:34", "content": "

@ks2019 thanks for your sharing. Can you share your code for training LGB/XGB. I'm not familiar with stacking and want to learn.

", "votes": null, "replies": [] } ] }, { "id": 1765975, "author_name": "anandparthiban", "author_url": "", "post_date": "04/24/2022 04:24:37", "content": "

Great job on the competition. I had heard that not very much computing resources went into your training pipeline. Is this true? How many iterations did your team go through to get such good results?

", "votes": null, "replies": [ { "id": 1767557, "author_name": "ks2019", "author_url": "", "post_date": "04/25/2022 13:21:23", "content": "

No, It's not that. There is some miscommunication. We didn't have any personal hardware (GPUs). We used TPUs to train our models, as some other top teams did. But, TPUs are good compute resources, available on Kaggle and Colab. Almost all of my kaggle finishes are with the models trained on TPUs/GPUs provided by Kaggle and Colab, except few in which I teamed up with someone who has good hardware. This is what Tanul wanted to convey in his post.
\nWe have been competing actively in this competition for last 1.5 months. So, our results has been results of many iterations. At first, our postprocessing was fixed (the same which I released in first week of start of the competition), and we tuned the architecture/training strategy for both DOLG & Non-DOLG models. Then, a lot of time went in understanding which preprocessing works better and designing data recipe, experimenting with psuedolabelling. Last week went in mostly designing & optimising postprocessing.

", "votes": null, "replies": [] } ] }, { "id": 1852856, "author_name": "mrinath", "author_url": "", "post_date": "07/12/2022 12:39:06", "content": "

I wanted to ask about the your 2nd stage model, I bit confused what was the target used for it
\nLike suppose you have 1 image and its 100 possible matches (founded via knn), then you generate features for that image using the matches, now here what target do you use, and is it the knn distance or the cosine similarity?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1761241": "Hi all, \n\nCongratulations to all the winners. And thank you to kaggle host and all participants for this exciting competition.It was as always very tough last week with huge LB changing everybody , fortunately for us , our ideas worked and we ended on right side of the private LB.\n\nCongratulations to @ks2019 for becoming Kaggle Competitions GM. Thanks a lot to my teammates @nischaydnk @tanulsingh077 @navjotbansal. Such a great team effort 🤜.\n\nOur solution consists of the following major components:\n1. Data Recipe\n2. Modelling\n3. Progressive Psuedo Labelling\n4. PostProcessing\n5. Ensemble\n\nAll our models are trained on TPUs using more or less the same tensorflow pipeline that KS shared in the beginning of the competition.\n\n## Data Recipe\nThere were different datasets available publically, which we thought could help us in diversity. Inspired from @thedrcat 's solution in Chaii, we decided to make our own data recipe. We used the following combination of datasets to train our models:\n- Fullbody Annotations\n- Fullbody Annotations + Backfins concatenated horizontally \n- Fullbody Annotations + Original Images concatenated horizontally\n\nFullbody Annotations\n\"fullbody\"

\n\nFullbody Annotations + Backfins concatenated horizontally \n\"fullbody-backfin\"

\n\nFullbody Annotations + Original Images concatenated horizontally\n\"fullbody-original\"

\n\nWe used simple augmentations:\n- horizontal flip\n- random pixel based augmentation (brightness, contrast, HSV)\n- cutout\n\n## Modelling\nWe used a combination of DOLG (with EFFNet backbone) and normal EFFNets with CurricularFace loss. We used @christofhenkel 's implementation of DOLG model, [implemented in PyTorch](https://www.kaggle.com/competitions/landmark-retrieval-2021/discussion/277099) which we ported to TensorFlow because it worked better than the public implementation in this competition. We also used multiple heads in all the models: one for species classification and another for individual classification. Species classification head was trained with normal softmax loss while the individual classification head was trained with CurricularFace loss.\n\nModels in our final submission:\n- DOLG B5/B6/B7, image_sizes: (786, 786x2), (896, 896x2), dataset: Fullbody Annotations + Backfins\n- DOLG B6/B7, image_sizes: (896, 896x2), dataset: Fullbody Annotations + Original Images\n- DOLG B5/B6/B7, image_sizes: (1024, 1024), dataset: Fullbody Annotations\n- EFFNet B5/B6/B7, image_sizes: (786, 786x2), (896, 896x2), dataset: Fullbody Annotations + Backfins\n- EFFNet B5/B6/B7, image_sizes: (1024, 1024), dataset: Fullbody Annotations\n\nAll the models were trained on psuedo labelled data from our best ensemble. During inference we also use hflip as TTA.\n\n## PostProcessing\nWe used a second stage model for getting better confidence scores which is less susceptible to threshold changes. The idea was to use predictions/embeddings/features to get a better confidence score than just using nearest distances. After candidate generation (top100 neighbors based on knn distances) we engineered the following features for our 2nd stage model (features were generated from the models stated above):\n- species probabilites for each image_id\n- top3 nearest distances for each (image_id, unique individual_id) pair present in the candidates\n- distance of each image_id from centroid of each unique individual_id present in the candidates\n- rank of each unique individual_id present in the candidates\n- sum of top3 neighbor distances \n- OOF predictions\n\nWe trained a 5 folds XGB and LightGBM models on the above features and used a ensemble of their predicitons as final confidence scores.\n\n## Ensemble\nWe used simple weighted voting approach using the confidence scores obtained from above where the weights were optimized using 5 fold OOFs.\n\n## Acknowledgements\n- Thanks a lot to @jpbremer for sharing the datasets (The real hero 🔥)\n- Special mention to Google TPU Research Program (https://sites.research.google/trc/) which helped us in training bigger models in the last few days of the competition by supporting us with V3 TPUs on GCP.", "1761383": "Thanks for your sharing. Can you share your source code? I'm really interested in stacking part", "1761429": "Good job, I am also interested in PostProcessing part.", "1763110": "Hi, sorry for late reply. Taking some time off from kaggle after intense competition. We'll make the complete source code public in few days.", "1763118": "Here is the code for feature engineering part. We trained the LGBM and XGBoost model on these features. Hyperparameters tuned with optuna to optimise competition metric. This gave us ~ 0.01-0.015 boost in public & private lb.\n\n```\nKNN = 100\nnn_cols = [f'nn-{x}' for x in range(KNN)]\ndist_cols = [f'dist-{x}' for x in range(KNN)]\nreciprocals = 1/(np.arange(1,6))\n\ndef generate_preds(row,threshold=0.5):\n \n nns = list(zip(row[dist_cols],row[nn_cols].values))\n nns.append((threshold,-1))\n preds = []\n for dist,pred in sorted(nns):\n if pred not in preds:\n preds.append(pred)\n if len(preds)==5:\n break\n if len(preds)<5:\n preds = preds+[-1,-1,-1,-1,-1]\n preds = preds[:5]\n return np.array(preds)\n \n \ndef map_per_sample(row):\n return ((row.target==row['preds'])*reciprocals).sum()\n\ndef get_sample_neighbours(data,MODE):\n \n train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n allowed_targets = set(train_targets)\n neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n neigh.fit(train_embeddings)\n distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n targets = train_targets[idxs]\n ids = train_ids[idxs]\n\n val_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n val_predictions['nn_ids'] = train_ids[idxs].reshape(-1)\n val_predictions['nn_distances'] = distances.reshape(-1)\n val_predictions['nn_rank'] = val_predictions.index//KNN\n val_predictions['image'] = val_predictions['nn_rank'].apply(lambda x: val_ids[x])\n if MODE=='TRAIN':\n val_predictions['target'] = val_predictions['nn_rank'].apply(lambda x: val_targets[x])\n else:\n val_predictions['target'] = -1\n val_predictions['nn_rank'] = val_predictions.index%KNN\n return val_predictions\n\ndef get_centroid_neighbours(data):\n \n train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n train_embeddings_df = pd.DataFrame(train_embeddings)\n train_embeddings_df['target'] = train_targets\n centroid_embeddings = train_embeddings_df.groupby('target').mean()\n centroid_targets = centroid_embeddings.index.values\n centroid_embeddings = centroid_embeddings.values\n\n from sklearn.neighbors import NearestNeighbors\n neigh = NearestNeighbors(n_neighbors = KNN,metric='cosine')\n neigh.fit(centroid_embeddings)\n distances,idxs = neigh.kneighbors(val_embeddings, KNN, return_distance=True)\n targets = centroid_targets[idxs]\n\n centroid_predictions = pd.DataFrame(targets.reshape(-1,1),columns=['nn_target'])\n centroid_predictions['nn_distances'] = distances.reshape(-1)\n centroid_predictions['nn_rank'] = centroid_predictions.index//KNN\n centroid_predictions['image'] = centroid_predictions['nn_rank'].apply(lambda x: val_ids[x])\n centroid_predictions['nn_rank'] = centroid_predictions.index%KNN\n centroid_predictions = centroid_predictions.set_index(['image','nn_target']).nn_distances.to_dict()\n \n return centroid_predictions\n\ndef get_target_counts(data):\n train_embeddings,train_targets,train_ids,val_embeddings,val_targets,val_ids = data\n target_counts = pd.Series(train_targets).value_counts().to_dict()\n return target_counts\n\ndef engineer_features(train_pattern,test_pattern,fold=0,MODE='TEST'):\n \n print(\"Loading Data\")\n val_species_ids,val_species,test_species_ids,test_species = load_species(fold=fold)\n val_species = {x:y for x,y in zip(val_species_ids,val_species)}\n test_species = {x:y for x,y in zip(test_species_ids,test_species)}\n \n if MODE=='TEST':\n data = load_test_embeds(train_pattern,test_pattern,fold=fold)\n species = test_species\n else:\n data = load_embeds(train_pattern,test_pattern,fold=fold)\n species = val_species\n \n spec_df = pd.DataFrame(species).T.reset_index()\n spec_df.columns = ['image']+[f'spec_{i}' for i in range(0,26)]\n \n print(\"Getting Sample Neighbours\")\n val_predictions = get_sample_neighbours(data,MODE)\n \n print(\"Getting Centroid Neighbours\")\n centroid_predictions = get_centroid_neighbours(data)\n \n print(\"Getting Target Counts\")\n target_counts = get_target_counts(data)\n \n print(len(species),len(val_predictions))\n \n train_df = val_predictions.groupby(['image','nn_target']).head(3)\n train_df['nn_rank_target'] = train_df.groupby(['image','nn_target']).nn_distances.rank(method='first')\n train_df = pd.pivot_table(train_df, values='nn_distances', index=['image','target', 'nn_target'],\n columns=['nn_rank_target'], aggfunc=np.sum).reset_index().fillna(1)\n train_df = train_df.rename(columns={1:'nn1',2:'nn2',3:'nn3','nn_target':'individual'}).reset_index(drop=True)\n train_df['centroid'] = train_df.apply(lambda row:centroid_predictions[(row.image,row.individual)]\n if (row.image,row.individual) in centroid_predictions else 1\n ,axis=1)\n train_df['target_counts'] = train_df.apply(lambda row:target_counts[row.individual],axis=1)\n train_df['individual_species'] = train_df.individual.map(species_individual_mappings)\n train_df['species_conf'] = train_df.apply(lambda row:species[row.image][row.individual_species],axis=1)\n# train_df[[f'species_{i}' for i in range(26)]] = train_df.apply(lambda row: species[row.image],axis=1)\n return train_df,spec_df\n\n```", "1765975": "Great job on the competition. I had heard that not very much computing resources went into your training pipeline. Is this true? How many iterations did your team go through to get such good results?", "1766206": "Thanks for your reply！", "1767557": "No, It's not that. There is some miscommunication. We didn't have any personal hardware (GPUs). We used TPUs to train our models, as some other top teams did. But, TPUs are good compute resources, available on Kaggle and Colab. Almost all of my kaggle finishes are with the models trained on TPUs/GPUs provided by Kaggle and Colab, except few in which I teamed up with someone who has good hardware. This is what Tanul wanted to convey in his post. \nWe have been competing actively in this competition for last 1.5 months. So, our results has been results of many iterations. At first, our postprocessing was fixed (the same which I released in first week of start of the competition), and we tuned the architecture/training strategy for both DOLG & Non-DOLG models. Then, a lot of time went in understanding which preprocessing works better and designing data recipe, experimenting with psuedolabelling. Last week went in mostly designing & optimising postprocessing.", "1767615": "ks2019 thanks for your sharing. Can you share your code for training LGB/XGB. I'm not familiar with stacking and want to learn.", "1852856": "I wanted to ask about the your 2nd stage model, I bit confused what was the target used for it\nLike suppose you have 1 image and its 100 possible matches (founded via knn), then you generate features for that image using the matches, now here what target do you use, and is it the knn distance or the cosine similarity?" }, "source": "meta" }