{ "id": 366590, "title": "Open Problems | My solution and ideas", "url": "/competitions/open-problems-multimodal/discussion/366590", "author_name": "Dmitriy Ershov", "post_date": "2022-11-16T20:12:26.114000", "votes": 4, "comment_count": 0, "views": 0, "content": "

Hi! Thanks to all the organizers and kagglers for this interesting, important and hard competition. I want to share my solution and ideas with you because it may be helpful.

\n

Multiome

\n

Preprocessing

\n
    \n
  1. SVD with concatenated Feature Shop (best is 256 components)
    2. \n
  2. SVD with default data (128, 256)
    3. \n
  3. SVD for targets (best is 128 components)
    4. \n
  4. Feature selection with Random Forest
    5. \n
\n

Corr selection, KNN worked bad.

\n

Models

\n
    \n
  1. LGB single regression
    2. \n
  2. Ridge with pseudo-labelling
    3. \n
  3. NNs - dense NN with Bi-LSTM, dense NN, 1D CNN, 2D CNN (dense, reshape, 2D convs)
    4. \n
\n

TabNet, CB worked bad. NNs were trained as single regressions and multi regressions both.

\n

CITEseq

\n

Preprocessing

\n
    \n
  1. Concatenated Feature Shop for NN
    2. \n
  2. SVD with default data (256)
    3. \n
  3. Feature selection with Random Forest
    4. \n
  4. Feature selection based on correlation -> worked well for single regressions with flipping of negative correlated features
    5. \n
\n

Models

\n
    \n
  1. LGB single regression with RF and corrs features
    2. \n
  2. CB single regression with RF features
    3. \n
  3. NNs - dense NN, 1D CNN, 2D CNN, with/without cell embedings
    4. \n
\n

TabNet also worked bad.

\n

Blending

\n

I tried 4 different methods:

\n
    \n
  1. Normalized averaging
    2. \n
  2. Weighted normalized averaging by total oof correlation
    3. \n
  3. Weighted normalized averaging by single target oof correlation
    4. \n
  4. Stacking with Ridge
    5. \n
\n

Third method gives best results.

\n

Ideas I did not try or tried a little

\n
    \n
  1. Transform vectors to distance matrices and fit it to 2D CNN. It may help make connections between features. Also we can build model with dual input - one for feature vector and one for distance matrix
    2. \n
  2. Augmentations (flip, little noise, vec rotating) for CNN
    3. \n
  3. Autoencoders
    4. \n
  4. Clip values and make group of features
    5. \n
  5. Kalman filter (was too long to compute)
    6. \n
  6. Fit only non-zero features to NN as sequences
    7. \n
  7. Different models for different cell types or weighted averaging, because different models show different results on different cell types
    8. \n
\n

Some my notebooks with analysis

\n
    \n
  1. CITEseq targets EDA
    2. \n
  2. Playground modeling
    3. \n
  3. CITEseq sklearn different cells modeling
    4. \n
", "messages": [ { "id": 2032792, "postDate": "2022-11-16T20:12:26.113Z", "content": "

Hi! Thanks to all the organizers and kagglers for this interesting, important and hard competition. I want to share my solution and ideas with you because it may be helpful.

\n

Multiome

\n

Preprocessing

\n
    \n
  1. SVD with concatenated Feature Shop (best is 256 components)
    2. \n
  2. SVD with default data (128, 256)
    3. \n
  3. SVD for targets (best is 128 components)
    4. \n
  4. Feature selection with Random Forest
    5. \n
\n

Corr selection, KNN worked bad.

\n

Models

\n
    \n
  1. LGB single regression
    2. \n
  2. Ridge with pseudo-labelling
    3. \n
  3. NNs - dense NN with Bi-LSTM, dense NN, 1D CNN, 2D CNN (dense, reshape, 2D convs)
    4. \n
\n

TabNet, CB worked bad. NNs were trained as single regressions and multi regressions both.

\n

CITEseq

\n

Preprocessing

\n
    \n
  1. Concatenated Feature Shop for NN
    2. \n
  2. SVD with default data (256)
    3. \n
  3. Feature selection with Random Forest
    4. \n
  4. Feature selection based on correlation -> worked well for single regressions with flipping of negative correlated features
    5. \n
\n

Models

\n
    \n
  1. LGB single regression with RF and corrs features
    2. \n
  2. CB single regression with RF features
    3. \n
  3. NNs - dense NN, 1D CNN, 2D CNN, with/without cell embedings
    4. \n
\n

TabNet also worked bad.

\n

Blending

\n

I tried 4 different methods:

\n
    \n
  1. Normalized averaging
    2. \n
  2. Weighted normalized averaging by total oof correlation
    3. \n
  3. Weighted normalized averaging by single target oof correlation
    4. \n
  4. Stacking with Ridge
    5. \n
\n

Third method gives best results.

\n

Ideas I did not try or tried a little

\n
    \n
  1. Transform vectors to distance matrices and fit it to 2D CNN. It may help make connections between features. Also we can build model with dual input - one for feature vector and one for distance matrix
    2. \n
  2. Augmentations (flip, little noise, vec rotating) for CNN
    3. \n
  3. Autoencoders
    4. \n
  4. Clip values and make group of features
    5. \n
  5. Kalman filter (was too long to compute)
    6. \n
  6. Fit only non-zero features to NN as sequences
    7. \n
  7. Different models for different cell types or weighted averaging, because different models show different results on different cell types
    8. \n
\n

Some my notebooks with analysis

\n
    \n
  1. CITEseq targets EDA
    2. \n
  2. Playground modeling
    3. \n
  3. CITEseq sklearn different cells modeling
    4. \n
", "rawMarkdown": "Hi! Thanks to all the organizers and kagglers for this interesting, important and hard competition. I want to share my solution and ideas with you because it may be helpful. \n\n# Multiome\n\n### Preprocessing\n\n1. SVD with concatenated [Feature Shop](https://www.kaggle.com/datasets/stasborodynkin/feature-shop-for-mmscel-multiome) (best is 256 components)\n2. SVD with default data (128, 256)\n3. SVD for targets (best is 128 components)\n4. Feature selection with [Random Forest](https://www.kaggle.com/bejeweled/multiome-rf-feature-selection)\n\nCorr selection, KNN worked bad.\n\n### Models\n\n1. LGB single regression\n2. Ridge with pseudo-labelling \n3. NNs - dense NN with Bi-LSTM, dense NN, 1D CNN, 2D CNN (dense, reshape, 2D convs)\n\nTabNet, CB worked bad. NNs were trained as single regressions and multi regressions both.\n\n# CITEseq\n\n### Preprocessing\n\n1. Concatenated [Feature Shop](https://www.kaggle.com/datasets/alexandervc/feature-shop-for-multimodal-singlecell-competition) for NN\n2. SVD with default data (256)\n3. Feature selection with [Random Forest](https://www.kaggle.com/code/bejeweled/siteseq-rf-feature-selection)\n4. Feature selection based on [correlation](https://www.kaggle.com/code/bejeweled/siteseq-corr-feature-selection) -> worked well for single regressions with flipping of negative correlated features\n\n### Models\n\n1. LGB single regression with RF and corrs features\n2. CB single regression with RF features\n3. NNs - dense NN, 1D CNN, 2D CNN, with/without cell embedings\n\nTabNet also worked bad. \n\n# Blending\n\nI tried 4 different methods:\n1. Normalized averaging\n2. Weighted normalized averaging by total oof correlation\n3. Weighted normalized averaging by single target oof correlation\n4. Stacking with Ridge\n\nThird method gives best results.\n\n# Ideas I did not try or tried a little\n\n1. Transform vectors to distance matrices and fit it to 2D CNN. It may help make connections between features. Also we can build model with dual input - one for feature vector and one for distance matrix\n2. Augmentations (flip, little noise, vec rotating) for CNN\n3. Autoencoders\n4. Clip values and make group of features \n5. Kalman filter (was too long to compute) \n6. Fit only non-zero features to NN as sequences\n7. Different models for different cell types or weighted averaging, because different models show different results on different cell types\n\n# Some my notebooks with analysis\n\n1. CITEseq targets [EDA](https://www.kaggle.com/code/bejeweled/mmscel-citeseq-targets-eda?scriptVersionId=109777501)\n2. Playground [modeling](https://www.kaggle.com/code/bejeweled/mmscel-all-targs-modeling-playground-fold/notebook)\n3. CITEseq sklearn different cells [modeling](https://www.kaggle.com/code/bejeweled/citeseq-sklearn-cells-feature-shop?scriptVersionId=110648831)", "votes": 4 } ], "comments": [], "raw_markdown_by_id": { "2032792": "Hi! Thanks to all the organizers and kagglers for this interesting, important and hard competition. I want to share my solution and ideas with you because it may be helpful. \n\n# Multiome\n\n### Preprocessing\n\n1. SVD with concatenated [Feature Shop](https://www.kaggle.com/datasets/stasborodynkin/feature-shop-for-mmscel-multiome) (best is 256 components)\n2. SVD with default data (128, 256)\n3. SVD for targets (best is 128 components)\n4. Feature selection with [Random Forest](https://www.kaggle.com/bejeweled/multiome-rf-feature-selection)\n\nCorr selection, KNN worked bad.\n\n### Models\n\n1. LGB single regression\n2. Ridge with pseudo-labelling \n3. NNs - dense NN with Bi-LSTM, dense NN, 1D CNN, 2D CNN (dense, reshape, 2D convs)\n\nTabNet, CB worked bad. NNs were trained as single regressions and multi regressions both.\n\n# CITEseq\n\n### Preprocessing\n\n1. Concatenated [Feature Shop](https://www.kaggle.com/datasets/alexandervc/feature-shop-for-multimodal-singlecell-competition) for NN\n2. SVD with default data (256)\n3. Feature selection with [Random Forest](https://www.kaggle.com/code/bejeweled/siteseq-rf-feature-selection)\n4. Feature selection based on [correlation](https://www.kaggle.com/code/bejeweled/siteseq-corr-feature-selection) -> worked well for single regressions with flipping of negative correlated features\n\n### Models\n\n1. LGB single regression with RF and corrs features\n2. CB single regression with RF features\n3. NNs - dense NN, 1D CNN, 2D CNN, with/without cell embedings\n\nTabNet also worked bad. \n\n# Blending\n\nI tried 4 different methods:\n1. Normalized averaging\n2. Weighted normalized averaging by total oof correlation\n3. Weighted normalized averaging by single target oof correlation\n4. Stacking with Ridge\n\nThird method gives best results.\n\n# Ideas I did not try or tried a little\n\n1. Transform vectors to distance matrices and fit it to 2D CNN. It may help make connections between features. Also we can build model with dual input - one for feature vector and one for distance matrix\n2. Augmentations (flip, little noise, vec rotating) for CNN\n3. Autoencoders\n4. Clip values and make group of features \n5. Kalman filter (was too long to compute) \n6. Fit only non-zero features to NN as sequences\n7. Different models for different cell types or weighted averaging, because different models show different results on different cell types\n\n# Some my notebooks with analysis\n\n1. CITEseq targets [EDA](https://www.kaggle.com/code/bejeweled/mmscel-citeseq-targets-eda?scriptVersionId=109777501)\n2. Playground [modeling](https://www.kaggle.com/code/bejeweled/mmscel-all-targs-modeling-playground-fold/notebook)\n3. CITEseq sklearn different cells [modeling](https://www.kaggle.com/code/bejeweled/citeseq-sklearn-cells-feature-shop?scriptVersionId=110648831)" } }