I finally finished the notebook that demonstrates how to train a convolutional AutoEncoder to learn latent features from the 3D CT scans dataset:
\n\nhttps://www.kaggle.com/carlossouza/osic-autoencoder-training
\n\nIn the example, training is only done for 10 epochs (~6min). To have better results, I recommend training for at least 500 epochs (~5 hours); I didn't do it because of GPU credits (I actually spent quite some time trying to make this training in TPUs, a very frustrating experience).
\n\nI'd to love to hear what you think. Cheers!
", "messages": [ { "id": 949611, "postDate": "2020-07-28T18:39:20.127Z", "content": "I finally finished the notebook that demonstrates how to train a convolutional AutoEncoder to learn latent features from the 3D CT scans dataset:
\n\nhttps://www.kaggle.com/carlossouza/osic-autoencoder-training
\n\nIn the example, training is only done for 10 epochs (~6min). To have better results, I recommend training for at least 500 epochs (~5 hours); I didn't do it because of GPU credits (I actually spent quite some time trying to make this training in TPUs, a very frustrating experience).
\n\nI'd to love to hear what you think. Cheers!
", "rawMarkdown": "I finally finished the notebook that demonstrates how to train a convolutional AutoEncoder to learn latent features from the 3D CT scans dataset: \n\nhttps://www.kaggle.com/carlossouza/osic-autoencoder-training\n\nIn the example, training is only done for 10 epochs (~6min). To have better results, I recommend training for at least 500 epochs (~5 hours); I didn't do it because of GPU credits (I actually spent quite some time trying to make this training in TPUs, a very frustrating experience). \n\nI'd to love to hear what you think. Cheers!", "votes": 10 }, { "id": 966516, "postDate": "2020-08-11T13:36:49.210Z", "content": "@carlossouza Does creating metadata from the CT scans help in modelling the data for autoencoders? I have created one but I am not sure how to incorporate it what all aspects should I consider?
", "rawMarkdown": "@carlossouza Does creating metadata from the CT scans help in modelling the data for autoencoders? I have created one but I am not sure how to incorporate it what all aspects should I consider?", "replies": [ { "id": 968187, "postDate": "2020-08-12T18:55:44.130Z", "rawMarkdown": "", "isDeleted": true }, { "id": 968188, "postDate": "2020-08-12T18:56:15.187Z", "content": "In my understanding, in theory it shouldn't… I'm preparing a notebook with a VAE implementation, will share soon :)
", "rawMarkdown": "In my understanding, in theory it shouldn't... I'm preparing a notebook with a VAE implementation, will share soon :)" } ] }, { "id": 964688, "postDate": "2020-08-10T05:14:52.973Z", "content": "Hi Carlos, thanks for sharing your work. \nAre you planning to use these latent features long with meta data to build an XG Boost classifier?\nDid you try 3D CNN to directly find the FVC and Confidence?
", "rawMarkdown": "Hi Carlos, thanks for sharing your work. \nAre you planning to use these latent features long with meta data to build an XG Boost classifier?\nDid you try 3D CNN to directly find the FVC and Confidence?", "replies": [ { "id": 968193, "postDate": "2020-08-12T18:58:46.313Z", "content": "Hi! I tried directly estimating FVC and confidence from the tabular data using a NN. The problem with this approach is that any discriminative machine learning approach will not estimate good sigmas. That's a job for generative machine learning (bayesian inference). I'm preparing a notebook about that, will share soon! :)
", "rawMarkdown": "Hi! I tried directly estimating FVC and confidence from the tabular data using a NN. The problem with this approach is that any discriminative machine learning approach will not estimate good sigmas. That's a job for generative machine learning (bayesian inference). I'm preparing a notebook about that, will share soon! :)" } ] } ], "comments": [ { "id": 966516, "author_name": "Digvijay Yadav", "author_url": "", "post_date": "2020-08-11T13:36:49.210000", "content": "@carlossouza Does creating metadata from the CT scans help in modelling the data for autoencoders? I have created one but I am not sure how to incorporate it what all aspects should I consider?
", "votes": 0, "replies": [ { "id": 968187, "author_name": "", "author_url": "", "post_date": "2020-08-12T18:55:44.130000", "content": "", "votes": 0, "replies": [] }, { "id": 968188, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-08-12T18:56:15.187000", "content": "In my understanding, in theory it shouldn't… I'm preparing a notebook with a VAE implementation, will share soon :)
", "votes": 0, "replies": [] } ] }, { "id": 964688, "author_name": "AjayKumar", "author_url": "", "post_date": "2020-08-10T05:14:52.973000", "content": "Hi Carlos, thanks for sharing your work. \nAre you planning to use these latent features long with meta data to build an XG Boost classifier?\nDid you try 3D CNN to directly find the FVC and Confidence?
", "votes": 0, "replies": [ { "id": 968193, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-08-12T18:58:46.313000", "content": "Hi! I tried directly estimating FVC and confidence from the tabular data using a NN. The problem with this approach is that any discriminative machine learning approach will not estimate good sigmas. That's a job for generative machine learning (bayesian inference). I'm preparing a notebook about that, will share soon! :)
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "949611": "I finally finished the notebook that demonstrates how to train a convolutional AutoEncoder to learn latent features from the 3D CT scans dataset: \n\nhttps://www.kaggle.com/carlossouza/osic-autoencoder-training\n\nIn the example, training is only done for 10 epochs (~6min). To have better results, I recommend training for at least 500 epochs (~5 hours); I didn't do it because of GPU credits (I actually spent quite some time trying to make this training in TPUs, a very frustrating experience). \n\nI'd to love to hear what you think. Cheers!", "966516": "@carlossouza Does creating metadata from the CT scans help in modelling the data for autoencoders? I have created one but I am not sure how to incorporate it what all aspects should I consider?", "964688": "Hi Carlos, thanks for sharing your work. \nAre you planning to use these latent features long with meta data to build an XG Boost classifier?\nDid you try 3D CNN to directly find the FVC and Confidence?" } }