![\"\"](\"https://i.ibb.co/ZJgR3dQ/Selection-042.png\")
![\"\"](\"https://i.ibb.co/CnBPyy3/Selection-041.png\")
![\"\"](\"https://i.ibb.co/55Fs2Rd/Selection-043.png\")
![\"\"](\"https://i.ibb.co/WBHTbjZ/Selection-044.png\")
![\"\"](\"https://i.ibb.co/Dzz9ZMc/Selection-046.png\")
I've Implemented CNN-TCN with image attention. Comparing to CNN-LSTM it becomes terribly slow and unusable with my setup. I was thinking about transformer as decoder but I guess performance will be terrible.
", "rawMarkdown": "I've Implemented CNN-TCN with image attention. Comparing to CNN-LSTM it becomes terribly slow and unusable with my setup. I was thinking about transformer as decoder but I guess performance will be terrible.", "votes": null }, { "id": "1258340", "postDate": "03/31/2021 14:44:03", "content": "![\"\"](\"https://i.ibb.co/mSdN0Gf/Selection-047.png\")
Thanks for sharing such nice contents, May I ask for the tools you used for drawing these pictures? 🙏🙏
", "rawMarkdown": "Thanks for sharing such nice contents, May I ask for the tools you used for drawing these pictures? 🙏🙏", "votes": null }, { "id": "1259167", "postDate": "04/01/2021 07:29:12", "content": "@xzy777 They usually use Microsoft Power point to draw these architectures
", "rawMarkdown": "xzy777 They usually use Microsoft Power point to draw these architectures", "votes": null }, { "id": "1259399", "postDate": "04/01/2021 11:17:23", "content": "![\"\"](\"https://i.ibb.co/J7KyXfV/Selection-063.png\")
the original CNN-attention-LSTM is slow and cannot be parallelised (e.g. for beam search)
\ni attempt to redesign it (this is concept only. i have not implemented it)
the above shows stacked of 2 LSTM. One can stack up more layers. It is like iterative refinement using attention
", "rawMarkdown": "\n\nthe original CNN-attention-LSTM is slow and cannot be parallelised (e.g. for beam search)\ni attempt to redesign it (this is concept only. i have not implemented it)\n\n\nthe above shows stacked of 2 LSTM. One can stack up more layers. It is like iterative refinement using attention", "votes": null }, { "id": "1259869", "postDate": "04/01/2021 17:56:15", "content": "Good idea! Why trasformers are so powerful in computer vision task? Where do their benefits in quality come from?
", "rawMarkdown": "Good idea! Why trasformers are so powerful in computer vision task? Where do their benefits in quality come from?", "votes": null }, { "id": "1262225", "postDate": "04/04/2021 01:47:53", "content": "![\"\"](\"https://d3i71xaburhd42.cloudfront.net/9fb5e3db385588f671b11cfc8bf18efb90ee7b19/4-Figure4-1.png\")
it turns out that there is a paper that uses 1d conv to replace RNN in image captioning
\nhttps://arxiv.org/pdf/1711.09151.pdf
\nConvolutional Image Captioning
http://visal.cs.cityu.edu.hk/research/convolutional-decoders-for-image-captioning/
", "rawMarkdown": "\n\nit turns out that there is a paper that uses 1d conv to replace RNN in image captioning\nhttps://arxiv.org/pdf/1711.09151.pdf\nConvolutional Image Captioning\n\nhttp://visal.cs.cityu.edu.hk/research/convolutional-decoders-for-image-captioning/", "votes": null }, { "id": "1262292", "postDate": "04/04/2021 04:36:42", "content": "adventureous kagglers can try CLIP for pretraining
\nhttps://arxiv.org/pdf/2103.00020.pdf
\nhttps://openai.com/blog/clip/
Learning Transferable Visual Models From Natural Language Supervision
\n\" We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, \"
\"CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. \"
", "rawMarkdown": "adventureous kagglers can try CLIP for pretraining\n\nhttps://arxiv.org/pdf/2103.00020.pdf\nhttps://openai.com/blog/clip/\n\nLearning Transferable Visual Models From Natural Language Supervision\n\" We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, \"\n\n\n\"CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. \"", "votes": null }, { "id": "1264603", "postDate": "04/06/2021 09:46:28", "content": "template for tf user
\nhttps://guillaumegenthial.github.io/image-to-latex.html
\n(check both part 1 and 2 of the web article)
![\"\"](\"https://i.ibb.co/stbvcQY/Selection-040.png\")
I have tried this, it is quite difficult on the full dataset but seems like a really interesting way to go. I’m hoping I will have some time to work on it some more before this competition is over.
", "rawMarkdown": "I have tried this, it is quite difficult on the full dataset but seems like a really interesting way to go. I’m hoping I will have some time to work on it some more before this competition is over.", "votes": null }, { "id": "1265226", "postDate": "04/06/2021 17:37:21", "content": "i wonder if the followings make sense:
\nif both the model gives the same results, then you know it is correct.
\nthere are several ways that you can modify the ground truth. based on the prediction of these several modification, can we get a code that can error correct itself?
\nthe idea is error correction of transmission codes in noisy channels, which is common in communication theory.
\ne.g.
\nhttps://sandipanweb.wordpress.com/2017/05/06/some-nlp-spelling-correction-with-noisy-channel-model/
eg druchok_design_correction_control.pdf
\n2.6 Error correction model
\nError correction in SMILES strings can be framed as a standard sequence-to-sequence learning problem,
\nwhich is traditionally solved with attention-based encoder-decoder models
imagine if you use GAN or autoencoder to synthesis chemical formula, i wonder how do one ensure that the generated sample is of a valid molecule?
", "rawMarkdown": "i wonder if the followings make sense:\n- train a normal seq predictor\n- now modify the ground truth, e.g. reverse the sequence. train a model to predict the reverse sequence.\n\n\nif both the model gives the same results, then you know it is correct.\n\nthere are several ways that you can modify the ground truth. based on the prediction of these several modification, can we get a code that can error correct itself?\n\nthe idea is error correction of transmission codes in noisy channels, which is common in communication theory.\n\ne.g.\nhttps://sandipanweb.wordpress.com/2017/05/06/some-nlp-spelling-correction-with-noisy-channel-model/\n\neg druchok_design_correction_control.pdf\n\n2.6 Error correction model\nError correction in SMILES strings can be framed as a standard sequence-to-sequence learning problem,\nwhich is traditionally solved with attention-based encoder-decoder models\n\nimagine if you use GAN or autoencoder to synthesis chemical formula, i wonder how do one ensure that the generated sample is of a valid molecule?", "votes": null }, { "id": "1265241", "postDate": "04/06/2021 17:51:33", "content": "is an ensemble of several seq2seq models = another seq2se model
", "rawMarkdown": "is an ensemble of several seq2seq models = another seq2se model", "votes": null }, { "id": "1265308", "postDate": "04/06/2021 19:28:00", "content": "There was a mention of MolGAN in another discussion thread (couldn't find it anymore).
\ntldr: MolGAN was able to generate valid molecule with high probability. Maybe one can leverage the idea in MolGAN to regularize GAN or autoencoder to generate valid molecule.
", "rawMarkdown": "There was a mention of [MolGAN](https://arxiv.org/abs/1805.11973) in another discussion thread (couldn't find it anymore).\n\ntldr: MolGAN was able to generate valid molecule with high probability. Maybe one can leverage the idea in MolGAN to regularize GAN or autoencoder to generate valid molecule.", "votes": null }, { "id": "1265489", "postDate": "04/06/2021 23:58:04", "content": "an interesting way to decode:
\nhttps://iconictranslation.com/2020/05/issue-82-constrained-decoding-using-levenshtein-transformer/
levenshtein-transformer
\n![\"\"](\"https://iconictranslation.com/wp-content/uploads/2020/05/NMT-82-LVT-fig-1.jpg\")
test-time augmentation
\n![\"\"](\"https://i.ibb.co/tYnNsNV/Selection-050.png\")
i almost forget there is this image bert that you can do multi-task training
\nwe now have a unique problem:
\nhow to use all data for pretraining?
", "rawMarkdown": "\n\ni almost forget there is this image bert that you can do multi-task training\n\nwe now have a unique problem:\n- we have image,text pair\n- we have additional images without text (test data)\n- we have additional text without images (external csv)\n\nhow to use all data for pretraining?", "votes": null }, { "id": "1269822", "postDate": "04/11/2021 00:20:48", "content": "best of both worlds?
\nDeepMind, Microsoft, Allen AI & UW Researchers Convert Pretrained Transformers into RNNs, Lowering Memory Cost While Retaining High Accuracy
https://arxiv.org/pdf/2103.13076.pdf
\nwould use this if it is a code challenge
MOCO-v3
\n\"This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know
\nbaseline given the recent progress in computer vision: self-supervised learning for Visual Transformers (ViT).\"
\"We also verify our models in GPUs using PyTorch. It takes 24 hours for ViT-B in 128 GPUs (vs. 2.1 hours in 256
\nTPUs).\"
more here:
\nhttps://github.com/dk-liang/Awesome-Visual-Transformer
CPTR: FULL TRANSFORMER NETWORK FOR IMAGE CAPTIONING
\n![\"\"](\"https://pbs.twimg.com/media/EstJICeXEAMnt6y.jpg\")
![\"\"](\"https://github.com/Sara-Ahmed/SiT/raw/main/SiT.png\")
SiT: Self-supervised image Transformer
", "rawMarkdown": "\n\nSiT: Self-supervised image Transformer", "votes": null }, { "id": "1269842", "postDate": "04/11/2021 01:07:54", "content": "something like this
\n![\"\"](\"https://i.ibb.co/KFqdVkp/Selection-098.png\")
![\"\"](\"https://i.ibb.co/sF077j1/Selection-101.png\")
![\"\"](\"https://i.ibb.co/KXpfd5z/Selection-104.png\")
is there a way to augment by combing 2 small molecules to a larger one?
", "rawMarkdown": "is there a way to augment by combing 2 small molecules to a larger one?", "votes": null }, { "id": "1270090", "postDate": "04/11/2021 09:35:42", "content": "I used an autoencoder to synthesise novel SMILES during my Masters dissertation. By sampling the latent space around known molecules it was possible to generate novel molecules. The closer the sampling to the known molecule, the more valid SMILES were produced, but at a lower uniqueness rate.
\nNote that it is possible to write a chemical formula (e.g. SMILES or InChI) which is completely valid, but the molecule is unlikely to exist or be stable in the physical world.
", "rawMarkdown": "I used an autoencoder to synthesise novel SMILES during my Masters dissertation. By sampling the latent space around known molecules it was possible to generate novel molecules. The closer the sampling to the known molecule, the more valid SMILES were produced, but at a lower uniqueness rate. \n\nNote that it is possible to write a chemical formula (e.g. SMILES or InChI) which is completely valid, but the molecule is unlikely to exist or be stable in the physical world.", "votes": null }, { "id": "1270283", "postDate": "04/11/2021 13:42:06", "content": "![\"\"](\"https://github.com/clovaai/SATRN/raw/master/figures/architecture.png\")
\nOn Recognizing Texts of Arbitrary Shapes with 2D Self-Attention
\"Adaptive 2D positional encoding (A2DPE) This new positional encoding is necessary for dynamically adapting to
\nthe inherent aspect ratios incurred by overall text alignment (horizontal, diagonal, or vertical). As alternative options, we consider not doing any positional encoding at all (“None”) (Zhang et al. 2019; Wang et al. 2017), using
\n1D positional encoding over flattened feature map (“1DFlatten”), using concatenation of height and width positional encodings (“2D-Concat”) (Parmar et al. 2018), and the A2DPE that we propose. See Table 3a for the results.
\nWe observe that A2DPE provides the best accuracy among four options considered\"
this reminds me of the \"Spatial Transformer Networks\" which is used to align input using affine transformation.
\nhere we can learn a 2d positional encoding to make input image affine invariant
", "rawMarkdown": "\nOn Recognizing Texts of Arbitrary Shapes with 2D Self-Attention\n\n\"Adaptive 2D positional encoding (A2DPE) This new positional encoding is necessary for dynamically adapting to\nthe inherent aspect ratios incurred by overall text alignment (horizontal, diagonal, or vertical). As alternative options, we consider not doing any positional encoding at all (“None”) (Zhang et al. 2019; Wang et al. 2017), using\n1D positional encoding over flattened feature map (“1DFlatten”), using concatenation of height and width positional encodings (“2D-Concat”) (Parmar et al. 2018), and the A2DPE that we propose. See Table 3a for the results.\nWe observe that A2DPE provides the best accuracy among four options considered\"\n\n\nthis reminds me of the \"Spatial Transformer Networks\" which is used to align input using affine transformation.\n\nhere we can learn a 2d positional encoding to make input image affine invariant", "votes": null }, { "id": "1270345", "postDate": "04/11/2021 14:51:15", "content": "i realise that you actually don't have to input the image. you can just input the non-empy block.
\nhere you will have a variable length encoder.
Thanks for share! :)
", "rawMarkdown": "Thanks for share! :)", "votes": null }, { "id": "1270366", "postDate": "04/11/2021 15:21:45", "content": "![\"\"](\"https://i.ibb.co/CQ980Hv/Selection-106.png\")
kinda of error correction?
\n(the input is noisy of course)
good shares
", "rawMarkdown": "good shares", "votes": null }, { "id": "1271925", "postDate": "04/13/2021 04:24:38", "content": "![\"\"](\"https://i.ibb.co/55DJqBc/Selection-029.png\")
\ntransformer graph decoder
https://arxiv.org/pdf/2006.05213.pdf
", "rawMarkdown": "\ntransformer graph decoder\n\nhttps://arxiv.org/pdf/2006.05213.pdf", "votes": null }, { "id": "1273003", "postDate": "04/14/2021 02:09:34", "content": "graph-based encoder
\n![\"\"](\"https://i.ibb.co/m5c9TBJ/Selection-032.png\")
https://molvec.ncats.io/
\nhttps://github.com/ncats/molvec-web
this should be combined with a transformer
\n![\"\"](\"https://i.ibb.co/x6j7Hjp/Selection-031.png\")
https://davide-belli.github.io/generative-graph-transformer.html
\n![\"\"](\"https://davide-belli.github.io/images/blog-ggt/architecture.png\")
going multi-task
\n![\"\"](\"https://i.ibb.co/qyBj8cL/Selection-092.png\")
multi-task learning is probably
\n\"The Feynman Technique\"
\nhttps://www.youtube.com/watch?v=6QUjy0B3lEw
为什么费曼技巧被称为终极学习法 (Why feynman technique works)
\nhttps://www.youtube.com/watch?v=7iNJyEbYDdc
\"let the machine teach the human and not human teaches machine\"
\nknowledge distillation should benefit the teacher and not the student …
in multi-task we care about input encoding. we care less about the task.
\nwe are forced to seek simple features (information compression) that can solve many tasks …
", "rawMarkdown": "multi-task learning is probably \n\"The Feynman Technique\"\nhttps://www.youtube.com/watch?v=6QUjy0B3lEw\n\n为什么费曼技巧被称为终极学习法 (Why feynman technique works)\nhttps://www.youtube.com/watch?v=7iNJyEbYDdc\n\n\"let the machine teach the human and not human teaches machine\"\nknowledge distillation should benefit the teacher and not the student ...\n\n---\n\nin multi-task we care about input encoding. we care less about the task.\n\nwe are forced to seek simple features (information compression) that can solve many tasks ...", "votes": null }, { "id": "1295834", "postDate": "05/06/2021 18:19:54", "content": "How do think MLP-mixer for this application? Can it replace Transfomer? Thanks.
\nhttps://arxiv.org/pdf/2105.01601.pdf
![\"\"](\"https://blogs.biomedcentral.com/on-physicalsciences/wp-content/uploads/sites/14/2020/02/JCheminf_figure_1_rnn_sampling.gif\")
I've Implemented CNN-TCN with image attention. Comparing to CNN-LSTM it becomes terribly slow and unusable with my setup. I was thinking about transformer as decoder but I guess performance will be terrible.
", "votes": null, "replies": [] }, { "id": 1259037, "author_name": "xzy777", "author_url": "", "post_date": "04/01/2021 05:24:04", "content": "Thanks for sharing such nice contents, May I ask for the tools you used for drawing these pictures? 🙏🙏
", "votes": null, "replies": [ { "id": 1259167, "author_name": "sayedathar11", "author_url": "", "post_date": "04/01/2021 07:29:12", "content": "@xzy777 They usually use Microsoft Power point to draw these architectures
", "votes": null, "replies": [] } ] }, { "id": 1259399, "author_name": "hengck23", "author_url": "", "post_date": "04/01/2021 11:17:23", "content": "
the original CNN-attention-LSTM is slow and cannot be parallelised (e.g. for beam search)
\ni attempt to redesign it (this is concept only. i have not implemented it)
the above shows stacked of 2 LSTM. One can stack up more layers. It is like iterative refinement using attention
", "votes": null, "replies": [] }, { "id": 1259869, "author_name": "zavodrobotov", "author_url": "", "post_date": "04/01/2021 17:56:15", "content": "Good idea! Why trasformers are so powerful in computer vision task? Where do their benefits in quality come from?
", "votes": null, "replies": [] }, { "id": 1262225, "author_name": "hengck23", "author_url": "", "post_date": "04/04/2021 01:47:53", "content": "
it turns out that there is a paper that uses 1d conv to replace RNN in image captioning
\nhttps://arxiv.org/pdf/1711.09151.pdf
\nConvolutional Image Captioning
http://visal.cs.cityu.edu.hk/research/convolutional-decoders-for-image-captioning/
", "votes": null, "replies": [] }, { "id": 1262292, "author_name": "hengck23", "author_url": "", "post_date": "04/04/2021 04:36:42", "content": "adventureous kagglers can try CLIP for pretraining
\nhttps://arxiv.org/pdf/2103.00020.pdf
\nhttps://openai.com/blog/clip/
Learning Transferable Visual Models From Natural Language Supervision
\n\" We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, \"
\"CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. \"
", "votes": null, "replies": [ { "id": 1264858, "author_name": "interneuron", "author_url": "", "post_date": "04/06/2021 13:23:54", "content": "I have tried this, it is quite difficult on the full dataset but seems like a really interesting way to go. I’m hoping I will have some time to work on it some more before this competition is over.
", "votes": null, "replies": [] } ] }, { "id": 1264603, "author_name": "hengck23", "author_url": "", "post_date": "04/06/2021 09:46:28", "content": "template for tf user
\nhttps://guillaumegenthial.github.io/image-to-latex.html
\n(check both part 1 and 2 of the web article)
i wonder if the followings make sense:
\nif both the model gives the same results, then you know it is correct.
\nthere are several ways that you can modify the ground truth. based on the prediction of these several modification, can we get a code that can error correct itself?
\nthe idea is error correction of transmission codes in noisy channels, which is common in communication theory.
\ne.g.
\nhttps://sandipanweb.wordpress.com/2017/05/06/some-nlp-spelling-correction-with-noisy-channel-model/
eg druchok_design_correction_control.pdf
\n2.6 Error correction model
\nError correction in SMILES strings can be framed as a standard sequence-to-sequence learning problem,
\nwhich is traditionally solved with attention-based encoder-decoder models
imagine if you use GAN or autoencoder to synthesis chemical formula, i wonder how do one ensure that the generated sample is of a valid molecule?
", "votes": null, "replies": [ { "id": 1265241, "author_name": "hengck23", "author_url": "", "post_date": "04/06/2021 17:51:33", "content": "is an ensemble of several seq2seq models = another seq2se model
", "votes": null, "replies": [] }, { "id": 1265308, "author_name": "jy2tong", "author_url": "", "post_date": "04/06/2021 19:28:00", "content": "There was a mention of MolGAN in another discussion thread (couldn't find it anymore).
\ntldr: MolGAN was able to generate valid molecule with high probability. Maybe one can leverage the idea in MolGAN to regularize GAN or autoencoder to generate valid molecule.
", "votes": null, "replies": [] }, { "id": 1270090, "author_name": "talktocharles", "author_url": "", "post_date": "04/11/2021 09:35:42", "content": "I used an autoencoder to synthesise novel SMILES during my Masters dissertation. By sampling the latent space around known molecules it was possible to generate novel molecules. The closer the sampling to the known molecule, the more valid SMILES were produced, but at a lower uniqueness rate.
\nNote that it is possible to write a chemical formula (e.g. SMILES or InChI) which is completely valid, but the molecule is unlikely to exist or be stable in the physical world.
", "votes": null, "replies": [] } ] }, { "id": 1265489, "author_name": "hengck23", "author_url": "", "post_date": "04/06/2021 23:58:04", "content": "an interesting way to decode:
\nhttps://iconictranslation.com/2020/05/issue-82-constrained-decoding-using-levenshtein-transformer/
levenshtein-transformer
\n
test-time augmentation
\n
i almost forget there is this image bert that you can do multi-task training
\nwe now have a unique problem:
\nhow to use all data for pretraining?
", "votes": null, "replies": [ { "id": 1269842, "author_name": "hengck23", "author_url": "", "post_date": "04/11/2021 01:07:54", "content": "something like this
\n
best of both worlds?
\nDeepMind, Microsoft, Allen AI & UW Researchers Convert Pretrained Transformers into RNNs, Lowering Memory Cost While Retaining High Accuracy
https://arxiv.org/pdf/2103.13076.pdf
\nwould use this if it is a code challenge
MOCO-v3
\n\"This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know
\nbaseline given the recent progress in computer vision: self-supervised learning for Visual Transformers (ViT).\"
\"We also verify our models in GPUs using PyTorch. It takes 24 hours for ViT-B in 128 GPUs (vs. 2.1 hours in 256
\nTPUs).\"
more here:
\nhttps://github.com/dk-liang/Awesome-Visual-Transformer
CPTR: FULL TRANSFORMER NETWORK FOR IMAGE CAPTIONING
\n
SiT: Self-supervised image Transformer
", "votes": null, "replies": [] }, { "id": 1269844, "author_name": "hengck23", "author_url": "", "post_date": "04/11/2021 01:21:46", "content": "
i realise that you actually don't have to input the image. you can just input the non-empy block.
\nhere you will have a variable length encoder.
is there a way to augment by combing 2 small molecules to a larger one?
", "votes": null, "replies": [] }, { "id": 1270283, "author_name": "hengck23", "author_url": "", "post_date": "04/11/2021 13:42:06", "content": "
\nOn Recognizing Texts of Arbitrary Shapes with 2D Self-Attention
\"Adaptive 2D positional encoding (A2DPE) This new positional encoding is necessary for dynamically adapting to
\nthe inherent aspect ratios incurred by overall text alignment (horizontal, diagonal, or vertical). As alternative options, we consider not doing any positional encoding at all (“None”) (Zhang et al. 2019; Wang et al. 2017), using
\n1D positional encoding over flattened feature map (“1DFlatten”), using concatenation of height and width positional encodings (“2D-Concat”) (Parmar et al. 2018), and the A2DPE that we propose. See Table 3a for the results.
\nWe observe that A2DPE provides the best accuracy among four options considered\"
this reminds me of the \"Spatial Transformer Networks\" which is used to align input using affine transformation.
\nhere we can learn a 2d positional encoding to make input image affine invariant
", "votes": null, "replies": [] }, { "id": 1270358, "author_name": "dasmehdixtr", "author_url": "", "post_date": "04/11/2021 15:10:54", "content": "Thanks for share! :)
", "votes": null, "replies": [] }, { "id": 1270366, "author_name": "hengck23", "author_url": "", "post_date": "04/11/2021 15:21:45", "content": "
kinda of error correction?
\n(the input is noisy of course)
good shares
", "votes": null, "replies": [] }, { "id": 1271925, "author_name": "hengck23", "author_url": "", "post_date": "04/13/2021 04:24:38", "content": "
\ntransformer graph decoder
https://arxiv.org/pdf/2006.05213.pdf
", "votes": null, "replies": [] }, { "id": 1273003, "author_name": "hengck23", "author_url": "", "post_date": "04/14/2021 02:09:34", "content": "graph-based encoder
\n
https://molvec.ncats.io/
\nhttps://github.com/ncats/molvec-web
this should be combined with a transformer
\n
https://davide-belli.github.io/generative-graph-transformer.html
\n
going multi-task
\n
multi-task learning is probably
\n\"The Feynman Technique\"
\nhttps://www.youtube.com/watch?v=6QUjy0B3lEw
为什么费曼技巧被称为终极学习法 (Why feynman technique works)
\nhttps://www.youtube.com/watch?v=7iNJyEbYDdc
\"let the machine teach the human and not human teaches machine\"
\nknowledge distillation should benefit the teacher and not the student …
in multi-task we care about input encoding. we care less about the task.
\nwe are forced to seek simple features (information compression) that can solve many tasks …
", "votes": null, "replies": [] } ] }, { "id": 1295834, "author_name": "ybwu01", "author_url": "", "post_date": "05/06/2021 18:19:54", "content": "How do think MLP-mixer for this application? Can it replace Transfomer? Thanks.
\nhttps://arxiv.org/pdf/2105.01601.pdf