publications

2024

1. ICLR

   Amortizing intractable inference in large language models

   Edward Hu*, Moksh Jain*, Eric Elmoznino, and 4 more authors

   In International Conference on Learning Representations, 2024

   Abs arXiv Bib Code

   Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest—including sequence continuation, infilling, and other forms of constrained generation—involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.

   @inproceedings{hu2023amortizing,
     title = {Amortizing intractable inference in large language models},
     author = {Hu*, Edward and Jain*, Moksh and Elmoznino, Eric and Kaddar, Younesse and Lajoie, Guillaume and Bengio, Yoshua and Malkin, Nikolay},
     booktitle = {International Conference on Learning Representations},
     year = {2024},
   }

2. ICLR

   Pre-Training and Fine-Tuning Generative Flow Networks

   Ling Pan, Moksh Jain, Kanika Madan, and 1 more author

   In International Conference on Learning Representations, 2024

   Abs arXiv Bib

   Generative Flow Networks (GFlowNets) are amortized samplers that learn stochastic policies to sequentially generate compositional objects from a given unnormalized reward distribution. They can generate diverse sets of high-reward objects, which is an important consideration in scientific discovery tasks. However, as they are typically trained from a given extrinsic reward function, it remains an important open challenge about how to leverage the power of pre-training and train GFlowNets in an unsupervised fashion for efficient adaptation to downstream tasks. Inspired by recent successes of unsupervised pre-training in various domains, we introduce a novel approach for reward-free pre-training of GFlowNets. By framing the training as a self-supervised problem, we propose an outcome-conditioned GFlowNet (OC-GFN) that learns to explore the candidate space. Specifically, OC-GFN learns to reach any targeted outcomes, akin to goal-conditioned policies in reinforcement learning. We show that the pre-trained OC-GFN model can allow for a direct extraction of a policy capable of sampling from any new reward functions in downstream tasks. Nonetheless, adapting OC-GFN on a downstream task-specific reward involves an intractable marginalization over possible outcomes. We propose a novel way to approximate this marginalization by learning an amortized predictor enabling efficient fine-tuning. Extensive experimental results validate the efficacy of our approach, demonstrating the effectiveness of pre-training the OC-GFN, and its ability to swiftly adapt to downstream tasks and discover modes more efficiently. This work may serve as a foundation for further exploration of pre-training strategies in the context of GFlowNets.

   @inproceedings{pan2023pretraining,
     title = {Pre-Training and Fine-Tuning Generative Flow Networks},
     author = {Pan, Ling and Jain, Moksh and Madan, Kanika and Bengio, Yoshua},
     booktitle = {International Conference on Learning Representations},
     year = {2024},
   }

3. ICLR

   PhyloGFN: Phylogenetic Inference with Generative Flow Networks

   Ming Yang Zhou, Zichao Yan, Elliot Layne, and 5 more authors

   In International Conference on Learning Representations, 2024

   Abs Bib

   Phylogenetics is a branch of computational biology that studies the evolutionary relationships among biological entities. Its long history and numerous applications notwithstanding, inference of phylogenetic trees from sequence data remains challenging: the high complexity of tree space poses a significant obstacle for the current combinatorial and probabilistic techniques. In this paper, we adopt the framework of generative flow networks (GFlowNets) to tackle two core problems in phylogenetics: parsimony-based and Bayesian phylogenetic inference. Because GFlowNets are well-suited for sampling complex combinatorial structures, they are a natural choice for exploring and sampling from the multimodal posterior distribution over tree topologies and evolutionary distances. We demonstrate that our amortized posterior sampler, PhyloGFN, produces diverse and high-quality evolutionary hypotheses on real benchmark datasets. PhyloGFN is competitive with prior works in marginal likelihood estimation and achieves a closer fit to the target distribution than state-of-the-art variational inference methods.

   @inproceedings{zhou2023phylogfn,
     title = {PhyloGFN: Phylogenetic Inference with Generative Flow Networks},
     author = {Zhou, Ming Yang and Yan, Zichao and Layne, Elliot and Malkin, Nikolay and Zhang, Dinghuai and Jain, Moksh and Blanchette, Mathieu and Bengio, Yoshua},
     booktitle = {International Conference on Learning Representations},
     year = {2024},
   }

2023

1. UAI

   Stochastic Generative Flow Networks

   Ling Pan*, Dinghuai Zhang*, Moksh Jain, and 2 more authors

   In Uncertainty in Artificial Intelligence, 2023

   Abs arXiv Bib Code

   Generative Flow Networks (or GFlowNets for short) are a family of probabilistic agents that learn to sample complex combinatorial structures through the lens of "inference as control". They have shown great potential in generating high-quality and diverse candidates from a given energy landscape. However, existing GFlowNets can be applied only to deterministic environments, and fail in more general tasks with stochastic dynamics, which can limit their applicability. To overcome this challenge, this paper introduces Stochastic GFlowNets, a new algorithm that extends GFlowNets to stochastic environments. By decomposing state transitions into two steps, Stochastic GFlowNets isolate environmental stochasticity and learn a dynamics model to capture it. Extensive experimental results demonstrate that Stochastic GFlowNets offer significant advantages over standard GFlowNets as well as MCMC- and RL-based approaches, on a variety of standard benchmarks with stochastic dynamics.

   @inproceedings{pan2023stochastic,
     title = {Stochastic Generative Flow Networks},
     author = {Pan*, Ling and Zhang*, Dinghuai and Jain, Moksh and Huang, Longbo and Bengio, Yoshua},
     booktitle = {Uncertainty in Artificial Intelligence},
     year = {2023},
   }

2. ICML

   Multi-objective gflownets

   Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-Garcı́a, and 4 more authors

   In International Conference on Machine Learning, 2023

   Abs arXiv Bib Code

   In many applications of machine learning, like drug discovery and material design, the goal is to generate candidates that simultaneously maximize a set of objectives. As these objectives are often conflicting, there is no single candidate that simultaneously maximizes all objectives, but rather a set of Pareto-optimal candidates where one objective cannot be improved without worsening another. Moreover, in practice, these objectives are often under-specified, making the diversity of candidates a key consideration. The existing multi-objective optimization methods focus predominantly on covering the Pareto front, failing to capture diversity in the space of candidates. Motivated by the success of GFlowNets for generation of diverse candidates in a single objective setting, in this paper we consider Multi-Objective GFlowNets (MOGFNs). MOGFNs consist of a novel Conditional GFlowNet which models a family of single-objective sub-problems derived by decomposing the multi-objective optimization problem. Our work is the first to empirically demonstrate conditional GFlowNets. Through a series of experiments on synthetic and benchmark tasks, we empirically demonstrate that MOGFNs outperform existing methods in terms of Hypervolume, R2-distance and candidate diversity. We also demonstrate the effectiveness of MOGFNs over existing methods in active learning settings. Finally, we supplement our empirical results with a careful analysis of each component of MOGFNs.

   @inproceedings{jain2023multi,
     title = {Multi-objective gflownets},
     author = {Jain, Moksh and Raparthy, Sharath Chandra and Hern{\'a}ndez-Garc{\i}́a, Alex and Rector-Brooks, Jarrid and Bengio, Yoshua and Miret, Santiago and Bengio, Emmanuel},
     booktitle = {International Conference on Machine Learning},
     pages = {14631--14653},
     year = {2023},
   }

3. ICML

   GFlowNet-EM for learning compositional latent variable models

   Edward J Hu*, Nikolay Malkin*, Moksh Jain, and 3 more authors

   In International Conference on Machine Learning, 2023

   Abs arXiv Bib Code

   Latent variable models (LVMs) with discrete compositional latents are an important but challenging setting due to a combinatorially large number of possible configurations of the latents. A key tradeoff in modeling the posteriors over latents is between expressivity and tractable optimization. For algorithms based on expectation-maximization (EM), the E-step is often intractable without restrictive approximations to the posterior. We propose the use of GFlowNets, algorithms for sampling from an unnormalized density by learning a stochastic policy for sequential construction of samples, for this intractable E-step. By training GFlowNets to sample from the posterior over latents, we take advantage of their strengths as amortized variational inference algorithms for complex distributions over discrete structures. Our approach, GFlowNet-EM, enables the training of expressive LVMs with discrete compositional latents, as shown by experiments on non-context-free grammar induction and on images using discrete variational autoencoders (VAEs) without conditional independence enforced in the encoder.

   @inproceedings{hu2023gflownet,
     title = {GFlowNet-EM for learning compositional latent variable models},
     author = {Hu*, Edward J and Malkin*, Nikolay and Jain, Moksh and Everett, Katie E and Graikos, Alexandros and Bengio, Yoshua},
     booktitle = {International Conference on Machine Learning},
     pages = {13528--13549},
     year = {2023},
   }

4. ICML

   Gflowout: Dropout with generative flow networks

   Dianbo Liu, Moksh Jain, Bonaventure FP Dossou, and 8 more authors

   In International Conference on Machine Learning, 2023

   Abs arXiv Bib Code

   Bayesian Inference offers principled tools to tackle many critical problems with modern neural networks such as poor calibration and generalization, and data inefficiency. However, scaling Bayesian inference to large architectures is challenging and requires restrictive approximations. Monte Carlo Dropout has been widely used as a relatively cheap way for approximate Inference and to estimate uncertainty with deep neural networks. Traditionally, the dropout mask is sampled independently from a fixed distribution. Recent works show that the dropout mask can be viewed as a latent variable, which can be inferred with variational inference. These methods face two important challenges: (a) the posterior distribution over masks can be highly multi-modal which can be difficult to approximate with standard variational inference and (b) it is not trivial to fully utilize sample-dependent information and correlation among dropout masks to improve posterior estimation. In this work, we propose GFlowOut to address these issues. GFlowOut leverages the recently proposed probabilistic framework of Generative Flow Networks (GFlowNets) to learn the posterior distribution over dropout masks. We empirically demonstrate that GFlowOut results in predictive distributions that generalize better to out-of-distribution data, and provide uncertainty estimates which lead to better performance in downstream tasks.

   @inproceedings{liu2023gflowout,
     title = {Gflowout: Dropout with generative flow networks},
     author = {Liu, Dianbo and Jain, Moksh and Dossou, Bonaventure FP and Shen, Qianli and Lahlou, Salem and Goyal, Anirudh and Malkin, Nikolay and Emezue, Chris Chinenye and Zhang, Dinghuai and Hassen, Nadhir and others},
     booktitle = {International Conference on Machine Learning},
     pages = {21715--21729},
     year = {2023},
   }

5. ICML

   Learning GFlowNets from partial episodes for improved convergence and stability

   Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, and 6 more authors

   In International Conference on Machine Learning, 2023

   Abs arXiv Bib

   Generative flow networks (GFlowNets) are a family of algorithms for training a sequential sampler of discrete objects under an unnormalized target density and have been successfully used for various probabilistic modeling tasks. Existing training objectives for GFlowNets are either local to states or transitions, or propagate a reward signal over an entire sampling trajectory. We argue that these alternatives represent opposite ends of a gradient bias-variance tradeoff and propose a way to exploit this tradeoff to mitigate its harmful effects. Inspired by the TD(λ) algorithm in reinforcement learning, we introduce subtrajectory balance or SubTB(λ), a GFlowNet training objective that can learn from partial action subsequences of varying lengths. We show that SubTB(λ) accelerates sampler convergence in previously studied and new environments and enables training GFlowNets in environments with longer action sequences and sparser reward landscapes than what was possible before. We also perform a comparative analysis of stochastic gradient dynamics, shedding light on the bias-variance tradeoff in GFlowNet training and the advantages of subtrajectory balance.

   @inproceedings{madan2023learning,
     title = {Learning GFlowNets from partial episodes for improved convergence and stability},
     author = {Madan, Kanika and Rector-Brooks, Jarrid and Korablyov, Maksym and Bengio, Emmanuel and Jain, Moksh and Nica, Andrei Cristian and Bosc, Tom and Bengio, Yoshua and Malkin, Nikolay},
     booktitle = {International Conference on Machine Learning},
     pages = {23467--23483},
     year = {2023},
   }

6. SPIGM@ICML

   BatchGFN: Generative Flow Networks for Batch Active Learning

   Shreshth A Malik, Salem Lahlou, Andrew Jesson, and 5 more authors

   In Structured Probabilistic Inference and Generative Modeling (SPIGM) workshop @ ICML, 2023

   Abs arXiv Bib Code

   We introduce BatchGFN – a novel approach for pool-based active learning that uses generative flow networks to sample sets of data points proportional to a batch reward. With an appropriate reward function to quantify the utility of acquiring a batch, such as the joint mutual information between the batch and the model parameters, BatchGFN is able to construct highly informative batches for active learning in a principled way. We show our approach enables sampling near-optimal utility batches at inference time with a single forward pass per point in the batch in toy regression problems. This alleviates the computational complexity of batch-aware algorithms and removes the need for greedy approximations to find maximizers for the batch reward. We also present early results for amortizing training across acquisition steps, which will enable scaling to real-world tasks.

   @inproceedings{malik2023batchgfn,
     title = {BatchGFN: Generative Flow Networks for Batch Active Learning},
     author = {Malik, Shreshth A and Lahlou, Salem and Jesson, Andrew and Jain, Moksh and Malkin, Nikolay and Deleu, Tristan and Bengio, Yoshua and Gal, Yarin},
     booktitle = {Structured Probabilistic Inference and Generative Modeling (SPIGM) workshop @ ICML},
     year = {2023},
   }

7. SPIGM@ICML

   Thompson Sampling for Improved Exploration in GFlowNets

   Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, and 5 more authors

   In Structured Probabilistic Inference and Generative Modeling (SPIGM) workshop @ ICML, 2023

   Abs arXiv Bib

   Generative flow networks (GFlowNets) are amortized variational inference algorithms that treat sampling from a distribution over compositional objects as a sequential decision-making problem with a learnable action policy. Unlike other algorithms for hierarchical sampling that optimize a variational bound, GFlowNet algorithms can stably run off-policy, which can be advantageous for discovering modes of the target distribution. Despite this flexibility in the choice of behaviour policy, the optimal way of efficiently selecting trajectories for training has not yet been systematically explored. In this paper, we view the choice of trajectories for training as an active learning problem and approach it using Bayesian techniques inspired by methods for multi-armed bandits. The proposed algorithm, Thompson sampling GFlowNets (TS-GFN), maintains an approximate posterior distribution over policies and samples trajectories from this posterior for training. We show in two domains that TS-GFN yields improved exploration and thus faster convergence to the target distribution than the off-policy exploration strategies used in past work.

   @inproceedings{rectorbrooks2023thompson,
     title = {Thompson Sampling for Improved Exploration in GFlowNets},
     author = {Rector-Brooks, Jarrid and Madan, Kanika and Jain, Moksh and Korablyov, Maksym and Liu, Cheng-Hao and Chandar, Sarath and Malkin, Nikolay and Bengio, Yoshua},
     booktitle = {Structured Probabilistic Inference and Generative Modeling (SPIGM) workshop @ ICML},
     year = {2023},
   }

8. DD

   GFlowNets for AI-driven scientific discovery

   Moksh Jain, Tristan Deleu, Jason Hartford, and 3 more authors

   Digital Discovery, 2023

   Abs arXiv Bib

   Tackling the most pressing problems for humanity, such as the climate crisis and the threat of global pandemics, requires accelerating the pace of scientific discovery. While science has traditionally relied on trial and error and even serendipity to a large extent, the last few decades have seen a surge of data-driven scientific discoveries. However, in order to truly leverage large-scale data sets and high-throughput experimental setups, machine learning methods will need to be further improved and better integrated in the scientific discovery pipeline. A key challenge for current machine learning methods in this context is the efficient exploration of very large search spaces, which requires techniques for estimating reducible (epistemic) uncertainty and generating sets of diverse and informative experiments to perform. This motivated a new probabilistic machine learning framework called GFlowNets, which can be applied in the modeling, hypotheses generation and experimental design stages of the experimental science loop. GFlowNets learn to sample from a distribution given indirectly by a reward function corresponding to an unnormalized probability, which enables sampling diverse, high-reward candidates. GFlowNets can also be used to form efficient and amortized Bayesian posterior estimators for causal models conditioned on the already acquired experimental data. Having such posterior models can then provide estimators of epistemic uncertainty and information gain that can drive an experimental design policy. Altogether, here we will argue that GFlowNets can become a valuable tool for AI-driven scientific discovery, especially in scenarios of very large candidate spaces where we have access to cheap but inaccurate measurements or to expensive but accurate measurements. This is a common setting in the context of drug and material discovery, which we use as examples throughout the paper.

   @article{jain2023gflownets,
     title = {GFlowNets for AI-driven scientific discovery},
     author = {Jain, Moksh and Deleu, Tristan and Hartford, Jason and Liu, Cheng-Hao and Hernandez-Garcia, Alex and Bengio, Yoshua},
     journal = {Digital Discovery},
     volume = {2},
     number = {3},
     pages = {557--577},
     year = {2023},
     publisher = {Royal Society of Chemistry},
   }

9. TMLR

   DEUP: Direct Epistemic Uncertainty Prediction

   Salem Lahlou*, Moksh Jain*, Hadi Nekoei, and 5 more authors

   Transactions on Machine Learning Research, 2023

   Abs arXiv Bib Code

   Epistemic Uncertainty is a measure of the lack of knowledge of a learner which diminishes with more evidence. While existing work focuses on using the variance of the Bayesian posterior due to parameter uncertainty as a measure of epistemic uncertainty, we argue that this does not capture the part of lack of knowledge induced by model misspecification. We discuss how the excess risk, which is the gap between the generalization error of a predictor and the Bayes predictor, is a sound measure of epistemic uncertainty which captures the effect of model misspecification. We thus propose a principled framework for directly estimating the excess risk by learning a secondary predictor for the generalization error and subtracting an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. We discuss the merits of this novel measure of epistemic uncertainty, and highlight how it differs from variance-based measures of epistemic uncertainty and addresses its major pitfall. Our framework, Direct Epistemic Uncertainty Prediction (DEUP) is particularly interesting in interactive learning environments, where the learner is allowed to acquire novel examples in each round. Through a wide set of experiments, we illustrate how existing methods in sequential model optimization can be improved with epistemic uncertainty estimates from DEUP, and how DEUP can be used to drive exploration in reinforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic image classification and predicting synergies of drug combinations.

   @article{lahlou2023deup,
     title = {DEUP: Direct Epistemic Uncertainty Prediction},
     author = {Lahlou*, Salem and Jain*, Moksh and Nekoei, Hadi and Butoi, Victor I and Bertin, Paul and Rector-Brooks, Jarrid and Korablyov, Maksym and Bengio, Yoshua},
     journal = {Transactions on Machine Learning Research},
     year = {2023},
   }

10. arXiv

    Multi-Fidelity Active Learning with GFlowNets

    Alex Hernandez-Garcia, Nikita Saxena, Moksh Jain, and 2 more authors

    arXiv preprint arXiv:2306.11715, 2023

    Abs arXiv Bib Code

    In the last decades, the capacity to generate large amounts of data in science and engineering applications has been growing steadily. Meanwhile, the progress in machine learning has turned it into a suitable tool to process and utilise the available data. Nonetheless, many relevant scientific and engineering problems present challenges where current machine learning methods cannot yet efficiently leverage the available data and resources. For example, in scientific discovery, we are often faced with the problem of exploring very large, high-dimensional spaces, where querying a high fidelity, black-box objective function is very expensive. Progress in machine learning methods that can efficiently tackle such problems would help accelerate currently crucial areas such as drug and materials discovery. In this paper, we propose the use of GFlowNets for multi-fidelity active learning, where multiple approximations of the black-box function are available at lower fidelity and cost. GFlowNets are recently proposed methods for amortised probabilistic inference that have proven efficient for exploring large, high-dimensional spaces and can hence be practical in the multi-fidelity setting too. Here, we describe our algorithm for multi-fidelity active learning with GFlowNets and evaluate its performance in both well-studied synthetic tasks and practically relevant applications of molecular discovery. Our results show that multi-fidelity active learning with GFlowNets can efficiently leverage the availability of multiple oracles with different costs and fidelities to accelerate scientific discovery and engineering design.

    @article{hernandez2023multi,
      title = {Multi-Fidelity Active Learning with GFlowNets},
      author = {Hernandez-Garcia, Alex and Saxena, Nikita and Jain, Moksh and Liu, Cheng-Hao and Bengio, Yoshua},
      journal = {arXiv preprint arXiv:2306.11715},
      year = {2023},
    }

2022

1. NeurIPS

   Trajectory balance: Improved credit assignment in gflownets

   Nikolay Malkin, Moksh Jain, Emmanuel Bengio, and 2 more authors

   Advances in Neural Information Processing Systems, 2022

   Abs arXiv Bib

   Generative flow networks (GFlowNets) are a method for learning a stochastic policy for generating compositional objects, such as graphs or strings, from a given unnormalized density by sequences of actions, where many possible action sequences may lead to the same object. We find previously proposed learning objectives for GFlowNets, flow matching and detailed balance, which are analogous to temporal difference learning, to be prone to inefficient credit propagation across long action sequences. We thus propose a new learning objective for GFlowNets, trajectory balance, as a more efficient alternative to previously used objectives. We prove that any global minimizer of the trajectory balance objective can define a policy that samples exactly from the target distribution. In experiments on four distinct domains, we empirically demonstrate the benefits of the trajectory balance objective for GFlowNet convergence, diversity of generated samples, and robustness to long action sequences and large action spaces.

   @article{malkin2022trajectory,
     title = {Trajectory balance: Improved credit assignment in gflownets},
     author = {Malkin, Nikolay and Jain, Moksh and Bengio, Emmanuel and Sun, Chen and Bengio, Yoshua},
     journal = {Advances in Neural Information Processing Systems},
     volume = {35},
     pages = {5955--5967},
     year = {2022},
   }

2. HILL@NeurIPS

   Consistent Training via Energy-Based GFlowNets for Modeling Discrete Joint Distributions

   Chanakya Ekbote, Moksh Jain, Payel Das, and 1 more author

   In Workshop on Human in the Loop Learning @ NeurIPS, 2022

   Abs arXiv Bib

   Generative Flow Networks (GFlowNets) have demonstrated significant performance improvements for generating diverse discrete objects given a reward function , indicating the utility of the object and trained independently from the GFlowNet by supervised learning to predict a desirable property given . We hypothesize that this can lead to \textitincompatibility between the inductive optimization biases in training and in training the GFlowNet, potentially leading to worse samples and slow adaptation to changes in the distribution. In this work, we build upon recent work on jointly learning energy-based models with GFlowNets and extend it to learn the joint over multiple variables, which we call Joint Energy-Based GFlowNets (JEBGFNs), such as peptide sequences and their antimicrobial activity. Joint learning of the energy-based model, used as a reward for the GFlowNet, can resolve the issues of incompatibility since both the reward function and the GFlowNet sampler are trained jointly. We find that this joint training or joint energy-based formulation leads to significant improvements in generating anti-microbial peptides. As the training sequences arose out of evolutionary or artificial selection for high antibiotic activity, there is presumably some structure in the distribution of sequences that reveals information about the antibiotic activity. This results in an advantage to modeling their joint generatively vs. pure discriminative modeling. We also evaluate JEBGFN in an active learning setting for discovering anti-microbial peptides.

   @inproceedings{ekbote2022consistent,
     title = {Consistent Training via Energy-Based GFlowNets for Modeling Discrete Joint Distributions},
     author = {Ekbote, Chanakya and Jain, Moksh and Das, Payel and Bengio, Yoshua},
     booktitle = {Workshop on Human in the Loop Learning @ NeurIPS},
     year = {2022},
   }

3. ICML

   Biological Sequence Design with GFlowNets

   Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, and 8 more authors

   In International Conference on Machine Learning, 2022

   Abs arXiv Bib Code

   Design of de novo biological sequences with desired properties, like protein and DNA sequences, often involves an active loop with several rounds of molecule ideation and expensive wet-lab evaluations. These experiments can consist of multiple stages, with increasing levels of precision and cost of evaluation, where candidates are filtered. This makes the diversity of proposed candidates a key consideration in the ideation phase. In this work, we propose an active learning algorithm leveraging epistemic uncertainty estimation and the recently proposed GFlowNets as a generator of diverse candidate solutions, with the objective to obtain a diverse batch of useful (as defined by some utility function, for example, the predicted anti-microbial activity of a peptide) and informative candidates after each round. We also propose a scheme to incorporate existing labeled datasets of candidates, in addition to a reward function, to speed up learning in GFlowNets. We present empirical results on several biological sequence design tasks, and we find that our method generates more diverse and novel batches with high scoring candidates compared to existing approaches.

   @inproceedings{jain2022biological,
     title = {Biological Sequence Design with GFlowNets},
     author = {Jain, Moksh and Bengio, Emmanuel and Hernandez-Garcia, Alex and Rector-Brooks, Jarrid and Dossou, Bonaventure FP and Ekbote, Chanakya Ajit and Fu, Jie and Zhang, Tianyu and Kilgour, Michael and Zhang, Dinghuai and others},
     booktitle = {International Conference on Machine Learning},
     pages = {9786--9801},
     year = {2022},
   }

4. MLDD@ICLR

   Evaluating Generalization in GFlowNets for Molecule Design

   Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, and 4 more authors

   In Machine Learning for Drug Discovery workshop @ ICLR, 2022

   Abs Bib PDF

   Deep learning bears promise for drug discovery problems such as de novo molecular design. Generating data to train such models is a costly and time-consuming process, given the need for wet-lab experiments or expensive simulations. This problem is compounded by the notorious data-hungriness of machine learning algorithms. In small molecule generation the recently proposed GFlowNet method has shown good performance in generating diverse high-scoring candidates and has the interesting advantage of being an off-policy offline method. Finding an appropriate generalization evaluation metric for such models, one predictive of the desired search performance (i.e. finding high-scoring diverse candidates), will help guide online data collection for such an algorithm. In this work, we develop techniques for evaluating GFlowNet performance on a test set, and identify the most promising metric for predicting generalization. We present empirical results on several small-molecule design tasks in drug discovery, for several GFlowNet training setups, and we find a metric strongly correlated with diverse high-scoring batch generation. This metric should be used to identify the best generative model from which to sample batches of molecules to be evaluated.

   @inproceedings{nica2022evaluating,
     title = {Evaluating Generalization in GFlowNets for Molecule Design},
     author = {Nica, Andrei Cristian and Jain, Moksh and Bengio, Emmanuel and Liu, Cheng-Hao and Korablyov, Maksym and Bronstein, Michael M and Bengio, Yoshua},
     booktitle = {Machine Learning for Drug Discovery workshop @ ICLR},
     year = {2022},
   }

2021

1. NeurIPS

   Flow network based generative models for non-iterative diverse candidate generation

   Emmanuel Bengio, Moksh Jain, Maksym Korablyov, and 2 more authors

   Advances in Neural Information Processing Systems, 2021

   Abs arXiv Bib Code

   This paper is about the problem of learning a stochastic policy for generating an object (like a molecular graph) from a sequence of actions, such that the probability of generating an object is proportional to a given positive reward for that object. Whereas standard return maximization tends to converge to a single return-maximizing sequence, there are cases where we would like to sample a diverse set of high-return solutions. These arise, for example, in black-box function optimization when few rounds are possible, each with large batches of queries, where the batches should be diverse, e.g., in the design of new molecules. One can also see this as a problem of approximately converting an energy function to a generative distribution. While MCMC methods can achieve that, they are expensive and generally only perform local exploration. Instead, training a generative policy amortizes the cost of search during training and yields to fast generation. Using insights from Temporal Difference learning, we propose GFlowNet, based on a view of the generative process as a flow network, making it possible to handle the tricky case where different trajectories can yield the same final state, e.g., there are many ways to sequentially add atoms to generate some molecular graph. We cast the set of trajectories as a flow and convert the flow consistency equations into a learning objective, akin to the casting of the Bellman equations into Temporal Difference methods. We prove that any global minimum of the proposed objectives yields a policy which samples from the desired distribution, and demonstrate the improved performance and diversity of GFlowNet on a simple domain where there are many modes to the reward function, and on a molecule synthesis task.

   @article{bengio2021flow,
     title = {Flow network based generative models for non-iterative diverse candidate generation},
     author = {Bengio, Emmanuel and Jain, Moksh and Korablyov, Maksym and Precup, Doina and Bengio, Yoshua},
     journal = {Advances in Neural Information Processing Systems},
     volume = {34},
     pages = {27381--27394},
     year = {2021},
   }

2020

1. ICML

   DROCC: Deep robust one-class classification

   Sachin Goyal, Aditi Raghunathan, Moksh Jain, and 2 more authors

   In International Conference on Machine Learning, 2020

   Abs arXiv Bib Code

   Classical approaches for one-class problems such as one-class SVM and isolation forest require careful feature engineering when applied to structured domains like images. State-of-the-art methods aim to leverage deep learning to learn appropriate features via two main approaches. The first approach based on predicting transformations (Golan & El-Yaniv, 2018; Hendrycks et al., 2019a) while successful in some domains, crucially depends on an appropriate domain-specific set of transformations that are hard to obtain in general. The second approach of minimizing a classical one-class loss on the learned final layer representations, eg, DeepSVDD (Ruff et al., 2018) suffers from the fundamental drawback of representation collapse. In this work, we propose Deep Robust One Class Classification (DROCC) that is both applicable to most standard domains without requiring any side-information and robust to representation collapse. DROCC is based on the assumption that the points from the class of interest lie on a well-sampled, locally linear low dimensional manifold. Empirical evaluation demonstrates that DROCC is highly effective in two different one-class problem settings and on a range of real-world datasets across different domains: tabular data, images (CIFAR and ImageNet), audio, and time-series, offering up to 20% increase in accuracy over the state-of-the-art in anomaly detection. Code is available at https://github.com/microsoft/EdgeML

   @inproceedings{goyal2020drocc,
     title = {DROCC: Deep robust one-class classification},
     author = {Goyal, Sachin and Raghunathan, Aditi and Jain, Moksh and Simhadri, Harsha Vardhan and Jain, Prateek},
     booktitle = {International Conference on Machine Learning},
     pages = {3711--3721},
     year = {2020},
   }

2019

1. SGO@NeurIPS

   Proximal Policy Optimization for Improved Convergence in IRGAN

   Moksh Jain, and Sowmya Kamath

   Smooth Games Optimization and Machine Learning, NeurIPS 2019, 2019

   Abs arXiv Bib

   IRGAN is an information retrieval (IR) modeling approach that uses a theoretical minimax game between a generative and a discriminative model to iteratively optimize both of them, hence unifying the generative and discriminative approaches. Despite significant performance improvements in several information retrieval tasks, IRGAN training is an unstable process, and the solution varies largely with the random parameter initialization. In this work, we present an improved training objective based on proximal policy optimization objective and Gumbel-Softmax based sampling for the generator. We also propose a modified training algorithm which takes a single gradient update on both the generator as well as discriminator for each iteration step. We present empirical evidence of the improved convergence of the proposed model over the original IRGAN and a comparison on three different IR tasks on benchmark datasets is also discussed, emphasizing the proposed model’s superior performance.

   @article{jain2019proximal,
     title = {Proximal Policy Optimization for Improved Convergence in IRGAN},
     author = {Jain, Moksh and Kamath, Sowmya},
     journal = {Smooth Games Optimization and Machine Learning, NeurIPS 2019},
     year = {2019},
   }