Back to News

InstaDeep presents record 13 papers at NeurIPS 2023

Published

Categories

NeurIPS
Research

Share

Share on LinkedInShare on RedditShare on Pocket

• An InstaDeep record of 13 papers are being presented at world’s top AI and machine learning conference

• Breadth of research ranges from Decision-Making AI to breakthroughs in Machine Learning for Biology

• Highlights include accelerated engineering design, protein graph neural network innovations, combinatorial optimization problem-solving and more

As a pioneer in AI research and innovation, InstaDeep has always prioritized scientific exploration. 

Since our NeurIPS debut in 2018, we’ve scaled our research efforts, becoming a key player in artificial intelligence. In 2023, we’re excited to mark a new milestone in our journey by presenting a record 13 papers at NeurIPS, the world’s leading conference for machine learning and AI research.

This year’s submissions reflect the depth and breadth of InstaDeep’s research efforts, ranging from groundbreaking developments in Decision-Making AI to breakthroughs in Machine Learning for Biology. We believe each paper represents a valuable step forward.

“We owe much of our success to the scientific and open-source communities, which have been incredible drivers of progress, ” InstaDeep’s Head of Research, Alexandre Laterre says. “I am extremely proud of our teams for their outstanding contributions in crucial areas like machine learning for biology and decision-making AI — and making them broadly available through open-source.” 

Here is a quick digest running down this year’s full InstaDeep NeurIPS paper line-up. If anything peaks your interest, we’ve provided links to the full papers so you can dive deeper. 

Main Track Papers

Nonparametric Boundary Geometry in Physics Informed Deep Learning

To accelerate engineering design, we developed a model to predict partial differential equation solutions for arbitrary boundary meshes. In our experiments, we discovered training on a diverse array of meshes significantly enhances accuracy. This represents a substantial improvement over standard physics-informed neural networks, which typically train on a single domain.

Winner Takes It All: Training Performant RL Populations for Combinatorial Optimization

Solving NP-hard combinatorial optimization problems, such as the traveling salesman problem, requires efficiently searching the massive space of possible solutions. We propose a population-based Reinforcement Learning algorithm that trains a diverse set of complementary strategies, and demonstrates that our approach can outperform state-of-the-art single-agent methods.

Combinatorial Optimization with Policy Adaptation using Latent Space Search

Building on our “Winner Takes It All” work, we take it a step further by learning a continuous distribution of complementary strategies without requiring an explicit population of agents during training. At inference time, this approach maintains few-shot performance while also allowing simple evolutionary algorithms to generate increasingly effective strategies when a problem has a larger search budget.

Papers presented at Workshops

BioCLIP: Contrasting Sequence with Structure: Pre-training Graph Representations with Protein Language Models

We show how protein language models trained on sequence-only data can be used to train a protein structure model, a graph neural network, using contrastive learning. The structural embeddings are additive to sequence-only embeddings on a variety of downstream tasks including residue-level protein interactions, gene ontology terms and enzyme commission number.

Generalisable Agents for Neural Network Optimisation

We leverage multi-agent reinforcement learning to produce layerwise learning rate schedules for supervised learning problems by observing its neural network dynamics. These schedules are effective, adaptive, and generalisable to increasingly complex problems.

LightMHC: A Light Model for pMHC Structure Prediction with Graph Neural Networks 

LightMHC is a lightweight model that outperforms AlphaFold 2 and ESMFold in predicting full-atom peptide-major histocompatibility complex structures (“pMHC”). It provides comparable accuracy while being five times faster, making it a highly efficient tool for immune protein structure prediction and immunotherapy design

FrameDiPT: SE(3) Diffusion Model for Protein Structure Inpainting

We proposed FrameDiff inPainTing (“FrameDiPT”), an SE(3) diffusion model for protein structure inpainting. With only 18M parameters, trained on 32K monomer structures, FrameDiPT achieved state-of-the-art performance on TCR CDR3 loop design compared to existing diffusion models ProteinGenerator and RFdiffusion, demonstrating strong generalization capacity while capturing structural conformational distributions.

Preferential Bayesian Optimisation for Protein Design with Ranking-Based Fitness Predictors

Bayesian Optimization (BO) is a family of approaches for efficiently optimizing black-box functions. BO relies on a surrogate model to predict function values while estimating uncertainty. We find that ranking-based surrogate models based on ensembles of CNNs or fine-tuned protein language models are superior predictors of protein fitness, and show how they can be used to significantly accelerate protein design guided by BO.

  • Workshop: Machine Learning in Structural Biology
  • Paper will be available on workshop website (mlsb.io)

Offline RL for generative design of protein binders

We use offline reinforcement learning (RL) to design small molecules. Our approach has two steps: First, we train a transformer encoder-decoder model using supervised learning with the PDBBind dataset to generate the corresponding ligand for each complex conditioned on the protein’s amino acid sequence. Then, we fine-tune this model using offline RL to optimize the chemical and molecular docking properties of the generated molecules to their respective targets.

Are we going MAD? Benchmarking Multi-Agent Debate between Language Models for Medical Q&A

We benchmarked Multi-Agent Debate (MAD) strategies to enhance accuracy and reliability of large language models in medical Q&A. We improve over baseline debate-prompting strategy by modulating agent agreement, and show that it outperforms existing strategies in the MedQA medical dataset. 

PASTA: Pretrained Action-State Transformer Agents

What ingredients contribute to the success of a foundational model in RL? In PASTA, we systematically compared design choices (tokenization-level, pre-training objective), uncovering valuable insights for practitioners. Our study encourages simple pre-training methods (e.g. masked language modeling on tokenized observation components) and offers adaptable models for a range of tasks, fostering robust policy learning.

CSSL-MHTR: Continual Self-Supervised Learning for Scalable Multi-script Handwritten Text Recognition

We used a continual-learning framework to improve the recognition of multi-script handwritten documents using a single model. The framework benefits from a self-supervised step that helps with catastrophic forgetting that occurs when a new script or language is added to the model.

From Humans to Agents: Reinventing Team Dynamics and Leadership in Multi-Agent RL

What happens when you leverage competition among agents to lead others while at the same time encouraging teamwork? While MultiAgent RL is built on top of economic theories, this paper introduces an economic language for agents to communicate with, this formulation not only enhances time convergence but also reduces sensitivity to hyperparameters compared to MAPPO.

  • Workshop: North Africans in Machine Learning
  • Read the full paper: Poster only, research ongoing

Follow us on X for #NeurIPS2023 updates and if you’re interested in joining our AI research team, visit instadeep.com/careers