The Growing Momentum for AI Foundation Models in Biotech and 12 Notable Companies

The most prominent examples of general purpose foundation models are the GPT-3 and GPT-4 models, which form the basis of ChatGPT, and BERT, or Bidirectional Encoder Representations from Transformers.

These are gigantic models trained on enormous volumes of data, often in a self-supervised or unsupervised manner (without the need for labeled data). 

Thanks to special model design, including transformer architecture and attention algorithms, foundation models are inherently generalizable, allowing their adaptation to a diverse array of downstream tasks, unlike traditional AI models that excel in single tasks like, say, predicting molecule-target interaction.

The "foundation" aspect comes from their generalizability: once pre-trained, they can be fine-tuned with smaller, domain-specific datasets to excel in specific tasks, reducing the need for training new models from scratch. This approach enables them to serve as a versatile base for a multitude of applications, from natural language processing to bioinformatics, by adapting to the nuances of particular challenges through additional training. 

Foundation models in bio

A number of companies are racing towards building more domain-specific foundation models, with even more accuracy and relevance than all-purpose models. 

For instance, in September 2023, Deep Genomics unveiled BigRNA, a pioneering AI foundation model for uncovering RNA biology and therapeutics. 

According to Deep Genomics, it is the first transformer neural network engineered for transcriptomics. BigRNA is informed by nearly two billion adjustable parameters and has been trained on thousands of datasets, totaling over a trillion genomic signals.

A month earlier, Ginkgo Bioworks and Google Cloud announced a 5-year partnership where Ginkgo would work to develop new, state-of-the-art large language models (LLMs). 

The Ginkgo’s AI foundation model would be focused on genomics, protein function, and synthetic biology and would be running on Google Cloud's Vertex AI platform. The model is supposed to help Ginkgo's customers accelerate innovation and discovery in fields as diverse as drug discovery, agriculture, industrial manufacturing, and biosecurity.

In February 2024, Ginkgo committed even further to building next-generation biological foundation models by the acquisition of key assets of Reverie Labs, a startup that builds and uses AI/ML tools to accelerate drug discovery.

Ginkgo has acquired Reverie's infrastructure and software for training large-scale AI foundation models and four of Reverie's key AI team members will join Ginkgo.