Protein Language Models: Builders & Pharma Deals

Proteins & Language

There’s a long-running analogy between proteins and natural language. Amino-acid sequences are strings over a 20-letter alphabet; they fold into secondary, tertiary, and quaternary structures in roughly the same way letters form words, sentences, and longer texts that carry meaning (in this case, function). Local context matters, but so do long-distance effects: amino acids that are far apart along the sequence can contact each other in 3D, much like dependencies between distant words or clauses in a document. Because of these parallels, researchers have been importing Natural Language Processing (NLP) tools into protein work for decades.

But the analogy has its limits. Proteins don’t have clean “word” boundaries, the underlying “grammar” is poorly understood, and evolution introduces all kinds of irregularities. At the same time, sequence databases have exploded in size, while structures and functional labels lag behind, creating sequence–structure gap. That mix of massive sequence datasets and limited structural or functional labels looks a lot like the setting where self-supervised language models first proved useful in NLP. So it is natural to try similar objectives on protein sequences, even though this isn’t a direct port of NLP into biology, rather, it’s more a case of borrowing inductive biases and training strategies to explore the protein sequence–function landscape in new ways.

Transformers are the latest step in that cross-over, where instead of hand-crafted features or shallow models, large PLMs are trained directly on raw sequences at scale. Some are set up to fill in masked amino acids and learn embeddings that have been used for tasks like contact prediction or estimating the effect of mutations. Others are built as generative models that can propose entirely new sequences and, in some cases, can be steered with broad tags for things like function or cellular compartment. In parallel, researchers are starting to look at the attention patterns inside these models as one possible way to probe how they relate distant positions in a sequence and what that might say about folding and design.

Overview of tasks in PLM. From: “Protein Large Language Models: A Comprehensive Survey” License: CC BY 4.0

Roughly, we can think of three generations of protein modelling:

1. First-generation methods hand-crafted features from amino-acid composition and position, with ad-hoc corrections for dataset biases, and powered early secondary-structure and signal-peptide predictors.
2. Second-generation methods coupled machine learning to evolutionary information from multiple-sequence alignments, starting in secondary-structure work and later becoming central to 3D prediction in models like AlphaFold2.
3. Third-generation methods train large PLMs directly on raw sequences with auto-encoding and autoregressive objectives, treating amino acids like words and learning reusable representations for sequence/structure prediction, function annotation, and protein design.

Players & Pharma Collaborations

There are multiple protein design companies developing general or specific PLMs. Many of them co-develop models with pharma enterprises or use tech cloud facilities to maintain and upscale the existing protein systems.