What does Haiku add to digital pathology?
A team from the University of Pennsylvania and Enable Medicine has introduced a new model called Haiku, a trimodal foundation model that links H&E images, spatial proteomics data from multiplexed immunofluorescence, and clinical text within one embedding space. The paper is available as an arXiv preprint titled Linking spatial biology and clinical histology via Haiku.
The idea is not just to add another data channel to an existing model. Haiku’s value comes from making morphology, spatial protein signal, and clinical context comparable and mutually retrievable. For pathologists, the practical question is clear: can an H&E image point toward possible protein markers? Can a clinical description retrieve relevant tissue regions or mIF patterns?
The data used to build the model
According to the authors, Haiku was trained on one of the largest published datasets in this area. The dataset includes more than 26.7 million spatial proteomics patches from 3,218 tissue sections linked to 1,606 patients. It covers 11 organs, 11 diseases, and 120 biomarkers, with alignment between H&E, multiplexed immunofluorescence, and clinical data.
This matters because many digital pathology models handle H&E alone, or add text and clinical data at a later stage. Haiku tries to train the representation itself on the relationship between the three channels, rather than treating each channel separately.
How does it link H&E, mIF, and text?
The model uses contrastive learning to bring matched samples from different modalities closer together and push unmatched samples apart. In this setup, an H&E image, an mIF image, and the associated clinical description are represented in a shared space. This enables cross-modal retrieval in three directions: from H&E to mIF, from mIF to text, and from text to images.
In the paper’s results, Recall@50 reached 0.611 in some retrieval tasks, compared with a baseline close to zero. This number does not mean the model is ready as a diagnostic tool. It shows that the learned representations carry a measurable relationship between morphology, spatial immunology, and clinical context.
Clinical outcomes and biomarkers without direct measurement
The team tested Haiku in downstream tasks including survival prediction in colorectal cancer, where it achieved a C-index of 0.737, a 7.91% relative improvement over unimodal models reported in the paper. The results also included treatment response prediction, with an AUPRC of 0.660 in melanoma and 0.775 in colorectal cancer.
The part most relevant to laboratories thinking about linking H&E with spatial biology is zero-shot biomarker inference. In this test, Haiku inferred signals across 52 biomarker channels with a mean Pearson correlation of 0.718, without including direct biomarker information in the text query. This does not replace laboratory measurement, but it may help triage hypotheses and identify cases or regions that deserve deeper measurement.
Counterfactual prediction: what happens if the tissue is fixed and the clinical context changes?
The paper presents an analytical framework that keeps morphology unchanged and changes only the clinical data inside the query. The goal is to observe the expected molecular shifts associated with a different clinical context. In a lung adenocarcinoma example, the model retrieved changes associated with better prognosis, including higher CD8 and granzyme B and lower PD-L1 and Ki67.
These results should be read as the authors frame them: exploratory signals for hypothesis generation, not mechanistic claims or substitutes for experiments. The strength of this type of analysis is that it gives researchers a structured way to ask: if the tissue stays the same, what molecular pattern changes when a specific clinical variable is modified?
Why does this matter for pathologists?
As spatial omics expands, the main problem is no longer only producing data. The problem is interpreting those data and linking them to tissue morphology and clinical context. Haiku proposes a different path: instead of viewing H&E, mIF, and clinical data as separate tables and images, they can be represented in one space that supports retrieval, comparison, and hypothesis generation.
For digital pathology, this type of model may be useful in three areas: selecting regions of interest before costly measurement, linking morphologic patterns to immune microenvironment signals, and building research tools that help interpret large cohorts instead of analyzing each data layer in isolation.
Clinical use still needs external validation, independent cohorts, and prospective evaluations that connect retrieval and prediction to actionable results inside the pathology workflow. The paper is strong as a research direction, not as a finished clinical product.
Source links
Paper on arXiv: https://arxiv.org/abs/2605.00925v1
Code and checkpoints: https://github.com/zhihuanglab/Haiku
Suggested internal links
- Foundation models in digital pathology
- Using spatial biology to interpret tumors
- How is AI changing H&E interpretation?