From images alone to deeper interpretation
Most AI systems in pathology rely on image analysis alone. A conventional MIL model takes digital tissue slides and classifies them according to predefined labels. The problem is that this approach is limited to the labels used during training, and it does not handle rare or unexpected cases well.
PathAI has taken a different approach. Instead of relying on images alone, its new Multimodal Pathology system combines the visual capabilities of its PLUTO-4 foundation model with language-based tissue descriptions. The goal is not only better accuracy, but a more flexible system that can adapt to new diagnoses without full retraining.
How the system works
The system uses contrastive learning to connect image representations from PLUTO with detailed histologic descriptions. The model learns to align visual features directly with descriptive text, creating a shared representation space between image and language.
This means the system does more than classify slides into fixed categories. It understands the relationship between what it sees in the image and how pathologists describe it. The result is a system that can support open-vocabulary prediction and search slide databases using natural language.
Clinical results
In dermatopathology, the system achieved a 4-6% improvement compared with image-only MIL models. In gastrointestinal pathology, the gain was larger, around 8-10%.
The important point is that this improvement came with a similar inference cost, so the better performance does not require extra computing resources during practical deployment.
Why this matters for practicing pathologists
The system opens the door to diagnosis guided by natural language. Instead of retraining the model whenever a new diagnostic category appears, the case can be described in words and the system can interpret the relationship. This brings AI closer to the way pathologists work, combining microscopic findings with linguistic and clinical knowledge.
Future directions
The ability to search hundreds of thousands of slides with natural language could change how pathology laboratories work. A pathologist could search for cases similar to a current case using a verbal description rather than predefined classifications.
The model can also handle rare cases that were not included in the training data, because the shared language space allows generalization to new diagnoses. This addresses one of the central problems in medical AI: overdependence on the data available during training.
Source: Pathology News – PathAI