Some AI Tool Now Reads Biopsy Images

Cancer is a multifaceted disease that typically requires costly genetic profiling for effective treatment. However, recent developments in deep learning have enabled more affordable predictions of genetic alterations from whole-slide images (WSIs). Despite the success of transformers in non-medical applications, their adoption for WSI analysis remains limited due to their complexity and the scarcity of large datasets.

Researchers at Stanford Medicine have developed an AI-based program that predicts the activity of thousands of genes in tumor cells using only standard microscopy images of biopsies. This tool, trained on data from over 7,000 diverse tumor samples, can predict genetic variations in breast cancers and forecast patient outcomes based on routinely collected biopsy images.

This software could rapidly identify gene signatures in patients’ tumors, accelerating clinical decision-making and potentially saving the healthcare system thousands of dollars by reducing the need for costly genetic tests.

Researchers recognized that gene activity within individual cells can subtly affect their appearance in ways that are difficult for the human eye to detect. To uncover these patterns, they turned to artificial intelligence. The team started with 7,584 cancer biopsy samples from 16 different cancer types.

Each biopsy was sliced into thin sections and stained using the standard hematoxylin and eosin method, which highlighted the cancer cells’ overall appearance. The researchers also had access to data on the tumors’ transcriptomes, which showed which genes the cells were actively using.

After integrating new cancer biopsy data with other datasets, including transcriptomic data and images from healthy cells, the researchers developed an AI program called SEQUOIA (Slide-based Expression Quantification Using Linearized Attention).

SEQUOIA successfully predicted the expression patterns of over 15,000 genes from stained biopsy images. In some cancer types, the AI’s predictions correlated with accurate gene activity data by more than 80%. The model performed better as more samples from a given cancer type were included in the training data.

It took several iterations to optimize the model, but eventually, it reached a level that made it useful in clinical settings.

Dr. Gevaert emphasized that doctors typically consider gene signatures—groups of genes associated with processes like inflammation or cell growth—rather than individual genes when making clinical decisions. SEQUOIA was particularly accurate at predicting the activation of these large genomic programs, surpassing its ability to predict individual gene expression.

To make the data more accessible and interpretable, the researchers designed SEQUOIA to display the genetic findings as a visual map of the tumor biopsy. This allows scientists and clinicians to visually explore how genetic variations may differ across various tumor areas, enhancing their ability to understand the tumor’s complexity and make more informed decisions.

To test SEQUOIA’s potential for clinical decision-making, Dr. Gevaert and his team focused on breast cancer genes that the model could accurately predict, which are already used in commercial genomic tests. For example, the FDA-approved MammaPrint test analyzes the expression of 70 breast cancer-related genes to calculate a risk score for cancer recurrence.

“Breast cancer has well-established gene signatures linked to treatment responses and patient outcomes,” Gevaert explained, making it an ideal test case for their model.

The team demonstrated that SEQUOIA could generate the same genomic risk score as MammaPrint using only stained biopsy images. This was validated across several patient groups, and those identified as high risk by SEQUOIA experienced worse outcomes—higher recurrence rates and shorter times before their cancer returned—confirming the model’s clinical relevance.

While SEQUOIA isn’t yet ready for clinical use, as it still requires clinical trial validation and FDA approval before it can guide treatment decisions, Dr. Gevaert and his team are refining the algorithm and exploring its potential applications. In the future, SEQUOIA could help reduce the reliance on costly gene expression tests, offering a more affordable alternative for assessing genetic variations in cancer.

Gevaert said, “We’ve shown how useful this could be for breast cancer, and we can now use it for all cancers and look at any gene signature out there. It’s a whole new data source that we didn’t have before.”

Journal Reference:

1. Pizurica, M., Zheng, Y., Carrillo-Perez, F. et al. Digital profiling of gene expression from histology images with linearized attention. Nat Commun 15, 9886 (2024). DOI: 10.1038/s41467-024-54182-5