Artificial Intelligence Extracts Pancreatic Cyst Data with High Accuracy

Choubey AP, Eguia E, Hollingsworth A, et al. Data Extraction and Curation from Radiology Reports for Pancreatic Cyst Surveillance Using Large Language Models. J Am Coll Surg. November 2025.

Manually extracting findings for pancreatic cyst registries takes time and can limit research. Choubey et al. tested whether a large language model (LLM) could accurately extract surveillance data from radiology reports.

The single-center retrospective study included reports from 3,198 scans of 991 patients undergoing pancreatic cyst surveillance. Investigators used OpenAI’s GPT-4 platform with instructions that required no task-specific training examples. Results were compared with a manually annotated institutional cyst database.

The model extracted nine features, including cyst size and number, main pancreatic duct size and dilation, branch duct dilation, solid components, calcifications, pancreatic atrophy, and pancreatitis.

Accuracy for categorical variables ranged from 97% for solid components to 99% for calcific lesions. Accuracy was 92% for cyst size and 97% for main pancreatic duct size. However, agreement beyond chance was stronger for cyst size, with Cohen’s kappa values of 0.92 and 0.82, respectively. Identifying the number of cysts was less accurate, at 81%.

The results suggest that LLM-assisted extraction could help assemble longitudinal pancreatic cyst databases. The study assessed extraction from written reports, rather than image interpretation or prediction of cancer progression, at a single institution.