Application of Large Language Models in Clinical Research: A case of Clinical Trial Eligibility Extraction from Unstructured Clinical Notes

Abstract

Background: Clinical research depends on unstructured text such as progress notes, pathology reports and trial protocols, which has traditionally required manual chart review or brittle rule-based natural language processing (NLP). Objective: To compare rule-based NLP, a fine-tuned encoder and decoder-only large language models (LLMs) for extracting phenotypes and clinical trial eligibility criteria from clinical notes. Methods: We evaluated MedSpaCy rules, ClinicalBERT, and open-weight 8B and 70B LLMs (zero-shot, few-shot chain-of-thought, and retrieval-augmented generation, RAG) on MIMIC-IV discharge summaries (N = 1,200, double-annotated, Cohen's kappa = 0.88), using a held-out test split of 300 notes and four target phenotypes. Results: The 70B model with RAG performed best (precision 0.938, recall 0.915, F1 0.926, hallucination rate 0.6%), ahead of ClinicalBERT (F1 0.846) and MedSpaCy (F1 0.717). Zero-shot inference with the 8B model produced a 6.8% hallucination rate. The best model required 6.15 s per note. Conclusions: LLMs combined with structured output schemas, retrieval and clinician verification outperform legacy NLP baselines for eligibility screening and phenotyping, but need human-in-the-loop review, source provenance and locally hosted, privacy-compliant deployment.

Keywords: large language models; clinical natural language processing; electronic health records; clinical trial eligibility; phenotyping; retrieval-augmented generation; pharmacovigilance; MIMIC-IV