상세 보기
Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study
- Seo, Junhyuk;
- Kim, Taerim;
- Kim, Ju-Hyun
WEB OF SCIENCE
1SCOPUS
0초록
Background: Administrative costs in the health care system are driven in part by complex insurance eligibility determinations. Large language models (LLMs) are increasingly used for health insurance-related queries, yet their reliability for structured logical reasoning over coverage criteria has not been systematically evaluated. Objective: This study aimed to develop a benchmark for anticancer drug reimbursement eligibility determination and evaluate whether LLMs can reliably perform eligibility verification. Methods: We constructed a benchmark based on South Korea's National Health Insurance reimbursement guidelines for 3 gynecologic cancers (cervical, uterine, and ovarian), using a tristate adjudication framework (eligible, ineligible, and undeterminable). Three gynecologic oncology experts and a utilization review nurse validated the benchmark. Six LLMs from 3 providers (Anthropic, Google, and OpenAI) were evaluated using the official guideline document as input. Each case was evaluated 3 times per model, with final predictions determined by majority vote, and performance was compared across the 3 outcome classes. Results: The benchmark comprises 74 anticancer regimens with 222 cases. Overall verification accuracy ranged from 77.9% to 88.7% across the 6 models. Eligible and ineligible cases were classified with high recall (86.5%-98.6%), but undeterminable cases showed a marked decline across all models (44.6%-70.3%). Performance varied by cancer type, with uterine cancer showing the lowest undeterminable recall (16.7%), corresponding to the highest guideline complexity. Undeterminable cases were predominantly misclassified as eligible rather than ineligible. The tristate framework enabled logic-based error analysis of 235 incorrect predictions, revealing information gap-filling as the dominant failure pattern (n=196, 83.4%), followed by criterion misapplication (n=20, 8.5%) and false uncertainty (n=19, 8.1%). Subtype analysis indicated that information gap-filling errors were concentrated at hierarchical elements of the guideline. Sensitivity analyses showed that converting the guideline document to structured text degraded performance, while web search-enabled condition (0%-3.2% tool invocation across models) and structure-guided prompting did not produce significant changes from baseline. Conclusions: In this benchmark, LLMs classified clearly eligible and ineligible cases with relatively high recall but showed limited reliability on undeterminable cases. The dominant error pattern was information gap-filling, in which models inferred eligibility rather than withholding judgment. These findings indicate that LLMs, in their current form, should be deployed as supervised decision-support tools rather than as independent adjudicators in reimbursement review.
키워드
- 제목
- Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study
- 저자
- Seo, Junhyuk; Kim, Taerim; Kim, Ju-Hyun
- DOI
- 10.2196/95877
- 발행일
- 2026-06-15
- 유형
- Article
- 권
- 28
- 언어
- ENG
- 출판사
- JMIR PUBLICATIONS, INC
- 발행국가
- 캐나다
- ISSN
- E 1438-8871
P 1439-4456