(OR11-05-25) Leveraging AI to Enhance Nutrition Education for Cancer Survivors

Objectives: To assess the role that generative AI models could play in developing educational quizzes to assess patients’ understanding of nutrition and healthy lifestyles during their cancer survivorship journey.

Methods: Four AI models (GPTo1, Claude3.5 Sonnet, Grok2, Gemini1.5) were first provided nutrition information based on NCCN “Healthy Lifestyles” Survivorship Guidelines. (1) Using that info, each model was asked to design a 20-question quiz across 7 domains at a patient comprehension level. Quiz quality was evaluated on four criteria (style, topics, content adherence, clarity) by each model using a 10-point scale. (2) Each model’s answer key was verified for accuracy against the original info. (3) We then compiled a 20-question "clinic-ready" quiz for cancer patients from the highest-rated questions and tested fresh AI model instances (without prior nutrition info) on nutrition knowledge, accuracy, and consistency across 30 attempts per model (600 questions each).

Results: (1) Quiz quality scores were highest for GPTo1 (9.0), followed by Claude3.5 (8.0), Grok2 (7.2), and Gemini1.5 (7.0), with some variations across style, topics, content, and clarity. (2) All four models produced correct (100%) answer keys to their quizzes, with no “hallucinations” (made-up answers). (3) On the “clinic-ready” quiz, Claude3.5 achieved 100% accuracy on all attempts (30/30), followed by Grok2 (15/30), Gemini1.5 (10/30), and GPTo1 (4/30). Lowest single-attempt accuracy was 100%, 95%, 85%, and 85%, respectively. The “Specific Populations” guideline domain had the worst performance, at 100%, 50%, 50%, and 70%, respectively.

Conclusions: AI models were able to develop high-quality assessment quizzes to evaluate patient knowledge based on cancer nutrition information provided to them, with some variations between models. The accuracy of their independent clinical nutrition knowledge on a “clinic-ready” patient quiz was reasonable, but the consistency in answers varied significantly. These findings suggest the potential of AI models as “tools” for helping to develop clinical patient education materials, though further validating their reliability, accuracy, and sourcing before deploying them as “patient educators” in the future will be crucial.

Funding Sources: None