(P14-001-25) Benchmarking Four Large Generative AI Models on Pediatric Nutrition Queries: A Comparative Study

Objectives: Generative AI is being rapidly adopted in consumer health, yet its accuracy in pediatric nutrition remains largely untested. We evaluated four current large language models (GPT o1, Claude 3.5 Sonnet, Grok 2, and Gemini 1.5) for their pediatric nutrition knowledge.

Methods: We compiled a test of 151 nutrition questions from the US Department of Agriculture (USDA) children’s quizzes, spanning 16 categories (e.g., Toddler Nutrition, Fruits, School Breakfast, etc.), and tested fresh AI model instances for nutrition accuracy and consistency across 30 test attempts per model, totaling 4,530 questions each (18,120 answers in total). We calculated overall statistics by model and by category.

Results: Over 30 attempts, GPTo1 had a mean accuracy of 94.8% (range: 92.0–96.7, SD: 0.98), Claude3.5 of 92.0% (90.0–94.0, SD: 1.14), Grok2 of 90.4% (90.1–91.4, SD: 0.41), and Gemini1.5 of 85.7% (83.4–88.1, SD: 1.04); Gemini1.5 performed significantly worse (p < 0.001) compared to the other three models. Performance across attempts was stable (p ≈ 1.00), and no model achieved perfect accuracy in any attempt.

Performance across nutrition categories varied significantly (p < 0.05) from 100% in some foundational categories (Fruits, Vegetables, Proteins), to ~75% in some specialized categories (Meal Components, Infant Nutrition, Child Nutrition Labels), with a mean category accuracy of 90.7%. Meal Components (SD: 9.9) and Child Nutrition Labels (SD: 9.7) had the worst consistency across attempts.

Conclusions: All four models show promise for child-focused nutrition knowledge and advice but exhibit variability and gaps in specialized pediatric categories. GPT o1 had the highest average accuracy, with some fluctuation from session to session, while Grok 2 demonstrated the best consistency. However, there were significant differences across categories, with all models achieving perfect 100% scores on some and relatively low scores (75–80%) on others. The “best” model depends on the goals of pediatric nutrition education; for general children’s and parents’ queries, GPT o1, Grok 2, and Claude 3.5 are all reasonable choices, but they are not reliable for specialized categories. These findings highlight the importance of evaluating accuracy and consistency in the clinical and educational applications of AI-driven pediatric nutrition resources.

Funding Sources: None