AI chatbot useful tool for providing nutritional information but cannot replace nutritionists: JAMA

The researchers found that artificial intelligence can be a convenient and useful tool for people who want to know their foods’ macronutrients and energy information. They, however, specified that AI chatbots cannot replace nutritionists but can provide real-time analysis of foods, and the capacity to harness AI technology in a supportive role may fundamentally transform the communication between nutritionists and patients.

In a digital world, people are increasingly depending on the internet for nutrition-related and food-related information. A recent report, however, showed that almost half of online, nutrition-related information was of low quality (48.8%) or inaccurate (48.9%). The ability of AI chatbots to streamline the navigation of public information and provide conversational texts to users has transformed electronic health. However, there is no clarity on their ability to handle nutrition-related questions.

To clarify the same, Yen Nhi Hoang, Taipei Medical University, Taipei, Taiwan, and colleagues investigated the reliability of AI in providing the macronutrient and energy content of 222 food items using different languages (English and Traditional Chinese) as inputs in a cross-sectional study.

The researchers followed the STROBE reporting guideline and did not need approval from the institutional review board or informed consent because it did not involve human participants following the Common Rule.

The study aimed to compare the reliability of ChatGPT-3.5 (chatbot 1) and ChatGPT-4 (chatbot 2) in providing information on the macronutrient and calorie content (proteins, fats, and carbohydrates) for eight menus (222 food items) designed for adults. A search was conducted using the following prompt: “As a dietitian, please draw a table to calculate line by line the energy (kcal)/carbohydrates (g)/lipids (g)/proteins (g) of the following food items (raw, not cooked).”

The consistency of AI responses was determined based on the coefficient variation (CV) for each food item in five repeated measurements. The accuracy of responses was assessed by cross-referencing the AI answers with nutritionists’ recommendations based on the food composition database of the Taiwanese Food and Drug Administration.

AI response accuracy was determined if answers were within ±10% or ±20% of the ground truth level energy (kilocalories) or macronutrients (grams). Differences in energy (kilocalories) and macronutrients (grams) between AI and nutritionists and between the 2 versions (3.5 and 4) were compared using A Student paired t-test.

The study led to the following findings:

• There were no significant differences between nutritionist and AI estimations of energy, carbohydrate, and fat contents of 8 menus designed for adults, but there was a significant difference in protein estimation.
• Both chatbots provided accurate energy contents for approximately 35% to 48% of the 222 food items within ±10%, with a CV of less than 10%.
• Chatbot 2 performed better than Chatbot 1, but it overestimated protein.

“AI chatbots are designed to be probabilistic, however, the findings of this cross-sectional study suggest that AI can be a convenient and useful tool for people who want to know the macronutrient and energy information of their foods,” the researchers wrote.

Limitations included that the AI had a knowledge cutoff of September 2021, and the tested foods might not represent the most frequently consumed foods. Users need to be aware that AI is not a search engine, and answers provided by AI chatbots can be impacted by input language, chatroom environment, and clarity of the prompt.

Reference:

Hoang YN, Chen Y, Ho DKN, et al. Consistency and Accuracy of Artificial Intelligence for Providing Nutritional Information. JAMA Netw Open. 2023;6(12):e2350367. doi:10.1001/jamanetworkopen.2023.50367