Abstract
To reduce meal logging burden in diet interventions, we fine-tuned OpenAI’s GPT-4o on 1269 Japanese meal photographs (train/val/eval: 912/252/105) to estimate nutrients, using weighed food records or dietitian estimates as ground truth, and compared it with 27 non–fine-tuned models and a human dietitian. Non–fine-tuned models did poorly for fiber. Most models did well for carbohydrates, protein, and energy, while performance for salt and fat varied by model. GPT-5.1 (minimal reasoning) and non-fine-tuned GPT-4o models both provided strong accuracy, though not universally better than dietitian performance. The fine-tuned GPT-4o model’s accuracy exceeded that of the dietitian for all nutrients, with the intra-class correlation coefficient for fiber of 0.79 (95% CI 0.782-0.797) greatly exceeded the dietitian performance of 0.68, validating the accuracy of the model.
Introduction
Improving diet is a key part of treating diabetes1,2 and other diseases, but interventions that rely on measurement of dietary intake suffer from low meal logging rates, even if using an app.3,4 Anecdotally, patients who log meals often photograph meals. Previous artificial intelligence (AI)-based approaches to assessing photographs of meals have identified the name of the dish and then referenced external food databases to estimate nutritional content. We are researching interventions to increase fiber intake, 5 and we are particularly interested in assessments of fiber. We assessed the accuracy of logging a meal by taking a photograph and performing a direct nutritional assessment using an AI model, GPT-4o, fine-tuned on a database of Japanese meals. We gave the same task to a human dietitian and to 27 non–fine-tuned AI models or sub-models.
Methods
There is a long history of assessing nutrition from photographs of meals,6-12 with a recent focus on generative AI. We previously assessed a non–fine-tuned AI’s performance for nutrition, 13 finding good performance for fiber but not for other nutrients. Our further research expanded the data set and added fine-tuning. This research extends our previously reported work. 14
In this research, we developed a one-shot chain of thought (COT) prompt that tasked the AI with returning a nutrition estimate based on a meal photograph. For fine-tuning, we developed a meal database (Figure 1). We examined meal photographs and associated nutrient assessments from three sources:
(a) Meals from a dietitian’s daily life over 90 days (unpublished data) and from their lab-prepared meals with natural looking photographs, 15 all with high-accuracy reference nutrition data from weighed food records (WFRs) or meal-label nutrition values.
(b) Meals from a research project 16 with meals cooked in a laboratory setting by home cooks with their unique recipes, with high-accuracy reference nutrition data via WFR.
(c) Meals from daily life of participants in an earlier study, 17 with fully natural settings but reference nutrition based on less accurate assessment of the meal photographs by dietitians, not WFR. Participants in the earlier study were informed about the study through the official website and were given the opportunity to decline the use of their data. To protect privacy, all images were anonymized and de-identified prior to analysis.
Fine tuning database with 1269 meal images from three sources. Description of the three sources, sample images, and details on the train/val/eval data split.
We assessed accuracy using intra-class correlation coefficients (ICCs) between the estimate and the reference value (WFR or dietitian estimate). All ICCs are ICC (2,1) two-way random effects, absolute agreement, single rater, 18 specifically ICC(2) in both the Python library pingouin and R. Intra-class correlation coefficients were calculated independently for each nutrient. As AI performance varies slightly run-to-run, we performed multiple runs for all but the highest latency/cost models reporting the mean ICC with an estimate of the 95% confidence interval (CI) based on the observed standard deviation (95% CI is mean ± 1.96 estimated SD). We used CIs for descriptive comparisons to understand estimate repeatability, and we did not do any assessments of statistical significance of any differences in ICCs. We calculated ICCs for estimates of carbohydrates, fat, protein, energy, salt, and fiber. We also collected the mean latency to assess one image.
We examined the meals available from each source, assessing a total of 1923 images. We removed near duplicates. We picked a source-balanced set of 1269 meal photos – 422 from groups A and B, and 425 from group C. We then split the resulting 1269 meals into 912 meals for training (used to fine-tune the model), 252 meals for validation (used during fine-tuning to assess model progress), and 105 meals for evaluation (separate data used to evaluate the performance of the resulting model). We balanced the meal photos between sources (A, B, C) and meal types (breakfast/lunch/dinner/snack) and also chose meals to give similar nutrient ICCs for the non–fine-tuned model GPT-4o for training, validation, and evaluation subsets. This result yielded well-balanced subsets of meals for training (912), validation (252), and evaluation (105) (Table 1).
Image Balancing Approach.
We fine-tuned the GPT-4o model (gpt-4o-08-06-24) on the 912 train meals and the 252 validation meals, using standard OpenAI fine-tuning methods (Table 2). We then assessed the ICC of the resulting fine-tuned model’s assessments of nutrients in the 105-meal evaluation set. We similarly used this 105-meal evaluation subset to assess, via ICC, the performance of 27 non–fine-tuned commercial AI models (or model and reasoning level): 18 from OpenAI (OpenAI. Models. OpenAI.com. https://platform.openai.com/docs/models; accessed November 22, 2025) and 9 from Anthropic (Anthropic. Models Overview. Anthropic.com. https://docs.anthropic.com/en/docs/about-claude/models/overview; accessed November 24, 2025). For 21 non-reasoning models (GPT-4o/4o mini, GPT-4.1/4.1 mini/4.1 nano, GPT-5 [minimal, low, medium], GPT-5 mini/nano, GPT-5.1 [minimal, low, medium], Claude Sonnet 3.5/3.7/4/4.5, Claude Haiku 3.5/4.5), we used a COT prompt identical to that used in fine-tuning. For eight reasoning models (GPT-5/5.1 [high reasoning], Claude Opus 4/4.1/4.5, o1, o3, and o4 mini), we used a shorter non-COT prompt, as the models implement COT reasoning internally. We also tasked an experienced human dietitian to assess the evaluation subset.
Summary of Fine-Tuning Procedure and Model Configuration.
Results
Performance varied greatly across models and nutrients (Table 3, Figure 2). Comparing our fine-tuned GPT-4o model with the dietitian performance (Figure 2A), the fine-tuned model’s performance is similar but slightly better for fat, better for protein/energy/carbohydrates, and significantly superior for salt and fiber. Error plots of the fine-tuned model show good performance overall, with occasional outliers and with some tendency to err on the high side for low levels and err on the low side for high levels (Figure 3). The fine-tuned model is significantly better than the base GPT-4o model for salt, fat, and especially fiber – fine-tuning was effective. Comparing the performance of all evaluated models (Figure 2B), we found varied performance. For salt, some models did well, especially newer models. Performance varied for fiber and fat, with the fine-tuned GPT-4o model’s performance being significantly better than all models for fiber. Most models gave similar performance for the other nutrients (carbohydrates, protein, and energy).
Comparison of ICCs of Different Estimators Relative to Reference Nutrient Values.
95% CIs were not calculated for dietitian, GPT-5 high, GPT-5.1 high, o1, Claude Opus 4, and Claude Opus 4.1 because only a single run was performed for each.
95% confidence interval calculated by mean ± 1.96 (standard error of intra-class correlation coefficients’ estimates of five runs).
Comparison of ICCs for 28 AI models relative to a human dietitian. (A) ICCs for the fine-tuned GPT-4o and base GPT-4o models versus ICCs for the human dietitian. (B) ICCs for 28 models versus ICCs for the human dietitian. (C) ICCs for the latest OpenAI models and fine-tuned GPT-4o model versus ICCs for the human dietitian. (D) ICCs for the latest Anthropic models and fine-tuned GPT-4o versus ICCs for the human dietitian.
Plots of differences between estimate and reference versus reference for the fine-tuned GPT-4o model. Individual plots show results for the case achieving the median ICC for that nutrient. Red line indicates the mean error. Green lines indicate the 95% prediction interval. (A) Carbohydrates (g). (B) Protein (g). (C) Fat (g). (D) Energy (kcal). (E) Fiber (g). (F) Estimated salt (g) versus reference value (g).
Focusing on current OpenAI (Figure 2C) and Anthropic (Figure 2D) models, we see that the OpenAI models are well suited to this task, while the Anthropic models generally do poorly. For the OpenAI models, GPT-5.1 with minimal reasoning does well, especially for fiber, but falls short of the fine-tuned model. The current OpenAI models all do well at salt, do fairly well at fat and protein, and generally do well for carbohydrates and energy. Claude Opus 4.5 does well for all but fiber. The higher cost and latency of reasoning models does not translate into better performance. The cost-optimized models such as GPT-5 mini or nano do well for some nutrients.
We assessed the ICCs for the full 105-meal set, the 70-meal WFR-rated subset (i.e. A+B), and the 35-meal dietitian-rated subset (i.e. C) (Table 3). In general, the ICC performance was much better with the WFR-rated subset for all nutrients other than for fat for the fine-tuned model – for fat, the WFR-rated subset is slightly worse than the dietitian-rated subset.
Focusing on fiber and latency, key measurands for some diet interventions in our laboratory research, the models that had performance above the dietitian’s ICC of 0.68 (GPT-4o fine-tuned, GPT-4.1 mini, GPT-4o, GPT-5 minimal, GPT-5.1 minimal, GPT-5 mini, and Claude Sonnet 3.7) all had similar latencies of 2.9 to 4.9 seconds per image (Table 3, Figure 4), except for the nearly 20-second latency of GPT-5 mini. The GPT-5 medium/high models had extremely high latency (>50 seconds) with moderate performance. The most current reasoning models (GPT-5.1 high and Claude Opus 4.5) provided moderate performance, with Claude Opus 4.5 achieving low latency (under 6 seconds) while GPT-5.1 high had high latency (>20 seconds).
Fiber ICC relative to ground truth and latency for 28 AI models. Fiber ICC versus latency for fine-tuned GPT-4o, general models, cost-optimized models, and reasoning models.
Discussion
Our fine-tuned GPT-4o model has excellent accuracy, exceeding dietitian performance for all nutrients assessed, and our results provide good validation that this model is accurate, subject to the limitations discussed below. Estimating nutrition from a photograph is difficult, with challenges including identifying meal names, estimating portion sizes, and identifying ingredients and their quantities, with some contributors being invisible (such as salt) or hard to see (such as fat or added sugar). Our approach of fine-tuning on nutrients, and not on intermediate factors like portion size or ingredient identification, worked well. Our model broadly tracks the performance of the human dietitian, with moderate ICCs for salt/fiber/fat and higher ICCs for protein/energy/carbohydrates. Human dietitians provide feedback from meal photographs, with an accuracy that is useful in many applications, and our model meets and exceeds that performance. We expect our model to be useful in any situation where a human dietitian’s estimate would be useful, such as providing general feedback on dietary choices. We observed variability across nutrients, and caution in interpretation is warranted. Given limitations including portion size estimation challenges and our validation limitations, tools like this should be considered supportive rather than decision-determinative. However, given the observed short latencies and costs that are below $0.02/image even for the most expensive models, this supportive role should scale well to enable fast, inexpensive, and reasonably accurate meal logging for wide populations.
A strength of this study was the large number of meals (1269) and the diversity of meal photographs, though it is not clear how well this diversity matches the diversity in the broader real world. All meals were of food in Japan, and we would expect to need to fine-tune to other cuisines due to issues such as differences in diet and meal presentation as well as ingredient variability. Future work to validate our approach in other environments is warranted. We used nutritional databases and food labels for the WFR reference values, and these may include biases or errors. The heterogeneity of our WFR reference values and dietitian-estimated reference values, while reflecting real-world variability, might have induced errors or biases, and future work exploring this heterogeneity is warranted. We saw lower ICCs for the dietitian-rated subset of meals than the dietitian-rated subset for both the base model and, for all but fat, for the fine-tuned model. This raises the possibility that inaccuracies in the dietitian estimates are artificially suppressing the ICCs from what they would be if all meal labels had the higher accuracy of WFR measurements, so that the models are even more accurate than assessed in this work. Future work exploring how best to use the real-world meal images assessed by dietitians is warranted. We used a specific prompt design, and other prompts might achieve better results. Although we have strict separation between train/val images and the evaluation images, avoiding any true data leakage, with such a large number of images some similar meals are in both sets. To the extent that this represents reality in a population, this similarity should lead to true fine-tuning benefits. However, if this similarity is specific to this study sample, the training benefit from this similarity will not apply fully to a broader population, and we would expect less benefit from fine-tuning than shown in our results. We were unable to address issues such as meal complexity, portion size uncertainty, mixed dishes, unknown sauces, hidden ingredients, and image conditions; future exploration of these factors is warranted. Our evaluation set size of 105 images was determined as a set fraction of the overall image set size, and further work to explore the stability of results for different sized evaluation sets is warranted.
None of the non–fine-tuned models met or exceeded the dietitian performance for all nutrients, but many delivered good accuracy useful for some applications. Most did well for protein, energy, and carbohydrates. Many struggled with salt, fiber, and fat, where ICCs are lower overall for both the dietitian and the models, though newer models generally did better for salt than older models. A good all-around choice is GPT-5.1 with minimal reasoning, which exceeded dietitian performance for all but fat and protein. The non–fine-tuned GPT-4o also provided good performance, though below dietitian performance for fiber and fat. The latest Anthropic models did poorly at this task, though Claude Opus 4.5 did fairly well for all but fiber. We cannot explain why different families of models varied so much. One hypothesis is that some families used more and deeper training on Japanese sources of data, while others may have emphasized American versions, with their different nutrient makeup, of similar dishes. Further research is warranted. Reasoning models delivered only moderate performance and, given generally high latencies, are not good options for this task – the moderate performance here is somewhat surprising, and further research investigating this is warranted. Some of the ICC differences were small, and we did not assess statistical significance of any differences – future work to explore this aspect is warranted. Even large differences, such as the improvement in ICC for fiber from 0.68 for the dietitian to 0.79 for the fine-tuned model, may not be clinically significant in all cases, though we expect feedback using improved accuracy to incrementally improve intervention efficacy.
Conclusion
Our study provides objective measurements of the accuracy of estimating nutrients from meal photographs. Our evaluation data set supports comparison of current and future AI models, benchmarked to dietitian performance. Our initial fine-tuned GPT-4o model performs well, as do some non-fine-tuned models.
Footnotes
Acknowledgements
The authors thank Ryohei Nakada of the University of Tokyo for his help with the table and figures.
Abbreviations
AI, artificial intelligence; CI, confidence interval; COT, chain of thought; ICC, intra-class correlation coefficients; WFR, weighed food records.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by internal laboratory funds.
Declaration of Conflicting Interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: KW and DRL have ownership in WaShiLa Health, a start-up which seeks to apply this technology commercially.
Data Availability Statements
The data are available upon reasonable request for non-commercial use.
Code Availability
The prompts and the analysis code are available upon reasonable request for non-commercial use.