[D] Working on a photo-based calorie tracker app

Hey,

I’m building a photo-based calorie tracking app. Apps like CalAI already do this, but from what I’ve seen they often struggle with mixed dishes, portion size estimation, and general hiccups with calorie estimates.

I’m trying to approach it a bit more seriously from an ML perspective and i want to hear your thoughts. I really want to make the scan part as accurate as possible. I don’t want it to be something simple as an OpenAI API call. I’m wondering if there is another approach for this using classic ML or specific food datasets which will give me an edge for the calculations.

Right now I’m experimenting with YOLOv8 for multi-food detection, and thinking about adding segmentation or some kind of regression model for portion/volume estimation.

Curious what others here think:

• Would you model this as detection + regression, or go full segmentation?
• Any good datasets for portion-aware food recognition?
• Is monocular depth estimation practical for something like this on mobile?

Would appreciate any thoughts, especially from anyone who’s worked on food recognition or similar real-world CV problems.