The Review Is the Feature

I did the thing you’re not supposed to do to your own AI feature: I benchmarked it, and it lost.

Twelve weighed meals from the public Nutrition5k dataset, fed to the meal-photo feature I’m building into Dayplan — photos only, no labels, no weights, no hints. Five calorie estimates landed within 20% of the reference. The model returned complete nutrition for eleven meals and refused the twelfth because it couldn’t identify a protein. Mean calorie error across the completions: 31.9%. Macros were worse — carbohydrates landed within 20% exactly once.

One run, one small fixed subset. This is a baseline, not a validation — but it’s a measured baseline, and the misses say more than the average does:

• A plate of plain brown rice was 112 g. The model said 150 g, plus “additional unidentified bits”. Portions are guesses wearing decimal points.
• Grapes became grape tomatoes. The calorie total was nearly right, which is its own warning: a plausible number doesn’t mean the food was identified.
• A salad came back as leafy greens, 14 kcal against a reference of 52 — the berries, granola and vinaigrette simply hid.

The tempting response is to chase the number: better prompts, a better model, wait a year. Maybe. But when a model is wrong a third of the time, the product question isn’t how do I make it right — it’s how do I make being wrong cheap.

So in Dayplan, nothing a photo produces is a record. It’s a draft: ingredients, estimated portions, totals, and whatever the model wanted to ask — 36 clarification questions across those twelve meals. I correct the food, adjust the grams, answer a question and request a re-estimate; it shows me before and after and keeps my manual edits. When the identity is right but the numbers deserve a better source, a USDA match replaces the guess. Only then does anything save. And when the model can’t identify something, it says unknown and refuses to produce a total — that refused twelfth dish wasn’t a failure, it was the system declining to make something up.

None of this makes the first guess more accurate. I wrote that in the dev note and it’s still true. What the review step does is make the error cheap: a bad estimate costs me one correction instead of quietly poisoning weeks of data I’d later trust.

That’s also why these numbers are in a blog post instead of a drawer. Every AI feature ships with a demo reel; very few ship with their error table. If you’re deciding whether to trust a feature — mine included — you deserve the measured number, not the marketing one. When I rerun the benchmark after the correction tooling matures, I’ll publish that too. Better or worse.

Dayplan is still on my dev server, being tested on my own meals. The model guesses. I review. That’s the feature.