Guardrails and evaluations: how health AI is tested before you see it

A guardrail is a rule enforced outside the model. If an ingredient matches a stated allergy, it is filtered out by a database query before the model ever sees it. If a calorie target would fall below a floor, code raises it to the floor. The model cannot argue with either, because neither goes through the model.

The strongest guardrails are boring on purpose: plain code, plain data, and rules that can be read, tested, and audited.

An evaluation, or eval, is a fixed set of test inputs with known correct outcomes, run against the system every time something changes. Good eval sets deliberately include the awkward cases: users with multiple allergies, conflicting goals, missing data, and prompts that try to push the system out of bounds.

Because the set is fixed, results can be compared over time. If a new prompt or model makes any hard case fail, that is visible immediately.

Some failures are tolerable in small numbers, such as a slightly awkward phrasing. Others are not. For the absolute guardrails, such as serving an allergen, exceeding an intensity ceiling, or leaving a red-flag safety event unresolved, the bar is zero failures, and a single one blocks the release.

Foviq's project rules name these as absolute guardrails with a target of zero, not merely low.

Passing every test in a fixed set tells you the system handles those cases. It does not prove that it handles every case a real person will bring. That is why evals are paired with structural guardrails, validators on live output, monitoring after release, and a way to turn off a feature quickly.

Treat any claim of a health AI being risk-free with suspicion. The honest claim is layered: certain harms are prevented by design, others are tested for and monitored, and the remainder is disclosed.

If the model is unavailable, slow, or returns something that fails validation, a good system does not improvise. It falls back to a deterministic, pre-approved answer, such as a standard workout or a stock explanation. The fallback should itself be tested regularly, because it is the last line of defence.

Testing before release cannot anticipate everything, so responsible products keep watching afterwards. That means logging failures and near-misses, reviewing user reports, tracking how often fallbacks are used, and having a documented way to switch a feature off quickly if something goes wrong.

Every problem that is found should leave something permanent behind, such as a new test case, so the same mistake cannot return unnoticed.

Every AI-generated output should be labelled as such, carry a short note about how strong the underlying evidence is, and indicate when professional advice is appropriate. This is a requirement in Foviq's own product specification rather than an optional extra.