Scenarios & fixtures

A fixture is a deterministic observation timeline (.jsonl) plus the exact semantic output it must produce (.expected.json). Fixtures are the behavioral contract: they read like scenarios — given this timeline, the world must look exactly like this — and they run against the same deterministic evaluator that runs production. There is no step-definition glue layer where classic BDD drifts.

The proof loop

tarski replay fixtures/happy-path.jsonl
tarski verify
tarski verify --cross-engines

tarski verify replays every fixture on a clean database and layers checks over the replay: golden expected outputs, invariants, candidate-authority lints, provenance bundles, replay determinism, generated scenarios, shadow reference agreement, redaction, and composition checks where configured. --cross-engines replays every fixture through both evaluation engines and requires bit-identical canonical output — see Determinism & replay.

What a good fixture proves

• Observations are accepted in the expected order, and mappers emit stable atom batches.
• Facts assert and retract correctly; invariants reject invalid states.
• Candidate evidence reaches trusted facts only through decision.accepted.*, and proposal evidence reaches reviewed intents only through decision.approved.*.
• Rejected, refuted, and review-required decisions stay auditable.
• Unsafe paths produce no executable effects.
• Provenance exists for every candidate, proposal, decision, fact, and intent.
• Redaction prevents secret leaks in exports, and deterministic replay reproduces the result on a clean database.

Recommended coverage for a real app: the happy path, a contradiction path, a policy bypass attempt, a human-review path, multi-agent coordination, stale provider data, retry/reconcile, and redaction.

Blessing expected outputs

Expected outputs are written from verified replays, never by hand:

tarski test --bless fixtures/happy-path.jsonl

Blessing refuses to freeze untrustworthy output: when replay or package load fails, when the product and shadow evaluators disagree, when repeated runs are non-deterministic, when redaction finds secret material, or when a recorded query result carries stale or unverified identity. Suite-wide rewriting requires an explicit --all. When a golden comparison fails, the actual output renders in expectation-file shape — the next edit is mechanical.

Query expectation sidecars

Fixtures can also assert declared-query behavior with a <fixture>.queries.json sidecar: bind parameters, pin a committed head (or use the final head), and assert canonical rows or a result digest, a required typed denial before any worldview read, and a row-support shape (supporting fact ids, aggregate counts, or absence witnesses). Query checks land in the verification bundle as first-class audit evidence.

For agent-relay fixtures, templates mint a complete request/result lifecycle:

tarski fixture template query-relay <query-name>
tarski test --bless --refresh-recorded-digests fixtures/query-relay.jsonl

Digest refresh re-derives every recorded digest from verified replay — it cannot launder a missing request or an unacknowledged result change, and refusals name the exact field with both values.

From production misbehavior to fixture

The counterexample flywheel: when a contained agent misbehaves, provenance pins the exact observations that caused it; that pinned timeline promotes into a fixture; an authoring agent iterates until it passes; verification confirms the fix across the whole corpus. The corpus grows exactly where reality found a gap — and a fixture, being deterministic, can never flake.

Proof artifacts

Verification is exportable evidence, not terminal output: evaluation packages, observations, facts, projections, verification bundles, graph bundles, and composition analysis all export as artifacts (tarski export …). Generated artifacts are raw truth — derived views may group or relabel, but always preserve traceability to stable ids and provenance.