Scenarios & fixtures
Fixtures pair deterministic observation timelines with exact expected worlds and form the proof model behind tarski verify, blessing, query expectation sidecars, and proof artifacts.
A fixture is a deterministic observation timeline (.jsonl) plus the exact semantic
output it must produce (.expected.json). Fixtures are the behavioral contract for your
app, and they are how you check that your rules and invariants do what your experts
intended. Each fixture reads as a scenario: given this timeline, the derived world must
match the expected output. Fixtures run against the same deterministic evaluator that runs
production, and they have no step-definition glue layer, which is where classic BDD
drifts.
The proof loop
tarski replay fixtures/happy-path.jsonl
tarski verify
tarski verify --cross-engines
tarski verify replays every fixture on a clean database and runs these checks against
the replay: golden expected outputs, invariants, candidate-authority lints, provenance
bundles, replay determinism, generated scenarios, shadow reference agreement, redaction,
and composition checks where configured. --cross-engines replays every fixture through
both evaluation engines and requires bit-identical canonical output. See
Determinism & replay.
What a good fixture proves
- Observations are accepted in the expected order, and mappers emit stable atom batches.
- Facts assert and retract correctly; invariants reject invalid states.
- Candidate evidence reaches trusted facts only through
decision.accepted.*, and proposal evidence reaches reviewed intents only throughdecision.approved.*. - Rejected, refuted, and review-required decisions stay auditable.
- Unsafe paths produce no executable effects.
- Provenance exists for every candidate, proposal, decision, fact, and intent.
- Redaction prevents secret leaks in exports, and deterministic replay reproduces the result on a clean database.
Recommended coverage for a real app: the happy path, a contradiction path, a policy bypass attempt, a human-review path, multi-agent coordination, stale provider data, retry/reconcile, and redaction.
Blessing expected outputs
Tarski writes expected outputs from verified replays; you don’t write them by hand. Bless a fixture to generate its expected output:
tarski test --bless fixtures/happy-path.jsonl
Blessing refuses to write output that can’t be trusted. It refuses when replay or package
load fails, when the product and shadow evaluators disagree, when repeated runs are
non-deterministic, when redaction finds secret material, or when a recorded query result
carries stale or unverified identity. Suite-wide rewriting requires an explicit --all.
When a golden comparison fails, the actual output is rendered in the shape of the
expectation file, so the next edit is mechanical.
Query expectation sidecars
Fixtures can also assert declared-query behavior with a <fixture>.queries.json sidecar.
A sidecar binds parameters, pins a committed head (or uses the final head), and asserts
canonical rows or a result digest, a required typed denial before any worldview read, and
a row-support shape (supporting fact ids, aggregate counts, or absence witnesses). Query
check results are recorded in the verification bundle as audit evidence.
For agent-relay fixtures, a template generates a complete request/result lifecycle:
tarski fixture template query-relay <query-name>
tarski test --bless --refresh-recorded-digests fixtures/query-relay.jsonl
Digest refresh re-derives every recorded digest from verified replay. It cannot make a missing request or an unacknowledged result change pass, and each refusal names the field and shows both values.
From production misbehavior to fixture
When a contained agent misbehaves, provenance identifies the specific observations that caused it. You promote that timeline into a fixture as a counterexample, an authoring agent iterates until the fixture passes, and verification confirms the fix across the whole corpus. Each new fixture covers a gap that production exposed. Fixtures are deterministic, so they do not flake.
Actor context and expected write denials
Use a fixture.write_policy_attempt control record to run the real pre-append boundary
with fixture actor context and assert a typed denial. The control record and denied
payload never enter the lineage. See
Write policies for
the JSONL shape.
Pair a denied attempt with an admitted one after the relevant access observations. This proves the policy reads current ontology-derived authorization and does not depend on a test-only stub.
Portable worlds
tarski fixture export, fixture import, and fixture compare move canonical
observation-only worlds between clean evaluators. They preserve replay-relevant metadata
but reject precomputed facts, intents, effects, or provenance as imported truth. Use the
compare receipt to keep a demo seed and an app-generated world in correspondence. See
Codegen & portable fixtures.
Proof artifacts
Verification produces exportable evidence. Evaluation packages, observations, facts,
projections, verification bundles, graph bundles, and composition analysis all export as
artifacts (tarski export …). Generated artifacts are raw truth. Derived views may group
or relabel them, but they always preserve traceability to stable ids and provenance.