Storage
Content-addressed blob storage with backend invariance, quotas, and GC, plus semantic tiering that keeps observation history at object-storage cost.
Tarski separates two storage concerns: blobs (large raw content referenced by observations) and observation history (the append-only log itself, kept forever and tiered by semantics).
Blob storage
Large raw content, such as uploads, fetched documents, and archived reports, is stored as
content-addressed blobs. A blob’s identity is sha256:<hex> of its canonical bytes,
computed before any backend write. Observations and receipts carry digest-only
BlobRefs.
Where bytes physically live is deployment configuration with no semantic authority.
Replaying the same timeline must produce byte-identical digests, receipts, derived facts,
and exports on every supported backend. The supported backends are sqlite (zero-config
local default), memory, mock (deterministic test double with scripted faults), and
s3 (any S3-compatible endpoint).
[storage]
blob_backend = "s3"
max_blob_bytes = 26214400
[storage.s3]
bucket = "my-app-blobs"
endpoint = "https://s3.example.com"
access_key_env = "MY_S3_ACCESS_KEY" # credentials by env-var name only
secret_key_env = "MY_S3_SECRET_KEY" # inline values are rejected at load
Selection priority is flag > env (TARSKI_BLOB_STORE) > file > sqlite, with a
--blob-store flag on dev, serve, replay, and verify. A configured S3 target
runs a write/read/head/delete probe at startup and fails closed. It never falls back
silently to another store. Managed Cloud ignores app-level storage configuration
entirely, and the platform owns the target there.
Apps reach storage through declared blob.put / blob.get capabilities. Typed errors
cover each failure: blob.size_limit_exceeded, blob.quota_exceeded,
blob.not_found, blob.corrupt_content, blob.tombstoned, blob.restricted,
blob.range_unsatisfiable.
Operate it with:
tarski stats # usage, egress, quota pressure
tarski gc --blobs --dry-run # unreferenced-blob collection, tombstoned
tarski storage migrate --to s3 # copy with digest verification
tarski blob put ./report.pdf # plus: hash, get, inspect
Client blob contract
@tarski/client 0.8.0 exposes generated upload, inventory, and metadata operations.
client.downloadBlob() handles full or ranged raw-byte downloads. Each request is
lineage-scoped and uses the same in-memory runtime or verified end-user bearer as other
SDK calls.
Possessing a digest does not authorize access to its blob. The runtime binds the authenticated organization, tenant, workspace/lineage, active redaction policy, and caller scope before touching storage. A digest from another tenant cannot be used to probe existence. Listing is filtered before pagination, and restricted content may leave its digest visible while withholding bytes.
The default per-blob cap is 256 MiB; deployments may set a smaller cap. The workspace hard quota is dedup-aware: uploading bytes already stored under the same digest is idempotent and does not consume quota again. Hosted transfer and egress quotas can also return typed, retryable throttling with a retry window.
Client-visible failures are stable:
| Code | HTTP | Meaning |
|---|---|---|
blob.size_limit_exceeded |
413 | This object exceeds the configured per-blob cap |
blob.quota_exceeded |
413 | The workspace hard quota would be exceeded |
blob.not_found |
404 | No authorized object resolves for the digest |
blob.restricted |
403 | Redaction/authorization permits metadata but withholds bytes |
blob.tombstoned |
410 | Physical purge is recorded by a durable tombstone |
blob.range_unsatisfiable |
416 | The requested range lies outside the byte length |
blob.corrupt_content |
502 | Stored bytes failed digest verification and were not served |
Use capability discovery before showing an upload control. An authority that omits the
generated blob operations resolves to unsupported_runtime, and the SDK does not guess an
older route.
Quota pressure reports as under, soft_exceeded, or hard_exceeded, with dedup-aware
usage. Studio’s Storage surface adds a blob browser, uploads, downloads, and
dry-run-gated reclamation.
Semantic tiering
The observation log is canonical truth, so Tarski keeps all of it, at object-storage cost. Semantic tiering makes that economical without a recency heuristic. Sealed, hash-chained archive segments carry semantic coverage, computed from the same canonical mappers that replay uses. Coverage records which namespaces, entity keys, and strata each segment can affect, under which evaluator digest.
Cold reads request history in ontology terms and fetch only matching segments. A read can ask for everything (replay), a namespace reduct (a declared query’s read set), or an entity neighborhood (provenance traversal). The rules are conservative:
- Pruning fails safe. Absent coverage, a foreign evaluator digest, or an overflowing neighborhood index always degrades to fetch-everything. Pruning is only an optimization, and correctness does not depend on it.
- Integrity fails closed. Digests verify before decoding, and tampered bytes fail.
- Identity is tier-invariant. Archive replay produces the same revision digest as hot replay, and pruned query execution produces the same result digest as full hydration. Every hydration returns a receipt (segments fetched, observations, bytes).
tarski stats folds validated segment manifests into per-lineage archive cost lines:
segment count, archived observations, archived bytes, and floor bytes-per-observation.