Durable product records

Infrahub Sync provides one storage contract for compact product records and immutable artifacts. It is independent of Prefect: a product run can retain any number of purpose-labelled execution links after Prefect no longer has the corresponding flow-run detail.

The Sync HTTP API and its worker store records in PostgreSQL and immutable artifacts in S3-compatible object storage. Each process creates its own clients from the same environment settings. CLI and public Python callers access these records only through the Sync API.

Service execution reads no cache setting. Each stage works in a private scratch directory it creates and removes, and hands the next stage its plan through an internal artifact in the same object storage. Internal artifacts are never enumerated or served by the public run resource.

This is minimum product-projection configuration. It does not select a new provider type or add a release-management API.

For configured direct sync, the immutable review artifact is published after the saved plan commits and before the first destination write. If publication fails, synchronization stops before contacting the destination write surface and retains typed failure evidence on a terminal failed ProductRun. A reserved artifact remains unavailable until an exact publication retry completes it.

Configuration registry

The product store holds an append-only registry of declared configuration packages. A package is a strict, JSON-native envelope containing declared content only: adapter names, settings, and references to credentials. It never contains a credential value.

Registration validates before it persists, so an invalid package is refused and no version is written. validate re-reads a version that is already registered and checks it against the adapter declarations installed now — which is why a package accepted at registration can report findings later, after an adapter's declared setting surface changes.

A package carrying several independent defects yields one finding per defect, not just the first. Each finding carries a stable machine-readable code, a severity of error or warning, a JSON Pointer location into the declared package, and a message. An error prevents execution; a warning records declared intent or a qualification gap and does not.

Finding codes

The code is the stable part of a finding. The message wording and the printed layout may change; the code will not. Three enumerations are emitted: the declared-content core's fifteen error codes below, the four destination-schema codes (emitted only when validate is given an explicit destination-schema opt-in), and the three warning-channel codes. One location can report more than one of them — a credential declaration naming an uninstalled provider and an invalid identifier reports both at /credentials/<name> — but only ever from one check: when two checks reach the same pointer, the first to judge it is the one that reports.

Code	What it means	Where it points
adapter-role-mismatch	The adapter named for this role cannot serve it — most often a source-only adapter declared as the destination.	/configuration/source or /configuration/destination
adapter-validator-finding	The adapter's own configuration check failed, or handed back something unusable. Nothing the adapter said is carried into the message, so read the adapter's own findings for detail.	/configuration/<role>
credential-path-not-declared	A $credential reference sits somewhere that does not accept one. Usually a misspelled setting name, or a reference placed in schema_mapping or order.	the referencing node
endpoint-not-absolute	A url or base_url setting is not an absolute http or https URL.	the setting
endpoint-not-relative	An api_endpoint or endpoint setting carries a scheme or a host. It names a path beneath the absolute URL, not a second address.	the setting
finding-limit-reached	The package carries more than 256 defects and the rest were not reported. It is reported first, not last, and it is counted as a finding: when it fires the reported set holds 257 items, not 256. Its severity is the highest one among the findings it stands for: error when any suppressed finding is an error, warning when they are all warnings — so a cut can neither hide an error nor invent one.	the whole package
inline-credential-value	A credential-bearing setting holds a literal value instead of a {"$credential": "<name>"} reference. A package never contains a credential value.	the setting
malformed-credential-reference	A credential declaration's environment identifier is not a valid variable name, or a $credential node carries keys beyond the reference itself.	the declaration or the node
missing-adapter	No adapter is installed under the declared name — check the spelling and the case. That role's settings are not judged at all, because there is no declared surface to judge them against, so expect exactly one finding for the role.	/configuration/<role>
missing-store-capabilities	The declared store type is unknown and carries settings. An unknown store type with no settings declares nothing unsafe and is accepted.	/configuration/store
setting-contains-credential-material	An endpoint setting carries user information, a query string, or a fragment. Credentials belong in a reference, not in a URL.	the setting
setting-not-a-string	An endpoint setting is declared as something other than a string.	the setting
undeclared-setting	The adapter or store does not declare that setting name. One finding per name, each at its own pointer.	the setting
unknown-credential-provider	A credential declaration names a provider that is not installed. env is the installed provider.	/credentials/<name>
unknown-credential-reference	A $credential names a reference the package's own credentials block does not declare.	the referencing setting

An adapter's own configuration check keeps its own codes, which are outside this set — unsafe-rest-request-endpoint is the one shipped today.

Destination schema validation codes

These five codes are emitted only on the explicit opt-in: validate given a destination-schema options object. The default validate path judges declared content only, performs no schema read and no network I/O, and never emits them. All five carry an error severity.

Code	What it means	Where it points
destination-schema-mismatch	A declared schema mapping disagrees with the destination's schema snapshot: an undeclared kind, a field that is neither an attribute nor a relationship, a relationship reference on an attribute, or a static value whose shape disagrees with the relationship's cardinality.	the mapping entry, field, reference, or static value
destination-schema-read-failed	The destination schema could not be read: a timeout, refused credentials, an unreachable server, a rejected or unusable response, an unresolvable declared token, or unusable declared client settings. The message names the failure class.	/configuration/destination
destination-schema-unsupported-semantics	The destination schema was read, but it declares semantics outside the supported schema domain — an unknown relationship cardinality, a member shape the domain does not define, or a default no JSON encoding can carry. A run of this configuration refuses the same schema.	/configuration/destination
destination-schema-validation-unsupported	Schema validation was explicitly requested against a destination adapter that does not declare it. A missing capability needed to determine safety is an error, not a warning.	/configuration/destination
unsupported-destination-write	The configuration requests destination write operations the destination adapter does not declare support for.	/configuration/destination

A successful opt-in read also returns destination_schema_fingerprint: the full SHA-256 digest of the schema semantics this configuration consumes — each mapped kind, its DiffSync identifiers, its ordered human-friendly ID and uniqueness-constraint component paths, every mapped field's type and required/default/unique properties, and every mandatory-without-default field on those kinds. Unmapped destination growth and differences in schema delivery order leave it unchanged. It is null whenever no snapshot was read — the default path, a non-declaring destination, or a failed read.

Warning-channel codes

The warning channel is closed: warnings are limited to intentional omissions and explicitly unqualified optional features, and nothing else.

Code	Severity	What it means	Where it points
intentional-omission	warning	An omissions entry declares that destination content is intentionally not synchronized. The declared reason, when present, is carried verbatim.	/omissions/<index>
omission-contradicts-mapping	error	An omission names content a schema mapping also maps. A contradictory declaration is a package defect, not a preference; the error replaces the warning at that location.	/omissions/<index>
optional-feature-unqualified	warning	The optional incremental feature is declared against a source adapter whose capability declaration does not qualify it, so extraction silently runs full.	/configuration/incremental

Storage profiles

Service deployment

The Sync API and worker construct the PostgreSQL/S3 profile from these settings:

Variable	Requirement
INFRAHUB_SYNC_DATABASE_URL	Non-empty PostgreSQL connection string accepted by Psycopg for product records.
INFRAHUB_SYNC_S3_BUCKET	Non-empty S3-compatible bucket for immutable artifacts.
INFRAHUB_SYNC_S3_PREFIX	Optional object-key prefix; defaults to infrahub-sync.
INFRAHUB_SYNC_S3_ENDPOINT_URL	Optional absolute http or https URL with no userinfo. The value reaches Boto3 unchanged; Boto3 owns any narrower SDK compatibility.
INFRAHUB_SYNC_S3_REGION	Optional region passed to Boto3.

S3 credentials use Boto3's standard credential-provider chain. Configure credentials in the API, worker, CLI, or smoke-process environment that constructs the client. Sync does not define access-key or secret-key settings.

Constructing a service process initializes the PostgreSQL schema. The PostgreSQL role must have the DDL privileges required to create the product-record tables in its configured schema, plus permission to read and write their rows.

Test injection

The local SQLite/filesystem projection is an injected test seam. It is not a service deployment option or a public CLI/Python execution mode. In-process service and store tests can inject it directly:

from pathlib import Path

from infrahub_sync.product_store import local_product_projection

records = local_product_projection(Path("/var/lib/infrahub-sync/product-cache"))

The path must be absolute after ~ expansion. Relative paths are rejected, and the seam never falls back to the process working directory.

The underlying PostgreSQL/S3 library constructor accepts a PostgreSQL DB-API connection factory and an S3-compatible client implementing the small S3Client protocol (put, get, head, get_bounded, copy, and delete). head returns the stored object's length from metadata, and get_bounded returns at most one byte past the limit it is given, so an oversized object is refused without being buffered:

from infrahub_sync.product_store import production_product_projection

records = production_product_projection(
    connect=postgres_connection_factory,
    s3_client=object_client,
    bucket="sync-product-artifacts",
    prefix="production",
)

Psycopg and Boto3 are installed only by the service extra. The record-store and artifact-store protocols accepted by ProductProjection are internal implementation seams, not a public custom-provider compatibility contract.

Record and identity contract

ProductRun owns the stable Sync run_id, requested operation, immutable configuration reference, actor and audit links, product phase and outcome, timings, summary and results, write-safety evidence, artifact references, and Prefect correlations. Relational child tables store artifact references and Prefect execution links separately from the compact run row.

reconciliation_required is the run's one authoritative write-safety fact: a non-null Boolean, false on every new run, exposed directly on the public run resource rather than buried in results. It becomes true in the same transaction that makes a claimed apply or sync execution terminal as interrupted / ambiguous — whether that verdict is written by the worker itself, by liveness reconciliation, or by cancellation. The store derives the condition from the verdict and the execution's purpose rather than taking it as an argument, so no caller can write it false, and no result merge, replacement, or later terminal transition clears it.

Each PrefectExecutionLink records:

• a distinct flow_run_id;
• an optional deployment_id;
• its purpose or stage;
• its attempt number;
• optional last-observed state and timestamp.

Duplicate Sync run IDs and duplicate flow-run IDs within one record are rejected. A confirmed sync creates its own record. A reviewed-plan apply instead advances the original planning record and attaches its result artifacts to the same run_id; it does not create a second Sync identity. add_prefect_execution appends stage links as they become known without changing the Sync record's identity.

Every execution link names the unresolved mutation receipt that appended it, through an internal uniquely-constrained column that appears in no public resource. That relation is the one-append rule: a receipt appends at most one execution, an execution belongs to a submission still in flight, a write append additionally requires the run's write-admission row to name that exact receipt and that exact operation, and a run holding a write admission accepts no further append at all.

Mutations against a missing Sync run raise RunNotFoundError. Read operations continue to return LookupResult with reason="run-not-found", so absence remains a normal, non-exceptional lookup result.

create_run accepts an unfinished ProductRun, including initial actor, audit, summary, results, phase, and Prefect-link metadata. It rejects records that already have a finish timestamp, outcome, or artifact reference; those completion fields must be added through the publication and finish operations so their integrity checks cannot be bypassed.

Sync API mutations reserve a MutationReceipt unique by actor and SHA-256 digest of the client idempotency key. A receipt binds the operation, target, request fingerprint, reason, Sync run, opaque Prefect key, state, and exact accepted response. The raw client key is not stored. Run creation commits its receipt and unfinished product run in one relational transaction.

Reserving a stage against an existing run arbitrates it in that same transaction, after locking the run's authoritative row: a write reservation requires no existing write admission, no other unresolved plan, verify, apply, or sync receipt, and no unfinished execution, while a plan or verify reservation requires only that no write has been admitted. A cancellation is deliberately outside this arbitration, so a claimed write stays cancellable. A losing request never reaches Prefect; its receipt is stored already answered, carrying 409 run-execution-conflict, so replaying that client key replays the refusal. An accepted run receipt therefore carries a Prefect flow-run ID only when its stored response is a successful submission. AuditEvent records secret-safe actor, reason, operation, and outcome evidence for accepted mutations and refusals.

record_results updates retained result evidence without changing product phase, outcome, or finish time. Service verification uses this operation because verification is read-only for both the destination and product lifecycle.

Artifact publication and lookup

Artifact keys contain their SHA-256 digest and never change. Publication first reserves a non-readable relational reference for the run-owned artifact identity. It then writes artifact data and its manifest, and finally marks that exact reference published in a second relational transaction. The injected local seam commits data and manifest using an atomic directory rename. The service S3-compatible profile copies staged data to its immutable key and uses a create-only manifest put as the object-store commit point. A crash before the final relational mark leaves durable pending evidence rather than exposing the artifact. The run cannot be finished successfully while any pending publication exists. It can be finished with outcome="failed" so publication failure evidence reaches a safe terminal state while the pending artifact remains unavailable.

An S3Client implementation must make put(..., if_absent=True) an atomic, create-only write and translate an existing-key conflict to DuplicateArtifactError. This prevents a racing publisher from replacing the immutable manifest.

To recover from an interrupted publication, retry publish_artifact with the same run and artifact IDs, content, kind, and media type. Matching is performed after secret redaction. If data and manifest are already complete, the retry verifies them and performs only the missing relational mark. If the manifest is absent, the retry resumes object publication and then marks the reference published when needed. On S3-compatible storage, that resume re-uploads the already-redacted data to its same content-addressed object key before the create-only manifest commit. This exact-match repair also covers a relational row already marked published whose manifest is missing. Different content or metadata is rejected without changing the pending reservation or writing to the artifact provider; correct the caller input and retry the original publication. Once published, the artifact remains immutable and any further publication with that identity is rejected as already published.

A lookup returns LookupResult rather than treating absence as an empty record. Reasons include run-not-found, artifact-reference-not-found, manifest-unavailable, artifact-publication-incomplete, data-unavailable, artifact-expired, artifact-too-large, and integrity failures. Normal run lookup includes only published public references. An unavailable or pending artifact does not prevent the remaining product record from being read.

artifact-expired is a forward-compatible read behavior for references that already contain an expiry timestamp. The public publication operation does not set expiry, and the MVP does not expire or delete product records or artifacts automatically. Prefect retention is independent.

Internal artifacts and run bundles

An artifact is either public or internal. Public artifacts are the operator-facing content a run retains, such as the plan review document. Internal artifacts are the product's own stage handoff state: the run bundle one stage writes so the next can rebuild what it needs without sharing a filesystem.

Internal artifacts are not reachable from any public path. They are absent from the run resource's artifact_refs, absent from GET /runs/{run_id}/artifacts, and GET /runs/{run_id}/artifacts/{artifact_id} refuses them exactly as it refuses an identifier that was never published. The refusal is identical on purpose: bundle identifiers are fixed by the protocol, so a distinguishable refusal would confirm a guess. The public response schema does not carry a visibility field.

A run bundle is an uncompressed ZIP holding a manifest and the stage's files. Identical members always produce identical bytes, on any host and in any process, which is what makes a fixed checkpoint identifier safe to retry: an equal retry is the same publication, and an unequal one is rejected as a content conflict. Members are stored uncompressed because zlib output is not identical across zlib builds or Python versions, and the members are Parquet and JSONL that are already compressed.

Because the two checkpoint identifiers have one writer — the product stage that owns them — repeating a completed publication returns the existing reference instead of conflicting. That happens only once the request's metadata equals the committed reference and the stored bytes have been read back and validated complete against it, so a stage that lost the response rather than the write can repeat itself. Different content still conflicts, and a public artifact still conflicts on a repeat regardless of its identifier, because a public artifact has no single product writer.

Bundles are published and looked up through the same providers as public artifacts, so a deployment needs no additional storage settings.

The two checkpoints, and when each is published

A run has at most two internal checkpoints, and their membership is fixed:

Checkpoint	Members	Written by	Read by
run-bundle-v1-plan	plan/manifest.json, plan/operations.jsonl, and the A/*.parquet source snapshots the manifest declares	plan, sync	verify, apply
run-bundle-v1-final	the applied run.json	apply, sync	nothing

The plan checkpoint publishes before the run's plan becomes publicly reviewable. A run whose checkpoint publication fails has no plan-review artifact and no planned result, so an operator never reviews a plan a later stage on another worker could not read. A sync publishes the same checkpoint before its first destination operation, so a sync interrupted mid-write still leaves the plan its writes were derived from.

Verify and apply rehydrate that checkpoint into their own private scratch. Bounded read, stored size and digest validation, whole-archive validation, and extraction all complete before either stage constructs anything that could reach a destination. A checkpoint that is missing, oversized, corrupt, digest-mismatched, or another run's is refused at that boundary, having contacted nothing.

The final checkpoint publishes after confirmed guard release and before the run's success is committed. It is current-run evidence and nothing more: no stage reads it, and its presence never implies a run succeeded. Whether a write landed is answered only by a success transaction that actually committed. Once a destination operation may have started, any failure up to and including that commit — a final-checkpoint publication that fails, or a commit that stores nothing — finishes the run as interrupted and ambiguous with reconciliation required.

Storage limits

A bundle is at most 64 MiB and holds at most 1024 members, not counting its manifest. Both bounds are enforced on both sides. Writing past either is refused rather than truncated, and the size bound is applied to the completed archive, so the ZIP headers and central directory count against it exactly as they do when the archive is read back. Every archive the writer returns is one the reader accepts.

The member limit sits far above any real configuration — a checkpoint carries the plan's two files plus one Parquet file per extracted resource kind. It exists because the byte bound does not bound the entry count on its own: an archive of empty members stays small while still costing one entry each to read. The limit is applied before the reader builds any collection over the entries, and again before it builds the manifest's member table.

Reading is bounded three times: the committed reference's recorded size is checked before the provider is contacted at all, the provider checks the stored object's length from metadata before transferring it, and the transfer itself stops one byte past the bound. Stored size and SHA-256 are verified before any archive structure is read.

An archive is validated as a whole before extraction writes anything: member count, member path grammar, duplicate and case-colliding names, entry types, compression method, encryption, declared members against present members, per-member and total size, and manifest usability. The manifest's format version and member lengths must be plain integers and its lengths non-negative; a JSON Boolean is not a number here even though it compares equal to one. Extraction is all-or-nothing into a new directory, so a destination either holds the whole bundle or does not exist. A configuration whose handoff state would exceed either bound cannot use this path; the refusal is explicit.

Configuration safety baseline

A successful managed apply or sync records the source row counts its plan was computed against, and that it extracted in full. That baseline is stored per configuration and updated inside the same transaction that commits the execution's verdict, so a run's counts and its success cannot come apart. Only a completed / succeeded verdict updates it. A failed or ambiguous terminal leaves the previous baseline in place, because that is still the last state known good. A read stage never advances it. Counts replace rather than merge, so a resource the configuration no longer extracts stops being compared against.

Nothing reads the baseline yet. The managed write path performs no row-count refusal, no lookup before dispatch, and offers no override or lockout; the baseline is the durable input a later guardrail feature would need.

Secret boundary

Pass the credential values collected by collect_secret_values to mutation, audit, run, execution-link, result, artifact, and finish operations. Values are redacted from nested record data and from public artifact bytes before either provider is called.

Internal artifact bytes are stored exactly as supplied. Redaction rewrites a byte sequence wherever it occurs, which inside a binary archive corrupts the container instead of protecting anything, and would make a bundle's digest depend on whether a configured secret happened to appear in compressed Parquet. No internal artifact is served to a client, and its integrity is established by stored size and digest before anything parses it. Treat a run bundle as internal state and do not hand one to a caller.

Do not put secrets into identifiers; credentials should continue to come from environment variables or a secret manager.

Reproducible sizing evidence

The sizing test repeats VAL-8's fixed-density 88k retained-artifact shape and the smaller 10k workload from the same source. Decimal bytes reproduce the published Parquet sizes: two payload stores, two hash indexes, and one plan. The separate 12.61 MB raw JSONL current-side wire measurement is not a retained artifact in this fixture.

Run:

uv sync --extra dev
uv run pytest -q -s tests/product_store/test_sizing.py

Observed on macOS with Python 3.13.3 on 2026-08-09:

Fixture	Records (baseline/current)	Artifact payload	Manifests	Relational store	Total files	Total logical bytes
VAL-8 88k	88,117 / 87,868	23,015,700 B	2,732 B	40,960 B	11	23,059,392 B
Representative 10k	10,051 / 10,023	2,722,700 B	2,877 B	40,960 B	11	2,766,537 B

The test fixture stores ten object files (data plus manifest) and one SQLite run/reference/link database. Filesystem allocation, database page size, and service object-store metadata can change physical billing; the payload and manifest byte counts are the portable sizing inputs.