Provenance and Reproducibility

Provenance Goals

ClearEx provenance is designed to answer:

  • what was run,

  • with which exact parameters and backend settings,

  • against which input data,

  • producing which latest output references.

Storage Layout

Within a canonical OME-Zarr store:

  • run records: clearex/provenance/runs/<run_id>

  • latest output references: clearex/provenance/latest_outputs/<analysis>

  • structured audit logs: clearex/provenance/event_logs/<run_id>.jsonl

  • public image collections: root source HCS collection and results/<analysis>/latest for image-producing analyses

  • internal image outputs: clearex/runtime_cache/results/<analysis>/latest

  • ClearEx-owned non-image latest outputs: clearex/results/<analysis>/latest

Run records are append-only. Large image outputs remain latest-only in the runtime-cache/public-output split to control storage growth.

Run Record Content

persist_run_provenance stores:

  • run identifiers, index, status, and timestamps,

  • input summary and input fingerprint hash,

  • normalized workflow settings,

  • effective Dask backend payload and chunk/pyramid settings,

  • effective spatial-calibration payload and canonical text form,

  • selected analyses and per-analysis parameters,

  • ordered step records and output references,

  • software metadata (package version, git commit/branch/dirty),

  • environment fingerprint (Python/platform/lockfile hash/argv).

  • structured audit-log manifest when a file-backed audit log was written.

Structured Audit Events

For file-backed workflow runs, ClearEx writes a sibling *.events.jsonl file next to the plain text log. Each JSONL record contains a schema/version, sequence number, UTC timestamp, execution id, optional run id, event type, status, operation, progress, message, and redacted metadata.

The workflow dispatcher emits events for workflow start/end, input resolution, store materialization, analysis sequence resolution, per-analysis start/progress/skip/complete/failure/cancel states, and provenance persistence. Sensitive credential-like keys such as tokens, passwords, secrets, API keys, private keys, and authorization headers are replaced with [REDACTED] before the record is written.

After provenance persistence succeeds, the final JSONL file is copied into the canonical store under clearex/provenance/event_logs/<run_id>.jsonl. The latest run record is updated with the audit manifest and its hash is recomputed so verify_provenance_chain still validates the run history.

Hash Chain Integrity

Each run includes chained integrity values:

  • prev_hash: prior run hash,

  • self_hash: hash of current run record payload.

verify_provenance_chain recomputes hashes and validates chain continuity, returning (is_valid, issues).

Latest Output References

register_latest_output_reference writes lightweight metadata for discovery:

  • analysis name,

  • latest component path,

  • update timestamp,

  • optional producing run_id,

  • optional output metadata payload.

This decouples large arrays from the append-only provenance history while keeping latest output pointers searchable.

Registration latest output references point to metadata-only results under clearex/results/registration/latest. Affine transform/layout arrays remain the stable compatibility contract; opt-in deformable runs also record the deformable parameter payload and the optional displacement-lattice metadata needed for reproducible fusion.

Spatial Placement Reproducibility

Store-level Navigate placement metadata is part of the reproducibility record:

  • canonical store metadata persists spatial_calibration in clearex/metadata,

  • workflow provenance stores the effective payload, canonical text form, and whether it was explicitly supplied by the operator,

  • visualization latest metadata stores the effective spatial calibration used for multiposition placement.

This keeps historical runs interpretable even when microscope stage axes do not match camera/world axes.

History Summaries and Dedup-Aware Execution

summarize_analysis_history provides per-analysis history summaries and parameter-match checks. Runtime orchestration uses this for dedup-aware execution:

  • if a matching successful run exists and required output components are present, the operation can be skipped;

  • if required outputs are missing, the operation is re-run.

Current runtime applies this matching logic to flatfield, deconvolution, particle detection, registration, and other latest-only analysis steps.