Architecture Overview

Design Goals

The runtime architecture is built around these constraints:

  • GUI-first operator workflow, with fully supported headless execution.

  • Canonical persisted store format is OME-Zarr v3 (*.ome.zarr).

  • Public image interoperability uses OME-Zarr HCS collections.

  • ClearEx internal analysis arrays keep canonical (t, p, c, z, y, x) layout for execution kernels and workflow chaining.

  • Metadata-only spatial calibration for Navigate multiposition placement.

  • Deterministic latest-output publication for image and non-image artifacts.

  • Append-only, FAIR-oriented provenance records.

  • Shared configuration model between GUI and headless paths.

Layered Runtime Model

ClearEx is intentionally split into layers that can evolve independently:

  1. Interface layer: GUI in clearex.gui.app, localhost dashboard relay in clearex.gui.dask_dashboard_proxy, and CLI parsing in clearex.io.cli.

  2. Configuration layer: typed runtime schema in clearex.workflow.WorkflowConfig (plus DaskBackendConfig, ZarrSaveConfig, and SpatialCalibrationConfig).

  3. Orchestration layer: workflow entrypoint and execution coordinator in clearex.main.

  4. Data and metadata layer: ingestion in clearex.io.experiment, OME publication and migration in clearex.io.ome_store, and provenance persistence in clearex.io.provenance.

  5. Analysis layer: analysis routines (flatfield, deconvolution, shear transform, registration, fusion, display pyramid, particle detection, uSegment3D, visualization, render_movie, compile_movie, volume export, and MIP export) via clearex.<analysis>.pipeline.

End-to-End Execution Flow

At runtime, control flows through one orchestrator path:

  1. Build a WorkflowConfig from CLI arguments and/or GUI state.

  2. Optionally launch GUI and let the operator finalize settings.

  3. Resolve acquisition source data or target store.

  4. Materialize or validate canonical OME-Zarr store state.

  5. Resolve analysis sequence and per-operation logical inputs.

  6. Run selected analyses against runtime-cache image components.

  7. Publish public OME image outputs and append one provenance run record.

Operational Invariants

These contracts are stable and expected by multiple modules:

  • Canonical public source image collection is the root OME HCS layout: A/1/<field>/<level>.

  • Canonical internal source component is clearex/runtime_cache/source/data.

  • Canonical internal image shape is always six-dimensional in (t, p, c, z, y, x) order.

  • Internal multiscale source levels are stored under clearex/runtime_cache/source/data_pyramid/level_<n>.

  • Public image-producing analysis outputs are published under results/<analysis>/latest.

  • Internal image-producing analysis arrays live under clearex/runtime_cache/results/<analysis>/latest.

  • volume_export is hybrid: OME-Zarr runs publish results/volume_export/latest; OME-TIFF runs keep artifacts under clearex/results/volume_export/latest/files.

  • Registration is metadata-only under clearex/results/registration/latest. Affine arrays remain the compatibility contract, and optional deformable lattices are ClearEx-owned non-image artifacts consumed by fusion.

  • ClearEx-owned metadata, provenance, GUI state, and non-image artifacts live under clearex/....

  • Store-level placement metadata is persisted in clearex/metadata["spatial_calibration"].

  • Provenance run history is append-only under clearex/provenance/runs.

  • Legacy root data, root data_pyramid, and results/<analysis>/latest/data layouts are migration-only and are not the canonical public contract.

Analysis Composition Model

Selected operations are not hard-coded into one fixed pipeline. Composition is driven by normalized per-operation parameters in analysis_parameters:

  • execution_order controls relative ordering between selected operations.

  • input_source controls which logical upstream image source an operation reads.

  • force_rerun lets operators bypass provenance-based dedup logic.

This allows one run to execute only one step, or a custom chain of steps, without changing orchestration code.

GUI-triggered Dask dashboard access is intentionally mediated through the localhost relay layer instead of opening scheduler dashboard URLs directly. That keeps browser access scoped to ClearEx-managed clients and lets the GUI fail with a warning when the upstream dashboard is unreachable.