CLI and Execution Modes ======================= Command Surface --------------- ClearEx installs the ``clearex`` command. Current primary arguments are: - ``--flatfield`` - ``--deconvolution`` - ``--particle-detection`` - ``--usegment3d`` - ``--channel-indices`` - ``--input-resolution-level`` - ``--shear-transform`` - ``--registration`` - ``--fusion`` - ``--display-pyramid`` - ``--visualization`` - ``--render-movie`` - ``--compile-movie`` - ``--volume-export`` - ``--mip-export`` - ``--file`` - ``--migrate-store`` - ``--migrate-output`` - ``--migrate-overwrite`` - ``--dask`` / ``--no-dask`` - ``--chunks`` - ``--stage-axis-map`` - ``--gui`` / ``--no-gui`` - ``--headless`` Execution Modes --------------- The entrypoint is GUI-first by default: - ``clearex`` attempts GUI launch. - ``--headless`` forces non-interactive mode. - ``--no-gui`` disables GUI launch attempts. - GUI launch failures (missing display/runtime issues) fall back to headless mode with warnings. GUI Setup Flow -------------- The first GUI window is an experiment-list driven setup flow: - ``Load Experiment`` adds one Navigate ``experiment.yml``/``experiment.yaml``. - ``Create Experiment List`` can either: - recursively scan a folder for Navigate experiment descriptors, or - reload a saved ClearEx list file (``.clearex-experiment-list.json``). - Drag and drop accepts individual experiment descriptors, folders to scan, and saved list files. - Selecting an entry in the list automatically loads the metadata panel. - Double-clicking a list item reloads that experiment's metadata explicitly. - The current ordered list can be saved back to a reusable ``.clearex-experiment-list.json`` file. - ``Spatial Calibration`` edits store-level world ``z/y/x`` placement mapping for the currently selected experiment without rewriting image data. - Spatial-calibration drafts are tracked per experiment while setup is open. - Existing canonical stores prefill the spatial-calibration control when metadata is already present. - Pressing ``Next`` batch-prepares canonical ``data_store.ome.zarr`` stores for every listed experiment that is missing a complete store, persists the resolved spatial calibration to every reused or newly prepared store, then opens analysis selection for the currently selected experiment. - ``Rebuild Canonical Store`` forces the listed stores to be rebuilt as canonical OME-Zarr outputs with the current chunk and pyramid settings. .. figure:: ../../screenshots/setup_dialog_experiment_list.png :alt: ClearEx setup dialog showing the themed experiment list pane and metadata panel :width: 100% Setup dialog with the experiment list pane, automatic metadata loading, and themed list-management controls. Workflow Selection ------------------ Each analysis flag is independent. You can run a single operation or multiple operations in one run. In the GUI analysis window: - ``Analysis Scope`` lets you choose which loaded ``experiment.yml`` is the active analysis target. - Enabling the batch checkbox runs the same selected operation set across all experiments from the setup list instead of only the selected one. - Per-dataset analysis widget state is restored automatically when available: saved GUI state is preferred, and ``Restore Latest Run Parameters`` falls back to the latest completed provenance-backed run for the active store. Examples -------- .. code-block:: bash # GUI-first default clearex .. code-block:: bash # Headless chained run clearex --headless \ --file /path/to/experiment.yml \ --flatfield --deconvolution --particle-detection .. code-block:: bash # Headless visualization against an existing canonical OME-Zarr store clearex --headless \ --file /path/to/data_store.ome.zarr \ --visualization .. code-block:: bash # Headless movie workflow against an existing canonical OME-Zarr store clearex --headless \ --file /path/to/data_store.ome.zarr \ --render-movie clearex --headless \ --file /path/to/data_store.ome.zarr \ --compile-movie .. code-block:: bash # Headless Navigate run with explicit stage-to-world placement mapping clearex --headless \ --file /path/to/experiment.yml \ --visualization \ --stage-axis-map z=+x,y=none,x=+y .. code-block:: bash # Headless volume export against an existing canonical OME-Zarr store clearex --headless \ --file /path/to/data_store.ome.zarr \ --volume-export .. code-block:: bash # Migrate one legacy ClearEx store clearex --migrate-store /path/to/legacy_store.zarr Spatial Calibration ------------------- Spatial calibration is a store-level mapping from world ``z/y/x`` placement axes to Navigate multiposition stage coordinates. - Canonical text form is ``z=...,y=...,x=...``. - Allowed bindings are ``+x``, ``-x``, ``+y``, ``-y``, ``+z``, ``-z``, ``+f``, ``-f``, and ``none``. - Default identity mapping is ``z=+z,y=+y,x=+x``. - ``none`` disables translation on that world axis. - ``THETA`` remains interpreted as rotation of the ``z/y`` plane about world ``x``. GUI and headless flows share the same normalized parser and storage policy: - GUI setup writes the resolved mappings to the listed experiment stores on ``Next``. - ``--stage-axis-map`` writes an explicit override to Navigate-materialized stores and existing canonical OME-Zarr stores before analysis starts. - If no explicit override is supplied, existing store calibration is preserved. - The mapping changes placement metadata only; image payloads remain unchanged. Interchangeable Routine Composition ----------------------------------- In orchestration, routines are composed from normalized ``analysis_parameters`` rather than hard-coded fixed order: - ``execution_order`` decides sequence among selected routines. - ``input_source`` decides which logical upstream component each routine reads. - ``force_rerun`` can override provenance-based skip behavior. This allows operators to rerun one stage, swap stage order, or run partial chains without changing the code path in ``main.py``. ``registration`` and ``fusion`` are intentionally split so operators can run transform estimation and final stitched rendering in separate executions with different backend sizing or worker-memory limits. ``registration`` always persists affine transform/layout metadata; when the experimental deformable follow-up is enabled it also persists a coarse displacement lattice that ``fusion`` consumes automatically. Older affine-only registration results remain valid fusion inputs. Input Source Resolution ----------------------- Runtime source aliases currently include: - ``data`` -> ``clearex/runtime_cache/source/data`` - ``flatfield`` -> ``clearex/runtime_cache/results/flatfield/latest/data`` - ``deconvolution`` -> ``clearex/runtime_cache/results/deconvolution/latest/data`` - ``shear_transform`` -> ``clearex/runtime_cache/results/shear_transform/latest/data`` - ``fusion`` -> ``clearex/runtime_cache/results/fusion/latest/data`` - ``usegment3d`` -> ``clearex/runtime_cache/results/usegment3d/latest/data`` - ``registration`` -> ``clearex/results/registration/latest`` (metadata-only; consumed by ``fusion``; may include optional deformable lattice metadata) - ``visualization`` -> ``clearex/results/visualization/latest`` (metadata-only; consumed by ``render_movie``) - ``render_movie`` -> ``clearex/results/render_movie/latest`` (metadata-only; consumed by ``compile_movie``) - ``compile_movie`` -> ``clearex/results/compile_movie/latest`` (metadata-only; terminal export metadata) Public OME image collections at the root and under ``results//latest`` exist for interoperability and visualization. Analysis kernels should not write into those public arrays directly. Volume Export ------------- ``volume_export`` is a visualization-family export workflow for one selected image-producing source component. - ``input_source`` resolves the upstream image-producing component to export. It accepts the usual runtime aliases such as ``data``, ``flatfield``, ``deconvolution``, ``shear_transform``, ``fusion``, and explicit internal component paths. - ``force_rerun`` bypasses provenance-based reuse and rewrites the latest export payload even when matching latest metadata already exists. - ``export_scope=current_selection`` exports one explicit ``(t, p, c)`` volume. - ``export_scope=all_indices`` exports every available ``(t, p, c)`` volume. - ``t_index``, ``p_index``, and ``c_index`` select the exported volume when ``export_scope=current_selection``. They are ignored for ``export_scope=all_indices``. - ``resolution_level`` can reuse discovered source-adjacent pyramid levels or generate missing deeper levels during export. - ``export_format=ome-zarr`` publishes ``results/volume_export/latest`` from the runtime cache. - ``export_format=ome-tiff`` writes in-store artifacts under ``/clearex/results/volume_export/latest/files``. - ``tiff_file_layout=single_file`` writes one current-selection ``ZYX`` OME-TIFF or one all-position BigTIFF with one ``TCZYX`` series per position. - ``tiff_file_layout=per_volume_files`` writes one ``ZYX`` OME-TIFF per exported ``(t, p, c)`` volume. - ``memory_overhead_factor`` is preserved in the normalized workflow payload as a scheduler hint, even though the current export implementation does not use it directly. - ``chunk_basis``, ``detect_2d_per_slice``, ``use_map_overlap``, and ``overlap_zyx`` are normalized compatibility fields inherited from other volume analyses and are pinned to 3D/no-overlap behavior for ``volume_export``. When a requested source component does not exist, runtime raises an input dependency error instead of silently falling back. The CLI currently exposes ``--volume-export`` as the operation flag. Detailed parameter editing is currently done through the GUI or a programmatic ``WorkflowConfig.analysis_parameters["volume_export"]`` payload. The full parameter-by-parameter reference lives in :doc:`../workflows/analysis-volume-export`. Progress and Run Lifecycle -------------------------- Execution progresses through these coarse stages: 1. Resolve workflow and inputs. 2. Materialize canonical OME-Zarr store when needed. 3. Execute selected analyses in resolved order. 4. Publish latest outputs and append provenance run record. GUI execution uses explicit progress callbacks and per-run logging in the resolved workflow log directory. File-backed runs also write a sibling ``*.events.jsonl`` structured audit log containing chronological workflow, input, analysis-step, progress, skip, failure, cancellation, and provenance events. Credential-like metadata keys are redacted before writing. When a run persists provenance in a canonical store, the final JSONL log is copied to ``clearex/provenance/event_logs/.jsonl`` and the run record stores its manifest, event count, checksum, execution id, and redaction policy. The ``Running Analysis`` dialog exposes an ``Open Dask Dashboard`` button. GUI dashboard launch behavior is: - ClearEx opens a localhost-only tokenized relay rather than the raw scheduler dashboard URL directly. - The relay is available only while a ClearEx-managed Dask client for the active analysis workload is alive. - The ``Running Analysis`` dialog enables its dashboard button only while the current run owns a live analysis client. - If ClearEx cannot reach the upstream dashboard or cannot start the relay, the GUI shows a warning and does not open the raw dashboard URL. The ``Running Analysis`` dialog also includes a ``Stop Analysis`` button. Cancellation is cooperative: ClearEx stops at the next progress checkpoint and persists the interrupted run in provenance with ``status=cancelled``. Visualization Keyframe Capture ------------------------------ When visualization launches napari, keyframe capture is enabled by default: - Press ``K`` to capture a keyframe. - Press ``Shift-K`` to remove the most recent keyframe. The keyframe manifest path defaults to: - ``/clearex/results/visualization/latest/keyframes.json`` and can be overridden with ``keyframe_manifest_path`` in visualization parameters. Each keyframe stores enough state to recreate the current scene for movie generation, including: - camera values (angles, zoom, center, perspective), - dims state (current step, axis labels, order, and 2D/3D mode), - layer order and selected/active layers, - per-layer display configuration (visibility, LUT/colormap, rendering mode, blending, opacity, contrast, and transforms when available). The GUI provides a popup editor (``Layer/View Table...``) for optional per-layer overrides with columns: - ``Layer``, ``Visible``, ``LUT/Colormap``, ``Rendering``, ``Annotation``. Movie Rendering and Compilation ------------------------------- ClearEx now separates movie generation into two explicit operations: - ``render_movie``: reconstructs the visualization scene from the keyframe manifest and renders PNG frames for one or more selected resolution levels. - ``compile_movie``: validates one rendered frame set and encodes it through ``ffmpeg`` into MP4, ProRes MOV, or both. Runtime storage: - ``render_movie`` latest metadata lives under ``clearex/results/render_movie/latest``. - ``compile_movie`` latest metadata lives under ``clearex/results/compile_movie/latest``. - Default in-store artifacts include ``/clearex/results/visualization/latest/keyframes.json``, ``/clearex/results/render_movie/latest/render_manifest.json``, ``/clearex/results/render_movie/latest/level__frames/frame_000000.png``, ``/clearex/results/compile_movie/latest/*.mp4``, and ``/clearex/results/compile_movie/latest/*.mov``. - ``output_directory`` can still redirect ``render_movie`` or ``compile_movie`` to an external export tree when desired. Practical guidance: - Use coarse levels such as ``[1]`` or ``[2]`` plus moderate frame sizes for preview renders. - Use level ``0`` and the final frame size for publication renders. - ``default_transition_frames`` around ``48`` is a good default for smooth motion. - ``mp4_crf`` in the ``16`` to ``24`` range is a reasonable review/final quality band, with lower values trading size for quality. - Rebuild timing and codec settings with ``compile_movie`` first, because it is much faster than rerendering napari screenshots. Captured napari ``Points`` and ``Tracks`` layers are serialized into the keyframe manifest and rebuilt during ``render_movie`` so common particle/track overlays can survive beyond the interactive session. ``render_movie`` now captures from a visible napari viewer by default. This avoids the empty-frame failures seen with hidden/offscreen capture while keeping CPU/software rendering and GPU-backed rendering usable. When ClearEx is already running inside a Qt GUI, the visible movie-capture viewer is launched in a dedicated subprocess so rendering does not touch Qt or OpenGL from the GUI worker thread. The CLI currently exposes ``--render-movie`` and ``--compile-movie`` as operation flags. Detailed movie parameter editing is currently done through the GUI or a programmatic ``WorkflowConfig.analysis_parameters`` payload.