Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
162 lines (95 loc) · 10.6 KB

ARCHITECTURE.md

File metadata and controls

162 lines (95 loc) · 10.6 KB
Raw

Architecture

This document describes the implemented architecture of the LSST conda repackager as of its initial working state.

Problem statement

The LSST Science Pipelines use EUPS, a custom package manager, for both distribution and runtime environment management. A standard installation requires users to run lsstinstall, then eups distrib install, then source loadLSST.sh && setup lsst_distrib in every new shell. This is friction-heavy for HPC users who want to treat the stack as a black-box tool.

Conda already solves both problems EUPS solves — package distribution and environment activation — so we repackage the EUPS-installed products into a conda package that installs into standard conda paths.

Design choice: monolithic repack

We considered two strategies:

Per-package conda recipes — generate a conda recipe for each of the ~100 EUPS products, build them individually, and create a metapackage. This is more "conda-native" but creates a combinatorial dependency-solving burden, requires maintaining 100+ recipes as the dependency graph evolves, and was the approach tried by mjuric/conda-lsst circa 2016–2018 (which also still required eups setup at runtime).

Install-then-repack — perform a standard EUPS install, relocate all files into conda-native paths, and package as a single conda artifact. This is simpler, faster, and eliminates EUPS from the user's runtime entirely.

We chose the monolithic repack. The tradeoff is that users cannot install individual LSST products — they get the full lsst_distrib stack. In practice, HPC users want the full stack anyway.

System components

┌──────────────────────────────────────────────────────────────────┐
│  build-lsst-conda.sh  (orchestrator)                             │
│                                                                  │
│  1. setup-conda.sh          Isolated local Miniforge3            │
│  2. install-patchelf.sh     Local patchelf binary                │
│  3. lsstinstall + eups      Standard LSST install in container   │
│  4. lsst_relocator.py       Flatten into conda prefix layout     │
│  5. conda-build             Package into .conda                  │
│  6. conda index             Update channel repodata.json         │
│                                                                  │
│  Output: local conda channel with lsst-distrib package           │
└──────────────────────────────────────────────────────────────────┘

Component details

setup-conda.sh

Ensures a clean, isolated conda installation exists at $SCRIPT_DIR/miniforge3. On first run, downloads and installs Miniforge3. On subsequent runs, activates the existing install.

Key behavior: before doing anything, it deactivates any active conda environment, unsets all conda/mamba environment variables (CONDA_EXE, CONDA_PREFIX, CONDA_DEFAULT_ENV, MAMBA_EXE, etc.), and strips all conda-related paths from $PATH. This prevents lsstinstall from discovering the user's personal conda environments.

Also installs conda-build into the base environment if not already present.

install-patchelf.sh

Downloads a prebuilt patchelf binary from the NixOS/patchelf GitHub releases into $SCRIPT_DIR/patchelf/bin/. Idempotent — skips if already installed and functional. No root required.

build-lsst-conda.sh

The top-level orchestrator. Accepts --tag, --channel, --product, --build-dir, and --keep-build arguments.

Path resolution. Because source loadLSST.sh and setup lsst_distrib aggressively modify $PATH and environment variables, all external tools (conda, conda-build, patchelf) are resolved to absolute paths before the LSST environment is sourced. These are exported as $CONDA, $CONDA_BUILD, and $PATCHELF for use by later stages.

LSST environment compatibility. LSST's shell scripts do not tolerate set -u (nounset). The script runs set +u before source loadLSST.sh and leaves it off for the remainder of execution.

Product manifest. After setup lsst_distrib, the script dumps a manifest file by iterating eups list -s output and resolving each product's directory via indirect shell variable expansion (${!env_var}). The manifest is a pipe-delimited file: name|version|ENV_VAR|/path/to/dir, one product per line. This is passed to the relocator, which avoids having to call EUPS from Python.

Idempotency. If a package with the same name and version already exists in the channel directory, the build is skipped.

lsst_relocator.py

The core of the system. Takes a product manifest and an output directory, and merges all EUPS product trees into a flat conda-compatible layout.

Product discovery

Products are loaded from the manifest file generated by the build script. Each line provides the product name, version, PRODUCT_DIR environment variable name, and installed directory path. A fallback mode scans environment variables matching *_DIR that point into EUPS-managed paths.

File relocation

For each product, files are relocated by category:

Python modules (<product>/python/) — Copied to lib/python3.X/site-packages/. The top level of python/ is iterated selectively: only actual Python package directories are copied. Build system files (meson.build, pyproject.toml, setup.py, README.rst, VERSION) and non-package directories (build-release/, doc/, tests/, src/) at the python/ root are skipped.

Shared libraries (<product>/lib/*.so) — Copied to lib/. RPATHs are patched using patchelf to $ORIGIN:$ORIGIN/../lib, making them relocatable. The $PATCHELF environment variable provides the absolute path to the patchelf binary.

Executables (<product>/bin/) — Copied to bin/. Python shebangs are rewritten to #!/usr/bin/env python3. Files are made executable.

Headers (<product>/include/) — Copied to include/. Needed if users want to compile against the stack.

Resource files (<product>/{policy,config,data,schema,pipelines}/) — Copied to share/lsst/<product>/. These are files accessed at runtime via PRODUCT_DIR environment variables or lsst.utils.getPackageDir().

Catch-all resources — Any directory at the product root that isn't one of the standard categories (python, lib, bin, include) and isn't a known build artifact is copied to share/lsst/<product>/.

Build artifact filtering

Filtering operates at three levels:

  1. merge_tree (global) — Uses os.walk with in-place directory pruning. Directories named build-release, CMakeFiles, .pytest_cache, meson-private, meson-info, meson-logs, __pycache__, or .git are pruned (not descended into). Files with .o or .a extensions are skipped.

  2. Python root filtering — The top level of each product's python/ directory is iterated item-by-item. Known build system files and non-package directories are skipped rather than blindly copying the entire tree.

  3. Resource directory filtering — The catch-all resource copy skips directories named build-release, build, tests, doc, ups, .git, .github, __pycache__, and .pytest_cache.

Dangling symlinks

Some EUPS products contain dangling symlinks (notably ups/eupspkg). The relocator detects and skips these: is_symlink() and not exists().

EUPS metadata

The ups/ directory in each product contains EUPS-specific files (.table dependency descriptors, eupspkg build scripts). These have no role in a conda environment and are not copied.

Namespace packages

The lsst Python namespace is shared across dozens of sub-packages. When merging into a single site-packages/lsst/ tree, __init__.py files and their __pycache__ bytecode collide. These are expected duplicates (LSST uses namespace packages) and are silently overwritten without warning.

Activation scripts

After relocation, the script generates:

  • etc/conda/activate.d/lsst-product-dirs.sh — Exports a <PRODUCT>_DIR environment variable for every product, pointing to $CONDA_PREFIX/share/lsst/<product>. Also sets LSST_STACK_VERSION.

  • etc/conda/deactivate.d/lsst-product-dirs.sh — Unsets all of the above.

These scripts run automatically on conda activate / conda deactivate.

Conda recipe generation

A meta.yaml and build.sh are generated. The recipe pins the exact rubin-env version detected in the build environment (e.g. rubin-env ==12.2.0). The build script copies the relocated file tree into $PREFIX.

Conda build and channel indexing

conda-build packages the relocated files in the modern .conda package format. The --no-test flag is used to skip the test phase (tests can be run separately after install). The resulting .conda package is placed in the channel directory, and conda index regenerates repodata.json.

Dependency model

lsst-distrib==30.0.7
  └── rubin-env==12.2.0  (from conda-forge)
        ├── python==3.13
        ├── numpy, astropy, scipy, ...
        ├── libast, cfitsio, fftw, gsl, ...
        ├── boost, log4cxx, ...
        └── (all other external dependencies)

The lsst-distrib package contains only LSST-authored code. All external dependencies come transitively through rubin-env, which is maintained on conda-forge by the Rubin Observatory team. Binary compatibility is guaranteed because the EUPS install is performed against the same rubin-env version that the conda package declares as a dependency.

Runtime behavior

After conda activate, the user has:

  • All LSST Python packages importable via standard Python mechanisms (import lsst.afw, etc.) — they live in site-packages/.
  • All LSST shared libraries in $CONDA_PREFIX/lib — on LD_LIBRARY_PATH via conda's activation.
  • All LSST executables in $CONDA_PREFIX/bin — on PATH via conda's activation.
  • All <PRODUCT>_DIR variables set via the activation script — getPackageDir() works.

No EUPS commands are needed or available.

Version multiplexing

Multiple stack versions coexist as separate conda environments:

conda create -n lsst-v30  lsst-distrib==30.0.7
conda create -n lsst-v29  lsst-distrib==29.2.1

Each environment is fully isolated, including its own rubin-env and all transitive dependencies. This is arguably better than EUPS version multiplexing because the entire dependency tree is isolated, not just the LSST products.