V1 prep

[parashardhapola]

Scarf 1.0 redesigns storage and I/O for local and cloud workflows, adds faster feature-wise analytics, a new plotting API, batch-aware reference mapping, stronger integration metrics, and much broader tests.

Existing Zarr v2 stores remain supported. No store conversion is required.

---

Quick upgrade checklist

Step	Action	Required?
1	Create a Python 3.12+ environment	Yes
2	Install scarf[extra]	Yes
3	Open an existing .zarr and smoke-test your workflow	Yes
4	Remove Dask / Polars / Nabo / prenormed usage	If used
5	Switch renamed metrics / merge APIs	Recommended
6	Rebuild WNN graphs	If you use WNN
7	Adopt scarf.plotting	Optional
8	Add countsT to existing v3 stores	Optional (speed)
9	Run repack_zarr	Optional (I/O layout only)

---

Environment

Item	0.32.x	1.0.0
Python	≥3.11 (docs often said 3.10+)	≥3.12 (tested: 3.12 / 3.13 / 3.14)
Packaging	requirements.txt + setup.py	PEP 621 pyproject.toml
Install	pip install scarf[extra]	uv pip install scarf[extra]

conda create -n scarf_v1 python=3.12
conda activate scarf_v1
uv pip install 'scarf[extra]'

Dependency changes

Change	Packages
Upgraded	zarr>=3.1.2, numpy>=2, numcodecs>=0.16
Added	obstore>=0.5.1
Removed	dask, polars, pyarrow, joblib, importlib_metadata, gensim, cmocean

---

Existing Zarr stores

Question	Answer
Do I need to convert my store?	No
Can I open Zarr v2 with Scarf 1.0?	Yes
Will Scarf rewrite my counts on open?	No
Will Scarf add sharding to my old store?	No
Will Scarf add countsT automatically to old stores?	No
What do new writes use?	Zarr v3 + sharding + countsT

Optional performance tools

Tool	What it does	When to use
repack_zarr	Copies store to a new v3/sharded layout	Large remote/local I/O gains; you can afford a full copy
write_counts_t	Adds feature-major countsT to an existing v3 assay	HVG/markers are I/O-bound on v3 stores

uv run python -m scarf.tools.repack_zarr old.zarr new.zarr --profile fast_local
# or --profile cloud

repack_zarr does not automatically create countsT.

---

Breaking changes

Area	Old	New	Action
Python	3.11-compatible	3.12+ required	New env
Lazy arrays	dask.array.Array	ChunkedArray	Drop Dask-specific APIs
Nabo	NaboH5Reader / NaboH5ToZarr	Removed	Use other readers/writers
Polars metadata	to_polars_dataframe	Removed	Use to_pandas_dataframe
Prenormed markers	save_normed_for_query, use_prenormed, precache kwargs	Removed	Use standard marker search
Package imports	Star imports from submodules	Explicit __all__	Import from defining modules
New writes	Zarr v2 / float64 normed	Zarr v3 / float32 normed	Existing arrays unchanged
Synchronizer	ThreadSynchronizer used	Ignored under Zarr v3	Do not rely on it for new stores

Dask migration

# before
result = ds.RNA.rawData.sum(axis=0).compute()

# after
result = ds.RNA.rawData.sum(axis=0).compute(nthreads=ds.nthreads)

# or
from scarf.utils import controlled_compute
result = controlled_compute(ds.RNA.rawData.sum(axis=0), ds.nthreads)

Custom norm methods must be (Assay, ChunkedArray) -> ChunkedArray.

Kept names (no Dask underneath): dask_to_zarr, show_dask_progress.

---

Renames and deprecations

Old	Prefer	Notes
ZarrMerge	AssayMerge	Alias still works, warns
metric_integration	metric_label_concordance	Same ARI/NMI meaning
(assumed batch mixing)	metric_batch_mixing	New API for neighborhood mixing
metrics.integration_score	metrics.label_concordance_score	Alias kept
exclude_missing=True	missing_feature_policy="intersection"	Preferred explicit policy
run_coral=True	Avoid	Unsupported for harmonized refs
ref_mu / ref_sigma	Remove	Ignored; reference stats always used
ds.plot_* as primary docs path	scarf.plotting	Legacy plots still default

---

Behaviour / default changes

API / area	Change	Impact
Mapping cell_key=None	Resolves latest graph key, not always "I"	Pass cell_key="I" if needed
missing_feature_policy	"zero" / "intersection" / "error" (+ "reference_mean" for refs)	Explicit feature handling
mark_hvgs(max_cells=...)	Default None (unlimited)	Same effective behaviour as old np.inf
run_marker_search(gene_batch_size=...)	Default None (auto)	Better I/O alignment
make_graph(local_cache=...)	Default "auto"	Remote stores may stage normed data locally
run_pseudotime_scoring(component_policy=...)	Default "largest"	Disconnected graphs score largest component
nan_cluster_value	Must be int	str no longer allowed
Heatmap cmap	magma_r	cmocean removed
New normed writes	float32	Existing float64 arrays unchanged
H5ad sparse matrices	Stricter validation	Bad encodings raise ValueError

When to recompute

Result	Must rebuild store?	Recompute?
Counts / Zarr root	No	No
WNN / multimodal graph	No	Yes, for corrected weights
Markers	No	Optional (legacy still readable)
Harmonized atlas mapping	No	Prefer build_mapping_reference
UMAP / Leiden on unchanged graph	No	Usually no
Plots	No	No

---

Storage and performance

Resource budget knobs

Knob	Arg	Env var	Role
Memory budget	mem_budget	SCARF_MEM_BUDGET	Bounds tile/shard working set
Workers	nthreads / writer nthreads	SCARF_WORKERS	CPU + I/O concurrency
Working copies	working_copies	SCARF_WORKING_COPIES	Peak temporary buffers
Profile	zarrProfile	SCARF_ZARR_PROFILE	fast_local or cloud

ds = DataStore(
    "s3://bucket/data.zarr",
    storage_options={...},
    zarrProfile="cloud",
    mem_budget="8G",
    working_copies=8,
    nthreads=8,
)

What changes for new v3 writes

Layer	Behaviour
Counts	Sharded Zarr v3; geometry from memory budget
countsT	Feature-major companion (nFeatures, nCells)
Normed matrices	Full-width row chunks, float32, no shards
Local compressors	Blosc / LZ4 + bitshuffle
Cloud compressors	Zstd
ANN index	Stored in Zarr as ann_idx_bytes
Sparse ingest	Shard-aligned flushes

countsT

Store	Gets countsT?	Effect
Existing Zarr v2	No	Uses counts
Existing Zarr v3 without countsT	No (unless you add it)	Silent fallback to counts
New Zarr v3 ingest	Yes, automatic	Faster HVG + markers; more write time/disk

Used by:

Workflow	Benefit
HVG summary stats	Feature-column reads
Marker search	Feature-column reads
Auto gene_batch_size	Uses countsT feature chunks when present

Trade-off: roughly one extra compressed copy of counts at ingest. No opt-out on new v3 writes.

from scarf.storage.zarr_store import write_counts_t
write_counts_t(assay_group["counts"], assay_group)

Cloud support

Capability	Detail
URIs	s3://, gs://, other obstore-backed URLs
Auth / options	storage_options=...
Graph build	local_cache="auto" stages normed data locally
Markers / feature reads	Remote-aware prefetch
Merge / writers	Accept storage_options

---

New features

Plotting (scarf.plotting)

Entry point	Purpose
embedding	Main UMAP/tSNE-style plots
embedding_raster	Large-cell rasterized embeddings
unified_embedding	Reference + query layouts
dotplot / matrixplot	Marker expression panels
composition / distribution	Composition and QC-style distributions
Facades	qc, elbow, graph_qc, highly_variable_features, marker_heatmap, cluster_tree, pseudotime_heatmap

import scarf.plotting as splt

splt.embedding(
    ds,
    layout_key="RNA_UMAP",
    color_by="RNA_leiden_cluster",
    theme="notebook",      # paper | minimal | dark
    legend_loc="auto",
    frame="minimal",
)

Setting	New API default	Legacy ds.plot_layout
Continuous cmap	viridis	Unchanged
Missing color	#bdbdbd	Often k
Point size	Auto from n cells	Fixed default
Frame	minimal	Unchanged
Rasterize	Above 50k cells	Shading path separate
Renderer	Opt-in	Default remains legacy

ds.plot_layout(..., use_plotting=True)  # opt-in bridge; may return PlotResult

Mapping reference + Symphony

API	Role
build_mapping_reference	Persist batch-aware reference artifacts
get_mapping_reference	Reload saved reference
MappingReference.map_query	Query projection (+ optional Symphony correction)
run_mapping(..., query_batches=...)	DataStore entry point
project_mapping_layout	Place query on fixed reference layout
get_target_label_evidence	Votes / entropy / conformal sets
calibrate_label_transfer_threshold	Threshold calibration

ref = ds.build_mapping_reference(
    from_assay="RNA",
    feat_key="hvgs",
    batch_columns=["batch", "donor"],
)
result = ref.map_query(
    query_assay, "query", "hvgs",
    query_batches=query_meta[["batch"]],
)

Doublet detection

ds.run_doublet_detection(
    cluster_key="RNA_leiden_cluster",
    label="doublet_score",
)

Param	Default	Notes
cluster_key	required	Simulation pools
simulation_ratio	1.0	Simulated vs observed scale
heterotypic_fraction	0.8	Cross-cluster doublets
label	doublet_score	Written to cell metadata

Integration metrics

Method	Measures
metric_label_concordance	ARI/NMI between two label columns
metric_batch_mixing	Neighborhood batch mixing (0–1)
metric_lisi	LISI (+ perplexity=30)
metric_silhouette	Cluster separation (+ seed/sample size)

Harmony

API	Notes
run_harmony / fit_harmony	Documented public API
HarmonyResult	Structured result object
make_graph(..., harmony_params=...)	Passed into graph build / cache identity

---

Other notable API changes

Area	Detail
Markers	New writes use compact compact_v2; old marker trees still load
Pseudotime	component_policy; stricter source/sink / regressor checks
Meld assay	Sparse CSC mapping; optional peaks_coords
Merge	Shared row plan; pandas feature order; remote storage_options
ANN	In-Zarr ann_idx_bytes; legacy filesystem index still readable
H5ad	Explicit sparse encoding validation

---

Bug fixes

Area	Problem	Impact	Fix
WNN	Fragile affinities / unequal k	Wrong multimodal weights, NaNs	Rewrite + validation
Pseudotime	Disconnected graphs	Crash / bad scores	component_policy
Pseudotime markers	Weak regressor checks	Misleading markers	Finite / uniqueness checks
Meld assay	Chromosome / dense map bugs	Wrong peak–gene aggregation	Sparse CSC mapping
Mapping projections	Incomplete writes readable	Corrupt layouts	Schema + complete flag
Harmonized mapping	Missing portable artifacts	Non-reproducible maps	MappingReference persistence
Merge	Divergent row/feature plans	Misaligned merges	Shared row plan
Wide matrices	Blosc / memory limits	Failures on many features	Budget + codec caps
HNSW / Py3.12+	Build/runtime issues	ANN failures	CI/build pins
Metrics naming	“integration” sounded like batch mixing	Misleading QC	Split APIs
H5ad sparse	Bad encodings	Silent bad reads	Hard validation errors

---

Tests and CI

Item	Before	After
Test modules	~9 under scarf/tests	40+ under tests/
Fixtures	Large binaries in-repo	Downloaded in CI
Python matrix	Mostly 3.11-era	3.12 / 3.13 / 3.14 + lowest-direct
Runner	Serial-ish	Parallel pytest-xdist
Quality gates	Limited	Ruff + mypy + coverage on scarf
Integration tests	Mixed in	Marked, skipped by default

Major new coverage areas: Zarr/countsT/budget/chunked/parallel, mapping + Symphony, WNN, markers, doublets, pseudotime, merge, plotting, readers/writers, metrics, Harmony/UMAP.

---

Docs and packaging

Item	Change
Version	0.32.3 → 1.0.0
README	Markdown root README
Install docs	Python 3.12+, uv, Zarr compatibility, budgets
API docs	MappingReference, Harmony, metrics, merge, plotting; Nabo removed
Vignettes	Reference mapping, graph imputation, plotting showcase
Packaging	pyproject.toml, dynamic VERSION, uv lockfile

---

Import cheat sheet

from scarf import DataStore, CrH5Reader, CrToZarr, AssayMerge, DatasetMerge
from scarf.mapping_reference import MappingReference, MappingResult
from scarf.harmony import run_harmony, fit_harmony, HarmonyResult
from scarf.metrics import (
    compute_lisi,
    label_concordance_score,
    lisi_batch_mixing_score,
)
import scarf.plotting as splt

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.