feat!: absence propagates and drops the row, on both lanes

Adopts linopy's v1 reading of absence as the language's own, and makes v1 the
oracle rather than a mode we happen to survive.

**What changes.** A term whose variable is masked out no longer contributes
zero — it makes the row absent, so `x + y >= 10` is *no constraint* where `y`
is masked rather than `x >= 10`. The old reading is how

    x - rel_max * size <= 0

silently became `x <= 0` on an unsized component: feasible model, plausible
answer, no error. That is goal 1 of the v1 convention ("no silent wrong
answers") and the whole of PyPSA/linopy#712, and it was reachable here.

**Two things deliberately do not propagate.** A *reduction* skips absent slots
(§13), so `sum(x, over=d)` stays defined when only some of `d` exists — without
that, one masked component would delete a system-wide accounting row. A
*parameter* covering only some coordinates is sparse encoding, not absence: its
missing rows mean a zero coefficient (SPEC §8), which is what lets a coefficient
table hold live entries only. Absence is a property of variables.

**How the engine tells them apart.** `TermFragment.presence` carries the
variable's own coordinates beside the term stream, because once `coeff x var`
are multiplied the frame cannot say which side removed a row. It is set only
for a variable whose declaration has a `where` — decided off the plan, before
data — so an unmasked variable never imposes the cost, and `_label_frame` keeps
both of its arithmetic paths (#152, #178) for every equation that does not turn
on the difference.

**The oracle is v1, and it raises rather than skips.** A skip would be the worst
outcome available: the suite would go green having stopped comparing the lanes
on exactly the cases the convention changed. No release carries the option yet,
so `[tool.uv.sources]` pins PyPSA/linopy#717 by branch — by branch and not by
rev on purpose, since a stale rev would measure us against a spec that has moved.

Not included, and it is the follow-up this needs: `defined(v)` (#219). Dropping
the row is now the only reading available, and the way to ask for the other one
is complementary `where` clauses over a variable's existence — which is not yet
sayable. Until it lands, a model wanting "keep the row, treat the term as zero"
has to carry a parameter mirroring the variable's mask.

`test_a_constraint_row_left_with_no_variables` stays xfailed and is *not* this:
raw linopy builds a term-less row under both conventions (`labels=[0,1]`,
`vars=[-1]`), so that divergence lives in our own eager lane and wants its own
diagnosis.

Refs #8, #219

Author: FBumann

SPEC.md

@@ -247,6 +247,25 @@ A boolean mask; true means "this coordinate exists". Semantics are **row
 absence**, not zero-fill: a masked-out variable is not created, a masked-out
 constraint row is not built.
 
+**Absence spreads.** A term whose variable does not exist at a coordinate does
+not contribute zero there — it makes the whole row absent, so `x + y >= 10` is
+*no constraint* where `y` is masked, not `x >= 10`. Zero-filling instead is how
+`x - rel_max * size <= 0` silently becomes `x <= 0` on an unsized component: a
+feasible model, a plausible answer, no error. To keep the row and treat the
+missing term as zero, say so — write the two cases as separate equations with
+complementary `where` clauses.
+
+Two things deliberately do **not** spread. A **reduction** skips what is absent
+rather than propagating it, so `sum(x, over=d)` is defined when only some of `d`
+exists and the sum of nothing is zero — without that, one masked component would
+delete a system-wide accounting row. And a **parameter** covering only some
+coordinates is sparse *encoding*, not absence: its missing rows mean a zero
+coefficient (§8), which is why a coefficient table may hold live entries only.
+Absence is a property of variables.
+
+This matches linopy's v1 arithmetic convention, which both lanes are built
+against; `farkas.linopy.semantics` is where the eager lane answers it.
+
 ```text
 where_expr ::= atom | "NOT" where_expr | where_expr ("AND"|"OR") where_expr
             |  "(" where_expr ")"

pyproject.toml

@@ -50,8 +50,14 @@ dependencies = [
 # The relational engine never imports them, and `dev` installs this extra by
 # name rather than repeating it — one place for the pins, so what CI tests the
 # lane against is what a user of the extra gets.
+#
+# The lower bound is 0.8, not 0.9, because the v1 arithmetic convention this
+# lane requires is not in a release yet: PyPSA/linopy#717 branches from 0.8.x
+# and reports a 0.8 dev version. `[tool.uv.sources]` below is what actually
+# resolves it for development and CI. When v1 ships, this becomes an ordinary
+# floor on the release that carries it and the source pin goes away.
 linopy = [
-    "linopy>=0.9.0,<0.10",
+    "linopy>=0.8.0.post1.dev0",
     "xarray>=2024.2.0",
     # also what `Result.to_pandas` / `to_dataarray` need. Declared here rather
     # than as a runtime dependency: pandas is a bridge out of the engine, not
@@ -191,3 +197,12 @@ unresolvable-dunder-all = true
 untyped-import = true
 unused-ignore = true
 variance-mismatch = true
+
+# The v1 arithmetic convention is not released. farkas's eager lane is written
+# against it — absence propagates and drops a constraint row rather than being
+# filled with zero — so the oracle is only an oracle on a linopy that speaks it.
+# Pinned by branch rather than by rev deliberately: this is tracking active
+# upstream work, and a stale rev would quietly measure us against a spec that
+# has since moved. Removed once v1 ships.
+[tool.uv.sources]
+linopy = { git = "https://github.com/PyPSA/linopy", branch = "feat/arithmetic-convention" }

src/farkas/linopy/builder.py

@@ -305,7 +305,7 @@ def _eval_ast(
         return node.value
 
     if isinstance(node, VariableNode):
-        return semantics.present(ctx.model.variables[node.name])
+        return ctx.model.variables[node.name]
 
     if isinstance(node, ParameterNode):
         return semantics.coefficient(ctx.dataset[node.name])

src/farkas/linopy/semantics.py

@@ -38,7 +38,7 @@
 
 from typing import Any
 
-__all__ = ['coefficient', 'present', 'vacated']
+__all__ = ['coefficient', 'vacated']
 
 
 def coefficient(parameter: Any) -> Any:
@@ -61,34 +61,6 @@ def coefficient(parameter: Any) -> Any:
     return parameter.fillna(0.0)
 
 
-def present(variable: Any) -> Any:
-    """A masked variable as a term contributing **zero** where it is absent.
-
-    Today's language rule: a term carrying a masked-out variable drops out while
-    the row it sits in survives, so ``x + y >= 10`` is ``x >= 10`` wherever ``y``
-    is masked. The relational lane gets that from row absence — the join finds no
-    row, the term is not emitted, the constraint row still is.
-
-    Filling at the **leaf** is what preserves it. Fill any later and §2 has
-    already absorbed the live terms sharing the slot (``(x + y).fillna(0)``
-    yields a bare ``0 >= 10`` with ``x``'s coefficient gone); fill here and ``x``
-    keeps it.
-
-    ``to_linexpr()`` first, because ``Variable.fillna`` means two different
-    things across the versions we support — a label fill routed through
-    ``.where()`` on the released line, an expression fill on the v1 branch. The
-    expression method is the stable one, and the one we want.
-
-    .. note::
-       This is the interim answer, not the settled one. v1 §6/§12 would drop the
-       row instead, and its goal 1 ("no silent wrong answers") indicts the fill:
-       ``x - relmax * size <= 0`` with ``size`` masked silently becomes
-       ``x <= 0``. Adopting that reading is a change to *both* lanes and to
-       SPEC §6, so it is its own commit — and it deletes this function.
-    """
-    return variable.to_linexpr().fillna(0)
-
-
 def vacated(expression: Any) -> Any:
     """A shifted expression with its vacated edge positions at **zero**.
 
@@ -108,5 +80,12 @@ def vacated(expression: Any) -> Any:
     ``x <= shift(dt, t=1)`` forces ``x <= 0`` at the first position unless it is
     masked. Until that is decided, this keeps the documented rule true by
     construction rather than by the legacy convention's accident.
+
+    ``to_linexpr()`` first when the operand is still a bare ``Variable``:
+    ``Variable.fillna`` means two different things across the versions we
+    support — a label fill routed through ``.where()`` on the released line, an
+    expression fill on the v1 branch — and only the expression method is stable.
     """
+    if hasattr(expression, 'to_linexpr'):
+        expression = expression.to_linexpr()
     return expression.fillna(0)

src/farkas/relational/compiler.py

@@ -313,7 +313,12 @@ def _variable_fragment(self, name: str) -> TermFragment:
 
         dims = self.program.variable(name).dims
         frame = self.variables[name].select(*dims, 'var_label', pl.lit(1.0, dtype=pl.Float64).alias('coeff'))
-        presence = self.variables[name].select(*dims)
+        # An *unmasked* variable exists at every coordinate of its foreach, so
+        # its presence could only ever restrict nothing. Leaving it None is not
+        # an optimisation detail: a presence frame is data, and carrying one
+        # costs `_label_frame` both of its arithmetic paths. Whether it is
+        # needed is decided here, off the declaration, before any data is read.
+        presence = self.variables[name].select(*dims) if self.program.variable(name).where is not None else None
         return TermFragment(dims, frame, True, label_dims=frozenset(dims), presence=presence)
 
     def _product(self, a: CompiledExpression, b: CompiledExpression, context: str) -> CompiledExpression:
@@ -414,14 +419,46 @@ def _translate_fragment(self, p: TermFragment, s: plan.Translate, context: str)
         moved = pl.col(_ORD_IN) + s.by
         if s.wrap:
             moved = (moved % card + card) % card
-        frame = (
-            p.frame.join(incoming, on=s.dimension, how='inner')
-            .drop(s.dimension)
-            .with_columns(moved.alias(_ORD_OUT))
-            .join(outgoing, on=_ORD_OUT, how='inner')
-            .select(*others, s.dimension, *p.carried)
+
+        def remap(source: pl.LazyFrame, carried: list[str]) -> pl.LazyFrame:
+            return (
+                source.join(incoming, on=s.dimension, how='inner')
+                .drop(s.dimension)
+                .with_columns(moved.alias(_ORD_OUT))
+                .join(outgoing, on=_ORD_OUT, how='inner')
+                .select(*others, s.dimension, *carried)
+            )
+
+        frame = remap(p.frame, p.carried)
+        presence = None
+        if p.presence is not None:
+            # Presence is a coordinate set, so it travels through the same map.
+            presence = remap(p.presence, [])
+            if not s.wrap:
+                presence = pl.concat([presence, self._vacated(p, s, card, others)], how='vertical_relaxed').unique()
+        return TermFragment(p.dims, frame, p.is_term, p.keyed, p.label_dims, presence)
+
+    def _vacated(self, p: TermFragment, s: plan.Translate, card: int, others: list[str]) -> pl.LazyFrame:
+        """The edge positions ``shift`` leaves with nothing to move in.
+
+        They are *not* absent. SPEC §7 fixes what they contribute — "vacated
+        positions contribute **zero**" — which is a declared rule of the
+        language, so they stay present and the row survives. That is the same
+        answer the eager lane gives them (``semantics.vacated``), and it is why
+        the two lanes agree about an acyclic recurrence's first step.
+
+        Only the ``shift`` edge qualifies. A coordinate the variable's own mask
+        removed is genuinely absent, and remapping already dropped it above.
+        """
+        table = self.dimensions[s.dimension]
+        edge = table.filter(((pl.col('ord') - s.by) < 0) | ((pl.col('ord') - s.by) >= card)).select(
+            pl.col('val').alias(s.dimension)
         )
-        return TermFragment(p.dims, frame, p.is_term, p.keyed, p.label_dims)
+        if not others:
+            return edge
+        # One vacated row per other-dim combination the variable actually has:
+        # a coordinate it never covers gains nothing from an edge it never sees.
+        return p.presence.select(*others).unique().join(edge, how='cross') if p.presence is not None else edge
 
     # ------------------------------------------------------------------
     # assembly helpers used by the executor

src/farkas/relational/executor.py

@@ -497,6 +497,7 @@ def _label_frame(
         where: plan.Predicate | None,
         label: str,
         start: int,
+        restrictions: Sequence[tuple[tuple[str, ...], pl.LazyFrame]] = (),
     ) -> tuple[pl.DataFrame, int]:
         """The masked coord product of *dims* with a dense *label* from *start*.
 
@@ -514,19 +515,30 @@ def _label_frame(
         Both return ``(dims…, label)`` in that column order. A mask that
         removes nothing has to be indistinguishable from no mask, down to the
         schema.
+
+        *restrictions* are variable-presence frames a constraint row must be
+        contained in: absence propagates into a comparison and drops the row
+        (v1 ``convention.rst`` §6, §12). They are semi-joins, so they can only
+        remove rows — but which rows is not known until data is read, and that
+        is what costs the two fast paths, so the caller passes them only when a
+        variable in the equation is actually masked (:func:`_absence_restrictions`).
         """
-        if where is None:
-            frame = self._q.frame(dims, None)
-            rows = math.prod(self._dim_card[d] for d in dims)
-            return self._positional(frame, dims, label, start), start + rows
+        if not restrictions:
+            if where is None:
+                frame = self._q.frame(dims, None)
+                rows = math.prod(self._dim_card[d] for d in dims)
+                return self._positional(frame, dims, label, start), start + rows
+
+            free = _free_prefix(dims, _predicate_dims(where, self._param_dims()))
+            if free:
+                return self._factored(dims, free, where, label, start)
 
-        free = _free_prefix(dims, _predicate_dims(where, self._param_dims()))
-        if free:
-            return self._factored(dims, free, where, label, start)
+        restricted = self._q.frame(dims, where)
+        for on, presence in restrictions:
+            restricted = restricted.join(presence.unique(), on=list(on), how='semi')
 
         materialised = (
-            self._q.frame(dims, where)
-            .sort([_ordinal(d) for d in dims])
+            restricted.sort([_ordinal(d) for d in dims])
             .select(*dims)
             .with_row_index(label, offset=start)
             .select(*dims, pl.col(label).cast(pl.Int64))
@@ -674,7 +686,8 @@ def _build_constraint(self, c: plan.ConstraintDeclaration) -> tuple[pl.DataFrame
                     f'foreach {list(c.dims)} — missing a Sum/GroupSum?'
                 )
 
-        labelled, self._n_rows = self._label_frame(c.dims, c.where, 'row', self._n_rows)
+        restrictions = _absence_restrictions([p for p, _ in terms])
+        labelled, self._n_rows = self._label_frame(c.dims, c.where, 'row', self._n_rows, restrictions)
         frame = labelled.lazy()
         self._constraints[c.name] = frame  # kept for the dual read-back
 
@@ -1000,3 +1013,30 @@ def _stack(frames: list[pl.DataFrame], columns: tuple[str, ...]) -> pl.DataFrame
     if frames:
         return pl.concat(frames)
     return pl.DataFrame(schema={name: _DTYPES[name] for name in columns})
+
+
+def _absence_restrictions(terms: Sequence[TermFragment]) -> list[tuple[tuple[str, ...], pl.LazyFrame]]:
+    """The presence frames a constraint's rows have to be contained in.
+
+    Absence propagates into a comparison and drops the row there (v1
+    ``convention.rst`` §6 and §12): ``x + y >= 10`` is not ``x >= 10`` where
+    ``y`` is masked, it is no constraint at all. A term whose variable is absent
+    therefore restricts the row set rather than merely contributing nothing.
+
+    Only *variable* absence counts. A sparse parameter is a compressed dense
+    array whose missing rows mean a zero coefficient (SPEC §8), which is why the
+    fragment carries :attr:`~farkas.relational.compiler.TermFragment.presence`
+    separately from its frame, and why this reads that rather than the frame.
+
+    **A fragment with nothing to restrict is skipped**, and that is load-bearing
+    rather than tidy: a restriction is data — which rows survive is unknown until
+    the presence frames are read — so it costs ``_label_frame`` both of its
+    arithmetic paths. An unmasked variable's presence is its whole coordinate
+    product and would remove nothing, so it never gets to impose that cost.
+    """
+    out: list[tuple[tuple[str, ...], pl.LazyFrame]] = []
+    for p in terms:
+        if p.presence is None or not p.dims:
+            continue
+        out.append((p.dims, p.presence))
+    return out

tests/oracle.py

@@ -30,6 +30,26 @@
 xr = pytest.importorskip('xarray', reason=_REASON)
 pd = pytest.importorskip('pandas', reason=_REASON)
 
+# **The oracle is v1, and only v1.** A differential test is an oracle only if
+# the thing it compares against is the convention we implement. Legacy is the
+# one linopy is retiring: it fills every absent slot with 0, so it agrees with a
+# lane that keeps a constraint row whose variable is masked, and disagrees with
+# one that drops it. Measuring against legacy would pin this package to the
+# behaviour v1 classifies as a bug (PyPSA/linopy#712).
+#
+# So this raises rather than skipping. A skip here would be the worst outcome
+# available — the suite would go green having quietly stopped checking the lanes
+# against each other on precisely the cases the convention changed.
+if 'semantics' not in getattr(linopy.options, '_defaults', {}):
+    raise RuntimeError(
+        f'linopy {linopy.__version__} has no options["semantics"], so it cannot speak the v1 '
+        f'arithmetic convention this package is written against. The oracle would silently '
+        f'measure against the legacy convention instead. Install the pin in pyproject.toml '
+        f'([tool.uv.sources]: PyPSA/linopy@feat/arithmetic-convention) — `uv sync`.'
+    )
+
+linopy.options['semantics'] = 'v1'
+
 # Spelled out rather than aliased to something shorter: this module also
 # re-exports the *real* ``linopy`` above, so the shim needs a name that cannot
 # be confused with it. ``farkas_linopy`` names the module it actually is, which

tests/test_relational.py

@@ -707,3 +707,43 @@ def test_a_parameter_covering_a_subset_of_its_dims_means_zero_on_both_lanes():
         # t=0 carries `0 * x <= 0` — a row that exists and constrains nothing,
         # which is what a zero coefficient and a zero right-hand side mean.
         assert run.result.objective == pytest.approx(10.0 + 4.0 + 5.0, rel=RTOL)
+
+
+ABSENT_VARIABLE_MODEL = {
+    'dimensions': {'f': {'values': ['a', 'b']}},
+    'parameters': {'gate': {'dims': ['f'], 'dtype': 'bool'}, 'relmax': {'dims': ['f']}, 'cost': {'dims': ['f']}},
+    'variables': {
+        'x': {'foreach': ['f'], 'bounds': {'lower': 0, 'upper': 100}},
+        'size': {'foreach': ['f'], 'where': 'gate', 'bounds': {'lower': 0, 'upper': 50}},
+    },
+    'constraints': {'envelope': {'foreach': ['f'], 'equations': [{'expression': 'x - relmax * size <= 0'}]}},
+    'objectives': {'total': {'sense': 'maximize', 'equations': [{'expression': 'sum(x * cost, over=f)'}]}},
+}
+
+
+def test_a_term_whose_variable_is_absent_drops_the_row_on_both_lanes():
+    """Absence propagates into the comparison; it does not zero the term.
+
+    ``x - relmax * size <= 0`` where ``size`` is masked out used to build
+    ``x <= 0`` — a row that silently pinned the flow to zero. Plausible answer,
+    no error, which is goal 1 of linopy's v1 convention ("no silent wrong
+    answers") and the whole of PyPSA/linopy#712. Under §6 the slot is absent and
+    §12 drops the row instead, so ``x`` is left free at ``f=b`` and bounded only
+    by its own declaration.
+
+    The oracle is the point: the eager lane gets this from linopy's own v1
+    semantics, the relational lane from carrying variable presence apart from
+    the term stream. Two independent implementations, one answer.
+    """
+    data = {
+        'gate': pd.Series({'a': True}),
+        'relmax': pd.Series({'a': 0.5, 'b': 0.5}),
+        'cost': pd.Series({'a': 1.0, 'b': 1.0}),
+    }
+    with differential(ABSENT_VARIABLE_MODEL, data, lp=True) as run:
+        # one call, then zip: `primal` is a label join and does not promise row order,
+        # so reading it twice and pairing the columns can mismatch them.
+        solved = run.result.primal('x')
+        x = dict(zip(solved['f'], solved['value'], strict=True))
+        assert x['a'] == pytest.approx(25.0, rel=RTOL), 'sized: x <= 0.5 * size, size <= 50'
+        assert x['b'] == pytest.approx(100.0, rel=RTOL), 'unsized: the row is gone, so only the bound holds'