Configurable way to allow entry/federation-relationship creation when a federated bundle is missing (revisiting #1256 / #4445)

[SpectralHiss]

Summary

When a registration entry's federatesWith references a trust domain with no corresponding bundles row, SPIRE Server rejects the entry create/update entirely:

unable to find federated bundle "spiffe://some-trust-domain"

This is enforced in makeFederatesWith
(pkg/server/datastore/sqlstore/sqlstore.go), which requires every trust domain listed in federatesWith to resolve to an existing bundles row before the entry write succeeds.

Real-world impact

We run federation from several per-cluster trust domains into a central cluster, managed by an operator reconciling ClusterFederatedTrustDomainresources.
During an incident, one federated cluster's bundle failed to publish to its storage backend (S3 404 / NoSuchKey).
SPIRE Server had a FederatedTrustDomain relationship for that trust domain registered, but no bundles row.
Our default ClusterSPIFFEID's federatesWith included that trust domain among several others, so every entry generated from that CSID failed to create — surfacing to workloads as PermissionDenied: no identity issued even though only one of several federated trust domains was actually unhealthy.

History

This is actually a known issue initially raised in #1256 but due to small complications in an initial fix PR was later abandoned due to lack of perceived demand for this edge case.

• Creating entries with a FederatesWith for an unknown trust domain should succeed #1256 (2019): proposed a fix. Discussion converged on a "sentinel bundle" design (@azdagron's comment): auto-create an empty bundle row (trust domain set, data column NULL) as a placeholder so the federated_registration_entries many-to-many join always resolves, with fetch/list operations filtering out NULL-data rows so placeholders are never mistaken for real bundles.
• This was chosen specifically to avoid the schema/data migration a "proper" fix would otherwise need (a decoupled table mapping entry ID -> trust domain strings, discussed here).
• @dfeldman implemented an auto-create-on-missing-bundle approach in If a bundle doesn't exist yet when creating a federation reg. entry, create it #1448
  (dfeldman:empty-bundle-federation).
  Note that this always created a real bundles row (never left a literal dangling foreign key — every entry's federatesWith still pointed at something that existed), but left two things unresolved: placeholder bundles showed up in bundle list operations indistinguishable from real ones, and nothing cleaned them up when the owning relationship was removed.

The issue and PR were ultimately closed won't-fix @evan2645's closing rationale points back to the
original 2019 intent behind the strict check: "we did this consciously with the intention of signaling to the registrant that something is up."

Auto-creating a bundle on any federatesWith value removes that typo/misconfiguration signal, so a mistyped or stale trust domain string would succeed silently and leave a permanent, unintentional bundle row behind, with nothing to prompt an operator to notice.

• spire-oidc does not start if federatesWith contains a down federated endpoint  #4445 (2023): same failure recurred via spire-controller-manager /
  ClusterSPIFFEID. @evan2645 called it "pretty tough to fix" and asked for
  a volunteer; none appeared, closed without a fix.

Proposal

Make this opt-in and server-wide (no per-relationship granularity, no CRD
or spire-api-sdk changes, no schema migration), addressing the concerns
that sank #1256/#1448:

1. Revive @dfeldman 's original approach of setting empty bundles as this is valid today
2. Filter placeholders from FetchBundle/ListBundles, per @azdagron's original proposal, so they don't show up in spire-server bundle list/bundle show (or via GetFederatedBundle/ListFederatedBundles) as if they were real (if empty) published
   bundles.
3. No new cleanup mechanism on relationship deletion. Placeholder bundles are ordinary bundles rows. Deleting a FederationRelationship doesn't touch bundles today either, and shouldn't start now —spire-server bundle delete already defaults to RESTRICT, so any real entry dependency blocks accidental removal.
   Building bespoke cleanup here would risk recreating the exact dangling-reference bug this proposal avoids.
4. Operator UX parity. Losing the hard failure removes the original typo/misconfiguration signal (see "History"). Getting equivalent visibility back isn't a single change: today the error reaches CLI/API callers as an immediate, hard failure, and reaches spire-controller-manager as both a specific log line and a bump to the existing ClusterSPIFFEID status stats.
   Matching that means work in a few places: SPIRE Server surfacing a non-error warning on the affected create/update result, the spire-server CLI actually displaying it (today it only reads the result message on failure, not success), and spire-controller-manager surfacing the same signal somewhere operators already look (e.g. ClusterSPIFFEID/ClusterFederatedTrustDomain status). None of this comes for free just by adding the server-side flag and empty sentinel bundle.

Happy to put together a WIP PR if this direction seems reasonable to
maintainers.

[SpectralHiss]

In our case there is a workaround of introducing health checks in the operator generating ClusterFederatedTrustdomains, this is probably the quickest win for us at this point, waiting to see what the interest of community in the generic fix proposed to this issue

[sorindumitru]

@SpectralHiss The problem identified here only happens when adding a new federation relationship, right? Once that refreshes once successfully you will have a bundle in the database that remains available even if the endpoint is unavailable.

One possible workaround for this could be implementing the federation groups that we have been discussing in #2272. The entry would point to a group and it wouldn't depend that much on bundles being available (somewhat dependent on the implementation). It's not a full solution to the problem, though.