Feature Request: Support a Kubernetes Deployment for TCP-only Broker API

[matheuscscp]

PRs: #7122, #7123, #7141, #7142

Unblocks: spiffe/helm-charts-hardened#868

My plan for Deployment agents in terms of registration entries is as follows. It supports multiple Deployment agents in the Kubernetes cluster, in any namespaces. (The below is fully written by hand, no AI, please read everything!)

k8s_psat node attestor cluster entries

Cluster entries in the k8s_psat node attestor configuration are just named rules the agents can choose among for defining the format of the SPIFFE ID they will get, they do not define selectors or entry authorization (see next sections). The selectors emitted for agents attesting by this method are fixed.

In a Kubernetes cluster with both DaemonSet agents and Deployment agents, two entries are needed in the server-side config of k8s_psat:

• The usual one, defaulting to node-UID SPIFFE IDs for agents
• An additional entry using use_pod_uid_for_agent_id

If the cluster has multiple Deployments acting as SPIRE agents, all of them can reference the second (pod-UID-based) cluster entry without conflicts.

The DaemonSet agent uses the first cluster entry (node-UID-based). If there's only one DaemonSet agent in the cluster (e.g. the one deployed by the SPIRE Helm chart), this is enough. If there are multiple DaemonSets, then only one can use the first node-UID-based cluster entry, all the others have to use the pod-UID-based entry or create a new node-UID-based cluster entry.

Alias entries

A single alias is needed for each SPIRE agent, regardless if Deployment or DaemonSet. Multiple aliases for the same group-of-pods SPIRE agent do not improve security, because at the end all of them have to select the same set of agent pods. In other words, from the perspective of alias entries, a group of pods representing the same logical SPIRE agent is the atomic unit. In Kubernetes, that is represented by the ServiceAccount being shared by all the pods in that group.

So for each Deployment or DaemonSet that is a SPIRE agent in the cluster, I'd recommend a single alias entry:

• Parent ID: SPIRE server ID
• SPIFFE ID: the alias ID for a specific group of pods (Deployment or DaemonSet)
• Selectors:
  • k8s_psat:cluster:<cluster> (The server-side cluster entry the agent has chosen to join, see previous section)
  • k8s_psat:agent_ns:<namespace>
  • k8s_psat:agent_sa:<service-account>

In the Helm chart for deploying a Deployment-based Broker API agent, a ClusterStaticEntry can be used for the alias entry.

Broker entries

Usual Workload API business. Brokers get SVIDs just like any other workload, via Workload API. The existing governance applies. The Broker-over-TCP chart can facilitate this as well, by creating a ClusterStaticEntry for each broker ID with the parent set to the alias for DaemonSet agents.

Broker access to object entries

Opt into SubjectAccessReview (access_policy = "enforced"). In this case, the broker ID is used as a Kubernetes username for a SubjectAccessReview with verb impersonate-via-spire on the referenced object. The Broker-over-TCP chart can facilitate this as well, by creating RBAC resources to satisfy the required access relationships.

Object entries

Can be initially created via ClusterStaticEntry, and later via #7075. They will usually point to a single parent alias, but there can be multiple entries for the same object pointing to multiple parent aliases. (This should be a requirement for #7075.)

[kfox1111]

psat isnt the right way to solve this imo.

I've had the same class of problem with the spire-identity-exchange providing its own agent. see this pr: spiffe/helm-charts-hardened#860

I had to work around it by making a credential composer:
https://github.com/spiffe/spire-identity-exchange/blob/main/cmd/spire-credentialcomposer-identity-exchange/main.go

I'd rather fix it like:
#7002

[matheuscscp]

@kfox1111 See what you think of #7118, which solves a different problem but still related to deploying a TCP-only Broker API-only SPIRE agent 🙏

[matheuscscp]

After deepening my understanding of the problem, I concluded that the solution proposed here is in fact the best alternative, as explained here: #7001 (comment)

@amartinezfayo Your suggestion was a more flexible alternative of what I'm proposing here based on templates. I think we really don't need templates, just a pod-UID-based alternative is enough. The goal of a node attestor is to bootstrap a workload in the target platform as a SPIRE agent, and workloads in Kubernetes are Pods. To satisfy SPIRE's uniqueness requirement for SPIFFE IDs assigned to SPIRE agents, this solution is effectively (necessary and) sufficient. Please take a look at the PR I raised for implementing this approach: #7118

[kfox1111]

Strongly disagree. I dont think you went deep enough in your analysis. See #7001 (comment). tl;dr: you must explain how you will modify spire-controller-manager to hook this up so that svids are deployed only to the correct agents. the tcp brokers must not have access to the entire k8s nodes svids. How do you do this with the proposed solution?

[kfox1111]

I've been thinking about what this would take to implement end to end securely...

It would require an agent id template like:

spiffe://<trust-domain>/spire/agent/k8s_psat/<cluster>/<serviceaccount ns>/<service account name>/<pod UID>

Then, it would require https://github.com/spiffe/spire-controller-manager-dynamic-registration configured to allow only the prefix:

spiffe://<trust-domain>/spire/agent/k8s_psat/<cluster>/<service account ns>/<service account name>

along with the service account <service account ns>:<service account name>

This would allow the daemonset agents to register their names and configure the spire-controller-manager parentIDTemplate to be

"spiffe://{{ .TrustDomain }}/k8s_psat/{{ .ClusterName }}/{{ .NodeMeta.UID }}"

This is more complicated then just using the x509pop attestor for the spire-identity-exchange/flux use case, but can be made to work.

I still will be using the x509pop route though due to previously discussed issues.

---

So at minimum, the proposal of just appending pod uid is insufficent to safely implement this feature. It needs full template support with at least cluster name, service account ns, service account name, and pod uid.

[matheuscscp]

Strongly disagree. I dont think you went deep enough in your analysis. See #7001 (comment). tl;dr: you must explain how you will modify spire-controller-manager to hook this up so that svids are deployed only to the correct agents. the tcp brokers must not have access to the entire k8s nodes svids. How do you do this with the proposed solution?

Simple: With an alias for the Broker-only agents. Just like the DaemonSet agents have their own alias.

[kfox1111]

your focused too closely on the wrong side. thats the easy part. its the daemonset that has the problem.

[matheuscscp]

its the daemonset that has the problem.

Which problem? Pod-UID-based SPIFFE IDs for DaemonSet pods also makes them unique, no?

[kfox1111]

They are unique too. That is not the problem. If all you do is change the template that the k8s_psat uses for the agent spiffe ids, the controller rmanager's ClusterSPIFFEID's will break. And then the ability for flux to get a spiffeid to give to the broker will also be broken. The controller manager has to be configured to point at a reliable identity that maps to every k8s node agent only using a parent spiffe id. see https://github.com/spiffe/helm-charts-hardened/blob/main/charts/spire/charts/spire-server/values.yaml#L643

Its not the unique name for an agent that is the only concern. Its also how does the controller manager create the right registration entries for the new name.

Give it a try if you don't believe me. Build your code to a container, override the image in the chart and stand it up with your config. I know its a problem because I ran into this same issue a while ago. I'm trying to help save you a bunch of time going down a path I've already suffered through. Its not me being stubborn trying to block you from getting to a solution.

[matheuscscp]

The strategy I used in #7118 and in the chart PR is to split by the cluster entry in the server-side k8s_psat config. The spire-broker-over-tcp chart uses a dedicated value for cluster. This way DaemonSet agents keep using the node-UID SPIFFE ID format, while the Deployment agents will use pod-UID.

[kfox1111]

Ah, interesting. I did not think the same cluster could be specified twice in a k8s_psat config. It looks like you made the template different per cluster entry, and then use the service account to restrict access....

If you can do that, that does seem like a viable strategy.