Change the ytt fields to accept arrays as well as single entries, such that you can append an overlay or other modifications to the ytt steps without modifying the previous ones.

[voor]

Description of problem

Currently whenever you see a ytt field in a spec if you want to modify it you're stuck doing ytt overlays to embedded ytt, which is very difficult in Carvel.

Here are examples of all 4 in the same file even, each more gruesome and painful as we go on!

1. If you are fortunate enough that you can just stick what you need at the end of the spec (because yaml!) then just go ahead and slap it at the end:
   Just adding to the end
2. If you need to change something in the middle, you can do a find and replace:
   Surgical find/replace
3. If you are making massive changes just copy and paste the whole thing, making the changes you need and then rigorously and continuously validating it every time there's a change:
   Just replace the whole thing manually
   (this is probably the worst one)
4. I want this to be my favorite, and I feel like it's the most inventive, but unfortunately this one is just as hard to maintain as the others, but append an overlay that modifies the other ytt preceding it:
   Append an overlay of ytt to the end

Proposed solution

As mentioned in approach 4, if ytt instead took an array (or map?) of things instead of a single entry then case 4 could be approached by appending the overlay to the ytt entries instead of sticking it to the end of the string. Or instead of modifying the ytt field offer another way to easily add overlays -- another field, or similar.

Given the ytt field accepts an array (or there is a new overlays field or additional_ytt or similar)
When I need to add an overlay to an existing entry
Then I can just add that ytt as another entry to the fields

Example

Basically take the example listed under 4 and change it to be an append to an array instead of appending to a string.

#@ def source_testingpipeline():
kind: ClusterSourceTemplate
metadata:
  name: testing-pipeline
#@ end

#@overlay/match by=overlay.subset(source_testingpipeline())
---
spec:
  overlays:
  - |-
    #@ def source_pipelinerun():
    kind: Runnable
    #@ end

    #@ def languagelabel():
      #@ if hasattr(data.values.workload.metadata.labels, "apps.tanzu.vmware.com/language"):
        #@ return data.values.workload.metadata.labels["apps.tanzu.vmware.com/language"]
      #@ else:
        #@ return "java"
      #@ end
    #@ end

    #@ load("@ytt:overlay", "overlay")

    #@overlay/match by=overlay.subset(source_pipelinerun())
    ---
    spec:
      selector:
        matchingLabels:
          #@overlay/match missing_ok=True
          apps.tanzu.vmware.com/language: #@ languagelabel()

Additional Context

[waciumawanjohi]

Adding context from some conversations today. Considering this approach:

Cartographer templates gain a new field named along the lines of .spec.additional_ytt_files. This field contains a list of selectors and namespaces of configMaps. Any such configMap found would be expected to contain an additional file to pass to ytt (e.g. ytt -f some-file.yaml -f some-additional-file.yaml)

This is a rough sketch, syntax can change:

---
kind: ClusterSourceTemplate
metadata:
  name: testing-pipeline
spec:
  ytt:
    some_template_def
  additional_ytt_files:
    - namespace: tap-install
      selector:
        match_labels:
        - tap
        - testing-pipeline

---
kind: ConfigMap
metadata:
  name: some-new-overlay
  namespace: tap-install
  labels:
  - tap
  - testing-pipeline
data:
  Some ytt