pvc-autoresizer with GitOps workflows

[kaitimmer]

I'm currently testing around with pvc-autoresizer. So far, I like it.

Just one question that comes to mind. Since we are using Flux, we do have the issue that whenever pvc-autoresize adds storage to the disk, every subsequent Flux (or all other GitOps tools) error is with:

spec.resources.requests.storage: Forbidden: field can not be less than previous value

Currently, this would mean that someone needs to manually fix this after the pvc-autoresizer did something (which is still a lot better than before, where the disk would just be full :) ), but I'm thinking:

Wouldn't it be possible to use the MutatingWebhook with the group resize feature on update and patch operations? I'm unsure how Kubernetes behaves, though. Is the size test done before or after the Mutating Webhook is called?

I would like your thoughts or alternative solutions to this.

[daichimukai]

Hi @kaitimmer, thank you for trying out the pvc-autoresizer!

Does this issue occur only when the PVCs are resized using the initial resize feature?

Wouldn't it be possible to use the MutatingWebhook with the group resize feature on update and patch operations? I'm unsure how Kubernetes behaves, though. Is the size test done before or after the Mutating Webhook is called?

The initial group resize feature is implemented in the mutating webhook. The size test occurs during the webhook execution. Since this webhook operates before the resource is created, I believe we are not able to use update or patch operations.

I'm not familiar with Flux, so the following might not be very useful, but I hope it provides some hints.

We're also using a GitOps tool, Argo CD, and we are not experiencing the same issue. The main reason is that we maintain almost no PVCs in the Git repository. Instead, we define StatefulSets or similar resources, and the controllers in the Kubernetes clusters generate PVCs. In this case, the PVCs are not directly controlled by the GitOps tool, so no such issue occurs.

On the other hand, when we want to ignore specific fields in the resource definitions, we use the ignoreDifferences setting, which forces Argo CD to ignore the differences between the actual field value and its definition in the Git repository. I'm not sure if there's an equivalent feature in Flux, though.

[kaitimmer]

You are right. I was testing directly with a PVC, but that's not what happens in a typical scenario. Usually, there is a controller in between, like StatefulSet, so this is not a big problem.

Thanks for pointing it out.