feat: allow wildcards as part of annotations and labels in allowlists by skoef · Pull Request #2873 · kubernetes/kube-state-metrics

skoef · 2026-02-12T16:22:19Z

What this PR does / why we need it:
In my clusters, I've add some annotations to nodes, pods and PVCs. I want to use these annotations to filter on in my alertmanager queries. For instance: I want to alert if a pod has high CPU usage, except when an annotation on that pod is set like monitoring.mycompany.org/ignore-cpu or whatever. I have a whole bunch of these annotations.

Currently, if I want to add these annotations to KSMs allowlist, I have to either use * for all annotations (I'd rather not do this) or mention all my annotations, for nodes=, pods=, and persistentvolumeclaims=. This could get rather annoying so I figured: why can't I use partial wildcards instead? Something like nodes=[monitoring.mycompany.org/*],pods=[monitoring.mycompany.org/*],persistentvolumeclaims=[monitoring.mycompany.org/*] or perhaps even *=[monitoring.mycompany.org/*].

With this MR this would be possible, to the limit of 1 * in every match. I would like to get some feedback on this idea! 🙏

How does this change affect the cardinality of KSM: does not change cardinality

k8s-ci-robot · 2026-02-12T16:22:30Z

Welcome @skoef!

It looks like this is your first PR to kubernetes/kube-state-metrics 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kube-state-metrics has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-02-12T16:22:30Z

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bhope · 2026-03-01T03:27:42Z

Thanks for the PR -- this solves a real issue with namespaced annotations. Left a few inline comments that need attention.

bhope · 2026-03-02T03:07:44Z

/assign @bhope

bhope

Thanks for addressing the feedback. Left one inline nit, otherwise it's in pretty good shape.

However, the docs CI is failing -- cli-arguments.md is auto-generated from the cli flag usage strings. The wildcard documentation needs to be added to the flag descriptions in the code and then regenerate by running make doccheck locally.

k8s-ci-robot · 2026-03-08T03:35:45Z

@bhope: changing LGTM is restricted to collaborators

Details

In response to this:

Thanks for addressing the feedback. Left one inline nit, otherwise it's in pretty good shape.
/lgtm
cc @mrueg @dgrisonnet for final review/approval.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bhope

Looks good to me.
We will wait for the CI to be green and one of the maintainers, if they have any comments.

bhope · 2026-03-17T06:58:25Z

/lgtm

k8s-ci-robot · 2026-03-17T06:58:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bhope, skoef
Once this PR has been reviewed and has the lgtm label, please assign catherinef-dev for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

skoef · 2026-03-25T16:35:18Z

@CatherineF-dev could you review this MR please?

Copilot

Pull request overview

Adds support for partial * wildcards in --metric-labels-allowlist / --metric-annotations-allowlist entries so users can allow groups of labels/annotations (e.g. monitoring.mycompany.org/*) without allowing everything.

Changes:

Allow wildcard patterns in allowlist entries and enforce a max of 1 * per label/annotation key.
Implement wildcard matching in label/annotation extraction via regex expansion.
Add unit tests and update CLI flag docs/help text.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/options/types_test.go	Adds allowlist parsing tests for partial wildcards and invalid multi-wildcard patterns.
pkg/options/types.go	Validates that each allowed label/annotation key contains at most one `*`.
pkg/options/options.go	Updates `--help` text to describe wildcard usage.
pkg/constant/utils.go	Introduces `MaxPartialWildcardsPerLabel` constant used by parsing/matching.
internal/store/utils.go	Matches allowlisted labels/annotations using wildcard-to-regex expansion.
internal/store/utils_test.go	Adds tests covering wildcard matching behavior and regex expansion.
docs/developer/cli-arguments.md	Updates documented CLI arguments to mention wildcard support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-21T10:50:56Z

+		// prepare regular expression based on potential wildcards
+		re, err := regexp.Compile(expandWildcard(l, constant.MaxPartialWildcardsPerLabel))
+		if err != nil {
+			continue
+		}


createPrometheusLabelKeysValues now compiles a regexp and scans allKubeData for every allowList entry, even when the allowList entry is an exact key (no '*'). This changes the complexity from O(len(allowList)) map lookups to O(len(allowList)len(allKubeData)) and adds per-object regexp compilation. Consider fast-pathing entries without '' using a direct map lookup, and only compiling/scanning for patterns that actually contain a wildcard (or precompiling patterns once when parsing options).

Since we use the label(-description) as the basis for the regex, we can't prepare the regex outside of this loop.

rexagod · 2026-04-21T11:14:34Z

+		if err != nil {
+			continue
+		}


We'd want to surface these to the user.

How do you suggest I'd implement that? The function createPrometheusLabelKeysValues currently doesn't return an error. I could go over the labels/annotations and throw this error in LabelsAllowList.Set instead during startup?

rexagod · 2026-04-21T11:28:54Z

Also PTAL at the Copilot reviews.

Partial labels and annotations are possible in --metric-annotations-allowlist and --metric-labels-allowlist with this change Signed-off-by: Reinier Schoof <reinier@skoef.nl>

k8s-ci-robot · 2026-04-21T12:06:55Z

New changes are detected. LGTM label has been removed.

skoef · 2026-05-07T07:36:11Z

@rexagod Hi, could you perhaps re-review my changes? Thanks!

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

 	for _, test := range tests {
 		lal := &LabelsAllowList{}
 		gotError := lal.Set(test.Value)
-		if gotError != nil && !test.err || !reflect.DeepEqual(*lal, test.Wanted) {
+		if gotError != nil && test.err == nil || !reflect.DeepEqual(*lal, test.Wanted) {
 			t.Errorf("Test error for Desc: %s\n Want: \n%+v\n Got: \n%#+v\n Got Error: %#v", test.Desc, test.Wanted, *lal, gotError)
 		}


+	o.cmd.Flags().Var(&o.AnnotationsAllowList, "metric-annotations-allowlist", "Comma-separated list of Kubernetes annotation keys that will be used in the resource' labels metric. By default the annotations metrics are not exposed. To include them, provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. A single '*' can be provided per resource instead to allow any annotations, but that has severe performance implications (Example: '=pods=[*]'). Additionally, an asterisk (*) can be provided for resources, which will resolve to all resources, i.e., assuming '--resources=deployments,pods', '=*=[*]' will resolve to '=deployments=[*],pods=[*]'. Wildcards can also be use to match multiple annotations, i.e., '=pods=[something.example.org/foo-*]' will resolve to '=pods=[something.example.org/foo-bar,something.example.org/foo-baz,...]'")
+	o.cmd.Flags().Var(&o.LabelsAllowList, "metric-labels-allowlist", "Comma-separated list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the labels metrics are not exposed. To include them, provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. A single '*' can be provided per resource instead to allow any labels, but that has severe performance implications (Example: '=pods=[*]'). Additionally, an asterisk (*) can be provided as a key, which will resolve to all resources, i.e., assuming '--resources=deployments,pods', '=*=[*]' will resolve to '=deployments=[*],pods=[*]'. Wildcards can also be used to match multiple labels, but only a single '*' is supported per key pattern, i.e., '=pods=[something.example.org/foo-*]' will match labels such as 'something.example.org/foo-bar' and 'something.example.org/foo-baz'.")


+	for i, l := range allowList {
+		// only the first label can be the wildcard label
+		if l == options.LabelWildcard {
+			if i == 0 {
+				return kubeMapToPrometheusLabels(prefix, allKubeData)
+			}
+			continue
 		}

-		for _, l := range allowList {
-			v, found := allKubeData[l]
-			if found {
-				allowedKubeData[l] = v
+		// prepare regular expression based on potential wildcards
+		re, err := regexp.Compile(expandWildcard(l, options.MaxPartialWildcardsPerLabel))
+		if err != nil {
+			continue
+		}
+		for k, v := range allKubeData {
+			if re.MatchString(k) {
+				allowedKubeData[k] = v
 			}
 		}


+// expandWildcard expands wildcards (*) to regular expressions, up to a limited number of wildcards
+func expandWildcard(pattern string, limit uint) string {
+	var result strings.Builder
+	var replacements uint
+	for i, literal := range strings.Split(pattern, options.LabelWildcard) {
+		if i > 0 {
+			result.WriteString(".*")
+			replacements++
+		}
+
+		result.WriteString(regexp.QuoteMeta(literal))
+		if replacements >= limit {
+			break
+		}
+	}
+	return "^" + result.String() + "$"
+}


k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 12, 2026

k8s-ci-robot requested a review from CatherineF-dev February 12, 2026 16:22

k8s-ci-robot requested a review from mrueg February 12, 2026 16:22

github-project-automation Bot added this to SIG Instrumentation Feb 12, 2026

github-project-automation Bot moved this to Needs Triage in SIG Instrumentation Feb 12, 2026

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 12, 2026

skoef force-pushed the annotations-wildcard branch from 45d6189 to 1330742 Compare February 12, 2026 16:23

skoef changed the title ~~allow wildcards as part of annotations and labels in allowlists~~ allow wildcards as part of feat: annotations and labels in allowlists Feb 12, 2026

skoef changed the title ~~allow wildcards as part of feat: annotations and labels in allowlists~~ feat: allow wildcards as part of annotations and labels in allowlists Feb 12, 2026

skoef force-pushed the annotations-wildcard branch from 1330742 to 411c93f Compare February 12, 2026 19:42

bhope reviewed Mar 1, 2026

View reviewed changes

Comment thread internal/store/utils_test.go Outdated

bhope reviewed Mar 1, 2026

View reviewed changes

Comment thread internal/store/utils.go

bhope reviewed Mar 1, 2026

View reviewed changes

Comment thread internal/store/utils.go

k8s-ci-robot assigned bhope Mar 2, 2026

skoef force-pushed the annotations-wildcard branch from 411c93f to 7b8a235 Compare March 2, 2026 10:51

skoef requested a review from bhope March 2, 2026 13:03

skoef force-pushed the annotations-wildcard branch from 7b8a235 to fe85b50 Compare March 2, 2026 13:11

bhope suggested changes Mar 3, 2026

View reviewed changes

Comment thread internal/store/utils.go

Comment thread internal/store/utils.go

github-project-automation Bot moved this from Needs Triage to In Progress in SIG Instrumentation Mar 3, 2026

skoef force-pushed the annotations-wildcard branch 3 times, most recently from fbd4fe8 to e621031 Compare March 4, 2026 10:30

skoef requested a review from bhope March 4, 2026 10:30

bhope reviewed Mar 8, 2026

View reviewed changes

Comment thread docs/developer/cli-arguments.md Outdated

Comment thread docs/developer/cli-arguments.md Outdated

skoef force-pushed the annotations-wildcard branch 2 times, most recently from bcd03f1 to 06049ee Compare March 9, 2026 10:44

skoef requested a review from bhope March 9, 2026 15:21

bhope mentioned this pull request Mar 9, 2026

REQUEST: New membership for @bhope kubernetes/org#6196

Closed

11 tasks

bhope approved these changes Mar 9, 2026

View reviewed changes

bhope mentioned this pull request Mar 13, 2026

chore: add @bhope as a reviewer #2897

Merged

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 17, 2026

rexagod requested a review from Copilot April 21, 2026 10:45

Copilot started reviewing on behalf of rexagod April 21, 2026 10:46 View session

Copilot AI reviewed Apr 21, 2026

View reviewed changes

rexagod reviewed Apr 21, 2026

View reviewed changes

feat: allow wildcards as part of annotations and labels in allowlists

a59160e

Partial labels and annotations are possible in --metric-annotations-allowlist and --metric-labels-allowlist with this change Signed-off-by: Reinier Schoof <reinier@skoef.nl>

skoef force-pushed the annotations-wildcard branch from 06049ee to a59160e Compare April 21, 2026 12:06

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 21, 2026

skoef requested a review from rexagod April 21, 2026 12:13

rexagod mentioned this pull request Apr 22, 2026

Propose release v2.19.0 #2919

Closed

rexagod requested a review from Copilot May 11, 2026 03:32

Copilot started reviewing on behalf of rexagod May 11, 2026 03:33 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

		o.cmd.Flags().Var(&o.AnnotationsAllowList, "metric-annotations-allowlist", "Comma-separated list of Kubernetes annotation keys that will be used in the resource' labels metric. By default the annotations metrics are not exposed. To include them, provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. A single '' can be provided per resource instead to allow any annotations, but that has severe performance implications (Example: '=pods=[]'). Additionally, an asterisk () can be provided for resources, which will resolve to all resources, i.e., assuming '--resources=deployments,pods', '==[]' will resolve to '=deployments=[],pods=[]'. Wildcards can also be use to match multiple annotations, i.e., '=pods=[something.example.org/foo-]' will resolve to '=pods=[something.example.org/foo-bar,something.example.org/foo-baz,...]'")
		o.cmd.Flags().Var(&o.LabelsAllowList, "metric-labels-allowlist", "Comma-separated list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the labels metrics are not exposed. To include them, provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. A single '' can be provided per resource instead to allow any labels, but that has severe performance implications (Example: '=pods=[]'). Additionally, an asterisk () can be provided as a key, which will resolve to all resources, i.e., assuming '--resources=deployments,pods', '==[]' will resolve to '=deployments=[],pods=[]'. Wildcards can also be used to match multiple labels, but only a single '' is supported per key pattern, i.e., '=pods=[something.example.org/foo-*]' will match labels such as 'something.example.org/foo-bar' and 'something.example.org/foo-baz'.")

Conversation

skoef commented Feb 12, 2026

Uh oh!

k8s-ci-robot commented Feb 12, 2026

Uh oh!

k8s-ci-robot commented Feb 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bhope commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bhope commented Mar 2, 2026

Uh oh!

Uh oh!

Uh oh!

bhope left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

k8s-ci-robot commented Mar 8, 2026

Uh oh!

bhope left a comment

Choose a reason for hiding this comment

Uh oh!

bhope commented Mar 17, 2026

Uh oh!

k8s-ci-robot commented Mar 17, 2026

Uh oh!

skoef commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

skoef Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rexagod Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

skoef Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

rexagod commented Apr 21, 2026

Uh oh!

k8s-ci-robot commented Apr 21, 2026

Uh oh!

skoef commented May 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bhope commented Mar 1, 2026 •

edited

Loading

bhope left a comment •

edited

Loading

skoef Apr 21, 2026 •

edited

Loading