High Cardinality Metrics

[mattrussi]

Hi @AndreyVMarkelov, we're running into issues with high cardinality of unbounded metrics like the confluence_user_login_count, confluence_user_logout_count, and confluence_user_failed_login_count metrics which have the username and ip as labels. We have ~7,000 active Confluence users so this causes a huge influx of unique time-series.

The Prometheus devs recommend against this practice for labels.

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

https://prometheus.io/docs/practices/naming/

Instead, could there be an aggregate counter, such as confluence_user_login_total, confluence_user_logout_total, and confluence_user_failed_login_total? This would follow some of the Prometheus best-practices like using total as the suffix for an accumulating count.

Example

From:

confluence_user_login_count{username="usera",ip="10.0.0.1",} 1.0
confluence_user_login_count{username="userb",ip="10.0.0.2",} 20.0

To:

confluence_user_login_total 30.0

[AndreyVMarkelov]

Sure guys. Will do чт, 23 авг. 2018 г., 21:51 mattrussi <notifications@github.com>:

…

Hi @AndreyVMarkelov <https://github.com/AndreyVMarkelov>, we're running into issues with high cardinality of unbounded metrics like the *confluence_user_login_count*, *confluence_user_logout_count*, and *confluence_user_failed_login_count* metrics which have the username and ip as labels. We have ~7,000 active Confluence users so this causes a huge influx of unique time-series. The Prometheus devs recommend against this practice for labels. CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values. https://prometheus.io/docs/practices/naming/ Instead, could there be an aggregate counter, such as *confluence_user_login_total*, *confluence_user_logout_total*, and *confluence_user_failed_login_total*? This would follow some of the Prometheus best-practices like using total as the suffix for an accumulating count. Example From: confluence_user_login_count{username="usera",ip="10.0.0.1",} 1.0 confluence_user_login_count{username="userb",ip="10.0.0.2",} 20.0 To: confluence_user_login_total 30.0 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB92cDf16Xy2AEGr8UOqsgyIo26rD2rRks5uTvm3gaJpZM4WKFyk> .

[AndreyVMarkelov]

In the progress

[adamdmharvey]

Is there any update on this one? Indeed, we see the same thing, and our polling for the Prometheus end point output is over 15,000 lines and 1 meg, every 5 minutes.