From 198aeeb919c41af4bb5f1bc0a443a131fa5f89f6 Mon Sep 17 00:00:00 2001 From: Shelly Matskel Date: Fri, 22 May 2026 11:52:24 -0400 Subject: [PATCH 1/5] Add Kafka Monitoring Data Collection page Document the metrics, configurations, and capabilities collected by the Kafka Consumer integration when enable_cluster_monitoring is set, and add a side nav entry under Kafka Monitoring. Co-Authored-By: Claude Opus 4.7 (1M context) --- config/_default/menus/main.en.yaml | 5 ++ .../en/data_streams/kafka/data_collection.md | 67 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100644 content/en/data_streams/kafka/data_collection.md diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 45c293b10b2..6b56c0a9abf 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -5133,6 +5133,11 @@ menu: identifier: data_streams_kafka_monitors_and_automation parent: data_streams_kafka weight: 2 + - name: Data Collection + url: data_streams/kafka/data_collection + identifier: data_streams_kafka_data_collection + parent: data_streams_kafka + weight: 3 - name: Schema Tracking url: data_streams/schema_tracking identifier: data_streams_schema_tracking diff --git a/content/en/data_streams/kafka/data_collection.md b/content/en/data_streams/kafka/data_collection.md new file mode 100644 index 00000000000..d3c69054687 --- /dev/null +++ b/content/en/data_streams/kafka/data_collection.md @@ -0,0 +1,67 @@ +--- +title: Kafka Monitoring Data Collection +description: Metrics, configurations, and capabilities collected by Data Streams Monitoring's Kafka Monitoring when cluster monitoring is enabled. +--- + +Data Streams Monitoring's Kafka Monitoring collects cluster health data through the [Kafka Consumer integration][1]. Setting `enable_cluster_monitoring: true` in the Kafka Consumer check expands the integration's capabilities and unlocks the additional data described on this page. + +## Metrics + +With `enable_cluster_monitoring: true`, the Kafka Consumer integration sends the following metrics in addition to its default set: + +| Metric | Description | +|--------------------------------------------------|--------------------------------------------------------------| +| `kafka.cluster.controller_id` | ID of the broker acting as the cluster controller | +| `kafka.broker.count` | Total number of brokers in the cluster | +| `kafka.broker.leader_count` | Number of partitions for which this broker is the leader | +| `kafka.broker.partition_count` | Total number of partitions on this broker including replicas | +| `kafka.broker.config.log_retention_bytes` | Broker configuration for log retention in bytes | +| `kafka.broker.config.log_retention_ms` | Broker configuration for log retention in milliseconds | +| `kafka.broker.config.log_segment_bytes` | Broker configuration for log segment size in bytes | +| `kafka.broker.config.num_partitions` | Broker configuration for default number of partitions | +| `kafka.broker.config.num_network_threads` | Broker configuration for number of network threads | +| `kafka.broker.config.num_io_threads` | Broker configuration for number of I/O threads | +| `kafka.broker.config.default_replication_factor` | Broker configuration for default replication factor | +| `kafka.broker.config.min_insync_replicas` | Broker configuration for minimum in-sync replicas | +| `kafka.topic.count` | Total number of topics in the cluster | +| `kafka.topic.partitions` | Number of partitions for this topic | +| `kafka.topic.size` | Total number of messages in the topic | +| `kafka.topic.message_rate` | Message production rate for this topic | +| `kafka.topic.config.retention_ms` | Topic configuration for retention time in milliseconds | +| `kafka.topic.config.retention_bytes` | Topic configuration for retention size in bytes | +| `kafka.topic.config.max_message_bytes` | Topic configuration for maximum message size in bytes | +| `kafka.partition.beginning_offset` | The earliest offset in the partition | +| `kafka.partition.replicas` | Number of replicas for this partition | +| `kafka.partition.isr` | Number of in-sync replicas for this partition | +| `kafka.partition.size` | Number of messages in the partition | +| `kafka.partition.under_replicated` | Whether this partition is under-replicated (1) or not (0) | +| `kafka.partition.offline` | Whether this partition is offline (1) or not (0) | +| `kafka.consumer_group.count` | Total number of consumer groups | +| `kafka.consumer_group.members` | Number of members in the consumer group | +| `kafka.consumer_group.member.partitions` | Number of partitions assigned to this consumer group member | + +Enabling the flag also collects these metrics across the entire cluster, rather than for selected topics only: + +| Metric | Description | +|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `kafka.broker_offset` | Current message offset on broker. *Shown as offset* | +| `kafka.consumer_offset` | Current message offset on consumer. *Shown as offset* | +| `kafka.consumer_lag` | Lag in messages between consumer and broker. *Shown as message* | +| `kafka.estimated_consumer_lag` | Lag in seconds between consumer and broker. This metric is provided through Data Streams Monitoring. Additional charges may apply. *Shown as second* | + +## Configurations and schemas + +With `enable_cluster_monitoring: true`, the Kafka Consumer integration also collects: + +- Broker configurations +- Topic configurations +- Schemas from a schema registry + +These appear in Data Streams Monitoring alongside the metrics above. + +## Capabilities + +Enabling cluster monitoring also unlocks the ability to read live messages on topics on demand. See [Enable message inspection][2] for the additional setup steps. + +[1]: /integrations/kafka-consumer/?tab=containerized +[2]: /data_streams/kafka/setup/#enable-message-inspection From 70c66ca59dac914cd2e8558b284a39b5c951c52f Mon Sep 17 00:00:00 2001 From: Shelly Matskel Date: Fri, 22 May 2026 12:56:42 -0400 Subject: [PATCH 2/5] Refine Data Collection page after review - Move Data Collection above Monitors and Automation in the side nav - Rename page from "Kafka Monitoring Data Collection" to "Data Collection" - Link to Kafka Monitoring Setup from the intro - Link "read live messages on topics" to the Kafka Monitoring overview section about schemas and messages - Tighten section openers and remove repetitive references to the enable_cluster_monitoring flag Co-Authored-By: Claude Opus 4.7 (1M context) --- config/_default/menus/main.en.yaml | 10 +++++----- content/en/data_streams/kafka/data_collection.md | 16 ++++++++-------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 6b56c0a9abf..878ec3dd24d 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -5128,15 +5128,15 @@ menu: identifier: data_streams_kafka_setup parent: data_streams_kafka weight: 1 - - name: Monitors and Automation - url: data_streams/kafka/monitors_and_automation - identifier: data_streams_kafka_monitors_and_automation - parent: data_streams_kafka - weight: 2 - name: Data Collection url: data_streams/kafka/data_collection identifier: data_streams_kafka_data_collection parent: data_streams_kafka + weight: 2 + - name: Monitors and Automation + url: data_streams/kafka/monitors_and_automation + identifier: data_streams_kafka_monitors_and_automation + parent: data_streams_kafka weight: 3 - name: Schema Tracking url: data_streams/schema_tracking diff --git a/content/en/data_streams/kafka/data_collection.md b/content/en/data_streams/kafka/data_collection.md index d3c69054687..35a2c1375f1 100644 --- a/content/en/data_streams/kafka/data_collection.md +++ b/content/en/data_streams/kafka/data_collection.md @@ -1,13 +1,13 @@ --- -title: Kafka Monitoring Data Collection +title: Data Collection description: Metrics, configurations, and capabilities collected by Data Streams Monitoring's Kafka Monitoring when cluster monitoring is enabled. --- -Data Streams Monitoring's Kafka Monitoring collects cluster health data through the [Kafka Consumer integration][1]. Setting `enable_cluster_monitoring: true` in the Kafka Consumer check expands the integration's capabilities and unlocks the additional data described on this page. +Data Streams Monitoring's Kafka Monitoring collects cluster health data through the [Kafka Consumer integration][1]. With `enable_cluster_monitoring: true`, the integration collects additional metrics and unlocks the extra capabilities described on this page. -## Metrics +See [Kafka Monitoring Setup][4] for how to configure the check. -With `enable_cluster_monitoring: true`, the Kafka Consumer integration sends the following metrics in addition to its default set: +## Metrics | Metric | Description | |--------------------------------------------------|--------------------------------------------------------------| @@ -40,7 +40,7 @@ With `enable_cluster_monitoring: true`, the Kafka Consumer integration sends the | `kafka.consumer_group.members` | Number of members in the consumer group | | `kafka.consumer_group.member.partitions` | Number of partitions assigned to this consumer group member | -Enabling the flag also collects these metrics across the entire cluster, rather than for selected topics only: +Now collected across the entire cluster instead of for selected topics only: | Metric | Description | |---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -51,8 +51,6 @@ Enabling the flag also collects these metrics across the entire cluster, rather ## Configurations and schemas -With `enable_cluster_monitoring: true`, the Kafka Consumer integration also collects: - - Broker configurations - Topic configurations - Schemas from a schema registry @@ -61,7 +59,9 @@ These appear in Data Streams Monitoring alongside the metrics above. ## Capabilities -Enabling cluster monitoring also unlocks the ability to read live messages on topics on demand. See [Enable message inspection][2] for the additional setup steps. +Enabling cluster monitoring also unlocks the ability to [read live messages on topics][3] on demand. See [Enable message inspection][2] for the additional setup steps. [1]: /integrations/kafka-consumer/?tab=containerized [2]: /data_streams/kafka/setup/#enable-message-inspection +[3]: /data_streams/kafka/#inspect-topic-schemas-and-messages +[4]: /data_streams/kafka/setup/ From ddf51fb1c4f8d31c3c34f1283a15e8ff376194f7 Mon Sep 17 00:00:00 2001 From: Shelly Matskel Date: Fri, 22 May 2026 12:59:21 -0400 Subject: [PATCH 3/5] Break unit hint onto its own line in expanded-scope metrics table Co-Authored-By: Claude Opus 4.7 (1M context) --- content/en/data_streams/kafka/data_collection.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/data_streams/kafka/data_collection.md b/content/en/data_streams/kafka/data_collection.md index 35a2c1375f1..7caab52c0ac 100644 --- a/content/en/data_streams/kafka/data_collection.md +++ b/content/en/data_streams/kafka/data_collection.md @@ -44,10 +44,10 @@ Now collected across the entire cluster instead of for selected topics only: | Metric | Description | |---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `kafka.broker_offset` | Current message offset on broker. *Shown as offset* | -| `kafka.consumer_offset` | Current message offset on consumer. *Shown as offset* | -| `kafka.consumer_lag` | Lag in messages between consumer and broker. *Shown as message* | -| `kafka.estimated_consumer_lag` | Lag in seconds between consumer and broker. This metric is provided through Data Streams Monitoring. Additional charges may apply. *Shown as second* | +| `kafka.broker_offset` | Current message offset on broker.
*Shown as offset* | +| `kafka.consumer_offset` | Current message offset on consumer.
*Shown as offset* | +| `kafka.consumer_lag` | Lag in messages between consumer and broker.
*Shown as message* | +| `kafka.estimated_consumer_lag` | Lag in seconds between consumer and broker. This metric is provided through Data Streams Monitoring. Additional charges may apply.
*Shown as second* | ## Configurations and schemas From 701794f9be6763646fbd03956448065a580ab5a3 Mon Sep 17 00:00:00 2001 From: Shelly Matskel Date: Fri, 22 May 2026 13:01:15 -0400 Subject: [PATCH 4/5] Break unit hint onto its own line in expanded-scope metrics table Also widen the first column to match the metrics table above. Co-Authored-By: Claude Opus 4.7 (1M context) --- content/en/data_streams/kafka/data_collection.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/data_streams/kafka/data_collection.md b/content/en/data_streams/kafka/data_collection.md index 7caab52c0ac..9937968b2fc 100644 --- a/content/en/data_streams/kafka/data_collection.md +++ b/content/en/data_streams/kafka/data_collection.md @@ -42,12 +42,12 @@ See [Kafka Monitoring Setup][4] for how to configure the check. Now collected across the entire cluster instead of for selected topics only: -| Metric | Description | -|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `kafka.broker_offset` | Current message offset on broker.
*Shown as offset* | -| `kafka.consumer_offset` | Current message offset on consumer.
*Shown as offset* | -| `kafka.consumer_lag` | Lag in messages between consumer and broker.
*Shown as message* | -| `kafka.estimated_consumer_lag` | Lag in seconds between consumer and broker. This metric is provided through Data Streams Monitoring. Additional charges may apply.
*Shown as second* | +| Metric | Description | +|--------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| +| `kafka.broker_offset` | Current message offset on broker.
*Shown as offset* | +| `kafka.consumer_offset` | Current message offset on consumer.
*Shown as offset* | +| `kafka.consumer_lag` | Lag in messages between consumer and broker.
*Shown as message* | +| `kafka.estimated_consumer_lag` | Lag in seconds between consumer and broker. This metric is provided through Data Streams Monitoring. Additional charges may apply.
*Shown as second* | ## Configurations and schemas From ac55aea1a952a3b94a71da0f47ab0e4836ba0f82 Mon Sep 17 00:00:00 2001 From: Shelly Matskel Date: Fri, 22 May 2026 14:09:51 -0400 Subject: [PATCH 5/5] Refactor Metrics section to point at integration source of truth After review, replace the inline metrics tables with a pointer to the Kafka Consumer integration's metric list, where Data Streams Monitoring metrics are marked DSM only. Tighten the intro and Configurations section accordingly. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../en/data_streams/kafka/data_collection.md | 50 ++----------------- 1 file changed, 5 insertions(+), 45 deletions(-) diff --git a/content/en/data_streams/kafka/data_collection.md b/content/en/data_streams/kafka/data_collection.md index 9937968b2fc..ac173cdaab4 100644 --- a/content/en/data_streams/kafka/data_collection.md +++ b/content/en/data_streams/kafka/data_collection.md @@ -3,51 +3,11 @@ title: Data Collection description: Metrics, configurations, and capabilities collected by Data Streams Monitoring's Kafka Monitoring when cluster monitoring is enabled. --- -Data Streams Monitoring's Kafka Monitoring collects cluster health data through the [Kafka Consumer integration][1]. With `enable_cluster_monitoring: true`, the integration collects additional metrics and unlocks the extra capabilities described on this page. - -See [Kafka Monitoring Setup][4] for how to configure the check. +Data Streams Monitoring's [Kafka Monitoring][6] uses the [Kafka Consumer integration][1] to collect cluster health data. On top of the integration's defaults, it collects additional data and adds ability to perform actions described in this doc. ## Metrics -| Metric | Description | -|--------------------------------------------------|--------------------------------------------------------------| -| `kafka.cluster.controller_id` | ID of the broker acting as the cluster controller | -| `kafka.broker.count` | Total number of brokers in the cluster | -| `kafka.broker.leader_count` | Number of partitions for which this broker is the leader | -| `kafka.broker.partition_count` | Total number of partitions on this broker including replicas | -| `kafka.broker.config.log_retention_bytes` | Broker configuration for log retention in bytes | -| `kafka.broker.config.log_retention_ms` | Broker configuration for log retention in milliseconds | -| `kafka.broker.config.log_segment_bytes` | Broker configuration for log segment size in bytes | -| `kafka.broker.config.num_partitions` | Broker configuration for default number of partitions | -| `kafka.broker.config.num_network_threads` | Broker configuration for number of network threads | -| `kafka.broker.config.num_io_threads` | Broker configuration for number of I/O threads | -| `kafka.broker.config.default_replication_factor` | Broker configuration for default replication factor | -| `kafka.broker.config.min_insync_replicas` | Broker configuration for minimum in-sync replicas | -| `kafka.topic.count` | Total number of topics in the cluster | -| `kafka.topic.partitions` | Number of partitions for this topic | -| `kafka.topic.size` | Total number of messages in the topic | -| `kafka.topic.message_rate` | Message production rate for this topic | -| `kafka.topic.config.retention_ms` | Topic configuration for retention time in milliseconds | -| `kafka.topic.config.retention_bytes` | Topic configuration for retention size in bytes | -| `kafka.topic.config.max_message_bytes` | Topic configuration for maximum message size in bytes | -| `kafka.partition.beginning_offset` | The earliest offset in the partition | -| `kafka.partition.replicas` | Number of replicas for this partition | -| `kafka.partition.isr` | Number of in-sync replicas for this partition | -| `kafka.partition.size` | Number of messages in the partition | -| `kafka.partition.under_replicated` | Whether this partition is under-replicated (1) or not (0) | -| `kafka.partition.offline` | Whether this partition is offline (1) or not (0) | -| `kafka.consumer_group.count` | Total number of consumer groups | -| `kafka.consumer_group.members` | Number of members in the consumer group | -| `kafka.consumer_group.member.partitions` | Number of partitions assigned to this consumer group member | - -Now collected across the entire cluster instead of for selected topics only: - -| Metric | Description | -|--------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| -| `kafka.broker_offset` | Current message offset on broker.
*Shown as offset* | -| `kafka.consumer_offset` | Current message offset on consumer.
*Shown as offset* | -| `kafka.consumer_lag` | Lag in messages between consumer and broker.
*Shown as message* | -| `kafka.estimated_consumer_lag` | Lag in seconds between consumer and broker. This metric is provided through Data Streams Monitoring. Additional charges may apply.
*Shown as second* | +To collect the **DSM only** [Kafka Consumer integration metrics][5], set `enable_cluster_monitoring: true`. See [Kafka Monitoring Setup][4] for how to configure the check. ## Configurations and schemas @@ -55,13 +15,13 @@ Now collected across the entire cluster instead of for selected topics only: - Topic configurations - Schemas from a schema registry -These appear in Data Streams Monitoring alongside the metrics above. - ## Capabilities -Enabling cluster monitoring also unlocks the ability to [read live messages on topics][3] on demand. See [Enable message inspection][2] for the additional setup steps. +Enabling cluster monitoring unlocks the ability to [read live messages on topics][3] on demand. See [Enable message inspection][2] for the additional setup steps. [1]: /integrations/kafka-consumer/?tab=containerized [2]: /data_streams/kafka/setup/#enable-message-inspection [3]: /data_streams/kafka/#inspect-topic-schemas-and-messages [4]: /data_streams/kafka/setup/ +[5]: /integrations/kafka-consumer/?tab=containerized#metrics +[6]: /data_streams/kafka/