Skip to content

promhttp: don't panic when instrumenting with non-exemplar observers#2005

Open
spor3006 wants to merge 1 commit into
prometheus:mainfrom
spor3006:fix/1258-summary-observer-panic
Open

promhttp: don't panic when instrumenting with non-exemplar observers#2005
spor3006 wants to merge 1 commit into
prometheus:mainfrom
spor3006:fix/1258-summary-observer-panic

Conversation

@spor3006
Copy link
Copy Markdown

What

InstrumentHandlerDuration and InstrumentHandlerCounter panic at request time when their ObserverVec/CounterVec is backed by a type that does not implement ExemplarObserver/ExemplarAdder. The most common trigger is a SummaryVec, which is a valid prometheus.ObserverVec but whose underlying *summary does not implement ObserveWithExemplar — summaries cannot carry exemplars in the Prometheus exposition format.

Reproduction from #1258:

duration := prometheus.NewSummaryVec(
    prometheus.SummaryOpts{
        Name:       "request_duration_seconds",
        Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
    },
    []string{"code", "method"},
)
http.Handle("/", promhttp.InstrumentHandlerDuration(duration, handler))

panics on the first request with:

interface conversion: *prometheus.summary is not prometheus.ExemplarObserver: missing method ObserveWithExemplar

Why

The unsafe type assertion was inconsistent with the rest of the codebase. Timer.ObserveDurationWithExemplar (prometheus/timer.go:72) has handled the same situation correctly for years using a safe-cast + fallback:

eo, ok := t.observer.(ExemplarObserver)
if ok && exemplar != nil {
    eo.ObserveWithExemplar(d.Seconds(), exemplar)
    return d
}
if t.observer != nil {
    t.observer.Observe(d.Seconds())
}

The two helpers in promhttp/instrument_server.go (observeWithExemplar, addWithExemplar) did obs.(prometheus.ExemplarObserver) without the ok check, turning a documented exemplar-not-supported case into a runtime panic in the request path.

Changes

  1. observeWithExemplar / addWithExemplar use the safe-cast + fallback pattern. If the observer/counter implements the exemplar interface, the exemplar is attached; otherwise the exemplar is silently dropped and the value is recorded with the plain Observe/Add path. No public API change.
  2. Doc comments on both helpers describe the fallback explicitly and cross-reference Timer.ObserveDurationWithExemplar for consistency.
  3. CHANGELOG [BUGFIX] entry under Unreleased.

Tests

Three regression tests in instrument_server_test.go:

  • TestMiddlewareAPI_SummaryWithExemplars — exact issue reproduction: InstrumentHandlerDuration(summaryVec, …, WithExemplarFromContext(…)) + one HTTP request. Fails on main with the panic; passes here.
  • TestObserveWithExemplar_NonExemplarObserverFallsBack — unit-level contract for the helper.
  • TestAddWithExemplar_NonExemplarAdderFallsBack — same for the counter helper.

All three are stable under go test -count=10 -race.

Verification

  • go test ./prometheus/promhttp/... — passes (existing + new).
  • go test -race ./prometheus/promhttp/... — passes.
  • go vet ./prometheus/promhttp/... — clean.
  • gofmt — clean.

Note: go test ./prometheus/collectors/... fails on this branch and on main because Go 1.25 added runtime metrics (htmlmetacontenturlescape, httpcookiemaxnum, urlmaxqueryparams) that aren't in the expectedMetrics fixture. Pre-existing, unrelated to this fix.

Trade-offs considered

  • Silently dropping the exemplar. The helpers have no error return and promhttp has no logger. Silent fallback matches Timer.ObserveDurationWithExemplar. The new behavior is documented in the godoc.
  • Making *summary implement ExemplarObserver. Rejected: the Prometheus exposition format does not support exemplars on summaries, so accepting them would either silently drop at write time (worse — bug moves to a less visible site) or fabricate a different code path. Keeping the interface as the contract is the right design.

Closes #1258.

/cc @bwplotka @kakkoyun @vesari

InstrumentHandlerDuration and InstrumentHandlerCounter unconditionally
type-asserted their observer/counter to ExemplarObserver / ExemplarAdder
when an exemplar was provided. This panicked at request time if the
caller passed a SummaryVec — a valid prometheus.ObserverVec whose
underlying *summary does not implement ObserveWithExemplar, because
summaries cannot carry exemplars in the Prometheus exposition format.

Switch to the safe-cast + fallback pattern already used by
Timer.ObserveDurationWithExemplar (prometheus/timer.go): if the
observer/counter does not implement the exemplar interface, drop the
exemplar and record the value with the plain Observe/Add path. No
public API change.

Adds three regression tests:
  - TestMiddlewareAPI_SummaryWithExemplars: exact panic reproduction
    from the upstream issue, through InstrumentHandlerDuration.
  - TestObserveWithExemplar_NonExemplarObserverFallsBack: unit-level
    contract for the helper.
  - TestAddWithExemplar_NonExemplarAdderFallsBack: same, for the
    counter helper.

Fixes prometheus#1258

Signed-off-by: Sparshal Kothari <41056517+spor3006@users.noreply.github.com>
@spor3006 spor3006 force-pushed the fix/1258-summary-observer-panic branch from 3928966 to eb1ce2c Compare May 13, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can't use an ObserverVec with promhttp.InstrumentHandlerDuration()

1 participant