Skip to content

v0.9.0 - Streaming Integrations

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 May 19:00
v0.9.0
84cf9b6

ExDataSketch v0.9.0 Release Notes

Release date: 2026-05-20
Theme: Streaming Integrations -- the BEAM-native streaming approximate analytics infrastructure layer.

Highlights

Stream and Collectable Integration

All 13 mergeable sketch types now implement the Collectable protocol and support lazy stream consumption via ExDataSketch.Stream:

# Lazy stream consumption
sketch = 1..100_000 |> Stream.map(&to_string/1) |> ExDataSketch.Stream.hll(p: 14)

# Collectable protocol
sketch = Enum.into(1..50_000, ExDataSketch.ULL.new(p: 14))

# Partitioned parallel reduction
sketch = ExDataSketch.Stream.reduce_partitioned(1..1_000_000, ExDataSketch.HLL, partitions: 8, p: 14)

Broadway, GenStage, and Flow

Production-grade integration for streaming pipelines:

  • ExDataSketch.Broadway.accumulate/3 -- build sketches from Broadway message batches
  • ExDataSketch.Broadway.PeriodicAggregator -- periodic accumulation with timer flush
  • ExDataSketch.GenStage.SketchConsumer -- back-pressure-aware sketch accumulation
  • ExDataSketch.GenStage.SketchProducer -- emit accumulated sketches on demand
  • ExDataSketch.Flow.reduce/3 and merge/2 -- parallel partition-local reduction

Five Persistence Backends

Save, load, merge, and delete sketches with atomic operations:

# ETS (in-memory)
ExDataSketch.Storage.ETS.save(sketch, table, "daily:2024-01-15")
{:ok, loaded} = ExDataSketch.Storage.ETS.load(ExDataSketch.HLL, table, "daily:2024-01-15")
ExDataSketch.Storage.ETS.merge(partial, table, "daily:2024-01-15")  # atomic merge

# Also: DETS, CubDB, Mnesia, Ecto

Production-Grade Telemetry

Structured :telemetry events at batch boundaries with category-based enable/disable:

:telemetry.attach("my-handler", [:ex_data_sketch, :sketch, :ingest], fn _name, meas, meta, _ ->
  Logger.info("#{meta.sketch_type}: #{meas.count} items in #{meas.duration}ns")
end, nil)

# OpenTelemetry bridge (optional)
ExDataSketch.Telemetry.OpenTelemetry.setup()

ULL Accuracy Fix

UltraLogLog estimates with few registers now use linear counting correction (when zeros > 0) and large-range bias correction, improving accuracy from 62.5% error to 0.8% at p=8/n=1000.

v1 Serialization Escape Hatch

# Produce backward-compatible v0.7.x binary (requires :phash2 hash strategy)
binary = ExDataSketch.HLL.serialize(sketch, format: :v1)

Nine Production-Oriented Livebooks

From streaming cardinality to AI token analytics. See guides/livebooks.md for the recommended reading order.

Breaking Changes

None. All v0.8.0 APIs are fully backward compatible.

ULL estimate change: ULL estimates at very low cardinalities (p < 12, n < 500) may differ from v0.8.0. The v0.9.0 estimates are more accurate. Add tolerance for small cardinalities in tests.

New Dependencies

  • :telemetry ~> 1.0 (required, was already a transitive dependency)
  • :broadway, :flow, :cubdb, :ecto_sql, :opentelemetry_api (optional, only loaded when available)

Full Changelog

See CHANGELOG.md for the complete list of changes.

Upgrade Guide

From v0.8.0: No code changes required. Add {:ex_data_sketch, "~> 0.9.0"} to your dependencies.

New features are opt-in:

  • Telemetry is enabled by default; disable with config :ex_data_sketch, telemetry_enabled: false
  • Persistence backends are enabled when their runtime dependency is available
  • Stream/Broadway/GenStage/Flow modules are available when their dependencies are loaded
Assets 18
Loading