Zipline - Observability

Numbers that matter, always visible.

Every connector exposes live metrics at the process level. No agents, no sidecars, no configuration beyond a scrape target.

P50 end-to-end latency

142ms

orders-postgres → kafka-bus
last 5 minutes

Within SLO

Throughput · all connectors

1.24M/s

events/sec · right now
peak today: 1.89M/s

Nominal

Messages lost · since deploy

847B total delivered
uptime: 99.997%

Exactly-once active

Four signal types. Zero setup.

Zipline connectors are self-describing. The moment a connector starts, all four signal types are active and ready to scrape, export, or tail.

Prometheus metrics

A /metrics endpoint on every connector pod. Scrape with any Prometheus-compatible stack - Grafana, Datadog agent, New Relic, or a bare Prometheus server. Histograms for latency, counters for throughput and errors, gauges for lag and offset.

zipline_connector_lag_seconds · zipline_events_total · zipline_latency_bucket

OpenTelemetry traces

OTLP export over gRPC or HTTP. Distributed traces span from the source commit detection all the way to the sink write acknowledgement. Set OTEL_EXPORTER_OTLP_ENDPOINT and traces flow to Jaeger, Tempo, Honeycomb, or any OTLP-compatible backend.

W3C trace context · baggage propagation · span events

Structured JSON logs

Every event is logged with connector name, table, offset, latency, and delivery status. Ship to any log aggregator: Elastic, Splunk, Loki, Datadog Logs, or CloudWatch. Filter on level, connector, or table without parsing free-form text.

JSON · log level · connector · table · offset · latency_ms

Built-in alerting rules

The Zipline Helm chart ships pre-configured Prometheus alerting rules. Lag exceeds threshold, error rate above 0.1%, connector unreachable - all covered without writing a single PromQL expression. Edit thresholds in values.yaml.

lag alert · error rate alert · connector down · custom thresholds

Alerting rules

Alerts you didn't have to write.

The Helm chart installs a complete set of Prometheus alerting rules tuned to real-world streaming workloads. Override any threshold in values.yaml. Add your own rules alongside them.

Rules fire into Alertmanager. Route to PagerDuty, Slack, OpsGenie, or any webhook - Zipline doesn't lock you into a notification channel.

View all alert rules Helm chart values

# Included in zipline Helm chart
groups:
  - name: zipline.rules
    rules:

    - alert: ZiplineLagHigh
      expr: zipline_connector_lag_seconds > 30
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "Connector {{ $labels.connector }} lag {{ $value }}s"
        runbook_url: "https://docs.zipline.run/runbooks/lag"

    - alert: ZiplineErrorRateHigh
      expr: rate(zipline_errors_total[5m]) > 0.001
      for: 1m
      labels:
        severity: critical

    - alert: ZiplineConnectorDown
      expr: up{job="zipline"} == 0
      for: 1m
      labels:
        severity: critical