Gallery
Terminal Ledger Incident Record Data & Reports
Palette
Primary
Accent
Background
Card
Foreground
SEV-2

INC-0418 · QUEUE.INGEST CONSUMER STALL

DURATION
2h 33m
DETECTED
Alert, 27m after onset
IMPACT
4.2M messages delayed, 0 lost
OWNER
platform-ingest
  1. 09:14
    ONSET

    Partition 11 added by autoscaler. Consumer group begins rebalance.

  2. 27m undetected
  3. 09:41
    DETECTED

    Lag alert fires at 500K threshold. Page delivered to on-call.

  4. 09:48
    ACKNOWLEDGED

    On-call acknowledges. Initial hypothesis: upstream producer spike. Incorrect.

  5. 1h 46m mitigating the wrong thing
  6. 11:34
    DIAGNOSED

    Member 7f2a identified as unresponsive; rebalance never completed.

  7. 11:47
    MITIGATED

    Member evicted. Rebalance completes in 9s. Lag drains over 14m.

  8. 12:01
    RESOLVED

    Lag below threshold. Monitoring for 45m before closing.

ACTIONS

  • P1Alert on rebalance duration, not only on lag — the lag alert fired 27m late by construction.platform-ingest
  • P1Evict unresponsive group members automatically after 2 failed heartbeats.platform-ingest
  • P2Runbook: rule out rebalance before investigating producer volume.sre
  • P3Dashboard: put group membership on the ingest panel.sre