What to do when something is on fire.
Degraded means a check is failing, not that the service is down. Look at status first to see which check, then at that service's page. Do not restart a degraded service as a first move — most degradations here are downstream and a restart discards in-flight work for no benefit.
Check the schema version in the envelope. The most common cause by some distance is a partner pinning to a version that has since been removed. The reject row names the failing field.
ingestctl rejects list --since 24h ingestctl rejects replay <id>
Replaying is idempotent. Replaying a batch that already succeeded is a no-op rather than a duplicate.
Almost always one bad destination consuming retry budget. Find it before raising the retry ceiling.
notifyctl queue top --by destination
See on-call. If you are reading this at 03:00 and nothing above matches, escalating is the right call and nobody will mind.
Console