Skip to content

Failure Taxonomy

Failure taxonomy

Failure taxonomy turns raw exceptions into operational meaning. ChokaQ stores a FailureReason when a job lands in DLQ so operators can triage by category instead of reading every stack trace first.

Why Taxonomy Matters

Two failed jobs can require very different actions:

  • a transient HTTP timeout may be safe to retry;
  • a malformed payload should be fixed before retry;
  • a zombie job may have already sent an email or charged a card;
  • an admin-cancelled job may be intentionally stopped.

The taxonomy preserves that distinction in data.

Common Reasons

ReasonMeaningTypical response
MaxRetriesExceededRetry budget exhausted.Fix cause, then sample resurrection.
FatalErrorClassified as non-transient.Fix code or payload first.
TimeoutHandler exceeded timeout.Tune handler or timeout; inspect side effects.
CancelledOperator/runtime cancellation.Confirm intent.
ZombieHeartbeat expired during processing.Inspect side effects before retry.
CircuitBreakerOpenExecution blocked by open circuit.Fix downstream before forcing recovery.
ThrottledDownstream overload/rate limit.Respect retry-after and reduce pressure.
TransientRetryable family eventually exhausted.Check downstream instability.
HeartbeatFailureHeartbeat write failure crossed policy.Check SQL/storage pressure.
RetryLifetimeExpiredJob became too old to retry.Treat as stale work.

Smart Worker Relationship

Smart Worker classification decides whether a failure should retry or fast-fail to DLQ. Failure taxonomy is the persisted operator-facing result.

Architecture Decision

Why this pattern?

Operators need routing metadata. Raw exception text is useful for debugging, but safe bulk actions, dashboards, and incident triage also need stable failure categories.

Trade-offs

Classification can be wrong if application exceptions are too generic. Apps should throw meaningful fatal/throttled/transient signals where possible.

Alternatives considered

AlternativeBenefitCost
Store stack traces onlySimple.Poor filtering and unsafe bulk decisions.
Use exception type onlyAutomatic.Loses operational categories like zombie/cancelled.
App-owned categories onlyFlexible.No consistent runtime semantics.

Additional Questions

Why persist failure reason separately from error details?
Because operators need stable categories for filtering, dashboards, and actions.

Which failures need inspection before retry?
Zombie, fatal payload/code errors, cancelled jobs, and stale jobs beyond lifetime policy.

How does this help The Deck?
The Deck can group, filter, badge, and bulk-preview based on reason instead of free-form exception text.

Apache 2.0 Licensed