Skip to content

Notifications

SLOzy delivers multi-channel alert notifications through internal/notifications/.

Architecture

SLO Burn Rate → Notification Service → Queue → Workers → Channel Delivery

The Service (internal/notifications/service.go) manages delivery with background workers, deduplication, and retry logic.

Channels

ChannelDelivery ImplementationConfiguration Required
EmailSMTP or HTTP API (deliverViaAPI)to, from, provider endpoint
SlackSlack webhook with rich attachmentswebhook_url
WebhookGeneric HTTP POSTwebhook_url, optional headers

Each channel implements the ChannelDelivery interface:

go
type ChannelDelivery interface {
    Deliver(ctx context.Context, notification *Notification, config map[string]string) error
    Validate(config map[string]string) error
    GetName() string
}

Alert Rules & Priority Levels

Notifications carry a priority level defined in NotificationPriority:

go
const (
    NotificationPriorityLow      = "low"
    NotificationPriorityMedium   = "medium"
    NotificationPriorityHigh     = "high"
    NotificationPriorityCritical = "critical"
)

Alert types: violation, warning, recovery, manual.

Slack messages use color-coded attachments based on priority (red for critical, orange for high, yellow for medium, green for low).

Retry Logic

The service uses exponential backoff:

go
func (s *Service) calculateRetryDelay(attempt int) time.Duration {
    delay := float64(baseDelay) * (backoffMultiplier ^ attempt)
    return min(delay, 5 * time.Minute)
}
  • Configurable MaxRetries, RetryDelay, and RetryBackoffMultiplier
  • Delivery capped at 5 minutes max per attempt

Deduplication

When enabled, the service checks for duplicate notifications within a configurable window using a cache key:

notif:{slo_id}:{channel_id}:{alert_type}:{priority}

Configuration

go
type NotificationConfig struct {
    MaxRetries            int
    RetryDelay            time.Duration
    RetryBackoffMultiplier float64
    AsyncQueueSize        int
    EnableAsyncDelivery   bool
    DeliveryTimeout       time.Duration
    EnableDeduplication   bool
    DeduplicationWindow   time.Duration
}