SLOzy Architecture

High-level architecture diagrams
Service communication flows (4 key flows)
Database schema (15+ tables with indexes)
Deployment options: Docker Compose, Kubernetes, VM
Security architecture (OAuth, RBAC, encryption)
Scaling strategy (single-tenant → multi-tenant path)
Monitoring and operational requirements

Version: 1.0-alpha
Date: 2025-06-09
Status: Draft
Target Audience: Technical stakeholders, DevOps engineers, SRE leads

💡 See also: Getting Started · Deployment · Features · API Reference · Русская версия

High-Level Architecture
Component Diagram
Service Architecture
Data Architecture
Deployment Architecture
Technology Stack
Integration Points
Security Architecture
Scalability Strategy
Operational Requirements

High-Level Architecture

Architecture Philosophy

SLOzy Commercial follows a stateless application architecture with externalized state (PostgreSQL + Redis). This enables horizontal scaling, easier updates, and separation of concerns.

Key Architectural Principles:

Service-Oriented: Separate API service, worker service, and monitoring stack
Stateless Application Layer: No local state; all state in database/cache
Event-Driven: Ingestion worker triggered by cron, not direct API calls
Multi-Process for Resilience: Compute-intensive tasks (metric ingestion) run in separate process
Future-Proof: Architecture supports both single-tenant (MVP) and multi-tenant (future) deployments

Architectural Decision Records (ADRs)

ADR-001: Use PostgreSQL Instead of Only Files

Status: Decided
Context: Current SLOzy uses file storage; commercial product needs querying, versioning, search
Decision: PostgreSQL for primary storage, with Git export for declarative config
Rationale: File storage can't support: versioning, audit logs, complex queries, multi-user conflicts. PostgreSQL provides ACID compliance and rich query capabilities while Git provides version control for declarative configuration.

ADR-002: Separate API and Ingestion Services

Status: Decided
Context: API handles user requests; ingestion worker queries Prometheus continuously
Decision: Separate binaries/deployments: slozy-web (API), slozy-ingestor (worker)
Rationale: API requires low latency (<200ms); ingestion involves slow Prometheus queries (15-30s). Separating allows scaling independently: scale API horizontally, keep worker as 1-2 instances.

ADR-003: Polling for Real-Time Updates (Instead of WebSockets)

Status: Decided (MVP only)
Context: Need real-time dashboard updates for burn rates
Decision: Poll every 30 seconds from frontend
Rationale: WebSockets increase complexity (connection management, scaling, load balancing). Yes, reduces server load. For MVP, acceptable 30-second latency. Defer WebSocket to P1.

ADR-004: OAuth 2.0 for Authentication (No Passwords)

Status: Defer SAML to P1
Context: Need authentication for multiple users in an organization
Decision: OAuth 2.0 with GitHub and Google providers
Rationale: Industry standard, well-scoped permissions, no password management. Defer SAML/LDAP to post-MVP (too complex for single developer, 3-month timeline).

Component Diagram

System Architecture Overview (MVP - Single-Tenant)

┌─────────────────────────────────────────────────────────────────────────┐
│                          SLOzy Deployment                               │
│                    (self-hosted at customer premise)                    │
└─────────────────────────────────────────────────────────────────────────┘

                    ┌───────────────────────┐
                    │   External Traffic    │
                    │   (Users, Integrations)│
                    └───────────┬───────────┘
                                │ HTTPS (Port 443)
                                │
                ┌───────────────▼────────────────┐
                │           Nginx                │  │
                │   (SSL Termination,           │  │
                │    Reverse Proxy, Rate Limit)  │  │
                └───────────────┬────────────────┘  │
                                │ Port 8080           │
                                │                     │
┌────────────────────────────────────────────────────┐
│                 Application Layer                   │
│                                                     │
│  ┌──────────────┐         ┌───────────────────┐  │
│  │ slozy-web    │         │  slozy-ingestor   │  │
│  │ (API Server) │         │  (Worker Service)  │  │
│  │              │         │                   │  │
│  │ - SLO CRUD   │         │ - Prometheus      │  │
│  │ - Auth       │         │   Queries         │  │
│  │ - GitOps     │         │ - Burn Rate Comp.  │  │
│  │ - Dashboard  │         │ - Metrics Store   │  │
│  │  Endpoint    │         │                   │  │
│  └──────┬───────┘         └─────────┬─────────┘  │
│         │                          │             │
└─────────┼──────────────────────────┼─────────────┘
          │                          │
          │                          │ SQL + TCP
          ▼                          ▼
┌─────────────────┐      ┌─────────────────┐
│   PostgreSQL    │      │   (Optional)    │
│   (Primary DB)  │      │   PostgreSQL     │
│                 │      │   Read Replica  │
│   - SLOs        │      │                 │
│   - Users       │      │   - UI Queries  │
│   - Teams       │      │   - Reports     │
│   - Metrics     │      │                 │
│   - Audit Log   │      │                 │
└────────┬────────┘      └─────────────────┘
         │
         │ TCP (6379)
         ▼
┌─────────────────┐
│     Redis       │
│  (Cache Layer)  │
│                 │
│  - SLO Status   │
│  - Sessions     │
│  - Prom Queries │
└─────────────────┘


┌────────────────────────────────────────────────────┐
│              External Dependencies                  │
│                                                     │
│  ┌────────────┐   ┌─────────────┐   ┌──────────┐ │
│  │  GitHub    │   │  Grafana    │   │   Alert  │ │
│  │ Integration│   │ (Generated  │   │ Manager  │ │
│  │            │   │  Dashboards)│   │          │ │
│  └────────────┘   └─────────────┘   └──────────┘ │
│                                                     │
│  ┌──────────────────────┐   ┌────────────────────┐ │
│  │   Prometheus        │   │   CloudWatch       │ │
│  │   (Metrics Source)  │   │   (Optional)       │ │
│  │                     │   │   (AWS Native)     │ │
│  └──────────────────────┘   └────────────────────┘ │
└────────────────────────────────────────────────────┘

Component Responsibilities

External Layer

Nginx (Reverse Proxy)

SSL/TLS termination
Request routing to application
Rate limiting (10 req/s per endpoint)
Static file serving (if needed)
Request/response logging

Application Layer

slozy-web (API Server)

RESTful API for SLO management
Authentication (OAuth 2.0)
Authorization (RBAC)
SLO CRUD operations
GitOps integration (GitHub push)
Dashboard endpoints (for frontend)
Prometheus metrics export (/metrics endpoint)

slozy-ingestor (Worker Service)

Scheduled by cron (every 1 minute)
Queries Prometheus API for each SLO
Computes burn rates and error budgets
Stores metrics in PostgreSQL
Updates Redis cache
Triggers alerts on violations

Data Layer

PostgreSQL (Primary)

SLO metadata (service name, target, metric query)
Users, teams, RBAC tables
SLO versions and audit log
Metrics time-series (with partitioning)

PostgreSQL (Read Replica - Optional)

Scalable UI queries (not critical for MVP)
Reduces load on primary database
1 replica recommended for customers with > 50 SLOs

Redis (Cache)

SLO status cache (burn rate, error budget, compliance status)
Prometheus query cache (to avoid rate limits)
Session storage (can be in-memory stored in application state for MVP)
Temporary data (e.g., webhook payloads)

External Services

GitHub (GitOps Integration)

Repository hosting for SLO files
Webhook triggers for validation
Git history for version tracking

Prometheus (Metrics Source)

Time-series database for monitoring metrics
Query interface via HTTP API
One or more instances per customer

Grafana (Optional)

Dashboard visualization
Exports generated JSON dashboards
Queries Prometheus directly (bypasses SLOzy API in some cases)

Service Architecture

Service Communication Flow

Flow 1: User Creates New SLO

mermaid

sequenceDiagram
    participant User
    participant Nginx
    participant WebAPI as slozy-web (API)
    participant PG as PostgreSQL
    participant Redis as Redis Cache
    participant GitHub as GitHub API

    User->>Nginx: POST /api/slos (SLO data)
    Nginx->>WebAPI: Forward request
    WebAPI->>PG: Check team permissions (RBAC)
    PG-->>WebAPI: Permission granted
    WebAPI->>PG: Create SLO record
    PG-->>WebAPI: SLO ID created
    WebAPI->>PG: Create initial version (type 'create')
    PG-->>WebAPI: Version stored
    WebAPI->>PG: Log to audit table
    PG-->>WebAPI: Audit log created
    WebAPI->>GitHub: Generate OpenSLO YAML file
    GitHub-->>WebAPI: File created/committed
    WebAPI->>Redis: Invalidate SLO status cache (if any)
    Redis-->>WebAPI: Cache invalidated
    WebAPI-->>Nginx: Return SLO data with ID
    Nginx-->>User: Return SLO creation success

Flow 2: Metrics Ingestion (Cron Job)

mermaid

sequenceDiagram
    participant Cron as Cron Scheduler
    participant Worker as slozy-ingestor
    participant PG as PostgreSQL
    participant Prom as Prometheus API
    participant Redis as Redis Cache

    Cron->>Worker: Trigger update every 1 minute
    Worker->>PG: Query all active SLOs
    PG-->>Worker: Return SLO list (50 items)
    
    loop For each SLO (parallel execution)
        Worker->>PG: Get Prometheus data source for SLO
        PG-->>Worker: Return Prometheus URL
        
        Worker->>Prom: Query request count (last 1 hour)
        Prom-->>Worker: Return 10000 requests
        Worker->>Prom: Query error count (last 1 hour)
        Prom-->>Worker: Return 50 errors
        
        Worker->>Worker: Compute burn rate: 50 / (10000 * 0.001) = 5x
        Worker->>Worker: Compute error budget: 100% - 99.9% - (50/10000)% = 0.4%
        Worker->>Worker: Determine status: warning (burn_rate > 2x)
        
        Worker->>PG: Store metrics (slo_id, timestamp, burn_rate, etc.)
        PG-->>Worker: Metrics stored
        Worker->>Redis: Update cache: status:slo:123 = {burn_rate: 5x, status: warning}
        Redis-->>Worker: Cache updated
    end
    
    Worker-->>Cron: Update completed (took 45 seconds)

Flow 3: User Views SLO Dashboard (Real-Time)

mermaid

sequenceDiagram
    participant User
    participant Frontend as React Frontend
    participant WebAPI as slozy-web (API)
    participant Redis as Redis Cache
    participant PG as PostgreSQL

    User->>Frontend: Open SLO dashboard page
    Frontend->>WebAPI: GET /api/slos?team_id=1
    WebAPI->>PG: Query SLOs for team
    PG-->>WebAPI: Return 25 SLOs
    WebAPI-->>Frontend: Return SLO list with IDs
    
    loop Every 30 seconds
        Frontend->>WebAPI: GET /api/slos/123/status
        WebAPI->>Redis: Check cache: status:slo:123
        Redis-->>WebAPI: Found (cache hit): {burn_rate: 5.2x, status: warning}
        WebAPI-->>Frontend: Return SLO status
        Frontend->>User: Update dashboard (burn rate gauge)
    end
    
    Frontend->>WebAPI: GET /api/slos/45



    WebAPI->>Redis: Check cache: status:slo:45
    Redis-->>WebAPI: Cache miss (expired or not found)
    WebAPI->>PG: Query latest metrics from slo_metrics table
    PG-->>WebAPI: Return metrics (last 5 minutes)
    WebAPI->>Redis: Populate cache: status:slo:45
    WebAPI-->>Frontend: Return SLO status
    Frontend->>User: Update dashboard (burn rate gauge)

Flow 4: GitHub Webhook (Pull Request Validation)

mermaid

sequenceDiagram
    participant GitHub as GitHub.com
    participant Webhook as slozy-web Webhook
    participant PG as PostgreSQL
    participant GitHub API as GitHub API

    GitHub->>Webhook: POST /api/webhooks/github (pull_request event)
    Webhook->>GitHub: Verify webhook signature
    GitHub-->>Webhook: Signature valid
    Webhook->>Webhook: Parse PR diff (changed files)
    Webhook->>GitHub API: Extract modified SLO YAML files
    GitHub API-->>Webhook: SLO file content

    loop For each changed SLO file
        Webhook->>Webhook: Parse OpenSLO YAML
        Webhook->>Webhook: Validate against schema
        Webhook->>PG: Check for duplicate SLO names
        PG-->>Webhook: No duplicates found

        Webhook->>GitHub API: Comment on PR with validation results
        GitHub API-->>Webhook: Comment posted
    end

    Webhook->>GitHub API: Check if merging is blocked (failed validation)
    GitHub API-->>Webhook: Merge allowed (validation passed)
    Webhook-->>GitHub: ACK (successful webhook processing)

API Endpoint Structure

RESTful API Design

Authentication & Authorization:

POST   /auth/github                    - Sign in with GitHub
GET    /auth/github/callback          - OAuth callback
POST   /auth/google                   - Sign in with Google
GET    /auth/google/callback          - OAuth callback
GET    /api/me                        - Get current user info
GET    /api/me/teams                  - Get user's teams

SLO Management:

POST   /api/slos                      - Create new SLO
GET    /api/slos?team_id=1&status=ok  - List SLOs (with filters)
GET    /api/slos/:id                  - Get SLO details
GET    /api/slos/:id/status           - Get SLO status (burn rate, error budget)
PUT    /api/slos/:id                  - Update SLO
DELETE /api/slos/:id                  - Delete SLO

SLO Versioning:

GET    /api/slos/:id/versions          - List all versions
GET    /api/slos/:id/versions/:vid    - Get specific version details
POST   /api/slos/:id/versions/:vid/restore  - Restore from version
GET    /api/slos/:id/versions/:vid/diff    - Get diff between versions

Metrics & Monitoring:

GET    /api/slos/:id/metrics          - Get metrics time series
GET    /api/slos/:id/anomalies        - Get anomaly detections
GET    /api/metrics/summary           - Aggregate metrics across org

Teams & RBAC:

POST   /api/teams                     - Create team
GET    /api/teams                     - List teams
GET    /api/teams/:id                 - Get team details
PUT    /api/teams/:id                 - Update team
DELETE /api/teams/:id                 - Delete team
POST   /api/teams/:id/members         - Add member to team
DELETE /api/teams/:id/members/:uid   - Remove member from team

Integrations:

POST   /api/integrations/github       - Connect GitHub repo
GET    /api/integrations/github       - Get GitHub integration status
PUT    /api/integrations/github       - Update GitHub integration
DELETE /api/integrations/github       - Remove GitHub integration
GET    /api/integrations/github/status - Check webhook status

POST   /api/integrations/prometheus    - Add Prometheus data source
GET    /api/integrations/prometheus    - List Prometheus data sources
PUT    /api/integrations/prometheus/:id - Update Prometheus data source
DELETE /api/integrations/prometheus/:id - Remove Prometheus data source

Audit & Admin:

GET    /api/audit-logs                - List audit logs (with filters)
GET    /api/audit-logs/:id            - Get specific audit log entry
GET    /api/admin/metrics             - System metrics (admin only)

Operational:

GET    /health                        - Health check dependency status
GET    /ready                         - Readiness check
GET    /metrics                       - Prometheus metrics (application stats)

Webhooks:

POST   /api/webhooks/github           - GitHub webhook handler
POST   /api/webhooks/gitlab           - GitLab webhook handler (future)

Data Architecture

Database Schema (Core Tables)

Users & Teams

organizations (level 1)
└── teams (level 2)
    └── users (level 2 - can belong to multiple teams)
    └── team_memberships (level 3 - user-team join table)

Relationships:
- One organization can have multiple teams
- One user belongs to one organization (for now MVP)
- One user can belong to multiple teams
- One team can have multiple users

Key Indexes:

organizations.slug (unique)
teams.organization_id + teams.slug (unique composite)
users.email (unique)
team_memberships.team_id + team_memberships.user_id (unique composite)

SLOs

slos (level 2)
├── team_id (FK)
├── service_name
├── slo_name
├── indicator_type (latency, availability, throughput)
├── metric_source (prometheus, cloudwatch)
├── metric_query (PromQL or CloudWatch query)
├── target (0.9999 = 99.99%)
├── budgeting_period (28d, 7d, etc.)
└── enabled (boolean)

Relationships:
- SLOs belong to teams (via team_id)
- SLOs have one or more versions (foreign key)
- SLOs generate metrics (stored in slo_metrics table)

Key Indexes:

slos.team_id
slos.service_name
slos.indicator_type
slos.team_id + slos.service_name (unique composite)
slos.slo_name (full-text search index with pg_trgm)

SLO Versions & Audit Logs

slo_versions (level 3)
├── slo_id (FK to slos)
├── parent_slo_id (self-referencing FK for restore)
├── diff_type (create, update, delete)
├── changes (JSONB with diff)
├── snapshot (JSONB with full SLO state)
└── created_by (FK to users)

slo_audit_log (level 3)
├── slo_id (FK to slos)
├── user_id (FK to users)
├── team_id (FK to teams, nullable)
├── action (create_slo, update_slo, delete_slo, restore_slo)
├── changes (JSONB)
├── ip_address
└── user_agent

Relationships:
- SLOs have multiple versions (one-to-many)
- SLOs have multiple audit log entries (one-to-many)
- Audit log tracks all user-level modifications
- Versions track system-level changes (auto-generated)

Key Indexes:

slo_versions.slo_id + slo_versions.created_at (composite)
slo_audit_log.slo_id + slo_audit_log.created_at (composite)
slo_audit_log.user_id + slo_audit_log.created_at (composite)

Time-Series Metrics

slo_metrics (level 3 - heavily indexed table)
├── slo_id (FK to slos)
├── timestamp (TIMESTAMP)
├── time_window (VARCHAR(10), e.g., "1h", "5m")
├── metric_type (request_count, error_count, burn_rate, error_budget, latency_p99)
├── metric_value (NUMERIC(20, 6))
├── metadata (JSONB)
└── created_at (TIMESTAMP)

Partitioning strategy:
- Monthly partitions: slo_metrics_2025_06, slo_metrics_2025_07, etc.
- Partitions created automatically by background job
- Retention: Keep high-resolution (1-minute) for 30 days
- Downsampling: After 30 days, roll up to 5-minute resolution, archive after 90 days

Relationships:
- SLOs have metrics time-series (one-to-many)
- Metrics heavily indexed for time-series queries

Key Indexes:

slo_metrics.slo_id + slo_metrics.timestamp (composite)
slo_metrics.timestamp (for partition pruning)
slo_metrics.metric_type

Integration Tables

prometheus_data_sources (level 2)
├── organization_id (FK to organizations, MVP default 1)
├── name
├── url
├── is_default
├── authentication_type (none, basic, api_token)
├── credentials_encrypted
└── created_by (FK to users)

github_integrations (level 2)
├── organization_id (FK)
├── github_token_encrypted
├── repository (e.g., "owner/repo")
├── branch (default 'main')
├── file_path_pattern
└── enabled (boolean)

Relationships:
- Organizations can have multiple Prometheus data sources
- Organizations can have one GitHub integration (for MVP)
- SLOs reference Prometheus data sources
- GitHub integration used for GitOps

Key Indexes:

prometheus_data_sources.organization_id
prometheus_data_sources.name (unique per org)
github_integrations.organization_id (unique per org)

Data Flow Patterns

Pattern 1: Write-Through for SLO Creation

1. User POSTs to /api/slos with SLO data
2. API validates SLO data
3. API checks user RBAC permissions (can create in this team)
4. API creates SLO record in PostgreSQL (INSERT INTO slos)
5. API creates initial version record (INSERT INTO slo_versions, diff_type='create')
6. API creates audit log entry (INSERT INTO slo_audit_log, action='create_slo')
7. API generates OpenSLO YAML file
8. API pushes YAML to GitHub via GitHub API
9. API invalidates Redis cache (DELETE status:slo:{id})
10. API returns success with SLO ID to user

Pattern 2: Read-Through with Cache Miss for SLO Status

1. User requests GET /api/slos/:id/status
2. API checks Redis cache: GET status:slo:{id}
3. IF cache exists:
    - API returns cached data to user (fast path)
4. ELSE (cache miss):
    - API queries PostgreSQL for latest metrics (SELECT * FROM slo_metrics WHERE slo_id = {id} ORDER BY timestamp DESC LIMIT 60)
    - API aggregates metrics: compute averages, detect anomalies
    - API puts data into Redis cache: SET status:slo:{id} {data} EX 60
    - API returns data to user (slow path)

Pattern 3: Write-Behind for Metrics Ingestion

1. Cron triggers ingestion worker every 1 minute
2. Worker queries active SLOs from PostgreSQL
3. Worker queries Prometheus API for each SLO (parallel)
4. Worker computes burn rates, error budgets offline (no external DB needed)
5. Worker inserts computed metrics into PostgreSQL (INSERT or UPDATE if exists)
6. Worker updates Redis cache (possible parallel update)
7. Worker checks for anomalies: compare current burn rate to 7-day baseline
8. IF anomalies detected:
    - Worker sends alerts (email, Slack)
    - Worker stores anomaly detection in slo_audit_log

Pattern 4: Event-Driven for GitOps Push

1. User updates SLO (PUT /api/slos/:id)
2. API updates SLO record in PostgreSQL
3. API updates version record (new slo_versions entry, diff_type='update')
4. API logs to audit table
5. API triggers background job (or inline) to push to GitHub
6. GitHub job generates updated YAML file
7. GitHub job commits to GitHub API with new version
8. GitHub job invalidates SLO status cache (since target or query may have changed)

Deployment Architecture

Self-Hosted Deployment Options

Option A: Docker Compose (Recommended for MVP)

Architecture: Single-node deployment

customer-host:
├── docker-compose.yml (orchestrates containers)
│   ├── slozy-web (API server)
│   │   ├── Port: 8080 (internal)
│   │   └── Health check: /health
│   ├── slozy-ingestor (worker)
│   │   └── Cron schedule: */1 * * * *
│   ├── postgres
│   │   ├── Port: 5432
│   │   ├── Volume: postgres_data
│   │   └── As: internal network only
│   ├── redis
│   │   ├── Port: 6379
│   │   ├── Volume: redis_data
│   │   └── As: internal network only
│   └── nginx (reverse proxy)
│       ├── Port: 80, 443 (external)
│       └── SSL certificates (Let's Encrypt or customer-provided)

Configuration:

Environment variables for all services
POSTGRES_HOST=postgres (internal Docker network)
REDIS_HOST=redis
PROMETHEUS_URL (external customer Prometheus URL)
Secret management: .env file (or secrets for production)

Advantages:

Simple deployment
All containers on single host
Easy to upgrade (restart containers)
Good for small customers (1-5 teams)

Limitations:

Single point of failure (if host down)
No horizontal scaling
Limited resource isolation

Option B: Kubernetes (Recommended for Production/Scaled Customers)

Architecture: Highly available deployment

customer-k8s-cluster:
├── Namespaces:
│   ├── slozy-system (application)
│   └── slozy-monitoring (if deploying Prometheus + Grafana with SLOzy)
│
├── Deployments:
│   ├── slozy-web (replicas: 2, autoscaling: min 2, max 5)
│   │   ├── Health checks: /health (liveness), /ready (readiness)
│   │   ├── Horizontal Pod Autoscaler (scale on CPU/memory)
│   │   └── Pod Anti-Affinity (avoid same node)
│   └── slozy-ingestor (replicas: 1 - cron-based, no autoscaling)
│
├── Services + Ingress:
│   ├── slozy-web-service (ClusterIP, port 8080)
│   ├── slozy-ingestor-service (ClusterIP for metrics export)
│   └── slozy-ingress (Ingress resource, TLS termination)
│
├── StatefulSets:
│   ├── postgres (replicas: 1 primary, optional 1 replica)
│   │   ├── Persistent Volume Claims (database storage)
│   │   └── Read replicas for UI queries (if >50 SLOs)
│   └── redis (replicas: 1)
│       ├── Persistent Volume Claims (cache storage)
│       └── Redis Sentinel for high availability (optional)
│
├── ConfigMaps:
│   ├── slozy-config (application configuration)
│   ├── slozy-ingestor-config (worker configuration)
│   └── postgres-config (PostgreSQL config)
│
├── Secrets:
│   ├── slozy-secrets (database password, OAuth secrets)
│   ├── slozy-prometheus-secrets (Prometheus auth for data sources)
│   └── slozy-github-secrets (GitHub personal access token)
│
└── NetworkPolicies:
    ├── slozy-postgres-policy (postgres-DB → slozy-web/ingestor)
    ├── slozy-redis-policy (redis-cache → slozy-web/ingestor)
    └── slozy-dns-policy (all pods → external Prometheus, GitHub)

Deployment Strategy:

Rolling updates (zero downtime)
Blue-green deployment (for risky changes)
Canary release (for testing with subset of traffic)
Rolling restart for ingestion worker (uninterrupted due to cron tolerance)

Advantages:

High availability
Horizontal scaling
Easier monitoring (use existing Kubernetes monitoring stack)
Day 2 operations (resource quotas, network policies, RBAC)

Limitations:

More operational complexity
Requires Kubernetes expertise
More infrastructure to manage

Option C: VM-Based Deployment (Traditional Infrastructure)

Architecture: Systemd-managed services

customer-vm:
├── systemd services:
│   ├── slozy-web.service (API server)
│   │   └── Restart: always
│   ├── slozy-ingestor.service (worker)
│   │   └── Restart: always
│   └── postgres.service (database)
│       └── Restart: always
│
├── nginx reverse proxy (systemd-managed)
│   └── Sites: /etc/nginx/sites-available/slozy
│
├── Filesystem:
│   ├── /opt/slozy/bin/ (binaries)
│   ├── /opt/slozy/data/ (PostgreSQL data directory)
│   ├── /opt/slozy/logs/ (application logs)
│   └── /var/log/nginx/ (nginx logs)
│
├── Authentication:
│   └── OAuth providers (GitHub, Google) - external
│
└── Monitoring:
    └── Prometheus + Grafana (manual setup or existing)

Deployment Process:

Download release artifacts (static binaries)
Extract to /opt/slozy/
Run ./scripts/setup.sh (creates user, directories, systemd services)
Run sudo systemctl start slozy-web
Run sudo systemctl start slozy-ingestor
Configure Nginx reverse proxy (copy deploy/nginx.conf)
Obtain SSL certificate (Let's Encrypt or customer-provided)
Configure OAuth app credentials in environment files
Start services

Advantages:

Familiar deployment (DevOps engineers comfortable with systemd)
Less complex than Kubernetes
Works with existing infrastructure monitoring (Nagios, Zabbix)

Limitations:

Higher operational overhead (manual service restarts, dependency management)
No built-in scaling
Requires more maintenance

Deployment Comparison Matrix

Aspect	Docker Compose	Kubernetes	VM
Ease of Deployment	★★★★☆ (easiest)	★★☆☆☆ (moderate)	★★★☆☆ (easy)
High Availability	★☆☆☆☆ (none)	★★★★☆ (excellent)	★★☆☆☆ (requires clustering)
Scalability	★☆☆☆☆ (vertical scaling only)	★★★★☆ (horizontal scaling)	★★☆☆☆ (limited)
Monitoring Integration	★★★☆☆ (expose metrics manually)	★★★★☆ (native integration)	★★★☆☆ (needs setup)
Operational Overhead	★☆☆☆☆ (minimal)	★★★☆☆ (moderate)	★★★☆☆ (high)
Resource Overhead	★★★☆☆ (Docker overhead only)	★★☆☆☆ (Kubernetes overhead)	★★★★☆ (minimal)
Day 2 Operations	★★☆☆☆ (manual)	★★★★☆ (rolling updates, rollbacks)	★★☆☆☆ (manual)
Target Customers	Small teams, testing, MVP	Production, large customers	Traditional organizations, legacy infra
Recommended for MVP	✅ Yes	❌ No (too complex)	✅ Yes (if customer prefers)

Resource Estimates

Docker Compose (Small Team - 5-10 SLOs)

Service	vCPU	Memory	Disk
slozy-web	1.0	2GB	10GB
slozy-ingestor	0.5	1GB	5GB
postgres	1.0	2GB	50GB (data) + 10GB (backups)
redis	0.5	1GB	5GB
nginx	0.5	512MB	1GB
Total	3.5 vCPU	6.5GB	81GB

Kubernetes (Medium Team - 20-50 SLOs)

Service	vCPU	Memory	Disk	replicas
slozy-web	2.0	4GB	10GB	2 (min) - 5 (max)
slozy-ingestor	1.0	2GB	5GB	1
postgres (primary)	2.0	8GB	100GB	1
postgres (read replica)	1.0	8GB	100GB	0-1 optional
redis	1.0	2GB	10GB	1
Total (max)	~7 vCPU	24GB	235GB

Notes:

CPU usage increases with SLO count (more queries to Prometheus)
Disk growth: ~1GB per month for 50 SLOs with 1-minute resolution
Redis memory: 100MB for 50 SLO status cache + overhead
PostgreSQL memory: 8GB to cache frequently queried metrics

Technology Stack

Backend Technologies

Component	Technology	Version	Rationale
Language	Go	1.21+	Performance-focused, compiles to single binary, strong typing reduces bugs
Web Framework	Chi or Gin	Latest	Lightweight, stdlib-compatible (Chi) OR feature-rich (Gin). Choose based on team preference
Database Driver	pgx	v4+	Excellent PostgreSQL driver with connection pooling
ORM	GORM or Raw SQL	Latest	GORM for rapid development OR raw SQL for performance-critical paths. Use raw SQL with `sqlc` if performance issues arise
Migrations	golang-migrate	Latest	Industry-standard, CLI and Go library, supports up/down migrations
OAuth 2.0	golang.org/x/oauth2	Latest	Official Go OAuth implementation, supports GitHub, Google
HTTP Client	net/http + retry	Stdlib + github.com/sethvargo/go-retry	Standard library with retry library for resilience
JSON/Webhook Parsing	gopkg.in/yaml.v3	v3+	Good YAML parser, used in existing codebase
Redis Client	go-redis	v9+	High-performance Redis client, supports clustering, Redis Sentinel
Cron Scheduling	robfig/cron	Latest	Cron job scheduling in Go, robust and well-tested
Logging	zerolog	Latest	Zero-allocation structured logging, fast JSON logging
Metrics Export	Prometheus client	Latest	Standard Prometheus client for Go, metrics exposed on /metrics
Configuration	Viper	Latest	Configuration management with env vars, files, flags
Secrets Encryption	AES-256-GCM	Stdlib	Encrypt sensitive data in database (credentials, tokens)
UUID	github.com/google/uuid	Latest	UUID generation for record IDs
Time Parsing	github.com/golang/protobuf/ptypes/timestamp	Stdlib	Timestamp parsing for metrics

Frontend Technologies

Component	Technology	Version	Rationale
Framework	React	18+ with TypeScript	Industry-standard React, TypeScript for type safety
Build Tool	Vite	Latest	Fast build, modern tooling, better than Create React App
UI Library	MUI (Material-UI)	v5+	Rich component library, enterprise-friendly, accessible
Chart Library	Recharts	Latest	D3-powered charts, React-friendly, excellent documentation
HTTP Client	Axios	Latest	Promise-based HTTP client with interceptors, retry logic is plugin-based
State Management	Zustand OR Zustand Redux Toolkit	Latest	Zustand (lightweight) OR Redux Toolkit (if complex state needed)
Routing	react-router-dom	Latest	Standard React
Routing	TanStack Query	Latest	Data fetching, caching, synchronization with server state
Forms	react-hook-form + Zod	Latest	Form validation with Zod schema, TypeScript integrated
Date Handling	date-fns	Latest	Immutable date utilities, time zone support
Testing	Vitest + React Testing Library	Latest	Fast unit testing, component testing
Mock API	MSW (Mock Service Worker)	Latest	API mocking in tests and dev mode
Deployment	Vercel OR Docker Build	Latest	Vercel for SaaS OR Docker for self-hosted

Database Technologies

Component	Technology	Version	Rationale
Primary Database	PostgreSQL	15+	Mature, reliable, excellent JSONB support, time-series functions (percentiles)
Partitioning	Native PostgreSQL Partitioning	15+	Native partitioning (no TimescaleDB dependency for MVP)
Cache	Redis	7+	Fast in-memory cache, excellent for status cache
Full-Text Search	pg_trgm Extension	Latest	Trigram-based fuzzy search, integrated into PostgreSQL
Backup	pg_dump + Cron	Latest	Simple backup strategy for MVP (dumps daily to S3)

Infrastructure Technologies

Component	Technology	Version	Rationale
Containerization	Docker	Latest + Docker Compose v2	Standard container format, Compose for orchestration
Orchestration	Kubernetes	1.25+ (optional)	For scalable, HA deployments (post-MVP)
Ingress/Load Balancer	Nginx OR Traefik	Latest OR 2+	Nginx (industry standard) OR Traefik (auto-discovery)
SSL/TLS	Let's Encrypt Certbot	Latest	Automated certificate management
Monitoring	Prometheus + Grafana	Latest	Industry-standard metrics stack
Logging	Loki (Grafana) OR Journalctl	Latest	Cloud-native logging OR systemd journal (traditional)
Secrets	Vault OR Environment Variables	Latest OR Simple	Kubernetes Secrets (for k8s) OR .env files (for Docker Compose)
CI/CD	GitHub Actions OR GitLab CI	Latest	Integrated with Git repository

Integration Points

External System Integrations

1. Prometheus Integration

Purpose: Query metrics for SLO monitoring

Integration Type: Pull-based (queries Prometheus HTTP API) — SLOzy is a read-only consumer

Connectivity: SLOzy must have network access to the metrics endpoint. Typical setups:

Scenario	URL in Data Source	Setup
Same Docker network	`http://prometheus:9090`	Internal Docker DNS
Another server in the network	`http://10.0.1.50:9090`	Direct HTTP
VictoriaMetrics	`http://victoria:8428`	Compatible API (`/api/v1/query`)
Thanos	`http://thanos:19194`	Query frontend
Behind reverse proxy	`https://prom.example.com`	Basic Auth / Bearer Token

SLOzy does not store raw metrics — only computed results (SLI, burn rate, error budget) in its own PostgreSQL. Prometheus/VictoriaMetrics remains the single source of truth.

API Endpoints Used:

/v1/query - Instant query (single timestamp)
/v1/query_range - Range query (time range)
/v1/label/<name>/values - Label values (for autocomplete)

Query Patterns:

# Request count
sum(increase(http_requests_total{service="user-api"}[5m]))

# Error count
sum(increase(http_errors_total{service="user-api"}[5m]))

# Error rate (as percentage)
(sum(increase(http_errors_total{service="user-api"}[5m])) / 
 sum(increase(http_requests_total{service="user-api"}[5m]))) * 100

# Latency percentile
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="user-api"}[5m]))

# Availability ratio (success / total)
sum(increase(http_requests_total{service="user-api",status!~"5.."}[5m])) / 
sum(increase(http_requests_total{service="user-api"}[5m]))

Authentication:

None (if Prometheus public)
Basic Auth (username:password)
API Token (Bearer token)
OAuth 2.0 (rare, but supported by Prometheus)

Rate Limiting:

Prometheus default: 1 req/sec (global)
SLOzy Mitigation: Cache aggressively (30-60s TTL), batch queries

Data Retention:

Prometheus internal: 15 days default
SLOzy strategy: Query Prometheus for 7 days max, archive in PostgreSQL

Configuration in SLOzy:

yaml

prometheus_data_source:
  name: "production-prometheus"
  url: "https://prometheus.example.com"
  authentication_type: "api_token"
  credentials_encrypted: "<encrypted-token>"
  is_default: true

2. GitHub Integration (GitOps)

Purpose: Store/generated files, version tracking, PR validation

Integration Type: Push-based (GitHub API) + Webhooks (pull-based)

API Endpoints Used:

POST /repos/{owner}/{repo}/contents/{path} - Create/update file
POST /repos/{owner}/{repo}/git/commits - Create commit
POST /repos/{owner}/{repo}/git/tags - Tag (optional)
POST /repos/{owner}/{repo}/pulls/{pull_number}/comments - Comment on PR
GET /repos/{owner}/{repo}/pulls/{pull_number}/files - List changed files in PR
GET /repos/{owner}/{repo}/contents/{path} - Get file content

Workflow:

1. User creates/updates SLO in SLOzy
2. SLO generates OpenSLO YAML file
3. SLOzy creates commit in GitHub repo via API
4. GitHub triggers webhook to SLOzy (pull request event)
5. SLOzy validates SLO schema, syntax, checks duplicate names
6. SLOzy comments on PR with validation status
7. PR merges (if validation passes)
8. If merge fails, show error in SLOzy UI, retry manually

File Pattern:

slozy/{service_name}/{slo_name}.yaml

Example:
slozy/user-api/api-latency.yaml
slozy/payment-service/checkout-availability.yaml

Webhook Payload (copied from GitHub):

json

{
  "action": "opened",
  "number": 123,
  "pull_request": {
    "html_url": "https://github.com/example/repo/pull/123",
    "changed_files": ["slozy/user-api/api-latency.yaml"]
  },
  "repository": {
    "name": "repo",
    "full_name": "example/repo"
  }
}

Configuration in SLOzy:

yaml

github_integration:
  repository: "example/repo"
  branch: "main"
  file_path_pattern: "slozy/{service}/{slo}.yaml"
  github_token_encrypted: "<encrypted-token>"
  enabled: true

3. Grafana Integration

Purpose: Export dashboards for monitoring

Integration Type: Export-only (generate JSON dashboards, import manually or via API)

API Endpoints Used:

POST /api/dashboards/db - Import dashboard (automate via GitHub Actions or manual)

Dashboard Templates:

SLO Details Dashboard: Burn rate, error budget, compliance status
Multi-SLO Dashboard: View all SLOs for service or team
Anomaly Detection Dashboard: Show anomalies for SLOs

Dashboard JSON Generation:

json

{
  "dashboard": {
    "title": "User API Latency SLO",
    "uid": "slozy-user-api-latency-123",
    "panels": [
      {
        "title": "Burn Rate (Last 24h)",
        "targets": [
          {
            "expr": "slozy_burn_rate{slo_id=\"123\"}"
          }
        ]
      },
      {
        "title": "Error Budget Remaining",
        "targets": [
          {
            "expr": "slozy_error_budget{slo_id=\"123\"}"
          }
        ]
      }
    ]
  }
}

Export to Grafana:

Automatic: GitHub Action triggers Grafana import
Manual: User downloads JSON and imports in Grafana UI

4. Email/Slack Integration (Alerting)

Purpose: Send alerts on SLO violations, anomalies

Integration Type: Push-based (SMTP for email, Webhook for Slack)

SMTP Configuration:

yaml

email_integration:
  smtp_host: "smtp.gmail.com"
  smtp_port: 587
  smtp_from: "slozy@example.com"
  smtp_username: "slozy@example.com"
  smtp_password_encrypted: "<encrypted-password>"

Slack Configuration:

yaml

slack_integration:
  webhook_url: "https://hooks.slack.com/services/..."
  default_channel: "#slozy-alerts"
  severity_mapping:
    warning: "#slozy-warnings"
    critical: "#slozy-critical"

Alert Template:

text

ALARM: High Burn Rate Detected

Service: User API
SLO: API Latency (p99 < 200ms)
Current Burn Rate: 5.2x (threshold: 2x)
Error Budget Remaining: 20%
Time Window: Last 1 hour

Anomaly Detected: Burn rate increased significantly!
Recommended Action: Investigate degradations in user-api latency.

View in SLOzy: https://slozy.example.com/slos/123

Security Architecture

Authentication & Authorization

OAuth 2.0 Flow

┌─────────────┐
│   User      │
└──────┬──────┘
       │ 1. User clicks "Sign in with GitHub"
       │
       ▼
┌──────────────────────┐
│  slozy-web (API)     │
│                      │
│  - Redirect URL:     │
│    https://github.com│
│    /login/oauth/     │
│    authorize?        │
│    client_id=...     │
│    scope=user:email  │
│    state=<random_string>│
│    redirect_uri=     │
│      https://slozy...│
│      /auth/github/   │
│      callback         │
└──────────┬───────────┘
           │ 2. User authorizes in GitHub
           │
           ▼
┌──────────────────────┐
│   GitHub OAuth       │
│   (Authorization     │
│    Endpoint)         │
│                      │
│  - User authorizes   │
│  - GitHub generates  │
│    temporary code     │
│  - Redirects back to:│
│    callback URL +    │
│    code + state      │
└──────────┬───────────┘
           │ 3. GitHub redirects with code
           │
           ▼
┌──────────────────────┐
│  slozy-web (API)     │
│                      │
│  - Verifies state    │
│  - Exchanges code for│
│    access token via: │
│    POST https://github│
│    .com/login/oauth/ │
│    access_token      │
│  - Fetches user info │
│    via:              │
│    GET https://github│
│    .com/api/v3/user  │
│  - Creates user in   │
│    DB if new user    │
│  - Creates session   │
│    token (JWT)       │
│  - Redirects to      │
│    dashboard         │
└──────────┬───────────┘
           │ 4. User redirected with session
           │
           ▼
┌─────────────┐
│   Browser   │
└─────────────┘

RBAC Implementation

Roles:

Role	Permissions
Admin	- Manage organization settings - Create/delete teams - Manage all SLOs - Add/remove team members - View all organizational metrics - Manage integrations (GitHub, Prometheus)
Editor	- Create SLOs in assigned teams - Edit SLOs in assigned teams - Delete own SLOs in assigned teams - View team metrics - Add/edit SLO versions - View audit logs for own SLOs
Viewer	- View all SLOs in assigned teams - View metrics and dashboards - View SLOs, versions - No creation, editing, deletion rights

Permission Matrix:

Resource	Admin	Editor	Viewer
View SLOs	✅ All	✅ Team	✅ Team
Create SLO	✅ Any team	✅ Team	❌
Edit SLO	✅ Any	✅ Team & owned	❌
Delete SLO	✅ Any	✅ Team & owned	❌
Restore SLO	✅ Any	✅ Team & owned	❌
View Versions	✅ All	✅ Team	✅ Team
View Audit Logs	✅ All	✅ Team & owned	✅ Team & owned
Manage Teams	✅	❌	❌
Add Team Members	✅	❌	❌
Remove Team Members	✅	❌	❌
Manage Integrations	✅	❌	❌
View Org Metrics	✅	❌	❌

Implementation:

Middleware:

1. Extract user from JWT (from Authorization header or session cookie2. Load user and team memberships from database
3. Check user role (Admin/Editor/Viewer)
4. For SLO operations: Verify user's team membership
5. Return 403 Forbidden if insufficient permissions

Examples:

// Editor can edit SLO if team_id in user.team_ids
if user.Role != "admin" && slo.TeamID not in user.TeamIDs {
  return http.Error("Insufficient permissions", 403)
}

// Viewer can only read
if user.Role == "viewer" && request.Method != "GET" {
  return http.Error("Read-only access", 403)
}

Data Security

Encryption at Rest

During MVP (Simplification):

Database credentials: Stored in environment variables (encrypted at OS level)
OAuth tokens: Stored in plain text in database (access token has 1-hour expiry)
SLO secrets (e.g., API keys for Prometheus queries): By client, not stored in SLOzy

Post-MVP Enhancement:

Use PostgreSQL TDE (Transparent Data Encryption) if supported
Encrypt OAuth refresh tokens using AES-256-GCM

Encrypt Prometheus data source credentials before storing:

func EncryptCredentials(plaintext string, key []byte) (string, error) {
  block, err := aes.NewCipher(key)
  if err != nil {
    return "", err
  }
  
  gcm, err := cipher.NewGCM(block)
  if err != nil {
    return "", err
  }
  
  nonce := make([]byte, gcm.NonceSize())
  if _, err = rand.Read(nonce); err != nil {
    return "", err
  }
  
  ciphertext := gcm.Seal(nonce, nonce, []byte(plaintext), nil)
  return base64.StdEncoding.EncodeToString(ciphertext), nil
}

Store encryption key in environment variable or Vault (post-MVP)

Encryption in Transit

TLS Enforcement:

All external traffic: HTTPS (TLS 1.2+)
Internal traffic (Docker network): May use HTTP but not recommended (use TLS in production)
Database connections: TLS if PostgreSQL supports (configure in connection string: sslmode=require)
Prometheus API: HTTPS if available, otherwise HTTP with basic auth
GitHub API: Always HTTPS

Nginx Configuration:

nginx

server {
  listen 443 ssl http2;
  ssl_certificate /etc/nginx/ssl/slozy.crt;
  ssl_certificate_key /etc/nginx/ssl/slozy.key;
  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!aNULL:!MD5;
  
  # Redirect HTTP to HTTPS
  if ($scheme != "https") {
    return 301 https://$host$request_uri;
  }
}

Keys and Secrets Management

MVP (Simplest):

Environment variables: .env file (Docker Compose) or Kubernetes Secrets
Secrets: Database password, OAuth app secrets, Prometheus basic auth tokens
Commit .env.example to Git (without actual secrets), add .env to .gitignore

Production (Recommended):

Kubernetes Secrets (with RBAC: restrict read access)
AWS Secrets Manager (if deploying to AWS)
HashiCorp Vault (for enterprise customers)

Input Validation & Sanitization

SQL Injection Prevention:

Use prepared statements (Go SQL library handles this)
Never build SQL queries with string concatenation

Example:

// ❌ DON'T DO THIS
query := fmt.Sprintf("SELECT * FROM slos WHERE service_name = '%s'", userInput)

// ✅ DO THIS
query := "SELECT * FROM slos WHERE service_name = $1"
db.Query(query, userInput)

XSS Prevention:

Sanitize user input before rendering in HTML (especially SLO names, descriptions)
Use html/template auto-escaping (Go templates escape by default)

Set Content Security Policy (CSP) header:

w.Header().Set("Content-Security-Policy", "default-src 'self'")

CSRF Protection:

For state-changing operations (POST, PUT, DELETE), require anti-CSRF token for web requests
For API clients, require authentication (OAuth token) which already prevents CSRF

File Upload Security:

Currently: No file uploads (SLO created via API form)
Future: If supporting custom templates, validate file type, scan for malware before storing

Network Security

Network Segmentation

Docker Network:

slozy-network: Internal network for services (postgres, redis, slozy-web, slozy-ingestor)
Only nginx has external access from internet
All internal services communicate via slozy-network only

Kubernetes Network Policies:

yaml

# Allow slozy-web to access postgres
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: slozy-web-to-postgres
spec:
  podSelector:
    matchLabels:
      app: slozy-web
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432

# Allow slozy-web to access redis
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: slozy-web-to-redis
spec:
  podSelector:
    matchLabels:
      app: slozy-web
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: redis
    ports:
    - protocol: TCP
      port: 6379

Firewall Rules (VM Deployment)

UFW Configuration:

bash

# Allow SSH
ufw allow 22/tcp

# Allow HTTP and HTTPS
ufw allow 80/tcp
ufw allow 443/tcp

# Allow all outgoing traffic
ufw default allow outgoing

# Deny all incoming except specified ports
ufw default deny incoming

# Enable firewall
ufw enable

PostgreSQL Security:

Listen only on localhost (bind to 127.0.0.1)
Don't expose to internet

Use pg_hba.conf for host-based access control:

host    all             all             0.0.0.0/0            trust   # DISALLOW (example)
host    all             slozy_user      172.17.0.0/16         trust   # ALLOW Docker network

Rate Limiting

Nginx Rate Limiting:

nginx

# Define rate limit zones (per IP)
limit_req_zone $binary_remote_addr zone=slozy_api:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=slozy_create:10m rate=5r/s;

# Apply to API endpoints
location /api/ {
  limit_req zone=slozy_api burst=20 nodelay;
  proxy_pass http://slozy-web;
}

location /api/slos {
  limit_req zone=slozy_create burst=10 nodelay;
  client_max_body_size 1M;
  proxy_pass http://slozy-web;
}

Application-Level Rate Limiting:

Redis-based rate limiting for global rate limits (persists across API restarts)

In-memory rate limiting for simpler implementation in Go:

rateLimiter := NewRateLimiter(5, time.Minute) // 5 requests per minute
if !rateLimiter.Allow(user_id) {
  return http.Error("Rate limit exceeded", 429)
}

Audit Logging

What to Audit:

User login/logout
SLO create/update/delete
Team create/update/delete
User add/remove from team
Integration changes
Configuration changes

Audit Log Entry Schema:

type AuditLogEntry struct {
  ID        int
  SLOID     int      // nil if not related to SLO
  UserID    int      // nil if system action
  TeamID    int      // nil if not related to team
  Action    string   // e.g., "create_slo", "delete_slo", "add_team_member"
  Changes   map[string]interface{}  // JSON diff
  IPAddress net.IP
  UserAgent string
  Timestamp time.Time
}

Audit Log API:

GET /api/audit-logs?user_id=1&team_id=2&start_date=2025-06-01&end_date=2025-06-30
Returns paginated audit log entries
Requires Admin role (user_id can filter by others)
Export to CSV: GET /api/audit-logs/export (Admin only)

Scalability Strategy

Single-Tenant Scalability (MVP)

Horizontal Scaling Strategy

Service Splitting:

Phase 1 (MVP): Single-host deployment
  ├─ slozy-web      (1 instance)
  ├─ slozy-ingestor (1 instance)
  ├─ postgres       (1 instance)
  └─ redis          (1 instance)

Phase 2 (Post-MVP): Horizontal scaling
  ├─ slozy-web      (2+ instances, load balanced)
  ├─ slozy-ingestor (1-2 instances)
  ├─ postgres       (1 primary + 1 replica)
  └─ redis          (1 instance, Redis Sentinel for HA)

Why No Horizontal Scaling for Ingestor:

Ingestor is stateless and idempotent (can run multiple instances)
But requires synchronization to avoid duplicate Prometheus queries
Simpler to run single instance with cron (every 1 minute)
If scaling needed, use distributed lock (Redis) to coordinate queries

Why Multiple Web Instances:

API is stateless (sessions in Redis)
Can scale horizontally with Nginx load balancing
Horizontal Pod Autoscaler (K8s) or manual scaling (Docker)
Good for UI-heavy usage (many concurrent dashboard viewers)

Database Scaling Strategy

Phase 1 (MVP): Single PostgreSQL Instance

Single primary database
Connection pooling (PgBouncer or internal Go pool)
Partitioning for metrics table (monthly partitions)

Phase 2 (Post-MVP): Read Replicas

Primary database for writes (ingestor, SLO CRUD)
Read replica for reads (UI queries, analytics)
pgBouncer to route queries automatically to replicas
1-3 replicas depending on read-heavy usage

Phase 3 (Large Scale): Sharding or Clustering

If customer has > 500 SLOs, consider sharding by service or team
Alternative: Use TimescaleDB (PostgreSQL extension) for better time-series performance
Alternative: Move to专用 time series database like ClickHouse for only metrics

Cache Scaling Strategy

Phase 1 (MVP): Single Redis Instance

Single Redis cache for SLO status, sessions, Prometheus queries
Redis memory: 2GB for 50 SLOs (100MB for data + overhead)

Phase 2 (Post-MVP): Redis Sentinel

1 master + 2 sentinels for high availability
Automatic failover if master fails
Clients connect to sentinel cluster

Phase 3 (Large Scale): Redis Cluster

Redis Cluster (multiple shards) for memory > 64GB
Horizontal scaling for cache reads/writes
Client library (go-redis) supports Redis Cluster

Multi-Tenant Scalability (Future - Post MVP)

Tenant Isolation:

Database-Level Isolation (Preferred for Security):

-- Option 1: Separate database per tenant (organization)
CREATE DATABASE slozy_org_1;
CREATE DATABASE slozy_org_2;
-- Pros: Complete isolation, easier deletion of tenant data
-- Cons: Higher operational complexity, hard cross-tenant queries

-- Option 2: Row-level security (single database, tenant_id column)
CREATE TABLE slos (
  id SERIAL,
  tenant_id INT NOT NULL,
  service_name VARCHAR(255),
  --  ... other columns
  CONSTRAINT fk_tenant FOREIGN KEY (tenant_id) REFERENCES organizations(id)
);
ALTER TABLE slos ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation_policy ON slos
  USING (tenant_id = current_user_tenant());
-- Pros: Single database, cross-tenant queries possible
-- Cons: Risk of cross-tenant data leakage if RLS misconfigured

Application-Level Isolation (Add service_id to all queries):

// Middleware extracts tenant from subdomain or header
tenantID := getTenantFromDomain("tenant1.slozy.net") or getTenantFromHeader()

// All queries include tenant_id filter
SELECT * FROM slos WHERE tenant_id = 123 AND ...

Resource Quotas:

Per-Tenant Quotas (Enforcement):

Plan Tiers:
- Free: 5 SLOs, 1 team, 3 users
- Standard: 50 SLOs, 5 teams, 10 users
- Pro: Unlimited SLOs, 15 teams, unlimited users

Implementation:
- Check quota before creating SLO: count by tenant_id
- Middleware to intercept Create operations
- Return error if quota exceeded

Billing Integration:

Usage Tracking:

SLO Usage Query:
SELECT COUNT(ID) as slo_count, tenant_id 
FROM slos 
WHERE created_at >= '2025-06-01' 
  AND tenant_id = 123 
GROUP BY tenant_id

User Count Query:
SELECT COUNT(DISTINCT user_id) as user_count
FROM team_memberships 
JOIN teams ON teams.id = team_memberships.team_id 
WHERE teams.organization_id = 123

Billing Integration:

Monthly cron job or webhook to billing provider (Stripe)
Usage report: SLO count, user count, active integrations
Generate invoice based on plan tier
Overages charged per SLO or per user

Operational Requirements

Monitoring Stack

Metrics to Collect

Application Metrics (slozy-web, slozy-ingestor):

HTTP request rate per endpoint: slozy_http_requests_total{method, route, status_code}
HTTP request latency (histogram): slozy_http_request_duration_seconds{method, route}
Database query latency: slozy_db_query_duration_seconds{query_type}
Redis operation latency: slozy_redis_operation_duration_seconds{operation}
Prometheus query latency: slozy_prometheus_query_duration_seconds
Cache hit ratio: slozy_cache_hit_ratio{cache_type}
Active users: slozy_active_users{tenant_id}
Concurrent SLOs: slozy_concurrent_slos_count

Business Metrics (Aggregated from database):

SLOs created today: slozy_slos_created_total{date=2025-06-09}
SLOs active today: slozy_slos_active_total{date=2025-06-09}
Total burn rate violations: slozy_burn_rate_violations_total{severity}
Average error budget across org: slozy_avg_error_budget{tenant_id}
GitOps sync failures: slozy_github_sync_failures_total

System Metrics (Infrastructure):

CPU utilization: cpu_usage_percent{core=0,1,2,3}
Memory utilization: memory_usage_bytes{type=used,free,buffers,cache}
Disk I/O: disk_io_bytes{device=sda,operation=read,write}
Network traffic: network_bytes{interface=eth0,direction=in,out}
Filesystem usage: filesystem_usage_bytes{mount=/opt/slozy/data,type=used,free}
Open file descriptors: process_open_fds

Alerting Rules

Critical Alerts (PagerDuty/Email):

# Service Down (if slozy-web is down)
ALERT SlozyWebDown
IF up{job="slozy-web"} == 0
FOR 5m
LABELS { severity="critical" }
ANNOTATIONS { summary="slozy-web service is down" }

# Database Query Latency Too High
ALERT PostgreSQLSlowQueries
IF rate(pg_stat_statements_calls_total[5m]) > 0 AND
   pg_stat_statements_mean_exec_time_ms > 1000
FOR 5m
LABELS { severity="critical" }
ANNOTATIONS { summary="PostgreSQL queries taking >1 second on average" }

# Prometheus API Error Rate High
ALERT PrometheusAPIErrorRate
IF rate(slozy_prometheus_query_errors_total[5m]) / rate(slozy_prometheus_query_total[5m]) > 0.05
FOR 5m
LABELS { severity="critical" }
ANNOTATIONS { summary="Prometheus API error rate > 5% for 5 minutes" }

Warning Alerts (Email/Slack):

# High CPU Usage
ALERT HighCPUUsage
IF cpu_usage_percent > 80
FOR 5m
LABELS { severity="warning" }
ANNOTATIONS { summary="CPU usage high" }

# Disk Space Low
ALERT LowDiskSpace
IF filesystem_usage_bytes{type=used} / filesystem_usage_bytes > 0.85
FOR 5m
LABELS { severity="warning" }
ANNOTATIONS { summary="Disk space low" }

# SLO Ingestion Not Running
IF up{job="slozy-ingestor"} == 0
FOR 10m
LABELS { severity="warning" }
ANNOTATIONS { summary="slozy-ingestor stopped, metrics not updating" }

Dashboards

System Health Dashboard:

CPU, Memory, Disk usage over time
HTTP request rate and latency
Database query latency
Redis metrics (cache hit ratio, connections)
Prometheus metrics (queries/sec, errors)

Business Metrics Dashboard:

SLOs created/updated daily
Active SLOs trend
GitOps sync status
Burn rate violations summary
Error budget health across organization

SLO Details Dashboard (per SLO):

Burn rate over time (line chart)
Error budget remaining (gauge)
Prominent metrics (request count, error count, latency p99)
Anomalies detected (list)
Implementation trace (versions history)

Logging Strategy

Structured Logging

Log Format:

json

{
  "timestamp": "2025-06-09T10:30:45Z",
  "level": "info",
  "service": "slozy-web",
  "correlation_id": "550e8400-e29b-41d4-a716-446655440000",
  "user_id": 123,
  "tenant_id": 1,
  "action": "slo.create",
  "slo_id": 456,
  "duration_ms": 245,
  "prometheus_query_duration_ms": 120,
  "prometheus_query_cache_hit": false
}

Log Levels:

debug: Detailed diagnostic info (prometheus queries, DB queries)
info: Normal operations (SLO created, user logged in)
warn: Warning conditions (Prometheus query cached data, database slow query)
error: Errors but service continues (Prometheus API error, DB connection lost)
fatal: Fatal errors (application cannot continue, database unreachable)

Logging Configuration:

Development: debug level, console output
Staging: info level, file output
Production: warn level, file output, send errors to Sentry or similar

Centralized Logging Options

Option 1: Loki + Grafana (Recommended)

  slozy-web --(stdout)--> Loki (log aggregator) --(queried)--> Grafana UI
  slozy-ingestor --(stdout)/

Simple to set up
Works with Kubernetes
Logs stored in object storage (S3)
Query with LogQL

Option 2: Oracle Cloud Logging (Traditional)

  slozy-web --(journalctl)--> systemd journal --> systemd-journal remote --> Elastic/Opensearch --> Kibana UI

Works with systemd-managed services
More complex to set up
More powerful querying capabilities

Option 3: journalctl only (Simplest)

  slozy-web/slozy-ingestor logs → systemd journal → journalctl to view

Simpler for small deployments
No centralized storage (kept on VM/instance)
Query with journalctl -u slozy-web -f

Backup & Disaster Recovery

Database Backup Strategy

Full Daily Backups (pg_dump):

bash

# Daily cron job: 2am
pg_dump -h localhost -U slozy_user -d slozy_db | gzip > /backups/slozy_db_$(date +%Y%m%d).sql.gz

Incremental Backups (PITR - Point-In-Time Recovery):

Configure PostgreSQL WAL archiving
Archive WAL files to S3 or backup storage
Allow recovery to any point in time (within WAL retention period)

Retention Policy:

Daily backups: Keep 30 days
Weekly backups: Keep 12 weeks (every weekly backup for 3 months)
Monthly backups: Keep 12 months (every monthly backup for 1 year)
WAL files: Keep only last 30 days

Redis Backup Strategy

RDB Snapshots:

Redis creates .rdb snapshots every X seconds if at least Y writes occurred
Default: 900 seconds (15 minutes) if at least 1 key changed
Backup .rdb files to persistent storage
Retention: Keep 7 daily backups

Git-Based Backup (for SLO files):

SLO files are stored in GitHub repository
GitHub already has history and redundancy
No additional backup needed for generated files
Backup GitHub repository to local Git mirror (weekly)

Disaster Recovery Plan

Scenario 1: PostgreSQL Instance Failure

Restore from latest full backup + WAL logs (for point-in-time recovery)
Recovery Time Objective (RTO): 2-4 hours
Recovery Point Objective (RPO): < 1 hour (manual triggering)

Scenario 2: Complete Data Center Loss

Restore from offsite backup (S3 or similar)
Re-provision infrastructure (using Terraform or similar)
Restore database from backup
RTO: 4-8 hours (depends on infrastructure provisioning)
RPO: < 1 day (last backup)

Scenario 3: Application Corruption (Bad Update)

Rollback application to previous version (Docker image tag)
If database migration corrupts data: restore from point-in-time recovery (before migration)
RTO: 30 minutes (application rollback) to 4 hours (database restore)
RPO: < 5 minutes (if rolling back after user reports)

Maintenance Windows

Scheduled Maintenance:

Database Maintenance (Monthly):

sql

-- Vacuum to reclaim space and update statistics
VACUUM VERBOSE ANALYZE;

-- Reindex to rebuild indexes
REINDEX TABLE slo_metrics;

-- Archive old metrics (older than 90 days)
CREATE TABLE slo_metrics_archive_2025_03 AS
SELECT * FROM slo_metrics WHERE timestamp >= '2025-03-01' AND timestamp < '2025-04-01';
DROP TABLE slo_metrics_archive_2025_03; -- backed up to archive storage

Application Updates (Bi-weekly):

Zero-downtime updates (rolling updates in Kubernetes)
Load balancer drains connections from old version
New version receives all new connections
Old version shuts down after 30s grace period

Security Updates (As Needed):

Docker OS package updates (weekly)
PostgreSQL security patches (ASAP)
Go dependencies updates (monthly)

Downtime Communication:

Maintenance Mode Page:

When maintenance scheduled:
Nginx shows maintenance page at https://slozy.example.com
Message: "SLOzy is under scheduled maintenance from 2:00 UTC to 3:00 UTC.
SLO monitoring continues, UI and API temporarily unavailable."

Email Notifications (24 hours and 1 hour before):

Subject: [UPCOMING] Scheduled Maintenance - SLOzy Deployment

SLOzy deployment scheduled maintenance:
- Start: 2025-06-10 02:00 UTC
- Duration: ~1 hour

Impact:
- UI and API will be unavailable during maintenance
- Existing SLO monitoring will continue (ingestor still running)
- No data loss expected

Questions? Contact support@slozy.net

Document Change Log

Version	Date	Author	Description
1.0-alpha	2025-06-09	SLOzy Team	Initial architecture draft for MVP

Document Status: Draft
Next Steps: Review with technical stakeholders, refine deployment architecture, finalize technology stack choices
Planned Updates: Add multi-tenant design (post-MVP), advanced monitoring setup, disaster recovery runbooks

Next Steps

Getting Started — set up SLOzy for the first time
Deployment Guide — deploy to Docker, Kubernetes, or VM
Features Overview — explore SLO management, notifications, caching
API Reference — full API documentation
Русская версия — документация на русском

SLOzy Architecture ​

Table of Contents ​

High-Level Architecture ​

Architecture Philosophy ​

Architectural Decision Records (ADRs) ​

Component Diagram ​

System Architecture Overview (MVP - Single-Tenant) ​

Component Responsibilities ​

External Layer ​

Application Layer ​

Data Layer ​

External Services ​

Service Architecture ​

Service Communication Flow ​

Flow 1: User Creates New SLO ​

Flow 2: Metrics Ingestion (Cron Job) ​

Flow 3: User Views SLO Dashboard (Real-Time) ​

Flow 4: GitHub Webhook (Pull Request Validation) ​

API Endpoint Structure ​

RESTful API Design ​

Data Architecture ​

Database Schema (Core Tables) ​

Users & Teams ​

SLOs ​

SLO Versions & Audit Logs ​

Time-Series Metrics ​

Integration Tables ​

Data Flow Patterns ​

Pattern 1: Write-Through for SLO Creation ​

Pattern 2: Read-Through with Cache Miss for SLO Status ​

Pattern 3: Write-Behind for Metrics Ingestion ​

Pattern 4: Event-Driven for GitOps Push ​

Deployment Architecture ​

Self-Hosted Deployment Options ​

Option A: Docker Compose (Recommended for MVP) ​

Option B: Kubernetes (Recommended for Production/Scaled Customers) ​

Option C: VM-Based Deployment (Traditional Infrastructure) ​

Deployment Comparison Matrix ​

Resource Estimates ​

Docker Compose (Small Team - 5-10 SLOs) ​

Kubernetes (Medium Team - 20-50 SLOs) ​

Technology Stack ​

Backend Technologies ​

Frontend Technologies ​

Database Technologies ​

Infrastructure Technologies ​

Integration Points ​

External System Integrations ​

1. Prometheus Integration ​

2. GitHub Integration (GitOps) ​

3. Grafana Integration ​

4. Email/Slack Integration (Alerting) ​

Security Architecture ​

Authentication & Authorization ​

OAuth 2.0 Flow ​

RBAC Implementation ​

Data Security ​

Encryption at Rest ​

Encryption in Transit ​

Keys and Secrets Management ​

Input Validation & Sanitization ​

Network Security ​

Network Segmentation ​

Firewall Rules (VM Deployment) ​

Rate Limiting ​

Audit Logging ​

Scalability Strategy ​

Single-Tenant Scalability (MVP) ​

Horizontal Scaling Strategy ​

Database Scaling Strategy ​

Cache Scaling Strategy ​

Multi-Tenant Scalability (Future - Post MVP) ​

Tenant Isolation: ​

Resource Quotas: ​

Billing Integration: ​

Operational Requirements ​

Monitoring Stack ​

Metrics to Collect ​

Alerting Rules ​

Dashboards ​

SLOzy Architecture

Table of Contents

High-Level Architecture

Architecture Philosophy

Architectural Decision Records (ADRs)

Component Diagram

System Architecture Overview (MVP - Single-Tenant)

Component Responsibilities

External Layer

Application Layer

Data Layer

External Services

Service Architecture

Service Communication Flow

Flow 1: User Creates New SLO

Flow 2: Metrics Ingestion (Cron Job)

Flow 3: User Views SLO Dashboard (Real-Time)

Flow 4: GitHub Webhook (Pull Request Validation)

API Endpoint Structure

RESTful API Design

Data Architecture

Database Schema (Core Tables)

Users & Teams

SLOs

SLO Versions & Audit Logs

Time-Series Metrics

Integration Tables

Data Flow Patterns

Pattern 1: Write-Through for SLO Creation

Pattern 2: Read-Through with Cache Miss for SLO Status

Pattern 3: Write-Behind for Metrics Ingestion

Pattern 4: Event-Driven for GitOps Push

Deployment Architecture

Self-Hosted Deployment Options

Option A: Docker Compose (Recommended for MVP)

Option B: Kubernetes (Recommended for Production/Scaled Customers)

Option C: VM-Based Deployment (Traditional Infrastructure)

Deployment Comparison Matrix

Resource Estimates

Docker Compose (Small Team - 5-10 SLOs)

Kubernetes (Medium Team - 20-50 SLOs)

Technology Stack

Backend Technologies

Frontend Technologies

Database Technologies

Infrastructure Technologies

Integration Points

External System Integrations

1. Prometheus Integration

2. GitHub Integration (GitOps)

3. Grafana Integration

4. Email/Slack Integration (Alerting)

Security Architecture

Authentication & Authorization

OAuth 2.0 Flow

RBAC Implementation

Data Security

Encryption at Rest

Encryption in Transit

Keys and Secrets Management

Input Validation & Sanitization

Network Security

Network Segmentation

Firewall Rules (VM Deployment)

Rate Limiting

Audit Logging

Scalability Strategy

Single-Tenant Scalability (MVP)

Horizontal Scaling Strategy

Database Scaling Strategy

Cache Scaling Strategy

Multi-Tenant Scalability (Future - Post MVP)

Tenant Isolation:

Resource Quotas:

Billing Integration:

Operational Requirements

Monitoring Stack

Metrics to Collect

Alerting Rules

Dashboards