System Architecture

DSO is a layered system designed for reliability and safety. Each layer has a specific responsibility. Failures are handled gracefully at every stage.

Secrets Provider

Source of truth for secrets

VaultAWS Secrets ManagerAzure Key VaultHuawei Cloud KMSLocal Vault

DSO Agent

Watches for changes, orchestrates rotation

Change detectionSchedulerState machineProvider API client

Checkpoint Manager

Persistence layer for crash recovery

On-disk state trackingRollback safetyConsistency guarantees

Validation Layer

Ensures safety before swap

Health checksConnection verificationReadiness validation

Runtime Engine

Executes container updates

Atomic swapCleanupFailure handling

Application

Consumes rotated secrets

Zero downtimeNo restart requiredAutomatic reconnection

Architectural Principles

  • Layered design: Each layer handles one responsibility. Failures in one layer don't cascade.
  • Safety before speed: Checkpoint-based recovery preferred over fast recovery. Consistency over performance.
  • Expected failures: Every layer assumes downstream failures. Automatic recovery is built in.
  • State persistence: Checkpoints allow recovery from any failure point. State is always recoverable.

System Architecture

How DSO orchestrates secret rotation from provider to container

Secret Provider

Vault, AWS Secrets Manager, Azure Key Vault, Huawei Cloud, or Local encrypted storage

DSO Agent/Watcher

Detects secret changes and spawns rotation container with health validation

Secure Storage (tmpfs)

Secrets loaded into encrypted memory, never written to disk

Application Container

Receives secrets via environment variables or mounted files

Health Checks & Rollback

Validates rotation success; automatically reverts on failure

Zero Disk Persistence

Secrets exist only in encrypted memory (tmpfs). Cleared on container exit.

Atomic Rotation

Secrets updated atomically. Application sees no partial state.

Automatic Rollback

If rotation fails, previous state is automatically restored.

Provider Agnostic

Same process works with any secret provider. No code changes needed.

Request Lifecycle

Core rotation operation from secret change detection to completion. Each step is atomic and checkpointed for recovery.

Detect

Provider change detected via watcher or poll

Lock

Acquire distributed lock to prevent concurrent rotations

Checkpoint

Save rotation state to persistent storage

Spawn

Launch container with new secrets from provider

Health Check

Validate container readiness and health signals

Swap

Atomically switch container to new secret version

Cleanup

Remove old container and release lock

Guarantees

  • Atomic: Rotation succeeds completely or reverts
  • Recoverable: Checkpoint allows resume after crash
  • Zero-downtime: Old container runs until swap
  • Isolated: Secrets never touch disk during rotation

Failure Handling

  • Health check fails: Rollback to old container
  • Agent crash: Checkpoint allows safe resume
  • Provider unavailable: Retry with exponential backoff
  • Swap fails: Previous version remains running

System Scope

DSO Manages

  • Secret rotation from provider
  • Container lifecycle during rotation
  • Atomic swap and rollback on failure

DSO Does Not Manage

  • Kubernetes or container orchestration
  • RBAC or identity management
  • Multi-cluster or high-availability