Skip to content
ArtificialFlow

Operations

Process health is not a functional check. Confirm identity, the connector, and a write that shows up in the query API before calling a deployment ready.

Health

  • /livez is process-only. Command /readyz checks Postgres and the migration ledger. Query /readyz checks Elasticsearch. /health remains a compatibility alias for readiness.
  • Sync /readyz stays 503 until topic and connector bootstrap finishes, and it checks search on every probe. It can also fail when consumption is stale and SYNC_HEALTH_FAIL_ON_STALE=true.
  • Gateway /readyz proxies the frontend check. Gateway /livez is local.

The command service also exposes compact counters for outbox relay activity, idempotency records, publish lag, and retry exhaustion. The sync worker reports freshness on its health endpoint.

A deployment is ready when

  1. The browser login succeeds.
  2. /api/identity/me returns the expected role.
  3. A workflow can be deployed and started.
  4. That instance appears through /api/v1/query/.

The Debezium connector and its tasks report RUNNING. The status URL still uses the connector name artificialflow-postgres-connector, which is the legacy wire identifier kept for rolling upgrades. Pin image tags or digests and review signatures, SBOM, and provenance before a production rollout. The release line pins 1.2.0 and v1.2.0. Those tags are not on Docker Hub yet. New installs pull artificialflow/*.

Backups

Back up Postgres, the search indexes or the data required to rebuild them, IAM configuration, secrets, Helm values, and rendered manifests. Test a restore.

Recovery

  • Rebuild search projections from source events or CDC topics when you can.
  • make init-connector is a manual recovery path for local Compose profiles, used when the connector is missing after the command service has initialized the database.
  • Keep the Debezium connector configuration versioned. Do not place a database password in a ConfigMap.

Replicas

The supported production shape is two or more gateway, frontend, command, and query pods. The sync worker scales with Kafka partitions. Replicas beyond kafka.topicPartitions (default 3) sit idle, and existing topics keep their partition count. The runtime default is one replica in compat scheduler mode. HA uses multi-active with three runtime replicas. Disruption budgets apply to components that run at least two replicas.

Command pod loss fails the in-flight request. Clients retry with Idempotency-Key. Sync worker loss grows projection lag while writes stay in Postgres. Search loss degrades the query API and leaves the command path up. Postgres loss stops writes and runtime ticks.

Upgrade

  1. Back up Postgres and record the image digests you can roll back to.
  2. Drain the outbox. artificialflow_outbox_pending should be near zero.
  3. Apply the Helm or Compose upgrade. Command, query, and frontend can roll. In compat mode, runtime ticks pause until a replica holds the lease again. Switch every runtime replica to multi-active together.
  4. Check health, an OIDC login when IAM is on, a deployed process, a started instance, a completed job, a user task, and that instance in the query API. Command and runtime migrate before they are ready. A failed migration leaves the old pods serving.

Roll back by redeploying the previous image digests. Restore Postgres only when the schema change cannot run on the old image, and rebuild search afterward. Admins read outbox and idempotency counters at GET /internal/metrics on the command service (Compose port 8080). The public gateway answers 404 for that path. The runtime process exposes its private health listener on port 9091.

Published latency figures in the product repository were measured on a single-node Compose stack at v0.3.0, not on a production cluster and not on this 1.2.0 line. Re-measure with make test-perf before using them for capacity. The aspirational targets are outbox age and CQRS lag under 15–30 seconds at p99, and runtime failover inside the lease TTL plus one tick.

Production checklist

  • TLS on public endpoints, and network policy around any internal plaintext route.
  • Example hostnames and Compose credentials replaced.
  • Audience validation left on, with short-lived SDK tokens.
  • Secrets in a secret manager, including SDK profiles. Those profiles never belong in a browser.
  • Resource requests, limits, and a tested backup.
  • Alerts on the gateway, command, runtime, query, sync, IAM, database, bus, connector, and search.

Schema changes need care: command and runtime apply the embedded migrations at startup under a Postgres advisory lock. There is no migration Job. Pair this list with Troubleshooting when an install does not become ready.