"Deployed" is not a health check

Disclosure: this article is written by the Appaloft team and describes our own implementation.

A deployment can finish cleanly while users still receive the wrong thing. The job can be green, the process can be alive, and a health endpoint can return HTTP 200 while an old container or stale proxy target continues serving traffic.

Recent public reports made the boundaries unusually clear. A Dokploy Swarm report from July 10 described a pinned image whose config-only deployment displayed “Deployed” without recreating the task. A July 12 Dokploy report described a healthy Compose container while Traefik selected a stale or wrong network address and returned 502. A July 11 Coolify report described a self-hosted production service reaching its restart limit and stopping. A July 15 Dokploy report described the platform’s own UI cycling through readiness states after an update while users saw Bad Gateway.

These are user reports, not independent root-cause audits, and they affect different products and layers. We did not treat them as comparison-advertising material. We used them as an audit checklist for Appaloft.

Four facts that “deployed” must prove

1. Artifact applied. The requested image, revision, or configuration reached a new workload generation.
2. Process live. The process stays running instead of entering a restart loop or stopping after a nominal start.
3. Service healthy. The current workload passes its internal health policy.
4. Route reachable and current. The real domain, TLS, and proxy path serve that exact deployment—not merely any healthy response.

The fourth fact cannot be inferred from a label on the container. A container can truthfully identify itself while the edge proxy still targets an older workload.

What our Appaloft audit found

The config-only Swarm class did not expose the same no-op path in Appaloft. Appaloft creates a deployment-specific candidate stack or service, stamps a new deployment identity and safe configuration fingerprint, verifies the candidate, then promotes it. Deployment Proof compares the observed generation and fingerprint with the accepted plan, so a healthy old task becomes stale, not verified.

Stopped and unhealthy workloads already produced failed health evidence, covering the workload-side symptom in the restart-limit report. The control-plane self-update loop is a separate platform-readiness boundary; workload Deployment Proof does not claim to solve it.

The route case did reveal a real Appaloft defect. Our Docker readback used the current container’s appaloft.deployment-id label as evidence that the route targeted that deployment. That was workload identity masquerading as route evidence. A stale Traefik or Caddy target could make the claim false.

We changed the contract:

• Appaloft-managed Caddy and Traefik serve routes stamp responses with X-Appaloft-Deployment-Id.
• deployments.proof probes the configured public health path through the real managed domain.
• A missing or different deployment identity fails access evidence, even when HTTP status is 200.
• Redirect-only aliases never claim workload identity.

The header contains an opaque deployment identifier, not configuration or secret data. Configuration evidence remains a SHA-256 fingerprint; raw values never enter the proof response.

One query, five verdicts

The query is available through the same operation catalog as the rest of Appaloft:

appaloft deployments proof <deploymentId>

The HTTP route is GET /api/deployments/{deploymentId}/proof. The generated SDK and MCP tool expose the same deployments.proof operation, and the Web console renders the same DTO on deployment detail pages.

The verdict deliberately avoids turning missing evidence into success:

• verified means the required artifact, workload, configuration, health, access, and recovery evidence agrees with the deployment intent.
• partially-verified means some evidence agrees, but a required readback is unavailable.
• unverified means there is not enough runtime evidence to make the claim.
• stale means the runtime is healthy but no longer represents this deployment.
• failed means observed evidence directly contradicts the intended result, such as failed health or an access route attached to the wrong workload.

This distinction matters. “We cannot read the artifact digest” and “we read a different digest” are different operational problems.

The tests include the uncomfortable cases

The existing real-Docker smoke deploys v1, changes APP_VERSION through the formal Resource configuration operation, redeploys v2, and then externally replaces the current container with a healthy v1 container carrying the old deployment and configuration labels. The proof reports the workload as stale even though health still passes.

We added a second smoke with a real Traefik container. Its route returns HTTP 200 and stamps dep_v1; the proof evaluates dep_v2. The result is deliberately not success:

{
  "httpStatus": 200,
  "expectedDeployment": "dep_v2",
  "observedDeployment": "dep_v1",
  "accessStatus": "failed",
  "reasonCode": "public_route_deployment_identity_mismatch"
}

Provider tests also assert that Caddy and Traefik add the response marker only to served routes, while adapter tests cover matching, missing, mismatched, unreachable, and non-2xx responses.

Where the evidence comes from

For local Docker and Docker Compose deployments, Appaloft finds the container by its Resource label and inspects its image identity, workload labels, start time, health, and safe configuration fingerprint. Docker Swarm uses the corresponding service identity and update generation. Managed public-route evidence now comes from the proxy response, independently of container inspection.

The application layer does not know Docker commands. It asks a runtime evidence port for normalized observations, then evaluates those observations against deployment state. This keeps the verdict logic testable and lets another runtime adapter provide equivalent evidence without changing the query contract.

Generic SSH deployments currently report an explicit evidence gap when no capable runtime readback adapter is present. Direct-port workloads do not claim a managed-proxy identity check. We prefer visible boundaries over inferring success from an execution record.

AI and GitHub checks use the same rule

The Appaloft skill now tells agents to read deployments.proof after observing deployment progress. An agent may describe a terminal state, but it may only claim success when the verdict is verified.

Cloud’s GitHub-check summary follows the same mapping. A green workflow is input evidence, not the final judgment. The control plane still has to reconcile what was requested with what is running.

That gives CLI users, the Web console, an MCP client, and an automated check one answer instead of four loosely related interpretations.

Read the deployment lifecycle documentation for the command and verdict model. For the provenance side of the problem, see what evidence a GitHub Actions deployment should leave behind.

The next adapters will have to earn verified with their own runtime evidence. That is the point of the verdict: unsupported readback remains visible instead of disappearing behind a green status.