Add managed registry provisioning, pruning, and readiness tracking

docs/managed-registry.md

@@ -1,5 +1,16 @@
+---
+status: in-progress
+implemented_slice: app-side-operation-generation-maintenance-readiness
+---
+
 # Managed Registry Plan
 
+## Implementation Status
+
+Keystone now has app-side managed registry provisioning, auth, smoke-check, and pruning operations. The generated remote scripts install and run `registry:2` with local storage, htpasswd auth, deletion enabled, Caddy HTTPS proxy snippets, Docker auth setup, push/pull smoke checks, manifest deletion, registry garbage collection, and artifact pruning status transitions.
+
+This document remains `in-progress` until the install flow runs these operations as part of first-run Keystone setup and the generated remote scripts have been validated against the supported production host layouts.
+
 Keystone should be self-hosted first. A fresh install should include a working build and image pipeline without requiring the user to bring an external Docker registry, S3 bucket, or separate build server.
 
 ## Product Principles
@@ -8,6 +19,7 @@ Keystone should be self-hosted first. A fresh install should include a working b
 - Keystone provides a first-party managed Docker registry by default.
 - The managed registry stores images on local disk first.
 - The registry storage path must be configurable for mounted VPS volumes.
+- Multi-server deployments using the managed registry require an HTTPS registry URL trusted by every build and runtime node.
 - External registries, S3-backed storage, and dedicated build nodes are optional advanced features.
 - Multi-server deployments should work out of the box after Keystone is installed.
 - Registry credentials must not be persisted in operation scripts, logs, or UI-visible output.
@@ -38,10 +50,11 @@ Default settings:
 ```text
 Build node: Keystone control node
 Registry: registry:2 managed by Keystone
+Registry URL: install-provided HTTPS hostname, for example registry.example.com
 Registry storage driver: local
 Registry storage path: /home/keystone/registry/data
 Image retention: latest 3 successful artifacts per environment
-Auth: generated htpasswd credentials managed by Keystone
+Auth: generated htpasswd build and runtime credentials managed by Keystone
 ```
 
 The install flow should allow overriding the storage path, for example:
@@ -52,6 +65,8 @@ The install flow should allow overriding the storage path, for example:
 
 This lets users place registry image data on a mounted VPS volume while keeping Keystone's default behavior simple.
 
+The install flow should also require a registry hostname for normal multi-server managed-registry use. Keystone should configure that hostname with HTTPS, usually through the control node's web proxy. A plain HTTP `host:5000` registry should only be available as an explicit local development or advanced fallback because it requires insecure-registry configuration on every Docker daemon that builds or pulls images.
+
 ## Default Image Flow
 
 ```text
@@ -78,25 +93,62 @@ If Keystone later supports HA control planes, the control node concept should be
 - The default build node.
 - Runtime nodes used for deployed applications.
 
-## Registry Exposure
+## Image References
 
-The managed registry should be exposed over HTTPS where possible, ideally behind the control node's web proxy, for example:
+Managed registry image names should be stable and collision-resistant. Use IDs in the repository path so renaming an application or environment does not move the image repository.
+
+Default tag format:
+
+```text
+registry.example.com/keystone/{application_uuid}/{environment_uuid}:{git_sha}
+```
+
+Deployment reference format:
+
+```text
+registry.example.com/keystone/{application_uuid}/{environment_uuid}@sha256:...
+```
+
+Each successful build artifact should store:
+
+- The registry host.
+- The full pushed tag.
+- The registry manifest digest.
+- The application and environment IDs.
+- The source commit SHA.
+
+Deployments should consume the stored digest reference. Tags are useful for humans and registry lookup, but deployments should not depend on mutable tags such as `latest`.
+
+## Registry URL And TLS
+
+The managed registry must be exposed over HTTPS for the normal multi-server path, ideally behind the control node's web proxy, for example:
 
 ```text
 registry.example.com
 ```
 
-Avoid defaulting to a plain `host:5000` registry if possible. Plain HTTP registries require Docker daemon insecure-registry configuration on every build and target server, which adds onboarding friction.
+The install flow should ask for the registry hostname and configure TLS before marking the managed registry ready. Target servers and build nodes must be able to resolve the hostname and trust the certificate before they can build, push, or deploy images.
+
+Avoid defaulting to a plain `host:5000` registry. Plain HTTP registries require Docker daemon insecure-registry configuration on every build and target server, which adds onboarding friction and weakens the default security posture. If Keystone supports this fallback, it should be clearly labelled as local development or advanced use.
 
 Target servers must be able to reach the registry URL before they can deploy images built by Keystone.
 
+Managed registry health checks should verify:
+
+- The registry service is running.
+- The registry URL is reachable over HTTPS from the control node.
+- The registry URL is reachable over HTTPS from the selected build node.
+- The registry URL is reachable over HTTPS from target runtime servers.
+- Build credentials can log in and push a small test manifest or image.
+- Runtime credentials can log in and pull the pushed test artifact.
+
 ## Authentication
 
 Use `registry:2` htpasswd authentication for the first version.
 
 Keystone should:
 
-- Generate registry credentials.
+- Generate separate build and runtime registry credentials.
 - Write the registry htpasswd file during provisioning.
 - Store credentials encrypted.
 - Configure build and target servers for registry access.
@@ -104,12 +156,22 @@ Keystone should:
 
 Do not inline registry passwords into persisted operation scripts. Operation steps are stored and may be visible in the UI or logs.
 
+The build node should receive build credentials. Runtime target servers should receive runtime credentials. With `registry:2` htpasswd authentication alone, these credentials are not truly push- or pull-scoped; any authenticated registry user can push and pull. The separation is still useful for rotation, auditing, and limiting which credential is distributed to each machine, but Keystone should not present runtime credentials as read-only until it adds token auth or another authorization layer.
+
+When Keystone configures Docker auth on a server, it should do so idempotently and with explicit ownership. For the default `keystone` user model, registry auth should live in that user's Docker config:
+
+```text
+/home/keystone/.docker/config.json
+```
+
+The file should be owned by `keystone:keystone` and readable only by that user where possible. If a root-owned Docker context is required for a specific operation, Keystone should write the equivalent root-owned config intentionally rather than relying on whichever user happened to run `docker login`.
+
 Preferred approaches:
 
 - Configure Docker auth on each server through a separate secure action.
 - Or write root-owned / user-owned credential files on the server and have deployment scripts read from those files.
 
-Token auth can be considered later if Keystone needs per-repository or per-server scoped credentials. It should not be part of the first implementation.
+Token auth can be considered later if Keystone needs per-repository, per-server, or true push/pull scoped credentials. It should not be part of the first implementation.
 
 ## Build Planning
 
@@ -146,13 +208,21 @@ The default build execution should:
 7. Push the image.
 8. Resolve and store the registry manifest digest.
 
+Control-node builds should have guardrails so the default path does not destabilize Keystone itself:
+
+- Limit concurrent builds on the control node, defaulting to one at a time.
+- Check available disk before cloning, building, and pushing.
+- Remove temporary clone/build directories after each build.
+- Prune local build images and intermediate layers separately from registry artifact retention.
+- Surface disk pressure as a build-node health problem before accepting more builds.
+
 Example flow:
 
 ```bash
-docker login registry.example.com --username keystone --password-stdin
-docker build --file Dockerfile.keystone --tag registry.example.com/application:aaaaaaaaaaaa .
-docker push registry.example.com/application:aaaaaaaaaaaa
-docker manifest inspect registry.example.com/application:aaaaaaaaaaaa
+docker login registry.example.com --username keystone-build --password-stdin
+docker build --file Dockerfile.keystone --tag registry.example.com/keystone/app-uuid/env-uuid:aaaaaaaaaaaa .
+docker push registry.example.com/keystone/app-uuid/env-uuid:aaaaaaaaaaaa
+docker manifest inspect registry.example.com/keystone/app-uuid/env-uuid:aaaaaaaaaaaa
 ```
 
 The stored digest must be the registry manifest digest, not a local image ID. Digest-based pulls and registry manifest deletion depend on this being correct.
@@ -173,7 +243,7 @@ Deploy execution should:
 Example pull reference:
 
 ```text
-registry.example.com/application@sha256:...
+registry.example.com/keystone/app-uuid/env-uuid@sha256:...
 ```
 
 Compose should use the full registry reference, not only `sha256:...`.
@@ -207,7 +277,7 @@ Pruning should remove old registry manifests first, then run registry garbage co
 REGISTRY_STORAGE_DELETE_ENABLED=true
 ```
 
-Garbage collection is safest when the registry is not accepting writes. The first implementation should run cleanup during a controlled maintenance window, using a lock so pruning does not race with active builds or pushes.
+Garbage collection is safest when the registry is not accepting writes. The first implementation should treat manifest deletion and blob garbage collection as separate steps: delete old manifests under the normal retention policy, then run blob garbage collection only during a controlled maintenance window, using a lock so pruning does not race with active builds or pushes.
 
 Suggested cleanup flow:
 
@@ -227,7 +297,7 @@ These should be optional settings, not onboarding requirements:
 - Dedicated build nodes.
 - S3-compatible registry storage.
 - External registries such as GHCR, Gitea, Docker Hub, or generic registries.
-- Separate push and pull credentials.
+- True push- and pull-scoped credentials.
 - Credential rotation.
 - Per-server or per-repository scoped auth.
 - Configurable retention per application or environment.