harry/keystone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement Keystone environment deployments

Browse Source
This commit is contained in:
Harry Bayliss
2026-05-13 16:11:23 +01:00
parent 65d3142d03
commit aa680b25fd
175 changed files with 10258 additions and 740 deletions
Download Patch File Download Diff File Expand all files Collapse all files

726
docs/implementation-spec.md Normal file
View File

@@ -0,0 +1,726 @@
# Keystone Implementation Spec
## 1. Product Scope
Keystone is a Laravel Forge-like deployment platform that runs applications and services with Docker. The v1 product is intentionally narrow:
- Laravel is the only first-class application framework.
- Application containers use a Keystone-managed Dockerfile based on `serversideup/php` with FrankenPHP.
- Services are explicitly coded drivers, not arbitrary Docker images.
- v1 is agentless and executes operations over SSH.
- Docker Compose is used as the generated runtime artifact.
- Caddy 2 is the default and only gateway for v1.
- The Keystone database is the source of truth. Server files are generated artifacts.
V1 should make the simple path robust before adding generic Docker support, distributed agents, HA databases, edge routing, or additional frameworks.
## 2. Core Domain Model
### Organisation
Owns users, providers, registries, applications, servers, services, and environments.
### Application
A source-code project. In v1, first-class applications are Laravel repositories.
Recommended fields:
- `organisation_id`
- `name`
- `repository_url`
- `repository_type`
- `default_branch`
- `deploy_key_public`
- `deploy_key_private` encrypted
- `deploy_key_fingerprint`
- `deploy_key_installed_at` nullable
### Environment
The primary application deployment unit. An application has environments such as production, staging, or dev.
Recommended fields:
- `application_id`
- `name`
- `branch`
- `status`
- `scheduler_enabled`
- `scheduler_target_service_id` nullable
- `scheduler_mode`: `single` or `every_replica`
- `build_config` json
Default for Laravel environments:
- Scheduler enabled.
- Scheduler target is the primary web service.
- Scheduler mode is `single`.
### Service
Every deployable thing is represented as a `Service`.
Examples:
- Laravel web runtime
- Laravel worker runtime
- Laravel websocket runtime
- Caddy gateway
- Postgres
- Valkey
- Future standalone services
Recommended fields:
- `organisation_id`
- `environment_id` nullable
- `server_id` nullable for single-placement legacy convenience only; long term use replicas
- `name`
- `category`
- `type`
- `version_track`
- `driver_name`
- `status`
- `desired_replicas`
- `desired_revision`
- `deploy_policy`
- `process_roles` json
- `current_image_digest` nullable
- `available_image_digest` nullable
- `update_status`
- `default_cpu_limit` nullable
- `default_memory_limit_mb` nullable
- `config` json
Deploy policy defaults:
- Laravel web: `with_environment`
- Laravel worker: `with_environment`
- Laravel websocket: `with_environment`
- Database/cache/storage: `dependency_only`
- Gateway: `manual_or_on_route_change`
- Standalone services: `manual`
The user should not need to configure these defaults during normal setup.
### ServiceReplica
A running instance of a service on a server. A service is logical; a replica is runtime placement.
Recommended fields:
- `service_id`
- `server_id`
- `operation_id` nullable
- `container_name`
- `container_id` nullable
- `image_digest`
- `internal_host`
- `internal_port`
- `public_port` nullable
- `status`
- `health_status`
- `cpu_limit` nullable
- `memory_limit_mb` nullable
- `config` json
Replica resource limits override service defaults. Null means unrestricted except host capacity.
### ServiceSlice
A logical sub-resource inside a service. Slices belong to `Service`, not `ServiceReplica`.
Examples:
- Database and user inside Postgres
- Logical database or namespace inside Valkey
- Route inside Caddy
- Future bucket, topic, vhost, etc.
Recommended fields:
- `service_id`
- `environment_id` nullable
- `name`
- `type`
- `status`
- `config` json
- `credentials` encrypted json nullable
Slices are not containers and should not be used for scaling. They are stable logical resources that survive service replica replacement.
### EnvironmentAttachment
Connects an environment to managed service slices.
Recommended fields:
- `environment_id`
- `service_id`
- `service_slice_id` nullable
- `role`: `database`, `cache`, `queue`, `storage`, `gateway`, `custom`
- `env_prefix` nullable
- `is_primary`
Attachments should point to slices whenever a slice exists. For example, a Laravel environment attaches to a Postgres database/user slice, not merely to the Postgres service.
### EnvironmentVariable
Represents user-defined and Keystone-managed runtime environment values.
Recommended fields:
- `environment_id`
- `key`
- `value` encrypted
- `source`: `user`, `managed_attachment`, `system`
- `service_slice_id` nullable
- `overridable` boolean
Managed values should be regenerated from attachments and slices.
## 3. Operations Model
Rename `Deployment` to `Operation`.
An operation is the generic audit and execution object for all state-changing work.
### Operation
Recommended fields:
- `id`
- `parent_id` nullable
- `hash`
- `kind`
- `target_type`
- `target_id`
- `status`
- `started_at`
- `finished_at`
- timestamps
Operation kinds:
- `server_provision`
- `service_deploy`
- `replica_deploy`
- `slice_provision`
- `slice_configure`
- `environment_deploy`
- `gateway_cutover`
- `config_change`
- `credential_rotation`
### OperationStep
Rename `Step` to `OperationStep`.
Recommended fields:
- `operation_id`
- `name`
- `order`
- `status`
- `script`
- `logs`
- `error_logs`
- `secrets` encrypted json nullable
- `started_at`
- `finished_at`
- timestamps
### Parent-Child Operations
Environment deploys are parent operations that create child operations.
Example:
- `environment_deploy`
- child `service_deploy` for web
- child `replica_deploy` for each web replica
- child `slice_configure` for Caddy route updates
- child `gateway_cutover`
Standalone service deploys and slice operations can also run independently.
## 4. Server Provisioning
V1 remains agentless over SSH.
Provisioning flow:
1. Create server through provider API.
2. Wait for root SSH to become available.
3. Execute provisioning script over SSH.
4. Create Keystone management user.
5. Install Docker Engine, Docker Compose plugin, UFW, fail2ban, and required runtime packages.
6. Install Keystone SSH public key.
7. Disable password login.
8. Enable UFW with SSH open.
9. Callback or SSH verification marks server active.
Server permanent keys are for Keystone management only. Repository deploy keys must not be permanently installed on servers.
## 5. Source Providers And Repository Access
V1 source support:
- Self-hosted Gitea
- GitHub
- Generic Git over SSH
Repository access uses a Keystone-generated deploy key per application/repository.
V1 flow:
1. User enters repo SSH URL.
2. Keystone generates an ed25519 deploy key.
3. UI shows the public key.
4. User adds it to Gitea/GitHub as read-only.
5. Keystone verifies access with `git ls-remote`.
During build operations, Keystone injects the encrypted private key into a temporary operation directory and uses `GIT_SSH_COMMAND`. The key is removed after the build. Repo keys are never permanently stored on target servers or builder services.
## 6. Registry And Build Artifacts
An external registry is required for multi-server application deployments.
Single-server deployments may build and run a local image without a registry.
Multi-server deployments must:
1. Build once.
2. Push the image to the configured external registry.
3. Pull the exact same image digest on each target server.
Supported registry types:
- Generic Docker registry
- Gitea registry
- GHCR
- Docker Hub
### Build Service
Building is a service capability, not a server type.
A dedicated builder is represented as a `Service` with category `builder`. If no builder service exists, Keystone may build on the target server for single-server deployments.
Build strategies:
- `target_server`: build on selected target server. Valid for single-server.
- `dedicated_builder`: build on builder service, then push/export artifact.
- `external_registry`: pull prebuilt image from registry.
For v1:
- Single-server default: build on target server.
- Multi-server: require configured registry and build once.
- Do not rebuild independently on each server.
### BuildArtifact
Recommended fields:
- `environment_id`
- `commit_sha`
- `image_tag`
- `image_digest`
- `registry_ref` nullable
- `built_by_operation_id`
- `built_by_service_id` nullable
- `status`
- `metadata` json
## 7. Managed Laravel Runtime
V1 uses Keystone-managed Dockerfile templates only. Custom Dockerfiles are deferred.
Laravel runtime defaults:
- Base: `serversideup/php` FrankenPHP image
- PHP version configurable
- Document root default: `public`
- Health path default: `/up`, fallback `/`
- Composer install with production defaults
- JS build step configurable
- Bun/Node strategy configurable
The same build artifact is used by web, worker, and websocket services. Runtime services differ by entrypoint/command.
Default topology:
- One web service.
- No worker service by default.
- Scheduler enabled on the web service by default.
- Dedicated worker service is recommended when queues are used, but created only when the user opts in.
Worker options:
- Dedicated worker service, recommended.
- Embedded worker in web service, allowed for low-throughput apps but not recommended for production.
- No workers, default.
Keystone should warn against deployed environments using `QUEUE_CONNECTION=sync`, but it should not automatically create worker services.
## 8. Scheduler Model
Mirror Laravel Cloud's scheduler model.
Scheduler is not a standalone service by default. It is a role/capability attached to a selected web or worker service.
Defaults:
- `scheduler_enabled`: true for Laravel templates.
- `scheduler_target_service_id`: primary web service.
- `scheduler_mode`: `single`.
Runtime behavior:
- `single`: run `schedule:run` every minute on exactly one selected replica.
- `every_replica`: run on each replica. This is advanced and explicit.
Keystone should enforce one scheduler runner per environment by default. Users may still use Laravel's `onOneServer()` for application-level safety.
## 9. Service Drivers
V1 services are explicitly coded drivers only. No arbitrary Docker image service in the v1 happy path.
Driver contract should define:
- service type and version track
- default image policy
- ports
- volumes
- environment schema
- health checks
- resource defaults
- supported slice types
- Compose rendering
- operation steps
- env var exports
- firewall requirements
- update behavior
V1 driver list:
- Caddy 2 gateway
- Laravel managed runtime using `serversideup/php` FrankenPHP
- Postgres 18
- Valkey 8
Use latest minor versions for new service deploy/update operations by resolving image tags to digests. Store the resolved digest on the operation/service/replica for reproducible rollbacks.
Do not silently update managed service images. Show updates in the UI and require an explicit service update/redeploy operation.
## 10. Persistent Storage
Use named Docker volumes for persistent service-local data.
Examples:
- Postgres: `keystone_service_<id>_postgres_data`
- Valkey: named volume when persistence is enabled
- Caddy: named volumes for `/data` and `/config`
Avoid distributed storage in v1. Moving a stateful service to another server requires an explicit migration operation.
## 11. Stateful Service Updates
V1 accepts downtime for single-node stateful updates.
Postgres/Valkey update flow:
1. User explicitly triggers update/redeploy.
2. Keystone warns about downtime and data risk.
3. Optional backup checkbox appears only if backup capability exists.
4. Stop container.
5. Preserve named volume.
6. Start new container with updated image digest.
7. Health check.
8. Mark operation complete.
Rolling stateful updates and HA clusters are v2.
## 12. Slices And Attachments
Attaching a managed service to an environment should create sensible default slices automatically.
Postgres attachment:
- Create database/user slice by default.
- Generate credentials.
- Wire `DB_*` environment variables.
Valkey attachment:
- Create/select logical slice if supported.
- Wire `REDIS_*`.
- Recommend `CACHE_STORE=redis`, `SESSION_DRIVER=redis`, or `QUEUE_CONNECTION=redis` depending on role.
- Do not silently change queue behavior without confirmation.
Caddy/domain attachment:
- Create route slice.
- Wire gateway route to environment web service.
Advanced users can select existing slices or create slices manually from service detail pages.
Slice operations should be independent from service container deployments. Creating a Postgres database/user should run as a slice operation against an existing Postgres replica, not redeploy the Postgres container.
## 13. Environment Variables
Keystone manages env vars from attachments and slices.
Postgres slice should export:
- `DB_CONNECTION=pgsql`
- `DB_HOST`
- `DB_PORT=5432`
- `DB_DATABASE`
- `DB_USERNAME`
- `DB_PASSWORD`
Valkey slice/service should export:
- `REDIS_HOST`
- `REDIS_PORT=6379`
- optional `CACHE_STORE=redis`
- optional `SESSION_DRIVER=redis`
- optional `QUEUE_CONNECTION=redis`
User-defined variables remain editable. Managed variables should show their source and whether they are overridable.
## 14. Networking And Internal Aliases
Support both same-server Docker networking and cross-server private networking.
Routing preference:
1. Same server: Docker network aliases/container DNS.
2. Same provider private network: private IP and internal port.
3. Public fallback only if explicitly allowed.
V1 should not build distributed DNS. Use deterministic internal hostnames and generated env vars. Where Keystone controls Docker networks, use network aliases. For cross-server communication, inject private IP/port endpoints.
Future agent/DNS systems should be possible, but are out of scope for v1.
Recommended endpoint model:
- `service_id`
- `service_replica_id` nullable
- `scope`: `docker_network`, `private_network`, `public`
- `hostname`
- `ip_address` nullable
- `port`
- `priority`
- `health_status`
## 15. Gateway And Cutover
There must be exactly one gateway service per server for v1.
Caddy owns public ports `80` and `443`. Application runtime containers should bind only to internal Docker networks or assigned internal ports.
Zero-downtime deployment happens at the gateway layer:
1. Render/start new service replica with unique container/project name.
2. Health check new replica.
3. Update Caddy upstreams to include the new healthy replica.
4. Reload Caddy.
5. Drain/remove old replica from Caddy upstreams.
6. Stop old container after the drain window.
For same-server upstreams, Caddy can use Docker network names. For cross-server upstreams, Caddy uses private IP and assigned internal port.
Web services may span multiple servers in v1. Keystone provides load balancing through Caddy upstreams but does not optimize global latency or regional placement.
Future v2 doctor page can flag:
- cross-region upstreams
- public-network fallbacks
- missing workers for async queues
- scheduler every-replica risks
- inefficient database/cache placement
## 16. Docker Compose Runtime
Use generated Docker Compose files, not raw `docker run`, for v1 runtime management.
Suggested server layout:
- `/home/keystone/services/<service-id>/compose.yml`
- `/home/keystone/services/<service-id>/.env`
- `/home/keystone/gateway/Caddyfile`
- `/home/keystone/operations/<operation-hash>/`
Compose files are generated artifacts. The Keystone database is canonical.
Compose should be used for:
- container definitions
- env files
- named volumes
- networks
- health checks
- restart policies
- resource limits
- labels
Resource controls:
- Use plain Docker runtime constraints such as `cpus`, `mem_limit`, and `memswap_limit`.
- Avoid relying on Swarm-only `deploy.resources` semantics for v1.
Example:
```yaml
services:
web:
image: registry.example.com/app:abc123
cpus: "1.0"
mem_limit: 1024m
memswap_limit: 1024m
```
## 17. Environment Deployment Flow
Environment deployment creates a parent `environment_deploy` operation.
High-level flow:
1. Resolve target commit.
2. Create or reuse build artifact.
3. Compute desired service changes.
4. Include only services with `deploy_policy=with_environment` and changed revision/config.
5. Check dependency-only services and attached slices.
6. Run pre-switch service steps.
7. Run application migrations according to service migration policy.
8. Deploy new web/worker/websocket replicas.
9. Health check new replicas.
10. Update gateway routes.
11. Reload Caddy.
12. Drain and stop old replicas.
13. Mark operation complete.
Database/cache services attached to the environment are checked but not redeployed unless the user explicitly deploys or updates them.
## 18. Migrations
Database migrations are owned by the application runtime service deployment.
Recommended fields on service config:
- `migration_mode`: `auto`, `manual`, `disabled`
- `migration_timing`: `pre_switch`, `post_switch`
- `migration_command`: default `php artisan migrate --force`
Default for Laravel web services:
- `migration_mode=auto`
- `migration_timing=pre_switch`
- command `php artisan migrate --force`
Manual mode should allow the user to run migration operation explicitly.
## 19. Onboarding
Onboarding should guide users through:
1. Organisation creation.
2. Server provider setup, Hetzner first.
3. Source provider/repository setup, including Gitea/GitHub/generic Git.
4. Deploy key installation and verification.
5. Registry setup. Optional for single-server, required for multi-server.
6. Server creation/provisioning.
7. Application/environment creation.
8. Optional service attachments: Postgres, Valkey, domain/gateway.
If an environment spans more than one server and no registry exists, deployment should be blocked with a registry setup prompt.
## 20. Current Code Migration Plan
The current code already has useful pieces:
- Provider abstraction
- Hetzner server creation
- Server provisioning jobs
- Service drivers
- Polymorphic deployments
- Step execution over SSH
Refactor in phases.
### Phase 1: Schema Alignment
- Add `environments` table.
- Rename `deployments` to `operations`.
- Rename `steps` to `operation_steps`.
- Add `operations.parent_id`.
- Add `operations.kind`.
- Add `service_replicas`.
- Add `service_slices`.
- Add `environment_attachments`.
- Add `environment_variables`.
- Add registry/source/build artifact tables.
### Phase 2: Model Cleanup
- Replace `Application::instances()` as the primary deployment path with `Application::environments()`.
- Keep or migrate `Instance` into `ServiceReplica` depending on implementation cost.
- Replace `Service::slices` references with real `ServiceSlice` relationship.
- Replace `Deployment` references with `Operation`.
- Replace deployment step jobs with operation step jobs.
### Phase 3: Driver Contract
- Define formal driver interfaces for service deployment, replica rendering, slices, health checks, and env exports.
- Implement Caddy 2 driver.
- Implement Postgres 18 driver with database/user slice provisioning.
- Implement Valkey 8 driver.
- Implement Laravel runtime driver/template.
### Phase 4: Compose Renderer
- Render Compose files from DB state.
- Upload generated files over SSH.
- Run `docker compose` operations.
- Capture container IDs and health state into `ServiceReplica`.
### Phase 5: Environment Deploy
- Build application artifact.
- Deploy web replicas.
- Run migrations.
- Health check.
- Cut over Caddy.
- Stop old replicas.
### Phase 6: UI Simplification
- Present environments as the primary application surface.
- Present services under an environment with sensible defaults.
- Hide deploy policies by default.
- Provide one-click add worker.
- Provide managed attachment flows for Postgres/Valkey/Caddy.
## 21. Explicit V2 Deferrals
Out of scope for v1:
- Server agent.
- Distributed internal DNS.
- Edge routing or anycast.
- Automatic regional topology optimization.
- Custom Dockerfiles.
- Arbitrary Docker image services.
- Non-Laravel first-class app frameworks.
- Managed Docker registry.
- HA Postgres/Valkey.
- Rolling stateful updates.
- Distributed storage.
- Full backup orchestration.
- Automatic deploy key installation via Gitea/GitHub API.