keystone/docs/implementation-spec.md

# Keystone Implementation Spec

## 1. Product Scope

Keystone is a Laravel Forge-like deployment platform that runs applications and services with Docker. The v1 product is intentionally narrow:

- Laravel is the only first-class application framework.
- Application containers use a Keystone-managed Dockerfile based on `serversideup/php` with FrankenPHP.
- Services are explicitly coded drivers, not arbitrary Docker images.
- v1 is agentless and executes operations over SSH.
- Docker Compose is used as the generated runtime artifact.
- Caddy 2 is the default and only gateway for v1.
- The Keystone database is the source of truth. Server files are generated artifacts.

V1 should make the simple path robust before adding generic Docker support, distributed agents, HA databases, edge routing, or additional frameworks.

## 2. Core Domain Model

### Organisation

Owns users, providers, registries, applications, servers, services, and environments.

### Application

A source-code project. In v1, first-class applications are Laravel repositories.

Recommended fields:

- `organisation_id`
- `name`
- `repository_url`
- `repository_type`
- `default_branch`
- `deploy_key_public`
- `deploy_key_private` encrypted
- `deploy_key_fingerprint`
- `deploy_key_installed_at` nullable

### Environment

The primary application deployment unit. An application has environments such as production, staging, or dev.

Recommended fields:

- `application_id`
- `name`
- `branch`
- `status`
- `scheduler_enabled`
- `scheduler_target_service_id` nullable
- `scheduler_mode`: `single` or `every_replica`
- `build_config` json

Default for Laravel environments:

- Scheduler enabled.
- Scheduler target is the primary web service.
- Scheduler mode is `single`.

### Service

Every deployable thing is represented as a `Service`.

Examples:

- Laravel web runtime
- Laravel worker runtime
- Laravel websocket runtime
- Caddy gateway
- Postgres
- Valkey
- Future standalone services

Recommended fields:

- `organisation_id`
- `environment_id` nullable
- `server_id` nullable for single-placement legacy convenience only; long term use replicas
- `name`
- `category`
- `type`
- `version_track`
- `driver_name`
- `status`
- `desired_replicas`
- `desired_revision`
- `deploy_policy`
- `process_roles` json
- `current_image_digest` nullable
- `available_image_digest` nullable
- `update_status`
- `default_cpu_limit` nullable
- `default_memory_limit_mb` nullable
- `config` json

Deploy policy defaults:

- Laravel web: `with_environment`
- Laravel worker: `with_environment`
- Laravel websocket: `with_environment`
- Database/cache/storage: `dependency_only`
- Gateway: `manual_or_on_route_change`
- Standalone services: `manual`

The user should not need to configure these defaults during normal setup.

### ServiceReplica

A running instance of a service on a server. A service is logical; a replica is runtime placement.

Recommended fields:

- `service_id`
- `server_id`
- `operation_id` nullable
- `container_name`
- `container_id` nullable
- `image_digest`
- `internal_host`
- `internal_port`
- `public_port` nullable
- `status`
- `health_status`
- `cpu_limit` nullable
- `memory_limit_mb` nullable
- `config` json

Replica resource limits override service defaults. Null means unrestricted except host capacity.

### ServiceSlice

A logical sub-resource inside a service. Slices belong to `Service`, not `ServiceReplica`.

Examples:

- Database and user inside Postgres
- Logical database or namespace inside Valkey
- Route inside Caddy
- Future bucket, topic, vhost, etc.

Recommended fields:

- `service_id`
- `environment_id` nullable
- `name`
- `type`
- `status`
- `config` json
- `credentials` encrypted json nullable

Slices are not containers and should not be used for scaling. They are stable logical resources that survive service replica replacement.

### EnvironmentAttachment

Connects an environment to managed service slices.

Recommended fields:

- `environment_id`
- `service_id`
- `service_slice_id` nullable
- `role`: `database`, `cache`, `queue`, `storage`, `gateway`, `custom`
- `env_prefix` nullable
- `is_primary`

Attachments should point to slices whenever a slice exists. For example, a Laravel environment attaches to a Postgres database/user slice, not merely to the Postgres service.

### EnvironmentVariable

Represents user-defined and Keystone-managed runtime environment values.

Recommended fields:

- `environment_id`
- `key`
- `value` encrypted
- `source`: `user`, `managed_attachment`, `system`
- `service_slice_id` nullable
- `overridable` boolean

Managed values should be regenerated from attachments and slices.

## 3. Operations Model

Rename `Deployment` to `Operation`.

An operation is the generic audit and execution object for all state-changing work.

### Operation

Recommended fields:

- `id`
- `parent_id` nullable
- `hash`
- `kind`
- `target_type`
- `target_id`
- `status`
- `started_at`
- `finished_at`
- timestamps

Operation kinds:

- `server_provision`
- `service_deploy`
- `replica_deploy`
- `slice_provision`
- `slice_configure`
- `environment_deploy`
- `gateway_cutover`
- `config_change`
- `credential_rotation`

### OperationStep

Rename `Step` to `OperationStep`.

Recommended fields:

- `operation_id`
- `name`
- `order`
- `status`
- `script`
- `logs`
- `error_logs`
- `secrets` encrypted json nullable
- `started_at`
- `finished_at`
- timestamps

### Parent-Child Operations

Environment deploys are parent operations that create child operations.

Example:

- `environment_deploy`
- child `service_deploy` for web
- child `replica_deploy` for each web replica
- child `slice_configure` for Caddy route updates
- child `gateway_cutover`

Standalone service deploys and slice operations can also run independently.

## 4. Server Provisioning

V1 remains agentless over SSH.

Provisioning flow:

1. Create server through provider API.
2. Wait for root SSH to become available.
3. Execute provisioning script over SSH.
4. Create Keystone management user.
5. Install Docker Engine, Docker Compose plugin, UFW, fail2ban, and required runtime packages.
6. Install Keystone SSH public key.
7. Disable password login.
8. Enable UFW with SSH open.
9. Callback or SSH verification marks server active.

Server permanent keys are for Keystone management only. Repository deploy keys must not be permanently installed on servers.

## 5. Source Providers And Repository Access

V1 source support:

- Self-hosted Gitea
- GitHub
- Generic Git over SSH

Repository access uses a Keystone-generated deploy key per application/repository.

V1 flow:

1. User enters repo SSH URL.
2. Keystone generates an ed25519 deploy key.
3. UI shows the public key.
4. User adds it to Gitea/GitHub as read-only.
5. Keystone verifies access with `git ls-remote`.

During build operations, Keystone injects the encrypted private key into a temporary operation directory and uses `GIT_SSH_COMMAND`. The key is removed after the build. Repo keys are never permanently stored on target servers or builder services.

## 6. Registry And Build Artifacts

An external registry is required for multi-server application deployments.

Single-server deployments may build and run a local image without a registry.

Multi-server deployments must:

1. Build once.
2. Push the image to the configured external registry.
3. Pull the exact same image digest on each target server.

Supported registry types:

- Generic Docker registry
- Gitea registry
- GHCR
- Docker Hub

### Build Service

Building is a service capability, not a server type.

A dedicated builder is represented as a `Service` with category `builder`. If no builder service exists, Keystone may build on the target server for single-server deployments.

Build strategies:

- `target_server`: build on selected target server. Valid for single-server.
- `dedicated_builder`: build on builder service, then push/export artifact.
- `external_registry`: pull prebuilt image from registry.

For v1:

- Single-server default: build on target server.
- Multi-server: require configured registry and build once.
- Do not rebuild independently on each server.

### BuildArtifact

Recommended fields:

- `environment_id`
- `commit_sha`
- `image_tag`
- `image_digest`
- `registry_ref` nullable
- `built_by_operation_id`
- `built_by_service_id` nullable
- `status`
- `metadata` json

## 7. Managed Laravel Runtime

V1 uses Keystone-managed Dockerfile templates only. Custom Dockerfiles are deferred.

Laravel runtime defaults:

- Base: `serversideup/php` FrankenPHP image
- PHP version configurable
- Document root default: `public`
- Health path default: `/up`, fallback `/`
- Composer install with production defaults
- JS build step configurable
- Bun/Node strategy configurable

The same build artifact is used by web, worker, and websocket services. Runtime services differ by entrypoint/command.

Default topology:

- One web service.
- No worker service by default.
- Scheduler enabled on the web service by default.
- Dedicated worker service is recommended when queues are used, but created only when the user opts in.

Worker options:

- Dedicated worker service, recommended.
- Embedded worker in web service, allowed for low-throughput apps but not recommended for production.
- No workers, default.

Keystone should warn against deployed environments using `QUEUE_CONNECTION=sync`, but it should not automatically create worker services.

## 8. Scheduler Model

Mirror Laravel Cloud's scheduler model.

Scheduler is not a standalone service by default. It is a role/capability attached to a selected web or worker service.

Defaults:

- `scheduler_enabled`: true for Laravel templates.
- `scheduler_target_service_id`: primary web service.
- `scheduler_mode`: `single`.

Runtime behavior:

- `single`: run `schedule:run` every minute on exactly one selected replica.
- `every_replica`: run on each replica. This is advanced and explicit.

Keystone should enforce one scheduler runner per environment by default. Users may still use Laravel's `onOneServer()` for application-level safety.

## 9. Service Drivers

V1 services are explicitly coded drivers only. No arbitrary Docker image service in the v1 happy path.

Driver contract should define:

- service type and version track
- default image policy
- ports
- volumes
- environment schema
- health checks
- resource defaults
- supported slice types
- Compose rendering
- operation steps
- env var exports
- firewall requirements
- update behavior

V1 driver list:

- Caddy 2 gateway
- Laravel managed runtime using `serversideup/php` FrankenPHP
- Postgres 18
- Valkey 8

Use latest minor versions for new service deploy/update operations by resolving image tags to digests. Store the resolved digest on the operation/service/replica for reproducible rollbacks.

Do not silently update managed service images. Show updates in the UI and require an explicit service update/redeploy operation.

## 10. Persistent Storage

Use named Docker volumes for persistent service-local data.

Examples:

- Postgres: `keystone_service_<id>_postgres_data`
- Valkey: named volume when persistence is enabled
- Caddy: named volumes for `/data` and `/config`

Avoid distributed storage in v1. Moving a stateful service to another server requires an explicit migration operation.

## 11. Stateful Service Updates

V1 accepts downtime for single-node stateful updates.

Postgres/Valkey update flow:

1. User explicitly triggers update/redeploy.
2. Keystone warns about downtime and data risk.
3. Optional backup checkbox appears only if backup capability exists.
4. Stop container.
5. Preserve named volume.
6. Start new container with updated image digest.
7. Health check.
8. Mark operation complete.

Rolling stateful updates and HA clusters are v2.

## 12. Slices And Attachments

Attaching a managed service to an environment should create sensible default slices automatically.

Postgres attachment:

- Create database/user slice by default.
- Generate credentials.
- Wire `DB_*` environment variables.

Valkey attachment:

- Create/select logical slice if supported.
- Wire `REDIS_*`.
- Recommend `CACHE_STORE=redis`, `SESSION_DRIVER=redis`, or `QUEUE_CONNECTION=redis` depending on role.
- Do not silently change queue behavior without confirmation.

Caddy/domain attachment:

- Create route slice.
- Wire gateway route to environment web service.

Advanced users can select existing slices or create slices manually from service detail pages.

Slice operations should be independent from service container deployments. Creating a Postgres database/user should run as a slice operation against an existing Postgres replica, not redeploy the Postgres container.

## 13. Environment Variables

Keystone manages env vars from attachments and slices.

Postgres slice should export:

- `DB_CONNECTION=pgsql`
- `DB_HOST`
- `DB_PORT=5432`
- `DB_DATABASE`
- `DB_USERNAME`
- `DB_PASSWORD`

Valkey slice/service should export:

- `REDIS_HOST`
- `REDIS_PORT=6379`
- optional `CACHE_STORE=redis`
- optional `SESSION_DRIVER=redis`
- optional `QUEUE_CONNECTION=redis`

User-defined variables remain editable. Managed variables should show their source and whether they are overridable.

## 14. Networking And Internal Aliases

Support both same-server Docker networking and cross-server private networking.

Routing preference:

1. Same server: Docker network aliases/container DNS.
2. Same provider private network: private IP and internal port.
3. Public fallback only if explicitly allowed.

V1 should not build distributed DNS. Use deterministic internal hostnames and generated env vars. Where Keystone controls Docker networks, use network aliases. For cross-server communication, inject private IP/port endpoints.

Future agent/DNS systems should be possible, but are out of scope for v1.

Recommended endpoint model:

- `service_id`
- `service_replica_id` nullable
- `scope`: `docker_network`, `private_network`, `public`
- `hostname`
- `ip_address` nullable
- `port`
- `priority`
- `health_status`

## 15. Gateway And Cutover

There must be exactly one gateway service per server for v1.

Caddy owns public ports `80` and `443`. Application runtime containers should bind only to internal Docker networks or assigned internal ports.

Zero-downtime deployment happens at the gateway layer:

1. Render/start new service replica with unique container/project name.
2. Health check new replica.
3. Update Caddy upstreams to include the new healthy replica.
4. Reload Caddy.
5. Drain/remove old replica from Caddy upstreams.
6. Stop old container after the drain window.

For same-server upstreams, Caddy can use Docker network names. For cross-server upstreams, Caddy uses private IP and assigned internal port.

Web services may span multiple servers in v1. Keystone provides load balancing through Caddy upstreams but does not optimize global latency or regional placement.

Future v2 doctor page can flag:

- cross-region upstreams
- public-network fallbacks
- missing workers for async queues
- scheduler every-replica risks
- inefficient database/cache placement

## 16. Docker Compose Runtime

Use generated Docker Compose files, not raw `docker run`, for v1 runtime management.

Suggested server layout:

- `/home/keystone/services/<service-id>/compose.yml`
- `/home/keystone/services/<service-id>/.env`
- `/home/keystone/gateway/Caddyfile`
- `/home/keystone/operations/<operation-hash>/`

Compose files are generated artifacts. The Keystone database is canonical.

Compose should be used for:

- container definitions
- env files
- named volumes
- networks
- health checks
- restart policies
- resource limits
- labels

Resource controls:

- Use plain Docker runtime constraints such as `cpus`, `mem_limit`, and `memswap_limit`.
- Avoid relying on Swarm-only `deploy.resources` semantics for v1.

Example:

```yaml
services:
  web:
    image: registry.example.com/app:abc123
    cpus: "1.0"
    mem_limit: 1024m
    memswap_limit: 1024m
```

## 17. Environment Deployment Flow

Environment deployment creates a parent `environment_deploy` operation.

High-level flow:

1. Resolve target commit.
2. Create or reuse build artifact.
3. Compute desired service changes.
4. Include only services with `deploy_policy=with_environment` and changed revision/config.
5. Check dependency-only services and attached slices.
6. Run pre-switch service steps.
7. Run application migrations according to service migration policy.
8. Deploy new web/worker/websocket replicas.
9. Health check new replicas.
10. Update gateway routes.
11. Reload Caddy.
12. Drain and stop old replicas.
13. Mark operation complete.

Database/cache services attached to the environment are checked but not redeployed unless the user explicitly deploys or updates them.

## 18. Migrations

Database migrations are owned by the application runtime service deployment.

Recommended fields on service config:

- `migration_mode`: `auto`, `manual`, `disabled`
- `migration_timing`: `pre_switch`, `post_switch`
- `migration_command`: default `php artisan migrate --force`

Default for Laravel web services:

- `migration_mode=auto`
- `migration_timing=pre_switch`
- command `php artisan migrate --force`

Manual mode should allow the user to run migration operation explicitly.

## 19. Onboarding

Onboarding should guide users through:

1. Organisation creation.
2. Server provider setup, Hetzner first.
3. Source provider/repository setup, including Gitea/GitHub/generic Git.
4. Deploy key installation and verification.
5. Registry setup. Optional for single-server, required for multi-server.
6. Server creation/provisioning.
7. Application/environment creation.
8. Optional service attachments: Postgres, Valkey, domain/gateway.

If an environment spans more than one server and no registry exists, deployment should be blocked with a registry setup prompt.

## 20. Current Code Migration Plan

The current code already has useful pieces:

- Provider abstraction
- Hetzner server creation
- Server provisioning jobs
- Service drivers
- Polymorphic deployments
- Step execution over SSH

Refactor in phases.

### Phase 1: Schema Alignment

- Add `environments` table.
- Rename `deployments` to `operations`.
- Rename `steps` to `operation_steps`.
- Add `operations.parent_id`.
- Add `operations.kind`.
- Add `service_replicas`.
- Add `service_slices`.
- Add `environment_attachments`.
- Add `environment_variables`.
- Add registry/source/build artifact tables.

### Phase 2: Model Cleanup

- Replace `Application::instances()` as the primary deployment path with `Application::environments()`.
- Keep or migrate `Instance` into `ServiceReplica` depending on implementation cost.
- Replace `Service::slices` references with real `ServiceSlice` relationship.
- Replace `Deployment` references with `Operation`.
- Replace deployment step jobs with operation step jobs.

### Phase 3: Driver Contract

- Define formal driver interfaces for service deployment, replica rendering, slices, health checks, and env exports.
- Implement Caddy 2 driver.
- Implement Postgres 18 driver with database/user slice provisioning.
- Implement Valkey 8 driver.
- Implement Laravel runtime driver/template.

### Phase 4: Compose Renderer

- Render Compose files from DB state.
- Upload generated files over SSH.
- Run `docker compose` operations.
- Capture container IDs and health state into `ServiceReplica`.

### Phase 5: Environment Deploy

- Build application artifact.
- Deploy web replicas.
- Run migrations.
- Health check.
- Cut over Caddy.
- Stop old replicas.

### Phase 6: UI Simplification

- Present environments as the primary application surface.
- Present services under an environment with sensible defaults.
- Hide deploy policies by default.
- Provide one-click add worker.
- Provide managed attachment flows for Postgres/Valkey/Caddy.

## 21. Explicit V2 Deferrals

Out of scope for v1:

- Server agent.
- Distributed internal DNS.
- Edge routing or anycast.
- Automatic regional topology optimization.
- Custom Dockerfiles.
- Arbitrary Docker image services.
- Non-Laravel first-class app frameworks.
- Managed Docker registry.
- HA Postgres/Valkey.
- Rolling stateful updates.
- Distributed storage.
- Full backup orchestration.
- Automatic deploy key installation via Gitea/GitHub API.