Hosting — VM tiers and client-VM lifecycle

Operator-facing reference for the hosted edition: one fly.io micro-VM per client (stage 2 of the hosting roadmap), how the tiers map to concrete fly parameters, and the lifecycle rules for deploy, up/downgrade and termination. The customer-facing short version lives in templates/help.html (section 19, "Hosting tiers"). Design context: docs/plan/hosting-vm-tiers-and-lifecycle-2026-08-11.md.

Measured sizing facts (2026-08-11, image sizes measured 2026-08-15)

• Local venv = 1.4 GB (Windows wheels); a full Linux container build incl. torch/sentence-transformers lands ~2.5–3 GB.
• Built images (2026-08-15, trimmed .dockerignore): slim = 883 MB, full = 2.67 GB (both validated with local docker build; torch 2.13.0+cpu present in full only) — see the build section below.
• data/ = < 1 MB (SQLite DBs + JSONL logs grow slowly; the client workspace is the real growth variable).
• No local model inference — CPU is mostly idle; memory is the binding constraint. Steady-state RSS is ~250–400 MB.
• Heavy optional deps: sentence-transformers + torch (~2 GB), chromadb + onnxruntime. The slim image (remote or keyword embeddings) installs in ~600–800 MB of packages.

Tier table

Pricing basis: fly.io pricing page, Toronto (yyz), 2026-08-14. All fly.io prices are USD (Canadian cards are converted by the bank — quote customers in USD). Machines: shared-cpu-1x = $2.17 (256 MB) / $3.56 (512 MB) / $6.35 (1 GB) / $11.93 (2 GB) per month; extra RAM $5.58/GB/mo; volumes $0.15/GB/mo, 1–500 GB, extend-only.

Notes:

• Micro is genuinely tight: measured steady-state RSS is 250–400 MB, so 256 MB survives only on the slim image and light workloads. Its second job is hosting the small Kucatoo-Sites registry apps (Apps-Pids, Kucatoo-Git, Tele-Prompter, …) — those need a containerization sweep first (Dockerfile + start.sh ENTRYPOINT per app).
• The 2026-08-11 draft understated costs (Basic "~$2/mo" was the 256 MB machine price; Plus "~$5–6" is really $6.35 + volume) — the table above is the live yyz pricing.
• shared-cpu-1x tops out at 2 GB RAM; more RAM forces more CPU (4 GB → 2x, 8 GB → 4x), so the economical tiers all sit on 1x.

A tier is really a triple: (Docker image, memory, volume). The slim/full split is a build-time decision — pip install happens only at image build time, never on the VM (a 512 MB machine OOMs on builds; this matches fly.io's build-once/deploy-many model). 512 MB serves steady state fine; it OOMs only if torch is loaded, so torch must stay out of the slim image.

Implemented 2026-08-15: one Dockerfile with ARG BUILD (full is the default so the operator's own deploys keep today's behavior):

• slim = requirements.txt alone (~600–800 MB, no torch). The heavy local-embedding stack lives in requirements-full.txt (sentence-transformers, chromadb, onnxruntime, hf chain); every import of those is lazy and fails soft (remote/keyword embeddings, in-memory LTM index).
• full = CPU torch (installed first from the CPU wheel index so pip never pulls CUDA) + requirements-full.txt (~2.5–3 GB).
• Build both: docker-build.sh / docker-build.bat → kucatoo-code:slim and kucatoo-code:full. The split is pinned by offline tests in tests/test_hosting_tiers.py.
• .dockerignore is part of the mechanism: venv311/, .kucatoo_backups/ and the artifact dirs (audio/, images/, guests/, …) must stay excluded — the first slim build weighed 2.91 GB with them in the context; the trimmed build is 883 MB.

Rules:

• Volumes can be extended upward without rebuild (fly volumes extend); fly.io cannot shrink a volume — downgrades adjust RAM/CPU only.
• RAM/CPU changes are a machine update (fly machines update --vm-memory).
• Deploy is blocked unless the client has an active subscription for the matching price. Since 2026-08-14 the gate asks the central pay hub (Kucatoo-Pay) for the entitlement instead of talking to Stripe directly — see "Central pay hub" below. Basic price exists (price_1U2vnzLHj4B38tEsPP2oHvnH); Plus/Micro gating is env-driven until their prices exist (STRIPE_PRICE_ID_PLUS / STRIPE_PRICE_ID_MICRO).

Current provisioning path (routes_fly.py)

Two provision paths, chosen by env (all endpoints operator-only):

• REST client (FLY_API_TOKEN set) — the ported fly_client.py (hosting plan §8, from the retired Kucatoo-App-Deploy-Tool): Machines API + GraphQL for IPs/certs, no flyctl, cross-platform. Provision is synchronous: app → volume (tier-sized) → machine (tier image + memory, autostop/autostart, min_machines_running=0) → wait started → shared IPv4 + cert best-effort. Destroy = machines → volumes → app. The client also carries the §3/§4 primitives: set_machine_memory, extend_volume (extend-only), suspend_app / resume_app.
• flyctl fallback (no token) — the legacy subprocess wrapper (~/.fly/bin/flyctl.exe), kept for local development; background deploy
  • post-deploy best-effort machines update --vm-memory.

Tier-aware since 2026-08-15: provision takes tier + volume_gb (+ optional customer + subdomain), validated by hosting_tiers.resolve_tier against the catalog, the tier_config toggles and the price env. The chosen shape is persisted on the clients row and audited (provision/destroy events). With billing configured, deploy requires a client row with an active/trialing subscription for the tier's price (fail-open otherwise). GET /api/fly/apps annotates each app with its clients.db tier/memory/volume/customer.

Tier toggles (owner)

Implemented 2026-08-15: a tier_config table in clients.db (tier, enabled, price_id, memory_mb, volume_gb, image_tag, updated_at) is seeded from the hosting_tiers.py catalog on every connect, owner-only GET/PATCH /api/settings/tiers flips the switch, and the Settings page's Operator block renders the toggle list. Toggle-off is soft — running deployments on a disabled tier keep running and billing; the tier only disappears from new deploys, checkouts and upgrade targets. Toggles are audited in deployment_events (tier_toggle). The deploy picker, resolve_tier validation and the checkout price lookup all read the same table; the Stripe price ids remain env-driven (STRIPE_PRICE_ID_*).

Subscriber shared-model caps (pricing locked 2026-08-16)

Paid plans include the tier's shared DeepSeek models on the operator's key within a monthly allowance. The budget is dollar-denominated internally, so provider price increases shrink the token-equivalent, not the operator's exposure (the grandfathered-price back-door — see docs/plan/hosting-roadmap-2026-08-10.md for the unit economics, which is staff-only). Customer-facing unit = credits (1 credit = $0.01 of included usage; Flash and Pro runs debit different credit counts). subscriber_limits.py + the subscriber_usage ledger + the query gate (query_gates.py): model gating, over-cap 402 with a BYOK/upgrade hint, and a user's own key always bypasses.

Founding prices $12 / $19 / $29 / $49 (anchors $16/$24/$39/$69; quarterly and annual billing −15%; no top-ups v1) — see the roadmap's product-tiers section.

Central pay hub (Kucatoo-Pay)

Kucatoo-Pay (FastAPI/SQLAlchemy) is the central payment app: checkout, signature-verified Stripe webhooks (7 event types, idempotent event table), Customer Portal, per-product /v1/entitlements API, migration runner, Dockerfile + fly.toml, 69 tests.

Kucatoo-Code side (wired 2026-08-15):

• Entitlement poll (pay_hub.py): with PAY_HUB_URL + PAY_HUB_API_KEY set, the deploy gate asks POST /v1/entitlements ({customer_email, product=coder}, bearer per-product key) and treats the hub as authoritative; a hub outage fails open to the embedded clients.db subscription check (with a note). Responses cache for 60 s.
• Provision hook receiver (routes_hosting.py): the hub's DEPLOY_HOOK_URL targets POST /api/hosting/hook (guard-exempt, shared HOSTING_HOOK_TOKEN bearer). grant provisions the client VM through the same tier path as the Sites tab — the hub verified the entitlement, so the env price gate is skipped and the tier is picked from the client's stored price (else Basic); revoke suspends the app (REST suspend_app; the flyctl fallback records the event for the operator). All hook actions are audited (operator="pay-hub").
• clients.db keeps only deployment state, the trial ledger, BYOK subscriber keys and the deployment_events audit; the hub's customers/entitlements tables are the billing record. Hub-side ops (operator): Fly Postgres in prod, real price ids, per-product ENTITLEMENT_API_KEYS, live webhook endpoint.
• Open: the hub has no Clerk — the identity bridge (email + metadata[clerk_user_id] round-trip vs Clerk JWT in the hub) is the operator's call; never run two Stripe accounts.

State: clients.db

clients rows carry the deployment shape alongside the subscription: tier, vm_size, memory_mb, volume_gb (added by _migrate() for databases created before the hosting-tiers schema — SQLite has no ADD COLUMN IF NOT EXISTS, so the migration checks PRAGMA table_info(clients) on connect and applies missing columns). set_client_deployment() updates these fields.

deployment_events is the append-only audit trail: every deploy, tier change and termination (including near-terminations) is recorded with operator identity, reason, a free-form detail (JSON old→new values, backup URL, or error) and a timestamp. Rows are never deleted — the clients row and its events remain the billing/audit record after a site is terminated (status='terminated', row kept).

Upgrade / downgrade

Implemented 2026-08-15 (POST /api/fly/change-tier, operator-only):

• RAM/CPU/volume change, same build: REST set_machine_memory + extend_volume. Disk is extend-only — a shrink is refused up front (a real shrink needs a new volume + data copy, out of scope v1).
• Build change (slim ↔ full): the same machine update swaps the image tag (kucatoo-code:slim ↔ :full). State lives on the volume so data survives; expect ~30–60 s downtime — the Sites tab warns.
• Order of operations: billing first — the subscription price switches with proration (update_subscription_price, embedded spine); the pay hub owns billing when configured (recorded, proration hub-side — the hub has no S2S plan-change endpoint yet). Only on success are the machines/volumes touched; a fly failure reverts the Stripe change and audits tier_change_failed. Then clients tier fields update and tier_change logs old→new; finishing with a best-effort health poll (any HTTP response = up) reported in the Sites tab.

Termination safeguards

Implemented 2026-08-15. Termination destroys customer data; the safeguards, in order (POST /api/fly/terminate):

1. Pre-checks — only when the subscription is cancelled/expired, or an explicit operator override with a written reason (active-subscription termination returns 409 with requires_override).
2. Backup first, always — the client volume's /data is pulled operator-side via fly-ssh (tar czf - -C /data . streamed to a temp file — nothing is added to the client app; the VM must be running), then uploaded to the operator's Drive folder. No verified upload → no terminate (override requires the written reason).
3. Two-step confirm — dialog states exactly what is destroyed and that it is irreversible; the app name must be typed.
4. Soft-delete window — suspend_app now, destroy scheduled at deletion_scheduled_at (+ TERMINATION_GRACE_DAYS, default 7); POST /api/fly/terminate-undo restores + resumes. The sweep (POST /api/fly/sweep, also lazily on /api/fly/apps) destroys apps past the window.
5. Audit — terminate_requested, pending_deletion, terminate_undone, terminated (and failures) to deployment_events with operator identity, reason, backup detail, timestamps.
6. Post-destroy cleanup — the subdomain's Alibaba DNS record removal is recorded as dns_cleanup_pending (manual v1 — no Alibaba API client), clients.status='terminated', row kept.

Immediate destroy (/api/fly/destroy) stays available for accidental/test resources and now REFUSES apps with a client row — terminate is the only path for client sites.

Settings provider links

routes_settings.py ships per-provider console links (_PROVIDER_LINKS) rendered on the Settings cards: "Get a key" and (where separate) "Manage plan" — the providers' own consoles, correct for BYOK (customer signs up with the provider, pastes the key into Settings). Qwen URLs are region-pinned to ap-southeast-1 (international token-plan console); when mainland endpoints land, add a mainland link pair alongside. The Settings page also shows an owner-only "Operator dashboards" block (Fly.io / Stripe / Clerk), gated on g.user_role == ROLE_OWNER.

verified against code: 2026-08-15