vrp-ops · the new release system (draft)

Customers get code we have used for a week. Hostaway gets code production has run for a week. Fixes still ship daily.

Today a merge reaches every customer within minutes, all at once, and Hostaway gets it at the same moment. In the new system, new code lives on staging for a week before production gets it, and Hostaway gets it another week later. Every release is built once, tested against the real production setup before any customer reaches it, and moved to customers in steps that roll back on their own. Bug fixes skip the wait: an agent copies them onto the live releases and they ship the same day.

how it works

One branch, main. Small PRs, reviewed by AI. Staging runs the newest build.
Every Monday, production moves to main of one week ago; Hostaway moves to production's previous release.
PRs labelled fix are cherry-picked onto both live releases and ship in the daily fix release.
Every release goes 0 → 10 → 30 → 50 → 100 % and rolls back as a whole if a check fails.
Features ship switched off; flags decide who sees them.

why it is the better choice

✓ The 2026-10-12 hang hit every customer for 30 min. Here it would sit on staging for a week, and the 0 % step stops it before any customer.
✓ The expense list crash (CUS-3072) came from an API running ahead of its UI. Here they always move and roll back together.
✓ Hostaway lost fields without warning. Here they get a changelog before every move and removals wait out a grace period.
✓ No release blocked by a missing build, and no Macroscope bill.
! The price: features reach customers a week after merge, Hostaway two; each fix is cherry-picked onto up to two release branches; and we build and run more tooling ourselves.

0 · branches, tags, environments, flagsdeploying code and turning on a feature are separate steps

branch: main

Where all work merges, small and often. Staging always runs its newest build.

The branches staging, production and hostaway are gone.

release branch

One per weekly cut: release/r-1005 starts at main of Mon Oct 5. It lives two weeks: one on production, one on Hostaway.

Only fixes are added to it, as cherry-picks. Deleted when Hostaway moves on.

tag

A built release on a release branch: r-1005, then r-1005-fix1, -fix2 as fixes arrive. Never moves.

One manifest per tag in R2 lists its builds.

environment

Not a branch. A pointer to the tag it runs: environments/production.json → r-1005-fix2.

Deploy = move the pointer forward. Rollback = move it back, for API, UI and tasks together.

feature flag

Decides who sees a feature: per environment, per team, or a share of teams. Owner and expiry date.

Can be turned on only where the feature's code already runs.

1 · three weeks: git on top, environments belowproduction = main of a week ago + fixes · hostaway = main of two weeks ago + fixes

main

Every PR merges here after AI review and tests. PRs that fix a bug carry the label fix. Features merge behind a flag.

staging

Runs every build of main, 10 min after merges settle. The team works on staging daily, so new code gets a week of real use before production.

production

Monday 10:00: moves to a new release branch cut from main of the previous Monday, plus every fix merged since. Fix releases daily at 10:00; urgent fixes at once.

hostaway

Monday 10:00: moves to the release branch production just left, with all its fixes. Same fix lane. Changelog to Hostaway's developers and Frank the Friday before.

2 · building a featurethe code ships dark; the flag decides who sees it, once the code is there

Small PRs to main, flag off. No long-lived feature branches. A preview deployment per PR lets people try a change before it merges.
The code follows the weekly cuts: staging at once, production one week after merge, Hostaway two weeks after.
Data changes add only. New tables and columns ship early; nothing old is dropped while the flag exists. Turning the flag off must leave every row it created valid.
API changes add only. Replacing a field means: add the new one, deprecate the old one, then a grace period of at least the two weeks Hostaway needs to see it.
Turn it on in steps from the console: staging → our own team → pilot customers → 25 % of production teams → all. Errors and accounting checks are compared between teams with the flag on and off. Bugs found in a step are fixed with fix PRs, which reach production the same day.
Hostaway last, after its developers had the changelog.
Remove the flag within 30 days of 100 %. Drops of old columns wait until no live release needs them.

3 · the path of one changeeach step has a check; a failed check stops the change there

4 · build once, promote the buildnothing is rebuilt per environment

builds every commit on main and every tag; only components whose content changed

api · webhook · tracking
portal routers · vrt-mcp
ecosystem workers · docs
ui app · hostaway embed

No build bakes in its environment. The UI reads its API host at runtime, so one UI build serves every environment.

──▶

R2 · vrp-releases (write once, never overwritten)

builds/<component>/<content-hash>/

bundle · source map · commit · lockfile hash

releases/r-1005-fix2.json

builds per component · wrangler config per env · openapi.json · migrations (add / drop) · PRs · risk tags

environments/production.json

live tag · Cloudflare version ID per Worker · UI deployment · rollout step

│ promote = upload the same bundle as a new version, then move traffic ▼

Not portable: Trigger.dev tasks are built per environment from the tag's commit and lockfile, staged, then promoted once the API reaches 100 %. Every Sentry event and metric carries the tag.

5 · one release runsame steps for weekly moves and fix releases

6 · fixes: production, hostaway, or botha fix goes to every live release that has the bug

The fix PR merges to main with the label fix, so no later release can lose it.
An agent checks which live release branches have the bug: production's, Hostaway's, or both. It cherry-picks the fix onto each, resolving conflicts on the older branch.
Each branch gets a new tag (r-1012-fix1, r-1005-fix3). CI runs the end-to-end tests on that tag, since staging runs main, not these tags.
Fix tags ship in the daily fix release at 10:00, or at once when urgent. Production first; Hostaway once production's fix passed, unless Hostaway is the one in trouble.
A bug only in Hostaway's older code, already fixed on main: cherry-pick that existing fix onto Hostaway's branch only.

Fixes may only add (columns, indexes). No drops, no API schema change. Features are never cherry-picked; they wait for the next cut.

7 · when an alert fires after a rolloutroll back only what the release caused

8 · shared parts and version skewshared parts must work with code that is two weeks old

Three versions are live at once: main on staging, one week old on production, two weeks old on Hostaway. CI checks each shared part against the manifests of every live tag.

9 · where things runthe devbox works; it is never the only way to stop a bad release