Measure Before You Optimize: The Deploy Got Faster, Just Not From the Idea I Expected

I had a number in my hands: 3× faster.

It was the measurement of a deploy where only one micro-frontend changed: 70.6 seconds the new way, against 216.6 seconds rebuilding and shipping everything. With that number, shipping each micro-frontend independently looked like the big lever for a pipeline that took close to 20 minutes per run.

Then I replayed, locally, the last 24 real deploys of that pipeline, each at the commit that actually went live, the old way and the new way. The result was 6.5% faster overall. On the production branch, the new way was 13.8% slower.

This post is about the gap between the number that anchored my expectations and the number reality gave back. It is also about what actually made the deploy faster: two changes far less interesting than the idea I wanted to build.

The system, in plain terms

The front end is a monorepo with a Shell (the root application plus a "hoster") and eight Remotes, micro-frontends loaded at runtime through webpack Module Federation.

All of it ships as a single nginx image to a Kubernetes cluster. The pipeline runs on Azure DevOps in two jobs: CI builds the image and pushes it to the registry; CD updates the deployment and moves the ticket.

Two consequences matter for the rest of this post:

• every push to dev, hml or main rebuilds and redeploys everything, even when the change is documentation only;
• changing one line in a Remote costs the same as changing the whole Shell.

Where the time actually went

Before touching anything, I looked at the 60 most recent pipeline runs. A run took P50 18.4 min and P90 29.7 min. Counting time in the queue, the median was 21.8 min.

Broken down by step, using the medians of the 24 most recent runs:

Two things stood out.

The step medians sum to about 20.1 minutes, including 7.9 minutes of agent waits. After CI finished building, the separate CD job needed another agent and another checkout.

Second, inside the Docker build, what weighed was the app builds. In a probe on the root app, minification (Terser, webpack's default) accounted for about 58% of build time.

Neither of these has anything to do with shipping micro-frontends independently. I only understood how much that mattered later.

The sum of step medians is an approximate time budget, not the median total duration of a run. The 20.1-minute scenarios and savings derived from them should therefore be read as estimates. Local results also do not establish a reduction already achieved in the production pipeline.

The ground rule: everything local

The system is in production and the pipeline is shared. So I set one rule: no calls to the cloud, the real pipeline or the ticket tracker during the work. Everything would be proven locally before anyone decided to take it to production.

For that I built a local harness:

• a kind cluster in place of the real Kubernetes;
• a local Docker registry;
• an HTTP stub of the ticket tracker;
• LocalStack standing in for S3;
• a guard that refuses to run if it finds cloud credentials or a kubectl context other than kind's.

The Azure DevOps data was read once, with a read-only token, and frozen into JSON. After that, no analysis queried Azure again.

The other rule was about labels. Every number in the reports carries one: [M] measured (with command, machine and commit) or [E]/[P] estimated or projected (with the formula). That feels like bureaucracy until the day you need to explain why an estimate was wrong.

Every local measurement ran on an ordinary laptop: Intel i5-1235U, 12 threads, 10 GiB of RAM, under WSL2.

L1: one job instead of two

The first lever was as boring as it gets: move the CD steps into the CI job. The YAML diff was +1 / −15 lines, and the scripts of every step stayed byte-for-byte identical.

# before: a second stage, with its own agent and checkout
- stage: Continuos_Deploy
  dependsOn: Continuos_Integrations
  jobs:
    - job: Publish
      steps:
        - checkout: self
        # ...kubeconfig, set image, rollout, ticket

# after: the same steps at the end of the CI job

That removes the second agent wait and the second checkout. From the medians, that is −5.1 min per run (estimated).

There is an honest limit here. I could prove locally that the single job works end to end on dev, hml and main: the right tag per branch, the image pushed, the rollout done, the ticket moved, and failures stopping the job at the right step. But the time gain cannot be proven locally, because it comes from Azure's agent queue. Until it runs on the real pipeline, that number is an estimate.

L2: SWC instead of Terser

The second lever was swapping the minifier. I tried esbuild and SWC in a scratch folder. esbuild was fast, but it grew the gzipped bundle by 25% to 39%, so it was out. SWC cut build time without changing size.

The change became one module shared by the ten webpack configs, with an environment variable to fall back to Terser:

// Production JS minimizer shared by every webpack config.
// BUILD_MINIFIER=terser restores webpack's default minifier (comparison and rollback switch).
const MINIFIERS = {
  swc: { minify: TerserPlugin.swcMinify },
  terser: { terserOptions: { compress: { passes: 2 } } },
};

export function jsMinimizer() {
  return new TerserPlugin(MINIFIERS[resolveMinifier(process.env.BUILD_MINIFIER)]);
}

Measured locally:

• sequential builds of every app: 286.2 s → 166.7 s (−42%);
• the real dev Docker image, no cache: the app build batches went from 204.7 s → 148.9 s (−27%), and the whole docker build from 274 s → 232 s (−15%);
• gzipped JS size within ±1% for every app.

Speed without correctness is worth nothing, so I also checked behaviour:

• every emitted JS file passes a syntax check;
• every Remote still emits its remoteEntry.js;
• the image returns 200 on every entry route;
• seven Remote routes executed in jsdom produce exactly the same errors, files and rendered text before and after.

What was left out was a real browser, and that was recorded as a post-merge step.

Putting L1 and L2 together, the estimate for the real pipeline came to ~20.1 min → ~11.5–12.7 min per run (−37% to −43%). It is an estimate: the formula is in the report, and the hosted agent's core count is unknown.

The ambitious idea: Microdeploy

With L1 and L2 ready, I moved on to the part I actually wanted to build. The idea was to take Remotes out of the Shell image and ship each one independently. I ran a pilot with one Remote, which in the project we call a Migrated Remote.

The design had five pieces:

• Affected-set detector. For each commit range, a script decides what needs to ship: only the Shell, only the Remote, both, or nothing. The rules live in a versioned, tested file, and when in doubt everything is affected.
• Immutable Release. The Remote becomes an immutable snapshot in object storage, one per commit and environment. Hashed files are published with a one-year cache. Only remoteEntry.js is mutable.
• Promotion. Making a Release active means swapping remoteEntry.js, behind two guards: the candidate Release must descend from the active one (git merge-base --is-ancestor) and pass an integrity check (manifest sha256, each file's MIME type, a filename allowlist).
• Post-Promotion verification. Done through the Shell: HTTP, mounting ./App in jsdom, and a stale-tab check. A tab opened before a Promotion asks for the previous Release's chunks afterwards, so deleting old files would break anyone already browsing.
• Automatic Rollback to the previous Release when any check fails, with no rebuild.

Everything was built test-first, in nine tasks, from a PRD with acceptance criteria. One finding of that process was that the PRD's own Promotion algorithm did not satisfy the concurrency criterion without a lock. Two simultaneous Promotions could end on the older version, which I reproduced in 1 of 3 runs. The test caught it before it became an incident.

The number that anchored expectations

Measured on the same machine and the same commit, as the mean of two rounds:

Three times faster. That was the number that stuck.

The same report, though, already carried the warning. I ran the detector over every commit since July and checked it against an oracle built from the pnpm dependency graph, with zero false negatives. Only 15 of 165 runs on dev would have skipped the Shell, for about 8% projected savings. On main it was 0 of 45, so 0%.

The warning was there, in a table under the headline. The headline won.

Reality check: replay the last 24 deploys

To settle it, I replayed locally the 24 most recent deploys of the real pipeline, fairly:

• each run uses the commit that actually went live;
• on top of that commit goes only today's deploy mechanism (scripts, config and the single-job YAML), with the app code untouched;
• the detector gets the merge's range (first parent to commit), exactly as it would in the pipeline;
• per run: an untimed setup, then the new way, then the old way, on the same machine under the same conditions.

The batch took about 3.5 hours. It ran in the background, wrote every result to a JSON file, and could resume where it stopped, which paid off when I had to fix the setup halfway through.

The result, as local totals of the old way against the new way:

The revealing part was the kind of deploy the detector picked:

• 18 needed only the Shell;
• 4 needed the Shell and the Remote;
• 2 had nothing to ship;
• 0 were Remote-only, the one case where "3×" applies.

Of the 465.4 s saved in total:

• +543.8 s came from the two "nothing to ship" deploys, which became an ≈8 s plan instead of an ≈280 s build;
• +201.5 s came from the 18 Shell-only deploys, ≈4% per run and with as much noise as signal;
• −279.9 s came from the four Shell + Remote deploys, each ≈72 s slower.

Why the estimate missed

I rebuilt the reasoning piece by piece, because "the estimate was wrong" teaches nothing on its own.

1. The new way is not slower in most cases. On Shell-only deploys it ties or wins by about 4%, and on nothing-to-ship deploys it is dramatically better. Only the Shell + Remote case gets worse. main came out slower because, in a sample of four, two were of that kind.

2. "3×" measured one path, not a typical deploy. It is the Remote-only path, which did not show up once in the 24. The history projection already said ≈8% on dev and 0% on main, but a big round number anchors expectations more than a table does.

3. The real unit of deploy is the PR merge, not the commit. All 24 runs were merges, and a merge bundles many commits. One of them touching Shell code, the shared package or package.json is enough to rebuild the whole Shell. That is why 22 of 24 needed it.

4. Only one Remote was migrated. The Shell build runs in parallel batches and is bound by its slowest app, the root app. Taking one small Remote out of the batch barely changes the total, as the 211.6 s vs 216.6 s measurement already showed.

5. Shell + Remote does more work, in sequence. The orchestrator runs the Shell job, then the Remote job: build, Release, Promotion and checks, some 80–90 s more. The old way built everything in parallel inside one image.

6. Azure time is mostly outside the build. Locally, these deploys take ≈4.5–7 min. On Azure, the median start-to-finish was 18 min for the same runs, and, from the pipeline medians, agent waits sum to almost 8 minutes in the step-median budget. Microdeploy does not touch that.

The 70.6-second measurement was correct. I attached it to the wrong question: I measured the path the idea optimized, instead of the distribution of deploys it would actually meet.

The bugs the replay found

Replaying old deploys with the new mechanism turns out to be a great regression test by accident.

On the first pass, the four Shell + Remote deploys failed to publish the Release. The old commits were still minified with Terser, and Terser extracts license comments into files like 245.9aad9037.js.LICENSE.txt. The Release allowlist required the hash right before the last extension, and this name has the hash before .js.LICENSE.txt.

The PRD said to publish "names containing a content hash". That file contains one, so the bug was a regex that was too narrow, not the design. The sequence was:

1. a failing regression test first;
2. a third pattern in the allowlist, with near-miss names still refused;
3. the four deploys replayed, now passing both ways.

"hashedPatterns": [
  "^(.*/)?[^/]+\\.[0-9a-f]{8}\\.[a-z0-9]+$",
  "^(.*/)?[0-9a-f]{20}\\.[a-z0-9]+$",
  "^(.*/)?[^/]+\\.[0-9a-f]{8}\\.js\\.LICENSE\\.txt$"
]

There was a second bug, mine, in the replay itself: the setup built an Active Release even for Shell-only deploys, which did not need one. On the first pass, 22 rows failed in setup. I fixed it, dropped those rows and resumed the batch from the JSON. That kind of thing does not go in a footnote. It is exactly why the batch was resumable and the raw results were versioned.

Saying no to the next optimization

The obvious next step was running the Shell and Remote jobs in parallel when both are needed. I got as far as designing it:

• the Remote builds and publishes its Release while the Shell is being built;
• Promotion waits for the Shell verification;
• Promotions run one at a time;
• a --sequential flag keeps today's behaviour.

Before implementing it, I did the math. It would recover ≈40–50 s on 4 of 24 deploys: about 3 minutes out of ≈467 minutes of real pipeline time, or ≈0.6%.

I deferred it. The decision was recorded with the triggers to reopen it: when Microdeploy moves into the real pipeline, or when more Remotes are migrated, because then Shell + Remote deploys become more common. Knowing what not to build is also a result.

What was prepared for review

Only L1 and L2 went up for review. I took the three commits for those changes, applied them on top of the current development branch in a clean branch, regenerated the lockfile and kept only code and tests: 17 files. The image was built locally and passed the smoke check.

Microdeploy stays as a locally tested mechanism: immutable Releases, guarded Promotion, verification and automatic Rollback, all under tests. Its value today is release safety, the ability to ship and undo a Remote without a rebuild and preserving open tabs in the tested scenarios, not pipeline time. The pilot's largest gain appears on Remote-only deploys. Migrating more Remotes could make that path more common, but the net pipeline gain remains unmeasured.

Along the way a few out-of-scope findings also came up: a rollout that could go green with a container that was not serving, and configuration that took the whole front end down if a single backend did not resolve. Those were reported separately, each with a local reproduction.

The biggest remaining lever is not even in the build: it is the agent queue. More parallel jobs or a self-hosted agent go after the minutes no code change touches. After that come a shallow checkout and a leaner Docker context, smaller and low-risk gains.

Before the next optimization

If I had to redo this project with one rule, it would be: before measuring how fast a new path is, count how often that path happens. The "3×" was true for a case that did not occur once in 24 real deploys. My own report already had the projection over commit history that pointed there, and classifying merges instead of commits would have pointed even closer. That frequency table belongs above the headline number, not below it.

Two practices made it possible to see that at all, and I would keep both:

• Replay real history, not synthetic cases. The 24 replays exercised changes that actually happened and found a real bug in the Release allowlist. They are still local measurements: execution order, cache state and the lack of repeated controlled trials limit how much of a gain can be attributed to the architecture.
• Label every number as measured or estimated. That separation is what let me trace exactly where the estimate went wrong instead of replacing one number with another.

The end result splits the work by what it is worth. One job fewer and a faster minifier are an estimated 37% to 43% reduction in pipeline time, still to be confirmed on the real pipeline. Microdeploy measured 6.5% locally. It stays in the project for the release safety it adds, and it should be judged by that.

What I still do not know is how much of the remaining time is the agent queue on the real pipeline after L1 ships. That measurement decides what comes next.