Testing and safe deployments
Test services at the right levels, catch breaking API changes with contract tests, and release safely with canaries, blue-green and feature flags.
- Shape a test suite for microservices, from unit to end-to-end
- Use consumer-driven contract tests to catch breaking changes before deploying
- Release with canaries, blue-green deployments and feature flags
Each Galactic Noodle Express service deploys on its own, many times a day. That only works if a deploy can’t quietly break the services that depend on it - and if a bad deploy is caught and undone fast.
Testing, from many-and-fast to few-and-slow:
- Unit tests of each service’s logic.
- Integration tests of a service with its real database or broker (often in throwaway containers).
- Contract tests between services (below).
- A few end-to-end tests of critical journeys (“order noodles to Mars”) across the deployed system. Keep these few: they’re slow, flaky, and need everything up at once.
Consumer-driven contract tests
Ordering consumes the Menu API. Instead of an end-to-end test, Ordering’s team writes a contract: “for GET /dishes/42, I need id (int), name (string) and price_cents (int)”. The contract is shared (tools like Pact use a broker), and Menu’s CI runs every consumer’s contract against Menu’s code. If a Menu change would break Ordering, Menu’s build fails - before anything is deployed, and without starting Ordering at all.
Contracts only list what the consumer actually uses, so the provider stays free to add fields or change the ones nobody reads.
Releasing safely
| Strategy | How | Rollback |
|---|---|---|
| Rolling update | replace instances gradually | roll forward/back gradually |
| Blue-green | run the new version (green) beside the old (blue), switch all traffic at once | switch back instantly |
| Canary | send a small share of traffic (5%) to the new version, compare its metrics with the old, then increase stepwise | move the share back to 0 |
| Feature flags | deploy code switched off; turn features on for some users at runtime | flip the flag off - no deploy |
Canary analysis can be automated: at each stage, compare the canary’s error rate and latency with the baseline, and automatically promote or roll back. Decoupling deploy (code is running) from release (users see it) with flags makes both safer.
1def judge(baseline, canary):
2 if canary["error_rate"] > baseline["error_rate"] + 0.5:
3 return "rollback: errors"
4 if canary["p95_ms"] > baseline["p95_ms"] * 1.10:
5 return "rollback: latency"
6 return "promote"
7
8baseline = {"error_rate": 0.4, "p95_ms": 210}
9print(judge(baseline, {"error_rate": 0.5, "p95_ms": 220}))
10print(judge(baseline, {"error_rate": 0.6, "p95_ms": 260}))
11print(judge(baseline, {"error_rate": 2.1, "p95_ms": 200}))promote rollback: latency rollback: errors
Key takeaways
Mostly unit and integration tests, contract tests between services, and only a few end-to-end tests.
Consumer-driven contracts let a provider’s CI catch changes that would break its consumers.
Canaries compare a small slice of traffic against the baseline before promoting; blue-green switches all at once.
Feature flags separate deploying code from releasing features.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: simulate microservice patterns in Python
Build small Python simulations of the patterns - routers, sagas, outboxes, circuit breakers, traces - and run them against sample inputs. They run locally in your browser; no servers or containers needed.
Run consumer contracts
The input has consumer contracts, ---, then the provider’s actual response as JSON. Each contract line is consumer: field:type field:type ... with types int, str, float, bool or list; nested fields use dots, like price.cents:int.
For each consumer print ordering: PASS or kitchen: FAIL (missing spicy_level; price.cents is str, expected int), listing problems in contract order. Then provider can deploy: yes or no. Extra fields in the response never matter.
- A breaking menu change
- Safe to ship
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Automated canary analysis
The first input line is the baseline error_rate p95_ms. Each following line is a canary stage traffic% error_rate p95_ms. Promote through the stages while the canary’s error rate is at most baseline + 0.5 points and its p95 latency is at most 110% of the baseline; print 5%: ok. At the first failing stage print 25%: ROLLBACK (error rate 1.2 > 0.9) or (p95 260 ms > 231 ms) - errors are checked first - and stop. If every stage passes, print promoted to 100%.
- Latency regression
- Error spike
- Clean rollout
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…