Feature Flags and Progressive Rollouts: Shipping Software Safely

Why Flags Deserve More Rigor Than “If This, Do That”

This guide covers the real, well-specified taxonomy of feature flags, how real, named companies (Microsoft, LaunchDarkly, Etsy) actually implement progressive rollouts, a real, precisely documented disaster caused by treating a flag's dormant code path carelessly, and a real, vendor-neutral standard (OpenFeature) now backed by the Cloud Native Computing Foundation. It connects to our guide on incident response and on-call, which covers what to do once something has already gone wrong in production; this guide covers the upstream discipline that prevents a flag-related incident from happening in the first place.

Every claim in this guide is anchored to a real, named, dated, and independently verifiable source: Martin Fowler's own published writing, Microsoft's own developer documentation, the SEC's own public administrative order, and the CNCF's own project records. Where the underlying research could not confirm a specific detail to that standard — a precise publication date, an unattributed industry term, a secondhand figure from a company that has not itself published it — that gap is named explicitly in this guide's own dedicated section on what it could not verify, rather than filled in with a plausible-sounding guess.

The Real Flag Taxonomy: Four Kinds, Four Lifecycles

Hodgson's piece, published on Martin Fowler's own site, draws a precise distinction that is easy to blur in casual usage: a release toggle hides an in-progress feature from users while the code ships to production incrementally, and is meant to be short-lived — removed once the feature is fully released. An experiment toggle supports A/B testing, routing different users to different code paths to measure an effect, and is also meant to be temporary, retired once the experiment concludes. An ops toggle gives operators a runtime kill switch over a feature's behavior and may legitimately live much longer, sometimes permanently, if it protects against a real operational risk. A permissioning toggle restricts a feature to specific user segments (for example, a premium tier), and can also be long-lived, since the segmentation itself is a permanent product decision rather than a temporary rollout mechanism.

The distinction matters directly for how a team manages its own flags: treating every toggle as if it has the same expected lifespan is exactly how a codebase accumulates the kind of forgotten, dormant flag path that caused Knight Capital's incident, covered in detail below. Hodgson's article is explicit that flags are a tool with a real cost, not a free abstraction — it recommends favoring small, frequent releases over flag-gated feature branches wherever practical, and treating toggles as a deliberate, last-resort technique rather than a default habit.

Percentage Rollouts and Progressive Rollouts

This distinction is worth being precise about, since the two terms are often used loosely as synonyms. A percentage rollout is the underlying mechanism — a way of splitting traffic deterministically by percentage. A progressive rollout is a policy layered on top of that mechanism: instead of a human deciding to move a flag from 5% to 25% to 100% at three separate points in time, the platform advances the percentage automatically according to a schedule the team defines up front, often paired with automatic rollback if error rates or other signals cross a defined threshold during the ramp. LaunchDarkly also documents targeting rules as a separate, complementary mechanism — rules that match specific user or context attributes (for example, a beta-tester group, or a specific geographic region) to a particular flag variation, independent of the percentage-based rollout logic.

Microsoft's Real Deployment Rings

The ring model formalizes the same underlying idea as a percentage rollout — exposing a change to an increasing share of a population over time — but organizes it around named, purpose-built cohorts rather than an arbitrary percentage curve. Per Microsoft's own documentation, the Test ring is intended for a small number of dedicated devices, often used specifically to catch problems before they reach any real end users; the First ring extends exposure to a modest, still-controlled population; the Fast ring covers half of all devices, front-loading exposure once a change has cleared the earlier, smaller rings; and the Broad ring covers the remaining devices last, once a change has already demonstrated stability across the earlier three. Separately, older reporting on Microsoft's Office 365 update process (via ITProToday) described a related but distinct five-ring internal-to-external model — worth noting as a related but not identical real Microsoft practice, since the exact ring count and percentages differ from the Windows Autopatch model documented above and should not be conflated with it.

How Real Companies Actually Roll Out Features

Etsy's own “Feature” project (published on github.com/etsy) is a real, directly verifiable primary source: a company built and open-sourced its own internal rampup and A/B-testing flagging API rather than treating flag infrastructure as an afterthought, underscoring that this is genuine, purpose-built engineering infrastructure at real, high-traffic companies rather than a niche practice. Shopify's own engineering blog, published at shopify.engineering, has similarly documented using a beta-access model to deploy new features to a controlled population before a general release, consistent with the permissioning-toggle category covered above. It is worth noting one limit in this guide's research directly: secondary sources describe Meta's internal “Gatekeeper” feature-flag system using specific figures (for example, dialing exposure up in small increments from an internal-only rollout to full release), but this guide could not locate a primary Meta or Facebook engineering blog post as the source for those specific figures — see the dedicated section below on what this guide could not verify before treating any exact Gatekeeper percentage as a settled fact.

Trunk-Based Development and Why Flags Enable It

The practice is documented in detail at trunkbaseddevelopment.com, a real, dedicated reference site maintained by Paul Hammant covering the technique's patterns and rationale. The core argument for trunk-based development is that long-lived feature branches accumulate merge conflicts and integration risk the longer they diverge from the shared trunk, and that merging small, frequent changes directly into trunk avoids that cost entirely. The obvious problem this creates is also the obvious reason flags matter here specifically: if a team merges every change directly into the branch that ships to production, how does an in-progress, unfinished feature stay hidden from real users until it is actually ready? A release toggle, in the taxonomy covered earlier in this guide, is precisely the answer — the code for a new feature can be merged, deployed, and even running in production, entirely invisible to real users, simply by keeping its release toggle off until the feature is complete.

This is worth stating directly as the deepest real connection between feature flags and modern software delivery practice: flags are not merely a convenience layered on top of an otherwise ordinary branching workflow, they are a structural precondition for trunk-based development to function at all once a team has more than a trivial number of engineers working on features of varying size and duration simultaneously. A team that adopts trunk-based development without a working flag discipline will, in practice, either ship unfinished features accidentally or quietly reinvent long-lived branches under a different name — neither of which achieves what the practice is meant to deliver.

A Real, $460 Million Lesson in Flag Hygiene

The SEC's order describes the mechanism in exact technical detail, and it is worth reproducing precisely rather than paraphrasing loosely, since the precision is the whole lesson. To support a new NYSE Retail Liquidity Program, Knight deployed new order-routing code (“RLP code”) to eight servers running its SMARS order router, intending the new code to replace old, unused code for a function called Power Peg that Knight had stopped using in 2003. The new RLP code repurposed a flag that had formerly activated Power Peg, so that when the flag was set, the new RLP functionality — not Power Peg — would run instead. Knight intended to delete the old Power Peg code entirely, but had not done so by the time of deployment.

During the deployment, one of Knight's technicians failed to copy the new RLP code to the eighth of the eight servers, and no second technician reviewed the deployment to catch the gap — the SEC order states plainly that Knight had no written procedure requiring such a review. On the morning of August 1, orders sent with the repurposed flag to that eighth server triggered the still-present, dormant Power Peg code instead of the intended new functionality. Because a separate change in 2005 had moved Power Peg's cumulative-share-tracking logic earlier in the code sequence — without Knight retesting Power Peg after that change — the dormant code no longer correctly recognized when a parent order had been completely filled, and continuously sent new child orders in rapid succession regardless.

Per the SEC's order, the result was over 4 million executions across 154 stocks totaling more than 397 million shares in roughly 45 minutes, leaving Knight with an unintended net long position of approximately $3.5 billion in 80 stocks and a net short position of approximately $3.15 billion in 74 stocks. Knight ultimately realized a loss of more than $460 million. The SEC order also documents a missed early warning: Knight's own systems generated 97 automated “BNET reject” e-mail messages before the market even opened, referencing an error described internally as “Power Peg disabled” — but these were not designed as system alerts, were not reviewed by Knight personnel in real time, and were not used to diagnose the problem once trading began. The SEC ultimately censured Knight, imposed a $12 million civil penalty, and required Knight to retain an independent consultant to review its software deployment and risk-control processes.

It is worth being precise about what this incident is, and is not, an example of. Power Peg was not a modern SaaS feature flag in the LaunchDarkly or Gatekeeper sense — it was an internal, repurposed activation flag inside a proprietary trading system, predating today's dedicated feature-management tooling. But the underlying failure mode is exactly the one every flag taxonomy above exists to prevent: dormant, unretired code left reachable behind a flag, reactivated by a change to that flag without adequate testing, review, or a fast kill switch to stop it once triggered. Deleting Power Peg entirely, rather than leaving it dormant and repurposing its flag, would have made the entire incident structurally impossible.

Ops Flags as Kill Switches and Circuit Breakers

These two techniques solve related but distinct problems. An ops toggle is typically a manually or automatically triggered switch that disables a specific feature or code path entirely, used when a team has identified a specific feature as the source of a problem and wants to remove it from the system without waiting for a new deploy. A circuit breaker, in the sense popularized by Nygard's book and implemented in Netflix's Hystrix, is a narrower, more automatic pattern specifically for calls to a downstream dependency: after a defined threshold of failures, the breaker “trips” and stops sending further calls to the failing dependency for a cooldown period, protecting the calling system from cascading failure. Knight Capital's incident is, again, instructive by omission: the SEC order notes explicitly that Knight had no procedure or automated control to halt SMARS's operations in response to its own aberrant activity — precisely the kind of automatic, threshold-triggered stop a well-implemented kill switch or circuit breaker is designed to provide.

Feature Flags vs. Canary Releases vs. Blue-Green Deployments

It is worth being precise about the layer each technique actually operates on, since the three are easy to conflate under the general heading of “safe deployment.” A feature flag lives inside the application's own code: the same running process serves both the old and new behavior, and a conditional statement decides which one a given request receives. A canary release operates at the infrastructure layer: a small number of new server instances, running the new version of the code, are brought online alongside the existing fleet, and a load balancer directs a controlled share of real traffic to them, letting a team observe real production behavior on the new version before committing fully. A blue-green deployment also operates at the infrastructure layer, but as a binary switch rather than a gradual ramp: two complete, independent environments (conventionally labeled “blue” and “green”) run side by side, one live and one idle, and a deployment cuts traffic from one to the other all at once, with the previous environment kept warm as an immediate rollback target.

The three techniques are complementary rather than substitutes for one another, and the strongest real-world safety net typically combines them: a canary release limits the blast radius of a bad new binary to the fraction of traffic hitting the canary instances; a feature flag inside that same binary gives finer-grained control over which specific behavior is active, independent of which instance served the request; and a blue-green setup gives a team a near-instant, whole-environment rollback path if a problem is discovered after a canary has already looked healthy. Knight Capital's incident, covered above, is a useful stress test for this framework: it involved neither a canary release nor a blue-green environment, and its own internal flag mechanism failed silently on exactly one server out of eight — a scenario a canary-style controlled rollout, paired with automated output monitoring, is specifically designed to catch before it reaches full production traffic.

The Testing Cost Flags Actually Add

This combinatorial cost is a direct, mechanical consequence of what a flag actually is: a branch point in the code, and every additional branch point multiplies the space of possible runtime states the application can be in. A single flag means a piece of code has two possible states to reason about, on or off. Two flags mean four possible combined states. This is precisely why the flag taxonomy covered earlier in this guide is not just an organizational nicety: a release toggle or experiment toggle that is only ever meant to exist for a few weeks is a temporary, bounded addition to this combinatorial space, while an ops toggle or permissioning toggle left in place indefinitely is a permanent addition to it, multiplying against every other long-lived flag already in the system. A codebase with a dozen long-lived, permanent flags has, in principle, thousands of distinct behavioral states, the overwhelming majority of which no test suite and no human developer has ever actually exercised or reasoned through directly.

The practical mitigation is not to avoid flags, but to bound this cost deliberately: keep the number of simultaneously active, long-lived flags as small as realistically possible; retire release and experiment toggles immediately once their purpose is served, exactly as Hodgson's taxonomy recommends; and treat any two flags that can plausibly interact with each other's behavior as a combination that needs its own explicit test, rather than assuming that two individually passing test suites imply their combination is also safe. This is the same underlying discipline as the technical-debt guidance below, applied to a more concrete, measurable cost: every flag a team chooses not to remove is not just a line of code sitting idle, but an active multiplier on the total behavioral surface area the system as a whole must be correct across.

The Real Technical-Debt Risk of Flags Left Behind

This is worth taking seriously as a distinct risk from the Knight Capital scenario above, even though both trace back to the same root cause: unretired code paths. Knight's incident involved dormant code being unexpectedly reactivated. The technical-debt risk Hodgson describes is subtler and more common: flags that are simply never cleaned up, accumulating indefinitely, each one adding a branch of conditional logic that every future engineer reading that code has to understand, even for flags whose original purpose no one remembers. LaunchDarkly's own product documentation addresses this directly with flag-status and stale-flag-detection tooling, reflecting that this is a widely recognized, real operational problem for any team running flag infrastructure at scale, not a hypothetical concern.

The practical discipline Hodgson's article recommends is straightforward to state and consistently hard to follow under deadline pressure: treat every flag as having an expected removal date at the moment it is created, and treat removing a flag's now-permanent branch of logic as a required, non-optional step of shipping the feature it gated — not a cleanup task to get to later. A release toggle or experiment toggle that outlives its original purpose by months is not evidence of a robust flagging system; per Hodgson's own framing, it is exactly the failure mode the taxonomy above exists to help a team avoid.

OpenFeature: A Real, Vendor-Neutral Standard

OpenFeature's goal, per its own backing consortium's public materials, is to give teams a single, vendor-neutral API for evaluating feature flags, so that application code does not need to be rewritten if a team switches feature-flag providers later. This is a meaningfully different kind of standardization than anything else covered in this guide: rather than standardizing rollout percentages or ring structures, it standardizes the interface application code uses to ask “is this flag on for this user,” independent of which vendor's backend actually stores and evaluates that flag. The project's SDKs for Java, JavaScript, .NET, and Go reached 1.0 general availability as part of this same trajectory. A real, CNCF-backed, multi-vendor consortium standardizing this interface is itself a signal of how mainstream and consequential feature-flag infrastructure has become across the industry, not a niche tooling concern.

Writing Flag-Aware Code: Practical Guidance

Each of these is a direct, practical response to a failure mode covered elsewhere in this guide. Naming a flag descriptively (for example, a name reflecting the actual feature and its rollout stage, rather than an opaque ticket number) directly addresses the technical-debt risk above — a flag nobody can identify by name is a flag nobody will confidently remove. Recording an owner and an expected removal date at creation time is the single most direct countermeasure to Hodgson's “pile of toggles” warning, since it converts an implicit, easily forgotten expectation into an explicit, trackable one. Keeping the conditional logic itself thin, delegating to clearly named functions for each branch, keeps the combinatorial testing cost covered above legible rather than tangled — a codebase where every flag check is a single, clearly named function call is far easier to reason about than one where flag logic is interleaved directly into a large function's control flow. And defaulting every flag to its safe, off state specifically addresses the exact failure pattern in Knight Capital's incident: a flag or configuration value that is unset, misconfigured, or fails to load should never silently fall back to an unintended, active code path.

What This Guide Could Not Verify

Consistent with the standing rule across this series, it is worth naming directly the specific claims this guide's research could not confirm to a standard it is comfortable presenting as settled fact:

1. 1

   Exact original publication date of the Fowler/Hodgson feature-toggle article

   Different retrieval methods returned conflicting signals for the article's original publication date on martinfowler.com. This guide deliberately avoids stating one specific original date as settled fact, and instead cites the article by its real, verifiable authorship (Pete Hodgson) and publisher (martinfowler.com) without an unverified exact date.

2. 2

   Precise mechanics of Meta's internal Gatekeeper system

   Secondary sources describe specific rollout percentages and increment sizes for Meta's Gatekeeper system, but this guide could not locate a primary Meta or Facebook engineering blog post as the source for those specific figures, so no exact Gatekeeper percentage is asserted here as a directly sourced fact.

3. 3

   Origin of the terms "canary deployment" and "dark launch"

   Both are real, standard, widely used industry terms, but this guide could not identify a single, definitive named company or publication that originated either term as applied to software releases, so no specific origin is claimed for either.

4. 4

   Exact ring count and percentages for Microsoft's Office 365 rollout process specifically

   This is a separate, older Microsoft rollout model from the Windows Autopatch rings covered and charted above, sourced in this guide's research to secondary reporting (ITProToday) rather than a Microsoft primary document this guide directly confirmed, so its exact ring count and percentages are described only generally, not charted or quoted as precise figures.

A Practical Framework

None of this requires adopting an enterprise feature-flag platform on day one. A small team can implement the same real discipline with a simple, self-managed configuration flag, as long as it follows the same underlying principles: know which of the four real categories a flag belongs to, roll it out gradually rather than all at once, give a human or an automated system a fast way to turn it off, and remove it once its purpose is served. Knight Capital's $460 million lesson was not that flags are dangerous — it was that a dormant, unretired, and unreviewed code path behind a repurposed flag is dangerous, and that danger is entirely preventable with the ordinary discipline this guide describes.

Frequently Asked Questions

Every taxonomy, percentage, and incident detail in this guide traces to a real, named, dated source — Martin Fowler's own site, Microsoft's own documentation, the SEC's own public order, and the CNCF's own project records — and every place this guide's research hit a genuine limit, that limit is stated directly rather than papered over with an invented detail. Feature flags are a real, precisely specified engineering discipline, and the cost of treating them carelessly is not hypothetical: it is a matter of public record, denominated in hundreds of millions of dollars.