Running a Beta Program: Recruiting, Managing Feedback, and Graduating Users to GA

This is a new topic for this series, though it picks up naturally from where our MVP development guide leaves off — that guide covers building and shipping a first version; a structured beta program is often the real step between having an MVP and calling a product generally available. Our product strategy frameworks guide covers general prioritization discipline that applies once beta feedback starts arriving in volume, though this guide covers something more specific: the real mechanics of recruiting the right testers, structuring how their feedback actually gets collected, communicating instability honestly, and deciding — with real, checkable criteria rather than a vague feeling — when a beta is actually done.

Gmail's Real Five-Year Beta

Gmail's launch itself is a real, well-documented event worth knowing precisely: announced April 1, 2004, which caused widespread initial disbelief that it was a real product rather than an April Fools' joke, given Google's history of releasing joke products on that date (Columbia University Library, “Today In History: Google Launches Gmail”). Gmail remained invite-only for roughly its first three years before opening to public sign-up in February 2007, still carrying the beta label, and finally exited beta on July 7, 2009 — the same day Google Calendar, Google Docs, and Google Talk also dropped the label as part of a broader push around Google Apps (Google, “Gmail leaves beta, launches ‘Back to Beta’ Labs feature,” Official Gmail Blog, July 7, 2009). Contemporaneous coverage from Slate specifically examined why Google kept the label for so long, noting the practical reason wasn't that the product was unfinished by the time businesses started adopting it — it was that enterprise procurement processes at many companies formally prohibited adopting “beta” software, which eventually made removing the label a real business necessity rather than a cosmetic choice (Slate, “Why Google kept Gmail in ‘beta’ for so many years,” July 2009).

The invite mechanic itself is worth describing carefully and honestly: existing users could invite others in limited batches, and Google adjusted and periodically replenished how many invites a given user had over the roughly three years the product was invite-gated, partly to manage server capacity during a real period of rapid growth and partly to slow spam-account creation. It's worth being direct that this guide could not verify a single, fixed number of invites every user received at any given moment — secondary accounts describe the number changing over 2004–2006, and no single official Google source gives one canonical figure, so no specific invite count is presented here as a fixed fact. The invite scarcity itself, real and well-documented independent of the exact count, is the detail worth taking from this case: constrained access created a genuine sense of exclusivity around a product still being actively refined, a dynamic several later companies have replicated deliberately.

Invite-only versus fully open betas

Gmail's specific choice — invite-only for roughly three years before opening to public sign-up — is worth examining as one real, concrete point on a spectrum a founder running a beta today still has to choose a position on. An invite-only or otherwise gated beta gives a team direct, ongoing control over exactly who gets in, which is precisely the mechanism that makes Des Traynor's “focused group experiencing the same problem” recruiting criterion actually enforceable in practice — a team reviewing and approving each individual beta applicant can check that criterion directly, applicant by applicant, before granting access. A fully open beta, by contrast, sacrifices that direct control in exchange for a much larger, faster-growing volume of testers and feedback, at the real cost of a higher share of that feedback coming from people who don't actually match the focused profile the recruiting criterion describes.

The practical decision, in the absence of a single correct universal answer, comes down to which failure mode a specific team can tolerate less. A team with limited capacity to process and act on feedback — the common situation for an early-stage startup with a small team — is generally better served by the gated approach, since it can't meaningfully act on a flood of feedback from an unfiltered open beta even if that feedback arrives faster. A team with more capacity to triage volume, or one specifically trying to stress-test a product under real-world load rather than collect qualitative feedback, may reasonably prefer the openness and scale a fully public beta provides. Gmail's own multi-year invite-gated period, running during a phase when Google was managing genuinely constrained server capacity in addition to wanting focused feedback, illustrates that the gating decision can serve more than one purpose simultaneously — it isn't only a recruiting-quality lever, it can be a real operational and capacity lever as well.

Recruiting the Right Beta Testers

Traynor's piece, “Running Closed Betas: Which Users, and How Long?,” published on the Intercom Blog on March 30, 2012, is worth reading directly for its specificity (Des Traynor, “Running Closed Betas: Which Users, and How Long?,” Intercom Blog, March 30, 2012). His central argument on recruiting cuts directly against the instinct to open a beta as widely as possible to maximize feedback volume: a broad, unfocused group produces feedback scattered across too many different underlying problems and use cases to actually act on coherently, while a narrower, genuinely focused group experiencing the same real problem produces feedback a team can synthesize into concrete, shippable decisions. This is a directly practical, checkable recruiting criterion worth applying before opening any beta: does this specific candidate tester actually have the problem this product solves, right now, or are they simply interested in trying something new? Only the former group produces the kind of feedback a beta program actually needs.

A related, real, well-documented recruiting channel worth naming: Superhuman, the email-client company, built its beta and early-access cohort from an enormous public waitlist rather than open sign-up — reported by TechCrunch to have exceeded 275,000 people by February 2020 (TechCrunch, “Superhuman CEO Rahul Vohra on waitlists, freemium pricing and future products,” February 28, 2020). It's worth being precise that this figure describes the company's broader waitlist at a specific later date, not a confirmed count for its earliest beta cohort specifically — this guide could not verify a separate, dated figure for that earlier phase, so it isn't presented as one. What is verifiable and useful regardless of the exact early number: a waitlist, built from genuine organic interest rather than paid acquisition, gives a founder a real, pre-qualified pool to recruit a focused beta cohort from — exactly the kind of “people who already want this” population Traynor's recruiting criterion describes.

Existing customers versus entirely new testers

A related, practical recruiting decision worth naming directly, since it applies to any product that already has some existing customer base before a new beta begins: whether to recruit beta testers from that existing base, from entirely new prospective users, or some deliberate mix of both. Recruiting from an existing customer base carries a real advantage Traynor's own criterion points toward directly — these are people already demonstrated to have the underlying problem the company solves, which means less recruiting effort is spent confirming relevance and more can go directly into collecting substantive feedback. It carries a real, corresponding risk worth naming honestly: existing customers who already like the current product may evaluate a new beta feature relative to what they're used to, rather than fresh, which can produce feedback skewed toward incremental refinement of the familiar rather than a genuinely fresh read on whether a new feature or product direction actually works on its own terms. Recruiting entirely new testers avoids that particular bias but reintroduces the relevance-screening work Traynor's criterion is meant to solve in the first place. Neither approach is universally correct; the practical decision depends on whether the beta is testing a genuinely new capability (where fresh eyes matter more) or refining something an existing customer base already uses regularly (where their informed, comparative perspective is exactly what's valuable).

Structuring Feedback Collection

Vohra's account, published in First Round Review under the title “How Superhuman Built an Engine to Find Product Market Fit,” is worth reading in full for its specific methodology (Rahul Vohra, “How Superhuman Built an Engine to Find Product Market Fit,” First Round Review). The method's specific insight, worth taking directly: the “very disappointed” segment is already satisfied and doesn't reveal what's missing, while the “not disappointed” segment often represents a poor product-market fit that no amount of feature work will fix — the “somewhat disappointed” middle segment is where real, actionable signal lives, since these are users who see genuine value but haven't yet gotten what they need to become fully committed. Vohra describes building what the piece calls a roughly 50/50 roadmap from this data: half the roadmap addresses what would most increase the “very disappointed” response if the product went away (deepening the core value for people already convinced), and half addresses the specific blockers the “somewhat disappointed” segment names directly.

The practical translation for a founder running a beta specifically, rather than an already-launched product refining fit: this same segmentation question can be asked directly of beta testers on a recurring basis, not just once. A beta cohort that skews heavily “not disappointed” early on is a real, actionable signal that the recruiting criteria covered above may need revisiting — the wrong people may have been let into the beta — rather than a signal that the product itself needs more features. A cohort skewing toward “somewhat disappointed” is closer to the ideal beta population: people who genuinely have the problem and can articulate, specifically, what's still missing.

Communicating Instability to Beta Users

This language comes directly from Stripe's own Services Agreement, a real, current, live legal document rather than a marketing summary (Stripe Services Agreement, General Terms), and it's worth reading it as a real example of how a well-known, technically sophisticated company sets expectations formally and explicitly rather than leaving them implicit. The agreement also states that users of Preview services are expected to provide “timely Feedback…in response to Stripe requests” — making participation in feedback collection an explicit, stated expectation of beta access, not an optional courtesy left to a tester's own initiative. This is a genuinely useful structural detail for a smaller company's own beta terms to borrow directly: stating plainly, in writing, both what a beta user should expect (instability, incomplete features, no production guarantee) and what's expected of them in return (actually giving feedback when asked) sets a real, mutual, two-way expectation rather than a one-directional disclaimer.

The practical lesson for a smaller team without Stripe's legal resources is the underlying discipline, not the specific legal language: write down, in plain terms a non-technical beta user can actually understand, what “beta” means for this specific product — what might break, what's not yet built, whether data created during the beta will carry over to the general release — and share it before someone joins, not apologetically after something breaks. A beta user surprised by instability they were never warned about reasonably reads it as a quality problem; the same instability, disclosed honestly in advance, reads as exactly what a beta is supposed to be.

Handling Beta User Disengagement

A practical, frequently underestimated problem worth naming directly: a real share of people who join a beta will stop actively participating well before the program ends, regardless of how carefully they were recruited. This isn't necessarily evidence the recruiting criteria covered earlier failed — a genuinely relevant tester can still disengage simply because using an unfinished, occasionally broken product takes real, ongoing effort that competes with everything else in their day, and that effort naturally fades once the initial novelty of early access wears off. The practical mistake worth avoiding is treating silence from a beta tester as a neutral, uninformative absence of feedback rather than a real signal worth investigating directly. A tester who stopped logging in three weeks into a beta is communicating something — that the product didn't clear the bar for continued, voluntary effort — even if they never say so explicitly in a support ticket or survey response.

The practical discipline worth building specifically to catch this: track actual usage among beta participants, not just who initially signed up or agreed to join, and treat a meaningful drop-off in active engagement as a trigger for direct, personal outreach rather than a number to note passively in a dashboard. A short, genuinely curious message asking what stopped a disengaged tester from continuing often surfaces exactly the kind of blocking, disqualifying feedback the Superhuman-style segmentation question covered above is designed to surface from active users — except this feedback comes from people who never got far enough to answer a survey about it. Treating disengagement data with the same seriousness as active feedback, rather than as an unfortunate but uninformative attrition number, closes a real gap most beta programs leave open by only listening to the testers who are still actively talking.

Exit Strategies: Scope, Time, and Budget

It's worth being direct about why the third option fails specifically, since it's the one a team drifts into by default when it hasn't deliberately chosen one of the first two. A team with no explicit scope or time commitment tends to keep finding one more thing worth fixing before calling the beta complete, which is a reasonable instinct in isolation but produces an indefinitely extending beta with no real endpoint, until external pressure (running out of money, a competitor shipping first, team fatigue) forces an exit under worse conditions than a deliberately chosen one would have. Choosing explicitly between scope and time, before a beta even begins, forces the harder but more useful conversation upfront: either agree on the specific, minimum feature set that constitutes “done,” or agree on the calendar date the beta ends regardless of what's still unfinished at that point. Either produces a real, plannable endpoint; drifting toward exhaustion produces neither.

Choosing between scope and time for a specific situation

It's worth going a level deeper on how a team should actually choose between Traynor's two legitimate strategies, since the right choice genuinely depends on the specific situation rather than one strategy being universally preferable. Exiting via scope fits best when the remaining work is genuinely well-understood and boundable — a team can specifically name the handful of features or fixes that constitute “done,” and reasonably estimate when that defined list will actually be complete. It fits poorly when the beta's own feedback keeps surfacing genuinely new, unanticipated requirements — in that situation, a scope-based exit keeps sliding indefinitely as the list keeps growing, which is functionally indistinguishable from the budget-based failure mode Traynor warns against, just dressed up as if it were principled. Exiting via time fits best precisely in that situation — when the team can't confidently bound the remaining scope in advance, but does have a real, external reason a decision needs to be made by a specific date (a fundraising milestone, a competitive launch pressure, a committed customer waiting on the general release). The practical test worth applying honestly: if a team can't currently write down the specific, bounded feature list that would constitute “done,” a scope-based exit isn't actually available to them yet, no matter how much they'd prefer it — a time-based exit, even though it feels less satisfying, is the more honest choice.

Graduation Criteria: Real Launch-Stage Definitions

This is real, current, directly verifiable documentation, not a paraphrase (Google for Developers, “Google Maps Platform launch stages”). The specific, checkable distinctions worth taking from Google's own definitions: Experimental releases “typically last up to 12 months” and are meant for test environments only, not production use of any kind; Preview releases are “not necessarily feature-complete” and still carry no SLA or support commitment, though Google states Preview offerings are “typically expected to reach GA within 12 months”; and General Availability means the product is “production ready,” covered by Google's actual terms of service, SLA, and support guidelines. Google is also explicit that GA doesn't always mean universal availability — a GA release can still be limited to a specific group of customers while still carrying the full support and SLA commitments that come with the label.

The practical, transferable graduation criterion worth borrowing directly from this real, published framework, regardless of a company's size: a product should only graduate to a “generally available” label once it can genuinely support the concrete commitments that label implies — a real support process, a real (even if informal) reliability standard, and a real willingness to stand behind the product for production use. A team calling a product “GA” simply because the beta has run for a while, without having built the actual support and reliability infrastructure the label implies to customers, is making the same mistake Gmail's multi-year beta specifically avoided: promising something the underlying operational reality doesn't yet back up.

Internal testing before any external beta begins

A related practice worth naming explicitly, since it sits chronologically before any of the external recruiting or feedback-collection steps covered above: internal testing, sometimes called “dogfooding,” where a company's own employees use a product or feature before any external tester sees it. The practical value this adds specifically, distinct from what external beta testers provide, is speed and directness — an internal team can catch a genuinely broken workflow or an obviously confusing interaction within hours of a build going out, feedback that would otherwise take days to surface through an external beta's slower recruiting, onboarding, and reporting cycle. It also protects a company's limited supply of genuinely relevant external testers, per Traynor's own recruiting criterion, from being spent discovering the kind of obvious, easily internally-caught defects that don't actually require an outside perspective to find.

It's worth being direct about what internal testing can't substitute for, though: an internal team, by definition, already understands the product's intended design and context in a way a genuinely new external user doesn't, which means internal testers systematically miss the exact category of confusion a first-time, unfamiliar user experiences. The two practices are complementary, not interchangeable — internal testing catches obvious breakage fast and cheaply, before it ever reaches an external tester whose time and goodwill are limited resources; external beta testing, recruited and structured per the criteria covered throughout this guide, is what actually reveals whether a genuinely new user without the team's own context can understand and get value from the product on their own. A team that only dogfoods internally, and never runs a real external beta before calling something generally available, is skipping the specific kind of validation internal testing structurally cannot provide.

A Real Example: Notion's Custom Agents Beta

Notion's post, “What we learned during the Custom Agents beta,” published on Notion's own blog, is a real, current, named example worth knowing specifically because it's recent and directly on-topic (Notion, “What we learned during the Custom Agents beta”). The post documents the beta directly informing what shipped at general availability, rather than the beta period being treated as a formality before an already-finalized feature launched unchanged — a real, current, first-party confirmation that the underlying discipline this guide covers (recruit genuinely relevant testers, collect and act on their feedback, use that feedback to inform what actually ships) is still standard practice at a well-known, current product company, not a historical artifact from Gmail's era.

It's worth naming what makes this specific case a genuinely useful, current data point rather than just a restatement of the same principle Gmail already illustrated two decades earlier: Custom Agents is an AI-powered feature, exactly the category our AI feature integration guide covers as carrying its own specific, distinct risks — non-deterministic output, real cost unpredictability, a faster deprecation cycle than typical software. A beta period is arguably more valuable, not less, for exactly this category of feature, since the real-world variability an AI feature exhibits under genuine, varied user input is precisely the kind of thing a controlled internal test environment struggles to surface fully before real, diverse users interact with it. Notion choosing to run a real, documented beta specifically for this kind of feature, rather than shipping it directly to general availability, is a direct, current illustration of the same underlying judgment call this guide argues for throughout: a feature carrying genuine uncertainty about how it will actually perform once real users depend on it is precisely the kind of feature that benefits most from the structured, deliberate beta process this guide describes, not the kind that can safely skip it.

What This Guide Could Not Verify

Consistent with the standing rule across this series, it's worth naming directly several categories of claim that circulate widely in beta-program marketing content but that this guide's research could not trace to a credible, verifiable primary source:

1. 1

   A specific "right" number of beta testers

   Widely repeated figures like "200-300 testers" appear across unattributed marketing-blog content with no named author or underlying study — no credible, named research establishing a specific sufficient number was located.

2. 2

   Specific beta program length benchmarks

   Claims that "most beta programs run 6-12 weeks" or similarly specific durations circulate without a traceable, credible, named source — Gmail's own multi-year beta is direct, real evidence against treating any single duration as a universal norm.

3. 3

   Beta-tester recruiting conversion-rate statistics

   This guide could not verify a specific, credible, named source for what percentage of waitlist signups or invited users typically become active beta testers.

4. 4

   A single fixed Gmail invite count

   Secondary accounts describe the number of invites Gmail users received changing over 2004-2006, with no single official Google source giving one canonical figure — this guide names the invite mechanic qualitatively rather than citing a specific, fixed number.

As with every prior guide in this series, the reason for naming these gaps explicitly, rather than silently omitting the topic or repeating an unsourced figure because it circulates widely, is the same: a plausible-sounding statistic is not the same as a verified one, and a founder making a real decision deserves to know exactly which guidance in this guide rests on a real, named source and which topics simply don't have one available yet.

A Practical Framework

Bringing the research above together into an actual sequence for a founder running a structured beta program for the first time:

None of this requires Google's formal launch-stage infrastructure or Stripe's legal team to apply at a small scale. A team of two or three can write a one-paragraph, plain-language beta disclosure, ask Vohra's single segmentation question in a simple recurring survey, and commit in advance to either a feature list or a calendar date as the real exit condition. What separates a beta program that produces genuinely useful signal from one that just delays a launch indefinitely isn't scale or resources — it's whether these decisions were made deliberately, in advance, the way every real case in this guide made them, rather than drifting by default toward Traynor's explicitly named failure mode.

Frequently Asked Questions

Every real case in this guide — Gmail's five-year beta, Intercom's recruiting and exit-strategy guidance, Superhuman's feedback segmentation, Stripe's Preview disclosures, Google's launch-stage definitions, Notion's recent Custom Agents beta — shares one underlying trait worth closing on directly: none of them treated “beta” as a vague, indefinite holding pattern. Each had a real, specific reason for the label, a real plan for who was in it and why, and a real, deliberate answer for when and how it would end. A beta program without those three things isn't really a beta program at all — it's just an unfinished product wearing a label that implies more intention than actually exists behind it.

The label itself carries no inherent value — calling something “beta” doesn't automatically buy a team goodwill for shipping something unfinished, and removing the label doesn't automatically make a product production-ready. What each real case in this guide actually demonstrates is that the label is only as meaningful as the real, deliberate process standing behind it. A founder running a first structured beta doesn't need Google's scale, Stripe's legal team, or Gmail's five years to get real value from this discipline — the same underlying decisions, made deliberately and in writing rather than left implicit, are exactly what separate a genuinely useful beta from a product quietly sitting in an unfinished, indefinitely extended holding pattern with a reassuring label attached to it.