Experiment Table exists because App Analytics made it easy to ship a test and left teams alone with the conversation that follows. A director asks whether it “won.” Someone opens a dashboard on a laptop whose brightness is fighting the window. A p-value appears like a weather report.
We teach a different opening: state the decision the test was meant to inform, state what would have counted as a change of mind, then show the number. If you cannot name the decision, you ran a decoration.
Non-results are information
A shrug is allowed. It means the change was too small to see at this sample, or the metric was too noisy, or the idea did not matter in the way the team hoped. All three are useful. What is not useful is hunting a secondary metric until something twitches. That is how a product accumulates folklore about buttons that “probably helped.”
Write the secondary metrics down before the test starts if they are truly guardrails — crashes, refunds, time-to-job. Guardrails can stop a ship. They should not be promoted into winners after the fact.
Peeking is a social habit
People peek because meetings exist on Tuesdays. Sequential testing methods exist; so does the simpler discipline of not putting a live experiment on the weekly pack until the pre-declared sample arrives. In United Kingdom product teams we have watched peeking become a status ritual: the person who checked this morning gets to speak first. Experiment Table treats that as a process bug, not a personality flaw.
What to bring into the room
One chart of the primary metric with intervals. One table of guardrails. One sentence on who was excluded (internal users, a geo, a broken assignment). One recommendation that a sceptic could disagree with. If your App Analytics tool offers a “significance badge,” leave it in the tool. Badges short-circuit argument.
If you want practice, the sitting is listed on programmes. Bring a test you already ran, including the ones that embarrassed you.