— views

Blogchain: Goal → Expectations → Updates

Impact Expectations. As a goal owner you're like the captain of a ship, steering toward your goal. Along the way you gauge the position of the stars so that the moment you veer off course, you can inform the crew and make adjustments to get back on course. The longer it takes to spot an unexpected detour, the harder it will be to recover.

We expect teams to provide updated impact expectations every 2 weeks in order to spot those unavoidable detours as soon as they happen.

If you're good at course correcting, being wrong may be less costly than you think, whereas being slow is going to be expensive for sure. - Jeff Bezos

How to get an impact estimate. Track your estimated impact and confidence in each experiment you plan to run in a table. Then multiply the impact by the confidence to estimate the expected impact.

Expected Impact = ∑ (Impact × Confidence)
ExperimentImpactConfidenceExpected Impact
Feature #10.5%50%0.25%
Feature #21.0%80%0.80%
Feature #31.0%25%0.25%
Total--1.30%

Use base rates to improve your estimates.

  • Impact: calculate the median impact of previously successful tests.
  • Confidence: calculate your hit rate. Of all the hypotheses you tested with experiments, what percent launched and drove a stat sig win in your goal metric?

Adjust up or down from the base rate using your intuition, which is informed by experience as well as a synthesis of all the insight at hand, including prior experiment results, research, and analyses. Don't worry about being precise. This exercise is about order of magnitude thinking — it's more important to be approximately right than exactly wrong.