How to Set a Goal Target with a Monte Carlo Simulation

1. Find your impact base rate. To get a sense of how much we can move our success metric, we first need to understand how much we have it moved it in the past. I recommend recording all statistically significant send wins in a results tracker. If you’re new to your team, try compiling previous wins from other PMs. Don’t just look at what your team has done. Also look at what’s been successful across your company and your industry.

2. Estimate the impact of new features. List out all of the features on your roadmap for the next half and estimate the impact of each. Use your base-rate as a starting point and adjust up or down by how confident you are in each feature. Your aim is accuracy.

Feature       Est. Impact     
Feature #1          1.0%
Feature #2          4.5%
Feature #3          2.0%
Feature #4          2.0%
Feature #5          0.5%
Feature #6          1.5%

Note: if you have a large team, consider breaking down your analysis by themes instead of features.

Step 3. Record your confidence in each feature. In addition to estimating the impact of each feature, you’ll need to estimate your confidence that the feature will launch and generate the impact you expect. Once again your aim is accuracy. If your confidence is 20%, that means only 1 out of 5 features you mark with 20% confidence should work. If too few of those 20% bets work out, you’re overconfident. And if too many work out you might be sandbagging.

Feature       Est. Impact   Confidence  
Feature #1          1.0%    80%
Feature #2          4.5%    20%
Feature #3          2.0%    60%
Feature #4          2.0%    90%
Feature #5          0.5%    50%
Feature #6          1.5%    30%

Step 4. Run a simulation. As a final step, run a bunch of simulations using your estimated impact and confidence to create a distribution of possible outcomes.

First start R, install the tidyverse package, and create a dataframe containing your impact and confidence estimations.

require(tidyverse)

dat <- tribble(
  ~feature, ~impact, ~confidence,
  #--------|--------|------------
  "feature1", 1.0, 0.80,
  "feature2", 4.5, 0.60,
  "feature3", 2.0, 0.60,
  "feature4", 2.0, 0.90,
  "feature5", 0.5, 0.50,
  "feature6", 1.5, 0.30
)

Then run 10,000 simulations by copying and pasting the following code.

# Returns a 50/50 goal target
get_target <- function(df, p = 0.5, trials = 1e4) {
  simulations <- tibble(rep(NA, trials))

  for (row in 1:nrow(df)) {
    impact <- df$impact[row]
    confidence <- df$confidence[row]

    flips <- sample(
      c(0,1),
      size = trials,
      replace = TRUE,
      prob = c(1 - confidence, confidence)
    )

    simulations[row] <- flips * impact
  }

  target <- rowSums(simulations) %>%
    quantile(p)
  return(target)
}

# Get 50/50 goal target
dat %>% get_target(0.5)

When we plot the distribution generated by these simulations and pick the 50/50 cutoff, we get a target of +7.5%.

At the end of each half, compare how your predictions fared against your experimental results. Instead of looking at the aggregate results, dig into feature by feature predictions and outcomes.

You may find that all of the feature wins came from a single product theme. Or you may find that you’re overconfident about your own feature ideas but your new Android engineer has a killer sense for impact. There’s no better way to improve your product sense than to evaluate how your product predictions turned out.