# Tag Archives: Pipeline

## Using Triangulation Forecasts For Improved Forecast Accuracy and Better Conversations

Ever been in this meeting?

CEO:  What’s the forecast?
CRO:  Same as before, \$3,400K.
Director 1:  How do you feel about it?
CRO:  Good.
Director 2:  Where will we really land?
CRO:  \$3,400K.  That’s why that’s the forecast.
Director 1:  But best case, where do we land?
CRO:  Best case, \$3,800K.
Director 2:  How do you define best case?
CRO:  If the stars align.

Not very productive, is it?

I’ve already blogged about one way to solve this problem:  encouraging your CRO think probabilistically about the forecast.  But that’s a big ask.  It’s not easy to change how sales leaders think, and it’s not always the right time to ask.  So, somewhat independent of that, in this series I’ll introduce three concepts that help ensure that we have better conversations about the forecast and ultimately forecast better as a result:  triangulation forecasts, to-go pipeline coverage, and this/next/all-quarter pipeline analysis.  In this post, we’ll cover triangulation forecasts.

Triangulation Forecasts

The simplest way to have better conversations about the forecast is to have more than one forecast to discuss.  Towards that end, much as we might take three or four bearings to triangulate our position when we’re lost in the backcountry, let’s look at three or four bearings to triangulate our position on the new annual recurring revenue (ARR) forecast for the quarter.

In this example [1] we track the forecast and its evolution along with some important context such as the plan and our actuals from the previous and year-ago quarters.  We’ve placed the New ARR forecast in its leaky bucket context [2], in bold so it stands out.  Just scanning across the New ARR row, we can see a few things:

• We sold \$3,000K in New ARR last quarter, \$2,850K last year, and the plan for this quarter is \$3,900K.
• The CRO is currently forecasting \$3,400K, or 87% of the New ARR plan.  This is not great.
• The CRO’s forecast has been on a steady decline since week 3, from its high of \$3,800K.  This makes me nervous.
• The CRO is likely pressuring the VP of Customer Success to cut the churn forecast to protect Net New ARR [3].
• Our growth is well below planned growth of 37% and decelerating [4].

I’m always impressed with how much information you can extract from that top block alone if you’re used to looking at it.  But can we make it better?  Can we enable much more interesting conversations?  Yes.  Look at the second block, which includes four rows:

• The sum of the sales reps’ forecasts [5]
• The sum of the sales managers’ forecasts [6]
• The stage-weighted expected value (EV) of the pipeline [7] [8]
• The forecast category-weighted expected value of the pipeline [9]

Each of these tells you something different.

• The rep-level forecast tells you what you’d sell if every rep hit their current forecast.  It tends to be optimistic, as reps tend to be optimistic.
• The manager-level forecast tells you how much we’d sell if every CRO direct report hit their forecast.  This tends to be the most accurate [10] in my experience.
• The stage-weighted expected value tells you the value of pipeline when weighted by probabilities assigned to each stage. A \$1M pipeline consisting of 10 stage 2 \$100K oppties has a much lower EV than a \$1M pipeline with 10 stage 5 \$100K oppties — even though they are both “\$1M pipelines.”
• The forecast category-weighted expected value tells you the value of pipeline when weighted by probabilities assigned to each forecast category, such as commit, forecast, or upside.

These triangulation forecasts provide different bearings that can help you understand your pipeline better, know where to focus your efforts, and improve the accuracy of predicting where you’ll land.

For example, if the rep- and manager-level forecasts are well below the CRO’s, it’s often because the CRO knows about some big deal they can pull forward to make up any gap.  Or, more sinisterly, because the CRO’s expense budget is automatically cut to preserve a target operating margin and thus they are choosing to be “upside down” rather face an immediate expense cut [11].

If the stage-weighted forecast is much lower than the others, it indicates that while we may have the right volume of pipeline that it’s not far enough along in its evolution, and ergo we should focus on velocity.

Now, looking at our sample data, let’s make some observations about the state of the quarter at SaaSCo.

• The reps are calling \$3,400K vs. a \$3,900K plan and their aggregate forecast has been fairly consistently deteriorating.  Not good.
• The managers, who we might notice called last quarter nearly perfectly (\$2,975K vs. \$3,000K) have pretty consistently been calling \$3,000K, or \$900K below plan.  Worrisome.
• The stage-weighted EV was pessimistic last quarter (\$2,500K vs. \$3,000K) and may need updated probabilities.  That said, it’s been consistently predicting around \$2,600K which, if it’s 20% low (like it was last quarter), it suggests a result of \$3,240K [12].
• The forecast category-weighted expected value, which was a perfect predictor last quarter, is calling \$2,950K.  Note that it’s jumped up from earlier in the quarter, which we’ll get to later.

Just by these numbers, if I were running SaaSCo I’d be thinking that we’re going to land between \$2,800K and \$3,200K [13].  But remember our goal here:  to have better conversations about the forecast.  What questions might I ask the CRO looking at this data?

• Why are you upside-down relative to your manager’s forecast?
• In other quarters was the manager-level forecast the most accurate, and if so, why you are not heeding it better now?
• Why is the stage-weighted forecast calling such a low number?
• What’s happened since week 5 such that the reps have dropped their aggregate forecast by over \$600K?
• Why is the churn forecast going down?  Was it too high to begin with, are we getting positive information on deals, or are we pressuring Customer Success to help close the gap?
• What big/lumpy deals are in these numbers that could lead to large positive or negative surprises?
• Why has your forecast been moving so much across the quarter?  Just 5 weeks ago you were calling \$3,800K and how you’re calling \$3,400K and headed in the wrong direction?
• Have you cut your forecast sufficiently to handle additional bad news, or should I expect it to go down again next week?
• If so, why are you not following the fairly standard rule that when you must cut your forecast you cut it deeply enough so your next move is up?  You’ve broken that rule four times this quarter.

In our next post in the series we’ll discuss to-go pipeline coverage.  A link to the spreadsheet used to the example is here.

# # #

Notes

[1] This is the top of the weekly sheet I recommend CEOs to start their weekly staff meeting.

[2] A SaaS company is conceptualized as a leaky bucket of ARR.

[3] I cheated and look one row down to see the churn forecast was \$500K in weeks 1-6 and only started coming down (i.e., improving) as the CRO continued to cut their New ARR forecast.  This makes me suspicious, particularly if the VP of Customer Success reports to the CRO.

[4] I cheated and looked one row up to see starting ARR growing at 58% which is not going to sustain if New ARR is only growing at ~20%.  I also had to calculate planned growth (3900/2850 = 1.37) as it’s not done for me on the sheet.

[5] Assumes a world where managers do not forecast for their reps and/or otherwise cajole reps into forecasting what the manager thinks is appropriate, instead preferring for managers to make their own forecast, loosely coupling rep-level and the manager-level forecasts.

[6]  Typically, the sum of the forecasts from the CRO’s direct reports.  An equally, if perhaps not more, interesting measure would be the sum of the first-line managers’ forecasts.

[7] Expected value is math-speak for probability * value.  For example, if we had one \$100K oppty with a 20% close probability, then its expected value would be \$100K * 0.2 = \$20K.

[8] A stage-weighted expected value of the (current quarter) pipeline is calculated by summing the expected value of each opportunity in the pipeline, using probabilities assigned to each stage.  For example, if we had only three stages (e.g., prospect, short-list, and vendor of choice) and assigned a probability to each (e.g., 10%, 30%, 70%) and then multiplied the new ARR value of each oppty by its corresponding probability and summed them, then we would have the stage-weighted expected value of the pipeline.  Note that in a more advanced world those probabilities are week-specific (and, due to quarterly seasonality, maybe week-within-quarter specific) but we’ll ignore that here for now.  Typically, one way I sidestep some of that hassle is to focus my quarterly analytics by snapshotting week 3, creating in effect week 3 conversion rates which I know will work better earlier in the quarter than later.  In the real world, these are often eyeballed initially and then calculated from regressions later on — i.e., in the last 8 quarters, what % of week 3, stage 2 oppties closed?

[9]  The forecast category-weighted expected value of the pipeline is the same the stage-weighted one, except instead of using stage we use forecast category as the basis for the calculation.  For example, if we have forecast categories of upside, forecast, commit we might assign probabilities of 0.3, 0.7, and 0.9 to each oppty in that respective category.

[10] Sometimes embarrassingly so for the CRO whose forecast thus ends up a mathematical negative value-add!

[11] This is not a great practice IMHO and thus CEOs should not inadvertently incent inflated forecasts by hard-coding expense cuts to the forecast.

[12] The point being there are two ways to fix this problem.  One is to revise the probabilities via regression.  The other is to apply a correction factor to the calculated result.  (Methods with consistent errors are good predictors that are just miscalibrated.)

[13]  In what I’d consider a 80% confidence interval — i.e., 10% chance we’re below \$2,800K and 10% chance we’re above \$3,200K.

## Detecting and Eliminating the Rolling Hairballs in your Sales Pipeline

Quick:  what’s the biggest deal in this quarter’s sales pipeline?  Was that the biggest deal in last quarter’s pipeline?  How about the quarter before?  Do you have deals in your pipeline older than your children?

If you’re answering yes to these questions, then you’re probably dealing with “rolling hairballs” in your pipeline.  Rolling hairballs are bad:

• They exaggerate the size of the pipeline.
• They distort coverage and conversion ratios.
• They mess up expected-value forecasts, like a forecast-category or stage-weighted sales forecast.

Maybe they’re real deals; maybe they’re figments of a rep’s imagination.  But, if you’re not careful, they pollute your pipeline and your metrics.

Let’s define a rolling hairball

A rolling hairball is a typically large opportunity that sits in your current-quarter pipeline every quarter, with a close date that slips every quarter.  At 2 quarters it’s a suspected rolling hairball; at 3 or more quarters it’s a confirmed one.

### Rolling Hairball Detection

The first thing you need to do is find rolling hairballs.  They’re tricky because salesreps always swear they’re real deals that are supposed to finally close this quarter.  What makes rolling hairballs obvious is their ever-sliding close dates.  What makes them dangerous is their size (including an accumulation of them that aggregate to a material fraction of the pipeline).

If you want to find rolling hairballs, look for opportunities in the current-quarter pipeline that were also in last-quarter’s pipeline.  That will find numerous bona fide slipped deals, but it will also light-up potential rolling hairballs.  To determine if an opportunity is  a rolling hairball, for sure, you can do one of two things:

• See if it also appeared in the current-quarter pipeline in any quarters prior to the previous one.
• Look at its stage or forecast category.  If either of those suggest it won’t be closing this quarter, it’s another big hairball indicator.

The more sophisticated way to find them is to examine “stuck opportunity” reports that light-up deals that are moving through pipeline stages too slowly compared to your norms.

But typically, the hairball is a big opportunity hiding in plain sight.  You know it was in last quarter’s pipeline and the quarter before that.  You’ve just been deluded into believing it’s not a hairball.

### Fixing Rolling Hairballs

There are two ways to fix rolling hairballs:

• Fix the close date.  Reps are subtly incented to put deals in the current quarter (e.g., to show they’re working on something, to show they might bring in some big sales this quarter). The manager needs to get on the phone with the customer and, after having verified it’s a real opportunity, get the real timeframe in which it might close.  Assigning a realistic close date to the opportunity makes your pipeline more real and reminds the rep that they need to be working on other shorter-term opportunities as well.  (There is no mid-term if you fail enough in the short term.)  The deal will still remain in the all-quarters pipeline, but it won’t always be in the current-quarter pipeline, ever-sliding, and distorting metrics and ratios.

• Fix the size. While a realistic close date is the best solution, what makes rolling hairballs dangerous is their size.  So, if the salesrep really believes it’s a current-quarter opportunity, you can either reduce its size or split it into two opportunities (particularly if that’s a possible outcome), a small one in the current quarter along with an upsell in the future.  Note that this approach can be dangerous, with lots of little hairball-lets flying below radar, so you should only try if it you’re sure your salesops team can produce the reports to find them and if you believe it reflects real customer buying patterns.

Don’t let rolling hairballs pollute your pipeline metrics and ratios.  Admit they exist, find them, and fix them.  Your sales and sales forecasting will be more consistent as a result.

## How to Train Your VP of Sales to Think About the Forecast

Imagine a board meeting.

Director:  What’s the forecast for new ARR this quarter?

Sales VP:  \$4.3M, with a best case of \$5.0M.

Director:  So what’s the most likely outcome?

Sales VP:  \$4.3M.

Director:  What are you really going to do?  (The classic newb trap question.)

Sales VP:  I think we can come in North of that.

Director:  What’s the worst case?

Sales VP:  \$3.5M.

Director:  What are the odds of coming in at or above the forecast?

Sales VP:  I always make my forecast.

Director:   What do you mean by worst case?

Sales VP:  You know, well, if the stars align in a bad way – a lot of stuff would have to go wrong – but if that happened, then we could end up at \$3.5M.

Director:  So, let’s say a 10% chance of being at/below the worst case?

Sales VP:  I’d say more like 5%.

Director:  What do you mean by best case?

Sales VP:  Well, if we really struck it rich and everything lined up just the way I wanted, that would be best case.

Director:  You mean if all the deals came in — so best case basically equals pipeline?

Sales VP:  No, that never happens, I’ve made about 10 scenarios of different deal closing combinations and in 2 of them I can get to the best case.

You see the problem?  Does it sound familiar?  Do you realize how much time we spend talking in board meetings about “forecast,” “best case,” and “worst case” without every discussing what we mean by those terms?

Do you see how this is compounded by the sales VP’s natural, intuitive view of the outcomes?  Do you see the obvious mathematical contradictions?  “I always make my forecast” says it’s a 100% number, but then the VP says it’s the “most likely” number which implies 50%.  Then the VP says there’s a 5% chance of coming in at/less than worst case (which is much lower) and then kind of implies that there’s a 20% chance of beating best case – but the 2 out of 10 is meaningless because it’s not a probability, it’s just a count of scenarios.  Nothing adds up.

The result is, if you’re not careful, the board ends up counting angels on pinheads.  What can we do to fix this?  It’s simple:  teach (and if need be, force) your sales VP to think probabilistically.  Ask him/her how often:

• It is reasonable to miss the forecast.  A typical answer might be 10%.
• It is likely to come in at/below the worst case? Typical answer, 5%.
• It is likely to meet/beat the best case? Typical answer, 20%.

So, with those three questions, we’ve now established that we want the sales VP to give us:

• A 90% number on being at/above the forecast
• A 20% number on being at/above the best case
• A 5% number on being at/below the worst case

Put differently, when the sales VP decides what number to forecast that they should be thinking:

• I should come in under my forecast once every 2.5 years (10 quarters).
• I should hit/beat the best case about once every 5 quarters (a bit less than once a year).
• I should come in/under the worst case once every 20 quarters (once every 5 years, or for most minds, basically never).

The beauty here is that when you work at a company a long time you can get enough quarters under your belt, to start really seeing how you’re doing relative to these frequencies.  What’s more, by converting the probabilities into frequencies (e.g., once every 10 quarters) you make it more intuitive for the sales VP and the organization to think this way.

In addition, you have a basis for conversations like this one which, among other things, is about overconfidence:

CEO:  You need to work on your forecasting.

Sales VP:  You know it’s hard out there, very competitive, and we don’t have much deal flow.  Back when I was at { Salesforce | Oracle | SAP }, I was much better at forecasting because we had more volume.

CEO:  But we agreed your forecast should be a 90% number and you’ve missed it 2 out of the past 4 quarters.

Sales VP:  Yes, but as I’ve said it’s tough to forecast in this market.

CEO:  Then forecast a lower number so you can beat it 90% of the time.  I’m asking you for a 90% number and empirically you’re giving me a 50% number.

Sales VP:  OK.

CEO:  Plus, when those two big deals slipped last quarter you didn’t drop your forecast, why?

Sales VP:  Because where I grew up, you don’t cut the forecast.  You try like crazy to hold it.  Do you know the morale problems it causes when I cut the forecast – especially if it’s below plan? So, yes, when those two deals slipped it added more risk to the forecast – and I told you and the board that — but I didn’t cut forecast, no.

CEO:  But “adding risk” here is meaningless.  In reality, “adding risk” means it’s not a 90% number anymore.  You’ve taken what was a 90% number and it’s now more like a 60% or 70% number.  So I want you to forget what they taught you growing up in sales and always – every week – give me a number that based on all available information you are 90% sure you can beat.  If that means dropping the forecast so be it.

This also helps with the board and the inevitable sandbagger issue.  In my experience (and with a bit of exaggeration) you always seem to be in one of two situations:  (1) intermittently missing plan and in trouble or (2) consistently making plan and a “sandbagger” – it feels like there’s nothing in between.

Well, if you establish with the board that your company forecast is a 90% number it means you are supposed to beat it 9 times out of 10 so you can only really be labelled a sandbagger when you’re 15 for 15 or 20 for 20.  It also reminds them that you’re supposed to arrive at the forecast so that you miss once every 10 quarters so they shouldn’t freak out if once every 2.5 years if that happens — it’s supposed to happen in this system.  (Just don’t let a once-in-ten-quarter event happen twice in a row.)

I like this quantitative basis for sales forecasting and I carry it down to the salesrep and pipeline level.  I believe that each “forecast category” should have a probability associated with it.  For example, at the opportunity level, you should link probabilities to categories, such as:

• Commit = 90%
• Forecast = 70%
• Upside = 30%

This, in turn, means that over time, a given salesrep should close 90% of their committed deals, 70% of their forecast deals, and 30% of their upside.  Deviations from this over time indicate that the rep is mis-categorizing the deals because the probability should be the basis for the forecast category assignment [1].

Finally, I do believe that salesreps should give quarterly forecasts [2] that reflect their sense for how things will come in given all the odd things that can happen to deals (e.g., size changes, acceleration, slippage).  I believe those forecasts should be a 70% number because the sales manager will be managing across a  portfolio of them and while there is little room for a company to miss at the VP of Sales level, there is more room for and more variance in performance across salesreps.

While I know this will not necessarily come naturally to all sales VPs — and some may push-back hard — this is a simple, practical, and rigorous way to think about the forecast.

# # #

[1] Some people do this through an independent (orthogonal) field in the CRM system called probability.  I think that’s unnecessary because in my mind forecast category should effectively equal probability and your options for picking a probability should be bucketed.  No one can say a deal is 43% vs. 52% and forecast category doesn’t indicate some probability of closing, then … what use is it and on what basis should you classify something as forecast vs. upside?

[2] Some people believe that only managers should make forecasts, but I believe both reps and managers should forecast for two reasons:  (1) provided it’s left independent and not “managed” by the managers, the aggregated salesrep-level forecast provides another, Wisdom of Crowds-y, view into the sales forecast and (2) it’s never too early to teach salesreps how to forecast which is best learned through the experience of trial and error over many quarters.

## The Evolution of Marketing Thanks to SaaS

I was talking with my friend Tracy Eiler, author of Aligned to Achieve, the other day and she showed me a chart that they were using at InsideView to segment customers.  The chart was a quadrant that mapped customers on two dimensions:  renewal rate and retention rate.  The idea was to use the chart to plot customers and then identify patterns (e.g., industries) so marketing could identify the best overall customers in terms of lifetime value as the mechanism for deciding marketing segmentation and targeting.

Here’s what it looked like:

While I think it’s a great chart, what really struck me was the thinking behind it and how that thinking reflects a dramatic evolution in the role of marketing across my career.

• Back two decades ago when marketing was measured by leads, they focused on how to cost-effectively generate leads, looking at response rates for various campaigns.
• Back a decade ago when marketing was measured by opportunities (or pipeline), they focused on how to cost-effectively generate opportunities, looking at response and opportunity conversion rates.
• Today, as more and more marketers are measured by marketing-sourced New ARR, they are focused on cost-effectively generating not just opportunities, but opportunities-that-close, looking all the way through the funnel to close rates.
• Tomorrow, as more marketers will be measured on the health of the overall ARR pool, they will be focused on cost-effectively generating not just opportunities-that-close but opportunities that turn into the best long-term customers. (This quadrant helps you do just that.)

As a company makes this progression, marketing becomes increasingly strategic, evolving in mentality with each step.

• Starting with, “what sign will attract the most people?” (Including “Free Beer Here” which has been used at more than one conference.)
• To “what messages aimed at which targets will attract the kind of people who end up evaluating?”
• To “who are we really looking to sell to — which people end up buying the most and the most easily – and what messages aimed at which targets will attract them?”
• To “what are the characteristics of our most successful customers and how can we find more people like them?”

The whole pattern reminds me of the famous Hubspot story where the marketing team was a key part forcing the company to focus on either “Owner Ollie” (the owner of a <10 person business) or “Manager Mary” (a marketer at a 10 to 1000 person business).  For years they had been serving both masters poorly and by focusing on Manager Mary they were able to drive a huge increase in their numbers that enabled cost-effectively scaling the business and propelling them onto a successful IPO.

What kind of CMO does any CEO want on their team?  That kind.  The kind worried about the whole business and looking at it holistically and analytically.